0.02 / sqrt(24) / sqrt(65535) = 1 / sqrt(2500 * 24 * 65535)
46def gpt2ResidualInitializationScaleWord : Nat =
47 (pipe (nest naturalMultiply
48 (naturalMultiply
49 gpt2InitializationDeviationDenominator
50 gpt2InitializationDeviationDenominator)
51 2 (learnedPositionDecoderLayersOf gpt2Geometry))
52 (naturalMultiply 65535)
53 float32ExactInverseRoot
54 compile-time)The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.