GPT-2 initializes every weight from N(0, 0.02^2), the residual output
projections (attention output and feed reduction, c_proj) from
N(0, (0.02 / sqrt(2 layers))^2), every bias at zero and every LayerNorm's
gain at one and shift at zero. The device normal image uses twelve
uniform bytes centred at zero (variance 65535): its scale is the
deviation divided by sqrt(65535). This is a CLT approximation to a
Gaussian, not PyTorch's RNG stream.
38def gpt2InitializationDeviationDenominator : Nat = 50The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.