Source/Systems

GPT2.Model

systems/gpt2/src/GPT2/Model.alpha

75 lines16 declarations3.3 KiBSHA-256 de41e63ea477

def · lines 38–38

gpt2InitializationDeviationDenominator

Full file
GPT-2 initializes every weight from N(0, 0.02^2), the residual output projections (attention output and feed reduction, c_proj) from N(0, (0.02 / sqrt(2 layers))^2), every bias at zero and every LayerNorm's gain at one and shift at zero. The device normal image uses twelve uniform bytes centred at zero (variance 65535): its scale is the deviation divided by sqrt(65535). This is a CLT approximation to a Gaussian, not PyTorch's RNG stream.
38def gpt2InitializationDeviationDenominator : Nat = 50

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.