---- the parameter layout ----
Element offsets, independent of F32/BF16 storage, in two contiguous
regions so a learner can treat them differently (GPT-2 practice decays
the matrices and embeddings, not the biases and LayerNorm parameters):
matrices: the token embedding [vocabulary][width], the position
embedding [context][width], then per block the Q/K/V weight
[3 width][width], the attention output weight [width][width], the
expansion weight [feed][width] and the reduction weight [width][feed];
vectors: per block the attention LayerNorm's gain and shift, the Q/K/V
bias, the output bias, the feed LayerNorm's gain and shift, the
expansion bias and the reduction bias; then the final LayerNorm's gain
and shift.
A weight is stored [output][input], as the forward product reads it.
103def learnedPositionDecoderEmbeddingBase : Nat = 0The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.