Source/Packages

Model.LearnedPositionDecoder

packages/components/src/Model/LearnedPositionDecoder.alpha

233 lines48 declarations12.8 KiBSHA-256 58c267358560

def · lines 103–103

learnedPositionDecoderEmbeddingBase

Full file
---- the parameter layout ---- Element offsets, independent of F32/BF16 storage, in two contiguous regions so a learner can treat them differently (GPT-2 practice decays the matrices and embeddings, not the biases and LayerNorm parameters): matrices: the token embedding [vocabulary][width], the position embedding [context][width], then per block the Q/K/V weight [3 width][width], the attention output weight [width][width], the expansion weight [feed][width] and the reduction weight [width][feed]; vectors: per block the attention LayerNorm's gain and shift, the Q/K/V bias, the output bias, the feed LayerNorm's gain and shift, the expansion bias and the reduction bias; then the final LayerNorm's gain and shift. A weight is stored [output][input], as the forward product reads it.
103def learnedPositionDecoderEmbeddingBase : Nat = 0

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.