Source/Packages

Model.LearnedPositionDecoder

packages/components/src/Model/LearnedPositionDecoder.alpha

233 lines48 declarations12.8 KiBSHA-256 58c267358560

constructor · lines 18–18

LearnedPositionDecoderGeometryValue

Full file
The GPT-2 decoder family: a tied token embedding plus a learned position embedding, then pre-LayerNorm blocks (LayerNorm with gain and shift; multi-head causal attention through one fused, biased Q/K/V projection and a biased output projection; a biased feed-forward expansion, the cubic-tanh GELU and a biased reduction), a final LayerNorm, and logits through the tied embedding. Radford et al. 2019 and the released openai-community/gpt2 config.json fix the instance (Model.GPT2). `vocabulary` is the stored class count, `tokens` the classes that exist: a realization may pad the vocabulary to a multiple its objective needs; padded classes never occur as inputs or targets and a realization must hold their probabilities at zero.
18constructor LearnedPositionDecoderGeometryValue

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.