Source/Packages

Model.LearnedPositionDecoder

packages/components/src/Model/LearnedPositionDecoder.alpha

233 lines48 declarations12.8 KiBSHA-256 58c267358560

field · lines 25–25

learnedPositionDecoderHeadWidth

Full file
The GPT-2 decoder family: a tied token embedding plus a learned position embedding, then pre-LayerNorm blocks (LayerNorm with gain and shift; multi-head causal attention through one fused, biased Q/K/V projection and a biased output projection; a biased feed-forward expansion, the cubic-tanh GELU and a biased reduction), a final LayerNorm, and logits through the tied embedding. Radford et al. 2019 and the released openai-community/gpt2 config.json fix the instance (Model.GPT2). `vocabulary` is the stored class count, `tokens` the classes that exist: a realization may pad the vocabulary to a multiple its objective needs; padded classes never occur as inputs or targets and a realization must hold their probabilities at zero.
25field unrestricted learnedPositionDecoderHeadWidth : Nat

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.