The GPT-2 decoder family: a tied token embedding plus a learned position
embedding, then pre-LayerNorm blocks (LayerNorm with gain and shift;
multi-head causal attention through one fused, biased Q/K/V projection
and a biased output projection; a biased feed-forward expansion, the
cubic-tanh GELU and a biased reduction), a final LayerNorm, and logits
through the tied embedding. Radford et al. 2019 and the released
openai-community/gpt2 config.json fix the instance (Model.GPT2).
`vocabulary` is the stored class count, `tokens` the classes that exist:
a realization may pad the vocabulary to a multiple its objective needs;
padded classes never occur as inputs or targets and a realization must
hold their probabilities at zero.
22field unrestricted learnedPositionDecoderLayers : NatThe compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.