GPT-2 small, openai-community/gpt2 config.json (Radford et al. 2019):
vocabulary 50257 (byte-level BPE, <|endoftext|> = 50256), context 1024,
12 blocks of width 768, 12 heads of 64, feed-forward width 3072
(n_inner = 4 n_embd), gelu_new, LayerNorm epsilon 1e-5, tied input and
output embeddings, learned positions, biases on every linear map.
The weights start fresh; the released checkpoint is not used.
The stored vocabulary is 50432 = 197 x 256, the multiple of 256 the exact
objective's row kernels need. The 175 padded classes are never inputs or
targets and their logits are filled with -1e30 before the objective
(Realization.Nvidia.SM86.ColumnFillSM86), so the model is GPT-2's.
21def gpt2TokenVocabulary : Nat = 50257The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.