Source/Systems

GPT2.Model

systems/gpt2/src/GPT2/Model.alpha

75 lines16 declarations3.3 KiBSHA-256 de41e63ea477

def · lines 21–21

gpt2TokenVocabulary

Full file
GPT-2 small, openai-community/gpt2 config.json (Radford et al. 2019): vocabulary 50257 (byte-level BPE, <|endoftext|> = 50256), context 1024, 12 blocks of width 768, 12 heads of 64, feed-forward width 3072 (n_inner = 4 n_embd), gelu_new, LayerNorm epsilon 1e-5, tied input and output embeddings, learned positions, biases on every linear map. The weights start fresh; the released checkpoint is not used. The stored vocabulary is 50432 = 197 x 256, the multiple of 256 the exact objective's row kernels need. The 175 padded classes are never inputs or targets and their logits are filled with -1e30 before the objective (Realization.Nvidia.SM86.ColumnFillSM86), so the model is GPT-2's.
21def gpt2TokenVocabulary : Nat = 50257

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.