All training runs / GPT-2 small / Chat
Alpha GPT-2 small, in conversation
124,439,808 parameters · pretrained on OpenWebText, fine-tuned on SmolTalk conversations, then on hand-written conversations in the spirit of Nietzsche · served for now by a CPU stand-in, not Alpha
Stand-in, not Alpha's predictor. Alpha's chat runs on this server's CPU so its RTX 3070 can train GPT-2. Replies come from the same checkpoints, run by the independent PyTorch reference model Alpha's GPT-2 runs are checked against, not by Alpha's compiled predictor. Asked again with greedy decoding, it gave 16 of the 21 replies the card had given token for token (verify-greedy.json). (PyTorch 2.8.0 on the lsalpha.com server's CPU.)
System prompt
Ask anything. Alpha is small (124M parameters), so expect short, confident and often wrong answers. It answers in the spirit of Nietzsche, and each reply is sampled, so asking again gives another.
How it was made
The same Alpha GPT-2 small as the OpenWebText run, continued on chat data under its own checkpoint contract.
- Pretraining
- One pass over OpenWebText: 9.03B tokens in 441 invocations on an RTX 3070. Its final checkpoint (update 2,205,000) is the “pretrained base” above.
- Chat fine-tune
- HuggingFaceTB/smol-smoltalk@f73fe857 (2 epochs): the SmolTalk mixture Hugging Face assembled for models under 1B parameters (SmolLM2-135M/360M-Instruct). 460,341 conversations, 364M tokens once rendered, two passes in 37 invocations. The result is the “SmolTalk only” model above.
- Persona fine-tune
- ajaxdavis/alpha-nietzsche-smoltalk@9f4a939a (v4, 58,950 conversations) mixed with HuggingFaceTB/smol-smoltalk@f73fe857: 58,950 conversations written by hand by Claude in which the assistant is Alpha, names itself correctly, describes how it was made, and answers in the spirit of Nietzsche, across advice, philosophy, science, history, math, code, writing and more. The first three invocations trained on the dataset's version 2 (9,942 conversations, two passes each), the next three on version 3 (21,546, one pass each), the run then continued on version 4, one pass per invocation over its 56,026 training conversations, and it finished with three invocations of supervised fine-tuning on the same mixture (Recipe, below). Every invocation mixes them with SmolTalk, so the voice arrives without losing general chat. 14 invocations, 286.72M tokens.
- Chat format
- GPT-2 has no chat tokens, so roles are plain text:
System: …,User: …,Assistant: …, with<|endoftext|>closing every reply, which is where the predictor stops. Conversations longer than the 1,024-token context are split at turn boundaries and packed whole into windows. - Recipe
- The fine-tunes keep the pretraining recipe (AdamW at a constant 2.4e-4, weight decay 0.1, clipping at 1.0) and train on every token, as in nanochat's mid-training, until the last stage: supervised fine-tuning, in which only the assistant's replies are trained on. Each reply token's word carries a flag; a kernel on the GPU clears the flags and weighs every flagged target by one over the pass's flagged targets, so the loss is the mean over the replies, and prompts and padding add nothing to it or to the gradient. The optimizer's moments start again, and the rate warms up over 556 updates to 2.4e-4, then follows a cosine to 0 over 15,000. Each stage has its own checkpoint contract; an adoption step carries the previous stage's checkpoint across, so no stage's executables accept another stage's checkpoints.
- Held-out loss
- Nats per token on the assistant replies of conversations never trained on, measured by the independent PyTorch reference. SmolTalk test: pretrained 3.836, SmolTalk only 2.204, Alpha 2.005. Alpha's persona test (the 2,924 held-out conversations of version 4): pretrained 5.201, SmolTalk only 5.335, Alpha 3.702.
- Decoding
- Sampled on the GPU by Alpha's own kernels: the logits of tokens already in the reply lose a frequency and presence penalty, tokens whose probability is below a fraction (min-p) of the most likely one are excluded, and the token is the largest of the logits over the temperature plus Gumbel noise drawn from a counter hash of the seed and step (the Gumbel-max draw from the softmax). While the stand-in serves, PyTorch computes the same decoding from its specification on the CPU, so a reply can differ from the one the card would give. The page samples at temperature 0.7, min-p 0.1 and frequency penalty 0.5; Greedy picks the most likely token every step, so the same message always gets the same reply. Up to 256 tokens, from a prompt of at most 768 tokens; long conversations keep their newest exchanges.
- Executables
- predictor
6200c70b29434629(not running while the stand-in serves) · checkpoint step-2460000.ckpt (update 2,460,000, invocation 492)
Supervised fine-tune loss
Cross-entropy per assistant-reply token over each update's four windows, before the update: the loss the fine-tune minimises. Prompts and padding carry none. The training conversations are seen once per invocation, so this falls below the held-out figures above.