GPT-2
GPT-2 small (124M): learned positions, LayerNorm, GELU, tied embeddings, trained from fresh weights. The same system states nanochat's d24 (1.38B parameters: value embeddings, sliding windows, ReLU², softcap, Muon and AdamW, FP8) as GPT2.Speedrun.
- Runs on
- x86-64 host → sm_86 (RTX 3070) and sm_90 (H100 SXM)
- Status
- GPT-2 small trained on 9.03B tokens of OpenWebText on the RTX 3070, then on llm.c's FineWeb recipe; on one H100 an update takes a median 11.01 ms against llm.c's 11.37. The d24 speedrun trained the record's full schedule on two H100s (5,568 steps); its CORE of 0.2089 is still short of the record's 0.2626.
- Namespaces
- GPT2 · GPT2.Build · GPT2.Speedrun · GPT2.SpeedrunBuild