All training runs / nanochat d24 speedrun / H100
nanochat d24 speedrun Training monitor
1,384,122,122 parameters · Andrej Karpathy's record recipe, trained by a trainer Alpha compiled to sm_90 machine code with no CUDA, NCCL or PyTorch on the training path
nanochat's d24 model (24 layers, width 1,536, value embeddings, FP8 matrix products, the Muon and AdamW optimizers) trained from nanochat's own initialization on nanochat's own data (ClimbMix, nanochat's tokenizer and loader) for its record's whole schedule: 5,568 steps of 220 tokens, the run that beats GPT-2 1.6B on CORE. One uninterrupted invocation over every card of the pod, one rank a card. The weights go to nanochat's checkpoint format for CORE, midtraining and post-training.
- Progress
- 5,568 / 5,568 100.0% of the steps · 5.84B tokens
- Training loss
- 2.456 the latest step's mean · 10.398 at the first step
- Throughput
- 247,865 tok/s 4.230 s a step · nanochat on two H100s: 247,886 tok/s
- Training time
- 6 h 33 min 6 h 33 min for the whole run at this rate
- Expected finish
- Done 0 min of training left
- Rented GPU cost
- $90.81 2 H100 SXM at $7.98 an hour, since 22 Oct 09:51 UTC
Training loss
Live from the trainer's result file: each micro-batch's loss is written as its pass completes, so this curve is never more than a minute behind the cards.
Against nanochat
The leaderboard's time is nanochat's own total_training_time on Karpathy's eight H100s; this run's is the trainer's clock on 2 rented H100s. The gate the run is for is CORE ≥ 0.256525 (GPT-2 1.6B), judged by nanochat's own base_eval on the exported weights after the run.
| Run | Cards | Steps | Tokens/s | Training time |
|---|---|---|---|---|
| nanochat Run 6 (leaderboard record) | 8× H100 | 5,568 | 982,891 | 1 h 39 min |
| nanochat a825e63 on two H100s (measured) | 2× H100 | — | 247,886 | 6 h 33 min for the record's steps |
| Alpha, this run | 2× H100 | 5,568 of 5,568 | 247,865 | 6 h 33 min |
On the cards
nvidia-smi on the pod at the last observation.
| GPU | Utilization | Memory | Power | Temperature |
|---|---|---|---|---|
| 0 | 0% | 0 MiB | 72 W | 33 °C |
| 1 | 0% | 0 MiB | 70 W | 28 °C |
Events
The run script's own log on the pod (newest first): the cards it found, the data, the artifact builds, the run and the export.
- phase done
- export: meta_005568.json model_005568.pt
- image 5536488544 bytes; rank images equal
- phase export
- run rc=0 seconds=23555 result=11085816672
- phase run
- input: 52247434336 bytes
- phase input
- batches: wrote 89088 micro-batches e1b63c2ef1d02ac8e693886b4bebd6b1374631ce28f22bce7830fb1a520cdc52
- ranks-full.elf: artifact /workspace/full/ranks-full.elf (9870598 bytes, sha256 ee64abbb647f5989cbea04e5b94e233f1f6916338c5d39aca5340befcc7c3c16) from checkArtifact ·
- phase build
- schedule: 5568 steps, 5838 M tokens
- words-full.bin: artifact /workspace/full/words-full.bin (1603584 bytes, sha256 5ede382473a4dd234ea0280498bb4f3f8ea0cc9515a16d2e5c204104cfdb0bd9) from fullArtifact ·
- data: 170 shards downloaded, tokenizer trained
- phase data
- cards: 2 3 (world 2, 16 micro-batches a rank a step)
The recipe
- Model: nanochat d24 at
a825e63: 24 layers, width 1,536, 12 heads, context 2,048, vocabulary 32,768, value embeddings, backout and smear (the record's architecture). - Schedule: base_train's derived horizon at data:params ratio 8, 5,568 steps of 1,048,576 tokens, its warmup, warmdown and Muon momentum schedules computed in Alpha in nanochat's own double arithmetic.
- Precision: FP8 (e4m3) matrix products with nanochat's scaling, bfloat16 activations, binary32 masters; most kernels bit-identical to nanochat's compiled record step.
- Distribution: the optimizer state sharded across ranks as nanochat's DistMuonAdamW does; gradients reduced in rank order, each rank's owned update, then gathered (proved equal to the whole update in Lean).
- Source: tools/ranks-full.sh (the run), ops/speedrun-publish.mjs (this page's publisher) and the plan.