GPT-2 small Training monitor

Training loss is the mean of an invocation's last 2,000 windows (for the running one, of its newest 2,000). The first five invocations ran on another pod, whose run directory was later deleted; their timings, losses and GPU samples survive in the controller's log and a copy of their monitor files.

GPU utilization

100%

sampled 15 Oct 01:36 UTC

Power draw

219 W

every 30 s by nvidia-smi

Temperature

71 °C

memory 5.04 GiB of 8 GiB

This run so far

49.2 h

10.70 kWh at the card · about $6.40 of rent

The current invocation's GPU samples appear here while it runs.

1. 6 Oct 10:06The OpenWebText run starts from fresh weights on one RTX 3070: one 1,024-token window an update at a learning rate of 6e-5.
2. 1.37B tokensInvocation 67: the adoption moves the run to four windows an update at 2.4e-4, keeping its weights and AdamW state. Loss drops 0.2 nats in the next 40M tokens, after 0.7B tokens flat at 4.81.
3. 2.49B tokensThe Chinchilla point, 20 tokens a parameter. Training loss is about 4.33, and barely moves after.
4. 10 Oct 07:21One full pass over OpenWebText: 9.03B tokens, 441 invocations, training loss about 4.29, HellaSwag 25.9% (chance is 25%).
5. 10 OctThe plateau is diagnosed as the recipe's, not a bug (below). The next run continues the same weights under llm.c's GPT-2 124M recipe on FineWeb.
6. 10 Oct 23:27The FineWeb recipe begins at invocation 441. After five invocations, validation loss is 3.617 (from 5.087) and held-out OpenWebText loss 3.746 (from 4.278).
7. 11 Oct 04:44Stopped 12 seconds into invocation 446; the card goes to another experiment and the pod's run directory is later deleted, all but the last checkpoint.
8. 13 Oct 03:08Resumed on its own pod, after showing every input is the original: the checkpoint's SHA-256, the executables' pins, FineWeb tokenized again with two invocations' inputs byte for byte the recorded ones, and the checkpoint's validation loss reproduced exactly (3.617338).
9. 16 Oct 00:52Expected end: invocation 517, update 18,865 past the origin, the learning rate at 0.

• Each update was mostly noise. The gradient noise scale (the batch size at which signal and noise are equal) measured 51,000 ± 7,000 tokens at 2.6B tokens and 63,000 ± 31,000 at the end, about 13 times the 4,096-token update. Only 6–7% of each update's direction was signal.
• The learning rate never decayed. At a constant rate, a noisy optimizer settles into a band around the minimum rather than into it. The run reached that band after 3–4B tokens and spent the remaining 5B tokens, 55% of its compute, moving around it: 6.5B tokens improved the loss on 32 fixed windows by 0.048 nats.
• Its own optimizer state agrees. Gradient clipping fired on every update, successive updates were slightly anti-correlated, and Adam's per-coordinate signal-to-noise matched pure noise (0.15–0.19 against 0.183): the signature of an iterate bouncing at a constant rate's floor.
• Not a bug. Alpha's arithmetic had already been checked against the reference (below), and the adoption to four-window updates was itself a controlled experiment: the same model on the same data dropped 0.2 nats as soon as each update got less noisy.
• The fix is llm.c's recipe. 524,288-token updates, 128 times larger and about 9 times the noise scale, a peak rate 2.5 times higher that such updates allow, and a cosine decay to zero.

Architecture

12 blocks × 768 wide, 12 heads of 64, feed-forward 3,072, context 1,024

Blocks

pre-LayerNorm, gain and shift, epsilon 1e-5; gelu_new (cubic tanh); biases on every linear map; learned positions; tied input and output embeddings

Vocabulary

50,257 byte-level BPE tokens, stored as 50,432 for the objective's multiple of 256; the padded classes' logits are held at −10³⁰, so loss and gradients are the unpadded model's

Parameters

124,439,808 (124,574,208 stored with the padded vocabulary)

OpenWebText trainer

On the four-window recipe's first update: loss 10.89689 against the reference's 10.89694; all 148 parameter tensors' gradients within 0.28% (every cosine at least 0.99999); AdamW on the native moments reproduces the native weights to 0.012%. Teacher-forced generation matches the reference's argmax at every token (511 of 512 exactly, one bfloat16 near-tie).

FineWeb recipe, from scratch

Two updates of the recipe from fresh weights: all 148 tensors within 0.29% of the reference, cosines at least 0.999996.

The continuation

The adopted checkpoint's weights are bit-identical to the OpenWebText run's last, both moment banks zero, the counts restated to 441 invocations of 245 updates.

Trained weights

On trained weights the device's gradients differ from the reference's by depth (1–2% at block 11 to tens of percent at blocks 0–1). The reference differs from itself about as much when only its products' summation changes from binary32 to binary64: bfloat16's inherent sensitivity on a heavily cancelling small-batch gradient, far below the sampling noise training absorbs. From fresh weights both agree within 0.3%.

HellaSwag scoring

On the GPU by Alpha's scoring executable, as llm.c's dev/data/hellaswag.py scores it; each position's loss on the first four examples agrees with the reference's to 0.03 nats on average.

The resume

Checkpoint SHA-256 cda20c33… as recorded; FineWeb tokenized again: 14,868,862 documents and 10,251,184,259 training tokens as before, invocations 441 and 446's inputs byte-identical to the recorded ones; HellaSwag's prepared input identical; the checkpoint's validation loss reproduced on the new card exactly (3.617338).

1. SmolTalkConversations from HuggingFaceTB/smol-smoltalk, from the OpenWebText checkpoint (update 2,390,000).
2. PersonaFour rounds of conversations in Alpha's voice, written for it (update 2,445,000).
3. Replies onlySupervised fine-tuning on the assistant's replies alone, with a warmup and a cosine decay (update 2,460,000): persona test loss 3.70 against the base model's 5.20, SmolTalk test loss 2.00.
4. NextThe same three stages on this run's final checkpoint, compared with the current chat model on the same held-out tests before it replaces it.

Chat with the current model → Its replies now come from a labelled CPU stand-in (PyTorch, the same checkpoints), so this card can train.