Training

Models training on real hardware.

Each dashboard reads the run's own records: loss at every step, predictions, where each step's device time goes, optimizer internals, and GPU and host telemetry. The executables come from the Alpha compiler alone, with no CUDA runtime in the path.

Ran on the GPU

AlphaOS on two RTX 3070s

two rented RTX 3070s (RunPod) · sm_86 · one launch per GPU

An operating system compiled by Alpha, resident on each GPU: dataflow tasks, programs loaded while it runs, and Midget training on one card and on two, with every host command recorded.

Open run record →
Finished

nanochat d24 speedrun on H100s

rented H100 SXMs (RunPod) · sm_90 · ClimbMix, FP8

Karpathy's record recipe (1.38B parameters, GPT-2 grade), every step trained by an Alpha-compiled trainer across the pod's cards. 5,568 steps on 2 H100s.

Open live dashboard →
Stopped

GPT-2 small on the RTX 3070

a rented RTX 3070 (RunPod) · sm_86 · OpenWebText, then FineWeb

The newest snapshot is 49671 minutes old.

Open dashboard →

Milestones

Recorded results

Each links to the run record it summarizes.

  1. GPT-2 trained as AlphaOS tasks

    A GPT-2-small-shaped model trained from scratch as resident AlphaOS tasks rather than kernel launches, with bit-exact arithmetic: on an RTX 3070, then to 30,000 updates on an RTX 3090 (held-out loss 4.77). The knots research runs on it.

  2. AlphaOS runs on two RTX 3070s

    A GPU-resident operating system compiled by Alpha boots once per card, schedules dataflow tasks, installs programs while it runs and trains Midget on one card and on two. The first runs found a predicate-latency bug and a reserved-register hang, both now fixed.

  3. nanochat d24 trained end to end on two H100s

    The record's full schedule, 5,568 steps in 6 h 32 min at 247,865 tokens a second, every step trained by an Alpha-compiled trainer across two NVLinked H100s. CORE 0.2089 against the record's 0.2626: the schedule, data and kernels are reproduced, the result not yet.

  4. GPT-2 small on Alpha's own Hopper kernels

    GPT-2 124M on one H100 through the sm_90 back end: a median 11.01 ms an update over four alternating rounds, against 11.37 ms for llm.c with cuDNN attention, on the same model, data, start and recipe.

  5. GPT-2 small scored in bits per byte

    Every retained checkpoint of Alpha's GPT-2 small scored on the same 18.5 MB of held-out OpenWebText on the RTX 3070: 1.396 bits per byte after 9B tokens at a constant learning rate, 1.223 after llm.c's recipe on FineWeb.

  6. GPT-2 small trains on the RTX 3070

    9.03 billion tokens of OpenWebText through Alpha's sm_86 trainer at 36,000 tokens a second, then llm.c's 124M recipe on FineWeb, from the same weights.