All training runs / AlphaOS / 2 × RTX 3070

AlphaOS On the GPU

One launch per GPU · 46 nodes, one per SM · programs loaded while it runs · the host as its internet

AlphaOS is an operating system that Alpha compiles into one resident sm_86 program. The host launches it once per GPU. From then on the GPU's 46 nodes claim tasks from queues and dataflow graphs, install programs the host sends, and run them. The nodes talk to the host only through protocol words in shared memory. No CUDA, no kernels, no relaunch.

13 of 13 runs finished2 × RTX 3070 · driver 590.48.01 · 2026-10-23
RunPod pod jjjz6n5bejqe5e. A recorded run; this page does not update.
GPUs
2 × RTX 3070
GA104 · sm_86 · PCI 0000:46:00.0, 0000:81:00.0
Nodes per GPU
46
online 46 · booted 46 · halted 46
Midget on one card
5.58 → 3.41
nats per byte over 8 steps
Held-out loss
5.65 → 2.99
on text the learner never trained on

What the host said to the GPU

Each card host recorded its own conversation with AlphaOS. Every write is a protocol word or a program's code going into memory the GPU reads. Every wait is the host holding until the GPU's nodes answered. Choose a run, then replay it.

exec: programs loaded at run time, CALL.REL/RET.REL · 17 host events over 84.3 ms · exit 0

  1. 0.000 mshosthost process starts
  2. 29.583 mshostGPU channel ready (RM client, device, memory)
  3. 32.645 mshostAlphaOS image and tables written to the GPU
  4. 35.677 mshostAlphaOS launched (the one launch of this GPU's life)
  5. 38.587 mswaitwaited for GPU 0 booted = 46 · the GPU answered
  6. 41.499 mswritewrote GPU 0 code area +0x0 · 48 bytes
  7. 44.756 mswritewrote GPU 0 program table +0x0 · 32 bytes
  8. 48.499 mswritewrote GPU 0 code area +0x400 · 176 bytes
  9. 51.407 mswritewrote GPU 0 program table +0x20 · 32 bytes
  10. 54.280 mswritewrote GPU 0 code area +0x800 · 208 bytes
  11. 57.154 mswritewrote GPU 0 program table +0x40 · 32 bytes
  12. 60.102 mswritewrote GPU 0 task ring +0x0 · 1,920 bytes
  13. 63.044 mswritewrote GPU 0 ring tail = 120 · 8 bytes
  14. 73.809 mswaitwaited for GPU 0 done = 120 · the GPU answered
  15. 77.118 mswritewrote GPU 0 command = 1 · 8 bytes
  16. 80.921 mshostevery node halted; the completion fence released
  17. 84.292 mshostmailbox read back to stdout
Times from the card host's CLOCK_MONOTONIC, from its first record. Recording a step costs the host about 3 ms (each record is made durable before the next step), so steps look slower than the GPU is.

Every run on the cards

Each run is one Alpha-compiled executable, started on the pod: one launch, one conversation, one read-back of what the GPU left in its mailbox.

RunWhat it exercisesExitSecondsHost events
AlphaOSBootboot/1: 46 nodes online, booted, halted0<18
AlphaOSRunQueuerun queue: CAS claim, BRX dispatch0<111
AlphaOSExecexec: programs loaded at run time, CALL.REL/RET.REL0<117
AlphaOSCatgpu-file and gpu-term: a file read and printed0<163
AlphaOSGraphgraph/1: 240 dataflow tasks0<117
AlphaOSProcessesproc/1 and event/1: exit, kill, events0125
AlphaOSLoaderloader/1: one image installed, nine refused by name0<124
AlphaOSSignalsignal/1: a program supplies a task's input0<121
AlphaOSObjectobject/1: versioned objects, no torn snapshot0<116
MidgetLearnsMidget (tiny) learns inside AlphaOS01227
MidgetDeepLearnsMidget.Deep learns; held-out loss08628
AlphaOSNetnet/1: a cross-GPU pipe by remote name (two cards)01145
MidgetPeerstwo RTX 3070s train one Midget (relay/1)015106

The loader test sent ten program images. The card installed one, refused nine by name, and ran the installed one: cube(3) = 27. The refusal codes it returned were 0, 1, 2, 5, 4, 3, 6, 7, 7, 8.

Midget learns on the GPU

Midget is a byte-level transformer written for AlphaOS. Its trainer is itself a program running on the GPU: it publishes every learner, evaluator and updater task, and the nodes claim them. Training text is Shakespeare's Sonnets and Thus Spake Zarathustra.

Step 7Training 3.3678Held-out 2.9859
2.733.524.305.095.8701234567
Midget.Deep on one RTX 3070: training loss, and loss on held-out windows (never trained on), before each step.
Step 7GPU 0 3.4010GPU 1 3.3053
3.123.784.445.105.7601234567
Two RTX 3070s train one Midget: each card computes gradients on its own batch and the host relays them between the cards every step.
Samples
Generated on the card, greedily, from the same prompt. Before training: nnnn····························································
After 8 steps: ␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣
After 8 steps of 8 sequences of 16 bytes, Midget has learned the commonest byte, the space (␣); it needs far more steps before it writes words. A non-printing byte shows as ·.
Two cards agree
Both GPUs ended with the same parameters and wrote the same sample.

Where the 46 nodes ran

Each node records the SM it found itself on (SR_VIRTUALSMID). The card fills one SM of each TPC first: 0, 2, 4 … then 1, 3, 5 ….

  1. node 0SM 0
  2. node 1SM 2
  3. node 2SM 4
  4. node 3SM 6
  5. node 4SM 8
  6. node 5SM 10
  7. node 6SM 12
  8. node 7SM 14
  9. node 8SM 16
  10. node 9SM 18
  11. node 10SM 20
  12. node 11SM 22
  13. node 12SM 24
  14. node 13SM 26
  15. node 14SM 28
  16. node 15SM 30
  17. node 16SM 32
  18. node 17SM 34
  19. node 18SM 36
  20. node 19SM 38
  21. node 20SM 40
  22. node 21SM 42
  23. node 22SM 44
  24. node 23SM 1
  25. node 24SM 3
  26. node 25SM 5
  27. node 26SM 7
  28. node 27SM 9
  29. node 28SM 11
  30. node 29SM 13
  31. node 30SM 15
  32. node 31SM 17
  33. node 32SM 19
  34. node 33SM 21
  35. node 34SM 23
  36. node 35SM 25
  37. node 36SM 27
  38. node 37SM 29
  39. node 38SM 31
  40. node 39SM 33
  41. node 40SM 35
  42. node 41SM 37
  43. node 42SM 39
  44. node 43SM 41
  45. node 44SM 43
  46. node 45SM 45

What the cards taught us

The first runs found four things that the CPU emulator had accepted. Each was fixed in the compiler or the emulator, then confirmed on the card.

Every lane stayed
The first boot counted 1,472 nodes online instead of 46: all 32 threads of every node had run past their exit. The exit read its predicate 7 cycles after the comparison that wrote it; on the card a branch or exit needs 13. Alpha's assembler now waits 13, and the emulator reports the hazard.
Two reserved registers
The signal test hung at 100% with its producer stuck before its first store. The producer used R70:R71, the two registers the hardware reserves above a 72-register allocation. The image now allocates what its programs need, and the emulator faults on a reserved register.
A new driver
The pod ran driver 590, which Alpha had never run on. Every structure the host sends the driver has the same layout in NVIDIA's 590 and 580 sources, so the 590 branch is now admitted. Before that change, the host refused 590 by name.
Cards by their real names
The container exposed the GPUs as /dev/nvidia2 and /dev/nvidia5, not 0 and 1. Each AlphaOS build now names the cards it runs on.
Record
Run record ↗ · Where AlphaOS is going ↗