AlphaGitHub

All training runs / Diakosmos / RTX 3070

Diakosmos Live measurement

99,823,934 parameters · bytes, no tokenizer · OpenWebText · Alpha native execution on a rented RTX 3070

διάκοσμος, the ordering of a world. A language model with no attention matrix and no learned embedding: each byte enters as a fixed constellation of 665 phasors, and each of 14 blocks blends a decaying harmonic recurrence over the past with a planar rotation between its wire groups. It is harmonic-gpt's leading design, read to its equations and being written in Alpha, measured here against Alpha's own GPT-2 small on the same held-out bytes.

Chat with Diakosmos: the run's model fine-tuned on conversations, a byte a step on the same card.

CompletedNVIDIA GeForce RTX 3070 · driver 580.126.20 · 11 of 11 gates passed
stale · Snapshot 23:14:04 UTC

What to expect

How long until Diakosmos writes text, and how long GPT-2 took

Diakosmos was built in Alpha, shown equal to an independent reference step by step, and trained: all 15,360 updates. Its run's time was measured from its own invocations, as GPT-2 small's was on this same RTX 3070.

Trained
100.0%
15,360 of 15,360 updates · 0 min left at its measured rate
The training run
24 h 31 min
2.01 GB of text at 22.8 KB/s, measured
Diakosmos's text
update 15,360
greedy, GPT-2's six prompts, at its scored checkpoints: below
GPT-2 small, measured
1 h 47 min
to its first text · 49 GPU hours to 1.40 bits/byte · 92 h for 9B tokens
harmonic-gpt's leading model read to its equationsdone · took 8 min
GPT-2 baseline measured on the held-out textdone · took 13 min of GPU time
Pod cleared; telemetry recordingdone · took 5 min
OpenWebText as bytesdone · took 25 min
Reference implementationdone · took 35 min
Alpha forward pass equals the referencedone · took 6.5 h
Causalitydone · took 5 min
Gradient equals the referencedone · took —
Optimizer update equals the referencedone · took —
Throughput and energy measureddone · took —
Training rundone · took 24.5 GPU hours of training and 69 min of held-out scoring
done (11)nextbuild and check in Alphatraining
  1. 1harmonic-gpt's leading model read to its equationstook 8 min
  2. 2GPT-2 baseline measured on the held-out texttook 13 min of GPU time
  3. 3Pod cleared; telemetry recordingtook 5 min
  4. 4OpenWebText as bytestook 25 min
  5. 5Reference implementationtook 35 min
  6. 6Alpha forward pass equals the referencetook 6.5 h
  7. 7Causalitytook 5 min
  8. 8Gradient equals the referencetook —
  9. 9Optimizer update equals the referencetook —
  10. 10Throughput and energy measuredtook —
  11. 11Training runtook 24.5 GPU hours of training and 69 min of held-out scoring
Widths are each step's time: measured for finished steps and for the run (from its invocations), the middle of the estimate for the rest. Estimates were working-time ranges for each open gate, set on 2026-10-11 from the GPT-2 system this reuses; each was replaced by the measured duration when its gate closed. The run's length is the throughput gate's measurement.

How long GPT-2 small took here

GPT-2 small's run on this card, by GPU hour: each invocation's loss on 2,000 windows it had not yet trained on, as bits per byte, with the held-out measurements as points.

latestunseen windows (estimated bits/byte) 1.40 bits/byteheld-out bytes (measured) 1.40 bits/byteafter llm.c's recipe 1.22 bits/byte
1.201.401.601.8020 h40 h60 h80 hfirst text4-window updatesllm.c recipe

What the hours bought

  1. 1.88 bits per byte after its first invocation (20 million tokens).
  2. First generated text, at update 100,000.
  3. 1.60 bits per byte.
  4. Then 20 hours near 1.58: with one window an update, gradient noise held it there (the diagnosed plateau). Four-window updates reached 1.50 within the hour.
  5. 1.40 bits per byte, and flat for the remaining 43 hours: a constant learning rate's plateau.
  6. 1.223 after 4 h 40 min of llm.c's recipe on FineWeb: more than the previous 68 hours had bought. The recipe mattered more than the hours.

Diakosmos matches GPT-2 small when its curve reaches 1.396 (the OpenWebText run) and 1.223 (after the recipe) on the same held-out bytes; it beats it if it gets there in fewer GPU hours.

What GPT-2 small wrote after 92 GPU hours

Its own text at update 2,200,000 (the OpenWebText run near its end (invocation 440)), greedy, up to 256 tokens. At this size, greedy decoding repeats itself once it loses the thread. Diakosmos's text for the same prompts, decoded the same way, is beside it: its newest, at update 15,360 of 15,360 (2.01 GB of text), greedy, up to 1131 bytes.

GPT-2 small

In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. “We found that the species was very close to the top of the ocean,” says Jennifer G. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H. H

Diakosmos · update 15,360

In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. The scientists are still trying to convince the scientists that the scientists were trying to do the same thing and that they were the ones that were the same as the scientists. They were the only ones that were the same as the scientists. They were the same scientists who were the same as the scientists. They were the same scientists who were the same scientists as the scientists. They were the same scientists who were the same scientists as the scientists. They were the same scientists who were the same scientists as the scientists. They were the same scientists who were scientists who were scientists and scientists who were scientists who were scientists and scientists who were the scientists of the scientists. The scientists were the scientists who were the scientists and the scientists who were the scientists of the scientists. The scientists were the scientists who were the scientists and the scientists who were the scientists of the scientists. The scientists were the scientists who were the scientists and the scientists who were the scientists of the scientists. The scientists were the scientists who wer

State
Completed
every gate passed
Held-out bits per byte
1.4194
final, after 24.5 GPU hours · GPT-2 small: 1.4417 after 25.2 GPU hours · 1.2228 best · 1.3964 OpenWebText only
Parameters
99.82 M
GPT-2 small has 124.44 M
Held-out text
18.54 MB
4,096 windows of OpenWebText never trained on

Decided before training

The measurement

One number decides whether Diakosmos learns language better than GPT-2 here: bits per byte on held-out OpenWebText, against the GPU time and energy it took to reach, on this card, beside Alpha's GPT-2 small measured the same way.

  1. Bits per byte, not loss per token. Diakosmos predicts bytes; GPT-2 predicts BPE tokens of about 4.4 bytes. Their losses per prediction are not comparable; the total information each assigns to the same text is. Bits per byte is the summed loss over the same bytes, divided by ln 2 and their count.
  2. Held-out text, fixed once. 4,096 windows (18,538,212 bytes) of OpenWebText documents split out by hash before any training, in a fixed order, the same bytes for every checkpoint of every model. The standard error over windows is about 0.002 bits per byte. harmonic-gpt's own leaderboard reported the best of several 82 KB random samples, which favours whichever run was lucky.
  3. Against cost, not steps. A byte model makes 4.4 predictions for each of GPT-2's; a step, a parameter or a token means different work in each. GPU hours and joules on the same RTX 3070, the same harness, the same data, are what a reader can compare. The curve is drawn against all three: hours, kilowatt-hours and bytes of text.
  4. The baseline was measured first. Every retained checkpoint of Alpha's GPT-2 runs was scored on these bytes before the pod was cleared. GPT-2 small reached 1.3950 bits per byte on OpenWebText alone after 92.3 GPU hours (at its constant learning rate it stopped improving after about 48 of them), and 1.2228 after llm.c's recipe on FineWeb.
  5. Correctness before speed. No loss counts until the gates below hold: the executable equals an independent reference, and no output can see a later byte. A recurrence that leaks the future reports an impossibly good loss; causality is checked bit for bit first.
  6. Secondary measures. HellaSwag with endings scored per byte (chance 25%), generated text, and everything the card reports, all on this page. None of them replaces the curve.

Recorded · the pod is stopped

The card, as the run left it

Every field the RTX 3070 answered, sampled once a second on the pod by scripts/alpha/diakosmos-telemetry.py. The run is complete and the pod is stopped, so these are the last samples it recorded: the seconds after training, and the minute and hour history of the whole run.

Last sample 23:46:09 UTC900 one-second samples held
GPU busy
0 %
share of the last sample period a kernel ran
Power
21 W
limit 220.00 W
Core temperature
33 °C
SM clock
210 MHz
max 2100 MHz
Device memory
0.00 GiB
of 8 GiB
Energy since collector start
4.446 kWh
NVML's cumulative counter
PCIe receive
0.341 MB/s
gen 1 × 8
Fan
40 %

Utilization

Kernel activity and memory-controller activity, percent of each sample period.

latestGPU 0 %memory controller 0 %
025507510023:31:0423:34:0523:37:0623:40:0723:43:0823:46:09

Power

Board power: NVIDIA's reported draw, its one-second average and the instantaneous reading, against the enforced limit.

latestdraw 21 W1 s average 21 Winstant 20.9 Wlimit 220 W
10020023:31:0423:34:0523:37:0623:40:0723:43:0823:46:09

Temperature and cooling

Core temperature, distance to the slowdown threshold and fan duty.

latestcore °C 33fan % 40
3436384023:31:0423:34:0523:37:0623:40:0723:43:0823:46:09

Clocks

Streaming-multiprocessor, graphics, memory and video clocks in MHz; the performance state is P0 (fastest) to P8 (idle).

latestSM 210 MHzgraphics 210 MHzmemory 405 MHzvideo 555 MHz
20030040050023:31:0423:34:0523:37:0623:40:0723:43:0823:46:09

Device memory

Used, free and driver-reserved memory of the 8 GiB card.

latestused 1 MiBfree 7,840 MiBreserved 352 MiB
02,5005,0007,50023:31:0423:34:0523:37:0623:40:0723:43:0823:46:09

Energy

NVML's cumulative energy counter since the collector started, in joules: the integral every joules-per-byte figure on this page divides.

latestenergy 1.60e+7 J
1.60e+71.60e+723:31:0423:34:0523:37:0623:40:0723:43:0823:46:09

PCIe

Host ↔ device traffic as NVML samples it (20 ms windows), MB/s.

latestto the GPU 0.341 MB/sfrom the GPU 0.39 MB/s
02.557.523:31:0423:34:0523:37:0623:40:0723:43:0823:46:09

Performance state

P-state over time; 0 is full performance, 8 the idle floor.

latestP-state P8
P0P5P1023:31:0423:34:0523:37:0623:40:0723:43:0823:46:09

Why the clocks are what they are

NVIDIA's clock-event reasons for the last 15 minutes, one lane per reason. Software power cap during training means the card is at its power limit; thermal lanes mean heat, not the program, set the pace.

GPU idleapplication clockssoftware power caphardware slowdownsync boostsoftware thermalhardware thermalpower brakedisplay clocks23:31:0423:46:09
software power cap
0.04 h
since the driver loaded
software thermal
0.00 h
since the driver loaded
hardware thermal
0.00 h
since the driver loaded
power brake
0.00 h
since the driver loaded
sync boost
0.00 h
since the driver loaded

Container processor and memory

The container's own CPU time (cores busy) and memory, from its cgroup; the host machine is shared, so its load is not ours.

latestCPU cores 0.0175memory GiB 7.75
02.557.523:31:0423:34:0523:37:0623:40:0723:43:0823:46:09

Disk and network

Block reads and writes of the container and eth0 traffic, MB/s; the run streams its bytes from disk.

latestdisk read 0 MB/sdisk write 0 MB/snetwork in 8.17e-4 MB/snetwork out 7.54e-4 MB/s
051023:31:0423:34:0523:37:0623:40:0723:43:0823:46:09
Every field and its source
FieldLatestUnit15-minute min15-minute maxSource
gpuUtilization0%00nvidia-smi utilization.gpu
memoryUtilization0%00nvidia-smi utilization.memory
memoryUsedMiB1MiB11nvidia-smi memory.used
memoryFreeMiB7,840MiB7,8407,840nvidia-smi memory.free
memoryReservedMiB352MiB352352nvidia-smi memory.reserved
powerW20.96W20.7221.58nvidia-smi power.draw
powerAverageW20.96W20.7221.58nvidia-smi power.draw.average
powerInstantW20.93W20.721.64nvidia-smi power.draw.instant
powerLimitW220W220220nvidia-smi enforced.power.limit
temperatureC33°C3335nvidia-smi temperature.gpu
fanPercent40%4040nvidia-smi fan.speed
smClockMHz210MHz210210nvidia-smi clocks.sm
memoryClockMHz405MHz405405nvidia-smi clocks.mem
graphicsClockMHz210MHz210210nvidia-smi clocks.gr
videoClockMHz555MHz555555nvidia-smi clocks.video
performanceState8P88nvidia-smi pstate
clockEventReasons1mask11nvidia-smi clocks_event_reasons.active
powerCapMicroseconds1.34e+8µs1.34e+81.34e+8nvidia-smi clocks_event_reasons_counters.sw_power_cap
softwareThermalMicroseconds0µs00nvidia-smi clocks_event_reasons_counters.sw_thermal_slowdown
hardwareThermalMicroseconds0µs00nvidia-smi clocks_event_reasons_counters.hw_thermal_slowdown
powerBrakeMicroseconds0µs00nvidia-smi clocks_event_reasons_counters.hw_power_brake_slowdown
syncBoostMicroseconds0µs00nvidia-smi clocks_event_reasons_counters.sync_boost
pcieGeneration1gen11nvidia-smi pcie.link.gen.current
pcieWidth8lanes88nvidia-smi pcie.link.width.current
encoderUtilization0%00nvidia-smi utilization.encoder
decoderUtilization0%00nvidia-smi utilization.decoder
energyJ1.60e+7J1.60e+71.60e+7NVML total energy since this collector started
pcieTxMBps0.39MB/s0.3419.423NVML PCIe transmit throughput
pcieRxMBps0.341MB/s0.2922.197NVML PCIe receive throughput
cpuCores0.01752cores0.01181.017container CPU time per second (cgroup cpuacct)
containerMemoryGiB7.747GiB7.747.886container memory in use (cgroup)
diskUsedGiB92.8GiB92.892.8/workspace in use
diskReadMBps0MB/s00.004071container block reads (cgroup blkio)
diskWriteMBps0MB/s00container block writes (cgroup blkio)
networkRxMBps8.17e-4MB/s00.1413eth0 received
networkTxMBps7.54e-4MB/s013.14eth0 transmitted
trainerCpuCoresnot answeredcores——the training executable's CPU time per second
trainerRssMiBnot answeredMiB——the training executable's resident memory

The headline

Learning curve

Held-out bits per byte of every checkpoint against the cumulative training cost on this card: GPT-2 small's runs, and Diakosmos's checkpoints as they are scored (at updates 0, 160, 320, 640, then every 1,280).

latestGPT-2 small · OpenWebText, constant learning rate 1.396 bits/byteGPT-2 small · then FineWeb under llm.c's recipe 1.223 bits/byteDiakosmos 1.419 bits/byte
2.0004.0006.0008.0000 h20 h40 h60 h80 h
Each point scores a checkpoint on the same 18,538,212 held-out bytes. The grey band is ±2 standard errors (over windows). Cost is cumulative training time and energy on the RTX 3070 (GPU hours); text bytes for the GPT-2 runs are their tokens × the held-out text's 4.420 bytes per token. Diakosmos's points appear here as it trains.
latestGPT-2 small · windows not yet trained on (estimated bits/byte) 1.40 bits/byteDiakosmos · each update's bytes, before it trains on them 1.43 bits/byte
2.004.0020 h40 h60 h80 h
Diakosmos reads every byte once, so each update's loss is measured on text it has not seen: every 16th update is drawn. GPT-2 small's is its invocations' loss on 2,000 windows it had not yet trained on, divided by the held-out text's 4.420 bytes per token. The held-out points above are the measurement; this is the run as it goes.

11 of 11

Gates before the run

Each must hold, with its evidence, before the next starts. A failed gate stays on this list as failed.

  1. passed

    harmonic-gpt's leading model read to its equations

    spectral_kan_unified_scan_planar (MonumentalSystems/harmonic-gpt at bd71df2, MIT): every forward equation, initialization, optimizer partition and measurement convention traced to file and line. Findings: no polynomial or tanh in the harmonic path; the recurrence decay starts at sigmoid(3); its PyTorch fallback is not the scan's mathematics beyond t |ln γ| ≈ 83; its bakeoff BPB is the minimum over 82 KB random evaluations with one seed.

    MonumentalSystems/harmonic-gpt scripts/wire_lm_variants.py:394-615, harmonic_gpt/layers/spectral_kan_attention.py:222-438

  2. passed

    GPT-2 baseline measured on the held-out text

    Every retained checkpoint of Alpha's GPT-2 small runs scored on the same 4,096 held-out OpenWebText windows (18,538,212 bytes) by Alpha scoring executables on the RTX 3070, before the pod was cleared.

    Proof.GPT2ScoringArtifacts owtValidationStream (9aa5f7ed), owtValidationRecipe (327a29cd); scripts/alpha/gpt2-validation-loss.py

  3. passed

    Pod cleared; telemetry recording

    Everything on the pod removed except the OpenWebText token shards; the collector records every field the card answers each second.

    scripts/alpha/diakosmos-telemetry.py

  4. passed

    OpenWebText as bytes

    The 80 GPT-2 token shards decoded to 39,821,918,911 training bytes (8,009,779 documents) and 19,696,698 held-out bytes (3,990 documents), 4.41 bytes a token. Every shard equals its source parquet's documents byte for byte, each document's UTF-8 text followed by the 13 bytes of <|endoftext|>, split by the same hash. The held-out windows decode from exactly the baseline's input ids (sha256 d0684974…), so both models are scored on the same 18,538,212 bytes.

    scripts/alpha/diakosmos-prepare-bytes.py; manifest diakosmos-openwebtext-bytes/1, held-out text sha256 20669845…

  5. passed

    Reference implementation

    An independent PyTorch Diakosmos written from the equations, with the recurrence stepped byte by byte rather than by harmonic-gpt's cumulative-sum shortcut. On the same weights its logits equal harmonic-gpt's own modules to 4.7e-15 and its wire table to 2.2e-7; 24 entries of every parameter tensor (all of the small ones) agree with central differences to 1.4e-6; no output depends on a later byte; 99,823,934 parameters. One 100M forward pass over 2,048 bytes takes 3.9 s on the CPU; untrained, it scores 8.25 bits a byte. Dropout is 0: each byte is read once.

    scripts/alpha/diakosmos-oracle.py check --harmonic-gpt --full

  6. passed

    Alpha forward pass equals the reference

    Alpha's Diakosmos scoring executable (sm_86, 126 launches a pass of four 2,048-byte windows; seven new kernels) scores 16 held-out windows on the RTX 3070 with the reference's own initialized weights: mean loss 5.731625 nats against the reference's 5.731665 (bfloat16 product operands, as the device's), per position within 0.0027 on average and 0.013 at the 99.9th percentile; the reference without the bfloat16 rounding is 4.5 times further off. On weights perturbed to larger values (11.5 nats a byte) the means agree to 3e-5. The first runs hung: an Ampere late register read in the LayerNorm and a misaligned 128-bit load, both found by bisecting the launch graph.

    scripts/alpha/diakosmos-score-check.sh; Diakosmos.Build.Scoring; Proof.DiakosmosKernels

  7. passed

    Causality

    Every byte from position 1024 on changed in 16 windows: the device's losses at positions 0 .. 1022 are bit-identical, and every loss from 1023 on (which predicts a changed byte) differs. The reference's own check (every position against every later byte) passed before.

    check-causal on the pod: a.result, b.result

  8. passed

    Gradient equals the reference

    Alpha's Diakosmos gradient executable (one pass of two 2,048-byte windows: the forward's 122 launches, then the pullback's 416, the reverse recurrence included) writes the mean loss's gradient as a bank; every one of the 199 parameter tensors equals the reference's (bfloat16 product operands and incoming gradients, as the device's) with cosine at least 0.99997 and largest difference at most 1.0e-2 of the tensor's largest entry, on the initialized weights and on weights perturbed off their uniform start (so each head's and harmonic's indexing shows). 4.8 s on the RTX 3070 for the pass and the 409 MB read-back. Two faults found by splitting the comparison: the key projection's pullback read the query's harmonic weights (an argument index from the forward kernel), and the decay rates' reduction gave shared memory byte addresses where LDS and STS take word registers (nvdisasm on the emitted kernel showed it).

    scripts/alpha/diakosmos-gradient-check.sh (seed 0; seed 1 PERTURB=0.3); Diakosmos.Build.GradientReadback; scripts/alpha/diakosmos-oracle.py gradient, compare-gradient

  9. passed

    Optimizer update equals the reference

    Alpha's Diakosmos update executable loads a training checkpoint (the master, Adam's moments, each rotational row's first norm and running mean, the run's schedule and its step), takes a gradient bank as its input, runs one update in 597 launches on the RTX 3070 and writes the checkpoint after it. Given the reference's own gradient, its result equals the reference's update at the run's first update and at update 200 (the peak rate, from a state with nonzero moments): the Adam tensors, the moments and the rows' state to binary32 rounding (about 1e-7); the rotational AdamW rows to 6e-5 of their change; Muon's output projections to 0.6% (cosine 0.99999), both Newton-Schulz iterations rounding their products' operands to bfloat16 (harmonic-gpt's ran in TF32). The global clip, the bias corrections and the warmup-cosine schedule are the reference's: the schedule is computed by it in binary64 and carried in the checkpoint. About 4 s for the update with the 1.2 GB checkpoint in and out.

    scripts/alpha/diakosmos-update-check.sh; Diakosmos.Build.UpdateCheck, Diakosmos.Build.OptimizerGraph; Realization.Nvidia.SM86.Diakosmos.OptimizerSM86; scripts/alpha/diakosmos-oracle.py update, training-checkpoint

  10. passed

    Throughput and energy measured

    Alpha's Diakosmos training executable on the RTX 3070, from the run's first checkpoint on the training stream's first windows: an update (64 windows of 2,048 bytes, 32 passes of two, and the optimizer) takes 5.63 s and 1,242 J (about 220 W), 23.3 KB of text a second and 9.5 mJ a byte, measured as the difference between invocations of 16 and of 4 updates so the 1.2 GB checkpoint's load and save cancel. Runs repeat bit for bit; the training loss fell from 8.25 to 5.28 bits a byte over the first 16 updates. The run's budget follows: harmonic-gpt's 20 bytes a parameter, 15,360 updates (2.01 GB), about 24 GPU-hours and 5.3 kWh of training, beside about an hour of held-out scoring (234 s each).

    scripts/alpha/diakosmos-throughput.sh; Diakosmos.Build.Training (Proof.DiakosmosArtifacts train4, train16)

  11. passed

    Training run

    Started 12 October 2026, 01:31 UTC, completed 13 October, 23:03 UTC: 15,360 updates of 64 windows read once and in order from the training stream (2.01 GB, harmonic-gpt's 20 bytes a parameter), 160 updates an invocation (train160), from the reference's initialization with the reference's schedule. Held-out bits a byte: untrained 8.2551, 1.9806 at update 640, 1.6842 at 2,560, 1.5712 at 5,120, 1.4673 at 10,240, and 1.4194 ± 0.0023 at the end, after 24.52 GPU hours and 5.29 kWh of training. GPT-2 small, on the same bytes and card, was at 1.4417 after 25.23 GPU hours and 5.25 kWh, and reached 1.4194 at about 30 GPU hours by interpolation between its scored checkpoints. RunPod exited the pod on 12 October at 07:06 UTC, during invocation 19; the run resumed from its last checkpoint 18.8 hours later and that invocation ran again.

    scripts/alpha/diakosmos-run.sh; Proof.DiakosmosArtifacts train160, scoringHeldout; the pod's runs/points.jsonl and runs/invocations.jsonl

14 blocks × 1,330 wide

Model

Seven wire groups of 190 numbers: 95 phasors each for the byte's four bit pairs and three parities. The stream stays in those groups through every block.

stream x7 × 190LayerNormq, k, v1330 → 3990harmonic recurrenceS ← γ S + (c·k̃) vy = Σℓ q̃ℓ Sℓ · 7 heads × 8 ℓout_projMuonplanar rotation of group pairs (0,1) (2,3) (4,5)on each group's unit vector · angle 0.3 θ · gain per groupx + σ(β)·T+ (1−σ(β))·Sresidual
One block. The temporal path (top) is causal: the state S of each head and harmonic carries every earlier byte, decayed by γ per byte. The spatial path (bottom) rotates within one position only. Fourteen blocks, then LayerNorm and a head over the stream, each group's alignment with its running mean and that mean's length.

Where the parameters are

99,823,934 in all; the input projection holds three quarters.

  • Muon24,764,600 24.81%

    momentum 0.95 (Nesterov), five quintic Newton–Schulz iterations (3.4445, −4.7750, 2.0315) on the Frobenius-normalized update; no weight decay; peak learning rate 0.002

  • Rotational AdamW74,637,864 74.77%

    Adam (β 0.9, 0.999, ε 1e-8) direction, row-centred and with its radial part removed, an angular step normalized by its running RMS, then each row rescaled to its initial norm; decay 0.01; peak 0.003

  • Adam421,470 0.42%

    β 0.9, 0.999, ε 1e-8, no weight decay; peak 0.003

Memory of the recurrence at initialization

γ = σ(3 + oℓ) with offsets 0 … −1: half-lives from 14.3 bytes (ℓ0) to 5.5 (ℓ7), identical in every head until training separates them. harmonic-gpt measured trained half-lives of 1–2.4 positions at 125M; the run's trained ones are below.

Point at a cell
ℓ0ℓ1ℓ2ℓ3ℓ4ℓ5ℓ6ℓ7head 0head 1head 2head 3head 4head 5head 6
5.46 – 14.27 bytes

Memory of the recurrence as it trains

γ is learned: each block's g (one per head) and o (one per harmonic) move under Adam, so all 784 half-lives are read from every kept checkpoint and the newest. Most shorten, as harmonic-gpt found; a few lengthen far past where they began. The recurrence starts from zero at every 2,048-byte window, in training and in the held-out score, so no memory longer than a window is used. The colour scale is logarithmic and the same at every update.

Point at a cell
h0h1h2h3h4h5h6block 0block 1block 2block 3block 4block 5block 6block 7block 8block 9block 10block 11block 12block 13
0.32 – 754 bytes
UpdateMedian half-lifeLongestOver 16 bytesUnder 2 bytes
08.81 bytes14.3 bytes0 of 7840 of 784
1608.01 bytes13.6 bytes0 of 7840 of 784
3205.17 bytes11.9 bytes0 of 78448 of 784
6404.05 bytes24.9 bytes3 of 784184 of 784
1,2803.61 bytes105 bytes15 of 784214 of 784
2,5603.07 bytes332 bytes23 of 784244 of 784
3,8402.83 bytes516 bytes29 of 784265 of 784
5,1202.65 bytes630 bytes29 of 784279 of 784
6,4002.55 bytes661 bytes33 of 784290 of 784
7,6802.47 bytes686 bytes33 of 784305 of 784
8,9602.42 bytes704 bytes33 of 784312 of 784
10,2402.35 bytes712 bytes33 of 784324 of 784
11,5202.29 bytes733 bytes35 of 784330 of 784
12,8002.25 bytes740 bytes37 of 784335 of 784
14,0802.21 bytes748 bytes37 of 784343 of 784
15,3602.18 bytes754 bytes37 of 784350 of 784
Every parameter tensor
TensorShapePer blockIn the modelOptimizerRole
temporal.ln.weight13301,33018,620AdamLayerNorm gain before the recurrence
temporal.ln.bias13301,33018,620AdamLayerNorm shift
attn.qkv_proj.weight3990 × 13305,306,70074,293,800Rotational AdamWquery, key and value from the normalized stream
attn.qkv_proj.bias39903,99055,860Adamtheir biases
attn.q_harmonic_weight7 × 8 × 19010,640148,960Adameach head's query onto its 8 harmonic scalars
attn.k_harmonic_weight7 × 8 × 19010,640148,960Adameach head's key onto its 8 harmonic scalars
attn.spectral_coeffs7 × 856784Adameach harmonic's weight on its key (initially 1/(1+ℓ))
attn._gate_logit7798Adameach head's decay logit (initially 3)
attn._harmonic_decay_offset88112Adameach harmonic's decay offset (initially 0 … −1)
attn.out_proj.weight1330 × 13301,768,90024,764,600Muonthe recurrence's read-out onto the stream
attn.out_proj.bias13301,33018,620Adamits bias
spatial.theta3 × 1905707,980Adamplanar rotation angles of group pairs (0,1), (2,3), (4,5)
spatial.coupling7798Adameach group's rotation gain (initially 0.5)
gate_logit7798Adameach group's temporal–spatial blend (initially 0: half and half)
ln_f.weight1330—1,330Adamfinal LayerNorm gain
ln_f.bias1330—1,330Adamfinal LayerNorm shift
head.weight256 × 1344—344,064Rotational AdamWbyte logits from the stream, each group's alignment with its running mean, and that mean's length
Total99,823,934harmonic-gpt's 100M configuration: 99,823,934

No learned embedding

How a byte enters

Each byte is split into four bit pairs, Gray-coded onto QPSK phases (π/4, 3π/4, −3π/4, −π/4), with three Hamming parities as sums of those phases. Three Haar scales add averages and differences; the 95 harmonics are those four scales rotated by multiples of π/8. Type bytes to see where they land.

Constellation

All 256 bytes in one group and scale; the number beside a point counts the bytes that share it. The ring marks are the bytes you type, and the table their phases in every group.

64646464
ByteCharb7–6b5–4b3–2b1–0p0p1p2
68D0.750π0.250π0.750π0.250π-0.250π-0.750π-0.750π
105i0.750π-0.750π-0.750π0.750π0.750π0.750π-0.750π
97a0.750π-0.750π0.250π0.750π-0.250π0.750π0.250π
107k0.750π-0.750π-0.750π-0.250π-0.250π-0.250π0.250π
111o0.750π-0.750π-0.250π-0.250π0.250π-0.250π0.750π
115s0.750π-0.250π0.250π-0.250π0.750π0.250π-0.250π
109m0.750π-0.750π-0.250π0.750π-0.750π0.750π-0.250π
111o0.750π-0.750π-0.250π-0.250π0.250π-0.250π0.750π
115s0.750π-0.250π0.250π-0.250π0.750π0.250π-0.250π

Every byte's phases

The 28 phases that determine all 665 phasors of a byte, as hue around the colour wheel: one column per byte, one row per group and scale.

Point at a cell
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~base bits 7–6base bits 5–4base bits 3–2base bits 1–0base parity 0base parity 1base parity 2H1 bits 7–6H1 bits 5–4H1 bits 3–2H1 bits 1–0H1 parity 0H1 parity 1H1 parity 2H2 bits 7–6H2 bits 5–4H2 bits 3–2H2 bits 1–0H2 parity 0H2 parity 1H2 parity 2H3 bits 7–6H3 bits 5–4H3 bits 3–2H3 bits 1–0H3 parity 0H3 parity 1H3 parity 2
0.000π – 2.000π rad

harmonic-gpt's recipe

Optimizer and schedule

Three optimizers by parameter role, one schedule. The budget is harmonic-gpt's 20 bytes per parameter: 15,360 updates, 2.01 GB of text read once and in order, set after the throughput gate measured 23.3 KB/s.

Learning rate

latestAdam and rotational AdamW (peak 0.003) 3.0e-4Muon (peak 0.002) 2.0e-4
0.0e+01.0e-32.0e-33.0e-305,00010,00015,000
15,259 updates of 131,072 bytes (2 × 32 windows of 2,048 bytes): 200 warm-up updates, then a cosine to a tenth of the peak.
Muon
momentum 0.95 (Nesterov), five quintic Newton–Schulz iterations (3.4445, −4.7750, 2.0315) on the Frobenius-normalized update; no weight decay; peak learning rate 0.002
Rotational AdamW
Adam (β 0.9, 0.999, ε 1e-8) direction, row-centred and with its radial part removed, an angular step normalized by its running RMS, then each row rescaled to its initial norm; decay 0.01; peak 0.003
Adam
β 0.9, 0.999, ε 1e-8, no weight decay; peak 0.003
Update
131,072 bytes: 2 sequences × 32 accumulated passes × 2,048 bytes; global gradient clipping at 1.0
Numerics
binary32 master weights and recurrence state; bfloat16 matrix operands with binary32 accumulation, as Alpha's GPT-2
Data
OpenWebText (Skylion007/openwebtext at 79d93d786212) as its exact UTF-8 bytes, recovered from the GPT-2 token shards (byte-level BPE is lossless); held-out documents kept apart

Measured 11 October 2026

The GPT-2 baseline

step-2205000 under the stream contract and its adoption (step-108045) under the recipe contract, scored by two different executables, agree to the last digit.

CheckpointTokensGPU hoursGPU kWhLoss (nats/token)± s.e.Bits per byte
owt-step-3750001.536 B25.25.254.41690.00711.4417
owt-step-5000002.048 B31.16.474.33550.00711.4151
owt-step-6250002.560 B36.97.734.32170.00701.4107
owt-step-7500003.072 B42.88.954.32210.00721.4108
owt-step-8750003.584 B48.210.104.28810.00701.3997
owt-step-10000004.096 B52.911.084.30130.00711.4040
owt-step-11250004.608 B57.612.074.28720.00701.3994
owt-step-12500005.120 B62.012.984.29080.00701.4006
owt-step-13750005.632 B66.213.844.29000.00711.4003
owt-step-15000006.144 B70.214.684.27990.00701.3970
owt-step-16250006.656 B74.115.494.27370.00711.3950
owt-step-17500007.168 B78.016.304.27590.00701.3957
owt-step-18750007.680 B81.917.114.28310.00701.3980
owt-step-20000008.192 B85.917.914.29630.00711.4024
owt-step-21250008.704 B89.818.724.28390.00711.3983
owt-step-22050009.032 B92.319.234.27810.00701.3964
fineweb-step-1092709.674 B96.920.253.74610.00541.2228
Evidence Sources, executables and documents
Design
MonumentalSystems/harmonic-gpt at bd71df2 (MIT): scripts/wire_lm_variants.py SpectralKANUnifiedWireLM, harmonic_gpt/layers/spectral_kan_attention.py
Held-out input
d068497406362f161eac476b733bca5e0fd525f7e1f8f896bb0fe546eb5ae5ba · documents with sha256(text)[:4] % 2000 == 0, never trained on
Baseline scorers
Proof.GPT2ScoringArtifacts owtValidationStream 9aa5f7ed… and owtValidationRecipe 327a29cd…; scripts/alpha/gpt2-validation-loss.py
Telemetry
scripts/alpha/diakosmos-telemetry.py on the pod → website/ops/diakosmos-publish.mjs → this page
Publisher source
6777d94a6f9bd118a4857a8b745f34e00230ec61
name
NVIDIA GeForce RTX 3070
driver_version
580.126.20
vbios_version
94.04.3A.40.65
uuid
GPU-3ae3ac93-2217-3f2d-0274-fb92cf9eef13
pci.bus_id
00000000:01:00.0
memory.total
8192
power.default_limit
220.00
power.min_limit
100.00
power.max_limit
220.00
clocks.max.sm
2100
clocks.max.mem
7001
clocks.max.gr
2100
pcie.link.gen.max
4
pcie.link.width.max
16
compute_cap
8.6
persistence_mode
Enabled
compute_mode
Default