Benefits and tradeoffs#

Alpha's distributed training is built on a bet: that the compiler, which already knows what every value means, who writes it, and which instruction computes it, should also decide how a learning system is split across machines, and should be able to prove that the split changes nothing.

This page makes the case for that design, then weighs it honestly. Every benefit below comes with the evidence behind it and where it stops today. Nothing here is a performance claim: no distributed Alpha run has been timed against another stack yet.

The case for it#

Every run is the same run#

Most distributed training is reproducible only approximately. The communication library chooses how to reduce gradients: a ring or a tree, how to chunk, which bucket goes first. That choice can change with the GPU count, the topology or the library version. Change any of those and the floating-point result changes, by a little, every step.

In Alpha the reduction order is part of the program. A combine treeDistributed · explanationCombining contributionsOrdered combine trees instead of opaque collectives, delivery to every owner or one, numerical contracts, and how the declared order is shown to have run. fixes which values are added in which order, at which owner, with which rounding. The topology is part of the numerical contract, and every message names that contract. The results:

• Resume is exact. Five uninterrupted updates and three updates, a checkpoint and two more in fresh processes end in the same bits, on GPUs and on hosts (Distributed checkpoints and resumeDistributed · explanationDistributed checkpoints and resumeLogical shard manifests, immutable publication, manifest trees, path-only restore, and resuming on a different owner count.).
• Instruction sets don't matter. Owners on x86-64 and AArch64 publish exactly the bytes of an all-x86-64 run, including after resuming with every owner swapped to the other architecture.
• Splitting the state changes nothing. Partitioned optimizer state, parameters and gradients equal the replicated learner bit for bit.
• Executables reproduce. All 31 executables from the latest GPU qualification rebuild byte-identically from the commit they were built from.

A training curve that moves therefore moved because the learning changed, never because the cluster did.

Today: shown on small learners: a scalar regression, a two-coordinate predictor, a non-gradient learner, and least squares with momentum and Adam.

Distribution is a compiler transformation, not a framework#

In a conventional stack, distribution is a wrapper around a model, and the wrapper's behaviour lives in a library you cannot inspect from the model's code. In Alpha it is typed source that can be read and changed: roles, membership and protocolsDistributed · explanationParticipants, endpoints and framesRoles and compact membership, star and hierarchy topologies, native connection setup, bounded identity-checked frames, and protocols projected to every role., combine trees, ownership, and checkpoint layouts. An agent or a researcher can:

• see exactly which participant holds which word;
• change a placement and check it still covers the state exactly once;
• try a new decomposition without touching the training loop.

The familiar strategies are not privileged switches. Data parallelism, optimizer-state partitioning, parameter partitioning and gradient partitioning (the ideas behind ZeRO stages 1 to 3) are compositions of five operations: partition, replicate, transfer, combine and advance a version. A new strategy is a new composition of the same checked parts.

Today: fixed membership and explicit placement; no automatic placement search.

One executable per role, and nothing else to install#

Each participant is a self-contained executable that the Alpha compiler emitted, making its own system calls. A GPU machine needs only:

• Linux;
• the NVIDIA kernel driver of the release the profile was measured on (580).

It does not need CUDA, NCCL, MPI, Python or a container image with a version matrix.

What runs is exactly what was built: the qualification driver uploads each executable, checks its SHA-256 on the remote machine, and runs it. There is no runtime library version to drift between the build and the run, and no launcher that decides anything about the learning.

Today: NVIDIA SM86 GPUs (RTX 3070 and 3090) for multi-GPU runs.

Failures are refused by name, before they cost anything#

A distributed run that silently mixes the wrong things is worse than one that crashes. Alpha refuses mismatches at the earliest point that can see them.

When the build is compiled, it refuses:

• missing target support or an unsupported operation;
• a membership with overlaps or gaps;
• a combine tree that misses or duplicates a contribution;
• memory demands one byte over the supply.

On the first message, a peer compiled for a different learner, rate, numerical contract, combine order or membership is refused, before any combined value exists.

At start-up on a GPU, the wrong card or the wrong product exits with device-uuid or nvidia-plan-host. It writes no checkpoint.

At resume, these are refused before any owner trains:

• a corrupted shard;
• another learner's checkpoint;
• a stale epoch;
• a NaN hidden behind a repaired checksum.

Each refusal is tested, many of them by executed mutants: deliberately wrong programs that the gates must catch.

Today: a failed round aborts the whole run. There is no automatic recovery yet.

State you can move#

A checkpoint is indexed by what the state is, not by where it lived. Shards bind logical regions and one capture, and the learner's identity deliberately excludes the owner count and the sample split. Saved state can therefore resume on a different number of owners:

• momentum moves from four owners to three, then to five;
• Adam moves from four to two, with its moments and bias-correction powers going to their new owners.

Each restore matches an independent oracle bit for bit. On GPUs, a checkpoint written on two RTX 3070s was resumed on a pair whose coordinator was an RTX 3090, then used for prediction.

On a hierarchy no participant ever holds another's tensor bytes. Restore reads one path from the root to its owner, never a sibling subtree.

Today: repartitioned restore works on a star for partitioned-parameter checkpoints. It is not yet built for hierarchies or for replicated checkpoints.

Unequal devices, as a first-class case#

Equal shards across identical GPUs is the easy case. Alpha's combine carries (Σ gradient, Σ samples) pairs, so owners can hold unequal shares and the update stays exact. In the qualification, one GPU holds one sample and the other two.

On a separate-host RTX 3070 + RTX 3090 pair the run passed in both role assignments, so correctness cannot depend on which product holds which share. A mutant that averages the two devices instead of weighting them is refused.

Each GPU's memory is admitted separately. A law states that neither process may borrow the other card's video memory, even though both share one host's RAM.

Today: two GPUs per run. The 3070 + 3090 pair ran the 3 October 2026 implementation and has not been rerun since.

Not just gradients#

The same publication, checkpoint and resume machinery serves a learner with no gradient at all. In the independent-state learner, each owner evolves its own state by a local rule. It ran on two RTX 3070s with persistent finite binary64 updates in video memory, continuation, and fresh GPU evaluation. Distribution is not a feature of the optimizer, so research that leaves backpropagation behind keeps the same tooling.

Designed for the fleet from the first two GPUs#

Small distributed systems are usually built around a coordinator, a rank table and an all-to-all control mesh, and those are exactly the designs that fail at scale. Alpha's representations were built for a 200,000-device target from the start:

• membership is structural: 200,000 roles in four index nodes;
• the hierarchy has bounded fan-out: three levels at fan-out 64;
• each role has at most fan-out + 1 peers;
• checkpoint metadata is a manifest tree.

Each is checked by the type checker at that size (Toward 200,000 devicesDistributed · analysisToward 200,000 devicesThe heterogeneous fleet design target, the five kinds of scale claim, and the bounded-fan-out representations that already exist for it.). The two-GPU runs use the same mechanisms the fleet profile would scale, not a prototype to be thrown away.

Today: representation only at that scale. Ten participants is the most that has executed.

Evidence you can audit#

Every physical claim on this site points to a retained record. Each record names:

• the pod and the GPU UUIDs;
• the driver;
• the SHA-256 of every artifact;
• sealed observability streams for the session.

The trackerDistributed · explanationQualifying on real hardwareThe physical qualification driver, its refusal and host-shadow controls, the tracker, and every two-GPU result to date. refuses to record a physical pass without a completed run and a linked stream. Implemented is never reported as verified, and on 13 October 2026 nothing is marked verified yet. The site and the records say precisely what has run, on what, and what has not.

Pros and cons#

Which to choose#

Choose Alpha's distributed path when you need to:

• know exactly what was computed, and reproduce it;
• change how state is split and see the consequences checked;
• mix unequal devices or host architectures in one learner;
• study learners other than gradient descent;
• build on mechanisms designed to scale without a central coordinator.

Choose a conventional stack (PyTorch with NCCL, DeepSpeed, JAX) today when you need:

• throughput on many GPUs now;
• large-model training recipes, or elastic and fault-tolerant clusters;
• AMD or Apple hardware in the run;
• an existing ecosystem.

Alpha does not yet compete there, and this page does not pretend it does.

What would change the balance#

The weaknesses above are mostly unbuilt work, not design limits. The next items in the programme each remove one:

• a GPU realization of data parallelism and partitioning;
• a bandwidth-optimal reduce-scatter, and direct device transfers;
• more than two GPUs per run, and native AArch64 participants;
• the first cross-vendor run, NVIDIA with AMD;
• authenticated membership, elastic epochs and durable authority beyond one publisher;
• a matched performance measurement against a conventional baseline, including a negative result if that is what it shows.

Read next#

• Distributed learningDistributed · explanationDistributed learningHow one learning system becomes cooperating executables across several GPUs, hosts and instruction sets, and what has been checked, run and physically qualified.: how the pieces fit together.
• Qualifying on real hardwareDistributed · explanationQualifying on real hardwareThe physical qualification driver, its refusal and host-shadow controls, the tracker, and every two-GPU result to date.: every physical result.
• Fit and limitsEvaluate · analysisFit and limitsWhat Alpha is good at, what it is bad at, and the central tradeoff.: the same trade for Alpha as a whole.