Distributed learning#

Alpha compiles one learning system into a set of cooperating native programs, one for each participant in a run. Distribution is not a library wrapped around a model. It is a transformation the compiler performs and checks: of what the state means, of who owns each part of it, and of the schedule that moves it.

The same system can be paired with:

• one GPU, or two identical GPUs;
• two unequal GPUs on separate machines;
• host processes on different instruction sets.

Changing the pairing changes how the system is realized. Changing the learner changes what is learned. The compiler keeps the two apart.

This section explains how that works today, piece by piece, and is exact about what has been checked, executed and physically run.

Why it matters#

• Every run is the same run. The reduction order is part of the program, so resume, repartitioning and mixed instruction sets reproduce the same bits.
• Nothing to install. Each participant is one self-contained executable. A GPU machine needs Linux and the NVIDIA kernel driver, not CUDA, NCCL, MPI or Python.
• Failures are refused by name, at compile time, on the first message, at start-up and at resume, before they can corrupt state.
• State is portable. Checkpoints are indexed by logical identity and can resume on a different number of owners.
• Built to scale without a coordinator. Membership, hierarchy and checkpoint metadata are bounded by fan-out, and are checked at 200,000 roles.

The cost is real: distributed runs are small, host-staged and not yet timed. Benefits and tradeoffsDistributed · analysisBenefits and tradeoffsThe case for compiler-owned distributed training (exact, inspectable, self-contained, refusal by name) weighed against what it cannot do yet. makes the case in full and weighs it honestly.

Three kinds of "multi"#

Different vendors (NVIDIA with AMD or Apple) in one run is the next physical milestone. It is not built yet.

How a run fits together#

1. The system declares its model, learner and state, with no device, host or address anywhere in them.
2. The distribution plan names:
   • the roles, the membership and the topology (a pair, a star or a hierarchy);
   • the protocol: what each role sends, receives and computes in each round;
   • the combine tree, the numerical contract, and which owner holds which state.
3. A pairing per role chooses the host target (x86-64 or AArch64), and for a GPU role the device profile, and admits the role's demands against them.
4. Deployment inputs (the exact card's minor and UUID, the peer address and port, resume or fresh) are compile-time arguments.
5. The compiler emits one executable per role. Each contains its own connection setup, its projected schedule, its frame checks, its GPU host program and its checkpoint code.
6. At run time the participants connect to each other, exchange bounded frames, combine contributions in the declared order, and apply the update. A checkpoint is published only when every owner's piece belongs to one capture.
7. A fresh process resumes from the published checkpoint, or evaluates from it.

There is no coordinator service, no Python controller, no MPI or NCCL, and no vendor runtime. A deployment tool copies the executables and starts them. It does not decide what is computed.

The invariants#

These hold for every distributed composition. Most are enforced by the compiler; the rest are checked by the runtime guards it emits.

• Model and learner semantics contain no device product, driver, endpoint, command descriptor or launch address.
• Every state mutation has one logical owner and a version. A replica is never a second authority.
• Communication is explicit in the typed plan, including completion, visibility and failure.
• A distributed rewrite declares exact or approximate semantics; a schedule cannot silently change the mathematics.
• Missing target support fails by name. Host transport is allowed. Computing an unsupported accelerator operation on the host is not.
• Persisted state is indexed by logical identity, not by rank, address or shard count.
• A failed round commits nothing.

Vocabulary#

What has been established#

The 3070 + 3090 runs (3 October 2026) qualified the implementation of that date. The merged tree was requalified on two RTX 3070s on 13 October 2026 across all three GPU lanes. On 14 October the host executables were made about twenty times smaller (shared routine code, compact SHA-256, a compressed image unpacked at start-up); executables built since then have passed every host gate but have not yet run on GPUs. No result on this page is a performance claim. Qualifying on real hardwareDistributed · explanationQualifying on real hardwareThe physical qualification driver, its refusal and host-shadow controls, the tracker, and every two-GPU result to date. lists every physical record.

Where the code lives#

Read the section#

1. Benefits and tradeoffsDistributed · analysisBenefits and tradeoffsThe case for compiler-owned distributed training (exact, inspectable, self-contained, refusal by name) weighed against what it cannot do yet.: the case for this design, and its pros and cons.
2. Participants, endpoints and framesDistributed · explanationParticipants, endpoints and framesRoles and compact membership, star and hierarchy topologies, native connection setup, bounded identity-checked frames, and protocols projected to every role.: roles, membership, topologies, connections, frames and protocols.
3. Combining contributionsDistributed · explanationCombining contributionsOrdered combine trees instead of opaque collectives, delivery to every owner or one, numerical contracts, and how the declared order is shown to have run.: ordered combine trees, delivery, and numerical contracts.
4. Distributing learning stateDistributed · explanationDistributing learning stateThe distributed learners, and partitioning of optimizer state, parameters and gradients, each equal bit for bit to the replicated learner.: the learners, and partitioning of optimizer state, parameters and gradients.
5. Distributed checkpoints and resumeDistributed · explanationDistributed checkpoints and resumeLogical shard manifests, immutable publication, manifest trees, path-only restore, and resuming on a different owner count.: publication, manifest trees, restore and repartitioning.
6. Hybrid hosts and multiple GPUsDistributed · explanationHybrid hosts and multiple GPUsOne executable per role and target, x86-64 and AArch64 participants, binding a GPU by UUID, unequal devices, and memory admission across processes.: mixed instruction sets, device binding, unequal devices and memory admission.
7. Qualifying on real hardwareDistributed · explanationQualifying on real hardwareThe physical qualification driver, its refusal and host-shadow controls, the tracker, and every two-GPU result to date.: the physical driver, its controls, and every result.
8. Toward 200,000 devicesDistributed · analysisToward 200,000 devicesThe heterogeneous fleet design target, the five kinds of scale claim, and the bounded-fan-out representations that already exist for it.: the fleet design target and what already exists for it.

The full product requirements and design are in docs/multi-device-learning/: 22 chapters, the implementation handoff, and the tracker.