Distributing learning state#

A distributed learner has to answer three questions: who computes which part of an update, who holds which part of the state, and who is allowed to change it. Alpha answers them in the types. Every mutable word has one logical owner and a version, replicas are read-only copies tagged with the version they mirror, and the compiler checks that the owners' pieces cover the state exactly once.

This page covers:

• the learner shapes that exist today;
• how optimizer state, parameters and gradients can each be partitioned;
• how a partitioned learner is shown to equal the replicated one;
• which shapes have run on GPUs.

The rule underneath#

The design (chapter 03 of the PRD) separates a logical region, identified by schema, field, coordinate domain, representation and version, from its physical fragments: an owning participant, an address space, a layout and a lifetime. A shard map from coordinates to fragments must satisfy three conditions:

coverage(P) = required logical domain
intersection(P[i], P[j]) = empty for distinct mutable owners
decode(assemble(P, fragments)) = logical state

Duplicate or missing coverage is refused even when the byte totals happen to match. Empty shards are legal. Replication replaces disjointness with a read-only relation, and a replica is never a second authority.

Five operations compose every distribution:

• partition a domain;
• replicate a version;
• transfer a region;
• combine contributions;
• advance a state version.

Data, tensor, pipeline and optimizer-state partitioning are names for compositions of these five, not compiler switches.

The consistency mode implemented today is synchronous versioning. Every contribution to update u reads the declared version of its inputs, and the update becomes authoritative only when every required contribution has arrived. A contribution computed against version 12 is never mixed into a version-13 update.

Learner shapes#

* The RTX 3070 + RTX 3090 runs (3 October 2026) qualified the implementation of that date. The merged tree was requalified on two RTX 3070s on 13 October 2026; the mixed pair has not been rerun since. Executables built after 14 October use the smaller packed host image and have not yet run on GPUs.

Samples-partitioned scalar learner#

PartitionedRegression.NativeHost holds the frames, the sample-weighted combine and the checkpoint. PartitionedRegression.Build.NativeHost adds the SM86 gradient program.

Each GPU computes the gradients for its own samples. The host participants combine (Σ gradient, Σ samples) and update the shared parameter.

The coordinator holds one sample and the worker holds two, so the work is unequal by construction. A mutant that takes the unweighted mean of the two devices is refused by the oracle.

Coordinate-owned learner#

Each owner holds one coordinate of a two-coordinate linear predictor, together with its accumulator. Build.CoordinateDevice and Build.CoordinateMemory keep that state resident in video memory:

• it is uploaded once;
• it is updated in place by compiled SM86 kernels;
• it is captured explicitly before a checkpoint is published.

The host keeps a shadow copy for checkpointing, and the qualification's host-shadow control proves that the shadow is not the authority. The control is described in Qualifying on real hardwareDistributed · explanationQualifying on real hardwareThe physical qualification driver, its refusal and host-shadow controls, the tracker, and every two-GPU result to date.. The model is PartitionedRegression.Coordinates, and CoordinateHost is a binary64 host realization of the same learner.

Independent-state learner#

LocalCorrelation.System is a non-gradient learner. Each owner evolves its own state region by a local rule, written as typed assignments, on immutable input pairs. The rule is built from Learning.IndependentState and Learning.ScaledProductUpdate.

It runs on a star (Runtime.NativeIndependentState) or a hierarchy (Runtime.NativeIndependentTree). On GPUs each owner holds its state persistently in video memory and updates it with finite binary64 SM86 arithmetic.

It exists to show that the distribution mechanisms are not specific to gradients. The same publication, checkpoint and resume machinery serves it unchanged.

Data-parallel learners#

Learning.DataParallel describes the learner, and Runtime.NativeDataParallel realizes it:

-- Slots, by position, with k parameters, m optimizer-state words and samples
-- of width s. The contribution program reads parameters [0, k) and one
-- sample [k, k + s) and writes only its contribution [k + s, 2k + s). The
-- update program reads parameters [0, k), the state [k, k + m), the mean
-- [k + m, 2k + m) and the rate at 2k + m, and writes only parameters and
-- state [0, k + m). The state is persistent learner state: checkpointed with
-- the parameters and required to agree across owners. The same typed
-- assignment semantics (Component.Pointwise) serve gradient and non-gradient
-- rules: a contribution need not be a gradient.
family DataParallelSystem : Type 0
parameter erased dataParallelContributionSlots : Nat
parameter erased dataParallelUpdateSlots : Nat
constructor DataParallelSystemOf
field unrestricted dataParallelParameterCount : Nat
field unrestricted dataParallelSampleWidth : Nat
field unrestricted dataParallelContribution : (family StdList (family PointwiseAssignment StdU64 dataParallelContributionSlots))
field unrestricted dataParallelUpdate : (family StdList (family PointwiseAssignment StdU64 dataParallelUpdateSlots))
field unrestricted dataParallelInitial : (family StdList StdU64)
-- the optimizer state's initial words (m = its length; none for plain steps)
field unrestricted dataParallelInitialState : (family StdList StdU64)
field unrestricted dataParallelRate : StdU64
-- per owner, per sample, the sample's s words
field unrestricted dataParallelSamples : (family StdList (family StdList (family StdList StdU64)))

end-family

A system is two programs of typed pointwise assignments (a per-sample contribution and an update), initial parameters and optimizer state, a rate, and the samples, listed per owner. The gates use least squares with plain steps, heavy-ball momentum, and Adam (moments and bias-correction powers, with native square root and reciprocal).

Samples may be split unequally. In the gates the owners hold one, two, three and zero samples, and the owner with none still contributes (0, 0).

Each round runs four steps:

1. Every owner sums its contributions in sample order.
2. The declared combine runs: left-deep on the star, or up and down the tree on a hierarchy (Combining contributionsDistributed · explanationCombining contributionsOrdered combine trees instead of opaque collectives, delivery to every owner or one, numerical contracts, and how the declared order is shown to have run.).
3. The total is replicated.
4. Every owner applies the typed update.

Before anything is published, owner 0 (or each parent, on a hierarchy) compares every owner's replicated state. Replicas that diverge publish nothing.

Partitioning#

For an update that is separable, where each coordinate's new value depends only on that coordinate's state and mean, three things can be partitioned independently. Runtime.NativeDataParallelOwned realizes all three on a star, and Runtime.NativeDataParallelOwnedTree on a hierarchy. Coordinate i belongs to owner i mod n.

dataParallelSeparable checks the update before any partitioning is admitted:

• Admitted: plain steps; momentum; Adam, whose two bias-correction powers are declared shared.
• Refused: declaring Adam's second moment shared, or overclaiming the words per coordinate.

Optimizer state#

Each coordinate's optimizer state (momentum's velocity, Adam's two moments) belongs to one owner, which alone updates it. The parameters are still replicated. After each update they are assembled by a selection combine, never by arithmetic.

Parameters, through an explicit read replica#

Each owner also holds only its own coordinates' parameters, so no owner holds another owner's parameter. Every round gathers them into a short-lived read replica of the current version:

1. Offer. Every owner offers its own parameters, with its ownership mask.
2. Materialize. The selection combine gathers and replicates them, checks coverage, and installs the result as the owner's replica, tagged with the version it was gathered at.
3. Contribute. The contribution reads only a held replica of the current version, for all of the owner's samples. A released replica refuses as data-parallel-replica-released, and a stale one as data-parallel-replica-version.
4. Release. The replica is cleared before the update. The version cannot advance while a replica is still held (data-parallel-replica-retained).

The replica counts toward the owner's peak memory, never toward its persistent state or checkpoint. For four owners the laws fix the persistent bytes per owner:

Gradients#

A reduction can deliver its result to one owner along the tree path (CombineDeliverTo). With gradient partitioning, every owner projects its contribution onto each coordinate owner's coordinates, and one reduction per coordinate owner delivers those sums to that owner alone.

Each coordinate's sum runs over the same tree, in the same order, as the full sum, so it is bit-identical to the replicated sum's word. The update then reads only its own coordinates' means.

Gradient partitioning changes no persistent state:

• A checkpoint resumes with or without it.
• With it switched off, every executable is byte-identical to the build without it.

It combines with replicated parameters, or with everything partitioned, on the star and on the hierarchy.

How equivalence is shown#

Every partitioned learner is compared with the replicated learner, and with an independent binary64 oracle, bit for bit:

• Partitioned momentum and Adam equal the replicated learner's five updates on four owners that alternate x86-64 and emulated AArch64. The run is resumed with every owner on the other instruction set.
• Fully partitioned Adam on seven owners at fan-out 2 equals the replicated hierarchy learner across a resume.
• Every owner's state holds exactly its own words, and every checkpoint shard holds exactly its word.
• Executed mutants complete their communication and are refused. Examples: delivering each owner's sums to the next owner, releasing a replica before it is read, never releasing it, and mixing an unweighted mean.

Gates#

Not built yet#

• GPU realizations. No data-parallel or partitioned learner has a GPU realization yet; all of them run on hosts.
• Physical AArch64. Every AArch64 owner so far ran under QEMU.
• Partitioning limits:
  • a bandwidth-optimal reduce-scatter (each owner still sends one slice per coordinate owner);
  • coordinate blocks larger than one word.
• Learner kinds: asynchronous and bounded-staleness learners.

Read next#

• Distributed checkpoints and resumeDistributed · explanationDistributed checkpoints and resumeLogical shard manifests, immutable publication, manifest trees, path-only restore, and resuming on a different owner count.: how this state is saved, and restored onto a different owner count.
• Hybrid hosts and multiple GPUsDistributed · explanationHybrid hosts and multiple GPUsOne executable per role and target, x86-64 and AArch64 participants, binding a GPU by UUID, unequal devices, and memory admission across processes.: where these owners run.