Toward 200,000 devices#

Alpha's distribution design targets an envelope that runs from two GPUs to a possible 200,000 heterogeneous devices cooperating on one learning system: NVIDIA and AMD GPUs, Apple silicon, CPUs and future accelerators.

That is a design target, not present support. The largest run to date has two GPUs, or ten host processes. But the target already shapes the mechanisms that exist, because the easiest designs for two devices are the ones that fail at scale:

• a flat coordinator;
• an all-to-all control mesh;
• a central process that collects every tensor;
• a rank table in every participant.

This page covers what the target demands, which of its representations already exist and are checked, and what each kind of scale claim would have to show. The full design is chapter 22 of the PRD.

Five different claims about scale#

Scale claims are easy to inflate, so the PRD separates them, and the tracker keeps each as its own task.

Only the first applies to 200,000 today.

Counting devices. The count is of distinct physical compute endpoints. Processes, streams, virtual partitions, CPU cores and repeated aliases of one device do not inflate it, and control-only hosts and idle registered devices do not count as contributors. A file of synthetic identities is not evidence that those devices ran.

What the final run must show. The final physical acceptance must be hybrid: at least two genuinely different target families doing necessary work. A homogeneous run of 200,000 devices would establish only its own scope.

Hierarchy instead of a mesh#

At 200,000 participants a directed peer mesh has 39,999,800,000 relationships. Alpha's answer is a bounded-fan-out hierarchy in which every structure a participant holds is bounded by its fan-out and depth, never by the total count.

The hierarchy laws, in ParticipantTreeLaws, are checked by definitional equality:

• 200,000 roles at fan-out 64 need exactly three edge levels, because 64² < 200,000 ≤ 64³.
• The root has 64 children; the first subtree holds 3,125 roles and the last holds 3,124.

Partial groups keep exact coverage and never allocate padding participants.

These are representations, checked by the type checker. They are what a fleet profile would scale; executing them at fleet scale is separate work.

What heterogeneity changes#

Admission at scale works on each region's requirements, not on the intersection of every device's instruction set. One region may use a specialized matrix instruction, another a checked scalar realization, and a third a non-gradient state transition. Each is admitted on the targets that can realize it.

Unsupported regions are still refused. Missing GPU support is never reinterpreted as CPU work, but an explicitly admitted, Alpha-compiled CPU realization is a legitimate participant.

Placement has to weigh several things at once:

• compute;
• resident state;
• shared memory: a unified-memory host cannot promise its capacity separately to its CPU and GPU;
• transfer staging;
• communication rate;
• synchronization frequency.

Grouping by vendor is neither required nor sufficient. Network proximity and capability can matter more.

A synchronous learner is still bounded by its slowest necessary contribution. Five mechanisms could hide stragglers or raise utilization:

• speculative replay;
• local learning steps;
• reduced participation;
• asynchronous updates;
• communication compression.

They are distinct, and the last four can change what is learned. Each needs an explicit contract of its own, and none may be switched on automatically to make utilization look better.

Communication and failure at scale#

Several design rules cover communication and failure:

• No central dispatch. There is no fleet-wide RPC per kernel. Each role runs its own compiled schedule and launches work when its versioned dependencies are ready. Hierarchical control handles admission, checkpoints and failures, not every tensor operation.
• Scoped collectives. A collective covers only the group its operation names; nothing becomes fleet-wide by default. Native plans must account for NIC affinity, staging, descriptor bounds and incast, with credits so a receiver cannot be overwhelmed.
• The socket path is a baseline. The host-staged TCP path that exists today is a correctness baseline. A high-performance fleet would need Alpha-owned NIC, RDMA or registered-memory realizations, each with narrow ABI and memory-safety contracts. No external collective library becomes the hidden definition of distributed learning.
• Failure is a normal planning input. Membership epochs, durable authority beyond one publisher, and shard placement across failure domains are part of the fleet profile, not afterthoughts.

How it would be qualified#

The PRD qualifies scale in steps that each expose a new boundary, rather than in a sweep over every power of two:

1. Two devices. The requested 3070s, then an unequal pair. Both have passed; the unequal pair has not been rerun since 3 October.
2. A small group. Combined partitions, routing, and native failure and recovery.
3. Multiple groups. Hierarchical control, distributed checkpoint storage and correlated failures.
4. 200,000 represented hybrid participants. Irregular capability mixes and simulated protocols, using compact generators rather than 200,000 processes on one host.
5. A larger physical fleet.
6. 200,000 physical heterogeneous devices. Full learning, a distributed checkpoint, recovery and resumed evaluation, with exact evidence of membership and the device mix.

The tracker holds this as milestones M9 (composable parallelism mechanisms), M10 (scalable representations, control and storage) and M11 (physical fleet qualification, G01–G08). All of M11 is planned. No modelled time, assumed bandwidth or handful of small runs substitutes for the physical acceptance.

Read next#

• Participants, endpoints and framesDistributed · explanationParticipants, endpoints and framesRoles and compact membership, star and hierarchy topologies, native connection setup, bounded identity-checked frames, and protocols projected to every role.: the membership and hierarchy representations in use.
• Distributed checkpoints and resumeDistributed · explanationDistributed checkpoints and resumeLogical shard manifests, immutable publication, manifest trees, path-only restore, and resuming on a different owner count.: the manifest tree.
• Distributed learningDistributed · explanationDistributed learningHow one learning system becomes cooperating executables across several GPUs, hosts and instruction sets, and what has been checked, run and physically qualified.: the overview.