Participants, endpoints and frames#

A distributed Alpha run is a set of participants. Each is a separate native executable, compiled from the same learning system for one logical role. No coordinator process, launcher service or communication library gives the run its meaning: every participant's executable already contains its own connections, its own schedule of sends and receives, and the checks that refuse a message it was not compiled to accept.

This page covers:

• how roles are named;
• who connects to whom, and how bytes move between processes and hosts;
• what a message carries;
• what happens when something does not match.

Roles, not ranks#

A role is a logical identity, 0 to n − 1. It says which part of the computation a participant performs and which peers it exchanges with. It does not name a device or a host: the pairing decides which device a role runs on, and the deployment decides which machine runs its executable (Hybrid hosts and multiple GPUsDistributed · explanationHybrid hosts and multiple GPUsOne executable per role and target, x86-64 and AArch64 participants, binding a GPU by UUID, unequal devices, and memory admission across processes.).

Membership is fixed for a run. The participant count, every edge and every peer address are compiled into the executables. A peer that never appears exhausts its bounded connect and accept budgets, and the run aborts.

Compact membership#

Large memberships are described structurally rather than as a rank table. Representation.ParticipantGroup builds a selection from three forms:

A group is a universe, an epoch, identity bytes and a selection. Admission refuses:

• an empty selection;
• overlapping parts or out-of-order joins;
• a role outside the universe, or arithmetic overflow;
• a selection that exceeds its index budget.

Looking up a role follows one path through the structure, never a list of every member. The laws in ParticipantGroupLaws cover two shapes of 200,000 roles each:

def regular =
  (group (repeat 625 320 (repeat 8 40 (repeat 2 20 (stride 0 1 20)))))

def irregular =
  (group (join (stride 0 2 3) (repeat 3 11 (stride 10 3 2))))

The regular group covers all 200,000 roles with four index nodes; groups of unequal intervals are admitted the same way. This is representation, checked by the type checker. The largest membership that has actually executed is ten participants.

Topologies#

Three shapes are implemented. Each is an instance of the same edge mechanism.

The hierarchy is Representation.ParticipantTree. Roles are numbered in preorder, so every subtree is one contiguous range of roles. A node's descendants are split into at most F chunks whose sizes differ by at most one, larger chunks first.

Two consequences follow:

• A parent certifies that its children cover its subtree with O(F) range checks, without ever collecting the membership.
• Finding a role costs depth × fan-out, not the participant count.

ParticipantTreeLaws fixes the fleet design point by definitional equality. 200,000 roles at fan-out 64 need three edge levels (64² < 200,000 ≤ 64³). The root has 64 children, whose subtrees hold 3,125 roles each, except the last, which holds 3,124. Runtime.NativePeerTree gives each role at most F + 1 edges.

Establishing connections#

Runtime.NativePeerEndpoint owns connection setup. A participant's sessions are its incident edges, in ascending peer order. Each edge has one IPv4 address. The lower role of an edge listens; the higher role connects. Every edge has a static listener slot and a static connection slot, so the protocol layer names a peer's descriptor by role, never by a runtime lookup.

Every participant establishes its edges in the same order:

1. Bind and listen on every edge to a higher peer.
2. Announce those ports: two bytes each, big-endian, in edge order.
3. Connect to each lower peer.
4. Accept each higher peer, then close that listener.

Launching participants in role order, each after the previous one has announced, therefore never connects to an absent listener. A star and a pair are both special cases of this one mechanism.

The socket layer below it is shared, with an AArch64 twin in Platform.AArch64.SocketRoutine:

These are system calls emitted by the Alpha compiler. There is no MPI, NCCL or RCCL in the build or the run.

Same host and separate hosts#

On one host, each participant is its own process with at most one bound GPU, and participants talk over loopback TCP.

Across hosts, the qualification driverDistributed · explanationQualifying on real hardwareThe physical qualification driver, its refusal and host-shadow controls, the tracker, and every two-GPU result to date. has an ssh-tunnel transport, described in the partitioned-regression README:

• The deployment host opens two authenticated SSH sessions: a local forward to the coordinator and a remote forward from the worker.
• Every training listener binds loopback.
• The relay moves bytes only. The Alpha executables own the framing, the computation, the update and the checkpoint.

{
  "transport": "ssh-tunnel",
  "remote_root": "/tmp/alpha-hybrid-your-unique-run",
  "nodes": [
    {"pod_id": "owned-coordinator-pod", "host": "root@coordinator-ip", "port": 12345,
     "ssh_key": "/absolute/private-key-path", "minor": 0},
    {"pod_id": "owned-worker-pod", "host": "root@worker-ip", "port": 23456,
     "ssh_key": "/absolute/private-key-path", "minor": 2}
  ]
}

nodes are listed in logical role order; reversing them swaps which machine runs which role. In this realization the relay is a trusted, necessary component: its loss aborts the run. It is not a direct link between machines, and it is not Alpha-owned cryptography.

Host staging is explicit. A GPU learner reads its contribution back to the host, the host sends it over TCP, a host participant combines, and the result is uploaded again. Staging bytes through a host never licenses computing an unsupported GPU operation there; a missing GPU lowering is still a compile error.

Frames#

Every message between participants is a bounded frame, owned by Runtime.RegionFrame, with offsets derived by Compiler.Planning.RegionFrame. Version 1 contains:

A scheduled receive already knows every identity and extent, so the receiver derives the expected header itself. No length, address, allocation or route is ever taken from peer bytes. Runtime.NativeRegionFrame compares the header before it reads the body, and exposes the payload only after the digest checks.

A frame is refused if any of these is wrong:

• the frame or edge;
• the version: a replay, a skipped version, a double rearm, or overflow;
• the bytes: corruption, truncation or trailing bytes.

The SHA-256 trailer detects corruption. It does not authenticate an untrusted peer.

What a frame's identity binds#

The run identity of every frame names what the participant was compiled for:

• The learner. The data-parallel learners hash their programs, initial state, rate and samples once per artifact, and every frame names that digest.
• The numerical contract. The combine adds the contract's digest, including the topology and fan-out, which fix the floating-point order (Combining contributionsDistributed · explanationCombining contributionsOrdered combine trees instead of opaque collectives, delivery to every owner or one, numerical contracts, and how the declared order is shown to have run.).
• The membership. A group-covered combine adds the participant group's digest: universe, epoch, identity and selection.

A peer compiled for a different learner, rate, contract, combine tree or membership therefore refuses on the first frame, before any combined value exists.

One protocol, projected to every role#

Communication is written once, as a global protocol, and projected to a schedule per participant. Runtime.Protocol defines the global forms:

Compiler.Planning.Protocol checks the protocol before projecting it. The ProtocolFailure refusals include:

• a participant, sender or receiver out of range, or a self-transfer;
• an exceeded repeat bound;
• a local operation or transfer with no realization;
• a choice without a valid authority, with bad routes, or that leaves a participant uninformed.

Each participant receives only its own sends, receives and local work, and Runtime.NativeProtocol lowers that schedule to native code. Repetition stays compact: a 200,000-round repeat produces an artifact the same size as a four-round one.

Agreed choices. A choice names its authority, its selector, an ordered dissemination tree (each receiver introduced once) and two arms. The authority checks that its selector is 0 or 1 and sends the canonical word. Forwarders relay the word they received and never decide independently.

The protocol-choice gate refuses inverted choices, independent forwarders, noncanonical words and mismatched frame bindings, across three processes, one of them an emulated AArch64 participant.

When something goes wrong#

The failure policy is deliberately simple: a failed round aborts the whole round, and no new state is committed. There is no continuation with fewer participants, and no partial update over a smaller denominator. A timeout, a dropped message or a missing shard is a failed execution, never an approximation.

The checkpoint pageDistributed · explanationDistributed checkpoints and resumeLogical shard manifests, immutable publication, manifest trees, path-only restore, and resuming on a different owner count. explains what survives an abort: the previous published generation, unchanged.

The gates#

These run in CI (shards 7 to 9 of .github/workflows/ci.yml) and locally through alpha observe gate:

Not built yet#

• Membership and failure:
  • authenticated membership and admission;
  • dynamic membership, epoch transitions and role replacement;
  • continuing after a partial failure.
• Transport:
  • Alpha-owned transport cryptography;
  • credit-based flow control, and chunk retry with stable transfer identities;
  • a session-wide deadline (today there are only poll budgets);
  • a shared-memory transport;
  • direct GPU peer, DMA or RDMA paths.
• Planning: a planner whose cost does not grow with the participant count. Some route checks are still quadratic, and every participant builds the whole combine tree at compile time.

Read next#

• Combining contributionsDistributed · explanationCombining contributionsOrdered combine trees instead of opaque collectives, delivery to every owner or one, numerical contracts, and how the declared order is shown to have run.: ordered reductions and numerical contracts.
• Distributing learning stateDistributed · explanationDistributing learning stateThe distributed learners, and partitioning of optimizer state, parameters and gradients, each equal bit for bit to the replicated learner.: what the participants actually compute and own.
• Distributed learningDistributed · explanationDistributed learningHow one learning system becomes cooperating executables across several GPUs, hosts and instruction sets, and what has been checked, run and physically qualified.: the overview and the current evidence.