Qualifying on real hardware#

Most of Alpha's distributed machinery is checked by the type checker and exercised by native host processes in CI. None of that shows that a real driver, a real GPU and a real network behave as the compiled plan assumes. A physical qualification answers that question for one learner on one pair of devices. It runs the exact executables the compiler produced, checks every result against an independent oracle, and keeps the evidence.

This page covers how a qualification runs, what it proves and does not prove, and every physical result to date.

The sequence#

The physical-qualification chapter fixes the order. Each step proves something the previous one could not.

A 3070 + 3090 run proves unequal NVIDIA placement. It does not prove cross-vendor support.

On 14 October 2026 the host executables became much smaller without changing what they execute. Repeated routine code is stored once, unused operand slots are no longer stored, SHA-256 is a compact loop, and the x86-64 command image is compressed and unpacked once at start-up. A GPU participant went from 1,272 KiB to about 55 KiB, and the GPU programs are unchanged, byte for byte. GPU executables built since then have passed every host gate but have not yet run on GPUs; the next physical run will cover them.

The driver#

scripts/alpha/partitioned-device-qualification.py deploys the executables and collects the evidence. It is outside the compilation and execution path:

• It supplies no computation to the learner.
• It never rents, stops or deletes a pod.
• A failure preserves every output and never touches anyone else's pods.

It runs one of three lanes on two task-owned GPUs:

A lane runs eight steps:

1. Preflight. Over SSH (in BatchMode), it captures nvidia-smi CSV and XML and the list of compute processes. It requires that:

   • the selected minor is unique;
   • the product is an RTX 3070 or 3090;
   • the driver is a 580 release;
   • no other compute process is on the GPU;
   • the two UUIDs differ.

   It writes binding.json.

2. Build. It builds each role locally with alpha build --artifact-value, binding the card's minor and UUID (Hybrid hosts and multiple GPUsDistributed · explanationHybrid hosts and multiple GPUsOne executable per role and target, x86-64 and AArch64 participants, binding a GPU by UUID, unequal devices, and memory admission across processes.), and runs alpha verify elf. It uploads each executable and checks its SHA-256 on the remote.

3. Refusal controls. One build for the wrong UUID and one for the wrong product must exit with device-uuid and nvidia-plan-host, write no checkpoint, and produce no output.

4. Uninterrupted run. Five updates, from fresh processes.

5. Split run. Three updates, a checkpoint, then two more in new processes resumed from it. The resumed checkpoint must equal the uninterrupted one, with the membership epoch advanced from 1 to 2.

6. Host-shadow control (coordinates and independent lanes). After the first device upload, a control build overwrites the host's copies of the parameters and accumulators with distinct finite values before every forward pass. The GPU trajectory and the published checkpoint must not change. This proves the video-memory state is the authority, not the host shadow.

7. Fresh evaluation. A new process predicts or evaluates from the resumed checkpoint, on the GPU for the coordinates and independent lanes. A corrupted checkpoint must be refused (checkpoint-chunk).

8. Record. It writes qualification.json, with the SHA-256 of every file it produced.

For the coordinates and independent lanes, each process's host and video quotas from resource_reservations become resource-graphDistributed · explanationHybrid hosts and multiple GPUsOne executable per role and target, x86-64 and AArch64 participants, binding a GPU by UUID, unequal devices, and memory admission across processes. data, so admission runs against the real machine's limits.

Configuring a run#

Two GPUs in one pod use the default loopback transport:

{
  "owner": "your-task-name",
  "pod_ids": ["your-pod-id"],
  "host": "root@pod-address",
  "port": 22,
  "ssh_key": "/absolute/private-key-path",
  "minors": [2, 3],
  "remote_root": "/tmp/alpha-hybrid-your-unique-run",
  "transport": "loopback"
}

Two hosts use "transport": "ssh-tunnel" and a nodes list in role order (Participants, endpoints and framesDistributed · explanationParticipants, endpoints and framesRoles and compact membership, star and hierarchy topologies, native connection setup, bounded identity-checked frames, and protocols projected to every role.).

Run a lane through its gate, so the session is recorded as a sealed AOS stream (ObservabilityBuild · explanationObservabilityRecording any command as an AOS stream, reading it back, and how receipts and gates reach their verdicts.):

ALPHA_COORDINATE_DEVICE_CONFIG=config.json \
  alpha observe gate coordinate-device -- bash scripts/ci/coordinate-device.sh OUT

The three device gates (distributed-device, coordinate-device and independent-device) always build and check their static GPU halves. They run the physical lane only when ALPHA_PARTITIONED_DEVICE_CONFIG, ALPHA_COORDINATE_DEVICE_CONFIG or ALPHA_INDEPENDENT_DEVICE_CONFIG names a config. CI sets none of them, so CI never rents or touches a GPU.

Running a session responsibly#

The chapter's rules for a rental, which every run so far has followed:

• Before renting:
  • Give every pod, output directory and evidence path a unique, task-owned name.
  • Set a stop condition, a maximum duration and a spending cap.
• While running: check once that real updates advance, then let the bounded run finish.
• Afterwards:
  • Copy results off the pod before releasing it.
  • Stop or delete only this task's pod, by its ID. Never use a broad "stop all pods".
  • Record the provider's acceptance of a stop separately from the verified stopped state.

What a pass establishes#

A passing lane establishes that:

• two named physical GPUs, identified by UUID, each executed their necessary share of every update;
• the committed state equals the oracle's;
• the state survived a checkpoint and fresh processes;
• the wrong device, the wrong product and corrupted state were refused.

It does not establish:

• performance (no throughput claim is made from any of these runs);
• behaviour at other sizes, devices or drivers;
• a direct GPU-to-GPU path (transfers are host-staged);
• cross-vendor support.

The pod's preflight records other load on the device, but no claim about interference is inferred from it.

Rebuild identity. After the 13 October run every one of the 31 executables that ran was rebuilt from the committed merge (6becd83c9) and compared byte for byte, so the evidence describes committed code, not an uncommitted tree.

Every physical result#

All records are under docs/multi-device-learning/evidence/.

Every pod used by these runs has been deleted.

The tracker#

docs/multi-device-learning/tracker.py keeps the programme's state in progress.sqlite3, outside Alpha's compilation and execution path:

• milestones, work items and dependencies;
• experiments, runs and devices;
• evidence and measurements.

Its rules:

• Runs are immutable once they reach a terminal state.
• A physical pass needs a completed run with a linked AOS stream.
• Status is honest. Implemented means code exists and focused checks pass; verified means acceptance evidence is attached. A hardware item needs physical evidence.

cd docs/multi-device-learning
python3 tracker.py status
python3 tracker.py next
python3 tracker.py show N10

As of 13 October 2026 it holds:

• 160 work items across 11 milestones: 5 implemented, 46 in progress, 109 planned;
• none verified;
• 11 completed runs, and 3 failed ones that were kept;
• 10 passing physical evidence records and 3 inconclusive ones.

Read next#

• Distributed learningDistributed · explanationDistributed learningHow one learning system becomes cooperating executables across several GPUs, hosts and instruction sets, and what has been checked, run and physically qualified.: what the whole programme covers, and its status.
• Evidence and statusEvaluate · analysisEvidence and statusHow to read checked, run and measured claims, and what each one does not establish.: Alpha's assurance ladder.