Hybrid hosts and multiple GPUs#

The same learning system can put its participants on different kinds of machine:

• two identical GPUs;
• two unequal GPUs on separate hosts;
• a mix of x86-64 and AArch64 host processes.

None of that is written into the model or the learner. It is decided by pairings (which target and device profile each role is built for) and by deployment inputs (which card, which address), and the compiler checks the result before it emits a byte.

This page covers:

• how one system becomes one executable per role and per target;
• how a process binds exactly one GPU and refuses any other;
• how unequal devices share work;
• how memory is admitted across several processes and devices.

Targets, devices and pairingsBuild · explanationTargets, devices and pairingsTarget packages, evidence-carrying device profiles, admission of demands against supplies, and the derived host. explains profiles, demands and admission for a single device. This page builds on it.

One program, one executable per role and target#

A distributed system derives one target-neutral program per role, NativePhysicalProgram. A Runtime.NativeHostTarget then turns that program into a self-contained executable for one instruction set and Linux ABI. Two host targets exist:

Every participant's program comes from the same derivation, and the bytes they exchange and publish are defined by that program, not by the target: wire and checkpoint encodings are little-endian, fixed-width and identical on every admitted host. That is why one run can mix architectures:

• Every learner in hybrid-host-learning trains with x86-64 and AArch64 roles together. It checkpoints, resumes with every owner swapped to the other instruction set, and evaluates.
• The checkpoints are byte-identical to an all-x86-64 run.

A target that cannot realize some command refuses the whole program; the ELF writer then rejects the empty bytes. No executable mixes instruction sets, and nothing is silently dropped or routed through another target. A new host architecture is a new instance beside its platform owner, never a branch inside the shared module.

AArch64 execution so far is emulated: it is evidence about encoding, ABI and protocol, not about physical cores, and it carries no timing claim. The DGX Spark's AArch64 host has run single-device GB10 work, but not a multi-participant run.

A GPU per process, bound by identity#

Each GPU participant is a process with exactly one bound GPU. Platform.Linux.Nvidia.DeviceBinding keeps deployment selection separate from target capability:

-- Deployment selection is distinct from target capability and memory layout.
-- A Linux minor is not an RM device instance. The host derives the latter
-- through CARD_INFO -> GPU_GET_ID_INFO_V2; it never probes allocation handles.
-- UUID is empty only for an explicit node-only binding. Distributed deployments
-- should pin the 16-byte binary UUID captured in their admitted inventory.
family NvidiaDeviceBinding : Type 0
constructor NvidiaDeviceBindingValue
field unrestricted nvidiaDeviceMinor : Nat
field unrestricted nvidiaDeviceExpectedUUID : Bytes

end-family

A binding names a Linux minor (the /dev/nvidiaN node, 0 to 247) and the 16-byte binary UUID expected there. At start-up the derived host checks four things:

1. The inventory. It reads the driver's card inventory (at most 32 rows) and requires exactly one row matching the minor, so a duplicate can never select an arbitrary GPU.
2. The instance. It obtains that card's driver instance through GPU_GET_ID_INFO_V2, never by probing allocation handles.
3. The UUID. It compares the card's GID with the binding's UUID. A mismatch exits with device-uuid.
4. The product. It compares the card's name string with the target profile's product identity. A mismatch exits with nvidia-plan-host.

A refused process exits nonzero with empty standard output, and its standard error is just the label. It writes no candidate and no committed checkpoint. The physical runs record these refusals as controls; for example, wrong-uuid.stderr contains only device-uuid.

The older single-device entry points mean exactly card 0 (nvidiaDeviceZero). They never fall through to whichever card opened last.

Pairing each role#

In a multi-device system each per-device build module applies one shared artifact function to one profile. From PartitionedRegression.Build.RTX3070:

module PartitionedRegression.Build.RTX3070

import PartitionedRegression.Build.NativeHost
import Platform.RunPod.Nvidia.SM86

-- The system's requirements and the target's facts meet at this pairing.
-- Deployment supplies exact UUID/minor, role, endpoint and run policy.
def partitionedRTX3070Artifact =
  (partitionedDeviceArtifact runPodRTX3070TargetProfile)

RTX3090.alpha is the same module with runPodRTX3090TargetProfile. The profiles in Platform.RunPod.Nvidia.SM86 carry each card's product identity, compute capability 8.6, total video memory (8 GiB and 24 GiB), and the video memory the driver reserves (352 MiB and 452 MiB, measured with driver 580.126.20).

partitionedDeviceArtifact takes the rest as compile-time deployment inputs:

• the profile and the device binding;
• the role, the peer address and port;
• whether to resume, and how many steps to run.

The qualification driverDistributed · explanationQualifying on real hardwareThe physical qualification driver, its refusal and host-shadow controls, the tracker, and every two-GPU result to date. builds each role with one command, then verifies the result:

alpha build role-1.elf --profile release --artifact-format elf --root ROOT \
  --artifact-value '(partitionedDeviceArtifact runPodRTX3070TargetProfile
    (constructor NvidiaDeviceBinding NvidiaDeviceBindingValue 2 b"\x1f\x8a...")
    1 ipv4Loopback 41733 0 5)'
alpha verify elf role-1.elf

The UUID is written as its sixteen bytes. Here role 1 binds minor 2, connects to its peer's loopback port, starts fresh (0), and runs five updates.

So a 3070 and a 3090 in one run are two pairings of one system, each admitted against its own profile, and each executable is bound to one card.

Unequal devices, unequal work#

Equal work on unequal devices proves little, so the samples-partitioned learner is deliberately uneven:

• The coordinator holds one sample and the worker holds two.
• The combine adds (Σ gradient, Σ samples), so the update divides by the total mass, 3, rather than averaging the two devices.
• The driver checks that each role's GPU issued steps × (role + 1) gradient submissions, so both devices did their necessary share.

On the RTX 3070 + RTX 3090 pair the run was repeated with the roles swapped, so correctness cannot depend on which product holds which share. A mutant that takes the unweighted device mean (9/16) is refused by the oracle (1/2).

The coordinate-owned learner adds the memory side of placement: each owner's coordinate, parameter and accumulator live in that GPU's video memory. In one qualification run the RTX 3090 acted as coordinator and resumed a checkpoint written on two RTX 3070s, then predicted from it. A placement change across products is a restore, not a special case.

Admitting memory across devices#

Each executable admits its own resources at compile time. Across processes the question becomes which memory is shared and which is not. Two layers answer it.

A typed resource graph. Execution.ResourceGraph describes endpoints, directed links with size envelopes, capacity domains and relation domains. A fact in it is either established with a reference or absent with a reason. Placements map roles to endpoints, and leases occupy half-open intervals of a schedule.

admitResourceSchedule in Compiler.Planning.Resources refuses, in order:

1. an exceeded planning budget;
2. a malformed graph;
3. an invalid placement or lease;
4. a capacity whose limit is unknown;
5. an exceeded capacity.

A route needs an explicit link. Two participants on the same host do not thereby gain a peer path, and a link in one direction does not imply its reverse.

NVIDIA leases. Platform.Linux.Nvidia.ResourcePlan charges every allocation the derived host makes:

• each device buffer, to the video or host domain its backing names;
• the parameter mapping;
• the error notifier, charged even when it is unmapped;
• recipe staging;
• the interpreter's own mapping, charged once.

Under UVM, an alias of one buffer through another mapping is charged once, not twice. If admission fails, the host generator emits no executable.

The laws in ResourcePlanContract state the multi-process cases exactly:

On the GPU pods the qualification driver supplies each process's host and video quotas as resource-graph data for the coordinate and independent lanes, so the same admission runs against the real machine's limits.

Not built yet#

• More than one GPU session in a process, and more than two GPU participants in one run.
• A GPU realization of the data-parallel and partitioned learners. They are host-only today.
• Native AArch64 participants on physical cores.
• Other vendors. AMD has a measured device profile and no realization or host; there is no Apple target. Cross-vendor execution is the next physical milestone after the NVIDIA pairs.
• Placement and transport:
  • automatic placement: ranking by estimates, bounded search, adaptive shares and pipeline cuts;
  • distributed capacity enforcement across hosts;
  • direct GPU peer, P2P or RDMA transfers.

Read next#

• Qualifying on real hardwareDistributed · explanationQualifying on real hardwareThe physical qualification driver, its refusal and host-shadow controls, the tracker, and every two-GPU result to date.: how these executables are run and judged on rented GPUs.
• Targets, devices and pairingsBuild · explanationTargets, devices and pairingsTarget packages, evidence-carrying device profiles, admission of demands against supplies, and the derived host.: single-device admission.
• Hardware: every device and its evidence.