Per-coordinate ownership for a data-parallel learner whose update is
separable (Learning.DataParallel.dataParallelSeparable) with q words of
optimizer state per coordinate: coordinate i is owned by owner i mod n,
and only its owner holds that coordinate's optimizer state and updates it.
The shared state words stay replicated. `p` chooses the parameters'
residency: replicated on every owner (p = 0, optimizer-state
partitioning) or partitioned with the optimizer state (p = 1, parameter
partitioning: a coordinate's parameter is one more word its owner holds).
One update with replicated parameters, on a star of owners 0..n-1:
every owner: its samples' (numerator, mass) leaf;
sum: the declared left-deep binary64 sum at owner 0, replicated
(Runtime.NativeCombine);
every owner: means, then only the shared assignments and its own
coordinates' assignments of the update program
(dataParallelOwnedProgram), in program order;
assemble: every owner offers its parameters with an ownership mask; a
second ordered combine assembles them by selection, never by
arithmetic, refusing two owners for one coordinate, and replicates the
result; every owner checks that every coordinate was owned and installs
the assembled parameters.
With partitioned parameters the assembly gathers them before they are
read, and the gathered copy is an explicit, short-lived read replica:
assemble: every owner offers its own parameters (the others +0) with its
ownership mask; the assembly gathers and replicates them;
materialize: every coordinate was owned; the result becomes this owner's
read replica, tagged with the version it was gathered at
(MaterializeReadVersion);
every owner: its leaf, reading only a held replica of the current version;
release: the replica is cleared and no longer held (ReleaseReadReplica);
sum, then means and the owned assignments on the owner's own parameter
and state words; a replica still held when the version advances
refuses (`data-parallel-replica-retained`).
The replica is held across all of the owner's samples (one gathered version
per update) and is part of the owner's peak storage, not its persistent
state. Either way the results equal the replicated realization's bit for
bit: separability makes each owned coordinate's results independent of the
others.
Persistence: owner 0 publishes the replicated words (parameters when
replicated, shared state); every owned word is a shard of its owner (the
star publication prepares each owner's shards from that owner's capture).
Restore reads the replicated words and only the restoring owner's own
words. Publication requires every owner's capture prefix and replicated
words to agree. Partitioned checkpoints bind the learner's
placement-independent identity, not its split of samples or its owner
count, so a saved state resumes on another number of owners.
110field unrestricted dataParallelOwnedGradientOperation : (family OrderedCombineOperation (family RegionTransferIdentity))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.