A data-parallel learner: one parameter vector and its optimizer state
(for instance momentum), both replicated on every owner; immutable samples
partitioned among owners in possibly unequal shares; a per-sample
contribution program; and an update program every owner applies
identically to the mean contribution.
One update, as a mathematical definition independent of placement:
numerator = sum over all samples of contribution(parameters, sample)
mass = number of samples
mean = numerator / mass
(parameters, state) := update(parameters, state, mean, rate)
The sum's evaluation order is not part of this definition. A realization
fixes it explicitly (an ordered combine tree, Runtime.OrderedCombine) and
its numerical contract names that order; a different order is a different
floating-point program, never a silent reassociation.
Slots, by position, with k parameters, m optimizer-state words and samples
of width s. The contribution program reads parameters [0, k) and one
sample [k, k + s) and writes only its contribution [k + s, 2k + s). The
update program reads parameters [0, k), the state [k, k + m), the mean
[k + m, 2k + m) and the rate at 2k + m, and writes only parameters and
state [0, k + m). The state is persistent learner state: checkpointed with
the parameters and required to agree across owners. The same typed
assignment semantics (Component.Pointwise) serve gradient and non-gradient
rules: a contribution need not be a gradient.
37family DataParallelSystem : Type 0
38parameter erased dataParallelContributionSlots : Nat
39parameter erased dataParallelUpdateSlots : Nat
40constructor DataParallelSystemOf
41field unrestricted dataParallelParameterCount : Nat
42field unrestricted dataParallelSampleWidth : Nat
43field unrestricted dataParallelContribution : (family StdList (family PointwiseAssignment StdU64 dataParallelContributionSlots))
44field unrestricted dataParallelUpdate : (family StdList (family PointwiseAssignment StdU64 dataParallelUpdateSlots))
45field unrestricted dataParallelInitial : (family StdList StdU64)
46-- the optimizer state's initial words (m = its length; none for plain steps)
47field unrestricted dataParallelInitialState : (family StdList StdU64)
48field unrestricted dataParallelRate : StdU64
49-- per owner, per sample, the sample's s words
50field unrestricted dataParallelSamples : (family StdList (family StdList (family StdList StdU64)))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.