Learning.DataParallel

A data-parallel learner: one parameter vector and its optimizer state (for instance momentum), both replicated on every owner; immutable samples partitioned among owners in possibly unequal shares; a per-sample contribution program; and an update program every owner applies identically to the mean contribution. One update, as a mathematical definition independent of placement: numerator = sum over all samples of contribution(parameters, sample) mass = number of samples mean = numerator / mass (parameters, state) := update(parameters, state, mean, rate) The sum's evaluation order is not part of this definition. A realization fixes it explicitly (an ordered combine tree, Runtime.OrderedCombine) and its numerical contract names that order; a different order is a different floating-point program, never a silent reassociation. Slots, by position, with k parameters, m optimizer-state words and samples of width s. The contribution program reads parameters [0, k) and one sample [k, k + s) and writes only its contribution [k + s, 2k + s). The update program reads parameters [0, k), the state [k, k + m), the mean [k + m, 2k + m) and the rate at 2k + m, and writes only parameters and state [0, k + m). The state is persistent learner state: checkpointed with the parameters and required to agree across owners. The same typed assignment semantics (Component.Pointwise) serve gradient and non-gradient rules: a contribution need not be a gradient.

42field unrestricted dataParallelSampleWidth : Nat