Learning.Learner.MomentNormalized

A CPU (Std.Float F32) learner that moves each parameter by its bias-corrected first moment divided by the root of its bias-corrected second moment: m' = beta1 m + (1 - beta1) g firstMomentValue v' = beta2 v + (1 - beta2) (g g) secondMomentValue p' = p beta momentDecayPowerNext m^ = m' / (1 - p1'), v^ = v' / (1 - p2') correctMomentBias w' = w - (rate m^) / (sqrt v^ + epsilon) normalizedStepValue This is the update known as Adam (Kingma & Ba 2015, Algorithm 1) without weight decay. It differs from Learning.Checked.AdamW, the SM86 owner, on purpose: there the moments use fused multiply-adds, the reciprocal square root is the MUFU approximation and the bias corrections are folded into a precomputed step size; here every operation is a correctly rounded binary32 add, subtract, multiply, divide or sqrt in the order written above, so a CPU run is reproducible bit for bit by any IEEE-754 implementation that keeps that order (systems/manfred checks it against a Nim reference). The complements 1 - beta are derived here in F32 rather than passed in: 1 - 0.9 rounds to 0.100000024, not to the literal 0.1, and a learner that took the complement as a separate hyperparameter could silently disagree with its own beta. The bias powers beta^t are carried by the caller as running products (momentDecayPowerNext), which needs no pow.

30family MomentNormalizedHyper : Type 0
31constructor MomentNormalizedHyperValue
32field unrestricted momentNormalizedRate : F32
33field unrestricted momentNormalizedBeta1 : F32
34field unrestricted momentNormalizedBeta2 : F32
35field unrestricted momentNormalizedEpsilon : F32