---- AdamW's step-dependent scalars ----
Coppelius's learner defines them exactly (Coppelius.Build.Graph: the
binary32 nearest lr sqrt(1 - beta2^t) / (1 - beta1^t)); the host computes
them in binary64 from the same ratios, continuing the step count from the
checkpoint's header (scripts/ci/adamw-schedule.sh holds the two to each
other): the constants and the powers first, then one update's scalars
per step, written into the update's AdamW launches.
465def coppeliusHostAdamW : (family NvidiaPlanHostAdamW) =
466 (constructor NvidiaPlanHostAdamW NvidiaPlanHostAdamWValue
467 (constructor NvidiaPlanHostArithmetic NvidiaPlanHostBinary64)
468 (constructor NvidiaPlanHostConstant NvidiaPlanHostRatio coppeliusLearningRateNumerator coppeliusLearningRateDenominator)
469 (constructor NvidiaPlanHostConstant NvidiaPlanHostRatio coppeliusBeta1Numerator coppeliusBeta1Denominator)
470 (constructor NvidiaPlanHostConstant NvidiaPlanHostRatio coppeliusBeta2Numerator coppeliusBeta2Denominator)
471 (constructor NvidiaPlanHostConstant NvidiaPlanHostRatio (naturalMultiply coppeliusEpsilonNumerator coppeliusLossScale) coppeliusEpsilonDenominator))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.