Source/Packages

Component.Pointwise

packages/components/src/Component/Pointwise.alpha

408 lines49 declarations15.8 KiBSHA-256 b4c565ce95d3

def · lines 368–375

pointwiseReluSquaredRounded

Full file
Squared ReLU and its backward are ordinary compositions, written once with the compute format's rounding as a parameter (`round` wraps every value the source materializes in that format). Upstream rounds both ReLU's result and the square; backward rounds 2*relu(x), then its product with dy, then applies threshold-backward. Multiplying by a zero mask would propagate NaNs and is not the same derivative program. In binary32 (no rounding boundary, nanochat's MLP on the CPU: F.relu(x).square()) the identity is passed; in BF16 the explicit PointwiseRoundBF16.
368def pointwiseReluSquaredRounded =
369  (lambda erased inputs : Nat .
370    unrestricted round : (pi unrestricted value : (family PointwiseExpression StdU32 inputs) . family PointwiseExpression StdU32 inputs) .
371    unrestricted x : (family PointwiseExpression StdU32 inputs) .
372    let unrestricted relu =
373      (round (pointwiseBinary StdU32 inputs PointwisePositiveGate x x))
374      in
375      round (pointwiseUnary StdU32 inputs PointwiseSquare relu))

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.