Squared ReLU and its backward are ordinary compositions, written once
with the compute format's rounding as a parameter (`round` wraps every
value the source materializes in that format). Upstream rounds both
ReLU's result and the square; backward rounds 2*relu(x), then its product
with dy, then applies threshold-backward. Multiplying by a zero mask would
propagate NaNs and is not the same derivative program. In binary32 (no
rounding boundary, nanochat's MLP on the CPU: F.relu(x).square()) the
identity is passed; in BF16 the explicit PointwiseRoundBF16.
368def pointwiseReluSquaredRounded =
369 (lambda erased inputs : Nat .
370 unrestricted round : (pi unrestricted value : (family PointwiseExpression StdU32 inputs) . family PointwiseExpression StdU32 inputs) .
371 unrestricted x : (family PointwiseExpression StdU32 inputs) .
372 let unrestricted relu =
373 (round (pointwiseBinary StdU32 inputs PointwisePositiveGate x x))
374 in
375 round (pointwiseUnary StdU32 inputs PointwiseSquare relu))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.