The activation commonly called SiLU or swish, expressed through the
available exp2 and reciprocal operations. The provider determines the
approximations and every intermediate rounding; the GPU realization must
keep this order so the semantic and machine models agree. In particular,
the derivative below is analytic, not a derivative of rounded f32 bits.
25def sigmoidFromExp2With =
26 (lambda unrestricted operations : Float32Operations . unrestricted x : Nat .
27 let unrestricted scaled =
28 (f32Multiply operations (f32Negate operations x) float32Log2E)
29 in
30 let unrestricted exponential =
31 (f32Approximate operations sigmoidWeightedGateExp2 scaled)
32 in
33 f32Approximate
34 operations
35 f32ApproximateReciprocal
36 (f32Add operations sigmoidWeightedGateOne exponential))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.