Source/Packages

Component.SigmoidWeightedGate

packages/components/src/Component/SigmoidWeightedGate.alpha

181 lines14 declarations6.6 KiBSHA-256 7302f612c9f7

def · lines 25–36

sigmoidFromExp2With

Full file
The activation commonly called SiLU or swish, expressed through the available exp2 and reciprocal operations. The provider determines the approximations and every intermediate rounding; the GPU realization must keep this order so the semantic and machine models agree. In particular, the derivative below is analytic, not a derivative of rounded f32 bits.
25def sigmoidFromExp2With =
26  (lambda unrestricted operations : Float32Operations . unrestricted x : Nat .
27    let unrestricted scaled =
28      (f32Multiply operations (f32Negate operations x) float32Log2E)
29      in
30      let unrestricted exponential =
31        (f32Approximate operations sigmoidWeightedGateExp2 scaled)
32        in
33        f32Approximate
34          operations
35          f32ApproximateReciprocal
36          (f32Add operations sigmoidWeightedGateOne exponential))

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.