Source/Packages

Component.CubicTanhGateSigmoid

packages/components/src/Component/CubicTanhGateSigmoid.alpha

179 lines15 declarations6.9 KiBSHA-256 d73f403dc6e0

def · lines 28–29

cubicTanhSigmoidScale

Full file
GPT-2's activation ("gelu_new"), x/2 (1 + tanh u) with u = sqrt(2/pi) (x + 0.044715 x^3), evaluated through the identity (1 + tanh u) / 2 = sigmoid(2u). The value is x sigmoid(v) with v = x (c1 + c2 x^2), c1 = 2 sqrt(2/pi), c2 = 0.044715 c1, and its pullback dy (s + x s (1 - s) (c1 + 3 c2 x^2)) with s = sigmoid(v). The sigmoid uses the exp2 and reciprocal operations of Component.SigmoidWeightedGate, so no approximate tanh enters: on SM86 MUFU.TANH carries a relative error near 2^-11, while EX2 and RCP are within a few ulp. Every intermediate rounding is explicit, and the pointwise forms below keep exactly this operation order. The three binary32 words are the nearest to the real constants: each rational below is within 10^-16 of its constant and at least 0.017 ulp from a rounding boundary, so its correctly rounded binary32 word is the constant's (c1 0x3fcc422a, c2 0x3d922279, 3 c2 0x3e5b33b6).
28def cubicTanhSigmoidScale : Nat =
29  (nat-add (float32ExactRational 15957691216057307 10000000000000000) 0)

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.