the up (then the gate) projection's binary16 weights transposed (width x
ffn): the second norm's gradient is dup Wup + dgate Wgate, and the product
takes its right operand as B^T -- the ffn x width copy itself would be
read as W^T
743def cgFfnWeightT : Nat =
744 (cgNext cgNorm2UpGradient cgPlaneBytes)The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.