module Model.LearnedPositionDecoder import Std.Natural -- The GPT-2 decoder family: a tied token embedding plus a learned position -- embedding, then pre-LayerNorm blocks (LayerNorm with gain and shift; -- multi-head causal attention through one fused, biased Q/K/V projection -- and a biased output projection; a biased feed-forward expansion, the -- cubic-tanh GELU and a biased reduction), a final LayerNorm, and logits -- through the tied embedding. Radford et al. 2019 and the released -- openai-community/gpt2 config.json fix the instance (Model.GPT2). -- -- `vocabulary` is the stored class count, `tokens` the classes that exist: -- a realization may pad the vocabulary to a multiple its objective needs; -- padded classes never occur as inputs or targets and a realization must -- hold their probabilities at zero. family LearnedPositionDecoderGeometry : Type 0 constructor LearnedPositionDecoderGeometryValue field unrestricted learnedPositionDecoderVocabulary : Nat field unrestricted learnedPositionDecoderTokens : Nat field unrestricted learnedPositionDecoderContext : Nat field unrestricted learnedPositionDecoderLayers : Nat field unrestricted learnedPositionDecoderWidth : Nat field unrestricted learnedPositionDecoderHeads : Nat field unrestricted learnedPositionDecoderHeadWidth : Nat field unrestricted learnedPositionDecoderFeedForwardWidth : Nat end-family def learnedPositionDecoderVocabularyOf = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . eliminate LearnedPositionDecoderGeometry (lambda unrestricted current : LearnedPositionDecoderGeometry . Nat) geometry (branch LearnedPositionDecoderGeometryValue vocab tokens context layers width heads head feed . vocab)) def learnedPositionDecoderTokensOf = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . eliminate LearnedPositionDecoderGeometry (lambda unrestricted current : LearnedPositionDecoderGeometry . Nat) geometry (branch LearnedPositionDecoderGeometryValue vocab tokens context layers width heads head feed . tokens)) def learnedPositionDecoderContextOf = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . eliminate LearnedPositionDecoderGeometry (lambda unrestricted current : LearnedPositionDecoderGeometry . Nat) geometry (branch LearnedPositionDecoderGeometryValue vocab tokens context layers width heads head feed . context)) def learnedPositionDecoderLayersOf = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . eliminate LearnedPositionDecoderGeometry (lambda unrestricted current : LearnedPositionDecoderGeometry . Nat) geometry (branch LearnedPositionDecoderGeometryValue vocab tokens context layers width heads head feed . layers)) def learnedPositionDecoderWidthOf = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . eliminate LearnedPositionDecoderGeometry (lambda unrestricted current : LearnedPositionDecoderGeometry . Nat) geometry (branch LearnedPositionDecoderGeometryValue vocab tokens context layers width heads head feed . width)) def learnedPositionDecoderHeadsOf = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . eliminate LearnedPositionDecoderGeometry (lambda unrestricted current : LearnedPositionDecoderGeometry . Nat) geometry (branch LearnedPositionDecoderGeometryValue vocab tokens context layers width heads head feed . heads)) def learnedPositionDecoderHeadWidthOf = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . eliminate LearnedPositionDecoderGeometry (lambda unrestricted current : LearnedPositionDecoderGeometry . Nat) geometry (branch LearnedPositionDecoderGeometryValue vocab tokens context layers width heads head feed . head)) def learnedPositionDecoderFeedForwardWidthOf = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . eliminate LearnedPositionDecoderGeometry (lambda unrestricted current : LearnedPositionDecoderGeometry . Nat) geometry (branch LearnedPositionDecoderGeometryValue vocab tokens context layers width heads head feed . feed)) -- the fused Q/K/V projection's output width def learnedPositionDecoderProjectionWidth = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalMultiply 3 (learnedPositionDecoderWidthOf geometry)) -- the heads tile the width; padding never removes a real class def learnedPositionDecoderAdmitted = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . eliminate LearnedPositionDecoderGeometry (lambda unrestricted current : LearnedPositionDecoderGeometry . Nat) geometry (branch LearnedPositionDecoderGeometryValue vocab tokens context layers width heads head feed . nest naturalAnd (naturalNonzero tokens) (naturalLessOrEqual tokens vocab) (naturalNonzero context) (naturalNonzero layers) (naturalNonzero heads) (naturalEqual width (naturalMultiply heads head)) (naturalNonzero feed))) -- ---- the parameter layout ---- -- Element offsets, independent of F32/BF16 storage, in two contiguous -- regions so a learner can treat them differently (GPT-2 practice decays -- the matrices and embeddings, not the biases and LayerNorm parameters): -- matrices: the token embedding [vocabulary][width], the position -- embedding [context][width], then per block the Q/K/V weight -- [3 width][width], the attention output weight [width][width], the -- expansion weight [feed][width] and the reduction weight [width][feed]; -- vectors: per block the attention LayerNorm's gain and shift, the Q/K/V -- bias, the output bias, the feed LayerNorm's gain and shift, the -- expansion bias and the reduction bias; then the final LayerNorm's gain -- and shift. -- A weight is stored [output][input], as the forward product reads it. def learnedPositionDecoderEmbeddingBase : Nat = 0 def learnedPositionDecoderEmbeddingElements = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalMultiply (learnedPositionDecoderVocabularyOf geometry) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderPositionBase = learnedPositionDecoderEmbeddingElements def learnedPositionDecoderPositionElements = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalMultiply (learnedPositionDecoderContextOf geometry) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderProjectionElements = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalMultiply (learnedPositionDecoderProjectionWidth geometry) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderOutputElements = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalMultiply (learnedPositionDecoderWidthOf geometry) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderFeedElements = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalMultiply (learnedPositionDecoderFeedForwardWidthOf geometry) (learnedPositionDecoderWidthOf geometry)) -- ---- the matrices ---- def learnedPositionDecoderBlockMatrixElements = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . nest naturalAdd (learnedPositionDecoderProjectionElements geometry) (learnedPositionDecoderOutputElements geometry) (naturalMultiply 2 (learnedPositionDecoderFeedElements geometry))) def learnedPositionDecoderProjectionBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . nest naturalAdd (learnedPositionDecoderPositionBase geometry) (learnedPositionDecoderPositionElements geometry) (naturalMultiply layer (learnedPositionDecoderBlockMatrixElements geometry))) def learnedPositionDecoderOutputBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderProjectionBase geometry layer) (learnedPositionDecoderProjectionElements geometry)) def learnedPositionDecoderFeedUpBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderOutputBase geometry layer) (learnedPositionDecoderOutputElements geometry)) def learnedPositionDecoderFeedDownBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderFeedUpBase geometry layer) (learnedPositionDecoderFeedElements geometry)) -- the first vector, and the matrices' count def learnedPositionDecoderVectorBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . learnedPositionDecoderProjectionBase geometry (learnedPositionDecoderLayersOf geometry)) def learnedPositionDecoderMatrixElements = learnedPositionDecoderVectorBase -- ---- the vectors ---- def learnedPositionDecoderBlockVectorElements = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . nest naturalAdd (naturalMultiply 6 (learnedPositionDecoderWidthOf geometry)) (learnedPositionDecoderProjectionWidth geometry) (learnedPositionDecoderFeedForwardWidthOf geometry)) def learnedPositionDecoderNormInputGainBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderVectorBase geometry) (naturalMultiply layer (learnedPositionDecoderBlockVectorElements geometry))) def learnedPositionDecoderNormInputShiftBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderNormInputGainBase geometry layer) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderProjectionBiasBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderNormInputShiftBase geometry layer) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderOutputBiasBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderProjectionBiasBase geometry layer) (learnedPositionDecoderProjectionWidth geometry)) def learnedPositionDecoderNormFeedGainBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderOutputBiasBase geometry layer) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderNormFeedShiftBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderNormFeedGainBase geometry layer) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderFeedUpBiasBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderNormFeedShiftBase geometry layer) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderFeedDownBiasBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . unrestricted layer : Nat . naturalAdd (learnedPositionDecoderFeedUpBiasBase geometry layer) (learnedPositionDecoderFeedForwardWidthOf geometry)) def learnedPositionDecoderFinalNormGainBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . learnedPositionDecoderNormInputGainBase geometry (learnedPositionDecoderLayersOf geometry)) def learnedPositionDecoderFinalNormShiftBase = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalAdd (learnedPositionDecoderFinalNormGainBase geometry) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderParameterCount = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalAdd (learnedPositionDecoderFinalNormShiftBase geometry) (learnedPositionDecoderWidthOf geometry)) def learnedPositionDecoderVectorElements = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalSaturatingSubtract (learnedPositionDecoderParameterCount geometry) (learnedPositionDecoderVectorBase geometry)) -- both regions are dense: a block's last matrix and vector end where the -- next block's begin def learnedPositionDecoderLayoutDense = (lambda unrestricted geometry : LearnedPositionDecoderGeometry . naturalAnd (pipe geometry (learnedPositionDecoderFeedDownBase 0) (naturalAdd (learnedPositionDecoderFeedElements geometry)) (naturalEqual (learnedPositionDecoderProjectionBase geometry 1))) (pipe geometry (learnedPositionDecoderFeedDownBiasBase 0) (naturalAdd (learnedPositionDecoderWidthOf geometry)) (naturalEqual (learnedPositionDecoderNormInputGainBase geometry 1))))