layer l's backward: `current` holds dy entering the layer from above and
`other` the other residual gradient plane; the layer is recomputed first,
then in order: the feed-forward output projection; the gated GELU, the up
and gate projections and the second norm; the attention output
projection; the heads' shared layouts, then the heads' attention
backward (cgStreamingAttentionBackward); the qkv projection and the first
norm
2353def cgLayerBackward =
2354 (lambda unrestricted l : Nat .
2355 (lambda unrestricted current : Nat .
2356 (lambda unrestricted other : Nat .
2357 (let unrestricted x =
2358 (cgBoundary l)
2359 in
2360 (cgThen
2361 (cgPhase
2362 cgPhaseFfnDown
2363 (cgThen (cgCastDown cgDownDyHalf current (naturalMultiply cgSeq cgWidth)) (cgThen (cgProductData cgSeq cgFfn cgWidth cgGatedGradient cgDownDyHalf (cgLayerHalf l cgMatrixDown)) (cgProductWeight cgWidth cgFfn cgSeq (cgLayerGradient l cgItemDown) cgDownDyHalf (cgGatedHalfOf l)))))
2364 (cgThen
2365 (cgPhase
2366 cgPhaseFfnGate
2367 (cgThen (cgGatedGeluBackward cgFfnGradientHalf cgGateGradientHalf cgGatedGradient (cgGeluOf l) (cgUpOf l) (cgGateOf l) (naturalMultiply cgSeq cgFfn)) (cgThen (cgProductWeight cgFfn cgWidth cgSeq (cgLayerGradient l cgItemUp) cgFfnGradientHalf (cgNorm2HalfOf l)) (cgThen (cgProductData cgSeq cgWidth cgFfn cgNorm2UpGradient cgFfnGradientHalf (cgLayerHalf l cgMatrixUp)) (cgThen (cgProductWeight cgFfn cgWidth cgSeq (cgLayerGradient l cgItemGate) cgGateGradientHalf (cgNorm2HalfOf l)) (cgThen (cgProductDataAdding cgSeq cgWidth cgFfn cgNorm2Gradient cgGateGradientHalf (cgLayerHalf l cgMatrixGate) cgNorm2UpGradient) (cgThen (cgLayerNormBackward
2368 (cgLayerParameter l cgItemNorm2Gain)
2369 (cgLayerGradient l cgItemNorm2Gain)
2370 (cgLayerGradient l cgItemNorm2Shift)
2371 (cgProjectionOf l)
2372 cgNorm2Gradient
2373 (cgNorm2StatisticsOf l)
2374 cgNorm2InputGradient) (cgBinary
2375 cgImageAdd
2376 other
2377 current
2378 cgNorm2InputGradient
2379 (naturalMultiply cgSeq cgWidth)))))))))
2380 (cgThen
2381 (cgPhase
2382 cgPhaseAttentionOutput
2383 (cgThen (cgCastDown cgOutputDyHalf other (naturalMultiply cgSeq cgWidth)) (cgThen (cgProductWeight cgWidth cgWidth cgSeq (cgLayerGradient l cgItemOutput) cgOutputDyHalf (cgMergedHalfOf l)) (cgProductData cgSeq cgWidth cgWidth cgMergedGradient cgOutputDyHalf (cgLayerHalf l cgMatrixOutput)))))
2384 (cgThen
2385 (cgPhase
2386 cgPhaseHeadLayout
2387 (cgThen
2388 (cgLaunch
2389 cgImageTokenToHead
2390 cgSeq
2391 1
2392 1
2393 (cgPointer
2394 (cgArgument 0)
2395 cgContextGradientHeads
2396 (cgPointer (cgArgument 2) cgMergedGradient cgNoSlots)))
2397 (cgThen
2398 (cgHeadTranspose cgHeadWidth cgSeq cgValuesHeads (cgValuesTOf l))
2399 (cgThen
2400 (cgHeadTranspose cgSeq cgHeadWidth cgQueriesT (cgQueriesOf l))
2401 (cgThen
2402 (cgHeadTranspose cgSeq cgHeadWidth cgKeysT (cgKeysOf l))
2403 (cgHeadTranspose
2404 cgSeq
2405 cgHeadWidth
2406 cgContextGradientT
2407 cgContextGradientHeads))))))
2408 (cgThen
2409 (cgPhase cgPhaseAttentionHead (cgStreamingAttentionBackward l))
2410 (cgPhase
2411 cgPhaseQkv
2412 (cgThen (cgLaunch
2413 cgImageInverseRope
2414 cgSeq
2415 1
2416 1
2417 (cgPointer
2418 (cgArgument 0)
2419 cgQkvGradientHalf
2420 (cgPointer
2421 (cgArgument 1)
2422 cgQueriesGradient
2423 (cgPointer
2424 (cgArgument 2)
2425 cgKeysGradient
2426 (cgPointer
2427 (cgArgument 3)
2428 cgCosine
2429 (cgPointer
2430 (cgArgument 4)
2431 cgSine
2432 (cgPointer (cgArgument 5) cgValuesGradient cgNoSlots))))))) (cgThen (cgProductWeight (naturalMultiply cgQkvParts cgWidth) cgWidth cgSeq (cgLayerGradient l cgItemQkv) cgQkvGradientHalf (cgNorm1HalfOf l)) (cgThen (cgProductData cgSeq cgWidth (naturalMultiply cgQkvParts cgWidth) cgNorm1Gradient cgQkvGradientHalf (cgLayerHalf l cgMatrixQkv)) (cgThen (cgLayerNormBackward
2433 (cgLayerParameter l cgItemNorm1Gain)
2434 (cgLayerGradient l cgItemNorm1Gain)
2435 (cgLayerGradient l cgItemNorm1Shift)
2436 x
2437 cgNorm1Gradient
2438 (cgNorm1StatisticsOf l)
2439 cgNorm1InputGradient) (cgBinary
2440 cgImageAdd
2441 current
2442 other
2443 cgNorm1InputGradient
2444 (naturalMultiply cgSeq cgWidth))))))))))))))))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.