layer l's backward: `current` holds dy entering the layer from above and
`other` the other residual gradient plane; the layer is recomputed first,
then in order: the feed-forward output projection; the gated GELU, the up
and gate projections and the second norm; the attention output
projection; the heads' shared layouts, then the heads' attention
backward (cgStreamingAttentionBackward); the qkv projection and the first
norm
2352def cgLayerBackward =
2353 (lambda unrestricted l : Nat .
2354 (lambda unrestricted current : Nat .
2355 (lambda unrestricted other : Nat .
2356 (let unrestricted x =
2357 (cgBoundary l)
2358 in
2359 (cgThen
2360 (cgPhase
2361 cgPhaseFfnDown
2362 (cgThen (cgCastDown cgDownDyHalf current (naturalMultiply cgSeq cgWidth)) (cgThen (cgProductData cgSeq cgFfn cgWidth cgGatedGradient cgDownDyHalf (cgLayerHalf l cgMatrixDown)) (cgProductWeight cgWidth cgFfn cgSeq (cgLayerGradient l cgItemDown) cgDownDyHalf (cgGatedHalfOf l)))))
2363 (cgThen
2364 (cgPhase
2365 cgPhaseFfnGate
2366 (cgThen (cgGatedGeluBackward cgFfnGradientHalf cgGateGradientHalf cgGatedGradient (cgGeluOf l) (cgUpOf l) (cgGateOf l) (naturalMultiply cgSeq cgFfn)) (cgThen (cgProductWeight cgFfn cgWidth cgSeq (cgLayerGradient l cgItemUp) cgFfnGradientHalf (cgNorm2HalfOf l)) (cgThen (cgProductData cgSeq cgWidth cgFfn cgNorm2UpGradient cgFfnGradientHalf (cgLayerHalf l cgMatrixUp)) (cgThen (cgProductWeight cgFfn cgWidth cgSeq (cgLayerGradient l cgItemGate) cgGateGradientHalf (cgNorm2HalfOf l)) (cgThen (cgProductDataAdding cgSeq cgWidth cgFfn cgNorm2Gradient cgGateGradientHalf (cgLayerHalf l cgMatrixGate) cgNorm2UpGradient) (cgThen (cgLayerNormBackward
2367 (cgLayerParameter l cgItemNorm2Gain)
2368 (cgLayerGradient l cgItemNorm2Gain)
2369 (cgLayerGradient l cgItemNorm2Shift)
2370 (cgProjectionOf l)
2371 cgNorm2Gradient
2372 (cgNorm2StatisticsOf l)
2373 cgNorm2InputGradient) (cgBinary
2374 cgImageAdd
2375 other
2376 current
2377 cgNorm2InputGradient
2378 (naturalMultiply cgSeq cgWidth)))))))))
2379 (cgThen
2380 (cgPhase
2381 cgPhaseAttentionOutput
2382 (cgThen (cgCastDown cgOutputDyHalf other (naturalMultiply cgSeq cgWidth)) (cgThen (cgProductWeight cgWidth cgWidth cgSeq (cgLayerGradient l cgItemOutput) cgOutputDyHalf (cgMergedHalfOf l)) (cgProductData cgSeq cgWidth cgWidth cgMergedGradient cgOutputDyHalf (cgLayerHalf l cgMatrixOutput)))))
2383 (cgThen
2384 (cgPhase
2385 cgPhaseHeadLayout
2386 (cgThen
2387 (cgLaunch
2388 cgImageTokenToHead
2389 cgSeq
2390 1
2391 1
2392 (cgPointer
2393 (cgArgument 0)
2394 cgContextGradientHeads
2395 (cgPointer (cgArgument 2) cgMergedGradient cgNoSlots)))
2396 (cgThen
2397 (cgHeadTranspose cgHeadWidth cgSeq cgValuesHeads (cgValuesTOf l))
2398 (cgThen
2399 (cgHeadTranspose cgSeq cgHeadWidth cgQueriesT (cgQueriesOf l))
2400 (cgThen
2401 (cgHeadTranspose cgSeq cgHeadWidth cgKeysT (cgKeysOf l))
2402 (cgHeadTranspose
2403 cgSeq
2404 cgHeadWidth
2405 cgContextGradientT
2406 cgContextGradientHeads))))))
2407 (cgThen
2408 (cgPhase cgPhaseAttentionHead (cgStreamingAttentionBackward l))
2409 (cgPhase
2410 cgPhaseQkv
2411 (cgThen (cgLaunch
2412 cgImageInverseRope
2413 cgSeq
2414 1
2415 1
2416 (cgPointer
2417 (cgArgument 0)
2418 cgQkvGradientHalf
2419 (cgPointer
2420 (cgArgument 1)
2421 cgQueriesGradient
2422 (cgPointer
2423 (cgArgument 2)
2424 cgKeysGradient
2425 (cgPointer
2426 (cgArgument 3)
2427 cgCosine
2428 (cgPointer
2429 (cgArgument 4)
2430 cgSine
2431 (cgPointer (cgArgument 5) cgValuesGradient cgNoSlots))))))) (cgThen (cgProductWeight (naturalMultiply cgQkvParts cgWidth) cgWidth cgSeq (cgLayerGradient l cgItemQkv) cgQkvGradientHalf (cgNorm1HalfOf l)) (cgThen (cgProductData cgSeq cgWidth (naturalMultiply cgQkvParts cgWidth) cgNorm1Gradient cgQkvGradientHalf (cgLayerHalf l cgMatrixQkv)) (cgThen (cgLayerNormBackward
2432 (cgLayerParameter l cgItemNorm1Gain)
2433 (cgLayerGradient l cgItemNorm1Gain)
2434 (cgLayerGradient l cgItemNorm1Shift)
2435 x
2436 cgNorm1Gradient
2437 (cgNorm1StatisticsOf l)
2438 cgNorm1InputGradient) (cgBinary
2439 cgImageAdd
2440 current
2441 other
2442 cgNorm1InputGradient
2443 (naturalMultiply cgSeq cgWidth))))))))))))))))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.