the residual gradient: `current` (dy0) carries it down the layers and
`other` (dy1) holds the attention branch's share; each layer reads and
rewrites both, in place
2449def cgBackward : (family NvidiaLaunchSchedule) =
2450 (cgThen
2451 cgLossForward
2452 (cgThen
2453 (cgPhase
2454 cgPhaseOutput
2455 (cgLaunch
2456 cgImageCrossEntropyBackward
2457 cgSeq
2458 (naturalDivideUnchecked cgVocab cgThreads)
2459 1
2460 (cgPointer
2461 (cgArgument 0)
2462 cgLogitsGradient
2463 (cgPointer
2464 (cgArgument 1)
2465 cgLogits
2466 (cgPointer
2467 (cgArgument 2)
2468 cgLoss
2469 (cgPointer
2470 (cgArgument 3)
2471 cgCorrect
2472 (cgPointer
2473 (cgArgument 4)
2474 cgTargets
2475 (cgSlot (cgScalar 0) float32Log2E (cgSlot (cgScalar 2) cgLossGradientScale cgNoSlots)))))))))
2476 (cgThen
2477 (cgPhase
2478 cgPhaseOutputGradient
2479 (cgThen (cgCastDown cgLogitsGradientHalf cgLogitsGradient (naturalMultiply cgSeq cgVocab)) (cgThen (cgProductWeight cgVocab cgWidth cgSeq (cgParameter cgBankG cgEmbeddingEntry) cgLogitsGradientHalf cgFinalHalf) (cgThen (cgProductData cgSeq cgWidth cgVocab cgFinalGradient cgLogitsGradientHalf (cgHalf cgEmbeddingMatrix)) (cgLayerNormBackward
2480 (cgParameter cgBankP cgFinalGamma)
2481 (cgParameter cgBankG cgFinalGamma)
2482 (cgParameter cgBankG cgFinalBeta)
2483 (cgBoundary cgLayers)
2484 cgFinalGradient
2485 cgFinalStatistics
2486 (cgArena cgDy0))))))
2487 (cgThen
2488 (nvidiaAffineLaunchRepeat
2489 cgLayers
2490 (lambda unrestricted fromTop : Nat .
2491 (cgLayerBackward
2492 (naturalSaturatingSubtract (naturalSaturatingSubtract cgLayers 1) fromTop)
2493 (cgArena cgDy0)
2494 (cgArena cgDy1))))
2495 (cgPhase
2496 cgPhaseScatter
2497 (cgLaunch
2498 cgImageScatter
2499 cgSeq
2500 1
2501 1
2502 (cgPointer
2503 (cgArgument 0)
2504 (cgParameter cgBankG cgEmbeddingEntry)
2505 (cgPointer
2506 (cgArgument 1)
2507 (cgArena cgDy0)
2508 (cgPointer (cgArgument 2) cgIds cgNoSlots)))))))))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.