the residual gradient: `current` (dy0) carries it down the layers and
`other` (dy1) holds the attention branch's share; each layer reads and
rewrites both, in place
2448def cgBackward : (family NvidiaLaunchSchedule) =
2449 (cgThen
2450 cgLossForward
2451 (cgThen
2452 (cgPhase
2453 cgPhaseOutput
2454 (cgLaunch
2455 cgImageCrossEntropyBackward
2456 cgSeq
2457 (naturalDivideUnchecked cgVocab cgThreads)
2458 1
2459 (cgPointer
2460 (cgArgument 0)
2461 cgLogitsGradient
2462 (cgPointer
2463 (cgArgument 1)
2464 cgLogits
2465 (cgPointer
2466 (cgArgument 2)
2467 cgLoss
2468 (cgPointer
2469 (cgArgument 3)
2470 cgCorrect
2471 (cgPointer
2472 (cgArgument 4)
2473 cgTargets
2474 (cgSlot (cgScalar 0) float32Log2E (cgSlot (cgScalar 2) cgLossGradientScale cgNoSlots)))))))))
2475 (cgThen
2476 (cgPhase
2477 cgPhaseOutputGradient
2478 (cgThen (cgCastDown cgLogitsGradientHalf cgLogitsGradient (naturalMultiply cgSeq cgVocab)) (cgThen (cgProductWeight cgVocab cgWidth cgSeq (cgParameter cgBankG cgEmbeddingEntry) cgLogitsGradientHalf cgFinalHalf) (cgThen (cgProductData cgSeq cgWidth cgVocab cgFinalGradient cgLogitsGradientHalf (cgHalf cgEmbeddingMatrix)) (cgLayerNormBackward
2479 (cgParameter cgBankP cgFinalGamma)
2480 (cgParameter cgBankG cgFinalGamma)
2481 (cgParameter cgBankG cgFinalBeta)
2482 (cgBoundary cgLayers)
2483 cgFinalGradient
2484 cgFinalStatistics
2485 (cgArena cgDy0))))))
2486 (cgThen
2487 (nvidiaAffineLaunchRepeat
2488 cgLayers
2489 (lambda unrestricted fromTop : Nat .
2490 (cgLayerBackward
2491 (naturalSaturatingSubtract (naturalSaturatingSubtract cgLayers 1) fromTop)
2492 (cgArena cgDy0)
2493 (cgArena cgDy1))))
2494 (cgPhase
2495 cgPhaseScatter
2496 (cgLaunch
2497 cgImageScatter
2498 cgSeq
2499 1
2500 1
2501 (cgPointer
2502 (cgArgument 0)
2503 (cgParameter cgBankG cgEmbeddingEntry)
2504 (cgPointer
2505 (cgArgument 1)
2506 (cgArena cgDy0)
2507 (cgPointer (cgArgument 2) cgIds cgNoSlots)))))))))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.