A layer's forward with `attention` the heads' attention into the layer's
merged half (cgStreamingAttention); what the backward reads it writes into
the layer's saved planes (cgSaved)
2091def cgLayerForwardWith =
2092 (lambda unrestricted attention : (pi unrestricted layer : Nat . (family NvidiaLaunchSchedule)) .
2093 (lambda unrestricted l : Nat .
2094 (lambda unrestricted x : Nat .
2095 (lambda unrestricted out : Nat .
2096 (cgPhase
2097 cgPhaseLayer
2098 (cgThen
2099 (cgLayerNorm
2100 cgNorm1
2101 x
2102 (cgLayerParameter l cgItemNorm1Gain)
2103 (cgLayerParameter l cgItemNorm1Shift)
2104 (cgNorm1StatisticsOf l))
2105 (cgThen
2106 (cgCastDown (cgNorm1HalfOf l) cgNorm1 (naturalMultiply cgSeq cgWidth))
2107 (cgThen
2108 (cgProductForward
2109 cgSeq
2110 (naturalMultiply cgQkvParts cgWidth)
2111 cgWidth
2112 cgQkv
2113 (cgNorm1HalfOf l)
2114 (cgLayerHalf l cgMatrixQkv))
2115 (cgThen
2116 (cgLaunch
2117 cgImageQKRope
2118 cgSeq
2119 1
2120 1
2121 (cgPointer
2122 (cgArgument 0)
2123 (cgQueriesOf l)
2124 (cgPointer
2125 (cgArgument 1)
2126 (cgKeysOf l)
2127 (cgPointer
2128 (cgArgument 2)
2129 cgQkv
2130 (cgPointer
2131 (cgArgument 3)
2132 cgCosine
2133 (cgPointer (cgArgument 4) cgSine cgNoSlots))))))
2134 (cgThen
2135 (cgLaunch
2136 cgImageValueTranspose
2137 (naturalDivideUnchecked cgSeq 2)
2138 1
2139 1
2140 (cgPointer
2141 (cgArgument 0)
2142 (cgValuesTOf l)
2143 (cgPointer (cgArgument 2) cgQkv cgNoSlots)))
2144 (cgThen
2145 (attention l)
2146 (cgThen (cgProductForwardAdding cgSeq cgWidth cgWidth (cgProjectionOf l) (cgMergedHalfOf l) (cgLayerHalf l cgMatrixOutput) x) (cgThen (cgLayerNorm
2147 cgNorm2
2148 (cgProjectionOf l)
2149 (cgLayerParameter l cgItemNorm2Gain)
2150 (cgLayerParameter l cgItemNorm2Shift)
2151 (cgNorm2StatisticsOf l)) (cgThen (cgCastDown (cgNorm2HalfOf l) cgNorm2 (naturalMultiply cgSeq cgWidth)) (cgThen (cgProductForward
2152 cgSeq
2153 cgFfn
2154 cgWidth
2155 (cgUpOf l)
2156 (cgNorm2HalfOf l)
2157 (cgLayerHalf l cgMatrixUp)) (cgThen (cgProductForward
2158 cgSeq
2159 cgFfn
2160 cgWidth
2161 (cgGateOf l)
2162 (cgNorm2HalfOf l)
2163 (cgLayerHalf l cgMatrixGate)) (cgThen (cgGatedGeluForward (cgGeluOf l) (cgGatedHalfOf l) (cgUpOf l) (cgGateOf l) (naturalMultiply cgSeq cgFfn)) (cgProductForwardAdding cgSeq cgWidth cgFfn out (cgGatedHalfOf l) (cgLayerHalf l cgMatrixDown) (cgProjectionOf l)))))))))))))))))))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.