A layer's forward with `attention` the heads' attention into the layer's
merged half (cgStreamingAttention); what the backward reads it writes into
the layer's saved planes (cgSaved)
2090def cgLayerForwardWith =
2091 (lambda unrestricted attention : (pi unrestricted layer : Nat . (family NvidiaLaunchSchedule)) .
2092 (lambda unrestricted l : Nat .
2093 (lambda unrestricted x : Nat .
2094 (lambda unrestricted out : Nat .
2095 (cgPhase
2096 cgPhaseLayer
2097 (cgThen
2098 (cgLayerNorm
2099 cgNorm1
2100 x
2101 (cgLayerParameter l cgItemNorm1Gain)
2102 (cgLayerParameter l cgItemNorm1Shift)
2103 (cgNorm1StatisticsOf l))
2104 (cgThen
2105 (cgCastDown (cgNorm1HalfOf l) cgNorm1 (naturalMultiply cgSeq cgWidth))
2106 (cgThen
2107 (cgProductForward
2108 cgSeq
2109 (naturalMultiply cgQkvParts cgWidth)
2110 cgWidth
2111 cgQkv
2112 (cgNorm1HalfOf l)
2113 (cgLayerHalf l cgMatrixQkv))
2114 (cgThen
2115 (cgLaunch
2116 cgImageQKRope
2117 cgSeq
2118 1
2119 1
2120 (cgPointer
2121 (cgArgument 0)
2122 (cgQueriesOf l)
2123 (cgPointer
2124 (cgArgument 1)
2125 (cgKeysOf l)
2126 (cgPointer
2127 (cgArgument 2)
2128 cgQkv
2129 (cgPointer
2130 (cgArgument 3)
2131 cgCosine
2132 (cgPointer (cgArgument 4) cgSine cgNoSlots))))))
2133 (cgThen
2134 (cgLaunch
2135 cgImageValueTranspose
2136 (naturalDivideUnchecked cgSeq 2)
2137 1
2138 1
2139 (cgPointer
2140 (cgArgument 0)
2141 (cgValuesTOf l)
2142 (cgPointer (cgArgument 2) cgQkv cgNoSlots)))
2143 (cgThen
2144 (attention l)
2145 (cgThen (cgProductForwardAdding cgSeq cgWidth cgWidth (cgProjectionOf l) (cgMergedHalfOf l) (cgLayerHalf l cgMatrixOutput) x) (cgThen (cgLayerNorm
2146 cgNorm2
2147 (cgProjectionOf l)
2148 (cgLayerParameter l cgItemNorm2Gain)
2149 (cgLayerParameter l cgItemNorm2Shift)
2150 (cgNorm2StatisticsOf l)) (cgThen (cgCastDown (cgNorm2HalfOf l) cgNorm2 (naturalMultiply cgSeq cgWidth)) (cgThen (cgProductForward
2151 cgSeq
2152 cgFfn
2153 cgWidth
2154 (cgUpOf l)
2155 (cgNorm2HalfOf l)
2156 (cgLayerHalf l cgMatrixUp)) (cgThen (cgProductForward
2157 cgSeq
2158 cgFfn
2159 cgWidth
2160 (cgGateOf l)
2161 (cgNorm2HalfOf l)
2162 (cgLayerHalf l cgMatrixGate)) (cgThen (cgGatedGeluForward (cgGeluOf l) (cgGatedHalfOf l) (cgUpOf l) (cgGateOf l) (naturalMultiply cgSeq cgFfn)) (cgProductForwardAdding cgSeq cgWidth cgFfn out (cgGatedHalfOf l) (cgLayerHalf l cgMatrixDown) (cgProjectionOf l)))))))))))))))))))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.