Source/Packages

Realization.Nvidia.SM86.StreamingAttentionSM86

packages/realizations/cooperative/nvidia-sm86/src/Realization/Nvidia/SM86/StreamingAttentionSM86.alpha

3,230 lines493 declarations193.3 KiBSHA-256 cc0366b3d857

def · lines 1457–1482

sqrPairs

Full file
row half h, n-tiles j and j + 4: the inverse rotation, both pairs packed into their first accumulators and stored
1457def sqrPairs = (lambda unrestricted width : Nat . (lambda unrestricted h : Nat . (lambda unrestricted j : Nat .
1458  (lambda unrestricted tail : (family SM86Program) .
1459  (let unrestricted first = (saElem (saO j) h 0) in
1460  (let unrestricted second = (saElem (saO (naturalAdd j 4)) h 0) in
1461  (saFor 2 (lambda unrestricted e : Nat . (lambda unrestricted rest : (family SM86Program) .
1462    (let unrestricted x1 = (saElem (saO j) h e) in
1463    (let unrestricted x2 = (saElem (saO (naturalAdd j 4)) h e) in
1464    (let unrestricted cosine = (sqrWord h 0 j e) in
1465    (let unrestricted sine = (sqrWord h 1 j e) in
1466    (saFmul (sqrOut e) x2 sine saWaitNone
1467    (saFneg (sqrOut e) (sqrOut e)
1468    (saFfma (sqrOut e) x1 cosine (sqrOut e)
1469    (saFmul (sqrOut (naturalAdd 2 e)) x1 sine saWaitNone
1470    (saFfma (sqrOut (naturalAdd 2 e)) x2 cosine (sqrOut (naturalAdd 2 e))
1471      rest)))))))))))
1472  (saOp (constructor SM86InstructionBody SM86FloatPairToPackedBFloat16Pair
1473    (saR first) (saR (sqrOut 1)) (saR (sqrOut 0)) saPlain)
1474  (saOp (constructor SM86InstructionBody SM86FloatPairToPackedBFloat16Pair
1475    (saR second) (saR (sqrOut 3)) (saR (sqrOut 2)) saPlain)
1476  (saOp (saStore sqrPointer first
1477    (naturalAdd (naturalMultiply h (naturalMultiply 8 (naturalMultiply width saHalfBytes)))
1478      (naturalMultiply j 16)) saWaitNone)
1479  (saOp (saStore sqrPointer second
1480    (naturalAdd (naturalMultiply h (naturalMultiply 8 (naturalMultiply width saHalfBytes)))
1481      (naturalMultiply (naturalAdd j 4) 16)) saWaitNone)
1482    tail)))))))))))

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.