grouped attention key backward120.77 ms
160 static launches18.4% of recorded time in this group
grouped attention query backward87.89 ms
80 static launches13.4% of recorded time in this group
grouped attention forward65.32 ms
80 static launches10.0% of recorded time in this group
restore forward workspace31.99 ms
79 static launches4.9% of recorded time in this group
save forward workspace31.93 ms
79 static launches4.9% of recorded time in this group
feed down weight pullback12.26 ms
80 static launches1.9% of recorded time in this group
feed gate weight pullback11.91 ms
80 static launches1.8% of recorded time in this group
feed up weight pullback11.91 ms
80 static launches1.8% of recorded time in this group
parameter update11.31 ms
1 static launches1.7% of recorded time in this group
norm input10.87 ms
160 static launches1.7% of recorded time in this group
feed reduction residual10.80 ms
80 static launches1.6% of recorded time in this group
feed gate gate pullback10.54 ms
80 static launches1.6% of recorded time in this group
feed gate input pullback10.36 ms
80 static launches1.6% of recorded time in this group
feed up input pullback10.36 ms
80 static launches1.6% of recorded time in this group
input norm input pullback10.04 ms
80 static launches1.5% of recorded time in this group
feed norm input pullback9.56 ms
80 static launches1.5% of recorded time in this group
feed down input pullback9.54 ms
80 static launches1.5% of recorded time in this group
feed up forward9.43 ms
80 static launches1.4% of recorded time in this group
feed gate forward9.42 ms
80 static launches1.4% of recorded time in this group
tied vocabulary forward8.29 ms
1 static launches1.3% of recorded time in this group
projection weight pullback7.98 ms
80 static launches1.2% of recorded time in this group
feed gated product7.34 ms
80 static launches1.1% of recorded time in this group
attention output weight pullback7.34 ms
80 static launches1.1% of recorded time in this group
feed gate up pullback7.22 ms
80 static launches1.1% of recorded time in this group
projection input pullback6.79 ms
80 static launches1.0% of recorded time in this group
projection forward6.35 ms
80 static launches1.0% of recorded time in this group
tied vocabulary input pullback5.84 ms
1 static launches0.9% of recorded time in this group
norm feed5.63 ms
80 static launches0.9% of recorded time in this group
embedding gradient scatter5.13 ms
1 static launches0.8% of recorded time in this group
feed norm gain reduction4.90 ms
80 static launches0.7% of recorded time in this group
attention output residual4.82 ms
80 static launches0.7% of recorded time in this group
input norm gain reduction4.73 ms
80 static launches0.7% of recorded time in this group
attention output input pullback4.40 ms
80 static launches0.7% of recorded time in this group
tied vocabulary weight pullback4.34 ms
1 static launches0.7% of recorded time in this group
feed gate gradient to bfloat164.10 ms
80 static launches0.6% of recorded time in this group
feed gated to bfloat164.08 ms
80 static launches0.6% of recorded time in this group
feed up gradient to bfloat163.94 ms
80 static launches0.6% of recorded time in this group
query rows to columns3.73 ms
80 static launches0.6% of recorded time in this group
output gradient rows to columns3.73 ms
80 static launches0.6% of recorded time in this group
feed branch gradient sum3.08 ms
80 static launches0.5% of recorded time in this group
gradient apply scalar2.95 ms
1 static launches0.4% of recorded time in this group
block input gradient sum2.85 ms
80 static launches0.4% of recorded time in this group
attention residual gradient sum2.85 ms
80 static launches0.4% of recorded time in this group
exact vocabulary logit gradient2.84 ms
1 static launches0.4% of recorded time in this group
projection gradient to bfloat162.68 ms
80 static launches0.4% of recorded time in this group
query inverse rotate2.48 ms
80 static launches0.4% of recorded time in this group
exact vocabulary loss rows2.43 ms
1 static launches0.4% of recorded time in this group
scaled gradient square2.40 ms
1 static launches0.4% of recorded time in this group
attention gradient token to head2.36 ms
80 static launches0.4% of recorded time in this group
last input norm to bfloat162.05 ms
80 static launches0.3% of recorded time in this group
query rotate1.94 ms
80 static launches0.3% of recorded time in this group
residual gradient to bfloat161.90 ms
80 static launches0.3% of recorded time in this group
norm feed to bfloat161.89 ms
80 static launches0.3% of recorded time in this group
norm input to bfloat161.88 ms
80 static launches0.3% of recorded time in this group
logit gradient to bfloat161.85 ms
1 static launches0.3% of recorded time in this group
attention residual gradient to bfloat161.83 ms
80 static launches0.3% of recorded time in this group
value columns to rows1.78 ms
80 static launches0.3% of recorded time in this group
key rows to columns1.77 ms
80 static launches0.3% of recorded time in this group
output gradient to bfloat161.66 ms
80 static launches0.3% of recorded time in this group
value gradient group reduction1.63 ms
80 static launches0.2% of recorded time in this group
key gradient group reduction1.58 ms
80 static launches0.2% of recorded time in this group
value transpose1.45 ms
80 static launches0.2% of recorded time in this group
value inverse transpose1.33 ms
80 static launches0.2% of recorded time in this group
key inverse rotate1.14 ms
80 static launches0.2% of recorded time in this group
feed down weight to bfloat161.01 ms
80 static launches0.2% of recorded time in this group
feed gate weight to bfloat161.00 ms
80 static launches0.2% of recorded time in this group
feed up weight to bfloat160.99 ms
80 static launches0.2% of recorded time in this group
key rotate0.99 ms
80 static launches0.2% of recorded time in this group
projection weight to bfloat160.83 ms
80 static launches0.1% of recorded time in this group
attention output weight to bfloat160.71 ms
80 static launches0.1% of recorded time in this group
rotary table0.43 ms
32 static launches0.1% of recorded time in this group
tied embedding to bfloat160.27 ms
1 static launches0.0% of recorded time in this group
final norm input pullback0.12 ms
1 static launches0.0% of recorded time in this group
final norm0.07 ms
1 static launches0.0% of recorded time in this group
final norm gain reduction0.06 ms
1 static launches0.0% of recorded time in this group
final norm to bfloat160.02 ms
1 static launches0.0% of recorded time in this group
embed0.02 ms
1 static launches0.0% of recorded time in this group
gradient reduction zero0.02 ms
1 static launches0.0% of recorded time in this group
gradient partial level 10.02 ms
1 static launches0.0% of recorded time in this group
token window copy0.01 ms
1 static launches0.0% of recorded time in this group
loss mean readback0.01 ms
1 static launches0.0% of recorded time in this group
loss row partial sum0.01 ms
1 static launches0.0% of recorded time in this group
next token logit0.01 ms
1 static launches0.0% of recorded time in this group
gradient norm square0.01 ms
1 static launches0.0% of recorded time in this group
loss mean final sum0.01 ms
1 static launches0.0% of recorded time in this group
gradient partial level 20.01 ms
1 static launches0.0% of recorded time in this group
gradient bank padding0.01 ms
1 static launches0.0% of recorded time in this group
gradient clip scalar0.01 ms
1 static launches0.0% of recorded time in this group