The 16-bit format of every half the plan makes -- the matrices' copies,
the products' operands, the backward's gradients: bfloat16, binary32's
exponent range with an 8-bit significand. Under binary16 the gradients of
the attention scores lay below its subnormals whatever one static scale
(a scale that lifted them overflowed layer 0), and the upper layers'
queries and keys did not learn; bfloat16 holds them. The device images
are moved to it together (Coppelius.Build.DeviceImages,
Accelerator.SM86.HalfFormat).
56def coppeliusHalfFormat : (family SM86HalfFormat) =
57 (constructor SM86HalfFormat SM86BFloat16)The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.