One parameter's update, given the bias powers already advanced to this step
(power1 = beta1^t, power2 = beta2^t for the t-th update, t >= 1).
93def momentNormalizedUpdate =
94 (lambda unrestricted hyper : (family MomentNormalizedHyper) .
95 (lambda unrestricted power1 : F32 .
96 (lambda unrestricted power2 : F32 .
97 (lambda unrestricted element : (family MomentNormalizedElement) .
98 (lambda unrestricted gradient : F32 .
99 (eliminate MomentNormalizedHyper
100 (lambda unrestricted current : (family MomentNormalizedHyper) . (family MomentNormalizedElement))
101 hyper
102 (branch MomentNormalizedHyperValue rate beta1 beta2 epsilon .
103 (eliminate MomentNormalizedElement
104 (lambda unrestricted current : (family MomentNormalizedElement) . (family MomentNormalizedElement))
105 element
106 (branch MomentNormalizedElementValue weight first second .
107 (let unrestricted first1 = (firstMomentValue beta1 first gradient) in
108 (let unrestricted second1 = (secondMomentValue beta2 second gradient) in
109 (constructor MomentNormalizedElement MomentNormalizedElementValue
110 (normalizedStepValue rate epsilon weight
111 (correctMomentBias power1 first1)
112 (correctMomentBias power2 second1))
113 first1
114 second1))))))))))))The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.