proofs
Provedalphabaguettermsnormdeferred-normalization

Deferred RMSNorm: dW = g ⊙ dW_g and the gain gradient dg_j = Σᵢ W_ij·(dW_g)_ij, with no token-axis reduction

For all T,m,n∈N\htmlData{mx=1}{T, m, n \in \htmlData{mx=0}{\mathbb{N}}}, W:Fin⁡ m→Fin⁡ n→R\htmlData{mx=9}{W : \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{m}} \mathrel{\htmlData{mx=8}{\to}} \htmlData{mx=5}{\htmlData{mx=3}{\operatorname{Fin}}\,\htmlData{mx=4}{n}} \mathrel{\htmlData{mx=7}{\to}} \htmlData{mx=6}{\mathbb{R}}}, g:Fin⁡ n→R\htmlData{mx=5}{g : \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{n}} \mathrel{\htmlData{mx=4}{\to}} \htmlData{mx=3}{\mathbb{R}}}, x:Fin⁡ T→Fin⁡ n→R\htmlData{mx=9}{x : \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{T}} \mathrel{\htmlData{mx=8}{\to}} \htmlData{mx=5}{\htmlData{mx=3}{\operatorname{Fin}}\,\htmlData{mx=4}{n}} \mathrel{\htmlData{mx=7}{\to}} \htmlData{mx=6}{\mathbb{R}}}, dy:Fin⁡ T→Fin⁡ m→R\htmlData{mx=9}{\mathit{dy} : \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{T}} \mathrel{\htmlData{mx=8}{\to}} \htmlData{mx=5}{\htmlData{mx=3}{\operatorname{Fin}}\,\htmlData{mx=4}{m}} \mathrel{\htmlData{mx=7}{\to}} \htmlData{mx=6}{\mathbb{R}}}, r:Fin⁡ T→R\htmlData{mx=5}{r : \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{T}} \mathrel{\htmlData{mx=4}{\to}} \htmlData{mx=3}{\mathbb{R}}}, i∈Fin⁡ m\htmlData{mx=3}{i \in \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{m}}} and j∈Fin⁡ n\htmlData{mx=3}{j \in \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{n}}}:

(∑tdy(t,i)⋅(g(j)⋅(r(t)⋅x(t,j)))=g(j)⋅∑tr(t)⋅dy(t,i)⋅x(t,j))∧(∑t(∑i′W(i′,j)⋅dy(t,i′))⋅(r(t)⋅x(t,j))=∑i′W(i′,j)⋅∑tr(t)⋅dy(t,i′)⋅x(t,j))\begin{aligned} & \left(\right. \htmlData{mx=19}{\sum_{t} \htmlData{mx=18}{\htmlData{mx=5}{\htmlData{mx=2}{\mathit{dy}}(\htmlData{mx=3}{t}, \htmlData{mx=4}{i})} \cdot (\htmlData{mx=17}{\htmlData{mx=8}{\htmlData{mx=6}{g}(\htmlData{mx=7}{j})} \cdot (\htmlData{mx=16}{\htmlData{mx=11}{\htmlData{mx=9}{r}(\htmlData{mx=10}{t})} \cdot \htmlData{mx=15}{\htmlData{mx=12}{x}(\htmlData{mx=13}{t}, \htmlData{mx=14}{j})}})})}} \\ & \quad \htmlData{mx=1}{=} \htmlData{mx=37}{\htmlData{mx=22}{\htmlData{mx=20}{g}(\htmlData{mx=21}{j})} \cdot \htmlData{mx=36}{\sum_{t} \htmlData{mx=35}{\htmlData{mx=30}{\htmlData{mx=25}{\htmlData{mx=23}{r}(\htmlData{mx=24}{t})} \cdot \htmlData{mx=29}{\htmlData{mx=26}{\mathit{dy}}(\htmlData{mx=27}{t}, \htmlData{mx=28}{i})}} \cdot \htmlData{mx=34}{\htmlData{mx=31}{x}(\htmlData{mx=32}{t}, \htmlData{mx=33}{j})}}}} \left.\right) \\ & \htmlData{mx=0}{\land} \left(\right. \htmlData{mx=58}{\sum_{t} \htmlData{mx=57}{\left(\htmlData{mx=48}{\sum_{i'} \htmlData{mx=47}{\htmlData{mx=42}{\htmlData{mx=39}{W}(\htmlData{mx=40}{i'}, \htmlData{mx=41}{j})} \cdot \htmlData{mx=46}{\htmlData{mx=43}{\mathit{dy}}(\htmlData{mx=44}{t}, \htmlData{mx=45}{i'})}}}\right) \cdot (\htmlData{mx=56}{\htmlData{mx=51}{\htmlData{mx=49}{r}(\htmlData{mx=50}{t})} \cdot \htmlData{mx=55}{\htmlData{mx=52}{x}(\htmlData{mx=53}{t}, \htmlData{mx=54}{j})}})}} \\ & \quad \htmlData{mx=38}{=} \htmlData{mx=78}{\sum_{i'} \htmlData{mx=77}{\htmlData{mx=62}{\htmlData{mx=59}{W}(\htmlData{mx=60}{i'}, \htmlData{mx=61}{j})} \cdot \htmlData{mx=76}{\sum_{t} \htmlData{mx=75}{\htmlData{mx=70}{\htmlData{mx=65}{\htmlData{mx=63}{r}(\htmlData{mx=64}{t})} \cdot \htmlData{mx=69}{\htmlData{mx=66}{\mathit{dy}}(\htmlData{mx=67}{t}, \htmlData{mx=68}{i'})}} \cdot \htmlData{mx=74}{\htmlData{mx=71}{x}(\htmlData{mx=72}{t}, \htmlData{mx=73}{j})}}}}} \left.\right) \end{aligned}
What the notation means · or point at any symbol
NNatural numbers
The natural numbers 0, 1, 2, …. In Lean they start at 0. (ℕ)
x ∈ TVariables
T, m, n range over natural numbers: the statement must hold for every choice. ((T m n : ℕ))
Fin
Fin: a definition from the library, not special notation. Its name is its meaning; open it in Lean to see how it is defined. (Fin)
f(x)Function application
Fin applied to m. Lean writes f x; papers write f(x). (Fin m)
∧And
Both hold: (∑ t, dy(t, i) · (g(j) · (r(t) · x(t, j)))) = g(j) · (∑ t,…, and (∑ t, (∑ i', W(i', j) · dy(t, i')) · (r(t) · x(t, j))) = (…. ((∑ t, dy t i * (g j * (r t * x t j)) = …)
∑Summation
Add up dy(t, i) · (g(j) · (r(t) · x(t, j))) for every t. (∑ t, dy t i * (g j * (r t * x t j)))
=Equality
∑ t, dy(t, i) · (g(j) · (r(t) · x(t, j))) and g(j) · (∑ t, r(t) · dy(t, i) · x(t, j)) are the same. (∑ t, dy t i * (g j * (r t * x t j)) = g…)
⟹Implies
If Fin(n), then ℝ. (Between types, A → B is the type of functions from A to B.) (Fin n → ℝ)
RReal numbers
The real numbers: every point on the number line. (ℝ)
·Multiplication
r(t) times x(t, j). (r t * x t j)

Let dW_g = Σ_t (r_t·dy_t)·x_tᵀ, one weight GEMM over row-scaled cotangents and raw inputs. First conjunct: the projection-weight gradient Σ_t dy_ti·(g_j·r_t x_tj) is g_j·(dW_g)_ij. Second conjunct: the gain gradient Σ_t (Wᵀdy_t)_j·(r_t x_tj), which Alpha's sumGainCotangentsWith currently reduces over all T rows of an activation-sized tensor, equals Σᵢ W_ij·(dW_g)_ij. That is a weight-shaped contraction the AdamW kernel can compute while it reads W and its gradient. Part of the idea 'Deferred RMS normalization for Baguette'.

How it was proved

Proved
3 October 2026
By
Ajax, with the agent “Baguette math & proofs (Claude Code, 2026-10-03)”
Checked by
Lean's kernel (Lean 4.33.1, with Mathlib); the proof is signed and replays offline
Axioms
Classical.choice, Quot.sound, propext
Proof
16 lines of Lean
Sign in to see the proofThe proof, its signed report and its history.

More proved statements