Provedalphabaguettermsnorm

RMSNorm of c·x with ε equals RMSNorm of x with ε/c²: epsilon is the only scale-dependence

For all $\htmlData{mx=1}{n \in \htmlData{mx=0}{\mathbb{N}}}$ , $\htmlData{mx=1}{\varepsilon, c \in \htmlData{mx=0}{\mathbb{R}}}$ , $\htmlData{mx=5}{x : \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{n}} \mathrel{\htmlData{mx=4}{\to}} \htmlData{mx=3}{\mathbb{R}}}$ and $\htmlData{mx=3}{i \in \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{n}}}$ , if $0 \mathrel{\htmlData{mx=1}{<}} \htmlData{mx=0}{n}$ and $0 \mathrel{\htmlData{mx=1}{<}} \htmlData{mx=0}{c}$ , then

\begin{aligned} & \htmlData{mx=18}{\frac{\htmlData{mx=5}{\htmlData{mx=1}{c} \cdot \htmlData{mx=4}{\htmlData{mx=2}{x}(\htmlData{mx=3}{i})}}}{\htmlData{mx=17}{\sqrt{\htmlData{mx=16}{\htmlData{mx=14}{\frac{\htmlData{mx=12}{\sum_{j} \htmlData{mx=11}{(\htmlData{mx=10}{\htmlData{mx=6}{c} \cdot \htmlData{mx=9}{\htmlData{mx=7}{x}(\htmlData{mx=8}{j})}})^{2}}}}{\htmlData{mx=13}{n}}} + \htmlData{mx=15}{\varepsilon}}}}}} \\ & \quad \htmlData{mx=0}{=} \htmlData{mx=35}{\frac{\htmlData{mx=21}{\htmlData{mx=19}{x}(\htmlData{mx=20}{i})}}{\htmlData{mx=34}{\sqrt{\htmlData{mx=33}{\htmlData{mx=28}{\frac{\htmlData{mx=26}{\sum_{j} \htmlData{mx=25}{\htmlData{mx=24}{\htmlData{mx=22}{x}(\htmlData{mx=23}{j})}^{2}}}}{\htmlData{mx=27}{n}}} + \htmlData{mx=32}{\frac{\htmlData{mx=29}{\varepsilon}}{\htmlData{mx=31}{\htmlData{mx=30}{c}^{2}}}}}}}}} \end{aligned}

What the notation means · or point at any symbol

NNatural numbers: The natural numbers 0, 1, 2, …. In Lean they start at 0. (ℕ)
<Less than: 0 is strictly smaller than n. (0 < n)
RReal numbers: The real numbers: every point on the number line. (ℝ)
x ∈ TVariables: ε, c range over real numbers: the statement must hold for every choice. ((ε c : ℝ))
Fin: Fin: a definition from the library, not special notation. Its name is its meaning; open it in Lean to see how it is defined. (Fin)
f(x)Function application: Fin applied to n. Lean writes f x; papers write f(x). (Fin n)
⟹Implies: If Fin(n), then ℝ. (Between types, A → B is the type of functions from A to B.) (Fin n → ℝ)
=Equality: c · x(i)/√((∑ j, (c · x(j))^2)/n + ε) and x(i)/√((∑ j, x(j)^2)/n + ε/c^2) are the same. ((c * x i) / Real.sqrt ((∑ j, (c * x j) …)
·Multiplication: c times x(i). (c * x i)
√Square root: The non-negative number whose square is (∑ j, (c · x(j))^2)/n + ε. (Real.sqrt ((∑ j, (c * x j) ^ 2) / n + ε))
a/bDivision: ∑ j, (c · x(j))^2 divided by n. In Lean, dividing by 0 gives 0. ((∑ j, (c * x j) ^ 2) / n)
+Addition: (∑ j, (c · x(j))^2)/n plus ε. ((∑ j, (c * x j) ^ 2) / n + ε)
∑Summation: Add up (c · x(j))^2 for every j. (∑ j, (c * x j) ^ 2)
aᵇPower: c · x(j) raised to the power 2. ((c * x j) ^ 2)

theorem rmsnorm-scale-moves-epsilon (n : ℕ) (hn : 0 < n) (ε c : ℝ) (hc : 0 < c) (x : Fin n → ℝ) (i : Fin n) :
    (c * x i) / Real.sqrt ((∑ j, (c * x j) ^ 2) / n + ε) = x i / Real.sqrt ((∑ j, x j ^ 2) / n + ε / c ^ 2)

For c > 0, normalizing c·x with epsilon ε is exactly normalizing x with epsilon ε/c². RMSNorm is therefore scale-invariant except through ε. A residual stream that grows by c shrinks the effective epsilon by c², and the reverse. This bears on Baguette's recorded growth of the residual and moments (GOAL-PRD.md, updates 1100–1135), and on any rewrite that moves the normalization's scale (see the idea on deferred normalization).

How it was proved

Proved: 3 October 2026
By: Ajax, with the agent “Baguette math & proofs (Claude Code, 2026-10-03)”
Checked by: Lean's kernel (Lean 4.33.1, with Mathlib); the proof is signed and replays offline
Axioms: Classical.choice, Quot.sound, propext
Proof: 14 lines of Lean

RMSNorm of c·x with ε equals RMSNorm of x with ε/c²: epsilon is the only scale-dependence

How it was proved

More proved statements