Provedsecond-order-bias-lawstochastic-attentiondepthlaw

Second-order bias: mean-zero noise and a Taylor remainder bound give |E f(x+ξ) − f(x)| ≤ (M/2) E[ξ²]

For all $\htmlData{mx=1}{m \in \htmlData{mx=0}{\mathbb{N}}}$ , $\htmlData{mx=5}{w, \xi : \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{m}} \mathrel{\htmlData{mx=4}{\to}} \htmlData{mx=3}{\mathbb{R}}}$ , $\htmlData{mx=3}{f : \htmlData{mx=0}{\mathbb{R}} \mathrel{\htmlData{mx=2}{\to}} \htmlData{mx=1}{\mathbb{R}}}$ and $\htmlData{mx=1}{x, f_{1}, M \in \htmlData{mx=0}{\mathbb{R}}}$ , if $\htmlData{mx=4}{\forall i,\ 0 \mathrel{\htmlData{mx=3}{\leq}} \htmlData{mx=2}{\htmlData{mx=0}{w}(\htmlData{mx=1}{i})}}$ , $\htmlData{mx=3}{\sum_{i} \htmlData{mx=2}{\htmlData{mx=0}{w}(\htmlData{mx=1}{i})}} \mathrel{\htmlData{mx=4}{=}} 1$ , $\htmlData{mx=7}{\sum_{i} \htmlData{mx=6}{\htmlData{mx=2}{\htmlData{mx=0}{w}(\htmlData{mx=1}{i})} \cdot \htmlData{mx=5}{\htmlData{mx=3}{\xi}(\htmlData{mx=4}{i})}}} \mathrel{\htmlData{mx=8}{=}} 0$ and $\htmlData{mx=20}{\forall t,\ \htmlData{mx=13}{\left\lvert \htmlData{mx=12}{\htmlData{mx=8}{\htmlData{mx=4}{\htmlData{mx=0}{f}(\htmlData{mx=3}{\htmlData{mx=1}{x} + \htmlData{mx=2}{t}})} - \htmlData{mx=7}{\htmlData{mx=5}{f}(\htmlData{mx=6}{x})}} - \htmlData{mx=11}{\htmlData{mx=9}{f_{1}} \cdot \htmlData{mx=10}{t}}} \right\rvert} \mathrel{\htmlData{mx=19}{\leq}} \htmlData{mx=18}{\htmlData{mx=15}{\frac{\htmlData{mx=14}{M}}{2}} \cdot \htmlData{mx=17}{\htmlData{mx=16}{t}^{2}}}}$ , then

\htmlData{mx=16}{\left\lvert \htmlData{mx=15}{\htmlData{mx=11}{\sum_{i} \htmlData{mx=10}{\htmlData{mx=2}{\htmlData{mx=0}{w}(\htmlData{mx=1}{i})} \cdot \htmlData{mx=9}{\htmlData{mx=3}{f}(\htmlData{mx=8}{\htmlData{mx=4}{x} + \htmlData{mx=7}{\htmlData{mx=5}{\xi}(\htmlData{mx=6}{i})}})}}} - \htmlData{mx=14}{\htmlData{mx=12}{f}(\htmlData{mx=13}{x})}} \right\rvert} \mathrel{\htmlData{mx=29}{\leq}} \htmlData{mx=28}{\htmlData{mx=18}{\frac{\htmlData{mx=17}{M}}{2}} \cdot \htmlData{mx=27}{\sum_{i} \htmlData{mx=26}{\htmlData{mx=21}{\htmlData{mx=19}{w}(\htmlData{mx=20}{i})} \cdot \htmlData{mx=25}{\htmlData{mx=24}{\htmlData{mx=22}{\xi}(\htmlData{mx=23}{i})}^{2}}}}}

What the notation means · or point at any symbol

NNatural numbers: The natural numbers 0, 1, 2, …. In Lean they start at 0. (ℕ)
Fin: Fin: a definition from the library, not special notation. Its name is its meaning; open it in Lean to see how it is defined. (Fin)
f(x)Function application: Fin applied to m. Lean writes f x; papers write f(x). (Fin m)
⟹Implies: If Fin(m), then ℝ. (Between types, A → B is the type of functions from A to B.) (Fin m → ℝ)
x : TVariables: w, ξ range over Fin m → ℝ: the statement must hold for every choice. ((w ξ : Fin m → ℝ))
x ∈ TVariables: x, f1, M range over real numbers: the statement must hold for every choice. ((x f1 M : ℝ))
∀For all: The statement after the comma holds for every i. (∀ i, 0 ≤ w i)
∑Summation: Add up w(i) for every i. (∑ i, w i)
=Equality: ∑ i, w(i) and 1 are the same. (∑ i, w i = 1)
≤At most: 0 is less than or equal to w(i). (0 ≤ w i)
·Multiplication: w(i) times ξ(i). (w i * ξ i)
|x|Absolute value: The size of f(x + t) − f(x) − f1 · t without its sign. (|f (x + t) - f x - f1 * t|)
-Subtraction: f(x + t) minus f(x). On ℕ, Lean's subtraction stops at 0: 2 − 5 = 0. (f (x + t) - f x)
RReal numbers: The real numbers: every point on the number line. (ℝ)
+Addition: x plus t. (x + t)
a/bDivision: M divided by 2. In Lean, dividing by 0 gives 0. (M / 2)
aᵇPower: t raised to the power 2. (t ^ 2)

theorem sob-jensen-gap-bound {m : ℕ} (w ξ : Fin m → ℝ) (f : ℝ → ℝ) (x f1 M : ℝ) (hw : ∀ i, 0 ≤ w i) (hsum : ∑ i, w i = 1) (hmean : ∑ i, w i * ξ i = 0) (hR : ∀ t, |f (x + t) - f x - f1 * t| ≤ M / 2 * t ^ 2) :
    |∑ i, w i * f (x + ξ i) - f x| ≤ M / 2 * ∑ i, w i * ξ i ^ 2

Per layer, an unbiased stochastic attention estimate with variance V induces bias at most (M/2)V downstream of a smooth map (delta method). Part of 'Second-Order Bias Law: unbiased stochastic attention stays (almost) unbiased through smooth depth'.

How it was proved

Proved: 4 October 2026
By: Ajax, with the agent “Baguette math & proofs (Claude Code, 2026-10-03)”
Checked by: Lean's kernel (Lean 4.33.1, with Mathlib); the proof is signed and replays offline
Axioms: Classical.choice, Quot.sound, propext
Proof: 21 lines of Lean

Second-order bias: mean-zero noise and a Taylor remainder bound give |E f(x+ξ) − f(x)| ≤ (M/2) E[ξ²]

How it was proved

More proved statements