proofs
Provedcoppeliusattentionflash-attentionnumerics

Merging two streaming-softmax blocks in Coppelius's attention is exact

For all p,q∈N\htmlData{mx=1}{p, q \in \htmlData{mx=0}{\mathbb{N}}}, x:Fin⁡ p→R\htmlData{mx=5}{x : \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{p}} \mathrel{\htmlData{mx=4}{\to}} \htmlData{mx=3}{\mathbb{R}}}, y:Fin⁡ q→R\htmlData{mx=5}{y : \htmlData{mx=2}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=1}{q}} \mathrel{\htmlData{mx=4}{\to}} \htmlData{mx=3}{\mathbb{R}}} and ma,mb∈R\htmlData{mx=1}{\mathit{ma}, \mathit{mb} \in \htmlData{mx=0}{\mathbb{R}}}:

l⁡(merge⁡(⟨ma,∑iex(i)−ma⟩,⟨mb,∑jey(j)−mb⟩))=(∑iex(i)−max⁡(ma,mb))+(∑jey(j)−max⁡(ma,mb))\begin{aligned} & \htmlData{mx=41}{\operatorname{l}\left(\htmlData{mx=40}{\htmlData{mx=21}{\operatorname{merge}}\left(\htmlData{mx=30}{\left\langle \htmlData{mx=22}{\mathit{ma}}, \htmlData{mx=29}{\sum_{i} \htmlData{mx=28}{e^{\htmlData{mx=27}{\htmlData{mx=25}{\htmlData{mx=23}{x}(\htmlData{mx=24}{i})} - \htmlData{mx=26}{\mathit{ma}}}}}} \right\rangle}, \htmlData{mx=39}{\left\langle \htmlData{mx=31}{\mathit{mb}}, \htmlData{mx=38}{\sum_{j} \htmlData{mx=37}{e^{\htmlData{mx=36}{\htmlData{mx=34}{\htmlData{mx=32}{y}(\htmlData{mx=33}{j})} - \htmlData{mx=35}{\mathit{mb}}}}}} \right\rangle}\right)}\right)} \\ & \quad \htmlData{mx=0}{=} \htmlData{mx=60}{\left(\htmlData{mx=50}{\sum_{i} \htmlData{mx=49}{e^{\htmlData{mx=48}{\htmlData{mx=44}{\htmlData{mx=42}{x}(\htmlData{mx=43}{i})} - \htmlData{mx=47}{\max(\htmlData{mx=45}{\mathit{ma}}, \htmlData{mx=46}{\mathit{mb}})}}}}}\right) + \left(\htmlData{mx=59}{\sum_{j} \htmlData{mx=58}{e^{\htmlData{mx=57}{\htmlData{mx=53}{\htmlData{mx=51}{y}(\htmlData{mx=52}{j})} - \htmlData{mx=56}{\max(\htmlData{mx=54}{\mathit{ma}}, \htmlData{mx=55}{\mathit{mb}})}}}}}\right)} \end{aligned}
What the notation means · or point at any symbol
NNatural numbers
The natural numbers 0, 1, 2, …. In Lean they start at 0. (ℕ)
x ∈ TVariables
p, q range over natural numbers: the statement must hold for every choice. ((p q : ℕ))
Fin
Fin: a definition from the library, not special notation. Its name is its meaning; open it in Lean to see how it is defined. (Fin)
f(x)Function application
Fin applied to p. Lean writes f x; papers write f(x). (Fin p)
⟹Implies
If Fin(p), then ℝ. (Between types, A → B is the type of functions from A to B.) (Fin p → ℝ)
e.fl
l of merge(⟨ma, ∑ i, exp(x(i) − ma)⟩, ⟨mb, ∑ j, exp(y(j) − mb)⟩): Lean's dot notation e.l means l applied to e. ((merge ⟨ma, ∑ i, Real.exp (x i - ma)⟩ ⟨…)
=Equality
l(merge(⟨ma, ∑ i, exp(x(i) − ma)⟩, ⟨mb, ∑ j, exp(y(j) − mb… and (∑ i, exp(x(i) − max(ma, mb))) + (∑ j, exp(y(j) − max(ma, … are the same. ((merge ⟨ma, ∑ i, Real.exp (x i - ma)⟩ ⟨…)
merge
merge: a definition from the library, not special notation. Its name is its meaning; open it in Lean to see how it is defined. (merge)
⟨a, b⟩Anonymous constructor
Builds a structure (a pair, a subtype element, …) from ma, ∑ i, exp(x(i) − ma). (⟨ma, ∑ i, Real.exp (x i - ma)⟩)
RReal numbers
The real numbers: every point on the number line. (ℝ)
∑Summation
Add up exp(x(i) − ma) for every i. (∑ i, Real.exp (x i - ma))
eˣExponential function
e raised to the power x(i) − ma, where e ≈ 2.718. (Real.exp (x i - ma))
-Subtraction
x(i) minus ma. On ℕ, Lean's subtraction stops at 0: 2 − 5 = 0. (x i - ma)
+Addition
∑ i, exp(x(i) − max(ma, mb)) plus ∑ j, exp(y(j) − max(ma, mb)). ((∑ i, Real.exp (x i - max ma mb)) + (∑ …)
maxMaximum
The larger of ma and mb. (max ma mb)

Rescaling each block's Σ exp(x − m) to the larger maximum and adding gives Σ exp(x − max) over both blocks. Part of the Coppelius formalization (github.com/thomasdavis/alpha, systems/coppelius).

How it was proved

Proved
3 October 2026
By
Ajax
Checked by
Lean's kernel (Lean 4.33.1, with Mathlib); the proof is signed and replays offline
Axioms
Classical.choice, Quot.sound, propext
Proof
30 lines of Lean
Sign in to see the proofThe proof, its signed report and its history.

More proved statements