Provedllmlarge-language-modelsformal-prooflog-sum-exp

Log-sum-exp is at most the max logit plus log of the support size

For all $\htmlData{mx=1}{n \in \htmlData{mx=0}{\mathbb{N}}}$ and $\htmlData{mx=6}{w : \htmlData{mx=3}{\htmlData{mx=0}{\operatorname{Fin}}\,\htmlData{mx=2}{\htmlData{mx=1}{n} + 1}} \mathrel{\htmlData{mx=5}{\to}} \htmlData{mx=4}{\mathbb{R}}}$ :

\htmlData{mx=2}{\htmlData{mx=0}{\operatorname{lse}}(\htmlData{mx=1}{w})} \mathrel{\htmlData{mx=13}{\leq}} \htmlData{mx=12}{\htmlData{mx=7}{\htmlData{mx=3}{w}(\htmlData{mx=6}{\htmlData{mx=4}{\operatorname{argmax}}(\htmlData{mx=5}{w})})} + \htmlData{mx=11}{\log (\htmlData{mx=10}{\htmlData{mx=9}{\htmlData{mx=8}{n}} + 1})}}

What the notation means · or point at any symbol

NNatural numbers: The natural numbers 0, 1, 2, …. In Lean they start at 0. (ℕ)
Fin: Fin: a definition from the library, not special notation. Its name is its meaning; open it in Lean to see how it is defined. (Fin)
f(x)Function application: Fin applied to n + 1. Lean writes f x; papers write f(x). (Fin (n + 1))
⟹Implies: If Fin(n + 1), then ℝ. (Between types, A → B is the type of functions from A to B.) (Fin (n + 1) → ℝ)
lse: LLMSoftmax.lse: a definition from the library, not special notation. Its name is its meaning; open it in Lean to see how it is defined. (LLMSoftmax.lse)
≤At most: lse(w) is less than or equal to w(argmax(w)) + log(n + 1). (LLMSoftmax.lse w ≤ w (LLMDecode.argmax …)
+Addition: n plus 1. (n + 1)
RReal numbers: The real numbers: every point on the number line. (ℝ)
argmax: LLMDecode.argmax: a definition from the library, not special notation. Its name is its meaning; open it in Lean to see how it is defined. (LLMDecode.argmax)
logNatural logarithm: The logarithm of n + 1 to base e: the power e must be raised to to give n + 1. (Real.log ((n : ℝ) + 1))
(x : T)As real numbers: n read as an element of real numbers. Mathematicians leave this implicit; Lean needs it to know which kind of number is meant. ((n : ℝ))

theorem lse-le-max-plus-log-card (n : ℕ) (w : Fin (n + 1) → ℝ) :
    LLMSoftmax.lse w ≤ w (LLMDecode.argmax w) + Real.log ((n : ℝ) + 1)

lse w ≤ w (argmax w) + log(n+1): the classical LSE upper bound, finite form.

How it was proved

Proved: 28 September 2026
By: Ajax, with the agent “ompmuse”
Checked by: Lean's kernel (Lean 4.33.1, with Mathlib); the proof is signed and replays offline
Axioms: Classical.choice, Quot.sound, propext
Proof: 27 lines of Lean

Log-sum-exp is at most the max logit plus log of the support size

How it was proved

More proved statements