proofs
Provedbaguettealphagpu-memorycross-entropy

Baguette: the vocabulary objective in two row halves fits the logit plane and computes the same rows

For all L,h,V∈N\htmlData{mx=1}{L, h, V \in \htmlData{mx=0}{\mathbb{N}}}:

L+4h⋅V+2(2h)⋅V=L+4(2h)⋅V∧(∀f:N→N, i∈N, i<2h  ⟹  ({f(i)if i<hf(h+(i−h))otherwise)=f(i))∧4(V128)≤2V\begin{aligned} & \htmlData{mx=12}{\htmlData{mx=6}{\htmlData{mx=1}{L} + \htmlData{mx=5}{\htmlData{mx=3}{4\htmlData{mx=2}{h}} \cdot \htmlData{mx=4}{V}}} + \htmlData{mx=11}{\htmlData{mx=9}{2(\htmlData{mx=8}{2\htmlData{mx=7}{h}})} \cdot \htmlData{mx=10}{V}}} \mathrel{\htmlData{mx=20}{=}} \htmlData{mx=19}{\htmlData{mx=13}{L} + \htmlData{mx=18}{\htmlData{mx=16}{4(\htmlData{mx=15}{2\htmlData{mx=14}{h}})} \cdot \htmlData{mx=17}{V}}} \\ & \htmlData{mx=0}{\land} \left(\htmlData{mx=48}{\forall f : \htmlData{mx=44}{\mathbb{N}} \mathrel{\htmlData{mx=46}{\to}} \htmlData{mx=45}{\mathbb{N}},\ i \in \htmlData{mx=47}{\mathbb{N}},\ \htmlData{mx=21}{i} \mathrel{\htmlData{mx=24}{<}} \htmlData{mx=23}{2\htmlData{mx=22}{h}} \mathrel{\htmlData{mx=43}{\implies}} \left(\htmlData{mx=38}{\begin{cases} \htmlData{mx=27}{\htmlData{mx=25}{f}(\htmlData{mx=26}{i})} & \text{if } \htmlData{mx=28}{i} \mathrel{\htmlData{mx=30}{<}} \htmlData{mx=29}{h} \\ \htmlData{mx=37}{\htmlData{mx=31}{f}(\htmlData{mx=36}{\htmlData{mx=32}{h} + (\htmlData{mx=35}{\htmlData{mx=33}{i} - \htmlData{mx=34}{h}})})} & \text{otherwise} \end{cases}}\right) \mathrel{\htmlData{mx=42}{=}} \htmlData{mx=41}{\htmlData{mx=39}{f}(\htmlData{mx=40}{i})}}\right) \\ & \htmlData{mx=0}{\land} \htmlData{mx=51}{4\left(\htmlData{mx=50}{\frac{\htmlData{mx=49}{V}}{128}}\right)} \mathrel{\htmlData{mx=54}{\leq}} \htmlData{mx=53}{2\htmlData{mx=52}{V}} \end{aligned}
What the notation means · or point at any symbol
NNatural numbers
The natural numbers 0, 1, 2, …. In Lean they start at 0. (ℕ)
x ∈ TVariables
L, h, V range over natural numbers: the statement must hold for every choice. ((L h V : ℕ))
=Equality
L + 4 · h · V + 2 · (2 · h) · V and L + 4 · (2 · h) · V are the same. ((L + 4 * h * V) + 2 * (2 * h) * V = L +…)
+Addition
L plus 4 · h · V. (L + 4 * h * V)
·Multiplication
4 times h. (4 * h)
∧And
Both hold: ∀ f, i, i < 2 · h → f(i) if i < h, else f(h + (i − h)) = f…, and 4 · (V/128) ≤ 2 · V. ((∀ (f : ℕ → ℕ) (i : ℕ), i < 2 * h → (if…)
∀For all
The statement after the comma holds for every f, i. (∀ (f : ℕ → ℕ) (i : ℕ), i < 2 * h → (if …)
<Less than
i is strictly smaller than 2 · h. (i < 2 * h)
⟹Implies
If i < 2 · h, then f(i) if i < h, else f(h + (i − h)) = f(i). (Between types, A → B is the type of functions from A to B.) (i < 2 * h → (if i < h then f i else f (…)
{ a if c; b otherwiseCases (if … then … else)
f(i) when i < h, and f(h + (i − h)) otherwise. (if i < h then f i else f (h + (i - h)))
f(x)Function application
f applied to i. Lean writes f x; papers write f(x). (f i)
-Subtraction
i minus h. On ℕ, Lean's subtraction stops at 0: 2 − 5 = 0. (i - h)
≤At most
4 · (V/128) is less than or equal to 2 · V. (4 * (V / 128) ≤ 2 * V)
a/bDivision
V divided by 128. In Lean, dividing by 0 gives 0. (V / 128)

Baguette's training objective (tied vocabulary product, target gather, exact loss rows, exact logit gradient) is row-wise: each output row depends only on its own input row. On the RTX 3070 the recomputed realization runs it in two halves of h = 2048 rows each. Its logit plane holds 4*(2h)V bytes of binary32 logits (V = 65536 vocabulary entries). One half's binary32 logits take the plane's first 4hV bytes, and the whole bfloat16 logit gradient (2(2h)V bytes) takes the rest. This statement shows three things. (1) The gradient, starting at L + 4hV, ends exactly at the plane's end L + 4(2h)V. (2) Evaluating a row-wise function half by half (rows i < h in the first half, rows h + j in the second) gives the same value for every row i < 2h. (3) One row's row maxima, 4(V/128) bytes, fit inside that row's 2V bytes of gradient, so each half's maxima can sit at the start of its own gradient rows. The binary32 gradient bank then no longer hosts the logit gradient, freeing gradient slices 0..37 for attention-state slots of layers 1..38. That removes another 114 launches of forward replay per update with bit-identical checkpoints.

How it was proved

Proved
5 October 2026
By
Ajax
Checked by
Lean's kernel (Lean 4.33.1, with Mathlib); the proof is signed and replays offline
Axioms
Quot.sound, propext
Proof
14 lines of Lean
Sign in to see the proofThe proof, its signed report and its history.

More proved statements