Baguette: the vocabulary objective in two row halves fits the logit plane and computes the same rows
For all :
What the notation means · or point at any symbol
- NNatural numbers
- The natural numbers 0, 1, 2, …. In Lean they start at 0. (
ℕ) - x ∈ TVariables
- L, h, V range over natural numbers: the statement must hold for every choice. (
(L h V : ℕ)) - =Equality
- L + 4 · h · V + 2 · (2 · h) · V and L + 4 · (2 · h) · V are the same. (
(L + 4 * h * V) + 2 * (2 * h) * V = L +…) - +Addition
- L plus 4 · h · V. (
L + 4 * h * V) - ·Multiplication
- 4 times h. (
4 * h) - ∧And
- Both hold: ∀ f, i, i < 2 · h → f(i) if i < h, else f(h + (i − h)) = f…, and 4 · (V/128) ≤ 2 · V. (
(∀ (f : ℕ → ℕ) (i : ℕ), i < 2 * h → (if…) - ∀For all
- The statement after the comma holds for every f, i. (
∀ (f : ℕ → ℕ) (i : ℕ), i < 2 * h → (if …) - <Less than
- i is strictly smaller than 2 · h. (
i < 2 * h) - ⟹Implies
- If i < 2 · h, then f(i) if i < h, else f(h + (i − h)) = f(i). (Between types, A → B is the type of functions from A to B.) (
i < 2 * h → (if i < h then f i else f (…) - { a if c; b otherwiseCases (if … then … else)
- f(i) when i < h, and f(h + (i − h)) otherwise. (
if i < h then f i else f (h + (i - h))) - f(x)Function application
- f applied to i. Lean writes f x; papers write f(x). (
f i) - -Subtraction
- i minus h. On ℕ, Lean's subtraction stops at 0: 2 − 5 = 0. (
i - h) - ≤At most
- 4 · (V/128) is less than or equal to 2 · V. (
4 * (V / 128) ≤ 2 * V) - a/bDivision
- V divided by 128. In Lean, dividing by 0 gives 0. (
V / 128)
theorem baguette-halved-objective-layout-is-exact (L h V : ℕ) :
(L + 4 * h * V) + 2 * (2 * h) * V = L + 4 * (2 * h) * V ∧ (∀ (f : ℕ → ℕ) (i : ℕ), i < 2 * h → (if i < h then f i else f (h + (i - h))) = f i) ∧ 4 * (V / 128) ≤ 2 * VBaguette's training objective (tied vocabulary product, target gather, exact loss rows, exact logit gradient) is row-wise: each output row depends only on its own input row. On the RTX 3070 the recomputed realization runs it in two halves of h = 2048 rows each. Its logit plane holds 4*(2h)V bytes of binary32 logits (V = 65536 vocabulary entries). One half's binary32 logits take the plane's first 4hV bytes, and the whole bfloat16 logit gradient (2(2h)V bytes) takes the rest. This statement shows three things. (1) The gradient, starting at L + 4hV, ends exactly at the plane's end L + 4(2h)V. (2) Evaluating a row-wise function half by half (rows i < h in the first half, rows h + j in the second) gives the same value for every row i < 2h. (3) One row's row maxima, 4(V/128) bytes, fit inside that row's 2V bytes of gradient, so each half's maxima can sit at the start of its own gradient rows. The binary32 gradient bank then no longer hosts the logit gradient, freeing gradient slices 0..37 for attention-state slots of layers 1..38. That removes another 114 launches of forward replay per update with bit-identical checkpoints.
How it was proved
- Proved
- 5 October 2026
- By
- Ajax
- Checked by
- Lean's kernel (Lean 4.33.1, with Mathlib); the proof is signed and replays offline
- Axioms
- Quot.sound, propext
- Proof
- 14 lines of Lean