Three walkers on one valley: plain descent, momentum, Adam. The learning rate decides whether anyone arrives. On the right, the same thing on 1.3 million parameters.
three walkers, one valley
the same idea on 1.3 million parameters: the Bible GPT’s loss
–
What optimisation is
Find the lowest point of a surface you can only feel underfoot. Gradient descent takes a step against the slope, again and again. The learning rate is the step size: too small and you crawl, too large and you bounce out of the valley altogether — drag the slider and watch.
Momentum and Adam
In a long narrow valley plain descent zig-zags. Momentum keeps some of the previous step (a heavy ball). Adam also scales each coordinate by how noisy its slopes have been, so it moves fast where the ground is flat and carefully where it is steep. Adam trained the Bible GPT on the right; its curve is this picture in a million dimensions.
Evidence. Left: the three update rules are run live (plain: x ← x − η∇f; momentum: v ← 0.9v + ∇f, x ← x − ηv; Adam: Kingma & Ba 2015 with β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸) on f = ½(x² + 12y²), a round bowl, or the Rosenbrock function; the finishing order and step counts are computed as you watch. Right: training and validation loss (nats per token) from data/biblegpt/run_small/log.json, the actual run of biblegpt/train.py (AdamW, warm-up then cosine decay) on the 66-book World English Bible.
Nine seconds elsewhere
The foundations in learning order, then the series they lead to. Each lands in about nine seconds.