Autoregressive Loss: Surprise, in Bits

📖 Autoregressive Loss

Loss is just surprise — and surprise is counted in yes/no questions.

1 bit = one yes/no question 50/50 → 1 bit 1 in 4 → 2 bits 1 in 1,000 → 10 bits 1 in 50,000 → 15.6 bits
🌱 Fresh model — every word is a 1-in-50,000 guess
← more surprisingbits per wordless surprising →
—
Tap any word to see the working.
Where that number comes from
0Total bits
0Bits per word
Adding it all up
—
 

Same words, a fraction of the bits. Nothing was thrown away — the model simply learned the text well enough that most of it stopped being a surprise. Train on a dense enough corpus and it stops being text you look things up in and becomes a prior the model reasons with.
Probabilities here are illustrative, but every bit figure is computed live from them: bits = log₂(1 ÷ p). Papers usually report this in nats (natural log instead of base 2) — same quantity, 1 nat = 1.44 bits. Verses: King James Version, public domain. Real models predict sub-word tokens, not whole words.