A word costs log₂(1/p) bits. Price Genesis 1:1 word by word, watch the counter, and see why a better model is one that pays fewer bits.
Genesis 1:1, priced word by word — how many bits each word costs a model that knows only the previous word
0.0 bitsa fair coin is 1 bit · a certainty is 0 · rarer means dearer
entropy: how unsure a model is before it sees the next word
–
What information is
Surprise, measured in bits: a word with probability p costs log₂(1/p) bits. A word you were sure of is free; a one-in-a-thousand word costs ten bits. Entropy is the average surprise a source will hand you — after “in” the Bible is fairly predictable (– bits); after “and” almost anything can follow (– bits).
What it buys you
A yardstick for models. On six books it never saw, a model that knows word frequencies pays – bits per word; one that knows the previous word pays –; the small transformer trained on the Bible for this series pays –. Training is nothing but driving that number down — the loss curve of every language model is measured in these units.
Evidence. Surprisals for Genesis 1:1 use the corpus’s own bigram counts (first word: unigram). Held-out bits per word: unigram and add-0.1 interpolated bigram models fitted on 60 books and scored on Deuteronomy, Psalms, Micah, Luke, James and 1 Peter (– words); the transformer’s figure converts its validation loss on the same books (nats per BPE token) to bits per word using its tokens-per-word ratio — all in foundations/data/probability.json and data/biblegpt/run_small/log.json.
Nine seconds elsewhere
The foundations in learning order, then the series they lead to. Each lands in about nine seconds.