Singular value decomposition: rotate, stretch, rotate
Every matrix is those three moves. Rank the stretches, keep the biggest few, and you have the best possible compression of any table — here, the Bible’s words by books.
rotate · stretch · rotate
240 words × 66 books, rebuilt from k directions
–
What the SVD says
Every matrix, whatever it does, is a rotation, then a stretch along perpendicular axes, then another rotation: M = U Σ Vᵀ. The stretch factors σ₁ ≥ σ₂ ≥ … are the singular values. Watch the circle on the left: turned, stretched into an ellipse, turned again — that ellipse is the matrix.
What it buys you
Rank the directions by how much they stretch and keep only the top k: that is the best possible rank-k approximation (Eckart–Young). On the right, the Bible’s word-by-book table is rebuilt from a handful of directions; the first is “how common a word is”, the second separates the epistles from the histories — found by arithmetic, not by reading.
Evidence. Left: the 2×2 SVD is computed on the page (closed form from the eigen-decomposition of MᵀM). Right: rows are the 240 most frequent content words of the World English Bible, columns its 66 books, entries log(1 + rate per 1,000 tokens); U, Σ, Vᵀ from numpy.linalg.svd in foundations/compute_data.py, stored in foundations/data/svd.json; the page multiplies UkΣkVkᵀ. Energy = share of Σσ² kept; error = relative Frobenius norm.
Nine seconds elsewhere
The foundations in learning order, then the series they lead to. Each lands in about nine seconds.