"How to compress text to 1 bit per character | Compression is Intelligence Part 3" — 3Blue1Brown
Why this is in the vault
Closes the three-part compression trilogy already partly filed (parts 1-2) and gives the cleanest explanation we have of how a language model's next-token distribution becomes an actual compressor. It also reframes pre-training loss as "bits per example", a useful lens for reasoning about model quality and size.
Episode summary
Grant Sanderson shows how arithmetic coding turns any predictor into a near-optimal lossless compressor: text maps to a tiny sub-interval of [0,1] whose width is the model's probability for that text, so bits needed are about negative log2 of that probability. A small char-level toy model gets about 2 bits per character on a sample sentence, while a 0.5B open model gets under 1 bit per character and compresses a famous constitutional preamble by two orders of magnitude versus Huffman. He closes by arguing compression captures three facets of intelligence: prediction, parsimony (small decoder plus model), and fast learning.
Key arguments / segments
- [00:00:00] Teaser: sub-1-bit-per-character text compression; the Hutter Prize (compress a gigabyte of Wikipedia, scored including decoder size) and the claim that every winner uses this one technique.
- [00:01:02] Shannon recap: optimal bits is roughly the negative log2 of message probability; predictable passages should cost a handful of bits, yet fixed-code schemes (ASCII, Huffman) cannot get near because they ignore context.
- [00:05:01] The move: give sender and receiver the same language model; better model means better compression. Model size is not the point, and small ones have an advantage once decoder size counts.
- [00:07:00] Probability stacks on the unit interval: chain rule makes each string a narrow region; encoding is picking a number inside it, decoding replays the model and zooms in.
- [00:12:00] Random numbers decode to plausible English, a visual for how LMs sample text and why optimal compressed output looks like noise.
- [00:13:01] Binary representation: each bit halves the interval. "math" fits in 10 bits; "mathematics" is lucky (10 bits vs 18 nominal) and drops to about 0.91 bits per character before length overhead.
- [00:16:00] Guarantee: you can always land in a region of width p using about ceil(-log2 p) bits, matching Shannon up to rounding; the real gap to Shannon is model quality, since p is the model's probability, not the truth.
- [00:20:00] Pre-training reframed: summed negative log predictions is cross entropy, so the objective can be read as "build the best compressor" rather than "predict next token".
- [00:22:02] Demo: nanoGPT (3.7M params) gets 254 bits (~2 bits/char) on a math-history sentence; Qwen (0.5B) gets under 1 bit/char; the preamble costs about 10 bits after its opening phrase.
- [00:25:02] Practicalities: naive real numbers need unbounded precision; arithmetic coding (Rissanen/Pasco, 1976) renormalizes; asymmetric numeral systems (2013) is the integer-based, faster alternative.
- [00:27:00] Compression is intelligence, framed like Newton defining energy: reductive but productive. Three facets follow.
- [00:28:01] (1) Compression and prediction are equivalent in both directions (size delta d implies probability of 2^-d); ties to predictive processing in neuroscience.
- [00:30:01] (2) Understanding as shrinkage: Hutter-style scoring includes decoder plus model size, so a small model matching a huge one wins, penalizing memorization.
- [00:32:01] (3) Fast learning: fine-tune the model on the fly, with the decoder replaying identical deterministic updates, so adaptation speed improves the compression ratio.
- [00:35:00] Trilogy wrap: parts 1-2 plus this suggest studying compression forces engaging with intelligence; best compressors need a small, accurate, fast-learning predictor.
Notable claims
- Every Hutter Prize winner uses the predictor-plus-arithmetic-coder structure; gains came from better predictors (00:25:02-00:26:01).
- A 0.5B open model compresses a sample sentence to under 1 bit/char vs about 2 for a 3.7M toy model (00:22:02).
- Counting decoder size in the score is what blocks the "hide Wikipedia in the decoder" cheat (00:32:01).
- Online fine-tuning inside the coder is used in practice and makes learning speed part of the score (00:34:01).
Guests
None. Solo presentation by Grant Sanderson (3Blue1Brown).
Mapping against Ray Data Co
Medium. Not an operational input, but it supplies precise vocabulary for conversations about model quality (cross entropy as bits, small-vs-large capacity tradeoff, memorization vs understanding) that phData customers and the founder's cert study (Claude/AWS AI tracks) touch. The "decoder size counts" framing is a clean argument against equating bigger models with better. Mild support for a Sanity Check angle on "why smaller models can be smarter"; no action required.
Related
- [[2026-07-16-3blue1brown-cross-entropy-compression-part2]]
- [[2026-06-07-3blue1brown-reinventing-entropy]]
- [[2026-04-20-3blue1brown-large-language-models-explained-briefly]]
- [[2026-05-08-david-perell-expression-is-compression]]