"Reinventing Entropy" — 3Blue1Brown

Why this is in the vault

Part 1 of a 3B1B trilogy on Compression & Intelligence that rebuilds Shannon's information/entropy definitions from scratch by asking "what is the fundamental limit on compressing language?" — and lands on the load-bearing claim that prediction and compression are mathematically equivalent, which means LLM pretraining (cross-entropy loss) is literally an exercise in building the most efficient possible text compressor. This is the cleanest available citable derivation of the math under the tools Ray runs as COO, and it gives RDCO a rigorous frame for thinking about its own curation pipeline as a compression operation.

Episode summary

Grant motivates information theory by posing a single question — is there a hard floor on how compactly text can be encoded? — and walks the viewer through "rediscovering" the definitions of information and entropy rather than being handed them. A toy lunar-robot example (four movement instructions with probabilities 1/2, 1/4, 1/8, 1/8) shows that a variable-length prefix-free code beats naive fixed-width encoding (1.75 vs 2 bits/instruction), and that the optimal code allocates code-space to each symbol in exact proportion to its probability. A "head-in-the-clouds" student then argues abstractly that perfect compression must look like random noise (because random noise is incompressible), which forces the conclusion that a message taking n bits has probability 2^-n — and therefore the information of an event is −log₂(p), the founding formula of the field. Averaging information across a distribution yields entropy, H = Σ p·(−log₂ p), which the noiseless coding theorem establishes as the unbeatable lower bound on bits per symbol. Grant closes by retracing how Shannon actually estimated the entropy of English — n-gram tables, the Betty-Shannon letter-guessing game, and the 1950 "Prediction and Entropy of Printed English" paper — emphasizing that you cannot estimate language's compressibility without probing an intelligent model of language (a human brain then; an LLM now). The episode teases Part 2 (cross-entropy in LLM training, distillation, why gzip recovers structure across languages) and Part 3 (an LM-based compressor that hits Shannon's limit in practice).

Key arguments / segments

• [00:00:00] The motivating question. ASCII spends 8 bits/char; cleverness gets to ~4; sequence-pattern methods do better still — but is there a fundamental limit? The question dates to Shannon's 1940s work that founded information theory.
• [00:01:03] Prediction = compression. Cross-entropy loss (the LLM pretraining objective) has roots in information theory; one of the field's conclusions is that prediction and compression are two sides of the same coin, so pretraining can be reframed as building the most efficient text compressor. Some go further: "compression is intelligence."
• [00:03:02] Lunar-robot warm-up. Four instructions (up/down/left/right) sampled independently with non-uniform probabilities (1/2, 1/4, 1/8, 1/8); the puzzle is the most bit-efficient encoding of the instruction stream.
• [00:05:02] Variable-length prefix-free codes. Assigning 0 / 10 / 110 / 111 gives 1.75 bits/instruction vs the naive flat 2 bits. The decoding constraint: no codeword may be a prefix of another, so the receiver reads until a complete codeword forms.
• [00:08:03] The binary-tree code-space diagram. Each codeword consumes a share of the space of all binary strings equal to its probability — the "pleasing alignment" between optimal code-length allocation and the underlying probabilities is the founding insight of information theory.
• [00:10:02] Perfect compression looks like random noise. A theoretical student argues random noise is incompressible, so an optimal bitstream is indistinguishable from random; saving one bit on one message provably costs ≥2 bits elsewhere ("push down a bump on the rug").
• [00:15:01] Deriving information. A message that takes n bits under perfect compression has probability 2^-n; taking −log₂ of both sides gives bits = −log₂(p), defined as the information of an event. Unlikely events carry more information; near-certain events carry almost none.
• [00:18:04] Language is the hard case. Per-letter probabilities are heavily context-dependent and not clean powers of 2, so information is fractional; the chain rule plus logarithms turn the product of per-letter probabilities into a sum of per-letter information — so a message's total information adds up cleanly even when individual values are fractional.
• [00:21:03] How Shannon actually estimated it. n-gram statistics break down for long/unseen strings; he instead used his wife Betty as a language model (letter-guessing game) and, in his 1950 paper, recorded the number of guesses humans needed — treating the human brain as a black-box predictor. Today we design those black boxes instead of merely interrogating them.
• [00:25:02] Entropy reinvented. Average information per symbol, H = Σ p·(−log₂ p), visualized as the total area of probability-weighted information rectangles; the noiseless coding theorem says no encoding beats this and you can get arbitrarily close. Generalizing to context-dependent language requires the entropy rate of a stochastic process.

Notable claims

• [00:06:00] Variable-length prefix code yields 1.75 bits/instruction for the robot vs 2.0 bits for naive fixed-width encoding.
• [00:15:01] Information of an event = −log₂(p) — described as the fundamental formula of all information theory; the number of bits a message gets under perfect compression equals its information content, and information is a lower bound on compression averaged over all messages.
• [00:26:00] Entropy H = Σ p·(−log₂ p) = minimum bits/symbol to encode a source; the von Neumann "call it entropy — nobody knows what entropy is, so you'll always win the argument" naming anecdote (Grant flags it as probably apocryphal but with a grain of truth).
• [00:30:02] Shannon estimated English at ≈ 1 bit per character when guessers had 100+ characters of preceding context — i.e. English could in principle compress to one yes/no answer per character.
• [00:01:03] Cross-entropy (the LLM pretraining loss) is information-theoretic, and prediction/compression equivalence reframes "next-token prediction" as "most efficient text compressor."

Sponsorship

Self-promo only. [00:31:00] Grant plugs 3B1B's own virtual career fair at 3b1b.co/talent (featured interviews between him and hiring teams) and the support/early-access page at 3b1b.co/support. No third-party commercial sponsor; this is house/audience-funded promotion.

Mapping against Ray Data Co

Strong relevance on three load-bearing fronts:

1. The curation pipeline IS a compression operation. RDCO's /process-newsletter and /process-youtube skills take high-volume, low-density sources and emit the highest-information-per-token assessment note. Grant's "perfect compression looks like random noise" insight is a precise frame for what a good summary does: strip every predictable, low-information token (filler, restated framing, expected boilerplate) so only the surprising, high-information claims survive. The CLAUDE.md context-rot discipline (route long artifacts through subagents, keep only the extract) is the same move — minimize bits in parent context while preserving information content. This is a genuinely original lens on work RDCO already does, not a restatement of the source.

2. It is the math under the tools Ray runs. Cross-entropy loss, next-token-prediction-as-compression, and entropy-as-lower-bound are the theoretical substrate of every LLM in the COO stack. This note pairs directly with the existing "[[~/rdco-vault/06-reference/2026-04-20-3blue1brown-large-language-models-explained-briefly.md]]" primer (which covers pretraining-vs-RLHF at the structural level) and gives the deeper "why cross-entropy" answer that primer leaves implicit. Citable for any Sanity Check piece or client briefing that touches how/why LLMs are trained.

3. Clean translation to the founder's data-engineering muscle. Prefix-free codes, the −log₂(p) information measure, and dictionary/entropy coding are exactly the primitives behind gzip, columnar dictionary encoding, and Parquet/ORC compression — tools the founder already knows cold. The trilogy even teases gzip in Part 2. The "compression ↔ intelligence" thesis also feeds the founder's harness-thesis / AI-capability interests: it's the rigorous version of the squishy "scaling = intelligence" claims, anchored to a theorem rather than vibes.

Watch for Parts 2 and 3 — they (cross-entropy in training, distillation, gzip-across-languages; then an LM-as-compressor hitting ~1 bit/char) are even more directly RDCO-relevant than Part 1.

Related

• [[~/rdco-vault/06-reference/transcripts/2026-06-07-3blue1brown-reinventing-entropy-transcript.md]]
• [[~/rdco-vault/06-reference/2026-04-20-3blue1brown-large-language-models-explained-briefly.md]]
• [[~/rdco-vault/06-reference/2026-04-20-3blue1brown-volume-higher-dim-spheres-most-beautiful-formula.md]]