"But what is cross-entropy? | Compression is Intelligence Part 2" — 3Blue1Brown

Why this is in the vault

Grant Sanderson derives cross-entropy loss from first principles as a mathematical inevitability — not an arbitrary design choice — making this the cleanest available citable foundation for understanding why LLMs are trained the way they are. Part 2 of a trilogy connecting compression to intelligence; directly relevant to LLM internals, phData AI consulting, and both cert study tracks (Snowflake GenAI + Anthropic Claude foundations).

Episode summary

Grant builds cross-entropy from Shannon information theory and a toy robot-instruction encoding example, then shows how cross-entropy naturally measures the cost of using a code optimized for one distribution (Q) when reality follows a different distribution (P). He then derives why the negative log is the only function that forces a loss to be minimized exactly when a model's output matches the statistics of training data — making cross-entropy loss a mathematical inevitability, not a choice. The video closes by introducing distillation as a richer application of cross-entropy and KL divergence as its natural complement, teasing the compression-as-intelligence thesis to be completed in Part 3.

Key arguments / segments

• [00:00:00] Hook: 2002 "Language Trees and Zipping" paper — using gzip co-compression to cluster languages and recover their evolutionary tree without any linguistic knowledge
• [00:02:01] Video roadmap: compression fundamentals (Part 1) → cross-entropy derivation → LLM pre-training connection → compression-as-intelligence reframe
• [00:03:02] Toy example revisited: biased robot-instruction distribution; optimal encoding allocates −log₂(p) bits per symbol (Shannon information content from Part 1)
• [00:06:01] Cross-entropy formula generalized with P (true/new distribution) and Q (code-optimized distribution); visualized as bar chart with width=p, height=−log(q)
• [00:10:00] Key property established: H(P,Q) is minimized exactly when P=Q, and that minimum equals entropy H(P); entropy is the theoretical compression floor
• [00:15:01] Transition to LLM training: models as next-token probability functions; loss function design problem framed from scratch
• [00:21:02] Why logarithm is forced, not chosen: "my name is blank" argument with Lagrange multiplier proof — any function satisfying the constraints must be log
• [00:26:00] Distillation introduced: training a small model against a large model's full output distribution rather than one-hot true tokens; richer signal analogized to chess coaching
• [00:30:00] KL divergence defined as cross-entropy minus entropy; asymmetric "wasted bits" measure; closes with three open questions for viewer to ponder

Notable claims

• [00:04:00] Optimal bit allocation per symbol = −log₂(p); Shannon named this the "information content" of an event — the quantity minimized by good compression
• [00:06:01] Cross-entropy H(P,Q) is the concrete average bits per symbol when a code built for Q faces a reality distributed as P — not abstract formula but a compression cost
• [00:10:00] H(P,Q) ≥ H(P) always, with equality iff P=Q; entropy is the strict lower bound of cross-entropy for any fixed P
• [00:13:02] The 2002 gzip language-tree trick is an empirical approximation of cross-entropy, not exact, because gzip is far from Shannon-optimal compression
• [00:19:01] Natural log vs. log base 2 differs only by a constant factor absorbed into the learning rate; natural log preferred because its derivative is cleaner for backpropagation
• [00:25:00] Forcing f to be logarithmic is provable via Lagrange multiplier: minimizing weighted sum subject to sum-of-Q=1 requires the derivative of f w.r.t. q to equal constant/q, a property unique to logarithms
• [00:32:02] KL divergence is asymmetric: KL(Q‖P) ≠ KL(P‖Q); it measures bits wasted per symbol by using the wrong code, and equals zero iff distributions are identical

Guests

Grant Sanderson (solo — no guests)

Sponsorship

Doppel is named as a partner company in a promotional segment (~[00:30:00]–[00:31:01]). The video also promotes 3b1b.co/talent (Grant's own job-board for companies hiring from his audience).

Mapping against Ray Data Co

Cross-entropy loss is the literal training objective for Claude, GPT, and every major LLM, so this video provides the mathematical grounding to explain why models behave as they do during pre-training and why distillation produces smaller but capable models — directly applicable to Snowflake GenAI Specialty exam questions on model training mechanics and to the Anthropic Claude cert's expectation that practitioners understand model internals. For phData consulting, the "cross-entropy as distribution distance" framing is immediately useful when evaluating fine-tuned model quality, explaining LLM evaluation metrics to clients, or designing loss functions for domain-adapted models. The distillation segment maps cleanly to the enterprise pattern of deploying a large frontier model to generate training signal for a cheaper inference-time model — a cost-optimization argument phData can make to Snowflake and Azure customers.

Related

• [[~/rdco-vault/06-reference/2026-06-07-3blue1brown-reinventing-entropy.md]] — Part 1: entropy/compression foundations this video builds on
• [[~/rdco-vault/06-reference/2026-04-20-3blue1brown-large-language-models-explained-briefly.md]] — Grant's high-level LLM explainer; cross-entropy connects the math under that treatment
• [[~/rdco-vault/06-reference/2026-06-30-dwarkesh-grant-sanderson-3blue1brown-ai-future-math.md]] — Grant on AI and math education; context for his pedagogical approach to ML internals