"What does the next training paradigm look like?" — Dwarkesh Patel

Why this is in the vault

Dwarkesh argues that RLVR (RL with verifiable rewards) is the current training paradigm but has a hard ceiling: it only works in "grindable" domains where you can run thousands of parallel rollouts from a deterministic starting state. Everything beyond that — politics, markets, company-building — requires sample efficiency and continual learning that RLVR alone cannot deliver. He then maps two credible near-term techniques (OPSD and "dreaming") that could unlock weight-level learning from deployment data. This is required background for anyone advising on AI strategy or sitting for the Anthropic Claude Certified Architect exam — it frames exactly where the current generation of models hits its ceiling and what research bets the labs are making to break through it.

Episode summary

This is a solo video essay (no guest). Dwarkesh walks through three nested questions: (1) Why is the current RLVR training paradigm probably not sufficient for general AGI? (2) Why has computer use lagged other domains, and what does that reveal about RL's structural constraints? (3) What training techniques might fill the gap — specifically on-policy self-distillation (OPSD) and "dreaming" (model-generated synthetic RL environments)? He closes with a 2027-2028 scenario where deployment experience becomes the primary driver of model improvement, replacing pre-training and RL as the dominant signal.

Key arguments / segments

• [00:00:00] The RLVR bet. Labs believe that training on millions of verifiable tasks across diverse RL environments will produce a general problem-solving agent. Optimists say data-inefficiency and lack of continual learning will be steamrolled by scale, just as NLP problems collapsed under LLM compute.

• [00:01:00] The amortized cost argument. Training is a one-time cost spread across billions of inference sessions. What matters is session-time sample efficiency, which is visibly improving with more RL training. Long context windows may make continual learning unnecessary if in-context learning scales far enough.

• [00:02:00] Computer use as a diagnostic. Computer use is clearly verifiable, yet progress has been far slower than coding or math. The reason is grindability: you need parallel rollouts from identical starting states. You can containerize a code repo 1,000 ways; you cannot run 1,000 bots through the same Amazon checkout.

• [00:04:00] The grindability constraint generalizes. Once AIs can build high-fidelity app clones, computer use will accelerate. But most high-value domains — court cases, elections, business-building — have months-long outer-loop feedback that cannot be replicated inside a data center.

• [00:06:00] RLVR generalization is an open empirical question. Labs are betting that enough containerized training generalizes to open-ended real-world tasks. Dwarkesh cites Dario Amodei's comment that short-context RL training degrades at long-context serving as evidence that generalization may not be "infinitely strong."

• [00:07:00] Deployment compute is wasted. 30-50% of a lab's compute goes to inference, currently generating zero weight updates. The most valuable training signal — what actually happens in real deployments — is being discarded. Dwarkesh's analogy: a genius grad student denied any real internship, only ever given classroom case studies.

• [00:09:00] Why continual learning is hard. KV cache growth doesn't scale and doesn't compress like human memory. Gradient updates are sample-inefficient. Online learning (e.g., Cursor's tab model at 400M requests/day) only works when millions of users are all learning the same objective — it breaks for per-user or per-org specialization.

• [00:12:00] OPSD (on-policy self-distillation). The technique distills what a "veteran" model (with full session context) learned back into the base model's weights, using per-token probability discrepancy as the supervision signal. Advantages over naive SFT: targets relevant knowledge rather than verbatim replay. Advantage over RLVR: no outer-loop verifiable reward needed.

• [00:15:00] Dreaming. A more speculative fourth scaling axis: the model generates its own RL training environments and trains against them between sessions. Analogous to EfficientZero's internal simulation for Atari. Would allow orders-of-magnitude more synthetic samples per wall-clock hour. Dwarkesh frames it as the difference between hitting "compact" vs hitting "dream" at session end.

• [00:17:00] 2027-2028 scenario. RLVR produces an agent capable of operating in the real world. Extended context allows week-long co-work sessions. After each session, the base model distills via OPSD or dreaming. The AI's skill envelope expands outward from verifiable domains into adjacent open-ended ones. Eventually, deployment experience — not pre-training or RL — becomes the primary improvement driver.

Notable claims

• Current models are roughly 1/1,000,000th as sample-efficient as humans during training (carried over from a prior Dwarkesh essay).
• 30-50% of lab compute goes to inference with zero weight-improvement payoff.
• Cursor's tab model processes 400M+ daily requests, all learning the same single objective (edit acceptance).
• Short-horizon RL training may not generalize to long-horizon RL performance (inferred from Dario Amodei quote on context-length degradation).
• RL updates very few parameters per step — which is actually desirable for continual learning because it minimizes catastrophic forgetting.

Guests

None. This is a solo video essay by Dwarkesh Patel, presented as a narrated version of a blog post also published at dwarkesh.com.

Sponsorship

Sponsored by Mercury (fintech banking platform). Mercury is presented as automating Dwarkesh's contractor bill-pay via an invoice-scanning email inbox feature. Standard disclosure: Mercury is a fintech company, not an FDIC-insured bank; banking services via Choice Financial Group and Column NA.

Mapping against Ray Data Co

Strong relevance. Three direct connections:

1. Anthropic cert prep (Claude Certified Architect — Foundations, target 2026-11-22). The cert covers model training paradigms and agent architecture. This essay is a compact mental model of where RLVR sits in the scaling stack, what its limits are, and what the next paradigm (OPSD + dreaming) looks like. That framing will appear in architecture questions about when to trust model capability vs. fine-tune vs. rely on in-context learning.

2. phData AI advisory work. When scoping AI solutions for clients, the grindability constraint is a practical ceiling: enterprise use cases (sales forecasting, org-specific knowledge) often fall outside verifiable/grindable domains. This essay gives language for explaining why RAG + fine-tuning is the practical path for those domains today, while continual learning is the 2027+ horizon.

3. RDCO agent infrastructure. The "deployment compute is wasted" argument maps directly to the always-on COO agent design: the agent accumulates session-specific context but currently cannot distill it to weights. OPSD would be the mechanism that makes session learning durable. Worth flagging when evaluating whether to persist structured memory vs. wait for native continual learning.

Related

• [[06-reference/2026-04-15-thariq-claude-code-session-management-1m-context]] — context management and the session-length problem that Dwarkesh's "dreaming" proposal addresses
• [[02-sops/2026-06-09-claude-md-prompt-precedence-full]] — agent architecture decisions that would be affected by continual learning becoming available
• [[06-reference/transcripts/2026-06-26-dwarkesh-next-training-paradigm-transcript]] — full transcript