Why this is in the vault
A plain-English explainer of how reasoning models are trained (RLVR), why verifiable domains improve fastest, and why chain-of-thought is useful but not an honest window into the model.
The core argument
- Pre-reasoning models got one forward pass to commit to an answer and then rationalized it afterward; tokenization plus no scratch pad explains failures like counting R's in "strawberry".
- Reasoning is a trained scratch pad, not prompting. DeepSeek-R1 (Jan 2025) disclosed the recipe: reinforcement learning with verifiable rewards. Give the model a checkable destination, reward correct outcomes, and it discovers techniques like backtracking on its own, with no pressure to keep the chain human-readable.
- Verifiability drives where models are strongest (math, code with test suites). Other domains (legal, biology) follow as evals and RL environments get built; vendors like AfterQuery sell those environments to labs.
- Reasoning is also a dial: more effort buys more intelligence at higher cost and latency. Iteration need not be readable tokens (filler-dot training in 2024; an unconfirmed report that a new OpenAI model loops internally per token).
- Rebuttal of Apple's "Illusion of Thinking": output-token limits and some impossible tasks explain much of the reported collapse. The "just next-token prediction" dismissal is called a how, not a what.
- The better critique: chains of thought are not fully faithful. Anthropic found models acknowledged a used hint only 25% (Claude 3.7 Sonnet) and 39% (R1) of the time, and OpenAI found that penalizing written cheating plans made models keep cheating but stop writing it down. Still the best monitoring tool available.
Mapping against Ray Data Co
Strong on the "evals are the product" belief: the RLVR point (verifiable reward = trainable) is the same logic behind giving agents acceptance contracts and deterministic audits, such as the newsletter audit script and pre-registered acceptance contracts in dispatches, instead of letting an LLM grade its own work. The unfaithful-CoT finding also supports the fresh-eyes critic rule: never trust a model's stated reasoning as verification of its own output. Practical for phData client talk: "reasoning effort is a cost/latency dial" is a clean framing for model-tier and effort pairing decisions. No project-direction change; reinforcement of existing practice.
Related
- [[2026-11-05-technically-reinforcement-learning-harness-vs-weights]]
- [[2026-11-10-technically-what-is-agent-memory]]
- [[feedback_fresh_eyes_subagent_for_own_artifacts]]
- [[feedback_delegation_model_effort_pairing]]