"Adam Marblestone – AI is missing something fundamental about the brain" — Dwarkesh Podcast

Why this is in the vault

Marblestone offers a concrete theoretical frame — the learning/steering subsystem split — for why current LLMs may hit a sample-efficiency ceiling, which directly informs how RDCO thinks about AI agent reward design and what "aligned" agent behavior actually requires at an architectural level.

Episode summary

Adam Marblestone, CEO of Convergent Research, argues that AI researchers have underweighted the role of specific, evolutionarily-shaped loss functions in explaining the brain's sample efficiency advantage over LLMs. He introduces neuroscientist Steve Burns' framework of a cortical "learning subsystem" paired with a subcortical "steering subsystem" whose bespoke reward circuits evolution encodes compactly — analogous to a few hundred lines of Python — rather than pre-training the learner. The conversation ranges from connectomics roadmaps and formal mathematics to alignment risks and the gap map of infrastructure missing from modern science.

Key arguments / segments

• [00:01:00] The brain's missing ingredient is probably not architecture or the learning algorithm — it is the specificity of loss functions and cost functions; evolution may have packed an entire curriculum into the reward signal rather than into weights
• [00:06:00] Steve Burns' learning/steering framework: the cortex is a general-purpose learning subsystem; the hypothalamus, amygdala, and basal ganglia form a steering subsystem with genetically pre-wired innate reward heuristics that bootstrap cortical learning
• [00:09:00] How abstract generalization emerges from innate rewards: the cortex trains predictors of the steering subsystem's responses, allowing concepts like "social embarrassment" to generalize far beyond the situations evolution ever witnessed (the spider/Yan LeCun example)
• [00:22:00] Omnidirectional inference hypothesis: the cortex may natively compute conditional probabilities in any direction across any subset of inputs — unlike LLMs, which are hardcoded to predict the next token — making it closer to an energy-based model or probabilistic inference engine
• [00:28:00] Genome compactness argument: the human genome is small because reward functions are algorithmically cheap to specify; most genomic real estate encodes the steering subsystem's bespoke circuits, not the learning algorithm itself
• [00:36:00] Why hominid brains scaled so fast: social learning raised the return on having a larger cortex, and scaling the cortex required only a small number of additional genes — it is essentially replicating what mammals already had
• [00:43:00] RL in the brain: the basal ganglia does something like model-free temporal-difference RL with a small finite action space; the cortex builds a world model that contains a model of its own reward function, giving model-based RL-like behavior for free
• [00:57:00] Alignment implication: minimum viable intelligence requires far less steering subsystem complexity than a human; a highly capable model may be achievable without human social instincts, making the paperclip-maximizer failure mode genuinely possible
• [01:04:00] Reframing interpretability: rather than tracing individual circuits, the goal should be describing the brain's architecture, learning rules, cost functions, and initialization — the same language used to describe neural networks we built
• [01:19:00] Brain-data-augmented AI: training models with neural activity patterns as auxiliary loss functions could improve generalization and robustness, analogous to how the cortex predicts the steering subsystem as a training signal
• [01:23:00] Lean and formal mathematics: RLVR on mechanically verifiable proofs is working and will accelerate math; formal verification extends to software security (provably unhackable code), and Convergent Research is incubating an FRO on this
• [01:44:00] The gap map: Convergent Research mapped a few hundred fundamental science infrastructure gaps across biology, math, and other fields; each is roughly series-A scale — the total cost to address all of them is far below what AI hardware spending will reach

Notable claims

• Current LLM training uses "the dumbest form of RL" — no value functions, no temporal difference learning — yet it works, which Marblestone finds both impressive and theoretically surprising [00:43:00]
• The number of distinct cell types in the hypothalamus and steering subsystem regions exceeds those in the cortex; this asymmetry is empirical evidence that more genomic real estate encodes reward structure than learning algorithm [00:30:00]
• First full mouse connectome would cost ~$1-3B with electron microscopy; E11 Bio's optical approach targets $10-30M, paralleling the cost collapse in genome sequencing [01:11:00]
• Temporal difference learning (dopamine as reward prediction error) appears to have been independently discovered by neuroscience and AI; Sutton described the math before the neuroscience evidence confirmed it [01:03:00]
• Marblestone's AGI timeline estimate: 10+ years, contingent on the current LLM paradigm proving inadequate and a brain-informed paradigm emerging; he holds this view with low conviction and treats Dwarkesh's guest distribution as his data [01:09:00]
• A provably secure quadcopter has already been built via DARPA formal verification; Marblestone believes LLM-aided Lean proof synthesis could make formally verified software mainstream within a few years [01:28:00]

Guests

Adam Marblestone — CEO, Convergent Research (focused research organization incubator); former Research Scientist at Google DeepMind's neuroscience team; PhD advised by George Church at Harvard; board member, Lean (formal mathematics language). Works at the intersection of neurotechnology, AI, and scientific infrastructure. Blogs at Longitudinal Science.

Mapping against Ray Data Co

Agent reward design: The learning/steering split is the most actionable frame for RDCO. When building AI agents with Claude, the "steering subsystem" analog is the system prompt plus tool affordances — the pre-specified reward heuristics the agent can't modify. Marblestone's argument implies that getting this specification precise and rich (many targeted loss functions, not one vague objective) matters more than scaling compute.

Test-time compute and amortization: The discussion of amortized inference vs. real probabilistic inference maps directly to how RDCO uses reasoning models. The insight that test-time compute is de-amortizing a previously hard inference problem — and that distillation re-amortizes it — clarifies when to prefer extended thinking vs. a fast pass in production agent workflows.

Synthetic data strategy: The brain-data-augmented training idea is a useful analogy for synthetic data design: auxiliary prediction tasks that force a model to represent the world in a way consistent with how domain experts represent it may generalize better than label-only training. Relevant if RDCO builds fine-tuned or distilled models for specific verticals.

Formal verification for agent outputs: The Lean/RLVR thread has near-term relevance. For any RDCO agent operating in a domain with a verifiable ground truth (financial calculations, data transforms, code generation), adding a formal-verification check layer aligns with Marblestone's prediction that this capability will become mainstream in 2-3 years.

Alignment posture: The steering subsystem alignment argument is sobering: a capable AI system may need only minimal social instincts — curiosity and interest in others — to function well, meaning alignment cannot be assumed from capability alone. This reinforces RDCO's existing posture of treating Claude as a tool with explicit guardrails rather than an inherently value-aligned collaborator.

Related

• [[2026-04-15-thariq-claude-code-session-management-1m-context]]
• [[02-sops/2026-05-19-verification-as-independent-worker-pattern]]