06-reference

stratechery inference shift agentic

2026-05-11·reference·source: Stratechery·by Ben Thompson
agentic-inferencecompute-substrateharness-engineeringai-infrastructurecerebrasnvidia

"The Inference Shift" - @Ben Thompson

Why this is in the vault

Thompson splits "inference" into two regimes (answer inference vs agentic inference) and argues the agentic regime makes the compute substrate look fundamentally different - memory hierarchy over compute speed, capacity over latency, "good enough" silicon over bleeding edge. This is a substrate-level claim that lives directly underneath the harness-engineering thesis cluster RDCO has been building this week (Osmani, Tobi, Avedissian, Tan, AlphaSignal, AVB, DEW 269). Filing because it changes the question of what the harness is actually orchestrating.

The core argument

Three inflection points so far: ChatGPT (token prediction utility), o1 (reasoning - more tokens = better answers), Opus 4.5 + Claude Code (first usable agents via reasoning model plus harness that uses tools and verifies work).

Compute substrate today is GPU-dominant because:

Cerebras represents the wafer-scale-engine alternative: 44GB on-chip SRAM at 21 PB/s vs an H100's 80GB HBM at 3.35 TB/s. Half the memory, 6,000x the bandwidth. Built for "answer inference" where speed matters, but bottlenecks the moment KV cache or model exceeds on-chip memory.

The pivot: "agentic inference" is structurally different. When there is no human in the loop, latency stops being the binding constraint. What matters instead is the memory hierarchy around the model: active KV cache, host memory, SSDs, databases, logs, embeddings, object stores. Speed for capacity is the right trade.

Three claims that fall out:

  1. Training keeps Nvidia's current architecture dominant.
  2. Answer inference becomes a meaningful but small market for Cerebras/Groq-style speed plays.
  3. Agentic inference is the largest market by far, because it "scales not with humans but with compute" - and it unbundles the GPU, favoring cheaper DRAM, "good enough" compute, and faster CPUs for tool use over leading-edge GPU bandwidth.

Secondary implications: China has "everything it needs for agentic inference" since it does not require leading-edge silicon. Space data centers become viable because slower/older nodes run cooler, handle radiation better, and use less power. Moore's Law stops mattering because the compute we have is already enough.

Mapping against Ray Data Co

This is the strongest substrate-level data point of the week and lands as a complement-with-an-edge to the harness-engineering thesis cluster.

Where it reinforces the thesis:

Thompson's framing of agentic inference puts the harness layer at the center of where value is created. Quote: "the future of computing speed-ups will be a function of systems innovation." If the substrate fragments into training silicon + answer-inference silicon + agentic-inference memory hierarchy, then the layer that orchestrates which workload runs where becomes load-bearing. That is exactly the harness. The Osmani / Avedissian / AVB position - that the harness is where moat accrues - looks more durable, not less, in a world of heterogeneous substrate, because heterogeneity creates surface area for orchestration to differentiate.

His point that agentic inference is mostly "waiting on memory" and that CPU speed for tool use matters more than GPU speed maps cleanly onto how harnesses actually run: most wall-clock time in a long-horizon agent run is I/O, retrieval, and verification, not raw token generation. The harness manages that hierarchy. See [[06-reference/2026-05-10-addy-osmani-agent-harness-engineering]] and [[06-reference/concepts/2026-05-10-harness-moat-two-layers-portability]].

Where it cuts the other way:

Thompson's other claim is that agentic inference is the largest market by far because it "scales not with humans but with compute." If that compounds the way he expects, the supply of agentic capability grows much faster than RDCO's personal-fit moat assumes. The Avedissian / AVB version of the thesis ("a harness encodes a specific person's workflow and is non-portable") survives, but the AlphaSignal / Tan version ("anyone can spin up an agent in 12 hours") gets cheaper still. Net: the personal-fit moat narrows from "12 hours of setup" toward "a few minutes," which raises the bar on what makes a particular founder-harness sticky beyond just having one at all.

A second tension: if agentic inference does unbundle into commodity memory + good-enough compute, the substrate cost curve falls fast, which means the COO-agent operating cost drops fast too. That is unambiguously good for RDCO's "small bet portfolio + COO-agent" cost structure, but it also means competitors face the same drop. Capability is not the moat. The harness shape and the founder-fit are.

What it does not change:

The two-layer harness moat argument ([[06-reference/concepts/2026-05-10-harness-moat-two-layers-portability]]) is orthogonal to where the inference happens. Whether agentic inference runs on Nvidia, Cerebras, a DRAM-heavy custom rack, or in low-orbit, the harness still has to encode the founder's workflow, decision authority, and judgment surface. Thompson's piece is upstream of that question, not a substitute for it.

Action implication: none yet. This is a substrate forecast, not a near-term operational input. Worth re-reading in 6 months against actual hyperscaler capex disclosures to see whether the "GPU unbundling for agentic" call is playing out.

Related