06-reference/research

agent scaffolding generation survival

2026-08-19·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)

Which agent scaffolding survives a generation jump — few-shot dies, decomposition doesn't, verification splits in half

The question

"Which categories of agent capability scaffolding (few-shot examples, task decomposition, verification steps) empirically survive multiple frontier-model generation jumps, versus needing to be rebuilt each time?"

This sharpens the open question left unanswered in [[2026-08-01-scaffolding-vs-harness-the-erosion-axis]] — whether category-1 capability scaffolding erodes uniformly, or whether some kinds outlive others. The note's stated guess was that verification persists longest. That guess is half right, and the half that is wrong is the half we were about to sell to a client.

What we already know (from the vault)

What the web says

Convergences and contradictions

Synthesis for RDCO

The operating test in the erosion-axis note — is this here because the model is dumb, or because the model can't know this? — mis-sorts two of the three categories in this question. Decomposition and artifact verification are both there "because the model is dumb," by that phrasing, and neither eroded. The test needs replacing, and the replacement is a better question anyway: what is this scaffolding's input?

Scaffolding whose input is the model's own output distribution erodes as that distribution tightens. Few-shot exemplars shape token continuations; self-consistency harvests sample variance; a same-family critic reads a same-family producer. All three get their leverage from model error, and every generation jump takes some of that supply away. This category should be built cheap, instrumented, and deleted on schedule — and self-consistency specifically answers the note's second open question about a cost-driven fourth category: it is the fourth category, eroding on the capability curve and the price curve simultaneously, which is why it is the first thing to cut.

Scaffolding whose input is outside the model does not erode, because a smarter model does not change the input. A filesystem either has the plan file or it doesn't. A test either passes or it doesn't. KwikTrip's hot-dog-roller failure modes are either written down or they aren't. This is the same set the note already called particular knowledge, extended with a category it had misfiled — external-ground-truth verification. Decomposition sits in a third position: its input is the task's structure, not the model's, which is why GAIA found its value holding or rising with capability. On the note's own harness/scaffolding line, decomposition belongs closer to harness than to scaffolding, and re-filing it there is the cleanest fix to the framework.

For the KwikTrip conversation, this changes what we tell them to harden. The call on the document-preprocessing pipeline stands — that is genuine capability scaffolding against weak multimodal diagram retrieval, and it is a wasting asset. But the note as written implies the orchestration around Sprocket is disposable too, and we should not say that onsite. The planner/executor split and any artifact-level phase verification are the parts to harden, and they are unusually easy to defend to a client because their benefit is measurable in operational units — tool calls per resolved ticket, percentage of failed trajectories that self-recover — rather than in eval points that a model release can erase. What we should tell them explicitly not to harden: exemplar libraries of "good" answers, and any self-review pass where the reviewing model is the same model that produced the work.

The mirror is uncomfortable. Our own critic stack is largely the fragile kind: station-critic and the verify-* family are model-checking-model. The differentiated-rubric work is what rescues them, because a rubric is an external anchor — but the ensembling itself buys less every release, on exactly the self-consistency curve. Meanwhile the durable kind is missing where we know it should be: the Opus-5 breakage note already flagged that our rewritten deep-research skill asserts "<N> briefs filed overnight" with nothing checking that a brief file exists. That is the one verification category with no measured decay, and we do not have it. The honest read is that our recent skills-retune spent effort on the category that depreciates and skipped the category that doesn't.

Calibration on the evidence itself, plainly. No published study measures the same scaffold across successive generations of one model lineage. GAIA is cross-sectional — five models at one moment, capability tiers standing in for generation jumps — and the GPT-5.5 outlier is direct evidence that the stand-in is leaky. The few-shot ablation runs on open-weight models topping out at Qwen2.5-72B, not on frontier closed models; the frontier corroboration is Anthropic's product decision, which is first-party but not a controlled experiment. METR's number is 2024-era and its non-significant 8pp is a weak instrument. The self-consistency result is the cleanest of the four and it is also the narrowest. So: high confidence on the few-shot and self-consistency verdicts, where mechanism and measurement agree. Moderate confidence on decomposition — one good study, contradicting our prior, on a benchmark that rewards multi-step tool use. Low confidence on any specific decay rate for anything. Nobody should quote a half-life.

Why this is in the vault

It resolves the first of the two open questions in [[2026-08-01-scaffolding-vs-harness-the-erosion-axis]] and corrects that note's decomposition filing, which is the note we are drawing on for the KwikTrip Sprocket harden-vs-disposable conversation — specifically the recommendation on whether the client should treat orchestration structure as a durable asset alongside the knowledge base. It also names a concrete gap in RDCO's own skill corpus: artifact-existence verification is the one category with no measured decay, and deep-research currently reports brief counts without it.

Open follow-ups

Related

Sources

Vault

Web