06-reference/research

extraction faithfulness vs source length

2026-11-12·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
extraction-faithfulnesscontext-rotsubagentsagents-mdagent-output-integrity

Extraction faithfulness vs source length: the curve exists in pieces, the crossover has never been measured, and the best estimate puts it far above 5KB

The question

Verbatim: "Is there published measurement of summarization/extraction FAITHFULNESS as a function of source-artifact length — does a subagent's handback get less faithful as the source grows, and at what size does that fidelity loss exceed the context-rot loss it was meant to avoid?"

Context: follow-up #2 from [[2026-09-27-context-rot-mechanism]]. AGENTS.md hard rule #4 sends anything over ~5KB to a subagent that returns only the extract. That brief priced the context-rot side of the trade and flagged the other side, lossy extraction with no error signal, as unpriced.

Short answer. Yes, partial curves are published; no, nobody has measured the crossover. Faithfulness is roughly flat from 1K to 8K tokens, trends worse with length inside a 32K-token benchmark, and fabrication roughly triples from 32K to 128K and passes 10% for every tested model at 200K. The crossover figure below is Ray's inference from two separately measured curves, not a published result.

Provenance labels used below: (opened) = primary page fetched this run; (snippet) = search-result excerpt only; (memory) = recalled from model training, not fetched or verified this run.

What we already know (from the vault)

What the web says

Convergences and contradictions

Synthesis for RDCO

What is established. Three things, at different confidence. First, a delegation fidelity tax exists at every length: even on short sources a summarizing model produces some unfaithful claims, and that rate does not move between 1K and 8K tokens (one small-model study). Second, the tax rises with length: directionally inside 32K for frontier models (Vectara, unquantified, confounded with complexity), and by about 3x between 32K and 128K with every tested model past 10% at 200K (one preprint, open-weight models only). Third, the handback's own length matters independently: claims late in a long extract are measurably less faithful than claims early in it. None of these sources tested a Claude 5-generation model on an extraction task with a length sweep, so every number here is a transfer, not a measurement of our setup.

The crossover, labeled as Ray's inference. Nobody has published it. The estimate rests on four assumptions: (a) the context-rot onset is 16-32K tokens of accumulated context, per the 09-27 brief's sources; (b) the delegation tax below 8K tokens is roughly constant and nonzero, at a few percent of claims for a strong model; (c) the subagent's length-driven error above the onset is error the parent would also incur on a direct read, so it does not count against delegation; (d) fabrication rates from open-weight Q&A probes transfer in shape, not level, to frontier extraction. Under those assumptions there are two thresholds, not one. Lower threshold: in a fresh or lightly loaded session, direct reading dominates until the read would push live context into the 16-32K onset zone, which puts the break-even for a single artifact at roughly 15-30K tokens (about 60-120KB of prose). Below that, delegation buys no context-rot relief and costs a fidelity tax, the round-trip tokens, and the source. In a session already past ~30K tokens the break-even drops toward the 8-10K token-cost floor the 09-27 brief proposed. Upper threshold: somewhere around 100K tokens, single-pass delegation is still better than a direct read but the handback should no longer be treated as trustworthy without chunking and quote grounding, given the ~3x fabrication increase by 128K. Confidence in the lower threshold's order of magnitude is moderate; in the specific numbers, low. What would change it: Vectara's bin-level data, or a length sweep on a current Claude model.

What this implies for hard rule #4, as a recommendation for the founder. The 5KB trigger is fidelity-negative as well as token-negative: at 1,300 tokens the rule trades a context cost that no study can detect for an extraction tax that every study can. The evidence supports the 09-27 brief's direction and suggests its 8-10K floor is, if anything, conservative for fresh sessions. More useful than moving the number is changing what a handback must contain, because the vault's own audits show length is not where RDCO's fidelity loss comes from. Three low-cost practices follow from the evidence. (1) Quote-grounded handbacks with locators. Every load-bearing claim carries a verbatim quote and a path or line reference. A quote can be string-matched against the source with grep in milliseconds, which converts an unverifiable claim into a mechanically verifiable one and is the only verification method here that does not inherit long-context weakness. Cost: longer handbacks, and it only works while the source is still on disk or re-fetchable. (2) Short handbacks, critical claims first. The end-of-output faithfulness drop (0.90 to 0.70) argues for capping extract length and ordering by importance. (3) Chunk above ~100K tokens. Chunk-and-merge is the one mitigation with a measured faithfulness gain (90.3 to 94.3) and it removes the positional decline; it costs multiple calls and loses cross-chunk reasoning.

What not to build. A second-LLM "faithfulness critic" over the source is the intuitive fix and the evidence is against it: FABLES found auto-raters unreliable even with the full text, the vault's critic layer costs about four minutes a run with a poor measured overturn record, and memory mode 5 shows critics passing a fabricated claim twice. The pointer-plus-content return (extract in the message, source path alongside) is the cheap structural protection. It differs from the pointer-not-content failure mode in the memory note, where the pointer replaced the content.

Why this is in the vault

It is the second half of the evidence base for the pending founder decision on amending AGENTS.md hard rule #4: the 09-27 brief priced what delegation saves, this one prices what it costs, and together they bound the delegation threshold. It also gives the deep-research and process-newsletter skills a measured reason to require quote-plus-locator handbacks instead of adding another critic gate.

Open follow-ups

Related

Sources

Vault

Web, opened this run

Web, search snippet only

Referenced but NOT fetched or verified this run (model memory)