Extraction faithfulness vs source length: the curve exists in pieces, the crossover has never been measured, and the best estimate puts it far above 5KB
The question
Verbatim: "Is there published measurement of summarization/extraction FAITHFULNESS as a function of source-artifact length — does a subagent's handback get less faithful as the source grows, and at what size does that fidelity loss exceed the context-rot loss it was meant to avoid?"
Context: follow-up #2 from [[2026-09-27-context-rot-mechanism]]. AGENTS.md hard rule #4 sends anything over ~5KB to a subagent that returns only the extract. That brief priced the context-rot side of the trade and flagged the other side, lossy extraction with no error signal, as unpriced.
Short answer. Yes, partial curves are published; no, nobody has measured the crossover. Faithfulness is roughly flat from 1K to 8K tokens, trends worse with length inside a 32K-token benchmark, and fabrication roughly triples from 32K to 128K and passes 10% for every tested model at 200K. The crossover figure below is Ray's inference from two separately measured curves, not a published result.
Provenance labels used below: (opened) = primary page fetched this run; (snippet) = search-result excerpt only; (memory) = recalled from model training, not fetched or verified this run.
What we already know (from the vault)
- [[2026-09-27-context-rot-mechanism]] puts the measured onset of context-length degradation at 16K-32K tokens, calls the 5KB (~1,300 token) trigger one to two orders of magnitude below the evidence, and proposes a floor of roughly 8-10K tokens on token-cost grounds alone. It named extraction fidelity as the first unpriced cost and had no data on it.
- [[feedback_workflow_agent_output_integrity]] records seven observed handback failure modes. The one that matters most here is mode 5: a station hallucinated a regulation's content, stamped it "verified against primary text: yes", and two independent critics passed it because both checked against the same tainted intermediate. Rule adopted: one gate per chain must hit the primary source.
- [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] and [[2026-10-20-non-arxiv-citation-exposure-sample]] are RDCO's own measured handback fidelity: roughly 1 in 5 cited claims defective (2 of 10, then 7 of 36), with zero failures in 19 claims from primary-tier sources and 7 of 17 from secondary sources. The defect shape is "a good citation with passengers": a verified figure followed by unverified ones behind the same link. Neither audit sliced by source length.
- [[2026-10-20-critic-layer-review-time-defect-escape]] prices the verification pass: median 254 seconds per critic run, and the only slice with ground truth showed 8 of 9 agent ACCEPTs overturned by the founder. A second-LLM gate is neither cheap nor reliable.
- [[2026-05-19-verification-as-independent-worker-pattern]] is the existing SOP for independent verification; it does not distinguish mechanical checks (string match) from judgment checks (critic read).
What the web says
- Vectara's refreshed hallucination leaderboard (Nov 19, 2025) is the only frontier-model summarization benchmark found that bins by source length, and the direction is clear. The new set has 7,700+ articles up to 32K tokens. The authors split articles into 10 length bins and report "the hallucination rate for longer articles is higher." Rates on the new set: best model 3.3% (Gemini 2.5 Flash Lite); Claude Sonnet 4.5, GPT-5, Grok-4 and DeepSeek-R1 all above 10%; Gemini 3 Pro 13.6%. Bin-level numbers are only in a figure, not in the text, so the slope is unquantified here. The authors attribute the jump from the old set to the articles being "longer and more challenging" jointly; length is not isolated from complexity. (opened) (vectara.com)
- Fabrication under long context roughly triples from 32K to 128K and exceeds 10% for every model at 200K. Roig, "How Much Do LLMs Hallucinate in Document Q&A Scenarios?" (arXiv 2603.08274, Kamiwaza AI, preprint, not peer reviewed): 35 open-weight models, document-grounded extraction. Fabrication is measured on probes asking for entities or fields that are not in the corpus, which is exactly the "confident fabrication" failure. Best model: 1.19% at 32K, 3.19% at 128K; best case at 200K is 10.25%. Top-tier models sit at 5-7% at 32K and the median model near 25%. Caveats: no closed frontier models, one evaluation framework, single vendor author. A model with 90% grounding accuracy still fabricated 49.5% on probes, so extraction accuracy and fabrication resistance are separate properties. (opened) (arxiv.org/abs/2603.08274)
- Below ~8K tokens, source length has no measurable effect on faithfulness. Yang et al., "Hallucinate at the Last in Long Response Generation" (arXiv 2505.15291, preprint): Llama 3.1 8B on Wikipedia, faithfulness by input length 4K to 8K reads 82.9, 86.6, 85.9, 83.9, 86.8. That is noise around a constant. A search excerpt gives the 1K-3K values as 83.3, 86.7, 78.9 (snippet). Small open model, 20 documents per dataset. (opened) (arxiv.org/abs/2505.15291)
- The same paper finds the stronger effect is the length of the handback, not the source. Faithfulness falls toward the end of long summaries: human evaluation by output fifth reads 0.90, 0.87, 0.84, 0.84, 0.70. GPT-4o mini shows the same shape (0.83 falling to 0.76). Chunk-and-merge summarization (the BooookScore method) removed the positional decline (sensitivity 10.3 to 0.0) and raised overall faithfulness from 90.33 to 94.33. The paper reports no cost figures for any mitigation. (opened)
- At book length, verification is itself unreliable. FABLES (Kim et al., COLM 2024, arXiv 2404.01261): 3,158 human-annotated claims from summaries of 26 books averaging ~121K tokens. Claude 3 Opus was the most faithful summarizer. LLM auto-raters could not reliably flag unfaithful claims even when given the full book, and most unfaithful claims required reasoning across the narrative to catch. The paper reports no faithfulness-vs-length curve. (snippet) Recalled but not re-verified: roughly 90% of Claude 3 Opus claims rated faithful against ~70-78% for other models, and the paper found omission and over-weighting of the book's ending to be common content-selection errors. (memory)
- Referenced but not fetched or verified this run (memory): Laban et al., "Summary of a Haystack" (EMNLP 2024, arXiv 2407.01370), where long-context models summarizing ~100K-token haystacks with citations scored far below human on joint coverage-plus-citation; Ravaut et al., "On Context Utilization in Summarization with LLMs" (ACL 2024, arXiv 2310.10570), a U-shaped input-position bias favoring the start and end of the source; Karpinska et al., NoCha (EMNLP 2024, arXiv 2406.16264), claim verification over whole books near chance for most models. Treat all three as leads, not evidence.
Convergences and contradictions
- Convergence: web and vault agree that the dangerous property of a handback is not its error rate but its lack of error signal. FABLES' auto-rater failure and memory mode 5 (critics inheriting a tainted intermediate) are the same finding from two directions: a second model reading the summary, or even the source, does not reliably catch the unfaithful claim.
- Contradiction with the framing of the question: the question assumes fidelity loss and context-rot loss are opposing curves that cross once. They are not independent. The subagent pays the same single-pass length penalty the parent would pay reading the artifact itself, so past the context-rot onset both curves rise together. The published evidence supports a crossover at the low end only (where delegation adds a fidelity tax and saves nothing), plus a separate high end where single-pass delegation still beats a direct read but stops being trustworthy.
- Contradiction between benchmark and vault rates: benchmark fabrication for good models at RDCO's typical artifact sizes is in the low single digits to low teens. RDCO's measured defect rate is about 1 in 5 claims. The gap is explained by source tier and "passenger" claims, per [[2026-10-20-non-arxiv-citation-exposure-sample]], not by length. Length is not the main driver of the fidelity loss RDCO actually experiences.
Synthesis for RDCO
What is established. Three things, at different confidence. First, a delegation fidelity tax exists at every length: even on short sources a summarizing model produces some unfaithful claims, and that rate does not move between 1K and 8K tokens (one small-model study). Second, the tax rises with length: directionally inside 32K for frontier models (Vectara, unquantified, confounded with complexity), and by about 3x between 32K and 128K with every tested model past 10% at 200K (one preprint, open-weight models only). Third, the handback's own length matters independently: claims late in a long extract are measurably less faithful than claims early in it. None of these sources tested a Claude 5-generation model on an extraction task with a length sweep, so every number here is a transfer, not a measurement of our setup.
The crossover, labeled as Ray's inference. Nobody has published it. The estimate rests on four assumptions: (a) the context-rot onset is 16-32K tokens of accumulated context, per the 09-27 brief's sources; (b) the delegation tax below 8K tokens is roughly constant and nonzero, at a few percent of claims for a strong model; (c) the subagent's length-driven error above the onset is error the parent would also incur on a direct read, so it does not count against delegation; (d) fabrication rates from open-weight Q&A probes transfer in shape, not level, to frontier extraction. Under those assumptions there are two thresholds, not one. Lower threshold: in a fresh or lightly loaded session, direct reading dominates until the read would push live context into the 16-32K onset zone, which puts the break-even for a single artifact at roughly 15-30K tokens (about 60-120KB of prose). Below that, delegation buys no context-rot relief and costs a fidelity tax, the round-trip tokens, and the source. In a session already past ~30K tokens the break-even drops toward the 8-10K token-cost floor the 09-27 brief proposed. Upper threshold: somewhere around 100K tokens, single-pass delegation is still better than a direct read but the handback should no longer be treated as trustworthy without chunking and quote grounding, given the ~3x fabrication increase by 128K. Confidence in the lower threshold's order of magnitude is moderate; in the specific numbers, low. What would change it: Vectara's bin-level data, or a length sweep on a current Claude model.
What this implies for hard rule #4, as a recommendation for the founder. The 5KB trigger is fidelity-negative as well as token-negative: at 1,300 tokens the rule trades a context cost that no study can detect for an extraction tax that every study can. The evidence supports the 09-27 brief's direction and suggests its 8-10K floor is, if anything, conservative for fresh sessions. More useful than moving the number is changing what a handback must contain, because the vault's own audits show length is not where RDCO's fidelity loss comes from. Three low-cost practices follow from the evidence. (1) Quote-grounded handbacks with locators. Every load-bearing claim carries a verbatim quote and a path or line reference. A quote can be string-matched against the source with grep in milliseconds, which converts an unverifiable claim into a mechanically verifiable one and is the only verification method here that does not inherit long-context weakness. Cost: longer handbacks, and it only works while the source is still on disk or re-fetchable. (2) Short handbacks, critical claims first. The end-of-output faithfulness drop (0.90 to 0.70) argues for capping extract length and ordering by importance. (3) Chunk above ~100K tokens. Chunk-and-merge is the one mitigation with a measured faithfulness gain (90.3 to 94.3) and it removes the positional decline; it costs multiple calls and loses cross-chunk reasoning.
What not to build. A second-LLM "faithfulness critic" over the source is the intuitive fix and the evidence is against it: FABLES found auto-raters unreliable even with the full text, the vault's critic layer costs about four minutes a run with a poor measured overturn record, and memory mode 5 shows critics passing a fabricated claim twice. The pointer-plus-content return (extract in the message, source path alongside) is the cheap structural protection. It differs from the pointer-not-content failure mode in the memory note, where the pointer replaced the content.
Why this is in the vault
It is the second half of the evidence base for the pending founder decision on amending AGENTS.md hard rule #4: the 09-27 brief priced what delegation saves, this one prices what it costs, and together they bound the delegation threshold. It also gives the deep-research and process-newsletter skills a measured reason to require quote-plus-locator handbacks instead of adding another critic gate.
Open follow-ups
- On RDCO's own corpus, does handback defect rate vary with source length at all once source tier is controlled? The two citation audits have the claims and the sources; re-slicing the existing 46 audited claims by source token count would answer it without new sampling, and would test whether length matters in practice or only on benchmarks.
- What does a length sweep look like for a current Claude model on quote-grounded extraction specifically (1K to 200K tokens, quote string-match rate and fabricated-quote rate as the metrics)? No published study covers frontier closed models with this design, and it is the measurement that would replace the inferred crossover with a real one.
- Do Vectara's bin-level numbers (Figure 11 of the Nov 2025 post, or the public leaderboard repository) separate length from complexity, and what is the slope for Claude models between the shortest and longest bins?
Related
- [[2026-09-27-context-rot-mechanism]] - the context-rot side of the trade-off; this brief answers its follow-up #2
- [[2026-11-12-accumulated-agentic-context-degradation-curve]] - sibling brief written the same night on accumulated multi-turn context (follow-up #1)
- [[feedback_workflow_agent_output_integrity]] - the seven handback failure modes; mode 5 is the tainted-intermediate case cited above
- [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] - RDCO's first measured handback defect rate (2 of 10)
- [[2026-10-20-non-arxiv-citation-exposure-sample]] - source tier, not domain or length, predicts defects (0 of 19 vs 7 of 17)
- [[2026-10-20-critic-layer-review-time-defect-escape]] - cost and reliability of the second-LLM verification pass
- [[2026-05-19-verification-as-independent-worker-pattern]] - the verification SOP that quote string-matching would complement
- [[context-rot]] - vault concept page
Sources
Vault
06-reference/research/2026-09-27-context-rot-mechanism.md06-reference/research/2026-10-08-finding-level-citation-audit-arxiv-gloss.md(synthesis section only)06-reference/research/2026-10-20-non-arxiv-citation-exposure-sample.md(synthesis section only)06-reference/research/2026-10-20-critic-layer-review-time-defect-escape.md(synthesis section only)02-sops/2026-05-19-verification-as-independent-worker-pattern.md(search snippet only)06-reference/concepts/context-rot.md(via the 09-27 brief; not re-read)~/.claude/projects/-Users-ray/memory/feedback_workflow_agent_output_integrity.md
Web, opened this run
- Vectara, "Introducing the Next Generation of Vectara's Hallucination Leaderboard," Nov 19, 2025 - https://www.vectara.com/blog/introducing-the-next-generation-of-vectaras-hallucination-leaderboard (read through the fetch tool's summarizer; bin-level figures not visible)
- Roig, "How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study," arXiv:2603.08274 (preprint, Kamiwaza AI) - https://arxiv.org/html/2603.08274
- Yang et al., "Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarization," arXiv:2505.15291v3 (preprint) - https://arxiv.org/html/2505.15291v3 (first 100K of 132K characters read)
Web, search snippet only
- Kim et al., "FABLES: Evaluating faithfulness and content selection in book-length summarization," COLM 2024, arXiv:2404.01261 - https://arxiv.org/abs/2404.01261
Referenced but NOT fetched or verified this run (model memory)
- Laban et al., "Summary of a Haystack," EMNLP 2024, arXiv:2407.01370
- Ravaut et al., "On Context Utilization in Summarization with LLMs," ACL 2024, arXiv:2310.10570
- Karpinska et al., "One Thousand and One Pairs (NoCha)," EMNLP 2024, arXiv:2406.16264
- Chang et al., "BooookScore," ICLR 2024, arXiv:2310.00785 (the chunk-and-merge method tested in arXiv:2505.15291)
- FABLES per-model faithfulness percentages and content-selection findings