Finding-level arXiv citation audit: 2 of 10 glosses carry a defect, and the provenance label is what separates the clean ones
The question
"Finding-level citation audit: sample 10 of the 44 gloss-only arXiv citations in the vault, read each primary source, and measure the rate at which the vault mischaracterizes what the paper actually found."
Context: [[2026-07-27-vault-arxiv-citation-audit]] found 0 live wrong-author defects across 60 arXiv IDs but noted that 44 of 60 name no author at all, so the clean result measured citation style, not verification. The 2026-07-19 incident had two halves — wrong byline and wrong finding — and the mechanical audit could only detect the first. This brief measures the second, the residual and more expensive error class.
What we already know (from the vault)
- [[2026-07-27-vault-arxiv-citation-audit]] measured the checkable half: 60 IDs, 0 live wrong-author defects, 1 historical defect that propagated to 4 documents. It closed by naming this brief's question as the one it could not answer.
- [[2026-07-19-constitution-or-collapse-citation-verification]] established that the wrong finding — not the wrong byline — is what moved a critic-graduation sizing decision. That is the class measured here.
- [[2026-08-01-unopened-source-citation-audit]] already ran the non-arXiv equivalent on named indexes, reports, filings, and transcripts: 13 checked, 8 verified, 4 failed, 1 unreachable. The vault therefore already had a comparator baseline, which this brief uses.
- [[feedback_workflow_agent_output_integrity]] holds the governing rule — one gate per chain MUST hit the primary source — and catalogues the false "verified against primary text" stamp as a known factory failure mode.
/verify-vault-writechecks citation presence, not correctness;deep-research/SKILL.mdandbrief-template.mdstill contain no primary-source or provenance-label step. The July brief's recommendation to require aReferenced but NOT fetchedblock was never written into either file.
Method, and the corrections it forced on the premise
grep -rnoE '(arXiv|arxiv)([ :]|\.org/(abs|pdf)/)+[0-9]{4}\.[0-9]{4,5}(v[0-9]+)?'across all vault markdown. 98 unique arXiv IDs, 253 occurrences, 58 files — not 60/146/39. The July figures are not inheritable; the arXiv-citing corpus grew ~63% in IDs over ten weeks.- All 98 IDs resolved against
export.arxiv.org/api/query. 98 of 98 resolve to a real paper. The vault still has never invented an arXiv number. - Classified each ID as byline-bearing or gloss-only by scanning a ±4-line window around every occurrence. Under a byline regex matching
et al.,Kalra & Tang,Deng/Farber/Lee,(Song, Kim, Eo, Park;, andby Margaret Mitchell: 34 byline-bearing, 64 gloss-only. So the gloss-only population today is 64 of 98 (65%), not 44 — the proportion fell slightly from July's 73% while the absolute count grew. - Sampled 10 by seeded shuffle (
random.Random(20261008)) of the gloss-only list, taking the first 10 in draw order. No post-hoc swaps, no filtering for "interesting" cases. - For each: recorded the vault's characterization, then checked the primary source. Where the abstract settled it, the abstract settled it. Where it did not, I fetched the LaTeXML HTML full text via
curland ran targeted greps rather than reading papers into context.
Two honest corrections to my own method. First, my draw used a looser byline regex than step 3's final one, so 4 of the 10 sampled turn out to carry a byline (2404.18796, 2502.18018, 2608.16003, 2608.23642). The sample is therefore a sample of finding-glosses, 6 strictly byline-free and 4 byline-bearing — which turned out to be more informative than a pure gloss sample, because the one byline-bearing defect has a perfectly correct byline. Second, judging a finding from an abstract alone is genuinely unsafe, and I have a worked example below where it would have produced a false defect.
Results — the 10 sampled
| # | arXiv ID | Vault's characterization | Primary source | Verdict |
|---|---|---|---|---|
| 1 | 2404.18796 (PoLL) | "motivates panels as a fix for a single judge's intra-model bias, using multiple smaller models — a model-diversity argument, not a rubric-diversity one"; labelled not fetched | Abstract: panel of smaller models "outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families," 7x cheaper | Accurate |
| 2 | 2410.02736 (Justice or Prejudice?) | "Verbosity/length bias inflates preference for longer, fluent answers by 15-30 points across GPT-4/Claude/PaLM judges" | Paper never evaluates PaLM — 0 hits for "palm" in v1 and v2 full text; evaluated judges are GPT-4-turbo, GPT-4o, Claude-3.5, Qwen2, LLaMA3, ChatGLM. The paper's metric is robustness rate (30 occurrences), not a point-scale preference delta | Mischaracterized |
| 3 | 2502.18018 (Verdict) | "exists to scale judge-time compute compositionally"; explicitly notes the comparison wanted "was not extractable from the PDF" | Abstract: library for "scaling judge-time compute… composition of modular reasoning units (verification, debate, aggregation)" | Accurate |
| 4 | 2509.14034 (ConfMAD) | "Confidence-weighted voting across different-backbone LLMs improves both accuracy and calibration; calibration post-hoc (Platt scaling, temperature scaling) layered on top helps further" | Abstract mentions none of this. Full text does: "we applied three calibration methods: Platt Scaling, Histogram Binning, and Temperature Scaling"; 14 ECE references; backbones are GPT-4o-mini + LLaMA-3.1; "SV+Platt achieved the best accuracy on BIGGSM (0.655)" | Accurate (see caveat) |
| 5 | 2605.16608 (To MRL or not to MRL) | "robust to plain truncation even without MRL, except under heavy truncation — MRL's measurable edge concentrates at aggressive dimension cuts" | Abstract: "unless heavily truncating embeddings (i.e. reducing their size by at least 80%), truncated embeddings of non-MRL models are competitive with, and often outperform models trained with MRL" | Accurate |
| 6 | 2601.14691 (Gaming the Judge) | "Manipulated reasoning traces alone inflated false-positive rates up to 90%" | Abstract: "manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories" | Accurate (near-verbatim) |
| 7 | 2608.19741 (Thinkingbox) | "the strongest coding model passed 65% of single attempts but dropped to 25% success when required to perform reliably across 20 consecutive attempts" | v1 abstract: "the strongest achieves 65.36% pass@1, but only 25.25% pass^20." Current (v4) abstract reports Opus 5 at 66.50 → 47.53 and Kimi-K3 at 57.37 → 17.60 | Accurate as of cited version |
| 8 | 2606.08529 (Scaffold Effects on GAIA) | "Opus 27.9pp… the most capable Anthropic model was the most scaffold-sensitive… the shrinking-sensitivity hypothesis was contradicted"; plus tool-call and error-recovery deltas | Abstract: "as much as 28 percentage points within a single model (Opus, Level 2, robust slice)"; "the pre-registered prediction that more capable models would be less scaffold-sensitive is rejected in direction"; "structured scaffolds make fewer tool calls yet recover more often" | Accurate |
| 9 | 2608.23642 (AI Agents Push Humans Out of the Loop) | "a paper by Margaret Mitchell, Avijit Ghosh, and Samir Passi finding that extended AI-agent use measurably erodes the vigilance, critical thinking, and domain skill that human oversight depends on" | Self-described position paper (2 occurrences, 0 hits for "we conduct / we measure / our experiment / our participants"). Sections are argumentative; the erosion claim is argued from Bainbridge's 1983 "irony of automation" and a secondary-literature appendix. Byline is correct | Overstated |
| 10 | 2608.16003 (Prior Audit-Repair Context) | 2026-09-25 made no content claim ("titles only; no claim made about their contents"); 2026-09-29 then read it in full and reported 15 of 15 combinations, 2.8-11.5pp, 9-25% relative, criterion survives in 13 of 15, d' in 0 of 15, 82% hand-audit | Abstract matches every one of those figures | Accurate |
The rate
2 of 10 sampled glosses carry a finding-level defect — one mischaracterization (#2) and one tier overstatement (#9). 8 of 10 are accurate within the bound the vault stated for them.
Split by byline status, on a sample far too small to support the split as a result: 1 of 6 strictly gloss-only citations is defective, and 1 of 4 byline-bearing citations is defective. No percentage is reported on a denominator of 10, 6, or 4, and no interval is claimed. The honest reading is "roughly one in five, with a plausible true value anywhere from one in twenty to one in three."
Methodological caveats that are load-bearing, not boilerplate
- An abstract is not enough, and I can prove it. Sample #4 (ConfMAD) looked overstated from its abstract, which says only that the paper "analyzes how confidence influences debate dynamics" and mentions no calibration machinery at all. The full text contains Platt scaling, temperature scaling, ECE, and two distinct backbones — exactly what the vault claimed. Abstract-only adjudication would have scored a false defect here. Any future run of this audit must budget full-text greps.
- Version drift cuts both ways. Sample #7 would score as a hard numeric defect against the current abstract (65→25 matches no model in v4). Against v1 — the version that existed on 2026-08-20, five days before the vault cited it — it is exact to two decimal places. The paper has four versions. The July audit was exonerated on 2507.06419 by the same mechanism in the other direction (title changed post-citation). A finding-level audit that does not pin to the version-at-time-of-citation manufactures defects.
- "Overstated" is a judgment call. Reclassifying #9 as accurate-but-loose would make the rate 1 of 10. I am keeping it as a defect because the specific word the vault added was "measurably," and a position paper measures nothing. That distinction is exactly the claim-tier error the July 19 post-mortem identified.
- The denominator is glosses, not claims. Several sampled IDs carry multiple claims across multiple bullets; I scored the citation as defective if any claim attributed to it failed.
Amplification — measured, and it did not fire
| Defect | Documents repeating the claim | Downstream docs citing the carrying doc | Claim repeated downstream? |
|---|---|---|---|
| #2 (PaLM / "15-30 points") | 1 — [[2026-07-03-verify-skills-pass-disagreement-audit]] | 6 (incl. [[2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics]], [[2026-09-25-llm-as-judge-surface-heuristic-bias]], [[2026-09-29-rereview-critic-diff-visibility-vs-zero-context]]) | No. grep -rn 'PaLM' returns 3 vault hits: this one plus two unrelated 2023/2024 notes. grep -rniE '15.?(-|to |–)30 (points|pts)' returns this one plus one unrelated memory-fab figure |
| #9 (position paper → "finding… measurably") | 1 — [[2026-09-08-every-to-read-or-not-to-read-the-code]] | 1 — [[2026-09-13-every-what-to-make-of-the-anthropic-warning]] | No. That doc contains zero hits for erod/atroph/oversight/vigilan/Mitchell |
Amplification = 1x on both defects. The 4x propagation measured in July is real — it happened — but it did not recur here. The honest statement is that the mechanism is untested by this sample, not absent. Both defects live in the "What the web says" bullet layer, which is the layer downstream briefs quote; neither happened to get quoted. The pre-2026-04 skill-atrophy and deskilling notes that look like propagation of #9 all predate 2608.23642 and are independent.
What the web says
- Peer-reviewed human literature runs at a comparable rate. The 2025 systematic review and meta-analysis of quotation accuracy in medicine pooled 46 studies and ~32,000 quotations/references: 16.9% of quotations incorrect (95% CI 14.1-20.0%), with about half major at 8.0% (6.4-10.0%), and no significant improvement over recent years (PMC12285159).
- Earlier pooled estimates are higher and definition-dependent: Jergas & Baethge (2015, 28 studies) put total quotation error at 25.4% (19.5-32.4), major at 11.9%; Wager & Middleton (66 studies, 3,836 references, 74 journals) reported a median of 20%, range 0-50% (PMC4627914).
- Mogull's 2017 recalculation argues the usual method inflates the figure by dividing errors by references rather than quotations examined, and lands at 14.5% (10.5-18.6%) — with roughly 65% of content errors major, meaning the cited source fails to support, has no relation to, or contradicts the claim (PMC5599002).
- A 2025 author survey (2,648 corresponding authors asked to evaluate a real citation of their own work) found a 16.6% quotation error rate with no significant difference across disciplines, and that only 11.3% of authors ever acted on an inaccurate citation of their work.
- The most consistent cross-study finding is not the headline rate but its composition: the dominant failure is the source not supporting the claim, not a transcription slip. That is the same shape as both defects found here.
Convergences and contradictions
- Convergence, and it is the deflating kind. 2 of 10 sits inside the band that peer-reviewed, editorially-gated, human-authored medical literature has held for thirty years (~17% total, ~8% major, no improvement over time). The vault's finding-level citation hygiene is not an agentic pathology — it is the normal rate for a literature that cites faster than it verifies. That reframes the harness question from "fix a defect" to "decide what rate is acceptable and buy it deliberately."
- Contradiction with the vault's own non-arXiv comparator. [[2026-08-01-unopened-source-citation-audit]] checked 13 claims attributed to named indexes, reports, filings, and transcripts: 8 verified, 4 failed, 1 unreachable — roughly 1 in 3 of the checkable ones. Against 2 of 10 here, that supports the July brief's hypothesis directly: arXiv is the best-behaved corner of the vault, because it is the corner with an API. The exposure is not where the July audit pointed.
- The strongest in-sample signal is provenance labelling, and it runs the right way. All four samples carrying an explicit provenance label — "not fetched, cited from search summary" (#1), "not extractable from the PDF" (#3), "titles only; no claim made about their contents" (#10), and #3's and #10's later full-text graduation — were accurate within the bound they declared. Both defects come from documents carrying no provenance label: a July 3 brief that predates the convention, and a newsletter-ingestion note where the paper is glossed secondhand from Every's gloss. On n=10 this is a signal, not a result. But it is a cheap, checkable, directionally-correct signal.
- Secondhand-of-secondhand is the shared provenance of both defects. #2 is a search-snippet gloss; #9 is a gloss of a newsletter's gloss. Neither originated from a run that opened the paper. Every citation in the sample that was opened — #4, #8, #10 — is accurate.
Synthesis for RDCO
The number the July audit could not produce is 2 of 10, and the most useful thing about it is that it is unremarkable. A ~17% total / ~8% major quotation-error rate is what the medical literature has run at across 46 pooled studies and three decades of trying, and RDCO's arXiv glosses land in that band on first measurement. That kills the implicit premise behind the whole thread — that the Constitution-or-Collapse incident revealed an agent-specific sloppiness fixable by adding a gate. It did not. It revealed that a knowledge base which cites faster than it verifies has the same error rate as every other knowledge base that does the same thing, and that the July 19 incident was costly not because the rate is high but because that particular claim was load-bearing. The decision variable is not the rate. It is which claims get promoted to decision-bearing without anyone opening the source.
The in-sample signal points at a fix that is already half-built and costs nothing. The July brief's recommendation (1) — adopt the Referenced but NOT fetched or verified this run block — was never written into deep-research/SKILL.md or brief-template.md; the only change the skill's changelog records since is a 2026-08-03 grep for the ## Synthesis for RDCO header. And yet the convention spread anyway, from 1 document in July to 78 documents today, by imitation across runs. That is a convention that propagated without a harness, and in this sample every labelled citation held within its declared bound while both defects sat in unlabelled text. The cheapest available intervention is therefore not a new verification gate but making the existing convention mandatory: a required provenance line per external claim, with three values — opened, snippet-only, title-only. It does not make any claim truer. It makes the untrusted ones legible to the next agent, which is the thing that actually broke in July when four documents cited one unverified claim and the convergence looked real.
Two findings argue against building anything heavier than that. First, a correct byline is no protection: sample #9 has the right three authors, the right arXiv ID, the right 1983 antecedent, and a wrong claim tier. An author-check would have passed it. Second, a finding-level audit is not cheap and not mechanical. It cost two full-text fetches to avoid one false positive (#4) and a version-pinned re-read to avoid another (#7). Three of 98 IDs needed full text to adjudicate; at that cost this audit is a quarterly-at-most exercise, not a per-brief gate. The July brief's instinct was right for the wrong reason: don't add a blanket check, because the measured yield does not justify it — but the reason is that the checkable things are already clean (98 of 98 IDs resolve, 0 live wrong authors) and the uncheckable thing is irreducibly expensive.
Where the real exposure sits is now measured rather than hypothesised, and it is not arXiv. The vault's own non-arXiv comparator failed 4 of 12 checkable claims attributed to named indexes, reports, filings, and transcripts — roughly double the rate found here — and there is no API to audit against. The SSRN mix-up in [[2026-06-26-strategy-discovery-loop-architecture]] (PBO/CSCV cited to 2460551, which is The Deflated Sharpe Ratio; the primary is 2326253, caught 2026-09-05) is the same defect class in a corner with no metadata service. If the next unit of verification budget goes anywhere, it goes to figure-bearing non-arXiv citations in investing and positioning docs, where a wrong number has a dollar consequence and no mechanical check exists. arXiv is the corner that was easy to audit, which is exactly why it was audited twice and why both audits came back cleaner than the corpus deserves.
Why this is in the vault
This closes the measurement [[2026-07-27-vault-arxiv-citation-audit]] opened and could not perform, giving /improve the finding-level base rate (2 of 10, against a human-literature benchmark of ~17% total / ~8% major) needed to judge whether any verification gate added to /deep-research or the verify-* family earns its cost. It also redirects the next unit of citation-verification budget off arXiv and onto figure-bearing non-arXiv claims, where the vault's own comparator failed at roughly double this rate and no API check exists.
Open follow-ups
- Make the provenance line mandatory in
brief-template.mdwith three values (opened / snippet-only / title-only), then re-run this audit on briefs written after the change and compare defect rates by label. That converts the n=10 correlation found here into a controlled before/after on the one intervention that costs nothing. - Remediate the two defects in place using the July 19 strikethrough-plus-dated-marker pattern: #2's PaLM attribution and "15-30 points" magnitude in [[2026-07-03-verify-skills-pass-disagreement-audit]], and #9's "finding… measurably" framing in [[2026-09-08-every-to-read-or-not-to-read-the-code]]. This brief is report-only; no vault files were edited this run.
- Audit the newsletter-ingestion path specifically. Both defects are secondhand-of-secondhand, and #9 originates in a
/process-newsletternote glossing Every's gloss of a paper. Measure the defect rate on arXiv IDs whose only vault appearance is inside a newsletter note, versus those cited by a/deep-researchbrief. That is a sharper cut than gloss-vs-byline. - Extend the version-pinning discipline into a check: for every arXiv ID in the vault, compare the citing document's date against the paper's version history and flag any citation of a paper that has been revised since. Sample #7 shows the vault can be right about a version arXiv no longer serves, and a future reader has no way to tell.
- Size the non-arXiv figure-citation exposure properly. [[2026-08-01-unopened-source-citation-audit]] checked 13 of 100+ candidates. The arXiv corner is now audited twice; the corner with the dollar consequences is sampled once at n=13.
- Test the amplification mechanism deliberately rather than waiting for it. Measure, across all of
06-reference/research/, how often a quantitative claim in a later brief traces to a vault doc rather than to the primary source. That sizes citation-through-the-vault as a structural property instead of as a single 4x anecdote.
Related
- [[2026-07-27-vault-arxiv-citation-audit]]
- [[2026-07-19-constitution-or-collapse-citation-verification]]
- [[2026-08-01-unopened-source-citation-audit]]
- [[2026-07-03-verify-skills-pass-disagreement-audit]]
- [[2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics]]
- [[2026-09-25-llm-as-judge-surface-heuristic-bias]]
- [[2026-09-29-rereview-critic-diff-visibility-vs-zero-context]]
- [[2026-09-08-every-to-read-or-not-to-read-the-code]]
- [[2026-09-13-every-what-to-make-of-the-anthropic-warning]]
- [[2026-08-25-every-benchmarks-dont-know-your-job]]
- [[2026-06-26-cortex-search-compression-strategies]]
- [[2026-06-16-multi-agent-ensembles-conviction-calibration]]
- [[2026-08-19-agent-scaffolding-generation-survival]]
- [[2026-06-26-strategy-discovery-loop-architecture]]
- [[2026-07-16-constitutional-ai-v2-reward-shaping]]
- [[feedback_workflow_agent_output_integrity]]
Sources
Vault (audited or cited):
~/rdco-vault/06-reference/research/2026-07-27-vault-arxiv-citation-audit.md— parent brief; source of the 60/44 figures this run corrected to 98/64~/rdco-vault/06-reference/research/2026-07-19-constitution-or-collapse-citation-verification.md— the originating two-part incident~/rdco-vault/06-reference/research/2026-08-01-unopened-source-citation-audit.md— non-arXiv comparator: 13 checked, 8 verified, 4 failed, 1 unreachable~/rdco-vault/06-reference/research/2026-07-03-verify-skills-pass-disagreement-audit.md— carries defect #2 (2410.02736)~/rdco-vault/06-reference/2026-09-08-every-to-read-or-not-to-read-the-code.md— carries defect #9 (2608.23642)~/rdco-vault/06-reference/2026-08-25-every-benchmarks-dont-know-your-job.md— sample #7 (2608.19741), accurate against v1~/rdco-vault/06-reference/research/2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics.md— samples #1, #3~/rdco-vault/06-reference/research/2026-06-16-multi-agent-ensembles-conviction-calibration.md— sample #4~/rdco-vault/06-reference/research/2026-06-26-cortex-search-compression-strategies.md— sample #5~/rdco-vault/06-reference/research/2026-08-19-agent-scaffolding-generation-survival.md— sample #8~/rdco-vault/06-reference/research/2026-09-25-llm-as-judge-surface-heuristic-bias.mdand2026-09-29-rereview-critic-diff-visibility-vs-zero-context.md— sample #10, title-only then full-text~/rdco-vault/06-reference/research/2026-06-26-strategy-discovery-loop-architecture.md— non-arXiv SSRN ID defect, remediated 2026-09-05
Harness:
~/.claude/skills/deep-research/SKILL.mdandreference/brief-template.md— still contain no primary-source or provenance-label requirement; changelog's only citation-adjacent addition since July is the 2026-08-03## Synthesis for RDCOheader grep
Primary sources (arXiv; metadata API for all 98 IDs, full-text LaTeXML HTML via curl where noted):
https://export.arxiv.org/api/query?id_list=<ids>— 98 IDs resolved in 4 batched calls, 2026-10-08https://arxiv.org/abs/2404.18796— Replacing Judges with Juries (PoLL), Verga et al.https://arxiv.org/html/2410.02736v1andv2— Justice or Prejudice?, Ye et al. Full text grepped both versions; 0 occurrences of "PaLM"https://arxiv.org/abs/2502.18018— Verdict, Kalra & Tanghttps://arxiv.org/html/2509.14034v1— Enhancing Multi-Agent Debate via Confidence Expression (ConfMAD), Lin et al. Full text; Platt/Histogram/Temperature scaling confirmedhttps://arxiv.org/abs/2605.16608— To MRL or not to MRL, Takeshita et al.https://arxiv.org/abs/2601.14691— Gaming the Judge, Khalifa et al.https://arxiv.org/html/2608.23642v1— AI Agents Push Humans Out of the Loop, Mitchell/Ghosh/Passi. Full text; self-described position paper, 0 own-measurement phraseshttps://arxiv.org/html/2608.19741v1— Thinkingbox, Li et al. v1 abstract: 65.36% pass@1 → 25.25% pass^20; 4 versions exist, current abstract differshttps://arxiv.org/abs/2606.08529— Scaffold Effects on GAIA, Staracehttps://arxiv.org/abs/2608.16003— Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency, Mazaheri & Mazaheri
Web (citation-error base rates):
- Quotation inaccuracy in medicine: systematic review and meta-analysis (2025) — 46 studies, ~32,000 quotations, 16.9% total / 8.0% major
- Quotation accuracy in medical journal articles — systematic review and meta-analysis (Jergas & Baethge, 2015) — 25.4% total, 11.9% major
- Mogull, Accuracy of cited "facts" in medical research articles (PLoS ONE 2017) — 14.5% recalculated; ~65% of content errors major