06-reference/research

finding level citation audit arxiv gloss

2026-10-08·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
vault-integritycitation-verificationdeep-researchharness-qualityarxiv

Finding-level arXiv citation audit: 2 of 10 glosses carry a defect, and the provenance label is what separates the clean ones

The question

"Finding-level citation audit: sample 10 of the 44 gloss-only arXiv citations in the vault, read each primary source, and measure the rate at which the vault mischaracterizes what the paper actually found."

Context: [[2026-07-27-vault-arxiv-citation-audit]] found 0 live wrong-author defects across 60 arXiv IDs but noted that 44 of 60 name no author at all, so the clean result measured citation style, not verification. The 2026-07-19 incident had two halves — wrong byline and wrong finding — and the mechanical audit could only detect the first. This brief measures the second, the residual and more expensive error class.

What we already know (from the vault)

Method, and the corrections it forced on the premise

  1. grep -rnoE '(arXiv|arxiv)([ :]|\.org/(abs|pdf)/)+[0-9]{4}\.[0-9]{4,5}(v[0-9]+)?' across all vault markdown. 98 unique arXiv IDs, 253 occurrences, 58 files — not 60/146/39. The July figures are not inheritable; the arXiv-citing corpus grew ~63% in IDs over ten weeks.
  2. All 98 IDs resolved against export.arxiv.org/api/query. 98 of 98 resolve to a real paper. The vault still has never invented an arXiv number.
  3. Classified each ID as byline-bearing or gloss-only by scanning a ±4-line window around every occurrence. Under a byline regex matching et al., Kalra & Tang, Deng/Farber/Lee, (Song, Kim, Eo, Park;, and by Margaret Mitchell: 34 byline-bearing, 64 gloss-only. So the gloss-only population today is 64 of 98 (65%), not 44 — the proportion fell slightly from July's 73% while the absolute count grew.
  4. Sampled 10 by seeded shuffle (random.Random(20261008)) of the gloss-only list, taking the first 10 in draw order. No post-hoc swaps, no filtering for "interesting" cases.
  5. For each: recorded the vault's characterization, then checked the primary source. Where the abstract settled it, the abstract settled it. Where it did not, I fetched the LaTeXML HTML full text via curl and ran targeted greps rather than reading papers into context.

Two honest corrections to my own method. First, my draw used a looser byline regex than step 3's final one, so 4 of the 10 sampled turn out to carry a byline (2404.18796, 2502.18018, 2608.16003, 2608.23642). The sample is therefore a sample of finding-glosses, 6 strictly byline-free and 4 byline-bearing — which turned out to be more informative than a pure gloss sample, because the one byline-bearing defect has a perfectly correct byline. Second, judging a finding from an abstract alone is genuinely unsafe, and I have a worked example below where it would have produced a false defect.

Results — the 10 sampled

# arXiv ID Vault's characterization Primary source Verdict
1 2404.18796 (PoLL) "motivates panels as a fix for a single judge's intra-model bias, using multiple smaller models — a model-diversity argument, not a rubric-diversity one"; labelled not fetched Abstract: panel of smaller models "outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families," 7x cheaper Accurate
2 2410.02736 (Justice or Prejudice?) "Verbosity/length bias inflates preference for longer, fluent answers by 15-30 points across GPT-4/Claude/PaLM judges" Paper never evaluates PaLM — 0 hits for "palm" in v1 and v2 full text; evaluated judges are GPT-4-turbo, GPT-4o, Claude-3.5, Qwen2, LLaMA3, ChatGLM. The paper's metric is robustness rate (30 occurrences), not a point-scale preference delta Mischaracterized
3 2502.18018 (Verdict) "exists to scale judge-time compute compositionally"; explicitly notes the comparison wanted "was not extractable from the PDF" Abstract: library for "scaling judge-time compute… composition of modular reasoning units (verification, debate, aggregation)" Accurate
4 2509.14034 (ConfMAD) "Confidence-weighted voting across different-backbone LLMs improves both accuracy and calibration; calibration post-hoc (Platt scaling, temperature scaling) layered on top helps further" Abstract mentions none of this. Full text does: "we applied three calibration methods: Platt Scaling, Histogram Binning, and Temperature Scaling"; 14 ECE references; backbones are GPT-4o-mini + LLaMA-3.1; "SV+Platt achieved the best accuracy on BIGGSM (0.655)" Accurate (see caveat)
5 2605.16608 (To MRL or not to MRL) "robust to plain truncation even without MRL, except under heavy truncation — MRL's measurable edge concentrates at aggressive dimension cuts" Abstract: "unless heavily truncating embeddings (i.e. reducing their size by at least 80%), truncated embeddings of non-MRL models are competitive with, and often outperform models trained with MRL" Accurate
6 2601.14691 (Gaming the Judge) "Manipulated reasoning traces alone inflated false-positive rates up to 90%" Abstract: "manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories" Accurate (near-verbatim)
7 2608.19741 (Thinkingbox) "the strongest coding model passed 65% of single attempts but dropped to 25% success when required to perform reliably across 20 consecutive attempts" v1 abstract: "the strongest achieves 65.36% pass@1, but only 25.25% pass^20." Current (v4) abstract reports Opus 5 at 66.50 → 47.53 and Kimi-K3 at 57.37 → 17.60 Accurate as of cited version
8 2606.08529 (Scaffold Effects on GAIA) "Opus 27.9pp… the most capable Anthropic model was the most scaffold-sensitive… the shrinking-sensitivity hypothesis was contradicted"; plus tool-call and error-recovery deltas Abstract: "as much as 28 percentage points within a single model (Opus, Level 2, robust slice)"; "the pre-registered prediction that more capable models would be less scaffold-sensitive is rejected in direction"; "structured scaffolds make fewer tool calls yet recover more often" Accurate
9 2608.23642 (AI Agents Push Humans Out of the Loop) "a paper by Margaret Mitchell, Avijit Ghosh, and Samir Passi finding that extended AI-agent use measurably erodes the vigilance, critical thinking, and domain skill that human oversight depends on" Self-described position paper (2 occurrences, 0 hits for "we conduct / we measure / our experiment / our participants"). Sections are argumentative; the erosion claim is argued from Bainbridge's 1983 "irony of automation" and a secondary-literature appendix. Byline is correct Overstated
10 2608.16003 (Prior Audit-Repair Context) 2026-09-25 made no content claim ("titles only; no claim made about their contents"); 2026-09-29 then read it in full and reported 15 of 15 combinations, 2.8-11.5pp, 9-25% relative, criterion survives in 13 of 15, d' in 0 of 15, 82% hand-audit Abstract matches every one of those figures Accurate

The rate

2 of 10 sampled glosses carry a finding-level defect — one mischaracterization (#2) and one tier overstatement (#9). 8 of 10 are accurate within the bound the vault stated for them.

Split by byline status, on a sample far too small to support the split as a result: 1 of 6 strictly gloss-only citations is defective, and 1 of 4 byline-bearing citations is defective. No percentage is reported on a denominator of 10, 6, or 4, and no interval is claimed. The honest reading is "roughly one in five, with a plausible true value anywhere from one in twenty to one in three."

Methodological caveats that are load-bearing, not boilerplate

Amplification — measured, and it did not fire

Defect Documents repeating the claim Downstream docs citing the carrying doc Claim repeated downstream?
#2 (PaLM / "15-30 points") 1 — [[2026-07-03-verify-skills-pass-disagreement-audit]] 6 (incl. [[2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics]], [[2026-09-25-llm-as-judge-surface-heuristic-bias]], [[2026-09-29-rereview-critic-diff-visibility-vs-zero-context]]) No. grep -rn 'PaLM' returns 3 vault hits: this one plus two unrelated 2023/2024 notes. grep -rniE '15.?(-|to |–)30 (points|pts)' returns this one plus one unrelated memory-fab figure
#9 (position paper → "finding… measurably") 1 — [[2026-09-08-every-to-read-or-not-to-read-the-code]] 1 — [[2026-09-13-every-what-to-make-of-the-anthropic-warning]] No. That doc contains zero hits for erod/atroph/oversight/vigilan/Mitchell

Amplification = 1x on both defects. The 4x propagation measured in July is real — it happened — but it did not recur here. The honest statement is that the mechanism is untested by this sample, not absent. Both defects live in the "What the web says" bullet layer, which is the layer downstream briefs quote; neither happened to get quoted. The pre-2026-04 skill-atrophy and deskilling notes that look like propagation of #9 all predate 2608.23642 and are independent.

What the web says

Convergences and contradictions

Synthesis for RDCO

The number the July audit could not produce is 2 of 10, and the most useful thing about it is that it is unremarkable. A ~17% total / ~8% major quotation-error rate is what the medical literature has run at across 46 pooled studies and three decades of trying, and RDCO's arXiv glosses land in that band on first measurement. That kills the implicit premise behind the whole thread — that the Constitution-or-Collapse incident revealed an agent-specific sloppiness fixable by adding a gate. It did not. It revealed that a knowledge base which cites faster than it verifies has the same error rate as every other knowledge base that does the same thing, and that the July 19 incident was costly not because the rate is high but because that particular claim was load-bearing. The decision variable is not the rate. It is which claims get promoted to decision-bearing without anyone opening the source.

The in-sample signal points at a fix that is already half-built and costs nothing. The July brief's recommendation (1) — adopt the Referenced but NOT fetched or verified this run block — was never written into deep-research/SKILL.md or brief-template.md; the only change the skill's changelog records since is a 2026-08-03 grep for the ## Synthesis for RDCO header. And yet the convention spread anyway, from 1 document in July to 78 documents today, by imitation across runs. That is a convention that propagated without a harness, and in this sample every labelled citation held within its declared bound while both defects sat in unlabelled text. The cheapest available intervention is therefore not a new verification gate but making the existing convention mandatory: a required provenance line per external claim, with three values — opened, snippet-only, title-only. It does not make any claim truer. It makes the untrusted ones legible to the next agent, which is the thing that actually broke in July when four documents cited one unverified claim and the convergence looked real.

Two findings argue against building anything heavier than that. First, a correct byline is no protection: sample #9 has the right three authors, the right arXiv ID, the right 1983 antecedent, and a wrong claim tier. An author-check would have passed it. Second, a finding-level audit is not cheap and not mechanical. It cost two full-text fetches to avoid one false positive (#4) and a version-pinned re-read to avoid another (#7). Three of 98 IDs needed full text to adjudicate; at that cost this audit is a quarterly-at-most exercise, not a per-brief gate. The July brief's instinct was right for the wrong reason: don't add a blanket check, because the measured yield does not justify it — but the reason is that the checkable things are already clean (98 of 98 IDs resolve, 0 live wrong authors) and the uncheckable thing is irreducibly expensive.

Where the real exposure sits is now measured rather than hypothesised, and it is not arXiv. The vault's own non-arXiv comparator failed 4 of 12 checkable claims attributed to named indexes, reports, filings, and transcripts — roughly double the rate found here — and there is no API to audit against. The SSRN mix-up in [[2026-06-26-strategy-discovery-loop-architecture]] (PBO/CSCV cited to 2460551, which is The Deflated Sharpe Ratio; the primary is 2326253, caught 2026-09-05) is the same defect class in a corner with no metadata service. If the next unit of verification budget goes anywhere, it goes to figure-bearing non-arXiv citations in investing and positioning docs, where a wrong number has a dollar consequence and no mechanical check exists. arXiv is the corner that was easy to audit, which is exactly why it was audited twice and why both audits came back cleaner than the corpus deserves.

Why this is in the vault

This closes the measurement [[2026-07-27-vault-arxiv-citation-audit]] opened and could not perform, giving /improve the finding-level base rate (2 of 10, against a human-literature benchmark of ~17% total / ~8% major) needed to judge whether any verification gate added to /deep-research or the verify-* family earns its cost. It also redirects the next unit of citation-verification budget off arXiv and onto figure-bearing non-arXiv claims, where the vault's own comparator failed at roughly double this rate and no API check exists.

Open follow-ups

Related

Sources

Vault (audited or cited):

Harness:

Primary sources (arXiv; metadata API for all 98 IDs, full-text LaTeXML HTML via curl where noted):

Web (citation-error base rates):