Vault-wide arXiv citation audit: the "Bai et al." defect was a singleton, and the real exposure is elsewhere

The question

"Vault-wide citation audit: where else has a deep-research run attributed a real paper to the wrong authors? Extract every arXiv-ID-plus-claimed-author pair in the vault and check each against arXiv metadata."

Context: the 2026-07-19 verification run proved this failure mode is real and live in the vault. "Constitution or Collapse?" (arXiv:2504.04918) was carried for two months across three docs as "Bai et al." when the paper has a single author (Xue Zhang), and the finding itself was mischaracterized. A load-bearing sizing decision rested on it. This audit asks whether that was a one-off or the visible instance of a defect class.

What we already know (from the vault)

• [[2026-07-19-constitution-or-collapse-citation-verification]] established the two-part failure: a fused citation (the citing paper's title welded to the cited paper's authorship — "Bai et al." is the original Anthropic CAI paper, arXiv:2212.08073, which Zhang replicates) plus a mischaracterized finding (Zhang reports degenerate token repetition in the policy model from emoji-noise in SFT data, not critic mode-collapse from low label volume).
• The defect propagated: one bad pair appeared in [[2026-05-19-cai-critic-graduation-per-axis-threshold]], [[2026-05-22-reward-hacking-patterns-llm-critic-systems]], and [[2026-07-16-constitutional-ai-v2-reward-shaping]] before anyone checked the primary source.
• [[feedback_workflow_agent_output_integrity]] already names the governing rule: one gate per chain MUST hit the primary source. FALSE "verified against primary text" stamps are a catalogued factory failure mode.
• /verify-vault-write checks citation presence (rubric item 5: "claims with no inline wiki-link or footnote"), never citation correctness. /deep-research SKILL.md contains no primary-source verification step at all. The gate that would have caught this does not exist.

What the web says

This run queried the authoritative arXiv API (export.arxiv.org/api/query) rather than the open web. Method: grep the full vault (3,505 markdown files) for arXiv ID patterns, resolve every unique ID to canonical title + full author list, then diff the claimed attribution in a ±4-line context window against ground truth.

• 60 unique arXiv IDs, 146 occurrences, 39 vault files. All 60 resolved. Zero hallucinated or non-existent IDs — the vault has never invented an arXiv number.
• Zero live wrong-author attributions. Every currently-standing author claim matches the real byline.
• Zero live wrong-title claims. The one apparent mismatch — 2507.06419 cited as "Reward Models Can Improve Themselves (RGFMD)" against a current title of "Teach a Reward Model to Correct Itself…" — is not a defect: the arXiv v1 title was exactly "Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling." The authors retitled at a later version. The vault is right; arXiv moved.
• The known defect is fully remediated. All four documents carrying 2504.04918 now show explicit in-place strikethrough plus [CORRECTED 2026-07-19] / [STRUCK 2026-07-19] markers naming Zhang, Xue and linking the verification note. No residual live claim survives anywhere in the vault.
• The structural finding: 44 of 60 IDs (73%) make no author claim at all. They are cited as a bare URL plus a descriptive gloss ("arXiv 2509.21051, 10 LLMs, up to 10 instructions"). Only 16 IDs carry an explicit named attribution.
• One near-miss worth naming: [[2026-05-26-skillopt-self-evolving-agent-skills]] frontmatter reads "Yang, Gong, Huang et al." — a three-name list where the first two are correct and the third is the third author, not a second lead. Technically accurate, stylistically unusual, not a defect.

Defect table

Base rate

One excluded pattern: arxiv-2403.12345-attention-is-all-you-need in [[2026-04-30-bookshelf-source-material-architecture-gap]] is an illustrative directory-naming example, not a citation. (It is also wrong on its own terms — 2403.12345 is not the Transformer paper — but it claims nothing.)

Convergences and contradictions

• Convergence: the backlog's premise that this is a defect class is half right. The mechanism is real and the remediation was correct. But the class has exactly one member.
• Contradiction — the important one: the audit's clean result is not evidence of a healthy citation discipline. It is an artifact of citation style. 73% of vault arXiv references never name an author, so they cannot be wrong about one. The vault is protected by omission, not by verification. Nothing in the harness would have stopped a second "Bai et al." — the first one survived two months and three documents until a human-triggered spot-check found it.
• Contradiction — the uncomfortable one: the 2026-07-19 incident had two errors, and this audit can only detect one of them. Wrong author is mechanically checkable. Mischaracterized finding is not. Those 44 descriptive glosses ("degrades monotonically as instructions are added," "0.83–0.92 continuity across Claude/GPT-4/Gemini") are exactly the shape of the error that actually moved the sizing decision, and every one of them is unverified.

Synthesis for RDCO

The headline is good news framed the wrong way. A 0% live wrong-author rate across 60 IDs says the vault is not riddled with fabricated bylines, and it retires the worry that the Constitution-or-Collapse incident was the tip of something. It also confirms the July 19 remediation was thorough rather than cosmetic — all four carrying documents show in-place corrections with strikethrough and dated markers, which is the right pattern (correct in place, leave the error visible, link the verification) and should be the standard.

But the audit measured the cheap thing because the cheap thing is measurable. Author attribution is one API call away from ground truth; findings are not. The July 19 post-mortem was explicit that the finding error — not the byline error — is what carried the sizing decision. A wrong author is embarrassing. A wrong finding is load-bearing. And the vault currently has 44 arXiv references whose entire epistemic content is a one-line gloss written by an agent that read a search-result snippet, with no gate anywhere in /deep-research or /verify-vault-write that requires touching the paper.

That reframes the harness gap. The fix is not "add an arXiv author-check to deep-research" — this audit just demonstrated that check would have caught one defect in three months at a cost of running it against every brief. The fix is a claim-tier discipline: when a brief attributes a specific quantitative result to a specific paper, and that result is going to be cited in a decision, one gate in the chain must open the primary source. That is exactly the rule already sitting in [[feedback_workflow_agent_output_integrity]] — this audit is the first vault-wide measurement of how widely it is violated. The answer is 44 times.

There is a cheap structural mitigation available immediately, independent of any new gate: the 4× propagation number. One bad pair reached four documents because downstream briefs cited the vault rather than the source, and the vault looked authoritative because it was internal. Citation-through-the-vault is how a single unverified claim becomes four corroborating ones and then reads as convergence. Marking secondhand vault citations as secondhand — the Referenced but NOT fetched or verified this run block that [[2026-07-16-constitutional-ai-v2-reward-shaping]] already uses — costs nothing and breaks the amplification. That block is the single best pattern this audit found in the vault, and it is currently used in exactly one document.

Remediation recommendation (report-only; no vault files were edited this run): (1) adopt the Referenced but NOT fetched or verified this run block as a required /deep-research output section; (2) add a verified: primary-source | secondhand | gloss-only marker to arXiv citations that carry a quantitative claim; (3) do not add a blanket arXiv author-check — the measured yield does not justify it; (4) re-run this mechanical audit quarterly, since it is a ~10-minute script and the base rate is the actual signal.

Why this is in the vault

This sets the empirical baseline for citation integrity as a harness-quality metric: 0% live author-error, 1.7% historical per-ID defect rate, 4× propagation. That baseline is the measurement /improve needs to judge whether any future verification gate added to /deep-research or /verify-vault-write is earning its cost, and it directly answers the open question left by [[2026-07-19-constitution-or-collapse-citation-verification]] about whether that incident warranted a systemic fix.

Open follow-ups

• The 44 unverified descriptive glosses are the real exposure. A finding-level audit — sample 10 glosses, read the primary source, measure the mischaracterization rate — would produce the number that actually matters. This audit cannot produce it.
• Are non-arXiv citations worse? The vault cites Medium, Substack, vendor blogs, and SEC filings far more often than arXiv, with no DOI-equivalent to check against. arXiv may be the best-behaved corner precisely because it is checkable.
• Does the 4× propagation factor hold generally? Measuring how often a vault claim is cited by a later vault doc without re-verification would size the amplification risk across the whole knowledge base, not just citations.
• Which agent runs produced the 44 gloss-only citations, and did any of them stamp "verified"? A false verification stamp is a strictly worse defect than an honest gloss.

Related

• [[2026-07-19-constitution-or-collapse-citation-verification]]
• [[2026-05-19-cai-critic-graduation-per-axis-threshold]]
• [[2026-05-22-reward-hacking-patterns-llm-critic-systems]]
• [[2026-07-16-constitutional-ai-v2-reward-shaping]]
• [[2026-07-03-verify-skills-pass-disagreement-audit]]
• [[2026-05-26-skillopt-self-evolving-agent-skills]]
• [[2026-04-30-bookshelf-source-material-architecture-gap]]
• [[feedback_workflow_agent_output_integrity]]

Sources

Vault:

• ~/rdco-vault/06-reference/research/2026-07-19-constitution-or-collapse-citation-verification.md — the originating incident; both error types documented
• ~/rdco-vault/06-reference/research/2026-05-19-cai-critic-graduation-per-axis-threshold.md — carrying doc, remediated in place
• ~/rdco-vault/06-reference/research/2026-05-22-reward-hacking-patterns-llm-critic-systems.md — carrying doc, remediated in place
• ~/rdco-vault/06-reference/research/2026-07-16-constitutional-ai-v2-reward-shaping.md — carrying doc; source of the Referenced but NOT fetched pattern
• ~/rdco-vault/06-reference/research/2026-07-03-verify-skills-pass-disagreement-audit.md — densest arXiv-citing doc (7 IDs)
• ~/rdco-vault/06-reference/2026-05-26-skillopt-self-evolving-agent-skills.md — attribution near-miss
• ~/rdco-vault/06-reference/2026-04-30-bookshelf-source-material-architecture-gap.md — excluded placeholder ID

Harness:

• ~/.claude/skills/verify-vault-write/SKILL.md — rubric item 5 checks citation presence, not correctness
• ~/.claude/skills/deep-research/SKILL.md — no primary-source verification step

Authoritative metadata:

• arXiv API, https://export.arxiv.org/api/query?id_list=<ids> — 60 IDs resolved in 3 batched calls, 2026-07-27. Note: http:// returns a 301; HTTPS required.
• arXiv API v1-pinned query for 2507.06419 — confirmed the title changed between versions, exonerating the vault's citation.