Non-arXiv citation exposure, resampled at n=36: the rate is indistinguishable from the arXiv corner, and every single failure came from a secondary source
The question
"Size the non-arXiv citation exposure properly: extend the 2026-08-01 unopened-source audit from 13 checked to a defensible sample, and measure how often a quantitative claim in a later brief traces to a vault doc rather than to the primary source."
The premise carried on the backlog item was that the non-arXiv corner fails at roughly double the arXiv rate (31% vs 20%) and is the corner with the dollar consequences, sampled only once at n=13. This brief triples that sample using random rather than thesis-weighted selection. The premise does not survive.
What we already know (from the vault)
- [[2026-08-01-unopened-source-citation-audit]] is the parent: 13 claims checked, 8 verified, 4 failed, 1 unreachable, so 4 of 12 checkable (33%). It explicitly warned against the extrapolation the backlog item then made: "this sample cannot be extrapolated to a folder-wide failure rate with confidence given its non-random, thesis-weighted selection." That warning was correct.
- [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] measured the arXiv comparator at 2 of 10 and refused to report a percentage on a denominator of 10. It also established two method rules this brief reuses: pin to the version-at-time-of-citation, and never adjudicate a finding from an abstract alone.
- [[2026-07-27-vault-arxiv-citation-audit]] found the mechanically-checkable defect (wrong byline) is rare, and named claim-tier discipline rather than an author-check as the real fix.
- The
feedback_workflow_agent_output_integritymemory holds the governing rule: one gate per chain MUST hit the primary source. It lives in agent memory rather than the vault, so it is deliberately not wikilinked here. - [[citation-version-discipline]] already names the failure shape as storing a number as a bare fact rather than fact-plus-provenance.
Method and sampling frame
Reproducing the parent's adjudication standard (does the claimed figure exist in the cited source at all), with the 2026-10-08 additions (version pinning, full-text checks, skips counted separately).
1. Population. 06-reference/research/ held 426 markdown files at enumeration time, up from 226 in August (427 once this brief lands). A quantitative-claim signature of a decimal-precision percentage, a precise dollar magnitude, or a decimal multiplier matches 1,509 lines across 219 files. Of those, 40 sit next to an arXiv reference and are excluded as already twice-audited, leaving 1,469 non-arXiv quantitative claim-lines.
2. Frame. A claim is in frame when it (a) matches the quantitative signature, (b) has no arXiv reference within plus-or-minus 3 lines, and (c) has at least one non-arXiv HTTP(S) URL within plus-or-minus 3 lines, so the claim and a candidate source are co-located and the attribution is unambiguous enough to check. That is 343 claim-citation pairs across 127 files. Removing the 14 documents already sampled in August, so this extends rather than re-checks, leaves a sampling frame of 314.
3. Draw. Seeded shuffle, random.Random(20261020), first 40 in draw order. No post-hoc swaps and no filtering for claims that looked suspicious, which is the bias that inflated the August figure. Enumeration and draw scripts are reproducible from the frame definition above.
4. Adjudication. Eight parallel workers, one per 5 claims, each resolving the authority the vault actually gives for that specific number (inline link first, then the file's Sources entry matching the named publisher), then opening it. Verdicts: VERIFIED, VERIFIED-AS-HEDGED (figure checks out and the doc's own stated confidence matches reality), FAILED with one of the parent's three shapes, SKIPPED-PAYWALL, UNREACHABLE.
Three honest corrections to the frame, found by running it.
- Co-location is a bad proxy for attribution. In the large majority of sampled claims the co-located URL was the wrong source for that number, and the vault's actual inline citation was different and correct. Four independent workers reported this unprompted. The frame is therefore a frame of checkable claims, not of claim-source pairs; resolution had to happen per claim, by hand.
- 4 of the 40 drawn fell out of frame on inspection: 2 were explicitly labeled as our own arithmetic, 1 gave a vault doc as its authority, 1 was unattributable. Adjudicated n is therefore 36.
- The denominator is claim-lines, not figures. Several sampled lines bundle four or five figures. Scoring is per line, failing the line if any figure attributed to the resolved source is absent. The resulting 19% is therefore a per-line rate and is not a per-figure rate, and the direction of the gap between them is not determined by this design: bundling pushes a per-line rate above the per-figure rate, while unchecked figures in the same bullets push the true per-figure defect count up. Both effects are present in this sample. No per-figure rate is claimed anywhere in this brief.
The rates
| Measure | Result | 95% CI (Wilson) |
|---|---|---|
| This audit, random sample | 7 of 36 = 19% | 10% - 35% |
| Parent 2026-08-01, thesis-weighted | 4 of 12 = 33% | 14% - 61% |
| Pooled August + October non-arXiv | 11 of 48 = 23% | 13% - 37% |
| Comparator, 2026-10-08 arXiv gloss | 2 of 10 = 20% | 6% - 51% |
Skipped and unreachable: zero of 36. No paywall was hit and no cited source was dead. Every source in the sample opened, several only via a workaround (NCBI E-utilities for reCAPTCHA-walled PubMed Central, Europe PMC fullTextXML, a browser User-Agent for sec.gov, Wayback for version pinning). Contrast the August sample, which hit one 410 and one hard 403.
Failure shapes, the same three the parent named, in the same rough proportions: absent-from-source 4, misattributed 2, temporal-conflation 1. No new shape appeared in a 3x larger sample, which is a meaningful negative result: the taxonomy is stable.
The premise fails. 19% (10-35%) against the arXiv corner's 20% (6-51%) is not double, and the two intervals almost entirely overlap. Nor is this audit distinguishable from its own parent: Fisher exact two-tailed on 7-of-36 against 4-of-12 gives p = 0.43. The August 31-33% was not wrong, it was a non-random sample behaving exactly as its own caveat predicted. The vault has been carrying "the non-arXiv corner is twice as bad" as a load-bearing fact for eleven weeks on evidence that never supported it.
The finding that does survive: it is a source-tier rate, not a corpus rate
Splitting the 36 by the tier of the authority the vault actually cited. Tier A is an official filing, a government or regulator publication, a peer-reviewed journal, or an organization's own first-party page about itself. Tier B is trade press, general news, aggregator sites, newsletters, and vendor content-marketing blogs.
| Tier | Failures | Rate | 95% CI |
|---|---|---|---|
| A: filing / regulator / journal / first-party | 0 of 19 | 0% | 0% - 17% |
| B: trade press / news / aggregator / marketing blog | 7 of 17 | 41% | 22% - 64% |
Fisher exact, two-tailed: p = 0.0023.
The 19 Tier A claims, enumerated so the boundary is auditable: sec.gov three times (Snowflake FY2026 8-K twice, the 2006 Pixar merger exhibits); a Virginia State Corporation Commission report; three journal records via PubMed Central and PubMed; Alphabet investor relations; Databricks newsroom; claude.com/pricing; an Anthropic news post; an Anthropic engineering blog reporting its own telemetry; two Forrester Total Economic Impact pages; a Colorado School of Mines institute explainer; TrendForce; MarketsandMarkets; haus.io publishing its own holdout results; and the United States Census Bureau page, cited correctly as a negative. Every one of the seven failures sat on Tier B.
Sensitivity on the boundary, because 2 to 6 of those 19 assignments are genuinely contestable. Vendor-commissioned research and commercial market-sizing firms are first-party about themselves but have an obvious interest, so the split was re-run with them demoted. Moving MarketsandMarkets and TrendForce to Tier B gives 0 of 17 against 7 of 19, p = 0.0084. Also moving both Forrester studies gives 0 of 15 against 7 of 21, p = 0.0273. A worst case that additionally demotes the two first-party-telemetry pages gives 0 of 13 against 7 of 23, p = 0.0343. The split survives every reassignment tested, which is the claim being made here; the exact p-value is not.
This matters more than the headline rate because it is actionable, and because of the composition of the corpus: classifying the first URL of all 343 in-frame pairs gives roughly 68% secondary, 21% primary or official, 11% vendor first-party. The exposure is concentrated in the two-thirds of citations pointed at content that was itself written to be linked.
The failure unit is the bullet, not the citation. Workers converged independently on one shape: a bullet bundles several figures behind one or two links, the explicitly-cited figures verify, and the un-cited bundled figures are the ones that break. The forward-deployed-engineer bullet's headline surge (643 to 5,330 postings, 729%) verifies verbatim while four comp figures in the same sentence appear in neither cited source. The telecom vendor-financing bullet verifies Lucent's $8.1B committed, Nortel's $3.1B and Cisco's $2.4B, then adds "a later McKinsey tally put combined exposure across nine suppliers at about $25.6B" which is in neither cited document and has no McKinsey entry in that file's Sources at all. Per-figure attribution, not per-bullet, is what would have caught both.
There are defects this design never looked at. Workers incidentally flagged defects in clauses adjacent to a verified sampled figure, which the per-line scoring rule does not reach because those figures were not attributed to the resolved source: a per-asset price derived by dividing three-year licence cost by a terminal asset count from a ramping series (roughly $53 where $149 is honest); a bullet pair that silently switches between nominal and present-value cost basis; a "fab utilization sustained 95%+" clause borrowing a market-share number; a rare-earth supply-chain partner list credited to a page naming only one of five; a "40,000+ firms applied" figure attributed to an Anthropic page containing neither number. Six such flags, found opportunistically by workers who were not looking for them rather than by any systematic pass, so no rate can be computed from them and none is claimed. What they establish is only that a non-empty class of defect sits outside this design's reach.
Amplification, measured separately
Operationalized on claim bodies only, excluding bibliography sections, by what authority sits within plus-or-minus 2 lines of a quantitative claim.
| Authority offered | Count |
|---|---|
| Vault doc only | 341 |
| External URL only | 204 |
| Both | 15 |
| Self-labeled own arithmetic | 34 |
| None co-located | 719 |
Structural amplification: 341 of 560 authority-bearing body claims, or 61%, offer a vault doc and no primary. That number is real but it is mostly the brief template working as designed, because ## What we already know (from the vault) is a required section whose whole job is to cite vault docs.
So I hand-traced a seeded random 15 of the 341. 8 carried a number of external origin. Of those 8, 6 gave the vault doc as the sole authority and 2 restated the primary provenance explicitly, one of them verbatim ("read the 2024 five-year paper's full text directly and recorded the medication figures with their basis"). No interval is claimed on a denominator of 8.
Then I traced whether those chains terminate. I followed 4 chains to the parent document. In 4 of 4, the chain did reach an external attribution one hop up, so amplification here lengthens the chain rather than breaking it. But 3 of those 4 terminated at content-marketing or aggregator content, not a primary: a $575B Korea fab figure resolving to a Motley Fool article; a course-launch case resolving to a blog the parent file itself labels "primary source for ... Aamir A/B"; and a paid-pilot conversion figure resolving to Monetizely, which in turn credits McKinsey and Forrester. That last one is the right behavior rather than a defect, because the parent credits the intermediary it actually read instead of the original it did not.
Amplification is therefore not the exposure. It routes back to Tier B. The amplification measure and the failure-rate measure are the same finding seen twice.
The correction loop is the actual broken loop
Re-checking the four defects the August audit identified, 80 days on:
- ShipSquad Solo Founder Index (23.7% to 36.3%): still live, stated flat at line 34 of [[2026-07-23-solo-operator-agent-fleet-org-shape-moat]] under a blanket trailing caveat ("These are secondary citations; I did not verify the index itself"), despite exact replacement text having been specified in August. The caveat does not say what August established, which is that the numbers are not on the cited page at all.
- Rigetti 99.5% two-qubit fidelity: still live, still inside a leaderboard headed "(current)" alongside competitors' achieved figures, still cited to the
heygotrade.comURL that was already HTTP 410 in August. - CXMT DRAM incumbent trio (33.2 / 32.6 / 25.7): still live at line 25, still credited to SemiAnalysis, which August confirmed does not contain them.
- Zep/Mem0 63.8% / 49.0% misattribution: still live, at line 31 of [[2026-07-25-agent-memory-infra-commoditization]], still in the same sentence that credits the mem0.ai "State of AI Agent Memory 2026" post, which is the exact misattribution August logged.
Four of four identified defects are still in the vault. Four citation audits have produced zero corrections. Measurement is not the constraint.
A note against this brief's own credibility, kept in deliberately: the first draft of this section reported the Zep/Mem0 defect as resolved by deletion. It was not. The grep that produced that conclusion had errored out on a shell-glob problem and returned nothing, and the empty result was read as absence rather than as a failed command. A fresh-eyes critic pass caught it with one grep. That is this brief committing, in a section about unverified numbers, the precise error class it is measuring, and it is the strongest available argument that the gate proposed below has to be mechanical rather than attentional.
What the web says
The external benchmark for this class of error is the human quotation-error literature. [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] already brought that benchmark into the vault, and this run re-opened its anchor paper independently, because that paper happened to come up as sample item S16.
- Jergas and Baethge 2015 (PeerJ 3:e1364, 28 studies) states total quotation error at 25.4%, CI 19.5-32.4, with major errors at 11.9% - retrieved this run via Europe PMC
fullTextXMLfor PMC4627914, because PubMed Central's own page is currently behind a reCAPTCHA. - The same paper's discussion confirms the 20% median attributed to Wager and Middleton.
- The re-read produced something the vault did not already have, which is why it counts as a read rather than a restatement: the descriptors our own vault attaches to the Wager and Middleton figure ("66 studies, 3,836 references, 74 journals") are not in the Jergas and Baethge paper credited for them - zero substring hits for "3,836", "66 studies", or "74 journals". Those descriptors belong to Wager and Middleton's own paper. So the vault's instance of this benchmark carries a live misattribution of exactly the bundled-passenger shape described above, and it sits inside the brief that introduced the benchmark.
- No sponsor or commercial interest attaches to either paper; both are peer-reviewed meta-research with no disclosed funding interest in the result.
- Our pooled non-arXiv rate of 23% (13-37%) and our arXiv rate of 20% (6-51%) both sit inside the human total-error band and above the human major-error rate. The definitions are not strictly commensurable: their "total" includes minor paraphrase drift we would score as verified, while our FAILED is closer to their "major."
Convergences and contradictions
- Convergence. Three audits now agree the vault does not invent sources or numbers. Zero fabricated URLs, zero dead-on-arrival citations, and no new failure shape in a 3x larger sample. The failure is always attribution, never invention.
- Contradiction, resolved against the vault. The backlog premise that non-arXiv fails at double the arXiv rate is not supported: 19% vs 20%, overlapping intervals, and indistinguishable from its own parent at p = 0.43. The driver was sampling method, not subject matter.
- Contradiction, newly opened. [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] concluded the clean arXiv result was partly an artifact of arXiv being "the corner with an API." The tier split argues something different: arXiv is clean because it is Tier A. Having an API and being a primary source are confounded in all three audits to date.
Synthesis for RDCO
The honest summary is that we have been measuring the wrong axis. Three audits sliced the corpus by subject domain, asking whether the arXiv corner is cleaner than the named-report corner. On a random sample those two corners are statistically indistinguishable, both near one in five. Slice the same 36 claims by the tier of the source instead and the picture resolves sharply: zero failures in 19 claims citing filings, regulators, journals, and first-party pages, against 7 failures in 17 claims citing trade press, newsletters, and vendor marketing blogs, at p = 0.0023. Subject domain was a proxy for source tier all along, and a lossy one. The arXiv corner keeps coming back clean because arXiv papers are primary documents, not because arXiv has an application programming interface.
That reframes the harness fix, and makes it cheaper than anything the prior three audits proposed. A verification gate applied to every numeric claim is expensive and, on this evidence, would spend most of its budget confirming filings and journal abstracts that were already right. A gate conditioned on source tier is close to free: when a figure's resolved authority is not a filing, a regulator, a journal, or the subject organization's own page, that figure needs either a primary-source hop or an explicit unverified label before it can be stated flat. Nineteen of our 36 claims would skip the gate entirely, and all seven defects would have been routed into it. That second number is in-sample and should not be read as a yield estimate: the tier boundary was drawn after seeing which claims failed, so a 100% catch rate on the claims that defined the boundary is close to tautological. What is not in-sample is the direction and size of the effect, which survives every boundary reassignment tested above. The honest claim is that tier is a strong predictor worth gating on, not that gating on it catches everything. The same rule handles the amplification result without a second mechanism, because the chains that terminate badly terminate at Tier B.
The second finding is about the unit of verification, and it cuts against how our briefs are actually written. Our defects do not look like a bad citation; they look like a good citation with passengers. A bullet opens with a figure that is verbatim correct and correctly linked, then accumulates three or four more figures from elsewhere behind that same link. The sourced figure holds, the passengers fail. This is why per-bullet review keeps passing documents that per-figure review would stop, and it is also why our measured 19% is a floor rather than an estimate: six defective clauses turned up incidentally in the immediate neighborhood of figures that verified, found by workers who were not looking for them. A per-figure attribution requirement in the brief template is a larger change than a provenance label, and it is the one the evidence actually asks for.
The third finding is the uncomfortable one and it is not a measurement finding. All four of the defects the August audit identified, including one with exact replacement text already drafted, are still sitting in the vault 80 days later. Four citation audits have produced zero corrections. Whatever the marginal value of a fifth audit, it is smaller than the value of applying the output of the first four, and this brief should not be read as an argument for more measurement. It is worth noting what the correction backlog costs in compounding terms: the misattributed benchmark descriptors found in the "What the web says" section above sit inside the 2026-10-08 audit itself, so the audit series is now generating the defect class it exists to measure, faster than it retires it. The measured exposure is roughly one in five non-arXiv claims, concentrated almost entirely in secondary sources, stable in shape across three samples, and no longer in serious doubt about its order of magnitude.
On defensibility, stated plainly: n=36 gives a 95% interval of 10-35%, which is wide enough that this brief can establish the order of magnitude and the tier split but cannot pin the rate. Resolving a 20%-versus-40% difference at 80% power needs 82 per arm. Getting a single rate to plus-or-minus 5 percentage points needs 246. Neither is worth buying. The tier split reached significance at n=36 because the effect is large, and that is the result worth acting on.
Why this is in the vault
This retires the "non-arXiv fails at double the rate" premise that [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] put into circulation and that the Research Backlog used to justify High priority, and it replaces it with a source-tier rule that /deep-research and /verify-vault-write can both implement as a conditional gate rather than a blanket check. It also puts a number on the correction backlog: four of four known defects are live and four audits have yielded zero fixes, which is the input to deciding whether /self-review should be allowed to auto-fix flagged figures rather than only flagging them.
Open follow-ups
- Does the tier effect hold out-of-sample? The boundary here was drawn after seeing which claims failed, so the 0-of-19 figure cannot be read as a gate yield. A small replication that stratifies the draw by tier before adjudicating would settle whether the gate is worth wiring in, and it is the only further measurement this brief endorses.
- Is arXiv clean because it has an application programming interface, or because it is Tier A? [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] attributed the clean arXiv result to auditability. The tier split suggests the cause is primary-document status instead. All three audits to date confound the two, and the answer changes which corner to worry about next.
- The 719 body claims with no authority co-located within plus-or-minus 2 lines are unmeasured. Most are presumably resolvable from their file's Sources section, which is how the briefs are written, but the proportion that is genuinely unsourced is unknown, and the Discord valuation case in this sample shows the category is not empty.
- Not a research question but the actionable one: four of four known defects are live. Whether
/self-reviewgets authority to apply a flagged numeric correction, rather than only flag it, is a policy decision for the founder and belongs on the board, not in this queue.
Related
- [[2026-08-01-unopened-source-citation-audit]]
- [[2026-10-08-finding-level-citation-audit-arxiv-gloss]]
- [[2026-07-27-vault-arxiv-citation-audit]]
- [[2026-08-01-solo-operator-moat-figures-fact-check]]
- [[2026-07-19-constitution-or-collapse-citation-verification]]
- [[citation-version-discipline]]
- [[2026-07-23-solo-operator-agent-fleet-org-shape-moat]]
- [[2026-07-25-agent-memory-infra-commoditization]]
- [[2026-05-24-quantum-computing-2026-milestone-thresholds]]
- [[2026-07-03-cxmt-dram-capacity-memory-oligopoly-risk]]
- [[2026-10-08-verification-ratio-senior-signoff-throughput]]
Sources
Vault (the audited corpus and the method parents):
- ~/rdco-vault/06-reference/research/2026-08-01-unopened-source-citation-audit.md
- ~/rdco-vault/06-reference/research/2026-10-08-finding-level-citation-audit-arxiv-gloss.md
- ~/rdco-vault/06-reference/research/2026-07-27-vault-arxiv-citation-audit.md
- ~/rdco-vault/06-reference/research/2026-08-01-solo-operator-moat-figures-fact-check.md
- ~/rdco-vault/06-reference/concepts/citation-version-discipline.md
- ~/rdco-vault/06-reference/research/ - all 426 files present at enumeration time; 40 claim-lines sampled across 35 distinct documents
- ~/.claude/skills/deep-research/reference/brief-template.md (output contract)
External sources opened this run (36 adjudicated claims; representative, full per-claim list held in the audit working notes):
- sec.gov EDGAR: Snowflake FY2026 Q4 earnings 8-K exhibit (CIK 1640147); Disney/Pixar 2006 merger agreement Ex-2.1, Ex-10.1, Ex-99.1 (CIK 1001039)
- rga.lis.virginia.gov/Published/2024/RD712/PDF - Virginia Bureau of Insurance mental-health parity report
- Jergas H, Baethge C. 2015, PeerJ 3:e1364 (PMC4627914); Wilding et al. 2022, Diabetes Obes Metab (PMC9542252); Cornford E et al. 2021, European Radiology, PMID 33630161 - all via NCBI E-utilities / Europe PMC
- abc.xyz investor relations, 2026-06-01 equity raise announcement; databricks.com newsroom Feb 2026 run-rate release; claude.com/pricing; anthropic.com/news/claude-partner-network; claude.com/blog/auto-mode-default-in-claude-code
- tei.forrester.com Augury Machine and Process Health Total Economic Impact study - sponsor flag: commissioned by Augury, the vendor whose economics it reports
- marketsandmarkets.com post-quantum cryptography market report page - sponsor flag: commercial market-sizing vendor selling the report it is cited for
- trendforce.com press releases; payneinstitute.mines.edu MP Materials/DoD explainer; census.gov C30 index
- Tier B sources carrying the seven failures: capital.com, sedric.ai/blog, thomsonreuters.com/en-us/posts/legal/alsp-report-2023 (301s today; checked via Wayback 20260608), thenewstack.io + finance.biggo.com, growthspreeofficial.com + discover.gtmplaybook.co + cloudnuro.ai, tomtunguz.com + americanaffairsjournal.org, seomator.com/blog - all content-marketing or trade content; several are vendor blogs with an interest in the figures they publish