06-reference/research

non arxiv citation exposure sample

2026-10-20·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
vault-integritycitation-verificationdeep-researchharness-qualitysampling

Non-arXiv citation exposure, resampled at n=36: the rate is indistinguishable from the arXiv corner, and every single failure came from a secondary source

The question

"Size the non-arXiv citation exposure properly: extend the 2026-08-01 unopened-source audit from 13 checked to a defensible sample, and measure how often a quantitative claim in a later brief traces to a vault doc rather than to the primary source."

The premise carried on the backlog item was that the non-arXiv corner fails at roughly double the arXiv rate (31% vs 20%) and is the corner with the dollar consequences, sampled only once at n=13. This brief triples that sample using random rather than thesis-weighted selection. The premise does not survive.

What we already know (from the vault)

Method and sampling frame

Reproducing the parent's adjudication standard (does the claimed figure exist in the cited source at all), with the 2026-10-08 additions (version pinning, full-text checks, skips counted separately).

1. Population. 06-reference/research/ held 426 markdown files at enumeration time, up from 226 in August (427 once this brief lands). A quantitative-claim signature of a decimal-precision percentage, a precise dollar magnitude, or a decimal multiplier matches 1,509 lines across 219 files. Of those, 40 sit next to an arXiv reference and are excluded as already twice-audited, leaving 1,469 non-arXiv quantitative claim-lines.

2. Frame. A claim is in frame when it (a) matches the quantitative signature, (b) has no arXiv reference within plus-or-minus 3 lines, and (c) has at least one non-arXiv HTTP(S) URL within plus-or-minus 3 lines, so the claim and a candidate source are co-located and the attribution is unambiguous enough to check. That is 343 claim-citation pairs across 127 files. Removing the 14 documents already sampled in August, so this extends rather than re-checks, leaves a sampling frame of 314.

3. Draw. Seeded shuffle, random.Random(20261020), first 40 in draw order. No post-hoc swaps and no filtering for claims that looked suspicious, which is the bias that inflated the August figure. Enumeration and draw scripts are reproducible from the frame definition above.

4. Adjudication. Eight parallel workers, one per 5 claims, each resolving the authority the vault actually gives for that specific number (inline link first, then the file's Sources entry matching the named publisher), then opening it. Verdicts: VERIFIED, VERIFIED-AS-HEDGED (figure checks out and the doc's own stated confidence matches reality), FAILED with one of the parent's three shapes, SKIPPED-PAYWALL, UNREACHABLE.

Three honest corrections to the frame, found by running it.

The rates

Measure Result 95% CI (Wilson)
This audit, random sample 7 of 36 = 19% 10% - 35%
Parent 2026-08-01, thesis-weighted 4 of 12 = 33% 14% - 61%
Pooled August + October non-arXiv 11 of 48 = 23% 13% - 37%
Comparator, 2026-10-08 arXiv gloss 2 of 10 = 20% 6% - 51%

Skipped and unreachable: zero of 36. No paywall was hit and no cited source was dead. Every source in the sample opened, several only via a workaround (NCBI E-utilities for reCAPTCHA-walled PubMed Central, Europe PMC fullTextXML, a browser User-Agent for sec.gov, Wayback for version pinning). Contrast the August sample, which hit one 410 and one hard 403.

Failure shapes, the same three the parent named, in the same rough proportions: absent-from-source 4, misattributed 2, temporal-conflation 1. No new shape appeared in a 3x larger sample, which is a meaningful negative result: the taxonomy is stable.

The premise fails. 19% (10-35%) against the arXiv corner's 20% (6-51%) is not double, and the two intervals almost entirely overlap. Nor is this audit distinguishable from its own parent: Fisher exact two-tailed on 7-of-36 against 4-of-12 gives p = 0.43. The August 31-33% was not wrong, it was a non-random sample behaving exactly as its own caveat predicted. The vault has been carrying "the non-arXiv corner is twice as bad" as a load-bearing fact for eleven weeks on evidence that never supported it.

The finding that does survive: it is a source-tier rate, not a corpus rate

Splitting the 36 by the tier of the authority the vault actually cited. Tier A is an official filing, a government or regulator publication, a peer-reviewed journal, or an organization's own first-party page about itself. Tier B is trade press, general news, aggregator sites, newsletters, and vendor content-marketing blogs.

Tier Failures Rate 95% CI
A: filing / regulator / journal / first-party 0 of 19 0% 0% - 17%
B: trade press / news / aggregator / marketing blog 7 of 17 41% 22% - 64%

Fisher exact, two-tailed: p = 0.0023.

The 19 Tier A claims, enumerated so the boundary is auditable: sec.gov three times (Snowflake FY2026 8-K twice, the 2006 Pixar merger exhibits); a Virginia State Corporation Commission report; three journal records via PubMed Central and PubMed; Alphabet investor relations; Databricks newsroom; claude.com/pricing; an Anthropic news post; an Anthropic engineering blog reporting its own telemetry; two Forrester Total Economic Impact pages; a Colorado School of Mines institute explainer; TrendForce; MarketsandMarkets; haus.io publishing its own holdout results; and the United States Census Bureau page, cited correctly as a negative. Every one of the seven failures sat on Tier B.

Sensitivity on the boundary, because 2 to 6 of those 19 assignments are genuinely contestable. Vendor-commissioned research and commercial market-sizing firms are first-party about themselves but have an obvious interest, so the split was re-run with them demoted. Moving MarketsandMarkets and TrendForce to Tier B gives 0 of 17 against 7 of 19, p = 0.0084. Also moving both Forrester studies gives 0 of 15 against 7 of 21, p = 0.0273. A worst case that additionally demotes the two first-party-telemetry pages gives 0 of 13 against 7 of 23, p = 0.0343. The split survives every reassignment tested, which is the claim being made here; the exact p-value is not.

This matters more than the headline rate because it is actionable, and because of the composition of the corpus: classifying the first URL of all 343 in-frame pairs gives roughly 68% secondary, 21% primary or official, 11% vendor first-party. The exposure is concentrated in the two-thirds of citations pointed at content that was itself written to be linked.

The failure unit is the bullet, not the citation. Workers converged independently on one shape: a bullet bundles several figures behind one or two links, the explicitly-cited figures verify, and the un-cited bundled figures are the ones that break. The forward-deployed-engineer bullet's headline surge (643 to 5,330 postings, 729%) verifies verbatim while four comp figures in the same sentence appear in neither cited source. The telecom vendor-financing bullet verifies Lucent's $8.1B committed, Nortel's $3.1B and Cisco's $2.4B, then adds "a later McKinsey tally put combined exposure across nine suppliers at about $25.6B" which is in neither cited document and has no McKinsey entry in that file's Sources at all. Per-figure attribution, not per-bullet, is what would have caught both.

There are defects this design never looked at. Workers incidentally flagged defects in clauses adjacent to a verified sampled figure, which the per-line scoring rule does not reach because those figures were not attributed to the resolved source: a per-asset price derived by dividing three-year licence cost by a terminal asset count from a ramping series (roughly $53 where $149 is honest); a bullet pair that silently switches between nominal and present-value cost basis; a "fab utilization sustained 95%+" clause borrowing a market-share number; a rare-earth supply-chain partner list credited to a page naming only one of five; a "40,000+ firms applied" figure attributed to an Anthropic page containing neither number. Six such flags, found opportunistically by workers who were not looking for them rather than by any systematic pass, so no rate can be computed from them and none is claimed. What they establish is only that a non-empty class of defect sits outside this design's reach.

Amplification, measured separately

Operationalized on claim bodies only, excluding bibliography sections, by what authority sits within plus-or-minus 2 lines of a quantitative claim.

Authority offered Count
Vault doc only 341
External URL only 204
Both 15
Self-labeled own arithmetic 34
None co-located 719

Structural amplification: 341 of 560 authority-bearing body claims, or 61%, offer a vault doc and no primary. That number is real but it is mostly the brief template working as designed, because ## What we already know (from the vault) is a required section whose whole job is to cite vault docs.

So I hand-traced a seeded random 15 of the 341. 8 carried a number of external origin. Of those 8, 6 gave the vault doc as the sole authority and 2 restated the primary provenance explicitly, one of them verbatim ("read the 2024 five-year paper's full text directly and recorded the medication figures with their basis"). No interval is claimed on a denominator of 8.

Then I traced whether those chains terminate. I followed 4 chains to the parent document. In 4 of 4, the chain did reach an external attribution one hop up, so amplification here lengthens the chain rather than breaking it. But 3 of those 4 terminated at content-marketing or aggregator content, not a primary: a $575B Korea fab figure resolving to a Motley Fool article; a course-launch case resolving to a blog the parent file itself labels "primary source for ... Aamir A/B"; and a paid-pilot conversion figure resolving to Monetizely, which in turn credits McKinsey and Forrester. That last one is the right behavior rather than a defect, because the parent credits the intermediary it actually read instead of the original it did not.

Amplification is therefore not the exposure. It routes back to Tier B. The amplification measure and the failure-rate measure are the same finding seen twice.

The correction loop is the actual broken loop

Re-checking the four defects the August audit identified, 80 days on:

Four of four identified defects are still in the vault. Four citation audits have produced zero corrections. Measurement is not the constraint.

A note against this brief's own credibility, kept in deliberately: the first draft of this section reported the Zep/Mem0 defect as resolved by deletion. It was not. The grep that produced that conclusion had errored out on a shell-glob problem and returned nothing, and the empty result was read as absence rather than as a failed command. A fresh-eyes critic pass caught it with one grep. That is this brief committing, in a section about unverified numbers, the precise error class it is measuring, and it is the strongest available argument that the gate proposed below has to be mechanical rather than attentional.

What the web says

The external benchmark for this class of error is the human quotation-error literature. [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] already brought that benchmark into the vault, and this run re-opened its anchor paper independently, because that paper happened to come up as sample item S16.

Convergences and contradictions

Synthesis for RDCO

The honest summary is that we have been measuring the wrong axis. Three audits sliced the corpus by subject domain, asking whether the arXiv corner is cleaner than the named-report corner. On a random sample those two corners are statistically indistinguishable, both near one in five. Slice the same 36 claims by the tier of the source instead and the picture resolves sharply: zero failures in 19 claims citing filings, regulators, journals, and first-party pages, against 7 failures in 17 claims citing trade press, newsletters, and vendor marketing blogs, at p = 0.0023. Subject domain was a proxy for source tier all along, and a lossy one. The arXiv corner keeps coming back clean because arXiv papers are primary documents, not because arXiv has an application programming interface.

That reframes the harness fix, and makes it cheaper than anything the prior three audits proposed. A verification gate applied to every numeric claim is expensive and, on this evidence, would spend most of its budget confirming filings and journal abstracts that were already right. A gate conditioned on source tier is close to free: when a figure's resolved authority is not a filing, a regulator, a journal, or the subject organization's own page, that figure needs either a primary-source hop or an explicit unverified label before it can be stated flat. Nineteen of our 36 claims would skip the gate entirely, and all seven defects would have been routed into it. That second number is in-sample and should not be read as a yield estimate: the tier boundary was drawn after seeing which claims failed, so a 100% catch rate on the claims that defined the boundary is close to tautological. What is not in-sample is the direction and size of the effect, which survives every boundary reassignment tested above. The honest claim is that tier is a strong predictor worth gating on, not that gating on it catches everything. The same rule handles the amplification result without a second mechanism, because the chains that terminate badly terminate at Tier B.

The second finding is about the unit of verification, and it cuts against how our briefs are actually written. Our defects do not look like a bad citation; they look like a good citation with passengers. A bullet opens with a figure that is verbatim correct and correctly linked, then accumulates three or four more figures from elsewhere behind that same link. The sourced figure holds, the passengers fail. This is why per-bullet review keeps passing documents that per-figure review would stop, and it is also why our measured 19% is a floor rather than an estimate: six defective clauses turned up incidentally in the immediate neighborhood of figures that verified, found by workers who were not looking for them. A per-figure attribution requirement in the brief template is a larger change than a provenance label, and it is the one the evidence actually asks for.

The third finding is the uncomfortable one and it is not a measurement finding. All four of the defects the August audit identified, including one with exact replacement text already drafted, are still sitting in the vault 80 days later. Four citation audits have produced zero corrections. Whatever the marginal value of a fifth audit, it is smaller than the value of applying the output of the first four, and this brief should not be read as an argument for more measurement. It is worth noting what the correction backlog costs in compounding terms: the misattributed benchmark descriptors found in the "What the web says" section above sit inside the 2026-10-08 audit itself, so the audit series is now generating the defect class it exists to measure, faster than it retires it. The measured exposure is roughly one in five non-arXiv claims, concentrated almost entirely in secondary sources, stable in shape across three samples, and no longer in serious doubt about its order of magnitude.

On defensibility, stated plainly: n=36 gives a 95% interval of 10-35%, which is wide enough that this brief can establish the order of magnitude and the tier split but cannot pin the rate. Resolving a 20%-versus-40% difference at 80% power needs 82 per arm. Getting a single rate to plus-or-minus 5 percentage points needs 246. Neither is worth buying. The tier split reached significance at n=36 because the effect is large, and that is the result worth acting on.

Why this is in the vault

This retires the "non-arXiv fails at double the rate" premise that [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] put into circulation and that the Research Backlog used to justify High priority, and it replaces it with a source-tier rule that /deep-research and /verify-vault-write can both implement as a conditional gate rather than a blanket check. It also puts a number on the correction backlog: four of four known defects are live and four audits have yielded zero fixes, which is the input to deciding whether /self-review should be allowed to auto-fix flagged figures rather than only flagging them.

Open follow-ups

Related

Sources

Vault (the audited corpus and the method parents):

External sources opened this run (36 adjudicated claims; representative, full per-claim list held in the audit working notes):