06-reference/research

unopened source citation audit

2026-08-01·research-brief·source: deep-research
fact-checkvault-integritycitation-hygieneprimary-sourcedeep-research

Unopened-Source Citation Audit: 13 Checked, 8 Verified, 4 Failed, 1 Unreachable

The question

How many other vault documents cite a statistic to a named index or report whose primary was never opened? A single grep-for-digits sweep over 06-reference/research/ would size the contamination [[2026-08-01-solo-operator-moat-figures-fact-check]] is a sample of.

This is the follow-up question that closed that brief, dispatched as its own ticket: sweep 06-reference/research/ for statistics/figures/quotes attributed to a named index, report, filing, or transcript, and spot-check whether the primary source actually contains the claimed figure. This is not an exhaustive audit — 226 files in the folder, ~100+ candidate numeric citations found by grep. The sample below was prioritized per the ticket's own criteria: (a) unusual-precision figures, (b) figures central to a synthesis doc's thesis, (c) direct quotes attributed to named people/companies.

Method

  1. Grepped 06-reference/research/*.md for decimal-precision percentages ([0-9]\.[0-9]% pattern) and for named-index/report/transcript/filing language — 100+ candidate occurrences across ~44 files.
  2. Selected 13 claims spanning health, competitive-positioning, memory-infra, LLM-behavior research papers, semiconductor/power-grid investing theses, healthcare outcomes data, and quantum computing — chosen for precision, thesis-centrality, or (in two cases already resolved by the prior brief) direct quote attribution to a named person.
  3. Dispatched two parallel research passes that fetched each primary source directly (WebFetch, or WebSearch/cross-reference when the primary was dead or paywalled) and checked for the literal claimed digits — the same standard the 2026-08-01 fact-check brief established: a live, correctly-linked source is not itself verification.
  4. Recorded VERIFIED / FAILED / UNCHECKED per claim, with a one-line reason.

Results — full claim table

# Doc Claim Cited source Verdict Note
1 [[2026-07-23-solo-operator-agent-fleet-org-shape-moat]] ShipSquad Solo Founder Index 23.7% (2019) → 36.3% (mid-2025) shipsquad.ai/blog/solo-founder-index-2026 via Foundra FAILED Numbers absent from the linked page (confirmed in [[2026-08-01-solo-operator-moat-figures-fact-check]] by direct HTML grep — zero occurrences of "23.7", "36.3", "2019").
2 [[2026-07-23-solo-operator-agent-fleet-org-shape-moat]] Amodei 70-80% chance of a one-person billion-dollar company Inc. (Ben Sherry), via an unretrieved aibusiness.vc snippet UNCHECKED Inc.com 403-blocks both WebFetch and browser-UA curl; figure only reaches the vault via search-engine summaries, never a primary open (per prior brief). Real but attribution chain broken.
3 2026-06-15-tirzepatide-uric-acid-gout-flare-window.md SURMOUNT-1 post hoc mediation: weight loss explained 72.7% of uric-acid reduction (PubMed 41198460) pubmed.ncbi.nlm.nih.gov/41198460 VERIFIED PubMed record states "weight reduction explained 72.7% of SUA reduction" — exact match.
4 2026-05-24-openai-workspace-agents-vs-claude-substrate-30day-check.md Terminal-Bench: GPT-5.5 82.7% vs Opus 4.7 69.4%; OSWorld GPT-5.5 78.7% MindStudio, DataCamp, BuildFastWithAI blogs VERIFIED (caveat) Not on the MindStudio page (star ratings only, no such numbers — one of three cited sources is a dud). DataCamp and BuildFastWithAI both confirm the exact figures independently.
5 2026-07-25-agent-memory-infra-commoditization.md Mem0 self-reported: 92.5 LoCoMo, 94.4 LongMemEval, ~6,900 tok/query vs ~26,000 full-context mem0.ai/blog/state-of-ai-agent-memory-2026 VERIFIED Blog post contains these figures almost exactly (~6,956/~6,787 avg per query — "~6,900" is a fair rounding).
6 2026-07-25-agent-memory-infra-commoditization.md "Zep leads temporal benchmarks (63.8% LongMemEval vs Mem0's 49.0%)" attributed to the same mem0.ai "State of AI Agent Memory 2026" post FAILED Those two numbers are not in that post — Zep is mentioned there with no scores attached. The 63.8%/49.0% figures are real (found via third-party sites, e.g. atlan.com, independent GPT-4o eval) but are misattributed to the wrong source in the vault doc — a citation-laundering pattern, not a fabrication.
7 2026-06-16-multi-agent-ensembles-conviction-calibration.md "Wisdom of the Silicon Crowd": ensemble Brier 0.20, human crowd 0.19, coin-flip 0.25, best single model (GPT-4) 0.15, mean forecast 57% vs 45% base rate arXiv 2402.19379 VERIFIED Full text confirms every figure exactly, including the 14/31 (~45%) base-rate detail.
8 2026-07-20-prompt-over-specification-capable-model-degradation.md IFScale: o3 97.8%@250 → 62.8%@500 instructions; 98-100%@10; 98.8-99.6%@50 arXiv 2507.11538 VERIFIED Table 3 of the paper: o3(high) 100.0%@10, 99.6%@50, 97.8%@250, 62.8%@500 — exact match.
9 2026-07-03-cxmt-dram-capacity-memory-oligopoly-risk.md DRAM bit-share trio: SK Hynix 33.2% / Samsung 32.6% / Micron 25.7% SemiAnalysis, "China's CXMT Is Set to Challenge DRAM Incumbents" FAILED Article fetched directly — these three numbers do not appear anywhere in it (the piece covers CXMT's own bit-share trajectory 9%→12% and ~17% wafer capacity, not an incumbent breakdown). Independently found real-world figures (SemiAnalysis / TrendForce) are meaningfully different: Samsung ~36-38.5%, SK Hynix ~28.8-32.1%, Micron ~22.4%. The cited trio matches neither the cited source nor any other primary found.
10 2026-07-28-pjm-peak-load-forecast-power-cycle-corroboration.md PJM 2030 forecast rose ~19.0% across four vintages (Jan-2022→Jan-2026); Jan-2026 vintage cut the 2028 target-year forecast −2.6% vs Jan-2025 PJM Load Report Tables (2025, 2026 vintage XLSX) VERIFIED Downloaded both source workbooks and re-derived the numbers directly from Table B1 (PJM RTO row) — every figure in the doc, including all five year-over-year deltas, matches the raw data exactly. Strongest verification in this sample.
11 2026-05-25-virta-health-5yr-cohort-reversal-durability.md Virta DRCP 5-year cohort: 7.6% average weight loss, cohort chain 262→194→169→122 Virta DRCP 2024 extension study VERIFIED Study abstract confirms "-7.6%" body-mass change and the exact cohort attrition chain cited.
12 2026-07-08-cortex-sense-semantic-layer-wedge-caf.md Cortex Sense raises agent accuracy from ~24% to 86.3% Snowflake blog (snowflake.com/en/blog/enterprise-ai-agents-grounded-context) VERIFIED Blog states "improved accuracy from 24.1% to 86.3% on our benchmark" — exact match.
13 2026-05-24-quantum-computing-2026-milestone-thresholds.md Rigetti Cepheus-1 (108 qubits) listed at 99.5% current two-qubit gate fidelity heygotrade.com blog (now dead, HTTP 410) FAILED Original source is gone. Cross-verified via Rigetti's own press materials: actual current fidelity as of the April 2026 GA announcement is 99.1%; 99.5% is explicitly a stated future target ("expects to reach... later this year"), not an achieved figure. The doc presents a target as a present-tense leaderboard entry alongside competitors' actually-achieved numbers — a different failure shape than the others (temporal conflation, not absence).

Summary count

13 claims checked: 8 VERIFIED / 4 FAILED / 1 UNCHECKED.

Failure rate on this sample: 4/13 (31%), well above what the folder-wide arXiv audit found for author-attribution defects ([[2026-07-27-vault-arxiv-citation-audit]], 1.7% per-ID historical rate) — but that audit checked a mechanically cheap thing (byline match); this one checked the harder thing (does the number exist at all), on a sample deliberately weighted toward high-precision, thesis-load-bearing figures rather than a random draw. Not a directly comparable base rate, but consistent with the arxiv audit's own finding that the mechanically-checkable defect (wrong byline) is rare while the harder-to-check defect (wrong or absent numbers, mischaracterized findings) is where the real exposure sits.

Failure shapes observed, three distinct patterns:

  1. Absent from source (ShipSquad, CXMT DRAM) — the cited document exists, resolves, and simply does not contain the claimed number.
  2. Misattributed source (Zep/Mem0 comparison) — the number is real and findable elsewhere, but the vault credits the wrong document for it, so anyone re-checking the given citation gets a false negative.
  3. Temporal conflation (Rigetti quantum fidelity) — a forward-looking target figure presented as a currently-achieved result.

None of the 8 verified claims required more than one fetch to confirm; several (PJM, Cortex Sense, tirzepatide, IFScale, Brier scores) matched exactly on the first primary-source read, which is a reminder that most of the vault's numeric citation discipline is sound — the failures are concentrated, not diffuse, and cluster in exactly the shape the 2026-08-01 fact-check brief predicted: a live, correctly-formatted, plausible-looking citation that simply isn't backing the number when actually opened.

What this does NOT cover

This is a spot-check, not an exhaustive sweep. ~100+ additional decimal-precision figures were located by the initial grep and not individually checked (time/effort budget). Direct-quote-to-named-person citations (beyond the Amodei case already covered) were grep-surveyed but no comparably load-bearing, unusually-specific quote-plus-number pair was found to justify inclusion in this sample — that category may still hide instances and was not exhaustively checked either.

Follow-up items status (not acted on, per this ticket's scope — decision items, not action items)

1. Correction to [[2026-07-23-solo-operator-agent-fleet-org-shape-moat]] (exact replacement text specified in [[2026-08-01-solo-operator-moat-figures-fact-check]]): NOT APPLIED. Confirmed by direct read of the live file — line 34 still carries the original unhedged "ShipSquad Solo Founder Index (via Foundra): solo-founded startups 23.7% (2019) → 36.3% (mid-2025)" text and the Amodei sentence is unchanged. The prior brief explicitly flagged this edit as "not applied — flagging for approval, since editing a published brief is outside this dispatch's scope." Still awaiting founder decision.

2. 11-file ranked edit list in [[2026-08-01-above-the-platform-tier-corroboration]] for downgrading $15K-$30K tier provenance: NOT APPLIED. Confirmed by direct read — the doc's own "Files that need editing" section is explicitly headed "do NOT auto-edit — founder review first." No edits have been made to any of the 11 listed files. Awaiting founder decision.

Both remain open decision items for the founder, not blockers on this audit's completion.

Synthesis for RDCO

This closes the sizing question the 2026-08-01 fact-check brief raised in its own follow-up ("how many other vault documents cite a statistic to a named index or report whose primary was never opened?") with a concrete, if partial, answer: on a prioritized 13-claim sample, roughly a third failed primary-source verification, in three distinct failure shapes. It gives /deep-research and future citation-hygiene passes a working example set of what "opening the primary and grepping for the digits" catches that link-liveness and domain-reputation checks do not.

Open follow-ups

Related

Sources

Vault:

Web — retrieved this run (see individual claim rows for exact URLs):