06-reference/research

verification ratio senior signoff throughput

2026-10-08·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
verification-bandwidthleverage-ceilingprofessional-liabilityagents-in-productionthroughput-capacity

There is no verification ratio — the published analogues span four thousand-fold, and the analogue closest to agent work products buys its throughput by sampling judgment rather than checking the artifact

The question

"What is the real verification ratio - how many agent-produced work products can one senior professional sign off per week at professional-liability standard? Use analogues where published: audit review rates, radiology second-read throughput, code-review throughput."

Filed as a derivative of [[2026-07-27-psf-pyramid-economics-post-agent-analyst-tier]], whose central finding was that the scarce layer in professional services moved from execution to verification. That brief's "new leverage ceiling" argument rests entirely on this number, and it closed by noting no professional-services firm has published it.

⚠️ Verification status of the numbers in this brief

Source-verification pass completed 2026-10-08. The brief was originally closed out before its primary-source pass: the three analogue researchers (audit, radiology, code review) had been dispatched but had not returned. All three have since returned with directly-retrieved sources, and a targeted verification pass re-pulled the brief's most load-bearing figures from their own documents. Every number below still carries one of three labels, and the labels remain load-bearing:

Now verified. All three analogues have a primary-verified spine. Code review: the METR RCT and its 2026 sequel, Cohen's Cisco study (article and book chapter), the Google ICSE-SEIP paper. Radiology: NHSBSP Publication No 59, Partridge 2024 reading times, the Cornford 2021 volume-to-quality curve, MASAI, PRAIM, and the Dratsch automation-bias trial. Audit: the text of AS 1220 itself, PCAOB's own post-implementation white paper with measured EQ-reviewer hours, the Korean EQCR-hour disclosure series, and FRC partner-load data.

Still unverified, and still not quotable. Fagan 1976 (paywalled; the circulating LOC/hour rates are secondary restatements that disagree with each other); Goodwin & Wu and López & Peters on audit-partner busyness (paywalled); McDonald et al. on cross-sectional imaging workload; Rigby & Bird FSE 2013 (paywalled — available only as quoted inside the Google paper, which is peer-reviewed secondhand rather than primary); NBER w35275; and all four vendor-telemetry figures. There is also no clean published mean of engagements per signing audit partner anywhere — the figure below is derived, not reported.

Five figures were corrected rather than confirmed, and one headline was downgraded. Wrong as stated: the Cisco LOC/hour threshold; the NHSBSP 5,000-case framing; MASAI's workload reduction; PRAIM's reading-time comparison; and the audit analogue's engagements-per-partner-week rate, which had no retrievable source at all. Downgraded: METR's 19% slowdown, which is correctly stated but which METR itself has since withdrawn confidence in. Each is marked CORRECTED inline with its prior value. That hit rate — five wrong in roughly twenty checkable figures — matches the 2-in-10 finding-level defect rate in [[2026-10-08-finding-level-citation-audit-arxiv-gloss]] almost exactly.

No number here should be put in front of a client, a prospect, or a public post unless it is SESSION-VERIFIED. This is exactly the false-verified-stamp failure mode in feedback_workflow_agent_output_integrity, so the labels stay in the document rather than being smoothed out.

What we already know (from the vault)

What the web says

Analogue 1 — audit partner sign-off. The unit is an engagement, and the find of this pass is that liability-grade sign-off is cheap as a share of the work product, not expensive.

Analogue 2 — radiology second reading. The unit is a study, the rate is per-hour, and this is now the best-measured analogue on the board.

Analogue 3 — code review. The unit is a change, and the best-grounded weekly throughput number in this entire brief lives here.

Convergences and contradictions

Synthesis for RDCO

The honest answer is that this question cannot be answered as asked, and that is a finding rather than a failure — and the verification pass made it stronger, not weaker. There is no verification ratio. There is a family of ratios set by the shape of the unit being verified, and in measured reviewer-time per unit they span roughly four thousand-fold (0.98 minutes for a screening mammogram, 66 hours for an audit engagement quality review). The sharpest support is that the single most-cited rate in this entire literature comes from the study that measured that the rate is not a constant: Cisco's best fit for a reviewer's inspection rate was R²=0.29, with rates varying "widely, even with similar authors, reviewers, files, and size of the review." Anyone quoting a verification ratio is quoting the headline of a paper whose finding was that there isn't one.

One correction to this brief's own central claim, stated plainly rather than softened. The brief originally argued that professional-services deliverables resemble audit partner sign-off, "the slowest analogue on the board." The measured audit data does not support "slowest." Engagement quality review runs 66 hours against 11,277 total engagement hours — 0.6% in the US, 1.8% in Korea — and AS 1220 achieves that by having the reviewer evaluate the engagement team's significant judgments and explicitly not re-perform the audit, with four tiers of filtering (team sampling → staff review → manager review → partner review) between the underlying transactions and the reviewer's desk. Liability-grade sign-off is cheap as a share of the work product, precisely because it is not comprehensive. Any framing in which one reviewer reads one deliverable end-to-end overstates audit's actual coverage by orders of magnitude — including the framing this brief used.

The claim that survives is better than the one it replaces. The structural resemblance to audit holds: professional-services deliverables are heterogeneous, one-off, have no base rate, no compiler, and no registry to check against later. But what audit teaches is not "expect to be slow." It is that high-throughput liability-grade sign-off is achieved by sampling the judgment layer rather than verifying the artifact — and that this is only defensible when the tiers beneath the reviewer, and the escalation path out of it, actually exist. That is the fourth checkability mechanism, alongside unit homogeneity, a trained error prior, and a machine pre-filter. If a number has to be stated, state it in audit's shape rather than audit's speed: a senior professional can sign off a meaningful number of liability-bearing work products per week only to the extent that something underneath has already filtered them, and the review is scoped to judgments rather than to the artifact — and state that RDCO has not measured its own rate and no published source gives it.

This disconfirms the parent brief, and that should be written down rather than softened. [[2026-07-27-psf-pyramid-economics-post-agent-analyst-tier]] argued that the scarce layer moved from execution to verification, and treated that as establishing a new, higher leverage ceiling. The first half holds — the vault, the labor-market data, and the lateral-hiring shift all support it. The second half does not survive this pass. Agent leverage only converts into a higher ceiling if per-unit verification gets cheaper, and the only measurements available say it gets more expensive per unit while getting much cheaper per unit generated. That is a worse position than the parent brief implied: generation goes up 741%, releases go up 20%, and the reviewer's per-unit cost goes up. The leverage ceiling may be lower after agents than the parent brief assumed, not higher. Our confidence that "verification is the new ceiling" should stay high; our confidence that we know where that ceiling sits should drop to near zero, and nothing downstream should be priced or promised against it.

The mechanism that actually moves the ratio is the one all three analogues share, and it is not agent count. In every analogue where the ratio is high, something upstream has made the unit cheap to check: tests and CI for code, a homogeneous image type plus a trained prior plus per-reader audit for mammography, tiered staff review plus judgment-scoped sampling for audit. The lever is manufacturing checkability, not adding reviewers or adding agents. Three measured findings sharpen this into operating rules. (1) Batching buys nothing. Cisco's defect discovery rate is roughly constant at ~13/hour regardless of artifact size, so a bigger review package yields the same findings and more escapes — throughput scales with reviewer-hours, never with artifact volume. (2) Terseness is a throughput feature. Machine output in the second-reader slot costs +33 to +41 seconds per case; in the first-pass slot it saves 77-104 seconds; CAD was faster only when it emitted ≤20 marks. Put the agent before the human as a filter, not beside the human as a second opinion, and make it brief. (3) Self-annotation is the cheapest lever available. Cisco reviews with author annotations never exceeded 30 defects/kLOC versus up to ~130 without — correlational, but the agent analogue is exact: an agent that flags its own uncertainty restores the triage signal that uniformly-confident output destroys. And one hard constraint on all of it: the automation-bias trial measured the most experienced radiologists dropping to 45.5% accuracy when the machine was confidently wrong. Reviewer seniority does not protect against a confident wrong machine, so no amount of "a senior will catch it" substitutes for a mechanical gate. [[2026-06-03-parallel-agent-fde-capacity]] reached the same conclusion from the other direction — the way to raise N is to collapse per-client review cost so the human reviews exceptions and rejections rather than artifacts — and this brief's analogue spread is independent support for it. Concretely, for RDCO's own stack: the value of the verify-* fresh-eyes critic layer is not that it reviews things, it is that it converts a heterogeneous artifact into a unit with a pass/fail contract, which is what moves the ratio from the audit regime toward the code regime.

For the founder's live positioning — credibility in agents in production, feeding phData sales — the asset here is the measurement method, not the number. The verified negative result is the opportunity: every buyer deploying agents in production needs this number to staff and price against, no professional-services firm has published it, and the available substitutes are vendor telemetry from companies selling review tools. That is an unusually clean gap for the exact niche the founder has chosen. The credible contribution is not "the verification ratio is N" — we would be inventing it, and feedback_calibrate_overconfidence applies directly. The credible contribution is publishing the instrumentation: define the unit, define what sign-off means, log per-artifact review minutes and defect escapes on real work, and report the measured rate with its degradation curve. That is a "here is how we measured it on a live deployment" artifact, which is the shape of content that builds the agents-in-production credibility the founder is actually building toward, and it is ours to produce because we run the agent stack daily and nobody else is logging it. Four cautions, all of which this verification pass earned the hard way. Do not cite MASAI's 44% as a workload time saving — it is a count of readings and the trial measured no times at all. Do not cite PRAIM's reading-time numbers as AI-versus-unaided; the paper says the control arm was never timed. Do not restate Every's 15-minute figure as a throughput benchmark. And do not cite METR's 19% slowdown as settled — METR itself withdrew confidence in February 2026, and a technical reader will know that. The defensible version of each is the structural point underneath it, not the percentage on top of it.

Why this is in the vault

It closes the explicit open follow-up left in [[2026-07-27-psf-pyramid-economics-post-agent-analyst-tier]] ("what is the actual verification ratio... this is the number the whole new-leverage-ceiling argument depends on") with a disconfirming answer: the number does not exist, the published analogues differ roughly four thousand-fold in reviewer-time per unit because the ratio is a property of the unit rather than the reviewer, and the analogue nearest to professional-services work products reaches its throughput by scoping review to judgments rather than to artifacts. It therefore retires the parent brief's leverage-ceiling claim as unsupported, confirms [[2026-06-03-parallel-agent-fde-capacity]]'s N=2-3 as the operative planning number, and converts the gap into a specific credibility artifact for the founder's agents-in-production positioning: publish the instrumentation, not a borrowed figure.

Open follow-ups

Related

Sources

Vault

Session-verified (web) — retrieved and read 2026-10-08

Code review

Audit

Radiology

Verified negatives

Search-relayed — vendor-published, not fetched, commercial interest in review tooling

Recalled, named for checking — still NOT retrieved

Peer-reviewed secondhand — quoted inside a primary source, not opened directly

Search-relayed — weak, directional only

Explicitly rejected — do not use