06-reference/research

critic layer review time defect escape

2026-10-20·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
verify-stackinstrumentationcritic-calibrationmeasurementpositioning

Our critic layer costs about four minutes an artifact, and the only part of it with a ground truth was overturned 8 times in 9

The question

"What is RDCO's own measured per-artifact review time and defect-escape rate across the verify-* critic layer, computed from the existing dispatch logs?"

This is an audit of our own operational logs, not a literature review. It is the self-measurement that [[2026-10-08-verification-ratio-senior-signoff-throughput]] proposed selling as a method, and the measurement that [[2026-07-03-verify-skills-pass-disagreement-audit]] attempted and abandoned.

What we already know (from the vault)

What the web says

No external search was run for the core numbers: every figure below traces to a file on local disk, and the question is answerable only from our own logs. Two points of external context carry over from [[2026-10-08-verification-ratio-senior-signoff-throughput]] and are relayed from that brief rather than re-verified here:

One external-shaped claim in the backlog note does not survive contact with the disk, and it is ours, not the web's: the premise that "we have months of real dispatch logs" is false for session transcripts. See the next section.

Convergences and contradictions

Synthesis for RDCO

Per-artifact review time, defined precisely. The logs support exactly one quantity: elapsed wall-clock of an autonomous critic subagent, measured as the last timestamp minus the first timestamp inside its own transcript. Across 473 classified critic runs in the 32-day window, the median is 254 seconds, about 4.2 minutes. Trimming the 4 aborted runs under 20 seconds and the 3 over an hour leaves n=466 and a median of 256 seconds, so the figure is robust to its tails. The interquartile range is 116 to 619 seconds and the 90th percentile is 905 seconds, about 15 minutes. The method was hand-checked against one transcript: agent a51144b31b0f5c9b6 ran 10:56:12.260Z to 10:57:48.151Z on 2026-09-30, which is the 95.891 seconds the script reported. Per family the spread is wide and interpretable: verify-dispatch 134 seconds (n=97), ad-hoc design critics 233 seconds (n=96), the design-critic skill 367 seconds (n=112), verify-vault-write 645 seconds (n=30), verify-strategic-output 652 seconds (n=36).

What that number is not, and this matters more than the number. It is machine elapsed time with no human attention in it, inflated by model latency and tool input/output and in three cases by idle resume. It is not the quantity a buyer means by "review time," which is human attention per artifact. Conflating the two would produce a benchmark-shaped artifact that is not a benchmark, and that is the single easiest way to lose credibility on this topic. Tokens and metered cost are not extractable for this corpus: the ledger's cost_micros_api column is null for subscription studio work by design. Tool-call count is available as an alternative unit if a load proxy is ever wanted (median about 26 calls for design critics, 14 for verify-dispatch).

The human unit, measured once. There is exactly one clean founder-side review-time observation on disk, and it is worth more than the machine figure. On 2026-10-04 the founder adjudicated a prepared packet and working-context.md logs 13 distinct decisions between 08:40 and 08:46 ET, a seven-minute span, or roughly 30 seconds per decision. Several decisions covered English and Spanish sibling pairs, so per artifact it is nearer 20 seconds. The honest caveat is large: he was ruling on pre-rendered screenshots assembled into a review packet, not inspecting artifacts, so this is decision time on a prepared summary. Read carefully, it says the expensive part of our pipeline is packet preparation, not founder attention, and the critic spends roughly eight times the founder's wall-clock per artifact while reaching a different answer.

Defect-escape rate: measurable, small, and ugly. The ledger holds 80 critiques over 2026-10-02 to 2026-10-07, 66 from agents and 14 from the founder. Fourteen iterations were graded by both. The confusion matrix: agent ACCEPT with founder REVISE, 8; ACCEPT with ACCEPT, 1; REVISE with ACCEPT, 1; REVISE with REVISE, 4. So of the 9 agent ACCEPTs the founder also ruled on, 8 were overturned. As a rate that is 0.89 with a Wilson 95% interval of 0.56 to 0.98, and the raw counts carry the meaning better than the decimal does. All 9 were design-critic. The reverse direction is also measured and is not zero: 1 of 5 agent REVISEs was overturned to ACCEPT, so the critic is not simply strict or simply lenient, it is decorrelated.

Two corrections that stop this being quoted as a headline. First, the 8 escapes are not 8 independent events. Five share a single founder judgment at 2026-10-01 21:25 ("art still basic; use Quiver/Grok", all on PR #387) and a sixth is the 21:27 ruling on PR #388. Only two are independent later judgments. At the level of distinct founder judgments the rate is 4 of 5, interval 0.38 to 0.96, and with n that small the interval is the finding. Second, the escapes decompose into two different failures with different fixes. Six of eight are criterion-absent: the founder applied an art-quality bar the rubric never contained, which is the same standard recorded in memory as the 2026-10-01 real-art-tools correction. Two of eight are criterion-present-but-mis-scored, and those are the damaging ones: sink-or-float ACCEPT at 19/19 against "counting circles and instructions confusing", and what-comes-next ACCEPT at 17/19 against "all 4 rows same AB alternation; pre-shaded example apple read as the thing to color". Adding criteria fixes the first six. Nothing about adding criteria fixes a 19/19.

The scope limit that is the real answer to the question as asked. The question asks about the verify-* layer. The measurable escape rate above is not from that layer. It is from the studio design-critic path on children's printables, a single domain, a single human grader, six days. The verify-* stack proper accounts for 352 of the 473 runs and has zero paired founder adjudications in the window, so for the layer the question actually names, the escape rate remains unmeasurable. It is no longer unmeasurable for want of a schema, which is the update: the schema exists and works, and the verify-* skills simply do not write to it. A second candidate ground-truth event exists and is deliberately excluded from the arithmetic above, because the log that records it also undercuts it: the founder killed moon-path-maze on 2026-10-04 ("this maze isn't valid. It looks like junk.") and the same ledger line notes the game "passed a critic on 09-06 (pre-v0.2)", while a nearby line concedes "only some had studio critic gates; 09-06 batch provenance unverified." An escape we cannot attribute to a specific PASS does not belong in a numerator.

The gate is not a rubber stamp, which was the other worry. Across 473 runs the distribution is ITERATE 270, PASS 77, REVISE 47, ACCEPT 43, SCRAP 4, FAIL 1, 5 runs that returned no verdict at all, and 26 whose verdict could not be parsed. Pass-equivalents number 120 against 322 fail-equivalents, a first-pass pass rate of 0.27. SCRAP has fired, 4 times, so the severe tier is not dead. But all 4 were triggered by the mechanical failure-count threshold rather than by judgment, and in 3 of the 4 the critic disputes its own verdict in the same breath: "SCRAP under the skill's mechanical rule (4 rubric items fail, and the threshold is 4 or more). In practice, though, the brief needs edits, not a rewrite"; "No SCRAP-class defect"; "triggered by the failure count, not by a SCRAP-class defect." Gate 2 is doing arithmetic, not triage. Two further hygiene findings fall out of the same tally. The layer runs two incompatible verdict vocabularies, PASS/ITERATE/SCRAP across 352 runs and ACCEPT/REVISE/REJECT across 95, which is why no single escape-rate query spans the layer; the literal string "Gate 1" appears in only 15 outputs, so the 05-20 two-gate rollout substantially did not land. And 5 runs ended with "You've hit your monthly spend limit" instead of a verdict, which is a gate that fails open and silently.

What this changes. The sellable asset from [[2026-10-08-verification-ratio-senior-signoff-throughput]] survives and gets more specific. We can now publish a unit definition, a measured machine cost per artifact with its distribution, a measured human decision cost, and a paired overturn rate with an honest interval and an honest selection caveat. That is more than any professional-services firm has published and more than the review-tool vendors can publish, because their incentive runs the other way. What we cannot publish is a layer-wide escape rate, and the reason is a one-line omission in each skill rather than a missing system.

Why this is in the vault

This is the self-measurement that gates the positioning bet in [[2026-10-08-verification-ratio-senior-signoff-throughput]]: we proposed publishing the agent-verification instrumentation method as credibility for phData sales, and this brief is the test of whether we can actually run it on ourselves. It also retires the open finding in [[2026-07-03-verify-skills-pass-disagreement-audit]] by locating the ground truth that audit could not find, and it supplies the first real overturn numbers for any future calibration or regrade work on the critic layer.

Open follow-ups

Deliberately not filed as research, because it is a build: wiring the existing verify-* skills to write a row into the eval-ledger schema that already exists. That belongs in the recommendation above and on the Notion task board.

Related

Sources

Local operational data, all read on 2026-10-20:

Vault documents: