06-reference/research

verify skills pass disagreement audit

2026-07-03·research-brief·source: deep-research
verify-stackllm-judgecritic-blind-spotsimprove-croncalibration

Where the verify-* critics emit false PASSes — and why we can't yet prove it

The question

Across the seven verify-* skills (verify-vault-write, verify-strategic-output, verify-dispatch, verify-pdf-output, verify-action, /design-critic, /video-critic), where has the founder disagreed with a PASS verdict in the last 60 days (2026-05-04 → 2026-07-03), and what are the recurring failure-mode patterns the critic is blind to? Context: the v3 SpaceX memo was a verify-pdf-output gap that triggered a skill build; the /improve cron is scoped to monthly sample-and-regrade but the historical failure inventory has never been compiled.

What we already know (from the vault)

What the web says

Convergences and contradictions

Synthesis for RDCO

Be explicit: the honest count of founder-vs-PASS disagreements in the last 60 days is zero, and that is a finding, not a reassurance. The disagreement inventory this brief was meant to compile cannot be compiled because the verify-* stack never logged its verdicts to a durable per-axis file. "Zero overrides" and "critics are well-calibrated" are indistinguishable given the current instrumentation. So the first thing /improve must test is not drift — it is whether a disagreement corpus can exist at all. Blind spot #0 (meta): the stack has no verdict log; stand up improve-cron-log.md / a per-axis flash-review-log.jsonl before any regrade math is meaningful.

Ranking the substantive blind-spot patterns the critics are structurally exposed to, most-to-least load-bearing: (1) Surface-over-substance ("Evaluation Illusion"). verify-pdf-output scores 12 layout invariants and nothing about whether the memo's argument holds; verify-vault-write scores whether an RDCO-mapping/cross-link is present, not whether it is insightful or points at the right thing; design-critic scores tells, not taste. This is where the SpaceX and video gaps both live, and where the literature says single-model judges fail hardest. (2) Rule-letter vs rule-intent (constitutional loophole). Presence axes — "cite a source," "has cross-links," "has an RDCO mapping," "has a bear case" — are satisfiable by hollow compliance (boilerplate mapping, links to the wrong doc, a token bear paragraph). (3) Self-preference / no capability differential. Producer and critic are the same Claude; the literature says this inflates false-PASS, and RDCO's only structural counter is the fresh-eyes zero-context dispatch — which defeats confirmation bias but not shared surface heuristics.

What the /improve cron should test (concrete, falsifiable hypotheses): (H1) Run the RGFMD adversarial-self-generation step already specced in [[2026-05-23-improve-cron-design-spec]] against verify-pdf-output and verify-vault-write first — manufacture artifacts that pass every mechanical/presence check but are substantively wrong or hollow, and measure how many the critic still PASSes. A high pass rate confirms the surface-over-substance blind spot empirically. (H2) Treat the two in-window gaps (MAC video C v2, SpaceX v3) as the seed negative-example corpus and back-fill them as gold-set FAILs so the regrade has something to score against. (H3) On the single highest-stakes axis (verify-strategic-output, where a false PASS ships a wrong thesis), pilot a cross-model or differently-prompted critic and measure the PASS-rate delta vs the same-model critic — the cheapest test of the self-preference hypothesis. (H4) Add one semantic axis on top of the presence axes in verify-vault-write ("does the RDCO mapping say something non-obvious?") and see whether it diverges from the presence check on real traffic. The design spec's own metric note applies: at RDCO's small N, founder-override events are the primary drift signal and Cohen's kappa is only confirmation — which is precisely why the missing log is the binding constraint.

Open follow-ups

Related

Sources

Vault:

Web: