Mental model: Verification is a system, not a step. The verification layer has to be kept out of reach of the changes it judges, graded against the bugs that actually mattered, fed with rules mined from misses, and pruned. A team that does this can stop reading diffs and start defining "done" and reviewing evidence, which is what lets one person run many agents at once.
⚠️ Sponsorship
Anthropic's own engineering blog about Anthropic's own product. Every recommended mechanism maps to a Claude Code feature (skills, hooks, /goal, /code-review, /verify, REVIEW.md, Code Review, the Chrome extension). The headline numbers (about 90% of team code written and reviewed by Claude; a separate rubric grader adding "up to 10 points" of task success on structured file generation) are internal and unaudited. The practices themselves are product-agnostic and line up with independent sources already in the vault, so the bias is in framing and tool choice more than in substance.
Why this is in the vault
This is the most concrete first-party description yet of how a frontier lab grades its own critics, and several of its specifics (test the reviewers against an incident answer key, diff-triggered checklist, observe-mode gates) are things Ray's critic fleet does not do. It is also current phData/Anthropic-enablement material for the founder's day job.
Summary (author's terms, paraphrased)
Premise. Verification loops are what let you delegate. Without them a person can supervise one or two agents; with them, dozens. Results should close the loop: a repeated failure becomes a hook or a CLAUDE.md rule.
Three test types worth investing in.
- End-to-end first. "If you have to focus on one test type, pick end-to-end." E2E tests are hard to fool and fail with traces and screenshots. Give the agent a skill that boots, drives and inspects the running app, plus a "feature map" of user paths and expected results that doubles as a rubric.
- Property tests (fast-check, Hypothesis, proptest): state an invariant and throw thousands of generated inputs at it. The value is that the edge cases are not picked with the same bias as the agent that wrote the code.
- Differential tests: old vs new system on the same inputs, for migrations, with a normalizer and an approved-changes list. Flaky tests teach the agent to rerun instead of fix; slow tests cut runs per batch.
How the team runs it (nine practices).
- Verification means running the change, via a
verifyskill kept next to the code that knows how to boot the app/simulator. - Three strikes then a human: Claude tries to fix failing checks; after three failed rounds it stops and escalates.
- Code owners write their own checks and tune them until they flag what the owner would flag.
- Diff-triggered reliability checklist: 39 items, each with a diff-readable trigger, a check, and the incident or PR that created it. Claude runs only the matching 3-10.
- One reviewer, many rounds: tune one primary reviewer (with a
REVIEW.mdof focus areas) rather than stacking reviewers; a second, costlier reviewer is often worse. - Changes can't modify their own checks:
REVIEW.mdis read from the base branch, never the PR; reviews are pinned to a commit and go stale on push. "If a change can modify the check that judges it, the check is broken." - New gates start in observe mode: every commit is stamped with how it was verified, including "none"; a commit gate has off / observe / enforce modes; after two blocks in a session the next commit passes so the agent can't get stuck.
- Test the reviewers: rebuild the repo at review time for past incidents, with a one-sentence answer key; credit only findings that describe the real bug; use clean merged changes as over-flagging controls; keep tuning and test sets separate; measure recall at raw-finding, post-verification and final-output stages; grade severity (Must-fix / Should-fix / Nit / Pre-existing).
- Turn misses into rules: mine review comments and reverts for patterns seen in at least two PRs, have three agents try to disprove each candidate, human approves; replay bug finders against incident PRs to see what is detectable; flaky tests may not be skipped, slept or retried, only fixed or deleted with a reason; prune rules and docs regularly because they only grow.
Mapping against Ray Data Co
What Ray already does (with receipts):
- Independent grader, not self-review. [[feedback_fresh_eyes_subagent_for_own_artifacts]] and the
/verify-*family ([[2026-05-19-verification-as-independent-worker-pattern]], [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]]) are the same "separate grader against a rubric" pattern the article credits with its 10-point gain. - Run the thing (their practice 1, and E2E-first).
~/.claude/skills/behavior-critic/is a source-blind critic that runs the script, clicks the page or calls the endpoint against a behavior contract. It came out of [[2026-07-26-autoreview-skill-teardown-second-model-critic-design]]. Caveat: the skill describes itself as dormant-until-activated, so it is not a default gate the way theirverifyskill is. - Bounded retries then a human (practice 2). Ray caps critic loops at 2 ITERATE rounds (fresh-eyes memory).
station-criticescalates to an HQ decision page atmax_iterations. That's the same shape as their three-strikes rule. - Read the evidence before claiming done. [[feedback_no_batched_result_declaration]] and [[feedback_verify_blockers_against_source_not_notes]] are the session-level version of "review evidence, not assertions."
- Cheap deterministic checks first (practice 6, tail).
audit-newsletter-outputs.pyruns I-checks (I3source:, I11 Sponsorship header) on 06-reference notes before the semantic/self-reviewpass. - Misses become rules (practice 9, loosely). Founder corrections become
feedback_*memory files, and/improvediarizes mediocre outputs into skill edits.
What Ray does not do yet (worth adopting, in priority order):
- Test the critics against an answer key (practice 8). None of the source-aware critics (
verify-vault-write,verify-strategic-output,verify-dispatch,design-critic) has a held-out eval. Onlybehavior-criticships fixtures, and those are four synthetic files (test-fixtures/broken_cli.shetc.). Ray already has the incident corpus: every founder-caught miss in the memory folder (the MAC video frames, the false Codex-blocker, the fabricated A2A failure) is a real defect with a one-sentence answer key. Change: build~/.claude/skills/verify-*/evals/with ~10 past misses plus ~10 clean controls, and score recall and over-flagging before any critic-rubric edit lands. This also gives [[2026-06-14-open-ended-output-verifier]] the eval set it was missing. - Make lessons trigger-indexed, not recall-indexed (practices 4 and 9). Ray's lessons live as prose
feedback_*memories that load by relevance and index title. Their checklist items are trigger / check / origin-incident triples that the reviewer matches against the diff. Change: convert the highest-frequency memories (em-dash count,source:key, Sponsorship header, iMessage plain text,datebefore time claims) into a checklist with a mechanical trigger, so critics run the 3-10 matching items instead of hoping the right memory surfaces. Add the pruning pass they describe. The MEMORY.md index is already long. - Stamp how each artifact was verified, with an observe-mode gate (practices 6 and 7). Today nothing records whether a vault write, PR or founder-facing message passed a critic. Settings has only
PreCompact,SessionStartandPreToolUsehooks. The 2026-07-28 Codex/Grok SOP shipped to the founder without a critic, and he had to ask ([[2026-07-28-external-model-delegation-codex-grok]]). Change: add averified_by:frontmatter field (critic name, ornone) and a hook that logs unverified 06-reference/02-sops writes in observe mode for a couple of weeks before enforcing. Related:/improveedits the same skill rubrics that judge its output, which is the "change modifies its own check" smell from practice 6. It's worth checking whether [[2026-05-26-improve-validation-gate]] actually pins the rubric version it evaluates against.
Where the article pushes back on us. Practice 5 ("one reviewer, many rounds; stacking reviewers is often worse") sits uneasily with station-critic's fan-out of one subagent per axis and the vault's enthusiasm for "council of Claudes" ([[2026-09-09-every-writers-who-use-ai]]). Their claim is from code review, so it may not transfer, but it is a testable claim once item 1 exists.
Open questions
- If we built the critic eval set, what is
verify-vault-write's actual recall on founder-caught defects? We have never measured it. - Is per-axis fan-out in
station-criticbuying recall, or just cost and noise? Their evidence says a single tuned reviewer wins. - Does the "two blocks then pass" escape hatch belong in Ray's cron-mode gates, where nobody is watching to unstick a blocked loop?
Related
- [[2026-05-19-verification-as-independent-worker-pattern]]
- [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]]
- [[2026-07-26-autoreview-skill-teardown-second-model-critic-design]]
- [[2026-05-12-zach-lloyd-warp-verify-then-build-test-harness-agentic-coding]]
- [[2026-04-20-every-ai-autopilot-verification-decay]]
- [[2026-06-14-open-ended-output-verifier]]
- [[feedback_fresh_eyes_subagent_for_own_artifacts]]
- [[feedback_no_batched_result_declaration]]