02-sops

verify stack two gate pass fail architecture

2026-05-20·sop·status: active
verify-stackfresh-eyes-patternverdict-shapepass-failiterate-scraplangfuse-evalsbinary-over-gradedarchitecture-decision

Verify-* stack — two-gate pass/fail architecture (replaces 3-state PASS/ITERATE/SCRAP)

What this changes

The verify-* fresh-eyes critic stack has shipped with a 3-state verdict surface (PASS | ITERATE | SCRAP) across every skill in the family. The Lotte Verheyden Langfuse-Academy post on AI eval rubrics ([[../06-reference/2026-05-20-lotte-verheyden-evals-explained-langfuse-academy]]) recommended binary over graded scales — surfacing the question of whether 3-state is graded-scale ambiguity in disguise.

Founder's reframe (2026-05-20 08:36 ET): two gates, not three states.

This is mechanically equivalent to the prior 3-state surface (same 3 outcomes) but sharper as a decision tree. The cognitive load on the critic shrinks: it's making one binary call, then conditionally a second. The signal to the parent (and to the founder) is therefore cleaner.

The new verdict definitions

Why ITERATE-vs-SCRAP at the second gate is mechanically useful

The choice between iterate and scrap is itself a real decision. They produce different downstream actions:

Confusing these is expensive in both directions. Iterating on a dead-end artifact wastes cycles AND produces calibration drift (the next critic sees "we kept iterating, must be close" when actually the artifact was always wrong). Scrapping a near-correct artifact wastes the prior work for no gain.

Keeping the iterate/scrap call inside the verify-* skills (rather than punting to the parent) preserves the fresh-eyes principle — the critic that just read the artifact with zero context is the right place to make the keep-vs-discard call.

What this means for the 6 affected skill files

Each of these skills currently emits PASS / ITERATE / SCRAP:

Update they each need (when this SOP rolls out):

  1. Reframe the verdict section as two-gate, not three-state. Sample wording:

    "Emit a verdict in two gates. Gate 1: PASS or FAIL. If FAIL, then Gate 2: ITERATE (close, needs cleanup) or SCRAP (dead end / fresh-eyes reset)."

  2. Output shape: PASS | FAIL → ITERATE | FAIL → SCRAP. Still 3 distinct outcomes for parent-side branching, but written as the two-gate decision tree the critic walked.
  3. The per-critic prompt scaffolding stays mostly identical — only the "now decide the verdict" section reframes.

Rollout discipline: apply the same edit across all 6 skills as a single batch so the verify-* surface stays consistent. Don't half-migrate. Either all 6 use two-gate by end of the rollout, or none do.

Calibration against Lotte's "manual-first" warning

The other Langfuse-rubric thread Lotte names — manual-first review is the prerequisite for trusting automated evaluators — still applies. The two-gate reframe sharpens the verdict shape but doesn't address the calibration-drift risk where the critic-skills themselves start measuring things that don't matter. That's tracked separately as the "missing manual-tier calibration loop" finding in the Lotte note's mapping section.

Short version: founder eyeballs N critic outputs/week to confirm they're catching real defects. Two-gate-verdict-shape is independent of that calibration practice; both need to ship.

Audit-newsletter-outputs.py is the deterministic-eval tier — already correctly placed

The verify-* stack is the LLM-as-judge tier of Lotte's eval rubric. The audit-newsletter-outputs.py script (13 invariant checks, zero LLM calls) is the code-based eval tier. Both tiers shipping together is structurally what Lotte recommends; the two-gate verdict-shape decision only affects the LLM-as-judge tier. The audit script's pass/fail-with-violation-count output is already binary-shaped — no change needed there.

Forward thread — Karpathy-style autoresearch optimizer (founder bookmark 2026-05-20 08:36 ET)

Founder flagged that the recursive-self-improvement loop for the verify-* stack itself might be a candidate for the Karpathy autoresearch / open-source optimizer pattern that's been gaining traction. The framing would be: treat "the verify-* stack as a system that should improve itself over time using its own outputs as training signal."

This is a longer-arc thought, not an immediate task. Worth tracking. Conditions to revisit:

Not now. Bookmarked.

Related