Which automated verifier is reliable enough to GATE an /improve skill-edit on RDCO's ungraded qualitative skills

The question

What concrete designs exist (or could be built) for an automated reward/verifier for OPEN-ENDED agent outputs (writing, design, strategy) — LLM-as-judge calibration, rubric-grading, pairwise preference — and which is reliable enough to gate /improve skill-edits on RDCO's ungraded skills? (Context: feeds increment 2 of the greenlit /improve validation-gate proposal — SkillOpt's gate assumes an auto-grader RDCO's qualitative skills lack; the open-ended verifier is the named frontier, SkillOpt lesson 6.)

What we already know (from the vault)

• The [[2026-05-26-improve-validation-gate]] proposal already specs the answer in skeleton: RDCO's gate = "a fresh-eyes critic as the reward model, plus a deterministic audit where one exists" — an independent worker judging blind whether the edit improves the skill, NOT numeric auto-grading. Increment 1 (bounded-edit budget + protected-section invariant + process-newsletter before/after audit) ships first; increment 2 is the universal verify-skill-edit critic for skills lacking a deterministic harness.
• [[2026-05-26-skillopt-self-evolving-agent-skills]] is the source: lesson 6 names the open-ended verifier as the frontier ("whoever builds the verifier for open-ended tasks owns the next stage"), confirms RDCO's verify-* family IS that verifier in embryo, and flags the description-vs-body two-surface check the gate must enforce.
• RDCO already runs the fresh-eyes-critic pattern at production scale: [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]] (binary Gate 1 PASS/FAIL, then conditional Gate 2 ITERATE/SCRAP — chosen over a 3-state graded scale precisely because binary is the more reliable judge surface) across verify-vault-write, verify-strategic-output, verify-dispatch, design-critic, video-critic.
• [[2026-05-23-improve-cron-design-spec]] already set RDCO's calibration bar: monthly sample-and-regrade, Cohen's kappa >= 0.6 (not raw % agreement, which is fooled by PASS-skewed distributions), revert an axis to "assisted" on kappa < 0.6 OR repeated founder overrides. Small-N reality means founder-override events are the primary drift signal, kappa is confirmation.
• [[2026-05-22-reward-hacking-patterns-llm-critic-systems]] catalogues the failure modes a gate must survive: critic blind-spot exploitation, sycophancy/style-mimicry, the RGFMD adversarial-self-generation defense, and the "rationale alongside verdict" requirement (Critique-GRPO) that RDCO already implements.

What the web says

• Rubric > holistic, and the rubric matters more than the judge. The 2026 methodology survey is blunt: "a weak judge on a great rubric outperforms a great judge on a weak rubric." Decomposing into explicit criterion-separated dimensions with a form-filling workflow (reason per-criterion, then score) cuts quantization noise far more than swapping in a stronger model (Masood, rubric-based evals, Apr 2026).
• Pairwise beats absolute scoring for reliability because relative judgments are easier and lower-variance — but pairwise loses diagnostic signal (it can't tell you WHICH dimension regressed). Rubric pointwise scoring keeps that diagnostic; the two are complementary, not rivals (Masood 2026).
• Off-the-shelf judges are NOT reliable on specialized domains without calibration. Frontier models exceeded 50% error on bias tests; divergence above ~20-25% vs human labels signals the rubric needs recalibration. An untuned judge validated on generic chat does not transfer (FutureAGI, LLM-as-a-Judge 2026; Adaline).
• The three named biases have concrete controls: position bias → swap-and-average (run both orderings); verbosity bias → length-controlled metric / explicit concision criterion; self-preference bias → cross-family judge. These are cheap and should be defaults (Masood 2026).
• SCOPE (Feb 2026) is the most gate-relevant primitive: selective conformal pairwise judging. It wraps a pairwise judge in conformal abstention — the judge returns a verdict ONLY when confidence clears a calibrated threshold, and conformal prediction gives a finite-sample, user-specified bound on the error rate among accepted judgments. Plain thresholding gives no such guarantee. It also folds in position-bias correction (SCOPE, arXiv 2602.13110). Caveat: needs representative calibration data and may not transfer across model pairs/domains.
• Reference-free reward models for open-ended generation work but are domain-fragile. OpenGenAlign trains a reference-free reward model (no gold answer) scoring hallucination / comprehensiveness / verbosity / attribution; preference data hits 81% human agreement, the trained RM reaches 85.9% on held-out — but off-the-shelf RMs that score ~90% on RewardBench drop below 80% here, and even the trained RM varies significantly by task (OpenGenAlign, arXiv 2501.13264). Training a bespoke RM is a real project, not a prompt.
• Calibration thresholds converge: Cohen's kappa > ~0.6 acceptable, Krippendorff's alpha ~0.8 for high-confidence use; recalibrate monthly or judges drift in 60-90 days (Masood 2026; FutureAGI; matches the vault's improve-cron spec).

Convergences and contradictions

• Convergence: every 2026 source agrees the reliable architecture is rubric-decomposed + bias-controlled + human-calibrated + cadenced — which is exactly the verify-stack RDCO already shipped. RDCO is not behind the frontier on architecture; it is behind on measured calibration (kappa per axis) and on the conformal-style "abstain when unsure" move.
• Contradiction (sharp, and it's the crux): pairwise is the most reliable judging mode, but an /improve gate needs a diagnostic verdict — WHY this edit fails, WHICH protected section it touched, whether description still matches body. Pure pairwise can't say that. So the gate can't be pairwise-only; it must be rubric-pointwise for diagnosis with a pairwise A/B (old skill vs edited skill) for the accept decision.
• Contradiction (reliability ceiling): OpenGenAlign shows even a purpose-trained open-ended reward model tops out ~86% and is domain-fragile. No verifier of open-ended work is reliable enough to be the SOLE autonomous gate at RDCO's stakes. The honest ceiling forces a confidence-gated design, not a blind one.

Synthesis for RDCO

Recommended design for increment 2's verify-skill-edit critic: a rubric-anchored pairwise critic with a conformal-style abstention band. Concretely, three composed parts. (1) A rubric-pointwise pass on the edited skill scoring the four axes the proposal already names — strictly-improves-clarity/correctness (not lateral churn), within edit budget, respects protected sections, description-still-matches-body — each as a binary criterion with a written rationale (Critique-GRPO discipline RDCO already runs). This gives the diagnostic the gate needs. (2) A pairwise A/B between the pre-edit and post-edit skill, run swap-and-averaged across both orderings to kill position bias, deciding "is the edited version better, and is it strictly better (ties rejected, per SkillOpt)." Pairwise is the more reliable accept/reject signal; the rubric pass tells us why. (3) A confidence band: borrow SCOPE's selective-abstention idea — when the pairwise judge isn't confident the edit strictly improves, the critic does NOT auto-accept; it ABSTAINS and routes the edit to the founder rather than guessing. Abstain-to-human is the safe failure mode for a gate; an advisory critic can guess, a gate cannot.

This composes natively with the existing fresh-eyes stack. verify-skill-edit is a sibling of verify-vault-write/design-critic, dispatched with zero context on why the edit was proposed (the proposer has confirmation bias — the entire reason the producer can't be the judge). It inherits the two-gate PASS/FAIL surface and the monthly kappa regrade from the improve-cron, so it is auto-calibrated and auto-de-graduated like every other axis. Crucially, the proposal's increment 1 (process-newsletter before/after deterministic audit) should run AHEAD of the critic wherever a harness exists: a deterministic audit is a real reward, an LLM critic is a proxy reward. Use the proxy only where no deterministic signal exists, and even then prefer it as a tripwire, not a green light.

The calibration bar a GATE needs is strictly higher than an advisory critic's, and this is the load-bearing distinction. An advisory critic (today's verify-* stack, which returns a verdict the parent can weigh) is useful at kappa ~0.6. A gate that silently blocks or accepts an edit with no human in the loop needs both higher agreement AND an abstention escape hatch, because its errors are invisible and compounding — a wrongly-accepted edit poisons the skill for every future run; a wrongly-blocked edit silently stalls improvement. The defensible bar: kappa >= 0.7 on that axis's regrade history before the gate is allowed to auto-act, swap-and-average + cross-family judge as mandatory bias controls, and conformal abstention so the accept-set error stays bounded with the uncertain middle routed to the founder. Below that bar, verify-skill-edit runs in "assisted" mode — it advises, the founder accepts.

On the false-confidence risk — using an LLM to grade open-ended work it might itself be wrong about. This is the real hazard and it cannot be fully eliminated, only bounded. Three structural mitigations, all already in RDCO's toolkit: (a) the critic is a different instantiation/model-family than the producer (self-preference bias control + the "Critical Evaluation of AI Feedback" finding that critic leverage needs asymmetry); (b) RGFMD adversarial self-generation runs quarterly — the critic generates artifacts that would PASS its own rubric but should semantically FAIL, surfacing blind spots before the policy finds them; (c) the gate ABSTAINS rather than asserts when unsure, so the failure mode of "confidently wrong" is structurally converted into "defer to human." The honest conclusion: no open-ended verifier is reliable enough to be a fully autonomous gate at RDCO's stakes today (OpenGenAlign's ~86% ceiling proves it). It IS reliable enough to be a confidence-gated gate — auto-accept the high-confidence clear-improvements and the high-confidence rejects, abstain the uncertain middle to the founder. That is increment 2: ship the rubric-pairwise critic in assisted mode, accumulate a regrade history, and only let it auto-act on axes once their kappa crosses 0.7. Bias toward abstention; an /improve gate that occasionally asks the founder is working as designed, not failing.

Open follow-ups

• Pilot SCOPE-style conformal abstention on ONE existing verify-* axis (verify-vault-write has the most verdict history) — what abstention rate is needed to bound accepted-set error under, say, 5%, and is the resulting coverage high enough to be useful?
• Does RDCO have enough per-axis verdict volume to estimate kappa at the 0.7 gate bar with tight-enough confidence intervals, or is the small-N problem (already flagged in improve-cron) fatal to an auto-acting gate for months?
• Should verify-skill-edit use a deliberately different model family than the /improve proposer to bank the self-preference + asymmetry mitigation, and does that cost/latency pencil out for a background loop?
• Is a tiny bespoke reference-free reward model (OpenGenAlign-shaped) ever worth training for RDCO's highest-volume skill, or does the rubric-pairwise-critic dominate it at RDCO's scale?
• Meta-recursion: when verify-skill-edit itself becomes an /improve target, what gates ITS edits? (The protected-section invariant is the stopgap; the regrade cron is the real answer.)

Related

• [[2026-05-26-improve-validation-gate]]
• [[2026-05-26-skillopt-self-evolving-agent-skills]]
• [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]]
• [[2026-05-23-improve-cron-design-spec]]
• [[2026-05-22-reward-hacking-patterns-llm-critic-systems]]

Sources

Vault:

• ~/rdco-vault/01-projects/skill-improvements/2026-05-26-improve-validation-gate.md
• ~/rdco-vault/06-reference/2026-05-26-skillopt-self-evolving-agent-skills.md
• ~/rdco-vault/02-sops/2026-05-20-verify-stack-two-gate-pass-fail-architecture.md
• ~/rdco-vault/06-reference/research/2026-05-23-improve-cron-design-spec.md
• ~/rdco-vault/06-reference/research/2026-05-22-reward-hacking-patterns-llm-critic-systems.md

Web:

• https://arxiv.org/pdf/2602.13110 (SCOPE: Selective Conformal Optimized Pairwise LLM Judging, Feb 2026)
• https://medium.com/@adnanmasood/rubric-based-evals-llm-as-a-judge-methodologies-and-empirical-validation-in-domain-context-71936b989e80 (Masood, Rubric-Based Evals & LLM-as-a-Judge, Apr 2026)
• https://futureagi.com/blog/llm-as-a-judge/ (FutureAGI, LLM-as-a-Judge in 2026: How It Works, When It Fails)
• https://www.adaline.ai/blog/llm-as-a-judge-reliability-bias (Adaline, Why Frontier Models Fail 50%+ Bias Tests)
• https://arxiv.org/html/2501.13264 (OpenGenAlign: reference-free reward modeling for open-ended long-context generation)