Proposal: add a validation gate to /improve
The problem (from SkillOpt)
/improve autonomous today reads the review-log + improve-queue, classifies findings low-risk vs structural, and applies low-risk edits silently — with no check that an edit actually improves the skill. SkillOpt's lesson #1: a self-editing loop that accepts most of what it proposes is shipping slop; their best skills land only 1–4 accepted edits total, each gated on a strict held-out improvement (ties rejected). We have no gate, no edit budget, and only-additive edits (skills bloat).
The RDCO adaptation (the key move)
SkillOpt assumes a per-skill auto-grader / numeric benchmark. RDCO mostly lacks that — our skills emit qualitative outputs (content, design, strategy, vault notes). But we already have the thing the paper calls the next frontier: a verifier for open-ended work — the fresh-eyes critic subagents (verify-vault-write, verify-pdf-output, design-critic, verify-strategic-output) + deterministic audit scripts (audit-newsletter-outputs.py, rdco-doctor.py). So:
RDCO's validation gate = a fresh-eyes critic as the reward model, plus a deterministic audit where one exists. Not "auto-grade numerically" — "have an independent worker judge whether the edit is an improvement, blind."
This is the same independent-worker pattern we already codified ([[feedback_verification_independent_worker_pattern]]); SkillOpt says it's the move that wins.
The design — 4 mechanisms
1. Critic gate (universal, the core). /improve no longer applies a skill edit silently. Each proposed edit goes to a fresh-eyes verify-skill-edit critic subagent (zero context on why the edit was proposed) that returns ACCEPT/REJECT on four axes:
- strictly improves clarity/correctness (not lateral churn),
- within the bounded-edit budget,
- respects protected sections,
description:(router surface) still matches the body (agent surface) — SkillOpt lesson 6b. Apply only on ACCEPT; otherwise drop or hold for founder. Confirmation bias is why the proposer can't be the judge.
2. Deterministic before/after where a harness exists (stronger). If the edited skill has an audit/eval script (process-newsletter → audit-newsletter-outputs.py), run it on a fixed sample corpus before + after the edit and keep the edit ONLY if pass-rate strictly improves or holds with strictly-fewer violations. Doable today for process-newsletter; extends to any skill we later give an audit.
3. Bounded-edit budget (all runs). Cap /improve to ~4–8 accepted edits per run; reject ties; log the accepted-edit count — a run that accepts many edits is itself a red flag (textual learning-rate). Allow net-negative diffs (compaction) — currently /improve only adds; the ~920-token-median lesson says trimming bloat is a valid improvement.
4. Protected-section invariant (structural, not by-convention). /improve must never edit CLAUDE.md hard-rules or the working-context Errors-&-Corrections ledger (slow state). SkillOpt: removing this invariant cost them 22 pts. We have it by convention + classifier block; make it explicit in the skill so a fast edit can't clobber a slow lesson.
Feasibility tiering
| Increment | Effort | Ship? |
|---|---|---|
| #3 bounded-edit budget + #4 protected-section invariant (doc-level rules in improve SKILL.md) | low | first |
| #2 process-newsletter before/after audit gate (harness already exists) | low-med | first |
#1 verify-skill-edit critic subagent (new skill, modeled on existing verify-*) |
med | second |
| Extend #2 to more skills (build per-skill audits) | high, ongoing | later |
Recommended first increment
Ship #3 + #4 + #2 together (all low/low-med, all use existing infra): a bounded-edit budget, an explicit protected-section invariant, and a real before/after audit gate for the one skill that already has an audit harness. Then build the verify-skill-edit critic (#1) as the universal gate for skills without a deterministic harness.
Open questions for founder
- Greenlight the first increment (#3+#4+#2), or want the full thing (incl. the critic) specced before any edit?
- Bounded-edit cap: 4–8/run (SkillOpt's number) or tighter to start?
- The
verify-skill-editcritic is a new skill — fine to add, or fold the check into the existing/improveflow without a separate skill file?
(Implementation touches ~/.claude/skills/improve/SKILL.md — self-mod, needs explicit founder OK to edit, per the 2026-05-26 classifier block.)
Related
- [[../../06-reference/2026-05-26-skillopt-self-evolving-agent-skills]] — the source paper + RDCO mapping
- [[feedback_verification_independent_worker_pattern]] / [[feedback_fresh_eyes_subagent_for_own_artifacts]]
- [[../../06-reference/2026-04-11-garry-tan-thin-harness-fat-skills]]