"SkillOpt: Executive Strategy for Self-Evolving Agent Skills" — paper + @koylanai breakdown

Why this is in the vault

This is the most on-thesis external artifact RDCO has filed on its own architecture: it formalizes "the SKILL.md file is the trainable unit, the harness is secondary" — the exact bet RDCO runs (fat skills, /improve self-edit loop, verify-* critics). It both validates our direction AND names concrete places our current loop is sloppier than the method. Founder shared both links cold 2026-05-26.

The core argument (SkillOpt)

Treat one skill document as the external "weights" of a frozen agent and optimize it like a parameter, not by vibes. A separate optimizer model turns scored rollouts into bounded add/delete/replace edits, and accepts an edit ONLY if it strictly improves a held-out validation score (ties rejected). Borrows optimizer discipline: a textual "learning-rate" edit budget, a rejected-edit buffer, epoch-wise slow/meta updates, and a protected-section invariant (fast edits can't overwrite slow lessons). Zero inference-time cost at deploy. Reported best-or-tied across 52 (model × benchmark × harness) cells, beating human / one-shot / TextGrad / GEPA / EvoSkill; +19.1 pts inside Claude Code (named as a harness); optimized skills transfer across model scales and between Codex ↔ Claude Code.

Transferable lessons (Koylan's distillation, paraphrased)

1. The validation gate is the whole game. Held-out set, strict improvement, ties rejected. Best skills land with only 1–4 accepted edits total. If a self-improving loop accepts most of what it proposes, it's shipping slop.
2. Bounded edits beat full rewrites (~4–8 edits/step). Remove the budget and performance collapses — the textual analog of learning rate. Cap the diff on any LLM-as-author loop.
3. Compactness wins. Median final skill ≈ 920 tokens. Length feels like effort but isn't; high-signal > long.
4. Skill > harness. Procedural knowledge generalizes across runtimes (a Codex-trained skill ported to Claude Code jumped +59.7 on SpreadsheetBench).
5. Frozen model + trained context is the practical adaptation path for anyone not training their own model (small model + optimized skill ≈ frontier on procedural tasks).
6. Verification is the bottleneck. Every gate here leans on an auto-grader — works for benchmarks, fails for writing/design/strategy. "Whoever builds the verifier for open-ended tasks owns the next stage." Plus: description vs body are two surfaces (router sees description, agent sees body — they can silently disagree); aggregate accuracy hides per-skill effect sizes (one description rewrite moved a single skill 23–25pp while the corpus average barely moved).

Mapping against Ray Data Co

Validates:

• The verify- critic pattern* (verify-vault-write / verify-pdf-output / design-critic / fresh-eyes subagents) is exactly lesson 6's "build the verifier" — and the paper frames that as the frontier, not a side-quest. RDCO is early on the thing Koylan says wins the next stage.
• Skills-as-the-unit + frozen-model-trained-context is RDCO's whole posture (Tan "thin harness, fat skills"). Confirmed, with mechanism.

Critiques (the useful part):

• /improve autonomous has no validation gate. It "applies low-risk fixes silently" — i.e. accepts most of what it proposes, the precise failure mode lesson 1 names. Today there's no held-out check that an edit strictly improves anything. This is the sharpest actionable gap.
• No bounded-edit budget. /improve can apply many edits per run; the paper says cap to ~4–8 and reject ties.
• Our SKILL.md files are bloated. Many run 200+ lines / well past 920 tokens (process-newsletter, process-youtube, check-board). Length has felt like rigor; the paper says it's the opposite. Compaction pass candidate.
• We score skills by vibes, not per-skill effect size. /self-review is qualitative; lesson 6b says per-skill deltas are where the signal is — but RDCO's open-ended outputs (content, design, strategy) lack the auto-grader the method depends on. That's the adoption blocker AND the moat opportunity (build the open-ended verifier).
• Protected-section invariant ≈ CLAUDE.md hard-rules + the working-context Errors-&-Corrections ledger (slow state that fast edits must not clobber). We have it by convention; the paper makes it structural and shows removing it cost 22 pts.

Honest caveat: preprint, not peer-reviewed; the whole method assumes a scored benchmark per skill, which RDCO's qualitative skills mostly don't have — so adoption means building eval harnesses first, not bolting SkillOpt on. Numbers are the authors' own.

Related

• [[../../.claude/skills/improve/SKILL.md]] — the loop that most needs the validation-gate + bounded-edit discipline
• [[2026-05-18-agentway-harness-engineering-claude-code-design-guide]] — the harness-engineering thesis this extends
• [[2026-04-11-garry-tan-thin-harness-fat-skills]] — "fat skills" source; SkillOpt is its optimizer-shaped version
• [[feedback_fresh_eyes_subagent_for_own_artifacts]] / [[feedback_verification_independent_worker_pattern]] — RDCO's verifier work = lesson 6's frontier
• [[2026-05-25-karpathy-llm-coding-failure-modes]] — adjacent "surgical changes / bounded edits" discipline