/improve cron - design spec for monthly verify-* axis sample-and-regrade

The question

Set up a /improve cron - monthly sample-and-regrade pass over each verify-* axis (10% of PASS verdicts re-graded by founder; revert axis to assisted status if disagreement >15%). Critic drift documented at 500-1000-example interval in production. Folds into existing scheduled-jobs.txt rhythm. (Source: deep-research-derivative from 2026-05-22-reward-hacking-patterns-llm-critic-systems.md. Priority: High, auto-promoted 2026-05-23 at 12/15.)

What we already know (from the vault)

• [[2026-05-22-reward-hacking-patterns-llm-critic-systems]] - PARENT brief. Established the need for a /improve cadence (currently only a trigger). Recommended 10% sample of prior month's PASS verdicts; >15% disagreement reverts axis to "assisted" status. Also named the RGFMD adversarial self-generation pattern as quarterly add-on.
• [[2026-05-19-cai-critic-graduation-per-axis-threshold]] - Established the per-axis CAI graduation thresholds. The /improve cron is the de-graduation mechanism that mirrors the graduation gate.
• [[2026-05-12-rdco-pipeline-rlhf-shaped]] - The pipeline IS RLHF. Each verify-* axis is a critic; /improve is the human-in-loop calibration that keeps the critic anchored to founder taste.
• [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]] - Verify-* skills now use two-gate PASS/FAIL (replaced 3-state PASS/ITERATE/SCRAP). /improve regrade should respect the two-gate architecture: founder grades binary PASS/FAIL, agreement-rate is binary-agreement-rate (Cohen's kappa for binary judgments).
• [[2026-05-19-verification-as-independent-worker-pattern]] (SOP) - Names verify-vault-write, verify-strategic-output, verify-dispatch as the canonical verify-* skill family. Initial /improve coverage scope.
• [[2026-05-12-multi-agent-pipeline-config-schema]] (SOP) - Per-axis YAML schema. /improve outputs should write back into this schema (axis status, last-calibration date, last-kappa, founder-override count).
• [[2026-04-14-commentary-tan-fat-skills-thin-harness]] - "Self-rewriting skills - /improve skill exists. Currently interactive, not fully autonomous. Opportunity to push further." The cron is the autonomy step.

What the web says

• Industry-converged calibration cadence is monthly. "Run the production judge over the gold-set monthly thereafter (verify the agreement holds). Without this discipline, judges drift in 60-90 days" (Future AGI 2026 LLM-as-Judge Best Practices). The parent brief's "monthly cron" call was guessed correctly against this benchmark.

• Recommended sampling rate is 5-20% of production traces + 100% of errors + low-score outputs + outlier-cost traces. "Marginal coverage past 20% rarely justifies the cost." Cost target: judge expenses under 10-15% of production LLM expenditures. The parent brief's 10% number sits in the middle of the recommended band.

• Agreement-rate metric should be Cohen's kappa, not raw % agreement. Kappa corrects for chance agreement (critical when most verdicts are PASS in a healthy axis - raw agreement looks artificially high). Recommended threshold: kappa >= 0.6. "Some rubrics tolerate lower, some require higher." Critically, kappa = 0.6 is roughly equivalent to ~80% raw agreement on a balanced PASS/FAIL distribution but only ~95% raw agreement on a 90/10 PASS-skewed distribution. The parent brief's "15% disagreement" threshold needs to be re-expressed in kappa terms to be axis-distribution-robust.

• Drift detection at 60-90 days even with no apparent issues. Anthropic + Future AGI guidance: cadence is non-negotiable BEFORE drift visibly hurts production. The cron should fire even when no failures are reported in the prior month.

• Stealthy preference drift (RIPD) - rubric edits that pass benchmark validation can still systematically shift judge preferences on target domains. (arXiv 2602.13576, Rubrics as an Attack Surface). Implication for /improve: when founder edits a rubric mid-month, the next /improve cycle must re-baseline kappa rather than treating the post-edit period as continuous with the pre-edit period.

• Gold-set maintenance: 200-500 entries recommended regardless of production volume. RDCO is well under this scale per axis - probably 20-100 verdicts/month per axis after launch. The small-N reality is that monthly kappa estimates will have wide confidence intervals; founder-override events should be treated as the primary drift signal, with kappa as confirmation.

Convergences and contradictions

Convergence: the parent brief's design parameters (monthly cadence, 10% sample, threshold-based reversion) align with industry-standard 2026 LLM-as-judge best practices. The design needs no major rework; the implementation needs refinement on metric choice (kappa) and the small-N reality (founder-override as primary signal).

Sharp contradiction with parent brief's "15% disagreement" framing: raw % disagreement is the wrong metric on PASS-skewed binary verdicts. A 95% PASS axis with 5% true FAIL has ~95% baseline raw agreement just from base-rate luck; "15% disagreement" becomes either trivially-met (axis dies on noise) or unreachable (no drift detected). Cohen's kappa fixes this. New recommendation: kappa < 0.6 OR three consecutive months of single-instance founder override = revert to assisted.

Convergence-but-amplified: the parent brief recommended quarterly RGFMD adversarial self-generation as separate. The web data on stealthy preference drift suggests this should be tightened to monthly co-fire with /improve - the regrade catches drift after the fact; RGFMD catches blind spots before the policy finds them. Combined, they cover both failure surfaces in one cron pass.

No contradiction with existing schema work: the two-gate PASS/FAIL architecture (from 2026-05-20 SOP) is compatible with kappa scoring. Binary kappa formula works directly on PASS/FAIL pairs.

Synthesis for RDCO

Concrete design spec for /improve cron:

schedule: monthly, 1st of month, 09:00 ET (Sunday-safe)
cron_line: "0 9 1 * * /Users/ray/.claude/scripts/run-improve-cron.sh"

per_axis_pass:
  1. enumerate PASS verdicts from prior month (read from per-axis log file)
  2. if N_verdicts < 10: skip axis (insufficient signal); log "below-threshold"
  3. sample = max(5, ceil(0.10 * N_verdicts))  # ensure minimum 5 samples
  4. queue sample for founder review (Notion task: "/improve regrade Q for axis X")
  5. RGFMD step: prompt critic to generate 3 artifacts that would PASS its rubric but should semantically FAIL. Append to /improve task notes.

per_axis_signal_aggregation (after founder responds):
  - compute Cohen's kappa on PASS/FAIL agreement
  - count single-instance founder overrides
  - emit revert signal if kappa < 0.6 OR (overrides >= 1 for 3 consecutive months)

revert_action:
  - update per-axis YAML: status: graduated -> assisted
  - notify founder via iMessage (this IS a decision, not just status)
  - log to vault: ~/rdco-vault/01-projects/self-review/improve-cron-log.md

output_format:
  - per axis: name, N_verdicts, N_sampled, kappa, override_count, status, action
  - aggregate: total review burden (minutes), axes-at-risk count

Initial axis coverage (3 axes from verify- stack):*

• verify-vault-write
• verify-strategic-output
• verify-dispatch

Add as new verify-* skills graduate. Out-of-scope for v1: pipeline-critic seat in build-haiku-generator (different shape - test/code review, not artifact verification).

Review-burden budget: 10% of low-volume axes at <50 verdicts/month = <5 regrades per axis. Three axes = <15 founder regrades per month = ~20-30 minutes founder time. RGFMD adds ~10 min per axis = ~30 min. Total: ~1 hour/month founder time. This is the calibration tax; it is non-negotiable per Future AGI 2026 guidance.

Execution path - smallest-possible build:

1. Create /Users/ray/.claude/skills/improve-cron/SKILL.md with the above design.
2. Wrapper script at ~/.claude/scripts/run-improve-cron.sh that triggers the skill in non-interactive mode + drops the regrade task on the Notion board.
3. Append cron line to ~/.claude/scripts/scheduled-jobs.txt.
4. First scheduled run: 2026-06-01 09:00 ET (8 days from now). Initial sample will be small (May 22 onwards = ~10 days of verdicts); manually adjust sample target downward for first month.
5. Self-bootstrapping: the skill itself becomes a verify-* axis candidate after 3 months of clean operation (does /improve grade its own grading? recurse with discipline).

Strategic implication beyond the immediate build: /improve cron is the first piece of RDCO infrastructure that explicitly closes the autonomous-but-calibrated loop. Self-rewriting skills + cadenced human-in-loop check IS the pattern Tan's "Thin Harness, Fat Skills" gestures at. The implementation should be designed to generalize: when MAC ships and clients deploy their own verify-* equivalents, /improve cron becomes the operational template they need. This is content + product, not just internal hygiene.

Open follow-ups

• Should /improve cron consume the RGFMD output as gold-set additions? (Adversarial-self-generated artifacts that founder confirms as semantically-FAIL become gold-set negative examples for the rubric. Compounds.)
• What's the right backoff when an axis reverts to assisted - does it require 3 clean weeks before re-graduating, or do we re-graduate on founder approval after the underlying rubric edit ships?
• Should /improve cron also evaluate the inverse - founder PASS that the critic SCRAP'd? Asymmetric over-restriction is a softer drift but still costs founder time on appeals.
• Cost-bound consideration: at 30-min founder time/month, this is acceptable. If verify-* family grows past 6-8 axes, the linear scaling becomes prohibitive. Should /improve adopt the same 5-20% sample rate at the axis-aggregation layer (sample 3 of 8 axes per month) once the family is large?
• Sanity Check angle: "We built a critic for our critics" - the meta-calibration story is publishable once the cron has 3 months of data showing it caught a real drift.

Sources

Vault:

• ~/rdco-vault/06-reference/research/2026-05-22-reward-hacking-patterns-llm-critic-systems.md
• ~/rdco-vault/06-reference/research/2026-05-19-cai-critic-graduation-per-axis-threshold.md
• ~/rdco-vault/06-reference/concepts/2026-05-12-rdco-pipeline-rlhf-shaped.md
• ~/rdco-vault/02-sops/2026-05-20-verify-stack-two-gate-pass-fail-architecture.md
• ~/rdco-vault/02-sops/2026-05-19-verification-as-independent-worker-pattern.md
• ~/rdco-vault/02-sops/2026-05-12-multi-agent-pipeline-config-schema.md
• ~/rdco-vault/06-reference/commentary-tan-fat-skills-thin-harness-2026-04-14.md

Web:

• https://futureagi.com/blog/llm-as-judge-best-practices-2026
• https://arxiv.org/abs/2602.13576 (Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges)
• https://stackpulsar.com/blog/llm-model-drift-detection/
• https://calmops.com/testing/llm-as-judge-testing-guide-2026/
• https://deepchecks.com/llm-judge-calibration-automated-issues/