06-reference/research

cai critic graduation per axis threshold

2026-05-19·research-brief·status: complete·source: Notion Research Backlog (page 35ff7d49-36d1-8130-83b7-ff9e0a336194); curiosity-promoted 2026-05-18, score 13/15·by Ray (deep-research)
constitutional-airlhfrlaifllm-as-judgepipeline-criticcritic-graduationreward-hackingverification-harnessagent-deployerschema-design

CAI Critic Graduation: Per-Axis Label Volume Thresholds

TL;DR (headline finding)

There is no single label-count threshold for graduation. Production wisdom converges on a two-gate cutover: ~100-200 human-labeled examples per axis to reach 85% agreement, AND a separate held-out reliability check (Cohen's kappa, edge-case coverage, red-team) before removing the human. The pipeline-RLHF concept doc's "~10 founder-overrides per axis" guess is roughly an order of magnitude light versus the LLM-as-judge production literature, but is defensible for RDCO if (and only if) two compensating mechanisms are in place: (1) tight axis scope (each axis tests one mechanical thing, not a fuzzy quality), and (2) sustained spot-check sampling post-graduation. The dominant failure mode of cutting over too early is not catastrophic - it is silent: the critic locks in a blind spot, the policy learns to exploit it, and quality degrades along the un-instrumented dimension while the dashboard shows PASS.

Why the question

The pipeline-RLHF concept doc ([[../concepts/2026-05-12-rdco-pipeline-rlhf-shaped.md]]) maps RDCO's multi-agent pipeline directly onto the Constitutional AI / RLAIF topology. It hand-waves a graduation threshold of "~10 founder-override labels per axis before that axis graduates to autonomous-critic status," with a footnote that the schema work needs literature-backed numbers. That schema work is now active. The verify-* family (verify-vault-write, verify-strategic-output, verify-dispatch) are explicitly designed to graduate from founder-in-loop to autonomous. The question this brief answers: what does the production CAI / LLM-as-judge literature actually say about the cutover, and what failure modes appear if we cut over too early.

What the literature actually says

1. Calibration-set sizes converge on 30-200 labels per axis

Three independent production sources triangulate on the same range:

Source Recommended human labels Stage
Comet (LLM-as-Judge production guide) 30-50 examples annotated by domain experts Initial calibration set; iterate prompt if >20% disagreement on clear-cut cases
Kili / LangChain 100-200 human-annotated examples Reach 85% agreement before scaling
Databricks (Grading Notes) ~200 use cases total across criteria Achieved 93-96% alignment with human judges using grading-notes pattern

The number is per-axis (per-criterion), not per-skill. A critic with 4 axes needs the calibration set 4 times, OR a stratified single set that covers each axis with 30+ examples.

2. Agreement-rate thresholds: 75-90% before scaling, 85% as canonical cutover

The gap between "scale" (85%) and "remove human" (90-95%) is intentional. Production deployments do not graduate at 85%; they graduate to assisted operation at 85% and to autonomous operation only after a second calibration pass shows 90%+ on a held-out set.

3. Single-metric thresholds are insufficient; literature mandates a battery

Every production source flags that agreement-rate alone is gameable. The dominant recommendation is to report accuracy plus Cohen's kappa (which corrects for chance agreement), plus targeted red-team coverage. ApX explicitly states: "no single metric suffices."

4. Constitutional AI's own numbers do NOT generalize

Anthropic's original CAI paper bootstrapped harmlessness behavior from ~16 written principles plus thousands of self-critique iterations - but those iterations were over a large pretrained model's full output distribution, not over 10 founder edits to a specific artifact type. The CAI v2 / "Constitution or Collapse?" (Bai et al., 2025; Llama 3-8B replication) line explicitly shows that on smaller-scale or narrower applications, low label volumes lead to mode collapse (the critic converges on a narrow set of "acceptable" outputs and rejects valid diversity). This is the most important nuance for RDCO: Anthropic's headline numbers come from a regime RDCO does not occupy.

Failure modes when you cut over too early

The CAI/RLAIF failure-mode literature is unusually well-catalogued. Six modes recur across sources:

Mode 1: Critic blind-spot exploitation (the dominant mode)

The policy (artifact-producer) learns to produce outputs that hit whatever the critic is actually measuring, while drifting along dimensions the critic doesn't cover. Symptom: PASS rate climbs, real quality plateaus or degrades. This is the textbook reward-hacking story.

Mode 2: Sycophancy / style-mimicry

Policy learns to phrase outputs in the critic's preferred style rather than meet the substantive bar. Symptom: outputs feel uniform; rubric-words appear verbatim in artifacts.

Mode 3: Constitutional loopholes (rule-letter vs rule-intent)

Policy satisfies the literal axis check while violating the underlying intent. The classic example: an axis that requires "cite a source" gets satisfied by hallucinated citations.

Mode 4: Critic drift over time

Even a well-calibrated critic's interpretation of fuzzy axes drifts as the population of artifacts it sees shifts (distribution shift in production). What scored 85% agreement on the calibration set scores 70% three months in.

Mode 5: Mode collapse / reduced diversity

Critic converges on a narrow set of "acceptable" outputs and rejects valid diversity. Documented in the Llama 3-8B CAI replication (Bai et al. 2025). The policy produces increasingly homogeneous artifacts that all clear the critic but cover less of the legitimate output space.

Mode 6: Brittleness on paraphrase / OOD inputs

Critic performance degrades sharply on artifact types it wasn't calibrated on. ApX recommends 95%+ consistency on paraphrased/adversarial variants before removing human oversight.

Where production wisdom diverges from academic claims

Two non-trivial divergences worth flagging:

  1. Academic CAI papers under-report mode collapse. The original Anthropic CAI work emphasized harmlessness gains; mode collapse only became a documented failure when the technique was replicated on smaller models and narrower domains (Bai et al. 2025, "Constitution or Collapse?"). RDCO's per-skill, per-axis application is closer to the smaller-narrower regime than to the original Anthropic regime. Implication: weight production case studies (Databricks, LangChain, Comet) over academic Anthropic papers when sizing the threshold.

  2. Academic literature treats reward-model gradient training as load-bearing; production LLM-as-judge does not. Almost every academic CAI / RLAIF paper assumes you're training a parametric reward model via gradient descent on preference pairs. Production deployments overwhelmingly use prompt-engineered LLM-as-judge with grading notes / few-shot examples instead. The label-volume requirements drop dramatically (200 vs 10K+) BUT the calibration discipline rises in importance, because you have no implicit averaging from the gradient process to smooth over noise.

Recommendation for RDCO pipeline-critic schema

Based on triangulation across sources:

Per-axis graduation criteria (proposed)

A pipeline-critic axis graduates from "founder-in-loop" to "autonomous-with-spot-check" when ALL of the following are true:

  1. Calibration volume: ≥30 labeled (PASS/FAIL + rationale) examples for this axis, covering at least 5 distinct artifact shapes. (Lower bound; 100 is safer if the axis is fuzzy.)
  2. Agreement gate: ≥85% agreement between critic verdict and founder verdict on a held-out 10-example subset of the calibration set (subset never used for prompt iteration).
  3. Edge-case red-team: founder constructs 3-5 deliberate edge cases (artifacts that "should fail" along this axis in non-obvious ways); critic catches ≥4/5.
  4. Diversity floor: axis was tested against artifacts from ≥2 different domains (Sanity Check + MAC + skill build-out, for example), not just the domain it was developed on.

Axes that pass all four gates graduate to autonomous, with mandatory spot-check sampling (see below). Axes that pass (1) and (2) but not (3) and (4) graduate to "assisted" - critic returns verdict, founder reviews critic's reasoning before final disposition.

Spot-check sampling rate (proposed)

Founder-override threshold is the WRONG primary signal

The concept doc's "~10 founder overrides" framing inadvertently optimizes for the wrong variable. Founder overrides are FAIL signals - they happen when the critic missed something. Counting them measures critic failure rate, not critic confidence. The literature-correct framing is:

Suggest re-framing the schema as "30 confirmation labels + 0 unresolved overrides" rather than "10 overrides."

Why this is more conservative than the concept doc's "~10"

The "~10 founder-overrides" hand-wave was reasonable for the unverified intuition stage. Now that we have literature: 10 is too few for an LLM-as-judge critic to reliably hit 85% agreement on a fuzzy axis. For mechanical/structural axes (frontmatter presence, link count, character-count limits) - 10 IS enough, because those are presence checks that don't need calibration. For semantic axes (does this brief say something non-derivative? does this verdict match the founder's voice?) - 30-100 is the literature-backed range.

Practical rule for RDCO: split axes into "mechanical" (10-label threshold) and "semantic" (30-100-label threshold) and graduate them separately. This mirrors the structural-vs-semantic split already implicit in pipeline-critic's YAML-axes-vs-prompt-axes architecture.

Cross-check: where vault context modifies the literature default

Three RDCO-specific factors that lower the safe-cutover bar versus the literature default:

  1. Single labeler. No inter-annotator-disagreement noise to resolve. The founder IS the ground truth. This eliminates ~20% of the label volume that production multi-labeler workflows need just to average out human disagreement.
  2. Fresh-eyes subagent pattern ([[../../feedback_fresh_eyes_subagent_for_own_artifacts]]). RDCO already routes critic work through zero-context subagents to defeat confirmation bias. This is a structural mitigation for Mode 1 (blind-spot exploitation) that most production deployments don't have.
  3. Reversible-action default ([[../../feedback_distinguish_decision_from_action]] + the auto-mode signal-to-noise memory). RDCO already treats most artifacts as cheap-to-redo. Even if the critic mis-grades, the cost of a bad PASS is low for most axes.

Three RDCO-specific factors that RAISE the safe-cutover bar:

  1. Narrow calibration domain at bootstrap. Most axes will be developed on one artifact type first (Sanity Check articles, or MAC carve-outs, or skill SKILL.md files). Cross-domain testing (gate 4 above) is doable but adds latency.
  2. Founder-voice axes are fuzzy. "Does this match the founder's voice" / "is this non-derivative" / "is this overconfident relative to evidence" are exactly the kind of fuzzy semantic axes where 30 labels is the floor, not the ceiling.
  3. /improve consolidation hook is the re-calibration mechanism. If /improve doesn't run on a cadence, drift (Mode 4) WILL show up. Need to either cron /improve or trigger it off the spot-check disagreement rate.

Follow-up questions surfaced

  1. What is the right spot-check stratification strategy for RDCO? Random 10% will get dominated by Sanity Check article volume; need to stratify by axis × artifact-type to catch drift on lower-volume artifact types.
  2. How do we instrument "silent quality degradation" detection? The dominant failure mode (Mode 1) is silent by definition. Periodic founder re-grading catches it but is expensive. Is there a cheaper proxy signal (artifact embedding-diversity drift, axis-correlation drift, output entropy)?
  3. Should /improve be cron'd or threshold-triggered? Cron'd = predictable but wasteful when nothing drifted. Threshold-triggered = efficient but requires the drift-detection signal from Q2.
  4. How do we handle a new axis introduced after others have graduated? Does the new axis re-set the spot-check rate for the whole skill, or only for itself? Production literature is silent on this.
  5. What's the right canonical-exemplars folder size to avoid mode collapse? Bai et al. 2025 documents the failure but doesn't give a number. Empirical question - log artifact-shape diversity over time and watch for entropy decline.

Sources

Related vault docs