06-reference/research

cai critic graduation per axis threshold

2026-05-19·research-brief·status: complete·source: Notion Research Backlog (page 35ff7d49-36d1-8130-83b7-ff9e0a336194); curiosity-promoted 2026-05-18, score 13/15·by Ray (deep-research)
constitutional-airlhfrlaifllm-as-judgepipeline-criticcritic-graduationreward-hackingverification-harnessagent-deployerschema-design

CAI Critic Graduation: Per-Axis Label Volume Thresholds

TL;DR (headline finding)

There is no single label-count threshold for graduation. Production wisdom converges on a two-gate cutover: ~100-200 human-labeled examples per axis to reach 85% agreement, AND a separate held-out reliability check (Cohen's kappa, edge-case coverage, red-team) before removing the human. The pipeline-RLHF concept doc's "~10 founder-overrides per axis" guess is roughly an order of magnitude light versus the LLM-as-judge production literature, but is defensible for RDCO if (and only if) two compensating mechanisms are in place: (1) tight axis scope (each axis tests one mechanical thing, not a fuzzy quality), and (2) sustained spot-check sampling post-graduation. The dominant failure mode of cutting over too early is not catastrophic - it is silent: the critic locks in a blind spot, the policy learns to exploit it, and quality degrades along the un-instrumented dimension while the dashboard shows PASS.

Why the question

The pipeline-RLHF concept doc ([[../concepts/2026-05-12-rdco-pipeline-rlhf-shaped.md]]) maps RDCO's multi-agent pipeline directly onto the Constitutional AI / RLAIF topology. It hand-waves a graduation threshold of "~10 founder-override labels per axis before that axis graduates to autonomous-critic status," with a footnote that the schema work needs literature-backed numbers. That schema work is now active. The verify-* family (verify-vault-write, verify-strategic-output, verify-dispatch) are explicitly designed to graduate from founder-in-loop to autonomous. The question this brief answers: what does the production CAI / LLM-as-judge literature actually say about the cutover, and what failure modes appear if we cut over too early.

What the literature actually says

1. Calibration-set sizes converge on 30-200 labels per axis

Three independent production sources triangulate on the same range:

Source Recommended human labels Stage
Comet (LLM-as-Judge production guide) 30-50 examples annotated by domain experts Initial calibration set; iterate prompt if >20% disagreement on clear-cut cases
Kili / LangChain 100-200 human-annotated examples Reach 85% agreement before scaling
Databricks (Grading Notes) ~200 use cases total across criteria Achieved 93-96% alignment with human judges using grading-notes pattern

The number is per-axis (per-criterion), not per-skill. A critic with 4 axes needs the calibration set 4 times, OR a stratified single set that covers each axis with 30+ examples.

2. Agreement-rate thresholds: 75-90% before scaling, 85% as canonical cutover

The gap between "scale" (85%) and "remove human" (90-95%) is intentional. Production deployments do not graduate at 85%; they graduate to assisted operation at 85% and to autonomous operation only after a second calibration pass shows 90%+ on a held-out set.

3. Single-metric thresholds are insufficient; literature mandates a battery

Every production source flags that agreement-rate alone is gameable. The dominant recommendation is to report accuracy plus Cohen's kappa (which corrects for chance agreement), plus targeted red-team coverage. ApX explicitly states: "no single metric suffices."

4. Constitutional AI's own numbers do NOT generalize

Anthropic's original CAI paper bootstrapped harmlessness behavior from ~16 written principles plus thousands of self-critique iterations - but those iterations were over a large pretrained model's full output distribution, not over 10 founder edits to a specific artifact type. This is the most important nuance for RDCO: Anthropic's headline numbers come from a regime RDCO does not occupy.

[CORRECTION 2026-07-19] This paragraph originally continued: "The CAI v2 / 'Constitution or Collapse?' (Bai et al., 2025; Llama 3-8B replication) line explicitly shows that on smaller-scale or narrower applications, low label volumes lead to mode collapse (the critic converges on a narrow set of 'acceptable' outputs and rejects valid diversity)." That claim does not survive contact with the primary source and has been struck. See [[2026-07-19-constitution-or-collapse-citation-verification]]. Two separate errors:

  • Attribution (hard-verified). arXiv:2504.04918 resolves and the title is right, but the paper has exactly one author — Zhang, Xue (submitted 2025-04-07). There is no "Bai et al." The vault fused the citing paper's title with the cited paper's authorship (Bai et al. = the original Anthropic CAI paper, arXiv:2212.08073, which Zhang replicates).
  • Finding (per brief's reading of the paper body). Zhang reports collapse in the policy model, not the critic; the phenomenon is degenerate token repetition, not rejection of valid diversity; and the root cause is emoji-noise in the SFT stage-1 revision data plus small-model quality, not low label volume (he used ~5,000 training examples).

Net effect on this brief: the graduation thresholds (≥30 labels/axis, ≥85% agreement) are unaffected — those came from Databricks / LangChain / Comet / ApX production material. What loses its evidentiary basis is Failure Mode 5 below and follow-up Q5. Critic-side mode collapse at low label volume remains a plausible and worth-watching RDCO hypothesis, but it is now an untested hypothesis, not a documented finding, and must not be cited as literature-backed.

Failure modes when you cut over too early

The CAI/RLAIF failure-mode literature is unusually well-catalogued. Six modes recur across sources:

Mode 1: Critic blind-spot exploitation (the dominant mode)

The policy (artifact-producer) learns to produce outputs that hit whatever the critic is actually measuring, while drifting along dimensions the critic doesn't cover. Symptom: PASS rate climbs, real quality plateaus or degrades. This is the textbook reward-hacking story.

Mode 2: Sycophancy / style-mimicry

Policy learns to phrase outputs in the critic's preferred style rather than meet the substantive bar. Symptom: outputs feel uniform; rubric-words appear verbatim in artifacts.

Mode 3: Constitutional loopholes (rule-letter vs rule-intent)

Policy satisfies the literal axis check while violating the underlying intent. The classic example: an axis that requires "cite a source" gets satisfied by hallucinated citations.

Mode 4: Critic drift over time

Even a well-calibrated critic's interpretation of fuzzy axes drifts as the population of artifacts it sees shifts (distribution shift in production). What scored 85% agreement on the calibration set scores 70% three months in.

Mode 5: Mode collapse / reduced diversity

Critic converges on a narrow set of "acceptable" outputs and rejects valid diversity. The policy produces increasingly homogeneous artifacts that all clear the critic but cover less of the legitimate output space.

[CORRECTION 2026-07-19] The supporting citation here — "Documented in the Llama 3-8B CAI replication (Bai et al. 2025)" — has been struck. It was neither correctly attributed nor correctly characterized; see the correction block above and [[2026-07-19-constitution-or-collapse-citation-verification]]. This failure mode is now an RDCO hypothesis with no literature backing. It is still worth instrumenting for (the mechanism is plausible and the cost of watching is low), but it should not be presented to a client, in a Sanity Check issue, or in any internal sizing argument as a documented research finding.

Mode 6: Brittleness on paraphrase / OOD inputs

Critic performance degrades sharply on artifact types it wasn't calibrated on. ApX recommends 95%+ consistency on paraphrased/adversarial variants before removing human oversight.

Where production wisdom diverges from academic claims

Two non-trivial divergences worth flagging:

  1. Academic CAI papers under-report mode collapse. The original Anthropic CAI work emphasized harmlessness gains; mode collapse only became a documented failure when the technique was replicated on smaller models and narrower domains (Bai et al. 2025, "Constitution or Collapse?"). [STRUCK 2026-07-19] — the replication in question (Zhang 2025, arXiv:2504.04918) does not document critic-side mode collapse at all; see [[2026-07-19-constitution-or-collapse-citation-verification]]. There is no known paper establishing this "under-reporting" pattern, so the premise of this convergence point is gone. RDCO's per-skill, per-axis application is still closer to the smaller-narrower regime than to the original Anthropic regime, and the practical implication survives on its own merits: weight production case studies (Databricks, LangChain, Comet) over academic Anthropic papers when sizing the threshold — because the production sources match RDCO's regime, not because the academic literature was shown to under-report.

  2. Academic literature treats reward-model gradient training as load-bearing; production LLM-as-judge does not. Almost every academic CAI / RLAIF paper assumes you're training a parametric reward model via gradient descent on preference pairs. Production deployments overwhelmingly use prompt-engineered LLM-as-judge with grading notes / few-shot examples instead. The label-volume requirements drop dramatically (200 vs 10K+) BUT the calibration discipline rises in importance, because you have no implicit averaging from the gradient process to smooth over noise.

Recommendation for RDCO pipeline-critic schema

Based on triangulation across sources:

Per-axis graduation criteria (proposed)

A pipeline-critic axis graduates from "founder-in-loop" to "autonomous-with-spot-check" when ALL of the following are true:

  1. Calibration volume: ≥30 labeled (PASS/FAIL + rationale) examples for this axis, covering at least 5 distinct artifact shapes. (Lower bound; 100 is safer if the axis is fuzzy.)
  2. Agreement gate: ≥85% agreement between critic verdict and founder verdict on a held-out 10-example subset of the calibration set (subset never used for prompt iteration).
  3. Edge-case red-team: founder constructs 3-5 deliberate edge cases (artifacts that "should fail" along this axis in non-obvious ways); critic catches ≥4/5.
  4. Diversity floor: axis was tested against artifacts from ≥2 different domains (Sanity Check + MAC + skill build-out, for example), not just the domain it was developed on.

Axes that pass all four gates graduate to autonomous, with mandatory spot-check sampling (see below). Axes that pass (1) and (2) but not (3) and (4) graduate to "assisted" - critic returns verdict, founder reviews critic's reasoning before final disposition.

Spot-check sampling rate (proposed)

Founder-override threshold is the WRONG primary signal

The concept doc's "~10 founder overrides" framing inadvertently optimizes for the wrong variable. Founder overrides are FAIL signals - they happen when the critic missed something. Counting them measures critic failure rate, not critic confidence. The literature-correct framing is:

Suggest re-framing the schema as "30 confirmation labels + 0 unresolved overrides" rather than "10 overrides."

Why this is more conservative than the concept doc's "~10"

The "~10 founder-overrides" hand-wave was reasonable for the unverified intuition stage. Now that we have literature: 10 is too few for an LLM-as-judge critic to reliably hit 85% agreement on a fuzzy axis. For mechanical/structural axes (frontmatter presence, link count, character-count limits) - 10 IS enough, because those are presence checks that don't need calibration. For semantic axes (does this brief say something non-derivative? does this verdict match the founder's voice?) - 30-100 is the literature-backed range.

Practical rule for RDCO: split axes into "mechanical" (10-label threshold) and "semantic" (30-100-label threshold) and graduate them separately. This mirrors the structural-vs-semantic split already implicit in pipeline-critic's YAML-axes-vs-prompt-axes architecture.

Cross-check: where vault context modifies the literature default

Three RDCO-specific factors that lower the safe-cutover bar versus the literature default:

  1. Single labeler. No inter-annotator-disagreement noise to resolve. The founder IS the ground truth. This eliminates ~20% of the label volume that production multi-labeler workflows need just to average out human disagreement.
  2. Fresh-eyes subagent pattern (../../feedback_fresh_eyes_subagent_for_own_artifacts). RDCO already routes critic work through zero-context subagents to defeat confirmation bias. This is a structural mitigation for Mode 1 (blind-spot exploitation) that most production deployments don't have.
  3. Reversible-action default (../../feedback_distinguish_decision_from_action + the auto-mode signal-to-noise memory). RDCO already treats most artifacts as cheap-to-redo. Even if the critic mis-grades, the cost of a bad PASS is low for most axes.

Three RDCO-specific factors that RAISE the safe-cutover bar:

  1. Narrow calibration domain at bootstrap. Most axes will be developed on one artifact type first (Sanity Check articles, or MAC carve-outs, or skill SKILL.md files). Cross-domain testing (gate 4 above) is doable but adds latency.
  2. Founder-voice axes are fuzzy. "Does this match the founder's voice" / "is this non-derivative" / "is this overconfident relative to evidence" are exactly the kind of fuzzy semantic axes where 30 labels is the floor, not the ceiling.
  3. /improve consolidation hook is the re-calibration mechanism. If /improve doesn't run on a cadence, drift (Mode 4) WILL show up. Need to either cron /improve or trigger it off the spot-check disagreement rate.

Follow-up questions surfaced

  1. What is the right spot-check stratification strategy for RDCO? Random 10% will get dominated by Sanity Check article volume; need to stratify by axis × artifact-type to catch drift on lower-volume artifact types.
  2. How do we instrument "silent quality degradation" detection? The dominant failure mode (Mode 1) is silent by definition. Periodic founder re-grading catches it but is expensive. Is there a cheaper proxy signal (artifact embedding-diversity drift, axis-correlation drift, output entropy)?
  3. Should /improve be cron'd or threshold-triggered? Cron'd = predictable but wasteful when nothing drifted. Threshold-triggered = efficient but requires the drift-detection signal from Q2.
  4. How do we handle a new axis introduced after others have graduated? Does the new axis re-set the spot-check rate for the whole skill, or only for itself? Production literature is silent on this.
  5. What's the right canonical-exemplars folder size to avoid mode collapse? Bai et al. 2025 documents the failure but doesn't give a number. [REVISED 2026-07-19] No paper documents this failure — the citation this question was built on was misattributed and mischaracterized ([[2026-07-19-constitution-or-collapse-citation-verification]]). The question is now upstream of a number: does critic-side mode collapse occur in RDCO's regime at all? Empirical either way — log artifact-shape diversity over time and watch for entropy decline.

Sources

Related vault docs