Do differentiated rubrics buy the independence that differentiated vendors did not?

The question

Verbatim: "Does published evidence on multi-LLM critic ensembles support using differentiated rubrics per model (rather than differentiated weights) to extract independent signal, given that Codex/Grok/Claude were found correlated-not-independent in RDCO's own seeded-defect benchmark?"

Context: [[2026-07-28-seeded-defect-benchmark-results]] closed the "wire external critics" question with a negative result and left exactly this door open — "if cross-model value exists, it may live in differing rubrics rather than differing weights — untested here."

What we already know (from the vault)

• The benchmark's actual finding, stated at its own confidence level. [[2026-07-28-seeded-defect-benchmark-results]]: Claude, Codex (gpt-5.6-sol) and Grok (grok-4.5-build) each caught 10/10 defects; unique catches by an external model that Claude missed were zero against a pre-registered threshold of ≥2. But the note self-limits hard: n=10 defects, single pass, no repeats, per-reviewer variance unmeasured, defects planted by the same agent whose bias the test exists to check, and an explicit ceiling effect — all three scored 100%, so the corpus was too easy to separate them. Its own words: "This rules out a large effect at this difficulty; it does not rule out a real effect on subtler defects." The correlation claim is a secondary, non-gating measure on one easy corpus, not a settled property of the three models.
• The one genuinely informative correlation datapoint was accidental: artifact-10 was authored as a clean control and all three reviewers independently flagged the same real flaw (a lexical-glob prefix allowlist defeated by ../) by the same reasoning. Converging on an unplanted defect is stronger evidence of shared reasoning than converging on planted ones ([[2026-07-28-seeded-defect-benchmark-results]]).
• The pre-registration already named the successor design. [[2026-07-28-seeded-defect-benchmark-preregistration]] fixed "identical prompt and schema for all three" as a design constant; the results note's own "what follows" item 3 says a revisit needs a harder corpus of real historical misses plus differentiated rubrics per reviewer. So the rubric hypothesis is RDCO's own, pre-existing, and untested.
• The independence constraint is already vault doctrine. [[verifier-as-epistemology]] (Kingsbury): two correlated estimators voting is uninformative; an LLM checking an LLM under the same discipline is "a single layer wearing a costume." [[2026-06-16-multi-agent-ensembles-conviction-calibration]] found every ensemble paper independently rediscovering this, and flagged that RDCO's verify-* stack is already the homogeneous-agents configuration.
• The rubric-sharding machinery already exists in RDCO — in one place only. station-critic is explicitly a fan-out coordinator: one subagent per critic axis, each axis subagent reading only its own axis fragment (~/rdco-vault/01-projects/skill-pipelines/axes/<name>.yaml) and the artifact, deliberately withheld from the spec and tests so axis verdicts stay independent — "parallel-isolation of critic axes is what prevents axis-coupling." That is differentiated-rubrics-on-one-model, already built. But only 3 axis fragments exist (image-coherence, novelty, syllable-count), and the whole /verify-* family (~250 lines of SKILL.md each) runs the opposite shape: one monolithic rubric, one subagent, one verdict.
• [[2026-07-03-verify-skills-pass-disagreement-audit]] already identified the structural exposure the rubric question bears on: verify-* critics run presence/surface checks, the literature's predicted "Evaluation Illusion" blind spot.

What the web says

• Cross-model panels buy far less independence than their headcount implies. "Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels" (arXiv:2605.29800, Guneet Kohli, May 2026, arXiv preprint, not peer-reviewed) measures an effective number of votes of roughly 2.0–2.4 out of 9 judges. Framing from the same work: "three correlated judges are one judge with 3× more requests." (https://arxiv.org/pdf/2605.29800 — fetched, but PDF text extraction was degraded; the rubric-vs-model decomposition I was looking for was not extractable from the fetched text and I am not claiming it is present.)
• Error correlation rises with model capability, across vendors. "Correlated Errors in Large Language Models" reports models agreeing ~60% of the time when both are wrong, with larger and more accurate models showing more correlated errors despite distinct architectures and providers (https://arxiv.org/html/2506.07962 — search-summary level only, full text not fetched). This is the mechanism behind RDCO's own result and it predicts the effect gets worse as frontier models converge, not better.
• Rubric/prompt variation demonstrably moves judge outputs. LLM-as-judge systems are brittle to prompt and rubric phrasing: semantically-preserving edits substantially alter judgments, and changes to scoring rubrics introduce anchoring effects that systematically shift score distributions. "Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness" (arXiv:2601.08843, Deng/Farber/Lee/Tang, Jan 2026, arXiv preprint) is built around exactly this axis (https://arxiv.org/pdf/2601.08843 — fetched; PDF extraction degraded, specific shift magnitudes were not extractable and I am not quoting numbers).
• Rubric decomposition's demonstrated payoff in the literature is coverage, not decorrelation. Recursive Rubric Decomposition and checklist-style frameworks recursively split a coarse rubric into finer dimensions "to enhance coverage and discrimination" and filter redundant criteria; decomposition into atomic criteria "prevents conflation when multiple quality dimensions are evaluated together" (https://arxiv.org/html/2603.00077v1, https://www.researchgate.net/publication/400506257 — search-summary level only). Note the stated failure mode runs the other way too: "overlapping or correlated rubrics can distort aggregation."
• The canonical panel result is about cost and intra-model bias, not rubric diversity. PoLL ("Replacing Judges with Juries," arXiv:2404.18796) motivates panels as a fix for a single judge's intra-model bias, using multiple smaller models — a model-diversity argument, not a rubric-diversity one (https://arxiv.org/pdf/2404.18796 — not fetched, cited from search summary).
• Practitioner tooling treats prompt/criteria ensembling on one model as a first-class option — Verdict (arXiv:2502.18018, Kalra & Tang, arXiv preprint, v2 Nov 2025) exists to scale judge-time compute compositionally, and practitioner write-ups describe "ensemble voting among several prompts or few-shot examples to stabilize a single model's outputs" alongside cross-model juries (https://arxiv.org/pdf/2502.18018 — fetched; the empirical comparison I wanted, single-model prompt-ensemble vs cross-model panel, was not extractable from the PDF. Practitioner framing: https://orq.ai/blog/llm-juries-in-practice).
• Nobody in what I found isolates rubric variation as the diversity source and measures it against model variation. Across three searches and four fetch attempts I found no study that holds the corpus fixed, varies only the rubric on one model, and reports whether error correlation drops relative to a vendor-swap arm. Stated plainly: the specific question is not answered in the published literature I could reach.

Convergences and contradictions

• Strong convergence on the negative half. RDCO's n=10 result and the panel literature agree that swapping vendors under an identical prompt buys little: zero unique catches here; ~2.0–2.4 effective votes from nine judges there; rising co-error rates as models get better. The vault's [[verifier-as-epistemology]] constraint and the benchmark's secondary measure are now corroborated by outside work at much larger N — which matters, because RDCO's own N cannot carry that conclusion alone.
• No convergence on the positive half — and a subtle trap. The literature establishes that rubric changes shift judge output (brittleness, anchoring). It does not establish that they shift errors into independence. Those are different claims: a rubric perturbation that reliably moves scores is evidence of sensitivity, which is equally consistent with generating independent noise (new false positives) as with generating independent signal (new true catches). RDCO's benchmark already saw the noise side — Grok, the reviewer that ran "slightly hotter," is the one that produced a marginal false positive on a clean control ([[2026-07-28-seeded-defect-benchmark-results]]).
• The literature's actual endorsement of differentiated rubrics is a coverage argument, which the RDCO benchmark is structurally blind to. Decomposition is recommended because a monolithic rubric conflates dimensions and under-covers. A 100%-detection ceiling corpus cannot test coverage — there was nothing left to cover. So the benchmark neither supports nor refutes the rubric hypothesis; it is not powered for it.

Synthesis for RDCO

Direct answer: no — published evidence does not support differentiated rubrics as an independence mechanism, because no one has tested that. What the evidence does support is weaker and different, and worth separating cleanly. (1) It strongly supports the conclusion RDCO already reached: cross-vendor panels under identical prompts are largely one judge billed several times, and the correlation gets worse as models improve. (2) It supports rubric decomposition as a coverage and discrimination mechanism — split a coarse rubric into atomic criteria so dimensions stop being conflated and under-checked. (3) It is silent on the sharp form of the question, whether varying the rubric on a single model recovers statistical independence of errors more cheaply than varying the vendor. Anyone who tells RDCO otherwise is extrapolating from brittleness results, and brittleness is as easily a false-positive generator as a true-positive generator.

That reframing changes what RDCO should do with the open thread, and it makes the answer cheaper rather than more expensive. The reason to shard /verify-* into differentiated axis-rubrics is not "we recover the independence we failed to get from Codex and Grok." It is "a 250-line monolithic rubric evaluated in one pass under-covers, and the fix is one narrow rubric per subagent." That is a coverage claim the literature does back, and — the important part — RDCO already built the machinery for it. station-critic is a per-axis fan-out with deliberate parallel isolation and a documented rationale ("axis-coupling") that is precisely the decomposition argument the rubric papers make, arrived at independently. The gap is not architectural, it is population: three axis fragments exist, all belonging to one narrow pipeline domain, and the /verify-* family never adopted the pattern. Porting the axis shape to verify-vault-write, verify-strategic-output and verify-dispatch requires no vendor spend, no ask-model.sh, no sandbox-exec confinement problem (the delegation work found neither external CLI can be read-confined by its own flags), and no new trust boundary. It is the strictly cheaper arm and it should be evaluated first on those grounds alone.

The cost asymmetry is worth stating explicitly because it inverts the usual framing. N rubric-shards on one model costs roughly the same inference as N vendors on one rubric — but the vendor arm additionally carries auth, wrapper, sandboxing, output-schema drift, and a second and third trust boundary. So even if rubric-diversity and vendor-diversity were equally effective at decorrelation (unknown), rubric-diversity wins on operational cost. And if the literature's coverage argument holds while the independence argument stays unproven, rubric-diversity wins on the one dimension that is actually supported. The honest posture for the critic layer: keep external models unwired, as [[2026-07-28-seeded-defect-benchmark-results]] concluded; treat rubric-sharding as a coverage upgrade with an unproven independence bonus, not as the independence fix.

One caution about re-running the benchmark to settle this. The ceiling effect is the binding constraint, not the rubric variable. A rubric-differentiated re-run on the same 12-artifact corpus will return 10/10 across every arm and settle nothing. Corpus difficulty has to be fixed first, and the results note already says how: real historical misses pulled from the vault log, not defects authored by the agent under test — which also retires the circularity the pre-registration flagged and left unresolved. Without that, any follow-up spends the effort and buys another negative-that-means-nothing.

Smallest experiment RDCO could run itself (pre-registered, per [[feedback_plan_tests_implementation_order]]): build a harder corpus of ~20 artifacts seeded with real logged misses — the seven observed factory failure modes are the natural source: pointer-not-content returns, shared-scratchpad contamination, false "verified against primary text" stamps, status: proposal laundered as live fact, cache-stale critic CRITICALs. Then run three arms on identical artifacts: (A) one Claude critic, one monolithic rubric (the current /verify-* shape, baseline); (B) N Claude critics, one differentiated axis-rubric each, mutually blind (the station-critic shape); (C) the already-run cross-vendor arm, identical prompt. Pre-register two numbers: unique catches by arm B that arm A missed, and false positives per clean control per arm. Arm B earns the port if it beats A on unique catches without inflating FP rate above A's. Comparing B against C then gives the rubric-vs-vendor answer nobody in the literature has published — which would make it a genuinely original result worth writing up publicly, not just an internal decision.

Why this is in the vault

This settles the explicitly-open thread in [[2026-07-28-seeded-defect-benchmark-results]] and decides the next move on the critic-layer architecture: it says do not revisit multi-vendor wiring for station-critic / /verify-*, and it re-grounds the rubric-sharding proposal as a coverage upgrade to the existing single-model axis machinery rather than as an independence fix — which changes both the justification and the experiment that would validate it.

Open follow-ups

• Does axis-decomposed critique on a single model produce unique catches over a monolithic rubric, or only relabel the same catches? (Arm A vs Arm B above — the load-bearing unknown.)
• Does rubric variation inflate false positives on clean controls faster than it adds true catches? (The Grok-ran-hotter signal suggests sensitivity and noise scale together.)
• What is the right number of axes before diminishing returns / aggregation distortion? The rubric literature warns that overlapping or correlated rubrics distort aggregation — is there an RDCO-measurable redundancy threshold?
• Can a harder corpus be built from real historical misses in the vault log at sufficient N, and does an audit of past /verify-* PASSes actually surface enough logged misses to seed 20 artifacts?
• Does the "effective number of votes" statistic from arXiv:2605.29800 transfer to a rubric-sharded panel — i.e. can RDCO compute effective-N over its own axis verdicts as a standing health metric on the critic layer rather than a one-off benchmark?

Related

• [[2026-07-28-seeded-defect-benchmark-results]]
• [[2026-07-28-seeded-defect-benchmark-preregistration]]
• [[2026-07-28-external-model-delegation-codex-grok]]
• [[verifier-as-epistemology]]
• [[2026-06-16-multi-agent-ensembles-conviction-calibration]]
• [[2026-07-03-verify-skills-pass-disagreement-audit]]
• [[2026-07-26-autoreview-skill-teardown-second-model-critic-design]]
• [[2026-07-25-multi-agent-claude-codex-grok-composition-patterns]]
• [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]]
• [[feedback_plan_tests_implementation_order]]
• [[feedback_fresh_eyes_subagent_for_own_artifacts]]

Sources

Vault (read in full this run):

• [[2026-07-28-seeded-defect-benchmark-results]] — ~/rdco-vault/02-sops/2026-07-28-seeded-defect-benchmark-results.md
• [[2026-07-28-seeded-defect-benchmark-preregistration]] — ~/rdco-vault/02-sops/2026-07-28-seeded-defect-benchmark-preregistration.md
• [[2026-06-16-multi-agent-ensembles-conviction-calibration]] — ~/rdco-vault/06-reference/research/2026-06-16-multi-agent-ensembles-conviction-calibration.md

Vault (cited from search snippets / referenced, not read in full this run):

• [[verifier-as-epistemology]] — ~/rdco-vault/06-reference/concepts/verifier-as-epistemology.md
• [[2026-07-03-verify-skills-pass-disagreement-audit]] — ~/rdco-vault/06-reference/research/2026-07-03-verify-skills-pass-disagreement-audit.md
• [[2026-07-26-autoreview-skill-teardown-second-model-critic-design]] — ~/rdco-vault/06-reference/2026-07-26-autoreview-skill-teardown-second-model-critic-design.md
• [[2026-07-25-multi-agent-claude-codex-grok-composition-patterns]] — ~/rdco-vault/06-reference/2026-07-25-multi-agent-claude-codex-grok-composition-patterns.md
• [[2026-07-28-external-model-delegation-codex-grok]] — ~/rdco-vault/02-sops/2026-07-28-external-model-delegation-codex-grok.md
• [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]] — ~/rdco-vault/02-sops/2026-05-20-verify-stack-two-gate-pass-fail-architecture.md

Harness (inspected directly this run): ~/.claude/skills/station-critic/SKILL.md (per-axis fan-out, axis-coupling rationale); ~/rdco-vault/01-projects/skill-pipelines/axes/ (3 fragments: image-coherence, novelty, syllable-count); ~/.claude/skills/verify-{vault-write,strategic-output,dispatch}/SKILL.md (251/255/264 lines, monolithic single-rubric shape).

Web — fetched this run (PDF text extraction degraded on all three; no numbers quoted beyond what is stated above):

• Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels — arXiv:2605.29800, Guneet Kohli, May 2026, arXiv preprint — https://arxiv.org/pdf/2605.29800
• Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness — arXiv:2601.08843, Deng, Farber, Lee, Tang, Jan 2026, arXiv preprint — https://arxiv.org/pdf/2601.08843
• Verdict: A Library for Scaling Judge-Time Compute — arXiv:2502.18018v2, Kalra & Tang, Nov 2025, arXiv preprint — https://arxiv.org/pdf/2502.18018

Web — search-summary level only, NOT fetched (claims labelled as such in the body):

• Correlated Errors in Large Language Models — https://arxiv.org/html/2506.07962
• Autorubric: A Unified Framework for Rubric-Based LLM Evaluation — https://arxiv.org/html/2603.00077v1
• Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks (Recursive Rubric Decomposition) — https://www.researchgate.net/publication/400506257
• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (PoLL) — https://arxiv.org/pdf/2404.18796
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape — https://arxiv.org/pdf/2606.08625 (fetch failed: response exceeded size limit; not retried)
• Weak judges, strong panel: an ensemble approach to LLM eval (practitioner blog, orq.ai) — https://orq.ai/blog/llm-juries-in-practice