Effective-N does not transfer to a rubric-sharded critic panel, and RDCO has no verdict data to feed it anyway
The question
Verbatim: "Does the 'effective number of votes' statistic (arXiv:2605.29800) transfer to a rubric-sharded critic panel — can RDCO compute effective-N over its own axis verdicts as a standing health metric on the critic layer, rather than a one-off benchmark?"
Context: this is the one open follow-up from [[2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics]] that would convert the critic layer from a periodically-benchmarked thing into a continuously-instrumented one. The parent brief flagged the identifier as fetched-but-degraded, so the citation itself needed resolving before anything was built on it.
What we already know (from the vault)
- The citation was real, and the parent brief described it accurately. arXiv:2605.29800 resolves. See "What the web says" below. The parent brief's attribution (title, single author Guneet Kohli, May 2026, arXiv preprint, not peer reviewed) and its reported figure of roughly 2.0-2.4 effective votes from 9 judges both check out against the paper itself (headline value 2.18, 95% bootstrap interval [2.07, 2.31]).
- RDCO's critics are strongly suspected to be correlated. The pre-registered 2026-07-28 benchmark had Claude, Codex and Grok catch the same 10/10 seeded defects with zero unique external catches against a pre-registered threshold of 2 ([[2026-07-28-seeded-defect-benchmark-results]]). Treated as a secondary, non-gating measure on a ceiling-effect corpus, not a settled property.
- The vault already holds the doctrine this question is trying to operationalise. [[verifier-as-epistemology]]: two correlated estimators voting is uninformative. [[2026-06-16-multi-agent-ensembles-conviction-calibration]] identified the verify-* stack as already the homogeneous-agents configuration.
- The critic layer's per-axis machinery exists on paper and has never executed. Verified on disk 2026-10-11:
~/rdco-vault/01-projects/skill-pipelines/runs/is an empty directory created 2026-05-12 and never written to; a filesystem-wide search forcritic-feedback-iter-*,final-state.mdandwarnings.mdreturns zero files. One per-domain config exists (haiku-generator.yaml, 3 axes: syllable-count, image-coherence, novelty), and all three axis fragments sit atlabel_count: 0. Thestation-criticfan-out has run zero times. - The verdict-logging gap is already a named RDCO finding, not a new discovery. [[2026-07-03-verify-skills-pass-disagreement-audit]] concluded that zero founder-vs-PASS disagreements are logged and that the absence "is an instrumentation gap, not a clean bill of health," naming as blind spot zero: "the stack has no verdict log." Re-verified today:
~/.claude/state/flash-review-log.jsonland~/.claude/state/improve-cron-log.mdstill do not exist. Six of the seven live critic skills (verify-vault-write, verify-strategic-output, verify-dispatch, verify-pdf-output, design-critic, video-critic) emit a structured per-axis PASS/FAIL block in their return text and persist none of it. The single exception is~/.claude/state/behavior-critic-log.jsonl: 5 rows, 2 incompatible key schemas, one of them a smoke test, two of them the same pull request, carrying an aggregate verdict plus a clause count rather than per-clause identities.
What the web says
- The paper exists and the parent brief's citation is correct. "Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels," arXiv:2605.29800v1, Guneet Kohli (Apple), submitted 28 May 2026, cs.CL, DOI 10.48550/arXiv.2605.29800. Single author, v1 only, arXiv preprint, no venue, not peer reviewed. The abstract page and the full LaTeXML HTML both fetched cleanly.
- The statistic is the Kish design-effect form, over error indicators, against gold labels. Section 3.3, quoted: "For each judge, we construct a binary error vector e_j in {0,1}^1000 where e_{j,i}=1 if judge j disagrees with the ChaosNLI majority label on item i. We compute the pairwise phi coefficient phi_jk between all 36 judge pairs, then apply the Kish formula: n_eff = k / (1 + (k-1) * phi_bar)". The gold standard is itself "the majority vote of 100 annotators." Attribution is to Kish, Leslie (1965), Survey Sampling, J. Wiley (verified independently). Headline results: k=9, N=1,000 items, phi_bar = 0.391, n_eff = 2.18 [2.07, 2.31], independence ratio 24.2%, and a Condorcet accuracy gap of 22.0 percentage points on MNLI.
- Ground truth is load-bearing, not optional. The quantity correlated is the error indicator, which is defined by disagreement with the gold label. No gold labels means no error vectors means no phi_bar means no n_eff. The paper reports gold-free inter-judge statistics (Krippendorff's alpha, panel entropy) but never feeds them into the Kish formula.
- The paper assumes all judges answer the same question against the same label set, and is silent on judges scoring different criteria. The error vector is indexed on one shared item index with one shared gold label per item; aggregation is majority vote over a common label space; section 3.2 specifies "a standardized NLI classification prompt." Searching the full text for rubric, criteria or dimension returns nothing relevant. Non-exchangeable judges scoring different constructs are not in the method and not in the limitations.
- Prompt perturbation did not decorrelate the panel, and chain-of-thought made it worse. Section 4.6, Table 4: baseline phi_bar = .391 / n_eff = 2.18; reframed prompt .394 / 2.17; reversed label order .399 / 2.15; chain-of-thought .456 / 1.94. Quoted: "Chain-of-thought actually increases correlation ... shared reasoning amplifies shared errors." Important scope limit: these were panel-wide prompt swaps, with all 9 judges receiving the same new prompt. Heterogeneous prompts across judges were not tested.
- Model-family diversity bought nothing either. Section 5.3: the three most-correlated pairs are all cross-family (Claude x Gemini phi = 0.603), and restricting to one judge per family (k=7) lowered n_eff to 1.93. This corroborates RDCO's zero-unique-catches result at far larger N.
- The paper's only monitoring-shaped recommendation is a threshold, not a time series. Quoted: "We recommend computing n_eff as a standard panel diagnostic: if n_eff/k < 0.5, results should be treated with caution." It contains no longitudinal, drift or streaming-estimation discussion. Sample-size guidance: n_eff "stabilizes by N ~ 200-300" items, with the interval narrowing substantially by N=500.
- The "effective number" algebra is old and well-established, which is why it is tempting. The same inverse-concentration form recurs across fields: Kish's effective sample size and the cluster design effect Deff = 1 + (n* - 1) * rho (Kish 1965, via the Wikipedia Design effect article); the effective number of parties, 1 / sum(p_i^2) (Laakso & Taagepera 1979, Comparative Political Studies 12(1):3-27, DOI 10.1177/001041407901200101); and the effective number of spatial degrees of freedom, N*_ef = (sum lambda_k)^2 / sum lambda_k^2 = (tr C)^2 / tr(C^2), bounded 1 <= N*_ef <= N, equal to N when all eigenvalues are equal and to 1 when one mode holds all variance (Bretherton, Widmann, Dymnikov, Wallace & Bladé 1999, Journal of Climate 12(7):1990-2009, DOI 10.1175/1520-0442(1999)012<1990:TENOSD>2.0.CO;2, equations 3 and 4). Kohli cites Bretherton for a robustness check,
n_eff^eigen = k / lambda_max, which is a different formula from Bretherton's own equation 3. The classic warning that the choice of intraclass correlation model changes the answer is Shrout & Fleiss (1979), Psychological Bulletin 86(2):420-428, DOI 10.1037/0033-2909.86.2.420. - Secondary use in the wild, not read directly. A search result for arXiv:2607.07469 ("SynthAVE", Scarinci et al.) credits Kohli (2026) for estimating effective votes from the mean pairwise correlation of judge error indicators, reporting about 2.6 effective votes from 21 judges at phi_bar = 0.36. I could not read that paper: WebFetch returned only PDF metadata and the body streams were unreadable. Treat this as a search-snippet-level corroboration of the formula's uptake, nothing more.
Convergences and contradictions
- Convergence, and it is strong. Kohli's cross-family results at k=9, N=1,000 say the same thing RDCO's n=10 benchmark said: swapping vendors under a shared prompt buys very little independence, and the best single judge matched or beat the full panel in every condition he tested. RDCO's own N could never carry that conclusion; this does.
- New evidence that bears directly on the parent brief's hypothesis, pointing the wrong way. The parent brief could find nothing published that isolates prompt or rubric variation as a decorrelation mechanism. Section 4.6 is the closest thing yet, and it is a null-to-negative result: reframing and label reordering moved phi_bar by less than 0.01, and chain-of-thought raised it by 0.065 while costing 2.8 points of panel accuracy. This does not settle the rubric-shard question, because every variant was applied uniformly to all nine judges rather than differentially across them. It does retire the weak version of the hypothesis, that merely perturbing the prompt decorrelates a panel.
- Contradiction between what the metric needs and what RDCO's architecture is for. Kohli's framework needs judges to be interchangeable answerers of one question. The explicit design rationale for
station-critic's per-axis fan-out is the opposite: "parallel-isolation of critic axes is what prevents axis-coupling." RDCO deliberately built non-exchangeable judges. That is not a flaw in either the paper or the architecture; it means the statistic and the architecture are addressing different problems.
Synthesis for RDCO
Direct answer: no. The statistic does not transfer, and the honest version of the answer has two independent halves, either of which is sufficient on its own.
Half one, the mathematical condition. Kohli's n_eff is derived from the variance of an average. The derivation that follows is mine, not his, though it is the standard one behind the Kish cluster form. Take k judges with unit-variance error indicators and common pairwise correlation rho. The variance of their mean error is (1/k) * [1 + (k-1) * rho]. Setting that equal to the variance of a mean over m independent judges, 1/m, gives m = k / (1 + (k-1) * rho), which is Kohli's equation 1 with rho = phi_bar. Four conditions are baked into that derivation. C1, a common estimand: all k judges must be estimating the same quantity, or the per-judge error indicator is not comparable across judges and their mean is not an estimator of anything. C2, averaging aggregation: the identity is a statement about the variance of a mean, so the panel must aggregate by mean or majority. C3, ground truth: rho is over error indicators, not vote indicators. C4, estimable correlations: non-degenerate per-judge variance and enough items; Kohli needs about 200 to 300 to stabilise at k=9.
RDCO's rubric-sharded panel violates all four. C1 fails by design: syllable-count, image-coherence and novelty have three different estimands and there is no single gold label an artifact could have that all three are noisy readings of. C2 fails: station-critic aggregates by conjunction, "Any axis returns FAIL → loop fails," which is an AND gate, not a vote. An AND gate has no mean whose variance you could match, so the quantity n_eff is purporting to measure is undefined for it; the relevant statistic for a conjunction is the miss probability, product over axes of (1 - detection rate), which depends on the per-axis detection rates and not just on rho. C3 fails: nothing in RDCO adjudicates whether a critic verdict was correct, which is the exact gap [[2026-07-03-verify-skills-pass-disagreement-audit]] already named. C4 fails hardest: there are zero logged critic runs to compute anything from.
A tempting workaround makes it worse rather than better. With no labels, the only correlation RDCO could compute is between axis votes, and substituting vote correlation for error correlation yields a different and non-conservative number. Vote agreement mixes shared signal with shared error; Kohli's own data shows the two decouple, with 319 of 1,000 items drawing unanimous votes yet only 90.9% accuracy on that subset. A vote-based effective-N would read reassuringly high precisely when the panel is agreeing for bad reasons. That inference is mine, supported by his figure.
There is one further reason to decline even if the conditions were met. At k=3, which is RDCO's only configured axis count, the entire plausible range of n_eff is 1.36 (at phi_bar = 0.6) to 2.14 (at phi_bar = 0.2), and n_eff/k sits at 0.45 to 0.71, straddling Kohli's caution threshold across the whole range. A one-number summary whose full dynamic range is 0.8 units, computed from exactly three pairwise correlations, is a reparameterisation of phi_bar that adds interpretive risk without adding information. At small k, report the pairwise agreement table and skip the transform.
Half two, what would work instead. Three things, all of them my derivation and none of them a published statistic for this setting. (1) Sole-FAIL attribution, the cheapest and most useful. Because aggregation is AND, an axis changed the outcome exactly when it was the only FAIL. So log, per window: for each axis j, its FAIL count, its sole-FAIL count, and u_j = sole_FAILs / FAILs. An axis with zero sole-FAILs across several dozen runs is a retirement candidate, and this answers the parent brief's open question about the right number of axes without needing a single label. (2) An axis-redundancy index, if a scalar is wanted. Build the matrix of FAIL indicators, one row per run and one column per axis, take the correlation matrix R, and compute the participation ratio N_part = k^2 / (k + 2 * sum_{j<k} R_jk^2). That is Bretherton's equation 3 rewritten using tr(R) = k and tr(R^2) = sum of squared entries, so it needs no eigendecomposition. It is bounded 1 to k and requires no ground truth, because redundancy is a property of the verdict vectors themselves. Two honesty requirements come with it. It must be called an axis-redundancy index and never "effective votes," because it cannot tell you the panel is any good: a panel of axes firing on independent noise maximises it. And note that under compound symmetry it equals k / (1 + (k-1) * rho^2), with rho squared, which is genuinely a different number from Kohli's k / (1 + (k-1) * rho). The algebra transfers; the meaning does not. (3) The only path to Kohli's actual statistic is labels. Error correlation requires an adjudicated verdict per axis per artifact, and for different-criteria axes that means one gold label per axis, which is strictly more labelling work than Kohli's single-gold case, not less. The nearest existing hook is the bootstrap: true path, which already routes axis verdicts to the founder and increments label_count. That hook is insufficient as specified: label_count is an anonymous counter that discards both the critic's verdict and the founder's, so the pair (critic said PASS on axis A, founder said FAIL) is structurally unrepresentable in today's schema.
What RDCO would have to start logging that it does not log today: effectively everything. There is no partial foundation to extend. A minimum viable verdict row is (run_id, ts, skill, domain, artifact_id, artifact_hash, iteration, axis_id, rubric_version, verdict, confidence, rationale), one row per axis per run rather than one row per artifact, which is the single biggest shape change from the only log that exists. Three non-obvious requirements. Axis identifiers must be stable and paired with a rubric version stamp, because today's axes are prose bullet labels that have been edited in place; without versioning, cross-run comparison silently pools incompatible rubrics. Confidence is emitted by nobody that actually runs, since high|medium|low and the PASS+weak state live only in the never-executed pipeline layer, so there is no graded signal to log until the live skills emit one. And the write must be an instructed step inside each skill's Process section, not a side-channel script: behavior-critic is the counterexample, where a working log path plus a working append script with no instructed step produced 5 rows in 3 months across 2 schemas. A second table is needed for adjudications, (run_id, axis_id, adjudicated_label, label_source, labelled_at), with the realistic label sources being founder-overruled FAILs and post-PASS escaped defects.
The recommendation in one line. Do not build an effective-N instrument. Build the per-axis verdict log that [[2026-07-03-verify-skills-pass-disagreement-audit]] already named as blind spot zero, add the sole-FAIL attribution table on top of it, and treat correlated-error statistics as gated on label capture rather than on statistical choice. The binding constraint on this layer was never which statistic to compute; it is that the layer emits structured verdicts and then throws them away.
Why this is in the vault
This closes the last open follow-up from [[2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics]] with a no, which removes "compute effective-N over axis verdicts" from the critic-layer roadmap and re-points that effort at the per-axis verdict log that [[2026-07-03-verify-skills-pass-disagreement-audit]] has been blocked on since July. It also records a resolved citation for arXiv:2605.29800, whose prompt-variation table is the first published evidence RDCO has on whether perturbing the prompt decorrelates a judge panel.
Open follow-ups
- Does differential prompting across judges, rather than the uniform panel-wide swaps Kohli tested in section 4.6, reduce mean pairwise error correlation on any published benchmark? This is the surviving form of the parent brief's rubric hypothesis and Kohli explicitly leaves it open ("More radical prompt reformulations ... could in principle alter the correlation structure").
- Does generalizability theory, with its person-by-item-by-rater facet decomposition and its decision-study estimates of how many raters a given facet needs, give a published framework for panels of non-exchangeable judges scoring different criteria, where the Kish and Condorcet machinery does not apply?
- At the sample sizes a small shop can realistically reach, under 50 labelled runs per axis on a PASS-skewed distribution, what does the methodological literature on sparse and imbalanced agreement data recommend over raw disagreement percentage and Cohen's kappa? The 2026-07-03 audit pre-registered this tension and left it unreconciled.
Related
- [[2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics]] - the parent brief; this answers its fifth open follow-up
- [[2026-07-03-verify-skills-pass-disagreement-audit]] - blind spot zero, the missing per-axis verdict log; the real blocker this brief re-points at
- [[2026-07-28-seeded-defect-benchmark-results]] - the zero-unique-external-catches result that Kohli's cross-family numbers corroborate
- [[2026-05-12-multi-agent-pipeline-config-schema]] - the axis-fragment and per-domain config schemas; where
label_countis specified as a bare counter - [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]] - the verdict-vocabulary SOP whose two-gate migration never shipped, which is why any pooled statistic would mix scales
- [[verifier-as-epistemology]] - the correlated-estimators constraint this question was trying to instrument
- [[2026-06-16-multi-agent-ensembles-conviction-calibration]] - identified the verify-* stack as the homogeneous-agents configuration
Sources
Web, fetched and read this run:
- Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels - arXiv:2605.29800v1, Guneet Kohli (Apple), 28 May 2026, cs.CL, arXiv preprint, not peer reviewed - https://arxiv.org/abs/2605.29800 (abstract page and full LaTeXML HTML at /html/2605.29800v1 both fetched; equations in section 3.3 quoted from the HTML's LaTeX alttext)
- Bretherton, C.S., Widmann, M., Dymnikov, V.P., Wallace, J.M. & Bladé, I. (1999). The Effective Number of Spatial Degrees of Freedom of a Time-Varying Field. Journal of Climate 12(7):1990-2009. DOI 10.1175/1520-0442(1999)012<1990:TENOSD>2.0.CO;2 (equations 3, 4 and 10 read from the author PDF at atmos.uw.edu)
- Design effect (tertiary source, for the Kish effective-sample-size and cluster design-effect formulas and their attribution to Kish 1965, pp. 162, 259) - https://en.wikipedia.org/wiki/Design_effect
Web, metadata verified but full text not read:
- Kish, L. (1965). Survey Sampling. J. Wiley. (Verified via OpenLibrary; cited as the origin of both the effective-sample-size and cluster design-effect formulas by Kohli and by the Design effect article. Book not consulted directly.)
- Laakso, M. & Taagepera, R. (1979). "Effective" Number of Parties: A Measure with Application to West Europe. Comparative Political Studies 12(1):3-27. DOI 10.1177/001041407901200101 (verified via Crossref; cited for the inverse-concentration form only)
- Shrout, P.E. & Fleiss, J.L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 86(2):420-428. DOI 10.1037/0033-2909.86.2.420 (verified via Crossref; cited for the model-choice warning only)
Web, unreachable:
- SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation - arXiv:2607.07469v2, Scarinci, Negri, Impata, Khan, Martinez & Federico. DOI 10.48550/arXiv.2607.07469. Body text could not be read: WebFetch returned only PDF metadata, the content streams being FlateDecode binary. Its reported citation of Kohli (2026) is from a search snippet only and is labelled as such above.
- arXiv:2606.08625 (From Holistic Evaluation to Structured Criteria), carried over from the parent brief as a failed fetch; not re-attempted this run.
Harness, inspected on disk 2026-10-11:
/Users/ray/.claude/skills/station-critic/SKILL.md/Users/ray/rdco-vault/01-projects/skill-pipelines/axes/{syllable-count,image-coherence,novelty}.yaml/Users/ray/rdco-vault/01-projects/skill-pipelines/configs/haiku-generator.yaml/Users/ray/rdco-vault/01-projects/skill-pipelines/runs/(empty; never written)/Users/ray/.claude/skills/{verify-vault-write,verify-strategic-output,verify-dispatch,verify-pdf-output,design-critic,behavior-critic,video-critic}/SKILL.md/Users/ray/.claude/state/behavior-critic-log.jsonl(5 rows; the only persisted critic verdict log on the machine)
Vault:
~/rdco-vault/06-reference/research/2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics.md~/rdco-vault/06-reference/research/2026-07-03-verify-skills-pass-disagreement-audit.md~/rdco-vault/02-sops/2026-07-28-seeded-defect-benchmark-results.md~/rdco-vault/02-sops/2026-05-12-multi-agent-pipeline-config-schema.md~/rdco-vault/02-sops/2026-05-20-verify-stack-two-gate-pass-fail-architecture.md~/rdco-vault/06-reference/concepts/verifier-as-epistemology.md~/rdco-vault/06-reference/research/2026-06-16-multi-agent-ensembles-conviction-calibration.md