06-reference/research

effective n votes rubric sharded critic panel

2026-10-11·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
critic-layereffective-sample-sizellm-as-judgeinstrumentationverification

Effective-N does not transfer to a rubric-sharded critic panel, and RDCO has no verdict data to feed it anyway

The question

Verbatim: "Does the 'effective number of votes' statistic (arXiv:2605.29800) transfer to a rubric-sharded critic panel — can RDCO compute effective-N over its own axis verdicts as a standing health metric on the critic layer, rather than a one-off benchmark?"

Context: this is the one open follow-up from [[2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics]] that would convert the critic layer from a periodically-benchmarked thing into a continuously-instrumented one. The parent brief flagged the identifier as fetched-but-degraded, so the citation itself needed resolving before anything was built on it.

What we already know (from the vault)

What the web says

Convergences and contradictions

Synthesis for RDCO

Direct answer: no. The statistic does not transfer, and the honest version of the answer has two independent halves, either of which is sufficient on its own.

Half one, the mathematical condition. Kohli's n_eff is derived from the variance of an average. The derivation that follows is mine, not his, though it is the standard one behind the Kish cluster form. Take k judges with unit-variance error indicators and common pairwise correlation rho. The variance of their mean error is (1/k) * [1 + (k-1) * rho]. Setting that equal to the variance of a mean over m independent judges, 1/m, gives m = k / (1 + (k-1) * rho), which is Kohli's equation 1 with rho = phi_bar. Four conditions are baked into that derivation. C1, a common estimand: all k judges must be estimating the same quantity, or the per-judge error indicator is not comparable across judges and their mean is not an estimator of anything. C2, averaging aggregation: the identity is a statement about the variance of a mean, so the panel must aggregate by mean or majority. C3, ground truth: rho is over error indicators, not vote indicators. C4, estimable correlations: non-degenerate per-judge variance and enough items; Kohli needs about 200 to 300 to stabilise at k=9.

RDCO's rubric-sharded panel violates all four. C1 fails by design: syllable-count, image-coherence and novelty have three different estimands and there is no single gold label an artifact could have that all three are noisy readings of. C2 fails: station-critic aggregates by conjunction, "Any axis returns FAIL → loop fails," which is an AND gate, not a vote. An AND gate has no mean whose variance you could match, so the quantity n_eff is purporting to measure is undefined for it; the relevant statistic for a conjunction is the miss probability, product over axes of (1 - detection rate), which depends on the per-axis detection rates and not just on rho. C3 fails: nothing in RDCO adjudicates whether a critic verdict was correct, which is the exact gap [[2026-07-03-verify-skills-pass-disagreement-audit]] already named. C4 fails hardest: there are zero logged critic runs to compute anything from.

A tempting workaround makes it worse rather than better. With no labels, the only correlation RDCO could compute is between axis votes, and substituting vote correlation for error correlation yields a different and non-conservative number. Vote agreement mixes shared signal with shared error; Kohli's own data shows the two decouple, with 319 of 1,000 items drawing unanimous votes yet only 90.9% accuracy on that subset. A vote-based effective-N would read reassuringly high precisely when the panel is agreeing for bad reasons. That inference is mine, supported by his figure.

There is one further reason to decline even if the conditions were met. At k=3, which is RDCO's only configured axis count, the entire plausible range of n_eff is 1.36 (at phi_bar = 0.6) to 2.14 (at phi_bar = 0.2), and n_eff/k sits at 0.45 to 0.71, straddling Kohli's caution threshold across the whole range. A one-number summary whose full dynamic range is 0.8 units, computed from exactly three pairwise correlations, is a reparameterisation of phi_bar that adds interpretive risk without adding information. At small k, report the pairwise agreement table and skip the transform.

Half two, what would work instead. Three things, all of them my derivation and none of them a published statistic for this setting. (1) Sole-FAIL attribution, the cheapest and most useful. Because aggregation is AND, an axis changed the outcome exactly when it was the only FAIL. So log, per window: for each axis j, its FAIL count, its sole-FAIL count, and u_j = sole_FAILs / FAILs. An axis with zero sole-FAILs across several dozen runs is a retirement candidate, and this answers the parent brief's open question about the right number of axes without needing a single label. (2) An axis-redundancy index, if a scalar is wanted. Build the matrix of FAIL indicators, one row per run and one column per axis, take the correlation matrix R, and compute the participation ratio N_part = k^2 / (k + 2 * sum_{j<k} R_jk^2). That is Bretherton's equation 3 rewritten using tr(R) = k and tr(R^2) = sum of squared entries, so it needs no eigendecomposition. It is bounded 1 to k and requires no ground truth, because redundancy is a property of the verdict vectors themselves. Two honesty requirements come with it. It must be called an axis-redundancy index and never "effective votes," because it cannot tell you the panel is any good: a panel of axes firing on independent noise maximises it. And note that under compound symmetry it equals k / (1 + (k-1) * rho^2), with rho squared, which is genuinely a different number from Kohli's k / (1 + (k-1) * rho). The algebra transfers; the meaning does not. (3) The only path to Kohli's actual statistic is labels. Error correlation requires an adjudicated verdict per axis per artifact, and for different-criteria axes that means one gold label per axis, which is strictly more labelling work than Kohli's single-gold case, not less. The nearest existing hook is the bootstrap: true path, which already routes axis verdicts to the founder and increments label_count. That hook is insufficient as specified: label_count is an anonymous counter that discards both the critic's verdict and the founder's, so the pair (critic said PASS on axis A, founder said FAIL) is structurally unrepresentable in today's schema.

What RDCO would have to start logging that it does not log today: effectively everything. There is no partial foundation to extend. A minimum viable verdict row is (run_id, ts, skill, domain, artifact_id, artifact_hash, iteration, axis_id, rubric_version, verdict, confidence, rationale), one row per axis per run rather than one row per artifact, which is the single biggest shape change from the only log that exists. Three non-obvious requirements. Axis identifiers must be stable and paired with a rubric version stamp, because today's axes are prose bullet labels that have been edited in place; without versioning, cross-run comparison silently pools incompatible rubrics. Confidence is emitted by nobody that actually runs, since high|medium|low and the PASS+weak state live only in the never-executed pipeline layer, so there is no graded signal to log until the live skills emit one. And the write must be an instructed step inside each skill's Process section, not a side-channel script: behavior-critic is the counterexample, where a working log path plus a working append script with no instructed step produced 5 rows in 3 months across 2 schemas. A second table is needed for adjudications, (run_id, axis_id, adjudicated_label, label_source, labelled_at), with the realistic label sources being founder-overruled FAILs and post-PASS escaped defects.

The recommendation in one line. Do not build an effective-N instrument. Build the per-axis verdict log that [[2026-07-03-verify-skills-pass-disagreement-audit]] already named as blind spot zero, add the sole-FAIL attribution table on top of it, and treat correlated-error statistics as gated on label capture rather than on statistical choice. The binding constraint on this layer was never which statistic to compute; it is that the layer emits structured verdicts and then throws them away.

Why this is in the vault

This closes the last open follow-up from [[2026-07-30-multi-llm-critic-ensembles-differentiated-rubrics]] with a no, which removes "compute effective-N over axis verdicts" from the critic-layer roadmap and re-points that effort at the per-axis verdict log that [[2026-07-03-verify-skills-pass-disagreement-audit]] has been blocked on since July. It also records a resolved citation for arXiv:2605.29800, whose prompt-variation table is the first published evidence RDCO has on whether perturbing the prompt decorrelates a judge panel.

Open follow-ups

Related

Sources

Web, fetched and read this run:

Web, metadata verified but full text not read:

Web, unreachable:

Harness, inspected on disk 2026-10-11:

Vault: