Can a multi-agent mechanism recover expert-level calibration that single LLMs lack — and should RDCO instrument conviction with one instead of a single eval?

The question

Does published work on agent ensembles / debate / market mechanisms recover expert-level calibration where single LLMs fail — could RDCO instrument conviction via a multi-agent targeting system rather than a single eval? (Direct follow-up to the 2026-06-12 brief, which found single-LLM conviction in fuzzy domains is overconfident and below expert calibration; this asks whether a multi-agent mechanism closes that gap enough to justify replacing single-pass /verify-* and investing-conviction evals.)

What we already know (from the vault)

• The seeding brief [[2026-06-12-agentic-targeting-conviction-calibrated-confidence]] found single frontier LLMs trail expert superforecasters ~6x on Brier (o3 0.135 vs expert-median 0.023), are systematically overconfident exactly where stakes are highest, and can verbalize uncertainty but not convert it into risk-sensitive decisions. It explicitly listed this question (ensembles/debate/markets recovering calibration) as an open follow-up.
• [[verifier-as-epistemology]] supplies the decisive prior constraint (Kingsbury): "two correlated estimators voting" is uninformative — agreement only earns epistemic weight when the layers' failure modes are independent. An LLM checking an LLM written under the same discipline is "a single layer wearing a costume." This is the hinge the whole ensemble question turns on.
• [[binary-decision-around-continuous-probability]] argues the calibrated signal lives in the distribution, and collapsing to a binary upstream destroys it — relevant because a multi-agent vote that emits only a majority answer throws away the conviction gradient the mechanism was supposed to produce.
• RDCO already runs the relevant machinery: the verify-* fresh-eyes critics (single-pass, single-LLM) and the 4-seat pipeline-spec/test/code/critic. The [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]] SOP flags an unaddressed "calibration-drift risk where the critic-skills themselves start measuring the wrong thing."

What the web says

• Ensembles partially recover calibration but DON'T reach expert level. "Wisdom of the Silicon Crowd" — a 12-model median-aggregate ensemble hits Brier 0.20, statistically tied with the human crowd (0.19), beating the 0.25 coin-flip baseline and most individual open models via error cancellation. But it did not reduce overconfidence (mean forecast 57% vs 45% base rate) and stays an order of magnitude worse than expert superforecasters (~0.023). Notably the best single frontier model (GPT-4, 0.15) beat the ensemble — aggregation rescues weak/diverse pools, it doesn't beat a strong single judge. (arxiv 2402.19379, Science Advances)
• Vanilla multi-agent debate underperforms simple majority vote despite far higher cost. Gains appear ONLY when two mechanisms are added: agent diversity and explicit calibrated-confidence communication. With both, "High-Diversity + Confidence" beats majority vote (GSM8K 93.2% vs 90.8% Qwen; +9.3 pts Llama) and cuts ECE ~50% and Brier 3-5x (0.217→0.069). (arxiv 2601.19921)
• Homogeneous same-model agents suffer "homogeneity collapse" / "echo chambers" — correlated errors converge on the same wrong rationale, so iterative critique "collapses to confirmation rather than correction." Heterogeneity (different companies' models) raises error-reduction rates; diversity-aware initialization lifts Pass@5 0.792→0.910 without model heterogeneity. (arxiv 2601.19921, ReConcile arxiv 2309.13007)
• Auditing the reasoning tree beats both majority vote and single LLM-as-judge. AgentAuditor: +3% avg over majority vote (peak +5.7%), recovers 65-82% of cases where the majority was wrong (vote recovers 0%), at 45% fewer tokens than naive LLM-as-judge. Mechanism: it discards vote counts and instead compares evidence at the critical divergence points between distinct reasoning branches — explicitly correcting the "confabulation consensus" of correlated errors. Multi-agent reasoning trees function as superior evaluators vs a single judge. (arxiv 2602.09341)
• Debate / deliberation improves calibration when confidence is made explicit and weighted. Confidence-weighted voting across different-backbone LLMs improves both accuracy and calibration; calibration post-hoc (Platt scaling, temperature scaling) layered on top helps further. (arxiv 2404.09127, arxiv 2509.14034)

Convergences and contradictions

• Strong convergence with the vault's central constraint. Every web source independently rediscovers Kingsbury's [[verifier-as-epistemology]] point: a multi-agent mechanism only buys calibration when the agents' errors are independent. Homogeneous ensembles = correlated estimators = no recovery (often worse than majority vote). The vault's philosophical claim is now empirically load-bearing, not just a thought experiment.
• Partial answer, with a ceiling. The literal question — "recover expert-level calibration" — answers no. Ensembles/debate close part of the gap to the human crowd, not to expert superforecasters, and don't fix overconfidence. They are a calibration improver, not a calibration solver.
• Sharpening, not contradiction: the best single strong model can beat a diverse-but-weak ensemble. So "multi-agent > single eval" is conditional on either (a) a genuinely diverse, independent agent pool, or (b) auditing the reasoning structure rather than counting votes — not on quantity of agents alone.

Synthesis for RDCO

The honest verdict is a qualified yes, with a hard precondition that maps exactly onto a known RDCO failure mode. A multi-agent mechanism does improve calibration over a single LLM eval — but only when it satisfies the independence condition the vault already named in [[verifier-as-epistemology]]: the agents' errors must be uncorrelated (diversity of model/viewpoint), and they must communicate explicit confidence, not just answers. Plain "spin up 5 copies of the same model and vote" underperforms a single majority vote and reproduces the same overconfidence. This is the single most important finding: RDCO's current verify-* and pipeline-critic stack runs homogeneous, same-model, same-prompt-discipline agents, which is precisely the "homogeneity collapse / echo chamber" configuration the 2601.19921 study shows yields no calibration gain. We would be paying multi-agent cost for single-agent calibration. So the question "should we replace single-pass evals with a multi-agent targeting system?" cannot be answered yes as currently architected — the lift is not "add more agents," it is "add independence and explicit confidence."

What this changes concretely, by decision class. (1) Instrumentable, gradeable surfaces (pipeline-code/test outputs, vault-write structural checks, PDF rubrics) — the [[verifier-as-epistemology]] answer still wins: a cheap deterministic verifier with LLM-independent failure modes beats any LLM ensemble, and AgentAuditor's result (audit the reasoning divergence, don't count votes) is the LLM-flavored version of the same principle. Don't build a debate system where a Python invariant suffices. (2) Strategic/fuzzy surfaces where verify-strategic-output operates (investing conviction, /decisions/ pages, paper-trade allocation) — here a diverse multi-agent panel with calibrated-confidence output is a real, evidence-backed upgrade over the current single fresh-eyes critic, if we enforce heterogeneity (different model families, deliberately divergent priors / bull-vs-bear seats) and have it emit a probability with a confidence interval rather than a PASS/FAIL bit (the [[binary-decision-around-continuous-probability]] fix). This is the cheapest high-value change: keep the single-agent gate for cost, but for the highest-stakes calls add a 3-seat heterogeneous panel whose spread is itself the conviction signal — wide spread = low conviction, tight independent agreement = earned conviction.

For the investing paper-trade conviction problem specifically, the ceiling matters: even a good ensemble lands near crowd calibration (Brier ~0.20), not superforecaster (~0.023), and does not cure overconfidence. So a multi-agent conviction score is usable as a relative ranker / disagreement detector (which of these theses do the independent seats fracture on?) but must not be sold to the founder as a calibrated probability he can size positions on. The disagreement signal is the asset; the point estimate is not. This is consistent with the 2026-06-12 conclusion that genuine conviction in the un-instrumented residual stays human — a heterogeneous panel narrows the fuzz by surfacing where independent reasoners diverge, which is exactly where the founder's judgment should be spent, but it does not manufacture the calibrated conviction.

Net targeting-system-filter call: a multi-agent conviction mechanism is ANCHORED for the strategic/investing-conviction niche with the bottleneck being "single-critic confirmation bias + binary-collapse of the conviction gradient" — provided the build is "heterogeneous panel + explicit confidence + reasoning-tree audit + probability output," not "N homogeneous voters." Built the naive way it is an un-anchored shiny object that buys cost and no calibration. The concrete next experiment is small: take a handful of past verify-strategic-output verdicts, re-run them through a 3-model heterogeneous panel that emits per-seat probabilities, and check whether panel-spread predicts the cases where the founder later overrode the single critic. If spread tracks overrides, the mechanism has earned its build.

Open follow-ups

• Does a heterogeneous panel's disagreement spread empirically predict founder overrides of the single verify-strategic-output critic? (the cheap validating experiment above — candidate /curiosity, also a concrete /improve task)
• Can RDCO get true independence cheaply without paying for 3 frontier vendors — e.g. does prompt-induced viewpoint diversity (forced bull/bear/base seats) on one model approximate cross-vendor heterogeneity for calibration purposes? (2601.19921 hints yes via diversity-aware init; unverified for strategic domains)
• Is there a deterministic / market-mechanism aggregator (proper scoring rule, log-opinion-pool) that beats naive median for combining the panel's probabilities, and is it worth the complexity at RDCO's volume?
• What is the right binary-collapse point for a multi-agent conviction score — i.e., where in the /decisions/ pipeline should the panel's probability distribution be preserved vs collapsed to a recommendation? (ties to [[binary-decision-around-continuous-probability]])
• Does confidence-alignment contamination (from the 2026-06-12 brief) get worse with a multi-agent system — does a confident-sounding panel import even more unwarranted conviction into the founder than a single critic?

Related

• [[2026-06-12-agentic-targeting-conviction-calibrated-confidence]]
• [[verifier-as-epistemology]]
• [[binary-decision-around-continuous-probability]]
• [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]]
• [[2026-05-22-reward-hacking-patterns-llm-critic-systems]]
• [[2026-05-12-rdco-pipeline-rlhf-shaped]]
• [[2026-04-24-targeting-system]]
• [[feedback_targeting_system_prioritization_filter]]

Sources

• Vault: [[2026-06-12-agentic-targeting-conviction-calibrated-confidence]] — ~/rdco-vault/06-reference/research/2026-06-12-agentic-targeting-conviction-calibrated-confidence.md (seeding brief; single-LLM conviction gap, overconfidence, confidence-alignment contamination)
• Vault: [[verifier-as-epistemology]] — ~/rdco-vault/06-reference/concepts/verifier-as-epistemology.md (Kingsbury independence constraint — correlated estimators voting is uninformative)
• Vault: [[binary-decision-around-continuous-probability]] — ~/rdco-vault/06-reference/concepts/binary-decision-around-continuous-probability.md (calibrated signal lives in the distribution; don't collapse upstream)
• Vault: [[2026-05-20-verify-stack-two-gate-pass-fail-architecture]] — ~/rdco-vault/02-sops/2026-05-20-verify-stack-two-gate-pass-fail-architecture.md (verify-* architecture + calibration-drift risk)
• Vault: [[2026-05-22-reward-hacking-patterns-llm-critic-systems]] — ~/rdco-vault/06-reference/research/2026-05-22-reward-hacking-patterns-llm-critic-systems.md (structured rationale + independence as the defensive primitive)
• Web: Wisdom of the Silicon Crowd — LLM Ensemble Prediction — https://arxiv.org/html/2402.19379v1 ; https://www.science.org/doi/10.1126/sciadv.adp1528
• Web: Demystifying Multi-Agent Debate — The Role of Confidence and Diversity — https://arxiv.org/html/2601.19921
• Web: AgentAuditor — Auditing Multi-Agent Reasoning Trees Outperforms Majority Vote and LLM-as-Judge — https://arxiv.org/html/2602.09341v1
• Web: Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation — https://arxiv.org/html/2404.09127v3
• Web: Enhancing Multi-Agent Debate System Performance via Confidence Expression — https://arxiv.org/html/2509.14034
• Web: ReConcile — Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs — https://arxiv.org/pdf/2309.13007