Does a principled aggregator (geometric mean of odds, log-opinion-pool, extremizing) beat the naive median for combining a panel's probabilities — and is it worth the complexity at RDCO's volume?

The question

Is there a deterministic / market-mechanism aggregator (proper scoring rule, log-opinion-pool) that beats naive median for combining a panel's probabilities, and is it worth the complexity at RDCO's volume? Surfaced as an open follow-up from the 2026-06-16 deep-research brief on multi-agent ensembles; RDCO panels (verify-strategic-output, /decisions/ conviction scoring) emit several independent probability estimates that currently collapse to a single recommendation.

What we already know (from the vault)

• The seeding brief [[2026-06-16-multi-agent-ensembles-conviction-calibration]] already established the load-bearing precondition: aggregation only buys calibration when the agents' errors are independent. It explicitly listed this aggregator question as an open follow-up, and concluded the conviction asset is panel disagreement spread, not the point estimate.
• [[brier-score]] is the vault's proper-scoring-rule anchor: it documents that Brier and log-loss are both proper (honest reporting minimizes expected score), that log-loss punishes confident-wrong far harder than Brier, and — critically — that small samples give noisy Brier estimates (N=20 means little). This directly governs whether RDCO can ever measure one aggregator beating another at its volume.
• [[binary-decision-around-continuous-probability]] argues the calibrated signal lives in the distribution; collapsing the panel to a single recommendation upstream destroys the conviction gradient. Any aggregator choice is downstream of this — the question only matters if we preserve the distribution long enough to aggregate it.
• [[verifier-as-epistemology]] (Kingsbury) supplies the independence constraint that the extremizing literature independently rediscovers: correlated estimators voting is uninformative.

What the web says

• Ranked empirically (Satopää/Mellers/Tetlock et al 2014, 1,300 forecasters / 69 geopolitics questions): extremized geometric-mean-of-odds > geometric mean of odds > arithmetic mean > median. The naive median is the worst of the standard family. (EA Forum: geometric mean of odds)
• The size of the win is small in the normal range. When member probabilities sit between 10-90%, the absolute difference between methods is typically 0-3%. The gap only becomes large (up to ~18%) with extreme inputs (sub-1% or >99%). (EA Forum: geometric mean of odds)
• Weighting by forecaster quality dominates the choice of aggregation method. On ~850 Metaculus questions, weighted-geometric beat unweighted methods by far more than geometric beat arithmetic. The practical lever is who you trust, not which mean. (EA Forum: how to aggregate forecasts)
• Geometric mean of odds is the recommended default — it outperforms arithmetic mean and median and is Bayesian-consistent (it behaves correctly as aggregation of independent prior-updates). It is a closed-form deterministic formula, not a market or training loop. (EA Forum: how to aggregate forecasts)
• Extremizing is justified ONLY by information diversity, and backfires on correlated/homogeneous panels. Tetlock's GJP found extremizing helped the diverse crowd but that superforecaster teams needed little or none — they already share each other's arguments. The theoretical condition (projective substitutes) fails when forecasters have overlapping information: "the less variance there is in experts' forecasts, the less it makes sense to extremize." (EA Forum: principled extremizing, researchgate: Two Reasons to Make Aggregated Forecasts More Extreme)
• Recent data says the extremizing gain is marginal and inconsistent. Rolling analysis on Metaculus showed extremization provides unstable, situation-dependent lift on recent questions — it is not a free win you can hardcode. (EA Forum: geometric mean of odds)

Convergences and contradictions

• Hard convergence with the vault's independence constraint. The extremizing literature's "projective substitutes / information diversity" requirement IS Kingsbury's [[verifier-as-epistemology]] independence condition, arrived at from forecasting rather than philosophy. Both say: the fancy aggregator's edge over the median is entirely a function of how independent the inputs are. A homogeneous same-model RDCO panel sits exactly where extremizing is documented to backfire.
• No contradiction, but a scope-narrowing. A deterministic aggregator (geometric mean of odds) does beat the median — that part is a clean yes. But the win is small in the normal probability range, and the extremizing layer (where the big wins live) is precisely the part that is unsafe to apply to RDCO's correlated panels.
• The "market-mechanism" half of the question is a red herring at this scale. Prediction-market scoring rules (LMSR, etc.) are mechanisms for incentivizing many self-interested participants to reveal beliefs; with 3-7 cooperative agents there is no incentive problem to solve, so the market machinery buys nothing the closed-form pool doesn't.

Synthesis for RDCO

The literal question answers yes, narrowly: geometric mean of odds is a deterministic, closed-form aggregator that reliably beats the naive median, it is the consensus default in the serious forecasting literature, and it is Bayesian-coherent. Replacing median(panel) with geometric-mean-of-odds(panel) in the /decisions/ conviction pipeline is a ~5-line change with no new infrastructure, no second model call, and no training loop. On pure cost-benefit that swap is worth doing — it is nearly free and weakly dominant. So if the pipeline is going to collapse the panel to one number anyway, it should collapse via geo-mean-of-odds, not median or arithmetic mean.

But the honest framing is that this is a rounding-error optimization relative to the two things that actually move calibration, and both were already named in the 2026-06-16 brief. First, weighting beats method choice by a wide margin — and RDCO has no per-seat track record to weight on, because at a handful of decisions per week the panel seats accumulate gradeable outcomes far too slowly to estimate trustworthy weights. Second, extremizing — the only part of the aggregator family with a large documented edge — is unsafe for RDCO's panels specifically, because our seats are homogeneous (same model, same prompt discipline) and therefore share information, which is exactly the regime where Tetlock's own data shows extremizing should be near-zero and where applying a hardcoded extremizing factor would manufacture false confidence. RDCO would be importing the literature's headline win in the one configuration where it inverts.

The volume question is decisive and cuts against complexity. From [[brier-score]]: you cannot distinguish two aggregators that differ by 0-3% absolute probability without a large held-out sample of resolved decisions. At a handful of decisions a week, most of them fuzzy strategic calls that may not cleanly resolve for months or ever, RDCO will never accumulate the N to prove one aggregator beat another. Building extremizing-calibration machinery, per-seat weighting, or a market mechanism would be optimizing a parameter we structurally cannot measure — pure motion, no instrumented feedback loop, which fails the targeting-system filter [[feedback_targeting_system_prioritization_filter]] on the instrumentation layer.

Net recommendation, concretely: (1) Do swap median → geometric mean of odds wherever the panel is already being collapsed to a point estimate — it is free and correct. (2) Do not build extremizing, per-seat weighting, or any market/scoring-rule training mechanism; the lift is unmeasurable at our volume and the one high-lift technique (extremizing) is contraindicated for homogeneous panels. (3) Reinvest the complexity budget upstream, exactly where the seeding brief pointed: preserve and surface the panel's disagreement spread rather than perfecting the aggregate point. The geo-mean is a fine summary stat; the spread is the conviction signal the founder should actually see. Don't polish the mean of a distribution we agreed not to collapse.

Open follow-ups

• Does the /decisions/ pipeline currently collapse the panel with median, arithmetic mean, or something else — i.e. is there even a median-to-geomean swap to make, or is the point already moot? (cheap code audit, not research)
• If RDCO ever moves to a genuinely heterogeneous panel (different model families / forced bull-bear-base seats, per the seeding brief), does a modest fixed extremizing factor (~1.5, well below GJP's 2.5) then earn its keep — and can we validate it on the back-catalog of resolved paper-trade calls? (candidate /curiosity once heterogeneity exists)
• Is there a principled non-parametric way to convert panel spread into a displayed confidence band for the founder, so the conviction gradient survives to the decision surface without a fake point-estimate precision? (ties to [[binary-decision-around-continuous-probability]])
• For the paper-trade conviction case specifically, would geo-mean-of-odds vs median have changed any past sizing decision by enough to matter — i.e. retro-test the swap on the existing decision log before committing to it?

Related

• [[2026-06-16-multi-agent-ensembles-conviction-calibration]]
• [[2026-06-12-agentic-targeting-conviction-calibrated-confidence]]
• [[brier-score]]
• [[binary-decision-around-continuous-probability]]
• [[verifier-as-epistemology]]
• [[feedback_targeting_system_prioritization_filter]]

Sources

• Vault: [[2026-06-16-multi-agent-ensembles-conviction-calibration]] — ~/rdco-vault/06-reference/research/2026-06-16-multi-agent-ensembles-conviction-calibration.md (seeding brief; independence precondition, disagreement-spread-as-asset, this question as open follow-up)
• Vault: [[brier-score]] — ~/rdco-vault/06-reference/concepts/brier-score.md (proper scoring rules, log-loss vs Brier, small-sample noise — governs measurability at RDCO volume)
• Vault: [[binary-decision-around-continuous-probability]] — ~/rdco-vault/06-reference/concepts/binary-decision-around-continuous-probability.md (preserve the distribution; don't collapse upstream)
• Vault: [[verifier-as-epistemology]] — ~/rdco-vault/06-reference/concepts/verifier-as-epistemology.md (independence constraint mirrored by extremizing's projective-substitutes condition)
• Vault: [[feedback_targeting_system_prioritization_filter]] — targeting-system filter (instrumentation layer fails at low decision volume)
• Web: When pooling forecasts, use the geometric mean of odds — https://forum.effectivealtruism.org/posts/sMjcjnnpoAQCcedL2/when-pooling-forecasts-use-the-geometric-mean-of-odds
• Web: My current best guess on how to aggregate forecasts — https://forum.effectivealtruism.org/posts/acREnv2Z5h4Fr5NWz/my-current-best-guess-on-how-to-aggregate-forecasts
• Web: Principled extremizing of aggregated forecasts — https://forum.effectivealtruism.org/posts/biL94PKfeHmgHY6qe/principled-extremizing-of-aggregated-forecasts
• Web: Satopää, Baron, Foster, Mellers, Tetlock, Ungar (2014) — Two Reasons to Make Aggregated Probability Forecasts More Extreme — https://www.researchgate.net/publication/275937752_Two_Reasons_to_Make_Aggregated_Probability_Forecasts_More_Extreme