06-reference/research

ensemble admission correlation threshold

2026-06-01·research-brief·source: deep-research
ensemble-admissionstrategy-correlationdiversificationoverfittingsystematic-trading

When are two systematic strategies "independent enough" to diversify — not double-count the same overfit signal?

The question

The 2026-05-31 ensembles brief settled that ensembles only reduce variance when each component is independently validated FIRST; correlated components just launder data-mining. It left an explicit follow-up (#2): decide and pre-register the constituent-correlation ceiling for ensemble admission. This brief defines that admission gate mechanically for Phase-2/Phase-5 of the automated-investing build. The hard part is that "correlation" is doing two different jobs — and a single pairwise return-correlation threshold (the obvious ρ < 0.7 rule) is provably insufficient on its own.

What we already know (from the vault)

What the web says

Convergences and contradictions

Strong convergence: every quality source agrees the unit of diversification is the effective number of independent bets, not raw pairwise ρ, and that correlation is a continuous, regime-dependent quantity — Carver explicitly declines a hard threshold and works from the whole correlation matrix; Meucci/ENB formalizes "how many bets do I actually have" as entropy over decorrelated factors. All sources agree ρ near 0.5 already destroys most of the benefit.

The contradiction that matters: the practitioner world offers a tidy "−0.3 to +0.3 = independent" rule of thumb, while the rigorous sources (Carver, Meucci) refuse a fixed cutoff because the honest answer is matrix-level and OOS-conditional. Resolution: RDCO needs a pre-registrable mechanical gate (the rule-of-thumb's job) but must back it with the rigorous logic — so the gate is a threshold PLUS a shared-feature veto PLUS an OOS-stability requirement, not a bare number. The single biggest trap, on which vault and web fully agree, is that low return-correlation does not imply independence when a shared overfit feature is present — return-ρ is necessary but not sufficient.

Honest gap: no source supplies a peer-reviewed, calibrated "admit if ρ < X" number for strategy ensembles specifically; the numbers cited (1.33 effective bets at ρ=0.5; −0.3/+0.3 rule) are practitioner figures, not validated constants. The threshold below is a defensible pre-registration, not an empirically optimal one.

Synthesis for RDCO — the Phase-2/Phase-5 admission rule

Admit a component into an ensemble only if it clears, in order, a three-part gate. All three are mandatory; passing return-correlation alone is explicitly not enough.

Gate A — Independent validation first (inherited, non-negotiable). Each constituent must already have passed the economic-prior + DSR + PBO + dual-buy-and-hold gates and earned exactly one clean holdout evaluation as a standalone strategy. No gate-failed cell is admissible — the exit_class-style hard-reject pattern generalizes here. Correlation is never a rescue path; it is only ever a filter applied to already-validated survivors. This is the load-bearing rule the parent brief established and it sits above any number.

Gate B — Shared-feature veto (the part a ρ rule misses). Before measuring any correlation, compare the constituents' declared required_features (the architecture already makes strategies declare these). If two candidates load on the same primary feature — both are momentum families, both key off realized_vol_21d, both are mechanical-exit variants of the same signal — they are the same bet regardless of measured return-ρ and are rejected as ensemble partners. This directly blocks the "light up together, fail together" failure and the vault's disproven-mechanical-exit trap. A low in-sample ρ between two same-feature strategies is treated as evidence of overfit-noise cancellation, not diversification.

Gate C — Out-of-sample return-correlation threshold AND effective-bet floor. Measure pairwise return correlation on the sealed-holdout segment only (vol-scaled subsystem returns, per Carver, not raw), never on the search partition. Use a pre-registered ceiling of ρ ≤ 0.5 absolute as the admission gate — chosen because ρ=0.5 already collapses two strategies to ~1.33 effective bets, so anything above it is paying for diversification you don't receive; a tighter |ρ| ≤ 0.3 is the preferred target consistent with the "independent" rule of thumb and the existing 2R/4R cluster-cap philosophy. Reject ρ < 0 only if it looks engineered (a strategy fit to be the negative of another is overfit, not diversifying). Then compute the ensemble-level Effective Number of Bets via MLT (not PCA — PCA produces the ENB≈1 false-negative shown above) and require ENB ≥ 0.7 × k for a k-constituent ensemble: the blend must deliver at least 70% of the independence its constituent count implies, or it is just leverage on one bet and is rejected. ENB is the right summary statistic because it generalizes the pairwise rule to the whole matrix, exactly as Carver and Meucci argue.

Why these specific numbers. ρ ≤ 0.5 is the outer admission bound (below it, diversification is at least net-positive after the √N math); |ρ| ≤ 0.3 is the target (where the practitioner consensus puts genuine independence); ENB ≥ 0.7k is the matrix-level backstop that catches the case where every pair is individually under 0.5 but the cluster still collapses (three strategies each pairwise-0.45 are not three bets). All three are pre-registered before the holdout is touched, all three count the ensemble as its own N-incrementing candidate per the existing multiple-testing ledger, and the whole gate runs on the holdout side of the import-level wall so the correlations can't be gamed in search. Default posture is unchanged: most ensembles should fail this gate and lose to the single best validated constituent — a clean rejection is a success of the discipline.

Open follow-ups

  1. Calibrate ρ ≤ 0.5 vs |ρ| ≤ 0.3 against RDCO's actual survivor set. Once Phase-3 produces a handful of validated survivors, measure their realized OOS cross-correlations to confirm 0.5 is the right admission outer-bound and 0.3 the right target — the numbers here are pre-registered defaults, not fitted constants.
  2. Pick the MLT-ENB implementation and add it to validation/. ENB-via-MLT needs the minimal-linear-torsion decorrelation; confirm whether to vendor an existing implementation (Meucci's reference code / PortfolioOptimizer-style) or build the thin version, and wire it as a Phase-5 gate output alongside holdout DSR.
  3. Define the shared-feature taxonomy. Gate B needs a machine-checkable notion of "same primary feature." Decide whether to tag each strategy with a feature_class at registration (momentum / mean-reversion / vol-regime / breakout / carry) so the veto is mechanical rather than a judgment call.
  4. OOS correlation-stability test. Add a check that in-sample and holdout pairwise correlations don't diverge beyond a band — a large in-sample-low / holdout-high gap is itself an overfit signal and should fail the constituent, not just the ensemble.

Sources

Vault:

Web: