06-reference/research

cpcv deflated sharpe autoinv validation

2026-06-01·research-brief·source: deep-research
cpcvdeflated-sharpebacktest-validationautomated-investingoverfitting

CPCV in autoinv: concrete implementation in validation.py, and the minimum trade count for a trustworthy deflated-Sharpe

The question

Direct build-spec input for the autoinv validation layer (GitHub RayDataCo/automated-investing). Two parts: (1) concretely, how does combinatorial purged cross-validation (CPCV) get implemented in validation.py — function signatures, the path-count math, purge/embargo mechanics; and (2) what is the minimum trade (observation) count below which a deflated-Sharpe number is not trustworthy and should not be reported. This feeds the backtest-thesis harness's honest-statistics requirement (no 2-decimal Sharpe on tiny trade counts) and the Phase-2 anti-data-mining GATE in the strategy-pipeline architecture.

Load-bearing finding up front: the current autoinv/validation.py does NOT implement CPCV, purging, embargo, or a deflated Sharpe at all. It wraps sklearn TimeSeriesSplit (plain walk-forward, single path) plus a four-bias audit checklist; there is no DSR/PBO anywhere in the repo (metrics.py, stats.py, tests confirm). So "how does it concretely implement" is answered as a gap + a spec, not a description of existing code. [verified from source, 2026-06-01]

What we already know (from the vault)

What the web says

Convergences and contradictions

Synthesis for RDCO

1. Concrete validation.py implementation sketch (Phase-2 build). The current module exposes only timeseries_cv(n_splits) → sklearn TimeSeriesSplit and a BiasAudit dataclass. CPCV is a net-new addition that should sit alongside (not replace) the walk-forward path, since the architecture wants both walk-forward and the combinatorial holdout distribution. Proposed signatures:

# autoinv/validation.py  (Phase-2 additions)

@dataclass
class CPCVSplit:
    train_idx: np.ndarray          # post-purge, post-embargo training rows
    test_idx: np.ndarray           # the k test-group rows
    test_groups: tuple[int, ...]   # which of the N groups are test here
    path_id: int                   # which reassembled backtest path this feeds

def combinatorial_purged_cv(
    n_obs: int,
    label_spans: np.ndarray,       # shape (n_obs, 2): [event_start, event_end] per obs
    n_groups: int = 6,             # N
    k_test: int = 2,               # k  -> C(6,2)=15 splits, phi=5 paths
    embargo_frac: float = 0.01,    # h: fraction of n_obs embargoed after each test block
) -> list[CPCVSplit]:
    # 1. partition [0..n_obs) into N contiguous, time-ordered groups
    # 2. for each of C(N,k) combinations of test groups:
    #    a. test_idx = union of those k groups' rows
    #    b. embargo: drop the next ceil(embargo_frac * n_obs) rows after each test block
    #    c. purge: drop any train row whose label_span overlaps any test row's span
    #    d. assign path_id round-robin so phi[N,k]=(k/N)*C(N,k) paths reassemble
    ...

def n_paths(n_groups: int, k_test: int) -> int:
    return (k_test * math.comb(n_groups, k_test)) // n_groups   # phi[N,k]

def deflated_sharpe(
    returns: np.ndarray,           # the realized return series (T observations)
    n_trials: int,                 # effective N (cumulative trial counter, not grid-cell count)
    trial_sharpe_var: float,       # V[SR_n] across trials -> from the CPCV path Sharpes
    benchmark_sr: float = 0.0,
) -> DSRResult:                    # returns observed SR, SR0, DSR, and MinTRL
    # SR0 = sqrt(trial_sharpe_var) * ((1-EM)*Phi_inv(1-1/N) + EM*Phi_inv(1-1/(N*e)))
    # DSR = Phi( (SR_obs - SR0)*sqrt(T-1) / sqrt(1 - skew*SR_obs + (kurt-1)/4 * SR_obs**2) )
    # MinTRL = 1 + (1 - skew*SR_obs + (kurt-1)/4*SR_obs**2) * (Phi_inv(conf)/(SR_obs-SR0))**2
    ...

Reuse what exists: metrics.sharpe_ratio for per-path Sharpes, stats.normality_report to source skew/kurtosis honestly (it already computes both), and feed the CPCV per-path Sharpes as the empirical estimate of trial_sharpe_var. PBO/CSCV is a sibling function (probability_of_backtest_overfitting) over the same path matrix. Keep the existing shuffle=True refusal rule; CPCV is the correct multi-path replacement people reach KFold for. Pair each function with a test the way Phase-2's V-model demands: a pure-noise strategy through a C(6,2) CPCV + DSR must be REJECTED (DSR low, MinTRL exceeding available T).

2. The minimum-trade-count answer (and why it's the wrong tool for RDCO's horizon). DSR is defined over T = number of return observations, and the honest gate is MinTRL: report a DSR only if your available T exceeds the MinTRL implied by your observed Sharpe, skew, kurtosis and benchmark. As a rule of thumb, a strategy whose edge sits near the benchmark needs thousands of daily observations; a clearly-separated edge can qualify in low-hundreds. There is no universal "N trades" cutoff — MinTRL is the principled cutoff and it depends on effect size. For a practical floor, marry it to the existing vault rule: <10 trades → no decimals on Sharpe and no DSR claim at all; 10-50 trades → DSR reportable only with its MinTRL and a wide bootstrap CI; 50+ → DSR as a gate. Below ~30 independent observations the skew/kurtosis estimates feeding the deflation term are themselves too noisy to trust, so the deflation is theater.

The capital-cycle caveat is decisive. RDCO is a position/capital-cycle investor: the chip-fab/memory thesis produces ~5-6 trades across ~40 years and the realized P&L is carried by one or two cycle-bottom entries. On that surface, neither CPCV nor deflated-Sharpe is the right instrument — CPCV needs many near-independent blocks (a thin 5-event history can't supply non-degenerate C(N,k) paths; purging eats most of the training set), and DSR over a handful of trade-returns is dominated by estimation noise in skew/kurtosis. This matches the architecture doc's own ruling: route CPCV/DSR/PBO at the large-sample liquid surfaces (cross-sectional equity momentum, the SPY vol-regime overlay) where C(6,2)=15 splits and 5 paths are well-populated, and keep the rare-state cyclical theses on the v2 multi-cycle harness — leave-one-out by stress episode, bootstrap CIs, dual buy-and-hold benchmarks, integer-Sharpe precision — NOT on DSR.

What to report instead for the capital-cycle book: per-cycle return distribution (not a mean), leave-one-out-by-episode (if dropping the AI-boom cycle flips the verdict, that's the headline), bootstrap CIs on aggregate return, both buy-and-hold benchmarks, and an explicit trade count with integer-precision Sharpe. The single honest number is "we have 5 cycles; here is each one and here is the leave-one-out," not a deflated-Sharpe to two decimals. DSR belongs on the swarm; the cycle thesis belongs on the v2 harness. Wiring DSR to a 5-trade thesis would be exactly the false-rigor the harness was built to forbid.

Open follow-ups

  1. Read the AFML primary (de Prado, Advances in Financial Machine Learning, ch. 7 purged CV + ch. 12 CPCV) and Bailey/de Prado SSRN 2460551 to confirm the embargo-fraction default and the exact PBO/CSCV partition count (the blogs assert "lower PBO / higher DSR" for CPCV but I did not verify magnitudes against the primary). [parent brief open-follow-up #1 still open]
  2. Decide the effective-N estimator. Counted grid-cell N under-deflates (correlated cells); CPCV path Sharpes give a cleaner V[SR_n]. Pre-register whether GATE-1's N comes from the cumulative trial counter, the CPCV path count, or a correlation-adjusted effective-N — this changes every DSR the pipeline ever reports.
  3. Set the MinTRL guard as a hard gate, not a report line. Wire validation.py to REFUSE to emit a DSR when available T < MinTRL(observed SR, skew, kurt, benchmark, confidence), returning "insufficient track record — report distribution instead." This operationalizes the founder's honest-stats rule mechanically.
  4. Pick N and k for the swarm surfaces. N=6,k=2 (15 splits / 5 paths) is the canonical default; confirm it against the available train/validation span length per universe so each of the N groups still holds enough bars after purge+embargo.

Sources

Vault:

Web: