Two CV regimes, not one: the autoinv confirm stage carries both anchored walk-forward and CPCV

The question

Should the autoinv confirm stage run CPCV in addition to anchored walk-forward, or instead of it? (v0 wants both — walk-forward for the time-ordered view, CPCV for the distribution feeding DSR/PBO.) This resolves open-follow-up #3 from the 2026-06-26 discovery-loop-architecture brief: confirm the Phase-2 build carries two cross-validation regimes rather than collapsing to one.

What we already know (from the vault)

• [[2026-06-26-strategy-discovery-loop-architecture]] already stakes out "v0 wants both" and flags the open question verbatim: walk-forward for the time-ordered view, CPCV for the distribution feeding DSR/PBO. It also names the honest correction the canon demands — estimate effective-N and V[SR_n] from CPCV path Sharpes at confirm, NOT from correlated grid-cell count at the vectorbt sweep.
• [[2026-06-01-cpcv-deflated-sharpe-autoinv-validation]] supplies the mechanics and the decisive gap finding: the current autoinv/validation.py implements NEITHER — it wraps sklearn TimeSeriesSplit (a single walk-forward path) plus a four-bias audit, with no CPCV/purge/embargo/DSR/PBO anywhere in the repo. Its build sketch explicitly adds CPCV "alongside (not replace) the walk-forward path, since the architecture wants both." It gives the path math (φ[N,k]=(k/N)·C(N,k); N=6,k=2 → 15 splits / 5 paths) and the DSR/MinTRL formulas.
• [[2026-05-29-strategy-pipeline-architecture-v0]] specs the confirm/validation layer as GATES-not-reports: GATE-0 economic prior → GATE-1 Deflated Sharpe (DSR<0.5 reject / >1.0 confidence) → GATE-2 PBO via CSCV, then exactly one sealed-holdout touch. Its walk-forward discipline (anchored, cycle-aligned or calendar-stepped, leave-one-out by stress episode, bootstrap CIs, future-leak mutation test) is inherited wholesale from the v2 harness and must not be regressed.
• Both briefs carve the capital-cycle book (chip/memory thesis, ~5-6 trades in ~40yr) OUT of this entire stack: CPCV can't form non-degenerate C(N,k) paths and DSR over a handful of trade-returns is skew/kurtosis noise. The two-regime confirm stage below applies only to the large-sample liquid swarm surfaces (cross-sectional equity momentum, SPY vol-regime overlay).

What the web says

• Walk-forward "preserves temporal order but evaluates the model on a single sequence of test sets... results are contingent on a specific historical path"; it is "the industry standard for realistic trading simulation" but tests "just one single scenario, which is easily overfit" and is high-variance (ScienceDirect controlled-environment comparison; search-tier syntheses).
• CPCV "systematically constructs multiple train-test splits, purging overlapping samples and enforcing an embargo... the result is a distribution of out-of-sample performance estimates, enabling robust statistical inference" — many reassembled backtest paths, each a plausible market scenario (Towards AI, CPCV method).
• The controlled study finds CPCV "demonstrates marked superiority in mitigating overfitting risks... lower Probability of Backtest Overfitting (PBO) and superior Deflated Sharpe Ratio (DSR)" versus walk-forward/k-fold — precisely because single-path methods cannot furnish the OOS distribution the metrics need (ScienceDirect).
• The recent framing states the trade-off as a division of labor, not a winner: "Combinatorial Purged Cross-Validation shows superiority in mitigating over-fitting risks, though walk-forward remains the industry standard for realistic trading simulation." CPCV prioritizes robustness across scenarios/distributions; WF prioritizes a realistic single historical sequence.
• The two-engine split RDCO already chose (vectorbt sweep → autoinv event-driven confirm) is the canonical practitioner pattern: "Vectorized for research. Event-driven for anything that will ever touch real money" (Hasan Javed, Best Python Backtesting Libraries 2026) — the confirm stage is where realism is bought, which is exactly why the realistic single path (WF) belongs there and is not redundant with the distribution (CPCV).
• Caveat on a weak source: one secondary post (insightbig, "Traditional Backtesting is Outdated. Use CPCV Instead") argues CPCV should replace traditional backtesting and implies purging forces training to precede testing. That overstates it — de Prado's CPCV assigns train/test groups combinatorially, so training for a given test block can include chronologically later groups; purge/embargo only clean label-overlap at block boundaries. That mechanic is the crux (below), and it is why CPCV does NOT subsume walk-forward.

Convergences and contradictions

• Convergence: vault and web agree the two regimes answer genuinely different questions — WF = one realistic, sequence-aware equity path (deployment-realistic simulation); CPCV = a distribution of OOS Sharpes (statistical inference / overfit measurement). Neither is a superset of the other.
• The load-bearing distinction (resolves the "does one subsume the other" tension): CPCV's combinatorial train/test assignment lets a test block be trained on future blocks (purged/embargoed only at the seams). That is deliberate — it maximizes independent OOS paths for a clean distribution — but it means a CPCV path is NOT how the strategy would have actually been deployed (you cannot train on the future in production). Anchored/expanding walk-forward only ever trains on the past, so it is the sole regime that honestly simulates live sequential retraining and preserves path-dependent risk (drawdown sequence, duration, real compounding). One weak source disagrees on the replace-vs-complement question; the methodological/controlled-study tier and the vault both land on complement.

Synthesis for RDCO

Recommendation: carry BOTH regimes in the confirm stage. This is not a hedge — each answers a question the other structurally cannot, and dropping either removes a distinct guarantee. v0's instinct is correct and should be locked. The 2026-06-01 build sketch already had it right: CPCV is added alongside walk-forward in validation.py, not in place of it.

What anchored walk-forward buys, and what is lost if you drop it. WF produces the single time-ordered equity path — the honest "what would actually have happened if we deployed this and retrained forward." Because it only ever trains on the past, it is the only regime that respects live-deployment causality, and it is the only one that preserves path-dependent metrics: max drawdown, drawdown duration, sequence-of-returns risk, realistic compounding, and (in the event-driven autoinv engine) realistic maker-limit fills, slippage, and after-tax/wash-sale net returns. A CPCV path is a reassembled chronology whose training used future blocks, so its drawdown sequence is a statistical artifact, not a deployable history. Drop WF and you lose deployment-realism and every path-dependent risk number — the exact realism the vectorbt→autoinv two-engine split exists to buy. WF also carries the inherited v2 discipline (leave-one-out by stress episode, bootstrap CIs, the future-leak mutation test) that must not regress.

What CPCV buys, and what is lost if you drop it. CPCV produces a distribution of OOS Sharpes across φ[N,k] reassembled paths (N=6,k=2 → 5 paths is the canonical default). That distribution is the only statistically valid input to the two multiple-testing gates: DSR needs V[SR_n] (the cross-sectional variance of trial Sharpes) and an effective-N, and PBO/CSCV needs many near-independent partitions to ask "does the in-sample winner fall below median OOS." A single walk-forward path is one number — it cannot furnish a variance, an effective-N, or a rank distribution, so it mathematically cannot feed DSR or PBO honestly. Drop CPCV and GATE-1/GATE-2 either become non-computable or fall back to the counted grid-cell N, which under-deflates because dense grid cells are correlated (v0's own red-team flag, and the canon's "effective N, not literal N"). CPCV is precisely the mitigation for that under-deflation: its near-independent path Sharpes are a cleaner V[SR_n] and effective-N than any grid-cell count.

Concrete build guidance for validation.py (Phase-2). Two coexisting functions, feeding different consumers. (1) The anchored walk-forward path (extend/keep the existing TimeSeriesSplit-based timeseries_cv into the anchored, cycle-aligned form the v2 harness specs) produces the single reported equity curve and all path-dependent metrics (drawdown, duration, sequence risk, after-cost/after-tax net return) plus the leave-one-out-by-episode table and bootstrap CIs. (2) combinatorial_purged_cv(n_obs, label_spans, n_groups=6, k_test=2, embargo_frac=0.01) produces the CPCV path matrix; deflated_sharpe(...) and probability_of_backtest_overfitting(...) consume the per-path Sharpe distribution off that matrix — this is what feeds GATE-1 (DSR) and GATE-2 (PBO). Pre-register that GATE-1's effective-N and V[SR_n] come from CPCV path Sharpes, not the cumulative grid-cell counter (open-follow-up #1 from the 2026-06-26 brief — resolve before coding). Both regimes run inside the event-driven autoinv confirm stage on candidates that already cleared the vectorbt sweep + GATE-0 economic prior; a candidate must pass the CPCV-fed DSR/PBO gates AND show an acceptable walk-forward equity path AND beat both buy-and-hold benchmarks before it earns its single sealed-holdout touch. Keep the capital-cycle book off this stack entirely — it stays on the v2 leave-one-out harness with integer-precision Sharpe.

Open follow-ups

• Pre-register the effective-N / V[SR_n] estimator as coming from CPCV path Sharpes (not the grid-cell counter) — this changes every DSR the pipeline reports (carried from 2026-06-26 #1; now blocking Phase-2 coding).
• Confirm N=6,k=2 holds enough bars per group after purge+embargo for each swarm surface's train/validation span, or pick a different (N,k) per universe (carried from 2026-06-26 #2).
• Decide whether the walk-forward equity path is itself a hard gate (e.g. a max-drawdown / drawdown-duration ceiling) or a report-only realism check feeding the founder-facing go/no-go — CPCV feeds the statistical gates, but WF's role as gate vs. report is unspecified.
• Read the AFML primary (de Prado ch. 7 purged CV + ch. 12 CPCV) and SSRN 2460551 to verify the embargo-fraction default and the exact CSCV partition count behind the "lower PBO / higher DSR" claim (still cited from secondary sources).
• Specify how the CPCV purge consumes label_spans from the anchor/phase-driven exit logic so purging is correct for hold-through-cycle strategies whose label horizons are long.

Related

• [[2026-06-26-strategy-discovery-loop-architecture]] — the parent control-flow brief; this resolves its open-follow-up #3 (confirm two CV regimes)
• [[2026-06-01-cpcv-deflated-sharpe-autoinv-validation]] — CPCV path math, DSR/MinTRL formulas, the "add alongside, not replace" build sketch, current validation.py gap
• [[2026-05-29-strategy-pipeline-architecture-v0]] — the three-GATE validation design and the inherited anchored-walk-forward discipline
• [[2026-06-01-ensemble-admission-correlation-threshold]] — where ensembles slot in (post-validation only), same confirm-stage surface
• [[01-projects/automated-investing/index]] — the RayDataCo/automated-investing repo pointer; Phase-1 spine built, Phase-2 (the gate) a founder go

Sources

Vault:

• ~/rdco-vault/06-reference/research/2026-06-26-strategy-discovery-loop-architecture.md — "v0 wants both" and the verbatim open question; CPCV-paths-for-effective-N correction
• ~/rdco-vault/06-reference/research/2026-06-01-cpcv-deflated-sharpe-autoinv-validation.md — validation.py gap (neither CPCV nor DSR built), φ[N,k] path math, "add alongside not replace," DSR/MinTRL formulas
• ~/rdco-vault/01-projects/investing/2026-05-29-strategy-pipeline-architecture-v0.md — three-GATE confirm design, anchored walk-forward + leave-one-out discipline, sealed-holdout, capital-cycle carve-out
• ~/rdco-vault/06-reference/research/2026-06-01-ensemble-admission-correlation-threshold.md — post-validation ensemble admission gate
• ~/rdco-vault/01-projects/automated-investing/index.md — repo pointer; Phase-2 gate deferred

Web:

• Backtest overfitting comparison, synthetic controlled environment — https://www.sciencedirect.com/science/article/abs/pii/S0950705124011110 (CPCV → lower PBO / higher DSR vs walk-forward; single-path methods can't furnish the OOS distribution; WF = realistic single sequence, high variance)
• The Combinatorial Purged Cross-Validation method (Towards AI) — https://towardsai.com/p/l/the-combinatorial-purged-cross-validation-method (distribution of OOS paths; C(N,k) splits; purge/embargo)
• Best Python Backtesting Libraries 2026 (Hasan Javed) — https://hasanjaved.me/blog/best-python-backtesting-libraries-2026/ (vectorized-for-research / event-driven-for-realism; confirm stage is where realism is bought)
• Traditional Backtesting is Outdated. Use CPCV Instead (insightbig) — https://www.insightbig.com/post/traditional-backtesting-is-outdated-use-cpcv-instead (weak-source replace-not-complement view; overstates purging as forcing past-only training — flagged, not relied on)