06-reference/research

strategy discovery loop architecture

2026-06-26·research-brief·source: deep-research
strategy-discoverywalk-forwarddeflated-sharpepbocpcvvectorbtautoinv

The canonical discovery-loop control flow: vectorbt sweeps, autoinv confirms, López de Prado gates

The question

What's the canonical architecture for a systematic-strategy discovery loop — candidate generation → walk-forward validation → deflated-Sharpe/PBO gating → paper-deploy — that vectorbt (vectorized sweep) and the autoinv event engine (high-fidelity confirm) implement together? Context: Phase 2 of RDCO's automated-investing pipeline is parked pending founder go; this brief gives Phase 2 a control-flow spec to build against, with honest statistical reporting (no inflated Sharpe on tiny trade counts) as a non-negotiable.

What we already know (from the vault)

What the web says

Convergences and contradictions

Synthesis for RDCO

The canonical loop, as a Phase-2 control-flow spec. Five stages, each with an explicit gate that a candidate must clear before consuming the next (more expensive) resource. (1) CANDIDATE GENERATION — vectorbt owns it. The discovery swarm sweeps param_space() × point-in-time universe × horizon on the TRAIN/VALIDATION partition only (holdout import-walled out), bounded by a pre-registered trial budget (e.g. ≤2000 cells/run) so N is known a priori, auto-incrementing the cumulative trial counter. Gate here = GATE 0, economic prior: a candidate with an empty economic_prior is hard-rejected at registration, and prior_strength scales the DSR bar it must later clear (a story-less candidate gets a harsher discount, never a free pass). This is the cheapest cull and it stops a pure pattern-miner from ever reaching the exhaustible holdout. vectorbt's job ends at producing a ranked table of {strategy, symbol, horizon, params, in-sample Sharpe, trial_id} — ranked by economic-gated candidacy, never by raw Sharpe.

(2) WALK-FORWARD VALIDATION and (3) DSR/PBO GATING — autoinv (the event engine) owns both. This is the high-fidelity confirm stage and the reason the two engines exist separately: vectorbt's instantaneous-fill, fixed-slippage abstraction generates false positives by construction, so every surviving candidate is re-executed event-driven with the real cost model (maker-limit fills, slippage bps, after-tax/wash-sale-aware net returns) and the fail-closed no-lookahead guard. Inside autoinv: anchored walk-forward (cycle-aligned windows for slow strategies, calendar-stepped for fast), leave-one-out by stress episode, bootstrap CIs, a deliberate future-leak mutation test that MUST change the result. Then the multiple-testing gates: GATE 1, Deflated Sharpe (DSR<0.5 = noise REJECT, DSR>1.0 = real-skill confidence; report raw SR and DSR side by side) and GATE 2, PBO via CSCV/CPCV (high PBO = the search process itself overfits → reject the whole family, not just the cell). The honest correction the canon demands: estimate effective-N and V[SR_n] from CPCV path Sharpes (N=6,k=2 → 5 paths is the canonical default; confirm each group still holds enough bars after purge+embargo), not from correlated grid-cell count. A candidate must clear GATE 0+1+2 AND beat BOTH mandatory buy-and-hold benchmarks before it earns the next stage.

(4) SEALED HOLDOUT — the single arbiter. A HoldoutManager with import-level read access is the only module that can touch the holdout segment; the discovery path receives a feed handle that physically excludes holdout dates and a test asserts it RAISES on any holdout read. A survivor is run on holdout EXACTLY ONCE; a second peek burns that segment (logged, retired). DSR/PBO are necessary-not-sufficient pre-screens — the sealed holdout is the real verdict. The pre-registered go/no-go bar (set BEFORE looking at OOS) is the three-bar wet rehearsal: null-test passes (structure vs iid) → calibration holds OOS within CI → risk-adjusted improvement after cost AND after tax beats both benchmarks across ≥2 leave-one-out episodes. (5) PAPER-DEPLOY GATING. Only a triple-gate + holdout-clearing survivor reaches sizing (vol-target spine → earn-your-Kelly, fixed-fractional 1% until OOS-calibrated edge graduates it to quarter-half Kelly → heat/correlation caps) and execution (Alpaca paper, fractional DAY LIMIT only, never OPG/market-on-open per [[feedback_alpaca_paper_opg_unreliable]]). Live capital is a separate founder-gated DECISION — canonical APPROVE or explicit "deploy" verb, structural agreement is not enough ([[feedback_paper_trade_deploy_authorization]]).

The statistical guardrails, made mechanical. Min trade count: <10 trades → no decimals on Sharpe and NO DSR claim at all; 10-50 → DSR only alongside its MinTRL and a wide bootstrap CI; 50+ → DSR as a hard gate. Below ~30 independent observations the skew/kurtosis feeding the deflation term are too noisy and the deflation is theater. Wire the MinTRL guard as a refusal, not a report line: validation.py returns "insufficient track record — report distribution instead" when available T < MinTRL(observed SR, skew, kurt, benchmark, confidence). Deflation thresholds: DSR<0.5 reject / >1.0 confidence. The mandatory new report section vs today's harness is the multiple-testing ledger (cumulative N, DSR, PBO, trial-Sharpe distribution). And the cultural guardrail the build pre-registers: MOST candidates should FAIL — a pipeline that "discovers" many winners is suspect, and a clean negative (nothing beats buy-and-hold-with-kill-switch after cost+tax) is a SUCCESS of the discipline. The capital-cycle book is explicitly carved OUT of this entire stack: it stays on the v2 leave-one-out-by-episode harness with integer-precision Sharpe, because DSR/CPCV on 5 trades is exactly the false rigor the harness forbids.

Open follow-ups

  1. Pre-register the effective-N estimator (changes every DSR the pipeline ever reports). Does GATE-1's N come from the cumulative trial counter, the CPCV path count, or a correlation-adjusted effective-N? The canon says CPCV paths; v0's sweep currently counts grid cells. Resolve before coding.
  2. Confirm N=6, k=2 against each swarm surface's train/validation span — verify each of the N groups still holds enough bars after purge + embargo, per universe, or pick a different (N,k).
  3. Decide whether the autoinv confirm stage runs CPCV in addition to anchored walk-forward, or instead of it — v0 wants both (walk-forward for the time-ordered view, CPCV for the distribution feeding DSR/PBO); confirm the build carries two CV regimes rather than one.
  4. Pin the holdout-refresh policy — touch-once-then-retire makes the holdout a depleting resource; is a periodic roll-forward reseal the sanctioned replenishment, or a different policy?
  5. Read the AFML primary (de Prado ch. 7 + ch. 12) and SSRN 2460551 to verify the embargo-fraction default and the exact CSCV partition count behind the "lower PBO / higher DSR" claim (cited from secondary sources here, not deep-read).

Related

Sources

Vault:

Web: