01-projects/investing/plugin/skills/backtest-thesis

SKILL

investing-backtest-thesis (v2)

Run a multi-cycle out-of-sample backtest against a documented thesis's locked entry/exit rules. Produces a backtest report the founder can sanity-check for overfit, selection bias, and structural validity before authorizing paper deployment.

Status: scaffold v2 (2026-05-18). Implementation (vectorbt or backtrader harness, multi-symbol universe constructor, per-cycle driver, leave-one-out aggregator) lands in a future build session. This file is the SOP shape future sessions execute against.

Why this version exists (founder critic, 2026-05-17)

The v1 skill produced a memory-cycle backtest at 01-projects/investing/backtests/2026-05-17-memory-cycle-v1-walk-forward.md (proxy) and 2026-05-17-memory-cycle-v1-real-anchor-rerun.md (real anchor). The founder critic correctly flagged seven load-bearing methodology problems. v2 is designed against each:

  1. Calendar-year fold cuts were a methodology artifact, not a strategy choice. v1 used "rolling 3yr train / 1yr test" walk-forward sliced on calendar years. The strategy is about memory CYCLES, not Jan-Dec windows. v2 aligns windows to thesis phases, which is what the strategy actually reasons over. See "Test windows" below.

  2. N=7 with one dominant outlier fold (2025) made parameter sweeps noisy. v2 requires reporting leave-one-out — if dropping the dominant cycle changes the headline, that IS the headline.

  3. Survivorship bias. v1 backtested MU and SMH, both chosen NOW knowing they won the AI-boom. v2 constructs universe at each window START using thesis-rule criteria (e.g. "any US-listed DRAM/NAND pure-play with >$5B market cap at window start"). Names that subsequently delisted or cratered are INCLUDED at their start-of-window state.

  4. Not really walk-forward. v1 was independent fixed-rule runs across folds, not parameter re-calibration on train then evaluation on test. v2 either DOES the train/test split honestly (with rolling re-estimation if any optimizable params exist), OR explicitly documents "fixed-rule out-of-sample replicated across N cycles" — no oversell.

  5. "Thesis empirically supported" was circular. v1 concluded the thesis was healthy because buy-and-hold was up. That's a buy-and-hold-Nasdaq statement, not a memory-cycle-theory statement. v2 separates THESIS validation (anchor data behaved as theory predicts) from STRATEGY validation (rules captured the predicted moves).

  6. Sharpe on ~3-6 trades reported with 2-decimal precision implied false rigor. v2 reports aggregates with confidence intervals, or shows distributions across cycles. Trade-count-aware precision: <10 trades → no decimals; 10-50 → 1 decimal; 50+ → 2 decimals.

  7. "Hold-to-fold-end" trap. v1's artificial exits at fold boundaries interacted with the AI-boom endpoint. v2 holds positions through window boundaries; only strategy exit logic or the backtest end date closes positions.

Plus: the v1 base case should have been "buy-and-hold + thesis-archival kill switch" (a hold-until-thesis-fails comparison), not v1's own strategy. v2 makes this base case mandatory.

What v2 requires

Inputs

The thesis file must declare:

If any are missing, fail loudly. Backtesting an under-specified thesis is the overfit trap.

Multi-symbol universe construction (handles survivorship bias)

At each cycle-window START date:

  1. Pull the list of US-listed equities (via a point-in-time symbol source — e.g. CRSP, or a curated dump) that satisfy the thesis's universe-construction rule on that date.
  2. Snapshot universe to ~/.claude/state/backtest-cache/<thesis-id>/universe-<window-start>.csv with the criteria check evidence.
  3. Universe membership stays fixed for the duration of that window — names that delist or get acquired mid-window get exited at last available price (recorded as a real exit).
  4. Universe REFRESHES at each new window start; a name absent in the prior window can enter, a name present in the prior window can drop.

Honest limitation: truly point-in-time fundamentals data (market cap on 1997-12-31) requires CRSP or Compustat subscriptions we don't have. v2 approximations: use Wikipedia historical S&P 500 / sector ETF constituent lists, Stooq delisted-ticker archives, and EDGAR 10-K filing dates. Document the data source per window so the gap is visible.

Test windows (aligned to thesis phases, not calendar years)

For the memory thesis, the minimum cycle set is:

Window Anchor event Phase coverage
1997-2002 Asian financial crisis crash → recovery 1 full memory cycle
2003-2008 Early-2000s cycle 1 full cycle
2008-2010 GFC crash → recovery 1 partial cycle (crash dominant)
2014-2019 Mid-decade cycle + 2018 crash 1+ cycles incl. major crash
2019-2022 COVID era 1 cycle (COVID crash + post-COVID recovery)
2022-2024 Post-COVID memory glut → AI recovery 1 cycle
2024-2025+ Current AI era 1 partial cycle (in progress)

That's 7+ cycle-windows, not 7 calendar-year folds. Each window is sized by the thesis-phase length, not by an arbitrary annual cut.

For other theses (power, space, longevity, elon-verse), windows are constructed analogously from phase-history.csv produced by /investing:label-historical-phases. Cycle windows are an INPUT to this skill, sourced from phase labels — not a hardcoded calendar grid.

No artificial fold-end exits

The backtest framework provides time windows; the strategy provides exit rules. A position opened mid-1997 in a 1997-2002 window holds through 1999, 2000, 2001 until either the strategy's exit logic fires or the window ends. If the window ends with the position open, that's a real held position carried into the next window's mark-to-market (NOT an artificial sell-to-cash). The "hold-to-fold-end" trap from v1 is the artifact this rule prevents.

Walk-forward only when parameters re-estimate

Two valid modes:

The skill MUST emit mode: A or mode: B in the report frontmatter.

Anchor-data-driven phase markers (not price-derived)

For thesis-faithful tests, phase markers come from anchor data, not price:

These anchors are consumed via phase-history.csv from /investing:label-historical-phases. Strategy rules reference phase-state, not raw anchor values, so the strategy is decoupled from anchor-data-format changes.

Honest statistical reporting

Buy-and-hold benchmarks (both required)

Strategy must beat BOTH benchmarks to claim alpha:

If strategy beats one but not the other, name which one and why. If strategy beats neither, ARCHIVE.

Procedure

Step 1 — Validate thesis spec

Read the thesis file. Confirm status: locked. Confirm universe-construction rule is machine-readable. Confirm exit rules are explicit (or explicitly "none — hold-to-thesis-archive"). Refuse if any gate fails.

Step 2 — Load phase history

Read ~/rdco-vault/01-projects/investing/anchors/<thesis-id>/phase-history.csv. If missing, instruct caller to run /investing:label-historical-phases <thesis-id> first and exit.

Cycle windows derive from phase-history transitions; for the memory thesis, each "demand → capacity-announce → capacity-online → down-cycle → recovery" sequence defines one window.

Step 3 — Construct per-window universes

For each cycle window start date, apply the thesis's universe-construction rule. Cache result to ~/.claude/state/backtest-cache/<thesis-id>/universe-<window-start>.csv with evidence trail (which data source, which date snapshot).

Step 4 — Fetch historical bars

Default: yfinance via pandas (free, covers delisted tickers reasonably for major US listings). Alpaca historical bars as a secondary source for survivors. Cache fetched bars under ~/.claude/state/backtest-cache/<thesis-id>/bars/<ticker>.parquet.

For delisted tickers that yfinance can't resolve: Stooq archive or skip with a documented gap (which IS a survivorship-bias confession the reader should see).

Step 5 — Run per-cycle simulations

For each cycle window:

  1. Open positions per entry rules (referencing phase markers from phase-history).
  2. Hold through the window. No artificial exits.
  3. Strategy-defined exits fire when their conditions trigger.
  4. Window-end mark-to-market for any still-open positions.
  5. Record per-cycle return, max DD, trade count, win rate.

Step 6 — Aggregate honestly

Compute:

Step 7 — Write the report

File at ~/rdco-vault/01-projects/investing/backtests/<YYYY-MM-DD>-<thesis-id>-v2.md.

Frontmatter:

---
date: YYYY-MM-DD
type: investing-backtest
thesis_id: <id>
thesis_file: 01-projects/investing/theses/<file>.md
harness: vectorbt-v<version> | backtrader-v<version>
data_source: yfinance + Stooq + alpaca-historical
universe_source: <wiki-constituents | crsp | etc>
phase_history: 01-projects/investing/anchors/<thesis-id>/phase-history.csv
mode: A | B
cycle_count: <N>
benchmark_universe_bh: <yes/no>
benchmark_broad_index: <ticker>
leave_one_out_done: yes
tags: [investing, backtest, v2, <thesis-id>]
---

Body sections (required, in order):

  1. ## Headline — one paragraph. Strategy vs both benchmarks. Lead with the leave-one-out result if it differs materially from the all-cycles result.
  2. ## Per-cycle table — one row per cycle: window dates, universe size, strategy return, BH-universe return, BH-broad return, max DD, trade count.
  3. ## Aggregate metrics with CI — mean, median, bootstrap 95% CI, leave-one-out table.
  4. ## Outlier cycle call-out — if any cycle contributes >50% to a headline metric, name it and re-state without it.
  5. ## Thesis vs strategy separation — TWO sub-sections: "Did the anchor data behave as the thesis predicts?" (thesis validation) and "Did the rules capture the predicted moves?" (strategy validation). These are different questions; v1 conflated them.
  6. ## Survivorship-bias confession — what universe data was missing per window, which delisted tickers couldn't be priced, what the gap likely does to results.
  7. ## Recommendation — one of: PROCEED to paper deploy, REVISE rules and re-backtest, ARCHIVE thesis — does not survive multi-cycle out-of-sample. Specific, not hedged. If recommendation depends on whether the outlier cycle is included, say that explicitly.

Step 8 — Return

One-line summary to the caller:

<thesis-id> v2 | aggregate return <X%> (CI <low>-<high>) | leave-one-out delta <Y%> | <PROCEED|REVISE|ARCHIVE>

Dependencies

Caveats (load-bearing — read every time)

Related

Changelog