Strategy Evaluation & Overfitting Discipline
Compare honestly: walk-forward, deflated Sharpe, bootstrap CIs, cost break-evens.
Why this theme exists
Given a working engine, can the agent evaluate strategies without fooling itself? Tasks require out-of-sample discipline, multiple-testing corrections, resampling that preserves dependence, and sensitivity analysis — and require the agent to state a conclusion the data was generated to make unambiguous.
Capability under test
Statistical honesty: reporting out-of-sample rather than in-sample, quantifying uncertainty correctly, resisting the pull to present the best-looking number.
Design notes
Synthetic data is generated so that the true answer (robust vs not robust) is known by construction, with a wide margin relative to estimation noise. Stochastic procedures are verified by property tests and oracle bands rather than exact draws. Conclusion fields carry explicit weight so that correct numbers with the wrong conclusion are penalized.
Tier ladder
- Tier 10
- Tier 22
- Tier 32
- Tier 40
| Task | Tier | Phase | Reward | Gates | Budget | Status |
|---|---|---|---|---|---|---|
| Break-even transaction cost for each active policy pf-eval-cost-breakeven For each active policy, root-find the transaction-cost level at which it stops beating buy-and-hold on CAGR. | T2 medium | Backtest | partial | — | 30 min | ready |
| Five-policy tournament with a lexicographic ranking rule pf-eval-strategy-tournament Backtest five specified policies with a shared engine and rank them by a lexicographic criterion, reporting a full comparison table. | T2 medium | Backtest | multi-metric | — | 40 min | ready |
| Stationary block bootstrap confidence intervals verified by properties, not draws pf-eval-bootstrap-ci Build stationary block bootstrap confidence intervals for CAGR and max drawdown of a policy, with results verified against oracle bands and dependence-preservation properties. | T3 hard | Backtest | multi-metric | — | 30 min | ready |
| Anchored walk-forward with the deflated Sharpe ratio and an honest conclusion pf-eval-walk-forward-dsr Run an anchored walk-forward over a parameter grid, compute the deflated Sharpe ratio for the in-sample winner, and conclude honestly whether the improvement is real. | T3 hard | Backtest | multi-metric | — | 50 min | ready |