Anchored walk-forward with the deflated Sharpe ratio and an honest conclusion
Run an anchored walk-forward over a parameter grid, compute the deflated Sharpe ratio for the in-sample winner, and conclude honestly whether the improvement is real.
instruction.md
What the agent sees (CONVENTIONS.md is appended automatically)
Walk-forward and the deflated Sharpe ratio
Grid: lookback in {63, 126, 189, 252} x top-N in {1, 2, 3} x band in {0, 0.05} — 24 variants of the momentum family in /app/strategies/momentum_family.yaml. Anchored walk-forward: train 2012-01-03..2016-12-30 and test 2017; then extend training by one year and test the next, through 2024. In each fold select the in-sample Sharpe maximiser and record its out-of-sample monthly returns. Stitch the OOS returns.
Report /app/output/walk_forward.json: per-fold selections, OOS Sharpe/CAGR/max drawdown, the in-sample Sharpe of the full-sample best variant, and its deflated Sharpe ratio (Bailey and Lopez de Prado 2014) using N = 24 trials, the variance of trial Sharpes, sample length in months, skewness and kurtosis of monthly returns, SR0 = 0.
conclusion must be "robust" or "not_robust": robust requires DSR >= 0.95 AND OOS Sharpe >= 0.5 x in-sample Sharpe.
Verification
reward.json metrics · weights sum to 1.00
| Metric | Weight | Check |
|---|---|---|
| fold_selections | 0.3 | Selected variant exact per fold. |
| oos_stats | 0.3 | rel 1e-4. |
| dsr | 0.2 | abs 1e-3. |
| conclusion | 0.2 | Exact; data is generated so the truth is unambiguous. |
Harbor scaffold
Generated from this record — task.toml, Dockerfile, verifier, oracle stub
schema_version = "1.4" [task] name = "portfolio-agent-evals/pf-eval-walk-forward-dsr" version = "1.0.0" description = "Run an anchored walk-forward over a parameter grid, compute the deflated Sharpe ratio for the in-sample winner, and conclude honestly whether the improvement is real." keywords = ["etf", "portfolio", "backtest", "strategy-evaluation", "walk-forward", "multiple-testing", "statistical-honesty"] [metadata] author_name = "portfolio-agent-evals" difficulty = "hard" category = "quant-finance" tags = ["strategy-evaluation", "tier-3", "backtest", "multi-metric"] theme = "Strategy Evaluation & Overfitting Discipline" tier = 3 reward_type = "multi-metric" [agent] timeout_sec = 3000.0 [verifier] timeout_sec = 300.0 [environment] # Offline by design: all data is synthetic and generated at build time. network_mode = "none" cpus = 2 memory_mb = 4096 storage_mb = 10240 build_timeout_sec = 900.0
Traps
Each must carry signal: a trap-blind solution must lose credit
- DSR must use monthly Sharpe and monthly T, not daily.
- Expected maximum Sharpe uses the Euler–Mascheroni constant and the inverse normal at 1 - 1/N and 1 - 1/(N e).
- In-sample selection must be recomputed per fold, not once.
- Half the seeds generate data where the momentum edge is real; the other half is pure noise — the agent must let the data decide.
Inputs
Fixtures mounted in the environment
- /app/strategies/momentum_family.yaml
- /app/data/prices/
- /app/data/factors.csv
- /app/data/dividends.csv
- /app/CONVENTIONS.md
Outputs
What the verifier reads from /app/output
- /app/output/walk_forward.jsonJSONFold log, OOS stats, DSR, conclusion.
Anti-gaming
Truth (real edge or not) is a seed-level coin flip; the generator ensures the DSR margin from 0.95 exceeds 0.1.
Oracle notes
solution/solve.sh must score 1.0 on five seeds
Oracle implements DSR per the paper's equations 10-12.
Reviews (0)
Design review before a task is marked ready