Audit and fix a look-ahead-inflated momentum backtest
Audit a provided backtest script whose reported Sharpe is inflated by several look-ahead and survivorship bugs; find, classify and fix them, then report the honest results.
instruction.md
What the agent sees (CONVENTIONS.md is appended automatically)
Audit a too-good-to-be-true backtest
/app/repo/momo_backtest.py reports a Sharpe of about 1.9 for a monthly momentum rotation across the ETF universe. It contains at least four distinct methodological bugs. Candidate classes: same-day-close execution of a same-day signal, adjusted prices computed with future dividends leaking into rankings, universe filtered on today's metadata, a shift in the wrong direction, benchmark misalignment, annualisation error.
Fix the script in place without changing its CLI or output schema. Write /app/output/findings.json: a list of {bug_class, file, line_range, description, impact_on_sharpe} and /app/output/stats_fixed.json from the fixed script. Do not "fix" things that do not change results — report only genuine methodological bugs.
Verification
reward.json metrics · weights sum to 1.00
| Metric | Weight | Check |
|---|---|---|
| fixed_stats | 0.5 | rel 1e-6 vs the oracle-corrected script. |
| findings_recall | 0.35 | Each planted bug class found with a line range overlapping the planted lines; the red herring must not be listed. |
| no_new_bugs | 0.15 | Fixed script runs on a hidden alternate dataset and matches oracle. |
Harbor scaffold
Generated from this record — task.toml, Dockerfile, verifier, oracle stub
schema_version = "1.4" [task] name = "portfolio-agent-evals/pf-bt-lookahead-audit" version = "1.0.0" description = "Audit a provided backtest script whose reported Sharpe is inflated by several look-ahead and survivorship bugs; find, classify and fix them, then report the honest results." keywords = ["etf", "portfolio", "backtest", "backtest-engine", "code-audit", "look-ahead-bias", "debugging"] [metadata] author_name = "portfolio-agent-evals" difficulty = "hard" category = "quant-finance" tags = ["backtest-engine", "tier-3", "backtest", "multi-metric"] theme = "Backtest Engine Correctness" tier = 3 reward_type = "multi-metric" [agent] timeout_sec = 2400.0 [verifier] timeout_sec = 300.0 [environment] # Offline by design: all data is synthetic and generated at build time. network_mode = "none" cpus = 2 memory_mb = 4096 storage_mb = 10240 build_timeout_sec = 900.0
Traps
Each must carry signal: a trap-blind solution must lose credit
- A stylistic red herring (redundant copy) that does not change results.
- Two bugs interact: fixing the shift exposes the same-day execution bug.
- Changing the output schema fails the hidden run even if numbers are right.
- The 'adjusted price' leak is subtle: total-return adjustment factors depend on future dividends.
Inputs
Fixtures mounted in the environment
- /app/repo/momo_backtest.py
- /app/data/prices/
- /app/data/etf_meta.csv
- /app/data/dividends.csv
Outputs
What the verifier reads from /app/output
- /app/output/findings.jsonJSONBug ledger with line ranges.
- /app/output/stats_fixed.jsonJSONHonest results.
- /app/repo/momo_backtest.pyPythonFixed script (same CLI).
Anti-gaming
Bug insertion points and the red herring location are seed-varied across five script variants.
Oracle notes
solution/solve.sh must score 1.0 on five seeds
Oracle is the pristine script; bugs are injected by patch files.
Reviews (0)
Design review before a task is marked ready