Exact band-monitored backtest over 400 ETFs in 60 seconds on 2 vCPU
Run a band-monitored backtest over 400 ETFs and 15 years within a strict time and memory budget while matching the oracle to 1e-6.
instruction.md
What the agent sees (CONVENTIONS.md is appended automatically)
Fast and exact
/app/data/big/prices.parquet contains 400 synthetic ETFs x 15 years daily. Backtest the equal-weight policy with 5% relative bands checked daily (engine conventions of pf-bt-calendar-rebalance: whole shares, 5 bps plus 1 USD per fill, next-open execution) and write /app/output/equity.csv and /app/output/stats.json.
Hard limits: your /app/run.sh must complete within 60 seconds wall-clock on 2 vCPU with 2 GB RAM. The verifier re-runs it under these limits.
Verification
reward.json metrics · weights sum to 1.00
| Metric | Weight | Check |
|---|---|---|
| correctness | 0.6 | rel 1e-6 every day; zero if any mismatch. |
| within_time | 0.25 | <= 60 s full; linear to 0 at 120 s. |
| within_memorygate | 0.15 | Peak RSS <= 2 GB. GATE: OOM = 0. |
Gates: within_memory. A gate failure caps or zeroes the trial reward regardless of other metrics.
Harbor scaffold
Generated from this record — task.toml, Dockerfile, verifier, oracle stub
schema_version = "1.4" [task] name = "portfolio-agent-evals/pf-perf-large-universe" version = "1.0.0" description = "Run a band-monitored backtest over 400 ETFs and 15 years within a strict time and memory budget while matching the oracle to 1e-6." keywords = ["etf", "portfolio", "cross-cutting", "debugging-tooling", "performance-engineering", "backtest-mechanics", "resource-limits"] [metadata] author_name = "portfolio-agent-evals" difficulty = "hard" category = "quant-finance" tags = ["debugging-tooling", "tier-4", "cross-cutting", "multi-metric"] theme = "Debugging, Tooling & Performance" tier = 4 reward_type = "multi-metric" [agent] timeout_sec = 3600.0 [verifier] timeout_sec = 600.0 [environment] # Offline by design: all data is synthetic and generated at build time. network_mode = "none" cpus = 2 memory_mb = 2048 storage_mb = 10240 build_timeout_sec = 900.0
Traps
Each must carry signal: a trap-blind solution must lose credit
- Per-day Python loops over 400 tickers are too slow.
- Band triggers and whole-share rounding are path-dependent, so full vectorisation is impossible: use vectorised drift checks between sparse rebalance events.
- Wide float64 frames with copies exceed 2 GB.
Inputs
Fixtures mounted in the environment
- /app/data/big/prices.parquet
- /app/CONVENTIONS.md
Outputs
What the verifier reads from /app/output
- /app/run.shshellEntry point re-run by the verifier.
- /app/output/equity.csvCSVEquity curve.
- /app/output/stats.jsonJSONStats.
Anti-gaming
Limit is 3x the oracle runtime on the reference hardware; verifier measures its own environment with a calibration run.
Oracle notes
solution/solve.sh must score 1.0 on five seeds
numpy hybrid: cumulative-return matrices between events; ~18 s.
Reviews (0)
Design review before a task is marked ready