Stationary block bootstrap confidence intervals verified by properties, not draws
Build stationary block bootstrap confidence intervals for CAGR and max drawdown of a policy, with results verified against oracle bands and dependence-preservation properties.
instruction.md
What the agent sees (CONVENTIONS.md is appended automatically)
Bootstrap uncertainty
For the monthly-calendar policy's daily return series, run a stationary block bootstrap (Politis and Romano; geometric block lengths with mean 20; circular wrapping) with 5,000 resamples of the original length, using numpy.random.default_rng(seed) with the seed from /app/data/seed.txt.
For each resample compute CAGR and max drawdown. Write /app/output/bootstrap.json with 5th/50th/95th percentiles of both, P(CAGR < 0) and P(max drawdown < -30%). Also write /app/output/resample_check.json with the mean lag-1 autocorrelation across the first 200 resampled series.
Verification
reward.json metrics · weights sum to 1.00
| Metric | Weight | Check |
|---|---|---|
| percentiles_within_band | 0.6 | Each percentile within 7.5% of the oracle interval width of the oracle value. |
| tail_probabilities | 0.25 | abs 0.02. |
| block_structure | 0.15 | Mean lag-1 autocorrelation of resamples within 0.02 of the original series (iid bootstrap fails this). |
Exact draws are not verified: different but valid implementations consume RNG differently.
Harbor scaffold
Generated from this record — task.toml, Dockerfile, verifier, oracle stub
schema_version = "1.4" [task] name = "portfolio-agent-evals/pf-eval-bootstrap-ci" version = "1.0.0" description = "Build stationary block bootstrap confidence intervals for CAGR and max drawdown of a policy, with results verified against oracle bands and dependence-preservation properties." keywords = ["etf", "portfolio", "backtest", "strategy-evaluation", "resampling", "uncertainty-quantification", "statistical-rigor"] [metadata] author_name = "portfolio-agent-evals" difficulty = "hard" category = "quant-finance" tags = ["strategy-evaluation", "tier-3", "backtest", "multi-metric"] theme = "Strategy Evaluation & Overfitting Discipline" tier = 3 reward_type = "multi-metric" [agent] timeout_sec = 1800.0 [verifier] timeout_sec = 300.0 [environment] # Offline by design: all data is synthetic and generated at build time. network_mode = "none" cpus = 2 memory_mb = 4096 storage_mb = 10240 build_timeout_sec = 900.0
Traps
Each must carry signal: a trap-blind solution must lose credit
- An iid bootstrap gives a much narrower drawdown distribution and fails block_structure.
- Circular wrapping is required at the series end.
- Block lengths are geometric, not fixed.
- Max drawdown of a resample is on its compounded path, not on sorted returns.
Inputs
Fixtures mounted in the environment
- /app/data/prices/
- /app/policy/ips.yaml
- /app/data/seed.txt
- /app/CONVENTIONS.md
Outputs
What the verifier reads from /app/output
- /app/output/bootstrap.jsonJSONPercentiles and tail probabilities.
- /app/output/resample_check.jsonJSONDependence diagnostics.
Anti-gaming
Oracle bands are computed from 50,000 resamples across five RNG streams to make the band robust.
Oracle notes
solution/solve.sh must score 1.0 on five seeds
arch.bootstrap.StationaryBootstrap matches the oracle within band.
Reviews (0)
Design review before a task is marked ready