Complete an under-specified backtest and be graded under your own declared assumptions
Complete an under-specified backtest by choosing defaults from an allowed menu, declaring them, and being graded against an oracle recomputed under your own declared assumptions.
instruction.md
What the agent sees (CONVENTIONS.md is appended automatically)
Declare your assumptions
Backtest the IPS policy with quarterly rebalancing from 2012 to 2024. This spec deliberately omits: dividend treatment, execution timing, cost model, whole vs fractional shares, and the rule for rebalance dates falling on holidays. Choose values from the enumerations in /app/ASSUMPTIONS_SCHEMA.json, write /app/output/assumptions.json, and produce equity.csv and stats.json consistent with your declared choices.
The verifier recomputes the oracle under YOUR declared assumptions. Choices are otherwise free, except you must not choose fractional shares if the holdings evidence indicates whole-share lots.
Verification
reward.json metrics · weights sum to 1.00
| Metric | Weight | Check |
|---|---|---|
| schema_valid | 0.15 | Every assumption is a member of its enumeration. |
| self_consistency | 0.7 | equity matches the oracle configured with the declared assumptions, rel 1e-6. |
| constraint_respected | 0.15 | share_mode == whole. |
A novel verifier: it rewards coherence between what the agent says and does rather than a single canonical answer.
Harbor scaffold
Generated from this record — task.toml, Dockerfile, verifier, oracle stub
schema_version = "1.4" [task] name = "portfolio-agent-evals/pf-judg-assumption-log" version = "1.0.0" description = "Complete an under-specified backtest by choosing defaults from an allowed menu, declaring them, and being graded against an oracle recomputed under your own declared assumptions." keywords = ["etf", "portfolio", "cross-cutting", "judgment-communication", "assumption-management", "self-consistency", "spec-completion"] [metadata] author_name = "portfolio-agent-evals" difficulty = "hard" category = "quant-finance" tags = ["judgment-communication", "tier-3", "cross-cutting", "multi-metric"] theme = "Judgment, Communication & Refusal" tier = 3 reward_type = "multi-metric" [agent] timeout_sec = 1800.0 [verifier] timeout_sec = 300.0 [environment] # Offline by design: all data is synthetic and generated at build time. network_mode = "none" cpus = 2 memory_mb = 4096 storage_mb = 10240 build_timeout_sec = 900.0
Traps
Each must carry signal: a trap-blind solution must lose credit
- Declaring one thing and implementing another (e.g. declaring next-open execution but trading at close).
- Choosing values outside the enumeration.
- Ignoring the whole-share evidence.
Inputs
Fixtures mounted in the environment
- /app/policy/ips.yaml, /app/ASSUMPTIONS_SCHEMA.json
- /app/data/prices/
- /app/data/dividends.csv
- /app/portfolio/holdings.csvWhole-share evidence.
Outputs
What the verifier reads from /app/output
- /app/output/assumptions.jsonJSONDeclared choices.
- /app/output/equity.csvCSVEquity curve.
- /app/output/stats.jsonJSONStats.
Anti-gaming
The oracle engine accepts all enumerated configurations; there is no single answer to leak.
Oracle notes
solution/solve.sh must score 1.0 on five seeds
Reference engine is fully parametrised over the assumption schema.
Reviews (0)
Design review before a task is marked ready