Quarterly review as four gated Harbor steps
A four-step Harbor task covering the full analyze, backtest, rebalance and communicate loop with gated step rewards and files carried between steps.
instruction.md
What the agent sees (CONVENTIONS.md is appended automatically)
Quarterly portfolio review (multi-step)
This task runs as four steps in one container. Each step's instruction is delivered in turn; later steps depend on files you produced earlier.
- canonicalize — build /app/output/close_adj.csv and anomalies.json from messy vendor data (conventions of pf-data-canonical-panel).
- analyze-and-backtest — drift.json, risk.json and a three-policy tournament.csv using your step-1 panel.
- rebalance — a tax-aware trade list within bands with lot selection and wash-sale checks (tax_summary.json, trades.csv).
- communicate — memo.md plus next_quarter_state.json (approved trades, blackout dates, declared assumptions).
Steps
Harbor multi-step layout; rewards aggregate by mean
- 1canonicalizemin_reward 0.8
Canonical panel and anomaly ledger from messy data.
- 2analyze-and-backtest
Drift, risk and a three-policy tournament from the step-1 panel.
- 3rebalance
Tax-aware trade list with lot selection and wash-sale checks.
- 4communicate
Client memo and next-quarter state file.
Verification
reward.json metrics · weights sum to 1.00
| Metric | Weight | Check |
|---|---|---|
| step1_panelgate | 0.25 | Panel and anomaly ledger as in canonical-panel; min_reward 0.8 gates the trial. |
| step2_analytics | 0.25 | Drift/risk/tournament vs oracle computed from the ORACLE panel (isolates step-2 skill). |
| step3_tradesgate | 0.25 | Feasibility gate, wash-sale gate, tax <= oracle + 1%. |
| step4_memo_state | 0.25 | Numeric consistency with step 3 outputs; state schema valid; judge rubric. |
multi_step_reward_strategy = mean; only step 1 is gated.
Gates: step1_panel, step3_trades. A gate failure caps or zeroes the trial reward regardless of other metrics.
Harbor scaffold
Generated from this record — task.toml, Dockerfile, verifier, oracle stub
schema_version = "1.4" [task] name = "portfolio-agent-evals/pf-e2e-quarterly-review-multistep" version = "1.0.0" description = "A four-step Harbor task covering the full analyze, backtest, rebalance and communicate loop with gated step rewards and files carried between steps." keywords = ["etf", "portfolio", "e2e", "end-to-end", "long-horizon", "state-management", "full-pipeline"] # Per-step rewards roll up by mean; gates are declared per step below. multi_step_reward_strategy = "mean" [metadata] author_name = "portfolio-agent-evals" difficulty = "hard" category = "quant-finance" tags = ["end-to-end", "tier-4", "e2e", "multi-metric"] theme = "End-to-End Multi-Step Reviews" tier = 4 reward_type = "multi-metric" [agent] timeout_sec = 1800.0 [verifier] timeout_sec = 300.0 [environment] # Offline by design: all data is synthetic and generated at build time. network_mode = "none" cpus = 2 memory_mb = 4096 storage_mb = 10240 build_timeout_sec = 900.0 [[steps]] name = "canonicalize" # Canonical panel and anomaly ledger from messy data. min_reward = 0.8 [steps.agent] timeout_sec = 1800.0 [steps.verifier] timeout_sec = 300.0 [[steps]] name = "analyze-and-backtest" # Drift, risk and a three-policy tournament from the step-1 panel. [steps.agent] timeout_sec = 1800.0 [steps.verifier] timeout_sec = 300.0 [[steps]] name = "rebalance" # Tax-aware trade list with lot selection and wash-sale checks. [steps.agent] timeout_sec = 1800.0 [steps.verifier] timeout_sec = 300.0 [[steps]] name = "communicate" # Client memo and next-quarter state file. [steps.agent] timeout_sec = 1800.0 [steps.verifier] timeout_sec = 300.0
Traps
Each must carry signal: a trap-blind solution must lose credit
- Errors propagate: a wrong panel makes everything downstream wrong, hence the gate.
- Step 3 must reuse the step-2 as-of date and prices.
- Step 4 memo numbers must match step 3 outputs, not step 2 estimates.
Inputs
Fixtures mounted in the environment
- /app/data/prices_messy/
- /app/portfolio/
- /app/policy/
- /app/strategies/
- /app/CONVENTIONS.md
Outputs
What the verifier reads from /app/output
- /app/output/close_adj.csvCSVStep 1.
- /app/output/{drift,risk}.json, tournament.csvJSON/CSVStep 2.
- /app/output/trades.csv, tax_summary.jsonCSV/JSONStep 3.
- /app/output/memo.md, next_quarter_state.jsonMD/JSONStep 4.
Anti-gaming
Each step's verifier regenerates truth from the seed; step-2 grading uses the oracle panel so a lucky step 1 does not inflate step 2.
Oracle notes
solution/solve.sh must score 1.0 on five seeds
Oracle solve.sh per step calls the reference modules in order.
Reviews (0)
Design review before a task is marked ready