Portfolio Agent EvalsHarbor task suite · ETF analyze → backtest → rebalance
End-to-End Multi-Step Reviews/pf-e2e-quarterly-review-multistep

Quarterly review as four gated Harbor steps

A four-step Harbor task covering the full analyze, backtest, rebalance and communicate loop with gated step rewards and files carried between steps.

T4expertEnd-to-endmulti-metricready4-step
Edit
readystatic
Agent budget
30 min / step
Verifier budget
5 min
Tier target
< 20% pass expected

instruction.md

What the agent sees (CONVENTIONS.md is appended automatically)

Quarterly portfolio review (multi-step)

This task runs as four steps in one container. Each step's instruction is delivered in turn; later steps depend on files you produced earlier.

  1. canonicalize — build /app/output/close_adj.csv and anomalies.json from messy vendor data (conventions of pf-data-canonical-panel).
  2. analyze-and-backtest — drift.json, risk.json and a three-policy tournament.csv using your step-1 panel.
  3. rebalance — a tax-aware trade list within bands with lot selection and wash-sale checks (tax_summary.json, trades.csv).
  4. communicate — memo.md plus next_quarter_state.json (approved trades, blackout dates, declared assumptions).

Steps

Harbor multi-step layout; rewards aggregate by mean

  1. 1
    canonicalizemin_reward 0.8

    Canonical panel and anomaly ledger from messy data.

  2. 2
    analyze-and-backtest

    Drift, risk and a three-policy tournament from the step-1 panel.

  3. 3
    rebalance

    Tax-aware trade list with lot selection and wash-sale checks.

  4. 4
    communicate

    Client memo and next-quarter state file.

Verification

reward.json metrics · weights sum to 1.00

MetricWeightCheck
step1_panelgate
0.25
Panel and anomaly ledger as in canonical-panel; min_reward 0.8 gates the trial.
step2_analytics
0.25
Drift/risk/tournament vs oracle computed from the ORACLE panel (isolates step-2 skill).
step3_tradesgate
0.25
Feasibility gate, wash-sale gate, tax <= oracle + 1%.
step4_memo_state
0.25
Numeric consistency with step 3 outputs; state schema valid; judge rubric.

multi_step_reward_strategy = mean; only step 1 is gated.

Gates: step1_panel, step3_trades. A gate failure caps or zeroes the trial reward regardless of other metrics.

Harbor scaffold

Generated from this record — task.toml, Dockerfile, verifier, oracle stub

schema_version = "1.4"

[task]
name = "portfolio-agent-evals/pf-e2e-quarterly-review-multistep"
version = "1.0.0"
description = "A four-step Harbor task covering the full analyze, backtest, rebalance and communicate loop with gated step rewards and files carried between steps."
keywords = ["etf", "portfolio", "e2e", "end-to-end", "long-horizon", "state-management", "full-pipeline"]
# Per-step rewards roll up by mean; gates are declared per step below.
multi_step_reward_strategy = "mean"

[metadata]
author_name = "portfolio-agent-evals"
difficulty = "hard"
category = "quant-finance"
tags = ["end-to-end", "tier-4", "e2e", "multi-metric"]
theme = "End-to-End Multi-Step Reviews"
tier = 4
reward_type = "multi-metric"

[agent]
timeout_sec = 1800.0

[verifier]
timeout_sec = 300.0

[environment]
# Offline by design: all data is synthetic and generated at build time.
network_mode = "none"
cpus = 2
memory_mb = 4096
storage_mb = 10240
build_timeout_sec = 900.0

[[steps]]
name = "canonicalize"
# Canonical panel and anomaly ledger from messy data.
min_reward = 0.8
[steps.agent]
timeout_sec = 1800.0
[steps.verifier]
timeout_sec = 300.0

[[steps]]
name = "analyze-and-backtest"
# Drift, risk and a three-policy tournament from the step-1 panel.
[steps.agent]
timeout_sec = 1800.0
[steps.verifier]
timeout_sec = 300.0

[[steps]]
name = "rebalance"
# Tax-aware trade list with lot selection and wash-sale checks.
[steps.agent]
timeout_sec = 1800.0
[steps.verifier]
timeout_sec = 300.0

[[steps]]
name = "communicate"
# Client memo and next-quarter state file.
[steps.agent]
timeout_sec = 1800.0
[steps.verifier]
timeout_sec = 300.0

Traps

Each must carry signal: a trap-blind solution must lose credit

  • Errors propagate: a wrong panel makes everything downstream wrong, hence the gate.
  • Step 3 must reuse the step-2 as-of date and prices.
  • Step 4 memo numbers must match step 3 outputs, not step 2 estimates.

Inputs

Fixtures mounted in the environment

Outputs

What the verifier reads from /app/output

  • /app/output/close_adj.csv
    CSV
    Step 1.
  • /app/output/{drift,risk}.json, tournament.csv
    JSON/CSV
    Step 2.
  • /app/output/trades.csv, tax_summary.json
    CSV/JSON
    Step 3.
  • /app/output/memo.md, next_quarter_state.json
    MD/JSON
    Step 4.

Anti-gaming

Each step's verifier regenerates truth from the seed; step-2 grading uses the oracle panel so a lucky step 1 does not inflate step 2.

Oracle notes

solution/solve.sh must score 1.0 on five seeds

Oracle solve.sh per step calls the reference modules in order.

Reviews (0)

Design review before a task is marked ready

    Reviews are read-only in static export.
    Tier 4 · Frontier< 20% pass expected. Created 2026-01-01, updated 2026-01-01.