Client memo whose every number is machine-checked, graded by a rubric judge
Write a client-facing rebalancing memo whose every cited number is machine-checked against the computed results and whose quality is graded by a rubric-driven judge.
instruction.md
What the agent sees (CONVENTIONS.md is appended automatically)
Write the memo
Using /app/output/results.json (already computed: drift, trades, costs, tax estimate, before/after risk), write /app/output/memo.md of at most 600 words for a non-specialist client covering: what changed and why, cost and tax impact, what risks remain, and what would trigger the next rebalance.
Every number you cite must appear in results.json (rounded to at most the precision shown). The verifier extracts numbers with units from the memo and matches them. Do not give individual investment advice beyond the IPS and do not forecast returns.
Verification
reward.json metrics · weights sum to 1.00
| Metric | Weight | Check |
|---|---|---|
| numeric_consistency | 0.5 | >= 95% of extracted numbers match a results.json value within rounding; any fabricated number scores 0. |
| judge_rubric | 0.5 | LLM judge with JSON-schema rubric: clarity, four sections present, no forecasts, no out-of-scope advice, <= 600 words. Combined multiplicatively with numeric_consistency. |
Multiplicative combination: a beautiful memo with a fabricated number earns zero.
Harbor scaffold
Generated from this record — task.toml, Dockerfile, verifier, oracle stub
schema_version = "1.4"
[task]
name = "portfolio-agent-evals/pf-judg-rebalance-memo"
version = "1.0.0"
description = "Write a client-facing rebalancing memo whose every cited number is machine-checked against the computed results and whose quality is graded by a rubric-driven judge."
keywords = ["etf", "portfolio", "cross-cutting", "judgment-communication", "communication", "numeric-grounding", "scope-discipline"]
[metadata]
author_name = "portfolio-agent-evals"
difficulty = "hard"
category = "quant-finance"
tags = ["judgment-communication", "tier-3", "cross-cutting", "llm-judge-hybrid"]
theme = "Judgment, Communication & Refusal"
tier = 3
reward_type = "llm-judge-hybrid"
[agent]
timeout_sec = 1200.0
[verifier]
timeout_sec = 300.0
[verifier.env]
# Judge credentials flow only to the verifier container, never to the agent.
JUDGE_API_KEY = "${JUDGE_API_KEY}"
[environment]
# Offline by design: all data is synthetic and generated at build time.
network_mode = "none"
cpus = 2
memory_mb = 4096
storage_mb = 10240
build_timeout_sec = 900.0
Traps
Each must carry signal: a trap-blind solution must lose credit
- Unit conversions (bps vs %) count as mismatches if wrong.
- Rounding that changes sign.
- The pull to add a market outlook (forecast) is penalised by the rubric.
Inputs
Fixtures mounted in the environment
- /app/output/results.jsonPre-computed by the environment.
- /app/policy/ips.yaml
Outputs
What the verifier reads from /app/output
- /app/output/memo.mdMarkdownClient memo.
Anti-gaming
results.json values are seeded; judge prompt includes the rubric and results.json but not the oracle memo.
Oracle notes
solution/solve.sh must score 1.0 on five seeds
Oracle memo is template-generated from results.json.
Reviews (0)
Design review before a task is marked ready