Build pfctl, verified on hidden datasets and metamorphic transformations
Build a pfctl command-line tool (analyze / backtest / rebalance) that is verified on hidden datasets and under metamorphic transformations of the inputs.
instruction.md
What the agent sees (CONVENTIONS.md is appended automatically)
Build pfctl
Implement /app/bin/pfctl with subcommands analyze, backtest and rebalance per /app/SPEC.md (arguments, JSON schemas, exit codes, conventions). The verifier will run it on datasets you have not seen and under transformations of those datasets:
- all prices multiplied by k: weights and returns invariant, share counts scale by 1/k
- calendar shifted by a constant offset: all dated outputs shift
- tickers renamed: outputs renamed
- column order shuffled and extra columns added: ignored
- a missing input file: exit code 2 with a JSON error object on stdout, no traceback
Each invocation must finish within 60 seconds.
Verification
reward.json metrics · weights sum to 1.00
| Metric | Weight | Check |
|---|---|---|
| hidden_correctness | 0.4 | Outputs match oracle on 5 hidden datasets (rel 1e-6). |
| metamorphic_invariance | 0.3 | Each transformation relation holds. |
| error_handling | 0.15 | Exit codes and JSON errors per SPEC. |
| schema_validity | 0.15 | All JSON outputs validate against SPEC schemas. |
Harbor scaffold
Generated from this record — task.toml, Dockerfile, verifier, oracle stub
schema_version = "1.4" [task] name = "portfolio-agent-evals/pf-tool-cli-metamorphic" version = "1.0.0" description = "Build a pfctl command-line tool (analyze / backtest / rebalance) that is verified on hidden datasets and under metamorphic transformations of the inputs." keywords = ["etf", "portfolio", "cross-cutting", "debugging-tooling", "cli-engineering", "generalisation", "robustness"] [metadata] author_name = "portfolio-agent-evals" difficulty = "hard" category = "quant-finance" tags = ["debugging-tooling", "tier-3", "cross-cutting", "multi-metric"] theme = "Debugging, Tooling & Performance" tier = 3 reward_type = "multi-metric" [agent] timeout_sec = 3000.0 [verifier] timeout_sec = 600.0 [environment] # Offline by design: all data is synthetic and generated at build time. network_mode = "none" cpus = 2 memory_mb = 4096 storage_mb = 10240 build_timeout_sec = 900.0
Traps
Each must carry signal: a trap-blind solution must lose credit
- Hard-coded tickers or sleeve names from the visible sample.
- Assuming column order.
- Non-deterministic dict ordering leaking into output.
- Uncaught exceptions producing tracebacks instead of exit code 2.
Inputs
Fixtures mounted in the environment
- /app/CONVENTIONS.md, /app/SPEC.md
- /app/data/prices/Visible sample dataset only.
- /app/portfolio/
- /app/policy/ips.yaml
Outputs
What the verifier reads from /app/output
- /app/bin/pfctlexecutableCLI entry point.
Anti-gaming
Hidden datasets have different universes, sleeves and calendars.
Oracle notes
solution/solve.sh must score 1.0 on five seeds
Oracle CLI wraps the reference engine.
Reviews (0)
Design review before a task is marked ready