Fix six planted bugs in pfkit without breaking its public API
Fix six planted bugs in a small backtesting library so a hidden test-suite passes, without changing its public API, and add regression tests that fail on the original code.
instruction.md
What the agent sees (CONVENTIONS.md is appended automatically)
Fix pfkit
/app/repo/pfkit is a small library (engine.py, metrics.py, rebalance.py) with a README and a partially passing test-suite. Users report that Sharpe ratios, drawdowns and rebalancing bands look wrong. /app/repo/ISSUES.md lists three reported issues; there are more bugs than issues, and not every issue is a bug.
Find and fix the bugs. Keep every public function signature and return type. Add regression tests under /app/repo/tests that fail on the original code and pass on yours. Hidden tests will exercise the public API on a different dataset.
Verification
reward.json metrics · weights sum to 1.00
| Metric | Weight | Check |
|---|---|---|
| hidden_tests | 0.7 | Fraction of six bug-specific hidden tests passing on a hidden dataset. |
| api_stablegate | 0.2 | inspect.signature of every public function unchanged. GATE at 0.3. |
| regression_tests_added | 0.1 | At least 3 new tests that fail against a pristine copy and pass against the fix. |
Gates: api_stable. A gate failure caps or zeroes the trial reward regardless of other metrics.
Harbor scaffold
Generated from this record — task.toml, Dockerfile, verifier, oracle stub
schema_version = "1.4" [task] name = "portfolio-agent-evals/pf-debug-buggy-backtester" version = "1.0.0" description = "Fix six planted bugs in a small backtesting library so a hidden test-suite passes, without changing its public API, and add regression tests that fail on the original code." keywords = ["etf", "portfolio", "cross-cutting", "debugging-tooling", "debugging", "regression-testing", "api-discipline"] [metadata] author_name = "portfolio-agent-evals" difficulty = "hard" category = "quant-finance" tags = ["debugging-tooling", "tier-3", "cross-cutting", "multi-metric"] theme = "Debugging, Tooling & Performance" tier = 3 reward_type = "multi-metric" [agent] timeout_sec = 2400.0 [verifier] timeout_sec = 300.0 [environment] # Offline by design: all data is synthetic and generated at build time. network_mode = "none" cpus = 2 memory_mb = 4096 storage_mb = 10240 build_timeout_sec = 900.0
Traps
Each must carry signal: a trap-blind solution must lose credit
- Annualisation with 365 instead of 252.
- pct_change followed by dropna misaligns returns with dates by one day.
- Dividends credited on both ex-date and pay-date.
- Costs charged on notional including cash.
- Drawdown computed on a price index instead of total return.
- Band check compares drift against target weight rather than current weight.
- One ISSUES.md item is user error; 'fixing' it breaks a hidden test.
Inputs
Fixtures mounted in the environment
- /app/repo/pfkit/, /app/repo/ISSUES.md
- /app/data/prices/
- /app/CONVENTIONS.md
Outputs
What the verifier reads from /app/output
- /app/repo/pfkit/PythonFixed library.
- /app/repo/tests/pytestNew regression tests.
Anti-gaming
Six bugs are drawn from a pool of ten per seed; hidden dataset differs from the visible one.
Oracle notes
solution/solve.sh must score 1.0 on five seeds
Oracle is the unpatched reference implementation.
Reviews (0)
Design review before a task is marked ready