Debugging, Tooling & Performance
Fix planted bugs, build a CLI verified on hidden and metamorphic inputs, hit hard time budgets.
Why this theme exists
Agents rarely start from a blank slate. These tasks provide a buggy library, a too-good-to-be-true script, a non-reproducible notebook, or a large universe, and test whether the agent can produce robust, general, fast tooling.
Capability under test
Code comprehension and debugging, generalization beyond the visible data (metamorphic testing), API discipline, reproducibility engineering, and performance under resource limits.
Design notes
Hidden datasets and metamorphic transformations (price scaling, calendar shifts, ticker renames, column shuffles) distinguish general solutions from overfit ones. Red-herring issues test whether the agent verifies before 'fixing'. The performance task's limit is set at 3x the oracle's runtime on the reference hardware.
Tier ladder
- Tier 10
- Tier 21
- Tier 32
- Tier 41
| Task | Tier | Phase | Reward | Gates | Budget | Status |
|---|---|---|---|---|---|---|
| Turn a non-reproducible notebook into a byte-identical pipeline pf-tool-reproducibility-pack Make an existing analysis notebook fully reproducible with a pinned environment, data checksums, deterministic seeds and a single make target that regenerates byte-identical outputs. | T2 medium | Cross-cutting | multi-metric | — | 30 min | ready |
| Fix six planted bugs in pfkit without breaking its public API pf-debug-buggy-backtester Fix six planted bugs in a small backtesting library so a hidden test-suite passes, without changing its public API, and add regression tests that fail on the original code. | T3 hard | Cross-cutting | multi-metric | 1 gate | 40 min | ready |
| Build pfctl, verified on hidden datasets and metamorphic transformations pf-tool-cli-metamorphic Build a pfctl command-line tool (analyze / backtest / rebalance) that is verified on hidden datasets and under metamorphic transformations of the inputs. | T3 hard | Cross-cutting | multi-metric | — | 50 min | ready |
| Exact band-monitored backtest over 400 ETFs in 60 seconds on 2 vCPU pf-perf-large-universe Run a band-monitored backtest over 400 ETFs and 15 years within a strict time and memory budget while matching the oracle to 1e-6. | T4 expert | Cross-cutting | multi-metric | 1 gate | 60 min | needs-oracle |