Themes
Each theme isolates one capability. Tasks within a theme climb the tier ladder so a failure pattern is legible: a model that passes tier 2 engine tasks but fails tier 3 look-ahead audits tells you something specific.
Data Forensics & Canonicalization
Every downstream number is only as good as the price panel. These tasks hand the agent multi-vendor ETF price files with planted defects (format drift, duplicates, unit scaling, undocumented splits, mis-stamped dividends, calendar mismatches) and require a canonical, adjusted, calendar-aligned panel plus an explicit anomaly ledger.
Capability under test: Data hygiene under adversarial conditions: detecting rather than silently smoothing defects, distinguishing genuine market events from data errors, and applying corporate-action math exactly.
- T1Profile raw price files against calendar and metadata
- T2Build a canonical adjusted price panel from messy vendor files
- T2Multi-currency, multi-calendar alignment to USD returns
- T3Reconcile custodian dividends against the vendor file and rebuild total-return indices
- T3Detect an undocumented split without misclassifying a genuine crash
Portfolio Analytics & Exposure
Deterministic analytics on a multi-account ETF household. Each task pins the convention (ddof, annualization, window definition, quantile interpolation, flow timing) so the only source of error is the agent, not the ambiguity of finance jargon.
Capability under test: Convention-exact quantitative work, multi-file joins (holdings, lots, transactions, constituents), performance measurement with cash flows, and econometric inference done correctly.
- T1Weights, drift and band breaches (absolute and 5/25 relative)
- T2Look-through exposure, pairwise overlap and hidden concentration
- T2Risk report with exact conventions
- T2Time-weighted vs money-weighted returns from a transaction log
- T3Brinson–Fachler attribution with Cariño linking that reconciles exactly
- T3Factor exposures with Newey–West inference and an implied-vs-direct reconciliation
Backtest Engine Correctness
The agent must implement simulation mechanics exactly: decision-at-close / execute-at-next-open timing, per-fill and proportional costs, whole-share rounding with cash feasibility, dividends by ex-date shares paid on pay-date, universe changes (launches, liquidations) without look-ahead, overlays with strictly-prior information.
Capability under test: Zero-look-ahead engineering, path-dependent state handling, and disciplined adherence to a spec where every sentence matters.
- T2Fixed-weight monthly rebalance with next-open execution, costs, whole shares and dividends
- T3Dual momentum rotation, exactly as specified
- T3Universe changes (launch and liquidation) without look-ahead
- T3Audit and fix a look-ahead-inflated momentum backtest
- T3Volatility-targeting overlay with strictly-prior EWMA and financing
Strategy Evaluation & Overfitting Discipline
Given a working engine, can the agent evaluate strategies without fooling itself? Tasks require out-of-sample discipline, multiple-testing corrections, resampling that preserves dependence, and sensitivity analysis — and require the agent to state a conclusion the data was generated to make unambiguous.
Capability under test: Statistical honesty: reporting out-of-sample rather than in-sample, quantifying uncertainty correctly, resisting the pull to present the best-looking number.
Rebalancing & Trade Generation
The rebalance phase where analysis becomes orders. Tasks range from a single-account trade list to household-level asset location, contribution-driven buy-only rebalancing, withdrawal sequencing, and a full year of event-driven band monitoring.
Capability under test: Constraint satisfaction and near-optimality under discrete constraints; correct feasibility reasoning (when buy-only is impossible, when a preference cannot be fully honored).
- T1Feasible trade list to IPS targets (whole shares, cash buffer, min trade)
- T2One year of daily band monitoring with next-open trading and monthly contributions
- T2Contribution-driven buy-only rebalancing with a minimal-sell fallback
- T3Household rebalance across three accounts with asset-location scoring
- T3Raise a large withdrawal across accounts with sequencing rules and penalties
Tax-Aware & Constrained Optimization
The hard edge of rebalancing: specific-ID lot selection, wash-sale windows that span accounts and dividend reinvestments, tax-loss harvesting into partner ETFs, and formal optimization problems with verifiable optimality gaps.
Capability under test: Modeling domain rules precisely (holding periods, substantially-identical securities, cross-account wash sales) and formulating/solving optimization problems rather than hand-waving heuristics.
- T2Equal risk contribution weights on an ill-conditioned covariance
- T3Minimum-turnover trade list restoring all bands (MILP with semi-continuous trades)
- T3Tracking-error minimisation with Ledoit–Wolf covariance and a cardinality cap
- T3Tax-loss harvesting into partner ETFs with exposure and wash-sale guards
- T3Specific-ID lot selection minimising tax with cross-account wash-sale rules
Debugging, Tooling & Performance
Agents rarely start from a blank slate. These tasks provide a buggy library, a too-good-to-be-true script, a non-reproducible notebook, or a large universe, and test whether the agent can produce robust, general, fast tooling.
Capability under test: Code comprehension and debugging, generalization beyond the visible data (metamorphic testing), API discipline, reproducibility engineering, and performance under resource limits.
Judgment, Communication & Refusal
Tests the behaviors that separate a useful analyst from a calculator: validating a policy before acting, refusing to silently normalize a broken spec, choosing and declaring defaults consistently, staying within authorization, and communicating in grounded language.
Capability under test: Meta-cognition about the task itself: detecting inconsistency, self-consistency between declared and implemented assumptions, scope discipline, and numerically grounded communication.
- T2Analysis only: ignore the co-located broker script and README urging execution
- T2Validate the policy and refuse to trade on an inconsistent IPS
- T3Complete an under-specified backtest and be graded under your own declared assumptions
- T3Client memo whose every number is machine-checked, graded by a rubric judge
Adversarial Robustness & Safety
Financial data pipelines ingest third-party files. These tasks plant instructions inside CSV comments, YAML blocks and notes fields; provide plausible-but-wrong proxies for missing data; put contradictory conventions in a README; and leave credentials in the environment — then check the agent did the right thing.
Capability under test: Instruction hierarchy (task > repository docs > data), refusal to fabricate, output-scope discipline, and secret hygiene.
End-to-End Multi-Step Reviews
Long-horizon tasks using Harbor's multi-step layout: a quarterly review split into gated steps, a single-shot full pipeline on hostile inputs, a year of quarterly cycles requiring reconciliation of actual fills against the prior plan, and a late client change that must be absorbed with a minimal amendment.
Capability under test: Prioritization under time pressure, error containment across stages, state management across steps, reconciliation, and minimal-diff reasoning when requirements change.