Data Forensics & Canonicalization
Turn hostile vendor exports into a panel you can trust — and prove what you found.
Why this theme exists
Every downstream number is only as good as the price panel. These tasks hand the agent multi-vendor ETF price files with planted defects (format drift, duplicates, unit scaling, undocumented splits, mis-stamped dividends, calendar mismatches) and require a canonical, adjusted, calendar-aligned panel plus an explicit anomaly ledger.
Capability under test
Data hygiene under adversarial conditions: detecting rather than silently smoothing defects, distinguishing genuine market events from data errors, and applying corporate-action math exactly.
Design notes
Anomaly placement is sampled from the trial seed at image build; the generator is removed from the image and re-run by the verifier. Scoring separates panel accuracy from anomaly recall/precision so that an agent which produces correct numbers by luck but cannot explain why still loses credit. Calibration: a 'trap-blind' cleaner (drop duplicates, forward-fill everything) must score below 0.5.
Tier ladder
- Tier 11
- Tier 22
- Tier 32
- Tier 40
| Task | Tier | Phase | Reward | Gates | Budget | Status |
|---|---|---|---|---|---|---|
| Profile raw price files against calendar and metadata pf-data-provenance-profile Profile every ticker's raw price file against the trading calendar and metadata and emit a structured data-quality report with exact counts. | T1 easy | Analyze | partial | — | 15 min | ready |
| Build a canonical adjusted price panel from messy vendor files pf-data-canonical-panel Turn messy multi-format price files plus a corporate-actions file into a canonical split- and dividend-adjusted close panel aligned to the trading calendar, and report every anomaly found. | T2 medium | Analyze | multi-metric | — | 40 min | ready |
| Multi-currency, multi-calendar alignment to USD returns pf-data-currency-and-calendar Merge a London-listed ETF quoted in pence with USD ETFs and FX rates, aligning trading calendars per an explicit rule, and produce USD daily returns. | T2 medium | Analyze | partial | — | 30 min | ready |
| Reconcile custodian dividends against the vendor file and rebuild total-return indices pf-data-dividend-reconciliation Reconcile dividends in the account transactions log against the vendor dividend file, find the ticker whose vendor records are stamped on pay-date instead of ex-date, and build a correct total-return index. | T3 hard | Analyze | multi-metric | — | 40 min | ready |
| Detect an undocumented split without misclassifying a genuine crash pf-data-hidden-split Detect a split missing from the corporate-actions file and adjust for it, while leaving a genuine one-day crash untouched. | T3 hard | Analyze | multi-metric | — | 40 min | ready |