Debugging, Tooling & Performance/pf-tool-reproducibility-pack

Turn a non-reproducible notebook into a byte-identical pipeline

Make an existing analysis notebook fully reproducible with a pinned environment, data checksums, deterministic seeds and a single make target that regenerates byte-identical outputs.

T2mediumCross-cuttingmulti-metricready

reproducibility engineering-hygiene

Edit

readystatic

Agent budget: 30 min
Verifier budget: 10 min
Tier target: 60–80% pass expected

instruction.md

What the agent sees (CONVENTIONS.md is appended automatically)

Make it reproducible

/app/repo/analysis.ipynb produces a results folder but yields different numbers each run and depends on an unpinned environment. Convert it into a script-based pipeline with: requirements.lock (exact versions), data/MANIFEST.sha256 covering every input file, deterministic seeds, a Makefile target reproduce that rebuilds /app/output/results from scratch, and RESULTS.md documenting the numbers.

The verifier runs make reproduce twice in fresh copies and compares output hashes, and validates the manifest against the inputs.

Verification

reward.json metrics · weights sum to 1.00

Metric	Weight	Check
byte_identical_runs	0.5	SHA-256 of every output identical across two fresh runs.
manifest_valid	0.2	Manifest covers all inputs and matches.
numbers_match_intent	0.2	Results within 1e-6 of oracle after seeding.
lockfile_complete	0.1	Every imported package pinned.

Harbor scaffold

Generated from this record — task.toml, Dockerfile, verifier, oracle stub

Download all (JSON)

schema_version = "1.4"

[task]
name = "portfolio-agent-evals/pf-tool-reproducibility-pack"
version = "1.0.0"
description = "Make an existing analysis notebook fully reproducible with a pinned environment, data checksums, deterministic seeds and a single make target that regenerates byte-identical outputs."
keywords = ["etf", "portfolio", "cross-cutting", "debugging-tooling", "reproducibility", "engineering-hygiene"]

[metadata]
author_name = "portfolio-agent-evals"
difficulty = "medium"
category = "quant-finance"
tags = ["debugging-tooling", "tier-2", "cross-cutting", "multi-metric"]
theme = "Debugging, Tooling & Performance"
tier = 2
reward_type = "multi-metric"

[agent]
timeout_sec = 1800.0

[verifier]
timeout_sec = 600.0

[environment]
# Offline by design: all data is synthetic and generated at build time.
network_mode = "none"
cpus = 2
memory_mb = 4096
storage_mb = 10240
build_timeout_sec = 900.0

Traps

Each must carry signal: a trap-blind solution must lose credit

Unseeded bootstrap; set iteration order; timestamps in output files.
BLAS thread nondeterminism in reductions — fix thread count or use stable reductions.
Floating-point summation order in groupby.

Inputs

Fixtures mounted in the environment

repo-artifacts
/app/repo/analysis.ipynb
prices-clean
/app/data/prices/
household
/app/portfolio/

Outputs

What the verifier reads from /app/output

/app/repo/Makefile
make
reproduce target.
/app/repo/requirements.lock
text
Pinned environment.
/app/repo/data/MANIFEST.sha256
text
Input checksums.
/app/output/results/
files
Regenerated outputs.

Anti-gaming

Nondeterminism sources are seeded from a pool of six.

Oracle notes

solution/solve.sh must score 1.0 on five seeds

Oracle pipeline uses OMP_NUM_THREADS=1 and sorted iteration.

Reviews (0)

Design review before a task is marked ready

Reviews are read-only in static export.

Tier 2 · Practitioner — 60–80% pass expected. Created 2026-01-01, updated 2026-01-01.