[scb_to_harbor] Held-out tests leak to the agent across checkpoints in Harbor multi-step runs

[cvbhjkl]

[scb_to_harbor] Held-out tests leak to the agent across checkpoints in Harbor multi-step runs

Summary

Harbor tasks generated by scripts/scb_to_harbor.py leave the full verifier
test suite at /tests inside the agent's container. Because Harbor reuses the
same container for every step of a trial, from checkpoint_2 onward the agent
can read all checkpoint test files — including hidden (functionality /
error) cases and the tests of future checkpoints that have not been
presented yet. This silently breaks the benchmark's core guarantee and
inflates scores.

This affects the published Harbor dataset
(gabeorlanski/slopcodebench on the Harbor registry) as well as any local
conversion produced from current main (observed on main@ef6a9dd).

Root cause

scripts/scb_to_harbor_lib/renderers.py::render_test_sh generates each step's
test.sh. The script runs pytest against /tests/test_checkpoint_N.py,
computes rewards, and then exits — it never removes /tests or the report
files (/tmp/ctrf.json, /tmp/pytest-report.json,
/tmp/scb-check-report.json). The generated _ctrf_to_reward.py shim itself
documents the shared-container assumption:

"Harbor keeps the same container for every step in a trial, so the history
file persists across steps …"

So after the checkpoint_1 verifier run, /tests contains
test_checkpoint_1..N.py, conftest.py, helpers, and all test data/case
files for the entire problem — readable by the agent in every subsequent
agent phase. /logs/verifier/reward*.json from prior steps is likewise
visible. In contrast, the native SCBench runner uses a fresh container per
checkpoint where only the workspace persists, so held-out tests are never
exposed.

Evidence from a real run

We ran the converted dataset (36 problems / 196 checkpoints) with a coding
agent and inspected the agent trajectories. In 8 of 36 problems the agent
found and used /tests during its own phase, for example:

• cat /tests/test_checkpoint_5.py /tests/test_checkpoint_4.py /tests/conftest.py
  — issued at checkpoint_4, i.e. reading the next checkpoint's tests
  before seeing its spec (file_query_tool)
• cat /tests/data/checkpoint_5/hidden/nested_structures.json — reading
  hidden test fixtures (file_query_tool)
• cd /tests && uv run ... -m pytest test_checkpoint_1.py ... --checkpoint checkpoint_4
  — running the held-out suite directly to iterate against it
  (log_query, recli, eve_jump_planner, eve_industry)
• cat /tests/cases/checkpoint_2/hidden/*.json, importing helper functions
  from conftest.py (eve_market_tools, eve_industry)
• cat /tmp/scbp/... ; cat /tests/test_checkpoint_2.py | head -120 plus
  cat /tests/test.sh (eve_jump_planner)

Consequences visible in scores: on one problem (eve_jump_planner) the agent
achieved a 100% strict solve (all checkpoints, all tests) — an outcome
the paper reports no agent achieves on any problem — and the top-scoring
problems in our run (eve_industry, log_query, file_query_tool ≈ 0.99 mean
pass rate) are exactly the ones where /tests was read. The contamination
makes Harbor-run scores non-comparable with native-runner results.

Suggested fixes

Any of these would close the hole; the first is a two-line change:

1. At the end of generated test.sh (after _ctrf_to_reward.py runs):

   rm -rf /tests /tmp/ctrf.json /tmp/pytest-report.json \
          /tmp/scb-check-report.json /tmp/scb-check.stderr

   (and re-copy fresh tests in at each verifier phase, which Harbor already
   does). Note /tmp/scb-check-history.jsonl must be kept since the shim
   relies on it across steps.
2. Ensure /logs/verifier is not readable by the agent user between steps
   (it contains per-group pass counts).
3. Longer term, match the native runner's isolation: run each checkpoint's
   verifier in a fresh container / overlay so no verifier state persists.

Happy to open a PR with fix (1) plus a converter regression test that asserts
/tests is absent after test.sh completes.

Environment

• scb-problems main@ef6a9dd (latest as of 2026-09-01)
• Harbor multi-step execution (msh AgentGym runtime), same-container steps
• Observed 2026-08-31