3/3 Add harness-engineering-bench (GAIA + benchmarks)

[varunursekar]

Third of three stacked PRs splitting #41. Base: pr2-add-vero.

Adds harness-engineering-bench/ with each benchmark as a top-level sibling (gaia, ale-bench, officeqa, swe-atlas-qna, tau3) + scripts. Pure adds (71 files).

Union of 1/3 + 2/3 + 3/3 == v0.5 plus a legacy/ archive (verified).

🤖 Generated with Claude Code

Greptile Summary

This PR adds harness-engineering-bench/ with five benchmark integrations — browsecomp-plus, gaia, officeqa, swe-atlas-qna, and tau3 — each as a self-contained directory with a baseline agent, partitions, and build scripts, alongside a new vero/src/vero/sandbox.py abstraction layer and structural tests in vero/tests/.

• browsecomp-plus is a new deep-research benchmark backed by a pinned BM25 corpus; its agent, task builder, verifier, and search tool are all new. The per_trial_tokens.py previously-flagged None key bug is resolved in this version (or "unattributed" at line 128).
• vero/src/vero/sandbox.py adds LocalSandbox and DockerSandbox with proper POSIX process-group cleanup and PID-file-based container termination.
• test_build_tasks.py contains a broken assertion that compares the number of partitioned tasks (165) against EXPECTED_TASKS (830), causing the test to always fail when run.

Confidence Score: 4/5

Safe to merge with the test assertion fix — the broken assertion in test_build_tasks.py will cause that test to fail whenever run, but does not affect production evaluation.

The broken test in test_build_tasks.py compares 165 partitioned tasks against EXPECTED_TASKS=830 and will always fail. Everything else in this pure-additions PR — the browsecomp-plus agent, sandbox abstraction, build scripts, and structural config tests — looks correct. The previously flagged issues in gaia/officeqa/tau3/ale-bench (from earlier stacked PRs) are unresolved but pre-existing relative to this diff.

Files Needing Attention: harness-engineering-bench/browsecomp-plus/scripts/test_build_tasks.py (broken assertion) and harness-engineering-bench/browsecomp-plus/task-template/tests/evaluate.py (format-string injection flagged in prior review, still unresolved).

Important Files Changed

Filename	Overview
harness-engineering-bench/browsecomp-plus/scripts/test_build_tasks.py	Test assertion == EXPECTED_TASKS will always fail — _partition returns 165 tasks, not 830
harness-engineering-bench/browsecomp-plus/baseline/target/src/browsecomp_plus_agent/agent.py	New BrowseComp-Plus Chat Completions agent; handles max-turns exhaustion correctly with a best-effort fallback; no parallel_tool_calls=False but lower impact here since submit_response terminates the loop
harness-engineering-bench/browsecomp-plus/task-template/tests/evaluate.py	BrowseComp-Plus verifier; GRADER_TEMPLATE.format(...) will raise KeyError if question/response/answer contain curly braces (previously flagged)
harness-engineering-bench/scripts/per_trial_tokens.py	Post-hoc token accounting script; previously-flagged None evaluation key issue is fixed via or 'unattributed' fallback at line 128
harness-engineering-bench/browsecomp-plus/scripts/build_tasks.py	Reproducible task builder with pinned dataset/index revisions, deterministic partitioning, atomic output rename, and safe shell quoting; looks correct
vero/src/vero/sandbox.py	New sandbox abstraction (Local + Docker) with proper process-group cleanup, PID-file-based container termination, and atomic file transfer; no issues found
vero/tests/test_v05_benchmark_configs.py	Structural tests verifying all five benchmarks route inference through the gateway, enforce model allow-lists, and scope optimizer/evaluation budgets independently

Sequence Diagram

sequenceDiagram
    participant H as Harbor Harness
    participant A as BrowseCompPlusAgent
    participant GW as Inference Gateway
    participant M as OpenAI Model
    participant E as Environment (corpus)
    participant V as Verifier (evaluate.py)

    H->>A: run(instruction, environment, context)
    loop Turn 1..MAX_TURNS
        A->>GW: chat.completions.create(messages, tools)
        GW->>M: proxied request (budget metered)
        M-->>GW: tool_calls or content
        GW-->>A: response
        alt tool_calls present
            loop each tool call
                A->>E: exec search.py / get-document
                E-->>A: JSON result
                A->>A: append tool result to messages
                note over A: break if submit_response called
            end
        else no tool calls (content only)
            A->>E: upload answer.txt
            A->>A: set context.metadata, break
        end
    end
    note over A: else branch (budget exhausted): force final answer call
    A->>E: upload answer.txt
    H->>V: python evaluate.py
    V->>M: responses.create (judge prompt)
    M-->>V: correct yes/no
    V->>V: write reward.txt

Loading

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
harness-engineering-bench/browsecomp-plus/scripts/test_build_tasks.py:35
The assertion `== EXPECTED_TASKS` will always fail because `_partition` intentionally returns only `sum(PARTITION_COUNTS.values()) = 165` partitioned tasks out of `EXPECTED_TASKS = 830` — the remaining rows are rendered as task dirs but left unassigned (per the `_partition` docstring). Running this test today would produce `AssertionError: assert 165 == 165 == 830`. The intent appears to be verifying no duplicates and that the total equals the sum of partition counts.

```suggestion
    assert len(flattened) == len(set(flattened)) == sum(PARTITION_COUNTS.values())
```

_{Reviews (42): Last reviewed commit: "Pin gaia held-out baseline (0.5736) with..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​tqdm@​4.68.4
	pypi/​tqdm@​4.69.0
	pypi/​click@​8.4.2
	pypi/​uvicorn@​0.51.0
	pypi/​tiktoken@​0.13.0
	pypi/​requests@​2.34.2
	pypi/​fastapi@​0.139.2
	pypi/​fastapi@​0.140.0

View full report

[greptile-apps]

The judge image is pulled by mutable tag (cpp20-202301) without a digest pin. Docker Hub tags can be silently overwritten, so two scoring runs could pull different binaries under the same tag — directly threatening benchmark reproducibility. In the worst case, a compromised yimjk account could push a malicious image that gets pulled at evaluation time. Pinning to the image digest locks the exact bytes regardless of tag mutations.

Suggested change

	docker pull yimjk/ale-bench:cpp20-202301
	docker tag yimjk/ale-bench:cpp20-202301 ale-bench:cpp20-202301
	docker pull yimjk/ale-bench:cpp20-202301@sha256:<DIGEST>
	docker tag yimjk/ale-bench:cpp20-202301@sha256:<DIGEST> ale-bench:cpp20-202301

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-engineering-bench/ale-bench/environment/sidecar/dind-entrypoint.sh
Line: 23-24

Comment:
The judge image is pulled by mutable tag (`cpp20-202301`) without a digest pin. Docker Hub tags can be silently overwritten, so two scoring runs could pull different binaries under the same tag — directly threatening benchmark reproducibility. In the worst case, a compromised `yimjk` account could push a malicious image that gets pulled at evaluation time. Pinning to the image digest locks the exact bytes regardless of tag mutations.

```suggestion
docker pull yimjk/ale-bench:cpp20-202301@sha256:<DIGEST>
docker tag yimjk/ale-bench:cpp20-202301@sha256:<DIGEST> ale-bench:cpp20-202301
```

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

endswith("ACCEPTED") will incorrectly match any string that ends with "ACCEPTED" — most notably "NOT_ACCEPTED" (or an enum-style repr like "JudgeResult.NOT_ACCEPTED"). If ALE-Bench returns such a value for a failing submission, the accepted metric would be written as 1.0 for a rejected run. Using exact equality is unambiguous regardless of what the library returns.

Suggested change

	accepted = 1.0 if judge.upper().endswith("ACCEPTED") else 0.0
	accepted = 1.0 if judge.upper() == "ACCEPTED" else 0.0

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-engineering-bench/ale-bench/environment/sidecar/harness/score.py
Line: 83

Comment:
`endswith("ACCEPTED")` will incorrectly match any string that ends with "ACCEPTED" — most notably "NOT_ACCEPTED" (or an enum-style repr like `"JudgeResult.NOT_ACCEPTED"`). If ALE-Bench returns such a value for a failing submission, the `accepted` metric would be written as `1.0` for a rejected run. Using exact equality is unambiguous regardless of what the library returns.

```suggestion
        accepted = 1.0 if judge.upper() == "ACCEPTED" else 0.0
```

How can I resolve this? If you propose a fix, please make it concise.

[shehabyasser-scale]

Approve with nits. The split is the part I care most about for experiment validity, and it checks out.

GAIA split is genuinely leak-safe (verified):

• Deterministic stratified partition keyed by sha256("vero-gaia-v1:{task}") (partition_gaia.py:116), no RNG, with a --check mode that fails CI if the committed manifest drifts.
• Verified disjoint: development 33 / validation 66 / test 66, zero pairwise overlap, union 165.
• Wiring is correct: development at disclosure: full, validation at aggregate (min_aggregate_cases: 5, expose_case_resources: false), selection on validation, and reward/target on the test partition which is never agent-visible. A proper held-out test set.
• k-anonymity floors are safe in every build.yaml (tau3 / officeqa / swe-atlas-qna / gaia all set 5). The ale-bench endswith("ACCEPTED") judge is fine too: upstream JudgeResult has no NOT_ACCEPTED member, so no failing verdict ends in that string.

One real defect (baseline quality, not integrity): the tau3 baseline drops conversation history after every plain-text turn. tau3_agent/agent.py:370 sets previous_response_id = None in the non-tool branch, while the tool branch threads it correctly (:347). After any non-terminal customer reply the model loses the domain policy (only present in the turn-0 input), prior tool results, and reasoning state, which depresses tau3 from the first follow-up onward. The Responses API supports threading after a message output, so None isn't required. Suggest previous_response_id = response.id on the text branch as well. Confirmed tau3-specific; the GAIA baseline has no such reset.

One thing to note for readers: every build.yaml ships infrastructure_max_attempts: 3 + aggregate_attempts: best, so the competitive-selection integrity of these benchmarks depends on the aggregate/retry fix over on #45, not on any change here.

[greptile-apps]

The assertion == EXPECTED_TASKS will always fail because _partition intentionally returns only sum(PARTITION_COUNTS.values()) = 165 partitioned tasks out of EXPECTED_TASKS = 830 — the remaining rows are rendered as task dirs but left unassigned (per the _partition docstring). Running this test today would produce AssertionError: assert 165 == 165 == 830. The intent appears to be verifying no duplicates and that the total equals the sum of partition counts.

Suggested change

	assert len(flattened) == len(set(flattened)) == EXPECTED_TASKS
	assert len(flattened) == len(set(flattened)) == sum(PARTITION_COUNTS.values())

Prompt To Fix With AI

This is a comment left during a code review.
Path: harness-engineering-bench/browsecomp-plus/scripts/test_build_tasks.py
Line: 35

Comment:
The assertion `== EXPECTED_TASKS` will always fail because `_partition` intentionally returns only `sum(PARTITION_COUNTS.values()) = 165` partitioned tasks out of `EXPECTED_TASKS = 830` — the remaining rows are rendered as task dirs but left unassigned (per the `_partition` docstring). Running this test today would produce `AssertionError: assert 165 == 165 == 830`. The intent appears to be verifying no duplicates and that the total equals the sum of partition counts.

```suggestion
    assert len(flattened) == len(set(flattened)) == sum(PARTITION_COUNTS.values())
```

How can I resolve this? If you propose a fix, please make it concise.