Repository navigation
Phase 0 baseline: strengthen turn-benchmark oracles (read-only verifiers, positive refactor oracle) #701
Description
Follow-up to the Phase 0 turn-trace baseline (#700). The baseline lands with a per-oracle-tier pass/fail split so the report cannot be misread as a blanket correctness verdict:
- Correctness (18 tasks: edit + fix) — positive oracles (substring grep, scoped
go test). The only pass rate that can move with model quality. - Build-only (6 tasks: refactor) —
go build ./..., reported asbuildPassRate, excluded fromcorrectnessPassRate. - Latency-only (24 tasks: nav/longproc/longctx/parallel) — no oracle; counted in
latencyOnlyTasks, excluded from every pass rate.
The tier split keeps the report honest, so this hardening is not a blocker — but the oracles are intentionally lightweight for Phase 0 and should be strengthened before a baseline report is treated as a correctness signal.
Work
- Edit grep oracles are substring checks. An add-field oracle (
grep -R Label .) only proves the string appears, not that it landed on the right struct. Addgo vet/go buildplus a structural assertion (e.g. the field on the named struct) for add-field/add-method tasks. Rename oracles already assert the old name is gone. - Refactor build oracle is non-positive.
go build ./...passes a no-op refactor. Add a structural verifier that proves the refactor happened (extracted helper exists, call sites updated), or document refactor as permanently structure-uncorrected. - 24 read-only tasks have no oracle. For nav, a deterministic verifier is realistic (
diff -q <(expected) <(agent-output)if the harness pipes the model's final text to a known path). longproc/longctx/parallel summarize with no canonical answer — document them as permanently latency-only.
Where
- Manifest:
internal/perfbench/manifests/baseline.json - Runner / tier logic:
internal/perfbench/turn_bench.go(NewTurnExecRunner,RunTurnBench) - Tier contract:
internal/perfbench/MANIFEST.md
Activity
Address CodeRabbit re-review: anti-averaging README + fixture-isolati…
Vasanthdev2004 commentedon Jul 16, 2026
Tracking one more non-blocker from the PR #700 re-review: rename the
LatencySourceJSON fieldshare→exclusiveShareso the contract makes explicit that the share is computed on exclusive time, not a raw duration sum. It's a v2 schema field (nothing downstream consumes it yet), so it's a safe deliberate rename for the hardening pass rather than mid-baseline. Pulling it in here alongside the refactor structural verifier so the next baseline report ships a self-describing contract.Add per-turn tracing and a turn-benchmark harness (Phase 0 baseline) (#…
Address gnanam1990 review on PR #712: close count=0 gameability + str…
Close nav-04/05 blind-count-one gameability (anandh8x review)
perf(turn-bench): Phase 0 — strengthen oracles so pass rate can't be …
Vasanthdev2004 commentedon Jul 27, 2026
Closing: this landed in #712, with #730, #737 and #763 refining it since. I checked each of the three asks against
mainrather than going by the PR title.Edit oracles are no longer bare substring greps. They now compile a stamped oracle and assert the old name is gone:
The refactor oracle is positive. A no-op refactor fails it, because the count is asserted rather than the build:
buildOnlyClassesis now[], so refactor has graduated out of the build tier into correctness, which is what made the build-only tier empty.The read-only tasks are settled both ways. nav carries a deterministic verifier against the stamped answer file and has graduated from latency-only to correctness. longproc, longctx and parallel remain oracle-free and are documented as permanently latency-only in
MANIFEST.md, with the reasoning stated: the answer is free-form prose and a contains-check would rubber-stamp it.Counts on
main: 48 tasks, 34 carrying an oracle (edit 10, fix 8, nav 10, refactor 6). The 14 without are exactly longproc, longctx and parallel.internal/perfbenchpasses.The part worth keeping is that the manifest documents why three classes have no oracle rather than leaving them silently unverified. A pass rate computed from oracles that cannot fail is the same defect as a test that passes with the code deleted, and harder to notice because it looks like a number.