Skip to content

Fixed bench-compare to run the comparison benchmarks - #27

Merged
alexeyco merged 3 commits into
masterfrom
chore/fix-bench-compare-target
Nov 18, 2026
Merged

alexeyco merged 3 commits into
masterfrom
chore/fix-bench-compare-target

Conversation

@alexeyco

Copy link
Copy Markdown
Owner

Summary

make bench-compare ran zero benchmarks. The recipe entered the isolated comparison module but invoked go test on that module's root package, which only holds main.go, so the command printed [no test files] and exited 0. AGENTS.md already claimed the target "runs the third-party comparison benchmarks", and the figures published in v1.2.1 had to be captured with an ad-hoc command instead of the Makefile target.

This change makes the target actually run the three comparison benchmarks and locks that selection against regression.

Changes

  • Makefile — bench-compare now runs cd $(BENCH_DIR) && go test -run '^$$' -bench . -benchmem ./bench: an explicit ./bench package and the same -run '^$$' guard the root bench target uses, so no unit test can execute under a benchmark invocation.
  • New make bench-compare-check — a lock that captures go test -list 'Benchmark.*' ./bench and fails with bench-compare-check: missing benchmark <name> unless BenchmarkSimpletable, BenchmarkTermtables, and BenchmarkUITable each appear as a whole line. Listing is asserted rather than the exit status, because go test -list exits 0 even when no benchmark matches. It is wired into make check after fmt-check.
  • Deliberately declined mechanisms: asserting that the output does not contain [no test files] (with ./... the benchmarks run and that line still prints, and an empty root test file would make it falsely pass), and set-equality on the benchmark list (an added fourth benchmark does not fail the lock; only one of the three locked names disappearing does).
  • Documentation surfaces — AGENTS.md names the bench package of _example/03-benchmarks-with-others; the harness README's ## Installation section drops the obsolete go get lines in favour of the module-aware statement that both comparators are pinned in that module's go.mod, fetched automatically, with go mod download from the module directory as the explicit step; its ## Run benchmarks section names make bench-compare as the repo-root invocation and keeps a repo-relative manual form. The ## Run benchmarks heading is byte-identical so the root README's #run-benchmarks anchor keeps resolving; the root README.md is not modified.
  • OpenSpec — adds requirements Comparison benchmark gate (five scenarios) and Comparison benchmark documentation (two scenarios) to the repository-quality-gates capability, with the change artifacts archived in this diff. The capability's ## Purpose is widened to cover the local make check aggregate as well as the GitHub Actions workflow, because the new requirement describes a local target that CI does not invoke.
  • Not included: running the lock in CI, and the harness README's ## Show section (its pre-module $GOPATH/src/... line is left as-is).

Test plan

  • make bench-compare — prints all three benchmark lines followed by PASS, exit 0.
  • make bench-compare-check — exit 0 on this tree; the failure path was proven by narrowing the -list pattern to 'BenchmarkSimpletable.*', which exited non-zero and reported missing benchmark BenchmarkTermtables, and the temporary edit was fully reverted.
  • make check — full local suite (test, build, lint and fmt for both modules, the new lock, OpenSpec validation) exits 0; make -n check shows the lock between fmt-check and openspec-validate.
  • openspec validate --all --strict --no-interactive — 5 specs pass, and --all --strict --archived passes for the archived change.
  • make help — lists all pre-existing targets plus bench-compare-check.
  • Anchor checks — grep -c '^## Run benchmarks$' _example/03-benchmarks-with-others/README.md prints 1, grep -n 'go get' finds nothing, and the root README link to #run-benchmarks is unchanged.

Known residual gap

The lock protects the three benchmark names in the bench package; it does not guard the shape of the bench-compare recipe itself. Reverting that recipe to the old argument-less form would still leave make check green, because the lock queries ./bench independently. This is recorded in the archived change's verification report.

@alexeyco
alexeyco merged commit a3414f9 into master Nov 18, 2026
2 checks passed
@alexeyco
alexeyco deleted the chore/fix-bench-compare-target branch November 18, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant