Repository navigation
Fixed bench-compare to run the comparison benchmarks - #27
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
make bench-compareran zero benchmarks. The recipe entered the isolated comparison module but invokedgo teston that module's root package, which only holdsmain.go, so the command printed[no test files]and exited 0.AGENTS.mdalready claimed the target "runs the third-party comparison benchmarks", and the figures published inv1.2.1had to be captured with an ad-hoc command instead of the Makefile target.This change makes the target actually run the three comparison benchmarks and locks that selection against regression.
Changes
bench-comparenow runscd $(BENCH_DIR) && go test -run '^$$' -bench . -benchmem ./bench: an explicit./benchpackage and the same-run '^$$'guard the rootbenchtarget uses, so no unit test can execute under a benchmark invocation.make bench-compare-check— a lock that capturesgo test -list 'Benchmark.*' ./benchand fails withbench-compare-check: missing benchmark <name>unlessBenchmarkSimpletable,BenchmarkTermtables, andBenchmarkUITableeach appear as a whole line. Listing is asserted rather than the exit status, becausego test -listexits 0 even when no benchmark matches. It is wired intomake checkafterfmt-check.[no test files](with./...the benchmarks run and that line still prints, and an empty root test file would make it falsely pass), and set-equality on the benchmark list (an added fourth benchmark does not fail the lock; only one of the three locked names disappearing does).AGENTS.mdnames thebenchpackage of_example/03-benchmarks-with-others; the harness README's## Installationsection drops the obsoletego getlines in favour of the module-aware statement that both comparators are pinned in that module'sgo.mod, fetched automatically, withgo mod downloadfrom the module directory as the explicit step; its## Run benchmarkssection namesmake bench-compareas the repo-root invocation and keeps a repo-relative manual form. The## Run benchmarksheading is byte-identical so the root README's#run-benchmarksanchor keeps resolving; the rootREADME.mdis not modified.Comparison benchmark gate(five scenarios) andComparison benchmark documentation(two scenarios) to therepository-quality-gatescapability, with the change artifacts archived in this diff. The capability's## Purposeis widened to cover the localmake checkaggregate as well as the GitHub Actions workflow, because the new requirement describes a local target that CI does not invoke.## Showsection (its pre-module$GOPATH/src/...line is left as-is).Test plan
make bench-compare— prints all three benchmark lines followed byPASS, exit 0.make bench-compare-check— exit 0 on this tree; the failure path was proven by narrowing the-listpattern to'BenchmarkSimpletable.*', which exited non-zero and reportedmissing benchmark BenchmarkTermtables, and the temporary edit was fully reverted.make check— full local suite (test, build, lint and fmt for both modules, the new lock, OpenSpec validation) exits 0;make -n checkshows the lock betweenfmt-checkandopenspec-validate.openspec validate --all --strict --no-interactive— 5 specs pass, and--all --strict --archivedpasses for the archived change.make help— lists all pre-existing targets plusbench-compare-check.grep -c '^## Run benchmarks$' _example/03-benchmarks-with-others/README.mdprints1,grep -n 'go get'finds nothing, and the root README link to#run-benchmarksis unchanged.Known residual gap
The lock protects the three benchmark names in the
benchpackage; it does not guard the shape of thebench-comparerecipe itself. Reverting that recipe to the old argument-less form would still leavemake checkgreen, because the lock queries./benchindependently. This is recorded in the archived change's verification report.