Benchmarks with base-branch comparison on pull requests

[johanrd]

mitata benchmarks (pnpm bench, test/validate.bench.mjs) and a comparison harness adapted from ember-estree (pnpm bench:compare --base <branch>; .github/workflows/bench-compare.yml posts the comparison on every pull request, or writes it to the job summary for fork PRs).

Two kinds of benchmark, because the regressions #51 fixes lived in different places and pnpm test could not see either:

• in-process: extractAttrTypeMap per fixture (small, medium, cross-file resolution), disk cache off
• whole process: dist/run.js over examples/ — cold (cache off), warm (all cached), one cached file, --no-glint

Main vs main on an M1 Max for the noise floor:

   Benchmark                       Control (p50)   Experiment (p50)        Δ
🟡 extract small template              255.38 µs          243.83 µs    -4.5%
🟢 extract medium template               1.69 ms            1.45 ms   -14.0%
🟡 extract cross-file resolution         1.49 ms            1.46 ms    -2.4%
🟢 cold run (cache off)               1375.98 ms         1225.22 ms   -11.0%
⚪ warm run (all cached)               863.43 ms          871.30 ms    +0.9%
🟡 one cached file                     255.66 ms          266.45 ms    +4.2%
🟡 no glint                           5960.16 ms         5735.63 ms    -3.8%

Also: tsconfig.json now declares types: ["node"]. The build only found @types/node through vite's type references (via vitest.config.ts matching include: ["*.ts"]); an export of the sources alone did not compile under the tsc that pnpm build resolves to, which is the typescript-7 alias's binary since #48.

Cowritten by Claude

[Copilot]

🔵 Needs a closer look

It changes build configuration (tsconfig.json types) affecting the shipped artifact and adds a CI workflow that builds/runs base-branch code, so it warrants human review alongside the flagged comment-pagination bug.

Pull request overview

This PR adds a performance-benchmarking harness (using mitata) with base-branch comparison, targeted at guarding the regressions fixed in #51/#48. It benchmarks both in-process extractAttrTypeMap calls and whole dist/run.js runs over examples/, and wires a CI workflow that posts (or, for fork PRs, summarizes) the comparison on every pull request. It also adds "types": ["node"] to tsconfig.json so the sources compile under the typescript-7 tsc used since #48.

Changes:

• New mitata benchmarks (test/validate.bench.mjs) plus a base-branch comparison harness (scripts/bench-compare.mjs) that git-archives, builds, and runs the base ref side by side.
• Formatting utilities for the comparison (format-bench-comment.mjs for PR markdown, format-bench-cli.mjs for terminal, local-bench-summary.sh for local runs) and a CI workflow to publish results.
• Supporting config: tsconfig.json types: ["node"], mitata dependency, package.json bench scripts, .gitignore, and README docs.

File summaries

File	Description
test/validate.bench.mjs	New mitata benchmarks (in-process extraction + whole-process runs) with optional control-dir comparison and JSON output.
scripts/bench-compare.mjs	Exports/installs/builds the base ref into a temp dir and runs the benchmarks against it, with taskset pinning.
scripts/format-bench-comment.mjs	Builds a GitHub markdown comment (summary table + collapsible mitata output) from the JSON/text results.
scripts/format-bench-cli.mjs	Renders a terminal-friendly comparison table from the JSON results.
scripts/local-bench-summary.sh	Local helper that warns on CPU tuning and prints the CLI summary.
.github/workflows/bench-compare.yml	PR workflow that runs the comparison and posts/summarizes it (fork-safe).
tsconfig.json	Declares types: ["node"] so sources compile under the TS7 tsc.
package.json / pnpm-lock.yaml	Adds the pinned mitata dev dependency and bench/bench:compare/bench:summary scripts.
README.md	Documents the new benchmark workflow.
.gitignore	Ignores bench-results.json.

Review details

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

• Files reviewed: 8/11 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🏎️ Benchmark Comparison

	Benchmark	Control (p50)	Experiment (p50)	Δ
⚪	extract small template	825.65 µs	818.70 µs	-0.8%
⚪	extract medium template	3.52 ms	3.57 ms	+1.3%
⚪	extract cross-file resolution	4.05 ms	4.13 ms	+1.8%
⚪	cold run (cache off)	1770.18 ms	1786.26 ms	+0.9%
⚪	warm run (all cached)	1398.21 ms	1431.58 ms	+2.4%
⚪	one cached file	525.07 ms	516.29 ms	-1.7%
⚪	no glint	4735.23 ms	4685.76 ms	-1.0%

🟢 faster · 🔴 slower · 🟡 5–10 %, possibly noise · ⚪ within 5 %

Full output

clk: ~2.77 GHz
cpu: AMD EPYC 9V74 80-Core Processor
runtime: node 24.19.0 (x64-linux)

benchmark                    avg (min … max) p75 / p99    (min … top 1%)
-------------------------------------------- -------------------------------
extract small template          1.20 ms/iter   1.19 ms  █                   
                       (688.61 µs … 6.77 ms)   3.91 ms ██                   
                     ( 24.84 kb …   4.31 mb) 315.06 kb ███▅▄▃▂▂▂▁▂▁▁▁▂▂▂▂▁▂▂

extract medium template         4.20 ms/iter   4.33 ms  █                   
                        (2.99 ms … 13.22 ms)  10.49 ms  █▂                  
                     (339.14 kb …   5.18 mb)   0.99 mb ███▄▃▃▂▂▃▄▂▁▁▁▁▁▁▁▁▁▁

extract cross-file resolution   4.62 ms/iter   5.03 ms  █▄                  
                        (3.46 ms … 10.69 ms)   8.96 ms  ██                  
                     (228.69 kb …   2.33 mb)   1.29 mb ███▇▆▄▄▃▃▂▆▃▃▂▁▁▁▂▂▂▂

whole process (min / p50 of 3 runs, 20 files)
  cold run (cache off)           1758 ms /   1770 ms
  warm run (all cached)          1390 ms /   1398 ms
  one cached file                 503 ms /    525 ms
  no glint                       4617 ms /   4735 ms
clk: ~2.75 GHz
cpu: AMD EPYC 9V74 80-Core Processor
runtime: node 24.19.0 (x64-linux)

benchmark                    avg (min … max) p75 / p99    (min … top 1%)
-------------------------------------------- -------------------------------
extract small template          1.19 ms/iter   1.11 ms  █                   
                       (655.83 µs … 6.15 ms)   4.11 ms  █                   
                     ( 24.84 kb …   3.53 mb) 311.82 kb ██▇▄▂▃▂▂▁▂▂▁▁▁▂▁▂▁▂▁▁

extract medium template         4.23 ms/iter   4.45 ms  █                   
                        (3.06 ms … 10.78 ms)   9.88 ms ▅█▄                  
                     (110.05 kb …   4.44 mb)   0.99 mb ███▅▄▂▄▂▄▂▃▂▃▁▁▁▁▁▁▁▁

extract cross-file resolution   4.59 ms/iter   4.86 ms  █                   
                         (3.53 ms … 9.26 ms)   8.93 ms ▂█▂▃                 
                     (433.97 kb …   2.09 mb)   1.28 mb ████▆▆▂▃▂▁▄▆▂▁▂▂▂▁▁▂▂

whole process (min / p50 of 3 runs, 20 files)
  cold run (cache off)           1783 ms /   1786 ms
  warm run (all cached)          1419 ms /   1432 ms
  one cached file                 513 ms /    516 ms
  no glint                       4662 ms /   4686 ms