Gate CI on benchmark regression thresholds

[medley56]

Summary

CI now fails the build when a benchmark in tests/benchmark/ regresses. Every benchmark test asserts its mean run time against a threshold declared next to it, enforced on ubuntu-latest for each supported Python version (3.10 to 3.14).

How the gate works

• Thresholds are in seconds on a nominal machine, not raw seconds. GitHub hands the same job to runners whose speed differs by up to ~2x from run to run, and that difference is common to every benchmark in the job. A machine_speed_factor fixture in tests/benchmark/conftest.py times a fixed pure-Python reference workload immediately before each benchmark and the threshold is scaled by it. This removed most of the run-to-run swing in the means (Linux CV fell from 10 to 35% to 1 to 12%).
• Linux only. macOS runners drift in speed within a job and stall for 100 to 200 ms mid-round, and Windows shows the same bimodal hardware classes as Linux with more residual noise. Both would need margins of 2x or more to avoid false positives, which defeats the purpose, so the gate runs only on ubuntu-latest. The other OSes still run the benchmark tests functionally under --benchmark-disable.
• Explicit opt-in. The check is enforced only when SPP_BENCHMARK_GATE is set, which the CI gate step does. A plain pytest tests/benchmark/ locally just reports timings, so developers on other hardware are not gated against x86-runner thresholds.
• Calibration. Thresholds come from the normalized means of six CI runs per Python version, set at about 1.5x the average and never below 1.2x the largest value observed. A single "default" per test covers all versions; a version-specific key can be added if one ever needs it. At 1.25x margins two of three validation runs produced one false positive each (a runner-class shift and a burst of slow rounds), which is why the margin is 1.5x.

CI changes

• The coverage-instrumented test step passes --benchmark-disable, since coverage tracing distorts timings.
• A separate uninstrumented "Benchmark regression gate" step runs tests/benchmark/ on ubuntu-latest with SPP_BENCHMARK_GATE=1.
• Each Linux job writes the per-test normalized mean and speed factor to the step summary and uploads benchmark.json as a benchmark-py<version>-attempt<n> artifact, so recalibration does not require log scraping.
• codecov.yml ignores tests/benchmark/, which by design never runs instrumented.

Benchmark changes

• The five bit-reading benchmarks run 1000 pedantic rounds instead of 3, so a stall of a few milliseconds moves the mean by a few percent rather than twenty.
• The two packet-parsing benchmarks let pytest-benchmark calibrate the round count instead of a fixed 20.

Documentation

• developers.md gains a short ### Benchmarks section describing the gate and how to recalibrate.
• benchmarking.md gains a ### Regression Guard note explaining the fixed bug the CTIM benchmark guards against and pointing at the developer docs.
• CHANGELOG entry under [Unreleased].

Verification

• All 9 benchmarks pass locally with the gate off, on, and under --benchmark-disable; forcing an absurd threshold fails the test.
• CI run #549 (same thresholds) passed on all five Linux jobs; the squashed commit re-runs the full matrix.
• make html builds with no new warnings.

Closes #147
Closes #287

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.72%. Comparing base (71bc3d0) to head (4581bcb).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #286      +/-   ##
==========================================
- Coverage   94.86%   94.72%   -0.14%     
==========================================
  Files          50       47       -3     
  Lines        4321     4209     -112     
==========================================
- Hits         4099     3987     -112     
  Misses        222      222              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟢 Approval recommended

The documentation changes are reviewed; the only finding is a non-blocking nit about an issue reference.

Pull request overview

Documents the CTIM XTCE parsing benchmark as a manual performance regression guard.

Changes:

• Adds CTIM benchmark context, history, budget, and comparison workflow.
• Documents benchmark guidance for contributors.
• Records the update in the unreleased changelog.

File summaries

File	Summary
docs/source/developers.md	Adds benchmark guidance and workflow.
docs/source/benchmarking.md	Documents CTIM’s regression-guard purpose.
CHANGELOG.md	Records the documentation update.

Review details

Suppressed comments (1)

docs/source/benchmarking.md:293

• The PR description says follow-up issue #285 tracks designing this CI gate, but repository issue #285 is actually about invalid CITATION.cff metadata and is unrelated. Please correct the issue reference in the PR description (or omit it) so the rationale for leaving enforcement manual remains traceable.

budget is currently a manual step. If you are touching `space_packet_parser/xtce/`, run the
comparison above before opening a PR.

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[medley56]

Ready to merge