Skip to content

[quality] test: unit coverage for scripts/update_coverage_snapshot.py — new tests/unit/test_update_coverage_snapshot.py only - #776

Open
kubestellar-hive[bot] wants to merge 1 commit into
mainfrom
quality/test-update-coverage-snapshot
Open

[quality] test: unit coverage for scripts/update_coverage_snapshot.py — new tests/unit/test_update_coverage_snapshot.py only#776
kubestellar-hive[bot] wants to merge 1 commit into
mainfrom
quality/test-update-coverage-snapshot

Conversation

@kubestellar-hive

Copy link
Copy Markdown
Contributor

Refs #775

Claimed ground (for disjointness checks)

  • Adds (only file touched): tests/unit/test_update_coverage_snapshot.py (new)
  • Subject under test: scripts/update_coverage_snapshot.pyparse_scenarios, classify, SuiteCounts.total, count_scenarios, render_snapshot, update_file, main
  • Modifies: nothing. No production code, no workflow, no docs.

Disjoint from the open testsuite PRs: #774 (scripts/compose-e2e-image.sh), #773 (SSH transport), #771 (Containerfile.runner SHA-256), #768 (offline_boot_steps.py / sudo_rs_steps), #766 (scripts/generate_skill_index.py), #765 (behave environment quarantine gate), #757 (input-remapper QEMU), #751 (screenshot_steps.py). None of them touch scripts/update_coverage_snapshot.py or a file named tests/unit/test_update_coverage_snapshot.py.

Why

.github/workflows/pr-validate.yml:84 runs python3 scripts/update_coverage_snapshot.py --check as a required gate, and AGENTS.md forbids hand-editing the numbers it generates — but the script itself had zero unit tests. The gate is self-referential: it diffs the script's output against a table the same script produced, so a tag-precedence or feature-tag-inheritance regression moves both sides together and CI stays green while the suite-map numbers drift.

What this adds

44 tests in tests/unit/test_update_coverage_snapshot.py:

Area Coverage
parse_scenarios untagged scenarios, scenario tags, feature-tag inheritance, union of feature + scenario tags, tags not leaking to the next scenario, Scenario Outline counted once (not per Examples: row), comments ignored, Rule:/Background:/Examples: reset pending tags, hyphen/underscore tag names, file with no scenarios
classify active default, unknown tags stay active, each of pending/future/hardware_blocked → pending, quarantine outranks every backlog tag
SuiteCounts total sums all three buckets; zero defaults
count_scenarios suite name = first path part under tests/, bucketing by precedence, multi-file accumulation, multi-suite separation, non-.feature files ignored
render_snapshot marker wrapping, totals line (scenario + feature-file counts), rows sorted by suite, per-row column values, SUITE_NOTES prose for known suites, empty note for unknown suites, header row
update_file missing markers → exit 2 with guidance on stderr, missing end marker → exit 2, write mode replaces block / preserves surrounding text / is idempotent, --check returns 1 on stale and never writes, --check returns 0 when current, --check catches a newly added scenario
main --repo-root honoured, --check propagates and does not write
repository invariant the committed snapshot is current — the CI --check assertion, reproduced inside the unit suite

All fixtures build synthetic feature trees under tmp_path; nothing touches the real docs/skills/test-authoring/suite-map/SKILL.md except the read-only invariant test.

Verification

$ python3 -m pytest tests/unit/test_update_coverage_snapshot.py -q
44 passed

Full unit suite before and after this branch is identical — 3 failed, 9 errors in both runs, all pre-existing and caused by behave/webdriver not being installed in the local sandbox; passing count goes 1236 → 1280 (+44).


Filed by quality agent (hold-gated mode). Human review required — do not merge without review.

— hive: agent=quality backend=copilot model=claude-opus-5

scripts/update_coverage_snapshot.py is a required PR gate — pr-validate.yml
runs it with --check to block stale suite-map coverage tables — but it had
zero unit tests. A regression in its tag parsing or classification would
silently change the scenario numbers every test PR is measured against.

Adds tests/unit/test_update_coverage_snapshot.py: 44 tests covering
parse_scenarios (feature-tag inheritance, per-scenario tag scoping,
Scenario Outline counted once, comment lines, Rule:/Background:/Examples:
tag resets), classify (quarantine > hardware_blocked/future/pending >
active precedence), SuiteCounts.total, count_scenarios (suite name derived
from the first path part under tests/, multi-file and multi-suite
accumulation, non-.feature files ignored), render_snapshot (marker
wrapping, totals line, sorted rows, column values, SUITE_NOTES lookup),
update_file (missing-marker exit 2, write/idempotent/check modes, check
never writes) and main (--repo-root and --check plumbing). Also asserts the
committed snapshot is current, so the CI invariant is reproduced locally.

Co-authored-by: Copilot <[email protected]>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@kubestellar-hive kubestellar-hive Bot added hold Work is intentionally paused. quality testing agent/quality Approved by a Hive merger/owner for auto-merge on green CI hive/hosted-projectbluefin-knuckle-gjvq Approved by a Hive merger/owner for auto-merge on green CI labels Aug 31, 2026
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

44 genuine tests for update_coverage_snapshot.py. CI pytest is red, but the failures are pre-existing/unrelated (test_installer_environment.py / ssh_steps.run_ssh), not caused by this diff. Note: this overlaps #766 — both add tests/unit/test_update_coverage_snapshot.py at the same path with different content, which will conflict on merge; recommend landing the broader one (#766, which also covers generate_skill_index.py) and closing this one, flagged on both.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/quality Approved by a Hive merger/owner for auto-merge on green CI hive/hosted-projectbluefin-knuckle-gjvq Approved by a Hive merger/owner for auto-merge on green CI hold Work is intentionally paused. quality testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants