Repository navigation
Description
scripts/unrooted_local_shape.py --no-raise-vs <base> returned green on a tree where --check reported REGRESSION: 563 findings exceeds baseline 561. The variant whose entire purpose is catching a regression against the merge base did not catch one.
Observed
Merge train 220 carried #10668, which takes crates/perry-ext-http/src/response_headers.rs from 1 to 6 unrooted-local shapes.
| invocation | result |
|---|---|
unrooted_local_shape.py --check on the train |
FAIL — REGRESSION: 563 findings exceeds baseline 561 |
unrooted_local_shape.py --no-raise-vs "$BASE_SHA" on the same train, same run_lint_gates.sh run |
ok |
unrooted_local_shape.py --check on origin/main |
ok — improved: 559 < baseline 561 |
So main is at 559, the train is at 563, and the vs-base check passed anyway. Both ran in the same run_lint_gates.sh invocation, seconds apart, against the same worktree.
Why this is the dangerous shape
CLAUDE.md lists four ways a gate can be unable to fail, and this is the fourth and worst: the job is genuinely green. An agent or a lane that validates a branch with the vs-base form — which is the natural choice for "did my change make this worse?" — gets a pass on a tree that the absolute check rejects. Nothing in the output suggests the comparison did not happen.
I do not know the mechanism, and that is the point of filing rather than patching: candidates are $BASE_SHA resolving to something other than the intended merge base (empty, or the branch tip itself), the comparison being made per-file against a truncated listing rather than on the total, or the ratchet's improved state on the base side masking a rise. The first is worth ruling out first, since run_lint_gates.sh passes "$BASE_SHA" through from the environment and an unset value would make the check vacuous while still exiting 0.
Suggested fixes, in order
- Make an unresolvable or empty base a hard error, not a pass.
raw_handle_debt.py'sgit_showalready documents exactly this reasoning — "a merge base the runner never fetched otherwise reports every file as absent, which reads as 'the base recorded nothing' — a comparison that did not happen, reported as a pass. It must be a RED build instead." The same guarantee belongs here. - Have
--no-raise-vsprint what it compared — the resolved base SHA and both totals — so a reader can tell a real comparison from a vacuous one. - Add a self-test that plants a regression and asserts
--no-raise-vsfails on it.--self-testcurrently passes while the live check is fooled, which means the self-test does not cover the mode that matters. Sabotage-testing the instrument is the standing pattern here (quarantine_catches_a_planted_stale_from_space_deref).
Until then the absolute --check is the only trustworthy form, and that is what this lane now runs.
Found while validating merge train 220; #10668 was removed from the train on the strength of the absolute check.
Activity
fix(tooling): close two blind spots in unrooted_local_shape.py