Skip to content

Keep a fixture's cleanup failure from aborting its test binary while a test is already failing - #1353

Merged
vjovanov merged 2 commits into
agent-grounds:mainfrom
vjovanov:fix/issue-1226
Nov 11, 2026
Merged

vjovanov merged 2 commits into
agent-grounds:mainfrom
vjovanov:fix/issue-1226

Conversation

@vjovanov

@vjovanov vjovanov commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1226

A test that fails while its fixture's tree cannot be removed is now reported as one failed test, and the rest of its binary still runs. Before this change, the fixture's Drop panicked a second time during unwinding. That aborted the whole test binary, and on windows-latest it did so silently with 0xC0000409, which is how rules_contract lost 205 tests in #1226.

Before, on main, the new pin holds a real Configured (the fixture of the report) whose tree cannot be removed, and fails the test:

thread '<unnamed>' panicked at crates/grund-cli/tests/rules_contract/offer_scope_cleanup.rs:45:13:
the test's own assertion failed while its fixture was alive

thread '<unnamed>' panicked at crates/grund-cli/tests/rules_contract/offer_scope.rs:224:37:
remove isolated configured fixture: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
...
panic in a destructor during cleanup
thread caused non-unwinding panic. aborting.
binary exit=134 listed=11 reported=1

Triage's reproducer injects the same failure into the whole rules_contract binary. On main it dies of SIGABRT with 66 of its 112 tests unrun.

After, the same reproducer on this branch:

OK (defect gone): the injected failures stayed 6 reported test failure(s),
and all 114 tests in the binary ran.

What changed

  • The cause. About two dozen fixture Drops under crates/*/tests called fs::remove_dir_all(..).unwrap() or .expect(..) with no std::thread::panicking() guard. If a test failed while its tree could not be removed, the Drop panicked during unwinding, and Rust aborts the process on that ("panic in a destructor during cleanup"). On Windows MSVC that abort is a fast-fail, exit 0xC0000409 (STATUS_STACK_BUFFER_OVERRUN), and libtest never flushes the output it captured. So the CI log carried an exit code and nothing else, and every test after it in the shard went unrun.

  • The rule. §AR-ci-fixture-cleanup is a new page under docs/architecture/, with a row in the architecture README. It says:

    • while unwinding, a fixture writes its removal error to stderr, naming the tree, and raises nothing (§AR-ci-fixture-cleanup.1);
    • on the passing path, cleanup stays strict: a removal failure still fails the test, with no ignoring, retrying or sleeping, as §AR-ci.10.4 asks (§AR-ci-fixture-cleanup.2);
    • one helper makes that decision, and a fixture with a passing-path rule of its own (support/case_copy.rs, which tolerates NotFound) guards inline (§AR-ci-fixture-cleanup.3);
    • what holds it: a source scan and a runtime pin (§AR-ci-fixture-cleanup.4).

    It is a page of its own rather than §AR-ci.10.6 because that number is already taken on main, and fissile refuses further edits to AR-ci.md (promotion limit). That is the same remedy the other AR-ci-* pages took. AR-ci.md is untouched.

  • The helper. remove_fixture_tree(path, what) in the new crates/grund-cli/tests/support/fixture_cleanup.rs. It removes the tree and, if that fails, writes <what> <path>: <err> to stderr while unwinding and panics with the same message otherwise.

  • Every copy. All 24 tree-removing fixture Drops now call the helper, each keeping its old message. Triage counted 18, and 6 more arrived from main since. 23 are in crates/grund-cli/tests, and binaries that do not include support reach the helper with #[path = ...] mod fixture_cleanup;. One is in grund-lsp (two_names_temp_roots.rs), and it reaches the same helper across crates through #[path = "../../grund-cli/tests/support/fixture_cleanup.rs"] instead of a second copy. value_home_unsearchable still resets its values directory to 0755 before removing the tree, but that reset no longer panics. The three copies that were already guarded (case_copy.rs, agreement_host.rs, agreement_editor_worker.rs) are left as they were.

  • Configured's root. The fixture of the report named its root from the pid and the clock, and adopted it with create_dir_all, so two roots named in one clock tick could share a tree. It now names its root grund-1001-configured-<pid>-<serial> from a process-wide counter (§AR-ci.10.1). It claims the root with an exclusive fs::create_dir before writing into it (§AR-ci.10.2), so a taken name panics before any Configured exists and cannot remove another test's tree.

No behaviour of the shipped grund changes: only test fixtures, test support and one architecture page.

How it was verified

  • The mechanism, at run time. offer_scope::cleanup in the rules_contract binary (#[cfg(unix)]) holds a real Configured on a thread of its own. It makes a directory inside the root read-only, so the removal fails without touching anything outside it.
    • a_failing_test_whose_configured_cannot_be_removed_reports_its_own_panic panics and asserts that the join returns the test's own panic and that the tree was left in place. On main it aborts the binary, as shown above.
    • a_passing_test_whose_configured_cannot_be_removed_still_fails is the control: with no first panic, the Drop must still fail with Permission denied. It catches a fix that silences the passing path.
    • Where read-only does not bind (running as root), both print skipped: … and return.
  • Coverage, from source. tests/integration/test_fixture_cleanup.py reads every impl Drop under crates/ that removes a tree. It requires each one to raise nothing, call the helper, or test std::thread::panicking() before its first raise, and holds the helper itself to both halves. On main it fails, naming all 24 sites. Ten ShapeTest cases show that it tells the shapes apart.
  • End to end. Triage's reproducer drives the real rules_contract binary on Linux with injected failures and unremovable trees. It went from exit 1 (SIGABRT, 66 of 112 tests unrun) to exit 0 (all 114 ran).
  • The gate. pre-commit run --all-files exited 0 on dc1c09ef04 (22m37s), with fmt, build with -Dwarnings, the cargo and Python tests, grund check --full, grund fmt, grund init --check, lychee and fissile.

Not established

  • What failed first in the CI run, and why that removal failed, were never observed. The failing log names no test. Re-dealing shard 2/4 of accbb882d7 puts four rules_contract tests in flight at the abort, and three of them build a Configured. The clock-named, adopted root fixed above is the likeliest trigger. A handle held by another process on the runner would also do it.
  • The Windows exit code rests on cited sources (Rust's abort_internal is a __fastfail with FAST_FAIL_FATAL_APP_EXIT), not on a Windows run. The mechanism was reproduced on Linux only.

Found and deliberately not fixed

  • R1-01 (minor). The coverage scan accepts two shapes that would still abort while unwinding: a raise reached through a method the Drop calls before the helper, and a panicking() call that gates nothing. No site in the tree has either shape today. An independent scan of every impl Drop under crates/ found no tree-removing Drop that still raises unguarded.
  • R1-02 (nit). value_home_unsearchable now ignores a failed mode reset (let _ = fs::set_permissions(..)). This is harmless: a reset that fails leaves the tree unremovable, and the strict removal that follows still fails the test. The one case it now passes silently is a missing values directory, where no tree is left behind anyway.
  • Adjacent, out of scope:
    • Reset (crates/grund-cli/src/cli_watch_observer.rs:160) and Harness (tests/support/watch_observer.rs:286) panic in Drop, through a mutex lock().unwrap(), but remove no tree.
    • rules_contract leaves its support::scratch repositories under target/rules-contract-work.
    • Fixtures other than Configured that still name their roots from the clock.
AI workflow: `rhei`, 14 agent invocations across 1 model; 6 tasks completed, 10 in progress.
  1. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket supervising (visit 1) — cld, anthropic/claude-opus-5-5 — 1.0s — 0 in / 0 out
  2. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket supervising (visit 1) — cld, anthropic/claude-opus-5-5 — 2.0s — 0 in / 0 out
  3. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket supervising (visit 1) — cld, anthropic/claude-opus-5-5 — 2m05s — 1.2M in / 11.8k out
  4. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket.triage assessing (visit 1) — cld, anthropic/claude-opus-5-5 — 30m00s — 17.2M in / 119.1k out
  5. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket.triage assessing (visit 1) — cld, anthropic/claude-opus-5-5 — 22m47s — 3.6M in / 24.2k out
  6. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket.triage reproducing (visit 1) — cld, anthropic/claude-opus-5-5 — 9m16s — 3.7M in / 50.1k out
  7. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket supervising (visit 2) — cld, anthropic/claude-opus-5-5 — 3m51s — 2.7M in / 21.6k out
  8. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket.specify specify — cld, anthropic/claude-opus-5-5 — 72m31s — 13.5M in / 75.1k out
  9. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket supervising (visit 3) — cld, anthropic/claude-opus-5-5 — 2m50s — 701.3k in / 4.9k out
  10. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket.implement implement — cld, anthropic/claude-opus-5-5 — 22m06s — 2.7M in / 17.0k out
  11. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket supervising (visit 4) — cld, anthropic/claude-opus-5-5 — 35.0s — 407.5k in / 3.6k out
  12. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket.review-1 review — cld, anthropic/claude-opus-5-5 — 2m11s — 1.4M in / 10.2k out
  13. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket supervising (visit 5) — cld, anthropic/claude-opus-5-5 — 40.0s — 518.3k in / 2.9k out
  14. github-issues-agent-grounds-grund-1226-implement-f358abf1.ticket supervising (visit 6) — cld, anthropic/claude-opus-5-5 — 28.0s — 313.9k in / 2.9k out
Accounting Value
cost $23.70
total tokens 48.2M
input tokens (incl. cache) 47.8M
input cache read 46.3M
input cache write 1.5M
output tokens (incl. cache) 343.5k
output cache read -
output cache write -
coverage Complete

…, and pin it failing on the fixture of the report and on every copy, red until agent-grounds#1226 (§AR-ci-fixture-cleanup)

A fixture Drop that runs `fs::remove_dir_all(..).unwrap()` or `.expect(..)` raises its removal failure even while the test holding it is already failing. Rust aborts the whole process on a panic during unwinding, so every test the binary has not yet reported goes unrun. On windows-latest the abort is the silent exit 0xC0000409 of agent-grounds#1226, which `rules_contract`'s `Configured` produced.

§AR-ci-fixture-cleanup is its own page under the architecture index, beside the other `AR-ci-*` pages, because AR-ci.md is over its fissile budget. It says a fixture writes the removal error to stderr while unwinding and raises nothing, that the passing path stays as strict as §AR-ci.10.4 asks, and that one helper, `remove_fixture_tree`, makes the decision.

Two pins, both red until the fix:

- `offer_scope::cleanup::a_failing_test_whose_configured_cannot_be_removed_reports_its_own_panic` holds a real `Configured` on its own thread, refuses its removal with a read-only directory inside the root, and fails the test. Today the whole `rules_contract` binary dies of SIGABRT, "panic in a destructor during cleanup". It is ignored until the fix, and `implement` removes the marker. Its control, `a_passing_test_whose_configured_cannot_be_removed_still_fails`, passes today and must keep passing.
- `tests/integration/test_fixture_cleanup.py` holds every tree-removing `impl Drop` under `crates/` to three shapes: it raises nothing, it calls the helper, or it tests `std::thread::panicking()` before it raises. It names the 24 that raise today. It lands as `expectedFailure`, which the fix takes off.
…hich writes a removal failure while unwinding and raises it otherwise, and claim Configured's root by pid and serial, so a failing test no longer aborts its binary (§AR-ci-fixture-cleanup)

Closes the 24 copies of the panicking fixture Drop under crates/, the one
that aborted rules_contract on windows-latest (agent-grounds#1226), and takes the red
markers off both pins.
@vjovanov vjovanov changed the title Specify that a fixture's cleanup failure never aborts its test binary, and pin it failing on the fixture of the report and on every copy, red until #1226 (§AR-ci-fixture-cleanup) Keep a fixture's cleanup failure from aborting its test binary while a test is already failing Nov 11, 2026
@vjovanov
vjovanov marked this pull request as ready for review November 11, 2026 00:04
@vjovanov
vjovanov requested a review from kimeta November 11, 2026 00:04
@vjovanov
vjovanov added this pull request to the merge queue Nov 11, 2026
Merged via the queue into agent-grounds:main with commit 0419145 Nov 11, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rules_contract aborts intermittently on windows-latest: process didn't exit successfully (exit code: 0xc0000409, STATUS_STACK_BUFFER_OVERRUN)

1 participant