ui-server: cancellation-cleanup control can time out before its deliberate suspension #2958

Out of scope

Changing production cancellation, widening the real suite's deadlines, dropping the following-case or late-finally isolation checks, or counting passing reruns as a repair.

Context

Main CI unit (1) on c670449d3bf1b0639e50ce554d728961f4313add failed the after-Git control in run 40238122480, job 121304128310. The job log was read during 2026-10-26 triage and corrects the original diagnosis:

• The selected cancellation case timed out during real git init (generated fixture line 428): exitCode was null, and the runner reported the 250 ms timeout after 638.52 ms. It had not reached the injected isolationProbe checkpoint.
• The following case failed immediately (0.14 ms) at CANCELLATION_SETUP_REACHED: expected after, received undefined.
• The late-finally case failed immediately (0.37 ms) because isolationProbe.resume was undefined.
• The outer assertion failed at line 72 while requiring the following case to pass. The siblings did not independently exceed their 250 ms deadlines; they lacked setup from the first case.

The generated control starts the 250 ms per-test deadline before fixture/Git preparation and before its deliberate suspension (const child = Bun.spawn, packages/ui-server/tests/maintenance-cancellation-cleanup.test.ts:63-66). The checkpoint is injected before createUiDb, then the outer control demands one expected timeout and both isolation checks passing (expect(code, output), packages/ui-server/tests/maintenance-cancellation-cleanup.test.ts:69-77). The original cancellation setup performs real Git initialization before reaching that checkpoint (for (const timing, packages/ui-server/tests/maintenance-startup-view.test.ts:424-430). The log establishes the premature setup timeout; it does not establish why Git took that long on the CI host.

Scope

Attribute and reproduce the wrong-phase timeout, then make the control prove cancellation only after its intended setup checkpoint is reached. Preserve the same-process following-case cleanup and late-finally ownership checks: running the induced case in a separate process alone would lose those guarantees. Select the smallest correction from the source-bound reproduction, without changing product deadlines.

Acceptance criteria

• A bounded natural or controlled run demonstrates timeout before the intended checkpoint, with source/runtime/load and the actual failed setup/sibling assertions recorded. Distinguish controlled delay from natural recurrence.
• The repaired control proves its deliberate suspension was reached before the induced timeout; slow preparation cannot masquerade as that timeout.
• Both following-case real-Git/global restoration and late-finally next-owner preservation checks execute in the same child process after the deliberate timeout and pass.
• A causal negative control breaks cleanup/ownership and fails the corresponding assertion, followed by a restored pass. Keep the one-expected-failure envelope and normal production/suite deadlines.

Verification

bun run test packages/ui-server/tests/maintenance-cancellation-cleanup.test.ts, a bounded source-bound wrong-phase reproduction, the targeted cleanup/ownership mutation and bun run check:pr --base origin/main.

At inspected main c670449d3bf1b0639e50ce554d728961f4313add, frozen dependencies and Bun 1.4.3, triage ran the unchanged focused command once normally (2 pass / 0 fail, 1.138 s runner time) and once pinned to CPU 0 with one competing CPU loop (2 pass / 0 fail, 1.56 s). Both outer during/after cases ran with offline guards. These are bounded non-reproductions on a different Bun version from CI's 1.4.2, not a repair or CI qualification. These bounded non-reproductions preceded the controlled wrong-phase receipt below; they do not establish the historical CI scheduling cause.

Contract impact

None selected. This issue concerns the cancellation proof harness; product behavior and machine surfaces remain unchanged.

2026-10-26 triage: corrected the asserted sibling-timeout cause against the actual CI log, supplied scope/exclusions/verification/contract impact, and retained the precise reproduction gap. The original triage left this work at p2; the 2026-10-28 release-blocking recurrence below supersedes that scheduling disposition. Standalone: #736 is already at its 100-direct-child limit; browser-only #2753 does not own this process-control outcome.

Controlled wrong-phase reproduction — 2026-10-26

Source: main c670449d3bf1b0639e50ce554d728961f4313add; Bun 1.4.3; frozen dependencies; ordinary guarded wrapper. Repository source stayed unchanged. A temporary copy of maintenance-cancellation-cleanup.test.ts used the original checkout as ROOT and the original maintenance-startup-view.test.ts as SOURCE. The sole behavioral diagnostic was to insert await new Promise<void>(resolve => setTimeout(resolve, 450)); immediately before the generated cancellation setup's const init = Bun.spawnSync([realGit, "-C", brain, "init", "-q"]);. This makes setup exceed the existing 250 ms deadline before the deliberate isolationProbe suspension, without changing the original assertions or deadlines.

Command: bun run test /tmp/brain-triage-pass2/2958-slow-setup.test.ts.

Observed: exit 1, 0 pass / 2 fail / 12 assertions, 1.166 s test phase. Both during/after outer controls failed on their original requirement that the following real-Git case pass. Each generated child reported 0 pass / 3 fail: the cancellation case timed out at about 251 ms; the following case failed CANCELLATION_SETUP_REACHED with undefined timing; the late-finally case failed because isolationProbe.resume was not a function. Thus the original one-expected-timeout envelope accepts a timeout string from the wrong phase but then fails its missing-setup siblings. This is a deterministic source-bound harness reproduction, not a reproduction or diagnosis of CI's slow Git operation.

The unchanged original command, bun run test ./packages/ui-server/tests/maintenance-cancellation-cleanup.test.ts, then passed 2/2, 18 assertions, 1.147 s test phase, exit 0. No product/test fix, deadline change, browser guard bypass or natural-load equivalence is claimed. An earlier dependency-resolution failure was excluded from behavioral evidence; the results above ran after frozen installation succeeded.

Diagnostic source SHA-256: d98e8d236b673f8f3741e2fc94243c17838bb4a49f3fad9196098341bcebd8e5. Failed log SHA-256: e47656f283c914068dafb78e9a96d3ff6a4b234f01ba6303fc5398f3ae65fcd0.

2026-10-26 readiness reconciliation: the controlled wrong-phase reproduction satisfies the missing harness prerequisite. Remove needs: repro and add agent-ready for the correction scope (subsequently promoted to p1/0.44.0 below). Preserve the historical CI failure, unknown host-delay cause and every same-process cleanup/ownership acceptance criterion. Implementation and its causal cleanup mutation remain required; no repair or completed issue is claimed.

Release-blocking hosted recurrence — 2026-10-28

Blocks #3030. This existing issue is now committed to 0.44.0, priority: p1, because the same harness failure blocks release preparation. It remains standalone; no epic membership is added.

PR #3083 candidate run 40676593252, unit (4), job 122776997145 failed at head f790b2bfb61dadd4e34c736c035d94f910cdc28d, base 47c55df72153bf742181186cad6d6f75b889d170. The job resource receipt identifies tested merge checkout 8647384c39a41ca7b30aa91d04e6e3615def866c, tree b6118cbff7e034ed3c57f30345aaf76bbabfde10, with those two parents. The retained raw job log has SHA-256 6664aadb484b1167e128552bfa0dd2d32c4b76127b2588b3c5fe650ee5ca0838.

Observed on Bun 1.4.2:

• The outer during control failed in 808.42 ms at maintenance-cancellation-cleanup.test.ts:72, requiring the following cancellation case to pass.
• The generated cancellation case timed out after 250 ms (reported duration 410.48 ms). At its real Git initialization, line 428, init.exitCode was null rather than 0; the log also reports one dangling process killed.
• The following case failed CANCELLATION_SETUP_REACHED (expected during, received undefined); the late-finally case failed because isolationProbe.resume was undefined. These are missing-setup failures, not independent sibling deadline overruns.
• The outer after control passed in 633.80 ms. The failed worker ended with 1336 pass / 1 fail across 122 files; that is a failed candidate, not an unsupported environment or passing qualification.

Both relevant test files are byte-identical between the supplied base and head: maintenance-cancellation-cleanup.test.ts blob a1caa8eff11b6e7a45dc24dcfde7bbce94a72fbd, maintenance-startup-view.test.ts blob fa2d206c1b1c9fdca0f03e3c1e24052a9f3c9336. The PR changes app-wiring coverage and its coverage document. No causal link from that cleanup change to this failure has been established. The evidence identifies the wrong timeout phase, not the reason real Git was delayed or terminated.

Source diagnosis and reviewed narrow correction

Citations verified at the head above:

• afterEach and fixture, packages/ui-server/tests/maintenance-startup-view.test.ts:19-27: reverse-order synchronous cleanup removes fixtures.
• for (const timing, packages/ui-server/tests/maintenance-startup-view.test.ts:424-453: real Git initialization precedes shim/spy installation and registration of the idempotent restore callback. Restore is registered last so it runs before fixture deletion.
• try, packages/ui-server/tests/maintenance-startup-view.test.ts:454-473: original body and finally; the control injects its suspension before database creation, so it does not exercise actual native cancellation at that checkpoint.
• for (const timing, packages/ui-server/tests/maintenance-cancellation-cleanup.test.ts:9-29: copied source and checkpoint injection; the probe is only assigned after real Git setup.
• test("following cancellation, packages/ui-server/tests/maintenance-cancellation-cleanup.test.ts:36-59: same-child shim deletion, real Git/global restoration, and a later finally preserving a new lookup/worker owner.
• const child, packages/ui-server/tests/maintenance-cancellation-cleanup.test.ts:63-77: 250 ms CLI timeout applies to every selected child test, with a separate 10-second child cap and the expected 2-pass/1-fail envelope.
• const tests, scripts/test.ts:203-213: guarded invocation joins process exit and both output streams before reporting completion. Preserve wrapper/offline/ownership behavior.

Reviewed harness-only correction, prepared in draft PR #3088: stage the selected original cancellation callback through its injected suspension in a separately bounded setup hook, with an explicit ready acknowledgement carrying timing, shim identity and the resume handle. Only after that acknowledgement should the selected test await the still-pending callback under an explicit 250 ms Bun test timeout. Keep the original callback suspended through Bun's timeout and afterEach, then run both existing sibling checks in the same child and release it from the late-finally case. Register cleanup before staging; setup failures must remain failures and cannot count as the induced timeout. Scope the staging hook to the induced case so it cannot reinstall globals before the siblings.

Use the existing normal setup/test budget for preparation; keep the induced 250 ms timeout and existing 10-second outer child cap. Give ordinary sibling checks their normal budget instead of the induced case's CLI-wide 250 ms budget. This separates phases rather than increasing the induced timeout. The Bun hook lifecycle and late callback unwinding have now been proven locally on CI-pinned Bun 1.4.2, as recorded below; independent review and hosted candidate proof remain outstanding. Do not substitute an explicit throw, Promise.race rejection or fake-clock tick for the genuine Bun timeout: the regression depends on Bun starting afterEach while the original callback's finally remains suspended.

Additional acceptance/verification requirements for this recurrence:

• A controlled setup delay longer than 250 ms reaches the explicit checkpoint under bounded setup and then produces exactly the intended Bun timeout; both existing siblings pass. A missing checkpoint or setup failure fails distinctly.
• Both timing arms preserve restoration-before-deletion, real Git execution, original lookup/PATH/worker identity and next-owner preservation after late finally; no assertion or original native cancellation deadline is removed.
• Preserve reverse cleanup order and idempotent restoration. Mutation receipts separately demonstrate that removing early restoration fails the following-case oracle and removing idempotence fails the late-finally ownership oracle, followed by restored passes.
• Retain the failed hosted receipt and run focused guarded proof plus required preflight/selected hosted proof only after reviewed implementation; no unchanged retry substitutes for correction.

This update is source/log research only: no tests, mutation, setup-hook experiment, CI retry or implementation were performed for this recurrence. Prior 2026-10-26 receipts above remain historical evidence. Binding guidance: docs/process/testing.md (failing first for the reason claimed), docs/process/test-waits.md (completion barriers and preserved failure caps), and docs/decisions/ci-utility.md (invocation ownership and failed selected proof). Contract impact remains none: test harness only, no product cancellation or machine-surface change.

2026-10-28 reconciliation: matched the new hosted failure to this existing issue, verified current citations and unchanged source, added the bounded phase-separation proposal and verification limits, and promoted scheduling to p1/0.44.0 as a blocker of #3030. No readiness label was added and no implementation is authorized by this research update.

Draft correction and local proof — 2026-10-28

Draft PR #3088 implements the reviewed harness-only correction. Head 04b4be3c4fd6ebb7a346b81ddc2a8a28dbf97074, base 47c55df72153bf742181186cad6d6f75b889d170, tree 300ce9d95bc318f061fbd52e9037e7255c6bc848; issue and PR retain milestone 0.44.0. This is partial delivery under Refs #2958, not closure or release qualification.

At this head, packages/ui-server/tests/maintenance-cancellation-cleanup.test.ts is the only changed file. Original native cancellation and runtime source are unchanged. The setup matrix and exact replacement anchors are at lines 9–36; scoped hook and selected 250 ms test at lines 59–105; following-case cleanup and late-finally assertions at lines 110–137; guarded child and one-timeout envelope at lines 140–158. The hook budget is 30 seconds; a separate 5-second acknowledgement watchdog diagnoses missing setup and restores/releases it within the unchanged 10-second outer child cap. Ordinary sibling tests use the wrapper's normal 30-second budget.

Bun 1.4.2+744846f84, frozen copyfile dependencies (59,516 regular files, all link count 1). Lock SHA-256 remains 5bbd5f6651f111db3fcfabeda512d44a21163cba54c58993f8878012d548cc14; actual SDK-relative native 0.3.293 binary remains 8968405e26db478af44eabc4635ab5ca557057b702a54460a59c13e1b253e978.

Local results:

• Original harness plus controlled 450 ms preparation delay fails both outer controls at the missing-checkpoint following-case requirement (exit 1). This is controlled phase-ordering evidence, not attribution of the historical real-Git delay.
• CI-pinned Bun lifecycle control completes 450 ms setup, then produces one genuine 250 ms timeout and two passing following cases; callback finally remains suspended through afterEach.
• Corrected committed harness passes all four ordinary/delayed timing combinations (64 assertions, exit 0). Every recorded child has exactly two passes and one expected genuine Bun timeout.
• Unchanged source cancellation cases plus three nearby siblings pass 5/5 (24 assertions), preserving AbortError, admission/storage release and the original less-than-600 ms reap assertion.
• Removing early restoration fails following-case real-Git/worker identity. Reversing restore order fails the explicit before-deletion observation. Removing idempotence fails the late-finally exactly-once assertion (2 versus 1); the restored harness passes. All variants cover both timing arms and ordinary/delayed setup.
• Bun's own mockRestore is insensitive to a repeated restoration on this runtime: the first identity-only idempotence mutant passed. The copied harness therefore also counts entry into the original restoration body after its guard. This makes missing idempotence fail without replacing real spy-identity assertions or changing product source.
• Missing acknowledgement and injected callback failure produce distinct setup diagnostics and no induced-timeout claim; the following real-Git/global/before-deletion cleanup oracle passes. No unhandled rejection appears in the retained child logs.
• Strict typecheck and default bun run check:pr --base origin/main pass, including lint/leakage, environment documentation, changeset policy, citation checks and 209 affected fast invariant tests (1,666 assertions). No consumer-visible change requires a changeset.

Exact mutant sources/diffs, generated children, runtime and log hashes, command exits and the original failed hosted log identity are retained for parent review; selected receipt hashes are in the PR body. Independent review and parent-owned selected hosted confirmation on the actual candidate are still required. No reservation, ready transition, CI retry, full release gate, merge, publication or deployment was performed. The original hosted result remains FAILED and no causal connection to the unrelated app-wiring correction is claimed. #3030 remains blocked by this issue.

2026-10-28 implementation reconciliation: replaced the pending proposal disposition with the reviewed draft correction, folded local causal and setup-failure evidence into this body, and retained all historical failed proof and outstanding hosted/review limits. The issue remains open; local proof alone does not complete all acceptance and release requirements.