Repository navigation
tests: attribute real-socket validation teardown hook timeout on main #2342
Description
2026-10-20 recurrence on a different case in the same lifecycle
Automatic main push run 39333870950, attempt 1, unit (4), job 118338611982, fails a real client on a real socket > completes the handshake the client half now reads after 30,220.60 ms, followed by a beforeEach/afterEach hook timed out for this test and three dangling processes killed. The actual checkout log and structured resource receipt both identify main squash 1f88a7a3018fc669f0dd8d97bd36572d8b315b03, tree 2b199682366c8009d60a79d3ead2c4e15b2a0bd2, parent cea53611ddea4916ac323060bfef42ec18118fea.
This file still declares only afterEach (packages/ui-server/tests/integration/real-socket.test.ts:40-48): server stop, awaited app.close and fixture removal. The handshake test starts at line 106. No handshake/schema assertion failure is printed; the log does not identify the blocking teardown operation. This extends the observed case coverage of the existing timeout investigation, not a claim that its root cause is established. The subsequent frame-validation and malformed-frame cases pass in 286.23/282.04 ms. The failing worker reports 1,705 pass / 24 explicit skips / 1 fail across 140 files, 36,285 assertions. Full log and actual resource receipt are retained.
The preceding ready-candidate run 39331233256 passed all 17 jobs on synthetic bd4388e36b1d116f7996587a768e94284f89c028 with the same tree and separate head/base identity. PR #2197 changes integration coordination/procedure/aggregate reporting and their tests; it changes neither this socket test nor the app/client teardown implementation. That comparison is not a repaired main result or root-cause attribution. No unchanged retry, deadline/assertion change, guard bypass or push to the merged branch is requested.
Keep the bounded phase/owned-child observation work and needs: repro: another hosted recurrence is now available, but the blocking operation and deterministic trigger remain unknown. The October 22 maintainer ruling removes this investigation as a prerequisite of #2111. Keep this p2 needs: repro item open; #2111 proceeds on its own required proof. Existing local non-reproductions below retain their own contexts.
Observation
Automatic main-push run39034501043 after #2070, unit3 job117403876905, reports one failure in a real client on a real socket > every frame the server sends passes the client's own validation (packages/ui-server/tests/integration/real-socket.test.ts:119). Actual selected checkout is squash 18a65206d4ed5f6fc4790728293ac2ce72858026; the job log records its checkout directly.
At13:27:02UTC the case reaches30,219ms and reports a beforeEach/afterEach hook timeout with three dangling processes killed. No frame-count/schema assertion failure is printed. This file declares only an afterEach lifecycle hook: it stops the socket server, awaits app.close, then removes fixture brains. The receipt does not identify which teardown operation stalled or establish a cause. The surrounding handshake and malformed-frame cases pass. Thirteen other selected product jobs pass; unit3 and the integrated aggregate fail. This main workflow is not green.
A previously executed separate UI confirmation39034594574, actual checkout c49d85cf4571e29d2b7675cfe015438416854ad9, passes this exact case in300.43ms. That is a different combined tree/job/context, not a repair or a reproduction of the failed teardown. No unchanged failed-main rerun or assertion/deadline edit was made. Complete failed job log is retained.
Part of #736. This is distinct from #2202's approval-readiness observation; no shared cause is inferred.
Scope
Attribute the actual blocking teardown operation with bounded phase and owned-child evidence at the next natural occurrence. Keep server stop, awaited app.close and fixture removal distinct; select a correction only after identifying the actor/lifecycle. The maintainer's nonblocking ruling remains in force.
Acceptance criteria
- Preserve the failing source/runtime/order and separate passing receipts.
- Capture bounded server-stop, app-close, owned-child exit/output and cleanup observations around the actual hook. Reproduce or observe recurrence before selecting a correction.
- Distinguish resource/actor/teardown conditions from a protocol assertion failure using evidence.
- Any correction retains the real socket/client validation assertions,30-second hook allowance, offline guards and owned-resource cleanup, with meaningful failing/restored and current selected proof.
Out of scope and contract
No raised timeout, skipped validation case, unrelated voice change or blind retry until green. No public contract change or production cause is established.
2026-10-22 triage: reconciled the active dependency statement with the later maintainer ruling. The current afterEach still stops the server, awaits app.close and removes fixture roots (packages/ui-server/tests/integration/real-socket.test.ts:40-48 at aa8c896). Historical failed runs remain failed; no blocking operation, causal repair or new runtime qualification is established.
2026-10-24: attached natively to #736 after #736's browser children moved to the new nested epic and freed capacity; replaced the earlier standalone-by-reference wording.
Verification
Use scripts/test-offline.sh -- bun run test ./packages/ui-server/tests/integration/real-socket.test.ts for a bounded isolated control, and retain the original ordered/hosted context when collecting recurrence. Record source, runtime, actual order, the server-stop/app-close/removal phase boundaries and owned-child exit/output before cleanup. An isolated pass does not attribute the original teardown timeout; do not repeat the already recorded load controls without a new observation dimension.
2026-10-24 continued triage at main dc5d3c73282c29a3f0098306d398743ee18ec791: Made the existing phase-attribution scope and focused offline verification explicit while preserving the 30-second allowance and actual socket/schema assertions. No new recurrence, test run, protocol failure or causal repair is claimed; needs: repro and the nonblocking ruling remain.
Activity
schlessera commentedon Oct 17, 2026
2026-10-17 — Not reproduced, original real-socket file with cleanup-stage observations at main
bfaf3be35408e7983f6b525430118fa867c19d6a.New approach: instrument the existing afterEach boundaries around server.stop(true), awaited app.close(), and removal of the temporary brain roots. This localizes a repeat hook stall without replacing native socket/app teardown, awaiting the previously unawaited stop, changing order, or extending the original 30 s hook budget. Logging can affect timing.
Command:
TMPDIR=<owned disk-backed scratch> timeout100s bun run test ./packages/ui-server/tests/integration/real-socket.test.ts. 7 pass / 0 fail / 16 assertions, exit 0, 3.38 s. All six started-server cleanups return from stop in 0.029–0.768 ms and reach app-close completion within 5.19–18.73 ms of the stop-entry marker; brain-root removals take 0.103–0.539 ms across all seven cases. Stop-return is only the synchronous call returning, not proof of asynchronous server shutdown completion.Bun 1.4.3, own frozen dependencies/full initial build, native local sockets and normal test wrapper/offline guards. This is a single-file observation on a shared local host, not the historical full-preflight load. Temporary diagnostic changes restored byte-for-byte; source and raw log retained locally.
Keep
needs: repro: the reported hook timeout and its ownership remain unresolved. No protocol or teardown fix inferred.schlessera commentedon Oct 17, 2026
2026-10-17 — Second round: Not reproduced. Refined original subject case passes / 2 assertions, 6 filtered, 1.246 s total. Native stop returns at 1.786 ms; app.close completes at 34.505 ms from stop entry, with all 8 added peers already CLOSED. The original stop call remains unawaited and the app close remains awaited. Initial full-file 32-peer arm saturated the configured 32-connection cap: 3 pass / 4 body failures, 15.50 s; no hook timeout. Those connection-refusal failures are excluded as teardown reproductions. at main
33cc244ea9f4f92114201a29d0d27edf24f89ce1.New approach: Keep eight additional native WebSockets open through the original validation case and server/app teardown, send their real client_hello and observe actual closure alongside stop/app-close boundaries. This varies real connection cleanup load rather than only adding stage timestamps.
Not reproduced. Refined original subject case passes / 2 assertions, 6 filtered, 1.246 s total. Native stop returns at 1.786 ms; app.close completes at 34.505 ms from stop entry, with all 8 added peers already CLOSED. The original stop call remains unawaited and the app close remains awaited. Initial full-file 32-peer arm saturated the configured 32-connection cap: 3 pass / 4 body failures, 15.50 s; no hook timeout. Those connection-refusal failures are excluded as teardown reproductions.
Command:
bun run test packages/ui-server/tests/integration/real-socket.test.ts -t "every frame the server sends passes". Own isolated frozen dependencies and successful full build; Bun 1.4.3, shared local Linux host. Browser controls use the unchanged pinned Playwright 1.63.0 noble image, Vitest 4.1.11 and verified locked fonts with network disabled. Original offline guards and deadlines retained. Existing connection cap and original 30 s hook bound are unchanged. Supplemental initial saturation is an expected different boundary, not a product defect; native extra peers are closed during cleanup. No historical teardown ownership or repair established.Keep
needs: repro: no historical cause or correction established. Temporary source edits restored; diagnostic inputs and output retained locally. No fix, PR or hosted dispatch.schlessera commentedon Oct 17, 2026
2026-10-17 — Third round: Not reproduced:original subject passes2assertions,6filtered,890ms total. Server-stop/app-close block completes in2.819ms.
New approach: Keep a real raw TCP peer with an incomplete HTTP request-header block pending during the original server.stop(true)/app.close teardown. This exercises pre-upgrade parser/socket cleanup rather than adding fully negotiated WebSocket peers.
Command:
bun run test packages/ui-server/tests/integration/real-socket.test.ts -t "every frame the server sends passes".Local test at main
2a0b77f675da2f01c32adba3f4920650a5ee09fa, isolated checkout with its own successful frozen install and full build, Bun 1.4.3. Browser probes use the pinned Playwright 1.63.0 noble image, Vitest 4.1.11, verified offline fonts and network disabled. Existing offline guards, assertions and deadlines retained. Shared Linux host; no historical runner equivalence or measured starvation claimed. The native client is paused and its request has no terminating header line. Its destroyed flag is still false immediately after app.close; finally explicitly destroys it. No claim that app.close awaited that client close event, or that every connection kind is qualified.Keep
needs: repro: this does not establish the historical cause or a correction. Diagnostic source restored; command, input and complete output retained locally. No implementation, PR, push or hosted dispatch.schlessera commentedon Oct 18, 2026
2026-10-18 — reproduction pass at
775eb17aab740fa759c2182f306c38fc17acbcbe.Different approach: Keep a real HTTP streaming response body pending through forced server/app teardown, rather than previous incomplete request headers or negotiated WebSocket peers; log native stop/close boundaries.
Observed: Not reproduced: original frame-validation case passes, 2 assertions (6 other cases filtered). A real response returns HTTP 200 and remains pending at teardown: 1 pending request and 1 WebSocket. Forced stop returns in 0.424 ms; awaited app close finishes by 20.694 ms. Outer invocation exit
0, elapsed 0.766 s; outer watchdog fired: false.Command (from repository root; any diagnostic source changes are described above):
An initial diagnostic adapter accidentally omitted the Bun server argument when forwarding app.fetch and failed before socket validation. That setup error is retained separately and excluded from subject evidence. The corrected adapter forwards both arguments, preserving the original upgrade behavior. Bun 1.4.2.
This is focused local evidence, not historical-run equivalence, a repair or full candidate proof. Existing assertions, deadlines and offline guards were retained. Temporary source instrumentation was restored byte for byte. Keep
needs: repro; no readiness or closure change. Full log, command/exit receipt and diagnostic patch are retained in task-owned scratch.schlessera commentedon Oct 18, 2026
2026-10-18 — second reproduction pass at
775eb17aab740fa759c2182f306c38fc17acbcbe.Different approach: Hold a paused real TCP reader against a bounded eight-MiB HTTP response and assert a pending native request before forced teardown. Unlike the earlier small indefinitely-open streaming body, this exercises native send-buffer backpressure. Keep original frame validation and stop/close boundaries.
Observed: Original real frame-validation case passes, 1 test / 3 assertions. Added coverage verifies one pending native request with the TCP reader paused against the 8 MiB response. Forced server stop returns in 0.335ms and app close by 6.595ms; no 5s teardown timeout. Outer invocation exit
0, elapsed 0.665 s; outer watchdog fired: false.Command (working directory: repository root; any diagnostic source changes are described above):
The paused native reader supplies actual bounded send-buffer backpressure. Original WebSocket validation, server stop and app close remain; this is a different induced connection state from the prior indefinitely-open small response.
This is focused local evidence, not historical-run equivalence, a repair or full candidate proof. Existing assertions, deadlines and offline guards were retained. Temporary source instrumentation was restored byte for byte. Keep
needs: repro; no readiness or closure change. Full log, command/exit receipt and diagnostic patch are retained in task-owned scratch.schlessera commentedon Oct 18, 2026
2026-10-18 — third reproduction pass at
775eb17aab740fa759c2182f306c38fc17acbcbe.Different approach: Supply an actual native TCP write-half-close after a complete real HTTP health request, receiving the server response before the original force-stop/app-close cleanup. Record actual socket state, response status and teardown stages. This differs from prior incomplete headers, WebSocket saturation and paused streaming-reader backpressure.
Observed: 1 pass; 0 fail; 3 expect() calls; actual TCP state readOnly with HTTP 200, stop call returned in 0.008ms and app close completed at 16.2ms. Outer invocation exit
0, elapsed 0.667 s; outer watchdog fired: false.Command (working directory: repository root; any diagnostic source changes are described above):
Explicit environment:
BRAIN_REQUIRE_CHROME=1,PUPPETEER_EXECUTABLE_PATH=$PLAYWRIGHT_CHROME,DEBUG=pw:browser,TMPDIR=/tmp/brain-repro-audit-3/native-state.Original broad invocation and its reported failure remain unqualified. No failure-cause or repair inference from these controls.
This is focused local evidence, not historical-run equivalence, a repair or full candidate proof. Existing assertions, deadlines and offline guards were retained. Temporary source instrumentation was restored byte for byte. Keep
needs: repro; no readiness or closure change. Full log, command/exit receipt and diagnostic patch are retained in task-owned scratch.schlessera commentedon Oct 19, 2026
Repro attempt, 2026-10-19, at main
09eb1f78e. Local Linux (WSL2): 32 CPUs, 47 GiB, Bun 1.4.2, 121 mountinfo entries. Chrome for Testing 153.0.8010.12 viaPUPPETEER_EXECUTABLE_PATH. CPU load came fromtaskset -c <cpus> yes > /dev/nullbusy loops.Not reproduced: 0 failures in 25 runs. Runs of
packages/ui-server/tests/integration/real-socket.test.ts(7 tests, about 7 s) with 64 busy loops across all 32 CPUs: no hook timeout and no dangling processes. The full unit3 shard was not replayed.schlessera commentedon Oct 19, 2026
2026-10-20 recurrence on a different case in the same lifecycle
Automatic main push run 39333870950, attempt 1, unit (4), job 118338611982, fails
a real client on a real socket > completes the handshake the client half now readsafter 30,220.60 ms, followed bya beforeEach/afterEach hook timed out for this testand three dangling processes killed. The actual checkout log and structured resource receipt both identify main squash1f88a7a3018fc669f0dd8d97bd36572d8b315b03, tree2b199682366c8009d60a79d3ead2c4e15b2a0bd2, parentcea53611ddea4916ac323060bfef42ec18118fea.This file still declares only
afterEach(packages/ui-server/tests/integration/real-socket.test.ts:40-48): server stop, awaited app.close and fixture removal. The handshake test starts at line 106. No handshake/schema assertion failure is printed; the log does not identify the blocking teardown operation. This extends the observed case coverage of the existing timeout investigation, not a claim that its root cause is established. The subsequent frame-validation and malformed-frame cases pass in 286.23/282.04 ms. The failing worker reports 1,705 pass / 24 explicit skips / 1 fail across 140 files, 36,285 assertions. Full log and actual resource receipt are retained.The preceding ready-candidate run 39331233256 passed all 17 jobs on synthetic
bd4388e36b1d116f7996587a768e94284f89c028with the same tree and separate head/base identity. PR #2197 changes integration coordination/procedure/aggregate reporting and their tests; it changes neither this socket test nor the app/client teardown implementation. That comparison is not a repaired main result or root-cause attribution. No unchanged retry, deadline/assertion change, guard bypass or push to the merged branch is requested.Keep the bounded phase/owned-child observation work and
needs: repro: another hosted recurrence is now available, but the blocking operation and deterministic trigger remain unknown. #2111's automatic-main verification is blocked on this investigation. Existing local non-reproductions below retain their own contexts.schlessera commentedon Oct 22, 2026
Maintainer ruling 2026-10-22: this issue no longer blocks #2111. It stays open as a standalone
needs: reproinvestigation.Why: the real-socket teardown hook timeout is a flake that bounded local work has not reproduced, including the 2026-10-19 attempt of 25 runs under 64 busy loops. Its hosted occurrence on main push run 39333870950, at #2197's squash 1f88a7a, hit a socket test that #2197 did not change. The real-socket file passed in later main runs, for example unit (4) of run 39369165918. Main CI is green with the same controller, for example push run 39699155671 on b17e349 and push run 39745246969 on 7b8a755.
Removed
Blocked by #2342from #2111.schlessera commentedon Oct 22, 2026
2026-10-22 triage: folded the maintainer’s nonblocking ruling into the active body. This remains a standalone p2 needs: repro investigation and no longer gates #2111. The current teardown still has the recorded server-stop/app-close/root-removal boundaries; original failed runs, later passing controls and the missing blocking-operation evidence remain separate. No runtime execution, repair or readiness is claimed.
schlessera commentedon Oct 24, 2026
2026-10-24 triage at main
dc5d3c73282c29a3f0098306d398743ee18ec791: Made the existing phase-attribution scope and focused offline verification explicit while preserving the 30-second allowance and actual socket/schema assertions. No new recurrence, test run, protocol failure or causal repair is claimed; needs: repro and the nonblocking ruling remain.