ci: attribute loaded external-speech teardown timeout after both behavior cases pass #2457

Activity

1. schlessera

   added

   type: testCoverage, harness or proof work with no behaviour change.Coverage, harness or proof work with no behaviour change.

   area: ui-reactReact chat/files/voice components, stores, WebSocket transport.React chat/files/voice components, stores, WebSocket transport.

   area: ciCI workflows, the gates they run, release scripts, packaging.CI workflows, the gates they run, release scripts, packaging.

   priority: p2Wanted, not scheduled. Pulled in when a milestone has room.Wanted, not scheduled. Pulled in when a milestone has room.

   needs: reproBlocked on a reproduction.Blocked on a reproduction.

   on Oct 20, 2026
2. schlessera

   added a parent issue on Oct 20, 2026

   • ci: attribute repeated hosted runtime setup hook timeouts #2279
3. schlessera commented on Oct 20, 2026

   

   schlessera

   on Oct 20, 2026

   OwnerAuthor

   More actions

   Attribution: the stalled await is app.close(), and on this host its cost comes almost entirely from sandbox launch latency (2026-10-20).

   Source 468618cf106c3e793cbb030f8fc90bf8d018913f, Bun 1.4.2, host Chrome 140.0.7339.127. The host had 3,685 mount-table entries, which is the Docker Desktop leak from #2370; a bare bwrap --ro-bind / / true takes about 330 ms. The load was 32 unpinned busy loops for the whole run. Command:

   unshare --user --map-current-user --keep-caps --net sh -c 'ip link set lo up && ip link add repro0 type dummy && ip addr add 192.0.2.1/32 dev repro0 && ip link set repro0 up && exec setpriv --inh-caps=-all --ambient-caps=-all --bounding-set=-all env BRAIN_REQUIRE_CHROME=1 BUN_JSC_useConcurrentJIT=false bun run test packages/ui-react/tests/external-speech-runtime.test.ts'

   Unchanged source, 3 loaded runs: 2 failed with the same shape as this issue. Both behaviour cases passed, then (fail) (unnamed) [30000.12ms] a beforeEach/afterEach hook timed out, plus Bun.serve() timed out a request after 10 seconds and dangling-process cleanup. The client bundle completed normally (exit 0, empty streams); the package build alone took 115 s under this load.

   For attribution, a temporary experiment wrapped each step of the afterAll at external-speech-runtime.test.ts:83 with timestamps and nothing else. It was reverted afterwards.

   Condition	browser.close()	server.stop / speechServer.stop	app.close()	Result
   Host, idle	99 ms	0 ms	6,158 ms	pass
   Host, 32 loops (3 runs)	118–121 ms	0 ms	27,276 ms; 29,457 ms; more than 30 s (timed out)	1 of 3 failed
   Same load, mount namespace without the leaked mounts (74 entries; nested bwrap about 2 ms), 3 runs	n/a	n/a	756 / 804 / 843 ms	3 of 3 passed, no Bun.serve warning

   The clean namespace binds every top-level directory except /mnt, through bwrap --unshare-user. Everything else matched: source, command, offline namespace and the 32-loop load.

   What app.close() waits on, read from the source and not separately instrumented:

   • close in packages/ui-server/src/app.ts:1059-1093 awaits, in sequence, every in-flight HTTP handler (app.ts:1073), keyedReindex.idle() (app.ts:1083 → packages/ui-server/src/brain/host-execution.ts:47) and governedTurns.close() (app.ts:1084 → packages/ui-server/src/brain/governed-turn.ts:52).
   • governedTurns.close() awaits the post-turn index refresh scheduled by host.governedTurns?.refresh(...) at packages/ui-server/src/ws/run-session.ts:768.
   • None of these is aborted, and none has a deadline. execBrain (packages/ui-server/src/brain/client.ts:225-263) passes a timeout only for bounded maintenance and a signal only for read-only search, so index, module list and briefing run unbounded.
   • Each such CLI run chains two or three sandboxed maintenance workers. Under the leak plus CPU load, each costs seconds.
   • The 10 s Bun.serve warning fits the test server's default idleTimeout, tripping on /api/status. That handler awaits getContentRenderers (packages/ui-server/src/routes/health.ts:61-63), which is brain.modules() → brain module list (app.ts:794). app.ts:1073 then waits for the same handler after server.stop(true).

   What this does and does not show:

   • It names the stalled step: app.close(), not browser.close() or the server stops.
   • It shows that on a host with leaked mounts, sandbox launch latency multiplied by CPU contention is the dominant cost: about 4.5× idle, and over 30 s against 0.8 s in the clean namespace.
   • It does not show which specific in-flight CLI run is live at teardown. That needs a per-run trace inside app.close.
   • It does not show the cause on a host without leaked mounts. Your 2026-10-20 receipt used 20 loops pinned to 4 CPUs, which can stretch the same unbounded waits without any leak. Recording the mount-table size alongside each local receipt would separate the two.
   • No correction is proposed here, per this issue's exclusions.
4. schlessera commented on Oct 20, 2026

   

   schlessera

   on Oct 20, 2026

   OwnerAuthor

   More actions

   2026-10-21 parent evidence reconciliation: recovered raw logs and the exact temporary markers refine the later experiment in the attribution comment.

   The measured app.close() is the test's HttpContractApp.close(): production app shutdown is followed in finally by native-fixture cleanup and directory removal (httpContractApp, packages/ui-server/tests/helpers/http-contract-app.ts:23-115, close at 110-113, source 468618cf106c3e793cbb030f8fc90bf8d018913f). The unfinished outer await therefore does not isolate the internal production shutdown operation.

   The raw markers are cumulative from hook entry. Approximate fixture-close durations are 27,155/29,336 ms for the two loaded successes, 6,059 ms idle, and 601/629/643 ms in the clean namespace. The failed instrumented run completes the preceding stop marker at 118 ms, then prints no fixture-close completion before the 30,016.47-ms hook failure. Server-stop durations round to zero; these are synchronous calls, not proof of awaited server shutdown. The earlier uninstrumented group selected both speech and stats-render files (2/3 failed); the instrumented speech-only group failed 1/3. Those denominators and command contexts remain distinct.

   The clean wrapper changes visible root, mounts and namespace construction, not only mount count. The serial timing difference is consistent with a sandbox-preparation hypothesis, but it does not identify the actual pending CLI/refresh/HTTP operation or its causal contribution. Neither /api/status nor a shared cause with #2342 is established.

   The later worker's observations identify bubblewrap 0.9.0. The separately retained private 0.13.1 qualification belongs to later #2279 preflight; it does not establish the original restored pressure arm's effective binary or native admission. That arm's complete PATH/worker-probe identity remains unrecorded. The new audit's contrary 0.13.1 attribution is rejected here; its original files are retained with a separate parent correction rather than silently rewritten.

   Parent verified 185 recovered/audit files (5,493,773 bytes) with zero checksum mismatches and checked the fixture source and raw timing deltas. A diagnostic overlay now distinguishes fixture/native removal from individual production close stages and correlates requests, refreshes and worker preparation/exit/EOF/publication. It has static/pure checks only; real observer integration remains pending. No #2457 runtime allocation is started while #2456 owns its one instrumented diagnostic.

   Keep needs: repro, original failure identities, assertions, deadlines and full-pressure acceptance. No production repair, complete pressure pass or hosted qualification follows from these corrections.
5. schlessera commented on Oct 20, 2026

   

   schlessera

   on Oct 20, 2026

   OwnerAuthor

   More actions

   2026-10-21 new evidence and bounded next diagnostic: #2456's single instrumented invocation at b751f30becd03af1b506e34f4f62f0cb155b83d3 plus observation edits passed both actual speech cases, then failed an unnamed post-case hook at 30,007.56 ms. It used the original 20 loops pinned to CPUs 0–3, system bubblewrap 0.9.0 and unchanged test budgets. All owned runtime resources are stopped. The typed path passed; its historical failure remains unresolved.

   Fifteen observed status request starts have no recorded return before that observer closed. The observer stopped before teardown, and ordinary maintenance workers bypassed its instrumented agent-launcher branch. Neither pending records nor the idle warning identify the stalled cleanup operation. This is additional failed-invocation evidence, not a causal repair or a completed #2457 attribution. Sampler overhead was substantial at 77.397 CPU seconds and remains an explicit measurement condition.

   One fresh #2457 diagnostic round now has the sole local runtime allocation. Before any full run, it must validate trace coverage through an actual ordinary maintenance worker and real delayed-output completion, distinguish fixture/production/native/removal boundaries, and retain tracing through shutdown. It will reduce broad polling overhead while retaining explicit observation limits and costs. If observer and native prerequisites pass, it may execute exactly one instrumented original-load invocation with unchanged assertions, deadlines, resources and guards. No unchanged retry, additional contrast, product correction, readiness or hosted request is allocated. Stop at its terminal result and retain every failed or incomplete observation.
6. schlessera commented on Oct 20, 2026

   

   schlessera

   on Oct 20, 2026

   OwnerAuthor

   More actions

   2026-10-21 terminal diagnostic and parent review.

   The single allocated invocation failed with exit 1 after both behavior cases passed (16,359.86 ms and 14,551.66 ms; 13 assertions). Its post-case hook timed out at 30,051.83 ms. Source was draft b751f30 plus a retained observation overlay. Original 20 loops on CPUs 0–3, setup/case deadlines, native/offline guards and speech case bodies were preserved. This is diagnostic evidence, not unmodified-candidate qualification.

   The new trace identifies the unfinished operation: production close awaits the admitted HTTP-handler promises at app.ts:1073, within the outer fixture close. The actual pending promise IDs join to 15 /api/status requests, then content-renderer discovery, module list --json and serialized per-brain maintenance execution. Five handlers completed while close waited; ten remained at the final snapshot, 29,434 ms after close began. Those ten had entered outputLock.withLock but not its callback. The unchanged lock implementation establishes the acquisition-wait interpretation; lock queue internals were not instrumented.

   A governed index refresh also competed for that lock, entering its body after approximately 44,476 ms and launching a worker about 28,860 ms after close began. Its final observed lifetime was only about 574 ms. The later governedTurns.close stage, other subsequent production shutdown stages and fixture native cleanup/removal were not reached. The observed queue was making progress; this is not proof of a permanently stuck process or an abnormally slow final index worker. The unlabelled idle warning remains unattributed.

   The operation trace stayed active through teardown with 1,875 sequential events and 13 ordinary maintenance launches. Actual ordinary-worker, delayed-output, fixture-nesting and cleanup controls admitted the observer; a classifier mutation produced the intended EOF assertion failure and restoration passed. External sampling stopped at its data cap before teardown. It recorded 18,186,777 bytes because its between-sample check overshot the nominal 16 MiB threshold. Thus no teardown-time OS wait-state or leaf-internal timing establishes whether sandbox preparation, payload execution or another stage accounts for the worker durations. Observer timing effects remain a limitation.

   The actual diagnostic used system bubblewrap 0.9.0 and passed the native worker-host guard. This does not establish the original historical arm's effective binary; later private 0.13.1 preflight remains separate. No common cause with #2456 or #2342 is claimed.

   Command exit and both output EOFs joined. All 20 loops exited; the final independent census found zero surviving or unreaped matches across 276 owned PID/start identities. Failed fixture scratch was preserved then removed after process cleanup; that does not make in-test teardown pass. Parent independently rehashed all 284 indexed files / 52,044,914 bytes without mismatch, manifest SHA256 84b4331c49d82e4d18b92b322b9ad0668678f14ad8351b655ee172bd8131be85.

   Builder wall was 223.829 s, with 67.062 s CPU. Command-through-exit/EOF/anchor wall was 315.725 s and reported waited-tree CPU 73.132 s, including the builder and potentially excluding interrupted unreaped descendants. Controller including cleanup took 316.757 s; loop CPU was 1,172.990 s and sampler CPU 0.618 s, each separately accounted. Complete observer CPU is unknown. No matched speedup or aggregate total is claimed.

   This runtime allocation is terminal. A fresh source/mechanism review now traces request cancellation, metadata discovery and maintenance ownership against existing guarantees and current main, to specify a decisive bounded causal control and justified correction. No further full pressure run, product correction, hosted dispatch or merge reservation is allocated. Keep needs: repro and all original acceptance criteria; #2279's complete-pressure qualification and #2456's historical typed failure remain open.
7. schlessera

   added

   type: fixSomething is broken and must behave the way it is documented.Something is broken and must behave the way it is documented.

   area: ui-serverHono app factory: turn coordinator, auth, session catalog, routes, cron.Hono app factory: turn coordinator, auth, session catalog, routes, cron.

   agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.Context and acceptance criteria are complete enough for an agent to take this unattended.

   and removed

   type: testCoverage, harness or proof work with no behaviour change.Coverage, harness or proof work with no behaviour change.

   needs: reproBlocked on a reproduction.Blocked on a reproduction.

   on Oct 20, 2026
8. 3 remaining items

   Load more

9. schlessera commented on Oct 21, 2026

   

   schlessera

   on Oct 21, 2026

   OwnerAuthor

   More actions

   2026-10-21 implementation handoff and independent qualification allocation.

   Draft PR #2464 is open at 063f5a0, tree 3b639c2ddf92125d266dfb762c9c5147e41f14ea, tested base e2deae1. It is linked to the coordinator thread, remains draft, and carries milestone 0.44.0 and the additive contract/minor changeset. The separate #2279 draft is unchanged.

   The correction forwards admission-only cancellation for status and GET modules. Queued lookups cancel on caller abort or app close; admitted execution retains its worker, publication and cleanup lifetime. The real app fails the named queue-before-lock-release oracle before the correction and when propagation is removed, then passes on restoration. Both routes cover pre-aborted, queued, shutdown and actual loopback abort. The real ordinary-worker test holds stdin, observes exit and both EOFs, then holds publication and cleanup; locks and database ownership remain until those stages finish. Forwarding shutdown into active execution causes the intended publication assertion failure. Unchanged HTTP-drain/browser/native owner tests and search cancellation tests pass.

   Required preflight passed once on the committed tree: 562 fast invariants, two separately invoked runtime tests, and 2,307 affected tests across 184 files with 20,421 assertions; zero failures. All nine final metadata tests ran. Strict types, API, citation/leakage and changeset checks passed. Wall 3,074.84 s, user CPU 1,534.99 s and system CPU 775.71 s. These are validation costs, not a performance gain. System bubblewrap 0.9.0 was used for causal controls; private 0.13.1 was separately qualified for the preflight prerequisite only.

   Parent independently verified 127 indexed files / 11,939,440 bytes, manifest SHA256 fd5f14cdc17590c75dbcb4b300aab0b00ee1ccd8c5723d40441532e1b079b901. Cleanup confirms the recorded owned identities absent, with sampling coverage limits retained. Complete pressure, hosted proof and historical #2456 attribution remain open.

   A fresh independent review now checks the exact cancellation candidate. If no unresolved correctness finding prevents qualification, it will prepare a separate local combined tree containing current main, this candidate and exact #2279 head b751f30. After merge-specific checks and controller/native admission, it has the sole allocation for exactly one original 20-loop/CPUs 0–3 speech invocation with all original assertions, deadlines and guards. The combined source receives its own identity; earlier proof is not relabelled. No diagnostic overlay, extra baseline/contrast, unchanged retry, private fast-native substitution or producer-audit weakening is allowed. A review or prerequisite failure stops before pressure.

   Parent claim e1849-2457-20261021-r4-cancel remains active. No readiness transition for either PR, hosted dispatch or merge-window reservation is allocated. #2319 activation stays deferred.
10. schlessera commented on Oct 21, 2026

    

    schlessera

    on Oct 21, 2026

    OwnerAuthor

    More actions

    2026-10-21 independent review and original-load qualification passed.

    The reviewer found no unresolved cancellation correctness gap and reran the exact #2464 metadata suite: 9 passed, 108 assertions. A separate local merge combines #2464 head 063f5a0, #2458 head b751f30 and main e2deae1. Combined commit 113cd7f1d56a9c7b182b0b69a2bb9bb183d74003 has tree dea94afaaf2f6168fe47f2d6f650ceeb723412c8. There were no merge conflicts; only the CI decision document combines content from both sides. Full source attribution retained all invocation, metadata, producer-audit and dormant-reuse guarantees.

    Exactly one original-load invocation passed on that combined tree. Twenty owned busy loops remained on CPUs 0–3 throughout; all original assertions, timeouts and guards remained unchanged. The actual audio/final-drain/no-grant case passed in 14,774.05 ms, unmount in 13,464.65 ms; inner 2 pass / 0 fail / 13 assertions, outer 1 pass / 0 fail / 1 assertion. Teardown completed. Whole command exit 0, signal null and both output EOFs joined; no safety watchdog fired. Strict types and 48 focused combined-tree tests also passed.

    This run used system bubblewrap 0.9.0, Bun 1.4.2, Node 24.21.0, Chrome 140.0.7339.127 and 3,689 mount entries, with actual native admission. No diagnostic overlay, private fast-native prefix, clean namespace, JIT change or retry was used. The unlabelled idle-request warning still appears and is not assigned a cause. Historical #2456 attribution remains open.

    Fresh invocation build: 198.053 s wall / 61.326 s waited CPU. Checked hook admission: 7.621 ms. Controller start through command exit/EOF/anchor: 260.514 s; command waited CPU 66.682 s includes the build. Controller including cleanup: 262.551 s wall / 1.029 s CPU. Load loops: 964.484 s CPU, separately accounted. These are overlapping scopes with declared exclusions, not a matched speedup or comprehensive total.

    Parent independently verified 177 indexed files / 44,245,801 bytes with no mismatch, manifest SHA256 b7031119f6c79c0906711c4bcd4f78dfb7881410adbdd998b05ce4ee18e515bd. All 153 recorded PID/start identities were absent, including zombies; one-second sampling cannot establish exhaustive descendant coverage. The unrelated sibling survived owned cleanup before its own owner stopped it.

    The runtime allocation is terminal. This is exact local combined-tree qualification, not hosted proof or delivery of either source PR. #2464 is now eligible for the repository's reserved candidate-confirmation procedure as a scoped Refs #2457 correction. Its current head and live main still match its successful preflight; the synthetic merge parents and tree were checked independently. Selected hosted proof and actual squash/main-push verification remain required. #2458 remains separate, historical #2456 attribution is retained, and #2319 activation remains deferred.
11. schlessera

    added

    blockedWaiting on another issue or an upstream dependency. Say which in the body.Waiting on another issue or an upstream dependency. Say which in the body.

    on Oct 21, 2026
12. schlessera

    added a sub-issue on Oct 21, 2026

    • ui-sdk: clean owned-prune fixture roots when source transformation refuses #2466
13. schlessera

    removed

    blockedWaiting on another issue or an upstream dependency. Say which in the body.Waiting on another issue or an upstream dependency. Say which in the body.

    on Oct 21, 2026
14. schlessera commented on Oct 21, 2026

    

    schlessera

    on Oct 21, 2026

    OwnerAuthor

    More actions

    Decision (maintainer, 2026-10-21): correct the attributed boundary by having app.close() cancel in-flight read-only (gate: "shared") brain runs after a short grace period. Runs that write Markdown or git are not interrupted. This is implemented under #2473, which carries the scope and acceptance criteria. This issue's own receipt requirements (#2279's full-pressure pass and the loaded re-run) still apply once #2473 lands.
15. schlessera

    added

    blockedWaiting on another issue or an upstream dependency. Say which in the body.Waiting on another issue or an upstream dependency. Say which in the body.

    on Oct 21, 2026
16. schlessera commented on Oct 21, 2026

    

    schlessera

    on Oct 21, 2026

    OwnerAuthor

    More actions

    2026-10-21 resumption: main advanced from c2f080f to 2db5626 through #2478. Its automatic main CI run 39675767925 reports success; this statement is an API conclusion, not a new independent checkout audit.

    #2478 adds retained planning observations and preserves the consumer assertions/deadlines. Its own remaining-work section says the timeout mechanism and targeted correction are still open. #2465 was closed immediately after that observer merge; closure alone is not interpreted here as repair or satisfaction of this candidate's failed selected proof. The original run 39595607928 remains failed and all diagnostic limitations remain recorded.

    One dedicated GPT-6-Astra child is refreshing the preserved #2464 source onto current main, reviewing the relevant observer/worker changes and obtaining required local affected proof. No hosted confirmation, pressure run, ready transition, push or integration reservation is allocated to that child. The exact previous candidate c2932a6 and its receipts remain preserved. Parent owns review and later proof disposition.

    #2473 is independently owned through #2475; its grace-then-cancel work and required loaded rerun remain separate. #2319 activation remains deferred. Finite integration-state inspection found no active reservation, so none was released or acquired. Existing parent claim e1849-2457-20261021-r4-cancel remains the scoped owner of this refresh.
17. schlessera commented on Oct 21, 2026

    

    schlessera

    on Oct 21, 2026

    OwnerAuthor

    More actions

    2026-10-21 refresh review: local head 7d4db2a26fbf08117dc9d45b5327e31d87f21c19 / tree e79af978ccf6944bcb558bd76e093373cdc140ff preserves the history merge of c2932a6 with main 2db5626. Final fetched main is unchanged. Parent independently verified all 89 evidence files / 1,971,530 bytes; manifest SHA256 30c263d1a0b7941dc9a074608510585e8867edd553eb42d0bd6c1c57ab4609a8.

    The focused run passed 62 cases and failed one new upstream observer control: its expected complete record was marked incomplete because more than ten host processes were busy. All admission, owned-prune and consumer controls passed, as did strict types. Required check:pr remains unrun; the plan-only output is not a preflight pass. The observer failure is separately tracked by #2481, attached to #1849, with one isolated GPT-6-Astra implementation worker. Production caps, truthfulness, deadlines and real accounting guarantees must remain.

    The clean local refresh is retained without push; PR #2464 remains draft at c2932a6. Resume required local qualification after the scoped observer correction, then reassess hosted prerequisites. No unchanged retry, pressure attempt, ready transition or merge reservation occurred. #2473 and the historical planning-timeout evidence remain separate.
18. schlessera

    added a commit that references this issue on Oct 21, 2026

    fix(ui-server): cancel in-flight shared brain runs after a close grac…

    ...

    2ddffb8
19. schlessera commented on Oct 21, 2026

    

    schlessera

    on Oct 21, 2026

    OwnerAuthor

    More actions

    Local qualification resumption — 2026-10-22

    The observer-control prerequisite #2481 is delivered by #2483 / dcf5341. Candidate proof passed; automatic main run 39688976449 passed all 17 jobs, with every actual checkout independently verified against the squash and the aggregate confirming all categories succeeded without retained results. The scoped correction preserves production observer limits and consumer assertions/deadlines; no historical timeout cause is inferred.

    The prior local refresh 7d4db2a26fbf08117dc9d45b5327e31d87f21c19 and its ambient observer assertion failure remain preserved. One GPT-6-Astra worker now owns the local merge of the delivered correction and required affected controls, strict types and check:pr under existing parent claim e1849-2457-20261021-r4-cancel. PR #2464 remains draft at c2932a6 until parent review of fresh local proof. No pressure allocation, hosted dispatch, ready transition or reservation is included in this local task.

    Blocked by #2473

    That independently owned shutdown correction and the maintainer-required loaded rerun still govern full issue completion. The historical failed hosted run and original pressure acceptance criteria are unchanged.
20. schlessera

    added 2 commits that reference this issue on Oct 21, 2026

    fix(ui-server): cancel in-flight shared brain runs after a close grac…

    ...

    a75636d

    fix(ui-server): cancel in-flight shared brain runs after a close grac…

    ...

    b17e349
21. schlessera commented on Oct 22, 2026

    

    schlessera

    on Oct 22, 2026

    OwnerAuthor

    More actions

    Unblocked: #2473, #2473, #2465 are closed, so scripts/sync-project.ts is removing the blocked label.
22. schlessera

    removed

    blockedWaiting on another issue or an upstream dependency. Say which in the body.Waiting on another issue or an upstream dependency. Say which in the body.

    on Oct 22, 2026