Skip to content

test(ui-server): retain held maintenance worker failure evidence - #2056

Merged
schlessera merged 6 commits into
mainfrom
test/2054-inner-worker-diagnostics-59b9eec5
Oct 20, 2026
Merged

schlessera merged 6 commits into
mainfrom
test/2054-inner-worker-diagnostics-59b9eec5

Conversation

@schlessera

@schlessera schlessera commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Refs #2054

What changed

The held-maintenance test fixture reports bounded inner stderr and resolved exit/signal evidence when its first receipt fails, before owned-worker cleanup. A real isolated worker passes its ordinary filesystem/masking checks, writes its marker, emits 1 MiB plus ODYSSEUS_INNER_STDERR_SENTINEL, and exits 23 before its held receipt. The outer observer must retain that evidence, and the control verifies owned native cleanup and an empty reservation ledger.

This is test-fixture observability work. No production repair, historical failure cause or common cause with #2016 is claimed. Successful receipts, admission/refusal/crash recovery, offline/native boundaries, 268435456-byte allowance/budget, 5000-ms receipt and 10000-ms worker deadlines remain unchanged. Inner tails remain 512 bytes per stream, fitting worst-case JSON escaping inside the outer 8192-byte diagnostic. No changeset: only three non-shipped test/helper files differ from main; no public contract changes.

Prepared candidate and current local proof

Candidate 76b629b8a892a99a618346a4b24e3511d6c6af8f, tree 2835ff7edf386d5ee66c37dbb22afe4f3b16a41d, has parents 497f0e4a906980046ae680b8db7bf31cb1d04d52 and independently verified base e9bd4621486fa07ced9c2c576e03603fd7f0eba1. The conflict-free ordinary merge preserves the original repair and campaign history. All three contribution paths retain their original blobs, modes, SHA-256 bytes and stable patch 871eacbf0f90b3bd82c3ac5dfcb547fbb8aabd2a.

  • One actual bun run check:pr --base origin/main --plan exits 0, 17.70 s. Its full parsed selection matches the accepted earlier plan: the same three changed paths, 14 fast files, packaging off, and complete affected test/browser/layout/capture/endurance/native-runtime hosted obligations. Flags do not mean those full categories ran locally.
  • After that comparison and fresh live-base/source/resource guards, one actual default bun run check:pr --base origin/main exits 0, 42.80 s. The Linux sandbox setup prerequisite, lint/leakage, environment documentation, pending changeset package names and contribution changeset gates pass. All 14 planned fast files execute: 235 tests / 2315 assertions, 5.43 s. Mandatory offline import-graph feedback resolves --changed=e9bd4621486fa07ced9c2c576e03603fd7f0eba1: 29 tests / 200 assertions across six files, 7.91 s. Raw per-file/case coverage reconciles with no failure or skip. Feedback includes the real maintenance budget/runtime, DB startup-lock, protocol, admission/recovery and inner-diagnostic controls. This is local preflight, not complete hosted proof.
  • The owned source is clean at that exact head/tree. The frozen private copyfile installation, lock, nineteen workspace links/manifests, 843 external manifests and Bun/Node executable hashes were reaudited. All 59516 installed regular files retain ordinary ownership and one link. No install, build, standalone typecheck or separately requested focused/mutation repeat ran in this refresh. All nineteen workspace dist directories remain absent; selection alone is not a build prerequisite.

Execution used Bun 1.4.3+c6da4a4d3, Node v26.7.0, ordinary UID 1000, original HOME, all capability sets zero, original /tmp, private TMP/cache/maintenance ledger, a loopback-only network namespace without default routes, and the repository-managed output lease. Resource floors and bounded owned write/read/fsync/delete probes passed before/after; Btrfs zero inode counters remained explicitly unknown. Owned processes/namespaces exited, the output lock released, reservation rows were verified empty read-only, and only owned scratch was removed. No explicit native-host/identity, Docker, browser, Btrfs/loop-device or provider proof was added. Live main remained e9bd4621486fa07ced9c2c576e03603fd7f0eba1 at final readback.

Retained proof and failed runs

  • Previous refreshed candidate 497f0e4a906980046ae680b8db7bf31cb1d04d52, tree e183209906ee16fdf2ea58ee5ac98a3b9c35ba29, base 510c2909b666b37880999fc50e5759c11591db33, retains its own actual plan (17.60 s) and default (43.03 s) receipts: 235 / 2315 fast and 29 / 200 feedback. These remain historical 497f/510c evidence.
  • Independent R2 review reported no findings on exact earlier candidate ab133d5b08b11a57bb44a6b8a1d5e71f0b65d7d6 / tree e357f39ea3454c77984482cf32d49f0fe8fd1e54 / base 9203e85966b409bfb4d645d48cc37344186d68e3, covering the three files and actual bootstrap/pipes/launch/run lifetimes and optional maintenance fields. Its fresh inner-only retention mutation fails exactly at INNER_WORKER_STDERR_RETAINED (exit 1, eight assertions), continuing stream draining while retaining inner exit 23/no signal and empty stderr; outer exit 1/no signal assertions pass before that failure. Exact restoration passes all three focused test files, 13 tests / 87 assertions, 6.38 s. Strict root tsc --noEmit then passes in 46.956 s without build/install. That head's default preflight separately passes 235 / 2315 plus 29 / 200. These receipts retain their actual ab133/9203 identities, not the refreshed candidate's.
  • Those focused assertions retain the worker marker and tmpfs/UI-DB masks, 1 MiB sentinel payload, exact 512-byte inner tail and bounded outer tail, original inner 23/no signal and outer 1/no signal, kill/drain ordering, native cleanup, empty reservation ledger and ordinary admission/refusal/crash recovery. Earlier campaign focused proof remains head 47216bb8bce99e8b51a663c0ad4534edfa307e16, base 685d83e22cc452e97a2c92ef7ae683f34ceb2371: 13 / 87 before and after intended retention failure. Earlier default remains head 043e7f4ea4f22f1ab93f4093af9b3951f9f24ed3, base 90a95123cfceb169bd2ecf7c213544db742eaed7: 235 / 2315 plus 29 / 200.
  • Original source/failing-first evidence remains base abe5086df48883bc270a36edb127b05519398332, repair head 7961c77ab24069046bd28581b9c0727564ac0af7, Bun 1.4.2+744846f84, Linux 6.18.33.2 and bubblewrap 0.9.0. The real worker reaches its marker; the original reporting path fails the named retention assertion with outer empty-EOF/exit 1/no signal and no retained inner sentinel. Targeted removal fails again and restoration passes. This controlled reporting-loss reproduction does not identify an unknown historical failure cause.
  • Hosted run 38878769506, attempt 1 remains incomplete/cancelled: eleven selected execution jobs and contract pass, but layout shard 2, job 116900394413, hits its annotated 25-minute outer limit after 580 passing tests without a successful category receipt. Aggregate 116909884784 fails on cancelled layout. Inspected checkout 46742ec927194508485d934b5c24890e3e94c152 has parents abe5086df48883bc270a36edb127b05519398332 + original repair head, tree 3941f9217d0d10d70e2f0431de7b31dcb9f6f7f4. Artifact 12038380221 and failed/cancelled logs remain retained.
  • Hosted run 39370921892, unit (3) job 118458753640, fails the same-base local CLI whole-case 30-second test; this is not a failing diagnostic assertion. Late execution-mode evidence and empty diagnostics do not establish the exact expired phase. Aggregate 118463864510 fails unit. Its actual synthetic checkout d524edc433080a1d519e877cea1bba4744fba3af, tree e357f39ea3454c77984482cf32d49f0fe8fd1e54, has parents [9203e85966b409bfb4d645d48cc37344186d68e3, ab133d5b08b11a57bb44a6b8a1d5e71f0b65d7d6]. Raw job/aggregate logs and metadata are retained; job-log SHA-256 is 2520b99dce74e460cf2f57da2d556e56cb2c969b45f80c9b396fb38b1d7a9171. No failed run is replaced by a passing unchanged retry.

Refresh scope and remaining work

Merged repair #2451 is main 510c2909b666b37880999fc50e5759c11591db33: only tests/local-checks.test.ts changes, isolating its byte-identical real CLI and real helper forwarders in a tiny owned Git fixture. All three modes, refusal/plan tripwires, streams/exit cleanup and original deadlines remain. At the 497f refresh, production maintenance/preload/bootstrap/pipes and the diagnostic contribution were unchanged from ab133. That direct source boundary comparison is not a generic passing cache, a new focused/type receipt, or a complete current-hosted proof claim. Automatic main-push run 39375337475, attempt 1 is independently verified green for exact main 510c2909b666b37880999fc50e5759c11591db33, tree c37eb2051b2bfb873c3ee7c7271286f823d02523, parent 9203e85966b409bfb4d645d48cc37344186d68e3. All seventeen jobs and actual checkout logs, thirteen artifact/resource pairs and fourteen artifact API digests reconcile. All 1120 unit files reconcile without omissions or duplicates: 14825 pass, no failure, and 28 justified exclusions (24 explicitly opt-in live-provider cases and four inverse SQLite diagnostic placeholders; the real vector cases pass). The repaired local-checks suite passes 20 tests / 134 assertions in 1.771 s; actual default/full/plan probes return the expected 1/1/0 in 212.769/256.892/228.995 ms. No final main-proof blocker remains for that exact main. This main receipt is separate from both the failed ab133 PR run and the refreshed PR's still-required selected proof; it does not retrospectively identify the old precise slow phase or qualify this PR.

Incoming #2449 is independently verified main e9bd4621486fa07ced9c2c576e03603fd7f0eba1, tree 14ec0d2d77cb10abfcbdaf18f33c733b961934d2, parent 510c2909b666b37880999fc50e5759c11591db33. All 28 incoming files were inspected (2255 additions / 142 deletions): the ordinary build scheduler moves into build-execution.ts; new fixed isolated-build, capture, producer-context, ownership/cleanup, trace and offline-consumer helpers plus their tests are delivered dormant. The shipped policy remains unqualified, CI intent remains off, and ordinary unsupported builds retain conservative rebuilding without Docker. Producer lookup binds actual daemon/kernel/CPU/Node context before reuse; explicit qualification paths grant no general test-result authority. These build/CI producer inputs are material changes, so older build or hosted passes are not treated as equivalent. Package source, diagnostic contribution, lock/manifests, preload, test wrapper, workspace lease implementation and preflight selection code are unchanged. The exact contribution patch is preserved; the current plan and default were executed fresh without a build or isolated producer invocation.

Automatic main-push run 39379334227, attempt 1 is independently verified green for that exact e9bd main/tree/parent: all seventeen jobs and checkout logs, thirteen resource/artifact pairs, fourteen artifact API digests and the canonical aggregate reconcile. Its eight workers cover all 1128 unit files with 14873 passes, zero failures and 28 justified exclusions, without omissions or duplicates. The repaired local-checks suite passes 20 tests / 134 assertions in 1.167 s; the real three CLI modes return the expected 1/1/0. This clears the main-proof hold for exact e9bd; it supplies no current PR confirmation, retrospective failure cause or blanket passing cache.

Earlier incoming production maintenance, Storage, Git preload and proof/lease changes remain material shared inputs, so older passes retain their own identities. The former layout prerequisite #1836 was delivered by #2062 (2e362cc12aa3d2bda38145b6da23d0ac232281b2); that separate proof likewise does not qualify this candidate.

Complete selected proof for the refreshed PR, actual checkout identity reconciliation and the parent-owned publication/confirmation/integration window remain required. The original unknown historical cause and future recurrence investigation stay separate from this controlled diagnostic correction. Keep partial Refs #2054; no delivery or closure is claimed. PR and issue retain milestone 0.44.0 and parent-held claim release044-2054-r2-20261019. This is a prepared body artifact; no push, body edit, readiness change, reservation, confirmation or merge was performed by this local refresh.

@schlessera schlessera added this to the 0.44.0 milestone Oct 16, 2026
@schlessera schlessera added type: test Coverage, harness or proof work with no behaviour change. area: ui-server Hono app factory: turn coordinator, auth, session catalog, routes, cron. labels Oct 16, 2026
@schlessera
schlessera marked this pull request as draft October 16, 2026 09:40
@schlessera
schlessera marked this pull request as ready for review October 20, 2026 04:41
@schlessera
schlessera marked this pull request as draft October 20, 2026 05:58
@schlessera
schlessera marked this pull request as ready for review October 20, 2026 06:52
@schlessera
schlessera merged commit 7c5c978 into main Oct 20, 2026
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ui-server Hono app factory: turn coordinator, auth, session catalog, routes, cron. type: test Coverage, harness or proof work with no behaviour change.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant