Skip to content

ui-server: held maintenance worker stderr is not reported when its first receipt fails #2054

Closed

Description

@schlessera

Out of scope

No change to the SQLite startup-lock correction in #2016 / PR #2019, maintenance allowance/budget/deadlines, receipt protocol, native isolation or production worker semantics. Do not claim a historical failure or common cause from this source inspection.

Context

The #2016 receipt-reader audit found a separate reporting gap in the server-hold branch of packages/ui-server/tests/maintenance-temp-boundary-child.ts:24-33 at PR #2019 head 3d749d7973d3e8702c0f3ce298f4cf21463c942d. diagnostic starts consuming the inner launched worker's stderr with new Response(running.stderr).text() at line 27. If ledgerLine(running.stdout) rejects at line 28, execution never reaches await diagnostic at line 33, and the collected inner stderr is not published. That collection is also unbounded. The outer boundary observer reports the fixture child's exception/phase, rather than recovering those inner worker bytes.

Observed evidence is source control flow only. No controlled inner-worker failure, leaked resource or historical runtime failure is claimed. Searches for the child filename plus stderr found #2016 only; that issue's reproduced failure is outer-child DB initialization before the inner worker launches. This follow-up is independent of that supported correction and does not block it. Related to #736's outcome; no native sub-issue attachment is inferred.

Scope

First establish an offline, owned real-worker control that fails before its held receipt with a distinctive stderr sentinel. Record the actual inner/outer exit and signal, original source/runtime identity and cleanup. If the reporting loss is reproduced, retain bounded inner stderr and original exit evidence on this failure path while preserving ordinary successful receipts.

Acceptance criteria

  • A real isolated worker emits a distinctive stderr sentinel and fails before its first held receipt; the current source is shown losing that diagnostic on the intended path, rather than failing at fixture/module startup.
  • A supported correction reports bounded inner stderr and the original exit/signal before owned-child cleanup changes them.
  • The successful held receipt, actual admission/refusal/crash recovery, native/network/filesystem masks, fixed allowance/budget, original deadlines and owned process/DB/filesystem cleanup remain exercised.
  • Removing diagnostic retention makes the named runtime assertion fail; restoring it passes. Appropriate local preflight and selected current-head hosted proof pass before completion.

Verification

Reproduced on source base abe5086df48883bc270a36edb127b05519398332, Bun 1.4.2+744846f84, Linux 6.18.33.2, bubblewrap 0.9.0. The owned offline real worker uses the unchanged 268435456-byte allowance/budget and 10000-ms worker/5000-ms receipt deadlines. After passing its actual tmpfs and UI-DB mask checks, it writes its ordinary marker, emits 1 MiB plus ODYSSEUS_INNER_STDERR_SENTINEL to stderr, and exits 23 before its held receipt. On the original reporting path, the named runtime assertion INNER_WORKER_STDERR_RETAINED fails because only the outer empty-EOF exception is reported. Outer exit is 1 with no signal. With bounded observation restored, the inner resolved exit is 23 with no boundary-reported signal, outer exit remains 1, the sentinel is retained in a 512-byte inner tail per stream (allowing worst-case JSON escaping) that fits the outer 8192-byte diagnostic, native fixture state is removed and the reservation ledger is empty. Discarding stderr retention reproduces the named assertion failure; restoring it passes. The normal server/cron admission, refusal and crash recovery control also passes. Local preflight and selected hosted proof remain required before completion.

Contract impact

Test fixture diagnostics only. No public contract, product change, release commitment or implementation is selected by this source-observed note.

Updated 2026-10-16: established the controlled real-worker reporting-loss reproduction; this does not establish a historical failure or common cause with #2016.

Current handoff (2026-10-16)

Prerequisite #1836 is resolved by merged PR #2062 (squash 2e362cc12aa3d2bda38145b6da23d0ac232281b2).

The reproduction and test-only correction are committed in draft PR #2056 at 7961c77ab24069046bd28581b9c0727564ac0af7. Local typecheck and preflight pass; the failing-first and targeted inner-stderr mutation/restoration receipts are recorded in the PR. Ready run 38878769506 attempt 1 passes eleven selected execution jobs and contract, including 9543 unit tests, both browser shards, native runtime, captures and layout shard 1. Layout shard 2 reaches its 25-minute job cap after reporting 580 passing tests and before emitting a successful category receipt; its check annotation explicitly identifies the outer limit. The aggregate fails and the workflow concludes cancelled. Full selected hosted proof is incomplete, so this issue remains open and the PR uses Refs.

The cancellation is recorded on the existing layout measurement/scheduling task #1836. No fixture failure or common cause is inferred. Preserve run 38878769506 and its exact checkout evidence. Layout scheduling qualification is complete: #2062 confirmation run 38895983513 attempt 1 passed all 15 selected jobs, including both ordinary layout shards and separate endurance. This does not qualify PR #2056. Resume by completing its own fresh verification: assess relevant main changes, refresh the PR if needed, rerun required local preflight and complete current-head selected hosted proof, inspect actual checkout identities, then restore readiness and merge with matching head and verified milestones. No scope expansion, deadline increase or unchanged retry was used to obtain green.

Updated 2026-10-16: resolved #1836 dependency after merge and exact squash tree verification. Draft PR #2056 remains unverified on current main and was not resumed or marked ready by this reconciliation.

Current campaign focused proof — 2026-10-19

Claim release044-2054-r2-20261019 preserves the original three-file repair byte-for-byte in local candidate 47216bb8bce99e8b51a663c0ad4534edfa307e16, integrated base 685d83e22cc452e97a2c92ef7ae683f34ceb2371. Private frozen copyfile installation and static gates pass. On Bun 1.4.3 and bubblewrap 0.13.1, the focused real-worker control passes 13 tests / 87 assertions. An inner-only stderr-retention mutation fails exactly at INNER_WORKER_STDERR_RETAINED, retaining the original inner exit 23 with no signal; byte-identical restoration passes 13 tests / 87 assertions. Bounded sentinel, outer exit 1 with no signal, masks, cleanup and empty ledger are exercised. These current focused receipts remain distinct from historical hosted runs.

The earlier focused native window returned with the candidate unpushed. In the subsequent authorized preflight window, the campaign branch integrates the unrelated frozen-native-input self-test from main and fast-forwards draft PR #2056 to 043e7f4ea4f22f1ab93f4093af9b3951f9f24ed3, base 90a95123cfceb169bd2ecf7c213544db742eaed7, tree 513c3231ab8a8fc6317b1d6385c2fa6b949b20ef. The original three repair files and frozen dependency inputs are unchanged. Actual default bun run check:pr --base origin/main exits 0: 235 fast invariants / 2,315 assertions plus mandatory import-graph feedback with 29 tests / 200 assertions. Metadata/lint/leakage gates pass. Focused receipts above retain their original source identity. Complete selected hosted confirmation remains required; PR #2056 remains draft under Refs. No delivery or issue completion is claimed. The parent campaign retains the assignment and owns final confirmation/readiness/merge windows. No privileged native-host, Btrfs, loop-device or browser-suite execution was added.

Updated 2026-10-19: recorded actual local preflight and the ancestry-preserving draft candidate refresh; historical incomplete hosted proof remains incomplete.

Activity

  1. added
    type: testCoverage, harness or proof work with no behaviour change.
    area: ui-serverHono app factory: turn coordinator, auth, session catalog, routes, cron.
    needs: reproBlocked on a reproduction.
    on Oct 16, 2026
  2. self-assigned this
    on Oct 16, 2026
  3. schlessera commented on Oct 16, 2026

    @schlessera
    OwnerAuthor

    Claiming #2054 at the explicit request to establish its reproduction and complete supported test-fixture repairs. Claim token: 2054-59b9eec5; intended branch: test/2054-inner-worker-diagnostics-59b9eec5. I will first exercise a real isolated worker failing before its held receipt; the missing reproduction remains explicit until demonstrated.

  4. added
    agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.
    and removed
    needs: reproBlocked on a reproduction.
    on Oct 16, 2026
  5. added this to the 0.44.0 milestone on Oct 16, 2026
  6. schlessera commented on Oct 16, 2026

    @schlessera
    OwnerAuthor

    2026-10-16 reproduction established on abe5086df48883bc270a36edb127b05519398332, Bun 1.4.2+744846f84, bubblewrap 0.9.0 and Linux 6.18.33.2. The control reaches the actual isolated maintenance command, passes its tmpfs/DB-mask checks and publishes its ordinary marker before emitting 1 MiB plus the stderr sentinel and exiting 23 without a held receipt. The unchanged reporting path fails INNER_WORKER_STDERR_RETAINED at the expected assertion; it reports outer exit 1/no signal and only the empty-EOF exception.

    The supported correction uses the existing bounded receipt observer for the inner worker, with a 4096-byte inner tail so the metadata and stderr fit its outer 8192-byte observer. The runtime assertion now inspects that complete nested path, including original inner resolved exit 23/no boundary-reported signal and outer exit 1/no signal. Native fixture cleanup and an empty reservation ledger are asserted. Discarding diagnostic retention makes the named assertion fail again; restoration passes. The normal server/cron admission/refusal/crash-recovery test also passes. Source-only readiness wording has been replaced in the body and needs: repro removed. This controlled failure does not explain any historical run or establish a common cause with #2016.

  7. added
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    on Oct 16, 2026
  8. schlessera commented on Oct 16, 2026

    @schlessera
    OwnerAuthor

    2026-10-16 handoff: reproduction and supported test-fixture correction are preserved in draft PR #2056, head 7961c77ab24069046bd28581b9c0727564ac0af7. Failing-first, targeted inner-stderr mutation/restoration, 13 focused tests (87 assertions), typecheck and local preflight pass. Ready run 38878769506 attempt 1 passes eleven selected execution jobs and contract; its unit shards pass 9543 tests. Layout shard 2 hits its annotated 25m0s job cap after reporting 580 passing tests and before a successful category receipt. Aggregate fails, workflow concludes cancelled, and required hosted proof remains incomplete. No retry or merge was performed.

    The body records exact checkout/runtime evidence, the remaining verification and canonical blocker #1836, where this additional layout capacity evidence is now recorded. The PR uses Refs and remains draft. Resume after the layout qualification work, then assess relevant main changes and complete current-head selected proof before matching-head merge. Releasing this invocation’s uncontested claim; no human-only action is required.

  9. removed their assignment
    on Oct 16, 2026
  10. removed
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    on Oct 16, 2026
  11. self-assigned this
    on Oct 18, 2026
  12. schlessera commented on Oct 18, 2026

    @schlessera
    OwnerAuthor

    2026-10-19 campaign claim release044-2054-r2-20261019, intended branch test/2054-campaign-r2.

    Fresh readback confirms the issue is open and unassigned, and the previous claim released custody. Explicitly selected independent refresh of the existing diagnostic correction in PR #2056; this selection does not invent a priority label. The prior claim released and the historical layout scheduling prerequisite is delivered. Preserve the existing implementation and original exit-23/sentinel/mutation receipts; assess current main, refresh and obtain required focused proof and preflight. No new implementation or historical-cause attribution is presumed.

    A dedicated GPT-6.1-Sol worker owns this scoped continuation. The campaign coordinator serializes claims, local heavy-runtime access, final hosted confirmation and merge windows. No full current-candidate proof or delivery is claimed. Other issue branches and stopped investigations remain untouched.

  13. schlessera commented on Oct 22, 2026

    @schlessera
    OwnerAuthor

    Maintainer ruling 2026-10-22: closed as delivered by #2056 (squash 7c5c978, 2026-10-20). Each acceptance criterion:

    • Failing first. On base abe5086, a real isolated worker passes its masks, writes its marker, emits 1 MiB plus ODYSSEUS_INNER_STDERR_SENTINEL and exits 23 before its held receipt. The original reporting path fails the named retention assertion with no retained sentinel, so this is not a fixture or startup failure.
    • Correction. Bounded inner stderr (512-byte tails within the 8,192-byte outer diagnostic) and the original exit 23 / no signal are reported before owned-child cleanup.
    • Preserved behaviour. The successful held receipt, admission/refusal/crash recovery, native/network/filesystem masks, the 268,435,456-byte allowance, the 5,000 / 10,000 ms deadlines and owned cleanup with an empty reservation ledger all remain exercised.
    • Mutation and proof. Removing inner retention fails INNER_WORKER_STDERR_RETAINED (packages/ui-server/tests/maintenance-temp-boundary-diagnostics.test.ts:66); restoring it passes 13 tests / 87 assertions. check:pr passed. The complete selected confirmation, run 39382960168, passed on head 76b629b, with packaging not selected. Main push run 39385119610 on 7c5c978 passed.

    Milestone 0.44.0 is kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.area: ui-serverHono app factory: turn coordinator, auth, session catalog, routes, cron.type: testCoverage, harness or proof work with no behaviour change.

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions