tests: shared storage scan intermittently returns two collection IDs #3013

Context

While proving #3001, two native processes share one collection and the same complete cached pages (packages/ui-server/tests/storage-shared-scan.test.ts:42-60) failed at STORAGE_SHARED_ONE_COLLECTION on line 54: the two successful child processes returned different collection IDs.

The complete bun run test packages/ui-server receipt is 4094 pass / 1 fail, 28408 assertions, 345 files, 2251.88 seconds, at head b9b7d099ba362838f22407902c9d3a4b9fc70cc6 based on c47483196f02c1098128207ce68c4f0de6b571ab, Bun 1.4.2 and frozen dependencies. This case took 2373.42 ms; its readiness and child-exit assertions passed, and this is not a timeout failure. All 11 new inbox placement/order tests passed in that full run.

The complete isolated storage-shared-scan file passes on the same head and on that base: 7 pass / 0 fail, 38 assertions each (8.93 and 8.91 seconds). The storage implementation, this test and its child fixture are unchanged between those sources and current main 07fbce896099b87903bb994e3b6a939f232669fd.

The child emits READY before waiting for stdin and calling coordinator.start (packages/ui-server/tests/fixtures/storage-shared-process.ts:18-21). Readiness alone does not record whether the active collection leases overlapped. That is an investigation lead, not attribution of the failed invocation. The first/second acquisition, collection, cache publication and release timeline was not captured.

Those full package controls have since completed: current main is 4101 pass / 1 unrelated fail (#3016), and the refreshed candidate is 4113 pass / 0 fail (details below). The existing failed receipt is retained; no source, assertion, timeout or offline guard was changed to obtain the isolated passes.

Out of scope

Inbox lifecycle worker migration, changing forced-scan reuse semantics, weakening the one-collection assertion, extending runtime deadlines or relaxing offline guards.

Scope

Repair the proved missing-overlap premise in the two-native-process fixture. Establish and observe a real owner/follower lease overlap before allowing the owning collection to finish, then keep the existing one-collection, complete-page and lease-cleanup assertions. Use an owned, bounded fixture barrier with cancellation/joined teardown; readiness must mean the required shared-lease state, not merely that both processes printed READY. Add a serial forced-scan control that correctly expects distinct collection IDs. Do not change production forced-scan or completed-result semantics.

Acceptance criteria

• Retain the original failed package receipt, the existing passing package/focused controls and the controlled nonoverlap receipt below as distinct evidence. Do not claim the historical run's exact schedule is known.
• The native positive test observes an actual owner/follower join before releasing collection completion; both callers then receive one collection and the same complete cached pages under the existing deadline.
• A deterministic negative control removes or violates only the overlap barrier and fails the intended STORAGE_SHARED_ONE_COLLECTION assertion, rather than a setup/module-load error. Restored proof passes.
• A serial forced-scan control completes the first lease before the second starts, expects two distinct fresh collections with complete matching pages, and preserves owner authorization and cleanup. Serial scans must not be coalesced to make the test pass.
• Every success, timeout, cancellation and assertion failure releases/joins owned children, streams, lease and temporary state. Preserve all existing collection/page, cancellation, ownership, offline and cleanup assertions.
• Focused complete-file proof and required affected/current-candidate proof pass for the eventual fixture correction. No wider product correction is bundled without separate causal evidence and scope.

Verification

bun run test packages/ui-server/tests/storage-shared-scan.test.ts
bun run test packages/ui-server
bun run check:pr --base origin/main

On the current machine, run bun run test packages/ui-server with the repository’s automatic checkout lock and run heavy verification commands one at a time. The original machine’s host-wide slot convention does not apply. Record the complete result and concurrency procedure.

Contract impact

None expected. Preserve the existing forced-scan semantics while finding the failed invocation's owner; determine the affected surface before choosing a product correction. The historical failed invocation remains unattributed; the controlled fixture premise below is sufficient for this narrow test-only repair, and does not establish a production scan defect.

2026-10-26 triage at main 2f10e0407769b685083c17a1c90bb43a060b94a0: Classified as priority: p2 (wanted, unscheduled work; no release commitment inferred). The recorded controls do not establish the missing cause; retain needs: repro. The closest general verification epic #736 is at its direct-child limit, so this investigation remains standalone by reference.

Completed comparison evidence — 2026-10-26

Reporter evidence from the completed comparison:

Completed controls from draft PR #3014 for #3001, with Bun 1.4.2 and frozen dependencies:

• Current main 07fbce896099b87903bb994e3b6a939f232669fd: the full bun run test packages/ui-server control finishes 4101 pass / 1 fail, 28421 assertions, 345 files, 2029.88 seconds. The shared-storage collection identity case passes. Its sole unrelated failure is the private-index advance observation filed as tests: private-index advance returns an empty rebound result in the package suite #3016.
• Refreshed candidate e77b90c2f9dd6682234f4ba426d44fb08d392a91, merged with that main: the complete package suite passes 4113/4113, 28462 assertions, 346 files, 2114.01 seconds, including the original shared-storage case.

Both full commands used the shared heavy-command slots. The storage implementation, shared-scan test and child fixture remain unchanged across these sources and the original failed candidate b9b7d099ba362838f22407902c9d3a4b9fc70cc6. No storage source, fixture, assertion, timeout or offline guard was changed to obtain these controls. The failed original full-head receipt remains retained. These are passing controls, not attribution or a repair; the missing first/second lease and publication timeline remains the readiness gap.

Bounded current-main control — 2026-10-29

On isolated main a8e80eea1ec045579ee7d69d6019bce5e78afa5e, Bun 1.4.4 (38afee6c8), with an independently copied frozen install and unchanged lock SHA-256 5bbd5f6651f111db3fcfabeda512d44a21163cba54c58993f8878012d548cc14:

GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=commit.gpgsign GIT_CONFIG_VALUE_0=false timeout --signal=TERM --kill-after=10s 120s bun run test ./packages/ui-server/tests/storage-shared-scan.test.ts

Exit 0; 7 pass / 0 fail / 38 assertions; 1.90 seconds, with no filtered or skipped cases. Both native child processes, complete cached-page identity and lease cleanup assertions pass. The ordinary repository invocation wrapper, network preload and original assertions/deadlines remain in force. The command-scoped Git setting isolates synthetic commits from ambient signing; no global Git setting changed. The outer watchdog did not fire. This does not capture the failed full-ordering acquisition/publication timeline or establish that its two leases overlapped. No full ui-server package run was performed. Retain needs: repro, original failed receipts and priority p2; no source repair, readiness or historical-cause claim follows.

Complete local log SHA-256: 45e04c8bbfd2b7de552e81d87981578d6ef5a61d4b7f02867b90eaac30820b4b. No source was changed for this control.

Controlled missing-overlap reproduction — 2026-10-29

At isolated main a8e80eea1ec045579ee7d69d6019bce5e78afa5e, Bun 1.4.4 with the same frozen lock and ordinary guarded test wrapper, a temporary control drives the unchanged original packages/ui-server/tests/fixtures/storage-shared-process.ts twice. Both native children print READY before either receives stdin. The parent releases only the first caller, awaits its successful exit and complete stdout, and verifies the shared lease has job_id: null and workers_json: "[]". Only then does it release the second caller. Both children exit 0, each performs one fresh collection, their complete page names match, and the final lease is again empty.

Keeping the original one-collection assertion in this schedule fails on STORAGE_SHARED_ONE_COLLECTION: expected one ID, received a different ID. Receipt: exit 1, 0 pass / 1 fail / 9 assertions, 358 ms test phase. Log SHA-256 7eb9a48081a8e0b1e6c956a3ea55958f071e7b1f4d38c2a0e1b1af8955e81ee4. A distinct serial-semantics control with the same schedule and the correct distinct-ID expectation passes: exit 0, 1 pass / 9 assertions, 366 ms; log SHA-256 488bedc9a57386a2ab60c73ad3de41d7986bc14e484ea5a7e868e14e140f4816. All children/output/database/temporary files are joined or cleaned in finally. No source was changed for either control.

This establishes causally that both processes being READY does not guarantee overlapping lease lifetimes, and that legitimate serial forced scans produce the exact ID-mismatch assertion. Source confirms that an active lease is joined while a fresh request after completion may allocate its own ID (packages/ui-server/src/storage/shared-scan-writes.ts, acquire; packages/ui-server/src/storage/coordinator.ts, start). The earlier unmodified full-file control is 7 pass / 38 assertions, not the deliberate negative control.

It does not identify the original package run's ordering or prove that production coalescing failed while leases overlapped. Those remain honest historical limits, rather than reasons to make a test rely on an unestablished premise. Scope is now the bounded native fixture-overlap repair above. Remove needs: repro and add agent-ready; preserve type test, area ui-server, priority p2 and no release commitment. No fixture or product repair is implemented by triage. The general verification epic #736 remains at its native direct-child limit, so no existing child is displaced to attach this issue.

Implementation handoff — 2026-10-29

Draft PR #3114, head eed1b0328789a6a5e085b74e214ea859883f5777, implements only the scoped fixture correction. The owner is held at a bounded, cancellable fixture barrier after acquiring its actual lease. The parent observes the owning PID and the follower's shared lease before releasing collection. Serial native forced scans correctly retain distinct IDs, and cancellation/timeout controls verify lease release. Production source and forced-scan semantics are unchanged.

Evidence at base a8e80eea1ec045579ee7d69d6019bce5e78afa5e, Bun 1.4.4, frozen dependencies:

• Original focused file: 7 pass, 38 assertions.
• Controlled nonoverlap mutation of the corrected positive test: both native children exit successfully; completing the first lease before starting the second fails specifically at STORAGE_SHARED_ONE_COLLECTION (0 pass, 1 fail, 7 assertions). Restored afterward.
• Restored complete focused file: 10 pass, 69 assertions, 12.49 seconds; no filtered or skipped cases. This includes the new serial, cancellation and actual bounded timeout controls.
• Final bun run check:pr --base origin/main: pass; 209 fast invariant tests, 1666 assertions; lint/leakage/environment/changeset gates pass.

All historical failed and passing package receipts above remain distinct and unchanged. This implementation does not identify the original failed package run's schedule.

Concurrency clarification (maintainer, 2026-10-29): this is a different machine. Use bun run test packages/ui-server with the repository's automatic checkout lock, and run heavy verification commands one at a time. The original machine's host-wide slot convention does not apply. Record the complete result and concurrency procedure used. This resolves the handoff prerequisite.

Resume and completion evidence: run the full package suite on the PR head under that procedure and retain its complete exit/count/failure receipt. Then obtain exact-candidate integration ownership and complete selected hosted confirmation before merging. Historical evidence remains unchanged.

Complete candidate package receipt — 2026-10-29

At PR #3114 head eed1b0328789a6a5e085b74e214ea859883f5777, based on a8e80eea1ec045579ee7d69d6019bce5e78afa5e, Bun 1.4.4 (38afee6c8), frozen dependencies and unchanged lock SHA-256 5bbd5f6651f111db3fcfabeda512d44a21163cba54c58993f8878012d548cc14:

bun run test packages/ui-server

Complete result: exit 1; 4395 pass / 2 fail; 31405 assertions; 4397 tests across 362 files; 1314.34 seconds (guarded invocation elapsed 1314839.744155 ms). No cases were filtered or skipped. The command ran once and was allowed to finish; no retry or guard, budget, assertion or deadline change was used to obtain a pass.

Concurrency procedure: this machine uses the repository's automatic checkout lock. The command printed Acquiring exclusive workspace output access; overlapping commands wait. No other heavy verification command was launched by this invocation during the full suite. Per maintainer clarification, the original machine's host-wide slot convention does not apply here.

All 10 cases in storage-shared-scan.test.ts passed inside this complete run, including actual owner/follower overlap, serial distinct forced scans, cancellation, timeout and lease cleanup. Both failures are in unchanged reference-transport.test.ts: the anonymous/private revision case (9.80 ms) and thousand-file acquisition case (60.90 ms). Both throw WorkerStateError at reserveMaintenanceTemp: a 256 MiB request sees one reservation holding the separate 256 MiB budget. This matches existing open #2859. No reservation owner/domain identity was captured at the failure; this receipt establishes recurrence, not its cause or repair.

Complete log SHA-256: 9a96848dc4720a46c1b942f62c9083f928fc9082df0c9db439aacc97f3dad941.

Blocked by #2859

The fixture implementation is preserved in draft #3114. Required complete-package proof is currently failing on the unrelated existing blocker; selected hosted candidate confirmation and merge have not been requested. Resume after #2859 is causally resolved: refresh the candidate if needed, obtain a passing complete package receipt under the recorded concurrency procedure, then run preflight and exact-candidate hosted confirmation under the integration reservation before merge. Keep the original historical receipts and this failed receipt distinct. Releasing claim 3013-overlap-20261029-r2; no human concurrency prerequisite remains.

Dependency delivered and resumed — 2026-10-29

#2859 was delivered by PR #3152, squash 76192ea488c96569a22a8b1ac87006940d0607e2, verified on origin/main with parent 286432d97e33a7b1d5faebbdcbfa030f5f9430d4 and tree a7c98747a3eaf6c7903acd2889143d5fffa769ae. Complete hosted candidate run 40938197657, attempt 1, passed and the shared integration command verified actual selected-job checkout identities before merging. Automatic main run 40941054676 is still in progress; it is not yet claimed green.

The blocker is closed. Resuming #3013 under claim 3013-overlap-20261029-r3, branch as/issue-3013, PR #3114. Refresh onto the delivered dependency, run focused/preflight/full-package verification sequentially under the automatic checkout lock, then qualify the exact current hosted candidate. All earlier failed/pass receipts retain their recorded identities.

Refreshed local verification — 2026-10-29

PR #3114 head 46aca2f2bc89909d58dbac6b2390823db7e35cd6 is based on delivered #2859 squash 76192ea488c96569a22a8b1ac87006940d0607e2. Focused shared-storage proof is 10 pass / 69 assertions; preflight is 209 pass / 1666 assertions. The complete package command ran once to completion under the automatic checkout lock, with heavy commands sequential: exit 1; 4411 pass / 1 fail; 31528 assertions; 362 files; 1288.39 seconds. All shared-storage and reference-transport cases passed. The sole failure is EDQUOT while copying a retained 100 MB turn-scope checkpoint onto the quota-constrained temporary filesystem. Full log SHA-256: 34ef7a2b1d136c5cb3b9e395041b5af2aa04a1b66e7daf88b13123d0bfdc8f3b.

Read-only inspection confirmed a per-user quota on that temporary filesystem, with less available capacity than filesystem-wide df reported. At the same unchanged head, the complete turn-scope file passes with an owned disk-backed TMPDIR with ample capacity: 15 pass / 173 assertions / 17.95 seconds, including the failed 100 MB case. Its root was removed after joined teardown. No source, assertion, guard, budget or deadline changed. Capacity-control log SHA-256: 9d5c6dad66a44e1d37aeae1b23c03866469cd23567359699f072ad121c387f55.

The failed complete local receipt and passing capacity control remain distinct. Required complete selected hosted confirmation will qualify the exact current head/base independently; no local full-green claim is made. #2859's automatic main run 40941054676, attempt 1, is now fully reconciled: all 16 successful jobs actually checked out its squash, every selected category passed, and packaging was the planner-intended test-only skip.