Repository navigation
ui-server: synchronous SQLite fsync commits stall the hosted event loop during turn admission #2789
Description
Context
Attribution for #2416 (HOSTED_ADMISSION_HEALTH_WITHIN_1_SECOND) found two classes of synchronous work that block the hosted server's event loop during turn admission and settlement. While either runs, /api/health and WebSocket pings wait. The class covered by this issue is synchronous SQLite commits that wait on fsync, on the event loop. (The other class, synchronous worker subprocesses, is moved off the loop by the PR that Refs #2416.)
The method: the native hosted proof's app child ran under bun --cpu-prof with a 20 ms timer-drift probe. Each lag window of at least 100 ms was attributed to the categorized frames sampled inside it. The same pi UID 1000 cell ran with no load, under CPU quotas (1, 0.5), on pinned contended cores, and with three dd … conv=fsync writers on the same filesystem as the app database.
| Condition | Journal SQLite (journal-accounting write) |
Maintenance temp budget DB | Activity SQLite |
|---|---|---|---|
| fsync load (3 writers) | 4,178 ms, 14 windows, max 604 ms | 1,966 ms, 7 windows, max 478 ms | 1,578 ms, 7 windows, max 552 ms |
| 1 CPU quota | 1,364 ms, 6 windows, max 274 ms | — | 548 ms, 3 windows, max 239 ms |
| 0.5 CPU quota | 904 ms, 6 windows, max 201 ms | — | 723 ms, 4 windows, max 213 ms |
On this host a 4 KiB fsync takes at most 25 ms when idle and up to 850 ms under the same write load. Lag windows in these frames held little CPU (for example 5 ms of samples in a 332 ms window), which is consistent with waiting in fsync.
The retained native failures put about half the real 1–1.5 s stalls in this class:
- a pi UID 1000 run whose
journal.finishtook 1,687 ms, with a 1,489 ms gap in WebSocket pongs; - a pi UID 0 run with a 1,380 ms gap during
journal.begin; - the pi UID 0 retained-index control (the tests: attribute recurring one-second native hosted health timeouts #2416 reopen evidence), with a 1,315 ms gap right after the automatic index started. Its maintenance worker preparation opens the temp budget DB.
The other half were worker starts, which the #2416 PR addresses.
Where the waits are:
- (
const write = <T>,packages/ui-server/src/brain/journal-accounting.ts:57). Every journal write setsPRAGMA synchronous = FULL(line 61) and runsPRAGMA wal_checkpoint(TRUNCATE)(line 66). It then runs a fulldbstatscan (function measure(), line 30, scan at line 40) at least twice inside an immediate transaction. Turn metadata updates such as (bindSession(turnId,packages/ui-server/src/brain/turn-journal.ts:357) andbindModelgo through it, as dobegin(line 323) andfinish(line 399). - (
export function reserveMaintenanceTemp,packages/ui-sdk/src/server/maintenance-temp-budget.ts:136) and itsrelease()(line 157). Each one opens the shared budget database next to the UI database (function open(, line 79), commits an immediate transaction and closes it. Every maintenance worker does this twice. - Activity lifecycle writes (
writeLifecycle(record),packages/ui-server/src/activity/store.ts:654).
Scope
Decide, then implement, one of:
- Accounting redesign. Keep the journal's capacity accounting (CONTRACT: feat(storage): deliver owner inventory, recovery and capacity guidance #2172) exact while taking the checkpoint,
FULLsync anddbstatscans off the per-write path. For example, account for metadata-only updates without a checkpoint, or batch accounting per observation window. The maintenance budget ledger should hold its database open, or batch, rather than open, commit and close per reservation. - SQLite off the main thread. Run the journal, activity and budget writers in a worker thread or process. The server's event loop then awaits commits instead of blocking on them, keeping the same transaction order and refusal semantics.
Either option keeps today's durability and capacity guarantees. The ruling goes on this issue before implementation.
Out of scope
- Raising
HOSTED_ADMISSION_HEALTH_WITHIN_1_SECONDor any other bound. - Synchronous worker subprocess waits: probe, sandbox git check, launch. Those are tests: attribute recurring one-second native hosted health timeouts #2416's PR and ui-sdk: remaining synchronous worker probes and launches block the hosted event loop #2790.
- Changing what the journal records or how its capacity is reported to users.
Acceptance criteria
- A maintainer ruling selects option 1 or 2 (or another shape), recorded here.
- Under the same fsync-load condition (three
dd … conv=fsyncwriters on the app database's filesystem), the native pi UID 1000 cell shows no event-loop lag window over 250 ms attributed to journal, budget or activity SQLite. This uses the same timer-drift probe and CPU-profile attribution, and the before and after receipts go in the PR. - Journal capacity accounting stays exact: the existing CONTRACT: feat(storage): deliver owner inventory, recovery and capacity guidance #2172 accounting tests pass unchanged, and a new test fails if a write escapes accounting.
- The native hosted proof passes
HOSTED_ADMISSION_HEALTH_WITHIN_1_SECONDwith the bound unchanged, under the flock, at least twice.
Verification
bun run test packages/ui-server packages/ui-sdk, plus the native hosted proof under the exclusive fixture lock, with and without the fsync-load condition. The attribution method is described in the #2416 attribution comment.
Contract impact
Option 1 may touch journal capacity accounting, which is covered by the storage contract component. That needs a check against docs/integration-contract.md before implementation. Option 2 has no wire or CLI contract impact.
2026-10-24: Option 1 is partly delivered in draft #2812 (Refs). The fsync-load acceptance criterion is still unmet: the remaining windows are the synchronous=FULL commit fsync itself. needs: decision on how that commit leaves the event loop; the options are in #2789 (comment).
2026-10-24: the maintainer rulings change this issue's scope. (1) The commit fsync leaves the event loop under #2814, which now carries the fsync-load lag criterion. (2) Here: no per-write dbstat walk at all, not even scoped to journal b-trees; journal capacity accounting must keep #2172's page-identity guarantee, proven by the page-identity oracle test and a mutation table, or the issue must show with evidence that this is impossible. #2812 is the first step (Refs); the design without a walk follows in its own PR.
2026-10-24: ruling 2 found no sound walk-free design in the shared UI database; evidence and options (dedicated journal database, a documented capacity-semantics change, or keeping the journal-scoped walk) are in #2789 (comment). needs: decision.
Activity
schlessera commentedon Oct 24, 2026
Maintainer ruling (2026-10-24): option 1, accounting redesign. Keep the journal capacity accounting (#2172) exact while taking the
wal_checkpoint(TRUNCATE),synchronous=FULLanddbstatscans off the per-write path, for example by accounting metadata-only updates without a checkpoint or batching per observation window. The maintenance budget ledger should hold its database open rather than opening, committing and closing per reserve/release. Durability and capacity guarantees stay as they are. Check the storage contract component indocs/integration-contract.mdbefore implementing and useCONTRACT:for any surface change.schlessera commentedon Oct 24, 2026
Claim
bk044-2789-1792825765: implementing the option 1 accounting redesign on branchfix/2789-sqlite-accounting-off-write-path-044.schlessera commentedon Oct 24, 2026
Handoff (2026-10-24): option 1 is implemented as far as it goes. The fsync-load criterion needs a decision.
Draft PR: #2812 (
Refs).What it delivers. A journal write no longer runs
wal_checkpoint(TRUNCATE). A ledger truncates once at construction, then charges the WAL file as it stands and truncates only when that would refuse admission. The measurement walks only the journal's own b-trees. The charge never falls below #2172's page-identity measurement; a page-identity oracle test and a mutation table back that. The #2172 tests pass unchanged. A connection-local guard refuses journal rows changed outside accounting. Commits staysynchronous=FULL. The maintenance budget ledger stays open between reservations; its four windows ≥ 250 ms before the change dropped to zero or one weakly attributed window per run.What it does not deliver. In each run, the native pi UID 1000 cell, loaded with three
dd … conv=fsyncwriters and measured with the #2416 method, still shows 2–3 windows ≥ 250 ms for the journal and 2–3 for Activity. I measured three accounting shapes, two runs each; the full table is in the PR.dbstat: 0 journal windows, but unsound. Independent review reproduced journal pages being discharged while still free, governance-expiry deletes inside journal transactions driving owned pages negative, and lost conversion pages.dbstatwalk itself waits on I/O under this load.The remaining windows hold 2–16 ms of CPU samples in 230–590 ms. The blocked time is unsampled, consistent with each
synchronous=FULLcommit's fsync. That holds for the journal, for Activity (writeLifecycle,packages/ui-server/src/activity/store.ts:654), and for the native lease DB. So the criterion fails on the durable commit itself, not on accounting. Option 1 cannot remove that without changing durability.Question for the maintainer: how should the commit fsync leave the event loop?
FULL, transaction order and refusal semantics. It is the only option that meets the criterion with durability unchanged. Cost:createTurnJournaland the Activity store expose synchronous APIs today, so their callers become asynchronous.synchronous=NORMAL, the journal staysFULL. Activity rows (lifecycle, spans, cost rollups and budget settlement) could lose their last commits on an OS crash or power loss; a process crash loses nothing. The journal's commit fsync stays on the loop, so its windows likely remain.HOSTED_ADMISSION_HEALTH_WITHIN_1_SECONDitself is unchanged and passes (receipts in the PR).Recommendation (not a ruling): 1, as its own issue. The PR can merge first as an independent step: it removes the per-write checkpoint and full-database walk, and the budget open/close.
A per-write
dbstatwalk remains, scoped to journal b-trees. The ruling asked for those scans off the per-write path too; counts alone cannot keep the page-identity rule (above). Whether a journal-scoped walk is acceptable is part of this decision.Next action: a maintainer ruling on 1–3. Resume from the PR's branch. Completion evidence: the same before/after runs in the PR body, with no window ≥ 250 ms attributed to journal, budget or Activity SQLite.
3 remaining items
schlessera commentedon Oct 24, 2026
Claim
bk044-2789-1792835940: ruling 2 (accounting without a per-write walk) on a follow-up branchfix/2789-walkless-journal-ledger-044stacked on #2812. Ruling 1 is filed as #2814.fix(ui-server): take the WAL checkpoint off the journal write path (#…
schlessera commentedon Oct 24, 2026
Ruling 2 report (2026-10-24): I found no sound accounting design without a per-write walk inside the current single-database layout. Evidence and options follow.
I built the closest design and pushed it as evidence on branch
fix/2789-walkless-journal-ledger-044(e58107654, stacked on #2812). It is not proposed for merge. On an ordinary write it walks no b-tree:mxFrame, salts) in-shm.data_version), on a schema or shrink change, and before any refusal.UPDATE/DELETEinside a journal write, and journal governance events now expire after the commit.An independent gpt-6-astra review then broke it; I reproduced the central case myself:
bun:sqliteexposes no per-transaction dirty-page or allocation count: nosqlite3_db_status, nowal_hook, and a commit's frames exist only after it commits. So admission can only bound the pages a commit will own. The bound must include pages freed earlier in the same transaction by other tables, and those frees are invisible:INSERT OR REPLACEinside an accounted write freed 97 pages. REPLACE's implicit delete bypasses DELETE triggers unlessrecursive_triggersis on, as SQLite documents.WITHOUT ROWIDb-tree freeing a page that the journal then reused.steering_change,roadblock), so they stay inside journal transactions.auto_vacuummaintains a pointer map. A page that is free now cannot be classified, after the fact, as formerly journal or formerly foreign. Any walk-free history therefore over-approximates. It charges foreign pages a journal commit happened to dirty, such as governance pages and foreign free-list trunks, that later become free. That moves refusals, and soretain()/collectOne()expiry, earlier than CONTRACT: feat(storage): deliver owner inventory, recovery and capacity guidance #2172.sqlite_sequencepages thatwalCapacitydoes not count. The reviewer measured 249 changed pages against a 77-frame budget for 500 journal changes with their governance events. This is inherited from CONTRACT: feat(storage): deliver owner inventory, recovery and capacity guidance #2172; walking only journal ownership does not fix it.Two more ordering bugs in the branch are fixable but moot given 1–2. The invalidation walk ran after the callback, so a deleting callback could erase identities first. A thrown post-commit file read did not invalidate tracking.
Options:
journal.dbbesideDB_PATH,ATTACHed to the UI connection). Every page in it is the journal's, and every free page is one the journal released. Page identity becomes exact by construction, with O(1) accounting:page_count,freelist_countand its own WAL size. No walk, ever. It fits ui-server: journal and Activity SQLite commits block the hosted event loop on fsync #2814: that writer thread or process can own the file. Cost: a migration that moves theturn_journal_*tables, and cross-file atomicity. WAL-mode commits spanning attached databases are atomic per file, not across them. The fix: the journal stays authoritative, and missingsteering_change/roadblockevents are replayed from journal rows at startup (their dedupe keys,turnId:sequenceandjournal-reader:…, already make replay idempotent). One semantic difference: a journal-released page stays charged until the journal reuses it, whereas today another component's reuse discharges it.Refs). Its cost under fsync load is the journal's own pages; ui-server: journal and Activity SQLite commits block the hosted event loop on fsync #2814 takes it off the event loop with the commit.Recommendation (not a ruling): A, sequenced with #2814: it removes the walk, the ownership question and the shared-WAL attribution together. Until then, C stands.
Next action: a maintainer ruling on A, B or C. Completion evidence for A or B: #2172's tests unchanged, the page-identity oracle, a mutation table, and the REPLACE/balancing cases above as tests.
schlessera commentedon Oct 24, 2026
Maintainer ruling (2026-10-24): option A, a dedicated journal database. The journal moves into its own
journal.db, next toDB_PATHandATTACHed to the UI connection. Every page in it belongs to the journal, so capacity accounting is exact by construction:page_count,freelist_countand its own WAL size, with no walk at all. Sequence it with #2814: the writer thread or process that #2814 introduces ownsjournal.db. Moving the journal tables needs a migration, with its proof. Until this lands, #2812's journal-scoped walk stays as it is.CONTRACT: docs: journal and Activity write surfaces under ruling 1 on #…
fix(ui-server): give the turn journal its own database file
fix(ui-server): the write worker owns journal.db beside the UI databa…
CONTRACT: fix(ui-server): commit journal, Activity and native-lease S…