Skip to content

refactor(ui-server): copy the turn data tree as a pool (#2891) - #2912

Merged
schlessera merged 3 commits into
mainfrom
refactor/2891-turn-tree-copy-044
Oct 25, 2026
Merged

schlessera merged 3 commits into
mainfrom
refactor/2891-turn-tree-copy-044

Conversation

@schlessera

@schlessera schlessera commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Closes #2891

What changed

Admission builds a strict, voice or autonomous turn's data tree about twice as fast. createTurnDataTree plans the tree in one walk (directories and links made, nothing else), then copies the view's files and the turn index seed sixteen at a time with fs/promises off the event loop, requesting a reflink (COPYFILE_FICLONE), and finally gives every directory its mode and times, deepest first. It is now async and takes the admission's abort signal: an aborted admission stops the copy, waits for the copies already started, and removes the tree and its index. What the tree holds is unchanged (the view's bytes, links, modes and times, minus what it always left out) and so are its guarantees: read-only to the worker, works with a separate worker identity, nothing but the view and the turn's own effects ever enters it.

Why this shape

Measured on the native fixture shape (an owned loop Btrfs, a read-only checkpoint snapshot, the hosted-app fixture's notes and a real brain index), with the real createTurnDataTree of origin/main and of this branch, as UID 1000 and as root, warm medians of five:

Brain Placement origin/main pooled copy
2,000 × 16 KiB, 115 MiB index server tmp (other fs), UID 1000 100 ms (cold 283) 51 ms (cold 59)
same server tmp, root 97 ms 55 ms
10,000 × 16 KiB, 574 MiB index server tmp, UID 1000 471 ms (cold 1,234) 245 ms (cold 247)
same server tmp, root 455 ms 224 ms
2,000 same Btrfs as the view (reflink) 46 ms 59 ms
10,000 same Btrfs as the view (reflink) 202 ms 272 ms

Constructions considered:

  • Reflink placement on the managed volume (the maintainer's "reflink when available"). Bun's copy already reflinks there one file after another (the index seed drops from 30-150 ms to 0.1 ms), so it is the cheapest in time and the only one that would also cut disk use. Not adopted: the server owns no directory on that volume outside the live brain. The manager refuses a store root that is not root-owned and protected (trustedDirectory, packages/ui-sdk/src/server/native-brain-manager.ts:109-113), its private directories are root's, and the live subvolume is what governed peers write and every checkpoint qualifies, so a tree there could be written by a peer. A manager-provisioned, server-owned directory on the store would also move turn copies (and every index CoW page an update dirties) into the store's fixed capacity, which needs a ruling. The pool keeps COPYFILE_FICLONE, so a tree placed on a reflinking filesystem still takes it; the pool is slower there than one file after another (table), which this PR accepts because no deployment can place it there today.
  • Hardlinks from the checkpoint fail across subvolumes (EXDEV, A5 spike); from a shared server-owned base they would raise link counts, and readers refuse multiply linked files (checkGovernancePolicy, the embedding policy, storage-attachment facts). Not adopted.
  • The manager stage (write through the overlay where the server owns it): already ruled out across UIDs by ui-sdk: a strict worker never sees its own turn's host applications or their index update #2848's gating proof, and two mechanisms by identity mode would double the surface.

The decision record (governed-writable-workers.md, "The copy runs as a pool (#2891)") keeps these figures and the residual: one copy per concurrent turn on the temporary directory.

Proof

Native hosted cell (bun scripts/prove-native-host.ts --backend=pi --app-uid=1000 --apply-timing, scratch worktrees, flock slot): admission turn_tree_seed 292 ms on origin/main @ 6814f2c → 84 ms on this branch; both cells pass (20/20 in-worker search, pi read_file/shell see the edits).
--backend=claude --app-uid=1000 --apply-timing: 241 ms → 100 ms, both pass (20/20 through the real Claude MCP, data root /tmp/brain-worker-data). Host load average 8-10 for all four runs.

Full native proof on head 0f419a3dd (scratch worktree, flock slot): bun scripts/prove-native-host.ts (claude/pi × UID 1000/0, then the deliberate index-namespace negative control) pass, "all four healthy cells remain qualified"; bun scripts/prove-worker-identity.ts (server 21001, worker 21002, separate-UID data tree, provider-index.native.ts masked turns) pass, owned cleanup pass. The identity proof patches fixture.py by anchors this PR does not touch; hosted-app-child.ts changes only its createTurnDataTree timing spy (now async).

New tests (packages/ui-server/tests/turn-data-tree.test.ts):

Test On origin/main
the pooled copy is the view, entry for entry: bytes, links, modes and times, and the seeded index passes (it pins the old copy's fidelity)
an aborted admission stops the copy and leaves no tree or index behind fails AN_ABORTED_COPY_REFUSES (the old copy had no abort)

Mutations (each breaks the code, the named test fails, restored):

Mutation Fails
directories never get their mode and times (finish dropped) THE_TREE_IS_THE_VIEW_ENTRY_FOR_ENTRY
a pool worker takes one job only THE_TREE_IS_THE_VIEW_ENTRY_FOR_ENTRY, AN_ABORTED_COPY_REFUSES
file times are not kept THE_TREE_IS_THE_VIEW_ENTRY_FOR_ENTRY
the index is not seeded THE_SEED_COPIED_THE_INDEX
the copy ignores the abort AN_ABORTED_COPY_REFUSES
a failed copy is not removed AN_ABORTED_COPY_LEAVES_NOTHING
the file chmod dropped still passes: Bun's copyFile already sets the source's mode (a 0664 file in the view keeps 0664); the chmod is kept as the copy's own guarantee

Suites on head 0f419a3dd (base origin/main @ 6814f2ce5):

  • bun run test packages/ui-server (rerun through the host's slot wrapper after a load spike turned earlier runs into 30 s timeout failures): 3983 pass, 0 fail (334 files).
  • bun run test ./tests: 1954 tests across 177 files, exit 0 (the two (fail) lines in the log are the runner's own nested fixture runs).
  • turn-data-tree.test.ts: 13 pass, 0 fail. Provider-history guards (packages/ui-backend-{claude,pi}/tests/provider-history.test.ts): 31 pass, 0 fail.
  • tsc --noEmit, bun run lint: clean. bun run api-report unchanged.
bun run check:pr --base origin/main

On head 0f419a3dd: Local preflight passed (draft gates on GitHub: sync, draft gates, contract, changeset pass).

Refresh onto main @ 8344571c9 (head 2fdfbcf0b, merge commits, no force-push)

Merged origin/main twice (b722bca83, then 8344571c9 with #2910, #2920, #2933). Conflicts: turn-data-tree.test.ts (imports; main's #2917 cache expectations kept with this PR's await) and the decision record, where this PR's earlier edit had split the "every other byte in the tree equals the view" paragraph; it is whole again (now naming #2892's image masks) and the pooled-copy bullet follows it. #2920's new private-index-stdio-release.test.ts awaits the now asynchronous createTurnDataTree. #2920's runOnTree stdio release and #2917's .brain-cache/ handling merged without conflict.

On 2fdfbcf0b: turn-data-tree, private-index-stdio-release, brain-mask-application, protected-host-writers, provider-mask-bridge, ./tests/policy-write-boundary, ./tests/decision-citations: 163 pass, 0 fail. Through heavy.sh: bun run test packages/ui-server 4051 pass, 0 fail (339 files); bun run test ./tests 1959 tests across 178 files, exit 0; bun run check:pr --base origin/main Local preflight passed. Native under native.lock (scratch worktree at 2fdfbcf0b): prove-native-host.ts full matrix pass; prove-worker-identity.ts pass. tsc --noEmit, bun run lint clean.

Checklist

  • Focused local preflight/behavioral receipts passed; hosted proof is the coordinator's.
  • Acceptance criteria: measured comparison on the native fixture (UID 1000 and root; separate-UID through the identity proof below) of the current copy against cheaper constructions, receipts above; the adopted construction keeps turn-data-tree.test.ts, the native --apply-timing and identity cells green with lower admission time.
  • Milestone 0.44.0 on the PR and ui-server: a strict turn's data tree copies the whole startup view at every admission #2891 (read back).
  • Changeset turn-tree-pooled-copy.md (patch).
  • No personal data; Odysseus fixtures only.
  • Nothing in the machine surface moved (createTurnDataTree is internal; API reports unchanged).
  • Docs: governed-writable-workers.md, agent-workers.md (cost figures).
  • Provider history: nothing on the provider-facing path changes; guard passes (above).

Anything left open

🤖 Generated with Claude Code

Admission copied the startup view into the turn's data tree one file
after another, synchronously, then the index seed: O(brain) on the event
loop. It now plans directories and links in one walk, copies the files
and the index seed sixteen at a time with fs/promises (requesting a
reflink), closes directory modes and times last, and takes the
admission's abort signal, removing whatever it made.

Native Btrfs checkpoint into the server's temporary directory, warm:
2,000 notes 100 -> 51 ms, 10,000 notes 471 -> 245 ms; cold first copy
283 -> 59 ms and 1,234 -> 247 ms. Reflink placement on the managed
volume and hardlinks were measured and not adopted (see the decision
record).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@schlessera schlessera added this to the 0.44.0 milestone Oct 25, 2026
@schlessera schlessera added type: refactor Structure changes; behaviour does not. No changeset-visible effect. area: ui-server Hono app factory: turn coordinator, auth, session catalog, routes, cron. labels Oct 25, 2026
schlessera and others added 2 commits October 25, 2026 21:06
Resolves against #2910 (#2892): the decision record keeps its paragraph
on what may differ from the view (now including image masks) whole,
and the pooled-copy bullet follows it. #2920's new stdio-release tests
await the now asynchronous createTurnDataTree.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@schlessera
schlessera marked this pull request as ready for review October 25, 2026 22:00
@schlessera
schlessera merged commit c670449 into main Oct 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ui-server Hono app factory: turn coordinator, auth, session catalog, routes, cron. type: refactor Structure changes; behaviour does not. No changeset-visible effect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ui-server: a strict turn's data tree copies the whole startup view at every admission

1 participant