Repository navigation
refactor(ui-server): copy the turn data tree as a pool (#2891) - #2912
Merged
Merged
Conversation
Admission copied the startup view into the turn's data tree one file after another, synchronously, then the index seed: O(brain) on the event loop. It now plans directories and links in one walk, copies the files and the index seed sixteen at a time with fs/promises (requesting a reflink), closes directory modes and times last, and takes the admission's abort signal, removing whatever it made. Native Btrfs checkpoint into the server's temporary directory, warm: 2,000 notes 100 -> 51 ms, 10,000 notes 471 -> 245 ms; cold first copy 283 -> 59 ms and 1,234 -> 247 ms. Reflink placement on the managed volume and hardlinks were measured and not adopted (see the decision record). Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Resolves against #2910 (#2892): the decision record keeps its paragraph on what may differ from the view (now including image masks) whole, and the pooled-copy bullet follows it. #2920's new stdio-release tests await the now asynchronous createTurnDataTree. Co-Authored-By: Claude Opus 5.5 <[email protected]>
schlessera
marked this pull request as ready for review
October 25, 2026 22:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2891
What changed
Admission builds a strict, voice or autonomous turn's data tree about twice as fast.
createTurnDataTreeplans the tree in one walk (directories and links made, nothing else), then copies the view's files and the turn index seed sixteen at a time withfs/promisesoff the event loop, requesting a reflink (COPYFILE_FICLONE), and finally gives every directory its mode and times, deepest first. It is nowasyncand takes the admission's abort signal: an aborted admission stops the copy, waits for the copies already started, and removes the tree and its index. What the tree holds is unchanged (the view's bytes, links, modes and times, minus what it always left out) and so are its guarantees: read-only to the worker, works with a separate worker identity, nothing but the view and the turn's own effects ever enters it.Why this shape
Measured on the native fixture shape (an owned loop Btrfs, a read-only checkpoint snapshot, the hosted-app fixture's notes and a real
brain index), with the realcreateTurnDataTreeof origin/main and of this branch, as UID 1000 and as root, warm medians of five:Constructions considered:
trustedDirectory,packages/ui-sdk/src/server/native-brain-manager.ts:109-113), its private directories are root's, and the live subvolume is what governed peers write and every checkpoint qualifies, so a tree there could be written by a peer. A manager-provisioned, server-owned directory on the store would also move turn copies (and every index CoW page an update dirties) into the store's fixed capacity, which needs a ruling. The pool keepsCOPYFILE_FICLONE, so a tree placed on a reflinking filesystem still takes it; the pool is slower there than one file after another (table), which this PR accepts because no deployment can place it there today.EXDEV, A5 spike); from a shared server-owned base they would raise link counts, and readers refuse multiply linked files (checkGovernancePolicy, the embedding policy, storage-attachment facts). Not adopted.The decision record (
governed-writable-workers.md, "The copy runs as a pool (#2891)") keeps these figures and the residual: one copy per concurrent turn on the temporary directory.Proof
Native hosted cell (
bun scripts/prove-native-host.ts --backend=pi --app-uid=1000 --apply-timing, scratch worktrees,flockslot): admissionturn_tree_seed292 ms on origin/main @ 6814f2c → 84 ms on this branch; both cells pass (20/20 in-worker search, piread_file/shell see the edits).--backend=claude --app-uid=1000 --apply-timing: 241 ms → 100 ms, both pass (20/20 through the real Claude MCP, data root/tmp/brain-worker-data). Host load average 8-10 for all four runs.Full native proof on head
0f419a3dd(scratch worktree,flockslot):bun scripts/prove-native-host.ts(claude/pi × UID 1000/0, then the deliberate index-namespace negative control) pass, "all four healthy cells remain qualified";bun scripts/prove-worker-identity.ts(server 21001, worker 21002, separate-UID data tree,provider-index.native.tsmasked turns) pass, owned cleanup pass. The identity proof patchesfixture.pyby anchors this PR does not touch;hosted-app-child.tschanges only itscreateTurnDataTreetiming spy (nowasync).New tests (
packages/ui-server/tests/turn-data-tree.test.ts):AN_ABORTED_COPY_REFUSES(the old copy had no abort)Mutations (each breaks the code, the named test fails, restored):
finishdropped)THE_TREE_IS_THE_VIEW_ENTRY_FOR_ENTRYTHE_TREE_IS_THE_VIEW_ENTRY_FOR_ENTRY,AN_ABORTED_COPY_REFUSESTHE_TREE_IS_THE_VIEW_ENTRY_FOR_ENTRYTHE_SEED_COPIED_THE_INDEXAN_ABORTED_COPY_REFUSESAN_ABORTED_COPY_LEAVES_NOTHINGchmoddroppedcopyFilealready sets the source's mode (a0664file in the view keeps0664); thechmodis kept as the copy's own guaranteeSuites on head
0f419a3dd(base origin/main @6814f2ce5):bun run test packages/ui-server(rerun through the host's slot wrapper after a load spike turned earlier runs into 30 s timeout failures): 3983 pass, 0 fail (334 files).bun run test ./tests: 1954 tests across 177 files, exit 0 (the two(fail)lines in the log are the runner's own nested fixture runs).turn-data-tree.test.ts: 13 pass, 0 fail. Provider-history guards (packages/ui-backend-{claude,pi}/tests/provider-history.test.ts): 31 pass, 0 fail.tsc --noEmit,bun run lint: clean.bun run api-reportunchanged.On head
0f419a3dd:Local preflight passed(draft gates on GitHub: sync, draft gates, contract, changeset pass).Refresh onto main @
8344571c9(head2fdfbcf0b, merge commits, no force-push)Merged origin/main twice (
b722bca83, then8344571c9with #2910, #2920, #2933). Conflicts:turn-data-tree.test.ts(imports; main's #2917 cache expectations kept with this PR'sawait) and the decision record, where this PR's earlier edit had split the "every other byte in the tree equals the view" paragraph; it is whole again (now naming #2892's image masks) and the pooled-copy bullet follows it. #2920's newprivate-index-stdio-release.test.tsawaits the now asynchronouscreateTurnDataTree. #2920'srunOnTreestdio release and #2917's.brain-cache/handling merged without conflict.On
2fdfbcf0b:turn-data-tree,private-index-stdio-release,brain-mask-application,protected-host-writers,provider-mask-bridge,./tests/policy-write-boundary,./tests/decision-citations: 163 pass, 0 fail. Throughheavy.sh:bun run test packages/ui-server4051 pass, 0 fail (339 files);bun run test ./tests1959 tests across 178 files, exit 0;bun run check:pr --base origin/mainLocal preflight passed. Native undernative.lock(scratch worktree at2fdfbcf0b):prove-native-host.tsfull matrix pass;prove-worker-identity.tspass.tsc --noEmit,bun run lintclean.Checklist
turn-data-tree.test.ts, the native--apply-timingand identity cells green with lower admission time.0.44.0on the PR and ui-server: a strict turn's data tree copies the whole startup view at every admission #2891 (read back).turn-tree-pooled-copy.md(patch).createTurnDataTreeis internal; API reports unchanged).governed-writable-workers.md,agent-workers.md(cost figures).Anything left open
needs: decision).🤖 Generated with Claude Code