diff --git a/.changeset/governance-scope-off-event-loop.md b/.changeset/governance-scope-off-event-loop.md new file mode 100644 index 0000000000..0ab2496b5c --- /dev/null +++ b/.changeset/governance-scope-off-event-loop.md @@ -0,0 +1,7 @@ +--- +"@schlessera/brain-ui-server": patch +"@schlessera/brain-ui-sdk": patch +--- + +Changed: the hosted server commits governance events raised outside a journal transaction, the session's first taint, roadblock records and turn scope rows on its SQLite write worker at `synchronous=FULL`, in submission order with journal and Activity writes, instead of on the event loop (#2850). A turn's egress connection sends its first byte only after its taint has committed, and admission counts that taint from the moment it is submitted. +Changed: the egress proxy awaits a promise returned from its `admitted` hook before the connection's first upstream byte. diff --git a/docs/decisions/answer-delivery.md b/docs/decisions/answer-delivery.md index 5edc7711c4..33518b7313 100644 --- a/docs/decisions/answer-delivery.md +++ b/docs/decisions/answer-delivery.md @@ -82,7 +82,7 @@ carries, no longer by position. The host also re-sends a session's pending questions after `session_resume` (`resendPendingAsks(host, ws, msg.sessionId)`, -`packages/ui-server/src/ws/dispatch.ts:640`). Before this, a resume replaced +`packages/ui-server/src/ws/dispatch.ts:641`). Before this, a resume replaced the transcript and left only a history card behind, which was the second symptom. diff --git a/docs/decisions/async-collaboration.md b/docs/decisions/async-collaboration.md index 8a60060f99..ea6afd8ed6 100644 --- a/docs/decisions/async-collaboration.md +++ b/docs/decisions/async-collaboration.md @@ -241,7 +241,7 @@ suppressed-per-model Action; unknown usage cannot become a guessed zero. Subscription billing moves the turn counter even when spend is zero. The activity sum exposes unpriced runs (`sumEffectiveCost`, `packages/ui-server/src/activity/store.ts:179-194`) and settles only after execution -(`store.rollupRun(runId);`, `packages/ui-server/src/activity/recorder.ts:723`). +(`store.rollupRun(runId);`, `packages/ui-server/src/activity/recorder.ts:746`). Retrospective totals alone cannot enforce admission. The maintainer's [2026-10-01 ruling on #678](https://github.com/schlessera/brain-kit/issues/678#issuecomment-5926691626) diff --git a/docs/decisions/claude-code-runtime.md b/docs/decisions/claude-code-runtime.md index 85631421eb..520caa983e 100644 --- a/docs/decisions/claude-code-runtime.md +++ b/docs/decisions/claude-code-runtime.md @@ -64,7 +64,7 @@ next to it. of this binary. The only version probe in the server is for the `brain` CLI (`Probe the selected content CLI`, `packages/ui-server/src/brain/client.ts:142-210`, called at - `probeBrainCliVersion(config.brainPath`, `packages/ui-server/src/app.ts:434`). + `probeBrainCliVersion(config.brainPath`, `packages/ui-server/src/app.ts:433`). `brain doctor` runs `claude mcp list` from `PATH` (`which("claude")`, `packages/core/src/cli/commands/doctor.ts:501-505`) — the user's own Claude Code on their own machine, to check the MCP registration, not the server's @@ -212,7 +212,7 @@ unnecessary. `node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs`. The boot probe below turns either into a refusal to start, for the same reason a missing backend refuses to boot (`A missing (or unrecognized) agent backend`, - `packages/ui-server/src/app.ts:372-377`). + `packages/ui-server/src/app.ts:371-376`). - **The published range is still a caret, and that bounds what this repo can guarantee.** `@schlessera/brain-backend-claude` depends on `^0.3.241`, and a host resolves it in its own lockfile. A host can bump the SDK — and so the diff --git a/docs/decisions/governed-writable-workers.md b/docs/decisions/governed-writable-workers.md index 93dde40c52..e0d283493f 100644 --- a/docs/decisions/governed-writable-workers.md +++ b/docs/decisions/governed-writable-workers.md @@ -1072,7 +1072,7 @@ principal authority remain the caller's responsibility. The policy publisher creates, writes, fsyncs and removes its operation-owned staging name entirely inside that synchronous callback, between the two serialized observations (`await options.observeUserPolicy`, -`packages/ui-server/src/brain/policy-proposal.ts:197-224`). First publication +`packages/ui-server/src/brain/policy-proposal.ts:198-225`). First publication also removes the temporary hardlink before the final observation. No filename prefix is ignored; a similarly named real agent change is still retained. Failed staging preserves the original policy, while failed recording after @@ -1334,7 +1334,7 @@ queues the capture (`function held`, `packages/ui-server/src/brain/turn-journal.ts:191-210`), and the scope monitor's hold freezes every active turn's productive workers until the capture completes (`before() {`, -`packages/ui-server/src/brain/turn-scope.ts:134-142`). Pi's agent loop runs in +`packages/ui-server/src/brain/turn-scope.ts:158-166`). Pi's agent loop runs in its worker, so a frozen worker cannot send its next provider request until the resume. On the native fixture the gap between a result and the next request tracked the hold, not the bridge call. @@ -1443,7 +1443,7 @@ unmet rather than relaxed. **Why.** Every governed turn queues an automatic index refresh after it (`host.governedTurns?.refresh(host.workerLock)`, -`packages/ui-server/src/ws/run-session.ts:899`). The refresh usually finishes +`packages/ui-server/src/ws/run-session.ts:909`). The refresh usually finishes while the next turn runs, so that turn's finish observation found a rewritten `brain.db` and copied a whole new version of it into retained objects, inside the observation's worker hold. #2831 measured 36 versions of a ~120 MB index in @@ -1484,10 +1484,10 @@ timestamps only, with no review action. Preview reports `metadata_only` and download has nothing to serve. History revert and Undo read that same presentation and exclude the family as `metadata_only`, so a version retained by an earlier release is not restored either -(`reason = "metadata_only"`, `packages/ui-server/src/brain/turn-revert.ts:43`). +(`reason = "metadata_only"`, `packages/ui-server/src/brain/turn-revert.ts:46`). The operation's ordinary index refresh rebuilds it, so that exclusion alone does not make the operation partial -(`const partial`, `packages/ui-server/src/brain/turn-revert.ts:145`). Existing +(`const partial`, `packages/ui-server/src/brain/turn-revert.ts:148`). Existing retained history is not erased; ordinary bounded retention expires it. **Measured.** `scripts/prove-native-host.ts --backend=pi --app-uid=1000 @@ -1531,7 +1531,7 @@ The brain's git history stays the durable record of its content. - **The Activity recorder and its rollups** price those events under the [cost-tracking rules](cost-tracking.md). An unknown price is never counted as zero. The session total that counts unknown as zero - (`msg.costUsd ?? 0`, `packages/ui-server/src/ws/session-catalog.ts:364`) + (`msg.costUsd ?? 0`, `packages/ui-server/src/ws/session-catalog.ts:368`) must not feed any budget. **Enforcement:** @@ -1724,7 +1724,7 @@ limits from the same policy inputs and states independent hard limits. **What exists.** The audit trail is the Activity span record: tool name, the full input up to its cap, output and approval decisions -(`"tool_input"`, `packages/ui-server/src/activity/recorder.ts:313`). +(`"tool_input"`, `packages/ui-server/src/activity/recorder.ts:334`). **What changes:** @@ -1805,7 +1805,7 @@ reports which applies as `governed.egress`. Making the relay work without managed storage is follow-up work. Each connection is recorded as a governance event, and a recording failure ends the turn. Without a governance recorder the policy still applies and only the records are absent (`prepareTurnEgress`, -`packages/ui-server/src/brain/turn-egress.ts:59-135`). The note says egress is +`packages/ui-server/src/brain/turn-egress.ts:60-166`). The note says egress is recorded only when the backend actually routed the worker through the proxy. API-key Claude profiles (2026-10-22) and pi turns (2026-10-23) are routed too. A pi turn whose credential the relay cannot carry is refused rather than @@ -2496,7 +2496,7 @@ every turn without a proven recipient: provenance shows every item and checkpoint of its origin thread compatible with those masks, checked once the masks are exact and before any thread text is read into the prompt (`admission = await admitOriginThread`, - `packages/ui-server/src/inbox/autonomous-turn.ts:373`). The dispatcher's + `packages/ui-server/src/inbox/autonomous-turn.ts:377`). The dispatcher's context is built afterwards, from the admission's arrival read and in the transaction that revalidates it, and both reads revalidate in their own transaction before building any payload (`function authorizeRead`, @@ -3310,7 +3310,7 @@ outward host skips the card only while every classified call in progress is a visible read, or an approved outward action that names that host. A GitHub write names the GitHub API hosts, and a push or HTTP write names its target's host. Any unknown or opaque call in progress withholds the exception -(`createOutwardCalls`, `packages/ui-server/src/brain/turn-egress.ts:35-57`). +(`createOutwardCalls`, `packages/ui-server/src/brain/turn-egress.ts:36-58`). A background process left by an earlier call can still connect inside such a window; the proxy records it either way. @@ -3324,7 +3324,7 @@ listen on other ports: `gmail.googleapis.com`, `graph.microsoft.com`, `smtp.sendgrid.net`, `api.mailgun.net`, `api.eu.mailgun.net`, `smtp.mailgun.org`, `smtp.eu.mailgun.org`, `api.postmarkapp.com`, `smtp.postmarkapp.com`, `api.resend.com` and `smtp.resend.com` -(`MAIL_SUBMISSION_PORTS`, `packages/ui-server/src/brain/turn-egress.ts:16-19`). +(`MAIL_SUBMISSION_PORTS`, `packages/ui-server/src/brain/turn-egress.ts:17-20`). Microsoft Graph and Exchange Online carry more than mail, so their reads by an opaque script also wait on the card. The coordinator chose this list for #1665; the maintainer may override it. @@ -3374,7 +3374,12 @@ server", following the rule above that erring towards taint costs only tighter rules (maintainer ruling, 2026-10-22). The first source is the session's one `taint` event. A write that fails stops the turn rather than leaving its outward rules loose. The check and write share a SQLite transaction -(`recordFirstTaint`, `packages/ui-server/src/governance/session-taint.ts:17-22`). +(`recordFirstTaint`, `packages/ui-server/src/governance/session-taint.ts:17-23`). +The hosted server runs that transaction on its write worker +(`insertFirstTaint`, `packages/ui-server/src/governance/session-taint.ts:29-35`; +[#2850](https://github.com/schlessera/brain-kit/issues/2850)). A taint counts +for admission from the moment it is submitted, and a connection's first byte +waits for its taint's commit. The admitting host hands both backends a live read of that record with the turn's command policy. While it reports a taint, no `outward.allow` rule @@ -3395,7 +3400,7 @@ session's proxies already admitted (a record with charged bytes; the proxy charges only after admission); otherwise it is refused and recorded as an `egress_denial` with reason `taint_allowlist`. The taint list narrows and never widens: deny and `allowlist` mode still apply first (`prepareTurnEgress`, -`packages/ui-server/src/brain/turn-egress.ts:59-135`). Hosts without managed +`packages/ui-server/src/brain/turn-egress.ts:60-166`). Hosts without managed native storage have no proxy, so the Bash classifier remains their shell rule and nothing narrows their egress. @@ -3543,7 +3548,7 @@ providers or a claim of capturing intervening external writes. The copy-out boundary is `const write`, `packages/ui-backend-claude/src/worker-state.ts:210-224`; retention/finalization is `persistClaudeMemory`, `packages/ui-server/src/brain/turn-journal.ts:524-554`; destination-bound recovery -is `const target`, `packages/ui-server/src/brain/turn-revert.ts:110-120`. +is `const target`, `packages/ui-server/src/brain/turn-revert.ts:113-123`. ### Concurrent edits @@ -3576,7 +3581,7 @@ exclusive "mine" or "theirs" ownership. A conflict requires two distinct observed after-versions sharing an overlapping turn; one version remains shared attribution. User choices use the existing authenticated history operation, comparing current raw state with the reviewed latest observation before effects -(`resolveSelection`, `packages/ui-server/src/brain/turn-revert.ts:52-75`). +(`resolveSelection`, `packages/ui-server/src/brain/turn-revert.ts:55-78`). Keep both creates the other version beside the selected file with sync's numbered `-remote` convention, using exclusive descriptor-rooted creation, existing CAS leases and explicit coverage for its final generated name diff --git a/docs/decisions/managed-worktrees.md b/docs/decisions/managed-worktrees.md index 478bafa431..589db92a76 100644 --- a/docs/decisions/managed-worktrees.md +++ b/docs/decisions/managed-worktrees.md @@ -97,7 +97,7 @@ unknown; what happens next is in [hygiene and cleanup](#hygiene-and-cleanup). Each governed or strict turn whose journal is recording opens one worker generation, and every worker of that turn binds the generation's root-owned directory whole at `/tmp/brain-worktrees`, inside the turn's own `/tmp` -(`// Managed worktrees (#2430): a journaled turn's workers bind one empty generation.`, `packages/ui-server/src/ws/run-session.ts:565-577`; +(`// Managed worktrees (#2430): a journaled turn's workers bind one empty generation.`, `packages/ui-server/src/ws/run-session.ts:566-578`; `// Managed worktrees (#2430): the whole access generation`, `packages/ui-sdk/src/server/worker-launcher.ts:848-859`). A granted worktree appears there as `/tmp/brain-worktrees//` on the worker's next path lookup; the worker cannot create entries beside it, and a @@ -199,7 +199,7 @@ The host side is `createReferenceTools` start, the host records its principal and what those workers bind: the public view, the principal's private view and the turn's worker generation, if one opened -(`// Reference tools (#2432): the turn's workers bind the principal's source`, `packages/ui-server/src/ws/run-session.ts:578-584`). +(`// Reference tools (#2432): the turn's workers bind the principal's source`, `packages/ui-server/src/ws/run-session.ts:579-585`). A call carries only a repository, a revision or a worktree identity. The bridge adds the turn's own principal, and a call for a turn the host did not prepare, or under another principal, refuses diff --git a/docs/decisions/scheduled-tasks.md b/docs/decisions/scheduled-tasks.md index b9f865c0b2..6eebc0737f 100644 --- a/docs/decisions/scheduled-tasks.md +++ b/docs/decisions/scheduled-tasks.md @@ -23,11 +23,11 @@ Source inspected at `b5c30c180df5584584f261b446c4d440380d2441`: (`else if (item.status === "scheduled"`, `packages/ui-server/src/inbox/actions.ts:196-197`). The runtime ticks every minute (`INBOX_TICK_MS`, `packages/ui-server/src/inbox/runtime.ts:11-13`). - Production app wiring supplies recovery and budgets, without a dispatcher - (`// Recovery/heartbeat only.`, `packages/ui-server/src/app.ts:491-507`). + (`// Recovery/heartbeat only.`, `packages/ui-server/src/app.ts:490-506`). A scripted nonempty ready-item test still reports no claim, attempt or Activity root. That proves disabled wiring, not safe execution of a schedule. - Headless plumbing checks backend support, a usable principal, lifetime and - reservation ownership (`runAutonomousTurn`, `packages/ui-server/src/inbox/autonomous-turn.ts:111-148`). + reservation ownership (`runAutonomousTurn`, `packages/ui-server/src/inbox/autonomous-turn.ts:112-149`). Capability flags and these checks do not prove credential/configuration, filesystem or egress containment at the actual backend boundary. - The generic core runner has a different interface diff --git a/docs/decisions/session-principals.md b/docs/decisions/session-principals.md index 820f6425ce..f3b4fd3bc2 100644 --- a/docs/decisions/session-principals.md +++ b/docs/decisions/session-principals.md @@ -102,7 +102,7 @@ Both reviewers verified every row below against the source. | WS upgrade | `ws/connection.ts:262-264` | `createWsUpgrade` ignores the request context | | WS admission | `ws/connection.ts:84` | `clients.add(ws)` — the socket's identity is unknown | | Turn record | `ws/run-session.ts:136-144` | recorder built from `{turnId, sessionId, billing}`; `RunningTurn` (`ws/turns.ts:55-77`) has no actor | -| Follow-up queue | `QueuedFollowUp`, `ws/turns.ts:61-103`; `slot.queue.push(entry)`, `run-session.ts:1090` | queued entries re-mint `turnId` in the same slot and may come from another socket | +| Follow-up queue | `QueuedFollowUp`, `ws/turns.ts:61-103`; `slot.queue.push(entry)`, `run-session.ts:1100` | queued entries re-mint `turnId` in the same slot and may come from another socket | | Password login | `acquirePasswordVerification(key)`, `auth.ts:749` | argon2id verify, failure counting, in-flight reservation, then `issueSessionCookie` | | Passkey login | `issueLoginSession`, `packages/ui-server/src/middleware/passkeys.ts:448` | assertion verified, then `issueSessionCookie`; `row.id` is in scope | | Passkey **registration** | `INSERT INTO passkey_credentials`, `passkeys.ts:547` | inserts a credential; calls **neither** helper | @@ -204,7 +204,7 @@ credentials, not people. (`upsertRollup`, `activity/sql.ts:19-24`) — the rollup is what survives span pruning, so cost-by-actor dies at prune time without it. The precedent is profile/billing: a root-span attr that the rollup reads (`spanId: rootSpanId`, -`activity/recorder.ts:222-237`). Nullable for cron (`origin: "cron"`, +`activity/recorder.ts:243-258`). Nullable for cron (`origin: "cron"`, `activity/span-sink.ts:102`, `cron/run-job.ts:113`) and for pre-migration rows; nothing is backfilled. diff --git a/docs/decisions/sqlite-write-worker.md b/docs/decisions/sqlite-write-worker.md index f9be1cfb10..19d575ad90 100644 --- a/docs/decisions/sqlite-write-worker.md +++ b/docs/decisions/sqlite-write-worker.md @@ -98,11 +98,28 @@ thread waits for it. snapshot as the app's pricing instance and reloads the table whenever the app's refresh replaces it. A refresh whose cache write failed leaves the worker on the previous table until the next successful write. -- **Other app-connection writers stay on the loop**: governance events written - outside a journal transaction, the writer gate, turn budgets and scope, the - session catalog, inbox and settings. Each commits at FULL on the loop, and - while a worker procedure holds its turn, such a write also waits there for - the worker's commit to finish. Those writers are tracked separately +- **Governance events and turn scope moved next** + ([#2850](https://github.com/schlessera/brain-kit/issues/2850)). Every + governance event raised outside a journal transaction, the session's first + taint, roadblock classification and turn scope rows commit on the same + worker (`packages/ui-server/src/governance/event-writes.ts`, + `packages/ui-server/src/brain/turn-scope-writes.ts`). An event is validated + and its clock taken where it is raised. Three reads that followed those + writes now wait for them: a connection's first byte waits for its taint's + commit, while admission counts a taint from submission; a scope decision + runs only after the turn's earlier scope writes commit, and settles after + its own; and a turn's outcome waits for its egress records. The scope + summary the client hears of is sent once its row commits. A process without + a write worker, such as the cron runner, runs the same procedures on its own + connection. +- **Other app-connection writers stay on the loop for now**: the writer gate, + turn budgets, the session catalog and message sources, the Activity + notifier and push ticks, auth sessions, inbox and settings. Each commits at + FULL on the loop, and while a worker procedure holds its turn, such a write + also waits there for the worker's commit to finish, including a journal + procedure that writes only `journal.db`, because the ledger's + `BEGIN IMMEDIATE` locks every attached file. Moving each of them to the + worker queue ends that wait too ([#2850](https://github.com/schlessera/brain-kit/issues/2850)). ## Rejected diff --git a/docs/decisions/voice-permission.md b/docs/decisions/voice-permission.md index 8bf770c2c8..d70b6f98ec 100644 --- a/docs/decisions/voice-permission.md +++ b/docs/decisions/voice-permission.md @@ -396,7 +396,7 @@ exchange over a security decision is a second chance for noise to produce a grant. **"Always allow" cannot be given by voice.** It is a persistent policy change -(the block computing `remembers`, `packages/ui-server/src/ws/dispatch.ts:438-444`) +(the block computing `remembers`, `packages/ui-server/src/ws/dispatch.ts:439-445`) and it is the one decision on the card with no keyboard shortcut, by D37's ruling 5, and the reason given there is exactly the one that applies here: *"a letter that grants standing @@ -405,7 +405,7 @@ permission by reflex is the one footgun in the vocabulary"* A microphone is a reflex surface with worse recognition than a keyboard. This costs nothing measurable: the server already refuses `always` for kind `command` requests -(the block computing `remembers`, `dispatch.ts:438-444`, and the lookup computing +(the block computing `remembers`, `dispatch.ts:439-445`, and the lookup computing `remembered`, `ws/bridge.ts:364-384`), and 192 of 192 measured approvals were kind `command`. @@ -460,7 +460,7 @@ An approval given by voice must be as reviewable afterwards as one given by tapping a card. Most of that already exists: every decision is written as an append-only `approval_decision` event carrying the principal, the decision and the request kind, and it patches the span -(`onApprovalDecision`, `packages/ui-server/src/activity/recorder.ts:651-666`), +(`onApprovalDecision`, `packages/ui-server/src/activity/recorder.ts:674-689`), fed from the bridge's `recorded()` wrapper (`const recorded`, `packages/ui-server/src/ws/bridge.ts:406-421`). @@ -475,7 +475,7 @@ provenance. The wire needs nothing new: a spoken refusal is an ordinary `tool_denial` (in `handleClientMessage`, the arm `case "tool_denial"`, -`packages/ui-server/src/ws/dispatch.ts:485-508`) with a message naming the +`packages/ui-server/src/ws/dispatch.ts:486-509`) with a message naming the phrase that produced it. ## Containment: shared with #51, deliberately not identical @@ -610,7 +610,7 @@ matters because a reader cannot otherwise tell a live hazard from a fixed one. left out (the lookup computing `remembered`, `packages/ui-server/src/ws/bridge.ts:364-384`, and the block computing `remembers`, - `packages/ui-server/src/ws/dispatch.ts:438-444`). + `packages/ui-server/src/ws/dispatch.ts:439-445`). The evaluation happens before the lookup, which is what this record asked for. diff --git a/docs/hosting/agent-workers.md b/docs/hosting/agent-workers.md index 6a91dae119..fb75961ca0 100644 --- a/docs/hosting/agent-workers.md +++ b/docs/hosting/agent-workers.md @@ -86,7 +86,7 @@ The explicitly prepared ordinary egress proxy follows that same helper selection. Its directory and socket remain owned by the server and use its shared primary group, with exact `0750`/`0660` modes; a setgid temporary parent cannot choose another IPC group (`const separateIdentity`, -`packages/ui-sdk/src/server/egress-proxy.ts:73-79`). Without a helper they retain +`packages/ui-sdk/src/server/egress-proxy.ts:77-83`). Without a helper they retain owner-only `0700`/`0600` modes. Before mounting the proxy, the launcher checks canonical paths, server ownership, the expected helper group, exact modes and socket type. Missing entries, symlinks and widened permissions refuse before @@ -1536,7 +1536,7 @@ wrappers do not borrow that exemption (`commandTaintSource`, `packages/ui-server/src/activity/command-taint.ts:2-27`). The first source is kept as a session `taint` governance event, using the same SQLite transaction for the existing-source check and write -(`recordFirstTaint`, `packages/ui-server/src/governance/session-taint.ts:17-22`). +(`recordFirstTaint`, `packages/ui-server/src/governance/session-taint.ts:17-23`). The recorder waits for a tool result and excludes known denied calls; failed calls can still return untrusted partial output. It records the source kind and tool name, without copying command text or content into the payload. If diff --git a/docs/investigations/live-conversation-investigation.md b/docs/investigations/live-conversation-investigation.md index 97b190d413..7123a7b143 100644 --- a/docs/investigations/live-conversation-investigation.md +++ b/docs/investigations/live-conversation-investigation.md @@ -279,12 +279,12 @@ matcher. An isolated ambiguous refusal denies all **current** pending requests; an exact refusal can name one. With none pending, it is ordinary input. Bind each denial to existing IDs, echoed turn and `channel:"voice"`; the server still rejects voice grants -(`if (msg.channel === "voice")`, `packages/ui-server/src/ws/dispatch.ts:398-421`). +(`if (msg.channel === "voice")`, `packages/ui-server/src/ws/dispatch.ts:399-422`). Never add a spoken approve or always-allow path. On the evidence date the denial handler removed/resolved a matched request but provided no dedicated denial receipt at that branch; #957 added one -(`case "tool_denial"`, `packages/ui-server/src/ws/dispatch.ts:485-508`). The +(`case "tool_denial"`, `packages/ui-server/src/ws/dispatch.ts:486-509`). The integration therefore needs host-authoritative resolution/replay: denied, already granted, expired, or unknown/disconnected. Speak a denial confirmation only for a confirmed denied outcome. A visual grant winning the race remains diff --git a/docs/plans/async-collaboration-requirements.md b/docs/plans/async-collaboration-requirements.md index 4c818bc755..88c68bf211 100644 --- a/docs/plans/async-collaboration-requirements.md +++ b/docs/plans/async-collaboration-requirements.md @@ -222,7 +222,7 @@ Stated before the requirements because five of them derive from it. ### Execution - R18. **The drain ticks in-server.** (`// manual triggers (triggerJob)`, `packages/ui-server/src/cron/scheduler.ts:11-14`) owns only manual triggers and - history, but (`export function createActivityRuntime(`, `packages/ui-server/src/activity/runtime.ts:57-196`) already runs an interval tick with a boot sweep — + history, but (`export function createActivityRuntime(`, `packages/ui-server/src/activity/runtime.ts:64-203`) already runs an interval tick with a boot sweep — that is the shape to copy, including its `close()` lifecycle. - R19. **The cron backstop has independent authorization before the general guard.** Mount it on the existing listener before @@ -253,10 +253,10 @@ Stated before the requirements because five of them derive from it. `persistSession: false`; pi provides `SessionManager.inMemory`. Drive this mode with a synthetic checkpoint bridge and server-selected authority; the recorder accepts autonomous origin and keeps runtime identity separate from an interactive session - (`export function createTurnRecorder(`, `packages/ui-server/src/activity/recorder.ts:115-193`). + (`export function createTurnRecorder(`, `packages/ui-server/src/activity/recorder.ts:125-203`). - R25. **Autonomous work gets its own pool** (`MAX_AUTONOMOUS_RUNS`, default 2) — but a second counter alone does not deliver "interactive always wins". The host cap applies only when - starting WS sessions (`const cap = host.maxConcurrentSessions();`, `packages/ui-server/src/ws/run-session.ts:1300-1312`), and an autonomous + starting WS sessions (`const cap = host.maxConcurrentSessions();`, `packages/ui-server/src/ws/run-session.ts:1310-1322`), and an autonomous turn can hold a path write lock while an interactive turn waits or is denied at 30 seconds (`export function createTurnLockBinding(`, `packages/ui-backend-claude/src/turn-lock.ts:27-119`). Required: an admission controller with reserved interactive capacity and hybrid yield at an explicit denial-risk @@ -343,7 +343,7 @@ Stated before the requirements because five of them derive from it. policy proposals, and discuss-session digest construction when those v2 features ship. Revision 1's "T2 is the entire bill" was false. - R41. **Budgets are enforced by reservation at claim time, not by summing history.** - `rollupRun` runs in `finish()` (`store.rollupRun(runId);`, `packages/ui-server/src/activity/recorder.ts:723`), so cost exists only after a + `rollupRun` runs in `finish()` (`store.rollupRun(runId);`, `packages/ui-server/src/activity/recorder.ts:746`), so cost exists only after a run ends: two runs can both start under the cap and finish over it, and unknown effective costs are excluded from the sum (`export function sumEffectiveCost(`, `packages/ui-server/src/activity/store.ts:179-194`) so the query **fails open**. Required: transactional reservations on claim, in-flight reservations counted, settlement at diff --git a/docs/plans/async-collaboration.md b/docs/plans/async-collaboration.md index cf586c8b19..7370bc8419 100644 --- a/docs/plans/async-collaboration.md +++ b/docs/plans/async-collaboration.md @@ -127,7 +127,7 @@ system gate. It is not a prerequisite to rebuild or a new independent package. - **Cursors and transactional snapshots:** (`CREATE TABLE IF NOT EXISTS activity_changes (`, `packages/ui-server/migrations/007_activity.sql:58-66`) and (`snapshotRun(runId) {`, `packages/ui-server/src/activity/store.ts:894-923`). Reuse the algorithm; the activity stream is bound to activity store methods and frame types. -- **Runtime lifecycle:** (`export function createActivityRuntime(`, `packages/ui-server/src/activity/runtime.ts:57-196`). Cron's scheduler +- **Runtime lifecycle:** (`export function createActivityRuntime(`, `packages/ui-server/src/activity/runtime.ts:64-203`). Cron's scheduler owns manual triggers/history; it does not tick the inbox. - **Two connections:** (`export function createUiDb(`, `packages/ui-server/src/db/client.ts:27-44`) sets WAL, foreign keys and a 5-second busy timeout. Claims are immediate transactions. @@ -146,7 +146,7 @@ system gate. It is not a prerequisite to rebuild or a new independent package. - **Pi resources and extension gate:** (`export function createSessionResources(`, `packages/ui-backend-pi/src/session-resources.ts:29-152`) and (`export function createPermissionGate(`, `packages/ui-backend-pi/src/permission-gate.ts:96-125`). Built-ins are disabled, but ambient resources/extensions and in-process execution still need containment. -- **Cost timing:** (`store.rollupRun(runId);`, `packages/ui-server/src/activity/recorder.ts:723`) settles after execution; +- **Cost timing:** (`store.rollupRun(runId);`, `packages/ui-server/src/activity/recorder.ts:746`) settles after execution; (`export function sumEffectiveCost(`, `packages/ui-server/src/activity/store.ts:179-194`) explicitly counts unpriced runs. Reservations must cover in-flight work, not only this retrospective sum. - **Notifications:** (`CREATE TABLE IF NOT EXISTS notification_intents (`, `packages/ui-server/migrations/007_activity.sql:99-114`), @@ -156,7 +156,7 @@ system gate. It is not a prerequisite to rebuild or a new independent package. - **Share provenance/limits:** (`const result = await stageShareAt(`, `packages/ui-server/src/inbox/intake.ts:106`) assigns the source in server code. (`export const SHARE_MAX_TEXT_BYTES =`, `packages/ui-sdk/src/protocol.ts:2348`) bounds text, not binary uploads; extracted T1 context needs its own byte/token bound. -- **Interactive locks:** (`const cap = host.maxConcurrentSessions();`, `packages/ui-server/src/ws/run-session.ts:1300-1312`) gates WS starts; +- **Interactive locks:** (`const cap = host.maxConcurrentSessions();`, `packages/ui-server/src/ws/run-session.ts:1310-1322`) gates WS starts; (`export function createTurnLockBinding(`, `packages/ui-backend-claude/src/turn-lock.ts:27-119`) owns tool locks. Shared-target priority and cooperative yield belong to the keyed lock and backend lifecycle. @@ -632,7 +632,7 @@ write-once outcome discipline, cursor emission); migration comment style of `007 - Test: `packages/ui-server/tests/inbox-runtime.test.ts` **Approach:** -- 60s interval with a `close()` lifecycle, modeled on (`export function createActivityRuntime(`, `packages/ui-server/src/activity/runtime.ts:57-196`), including +- 60s interval with a `close()` lifecycle, modeled on (`export function createActivityRuntime(`, `packages/ui-server/src/activity/runtime.ts:64-203`), including its boot sweep - **Gate first**: a SQL count of ready items; zero means return immediately — no model call and **no Activity run** (AE2) @@ -1193,7 +1193,7 @@ and a server-selected tool policy. - Modify: `packages/ui-backend-claude/src/backend.ts` (`persistSession: false`, synthetic bridge) - Modify: `packages/ui-server/src/activity/recorder.ts` (server-selected - origin defaults to `"session"` at (`export function createTurnRecorder(`, `packages/ui-server/src/activity/recorder.ts:115-193`)) + origin defaults to `"session"` at (`export function createTurnRecorder(`, `packages/ui-server/src/activity/recorder.ts:125-203`)) - Test: `packages/ui-backend-claude/tests/autonomous-turn.test.ts` **Approach:** an autonomous request carrying persistence, tool policy, origin, and prompt @@ -1274,7 +1274,7 @@ full-v1 enablement; predicates and schema tests cannot substitute for it. **Approach:** **hybrid — reserve capacity normally, yield only at denial risk.** `MAX_AUTONOMOUS_RUNS` (default 2) is necessary but not sufficient: the host cap applies only at -WS session start (`const cap = host.maxConcurrentSessions();`, `packages/ui-server/src/ws/run-session.ts:1300-1312`) and an autonomous turn can hold a path write lock +WS session start (`const cap = host.maxConcurrentSessions();`, `packages/ui-server/src/ws/run-session.ts:1310-1322`) and an autonomous turn can hold a path write lock while an interactive turn waits or is denied at 30s (`export function createTurnLockBinding(`, `packages/ui-backend-claude/src/turn-lock.ts:27-119`). diff --git a/docs/plans/navigation-recovery.md b/docs/plans/navigation-recovery.md index fcb5f00fd9..77b9dbdaa2 100644 --- a/docs/plans/navigation-recovery.md +++ b/docs/plans/navigation-recovery.md @@ -385,13 +385,13 @@ implementation. Any PR moving these lines owns their citation repairs. - Host reconnect: (`const runningTurns = [...coordinator.running].filter((t) => t.sessionId);`, `packages/ui-server/src/ws/connection.ts:126-191`). -- Host resume: (`case "session_resume": {`, `packages/ui-server/src/ws/dispatch.ts:620-674`). +- Host resume: (`case "session_resume": {`, `packages/ui-server/src/ws/dispatch.ts:621-675`). -- Catalog activity: (`persistSessionStub(sessionId, promptText, providerId, backendId) {`, `packages/ui-server/src/ws/session-catalog.ts:335-374`). +- Catalog activity: (`persistSessionStub(sessionId, promptText, providerId, backendId) {`, `packages/ui-server/src/ws/session-catalog.ts:339-378`). - Session summary shape: (`export interface ChatSession {`, `packages/ui-sdk/src/protocol.ts:1568-1584`). -- Run identity: (`const runId = turn.turnId;`, `packages/ui-server/src/activity/recorder.ts:144-145`). +- Run identity: (`const runId = turn.turnId;`, `packages/ui-server/src/activity/recorder.ts:154-155`). - Outcome vocabulary: (`export type ActivitySpanOutcome =`, `packages/ui-sdk/src/protocol.ts:2930-2953`). @@ -399,7 +399,7 @@ implementation. Any PR moving these lines owns their citation repairs. - API authorization: (`app.use("/api/*", authGuard(authMode, auth, db));`, `packages/ui-server/src/app.ts:877`). -- Queued acceptance: (`// Queue it as the session's next turn; report queued immediately.`, `packages/ui-server/src/ws/run-session.ts:1067-1112`). +- Queued acceptance: (`// Queue it as the session's next turn; report queued immediately.`, `packages/ui-server/src/ws/run-session.ts:1077-1122`). - Approval target: (`function mutateLastAssistant(`, `packages/ui-react/src/stores/chat-state.ts:1224-1254`). diff --git a/docs/plans/storage-accounting.md b/docs/plans/storage-accounting.md index 16a0193ab7..6f720d3ca8 100644 --- a/docs/plans/storage-accounting.md +++ b/docs/plans/storage-accounting.md @@ -67,7 +67,7 @@ Source anchors, checked against the reviewed commit: The completed #94 and #398 provide existing stats/maintenance foundations. - C: snapshot fields (`snapshotOf`, `packages/core/src/lib/stats-history.ts:132-172`), thinning (`thin`, `packages/core/src/lib/stats-history.ts:232-244`). -- D: effective path/journal resolution (`createApp`, `packages/ui-server/src/app.ts:293-329`), +- D: effective path/journal resolution (`createApp`, `packages/ui-server/src/app.ts:292-328`), WAL store (`createUiDb`, `packages/ui-server/src/db/client.ts:27-44`), and private locations (`statePaths`, `packages/ui-server/src/brain/operational-state.ts:108-114`). - E: live/archive paths (`transcriptPaths`, `packages/ui-backend-claude/src/transcript-archive.ts:30-39`). diff --git a/packages/ui-sdk/src/server/egress-proxy.ts b/packages/ui-sdk/src/server/egress-proxy.ts index 5eae9d9b5e..c01b5b86bc 100644 --- a/packages/ui-sdk/src/server/egress-proxy.ts +++ b/packages/ui-sdk/src/server/egress-proxy.ts @@ -44,8 +44,12 @@ export interface EgressProxyOptions { * `contacted()` reports; deny and the ordinary rules still apply first. */ taint?: { allow: readonly string[]; tainted(): boolean; contacted(host: string): boolean }; - /** A destination passed every check; called before its first upstream byte. */ - admitted?(destination: { host: string; port: number; method: string; inference: boolean }): void; + /** + * A destination passed every check; called before its first upstream byte. + * A returned promise is awaited first, so a durable record it makes commits + * before any byte flows; its rejection refuses the connection. + */ + admitted?(destination: { host: string; port: number; method: string; inference: boolean }): unknown; signal?: AbortSignal; } const hostname = (value: string) => value.toLowerCase().replace(/\.$/, ""); @@ -131,7 +135,8 @@ export async function startEgressProxy(options: EgressProxyOptions) { } if (narrowed()) { finish("denied", "taint_not_allowed"); return false; } // A recorder failure aborts the turn through its own signal; it cannot admit. - try { options.admitted?.({ host, port, method, inference }); } catch { finish("failed", "proxy_error"); return false; } + try { await options.admitted?.({ host, port, method, inference }); } catch { finish("failed", "proxy_error"); return false; } + if (stopped || controller.signal.aborted) { finish("cancelled", "turn_cancelled"); return false; } return true; }; return { finish, charge, admit, signal: controller.signal, outcome: () => finalOutcome }; diff --git a/packages/ui-sdk/tests/boot-prune-capture.test.ts b/packages/ui-sdk/tests/boot-prune-capture.test.ts index 0055c03f3b..47f3294d32 100644 --- a/packages/ui-sdk/tests/boot-prune-capture.test.ts +++ b/packages/ui-sdk/tests/boot-prune-capture.test.ts @@ -10,7 +10,7 @@ const source = await Bun.file(resolve(import.meta.dir, "../../ui-server/src/cron async function factory() { const root = mkdtempSync(join(tmpdir(), "odysseus-boot-prune-")); try { - const variant = bootPruneCaptureVariant(source).cron.replace('"../governance/events.js"', JSON.stringify(resolve(import.meta.dir, "../../ui-server/src/governance/events.ts"))); + const variant = bootPruneCaptureVariant(source).cron.replace('"../governance/event-writes.js"', JSON.stringify(resolve(import.meta.dir, "../../ui-server/src/governance/event-writes.ts"))); const file = join(root, "scratch-prune.ts"); writeFileSync(file, variant); return { start: (await import(file)).startScratchPrune, cleanup: () => rmSync(root, { recursive: true, force: true }) }; } catch (error) { rmSync(root, { recursive: true, force: true }); throw error; } diff --git a/packages/ui-sdk/tests/owned-prune-variant.test.ts b/packages/ui-sdk/tests/owned-prune-variant.test.ts index c9e2c15722..4e1efc8167 100644 --- a/packages/ui-sdk/tests/owned-prune-variant.test.ts +++ b/packages/ui-sdk/tests/owned-prune-variant.test.ts @@ -31,7 +31,7 @@ test("fixed owned-prune variants refuse missing/duplicate anchors and preserve g test("real variant tick joins existing work and starts a subsequent pass with the same receiver/signal/output", async () => { const root = mkdtempSync(join(tmpdir(), "odysseus-prune-variant-")); - const source = ownedPruneVariant(cron, boundary).cron.replace('"../governance/events.js"', JSON.stringify(resolve(import.meta.dir, "../../ui-server/src/governance/events.ts"))); + const source = ownedPruneVariant(cron, boundary).cron.replace('"../governance/event-writes.js"', JSON.stringify(resolve(import.meta.dir, "../../ui-server/src/governance/event-writes.ts"))); const file = join(root, "scratch-prune.ts"); writeFileSync(file, source); const descriptor = Object.getOwnPropertyDescriptor(console, "log")!; const records: unknown[] = []; Object.defineProperty(console, "log", { ...descriptor, value: (text: string) => records.push(JSON.parse(text)) }); diff --git a/packages/ui-server/src/activity/recorder.ts b/packages/ui-server/src/activity/recorder.ts index ca24eb9c3d..b32016dd29 100644 --- a/packages/ui-server/src/activity/recorder.ts +++ b/packages/ui-server/src/activity/recorder.ts @@ -38,11 +38,12 @@ import type { Logger } from "@opentelemetry/api-logs"; import type { Database } from "bun:sqlite"; import type { GovernanceEventInput, GovernanceProfile } from "../governance/events.js"; -import { recordTurnRoadblocks, type TracedCall, type UsageSample } from "../governance/roadblocks.js"; +import type { TracedCall, UsageSample } from "../governance/roadblocks.js"; +import { createGovernanceWrites, type GovernanceWrites } from "../governance/event-writes.js"; +import { recordFirstTaint } from "../governance/session-taint.js"; import { redactCredentials, type RuntimeStatus } from "./runtime-status.js"; import { inputTaintSource, outputTaintSource, type IngestionSource } from "./ingestion-taint.js"; -import { recordFirstTaint } from "../governance/session-taint.js"; import { MAX_PROVIDER_UNAVAILABLE_PER_TURN, providerUnavailablePayloads } from "../governance/provider-unavailable.js"; import type { AwaitableActivityStore, SpanOutcome, SpanUsage } from "./store.js"; import { orderedWrites } from "./store-writes.js"; @@ -54,14 +55,23 @@ export interface TurnRecorderDeps { /** Called after any committed write so the live stream can pump. */ onWrite?: () => void; log?: Logger; - /** The governance-event writer (#1651). Absent: interventions stay span events only. */ - governance?: (input: GovernanceEventInput) => void; + /** + * The governance-event writer (#1651). Absent: interventions stay span + * events only. A returned promise is the commit (#2850): `settled()` waits + * for it. + */ + governance?: (input: GovernanceEventInput) => unknown; /** * The server database the turn's roadblocks are classified against and * recorded in at finish (#1671): it reads the turn's other governance * events as well as writing. Absent records none. */ governanceDb?: Database; + /** + * Where the turn's taint and roadblocks commit (#2850): the hosted server's + * write worker. Absent, they commit on `governanceDb` itself. + */ + governanceWrites?: GovernanceWrites; /** The brain's host path, so an absolute file-tool path resolves to the area it reads (G6). */ brainPath?: string; /** Stops the turn when its untrusted-input taint cannot be persisted: outward rules must not stay loose. */ @@ -213,6 +223,17 @@ export function createTurnRecorder( const writes = orderedWrites(deps.store, dropped); const store = writes.store; const onWrite = wrote ? () => writes.after(wrote) : undefined; + // Governance rows are submitted when raised, so a later classification sees + // them; `settled()` also waits for their commits (#2850). + const governanceWrites = deps.governanceWrites ?? (deps.governanceDb ? createGovernanceWrites(deps.governanceDb) : undefined); + // Without a write worker the taint commits through `governance`, at once, as it always did. + const governanceDb = deps.governanceDb, record = deps.governance; + const firstTaint = deps.governanceWrites ? (input: GovernanceEventInput<"taint">) => deps.governanceWrites!.firstTaint(input) + : governanceDb && record ? (input: GovernanceEventInput<"taint">) => { recordFirstTaint(governanceDb, input, record); return Promise.resolve(); } : undefined; + const govern = (input: GovernanceEventInput) => { + const committed = deps.governance?.(input); + if (committed instanceof Promise) writes.after(() => committed); + }; function ensureRoot() { if (rootStarted) return; @@ -326,15 +347,17 @@ export function createTurnRecorder( const ingestion = ingestions.get(msg.toolUseId); ingestions.delete(msg.toolUseId); const source = ingestion && (ingestion.source ?? outputTaintSource(ingestion.tool, msg.output, deps.brainPath)); - if (ingestion && source && !traced?.denied && deps.governanceDb && deps.governance) { - try { - recordFirstTaint(deps.governanceDb, { kind: "taint", payload: { source, tool: ingestion.tool, detail: null }, - sessionId: sessionId ?? null, turnId: runId, actor: turn.posture ?? "interactive", - model: mainModel, profile: turn.profile ?? null }, deps.governance); - } catch (error) { + if (ingestion && source && !traced?.denied && firstTaint && deps.governance) { + const untracked = (error: unknown) => { log?.emit({ severityText: "WARN", body: `taint not recorded: ${error instanceof Error ? error.message : String(error)}` }); deps.abort?.(new Error("Untrusted input could not be recorded; the turn was stopped.")); - } + }; + try { + const committed = firstTaint({ kind: "taint", payload: { source, tool: ingestion.tool, detail: null }, + sessionId: sessionId ?? null, turnId: runId, actor: turn.posture ?? "interactive", + model: mainModel, profile: turn.profile ?? null }).then(() => {}, untracked); + writes.after(() => committed); + } catch (error) { untracked(error); } } // A `brain` command in the worker that needed a withheld provider key (#1666). if (turn.origin !== "autonomous" && deps.governance && !traced?.denied && providerUnavailableRecorded < MAX_PROVIDER_UNAVAILABLE_PER_TURN) { @@ -342,7 +365,7 @@ export function createTurnRecorder( if (providerUnavailableRecorded >= MAX_PROVIDER_UNAVAILABLE_PER_TURN) break; providerUnavailableRecorded++; try { - deps.governance({ kind: "provider_unavailable", payload, sessionId: sessionId ?? null, turnId: runId, + govern({ kind: "provider_unavailable", payload, sessionId: sessionId ?? null, turnId: runId, actor: turn.posture ?? "interactive", model: mainModel, profile: turn.profile ?? null }); } catch (error) { log?.emit({ severityText: "WARN", body: `governance event not recorded: ${error instanceof Error ? error.message : String(error)}` }); @@ -521,7 +544,7 @@ export function createTurnRecorder( ensureRoot(); store.appendEvent(rootSpanId, event.kind, event.payload); try { - deps.governance?.({ kind: event.kind, payload: event.payload, + govern({ kind: event.kind, payload: event.payload, sessionId: sessionId ?? null, turnId: runId, actor: turn.origin === "autonomous" ? "autonomous" : turn.posture ?? "interactive", model: turn.origin === "autonomous" ? resultUsage?.model ?? null : mainModel, profile: turn.profile ?? (turn.origin === "autonomous" ? "restricted" : null) }); @@ -542,7 +565,7 @@ export function createTurnRecorder( } if (deps.governance && event.limits.length) { try { - deps.governance({ kind: "resource_limit", payload: { limits: event.limits.slice(0, 20), fatal: event.fatal }, + govern({ kind: "resource_limit", payload: { limits: event.limits.slice(0, 20), fatal: event.fatal }, sessionId: sessionId ?? null, turnId: runId, actor: turn.origin === "autonomous" ? "autonomous" : turn.posture ?? "interactive", model: null, profile: turn.profile ?? (turn.origin === "autonomous" ? "restricted" : null) }); @@ -727,11 +750,11 @@ export function createTurnRecorder( }); // Separately guarded: a classification or storage failure loses the // roadblock records, never the turn's own Activity write above. - const governance = deps.governanceDb; - if (governance && !roadblocksRecorded) { + if (governanceWrites && !roadblocksRecorded) { roadblocksRecorded = true; guard(() => { - recordTurnRoadblocks(governance, { + // Submitted now, so it classifies exactly the events raised before finish. + const recorded = governanceWrites.roadblocks({ turnId: runId, sessionId: sessionId ?? null, actor: turn.origin === "autonomous" ? "autonomous" : turn.posture ?? "interactive", @@ -749,6 +772,7 @@ export function createTurnRecorder( cancelledAt, cancelledCallIds, }); + writes.after(() => recorded); }); } }, diff --git a/packages/ui-server/src/activity/runtime.ts b/packages/ui-server/src/activity/runtime.ts index 43479c4c3d..3a3af03a2c 100644 --- a/packages/ui-server/src/activity/runtime.ts +++ b/packages/ui-server/src/activity/runtime.ts @@ -24,7 +24,8 @@ import { digestRetentionFloor } from "./digest.js"; import { runActivityQuery } from "./query.js"; import { keepSubscriptionProof } from "../agent/subscription.js"; import { createRuntimeStatus, type RuntimeStatus } from "./runtime-status.js"; -import { recordGovernanceEvent, type GovernanceEventInput } from "../governance/events.js"; +import type { GovernanceEventInput } from "../governance/events.js"; +import { createGovernanceWrites, type GovernanceWrites } from "../governance/event-writes.js"; import { getDetailRetentionDays } from "../db/settings.js"; // Activity lifecycle cadence. The stale threshold must comfortably exceed @@ -49,8 +50,14 @@ export interface ActivityRuntime { runtime: RuntimeStatus; /** The agent-facing read seam (bridge.queryActivity). */ query(query: ActivityQuery): ActivityQueryResult; - /** The governance-event writer (#1651), for the turn recorders. Throws on an invalid event. */ - governance(input: GovernanceEventInput): void; + /** + * The governance-event writer (#1651), for the turn recorders. Throws on an + * invalid event. With a write worker the row commits there (#2850); the + * promise settles once it has, and never rejects: a failed commit is logged. + */ + governance(input: GovernanceEventInput): Promise; + /** Every governance write, committed on the write worker when there is one (#2850). */ + governanceWrites: GovernanceWrites; close(): void; } @@ -78,6 +85,7 @@ export function createActivityRuntime( ? createOffloopActivityStore(db, deps.writer, { ...(deps.pricing ? { pricing: deps.pricing } : {}), ...(deps.pricingConfig ? { pricingConfig: deps.pricingConfig } : {}) }) : createActivityStore(db, deps.pricing ? { pricing: deps.pricing } : {}); const stream = createActivityStream(store, log); + const governanceWrites = createGovernanceWrites(db, deps.writer); // The notifier resumes its persisted cursor, including terminal changes // committed while this server was down and interruptions from the boot sweep. @@ -204,7 +212,10 @@ export function createActivityRuntime( actionNotices, runtime: createRuntimeStatus(undefined, log, (at) => keepSubscriptionProof(db, at)), query: (query) => runActivityQuery(db, store, query, notifier), - governance: (input) => { recordGovernanceEvent(db, input); }, + governance: (input) => governanceWrites.record(input).then(() => {}, (err: unknown) => { + log.emit({ severityText: "WARN", body: `governance event not recorded: ${err instanceof Error ? err.message : String(err)}` }); + }), + governanceWrites, close() { clearInterval(tick); stream.close(); diff --git a/packages/ui-server/src/app.ts b/packages/ui-server/src/app.ts index c8135a567f..d500eb3197 100644 --- a/packages/ui-server/src/app.ts +++ b/packages/ui-server/src/app.ts @@ -12,7 +12,6 @@ import { createStorageMaintenanceRoutes } from "./routes/storage-maintenance.js" import { createStorageCoordinator } from "./storage/coordinator.js"; import { createStorageRoutes } from "./routes/storage.js"; import type { StorageStore } from "./storage/collect.js"; -import { recordGovernanceEvent } from "./governance/events.js"; import { createPolicyProposals } from "./brain/policy-proposal.js"; import { createAgentPolicyRoutes } from "./routes/agent-policy.js"; import { createPolicyProposalRoutes } from "./routes/policy-proposals.js"; @@ -573,7 +572,7 @@ export async function createApp(options: CreateAppOptions = {}): Promise scratchPins.close()); - const scratchPrune = startScratchPrune({ brain, pins: scratchPins, db, log: observability.logger("cron") }); + const scratchPrune = startScratchPrune({ brain, pins: scratchPins, governance: activity.governanceWrites, log: observability.logger("cron") }); bootCleanup.push(() => scratchPrune.close()); // The subscription token (#254): boot warnings now, the expiry check daily, // and what /api/status says about it. @@ -624,10 +623,10 @@ export async function createApp(options: CreateAppOptions = {}): Promise host.workerLock.withLock(key, fn, admission) }, observeUserPolicy: (turnId, write) => journal.observeUserPolicy(turnId, write), - recordEvent: (review, turn) => recordGovernanceEvent(db, { kind: "policy_change", actor: "user", turnId: turn.turnId, sessionId: turn.sessionId(), model: null, profile: null, + recordEvent: (review, turn) => activity.governanceWrites.record({ kind: "policy_change", actor: "user", turnId: turn.turnId, sessionId: turn.sessionId(), model: null, profile: null, payload: { requestId: review.requestId, baseHash: review.baseHash, proposedHash: review.proposedHash, sections: review.sections, outcome: "updated" }, dedupeKey: `${turn.turnId}:${review.requestId}` }), }); - const scope = createTurnScope(db, { brainPath: config.brainPath, journal, + const scope = createTurnScope(db, { brainPath: config.brainPath, journal, writer, governance: activity.governanceWrites, log: brainLog, connected: () => host.clients.hasClients(), changed(turnId, sessionId) { host.clients.broadcast({ type: "turn_journal", sessionId, turnId, journal: { ...journal.summary(turnId), ...(host.budgets?.saved(turnId) ? { budget: host.budgets.saved(turnId) } : {}) } }); } }); const host = new WsHost({ @@ -648,7 +647,7 @@ export async function createApp(options: CreateAppOptions = {}): Promise db, dbLog, scratchPins, { pricing, store: activity.store }), + catalog: createSessionCatalog(() => db, dbLog, scratchPins, { pricing, store: activity.store, governance: activity.governanceWrites }), // Exact-operation approvals remain fail-closed until the full-v1 engine // wires current server authority. Dismiss/cancel/snooze need no grant. inboxTools: createInboxBridgeTools(db, intake, hygieneReview.resolver, config.brainPath), @@ -693,6 +692,7 @@ export async function createApp(options: CreateAppOptions = {}): Promise { try { host.close(); } finally { await Promise.allSettled([...host.turnTasks]); } }); @@ -700,7 +700,7 @@ export async function createApp(options: CreateAppOptions = {}): Promise dispatchFixture.select(registry), budget: inboxBudget, fetch: dispatchFixture.fetch, now: dispatchFixture.now, classification: { apiKey: dispatchFixture.classificationKey ?? config.classification.apiKey }, - store: activity.store, runtime: activity.runtime, onWrite: () => activity.stream.pump(), log: observability.logger("inbox") }); + store: activity.store, runtime: activity.runtime, governanceWrites: activity.governanceWrites, onWrite: () => activity.stream.pump(), log: observability.logger("inbox") }); // The agent worker boundary, probed now (#1632) so the server knows at boot // whether this host can run a turn, and says so in the log, on /api/status // and in the client. Unlike a missing runtime, a failure does not refuse @@ -995,7 +995,7 @@ export async function createApp(options: CreateAppOptions = {}): Promise storageRemoval.close()); app.route("/api", createStorageRemovalRoutes(storageRemoval)); - const reverts = createTurnReverts(db, { brainPath: config.brainPath, journal, gate: hostExecution.gate()!, lock: host.workerLock, + const reverts = createTurnReverts(db, { brainPath: config.brainPath, journal, gate: hostExecution.gate()!, lock: host.workerLock, governance: activity.governanceWrites, isAuthorized(id) { const principal = resolvePrincipal(db, id); return !!principal && ["owner", "ambient"].includes(principal.kind) && isUsablePrincipal(principal, Date.now()); }, refresh() { host.governedTurns?.refresh(host.workerLock); } }); bootCleanup.push(() => reverts.close()); diff --git a/packages/ui-server/src/brain/policy-proposal.ts b/packages/ui-server/src/brain/policy-proposal.ts index ae0e4ed228..cfa94586b0 100644 --- a/packages/ui-server/src/brain/policy-proposal.ts +++ b/packages/ui-server/src/brain/policy-proposal.ts @@ -106,7 +106,8 @@ export function createPolicyProposals(options: { root: string; db: Database; lock: Pick; /** The journal service owns exact user/policy attribution and its live recording window. */ observeUserPolicy(turnId: string, approvedWrite: () => void): void | Promise; - recordEvent(review: PolicyChangeReview, turn: PolicyProposalTurn): void; + /** A returned promise is the event's commit (#2850); its failure is the receipt's. */ + recordEvent(review: PolicyChangeReview, turn: PolicyProposalTurn): unknown; /** Deterministic race/revalidation fixtures, never wired to a worker. */ beforeCommit?(): void; validate?: typeof parseGovernancePolicy; @@ -223,7 +224,7 @@ export function createPolicyProposals(options: { fsyncSync(target!.chain.at(-1)!.fd); }); review.state = "updated"; review.message = "Policy updated. Applies from the next turn."; - options.recordEvent(review, turn); + await options.recordEvent(review, turn); } finally { if (temp) { try { unlinkSync(temp); } catch { /* Rename consumed it. */ } } if (target) close(target); diff --git a/packages/ui-server/src/brain/turn-egress.ts b/packages/ui-server/src/brain/turn-egress.ts index 20c96aaee7..d508bd3334 100644 --- a/packages/ui-server/src/brain/turn-egress.ts +++ b/packages/ui-server/src/brain/turn-egress.ts @@ -5,6 +5,7 @@ import { requestToolPermission, type BackendActivityEvent, type BackendBridge } import type { Database } from "bun:sqlite"; import type { GovernanceEventInput } from "../governance/events.js"; import { contactedHosts, recordFirstTaint, sessionTaint } from "../governance/session-taint.js"; +import type { GovernanceWrites } from "../governance/event-writes.js"; type Classification = Extract["classification"]; const GITHUB_API_HOSTS = ["api.github.com", "uploads.github.com"] as const; @@ -60,14 +61,31 @@ export async function prepareTurnEgress(options: { policy: EffectiveGovernancePolicy; bridge: BackendBridge; controller: AbortController; turnId: string; sessionId: () => string | null; model: () => string | null; profile: "governed" | "strict"; origin: "interactive" | "voice"; - /** Absent when the host keeps no governance record: limits still apply. */ - record?(event: GovernanceEventInput): void; + /** + * Absent when the host keeps no governance record: limits still apply. A + * returned promise is the record's commit (#2850); its failure stops the + * turn like a synchronous one, and `settled()` waits for it. + */ + record?(event: GovernanceEventInput): unknown; /** The record the session's taint and contacted hosts are read from (G6). Absent: no taint narrowing. */ governanceDb?: Database; + /** + * Where the session's first taint commits, and how admission reads it back + * before that commit lands (#2850). Absent: the taint commits on + * `governanceDb` through `record`. + */ + governanceWrites?: Pick; }) { const calls = createOutwardCalls(); let recordingFailure: Error | undefined; const db = options.governanceDb; + // Without a write worker the taint commits through `record`, at once, as it always did. + const record = options.record; + const writes = options.governanceWrites ?? (db && record ? { + taint: (turnId: string | null, sessionId: string | null) => sessionTaint(db, turnId, sessionId), + firstTaint: (event: GovernanceEventInput<"taint">) => { recordFirstTaint(db, event, record); return Promise.resolve(true); }, + } : undefined); + const recording = new Set>(); // Hosts earlier turns of the session reached, then every host this turn admits. const contacted = new Set(db ? contactedHosts(db, options.sessionId()) : []); let tainted = false; @@ -77,19 +95,24 @@ export async function prepareTurnEgress(options: { recordingFailure ??= new Error("Turn egress recording is unavailable"); options.controller.abort(recordingFailure); }; + const track = (committed: unknown) => { + if (!(committed instanceof Promise)) return; + const pending = committed.catch(fail).finally(() => recording.delete(pending)); + recording.add(pending); + }; const report = (record: EgressRecord): void => { if (!options.record) return; try { - options.record({ ...base(), kind: "egress_connection", payload: record }); + track(options.record({ ...base(), kind: "egress_connection", payload: record })); const denied = options.policy.egress.deny.some(pattern => { const normalized = pattern.toLowerCase().replace(/\.$/, ""); return normalized.startsWith("*.") ? record.host.endsWith(normalized.slice(1)) && record.host !== normalized.slice(2) : record.host === normalized; }); const reason = record.reason === "destination_not_allowed" ? denied ? "denied_host" : "not_allowlisted" : record.reason === "max_bytes_per_turn" ? "byte_cap" : record.reason === "taint_not_allowed" ? "taint_allowlist" : undefined; - if (reason) options.record({ ...base(), kind: "egress_denial", payload: { + if (reason) track(options.record({ ...base(), kind: "egress_denial", payload: { host: record.host, port: record.port, method: record.method, reason, bytes: record.bytes, - } }); + } })); } catch { fail(); } }; /** @@ -98,19 +121,27 @@ export async function prepareTurnEgress(options: { */ const taint = db ? { allow: options.policy.taint.egress.allow, - tainted: () => tainted ||= sessionTaint(db, options.turnId, options.sessionId()) !== null, + // A taint submitted but not yet committed counts (#2850). + tainted: () => tainted ||= (writes ? writes.taint(options.turnId, options.sessionId()) : sessionTaint(db, options.turnId, options.sessionId())) !== null, contacted: (host: string) => contacted.has(host), } : undefined; - const admitted = (destination: { host: string; port: number; method: string; inference: boolean }): void => { + /** + * The proxy awaits this before the connection's first byte, so the taint + * has committed by then. It is in force for every later admission from the + * moment it is submitted. + */ + const admitted = async (destination: { host: string; port: number; method: string; inference: boolean }): Promise => { if (destination.inference) return; contacted.add(destination.host); - if (!db || !options.record || tainted) return; + if (!writes || tainted) return; + tainted = true; try { - recordFirstTaint(db, { ...base(), kind: "taint", payload: { source: "egress_connection", tool: null, - detail: `${destination.method} ${destination.host}:${destination.port}` } }, options.record); - tainted = true; + await writes.firstTaint({ ...base(), kind: "taint", payload: { source: "egress_connection", tool: null, + detail: `${destination.method} ${destination.host}:${destination.port}` } }); } catch (error) { fail(); throw error; } }; + /** Settles once every record submitted so far has committed or failed; `recordingFailure()` is final after it. */ + const settled = async () => { while (recording.size) await Promise.all([...recording]); }; const proxy = await startEgressProxy({ mode: options.policy.egress.mode, allow: options.policy.egress.allow, deny: options.policy.egress.deny, maxBytes: options.policy.egress.max_bytes_per_turn ?? Number.MAX_SAFE_INTEGER, outwardHosts: [...new Set([...GITHUB_API_HOSTS, ...MAIL_SUBMISSION_HOSTS, ...options.policy.outward.hosts])], @@ -131,5 +162,5 @@ export async function prepareTurnEgress(options: { emit(message) { calls.emitted(message); original.emit(message); }, activity(event) { calls.activity(event); original.activity?.(event); }, }; - return { ...proxy, bridge, recordingFailure: () => recordingFailure }; + return { ...proxy, bridge, recordingFailure: () => recordingFailure, settled }; } diff --git a/packages/ui-server/src/brain/turn-journal.ts b/packages/ui-server/src/brain/turn-journal.ts index 3923eb1064..1c67034ae3 100644 --- a/packages/ui-server/src/brain/turn-journal.ts +++ b/packages/ui-server/src/brain/turn-journal.ts @@ -4,7 +4,7 @@ import type { JournalVersion, TurnJournalDetails, TurnJournalSummary, JournalOut import { openSqliteWriter, type SqliteWriter } from "../db/write-worker.js"; import { JOURNAL_SCHEMA, journalDatabasePath } from "../db/journal-database.js"; import { installEscapeGuard, JournalStorageUnavailable } from "./journal-accounting.js"; -import { replaySteeringEvents } from "./journal-governance.js"; +import { createGovernanceWrites } from "../governance/event-writes.js"; import { createJournalObjects, JournalCapacityError } from "./journal-objects.js"; import * as journalStore from "./journal-store.js"; import { reviveJournalError, type JournalStatement, type JournalStoreOptions } from "./journal-store.js"; @@ -88,7 +88,7 @@ export function createTurnJournal(db: Database, options: TurnJournalOptions) { await ledger.call("recover"); // Governance events live in the UI database; a commit split between the // two files is completed from the journal's rows (#2789). - replaySteeringEvents(db); + await createGovernanceWrites(db, writer).replaySteering(); const interrupted = (db.query("SELECT turn_id,pid,pid_start FROM turn_journal_turns WHERE ended_at IS NULL").all() as { turn_id: string; pid: number; pid_start: string }[]) .filter(row => !processAlive(row.pid, row.pid_start)); for (const row of interrupted) await ledger.call("statements", [statement("UPDATE turn_journal_turns SET state='partial',ended_at=?,missing=? WHERE turn_id=?", now(), JSON.stringify([{ path: ".", reason: "interrupted_observation" }]), row.turn_id)]); diff --git a/packages/ui-server/src/brain/turn-revert.ts b/packages/ui-server/src/brain/turn-revert.ts index 99d5143ce4..2086ab3f34 100644 --- a/packages/ui-server/src/brain/turn-revert.ts +++ b/packages/ui-server/src/brain/turn-revert.ts @@ -10,7 +10,7 @@ import { journalStatement } from "./journal-store.js"; import type { WriterGate } from "./writer-gate.js"; import { processAlive, SELF } from "./writer-gate.js"; import { reversibleJournalPath, restoreJournalPath, createJournalSibling } from "./revert-files.js"; -import { recordGovernanceEvent } from "../governance/events.js"; +import { createGovernanceWrites, type GovernanceWrites } from "../governance/event-writes.js"; export class RevertUnavailable extends Error {} type Selected = TurnRevertPath & { before: JournalVersion | null; after: JournalVersion | null; beforeSequence: number; afterSequence: number; other?: JournalVersion; otherSequence?: number }; @@ -22,7 +22,10 @@ export function createTurnReverts(db: Database, options: { brainPath: string; journal: TurnJournal; gate: WriterGate; lock: KeyedLock; isAuthorized(principalId: string): boolean; refresh(): void; + /** Where a revert conflict is recorded: the write worker (#2850). Absent: on `db`. */ + governance?: Pick; }) { + const governance = options.governance ?? createGovernanceWrites(db); const journal = options.journal, running = new Map>(), shutdown = new AbortController(); function select(sessionId: string, turnId: string, action: "revert" | "restore"): Selected[] { const details = journal.details(sessionId, turnId); @@ -137,7 +140,7 @@ export function createTurnReverts(db: Database, options: { } } const conflicts = selected.filter(path => path.state === "conflict"); - for (let offset = 0; offset < conflicts.length; offset += 1) recordGovernanceEvent(db, {kind:"revert_conflict",sessionId:row.session_id,turnId:row.journal_turn_id,actor:"user",model:null,profile:source.profile, + for (let offset = 0; offset < conflicts.length; offset += 1) await governance.record({kind:"revert_conflict",sessionId:row.session_id,turnId:row.journal_turn_id,actor:"user",model:null,profile:source.profile, payload:{conflicts:conflicts.slice(offset,offset+1).map(path=>({path:path.path,state:"changed"})),restored:selected.filter(path=>path.state==="restored").length}}); })); await journal.finish(row.journal_turn_id); diff --git a/packages/ui-server/src/brain/turn-scope-writes.ts b/packages/ui-server/src/brain/turn-scope-writes.ts new file mode 100644 index 0000000000..f83713466e --- /dev/null +++ b/packages/ui-server/src/brain/turn-scope-writes.ts @@ -0,0 +1,42 @@ +/** + * Turn scope writes off the event loop (#2850). The hosted server commits + * every `turn_scope` row on the `ui.db` write worker at `synchronous=FULL`, + * in submission order with the journal's, Activity's and governance writes; + * the scope monitor keeps each live turn's summary in memory and reads its + * own rows back only for turns it no longer runs. + */ +import type { Database } from "bun:sqlite"; +import type { TurnScopeSummary } from "@schlessera/brain-ui-sdk/protocol"; + +import type { WriteProcedures } from "../db/write-worker.js"; +import { recordTurnRoadblocks } from "../governance/roadblocks.js"; +import { processAlive } from "./writer-gate.js"; + +export const url = import.meta.url; + +export function setup(db: Database): WriteProcedures { + return { + /** A restart cannot resume a kernel process or turn whose authority ended. */ + recover(now: number) { + for (const row of db.query("SELECT s.turn_id,s.session_id,s.summary,j.actor,j.profile,j.started_at,j.pid,j.pid_start FROM turn_scope s LEFT JOIN turn_journal_turns j ON j.turn_id=s.turn_id WHERE s.ended_at IS NULL").all() as { turn_id: string; session_id: string | null; summary: string; actor: "interactive" | "voice" | "user" | null; profile: "strict" | "governed" | null; started_at: number | null; pid: number | null; pid_start: string | null }[]) { + if (row.pid !== null && row.pid_start !== null && processAlive(row.pid, row.pid_start)) continue; + const summary = JSON.parse(row.summary) as TurnScopeSummary; + for (const receipt of summary.receipts) if (receipt.state === "paused") { + receipt.state = "ended_before_answer"; receipt.reason = "Host restarted; the turn was interrupted."; + } + if (summary.receipts.some(receipt => receipt.state === "ended_before_answer")) recordTurnRoadblocks(db, { turnId: row.turn_id, sessionId: row.session_id, actor: row.actor ?? "interactive", model: null, profile: row.profile }, + { startedAt: row.started_at ?? summary.observedAt, endedAt: now, calls: [], usage: [], finalUsage: null, outcomeDetail: null }); + db.query("UPDATE turn_scope SET summary=?,ended_at=? WHERE turn_id=?").run(JSON.stringify(summary), now, row.turn_id); + } + }, + insert(turnId: string, sessionId: string | null, principalId: string, summary: string) { + db.query("INSERT INTO turn_scope(turn_id,session_id,principal_id,summary) VALUES (?,?,?,?)").run(turnId, sessionId, principalId, summary); + }, + save(turnId: string, sessionId: string | null, summary: string) { + db.query("UPDATE turn_scope SET session_id=?,summary=? WHERE turn_id=?").run(sessionId, summary, turnId); + }, + end(turnId: string, endedAt: number) { + db.query("UPDATE turn_scope SET ended_at=? WHERE turn_id=?").run(endedAt, turnId); + }, + }; +} diff --git a/packages/ui-server/src/brain/turn-scope.ts b/packages/ui-server/src/brain/turn-scope.ts index 7b89d8788c..e4b2a64d02 100644 --- a/packages/ui-server/src/brain/turn-scope.ts +++ b/packages/ui-server/src/brain/turn-scope.ts @@ -5,13 +5,15 @@ import { statfsSync } from "node:fs"; import type { EffectiveGovernancePolicy } from "@schlessera/brain-common/internal/governance-policy"; import type { StartTurnRequest } from "@schlessera/brain-ui-sdk/server"; import type { ClientScopeDecision, ServerScopeDecisionReceipt, ScopeCapReceipt, ScopeDimension, TurnScopeSummary } from "@schlessera/brain-ui-sdk/protocol"; -import { recordTurnRoadblocks } from "../governance/roadblocks.js"; -import { recordGovernanceEvent } from "../governance/events.js"; -import { processAlive } from "./writer-gate.js"; +import type { Logger } from "@opentelemetry/api-logs"; +import type { GovernanceEventInput } from "../governance/events.js"; +import { createGovernanceWrites, type GovernanceWrites } from "../governance/event-writes.js"; +import { procedureCaller, type SqliteWriter } from "../db/write-worker.js"; +import { setup, url } from "./turn-scope-writes.js"; import type { TurnJournal } from "./turn-journal.js"; type Worker = Parameters>[0]; -type Live = { sessionId: string | null; principalId: string; origin: "interactive" | "voice"; profile: "strict" | "governed"; +type Live = { unrecorded?: boolean; sessionId: string | null; principalId: string; origin: "interactive" | "voice"; profile: "strict" | "governed"; startedAt: number; summary: TurnScopeSummary; workers: Set; calls: Set; disconnectedAt?: number; checking?: boolean; source: "policy" | "default"; stop: (reason: string) => void; current: () => boolean; model?: () => string | null }; export const SCOPE_DISCONNECTED_MS = 30 * 60_000; @@ -22,25 +24,47 @@ export function scopeLimits(policy: EffectiveGovernancePolicy) { } export function createTurnScope(db: Database, options: { brainPath: string; journal: TurnJournal; connected: (sessionId: string | null) => boolean; changed: (turnId: string, sessionId: string) => void; - now?: () => number; space?: () => { total: number; free: number } }) { + now?: () => number; space?: () => { total: number; free: number }; + /** Where scope rows and their governance events commit: the write worker (#2850). Absent: on `db`, at once. */ + writer?: SqliteWriter; governance?: Pick; log?: Logger }) { const now = options.now ?? Date.now; const space = options.space ?? (() => { const fs = statfsSync(options.brainPath); return { total: fs.blocks * fs.bsize, free: fs.bavail * fs.bsize }; }); const active = new Map(); - const save = (turnId: string, turn: Live) => { - db.query("UPDATE turn_scope SET session_id=?,summary=? WHERE turn_id=?").run(turn.sessionId, JSON.stringify(turn.summary), turnId); - if (turn.sessionId) options.changed(turnId, turn.sessionId); + const submit = procedureCaller(db, { url, setup }, undefined, options.writer); + // Each turn's latest scope write, until it settles: a decision reads the + // turn's row only once every write submitted before it has committed. + const tails = new Map>(); + const call = (turnId: string, name: string, ...args: unknown[]): Promise => { + const done = submit(name, turnId, ...args); + const tail = done.then(() => {}, () => {}); + tails.set(turnId, tail); + void tail.then(() => { if (tails.get(turnId) === tail) tails.delete(turnId); }); + return done; }; - // A restart cannot resume a kernel process or turn whose authority ended. - for (const row of db.query("SELECT s.turn_id,s.session_id,s.summary,j.actor,j.profile,j.started_at,j.pid,j.pid_start FROM turn_scope s LEFT JOIN turn_journal_turns j ON j.turn_id=s.turn_id WHERE s.ended_at IS NULL").all() as { turn_id: string; session_id: string | null; summary: string; actor: "interactive" | "voice" | "user" | null; profile: "strict" | "governed" | null; started_at: number | null; pid: number | null; pid_start: string | null }[]) { - if (row.pid !== null && row.pid_start !== null && processAlive(row.pid, row.pid_start)) continue; - const summary = JSON.parse(row.summary) as TurnScopeSummary; - for (const receipt of summary.receipts) if (receipt.state === "paused") { - receipt.state = "ended_before_answer"; receipt.reason = "Host restarted; the turn was interrupted."; - } - if (summary.receipts.some(receipt => receipt.state === "ended_before_answer")) recordTurnRoadblocks(db, { turnId: row.turn_id, sessionId: row.session_id, actor: row.actor ?? "interactive", model: null, profile: row.profile }, - { startedAt: row.started_at ?? summary.observedAt, endedAt: now(), calls: [], usage: [], finalUsage: null, outcomeDetail: null }); - db.query("UPDATE turn_scope SET summary=?,ended_at=? WHERE turn_id=?").run(JSON.stringify(summary), now(), row.turn_id); - } + + const governance = options.governance ?? createGovernanceWrites(db, options.writer); + /** + * A scope record that cannot be kept stops its turn: its caps could no + * longer be decided against what was recorded. + */ + const unrecorded = (turnId: string, error: unknown) => { + options.log?.emit({ severityText: "WARN", body: `turn scope not recorded: ${error instanceof Error ? error.message : String(error)}`, attributes: { "turn.id": turnId } }); + const turn = active.get(turnId); + if (!turn || turn.unrecorded || !turn.current()) return; + turn.unrecorded = true; + for (const worker of turn.workers) worker.kill("SIGKILL"); + turn.stop("The turn's scope could not be recorded; the turn stopped. Partial work remains."); + }; + // The summary is taken now; the client hears of it once the row has + // committed, because the journal summary it is sent with reads it back. + const save = (turnId: string, turn: Live): Promise => { + const sessionId = turn.sessionId; + return call(turnId, "save", sessionId, JSON.stringify(turn.summary)).then(() => { if (sessionId) options.changed(turnId, sessionId); }, error => unrecorded(turnId, error)); + }; + const scopeEvent = (turnId: string, input: GovernanceEventInput<"scope_cap">) => { + try { governance.record(input).catch(error => unrecorded(turnId, error)); } catch (error) { unrecorded(turnId, error); } + }; + void submit("recover", now()).catch(error => options.log?.emit({ severityText: "WARN", body: `turn scope recovery failed: ${error instanceof Error ? error.message : String(error)}` })); function stop(turnId: string, turn: Live, receipt: ScopeCapReceipt, state: "stopped" | "voice_stopped" | "ended_before_answer", reason: string) { receipt.state = state; receipt.reason = reason; save(turnId, turn); @@ -78,7 +102,7 @@ export function createTurnScope(db: Database, options: { brainPath: string; jour const receipt: ScopeCapReceipt = { eventId: randomUUID(), dimension: "free_space_floor", count: fs.free, cap: floor, overshoot: floor - fs.free, frozen: pending.frozen, observedAt: now(), state: "paused", reason: "Free space fell below its floor while this turn was paused. Stop only." }; turn.summary.receipts.push(receipt); - recordGovernanceEvent(db, { kind: "scope_cap", turnId, sessionId: turn.sessionId, actor: turn.origin, model: turn.model?.() ?? null, profile: turn.profile, + scopeEvent(turnId, { kind: "scope_cap", turnId, sessionId: turn.sessionId, actor: turn.origin, model: turn.model?.() ?? null, profile: turn.profile, dedupeKey: receipt.eventId, payload: { cap: receipt.dimension, limit: floor, observed: fs.free, overshoot: receipt.overshoot, frozen: receipt.frozen, outcome: "paused", source: "default" } }); } @@ -114,7 +138,7 @@ export function createTurnScope(db: Database, options: { brainPath: string; jour overshoot: Math.max(0, dimension === "free_space_floor" ? cap - observed : observed - cap), frozen, observedAt: now(), state: "paused" }; refreshHeadroom(turn, receipt); turn.summary.receipts.push(receipt); - recordGovernanceEvent(db, { kind: "scope_cap", turnId, sessionId: turn.sessionId, actor: turn.origin, model: turn.model?.() ?? null, profile: turn.profile, + scopeEvent(turnId, { kind: "scope_cap", turnId, sessionId: turn.sessionId, actor: turn.origin, model: turn.model?.() ?? null, profile: turn.profile, dedupeKey: receipt.eventId, payload: { cap: dimension, limit: cap, observed, overshoot: receipt.overshoot, frozen, outcome: turn.origin === "voice" ? "voice_stopped" : frozen ? "paused" : "stopped", source: turn.source } }); if (turn.origin === "voice") stop(turnId, turn, receipt, "voice_stopped", `I stopped because this turn reached its limit of ${cap} ${dimension === "tool_calls" ? "tool calls" : dimension === "minutes" ? "wall minutes" : dimension === "files_changed" ? "distinct changed paths" : "bytes"}${dimension === "free_space_floor" ? " of free space" : ""}. Ask again to carry on.`); @@ -176,8 +200,12 @@ export function createTurnScope(db: Database, options: { brainPath: string; jour const summary: TurnScopeSummary = { policySource: input.source, limits: scopeLimits(input.policy), hardTimeoutMs: input.hardTimeoutMs, calls: 0, minutes: 0, changedPaths: null, observedBytes: null, coverage: "partial", observedAt: now(), receipts: [] }; const turn: Live = { ...input, startedAt: now(), summary, workers: new Set(), calls: new Set() }; - db.query("INSERT INTO turn_scope(turn_id,session_id,principal_id,summary) VALUES (?,?,?,?)").run(turnId, input.sessionId, input.principalId, JSON.stringify(summary)); active.set(turnId, turn); + // The turn starts only once its row has committed; refused, it never runs. + return call(turnId, "insert", input.sessionId, input.principalId, JSON.stringify(summary)).then(() => {}, (error: unknown) => { + if (active.get(turnId) === turn) active.delete(turnId); + throw error; + }); }, workerStarted(turnId: string, worker: Worker) { const turn = active.get(turnId); if (!turn || !turn.current()) { worker.kill("SIGKILL"); return; } @@ -197,48 +225,65 @@ export function createTurnScope(db: Database, options: { brainPath: string; jour turn.calls.add(toolUseId); turn.summary.calls = turn.calls.size; check(turnId, turn); }, isPaused(turnId: string) { return active.get(turnId)?.summary.receipts.some(receipt => receipt.state === "paused") ?? false; }, - decide(msg: ClientScopeDecision, authorized: boolean): ServerScopeDecisionReceipt { - const response = (state: ServerScopeDecisionReceipt["state"], reason?: string): ServerScopeDecisionReceipt => ({ ...msg, type: "scope_decision_receipt", state, ...(reason ? { reason } : {}) }); - if (!authorized) return response("refused", "Current authenticated authority is required."); - const row = db.query("SELECT session_id,summary,ended_at FROM turn_scope WHERE turn_id=?").get(msg.turnId) as { session_id: string | null; summary: string; ended_at: number | null } | null; - if (!row || row.session_id !== msg.sessionId) return response("refused", "This decision does not match the turn and session."); - const summary = JSON.parse(row.summary) as TurnScopeSummary, receipt = summary.receipts.find(receipt => receipt.eventId === msg.eventId); - if (!receipt) return response("refused", "This cap event is not recognized."); - if (receipt.state === "ended_before_answer" || row.ended_at !== null && receipt.state === "paused") return response("ended_before_answer", receipt.reason); - if (receipt.submissionId === msg.submissionId) return response("accepted"); - if (receipt.state !== "paused") return response("answered_elsewhere"); - const turn = active.get(msg.turnId); - if (!turn || !turn.current()) return response("ended_before_answer"); - if (msg.action === "readback") return response("pending"); - if (msg.action === "continue" && receipt.newCap !== turn.summary.receipts.find(item => item.eventId === msg.eventId)?.newCap) - return response("refused", "This continuation offer no longer matches the live turn."); - // A grant cannot bypass a floor that another writer crossed while this worker waited. - if (msg.action === "continue") check(msg.turnId, turn); - if (!turn.current()) return response("ended_before_answer"); - const latestPause = turn.summary.receipts.findLast(receipt => receipt.state === "paused"); - if (msg.action === "continue" && latestPause?.eventId !== msg.eventId) return response("refused", "The free-space floor now requires Stop only."); - const liveReceipt = turn.summary.receipts.find(receipt => receipt.eventId === msg.eventId)!; - if (msg.action === "continue" && (liveReceipt.newCap === undefined || turn.checking || observationHolds > 0)) - return response("refused", liveReceipt.reason ?? "This limit permits Stop only while continuation headroom is unconfirmed."); - try { - if (msg.action === "continue") { - // Grant is consumed before resumption; a failed resume cannot issue another increment. - turn.summary.limits[liveReceipt.dimension as Exclude] = liveReceipt.newCap!; - liveReceipt.state = "continued"; liveReceipt.submissionId = msg.submissionId; save(msg.turnId, turn); - if (observationHolds === 0) for (const worker of turn.workers) worker.resume(); - } else { liveReceipt.submissionId = msg.submissionId; stop(msg.turnId, turn, liveReceipt, "stopped", "You stopped the turn at its scope limit."); } - return response("accepted"); - } catch (error) { - stop(msg.turnId, turn, liveReceipt, "ended_before_answer", "The host could not resume the worker; it stopped. Partial work remains."); - return response("failed", String(error)); - } + /** + * Settles once every scope write the turn made before the decision has + * committed, the decision has run against its committed row, and the + * writes the decision made have committed too (#2850). + */ + async decide(msg: ClientScopeDecision, authorized: boolean): Promise { + if (!authorized) return { ...msg, type: "scope_decision_receipt", state: "refused", reason: "Current authenticated authority is required." }; + // Checked again after every wait, with nothing awaited between the last + // check and the decision: a concurrent one waits for this one's writes. + for (let tail = tails.get(msg.turnId); tail; tail = tails.get(msg.turnId)) await tail; + const receipt = decideCommitted(msg); + for (let tail = tails.get(msg.turnId); tail; tail = tails.get(msg.turnId)) await tail; + return receipt; }, - finish(turnId: string) { - const turn = active.get(turnId); if (!turn) return; + /** Settles once the turn's final summary and end have committed. */ + finish(turnId: string): Promise { + const turn = active.get(turnId); if (!turn) return Promise.resolve(); for (const receipt of turn.summary.receipts) if (receipt.state === "paused") { receipt.state = "ended_before_answer"; receipt.reason = "The turn ended before an answer."; } - save(turnId, turn); db.query("UPDATE turn_scope SET ended_at=? WHERE turn_id=?").run(now(), turnId); active.delete(turnId); + const saved = save(turnId, turn), ended = call(turnId, "end", now()).catch(error => unrecorded(turnId, error)); + active.delete(turnId); + return Promise.all([saved, ended]).then(() => {}); }, close() { unsubscribe(); for (const [id, turn] of active) { for (const worker of turn.workers) worker.kill("SIGKILL"); turn.stop("Host closed; the turn was interrupted."); this.finish(id); } }, }; + /** A decision, synchronous from its read of the committed row through the writes it submits. */ + function decideCommitted(msg: ClientScopeDecision): ServerScopeDecisionReceipt { + const response = (state: ServerScopeDecisionReceipt["state"], reason?: string): ServerScopeDecisionReceipt => ({ ...msg, type: "scope_decision_receipt", state, ...(reason ? { reason } : {}) }); + const row = db.query("SELECT session_id,summary,ended_at FROM turn_scope WHERE turn_id=?").get(msg.turnId) as { session_id: string | null; summary: string; ended_at: number | null } | null; + if (!row || row.session_id !== msg.sessionId) return response("refused", "This decision does not match the turn and session."); + const summary = JSON.parse(row.summary) as TurnScopeSummary, receipt = summary.receipts.find(receipt => receipt.eventId === msg.eventId); + if (!receipt) return response("refused", "This cap event is not recognized."); + if (receipt.state === "ended_before_answer" || row.ended_at !== null && receipt.state === "paused") return response("ended_before_answer", receipt.reason); + if (receipt.submissionId === msg.submissionId) return response("accepted"); + if (receipt.state !== "paused") return response("answered_elsewhere"); + const turn = active.get(msg.turnId); + if (!turn || !turn.current()) return response("ended_before_answer"); + if (msg.action === "readback") return response("pending"); + if (msg.action === "continue" && receipt.newCap !== turn.summary.receipts.find(item => item.eventId === msg.eventId)?.newCap) + return response("refused", "This continuation offer no longer matches the live turn."); + // A grant cannot bypass a floor that another writer crossed while this worker waited. + if (msg.action === "continue") check(msg.turnId, turn); + if (!turn.current()) return response("ended_before_answer"); + const latestPause = turn.summary.receipts.findLast(receipt => receipt.state === "paused"); + if (msg.action === "continue" && latestPause?.eventId !== msg.eventId) return response("refused", "The free-space floor now requires Stop only."); + const liveReceipt = turn.summary.receipts.find(receipt => receipt.eventId === msg.eventId)!; + if (msg.action === "continue" && (liveReceipt.newCap === undefined || turn.checking || observationHolds > 0)) + return response("refused", liveReceipt.reason ?? "This limit permits Stop only while continuation headroom is unconfirmed."); + try { + if (msg.action === "continue") { + // Grant is consumed before resumption; a failed resume cannot issue another increment. + turn.summary.limits[liveReceipt.dimension as Exclude] = liveReceipt.newCap!; + liveReceipt.state = "continued"; liveReceipt.submissionId = msg.submissionId; save(msg.turnId, turn); + if (observationHolds === 0) for (const worker of turn.workers) worker.resume(); + } else { liveReceipt.submissionId = msg.submissionId; stop(msg.turnId, turn, liveReceipt, "stopped", "You stopped the turn at its scope limit."); } + return response("accepted"); + } catch (error) { + stop(msg.turnId, turn, liveReceipt, "ended_before_answer", "The host could not resume the worker; it stopped. Partial work remains."); + return response("failed", String(error)); + } + } } export type TurnScope = ReturnType; diff --git a/packages/ui-server/src/cron/scratch-prune.ts b/packages/ui-server/src/cron/scratch-prune.ts index 54b6e7f7ce..10f2c7ecd8 100644 --- a/packages/ui-server/src/cron/scratch-prune.ts +++ b/packages/ui-server/src/cron/scratch-prune.ts @@ -1,5 +1,5 @@ import type { Database } from "bun:sqlite"; -import { recordGovernanceEvent } from "../governance/events.js"; +import { createGovernanceWrites, type GovernanceWrites } from "../governance/event-writes.js"; import type { Logger } from "@opentelemetry/api-logs"; import type { BrainClient, MaintenanceRequest } from "../brain/client.js"; @@ -41,13 +41,17 @@ export type Every = (check: () => void, ms: number) => () => void; export function startScratchPrune(options: { brain: Pick; log?: Logger; + /** Where a scratch overage is recorded: on this connection, at once. */ db?: Database; + /** Where a scratch overage is recorded, in preference to `db`: the hosted server's write worker (#2850). */ + governance?: Pick; pins?: { file: string; refresh(signal: AbortSignal): Promise }; every?: Every; intervalMs?: number; deadlineMs?: number; }): ScratchPruner { const { brain, log } = options; + const governance = options.governance ?? (options.db ? createGovernanceWrites(options.db) : undefined); const deadlineMs = options.deadlineMs ?? SCRATCH_PRUNE_DEADLINE_MS; let inFlight: Promise | null = null; let active: AbortController | null = null; @@ -79,14 +83,14 @@ export function startScratchPrune(options: { request?.assertAuthorized?.(); return brain.scratchPrune({ ...(request ?? {}), signal: controller.signal, ...(options.pins ? { pinsFile: options.pins.file } : {}) }); })() - .then((output) => { + .then(async (output) => { try { const report = JSON.parse(output) as { bytes: number; overageBytes?: number; pinned: { files: number; bytes: number } }; if (report.overageBytes && report.overageBytes > 0) { let eventError: unknown; - if (options.db) { + if (governance) { try { - recordGovernanceEvent(options.db, { kind: "scratch_overage", actor: "server", sessionId: null, + await governance.record({ kind: "scratch_overage", actor: "server", sessionId: null, turnId: null, model: null, profile: null, payload: { bytes: report.bytes, pinnedFiles: report.pinned.files, pinnedBytes: report.pinned.bytes, overageBytes: report.overageBytes } }); return output; diff --git a/packages/ui-server/src/db/write-worker.ts b/packages/ui-server/src/db/write-worker.ts index 622bbb6b78..5ab7af92c3 100644 --- a/packages/ui-server/src/db/write-worker.ts +++ b/packages/ui-server/src/db/write-worker.ts @@ -46,6 +46,26 @@ export interface SqliteWriter { close(): Promise; } +/** Calls one module's procedures by name; each call settles once its write has committed. */ +export type ProcedureCall = (name: string, ...args: unknown[]) => Promise; + +/** + * A module's procedures on `writer` (#2850). Without one (a process that + * commits its own writes, or a test), the same procedures run on `db` at + * once, behind the same promises. + */ +export function procedureCaller(db: Database, module: WriteModule, options: O, writer?: SqliteWriter, + revive?: (error: WireError) => Error | undefined): ProcedureCall { + if (writer) { + const channel = writer.channel(module, options, revive); + return (name, ...args) => channel.call(name, ...args); + } + const procedures = module.setup(db, options); + return (name: string, ...args: unknown[]) => { + try { return Promise.resolve(Reflect.apply(procedures[name]!, procedures, args) as T); } catch (error) { return Promise.reject(error); } + }; +} + export class SqliteWriterClosed extends Error { constructor(message = "The SQLite write worker is closed.") { super(message); } } diff --git a/packages/ui-server/src/governance/event-writes.ts b/packages/ui-server/src/governance/event-writes.ts new file mode 100644 index 0000000000..73dfc64222 --- /dev/null +++ b/packages/ui-server/src/governance/event-writes.ts @@ -0,0 +1,82 @@ +/** + * Governance event writes off the event loop (#2850). The hosted server's + * app connection never commits a governance event itself: each one is + * validated where it is raised, then committed on the `ui.db` write worker at + * `synchronous=FULL`, in submission order with the journal's and Activity's + * writes. An event the journal records inside its own transaction stays + * there (`brain/journal-store.ts`). + */ +import type { Database } from "bun:sqlite"; + +import { procedureCaller, type SqliteWriter, type WriteProcedures } from "../db/write-worker.js"; +import { replaySteeringEvents } from "../brain/journal-governance.js"; +import { bindTurnEvents, governanceEventRow, insertGovernanceEvent, type GovernanceEventInput, type GovernanceEventRow } from "./events.js"; +import { recordTurnRoadblocks, type RoadblockAttribution, type TurnTrace } from "./roadblocks.js"; +import { insertFirstTaint, sessionTaint, type SessionTaint } from "./session-taint.js"; + +export const url = import.meta.url; + +/** Worker side: every procedure is one of the synchronous writers, run on the write connection. */ +export function setup(db: Database): WriteProcedures { + return { + record: (row: GovernanceEventRow) => insertGovernanceEvent(db, row), + firstTaint: (row: GovernanceEventRow) => insertFirstTaint(db, row), + bind: (turnId: string, sessionId: string) => bindTurnEvents(db, turnId, sessionId), + // Classification reads the turn's earlier events on the same connection, + // after every event submitted before it has committed. + roadblocks: (attribution: RoadblockAttribution, trace: TurnTrace) => recordTurnRoadblocks(db, attribution, trace), + replaySteering: () => replaySteeringEvents(db), + }; +} + +export interface GovernanceWrites { + /** + * Validates now, throwing `GovernanceEventError` as the synchronous writer + * did; the returned promise settles once the row commits, or with the + * database failure. + */ + record(input: GovernanceEventInput): Promise<{ id: number; recorded: boolean }>; + /** The session's first taint, checked and stored in one transaction; whether this call stored it. Validates now. */ + firstTaint(input: GovernanceEventInput<"taint">): Promise; + /** + * The session's first taint (G6) as admission must see it: the committed + * one, else one submitted and not yet committed. A pending taint has no row + * id yet (`eventId` 0). Taint is sticky, so a taint that later fails to + * commit stops its turn rather than loosening anything. + */ + taint(turnId: string | null, sessionId: string | null): SessionTaint | null; + /** Attaches a turn's session-less events to its session once its catalog row exists. */ + bind(turnId: string, sessionId: string): Promise; + /** Classifies a finished turn against its committed events and stores its roadblocks. */ + roadblocks(attribution: RoadblockAttribution, trace: TurnTrace): Promise; + /** Records any `steering_change` event a journal steering change lacks (#2789). */ + replaySteering(): Promise; +} + +/** The app side: every write commits on `writer` when there is one, else on `db` at once. */ +export function createGovernanceWrites(db: Database, writer?: SqliteWriter): GovernanceWrites { + const call = procedureCaller(db, { url, setup }, undefined, writer); + const pendingTaints = new Set<{ turnId: string | null; sessionId: string | null; source: SessionTaint["source"] }>(); + return { + record: input => call<{ id: number; recorded: boolean }>("record", governanceEventRow(input)), + firstTaint: input => { + const row = governanceEventRow(input); + const pending = { turnId: row.turnId, sessionId: row.sessionId, source: input.payload.source }; + pendingTaints.add(pending); + const stored = call("firstTaint", row); + void stored.finally(() => pendingTaints.delete(pending)).catch(() => {}); + return stored; + }, + taint(turnId, sessionId) { + const committed = sessionTaint(db, turnId, sessionId); + if (committed) return committed; + for (const pending of pendingTaints) { + if ((turnId !== null && pending.turnId === turnId) || (sessionId !== null && pending.sessionId === sessionId)) return { eventId: 0, source: pending.source }; + } + return null; + }, + bind: (turnId, sessionId) => call("bind", turnId, sessionId), + roadblocks: (attribution, trace) => call("roadblocks", attribution, trace), + replaySteering: () => call("replaySteering"), + }; +} diff --git a/packages/ui-server/src/governance/events.ts b/packages/ui-server/src/governance/events.ts index ff647457ea..5f200484b1 100644 --- a/packages/ui-server/src/governance/events.ts +++ b/packages/ui-server/src/governance/events.ts @@ -305,17 +305,19 @@ export function isGovernanceEventKind(value: string): value is GovernanceEventKi return Object.hasOwn(GOVERNANCE_EVENT_SCHEMAS, value); } +/** A validated event as it is stored: everything but the row id. */ +export interface GovernanceEventRow { + sessionId: string | null; turnId: string | null; at: number; kind: GovernanceEventKind; version: number; + actor: GovernanceActor; model: string | null; profile: GovernanceProfile | null; payload: string; dedupeKey: string | null; +} + /** - * The one writer. Throws `GovernanceEventError` on an unknown kind, a payload - * that does not match its kind's schema, bad attribution or an oversized - * payload; database failures propagate. Returns the row id, and whether this - * call stored it (false when `dedupeKey` already recorded the event). - * - * The session is attached only when its catalog row exists, so an emitter - * never has to know whether `session_info` has landed yet; `bindTurnEvents` - * attaches the rest when it does. + * Validates one event and builds its row. Throws `GovernanceEventError` on an + * unknown kind, a payload that does not match its kind's schema, bad + * attribution or an oversized payload. The clock default is taken here, when + * the event is submitted, not when its commit runs. */ -export function recordGovernanceEvent(db: Database, input: GovernanceEventInput): { id: number; recorded: boolean } { +export function governanceEventRow(input: GovernanceEventInput): GovernanceEventRow { if (typeof input?.kind !== "string" || !isGovernanceEventKind(input.kind)) { throw new GovernanceEventError(`Unknown governance event kind: ${String(input?.kind)}`); } @@ -333,22 +335,45 @@ export function recordGovernanceEvent(db: Database, input: GovernanceEventInput) throw new GovernanceEventError(`${input.kind} payload exceeds ${GOVERNANCE_PAYLOAD_MAX_BYTES} bytes`); } const m = meta.data; - const at = m.at ?? Date.now(); + return { sessionId: m.sessionId, turnId: m.turnId, at: m.at ?? Date.now(), kind: input.kind, version: spec.version, + actor: m.actor, model: m.model, profile: m.profile, payload: json, dedupeKey: m.dedupeKey ?? null }; +} + +/** + * Stores one validated row in its own immediate transaction. Database + * failures propagate. Returns the row id, and whether this call stored it + * (false when `dedupeKey` already recorded the event). + * + * The session is attached only when its catalog row exists, so an emitter + * never has to know whether `session_info` has landed yet; `bindTurnEvents` + * attaches the rest when it does. + */ +export function insertGovernanceEvent(db: Database, row: GovernanceEventRow): { id: number; recorded: boolean } { return db.transaction(() => { - db.query("DELETE FROM governance_events WHERE session_id IS NULL AND at < ?").run(at - UNBOUND_GOVERNANCE_RETENTION_MS); - const row = db.query(`INSERT INTO governance_events + db.query("DELETE FROM governance_events WHERE session_id IS NULL AND at < ?").run(row.at - UNBOUND_GOVERNANCE_RETENTION_MS); + const stored = db.query(`INSERT INTO governance_events (session_id, turn_id, at, kind, schema_version, actor, model, profile, payload, dedupe_key) VALUES ((SELECT id FROM sessions WHERE id = ?1), ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10) ON CONFLICT(kind, dedupe_key) WHERE dedupe_key IS NOT NULL DO NOTHING RETURNING id`) - .get(m.sessionId, m.turnId, at, input.kind, spec.version, m.actor, m.model, m.profile, json, m.dedupeKey ?? null) as { id: number } | null; - if (row) return { id: row.id, recorded: true }; + .get(row.sessionId, row.turnId, row.at, row.kind, row.version, row.actor, row.model, row.profile, row.payload, row.dedupeKey) as { id: number } | null; + if (stored) return { id: stored.id, recorded: true }; const existing = db.query("SELECT id FROM governance_events WHERE kind = ? AND dedupe_key = ?") - .get(input.kind, m.dedupeKey ?? null) as { id: number }; + .get(row.kind, row.dedupeKey) as { id: number }; return { id: existing.id, recorded: false }; }).immediate(); } +/** + * The one synchronous writer, for a connection that commits its own writes: + * the write worker's procedures and the cron runner. The hosted server's app + * connection submits through `governance/event-writes.ts` instead (#2850). + * Throws as `governanceEventRow` and `insertGovernanceEvent` do. + */ +export function recordGovernanceEvent(db: Database, input: GovernanceEventInput): { id: number; recorded: boolean } { + return insertGovernanceEvent(db, governanceEventRow(input)); +} + /** * Attach a turn's session-less events to its session, once the session's * catalog row exists. Returns how many rows it bound. diff --git a/packages/ui-server/src/governance/session-taint.ts b/packages/ui-server/src/governance/session-taint.ts index 1a15de7ad5..e88792657c 100644 --- a/packages/ui-server/src/governance/session-taint.ts +++ b/packages/ui-server/src/governance/session-taint.ts @@ -1,6 +1,6 @@ /** A session's untrusted-input state (G6), read from and written to its durable governance record. */ import type { Database } from "bun:sqlite"; -import type { GovernanceEventInput, GovernanceEventPayload } from "./events.js"; +import { insertGovernanceEvent, type GovernanceEventInput, type GovernanceEventPayload, type GovernanceEventRow } from "./events.js"; export type TaintSource = GovernanceEventPayload<"taint">["source"]; export interface SessionTaint { eventId: number; source: TaintSource } @@ -13,14 +13,27 @@ export function sessionTaint(db: Database, turnId: string | null, sessionId: str return row ? { eventId: row.id, source: (JSON.parse(row.payload) as { source: TaintSource }).source } : null; } -/** Same SQLite transaction owns the first-source check and durable write. */ +/** Same SQLite transaction owns the first-source check and durable write, on a connection that commits its own writes. */ export function recordFirstTaint(db: Database, event: GovernanceEventInput<"taint">, - record: (event: GovernanceEventInput<"taint">) => void): void { + record: (event: GovernanceEventInput<"taint">) => unknown): void { db.transaction(() => { if (!db.query(FIRST_TAINT).get(event.turnId ?? null, event.sessionId ?? null, event.sessionId ?? null)) record(event); })(); } +/** + * `recordFirstTaint` for an already validated row, as the hosted server's + * write worker runs it (`governance/event-writes.ts`, #2850). Returns whether + * this call recorded the taint. + */ +export function insertFirstTaint(db: Database, row: GovernanceEventRow): boolean { + return db.transaction(() => { + if (db.query(FIRST_TAINT).get(row.turnId, row.sessionId, row.sessionId)) return false; + insertGovernanceEvent(db, row); + return true; + }).immediate(); +} + /** * Hosts the session's turn proxies admitted before, excluding the inference * relay. The proxy charges bytes only after admission, so a charged record is diff --git a/packages/ui-server/src/inbox/autonomous-turn.ts b/packages/ui-server/src/inbox/autonomous-turn.ts index 757e5f5a82..1f8cec6732 100644 --- a/packages/ui-server/src/inbox/autonomous-turn.ts +++ b/packages/ui-server/src/inbox/autonomous-turn.ts @@ -24,7 +24,8 @@ import { privateIndexBuildFolders } from "../brain/private-index-cache.js"; import { readsIndex, type IndexBatch } from "../brain/index-batch.js"; import { createHostBoundary } from "../brain/host-boundary.js"; import { applicationToolName, inboxOperationAllowed } from "./envelope.js"; -import { recordGovernanceEvent } from "../governance/events.js"; +import type { GovernanceEventInput } from "../governance/events.js"; +import { createGovernanceWrites } from "../governance/event-writes.js"; import { governancePolicyReaderFor, ProviderPolicyRefusedError, providerDeniedTypes, providerFolderRoots, providerMaskedFolders, providerTypeSelectors, unsupportedProviderSelectors, type TurnGovernancePolicy } from "../brain/governance-policy.js"; import { createProviderMaskGuard, type ProviderMaskGuard } from "../brain/provider-mask-refusal.js"; import { translateProviderTypesInView } from "../brain/provider-types.js"; @@ -172,9 +173,12 @@ async function executeAutonomousTurn( if (!origin) throw new BackendRequestError("Autonomous inbox origin is unavailable."); // Own immutable request policy for the attempt; callbacks cannot widen it. // Governance events (#1651) and roadblocks (#1671) go to the same database as the attempt. - const governance = deps.governance ?? ((event: Parameters[1]) => { recordGovernanceEvent(deps.db, event); }); + const governanceWrites = deps.governanceWrites ?? createGovernanceWrites(deps.governanceDb ?? deps.db); + const governance = deps.governance ?? ((event: GovernanceEventInput) => governanceWrites.record(event).then(() => {}, (error: unknown) => { + deps.log?.emit({ severityText: "WARN", body: `governance event not recorded: ${error instanceof Error ? error.message : String(error)}` }); + })); const recorder = createTurnRecorder({ ...deps, governance, - governanceDb: deps.governanceDb ?? deps.db }, + governanceDb: deps.governanceDb ?? deps.db, governanceWrites }, { turnId: input.turnId, sessionId: null, principalId, origin: "autonomous", profileId: input.profileId, billingMode: input.billingMode, pricingRoute: input.pricingRoute }); diff --git a/packages/ui-server/src/inbox/dispatcher.ts b/packages/ui-server/src/inbox/dispatcher.ts index a9c2f92962..6441a687a8 100644 --- a/packages/ui-server/src/inbox/dispatcher.ts +++ b/packages/ui-server/src/inbox/dispatcher.ts @@ -30,7 +30,7 @@ export interface InboxDispatchSelection { maximumTokens?: InboxBudgetOperation["maximumTokens"]; } -export interface InboxDispatcherDeps extends Pick { +export interface InboxDispatcherDeps extends Pick { nativeManager?: AutonomousTurnDeps["nativeManager"]; privateIndex?: AutonomousTurnDeps["privateIndex"]; brainRoot: string; @@ -207,7 +207,7 @@ export function createInboxDispatcher(db: Database, deps: InboxDispatcherDeps) { let result: Awaited> | null = null; try { result = await runAutonomousTurn({ db, backend: chosen.backend, brainPath: deps.brainRoot, identitySentinelDirectory: deps.identitySentinelDirectory, nativeManager: deps.nativeManager, privateIndex: deps.privateIndex, store: deps.store, runtime: deps.runtime, - onWrite: deps.onWrite, log: deps.log, applicationPlanner: deps.applicationPlanner, indexBatch: deps.indexBatch, + ...(deps.governanceWrites ? { governanceWrites: deps.governanceWrites } : {}), onWrite: deps.onWrite, log: deps.log, applicationPlanner: deps.applicationPlanner, indexBatch: deps.indexBatch, checkpoint: (escalation) => { const current = store.getItem(item.id)!; const { action, allowed } = decision(item, backendId, escalation); diff --git a/packages/ui-server/src/schedules/admission.ts b/packages/ui-server/src/schedules/admission.ts index 005175d71f..167edeaeb2 100644 --- a/packages/ui-server/src/schedules/admission.ts +++ b/packages/ui-server/src/schedules/admission.ts @@ -34,7 +34,7 @@ export interface AdmissionTimers { clearTimeout(timer: Timer): void; } -export interface ScheduleAdmissionDeps extends Pick { +export interface ScheduleAdmissionDeps extends Pick { service: ScheduleService; /** The backend the approved execution policy names. */ backend: AgentBackend; @@ -409,6 +409,7 @@ export function createScheduleAdmission(db: Database, deps: ScheduleAdmissionDep try { result = await runAutonomousTurn({ db, backend: deps.backend, store: deps.store, runtime: deps.runtime, onWrite: deps.onWrite, log: deps.log, + ...(deps.governanceWrites ? { governanceWrites: deps.governanceWrites } : {}), identitySentinelDirectory: deps.identitySentinelDirectory, nativeManager: deps.nativeManager, privateIndex: deps.privateIndex, brainPath: deps.brainPath, applicationPlanner: deps.applicationPlanner, indexBatch: deps.indexBatch, // No durable decision path is wired for scheduled work yet: a run that // needs one fails closed instead of waiting on a live promise. diff --git a/packages/ui-server/src/ws/dispatch.ts b/packages/ui-server/src/ws/dispatch.ts index 69539a5129..d943f6e5f0 100644 --- a/packages/ui-server/src/ws/dispatch.ts +++ b/packages/ui-server/src/ws/dispatch.ts @@ -293,7 +293,8 @@ export async function handleClientMessage( } case "scope_decision": { - const receipt = host.scope?.decide(msg, connection.authorization.valid && connection.authorization.expiresAt > Date.now()) + // Resolves once the decision's own scope writes have committed, so the summary below reads them (#2850). + const receipt = await host.scope?.decide(msg, connection.authorization.valid && connection.authorization.expiresAt > Date.now()) ?? { ...msg, type: "scope_decision_receipt" as const, state: "refused" as const, reason: "This host cannot answer scope decisions." }; if (receipt.state !== "refused" && host.journal) host.sendMessage(ws, { type: "turn_journal", sessionId: msg.sessionId, turnId: msg.turnId, journal: { ...host.journal.summary(msg.turnId), ...(host.budgets?.saved(msg.turnId) ? { budget: host.budgets.saved(msg.turnId) } : {}) } }); host.sendMessage(ws, receipt); diff --git a/packages/ui-server/src/ws/host.ts b/packages/ui-server/src/ws/host.ts index 8d1e00b076..13956ccdb5 100644 --- a/packages/ui-server/src/ws/host.ts +++ b/packages/ui-server/src/ws/host.ts @@ -39,10 +39,12 @@ export interface ActivityRuntime { query?: (query: import("@schlessera/brain-ui-sdk/server").ActivityQuery) => Record; /** Where each turn's runtime report and auth failures are kept for /api/status. */ runtime?: Pick & Partial>; - /** The governance-event writer (#1651) the turn recorders emit through. */ - governance?: (input: GovernanceEventInput) => void; - /** The server database each turn's roadblocks are classified and recorded in (#1671); absent records none. */ + /** The governance-event writer (#1651) the turn recorders emit through. A returned promise is its commit (#2850). */ + governance?: (input: GovernanceEventInput) => unknown; + /** The server database each turn's roadblocks are classified against and its taint read from (#1671); absent records none. */ governanceDb?: import("bun:sqlite").Database; + /** Where governance rows commit: the write worker, off the event loop (#2850). Absent: on `governanceDb`. */ + governanceWrites?: import("../governance/event-writes.js").GovernanceWrites; } /** Host-side turn timeout. The backend no longer times out — the host owns it. */ diff --git a/packages/ui-server/src/ws/run-session.ts b/packages/ui-server/src/ws/run-session.ts index 80b7f592a6..c64de4d5ec 100644 --- a/packages/ui-server/src/ws/run-session.ts +++ b/packages/ui-server/src/ws/run-session.ts @@ -443,6 +443,7 @@ async function runRetainedSession( onWrite: () => host.activity!.stream.pump(), log: host.log, ...(host.activity.governanceDb ? { governanceDb: host.activity.governanceDb } : {}), + ...(host.activity.governanceWrites ? { governanceWrites: host.activity.governanceWrites } : {}), ...(host.brainPath ? { brainPath: host.brainPath } : {}), abort: error => abortController.abort(error), }, @@ -583,7 +584,7 @@ async function runRetainedSession( generation: worktreeGeneration ?? null, signal: abortController.signal }); } const scopeTurnId = turn.turnId; - if (governedHold && host.scope) host.scope.begin(scopeTurnId, { sessionId: turn.sessionId ?? null, principalId: turn.principalId, + if (governedHold && host.scope) await host.scope.begin(scopeTurnId, { sessionId: turn.sessionId ?? null, principalId: turn.principalId, origin, model: () => mainModel, profile: governedHold.sandbox.profile === "governed" ? "governed" : "strict", policy: governedHold.policy.policy, source: governedHold.policy.source === "default" ? "default" : "policy", hardTimeoutMs: host.turnTimeoutMs, current: () => turn.turnId === scopeTurnId && !abortController.signal.aborted && current.authorization.valid && current.authorization.expiresAt > Date.now() && host.isPrincipalAuthorized(current.principalId), @@ -713,11 +714,13 @@ async function runRetainedSession( if (governedHold?.sandbox.inferenceRecipient && !(governedHold.managedStorage && relayed)) throw new Error("This turn's provider folders were admitted for a recipient its inference relay cannot enforce; retry the turn."); if (governedHold?.managedStorage && relayed) { - const governance = host.activity?.governance; + const governance = host.activity?.governance, governanceWrites = host.activity?.governanceWrites; turnEgress = await prepareTurnEgress({ policy: governedHold.policy.policy, bridge, controller: abortController, turnId: turn.turnId, sessionId: () => turn.sessionId ?? null, model: () => mainModel, profile: governedHold.sandbox.profile === "governed" ? "governed" : "strict", origin, - ...(governance ? { record: (event: Parameters[0]) => governance(event) } : {}), + // A record's commit failure stops the turn, as a synchronous one did (#2850). + ...(governanceWrites ? { record: (event: Parameters[0]) => governanceWrites.record(event), governanceWrites } + : governance ? { record: (event: Parameters[0]) => governance(event) } : {}), ...(host.activity?.governanceDb ? { governanceDb: host.activity.governanceDb } : {}), }); // The CLI's paid providers reach the network only through this relay, @@ -753,7 +756,9 @@ async function runRetainedSession( // The session's recorded untrusted input, read live by outward admission and the note (G6). const governanceDb = host.activity?.governanceDb; const commandPolicy = governedHold?.sandbox.commandPolicy && governanceDb ? { ...governedHold.sandbox.commandPolicy, - taint: () => sessionTaint(governanceDb, turn.turnId, turn.sessionId ?? null) } : governedHold?.sandbox.commandPolicy; + // A taint submitted to the write worker counts before it commits (#2850). + taint: () => host.activity?.governanceWrites ? host.activity.governanceWrites.taint(turn.turnId, turn.sessionId ?? null) + : sessionTaint(governanceDb, turn.turnId, turn.sessionId ?? null) } : governedHold?.sandbox.commandPolicy; enteredBackend = true; turn.backendStarted = true; lifecycle?.mark("backendInvokedAt"); @@ -799,6 +804,8 @@ async function runRetainedSession( // Every relayed attempt reports before the turn's accounting closes. await providerRelay?.stop(); turnEgress?.stop(); + // Every egress record commits, or fails the turn, before its outcome (#2850). + await turnEgress?.settled(); const egressFailure = turnEgress?.recordingFailure(); if (egressFailure) throw egressFailure; // Final observation is part of this host outcome, after worker writes @@ -840,6 +847,7 @@ async function runRetainedSession( } catch (err) { lifecycle?.mark("failureObservedAt"); await settleIndex?.(); + await turnEgress?.settled(); const egressFailure = turnEgress?.recordingFailure(); const terminal = budgetFailure ?? journalFailure ?? egressFailure ?? err; if (terminal instanceof BudgetExceeded && terminal.stage === "admission") { @@ -869,6 +877,8 @@ async function runRetainedSession( } finally { await providerRelay?.stop(); turnEgress?.stop(); + // The session's next turn reads these hosts back as contacted (G6). + await turnEgress?.settled(); if (referenceTurn) host.referenceTools?.end(turn.turnId); // No grant after this point: the final observation covers everything granted. if (worktreeGeneration) await host.referenceWorktrees!.retireGeneration(worktreeGeneration.id).catch(() => {}); diff --git a/packages/ui-server/src/ws/session-catalog.ts b/packages/ui-server/src/ws/session-catalog.ts index e24b818f52..ed2e40f5c9 100644 --- a/packages/ui-server/src/ws/session-catalog.ts +++ b/packages/ui-server/src/ws/session-catalog.ts @@ -29,7 +29,8 @@ import { import * as retries from "./retry-requests.js"; import * as work from "./session-work.js"; import { attachTurnFailures, saveTurnFailure, type RecordedTurnFailure } from "./turn-failures.js"; -import { bindTurnEvents, recordGovernanceEvent, type GovernanceEventInput } from "../governance/events.js"; +import type { GovernanceEventInput } from "../governance/events.js"; +import { createGovernanceWrites, type GovernanceWrites } from "../governance/event-writes.js"; /** * Persistence seam for session ownership + accounting. The ws coordinator only @@ -162,8 +163,11 @@ const UPSERT_SESSION_SQL = `INSERT INTO sessions (id, title, created_at, last_ac /** SQLite-backed catalog over the app's own database (injected by createApp). */ export function createSessionCatalog(db: () => Database, log?: Logger, scratchPins?: { record(sessionId: string, backendId: string, messages: SessionHistoryMessage[]): void }, - accounting?: { pricing: RollupPricing; store: AwaitableActivityStore }): SessionCatalog { + accounting?: { pricing: RollupPricing; store: AwaitableActivityStore; governance?: Pick }): SessionCatalog { let accountingDeps = accounting; + // Governance rows commit on the write worker when the app has one (#2850). + let governanceWrites = accounting?.governance; + const governance = () => governanceWrites ??= createGovernanceWrites(db()); const getAccounting = () => accountingDeps ??= (() => { const pricing = createModelPricing(resolveStandalonePricingConfig()); return { pricing, store: createActivityStore(db(), { pricing }) }; @@ -464,13 +468,12 @@ export function createSessionCatalog(db: () => Database, log?: Logger, scratchPi }, bindGovernanceEvents(turnId, sessionId) { - try { bindTurnEvents(db(), turnId, sessionId); } + try { governance().bind(turnId, sessionId).catch((err: unknown) => reportWriteFailure(sessionId, err)); } catch (err) { reportWriteFailure(sessionId, err); } }, recordGovernanceEvent(event) { - try { recordGovernanceEvent(db(), event); } - catch (err) { + const notRecorded = (err: unknown) => { log?.emit({ severityText: "WARN", body: "governance event not recorded", @@ -481,7 +484,9 @@ export function createSessionCatalog(db: () => Database, log?: Logger, scratchPi error: err instanceof Error ? err.message : String(err), }, }); - } + }; + try { governance().record(event).catch(notRecorded); } + catch (err) { notRecorded(err); } }, addSessionCost(sessionId, costUsd) { diff --git a/packages/ui-server/tests/app-writes-offloop.test.ts b/packages/ui-server/tests/app-writes-offloop.test.ts new file mode 100644 index 0000000000..95808be93d --- /dev/null +++ b/packages/ui-server/tests/app-writes-offloop.test.ts @@ -0,0 +1,107 @@ +import { expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { resolveGovernancePolicy } from "@schlessera/brain-common/internal/governance-policy"; +import { createUiDb } from "../src/db/client.js"; +import { openSqliteWriter } from "../src/db/write-worker.js"; +import { createActivityRuntime } from "../src/activity/runtime.js"; +import { createGovernanceWrites } from "../src/governance/event-writes.js"; +import { createTurnJournal } from "../src/brain/turn-journal.js"; +import { createTurnScope } from "../src/brain/turn-scope.js"; +import { createTurnRecorder } from "../src/activity/recorder.js"; +import { createActivityStore } from "../src/activity/store.js"; +import { createNativeJournalFixture } from "./helpers/native-journal-fixture.js"; +import { watchWriteSites } from "./helpers/write-sites.js"; +import type { Logger } from "@opentelemetry/api-logs"; + +function fixture() { + const root = mkdtempSync(join(tmpdir(), "app-writes-odysseus-")), brain = join(root, "brain"), path = join(root, "ui.db"); + mkdirSync(join(brain, ".brain/scratch"), { recursive: true }); + const db = createUiDb(path), writer = openSqliteWriter(db); + db.query("INSERT INTO sessions(id,title,created_at,last_active_at) VALUES ('ithaca','Voyage',0,0)").run(); + return { root, brain, path, db, writer, async close() { await writer.close(); db.close(); rmSync(root, { recursive: true, force: true }); } }; +} +const quietLog = { emit() {} } as unknown as Logger; +const attribution = { turnId: "voyage", sessionId: null, actor: "interactive" as const, model: null, profile: "governed" as const }; + +test("governance events commit on the write worker at FULL, never on the app connection", async () => { + const f = fixture(); + try { + expect(f.writer.offloop).toBe(true); + const sites = await watchWriteSites(f.db, f.writer, ["governance_events"]); + const writes = createGovernanceWrites(f.db, f.writer); + const recorded = await writes.record({ ...attribution, kind: "scope_cap", payload: { cap: "tool_calls", limit: 3, observed: 3, overshoot: 0, frozen: true, outcome: "paused", source: "policy" } }); + expect(recorded.recorded, "GOVERNANCE_EVENT_COMMITS_ON_WORKER").toBe(true); + expect(await writes.firstTaint({ ...attribution, kind: "taint", payload: { source: "egress_connection", tool: null, detail: "GET troy.example:443" } }), "GOVERNANCE_TAINT_COMMITS_ON_WORKER").toBe(true); + await writes.bind("voyage", "ithaca"); + expect(f.db.query("SELECT count(*) AS n FROM governance_events WHERE session_id = 'ithaca'").get(), "GOVERNANCE_BIND_COMMITS_ON_WORKER").toEqual({ n: 2 }); + expect(await writes.roadblocks({ ...attribution, sessionId: "ithaca" }, { startedAt: 0, endedAt: 1, calls: [], usage: [], finalUsage: null, outcomeDetail: null })) + .toBeGreaterThanOrEqual(0); + const runtime = createActivityRuntime(f.db, { log: quietLog, writer: f.writer }); + try { await runtime.governance({ ...attribution, kind: "budget_warning", payload: { scope: "turn", unit: "usd", limit: 1, observed: 1, source: "policy" } }); } + finally { runtime.close(); } + const log = await sites.log(); + expect(log.length, "GOVERNANCE_WRITES_LOGGED_ON_WORKER").toBeGreaterThanOrEqual(4); + expect(new Set(log), "GOVERNANCE_WRITES_ONLY_TO_WATCHED_TABLE").toEqual(new Set(["governance_events"])); + expect((f.db.query("SELECT count(*) AS n FROM governance_events").get() as { n: number }).n).toBe(4); + } finally { await f.close(); } +}); + +test("a pending taint narrows admission before its commit lands", async () => { + const f = fixture(); + try { + const writes = createGovernanceWrites(f.db, f.writer); + const committed = writes.firstTaint({ ...attribution, kind: "taint", payload: { source: "egress_connection", tool: null, detail: "GET troy.example:443" } }); + expect(writes.taint("voyage", null)?.source, "GOVERNANCE_PENDING_TAINT_IS_IN_FORCE").toBe("egress_connection"); + await committed; + expect(writes.taint("voyage", null)!.eventId, "GOVERNANCE_COMMITTED_TAINT_HAS_ITS_ROW").toBeGreaterThan(0); + } finally { await f.close(); } +}); + +test("turn scope rows and their cap events commit on the write worker, in submission order with Activity", async () => { + const f = fixture(); + const native = createNativeJournalFixture(f.brain, f.root); + const journal = createTurnJournal(f.db, { nativeManager: native.manager, brainPath: f.brain, dbPath: f.path, objectsRoot: join(f.root, "objects"), writer: f.writer }); + const runtime = createActivityRuntime(f.db, { log: quietLog, writer: f.writer }); + try { + await journal.ready; + const sites = await watchWriteSites(f.db, f.writer, ["turn_scope", "governance_events", "activity_spans"]); + const changed: string[] = []; + const scope = createTurnScope(f.db, { brainPath: f.brain, journal, writer: f.writer, governance: runtime.governanceWrites, connected: () => true, + changed: turnId => { changed.push((f.db.query("SELECT summary FROM turn_scope WHERE turn_id=?").get(turnId) as { summary: string }).summary); } }); + let stopped = ""; + void runtime.store.startSpan({ spanId: "voyage:root", runId: "voyage", name: "invoke_agent", kind: "turn", origin: "session", sessionId: "ithaca" }); + await scope.begin("voyage", { sessionId: "ithaca", principalId: "athena", origin: "interactive", profile: "governed", + policy: resolveGovernancePolicy({ schema_version: 1, scope: { max_tool_calls: 1 } }), source: "policy", hardTimeoutMs: 60_000, + current: () => !stopped, stop: reason => { stopped = reason; } }); + // The cap is reached with no worker to freeze: its event, then the stopped summary. + scope.toolCall("voyage", "call-1"); + await scope.finish("voyage"); + const log = await sites.log(); + expect(log.slice(0, 2), "SCOPE_ORDER_ACTIVITY_THEN_SCOPE_ROW").toEqual(["activity_spans", "turn_scope"]); + expect(log.slice(2), "SCOPE_CAP_EVENT_BEFORE_ITS_SUMMARY").toEqual(["governance_events", "turn_scope", "turn_scope", "turn_scope"]); + expect(stopped, "SCOPE_CAP_STOPS_UNFROZEN_TURN").toContain("could not confirm a running worker"); + const row = f.db.query("SELECT summary, ended_at FROM turn_scope WHERE turn_id='voyage'").get() as { summary: string; ended_at: number | null }; + expect(row.ended_at, "SCOPE_END_COMMITTED").not.toBeNull(); + expect(changed.at(-1), "SCOPE_CHANGED_AFTER_COMMIT").toBe(row.summary); + } finally { runtime.close(); await journal.close(); await native.close(); await f.close(); } +}); + +test("a turn's settled() waits for its roadblock commit", async () => { + const db = createUiDb(":memory:"); + let release!: () => void; + const held = new Promise(done => { release = () => done(0); }); + const recorder = createTurnRecorder({ store: createActivityStore(db, { writer: "odysseus" }), governanceDb: db, + governanceWrites: { ...createGovernanceWrites(db), roadblocks: () => held } }, { turnId: "voyage", sessionId: null }); + try { + recorder.finish("success"); + let settled = false; + void recorder.settled().then(() => { settled = true; }); + await Bun.sleep(50); + expect(settled, "RECORDER_SETTLED_WAITS_FOR_ROADBLOCKS").toBe(false); + release(); + await recorder.settled(); + expect(settled).toBe(true); + } finally { db.close(); } +}); diff --git a/packages/ui-server/tests/fixtures/scope-tmpfs-probe.ts b/packages/ui-server/tests/fixtures/scope-tmpfs-probe.ts index 3f080074a0..cc12b0df8c 100644 --- a/packages/ui-server/tests/fixtures/scope-tmpfs-probe.ts +++ b/packages/ui-server/tests/fixtures/scope-tmpfs-probe.ts @@ -22,7 +22,7 @@ try { if (f.receipt().dimension !== "tool_calls" || f.receipt().newCap !== 2) throw Error("Fixture did not pause at its tool cap"); writeFileSync(join(brain, ".brain/scratch/disk-pressure"), Buffer.alloc(33 * 1024 ** 2)); // Fresh Continue admission itself observes the floor, without waiting for a new interval. - if (f.scope.decide(previous, true).state !== "refused") throw Error("SCOPE_PAUSED_CONTINUE_BYPASSED_FLOOR"); + if ((await f.scope.decide(previous, true)).state !== "refused") throw Error("SCOPE_PAUSED_CONTINUE_BYPASSED_FLOOR"); } const deadline = Date.now() + 32_000; if (clock) { @@ -47,8 +47,8 @@ try { if (receipt.dimension !== "free_space_floor" || receipt.state !== "paused" || !receipt.frozen || receipt.newCap !== undefined) throw Error(JSON.stringify(receipt)); await Bun.sleep(200); if (readdirSync(join(brain, ".brain/scratch")).length !== count) throw Error("SCOPE_TMPFS_LOOP_NOT_FROZEN"); - if (f.scope.decide(f.message("continue"), true).state !== "refused") throw Error("SCOPE_TMPFS_FLOOR_GRANTED"); - f.scope.decide(f.message("stop"), true); await child.exited; + if ((await f.scope.decide(f.message("continue"), true)).state !== "refused") throw Error("SCOPE_TMPFS_FLOOR_GRANTED"); + (await f.scope.decide(f.message("stop"), true)); await child.exited; console.log(JSON.stringify({ proof: "supported tmpfs worker freeze/Stop", receipt: f.receipt(), retainedFiles: count, timings })); } finally { const cleanup = [async () => { child?.kill("SIGKILL"); if (child) await child.exited; }, () => f.close()]; diff --git a/packages/ui-server/tests/fixtures/write-site-probe.ts b/packages/ui-server/tests/fixtures/write-site-probe.ts new file mode 100644 index 0000000000..54a2f9d176 --- /dev/null +++ b/packages/ui-server/tests/fixtures/write-site-probe.ts @@ -0,0 +1,22 @@ +/** + * A write-worker procedure module for tests (#2850): on the write connection, + * it refuses any write to the watched tables below `synchronous=FULL` and logs + * each one, in commit order. + */ +import type { Database } from "bun:sqlite"; + +export const url = import.meta.url; + +export function setup(db: Database, options: { tables: string[] }) { + db.exec("CREATE TEMP TABLE IF NOT EXISTS odysseus_write_log (n INTEGER PRIMARY KEY AUTOINCREMENT, tbl TEXT NOT NULL)"); + for (const table of options.tables) for (const event of ["INSERT", "UPDATE", "DELETE"]) { + db.exec(`CREATE TEMP TRIGGER IF NOT EXISTS ${table}_full_${event.toLowerCase()} BEFORE ${event} ON main.${table} + WHEN (SELECT synchronous FROM pragma_synchronous) IS NOT 2 BEGIN SELECT RAISE(ABORT, 'WRITE_BELOW_FULL ${table}'); END`); + db.exec(`CREATE TEMP TRIGGER IF NOT EXISTS ${table}_log_${event.toLowerCase()} AFTER ${event} ON main.${table} + BEGIN INSERT INTO odysseus_write_log (tbl) VALUES ('${table}'); END`); + } + return { + /** Every watched write the worker committed, in commit order. */ + log() { return (db.query("SELECT tbl FROM odysseus_write_log ORDER BY n").all() as { tbl: string }[]).map(row => row.tbl); }, + }; +} diff --git a/packages/ui-server/tests/helpers/write-sites.ts b/packages/ui-server/tests/helpers/write-sites.ts new file mode 100644 index 0000000000..8c7144bc0a --- /dev/null +++ b/packages/ui-server/tests/helpers/write-sites.ts @@ -0,0 +1,22 @@ +/** + * Where a moved write commits (#2850). The app connection refuses every write + * to the watched tables; the write worker's connection refuses one below + * `synchronous=FULL` and logs the rest, in commit order. Temporary triggers + * belong to the connection that made them, so each side checks only itself. + */ +import type { Database } from "bun:sqlite"; +import type { SqliteWriter } from "../../src/db/write-worker.js"; +import * as probe from "../fixtures/write-site-probe.js"; + +export async function watchWriteSites(db: Database, writer: SqliteWriter, tables: string[]) { + for (const table of tables) for (const event of ["INSERT", "UPDATE", "DELETE"]) { + db.exec(`CREATE TEMP TRIGGER IF NOT EXISTS ${table}_app_${event.toLowerCase()} BEFORE ${event} ON main.${table} + BEGIN SELECT RAISE(ABORT, 'APP_CONNECTION_WRITE ${table}'); END`); + } + const channel = writer.channel(probe, { tables }); + await channel.ready; + return { + /** The watched tables each committed write landed in, in commit order, once everything submitted so far has committed. */ + async log() { await writer.idle(); return channel.call("log"); }, + }; +} diff --git a/packages/ui-server/tests/interactive-share.test.ts b/packages/ui-server/tests/interactive-share.test.ts index c6f120ec10..249a504eae 100644 --- a/packages/ui-server/tests/interactive-share.test.ts +++ b/packages/ui-server/tests/interactive-share.test.ts @@ -1,4 +1,5 @@ import { afterEach, expect, test } from "bun:test"; +import { expectRejection } from "./helpers/expect-rejection.js"; import { existsSync, linkSync, mkdirSync, readFileSync, symlinkSync, unlinkSync, writeFileSync } from "node:fs"; import { join } from "node:path"; import { generateSignedCookie } from "hono/cookie"; @@ -93,7 +94,7 @@ test("real app/WS completion verifies filing and retained PDF, then a lost reply const repeated = await f.upload(); expect(repeated.id).toBe(share.id); expect(repeated.filingSessionId, "LOST_ADMISSION_RECOVERS_BOUND_SESSION_WITHOUT_DISPATCH").toBe(sid); await f.send(repeated, async req => { - await expect(req.bridge.completeShare!(request), "COMPLETION_REPLAY_IS_IDEMPOTENT").resolves.toEqual({ ...receipt, replay: true }); + expect(await req.bridge.completeShare!(request), "COMPLETION_REPLAY_IS_IDEMPOTENT").toEqual({ ...receipt, replay: true }); expect((await req.bridge.applyBrain!(application(share))).shareReceiptId).toBe(request.receiptIds[0]); }, sid); expect(f.frames.findLast(frame => frame.type === "session_info").sessionId, "REUPLOADED_SHARE_RESUMES_ORIGINAL_SESSION").toBe(sid); @@ -113,7 +114,7 @@ test("partial filing and duplicate application delivery recover the same item/se expect(result.ok).toBe(true); expect(await req.bridge.applyBrain!(application(share)), "FILING_REPLAY_HAS_ORIGINAL_RECEIPT").toEqual(result); expect(await req.bridge.applyBrain!({ ...application(share), expectedBaseHash: result.changes[0].contentHash } as BrainApplicationInput), "FRESH_BASE_CANNOT_DUPLICATE_FILING").toEqual(result); - await expect(req.bridge.completeShare!({ receiptIds: ["forged"], files: [] })).rejects.toThrow("forged"); + await expectRejection(req.bridge.completeShare!({ receiptIds: ["forged"], files: [] }), "forged"); }); expect(f.item(share).status).toBe("ready"); expect((await f.upload(false)).filingSessionId, "PARTIAL_FILING_RECOVERS_THE_SAME_SESSION").toBe(sid); @@ -137,25 +138,25 @@ test("unrelated writes, another item's receipt, changed binaries and stale autho await f.send(share, async req => { const unrelated = await req.bridge.applyBrain!({ operation: "write", path: "notes/unrelated.md", expectedBaseHash: null, content: "Telemachus sails.\n" }); const input = { receiptIds: [unrelated.shareReceiptId!], files: [] }; - await expect(req.bridge.completeShare!(input)).rejects.toThrow("source identity"); - await expect(req.bridge.completeShare!({ ...input, receiptIds: [foreign.shareReceiptId] })).rejects.toThrow("unrelated"); + await expectRejection(req.bridge.completeShare!(input), "source identity"); + await expectRejection(req.bridge.completeShare!({ ...input, receiptIds: [foreign.shareReceiptId] }), "unrelated"); const result = await req.bridge.applyBrain!(application(share)); const complete = { receiptIds: [result.shareReceiptId!], files: [{ sourcePath: share.files[0]!.path, disposition: "retained" as const, path: "assets/ithaca.pdf" }] }; - await expect(req.bridge.completeShare!(complete)).rejects.toThrow("differs"); + await expectRejection(req.bridge.completeShare!(complete), "differs"); // Equal-sized original tampering cannot replace intake evidence. writeFileSync(join(f.brainPath, share.files[0]!.path), pdf.replace("Odysseus", "Achilles")); writeFileSync(join(f.brainPath, "assets/ithaca.pdf"), pdf.replace("Odysseus", "Achilles")); - await expect(req.bridge.completeShare!(complete)).rejects.toThrow("verified intake evidence"); + await expectRejection(req.bridge.completeShare!(complete), "verified intake evidence"); writeFileSync(join(f.brainPath, share.files[0]!.path), pdf); unlinkSync(join(f.brainPath, "assets/ithaca.pdf")); symlinkSync(join(f.brainPath, share.files[0]!.path), join(f.brainPath, "assets/ithaca.pdf")); - await expect(req.bridge.completeShare!(complete), "ALIASED_RETENTION_IS_NOT_EVIDENCE").rejects.toThrow(); + await expectRejection(req.bridge.completeShare!(complete), undefined, "ALIASED_RETENTION_IS_NOT_EVIDENCE"); unlinkSync(join(f.brainPath, "assets/ithaca.pdf")); linkSync(join(f.brainPath, share.files[0]!.path), join(f.brainPath, "assets/ithaca.pdf")); - await expect(req.bridge.completeShare!(complete), "HARDLINKED_RETENTION_IS_NOT_EVIDENCE").rejects.toThrow("Unsupported binary evidence"); + await expectRejection(req.bridge.completeShare!(complete), "Unsupported binary evidence", "HARDLINKED_RETENTION_IS_NOT_EVIDENCE"); unlinkSync(join(f.brainPath, "assets/ithaca.pdf")); writeFileSync(join(f.brainPath, "assets/ithaca.pdf"), pdf); revokePrincipal(f.app.db, f.principal.id, Date.now()); - await expect(req.bridge.completeShare!(complete)).rejects.toThrow("principal authority"); + await expectRejection(req.bridge.completeShare!(complete), "principal authority"); }); expect(f.item(share).status).toBe("ready"); expect(existsSync(join(f.brainPath, share.dir))).toBe(true); }); @@ -180,9 +181,9 @@ test("share completion refuses a filed document or retained original under a pro const result = await req.bridge.applyBrain!(application(share)); const input = { receiptIds: [result.shareReceiptId!], files: [{ sourcePath: share.files[0]!.path, disposition: "retained" as const, path: "assets/ithaca.pdf" }] }; masked = ["notes"]; - await expect(req.bridge.completeShare!(input), "MASKED_FILED_DOCUMENT_REFUSED").rejects.toThrow("provider_masked: notes/ithaca-share.md"); + await expectRejection(req.bridge.completeShare!(input), "provider_masked: notes/ithaca-share.md", "MASKED_FILED_DOCUMENT_REFUSED"); masked = ["assets"]; - await expect(req.bridge.completeShare!(input), "MASKED_RETAINED_ORIGINAL_REFUSED").rejects.toThrow("provider_masked: assets/ithaca.pdf"); + await expectRejection(req.bridge.completeShare!(input), "provider_masked: assets/ithaca.pdf", "MASKED_RETAINED_ORIGINAL_REFUSED"); expect(f.item(share).status).toBe("claimed"); masked = []; expect(await req.bridge.completeShare!(input), "UNMASKED_COMPLETION_STILL_FILES").toMatchObject({ status: "done", replay: false }); @@ -238,7 +239,7 @@ test("client cancellation preserves partial effects and originals and never comm req.signal.addEventListener("abort", () => cancelled.resolve(), { once: true }); f.ws.send(JSON.stringify({ type: "cancel", sessionId: "odysseus-share-1" })); await cancelled.promise; - await expect(req.bridge.completeShare!({ receiptIds: [result.shareReceiptId!], files: [] })).rejects.toThrow("principal authority"); + await expectRejection(req.bridge.completeShare!({ receiptIds: [result.shareReceiptId!], files: [] }), "principal authority"); }); expect(f.item(share).status, "CANCEL_IS_NEVER_SUCCESSFUL_FILING").toBe("ready"); expect(existsSync(join(f.brainPath, share.dir))).toBe(true); @@ -254,8 +255,8 @@ test("uncertain application receipt refuses replay instead of duplicating a comm // journaled. The pending operation remains authoritative replay restriction. f.app.db.query("UPDATE inbox_share_applications SET result_json = NULL WHERE id = ?").run(result.shareReceiptId); await f.send(share, async req => { - await expect(req.bridge.applyBrain!(application(share))).rejects.toThrow("outcome is uncertain"); - await expect(req.bridge.completeShare!({ receiptIds: [result.shareReceiptId], files: [] })).rejects.toThrow("Unsupported"); + await expectRejection(req.bridge.applyBrain!(application(share)), "outcome is uncertain"); + await expectRejection(req.bridge.completeShare!({ receiptIds: [result.shareReceiptId], files: [] }), "Unsupported"); }, sid); expect(f.item(share).status).toBe("ready"); expect(existsSync(join(f.brainPath, share.dir))).toBe(true); expect(readFileSync(join(f.brainPath, "notes/ithaca-share.md"), "utf8")).toBe((application(share) as { content: string }).content); @@ -304,7 +305,7 @@ test("completed filing preserves success and uncertainty replay guards while all // Even after completion an interrupted reply remains a replay restriction. f.app.db.query("UPDATE inbox_share_applications SET result_json = NULL WHERE id = ?").run(filed.shareReceiptId); await f.send(share, async req => { - await expect(req.bridge.applyBrain!(application(share, "notes/filed-ithaca.md")), "COMPLETION_NEVER_REOPENS_UNCERTAIN_WRITES").rejects.toThrow("outcome is uncertain"); + await expectRejection(req.bridge.applyBrain!(application(share, "notes/filed-ithaca.md")), "outcome is uncertain", "COMPLETION_NEVER_REOPENS_UNCERTAIN_WRITES"); }, sid); expect(f.item(share).status).toBe("done"); }); diff --git a/packages/ui-server/tests/turn-egress.test.ts b/packages/ui-server/tests/turn-egress.test.ts index 3e17e73eaa..7766c6f8b9 100644 --- a/packages/ui-server/tests/turn-egress.test.ts +++ b/packages/ui-server/tests/turn-egress.test.ts @@ -12,6 +12,9 @@ import type { BackendBridge, PermissionDecision } from "@schlessera/brain-ui-sdk import { createUiDb } from "../src/db/client.js"; import { queryGovernanceEvents, recordGovernanceEvent, type GovernanceEventInput } from "../src/governance/events.js"; import { createOutwardCalls, prepareTurnEgress } from "../src/brain/turn-egress.js"; +import { openSqliteWriter } from "../src/db/write-worker.js"; +import { createGovernanceWrites } from "../src/governance/event-writes.js"; +import * as probe from "./fixtures/write-worker-probe.js"; function get(socketPath: string, target: string): Promise { return new Promise((resolve, reject) => { @@ -266,3 +269,47 @@ test("the first admitted connection taints the session before its bytes; later t expect(events("taint"), "TURN_EGRESS_FIRST_SOURCE_ONLY").toHaveLength(1); } finally { harbour.stop(true); island.stop(true); db.close(); } }); + +test("on the write worker, a connection's bytes wait for its taint commit and a pending taint narrows the next host", async () => { + const root = mkdtempSync(join(tmpdir(), "turn-egress-worker-odysseus-")), db = createUiDb(join(root, "ui.db")), writer = openSqliteWriter(db); + db.query("INSERT INTO sessions (id,title,created_at,last_active_at) VALUES ('ithaca','Ithaca',0,0)").run(); + const hits: number[] = []; + const harbour = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => { hits.push(Date.now()); return new Response("Ithaca"); } }); + const island = Bun.serve({ hostname: "127.0.0.2", port: 0, fetch: () => { hits.push(-1); return new Response("Aeaea"); } }); + const writes = createGovernanceWrites(db, writer); + const held = writer.channel(probe, { table: "odysseus_log" }); + const proxy = await prepareTurnEgress({ ...inputs(new AbortController()), turnId: "turn-harbour", sessionId: () => "ithaca", + record: event => writes.record(event), governanceDb: db, governanceWrites: writes }); + try { + expect(writer.offloop).toBe(true); + await held.ready; + // A slow commit ahead of the taint, as an fsync under load would be. + const slow = held.call("slow", "Calypso", 400), started = Date.now(); + const first = get(proxy.socketPath, harbour.url.href); + await Bun.sleep(100); + expect(writes.taint("turn-harbour", "ithaca")?.source, "TURN_EGRESS_PENDING_TAINT_IN_FORCE").toBe("egress_connection"); + expect(await get(proxy.socketPath, island.url.href), "TURN_EGRESS_PENDING_TAINT_NARROWS_NEW_HOST").toBe(403); + expect(await first).toBe(200); + await slow; + expect(hits, "TURN_EGRESS_NEW_HOST_NEVER_CONTACTED").toHaveLength(1); + expect(hits[0]! - started, "TURN_EGRESS_BYTES_AFTER_TAINT_COMMIT").toBeGreaterThanOrEqual(350); + proxy.stop(); + await proxy.settled(); + expect(queryGovernanceEvents(db, { sessionId: "ithaca", kinds: ["egress_connection", "egress_denial"], limit: 10 }).map(event => event.kind).sort(), + "TURN_EGRESS_RECORDS_COMMITTED_BY_SETTLED").toEqual(["egress_connection", "egress_connection", "egress_denial"]); + expect(proxy.recordingFailure()).toBeUndefined(); + } finally { proxy.stop(); harbour.stop(true); island.stop(true); await writer.close(); db.close(); rmSync(root, { recursive: true, force: true }); } +}); + +test("a record that fails to commit is the turn's recording failure once settled", async () => { + const origin = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => new Response("Ithaca") }); + const controller = new AbortController(); + const proxy = await prepareTurnEgress({ ...inputs(controller), record: () => new Promise((_, reject) => setTimeout(() => reject(new Error("Scylla took the record")), 50)) }); + try { + expect(await get(proxy.socketPath, origin.url.href)).toBe(200); + proxy.stop(); + await proxy.settled(); + expect(proxy.recordingFailure()?.message, "TURN_EGRESS_LATE_RECORD_FAILURE_IS_FINAL_AFTER_SETTLED").toBe("Turn egress recording is unavailable"); + expect(controller.signal.aborted).toBe(true); + } finally { proxy.stop(); origin.stop(true); } +}); diff --git a/packages/ui-server/tests/turn-scope-headroom.test.ts b/packages/ui-server/tests/turn-scope-headroom.test.ts index a42ead63c0..62e51c7b68 100644 --- a/packages/ui-server/tests/turn-scope-headroom.test.ts +++ b/packages/ui-server/tests/turn-scope-headroom.test.ts @@ -28,7 +28,7 @@ function fixture(origin: "interactive" | "voice" = "interactive") { const message = (action: ClientScopeDecision["action"] = "continue", submissionId = "athena-grant"): ClientScopeDecision => ({ type: "scope_decision", sessionId: "ithaca", turnId: "voyage", eventId: receipt().eventId, submissionId, action }); return { db, scope, summary, receipt, observe, message, get callbacks() { return { freezes, resumes, kills }; }, advance(ms: number) { clock += ms; }, disconnect() { connected = false; }, end() { current = false; }, floor() { free = 0; }, close() { scope.close(); db.close(); } }; } -for (const dimension of ["files_changed", "turn_bytes"] as const) for (const count of [1000, 1999, 2000, 5333]) test(`${dimension}: finite headroom at ${count}`, () => { +for (const dimension of ["files_changed", "turn_bytes"] as const) for (const count of [1000, 1999, 2000, 5333]) test(`${dimension}: finite headroom at ${count}`, async () => { const f = fixture(); try { f.observe(count, dimension); @@ -37,35 +37,35 @@ for (const dimension of ["files_changed", "turn_bytes"] as const) for (const cou if (count >= 2000) { expect(f.receipt().newCap, "SCOPE_HEADROOM_EXHAUSTED_OFFER_STOP_ONLY").toBeUndefined(); expect(f.receipt().reason).toContain("finite continuation ceiling"); - expect(f.scope.decide(f.message(), true).state).toBe("refused"); + expect((await f.scope.decide(f.message(), true)).state).toBe("refused"); expect(f.receipt().submissionId).toBeUndefined(); expect(f.callbacks.resumes).toBe(0); } else { expect(f.receipt().newCap).toBe(2000); const msg = f.message(); - for (const altered of [{ ...msg, sessionId: "ogygia" }, { ...msg, eventId: "other-event" }, { ...msg, turnId: "other-turn" }]) expect(f.scope.decide(altered, true).state).toBe("refused"); - expect(f.scope.decide(msg, false).state).toBe("refused"); - expect(f.scope.decide(msg, true).state).toBe("accepted"); expect(f.scope.decide(msg, true).state).toBe("accepted"); + for (const altered of [{ ...msg, sessionId: "ogygia" }, { ...msg, eventId: "other-event" }, { ...msg, turnId: "other-turn" }]) expect((await f.scope.decide(altered, true)).state).toBe("refused"); + expect((await f.scope.decide(msg, false)).state).toBe("refused"); + expect((await f.scope.decide(msg, true)).state).toBe("accepted"); expect((await f.scope.decide(msg, true)).state).toBe("accepted"); expect(f.callbacks.resumes).toBe(1); expect(f.summary().limits[dimension]).toBe(2000); - expect(f.scope.decide({ ...msg, submissionId: "another-device" }, true).state).toBe("answered_elsewhere"); + expect((await f.scope.decide({ ...msg, submissionId: "another-device" }, true)).state).toBe("answered_elsewhere"); f.observe(2000, dimension); expect(f.receipt().newCap).toBeUndefined(); - expect(f.scope.decide(f.message("stop", "athena-stop"), true).state).toBe("accepted"); + expect((await f.scope.decide(f.message("stop", "athena-stop"), true)).state).toBe("accepted"); } } finally { f.close(); } }); -for (const dimension of ["files_changed", "turn_bytes"] as const) test(`${dimension}: latest observation refuses stale submission without consuming grant`, () => { +for (const dimension of ["files_changed", "turn_bytes"] as const) test(`${dimension}: latest observation refuses stale submission without consuming grant`, async () => { const f = fixture(); try { f.observe(1000, dimension); const msg = f.message(); f.observe(5333, dimension); expect(f.receipt()).toMatchObject({ count: 5333, overshoot: 4333 }); - expect(f.scope.decide(msg, true).state, "SCOPE_HEADROOM_STALE_DECISION_REFUSED").toBe("refused"); - expect(f.scope.decide(msg, true).state).toBe("refused"); + expect((await f.scope.decide(msg, true)).state, "SCOPE_HEADROOM_STALE_DECISION_REFUSED").toBe("refused"); + expect((await f.scope.decide(msg, true)).state).toBe("refused"); expect(f.summary().limits[dimension]).toBe(1000); expect(f.receipt().state).toBe("paused"); expect(f.receipt().submissionId).toBeUndefined(); expect(f.callbacks.resumes).toBe(0); - expect(f.scope.decide({ ...msg, action: "readback" }, true).state).toBe("pending"); - expect(f.scope.decide({ ...msg, action: "stop", submissionId: "athena-stop" }, true).state).toBe("accepted"); + expect((await f.scope.decide({ ...msg, action: "readback" }, true)).state).toBe("pending"); + expect((await f.scope.decide({ ...msg, action: "stop", submissionId: "athena-stop" }, true)).state).toBe("accepted"); } finally { f.close(); } }); -test("submission revalidates elapsed time and refuses a tampered ceiling or incomplete observation", () => { +test("submission revalidates elapsed time and refuses a tampered ceiling or incomplete observation", async () => { for (const mode of ["time", "tampered", "partial"] as const) { const f = fixture(); try { @@ -77,12 +77,12 @@ test("submission revalidates elapsed time and refuses a tampered ceiling or inco const altered = f.summary(); altered.receipts[0]!.newCap = 9000; f.db.query("UPDATE turn_scope SET summary=? WHERE turn_id='voyage'").run(JSON.stringify(altered)); } - expect(f.scope.decide(msg, true).state, `SCOPE_HEADROOM_${mode.toUpperCase()}_REFUSED`).toBe("refused"); + expect((await f.scope.decide(msg, true)).state, `SCOPE_HEADROOM_${mode.toUpperCase()}_REFUSED`).toBe("refused"); expect(f.receipt().submissionId).toBeUndefined(); expect(f.callbacks.resumes).toBe(0); } finally { f.close(); } } }); -test("headroom never overrides voice, free-space floor, ended authority or disconnected lifetime", () => { +test("headroom never overrides voice, free-space floor, ended authority or disconnected lifetime", async () => { for (const mode of ["voice", "floor", "ended", "disconnected"] as const) { const f = fixture(mode === "voice" ? "voice" : "interactive"); try { @@ -90,7 +90,7 @@ test("headroom never overrides voice, free-space floor, ended authority or disco if (mode === "floor") f.floor(); if (mode === "ended") f.end(); if (mode === "disconnected") { f.disconnect(); f.observe(1000); f.advance(SCOPE_DISCONNECTED_MS); } - expect(f.scope.decide(msg, true).state).not.toBe("accepted"); + expect((await f.scope.decide(msg, true)).state).not.toBe("accepted"); expect(f.callbacks.resumes).toBe(0); expect(f.summary().hardTimeoutMs).toBe(600_000); } finally { f.close(); } } diff --git a/packages/ui-server/tests/turn-scope.test.ts b/packages/ui-server/tests/turn-scope.test.ts index 510beec8dd..102b67ecfe 100644 --- a/packages/ui-server/tests/turn-scope.test.ts +++ b/packages/ui-server/tests/turn-scope.test.ts @@ -34,7 +34,7 @@ test("real process tree freezes, resumes once per dimension, then stops with eve await entered.promise; expect(f.journal.isHeld(), "SCOPE_CAP_OBSERVATION_HELD").toBe(true); expect(observations, "SCOPE_CAP_OBSERVATION_NOT_COMPLETED_WHILE_HELD").toBe(0); - expect(f.scope.decide(f.message("continue", "held-observation"), true).state, "SCOPE_CONTINUE_REFUSED_WHILE_OBSERVATION_HELD").toBe("refused"); + expect((await f.scope.decide(f.message("continue", "held-observation"), true)).state, "SCOPE_CONTINUE_REFUSED_WHILE_OBSERVATION_HELD").toBe("refused"); const before = Bun.file(join(f.brain, "heartbeat")).size; await Bun.sleep(100); expect(Bun.file(join(f.brain, "heartbeat")).size, "SCOPE_RUNTIME_PAUSE_STOPS_RUNNING_DESCENDANT").toBe(before); @@ -45,17 +45,17 @@ test("real process tree freezes, resumes once per dimension, then stops with eve expect(observations, "SCOPE_CAP_OBSERVATION_COMPLETED").toBe(1); expect(f.journal.isHeld(), "SCOPE_CAP_OBSERVATION_RELEASED").toBe(false); const msg = f.message("continue"); - expect(f.scope.decide({ ...msg, sessionId: "ogygia" }, true).state).toBe("refused"); - expect(f.scope.decide({ ...msg, turnId: "other-turn" }, true).state).toBe("refused"); - expect(f.scope.decide(msg, false).state).toBe("refused"); - expect(f.scope.decide(msg, true).state, "SCOPE_CONTINUE_ACCEPTED_AFTER_OBSERVATION_COMPLETES").toBe("accepted"); - expect(f.scope.decide(msg, true).state).toBe("accepted"); - expect(f.scope.decide({ ...msg, submissionId: "penelope" }, true).state).toBe("answered_elsewhere"); + expect((await f.scope.decide({ ...msg, sessionId: "ogygia" }, true)).state).toBe("refused"); + expect((await f.scope.decide({ ...msg, turnId: "other-turn" }, true)).state).toBe("refused"); + expect((await f.scope.decide(msg, false)).state).toBe("refused"); + expect((await f.scope.decide(msg, true)).state, "SCOPE_CONTINUE_ACCEPTED_AFTER_OBSERVATION_COMPLETES").toBe("accepted"); + expect((await f.scope.decide(msg, true)).state).toBe("accepted"); + expect((await f.scope.decide({ ...msg, submissionId: "penelope" }, true)).state).toBe("answered_elsewhere"); await wait(() => Bun.file(join(f.brain, "heartbeat")).size > before); f.scope.toolCall("voyage", "loom-3"); f.scope.toolCall("voyage", "loom-4"); expect(f.receipt()).toMatchObject({ state: "paused", cap: 4 }); expect(f.receipt().newCap).toBeUndefined(); - expect(f.scope.decide(f.message("continue", "second-grant"), true).state, "SCOPE_SECOND_HIT_STOP_ONLY").toBe("refused"); - expect(f.scope.decide(f.message("stop", "stop-here"), true).state).toBe("accepted"); + expect((await f.scope.decide(f.message("continue", "second-grant"), true)).state, "SCOPE_SECOND_HIT_STOP_ONLY").toBe("refused"); + expect((await f.scope.decide(f.message("stop", "stop-here"), true)).state).toBe("accepted"); await child.exited; await f.journal.finish("voyage"); f.scope.finish("voyage"); expect(f.stopped).toContain("You stopped"); expect(f.journal.summary("voyage").state).toBe("complete"); expect(f.db.query("SELECT count(*) AS n FROM governance_events WHERE kind='scope_cap'").get()).toEqual({ n: 2 }); @@ -71,13 +71,13 @@ test("shared snapshots count ignored scratch, workspaces and repeated observed b await f.begin(); child = f.worker('while :; do sleep 1; done'); writeFileSync(join(f.brain, ".brain/scratch/loom"), "12345678"); await f.journal.observe(); expect(f.receipt()).toMatchObject({ dimension: "turn_bytes", count: 8, cap: 8, newCap: 16 }); - expect(f.scope.decide(f.message("continue"), true).state).toBe("accepted"); + expect((await f.scope.decide(f.message("continue"), true)).state).toBe("accepted"); writeFileSync(join(f.brain, "workspaces/shroud"), "x"); writeFileSync(join(f.brain, "sail"), "x"); await f.journal.observe(); expect(f.receipt()).toMatchObject({ dimension: "files_changed", count: 3, newCap: 6 }); - expect(f.scope.decide(f.message("continue", "files"), true).state, "SCOPE_OTHER_DIMENSION_RETAINS_GRANT").toBe("accepted"); + expect((await f.scope.decide(f.message("continue", "files"), true)).state, "SCOPE_OTHER_DIMENSION_RETAINS_GRANT").toBe("accepted"); writeFileSync(join(f.brain, ".brain/scratch/loom"), "87654321"); await f.journal.observe(); expect(f.receipt()).toMatchObject({ dimension: "turn_bytes", count: 18, cap: 16, overshoot: 2 }); expect(f.receipt().newCap).toBeUndefined(); - f.scope.decide(f.message("stop", "bytes-stop"), true); await child.exited; + (await f.scope.decide(f.message("stop", "bytes-stop"), true)); await child.exited; } finally { child?.kill("SIGKILL"); if (child) await child.exited; await f.close(); } }); @@ -170,7 +170,7 @@ test("connected pause has no scope expiry; disconnect lifetime, reconnect and re f.disconnect(); await f.journal.observe(); clock += SCOPE_DISCONNECTED_MS - 1; await f.journal.observe(); expect(f.receipt().state).toBe("paused"); f.reconnect(); await f.journal.observe(); expect(f.receipt().state, "SCOPE_RECONNECT_NEVER_RESUMES").toBe("paused"); f.disconnect(); await f.journal.observe(); clock += SCOPE_DISCONNECTED_MS; await f.journal.observe(); - expect(f.receipt().state).toBe("ended_before_answer"); expect(f.scope.decide(f.message("continue"), true).state).toBe("ended_before_answer"); await child.exited; + expect(f.receipt().state).toBe("ended_before_answer"); expect((await f.scope.decide(f.message("continue"), true)).state).toBe("ended_before_answer"); await child.exited; } finally { child?.kill("SIGKILL"); if (child) await child.exited; await f.close(); } }); @@ -180,7 +180,7 @@ test("true voice stops without a grant and preserves partial work", async () => await f.begin("voice"); child = f.worker('printf Ithaca > voice-note; while :; do sleep 1; done'); await wait(() => Bun.file(join(f.brain, "voice-note")).size > 0); f.scope.toolCall("voyage", "loom"); await child.exited; expect(f.receipt()).toMatchObject({ state: "voice_stopped", cap: 1, frozen: true }); expect(f.receipt().newCap).toBeUndefined(); - expect(f.scope.decide(f.message("continue"), true).state).toBe("answered_elsewhere"); expect(f.stopped).toContain("I stopped"); + expect((await f.scope.decide(f.message("continue"), true)).state).toBe("answered_elsewhere"); expect(f.stopped).toContain("I stopped"); expect(await Bun.file(join(f.brain, "voice-note")).text()).toBe("Ithaca"); } finally { child?.kill("SIGKILL"); if (child) await child.exited; await f.close(); } }); @@ -293,7 +293,7 @@ test("one long single Bash loop is frozen by the registered 15-second observatio await wait(() => progress().filter(sample => sample.at >= stableAt).length >= 5); expect(progress().filter(sample => sample.at >= stableAt).every(sample => sample.count === count), "SCOPE_EXTERNAL_FINAL_WINDOW_STAYS_FROZEN").toBe(true); expect(await Bun.file(join(f.brain, "finished")).exists()).toBe(false); - expect(f.scope.decide(f.message("stop"), true).state, "SCOPE_LONG_BASH_STOP_ACCEPTED").toBe("accepted"); + expect((await f.scope.decide(f.message("stop"), true)).state, "SCOPE_LONG_BASH_STOP_ACCEPTED").toBe("accepted"); expect(await child.exited, "SCOPE_LONG_BASH_STOP_EXITS_CHILD").toBe(137); f.native.manager.checkpoint = checkpoint; await f.journal.close(); @@ -372,7 +372,7 @@ test("real host restart kills its paused namespace, records interruption and nev const receipt = journal.summary("voyage").scope!.receipts[0]!; expect(receipt.state, "SCOPE_RESTART_INTERRUPTION_RECEIPT").toBe("ended_before_answer"); expect(receipt.reason).toContain("Host restarted"); expect(journal.summary("voyage").state).toBe("partial"); - expect(recovered.decide({ type: "scope_decision", eventId: receipt.eventId, turnId: "voyage", sessionId: "ithaca", submissionId: "late-device", action: "continue" }, true).state).toBe("ended_before_answer"); + expect((await recovered.decide({ type: "scope_decision", eventId: receipt.eventId, turnId: "voyage", sessionId: "ithaca", submissionId: "late-device", action: "continue" }, true)).state).toBe("ended_before_answer"); expect(db.query("SELECT count(*) AS n FROM governance_events WHERE kind='roadblock'").get(), "SCOPE_RESTART_RETAINS_ROADBLOCK").toEqual({ n: 1 }); expect(await Bun.file(join(packet.brain, "restart-note")).text()).toBe("Ithaca"); } finally { recovered.close(); await journal.close(); await native.close(); db.close(); } @@ -384,9 +384,9 @@ test("wall time uses the frozen policy independently of later edits and hard lim try { await f.begin(); child = f.worker('while :; do sleep 1; done'); f.policy.scope.max_minutes = 90; clock += 15_001; await f.journal.observe(); expect(f.receipt()).toMatchObject({ dimension: "minutes", cap: .25, newCap: .5, frozen: true }); - expect(f.scope.decide(f.message("continue"), true).state).toBe("accepted"); + expect((await f.scope.decide(f.message("continue"), true)).state).toBe("accepted"); clock += 15_001; await f.journal.observe(); expect(f.receipt()).toMatchObject({ cap: .5, state: "paused" }); expect(f.receipt().newCap).toBeUndefined(); - f.scope.decide(f.message("stop", "time-stop"), true); await child.exited; + (await f.scope.decide(f.message("stop", "time-stop"), true)); await child.exited; } finally { child?.kill("SIGKILL"); if (child) await child.exited; await f.close(); } }); @@ -409,7 +409,7 @@ test("namespace freeze never signals the host or an unrelated worker", async () expect(Bun.file(join(f.brain, "own-heartbeat")).size, "SCOPE_OWN_NAMESPACE_STAYS_FROZEN").toBe(ownBytes); } while (Bun.file(join(neighborBrain, "neighbor-heartbeat")).size <= neighborBytes && Date.now() < neighborDeadline); expect(Bun.file(join(neighborBrain, "neighbor-heartbeat")).size, "SCOPE_UNRELATED_NAMESPACE_STAYS_RUNNING").toBeGreaterThan(neighborBytes); - expect(f.scope.decide(f.message("stop"), true).state, "SCOPE_UNRELATED_WORKER_STOP_ACKNOWLEDGED").toBe("accepted"); await own.exited; + expect((await f.scope.decide(f.message("stop"), true)).state, "SCOPE_UNRELATED_WORKER_STOP_ACKNOWLEDGED").toBe("accepted"); await own.exited; } finally { own?.kill("SIGKILL"); neighbor?.kill("SIGKILL"); if (own) await own.exited; if (neighbor) await neighbor.exited; await f.close(); rmSync(neighborBrain, { recursive: true, force: true }); } }); @@ -424,8 +424,8 @@ test("failed freeze or resume stops the real worker, retains work and exposes an if (failure === "resume") { child.resume = () => { throw Error("Resume failed"); }; await f.journal.idle(); - const message = f.message("continue"); expect(f.scope.decide(message, true).state).toBe("failed"); - expect(f.scope.decide({ ...message, action: "readback" }, true).state, "SCOPE_FAILED_RESUME_ENDED_READBACK").toBe("ended_before_answer"); + const message = f.message("continue"); expect((await f.scope.decide(message, true)).state).toBe("failed"); + expect((await f.scope.decide({ ...message, action: "readback" }, true)).state, "SCOPE_FAILED_RESUME_ENDED_READBACK").toBe("ended_before_answer"); expect(f.receipt().state).toBe("ended_before_answer"); } else { expect(f.receipt()).toMatchObject({ state: "stopped", frozen: false }); expect(f.receipt().newCap).toBeUndefined(); diff --git a/packages/ui-server/tests/ws-main-model.test.ts b/packages/ui-server/tests/ws-main-model.test.ts index 3195a9147f..af1f25963c 100644 --- a/packages/ui-server/tests/ws-main-model.test.ts +++ b/packages/ui-server/tests/ws-main-model.test.ts @@ -39,6 +39,8 @@ for (const cancel of [false, true]) test(`runtime main identity preserves earlie } oldBridge = req.bridge; req.bridge.activity?.(promotion(`before-${started}`)); + // The row's attribution is taken when it is raised; it commits off the event loop (#2850). + await waitFor(() => queryGovernanceEvents(app.db, { limit: 100, turnId: req.turnId, kinds: ["promotion_refused"] }).length > 0); const before = queryGovernanceEvents(app.db, { limit: 100, turnId: req.turnId, kinds: ["promotion_refused"] }); expect(before[0]?.model, "MODEL_LIVE_EARLY_UNKNOWN").toBeNull(); const adapter = new StreamAdapter(req.bridge.activity); diff --git a/packages/ui-server/tests/ws-scope-caps.test.ts b/packages/ui-server/tests/ws-scope-caps.test.ts index 5f47b6ae71..0f3e5a4265 100644 --- a/packages/ui-server/tests/ws-scope-caps.test.ts +++ b/packages/ui-server/tests/ws-scope-caps.test.ts @@ -114,7 +114,10 @@ for (const { adapter, voice, overshoot } of cases) test(`${adapter}: ${overshoot return; } send({ type: "chat_message", text: "Record the return", requestId: "scope-request", source: "dictation" }); - await wait(paused, () => frames); + // Scope summaries reach the client once each commits (#2850): wait for the + // one that counts every call the worker has made so far. + const calls = () => new Set(frames.flatMap(frame => frame.type === "tool_use_start" || frame.type === "tool_use_complete" ? [frame.toolUseId] : [])).size; + await wait(() => paused() && latest()!.journal.scope!.receipts.at(-1)!.count === calls(), () => frames); const first = latest()!, receipt = first.journal.scope!.receipts.at(-1)!; expect(first.journal.budget, "COMBINED_SCOPE_PAUSE_RETAINS_BUDGET_FACTS").toBeDefined(); if (overshoot) {