[schlessera]

Closes #2789

What changed

The turn journal now keeps its tables in its own SQLite file, DB_PATH.journal.db. The file sits beside the UI database's physical file and is attached to every UI connection as journal. Every page in it is the journal's, so the 10 GiB cap now comes from that file's page_count, freelist_count and its own WAL. No b-tree walk runs on any write; #2812's journal-scoped walk is gone.

The move:

• createUiDb attaches the file before migrations. It moves every turn_journal_* table, with rows, indexes and triggers, before the first migration after 063, and again after the last.
• The copy writes only the journal file and the drop writes only the UI file. A crash between them leaves both copies, and the next start rebuilds the copy without firing a copied trigger before dropping.
• A later migration that changes journal tables is named *.journal.sql, creates its objects as journal.<name> and records its receipt in the journal file. The runner refuses one that changes the UI schema, and a test holds every migration after 063 to this rule.

Around the move:

• Journal commits set journal.synchronous=FULL.
• On start, a ledger returns released pages to the file system inside its first admitted write (incremental auto-vacuum).
• A ledger attaches the file and enforces foreign keys on any connection, such as ui-server: journal and Activity SQLite commits block the hosted event loop on fsync #2814's write worker. It refuses to start inside a transaction.
• On start, replaySteeringEvents restores a steering_change event whose UI-file commit was lost.
• The operational backup (brain-ui-inbox export) folds the journal tables back into its UI-database image, where earlier releases kept them, so its format is unchanged. A restore moves them out again on first start. Export also refuses the journal file, or the target of a link to it, as a destination.
• The file is masked from workers, refused inside the brain like the UI database, and inventoried as retained file history.

Why this shape

This is the maintainer's option-A ruling on #2789. A design without a walk was built and broken inside the shared file (evidence on the issue): a commit's new pages cannot be known before it commits, and other tables' frees in the same transaction feed pages to the journal. In its own file, ownership is exact by construction.

The interface with #2814 is agreed with that branch:

• createJournalAccounting(db, uiDbPath) keeps its signature and attaches the file itself.
• ui-server: journal and Activity SQLite commits block the hosted event loop on fsync #2814's writer attaches { schema: JOURNAL_SCHEMA, path: journalDatabasePath(config.dbPath) } on its own connection; the ledger's attach is then a no-op.
• Whichever PR lands second adds that attach entry.

Decision record: docs/decisions/journal-database.md.

Proof

Head 88688b3c9 on 15a5a3b4f. Package suites, the pack smoke, check:pr and both native proofs ran on the head. The mutation batch ran on 552545b14. The head differs from it by the native-proof script fix and a rebase onto #2840, whose turn-journal.ts queries are unqualified journal SQL.

• bun run test packages/ui-server packages/ui-sdk on the head: 5010 pass, 1 fail. The failure is http-supported-mounts.test.ts "a dirty failed mount discards its resources before the next independent case": 6 timers pending after an injected mount failure. It ran 3/3 green alone (109/109 each). This change adds no timers. On the previous base the same suites passed 5005/5005.
• packages/ui-react/tests/turn-journal-runtime.test.ts, whose fault-injection trigger is now qualified into the journal schema: 24/24 (on 479a0eba8).
• Pack smoke bun scripts/check-packages.ts on the head: every publishable package packed, installed and passed its probes.
• bun run check:pr --base origin/main on the head: passed, 1163 tests.
• The CONTRACT: feat(storage): deliver owner inventory, recovery and capacity guidance #2172 tests, packages/ui-server/tests/journal-accounting.test.ts, are unchanged and pass 10/10. The VACUUM case passes because a ledger returns released pages when it starts.
• The page-identity oracle walks both files and asserts three things: no journal b-tree is left in the UI database, the journal file holds only journal b-trees, and the charge covers every page the journal owns or released. Activity churn does not move the charge.

Native hosted proof, real harness, under flock …/native.lock. Only the proof command holds the lock; the three dd if=/dev/zero bs=1M count=1024 conv=fsync writers run outside it, started once the lock is free.

• With the fsync load, on the head: passed. All four cells, including HOSTED_ADMISSION_HEALTH_WITHIN_1_SECOND with the bound unchanged, plus the schedule-publication test; the retained-index reproduction failed as expected; exit 0.
• Without load, on the head: passed, same shape.
• Earlier attempts that did not pass:
  • On ae67c59d3, one no-load run hit the known intermittent invalid worker mount record refusal in the schedule-publication test (#2701), whose code this PR does not touch.
  • On the head, one fsync-load attempt timed out (180 s) waiting for the workspace output lease, which a host bun run test of mine held at the same time. It was rerun alone.
• The first attempts on this branch failed with no such table: turn_journal_reservations. The proof script itself opened ui.db raw at scripts/native-proof/hosted-app.test.ts:325, and now attaches the journal file there. A search of scripts/, every package src/ and every test fixture found no other raw UI connection that reads journal tables; the operational export attaches the file itself.

Mutations. Each row breaks one guard, runs the named suite and restores it. Every one fails.

Mutation	Failing assertion
J1 free pages uncharged	JOURNAL_FREED_METADATA_PAGES_REMAIN_CHARGED (#2172), oracle
J2 measure reads the UI file	JOURNAL_CHARGE_IGNORES_OTHER_COMPONENTS
J3 WAL of the UI file	JOURNAL_WAL_DEBT_STAYS_RESERVED
J4 journal commit not FULL	JOURNAL_COMMIT_FULL_ON_JOURNAL_FILE
J5 move keeps UI copies	legacy fixture: table … already exists
J6 rerun keeps the stale copy	interrupted-copy test
J7 indexes left behind	JOURNAL_MOVE_KEEPS_INDEXES
J8 ledger does not attach	bare-connection test
J9 no page return at start	JOURNAL_VACUUM_DROPS_NONEXISTENT_PAGE_DEBT (#2172)
J10 startup replay missing	JOURNAL_STARTUP_REPLAYS_STEERING
J11 replay under another key	JOURNAL_STEERING_REPLAY_IDEMPOTENT
J12 journal file not masked	OPERATIONAL_STATE_BOOT_REFUSAL (hardlinked journal layout)
J13 no escape guard	JOURNAL_GUARD_*
J14 backup without the journal	export refuses: inbox_snapshot_schema
J15 recovery command sees no journal	export refuses: inbox_snapshot_schema
J16 later migration before the move	no such table: journal.turn_journal_fixture
J17 journal receipt in the UI file	JOURNAL_MIGRATION_RECEIPT_IN_JOURNAL_FILE
J18 references unenforced	JOURNAL_LEDGER_ENFORCES_REFERENCES
J19 alias names its own journal	JOURNAL_ALIAS_SAME_FILE
J20 export overwrites a linked journal	JOURNAL_EXPORT_REFUSES_PHYSICAL
J21 journal migration may change the UI file	JOURNAL_MIGRATION_STAYS_IN_JOURNAL_FILE
J22 ledger starts inside a transaction	JOURNAL_LEDGER_REFUSES_INSIDE_TRANSACTION
J23 measurement walks a b-tree	JOURNAL_WRITE_WALKS_NO_BTREE

A migration that violates the convention (a probe 099_*.sql altering turn_journal_turns) fails UI_MIGRATION_TOUCHES_NO_JOURNAL_099_mutation_probe.sql.

Two independent gpt-6-astra review rounds ran.

• Round 1 found these problems, each now fixed and tested:
  • the backup omitted the journal, leaving a restore unbootable;
  • future migrations could commit apart from their receipts;
  • a rerun fired copied triggers;
  • symlinked paths broke WAL accounting;
  • the startup vacuum ran outside admission;
  • a bare connection had no FK enforcement.
• Round 2 found three more, also fixed and tested:
  • export could overwrite a journal file reached through a link;
  • unqualified creates in journal migrations;
  • construction inside a transaction.
• Accepted and documented in the decision record:
  • every writer must be stopped for the upgrade;
  • a crash between the two files' commits can keep an event without its change (SQLite commits the UI file first).

Checklist

• Focused local preflight/behavioral receipts and complete selected hosted proof passed; actual head/base/job checkout evidence is in Proof.
• Acceptance criteria from the issue are all met, or the gap is named below.
• The PR and completed/advanced issues have verified release milestones under docs/process/github.md.
• A changeset is included, or this changes nothing a consumer can see.
• No personal data anywhere in the diff — the leakage gate covers the whole tree.
• The integration contract index or relevant component is updated in this same commit, or nothing in the machine surface moved. The backup format, CLI and HTTP shapes are unchanged.
• Docs under docs/ reflect the change, or nothing documented moved.
• Provider history stays append-only (docs/decisions/provider-history.md): the provider-history guard passes, or nothing on a turn's path changed.

Anything left open

• No downgrade. A release before this one cannot open the database after the move; rolling back needs the backup taken before the upgrade. Self-hoster rollback guidance: schlessera/brain-hosting-template#39.
• Event timestamps. A replayed steering_change is timestamped with its turn's end, because the journal does not keep the original observation time. roadblock events are not replayed.
• ui-server: journal and Activity SQLite commits block the hosted event loop on fsync #2814 integration. The attach entry at openSqliteWriter is added by whichever PR lands second.
• The fsync-load lag criterion moved to ui-server: journal and Activity SQLite commits block the hosted event loop on fsync #2814 by ruling 1 and is not claimed here.

🤖 Generated with Claude Code