Skip to content

Index store: one shared store replaces the trigram, SQLite call-graph and views indexes (0.64) - #1405

Open
aft-alfonso[bot] wants to merge 570 commits into
mainfrom
land/index-store-0.64
Open

aft-alfonso[bot] wants to merge 570 commits into
mainfrom
land/index-store-0.64

Conversation

@aft-alfonso

@aft-alfonso aft-alfonso Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

This is the 0.64 index-store cutover. Text search, the call graph and meaning search move to one index store per repository, shared by its worktrees and clones; the old trigram index, SQLite call-graph store and content-addressed views are deleted (about 215,000 lines net).

What changes for users

  • Each repository builds its new indexes once on first use: text in about a second, the call graph in under a minute for large projects. Worktrees of one repository share them, and a new worktree starts from a sibling's indexes and processes only the files that differ.
  • Meaning search reuses old vectors for unchanged files embedded with the same model.
  • Old index folders stay on disk; npx @cortexkit/aft cache prune-legacy lists and removes them.
  • views.enabled, views.callgraph_copies and worktree.ram_overlay show a removal notice; views.idle_checkout_hours becomes idle.checkout_hours. Config names retired in 0.62 or earlier are refused in user config and load with a notice in project config.
  • trace_to and impact label each root (main, test, attribute, public API, no caller found).

Measured on AFT's own repository (7,300 files)
Call-graph build 207 s to 17 s; index disk 1.7 GB to 0.3 GB; typical edit catch-up 4.7 s to 0.1 s; dead code first run 72 s / +4.25 GB to 6-8 s / +0.8 GB; an edited worktree's call graph median 3 MiB between publications; a new worktree at the same commit under a second instead of about 160 CPU-seconds.

Verification

  • Full Rust suite on Linux: all pass except the 12 native-secret-shell tests and one rust-analyzer test the build server can't run (both fail the same way on main).
  • Incremental call graph equals a cold build after each of 430 real edits (edit, reuse, extract and checkout gates).
  • Dead-code findings unchanged across a 91-project corpus; search-quality benchmark rows byte-identical except five declared pattern-lane candidate counts (land-index-store-0.64.json).
  • macOS and Windows compile with warnings as errors.

The release notes are in review; the version bump and tag follow through scripts/release.sh after this merges.


Summary by cubic

This is the 0.64 index-store cutover: text search, the call graph, meaning search, dead code and TODO counts all move to one index store per repository, shared by its worktrees and clones. The old trigram index, SQLite call-graph store and views are deleted (about 215,000 lines net). The data layer is always on: there is no development switch, and configure no longer builds or opens today's indexes at all.

What changes for users

  • Each repository builds its new indexes once on first use; worktrees share them, and a new worktree starts from a sibling's indexes and processes only the files that differ.
  • Meaning search reuses stored vectors for unchanged files embedded with the same model, and a repository is filled by one process at a time so no chunk is embedded twice.
  • Editing a config file (Cargo.toml, tsconfig.json, package.json) rebinds only the files it changed: crate roots, found config files and workspace package names are tracked per name, not per workspace member walk.
  • write, edit and apply_patch return before the edited file's syntax tree is built: search sees the edit at once, and the call graph parses it on its next refresh.
  • A repository whose index-store files a newer build wrote is refused with storage_requires_newer_reader, naming the file and version, instead of being rebuilt over them; a store an older build published still serves text while the checkout's own bind rebuilds its call graph, and a read-only bind that still fails waits a minute before retrying.
  • A call-graph build or refresh a read asked for is admitted to its pool immediately instead of waiting behind a queued maintenance job, so concurrent reads no longer answer callgraph_building while the build sits queued.
  • Old index folders stay on disk; npx @cortexkit/aft cache prune-legacy lists and removes them, and status no longer sizes them.
  • views.enabled, views.callgraph_copies and worktree.ram_overlay show a removal notice; views.idle_checkout_hours becomes idle.checkout_hours.
  • Config names retired in 0.62 or earlier are refused in user config; a project config carrying one still loads, translating the name with a notice.
  • Configure's missing-tool and language-server warnings take their file list from the index-store checkout instead of walking the project once it has published; the same list feeds the background TODO refresh.
  • trace_to and impact follow callers to real roots and label each root: main, test, attribute, public API, or no caller found.
  • Index-store background work (builds, meaning fills, refreshes) runs in process-wide pools, so a fill round waiting on the embedding backend no longer holds every other root's builds and tool calls; all call-graph engines resolve on one shared thread pool.
  • When a root's index store fails to build or refresh, the daemon's health report names the root and its reason code instead of a count.
  • Minified bundles (*.min.js, or JS/CSS with almost no line breaks) are still searched by grep, but are not parsed for the call graph, cut into meaning-search chunks, or counted as files the call graph fails to cover.
  • Dead code and unused exports run on the index store's own integer graph, with per-file facts extracted at index time; TODO counts come from stored markers, and both list their files from the store's checkout instead of walking the project. A member kept alive by a test-code call is now a test seed, a call like service.render() resolves to the exact method, and a Python dotted import no longer records an unresolved call site.
  • Automatic dead code and unused exports follow their own schedule: they wait for a published call graph, run on one root at a time, and at most once every 30 minutes while edits arrive.
  • aft_search reads each candidate file once per search, no longer serializes concurrent searches' query embeddings, matches file-name searches against the index's path list instead of walking the project, and again carries each symbol hit's caller count and caller files from the store's current call graph.
  • A worktree that has stopped publishing no longer keeps every segment it ever read in the store: another checkout's publication copies what the idle one still reads into one segment and drops the rest, and meaning-search vector segments are combined into one once a repository has more than eight of them.

Measured and verified

  • On AFT's own repository (7,300 files): call-graph build 207 s to 17 s, index disk 1.7 GB to 0.3 GB, typical edit catch-up 4.7 s to 0.1 s, new worktree at the same commit under a second instead of about 160 CPU-seconds, and a busy worktree's call graph holds about a tenth of the memory after a broad edit (about 120 MB instead of 1 GB).
  • The store keeps at most 1,024 index files open and one descriptor per segment file, so busy daemons stay below macOS's open-file limit; a busy worktree's piled-up index changes are folded into one segment without rereading or reparsing any file (under a second on 7,000 files, where the rebuild took about 12 s and over 1 GB); decoded call-graph files share a 256 MB bounded cache; the segment block cache shards by block and serves hits under a read lock (43 ns per warm hit on one thread, 357 ns each on twelve); a finished build returns its freed pages to the allocator, so resident memory settles near the working set; the memory census carries the meaning fill's own timings into daemon health, and the health rollup reuses a checkout's memory figure until its recorded changes, refresh or publication move its epoch or 30 seconds pass.
  • A publication carries the call-graph engine that built it, so the refresh after an edit reuses it instead of building a new one; on 430 replayed edits one engine served all 88 publications, using about a third less CPU. A refresh never holds decoded copies of files it only checks: a crate-root edit held about 360 MB instead of 1.5 GB.
  • A refresh publishes once 256 files have been re-resolved since the last publication, so a broad edit's re-resolved files are released instead of being held until the checkout goes quiet.
  • Incremental call graph equals a cold build after each of 430 real edits; search-quality benchmark rows byte-identical except five declared pattern-lane candidate counts.
  • Replaying 6,956 recorded callers/call_tree/trace replies through the new store matches main's corpus except 34 reviewed differences (C++ type-matched method calls, macro-template calls and macro-generated self calls now resolve).
  • Full Rust suite passes on Linux except the 12 native-secret-shell tests and one rust-analyzer test the build server can't run (both fail the same way on main); macOS and Windows compile with warnings as errors.

Written for commit a4df3b3. Summary will update on new commits.

View guided diff Turn on auto-fix

ualtinok and others added 30 commits November 6, 2026 20:56
The health rollup measures every bound root every 3 seconds, and measuring
an edited checkout walks each decoded parse and resolution its call-graph
engine holds; on the live daemon that kept a core busy. A checkout now keeps
its last figure with a memory epoch that moves on every recorded change,
refresh and publication, and the census walks it again only when the epoch
moved or the figure is 30 seconds old.

Co-authored-by: alfonso-aft <[email protected]>
Delete tests of the old stores, legacy meaning loading and borrowed external
indexes; count the three pending path sets and thirteen rollup steps that
remain; point the held-tail bind test at the stage it holds. Tests that need a
building index store hold its build (or its call graph after text publishes)
instead of faking today's index channels, and the text-index test helper waits
for the whole build so answers cached by text generation stay comparable.

Co-authored-by: alfonso-aft <[email protected]>
…oved

The stale-answer check runs before an update removes deleted files. A file
deleted in the same batch as a manifest its type-qualified calls read kept a
stale dispatch answer, and pass 3 then looked up its parse, which no longer
exists, and panicked (seen live). Binding and resolution already kept only
parsed files; dispatch now skips files no longer resolved.

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit a089992208181f730c5ef082b5292308dda10215)
…nux, macOS)

Also drops a stale binary test import and applies rustfmt to files the
earlier automated deletions left unformatted.

Co-authored-by: alfonso-aft <[email protected]>
A checkout kept every call-graph file it ever decoded, with no limit, for
the life of its publication: on AFT's repository one impact or trace_to left
250-360 MB behind, and the fully decoded graph is 880 MB. Decoded files now
live in one least-recently-used cache for the whole process, 256 MB in all,
and a closed set of stored files releases its own at once. The memory census
reports it with the shared block cache (slice_* counters).

Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 690075bcd8f2da8b7b638d33f279e883660fb687)
Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 50e4faf006c318f9f7d5340fca9d17cda516d321)
Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit cd6f0b8fded6aed8b7f2716596876680dcac1f54)
Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit b78772eeef5b01f22d4167d84bbdccd7d97d51ec)
…from the schema

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 585062a350ae1e7541e8d687c33008342208c898)
… away in TypeScript

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit accb9e0af2e83bb0de2f1dec2d0fe69469fe5a31)
Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit beb70667e3cecfcead426792e03faab0024480d1)
…rom the model

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 688dc1bfc5dd8bbb192a7b78802e676b2e8e73eb)
Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 5c4ca0ed3938ca7b5747c2f09283d1ed94abe0a0)
From 0.64 every index lives in <storage>/data/ and nothing opens the old
stores, which can hold hundreds of gigabytes. `aft cache prune-legacy`
lists them with sizes (dry run); `--yes` deletes them. `npx @cortexkit/aft
cache prune-legacy` already forwards to it.

What it removes, under the proven storage root and under a configured
storage.cache_dir root that carries this storage root's owner marker:
views/, blobs/, callgraph/, index/, symbols/, artifact-owners/,
migration/, semantic-batch-leases-v1/, every child of semantic/ except
semantic/models/ (the live embedding and reranker model cache that
FASTEMBED_CACHE_DIR and semantic.local_cache_dir point at), each
harness folder's callgraph/, and .prune-legacy-* leftovers.

Safety: the storage root must hold aft.db or its owner marker; every
candidate is opened without following links and must sit on the root's
device and mount; a bounded size walk refuses mounts, hard links and
special files; each folder is renamed aside into
<root>/.prune-legacy-<time>-<pid>/ and only then deleted in bounded,
no-follow passes; a folder that cannot be renamed is skipped and listed.
A run holds retention/prune-legacy.lock and a shared hold on the daemon
storage lock, so it runs beside AFT daemons but never beside
`aft clean-old-caches`. Exit codes: 0 done, 1 error, 2 usage, 3 refused,
5 a folder was skipped.

Health and status no longer read the old stores:
- The health rollup no longer opens callgraph/<key>/build-breaker.sqlite,
  so suspended_domains stays empty and build_breaker_error is gone from
  diagnostics evidence. The tests that asserted breaker reuse and a
  standing root's durable breaker reason described that reader; the
  first now asserts the rollup never opens the breaker, the second is
  removed.
- `status` no longer sizes index/<key> and semantic/<key>; the
  trigram_disk_bytes and semantic_disk_bytes fields stay, at 0, for the
  plugins and status push contract that read them (the index store's
  size is data_store_bytes). The two tests that asserted those sizes now
  assert the old folders are not sized even when they hold data.

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 5d881ef8a38816f5b39986d686f8f3bf2eb2de7a)
Tests that pinned the old call-graph store, views, the old text and
meaning indexes, borrowed artifacts or their refresh workers are deleted.
Tests that pin answers still shipped now bind the index store's checkout.
views.enabled no longer reaches the configure params either plugin sends,
matching the Rust resolver; the three parity goldens that carried it are
recaptured.

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 8a91791a75a9d839eeb0e355ae26dc38d392c091)
…re (wip)

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit b77a23d958b0f0079a9a0117dd1882104dd7c26a)
…ing defects ignored (wip)

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 1f4d85e473334a44fd7c44da2bdd6770d13f254e)
Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit fe99a2a737d78e304828df21e61caad2f0b7b8a7)
…watcher test; format (wip)

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 384fa373183e195db55c82606e85efb2899bd24b)
Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit b4f9bd25ef4101935f6247accc03684d8f50822f)
Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit 985ad87a220ffe150c92c4b22a2d96349a5e4563)
…checkout

Co-authored-by: alfonso-aft <[email protected]>
(cherry picked from commit bcd820d1fa477aed040dc881585aab27e018b694)
…nothing pending

Edits still reached the retired SQLite store's refresh worker, which nothing
applies any more, so every call-graph answer about an edited file carried a
'call graph update pending' note for the life of the daemon (seen live). The
store's refresh, reconcile and pending-query paths now stand down while it is
retired. Dead code checked the same retired store to decide whether its graph
was busy, so it never was; it now asks the index store whether the root has a
published call graph, before a run opens the inspect cache and before a cached
answer is served. The two tests marked ignored for these causes run again.

Co-authored-by: alfonso-aft <[email protected]>
ualtinok and others added 30 commits November 9, 2026 11:40
…uest files, wait timers); URL-fetch scope drop empties its slot with RefCell::replace

Co-authored-by: alfonso-aft <[email protected]>
…ropping it

After publishing the changes it resolved, a checkout's engine is moved
onto the new publication (and onto a fold's) when it applied exactly the
changes published and the publication holds the paths, parse keys and
configuration it resolves over; otherwise it is dropped as before. It
keeps what it bound and every file's method-call inputs, and reads
everything else from the new publication, as a start from it would.

Engines are let go when their checkout goes ten minutes without an edit
or an applied refresh, when the engines of a data layer together hold
more than a budget (least recently used first), or when the checkout
unbinds.

Co-authored-by: alfonso-aft <[email protected]>
…s check query dependencies without building them

Walking an engine as the census counts it took 39 ms at the median and
up to 122 ms on the 430-edit pack, a few percent of a refresh, so a
refresh measures its engine only when it has not been measured in the
last 30 s. A replay asked every recorded surface query for its
dependency list, allocating each path again; the queries that depend on
their file and fixed domain keys now check those directly.

alloc_trace_probe prints each phase's user and system CPU, minor faults,
glibc's allocator figures and the first refreshes' block-cache reads;
engine_keep_probe replays the pack measuring the kept engine.

Co-authored-by: alfonso-aft <[email protected]>
…ses by reason

The budget stays 1 GiB: an engine of AFT's own checkout holds about
105 MB after its first refresh, and one kept across the 430-edit replay
held 158/307/344 MiB at the median, 90th percentile and most, so three
worktrees refreshing at once and a few more agents' worktrees fit. Each
release line now counts every release so far by reason (idle, over
budget, unbound), and an engine dropped with its unbound checkout is
logged too, so the daemon's log shows how often the budget sends one
away.

Co-authored-by: alfonso-aft <[email protected]>
…prints per-refresh percentiles

Co-authored-by: alfonso-aft <[email protected]>
…e fewer full drive flush (undo index fsynced, then committed by meta.json's full flush); pin both totals and full flushes

Co-authored-by: alfonso-aft <[email protected]>
…permission rejection

The bash permission loop (af0eab4) wraps the host's permission.assert
failure in BashPermissionLoopError, a plain Error, so OpenCode 2's tool
runtime refiled every shell refusal as tool.execution. Rebuild the host's
Tool.Error and its Permission.BlockedError/CorrectedError from the caught
instances, carrying AFT's unchanged refusal text where the host shows it,
and keep the plain error whenever the host's errors differ from that shape.

Co-authored-by: alfonso-aft <[email protected]>
…al OpenCode 2 host checks them

Co-authored-by: alfonso-aft <[email protected]>
…platform-form permission labels

The access_grant_rows fixtures saved grants at /work/x, which is relative on
Windows (no drive), so insert_from_card refused them with InvalidQuery as it
should. They now use folders under the test's temporary directory in the
stored policy_path form.

permissions_list_rows_for_agent_and_worker_grants expected ~/Work/site; the
handler shows the folder the way the platform writes it (~\Work\site on
Windows). permissions/v1 sets no separator for subject_label or scope_label,
so the expected labels are now built per platform; Unix still expects the
exact ~/Work/site text.

Co-authored-by: alfonso-aft <[email protected]>
… Duplicates; those analyses run only on aft_inspect

Co-authored-by: alfonso-aft <[email protected]>
…s once per route, save draining-wait notes at a coarse step

Co-authored-by: alfonso-aft <[email protected]>
… nested sync groups join the enclosing one

The undo mirror's own aft.db connection made set_db_pool run
db::open_with_mode on every configure, retry and pool attach. That
peeks the schema through a read-only connection on the resident
database and then keeps a second read-write connection open, so the
shared_db_tests that pin one resident connection per database failed
(macOS, Windows, and Linux too, where CI skipped the libtest jobs).
The mirror now writes through the shared connection. It waits at most
50 ms in total, first polling for the connection's mutex and then on
SQLite's busy handler for the remainder. Past that, the stack is
marked out of sync and the next mirror write for the file rewrites it
from disk.

snapshot_does_not_wait_for_the_shared_aft_db_connection_mutex asserted
that the mirror row commits while the shared mutex is held, which only
a second connection can do. It is replaced by
held_shared_connection_bounds_the_snapshot_and_undo_restores_after_repair.
That test checks the snapshot returns while the mutex is held, leaves
the row out, has the next snapshot repair the stack, and has undo from
a reopened store restore both edits.

The snapshot's own SyncGroup, which lets meta.json and its folder share
one flush, replaced aft_delete's enclosing group with a fresh pending
set and full-flushed it on finish. So every pre-delete snapshot paid its
own F_FULLFSYNC: 66 before the first unlink of a 64-file tree, and 5 for
five abandoned captures. A group opened while another is open on the
same thread now joins it, and its finish flushes nothing.

Co-authored-by: alfonso-aft <[email protected]>
…ch the call graph does not parse

Co-authored-by: alfonso-aft <[email protected]>
…gin and stops the row's service

Co-authored-by: alfonso-aft <[email protected]>
…y its reconciliation check

Co-authored-by: alfonso-aft <[email protected]>
Every bot-enabled shell launch stores one gh_speech_authorities row of about
84 KB (the start policy embeds the signed routing manifest as a JSON number
array) and revocation only marked it revoked, so the table grew without bound.

- The once-a-minute retention worker now deletes authorities revoked at least
  24 hours ago, in transactions of at most 64 rows on the shared connection,
  within 100 ms per tick, logging examined and deleted counts. Revocation rows
  are kept, so the incarnation stays refused; rows referenced by a legacy
  queued dispatch are kept for outcome-unknown replay.
- Revocation now rewrites the record without its start-policy copy, which only
  a live row's send gate reads.
- A database with incremental auto-vacuum releases freed pages a bounded
  number per tick; the default (NONE) keeps them for reuse.

Co-authored-by: alfonso-aft <[email protected]>
…, bytes by plane, and simulated folds and vector merges

Co-authored-by: alfonso-aft <[email protected]>
… merge vector segments

A checkout folds its own segments only after it publishes changes, so a
worktree left idle (folder still there) kept every segment it read. After a
change publication that calls for no fold of its own, a checkout now folds,
under the same writer, the one other checkout most due a fold by the
existing counts (Checkout::fold_other), publishing no state.

Each meaning fill round stored its vectors in a segment of its own, which
nothing folded. Past 8 vector segments a round now merges vector-only
segments older than 10 minutes, cut at 64 MiB of copied vectors, keeping
exactly the chunk maps and vectors a reader reaches: every published state
plus the current states of checkouts bound in this process.

On copies of two live stores: 213 -> 30 segments (5.2 -> 1.9 GB) and
335 -> 20 (1.6 -> 0.6 GB), meaning search identical. Measurement example:
crates/aft/examples/data (example 'data'); the store census example gains
AFT_CENSUS_RECLAIM=1.

Co-authored-by: alfonso-aft <[email protected]>
…r, which appears only when a count changes

Co-authored-by: alfonso-aft <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant