Repository navigation
Index store: one shared store replaces the trigram, SQLite call-graph and views indexes (0.64) - #1405
Open
aft-alfonso[bot] wants to merge 570 commits into
Open
aft-alfonso[bot] wants to merge 570 commits into
aft-alfonso[bot] wants to merge 570 commits into
Conversation
The health rollup measures every bound root every 3 seconds, and measuring an edited checkout walks each decoded parse and resolution its call-graph engine holds; on the live daemon that kept a core busy. A checkout now keeps its last figure with a memory epoch that moves on every recorded change, refresh and publication, and the census walks it again only when the epoch moved or the figure is 30 seconds old. Co-authored-by: alfonso-aft <[email protected]>
Delete tests of the old stores, legacy meaning loading and borrowed external indexes; count the three pending path sets and thirteen rollup steps that remain; point the held-tail bind test at the stage it holds. Tests that need a building index store hold its build (or its call graph after text publishes) instead of faking today's index channels, and the text-index test helper waits for the whole build so answers cached by text generation stay comparable. Co-authored-by: alfonso-aft <[email protected]>
…oved The stale-answer check runs before an update removes deleted files. A file deleted in the same batch as a manifest its type-qualified calls read kept a stale dispatch answer, and pass 3 then looked up its parse, which no longer exists, and panicked (seen live). Binding and resolution already kept only parsed files; dispatch now skips files no longer resolved. Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit a089992208181f730c5ef082b5292308dda10215)
Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
…uild Co-authored-by: alfonso-aft <[email protected]>
…nux, macOS) Also drops a stale binary test import and applies rustfmt to files the earlier automated deletions left unformatted. Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
…ully decoded graph Co-authored-by: alfonso-aft <[email protected]>
A checkout kept every call-graph file it ever decoded, with no limit, for the life of its publication: on AFT's repository one impact or trace_to left 250-360 MB behind, and the fully decoded graph is 880 MB. Decoded files now live in one least-recently-used cache for the whole process, 256 MB in all, and a closed set of stored files releases its own at once. The memory census reports it with the shared block cache (slice_* counters). Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 690075bcd8f2da8b7b638d33f279e883660fb687)
Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 50e4faf006c318f9f7d5340fca9d17cda516d321)
Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit cd6f0b8fded6aed8b7f2716596876680dcac1f54)
Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit b78772eeef5b01f22d4167d84bbdccd7d97d51ec)
…from the schema Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 585062a350ae1e7541e8d687c33008342208c898)
… away in TypeScript Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit accb9e0af2e83bb0de2f1dec2d0fe69469fe5a31)
Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit beb70667e3cecfcead426792e03faab0024480d1)
…rom the model Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 688dc1bfc5dd8bbb192a7b78802e676b2e8e73eb)
Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 5c4ca0ed3938ca7b5747c2f09283d1ed94abe0a0)
From 0.64 every index lives in <storage>/data/ and nothing opens the old stores, which can hold hundreds of gigabytes. `aft cache prune-legacy` lists them with sizes (dry run); `--yes` deletes them. `npx @cortexkit/aft cache prune-legacy` already forwards to it. What it removes, under the proven storage root and under a configured storage.cache_dir root that carries this storage root's owner marker: views/, blobs/, callgraph/, index/, symbols/, artifact-owners/, migration/, semantic-batch-leases-v1/, every child of semantic/ except semantic/models/ (the live embedding and reranker model cache that FASTEMBED_CACHE_DIR and semantic.local_cache_dir point at), each harness folder's callgraph/, and .prune-legacy-* leftovers. Safety: the storage root must hold aft.db or its owner marker; every candidate is opened without following links and must sit on the root's device and mount; a bounded size walk refuses mounts, hard links and special files; each folder is renamed aside into <root>/.prune-legacy-<time>-<pid>/ and only then deleted in bounded, no-follow passes; a folder that cannot be renamed is skipped and listed. A run holds retention/prune-legacy.lock and a shared hold on the daemon storage lock, so it runs beside AFT daemons but never beside `aft clean-old-caches`. Exit codes: 0 done, 1 error, 2 usage, 3 refused, 5 a folder was skipped. Health and status no longer read the old stores: - The health rollup no longer opens callgraph/<key>/build-breaker.sqlite, so suspended_domains stays empty and build_breaker_error is gone from diagnostics evidence. The tests that asserted breaker reuse and a standing root's durable breaker reason described that reader; the first now asserts the rollup never opens the breaker, the second is removed. - `status` no longer sizes index/<key> and semantic/<key>; the trigram_disk_bytes and semantic_disk_bytes fields stay, at 0, for the plugins and status push contract that read them (the index store's size is data_store_bytes). The two tests that asserted those sizes now assert the old folders are not sized even when they hold data. Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 5d881ef8a38816f5b39986d686f8f3bf2eb2de7a)
Tests that pinned the old call-graph store, views, the old text and meaning indexes, borrowed artifacts or their refresh workers are deleted. Tests that pin answers still shipped now bind the index store's checkout. views.enabled no longer reaches the configure params either plugin sends, matching the Rust resolver; the three parity goldens that carried it are recaptured. Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 8a91791a75a9d839eeb0e355ae26dc38d392c091)
…re (wip) Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit b77a23d958b0f0079a9a0117dd1882104dd7c26a)
…ing defects ignored (wip) Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 1f4d85e473334a44fd7c44da2bdd6770d13f254e)
Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit fe99a2a737d78e304828df21e61caad2f0b7b8a7)
…watcher test; format (wip) Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 384fa373183e195db55c82606e85efb2899bd24b)
Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit b4f9bd25ef4101935f6247accc03684d8f50822f)
Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit 985ad87a220ffe150c92c4b22a2d96349a5e4563)
…checkout Co-authored-by: alfonso-aft <[email protected]> (cherry picked from commit bcd820d1fa477aed040dc881585aab27e018b694)
…nothing pending Edits still reached the retired SQLite store's refresh worker, which nothing applies any more, so every call-graph answer about an edited file carried a 'call graph update pending' note for the life of the daemon (seen live). The store's refresh, reconcile and pending-query paths now stand down while it is retired. Dead code checked the same retired store to decide whether its graph was busy, so it never was; it now asks the index store whether the root has a published call graph, before a run opens the inspect cache and before a cached answer is served. The two tests marked ignored for these causes run again. Co-authored-by: alfonso-aft <[email protected]>
…uest files, wait timers); URL-fetch scope drop empties its slot with RefCell::replace Co-authored-by: alfonso-aft <[email protected]>
…ropping it After publishing the changes it resolved, a checkout's engine is moved onto the new publication (and onto a fold's) when it applied exactly the changes published and the publication holds the paths, parse keys and configuration it resolves over; otherwise it is dropped as before. It keeps what it bound and every file's method-call inputs, and reads everything else from the new publication, as a start from it would. Engines are let go when their checkout goes ten minutes without an edit or an applied refresh, when the engines of a data layer together hold more than a budget (least recently used first), or when the checkout unbinds. Co-authored-by: alfonso-aft <[email protected]>
…s check query dependencies without building them Walking an engine as the census counts it took 39 ms at the median and up to 122 ms on the 430-edit pack, a few percent of a refresh, so a refresh measures its engine only when it has not been measured in the last 30 s. A replay asked every recorded surface query for its dependency list, allocating each path again; the queries that depend on their file and fixed domain keys now check those directly. alloc_trace_probe prints each phase's user and system CPU, minor faults, glibc's allocator figures and the first refreshes' block-cache reads; engine_keep_probe replays the pack measuring the kept engine. Co-authored-by: alfonso-aft <[email protected]>
…ses by reason The budget stays 1 GiB: an engine of AFT's own checkout holds about 105 MB after its first refresh, and one kept across the 430-edit replay held 158/307/344 MiB at the median, 90th percentile and most, so three worktrees refreshing at once and a few more agents' worktrees fit. Each release line now counts every release so far by reason (idle, over budget, unbound), and an engine dropped with its unbound checkout is logged too, so the daemon's log shows how often the budget sends one away. Co-authored-by: alfonso-aft <[email protected]>
…prints per-refresh percentiles Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
…e fewer full drive flush (undo index fsynced, then committed by meta.json's full flush); pin both totals and full flushes Co-authored-by: alfonso-aft <[email protected]>
…permission rejection The bash permission loop (af0eab4) wraps the host's permission.assert failure in BashPermissionLoopError, a plain Error, so OpenCode 2's tool runtime refiled every shell refusal as tool.execution. Rebuild the host's Tool.Error and its Permission.BlockedError/CorrectedError from the caught instances, carrying AFT's unchanged refusal text where the host shows it, and keep the plain error whenever the host's errors differ from that shape. Co-authored-by: alfonso-aft <[email protected]>
… rejections Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
…al OpenCode 2 host checks them Co-authored-by: alfonso-aft <[email protected]>
…platform-form permission labels The access_grant_rows fixtures saved grants at /work/x, which is relative on Windows (no drive), so insert_from_card refused them with InvalidQuery as it should. They now use folders under the test's temporary directory in the stored policy_path form. permissions_list_rows_for_agent_and_worker_grants expected ~/Work/site; the handler shows the folder the way the platform writes it (~\Work\site on Windows). permissions/v1 sets no separator for subject_label or scope_label, so the expected labels are now built per platform; Unix still expects the exact ~/Work/site text. Co-authored-by: alfonso-aft <[email protected]>
… Duplicates; those analyses run only on aft_inspect Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
…s once per route, save draining-wait notes at a coarse step Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
…ilds Co-authored-by: alfonso-aft <[email protected]>
… release note Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
… nested sync groups join the enclosing one The undo mirror's own aft.db connection made set_db_pool run db::open_with_mode on every configure, retry and pool attach. That peeks the schema through a read-only connection on the resident database and then keeps a second read-write connection open, so the shared_db_tests that pin one resident connection per database failed (macOS, Windows, and Linux too, where CI skipped the libtest jobs). The mirror now writes through the shared connection. It waits at most 50 ms in total, first polling for the connection's mutex and then on SQLite's busy handler for the remainder. Past that, the stack is marked out of sync and the next mirror write for the file rewrites it from disk. snapshot_does_not_wait_for_the_shared_aft_db_connection_mutex asserted that the mirror row commits while the shared mutex is held, which only a second connection can do. It is replaced by held_shared_connection_bounds_the_snapshot_and_undo_restores_after_repair. That test checks the snapshot returns while the mutex is held, leaves the row out, has the next snapshot repair the stack, and has undo from a reopened store restore both edits. The snapshot's own SyncGroup, which lets meta.json and its folder share one flush, replaced aft_delete's enclosing group with a fresh pending set and full-flushed it on finish. So every pre-delete snapshot paid its own F_FULLFSYNC: 66 before the first unlink of a 64-file tree, and 5 for five abandoned captures. A group opened while another is open on the same thread now joins it, and its finish flushes nothing. Co-authored-by: alfonso-aft <[email protected]>
…ch the call graph does not parse Co-authored-by: alfonso-aft <[email protected]>
…gin and stops the row's service Co-authored-by: alfonso-aft <[email protected]>
…y its reconciliation check Co-authored-by: alfonso-aft <[email protected]>
Every bot-enabled shell launch stores one gh_speech_authorities row of about 84 KB (the start policy embeds the signed routing manifest as a JSON number array) and revocation only marked it revoked, so the table grew without bound. - The once-a-minute retention worker now deletes authorities revoked at least 24 hours ago, in transactions of at most 64 rows on the shared connection, within 100 ms per tick, logging examined and deleted counts. Revocation rows are kept, so the incarnation stays refused; rows referenced by a legacy queued dispatch are kept for outcome-unknown replay. - Revocation now rewrites the record without its start-policy copy, which only a live row's send gate reads. - A database with incremental auto-vacuum releases freed pages a bounded number per tick; the default (NONE) keeps them for reuse. Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
Co-authored-by: alfonso-aft <[email protected]>
…, bytes by plane, and simulated folds and vector merges Co-authored-by: alfonso-aft <[email protected]>
… merge vector segments A checkout folds its own segments only after it publishes changes, so a worktree left idle (folder still there) kept every segment it read. After a change publication that calls for no fold of its own, a checkout now folds, under the same writer, the one other checkout most due a fold by the existing counts (Checkout::fold_other), publishing no state. Each meaning fill round stored its vectors in a segment of its own, which nothing folded. Past 8 vector segments a round now merges vector-only segments older than 10 minutes, cut at 64 MiB of copied vectors, keeping exactly the chunk maps and vectors a reader reaches: every published state plus the current states of checkouts bound in this process. On copies of two live stores: 213 -> 30 segments (5.2 -> 1.9 GB) and 335 -> 20 (1.6 -> 0.6 GB), meaning search identical. Measurement example: crates/aft/examples/data (example 'data'); the store census example gains AFT_CENSUS_RECLAIM=1. Co-authored-by: alfonso-aft <[email protected]>
…r, which appears only when a count changes Co-authored-by: alfonso-aft <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the 0.64 index-store cutover. Text search, the call graph and meaning search move to one index store per repository, shared by its worktrees and clones; the old trigram index, SQLite call-graph store and content-addressed views are deleted (about 215,000 lines net).
What changes for users
npx @cortexkit/aft cache prune-legacylists and removes them.views.enabled,views.callgraph_copiesandworktree.ram_overlayshow a removal notice;views.idle_checkout_hoursbecomesidle.checkout_hours. Config names retired in 0.62 or earlier are refused in user config and load with a notice in project config.trace_toandimpactlabel each root (main, test, attribute, public API, no caller found).Measured on AFT's own repository (7,300 files)
Call-graph build 207 s to 17 s; index disk 1.7 GB to 0.3 GB; typical edit catch-up 4.7 s to 0.1 s; dead code first run 72 s / +4.25 GB to 6-8 s / +0.8 GB; an edited worktree's call graph median 3 MiB between publications; a new worktree at the same commit under a second instead of about 160 CPU-seconds.
Verification
land-index-store-0.64.json).The release notes are in review; the version bump and tag follow through
scripts/release.shafter this merges.Summary by cubic
This is the 0.64 index-store cutover: text search, the call graph, meaning search, dead code and TODO counts all move to one index store per repository, shared by its worktrees and clones. The old trigram index, SQLite call-graph store and views are deleted (about 215,000 lines net). The data layer is always on: there is no development switch, and configure no longer builds or opens today's indexes at all.
What changes for users
Cargo.toml,tsconfig.json,package.json) rebinds only the files it changed: crate roots, found config files and workspace package names are tracked per name, not per workspace member walk.write,editandapply_patchreturn before the edited file's syntax tree is built: search sees the edit at once, and the call graph parses it on its next refresh.storage_requires_newer_reader, naming the file and version, instead of being rebuilt over them; a store an older build published still serves text while the checkout's own bind rebuilds its call graph, and a read-only bind that still fails waits a minute before retrying.callgraph_buildingwhile the build sits queued.npx @cortexkit/aft cache prune-legacylists and removes them, andstatusno longer sizes them.views.enabled,views.callgraph_copiesandworktree.ram_overlayshow a removal notice;views.idle_checkout_hoursbecomesidle.checkout_hours.trace_toandimpactfollow callers to real roots and label each root: main, test, attribute, public API, or no caller found.*.min.js, or JS/CSS with almost no line breaks) are still searched by grep, but are not parsed for the call graph, cut into meaning-search chunks, or counted as files the call graph fails to cover.service.render()resolves to the exact method, and a Python dotted import no longer records an unresolved call site.aft_searchreads each candidate file once per search, no longer serializes concurrent searches' query embeddings, matches file-name searches against the index's path list instead of walking the project, and again carries each symbol hit's caller count and caller files from the store's current call graph.Measured and verified
Written for commit a4df3b3. Summary will update on new commits.