Repository navigation
perf(resolve): bound resolution-tail memory on very large graphs - #143
Merged
Merged
Conversation
Ports the applicable part of upstream colby #1212. Most of that change is TypeScript/Node-specific (event-loop yield points, the kotlin-expect-actual / C-fn-pointer / spring synthesizers this port does not have) or already shipped here (the WAL valve and the daemon session fixes landed in v0.29.1), so the Rust-relevant subset is narrow. Batched reference resolution built a SnapshotResolutionContext holding the whole node set across several in-memory maps so the rayon path could resolve batches in parallel. That is fine at normal scale but O(graph) in the heap at kernel scale. Add a large-graph fallback: above RESOLVE_STREAMING_NODE_THRESHOLD (1,000,000 nodes, overridable via CODEGRAPH_RESOLVE_STREAMING_THRESHOLD) each batch resolves serially through a per-batch StoreResolutionContext backed by the live store's bounded LRU caches instead of the whole-graph snapshot, so memory is bounded to one batch plus the caches. The rowid cursor, persistence order, deletion order, and post-resolution passes are unchanged, and prior-batch edges stay visible through the live store, so the resolved graph is byte-for-byte identical to the snapshot path — proven by a dual-path equivalence test that forces >=2 batches with a cross-batch edge and asserts identical nodes/edges/unresolved_refs. Normal projects never cross the threshold and keep the exact snapshot+rayon path. Also extend the existing WAL-checkpoint valve to fire after the resolution-batch writes (not only during spill replay), bounding WAL growth through the resolution tail. The valve is TRUNCATE-only and row-neutral, so index output is byte-identical; env CODEGRAPH_WAL_VALVE_MB / CODEGRAPH_NO_WAL_DEFER govern it as before. Golden .schema and all fixtures unchanged (no regen).
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #143 +/- ##
==========================================
+ Coverage 95.53% 95.55% +0.01%
==========================================
Files 115 115
Lines 57233 57429 +196
==========================================
+ Hits 54679 54877 +198
+ Misses 2554 2552 -2
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upstream-sync DEFER item E — ports the applicable part of colby #1212 (
a3f9008). Most of #1212 is TS/Node-specific (event-loop yield points; the kotlin-expect-actual / C-fn-pointer / spring synthesizers this port lacks) or already shipped here (WAL valve + daemon session fixes landed in v0.29.1), so the Rust-relevant subset is narrow. The 7-part classification is recorded in the local ledger.PORT 1 — large-graph streaming fallback. Batched resolution built a
SnapshotResolutionContextholding the whole node set across several in-memory maps (needed by the rayon parallel path) — O(graph) heap at kernel scale. New fallback: aboveRESOLVE_STREAMING_NODE_THRESHOLD(1,000,000 nodes; envCODEGRAPH_RESOLVE_STREAMING_THRESHOLD), each batch resolves serially through a per-batchStoreResolutionContextbacked by the live store's bounded LRU caches — memory bounded to one batch + caches. Rowid cursor, persistence order, deletion order, and post-resolution passes unchanged; prior-batch edges stay visible via the live store, so the resolved graph is byte-identical to the snapshot path. Normal projects never cross the threshold → exact same path as today.PORT 2 — WAL valve in the resolution write loop. The existing TRUNCATE-only, row-neutral valve now also fires after resolution-batch writes (not just spill replay), bounding WAL through the resolution tail.
CODEGRAPH_WAL_VALVE_MB/CODEGRAPH_NO_WAL_DEFERgovern it as before; index output byte-identical.Verification
make cigreen (fmt + clippy + test + guardrail + equivalence).streaming_fallback_matches_snapshot_across_multiple_batches— forces ≥2 batches (batch size 1) with a cross-batchSub extends Base/sub.ping()case, asserts byte-identicalnodes/edges/unresolved_refsbetween the snapshot+rayon and streaming paths on separate identical stores (teeth-proven: a deliberate perturbation failed the assertion, then reverted).wal_valve_fires_during_resolution_batch_writes— valve fires during resolution (telemetry counter), 0 when off, resolved graph byte-identical on vs off..schema+ fixtures byte-identical, NO regen (memory/streaming refactor only).DEFER item E of the 1.2.0→1.4.1 upstream sync (the remaining DEFER work is prompt-hook + the language expansions).