Skip to content

perf(resolve): bound resolution-tail memory on very large graphs - #143

Merged
sunerpy merged 1 commit into
mainfrom
feat/upstream-1.4.1-defer-e-streaming-resolution
Jul 14, 2026
Merged

sunerpy merged 1 commit into
mainfrom
feat/upstream-1.4.1-defer-e-streaming-resolution

Conversation

@sunerpy

@sunerpy sunerpy commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Summary

Upstream-sync DEFER item E — ports the applicable part of colby #1212 (a3f9008). Most of #1212 is TS/Node-specific (event-loop yield points; the kotlin-expect-actual / C-fn-pointer / spring synthesizers this port lacks) or already shipped here (WAL valve + daemon session fixes landed in v0.29.1), so the Rust-relevant subset is narrow. The 7-part classification is recorded in the local ledger.

PORT 1 — large-graph streaming fallback. Batched resolution built a SnapshotResolutionContext holding the whole node set across several in-memory maps (needed by the rayon parallel path) — O(graph) heap at kernel scale. New fallback: above RESOLVE_STREAMING_NODE_THRESHOLD (1,000,000 nodes; env CODEGRAPH_RESOLVE_STREAMING_THRESHOLD), each batch resolves serially through a per-batch StoreResolutionContext backed by the live store's bounded LRU caches — memory bounded to one batch + caches. Rowid cursor, persistence order, deletion order, and post-resolution passes unchanged; prior-batch edges stay visible via the live store, so the resolved graph is byte-identical to the snapshot path. Normal projects never cross the threshold → exact same path as today.

PORT 2 — WAL valve in the resolution write loop. The existing TRUNCATE-only, row-neutral valve now also fires after resolution-batch writes (not just spill replay), bounding WAL through the resolution tail. CODEGRAPH_WAL_VALVE_MB / CODEGRAPH_NO_WAL_DEFER govern it as before; index output byte-identical.

Verification

  • make ci green (fmt + clippy + test + guardrail + equivalence).
  • streaming_fallback_matches_snapshot_across_multiple_batches — forces ≥2 batches (batch size 1) with a cross-batch Sub extends Base / sub.ping() case, asserts byte-identical nodes/edges/unresolved_refs between the snapshot+rayon and streaming paths on separate identical stores (teeth-proven: a deliberate perturbation failed the assertion, then reverted).
  • wal_valve_fires_during_resolution_batch_writes — valve fires during resolution (telemetry counter), 0 when off, resolved graph byte-identical on vs off.
  • Golden .schema + fixtures byte-identical, NO regen (memory/streaming refactor only).
  • Independent momus review: APPROVE (identical-resolution guarantee verified: global node visibility via live store, prior-batch edges committed, LRUs cache identical ordered queries, ordering preserved).

DEFER item E of the 1.2.0→1.4.1 upstream sync (the remaining DEFER work is prompt-hook + the language expansions).

Ports the applicable part of upstream colby #1212. Most of that change is
TypeScript/Node-specific (event-loop yield points, the kotlin-expect-actual /
C-fn-pointer / spring synthesizers this port does not have) or already shipped
here (the WAL valve and the daemon session fixes landed in v0.29.1), so the
Rust-relevant subset is narrow.

Batched reference resolution built a SnapshotResolutionContext holding the whole
node set across several in-memory maps so the rayon path could resolve batches in
parallel. That is fine at normal scale but O(graph) in the heap at kernel scale.
Add a large-graph fallback: above RESOLVE_STREAMING_NODE_THRESHOLD (1,000,000
nodes, overridable via CODEGRAPH_RESOLVE_STREAMING_THRESHOLD) each batch resolves
serially through a per-batch StoreResolutionContext backed by the live store's
bounded LRU caches instead of the whole-graph snapshot, so memory is bounded to
one batch plus the caches. The rowid cursor, persistence order, deletion order,
and post-resolution passes are unchanged, and prior-batch edges stay visible
through the live store, so the resolved graph is byte-for-byte identical to the
snapshot path — proven by a dual-path equivalence test that forces >=2 batches
with a cross-batch edge and asserts identical nodes/edges/unresolved_refs. Normal
projects never cross the threshold and keep the exact snapshot+rayon path.

Also extend the existing WAL-checkpoint valve to fire after the resolution-batch
writes (not only during spill replay), bounding WAL growth through the
resolution tail. The valve is TRUNCATE-only and row-neutral, so index output is
byte-identical; env CODEGRAPH_WAL_VALVE_MB / CODEGRAPH_NO_WAL_DEFER govern it as
before.

Golden .schema and all fixtures unchanged (no regen).
@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.71362% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/codegraph-resolve/src/resolver.rs 96.71% 7 Missing ⚠️

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #143      +/-   ##
==========================================
+ Coverage   95.53%   95.55%   +0.01%     
==========================================
  Files         115      115              
  Lines       57233    57429     +196     
==========================================
+ Hits        54679    54877     +198     
+ Misses       2554     2552       -2     
Files with missing lines Coverage Δ
crates/codegraph-resolve/src/resolver.rs 96.31% <96.71%> (+0.26%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sunerpy
sunerpy merged commit 696c681 into main Jul 14, 2026
7 checks passed
@sunerpy
sunerpy deleted the feat/upstream-1.4.1-defer-e-streaming-resolution branch July 14, 2026 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant