perf(core): read only the projects in scope from the vector indexes

[phernandez]

Summary

Sixth PR in the #1558 stack, on top of #1568. First half of the "vector order and ANN pre-filter" item: the scope becomes a real pre-filter in both built-in vector adapters, and the pgvector statement actually uses its HNSW index. The second half (filling the window when structured filters reject most of the nearest candidates) follows as its own PR on top of this one.

What was wrong. Both adapters ranked the k nearest vectors across the whole database and applied the scope afterwards. In a shared database (a local vault with many projects, or a tenant with several) a small project could get an under-filled or empty page for a query its own notes answered, because a larger neighbour's vectors filled the window first. This predates the stack; the scoped route just makes it easier to hit.

sqlite-vec. project_id becomes a vec0 partition key. The KNN ranks each partition in scope and the outer ORDER BY merges them. Existing local storage is carried into the partitioned table through a temporary table inside one transaction: vectors and ready manifests survive, nothing is re-embedded. Dimension and source_hash mismatches still rebuild as before.

pgvector. Three things, found while measuring:

• The tie-break sort keys in the old statement kept the planner off the HNSW index entirely. Every Postgres vector query was an exact scan and sort of the whole table. The window is now taken by distance alone inside a materialized CTE and re-sorted with the tie-breaks outside it.
• hnsw.ef_search was never set, so an HNSW scan would have returned at most 40 rows whatever the candidate window asked for. It is now set per transaction to the window the query must fill (capped at the server maximum of 1000).
• hnsw.iterative_scan = relaxed_order lets the scan continue until the scope and manifest filters have admitted enough rows. That needs pgvector 0.8. An older extension raises SemanticDependenciesMissingError at first use, which the API reports as a bad request, instead of quietly returning short windows. Neon tenants run 0.8.0; the test image runs 0.8.5.

Measurements

20k chunks x 384 dims, three projects, project 1 holds 2% of the vectors. Window of 100 rows. Medians of 5 runs.

Engine	Statement	Scope	Rows	Median
sqlite-vec 0.1.9	before	project 1 (2%)	1	3.1 ms
sqlite-vec 0.1.9	after	project 1 (2%)	100	0.2 ms
sqlite-vec 0.1.9	before	projects 1,2	53	3.1 ms
sqlite-vec 0.1.9	after	projects 1,2	100	2.0 ms
pgvector 0.8.5	before (exact scan)	all	100	14.5 ms
pgvector 0.8.5	after, iterative off	projects 1,2	60	3.3 ms
pgvector 0.8.5	after, relaxed_order	projects 1,2	100	5.6 ms
pgvector 0.8.5	after, relaxed_order	all	100	3.8 ms

The "iterative off" row is why pgvector 0.8 is required: with the index in use and a scope covering half the table, an ef_search-sized scan alone under-fills the window. For the 2% scope the planner chose the project btree index and an exact sort on its own, so that case is exact under both statements.

Test plan

• just fast-check
• Adapter tests on SQLite: partition scope fills a small project's window while a neighbour's vectors sit closer; legacy storage is carried over with vectors and readiness intact; pgvector sizes ef_search to the window, always requests the iterative scan, and refuses pgvector < 0.8 before creating storage
• tests/repository tests/services tests/api on SQLite (2175 passed)
• Postgres container: repository, pgvector adapter, vector sync, scoped route, semantic service (132 passed)
• test-int vec0 and semantic coverage (7 passed)
• just doctor

Refs #1558

🤖 Generated with Claude Code

https://claude.ai/code/session_019YW9ysxugGGBCNEGzsxtFV

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[phernandez]

Superseded by #1574: same branch and commit, opened against main because GitHub refuses to retarget a PR that is part of a stack once its base has been merged.