Skip to content

refactor(core): bind vector adapters to the database and search them over a ProjectScope - #1566

Merged
phernandez merged 1 commit into
feat/1558-scoped-search-2b-readerfrom
feat/1558-scoped-search-3a-vector-scope
Sep 15, 2026
Merged

phernandez merged 1 commit into
feat/1558-scoped-search-2b-readerfrom
feat/1558-scoped-search-3a-vector-scope

Conversation

@phernandez

Copy link
Copy Markdown
Member

Part of #1558. Fourth PR in the stack, on top of #1565 (which is on #1563, on #1562). First half of PR 3: the adapters have to read a set of projects before a route can build a SearchReader over one.

What changes

The vector adapters were bound to one project through VectorIndexScope.project_id, so the reader could only ask an adapter for the project it was built for. The project was never part of a vector's identity (entity ids are database-wide primary keys); it is the partition an operation touches.

  • VectorIndexScope is now the database namespace plus the embedding schema (namespace, embedding_identity, dimensions). storage_key is gone.
  • Every write names its project: upsert(project_id, records), delete(project_id, records), delete_entity(project_id, entity_id), delete_orphans(project_id, live_keys).
  • search(query, *, limit, projects: ProjectScope). The built-in adapters bind the scope's ids through ProjectScope.predicate, so sqlite-vec and pgvector answer any set of projects with one statement. An empty scope returns nothing without touching storage.
  • Milvus keeps a collection per project. initialize() has nothing shared to prepare, so each project's collection is created or validated on first use, once per adapter instance. A search over several projects asks each collection for its own top limit and merges by similarity.
  • SemanticSearch passes its scope to the adapter, so a project repository's vector search runs the same statement as before with a scope of one. The repositories build the scope without a project; build_vector_index_scope and create_semantic_vector_index no longer take one.
  • Two unused lookup helpers in the built-in adapters go with the change.

No query behavior changes.

Tests

Adapter doubles implement the new signatures. New cases:

  • pgvector: a two-project scope binds e.project_id IN (:scope_0, :scope_1) and c.project_id IN (:scope_0, :scope_1); an empty scope makes no statement.
  • sqlite-vec: a real-database test seeds two projects' manifests and vectors and checks a two-project scope returns both rows, a single-project scope returns one, and an empty scope returns none.
  • Milvus: a two-project scope searches both collections and merges the top matches; initialize() touches nothing; first use creates or validates the collection; empty operations still touch nothing.
Check Result
just fast-check passed
tests/repository, SQLite 1103 passed, 38 skipped
tests/repository, Postgres (testcontainers) 1099 passed, 41 skipped, plus the Milvus and pgvector adapter files rerun green after a late assertion fix
tests/services, tests/api, tests/mcp, SQLite 2377 passed, 2 skipped
test-int/semantic/test_milvus_lite.py, test-int/test_embedding_status_vec0.py 3 passed
just doctor passed

Trade-off to note

Milvus used to validate the project's collection at startup through initialize(). With no project on the scope, that validation moves to the first operation against each project. A dimension mismatch on an existing collection now surfaces on the first write or search rather than at boot.

Next in the stack

PR 3b adds QUERY /v2/search/ and a scoped service method that builds a SearchReader over a multi-project ProjectScope without a repository.

🤖 Generated with Claude Code

https://claude.ai/code/session_019YW9ysxugGGBCNEGzsxtFV

…over a ProjectScope

The vector adapters were bound to one project through VectorIndexScope.project_id,
which made a multi-project search impossible to express: the reader could only ask
the adapter for the project it was built for. The project was never part of the
vector's identity (entity ids are database-wide primary keys); it is the partition
an operation touches.

VectorIndexScope is now the database namespace plus the embedding schema. Every
write names its project (upsert, delete, delete_entity, delete_orphans take
project_id first) and search takes a ProjectScope, so one sqlite-vec or pgvector
adapter answers any set of projects with one statement through the scope's IN
predicate. An empty scope returns nothing without touching storage. Milvus keeps a
collection per project, validates each collection on its first use instead of in
initialize(), and searches the collections in scope, merging by similarity.

SemanticSearch passes its own scope to the adapter, so a project repository's
vector search is unchanged. The repositories build the scope without a project and
the factory no longer takes one. Two dead lookup helpers in the built-in adapters
go with the change.

Test doubles implement the new signatures; the pgvector, sqlite-vec, and Milvus
suites gain multi-project and empty-scope cases. No query behavior changes.

Part of #1558.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_019YW9ysxugGGBCNEGzsxtFV
Signed-off-by: phernandez <[email protected]>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@phernandez
phernandez added this pull request to stack #1564 September 15, 2026 22:21
@phernandez
phernandez merged commit 2386890 into main Sep 15, 2026
34 checks passed
@phernandez
phernandez deleted the feat/1558-scoped-search-3a-vector-scope branch September 15, 2026 23:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant