perf(core): read only the projects in scope from the vector indexes

CHANGELOG.md

@@ -4,6 +4,38 @@
 
 ### Features
 
+- **#1558**: `QUERY /v2/search/` (and `POST /v2/search/` for clients that cannot send
+  QUERY) searches an explicit set of projects in one database with one query. The body
+  is the project search body plus `project_ids`, a required list of internal ids the
+  caller has already authorized; an empty list answers no rows and there is no way to
+  ask for every project. Full-text, vector, and hybrid retrieval run the same reader
+  the project route runs, so one project here ranks exactly as its own route does, and
+  results are one ranking over the union rather than merged per-project pages. Hits are
+  hydrated only from projects in scope. Every search result, on both routes, now
+  carries `project_id` and `project_external_id`.
+
+- **#1558**: `search_notes(search_all_projects=True)` runs one scoped query per database
+  instead of one search per project, so a local vault with many projects is one
+  query, and each cloud workspace is one query, with results attributed to their
+  project by the server rather than by which request they came back on. A new
+  `projects` parameter searches a chosen subset by name or external id; an unknown
+  name is an error. Cross-database results are still merged by score, one failing
+  database is skipped with a warning and an inexact total, and a retryable outage
+  or a server too old to attribute its results fails the whole page.
+
+- **#1558**: Vector retrieval reads only the projects in scope and fills the window it
+  asks for. The sqlite-vec table gains a `project_id` partition key, so a scoped
+  nearest-neighbour query ranks each project's own vectors instead of taking the k
+  nearest across the whole database and discarding the out-of-scope ones, which
+  could leave a small project with an empty page for a query its notes answered.
+  Existing local storage is carried into the partitioned table without re-embedding.
+  On Postgres the nearest-neighbour statement now runs on the HNSW index (its
+  tie-break sort keys had kept the planner on an exact scan of every vector), with
+  `hnsw.ef_search` sized to the candidate window and an iterative scan that keeps
+  going until the scope and manifest filters have admitted enough rows. That scan
+  needs pgvector 0.8 or later; an older extension is reported as a dependency error
+  instead of quietly returning short windows.
+
 - **#1512**: Word, PowerPoint, and CSV files get the same sidecar Markdown note a
   PDF gets. `bm import document <path>` indexes the project, extracts the file,
   and writes `<file>.<ext>.md` next to it plus a run note under
@@ -109,6 +141,49 @@
   Frontmatter is now classified once, by the parser, as present, absent, or
   malformed, and only the first two are ever written to.
 
+### Internal
+
+- **#1558**: Search filter compilation now runs over an explicit `ProjectScope` instead of
+  a repository-bound `project_id`. FTS term preparation and filter compilation moved out
+  of the SQLite and Postgres repositories into `sqlite_search_query` and
+  `postgres_search_query` as pure functions returning a `CompiledFilter`, and the filters
+  both backends share (scope, permalink, directory, item type, category, note type,
+  `after_date`, valid time, candidate keys) are compiled once in `search_filters`. The
+  note-type and valid-time predicates match search rows on their full
+  `(project_id, ...)` identity. Project repositories call the compilers with a scope of
+  one; no query behavior changes. First step of the shared single/multi-project reader.
+
+- **#1558**: Full-text execution leaves the project repositories. `SQLiteFts` and
+  `PostgresFts` run a compiled statement for any `ProjectScope` and own their engine's
+  failure semantics (FTS5 syntax errors answer empty, Postgres retries a malformed strict
+  tsquery relaxed inside a savepoint). `SearchRepositoryBase.search` and `count` are
+  concrete: shared vector/hybrid dispatch, then the engine's `FtsBackend`. The base read
+  path binds every statement to the repository's scope (manifest hydration, candidate row
+  fetch, readiness and drop classification). `PreparedSearchQuery` moves to the repository
+  layer with defaults, and the filter helpers both backends share move from the base into
+  `search_filters`. No query behavior changes.
+
+- **#1558**: Retrieval leaves `SearchRepositoryBase`. `SearchReader` runs one prepared
+  query over one `ProjectScope` in whichever mode it asks for, and `SemanticSearch` owns
+  vector and hybrid retrieval (adapter lookup, manifest hydration, the structured filter
+  pass, score fusion, reranking, pagination) over a `VectorRetrieval` that is present or
+  absent rather than probed with `hasattr`. The repository keeps what only it knows
+  (whether semantic search is enabled and its vector tables exist) and builds a reader
+  per call from its current state. Hydrated chunks are a typed `HydratedChunk`, which
+  retires the `best_distance` compatibility branch and the per-backend
+  `_distance_to_similarity` hooks the adapters had already replaced. Test doubles
+  construct the pipeline directly instead of subclassing the repository. No query
+  behavior changes.
+
+- **#1558**: Vector adapters are bound to the database, not to a project. `VectorIndexScope`
+  is the database namespace plus embedding schema; every write names the project it
+  touches (`upsert(project_id, ...)`, `delete(project_id, ...)`,
+  `delete_entity(project_id, ...)`, `delete_orphans(project_id, ...)`) and `search` takes
+  a `ProjectScope`, so one sqlite-vec or pgvector adapter answers a query across any set
+  of projects with one statement. Milvus keeps a collection per project and searches the
+  collections in scope. `SemanticSearch` passes its scope through, so a project
+  repository's vector search is unchanged. No query behavior changes.
+
 
 ## v0.23.2 (2026-08-25)
 

src/basic_memory/api/app.py

@@ -14,6 +14,7 @@
     knowledge_router as v2_knowledge,
     project_router as v2_project,
     memory_router as v2_memory,
+    scoped_search_router as v2_scoped_search,
     search_router as v2_search,
     resource_router as v2_resource,
     directory_router as v2_directory,
@@ -136,6 +137,8 @@ async def workspace_permalink_context_middleware(request: Request, call_next):
 app.include_router(v2_schema, prefix="/v2/projects/{project_id}")
 app.include_router(v2_inspect, prefix="/v2/projects/{project_id}")
 app.include_router(v2_project, prefix="/v2")
+# Database-scoped search: one query over an explicit set of projects.
+app.include_router(v2_scoped_search, prefix="/v2")
 
 # Legacy web app proxy paths (compat with /proxy/projects/projects)
 app.include_router(v2_project, prefix="/proxy/projects")

src/basic_memory/api/v2/routers/__init__.py

@@ -6,6 +6,7 @@
 from basic_memory.api.v2.routers.knowledge_router import router as knowledge_router
 from basic_memory.api.v2.routers.project_router import router as project_router
 from basic_memory.api.v2.routers.memory_router import router as memory_router
+from basic_memory.api.v2.routers.scoped_search_router import router as scoped_search_router
 from basic_memory.api.v2.routers.search_router import router as search_router
 from basic_memory.api.v2.routers.resource_router import router as resource_router
 from basic_memory.api.v2.routers.directory_router import router as directory_router
@@ -22,6 +23,7 @@
     "knowledge_router",
     "project_router",
     "memory_router",
+    "scoped_search_router",
     "search_router",
     "resource_router",
     "directory_router",

src/basic_memory/api/v2/routers/scoped_search_router.py

@@ -0,0 +1,114 @@
+"""Database-scoped search: one query over an explicit set of projects.
+
+The caller names the projects. Nothing here discovers a scope, and an empty set
+answers no rows; Cloud resolves authorization first and passes the effective ids.
+The route shares the project route's reader, so one project here ranks exactly as
+that project's own route does, and several projects are one ranking over the union
+rather than per-project pages merged afterwards.
+"""
+
+import asyncio
+
+import logfire
+from fastapi import APIRouter, Query, Response
+
+from basic_memory import db
+from basic_memory.api.v2.utils import (
+    load_temporal_metadata,
+    search_error_boundary,
+    to_search_results,
+)
+from basic_memory.deps import AppConfigDep, SessionMakerDep
+from basic_memory.repository.search_repository import create_search_reader
+from basic_memory.repository.search_scope import ProjectScope
+from basic_memory.schemas.search import ScopedSearchQuery, SearchResponse, SearchRetrievalMode
+from basic_memory.services.scoped_search_service import ScopedSearchService
+
+# App registration mounts this router at /v2.
+router = APIRouter(tags=["search"])
+
+
+@router.api_route(
+    "/search/",
+    methods=["QUERY"],
+    response_model=SearchResponse,
+    include_in_schema=False,
+)
+@router.post("/search/", response_model=SearchResponse)
+async def search_scope(
+    query: ScopedSearchQuery,
+    app_config: AppConfigDep,
+    session_maker: SessionMakerDep,
+    response: Response,
+    page: int = Query(1, ge=1),
+    page_size: int = Query(10, ge=1, le=1000),
+) -> SearchResponse:
+    """Search an explicit set of projects in this database.
+
+    Results are one ranking across the whole scope and carry ``project_id`` and
+    ``project_external_id``. Hydration reads only from projects in scope.
+    """
+    response.headers["Accept-Query"] = "application/json"
+    # The read cache is keyed by one project's generation. A set of projects has no
+    # single generation to invalidate on, so this route is never cached.
+    response.headers["Cache-Control"] = "no-store"
+
+    scope = ProjectScope.of(query.project_ids)
+    service = ScopedSearchService(
+        session_maker, scope, create_search_reader(session_maker, scope, app_config)
+    )
+    temporal_requested = query.has_temporal_filter()
+    exact_count_available = query.retrieval_mode == SearchRetrievalMode.FTS
+    offset = (page - 1) * page_size
+
+    with logfire.span(
+        "api.request.search_scope",
+        entrypoint="api",
+        domain="search",
+        action="search_scope",
+        project_count=len(scope.project_ids),
+        page=page,
+        page_size=page_size,
+        retrieval_mode=query.retrieval_mode.value,
+        has_temporal_filter=temporal_requested,
+    ):
+        with search_error_boundary():
+            if exact_count_available:
+                results, total = await asyncio.gather(
+                    service.search(query, limit=page_size, offset=offset),
+                    service.count(query),
+                )
+                has_more = offset + len(results) < total
+            else:
+                # Trigger: semantic modes would need another vector or hybrid pass to count.
+                # Why: a search should not pay for a second semantic retrieval.
+                # Outcome: probe one row past the page, leave total at 0, mark it inexact.
+                results = await service.search(query, limit=page_size + 1, offset=offset)
+                total = 0
+                has_more = len(results) > page_size
+                results = results[:page_size]
+
+        temporal_by_source = {}
+        project_external_ids: dict[int, str] = {}
+        if results:
+            async with db.scoped_session(session_maker) as session:
+                project_external_ids = await service.project_external_ids(session, results)
+                if temporal_requested:
+                    temporal_by_source = await load_temporal_metadata(service, session, results)
+        search_results = await to_search_results(
+            service,
+            results,
+            temporal_by_source=temporal_by_source,
+            project_external_ids=project_external_ids,
+        )
+        return SearchResponse(
+            results=search_results,
+            current_page=page,
+            page_size=page_size,
+            total=total,
+            total_is_exact=exact_count_available,
+            has_more=has_more,
+            # None, not False, when nothing was asked: an ordinary search payload stays
+            # exactly what it was before valid time existed.
+            temporal_applied=True if temporal_requested else None,
+        )

src/basic_memory/api/v2/routers/search_router.py

@@ -9,11 +9,15 @@
 from contextlib import nullcontext
 from typing import Annotated
 
-from fastapi import APIRouter, Depends, HTTPException, Path, Response
+from fastapi import APIRouter, Depends, Path, Response
 
 import logfire
 from basic_memory import db
-from basic_memory.api.v2.utils import load_temporal_metadata, to_search_results
+from basic_memory.api.v2.utils import (
+    load_temporal_metadata,
+    search_error_boundary,
+    to_search_results,
+)
 from basic_memory.deps import (
     EntityServiceV2ExternalDep,
     MemoryTimeIndexRepositoryV2ExternalDep,
@@ -32,12 +36,6 @@
     read_cache_request_digest,
 )
 from basic_memory.read_cache.policy import SEARCH_READ_CACHE_TTL_SECONDS
-from basic_memory.repository.semantic_errors import (
-    RerankProviderContractError,
-    RerankTransientError,
-    SemanticDependenciesMissingError,
-    SemanticSearchDisabledError,
-)
 from basic_memory.schemas.search import SearchQuery, SearchResponse, SearchRetrievalMode
 from basic_memory.services.search_guidance import unspaced_script_query_hint
 
@@ -91,6 +89,7 @@ async def search(
     session_maker: SessionMakerDep,
     read_cache: SearchReadCacheDep,
     response: Response,
+    internal_project_id: ProjectExternalIdPathDep,
     project_id: str = Path(..., description="Project external UUID"),
     page: int = 1,
     page_size: int = 10,
@@ -157,7 +156,7 @@ async def search(
 
             offset = (page - 1) * page_size
             exact_count_available = query.retrieval_mode == SearchRetrievalMode.FTS
-            try:
+            with search_error_boundary():
                 with logfire.span(
                     "api.search.search.execute_query",
                     domain="search",
@@ -176,21 +175,6 @@ async def search(
                             query, limit=page_size + 1, offset=offset
                         )
                         total = 0
-            except SemanticSearchDisabledError as exc:
-                raise HTTPException(status_code=400, detail=str(exc)) from exc
-            except SemanticDependenciesMissingError as exc:
-                raise HTTPException(status_code=400, detail=str(exc)) from exc
-            except RerankTransientError as exc:
-                # Returning raw retrieval order would make pagination inconsistent with
-                # earlier reranked pages. Preserve ordering semantics and make the outage
-                # explicitly retryable instead.
-                raise HTTPException(status_code=503, detail=str(exc)) from exc
-            except RerankProviderContractError as exc:
-                # Upstream reranker returned a malformed response — an upstream fault, not a
-                # client error and not a transient outage (those map to a retryable 503).
-                raise HTTPException(status_code=502, detail=str(exc)) from exc
-            except ValueError as exc:
-                raise HTTPException(status_code=400, detail=str(exc)) from exc
 
             with logfire.span(
                 "api.search.search.paginate_results",
@@ -228,7 +212,10 @@ async def search(
                             temporal_repository, session, results
                         )
                 search_results = await to_search_results(
-                    entity_service, results, temporal_by_source=temporal_by_source
+                    entity_service,
+                    results,
+                    temporal_by_source=temporal_by_source,
+                    project_external_ids={internal_project_id: project_id},
                 )
             with logfire.span(
                 "api.search.search.build_response",

src/basic_memory/api/v2/utils.py

@@ -1,11 +1,19 @@
 from collections import defaultdict
-from collections.abc import Mapping
+from collections.abc import Iterator, Mapping
+from contextlib import contextmanager
 from typing import Any, Protocol, Optional, List, Sequence
 
 import logfire
+from fastapi import HTTPException
 from sqlalchemy.ext.asyncio import AsyncSession
 from basic_memory.models import MemoryTimeIndex
 from basic_memory.repository.search_repository import SearchIndexRow
+from basic_memory.repository.semantic_errors import (
+    RerankProviderContractError,
+    RerankTransientError,
+    SemanticDependenciesMissingError,
+    SemanticSearchDisabledError,
+)
 from basic_memory.schemas.memory import (
     EntitySummary,
     ObservationSummary,
@@ -54,6 +62,28 @@ async def find_for_sources(
 type TemporalMetadataBySource = Mapping[tuple[str, int], list[TemporalResultMetadata]]
 
 
+@contextmanager
+def search_error_boundary() -> Iterator[None]:
+    """Map search failures onto HTTP statuses the same way on every search route."""
+    try:
+        yield
+    except SemanticSearchDisabledError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+    except SemanticDependenciesMissingError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+    except RerankTransientError as exc:
+        # Returning raw retrieval order would make pagination inconsistent with
+        # earlier reranked pages. Preserve ordering semantics and make the outage
+        # explicitly retryable instead.
+        raise HTTPException(status_code=503, detail=str(exc)) from exc
+    except RerankProviderContractError as exc:
+        # Upstream reranker returned a malformed response: an upstream fault, not a
+        # client error and not a transient outage (those map to a retryable 503).
+        raise HTTPException(status_code=502, detail=str(exc)) from exc
+    except ValueError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+
 async def get_entities_by_id_lookup(
     entity_service: EntityServiceBatchLookup,
     entity_ids: Sequence[int],
@@ -304,7 +334,13 @@ async def to_search_results(
     results: List[SearchIndexRow],
     *,
     temporal_by_source: TemporalMetadataBySource | None = None,
+    project_external_ids: Mapping[int, str] | None = None,
 ) -> list[SearchResult]:
+    """Shape one page of search rows into the public result.
+
+    ``project_external_ids`` maps each row's project to its external id; a caller
+    that knows the projects passes it and results carry both identities.
+    """
     with logfire.span(
         "search.hydrate_results",
         domain="search",
@@ -391,6 +427,12 @@ async def to_search_results(
                             if temporal_by_source
                             else None
                         ),
+                        project_id=result.project_id,
+                        project_external_id=(
+                            project_external_ids.get(result.project_id)
+                            if project_external_ids
+                            else None
+                        ),
                     )
                 )
         return search_results