Avoid duplicate hybrid/vector retrieval for stable reranker pagination

[phernandez]

Problem

With reranking enabled, a hybrid request whose result window exceeds the fixed reranking prefix performs full-text and vector retrieval twice. The second pass reconstructs a stable candidate pool for pagination. Reranking itself runs once.

Production Cloud 37d681c (Core fa30662) reproduced this with search_notes(query="tenant routing search architecture and memory usage", search_type="hybrid", search_all_projects=true, page_size=25) across 17 projects.

Evidence

Logfire project basic-memory-cloud, production trace 01a0a2f7674171c89b468c1eb6970d76, 2026-09-15 02:48:37–02:49:37 UTC:

• Server request 59.108s; project resolution 6.810s, followed by overlapping project searches through 58.601s.
• 34 FTS spans: 53.884s cumulative, 10.298s maximum.
• 34 query embedding spans: 24.863s cumulative, 7.328s maximum.
• 34 vector-query spans: 23.511s cumulative, 4.949s maximum.
• 17 rerank spans: 216.825s cumulative, 20.591s maximum.

Cumulative durations include concurrent work and must not be added as request wall time. Reranking is the largest cost; duplicate retrieval is an additional independently observed inefficiency. The repeat hit all 17 caches and took 10.762s client-side. This is not a controlled benchmark.

Code path

In src/basic_memory/repository/search_repository_base.py, _candidate_limit() expands the retrieval window beyond _rerank_candidate_limit() when the requested window exceeds _reranker_candidates. _search_hybrid() then recursively calls itself with the fixed candidate limit and _apply_rerank=False. This repeats FTS, query embedding and vector retrieval before the single _rerank_and_paginate() call. Inspect the analogous vector stable-pool path too.

The fixed pool protects pagination: expanding the retrieval universe can alter fused scores and must not reorder a previously returned prefix. Removing the recursion without preserving that invariant is not sufficient.

Desired outcome / acceptance

• Avoid duplicate retrieval work for expanded hybrid/vector windows while preserving fixed reranked-prefix membership, deterministic tail ordering and page completeness.
• Reuse query embeddings where the query/model are identical within the request.
• Cover first page, larger page sizes, deeper pages, chunk-to-document collapse, mixed FTS/vector matches and reranking disabled.
• Regression tests must demonstrate the duplicate work before the fix and fail if stable pagination breaks. Exercise real SQLite and Postgres retrieval and API/MCP pagination boundaries.
• Use stage spans to compare retrieval call counts and latency; distinguish inference execution from queue waiting if tuning reranker performance separately.
• No cross-worker semaphores or admission-limit system in this issue.

Cloud is disabling production reranking as an operational mitigation. Existing bounded previews and ONNX batch/arena tuning remain in place. No implementation is included in this issue.

[claude]

Claude finished @phernandez's task in 1m 12s —— View job

---

Triage

Type: Bug — duplicate FTS/embedding/vector retrieval when hybrid search with reranking expands the candidate window beyond _reranker_candidates (_search_hybrid() recursion in src/basic_memory/repository/search_repository_base.py); reranking itself already runs once.

Component: Core search/retrieval (repository/search_repository_base.py), not a Cloud-only code path — the defect is in shared retrieval logic used by local and cloud alike. Impact was observed and is currently most severe at Cloud scale (search_all_projects across many projects), where Cloud has applied an operational mitigation (disabling production reranking) while this is fixed.

Priority: High. Not a full outage, but it doubles retrieval cost (FTS + query embedding + vector) on any hybrid+rerank request whose page window exceeds the fixed rerank prefix, and has already forced disabling reranking in production as a stopgap.

Complexity: Complex. A correct fix must eliminate the redundant retrieval pass while strictly preserving the stable-pagination invariants (fixed reranked-prefix membership, deterministic tail ordering, page completeness) across both the hybrid and analogous vector stable-pool paths, reuse query embeddings within a request, and be validated with regression tests across SQLite and Postgres and at the API/MCP pagination boundary — plus tests that demonstrably show the duplicate work exists before the fix.

No duplicate issue found in the repo (searched for related terms around reranker performance, candidate limits, and hybrid pagination).

Labels applied: bug (component: none — this is core, not Cloud-infrastructure-specific).

[phernandez]

PR #1560 is closed as superseded. The fix moves onto the scope-parameterized SearchReader from the #1558 stack (#1562 first), so one retrieval feeds the fixed reranker prefix on every route. Tracking here until that PR opens.