fix(core): reuse retrieval for stable rerank pagination

[phernandez]

Why

Fixes #1557. Expanded hybrid/vector pages currently repeat retrieval to reconstruct the fixed reranker prefix. This adds a second query embedding and vector query (and a second FTS pass for hybrid) before the one reranker call.

Stacked on #1559, with base feat/1558-multi-project-search at 0159119c7328729b7ac69be342031e072c073a6d. Neither PR is to be merged as part of this task.

What Changed

• Reconstruct the fixed prefix from the one expanded retrieval, retaining its original chunk membership and matched passages before collapse, filtering, and reranking.
• Preserve a deterministic fused prefix and order new hybrid tail rows by the earliest eligible FTS/chunk source rank. A late vector-only document cannot jump ahead merely because many earlier chunks collapse into one document.
• Keep one reranker call on the fixed prefix, bounded previews/chunks, thresholds, filters, and request tracing. The trace correctly reports stable_pool_refetched=false.
• Verify the feat(core): add explicit multi-project API search #1559 database-scoped reader stays on its separate one-query path, with explicit project scope and no reranker invocation, even when the global reranker setting is enabled.

Implementation Details

The request-local vector window carries prefix rows and original chunk ranks; it is not cached or stored on the repository. Hydration and filter evaluation are shared by prefix and tail.

SQLite vec0's equal-distance top-k membership changes with k, so simply slicing a larger retrieval is insufficient. Reranker-enabled SQLite requests use vec0's existing 4,096-candidate ceiling as a repeatable tie universe, then trim the requested chunk window in SQL before project/manifest filtering. Original positions survive dropped matches. The disabled path retains its existing k sizing. PostgreSQL retrieval SQL is unchanged.

Testing

• Base reproduction: the updated call-count regressions failed with vector limits [18, 8] and hybrid limits [90, 80]; the fixed path uses [18] and [90].
• just fast-check: passed (ruff, formatting, full ty check).
• just doctor: passed.
• Relevant SQLite/Postgres regression suites: SQLite 336 passed; PostgreSQL 314 passed, 22 skipped (backend-specific skips).
• After extracting shared fixtures for standalone CI collection: just fast-check passed, the fixture-dependent SQLite selection passed 51 tests, and standalone test-int/test_stable_rerank_pagination.py passed 2 tests on each of SQLite and Postgres. ty uses pytest's existing tests import root.
• Exact suite commands:

uv run pytest --import-mode=importlib -q tests/repository/test_rerank_pipeline.py tests/repository/test_stable_rerank_pagination.py tests/repository/test_hybrid_fusion.py tests/repository/test_semantic_search_base.py tests/repository/test_search_trace.py tests/repository/test_semantic_vector_index.py tests/repository/test_vector_threshold.py tests/repository/test_vector_pagination.py tests/repository/test_vector_temporal_filter.py tests/repository/test_vector_filter_candidate_restriction.py tests/repository/test_sqlite_vector_search_repository.py tests/repository/test_pgvector_index.py tests/api/v2/test_search_router.py tests/mcp/test_tool_search.py tests/mcp/tools/test_search_notes_multi_project.py tests/mcp/tools/test_search_notes_multi_project_temporal.py test-int/test_stable_rerank_pagination.py test-int/test_multi_project_search.py --cov-report=json:/tmp/1557-coverage.json
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest --import-mode=importlib -q --no-cov tests/repository/test_rerank_pipeline.py tests/repository/test_stable_rerank_pagination.py tests/repository/test_hybrid_fusion.py tests/repository/test_semantic_search_base.py tests/repository/test_search_trace.py tests/repository/test_semantic_vector_index.py tests/repository/test_vector_threshold.py tests/repository/test_vector_pagination.py tests/repository/test_vector_temporal_filter.py tests/repository/test_vector_filter_candidate_restriction.py tests/repository/test_sqlite_vector_search_repository.py tests/repository/test_pgvector_index.py tests/api/v2/test_search_router.py tests/mcp/test_tool_search.py tests/mcp/tools/test_search_notes_multi_project.py tests/mcp/tools/test_search_notes_multi_project_temporal.py test-int/test_stable_rerank_pagination.py test-int/test_multi_project_search.py
uv run pytest --import-mode=importlib -q --no-cov test-int/semantic/test_semantic_coverage.py -k test_postgres_hybrid_preserves_candidate_windows

The real FastEmbed/Postgres test passed, including a 101-result request and deep pages. New regressions exercise real sqlite-vec/pgvector, API/MCP probes and later empty pages, tied similarities, mixed lexical/vector matches, multi-chunk collapse, fixed reranker document text, pending manifests, thresholds, filters, and reranking enabled/disabled. Stage-span assertions require exactly one embedding/vector span per request and one FTS span for hybrid. All 55 added executable production lines were covered by the selected SQLite run.

Risks / Follow-ups

SQLite reranker-enabled shallow requests do more bounded vec0 selection work to make ties repeatable. A local synthetic microbenchmark (384 dimensions, seed 1557, five warm iterations) compared the old two k queries (890 + 80) with one fixed-4096 query trimmed to 890: 10k vectors, 9.39 ms versus 9.98 ms; 50k vectors, 45.79 ms versus 45.48 ms. This isolates retrieval and is not a production latency or model-inference benchmark. A substantially slower scalar-distance scan was rejected.

Existing best-effort chunk headroom and backend candidate ceilings remain. No Cloud configuration, production reranker enablement, cache, locking, or admission system changes.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-15T06:39:33.792268Z	2e1b9a0	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[phernandez]

@codex review

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[phernandez]

Closing as superseded, together with its base #1559. The retrieval path this PR changes is being extracted into a scope-parameterized SearchReader in the #1558 stack (starting at #1562). The #1557 fix, one expanded retrieval reused for the fixed reranker prefix instead of a second embedding and vector query, will be reimplemented on that reader once it exists, so the fix lands in one place for the project route, the database-scoped route, and MCP.

What carries over from here when that happens: the sqlite-vec repeatable tie universe for reranker-enabled requests, the deterministic tail ordering by earliest eligible source rank, tests/semantic_search_helpers.py, and the standalone stable-pagination suites. #1557 stays open until then.