Repository navigation
Search cannot serve a backend: no token, no scores, no paging, no filter #1271
Description
We want a backend to search a base the way the chat does: hybrid full-text and vector retrieval, fused by rank. POST /kbs/{id}/search is close, but four things stop a service from using it:
- It takes a session JWT only; a personal token gets 401.
- It returns chunks without scores or ranks, so a caller cannot blend the results with its own or tell a strong hit from a weak one, and it cannot tell when the vector arm was skipped (no embedding model, or the call failed) because the fallback to full-text only is silent.
- There is no paging:
top_kand nothing else. - It searches the whole base. A base that holds several kinds of documents (code, docs, a memory log), each in its own source, cannot be searched one kind at a time.
Proposal, as small PRs:
- scores:
rrf_fusekeeps the fused score and each arm's rank; the route returns them withsource_id,external_id,has_moreandvector_used, takesoffset, capstop_kat 50, and accepts a token (read scope,kb_idsnarrowing, as document content does) - filters:
source_ids,filename_prefix,external_id_prefix,exclude_missing, pushed into both arms through the document ids they allow (the full-text index already storesdocument_id, so no reindex) - the same filters and scores on MCP
search_chunks
Activity
gedaiu commentedon Oct 16, 2026
Related: #1268 keeps an
apipush'sexternal_idas the document's identity, which theexternal_idfilter here relies on. Token reads of the graph are #1262; same backend use as #1267 and #1269.gedaiu commentedon Oct 16, 2026
Related: #1274 lets a write token create the
apisource these documents are pushed to; #1273 is the first part of this issue.gedaiu commentedon Oct 18, 2026
You mentioned on #1277 that this one comes first, so before going further: could we agree the shape here? What I'd propose is what draft #1273 does. A token reads search like a session does. Each hit carries its score, its BM25 and vector ranks, and its source. The request can filter by
source_idandexternal_idand pages byoffset. The response says whether the vector side ran (vector_used). If you'd rather cut it differently, for example scores and ranks first and filters in a second PR, I'll rework #1273 to match. Thanks.WaylandYang commentedon Oct 19, 2026
Thank you for narrowing this down. Yes, let's do it, in two PRs as you offered:
kb_idsnarrowing), and each hit carries its score, its BM25 and vector ranks, and its source; the response says whether the vector side ran. Paging has to slice one fused list: please give every page the same candidate count per channel (e.g.MAX_DEPTH + 1), since a deeper page currently fuses longer lists and pages can overlap or skip hits. A paging test on the two-channel fixture inretrieval_tests.rsshould fail on today's code. Fold the route into 0014's decision 7 with the other token reads.source_id,external_id) in a second PR.MCP
search_chunkscan follow later, once the route has settled. #1273 can be reworked into the first PR when you're ready.WaylandYang commentedon Oct 22, 2026
Design agreed on 10-19: the first PR is token access, per-hit score and ranks,
vector_used, and paging over one fused list with a fixed candidate count per channel; filters second; MCP after. #1273 is being reworked to that shape. Nothing on dev yet.