Repository navigation
Search answers a service with each hit's score, ranks and source, and pages - #1273
Conversation
|
Moving this to draft: CONTRIBUTING asks for a change to the search contract to be agreed in its issue first, and #1271 has no agreed design yet. I'll ask there, and rework this to whatever we settle on before marking it ready. Sorry for jumping ahead. |
3611670 to
f0b73cb
Compare
|
Thanks @WaylandYang for agreeing the design on #1271. I've rebased this onto dev, after #1266, and it covers each point:
Filters will come in the next PR. |
WaylandYang
left a comment
There was a problem hiding this comment.
Thanks, this matches what we agreed on #1271, and the paging test is exactly right. A few small things before merge: (1) on the page that ends at depth 100, has_more is true but the next page is 422; please return false at the cap (or document it) with a test. (2) Web search now fuses 101 per channel, so its ranking can differ from before and from chat; please say so in the PR body and chat-and-mcp.md. (3) Note that offset pages are consistent only while the base doesn't change, and that ranks are counted before the as_of filter. (4) A short section for integrators in web/src/docs/ingest.md, next to the read-back by external_id. (5) In 0014, date the search entry for when it lands and add #1271 to the status line. Optional: keep external_id derivation next to pushed_key, and reject unknown request fields until filters land.
f0b73cb to
08c810c
Compare
|
Thanks for the careful review, all fixed in the amended commit:
Of the optional ones: unknown request fields are now a 422 ( |
WaylandYang
left a comment
There was a problem hiding this comment.
Thanks, all five points check out, and the cap test is exactly what I wanted. One last nit: ingest.md says any field other than q/top_k/offset is a 422, but as_of is accepted. Please list it there with the note that ranks are counted before as_of. Moving the external_id derivation next to pushed_key can go in the filters PR.
… pages Signed-off-by: Bogdan Szabo <[email protected]>
08c810c to
fee7b51
Compare
|
Thanks! |
WaylandYang
left a comment
There was a problem hiding this comment.
Thanks, ingest.md now lists as_of with the note that ranks are counted before it filters, and nothing else changed. Approving.
PR 1 of #1271. Filters follow in a separate PR.
POST /kbs/{id}/searchnow accepts a read token on the same terms as the other reads in ADR 0014 decision 7: Viewer is the ceiling, and a base outsidekb_idsreturns 404.Each hit includes its RRF score, its 1-based BM25 and vector ranks (null when the hit did not come from that channel) and its source. A hit from a pushed source also carries its
external_id. The response reportsvector_used, which is false when the vector channel was skipped.offsetpages through one fused list. Every page asks each channel for the same 100 candidates, so pages never overlap or skip hits.offset + top_kabove 100 returns 422, and the page that reaches 100 answershas_more: false. A score ranks hits within one query only. Until filters land, an unknown request field returns 422 instead of being ignored.Two things change for callers. The search page now fuses 100 candidates per channel where it used to fuse 24, as chat still does, so the same query can rank differently than before and differently from chat. And paging has limits: pages join up only while the base does not change between requests, and each channel counts its ranks before the final
as_offilter, so a historical page can skip rank numbers.two_pages_of_fifteen_join_into_the_first_thirty_hitschecks that two pages of 15 equal the first 30 hits; it fails on the old code.a_page_ending_at_depth_100_of_101_hits_has_no_morecovers the cap. The token-read tests now cover search, including both 422s. ADR 0014, access-and-audit and chat-and-mcp describe the route, andweb/src/docs/ingest.mdhas a section for integrators next to the read-back byexternal_id.