diff --git a/README.md b/README.md index 9d4af90..3ae630b 100644 --- a/README.md +++ b/README.md @@ -283,7 +283,7 @@ Tests live under `tests/`: | Tool | Description | |-------------------------|------------------------------------------------------------------------------------------------------| | `search_code` | Hybrid (dense + BM25) search by query, with optional filters for language, service, symbol type | -| `find_symbol` | Look up a symbol by name — exact match, or case-insensitive substring when `exact=false` | +| `find_symbol` | Look up a symbol by name — exact match, or case-insensitive token match when `exact=false` | | `find_usages` | Find code that references a given symbol name (semantic search, then excludes the definition itself) | | `get_code_context` | Fetch the full source of a file — or a specific symbol within it — directly from GitHub | | `reindex` | Trigger code indexing of one or all services (incremental by default; `force` to re-embed) | @@ -293,6 +293,13 @@ Tests live under `tests/`: | `list_indexed_services` | List indexed services with chunk and file counts, languages, and last-indexed time | | `index_stats` | Show Qdrant collection statistics and configured services | +`find_symbol(exact=false)` matches against a full-text index over the symbol name's camelCase/snake_case tokens, so +`order` or `ord` finds `placeOrderRequest` in ~2 ms regardless of collection size. Mid-token fragments (`rder`) still +match, but fall back to a client-side scan that is linear in collection size. Collections indexed before this field +existed use that same fallback until reindexed — and because change detection skips unchanged files, populating the +field needs a **force** reindex (`POST /reindex {"force": true}`), which re-embeds every symbol. See +[docs/retrieval-rrf.md](docs/retrieval-rrf.md#name-lookup-find_by_name). + ## MCP Prompts | Prompt | Arguments | Description | diff --git a/docs/retrieval-rrf.md b/docs/retrieval-rrf.md index ef0a9d2..92b0dbc 100644 --- a/docs/retrieval-rrf.md +++ b/docs/retrieval-rrf.md @@ -79,17 +79,39 @@ Queries Qdrant with a keyword filter on the `symbol_name` payload field: FieldCondition(key="symbol_name", match=MatchValue(value=name)) ``` -Returns up to 20 exact matches via a scroll operation. Additional filters for `symbol_type` and `service` are stacked into the same `must` list. No vectors are fetched. +Returns up to 20 exact matches via a scroll operation. `symbol_name` carries a `KEYWORD` payload index, so this filter is served by Qdrant. Additional filters for `symbol_type` and `service` are stacked into the same `must` list. No vectors are fetched. -### Substring mode (`exact=False`, default) +### Partial mode (`exact=False`, default) -Qdrant has no native text-contains index for partial name matching. The implementation falls back to a **client-side substring scan**: +Matching is **token-aware and case-insensitive**, served server-side by a full-text payload index. -1. Scroll the collection in batches of 200 points -2. For each point, check whether `name.lower()` appears in `payload['symbol_name'].lower()` -3. Collect up to 50 matches, then stop +At index time, `_symbol_to_payload()` stores a derived `symbol_name_tokens` field built by `symbol_name_tokens()`. It contains three things: the original identifier, its camelCase/PascalCase/snake_case subwords (via the same `split_code_identifiers()` helper that feeds BM25), and *suffix joins* — the identifier re-joined from each subword boundary onwards. That field carries a `TEXT` index with the `PREFIX` tokenizer (`lowercase=True`, token length 2–30), and `find_by_name` queries it with a single `MatchText`-filtered scroll returning up to 50 matches. -This is **O(N)** in collection size — it scans every indexed symbol in the collection (or service subset, if filtered). On large codebases with hundreds of thousands of symbols, this can be slow. +**Why suffix joins.** A `PREFIX` index only matches a query that prefixes a stored token. Storing just the identifier and its subwords means `GetWebAuthnSession` is reachable by `GetWeb` (prefix of the whole name) and by `Authn` (a subword), but *not* by `WebAuth` — that query spans `Web` + `Authn` and is not anchored at the start of the name. The old client-side substring scan found it, so without the joins this would be a silent recall regression: `WebAuth` returned 12 of 18 real matches on a live collection. Emitting `WebAuthnSession`, `AuthnSession`, `Session` as their own tokens makes every subword-boundary query a real index hit. Joins are capped at the first `MAX_SUFFIX_JOIN_SUBWORDS` (8) subwords and skipped entirely for names containing whitespace — markdown headings, CSS selector lists and dependency coordinates are prose, whose words are already separate tokens. + +Matching a query against `placeOrderRequest`, measured against a real Qdrant: + +| Query | Matches | Latency | Why | +| --- | --- | --- | --- | +| `order` | ✅ | ~2 ms | full subword token — indexed lookup | +| `ord`, `plac`, `reques` | ✅ | ~2 ms | `PREFIX` tokenizer indexes every token prefix | +| `orderRequest` | ✅ | ~2 ms | suffix join — indexed lookup | +| `place order` | ✅ | — | `MatchText` requires all query tokens to match | +| `rder`, `quest` | ✅ | O(N) | mid-token: no index hit, served by the client-side fallback (see below) | + +Results are then ranked exact name → prefix → remainder, because Qdrant returns scrolled points in point-id order and would otherwise bury the exact hit. + +**Mid-token queries are not served by the index, and cost O(N) client-side.** A fragment that is not a prefix of any stored token (`rder` inside `placeOrderRequest`) produces **zero** rows from Qdrant — the `PREFIX` tokenizer indexes token prefixes only, and Qdrant does not silently scan on your behalf. Such queries land in `_find_by_name_scanning()` below. Qdrant offers no n-gram tokenizer, so arbitrary-substring matching cannot be made sublinear. What issue [#72](https://github.com/GoodbyePlanet/semcode/issues/72) removed is the client-side scan for *token and subword-boundary* queries, which are the overwhelming majority. + +**Fallback.** When the full-text filter returns zero results, `_find_by_name_scanning()` runs the pre-#72 behaviour — scroll in batches of 200 and substring-match `symbol_name` in Python. Two distinct cases reach it, indistinguishable from each other: a mid-token fragment, and a collection indexed before `symbol_name_tokens` existed (where `MatchText` on the absent field matches nothing). A genuinely unmatched query such as `zzz` also triggers it, and is the worst case — it scans the entire collection to return nothing. Measured cost of that scan: 0.9 s at 50k symbols, 8.8 s at 250k, linear thereafter. + +**Migrating an existing collection.** The payload indexes are created on every startup, including for collections that predate them, so no collection drop is needed. But populating `symbol_name_tokens` requires the points to be rewritten, and incremental indexing compares Git blob SHAs and skips unchanged files — a plain `make index-code` on an unchanged repo reports `{"files": 0, "chunks": 0, "skipped": N}` and leaves the field empty. Use a **force** reindex: + +```bash +curl -X POST http://localhost:8090/reindex -H 'Content-Type: application/json' -d '{"force": true}' +``` + +This re-embeds every symbol, so it costs a full pass of embedding-provider calls (709 symbols took ~4 minutes on a hosted provider). Until it runs, partial lookups keep working via the fallback scan. --- @@ -117,7 +139,7 @@ Each result includes: symbol name and type, RRF score, file location (path + lin find_symbol(name: str, symbol_type: str | None, service: str | None, chunk_tier: str | None, exact: bool = False) -> str ``` -Name-based lookup via `store.find_by_name()`. Does not use vectors or RRF. Supports filtering by `chunk_tier` (`"method"` or `"class"`) in addition to `symbol_type` and `service`. Returns up to 20 (exact) or 50 (substring) matches. Each result includes: name, type, location, package, parent class, and source (first 800 characters). +Name-based lookup via `store.find_by_name()`. Does not use vectors or RRF. Supports filtering by `chunk_tier` (`"method"` or `"class"`) in addition to `symbol_type` and `service`. Returns up to 20 (exact) or 50 (partial) matches, exact names first. Each result includes: name, type, location, package, parent class, and source (first 800 characters). ### `find_usages` @@ -174,7 +196,9 @@ public OrderResult processOrder(OrderRequest request) { **RRF constant is not configurable** — Qdrant's `k=60` default is used. There is no way to adjust this via configuration. The choice of `k` affects how strongly RRF rewards documents appearing in both lists versus only one. A lower `k` amplifies the benefit of appearing in both; a higher `k` makes the fusion more uniform. -**Substring scan is O(N)** — `find_by_name` with `exact=False` scans the entire collection client-side. On a codebase with 500,000 indexed symbols, every partial-name lookup scrolls through all symbols in batches. A Qdrant full-text index on `symbol_name` would solve this but is not currently implemented. +**Mid-token queries are still O(N), on the client** — `find_by_name` with `exact=False` is served by the `symbol_name_tokens` full-text index, but only a query prefixing the identifier, one of its subwords, or one of its suffix joins is a real index hit (~2 ms, flat from 50k to 250k symbols). A mid-token fragment such as `rder` is not a prefix of any stored token, so Qdrant returns nothing and `_find_by_name_scanning()` pages the collection over the wire instead — 0.9 s at 50k symbols, 8.8 s at 250k. Qdrant offers no n-gram tokenizer, so there is no index that would make arbitrary-substring matching sublinear. Note that a query matching *no* symbol pays this same full scan. + +**Token semantics differ from substring matching** — now that lookups are token-aware, `Auth` no longer matches `oauth_db` or `oauth2-session`: `auth` is not a prefix of the token `oauth2`. This is intentional, and differs from the pre-#72 substring scan. Subword-boundary queries (`WebAuth` → `GetWebAuthnSession`) *are* matched, via suffix joins. **`find_usages` depends on dense quality** — the "code that uses or references X" query wrapper is a heuristic. If the dense model doesn't associate the phrasing with caller patterns, results will be poor. There is no static call-graph analysis; the tool is entirely retrieval-based. diff --git a/server/embeddings/code_tokenizer.py b/server/embeddings/code_tokenizer.py index 37487aa..da97c23 100644 --- a/server/embeddings/code_tokenizer.py +++ b/server/embeddings/code_tokenizer.py @@ -2,6 +2,11 @@ import re +# Suffix joins are only emitted for the first N subwords of an identifier. The +# joins are O(k^2) in total characters for k subwords, and their value drops off +# fast — nobody starts a lookup nine subwords into a name. +MAX_SUFFIX_JOIN_SUBWORDS = 8 + def split_code_identifiers(text: str) -> str: """Split camelCase/PascalCase/snake_case into subwords; keep originals alongside. @@ -14,3 +19,34 @@ def split_code_identifiers(text: str) -> str: expanded = expanded.replace("_", " ") expanded = expanded.replace("-", " ") return text + "\n" + expanded + + +def symbol_name_tokens(name: str) -> str: + """Build the text indexed behind `find_symbol(exact=False)`. + + Extends `split_code_identifiers()` with *suffix joins* — the identifier + re-joined from each subword boundary onwards. Without them, a `PREFIX` + full-text index only matches a query that prefixes the whole identifier or + one single subword, so `WebAuth` would miss `GetWebAuthnSession`: the query + spans `Web` + `Authn` but is not anchored at the start of the name. Emitting + `WebAuthnSession` as its own token makes that a real index hit. + + Suffix joins are skipped for names containing whitespace (markdown headings, + CSS selector lists, dependency coordinates). Those are prose rather than + concatenated identifiers — their words are already separate tokens, so joins + would add nothing but index bulk. + """ + base = split_code_identifiers(name) + if re.search(r"\s", name): + return base + + subwords = base.split("\n", 1)[1].split() + if len(subwords) < 2: + return base + + capped = subwords[:MAX_SUFFIX_JOIN_SUBWORDS] + joins = ["".join(capped[i:]) for i in range(1, len(capped))] + # dict.fromkeys dedupes while preserving order; a join can repeat the + # original name (snake_case) or a lone trailing subword. + extra = [j for j in dict.fromkeys(joins) if j not in {name, *subwords}] + return base + ("\n" + " ".join(extra) if extra else "") diff --git a/server/indexer/pipeline.py b/server/indexer/pipeline.py index 7670458..25ebc3a 100644 --- a/server/indexer/pipeline.py +++ b/server/indexer/pipeline.py @@ -14,12 +14,13 @@ from server.embeddings import get_embedding_provider from server.embeddings.base import EmbeddingProvider from server.embeddings.bm25 import BM25SparseProvider, get_sparse_embedding_provider +from server.embeddings.code_tokenizer import symbol_name_tokens from server.indexer.cleanup import prune_orphaned_services from server.indexer.github_source import fetch_blob_content, list_github_files from server.parser.base import CodeSymbol, ParseError from server.parser.registry import parse_file from server.state import get_reindex_lock, get_service_registry -from server.store.qdrant import QdrantStore +from server.store.qdrant import SYMBOL_TOKENS_FIELD, QdrantStore from server.store.service_registry import ServiceRegistry, load_effective_services logger = logging.getLogger(__name__) @@ -136,6 +137,7 @@ def _symbol_to_payload( ) -> dict[str, Any]: return { "symbol_name": symbol.name, + SYMBOL_TOKENS_FIELD: symbol_name_tokens(symbol.name), "symbol_type": symbol.symbol_type, "language": symbol.language, "service": service_name, diff --git a/server/store/qdrant.py b/server/store/qdrant.py index 1655377..02d90e0 100644 --- a/server/store/qdrant.py +++ b/server/store/qdrant.py @@ -11,6 +11,7 @@ Fusion, FusionQuery, HnswConfigDiff, + MatchText, MatchValue, OptimizersConfigDiff, PayloadSchemaType, @@ -20,11 +21,38 @@ SparseIndexParams, SparseVector, SparseVectorParams, + TextIndexParams, + TextIndexType, + TokenizerType, VectorParams, ) from server.config import settings +# Payload field holding the tokenized form of symbol_name (original identifier plus +# its camelCase/snake_case subwords). Backed by a full-text index so partial-name +# lookups are served by Qdrant instead of a client-side scan. +SYMBOL_TOKENS_FIELD = "symbol_name_tokens" + +FUZZY_MATCH_LIMIT = 50 + + +def _rank_by_name(points: list[ScoredPoint], name: str) -> list[ScoredPoint]: + """Exact name matches first, then prefix matches, then the rest. + + Qdrant returns scrolled points in point-id order, which would otherwise bury an + exact hit underneath incidental partial matches. + """ + name_lower = name.lower() + + def rank(point: ScoredPoint) -> int: + symbol_name = (point.payload.get("symbol_name") or "").lower() + if symbol_name == name_lower: + return 0 + return 1 if symbol_name.startswith(name_lower) else 2 + + return sorted(points, key=rank) + def _symbol_point_id( service: str, file_path: str, symbol_name: str, start_line: int @@ -43,6 +71,9 @@ async def ensure_collection(self) -> None: exists = await self._client.collection_exists(self._collection) if exists: await self._validate_dimensions() + # Payload indexes are created unconditionally so that indexes added + # in later versions also reach collections created before them. + await self._create_payload_indexes() return await self._client.create_collection( collection_name=self._collection, @@ -85,6 +116,7 @@ async def _create_payload_indexes(self) -> None: "chunk_tier", "parent_name", "file_path", + "symbol_name", ] for field in keyword_fields: await self._client.create_payload_index( @@ -92,6 +124,18 @@ async def _create_payload_indexes(self) -> None: field_name=field, field_schema=PayloadSchemaType.KEYWORD, ) + # PREFIX tokenizer so a partial query ("Ord") matches a full token ("Order"). + await self._client.create_payload_index( + collection_name=self._collection, + field_name=SYMBOL_TOKENS_FIELD, + field_schema=TextIndexParams( + type=TextIndexType.TEXT, + tokenizer=TokenizerType.PREFIX, + min_token_len=2, + max_token_len=30, + lowercase=True, + ), + ) async def upsert_chunks( self, @@ -283,10 +327,47 @@ async def find_by_name( ) return list(results) + token_filter = Filter( + must=[ + *must, + FieldCondition(key=SYMBOL_TOKENS_FIELD, match=MatchText(text=name)), + ] + ) + results, _ = await self._client.scroll( + collection_name=self._collection, + scroll_filter=token_filter, + limit=FUZZY_MATCH_LIMIT, + with_payload=True, + with_vectors=False, + ) + matches = list(results) + if not matches: + # Two distinct cases reach here, both indistinguishable from an empty + # MatchText result: + # 1. The collection predates SYMBOL_TOKENS_FIELD, so the filter runs + # against an absent field and matches nothing until a force reindex. + # 2. A mid-token fragment ("rder", "asskey"). The PREFIX tokenizer only + # indexes token *prefixes*, so Qdrant returns nothing for these — + # it does not resolve them server-side. + matches = await self._find_by_name_scanning(name, base_filter) + return _rank_by_name(matches, name) + + async def _find_by_name_scanning( + self, name: str, base_filter: Filter | None + ) -> list[ScoredPoint]: + """Substring fallback: scrolls the collection and matches in Python. + + Pre-#72 behaviour, retained for collections indexed before + SYMBOL_TOKENS_FIELD existed and for mid-token fragments, which the + PREFIX index cannot serve. O(N) in collection size, and unlike the + indexed path it pages every payload over the wire — measured at 8.8 s + for a no-match query over 250k symbols, so it is a real cliff on large + collections, not a rounding error. + """ name_lower = name.lower() matches: list[ScoredPoint] = [] offset = None - while len(matches) < 50: + while len(matches) < FUZZY_MATCH_LIMIT: batch, offset = await self._client.scroll( collection_name=self._collection, scroll_filter=base_filter, @@ -295,9 +376,11 @@ async def find_by_name( with_payload=True, with_vectors=False, ) - for r in batch: - if name_lower in (r.payload.get("symbol_name") or "").lower(): - matches.append(r) + matches.extend( + r + for r in batch + if name_lower in (r.payload.get("symbol_name") or "").lower() + ) if offset is None: break return matches diff --git a/tests/test_code_tokenizer.py b/tests/test_code_tokenizer.py index 7198cbe..140434e 100644 --- a/tests/test_code_tokenizer.py +++ b/tests/test_code_tokenizer.py @@ -1,12 +1,28 @@ from __future__ import annotations -from server.embeddings.code_tokenizer import split_code_identifiers +import pytest + +from server.embeddings.code_tokenizer import ( + MAX_SUFFIX_JOIN_SUBWORDS, + split_code_identifiers, + symbol_name_tokens, +) def _tokens(text: str) -> set[str]: return set(split_code_identifiers(text).lower().split()) +def _indexed(name: str) -> set[str]: + """Tokens Qdrant's PREFIX tokenizer would see for a symbol name.""" + return set(symbol_name_tokens(name).lower().split()) + + +def _matches(name: str, query: str) -> bool: + """Whether a PREFIX-tokenized query would be a real index hit for `name`.""" + return any(token.startswith(query.lower()) for token in _indexed(name)) + + def test_pascal_case_splits_to_subwords() -> None: tokens = _tokens("PlaceOrderRequest") assert "place" in tokens @@ -48,3 +64,67 @@ def test_idempotent_token_set() -> None: once = set(split_code_identifiers(text).lower().split()) twice = set(split_code_identifiers(split_code_identifiers(text)).lower().split()) assert once == twice + + +def test_symbol_name_tokens_keeps_identifier_and_subwords() -> None: + tokens = _indexed("GetWebAuthnSession") + assert {"getwebauthnsession", "get", "web", "authn", "session"} <= tokens + + +def test_symbol_name_tokens_emits_suffix_joins() -> None: + tokens = _indexed("GetWebAuthnSession") + assert {"webauthnsession", "authnsession"} <= tokens + + +@pytest.mark.parametrize( + "name,query", + [ + # The regression this exists to prevent: queries spanning a subword + # boundary but not anchored at the start of the identifier. + ("GetWebAuthnSession", "WebAuth"), + ("RemoveWebAuthnSession", "WebAuthnSess"), + ("defaultSecurityFilterChain", "securityF"), + ("JwtTokenCustomizerConfig", "tokenCust"), + ("LoginBeginRequest", "BeginRe"), + ("JpaClientRepository", "ClientRep"), + ("NewInMemoryStore", "InMemory"), + # Still-supported existing behaviour. + ("GetWebAuthnSession", "GetWeb"), + ("GetWebAuthnSession", "session"), + ("placeOrderRequest", "ord"), + ("place_order_request", "orderRequest"), + ], +) +def test_subword_boundary_queries_are_index_hits(name: str, query: str) -> None: + assert _matches(name, query) + + +@pytest.mark.parametrize( + "name,query", + [ + # Mid-token fragments are deliberately not index hits — they fall through + # to the client-side scan in QdrantStore._find_by_name_scanning. + ("placeOrderRequest", "rder"), + ("GetRegisteredPasskeys", "asskey"), + # Token semantics: "auth" does not prefix the token "oauth2". + ("oauth2_session", "auth"), + ], +) +def test_mid_token_fragments_are_not_index_hits(name: str, query: str) -> None: + assert not _matches(name, query) + + +def test_prose_names_skip_suffix_joins() -> None: + """Headings and selector lists are already whitespace-tokenized.""" + name = "Registration and Authentication flow" + assert symbol_name_tokens(name) == split_code_identifiers(name) + + +def test_suffix_joins_are_capped() -> None: + name = "".join(f"Part{i}" for i in range(20)) + joins = symbol_name_tokens(name).split("\n")[2].split() + assert len(joins) <= MAX_SUFFIX_JOIN_SUBWORDS - 1 + + +def test_single_subword_name_adds_no_joins() -> None: + assert symbol_name_tokens("main") == split_code_identifiers("main") diff --git a/tests/test_pipeline.py b/tests/test_pipeline.py index 49d2721..1015d03 100644 --- a/tests/test_pipeline.py +++ b/tests/test_pipeline.py @@ -10,8 +10,10 @@ IndexPipeline, _build_bm25_text, _build_embedding_text, + _symbol_to_payload, ) from server.parser.base import CodeSymbol, ParseError +from server.store.qdrant import SYMBOL_TOKENS_FIELD _TRUNCATION_MARKER = "// ... (truncated)" @@ -61,6 +63,29 @@ def _sym(docstring: str) -> CodeSymbol: ) +def test_payload_carries_tokenized_symbol_name() -> None: + """find_symbol's full-text index matches against the split subwords, so the + payload must carry both the original identifier and its parts.""" + sym = CodeSymbol( + name="placeOrderRequest", + symbol_type="function", + language="java", + source="void placeOrderRequest() {}", + file_path="svc/Order.java", + start_line=1, + end_line=1, + ) + + payload = _symbol_to_payload(sym, "billing", "hash") + + tokens = payload[SYMBOL_TOKENS_FIELD].lower().split() + assert "placeorderrequest" in tokens + assert {"place", "order", "request"} <= set(tokens) + # Suffix joins, so a subword-boundary query ("orderReq") is an index hit + # rather than falling through to the client-side scan. + assert {"orderrequest", "request"} <= set(tokens) + + async def test_index_all_prunes_orphaned_services_before_indexing() -> None: store = AsyncMock() store.ensure_collection = AsyncMock() diff --git a/tests/test_store.py b/tests/test_store.py index 8d1f442..e101277 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -4,9 +4,16 @@ from types import SimpleNamespace from unittest.mock import AsyncMock, MagicMock -from qdrant_client.models import FieldCondition, Fusion, FusionQuery, SparseVector +from qdrant_client.models import ( + FieldCondition, + Fusion, + FusionQuery, + SparseVector, + TextIndexParams, + TokenizerType, +) -from server.store.qdrant import QdrantStore +from server.store.qdrant import SYMBOL_TOKENS_FIELD, QdrantStore def _make_record(symbol_name: str) -> SimpleNamespace: @@ -34,8 +41,48 @@ async def test_get_indexed_services_excludes_unknown_placeholder() -> None: assert services == ["billing"] -async def test_find_by_name_fuzzy_scans_all_pages() -> None: - """Non-exact search must paginate instead of relying on the first 20 results.""" +async def test_find_by_name_fuzzy_queries_the_text_index() -> None: + """Non-exact search must filter server-side instead of scanning the collection.""" + store = QdrantStore.__new__(QdrantStore) + store._collection = "test" + + store._client = MagicMock() + store._client.scroll = AsyncMock( + return_value=([_make_record("target_gamma")], None) + ) + + results = await store.find_by_name("target", exact=False) + + assert store._client.scroll.call_count == 1 + scroll_filter = store._client.scroll.call_args.kwargs["scroll_filter"] + assert any( + isinstance(c, FieldCondition) + and c.key == SYMBOL_TOKENS_FIELD + and c.match.text == "target" + for c in scroll_filter.must + ) + assert [r.payload["symbol_name"] for r in results] == ["target_gamma"] + + +async def test_find_by_name_fuzzy_keeps_other_filters_alongside_text_match() -> None: + store = QdrantStore.__new__(QdrantStore) + store._collection = "test" + + store._client = MagicMock() + store._client.scroll = AsyncMock( + return_value=([_make_record("target_gamma")], None) + ) + + await store.find_by_name("target", exact=False, service="billing") + + scroll_filter = store._client.scroll.call_args.kwargs["scroll_filter"] + keys = {c.key for c in scroll_filter.must if isinstance(c, FieldCondition)} + assert keys == {"service", SYMBOL_TOKENS_FIELD} + + +async def test_find_by_name_fuzzy_falls_back_to_scanning_when_index_misses() -> None: + """Collections indexed before symbol_name_tokens existed have nothing for + MatchText to hit, so lookups must still resolve via the paginated scan.""" store = QdrantStore.__new__(QdrantStore) store._collection = "test" @@ -45,18 +92,41 @@ async def test_find_by_name_fuzzy_scans_all_pages() -> None: store._client = MagicMock() store._client.scroll = AsyncMock( side_effect=[ + ([], None), # text index returns nothing (page1, "cursor_page2"), (page2, None), ] ) - results = await store.find_by_name("target", exact=False) + results = await store.find_by_name("arget", exact=False) - assert store._client.scroll.call_count == 2 + assert store._client.scroll.call_count == 3 assert len(results) == 1 assert results[0].payload["symbol_name"] == "target_gamma" +async def test_find_by_name_ranks_exact_then_prefix_matches_first() -> None: + store = QdrantStore.__new__(QdrantStore) + store._collection = "test" + + # Qdrant returns points in id order, which buries the exact match. + records = [ + _make_record("findOrderById"), + _make_record("OrderService"), + _make_record("Order"), + ] + store._client = MagicMock() + store._client.scroll = AsyncMock(return_value=(records, None)) + + results = await store.find_by_name("order", exact=False) + + assert [r.payload["symbol_name"] for r in results] == [ + "Order", + "OrderService", + "findOrderById", + ] + + async def test_find_by_name_exact_does_not_paginate() -> None: store = QdrantStore.__new__(QdrantStore) store._collection = "test" @@ -145,3 +215,44 @@ async def test_find_by_name_filters_by_chunk_tier() -> None: and c.match.value == "class" for c in scroll_filter.must ) + + +async def test_ensure_collection_indexes_existing_collection() -> None: + """Indexes added in later versions must reach collections created before them.""" + store = QdrantStore.__new__(QdrantStore) + store._collection = "test" + store._dimensions = 1024 + store._client = MagicMock() + store._client.collection_exists = AsyncMock(return_value=True) + store._client.create_collection = AsyncMock() + store._client.create_payload_index = AsyncMock() + store._validate_dimensions = AsyncMock() + + await store.ensure_collection() + + store._client.create_collection.assert_not_awaited() + indexed_fields = { + call.kwargs["field_name"] + for call in store._client.create_payload_index.call_args_list + } + assert "symbol_name" in indexed_fields + assert SYMBOL_TOKENS_FIELD in indexed_fields + + +async def test_symbol_name_tokens_index_uses_prefix_tokenizer() -> None: + """A partial query ('Ord') must match a full token ('Order').""" + store = QdrantStore.__new__(QdrantStore) + store._collection = "test" + store._client = MagicMock() + store._client.create_payload_index = AsyncMock() + + await store._create_payload_indexes() + + schema = next( + call.kwargs["field_schema"] + for call in store._client.create_payload_index.call_args_list + if call.kwargs["field_name"] == SYMBOL_TOKENS_FIELD + ) + assert isinstance(schema, TextIndexParams) + assert schema.tokenizer == TokenizerType.PREFIX + assert schema.lowercase is True