perf: serve partial symbol lookup from a Qdrant full-text index

README.md

@@ -283,7 +283,7 @@ Tests live under `tests/`:
 | Tool                    | Description                                                                                          |
 |-------------------------|------------------------------------------------------------------------------------------------------|
 | `search_code`           | Hybrid (dense + BM25) search by query, with optional filters for language, service, symbol type      |
-| `find_symbol`           | Look up a symbol by name — exact match, or case-insensitive substring when `exact=false`             |
+| `find_symbol`           | Look up a symbol by name — exact match, or case-insensitive token match when `exact=false`          |
 | `find_usages`           | Find code that references a given symbol name (semantic search, then excludes the definition itself) |
 | `get_code_context`      | Fetch the full source of a file — or a specific symbol within it — directly from GitHub              |
 | `reindex`               | Trigger code indexing of one or all services (incremental by default; `force` to re-embed)           |
@@ -293,6 +293,12 @@ Tests live under `tests/`:
 | `list_indexed_services` | List indexed services with chunk and file counts, languages, and last-indexed time                   |
 | `index_stats`           | Show Qdrant collection statistics and configured services                                            |
 
+`find_symbol(exact=false)` matches against a full-text index over the symbol name's camelCase/snake_case tokens, so
+`order` or `ord` finds `placeOrderRequest` in ~1.5 ms regardless of collection size. Mid-token fragments (`rder`) still
+match, but cost a linear server-side scan. Collections indexed before this field existed fall back to a client-side
+scan until reindexed once with `make index-code` — see
+[docs/retrieval-rrf.md](docs/retrieval-rrf.md#name-lookup-find_by_name).
+
 ## MCP Prompts
 
 | Prompt                   | Arguments | Description                                                                                                                                                                                 |

docs/retrieval-rrf.md

@@ -79,17 +79,28 @@ Queries Qdrant with a keyword filter on the `symbol_name` payload field:
 FieldCondition(key="symbol_name", match=MatchValue(value=name))
 ```
 
-Returns up to 20 exact matches via a scroll operation. Additional filters for `symbol_type` and `service` are stacked into the same `must` list. No vectors are fetched.
+Returns up to 20 exact matches via a scroll operation. `symbol_name` carries a `KEYWORD` payload index, so this filter is served by Qdrant. Additional filters for `symbol_type` and `service` are stacked into the same `must` list. No vectors are fetched.
 
-### Substring mode (`exact=False`, default)
+### Partial mode (`exact=False`, default)
 
-Qdrant has no native text-contains index for partial name matching. The implementation falls back to a **client-side substring scan**:
+Matching is **token-aware and case-insensitive**, served server-side by a full-text payload index.
 
-1. Scroll the collection in batches of 200 points
-2. For each point, check whether `name.lower()` appears in `payload['symbol_name'].lower()`
-3. Collect up to 50 matches, then stop
+At index time, `_symbol_to_payload()` stores a derived `symbol_name_tokens` field containing the original identifier plus its camelCase/PascalCase/snake_case subwords, produced by the same `split_code_identifiers()` helper that feeds BM25. That field carries a `TEXT` index with the `PREFIX` tokenizer (`lowercase=True`, token length 2–30), and `find_by_name` queries it with a single `MatchText`-filtered scroll returning up to 50 matches.
 
-This is **O(N)** in collection size — it scans every indexed symbol in the collection (or service subset, if filtered). On large codebases with hundreds of thousands of symbols, this can be slow.
+Matching a query against `placeOrderRequest`, measured against a real Qdrant on a 24,003-symbol collection:
+
+| Query | Matches | Latency | Why |
+| --- | --- | --- | --- |
+| `order` | ✅ | 2.0 ms | full subword token — indexed lookup |
+| `ord`, `plac`, `reques` | ✅ | ~1.5 ms | `PREFIX` tokenizer indexes every token prefix |
+| `place order` | ✅ | — | `MatchText` requires all query tokens to match |
+| `rder`, `quest` | ✅ | ~390 ms | mid-token: Qdrant cannot use the index and scans (see below) |
+
+Results are then ranked exact name → prefix → remainder, because Qdrant returns scrolled points in point-id order and would otherwise bury the exact hit.
+
+**Mid-token queries still cost O(N), server-side.** A fragment that is not a token prefix (`rder` inside `placeOrderRequest`) does still match — Qdrant falls back to scanning rather than returning nothing — but the cost scales linearly with collection size: measured 49 ms at 3,003 symbols and 387 ms at 24,003, whereas token-prefix queries stay flat at ~1.5 ms regardless of size. What issue [#72](https://github.com/GoodbyePlanet/semcode/issues/72) removed is the *client-side* scan: no payload is paged over the wire any more, and the common prefix query is now a genuine index hit.
+
+**Fallback.** When the full-text filter returns zero results, `_find_by_name_scanning()` runs the pre-#72 behaviour — scroll in batches of 200 and substring-match `symbol_name` in Python. Its only remaining purpose is collections indexed before `symbol_name_tokens` existed, where a `MatchText` filter on the absent field matches nothing. Run `make index-code` once to populate the field; until then every partial lookup pays that scan. (A genuinely unmatched query, e.g. `zzz`, also triggers it — one wasted scan on a path that returns nothing either way.)
 
 ---
 
@@ -117,7 +128,7 @@ Each result includes: symbol name and type, RRF score, file location (path + lin
 find_symbol(name: str, symbol_type: str | None, service: str | None, chunk_tier: str | None, exact: bool = False) -> str
 ```
 
-Name-based lookup via `store.find_by_name()`. Does not use vectors or RRF. Supports filtering by `chunk_tier` (`"method"` or `"class"`) in addition to `symbol_type` and `service`. Returns up to 20 (exact) or 50 (substring) matches. Each result includes: name, type, location, package, parent class, and source (first 800 characters).
+Name-based lookup via `store.find_by_name()`. Does not use vectors or RRF. Supports filtering by `chunk_tier` (`"method"` or `"class"`) in addition to `symbol_type` and `service`. Returns up to 20 (exact) or 50 (partial) matches, exact names first. Each result includes: name, type, location, package, parent class, and source (first 800 characters).
 
 ### `find_usages`
 
@@ -174,7 +185,7 @@ public OrderResult processOrder(OrderRequest request) {
 
 **RRF constant is not configurable** — Qdrant's `k=60` default is used. There is no way to adjust this via configuration. The choice of `k` affects how strongly RRF rewards documents appearing in both lists versus only one. A lower `k` amplifies the benefit of appearing in both; a higher `k` makes the fusion more uniform.
 
-**Substring scan is O(N)** — `find_by_name` with `exact=False` scans the entire collection client-side. On a codebase with 500,000 indexed symbols, every partial-name lookup scrolls through all symbols in batches. A Qdrant full-text index on `symbol_name` would solve this but is not currently implemented.
+**Mid-token queries are still O(N)** — `find_by_name` with `exact=False` is served by the `symbol_name_tokens` full-text index, but only a token or token-prefix query is a real index hit (~1.5 ms at 24k symbols). A mid-token fragment such as `rder` still matches, at a linear cost Qdrant absorbs server-side (387 ms at 24k, 49 ms at 3k). Qdrant offers no n-gram tokenizer, so there is no index that would make arbitrary-substring matching sublinear.
 
 **`find_usages` depends on dense quality** — the "code that uses or references X" query wrapper is a heuristic. If the dense model doesn't associate the phrasing with caller patterns, results will be poor. There is no static call-graph analysis; the tool is entirely retrieval-based.
 

server/indexer/pipeline.py

@@ -14,12 +14,13 @@
 from server.embeddings import get_embedding_provider
 from server.embeddings.base import EmbeddingProvider
 from server.embeddings.bm25 import BM25SparseProvider, get_sparse_embedding_provider
+from server.embeddings.code_tokenizer import split_code_identifiers
 from server.indexer.cleanup import prune_orphaned_services
 from server.indexer.github_source import fetch_blob_content, list_github_files
 from server.parser.base import CodeSymbol, ParseError
 from server.parser.registry import parse_file
 from server.state import get_reindex_lock, get_service_registry
-from server.store.qdrant import QdrantStore
+from server.store.qdrant import SYMBOL_TOKENS_FIELD, QdrantStore
 from server.store.service_registry import ServiceRegistry, load_effective_services
 
 logger = logging.getLogger(__name__)
@@ -136,6 +137,7 @@ def _symbol_to_payload(
 ) -> dict[str, Any]:
     return {
         "symbol_name": symbol.name,
+        SYMBOL_TOKENS_FIELD: split_code_identifiers(symbol.name),
         "symbol_type": symbol.symbol_type,
         "language": symbol.language,
         "service": service_name,

server/store/qdrant.py

@@ -11,6 +11,7 @@
     Fusion,
     FusionQuery,
     HnswConfigDiff,
+    MatchText,
     MatchValue,
     OptimizersConfigDiff,
     PayloadSchemaType,
@@ -20,11 +21,38 @@
     SparseIndexParams,
     SparseVector,
     SparseVectorParams,
+    TextIndexParams,
+    TextIndexType,
+    TokenizerType,
     VectorParams,
 )
 
 from server.config import settings
 
+# Payload field holding the tokenized form of symbol_name (original identifier plus
+# its camelCase/snake_case subwords). Backed by a full-text index so partial-name
+# lookups are served by Qdrant instead of a client-side scan.
+SYMBOL_TOKENS_FIELD = "symbol_name_tokens"
+
+FUZZY_MATCH_LIMIT = 50
+
+
+def _rank_by_name(points: list[ScoredPoint], name: str) -> list[ScoredPoint]:
+    """Exact name matches first, then prefix matches, then the rest.
+
+    Qdrant returns scrolled points in point-id order, which would otherwise bury an
+    exact hit underneath incidental partial matches.
+    """
+    name_lower = name.lower()
+
+    def rank(point: ScoredPoint) -> int:
+        symbol_name = (point.payload.get("symbol_name") or "").lower()
+        if symbol_name == name_lower:
+            return 0
+        return 1 if symbol_name.startswith(name_lower) else 2
+
+    return sorted(points, key=rank)
+
 
 def _symbol_point_id(
     service: str, file_path: str, symbol_name: str, start_line: int
@@ -43,6 +71,9 @@ async def ensure_collection(self) -> None:
         exists = await self._client.collection_exists(self._collection)
         if exists:
             await self._validate_dimensions()
+            # Payload indexes are created unconditionally so that indexes added
+            # in later versions also reach collections created before them.
+            await self._create_payload_indexes()
             return
         await self._client.create_collection(
             collection_name=self._collection,
@@ -85,13 +116,26 @@ async def _create_payload_indexes(self) -> None:
             "chunk_tier",
             "parent_name",
             "file_path",
+            "symbol_name",
         ]
         for field in keyword_fields:
             await self._client.create_payload_index(
                 collection_name=self._collection,
                 field_name=field,
                 field_schema=PayloadSchemaType.KEYWORD,
             )
+        # PREFIX tokenizer so a partial query ("Ord") matches a full token ("Order").
+        await self._client.create_payload_index(
+            collection_name=self._collection,
+            field_name=SYMBOL_TOKENS_FIELD,
+            field_schema=TextIndexParams(
+                type=TextIndexType.TEXT,
+                tokenizer=TokenizerType.PREFIX,
+                min_token_len=2,
+                max_token_len=30,
+                lowercase=True,
+            ),
+        )
 
     async def upsert_chunks(
         self,
@@ -283,10 +327,40 @@ async def find_by_name(
             )
             return list(results)
 
+        token_filter = Filter(
+            must=[
+                *must,
+                FieldCondition(key=SYMBOL_TOKENS_FIELD, match=MatchText(text=name)),
+            ]
+        )
+        results, _ = await self._client.scroll(
+            collection_name=self._collection,
+            scroll_filter=token_filter,
+            limit=FUZZY_MATCH_LIMIT,
+            with_payload=True,
+            with_vectors=False,
+        )
+        matches = list(results)
+        if not matches:
+            # The collection predates SYMBOL_TOKENS_FIELD, so a MatchText filter on the
+            # absent field matches nothing until it is reindexed. (Mid-token fragments
+            # such as "rder" do NOT need this path — Qdrant resolves those itself.)
+            matches = await self._find_by_name_scanning(name, base_filter)
+        return _rank_by_name(matches, name)
+
+    async def _find_by_name_scanning(
+        self, name: str, base_filter: Filter | None
+    ) -> list[ScoredPoint]:
+        """Substring fallback: scrolls the collection and matches in Python.
+
+        Pre-#72 behaviour, kept only for collections indexed before
+        SYMBOL_TOKENS_FIELD existed. O(N) in collection size, and unlike the
+        indexed path it pages every payload over the wire.
+        """
         name_lower = name.lower()
         matches: list[ScoredPoint] = []
         offset = None
-        while len(matches) < 50:
+        while len(matches) < FUZZY_MATCH_LIMIT:
             batch, offset = await self._client.scroll(
                 collection_name=self._collection,
                 scroll_filter=base_filter,
@@ -295,9 +369,11 @@ async def find_by_name(
                 with_payload=True,
                 with_vectors=False,
             )
-            for r in batch:
-                if name_lower in (r.payload.get("symbol_name") or "").lower():
-                    matches.append(r)
+            matches.extend(
+                r
+                for r in batch
+                if name_lower in (r.payload.get("symbol_name") or "").lower()
+            )
             if offset is None:
                 break
         return matches

tests/test_pipeline.py

@@ -10,8 +10,10 @@
     IndexPipeline,
     _build_bm25_text,
     _build_embedding_text,
+    _symbol_to_payload,
 )
 from server.parser.base import CodeSymbol, ParseError
+from server.store.qdrant import SYMBOL_TOKENS_FIELD
 
 _TRUNCATION_MARKER = "// ... (truncated)"
 
@@ -61,6 +63,26 @@ def _sym(docstring: str) -> CodeSymbol:
     )
 
 
+def test_payload_carries_tokenized_symbol_name() -> None:
+    """find_symbol's full-text index matches against the split subwords, so the
+    payload must carry both the original identifier and its parts."""
+    sym = CodeSymbol(
+        name="placeOrderRequest",
+        symbol_type="function",
+        language="java",
+        source="void placeOrderRequest() {}",
+        file_path="svc/Order.java",
+        start_line=1,
+        end_line=1,
+    )
+
+    payload = _symbol_to_payload(sym, "billing", "hash")
+
+    tokens = payload[SYMBOL_TOKENS_FIELD].lower().split()
+    assert "placeorderrequest" in tokens
+    assert {"place", "order", "request"} <= set(tokens)
+
+
 async def test_index_all_prunes_orphaned_services_before_indexing() -> None:
     store = AsyncMock()
     store.ensure_collection = AsyncMock()