perf: serve partial symbol lookup from a Qdrant full-text index

[GoodbyePlanet]

Closes #72.

Problem

find_by_name(exact=False) scrolled the entire code_symbols collection in pages of 200, lowercasing and substring-matching in Python until it collected 50 hits. Every partial-name find_symbol call pulled most of the collection's payload over the wire, at O(N) in collection size.

Change

Index a derived symbol_name_tokens payload field — the identifier plus its camelCase/PascalCase/snake_case subwords, built with the existing split_code_identifiers() helper that already feeds BM25 — with a TEXT index using the PREFIX tokenizer, and query it with a single MatchText-filtered scroll.

Three adjacent problems found while verifying the issue are fixed in the same pass:

• _create_payload_indexes() only ran when the collection was created, so any index added in a later version would never reach an existing deployment. It now runs on the existing-collection path too, following the CommitStore pattern.
• symbol_name had no payload index at all, so exact=True and get_code_context were filtering against an unindexed field. Added a KEYWORD index.
• Results came back in Qdrant point-id order with no ranking, which could bury an exact hit under incidental partial matches. Now ranked exact → prefix → remainder.

Measured against a real Qdrant (24,003-symbol collection)

Query against placeOrderRequest	Latency
order (full subword token)	2.0 ms	indexed lookup
ord, plac, reques (token prefixes)	~1.5 ms	flat regardless of collection size
rder, quest (mid-token)	~390 ms	Qdrant scans; 49 ms at 3k symbols, so linear

The PREFIX tokenizer earns its keep: truncated prefixes are real index hits, which WORD would not give.

Honest limitation: mid-token fragments still match, but at linear cost — the scan moved from the client into Qdrant rather than disappearing. Qdrant has no n-gram tokenizer, so arbitrary-substring lookup cannot be made sublinear. What this PR removes is the client-side scan and its wire traffic, and it turns the common prefix query into a genuine index hit.

Migration

Collections indexed before this change have no symbol_name_tokens, so MatchText matches nothing and _find_by_name_scanning() (the previous behaviour, kept as a fallback) handles them transparently. One make index-code populates the field and reaches the fast path. No collection drop or re-embedding needed.

Verification

• uv run pytest — 311 passed. tests/test_store.py's old test_find_by_name_fuzzy_scans_all_pages is replaced by four tests (indexed query, filters coexisting with the text match, fallback path, ranking), plus tests for idempotent index creation, the tokenizer config, and the payload token field.
• Live against a real Qdrant in throwaway collections: payload schema shows symbol_name_tokens: text and symbol_name: keyword; ensure_collection() is idempotent; ranking confirmed; zzz returns nothing; stripping the token field falls back correctly.

Why needs testing

The numbers above come from synthetic symbol names in a throwaway collection. Worth confirming on a real indexed repo before merge:

• find_symbol latency and result quality on an actual code_symbols collection, especially with service / symbol_type filters stacked alongside the text match
• that a live server booting against a pre-existing collection creates the two new indexes without disturbing it, and that partial lookups keep working before the reindex
• PREFIX index build cost and memory on a large collection — each token expands to up to 30 prefixes

🤖 Generated with Claude Code

[GoodbyePlanet]

Live test on a real indexed collection

Ran the needs testing items against a real Qdrant with the actual code_symbols collection: 709 symbols across 7 services (Java/Go/TypeScript/XML/SQL/Dockerfile), last indexed 2026-08-12 — genuinely pre-dating this change, so the migration path was exercised for real rather than simulated.

uv run pytest — 311 passed.

✅ Confirmed

Index creation against a pre-existing collection. Server booted, issued 8 PUT /collections/code_symbols/index calls, no errors, collection stayed green with all 709 points intact. Both new indexes appeared (symbol_name: keyword, symbol_name_tokens: text/prefix), and it's idempotent across restarts.

Pre-reindex fallback. Every partial lookup kept working before migration (~22 ms via the client scan), transparently.

Post-reindex latency. The headline claim holds on real data, ~11x:

Query	Before (scan)	After (indexed)
Controller (full token)	23.9 ms	1.8 ms
regist (prefix)	22.1 ms	1.9 ms
Registration (camel subword)	23.2 ms	2.0 ms
Auth + service filter	13.3 ms	3.6 ms
Registration + service + type filter	5.0 ms	2.0 ms

Filters compose correctly with the text match — identical hits, no interference. Exact-first ranking works.

---

❌ Finding 1 — silent recall regression

Post-migration, queries that span a subword boundary but aren't anchored at the identifier's start silently return fewer results than before. A sweep of 768 fragments derived from actual symbol names in the collection found 20 such queries:

'WebAuth'    indexed=12  scan=18   missed: GetWebAuthnSession, InitWebAuthn, RemoveWebAuthnSession, …
'securityF'  indexed=1   scan=3    missed: defaultSecurityFilterChain, authorizationServerSecurityFilterChain
'BeginRe'    indexed=1   scan=3    missed: LoginBeginRequest, RegistrationBeginRequest
'tokenCust'  indexed=1   scan=2    missed: JwtTokenCustomizerConfig
'ClientRep'  indexed=1   scan=2    missed: JpaClientRepository

Cause: split_code_identifiers("GetWebAuthnSession") stores "GetWebAuthnSession\nGet Registered Passkeys"-style text, so a query only matches when it prefixes either the whole identifier or one subword. webauth spans Web + Authn, so it misses — while the old substring scan found it.

The fallback does not rescue this, because it only fires when the indexed query returns zero rows. Here it returns 12, so _find_by_name_scanning() never runs and a third of the hits disappear with no signal. WebAuth finding 12 of 18 WebAuthn symbols is a plausible query with a quietly wrong answer.

Cheapest correct fix is probably to union the scan in whenever the indexed result count is below FUZZY_MATCH_LIMIT, rather than only when it's empty — keeps the fast path fast for broad queries, costs a scan on narrow ones. Alternative is indexing subword suffixes too, trading index size for recall.

❌ Finding 2 — the mid-token comment is inaccurate

server/store/qdrant.py:347 states mid-token fragments "do NOT need this path — Qdrant resolves those itself." Verified directly against the collection:

raw MatchText('asskey')     -> 0 hits
raw MatchText('gistration') -> 0 hits
raw MatchText('uthCont')    -> 0 hits
raw MatchText('WebAuth')    -> 15 hits

Mid-token returns nothing from Qdrant and always falls through to the client-side scan. Measured post-migration: asskey 23.0 ms, gistration 23.3 ms — unchanged from the pre-migration baseline.

So the PR description's "rder, quest → ~390 ms, Qdrant scans" is mis-attributed: that is the client-side scan this PR set out to remove, still running on fully migrated collections. Same applies to every zero-result query — zzzqqq costs a full 23 ms scan, and an empty result is indistinguishable from "collection not yet migrated".

❌ Finding 3 — the migration instructions don't work

The description says one make index-code populates the field, with no re-embedding needed. On the real collection:

POST /reindex {"service":"passkey-service"}
{"type": "done", "result": {"files": 0, "chunks": 0, "skipped": 15}}

Blob-SHA change detection skips every unchanged file, so nothing is upserted and symbol_name_tokens stayed at 0 points. Migration actually requires force: true, which re-embeds the entire collection — 709 chunks took 3m59s plus a full round of embedding API calls. Worth correcting in the README/docs, since the cost on a large repo is not incidental.

---

Not assessable at this scale

PREFIX index build cost and memory. At 709 points (35 MB collection, Qdrant at 481 MB RSS) there's no usable signal to extrapolate to a large repo. That item stays open.

Summary

The core performance claim is real and reproducible on live data, and the migration is non-destructive. Findings 2 and 3 are documentation/comment corrections. Finding 1 is a behavioural regression that I'd want resolved before merge — it trades correctness for latency without surfacing that it's doing so.