feat(relevant): grade lexical relevance with ts_rank_cd and cut the candidate window by rank

[dash0-dev]

Summary

GET /memories/relevant scored relevance as a binary 1-or-nothing, and cut its candidate window by recency. Both were the same limitation: ts_rank_cd is not projectable through PostgREST's query grammar, so from the PostgREST side there was no graded value to sort by and no rank to cut by. The consequence was that ordering among matches was decided entirely by recency and salience, and that on a store with more than CANDIDATE_LIMIT matches the best-matching lesson could fall outside the window and never be scored at all.

Migration 00112 moves the candidate fetch into lorekit_memory_relevant_candidates, which returns ts_rank_cd(fts, websearch_to_tsquery('english', q)) alongside each row and — when a query is present — orders the window by it. Retrieval stays deterministic and embedding-free: a pure Postgres function over the existing fts column, no model, no vector index, no external call on the request path.

This is PR 1 of a four-PR stack. PR 2 (RRF rank fusion) depends on it: binary relevance produced a degenerate rank list, and this is what gives RRF a real signal to fuse.

Changes

• supabase/migrations/00112_relevant_ranking.sql — new lorekit_memory_relevant_candidates(p_user_id, p_q, p_scopes, p_key_scopes, p_key_org_access, p_key_org_ids, p_limit). Returns the route's narrow projection (scope, key, value, seen_count, updated_at, tags, origin_pr) plus relevance double precision. STABLE / SECURITY DEFINER with a pinned search_path, following lorekit_memory_read_ranking (00085). No new index — the q path is served by memories_fts_idx (00001), the no-q path by memories_user_updated_at_id_idx (00096).
• relevant.ts — swaps the PostgREST query for the RPC, maps the raw rank through normalizeLexicalRank, and drops the now-stale binary-relevance comment. getMemberOrgIds + applyRestTenantScope go away: the RPC composes the same three SQL functions those helpers mirror, which also removes a round trip (membership becomes a join inside the one query). RELEVANT_SELECT is deleted from both schema mirrors: the RPC's returns table is now the route's only column list.
• lesson-rank.ts (+ byte mirror) — adds normalizeLexicalRank and LEXICAL_RANK_SATURATION. Purely additive; scoreLesson, rankLessons and selectDiverse are untouched, so lesson-rank-parity.spec.ts is unaffected.
• lexical-rank.spec.ts — pins the mapping by value, not merely by property.
• tenant-scope-usage.spec.ts — relevant moves from REST_READ_HANDLERS to RPC_BACKED_ROW_READS (the same move list made at 00067), plus a new key-tenancy usage guard for org-widening RPC reads.
• migrations.test.sql §112 — 11 acceptance criteria (AC-11: an authenticated caller's p_user_id is ignored) plus a function-shape block.
• Docs: decisions.md entry + CLAUDE.md index line, memories/CLAUDE.md, the relevant schema docblocks, llms/template.md → regenerated llms.txt.

Deliberate deviations from the plan

Three, each with a reason:

1. SECURITY DEFINER, not SECURITY INVOKER. The api_key tier calls this on the service-role client, where an invoker function applies no RLS and therefore no predicate at all. Definer gives one predicate serving both auth tiers, and it is the shape every sibling analytics RPC already uses.
2. Tenancy takes p_key_org_access / p_key_org_ids as well as p_key_scopes. applyRestTenantScope applies two narrowings — the scope allowlist and the key's org tenancy (effectiveOrgIds / ownRowsFragment). Moving the query into an RPC without the second would have let a personal-tenancy key read every org row its owner can see. The RPC composes lorekit_api_token_org_allowed, the SQL twin those helpers mirror, under the repo's existing p_key_org_* parameter names.
3. Saturation is 0.1, not 1. The plan suggested r / (r + 1). memories.fts applies no setweight, so every lexeme carries Postgres's default D weight of 0.1 and a single-term cover is a ts_rank_cd of 0.1 — meaning r / (r + 1) would squash the entire plausible range into [0.09, 0.5] and a "graded" signal would arrive with most of its say already spent. LEXICAL_RANK_SATURATION = 0.1 puts a single-term match at the factor's midpoint. Both choices are deterministic and strictly increasing; this one spends the range on values that occur. The constant is pinned in lexical-rank.spec.ts and its anchor is pinned in migrations.test.sql §112 AC-4, so if the 0.1 claim is wrong, CI says so rather than the constant drifting quietly.

Behaviour change callers should know about

min_score now gates across the whole range. Under binary relevance no matched score could drop below (1 + 0.5) / 4 = 0.375 — relevance at 1, outcome never below its 0.5 cold-start prior — so any min_score at or under 0.375 was a no-op. Grading removes that floor, which is the point, but it means a threshold tuned against the old behaviour is now stricter than it was. Documented in the schema docblock, memories/CLAUDE.md, and llms.txt.

One thing deliberately not changed: an all-stopword q (e.g. the) still matches nothing rather than being folded into the no-query path. An empty tsquery matching nothing is exactly what .textSearch did; rescuing it would silently answer a different question. Pinned by AC-6.

How to verify

pnpm nx test mcp-core                    # 1908 passed, 1 skipped
pnpm nx test schemas                     # 283 passed
pnpm nx run-many -t typecheck,lint -p mcp-core,schemas --parallel=1
NODE_USE_ENV_PROXY=1 node scripts/ci/deno-check-functions.mjs --node-modules-dir=none
node scripts/migrations/check-migration-order.mjs origin/main
node packages/cli/bin/lorekit.mjs obligations

All green locally. Lint reports pre-existing warnings only (0 errors).

migrations.test.sql was NOT run locally — no Postgres in the sandbox (docs/sandbox.md trap 4), and the throwaway PG16 path needs a cluster this box does not have. §112 — including the AC-4 anchor and AC-11 — is proven by CI's Integration smoke job instead, which is green at the PR head.

[github-actions]

The dashboard preview for this PR — redeployed on each push that changes the web app.

Project	Deployment	Actions	Updated
lorekit	Ready	Preview	Oct 11, 2026 6:15am UTC

_{Preview always points at this PR's latest commit; Deployment is the immutable build of 49dd558. · Comment /web-preview to force a redeploy. · Workflow logs}

[dash0-dev]

✅ No issues found

Re-review at 49dd558: both doc fixes land in both schema mirrors and the 112 header now indexes AC-11. No new findings; tenancy, the rank-cut window and CI are unchanged. PR body still says 10 ACs (cosmetic).

Checked: 14 of 14 changed files read · 3 of 6 dependent files traced · 6 possible issues → 0 confirmed → 0 posted

Progress: open review threads 4 → 1 → 1 → 2 → 0 across the last 5 reviews

What this change reaches — 8 changed exports · 6 dependent files · 3 checked · 3 not checked

flowchart LR
  subgraph pr["Changed in this PR"]
    s1["normalizeRelevance<br/>body changed"]:::changed
    s2["RelevantQuerySchema<br/>body changed"]:::changed
    s3["handleRelevant<br/>body changed"]:::changed
    s4["normalizeRelevance<br/>body changed"]:::changed
    s5["RelevantQuerySchema<br/>body changed"]:::changed
    s6["LEXICAL_RANK_SATURATION<br/>added"]:::changed
    smore["+2 more changed exports<br/>listed below"]:::changed
  end
  s1c1["packages/mcp-core/src/ranking/ · 2 files<br/>? not checked"]:::unknown
  s2c1["supabase/functions/memories/handlers/relevant.ts<br/>✓ checked"]:::ok
  s2c2["supabase/functions/_shared/schemas/openapi/spec.ts<br/>? not checked"]:::unknown
  s3c1["supabase/functions/memories/index.ts<br/>✓ checked"]:::ok
  s4c1["supabase/functions/memories/handlers/relevant.ts<br/>✓ checked"]:::ok
  s5c1["packages/schemas/src/openapi/spec.ts<br/>✓ checked"]:::ok
  s6c1["packages/mcp-core/src/ranking/lexical-rank.spec.ts<br/>✓ checked"]:::ok
  s1 -.-> s1c1
  s2 --> s2c1
  s2 -.-> s2c2
  s3 --> s3c1
  s4 --> s4c1
  s5 --> s5c1
  s6 --> s6c1
  classDef changed fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
  classDef ok fill:#e7f6ec,stroke:#16a34a,color:#14532d
  classDef partial fill:#fef9c3,stroke:#ca8a04,color:#713f12
  classDef unknown fill:#f3f4f6,stroke:#9ca3af,stroke-dasharray:4 3,color:#374151
  classDef bad fill:#fde8e8,stroke:#dc2626,color:#7f1d1d
  classDef warn fill:#fff4e5,stroke:#d97706,color:#78350f
  classDef imports fill:#f8fafc,stroke:#64748b,stroke-dasharray:4 3,color:#1e293b

Loading

• normalizeRelevance (packages/mcp-core/src/ranking/lesson-rank.ts) — body change · 2 consumer files · 2 not traced
• RelevantQuerySchema (supabase/functions/_shared/schemas/relevant.ts) — body change · 2 consumer files · 1 verified unaffected · 1 not traced
• handleRelevant (supabase/functions/memories/handlers/relevant.ts) — body change · 1 consumer file · 1 verified unaffected
• normalizeRelevance (supabase/functions/_shared/ranking/lesson-rank.ts) — body change · 1 consumer file · 1 verified unaffected
• RelevantQuerySchema (packages/schemas/src/shared/relevant.ts) — body change · 1 consumer file · 1 verified unaffected
• LEXICAL_RANK_SATURATION (packages/mcp-core/src/ranking/lesson-rank.ts) — added change · 1 consumer file · 1 verified unaffected
• normalizeLexicalRank (packages/mcp-core/src/ranking/lesson-rank.ts) — added change · 1 consumer file · 1 verified unaffected
• normalizeLexicalRank (supabase/functions/_shared/ranking/lesson-rank.ts) — added change · 1 consumer file · 1 verified unaffected

Review details

Gate	Status	Details
Description vs. code	✅	Matches the diff; body still names the removed RELEVANT_SELECT and says 10 §112 ACs, not 11 (cosmetic).
Prior review feedback	✅	Earlier review comments are resolved.
Documentation	✅	Migration header, docblocks, decisions.md, memories/CLAUDE.md and llms.txt explain grading, tenancy, min_score.
Self-review signals	✅	No debug logs, leftover TODOs, or unreviewed stubs.
Code review	✅	The multi-lens review found no blocking issues.

Found

Quality — produced 6 → posted inline 0 · cleared 0 · carried forward 0 · deferred 0 · below-bar 0
Dropped — 3 below-bar
Optimality — ran: 3 approach units (definer candidate RPC composing the tenancy functions, TS-side saturation, rank-ordered window) re-judged optimal; delta is docs-only; 0 proposals
Standards — ran: 5 governing docs, 123 bullets, all 14 files; 0 findings (llms.txt regenerated from template.md, schema mirrors in sync, decisions entry plus index line present)
Measurability — ran · 14 paths classified · 0 missing · 1 unlinked · no profile

Run

full · 8 lines in delta · tier deep · depth checkout
⚠️ hybrid topology ran in-context (no sub-agent dispatch on this host), so the intent finder and verifier were not isolated; SQL claims verified by reading only (no Postgres here); CI Integration smoke ran §112 green at this head
Memories — 60 indexed · 5 used (4 knowledge · 1 hotspot)

• knowledge knowledge::handleRelevant@supabase/functions/memories/handlers/relevant.ts — re-verified at 49dd558 (file unchanged since 0a59e5f): RPC args, null-to-0 mapping, no in-repo min_score caller, min_score absent from the span
• knowledge knowledge::lorekit_memory_relevant_candidates@supabase/migrations/00112_relevant_ranking.sql — re-verified at 49dd558 (migration unchanged): actor rule, tenancy predicate, rank order; perf history ambiguous twice
• knowledge knowledge::normalizeLexicalRank@packages/mcp-core/src/ranking/lesson-rank.ts — re-verified at 49dd558: n/(n+s), byte-identical mirrors, cover docblocks consistent
• knowledge knowledge::RelevantEntrySchema@packages/schemas/src/shared/relevant.ts — fact 1 (docblock disagreement) now stale: 51bc31d fixed both mirrors
• hotspot hotspot::supabase/tests/migrations.test.sql — checklist: test-gap and missing-update classes; AC index now complete through AC-11

^{Nothing to report — integrations (not activated), severity, 0 files skipped.}

^{pr-reviewer · commit 49dd558 · full review · how these findings are produced · updated Oct 11, 2026 6:39am UTC}