Skip to content

feat(relevant): record min_score and its filtered-hit count on the span - #688

Merged
dash0-dev[bot] merged 3 commits into
mainfrom
feat/relevant-span-min-score
Oct 11, 2026
Merged

dash0-dev[bot] merged 3 commits into
mainfrom
feat/relevant-span-min-score

Conversation

@dash0-dev

@dash0-dev dash0-dev Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Why

GET /memories/relevant drops ranked hits scoring below min_score, and since 00112 graded relevance, a threshold tuned against the old 0.375 floor can drop hits it used to keep. The span showed that a response came back short (result_count < min(limit, candidate_count)), but not the threshold or how many hits it removed.

What changed

  • lorekit.min_score on the handler span — always recorded, 0 included, bounded to [0,1] by RelevantQuerySchema.
  • lorekit.min_score_filtered_count — ranked candidates the threshold removed before diversification. Read with candidate_count and result_count, it separates "nothing matched" from "min_score dropped them" from "limit cut the list".
  • relevant-min-score-tracing.spec.ts pins both, as a source scan (the rest-auth-tracing.spec.ts pattern — the edge tree has no test harness).

How to verify

  • pnpm nx test mcp-core — the new spec fails 3 of 4 against main's handler and passes on this branch.
  • node scripts/ci/deno-check-functions.mjs --node-modules-dir=none — memories reports 0 errors.

Notes for reviewers

No user-facing docs or llms.txt change: span attributes alter nothing a user can do, see or configure. Nothing watches the new count yet; an alert on min_score_filtered_count > 0 is a follow-up once /relevant has a caller.

GET /memories/relevant now stamps `lorekit.min_score` (always, 0 included)
and `lorekit.min_score_filtered_count` (ranked candidates the threshold
removed before diversification). Since 00112 graded relevance, a min_score
tuned against the old 0.375 floor can drop hits it used to keep; the span
showed that something was filtered but not the threshold or how much.

Pinned by a source-scan spec in mcp-core, the rest-auth-tracing pattern,
because the Deno edge tree has no test harness of its own.

Co-authored-by: dash0-dev[bot] <257284812+dash0-dev[bot]@users.noreply.github.com>
@dash0-dev

dash0-dev Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor Author

✅ No issues found

Re-review of the two fix commits: the same-call test now checks membership without pinning key order (replayed: reorder passes, split still fails), and the comment names the filtered step. No new findings.

Checked: 2 of 2 changed files read · 0 of 1 dependent file traced · 2 possible issues → 0 confirmed → 0 posted

Progress: open review threads 2 → 0 across the last 2 reviews

What this change reaches — 1 changed export · 1 dependent file · 0 checked · 1 not checked
flowchart LR
  subgraph pr["Changed in this PR"]
    s1["handleRelevant<br/>body changed"]:::changed
  end
  s1c1["supabase/functions/memories/index.ts<br/>? not checked"]:::unknown
  s1 -.-> s1c1
  classDef changed fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
  classDef ok fill:#e7f6ec,stroke:#16a34a,color:#14532d
  classDef partial fill:#fef9c3,stroke:#ca8a04,color:#713f12
  classDef unknown fill:#f3f4f6,stroke:#9ca3af,stroke-dasharray:4 3,color:#374151
  classDef bad fill:#fde8e8,stroke:#dc2626,color:#7f1d1d
  classDef warn fill:#fff4e5,stroke:#d97706,color:#78350f
  classDef imports fill:#f8fafc,stroke:#64748b,stroke-dasharray:4 3,color:#1e293b
Loading
  • handleRelevant (supabase/functions/memories/handlers/relevant.ts) — body change · 1 consumer file · 1 not traced
Review details
Gate Status Details
Description vs. code ✅ Description matches the diff; the 3-of-4-fail-on-main claim re-reproduced against the new spec.
Prior review feedback ✅ Earlier review comments are resolved.
Documentation ✅ Body covers why, what and how to verify; the comment now names the filtered step.
Self-review signals ✅ No debug logs, leftover TODOs, or unreviewed stubs.
Code review ✅ The multi-lens review found no blocking issues.

Found

Quality — produced 2 → posted inline 0 · cleared 0 · carried forward 0 · deferred 0 · below-bar 0
Standards — ran — delta files; 2 docs, 102 bullets; 0 findings
Measurability — ran — quiet: gate 2 not met (delta is a comment rewording and a test-assertion refactor; no new action, operation or failure mode)

Run

incremental · 23 lines in delta · tier standard · depth checkout
Integrations — not activated (no manifest or lockfile in the diff)
CI — still pending: Typecheck, Test & Lint (affected) · 5 passed
Memories — 1 summary list (page 1 of >1, capped by the memory budget) + 1 diff-keyed search (0 hits); 0 lessons applied

Nothing to report — optimality (skipped), severity, 0 files skipped.

pr-reviewer · commit ba4ef2f · incremental review, delta since 64faf68 · how these findings are produced · updated Oct 11, 2026 7:51am UTC

Comment thread packages/mcp-core/src/telemetry/relevant-min-score-tracing.spec.ts Outdated
Comment thread supabase/functions/memories/handlers/relevant.ts Outdated
mthines and others added 2 commits October 11, 2026 07:39
…ithout pinning key order

Addresses the pr-reviewer bot's comment: #688 (comment)

Refs: #688
Co-authored-by: dash0-dev[bot] <257284812+dash0-dev[bot]@users.noreply.github.com>
…d of "the line above"

Addresses the pr-reviewer bot's comment: #688 (comment)

Refs: #688
Co-authored-by: dash0-dev[bot] <257284812+dash0-dev[bot]@users.noreply.github.com>
@dash0-dev
dash0-dev Bot marked this pull request as ready for review October 11, 2026 07:52
@dash0-dev
dash0-dev Bot requested a review from mthines as a code owner October 11, 2026 07:52
@dash0-dev

dash0-dev Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Darkplane auto-approval

Darkplane could not finish evaluating this pull request. It will try again on the next request.

@dash0-dev
dash0-dev Bot merged commit 5e413f8 into main Oct 11, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant