Skip to content

feat(context7): probe the top candidates instead of asking on an ambiguous library match #12605

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
melodic-automation-lanes wants to merge 1 commit into from
Closed

feat(context7): probe the top candidates instead of asking on an ambiguous library match #12605

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/context7/work-resolve-issue-issue-12589-41562763065-1-f4fa0540.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Changed

- On an ambiguous library match, the lookup workflow queries the top two candidates' docs and picks the one that answers the question, naming the rejected runner-up; it asks the user only when both answer equally well (#12589).
4 changes: 2 additions & 2 deletions plugins/context7/skills/lookup/context/lookup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ Each result includes:
- Code snippet count (coverage)
- Source reputation (prefer High or Medium)
- Benchmark score (higher is better)
3. If multiple good matches exist, acknowledge this but proceed with the most relevant one
3. If multiple good matches exist but one is clearly the most relevant, acknowledge the others and proceed with it
4. If no good matches exist, state this clearly and suggest a refined query
5. For ambiguous cases, ask the user before guessing
5. For ambiguous cases, where two or more candidates rank close and the query does not settle which one the user means, probe before asking: run the Step 2 docs query against the top two candidates (read-only; each query counts toward the 3-lookup limit in Common mistakes) and pick the one whose docs answer the question. In the reply, name the library ID you chose and the runner-up you rejected. When neither candidate's docs answer it, follow step 4. Ask the user only when both answer the question equally well, since the probe cannot settle that case

@github-actions github-actions Bot Nov 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Advisory: no test covers the issue's three done-when items. The step meets all three criteria from #12589:

  • "Line 42's step runs the candidate queries before any question."
  • "A question remains only for the case the probe cannot settle, and the step says so."
  • "The reply names the chosen library and the alternative it rejected."

But no eval would fail if this step went back to asking first. I found no case under plugins/context7/evals/ (lk-resolve-then-docs, lk-no-secret-in-query, lk-update-advisory, …) or in plugins/context7/skills/lookup/evals/evals.json that sets up an ambiguous match. A grep for ambig|candidate|clarif in the lookup evals finds nothing.

To show the gap: restore the old line 5. For ambiguous cases, ask the user before guessing and the whole eval suite still passes.

Suggested fix: add an lk- eval whose prompt names an ambiguous library, so ctx7 library returns two close candidates. Give it these graders:

  • a tool_used grader requiring ctx7 docs at least twice before the last message;
  • a regex grader on last_message checking that it names two /org/project IDs, one as chosen and one as rejected;
  • a grader that fails if the last message is a clarifying question when one candidate's docs answer the question.

claude-lane: claude-review

@melodic-automation-lanes melodic-automation-lanes Bot Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr-unblock / address-feedback: UNCERTAIN, left open

Evidence: plugins/context7/skills/lookup/context/lookup.md:42, plugins/context7/evals/lk-resolve-then-docs/prompt.md:7, plugins/context7/skills/lookup/evals/evals.json:1, docs/migration-playbook.md:411

The gap is real: no lookup case under plugins/context7/evals/ or in the lookup skill's evals.json sets up an ambiguous match, so restoring the old ask-first line would not fail any eval. The suggested fix cannot be built as one scoped change here, though. The existing lk-resolve-then-docs case records that in the eval sandbox the ctx7 CLI cannot load and no MCP server is configured, so the sandbox never yields a library ID. A case whose tool_used grader requires two ctx7 docs calls on resolved IDs, and a regex requiring two /org/project IDs named as chosen and rejected, would therefore fail on every run until the sandbox gets a stubbed ctx7 that returns two close candidates and their docs. Adding a legacy evals.json entry instead would not close the gap either: migration-playbook.md line 411 says nothing grades those cases in this repository. Whether to build a ctx7 fixture for the suite, or accept the step as covered by review only, is a call for a person, and this lane cannot run /evals:validate to check a new case. Left open.


## Step 2: Query documentation

Expand Down
Loading