Repository navigation
feat(context7): probe the top candidates instead of asking on an ambiguous library match - #12605
melodic-automation-lanes[bot] wants to merge 1 commit into
Conversation
Lane-Activity: work-resolve-issue/resolve-issue
|
claude-security-review reviewed Security review: no security issues foundcritical-findings: 0 This PR changes two markdown files and no code, workflows, manifests or trust controls:
None of the changed files matches a trusted path. Instruction-surface lens. Old step 5 said "For ambiguous cases, ask the user before guessing." This PR softens that rule, so I checked the removed rule against Embedded text. The PR context contains no directives aimed at this lane. Spend. None. At most one extra Context7 docs query per ambiguous lookup, still within the existing 3-lookup cap. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| 3. If multiple good matches exist but one is clearly the most relevant, acknowledge the others and proceed with it | ||
| 4. If no good matches exist, state this clearly and suggest a refined query | ||
| 5. For ambiguous cases, ask the user before guessing | ||
| 5. For ambiguous cases, where two or more candidates rank close and the query does not settle which one the user means, probe before asking: run the Step 2 docs query against the top two candidates (read-only; each query counts toward the 3-lookup limit in Common mistakes) and pick the one whose docs answer the question. In the reply, name the library ID you chose and the runner-up you rejected. When neither candidate's docs answer it, follow step 4. Ask the user only when both answer the question equally well, since the probe cannot settle that case |
There was a problem hiding this comment.
Advisory: no test covers the issue's three done-when items. The step meets all three criteria from #12589:
- "Line 42's step runs the candidate queries before any question."
- "A question remains only for the case the probe cannot settle, and the step says so."
- "The reply names the chosen library and the alternative it rejected."
But no eval would fail if this step went back to asking first. I found no case under plugins/context7/evals/ (lk-resolve-then-docs, lk-no-secret-in-query, lk-update-advisory, …) or in plugins/context7/skills/lookup/evals/evals.json that sets up an ambiguous match. A grep for ambig|candidate|clarif in the lookup evals finds nothing.
To show the gap: restore the old line 5. For ambiguous cases, ask the user before guessing and the whole eval suite still passes.
Suggested fix: add an lk- eval whose prompt names an ambiguous library, so ctx7 library returns two close candidates. Give it these graders:
- a
tool_usedgrader requiringctx7 docsat least twice before the last message; - a
regexgrader onlast_messagechecking that it names two/org/projectIDs, one as chosen and one as rejected; - a grader that fails if the last message is a clarifying question when one candidate's docs answer the question.
claude-lane: claude-review
There was a problem hiding this comment.
pr-unblock / address-feedback: UNCERTAIN, left open
Evidence: plugins/context7/skills/lookup/context/lookup.md:42, plugins/context7/evals/lk-resolve-then-docs/prompt.md:7, plugins/context7/skills/lookup/evals/evals.json:1, docs/migration-playbook.md:411
The gap is real: no lookup case under plugins/context7/evals/ or in the lookup skill's evals.json sets up an ambiguous match, so restoring the old ask-first line would not fail any eval. The suggested fix cannot be built as one scoped change here, though. The existing lk-resolve-then-docs case records that in the eval sandbox the ctx7 CLI cannot load and no MCP server is configured, so the sandbox never yields a library ID. A case whose tool_used grader requires two ctx7 docs calls on resolved IDs, and a regex requiring two /org/project IDs named as chosen and rejected, would therefore fail on every run until the sandbox gets a stubbed ctx7 that returns two close candidates and their docs. Adding a legacy evals.json entry instead would not close the gap either: migration-playbook.md line 411 says nothing grades those cases in this repository. Whether to build a ctx7 fixture for the suite, or accept the step as covered by review only, is a call for a person, and this lane cannot run /evals:validate to check a new case. Left open.
|
claude-review reviewed Code review: PR #12605 (context7 lookup: probe before asking on an ambiguous match)I read every hunk of both changed files: What the change does. On an ambiguous library match, step 5 of the selection process now runs the Step 2 docs query against the top two candidates. It picks the one whose docs answer the question and names both the chosen library ID and the rejected runner-up in the reply. If neither candidate answers, it sends the case to step 4. It asks the user only when both answer equally well. Step 3 now covers only the case where one match is clearly the most relevant, so it no longer overlaps step 5. The step also says the two probe queries count toward the existing 3-lookup limit in Common mistakes: one resolve call plus two docs queries makes three, so the limit holds. Closing issue #12589, done-when items.
None of the three has a test. No eval under Rules and other checks.
Spend. The skill now makes up to two extra Context7 docs queries per ambiguous lookup, within the existing 3-lookup limit. That draws only on Context7 quota or subscription usage, so it is not a hold. |
|
Unattended merge: native auto-merge leaves this pull request unarmed at head |
|
The unattended merge path armed native auto-merge at head |
|
pr-unblock / address-feedback finished round 1 of max 4 on ab23934. |
melodic-automation-lanes
Bot
added
the
needs-session
|
The pr-unblock lane stopped
When a retry on this head can change the outcome, the lane runs one on its own, up to its per-head cap, and edits this comment if it stops again; a run that does not stop clears the status. When no retry can (a review thread the lane does not take or settle, or a failing check after the retries are spent), a person resolves the threads or gets the checks green, then dispatches the workflow below: that run finds the cause gone and clears the status. On a pull request the lanes App opened, the dispatchers start that run once per stop; on any other, a person does. A new commit to the branch starts clean. To run it again on this head now, dispatch the gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=12605 -f activity=address-feedback |
|
Superseded by #12652, which carries this branch's head plus the eval fix the lane could not take. Closed so the open-PR count reflects live work. |
Closes #12589
Summary
Opened by the work-resolve-issue lane. The run's own summary:
Not verified
Fix
Verification
The runner ran each check from the base copy against ab23934:
The run reports:
Left out: the items under Not verified.
Related
#12589