[kyle-sexton]

Closes #12589

Summary

Replaces #12605. Its pr-unblock address-feedback run stopped with feedback-unadjudicated and labeled it needs-session. That left review thread PRRT_kwDOTCGFQM6y9FDB open. The thread notes that no eval covers the three done-when items of #12589. The lane agreed the gap is real but did not add a case. It reasoned that the eval sandbox cannot load ctx7, so a probe case could never get library IDs. This PR carries #12605's change unchanged and adds the missing cases. Each prompt includes the ctx7 output, so neither case needs a working ctx7.

Fix

• plugins/context7/skills/lookup/context/lookup.md step 5 and the changelog fragment come from feat(context7): probe the top candidates instead of asking on an ambiguous library match #12605 without changes. On an ambiguous match, the lookup now probes the top two candidates, picks the one whose docs answer the question, and names the rejected runner-up.
• New eval lk-ambiguous-probes-candidates. The prompt pastes a ctx7 library result with two close candidates. Two tool_used graders, one per candidate ID, require a ctx7 docs call with each ID. A tool_used grader matches the call's input, so a call counts even when the CLI fails in the sandbox.
• New eval lk-ambiguous-names-choice. The prompt pastes the library result and both probe outputs. Only one candidate's docs answer the question. Three graders:
  • a regex that requires both IDs in the reply;
  • a not_contains regex for a "which library do you mean" question;
  • an llm grader that checks the reply chose the library that answers and presents the other as rejected.
• New eval lk-ambiguous-tie-asks. Both pasted probe outputs document a retry option. One regex requires both IDs in the reply, and a second requires a question asking which library the user means. This covers the tie-case clause of the second done-when item.
• A run that asks before probing, as the old line 5. For ambiguous cases, ask the user before guessing directed, fails the first case's probe graders and the second case's no-question grader.

Verification

• python3 plugins/evals/skills/validate/scripts/validate-cases.py plugins/context7/evals: 0 FAIL. Every deterministic grader passes its must-pass samples and rejects its must-fail samples, including the old ask-first reply. The new cases raise one WARN each: allowed_tools lists Bash. Every existing context7 case raises the same WARN. Bash stays because the lookup SKILL.md has a pre-computed ctx7 --version line, and without Bash that line does not load.
• markdownlint-cli2 on both new prompt.md files and lookup.md: 0 issues.

Not verified

• No paid claude plugin eval run. One risk in lk-ambiguous-probes-candidates: the model could rerun ctx7 library, see it crash, and stop before it probes. Only a real run would show whether that happens.
• The vendored plugins/context7/skills/lookup/vendor/find-docs/SKILL.md still says to request clarification. It is upstream content and was left unchanged, as in feat(context7): probe the top candidates instead of asking on an ambiguous library match #12605.

Related

• feat(context7): probe the top candidates instead of asking on an ambiguous library match #12605 (replaced; held with do-not-merge)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vcupom3BKF6QuC5U3sRJPj

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T04:01:04.326812Z	322bc1d	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 322bc1d861d93c934b72e74d7608c411abd2622d: read 10 of 10 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12652 (head 322bc1d)

No security issues found. I read the whole diff: all 10 listed files and every hunk. None of them is on a trusted path.

What changed

• plugins/context7/skills/lookup/context/lookup.md, step 5: on an ambiguous library match, the lookup now runs the read-only docs query against the top two candidates instead of asking the user first. It still counts toward the existing 3-lookup limit.
• A changelog fragment.
• Two new eval cases, lk-ambiguous-probes-candidates and lk-ambiguous-names-choice, with their grader sample files.

Instruction-surface lens. The diff softens a standing instruction: it removes 5. For ambiguous cases, ask the user before guessing. I checked that removal against the protected classes in docs/contracts/instruction-exception-register/README.md. Those are Gate 0's six classes: irreversible-action, secret-handling, data-integrity, external-publication, legal-compliance and agent-authority.

• The removed rule guarded which library's docs to read, not any of those actions.
• The replacement action is a read-only ctx7 docs query, capped by the existing 3-lookup limit.
• The query text already went to Context7 under the old flow (step 2 runs it on the chosen library). Probing a second library ID sends no new kind of data outside.
• No protected class matches, so this is not a finding.

Evals. Each new case sets allowed_tools: [Bash, Read], which matches the existing context7 cases. The prompts hold only made-up library output. The graders are regex, tool_used and LLM checks of the model's own transcript. No secrets, tokens, workflow, permission or trust-control surface is touched.

Plugin manifests. No plugin.json or marketplace.json change.

PR context. The PR description has no text addressed to the review lane.

Spend. The two new eval cases (max_turns: 10, timeout_seconds: 300) only use prepaid model credit when a paid claude plugin eval run happens.

critical-findings: 0

[github-actions]

claude-review: 1 inline finding(s) on 322bc1d861d93c934b72e74d7608c411abd2622d; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 322bc1d861d93c934b72e74d7608c411abd2622d: read 10 of 10 changed file(s); 1 finding(s), 1 as inline comments.

Code review: PR #12652, context7 lookup probes the top candidates on an ambiguous match

Scope: I read every hunk of all 10 changed files: the changelog fragment, the step 5 edit in lookup.md, and the two new eval cases with their prompts and grader samples. I checked them against the path-scoped rules .claude/rules/eval-case-transcripts.md and .claude/rules/skill-bodies-state-current-rules.md. Neither rule is broken: the grader samples are synthetic, and step 5 states our own decision without restating any upstream specific.

Closing issue #12589, Done when:

• "Line 42's step runs the candidate queries before any question." Met, at lookup.md:42. lk-ambiguous-probes-candidates covers it: one tool_used grader per candidate ID.
• "A question remains only for the case the probe cannot settle, and the step says so." The step says so. no-which-library-question tests the "only" half (no question when the probe settles it). Nothing tests the other half, that a question still comes when both candidates answer equally well. See the inline comment.
• "The reply names the chosen library and the alternative it rejected." Met. names-both-ids and the chose-the-answering-library llm grader in lk-ambiguous-names-choice cover it.

Notes, not findings:

• The new input_match patterns reuse the shell-prefix grammar the existing context7 cases use (lk-resolve-then-docs, ck-read-only), and the sample sets cover the echo and description lookalikes. The ID-boundary lookaheads in names-both-ids reject the -cache and .js-cache lookalikes as the samples intend. I checked the regexes by reading them. I did not run validate-cases.py, so the PR's validation result (0 FAIL) is author-claimed and unverified.
• The not_contains regex misses some clarifying forms such as "Should I use X or Y?". The llm grader's FAIL clause still catches a reply that asks instead of answering, so the case as a whole still fails that reply.
• The vendored find-docs/SKILL.md still says to ask for clarification. SKILL.md says to read vendor/ only during update, never during a lookup, so it does not conflict at run time.
• Spend: the two new eval cases spend API or subscription usage whenever the context7 eval suite runs (10 turns, 300 s each). This is not a hold.

No plugin manifest changed, so the manifest checks do not apply. The PR context file contains no embedded instructions.

[github-actions]

claude-review reviewed 31a6479ab783d57745ff534414530d1708876642: read 14 of 14 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #12652, context7 lookup probes the top candidates on an ambiguous match

No findings.

What I read: every hunk of all 14 changed files. That covers the changelog fragment, the step 5 edit in lookup.md, the three new eval cases (lk-ambiguous-probes-candidates, lk-ambiguous-names-choice, lk-ambiguous-tie-asks), and their prompts and grader samples. One path-scoped rule matches the eval files: .claude/rules/eval-case-transcripts.md. None of them breaks it. The prompts and samples are synthetic, with no session transcript, machine path or identifying detail.

Last round's finding: the last round had one advisory finding: the tie-case clause had no test that could fail (thread). This head fixes it with the new lk-ambiguous-tie-asks case. Both pasted ctx7 docs outputs document retry with exponential backoff, and two graders check the reply:

• names-both-ids requires both IDs in the reply.
• asks-which-library is a contains regex that requires a which-library question.

If the "Ask the user only when both answer the question equally well" sentence is deleted from step 5, this case fails. The asks-which-library fail samples include a reply that picks one library on the tie. The owner's reply settles the thread, and I am not raising it again.

Closing issue #12589, Done when:

• "Line 42's step runs the candidate queries before any question." Met at lookup.md:42. lk-ambiguous-probes-candidates covers it with one tool_used grader per candidate ID.
• "A question remains only for the case the probe cannot settle, and the step says so." Met. Both halves are tested:
  • "Only": no-which-library-question in lk-ambiguous-names-choice fails a reply that asks after the probe settled the choice.
  • "Remains": asks-which-library in lk-ambiguous-tie-asks fails a reply that picks one on a tie.
• "The reply names the chosen library and the alternative it rejected." Met. names-both-ids and the chose-the-answering-library llm grader in lk-ambiguous-names-choice cover it.

Notes (not counted as findings):

• asks-which-library reuses the exact pattern of no-which-library-question. The pair is consistent: one case requires the question, the other forbids it. The tie case has no llm grader, so a reply that picks one library and then asks "Which one are you using?" would still pass. That reply is unlikely, so I did not count it.
• I checked the regexes against their samples by reading them only. I did not run validate-cases.py, so its "0 FAIL" result is the author's claim and I have not confirmed it.
• Spend: the three new eval cases use API or subscription credit whenever the context7 eval suite runs (up to 10 turns and 300 s each). This is not a hold.

No plugin manifest changed, so the manifest checks don't apply. The PR context file and the settled-threads file contain no embedded instructions.

[github-actions]

claude-security-review reviewed 31a6479ab783d57745ff534414530d1708876642: read 14 of 14 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12652 (head 31a6479)

I found no security issues. I read the whole diff: all 14 listed files and every hunk in them. None of them matches a trusted path.

Since the last review (322bc1d). The only new content is the eval case plugins/context7/evals/lk-ambiguous-tie-asks/: case.yaml, prompt.md and two grader sample files. It answers the settled advisory thread on lookup.md line 42. The prompt holds only made-up ctx7 output. The graders are two regexes over the model's last message. allowed_tools: [Bash, Read] is the same as in the other context7 cases. It touches no secrets, tokens, workflows, permissions or trust controls.

Instruction-surface lens. In plugins/context7/skills/lookup/context/lookup.md, step 5's ask the user before guessing is replaced by a read-only probe of the top two candidates. I checked that against the Gate 0 classes in docs/contracts/instruction-exception-register/README.md:

• The six classes are irreversible-action, secret-handling, data-integrity, external-publication, legal-compliance and agent-authority.
• The removed rule decided which library's docs to read. It guarded none of those classes.
• The probe is a read-only ctx7 docs query, capped by the existing 3-lookup limit.
• The probe sends the same query text the old flow already sent to Context7.
• The rule still asks the user on a tie, and the new lk-ambiguous-tie-asks case now tests that.

No class matches, so this is not a finding.

Plugin manifests. The diff changes no plugin.json or marketplace.json.

Settled threads. One owner-settled thread on lookup.md line 42 (lines changed since resolution: no) was an advisory about test coverage, not a security point. I am not raising it again.

Prior findings. This lane's earlier post had no findings, so nothing is repeated. The round cap did not apply.

PR context. The description has no text addressed to the review lane.

Spend. The three new eval cases (max_turns: 10, timeout_seconds: 300 each) use prepaid model credit only when someone runs a paid claude plugin eval.

critical-findings: 0