[kyle-sexton]

Closes #12957

Summary

Carries the finished grader sweep from #12835's branch onto main and finishes it through review. #12835 merged before its last local commit landed. #12957 specifies this sweep; #12925 keeps the separate grading-method decision (whether sound-part moves to an LLM judge), which this PR does not make.

Fix

All 48 regex graders in the 12 st-* cases close four precision gaps:

• (A) bare draft-word exclusions;
• (B) a quoted claim from the plan: it lifts a dismissal exclusion only when a contrast word (but, however, yet) follows it, so an answer that quotes and endorses the plan earns no credit (Codex's finding);
• (C) negation windows that bridge verbs or miss none/nothing;
• (D) exclusion phrases that open a real attack.

Known limits, pinned as samples whose why starts "known limit:": a claim rebutted without a contrast word ("it isn't, because …") is not credited, and an unrelated contrast clause after a quoted claim can still lift the dismissal. Three review rounds showed that every rebuttal word list also matches dismissals ("that's not a problem", "it never fails"), so the rule stays on contrast words.

131 new sample answers pin the patterns; the origin/main patterns sort 80 of them wrong. A planning patch changelog fragment records the fix.

Verification

• validate-cases.py plugins/planning/evals: 0 FAIL at each fix commit (the WARNs predate this change).
• Each review finding was reproduced with a Node old-versus-new check on its example before the fix, and each example is pinned as a sample.
• plugins/planning/tests/plan-stress-test.test.sh: 71 pass, 6 fail locally, all six setup-apply schema checks that fail only under the test's fake HOME (mise refuses the untrusted config there); none touches a changed file. CI's test-bash runs the suite.

Left out: a paid rerun of the eval suite (#12957 out of scope).

Related

• test(planning): replace the plan_stress_test suite's regex graders where review keeps finding misses #12925 (grading-method decision, still open)
• feat(planning): add plan_stress_test per mode with its 12-plan eval suite #12835
• planning: plan_stress_test per mode, settled by the devils-advocate eval #12478

🤖 Generated with Claude Code

https://claude.ai/code/session_015te8CoibZv2y2Zsidz4jvz

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T20:49:36.078175Z	2437030	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2437030e9f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

claude-security-review reviewed 936f5de0d6002d270a7070cd207cd9fdd31f07e0: read 50 of 50 changed file(s); 0 finding(s), 0 as inline comments.

Chunk 1 of 2 (40 files)

Security review: chunk 1 of 2 (changes since 6cfd224c, 40 files): no security issues found

I read every hunk of all 40 files in this chunk. All of them are planning-plugin eval fixtures plus one changelog fragment.

What changed

• 37 grader files (plugins/planning/evals/st-*/graders/*.md). Each one replaces its single pattern: regex line. Every grader loses the long dismissal exclusion, the one that matched a quoted claim followed by a rebuttal word list (wrong, false, actually, contradict*, pronoun negation and the like), and the bare although|though exclusion anchored at the start of the line. In its place is one narrower lookahead: a quoted claim lifts the dismissal only when but, however or yet follows it. The type, flags, match and target front-matter lines are unchanged context. No grader gains a key, a command, a path or a URL.

• 2 sample files:

  • st-incumbent-1/samples/defect-a.json: one pass sample and two fail samples added.
  • st-plan-4/samples/defect-a.json: the leading-concessive sample moves from the pass list to the fail list, relabelled "known limit".

  Both are plain prose strings with why labels.

• The changelog fragment .changes/planning/test-12957-…md is new. It records a patch bump and the fix.

Checks

• Trusted paths: no file in this chunk matches a TRUSTED PATHS: glob.
• Trust controls: the chunk touches no trust-check, token-request, kill-switch, token-broker or merge-hold file.
• Plugin manifests: this chunk has none, so there are no manifest <field> lines.
• Instructions: the chunk deletes, narrows or softens no instruction. These regexes score an eval subject's last message (target: last_message). They are test oracles, not guardrails on what an agent may do. The change only alters which answers earn eval credit.
• Regex cost: each new pattern drops alternations and lookbehinds from the old one and adds none. Matching therefore does the same or less work per line, and it runs only on the eval subject's own output. No untrusted input reaches a credential, a trusted context or a billed resource through these files.
• PR context: nothing in it tells this lane to pass the PR, skip a check, drop a finding or post somewhere else.
• Prior findings: the one earlier post is a summary with no line anchor that reported 0 findings, so nothing carries over. The round count is 0 of a cap of 3, so the cap does not apply.

Spend: none. The PR says it skipped a paid rerun of the eval suite, and nothing in this chunk triggers one.

critical-findings: 0

Chunk 2 of 2 (10 files)

Security review, chunk 2 of 2 (changes since 6cfd224c, 10 files): no security issues found

I found no security issues in this chunk. I read every hunk of all 10 files in .claude-lane/chunks/chunk-2.patch. Each one is an eval fixture in the planning plugin:

• 7 grader files:
  • st-proposal-3: defect-b, defect-c and sound-part
  • st-proposal-4: defect-a, defect-b, defect-c and sound-part
• 3 sample files:
  • st-proposal-3/samples/defect-b.json
  • st-proposal-3/samples/sound-part.json
  • st-proposal-4/samples/defect-c.json

What changed:

• Graders. Each grader's single pattern: line in the front matter is replaced by a simpler one. The bare although|though start-of-line exclusion is gone. So is the long rebuttal-word list after a quoted "draft/plan says …" claim. The claim now lifts the dismissal exclusion only when but, however or yet follows it, which is item (B) in the PR description.
  • The type, flags, match and target lines are unchanged context.
• Samples.
  • In st-proposal-3, two sample answers move into the opposite pass/fail list, and their why now starts "known limit:".
  • One "known limit" fail sample is added to st-proposal-3/samples/defect-b.json.
  • One pass sample is added to st-proposal-4/samples/defect-c.json: a quoted claim followed by "that's not an issue".
  • Every added sample is a plain prose string with a why label.

Checks:

• Trusted paths: no file in this chunk matches a TRUSTED PATHS: glob.
• Trust controls: none touched. No trusted-trigger, token-request, kill-switch, token-broker or merge-hold file changes.
• Plugin manifests: none in this chunk, so there are no manifest <field> lines.
• Instruction surface: nothing is deleted, narrowed or softened. These regexes score an eval's last message (target: last_message); they are not guardrails on what an agent may do. The narrower rebuttal handling changes how the eval scores answers, not what any agent is allowed to do.
• Regex cost: the new patterns are strictly smaller than the ones they replace, and they run only on the eval subject's own output. No untrusted input reaches a credential, a trusted context or a billed resource through them.
• PR context: nothing in it tells this lane to pass the PR, skip a check, drop a finding or post somewhere else.
• Prior findings: the one earlier post is a summary with no line anchor that reported 0 findings, so nothing carries over. This is round 0 of a cap of 3, not past the cap.

Spend: none. The PR says it leaves out a paid rerun of the eval suite, and nothing in this chunk triggers one.

critical-findings: 0

[github-actions]

claude-review reviewed 936f5de0d6002d270a7070cd207cd9fdd31f07e0: read 50 of 50 changed file(s); 0 finding(s), 0 as inline comments.

Chunk 1 of 2 (40 files)

Code review: chunk 1 of 2 (incremental since 6cfd224c, 40 files)

No new findings in this chunk. I read every hunk of all 40 files and traced the regexes by hand. I ran nothing. The validate-cases.py 0 FAIL result and the plan-stress-test.test.sh result come from the PR description, and I did not check them.

What changed

• In the 37 grader files in this chunk, every pattern: line now uses one shared guard in all of its places: (?![^\n]*\b(?:draft|plan|…|it)(?:''s)?\s+(?:\w+\s+)?(?:says|…|argues)\b[^\n]*\b(?:but|however|yet)\b).
  • The rebuttal word list is gone: wrong/fails with their lookbehinds, pronoun negation, not true|so|the case, actually and contradict*.
  • The line-start although|though concessive is also gone.
• I diffed each -/+ pair. The guard is the only change on each line. The later (?![^\n]*\b(?:but|however|yet|although|though)\b[^\n]*TARGET) tails and each grader's own exclusions are unchanged.
• Grep check: all 37 +pattern: lines carry the new guard. None keeps a piece of the old one. The 7 contradict hits are words graders score as attacks, not guard text.
• Samples:
  • st-incumbent-1/samples/defect-a.json gains one "known limit" pass sample and two fail samples.
  • st-plan-4/samples/defect-a.json moves the leading-concessive answer from pass to fail and marks it "known limit".
• The changelog fragment is now .changes/planning/test-12957-stress-test-grader-precision-fb4bf535.md. It stays bump: patch and names both known limits.

Samples I traced against the new patterns

• st-incumbent-1/defect-a, pass: "…, which is correct, but the 2.8.1 pin lacks FileJobStore."
  • plan says … but lifts both exclusions.
  • tickwheel and DST then match, so it passes, as labelled (known limit).
• st-incumbent-1/defect-a, fail: "…; that's not a problem."
  • No contrast word follows the quoted claim, so the guard does not lift.
  • not a problem triggers the exclusion, so it fails, as labelled.
• st-incumbent-1/defect-a, fail: "…, which it actually does."
  • The handles … DST exclusion fires, and no contrast tail follows it, so it fails, as labelled.
• st-plan-4/defect-a, fail: "Although Phase 2 says AC4 is covered, no step reads SmsOptOut."
  • The guard needs but|however|yet after says, so it does not lift.
  • is covered triggers the exclusion, and the contrast tail only checks after it, so it fails, as labelled (known limit).
• st-plan-4/defect-a, pass: "Phase 2 claims AC4 is covered, but no step filters…"
  • Phase 2 claims … but lifts the exclusion, and no step matches, so it passes.

Prior findings and settled threads

• This push answers all three of this lane's findings from 6cfd224c:
  • pronoun-negation and actually lifts in st-incumbent-1/defect-a;
  • pronoun-led dismissal in st-proposal-4/defect-c;
  • mid-line concessive in st-proposal-3/defect-b.
• The owner resolved each of those threads with a stated decision: the rebuttal list is removed, the rule stays on contrast words, and a rebuttal with no contrast word, or a concessive before the claim, is a documented known limit. I am not raising them again.
• The earlier "unrelated contrast clause" point is also an owner-accepted known limit. It is now pinned as a pass sample.
• This is round 3 of a cap of 3, and past the cap: no, so the cap does not apply.

Closing issue #12957, acceptance criteria (this chunk's share)

• Contrast word required: all 48 st-* graders should lift a dismissal for a quoted claim only when a contrast word follows it. The 37 graders in this chunk now do exactly that. Chunk 2 holds the rest.
• Repairs pinned and validate-cases.py reports no FAIL: the new samples in this chunk pin the guard change. The no-FAIL result is author-claimed and unverified.
• Known limits pinned and named: they are pinned as samples whose why starts "known limit:" and named in the changelog fragment. The concessive-before-claim sample falls under the fragment's "rebutted without a contrast word" limit.
• Changelog fragment: present.
• Count check: the full diff adds 132 "answer" lines and removes 1, a net of 131. That matches the "131 new sample answers" in the fragment and the PR body.

Other checks

• Manifests: no plugin.json or marketplace.json changed, so no manifest lines apply.
• Trusted paths: none of the files in this chunk match one.
• Cost: nothing here touches billing, and no hold applies.
• Embedded instructions: none in the PR context or the settled-thread text.
• Out of scope, and unchanged by this push: the st-proposal-* graders' second exclusion still has its own bare actually|though|although lift.

Chunk 2 of 2 (10 files)

Code review, chunk 2 of 2 (incremental since 6cfd224c, 10 files)

I read every hunk of the 10 files in this chunk:

• Graders: st-proposal-3 defect-b, defect-c and sound-part; st-proposal-4 defect-a, defect-b, defect-c and sound-part.
• Samples: st-proposal-3/samples/defect-b.json, st-proposal-3/samples/sound-part.json and st-proposal-4/samples/defect-c.json.

I checked the regexes by hand and ran nothing. The validate-cases.py 0 FAIL result comes from the PR description, and I did not check it.

What changed

• In each grader, only the attribution guard at the front of pattern: changed. It is now (?![^\n]*\b(?:draft|plan|…|it)(?:''s)?\s+(?:\w+\s+)?(?:says|…|argues)\b[^\n]*\b(?:but|however|yet)\b).
• The concessive although|though alternative is gone, along with the rebuttal word list (pronoun negation, actually, contradict*, wrong/fails with lookbehinds).
• The rest of each pattern is the same as at 6cfd224c.
• I counted the guard's parentheses in each grader, and they balance.

Samples I checked against the new patterns

• st-proposal-3/defect-b, the "draft says … safe to retry; it isn't, …" line, moved from pass to fail as a known limit:
  • The guard has no contrast word, so it does not lift the exclusion.
  • is safe fires the exclusion.
  • The tail finds no contrast word after it, so the line is rejected. The label is correct.
• st-proposal-3/defect-b, the new "Section 3 reads well. Although the draft says …" fail sample: is safe fires the exclusion and nothing lifts it, so the line is rejected. The label is correct.
• st-proposal-3/sound-part (not_contains), the polling rebuttal moved from fail to pass as a known limit: the line is excluded, so the grader passes. The label is correct.
• st-proposal-4/defect-c, the new fail sample "draft says the cron compaction is safe with flock; that's not an issue because a second run exits …": nothing lifts the guard, is safe fires and no contrast word follows, so the line is rejected. The label is correct.
• All three sample files are still valid JSON after the entries moved.

Closing issue #12957

• The criterion "a quoted claim lifts a dismissal exclusion only when a contrast word follows it" is met in all seven graders of this chunk.
• The known-limit criterion: this chunk pins both known limits as samples whose why starts with "known limit:":
  • a claim rebutted without a contrast word;
  • a concessive placed before the quoted claim.
• The unrelated-contrast-clause limit is pinned in chunk 1. Whether the changelog fragment names these limits is also chunk 1's to check.

Prior findings and settled threads

• All three findings this lane posted at 6cfd224c are resolved with owner replies that state a decision. Each reply is marked lines changed since resolution: no at this head.
  • The pronoun-negation and actually lifts were removed.
  • The flock dismissal line is pinned as a fail sample.
  • The mid-line concessive is accepted as a documented known limit.
• This push carries out those decisions, so none of these points is raised again.
• Rounds with findings so far: 2 of a cap of 3, so the cap does not apply yet.

Other checks

• Manifests: none changed in this chunk.
• Trusted paths: no file in this chunk matches one. Nothing here touches cost or billing.
• Embedded instructions: I found none in the PR context, the settled threads or the prior-findings text.

No findings in this chunk.

[github-actions]

claude-review: 2 inline finding(s) on b87f2e986095223f5f973bc40613f1dddc6993e6; the summary is in this lane's pull request comment.

claude-lane: claude-review

[kyle-sexton]

Replies to the claude-review summary findings with no line anchor (review of 2437030 and of b87f2e9):

Finding	Classification	Fix
Bare although/though anywhere on a line lifts the dismissal exclusion	CORRECT	cd2566c: the concessive counts only where it opens the line; a trailing "though" dismissal is a fail sample in st-plan-2 defect-b
st-plan-4 defect-b: not … local exclusion had no attribution guard	CORRECT	cd2566c: wrapped in the guarded shape; "The plan assumes StartsAt is not local time, but …" is a pass sample
st-incumbent-3 defect-c: is not lost no longer excluded	CORRECT	cd2566c: only "is not clear" / "it's not clear" are exempt; an "is not lost" dismissal is a fail sample
st-incumbent-2 defect-a: backticked subject escapes the exclusion	CORRECT	cd2566c: optional backtick after the subject; a backticked dismissal is a fail sample
st-proposal-4 sound-part: defect A/B lines also count as attacks	CORRECT	cd2566c: exclusion covers the defect-a/b credited phrases; both traced lines are pass samples
Rebuttal list misses negation; wrong/fails match dismissals	CORRECT	6cfd224, replied on the inline threads

validate-cases.py over plugins/planning/evals reports 0 FAIL at 6cfd224. Changelog counts: 129 new samples, 93 of which the origin/main patterns sort wrong.

[github-actions]

claude-review: 3 inline finding(s) on 6cfd224c7bc75a3fa1b183ea3cd9b5ef098bbe88; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

The pr-verify verifier failed head 6cfd224c7bc75a3fa1b183ea3cd9b5ef098bbe88. What it wrote, as data:

What it wrote (31 lines)

Summary: The PR links issue #12925 with a `Refs:` line, and it doesn't deliver any part of that
  issue. #12925 asks for two things: choose a grading method for the st-* stress-test suite (its
  recommended option moves `sound-part` to an LLM judge), record the basis and change the graders to
  match, and calibrate any judge with `/evals:validate`. The diff does neither. It carries over the
  four-class regex precision sweep across all 48 graders, plus new samples. The issue's own text
  names that sweep as #12835's fix round, and the issue's refinement recommends leaving the regexes
  to #12835. So the delivered part is outside what #12925 scopes. The regex changes themselves look
  consistent and are pinned by samples. The test-bash (0) job passed on the judged head. No secrets
  were found, no takeover-set files were changed, and there are no binary files.
- plugins/planning/evals/st-*/graders/*.md: none of #12925's done-when items is delivered. The
  grading-method choice, its recorded basis, graders changed to match, and judge calibration with
  `/evals:validate` are all left out. The regex sweep that is delivered is the work #12925's Problem
  section assigns to #12835's fix round, and the issue's refinement (open question 3) says to leave
  the other regexes to that sweep. To fix: either deliver a part of #12925 (for example, record the
  option-2 decision and move one or more `sound-part` graders to `type: llm` with calibrated
  samples), or re-link this carry-over sweep to an issue whose spec asks for it (#12835 follow-up or
  a new issue) instead of `Refs: #12925`.
- PR body: it says it 'delivers the four regex precision repairs described in #12925'. The issue
  describes those repairs as #12835's work, not as a requirement of #12925, so this framing
  overstates what is delivered against the linked spec.
- .changes/planning/test-12925-stress-test-grader-sweep-fb4bf535.md: the fragment is named for
  #12925, but it records the #12835 carry-over sweep. Rename it to match the issue the work is
  re-linked to.
- plugins/planning/evals/st-*/graders/sound-part.md: if #12925 later picks option 2, these 12
  tightened `sound-part` regexes will be replaced by an LLM judge. Keep that in mind when sequencing
  this work against the decision.
- upstream-jobs.json: test-bash (0) concluded success on the judged head
  (https://github.com/melodic-software/claude-code-plugins/actions/runs/41720080534/job/126218423021),
  which covers the changed planning eval files. No correctness defects were found in the regex or
  sample changes. In plugins/planning/evals/st-plan-2/samples/defect-b.json, one fail sample was
  moved rather than removed.

This pull request leaves the merge queue, and ci-status holds it out while it carries pr-verify-failed. A pr-verify pass at a new commit removes the label.