Repository navigation
test(planning): close the regex precision gaps in the st-* stress-test graders #12957
Description
Problem
#12835 (the #12478 eval) merged before its last fix round landed. That round tightens the 48 regex graders in the 12 st-* stress-test cases under plugins/planning/evals, which mis-sort answers in four ways:
- (A) a bare draft word (
draft,plan) in a line trips a dismissal exclusion; - (B) a quoted claim from the plan that the answer then disputes is graded as the answer's own dismissal;
- (C) negation windows bridge into the next verb, or miss
none/nothing; - (D) exclusion phrases broad enough to also swallow a real attack on the sound part.
#12925 keeps the separate question of moving sound-part (or the defect checks) to an LLM judge; this issue covers only the regex repairs, which #12925's Problem section assigns to #12835's fix round.
Done when
- All 48
st-*regex graders carry repairs for (A)-(D), and a quoted claim lifts a dismissal exclusion only when a contrast word follows it, so an answer that quotes and endorses the plan earns no credit. - Every repaired gap is pinned by
samples/<grader>.jsonanswers, andpython3 plugins/evals/skills/validate/scripts/validate-cases.py plugins/planning/evalsreports no FAIL. - Known limits the regex cannot separate (a claim rebutted with no contrast word, an unrelated contrast clause) are pinned as samples whose
whystarts "known limit:" and named in the changelog fragment. - A planning changelog fragment records the fix.
Out of scope
- test(planning): replace the plan_stress_test suite's regex graders where review keeps finding misses #12925's grading-method decision and any LLM judge grader.
- A paid rerun of the eval suite.
Filed by lane 2 from #12947's pr-verify finding: the sweep needs its own spec instead of Refs: #12925.
Metadata
Metadata
Assignees
Type
Fields
Work status
ready
Activity
chore(planning): name the grader sweep fragment for #12957
kyle-sexton commentedon Nov 3, 2026
github-actions commentedon Nov 3, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Understood: repair the 48 regex graders in the 12
st-*stress-test cases underplugins/planning/evalsfor gaps (A)-(D), pin each repair with samples, record known limits, and add a planning changelog fragment. The grading-method question and any LLM judge stay in #12925, which this issue does not duplicate. The done-when items are clear, so nothing is missing.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 3, 2026
Not admitted to the issue lane yet: open pull request #12947 is for it.