Repository navigation
feat(planning): add plan_stress_test per mode with its 12-plan eval suite - #12835
Conversation
… not yet measured Co-Authored-By: Claude Opus 5.5 <[email protected]>
…s-test # Conflicts: # plugins/planning/reference/config.md # plugins/planning/schemas/planning.schema.json
- comment-hygiene: the seed scripts write each phase's [TODO] status tag after the plan heredoc, so the scan no longer reads the markdown headings as comments; those eight seeded workspaces are byte-identical. - typos: American spellings in prose and samples; st-proposal-4 names libflate and a harbor station; grader regex stems take a \b at sites that already sit on a word boundary, and re-?runn is spelled out, so every grader sorts every sample as before. - shell-portability: the stress-test test matches with ' +', not \s. - exec-bit: the stress-test test is executable. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…efault The 144-run eval kept every mode at auto: its negative control detected no loss, so the suite cannot measure what the stress test adds. Closes #12478 Co-Authored-By: Claude Opus 5.5 <[email protected]>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6734fd56e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
claude-security-review reviewed Security review: incremental (PR #12835,
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
claude-review reviewed Code review: incremental (
|
…y fixture editorconfig-checker failed 64 lines in the st-incumbent-3, st-plan-2 and st-proposal-2 seeds: their Go heredocs were tab-indented, and the repo .editorconfig sets indent_style = space for *.sh. Leading tabs are now four spaces each; the diff is whitespace-only and Go accepts either indent. The orphaned-fixtures check found no grader naming devils-advocate-degraded-body.md.txt. The stress test now asserts the negative control's SKILL.md is the source frontmatter followed by that file. Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Point the re-plan and panel paths at Step 3's plan_stress_test resolution instead of always dispatching plan-reviewer. - Classify plan_stress_test modes by the draft's main decision, so proposal is reachable at Step 3; state that a direct /planning:devils-advocate run ignores the key. - Narrow the st-incumbent-3 and st-plan-3 sound-part graders so correct neighboring findings do not match, with samples for each case. - Make st-plan-4's sound part sound: Phase 3 retries only a refused send within the same tick and never picks up a row from an earlier tick. - Assert the schema defaults every mode to auto and that plan Step 3 resolves the key through config.md "Convention-home keys". Co-Authored-By: Claude Opus 5.5 <[email protected]>
…st test Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Reply to claude-review's summary findings on 6734fd5. Both are fixed in a8296c2:
|
st-incumbent-3: `unsafe` and `wrong` now count only in the same clause as the retry, replay, ON CONFLICT or insertSample claim, and not on a line naming the buffer, ring, watchDatabase or process exit, so a correct buffer-size or process-exit report no longer fails the grader. st-proposal-1: the exclusion drops bare start, window, AC1, AC2, insert and retry words, and keys on the three defects' own markers, so an attack on SQLite that says "start" still counts. Each grader gains samples for both directions. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…s-test # Conflicts: # plugins/planning/schemas/planning.schema.json
…correct lines - st-incumbent-4 sound-part: the negation allows four words and "nothing", and wrong/unverified count only in the same clause as the 4 KB target. - st-plan-2 defect-b: the crash exclusion drops only "no crash risk" style dismissals, not "no crash safety". - st-plan-4 defect-a and defect-c: a line that quotes the draft's claim and then refutes it with but/however is no longer dropped. - st-plan-4 defect-b: "does not convert local time" is no longer read as "is not local". - st-proposal-3 and st-proposal-4 sound-part: the context exclusions no longer skip attacks that use the draft's own words (queued, claim, segments, layout); "five seconds" now reaches the attack set. Each fix adds the reviewer's line as a sample that pins it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Closes #12478
Summary
Adds the
plan_stress_testsetting to the planning plugin (per draft mode:plan,proposal,incumbent; valuesauto,reviewer,devils-advocate,both; defaultauto) and a 12-case eval suite (plugins/planning/evals/st-*, tagplan-stress-test) that measures which value each mode should default to. The measured pass keptautoin every mode: the negative control detected no loss, so the suite cannot measure what the stress test adds.reference/config.mdcites the Probe result.Fix
plan_stress_testinplugins/planning/schemas/planning.schema.json,reference/config.mdand/planning:plan, beside main'swayfind_suggest_questions.sound-partmust-not-flag check, graded by deterministic regex graders onlast_message.evals/fixtures/stress-arm/prepare-arm.shbuilds one plugin copy per arm, plus the degraded-body negative control.tests/plan-stress-test.test.shproves no detection grader passes on a workspace's own text.Pre-registration (recorded before any paid run; not amended after a number is read)
Frozen suite:
56f217b92f. Nothing underplugins/planning/evals/changes after the first batch run.Arms. One plugin copy per arm, with
plan_stress_testgiving all three modes the same value. The prompt is byte-identical across arms.reviewerplan-revieweronlydevils-advocate/planning:devils-advocatein the draft's mode onlybothArms: owner decision, 2026-11-03 in the lane 2 session, choosing A / B / C over a no-review baseline and over a fourth no-review arm. #7586 Q165 left the arms undefined, and Q166 keeps
plan-revieweron, so the three arms are the settings the default chooses between.Numbers. Each run is a
claude plugin evalrun with--ablation none,--runs 3,-j 1,--model claude-opus-5-5,--judge-model claude-opus-5-5(no case calls a judge) and--threshold 0.5(no rule reads it). That gives 12 × 3 × 3 = 108 arm runs plus 36 control runs. A run's score is the fraction of its 4 graders it passed; a case's score is the mean of its conforming runs. Before the batch there are 3 Sonnet smoke runs, one each for arms C, B and A. They check that the setting resolves (each trace shows theplan_stress_testreport line), they show which modelplan-reviewerruns on given itsopuspin, and their measured cost sets the batch's--max-cost-usd.Exclusions.
run-validity.py --scaffoldedmust printverdict: VALIDfor each result file; the control file also gets--own-copy-search-warns.arm-report.py. That includes a devils-advocate run on anst-incumbent-*case that is not in incumbent mode.errormakes its case non-comparable. A usage or rate-limit error is rerun per case, never counted.autostays.Addendum 2026-11-03T16:5xZ (validity, written before any score was read). B and C first read INVALID on 3 denials, each the run searching its own plugin copy, which sits outside the sandbox but is readable in a real install. A rerun can't fix that cause, so: all four files are read with
--own-copy-search-warns(all VALID), and the 6 runs with an own-copy search denial are excluded in every arm. They are B st-incumbent-1 r2, B st-proposal-3 r2, C st-plan-4 r3, and the control's st-incumbent-3 r1, st-proposal-3 r2 and st-proposal-4 r3. Each affected case keeps 2 runs. Limit: these searches occur only in arms that run devils-advocate. Full text in the plan file.Variance (read first). If a mode's median per-case range across runs is above 0.25 in any arm,
autostays in that mode and the report says 3 runs do not separate the arms there.Negative control.
prepare-arm.sh devils-advocate --degrade-devils-advocatekeeps the skill's frontmatter and replaces its body with a body that has no procedure and no output format. It runs beside arm B. The control detects a loss whennoise-report.pycalls acase dropor the mean case score falls by 0.15 or more over all 12 cases. If it detects nothing, every mode keepsautoand the skill's contribution is reported as unmeasurable by this suite.Decision rule, per mode, read after the variance check and the control:
noise-report.py, margin 0.15) shows a change of +0.15 or more with an interval whose lower end is above 0:planandproposalstayauto(now measured), andincumbentbecomesboth.non-inferior at margin 0.15with nocase drop: the mode becomesreviewer.autostays, reported as not settled, with the intervals.B never sets a default. B against C at margin 0.05 is reported as a finding only. When C's
sound-partmisses exceed A's by 3 or more in a mode, that is reported beside the mode's result.Launch limits. 4 concurrent model runs (arms A, B, C and the control, each
-j 1) under the host cap of 8. Each launch needs a 1-minute load under 24 and at least 16 GB of memory available, with an EVAL CLAIM line in the hub outbox before the launch and a release line after.Verification
56f217b92f: check-affected (14 passed, 1 skipped over 157 files), reference-verify, pre-push-content.plugins/planning/tests/plan-stress-test.test.sh: PASS=71 FAIL=0. Locally the test's fakeHOMEbreaks the mise node shim, so it ran with node on PATH directly; CI runs a real node.validate-cases.py plugins/planning/evals: exit 0. It printed 12 WARNs, one per st-* case, because the prompt listsBashinallowed_tools; the eval runs grant Bash with--allow-tools Bash.[TODO]phase tags in 8 seeds are now filled in byseed.sh, and the generated workspaces are byte-identical to before.56f217b92f): C onst-plan-11.00 in 209 s; B onst-incumbent-10.75 in 128 s (missed defect-b); A onst-proposal-10.75 in 130 s (missed defect-a). All three conform perarm-report.py. Each trace showsplan_stress_testresolved from the copy'sdocs/conventions/planning.yamlto its arm's value. B ran devils-advocate in incumbent mode with noplan-reviewer; A ranplan-reviewerwith no devils-advocate.plan-reviewer's messages areclaude-opus-5-5under--model claude-sonnet-5-5, so itsopuspin holds, and the batch's reviewer runs on Opus in every arm. The batch keeps--max-cost-usd 200per pass: the smoke runs' highest cost times 36 runs is far below it at Sonnet rates, which leaves room for Opus without a partial pass (judgment).claude-opus-5-5, Claude Code 2.1.317), full record in the Probe result:auto.sound-partgrader false positive, and it covered one run, not the six pre-registered.Related
plan-revieweron).🤖 Generated with Claude Code