pstack: in Claude Code, poteto-mode cites principles without reading their SKILL.md files #726

Activity

1. boom-bang commented on Oct 19, 2026

   

   boom-bang

   on Oct 19, 2026

   Author

   More actions

   Tested the proposed sentence against two alternatives on omp, which shares the skill body and the disable-model-invocation leaves. The sentence helps a little. Putting the principle list inside each playbook helps far more.

   Setup

   • 5 arms, 12 prompts, 6 repeats each, 72 runs per arm, shuffled order. Every run had its own agent dir, session dir and copy of the fixtures.
   • pstack 0.15.28. The skill message sent to the model is byte-identical to what omp injects for /skill:poteto-mode (checked on a real session record, 20,703 characters).
   • Prompts are small read-only tasks (bug, feature, refactor, perf, design, tests, review, investigation). "Read a principle" means a read tool call on a principle-* skill file, and it matches a raw grep of the tool calls with 0 mismatches.
   • Pre-registered primary metric: at least one principle file read. Contrasts against base use Fisher exact with a Bonferroni threshold of 0.025.

   Arms

   Arm	Change
   base	0.15.28 as shipped
   patched	the sentence from this issue, applied to line 17
   checklist	a numbered "Before your first reply" block (read the playbook, read principles, open the todolist)
   chain	skill body reduced to one line ("read the playbook first, cite only what you read"). Each playbook opens with an unconditional list of the principle skills to read.
   gate	omp extension that blocks every tool except playbook and principle reads until one principle is read. Upper bound only.

   Results (all 72 runs per arm)

   Arm	Read a principle	Cited a principle it never read	Mean tool calls
   base	0/72	3	3.5
   patched	7/72 (p=0.013 vs base)	0	3.8
   checklist	27/72	12	5.4
   chain	68/72 (p<0.0001 vs checklist)	4	6.8
   gate	71/72	2	6.8

   • Base never read a principle in 72 runs, which agrees with the audit in this issue.
   • The patched sentence moved 7 of 72 runs, on 5 of 12 prompts. It hurt on none.
   • The checklist read the playbook in 72/72 runs, but only 27/72 read a principle and 12/72 called todo. Compliance fell with each added step.
   • The checklist's conditional wording ("for every principle you expect to shape a decision") failed mostly through the investigation playbook, which names no principles. Of 43 checklist runs that opened it, 8 read one. The bug-fix, feature and refactoring playbooks name principles, and those runs read far more often.
   • Chain beat checklist on 10 of 12 prompts, tied on 1 and lost on 1 (sign test p=0.012). That check treats each prompt as one unit, because repeats on the same prompt are not independent.
   • Chain and gate are indistinguishable (68 vs 71, p=0.37). A plain unconditional instruction in a file the model always opens matches a hard block.

   What this does not show

   • 4 chain runs read no principle. Three ended after 2 to 3 tool calls, and none was blocked. Reaching 72/72 is not realistic with text alone, and a gate cannot force a model to make a call either (one gate run made zero).
   • Reads are not quality. Nobody graded whether replies improved. Chain replies are about 25% longer.
   • Cited-but-unread principles are not eliminated. Chain had 4 and base had 3.
   • One model (Sonnet via omp), headless, tiny fixtures. This issue's audit used Claude Code with Opus. The adapter clause in the proposed sentence applies only in Claude Code and was read in 2 of 72 patched runs, so that half is essentially untested here.
   • 7 chain and gate runs have no final message in the session record. The table above uses all 72 runs and takes the reply text from stdout, so none were dropped.
   • The principle lists I put in the playbooks are my choices (for example investigation reads prove-it-works and attack-the-premise). Other lists were not tried.

   Suggested fix

   Instead of, or alongside, the sentence in this issue:

   1. Add an unconditional "Before step 1, Read these principle skills" line at the top of each playbook, naming that playbook's principles.
   2. Reduce the skill body line to "Read the playbook before anything else. Cite only principles whose SKILL.md you read."
   3. Add a small check that every principle-* named in a playbook exists under skills/, so the lists cannot rot.

   This is a few lines in six playbook files and needs no hook, so it works on every harness. It still needs a retest on Claude Code with Opus before anyone calls it a fix. The harness, per-run sessions and scoring scripts are local and I can share them.