Skip to content

feat(pstack): add /create-formal-verification-skill - #709

Merged
poteto merged 4 commits into
mainfrom
lauren/add-create-formal-verification-skill-213a
Oct 18, 2026
Merged

poteto merged 4 commits into
mainfrom
lauren/add-create-formal-verification-skill-213a

Conversation

@poteto

@poteto poteto commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Why

A test checks the cases that someone thought to write. Races, retries, state machines, parsers, and unsafe code often break on a case that nobody wrote. pstack already has /create-verification-skill, which proves app behavior with scripted runs. It has no skill that checks a property across every case in a stated scope.

What the skill does for you

Call /create-formal-verification-skill in your repo. If you name an area or a bug, the skill starts there. The skill then does these steps in order:

  1. It chooses the strongest target in the repo, not the easiest one to verify. A good target is a promise the code makes, where a break harms users and tests rarely reach the case that breaks it.
  2. It picks a verification tool for that question and tells you why.
  3. It verifies one property and shows that property fail on a broken variant of the code, such as a removed check or a reordered step.
  4. It writes a project-local skill. From that skill alone, the next agent can tell whether a change touches the verified code, rerun the check with one command, and tell a real bug from a modeling error.
  5. It follows the new skill once on a scratch change that breaks the property, confirms that the check catches the break, and then drops the change.

At handoff, the skill tells you what it verified, which tool it used and why, the bounds, what the check found, how to rerun it, and what it does not cover. If the current code breaks the property, you get the counterexample as a short sequence of steps in the code's words. The skill does not weaken the property or the model to get a pass, and it does not change product code unless you ask.

Why the skill stays goal-level

The skill says what a good result looks like. It leaves the choice of tool and model design to the agent, because the right choice depends on the question and the repo. A question about events in a bad order usually suits a model checker over a small model of the design. A question about the code for every input, such as memory safety or overflow, usually suits a verifier that runs on the code. A property that must hold at every size can need a proof. The tool must also be one that the repo's developers can install and run.

One prescribed tool or model shape fits some repos and gives the wrong check in the others. So the skill names the tool families, suggests TLA+ as a common start for concurrent and distributed designs, and asks the agent to explain its choice. What the skill fixes is the bar that every result must meet, whatever the tool:

  • Each part of the model or proof points to code that a reviewer can open.
  • The agent must see each property fail on a broken variant of the code before it trusts a pass.
  • The handoff states the bounds, the assumptions, and what the check leaves out.
  • The check reruns with one command and a reproducible tool version.
  • When verified code changes, something tells the next agent to look at the verification again.

What changed

  • The new skill in pstack/skills/create-formal-verification-skill/SKILL.md. It sets disable-model-invocation: true, so it runs only when you call it by name.
  • One row for the skill in pstack/README.md and one line in pstack/skills/poteto-help/SKILL.md.
  • The regenerated copies of these three files under harness/pstack/claude-code/.

Blast radius

This PR changes only prompt text and docs. No script, runtime code, or existing skill changes.

Verification

  • node scripts/validate-plugins.mjs, node scripts/check-pstack-docs.mjs, and node scripts/generate-pstack-harness-trees.mjs --check pass on the head commit and on a merge with current main.
  • The skill file under pstack/ and its copy under harness/pstack/claude-code/ are identical.
  • The docs check reports 0 problems. It notes that pstack/docs/guide does not name the new command yet. Three existing commands get the same note.
  • An eval tested this exact SKILL.md on six open-source repos that it never saw: timgit/pg-boss, vercel/swr, brettwooldridge/HikariCP, kubernetes/client-go, DaveGamble/cJSON, and bluss/arrayvec. It compared the skill with an earlier draft that prescribed TLA+ and came with file templates and a worked example. Blinded judges scored each attempt out of 30 on six criteria, such as target choice, tool fit, and checks.
    • On the four repos with a concurrency or distribution shape, the skill's mean was 29.0 and the draft's was 26.5. On cJSON and arrayvec, the means were 26.0 and 22.0. Two hand corrections to judge scores move the skill's means to 28.75 and 26.25. On every repo, the skill scored the same as the draft or higher when both used the same model.
    • On cJSON and arrayvec, where TLA+ fits poorly, the skill chose a verifier that runs on the code in 4 of 4 runs: CBMC on cJSON and Kani on arrayvec. The draft chose TLA+ on both repos.
    • The skill was weaker on one criterion. On cJSON and arrayvec, its checks score was 3.75 and the draft's was 4.5. One run did not confirm that its CBMC loop bounds were large enough, and another had no single run command.
    • The sample is small: n = 1 run per repo and version, and n = 2 for the skill on cJSON and arrayvec. Treat a one-run gap as a lead, not a finding.
Open in Web Open in Cursor 

Note

Low Risk
Docs and prompt text only; no runtime, scripts, or changes to existing skills.

Reviewed by Cursor Bugbot for commit 3da6ed8. Bugbot is set up for automated code reviews on this repo. Configure here.

Generates a project-local skill that formally verifies one property of the
code where a bug costs the most and tests reach the least. The skill picks
the verification tool from the question instead of prescribing one.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit feaf2fd. Configure here.

Comment thread pstack/skills/create-formal-verification-skill/SKILL.md Outdated
…ings

The proof step said everything must pass again after the scratch change is dropped, and to fix the skill or the verification when a step fails. When the current code already breaks the property, that pointed the agent at weakening the check. Accept a baseline that fails only on findings the agent will report, and limit the fix-it fallback to a missed break or a step that does not run.
@poteto
poteto marked this pull request as ready for review October 18, 2026 06:18
@poteto
poteto merged commit 4382e68 into main Oct 18, 2026
2 checks passed
@poteto
poteto deleted the lauren/add-create-formal-verification-skill-213a branch October 18, 2026 19:47
leegmoore added a commit to liminal-ai/pstack-lim that referenced this pull request Oct 22, 2026
* fix(pstack): make a brand with a type guard, not as

Upstream 0317eb8 (cursor#700), taken unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(pstack): port upstream's docs refresh and docs-vs-tree check

Port of upstream 019a6a2 (cursor#704). The guide and README take upstream's
plain-language pass, and scripts/check-pstack-docs.mjs comes in as is.
Left out: the Claude Code plugin install text and custom-mode wording
(Install row), and the generated harness tree.

The check found three skills the fork bundles that the README never
listed (control-cli, control-ui, deslop, copied from cursor-team-kit),
so the README table now lists them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pstack): list every waiting item so each ketchup call is decidable

Port of upstream 16541c6 (cursor#692) and bc14437 (cursor#708). Every item waiting
on the user is listed with its pick, evidence and risk, and items are
never cut to stay under a word count.

Adapted for T3: screenshots and recordings stay inline, and diagrams
and charts stay on the html_render page, in place of Mermaid and
/visualize (Visuals row). The description keeps the named-only ending
without disable-model-invocation (Skill visibility row).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(pstack): add the formal verification skills and offer them in setup

Port of upstream 4382e68 (cursor#709), f9be7aa (cursor#715), 6b78e9f (cursor#721),
7b353f7 (cursor#716) and the guide line from d457567 (cursor#724).
create-formal-verification-skill and maintain-formal-verification-skill
come in with upstream's text. setup-pstack gains step 8, the formal
verification offer. poteto-help, its recipes, the README and the guide
mention both skills.

Adapted for T3: both descriptions end with the named-only sentence and
drop disable-model-invocation (Skill visibility row). poteto-help keeps
the fork's per-user install steps and T3 note, with upstream's new
setup offer added to its step 2 (Install row). The guide's Claude Code
install line is left out (Install row). Version bumps are left out.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pstack): restore /sharpen to rewriting the prompt

Port of upstream 75fabac (cursor#712) and 88d4751 (cursor#729), taken over the
fork's do-the-work version at Lee's call in the 2026-10-20 retro.
/sharpen asks four questions, then hands back the rewritten prompt and
leaves the user's choices open. README, guide and poteto-help follow.

Adapted for T3: the description keeps the named-only ending without
disable-model-invocation (Skill visibility row). Version bump left out.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(fork): log sync 2 and move the sync point to f2bad52

Records each upstream commit from 101b274 to f2bad52 as take, adapt or
skip, and adds check-pstack-docs.mjs to the checks a sync runs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pstack): allow the negative-control break and name the project skill path in the formal verification skills

Codex review on PR 24: the maintain skill's edit scope forbade the scratch break its
own step 5 requires, and the create skill never said where the project skill lives.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pstack): let the create skill's handoff break the code, and let maintain report updated and bug together

Codex review on PR 24 (r2): the create skill forbade the scratch break its own handoff
check needs, and the maintain skill asked for one outcome while allowing two.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
leegmoore added a commit to liminal-ai/pstack-lim that referenced this pull request Oct 22, 2026
…te, workflow setup, three contradictions (#25)

* sync: upstream cursor/plugins 101b274 to f2bad52 (pstack 0.15.29) (#24)

* fix(pstack): make a brand with a type guard, not as

Upstream 0317eb8 (cursor#700), taken unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(pstack): port upstream's docs refresh and docs-vs-tree check

Port of upstream 019a6a2 (cursor#704). The guide and README take upstream's
plain-language pass, and scripts/check-pstack-docs.mjs comes in as is.
Left out: the Claude Code plugin install text and custom-mode wording
(Install row), and the generated harness tree.

The check found three skills the fork bundles that the README never
listed (control-cli, control-ui, deslop, copied from cursor-team-kit),
so the README table now lists them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pstack): list every waiting item so each ketchup call is decidable

Port of upstream 16541c6 (cursor#692) and bc14437 (cursor#708). Every item waiting
on the user is listed with its pick, evidence and risk, and items are
never cut to stay under a word count.

Adapted for T3: screenshots and recordings stay inline, and diagrams
and charts stay on the html_render page, in place of Mermaid and
/visualize (Visuals row). The description keeps the named-only ending
without disable-model-invocation (Skill visibility row).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(pstack): add the formal verification skills and offer them in setup

Port of upstream 4382e68 (cursor#709), f9be7aa (cursor#715), 6b78e9f (cursor#721),
7b353f7 (cursor#716) and the guide line from d457567 (cursor#724).
create-formal-verification-skill and maintain-formal-verification-skill
come in with upstream's text. setup-pstack gains step 8, the formal
verification offer. poteto-help, its recipes, the README and the guide
mention both skills.

Adapted for T3: both descriptions end with the named-only sentence and
drop disable-model-invocation (Skill visibility row). poteto-help keeps
the fork's per-user install steps and T3 note, with upstream's new
setup offer added to its step 2 (Install row). The guide's Claude Code
install line is left out (Install row). Version bumps are left out.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pstack): restore /sharpen to rewriting the prompt

Port of upstream 75fabac (cursor#712) and 88d4751 (cursor#729), taken over the
fork's do-the-work version at Lee's call in the 2026-10-20 retro.
/sharpen asks four questions, then hands back the rewritten prompt and
leaves the user's choices open. README, guide and poteto-help follow.

Adapted for T3: the description keeps the named-only ending without
disable-model-invocation (Skill visibility row). Version bump left out.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(fork): log sync 2 and move the sync point to f2bad52

Records each upstream commit from 101b274 to f2bad52 as take, adapt or
skip, and adds check-pstack-docs.mjs to the checks a sync runs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pstack): allow the negative-control break and name the project skill path in the formal verification skills

Codex review on PR 24: the maintain skill's edit scope forbade the scratch break its
own step 5 requires, and the create skill never said where the project skill lives.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(pstack): let the create skill's handoff break the code, and let maintain report updated and bug together

Codex review on PR 24 (r2): the create skill forbade the scratch break its own handoff
check needs, and the maintain skill asked for one outcome while allowing two.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>

* fix(retro): record each model a thread ran on, and stop the backup when gitleaks is missing

register takes --model <provider>/<model>@<from>, repeatable, where
<from> is a run ordinal from t3_thread_read's recentRuns or a date. Every
model is kept in order, so a thread that changed lead models shows each
one. REGISTRY.md and the round README list them.

The backup used to treat a missing gitleaks as no hits and push. It now
fails the scan step, pushes nothing, says so on stderr and heads
REGISTRY.md with the failure until a write backs up cleanly.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(reflect): record what happened and apply nothing

Lee's decision in the 2026-10-20 retro: reflect saves and summarizes in
a few lines, surfaces only product or ops decisions that can't wait,
and applies nothing. Approval, filing and skill edits move to the retro
round's open and close.

The three reviewer briefs now ask for what happened, with evidence and
a classification (bad port, bad application, contradiction, genuine
gap), not principles routed to skills. The synthesizer checks and merges
those observations by incident instead of sorting proposals into
Accepted, Rejected and Backlog. The save step records each model the
thread ran on. README, guide and poteto-help follow.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(poteto-mode): wake on schedule_task in every lead, and keep resume notes in the store

Three installed instructions contradicted the role file and the
resume-note hook (2026-10-20 retro, step 1):

- autonomous-run step 2, autopilot-full and multi-phase-plan told a
  Claude lead to use /loop or Monitor. The role file's Waking and
  polling section says those die when the session restarts, so every
  lead now uses T3's schedule_task. check-plan.mjs stops requiring the
  literal /loop 1h marker and requires schedule_task instead.
- pause-safely wrote the resume note to /tmp. It now goes to
  ~/.local/state/pstack/resume/<session>.md, which
  ~/lim-v2/bin/resume-note-hook reads after a compaction, and
  session-pickup looks there.

autonomous-run also says each heartbeat retests the blocker it reports
before repeating it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(poteto-mode): make the plan the record of what done means

From the 2026-10-20 retro. The plan already held the verify boxes, the
fix bar and the scope, but later stages did not read it back, so
deferrals piled into tickets and scope cuts reached Lee as his own
discovery.

- multi-phase-plan keeps the per-PR verify boxes and adds an optional
  Verify, model block for a PR whose risk is the order of events
  (queues, Stop, retries, restarts), run with the project's formal
  verification skill. A new Appendix E lists scope cuts and splits later
  work into decided no, each with its reason, and owed. Close the
  program files tickets only from owed and names every scope cut.
- autopilot-full closes out from that list and names the scope cuts in
  its reply.
- bugbot-triage: a deferral that says a race cannot happen carries the
  model check's result when the plan has a Verify, model block, and
  review cost alone is not a reason to defer.
- check-plan.mjs checks the new blocks and appendix, and prints each
  PR's verdict box ids. plan-boxes.mjs holds the fence and box rules so
  the merge gate reads plans the same way.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(merge-gate): refuse a verdict that omits a plan box or has an unsigned skip

Lee's merge-gate decision in the 2026-10-20 retro: each PR gets a short
must-verify list from its plan, the approver may waive an item with a
reason on the PR, and findings outside the list stay the approver's
call.

- shipping step 1: under the pstack-verdict line, the verdict lists each
  required box of the PR's plan section as done, n/a with a reason, or
  skipped. A skip needs the lead's signed pstack-skip line on the PR,
  and until then the verdict is not a pass.
- merge-gate takes --plan <file> --plan-pr <id> and adds a Plan boxes
  requirement. It refuses a verdict that omits a required box, a box
  line it cannot read, n/a or skipped with no reason, and a skip no one
  with a role in the repo signed outside the verdict comment. Skips are
  checked with no plan given too.
- Tested on GC2's real verdict
  (liminal-ai/t3code-v2#18, issuecomment-6262808190), kept verbatim in
  fixtures/gc2-verdict.md, against a plan fixture with GC2's 16 boxes.
  The gate refuses it. Run against the real plan in the pstack store,
  merge-gate --check refuses it the same way.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(poteto-mode): hand single-task work back through ketchup

The autopilots already end by catching the operator up per the ketchup
skill. The build playbooks that hand finished work back did not:
autonomous-run, feature, bug-fix, refactoring, perf-issue, hillclimb,
visual-parity and shipping now do, keeping each playbook's own content.
Read-only and planning playbooks (investigation, the forensics, eval,
prototype, multi-phase-plan) keep their replies, since the answer is
the reply.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(poteto-mode): set up a thread workflow right after picking the playbook

Lee's decision in the 2026-10-20 retro: once the playbook is picked and
the agent has decided how to use it, the agent sets up a T3 thread
workflow from a template or open-ended, fitted to the actual work in a
handful of consolidated steps, and tells the user it is set up. A step
that waits on the user is asked in chat while the user is there, and
the question tool is for getting their attention when they are away.

Every workflow ends with a cleanup step. The lead lists what it
remembers creating and the scope it worked in, a cleanup subagent
scours that scope and deletes what is clearly from the run, and the
user is asked only about the rest.

The pstack-store todo file stays as the fallback where workflows are
off. session-pickup reads the thread's workflow, and FORK.md's Todo
list row says so.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(retro): the backup finds gitleaks through GITLEAKS_PATH or ~/lim/service

The verifier's FAIL on #25 showed the installed scanner at
~/lim/service/gitleaks/8.30.1/bin is not on PATH, and GITLEAKS_PATH had no
effect. These tests fail at d59628c.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(retro): resolve gitleaks from GITLEAKS_PATH, PATH, then ~/lim/service

The backup ran bare `gitleaks`, so on a host where the scanner is installed
but not on PATH every push stopped at the scan step. A GITLEAKS_PATH that
does not run fails closed rather than falling back, and the not-found error
names each place it looked.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(merge-gate): a skip the verifier signs in a separate comment is refused

Reproduces the verifier's finding on #25: a Codex PASS+NOTES verdict with
perf skipped, then a separate 'pstack-skip: perf by=codex/gpt-6.1-sol' comment,
exits 0 with WOULD MERGE at d59628c. Every agent posts as one GitHub account,
so only by= tells the verifier from the lead. Also pins that a skip line with
no signer never counts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(merge-gate): a skip signed with the verifier's by= does not count

findSignedSkips excluded only the verdict comment itself, so the verifier
could accept its own skip by posting a second comment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(check-plan): plans written before the record of done keep their checks

At d59628c the running limhealth-next port plan fails the new Owed, scope
cut, schedule_task and Later work requirements, and an existing appendix
whose title mentions later work is taken for the new one. These tests fail
there: a plan without the template's opt-in keeps the checks it had, the
template opts in, and only the exact new title counts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(check-plan): the later-work checks apply only to plans that opt in

The template's How to read this now writes "This plan is the record of
done.", and only a plan with that line must carry the Owed, scope cut and
schedule_task close-out and the Later work appendix. Other plans keep the
checks they were written under, /loop 1h included. The appendix is
"## Appendix. Later work and scope cuts", found by that exact title, so a
plan's own Appendix E, such as the operator's reserved calls, is never
read as it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(merge-gate): plan box ids, stale box lines and duplicate boxes

Codex's four findings on acda474, each failing here: lanes indented in the
plan get no ids, bold text inside a verify block drops the boxes after it,
box lines above the counted verdict line still count, and a box listed twice
passes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(check-plan): check-plan and the merge gate read blocks and boxes with one parser

plan-boxes.mjs now owns the block and box reading both use. A block changes
only at a recognized block heading, so bold text such as **Linux.** inside a
verify block keeps the boxes after it. A box is nested only when it sits
inside the list item above it, so lanes indented one to three spaces get
their live ids, and check-plan checks lanes, perf items and empty blocks
against the same top-level boxes the gate requires.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(merge-gate): count only the box lines under the verdict line, each box once

Box lines above the counted verdict line belong to an older verdict an edit
left behind, so they no longer count toward the current one. A box listed
more than once is refused, since the verdict then does not say which status
holds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(merge-gate): the lead waives a skipped box on the gate run, not in a PR comment

Every agent posts as the same GitHub account, so no PR comment can show
that the lead accepted a skip. Two fixes that read a by= field in a
comment both failed: a verifier could sign its own skip, and then an
older verifier's comment could sign a later verifier's skip.

The lead is the only agent that runs the gate to merge, so the waiver
moves onto that run: --accept-skip <id>=<reason>, repeatable. The gate
accepts a skipped box only when the run waives it with a reason. Before
it merges, it posts one PR comment listing each waived box and reason,
and a new waived column in runs.tsv records them. An empty reason, a
bad id, a repeated id, or a box the verdict did not skip is a usage
error. pstack-skip: parsing is gone, and shipping and multi-phase-plan
tell the lead to use the flag.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant