feat(improvement): a verify disposition that can pass a tree and keep spending shots

[drewstone]

The gap

agenticGenerator's Verifier answered one boolean, and ok carried two meanings at once — "this tree is shippable" and "stop now". The shot loop returns inside if (result.ok), and AgenticGeneratorOptions exposed no field to override it. So a caller who wanted to spend the shot budget it was given and keep the best tree — ordinary best-of-n — could not say so.

The consumer's workaround, which is the motivation

agent-lab's playproof study (projects/playproof/verify-budget.ts) needed best-of-n and encoded it by hand:

1. reject every shot but the last, whatever it measured;
2. score each tree as it is produced;
3. physically write the best program back into the worktree just before the last shot is accepted.

Step 3 exists only because the substrate ships the worktree: with one boolean, "ship the best tree" has to mean "make the best tree BE the worktree". Their own module says so — "The upstream change that would remove the restore is one field on AgenticGeneratorOptions: a disposition richer than a boolean."

Their measured runs are why it matters. Under first-acceptance-wins, 1 shot of 3 fired and the program that shipped was never once run by its author. Under the workaround, 3 of 3 fired and every shot ran its own program.

The shape chosen

VerifyResult is widened with two optional fields, not split into a union — an existing verifier's return value is still a valid VerifyResult and still gets today's semantics.

field	question	omitted
ok	is this tree shippable	unchanged
keepGoing	should the budget stop here	the first passing tree ends the candidate, exactly as before
score	how does this tree rank against the other passing trees	every passing tree ties, so the later one wins

Two fields rather than one, because they are independent: keepGoing alone is "spend the budget, ship the final state"; score alone would be meaningless under first-acceptance-wins. The disposition lives on the RESULT rather than on AgenticGeneratorOptions, because it is a per-shot decision the caller already makes per shot — an option could only express a static policy, and could not say "this tree is good enough, stop early".

The generator owns the restore

That is the half that removes the workaround. A passing tree whose verifier asks for another shot is snapshotted as a Git tree object (git add --all into a private index file, so the index the driver commits from is untouched). When the budget ends, the highest-scoring tree is restored — read-tree -u --reset plus clean -f -d, so content, added files, and the removal of files only a losing shot wrote all land — and the restore is proved by re-snapshotting and comparing tree ids before the candidate is returned. The caller ranks; the runtime moves the bytes.

A last shot that breaks or reverts the change no longer costs the candidate: a banked tree passed verification, so it ships. The failing shot's own rejected disposition is still emitted first, so its evidence survives, and the invariant holds — a tree that failed verification is never what ships.

Compatibility is an explicit gate

tests/agentic-generator.test.ts drives the real agenticGenerator with a verifier returning today's shape and asserts today's behaviour:

• COMPATIBILITY: a verifier returning todays shape still stops at the first passing tree — 1 of 3 shots fires, the disposition stream is exactly ['accepted'] with restoredFromShot: null, and the accepted shot's own tree is what lands.
• COMPATIBILITY: a failing verifier still feeds the next shot and never ships — verification FAILED reaches shot 2, all three dispositions are rejected, applied: false.

Six more gates cover the new path: every shot fires under keepGoing; the best tree lands when the best is not the last (including the losing shot's extra file being removed); a banked tree ships when the last shot breaks the change; a banked tree ships when the last shot reverts every edit; nothing ships when no shot ever verifies; a set of trees that cannot be ordered (scored and unscored mixed) and a non-finite score both throw.

Surface changes

• VerifyResult: +keepGoing?: boolean, +score?: number.
• AgenticGeneratorShotDisposition: new kept variant (score, best, feedback); accepted gains restoredFromShot: number | null.
• New internal module src/improvement/worktree-tree.ts (not exported).
• cli-worktree, cli-in-place and commandVerifier are untouched.

Version bump

check:version-bump refused the branch for the two export-shape changes. pnpm run release:prepare 0.174.0 is an isolated final commit (chore(release): 0.174.0) — drop it if you would rather cut the version yourself. No tag was cut.

Proof

pnpm run lint       → Checked 658 files in 403ms. No fixes applied.
pnpm run typecheck  → tsc --noEmit && tsc --noEmit -p tsconfig.examples.json (clean)
pnpm test           → Test Files 1 failed | 252 passed | 3 skipped (256)
                      Tests 1 failed | 3115 passed | 9 skipped (3125)

The one failure is the known pre-existing macOS flake, tests/improvement-driver.test.ts > forks isolated generation-two candidates from the promoted generation-one surface, which times out at 20s under full-suite load and passes in isolation (11 passed (11), 13.31s). It fails identically at origin/main and is green on CI Linux. Untouched by this branch.

pnpm run docs:check          → docs freshness: OK — no drift detected
pnpm run check:api-surface   → 2224 exports across 18 entry points, record current
pnpm run check:version-bump  → paid for by 0.173.0 -> 0.174.0 (minor)
pnpm run check:testing-fixture / check:skills → current / valid

Both sides of every path comparison are canonicalised before comparing, per this repo's /var → /private/var history.

[tangletools]

Approved: widens VerifyResult with optional keepGoing/score, backward compatible with explicit COMPATIBILITY gates. agenticGenerator now owns the best-tree restore, proved by tree-id comparison before return.