fix(runtime): a source retry withdraws the prompts rendered from the withdrawn expansion

[aviggiano]

This PR was split out of draft #1216 so it can ship in v0.1.2. It ports #1216's second commit (1621598) to main on its own, with its comments and tests adapted to main's semantics. #1216's main change, keeping a published runtime prompt that the current build renders differently, is contentious and deferred to v0.1.3. This fix does not depend on it. On this branch a prompt that renders differently still fails the render as it does on main, so the tests check that the stale prompt is gone before the next render, not that it gets adopted.

Problem

resume --retry-failed on a dynamic source whose verifier failed withdraws the published expansion generation, so the group expands again from the source's new output. If the re-run source plans different items, a task that waits on the group and whose prompt names the group's children cannot render again. A custom join whose prompt uses {{artifact_path:<group>}} is one example. The re-expansion throws runtime rendered prompt changed for <join>. It publishes the new manifest before it renders, so every later render, and every lifecycle admission check that re-derives the dynamic controls, renders the same prompt again and throws the same error. The run stays stuck until someone deletes the join's artifacts/<attempt>/prompt.rendered.md by hand and resumes it.

The packaged topologies are not affected. Their deferred prompts belong to the nodes behind the goal groups: dedupe, triage, severity classification, test aggregation and the final report. Those prompts name only static nodes' directories, and they reach the goal children only through authority selectors. Authority selectors render the same bytes whatever the children are.

Root cause

archiveDynamicExpansionsForRetry (packages/runtime/src/dynamic-expansion-retry.ts) moves the manifests and the attempt state the generation owns (artifacts/<storage-id>, invariant snapshots, verification records) into dynamic-expansion-history/. It did not move artifacts/<attempt>/prompt.rendered.md for the sealed base tasks whose prompt waits on the group (deferredPromptGroups), although that prompt was rendered from the withdrawn children. renderReadyRuntimePrompts publishes a prompt that is missing but refuses one on disk whose bytes differ from a fresh render. So the stale file blocks the new expansion.

Change

• collectAttemptStateMoves also moves the published prompt.rendered.md of every sealed base task whose deferredPromptGroups names a withdrawn group.
  • It skips only a prompt that does not exist. The check uses lstat, so a dangling symlink is not mistaken for a prompt that was never rendered.
  • It refuses a path with symlink components, a symlink at the prompt path (live or dangling), a prompt that is not a regular file, and any other failed lookup of that path.
• planDynamicExpansionRetryArchive now reads the sealed runtime base before collecting the moves. These prompts are therefore validated with the rest of the attempt state before the Smithers timetravel reset, just as the generation's own entries already were.
• retry.json lists each moved prompt in archived_attempt_paths with the other archived paths. The next expansion renders it afresh from the new generation.
• docs/reference/topology-yaml.md now says that the generated children's artifacts, and the rendered prompts of later nodes that wait on the group, move with the manifests.

The first commit is the port. The second commit came out of review. It replaces the port's existsSync check, which follows symlinks and so skipped a dangling link, with lstat, and it adds the refusal test below.

Verification

The branch is based on origin/main 13af837. To run the tests against main, I copied origin/main's packages/runtime/src/dynamic-expansion-retry.ts over this branch's version, left the tests as they are, and recompiled.

• dynamic-expansion.test.ts, "explicit source retry re-derives the base runtime controls after archiving an expansion". The fixture's join now has a deferred prompt, Join {{artifact_path:fanout}}., with deferredPromptGroups: ["fanout"]. After the retry, the source plans a different item and the group expands again. The test checks four things: the join's prompt names the new child and not the withdrawn one; admission re-derives the expanded controls; the archive holds the withdrawn prompt byte for byte; and retry.json lists exactly the generation's artifact directory and the join's prompt.
  • origin/main: fails at the re-expansion with Error: runtime rendered prompt changed for join, thrown from renderReadyRuntimePrompts.
  • This branch: passes.
• dynamic-expansion.test.ts, "explicit source retry refuses a withdrawn prompt that is not a regular file before anything moves" (new). After the expansion, each case damages the join's published prompt in one way: a live symlink, a dangling symlink, a directory at the prompt path, or a symlinked attempt directory. In each case planDynamicExpansionRetryArchive must throw an ArtifactPathError with the expected code, the manifest must stay in dynamic-expansions/, and no dynamic-expansion-history/ may exist.
  • origin/main: fails at the first case with Missing expected exception: symlink, because main never checks these prompts.
  • The first commit alone (the existsSync check): fails with Missing expected exception: dangling-symlink.
  • This branch: passes.
• dynamic-lifecycle.test.ts, "explicit source retry prunes the withdrawn generation from run state and keeps observers admitted". This test runs a real resumeRun({ retryFailed: true }) on a launched run whose compiled base tasks mark the join's prompt as deferred. It checks that the join's prompt is in the archive after the resume, that the re-expansion renders it again, and that observers stay admitted.
  • origin/main: fails with AssertionError: Expected values to be strictly equal (true !== false), because the join's prompt is still in place after the resume.
  • This branch: passes.
  • This fixture's join prompt (Summarize completed work in {{artifact_path}}/report.md.) does not name the children. It re-renders to the same bytes, so this test cannot reproduce the throw. The unit test above does.
• A run already stuck on v0.1.1. I checked this with a throwaway lifecycle test that is not committed. After resume --retry-failed, I wrote a differing prompt back at the join's path, which is the state v0.1.1 leaves.
  • The re-expansion threw runtime rendered prompt changed for strict-join, and the new manifest stayed published. Strict admission failed with the same message.
  • A second resume --retry-failed created no new archive and left the stale prompt in place, because no verifier had failed.
  • I then deleted the prompt by hand. Strict admission reported it as missing until the next render, but resume still went through. The next render published the prompt again, and strict admission passed. The changelog entry says this.
• Full files on this branch: dynamic-expansion.test.ts 17/17 and dynamic-lifecycle.test.ts 19/19. The retry subset of runtime.test.ts (--test-name-pattern='resume retries|retry-failed|retryFailed|source retry|retry') passes 16/16. Three Bun adapter tests in that subset are skipped because Bun is not installed here.
• npx prettier --check and npx eslint on the changed files, CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci (exit 0), pnpm -w lint, pnpm --filter @ultrafuzz/runtime typecheck, pnpm -w knip and node scripts/docs-check.mjs all pass.

Risk

• Narrow path. The change only runs when resume --retry-failed resets a producer that owns the published expansion, which is the path that already withdraws the generation. It reads the sealed runtime base (smithers/runtime-base-tasks.json), which every run with a dynamic group launched on v0.1.0 or later has. With no sealed base, nothing new moves.

• Unchanged prompts move too. Every deferred prompt that waits on a withdrawn group moves, including one that would re-render to the same bytes, like the packaged final report. The cost is one rename and one retry.json entry. Smithers resets the task with the source's dependents, and the next expansion renders its prompt again before the task runs.

• New refusals before reset. Planning now refuses the retry, before any timetravel, in these cases for such a task:

  • a symlinked prompt.rendered.md, live or dangling;
  • a prompt.rendered.md that is not a regular file;
  • a symlinked attempt directory;
  • any other failed lookup of that path, such as an attempt path that is a regular file.

  In each case the operator removes the entry and retries. The new test covers every case except the last. ultrafuzz itself never creates a symlink under a run's artifacts/.

• Not atomic. The prompt moves join a window that already was not atomic. It opens after the Smithers reset and the manifest rename, and closes once the controls are re-derived, the run state is pruned and retry.json is written.

  • A crash in that window, or a rename that throws, leaves a partial archive with no retry.json, as the generation's own moves already can on main. I tested the throwing case by forcing EACCES on the join prompt in a throwaway test.
  • In that state the admission check fails with persisted dynamic runtime task plan does not match its sealed templates and manifests, and the join's prompt stays in place. A re-expansion that plans other items then fails as it does on main today, so this is no worse than main.

• Stuck runs are not repaired. Upgrading does not unstick a run that is already stuck on v0.1.1. The archive only runs for a failed verifier's producer, and such a run failed at render time instead. The changelog entry gives the manual step.

• The join's other artifacts, such as outputs of an earlier join attempt, stay where they are, as on main.

Changelog entry

• [runtime] [docs] When resume --retry-failed retries a dynamic source whose verifier failed, it now also withdraws the rendered prompts of later nodes that wait on the group, so they render again from the new expansion. Before, if the retried source planned different items, a node whose prompt names the group's children (for example with {{artifact_path:<group>}}) failed every render and lifecycle admission check with runtime rendered prompt changed for <attempt>, and the run could not continue. The packaged topologies were not affected. Upgrading does not repair a run that is already stuck this way: delete the stale artifacts/<attempt>/prompt.rendered.md by hand and resume the run, and the next render publishes it again.

Refs #1141

Greptile follow-up

• Kept: a dangling symlink in place of the run's whole artifacts/ directory is treated as absent by the attempt-state root check (comment). That check predates this PR; its existsSync use is noted under Risk. A run whose artifacts/ root is a dangling link is already corrupt: every other command that reads run artifacts fails on it, and repairing it is manual either way. This PR's own new paths, the per-attempt prompt and its attempt directory, use lstat and refuse a dangling link before the Smithers reset. The new test covers that.

🤖 Generated with Claude Code

[greptile-apps]

Dangling attempt link gets skipped If artifacts/<attemptId> is a dangling symlink, the path check treats it as missing, and the prompt lookup skips it. The retry then resets Smithers and archives the manifests rather than refusing the invalid directory before reset. The next expansion cannot create the prompt there, leaving the run in need of manual repair. Check parent directories with lstat before treating the prompt as absent.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/dynamic-expansion-retry.ts
Line: 279-280

Comment:
**Dangling attempt link gets skipped** If `artifacts/<attemptId>` is a dangling symlink, the path check treats it as missing, and the prompt lookup skips it. The retry then resets Smithers and archives the manifests rather than refusing the invalid directory before reset. The next expansion cannot create the prompt there, leaving the run in need of manual repair. Check parent directories with `lstat` before treating the prompt as absent.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.