fix(cli): recovery hints name ultrafuzz commands, not raw runner invocations

[aviggiano]

Refs #1145

Problem

For the failed smoke-main-2 smoke run, ultrafuzz why on main (2cacf4c) printed recovery steps an operator cannot run:

- verify:final-report#0 retries-exhausted: All retries exhausted. Last error: ... (attempt 1/1)
  unblock with: workflow runner up <target>/.smithers/workflows/ultrafuzz-smoke-main-2.tsx --run-id ultrafuzz-smoke-main-2 --resume true
Note: Last good checkpoint: frame 29. Resume in place with `workflow runner up <target>/.smithers/workflows/ultrafuzz-smoke-main-2.tsx --run-id ultrafuzz-smoke-main-2 --resume true` or replay from the checkpoint with `ultrafuzz replay` <target>/.smithers/workflows/ultrafuzz-smoke-main-2.tsx --run-id ultrafuzz-smoke-main-2 --frame 29`.

workflow runner is not a command. ultrafuzz-smoke-main-2 is the runner's run ID, not the Ultrafuzz one. ultrafuzz replay takes neither a workflow path nor --frame. Even a well-formed ultrafuzz resume smoke-main-2 would not help, because a plain resume leaves the rejected verifier failed. #1194 already keeps the path intact. This PR fixes the hint text.

ultrafuzz status on a paused run has the same fault. Its reason reads "run was gracefully paused; resume with workflow runner up --resume <runId>".

ultrafuzz node smoke-main-2 final-report failed with WORKFLOW_NODE_FAILED: Command failed: /proc/<pid>/fd/24 --config=/proc/<pid>/fd/22/controls/bunfig.toml ... /proc/<pid>/fd/22/dependencies/packages/000004/src/bin/smithers.js node final-report --run-id ultrafuzz-smoke-main-2 --format json --full-output. Other failed runner queries, apart from timeouts, were reported the same way. For example, why on a project whose store holds no run history showed a Command failed: <argv> line too.

Root cause

1. Runner suggestions were renamed, not rebuilt. publicWorkflowText (lifecycle-inspection.ts) rewrote a runner suggestion by its command name only. If Ultrafuzz had a command with the same name, smithers <cmd> became `ultrafuzz <cmd>`, and the runner's arguments stayed after the closing backtick. Otherwise it became workflow runner <cmd>. A rename cannot produce the right command here:

   • The runner's replay --frame N corresponds to ultrafuzz fork --frame N.
   • On a failed run, the runner's up --resume has to become ultrafuzz resume --retry-failed.
   • The runner's up, retry-task and logs have no Ultrafuzz command of the same name.

   These suggestions reach an operator in two places: why (summary, unblockers and notes) and the status health reason. fix(runtime): make run synchronization single-writer, idempotent, bounded and non-fatal for observers #1180's publicHealthReason rewrote the status reason's smithers why and smithers supervise -r <id>. It did not rewrite the paused reason's smithers up --resume <runId> (pinned runner, run-status.js:484). Run-level errors already leave out the resume commands the runner appends to them, because currentSmithersRunError keeps the cause or summary.

2. node final-report had two causes:

   • final-report is the topology node ID. The workflow's task IDs are prepare:final-report, node:final-report and verify:final-report, and ultrafuzz node smoke-main-2 verify:final-report works. So the runner answered {"ok":false,"error":{"code":"NODE_NOT_FOUND","message":"Node not found: final-report"}} on stdout, with exit 4. I captured this with a probe against a copy of the target.
   • runSmithersInspectionCommand stored Node's exec error (Command failed: <argv>) as the snapshot's error. The node diagnostic prints error first, so the operator saw the bun command line instead of the runner's reason.

3. The runner's own message carries its advice and its name. It follows the reason with its own next step. For a store with no run table it says "No Smithers run history found at . Run 'smithers up ' to start a run first. See https://smithers.sh/reference/errors" (openSmithersStore.js:262). For a missing store it says "No smithers.db found at ." followed by the same advice (line 115). The publicWorkflowText rename was case-insensitive, so it read the capitalized product name in "Smithers run history" as the command smithers run.

Change

• why rebuilds each runner recovery suggestion as the ultrafuzz command for the run, using the Ultrafuzz run ID. An unblocker is rewritten as one whole command. In the notes and the summary, only the commands the runner quotes in backticks are rewritten:

  Runner suggestion (0.35.0)	Raised for	why now says
  up … --resume true, retry-task … --node-id N on a failed run	retries-exhausted, dependency-failed, stalled, checkpoint note	ultrafuzz resume <run-id> --retry-failed
  up … --resume … on a live or paused run	stale-heartbeat, paused summary	ultrafuzz resume <run-id>
  retry-task … --node-id N on a live run	stale-task-heartbeat, retry-backoff, stalled	ultrafuzz resume <run-id> --reset-node N
  replay … --frame N	checkpoint note	ultrafuzz fork <run-id> --frame N
  logs <id>	engine-busy	ultrafuzz events <run-id> --watch
  inspect <id>	side-effect-boundary-crossed	ultrafuzz inspect <run-id>
  up --max-concurrency N (no --resume)	concurrency-saturation warning	unchanged: `workflow runner up --max-concurrency N`

  A rebuilt command skips the runner-name scrub, so a run ID that contains "smithers" stays intact. approve, deny and signal have no Ultrafuzz command and keep the existing neutral rename. The runner raises them only for approval and signal waits, and generated workflows contain no approval, signal or timer components.

• status reason for a paused run. publicHealthReason now also turns smithers up --resume <runId> into ultrafuzz resume <run-id>, as it already did for the orphaned run's supervise -r. This extends the pattern in fix(runtime): make run synchronization single-writer, idempotent, bounded and non-fatal for observers #1180's function by one alternative.

• Failed runner queries. When a query fails and the runner printed a JSON error envelope, the snapshot's error is the envelope's message instead of Command failed: <argv>. Any sentence in it that quotes a runner command is dropped. Timeouts, and failures with no envelope, report as before.

• The rename is lowercase only. publicWorkflowText renames only lowercase smithers <command>, which is how the runner writes its command suggestions. "Smithers run history" now scrubs to "workflow runner run history" instead of becoming `ultrafuzz run`.

• Docs. docs/reference/cli.md gets one paragraph listing which suggestions why rebuilds, and saying that approve and signal suggestions stay in the runner's words.

After the change, on the same copied project:

  unblock with: ultrafuzz resume smoke-main-2 --retry-failed
Note: Last good checkpoint: frame 29. Resume in place with `ultrafuzz resume smoke-main-2 --retry-failed` or replay from the checkpoint with `ultrafuzz fork smoke-main-2 --frame 29`.

error: WORKFLOW_NODE_FAILED: Node not found: final-report

why with the copy's smithers.db replaced by an empty file, which makes the pinned runner report missing run history:

# main (2cacf4ca)
warning: WORKFLOW_INSPECT_FAILED: Command failed: /proc/<pid>/fd/24 --config=/proc/<pid>/fd/22/controls/bunfig.toml ... smithers.js inspect ultrafuzz-smoke-main-2 --forma…
error: WORKFLOW_DIAGNOSIS_FAILED: Command failed: /proc/<pid>/fd/24 --config=/proc/<pid>/fd/22/controls/bunfig.toml ... smithers.js why ultrafuzz-smoke-main-2 --format js…
# this PR
warning: WORKFLOW_INSPECT_FAILED: No workflow runner run history found at <target>/smithers.db. See https://smithers.sh/reference/errors
error: WORKFLOW_DIAGNOSIS_FAILED: No workflow runner run history found at <target>/smithers.db. See https://smithers.sh/reference/errors

With smithers.db moved away, both lines read "No smithers.db found at /smithers.db. See https://smithers.sh/reference/errors".

Deliberately not built

• No per-kind unblocker table. The runner's command, plus whether the run failed, already picks the right command for every blocker the pinned runner raises on generated workflows. Generated verifiers have retries={0}, so they fail rather than stall. That means --reset-node on a live run never reruns a verifier on output it already rejected.
• One rewriter per surface, not one shared rewriter. status keeps fix(runtime): make run synchronization single-writer, idempotent, bounded and non-fatal for observers #1180's publicHealthReason, extended by one alternative, and why has ultrafuzzRecoveryCommand. Merging them would move code between state-export.ts and lifecycle-inspection.ts, and neither diff here needs that. If the runner changes how it words its suggestions, both need updating.
• No ultrafuzz form of up --max-concurrency N. That suggestion starts a run with a higher limit. ultrafuzz resume <run-id> --max-concurrency N does nothing on a live run, so no single ultrafuzz command is right for it, and the hint keeps main's wording. For a run with a pinned limit, the engine records this note only when a descendant run waits for a slot (engine.js:9595-9607). Submit always pins the limit (smithers.ts:4474), so the note should be rare on generated runs.
• No --force for an orphaned run. The runner suggests up … --force true for stale-heartbeat, but plain ultrafuzz resume is enough. The runner classifies such a run as orphaned (or stale when the owner cannot be checked), and neither is an active state, so ultrafuzz goes on to up --resume. The runner refuses that only for a fresh heartbeat or a live owner. This matches the fix(runtime): make run synchronization single-writer, idempotent, bounded and non-fatal for observers #1180 orphan hint in status.
• No bare topology IDs in ultrafuzz node. final-report could mean any of three tasks. In smoke-main-2 the failing one is verify:final-report, so guessing node:final-report would show a finished node. The error now names the ID the runner did not find.
• The runner's error-reference link stays. "See https://smithers.sh/reference/errors" is a link to the runner's documentation, not a command, so the advice filter keeps it.
• ultrafuzz replay is left alone, but it is broken on the pinned runner. It runs smithers replay <workflow> --run-id <id> --format json --full-output without --frame. Smithers 0.35.0 requires that option: running the pinned runner's replay without it returns VALIDATION_ERROR: Missing required option --frame with exit 4. The hints therefore name ultrafuzz fork <run-id> --frame N, which does the runner's fork and resume. Adding --frame to replay or retiring it is a CLI decision, and a parallel change in this batch reworks the replay and fork result types.
• why --json still rejects five blocker kinds (found, not fixed). The CLI result schema's blocker kind enum lists binding, side-effect-boundary and other. It does not list stale-task-heartbeat, bound-stale, binding-missing, approval-decided-resume-required or side-effect-boundary-crossed. validateCliResultEnvelope rejects a why result carrying any of them. Since envelope() throws on that, why --json fails for a hung task (stale-task-heartbeat) or after a forced crossing. Human why is unaffected. This needs a separate schema fix.
• The summary line is unchanged. It still names the linked workflow run ID (Run ultrafuzz-smoke-main-2 is failed), which why --json also reports as workflow_run_id. It is not a recovery hint.

Verification

• Tests that fail on origin/main (2cacf4c) and pass with the change. I ran them on clean worktrees of main and of this branch before the review fixes (0739a4d), with only the test files copied in:
  • lifecycle-inspection.test.ts › "diagnoseRun names the ultrafuzz command for each recovery the runner suggests". It uses the pinned runner's exact suggestion strings for a failed, a live and a paused diagnosis, under run ID smithers-probe.
    • On main, every asserted value differs. For example, the unblocker is workflow runner up /work/target/.smithers/workflows/ultrafuzz-smithers-probe.tsx --run-id ultrafuzz-workflow runner-probe --resume true.
    • On 0739a4d, the concurrency note failed: Remediation: `ultrafuzz resume smithers-probe`. instead of `workflow runner up --max-concurrency 8`.
  • runtime.test.ts › "getRunHealth accepts strict 0.35 orphan, cancel-pending, quota, and operation metadata shapes". It gains a paused case with the runner's exact reason. On both main and 0739a4d the reason was resume with `workflow runner up --resume <runId>` .
  • cli/lifecycle-commands.test.ts › "why reports a runner failure in the runner's words, without its advice". The fake runner answers why with the pinned runner's missing-history envelope and exit 1, and the test asserts the exact error: line.
    • On main: error: WORKFLOW_DIAGNOSIS_FAILED: Command failed: /proc/<pid>/fd/29 /proc/<pid>/fd/28 why ultrafuzz-lifecycle-cli-run --format json --full-output.
    • On 0739a4d: No `ultrafuzz run` history found at <db>. Run 'workflow runner up <workflow>' to start a run first. See ….
    • Either source change alone still fails it. I checked on 0739a4d with one change applied at a time: the advice filter alone leaves `ultrafuzz run`, and the lowercase-only rename alone leaves the advice sentence.
  • smithers-executable-capability.test.ts › "a failed runner query reports the runner's own reason, not the command line it ran". On main, error was Command failed: /proc/<pid>/fd/22 /proc/<pid>/fd/21 node final-report --run-id ultrafuzz-run --format json --full-output\n.
  • cli/lifecycle-commands.test.ts › "why reports the diagnosis in human and JSON output". The fixture uses the runner's real retry-task suggestion. On main, the rendered line is workflow runner retry-task /tmp/…/.smithers/workflows/ultrafuzz-lifecycle-cli-run.tsx --run-id … --iteration 0.
• Targeted suites with the change, all passing:
  • lifecycle-inspection, 34 tests: diagnoseRun|getWorkflowNode|watchWorkflowNode|getRunTimeline|listRunSnapshots|event queries|queryWorkflowEvents|watchWorkflowEvents|truncated event stream|cancelRun|lifecycle commands reject
  • smithers-executable-capability: all 19 tests
  • runtime.test.ts, 19 tests: getRunHealth|controller refresh|parseCurrentSmithersInspect|status reports a runner query|status keeps reporting runner health
  • cli lifecycle-commands, 12 tests (--test-concurrency=1): every test except doctor
• End to end. I reproduced the cases above with the built CLI (why smoke-main-2, node smoke-main-2 final-report, status smoke-main-2, and why with the store emptied or moved away). I ran them against an rsync copy of /home/ubuntu/targets/tiny-vault, so the real run directory was not touched. The outputs are above. I did not reproduce a paused status end to end; the runtime test covers it with the runner's exact reason string.
• Gates. prettier, pnpm -w lint, CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci, pnpm -w knip, pnpm -w docs:check, and typecheck for runtime and cli all pass. The complexity ceiling (83, verifyCoverageProductionInventory) is unchanged.

Risk / compatibility

• Text only. unblocker, notes, summary and the paused status reason change for runner suggestions only. The JSON shape and types are unchanged. Prose unblockers and non-command notes go through the same scrub as before.
• Failed-query error. A failed inspection query's error is now the runner's reason, without the runner's advice, instead of Command failed: <argv>. Nothing in the repository matches on that text. smithersSnapshotReportsMissingRun reads the raw envelope, not error. Synchronization and status diagnostics still prefer stderr when the runner wrote any.
• The advice filter is a heuristic. It drops any sentence that quotes a lowercase runner command ('smithers … or `smithers …). If a runner message puts its reason and its advice in one sentence, that sentence is dropped. If nothing is left, the error falls back to Command failed: <argv> as before.
• Lowercase-only rename. A capitalized Smithers <command> in runner text is no longer rewritten to an ultrafuzz command; it scrubs to workflow runner <word>. In the pinned runner, command suggestions are written in lowercase, and the capitalized name appears in prose such as "No Smithers run history".
• Unrun headline hint. ultrafuzz resume <run-id> --retry-failed is now the headline hint for a verifier rejection like smoke-main-2. I did not run it against that run. Making --retry-failed recover a verifier rejection is a separate fix in this batch ("resume --retry-failed can recover a run whose verifier rejected output"). This PR only makes the hint name the command.
• No --project in hints. Rebuilt hints omit --project, like fix(runtime): make run synchronization single-writer, idempotent, bounded and non-fatal for observers #1180's status hint, so they assume the operator's working directory is the target project.
• Conflict with test(runtime): delete vacuous and source-text tests, keep behavioural coverage #1203. This PR conflicts textually with test(runtime): delete vacuous and source-text tests, keep behavioural coverage #1203 in packages/runtime/test/lifecycle-inspection.test.ts. This PR adds a runId = "inspect-run" parameter to launchedProject, and test(runtime): delete vacuous and source-text tests, keep behavioural coverage #1203 restructures that function. Whichever lands second adds the parameter to the restructured launchedProject. git merge-tree shows no other conflict with the open PRs. feat(runtime): let agents record roadblocks in a run-local friction log #1162's conflict in smithers.ts already exists against main.

Changelog entry

ultrafuzz why now gives each recovery step as a runnable ultrafuzz command for the run, such as ultrafuzz resume <run-id> --retry-failed for a failed node or ultrafuzz fork <run-id> --frame <n> for the last good checkpoint, instead of a workflow runner invocation with its workflow path and run ID. ultrafuzz status on a paused run now suggests ultrafuzz resume <run-id>. A failed runner query, such as ultrafuzz node with an unknown node ID, now reports the runner's reason ("Node not found: final-report") without the runner's own advice, instead of the command line it ran.

Greptile follow-up

• Fixed: a runner logs suggestion now becomes ultrafuzz events <run-id> --watch --history. --watch alone streams only new events, so on a stalled or ended run it showed none of the events that explain the blocker (comment).
• Kept: on a failed run, a runner retry-task --node-id <task> suggestion still becomes ultrafuzz resume <run-id> --retry-failed (comment). Its node ID is a runner task ID, which may name a verifier. --retry-failed is the recovery path that re-runs a failed verifier from its producer (fix(runtime): resume --retry-failed can recover a run whose verifier rejected output #1205). On a failed run, the failed nodes are the work the campaign needs to finish. On a run that has not failed, the hint keeps the target (--reset-node <task>), as before.

Also rebased onto main after #1203. The rebase resolves the launchedProject helper conflict by keeping #1203's shared-run structure and adding this PR's runId parameter. lifecycle-inspection.test.ts: 53/53 on the rebased tree.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge, although the existing targeted-retry wording concern remains non-blocking.

Findings

1.  Targeted retry becomes broad ▶

Fix with agent prompt

### Issue 1
packages/runtime/src/lifecycle-inspection.ts:undefined-1985
On a failed run, a runner suggestion to retry one task becomes `ultrafuzz resume <run-id> --retry-failed`, dropping its `--node-id`. That command resets every failed task, so an operator following a single-node hint can also rerun unrelated failed work, including costly producers. The hint should preserve the target where possible or make its broader effect clear.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR rebuilds runner recovery suggestions as Ultrafuzz commands, updates the paused-status hint, and reports runner error messages instead of failed command lines. The change since the previous review adds --history to the events hint so it includes existing events.

_{Reviews (3) · Last reviewed commit: "fix(runtime): the events hint for a runn..."}

[greptile-apps]

Targeted retry becomes broad

On a failed run, a runner suggestion to retry one task becomes ultrafuzz resume <run-id> --retry-failed, dropping its --node-id. That command resets every failed task, so an operator following a single-node hint can also rerun unrelated failed work, including costly producers. The hint should preserve the target where possible or make its broader effect clear.

Knowledge Base Used: Runtime orchestration

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/lifecycle-inspection.ts
Line: 1985

Comment:
**Targeted retry becomes broad**

On a failed run, a runner suggestion to retry one task becomes `ultrafuzz resume <run-id> --retry-failed`, dropping its `--node-id`. That command resets every failed task, so an operator following a single-node hint can also rerun unrelated failed work, including costly producers. The hint should preserve the target where possible or make its broader effect clear.

**Knowledge Base Used:** [Runtime orchestration](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/runtime-orchestration.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.