fix(cli): plain status, inspect and why print a successful poll's warnings, and the e2e covers a sealed engine relaunch

[aviggiano]

Problem

• Plain status hid the warnings the deadline docs tell operators to act on. Greptile on #1157. docs: state that workflow_deadline_seconds is only enforced on sync #1157 (docs: state that workflow_deadline_seconds is only enforced on sync) added this advice to docs/reference/configuration.md: "run ultrafuzz status <run-id> periodically (for example from cron) and act on its warnings". But commandFromRuntime (packages/cli/src/command-shared.ts) rendered a result's diagnostics only when the result failed. getRunHealth returns ok unless a diagnostic is an error, and these deadline and sync diagnostics are warnings: WORKFLOW_DEADLINE_CANCEL_FAILED, WORKFLOW_STATE_SYNC_SKIPPED, WORKFLOW_CONTROL_EVIDENCE_DIVERGED and the transient synchronization codes. So a cron job polling a run whose deadline cancel keeps failing, or whose sync is skipped because its control evidence diverged (the No supported way to re-seal workflow control evidence: one divergent file permanently disables product-level status and reporting #674 shape), saw exit 0 and Status: running-healthy (running) on every poll, while nothing cancelled the run. inspect and why render through the same helper. Only resume and init appended successful diagnostics themselves.

• No test checked that a sealed engine killed on its own gets relaunched by its supervisor. Greptile on #1199. fix(runtime): the Smithers supervisor can relaunch a crashed engine #1199 (fix(runtime): the Smithers supervisor can relaunch a crashed engine) fixed the relaunch of both plain-path and sealed engines. A sealed engine is what every fresh ultrafuzz run starts. But:

  • its real-engine test runs a plain-path engine with no snapshot descriptor;
  • the runtime fd-transfer harness runs a toy script;
  • the CLI e2e kills the engine and its supervisor together, then recovers through ultrafuzz resume.

  The sealed relaunch was checked only by hand, in fix(runtime): the Smithers supervisor can relaunch a crashed engine #1199's manual experiment. A later change to the sealed launch path, such as Re-evaluate sealed execution snapshots: cost/benefit after repeated campaign losses #921 or feat(runtime)!: run lifecycle commands from the pnpm-patched install instead of per-command npm installs (#921 step 1) #1201, could leave every engine-only crash of a fresh run orphaned until an operator resumes it. No test takes a sealed engine through a supervisor relaunch, so none would fail.

Change

1. commandFromRuntime prints a successful result's diagnostics after the rendered value. Each one prints on stdout as <severity>: <code>: <message>, and the exit status stays 0. Failures still lead with them. The duplicate appenders in commands/resume.ts and commands/init.ts are deleted. Source is +5/−9 lines. The JSON envelope is unchanged. In the docs:
   • The status section of docs/reference/cli.md says where these lines appear and that a message can span several lines. It also says that warnings do not change the exit status, so scripts should read the --json diagnostics array.
   • The deadline advice in docs/reference/configuration.md now tells cron jobs to poll ultrafuzz status <run-id> --json and act on the warnings in its diagnostics. It notes that plain status prints them on stdout.
2. The campaign e2e (packages/cli/test/e2e/campaign-resume.test.ts) first kills only the engine. Test code only, no production change:
   • While summarize is held, it SIGKILLs only the detached engine: the process whose argv has up and not supervise.
   • With no ultrafuzz command, it waits for the relaunched engine to hold summarize again in a new process. A new heldCall helper serves both waits.
   • Then it kills the whole controller, which is now the relaunched engine and its supervisor, and resumes. So ultrafuzz resume now follows a relaunched engine, not the engine ultrafuzz run started (see Risk).
   • The final checks change to match: summarize starts 3 times, the event log has 3 RunStarted, and it must contain a RunAutoResumed. The test title now names both kills.

Deliberately not fixed

• The relaunched engine does not finish the run. Greptile's finding asked whether a sealed launch's run finishes after a relaunch.
  • What the new phase shows: the supervisor relaunches a sealed engine, and the relaunched engine re-runs the interrupted node.
  • What it does not show: the e2e SIGKILLs that engine while it holds summarize. RunFinished comes from the engine ultrafuzz resume starts, and that engine runs the project-relative workflow file recorded in run.json, not the sealed snapshot.
  • The gap: a regression that breaks a later node, or the run's completion, only in a sealed relaunch would stay green.
  • Why not close it here: resume runs from plain paths, so one campaign cannot both let the sealed relaunch finish and resume after a mid-node crash. A second campaign would roughly double this test's time, which was 329–611 s per run on this host.
• The held stub is not killed separately. Smithers 0.35.0 spawns its CLI agents, the stub codex included, under @smthrs/agents/src/BaseCliAgent/parentDeathWatchdog.js (runCommandEffect → parentDeathCommand). The watchdog checks the engine's pid every 100 ms and SIGKILLs the agent's process group once the engine is gone. So killing the engine kills the held stub. origin/main's existing wait for "the controller and its agent to exit" already relies on this.
• No sealed variant of smithers-supervisor-relaunch.integration.test.ts. Draft feat(runtime)!: run lifecycle commands from the pnpm-patched install instead of per-command npm installs (#921 step 1) #1201 rewrites that file. The e2e drives the CLI black-box, so it keeps guarding the sealed path through feat(runtime)!: run lifecycle commands from the pnpm-patched install instead of per-command npm installs (#921 step 1) #1201 and the Re-evaluate sealed execution snapshots: cost/benefit after repeated campaign losses #921 rewrites.
• Some lines now appear twice, and I left them. This is cosmetic, and fixing it would change the renderers or the envelope. I observed both cases:
  • During launch preparation, plain status prints the WORKFLOW_CONTROL_SEAL_PENDING warning, and its message is also the Reason: line.
  • A run can be terminal in Ultrafuzz while the runner is still live. In the repro's scenario C, the deadline cancel succeeds while the fake runner still reports running. Plain status then prints both Lifecycle divergence: Ultrafuzz is terminal timed-out, but the workflow runner is running; ... and warning: RUN_WORKFLOW_STATUS_DIVERGED: .... From reading the code, a poll that fails but still returns a health value (a non-transient sync error) already printed both on origin/main, diagnostics first. A follow-up could delete lifecycleDivergenceLines in commands/status.ts (about −7 lines), because the same condition now also prints the warning, and point cli.test.ts's divergence assertion at the warning line.
• RunAutoResumed must be present, but its count is not checked. A relaunch that needs a second supervisor attempt under load still finishes the campaign, and this test is not meant to measure activation latency. RunStarted is still required to be exactly 3.

Verification

• New CLI test "plain status prints the warnings of a successful poll" (packages/cli/test/lifecycle-commands.test.ts). It sets a past workflow_deadline_at, and the fake runner answers cancel outside its status contract.
  • On origin/main (2cacf4c) it fails with AssertionError [ERR_ASSERTION]: The input did not match the regular expression /^warning: WORKFLOW_DEADLINE_CANCEL_FAILED: /mu. The input is the plain status text, which ends at Pace: 0 finished in the last 10m with no warning line. Both reviews reproduced this failure with main's text logic in packages/cli/dist. oclif loads commands from there, so rebuilding only dist-test does not test main.
  • It passes here.
• Existing tests, passing here after the last change:
  • all of lifecycle-commands.test.ts (13/13);
  • 9 cli.test.ts tests selected by name, most of which run plain-text commands: the product-evidence test, plain init, ps text, status launch-incomplete, status lifecycle divergence, status quota parking, resume already-active, references status and symlinked --project.
  • A review also ran all of cli.test.ts plus audit-profile-commands (88/91):
    • The 3 failures were two report bundle --require-verified tests whose ultrafuzz run --json failed with WORKFLOW_SUBMISSION_FAILED (... changed while reading) on a dependency file. That is the launch-snapshot race that x03 fixes.
    • Both tests passed when rerun alone.
• The verifier's repro, which I re-ran against this branch's build. It drives runCli with a fake runner. Every scenario exits 0, and the JSON is unchanged. On origin/main the same script printed no warning line in any scenario.
  • A, past deadline with a failing cancel: plain status now ends with warning: WORKFLOW_DEADLINE_CANCEL_FAILED: workflow runner command failed (exit 1), followed by the message's own stderr: runner busy line. Plain why prints the same warning.
  • B, diverged snapshot workflow file: both polls print warning: WORKFLOW_CONTROL_EVIDENCE_DIVERGED: ... and warning: WORKFLOW_STATE_SYNC_SKIPPED: ..., and plain events prints the first.
  • C, the deadline cancel succeeds: plain status prints the Lifecycle divergence: line and the RUN_WORKFLOW_STATUS_DIVERGED warning described above.
• Campaign e2e. This change is test-only and covers behaviour that already works on main, so the new check passes on main's code. Two ablations show that it discriminates.
  • Passes here in 329 s (load average about 5–13). The two reviews ran the same test, and it passed again in 373 s and 611 s. An earlier run of the same checks, instrumented to print process argv and events, also passed (469 s, load average about 15–20). In that run:
    • the relaunched engine held summarize 45 s after the engine SIGKILL;
    • that engine ran from the sealed snapshot: its argv held /proc/self/fd/3/... paths, --log-dir <run>/smithers/logs and --resume;
    • the events were RunAutoResumed ×1, RunStarted ×3, summarize NodeStarted ×3, then RunFinished;
    • every status, stats and report check passed.
  • Ablation of the relaunch root. I reduced fix(runtime): the Smithers supervisor can relaunch a crashed engine #1199's supervisor_resume_root patch to upstream behaviour (return direct;) and rebuilt the runtime.
    • The new e2e fails with AssertionError [ERR_ASSERTION]: timed out after 15 minutes waiting for the relaunched engine to rerun summarize. The supervisor log repeats Cannot resume run ...: detached log is unavailable at <run>/smithers/execution-snapshots/<gen>/.smithers/workflows/.smithers/logs/<id>.log, the pre-fix(runtime): the Smithers supervisor can relaunch a crashed engine #1199 sealed failure.
    • origin/main's e2e passes under the same ablation (910 s).
    • This ablation also breaks the plain-path relaunch, which fix(runtime): the Smithers supervisor can relaunch a crashed engine #1199's integration test catches. fix(runtime): the Smithers supervisor can relaunch a crashed engine #1199's ablation table records that test failing without this patch.
    • I reverted the ablation and rebuilt before the final run.
  • Sealed-only ablation, run by a review. It broke only the inherited-descriptor branch of fix(runtime): the Smithers supervisor can relaunch a crashed engine #1199's resume_snapshot_transfer patch, by pointing a sealed relaunch at a missing --config=/proc/self/fd/3/controls/ablated-bunfig.toml. The plain-path relaunch was left alone.
    • The new e2e fails with the workflow stopped while waiting for the relaunched engine to rerun summarize. The review reports that the supervisor log showed ENOENT for that config on attempts 1–3, after which the supervisor gave up.
    • origin/main's e2e passes under the same ablation (536 s). This is the gap the finding describes.
• Gates, all passing after the last change:
  • npx prettier --check and npx eslint on the changed files;
  • CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci. It also passes with the merge base as the diff commit, which is what CI's pull-request diff sees; origin/main has two later commits that touch no file here;
  • pnpm -w lint;
  • pnpm --filter @ultrafuzz/cli typecheck;
  • pnpm -w knip;
  • pnpm -w docs:check, which includes node scripts/docs-check.mjs.

Risk / compatibility

• Human text changes for every command that renders through commandFromRuntime. A successful result with diagnostics now prints them on stdout after its text, which resume and init already did. The affected commands are status (each --watch poll too), inspect, why, events, node, timeline, snapshots, validate, doctor, run, ps, pause, cancel, replay, fork, clean, materialize and references status|sync|update. The output of resume and init is unchanged.
• Scripts that scrape plain text will see extra lines. A script that scrapes plain status stdout will now see warning: lines. docs/cli.md already says human text is not the automation contract, --json is unchanged, and the status and deadline docs now point scripts at its diagnostics.
• Plain doctor repeats what its check lines summarise. It now also lists the validation, temporary-directory and registry warnings behind those lines. This is from reading the code.
• The e2e's resume phase now starts from a supervisor-relaunched engine. Before, it started from the engine ultrafuzz run started, so the e2e no longer covers a host crash before any relaunch. Resume takes the workflow path from Ultrafuzz's run metadata (start-run.ts), not from the engine that ran last. What differs is the state the relaunched engine wrote to the Smithers database.
• The e2e is longer. It adds one relaunch wait, bounded at 15 minutes. The relaunch took about 45 s in the instrumented run. The wait needs the sealed relaunch to activate within the 30 s supervisor stale threshold, the same requirement production has (see fix(runtime): the Smithers supervisor can relaunch a crashed engine #1199's Risk section). The e2e runs in the cli-e2e release-validation lane. CI runs that lane on every pull request and on pushes to main, with a 60-minute lane timeout (scripts/ci/release-validation-lanes.mjs). A broken relaunch therefore fails the lane after the 15-minute wait instead of passing.
• Overlaps with other work. The branch merges without conflicts into x01–x05, refactor!: remove per-node cloud execution (execution.mode = "cloud") #1197, feat(runtime)!: run lifecycle commands from the pnpm-patched install instead of per-command npm installs (#921 step 1) #1201, feat(topology)!: a failed property lens no longer skips the rest of the campaign #1198 and fix(runtime): record final-report producer selections in the run instead of querying smithers #1183. feat(runtime): let agents record roadblocks in a run-local friction log #1162 conflicts only in files this branch does not touch. Textual overlaps:
  • x05 edits commands/resume.ts just above the deleted block.
  • x04 changes why hints, and plain why now also prints its sync warnings. x04 and x05 also edit docs/reference/cli.md.
  • refactor!: remove per-node cloud execution (execution.mode = "cloud") #1197 edits both reference docs, and feat(runtime): let agents record roadblocks in a run-local friction log #1162 edits configuration.md.

Changelog entry

Plain-text ultrafuzz status (including each --watch poll), inspect, why, events, node, timeline, snapshots, run, ps, pause, cancel, replay, fork, clean, materialize, validate, doctor and references status|sync|update now print the warnings of a successful result on stdout after their output, as init and resume already did. Examples are WORKFLOW_DEADLINE_CANCEL_FAILED and WORKFLOW_STATE_SYNC_SKIPPED, which before only --json showed; the exit status is unchanged. The end-to-end campaign test now also checks that the supervisor relaunches a SIGKILLed engine of a fresh run, and that the relaunched engine re-runs the interrupted node.

Greptile follow-up

• Kept as is: diagnostics are appended to a successful result's text without a separator (comment). Every renderer that reaches commandFromRuntime ends its text with a newline, so the output is correct. A separator helper would guard only a hypothetical renderer.

🤖 Generated with Claude Code

[greptile-apps]

Warnings depend on renderer newlines The helper appends diagnostics without adding a separator. All current renderers end with a newline, so output is correct today, but a future renderer that does not would attach warning: to its final line and make the warning harder to recognize. Adding the separator here would keep that formatting requirement in one place.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/cli/src/command-shared.ts
Line: 137

Comment:
**Warnings depend on renderer newlines** The helper appends diagnostics without adding a separator. All current renderers end with a newline, so output is correct today, but a future renderer that does not would attach `warning:` to its final line and make the warning harder to recognize. Adding the separator here would keep that formatting requirement in one place.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!