fix(runtime): resume keeps agent auth config and a no-op attach no longer rewrites run state

[aviggiano]

Problem

ultrafuzz resume (submitSmithersContinuation in packages/runtime/src/start-run.ts) had four problems:

1. Resume lost the agent auth config. Native continuation set ULTRAFUZZ_CONFIG_PATH=<run>/smithers/resolved-config.json. All seven stock adapters (codex, claude, kimi, pi, opencode, openrouter, deepseek) read that path with the TOML reader in agents/toml.tsx. On JSON that reader finds no [agents.*] table and returns {}, so every adapter fell back to its defaults and silently dropped auth, api_key_env and config_dir. With the default ultrafuzz.toml ([agents.CodexAgent] auth = "api-key"), resumed Codex tasks were built with subscription auth and an empty OPENAI_API_KEY/CODEX_API_KEY.

2. A resume that started nothing still rewrote run state (Make pause, hijack, and continuation handoff atomic #1153). When Smithers reports the run as active, resume only attaches (alreadyRunning). That covers two cases:

   • a run still draining a pause;
   • a run whose controller process exited less than 30 s ago. The pinned Smithers' deriveRunState keeps a run running until its heartbeat is RUN_STATE_HEARTBEAT_STALE_MS (30 000 ms) stale.

   recordNativeContinuationState still ran on those attaches. It rewrote state.json and pushed workflow_deadline_at forward, silently extending the deadline that workflow_deadline_seconds is only enforced while an operator runs a CLI command; unattended runs have no wall clock #1110/docs: state that workflow_deadline_seconds is only enforced on sync #1157 document. The CLI printed Submitted resume: <id> whether or not anything was submitted.

3. A failed trusted-CLI check on resume dropped the run's launcher (Pin required executables and runtime dependencies for detached runs #1143, part 2). When prepareTrustedCliEnvironment threw, the error was swallowed and every trusted-CLI variable was cleared. One way it throws is a run launched under a different Node binary path, because the launcher bytes then differ from a fresh render. composeSmithersCommandPath then left <run>/trusted-bin off the engine PATH. Every task's preflight-json-validator step then ran whatever ultrafuzz was on the operator's PATH, or hit ENOENT.

4. Stale-worktree cleanup could fail the whole resume. A failed git worktree remove in repairPrunableRunWorktreeRegistrations threw out of resume. So did a non-ENOENT lstat error.

Root cause

1. fix(runtime): restore native Smithers continuation #961 wired the continuation to the JSON file. Launch gives adapters a copy of <run>/smithers/execution-config.toml (the execution snapshot's controls/ultrafuzz.toml). That file is the TOML rendering of the same resolved config (serializeResolvedConfigToml).
2. The call site passed alreadyRunning into recordNativeContinuationState. That flag skipped only the lease update; status and deadline were still rewritten. resume.ts ignored submitted.
3. The catch block treated "cannot re-verify" as "drop the launcher". But the launcher re-verifies its closure on every dispatch anyway.
4. Best-effort cleanup had no error boundary.

Change

• Config path. Resume points ULTRAFUZZ_CONFIG_PATH at <run>/smithers/execution-config.toml. It sets the variable only when resolved-config.json parses as the current ultrafuzz.resolved-config.v4 schema (feat(runtime): default to best-effort runs with agent-written PARTIAL reports #1120, 2026-09-09). The only writer of that JSON (compileSmithersWorkflow) writes the TOML next to it, and has done so since fix(security): complete remaining high-priority AppSec controls #635 (2026-08-18). So every run that reaches this branch has the TOML. Runs whose resolved config does not parse get no ULTRAFUZZ_CONFIG_PATH, as before.

• No-op attach. recordNativeContinuationState runs only when result.alreadyRunning !== true. Its now-unused alreadyRunning parameter and branch are deleted. When nothing was submitted, ultrafuzz resume prints:

  Run already active: <id>; no new controller was started. If a pause is still draining, resume again once status reports paused; if its controller process just exited, resume again after 30 seconds.

  The second clause is needed because status maps Smithers stale/orphaned to running, so it never reports paused for a dead controller. After the 30 s window the run is no longer active and resume submits a continuation. JSON output is unchanged; it already carried submitted: false.

• Trusted launcher. When trusted-CLI preparation throws and <run>/trusted-bin/ultrafuzz exists, ULTRAFUZZ_TRUSTED_BIN keeps pointing at <run>/trusted-bin. Resume never substitutes an ambient ultrafuzz for it. The failure becomes a WORKFLOW_TRUSTED_CLI_UNVERIFIED warning, and the warning says what follows:

  • Launcher present: … (tasks still call <run>/trusted-bin/ultrafuzz, and their preflight-json-validator step fails while that launcher cannot verify itself): <error>.
  • Launcher absent: … (<path> does not exist, so tasks call whatever ultrafuzz is on PATH): <error>.

• Worktree repair. The worktree-registration repair is wrapped in a try/catch. A failure becomes a WORKFLOW_WORKTREE_REPAIR_FAILED warning.

• Warnings are returned. Resume returns its warnings as diagnostics; on a failed resume they follow the error. The CLI prints them in text mode, as init does.

• Docs.

  • docs/reference/cli.md: the config sentence is scoped to runs with a current resolved config, and the attach paragraph now covers the 30 s window.
  • docs/schemas.md: the existence condition, the failure modes of a kept launcher, and that --refresh-controller does not repair one.
  • docs/reference/configuration.md: one sentence; the deadline is re-recorded only by a resume that starts a controller.
  • A CHANGELOG entry.

Source change: start-run.ts +72/−31, much of it re-indentation of the unwrapped lease block, and resume.ts +9/−5. The rest is tests and docs.

Deliberately not built (and why)

• No controller-generation fence, owner-fenced pause, or blocking pause (Make pause, hijack, and continuation handoff atomic #1153's proposal). The pinned engine already refuses a second controller while the owner is alive:

  • up --resume --force and timetravel --force fail with RUN_OWNER_ALIVE;
  • --force does not bypass this; --steal-ownership does, and nothing in this repo passes it (git grep);
  • a graceful pause drains rather than aborts.

  This PR pins those facts with a real-Smithers integration test instead of re-implementing them.

• No probe of, fallback from, or rejection over the kept launcher. Greptile P1 Improve agent tools #2 and both reviews flagged that file existence does not prove the launcher works. That is correct, and the warning and docs now say so. I kept the spec'd behaviour:

  • A fallback to the ambient ultrafuzz is what Pin required executables and runtime dependencies for detached runs #1143 removes.
  • Rejecting the resume would make historical validator identity a resume gate again, against Restore native Smithers continuation by removing Ultrafuzz hard blocks #939.
  • Some causes of the throw leave a working launcher: no CLI entrypoint (tested below), or a preflight timeout under load. Rejecting would stop those runs for nothing.

• No change to trusted-cli.ts. This is the root cause of the unrecoverable cases and is a follow-up. Resume cannot re-render a launcher whose embedded interpreter path changed. prepareTrustedCliEnvironment throws "trusted Ultrafuzz CLI launcher changed" before it reaches rotation, even under --refresh-controller. Fixing it means either re-rendering for the current interpreter on refresh, or dropping the interpreter from the launcher identity. Both change launcher identity handling for in-flight runs, so neither belongs in this PR.

• No sealed-copy indirection for the agent config (Greptile P1 Cleanup whitelisted commands #1: "read controls/ultrafuzz.toml in the execution snapshot, or verify the live bytes"). Main already handed resumed adapters the equally mutable <run>/smithers/resolved-config.json, so this is not a regression. Resume also takes credential names, lease and deadline from that same mutable file. And by design it runs the mutable persisted workflow (native resume delegates the persisted workflow after mutable project sources are replaced). A verification layer is the kind of fail-closed machinery this effort is removing.

• No regeneration of a missing execution-config.toml. By the argument above, every run that reaches this branch has it. If someone deletes it, adapters now fail loudly (ENOENT on the named path) instead of silently running on default auth.

• No change to the pre-resume inspect gate. One option is to always pass --force and map RUN_OWNER_ALIVE to alreadyRunning, which would close the 30 s window. That changes lifecycle behaviour, so the CLI message documents the window instead.

• No attempt-ledger work. The pre-reset preservation hook is untouched; w02 owns it.

Verification

Discriminating tests

Each fails on origin/main and passes with this change:

Test	Failure on main	With change
runtime.test.ts "native continuation hands generated agents the run's TOML config, so CodexAgent keeps API-key auth" (new). Resumes, then builds the generated CodexAgent from the ULTRAFUZZ_CONFIG_PATH the fake runner received.	CODEX_API_KEY is '' (subscription fallback)	carries the key
runtime.test.ts "ordinary resume checks active-run ownership before detached preflight" (extended). state.json must be byte-identical after both no-op attaches.	workflow_deadline_at moved on a submitted: false resume	identical
cli.test.ts "resume of an already-active run says no controller was started instead of claiming a submission" (new)	printed Submitted resume: ultrafuzz-resume-already-active	Run already active: …
runtime.test.ts "a resume that cannot re-verify the trusted CLI leaves tasks on the run's own working launcher" (new, fixer round). Resumes without a CLI entrypoint, so preparation throws while the launcher is intact. Resolves ultrafuzz through the runner's PATH, requires the run's trusted-bin/ultrafuzz, then runs the task's json validate call through it and parses the success envelope.	the lookup finds no ultrafuzz (undefined instead of <run>/trusted-bin/ultrafuzz)	passes
runtime.test.ts "native continuation does not use historical trusted CLI identity as an authorization gate" (extended with a warning assertion only)	no WORKFLOW_TRUSTED_CLI_UNVERIFIED diagnostic	warning on both resumes
runtime.test.ts "resume continues with a warning when stale task-worktree cleanup fails" (new). A fake git lists a prunable registration owned by the run and fails worktree remove.	WORKFLOW_LIFECYCLE_FAILED: failed to remove stale task-worktree registration	resume submitted, warning reported

How the main runs were done:

• For the two trusted-CLI rows, I (the fixer) reverted only start-run.ts to origin/main and recompiled.
• The other rows were checked by the implementer and independently by both reviewers, the same way. For the CLI test, the CLI dist was rebuilt as well.

Tests rewritten because they locked in the bug

• "native resume delegates the persisted workflow after mutable project sources are replaced" asserted ^config=.*/smithers/resolved-config.json$. It now asserts execution-config.toml.
• The Pin required executables and runtime dependencies for detached runs #1143 test used to assert the runner PATH only in the corrupted-metadata scenario. In that scenario the kept launcher cannot help, so that assertion checked the mechanism rather than the outcome. It is replaced by the working-launcher test above.

Kept launcher that cannot verify itself

I checked this by execution. In the corrupted-metadata scenario (trusted-cli.json overwritten with {}), after an ordinary and a --refresh-controller resume:

• <run>/trusted-bin/ultrafuzz json validate --json exits 1;
• both resumes return ok: true with the new warning;
• trusted-cli.json is still {}.

The missing-interpreter case (exit 127, exec: … not found) was reproduced by a reviewer, not by me.

Guard test (not discriminating by design)

smithers-preparation-race.integration.test.ts "graceful pause drains in-flight tasks and refuses a second controller until the run parks" runs on the real pinned smithers:

• Two held tasks start, and pause is requested.
• up --resume --force and timetravel --force both exit non-zero with RUN_OWNER_ALIVE.
• While the owner drains, inspect still reports running.
• The tasks are released and finish in the owner's PID. The run parks paused, and the downstream task never started.
• A plain resume runs only the downstream task, in a new PID. Each task ran exactly once.

Timing:

• Observed durations range from 7 s to about 108 s, depending on host load; the 108 s run was one of the reviewers', at load ~45.
• My runs: 27.6 s isolated at load ~27, and 10.0 s inside the full file. All 4 tests in the file pass.
• The critical window, from both tasks started to release, is a few seconds, well inside the 120 s hold.

Mutation check: replacing --force with --steal-ownership in the takeover makes the test fail. Depending on timing, it fails on the RUN_OWNER_ALIVE match after DETACHED_ADMISSION_FAILED/RUN_RESUME_CLAIM_FAILED (my run and one reviewer's), or with status 0 (the implementer's run).

Teardown (fixer round): the finally now waits, bounded at 30 s, for every process that traced a task to exit before deleting the root. Previously, on a failure the engine kept running until the 120 s hold expired and wrote logs back under the deleted root. In the --steal-ownership mutant, the owner had exited when I checked 5 s after the test returned, and 2.5 minutes later (past the 120 s hold) the root had not reappeared.

Also ran, at the final head 10ce4e5d

• runtime.test.ts: the six tests above plus "native resume delegates the persisted workflow…". Earlier rounds also ran the 38 runtime.test.ts tests that call resumeRun, the dynamic-lifecycle.test.ts resume test, and the cli.test.ts lifecycle test. The fixer round changed only the warning text and a comment in start-run.ts, and I did not re-run those.
• cli.test.ts "resume of an already-active run…".
• The whole smithers-preparation-race.integration.test.ts file.
• npx prettier --check on all changed files; npx eslint on changed sources and tests; CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/main pnpm -w lint:strict:ci; pnpm --filter @ultrafuzz/runtime typecheck; pnpm --filter @ultrafuzz/cli typecheck; pnpm -w knip; node scripts/docs-check.mjs. All pass.

Not run

• The full runtime and CLI suites.
• The Bun adapter-contract lane (no adapter template changed).
• A live campaign with real agent credentials.

Risk / compatibility

A kept launcher that cannot verify itself fails every task preparation after resume. When <run>/trusted-bin/ultrafuzz exists but its metadata or closure is damaged, or the Node binary it names is gone, each task's preflight-json-validator step fails. A likely way to lose the Node binary is an upgrade that deletes the old versioned binary, because the launcher embeds process.execPath; this is inferred, not reproduced. Resume still returns ok, with the WORKFLOW_TRUSTED_CLI_UNVERIFIED warning, which now says so.

• On hosts with no ambient ultrafuzz, which is the documented tutorial setup, main failed these tasks too, with ENOENT.
• On hosts that also have an ambient ultrafuzz, main fell back to it and may have completed. There this is a regression, accepted in exchange for never running a different CLI than the one the run launched with.
• --refresh-controller does not recover these runs (pre-existing). The follow-up is the trusted-cli.ts fix under "Deliberately not built".

Resume can now return warnings. On a successful resume, diagnostics can be non-empty (WORKFLOW_TRUSTED_CLI_UNVERIFIED, WORKFLOW_WORKTREE_REPAIR_FAILED), and text output gains those lines. Both the CLI and the evmbench result schemas accept warning severity when ok is true. The JSON data shape is unchanged.

Resumed runs now use the configured auth. A resumed run authenticates as configured, so api-key mode reads the configured key variable. Operators who were unknowingly running resumed Codex tasks on a ChatGPT subscription will now need the key set, which is what their config asked for.

Overlap with other open work. Checked with git merge-tree against the current heads:

• feat(runtime): let agents record roadblocks in a run-local friction log #1162 merges cleanly.
• ci: validate every package on PRs, stop cancelling main runs, and add a global complexity ceiling #1184 (w21) conflicts in packages/cli/test/cli.test.ts, because it changes tempProject() to tempProject(t: TestContext). Whichever PR lands second should convert the new CLI test to async (t) => { const project = tempProject(t); … }.
• fix(runtime): cancel and pause work even when run control evidence has diverged #1170, fix: failed launches are recorded and reported, and topology validation cannot reject compiler output #1185 and fix(runtime): attempt and usage ledgers never block run synchronization #1186 conflict only in CHANGELOG.md.
• This PR does not touch the pre-reset attempt-ledger hook (w02) or workflow-control.ts (w03).

Closes #1153
Refs #1143, #1110

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR does not appear safe to merge while an existing but unusable trusted launcher can shadow a working CLI and leave resumed tasks unable to validate.

Findings

1.  Broken launcher shadows working CLI ▶

Fix with agent prompt

### Issue 1
packages/runtime/src/start-run.ts:608-611
If a Node upgrade removes the executable embedded in the run-owned launcher, the launcher file still passes this existence check and takes first place on `PATH`. Resume reports success with a warning, but schema-backed tasks cannot run their `ultrafuzz` validation commands, even if another working CLI is on `PATH`. This case needs a recovery or explicit rejection path rather than treating file existence as proof that the launcher works.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR makes resumed agents read the run’s TOML configuration, avoids rewriting state on an active-run attach, retains an existing run-owned trusted launcher when verification fails, and reports best-effort repair failures as warnings. It also updates CLI messaging, documentation, and lifecycle tests.

_{Reviews (3) · Last reviewed commit: "chore: move the changelog entry to the c..."}

[greptile-apps]

Broken launcher shadows working CLI If a Node upgrade removes the executable embedded in the run-owned launcher, the launcher file still passes this existence check and takes first place on PATH. Resume reports success with a warning, but schema-backed tasks cannot run their ultrafuzz validation commands, even if another working CLI is on PATH. This case needs a recovery or explicit rejection path rather than treating file existence as proof that the launcher works.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/start-run.ts
Line: 608-611

Comment:
**Broken launcher shadows working CLI** If a Node upgrade removes the executable embedded in the run-owned launcher, the launcher file still passes this existence check and takes first place on `PATH`. Resume reports success with a warning, but schema-backed tasks cannot run their `ultrafuzz` validation commands, even if another working CLI is on `PATH`. This case needs a recovery or explicit rejection path rather than treating file existence as proof that the launcher works.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.