fix(runtime): the Smithers supervisor can relaunch a crashed engine

[aviggiano]

Problem

Every Smithers engine Ultrafuzz submits runs with --supervise, so when the detached engine dies (SIGKILL, OOM kill) the Smithers supervisor should relaunch it and the campaign should continue. On origin/integration/wave1 the relaunch never finishes the run:

• An engine run from plain paths, the way ultrafuzz resume runs it: the supervisor relaunches three times, each relaunch dies before the engine activates, and the supervisor then marks the run failed (AUTO_RESUME_GAVE_UP). An engine crash that leaves the supervisor alive, such as an OOM kill of the engine alone, therefore turns into a failed run that only an operator can recover. The new real-engine test reproduces this on the base: RunStarted, NodeStarted, RunAutoResumed ×3, RunAutoResumeSkipped, RunFailed.
• A sealed launch (every fresh ultrafuzz run): the relaunch starts in <snapshot>/.smithers/workflows, and Smithers puts the relaunch's detached log below that directory, inside the read-only snapshot. Every poll fails with Cannot resume run …: detached log is unavailable at <snapshot>/.smithers/workflows/.smithers/logs/<id>.log. The failed spawn releases its claim, so the supervisor never relaunches, never gives up, and never emits RunAutoResumed. I reproduced this with a real ultrafuzz run on the base (manual experiment): 70 identical failures in the 12 minutes after the SIGKILL. ultrafuzz status then reported status: running, verdict: orphaned with 4 of 9 tasks finished, so the campaign stays stopped until an operator runs ultrafuzz resume.

Separately, Bun reads bunfig.toml (and runs its preload list) and .env from its working directory, which for every controller process is the target repository. The native-continuation command, and the patched detached-engine and supervisor spawns when they carry no snapshot descriptor, started Bun with no flags. A target's bunfig.toml preload therefore ran inside the controller processes, and its .env variables reached the engine. A probe on the base recorded the preload running in the top-level command, the detached engine and the supervisor, and the engine's task seeing the target's .env value.

Root cause

Three defects in the relaunch path, all in code Ultrafuzz patches or should patch:

1. ENOENT. The resume_snapshot_transfer patch (@smthrs/cli/src/resume-detached.js) always passed the /proc/self/fd/3/controls/... Bun startup flags. Only a process that inherited the snapshot descriptor has that path. A supervisor holding none relaunched into ENOENT: No such file or directory (open()) while reading config "/proc/self/fd/3/controls/bunfig.toml".
2. Wrong working directory. Upstream resolveResumeTarget relaunches a workflow file with cwd: dirname(workflowPath) (resume-target.js:99) and passes no --root. The workflow opens its store from its working directory (createSmithers: findSmithersAnchorDir(process.cwd()), smthrs/src/create.js:442). That anchor walk stops at $HOME (findSmithersAnchorDir.js:25), so with the target outside $HOME the relaunch created a new, empty smithers.db beside the workflow and failed with RUN_NOT_FOUND. The generated workflow also resolves its task paths from process.cwd() (workflow.tsx, e.g. dynamicRunRoot), so the working directory matters inside $HOME too. For a sealed launch, that directory is inside the read-only snapshot, and upstream resumeRunDetached derives the relaunch's detached log path from it (resolveDetachedRunLogFile(runId, { cwd })), so the relaunch can't even open its log.
3. Lost --log-dir. buildResumeArgs (resume-detached.js:40) rebuilds the relaunch argv without --log-dir. The relaunched engine then falls back to <root>/.smithers/executions/<id>/logs (@smthrs/engine/src/engine.js:2889), and <run>/smithers/logs/stream.ndjson stops receiving events, including the NodeFailed payloads (the comment in runSmithersLifecycleCommand records why that matters).

The Bun exposure comes from patched spawns passing [] when no descriptor is transferred, and from acquireSmithersExecutableAnchor passing no interpreter arguments for an unsealed native continuation.

Change

All runner changes go through the existing SMITHERS_COMPATIBILITY_PATCHES registry, and every new or rewritten text is covered by the existing anchor, helper-scope, patcher and doctor-posture tests.

• BUN_TARGET_CONFIGURATION_GUARD_ARGS (smithers-executable-capability.ts): --config=/dev/null --no-env-file --no-install --no-addons. Bun then reads no bunfig.toml, loads no .env, never auto-installs and loads no native addons, which the sealed startup controls already guarantee for sealed runs. This is the only copy of the flag list: the patch texts embed it (SMITHERS_BUN_GUARD_ARGS in smithers.ts), and the native-continuation branch and the new test import it.
• detached_snapshot_transfer and supervisor_descriptor: the no-transfer branch passes the guard instead of nothing.
• resume_snapshot_transfer: passes the descriptor-rooted startup flags only when a descriptor is inherited, and the guard otherwise (fixes 1).
• New supervisor_resume_root (src/supervisor.js, resolveResumeTargetEffect): a workflow-file relaunch starts in the rootDir the engine persisted in the run's config_json, read the way parsePersistedRootDir reads it for up --resume (fixes 2). Because the patch sits in the supervisor's shared target resolution, the stale-run, timer, approval and quota relaunch paths all use it. Without a persisted root it keeps upstream's target.
• New resume_log_dir (src/resume-detached.js, buildResumeArgs): the relaunch argv appends --log-dir $ULTRAFUZZ_SMITHERS_LOG_DIR. supervisor_descriptor sets that variable in the supervisor's environment from the launch's own --log-dir, so no Ultrafuzz environment plumbing is needed (fixes 3).
• applySmithersCompatibilityPatches applies the two new patches and requires src/supervisor.js.
• acquireSmithersExecutableAnchor: a Bun runner without startup controls (only a native continuation reaches that branch) gets the same guard flags.
• data-governance.ts: .smithers/logs joins controllerOwnedGovernancePaths. A relaunch now starts in the run's root, so upstream resumeRunDetached writes the relaunch's own output to <target>/.smithers/logs/<run-id>.log. That file is untracked in the target, so the next campaign's governed target identity would read the worktree as dirty, which a private campaign refuses (DATA_GOVERNANCE_PRIVATE_TARGET_UNBOUND).

Source: +114/-21 lines, most of it patch text and comments. Tests: +295/-34, including a shared test/patched-smithers-runner.ts that replaces the resume-reopen test's private copy of the same helper.

Deliberately not built

• No --root on the relaunch. With the working directory set to the persisted rootDir, up --resume already reuses that root for the task root. The store anchor and the workflow's own paths depend on the working directory, which --root would not change.
• No guard change on the manifest-conflict relaunch (manifest_relaunch). That relaunch chdirs into the @smthrs/cli package before re-executing, and the logical workspace is restored without a real chdir, so Bun never reads the target's bunfig.toml or .env there. I checked this with a jj-conflicted package.json plus a hostile bunfig.toml/.env on the base patches: the relaunch ran and no preload executed. A change there would be unobservable.
• No predecessor entries for the rewritten patch texts. Operator controllers are installed and patched fresh in each process. The only code that re-classifies runner sources patched by an older build is refreshedSmithersControllerSnapshot, which has no production caller (v03 deletes it).
• No BUN_OPTIONS guard. Detached children would inherit it, including agent CLIs and Bun-based target test commands. Explicit flags on the patched spawns do not leak.
• No re-routing of the relaunch's own launcher log into the run's log directory. That is the file the supervisor and the relaunched engine's detached child already append to. The upstream location is kept and excluded from governance instead.
• No other Smithers spawn sites (post-failure autopsy, monitor, gateway, hijack): Ultrafuzz suppresses or does not use them. No move to pnpm patchedDependencies (v10).
• Upstreaming the relaunch-root, --log-dir and supervisor-descriptor fixes to Smithers would shrink this patch set; that is outside this repository.

Verification

New real-engine test. packages/runtime/test/smithers-supervisor-relaunch.integration.test.ts uses the pinned Smithers 0.35.0 with every registry patch applied. The runner comes from test/patched-smithers-runner.ts, which smithers-resume-reopen.integration.test.ts now uses too. It copies every Smithers package and the store's hoisted node_modules, so each Smithers module loads once, from the copy. The store itself is never written. An strace of a toy smithers up on such a copy opened each of the 12 patched modules once, from the copy, and no Smithers module from the store. Without the hoisted node_modules copy, patched engine, scheduler and db modules also loaded a second, unpatched instance from the store. This was seen with strace both in review, on the first revision's helper, and again on the reopen test's helper. The setup:

• a toy two-task workflow in a run-owned directory, under a target outside $HOME;
• a hostile target bunfig.toml preload and .env;
• up --detach --supervise with a 1 s supervisor interval and a 30 s stale threshold, production's controller_lease_seconds default.

The test SIGKILLs the engine mid-task. It asserts:

• the run finishes;
• RunAutoResumed fires exactly once (resumeAttempt 1);
• the interrupted task re-runs in a new pid, and both engines ran with cwd equal to the launch root;
• <logDir>/stream.ndjson has two RunStarted and ends in RunFinished, and no <target>/.smithers/executions exists;
• no process ran the preload, and no engine saw the .env value.

Results:

• With the change: passes in 35–44 s (3 runs, load average 5–10).
• On origin/integration/wave1, with this test and helper copied in: fails in 125 s (RunAutoResumed ×3, RunAutoResumeSkipped, RunFailed; status failed). The wait for the run's end allows 180 s, so a regression reports the supervisor's give-up events instead of timing out.
• If the test fails, its cleanup kills every process whose command line names the fixture root (pkill -f on the regex-escaped root). The detached engine and supervisor run <root>/runner/@smthrs+cli@…/src/index.js, which the first revision's pattern (the smthrs runner path, whose + characters are regex quantifiers) never matched.

Ablations. Each run removes one change from the compiled registry, and each fails the final test. The log diagnoses in parentheses were recorded against the first revision of the test.

Ablation	Result
Resume flags always /proc/self/fd/3	125 s, RunAutoResumed ×3, RunAutoResumeSkipped, RunFailed (three relaunch logs with ENOENT)
No supervisor_resume_root	125 s, RunAutoResumed ×3, RunAutoResumeSkipped, RunFailed (every relaunch log says RUN_NOT_FOUND, stray smithers.db in the workflow's directory)
No resume_log_dir	36 s, the run finishes but the configured log directory has one RunStarted (the relaunched engine logs to <target>/.smithers/executions/<id>/logs/stream.ndjson)
No guard on the patched spawns	36 s, an engine loaded the target's .env

Other new tests (each passes with the change and fails on the base):

• smithers-executable-capability.test.ts, "native operator continuations run Bun without the target repository's bunfig.toml or .env". On the base, dotenv: 'loaded'.
• data-governance.test.ts, "a supervisor relaunch's log in the target leaves its governed identity unchanged". On the base, dirty: true.
• runtime.test.ts, "fixed fd transfer survives parent exit and fd reuse across Bun engine, supervisor, and resume" now also asserts that the sealed (fd-3) supervisor inherits ULTRAFUZZ_SMITHERS_LOG_DIR from the launch's --log-dir. This is the only automated check of that line on the sealed spawn chain. With the supervisor_descriptor environment line removed, it fails with actual: undefined.

Sealed launch mode (manual experiment on the first revision). The later revision changes nothing on this path; it only moves the guard flag list into one constant. I ran a real ultrafuzz run with the CLI e2e's three-agent topology, a stub codex and controller_lease_seconds = 30, so the engine ran from a sealed execution snapshot through the held descriptor:

• The detached engine was SIGKILLed while summarize was held. After the relaunch, the relaunched engine was SIGKILLed again while summarize was still held.
• The supervisor relaunched the engine both times, each on its first attempt (Resuming stale run … attempt 1 twice). summarize ran three times and the run ended RunFinished.
• ultrafuzz status reported succeeded with 9/9 tasks finished.
• <run>/smithers/logs/stream.ndjson holds all three RunStarted. No .smithers/executions directory and no stray smithers.db exist. The relaunch launcher's log is <target>/.smithers/logs/<id>.log, hence the governance change.

The same experiment on the base is the sealed-launch failure described under Problem.

CLI campaign e2e. packages/cli/test/e2e/campaign-resume.test.ts launches a campaign, SIGKILLs engine and supervisor, then resumes it through the native-continuation path, which gets the new Bun guard. It passes on the final revision in 296 s (load average 4–10), and its existing #1187 todo still reports as a todo. On the first revision it passed in 274 s at a load average of 12–16. An earlier attempt there, at a load average of about 45, failed before the resume step: an ultrafuzz status call printed nothing 350 s after the controller kill, which fits the helper's 5-minute kill timeout. status on a sealed run executes none of the changed code paths.

Existing tests:

• runtime.test.ts, the two harnesses that execute the supervisor_descriptor patch text: "the patched runner admits engine and supervisor process-owned execution snapshot descriptors" (Node) and "fixed fd transfer survives parent exit and fd reuse across Bun engine, supervisor, and resume" (Bun). The first revision broke both with ReferenceError: options is not defined: the patch reads options.logDir, which upstream's up -d scope binds and the harness scopes did not. Both harnesses now declare options, and both pass.
• runtime.test.ts patch tests, all passing:
  • compatibility patcher rewrites every described workaround
  • patches declare every ultrafuzz helper inside the module they patch
  • every runner compatibility patch still anchors in the pinned Smithers release
  • patched engine admits authenticated controller path changes
  • the 10 controller refresh tests
  • ordinary and refresh resume ownership tests
• runtime.test.ts native-continuation and native-resume tests: 9 pass. "native continuation hands generated agents the run's TOML config, so CodexAgent keeps API-key auth" fails as it does on the base (fixture ERR_MODULE_NOT_FOUND, fixed by fix: reconcile interactions between the merged stability batch and record it in the changelog #1191).
• lifecycle-inspection.test.ts: diagnoseProject reports a posture for every tracked compatibility patch.
• smithers-controller-engine.integration.test.ts (2 tests), which loads every registry patch into the engine.
• smithers-resume-reopen.integration.test.ts (3 tests), now on the shared runner helper.
• smithers-executable-capability.test.ts and data-governance.test.ts: every test passes.

Gates:

• pnpm -w format:check
• npx prettier --check and npx eslint on the changed files
• CI=1 ESLINT_PLUGIN_DIFF_COMMIT=origin/integration/wave1 pnpm -w lint:strict:ci
• pnpm -w lint
• pnpm --filter @ultrafuzz/runtime typecheck
• pnpm -w knip
• pnpm -w docs:check

Risk / compatibility

• Runs launched by earlier builds. Their running engines and supervisors keep the old patched code from their snapshot or retained controller, so their supervisor relaunch stays broken. ultrafuzz resume installs a fresh controller and gets the fix.
• Rewritten patch texts. In a runner patched by an older build, resume_snapshot_transfer, detached_snapshot_transfer and supervisor_descriptor would classify as incompatible. Only the dead refreshedSmithersControllerSnapshot re-classifies such sources.
• Relaunch working directory now comes from config_json.rootDir, which @smthrs/engine writes. If a future Smithers stops persisting it, the patch falls back to upstream's target. upstreamAbsent: ["rootDir", "parsePersistedRootDir"] flags a Smithers release whose supervisor starts reading the root itself.
• Activation window. A relaunch must activate within the supervisor stale threshold (controller_lease_seconds, default 30 s). If it doesn't, the supervisor claims another attempt and gives up after three. That is unchanged upstream behaviour, and it now matters because relaunches can succeed. In the sealed experiment, run at the 30 s lease on a host with load average above 30, both relaunches activated on their first attempt. The new test uses the same 30 s threshold. A sealed run whose relaunch misses activation three times now ends failed (AUTO_RESUME_GAVE_UP) instead of staying orphaned. Both states need ultrafuzz resume, and the engine accepts a failed run for resume (RESUMABLE_RUN_STATUSES, @smthrs/engine/src/engine.js:3434).
• Agents see the log directory. Agents inherit the controller environment minus a blocklist, so a relaunched engine's agents see ULTRAFUZZ_SMITHERS_LOG_DIR: the run's log directory path, not a credential.
• Governance exclusion. Untracked files under a target's .smithers/logs no longer count toward its governed identity. Tracked files there still do.
• Bun flags. Native continuations now run Bun with --no-install --no-addons, flags sealed runs already use.

Changelog entry

When the workflow engine dies mid-run, the Smithers supervisor now relaunches it and the run continues. Before, every relaunch failed: a freshly launched run stayed orphaned until an operator resumed it, and a resumed run was marked failed after three attempts. Relaunched engines keep writing events to the run's log directory, and controller Bun processes no longer load a target repository's bunfig.toml (or its preload scripts) or .env.

Refs #921

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking gap in automated sealed-launch relaunch coverage.

Findings

1.  Sealed relaunch lacks coverage ▶

Fix with agent prompt

### Issue 1
packages/runtime/test/smithers-supervisor-relaunch.integration.test.ts:45-59
The new real-engine test starts from plain paths without a snapshot descriptor. It therefore cannot catch a regression where a sealed launch fails to resume after the engine crashes. The descriptor test checks inheritance, but not whether the run finishes. An automated sealed-launch crash-and-relaunch test would cover that gap.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR patches Smithers’ supervisor relaunch to restore the persisted working directory and run log directory, and adds Bun startup guards for controller processes without snapshot controls.

• Adds a real-engine crash-and-relaunch test and shares its patched-runner helper with the resume-reopen tests.
• Excludes the supervisor’s untracked launcher logs from governed target identity.

_{Reviews (1) · Last reviewed commit: "test(runtime): the patch harnesses and r..."}

[greptile-apps]

Sealed relaunch lacks coverage The new real-engine test starts from plain paths without a snapshot descriptor. It therefore cannot catch a regression where a sealed launch fails to resume after the engine crashes. The descriptor test checks inheritance, but not whether the run finishes. An automated sealed-launch crash-and-relaunch test would cover that gap.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/test/smithers-supervisor-relaunch.integration.test.ts
Line: 45-59

Comment:
**Sealed relaunch lacks coverage** The new real-engine test starts from plain paths without a snapshot descriptor. It therefore cannot catch a regression where a sealed launch fails to resume after the engine crashes. The descriptor test checks inheritance, but not whether the run finishes. An automated sealed-launch crash-and-relaunch test would cover that gap.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.