An interrupted teardown hook leaves a managed environment retrying forever, with no backoff and no recovery command

[mattwyckhouse]

Environment. bb 0.43.3 desktop; git-worktree provider; repositories with a
.bb-env-teardown.sh hook.

Symptom. After a host-daemon restart interrupts a teardown hook, the environment stays in
status: error, lifecycle.teardown.status: failed with the message
Environment hook outcome is unknown after interruption. Automatic cleanup is blocked; inspect the workspace before recovering it.
and is retried once per minute indefinitely. On one host 18 environments are in this state with
attempt counters between 2,981 and 8,305 (about 5.8 days at one attempt per minute), each attempt
costing a host RPC with a 6 s timeout; 10 of their worktrees still exist (7.6 GB) and 8 are
already gone, yet none can complete. bb guide environments documents that a lost hook blocks
cleanup and "requires inspection before recovery", but there is no recovery verb, no filter to
find such environments, and no bound on the retries.

Evidence (installed bundle server/dist/start-server.js).

• cancelPendingEnvironmentHook (209864–209884): if the environment_hook_operations row has no
  finished_at, it sends environment.hook.cancel with TRANSPORT_GRACE_MS (6 s, 209894); when the
  host answers unknown it throws the message above before the finished_at update that
  follows, so the row stays pending.
• runEnvironmentHook's catch awaits that cancel before logging "continuing removal"
  (209849–209857), so the throw escapes to runRemove.
• runRemove (239037–239110) catches it and writes teardownStatus: "failed",
  retireAt: Date.now() + REMOVE_RETRY_MS (60 s, 239905). The lifecycle sweep re-enters as soon as
  retireAt passes (239186–239187), increments teardownAttempt (239189) and takes the identical
  path. No backoff, no cap, no terminal state.
• Recovery: no route or CLI clears teardown_status for an environment; the only reset is the
  provisioning upsert (34561), which is not reachable as an operation. bb environment delete
  enters the same removal path.

Query to find affected rows. SELECT count(*) FROM environment_hook_operations WHERE finished_at IS NULL;
and SELECT id, teardown_attempt FROM environments WHERE teardown_status = 'failed';

Reproduction. Commit a .bb-env-teardown.sh that sleeps for a minute, archive the last thread
of a worktree environment, and restart the host daemon while the hook is running. The environment
enters teardown failed and retries every 60 s from then on.

Proposed fix.

1. On the unknown branch, mark the hook operation finished with an unknown error (and record
   the message on the environment) so the next attempt can proceed to provider.remove; the
   guide already states there is no cross-restart process tracking, so the row cannot be resolved
   later anyway.
2. Exponential backoff on the retry interval and a cap on teardownAttempt, after which the
   environment enters a terminal teardown: blocked state that stops the RPCs.
3. bb environment recover <id> (clear the teardown state after confirming no live threads; treat
   a missing path as removed) and bb environment list --teardown failed.
4. Docs: state the retry cadence (currently 60 s, no backoff) and the recovery procedure next to
   the existing sentence about the unknown-outcome block.

---

Filed from an audit of bb 0.43.3 by the Factory plugin (github.com/mattwyckhouse/bb-plugin-factory). Method, re-derived numbers and the adversarial review of this report: plans/076-host-sustainable-line/bb-core-issues/REVIEW.md in that repository.

[bb-slop-cop]

🚨 SLOP COP 🚨 · new-issue-autopilot

Classified as Bug · Medium priority · High effort; applied workspaces and confirmed-repro.

REPRODUCED · high root-cause confidence. Report and regression test. Two clean checkouts reproduced four failed teardown attempts, each rescheduled after 60 seconds, with the hook unfinished and zero provider-removal calls. The nine existing daemon hook tests passed.

Unknown hook cancellation throws before finishing its operation; the removal catch then schedules the same unresolved operation again. The test models daemon memory loss; reporter fleet counts and disk usage were not verified. environment list --status error provides a broad filter.

No PR: safe recovery requires a product decision about unresolved hook processes, outside the simple-fix limits.

[RichieGP]

Hitting the same class of bug, slightly different trigger: a manual-provider machine whose physical hardware was permanently decommissioned (replaced, wiped, never coming back online).

• bb machine remove <id> --yes accepts the request, machine enters lifecycle.phase: "removing" with teardown: {"status":"failed","attempt":0}.
• bb machine retry-cleanup <id> returns success ("cleanup retried") but attempt never increments past 0 and status stays "failed" — tried multiple times over several minutes, no change.
• bb machine reconcile <id> → 409: Machine provider "manual" does not support suspend and resume.
• Tried bb machine reconnect <id> as a possible escape hatch (since it can adopt the host ID from different hardware per #<PR 4209>) → 409: Only active machines other than the server's own can be reconnected. So once a manual machine is in removing, reconnect is blocked too — no path back to active to retry reconnect, and no path forward to complete teardown. Dead end either direction.

No CLI/API force-delete exists for this state, matching what's described here. The machine just sits in bb machine list forever as a disconnected zombie entry. Would like either:

• a documented/CLI way to force-drop a manual machine's record when its host is known to be gone for good (skip teardown entirely, since manual machines have no cloud resource to tear down anyway), or
• bb machine reconnect to be allowed against a removing-phase machine so a live daemon (even on different hardware) can pick up the ID and let teardown complete normally.

[WyrdWerk]

Still reproduces on bb-app 0.45.0 (server + host daemon), on a remote Linux machine
(Ubuntu 24.04) enrolled through a machine-provider plugin, git-worktree environments, repository
without a .bb-env-teardown.sh.

Additional observations:

• It also blocks machine removal. bb machine remove puts the host in phase removing and
  waits for these environments; it never completes. If the machine is offline (stopped or deleted),
  the message becomes Host is not connected, and there is again no way out: no --force on
  machine remove, and bb machine retry-cleanup repeats the same path.
• bb environment cleanup <id> returns ok but changes nothing (attempts kept rising 8 → 14).
  bb thread unarchive doesn't reset the lifecycle either.
• Trigger in our case: archiving worktree threads and suspending the machine (or restarting its
  host daemon) seconds later, while the teardowns were still pending.

Workaround that worked, for anyone stuck (back up bb.db first; server running is fine). For the
affected host only, do in one transaction what markHostEnvironmentsDestroyed does, plus close
the pending hook rows:

UPDATE environments
   SET path = NULL, resource = NULL, retire_at = NULL, status = 'destroyed',
       teardown_message = NULL, teardown_status = 'removed',
       owner_thread_id = NULL, claim_path = NULL, updated_at = <now_ms>
 WHERE host_id = '<host_id>' AND status != 'destroyed';
UPDATE environment_hook_operations
   SET finished_at = <now_ms>, error = 'manually resolved: hook outcome lost'
 WHERE host_id = '<host_id>' AND finished_at IS NULL;

Then bb machine retry-cleanup <host>. Removal completed and the machine provider's remove ran.

+1 to a supported recovery verb (e.g. bb environment recover <id> --discard-hook and a
machine-level --force for offline/deleted machines), or bounding the unknown retries.