Skip to content

orchestrate: coordinator-side status writes pick the wrong tmux session, and a stale 'failed' phase has no documented reset path #97

Description

@dch0202-rsquare

Summary

Three related problems in the coordinator-side state channel. Together they mean: after a worker reports failed and the coordinator answers its question, there is no clean documented way to put the task back in flight — and the obvious attempt silently corrupts another task's record.

Observed while orchestrating NEWRTB-2786 (2 tmux workers, dev-loop 1.6.0, macOS).


(a) status-update.sh writes the WRONG session name when called from a non-tmux shell

status-update.sh resolves the session as:

sess="${STATUS_SESSION:-$(tmux display-message -p '#S' 2>/dev/null || true)}"

The coordinator runs outside tmux. There, tmux display-message -p '#S' does not fail — it returns the most recently active session, which is some other worker.

What I ran (resetting t1 after answering its question):

STATUS_DIR=... sh scripts/status-update.sh t1 pending worktree=...

What it wrote into status/t1.json:

{ "task": "t1", "phase": "pending", "session": "lo-4-2786" }

lo-4-2786 is t4's session. Since watch-status.sh uses the session field for dead-worker detection, t1 would now have been declared dead if t4 ever exited — and a genuinely dead t1 would have gone unnoticed.

STATUS_SESSION exists as an override and fixes it, but nothing in SKILL.md tells a coordinator to set it. The skill does instruct the coordinator to write status in other places, so the non-tmux caller is an expected caller.

Suggested fix: treat "not inside a tmux client" as unknown session rather than whatever is active. tmux display-message only means "this shell's session" when $TMUX is set:

if [ -n "${STATUS_SESSION:-}" ]; then sess="$STATUS_SESSION"
elif [ -n "${TMUX:-}" ]; then sess=$(tmux display-message -p '#S' 2>/dev/null || true)
else sess=""   # unknown - leave the field untouched rather than guessing
fi

(The existing jq already does if $s != "" then .session=$s else . end, so an empty value correctly leaves the prior value in place.) Plus a line in SKILL.md telling coordinator-side callers to pass STATUS_SESSION.


(b) watch-status.sh aborts on a stale failed phase, with no documented reset

Sequence:

  1. t1 verified the plan, found it wrong, called ask-coordinator.sh, and recorded phase=failed.
  2. watch-status.sh exited 6 (question pending). Correct.
  3. I answered via send-prompt.sh send, deleted questions/t1.json, and relaunched the watch — per the exit-6 playbook in SKILL.md.
  4. The watch immediately exited 3: [watch ->plan_ready] 0/1 | t1:failed / [watch] failed session detected — abort.

The worker was alive and actively re-planning (confirmed by capture-pane). The failed phase was simply stale — nothing in the answer-and-relaunch playbook clears it.

The exit-6 playbook says "answer, delete the record file, relaunch watch". That is incomplete when the worker also wrote failed, which is the normal shape for "I stopped because the plan was wrong" — the very case ask-coordinator exists for.

Suggested fix: either

  • have ask-coordinator.sh record a non-terminal phase (e.g. blocked) instead of leaving the task failed, so answering it is enough; or
  • extend the exit-6 playbook in SKILL.md to include resetting the phase (and say to pass STATUS_SESSION, see (a)).

(c) The error field survives into later phases, producing self-contradicting records

status-update.sh merges rather than replaces, so a task that recorded phase=failed with an error string and later reaches plan_ready carries the stale failure text:

{
  "task": "t1",
  "phase": "plan_ready",
  "error": "plan verification failed - not adopted, no code written. ..."
}

Nothing reads error today, so this is not a live bug — but the re-entry procedure in SKILL.md tells a resuming coordinator to "measure real state first" from these files, and a plan_ready record carrying a failure message is exactly the kind of thing that misleads that read. I cleared it by explicitly passing error="".

Suggested fix: clear error whenever the phase advances to a non-failed phase, or document error as valid only when phase == failed.


Environment

  • dev-loop 1.6.0
  • macOS (Darwin 25.5.0), tmux from Homebrew
  • substrate: tmux (chosen at Gate 1)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions