Skip to content

Make worker liveness heartbeat-based so executions stop being falsely finalized - #14

Merged
TBxy merged 1 commit into
mainfrom
feat/heartbeat-liveness
Oct 7, 2026
Merged

TBxy merged 1 commit into
mainfrom
feat/heartbeat-liveness

Conversation

@TBxy

@TBxy TBxy commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

What & why

Liveness for RUNNING executions was pid-based, but a pid is only meaningful inside its own PID namespace. In container deployments (web pods serving the admin, a separate task-cluster pod) the stale sweep checked pids from the wrong namespace: every watched long-running command was falsely finalized as FAILED "worker lost" minutes into a healthy run, while recycled pids kept dead rows RUNNING forever. Liveness is now heartbeat-based (database-backed), which stays meaningful across pods.

Changes

  • CommandExecution: new worker_host, worker_started_at, last_heartbeat_at fields (migration included); worker refreshes the heartbeat every ADMIN_RUNNER_HEARTBEAT_INTERVAL (default 30 s) on a daemon thread
  • Sweeper finalizes RUNNING rows whose heartbeat is older than ADMIN_RUNNER_HEARTBEAT_STALE_AFTER (default 180 s); pre-heartbeat rows keep the legacy pid rules, now with a PID-reuse guard (a pid occupied by an older process counts as dead, not alive)
  • stop()/force_stop() only send signals when the worker ran on the local host — previously a Force Stop from a web pod silently no-oped, or could signal an unrelated local process on pid collision; the cooperative stop flag remains the cross-host mechanism
  • finalize_stale looks up the django-q2 Task by id (pk) instead of name: task_id stores the UUID, Task.name holds a humanhash, so timeout attribution never matched
  • django-q2 tasks enqueue with ack_failure=True (no broker redelivery of failed tasks); the per-task retry=-1 option was dropped (django-q2 silently drops unknown q_options keys); tasks get a readable admin-runner:<pk> name
  • sys.stdin is redirected to an EOF, non-tty stream while a command runs: interactive prompts (input()/click) now fail fast with EOFError instead of hanging the worker forever

Testing

uv run --group test pytest tests/ → 386 passed (new tests/test_heartbeat_liveness.py covers heartbeat sweep verdicts, pid-reuse guard, host-gated signals, q2 attribution by id, stdin EOF guard; tests/test_runners.py / tests/test_cancellation.py updated to the new enqueue/signal semantics). ruff check + ruff format + pyright clean.

Risks / follow-ups

  • Dead workers are now detected after the heartbeat-staleness window (default 3 min) instead of instantly via pid checks — the trade-off for namespace-safe detection.
  • Wodore-side (config only, later): raise ADMIN_RUNNER_SWEEP_INTERVAL and set ADMIN_RUNNER_STALE_AFTER explicitly.

Liveness for RUNNING executions is now heartbeat-based instead of
pid-based. Workers refresh last_heartbeat_at every
ADMIN_RUNNER_HEARTBEAT_INTERVAL (default 30 s) on a daemon thread, and
the stale-run sweeper finalizes RUNNING rows whose heartbeat is older
than ADMIN_RUNNER_HEARTBEAT_STALE_AFTER (default 180 s).

Rationale: a pid is only meaningful inside its own PID namespace. In
container deployments (web pods serving the admin, a separate task
cluster pod) the old sweep checked pids from the wrong namespace, so
*every* watched long-running command was falsely finalized as FAILED
'worker lost' minutes into a healthy run; conversely, a recycled pid in
the cluster pod kept dead rows RUNNING forever. The heartbeat flows
through the shared database and stays meaningful across pods.

Rows recorded before this change keep the legacy pid rules, now with a
PID-reuse guard (a pid occupied by a process older than the execution
counts as dead, not alive).

Also:

- stop/force_stop signals are only sent when the worker ran on the
  local host (worker_host recorded at claim); the cooperative stop
  flag remains the cross-host mechanism. Previously a Force Stop from
  a web pod silently no-oped - or could signal an unrelated local
  process on pid collision.
- finalize_stale looks up the django-q2 Task by id (pk) instead of
  name: task_id stores the UUID, Task.name holds a humanhash, so
  timeout attribution never matched.
- django-q2 tasks are enqueued with ack_failure=True so failed tasks
  are not redelivered after the broker retry window; the per-task
  retry=-1 option was dropped (django-q2 silently ignores unknown
  q_options keys) and tasks get a readable 'admin-runner:<pk>' name.
- sys.stdin is redirected to an EOF, non-tty StringIO while a command
  runs: interactive prompts (input()/click) now fail fast with EOFError
  instead of hanging the worker forever (commands ran with a TTY-
  reporting stdout, so prompts believed a terminal was attached).

Adds ADMIN_RUNNER_HEARTBEAT_INTERVAL and
ADMIN_RUNNER_HEARTBEAT_STALE_AFTER settings (docs updated).
@TBxy
TBxy merged commit 9efeb32 into main Oct 7, 2026
6 checks passed
@TBxy
TBxy deleted the feat/heartbeat-liveness branch October 7, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant