Skip to content

Surface workflow tasks that keep losing their lease (#1231) - #1244

Merged
rmcdaniel merged 12 commits into
mainfrom
general/1231-lost-lease-attention
Nov 19, 2026
Merged

rmcdaniel merged 12 commits into
mainfrom
general/1231-lost-lease-attention

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Implements the stuck-run proposal approved on #1231. It is separate from #1235. This revision applies the third review round: the workflow task job timeout is opt-in, the worker warning is reworded, summary writes tolerate a schema rollback, and the docs example routes activities to the bounded worker.

Problem

A workflow task whose worker is killed, times out or loses its connection loses its lease, and its work rolls back. It stays leased until the 300 s lease expires, and then task repair returns it to ready. With a short worker --timeout (60 s by default with pcntl), a long workflow task was killed on every retry. attempt_count grew, no failure was recorded, and nothing was surfaced to the operator.

Change

  1. Opt-in workflow task job timeout. By default RunWorkflowTask carries no job timeout, so the worker's --timeout applies exactly as before. Only when workflows.v2.workflow_task_job_timeout_seconds (DW_V2_WORKFLOW_TASK_JOB_TIMEOUT_SECONDS) is set does every workflow task job carry that timeout, kept 10 s below its connection's retry_after; Laravel's worker then uses it in place of its --timeout for those jobs. There is no blanket --timeout=0. Workers of earlier releases that unserialize a new job ignore the extra property (SerializesModels::__unserialize restores only declared properties; checked under error_reporting=-1 with no deprecation), and the queue payload's timeout stays null when unset.
  2. Worker-start warning. It fires once, only for a worker on the package's workflow queue and connection with pcntl, when the timeout its workflow tasks run under (the worker --timeout, or the opt-in job timeout) is below the workflow task lease. It says such tasks are retried only after the lease expires and counted as lost leases, and how to raise --timeout (keeping retry_after above it) or set DW_V2_WORKFLOW_TASK_JOB_TIMEOUT_SECONDS. --timeout=0 does not warn.
  3. Typed diagnostic at claim. Task repair counts each expired workflow-task lease (lost_lease_count, last_lease_lost_at). The next claim logs a typed workflow_task_lease_lost diagnostic, worded neutrally: "worker was killed, timed out, or lost its connection". No history is written.
  4. Additive attention (M2). At workflows.v2.workflow_task_lost_leases.attention_threshold (3) consecutive lost leases, the run summary, list items and describe carry an additive task_attention field: {reason: workflow_task_lease_lost, consecutive, threshold, workflow_task_id, last_lease_lost_at}. Operator metrics add runs.task_attention, tasks.lease_lost_repeatedly and tasks.max_lost_lease_count. The five repair_blocked_reason codes stay frozen, and repair_attention and can_repair are unchanged. The test asserts that every field an existing consumer reads is identical just below and at the threshold. Follow-ups to display or parse the field: Display the run summary's additive task_attention field waterline#339 and Parse the run summary's additive task_attention field server#1255.
  5. Opt-in failure, off by default. With fail_run enabled, the claim at the threshold fails the run with a typed WorkflowTaskLeaseLostException (task_failure) before any workflow code runs. Boot refuses fail_run with a threshold below 2.
  6. Migration (M1). 2026_11_18_000100_add_lost_lease_tracking sets $withinTransaction = false. Each column on the hot workflow_tasks and workflow_run_summaries tables is added under withSchemaLockBudget, as for Add opt-in deadline-triggered durable cleanup with an immutable total budget #863. Nothing is backfilled, and a repeated run skips completed steps.
  7. Rollback tolerance. The schema check is cached for at most 60 s. A missing-column error, caught in its own savepoint so the caller's transaction stays usable, degrades to untracked, as does SQLite reading an unknown quoted column as a string. Metrics re-check the schema on every snapshot. Task writes happen only where the row has the columns. A run summary write that hits a task_attention column dropped after the cached check forgets the cache and writes again without the field; the retry sits outside the projection's own transaction, and inside a caller's open PostgreSQL transaction the error is rethrown so the caller's retry sees the current schema.
  8. Docs. The README explains how the worker timeout, the lease and retry_after relate. Its example raises --timeout for the workers serving workflow tasks and routes activities to a bounded worker through the activity class's $queue (activities otherwise run on their workflow's queue), and it describes the opt-in from Workflow 2.34.0. docs/architecture/operational-liveness.md describes task_attention beside the frozen codes. The docs site is updated in Document the queue worker job timeout for workflow tasks durable-workflow.github.io#578.
  9. CI fixes carried from the first round.
    • The upgrade journeys' refused rollback targets the ledger's batch (scripts/ci/upgrade-journey-lib.sh), so a PR that adds a migration no longer fails them.
    • No issue references appear in docs.

Tests and evidence

  • V2WorkflowTaskLostLeaseTest (weighted) covers:
    • counting lost leases through the real TaskWatchdog repair pass, and the additive field at the threshold;
    • unchanged consumer fields and describe repair fields, and the metrics;
    • history unchanged by lost leases, the claim diagnostic, and attention clearing after the task completes;
    • the opt-in failure and its threshold, and boot refusing a threshold below 2;
    • the job timeout: none by default (the dispatched job's timeout is null), and when configured its value, the retry_after cap, and the timeout carried by the dispatched job;
    • the worker warning only on the workflow queue, only below the lease, never without pcntl or for --timeout=0;
    • degrading to untracked after a task column drop inside a transaction, and a summary projection succeeding after the summary column is dropped.
  • V2LostLeaseTrackingMigrationTest (weighted): no transaction, the lock budget used for each column, no backfill, and repeatable on populated tables.
  • Both pass locally on MySQL 8.4, PostgreSQL 17 and SQLite.
  • Replay-regression evidence workflow-task-lease-lost-fails-run-opt-in: the cold runner replays it. The consumer reproduces that history only with this change.
  • Related suites pass locally on MySQL: operator metrics, run detail, task dispatch, configured models, failure liveness, watchdog repair, the watchdog no-op task test, the replay corpus, migrations, list items and visibility filters.

Not changed

With fail_run off, run semantics and every existing consumer field are unchanged. #1235 is untouched; the only overlap with it is .github/feature-test-timings.json and CHANGELOG.md.

🤖 Generated with Claude Code

A pcntl queue:work kills a job at --timeout (60 s by default). A workflow task
killed there rolled back, stayed leased until its 300 s lease expired, and
task repair returned it to ready for the next worker to repeat. A task that
always needs longer than the timeout retried forever: attempt_count grew, no
failure was recorded and nothing told the operator why.

- Task repair counts each expired workflow-task lease on the task
  (lost_lease_count, last_lease_lost_at; additive migration, written only
  where the columns exist). The next claim logs a typed
  workflow_task_lease_lost diagnostic. No history is written and the run
  behaves as before.
- At workflows.v2.workflow_task_lost_leases.attention_threshold (3) lost
  leases, RepairBlockedReason reports workflow_task_lease_lost ("Task Timing
  Out", badge visible), so the run summary, describe and Waterline show repair
  attention. Operator metrics add runs.workflow_task_lease_lost,
  tasks.lease_lost_repeatedly and tasks.max_lost_lease_count.
- Opt-in workflow_task_lost_leases.fail_run (off by default): at the
  threshold the next claim fails the run with WorkflowTaskLeaseLostException
  (task_failure) before workflow code runs.
- A queue worker starting with pcntl and --timeout below the workflow task
  lease logs one warning (WorkerStarting). The README documents running
  workflow workers with --timeout=0 or above the longest workflow task.

V2WorkflowTaskLostLeaseTest covers counting, attention, describe, metrics,
the claim diagnostic, unchanged history, the opt-in failure and its
threshold, and the worker warning. Replay-regression evidence
workflow-task-lease-lost-fails-run-opt-in records the opt-in failure; its
consumer reproduces that history only with this change.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel and others added 5 commits November 19, 2026 16:31
The public boundary scan forbids cross-tracker issue references in docs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The state-machine journey upgrades from 2.33.0, which already has the
compensation ledger, so a candidate that adds a migration runs it in a new
batch. A plain migrate:rollback then undid only that newer batch and never
reached the ledger's refusal, failing the journey for any added migration.
expect_rollback_refused now rolls back the batch that holds the ledger
(migrate:rollback --batch), which the rollback preflight already checks, so
the journeys keep proving that retained scopes refuse a schema rollback.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel and others added 6 commits November 19, 2026 18:09
…imeout

Review of the lost-lease change:

- M1, migration lock: the columns are added outside the migrator
  transaction, each under the schema lock budget, and repeatably
  (2026_11_18_000100_add_lost_lease_tracking, as for the #863 column), so an
  ALTER behind an open task transaction fails fast instead of queueing reads.
- M2, describe contract: the five repair_blocked_reason codes stay frozen,
  and can_repair and repair_attention are unchanged. Lost leases are reported
  in a separate additive task_attention field ({reason, consecutive,
  threshold, workflow_task_id, last_lease_lost_at}) on the run summary, list
  items and describe. Operator metrics add runs.task_attention (alongside
  tasks.lease_lost_repeatedly and tasks.max_lost_lease_count). With fail_run
  off, every field an existing consumer reads is identical at the threshold.
- M3, timeouts: no blanket --timeout=0. RunWorkflowTask carries its own job
  timeout (WorkflowTaskJobTimeout): the lease less 30 s, or the configured
  workflow_task_job_timeout_seconds if lower, always 10 s below the
  connection's retry_after. A short worker --timeout no longer kills workflow
  tasks, and activities keep the worker's bounded --timeout. The worker-start
  warning now fires only for workers on the package's workflow queue, and
  only when retry_after caps workflow tasks. The README explains the lease,
  retry_after and timeout relationship and recommends a dedicated
  workflow-task connection.
- Lows: neutral wording (the worker was killed, timed out, or lost its
  connection); fail_run refuses a threshold below 2 at boot; the schema
  check is re-done every 60 s and on any missing-column error (in its own
  savepoint, so it never aborts the caller's transaction), and degrades to
  untracked after a schema rollback, including SQLite's quirk of reading an
  unknown quoted column as a string.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The bounded claim, repair and fresh-run projections already hold the run's
open workflow task, so they derive task_attention from it instead of
querying the task table again. A bridge test pins that budget.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Workflow task jobs carry their own queue timeout only when
workflows.v2.workflow_task_job_timeout_seconds is configured; unset, the
worker's --timeout applies as before. The worker-start warning now fires
when the timeout workflow tasks run under is below the lease, and says such
tasks are retried after the lease expires and counted as lost leases.

Summary writes tolerate a task_attention column dropped after the cached
schema check. The README example routes activities to the bounded worker.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit a2e0a0d into main Nov 19, 2026
51 of 52 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general Work owned by General

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant