Skip to content

A 300-activity all() outlasts the default queue worker timeout, so the run never completes #1231

Open

Description

@rmcdaniel

Found while closing durable-workflow/.github#228 on published 2.32.0. Receipt and code.

A fresh Laravel 13.43.0 app with durable-workflow/workflow:2.32.0 from Packagist runs one all() of N activities with 10 KB arguments. Each workflow task's time grows roughly with the square of N:

Members Database Scheduling task Final task
100 SQLite 13 s 8 s
300 SQLite 134–140 s 80–85 s
100 MySQL 8.4.8 20 s 6 s
300 MySQL 8.4.8 135 s 58 s

The documented limit allows 1,000 members in one all().

With pcntl installed, the README's php artisan queue:work enforces Laravel's 60 s job timeout. Under it, the valid 300-member run never completes, even with the worker restarted after each stop:

  • The scheduling task is killed after it commits, with 115 of 300 activity jobs dispatched. Repair dispatches the rest after restart, and all 300 activities complete.
  • The final workflow task is killed at 60 s on every attempt and rolled back. It is leased again when the 4-minute lease expires. The run stays waiting, with no WorkflowCompleted and no typed failure, until the 25-minute observation ended.

This is the shape of .github#228's original defect: a fan-out within the documented limits blocks its run with no diagnosis.

Expected

  • Per-member cost is roughly constant, so a 1,000-member all() workflow task finishes well within 60 s on SQLite and MySQL.
  • A timing regression test runs at the documented width.
  • A task that still exceeds the worker's job timeout ends in a typed, named failure instead of retrying forever.
  • On the published release, a 300 × 10 KB all() completes under a default pcntl queue:work.

Possibly related: #1203 (quadratic child starts in one completion).

Activity

  1. rmcdaniel commented on Nov 19, 2026

    @rmcdaniel
    MemberAuthor

    #1235 makes the fan-out workflow tasks linear in width. 1,000 activities with 10 KB arguments now schedule and finish in about 11–15 s per task on hosted runners, down from about 1,900 s and 800–990 s locally on main. Starting 1,000 children is still 64–86 s on hosted runners (#1238).

    Stuck-run finding (proposal, not changed in #1235): RunWorkflowTask has $tries = 1, no $timeout and no failed() hook. The claim commits leased and attempt_count + 1 with a 300 s lease before the task executes. A pcntl timeout fails the job and exits the worker before the execution transaction commits. The task stays leased until its lease expires, and then TaskWatchdog redispatches it. attempt_count grows without bound and no failure is recorded.

    Proposed fix:

    1. At claim, detect a re-claim after an expired lease with no completion. Record a typed diagnostic and count consecutive lost leases.
    2. After N consecutive lost leases (configurable, default 3), make the run operator-visible: liveness workflow_task_repeatedly_lost with repair attention and a lifecycle event. Failing the run with a typed workflow_task_timeout failure changes semantics, so it should be opt-in.
    3. At worker start, warn once when pcntl is present and the job --timeout is below workflow_task_lease_seconds, recommending --timeout=0 or a value above the largest workflow task. Document this beside the README's queue:work.
  2. added 2 commits that reference this issue on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions