Skip to content

Keep fan-out workflow tasks linear in the member count (#1231, #1203) - #1235

Merged
rmcdaniel merged 16 commits into
mainfrom
general/1231-fanout-quadratic
Nov 19, 2026
Merged

rmcdaniel merged 16 commits into
mainfrom
general/1231-fanout-quadratic

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Fixes the quadratic fan-out workflow tasks in #1231 and #1203. The stuck-run part of #1231 is a proposal below and is not changed here.

This branch has current main merged in (2.33.0, including #1234's ParallelGroupAdmission). Admission counts the pending members once per group before any write. The per-member guards it keeps as a backstop use this PR's running bound, so the two compose. The only conflict was StructuralLimits.php: it keeps main's public pendingActivityCount/pendingChildCount/guard*Count and routes the per-member guards through the bound.

Profile before (2.32.0 main, real queue jobs, one task at a time)

Excimer sampling plus per-query logging on SQLite and MySQL 8.4 found these per-member steps that rescan the run:

Step Where Share before
Each published member task rebuilt the run's whole summary (TaskDispatcher::markDispatched → RunSummaryProjector::project), each in its own autocommit transaction scheduling task 90% (100 activities, SQLite)
WorkflowStepHistory::assertCompatible / assertParallelGroupCompatible scanned the whole history for every member final task 63–81%
WorkflowExecutor::activityCompletionEvent/activityOpenEvent, ParallelChildGroup::metadataPathForSequence/waitKindForSequence, ChildRunHistory::*ForSequence/latestLinkForSequence, RunLineageView::latestParentClosePolicyEventForChild scanned history or links per member final task; every child closure's parent projection child closures were cubic: 100 children did not finish in 10 min
ActivityReplayIdentity::assertCall scanned and decoded history until each replayed activity's ActivityScheduled event final task 57% at 500 activities
Each started child's call row, written after commit, committed by itself child scheduling ~17 ms per child on SQLite
StructuralLimits pending-child and pending-activity counts ran twice per member over a growing set scheduling 13 s of 58 s, 1,000 children on MySQL
ChildWorkflowNamespaceProjection read the parent run row twice per child child scheduling on PostgreSQL 40% at 300 children

Fixes

  • RunWorkflowTask publishes the tasks it committed in bounded batch transactions (50 per batch, one savepoint per task so one failure keeps the rest) and refreshes each affected run summary once. Atomic poll repair keeps its refresh inside its own transaction; faked runs keep the old path.
  • HistoryEventIndex memoizes per-sequence indexes over one loaded history collection, invalidated when the collection grows or is replaced. All the lookups above use it. For lookups that used to stop at the first match, find() indexes only as far as each lookup needs, so a single lookup still stops early and all lookups together read each event once.
  • Child-call rows written after commit join the same batched post-commit work, ahead of the tasks published after them. Each runs in its own savepoint and still reports a failure instead of throwing it.
  • Within one workflow task, the pending-member limit checks use a running upper bound. Below the limit or warning threshold the bound gives the same decision as the stored count; at or above it the rows are counted again. A new test fails a 5-member fan-out at a limit of 3 with the same kind, value and limit on main and on this branch.
  • Activity starts refresh only their own timeline points, like outcomes (one-line change in RunSummaryProjector's existing scoped-timeline condition). The scoped timeline and FailureSnapshots skip decoding payloads whose stored JSON cannot name the activity or a failure (StoredHistoryPayload).
  • TaskScopeMemo remembers the parent's namespace while one workflow task runs.

No limits or semantics change. Recorded history, and the final summary, wait, timeline, timer and lineage rows, are identical to main for activity and child fan-outs (compared field by field, masking generated ids and timestamps).

Numbers

Seconds per workflow task. Each cell is scheduling / final. These are local runs on lab4 (PHP 8.3, capped containers), with member tasks completed between them. The 30/300 rows use 1 KB arguments and the 1,000 rows use 10 KB. "Before" at 30/300 is current main (2.33.0). "Before" at 1,000 is 2.32.0, since a 1,000-member run on main takes hours.

Members Kind SQLite before SQLite after MySQL 8.4 before MySQL 8.4 after
30 activities 2.04 / 0.32 0.35 / 0.25 1.93 / 0.35 0.35 / 0.25
300 activities 133 / 23.5 2.6 / 1.7 141 / 22.7 3.9 / 2.1
1,000 activities 1,928 / 992 7.4 / 22.0 1,944 / 810 9.4 / 16.1
30 children 1.46 / 0.64 0.61 / 0.20 1.28 / 0.68 0.94 / 0.35
300 children 50 / 67 7.0 / 2.1 46 / 72 11.8 / 3.5
1,000 children not run (cubic closures) 28.2 / 11.8 not run 29.4 / 18.9

The 1,000-member "after" scheduling times were re-measured on the merged branch. The 1,000-member final times are from the branch before the merge (the merge does not touch the final task).

Hosted runners (Fan-out scaling jobs on this head, 1,000 members, 10 KB, main merged in): activities 11.2 / 14.4 s on SQLite and 12.6 / 9.9 s on MySQL; children 23.6 / 7.3 s and then 46.6 / 13.1 s on SQLite, and 33.3 / 10.9 s, 65.1 / 16.0 s and 44.6 / 13.5 s on MySQL across runs of the same code (head a29bf78: all four jobs pass). Activities are well under the 50 s bound. Children on hosted MySQL straddle it, so the child jobs stay non-blocking (#1238).

PostgreSQL 17, 300 children, scheduling: 41 s before the namespace memo and 27 s after it (1.8 s at 30). Before this change, completing 100 children did not finish within 10 minutes, because every child closure's parent projection was cubic. Now it takes 30 s.

Review fixes (head after review)

  • Failed push keeps its record. Publishing now runs in two phases. Inside the batch transaction each task gets the metadata a successful dispatch leaves, or the failure a refused dispatch records, and no step throws. After the batch commits, each job is pushed. A failed push records last_dispatch_error and an immediate repair_available_at on its own and restores the dispatch marks. The task is left as an unbatched failed publish leaves it.
  • No work continues in a dead transaction. A deadlock, lock wait or lost connection (TransactionFailure) aborts the batch, including from best-effort child-call writes. The whole batch then runs once more in a fresh transaction. A second such failure is thrown with nothing committed or pushed.
  • Push after commit. Jobs are pushed only after the rows they read, including child-call rows, have committed.
  • V2CoalescedTaskPublishTest covers each fault on MySQL and SQLite. Three of its four cases fail on the previous head.
  • The fan-out corpus consumer compares payloads independently of key order, because MySQL's JSON column reorders keys. It runs in the MySQL and PostgreSQL feature shards.

Tests and evidence

  • V2FanOutWidthScalingTest (feature, weighted): 30 vs 300 activities and children. It fails if either workflow task grows more than 25× (2.5× linear). On main the ratio is 60× (scheduling) and 65× (final) for activities.
  • tests/Performance/FanOutWidthLiteralTest.php and the new Fan-out scaling workflow: the literal 1,000-activity and 1,000-child fan-outs with 10 KB arguments, as separate jobs on SQLite and MySQL. It runs nightly, on dispatch and on PRs touching src/V2. Each workflow task must finish under 50 s and grow at most 25× from 100 members. The activity jobs are blocking; the child jobs report with continue-on-error until Starting 1,000 children in one workflow task still takes 64–86 s on hosted runners #1238.
  • Replay-regression evidence fan-out-activity-all-history-index and fan-out-child-all-history-index (recorded on this branch; history identical to main). The cold runner replays both to completion. The new consumer in V2EmbeddedReplayRegressionCorpusTest proves that the later members' lookups together read fewer than one pass over other members' events (8 here; 114 and 92 on main), and that a live run on this code records the evidence history unchanged.

The probes skip each member task's own run-summary refresh during the member phase (MemberPhaseProjectionRole) and refresh every summary once before the measured final task, so the tests stay bounded. See the remaining work below.

Remaining work (not in this PR)

  • Each member's own task (activity start, activity outcome, child closure) still rebuilds its run's summary from the whole run once. That is O(N) per member, so completing all members is still quadratic in total: about 1.4 s per activity at 300 members on SQLite. Those tasks are short and fit the job timeout, but a 1,000-member run does not complete in seconds. It needs an incremental summary/wait projection for member transitions. Tracked in Each fan-out member's task still rebuilds its run's whole summary #1240.
  • Starting 1,000 children is the slowest workflow task (Starting 1,000 children in one workflow task still takes 64–86 s on hosted runners #1238): about 28–29 s locally, but 24–65 s on hosted runners, at about 60 queries per child. Batching child creation would shrink it.
  • The final task still runs two full summary projections (claim and close). That is linear, but heavy at 1,000 × 10 KB.
  • On PostgreSQL, child scheduling at 300 is 27 s locally, mostly parent-row lock/update work per recorded event.

Stuck run (#1231): proposal only

RunWorkflowTask has $tries = 1, no $timeout and no failed() hook. The claim commits leased and attempt_count + 1 with a 300 s lease before execution. Under queue:work --timeout=60 with pcntl, the alarm handler fails the job and exits the worker. The execution transaction never commits. The task stays leased until its lease expires, and then TaskWatchdog redispatches it. Every retry repeats the same work and is killed again. attempt_count grows without bound and no failure is recorded.

Proposal:

  1. At claim, detect a re-claim after an expired lease with no completion. Record a typed diagnostic ("previous attempt ended without completing: worker timeout or crash") and count consecutive lost leases.
  2. After N consecutive lost leases (config, default 3), make the run operator-visible: liveness workflow_task_repeatedly_lost, with repair attention and a lifecycle event. Failing the run with a typed workflow_task_timeout failure would change semantics, so it should be opt-in.
  3. At worker start, warn once when pcntl is present and the job --timeout is below workflow_task_lease_seconds, recommending --timeout=0 or a value above the largest workflow task. Document this beside the README's queue:work.

Overlap

main is merged in. RunSummaryProjector.php has a one-line change in the existing scoped-timeline condition, outside #1229's hunks. StructuralLimits.php was resolved as described above. php.yml and validate-regression-corpus.py are untouched; the slow job is a separate workflow file.

🤖 Generated with Claude Code

rmcdaniel and others added 3 commits November 19, 2026 10:14
A 300-member all() of 10 KB activities took 134 s to schedule and 58-85 s
to finish on 2.32.0, so the default 60-second queue job timeout killed the
final workflow task on every retry. Profiling the real jobs on SQLite and
MySQL found per-member steps that rescanned the whole run:

- Publishing each member's task after commit rebuilt the run's whole
  summary, in its own autocommit transaction (90% of the scheduling task).
  RunWorkflowTask now publishes the tasks it committed in bounded batch
  transactions and refreshes each affected run summary once.
- Replay checked every member with WorkflowStepHistory scans over the whole
  history, and the executor, ParallelChildGroup, ChildRunHistory and
  RunLineageView looked up each member's events, links and policy outcomes
  the same way. Each lookup family now indexes the loaded history once
  (HistoryEventIndex), keyed by the collection and invalidated when it grows.
  A child closure's parent projection was cubic in the width before this.
- Activity starts refresh only their own timeline points, like outcomes, and
  the scoped timeline and failure snapshots skip decoding payloads whose
  stored text cannot name the activity or a failure.

Recorded history and the final projection rows are identical to 2.32.0 for
activity and child fan-outs. New replay-regression evidence records both
fan-outs; its consumer proves lookups for one member no longer read other
members' events (it fails on main) and that a live run records the evidence
history unchanged. V2FanOutWidthScalingTest bounds 300 members to 2.5 times
linear growth over 30, and the fan-out scaling job runs the literal
1,000-activity and 1,000-child cases.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Binding each new child to its parent's namespace read the parent run row
twice per child. On PostgreSQL those reads slowed as the scheduling task
kept updating that row, and they were 40% of a 300-child scheduling task
(41 s -> 27 s). TaskScopeMemo remembers the value only while one queued
workflow task runs, so no value is trusted across tasks.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Measured locally at 201 s on MySQL 8.4 and 297 s on PostgreSQL 17 under
load. The MySQL weight adds headroom for its pcov coverage shards and hosted
runners; the PostgreSQL weight adds headroom for hosted runners.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel added the agent:general Work owned by General label Nov 19, 2026
rmcdaniel and others added 6 commits November 19, 2026 10:29
The first run lost a package download to an HTTP 429 from codeload.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The literal 1,000-activity run showed the final workflow task growing
faster than linearly (1.5 s at 100 members, 40 s at 1,000). At 500 members
57% of it was ActivityReplayIdentity::assertCall, which scanned and decoded
the whole history for every replayed activity until it found that member's
ActivityScheduled event.

HistoryEventIndex::find now indexes a loaded history only as far as each
lookup needs: one lookup stops at its match as the old scan did, and all
lookups together read each event once. The replay identity check and the
first-match lookups (parallel metadata, wait kind, child scheduled and
resolution events, activity completion and open events) use it, so a member
task that looks up one sequence no longer decodes the whole history to build
an index.

Recorded history and final projection rows remain identical to main. The
fan-out corpus consumer now also runs the replay identity check.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…1203)

Every child a workflow task starts records its child-call row after the
task commits. Each row committed alone, so a 1,000-child scheduling task
spent about 17 ms per child waiting on its own commit on SQLite. Inside the
coalescing scope those best-effort writes now join the batched post-commit
work ahead of the tasks published after them, each in its own savepoint, and
still report rather than throw a failure. 1,000 children: 58 s -> 39 s.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
)

Before every fan-out member, the scheduling task counted the run's stored
pending activities or open child links twice (limit guard and warning). On
MySQL those counts were the largest growing step of a 1,000-child scheduling
task (13 s). Within one workflow task the first check counts the rows and
each later guard adds the member scheduled since; members that close
meanwhile only lower the stored count, so the value is an upper bound. Below
the limit (or warning threshold) the bound decides exactly like the stored
count; at or above it the rows are counted again and that count is used.

A new case fails a five-member fan-out at a pending limit of three with the
same kind, value and configured limit on main and on this branch.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Starting 1,000 children with 10 KB arguments is the slowest workflow task:
about 33 s on MySQL and 39 s on SQLite locally after this branch's fixes,
against more than 85 s on hosted runners before them. Keep the bound inside
the 60-second default job timeout with room for hosted-runner variance.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Hosted runners measured the literal fan-outs on this branch: 1,000 activities
schedule in 11-15 s and finish in 11-14 s, but starting 1,000 children takes
86 s on SQLite and 64 s on MySQL. Run activities and children as separate
jobs, and let the child job report without failing the build until #1238
brings it under the bound.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel and others added 2 commits November 19, 2026 14:51
…uadratic

# Conflicts:
#	src/V2/Support/StructuralLimits.php
Run as its own job on the branch merged with current main, hosted runners
start 1,000 children in 33.3 s on MySQL and 23.6 s on SQLite, inside the
50-second bound (they took 64 s and 86 s when the child case ran after the
activity case in the same process). The child jobs no longer need
continue-on-error (#1238).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The same code started 1,000 children in 33.3 s and then in 65.1 s on hosted
MySQL runners, so the child case is around the 50-second bound on hosted
hardware. Report it without failing the build until #1238 lands.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel and others added 3 commits November 19, 2026 16:52
…review)

Review of the coalesced publish found three faults:

- A failed push recorded last_dispatch_error and repair_available_at inside
  its savepoint and then rethrew, so the savepoint rolled the record back.
  Publishing now runs in two phases. Inside the batch transaction each task
  gets the metadata a successful dispatch leaves, or the failure a refused
  dispatch records, and nothing is thrown. After the batch commits each job
  is pushed. A failed push records its failure on its own and restores the
  dispatch marks, leaving the task as an unbatched failed publish does.
- A deadlock, lock wait or lost connection ends the whole transaction, but
  the batch caught it and kept going. Such failures (TransactionFailure) now
  abort the batch, including from best-effort child-call writes. The whole
  batch then runs once more in a fresh transaction, and a second such failure
  is thrown with nothing committed or pushed.
- Jobs were pushed while their batch was uncommitted, so a child could finish
  before its call row existed. Pushing after commit closes that gap.

V2CoalescedTaskPublishTest covers each fault on MySQL and SQLite; three of
its four cases fail on the previous head. The fan-out corpus consumer now
compares payloads independently of key order, which MySQL's JSON column
changes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit 09118f9 into main Nov 19, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general Work owned by General

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant