Skip to content

Starting 1,000 children in one workflow task still takes 64–86 s on hosted runners #1238

Open

Description

@rmcdaniel

Follow-up to #1203 and #1231, measured on #1235.

#1235 removes the quadratic steps from fan-out workflow tasks. A 1,000-activity all() now schedules in about 11–15 s and finishes in about 11–14 s on hosted runners. Starting 1,000 child workflows in one workflow task is still the slowest task:

100 children 1,000 children
Hosted runner, SQLite 3.0 s 86 s
Hosted runner, MySQL 4.7 s 64 s
lab4, SQLite — 28–39 s
lab4, MySQL 8.4 — 33 s

Under the default 60-second queue:work timeout, the 1,000-child task can still be killed on hosted-class hardware.

Each child costs about 60 queries inside the parent's transaction: the instance, run, link, child call, command, task and history rows; the child run's first projection; and the parent's own full summary/wait/timeline/lineage projection at the end, which writes about four projection rows per child. On hosted SQLite the per-child cost also grows with the transaction size (29× from 100 to 1,000 children, against 6.5× locally).

Required outcome: one workflow task that starts 1,000 children (10 KB arguments) finishes in well under 60 s on hosted runners on SQLite and MySQL. Recorded history and projections stay unchanged. The literal child case in the Fan-out scaling job (tests/Performance/FanOutWidthLiteralTest.php) passes without continue-on-error.

Related remaining cost, which is not this issue's completion condition: each member's own task (activity start/outcome, child closure) still rebuilds its run's whole summary once, so completing all members of a 1,000-member fan-out is quadratic in total even though every task fits the job timeout.

Activity

  1. rmcdaniel commented on Nov 19, 2026

    @rmcdaniel
    MemberAuthor

    On #1235's head (current main merged in, each kind in its own job), hosted runners start 1,000 children in 33.3 s on MySQL and 23.6 s on SQLite. Both are under the 50 s bound, so the child jobs are blocking again. The earlier 64–86 s came from running the child case after the activity case in the same process. #1235 now says "Closes #1238".

  2. rmcdaniel commented on Nov 19, 2026

    @rmcdaniel
    MemberAuthor

    Correction to my last comment: the next run of the same code took 65.1 s on hosted MySQL (it had taken 33.3 s), so 1,000 children are around the bound on hosted runners. #1235 keeps the child jobs non-blocking and no longer closes this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions