Skip to content

Large fan-outs within documented limits block their run with a 413 #228

Open

Description

@rmcdaniel

Current delivery and remaining acceptance

Native Server. Server 3.9.1 added the explicit 32 MiB completion allowance (producer qualification). Native Server 3.13.0 defaults the history byte budget to 4× the completion cap. A refusal is a typed 409 history_transaction_limit_exceeded with limit_kind, current_value and configured_limit.

SDKs (delivered).

Binding Published-package evidence
Python 2.17.1: 300 × 10 KB, structural over-limit failure, zero partial activities, same-worker follow-up. 2.18.0+ fails the run once on the 3.13.0 typed 409 (sdk-python#337).
Rust 4.16.0: fresh registry consumer receipt. 4.20.0 fails the run once on the 3.13.0 typed 409 (sdk-rust#291), with an installed test against 3.13.0.
PHP 3.18.1: normal-registry Worker acceptance (run 43440828565). 3.21.0 fails the run once on the 3.13.0 typed 409 (sdk-php#435), with an installed test against 3.13.0.

Docs (delivered). Root docs adoption. The structural limits page links the aggregate wire limit beside the per-command limits.

Embedded Workflow (partly delivered). Receipt on Packagist durable-workflow/workflow 2.32.0 (evidence):

  • 300 × 10 KB completes on SQLite and MySQL.
  • 1,001 members are refused atomically with command_batch_size.
  • An oversized member is refused with payload_size.
  • The same worker completes a follow-up run.

Remaining: workflow#1231. Workflow-task time grows quadratically with fan-out width (about 135 s to schedule 300 members). Under a default pcntl php artisan queue:work, whose job timeout is 60 s, the final task is killed on every attempt, and the run stays waiting without a typed failure. Close this issue when a released Workflow completes 300 × 10 KB under that default worker. workflow#1232 (non-atomic payload-size refusal inside all()) is follow-up work and does not block this issue.

Problem

A fan-out that stays within every documented limit can block its run permanently. The limits allow up to 1,000 operations in one all() and 2 MiB per argument. But a workflow task's commands are sent in one completion request, and the Server rejects any request body over 2 MiB.

Reproduced on published Server 2.9.9 with Python SDK 2.6.1:

  • A workflow that schedules 50 activities with 10 KB arguments in one parallel group completed.
  • One that schedules 300 such activities (about 3 MB in total) got 413 payload_too_large on workflow-task completion.
  • The worker recorded "workflow task completion failed after commands were produced". The run became workflow_replay_blocked with a failed task, no retry, no activities and only StartAccepted and WorkflowStarted in history.

Repair re-runs the same code into the same limit, so the run can't progress without a code change. The diagnosis says "replay blocked" rather than naming the size limit that was hit.

Acceptance criteria

  • A workflow task whose commands are each within the documented limits completes, in the PHP, Python and Rust SDKs and embedded mode. For example, large arguments are offloaded through the runtime payload transport, or commands are split across requests.
  • Anything that still can't be accepted fails with a typed structural-limit failure that names the limit and the measured size. It is not recorded as a replay block.
  • The structural limits page states any per-task aggregate limit beside command_batch_size and payload_size_bytes.
  • Tests against a published Server cover a 300 × 10 KB fan-out and the over-limit case in each SDK.

Activity

  1. added
    bugSomething isn't working
    kind:defectA public product behavior is incorrect
    kind:cross-repositoryWork spans more than one public repository
    priority:P2Normal-priority product work
    repo:workflowOwned by the Workflow engine repository
    repo:serverOwned by the standalone server repository
    repo:sdk-phpOwned by the PHP SDK repository
    repo:sdk-pythonOwned by the Python SDK repository
    repo:sdk-rustOwned by the Rust SDK repository
    on Oct 21, 2026
  2. rmcdaniel commented on Oct 21, 2026

    @rmcdaniel
    MemberAuthor

    Accepted as a cross-repository product defect. The published 300 × 10 KB reproduction and permanent replay block are the baseline; the documented per-command limits must compose into a task that can actually be delivered.

    The next implementation step is to trace the aggregate command serialization and runtime external-payload path in all three SDKs and the Server, then add a failing whole-task acceptance control. Preserve one atomic command batch and its deterministic ordering: splitting requests cannot introduce partial command acceptance or replay differences. An unsupported aggregate must produce a typed size-limit outcome with measured size, rather than a determinism/replay-block diagnosis. This is separate from oversized failure diagnostics in #225, though transport-sizing code may be shared after review.

    Keep the 300 × 10 KB and actual over-limit published-artifact controls, embedded coverage and explicit aggregate-limit documentation in acceptance. Documentation alone does not close this issue. Current P1 activity-lease delivery remains ahead of this P2; no implementation or released fix is claimed here.

  3. rmcdaniel commented on Oct 22, 2026

    @rmcdaniel
    MemberAuthor

    Scope note: the 2 MiB request cap also undercuts the documented per-payload limit for a single value, and for client requests as well as worker completions.

    Source-verified, not reproduced. The acceptance should state which size the limit measures (encoded or raw Avro bytes) and keep the request cap from rejecting a value that the documented per-payload limit allows.

  4. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    General open-item pass, November 4. Owner: Server for request/command boundary; Workflow and SDKs for producer behavior; General for shared size contract.

    Next action: Define encoded-envelope versus per-value limits and a bounded completion strategy with countercontrols for valid large fan-outs, avoiding silent drops or endless413 retries.

    Action taken in this pass: Asked Server and Workflow for exact existing issue/source ownership and the narrow contract decision; documented1000-operation/per-value limits cannot be dismissed by an unrelated request cap.

    General retains the cross-repository follow-through. Existing operator stops and exact source/release/production gates remain binding; the item stays open until its actual acceptance is evidenced.

  5. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    Workflow maps this to existing #865 (per-site payload/pending guards) and #930 Phase B (whole-transition events/bytes/rows/payload staging). Both remain active; neither is an excuse for the published valid-300x10KB failure. Current bridge pending guards from #933 and start-input guarding already exist and must not be reimplemented from the old report.

    Named next action: Server returns a raw Avro value versus base64/JSON envelope/request measurement table and a typed over-limit admission outcome; Workflow audits every normalized field/externalization site and transaction-local staging against it. Keep one atomic ordered completion, zero partial commands/puts on refusal, original deadlines and no endless413/replay-block loop. General owns any effective aggregate cap/capability change; SDKs must not split accepted semantics into independent partial requests. No new published cohort is started from this mapping, and #225 oversized diagnostics remains separate.

  6. rmcdaniel commented on Nov 11, 2026

    @rmcdaniel
    MemberAuthor

    Workflow: in progress. workflow#865: service-mode argument/result size guards merged (#1083, ships in 2.23.1). Next: the remaining payload sites (side-effect, local-activity and update results, search attributes), then the fan-out audit against Server's size table. workflow#930 Phase B is P3 hardening.

  7. rmcdaniel commented on Nov 11, 2026

    @rmcdaniel
    MemberAuthor

    Server: remaining, tracked in durable-workflow/server#1136 (native caps the whole completion body). Next: per-command limits with a typed structural_limit_exceeded. Owner: server.

  8. 8 remaining items

  9. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    sdk-php owns the PHP repeat of the published transport control now: installed SDK 3.17.0 and native Server 3.8.0 index sha256:7988a23525d9e50e997a9f6f02e801300ed4f626de3cf6977033657ae7e9f58e, amd64. One isolated SQLite runtime and one bounded PHP runner, synthetic completion task as in Python/Rust. This distinguishes actual typed-413 delivery from mock coverage; it is not full fan-out conformance. Results and exact toolchain follow here; server#1136 remains the response-delivery owner.

  10. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    PHP published-artifact result, 2026-11-13 approximately 09:41–09:48 UTC. Owner sdk-php. Tuple: Packagist SDK 3.17.0, source/dist reference daa842c277059471ff246803a8272165b4097c52; native Server 3.8.0, image index sha256:7988a23525d9e50e997a9f6f02e801300ed4f626de3cf6977033657ae7e9f58e; official PHP 8.3.36 image php@sha256:628102249d93494233b54490b785f12db9cd60f4dc89416dfca15973f2125f10, amd64. SDK loaded only from a separate installed consumer, with no source overlay. Disposable SQLite runtime and bounded PHP runner; all containers/volumes removed, diagnostics retained at most 24 hours.

    • Direct transport control PASS. Client::completeWorkflowTask() with the same synthetic opaque commands used in Python/Rust sends 36,005,171 bytes and receives typed WorkflowTaskCompletionTooLarge, HTTP 413, payload_too_large, limit 33,554,432 bytes, in 0.115 s. A 3,005,171-byte control reaches ordinary HTTP 422 command validation. This proves PHP receives the actual refusal; it does not prove complete fan-out behavior or the worker’s follow-up command.
    • Real 300 × 10 KB Worker fan-out PRODUCT FAIL before completion HTTP. Valid deferActivity() members are refused locally by RuntimePayloadUploads with ExternalPayloadException: Namespace runtime external payload storage is not available. Default native discovery advertises limits.max_payload_bytes = 2097152, external storage status: unconfigured, and no workflow-completion aggregate limit. The PHP planner applies that ordinary 2 MiB request limit to completion batches and demands offload even though native accepts a 32 MiB completion. The worker exits before scheduling the activities. Adding the documented synthetic server key did not change the unconfigured storage status; that correction did not produce a pass. The larger valid Worker case was not reached.

    The PHP slice remains incomplete. Next action: sdk-php owns a scoped completion-planning correction and regression coverage; Server owns an additive discovery field for the actual completion aggregate limit (proposed limits.max_workflow_task_completion_bytes) alongside its response-delivery correction in server#1136. Server/General are being asked to confirm that field and the unavailable-storage fallback before implementation. Then PHP repeats both installed Worker cases and the one-time structural failure case on the exact published tuple. Native’s Python/Rust early-response defect remains independently valid; PHP’s successful direct transport control does not close it.

  11. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    Published feature acceptance classification: PRODUCT-FAIL, owned by Rust SDK #254.

    The registry-only 4.15.0 consumer against published native Server 3.8.0 (index sha256:7988a23525d9e50e997a9f6f02e801300ed4f626de3cf6977033657ae7e9f58e) passes typed pending-update readback followed by completed JSON null, with exactly one UpdateAccepted event. Ordinary installed lifecycle/deadline checks also pass.

    The 300 × 10 KB inline activity fan-out fails before completion submission: the worker returns external_payload_unavailable: namespace runtime storage is not available. The Rust planner uses ordinary limits.max_payload_bytes (2 MiB) as the entire completion request limit, forcing individually valid values into external storage. This is a confirmed SDK product defect, separate from Server #1136's early HTTP 413 response-delivery defect. Neither failed cell is passing acceptance.

    Next action, owner sdk-rust: implement #254 using the explicit optional limits.max_workflow_task_completion_bytes discovery field agreed with Server, scoped to workflow and activity-continuation completion endpoints. Preserve per-value/ordinary limits and upload authority. Server owns discovery plus transport correction in #1136; repeat the feature cell with exact published SDK and corrected qualified Server artifacts after both fixes. No version or multiplier inference, oversized-batch success, or native timeout-cleanup claim is authorized by the current evidence.

  12. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    PR #255 merged at 3e8b28299528b0d49159632309f13a1ccdec2ad5 after Security's exact-head PASS and all required checks. The remote head is deleted and confirmed absent. The implementation and regression fixture are public; raw continuation offload/lease authority remain unchanged.

    Owner sdk-rust. Release draft #256 prepares 4.16.0 with this fix and revision-2 metrics. Exact package/lifecycle backward-compatibility qualification is running against published native 3.8, which cannot qualify the new negotiated-cap fan-out path. Publication and feature delivery remain held for Server #1203's qualified published producer and the real installed-worker within/over-cap cell. Server owner and Security have the dependency and test contract; sdk-rust owns the SDK package/installed repeat and downstream evidence. This issue stays open.

  13. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    The release dependency is now the combined corrected native Server 3.9.1 cohort (#1202 + #1203), as confirmed by the Server owner and Security. The tentative 3.9.2 split is superseded. The durable cohort decision retains combined CI, counter-review, final notes/tree and immutable publication gates. This is an unreleased plan, not a version-based capability assumption.

    SDK source afc2b2ff8960ac2e080ffec765c66f59200cd544 and candidate package 89f6418ca72e1c6e13495281aab131417c5ddcee1946bfc1a22b1e63cc2b11cc remain unchanged. Security has granted exact-head release-source PASS; all current checks are green. Final tag, approval record and publication remain held until Server provides its qualified immutable index/platform/source/version tuple and the candidate passes actual installed within/over-limit Worker acceptance. The consumer must see the explicit discovery field, satisfy per-value/ordinary limits and complete a same-worker follow-up. Neither older Server evidence nor the planned version number substitutes for that gate.

    Owners: Server drives combined CI/counter-review/publication and sends the exact tuple; sdk-rust runs the real feature repeat and protected package/publication/registry follow-through; Security retains final package review. Existing baseline and metrics candidate PASS evidence stays scoped to published Server 3.8.0.

    Release preparation PR #256 is now merged at 154fab4dcf5e503328a42c61932bcaa4b5af6523 after exact-source Security PASS and green required checks; its remote head is deleted and confirmed absent. The approved release-source candidate remains afc2b2ff, with unchanged package bytes. The real completion-limit fixture also compiles as a fresh archive-only consumer; execution remains held for the qualified corrected published Server tuple. No tag, approval record or publication exists.

  14. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    Corrected native 3.9.1 is published and exact-image qualified at source 7d7ad9d6c9aee2e4592d74389ba9efd36e741f44, index sha256:946c65eb70e8b4b7c4694be8a181c9669d46c3408b80d4e44d96f2030ea3cf51 in both registries. Linux/amd64 child is sha256:b0f2bf2a37e8b229282362acb26d9ed511596ebdd366ea4e53bd9f16008a356e; linux/arm64 child is sha256:2400a252f7877e3f4876ca96486b9c740bfab47e4d5184fdf3657a2d3a045c3a. Protocol 1.20 / control plane 2.

    Evidence: publication/scans43423285947, corpus43425102353/a1, all four Compose cells43425106728/a1, and Security exact five-receipt PASS. Protected alias promotion43428195557 is running; final GitHub Release/live alias readback is still pending. This is an immutable-image handoff for owner-controlled installed qualification, not a production adoption instruction.

    The producer now advertises limits.max_workflow_task_completion_bytes from the same HTTP admission function (default 33,554,432 encoded request-body bytes, both completion routes). Ordinary/per-value budgets and external-reference slot authority remain unchanged. The bounded discard preserves typed 413 delivery within its documented 64 MiB/two-second remainder bound.

    PHP, Python and Rust owners have the tuple and own actual installed real-Worker within-cap and oversized/follow-up repeats, plus their package/source/release gates. Record exact package/registry identity, valid 300×10 KiB behavior, an individually valid but over-wire-cap batch, no partial scheduled activity writes, one structural failure and continued same-worker operation. Retain malformed/absent/smaller cap and continuation-authority scope as applicable; no result-bound ref substitution into unsupported command slots. Server owns response/discovery diagnosis if any repeat fails. This issue remains open through those published-package results; source controls, transport-only 413 and the image corpus alone do not close it.

  15. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    Python owner sdk-python is running one installed PyPI 2.17.1 consumer against the newly qualified published native Server 3.9.1, source 7d7ad9d6c9aee2e4592d74389ba9efd36e741f44, index sha256:946c65eb70e8b4b7c4694be8a181c9669d46c3408b80d4e44d96f2030ea3cf51, linux/amd64 child sha256:b0f2bf2a37e8b229282362acb26d9ed511596ebdd366ea4e53bd9f16008a356e. Server's exact qualification handoff is recorded in server1205 / comment6853373077.

    The selected scope is actual Worker 300×10KB within-limit, an aggregate over-limit request with zero partial activities, and a small follow-up on the same worker. The earlier native3.8 transport nonpass remains intact. This is one bounded local acceptance with synthetic SQLite state, loopback only and no production credentials or source overlay. Results and exact cleanup follow here; no cross-SDK acceptance or final issue closure is inferred.

  16. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    Workflow searched this issue's retained evidence and the owning Workflow #865/#930/#809 records for an equivalent published embedded acceptance. No named-published 300 × 10 KB plus aggregate/budget-refusal atomicity receipt was found. The existing published 2.5.27 four-worker receipt covers 64 activities with small typed values and queue recovery. It is useful prior queue evidence, but it does not exercise this issue's aggregate payload or refusal control, so it will not be rerun or counted as equivalent.

    Owner: Workflow, after the current 2.29.0 release and timeout-cleanup installed gate. Reuse the ordinary embedded database-queue fan-out fixture with a normal registry installation and no runtime source override. The bounded acceptance will cover 300 individually valid 10 KB arguments, exact completion/effects and original deadlines, then a separately named over-budget control with a typed measured refusal and no partial scheduling/effects, followed by continued ordinary worker operation. Record the exact package/source archive, framework/PHP/backend, configured embedded limits, results and cleanup here.

    Embedded admission has no native HTTP completion envelope. Its history-event/byte/row and per-value limits must be measured and identified directly; native's 32 MiB wire cap and SDK controls are not embedded proof. Source-only budget tests do not replace the installed result. General retains cross-mode closure, and existing #1116 retains any already-scoped accounting/legacy-closure design work.

  17. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    Python published-artifact acceptance: PASS for the selected completion cases

    Owner sdk-python, linux/amd64 SQLite cell, November 13. Exact tuple: PyPI durable-workflow 2.17.1, wheel SHA-256 90040cb28906985b227988815b1b53aa3079378673e35c689392ac71edaf7b35; native Server 3.9.1, source 7d7ad9d6c9aee2e4592d74389ba9efd36e741f44, index sha256:946c65eb70e8b4b7c4694be8a181c9669d46c3408b80d4e44d96f2030ea3cf51, amd64 child sha256:b0f2bf2a37e8b229282362acb26d9ed511596ebdd366ea4e53bd9f16008a356e. Pulled the exact index, verified its amd64 selection and image revision, and compared all 46 installed SDK files byte-for-byte with the downloaded registry wheel. Dependencies: httpx 0.28.1 / fastavro 1.13.1. No source overlay.

    One actual Worker ran all three scenarios over direct native HTTP from 12:58:39 to 13:00:56 UTC:

    Case Observed result
    300×10,000-byte arguments A 4,164,455-byte completion with 300 schedule commands received 200. Exactly 300 ActivityScheduled/ActivityStarted/ActivityCompleted and one WorkflowCompleted; result [300,3000000].
    300×120,000-byte arguments The 48,164,855-byte completion received exactly one actual 413, reason: payload_too_large, limit_kind: workflow_task_completion_bytes, limit: 33554432. The SDK sent one 477-byte fail_workflow completion on the same original task and received 200. Exactly one WorkflowFailed with failure_category: structural_limit, non_retryable: true, and measured-size/limit guidance. Zero ActivityScheduled, ActivityCompleted or WorkflowTaskFailed; no repair/replay block.
    Same-worker follow-up, 3×10-byte arguments Completed with exactly 3 activities and one WorkflowCompleted; result [3,30].

    The first fixture attempt had an insufficient 120-second bound: 300 activities were scheduled and 241 completed before the harness stopped the Worker; a lost-lease failure was recorded at that shutdown. It remains runner-aborted/NONPASS. Only the later run with a 300-second bound qualifies above. No SDK or Server source changed between attempts.

    Cleanup verified: owned container absent, both anonymous volumes removed, zero remaining owned volumes, fixture credential file removed and no OOM. No production credentials, vault reads, builds or production changes.

    This qualifies Python 2.17.1's selected within/over/follow-up completion behavior on the exact published native 3.9.1 amd64 tuple. The earlier native 3.8 transport nonpass remains historical evidence. Other SDKs, architectures, legacy/request-key cohorts and cross-repository issue closure retain their own gates. Python 2.17.1 needs no new SDK release for this Server correction.

  18. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    Published 4.16.0 installed acceptance: PASS

    Exact published tuple: Rust 4.16.0, registry/archive SHA-256 89f6418ca72e1c6e13495281aab131417c5ddcee1946bfc1a22b1e63cc2b11cc, clean source/tag afc2b2ff8960ac2e080ffec765c66f59200cd544; native Server 3.9.1, source 7d7ad9d6c9aee2e4592d74389ba9efd36e741f44, index sha256:946c65eb70e8b4b7c4694be8a181c9669d46c3408b80d4e44d96f2030ea3cf51, executed Linux amd64 child sha256:b0f2bf2a37e8b229282362acb26d9ed511596ebdd366ea4e53bd9f16008a356e. Public runner fixture is reviewed commit b3480de83b30764b6d4237f5de243a49d9a22f61 (#257); package/source bytes remain the approved ones.

    Fresh registry consumer: exact lock source/checksum verified, no patch, vendor or local Cargo config. Actual tests against the same disposable published Server:

    • PASS: 300 × 10 KB inline arguments without payload storage, exact result and 300 ActivityCompleted events.
    • PASS: 60 × 450 KB individually valid arguments exceed the advertised aggregate budget; managed worker records one non-retryable structural-limit WorkflowFailed, with zero ActivityScheduled/ActivityCompleted from the rejected batch.
    • PASS: another workflow completes on the same worker, then bounded clean shutdown. The completion-limit test passed in 80.01 seconds.
    • PASS: all seven shared metrics, real workflow/activity ready-age observations and positive queued workflow age, paired counts/durations, completed/failed activities, HTTP-error observation, and official Prometheus exposition with the pinned tag contract. Metrics test passed in 1.12 seconds.

    Separate registry-only lifecycle/deadline acceptance also passes. Every run cleaned its owned containers/networks; no leftovers. The prior candidate shutdown NONPASS is preserved and is not substituted for this result.

    Publication 43432093783, final Release, and API docs 43432730516 all pass. Fresh API build identity is 4.16.0 / checksum above / source above / tooling ab8cae5eeb9c2aa701a997a2593e0ecd933a57a4.

    Scope: the larger allowance requires explicit limits.max_workflow_task_completion_bytes; older producers retain the conservative ordinary cap. This tuple qualifies the Rust completion/metrics behavior, not other SDKs, every backend, #216 caller-key retries or native timeout-cleanup. Root docs adoption is being driven by its owner. Sample App keeps its already-frozen 4.15 ordinary cohort, which does not claim this large-fanout slice; no redundant tuple run is required for that unchanged scope.

  19. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    PHP owner-installed acceptance is complete on published SDK3.18.1 / native Server3.9.1, registry-only43440828565. Exact SDK source/dist ffa67454b99e299624d46dc2c5a4f7eceab5b73f; Server index sha256:946c65eb70e8b4b7c4694be8a181c9669d46c3408b80d4e44d96f2030ea3cf51, source 7d7ad9d6c9aee2e4592d74389ba9efd36e741f44; protected stock PHP8.3.36, no source/candidate fallback.

    The actual Worker completes300×10KB activities from a4,158,899-byte completion without configured storage. A60×450KB completion is36,032,179 bytes and is submitted exactly once: authoritative HTTP413 names workflow_task_completion_bytes, limit33,554,432 and exact observed size. The run closes with structural_limit, zero partial activities and no repeated submission. The same Worker then completes fresh work; upload count0. Ordinary per-value limit remains2,097,152. Retained artifacts include body counts/statuses and bounded diagnostics, no payloads/credentials.

    PR411 links source fixes406/408, independent exact-package approval, immutable release/Packagist identity and live stable manual/API proof. This closes the PHP planner/Worker delivery slice. General owns cross-SDK issue closure; continuation slot authority, other transport cohorts and #216 are not inferred from this result.

  20. rmcdaniel commented on Nov 13, 2026

    @rmcdaniel
    MemberAuthor

    Root docs adoption is delivered via docs #536, merge 127993281e379985fcb3b20d27ca7e0d97543fea, protected Pages 43447139751 SUCCESS. All 16 changed public pages across eight locales, exact manifest and unchanged installers passed live readback. Stable selection now uses PHP SDK 3.18.1; manifest SHA256 0a8878f4dc1cd92af315d5c41887da1beddd3d20e8efd53b01a7ab1e46a88c36.

    The PHP guide explains typed/raw defaults, pending versus completed null, capability refusal and stable-ID recovery when the original key is unknown, linking the actual exact-tag SDK guide. The shared completion reference names only the qualified PHP 3.18.1/native 3.9.1 PHP8.3.36/linux-amd64/SQLite Worker receipt. Broader #216/native timeout and cross-product closure remain separate. This completes the root-docs downstream slice; SDK/General retain their actual upstream acceptance and closure decisions.

  21. rmcdaniel commented on Nov 18, 2026

    @rmcdaniel
    MemberAuthor

    Root cause and fixes (General, 2026-11-18). On published Server 3.12.0 with the Python 2.17.2 worker, a 1,000 × 22,000-byte all() produced a 29.9 MB completion. That is under the advertised 32 MiB max_workflow_task_completion_bytes, yet it was refused with 409 history_transaction_limit_exceeded and the run blocked with zero activities.

    Cause: native raised the completion cap to 32 MiB but kept PHP's 16 MiB per-transaction history budget, and ActivityScheduled events repeat their arguments. The docs were right; the Server didn't honor them.

    • Server #1231 (merge queue): when unset, the history budget defaults to 4× the completion cap. The same fan-out completes on the branch. The 409 now names limit_kind, current_value and configured_limit. Ships in native Server 3.13.0.
    • Python Pin native policy to accepted PHP configuration CI contexts #337 (merged): if the budget is still hit (an explicitly lowered limit), the worker fails the run once with a structural-limit failure instead of blocking. Ships in Python SDK 2.18.0.
    • Separate defects found: server#1234 (MariaDB's default max_allowed_packet makes completions over 16 MiB retry a 503 forever) and sdk-python#338 (all() with a failed member while siblings are pending blocks on 409 parallel_sequence_mismatch).
    • Remaining for this issue: both releases, PHP/Rust typed handling of the 409 on servers with lowered budgets, and confirming the fix with published packages.
  22. rmcdaniel commented on Nov 18, 2026

    @rmcdaniel
    MemberAuthor

    Server side released: native Server 3.13.0 sets the history byte budget to 4× the completion cap and types the 409 (aliases 3/latest = sha256:30ccaf2c…). Python SDK 2.18.0 (already published) fails the run cleanly if an explicitly lowered budget is still hit. Remaining: PHP and Rust typed handling of that 409, and an installed fan-out check on 3.13.0.

  23. rmcdaniel commented on Nov 19, 2026

    @rmcdaniel
    MemberAuthor

    Embedded Workflow receipt (General, 2026-11-19). No earlier named published-package receipt existed, so I ran one. Receipt, code and raw results.

    Setup. Fresh laravel/laravel (13.43.0), composer require durable-workflow/workflow:2.32.0 from Packagist (dist 8771162a), PHP 8.5.12, database queue, one php artisan queue:work, default structural limits. composer.lock SHA256 8c4797c1c1ce0dec0265439d1f66d805c94cc531b508c5d89aed897c328bd5ec.

    Results

    • Valid fan-out: 300 × 10 KB in one all() completed with 300/300 activities on SQLite (863 s) and on MySQL 8.4.8 (771 s).
    • Batch-size limit: 1,001 × 10 KB was refused atomically with structural_limit / command_batch_size, 1001 against 1000. Zero activities were scheduled. History holds only StartAccepted, WorkflowStarted and WorkflowFailed.
    • Payload-size limit: one 2.2 MB member was refused with structural_limit / payload_size, 2,933,364 against 2,097,152. The refusal isn't atomic: the 150 members before it were scheduled and then cancelled (0 attempts). Tracked in workflow#1232. This is not an acceptance item.
    • Follow-up: a second 300 × 10 KB run on the same worker process completed, with 0 failed jobs.
    • Blocker: each workflow task's time grows roughly quadratically with fan-out width. At 300 members, scheduling takes about 135 s and the final task 58–85 s. With pcntl, the README's default queue:work enforces a 60 s job timeout. That kills the final task on every attempt, and the run stays waiting with no typed failure. That is this issue's original defect in embedded mode. Tracked in workflow#1231.

    SDK handling of native Server 3.13.0's typed 409 history_transaction_limit_exceeded. All three SDKs now fail the run once with a structural-limit failure instead of blocking:

    • Python 2.18.0+ (sdk-python#337)
    • PHP 3.21.0 (sdk-php#435)
    • Rust 4.20.0 (sdk-rust#291)

    Each has installed-package tests against 3.13.0 with a lowered history budget.

    Acceptance status

    • Valid fan-outs complete in PHP, Python and Rust: met. In embedded mode: not met. The fan-out completes only when the worker doesn't enforce a job timeout.
    • Typed structural-limit failure that names the limit and measured size: met in all four modes.
    • Structural limits page states the per-task aggregate: met (docs #536).
    • Published tests cover 300 × 10 KB and the over-limit case in each SDK: met.

    Remaining: workflow#1231 fixed and released, then this receipt's run 05 (300 × 10 KB under a default pcntl queue:work) completing on that release. Leaving this issue open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingkind:cross-repositoryWork spans more than one public repositorykind:defectA public product behavior is incorrectpriority:P2Normal-priority product workrepo:sdk-phpOwned by the PHP SDK repositoryrepo:sdk-pythonOwned by the Python SDK repositoryrepo:sdk-rustOwned by the Rust SDK repositoryrepo:serverOwned by the standalone server repositoryrepo:workflowOwned by the Workflow engine repository

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions