Repository navigation
Rediscover a stale payload policy on 413 payload_too_large and resend the completion once - #360
Conversation
The worker resolves the runtime external payload policy once. When the Server lowered its limit mid-run, a result under the old threshold was sent inline and refused 413 payload_too_large; the worker logged "failed to complete" and the result was lost (the activity later failed as a lost lease). On that 413 for an activity completion, failure report or workflow-task completion, rediscover the policy (cluster info, retried through outages with the shared backoff while the lease lasts) and, if it changed, resend once re-encoded, externalising what the new threshold requires. The 413 is refused before processing, so the resend is safe. If the policy did not change or the resend is still too large, today's handling stands. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…arge When the result is still too large after rediscovery, there is no external storage to offload to, or it is over the upload limit, the attempt used to expire silently. Report it as a non-retryable failure with failure.type and failure.class "ActivityResultTooLarge" (the same wire shape as PHP and Rust) and a message naming the Server's limit; the worker keeps running. Co-Authored-By: Claude Opus 5.5 <[email protected]>
No behaviour change: read the refusal's limit without a redundant type check, move the rediscovery loop into _rediscover_cluster_info (cognitive complexity), log its final failure with log.exception, and split a composite test assertion. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
rmcdaniel
left a comment
There was a problem hiding this comment.
Reviewed the coordinated worker payload-limit fix against the PHP #451, Python #360 and Rust #306 changes. PHP, Python and Rust recover only from the Server's typed size refusal. The worker can rediscover the policy and re-encode the result without re-running the activity. A result that still cannot fit is reported with type and class ActivityResultTooLarge and non_retryable: true, and worker dispatch remains available. A discovery outage keeps the attempt eligible for recovery instead of recording a permanent size failure.
The target-branch and language CI jobs passed for all three reviewed heads. Python also passed the local focused transport suite, and Rust passed the 17 worker-outage tests. The release qualification and published-package checks follow the merges.



Python counterpart of the PHP finding: a stale payload policy that leads to a
413 payload_too_large.Is Python affected? Yes. It does not crash, but it loses the result.
Client._runtime_external_payload_transport(src/durable_workflow/client.py, around line 2435 on main) resolvesnamespace.external_payload_storagefrom cluster discovery once. That includesthreshold_bytes, which is native Server's payload ceiling when the threshold is absent (rust/src/runtime/payload_discovery.rs::threshold). The result is cached in_runtime_external_payload_transport_cache, and_externalize_runtime_payloadskeeps a payload inline whenlen(data) <= transport.threshold_bytes.413 {"reason":"payload_too_large","limit":N}(rust/src/runtime/payload_limits.rs::enforce)._deliver_activity_outcomeraises, and_run_activity_task_bodylogsfailed to complete activity taskand returnscomplete_error.limit_kind: workflow_task_completion_bytesfails the run straight away as a structural limit, even when only the cached policy was stale.Fix
On a
413 payload_too_largefor an activity completion, failure report or workflow-task completion:Rediscover the payload policy with
GET /api/cluster/info. Outages are retried with the shared backoff while the lease, or 60 s if the lease is unknown, allows. Held discovery is replaced only by a successful read.Drop the cached policy and re-resolve it.
If it changed, resend once. The request is re-encoded, so payloads that the new threshold requires are externalised. The 413 is refused before processing, so the resend is safe.
Some activity results still cannot be delivered: the policy did not change, the resend is still too large, there is no external storage, or the result is over the upload limit (
ExternalPayloadOversized). The attempt is then reported as a non-retryable failure with the same wire shape as PHP and Rust:failure.type = "ActivityResultTooLarge"failure.class = "ActivityResultTooLarge"non_retryable = trueThe attempt no longer expires silently.
An oversized workflow-task completion keeps the structural-limit run failure.
The worker never exits and never reruns the activity, so the computed result is kept and sent.
Tests (
tests/test_payload_policy_refresh.py)The tests use a fake runtime whose request limit and threshold are lowered mid-run, after the worker cached the 2 MiB policy.
completed, the activity ran once, no failure report, and the new policy is held.ActivityResultTooLargefailure naming the 100 KiB limit. The worker's client keeps serving requests.Local results (Python 3.14):
pytest tests/ -m "not integration": 7905 passed, 3 skipped.ruff: clean.mypy: clean.🤖 Generated with Claude Code