Repository navigation
Description
Problem
A byte-metered lab run of the PHP SDK on DW Standard Workflow v1 (one start, one activity, one completion, 1 KiB payloads) against native Server 3.15/3.16 found about 385 KB of responses per workflow. That was enough to saturate a Cloud Dev cell's 4 Mbit/s egress. Under that saturation, the PHP worker silently dropped activity claims (durable-workflow/sdk-php#445). The Python SDK (origin/main 0d5283c) has three of the same byte costs and the same claim-drop bug.
Measured: Python SDK, DW Standard Workflow v1
Lab only: one native server container on SQLite, capped at 1 CPU. One Python worker and one caller. 5 sequential workflows. Bytes were counted per request with httpx response hooks (num_bytes_downloaded, which is the wire size). Script: lab4 /home/lab4/fleet/tmp/general-egress-py/scripts/measure.py.
| Response bytes per workflow | stock durableworkflow/server:3.15.0 |
server build with durable-workflow/server#1261 gzip |
|---|---|---|
| worker | 151,007 | 42,592 |
of which server_capabilities (uncompressed) |
121,392 (7 copies × 16,860) | (same 7 copies, compressed) |
| caller | 42,792 | 7,498 |
| of which run history GET | 17,630 | 1,805 |
| total | 193,799 | 50,091 |
Per workflow, the worker made 2 workflow-task polls, 2 workflow-task completions, 1 activity poll, 1 activity renew-lease and 1 activity completion. Each of those 7 answers carried the full ~17 KB server_capabilities. The caller made 1 start, 2 describe reads and 1 history page.
1. Server capabilities digest (durable-workflow/server#1262): not supported, would cut most of the remaining bytes
Every worker-plane answer carries the full object. The SDK reads it from answers in only two places, and both read only one key:
src/durable_workflow/client.py:2314readsresult.get("server_capabilities")on an activity poll, foractivity_lease_renewal.src/durable_workflow/_activity_checkpoint.py:93(capture_poll) readsactivity_heartbeat_resume.
Every other reader uses the cached /cluster/info (worker.py:1091-1130, :1396, :1462-1547, _activity_lease.py:33, telemetry_carrier.py:44). The SDK does not use carried polls, so only the top-level copy matters.
Support would need these changes, following the contract in docs/contracts/server-capabilities-digest.md (durable-workflow/server#1262) and durable-workflow/sdk-php#446 (src/Transport/ServerCapabilitiesDigest.php):
- Send
X-Durable-Workflow-Server-Capabilities-Digest: <digest>|nonefrom_headers_with_token(worker=True)(client.py:2020). - In
_request, right afterresp.json()(client.py:2311) and before the two readers above, restore a matching digest string to the held object. - Store a full copy without its
digestmember. - Treat an unknown digest string as a protocol failure.
- Note that
_poll_transport.unwrap_responsekeeps onlyretry-afterand the protocol-version header. Inside a negotiated poll envelope, use the body'sdigestmember, not the response header.
Expected effect: about 37 KB per workflow left after gzip drops to almost nothing. PHP went from 385 KB to 18 KB with all three changes.
2. gzip (durable-workflow/server#1261): already handled, no change needed
Client builds a plain httpx.AsyncClient(base_url=..., timeout=..., verify=...) (client.py:1986-1988). Its per-request headers (client.py:2021) never override Accept-Encoding. httpx 0.28 therefore sends Accept-Encoding: gzip, deflate and decodes transparently (measured: accept-encoding sent: ['gzip, deflate'], and every answer came back content-encoding: gzip from the durable-workflow/server#1261 build). This gives about 194 → 50 KB per workflow with no SDK change. A regression test that asserts the header would be cheap insurance.
3. result() downloads history it does not need: present
Client.get_result (client.py:5506-5552) polls describe_workflow(handle.workflow_id), which is GET /workflows/{id}. When the status is terminal, it also pages the whole run history (get_history(..., page_size=1000), :5515) to find WorkflowCompleted.output. But describe_workflow already decodes output_envelope from the same status read (client.py:4137-4148).
Request count per completed result: N describe reads + ≥1 history pages (all pages for long runs). Measured: 17.6 KB uncompressed, or 1.8 KB gzip, per workflow, out of 42.8 KB of caller bytes.
Fix, as in durable-workflow/sdk-php#446: for completed, return the describe read's output_envelope (decoded with the run's payload_codec). Keep the history read for failed, cancelled, terminated and timed-out runs, which need the failure details. Fall back to history when no envelope is present (older servers).
Side note: get_result describes the current run of the instance (describe_workflow(handle.workflow_id)), not handle.run_id. A fix should use GET /workflows/{id}/runs/{run} when the handle has a run id.
4. Activity claim silently dropped when the initial lease confirmation times out: present (bug, same as durable-workflow/sdk-php#445)
When an activity poll returns a task, _request calls ActivityLeaseAuthority.confirm_delivery (client.py:2314-2320). That makes one renew-lease call with a min(5.0, remaining) budget and no retry (_activity_lease.py:74-89). Any failure becomes ActivityLeaseAuthorityLost("activity_initial_lease_unavailable"), including a transport timeout, a connection reset or a 503. The exception is raised out of poll_activity_task. The poll loop treats it as an ordinary poll error, releases the slot and sleeps (worker.py:4244-4277). The claimed task is gone: nothing is completed, failed or released. The server holds the 300 s lease and then settles ActivityLeaseExpired. With the default 1 attempt, the workflow fails.
The renew-lease answer itself carries the 17 KB capabilities (measured 17,663 B), which makes this exchange slow on a saturated link. Unlike PHP, Python does not do a cluster-info GET per renewal (client.py:7068 uses the cached discovery).
The steady-state path has a second silent drop. _maintain already retries after transient failures (_activity_lease.py:281-290). But if renewal never recovers, the local bound expires (activity_execution_budget_expired) and _run_activity_task returns claim_aborted without telling the server (worker.py:3478-3480).
Lower severity, not workflow-failing: one failed workflow-task heartbeat during a local activity (worker.py:1759) abandons the workflow task without fail_workflow_task (worker.py:2994). The server re-dispatches it after the workflow-task lease expires.
Acceptance criteria
- Item 4 (fix PR linked below): a transient failure or timeout of the initial confirmation keeps the claim. Renewal is retried with backoff inside the lease, and the activity's completion is accepted. A claim whose renewal never recovers is failed retryably through the server (typed
ActivityLeaseAuthorityLost/activity_lease_renewal_unavailable) while lease is left, never dropped silently. Tests simulate timeouts, resets and 503s, then success. - Item 3: a completed result needs no history request when the status read has
output_envelope. Non-success terminal states keep their typed errors. - Item 1: worker requests opt in to the digest. Answers with matching digests are restored before any reader runs. Older servers keep working. A lab rerun shows the per-workflow worker bytes drop.
- Item 2: a test asserts that SDK requests advertise gzip.
Related: durable-workflow/server#1261 (compression), durable-workflow/server#1262 (capabilities digest), durable-workflow/sdk-php#446 (PHP SDK gzip + digest + history-free result), durable-workflow/sdk-php#445 (PHP claim drop).
Activity
rmcdaniel commentedon Nov 20, 2026
Item 4 fix: #353 (head d6c1084). Items 1 and 3 are still open.
Release an unmaintainable claim as worker_lost; add lease_loss_reason (…
Release worker_lost only after the execution is confirmed stopped (#352)
Track activity threads with a race-free guard instead of the future (#…
Merge pull request #353 from durable-workflow/general/352-activity-cl…
Bound decoded response bodies at 128 MiB by default (#352)
Merge origin/main into the #352 egress branch
Merge pull request #354 from durable-workflow/general/352-egress-dige…