Skip to content

Python SDK: capabilities digest + history-free result to cut response bytes; activity claim silently dropped when the initial lease confirmation times out #352

Open

Description

@rmcdaniel

Problem

A byte-metered lab run of the PHP SDK on DW Standard Workflow v1 (one start, one activity, one completion, 1 KiB payloads) against native Server 3.15/3.16 found about 385 KB of responses per workflow. That was enough to saturate a Cloud Dev cell's 4 Mbit/s egress. Under that saturation, the PHP worker silently dropped activity claims (durable-workflow/sdk-php#445). The Python SDK (origin/main 0d5283c) has three of the same byte costs and the same claim-drop bug.

Measured: Python SDK, DW Standard Workflow v1

Lab only: one native server container on SQLite, capped at 1 CPU. One Python worker and one caller. 5 sequential workflows. Bytes were counted per request with httpx response hooks (num_bytes_downloaded, which is the wire size). Script: lab4 /home/lab4/fleet/tmp/general-egress-py/scripts/measure.py.

Response bytes per workflow stock durableworkflow/server:3.15.0 server build with durable-workflow/server#1261 gzip
worker 151,007 42,592
of which server_capabilities (uncompressed) 121,392 (7 copies × 16,860) (same 7 copies, compressed)
caller 42,792 7,498
of which run history GET 17,630 1,805
total 193,799 50,091

Per workflow, the worker made 2 workflow-task polls, 2 workflow-task completions, 1 activity poll, 1 activity renew-lease and 1 activity completion. Each of those 7 answers carried the full ~17 KB server_capabilities. The caller made 1 start, 2 describe reads and 1 history page.

1. Server capabilities digest (durable-workflow/server#1262): not supported, would cut most of the remaining bytes

Every worker-plane answer carries the full object. The SDK reads it from answers in only two places, and both read only one key:

  • src/durable_workflow/client.py:2314 reads result.get("server_capabilities") on an activity poll, for activity_lease_renewal.
  • src/durable_workflow/_activity_checkpoint.py:93 (capture_poll) reads activity_heartbeat_resume.

Every other reader uses the cached /cluster/info (worker.py:1091-1130, :1396, :1462-1547, _activity_lease.py:33, telemetry_carrier.py:44). The SDK does not use carried polls, so only the top-level copy matters.

Support would need these changes, following the contract in docs/contracts/server-capabilities-digest.md (durable-workflow/server#1262) and durable-workflow/sdk-php#446 (src/Transport/ServerCapabilitiesDigest.php):

  • Send X-Durable-Workflow-Server-Capabilities-Digest: <digest>|none from _headers_with_token(worker=True) (client.py:2020).
  • In _request, right after resp.json() (client.py:2311) and before the two readers above, restore a matching digest string to the held object.
  • Store a full copy without its digest member.
  • Treat an unknown digest string as a protocol failure.
  • Note that _poll_transport.unwrap_response keeps only retry-after and the protocol-version header. Inside a negotiated poll envelope, use the body's digest member, not the response header.

Expected effect: about 37 KB per workflow left after gzip drops to almost nothing. PHP went from 385 KB to 18 KB with all three changes.

2. gzip (durable-workflow/server#1261): already handled, no change needed

Client builds a plain httpx.AsyncClient(base_url=..., timeout=..., verify=...) (client.py:1986-1988). Its per-request headers (client.py:2021) never override Accept-Encoding. httpx 0.28 therefore sends Accept-Encoding: gzip, deflate and decodes transparently (measured: accept-encoding sent: ['gzip, deflate'], and every answer came back content-encoding: gzip from the durable-workflow/server#1261 build). This gives about 194 → 50 KB per workflow with no SDK change. A regression test that asserts the header would be cheap insurance.

3. result() downloads history it does not need: present

Client.get_result (client.py:5506-5552) polls describe_workflow(handle.workflow_id), which is GET /workflows/{id}. When the status is terminal, it also pages the whole run history (get_history(..., page_size=1000), :5515) to find WorkflowCompleted.output. But describe_workflow already decodes output_envelope from the same status read (client.py:4137-4148).

Request count per completed result: N describe reads + ≥1 history pages (all pages for long runs). Measured: 17.6 KB uncompressed, or 1.8 KB gzip, per workflow, out of 42.8 KB of caller bytes.

Fix, as in durable-workflow/sdk-php#446: for completed, return the describe read's output_envelope (decoded with the run's payload_codec). Keep the history read for failed, cancelled, terminated and timed-out runs, which need the failure details. Fall back to history when no envelope is present (older servers).

Side note: get_result describes the current run of the instance (describe_workflow(handle.workflow_id)), not handle.run_id. A fix should use GET /workflows/{id}/runs/{run} when the handle has a run id.

4. Activity claim silently dropped when the initial lease confirmation times out: present (bug, same as durable-workflow/sdk-php#445)

When an activity poll returns a task, _request calls ActivityLeaseAuthority.confirm_delivery (client.py:2314-2320). That makes one renew-lease call with a min(5.0, remaining) budget and no retry (_activity_lease.py:74-89). Any failure becomes ActivityLeaseAuthorityLost("activity_initial_lease_unavailable"), including a transport timeout, a connection reset or a 503. The exception is raised out of poll_activity_task. The poll loop treats it as an ordinary poll error, releases the slot and sleeps (worker.py:4244-4277). The claimed task is gone: nothing is completed, failed or released. The server holds the 300 s lease and then settles ActivityLeaseExpired. With the default 1 attempt, the workflow fails.

The renew-lease answer itself carries the 17 KB capabilities (measured 17,663 B), which makes this exchange slow on a saturated link. Unlike PHP, Python does not do a cluster-info GET per renewal (client.py:7068 uses the cached discovery).

The steady-state path has a second silent drop. _maintain already retries after transient failures (_activity_lease.py:281-290). But if renewal never recovers, the local bound expires (activity_execution_budget_expired) and _run_activity_task returns claim_aborted without telling the server (worker.py:3478-3480).

Lower severity, not workflow-failing: one failed workflow-task heartbeat during a local activity (worker.py:1759) abandons the workflow task without fail_workflow_task (worker.py:2994). The server re-dispatches it after the workflow-task lease expires.

Acceptance criteria

  • Item 4 (fix PR linked below): a transient failure or timeout of the initial confirmation keeps the claim. Renewal is retried with backoff inside the lease, and the activity's completion is accepted. A claim whose renewal never recovers is failed retryably through the server (typed ActivityLeaseAuthorityLost / activity_lease_renewal_unavailable) while lease is left, never dropped silently. Tests simulate timeouts, resets and 503s, then success.
  • Item 3: a completed result needs no history request when the status read has output_envelope. Non-success terminal states keep their typed errors.
  • Item 1: worker requests opt in to the digest. Answers with matching digests are restored before any reader runs. Older servers keep working. A lab rerun shows the per-workflow worker bytes drop.
  • Item 2: a test asserts that SDK requests advertise gzip.

Related: durable-workflow/server#1261 (compression), durable-workflow/server#1262 (capabilities digest), durable-workflow/sdk-php#446 (PHP SDK gzip + digest + history-free result), durable-workflow/sdk-php#445 (PHP claim drop).

Activity

  1. rmcdaniel commented on Nov 20, 2026

    @rmcdaniel
    MemberAuthor

    Item 4 fix: #353 (head d6c1084). Items 1 and 3 are still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions