Skip to content

Cut response bytes: gzip, server capabilities digest, history-free completed result (385 → 18 KB/workflow with server#1261/#1262) - #446

Merged
rmcdaniel merged 6 commits into
mainfrom
general/egress-sdk
Nov 20, 2026
Merged

rmcdaniel merged 6 commits into
mainfrom
general/egress-sdk

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Why

The Cloud Dev benchmark (native Server 3.15.0, SDK 3.22.0) was egress-bound at 1.5 wf/s: the cell's outbound is shaped to 4 Mbit/s. A byte-metered lab rerun found about 385 KB of responses per workflow for DW Standard Workflow v1. The SDK is responsible for part of that:

  • No gzip. Guzzle sends no Accept-Encoding by default.
  • No capability reuse. Every worker answer, and every carried poll, carries the ~17 KB server_capabilities object again.
  • Full history on every result. result() downloads the run's whole history (18.5 KB for 6 events) after the terminal status read, even though that read already returned output_envelope.

What

  • Psr18Transport and gzip.
    • When it owns a Guzzle client with cURL and zlib, it sends Accept-Encoding: gzip; cURL decodes the body.
    • A gzip body that a custom PSR-18 client left encoded (gzip magic plus Content-Encoding: gzip) is gunzipped.
    • Bodies that the client already decoded are left alone. This covers both the native control handler and Guzzle moving Content-Encoding aside.
  • Capabilities digest (new Transport\ServerCapabilitiesDigest).
    • Worker requests send X-Durable-Workflow-Server-Capabilities-Digest (server#1262). The value is none until the transport holds capabilities.
    • When the answer tags the capabilities with a digest member, the transport keeps them. When it elides them to that digest, the transport restores them in place.
    • Restoring covers success bodies, refusals (TransportException::$response) and poll-transport frames, at the top level and in carried answers, before any caller sees the answer.
    • Callers therefore see exactly what they see today, including CarriedPoll's exact-key validation.
    • Servers that don't support it ignore the header.
  • Client::workflowResult().
    • For a completed run whose terminal read has output_envelope for the same run, it decodes that, resolving external payloads as before, and skips the history read.
    • Failed, cancelled, terminated and timed-out runs, and Servers without output_envelope, still read the history.

Measured (lab)

Setup

  • Native build of server main, in a 1 vCPU / 1 GiB Dev-like slice.
  • DW Standard Workflow v1, 1 wf/s, 4 workers, 1 s task polls, 1 s result polling.
  • A byte meter that behaves like the Cloudflare edge (it asks the origin for br, gzip) and logs origin response bytes.
  • Window: 120 s. Every workflow was verified: result, status and typed history.
server SDK origin response bytes / wf
main 3.22.0 384.8 KB
main + server#1261 (compression) 3.22.0 106.0 KB
main + server#1262 (digest), compression off this PR 60.1 KB
main + server#1261 + server#1262 this PR 18.4 KB

The digest saves about 306 KB per workflow, and the history-free result saves 18.5 KB uncompressed (2.3 KB gzipped). Latency was unchanged (p50 1.04 s).

Tests

  • New: tests/ServerCapabilitiesDigestTest.php.
    • The tagged object is kept, and elided copies are restored at the top level, in a carried answer and in a 409 refusal.
    • The control plane is untouched, and custom clients aren't asked for gzip.
    • An unknown digest is a protocol failure.
    • A gzip body that a custom client left encoded is decoded.
    • The owned Guzzle/cURL client sends Accept-Encoding: gzip and decodes (the peer needs pcntl).
  • Updated: ClientWorkflowResultTest.
    • A completed run with output_envelope makes exactly two requests: the status read and the external payload. This holds for both the selected run and the current run.
    • Without output_envelope, the result still falls back to the terminal history payload.
  • Full suite (PHP 8.4 with pcntl): 9198 tests, 5 failures. The same 5 tests fail on a clean main checkout (CooperativeActivityExecutorTest SIGKILL cases and WorkerActivityLeaseMaintenanceTest::testMaintenanceStopsWhenTheOriginalOwnerIsKilled). Nothing new fails.
  • phpstan analyse on the changed files: no errors.

Not in this PR

Worker byte and robustness follow-ups I found but did not change here:

🤖 Generated with Claude Code

@rmcdaniel

Copy link
Copy Markdown
Member Author

I re-ran this against server main 43ad8d8f, which has server#1261 and #1262 merged and will be released as 3.17.0. The setup matches the earlier runs: native build in a 1 vCPU / 1 GiB lab slice, DW Standard Workflow v1 at 1 wf/s, 4 workers, 1 s task polls, a byte meter that asks the origin for br, gzip the way the Cloudflare edge does, and a 120 s window with every workflow verified.

SDK Origin response bytes per workflow
3.22.0 (published) 104.2 KB (compression only)
this PR (8d836528) 18.1 KB (17.6 KB on a repeat run)

The branch is already up to date with sdk-php main, so it needs no rebase. Slice CPU varied between 39% and 69% from run to run because other jobs were using the host, so I'm not presenting CPU as evidence for or against this PR.

A separate finding, not caused by this PR: on a repeat run, a freshly started stack answered one request with a transient 503 storage_unavailable during warm-up, and a 3.22.0 worker process died on the uncaught TransportException. I'll file that separately if it reproduces.

@rmcdaniel

Copy link
Copy Markdown
Member Author

I pushed fixes for the re-review in 93b66f58.

  • Blocker: a garbled completion answer killed the worker. The rule "this 409 is our own committed completion" (409 stale_attempt, recorded: false, our owner and attempt, every status completed) now applies to any answer: the first one, one after a transient failure, and one after a digest resend.
  • New end-to-end test: a garbled 200 to an activity completion, then the resend without the digest, then the 409. The worker keeps running, the activity runs once, and no failure report is sent.
  • Live check against durableworkflow/server:3.17.0: the first completion committed and its answer was garbled. The resend got 409 stale_attempt and was accepted. The workflow completed with exactly one ActivityCompleted.
  • Decode limit: the default is now min(32 MiB, memory_limit/4), and ResponseTooLarge reaches callers unwrapped.
  • Tests: the full suite passes (9216 tests, 0 failures) and phpstan is clean.

rmcdaniel and others added 5 commits November 20, 2026 09:26
…ory-free result

A lab run of DW Standard Workflow v1 against native Server moved about
385 KB of responses per workflow, enough to saturate a Cloud Dev cell's
4 Mbit/s egress at 1.5 workflows/second.

- The default transport sends Accept-Encoding: gzip when it owns a Guzzle
  client with cURL and zlib (cURL decodes), and gunzips a body a custom
  client left encoded (gzip magic plus Content-Encoding: gzip).
- Worker requests send X-Durable-Workflow-Server-Capabilities-Digest. A
  Server that supports it replaces repeated server_capabilities objects
  with their digest; ServerCapabilitiesDigest restores the held object in
  success bodies, refusals and poll frames (top level and carried answers)
  before any caller sees them. Older Servers ignore the header.
- result() of a completed run uses the terminal status read's
  output_envelope instead of downloading the whole history again; other
  terminal states and Servers without output_envelope still read history.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review fix. The transport held one digest, so a completion sent under D1
whose answer arrived after another response moved it to D2 (a rolling
deploy with overlapping requests, as completion pipelining allows) threw
"capabilities this worker does not hold" even though the server had
committed it. The server only ever replaces copies with the digest that
request sent, so each worker request now captures the digest and object it
is sent with, and its answer (success, refusal or poll frame) is restored
against that pair. A tagged copy still updates the pair used by later
requests.

Test: a completion in flight under D1 overlaps a poll that learns D2; the
completion restores the D1 object, the poll the D2 object, and the next
request sends D2. It fails on the previous single-digest code.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…tact

Review fixes:
- An answer naming capabilities the transport does not hold no longer ends
  the Worker. The transport forgets its pair and sends that request once
  more without the digest header, which returns the full object. Worker
  requests are fenced, so a request the Server already committed comes back
  as a duplicate 409 (task_not_leased / stale_attempt) that the Worker
  already handles.
- A digest the transport does not hold in an ERROR body is dropped; the HTTP
  error surfaces unchanged and is not retried.
- gzip bodies are inflated by the transport (cURL no longer inflates the
  bodies it asked for), in 8 KiB input steps, within maxDecodedResponseBytes
  (default 128 MiB, configurable on Psr18Transport). A larger body fails
  with the typed DurableWorkflow\Exception\ResponseTooLarge.
- Tests: garbled digest on a poll (retried in full; a managed Worker keeps
  running), on a committed completion (the duplicate 409 surfaces), in an
  error body, and a gzip bomb past a 16 KiB limit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Digest re-review:
- BLOCKER: a garbled 200 to an activity completion triggered the no-digest
  resend, the Server answered 409 stale_attempt for the completion it had
  already committed, and Worker::run exited. The committed-completion rule
  (409 stale_attempt, recorded false, our owner and attempt, every status
  completed) now applies to any answer: the first, after a transient
  failure, or after a digest resend. This block is identical to the one in
  #449, so whichever merges second merges cleanly. Workflow-task duplicates
  need nothing: Server 3.17.0 answers them 200 recorded:false
  already_completed, which the Worker already accepts.
- End-to-end test: garbled completion answer, then the resend's 409; the
  Worker keeps running and the activity runs once. Verified live against
  durableworkflow/server:3.17.0 as well.
- The default decode limit is min(32 MiB, memory_limit / 4), safe under
  PHP's default 128 MB, and ResponseTooLarge reaches callers unwrapped.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Digest re-review blocker: a workflow-task completion's answer can carry the
next leased activity. When its digest was garbled the transport resent the
completion; the Server answered the repeat as a duplicate without that
task, and the leased activity was lost (stuck Started until its lease
expired). Under a garbling proxy 9 of 15 workflows stalled.

On an unknown digest in a success answer the transport now keeps that
answer as received, reads the capabilities from the read-only
GET /api/cluster/info (worker_protocol.server_capabilities, the same
object) in their place, and forgets the held pair so the next request
learns a tagged copy. Only task polls, which are safe to repeat, are still
resent without the header. Errors are unchanged.

The reviewer's scenario is a test over the real Psr18Transport: a garbled
completion answer carrying the next activity. The completion is sent once,
cluster info is read, and the activity runs and completes exactly once. It
fails on the previous head.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit d99cae8 into main Nov 20, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant