Skip to content

Shrink worker egress: gzip and the server capabilities digest - #304

Merged
rmcdaniel merged 3 commits into
mainfrom
general/egress-digest-gzip
Nov 20, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
general/egress-digest-gzip

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Items 1 and 2 of #302: worker egress. It builds on durable-workflow/server#1261 (gzip) and durable-workflow/server#1262 (capabilities digest), both now merged to server main. The PHP reference is durable-workflow/sdk-php#446. This PR is separate from #303.

Change

gzip. reqwest is now built with its gzip feature, so every request sends Accept-Encoding: gzip and compressed answers are decoded transparently.

  • Cargo.lock adds only the compression crates: async-compression, compression-codecs/-core, flate2, zlib-rs/miniz_oxide, crc32fast, simd-adler32 and tokio-util. All of them build on Rust 1.86 (cargo +1.86.0 check --all-targets passes).

Capabilities digest (new src/server_capabilities_digest.rs, following docs/contracts/server-capabilities-digest.md):

  • Header. Worker-plane requests send X-Durable-Workflow-Server-Capabilities-Digest: <digest>|none. Control-plane requests don't send it.
  • Per-request resolution.
    • In send_json_with_timeout, the (digest, capabilities) pair is captured when the request is built, and storage-admission retries reuse it.
    • The answer is restored against that captured pair before any reader sees it. This covers success and error bodies, server_capabilities at the top level and up to two objects deep, as in a carried poll's answer.
    • So every existing reader (lease renewal, sticky execution, metrics, cooperative cancellation, query authority, signal_external) keeps seeing the full object.
  • Tagged copies. A copy carrying a digest member has that member stripped. The pair then becomes the one later requests send. Clones of a Client share it.
  • Unknown digest: recoverable. A digest string that matches neither the request's captured digest nor a tagged copy in the same answer no longer ends the worker.
    • The Client forgets its held pair and resends that same request once with none, which returns full, tagged capabilities.
    • If the resent request is a report the Server already committed, it gets the usual non-recorded receipt: 409 stale_attempt for an activity completion, task_not_leased for a workflow task. The worker already settles both.
    • Only an unknown digest on the resend is Error::Protocol with reason server_capabilities_digest_unknown.
  • Decoded body cap. Bodies stream with a cap: 128 MiB by default, configurable with the new ClientBuilder::max_response_bytes. A larger body stops being read and fails with the new Error::ResponseBodyTooLarge { limit_bytes }. This is additive because Error is #[non_exhaustive]. Raw requests use the same cap. It closes the review finding that a 1.5 MB gzip body expanded to 3.2 GB.
  • Rewriting. A body is re-serialized only when something was restored or stripped. Answers from older Servers keep their exact bytes.
  • Corpus guard. The module is named server_capabilities_digest partly because rustfmt sorts mod lines. Under the name capabilities_digest, the new line landed right after mod cancellation_scope_replay;, and the regression-corpus guard counted that neighbouring line as replay code. Nothing here touches replay.

API. Additive only: ClientBuilder::max_response_bytes and Error::ResponseBodyTooLarge. cargo semver-checks --baseline-version 4.21.0 (v0.51.0) reports 229 checks: 229 pass, 31 skip and Summary no semver update required.

Measurement

  • Server: native Server main 43ad8d8f (image general-egress-server:main-43ad8d8f, reports 3.17.0-main43ad8d8f), with sqlite and compression enabled by default. One container, capped at 1 CPU and 768 MB.
  • Workload: "DW Standard Workflow v1", run 20 times one after another. Each run is one start, one activity with a 1 KiB payload, and result() polling every 500 ms. The worker and the caller run in the same process.
  • Meter: a counting TCP proxy with separate ports for the worker client and the caller client. It counts every byte the Server sends to the SDK, HTTP headers included.
  • Same server, same binary harness. The only difference is the SDK path dependency: origin/main versus this branch.
Response bytes per workflow SDK main This branch Reduction
Worker 158.7 KB 8.8 KB −94%
Caller 19.3 KB 4.7 KB −75%
Total 178.0 KB 13.6 KB −92%

Both runs completed 20 out of 20 workflows with 0 failures in about 40 s. The worker's request bytes grew from 7.6 KB to 8.6 KB per workflow, because of the extra headers.

Tests

Test What it covers
server_capabilities_digest::tests (6) Full answers stay unchanged. A tagged copy is stripped and used later. Nested copies up to depth 3 and error bodies are restored; copies deeper than that or inside lists are left alone. Rolling deploy: a completion in flight under D1 resolves D1 after a poll already switched the client to D2, a later request resolves D2, and a D2 tag plus a D2 string in one answer both resolve. Unknown digests fail. Non-digest strings and non-object bodies are left alone.
tests/server_capabilities_digest.rs: worker_answers_are_restored_from_the_digest_each_request_sent A garbled poll answer is resent once with none and recovers. An unknown digest on the resend is Error::Protocol.
tests/server_capabilities_digest.rs: a_completion_resent_after_a_garbled_answer_gets_its_committed_receipt A garbled answer to a committed completion is resent and gets the 3.17 409 stale_attempt receipt.
tests/server_capabilities_digest.rs: decoded_response_bodies_are_capped 91 gzip bytes decode to 1574, over a 1000-byte cap, giving Error::ResponseBodyTooLarge.
tests/server_capabilities_digest.rs (public API) Requests carry Accept-Encoding: gzip and a gzip answer decodes. Worker polls send none, then the learned digest; the tagged copy is stripped, the digest string is restored, and an unknown digest raises Error::Protocol. Control-plane requests don't send the header.

Results:

  • cargo fmt --check: clean.
  • cargo clippy --all-targets --all-features -- -D warnings: clean.
  • cargo test: 1103 passed, 0 failed, 82 ignored (server-gated).
  • cargo +1.86.0 check --all-targets: passes.
  • The regression-corpus validator passes.

🤖 Generated with Claude Code

@rmcdaniel rmcdaniel added enhancement New feature or request agent:general labels Nov 20, 2026
Build reqwest with its gzip feature, so every request sends
Accept-Encoding: gzip and compressed answers are decoded (server#1261).

Worker-plane requests send X-Durable-Workflow-Server-Capabilities-Digest
with the digest this Client holds, or none (server#1262). The digest
and capabilities object are captured when each request is built, and
that request's answer (success or error, top level and up to two
objects deep) is restored against the captured pair before any reader
sees it. A tagged copy is stripped of its digest member and used for
later requests; clones share it. Overlapping requests across a rolling
deploy therefore each restore their own capabilities. A digest string
that matches neither the request's digest nor a tagged copy in the same
answer is Error::Protocol (server_capabilities_digest_unknown). Bodies
are re-serialized only when something was restored or stripped.

Lab run against native Server main 43ad8d8f, 20 standard one-activity
workflows: response bytes per workflow fell from 158.7 KB to 8.8 KB for
the worker and from 19.3 KB to 4.7 KB for the caller.

No public API change; cargo semver-checks against 4.21.0 reports no
semver update required.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel
rmcdaniel force-pushed the general/egress-digest-gzip branch from 4699b4e to 0b65da1 Compare November 20, 2026 05:26
rmcdaniel and others added 2 commits November 20, 2026 06:36
Review fixes for the digest and gzip change.

An answer naming a digest the request did not send used to be
Error::Protocol, which ended the worker on the first garbled poll. The
Client now forgets its held digest and resends that same request once
with "none", which returns full (tagged) capabilities. A resent report
the Server already committed comes back as its usual non-recorded
receipt (409 stale_attempt for an activity completion, task_not_leased
for a workflow task), which the worker already settles. Only an unknown
digest on the resend is a protocol failure.

Decoded bodies were read with bytes(), so a small gzip answer could
expand without bound. Bodies now stream with a cap, 128 MiB by default
and configurable with ClientBuilder::max_response_bytes; a larger body
fails with the new Error::ResponseBodyTooLarge (Error is
non_exhaustive). Raw requests use the same cap.

cargo semver-checks against 4.21.0: no semver update required.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
With full-jitter renewal backoff (#303) a retry can be drawn near zero
and fire before the callback settles, so the test now asserts at least
one renewal instead of exactly one. Seen on the Rust 1.86 CI job.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit 3897965 into main Nov 20, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant