Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Waits for expiry if it cannot stop:** if the execution does not stop in time, for example a thread that never heartbeats, nothing is sent. The Server settles the attempt at lease expiry (`lease_expired`), so two executions of one attempt never overlap. A claim that never started is released directly. A typed Server refusal (for example `lease_expired` or `run_closed`) still ends the claim as before. See "Activity lease renewal" in the SDK reference.
- **Replay safety:** worker-side lease handling only. No workflow commands or recorded history change.

### Changed

- Workers send far fewer response bytes on a Server that supports the capabilities digest (#352, durable-workflow/server#1262).
- **The header:** worker-plane requests now send `X-Durable-Workflow-Server-Capabilities-Digest` (`none` until the first full copy arrives).
- **The Server's answer:** the Server replaces each repeated `server_capabilities` object (about 17 KB on native 3.15/3.16) with its digest, or sends a full copy tagged with a `digest` member.
- **What callers see:** the SDK restores elided copies before any caller sees the answer, in success bodies, error bodies and negotiated poll envelopes, up to three levels deep. It strips the `digest` member, so callers still see complete capabilities.
- **Overlapping requests:** each answer is resolved against the digest and object its own request was sent with. A request in flight during a rolling deploy that changes the capabilities restores the object it was sent with, while later requests use the newer digest.
- **Errors:** a digest the worker does not hold raises the new `ServerCapabilitiesDigestError`, a protocol failure.
- **Older Servers** ignore the header and answer in full as before.
- Response bodies are bounded after decoding: 128 MiB by default, configurable with `Client(max_response_body_bytes=...)` (and the sync client). httpx decodes `Content-Encoding: gzip` transparently, so before this a small compressed answer could expand without limit in memory.
- **How:** each response stream is counted as it decodes, inflating at most 1 MiB at a time. It is refused with the new `ResponseBodyTooLarge` before httpx decodes past the limit, and that error is not retried. The bound also applies to runtime external-payload fetches.
- **Encodings:** the client now asks only for `gzip`, which is what native Server sends. Any other response `Content-Encoding` is refused with `httpx.DecodingError`.
- `WorkflowHandle.result()` and `Client.get_result()` no longer read the run's history for a completed run when the status read already carries that run's `output_envelope`. The status read is `GET /workflows/{id}`, which the result poll makes anyway. Failed, cancelled, terminated and timed-out runs still read the history for their typed failure, and so does a completed run whose status read has no `output_envelope` or names a different run.

## [2.20.0] - 2026-11-19

Both additions need **native Server 3.15.0 or later**, the first Server release that advertises `capabilities.signal_with_start` and the `signal_external_workflow` worker command. Each is discovered from the Server's capabilities and refused before anything is sent on an older Server, so the SDK's supported Server baselines are unchanged and every other feature keeps working on older Servers. Worker protocol (1.20) and control-plane (2) versions are unchanged.
Expand Down
84 changes: 84 additions & 0 deletions src/durable_workflow/_capabilities_digest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
"""Worker answers without repeated ``server_capabilities`` (server#1262).

A worker request sends the digest of the capabilities this client holds, or
``none``. A Server that supports the digest answers with each copy of its
current capabilities either tagged with a ``digest`` member, which is kept for
later requests, or replaced by the digest string, which is restored here before
any caller sees the answer. Older Servers ignore the header and answer in full,
so callers always observe complete capabilities.

The Server only replaces copies with the digest that request sent, so an answer
is resolved against the pair captured when its request was sent, not the pair
held when it arrives. Overlapping requests across a change of capabilities (a
rolling deploy) each restore their own object. Contract:
``docs/contracts/server-capabilities-digest.md`` in the Server repository.
"""

from __future__ import annotations

import copy
from typing import Any

from .errors import ServerCapabilitiesDigestError

HEADER = "X-Durable-Workflow-Server-Capabilities-Digest"
_KEY = "server_capabilities"
_PREFIX = "sha256:"
# The Server elides copies at the top level and up to two objects deep.
_MAX_DEPTH = 3

Captured = tuple[str | None, dict[str, Any] | None]


class ServerCapabilitiesDigest:
"""The newest (digest, capabilities) pair this client has learned."""

def __init__(self) -> None:
self._digest: str | None = None
self._capabilities: dict[str, Any] | None = None

def capture(self) -> Captured:
"""The pair a request is sent with; pass it to :meth:`restore` for its answer."""
return self._digest, self._capabilities

@staticmethod
def header(captured: Captured) -> str:
return captured[0] if captured[0] is not None and captured[1] is not None else "none"

def restore(self, answer: Any, captured: Captured) -> None:
"""Strip and learn tagged copies, then restore elided ones, in place."""
if not isinstance(answer, dict):
return
learned: dict[str, dict[str, Any]] = {}
self._learn(answer, learned, 1)
self._resolve(answer, captured, learned, 1)

def _learn(self, answer: dict[str, Any], learned: dict[str, dict[str, Any]], depth: int) -> None:
for key, value in answer.items():
if key == _KEY:
if isinstance(value, dict) and isinstance(value.get("digest"), str) \
and value["digest"].startswith(_PREFIX):
digest = value.pop("digest")
learned[digest] = value
# Later requests send the newest digest this client has seen.
# Callers own the answer, so the held pair is a private copy.
self._digest, self._capabilities = digest, copy.deepcopy(value)
elif isinstance(value, dict) and depth < _MAX_DEPTH:
self._learn(value, learned, depth + 1)

def _resolve(self, answer: dict[str, Any], captured: Captured, learned: dict[str, dict[str, Any]],

Check failure on line 69 in src/durable_workflow/_capabilities_digest.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this function to reduce its Cognitive Complexity from 16 to the 15 allowed.

See more on https://sonarcloud.io/project/issues?id=durable-workflow_sdk-python&issues=AaH3VI3rDKUrjaOBRNbA&open=AaH3VI3rDKUrjaOBRNbA&pullRequest=354
depth: int) -> None:
for key, value in answer.items():
if key == _KEY:
if isinstance(value, str) and value.startswith(_PREFIX):
if value == captured[0] and captured[1] is not None:
answer[key] = copy.deepcopy(captured[1])
elif value in learned:
answer[key] = copy.deepcopy(learned[value])
else:
raise ServerCapabilitiesDigestError(
"the Server elided server_capabilities this worker does not hold "
f"({value}; sent {self.header(captured)})"
)
elif isinstance(value, dict) and depth < _MAX_DEPTH:
self._resolve(value, captured, learned, depth + 1)
74 changes: 74 additions & 0 deletions src/durable_workflow/_response_limit.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
"""Bound the decoded size of every HTTP response body.

httpx decodes ``Content-Encoding: gzip`` transparently, so a small compressed
answer could expand without limit in memory (a 1.5 MB gzip body can inflate to
gigabytes). Each response stream is wrapped before httpx reads it: the wrapper
counts the bytes the body decodes to, inflating at most 1 MiB at a time, and
raises :class:`~durable_workflow.errors.ResponseBodyTooLarge` before httpx
decodes a chunk that would cross the limit.
"""

from __future__ import annotations

import zlib
from collections.abc import AsyncIterator, Awaitable, Callable

import httpx

from .errors import ResponseBodyTooLarge

#: Default bound on a decoded response body: 128 MiB.
DEFAULT_MAX_RESPONSE_BODY_BYTES = 128 * 1024 * 1024
#: Encodings this client asks for; httpx is not offered others.
ACCEPT_ENCODING = "gzip"
_INFLATE_STEP = 1 << 20


class _BoundedBody(httpx.AsyncByteStream):
def __init__(self, inner: httpx.AsyncByteStream, *, gzip: bool, limit: int, url: str) -> None:
self._inner = inner
self._gzip = gzip
self._limit = limit
self._url = url

async def __aiter__(self) -> AsyncIterator[bytes]:

Check failure on line 34 in src/durable_workflow/_response_limit.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this function to reduce its Cognitive Complexity from 19 to the 15 allowed.

See more on https://sonarcloud.io/project/issues?id=durable-workflow_sdk-python&issues=AaH3t1CHpTi907jwBdlB&open=AaH3t1CHpTi907jwBdlB&pullRequest=354
inflater = zlib.decompressobj(16 + zlib.MAX_WBITS) if self._gzip else None
decoded = 0
async for chunk in self._inner:
if inflater is None:
decoded += len(chunk)
else:
pending = chunk
while pending and not inflater.eof:
try:
decoded += len(inflater.decompress(pending, _INFLATE_STEP))
except zlib.error:
# Not ours to judge: httpx reports the invalid encoding.
inflater = None
break
if decoded > self._limit:
break
pending = inflater.unconsumed_tail
if decoded > self._limit:
raise ResponseBodyTooLarge(self._limit, self._url)
yield chunk

async def aclose(self) -> None:
await self._inner.aclose()


def response_body_limit(limit: int) -> Callable[[httpx.Response], Awaitable[None]]:
"""An httpx response hook that bounds the decoded body at ``limit`` bytes."""

async def bound(response: httpx.Response) -> None:

Check warning on line 63 in src/durable_workflow/_response_limit.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Use asynchronous features in this function or remove the `async` keyword.

See more on https://sonarcloud.io/project/issues?id=durable-workflow_sdk-python&issues=AaH3t1CHpTi907jwBdlC&open=AaH3t1CHpTi907jwBdlC&pullRequest=354
encoding = response.headers.get("content-encoding", "").strip().lower()
if encoding not in ("", "identity", "gzip"):
raise httpx.DecodingError(
f"unrequested response content-encoding {encoding!r}", request=response.request,
)
if isinstance(response.stream, httpx.AsyncByteStream):
response.stream = _BoundedBody(
response.stream, gzip=encoding == "gzip", limit=limit, url=str(response.request.url.path),
)

return bound
38 changes: 35 additions & 3 deletions src/durable_workflow/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
import warnings
from collections.abc import AsyncIterator, Awaitable, Callable, Mapping, Sequence
from dataclasses import dataclass
from dataclasses import field as dataclass_field
from importlib.metadata import PackageNotFoundError
from importlib.metadata import version as _pkg_version
from typing import Any, cast
Expand All @@ -49,6 +50,8 @@
CancellationScopeDeliveryReceipt,
scope_timestamp,
)
from ._capabilities_digest import HEADER as CAPABILITIES_DIGEST_HEADER
from ._capabilities_digest import ServerCapabilitiesDigest
from ._cooperative_cancellation import CancellationDelivery, CancellationRequest
from ._failure_reporting import (
DETAIL_BYTES,
Expand All @@ -64,6 +67,7 @@
from ._poll_transport import unwrap_response as _unwrap_poll_response
from ._poll_transport import warn_early_cutoff as _warn_poll_cutoff
from ._query_task_authority import QueryTaskAuthority, current_query_authority
from ._response_limit import ACCEPT_ENCODING, DEFAULT_MAX_RESPONSE_BODY_BYTES, response_body_limit
from ._typed_values import decode_payload_value, to_payload_values, validate_annotation
from .errors import (
REQUEST_ID_AMBIGUOUS,
Expand Down Expand Up @@ -486,6 +490,8 @@
payload_codec: str | None = None
memo: dict[str, Any] | None = None
search_attributes: dict[str, Any] | None = None
# Whether ``output`` was decoded from the status read's ``output_envelope``.
_output_from_envelope: bool = dataclass_field(default=False, init=False, repr=False, compare=False)


@dataclass
Expand Down Expand Up @@ -1923,7 +1929,7 @@
values raise ``ValueError`` naming the option, without exposing its value.
"""

def __init__(

Check failure on line 1932 in src/durable_workflow/client.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this function to reduce its Cognitive Complexity from 17 to the 15 allowed.

See more on https://sonarcloud.io/project/issues?id=durable-workflow_sdk-python&issues=AaH3t1I9pTi907jwBdlD&open=AaH3t1I9pTi907jwBdlD&pullRequest=354
self,
base_url: str,
*,
Expand All @@ -1943,6 +1949,7 @@
external_storage_threshold_bytes: int | None = None,
external_storage_cache: ExternalPayloadCache | None = None,
telemetry: ExecutionTelemetry | None = None,
max_response_body_bytes: int = DEFAULT_MAX_RESPONSE_BODY_BYTES,
) -> None:
_validate_static_credential("token", token)
_validate_static_credential("control_token", control_token)
Expand Down Expand Up @@ -1983,11 +1990,17 @@
if external_storage_cache is not None
else ExternalPayloadCache()
)
if type(max_response_body_bytes) is not int or max_response_body_bytes < 1:
raise ValueError("max_response_body_bytes must be a positive integer")
self.max_response_body_bytes = max_response_body_bytes
self._http = httpx.AsyncClient(
base_url=self.base_url, timeout=timeout, verify=tls_context if tls_context is not None else True,
headers={"Accept-Encoding": ACCEPT_ENCODING},
event_hooks={"response": [response_body_limit(max_response_body_bytes)]},
)
self._cluster_info: dict[str, Any] | None = None
self._cluster_info_lock = asyncio.Lock()
self._capabilities_digest = ServerCapabilitiesDigest()
self._runtime_external_payload_transport_cache: (
_RuntimeExternalPayloadTransport | None
) = None
Expand Down Expand Up @@ -2242,6 +2255,8 @@
status_code = "none"
outcome = "pending"
request_started = time.monotonic()
# The digest pair this request is sent with; its answer resolves against it.
capabilities_digest = self._capabilities_digest.capture() if worker else None

async def _do_request() -> httpx.Response:
nonlocal request_started
Expand All @@ -2250,6 +2265,8 @@
if worker and activity_authority is not None and activity_authority.governs(self, self.namespace, path):
request_timeout = min(request_timeout, activity_authority.remaining())
headers = self._discovery_headers() if discovery else self._headers(worker=worker)
if capabilities_digest is not None:
headers[CAPABILITIES_DIGEST_HEADER] = ServerCapabilitiesDigest.header(capabilities_digest)
if progress_poll:
headers["Accept"] = _POLL_ACCEPT
pending = self._http.request(
Expand Down Expand Up @@ -2290,6 +2307,8 @@
body = exc.response.json()
except ValueError:
body = exc.response.text
if capabilities_digest is not None:
self._capabilities_digest.restore(body, capabilities_digest)
# The typed completion-size refusal carries the limit; the
# size is the encoded request body this client sent.
if raise_for_status is _raise_for_status:
Expand All @@ -2305,6 +2324,8 @@
outcome = "ok"
return None
result = resp.json()
if capabilities_digest is not None:
self._capabilities_digest.restore(result, capabilities_digest)
if (worker and activity_authority is not None
and activity_authority.governs(self, self.namespace, path)
and path.endswith("/heartbeat") and isinstance(result, dict)
Expand Down Expand Up @@ -4132,6 +4153,7 @@
data = await self._request("GET", f"/workflows/{workflow_id}", context=workflow_id)
input_val = data.get("input")
output_val = data.get("output")
output_from_envelope = False
envelope_jobs: list[tuple[str, Any]] = []
if data.get("input_envelope"):
envelope_jobs.append(("input", data["input_envelope"]))
Expand All @@ -4143,12 +4165,13 @@
external_storage=self.external_storage,
external_storage_cache=self.external_storage_cache,
)
for (field, _), value in zip(envelope_jobs, decoded, strict=True):
if field == "input":
for (field_name, _), value in zip(envelope_jobs, decoded, strict=True):
if field_name == "input":
input_val = value
else:
output_val = value
return WorkflowExecution(
output_from_envelope = True
execution = WorkflowExecution(
workflow_id=data.get("workflow_id", workflow_id),
run_id=data.get("run_id"),
workflow_type=data.get("workflow_type", ""),
Expand All @@ -4163,6 +4186,8 @@
data.get("search_attributes") if isinstance(data.get("search_attributes"), dict) else None
),
)
execution._output_from_envelope = output_from_envelope
return execution

async def list_workflows(
self,
Expand Down Expand Up @@ -5508,6 +5533,13 @@
status = desc.status
if status in ("completed", "failed", "terminated", "canceled", "cancelled"):
run_id = handle.run_id or desc.run_id
if (status == "completed" and desc._output_from_envelope and desc.run_id is not None
and desc.run_id == run_id):
# The status read already carries this run's decoded output;
# the history is only needed for a non-success outcome.
if result_type is None:
return desc.output
return decode_payload_value(desc.output, result_type, "workflow result")
if run_id is None:
raise WorkflowFailed("no run_id available to fetch history")
page_token: str | None = None
Expand Down
22 changes: 22 additions & 0 deletions src/durable_workflow/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,28 @@ def __init__(self, query_task_id: str, reason: str) -> None:
self.reason = reason


class ResponseBodyTooLarge(DurableWorkflowError):
"""A response body decoded to more bytes than the client allows.

The bound (``Client(max_response_body_bytes=...)``, 128 MiB by default)
applies after ``Content-Encoding`` decoding, so a small compressed answer
cannot expand without limit in memory. Not retried.
"""

def __init__(self, limit: int, path: str = "") -> None:
super().__init__(f"response body{f' from {path}' if path else ''} exceeds {limit} decoded bytes")
self.limit = limit
self.path = path


class ServerCapabilitiesDigestError(DurableWorkflowError):
"""A worker answer elided ``server_capabilities`` with a digest this worker does not hold.

The Server only elides with the digest a request sent (server#1262), so
this is a protocol failure, never a transient condition.
"""


class ActivityWorkerLost(DurableWorkflowError):
"""An activity delivery lost its original live execution authority.

Expand Down
3 changes: 3 additions & 0 deletions src/durable_workflow/sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from collections.abc import Coroutine
from typing import Any, TypeVar

from ._response_limit import DEFAULT_MAX_RESPONSE_BODY_BYTES
from .client import Client as AsyncClient
from .client import (
NamespaceDescription,
Expand Down Expand Up @@ -320,6 +321,7 @@ def __init__(
external_storage: ExternalStorageDriver | None = None,
external_storage_threshold_bytes: int | None = None,
external_storage_cache: ExternalPayloadCache | None = None,
max_response_body_bytes: int = DEFAULT_MAX_RESPONSE_BODY_BYTES,
) -> None:
self._async = AsyncClient(
base_url,
Expand All @@ -334,6 +336,7 @@ def __init__(
external_storage=external_storage,
external_storage_threshold_bytes=external_storage_threshold_bytes,
external_storage_cache=external_storage_cache,
max_response_body_bytes=max_response_body_bytes,
)
self._runner = _LoopRunner()
self._closed = False
Expand Down
Loading