Skip to content
Merged
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- A worker answer that names a capabilities digest the worker does not hold no longer strands the request (PHP and Rust parity). Before, it raised `ServerCapabilitiesDigestError`, and a leased task sat until its lease expired. In a lab run with garbled digests, 0 of 10 workflows finished in 150 s.
- **Non-idempotent requests are never resent.** For a completion, failure report or heartbeat, the client keeps the answer as received, since everything but the capabilities object is valid. It reads the capabilities with a read-only `GET /api/cluster/info` (`worker_protocol.server_capabilities`, the same object) and forgets its held digest, so later requests send `none` and learn a fresh copy. If discovery is unavailable, the field is dropped; no caller reads capabilities from those answers.
- **Task polls** are safe to repeat, so a poll is instead resent once with `none` to get the full object. An unknown digest on that resend, or in an answer to a request that already sent `none`, is a protocol error.
- **Error bodies:** an unknown digest in an error body is dropped. The real HTTP error surfaces unchanged and is not resent.
- A worker no longer exits when the runtime is warming up or briefly unavailable while it registers (same issue as durable-workflow/sdk-php#448). Before, `Worker.run()` and `Worker.run_until()` raised once the client's three transport attempts had failed on `GET /api/cluster/info` or `POST /api/worker/register`. That covered native Server's `503 {"reason":"storage_unavailable"}` during warm-up, a refused connection and an upstream 502/503/504.
- **Retried:** discovery and registration are now repeated with the backoff shared across SDKs, `uniform(0, min(4, 0.25 * 2**(n - 1)))` seconds, logging a warning each time. The transient failures are: refused, reset or timed-out connections; upstream 502/503/504 without a protocol body; 503 `storage_unavailable` or `backend_unavailable`; and any 5xx or 429 with `retryable: true`.
- **Still fatal:** `retryable: false`, HTTP 401/403, other 4xx refusals and an incompatible Server still stop the worker. It now logs `worker <id> stopping: registration failed with a non-retryable error: ...` at error level first.
- **Activity outcomes:** an activity completion or failure report that hits one of those transients is now sent again with the same backoff while the attempt's lease covers the next wait. Before, it was dropped after the client's three quick attempts, with only a warning. A transient never turns a completion into a failure report. When the lease runs out, the worker stops resending and logs an error, and the Server settles the attempt at lease expiry.
- **Duplicate completions:** a resend answered `409 stale_attempt` with `recorded: false`, the same task, attempt and lease owner, and activity, attempt and task status all `completed` is this worker's committed completion and counts as delivered. That is what native Server 3.17.0 answers after a transient hid the committed first completion, and it is the same rule as PHP. Any other `stale_attempt` is a closed attempt and is never treated as success.
- **Poll-time auth refusals:** pollers still retry 401/403 so credentials can rotate under a running worker. PHP and Rust stop the worker instead. Every such refusal is now logged at ERROR level, not rate-limited as a warning, including refusals before a later recovery.
- **Poll backoff:** poll retry waits now follow `uniform(0, min(10, 0.25 * 2**(n - 1)))` seconds, the shared full jitter with a 10 s cap, while still honouring a bounded Server retry delay. Before, the ceiling started at 1 s and doubled to 30 s, which slowed recovery.
- **Long outages:** a poller failing continuously for 60 s now logs the outage at ERROR level, then every 5 minutes while it lasts. Before, it logged only a warning about once a minute. A successful poll resets this.
- **Heartbeats:** worker heartbeats already rode out these failures.
- `WorkflowHandle.result()` and `Client.get_result()` on a handle bound to a run that continued as new no longer silently return `None` (#356). They now follow the continue-as-new chain to the run that actually ended, as the PHP and Rust SDKs do. That run's result is returned, or its typed failure is raised. The chain is followed through each `WorkflowContinuedAsNew` event's `continued_to_run_id`; when no successor is recorded, the newest run named by the status read is used. A chain that cannot be followed raises `WorkflowFailed`.
- An activity worker no longer silently drops a claimed task when lease renewal fails transiently (#352, same bug as durable-workflow/sdk-php#445). Before, if the `renew-lease` confirmation sent right after an activity poll timed out, hit a connection reset or got a 5xx/429, the poll raised `ActivityLeaseAuthorityLost("activity_initial_lease_unavailable")` and the worker discarded the task. The Server then held the lease (300 s by default) until it expired and, with the default single attempt, failed the workflow.
- **Keeps working:** the activity now runs under the lease it already holds, and renewal is retried inside that lease. The backoff is the same in every SDK: the *n*-th retry waits `uniform(0, min(4, 0.25 * 2**n))` seconds, so 0.25 s doubling to a 4 s cap, with full jitter. Each renewal exchange is bounded to leave a reserve of the lease (at most 10 s, at most a fifth of the grant). An unconfirmed lease is charged from the poll's start, because the Server stamps it only after receiving the poll.
Expand Down
91 changes: 80 additions & 11 deletions docs/reference/worker.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,17 +90,31 @@ This backoff requires Python SDK 2.10.0 or later. Earlier packages use a fixed
one-second poll-error wait.

Workflow, activity and query pollers back off independently after failed polls,
including HTTP errors, rejected credentials and unreachable runtimes. The
retry ceiling starts at 1 second, doubles on each consecutive failure and stops
at 30 seconds. Each delay is chosen randomly between half and all of that
ceiling, so workers do not retry in lockstep. Any successful poll, including an
empty response, resets the ceiling to 1 second. Shutdown interrupts the wait.

The first failure is logged immediately. An unchanged failure is logged at most
once per minute per poller until polling succeeds. HTTP 401 identifies rejected
authentication; HTTP 403 identifies denied authorization. Check the worker
credentials and their permissions. Once requests succeed again, polling
recovers without a worker restart. Poll-error metrics
including HTTP errors, rejected credentials and unreachable runtimes. The wait
before the *n*-th consecutive retry is `uniform(0, min(10, 0.25 * 2**(n - 1)))`
seconds. That is the shared full-jitter schedule with a 10-second cap, so a
poller recovers quickly once the runtime is back, and workers do not retry in
lockstep. A bounded retry delay advertised by the Server is still honoured. Any
successful poll, including an empty response, resets the schedule. Shutdown
interrupts the wait.

The first failure is logged as a warning immediately. An unchanged failure is
logged at most once per minute per poller. Once a poller has failed
continuously for 60 seconds, the outage is logged at ERROR level, then again
every 5 minutes while it lasts. A successful poll ends the outage, and the next
failure starts again with a warning.

Authentication and authorization refusals are the exception to that limit:
**every** poll answered with HTTP 401 (rejected authentication) or HTTP 403
(denied authorization) is logged at ERROR level, including refusals that come
before a later recovery. Check the worker credentials and their permissions.
The poller keeps retrying with the same backoff, so rotated credentials
recover without a worker restart.

This differs from the PHP and Rust SDKs, which log an error and stop the
worker on a poll-time 401 or 403. Python retries so that credentials can be
rotated under a running worker. Registration at startup treats 401 and 403 as
fatal in every SDK (see below). Poll-error metrics
still count every failed poll, including failures whose warning is suppressed.

Starting with Python SDK 2.10.0, warnings name the exception class and static
Expand All @@ -114,6 +128,61 @@ Typed capacity backpressure follows the Server's advertised retry delay.
Storage-admission failures retain their separate identity-preserving retry and
worker-stop behavior. Neither uses this generic poll-failure backoff.

### Startup while the runtime is unavailable

`Worker.run()` and `Worker.run_until()` also ride out a runtime that is warming
up or briefly unavailable while they register. Cluster discovery and
registration are repeated whole after a transient failure:

- a refused, reset or timed-out connection
- an upstream 502, 503 or 504 without a protocol body
- a 503 whose reason is `storage_unavailable` or `backend_unavailable`
(native Server answers every storage failure with this body)
- any 5xx or 429 that says `retryable: true`

The wait before the *n*-th retry is `uniform(0, min(4, 0.25 * 2**(n - 1)))`
seconds. This is the backoff schedule shared by every Durable Workflow SDK. Each
retry logs a warning, and shutdown interrupts the wait.

Some failures are never retried:

- `retryable: false`
- HTTP 401 or 403
- any other 4xx protocol refusal
- an incompatible Server

For these, the worker logs `worker <id> stopping: registration failed with a
non-retryable error: ...` at error level, and `run()` raises the error.

### Activity outcomes through transient failures

An activity completion or failure report that hits one of the transient
failures listed above is sent again with the same backoff, as long as the
attempt's lease still covers the next wait. The lease is the renewed lease when
the Server supports activity lease renewal, and the claimed `lease_expires_at`
otherwise. A transient never turns a completion into a failure report.

When the lease cannot cover another wait, the worker stops resending and logs
`activity task <id> ... outcome not delivered after N transient failures ...:
its lease ran out` at error level. The Server then settles the attempt when its
lease expires.

A transient can hide an outcome the Server already committed. Native Server
answers the resend with `409` and this body:

```json
{"reason": "stale_attempt", "recorded": false, "outcome": "completed",
"lease_owner": "<this worker>", "activity_status": "completed",
"attempt_status": "completed", "task_status": "completed"}
```

The body also carries the same task and attempt IDs. That exact shape counts
as this worker's committed completion, so the activity is not run again and
no second outcome is recorded. Any other `stale_attempt` means the attempt is
closed. It is reported as such and never treated as success. After
registration, pollers keep the poll-failure backoff described above. Worker
heartbeats log a failure and try again at the next interval.

## Lifecycle and shutdown

`Worker.run()` and `Worker.run_until()` register through the worker protocol
Expand Down
34 changes: 27 additions & 7 deletions src/durable_workflow/_capabilities_digest.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,25 @@ def capture(self) -> Captured:
def header(captured: Captured) -> str:
return captured[0] if captured[0] is not None and captured[1] is not None else "none"

def restore(self, answer: Any, captured: Captured) -> None:
"""Strip and learn tagged copies, then restore elided ones, in place."""
def forget(self) -> None:
"""Drop the held pair, so the next request sends ``none`` and gets full capabilities."""
self._digest = None
self._capabilities = None

def restore(self, answer: Any, captured: Captured, *, drop_unknown: bool = False,
substitute: dict[str, Any] | None = None) -> bool:
"""Strip and learn tagged copies, then restore elided ones, in place.

An elided copy naming a digest neither captured for the request nor
tagged in this answer raises :class:`ServerCapabilitiesDigestError`.
With ``substitute`` it is replaced by a copy of that object; with
``drop_unknown`` it is removed. Either way ``True`` is returned.
"""
if not isinstance(answer, dict):
return
return False
learned: dict[str, dict[str, Any]] = {}
self._learn(answer, learned, 1)
self._resolve(answer, captured, learned, 1)
return self._resolve(answer, captured, learned, 1, drop_unknown, substitute)

def _learn(self, answer: dict[str, Any], learned: dict[str, dict[str, Any]], depth: int) -> None:
for key, value in answer.items():
Expand All @@ -67,18 +79,26 @@ def _learn(self, answer: dict[str, Any], learned: dict[str, dict[str, Any]], dep
self._learn(value, learned, depth + 1)

def _resolve(self, answer: dict[str, Any], captured: Captured, learned: dict[str, dict[str, Any]],
depth: int) -> None:
for key, value in answer.items():
depth: int, drop_unknown: bool, substitute: dict[str, Any] | None) -> bool:
dropped = False
for key, value in list(answer.items()):
if key == _KEY:
if isinstance(value, str) and value.startswith(_PREFIX):
if value == captured[0] and captured[1] is not None:
answer[key] = copy.deepcopy(captured[1])
elif value in learned:
answer[key] = copy.deepcopy(learned[value])
elif substitute is not None:
answer[key] = copy.deepcopy(substitute)
dropped = True
elif drop_unknown:
del answer[key]
dropped = True
else:
raise ServerCapabilitiesDigestError(
"the Server elided server_capabilities this worker does not hold "
f"({value}; sent {self.header(captured)})"
)
elif isinstance(value, dict) and depth < _MAX_DEPTH:
self._resolve(value, captured, learned, depth + 1)
dropped = self._resolve(value, captured, learned, depth + 1, drop_unknown, substitute) or dropped
return dropped
Loading