See Separate workflow and activity processes for a runnable deployment example with distinct registrations, independent slots, remote activity retries and signal-driven shutdown.
By default, ordinary synchronous def remote activities run in a dedicated
thread pool bounded by max_concurrent_activity_tasks. Each thread receives
its own activity.context(). A blocked synchronous call does not occupy the
worker event loop, so other activity slots, workflow tasks and worker
heartbeats can progress. Async async def activities run on that event loop
and must use nonblocking I/O. Blocking work inside an async function still
stalls the loop.
From a synchronous activity, call
activity.context().heartbeat_sync(details) to report progress with the
original task, attempt and worker identity. It waits for the owner's event
loop to send the request using the normal client transport timeouts and
raises ActivityCancelled when cancellation is reported. Async activities use
await activity.context().heartbeat(details). Calling the synchronous method
on the owner event loop raises an error instead of deadlocking it.
Threads suit blocking I/O; they do not provide hard isolation or parallel Python CPU execution. Use bounded downstream timeouts and cooperative heartbeat checks. For CPU-heavy or untrusted callbacks, use separately isolated worker processes and their resource limits. The separately negotiated cooperative-cancellation capability retains its supervised process execution; selecting a default thread does not enable that protocol.
This behavior is available in Python SDK 2.11.0 and later.
Failure reports retain at most 62 KiB of UTF-8 message text, 128 KiB of stack
text and 64 KiB of serialized details and diagnostics. These budgets include
truncation markers. The complete encoded JSON request body defaults to at most
1 MiB, counting JSON escaping, claim identities, command batches and metadata.
The message budget reserves 2 KiB below the original 64 KiB ceiling for
runtime-derived summaries.
A lower discovered Server limits.max_payload_bytes also applies. Configure
Client(..., failure_report_limit_bytes=...) when a proxy or transport has a
lower request limit. This option can lower the default cap; it cannot raise it.
Query-client clones inherit it. Payload warning thresholds remain advisory.
Oversized diagnostics produce a visible [truncated ...] message marker with
structured omitted-field names and original byte sizes. Activity failures also
carry this information in failure.diagnostics.failure_report; terminal
workflow failures use exception.diagnostics.failure_report. Other existing
report shapes keep the diagnostic JSON in the message marker. Unknown sizes
are null when optional details could not be encoded. Avro details are kept as
an intact valid envelope or omitted whole, before an unnecessary payload upload.
The original failure type, retry/non-retryable choice, authored retry delay and claim identity remain exact. Activity, workflow-task, query and update rejection reports, terminal workflow/update failures and local-activity attempt reports share the complete-request bound. A workflow-task replay failure still leaves the run open for repair; it does not become a terminal workflow failure.
A local activity's failure is bounded before workflow code observes it. First execution and cold replay therefore see the same persisted message. If a later command batch or newly discovered lower limit cannot fit that observed value, reporting fails explicitly rather than changing a value already used by the workflow. Prepared local activities observe the runtime's canonical history.
If essential identity or semantic fields cannot fit even without optional
text, the client raises FailureReportTooLarge before sending the report.
This reporting refusal is separate from the application failure. The worker's
existing reporting-error log identifies it; shortening a task ID or replacing
the application type would change which failure the runtime records.
Worker poll requests advertise optional JSON progress responses and also accept the ordinary JSON response from Servers that do not implement that transport. When a Server sends progress while waiting, each HTTP attempt still has an absolute deadline. Receiving another progress byte does not extend it. The existing transport retry budget and logical poll identity continue to apply. The SDK interprets the terminal status and retry headers before the ordinary error handling, including errors that arrive after HTTP 200 progress began.
If a connection failure, HTTP 502 or HTTP 504 ends a poll before its requested wait, a warning names the observed duration and configured wait. Check the proxy and HTTP timeout budgets. An intentional cancellation does not produce this cutoff warning. These transport observations do not establish that a task was delivered or authorize revoking its lease.
This backoff requires Python SDK 2.10.0 or later. Earlier packages use a fixed one-second poll-error wait.
Workflow, activity and query pollers back off independently after failed polls,
including HTTP errors, rejected credentials and unreachable runtimes. The wait
before the n-th consecutive retry is uniform(0, min(10, 0.25 * 2**(n - 1)))
seconds. That is the shared full-jitter schedule with a 10-second cap, so a
poller recovers quickly once the runtime is back, and workers do not retry in
lockstep. A bounded retry delay advertised by the Server is still honoured. Any
successful poll, including an empty response, resets the schedule. Shutdown
interrupts the wait.
The first failure is logged as a warning immediately. An unchanged failure is logged at most once per minute per poller. Once a poller has failed continuously for 60 seconds, the outage is logged at ERROR level, then again every 5 minutes while it lasts. A successful poll ends the outage, and the next failure starts again with a warning.
Authentication and authorization refusals are the exception to that limit: every poll answered with HTTP 401 (rejected authentication) or HTTP 403 (denied authorization) is logged at ERROR level, including refusals that come before a later recovery. Check the worker credentials and their permissions. The poller keeps retrying with the same backoff, so rotated credentials recover without a worker restart.
This differs from the PHP and Rust SDKs, which log an error and stop the worker on a poll-time 401 or 403. Python retries so that credentials can be rotated under a running worker. Registration at startup treats 401 and 403 as fatal in every SDK (see below). Poll-error metrics still count every failed poll, including failures whose warning is suppressed.
Starting with Python SDK 2.10.0, warnings name the exception class and static poll route even when an httpx exception has an empty message. Transport exception messages are omitted because they can contain private request details. Server refusals retain their status and bounded reason; authentication failures retain their credential-safe summary. See request outcomes and errors for local cancellation and transport retry behavior.
Typed capacity backpressure follows the Server's advertised retry delay. Storage-admission failures retain their separate identity-preserving retry and worker-stop behavior. Neither uses this generic poll-failure backoff.
Worker.run() and Worker.run_until() also ride out a runtime that is warming
up or briefly unavailable while they register. Cluster discovery and
registration are repeated whole after a transient failure:
- a refused, reset or timed-out connection
- an upstream 502, 503 or 504 without a protocol body
- a 503 whose reason is
storage_unavailableorbackend_unavailable(native Server answers every storage failure with this body) - any 5xx or 429 that says
retryable: true
Cluster discovery (GET /api/cluster/info) is a read with no side effects, so
any refused or failed connection, and any 5xx or 429 answer to it, counts as
transient unless the Server says retryable: false.
The wait before the n-th retry is uniform(0, min(4, 0.25 * 2**(n - 1)))
seconds. This is the backoff schedule shared by every Durable Workflow SDK.
Each retry logs a warning during the first minute. After 60 seconds the outage
is logged at ERROR level, then again every 5 minutes while it lasts, the same
escalation pollers use. Shutdown interrupts the wait.
After registration the worker uses the discovery it registered with. It does not read cluster info again, so a later cluster-info outage cannot stop it.
Some failures are never retried:
retryable: false- HTTP 401 or 403
- any other 4xx protocol refusal
- an incompatible Server
For these, the worker logs worker <id> stopping: registration failed with a non-retryable error: ... at error level, and run() raises the error.
An activity completion or failure report that hits one of the transient
failures listed above is sent again with the same backoff, as long as the
attempt's lease still covers the next wait. The lease is the renewed lease when
the Server supports activity lease renewal, and the claimed lease_expires_at
otherwise. A transient never turns a completion into a failure report.
When the lease cannot cover another wait, the worker stops resending and logs
activity task <id> ... outcome not delivered after N transient failures ...: its lease ran out at error level. The Server then settles the attempt when its
lease expires.
A transient can hide an outcome the Server already committed. Native Server
answers the resend with 409 and this body:
{"reason": "stale_attempt", "recorded": false, "outcome": "completed",
"lease_owner": "<this worker>", "activity_status": "completed",
"attempt_status": "completed", "task_status": "completed"}The body also carries the same task and attempt IDs. That exact shape counts
as this worker's committed completion, so the activity is not run again and
no second outcome is recorded. Any other stale_attempt means the attempt is
closed. It is reported as such and never treated as success. After
registration, pollers keep the poll-failure backoff described above. Worker
heartbeats log a failure and try again at the next interval.
Worker.run() and Worker.run_until() register through the worker protocol
before polling. After registration succeeds, shutdown stops every poller and
waits for accepted work within the shared shutdown_timeout deadline before
removing that worker-plane registration. Work still running at the deadline is
cancelled. Calling stop() more than once shares the same shutdown operation
and result. No deregistration is sent if registration failed.
Unreleased source behavior, prepared for SDK 2.10.1: the final HTTP call and any recovery use the
original remaining shutdown budget. If Server advertises
worker_deregistration_fencing, Worker retains its registration token and uses
the fenced receipt endpoint. Matching transient responses and transport losses
can retry that same generation. A completed receipt can replay safely even if
a replacement now owns the worker ID. Authentication, malformed or mismatched
receipts, lost authority and expired receipts fail without retrying. Without
advertised fencing, Worker keeps one attempt through the legacy worker-plane
DELETE and does not replay an uncertain result. That legacy route has no
generation fence. Keep worker IDs unique to a process, and do not reuse an ID
until the prior worker has fully stopped.
If drain consumes the deadline, no late deregistration request is sent. A final
HTTP timeout or exhausted recovery raises RuntimeError and leaves removal
unconfirmed. The Server might have committed a request whose response was lost;
an exception is not proof that the registration still exists. Inspect the
runtime before taking an operator recovery action. These changes are not in a
published package yet.
Cancelling an await cannot stop a running synchronous activity thread. Its
physical slot remains occupied until the callback returns, and a late return
does not submit a result after its await was cancelled. If the callback has
not stopped by the shared shutdown deadline, shutdown raises RuntimeError
and retains the registration. Python can keep the process alive while that
thread remains running. Configure callback I/O timeouts and a supervisor
termination policy; a timeout is not proof that an external side effect was
cancelled or that the worker stopped gracefully.
An external stop() also interrupts run_until(); the pending run_until()
call ends with asyncio.CancelledError after its accepted workflow or activity
work has drained or been cancelled. If an async poller, the query-poller thread,
or accepted work cannot be stopped, stop() raises RuntimeError and leaves
the registration active. Treat that as an incomplete shutdown requiring
operator attention rather than continuing as though the worker stopped cleanly.
For a long-running process, coordinate shutdown with the same worker instance
and await both stop() and the run task:
run_task = asyncio.create_task(worker.run())
try:
await shutdown_requested.wait()
finally:
try:
await worker.stop()
finally:
await run_taskDeregistration authentication, protocol, and HTTP failures are raised from
stop() and run(); a process must not treat those failures as a clean
shutdown. When a worker-loop error and deregistration both fail, the loop error
remains the raised exception and the deregistration error is available through
its __cause__.
The worker-plane cleanup operation is separate from
Client.deregister_worker(). That existing control-plane method remains an
operator management action for retiring or recovering worker records and is not
used by normal Worker shutdown.
run_until(timeout=...) includes Server discovery and registration in its
workflow wait budget. A timeout before startup finishes identifies the worker
and workflow in the error. A timeout after startup reports that the workflow
has not reached a terminal state. Neither timeout cancels the durable run.
A replacement worker can continue that same run.
When stop() is called during startup, it waits for registration to finish
before starting the shared drain deadline. Server can record a registration
before its reply reaches the SDK. A worker that never receives that reply has
not begun polling and becomes stale under the Server's heartbeat policy.
Use the existing workflow handle and queue when bringing up a replacement.
Worker.status() returns a read-only snapshot that separates readiness
(registered, every configured poller running, latest Server exchange of each
role succeeded) from liveness (event loop progressing, no poller exited, no
request outliving its bound). It performs no network I/O and can be called from
another thread. See Worker readiness and liveness
for states, probe configuration and shutdown guidance.
::: durable_workflow.worker
::: durable_workflow.worker_status