Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,30 @@ project follows [Semantic Versioning](https://semver.org/).

### Fixed

- A managed Worker no longer exits on a transient Server or network outage
(#448). For example, native Server answers `503 storage_unavailable` while
its database warms up. Polls retry untyped transients with the poll's
identity kept, and `Worker::run()` absorbs a transient failure that escapes a
tick (a completion, for example), using the shared jittered schedule (0.25 s
doubling to a 4 s cap). Authentication, authorization, protocol refusals and
explicit `retryable: false` still stop the Worker, after a
`worker.fatal_failure` diagnostic.
- A Worker started while the Server is down or warming up waits for it.
Registration retries refused or reset connections, empty replies, bodiless
502/503/504 answers from a load balancer, and `503 storage_unavailable` /
`backend_unavailable`. Registration is idempotent: Server 3.17.0 accepts the
same worker ID registering again. An outage lasting 60 s is reported once as
a `worker.outage_persisting` error, then every 5 minutes until a request
succeeds.
- A completion that meets an untyped outage is retried with the same attempt
fence, inside the claim's lease, instead of being reported as an activity
failure. A duplicate of a completion the Server already committed (Server
3.17.0: `409 stale_attempt`, `recorded: false`, attempt `completed`) counts as
committed. A result that cannot be published in time is logged
(`worker.outcome_unpublished`) and left to the Server's lease handling.

### Fixed

- An activity worker no longer drops a claimed task when the lease renewal at
activity start is slow or fails transiently (#445). The worker keeps working
under the poll's original lease, charged conservatively from the poll's own
Expand Down
8 changes: 8 additions & 0 deletions docs/portal/build/workers.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,14 @@ An activity option such as `['queue' => 'payments-v2']` selects the queue of a n

`Worker::run()` registers, negotiates heartbeat cadence, polls workflow/activity/query tasks, renews workflow-task leases, and reports terminal outcomes. Typed transient poll pressure is retried with capped backoff while registration heartbeats and shutdown checks remain responsive.

An untyped outage does not stop the Worker either. That covers a connection that is refused, reset or timed out; a 502/503/504 with no protocol body; a 503 `storage_unavailable` or `backend_unavailable`, for example while the Server's database warms up; and any 5xx or 429 with `retryable: true`. The Worker retries using the schedule all SDKs share: 0.25 s doubling to a 4 s cap, with full random jitter. This applies to polls, and to any request that fails a tick, such as a completion; the Server re-delivers a claimed task when its lease ends. Authentication and authorization failures, other protocol refusals, and explicit `retryable: false` stay fatal. They are reported as a `worker.fatal_failure` diagnostic before `run()` rethrows.

**Startup and long outages.** A Worker started while the Server or its load balancer is unavailable keeps retrying registration on the same schedule instead of exiting. Registration is safe to repeat: the Server accepts the same worker ID registering again. An outage that lasts 60 s is reported as a `worker.outage_persisting` error, then every 5 minutes until a request succeeds.

**Completions.** A completion that meets an outage is retried with the same attempt fence on that schedule. With activity lease maintenance, the retries stop when the claim's grant ends; without it, they stop after a bounded number of attempts. Retrying is safe: the Server records one outcome per attempt. A retry of a completion the Server already committed is answered `409 stale_attempt` with `recorded: false` and the attempt `completed`, and the Worker treats that as its own committed completion. A result that cannot be published before the retries end is logged as `worker.outcome_unpublished` and left to the Server, which re-delivers the attempt when its lease ends. It is never reported as an activity failure.

**Across SDKs.** PHP and Rust Workers stop on a 401 or 403 after logging it as an error. Python Workers keep retrying polls through a 401 or 403 so that credentials can be rotated without a restart, and log every refusal at error level.

### Activity continuations

Activity continuations are an explicit constructor opt-in:
Expand Down
Loading