Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 30 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,26 @@

## Unreleased

- A failed cluster-info read no longer stops a running worker. Startup
discovery (query-task authority, Nexus cooperation) and the inline-payload
policy read before a report retry a connection failure, timeout, 5xx or 429
with the shared jittered backoff, uniform in `[0, min(4 s, 0.25 s *
2^(n-1))]`, logging a `tracing` ERROR once the outage has lasted 60 s and
every 5 minutes after, until it succeeds or the worker stops. An explicit
`retryable: false`, 401, 403 and other refusals are still returned.
`Client::cluster_info` and Clients outside a running worker make one attempt.
When the cached payload policy (refreshed every 60 s) expires during such an
outage, reports keep using the last-known policy and discovery is tried
again 10 s later; only a worker that has no policy yet waits.
- An activity report the Server refuses as too large (413 `payload_too_large`)
because it lowered its inline limit since the policy was cached is no longer
lost. The worker forgets the policy, rediscovers it, encodes the same report
again (externalizing payloads the new limit requires) and resends it once.
A result that is still too large (refused again, above the upload limit, or
with no payload storage) no longer stops the worker: the attempt is failed
as a non-retryable `ActivityResultTooLarge` (`failure.type` and
`failure.class`), as in the PHP SDK, and the worker keeps running.

- **BEHAVIOUR CHANGE: `Worker::recover_transient_outages` now defaults to
`true`.** `run` and `run_until` keep running through transient Server
outages instead of returning the error once `WorkerRetryPolicy.max_retries`
Expand Down Expand Up @@ -90,11 +110,16 @@
used for later requests, so overlapping requests across a rolling deploy each
restore their own capabilities. A digest string that matches neither the
request's digest nor a tagged copy in the same answer makes the Client
forget its held digest and resend that same request once with `none`; a
resent report the Server already committed comes back as its usual
non-recorded receipt (for an activity completion, `409 stale_attempt`), which
the worker settles as before. Only an unknown digest on the resend is
`Error::Protocol` with reason `server_capabilities_digest_unknown`.
forget its held digest. Only a task poll, idempotent by its poll request ID,
is resent once with `none`. Any other request (a completion, failure,
heartbeat, start or signal) is never resent: its answer is kept as received,
because a completion's answer can carry the next leased task, and the full
capabilities are read with `GET /api/cluster/info`
(`worker_protocol.server_capabilities`) and put in place of the unknown
digest. If that read fails, the answer is still used without the field and
later requests learn the capabilities again; the request never fails. A
poll whose resend names an unknown digest again is `Error::Protocol` with
reason `server_capabilities_digest_unknown`.
Decoded response bodies are capped at 128 MiB by default
(`ClientBuilder::max_response_bytes`); a larger body stops being read and
fails with the new `Error::ResponseBodyTooLarge`, so a small gzip answer
Expand Down
21 changes: 16 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,11 +308,13 @@ digest and object when it sends each request and restores that request's
answer before anything reads it, so readers always see the full object, even
when requests overlap a rolling deploy. Clones share what they learned. An
answer naming a digest that request did not send, and that the answer itself
does not carry, makes the Client forget its digest and resend that request once
asking for full capabilities. A resent report the Server already committed
gets its usual non-recorded receipt, which the worker settles. Only a second
unknown digest is `Error::Protocol` with reason
`server_capabilities_digest_unknown`. Older Servers ignore both headers and
does not carry, makes the Client forget its digest. A task poll is resent once
asking for full capabilities. Any other request is never resent, since it may
already have committed: its answer is kept as received and the capabilities
are read from `GET /api/cluster/info`. If that read fails, the answer is still
used without the field and later requests learn the capabilities again. A
poll whose resend names an unknown digest again is `Error::Protocol` with
reason `server_capabilities_digest_unknown`. Older Servers ignore both headers and
answer in full.

A decoded response body is capped at 128 MiB by default. Set
Expand Down Expand Up @@ -1169,6 +1171,15 @@ task, attempt and owner with every status `completed` (or `attempt_status`
else. When the lease runs out first, the worker logs a `tracing` error and the
Server settles the attempt.

A running worker's cluster-info reads (startup discovery and the payload
policy read before a report) retry a connection failure, timeout, 5xx or 429
the same way. Once a payload policy is known, an outage does not hold reports
back: the last-known policy is used and discovery is tried again 10 s later.
If the Server refuses a report as too large because it lowered its inline
limit, the worker rediscovers the policy, encodes the report again and resends
it once. A result that is still too large, or cannot be externalized, fails the
attempt as a non-retryable `ActivityResultTooLarge`; the worker keeps running.

An explicit `retryable: false`, 401, 403 and other protocol refusals, codec
and handler failures stop the worker: it emits a `tracing` error event and
returns the error. `.recover_transient_outages(false)` restores bounded retries,
Expand Down
33 changes: 15 additions & 18 deletions src/cooperative_cancellation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1892,30 +1892,27 @@ impl Worker {
}
return Ok(ManagedPollOutcome::Handled);
}
let task_outcome = if result.is_ok() {
let mut task_outcome = if result.is_ok() {
"completed"
} else {
"failed"
};
let settled = match result {
Ok(value) => {
self.settle_activity_report(
crate::ActivityReport::Completed,
&guard.task_id,
&guard.attempt_id,
&guard.owner,
lease_deadline,
|| {
self.client.complete_activity_task(
&guard.task_id,
&guard.attempt_id,
&guard.owner,
value.clone(),
&task.payload_codec,
)
},
)
.await?
let (settlement, too_large) = self
.settle_activity_completion(
&guard.task_id,
&guard.attempt_id,
&guard.owner,
lease_deadline,
&value,
&task.payload_codec,
)
.await?;
if too_large {
task_outcome = "failed";
}
settlement
}
Err(error) if worker_storage_admission_body(&error).is_some() => return Err(error),
Err(error) => {
Expand Down
Loading
Loading