Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,25 @@ project follows [Semantic Versioning](https://semver.org/).

### Fixed

- An outage of `GET /api/cluster/info` no longer stops a managed Worker.
Previously a plain 500 from it, read before a workflow-task completion for
the payload policy, ended `Worker::run()`. Cluster info is read-only, so a
refused, reset or timed-out connection and any 5xx or 429 (unless
`retryable: false`) are retried with the shared jittered schedule and
reported as `worker.outage_persisting` when they last. Startup discovery
waits like registration. The transport keeps the last discovered payload
policy through an outage and tries discovery again later. Without one, the
request that needed it is not sent and is retried. Authentication and
authorization refusals still stop the Worker.
- A completion refused for its size (`413 payload_too_large`) no longer stops
the Worker. The Server's inline limit can be lowered while the Worker holds
an older payload policy. The Worker drops that policy, reads it again,
encodes the same outcome under it (externalising what the new limit
requires) and sends it once more, without recomputing the result. A
discovery outage then is retried under the outage rules. An activity result
that is still too large is reported as a non-retryable failure whose
`failure.type` and `failure.class` are `ActivityResultTooLarge`, as in the
other SDKs.
- A managed Worker no longer exits on a transient Server or network outage
(#448). For example, native Server answers `503 storage_unavailable` while
its database warms up. Polls retry untyped transients with the poll's
Expand Down
4 changes: 4 additions & 0 deletions docs/portal/build/workers.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,10 @@ An untyped outage does not stop the Worker either. That covers a connection that

**Startup and long outages.** A Worker started while the Server or its load balancer is unavailable keeps retrying registration on the same schedule instead of exiting. Registration is safe to repeat: the Server accepts the same worker ID registering again. An outage that lasts 60 s is reported as a `worker.outage_persisting` error, then every 5 minutes until a request succeeds.

**Cluster info.** Reads of `GET /api/cluster/info` are read-only, so any outage of them is retried: a refused, reset or timed-out connection, or any 5xx or 429 unless it says `retryable: false`. At startup, a Worker that needs discovery waits for it as it waits for registration. While running, the transport keeps the last payload policy it discovered and tries discovery again later. If it has none yet, the request that needed it is not sent and is retried. A lasting outage is reported as `worker.outage_persisting`. Authentication and authorization refusals stay fatal.

**A lowered payload limit.** If the Server refuses a completion for its size (`413 payload_too_large`), for example because its inline limit was lowered after the Worker read it, the Worker reads the payload policy again, encodes the same outcome under it (externalising large values when storage is configured) and sends it once more. The result is never recomputed. An activity result that is still too large fails the attempt as a non-retryable `ActivityResultTooLarge`; the Worker keeps running.

**Completions.** A completion that meets an outage is retried with the same attempt fence on that schedule. With activity lease maintenance, the retries stop when the claim's grant ends; without it, they stop after a bounded number of attempts. Retrying is safe: the Server records one outcome per attempt. A retry of a completion the Server already committed is answered `409 stale_attempt` with `recorded: false` and the attempt `completed`, and the Worker treats that as its own committed completion. A result that cannot be published before the retries end is logged as `worker.outcome_unpublished` and left to the Server, which re-delivers the attempt when its lease ends. It is never reported as an activity failure.

**Across SDKs.** PHP and Rust Workers stop on a 401 or 403 after logging it as an error. Python Workers keep retrying polls through a 401 or 403 so that credentials can be rotated without a restart, and log every refusal at error level.
Expand Down
89 changes: 72 additions & 17 deletions src/Client.php
Original file line number Diff line number Diff line change
Expand Up @@ -3472,24 +3472,37 @@ private function performRequest(
$poll = str_ends_with($path, '/poll') && is_int($body['timeout_seconds'] ?? null);
$pollSeconds = $poll ? max(0, min(60, $body['timeout_seconds'])) : 0;
$budget ??= $worker && $this->boundedWorkerRequests ? new RequestBudget($pollSeconds + 5) : null;
if ($body !== null) {
$body = $this->failureReports->request($path, $body, false);
$body = $this->payloadUploads->request($body, $method, $path, $worker, $headers, $budget,
$this->failureReports);
$body = $this->failureReports->request($path, $body);
}
$activityLease = $worker ? $this->activityLeaseBinding->current() : null;
if ($activityLease !== null && $activityLease->governs($path)) {
// Uploading an external result is not publication authority.
// Check again before its final Worker POST and bound the send
// by the latest acknowledged grant, not an old cached lease.
$activityLease->checkRequest($body);
$budget?->restrictAuthorityDeadline($activityLease->publicationDeadline());
}
$sourceBody = $body;
$body = $this->encodeRequestBody($sourceBody, $method, $path, $worker, $headers, $budget);
try {
$response = $this->sendTransportRequest($method, $path, $worker, $headers, $body, $budget);
} finally {
$observation?->captureStatus($this->transport);
try {
$response = $this->sendTransportRequest($method, $path, $worker, $headers, $body, $budget);
} finally {
$observation?->captureStatus($this->transport);
}
} catch (TransportException $refusal) {
if (!$worker || $method !== 'POST' || $sourceBody === null || !self::isCompletion($path)
|| !self::isPayloadSizeRefusal($refusal)) {
throw $refusal;
}
// The Server refused the body's size before reading it, so
// nothing was recorded. The payload policy this transport held
// may be older than the Server's limit: read it again and
// encode the same outcome under it (externalising what the new
// limit requires). Send once more only if that changes the
// body; otherwise the refusal stands. A discovery outage here
// fails as not sent, and the Worker repeats the request.
$this->payloadUploads->forgetPolicies();
$reencoded = $this->encodeRequestBody($sourceBody, $method, $path, $worker, $headers, $budget);
if ($reencoded === $body) {
throw $refusal;
}
$body = $reencoded;
try {
$response = $this->sendTransportRequest($method, $path, $worker, $headers, $body, $budget);
} finally {
$observation?->captureStatus($this->transport);
}
}

$budget?->remainingSeconds();
Expand Down Expand Up @@ -3532,6 +3545,48 @@ private function performRequest(
}
}

/**
* Apply failure-report limits and the payload policy to a request body,
* then check an activity outcome against its current lease.
*
* @param array<string, mixed>|null $body
* @param array<string, string> $headers
* @return array<string, mixed>|null
*/
private function encodeRequestBody(?array $body, string $method, string $path, bool $worker, array $headers, ?RequestBudget $budget): ?array
{
if ($body !== null) {
$body = $this->failureReports->request($path, $body, false);
$body = $this->payloadUploads->request($body, $method, $path, $worker, $headers, $budget,
$this->failureReports);
$body = $this->failureReports->request($path, $body);
}
$activityLease = $worker ? $this->activityLeaseBinding->current() : null;
if ($activityLease !== null && $activityLease->governs($path)) {
// Uploading an external result is not publication authority.
// Check again before its final Worker POST and bound the send
// by the latest acknowledged grant, not an old cached lease.
$activityLease->checkRequest($body);
$budget?->restrictAuthorityDeadline($activityLease->publicationDeadline());
}

return $body;
}

/** Activity and workflow-task completions, whose outcome is already computed. */
private static function isCompletion(string $path): bool
{
return preg_match('~\A/worker/(?:workflow-tasks/[^/]+/complete|activity-tasks/[^/]+/(?:continuation/)?complete)\z~', $path) === 1;
}

/** The Server's structural size refusal (413 payload_too_large), answered before the body is read. */
private static function isPayloadSizeRefusal(TransportException $refusal): bool
{
$details = $refusal->response;

return $refusal->status === 413 && is_array($details) && ($details['reason'] ?? null) === 'payload_too_large';
}

/**
* Server 3.6.0 bounds these requests by its workflow-task completion limit, and
* records each in one history transaction bounded by its history budget.
Expand Down
28 changes: 28 additions & 0 deletions src/Exception/ActivityResultTooLarge.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
<?php

declare(strict_types=1);

namespace DurableWorkflow\Exception;

/**
* An activity's result exceeds what the Server accepts, even after the Worker
* read the Server's current payload policy again. The attempt is reported as a
* non-retryable failure: running it again would compute the same result.
*/
final class ActivityResultTooLarge extends DurableWorkflowException
{
/** `failure.type` and `failure.class` on the wire, the same in every SDK. */
public const FAILURE_TYPE = 'ActivityResultTooLarge';

public function __construct(ServerException $refusal)
{
parent::__construct('The activity result exceeds the Server\'s payload limit: '.$refusal->getMessage(), 0, $refusal);
}

/** A size refusal of an activity result: the Server's 413, or an encoding the policy cannot carry. */
public static function matches(\Throwable $error): bool
{
return $error instanceof ServerException && !$error instanceof WorkflowTaskCompletionTooLarge
&& $error->status === 413 && in_array($error->reason, ['payload_too_large', 'external_payload_oversized'], true);
}
}
13 changes: 13 additions & 0 deletions src/Exception/ServerException.php
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,19 @@ public function isActivityTaskBackendUnavailable(string $taskId, string $attempt
&& $response['retry_after_seconds'] > 0;
}

/** Runtime discovery (cluster info) failed before this request was sent, so it may be repeated. */
public function isPayloadDiscoveryUnavailable(): bool
{
$details = $this->details;

return $this instanceof ExternalPayloadException
&& $this->status === 503
&& $this->reason === 'payload_discovery_unavailable'
&& $details !== null && !array_is_list($details)
&& ($details['request_admitted'] ?? null) === false
&& ($details['retryable'] ?? null) === true;
}

/** Discovery failed before any activity-completion request was submitted. */
public function isActivityCompletionPayloadDiscoveryUnavailable(string $taskId, string $attemptId, string $leaseOwner): bool
{
Expand Down
28 changes: 28 additions & 0 deletions src/Transport/RuntimePayloadUploads.php
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
use DurableWorkflow\Exception\TransportException;
use DurableWorkflow\Version;
use DurableWorkflow\Failure\FailureReportLimiter;
use DurableWorkflow\Worker\TransientFailure;

/** Namespace-scoped discovery and request-local, content-addressed uploads.
* @phpstan-type PayloadSlot array{path: list<int|string>, blob: string}
Expand All @@ -16,6 +17,9 @@
*/
final class RuntimePayloadUploads
{
/** How long a policy kept through a discovery outage is used before discovery is tried again. */
private const STALE_POLICY_RETRY_SECONDS = 10;

private const COMPLETION_SCHEMA = 'durable-workflow.v2.payload-completion-context.v1';
private const PREPARED_COMPLETION_SCHEMA = 'durable-workflow.v2.payload-completion-context.v2';
private const COMPLETION_HEADER = 'X-Durable-Workflow-Payload-Completion';
Expand Down Expand Up @@ -143,15 +147,30 @@
return $results;
}

/** @internal Forget discovered policies; the next request reads cluster info again. */
public function forgetPolicies(): void
{
$this->policies = [];
}

/** @param array<string, mixed> $body
* @param array<string, string> $headers
* @return UploadPolicy
*/
private function requestPolicy(array $body, string $path, bool $worker, array $headers, ?RequestBudget $budget): array

Check failure on line 160 in src/Transport/RuntimePayloadUploads.php

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this function to reduce its Cognitive Complexity from 21 to the 15 allowed.

See more on https://sonarcloud.io/project/issues?id=durable-workflow_sdk-php&issues=AaH4fUfrDQGyqZpw2nWG&open=AaH4fUfrDQGyqZpw2nWG&pullRequest=451
{
try {
return $this->policy($headers, $worker, $budget);
} catch (TransportException $exception) {
$key = (int) $worker;
if (isset($this->policies[$key]) && TransientFailure::isReadOutage($exception)) {
// Discovery is read-only and the Server validates every request,
// so an outage of it keeps the last policy this transport read.
// The read is tried again later, not before every request.
$this->policies[$key]['expires'] = time() + self::STALE_POLICY_RETRY_SECONDS;

return $this->policies[$key]['policy'];
}
$discovery = $exception->response;
if ($worker
&& preg_match('~\A/worker/activity-tasks/([^/]+)/complete\z~', $path, $matches) === 1
Expand Down Expand Up @@ -184,6 +203,15 @@
$exception,
);
}
if ($worker && TransientFailure::isReadOutage($exception)) {
// Nothing has been sent: the Worker may repeat this request
// once discovery answers again.
$retryAfter = is_array($discovery) ? ($discovery['retry_after_seconds'] ?? null) : null;
throw new ExternalPayloadException('Runtime payload policy discovery is temporarily unavailable.', 503,
'payload_discovery_unavailable', ['reason' => 'payload_discovery_unavailable', 'operation' => 'cluster_info',
'request_admitted' => false, 'retryable' => true,
'retry_after_seconds' => is_int($retryAfter) && $retryAfter > 0 ? $retryAfter : 1], $exception);
}

throw $exception;
}
Expand Down
Loading