Skip to content

Keep a managed Worker running through a transient Server or network outage (#448) - #449

Merged
rmcdaniel merged 5 commits into
mainfrom
general/transient-poll-retry
Nov 20, 2026
Merged

rmcdaniel merged 5 commits into
mainfrom
general/transient-poll-retry

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Fixes #448.

Problem

A managed Worker process died when native Server answered a workflow poll with 503 {"reason":"storage_unavailable","message":"storage_unavailable"} while its database warmed up. Two gaps combined:

  • Worker::pollFailurePolicy() retried a 503 only when it was a typed poll refusal or a typed backend/storage-admission refusal.
  • Worker::run() rethrew everything except a stale-registration recovery.

Evidence is in #448: the lab run with Server main 43ad8d8f and SDK 3.22.0, where worker-3.log ends with an uncaught TransportException.

Change

  • Worker\TransientFailure classifies untyped transients:

    • connection refused, reset or timed out;
    • an upstream 502, 503 or 504 with no protocol body;
    • 503 storage_unavailable or backend_unavailable;
    • any 5xx or 429 whose body says retryable: true.

    It never matches retryable: false, 401 or 403, or other 4xx protocol refusals.

  • Polls retry these and keep the poll identity, since an outage committed nothing. The delay follows the schedule all SDKs share: 0.25 s doubling to a 4 s cap, with full random jitter, and never shorter than a retry_after_seconds the Server sends. Typed refusals keep their existing protocol backoff and identity rules; a test guards the lock-pressure identity rule.

  • Worker::run() absorbs a transient failure that escapes a tick (a completion, for example) on the same schedule. It logs worker.transient_failure, keeps heartbeating while it waits, and continues. The Server re-delivers any claimed task when its lease ends. Fatal failures log worker.fatal_failure with status and reason before run() rethrows.

  • Docs and changelog: docs/portal/build/workers.md and CHANGELOG.md describe the behaviour.

  • Completions that meet an untyped outage are retried with the same attempt fence on the shared schedule.

    • Retries end at the lease's grant when activity lease maintenance is running, and after 8 attempts otherwise (worker.outcome_retry_exhausted).
    • A computed result that can't be published is logged as worker.outcome_unpublished and left to the Server's lease handling. It is never sent as an activity failure report.
  • Unchanged: an unclassified retryable: true answer still never authorizes repeating an acknowledgement (the existing contract tests in WorkerStorageAdmissionTest and WorkerWorkflowCompletionLockPressureTest). Typed refusals keep their own protocol checks, and Server-suggested delays for untyped outages are capped at 60 s.

Review fixes (live fault-proxy findings)

  1. Registration retries every outage on the shared jittered schedule: a refused or reset connection, an empty reply, a bodiless 502/503/504 from a load balancer, and 503 storage_unavailable / backend_unavailable. The cluster-info reads registration makes in query-authority and cooperative modes do the same, so a Worker started while the Server is down or warming up waits for it.
    • Why repeating is safe: registration is idempotent. Against durableworkflow/server:3.17.0, registering the same worker ID twice returns registered: true both times.
    • The two existing tests that forbade the repeat (WorkerConnectionRecoveryTest::testAmbiguousRegistrationIsNotRepeated and WorkerUpstreamRecoveryTest::testRegistrationDoesNotBlindlyRepeatAnUncertainRequest) are replaced. The new tests assert the repeat keeps the original identity.
    • Outage alarm: an outage lasting 60 s is reported once as a worker.outage_persisting error, then every 5 minutes until a request succeeds.
  2. Heartbeat in a retry wait. The heartbeat defers an untyped outage on the shared schedule instead of rethrowing it, so no network call in the wait path can end run().
  3. Committed completion on the first answer. If cURL resends a completion whose reply was lost, the first answer the Worker sees can already be the Server's duplicate: 409 stale_attempt, recorded: false, our owner and attempt, every status completed. That exact shape is accepted as our committed completion on the first answer (worker.completion_already_committed).

All three are reproduced by fake-transport tests that fail on the previous head.

Live check, Worker started with the Server down: the Server container was stopped for about 15 s at Worker start. The Worker retried 13 times, registered once the Server was back, and completed a workflow started afterwards. There was no Worker error and no fatal diagnostic.

Live check against durableworkflow/server:3.17.0, with faults injected by a Guzzle middleware:

  • 3 registration 503s, then a 12 s outage (6 poll and 7 heartbeat 503s);
  • no Worker error and no fatal diagnostic;
  • one workflow started before the outage and one after it, both completed.

Idempotency against published Server 3.17.0

This is a real Worker with a Guzzle middleware that forwards the first activity completion to durableworkflow/server:3.17.0. The Server commits it, and the middleware then shows the Worker 503 storage_unavailable.

first completion 200 recorded: true (committed, the reply hidden from the Worker)
retried completion 409 {"outcome":"completed","recorded":false,"reason":"stale_attempt","activity_status":"completed","attempt_status":"completed","task_status":"completed",…}
Worker treats it as its own committed completion (isCommittedActivityCompletion)
result workflow completed, one ActivityCompleted, no ActivityFailed, activity ran once

Python (#355) and Rust (#305) observed the same answer and apply the same rule.

Across SDKs

PHP and Rust stop on a 401 or 403 after logging an error. Python keeps retrying polls through a 401 or 403 so credentials can rotate, and logs every refusal at error level. This is documented in each SDK.

Tests (tests/WorkerTransientOutageTest.php, using a fake transport)

test on main with this PR
three 503 storage_unavailable polls, then recovery the Worker throws and exits it retries with waits of at most 0.25, 0.5 and 1 s, then completes the activity
a completion and its failure report both meet the outage the Worker throws and exits worker.transient_failure is logged and the Worker keeps polling
401, 403, 400, or 503 with retryable: false fatal still fatal, with exactly one worker.fatal_failure diagnostic
completion meets a 503, then the retry is recorded or answered as a duplicate the result becomes a failure report retried with the same fence; no failure report
completion meets the outage until retries run out the result becomes a failure report worker.outcome_unpublished; no failure report
classification table and jitter bounds n/a passes

Results:

  • Full suite (PHP 8.4 with pcntl): 9216 tests, 0 failures.
  • The poll, transient, outage and pipelining subsets all pass (155 tests).
  • phpstan: no errors.

🤖 Generated with Claude Code

…utage

Fixes #448. A Worker process died when native Server answered a workflow
poll with 503 {"reason":"storage_unavailable"} while its database warmed up:
pollFailurePolicy() retried only typed refusals, and Worker::run() rethrew
everything except a stale registration.

- Worker\TransientFailure classifies untyped transients: a connection that is
  refused, reset or timed out; an upstream 502/503/504 with no body; a 503
  storage_unavailable or backend_unavailable; and any 5xx or 429 with
  retryable: true. Never retryable: false, auth failures or other 4xx.
- Polls retry them and keep the poll identity (an outage committed nothing),
  using the shared jittered schedule: 0.25 s doubling to a 4 s cap, full
  jitter. Typed refusals keep their protocol backoff and identity rules.
- Worker::run() absorbs a transient failure that escapes a tick (a completion,
  for example) with the same schedule, logging worker.transient_failure, and
  keeps heartbeating while it waits. Fatal failures log worker.fatal_failure
  with status and reason before run() rethrows.
- Tests use a fake transport: 503s then recovery leaves the Worker running; a
  failure escaping a tick is absorbed; 401, 403, 400 and retryable: false stop
  the Worker with one fatal diagnostic; classification and jitter bounds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel added bug Something isn't working agent:general labels Nov 20, 2026
rmcdaniel and others added 3 commits November 20, 2026 07:08
Coordinator decisions on #448:
- A completion that meets an untyped outage (503 storage_unavailable or
  backend_unavailable) is retried with the same attempt fence on the shared
  jittered schedule, like the existing connection-failure retry. With
  activity lease maintenance the publication fence ends retries at the
  grant's end; without it, eight attempts do (worker.outcome_retry_exhausted).
- Idempotent: verified against published Server 3.17.0. The first completion
  committed, the Worker saw a 503, and the retry was answered 409
  stale_attempt, recorded false, attempt completed. The existing
  isCommittedActivityCompletion() rule now applies to it, so the run
  completed with one ActivityCompleted and the activity ran once.
- A computed result whose publication an outage defeats is logged
  (worker.outcome_unpublished) and left to the Server's lease handling,
  never sent as an activity failure report.
- An unclassified retryable answer still never authorizes repeating an
  acknowledgement, and typed refusals keep their own protocol checks, so an
  invalid typed retry_after cannot stall the Worker. Server-suggested delays
  for untyped outages are capped at 60 s.
- Docs: completion retry, and the cross-SDK difference (PHP and Rust stop on
  401/403 after an error; Python retries polls so credentials can rotate).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…answers

Review HIGHs on #449, found live against Server 3.17.0 through a fault proxy:
1. Registration (the original #448 incident) retried only typed refusals,
   so 503 storage_unavailable during warm-up still ended the Worker. It now
   uses the shared transient classification and jittered backoff, and so do
   the cluster-info reads registration makes in query-authority and
   cooperative modes.
2. The worker heartbeat inside a retry wait rethrew an untyped outage, so a
   12 s outage after registration ended run(). The heartbeat now defers it on
   the shared schedule like its other transient failures, so no network call
   in the wait path can end the run.
3. cURL can resend a completion whose reply was lost on a reused connection,
   so the first answer the Worker sees may already be the Server's duplicate:
   409 stale_attempt, recorded false, our owner and attempt, every status
   completed. That exact shape is accepted as our committed completion on the
   first answer (worker.completion_already_committed), not only after a seen
   failure.

Tests reproduce all three with a fake transport and fail on the previous
head. A live check against durableworkflow/server:3.17.0 with injected
faults (three registration 503s, then a 12 s outage of polls and heartbeats)
shows no Worker error, and workflows started before and after both complete.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The registration retry also retried connection failures, so the existing
contract that an ambiguous registration (the Server may have registered) is
not blindly repeated looped until the CI timeout. Registration now retries
only the Server's definite outage answer, 503 storage_unavailable or
backend_unavailable (the #448 warm-up incident), on the shared jittered
schedule. Full suite: 9216 tests, 0 failures.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel added a commit that referenced this pull request Nov 20, 2026
Digest re-review:
- BLOCKER: a garbled 200 to an activity completion triggered the no-digest
  resend, the Server answered 409 stale_attempt for the completion it had
  already committed, and Worker::run exited. The committed-completion rule
  (409 stale_attempt, recorded false, our owner and attempt, every status
  completed) now applies to any answer: the first, after a transient
  failure, or after a digest resend. This block is identical to the one in
  #449, so whichever merges second merges cleanly. Workflow-task duplicates
  need nothing: Server 3.17.0 answers them 200 recorded:false
  already_completed, which the Worker already accepts.
- End-to-end test: garbled completion answer, then the resend's 409; the
  Worker keeps running and the activity runs once. Verified live against
  durableworkflow/server:3.17.0 as well.
- The default decode limit is min(32 MiB, memory_limit / 4), safe under
  PHP's default 128 MB, and ResponseTooLarge reaches callers unwrapped.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel

Copy link
Copy Markdown
Member Author

The committed-completion block in retryStorageAdmission() (the first-answer rule) is now byte-for-byte the same as the one in #446, so the two PRs merge in either order without conflicting copies.

Re-check follow-up on #449: a Worker started during an outage still exited
on a refused connection, an empty reply or a bodiless 502/503/504 at
registration.

- Registration retries every outage on the shared jittered schedule:
  refused or reset connections, empty replies, bodiless 502/503/504 from a
  load balancer, and 503 storage_unavailable / backend_unavailable.
  Registration is idempotent: against durableworkflow/server:3.17.0,
  registering the same worker ID twice answers registered: true both times.
  The two tests that forbade the repeat are replaced by ones asserting it
  keeps the original identity.
- An outage lasting 60 s is reported once as a worker.outage_persisting
  error, then every 5 minutes, until a request succeeds (registration, poll
  retries and the run loop).
- Tests: server down at startup (refused connection, empty reply), a load
  balancer answering bare 502 and 504, a bodiless 503 behind a load balancer,
  and the 60 s / 5 min escalation on a simulated clock. Live: a Worker started
  with the Server stopped for about 15 s registered once it returned and
  completed a workflow. Full suite: 9221 tests, 0 failures.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit f17b035 into main Nov 20, 2026
61 of 63 checks passed
rmcdaniel added a commit that referenced this pull request Nov 20, 2026
Digest re-review:
- BLOCKER: a garbled 200 to an activity completion triggered the no-digest
  resend, the Server answered 409 stale_attempt for the completion it had
  already committed, and Worker::run exited. The committed-completion rule
  (409 stale_attempt, recorded false, our owner and attempt, every status
  completed) now applies to any answer: the first, after a transient
  failure, or after a digest resend. This block is identical to the one in
  #449, so whichever merges second merges cleanly. Workflow-task duplicates
  need nothing: Server 3.17.0 answers them 200 recorded:false
  already_completed, which the Worker already accepts.
- End-to-end test: garbled completion answer, then the resend's 409; the
  Worker keeps running and the activity runs once. Verified live against
  durableworkflow/server:3.17.0 as well.
- The default decode limit is min(32 MiB, memory_limit / 4), safe under
  PHP's default 128 MB, and ResponseTooLarge reaches callers unwrapped.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Managed Worker process dies on a transient 503 storage_unavailable (Server warm-up) instead of retrying

1 participant