Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Default off; enable it only after every worker on the task queue runs this release or later.** Published 2.20.0 and earlier cannot replay a marked run. The Server marks the workflow task replay-blocked with no retry, and the run waits until it is repaired, which can hand it to an old worker again. With recording off, new runs send exactly the commands 2.20.0 sends. Turning recording off again is safe. Rolling back below this release requires marked runs to be drained first. The next minor release turns recording on by default (#351).
- **Replay safety:** every worker of this release follows a marker a run already recorded, whether or not recording is on, for every workflow task, query and update. A marked run needs no per-group decision, because the first failure a task sees in append-only history is the first every later replay sees. A run whose first such group was recorded without the marker keeps the earlier rule for its whole life. Its settled groups raise the lowest-position failure, and a failure raised while siblings are open still records the #338 decision. This holds even when a task recorded no command after catching the failure, or another worker answered an update in between. A run with a cancellation request when it reaches its first group records no marker.
- **Regression evidence:** fixtures recorded by published 2.19.1 on a Server built from main replay under the earlier rule. They cover a settled group in flight, a catch followed by a task that recorded no command, an update answered by 2.19.1 before or between later tasks, and a select() cancellation. Marked runs recorded on native and PHP Server replay under the new rule without the setting. New fixtures cover the first task with recording on and off. Existing fixtures, including `parallel-group-metadata-emission`, the completion-limit fixtures and `business-clock-parallel-failure-selection-gap`, are unchanged.
- `ActivityFailed.lease_loss_reason()` tells workflow code whether an activity attempt failed because its worker lost the claim. It returns `"worker_lost"` when a worker released a claim it could not maintain (failure type `worker_lost`, see Fixed). For a lease the Server settled (`Workflow\V2\Exceptions\ActivityLeaseExpiredException`), it returns the recorded `failure_reason`, `"lease_expired"` or `"worker_deregistered"`, defaulting to `"lease_expired"`. It returns `None` for any other failure. `ActivityFailed.failure_reason` now carries that recorded field. The default activity retry policy is unchanged.

### Changed

- `ActivityLeaseAuthorityLost` is renamed `ActivityWorkerLost`. The old name stays importable from `durable_workflow.errors` as a deprecated alias of the same class, so existing `except ActivityLeaseAuthorityLost` handlers keep working. `durable_workflow.errors.WORKER_LOST_FAILURE_TYPE` (`"worker_lost"`) and `ACTIVITY_LEASE_EXPIRED_EXCEPTION_CLASS` name the two recorded forms of a lost claim.

### Fixed

- An activity worker no longer silently drops a claimed task when lease renewal fails transiently (#352, same bug as durable-workflow/sdk-php#445). Before, if the `renew-lease` confirmation sent right after an activity poll timed out, hit a connection reset or got a 5xx/429, the poll raised `ActivityLeaseAuthorityLost("activity_initial_lease_unavailable")` and the worker discarded the task. The Server then held the lease (300 s by default) until it expired and, with the default single attempt, failed the workflow.
- **Keeps working:** the activity now runs under the lease it already holds, and renewal is retried inside that lease. The backoff is the same in every SDK: the *n*-th retry waits `uniform(0, min(4, 0.25 * 2**n))` seconds, so 0.25 s doubling to a 4 s cap, with full jitter. Each renewal exchange is bounded to leave a reserve of the lease (at most 10 s, at most a fifth of the grant). An unconfirmed lease is charged from the poll's start, because the Server stamps it only after receiving the poll.
- **Stops before it releases:** if renewal still has not succeeded when only the reserve is left, the worker stops the execution. An async activity is cancelled. A thread activity is asked to stop: its next `heartbeat_sync()` raises `ActivityWorkerLost`. A callback process is stopped. The worker waits for that stop for up to half of the reserve.
- **Releases through the Server:** only once every execution of the claim is confirmed stopped does the worker release the claim, following the cross-SDK contract shared with PHP and Rust. It sends `POST /worker/activity-tasks/{id}/fail` with `failure.type = "worker_lost"` (recorded as `exception_type`, and matchable in `non_retryable_error_types`), `failure.class = "ActivityWorkerLost"`, `non_retryable = false`, the local cause `activity_lease_renewal_unavailable` in the message, and `details = {"reason": "worker_lost", "cause": "activity_lease_renewal_unavailable"}`. The retry policy then applies right away.
- **Waits for expiry if it cannot stop:** if the execution does not stop in time, for example a thread that never heartbeats, nothing is sent. The Server settles the attempt at lease expiry (`lease_expired`), so two executions of one attempt never overlap. A claim that never started is released directly. A typed Server refusal (for example `lease_expired` or `run_closed`) still ends the claim as before. See "Activity lease renewal" in the SDK reference.
- **Replay safety:** worker-side lease handling only. No workflow commands or recorded history change.

## [2.20.0] - 2026-11-19

Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,8 @@ after cancellation. The disposal guarantees require SDK 2.12.2 or later.

Both `await activity.context().heartbeat()` and synchronous
`activity.context().heartbeat_sync()` stop at an explicit Server refusal.
They raise `ActivityLeaseAuthorityLost` with its original `.reason` for an
They raise `ActivityWorkerLost` (formerly `ActivityLeaseAuthorityLost`, still
importable under that name) with its original `.reason` for an
elapsed deadline, closed attempt or closed run. Cancellation continues to raise
`ActivityCancelled`. The worker submits no result or failure for an attempt
whose heartbeat refused continuation, even if application code catches that
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/activity.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ The helper records the original exception, sets the cancellation event, and
stops heartbeating when cancellation, lease-authority loss or a transport error
occurs. After the work unwinds, it joins the helper and re-raises that exception
before a result can be returned. In particular, do not swallow `ActivityCancelled`
or `ActivityLeaseAuthorityLost`, replace them with an application error, or
or `ActivityWorkerLost`, replace them with an application error, or
return a successful result after either. `ActivityCancelled` inherits from
`BaseException`, so `except Exception` alone does not catch it. Ordinary return
and work exceptions also stop and join the helper.
Expand Down
49 changes: 48 additions & 1 deletion docs/sdk-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -1516,7 +1516,7 @@ storage-admission handling remain separate. See [Worker recovery](reference/work
Use `await activity.context().heartbeat(details)` in an async activity or
`activity.context().heartbeat_sync(details)` in a synchronous activity thread.
Both raise `ActivityCancelled` when cancellation is requested. An explicit
`can_continue=false` instead raises `ActivityLeaseAuthorityLost` with the
`can_continue=false` instead raises `ActivityWorkerLost` with the
original `.reason`, such as `activity_deadline_elapsed`, `attempt_closed` or
`run_closed`. This is lost execution authority, not a new activity failure.
The worker releases the slot after the callback exits and submits neither a
Expand All @@ -1528,6 +1528,53 @@ supervision. The SDK cannot forcibly stop a running native thread or undo its
external effects. Keep side effects idempotent and heartbeat at useful progress
boundaries.

## Activity lease renewal

When the Server advertises `activity_lease_renewal`, the worker keeps each
claimed activity's lease alive in the background. It confirms the lease right
after the poll, then renews it about every third of the lease, at most every
30 seconds. Each renewal exchange is limited to 5 seconds, and part of the
lease is always held in reserve: at most 10 seconds, and at most a fifth of
the grant.

A timeout, connection reset, 5xx or 429 on a renewal does not end the claim.
The activity keeps running under the lease it already holds. Renewal is retried
with the same backoff in every Durable Workflow SDK:

- The *n*-th retry waits `uniform(0, min(4, 0.25 * 2**n))` seconds: 0.25 s,
doubling to a 4 s cap.
- Full random jitter keeps workers from retrying in step after an outage.
- Retries always stay inside the lease, ahead of the reserve.

If the claim was never confirmed, its lease is charged from the start of the
poll, because the Server stamps the lease only after it receives the poll.

If renewal has still not succeeded when only the reserve is left, the worker
stops the activity:

- An async activity is cancelled.
- A thread activity is asked to stop: its next `heartbeat_sync()` raises
`ActivityWorkerLost`.
- A supervised callback process is stopped.

The worker waits for that stop for up to half of the reserve. Once every
execution of the claim has stopped, it releases the claim with
`POST /worker/activity-tasks/{id}/fail`. The failure has type `worker_lost`
(recorded as `exception_type`, and matchable in `non_retryable_error_types`),
class `ActivityWorkerLost`, `non_retryable = false`, the local cause in the
message, and `details = {"reason": "worker_lost", "cause": ...}`. The
activity's retry policy then applies right away.

If the execution does not stop in time, for example a thread that never
heartbeats, the worker sends nothing. The Server settles the attempt when its
lease expires (`lease_expired`), so two executions of one attempt never
overlap. A claim that never started running is released directly.

In workflow code, `ActivityFailed.lease_loss_reason()` tells these outcomes
apart. It returns `"worker_lost"` for a released claim, `"lease_expired"` or
`"worker_deregistered"` when the Server settled the lease, and `None` for any
other failure.

### Acknowledged heartbeat checkpoints

SDK 2.18.0 adds `activity.context().last_heartbeat_checkpoint()`,
Expand Down
Loading