Repository navigation
Keep activity claims through transient lease renewal failures; release an unrenewable claim as worker_lost (#352) - #353
Merged
Conversation
A transport timeout, connection reset or 5xx/429 on the renew-lease confirmation sent right after an activity poll raised ActivityLeaseAuthorityLost out of the poll, and the worker discarded the claimed task without completing, failing or releasing it. The Server then held the lease until expiry and, with the default single attempt, failed the workflow (same bug as durable-workflow/sdk-php#445). Now the activity runs under the poll's grant while confirmation is retried with backoff inside the lease. Every renewal is bounded to leave a report reserve (at most 10 s, at most a fifth of the grant). If renewal has still not succeeded when only that reserve is left, the worker stops the execution and fails the attempt retryably through the Server (ActivityLeaseAuthorityLost / activity_lease_renewal_unavailable) instead of dropping it. Typed Server refusals still end the claim. Worker-side lease handling only; no commands or history change. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…#352) Align the released-claim failure with the cross-SDK wire contract shared with PHP and Rust: failure.type "worker_lost", failure.class "ActivityWorkerLost", non_retryable false, the local cause (activity_lease_renewal_unavailable) in the message, and details {"reason": "worker_lost", "cause": <cause>}. Rename ActivityLeaseAuthorityLost to ActivityWorkerLost, keeping the old name as a deprecated alias of the same class so existing handlers keep working. Add ActivityFailed.lease_loss_reason(): "worker_lost" for a released claim, the recorded failure_reason ("lease_expired" or "worker_deregistered", default "lease_expired") for a Server-settled ActivityLeaseExpiredException, otherwise None. ActivityFailed now keeps the recorded failure_reason. The default retry policy is unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review follow-up, shared rule for all SDKs: a claim is released as worker_lost only after its execution has confirmed stopped inside the lease; otherwise nothing is sent and the Server settles lease_expired, so two executions of one attempt never overlap. - Thread futures and callback processes of a claim are tracked on its lease authority. On renewal exhaustion the execution is stopped (async cancelled, thread asked to stop via its next heartbeat_sync, callback stopped) and awaited for up to half of the release reserve. Only then is the release sent, within the remaining lease. - An unconfirmed grant is charged from the poll's start, the conservative bound. If it is already spent, the unstarted claim is released directly, and run() never starts an exhausted claim. - Renewal retries use the cross-SDK backoff: uniform(0, min(4, 0.25 * 2**n)) seconds (full jitter). Documented in the SDK reference. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…352) Wrap a claim's synchronous activity before it is handed to the thread pool. A thread that has not started yet is fenced when the release asks, so it never runs afterwards; a started one counts as stopped only once its function has returned or raised. This also keeps the shared thread-pool code path (a replay-guarded symbol) unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes item 4 of #352. This is the Python counterpart of durable-workflow/sdk-php#445.
Bug
When an activity poll returns a task,
ActivityLeaseAuthority.confirm_deliverysends onerenew-leaserequest with amin(5 s, remaining)budget and no retry. A transport timeout, a connection reset, or a 5xx/429 answer raised out ofpoll_activity_task. The poll loop then discarded the claimed task (worker.py:4244-4277on main). Nothing was completed, failed or released. The Server held the lease (300 s by default) and settled it as expired. With the default single attempt, that failed the workflow.In steady state, renewals that failed until the local bound ran out also ended in a silent
claim_aborted.Change
Keeps working. A transient failure keeps the claim, at start or later. The activity runs under the lease it already holds, and renewal is retried inside that lease.
uniform(0, min(4, 0.25·2^n))seconds, with full jitter.Conservative lease clock. An unconfirmed grant is charged from the poll's start, because the Server stamps the lease only after it receives the poll. If that grant is already spent, nothing has run, so the unstarted claim is released directly.
run()never starts an exhausted claim.Stops before it releases (shared rule for all SDKs). Each claim's activity threads and callback processes are tracked on its lease authority. Each thread is tracked by a race-free guard wrapped around the activity: a thread that has not started yet is fenced so it never runs, and a started one counts as stopped only once it returns. If renewal has not succeeded when only the reserve is left, the execution is stopped:
heartbeat_sync()raisesActivityWorkerLostThe worker then waits up to half of the reserve for every execution to actually stop. Only then does it release the claim, inside the remaining lease, with the cross-SDK wire contract:
POST /api/worker/activity-tasks/{id}/failfailure.type = "worker_lost"failure.class = "ActivityWorkerLost"failure.non_retryable = falseactivity_lease_renewal_unavailablefailure.details = {"reason": "worker_lost", "cause": ...}If the execution does not stop in time, for example a thread that never heartbeats or a coroutine that swallows cancellation, nothing is sent. The Server settles the attempt at lease expiry (
lease_expired), so two executions of one attempt never overlap. A stale or late release is refused by the Server anyway (409 stale_attempton 3.16.0).Unchanged: a typed Server refusal (
lease_expired,run_closed, other 4xx) still ends the claim as before. Fencing of late results is unchanged.Rename.
ActivityLeaseAuthorityLostbecomesActivityWorkerLost. The old name stays importable as a deprecated alias of the same class.Workflow side.
ActivityFailed.lease_loss_reason()returns:"worker_lost"for a released claimActivityLeaseExpiredException: the recordedfailure_reason("lease_expired"or"worker_deregistered"), defaulting to"lease_expired"Nonefor any other failureThe default retry policy is unchanged.
Docs. There is a new "Activity lease renewal" section in
docs/sdk-reference.md, covering the backoff, the reserve, the stop-then-release rule, andlease_loss_reason().Replay safety. Worker-side lease handling only. No workflow commands and no recorded history change.
Tests
tests/test_activity_lease_worker.pyruns against a synthetic lease server with faults injected (timeout,reset,503):/failis sent.worker_lost/fail, inside the lease.heartbeat_sync()claim_aborted. The thread is still running and holding its slot, and nothing is ever sent, so the lease is left to expire.lease_expired)tests/test_activity_lease_authority.pychecks:[0, ceiling]tests/test_activity_worker_lost.pychecks the alias andlease_loss_reason().Local results (Python 3.14, httpx 0.28.1):
pytest tests/ -m "not integration": 7807 passed, 3 skipped.ruff: clean.mypy: clean.🤖 Generated with Claude Code