ateapi: Fix various actor workflow races - #1444
Open
Eitan Yarmush (EItanya) wants to merge 2 commits into
Open
Conversation
Eitan Yarmush (EItanya)
force-pushed
the
fix/preserve-crashed-lifecycle
branch
from
September 3, 2026 18:41
60fbab0 to
a74f853
Compare
Eitan Yarmush (EItanya)
marked this pull request as ready for review
September 3, 2026 22:06
Eitan Yarmush (EItanya)
force-pushed
the
fix/preserve-crashed-lifecycle
branch
from
September 4, 2026 13:33
a74f853 to
359b899
Compare
Eitan Yarmush (EItanya)
requested review from
Haven Xia (HavenXia),
Julian Gutierrez Oschmann (juli4n) and
Zoe Zhao (zoez7)
September 4, 2026 13:35
Eitan Yarmush (EItanya)
force-pushed
the
fix/preserve-crashed-lifecycle
branch
from
September 5, 2026 22:39
359b899 to
ca33fa5
Compare
Signed-off-by: Eitan Yarmush <[email protected]>
Eitan Yarmush (EItanya)
force-pushed
the
fix/preserve-crashed-lifecycle
branch
from
September 5, 2026 22:48
ca33fa5 to
ad92263
Compare
A worker claim can commit before resume writes the actor backlink. Advance the actor version when deleting a drained worker with a pending claim so that resume cannot persist a reference to the deleted worker. Preserve suspended and paused state, and retry deletion if resume wins the version check. Signed-off-by: Eitan Yarmush <[email protected]>
Eitan Yarmush (EItanya)
changed the title
Eitan Yarmush (EItanya)
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Worker deletion can race both lifecycle finalization and the gap between a worker claim and its actor backlink. Preserve
CRASHEDduring resume and suspend finalization, returningFailedPreconditioninstead of reporting an invalid successful transition.When a drained worker still has a claim whose actor backlink is pending, advance the actor's version before deleting the worker. The pending resume write then loses its version check and retains the actor's suspended or paused state and snapshots for retry. If resume wins the version check, deletion returns
Abortedand retries against the committed assignment.Regression tests cover the lifecycle finalization races, both assignment/deletion orderings, snapshot preservation, and avoiding false crash metrics.
Fixes #1443
Validation:
-race.env -u NO_COLOR make verifyin a clean worktree (includesgo test -race ./...).