Repository navigation
Do not rerun a failed activity redispatched before its failure is recorded (#1241) - #1245
Merged
Merged
Conversation
…orded A v1 activity's failure is recorded by a separately queued Workflow\Exception job, after Laravel has released the activity's unique lock. A workflow replay in that gap dispatched the activity again and the copy executed it, adding an exception row and repeating its side effects (the #1241 flake: the second retry attempt ran twice). The failing job now marks the activity in the cache before its lock is released. Another job for the same index that finds the mark completes without executing or recording a result; the Exception job removes the mark once it has recorded the failure or let it go. History, payloads and replay are unchanged. Fixes #1241. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…n unit tests Track the Exception job's own release instead of asking the queue job, expose the copy's skip as an unserialized public property, and cover marking, skipping, same-job retries and removal in unit tests. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review of #1245: a mark kept forever stranded a workflow whose Workflow\Exception job was lost (flushed queue, undecryptable payload), because every later copy of the activity skipped. The mark now expires after workflows.activity_failure_mark_seconds (DW_V1_ACTIVITY_FAILURE_MARK_SECONDS, default 3600); after that a wake runs the activity again, as before. The key is top-level so applications with an older published workflows.php receive the default. ActivityMiddleware now checks the mark before ActivityStarted, so a skipped copy fires no activity events. Adds the lost-Exception-job regression and unit coverage; documents that timeout and max-attempts-after-crash failures write no mark. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Fixes #1241.
Cause: an engine defect, not a test race
ExceptionLoggingReplayTest::testSignalRetryLogsEachSequentialExceptionrecorded three exceptions because the second activity attempt really executed twice. Thelaravel-log-postgresql-12artifact of the failing run (44554854139) showsfirst failure,second failure,second failurefrom that test. The test's expectation of two exceptions is correct.A v1 activity's failure is recorded by a separately queued
Workflow\Exceptionjob. Laravel releases the activity'sShouldBeUniquelock when the job fails (CallQueuedHandler::failedreleases it before callingActivity::failed(), which queues the Exception job). Until that Exception job writes the log at the activity's index,ActivityStub::make()sees no log and the unique lock is free. Any replay in that gap (a signal, timer or sibling result) dispatches the activity again.Activity::handle()only skipped when a log already existed, so the copy ran the activity again. That added an exception row and repeated the activity's side effects. In the flaky test, the Exception job's wake and the signal's wake both replayed the workflow, and the second replay landed in that gap.Successful results have no such gap:
ActivityMiddleware::onUnlockwrites the log before the unique lock is released.Fix
Activity::handle()stores the failing queue job's UUID underlaravel-workflow-activity-failure:{workflow}:{index}. This runs before the worker can fail the job and release the unique lock.ActivityMiddlewarechecks the mark beforeActivityStarted, so a skipped copy fires no activity events and records no result. Retries of the same job (same UUID) still execute.workflows.activity_failure_mark_seconds(DW_V1_ACTIVITY_FAILURE_MARK_SECONDS, default 3600). If the Exception job is lost (a flushed queue during a deploy, an APP_KEY rotation that makes its payload undecryptable,failed()throwing before it dispatches, a manual delete), the mark expires and the next wake runs the activity again, as before this PR, instead of the workflow waiting forever. One hour is far above normal Exception-job latency: that is seconds, made of the 1 s lock and probe retries plus a wait behind earlier pending work in the same workflow. The setting is a top-level key so applications with an older publishedworkflows.php, whosev1array would shadow a nested key, still get the default.Workflow\Exceptionremoves the mark once it has recorded the failure or let it go: persisted, already logged, probe skip, or a workflow that is no longer running. While it is waiting (released), the mark stays.Activity::awaitsRecordedFailure(), into@internaltyped properties that are never serialized.Queue/Worker.phptimeout and max-attempts paths), fails without running the activity's catch block. It writes no mark, so the race stays open on those paths, no worse than before. The CHANGELOG says so too.Replay compatibility: recorded history, log/exception rows and job payloads are unchanged. The new flag is an uninitialized typed property, so it is never serialized. Jobs already in flight from older releases behave as before.
Evidence
FailedActivityRedispatchTest, uses--onceworkers on a private queue. It fails the activity's job, replays the workflow before the failure is recorded (the replay does dispatch the activity again), holds the Exception job, then runs the copy. Before the fix it failed 20/20 at "The failed activity ran again" (2 executions). After the fix it passed 20/20.testLostExceptionJobLeavesTheActivityRunnableAfterTheMarkExpiresdeletes the queued Exception job, runs a wake's copy while the mark lasts (it does not run), waits for a 2 s configured mark to expire, then wakes again. The activity runs a second time and the workflow completes. This is the reviewer's scenario (it stayed inWaitingStatusforever on the previous head), with expiry added.ActivityMiddlewareTest), the mark TTL setting, same-job retries clearing the mark, a released Exception job keeping it, and a recorded failure removing it.ExceptionLoggingReplayTest,ActivityCompletionWakeTest,ActivityDispatchBoundaryTest,ExceptionWorkflowTestandFailingWorkflowTestpass locally on PostgreSQL. ecs and phpstan pass on the changed files..github/feature-test-timings.jsonhas weights for the new file.verify-feature-shards.phppasses.🤖 Generated with Claude Code