Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/feature-test-timings.json
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@
"tests/Feature/V2/V2LifecycleEventTest.php": 3.430998,
"tests/Feature/V2/V2LocalActivityTest.php": 2.806176,
"tests/Feature/V2/V2LocalEntityReferenceAuthorityTest.php": 9.066242,
"tests/Feature/V2/V2LostLeaseTrackingMigrationTest.php": 10.0,
"tests/Feature/V2/V2MachineDefinitionTest.php": 13.210704,
"tests/Feature/V2/V2MemoUpsertTest.php": 10.058037,
"tests/Feature/V2/V2MessageCursorContinueAsNewTest.php": 4.673203,
Expand Down Expand Up @@ -244,6 +245,7 @@
"tests/Feature/V2/V2WorkflowStubContractTest.php": 0.653342,
"tests/Feature/V2/V2WorkflowStubFakeTest.php": 12.837294,
"tests/Feature/V2/V2WorkflowTaskBridgeTest.php": 269.332131,
"tests/Feature/V2/V2WorkflowTaskLostLeaseTest.php": 20.0,
"tests/Feature/V2/V2WorkflowTest.php": 161.306516,
"tests/Feature/V2/V2WorkflowTimeoutTest.php": 11.990609,
"tests/Feature/V2/V2WorkflowVersionPinningTest.php": 0.888561,
Expand Down Expand Up @@ -395,6 +397,7 @@
"tests/Feature/V2/V2LifecycleEventTest.php": 2.681959,
"tests/Feature/V2/V2LocalActivityTest.php": 1.202231,
"tests/Feature/V2/V2LocalEntityReferenceAuthorityTest.php": 10.707134,
"tests/Feature/V2/V2LostLeaseTrackingMigrationTest.php": 3.0,
"tests/Feature/V2/V2MachineDefinitionTest.php": 9.882914,
"tests/Feature/V2/V2MemoUpsertTest.php": 5.262701,
"tests/Feature/V2/V2MessageCursorContinueAsNewTest.php": 1.910908,
Expand Down Expand Up @@ -500,6 +503,7 @@
"tests/Feature/V2/V2WorkflowStubContractTest.php": 0.571702,
"tests/Feature/V2/V2WorkflowStubFakeTest.php": 10.15931,
"tests/Feature/V2/V2WorkflowTaskBridgeTest.php": 149.713579,
"tests/Feature/V2/V2WorkflowTaskLostLeaseTest.php": 15.0,
"tests/Feature/V2/V2WorkflowTest.php": 115.227937,
"tests/Feature/V2/V2WorkflowTimeoutTest.php": 6.892443,
"tests/Feature/V2/V2WorkflowVersionPinningTest.php": 0.853134,
Expand Down
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,33 @@
definitions, rules, budgets, child bindings and native child start in 2.29.0.
Docblocks and documentation only; no behavior changes.

- A workflow task that keeps losing its lease without completing no longer
retries silently forever (#1231). A lease is lost when the worker running
the task is killed, times out, or loses its connection.
- Opt-in `workflows.v2.workflow_task_job_timeout_seconds`
(`DW_V2_WORKFLOW_TASK_JOB_TIMEOUT_SECONDS`, unset by default) gives workflow
task jobs their own queue timeout, kept 10 s below the connection's
`retry_after`, in place of the worker's `--timeout`. Unset, the worker's
`--timeout` applies as before. A worker on the workflow queue logs one
warning at start when the timeout its workflow tasks run under is below
the workflow task lease.
- Task repair counts each lost lease on the task (`lost_lease_count`,
`last_lease_lost_at`). The next claim logs a typed
`workflow_task_lease_lost` diagnostic. No history is written, and run
semantics are unchanged.
- At three consecutive lost leases
(`workflows.v2.workflow_task_lost_leases.attention_threshold`), the run
summary, list items and describe carry an additive `task_attention` field
(`reason`, `consecutive`, `threshold`), and operator metrics add
`runs.task_attention`, `tasks.lease_lost_repeatedly` and
`tasks.max_lost_lease_count`. Repair-blocked reasons, repair attention
and `can_repair` are unchanged.
- Opt-in (`workflows.v2.workflow_task_lost_leases.fail_run`, off by default;
boot refuses a threshold below 2): at the threshold, the next claim fails
the run with a typed `WorkflowTaskLeaseLostException` (`task_failure`)
instead of retrying.
- The migration adds the columns outside a transaction under the schema
lock budget.
- Workflow definition and `await()` condition fingerprints are versioned, and
a new opt-in version 2 makes comment-only and whitespace-only edits replay-safe
(#1227). The prefix names the version:
Expand Down
49 changes: 49 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,55 @@ Run a Laravel queue worker or Horizon to execute workflows and activities:
php artisan queue:work
```

With the `pcntl` extension, a queue worker kills any job that runs longer than
its timeout: the worker's `--timeout`, 60 seconds by default. A workflow task
killed this way is retried only after its lease expires
(`workflows.v2.workflow_task_lease_seconds`, 300 seconds by default) and is
counted as a lost lease. A worker that serves the workflow queue logs a warning
when it starts if its timeout is below the lease. Wide `all()` fan-outs can need
longer than 60 seconds for one workflow task.

Raise `--timeout` on the workers that serve workflow tasks to cover your
longest workflow task, keeping the queue connection's `retry_after` above it;
Laravel requires `retry_after` to exceed every job's timeout. To keep a
bounded timeout for activities, give them their own queue and workers.
Activities run on their workflow's connection and queue unless the activity
class sets `$connection` or `$queue`, or the workflow passes `ActivityOptions`:

```php
// config/queue.php
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'retry_after' => 330, // above the workflow task worker's --timeout
],
```

```php
final class ChargeCard extends Activity
{
public ?string $queue = 'activities';
}
```

```bash
php artisan queue:work redis --queue=default --timeout=300 # workflow tasks
php artisan queue:work redis --queue=activities --timeout=60 # activities, bounded
```

From Workflow 2.34.0, `DW_V2_WORKFLOW_TASK_JOB_TIMEOUT_SECONDS` can instead give
workflow task jobs their own timeout, which Laravel's worker uses in place of
its `--timeout` for those jobs, while activities on the same workers keep the
worker's `--timeout`. It is kept 10 seconds below the connection's
`retry_after`, so raise `retry_after` above it. It is unset by default.

If a workflow task loses its lease three times in a row
(`DW_V2_WORKFLOW_TASK_LOST_LEASE_ATTENTION_THRESHOLD`), the run summary and
describe carry an additive `task_attention` field, and operator metrics count
it. Set `DW_V2_WORKFLOW_TASK_LOST_LEASE_FAIL_RUN=true` to fail such a run with a
typed task failure instead; that requires a threshold of at least 2.

For the default database queue on SQLite, failed reservation writes preserve
unclaimed work after a proven rollback. See [SQLite queue reservations](docs/sqlite-queue-reservations.md)
for the supported framework versions and custom-connector boundary.
Expand Down
8 changes: 8 additions & 0 deletions docs/architecture/operational-liveness.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,14 @@ codes that a caller may observe:
Adding a blocked reason is a protocol change; deleting one is a
protocol change.

A workflow task that keeps losing its lease without completing (its
worker was killed, timed out, or lost its connection) is not a blocked
repair: repair keeps returning it to ready and `can_repair` is unchanged.
It is reported in the separate, additive `task_attention` field of the
run summary, list items and describe, set once the open workflow task has
lost `workflows.v2.workflow_task_lost_leases.attention_threshold` leases
in a row (3 by default) and null otherwise.

## Repair cadence and bounded backoff

The repair path is bounded. An automated recovery pass cannot
Expand Down
21 changes: 19 additions & 2 deletions scripts/ci/upgrade-journey-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,27 @@ artisan_output() {

# Runs migrate:rollback, which must be refused because retained compensation
# scopes exist. Fails unless the refusal comes from the package preflight.
# The migration batch that holds the compensation ledger. A candidate that adds
# its own migrations runs them in a later batch, which a plain rollback would
# undo first without reaching the ledger.
ledger_batch() {
COLUMNS=1000 php artisan migrate:status --no-ansi --no-interaction 2>/dev/null \
| grep -E '_create_compensation_scopes' \
| grep -oE '\[[0-9]+\]' \
| tr -d '[]' \
| head -n 1 || true
}

expect_rollback_refused() {
local rollback_output
local rollback_output batch
local -a batch_option=()

batch=$(ledger_batch)
if [ -n "$batch" ]; then
batch_option=("--batch=$batch")
fi

if rollback_output=$(COLUMNS=1000 php artisan migrate:rollback --force --no-interaction 2>&1 | tr -s ' \n' ' '; exit "${PIPESTATUS[0]}"); then
if rollback_output=$(COLUMNS=1000 php artisan migrate:rollback "${batch_option[@]}" --force --no-interaction 2>&1 | tr -s ' \n' ' '; exit "${PIPESTATUS[0]}"); then
printf '%s\n' "$rollback_output" >&2
echo "$1" >&2
exit 1
Expand Down
20 changes: 20 additions & 0 deletions src/Providers/WorkflowServiceProvider.php
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,9 @@
use Workflow\V2\Support\SqliteSavepointRelease;
use Workflow\V2\Support\TaskFairnessState;
use Workflow\V2\Support\TypeRegistry;
use Workflow\V2\Support\WorkerTimeoutAdvisory;
use Workflow\V2\Support\WorkflowModeGuard;
use Workflow\V2\Support\WorkflowTaskLeaseLoss;
use Workflow\Watchdog;

final class WorkflowServiceProvider extends ServiceProvider
Expand Down Expand Up @@ -169,7 +171,7 @@
});
}

public function boot(): void

Check failure on line 174 in src/Providers/WorkflowServiceProvider.php

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this function to reduce its Cognitive Complexity from 17 to the 15 allowed.

See more on https://sonarcloud.io/project/issues?id=durable-workflow_workflow&issues=AaH1Q7A2BvnSQhuiF09k&open=AaH1Q7A2BvnSQhuiF09k&pullRequest=1244
{
SerializableClosure::setSecretKey(config('app.key'));

Expand Down Expand Up @@ -222,6 +224,7 @@
// Refuse an unsupported definition fingerprint version before any
// workflow can run; at runtime the setting never fails a run.
SourceFingerprint::validateConfiguration();
WorkflowTaskLeaseLoss::validateConfiguration();
$this->configureSqliteStorageConnection();

// Register long-poll wake signal observers
Expand All @@ -237,6 +240,23 @@
// Validate cache backend for multi-node deployments
$this->validateCacheBackend();

if (class_exists(\Illuminate\Queue\Events\WorkerStarting::class)) {
// A timeout below the workflow task lease kills workflow tasks that
// are then retried only after the lease; warn workflow-queue
// workers once.
Event::listen(
\Illuminate\Queue\Events\WorkerStarting::class,
static function (object $event): void {
$options = $event->workerOptions ?? null;
WorkerTimeoutAdvisory::workerStarting(
is_string($event->connectionName ?? null) ? $event->connectionName : null,
is_string($event->queue ?? null) ? $event->queue : null,
is_object($options) && is_numeric($options->timeout ?? null) ? (int) $options->timeout : null,
);
},
);
}

Event::listen(Looping::class, static function (Looping $event) use ($v1Enabled): void {
if ($v1Enabled) {
Watchdog::wake($event->connectionName, $event->queue);
Expand Down
29 changes: 29 additions & 0 deletions src/V2/Exceptions/WorkflowTaskLeaseLostException.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<?php

declare(strict_types=1);

namespace Workflow\V2\Exceptions;

use RuntimeException;

/**
* A run failed because its workflow task kept losing its lease without
* completing, with workflows.v2.workflow_task_lost_leases.fail_run enabled.
* A lease is lost when the worker running the task is killed, times out, or
* loses its connection before the task completes.
*/
final class WorkflowTaskLeaseLostException extends RuntimeException
{
public function __construct(
public readonly string $taskId,
public readonly int $lostLeases,
public readonly int $leaseSeconds,
) {
parent::__construct(sprintf(
'Workflow task %s lost its lease %d times in a row without completing (its worker was killed, '
. 'timed out, or lost its connection).',
$taskId,
$lostLeases,
));
}
}
25 changes: 25 additions & 0 deletions src/V2/Jobs/RunWorkflowTask.php
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@
use Workflow\V2\Support\WorkflowContinuationCache;
use Workflow\V2\Support\WorkflowExecutor;
use Workflow\V2\Support\WorkflowSignalReadScope;
use Workflow\V2\Support\WorkflowTaskJobTimeout;
use Workflow\V2\Support\WorkflowTaskLease;
use Workflow\V2\Support\WorkflowTaskLeaseLoss;

final class RunWorkflowTask implements ShouldQueue
{
Expand All @@ -46,6 +48,14 @@ final class RunWorkflowTask implements ShouldQueue

public int $tries = 1;

/**
* The job's own queue timeout, set at dispatch only when
* workflows.v2.workflow_task_job_timeout_seconds is configured
* ({@see WorkflowTaskJobTimeout}); null leaves the worker's --timeout in
* force. Workers of earlier releases ignore the serialized value.
*/
public ?int $timeout = null;

// Mixed so malformed telemetry cannot prevent an older job from loading.
public mixed $dispatchObservabilityContext = null;

Expand Down Expand Up @@ -164,6 +174,15 @@ private function executeClaimPhase(
return null;
}

if (WorkflowTaskLeaseLoss::failsRunAt($task)) {
// Opt-in: a task that keeps losing its lease fails its
// run with a typed task failure instead of retrying (#1231).
$continuation?->disable('workflow_task_lease_lost');
$executor->failAfterLostLeases($run, $task);

return null;
}

if ($continuation !== null && ! $continuation->hasAuthority($run, $task)) {
$continuation->disable('lost_task_authority');
return null;
Expand Down Expand Up @@ -286,6 +305,12 @@ private function claimTask(): array
ServiceCallWorkflowDelivery::recordForClaim($task, $run);
$this->projectRun($run->fresh(['instance', 'tasks', 'activityExecutions', 'failures']));

// A retry after lost leases records its typed diagnostic (#1231).
$leaseLoss = WorkflowTaskLeaseLoss::claimDiagnostic($task);
if ($leaseLoss !== null) {
WorkflowTaskLeaseLoss::reportClaim($leaseLoss);
}

return [$task->setRelation('run', $run), null, $claimedAttempt];
});
}
Expand Down
1 change: 1 addition & 0 deletions src/V2/Models/WorkflowRunSummary.php
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ class WorkflowRunSummary extends Model
protected $casts = [
'is_current_run' => 'bool',
'repair_attention' => 'bool',
'task_attention' => 'array',
'task_problem' => 'bool',
'visibility_labels' => 'array',
'projection_schema_version' => 'integer',
Expand Down
2 changes: 2 additions & 0 deletions src/V2/Models/WorkflowTask.php
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ class WorkflowTask extends Model
'payload' => 'array',
'queue_delivery_reference' => 'array',
'repair_count' => 'integer',
'lost_lease_count' => 'integer',
'priority' => 'integer',
'fairness_weight' => 'integer',
'sticky_until' => 'datetime',
Expand All @@ -50,6 +51,7 @@ class WorkflowTask extends Model
'last_dispatched_at' => 'datetime',
'last_claim_failed_at' => 'datetime',
'repair_available_at' => 'datetime',
'last_lease_lost_at' => 'datetime',
];

public function run(): BelongsTo
Expand Down
5 changes: 4 additions & 1 deletion src/V2/Support/FailureFactory.php
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
use Workflow\V2\Exceptions\StructuralLimitExceededException;
use Workflow\V2\Exceptions\UnresolvedWorkflowFailureException;
use Workflow\V2\Exceptions\UnsupportedWorkflowYieldException;
use Workflow\V2\Exceptions\WorkflowTaskLeaseLostException;

final class FailureFactory
{
Expand Down Expand Up @@ -361,10 +362,12 @@ private static function classifyFromThrowable(?Throwable $throwable): FailureCat
return FailureCategory::StructuralLimit;
}

// Task-level failures: determinism violations, unsupported yields, replay shape errors.
// Task-level failures: determinism violations, unsupported yields, replay shape errors,
// and a workflow task that keeps losing its lease (#1231).
if (
$throwable instanceof UnsupportedWorkflowYieldException
|| $throwable instanceof StraightLineWorkflowRequiredException
|| $throwable instanceof WorkflowTaskLeaseLostException
) {
return FailureCategory::TaskFailure;
}
Expand Down
Loading
Loading