Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- Opt-in first-recorded-failure selection for `all()` groups (`yield [...]`), as a two-phase rollout (#307). With `Worker(..., record_parallel_failure_order=True)` or `DURABLE_WORKFLOW_RECORD_PARALLEL_FAILURE_ORDER=true`, a new run records the version marker `durable_workflow.parallel_failure_order` (version 1) directly before its first group that contains an activity or child workflow. A marked run raises the failure recorded first in history, even when every member had settled before the workflow resumed. Workers up to 2.20.0 raise the lowest-position failure in that case, so the exception a workflow caught depended on whether its task ran before or after the other members finished. The first recorded failure is still raised as soon as it is recorded, and the other members keep running, are not cancelled, and have their outcomes ignored. Ties follow durable event order, as the shared parallel-group contract specifies. Workflow's embedded engine compares millisecond timestamps and then member index, so failures recorded in the same millisecond can be chosen differently (durable-workflow/.github#415). Any other value of the environment variable is refused when the `Worker` is created.
- **Default off; enable it only after every worker on the task queue runs this release or later.** Published 2.20.0 and earlier cannot replay a marked run. The Server marks the workflow task replay-blocked with no retry, and the run waits until it is repaired, which can hand it to an old worker again. With recording off, new runs send exactly the commands 2.20.0 sends. Turning recording off again is safe. Rolling back below this release requires marked runs to be drained first. The next minor release turns recording on by default (#351).
- **Replay safety:** every worker of this release follows a marker a run already recorded, whether or not recording is on, for every workflow task, query and update. A marked run needs no per-group decision, because the first failure a task sees in append-only history is the first every later replay sees. A run whose first such group was recorded without the marker keeps the earlier rule for its whole life. Its settled groups raise the lowest-position failure, and a failure raised while siblings are open still records the #338 decision. This holds even when a task recorded no command after catching the failure, or another worker answered an update in between. A run with a cancellation request when it reaches its first group records no marker.
- **Regression evidence:** fixtures recorded by published 2.19.1 on a Server built from main replay under the earlier rule. They cover a settled group in flight, a catch followed by a task that recorded no command, an update answered by 2.19.1 before or between later tasks, and a select() cancellation. Marked runs recorded on native and PHP Server replay under the new rule without the setting. New fixtures cover the first task with recording on and off. Existing fixtures, including `parallel-group-metadata-emission`, the completion-limit fixtures and `business-clock-parallel-failure-selection-gap`, are unchanged.

## [2.20.0] - 2026-11-19

Both additions need **native Server 3.15.0 or later**, the first Server release that advertises `capabilities.signal_with_start` and the `signal_external_workflow` worker command. Each is discovered from the Server's capabilities and refused before anything is sent on an older Server, so the SDK's supported Server baselines are unchanged and every other feature keeps working on older Servers. Worker protocol (1.20) and control-plane (2) versions are unchanged.
Expand Down
44 changes: 35 additions & 9 deletions docs/sdk-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -522,15 +522,41 @@ profile, (shipping, _) = results
```

A failed activity or child is thrown at the list-yield point as soon as it is
recorded. The other members keep running; their later outcomes do not change
the thrown failure or reach the workflow's next operations. If every member
had already settled when the workflow resumed, the failure with the lowest
durable input position is thrown. A group whose members are already scheduled
is never scheduled again on a later wake, such as a signal. When a failure is
raised while other members are open, the worker records which member it
raised as a side effect directly after the group, so every later replay raises
the same one. Code after such a group therefore starts one workflow sequence
later.
recorded. The other members keep running and are not cancelled; their later
outcomes do not change the thrown failure or reach the workflow's next
operations. A group whose members are already scheduled is never scheduled
again on a later wake, such as a signal.

When several members have failed, a run that recorded the failure-order marker
throws the failure recorded first in history, in durable event order, even if
every member had already settled when the workflow resumed. The marker is the
version marker `durable_workflow.parallel_failure_order`, recorded once, directly
before the run's first group that contains an activity or child workflow; code
after it starts one workflow sequence later. Workflow's embedded engine also
throws the earliest failure, but compares millisecond timestamps and then member
index, so two failures recorded in the same millisecond can be chosen
differently
([durable-workflow/.github#415](https://github.com/durable-workflow/.github/issues/415)).

Recording the marker is opt-in in this release: pass
`Worker(..., record_parallel_failure_order=True)` or set
`DURABLE_WORKFLOW_RECORD_PARALLEL_FAILURE_ORDER=true`. Turn it on only after
every worker that polls the task queue runs this release or later. Workers up
to 2.20.0 cannot replay a marked run: the Server marks its workflow task
replay-blocked, and the run waits until it is repaired, which can hand it to an
old worker again. Every worker of this release follows a marker a run already
recorded, whether or not recording is on, so turning recording off again is
safe. Rolling back below this release needs marked runs drained first. A later
minor release turns recording on by default
([#351](https://github.com/durable-workflow/sdk-python/issues/351)).

A run without the marker keeps the earlier rule for its whole life, on every
task, query and update: if every member had settled when the workflow resumed,
the failure with the lowest input position is thrown. When such a run raises a
failure while other members are open, the worker records which member it raised
as a side effect directly after the group, so every later replay raises the
same one. A run that has a cancellation request when it reaches its first group
records no marker.

## Saga compensation

Expand Down
32 changes: 32 additions & 0 deletions src/durable_workflow/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import json
import logging
import math
import os
import pickle
import random
import sys
Expand Down Expand Up @@ -1156,6 +1157,24 @@ def _contract_version_matches(value: Any, expected: int) -> bool:
return False


RECORD_PARALLEL_FAILURE_ORDER_ENV = "DURABLE_WORKFLOW_RECORD_PARALLEL_FAILURE_ORDER"


def _record_parallel_failure_order(value: bool | None) -> bool:
"""The worker's failure-order recording setting, refused when malformed."""
if value is None:
raw = os.environ.get(RECORD_PARALLEL_FAILURE_ORDER_ENV, "")
setting = raw.strip().lower()
if setting in ("", "false", "0"):
return False
if setting in ("true", "1"):
return True
raise ValueError(f"{RECORD_PARALLEL_FAILURE_ORDER_ENV} must be true or false")
if type(value) is not bool:
raise ValueError("record_parallel_failure_order must be a bool or None")
return value


class Worker:
"""Polls workflow and activity tasks and dispatches them to Python callables."""

Expand Down Expand Up @@ -1184,8 +1203,20 @@ def __init__(
external_storage_threshold_bytes: int | None = None,
external_storage_cache: ExternalPayloadCache | None = None,
require_query_task_authority: bool = False,
record_parallel_failure_order: bool | None = None,
) -> None:
"""Create a worker.

``record_parallel_failure_order`` lets new runs record the parallel
failure-order marker before their first all-group, so their groups
raise the first recorded member failure (sdk-python#307). ``None``
reads ``DURABLE_WORKFLOW_RECORD_PARALLEL_FAILURE_ORDER`` (``true`` or
``false``, default ``false``). Enable it only after every worker on the
task queue runs this release or later: older workers cannot replay a
marked run. Every worker follows a marker that a run already recorded.
"""
self.client = client
self.record_parallel_failure_order = _record_parallel_failure_order(record_parallel_failure_order)
self.require_query_task_authority = require_query_task_authority
self._query_authority_supported = False
self.task_queue = task_queue
Expand Down Expand Up @@ -1960,6 +1991,7 @@ async def _replay_workflow_claim(
local_activity_cancellation_policies=self._local_activity_cancellation_policies,
allow_cancellation_scope_authoring=self._allow_cancellation_scope_authoring,
allow_cancellation_scope_delivery=self._allow_cancellation_scope_delivery,
record_parallel_failure_order=self.record_parallel_failure_order,
)
if outcome.cancellation_scope_delivery is not None:
if outcome.commands:
Expand Down
92 changes: 92 additions & 0 deletions src/durable_workflow/workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -3363,6 +3363,7 @@ def replay(
external_storage: ExternalStorageDriver | None = None,
external_storage_cache: ExternalPayloadCache | None = None,
cancel_requested: bool = False,
record_parallel_failure_order: bool = False,
) -> ReplayOutcome:
events = _history_events_from_export(history)
selected_workflow_type = workflow_type or _workflow_type_from_history(events)
Expand All @@ -3387,6 +3388,7 @@ def replay(
external_storage=external_storage,
external_storage_cache=external_storage_cache,
cancel_requested=cancel_requested,
record_parallel_failure_order=record_parallel_failure_order,
)

def _workflow_cls(self, workflow_type: str | None) -> type:
Expand Down Expand Up @@ -3756,7 +3758,16 @@ def replay(
local_activity_cancellation_policies: tuple[str, ...] = (),
allow_cancellation_scope_authoring: bool = False,
allow_cancellation_scope_delivery: bool = False,
record_parallel_failure_order: bool = False,
) -> ReplayOutcome:
"""Replay ``history_events`` and return the next commands.

``record_parallel_failure_order`` lets a run that reaches its first
all-group now record ``PARALLEL_FAILURE_ORDER_MARKER`` and raise its first
recorded member failure (sdk-python#307). It is off by default, so new runs
send the same commands as 2.20.0. A history that already contains the
marker follows it whether or not this is set.
"""
return _replay_state(
workflow_cls,
history_events,
Expand All @@ -3775,6 +3786,7 @@ def replay(
local_activity_cancellation_policies=local_activity_cancellation_policies,
allow_cancellation_scope_authoring=allow_cancellation_scope_authoring,
allow_cancellation_scope_delivery=allow_cancellation_scope_delivery,
record_parallel_failure_order=record_parallel_failure_order,
).outcome


Expand Down Expand Up @@ -4265,6 +4277,12 @@ def _history_event_type(event: Mapping[str, Any]) -> str | None:
# absent-policy all-group it raised while other members were open
# (sdk-python#338). Replay raises exactly that member.
PARALLEL_FAILURE_DECISION = "durable_workflow.parallel_failure_decision"
# The version marker a run records, directly before its first all-group, when
# that group is first scheduled by a worker with recording enabled
# (sdk-python#307). A marked run raises the first recorded member failure. A run
# without it keeps the failure selection of the workers that ran it before, for
# its whole life. Every worker reads it; writing it is opt-in in this release.
PARALLEL_FAILURE_ORDER_MARKER = "durable_workflow.parallel_failure_order"


def _settled_at_close(event: Any) -> bool:
Expand Down Expand Up @@ -5272,6 +5290,7 @@ def _replay_state(
allow_cancellation_scope_authoring: bool = False,
allow_cancellation_scope_delivery: bool = False,
parallel_group_policies: tuple[str, ...] = (),
record_parallel_failure_order: bool = False,
) -> _ReplayState:
_validate_workflow_entrypoint(workflow_cls)
_workflow_handler_registries(workflow_cls)
Expand Down Expand Up @@ -6709,6 +6728,11 @@ def _resume_business(function: Callable[[Any], Any], value: Any) -> Any:
# Its terminal position proves whether a different-process duplicate belongs.
terminal_condition_chain_current: Mapping[str, Any] | None = None
version_decisions: dict[str, int] = {}
# Whether this run raises an all-group's first recorded member failure
# (sdk-python#307). Decided once, at the run's first ordinary all-group:
# True when the group is preceded by PARALLEL_FAILURE_ORDER_MARKER or is
# scheduled now, False when it was recorded without the marker.
recorded_failure_order: bool | None = None

def _version_error(detail: str, step: _RecordedStep | None = None) -> NonDeterministicReplayError:
return NonDeterministicReplayError(
Expand Down Expand Up @@ -6795,6 +6819,67 @@ def _resolve_version(command: RecordVersionMarker) -> Any:
version_decisions.setdefault(change_id, version)
return _version_marker_result(command, version)

def _ordinary_all_group(command: Any) -> bool:
"""A remote all-group whose failure selection the run's marker decides."""
if not isinstance(command, list) or "propagate" in parallel_group_policies:
return False
try:
leaves = _parallel_leaves(command)
except (TypeError, ValueError):
return False
if any(isinstance(leaf, WaitCondition | RecordLocalActivity) for leaf in leaves) or (
prepare_local_activities and _contains_local_activity(command)
):
return False
# A group of timers alone cannot fail, so it does not decide the rule.
return any(isinstance(leaf, ScheduleActivity | StartChildWorkflow) for leaf in leaves)

def _resolve_failure_order() -> bool:
"""Read or record the run's failure-order marker at its first all-group.

A run whose first all-group was recorded without the marker was
scheduled by an earlier worker, and keeps that worker's failure
selection for its whole life, on every later task, query and update.
A run reaching its first all-group now records the marker directly
before scheduling it. No all-group of the run has produced an outcome
before that point, so no earlier task can have chosen a failure under
the other rule. Recording is opt-in (``record_parallel_failure_order``)
because workers up to 2.20.0 cannot replay a marked run. A run with a
cancellation request that may replace the group records no marker and
keeps the earlier rule.
"""
nonlocal result_cursor, authored_sequence
marker = RecordVersionMarker(
change_id=PARALLEL_FAILURE_ORDER_MARKER, version=1, min_supported=1, max_supported=1,
)
# The marker is the next authored step; an open select() loser or
# condition wait recorded earlier can still be unconsumed.
step = recorded_steps[result_cursor] if result_cursor < len(recorded_steps) else None
if (step is not None and step.shape == "version marker"
and step.details.get("change_id") == PARALLEL_FAILURE_ORDER_MARKER):
_assert_step_matches(marker, step)
if _recorded_version(step) != 1:
raise _version_error("Unsupported parallel failure order marker version.", step)
result_cursor += 1
authored_sequence = max(authored_sequence, step.workflow_sequence + 1)
return True
step = _next_unconsumed_recorded_step()
closed = any(_history_event_type(event) in {
"WorkflowCompleted", "WorkflowFailed", "WorkflowCancelled", "WorkflowTerminated",
"WorkflowContinuedAsNew", "WorkflowTimedOut",
} for event in events)
interrupting = cancellation.request is not None or any(
sequence not in consumed_scope_deliveries for sequence in committed_scopes.deliveries
)
if record_parallel_failure_order and step is None and not closed and not interrupting:
pending.append(marker)
authored_sequence += 1
return True
# Recorded without the marker, scheduled now with recording off (the
# default until every worker reads the marker), or scheduled under a
# cancellation request that may replace it: the earlier rule.
return False

def _parallel_leaf_kind(command: Any) -> str:
if isinstance(command, ScheduleActivity | RecordLocalActivity):
return "activity"
Expand Down Expand Up @@ -7836,6 +7921,11 @@ def _absent_policy_failure_decision(commands: list[Any]) -> tuple[bool, int | No
if base <= recorded_steps[index].workflow_sequence < end]
if not any(isinstance(resolved_results[index], ActivityFailed | ChildWorkflowFailed) for index in slots):
return False, None, False
if recorded_failure_order:
# History is append-only, so the first recorded failure a task
# sees is the first one every later replay sees: no decision is
# needed, whether or not the other members have settled.
return True, None, False
moved_on = [index for sequence, index in first_command_index_by_sequence.items() if sequence >= end]
if authored_close_index is not None:
moved_on.append(authored_close_index)
Expand Down Expand Up @@ -7925,6 +8015,8 @@ def _advance_selection_clock(base: int, size: int, failure: BaseException | None
first = False
_apply_due_receivers()
_validate_local_activity_policies(cmd)
if recorded_failure_order is None and _ordinary_all_group(cmd):
recorded_failure_order = _resolve_failure_order()
if isinstance(cmd, RecordVersionMarker):
next_value = _resolve_version(cmd)
continue
Expand Down
18 changes: 18 additions & 0 deletions tests/failure_order.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
"""Helpers for the run's parallel failure-order marker (sdk-python#307)."""

from __future__ import annotations

from typing import Any

from durable_workflow.workflow import PARALLEL_FAILURE_ORDER_MARKER, RecordVersionMarker

FAILURE_ORDER_MARKER = RecordVersionMarker(
change_id=PARALLEL_FAILURE_ORDER_MARKER, version=1, min_supported=1, max_supported=1,
)


def failure_order_marker_event(sequence: int = 1, **extra: Any) -> dict[str, Any]:
return {"event_type": "VersionMarkerRecorded", **extra, "payload": {
"sequence": sequence, "change_id": PARALLEL_FAILURE_ORDER_MARKER, "version": 1,
"min_supported": 1, "max_supported": 1,
}}
Loading