You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Let failed-run redrive recover ordinary workflows and document it #864
Failed-run redrive (workflow#501) continues a failed run in a new linked run. It reuses completed activity results and retries the failed step, which is how users recover from a bad deploy without re-running paid or customer-visible work. Today it refuses most real runs, and users can't find it.
FailedRunRedrivePlan refuses any history containing anything other than activity events: signals, updates, timers, side effects, version markers and children.
The Server route and the PHP, Python and Rust client methods exist, but no docs page, API reference, CLI reference or Waterline action mentions redrive. The failures guide says a failed run is terminal and lists recovery paths without it.
When a run is refused, the only remaining option is to start over, which re-executes every completed activity.
Acceptance criteria
A failed run whose completed prefix includes signals, updates, fired timers, side effects, version markers, memos or search attributes can be redriven. The successor replays those facts in order and never re-executes completed activities or side effects, or re-waits fired timers.
A run that failed in workflow code or on a structural limit can be redriven onto a corrected workflow definition. A prefix the new code cannot replay is refused with a typed reason before any new activity is scheduled.
Histories that stay unsupported, such as child workflows and local activities, are refused with a stated reason and no mutation.
Run detail and describe say whether a failed run is redrivable and why not. Waterline offers redrive on eligible runs.
Embedded PHP and published PHP, Python and Rust workers pass the same redrive conformance cases, including cold replay of the successor.
The failures guide, the Server API and CLI references, the SDK guides and the Waterline operator API document redrive. They cover what is reused, what runs again, and every refusal reason.
Additional scope from review (Slack miner cases 47836ceb and b2817412): redrive lineage is poorly exposed. The engine creates a linked successor (run_number+1, continued_from_run_id, recovery_kind redrive, and a redrive link), and Server describe returns those fields. But:
Waterline labels redrive links as plain Parent/Child diagnostic-only fallbacks.
Python's typed WorkflowRun drops continued_from_run_id and recovery_kind.
The workflow-id page lists only start and continue-as-new as ways a new run is created.
Showing redrive lineage in Waterline, the SDK run types and the workflow-id docs fits this issue's docs and Waterline acceptance.
A1b (Report redrive eligibility in run detail and describe #990, draft): run detail and describe report can_redrive and redrive_blocked_reason from the same decision redrive() applies. Corpus fixture redrive-eligibility-report fails on A1's source and passes on head.
A2 is large because each history family has different state, so I'm splitting it into drafts that each add a corpus fixture:
A2a: side effects and version markers. History-only facts keyed by step sequence. The successor copies them in order. Like reused activity completions, they carry reused_from_run_id and reused_recorded_at, and replay reads reused_recorded_at so now() after the reused step matches the source run. That adds two keys to SideEffectRecorded and VersionMarkerRecorded in the payload contract. Embedded replay paths: WorkflowExecutor, QueryStateReplayer, WorkflowFiberRunner.
A2b: fired timers. The successor records the timer as already fired, with no new timer row to wait on.
A2c: signals and updates. These need their workflow_signals / workflow_updates rows and receipts, so they come after A2a/A2b settle the copy mechanics.
A2d: workflow-code upserts (memo and search attributes at upserted_at_sequence > 0). This lifts the remaining unsupported_visibility_metadata refusal.
Service-mode SDK replay has to honor reused_recorded_at on the same events for portability. That goes into (C) once A2a fixes the keys.
The defect that matters is in Python: ctx.now() in the new run uses the redrive time, not the failed run's start (replayed_started_at). That is reproduced by cold replay and is now sdk-python#232 (P2).
PHP and Rust have no clock that reads per-step time, so they only need cold-replay proof for the reused events: sdk-php#240 and sdk-rust#164 (P3).
(B) native parity remains server#980.
Still open here beyond B and C:
A2c (signals and updates), A2d (upserts), and A3 (redriving onto a corrected definition).
Status (workflow agent): A2d is draft #1001. Workflow-code memo and search-attribute upserts now redrive, with a replay corpus counterfactual. A2c (signals and updates) starts after #877 merges, because #877 changes signal and update receipts.
A3 proposal: redrive onto a corrected definition
Acceptance: a run that failed in workflow code or on a structural limit can be redriven onto corrected code, and a prefix the new code can't replay is refused with a typed reason before anything new is scheduled.
What blocks it today:
WorkflowFailed records failed_step_sequence / failed_step_kind only for an escaped activity failure, so the plan refuses with unrecorded_failure_boundary.
FailedRunRedriveEligibility::definitionMatches() requires the recorded fingerprint, so workflow_definition_unavailable_or_changed.
Proposed design:
Record the boundary. When workflow code throws, or a structural limit fails the run, WorkflowFailed records failed_step_sequence as the step cursor at the throw, with failed_step_kind set to workflow_code or structural_limit. Older failures without these keys keep refusing with unrecorded_failure_boundary. No backfill.
Plan. For those kinds, every step before the boundary is the reused prefix, and there is no failed activity step. The successor resumes the new code at the boundary.
Opt in to a changed definition. A different fingerprint is accepted only when the request says so: allow_definition_change: true on redrive(), and the matching Server/CLI option later. It is never silent. With the option off, behavior is unchanged.
Dry-run replay before any write. Inside the redrive transaction, build the candidate successor history in memory (start events plus reused steps) and replay it with the current definition, reusing the WorkflowReplayer/QueryStateReplayer path. If replay raises a step-history or identity mismatch, or yields a new command before the boundary, refuse with definition_incompatible_with_prefix and report the first mismatching step sequence and kind. Nothing is written.
Service mode. The server can't run SDK code, so a definition change on a service-mode run is refused with definition_change_requires_replay_check until server#980 (B) defines a worker-side preflight. Embedded runs get the full feature.
Corpus: a replay fixture where corrected code redrives a workflow-code failure, plus one where an incompatible prefix is refused without writes.
I'll start A3 after A2d unless someone objects to steps 3–5.
Activity
rmcdaniel commentedon Oct 20, 2026
Additional scope from review (Slack miner cases 47836ceb and b2817412): redrive lineage is poorly exposed. The engine creates a linked successor (run_number+1, continued_from_run_id, recovery_kind redrive, and a redrive link), and Server describe returns those fields. But:
WorkflowRundrops continued_from_run_id and recovery_kind.Showing redrive lineage in Waterline, the SDK run types and the workflow-id docs fits this issue's docs and Waterline acceptance.
test: add redrive start-visibility corpus evidence (#864)
test: add redrive eligibility report corpus evidence (#864)
rmcdaniel commentedon Nov 3, 2026
Progress and A2 slicing (workflow agent, now on a new session).
can_redriveandredrive_blocked_reasonfrom the same decisionredrive()applies. Corpus fixtureredrive-eligibility-reportfails on A1's source and passes on head.A2 is large because each history family has different state, so I'm splitting it into drafts that each add a corpus fixture:
reused_from_run_idandreused_recorded_at, and replay readsreused_recorded_atsonow()after the reused step matches the source run. That adds two keys toSideEffectRecordedandVersionMarkerRecordedin the payload contract. Embedded replay paths:WorkflowExecutor,QueryStateReplayer,WorkflowFiberRunner.workflow_signals/workflow_updatesrows and receipts, so they come after A2a/A2b settle the copy mechanics.upserted_at_sequence > 0). This lifts the remainingunsupported_visibility_metadatarefusal.Service-mode SDK replay has to honor
reused_recorded_aton the same events for portability. That goes into (C) once A2a fixes the keys.test: expect the redrive decision in the describe routing contract (#864
test: give the redrive eligibility fixture its own replay history (#864)
rmcdaniel commentedon Nov 3, 2026
Progress:
reused_recorded_atonSideEffectRecorded,VersionMarkerRecordedandTimerFired, will be filed once Redrive failed runs whose completed steps include side effects or version markers #991 and Redrive failed runs whose completed steps include a fired timer #992 merge and fix those keys.test: redrive reuses one marker for a repeated patch call (#864)
rmcdaniel commentedon Nov 3, 2026
Progress (General, handing the remainder to the Workflow agent):
docs/failures-and-recovery.mdwith all translations. Released behavior goes in the main text. Redrive failed runs that have start-time memos or search attributes #985 and Report redrive eligibility in run detail and describe #990–Redrive failed runs whose completed steps include a fired timer #992 go in a "Not yet released" subsection, to fold in once the release ships. Before this PR, no docs page mentioned redrive.reused_recorded_at, despite the Redrive failed runs whose completed steps include side effects or version markers #991 note.ctx.now()in the new run uses the redrive time, not the failed run's start (replayed_started_at). That is reproduced by cold replay and is now sdk-python#232 (P2).Still open here beyond B and C:
WorkflowRundropscontinued_from_run_idandrecovery_kind.test: give the unsupported child event its own step sequence (#864)
rmcdaniel commentedon Nov 3, 2026
Status (workflow agent): A2d is draft #1001. Workflow-code memo and search-attribute upserts now redrive, with a replay corpus counterfactual. A2c (signals and updates) starts after #877 merges, because #877 changes signal and update receipts.
A3 proposal: redrive onto a corrected definition
Acceptance: a run that failed in workflow code or on a structural limit can be redriven onto corrected code, and a prefix the new code can't replay is refused with a typed reason before anything new is scheduled.
What blocks it today:
WorkflowFailedrecordsfailed_step_sequence/failed_step_kindonly for an escaped activity failure, so the plan refuses withunrecorded_failure_boundary.FailedRunRedriveEligibility::definitionMatches()requires the recorded fingerprint, soworkflow_definition_unavailable_or_changed.Proposed design:
WorkflowFailedrecordsfailed_step_sequenceas the step cursor at the throw, withfailed_step_kindset toworkflow_codeorstructural_limit. Older failures without these keys keep refusing withunrecorded_failure_boundary. No backfill.allow_definition_change: trueonredrive(), and the matching Server/CLI option later. It is never silent. With the option off, behavior is unchanged.WorkflowReplayer/QueryStateReplayerpath. If replay raises a step-history or identity mismatch, or yields a new command before the boundary, refuse withdefinition_incompatible_with_prefixand report the first mismatching step sequence and kind. Nothing is written.definition_change_requires_replay_checkuntil server#980 (B) defines a worker-side preflight. Embedded runs get the full feature.Corpus: a replay fixture where corrected code redrives a workflow-code failure, plus one where an incompatible prefix is refused without writes.
I'll start A3 after A2d unless someone objects to steps 3–5.
40 remaining items
refactor: decide redrive eligibility in one place
test: run detail and describe must report the redrive decision (durab…
fix: report redrive eligibility in run detail and describe
test: redrive must reuse side effects and version markers at their re…
test: add redrive reused-marker corpus evidence (durable-workflow#864)
test: redrive must reuse a fired timer at its recorded time (durable-…
test: add redrive reused-timer corpus evidence (durable-workflow#864)
test: update the redrive plan test for supported side effects (durabl…
fix: redrive runs whose workflow code upserted memos or search attrib…
fix: redrive runs whose completed steps include an applied signal wait (
feat: redrive a workflow-code failure onto a corrected definition (du…
fix: refuse service redrive of reused steps its SDKs can't replay yet (…
fix: redrive runs whose workflow applied an update before the failure (…
fix: report a redrive link as typed lineage (durable-workflow#864)