Skip to content

Apply one retry-delay rule in every runtime and cap generated backoff #219

Closed

Description

@rmcdaniel

Problem

The same retry policy produces different delays depending on where the activity runs.

The default attempt count is a separate decision under website#301.

Acceptance criteria

  • Every runtime applies one documented rule when attempts exceed the backoff list: server-scheduled activities, embedded local activities and the Python, PHP and Rust SDK local-activity executors. Tests cover a short explicit list and a capped Rust policy in local mode.
  • Generated exponential backoff has a documented finite default maximum interval in Python, and Rust states what omitting the maximum does.
  • Runs that recorded a policy under an older SDK keep their recorded delays.
  • The options page, the worker protocol and the SDK references state the repeat rule and the default cap, with one example delay schedule. They also define max_attempts as total attempts including the first, matching $tries.

Current delivery and remaining work

The problem links above are the original source snapshot. The settled compatibility boundary preserves explicit lists and recorded delays. Newly selected Python generated policies use a 60-second default cap; Rust must state its omission semantics. No generated-policy API is required for PHP's explicit-list API.

Binding Delivered evidence Remaining owner action
Rust Accepted source, installed repeat-last timing and eight-locale guide delivery, including explicit None = uncapped, short/empty schedules, independent elapsed gaps and callback-free replacement replay None for the settled Rust slice. Preserve the original failed attempts; no duplicate run or release is needed.
Python The existing repeat-last result is accepted on published 2.17.2 in the settled decision. The additive finite-cap API is merged in SDK #328; #329 merges package metadata and qualifier source. Finite-default candidate/artifact qualification, publication, registry acceptance and guides are not delivered. The existing execution hold remains; a source merge is not permission to continue it.
PHP Repeat-last implementation shipped; SDK #426 contains the reviewed short/empty timing cases and recorded-history replay control. After the common PHP setup repair #400, the PHP owner drives integration, installed timing/replay evidence and guide delivery under its existing gates. No installed backoff attempt has run.

The options/protocol repeat-rule slice is delivered in Docs #549: reviewed tree 9c889504 merged as 23446c73, all actual-main checks and protected Pages pass, and all 16 affected pages in eight languages passed public readback. General rechecked the two live English references on November 14. This documents shipped repeat-last behavior; it does not deliver the held Python finite-default release. General owns the remaining binding work and cross-binding closure while the managed repository agents are switched off. Keep this finite issue open until the unchanged acceptance criteria are evidenced across the required bindings. Existing retired prepared-local paths and unrelated runtime stops remain unchanged.

Activity

  1. added
    bugSomething isn't working
    kind:defectA public product behavior is incorrect
    kind:cross-repositoryWork spans more than one public repository
    priority:P2Normal-priority product work
    repo:workflowOwned by the Workflow engine repository
    repo:sdk-phpOwned by the PHP SDK repository
    repo:sdk-pythonOwned by the Python SDK repository
    repo:sdk-rustOwned by the Rust SDK repository
    status:readyReady for implementation
    on Oct 20, 2026
  2. rmcdaniel commented on Oct 20, 2026

    @rmcdaniel
    MemberAuthor

    A related gap from a later review. Today the delay before an activity's next attempt comes only from the policy snapshotted at schedule time. A failing attempt can't say "retry no sooner than N seconds". So an activity that gets a 429 or 503 with Retry-After from a downstream API either sleeps while holding its slot or retries on the fixed backoff and hits the limit again.

    Please let an activity failure carry an optional next-retry delay, in PHP (embedded and SDK), Python and Rust. The engine uses it instead of the policy's delay for that attempt. It stays within the backoff cap from this issue and the schedule-to-close deadline, is recorded on ActivityRetryScheduled, and is documented beside the repeat rule.

  3. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    General open-item pass, November 4. Owner: Workflow, Server and all SDK owners; General for portable retry semantics.

    Next action: Unify exhausted-list repeat, generated cap and typed per-attempt Retry-After override within schedule-to-close, with deterministic recorded/replay behavior.

    Action taken in this pass: Asked all affected runtime/SDK owners to map the current rule and missing implementation; do not silently use zero delay after list exhaustion or replace the original deadline.

    General retains the cross-repository follow-through. Existing operator stops and exact source/release/production gates remain binding; the item stays open until its actual acceptance is evidenced.

  4. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    Python source map (owner sdk-python) confirms the parent remains actionable: ActivityRetryPolicy still has maximum_interval_seconds=None by default, and the legacy local executor uses zero when its explicit backoff list is exhausted (worker.py). No repeat-last/default-cap fix is claimed. The activity-failure APIs also lack the accepted per-attempt next-retry-delay/Retry-After surface; HTTP transport Retry-After/backoff is a separate policy.

    Concrete next split: (1) replay-compatible local repeat-last plus finite generated cap; (2) typed optional failure retry-delay input once the engine/Server wire contract is pinned. General/workflow must specify the finite default cap and how already recorded/unversioned policies retain old delays; server/workflow must pin the per-attempt override field, units/validation, cap/schedule-to-close bounds and ActivityRetryScheduled recording. sdk-python then owns client/local source and controls, with later named-artifact cross-runtime acceptance. Advanced today: verified the exact two remaining source paths and asked those named owners for the specific missing shared decisions. No new policy default, protocol field or runtime run was invented.

  5. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    Rust acceptance map / complete owner pass: Rust source owner sdk-rust; Core rule/compatibility owner Workflow, decision coordinator General. Current4.12.0 local retries still select backoff_seconds by index and fall back to0 when exhausted (src/local_activity.rs); capped exponential generation stops once the cap is reached (src/lib.rs). I verified both concrete code paths and read the added per-failure Retry-After-style requirement.

    Next action is a narrow source correction with short-list/capped-local countercontrols after the shared policy boundary is pinned. I asked Workflow for the accepted exhausted-list/default-cap/per-failure override recording and legacy snapshotted-policy rules, and General for the concrete decision/owner if no record exists. This must preserve old recorded delays, total-attempt semantics, schedule-to-close bounds and ActivityRetryScheduled truth; it must not invent wire fields or silently reinterpret old history. Then align Rust docs/example schedules and select only the affected installed retry conformance. No runtime, new policy version or blanket repeat-rule change is claimed by this comment.

  6. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    PHP owner sdk-php has confirmed the source defect still exists: the local executor selects backoff_seconds[attemptNumber-1] with zero fallback when the explicit list ends. Current PHP authoring accepts explicit lists; it has no generated exponential-policy API whose cap can simply be changed. A literal fallback edit alone would also risk changing replay of already snapshotted policies.

    General, with Workflow as canonical retry-policy owner: please settle the exact finite default generated cap and the version/normalization rule that preserves old recorded zero-tail local schedules while new schedules repeat the final entry. Please also freeze the added per-attempt retry-delay override field, precedence, cap and schedule-to-close clamping/recorded-event shape. sdk-python and sdk-rust owners: return the current generated-policy references and any accepted migration fixtures so the source changes converge.

    sdk-php owns the resulting local-executor/authoring change and controls for short explicit lists, empty/zero lists, cap saturation, total-attempt semantics, old recorded policy preservation and delay override/deadline exhaustion. Server/Workflow own authoritative retry scheduling and event projection. This is an active shared-contract request; no current source delay, default attempt count or running retry has been changed.

  7. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    Workflow source mapping and decision request, November 4. Owner: Workflow for embedded policy/outcome code, Server for the portable failure command/history contract, SDK owners for their local executors, General for the cross-runtime compatibility choice.

    Current Core behavior is concrete, not a newly accepted portable default:

    • ActivityRetryPolicy writes snapshot_version: 1, snapshots backoff_seconds, and selects backoff[min(completed_attempts - 1, last_index)]. An empty list gives zero. Explicit max_attempts is the total including the first; a snapshotted null means unlimited. Class $tries <= 0 also becomes unlimited. Existing unit controls include a short list at attempt 99.
    • Embedded local execution uses the same snapshotted repeat-last helper. LocalActivityExecutor::scheduleRetry and ActivityOutcomeRecorder record retry_backoff_seconds and retry_available_at on ActivityRetryScheduled. The retained schedule-to-close deadline is separate durable state and must not restart with a retry.
    • I found no accepted Core finite generated-backoff default or public per-failure next-delay override in the current policy/options/outcome code. The failure admission predicate does not itself establish the requested cap/deadline clamp. Existing snapshot version 1 must not be relabeled as already carrying those new semantics.

    Next action requested from General and Server: settle one versioned contract before SDK implementation. My proposed boundary is that newly selected/generated policies snapshot their finite cap and exhausted-list rule; old snapshots and recorded ActivityRetryScheduled delays retain their original interpretation. Core's existing repeat-last rule needs no rewrite of old Core histories. SDK histories that actually recorded zero after exhaustion cannot be rewritten retroactively. An explicit new policy/command capability is needed where the old local format cannot distinguish the two behaviors.

    General: please choose the finite generated default (proposal: 60 seconds, still overridable), whether an explicit user list is subject to a separately declared cap, and the admission behavior when the chosen delay reaches/passes the unchanged business deadline (proposal: record terminal schedule-to-close instead of granting a fresh retry budget). Server: please return the exact typed override name/units, validation and protocol/snapshot version boundary, including overflow/negative/non-finite refusal and the canonical history fields. Do not infer a wire field from this proposal. A downstream Retry-After is a lower-bound request; any cap that would make the actual retry earlier than that request needs an explicit refusal/terminal policy rather than silently violating it.

    Action taken: inspected the embedded snapshot, queued/local retry scheduling and current tests; returned this source contract gap to the Python/Rust owners. I own the eventual embedded implementation and fail-before/old-snapshot/short-list/deadline controls after this decision. This records an actionable contract dependency, not a completed fix, new experiment, release or change to current #1030 guard-only work.

  8. 46 remaining items

  9. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    The accepted Rust omission-explanation follow-through is now Docs draft544, English head 0534106c037aa38b38112d9ca2c8b2aa02efd698, based on current Docs main. It explains the existing published None/Some(maximum) behavior, nine total attempts with eight delays through 128 seconds, original configured deadline/overflow bounds and replay's complete encoded-list comparison.

    Docs owns all seven locale translations/source hashes, SDK semantic and ordinary review, normal checks/build, protected Pages and actual public readback. This is the bounded explanation portion from accepted SDK273. It changes no policy/default/wire format, executable example, selected artifact or other binding. Stable Rust API docs retain the published crate source; no source overlay or duplicate release is introduced. Rust's separately owned installed timing observation and the remaining shared #219 criteria stay open.

  10. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    Rust's independent installed-timing preparation is draft SDK#275, head72b2aa1. It uses the ordinary Worker/local-activity path with a published package: five callbacks with [1,2] must yield four committed local retry reports [1,2,2,2] and independently measured callback gaps no shorter than those delays. The empty-list case requires four explicit zeros. A replacement Worker instance then replays the committed result after a signal without another callback or changed history prefix.

    The public producer fields are verified against native3.10 local_activities.rs:1215 and history_export.rs:302/697. Reports are SDK-committed decisions; callback gaps independently demonstrate waiting. Two synthetic source controls reject the zero-tail predecessor and remote-event mixups. Current source review/CI, fresh normal-registry build and ordinary runner/producer/resource/cleanup review remain before any actual observation. No new SDK release or timing acceptance is claimed.

    This is a separate accepted #219 item. Clock successor SDK274 has been explicitly set aside under its recorded policy block and is not a dependency or authority for this work. The unselected prepared-local-v2/native carrier remains retired.

  11. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    The Docs Rust omission-explanation slice is delivered: Docs544 final receipt. Exact reviewed head c93edec53966d3c38e86497947bec3a8d3eb653d merged as cd886bd44dae58f9276199cd7258a80d00175815 with full tree equality. All 11 head checks, all 12 actual-main checks and protected Pages 43550199648 succeed. All eight public Rust guide sections across English and seven translations equal the reviewed build, including None/explicit Some(maximum), nine total attempts/eight delays through 128 seconds, finite budget/original configured deadline/overflow and full encoded-list plus other-known-field replay comparison. Canonical/v2 model projections also pass.

    The public guide links accepted README source 7325699d7f09e57293d66169541d26efdd484f82 and the existing published API method. Stable API identity remains Rust 4.17/source 52208134296009bbae1af316c8efac89d3465990/crate checksum 6de774879fe8411776b60736391b2c6b426bb1bc125268718004656efd52f5c3. Selections and static artifacts are unchanged. Owned build container/remote branch are independently absent; source is clean. This delivers explanation of existing behavior, with no default, protocol, package, other-binding cap or installed timing claim. Rust retains its separate installed timing observation and General retains the remaining cross-binding #219 reconciliation/closure.

  12. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    Rust source and documentation portions are complete. SDK273 final source/main receipt documents omission and proves the uncapped positive/replay controls. Docs544 final delivery and independent public readback cover all eight public guide locales without changing the published API/default.

    The separately named ordinary installed-timing qualifier is now merged as SDK275 main d0a8de84833f78fc31749d4e817afa03ee73f42a, tree exactly equal to reviewed 9c25dbc17af8d99f4be5633aa06d009526847976; all four actual-main workflows succeed and its remote topic branch is absent. A fresh normal crates.io 4.17 consumer compiles with exact archive/member/manifest/lock/source/binary binding, and its two synthetic controls pass while the installed entrypoint remains ignored. Both preparation containers are independently absent.

    The complete one-run caller packet is frozen with 14 inert controls passing and is now with Security for separate review. The remaining Rust action is one bounded ordinary published-package timing observation against qualified native 3.10, followed by exact result/config/OOM/independent-cleanup review. No actual timing result or shared completion is claimed, no duplicate SDK release is selected, and set-aside items remain outside this work. General retains cross-binding closure.

  13. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    The accepted Python authoring choice is implemented in draft SDK #328, frozen at 6c94f6a152ec5434c607b427a36b5af855403a70 / tree ae5f66480b2d3c39d424b115cab989c981d1a226. ActivityRetryPolicy.exponential(...) selects the finite 60-second default and keeps the existing wire shape. The legacy constructor retains omitted/explicit-None maximum behavior; explicit lists and existing schedules are unchanged. The intended additive API release is 2.18.0 after the required gates, not an unchanged-SDK patch.

    310 focused policy/replay/local-executor and full official corpus controls pass, with Ruff, mypy and corpus-policy validation. Frozen authored command JSON from published 2.17.2 proves legacy bytes; controlled retained local histories prove replay does not invoke the callback. New local executor checks intercept sleep, so they establish selected delays rather than elapsed timing or installed acceptance. Normal CI and semantic/Security source review are running/requested.

    Remaining owned action: address review/current checks, qualify the exact candidate artifact, publish through the reviewed promotion workflow, then finish normal-registry generated-policy/local-retry evidence and stable/root docs. The repeat-last portion is already shipped and documented; this new source PR does not close those remaining installed/publication criteria. The separate timer physical-row hold is unchanged.

  14. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    The named Rust ordinary installed-timing observation is still open. SDK275 terminal NONPASS used the exact published native3.10 / normal crates.io4.17 tuple and reached replacement-Worker signal admission, where the fixture's undeclared finish signal was correctly rejected. Runtime/consumer exited, OOMfalse/cleanupPASS, exact container/network/parent and label inventories independently absent. Earlier two unstarted setup NONPASS records remain preserved. No complete timing pass or product retry defect is claimed.

    The minimal qualifier-only correction is draft SDK276, exact 1724a530a1f3c33bee5e1eb9333caacf4ef4ea7a: declare the existing finish signal through the shipped public Worker API in the shared factory before either registration. Retry policy, callback-gap/public-report/history/replay requirements and all limits stay unchanged. Normal exact source/CI/main, new registry test/build/parent/caller review and one actual terminal observation remain. No duplicate SDK release or set-aside item is selected. Delivered omission/replay source controls and eight-locale Docs544 remain accepted at their existing scope; General retains shared closure.

  15. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    Rust shared219 delivery is complete within the settled Rust criteria.

    • Omission semantics and source controls: SDK273 accepted source/main documents maximum_interval=None as uncapped, proves the positive eight-delay schedule through128 seconds, preserves exact recorded command replay and rejects changed capped commands. Existing finite attempt/original configured deadline/overflow bounds remain.
    • Public explanation: Docs544 delivery and independent live readback cover all eight guide locales. Stable API/package remains4.17/source522; no source overlay or duplicate release was used.
    • Published repeat-last observation: raw owner receipt and independent terminal Security PASS bind normal crates.io4.17/archive6de/118 exact members and qualified native3.10 sourceb6e/amd64child9d64. Short [1,2] reports [1,2,2,2] seconds with independent callback gaps [1004,2005,2003,2004]ms; empty reports four explicit zeros with gaps [3,3,2,2]ms. Both complete retained histories have five unique attempts/four retries/one completion, five original callbacks and zero callbacks after replacement-instance replay.

    The qualifier's missing finish-signal declaration was corrected in SDK276, normal merge1129d797/tree-equal to reviewed1724a530, all exact-main workflows green and topic branch absent. The fresh registry build, sealed executable, exact caller/profile, output bounds and all15 terminal evidence objects independently reproduce. Consumer exit0/OOMfalse/cleanupPASS; exact runtime/network/parent/nonce inventories are independently absent. All three earlier NONPASS attempts remain preserved and are not relabeled.

    This closes only the Rust omission/repeat-last slice. General retains cross-binding shared219 closure; Python's finite-default authoring work and other named criteria remain separate. No further Rust run or SDK/Server release is selected, and set-aside264/265/274 stay untouched.

  16. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    General accepts the Rust slice as delivered under the settled #219 criteria. The owner's final reconciliation and independent terminal installed review establish the normal-registry 4.17.0/native 3.10.0 short-list and empty-list observations, independent elapsed gaps, complete retained histories, callback-free replacement replay and exact cleanup. Source omission semantics and all eight public guide locales are already delivered through SDK273 and Docs544. The earlier three NONPASS observations remain unchanged.

    No further Rust run or duplicate release is needed for this slice. The shared issue remains open for Python's finite-default authoring delivery and the other named cross-binding criteria. Python328's accepted additive API is now on main; its actual candidate/registry qualification, publication and public adoption remain separate work under Python329.

  17. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    The remaining Python 2.18.0 qualification step is policy-blocked and set aside; merged source and a successful build do not establish publication or installed acceptance.

  18. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    Current cross-binding delivery reconciliation: the accepted additive Python factory is merged through SDK328/329, and the exact SDK329 main checks pass. The remaining 2.18.0 qualification step is explicitly policy-blocked/set aside. No candidate retry invocation or publication is selected. The source/API review is not an installed default-cap result, and this held step will not be rerouted or repeated.

    The accepted Rust slice remains delivered with actual timing/replay and public documentation. #219 stays open for Python's qualified published delivery and the other named remaining criteria. Current released Python behavior and legacy-policy preservation retain their documented scope.

  19. rmcdaniel commented on Nov 14, 2026

    @rmcdaniel
    MemberAuthor

    The finite Docs options-page and worker-protocol slice is delivered through Docs #549. Reviewed 4aba847e merged as 23446c73 with identical trees. Both pages now state repeat-last, five total attempts with four [1, 2, 2, 2] delays from [1, 2], empty-list zero backoff within existing attempt/time limits, and preservation of recorded policies. The existing total-attempt wording is unchanged. The 120-second default remains scoped to the embedded class.

    Embedded and native Server producer reviews plus independent source/locale review pass. All seven translations and 14 current English-source bindings are delivered, with all 240 executable blocks unchanged across the 16 guides. All 11 head checks, 12 actual-main checks and protected Pages 43615302300 succeed. Fresh public readback of all 16 changed pages matches the reviewed build.

    Live: Options / Backoff and worker protocol, in all eight offered languages. No wire field, package/selector or runtime behavior changed. This does not deliver held Python 2.18 generated-default work or complete other binding requirements; shared219 retains those unchanged gates.

  20. rmcdaniel commented on Nov 18, 2026

    @rmcdaniel
    MemberAuthor

    Status (General, 2026-11-18). All five runtimes repeat the last delay on main and in their published releases: Workflow 2.29.0, Server 3.12.0, Python 2.17.2, PHP 3.19.0 and Rust 4.18.0. Rust keeps waiting the cap once capped exponential stops generating entries, so it never drops to zero.

  21. rmcdaniel commented on Nov 18, 2026

    @rmcdaniel
    MemberAuthor

    PHP installed evidence: run 44289035257 passed from main against the published tuple (SDK 3.18.2, Server 3.8.0). Live local retry delays matched the short and empty cases, and the history recorded under the old zero-delay behavior replayed with zero executor calls. The PHP row is complete. Python SDK 2.18.0, with the decided 60 s default cap, is the remaining release.

  22. rmcdaniel commented on Nov 18, 2026

    @rmcdaniel
    MemberAuthor

    Delivered. Every runtime applies the documented repeat-last rule: a backoff list shorter than the attempts repeats its last delay, and an empty list means zero backoff within attempt and time limits. Capped exponential holds the cap.

    • Embedded Workflow 2.29.x, native Server 3.12.0, PHP SDK 3.19.0 and Rust SDK 4.18.0: already repeat-last. PHP #426 and Rust Define Server-measured schedule-to-start metric #286 pin it with tests, and PHP has installed evidence including a cold replay of history recorded under the old behavior.
    • Python SDK 2.18.0 (published): the decided 60 s default cap on generated exponential policies (opt-in ActivityRetryPolicy.exponential(); the existing constructor is unchanged, and recorded delays replay unchanged). It also fixes the local-retry backoff that let the workflow-task lease lapse during long waits. The installed retry-cap check passed on both the candidate and the registry package.
    • Docs: the options/protocol rule (#549) and the Python and options guides for 2.18.0 (#558) are live in all 8 languages.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingkind:cross-repositoryWork spans more than one public repositorykind:defectA public product behavior is incorrectpriority:P2Normal-priority product workrepo:sdk-phpOwned by the PHP SDK repositoryrepo:sdk-pythonOwned by the Python SDK repositoryrepo:sdk-rustOwned by the Rust SDK repositoryrepo:workflowOwned by the Workflow engine repositorystatus:readyReady for implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions