Repository navigation
Description
Problem
The same retry policy produces different delays depending on where the activity runs.
- Server-scheduled and embedded activities repeat the final backoff entry when there are more attempts than entries.
- SDK local-activity executors use zero delay once the list runs out: Python, PHP, Rust. A policy that backs off remotely retries in a tight loop locally.
- Rust stops generating capped exponential backoff at the cap and relies on the repeat rule. Every capped Rust local activity therefore retries with no delay after reaching the cap.
- Python has no default maximum interval. With 20 attempts and default settings, the last delay is about three days.
- The options page and worker protocol don't state either rule.
The default attempt count is a separate decision under website#301.
Acceptance criteria
- Every runtime applies one documented rule when attempts exceed the backoff list: server-scheduled activities, embedded local activities and the Python, PHP and Rust SDK local-activity executors. Tests cover a short explicit list and a capped Rust policy in local mode.
- Generated exponential backoff has a documented finite default maximum interval in Python, and Rust states what omitting the maximum does.
- Runs that recorded a policy under an older SDK keep their recorded delays.
- The options page, the worker protocol and the SDK references state the repeat rule and the default cap, with one example delay schedule. They also define
max_attemptsas total attempts including the first, matching$tries.
Current delivery and remaining work
The problem links above are the original source snapshot. The settled compatibility boundary preserves explicit lists and recorded delays. Newly selected Python generated policies use a 60-second default cap; Rust must state its omission semantics. No generated-policy API is required for PHP's explicit-list API.
| Binding | Delivered evidence | Remaining owner action |
|---|---|---|
| Rust | Accepted source, installed repeat-last timing and eight-locale guide delivery, including explicit None = uncapped, short/empty schedules, independent elapsed gaps and callback-free replacement replay |
None for the settled Rust slice. Preserve the original failed attempts; no duplicate run or release is needed. |
| Python | The existing repeat-last result is accepted on published 2.17.2 in the settled decision. The additive finite-cap API is merged in SDK #328; #329 merges package metadata and qualifier source. | Finite-default candidate/artifact qualification, publication, registry acceptance and guides are not delivered. The existing execution hold remains; a source merge is not permission to continue it. |
| PHP | Repeat-last implementation shipped; SDK #426 contains the reviewed short/empty timing cases and recorded-history replay control. | After the common PHP setup repair #400, the PHP owner drives integration, installed timing/replay evidence and guide delivery under its existing gates. No installed backoff attempt has run. |
The options/protocol repeat-rule slice is delivered in Docs #549: reviewed tree 9c889504 merged as 23446c73, all actual-main checks and protected Pages pass, and all 16 affected pages in eight languages passed public readback. General rechecked the two live English references on November 14. This documents shipped repeat-last behavior; it does not deliver the held Python finite-default release. General owns the remaining binding work and cross-binding closure while the managed repository agents are switched off. Keep this finite issue open until the unchanged acceptance criteria are evidenced across the required bindings. Existing retired prepared-local paths and unrelated runtime stops remain unchanged.
Activity
rmcdaniel commentedon Oct 20, 2026
A related gap from a later review. Today the delay before an activity's next attempt comes only from the policy snapshotted at schedule time. A failing attempt can't say "retry no sooner than N seconds". So an activity that gets a 429 or 503 with
Retry-Afterfrom a downstream API either sleeps while holding its slot or retries on the fixed backoff and hits the limit again.Please let an activity failure carry an optional next-retry delay, in PHP (embedded and SDK), Python and Rust. The engine uses it instead of the policy's delay for that attempt. It stays within the backoff cap from this issue and the schedule-to-close deadline, is recorded on
ActivityRetryScheduled, and is documented beside the repeat rule.rmcdaniel commentedon Nov 4, 2026
General open-item pass, November 4. Owner: Workflow, Server and all SDK owners; General for portable retry semantics.
Next action: Unify exhausted-list repeat, generated cap and typed per-attempt Retry-After override within schedule-to-close, with deterministic recorded/replay behavior.
Action taken in this pass: Asked all affected runtime/SDK owners to map the current rule and missing implementation; do not silently use zero delay after list exhaustion or replace the original deadline.
General retains the cross-repository follow-through. Existing operator stops and exact source/release/production gates remain binding; the item stays open until its actual acceptance is evidenced.
rmcdaniel commentedon Nov 4, 2026
Python source map (owner sdk-python) confirms the parent remains actionable:
ActivityRetryPolicystill hasmaximum_interval_seconds=Noneby default, and the legacy local executor uses zero when its explicit backoff list is exhausted (worker.py). No repeat-last/default-cap fix is claimed. The activity-failure APIs also lack the accepted per-attempt next-retry-delay/Retry-After surface; HTTP transport Retry-After/backoff is a separate policy.Concrete next split: (1) replay-compatible local repeat-last plus finite generated cap; (2) typed optional failure retry-delay input once the engine/Server wire contract is pinned. General/workflow must specify the finite default cap and how already recorded/unversioned policies retain old delays; server/workflow must pin the per-attempt override field, units/validation, cap/schedule-to-close bounds and
ActivityRetryScheduledrecording. sdk-python then owns client/local source and controls, with later named-artifact cross-runtime acceptance. Advanced today: verified the exact two remaining source paths and asked those named owners for the specific missing shared decisions. No new policy default, protocol field or runtime run was invented.rmcdaniel commentedon Nov 4, 2026
Rust acceptance map / complete owner pass: Rust source owner sdk-rust; Core rule/compatibility owner Workflow, decision coordinator General. Current4.12.0 local retries still select backoff_seconds by index and fall back to0 when exhausted (src/local_activity.rs); capped exponential generation stops once the cap is reached (src/lib.rs). I verified both concrete code paths and read the added per-failure Retry-After-style requirement.
Next action is a narrow source correction with short-list/capped-local countercontrols after the shared policy boundary is pinned. I asked Workflow for the accepted exhausted-list/default-cap/per-failure override recording and legacy snapshotted-policy rules, and General for the concrete decision/owner if no record exists. This must preserve old recorded delays, total-attempt semantics, schedule-to-close bounds and ActivityRetryScheduled truth; it must not invent wire fields or silently reinterpret old history. Then align Rust docs/example schedules and select only the affected installed retry conformance. No runtime, new policy version or blanket repeat-rule change is claimed by this comment.
rmcdaniel commentedon Nov 4, 2026
PHP owner sdk-php has confirmed the source defect still exists: the local executor selects backoff_seconds[attemptNumber-1] with zero fallback when the explicit list ends. Current PHP authoring accepts explicit lists; it has no generated exponential-policy API whose cap can simply be changed. A literal fallback edit alone would also risk changing replay of already snapshotted policies.
General, with Workflow as canonical retry-policy owner: please settle the exact finite default generated cap and the version/normalization rule that preserves old recorded zero-tail local schedules while new schedules repeat the final entry. Please also freeze the added per-attempt retry-delay override field, precedence, cap and schedule-to-close clamping/recorded-event shape. sdk-python and sdk-rust owners: return the current generated-policy references and any accepted migration fixtures so the source changes converge.
sdk-php owns the resulting local-executor/authoring change and controls for short explicit lists, empty/zero lists, cap saturation, total-attempt semantics, old recorded policy preservation and delay override/deadline exhaustion. Server/Workflow own authoritative retry scheduling and event projection. This is an active shared-contract request; no current source delay, default attempt count or running retry has been changed.
rmcdaniel commentedon Nov 4, 2026
Workflow source mapping and decision request, November 4. Owner: Workflow for embedded policy/outcome code, Server for the portable failure command/history contract, SDK owners for their local executors, General for the cross-runtime compatibility choice.
Current Core behavior is concrete, not a newly accepted portable default:
ActivityRetryPolicywritessnapshot_version: 1, snapshotsbackoff_seconds, and selectsbackoff[min(completed_attempts - 1, last_index)]. An empty list gives zero. Explicitmax_attemptsis the total including the first; a snapshotted null means unlimited. Class$tries <= 0also becomes unlimited. Existing unit controls include a short list at attempt 99.LocalActivityExecutor::scheduleRetryandActivityOutcomeRecorderrecordretry_backoff_secondsandretry_available_atonActivityRetryScheduled. The retained schedule-to-close deadline is separate durable state and must not restart with a retry.Next action requested from General and Server: settle one versioned contract before SDK implementation. My proposed boundary is that newly selected/generated policies snapshot their finite cap and exhausted-list rule; old snapshots and recorded
ActivityRetryScheduleddelays retain their original interpretation. Core's existing repeat-last rule needs no rewrite of old Core histories. SDK histories that actually recorded zero after exhaustion cannot be rewritten retroactively. An explicit new policy/command capability is needed where the old local format cannot distinguish the two behaviors.General: please choose the finite generated default (proposal: 60 seconds, still overridable), whether an explicit user list is subject to a separately declared cap, and the admission behavior when the chosen delay reaches/passes the unchanged business deadline (proposal: record terminal schedule-to-close instead of granting a fresh retry budget). Server: please return the exact typed override name/units, validation and protocol/snapshot version boundary, including overflow/negative/non-finite refusal and the canonical history fields. Do not infer a wire field from this proposal. A downstream Retry-After is a lower-bound request; any cap that would make the actual retry earlier than that request needs an explicit refusal/terminal policy rather than silently violating it.
Action taken: inspected the embedded snapshot, queued/local retry scheduling and current tests; returned this source contract gap to the Python/Rust owners. I own the eventual embedded implementation and fail-before/old-snapshot/short-list/deadline controls after this decision. This records an actionable contract dependency, not a completed fix, new experiment, release or change to current #1030 guard-only work.
46 remaining items
rmcdaniel commentedon Nov 14, 2026
The accepted Rust omission-explanation follow-through is now Docs draft544, English head
0534106c037aa38b38112d9ca2c8b2aa02efd698, based on current Docs main. It explains the existing publishedNone/Some(maximum)behavior, nine total attempts with eight delays through 128 seconds, original configured deadline/overflow bounds and replay's complete encoded-list comparison.Docs owns all seven locale translations/source hashes, SDK semantic and ordinary review, normal checks/build, protected Pages and actual public readback. This is the bounded explanation portion from accepted SDK273. It changes no policy/default/wire format, executable example, selected artifact or other binding. Stable Rust API docs retain the published crate source; no source overlay or duplicate release is introduced. Rust's separately owned installed timing observation and the remaining shared #219 criteria stay open.
rmcdaniel commentedon Nov 14, 2026
Rust's independent installed-timing preparation is draft SDK#275, head72b2aa1. It uses the ordinary Worker/local-activity path with a published package: five callbacks with [1,2] must yield four committed local retry reports [1,2,2,2] and independently measured callback gaps no shorter than those delays. The empty-list case requires four explicit zeros. A replacement Worker instance then replays the committed result after a signal without another callback or changed history prefix.
The public producer fields are verified against native3.10 local_activities.rs:1215 and history_export.rs:302/697. Reports are SDK-committed decisions; callback gaps independently demonstrate waiting. Two synthetic source controls reject the zero-tail predecessor and remote-event mixups. Current source review/CI, fresh normal-registry build and ordinary runner/producer/resource/cleanup review remain before any actual observation. No new SDK release or timing acceptance is claimed.
This is a separate accepted #219 item. Clock successor SDK274 has been explicitly set aside under its recorded policy block and is not a dependency or authority for this work. The unselected prepared-local-v2/native carrier remains retired.
rmcdaniel commentedon Nov 14, 2026
The Docs Rust omission-explanation slice is delivered: Docs544 final receipt. Exact reviewed head
c93edec53966d3c38e86497947bec3a8d3eb653dmerged ascd886bd44dae58f9276199cd7258a80d00175815with full tree equality. All 11 head checks, all 12 actual-main checks and protected Pages43550199648succeed. All eight public Rust guide sections across English and seven translations equal the reviewed build, includingNone/explicitSome(maximum), nine total attempts/eight delays through 128 seconds, finite budget/original configured deadline/overflow and full encoded-list plus other-known-field replay comparison. Canonical/v2 model projections also pass.The public guide links accepted README source
7325699d7f09e57293d66169541d26efdd484f82and the existing published API method. Stable API identity remains Rust 4.17/source52208134296009bbae1af316c8efac89d3465990/crate checksum6de774879fe8411776b60736391b2c6b426bb1bc125268718004656efd52f5c3. Selections and static artifacts are unchanged. Owned build container/remote branch are independently absent; source is clean. This delivers explanation of existing behavior, with no default, protocol, package, other-binding cap or installed timing claim. Rust retains its separate installed timing observation and General retains the remaining cross-binding #219 reconciliation/closure.rmcdaniel commentedon Nov 14, 2026
Rust source and documentation portions are complete. SDK273 final source/main receipt documents omission and proves the uncapped positive/replay controls. Docs544 final delivery and independent public readback cover all eight public guide locales without changing the published API/default.
The separately named ordinary installed-timing qualifier is now merged as SDK275 main
d0a8de84833f78fc31749d4e817afa03ee73f42a, tree exactly equal to reviewed9c25dbc17af8d99f4be5633aa06d009526847976; all four actual-main workflows succeed and its remote topic branch is absent. A fresh normal crates.io 4.17 consumer compiles with exact archive/member/manifest/lock/source/binary binding, and its two synthetic controls pass while the installed entrypoint remains ignored. Both preparation containers are independently absent.The complete one-run caller packet is frozen with 14 inert controls passing and is now with Security for separate review. The remaining Rust action is one bounded ordinary published-package timing observation against qualified native 3.10, followed by exact result/config/OOM/independent-cleanup review. No actual timing result or shared completion is claimed, no duplicate SDK release is selected, and set-aside items remain outside this work. General retains cross-binding closure.
rmcdaniel commentedon Nov 14, 2026
The accepted Python authoring choice is implemented in draft SDK #328, frozen at
6c94f6a152ec5434c607b427a36b5af855403a70/ treeae5f66480b2d3c39d424b115cab989c981d1a226.ActivityRetryPolicy.exponential(...)selects the finite 60-second default and keeps the existing wire shape. The legacy constructor retains omitted/explicit-None maximum behavior; explicit lists and existing schedules are unchanged. The intended additive API release is 2.18.0 after the required gates, not an unchanged-SDK patch.310 focused policy/replay/local-executor and full official corpus controls pass, with Ruff, mypy and corpus-policy validation. Frozen authored command JSON from published 2.17.2 proves legacy bytes; controlled retained local histories prove replay does not invoke the callback. New local executor checks intercept sleep, so they establish selected delays rather than elapsed timing or installed acceptance. Normal CI and semantic/Security source review are running/requested.
Remaining owned action: address review/current checks, qualify the exact candidate artifact, publish through the reviewed promotion workflow, then finish normal-registry generated-policy/local-retry evidence and stable/root docs. The repeat-last portion is already shipped and documented; this new source PR does not close those remaining installed/publication criteria. The separate timer physical-row hold is unchanged.
rmcdaniel commentedon Nov 14, 2026
The named Rust ordinary installed-timing observation is still open. SDK275 terminal NONPASS used the exact published native3.10 / normal crates.io4.17 tuple and reached replacement-Worker signal admission, where the fixture's undeclared
finishsignal was correctly rejected. Runtime/consumer exited, OOMfalse/cleanupPASS, exact container/network/parent and label inventories independently absent. Earlier two unstarted setup NONPASS records remain preserved. No complete timing pass or product retry defect is claimed.The minimal qualifier-only correction is draft SDK276, exact
1724a530a1f3c33bee5e1eb9333caacf4ef4ea7a: declare the existing finish signal through the shipped public Worker API in the shared factory before either registration. Retry policy, callback-gap/public-report/history/replay requirements and all limits stay unchanged. Normal exact source/CI/main, new registry test/build/parent/caller review and one actual terminal observation remain. No duplicate SDK release or set-aside item is selected. Delivered omission/replay source controls and eight-locale Docs544 remain accepted at their existing scope; General retains shared closure.rmcdaniel commentedon Nov 14, 2026
Rust shared219 delivery is complete within the settled Rust criteria.
maximum_interval=Noneas uncapped, proves the positive eight-delay schedule through128 seconds, preserves exact recorded command replay and rejects changed capped commands. Existing finite attempt/original configured deadline/overflow bounds remain.[1,2]reports[1,2,2,2]seconds with independent callback gaps[1004,2005,2003,2004]ms; empty reports four explicit zeros with gaps[3,3,2,2]ms. Both complete retained histories have five unique attempts/four retries/one completion, five original callbacks and zero callbacks after replacement-instance replay.The qualifier's missing finish-signal declaration was corrected in SDK276, normal merge1129d797/tree-equal to reviewed1724a530, all exact-main workflows green and topic branch absent. The fresh registry build, sealed executable, exact caller/profile, output bounds and all15 terminal evidence objects independently reproduce. Consumer exit0/OOMfalse/cleanupPASS; exact runtime/network/parent/nonce inventories are independently absent. All three earlier NONPASS attempts remain preserved and are not relabeled.
This closes only the Rust omission/repeat-last slice. General retains cross-binding shared219 closure; Python's finite-default authoring work and other named criteria remain separate. No further Rust run or SDK/Server release is selected, and set-aside264/265/274 stay untouched.
rmcdaniel commentedon Nov 14, 2026
General accepts the Rust slice as delivered under the settled #219 criteria. The owner's final reconciliation and independent terminal installed review establish the normal-registry 4.17.0/native 3.10.0 short-list and empty-list observations, independent elapsed gaps, complete retained histories, callback-free replacement replay and exact cleanup. Source omission semantics and all eight public guide locales are already delivered through SDK273 and Docs544. The earlier three NONPASS observations remain unchanged.
No further Rust run or duplicate release is needed for this slice. The shared issue remains open for Python's finite-default authoring delivery and the other named cross-binding criteria. Python328's accepted additive API is now on main; its actual candidate/registry qualification, publication and public adoption remain separate work under Python329.
rmcdaniel commentedon Nov 14, 2026
The remaining Python 2.18.0 qualification step is policy-blocked and set aside; merged source and a successful build do not establish publication or installed acceptance.
rmcdaniel commentedon Nov 14, 2026
Current cross-binding delivery reconciliation: the accepted additive Python factory is merged through SDK328/329, and the exact SDK329 main checks pass. The remaining 2.18.0 qualification step is explicitly policy-blocked/set aside. No candidate retry invocation or publication is selected. The source/API review is not an installed default-cap result, and this held step will not be rerouted or repeated.
The accepted Rust slice remains delivered with actual timing/replay and public documentation. #219 stays open for Python's qualified published delivery and the other named remaining criteria. Current released Python behavior and legacy-policy preservation retain their documented scope.
rmcdaniel commentedon Nov 14, 2026
The finite Docs options-page and worker-protocol slice is delivered through Docs #549. Reviewed
4aba847emerged as23446c73with identical trees. Both pages now state repeat-last, five total attempts with four[1, 2, 2, 2]delays from[1, 2], empty-list zero backoff within existing attempt/time limits, and preservation of recorded policies. The existing total-attempt wording is unchanged. The 120-second default remains scoped to the embedded class.Embedded and native Server producer reviews plus independent source/locale review pass. All seven translations and 14 current English-source bindings are delivered, with all 240 executable blocks unchanged across the 16 guides. All 11 head checks, 12 actual-main checks and protected Pages 43615302300 succeed. Fresh public readback of all 16 changed pages matches the reviewed build.
Live: Options / Backoff and worker protocol, in all eight offered languages. No wire field, package/selector or runtime behavior changed. This does not deliver held Python 2.18 generated-default work or complete other binding requirements; shared219 retains those unchanged gates.
rmcdaniel commentedon Nov 18, 2026
Status (General, 2026-11-18). All five runtimes repeat the last delay on main and in their published releases: Workflow 2.29.0, Server 3.12.0, Python 2.17.2, PHP 3.19.0 and Rust 4.18.0. Rust keeps waiting the cap once capped exponential stops generating entries, so it never drops to zero.
local_backoffrun from main is in progress.[1,2]→[1,2,2,2],[]→0 and capped exponential holding the cap.rmcdaniel commentedon Nov 18, 2026
PHP installed evidence: run 44289035257 passed from main against the published tuple (SDK 3.18.2, Server 3.8.0). Live local retry delays matched the short and empty cases, and the history recorded under the old zero-delay behavior replayed with zero executor calls. The PHP row is complete. Python SDK 2.18.0, with the decided 60 s default cap, is the remaining release.
rmcdaniel commentedon Nov 18, 2026
Delivered. Every runtime applies the documented repeat-last rule: a backoff list shorter than the attempts repeats its last delay, and an empty list means zero backoff within attempt and time limits. Capped exponential holds the cap.
ActivityRetryPolicy.exponential(); the existing constructor is unchanged, and recorded delays replay unchanged). It also fixes the local-retry backoff that let the workflow-task lease lapse during long waits. The installed retry-cap check passed on both the candidate and the registry package.