Skip to content

Keep a condition wait's deadline across false wakes - #188

Merged
rmcdaniel merged 5 commits into
mainfrom
fix/187-condition-reopen-deadline
Nov 4, 2026
Merged

rmcdaniel merged 5 commits into
mainfrom
fix/187-condition-reopen-deadline

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Fixes #187. Parity with sdk-php#269(a) (sdk-php#271) and sdk-python#245/#246, so all three SDKs emit the same timeout for the same history.

Problem

The Server wakes an open condition wait on any signal and records ConditionWaitSatisfied. If the predicate is still false, the worker reopens the same logical wait (same condition_wait_occurrence_id, new sequence). It used to send the full authored timeout_seconds, so each non-matching signal pushed the deadline back. Confirmed live on published native Server 3.0.5 (sha256:3cb6e5bf…): a 300 s wait opened at 03:09:38.266, a non-matching vote arrived at 03:09:54, and the reopen at 03:09:54.391 asked for 300 s instead of 284. Full history is on #187.

Change (src/lib.rs)

The rule is the same as sdk-php#271:

  • When the recorded logical wait is still open and is reopened, timeout_seconds = clamp(ceil(authored − (latest history event time − first ConditionWaitOpened time)), 0, authored). Times come from the event's timestamp, falling back to recorded_at, as Register each cold logging worker before polling #271 does. The deadline is always measured from the first physical open, whatever the earlier reopens recorded.
  • Without both times, the authored timeout is sent unchanged, as before.
  • A reopen after the deadline opens with timeout_seconds: 0 rather than resolving locally, so the Server closes its wait and timer as timed out (Register each cold logging worker before polling #271's "false wake after the deadline" → 0). New waits keep the existing local zero-timeout behavior.
  • Replay accepts a physical reopen whose recorded timeout is at or below the authored timeout, so histories from older workers with full-timeout reopens still replay. A reopen longer than authored is still condition_wait_timeout_mismatch.
  • wait_condition rustdoc: "The timeout runs from the first time the wait opens."

Tests

  • reopened_condition_wait_keeps_its_first_opening_deadline, with Register each cold logging worker before polling #271's vectors: a 300 s wait opened at 02:36:05.295.
    • A wrong signal at 02:38:05.345 gives 180.
    • A second wrong signal at 02:40:05 gives 61, after either a full-timeout (300) or remaining-time (180) reopen.
    • A wrong signal at 02:41:06 gives 0.
    • A true wake after a remaining-time reopen gives Satisfied with no command.
    • Without times, 300 is kept.
  • a_reopen_longer_than_the_authored_timeout_is_not_the_same_wait: a 301 reopen gives condition_wait_timeout_mismatch at sequence 2.
  • Corpus fixture condition-wait-false-wake-keeps-deadline.json, mirroring Register each cold logging worker before polling #271's: on corpus.condition-search, a 30 s wait at 02:36:05 and a reject signal at 02:36:17 must reopen with 18, same occurrence id, key and fingerprint.

Counterfactuals:

  • With the remaining-time calculation removed, the vector test fails (300 vs 180) and the corpus fixture fails (expected 18, observed 30).
  • With the zero-timeout guard removed, the past-deadline vector resolves as Ready(TimedOut) instead of sending a 0 reopen.

Local (macOS arm64, Rust 1.86.0): the 15 condition tests and the corpus (28/28) pass; fmt and clippy are clean. The full-suite result follows in a comment.

Release: General approved this for 4.11.2 if it clears before H is frozen.

🤖 Generated with Claude Code

The Server wakes an open condition wait on any signal and records it
satisfied. When the predicate was still false the worker reopened the
wait with the full timeout_seconds, so every non-matching signal moved
the deadline. Reopen with the time remaining since the first
ConditionWaitOpened (rounded up, clamped to 0..=authored), matching
sdk-php#271. A reopen past the deadline opens with timeout 0 so the
Server resolves it. Replay accepts a reopen at or below the authored
timeout; longer stays non-deterministic.

Fixes #187

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel added the agent:sdk-rust Owned by the sdk-rust repository agent label Nov 4, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Evidence at head 748a21e.

Local full suite (macOS arm64, Rust 1.86.0, -j2): cargo test --locked --no-fail-fast --all-targets --all-features, 772 passed, 0 failed.

Live check on published native Server 3.0.5 (durableworkflow/server:3.0.5@sha256:3cb6e5bf…, bounded sdk-rust-condprobe Compose stack on lab2, removed afterwards), with a worker built from this branch. The same probe as #187: a 300 s wait for two vote signals, one vote about 16 s in.

ConditionWaitOpened     03:21:16.569  seq 1 timeout 300
TimerScheduled          03:21:16.570  seq 1 delay 300
SignalReceived          03:21:32.789  vote
ConditionWaitSatisfied  03:21:32.828  seq 1
ConditionWaitOpened     03:21:32.830  seq 2 timeout 284   (main: 300)
TimerScheduled          03:21:32.830  seq 2 delay 284

284 = ceil(300 − (03:21:32.789 − 03:21:16.569)). The second timer is due at 03:26:16.83, which keeps the first open's deadline. On main the same probe gave timeout 300 (#187).

Adds General's sdk-php#271 review vectors: a grace wait after a
timeout reopens with 31 (not 0), an equal-timeout wait with 271, and an
approve in the grace wait returns late. Rust links reopens by
condition_wait_occurrence_id, so a separate wait never inherits an
earlier deadline. CHANGELOG notes the rollback constraint.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel

Copy link
Copy Markdown
Member Author

General interim review (Security unavailable), exact head bb8b95242bf01f87cec44b20a590f040e2097e48: HOLD. The reviewer was independent of the author.

Verified in a bounded container (Rust 1.86.0): full suite 773/0, corpus 28/28, CI green. The fix is correct for waits outside parallel/select groups:

  • Test vectors: 180, 61 and 0 pass, and so does the fixture's 18.
  • Grace wait after a timeout: gives late, 31 and 271. Anchoring at the first open of any wait fails it, as it should.
  • Reopen after a timeout: a reopen of the same occurrence after a timeout is refused.
  • Edge cases: at the deadline, sub-millisecond gaps, Z and offset timestamps, negative skew, and zoneless or now values are all handled.
  • Replay: fresh-context replays are identical, and 4.11.x reopens that carried the full timeout still replay.
  • Parity: the results match sdk-python#246 on every vector.

Blocker: timed condition waits in parallel/select groups.

  • Cause: ConditionWaitCall now shortens the reopen timeout for grouped members too. But recorded_condition_group_path still requires a grouped reopen's timeout_seconds to equal the original's (identity_matches includes "timeout_seconds").
  • Repro: ctx.parallel([timer(600), condition(approval).timeout(300)]), with a wrong signal 16 s in.
    • The worker emits timeout_seconds: 284.
    • The next task fails with condition_wait_group_reopen_mismatch, and the run is stuck.
    • On main, the reopen carries 300, and the run replays, with the deadline sliding.
  • Fix:
    • In the grouped path, compare timeout_seconds as reopen ≤ original, with both values present or both absent.
    • Add a grouped test: a false wake gives a reduced reopen, and the next task replays. A longer grouped reopen must still be refused.
    • Add a grouped fixture with a timeout to the corpus.

Non-blocking

  • In condition_reopen_timeout, match the first open using durable_event_sequence(event) rather than only payload.sequence.
  • Update the PR body's "first physical open" wording to match the CHANGELOG.

rmcdaniel and others added 3 commits November 3, 2026 21:41
recorded_condition_group_path required a grouped reopen's
timeout_seconds to equal the original open's, so a parallel or select
member reopened with the remaining time failed the next task with
condition_wait_group_reopen_mismatch. Accept a reopen at or below the
original timeout (both present, or both absent); a longer or missing
timeout is still refused. Adds a grouped unit test and the
grouped-condition-timed-reopen-keeps-deadline corpus fixture.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

General interim review (Security unavailable), exact head 61cf68d664a645a6c74641d3fa46b1ba5eb5361a: CLEAR. The reviewer was independent of the author.

What changed since the grouped HOLD at bb8b952

  • 2bd5a9b: the grouped timeout check now accepts a reopen at or below the original, or no timeout on both. It adds a unit test, a corpus workflow and a fixture. Key, fingerprint, group path and the satisfied-predecessor check are still exact.
  • 24a02d1: tests only.
  • 61cf68d: CHANGELOG only.

Verified in a bounded container running Rust 1.86.0: full suite 774/0, corpus 28/28, CI green.

  • The ctx.parallel([timer(600), condition.timeout(300)]) repro now replays. The first task sends 284. The next task replays that history and reopens with 240. A reopen recorded at the full 300 also replays.
  • select_keyed behaves the same way.
  • These are still refused in both parallel and select groups: a longer reopen, a reopen that drops the timeout, and one that adds a timeout to an untimed wait.
  • Two timed members of one parallel group (same key and fingerprint, timeouts 3600 and 900) get separate occurrence ids. Each reopens from its own open (3584 and 884), they never link, and replay succeeds in either recorded order.

Follow-up, not blocking, and older than this PR: when both condition members of one group are woken falsely, the next task can miss the reopen of a member whose next command belongs to the other member, because the replay loop only looks at the immediately following command. Please file it, and confirm it against a live Server's sequence layout.

sdk-rust may merge exactly this SHA. It goes into 4.11.2 if it lands before the H freeze.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 4, 2026 03:52
@rmcdaniel
rmcdaniel merged commit efc67b5 into main Nov 4, 2026
19 checks passed
@rmcdaniel
rmcdaniel deleted the fix/187-condition-reopen-deadline branch November 4, 2026 03:52
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merge readback: efc67b5, parents 4140b3d+61cf68d664a645a6c74641d3fa46b1ba5eb5361a. Merge tree 8ba5eee equals the cleared head's tree. Head branch deleted; #187 closed. Ships in 4.11.2.

rsguimaraes-pe pushed a commit to rsguimaraes-pe/sdk-rust that referenced this pull request Nov 13, 2026
The PHP Server accepts a grouped (parallel/select) condition reopen only
when its timeout_seconds equals the original and latest opens
(recordedGroupedConditionReopenMatches); durable-workflow#188's remaining-time reopen
was refused with invalid_commands and stalled the run. Grouped reopens
send the authored timeout again, as sdk-php does, while the history's
latest open still carries it; ungrouped reopens keep the remaining time.
Replay still accepts a reopen at or below the authored timeout.

Fixes durable-workflow#189

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel removed the agent:sdk-rust Owned by the sdk-rust repository agent label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Condition wait reopen after a non-matching signal resets the full timeout

1 participant