Skip to content

Never resend a non-poll request to recover server capabilities - #306

Merged
rmcdaniel merged 5 commits into
mainfrom
general/digest-no-resend
Nov 21, 2026
Merged

rmcdaniel merged 5 commits into
mainfrom
general/digest-no-resend

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #304, applying the cross-SDK principle: never resend a non-idempotent request to recover server capabilities.

Problem

In #304, when an answer named an unknown capabilities digest, the SDK resent that request once with none. That covered every request kind, including completions, failures, heartbeats, starts and signals. A workflow-task completion's answer can carry the next leased task, and resending the completion only returns a duplicate receipt without that task. The PHP SDK lost leased activities exactly this way.

Change (send_json_with_timeout, src/server_capabilities_digest.rs)

  • Task polls (/worker/{workflow,activity,query}-tasks/poll) are idempotent by poll_request_id. They are still resent once with none, as before.

  • Every other request is never resent:

    • its answer is kept as received;
    • the full capabilities are read with a read-only GET /api/cluster/info, from worker_protocol.server_capabilities (the same object);
    • Digest::restore_with substitutes that object for the unknown digest string, at the same depth rules as before;
    • the held pair is forgotten, so later requests send none.
  • The received answer is never failed. If that single cluster-info read fails (a 5xx, or a refused or dropped connection), the answer, including any next task, is still used:

    • the unknown copies are dropped from it;
    • the held digest is forgotten;
    • later requests send none and learn the capabilities again.
      Cluster info never recurses into itself.
  • Cluster-info outages no longer stop a running worker. This covers startup discovery (query-task authority, Nexus cooperation) and the inline-payload policy read before a report.

    • A connection failure, timeout, 5xx or 429 is retried with the shared jittered backoff, uniform(0, min(4 s, 0.25 s·2^(n−1))).
    • An outage logs a tracing ERROR after 60 s, then every 5 minutes, and resets on success.
    • Retries continue until the read succeeds or the worker stops. At startup this waits the same way registration does.
    • Refusals (retryable: false, 401, 403 and other 4xx) still return.
    • Client::cluster_info, and any Client outside a running worker, makes one attempt.
  • CHANGELOG (Unreleased) and the README are updated.

  • API: no public change. cargo semver-checks --baseline-version 4.21.0 reports 229 checks: 229 pass, 31 skip and Summary no semver update required.

  • Stale payload policy during an outage. The 60 s payload-policy cache used to make reports wait for cluster info once it expired during an outage. Now an expired policy is refreshed with one attempt. On an outage the last-known policy is used and discovery is retried 10 s later, as in PHP #451. Only a worker with no policy yet waits under the outage rules.

  • Too-large report after a lowered limit. An activity report sent inline under a stale policy, then refused with 413 payload_too_large, is no longer lost:

    • the policy is forgotten and rediscovered (outage rules apply);
    • the same report is encoded again, externalizing what the new limit requires;
    • it is resent once.
      Workflow-task completions keep their typed size refusals (managed_worker_leaves_other_413_and_5xx_completion_refusals_unchanged).
  • Still too large: fail the attempt, not the worker. Before this change, a result that was refused again after rediscovery, or that could not be externalized (above the upload limit, or no storage), made the settlement return an error that the poll lane treated as fatal, and the worker exited. Such an attempt is now reported as a non-retryable ActivityResultTooLarge failure (failure.type and failure.class), as in PHP, and the worker keeps running.

Tests (tests/server_capabilities_digest.rs, src/tests/worker_transient_outage.rs)

Test What it covers
a_report_refused_after_a_lowered_limit_is_externalized_and_resent_once The policy is cached at 2 MB, then the Server lowers it to 100 KB. A ~500 KB result is sent inline and refused with 413. It is rediscovered once, uploaded once, resent with an external reference, and completes.
a_result_still_too_large_after_rediscovery_fails_the_attempt_not_the_worker Covers a result refused again after re-encoding, and a result with no storage after rediscovery. Exactly one non-retryable ActivityResultTooLarge fail is sent, and the poll lane keeps running.
an_expired_policy_keeps_reports_moving_through_a_cluster_info_outage The cache is aged past 60 s and cluster info then returns 500. Both reports are sent immediately, with one refresh attempt and then a 10 s pause.
a_garbled_completion_answer_is_kept_and_capabilities_come_from_cluster_info The completion is sent exactly once. Its own answer (recorded, next_task) is returned, with server_capabilities restored from GET /api/cluster/info. The cluster-info request and the next poll both send none.
a_garbled_answer_survives_unavailable_cluster_info Runs with cluster info returning a 500, and with a dropped connection. The completion's answer and its next task are used, the field is dropped, and the next poll re-learns the capabilities with none.
startup_cluster_info_failures_are_waited_out Startup discovery gets a 500, then a dropped connection, then success. The worker registers once and runs.
run_time_cluster_info_failures_are_waited_out The payload-policy read before a completion gets a 500, then a dropped connection. The completion is delivered once and the worker keeps running.
cluster_info_failures_other_than_refusals_are_transient Classification of cluster-info failures.
worker_answers_are_restored_from_the_digest_each_request_sent A garbled poll is still resent once with none, and a second unknown digest is Error::Protocol.

Results:

  • cargo fmt --check: clean.
  • cargo clippy --all-targets --all-features -- -D warnings: clean.
  • cargo test: 1120 passed, 0 failed, 82 ignored (server-gated).
  • cargo +1.86.0 check --all-targets: passes.
  • The regression-corpus validator passes.

🤖 Generated with Claude Code

#304 resent any request once with "none" when its answer named an
unknown capabilities digest. A completion's answer can carry the next
leased task, and a resent completion only gets a duplicate receipt
without it, so a worker could lose leased work.

Only task polls (idempotent by poll request ID) are still resent. For
any other request the answer is kept as received, the full capabilities
are read with GET /api/cluster/info (worker_protocol.server_capabilities)
and substituted for the unknown digest, and the held pair is forgotten
so later requests send none.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel added bug Something isn't working agent:general labels Nov 20, 2026
rmcdaniel and others added 4 commits November 20, 2026 10:17
Review fixes:

- After a garbled digest the one cluster-info read used `?`, so a failed
  read threw away an answer already received (including its next task)
  and stopped the worker. The answer is now used without the
  capabilities (the unknown copies are dropped), the held digest is
  forgotten, and later requests learn them again with `none`.
- Any cluster-info failure used to stop a running worker. Startup
  discovery (query-task authority, Nexus cooperation) and the inline
  payload policy read before a report now retry a connection failure,
  timeout, 5xx or 429 with the shared jittered backoff and the 60 s /
  5 min ERROR escalation until they succeed or the worker stops.
  Refusals still return; Clients outside a running worker make one
  attempt.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review fixes:

- When the 60 s payload-policy cache expired during a cluster-info
  outage, every report waited for cluster info. An expired policy is now
  refreshed with one attempt; on an outage the last-known policy is used
  and discovery is tried again 10 s later. Only a worker with no policy
  yet waits under the outage rules.
- An activity report sent inline under a stale policy, after the Server
  lowered its inline limit, was refused with 413 payload_too_large and
  lost. The worker now forgets the policy, rediscovers it (outage rules
  apply), encodes the same report again, externalizing what the new
  limit requires, and resends it once. A second refusal keeps today's
  handling. Workflow-task completions keep their typed size refusals.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A completion still refused as too large after the payload policy was
rediscovered, or one that could not be externalized (above the upload
limit, or no payload storage), returned an error from the settlement,
and the poll lane treated it as fatal: the worker exited. Such an
attempt is now reported as a non-retryable ActivityResultTooLarge
failure (failure.type and failure.class), as the PHP SDK does, and the
worker keeps running.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel rmcdaniel left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the coordinated worker payload-limit fix against the PHP #451, Python #360 and Rust #306 changes. PHP, Python and Rust recover only from the Server's typed size refusal. The worker can rediscover the policy and re-encode the result without re-running the activity. A result that still cannot fit is reported with type and class ActivityResultTooLarge and non_retryable: true, and worker dispatch remains available. A discovery outage keeps the attempt eligible for recovery instead of recording a permanent size failure.

The target-branch and language CI jobs passed for all three reviewed heads. Python also passed the local focused transport suite, and Rust passed the 17 worker-outage tests. The release qualification and published-package checks follow the merges.

@rmcdaniel
rmcdaniel merged commit 9847c8e into main Nov 21, 2026
21 checks passed
@rmcdaniel
rmcdaniel deleted the general/digest-no-resend branch November 21, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant