Repository navigation
Treat cluster-info read failures at worker startup as transient - #359
Merged
Merged
Conversation
GET /api/cluster/info has no side effects, so any connection failure and any 5xx or 429 is now transient for the startup discovery read (unless retryable: false); before, only the narrower transient set was, so a plain 500 stopped the worker. Startup retries use the shared backoff and now escalate like poll outages: ERROR after 60 s, then every 5 minutes. Typed refusals (401/403) stay fatal. After registration the worker uses the discovery it registered with and does not read cluster info again. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



A cluster-info read failure at worker startup stopped the Python worker. Startup now treats cluster-info reads as transient.
Change
Startup discovery.
GET /api/cluster/infois side-effect free, so_register_through_transients(Worker.run()andrun_until()) now treats as transient:retryable: falseBefore, only the narrower set from #355 counted, so a plain
500 {"message":"Server Error"}, a 500 with a reason, or a 501 stopped the worker. Retries use the shared backoffuniform(0, min(4, 0.25·2^(n−1)))seconds, and the worker waits until discovery answers.Escalation. This is the same as for poll outages:
worker registration has failed for N s: ...)Each startup is a fresh streak.
Typed refusals are unchanged. 401 and 403 on discovery stay fatal at startup, as decided, with the error-level stop log.
retryable: falseis fatal too.Run time. After registration the worker uses the discovery it registered with; the cached
_runtime_discoveryis not refetched. The one run-time read, #358's capability recovery after an unknown digest, already keeps the answer and drops the field if discovery fails. A cluster-info outage at run time therefore cannot stop the worker.Tests (
tests/test_worker_transient_runtime.py)500 {"message"},500 {"reason"},501,429and a refused connection on cluster info, repeated six times and then recovering:500 retryable: false, each served once and never retried.Local results (Python 3.14):
pytest tests/ -m "not integration": 7899 passed, 3 skipped.ruff: clean.mypy: clean.🤖 Generated with Claude Code