fix(k8s): recover a node the controller has forgotten - #809
Open
pre wants to merge 2 commits into
Open
Conversation
The node watcher registers a node on its initial list and when its resources change. A controller that lost its state, for example a Raft cluster that was built again, is neither event, so every node stayed unknown, sinfo reported none, and the cluster scheduled nothing until the operator was restarted by hand. The heartbeat already holds the RegisterAgentRequest of each tracked node. A heartbeat that comes back NOT_FOUND now registers the node again from that stored request.
A job from the Slurm-compatible CLI in Pod mode waited in the queue with "JobLaunchFailure (dispatch confirmation failed (0/1 confirmed): 1 agent unreachable)". The operator was running and reachable. It refused the launch because no SpurJob carries the job id, which only kubectl apply of a SpurJob creates. The operator hid the refusal behind its own retry budget and answered DEADLINE_EXCEEDED, which the controller reads as a transport failure. The operator now answers NOT_FOUND with what is missing when the budget runs out, and the controller classifies NOT_FOUND and FAILED_PRECONDITION as an explicit rejection. The queue reason now says the agent rejected the launch, and the operator's message reaches the controller log.
pre
requested review from
biluriuday,
sajmera-pensando,
sgopinath1,
shiv-tyagi and
yansun1996
as code owners
September 2, 2026 12:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related:
Motivation
Two faults that both come from a controller answering
NotFound.A node the controller has forgotten never comes back. The operator registers
each Kubernetes node once, at startup, then only sends heartbeats. If the
controller loses its record of a node, through
spur node removeor through arebuild from an older state, every later heartbeat fails with
NotFoundand theoperator merely logs it. The node stays missing from
sinfountil somebodyrestarts the operator, and the cluster quietly runs at reduced capacity.
A refused launch is reported as an unreachable agent. When an agent answers
NotFoundorFailedPrecondition, the controller maps it toUnreachable. Theuser sees "agent unreachable" although the agent answered, and the real reason is
lost. That sends anyone debugging it towards the network instead of the job.
Technical Details
The operator keeps the
RegisterAgentRequestit built at startup. When a heartbeatcomes back
NotFound, it registers again from that stored request and logs that itdid so. Nothing else changes: any other error keeps the existing behaviour.
On the controller side,
tonic::Code::NotFoundandtonic::Code::FailedPreconditionnow map to
DispatchError::AgentRejectedand carry the agent's own message, ratherthan falling through to
Unreachable. The two codes mean the agent answered anddeclined, which is the definition of a rejection.
The operator's "no
SpurJobcarries this job id" path is also made explicit, sothe rejection that reaches the user says what actually happened instead of
deadline_exceeded, which the controller used to read as unreachable.Related:
Test Plan
spur node remove, then wait for one heartbeat. Do not restart the operator.sinfo.the user gets and the reason in the controller log.
Test Result
Environment: three node RKE2 cluster on cloud VMs, 8 vCPU and 96 GiB each, no GPU.
spur node removetook a node away andsinfoshowed 2. One heartbeat later theoperator logged that the controller did not know the node and that it registered
it again, and
sinfoshowed 3. The operator was not restarted.heartbeat on every cycle.
1 agent rejected launch, and thecontroller log holds the agent's reason. Before the change it read
"agent unreachable".
cargo clippy --workspace --exclude spur-ffi --all-targets --lockedreportsnothing;
cargo test --lockedpasses 3253 tests.Submission Checklist