fix(spurctld)!: refuse to bootstrap a second raft cluster - #806
Open
pre wants to merge 2 commits into
Open
Conversation
BREAKING CHANGE: a spurctld with an empty Raft store now refuses to start
when a configured peer already runs a cluster that does not list it. This
turns a silent corruption into a clear startup failure, and it makes the
replica count of a running controller fixed.
A replica added to a running cluster starts with an empty store, exactly
like a first start, so it bootstrapped a NEW cluster. On Kubernetes a
StatefulSet rolling update starts at the highest ordinal and gives each
new ordinal an empty volume, so raising replicas from 1 to 3 made the two
new empty replicas form a cluster and out-vote the one holding the data.
Measured results: the replica with the data panicked in openraft
("range start is greater than range end in BTreeMap"), the job id counter
restarted at 1 and collided with ids already in accounting, and the
replicas answered sinfo differently behind one headless Service.
An empty store now asks each peer over the new ClusterProbe RPC whether it
belongs to a cluster. A peer whose membership contains this node means a
member lost its volume, so it starts and catches up. A peer whose
membership does not contain it means bootstrapping would split the
cluster, so startup fails with a message that says the replica count
cannot be raised. A peer that predates the RPC answers UNIMPLEMENTED,
which is read as "no cluster", so mixed versions behave as before.
ClusterProbe is added to raft_internal.proto, which is controller-to-
controller plumbing only. No field tag changes, and slurm.proto is
untouched.
The deployment page told the reader to apply spurd.yaml AND operator.yaml. Both register the same Kubernetes nodes, so a reader who followed it got two agents claiming every node. They are alternatives: the page now gives a table of the two topologies and says that Pod mode launches only a SpurJob, so sbatch and spur submit accept a job that never gets a Pod. Add a warning that the spurctld replica count is fixed at the first apply, which is what the controller now enforces. Example manifests: * spurd.yaml selected spur.amd.com/compute, but the operator default and every other reference is spur.amd.com/managed. A reader who labelled nodes for one got nothing scheduled by the other. * spur.conf holds the accounting password, so spurctld.yaml mounts a Secret. configmap.yaml keeps the readable template, with an empty database_url instead of an example credential.
pre
requested review from
biluriuday,
sajmera-pensando,
sgopinath1,
shiv-tyagi and
yansun1996
as code owners
September 2, 2026 12:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related:
Motivation
Raising the replica count of a running
spurctldStatefulSet silently destroys thecluster. A replica that starts with an empty Raft store bootstraps a second
one-member cluster instead of joining the existing one. The two then meet, and:
RaftCoreon a higher term;already exist;
whichever one the client reached.
Nothing in the current code stops this, and the operator gets no warning. The goal
is to make the unsafe transition impossible rather than merely documented.
Technical Details
A node with an empty store now asks every configured peer whether it already
belongs to a cluster, before it bootstraps.
raft_internal.protogains aClusterProbeRPC. It is internalcontroller-to-controller plumbing, so
slurm.protois untouched and FFI and RESTare unaffected. The RPC is appended after the existing three, so a peer that
predates it answers
UNIMPLEMENTED, which the caller reads as "unknown" andtreats as permission to continue.
The probe returns the peer's effective membership. Three verdicts follow:
start, with a message that explains why and what to do instead.
The refusal is deliberate. Real online growth needs
change_membership, which isa feature and not a defect fix; until then the replica count is fixed at first
apply, and the example manifests and
docs/deployment/kubernetes.rstnow say so.This is a breaking change: a controller that used to start now refuses to, in the
one case where starting corrupted the cluster. Hence the
!in the title.The docs commit also documents the two execution backends (Pod mode and Agent
mode) as a choice, and corrects the
spurdDaemonSetnodeSelectorfromspur.amd.com/computetospur.amd.com/managed, which is the operator's actualdefault selector (
crates/spur-k8s/src/main.rs). The old value matches no node.Related:
Test Plan
spurctldwith 1 replica, submit jobs,then raise the replica count to 3 and observe what the new replicas do.
healthy three member cluster, and cold start three replicas from empty PVCs.
ClusterProbeserver.Test Result
Environment: three node RKE2 cluster on cloud VMs, 8 vCPU and 96 GiB each, no GPU.
spurctld-0panicked inRaftCore, the next job id was 1 although jobs 1 and 2existed, and
sinfogave different answers depending on which replica served it.The panicked Pod stayed
1/1 Runningwith 0 restarts.spurctld-1andspurctld-2inCrashLoopBackOffwith the refusal message.spurctld-0keeps both jobs, logsno panic, and the next job id is 3.
replaced all three replicas with no refusal. A cold start of three replicas from
empty PVCs elected a leader, replicated, survived a leader kill, and readmitted
the restarted member.
cargo clippy --workspace --exclude spur-ffi --all-targets --lockedreportsnothing;
cargo test --lockedpasses 3525 tests, including three new ones thatdrive the verdicts against a stub
ClusterProbegRPC server.Submission Checklist