fix(spurctld): keep the node record when its agent shuts down

[pre]

spurd deregisters on SIGTERM, and the controller answered with remove_node(force = true), which deleted the node record. On a cluster with the WireGuard mesh, mesh_from_nodes builds the membership from that record, thus the k0s reconciliation pruned the peer of the node on every other node. The rebooted node could not register again, because the controllers are only reachable over the mesh it had just left. This is a deadlock: no part of it repairs itself, and the node stays out of Kubernetes and out of the batch cluster.

A stopping agent is Down, not gone. DeregisterAgent now marks the node Down and keeps the record, its wg_pubkey and its mesh IP. The jobs are evicted exactly as before, because the Down transition evicts them. An admin hold keeps its reason and its lock. check_node_health recovers the node when its heartbeat returns. Removal stays an operator action: spur node remove and spur k8s down.

Measured on a three-node cluster with k0s and the mesh: before the change, a reboot of a worker left sinfo at 2 nodes of 3, one peer on every node that stayed up, the Kubernetes node NotReady and spurd in a restart loop. After the change, two reboots in a row keep 3 nodes, 2 peers, the Kubernetes node Ready and no restart of spurd, and the node returns about 30 seconds after it comes up.

The fault fires only when the rebooted node is not the Raft leader: the removal of the leader cannot commit while the leader goes down.

Motivation

A reboot is a normal event, so a kernel update or a power cut loses a node for good.

Symptom

On a cluster with the WireGuard mesh and k0s, rebooting a node takes it out of Kubernetes and out of the batch cluster, permanently. Nothing repairs itself, and no signal names the network:

sinfo	2 nodes of 3
Kubernetes node	NotReady, still after 5 minutes
spurd on the rebooted node	restart loop, 44 restarts
WireGuard peers on the rebooted node	2, correct
WireGuard peers on every node that stayed up	1, the rebooted node pruned

The error on the rebooted node points at Raft, while the fault is in the mesh:

Error: registration failed
Caused by: code: 'The service is currently unavailable', message: "not the Raft leader"

The fault fires only when the rebooted node is not the Raft leader: the removal of the leader cannot commit while the leader goes down, so rebooting the leader is survivable by accident. On a cluster of N nodes, N-1 of them break the cluster when they reboot.

Change

A stopping agent is Down, not gone. DeregisterAgent now marks the node Down and keeps the record, its wg_pubkey and its mesh IP:

• Jobs are evicted exactly as before, because the Down transition evicts them.
• An admin hold keeps its reason and its lock, so an operator drain is not cleared by a service restart.
• check_node_health recovers the node when its heartbeat returns.
• Removal stays an operator action: spur node remove and spur k8s down.

Test Plan

agent_shutdown_marks_node_down_and_keeps_it fails if the record, the wg_pubkey or the automatic recovery is lost. 963 unit tests pass.

Verified on a three-node cluster with k0s and the WireGuard mesh, the k0s control plane on node 2 and a worker on node 3:

	before	after
node removed from cluster on the survivors	yes	none
node lifecycle	deleted	marked DOWN, then recovered about 30 s later
sinfo	2 of 3	3 of 3
WireGuard peers	1 / 1 / 2	2 / 2 / 2
Kubernetes node	NotReady	Ready, it never left
spurd restarts	37 and rising	0
repair by hand	wg syncconf needed	none

Two reboots in a row, plus a batch job afterwards. A reboot of all three nodes at the same time also comes back complete.

Test Result

Verification so far, since the workflow runs need maintainer approval:

Local, on this branch alone (rebased on 360d44c):

• cargo fmt --check --all clean
• cargo clippy -p spurctld no warnings
• 963 unit tests pass, including the new agent_shutdown_marks_node_down_and_keeps_it

On a three-node cluster with the WireGuard mesh and k0s, control plane on
node 2 and a worker on node 3, with the patched spurctld on all three:

• Before: rebooting the worker left sinfo at 2 nodes of 3, one peer on each
  node that stayed up, the Kubernetes node NotReady and spurd at 37
  restarts. Reproduced twice.
• After: two reboots in a row keep 3 nodes, 2 peers on every node, the
  Kubernetes node Ready throughout and spurd at 0 restarts. The node is
  back in sinfo about 30 s after it comes up, with no command by hand, and
  the journal shows node marked DOWN (agent shutdown), record kept followed
  by node recovered (heartbeat resumed).
• A batch job runs after the reboots, and a reboot of all three nodes at the
  same time also comes back complete.

The E2E suite has not run yet. Could a maintainer approve the workflow runs?

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.