fix(spurctld): keep the node record when its agent shuts down

[pre]

Symptom

On a cluster with the WireGuard mesh and k0s, rebooting a node takes it out of Kubernetes and out of the batch cluster, permanently. Nothing repairs itself, and no signal names the network:

sinfo	2 nodes of 3
Kubernetes node	NotReady, still after 5 minutes
spurd on the rebooted node	restart loop, 44 restarts
WireGuard peers on the rebooted node	2, correct
WireGuard peers on every node that stayed up	1, the rebooted node pruned

The error on the rebooted node points at Raft, while the fault is in the mesh:

Error: registration failed
Caused by: code: 'The service is currently unavailable', message: "not the Raft leader"

A reboot is a normal event, so a kernel update or a power cut loses a node for good.

Cause

spurd deregisters on SIGTERM (spurd/main.rs), and deregister_agent answered with remove_node(force = true), which deletes the node record. mesh_from_nodes builds the WireGuard membership from that record, so the membership loses the node, and reconcile_mesh then removes its peer on every node that stayed up. The rebooted node cannot register again, because the controllers are only reachable at their mesh addresses — the mesh it was just removed from. The local controller answers, is not the leader, and cannot forward to the leader over the broken mesh, which produces the message above.

The comment in cluster_k8s.rs says the prune exists so that node-local drift, "reboot, wg restart", can self-heal. A reboot is the case the prune makes unrecoverable.

The fault fires only when the rebooted node is not the Raft leader: the removal of the leader cannot commit while the leader goes down, so rebooting the leader is survivable by accident. On a cluster of N nodes, N-1 of them break the cluster when they reboot.

Change

A stopping agent is Down, not gone. DeregisterAgent now marks the node Down and keeps the record, its wg_pubkey and its mesh IP:

• Jobs are evicted exactly as before, because the Down transition evicts them.
• An admin hold keeps its reason and its lock, so an operator drain is not cleared by a service restart.
• check_node_health recovers the node when its heartbeat returns.
• Removal stays an operator action: spur node remove and spur k8s down.

Test

agent_shutdown_marks_node_down_and_keeps_it fails if the record, the wg_pubkey or the automatic recovery is lost. 963 unit tests pass.

Verified on a three-node cluster with k0s and the WireGuard mesh, the k0s control plane on node 2 and a worker on node 3:

	before	after
node removed from cluster on the survivors	yes	none
node lifecycle	deleted	marked DOWN, then recovered about 30 s later
sinfo	2 of 3	3 of 3
WireGuard peers	1 / 1 / 2	2 / 2 / 2
Kubernetes node	NotReady	Ready, it never left
spurd restarts	37 and rising	0
repair by hand	wg syncconf needed	none

Two reboots in a row, plus a batch job afterwards. A reboot of all three nodes at the same time also comes back complete.

Not in this PR

The agent still has no path to a controller that does not go through the mesh: spur.conf names only mesh addresses. This change removes the fault; an underlay fallback for registration would remove the whole class, and is a larger change worth its own discussion.

[pre]

Verification so far, since the workflow runs need maintainer approval:

Local, on this branch alone (rebased on 360d44c):

• cargo fmt --check --all clean
• cargo clippy -p spurctld no warnings
• 963 unit tests pass, including the new agent_shutdown_marks_node_down_and_keeps_it

On a three-node cluster with the WireGuard mesh and k0s, control plane on
node 2 and a worker on node 3, with the patched spurctld on all three:

• Before: rebooting the worker left sinfo at 2 nodes of 3, one peer on each
  node that stayed up, the Kubernetes node NotReady and spurd at 37
  restarts. Reproduced twice.
• After: two reboots in a row keep 3 nodes, 2 peers on every node, the
  Kubernetes node Ready throughout and spurd at 0 restarts. The node is
  back in sinfo about 30 s after it comes up, with no command by hand, and
  the journal shows node marked DOWN (agent shutdown), record kept followed
  by node recovered (heartbeat resumed).
• A batch job runs after the reboots, and a reboot of all three nodes at the
  same time also comes back complete.

The E2E suite has not run yet. Could a maintainer approve the workflow runs?

[pre]

Superseded by

• fix(spurctld): keep the node record when its agent shuts down #784