fix(gateway): keep a removed node out until an operator re-admits it

[kvinwang]

Stacked on #1099; review only the last commit until that merges (base is set to its branch).

Problem

A node removed via RemoveNode silently rejoined the cluster the next time it started. Every node re-registers its own sync address on boot (register_peer_url), the sync endpoints authenticate app identity (RA-TLS) but not membership, and neither handle_envelope nor apply_envelope asks whether the sender is still a member. So "removed but still powered" was never really removed.

With #1099 collecting tombstones, that comeback stops being an annoyance and becomes a resurrection. A removed node returning with its old data directory still holds, live, records whose deletes the cluster collected in its absence. Ordinary rounds do not leak them back — its request re-teaches the responder coverage of its origin, so nothing it authored is re-sent. But the digests disagree for as long as the zombies exist (no tombstone and no ack filter can reconcile them), and after digest_check_rounds wavekv's divergence repair answers with a full re-exchange that merges them in as live values: deregistered CVMs re-entering every node's WireGuard config. #1099's a_removed_node_returning_with_old_state_resurrects_collected_deletes pins that route. The conclusion it forces: the door, not the merge, is where a removed sender has to stop.

Fix

RemoveNode now writes a durable removal marker, and both sync routes enforce it: an envelope from a marked sender is refused with 403 before anything merges.

The marker is a live record, deliberately. The obvious marker — the __peer_addr tombstone the removal already writes — is food for the very GC this defends against: once collected, the key reads as "never registered", which is indistinguishable from an early-bootstrap peer and must be admitted. The lockout would evaporate at exactly the moment it starts to matter, because the resurrection window opens after collection (while the tombstone survives, LWW kills the zombies for free). A fact that must outlive every tombstone cannot be expressed as a deletion, so it is stored as data: __peer_removed/{id} → {removed_at, removed_by}.

This is also the embedder-side instance of the fix wavekv's own docs name but decline to build — "the collector needs to remember what it collected". For this one key family, the memory is the marker.

The rest of the surface

• prune_removed_peers reads the marker too. The address tombstone still counts — it is the only signal a removal by an older binary leaves, until the collector eats it. A second watcher prunes on marker replication alone: a node that was offline for the removal may only ever receive the marker, the tombstone having been collected everywhere else before it returned.
• A corrupt marker fails closed as "removed". Refusing sync is recoverable — the operator overwrites or clears the record; admitting a removed node's full dump is not.
• RemoveNode reports membership from a pre-write read. Both the marker and the address tombstone wake watchers whose prune races the call — the marker is the sharper race, being the very first write — so removed_from_peer_set is captured before anything is written, and the racing-watcher guard test now drives both watchers at full speed.
• Re-admission is one explicit operator decision: SetNodeUrl clears the marker. That is the escape hatch for a mistaken removal; the proto comment now spells out when re-admitting an old data directory is safe (never absent across a collected deletion; when in doubt, wipe first).
• A node that finds its own marker at startup warns and proceeds. Best-effort only: the marker rarely replicates to its own victim — the refusals it drives are what keep it from arriving — the local copy may be stale after a re-admission, and the lockout is enforced by the peers either way. Making the refusal properly observable from both ends (metrics on the refusing peers and on the refused node itself, which can only learn of its removal from the 403s) is split into a follow-up PR stacked on this one.

Verification

cargo test -p dstack-gateway — 285 passed (after rebasing onto the updated #1099 branch). Five new tests, plus assertions added to two existing ones:

test	property
a_removal_marker_survives_the_collector	the full removal sequence plus a collection pass: the address tombstone is gone, the marker is what remains
a_marked_peer_is_pruned_even_when_the_address_tombstone_is_long_gone	the post-collection state prunes; an early-bootstrap peer (no marker, no tombstone) is left alone
clearing_the_marker_re_admits_the_node	clear → admitted again; idempotent, and the retry reports there was nothing to clear
a_corrupt_removal_marker_reads_as_removed	fail-closed
a_removed_nodes_envelopes_are_refused_at_the_door	route-level: both sync routes answer 403 to a marked sender, and clearing the marker opens the door again

Plus: the existing RemoveNode test asserts removal leaves the marker; the racing-watcher guard test (remove_node_reports_peer_membership_despite_a_racing_watcher) now also drives the marker watcher, guarding the pre-write membership capture against the race the marker itself introduces.

Residual risk

The lockout takes effect where the marker has replicated. In the round or so between RemoveNode and the marker reaching a given peer, that peer still answers the removed node — the same window the removal itself has always had, and it closes by replication rather than staying open forever. The protocol-level fix (wavekv remembering collected watermarks, making resurrection impossible regardless of membership games) remains upstream work.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.