fix: only the other member of the pair gates a promote; the others are told after (keel#137)

.github/workflows/tests.yml

@@ -104,6 +104,23 @@ jobs:
           etcd-server etcd-client util-linux
         KEEL_VIP_CLAIM_NETNS=1 PYTHONPATH=. python3 -m pytest -q -s \
           tests/test_vip_claim_netns.py
+  # keel vip promote in cloud simple with a member outside the pair down
+  # (keel#137), on the vip job's nodes and links, every spec in cloud
+  # simple: a planned promote takes the VIP in less than 3 s, and no two
+  # nodes carry it at one instant (tests/test_vip_simple_netns.py). The
+  # check is "vip-simple / trixie".
+  vip-simple:
+    uses: keel-linux/.github/.github/workflows/lxc-trixie.yml@main
+    with:
+      systemd: true
+      timeout: 30
+      run: |
+        apt-get update -qq
+        apt-get install -y -qq --no-install-recommends python3-pytest \
+          python3-yaml wireguard-tools iproute2 iputils-ping openssl \
+          etcd-server etcd-client util-linux
+        KEEL_VIP_SIMPLE_NETNS=1 PYTHONPATH=. python3 -m pytest -q -s \
+          tests/test_vip_simple_netns.py
   # Upgrades on a pair whose VIP is active (the maintainer's requirement
   # of 2026-10-10), on the nodes and links of the vip job: keel-vip.service
   # try-restarted on the holder and the replica as keel-overlay-vip's

debian/changelog

@@ -1,3 +1,22 @@
+keel (0.23.15) unstable; urgency=medium
+
+  * In cloud simple, one member outside the pair that is down made every
+    keel vip promote wait two 10 s timeouts, for its epoch and for the
+    announcement: a planned promote took about 21 s on a real pair
+    (keel#137). Now the other member of the pair alone gates the
+    handover: its epoch, its release and its taking the claim, each
+    waited for in full, as nothing fences a live old primary without
+    etcd. The members outside the pair are told after, in the
+    background, and what each said is logged. With --old-primary-gone
+    every peer is asked and told, and the claim is carried on a
+    majority of those that answer, as before. With etcd the peers are
+    told in the background too. A replica that holds the primary's
+    claim asks the release at the next epoch at once, and asks the
+    epoch only when the primary refuses that as stale: two round trips
+    with the other member, not three.
+
+ -- Marcos Méndez <[email protected]>  Sat, 10 Oct 2026 12:00:00 +0000
+
 keel (0.23.14) unstable; urgency=medium
 
   * keel vip promote failed with "etcd did not take the claim: etcdctl

docs/vip.md

@@ -184,21 +184,28 @@ in the next.
 On the node that is to become the primary, which must hold the pair
 record:
 
-1. it finds the newest claim: before etcd, every peer is asked its epoch
-   at once; with etcd, the counter;
+1. it finds the newest claim: before etcd, the claim of the other
+   member of the pair that this node holds, else that member is asked
+   its epoch (every peer with `--old-primary-gone`); a release refused
+   as stale asks the epoch, then the release again; with etcd, the
+   counter;
 2. when another node holds the VIP, it asks it to release, for the next
    epoch: **the old primary drops the address first and answers after**.
    When the old primary does not answer, **it refuses, unless
    `--old-primary-gone`**. With etcd and `--old-primary-gone`, it waits
    for the old primary's lease to expire, up to its TTL, and **never
    revokes another node's lease** (third round, point 5);
-3. it claims at the next epoch. Before etcd, the claim is announced to
-   every peer, and **this node carries the VIP only when a majority of
-   the peers that answered took it**; with a majority refusing, or none
-   answering without `--old-primary-gone`, it exits non-zero and does
-   not carry it. With `--old-primary-gone` and no peer answering (a mesh
-   of the two nodes alone, the other one gone), the operator's flag is
-   the acceptance. With
+3. it claims at the next epoch. Before etcd, **the other member of the
+   pair alone gates the handover** (keel#137): this node carries the VIP
+   only once that member took the claim; when it refuses or does not
+   answer, it exits non-zero and does not carry it. The members outside
+   the pair are told after, in the background, and what each said is
+   logged: one that is down no longer holds up the promote (two 10 s
+   timeouts on keel 0.23.13). With `--old-primary-gone` the other member
+   is declared gone, so every peer is asked and told, and **this node
+   carries the VIP only when a majority of the peers that answered took
+   it**; with no peer answering (a mesh of the two nodes alone, the
+   other one gone), the operator's flag is the acceptance. With
    etcd, the claim is the counter's compare-and-swap, and the controller
    adds the address once it renewed the claim's lease. A transaction
    that etcd does not answer in time may have committed (keel#135): the
@@ -210,7 +217,8 @@ record:
    leases of the tries that did not win are revoked, so a late
    transaction with one of them fails in etcd;
 4. each peer routes the VIP with one `wg set`; one that did not answer
-   takes it at its next check.
+   takes it at its next check. With etcd, the peers are told in the
+   background too.
 
 With `--old-primary-gone`, the operator declared the other member of
 the pair gone. **With etcd**, its lease fences it: each ask of it in

keel/__init__.py

@@ -12,7 +12,7 @@
 # an image's tree read offline): debian/changelog's and pyproject.toml's,
 # which tests/test_version.py keeps in step. Installed, the metadata
 # dpkg put beside the package answers (keel#98).
-FALLBACK_VERSION = "0.23.14"
+FALLBACK_VERSION = "0.23.15"
 
 
 def version() -> str:

keel/mesh/vipnode.py

@@ -495,12 +495,15 @@ def drop_fenced(here: Here, vip: str, why: str,
              f"{'; ' + problem if problem else ''}")
 
 
-def announce(here: Here, claim: Claim) -> dict[str, str]:
-    """The claim told to every peer at once, as the new holder does; each
-    peer's address to what it said: "applied", "known" (both accepted
-    it), "refused: why" (it answered no) or "unreachable: why"
+def announce(here: Here, claim: Claim,
+             only: set[str] | None = None) -> dict[str, str]:
+    """The claim told to every peer at once (or to those of `only`), as
+    the new holder does; each peer's address to what it said: "applied",
+    "known" (both accepted it), "refused: why" (it answered no) or
+    "unreachable: why"
     """
-    peers = [one.address for one in here.node.peers("")]
+    peers = [one.address for one in here.node.peers("")
+             if only is None or one.address in only]
 
     def one(at: str) -> str:
         try:
@@ -518,10 +521,13 @@ def one(at: str) -> str:
         return dict(zip(peers, pool.map(one, peers)))
 
 
-def epochs(here: Here, vip: str) -> dict[str, Claim | None | str]:
-    """Every peer asked, at once, for the newest claim it took of `vip`:
-    its address to that claim, verified, None for none, or why not"""
-    peers = [one.address for one in here.node.peers("")]
+def epochs(here: Here, vip: str,
+           only: set[str] | None = None) -> dict[str, Claim | None | str]:
+    """Every peer (or those of `only`) asked, at once, for the newest
+    claim it took of `vip`: its address to that claim, verified, None
+    for none, or why not"""
+    peers = [one.address for one in here.node.peers("")
+             if only is None or one.address in only]
 
     def one(at: str) -> Claim | None | str:
         try:

keel/mesh/vippromote.py

@@ -36,6 +36,7 @@
 """
 
 import json
+import threading
 from collections.abc import Callable
 from datetime import datetime
 
@@ -252,9 +253,146 @@ def released(here: Here, vip: str, newest: Claim, epoch: int, gone: bool,
     return True
 
 
+# the announcements to the members outside the pair, told after the
+# handover in the background (keel#137); `announced` waits for them.
+# BACKGROUND False tells them before the promote returns (the tests that
+# read the peers right after a promote)
+PENDING: list[threading.Thread] = []
+BACKGROUND = True
+
+
+def pair_peer(here: Here, vip: str) -> str | None:
+    """The other member of the pair's overlay address, or None"""
+    try:
+        record = vipnode.kept(here, vip)
+        own = here.own_key()
+    except VipError:
+        return None
+    return next((vipnode.peer_at(here, key) for key in record.members
+                 if not same_key(key, own)), None)
+
+
 def promoted(here: Here, vip: str, gone: bool,
              out: Callable[[str], None]) -> int:
-    """Before etcd: release, take, announce"""
+    """Before etcd: release, take, announce. The other member of the
+    pair alone gates the handover (keel#137): its epoch, its release,
+    its taking the claim, each waited for in full (nothing fences a live
+    old primary without etcd); the members outside the pair are told
+    after, in the background. With --old-primary-gone the other member
+    is declared gone, so every peer is asked and the claim is carried
+    on a majority of those that answer, as before"""
+    other = None if gone else pair_peer(here, vip)
+    if other is None:
+        return promoted_by_all(here, vip, gone, out)
+    own = here.own_key()
+    held = vipnode.current(here, vip)
+    # the other member's claim known here: the release is asked at the
+    # next epoch at once, and its epoch only when it refuses that as
+    # stale (two round trips with it, not three)
+    known = held.claim is not None and not same_key(held.claim.holder, own)
+    newest = held.claim if known else asked_epoch(here, vip, other, held,
+                                                  out)
+    if newest is not None and same_key(newest.holder, own) and \
+            held.holds(own) and vipnet.carried(
+                here.iface(), vip, here.node.output):
+        out(f"this node holds {vip} already, at epoch {newest.epoch}")
+        return exits.OK
+    epoch = (newest.epoch if newest else 0) + 1
+    if newest is not None and not same_key(newest.holder, own):
+        why = asked_release(here, vip, newest, epoch, out)
+        if why == STALE and known:
+            newest = asked_epoch(here, vip, other, held, out)
+            epoch = (newest.epoch if newest else 0) + 1
+            why = None if newest is None or same_key(newest.holder, own) \
+                else asked_release(here, vip, newest, epoch, out)
+        if why:
+            out(f"the primary at {newest.address} did not release it: {why}."
+                " Nothing was changed. Only you know whether that node is"
+                " gone: if it is, run keel vip promote --old-primary-gone")
+            return exits.MESH_REFUSED
+    made = vipnode.signed_claim(here, vip, epoch)
+    vipnode.hold(here, made, False)
+    what = vipnode.announce(here, made, {other}).get(
+        other, "unreachable: not a peer of this node")
+    out(f"  {other}: {what}")
+    if what not in ACCEPTED:
+        vipnode.release(here, vip, lambda lease: None)
+        out(f"the other member of the pair, at {other}, did not take the"
+            f" claim at epoch {epoch}: this node does not carry {vip}")
+        return exits.MESH_REFUSED
+    if not vipnode.carry_held(here, vip):
+        out(f"the VIP could not be added to {here.iface()}")
+        return exits.APPLY_FAILED
+    out(f"this node holds {vip} at epoch {epoch}, on {here.iface()},"
+        " taken by the other member of the pair; the other peers are told"
+        " in the background")
+    in_background(here, made, {other})
+    return exits.OK
+
+
+STALE = "stale"
+
+
+def asked_epoch(here: Here, vip: str, other: str, held,
+                out: Callable[[str], None]) -> Claim | None:
+    """The newest claim this node and the other member of the pair know"""
+    answers = vipnode.epochs(here, vip, {other})
+    for at, one in answers.items():
+        if isinstance(one, str):
+            out(f"  {at}: {one}")
+    claims = [one for one in answers.values() if isinstance(one, Claim)]
+    return vipnode.newest(claims + ([held.claim] if held.claim else []))
+
+
+def asked_release(here: Here, vip: str, newest: Claim, epoch: int,
+                  out: Callable[[str], None]) -> str | None:
+    """The holder asked to release, for `epoch`: None once it did, STALE
+    when it knows that epoch or a newer one, else why not"""
+    out(f"asking the primary, {newest.holder} at {newest.address}, to"
+        f" release {vip}…")
+    try:
+        here.say("release", newest.address, {"vip": vip, "epoch": epoch})
+    except (LinkError, VipError, SigningError) as e:
+        if "stale release" in str(e):
+            out(f"the primary at {newest.address} knows a newer epoch;"
+                " asking it")
+            return STALE
+        return str(e)
+    out(f"released by {newest.address}")
+    return None
+
+
+def in_background(here: Here, made: Claim, told_already: set[str]) -> None:
+    """The claim told to every other peer in a thread of its own, not a
+    daemon (the process ends after it), what each said logged"""
+    peers = {one.address for one in here.node.peers("")} - told_already
+    if not peers:
+        return
+
+    def tell() -> None:
+        for at, what in sorted(vipnode.announce(here, made, peers).items()):
+            here.err(f"vip {made.vip}: {at}: {what}")
+    if not BACKGROUND:
+        tell()
+        return
+    worker = threading.Thread(target=tell)
+    PENDING.append(worker)
+    worker.start()
+
+
+def announced(timeout: float | None = None) -> int:
+    """Wait for the announcements in the background; how many there were"""
+    done = 0
+    while PENDING:
+        PENDING.pop(0).join(timeout)
+        done += 1
+    return done
+
+
+def promoted_by_all(here: Here, vip: str, gone: bool,
+                    out: Callable[[str], None]) -> int:
+    """Every peer asked and told, the claim carried on a majority of the
+    peers that answer (the flag is the acceptance when none does)"""
     own = here.own_key()
     held = vipnode.current(here, vip)
     answers = vipnode.epochs(here, vip)
@@ -393,7 +531,9 @@ def promoted_etcd(here: Here, vip: str, gone: bool,
             f" {CARRY_WAIT:g} s: is it running? Its lease expires"
             f" {vipetcd.TTL} s after its last renewal")
         return exits.APPLY_FAILED
-    told(here, made[0], out)
+    # etcd holds the claim; the peers are told in the background (keel#137)
+    out("the peers are told in the background")
+    in_background(here, made[0], set())
     return exits.OK
 
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "keel"
-version = "0.23.14"
+version = "0.23.15"
 authors = [{name = "KeelLinux maintainers"}]
 description = "Declarative management of one appliance instance"
 readme = "README.md"