Repository navigation
fix: the old primary turns its database read only before it releases the VIP (keel#138) - #142
Open
marcos-mendez wants to merge 1 commit into
Open
marcos-mendez wants to merge 1 commit into
marcos-mendez wants to merge 1 commit into
Conversation
marcos-mendez
force-pushed
the
fix/promote-announce-in-background
branch
from
October 10, 2026 18:19
9988ec4 to
22ab394
Compare
marcos-mendez
force-pushed
the
fix/planned-promote-quiesce
branch
from
October 10, 2026 18:19
e51ca7c to
544b2db
Compare
marcos-mendez
force-pushed
the
fix/promote-announce-in-background
branch
from
October 10, 2026 18:19
22ab394 to
191f4c4
Compare
marcos-mendez
force-pushed
the
fix/planned-promote-quiesce
branch
4 times, most recently
from
October 10, 2026 19:23
7bc35e3 to
c1ffb9f
Compare
…the VIP (keel#138) On a real pair, a write in flight at a planned promote waited 15 s for its answer, and the new primary took 10 s to take writes. The 10 s on the new primary: its follow waited for the VIP on wg0, which the promote added only after its announcement to web-2, a member outside the pair that was down, timed out. keel#137 takes that out of the handover. The write: it committed on the old primary and waited for the replica's acknowledgement (semi-synchronous), while the old primary dropped the VIP for the release, so its answer had no address to come from. vipserve.released now calls quiesced() before vipnode.release: on a node that declares a database, SET GLOBAL read_only = ON (dbreadonly.quiesce, at most 15 s) returns once the commits in flight ended, which the replica, still connected, acknowledges; then the address goes. No new write is taken. A failure is logged and the release goes on; the database follows the VIP's state after, as before. With etcd, the first measurement showed a 5.6 s planned write gap at 250 ms: the promote waited for the controller's next lease renewal before the VIP was on wg0. The majority granted the lease and wrote the claim with it after the moment the lease was asked for, as a renewal confirms it; so promoted_etcd now adds the address itself (carry_held with that age), for what is left of the release time, and the controller extends it once it renewed. carried_soon now waits for that renewal (the lease's time to live goes up), so a missing keel-vip.service is still reported. The peers are told as soon as the address is here, not after that check, so they route the VIP to it at once. A primary that released the VIP dropped the address and revoked its own lease before it answered, and the claim's compare-and-swap is on the counter alone: the promote now claims at once on the counter read before the release, and waits for the old lease only with --old-primary-gone when the old primary did not answer (measured: 4.6 s gap with the wait, at 250 ms). Tests: the read only comes before the address is dropped; the promote carries the address at once, a lease too old is left to the controller; a released primary's lease is not waited for; quiesced asks the database only where one is declared, and logs a failure; quiesce's statement, timeout and failures. In mariadb / trixie, two planned promotes (A to B and back) while C writes at the VIP: the longest gap between two acknowledged writes under 3 s, no write waiting 10 s, no acknowledged row lost.
marcos-mendez
force-pushed
the
fix/planned-promote-quiesce
branch
from
October 10, 2026 19:40
c1ffb9f to
afd6adb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #138. Stacked on #141 (both bump
debian/changelog); its base is #141's branch.What waited, and why
Evidence: Keel-Linux/handbook#63, 17:31:48 to 17:32:14.
wg0, which the promote adds only after its announcement. The announcement to web-2, a member outside the pair that was down, timed out (10 s). fix: only the other member of the pair gates a promote; the others are told after (keel#137) #141 (keel#137) takes that out of the handover.Change
keel/mesh/vipserve.py(released) now callsquiesced()beforevipnode.release. On a node that declares a database,SET GLOBAL read_only = ON(keel/system/dbreadonly.py,quiesce, at most 15 s) returns once the commits in flight have ended; the replica, still connected, acknowledges them. Only then does the address go, and no new write is taken. A failure is logged and the release goes on; the database follows the VIP's state after, as before.Also:
docs/replication.md,debian/changelog0.23.16.Tests
quiescedasks the database only where one is declared, and logs a failure.quiesce's statement, its timeout and its failures.tests/mariadb_netns.py: two planned promotes (A to B and back) while C writes at the VIP. For each: the longest gap between two acknowledged writes under 3 s, no write waiting 10 s, no acknowledged row lost. The measured numbers go here once CI ran.Test plan