Repository navigation
test: the unplanned failover of a cloud simple MariaDB pair, measured at 250 ms - #124
Merged
Merged
Conversation
marcos-mendez
force-pushed
the
fix/follow-writable-when-the-vip-lands
branch
from
October 9, 2026 05:55
e450cf5 to
a7bf219
Compare
marcos-mendez
force-pushed
the
test/mariadb-cloud-simple-crash
branch
from
October 9, 2026 05:55
455fcd7 to
dbd91da
Compare
marcos-mendez
force-pushed
the
fix/follow-writable-when-the-vip-lands
branch
from
October 9, 2026 06:18
a7bf219 to
0ab687a
Compare
marcos-mendez
force-pushed
the
test/mariadb-cloud-simple-crash
branch
from
October 9, 2026 06:18
dbd91da to
f9f8562
Compare
This was referenced Oct 9, 2026
Closed
marcos-mendez
force-pushed
the
fix/follow-writable-when-the-vip-lands
branch
from
October 9, 2026 06:53
0ab687a to
3cad7e8
Compare
marcos-mendez
force-pushed
the
test/mariadb-cloud-simple-crash
branch
from
October 9, 2026 06:53
f9f8562 to
029cea0
Compare
marcos-mendez
changed the base branch from
fix/follow-writable-when-the-vip-lands
to
main
October 9, 2026 08:36
marcos-mendez
force-pushed
the
test/mariadb-cloud-simple-crash
branch
from
October 9, 2026 08:38
029cea0 to
8f0549d
Compare
…#118) After keel vip promote --old-primary-gone on a real pair, the new primary refused writes for 11.7 s while it carried the VIP. keel database follow promoted the database 7.5 s after the VIP landed. The cause is STOP SLAVE IO_THREAD: it returns only when the I/O thread ends. Measured on MariaDB 11.8 with the primary gone: a thread that tries to connect ends when the try times out (slave_net_timeout, 10 s; 2.96 s with 3 s); a connected thread with semi-synchronous replication tries to connect to the primary to end its dump thread (5.06 s, 0.07 s without semi-synchronous replication). A KILL of the thread does not shorten either. The status shows "Ending IO thread" at once: the thread receives nothing from the moment the stop marks it killed. keel.system.dbreadonly.promote now runs the stop in a client of its own. Once the stop is in the server's killing phase (its process list STATE is "Killing slave"; INFO is set at dispatch, before the stop takes its lock) or ended without an error, the SQL thread drains, the received position and the killing phase are checked again, and read_only goes off; STOP SLAVE and RESET SLAVE ALL follow while the server already takes writes, and the stop is waited for. The SQL thread is not stopped first: it has nothing more to apply, and STOP SLAVE SQL_THREAD waits behind the I/O thread's stop (4.7 s measured). Against a frozen primary on 11.8 the server took writes 0.35 s after the promotion started (0.26 s with the thread connecting). When the position moved after the drain, or cannot be read, the old order applies: the stop waited for, the drain again, then the promotion. Follow looks for the VIP on wg0 every 0.2 s, not 0.5 s. A planned promote takes the same path; its stop ends at once. Spec apply's SeedReplica and PromoteReplica take follow's lock, so apply, follow and watch never interleave. A stop client that failed after the server may have taken the statement starts the I/O thread again. Tests: the writes before the stop ends; a stop not in the killing phase after the drain takes the old order; anything received after the drain takes the old order; a stop the server does not show is waited for first; each failure. In mariadb / trixie the promote no longer holds the agent's lock, as on a machine, where keel-database-follow.path runs follow while the promote goes on; the crash case checks that the server takes writes within 3 s after the VIP lands, with no acknowledged row lost. The crash case kills A only once C's writes were acknowledged, and runs the promote again, as an operator does, when etcd had no leader to answer (A, cut off, may have been etcd's leader), at most three times, the attempts kept.
… at 250 ms The real pair of the rollouts was in cloud simple: the VIP moves by the members' channel alone, with no etcd lease. The mariadb / trixie job runs with etcd, where the old primary's lease (20 s, decision 0049) comes before every claim. tests/mariadb_simple_netns.py runs the mariadb job's nodes and links (250 ms +-25 ms, 2 % loss, measured first) with every spec in cloud simple and etcd disabled. A primary, B replica, connected and semi-synchronous; C writes one row at the VIP every 0.1 s. A's server is killed and A is cut off; exactly 2 s after, keel database promote --old-primary-gone on B. Without etcd the promote waits the members' channel's full timeout for each of its three asks of the dead old primary (the safety review of keel#108), so the writes at the VIP must come back within 15 s plus three times 10 s of the promote's start, with no acknowledged row lost. The new job is "mariadb-simple / trixie".
marcos-mendez
force-pushed
the
test/mariadb-cloud-simple-crash
branch
from
October 9, 2026 08:53
8f0549d to
0be60b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #122 (rebased on main after #121 and #127). Measures #108 and #118 in cloud simple, the mode of the real pair: no etcd lease, the VIP moved by the members' channel.
Case
tests/mariadb_simple_netns.pyon themariadbjob's nodes and links (250 ms ±25 ms, 2 % loss, measured first), every spec in cloud simple with etcd disabled. A primary, B replica, connected and semi-synchronous; C writes one row at the VIP every 0.1 s. A's server killed and A cut off; exactly 2 s later,keel database promote --old-primary-goneon B. New job:mariadb-simple / trixie.Target
15 s from the promote's start, plus what the safety review of #121 keeps without etcd: the promote waits the channel's full 10 s connect timeout for each of its three asks of the dead old primary (epoch, release, announcement). So the assert is 15 + 3 × 10 = 45 s, and 0 acknowledged rows lost.
Measured (0be60b6)
Of the 32.5 s, 30 s are the three full timeouts to the dead old primary. Without them the case is at about 3 s from the promote's start. With the 1.5 s bound (the first version of #121, before the safety review) the same case measured 10.7 s.
CI record
Green at 0be60b6 at the first run, no rerun. The case kills A only once C's writes were acknowledged. 8f0549d (the rebase on #127) failed
mariadb / trixie(#122's harness, fixed there) andvip / trixieatg_promote_b(etcd did not answer, the known failure). Earlier heads: see the history of this PR.