fix: an old primary stays read only until it proves it holds the VIP (keel#104)

[marcos-mendez]

Fixes #104 (bug B2 of the real-node MariaDB pair test, Keel-Linux/handbook#60).

Problem

After a crash and a boot, the old primary read its own claim of the VIP from /var/lib/keel/vip and keel database follow set read_only OFF. The other node held the VIP at a newer epoch. The old primary was writable for 31 s, until keel vip check learned the newer epoch. A write in that window makes an errant GTID.

Change

• keel/system/dbfollow.py: on a node whose newest claim is its own, follow lifts read_only only when the VIP is on wg0. keel adds the VIP to wg0 only after a proof: keel vip promote after its new claim; keel vip check when a peer answered and no peer knows a newer claim; with etcd, the controller after a lease renewal that the majority confirmed (the kernel valid_lft ends it). A boot removes the address. Without the proof, follow sets read_only ON and the root lock, and writes no primary role file. Follow waits up to 15 s (PROOF_WAIT) for the address, because a promote records its claim before it adds the VIP.
• keel/mesh/vipnode.py (carry_held): when keel adds the VIP to wg0, it writes the VIP's state again, so keel-database-follow.path runs follow again. keel database watch also runs follow again within 30 s.
• No claim on a paired node (the VIP state file removed, or before the first promote): not proven either. A declared primary stays read only and locked, with no primary role file.
• keel/system/dbwatch.py (needs_follow): follows a server again when it takes writes with no claim, or while the VIP is not on wg0 (its valid_lft ran out).
• keel/system/dbfollow.py (follow): one run at a time, under /var/lib/keel/database/follow.lock, so the unit and keel database watch never interleave between the proof and SET read_only OFF.
• Without etcd the address has no lifetime, so the proof is weaker: keel vip check: without etcd, the proof that a node holds the VIP is weak #113 records it, and docs/replication.md says so.
• docs/replication.md: the follow table and the rule.
• debian/changelog, pyproject.toml, keel/__init__.py: 0.23.4.

Tests

• tests/test_system_dbfollow.py, TestTheOldPrimaryAtBoot: crash, the other node promoted at a newer epoch, the old node boots, stays read only, then follows as a replica (cloud simple, as on the real mesh); a server found writable without a proof is made read only; the holder proven again after its boot takes writes; no peer answers; the proof comes while follow waits; etcd does not answer at boot. The old code fails five of these with SET GLOBAL read_only = OFF.
• TestNoClaimAndNoAddress: the state file removed on a primary with a pair record, follow keeps it read only; watch follows a writable primary whose VIP left wg0; follow takes its lock.
• tests/test_mesh_vipmove.py: the state is written again when keel vip check adds the VIP, and not when it is there.
• tests/mariadb_netns.py: A now takes the VIP (and follow runs) before the application's data is written, because a paired node with no claim is read only.
• tests/mariadb_netns.py (mariadb / trixie): the primary's server killed, its VIP helper stopped, cut off; keel database promote --old-primary-gone on the replica; the old primary boots with its old claim: @@read_only is 1 at boot, after follow, and in every 0.5 s sample until it replicates from the new primary over TLS by GTID.

Test plan

• tests / coverage
• mariadb / trixie (the new crash case)
• vip, vip-upgrade, etcd, etcd-auth, mesh / trixie
• package / changelog

CI record

First run (617bfbb's first version): mariadb / trixie failed in the new case only: the replica's I/O thread was still Connecting when the status was read (read only held in every sample, replica after 6.4 s). The case now waits for the I/O thread to stream, as case (a) does (amended). vip / trixie failed at g_promote_b (etcdctl get: context deadline exceeded), the same failure as main's run 37833124848 and run 37843439840. No job was rerun: the amended push ran all checks again, all green.

Security review: H1, M2 and L1 are fixed in the amended commit; M1 is #113.

Head 7cbfab1 (after the security review): every check green but mesh / trixie. Attempt 1 timed out in test_a_spent_invite_is_refused (the inviter process). I reran the job once (run 37869774880, attempt 2); it failed in another test, test_c_joins_through_the_fallback, the same failure as main's run 37833124848. This PR changes nothing in keel.mesh's join or invite code. #112 holds this commit plus its own, and its mesh / trixie passed at 132a13a. I did not rerun a second time.