Repository navigation
Drill the failover gap of one pool address in front of two replicas #372
Description
What to do
Measure how long DNS goes unanswered when the node that announces a pool address fails, with two replicas behind that one address. Decide which conditions on the settled decisions change the model accepts that a Process spread across nodes behind a pool address is failover and not load sharing, on the condition that the gap is measured before the utility Project hands over.
In layer 2 mode all traffic for an address goes to one node, and under the Local traffic policy a replica on another node receives none. MetalLB documents failover after a node failure as "within a few seconds", and a client that caches the old hardware address waits longer (docs/research/unsupported-claims.md, claim C18). adguard-ha answers the same failure in the client today, with two addresses in the router. One address replaces that, so the whole LAN's DNS depends on this gap.
The drill needs a client on the Enschede LAN and stops a pod and then a node, so it is run with the maintainer present, against a throwaway Service on a spare pool address.
- Run two replicas of a UDP responder on two Enschede nodes, behind one LoadBalancer Service with the
Localtraffic policy. - From a LAN client, query the address several times a second and log every answer with its time.
- Delete the pod on the announcing node. Record the gap.
- Stop the announcing node, or its speaker and its pod together. Record the gap, from a client that held the old hardware address.
- Remove the Service and the workload.
Done when
- The gap is recorded for a pod failure and for a node failure.
- The result is posted here: either the gap is acceptable for DNS, or the number is given so the one-address decision can be reopened.
- Nothing the drill created is left in the cluster.
Notes
This runs before the utility Project (adguard-ha) hands over. It sits outside the map Map: re-establish deploy-kit's goals, rewrite its spec, and order the work until the estate is handed over, which is not held up by it.
Activity
ExtraToast commentedon Oct 29, 2026
Warning
The node-failure row and the verdict below are invalid. The drill's link cut on
enschede-pi-3never ran: its pod started three minutes late, after the drill had taken one stray lost query for the outage. pi-3 answered on both sides of that query, and Kubernetes never marked it NotReady. The pod-failure row (6.6 s) stands. A corrected run follows.Drill result: the failover gap of one pool address in front of two replicas
Run
372-20261029T112700Zofdrills/pool-address-failover.sh(estate a316c8b, with local changes), against kubectl contextpersonal, 2026-10-29 11:27–11:40 UTC.Two CoreDNS responders on
enschede-pi-3andenschede-pi-2, behind one LoadBalancer Service at192.168.0.93(pooldrill-372) withexternalTrafficPolicy: Local. A LAN client (192.168.0.204 on en0) askedversion.bindevery 100 ms with a 250 ms timeout; every answer names the node that gave it.enschede-pi-3deletedenschede-pi-3off the network192.168.0.93went fromnonetononeacross the node failure.enschede-pi-3NotReady not at all while it was off.probe.login the run's record.Verdict: pending the corrected node-failure run.
Done when
ExtraToast commentedon Oct 29, 2026
Drill result: the failover gap of one pool address in front of two replicas
Run
372-20261029T115711Zofdrills/pool-address-failover.sh(estate b8b9a48), against kubectl contextpersonal, 2026-10-29 11:57–12:07 UTC.Two CoreDNS responders on
enschede-pi-3andenschede-pi-2, behind one LoadBalancer Service at192.168.0.93(pooldrill-372) withexternalTrafficPolicy: Local. A LAN client (192.168.0.204 on en0) askedversion.bindevery 100 ms with a 250 ms timeout; every answer names the node that gave it.enschede-pi-3deletedenschede-pi-3off the networkenschede-pi-3NotReady 50s after it was taken off.probe.login the run's record.Verdict: the gap is acceptable for DNS; one pool address in front of two replicas stands as failover.
Done when