Repository navigation
feat(aws): make instance refresh safe on one-node clusters - #29
Merged
Merged
Conversation
A rolling refresh counted a new node healthy on EC2 status alone and terminated the old node, the only copy of the data, before the new one had joined Raft. Two lifecycle hooks, answered by a new `vault-utils lifecycle` daemon on each node, close that gap at any cluster size: - vault-join holds a new instance in Pending:Wait until it is unsealed, a Raft voter, and caught up. A node that never joins is abandoned after 30 minutes, which fails the refresh and rolls it back. - vault-leave holds a departing instance in Terminating:Wait until it has stepped down and removed itself from the peer set, so the survivors keep quorum. It terminates either way after two minutes. The operator policy gains sys/step-down and sys/storage/raft/remove-peer for the leave step. The ASG health check stays EC2 on purpose. Nodes now advertise api_addr as the URL clients use (user_vault_addr with a subdomain, else vault_addr) instead of https://vault.internal, which the NLB certificate never names. A standby redirect broke the web UI with "The request failed and the interceptors did not return an alternative response". Needs a fresh node AMI. Module 0.4.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A rolling instance refresh on
aws-ec2-vault-clustercounted a new node healthy on EC2 status alone and terminated the old node before the new one had joined Raft. On a one-node cluster that is the only copy of the data. It happened on thepreviews-sharedcluster during an upgrade; the new node sat uninitialized and the old root volume went with the instance.What
Two Auto Scaling lifecycle hooks, answered by a new
vault-utils lifecycledaemon (vault-lifecycle.service) on each node:vault-joinholds a new instance inPending:Waituntil it is unsealed, a Raft voter, and caught up. A node that never joins is abandoned after 30 minutes, which fails the refresh and rolls it back with the old node untouched.vault-leaveholds a departing instance inTerminating:Waituntil it has stepped down (if leader) and removed itself from the peer set, so the survivors keep quorum. It terminates either way after two minutes of retries; the sole voter is left in place.sys/step-downandsys/storage/raft/remove-peer. Goldens regenerated.autoscaling:DescribeAutoScalingInstances, andautoscaling:CompleteLifecycleActionscoped by the cluster tag (the group name is unknown at plan time).instance_warmup = 60; the hook, not warmup, proves the node joined. ASG health check staysEC2on purpose: anELBcheck would replace a one-node cluster on transient Vault unhealth.Rollout
Verification
go vet,go test -short ./..., Docker-backedTestOperatorBreakGlass,tofu fmt,tofu validate,tofu test(new hook assertion). Not exercised: a live instance refresh.Not fixed
Why the original join failed. The node that knew is gone. With the join gate a repeat is harmless and
journalctl -u vault-lifecyclewill show the join state next time.