feat(secrets): make verifying an age key backup a command - #40
Merged
Conversation
The private key at ~/.config/sops/age/keys.txt has one copy. If that disk dies, what is lost is not really the secrets — a password can be reset and a community string can be changed — but every device visit needed to do it: Grafana, the Alertmanager webhook, and four SNMP communities re-entered by hand on pfSense, the APC NMC, iLO and the MokerLink switch. The switch is the one that hurts, because its firmware will not persist a deletion from the community table and drops its SNMP agent on each attempt, which is why SECURITY.md still records an accepted residual there from the last rotation. Doing that again, unplanned, is the scenario this is against. The backup itself is manual and stays out of the repository. What is added is the part that is easy to get wrong. The obvious verification does not verify anything. On the host that already holds the key, `SOPS_AGE_KEY_FILE=<backup> sops -d secrets/observability.sops.yaml` passes whatever is in the file you name: sops also consults SOPS_AGE_KEY and the default ~/.config/sops/age/keys.txt. This was measured, not assumed — pointed at a freshly generated, entirely unrelated keypair with SOPS_AGE_KEY set, it exits 0 and prints every secret. A backup that has never been verified and one that has been verified wrongly are the same backup; only one of them feels safe. So scripts/verify-key-backup.sh clears SOPS_AGE_KEY, redirects HOME and XDG_CONFIG_HOME at an empty directory so a fallback finds nothing rather than silently succeeding, and refuses to run against the live key at all. It compares device and inode rather than paths, because the first version compared with `stat -c` and no -L, which reports the symlink instead of its target — a symlink straight back to the live key passed as a good backup, and the test that caught it is the reason the guard is written this way. Two further guards, both from watching how this fails rather than how it works: a decrypt failure alone cannot distinguish "wrong keypair" from "not an age file" and those have different fixes, so the recipient is checked against .sops.yaml first; and the exit status is not trusted alone — the plaintext is asserted to contain all six keys render-config.sh requires, the same reasoning bootstrap.sh already applies to its own encrypt. No secret value reaches stdout, a temp file, or another process's argv; the here-string that would have been the obvious way to search the plaintext is avoided because bash may back one with a temp file. The target sits under Maintenance next to snmp-verify rather than Validation. Everything under Validation is offline and safe for CI; this needs a private key on disk and must never end up inside `make validate`. Deliberately not done: no CI check, since CI holds no key and a check that always skips is noise. No roadmap entry — #11 says explicitly it is filed separately. No second age recipient, which is the stronger arrangement but re-keys the committed secrets file and is worth doing when the lab stops being a one-person project. And nothing here records where the backup lives: a public pointer to the location of the only key is worth less than the reminder would be. The three checkboxes on #11 are still the operator's to tick — this makes the middle one provable rather than performing it. Refs #11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Adds
make secrets-verify-backup KEY=<copy>, the script behind it, and a runbook for backing up the age private key off the monitoring host. Nothing that runs changes — no container, config or credential is touched.Why
~/.config/sops/age/keys.txtis the only copy of the key that decryptssecrets/observability.sops.yaml(#11). If that disk dies, the cost is not the secrets — it is re-entering four SNMP communities by hand on pfSense, the APC NMC, iLO and the switch whose firmware already fights rotation.The backup is manual and stays out of the repo. What is automated is proving it works, because the obvious verification does not verify anything: on the host that already holds the key,
SOPS_AGE_KEY_FILE=<backup> sops -d …also consultsSOPS_AGE_KEYand the default key path. Measured, not assumed — pointed at a freshly generated unrelated keypair withSOPS_AGE_KEYset, it exits 0 and prints every secret. The script clearsSOPS_AGE_KEY, redirectsHOME/XDG_CONFIG_HOMEat an empty directory, and refuses to run against the live key.Blast radius
None at runtime. The new target is under Maintenance next to
snmp-verify, deliberately outsidemake validate— it needs a private key on disk and CI has none.secrets/*.sops.yamlVerification
./scripts/validate.sh— all checks passed.shellcheckclean;markdownlint-cli20 issues across 23 files.Every failure mode exercised, since each is a way a verification could pass for the wrong reason:
KEY=the live keystatwithout-Lbug in the first version, which reported the link instead of its targetSOPS_AGE_KEYset to the real key/dev/shmok — 6/6 keys, shredded afterOutput of a passing run contains zero occurrences of any credential name or key material.
make validatepassesStill manual
The three boxes on #11 stay open: store the copy, verify it in its final location, confirm that location does not republish it. This makes the middle one provable rather than performing it. Deliberately nothing in the repo records where the backup lives.
🤖 Generated with Claude Code