feat(secrets): make verifying an age key backup a command

Makefile

@@ -219,6 +219,18 @@ snmp-verify: ## Check each SNMP device answers to its community (ARGS=--old)
 	@# it into `make validate`.
 	./scripts/snmp-verify.sh $(ARGS)
 
+.PHONY: secrets-verify-backup
+secrets-verify-backup: ## Check a backup age key decrypts the secrets (KEY=/path/to/keys.txt)
+	@# Under Maintenance rather than Validation for the same reason as
+	@# snmp-verify: everything under Validation is offline and safe for CI,
+	@# and this needs a private key on disk. It must never end up inside
+	@# `make validate`, where it would either always skip or ask CI for a key.
+	@#
+	@# KEY rather than ARGS because there is exactly one argument and it is
+	@# required — an empty ARGS would reach the script as no argument at all
+	@# and print usage, which reads like the target is broken.
+	./scripts/verify-key-backup.sh "$(KEY)" $(STACK)
+
 .PHONY: certs
 certs: ## Create the internal CA / issue a leaf (ARGS="--host x.matrix.elysium --ip 10.0.0.1")
 	@# certificates/ is gitignored. Nothing in the stack terminates TLS yet —

README.md

@@ -58,7 +58,7 @@ incident.
   `sha256:` digest, so a moved tag cannot change what deploys. CI enforces it;
   `make pin-digests` re-resolves them from the registry.
 - **Documented decisions and runbooks.** Five ADRs covering what was chosen and
-  what was rejected; five runbooks for the operations that are easy to get wrong
+  what was rejected; six runbooks for the operations that are easy to get wrong
   at 1am.
 
 ## Architecture
@@ -137,7 +137,7 @@ the internet and nothing more. Full topology and data flow in
 │   ├── architecture.md  network.md  hardware.md
 │   ├── observability.md  security.md  roadmap.md
 │   ├── adr/                  # 5 architecture decision records
-│   └── runbooks/             # deploy, add device, rotate creds, certs, purge history
+│   └── runbooks/             # deploy, add device, rotate creds, certs, key backup, purge
 └── Makefile                  # make help
 ```
 

docs/runbooks/back-up-the-age-key.md

@@ -0,0 +1,167 @@
+# Runbook: Back up the age private key
+
+**Target:** `~/.config/sops/age/keys.txt` on the monitoring host (10.0.99.20)
+**Time:** five minutes, once
+**You will need:** shell access to the monitoring host, and somewhere durable to
+put a copy — a password manager, or paper and a place to keep it
+
+This is the highest-value five minutes in the repository, and the only one whose
+downside is unbounded.
+
+## Why
+
+That file is the sole copy of the private key that decrypts
+[`secrets/observability.sops.yaml`](../../secrets/observability.sops.yaml). The
+public half in [`.sops.yaml`](../../.sops.yaml) is committed and can only
+encrypt; nothing in this repository, and nothing in any backup of this
+repository, can recover the private half.
+
+What is lost with that disk is not really the secrets — a password can be reset
+and a community string can be changed. What is lost is every device visit needed
+to do it: the Grafana admin password, the Alertmanager webhook, and four SNMP
+communities re-entered by hand on pfSense, the APC NMC, HPE iLO and the
+MokerLink switch. The switch is the one that hurts. Its firmware will not
+persist a deletion from the community table and drops its SNMP agent on each
+attempt, which is why [`SECURITY.md`](../../SECURITY.md) still records an
+accepted residual there from the last rotation. Doing that a second time,
+unplanned, because a 2012 MacBook Pro's disk failed, is the scenario this
+avoids.
+
+Background on the scheme itself is in
+[`secrets/README.md`](../../secrets/README.md).
+
+## 1. Copy the key
+
+```bash
+cat ~/.config/sops/age/keys.txt
+```
+
+Two lines of plain ASCII: a `# public key: age1…` comment and one
+`AGE-SECRET-KEY-1…` line. The secret line alone is sufficient — the comment is a
+convenience, and `age-keygen -y` can regenerate it from the secret half at any
+time.
+
+Put it somewhere that is not this machine:
+
+| Where | Good for | Watch out for |
+| --- | --- | --- |
+| Password manager, as a secure note | The default choice — encrypted, backed up, searchable | Losing the master password loses this too. Title it so you can find it in five years |
+| Printed, in a safe or with documents | Survives every digital failure at once | A printer with a spool, and a photocopier's memory |
+| Encrypted USB stick, stored elsewhere | Fast restore | Flash cells fade unpowered; re-verify yearly |
+
+Record *what it is and what it unlocks* alongside it. A bare
+`AGE-SECRET-KEY-1…` string found in a password manager in three years is
+indistinguishable from junk.
+
+The location itself is deliberately **not** written down in this repository. A
+public pointer to where the only key lives is worth less than the reminder it
+would provide.
+
+## 2. Verify it actually decrypts
+
+Not optional, and not the same as looking at it. From a checkout of this repo:
+
+```bash
+make secrets-verify-backup KEY=/path/to/the/copy
+```
+
+Expect:
+
+```console
+-- recipient matches .sops.yaml: age1yrdu996…
+-- decrypting observability.sops.yaml with the backup key only
+ok — keys.txt decrypts secrets/observability.sops.yaml (6/6 keys)
+```
+
+What it proves: the key parses, its public half is the recipient the file was
+encrypted to, it decrypts the real ciphertext, and the result contains all six
+keys `render-config.sh` requires. No secret value is printed, written to a
+temporary file, or passed to another process.
+
+What it deliberately does not prove: that where you put the copy will still
+exist after a fire, a theft, or a forgotten master password. That part is
+judgement, not a check.
+
+> **Why not just run `sops -d` by hand.** Because on the host that already holds
+> the key, it passes no matter what. `SOPS_AGE_KEY_FILE=<backup> sops -d …` also
+> consults `SOPS_AGE_KEY` and the default `~/.config/sops/age/keys.txt`; pointed
+> at a freshly generated, completely unrelated keypair it exits 0 and prints
+> every secret. [`verify-key-backup.sh`](../../scripts/verify-key-backup.sh)
+> clears `SOPS_AGE_KEY`, redirects `HOME` and `XDG_CONFIG_HOME` at an empty
+> directory, and refuses to run against the live key file at all — so a pass
+> means the backup did the work.
+
+### Verifying a paper backup
+
+Type it **back in** and verify that file. Verifying the copy you pasted from is
+verifying your clipboard; verifying the copy you typed is verifying your
+handwriting, which is the thing that will actually fail.
+
+```bash
+umask 077
+vi /dev/shm/restore-test.txt          # transcribe from the paper
+make secrets-verify-backup KEY=/dev/shm/restore-test.txt
+shred -u /dev/shm/restore-test.txt
+```
+
+age keys transcribe better than most secrets: uppercase Bech32, so the data part
+contains no lowercase, and no `1`, `b`, `i` or `o` at all. The `0`/`O` and
+`1`/`l` confusions that ruin handwritten passwords cannot occur here. A
+mis-transcription is caught by the checksum in the key itself, and the script
+reports it as "not a readable age identity file" rather than a decrypt failure.
+
+## 3. Confirm the copy is not republishing itself
+
+The third thing that goes wrong: a backup that is durable and also public.
+
+- **Not in a git repository.** `.gitignore` here matches `keys.txt` and
+  `age.key`, which protects nothing outside this tree and nothing under a
+  different filename.
+- **Not in a synced folder.** Dropbox, OneDrive, Google Drive, iCloud,
+  Nextcloud, Syncthing — a copy there is a copy on someone else's disk.
+  `make secrets-verify-backup` warns when the path looks like one, but it can
+  only see the path, not what the folder actually syncs to.
+- **Not on the monitoring host.** A second copy on the same disk is not a
+  backup. The script refuses to verify the live key for this reason.
+
+## Restoring on a new host
+
+```bash
+mkdir -p ~/.config/sops/age
+vi ~/.config/sops/age/keys.txt        # paste the backup
+chmod 600 ~/.config/sops/age/keys.txt
+git clone https://github.com/Gerrrt/HomeLab.git && cd HomeLab
+make render                           # decrypts; this is the proof it worked
+make up
+```
+
+**Do not run `make secrets-init` to restore.** It generates a *new* keypair
+whose public half does not match `.sops.yaml`, and then nothing decrypts. It
+refuses to overwrite an existing key or an existing encrypted secrets file, so
+the damage is recoverable — but the error it produces afterwards
+(`sops metadata not found`, or a failed decrypt) sends you looking in the wrong
+place. Restore is a file copy, nothing more.
+
+## What this still does not solve
+
+One key, one person, one copy plus the original. If the answer to "who else can
+recover this" needs to be more than one, the mechanism already exists: generate a
+second keypair that lives only offline, add its public half to `.sops.yaml` as an
+additional recipient, and re-key with
+`sops updatekeys secrets/observability.sops.yaml` from a host that can already
+decrypt. That is the same procedure as bringing a second host in, described in
+[`secrets/README.md`](../../secrets/README.md). It is worth doing on the day the
+lab stops being a one-person project, and not before — every extra recipient is
+another key that can leak.
+
+## If something goes wrong
+
+| Symptom | Cause | Fix |
+| --- | --- | --- |
+| `that is the live key on this host, not a backup of it` | `KEY=` points at `~/.config/sops/age/keys.txt`, or a symlink or hard link to it | Point it at the copy. This is the check working |
+| `this is a valid age key, but not one the secrets are encrypted to` | A different keypair — usually one `age-keygen` made by accident | Find the right backup. If there is none, the secrets must be rotated on the devices |
+| `is not a readable age identity file` | Truncated, wrapped, or mis-transcribed secret line | Re-copy. Check the whole `AGE-SECRET-KEY-1…` line landed on one line |
+| `decryption failed with this key` after the recipient matched | Right public half, damaged secret half — a partial copy | Re-copy from the original while the host still exists |
+| `the backup is inside this repository` | The copy was written into the working tree | Move it out, then `git log --all -- <path>` to confirm it was never committed |
+| `no encrypted secrets at secrets/observability.sops.yaml` | Wrong working directory, or a fresh clone that never ran `make secrets-init` | Run from the repo root |
+| `mode 644 — group or other can read this copy` | A copy written without `umask 077` | `chmod 600` it, and consider it seen by anything else on that machine |

docs/security.md

@@ -45,6 +45,12 @@ assumption consistent with what they are.
   and committed in encrypted form. See [`secrets/README.md`](../secrets/README.md).
 - The private key lives at `~/.config/sops/age/keys.txt` on the deployment host
   and is never in the repository.
+- That key is the single point of failure for every encrypted secret here, so it
+  is copied off the host and the copy is proven to decrypt with
+  `make secrets-verify-backup KEY=<copy>` — which refuses to run against the live
+  key and blanks the environment first, because the obvious hand-typed
+  equivalent passes even for an unrelated keypair. See
+  [`runbooks/back-up-the-age-key.md`](runbooks/back-up-the-age-key.md).
 - `scripts/render-config.sh` decrypts at deploy time into gitignored files.
   Nothing writes a plaintext secret into a tracked path.
 - CI runs `gitleaks` with rules specifically for SNMP communities, inline

scripts/bootstrap.sh

@@ -40,6 +40,8 @@ else
   chmod 600 "${KEY_FILE}"
   warn "BACK THIS FILE UP OFF THIS MACHINE. Without it the encrypted secrets"
   warn "in this repository are unrecoverable."
+  warn "  docs/runbooks/back-up-the-age-key.md"
+  warn "  make secrets-verify-backup KEY=<the copy>   # proves the copy decrypts"
 fi
 
 PUBLIC_KEY="$(grep -oE 'age1[a-z0-9]+' "${KEY_FILE}" | head -n1)"
@@ -163,4 +165,11 @@ Next steps:
   1. make secrets-edit     # replace every change-me value
   2. make validate         # confirm the configs are sound
   3. make up               # render config and start the stack
+
+And the one that has no second chance — back up ${KEY_FILE} off this
+machine, then prove the copy works:
+
+  make secrets-verify-backup KEY=/path/to/the/copy
+
+  docs/runbooks/back-up-the-age-key.md
 EOF