diff --git a/Makefile b/Makefile index f3834a2..195d09f 100644 --- a/Makefile +++ b/Makefile @@ -219,6 +219,18 @@ snmp-verify: ## Check each SNMP device answers to its community (ARGS=--old) @# it into `make validate`. ./scripts/snmp-verify.sh $(ARGS) +.PHONY: secrets-verify-backup +secrets-verify-backup: ## Check a backup age key decrypts the secrets (KEY=/path/to/keys.txt) + @# Under Maintenance rather than Validation for the same reason as + @# snmp-verify: everything under Validation is offline and safe for CI, + @# and this needs a private key on disk. It must never end up inside + @# `make validate`, where it would either always skip or ask CI for a key. + @# + @# KEY rather than ARGS because there is exactly one argument and it is + @# required — an empty ARGS would reach the script as no argument at all + @# and print usage, which reads like the target is broken. + ./scripts/verify-key-backup.sh "$(KEY)" $(STACK) + .PHONY: certs certs: ## Create the internal CA / issue a leaf (ARGS="--host x.matrix.elysium --ip 10.0.0.1") @# certificates/ is gitignored. Nothing in the stack terminates TLS yet — diff --git a/README.md b/README.md index bec58d9..fcbdbcb 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ incident. `sha256:` digest, so a moved tag cannot change what deploys. CI enforces it; `make pin-digests` re-resolves them from the registry. - **Documented decisions and runbooks.** Five ADRs covering what was chosen and - what was rejected; five runbooks for the operations that are easy to get wrong + what was rejected; six runbooks for the operations that are easy to get wrong at 1am. ## Architecture @@ -137,7 +137,7 @@ the internet and nothing more. Full topology and data flow in │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md │ ├── adr/ # 5 architecture decision records -│ └── runbooks/ # deploy, add device, rotate creds, certs, purge history +│ └── runbooks/ # deploy, add device, rotate creds, certs, key backup, purge └── Makefile # make help ``` diff --git a/docs/runbooks/back-up-the-age-key.md b/docs/runbooks/back-up-the-age-key.md new file mode 100644 index 0000000..f120369 --- /dev/null +++ b/docs/runbooks/back-up-the-age-key.md @@ -0,0 +1,167 @@ +# Runbook: Back up the age private key + +**Target:** `~/.config/sops/age/keys.txt` on the monitoring host (10.0.99.20) +**Time:** five minutes, once +**You will need:** shell access to the monitoring host, and somewhere durable to +put a copy — a password manager, or paper and a place to keep it + +This is the highest-value five minutes in the repository, and the only one whose +downside is unbounded. + +## Why + +That file is the sole copy of the private key that decrypts +[`secrets/observability.sops.yaml`](../../secrets/observability.sops.yaml). The +public half in [`.sops.yaml`](../../.sops.yaml) is committed and can only +encrypt; nothing in this repository, and nothing in any backup of this +repository, can recover the private half. + +What is lost with that disk is not really the secrets — a password can be reset +and a community string can be changed. What is lost is every device visit needed +to do it: the Grafana admin password, the Alertmanager webhook, and four SNMP +communities re-entered by hand on pfSense, the APC NMC, HPE iLO and the +MokerLink switch. The switch is the one that hurts. Its firmware will not +persist a deletion from the community table and drops its SNMP agent on each +attempt, which is why [`SECURITY.md`](../../SECURITY.md) still records an +accepted residual there from the last rotation. Doing that a second time, +unplanned, because a 2012 MacBook Pro's disk failed, is the scenario this +avoids. + +Background on the scheme itself is in +[`secrets/README.md`](../../secrets/README.md). + +## 1. Copy the key + +```bash +cat ~/.config/sops/age/keys.txt +``` + +Two lines of plain ASCII: a `# public key: age1…` comment and one +`AGE-SECRET-KEY-1…` line. The secret line alone is sufficient — the comment is a +convenience, and `age-keygen -y` can regenerate it from the secret half at any +time. + +Put it somewhere that is not this machine: + +| Where | Good for | Watch out for | +| --- | --- | --- | +| Password manager, as a secure note | The default choice — encrypted, backed up, searchable | Losing the master password loses this too. Title it so you can find it in five years | +| Printed, in a safe or with documents | Survives every digital failure at once | A printer with a spool, and a photocopier's memory | +| Encrypted USB stick, stored elsewhere | Fast restore | Flash cells fade unpowered; re-verify yearly | + +Record *what it is and what it unlocks* alongside it. A bare +`AGE-SECRET-KEY-1…` string found in a password manager in three years is +indistinguishable from junk. + +The location itself is deliberately **not** written down in this repository. A +public pointer to where the only key lives is worth less than the reminder it +would provide. + +## 2. Verify it actually decrypts + +Not optional, and not the same as looking at it. From a checkout of this repo: + +```bash +make secrets-verify-backup KEY=/path/to/the/copy +``` + +Expect: + +```console +-- recipient matches .sops.yaml: age1yrdu996… +-- decrypting observability.sops.yaml with the backup key only +ok — keys.txt decrypts secrets/observability.sops.yaml (6/6 keys) +``` + +What it proves: the key parses, its public half is the recipient the file was +encrypted to, it decrypts the real ciphertext, and the result contains all six +keys `render-config.sh` requires. No secret value is printed, written to a +temporary file, or passed to another process. + +What it deliberately does not prove: that where you put the copy will still +exist after a fire, a theft, or a forgotten master password. That part is +judgement, not a check. + +> **Why not just run `sops -d` by hand.** Because on the host that already holds +> the key, it passes no matter what. `SOPS_AGE_KEY_FILE= sops -d …` also +> consults `SOPS_AGE_KEY` and the default `~/.config/sops/age/keys.txt`; pointed +> at a freshly generated, completely unrelated keypair it exits 0 and prints +> every secret. [`verify-key-backup.sh`](../../scripts/verify-key-backup.sh) +> clears `SOPS_AGE_KEY`, redirects `HOME` and `XDG_CONFIG_HOME` at an empty +> directory, and refuses to run against the live key file at all — so a pass +> means the backup did the work. + +### Verifying a paper backup + +Type it **back in** and verify that file. Verifying the copy you pasted from is +verifying your clipboard; verifying the copy you typed is verifying your +handwriting, which is the thing that will actually fail. + +```bash +umask 077 +vi /dev/shm/restore-test.txt # transcribe from the paper +make secrets-verify-backup KEY=/dev/shm/restore-test.txt +shred -u /dev/shm/restore-test.txt +``` + +age keys transcribe better than most secrets: uppercase Bech32, so the data part +contains no lowercase, and no `1`, `b`, `i` or `o` at all. The `0`/`O` and +`1`/`l` confusions that ruin handwritten passwords cannot occur here. A +mis-transcription is caught by the checksum in the key itself, and the script +reports it as "not a readable age identity file" rather than a decrypt failure. + +## 3. Confirm the copy is not republishing itself + +The third thing that goes wrong: a backup that is durable and also public. + +- **Not in a git repository.** `.gitignore` here matches `keys.txt` and + `age.key`, which protects nothing outside this tree and nothing under a + different filename. +- **Not in a synced folder.** Dropbox, OneDrive, Google Drive, iCloud, + Nextcloud, Syncthing — a copy there is a copy on someone else's disk. + `make secrets-verify-backup` warns when the path looks like one, but it can + only see the path, not what the folder actually syncs to. +- **Not on the monitoring host.** A second copy on the same disk is not a + backup. The script refuses to verify the live key for this reason. + +## Restoring on a new host + +```bash +mkdir -p ~/.config/sops/age +vi ~/.config/sops/age/keys.txt # paste the backup +chmod 600 ~/.config/sops/age/keys.txt +git clone https://github.com/Gerrrt/HomeLab.git && cd HomeLab +make render # decrypts; this is the proof it worked +make up +``` + +**Do not run `make secrets-init` to restore.** It generates a *new* keypair +whose public half does not match `.sops.yaml`, and then nothing decrypts. It +refuses to overwrite an existing key or an existing encrypted secrets file, so +the damage is recoverable — but the error it produces afterwards +(`sops metadata not found`, or a failed decrypt) sends you looking in the wrong +place. Restore is a file copy, nothing more. + +## What this still does not solve + +One key, one person, one copy plus the original. If the answer to "who else can +recover this" needs to be more than one, the mechanism already exists: generate a +second keypair that lives only offline, add its public half to `.sops.yaml` as an +additional recipient, and re-key with +`sops updatekeys secrets/observability.sops.yaml` from a host that can already +decrypt. That is the same procedure as bringing a second host in, described in +[`secrets/README.md`](../../secrets/README.md). It is worth doing on the day the +lab stops being a one-person project, and not before — every extra recipient is +another key that can leak. + +## If something goes wrong + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `that is the live key on this host, not a backup of it` | `KEY=` points at `~/.config/sops/age/keys.txt`, or a symlink or hard link to it | Point it at the copy. This is the check working | +| `this is a valid age key, but not one the secrets are encrypted to` | A different keypair — usually one `age-keygen` made by accident | Find the right backup. If there is none, the secrets must be rotated on the devices | +| `is not a readable age identity file` | Truncated, wrapped, or mis-transcribed secret line | Re-copy. Check the whole `AGE-SECRET-KEY-1…` line landed on one line | +| `decryption failed with this key` after the recipient matched | Right public half, damaged secret half — a partial copy | Re-copy from the original while the host still exists | +| `the backup is inside this repository` | The copy was written into the working tree | Move it out, then `git log --all -- ` to confirm it was never committed | +| `no encrypted secrets at secrets/observability.sops.yaml` | Wrong working directory, or a fresh clone that never ran `make secrets-init` | Run from the repo root | +| `mode 644 — group or other can read this copy` | A copy written without `umask 077` | `chmod 600` it, and consider it seen by anything else on that machine | diff --git a/docs/security.md b/docs/security.md index c3e22d4..8242497 100644 --- a/docs/security.md +++ b/docs/security.md @@ -45,6 +45,12 @@ assumption consistent with what they are. and committed in encrypted form. See [`secrets/README.md`](../secrets/README.md). - The private key lives at `~/.config/sops/age/keys.txt` on the deployment host and is never in the repository. +- That key is the single point of failure for every encrypted secret here, so it + is copied off the host and the copy is proven to decrypt with + `make secrets-verify-backup KEY=` — which refuses to run against the live + key and blanks the environment first, because the obvious hand-typed + equivalent passes even for an unrelated keypair. See + [`runbooks/back-up-the-age-key.md`](runbooks/back-up-the-age-key.md). - `scripts/render-config.sh` decrypts at deploy time into gitignored files. Nothing writes a plaintext secret into a tracked path. - CI runs `gitleaks` with rules specifically for SNMP communities, inline diff --git a/scripts/bootstrap.sh b/scripts/bootstrap.sh index 0c576bb..a91c812 100755 --- a/scripts/bootstrap.sh +++ b/scripts/bootstrap.sh @@ -40,6 +40,8 @@ else chmod 600 "${KEY_FILE}" warn "BACK THIS FILE UP OFF THIS MACHINE. Without it the encrypted secrets" warn "in this repository are unrecoverable." + warn " docs/runbooks/back-up-the-age-key.md" + warn " make secrets-verify-backup KEY= # proves the copy decrypts" fi PUBLIC_KEY="$(grep -oE 'age1[a-z0-9]+' "${KEY_FILE}" | head -n1)" @@ -163,4 +165,11 @@ Next steps: 1. make secrets-edit # replace every change-me value 2. make validate # confirm the configs are sound 3. make up # render config and start the stack + +And the one that has no second chance — back up ${KEY_FILE} off this +machine, then prove the copy works: + + make secrets-verify-backup KEY=/path/to/the/copy + + docs/runbooks/back-up-the-age-key.md EOF diff --git a/scripts/verify-key-backup.sh b/scripts/verify-key-backup.sh new file mode 100755 index 0000000..fb61bd2 --- /dev/null +++ b/scripts/verify-key-backup.sh @@ -0,0 +1,237 @@ +#!/usr/bin/env bash +# +# Prove that a backup copy of the age private key can decrypt this stack's SOPS +# file — and that it decrypted because of *that* file rather than because the +# live key on this host was quietly picked up instead. +# +# The second half is the whole point. `SOPS_AGE_KEY_FILE= sops -d ...` +# typed by hand looks like a verification and is not one: sops consults +# SOPS_AGE_KEY and the default ~/.config/sops/age/keys.txt as well, so on the +# host that already holds the real key it prints the secrets whatever is in the +# file you named. Measured, not theorised — pointing that command at a freshly +# generated unrelated keypair with SOPS_AGE_KEY set exits 0 and prints every +# secret. A backup that has never been verified and a backup that has been +# verified wrongly are the same backup; only one of them feels safe. +# +# Nothing here writes a secret to disk, to a temp file, or to the terminal. +# +# Usage: scripts/verify-key-backup.sh [stack] +# make secrets-verify-backup KEY=/path/to/backup/keys.txt +# +# See docs/runbooks/back-up-the-age-key.md. + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +BACKUP="${1:-}" +STACK="${2:-observability}" +SECRETS_FILE="${REPO_ROOT}/secrets/${STACK}.sops.yaml" +SOPS_CONFIG="${REPO_ROOT}/.sops.yaml" + +# Where the *live* key is. Read before the decrypt below scrubs the environment, +# and only ever used to refuse to verify the original by mistake. +LIVE_KEY="${SOPS_AGE_KEY_FILE:-${HOME}/.config/sops/age/keys.txt}" + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +info() { printf '\033[0;34m--\033[0m %s\n' "$*"; } +warn() { printf '\033[0;33m!!\033[0m %s\n' "$*"; } + +if [[ -z "${BACKUP}" ]]; then + cat >&2 < [stack] + or: make secrets-verify-backup KEY=/path/to/backup/keys.txt + +Checks that the age key in decrypts secrets/${STACK}.sops.yaml. +Procedure: docs/runbooks/back-up-the-age-key.md +EOF + exit 2 +fi + +for tool in sops age-keygen; do + command -v "${tool}" >/dev/null 2>&1 || die "${tool} not found. + age: https://github.com/FiloSottile/age/releases + sops: https://github.com/getsops/sops/releases" +done + +[[ -f "${SECRETS_FILE}" ]] || die "no encrypted secrets at ${SECRETS_FILE} +There is nothing to verify a key against. Run 'make secrets-init' first." +[[ -f "${BACKUP}" ]] || die "no such file: ${BACKUP}" +[[ -r "${BACKUP}" ]] || die "${BACKUP} is not readable by $(id -un)" + +# Directory symlinks are resolved with `pwd -P`; the leaf is left alone so the +# stat comparison below is what catches a symlink pointing back at the live key. +BACKUP_ABS="$(cd "$(dirname "${BACKUP}")" && pwd -P)/$(basename "${BACKUP}")" + +# --------------------------------------------------------------------------- +# 1. It has to be a different file from the live key +# --------------------------------------------------------------------------- +# Verifying the original proves nothing at all, and it is an easy mistake: the +# obvious way to test the command before pointing it at the real backup is to +# point it at the key you already have. Device + inode rather than path, so a +# symlink or a hard link back to the original is caught too. +# -L dereferences: without it GNU stat reports the symlink's own inode, and a +# symlink pointing straight back at the live key sails through as a "backup". +file_id() { stat -Lc '%d:%i' "$1" 2>/dev/null || stat -Lf '%d:%i' "$1" 2>/dev/null; } + +if [[ -e "${LIVE_KEY}" ]]; then + backup_id="$(file_id "${BACKUP}" || true)" + live_id="$(file_id "${LIVE_KEY}" || true)" + live_abs="$(cd "$(dirname "${LIVE_KEY}")" && pwd -P)/$(basename "${LIVE_KEY}")" + + if { [[ -n "${backup_id}" ]] && [[ "${backup_id}" == "${live_id}" ]]; } \ + || [[ "${BACKUP_ABS}" == "${live_abs}" ]]; then + die "that is the live key on this host, not a backup of it: + ${BACKUP_ABS} + +A copy that lives on the same disk as the original is not a backup, and +verifying the original proves only that the original works. Point this at the +copy — a mounted USB stick, or a file you pasted back out of your password +manager. See docs/runbooks/back-up-the-age-key.md." + fi +fi + +# --------------------------------------------------------------------------- +# 2. It must not be somewhere that republishes it +# --------------------------------------------------------------------------- +# Issue #11's third checkbox. The repo case is a hard failure because this tree +# is a public repository and .gitignore's `keys.txt` rule is a filename match, +# not a guarantee — a copy named anything else is committable. +if [[ "${BACKUP_ABS}" == "${REPO_ROOT}/"* ]]; then + die "the backup is inside this repository: + ${BACKUP_ABS} + +This tree is published. Move it out before verifying it, and check that it was +never committed: git log --all --oneline -- $(printf '%q' "${BACKUP_ABS#"${REPO_ROOT}/"}")" +fi + +# Advice, not a verdict: the script cannot know what a directory syncs to, and +# for some people a synced vault *is* the durable copy. Saying so out loud is +# the useful part. +for pattern in Dropbox OneDrive 'Google Drive' Nextcloud ownCloud Syncthing iCloud 'Mobile Documents'; do + shopt -s nocasematch + if [[ "${BACKUP_ABS}" == *"${pattern}"* ]]; then + warn "the path contains '${pattern}' — if that folder syncs to a third party," + warn "the private key is now wherever that service keeps it." + fi + shopt -u nocasematch +done + +# --------------------------------------------------------------------------- +# 3. Is it even the right key? (offline, no ciphertext involved) +# --------------------------------------------------------------------------- +# A decrypt failure alone cannot distinguish "this is not the key the file was +# encrypted to" from "this file is not an age identity at all", and those have +# completely different fixes. +if ! PUBLIC_KEYS="$(age-keygen -y "${BACKUP}" 2>&1)"; then + die "${BACKUP_ABS} +is not a readable age identity file. + +age-keygen said: ${PUBLIC_KEYS} + +An age key file holds a line beginning AGE-SECRET-KEY-1. If you transcribed +this from paper, check for a truncated or wrapped line." +fi + +matched="" +while IFS= read -r pub; do + [[ -n "${pub}" ]] || continue + if grep -qF "${pub}" "${SOPS_CONFIG}"; then + matched="${pub}" + break + fi +done <<< "${PUBLIC_KEYS}" + +if [[ -z "${matched}" ]]; then + die "this is a valid age key, but not one the secrets are encrypted to. + + backup holds: $(printf '%s' "${PUBLIC_KEYS}" | tr '\n' ' ') + .sops.yaml wants: $(grep -oE 'age1[a-z0-9]+' "${SOPS_CONFIG}" | tr '\n' ' ') + +A freshly generated keypair looks exactly like this. If you meant to add a new +recipient rather than restore an old one, that is 'sops updatekeys' — see +secrets/README.md." +fi +info "recipient matches .sops.yaml: ${matched}" + +# --------------------------------------------------------------------------- +# 4. The real test: decrypt with nothing but the backup available +# --------------------------------------------------------------------------- +# Every element of the env line below is load-bearing: +# +# -u SOPS_AGE_KEY takes precedence over SOPS_AGE_KEY_FILE, so an +# inherited value would make any file "pass" +# HOME / XDG_CONFIG_HOME pointed at an empty directory, so that if sops falls +# back to the default ~/.config/sops/age/keys.txt it +# finds nothing and fails — rather than decrypting with +# the live key and reporting the backup good +# +# Remove any one of them and this script becomes the hand-typed command it +# exists to replace. +TMP_HOME="$(mktemp -d)" +trap 'rm -rf "${TMP_HOME}"' EXIT INT TERM + +info "decrypting $(basename "${SECRETS_FILE}") with the backup key only" + +# Plaintext is captured into a shell variable and never redirected anywhere. A +# here-string is deliberately not used to search it further down: bash may +# implement one as a temp file, which is exactly what this is avoiding. +if ! plaintext="$(env -u SOPS_AGE_KEY -u XDG_CONFIG_HOME \ + HOME="${TMP_HOME}" SOPS_AGE_KEY_FILE="${BACKUP}" \ + sops --decrypt "${SECRETS_FILE}" 2>"${TMP_HOME}/sops.err")"; then + die "decryption failed with this key. + +sops said: +$(sed 's/^/ /' "${TMP_HOME}/sops.err") + +The public half matched .sops.yaml, so the secret half is damaged or truncated — +the usual cause is a partial copy or a mis-transcribed line." +fi + +# --------------------------------------------------------------------------- +# 5. Do not trust the exit status alone +# --------------------------------------------------------------------------- +# Same reasoning as bootstrap.sh: a zero exit and an empty or partial result are +# indistinguishable from the caller's side, and this is the one check standing +# between a lost key and a lost network. +REQUIRED=( + GRAFANA_ADMIN_PASSWORD + ALERTMANAGER_WEBHOOK_URL + SNMP_COMMUNITY_PFSENSE + SNMP_COMMUNITY_APC + SNMP_COMMUNITY_MOKERLINK + SNMP_COMMUNITY_ILO +) +found=0 +missing=() +for var in "${REQUIRED[@]}"; do + # Pure bash matching: no grep, no process substitution, nothing that could put + # a decrypted value into another process's argv or into a file. + if [[ $'\n'"${plaintext}" == *$'\n'"${var}":* ]]; then + found=$((found + 1)) + else + missing+=("${var}") + fi +done +unset plaintext + +if ((${#missing[@]} > 0)); then + die "the key decrypted the file, but the result is missing keys that +render-config.sh requires: ${missing[*]} + +The backup is fine; the encrypted file is incomplete. Fix it with +'make secrets-edit' on a working host, then verify the backup again." +fi + +# --------------------------------------------------------------------------- +# 6. Housekeeping notes on the copy itself +# --------------------------------------------------------------------------- +mode="$(stat -Lc '%a' "${BACKUP}" 2>/dev/null || stat -Lf '%Lp' "${BACKUP}" 2>/dev/null || true)" +if [[ -n "${mode}" && "${mode: -2}" != "00" ]]; then + warn "mode ${mode} — group or other can read this copy: chmod 600 $(printf '%q' "${BACKUP_ABS}")" +fi + +printf '\033[0;32mok\033[0m — %s decrypts secrets/%s.sops.yaml (%d/%d keys)\n' \ + "$(basename "${BACKUP}")" "${STACK}" "${found}" "${#REQUIRED[@]}" +printf ' Proven: this key, on its own, recovers every secret in the repo.\n' +printf ' Not proven: that where you keep it will still exist after a fire,\n' +printf ' a theft, or a forgotten password. That part is your judgement.\n' diff --git a/secrets/README.md b/secrets/README.md index 41a2bd9..e4f6336 100644 --- a/secrets/README.md +++ b/secrets/README.md @@ -49,7 +49,14 @@ public half into `.sops.yaml`, and encrypts `observability.example.yaml` into `observability.sops.yaml` for you to fill in. **Back up `~/.config/sops/age/keys.txt` somewhere outside this machine.** -Without it the encrypted file is unrecoverable. +Without it the encrypted file is unrecoverable — not "reset with some effort", +but gone, with every secret in it re-entered by hand on four devices. The +procedure, and the way to prove the backup actually decrypts, are in +[`docs/runbooks/back-up-the-age-key.md`](../docs/runbooks/back-up-the-age-key.md): + +```bash +make secrets-verify-backup KEY=/path/to/the/copy +``` ## Editing