diff --git a/Makefile b/Makefile index 195d09f..c53918c 100644 --- a/Makefile +++ b/Makefile @@ -81,7 +81,14 @@ secrets-init: ## Generate an age keypair and create the encrypted secrets file .PHONY: secrets-edit secrets-edit: ## Edit the encrypted secrets in $$EDITOR - sops $(SECRETS) + @# Not a bare `sops $(SECRETS)`. sops decrypts to a temp file and opens + @# $$EDITOR on it, and a vim or neovim with `undofile` set then writes that + @# buffer — the decrypted secrets — into a permanent undodir. sops shreds + @# its own temp file on exit; nothing shreds the undo file. Found in the + @# wild on the monitoring host: three of them holding the live SNMP + @# community strings for pfSense, the APC NMC and iLO, mode 664, on an + @# unencrypted disk. scripts/secrets-edit.sh silences the editor first. + ./scripts/secrets-edit.sh $(STACK) .PHONY: secrets-show secrets-show: ## Print the decrypted secrets to stdout (careful) diff --git a/README.md b/README.md index fcbdbcb..09b6a81 100644 --- a/README.md +++ b/README.md @@ -165,6 +165,7 @@ $ make help reload Hot-reload Prometheus, Alertmanager and snmp-exporter (no restart) secrets-init Generate an age keypair and create the encrypted secrets file secrets-edit Edit the encrypted secrets in $EDITOR + secrets-verify-backup Check a backup age key decrypts the secrets validate Run every check CI runs backup Back up the stack's volumes to ./backups/ ... diff --git a/docs/runbooks/back-up-the-age-key.md b/docs/runbooks/back-up-the-age-key.md index f120369..53fd4ac 100644 --- a/docs/runbooks/back-up-the-age-key.md +++ b/docs/runbooks/back-up-the-age-key.md @@ -91,17 +91,52 @@ judgement, not a check. > directory, and refuses to run against the live key file at all — so a pass > means the backup did the work. +### Do not open the copy in an editor + +Both flows below write the key to a file with `cat >`, not with `vi`. That is +not a style preference. + +`sops` decrypts to a temp file and opens `$EDITOR` on it, and a vim or neovim +configured with `undofile` writes the buffer — the full text — into a permanent +undodir. The same is true of any file you open to paste a key into. `shred -u` +on the tmpfs file then removes the copy you can see and leaves the one you +cannot, on the unencrypted disk, indefinitely. + +This was not hypothetical: it is how three live SNMP community strings came to +be sitting in `~/.local/state/nvim/undodir/` on the monitoring host, which is +why [`make secrets-edit`](../../scripts/secrets-edit.sh) now hardens the editor +before handing it plaintext. `cat >` has no such machinery — it writes what you +give it and nothing else. + +If you genuinely need an editor, `vim -u NONE -i NONE -n` skips the config that +turns those features on. + +### Verifying a password-manager backup + +Verify what came back **out** of the vault, not what you put in — verifying the +copy you pasted from verifies your clipboard. + +```bash +umask 077 +cat > /dev/shm/restore-test.txt # paste from the vault, then Ctrl-D +make secrets-verify-backup KEY=/dev/shm/restore-test.txt +rm -f /dev/shm/restore-test.txt +``` + +`/dev/shm` is tmpfs, so the copy lives in RAM and never reaches the disk. That +is also why `rm` is enough here and `shred`'s overwrite would be theatre — +tmpfs has no stable blocks to overwrite. + ### Verifying a paper backup -Type it **back in** and verify that file. Verifying the copy you pasted from is -verifying your clipboard; verifying the copy you typed is verifying your -handwriting, which is the thing that will actually fail. +Type it **back in** and verify that file. Verifying the copy you typed is +verifying your handwriting, which is the thing that will actually fail. ```bash umask 077 -vi /dev/shm/restore-test.txt # transcribe from the paper +cat > /dev/shm/restore-test.txt # transcribe from the paper, then Ctrl-D make secrets-verify-backup KEY=/dev/shm/restore-test.txt -shred -u /dev/shm/restore-test.txt +rm -f /dev/shm/restore-test.txt ``` age keys transcribe better than most secrets: uppercase Bech32, so the data part @@ -117,10 +152,21 @@ The third thing that goes wrong: a backup that is durable and also public. - **Not in a git repository.** `.gitignore` here matches `keys.txt` and `age.key`, which protects nothing outside this tree and nothing under a different filename. -- **Not in a synced folder.** Dropbox, OneDrive, Google Drive, iCloud, - Nextcloud, Syncthing — a copy there is a copy on someone else's disk. - `make secrets-verify-backup` warns when the path looks like one, but it can - only see the path, not what the folder actually syncs to. +- **Not in plaintext in a synced folder.** Dropbox, OneDrive, Google Drive, + iCloud, Nextcloud, Syncthing — a plaintext copy there is a readable copy on + someone else's disk. `make secrets-verify-backup` warns when the path looks + like one, but it can only see the path, not what the folder actually syncs to, + which is why it warns rather than refuses. + + A password manager syncing an **end-to-end-encrypted** vault is a different + thing and is not disqualified by this — it is the default choice recommended + above. The provider holds a blob it cannot read. Its real failure modes are + losing the master password and losing the account recovery material, which is + the next bullet. +- **Not recoverable only from the host you are insuring.** If the vault's own + recovery material — a 1Password Emergency Kit, a Bitwarden export, the TOTP + seed guarding the account — exists nowhere but this machine, the backup is + circular: the disk that dies takes the means of opening the copy with it. - **Not on the monitoring host.** A second copy on the same disk is not a backup. The script refuses to verify the live key for this reason. diff --git a/docs/security.md b/docs/security.md index 8242497..075d7f7 100644 --- a/docs/security.md +++ b/docs/security.md @@ -53,6 +53,9 @@ assumption consistent with what they are. [`runbooks/back-up-the-age-key.md`](runbooks/back-up-the-age-key.md). - `scripts/render-config.sh` decrypts at deploy time into gitignored files. Nothing writes a plaintext secret into a tracked path. +- `make secrets-edit` hardens `$EDITOR` before handing it the decrypted file, so + the editor cannot persist the plaintext in an undo file, swap file or backup + that sops does not shred. See [`secrets/README.md`](../secrets/README.md). - CI runs `gitleaks` with rules specifically for SNMP communities, inline Grafana passwords, PEM private keys and age secret keys, and separately asserts that every `secrets/*.sops.yaml` is genuinely encrypted. @@ -67,6 +70,8 @@ repository must be treated as compromised: | --- | --- | --- | | SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` (now rewritten) | Purged from history. Replaced with four distinct per-device values, SOPS-encrypted. Rotated on all four. `morpheus`, `mjolnir` and `shiva` verified answering the new community and refusing the old; `neo` answers the new one but still accepts its previous community — accepted risk, see [`SECURITY.md`](../SECURITY.md) and the [runbook](runbooks/rotate-snmp-community.md) | | Grafana `admin` / `admin` with anonymous Admin access | compose file | Fixed: password from SOPS, anonymous auth disabled | +| Decrypted secrets in editor undo files | `~/.local/state/nvim/undodir/`, written by `make secrets-edit` | Found 2026-08-20: three files holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664, on an unencrypted disk. Shredded. `make secrets-edit` now hardens the editor first, so it cannot recur. Never committed, never left the host, so the communities were not rotated on that basis | +| Alertmanager webhook URL and the MokerLink SNMP community | a local Claude Code session transcript under `~/.claude/projects/` | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook was rotated because it is a one-line regenerate; the switch community was not, because rotating it means the `neo` residual below all over again | | Passphrase-encrypted TLS private keys | `certificates/`, added in `efb2632`, deleted in `647d90a` | Purged from history, and the CA replaced — see [runbook](runbooks/generate-certificates.md). Anything that trusted the old CA must be re-pointed at the new one | CI scans both the working tree and the full history, with no ignore file. Both diff --git a/scripts/secrets-edit.sh b/scripts/secrets-edit.sh new file mode 100755 index 0000000..c93197e --- /dev/null +++ b/scripts/secrets-edit.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# +# Edit the encrypted secrets, without the editor leaving a decrypted copy +# behind. +# +# `sops ` decrypts to a temp file under /tmp and opens $EDITOR on it. A +# modern vim or neovim config persists buffer state outside that temp file: +# `undofile` writes the full undo history — which is to say the buffer text — +# to a permanent undodir, and viminfo/shada records registers and marks. sops +# shreds its temp file on exit; nothing shreds the undo file. +# +# Measured on this repo's own monitoring host: three undo files in +# ~/.local/state/nvim/undodir/ named %tmp%%observability.sops.yaml, mode +# 664, holding the live SNMP community strings for pfSense, the APC NMC and +# iLO in plaintext on an unencrypted disk. They had been sitting there since +# the last `make secrets-edit`, and nothing in the repo knew. +# +# That is not the editor's fault and it is not fixable in the operator's +# dotfiles alone — this repository should not hand plaintext to a program that +# it has not told to keep quiet. So the hardening lives here, where it applies +# to whoever clones this and whatever their config says. +# +# Usage: scripts/secrets-edit.sh [stack] (default: observability) +# make secrets-edit +# +# Override with SECRETS_EDITOR if you need something else entirely; you are +# then responsible for that editor's persistence, and the warning below tells +# you what to check. + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +STACK="${1:-observability}" +SECRETS_FILE="${REPO_ROOT}/secrets/${STACK}.sops.yaml" + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +warn() { printf '\033[0;33m!!\033[0m %s\n' "$*" >&2; } + +command -v sops >/dev/null 2>&1 || die "sops not found. + https://github.com/getsops/sops/releases" + +[[ -f "${SECRETS_FILE}" ]] || die "no encrypted secrets at ${SECRETS_FILE} +Run 'make secrets-init' first." + +# --------------------------------------------------------------------------- +# Work out what to run, and silence its persistence +# --------------------------------------------------------------------------- +# sops joins $EDITOR and the temp path into one string and hands it to `sh -c`, +# so flags and quoting inside EDITOR survive. Verified against sops 3.9.4. +# +# The flags, and why each one: +# -n no swap file — .swp holds buffer text too +# -i NONE no viminfo (vim) / shada (nvim): no registers, no +# marks, no search history carrying secret fragments +# -c 'set noundofile …' the important one. `-c` runs after the user's config +# has been read, so it overrides a config that turned +# undofile on; --cmd would run too early and be undone. +# +# vi is included because on Debian and Ubuntu /usr/bin/vi is vim.basic and +# reads the same ~/.vimrc. +editor="${SECRETS_EDITOR:-${VISUAL:-${EDITOR:-vi}}}" + +if [[ -z "${SECRETS_EDITOR:-}" ]]; then + case "$(basename "${editor%% *}")" in + vi | vim | vim.basic | vim.tiny | nvim | view) + editor="${editor} -n -i NONE -c 'set noundofile nobackup nowritebackup'" + ;; + *) + warn "not hardening '$(basename "${editor%% *}")' — this script only knows how to" + warn "silence the vim family. Check that your editor does not write undo" + warn "history, backups or autosaves outside the file it was given; sops" + warn "shreds its temp file, but nothing shreds an editor's leftovers." + ;; + esac +fi + +# --------------------------------------------------------------------------- +# Run it +# --------------------------------------------------------------------------- +# sops exits 200 when the editor quit without changing anything. That is the +# normal outcome of opening the file to look at it, but as a bare recipe it +# surfaced as `make: *** [secrets-edit] Error 200` — which reads like the +# target is broken and sends you looking for a fault that is not there. Every +# other exit code is passed through untouched. +set +e +EDITOR="${editor}" sops "${SECRETS_FILE}" +status=$? +set -e + +if ((status == 200)); then + printf '\033[0;34m--\033[0m %s\n' "unchanged — nothing re-encrypted" + exit 0 +fi +exit "${status}" diff --git a/secrets/README.md b/secrets/README.md index e4f6336..72037d6 100644 --- a/secrets/README.md +++ b/secrets/README.md @@ -64,8 +64,16 @@ make secrets-verify-backup KEY=/path/to/the/copy make secrets-edit ``` -Decrypts to a temporary file, opens `$EDITOR`, re-encrypts on save. The -plaintext never lands on disk unencrypted. +Decrypts to a temporary file, opens `$EDITOR`, re-encrypts on save. + +The target runs [`scripts/secrets-edit.sh`](../scripts/secrets-edit.sh) rather +than `sops` directly, because "the plaintext never lands on disk" is only true +once the editor has been told not to keep its own copy. A vim or neovim with +`undofile` set writes the buffer into a permanent undodir that sops does not +shred — which is how three live SNMP community strings came to be sitting in +`~/.local/state/nvim/undodir/` on the monitoring host. The script disables undo +files, swap files, backups and viminfo/shada for that one invocation. Set +`SECRETS_EDITOR` to override it, and read the script's header first. ## How they reach the containers