Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,14 @@ secrets-init: ## Generate an age keypair and create the encrypted secrets file

.PHONY: secrets-edit
secrets-edit: ## Edit the encrypted secrets in $$EDITOR
sops $(SECRETS)
@# Not a bare `sops $(SECRETS)`. sops decrypts to a temp file and opens
@# $$EDITOR on it, and a vim or neovim with `undofile` set then writes that
@# buffer — the decrypted secrets — into a permanent undodir. sops shreds
@# its own temp file on exit; nothing shreds the undo file. Found in the
@# wild on the monitoring host: three of them holding the live SNMP
@# community strings for pfSense, the APC NMC and iLO, mode 664, on an
@# unencrypted disk. scripts/secrets-edit.sh silences the editor first.
./scripts/secrets-edit.sh $(STACK)

.PHONY: secrets-show
secrets-show: ## Print the decrypted secrets to stdout (careful)
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ $ make help
reload Hot-reload Prometheus, Alertmanager and snmp-exporter (no restart)
secrets-init Generate an age keypair and create the encrypted secrets file
secrets-edit Edit the encrypted secrets in $EDITOR
secrets-verify-backup Check a backup age key decrypts the secrets
validate Run every check CI runs
backup Back up the stack's volumes to ./backups/
...
Expand Down
64 changes: 55 additions & 9 deletions docs/runbooks/back-up-the-age-key.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,17 +91,52 @@ judgement, not a check.
> directory, and refuses to run against the live key file at all — so a pass
> means the backup did the work.

### Do not open the copy in an editor

Both flows below write the key to a file with `cat >`, not with `vi`. That is
not a style preference.

`sops` decrypts to a temp file and opens `$EDITOR` on it, and a vim or neovim
configured with `undofile` writes the buffer — the full text — into a permanent
undodir. The same is true of any file you open to paste a key into. `shred -u`
on the tmpfs file then removes the copy you can see and leaves the one you
cannot, on the unencrypted disk, indefinitely.

This was not hypothetical: it is how three live SNMP community strings came to
be sitting in `~/.local/state/nvim/undodir/` on the monitoring host, which is
why [`make secrets-edit`](../../scripts/secrets-edit.sh) now hardens the editor
before handing it plaintext. `cat >` has no such machinery — it writes what you
give it and nothing else.

If you genuinely need an editor, `vim -u NONE -i NONE -n` skips the config that
turns those features on.

### Verifying a password-manager backup

Verify what came back **out** of the vault, not what you put in — verifying the
copy you pasted from verifies your clipboard.

```bash
umask 077
cat > /dev/shm/restore-test.txt # paste from the vault, then Ctrl-D
make secrets-verify-backup KEY=/dev/shm/restore-test.txt
rm -f /dev/shm/restore-test.txt
```

`/dev/shm` is tmpfs, so the copy lives in RAM and never reaches the disk. That
is also why `rm` is enough here and `shred`'s overwrite would be theatre —
tmpfs has no stable blocks to overwrite.

### Verifying a paper backup

Type it **back in** and verify that file. Verifying the copy you pasted from is
verifying your clipboard; verifying the copy you typed is verifying your
handwriting, which is the thing that will actually fail.
Type it **back in** and verify that file. Verifying the copy you typed is
verifying your handwriting, which is the thing that will actually fail.

```bash
umask 077
vi /dev/shm/restore-test.txt # transcribe from the paper
cat > /dev/shm/restore-test.txt # transcribe from the paper, then Ctrl-D
make secrets-verify-backup KEY=/dev/shm/restore-test.txt
shred -u /dev/shm/restore-test.txt
rm -f /dev/shm/restore-test.txt
```

age keys transcribe better than most secrets: uppercase Bech32, so the data part
Expand All @@ -117,10 +152,21 @@ The third thing that goes wrong: a backup that is durable and also public.
- **Not in a git repository.** `.gitignore` here matches `keys.txt` and
`age.key`, which protects nothing outside this tree and nothing under a
different filename.
- **Not in a synced folder.** Dropbox, OneDrive, Google Drive, iCloud,
Nextcloud, Syncthing — a copy there is a copy on someone else's disk.
`make secrets-verify-backup` warns when the path looks like one, but it can
only see the path, not what the folder actually syncs to.
- **Not in plaintext in a synced folder.** Dropbox, OneDrive, Google Drive,
iCloud, Nextcloud, Syncthing — a plaintext copy there is a readable copy on
someone else's disk. `make secrets-verify-backup` warns when the path looks
like one, but it can only see the path, not what the folder actually syncs to,
which is why it warns rather than refuses.

A password manager syncing an **end-to-end-encrypted** vault is a different
thing and is not disqualified by this — it is the default choice recommended
above. The provider holds a blob it cannot read. Its real failure modes are
losing the master password and losing the account recovery material, which is
the next bullet.
- **Not recoverable only from the host you are insuring.** If the vault's own
recovery material — a 1Password Emergency Kit, a Bitwarden export, the TOTP
seed guarding the account — exists nowhere but this machine, the backup is
circular: the disk that dies takes the means of opening the copy with it.
- **Not on the monitoring host.** A second copy on the same disk is not a
backup. The script refuses to verify the live key for this reason.

Expand Down
5 changes: 5 additions & 0 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ assumption consistent with what they are.
[`runbooks/back-up-the-age-key.md`](runbooks/back-up-the-age-key.md).
- `scripts/render-config.sh` decrypts at deploy time into gitignored files.
Nothing writes a plaintext secret into a tracked path.
- `make secrets-edit` hardens `$EDITOR` before handing it the decrypted file, so
the editor cannot persist the plaintext in an undo file, swap file or backup
that sops does not shred. See [`secrets/README.md`](../secrets/README.md).
- CI runs `gitleaks` with rules specifically for SNMP communities, inline
Grafana passwords, PEM private keys and age secret keys, and separately
asserts that every `secrets/*.sops.yaml` is genuinely encrypted.
Expand All @@ -67,6 +70,8 @@ repository must be treated as compromised:
| --- | --- | --- |
| SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` (now rewritten) | Purged from history. Replaced with four distinct per-device values, SOPS-encrypted. Rotated on all four. `morpheus`, `mjolnir` and `shiva` verified answering the new community and refusing the old; `neo` answers the new one but still accepts its previous community — accepted risk, see [`SECURITY.md`](../SECURITY.md) and the [runbook](runbooks/rotate-snmp-community.md) |
| Grafana `admin` / `admin` with anonymous Admin access | compose file | Fixed: password from SOPS, anonymous auth disabled |
| Decrypted secrets in editor undo files | `~/.local/state/nvim/undodir/`, written by `make secrets-edit` | Found 2026-08-20: three files holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664, on an unencrypted disk. Shredded. `make secrets-edit` now hardens the editor first, so it cannot recur. Never committed, never left the host, so the communities were not rotated on that basis |
| Alertmanager webhook URL and the MokerLink SNMP community | a local Claude Code session transcript under `~/.claude/projects/` | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook was rotated because it is a one-line regenerate; the switch community was not, because rotating it means the `neo` residual below all over again |
| Passphrase-encrypted TLS private keys | `certificates/`, added in `efb2632`, deleted in `647d90a` | Purged from history, and the CA replaced — see [runbook](runbooks/generate-certificates.md). Anything that trusted the old CA must be re-pointed at the new one |

CI scans both the working tree and the full history, with no ignore file. Both
Expand Down
94 changes: 94 additions & 0 deletions scripts/secrets-edit.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
#!/usr/bin/env bash
#
# Edit the encrypted secrets, without the editor leaving a decrypted copy
# behind.
#
# `sops <file>` decrypts to a temp file under /tmp and opens $EDITOR on it. A
# modern vim or neovim config persists buffer state outside that temp file:
# `undofile` writes the full undo history — which is to say the buffer text —
# to a permanent undodir, and viminfo/shada records registers and marks. sops
# shreds its temp file on exit; nothing shreds the undo file.
#
# Measured on this repo's own monitoring host: three undo files in
# ~/.local/state/nvim/undodir/ named %tmp%<pid>%observability.sops.yaml, mode
# 664, holding the live SNMP community strings for pfSense, the APC NMC and
# iLO in plaintext on an unencrypted disk. They had been sitting there since
# the last `make secrets-edit`, and nothing in the repo knew.
#
# That is not the editor's fault and it is not fixable in the operator's
# dotfiles alone — this repository should not hand plaintext to a program that
# it has not told to keep quiet. So the hardening lives here, where it applies
# to whoever clones this and whatever their config says.
#
# Usage: scripts/secrets-edit.sh [stack] (default: observability)
# make secrets-edit
#
# Override with SECRETS_EDITOR if you need something else entirely; you are
# then responsible for that editor's persistence, and the warning below tells
# you what to check.

set -euo pipefail

REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
STACK="${1:-observability}"
SECRETS_FILE="${REPO_ROOT}/secrets/${STACK}.sops.yaml"

die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; }
warn() { printf '\033[0;33m!!\033[0m %s\n' "$*" >&2; }

command -v sops >/dev/null 2>&1 || die "sops not found.
https://github.com/getsops/sops/releases"

[[ -f "${SECRETS_FILE}" ]] || die "no encrypted secrets at ${SECRETS_FILE}
Run 'make secrets-init' first."

# ---------------------------------------------------------------------------
# Work out what to run, and silence its persistence
# ---------------------------------------------------------------------------
# sops joins $EDITOR and the temp path into one string and hands it to `sh -c`,
# so flags and quoting inside EDITOR survive. Verified against sops 3.9.4.
#
# The flags, and why each one:
# -n no swap file — .swp holds buffer text too
# -i NONE no viminfo (vim) / shada (nvim): no registers, no
# marks, no search history carrying secret fragments
# -c 'set noundofile …' the important one. `-c` runs after the user's config
# has been read, so it overrides a config that turned
# undofile on; --cmd would run too early and be undone.
#
# vi is included because on Debian and Ubuntu /usr/bin/vi is vim.basic and
# reads the same ~/.vimrc.
editor="${SECRETS_EDITOR:-${VISUAL:-${EDITOR:-vi}}}"

if [[ -z "${SECRETS_EDITOR:-}" ]]; then
case "$(basename "${editor%% *}")" in
vi | vim | vim.basic | vim.tiny | nvim | view)
editor="${editor} -n -i NONE -c 'set noundofile nobackup nowritebackup'"
;;
*)
warn "not hardening '$(basename "${editor%% *}")' — this script only knows how to"
warn "silence the vim family. Check that your editor does not write undo"
warn "history, backups or autosaves outside the file it was given; sops"
warn "shreds its temp file, but nothing shreds an editor's leftovers."
;;
esac
fi

# ---------------------------------------------------------------------------
# Run it
# ---------------------------------------------------------------------------
# sops exits 200 when the editor quit without changing anything. That is the
# normal outcome of opening the file to look at it, but as a bare recipe it
# surfaced as `make: *** [secrets-edit] Error 200` — which reads like the
# target is broken and sends you looking for a fault that is not there. Every
# other exit code is passed through untouched.
set +e
EDITOR="${editor}" sops "${SECRETS_FILE}"
status=$?
set -e

if ((status == 200)); then
printf '\033[0;34m--\033[0m %s\n' "unchanged — nothing re-encrypted"
exit 0
fi
exit "${status}"
12 changes: 10 additions & 2 deletions secrets/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,16 @@ make secrets-verify-backup KEY=/path/to/the/copy
make secrets-edit
```

Decrypts to a temporary file, opens `$EDITOR`, re-encrypts on save. The
plaintext never lands on disk unencrypted.
Decrypts to a temporary file, opens `$EDITOR`, re-encrypts on save.

The target runs [`scripts/secrets-edit.sh`](../scripts/secrets-edit.sh) rather
than `sops` directly, because "the plaintext never lands on disk" is only true
once the editor has been told not to keep its own copy. A vim or neovim with
`undofile` set writes the buffer into a permanent undodir that sops does not
shred — which is how three live SNMP community strings came to be sitting in
`~/.local/state/nvim/undodir/` on the monitoring host. The script disables undo
files, swap files, backups and viminfo/shada for that one invocation. Set
`SECRETS_EDITOR` to override it, and read the script's header first.

## How they reach the containers

Expand Down
Loading