From 843f946c96dfae42b5258c8c1b4a2d8ee5ee786b Mon Sep 17 00:00:00 2001 From: Your Name Date: Tue, 18 Aug 2026 23:34:33 +0000 Subject: [PATCH 1/2] docs: record which SNMP communities are actually rotated MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three of the four devices were rotated and nothing said so. pfSense, the APC NMC and iLO each answer to their own new community and refuse the old shared one; the MokerLink switch does not, and cannot be verified, because it has never answered an SNMP poll at all. SECURITY.md is the ledger and carries the per-device status. docs/security.md carries the detail behind it — it also claimed SNMP was reachable only on the management VLAN, which is wrong: neo is polled at 10.7.7.2 on the switch-management LAN. secrets/README.md gets a pointer rather than a fifth copy of the status to keep in sync. The roadmap item stays unchecked but is renamed to name the switch. An unchecked box reading "all four devices" reads as though nothing happened, which is materially false. Refs #9, #22 Co-Authored-By: Claude Opus 5 --- README.md | 6 ++++-- SECURITY.md | 9 ++++++++- docs/roadmap.md | 12 ++++++++++-- docs/security.md | 18 +++++++++++------- secrets/README.md | 4 ++++ 5 files changed, 37 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 57a16ea..05ab2a1 100644 --- a/README.md +++ b/README.md @@ -199,8 +199,10 @@ enforces it, and `make pin-digests` re-resolves them. ## Roadmap Open work is tracked in [`docs/roadmap.md`](docs/roadmap.md). The current top -items: rotate the SNMP communities, purge the old ones from git history, replace -the UPS battery, and get 64-bit interface counters off the switch. +items: finish the SNMP rotation on the switch, purge the old communities from +git history, replace the UPS battery, and get 64-bit interface counters off the +switch. The first and last of those share a blocker — the switch has never +answered an SNMP poll at all. ## License diff --git a/SECURITY.md b/SECURITY.md index 4e75cb0..9016e08 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -38,10 +38,17 @@ is a very different thing from an overlooked one. Full detail in | What | Status | | --- | --- | -| SNMP community committed in plaintext, shared across firewall, switch, UPS and BMC | Removed from `HEAD` and replaced with per-device SOPS-encrypted values. **Still present in git history, and not yet rotated on the devices.** Treat it as public. | +| SNMP community committed in plaintext, shared across firewall, switch, UPS and BMC | Removed from `HEAD` and replaced with four distinct per-device SOPS-encrypted values. Rotated and verified on the firewall, the UPS and the BMC: each answers to its own new community and refuses the old one. **Not rotated on the switch**, which has never answered an SNMP poll at all — so whether it still accepts the old community is unknown, not proven clean. **Still present in git history.** Treat the old string as public. | | Grafana `admin`/`admin` with anonymous Admin access enabled | Fixed — anonymous auth off, password from SOPS | | Passphrase-encrypted TLS private keys under `certificates/` | Removed from `HEAD`, still reachable in history. Purge tooling and a runbook are provided; not yet run. | +The switch is the honest gap. `10.7.7.2` has returned `up == 0` for every scrape +in the 30-day retention window, which predates the rotation — the target has +never worked, so its failure is not evidence that the rotation broke anything, +and its silence is not evidence that the old community was removed. It is +reachable at layer 3 from the monitoring host on ICMP and TCP/80; only UDP/161 +fails. Tracked separately in [#22](https://github.com/Gerrrt/HomeLab/issues/22). + Remediation is tracked in [`docs/roadmap.md`](docs/roadmap.md), with procedures in [`docs/runbooks/rotate-snmp-community.md`](docs/runbooks/rotate-snmp-community.md) and [`docs/runbooks/purge-git-history.md`](docs/runbooks/purge-git-history.md). diff --git a/docs/roadmap.md b/docs/roadmap.md index b1edb03..f59e9fc 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -5,8 +5,13 @@ inventory. Ordered roughly by how much it matters. ## Security -- [ ] **Rotate the SNMP communities on all four devices.** The previous shared - string was committed in plaintext and must be considered public. +- [ ] **Finish the SNMP rotation on the MokerLink switch (`neo`, 10.7.7.2).** + pfSense, the APC and iLO are done — each answers to its own new community + and refuses the old one. The switch is not, and cannot be verified, + because it has never answered SNMP at all: `up{job="snmp"}` for + `10.7.7.2` has been `0` for the full 30-day retention window, so the + failure predates the rotation rather than being caused by it. Blocked on + [#22](https://github.com/Gerrrt/HomeLab/issues/22). → [runbook](runbooks/rotate-snmp-community.md) - [ ] **Purge `certificates/` and the old community string from git history**, then regenerate the CA and leaf certificates. @@ -71,3 +76,6 @@ inventory. Ordered roughly by how much it matters. - [x] Add SECURITY.md with a disclosure policy and known-exposure summary - [x] Loki alerting rules (8) for auth, SSH brute force and disk/OOM events, validated in CI by booting the pinned Loki image against them +- [x] Give every SNMP device its own community, and rotate three of the four on + the hardware (pfSense, the APC NMC, iLO), confirming each refuses the old + shared string diff --git a/docs/security.md b/docs/security.md index 405eec0..e24be13 100644 --- a/docs/security.md +++ b/docs/security.md @@ -59,7 +59,7 @@ repository must be treated as compromised: | What | Where | Status | | --- | --- | --- | -| SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` | Replaced with per-device placeholders. **Rotate on the devices** — see [runbook](runbooks/rotate-snmp-community.md) | +| SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` | Replaced with four distinct per-device values, SOPS-encrypted. Rotated on `morpheus`, `mjolnir` and `shiva`, each verified answering its new community and refusing the old. **Not rotated on `neo`** — see [#22](https://github.com/Gerrrt/HomeLab/issues/22) and the [runbook](runbooks/rotate-snmp-community.md) | | Grafana `admin` / `admin` with anonymous Admin access | compose file | Fixed: password from SOPS, anonymous auth disabled | | Passphrase-encrypted TLS private keys | `certificates/`, added in `efb2632`, deleted in `647d90a` but reachable at `647d90a~1` | Still in history. **Purge and regenerate** — see [runbook](runbooks/purge-git-history.md) | @@ -77,12 +77,16 @@ stale and the file gets deleted. The devices are polled with SNMPv2c, which transmits the community string in cleartext. Anyone with a port on the management VLAN can read it off a single -packet. Two mitigations are in place and one is not: - -- **Done:** each device now has a distinct community, so one captured packet - does not grant read access to the whole fleet. -- **Done:** SNMP is only reachable on the management VLAN, which nothing but - specific trusted hosts can enter. +packet. Two mitigations are in place, one only partly, and one is not: + +- **Done, with one exception:** each device has its own community, so one + captured packet no longer grants read access to the whole fleet. Three are + confirmed live on the hardware. The switch's exists only in SOPS — it has + never been proven on the device, because `10.7.7.2` has never answered a poll + ([#22](https://github.com/Gerrrt/HomeLab/issues/22)). +- **Done:** SNMP is reachable only on the management VLAN and the + switch-management LAN, neither of which anything but specific trusted hosts + can enter. - **Not done:** SNMPv3 with authPriv. The MokerLink switch does not support it. Tracked in [roadmap](roadmap.md). diff --git a/secrets/README.md b/secrets/README.md index 78a1c70..41a2bd9 100644 --- a/secrets/README.md +++ b/secrets/README.md @@ -82,6 +82,10 @@ See [`docs/runbooks/rotate-snmp-community.md`](../docs/runbooks/rotate-snmp-comm `make snmp-verify` confirms a rotation landed without putting the community into your shell history. +Which devices are actually rotated is recorded in +[`SECURITY.md`](../SECURITY.md), not here. One place to correct when it changes +is the only arrangement that survives the next rotation. + > **Note on this repository's history.** Earlier commits contained a plaintext > SNMP community string shared across all four devices, and encrypted TLS > private keys. Both must be treated as compromised regardless of the current From 0025c1962d314d69ffe6679f66ca0bac6a50fe95 Mon Sep 17 00:00:00 2001 From: Your Name Date: Tue, 18 Aug 2026 23:36:32 +0000 Subject: [PATCH 2/2] docs: correct the SNMP rotation runbook from what the rotation taught us MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing in the runbook let an operator tell "my rotation broke this" from "this never worked", and that distinction decides whether to roll back. Adds the max_over_time(up[30d]) check to §2.4 and to the troubleshooting table. The neo row's hypothesis was right and is now evidence-backed, so it becomes a test rather than a hunch: ICMP and TCP/80 to 10.7.7.2 both succeed while UDP/161 does not, which narrows the pfSense rule from "missing" to "may not cover this protocol and port". §2.5 explained why --old skips a device but never what to do about it. A SKIP is an open question, not a pass deferred — a device that never answered may still accept the leaked community. Also records that --old refuses a pipe on purpose, so no script or agent can run it. The APC 15-character warning read as a property of the model. This lab's card is verified working at 16. The close-the-loop list was wrong by one: docs/security.md carries the detail SECURITY.md points at, and leaving it stale makes the two disagree. Refs #9, #22 Co-Authored-By: Claude Opus 5 --- docs/runbooks/rotate-snmp-community.md | 44 ++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/docs/runbooks/rotate-snmp-community.md b/docs/runbooks/rotate-snmp-community.md index a83e324..5980b57 100644 --- a/docs/runbooks/rotate-snmp-community.md +++ b/docs/runbooks/rotate-snmp-community.md @@ -90,6 +90,11 @@ make gen-secret ARGS="--snmp --length 15" `[A-Za-z0-9]` only — every other character has a specific way of going wrong quietly somewhere between `secrets-edit` and the wire. +The card in this lab accepts 16 and is verified working at that length, so the +cap is not universal to the model. Check yours rather than assuming it either +way: from the monitoring host, a silent truncation and a correct community are +indistinguishable — both present as the device refusing you. + Note that SNMPv2c sends these in cleartext on every poll. Distinct communities limit the blast radius of a captured packet; they do not make the protocol secure. Moving to SNMPv3 authPriv is tracked in [`roadmap.md`](../roadmap.md) — @@ -188,6 +193,11 @@ up{job="snmp"} The target should return within 60 seconds. The `SnmpTargetUnreachable` alert fires after 10 minutes, so a mistake here announces itself. +If a device fails here, establish whether you broke it before you roll anything +back — `max_over_time(up{job="snmp",instance=""}[30d])`. `1` means it was +working before you started and the rotation is the suspect; `0` means it never +worked, the rotation is not the cause, and reverting will not bring it back. + ### 2.5 Remove the old community, then prove it is gone Only once §2.4 is green. Delete the old entry in the device's UI — some UIs @@ -198,7 +208,9 @@ require deleting the row rather than blanking the field — then: ``` It prompts for the old community without echoing it, and asserts each device now -refuses it. +refuses it. It requires a terminal and refuses a pipe on purpose — `echo "$old" | +...` would put the old community into your shell history, which is the leak this +tooling exists to close. Run it yourself; no script or agent can. `--old` only checks devices that just passed their current-community check. SNMPv2c has no "wrong community" reply — a device that rejects you simply drops @@ -207,6 +219,12 @@ a timeout against a device that correctly refused you. Anything else is reported `SKIP`, never `PASS`: reporting it as success would be the tool agreeing with you rather than checking you. +A `SKIP` is not a pass deferred, it is an open question. If a device never +answered its current community, you do not know whether it still accepts the old +one — and if the old one leaked, that device is still exposed. Do not record the +rotation as complete while any device is `SKIP`. Fix the current-community check +first; `--old` only becomes meaningful for that device at that point. + ## 3. Commit ```bash @@ -217,11 +235,22 @@ git commit -m "chore(secrets): rotate SNMP communities" The diff shows *which* keys changed and nothing about their values — SOPS encrypts values and leaves keys in plaintext. -Then close the loop, because three files currently assert this has not happened: +Then close the loop, because these files carry the rotation's status and will +otherwise assert it never happened: + +- [`SECURITY.md`](../../SECURITY.md) — the "Known exposure" row. This is the + ledger; correct it here first. +- [`docs/security.md`](../security.md) — the historical-exposure row and the + SNMPv2c bullets, which `SECURITY.md` points at for detail. Leaving this one + stale makes the two disagree, which is worse than either being stale alone. +- [`docs/roadmap.md`](../roadmap.md) — the checkbox. + +[`secrets/README.md`](../../secrets/README.md) points at `SECURITY.md` rather +than restating the status, and needs no edit. Keep it that way. -- [`SECURITY.md`](../../SECURITY.md) — the "not yet rotated on the devices" note -- [`docs/roadmap.md`](../roadmap.md) — the unchecked box -- [`secrets/README.md`](../../secrets/README.md) — the rotation pointer +Record what `snmp-verify` actually printed — which devices passed, which refused +the old community, which were `SKIP` — on the tracking issue. §2.5's output is +the only evidence the rotation happened, and it lives in a terminal that closes. ## If something goes wrong @@ -266,8 +295,9 @@ care how many times you write it. | Symptom | Cause | Fix | | --- | --- | --- | | All four `FAIL` | Not the devices — you are not on `10.0.99.0/24`, or sops decrypted a stale file | `ping 10.0.99.1`; re-run `make render` | -| Only `neo` (`10.7.7.2`) fails | `10.7.7.0/24` is a separate segment; the pfSense rule permitting `10.0.99.20 → 10.7.7.2:161` is missing or was reset | Check pfSense **Firewall → Rules**; test from a host on that segment | -| One device fails right after you changed it | The UI truncated the community, or you removed the wrong entry | Re-enter it; check the field's max length (the APC NMC truncates at 15 on several firmwares) | +| Only `neo` (`10.7.7.2`) fails | `10.7.7.0/24` is a separate segment reached through pfSense; the rule permitting `10.0.99.20 → 10.7.7.2` either does not exist, was reset, or does not cover UDP/161 | `ping 10.7.7.2` and `nc -vz 10.7.7.2 80` from the monitoring host. If both succeed while SNMP times out, the return path is fine and the fault is protocol-specific — check the *protocol and port* on the pfSense rule (**Firewall → Rules**) before you touch the switch | +| A device `FAIL`s and you cannot tell whether you broke it | An SNMPv2c timeout looks identical for a wrong community, a filtered path, and a device that never worked | Ask Prometheus before rolling anything back: `max_over_time(up{job="snmp",instance=""}[30d])`. `1` means it worked before you started, so the rotation is the suspect. `0` means it never worked, the rotation is not the cause, and rolling back will not help | +| One device fails right after you changed it | The UI truncated the community, or you removed the wrong entry | Re-enter it; check the field's max length (the APC NMC truncates at 15 on several firmwares; the card in this lab does not — it is verified at 16) | | `--old` reports `STILL ACCEPTED` | The device added the new community alongside the old one | Delete the old entry explicitly — some UIs need the row deleted, not blanked | | `--old` reports `SKIP` | The current-community check failed for that device, so a timeout proves nothing | Fix the current check first | | `error: ... contains whitespace or '#'` | A community was typed with a space or `#` into SOPS | `make secrets-edit`; regenerate with `make gen-secret` |