docs: record which SNMP communities are actually rotated

README.md

@@ -199,8 +199,10 @@ enforces it, and `make pin-digests` re-resolves them.
 ## Roadmap
 
 Open work is tracked in [`docs/roadmap.md`](docs/roadmap.md). The current top
-items: rotate the SNMP communities, purge the old ones from git history, replace
-the UPS battery, and get 64-bit interface counters off the switch.
+items: finish the SNMP rotation on the switch, purge the old communities from
+git history, replace the UPS battery, and get 64-bit interface counters off the
+switch. The first and last of those share a blocker — the switch has never
+answered an SNMP poll at all.
 
 ## License
 

SECURITY.md

@@ -38,10 +38,17 @@ is a very different thing from an overlooked one. Full detail in
 
 | What | Status |
 | --- | --- |
-| SNMP community committed in plaintext, shared across firewall, switch, UPS and BMC | Removed from `HEAD` and replaced with per-device SOPS-encrypted values. **Still present in git history, and not yet rotated on the devices.** Treat it as public. |
+| SNMP community committed in plaintext, shared across firewall, switch, UPS and BMC | Removed from `HEAD` and replaced with four distinct per-device SOPS-encrypted values. Rotated and verified on the firewall, the UPS and the BMC: each answers to its own new community and refuses the old one. **Not rotated on the switch**, which has never answered an SNMP poll at all — so whether it still accepts the old community is unknown, not proven clean. **Still present in git history.** Treat the old string as public. |
 | Grafana `admin`/`admin` with anonymous Admin access enabled | Fixed — anonymous auth off, password from SOPS |
 | Passphrase-encrypted TLS private keys under `certificates/` | Removed from `HEAD`, still reachable in history. Purge tooling and a runbook are provided; not yet run. |
 
+The switch is the honest gap. `10.7.7.2` has returned `up == 0` for every scrape
+in the 30-day retention window, which predates the rotation — the target has
+never worked, so its failure is not evidence that the rotation broke anything,
+and its silence is not evidence that the old community was removed. It is
+reachable at layer 3 from the monitoring host on ICMP and TCP/80; only UDP/161
+fails. Tracked separately in [#22](https://github.com/Gerrrt/HomeLab/issues/22).
+
 Remediation is tracked in [`docs/roadmap.md`](docs/roadmap.md), with procedures
 in [`docs/runbooks/rotate-snmp-community.md`](docs/runbooks/rotate-snmp-community.md)
 and [`docs/runbooks/purge-git-history.md`](docs/runbooks/purge-git-history.md).

docs/roadmap.md

@@ -5,8 +5,13 @@ inventory. Ordered roughly by how much it matters.
 
 ## Security
 
-- [ ] **Rotate the SNMP communities on all four devices.** The previous shared
-      string was committed in plaintext and must be considered public.
+- [ ] **Finish the SNMP rotation on the MokerLink switch (`neo`, 10.7.7.2).**
+      pfSense, the APC and iLO are done — each answers to its own new community
+      and refuses the old one. The switch is not, and cannot be verified,
+      because it has never answered SNMP at all: `up{job="snmp"}` for
+      `10.7.7.2` has been `0` for the full 30-day retention window, so the
+      failure predates the rotation rather than being caused by it. Blocked on
+      [#22](https://github.com/Gerrrt/HomeLab/issues/22).
       → [runbook](runbooks/rotate-snmp-community.md)
 - [ ] **Purge `certificates/` and the old community string from git history**,
       then regenerate the CA and leaf certificates.
@@ -71,3 +76,6 @@ inventory. Ordered roughly by how much it matters.
 - [x] Add SECURITY.md with a disclosure policy and known-exposure summary
 - [x] Loki alerting rules (8) for auth, SSH brute force and disk/OOM events,
       validated in CI by booting the pinned Loki image against them
+- [x] Give every SNMP device its own community, and rotate three of the four on
+      the hardware (pfSense, the APC NMC, iLO), confirming each refuses the old
+      shared string

docs/runbooks/rotate-snmp-community.md

@@ -90,6 +90,11 @@ make gen-secret ARGS="--snmp --length 15"
 `[A-Za-z0-9]` only — every other character has a specific way of going wrong
 quietly somewhere between `secrets-edit` and the wire.
 
+The card in this lab accepts 16 and is verified working at that length, so the
+cap is not universal to the model. Check yours rather than assuming it either
+way: from the monitoring host, a silent truncation and a correct community are
+indistinguishable — both present as the device refusing you.
+
 Note that SNMPv2c sends these in cleartext on every poll. Distinct communities
 limit the blast radius of a captured packet; they do not make the protocol
 secure. Moving to SNMPv3 authPriv is tracked in [`roadmap.md`](../roadmap.md) —
@@ -188,6 +193,11 @@ up{job="snmp"}
 The target should return within 60 seconds. The `SnmpTargetUnreachable` alert
 fires after 10 minutes, so a mistake here announces itself.
 
+If a device fails here, establish whether you broke it before you roll anything
+back — `max_over_time(up{job="snmp",instance="<ip>"}[30d])`. `1` means it was
+working before you started and the rotation is the suspect; `0` means it never
+worked, the rotation is not the cause, and reverting will not bring it back.
+
 ### 2.5 Remove the old community, then prove it is gone
 
 Only once §2.4 is green. Delete the old entry in the device's UI — some UIs
@@ -198,7 +208,9 @@ require deleting the row rather than blanking the field — then:
 ```
 
 It prompts for the old community without echoing it, and asserts each device now
-refuses it.
+refuses it. It requires a terminal and refuses a pipe on purpose — `echo "$old" |
+...` would put the old community into your shell history, which is the leak this
+tooling exists to close. Run it yourself; no script or agent can.
 
 `--old` only checks devices that just passed their current-community check.
 SNMPv2c has no "wrong community" reply — a device that rejects you simply drops
@@ -207,6 +219,12 @@ a timeout against a device that correctly refused you. Anything else is reported
 `SKIP`, never `PASS`: reporting it as success would be the tool agreeing with you
 rather than checking you.
 
+A `SKIP` is not a pass deferred, it is an open question. If a device never
+answered its current community, you do not know whether it still accepts the old
+one — and if the old one leaked, that device is still exposed. Do not record the
+rotation as complete while any device is `SKIP`. Fix the current-community check
+first; `--old` only becomes meaningful for that device at that point.
+
 ## 3. Commit
 
 ```bash
@@ -217,11 +235,22 @@ git commit -m "chore(secrets): rotate SNMP communities"
 The diff shows *which* keys changed and nothing about their values — SOPS
 encrypts values and leaves keys in plaintext.
 
-Then close the loop, because three files currently assert this has not happened:
+Then close the loop, because these files carry the rotation's status and will
+otherwise assert it never happened:
+
+- [`SECURITY.md`](../../SECURITY.md) — the "Known exposure" row. This is the
+  ledger; correct it here first.
+- [`docs/security.md`](../security.md) — the historical-exposure row and the
+  SNMPv2c bullets, which `SECURITY.md` points at for detail. Leaving this one
+  stale makes the two disagree, which is worse than either being stale alone.
+- [`docs/roadmap.md`](../roadmap.md) — the checkbox.
+
+[`secrets/README.md`](../../secrets/README.md) points at `SECURITY.md` rather
+than restating the status, and needs no edit. Keep it that way.
 
-- [`SECURITY.md`](../../SECURITY.md) — the "not yet rotated on the devices" note
-- [`docs/roadmap.md`](../roadmap.md) — the unchecked box
-- [`secrets/README.md`](../../secrets/README.md) — the rotation pointer
+Record what `snmp-verify` actually printed — which devices passed, which refused
+the old community, which were `SKIP` — on the tracking issue. §2.5's output is
+the only evidence the rotation happened, and it lives in a terminal that closes.
 
 ## If something goes wrong
 
@@ -266,8 +295,9 @@ care how many times you write it.
 | Symptom | Cause | Fix |
 | --- | --- | --- |
 | All four `FAIL` | Not the devices — you are not on `10.0.99.0/24`, or sops decrypted a stale file | `ping 10.0.99.1`; re-run `make render` |
-| Only `neo` (`10.7.7.2`) fails | `10.7.7.0/24` is a separate segment; the pfSense rule permitting `10.0.99.20 → 10.7.7.2:161` is missing or was reset | Check pfSense **Firewall → Rules**; test from a host on that segment |
-| One device fails right after you changed it | The UI truncated the community, or you removed the wrong entry | Re-enter it; check the field's max length (the APC NMC truncates at 15 on several firmwares) |
+| Only `neo` (`10.7.7.2`) fails | `10.7.7.0/24` is a separate segment reached through pfSense; the rule permitting `10.0.99.20 → 10.7.7.2` either does not exist, was reset, or does not cover UDP/161 | `ping 10.7.7.2` and `nc -vz 10.7.7.2 80` from the monitoring host. If both succeed while SNMP times out, the return path is fine and the fault is protocol-specific — check the *protocol and port* on the pfSense rule (**Firewall → Rules**) before you touch the switch |
+| A device `FAIL`s and you cannot tell whether you broke it | An SNMPv2c timeout looks identical for a wrong community, a filtered path, and a device that never worked | Ask Prometheus before rolling anything back: `max_over_time(up{job="snmp",instance="<ip>"}[30d])`. `1` means it worked before you started, so the rotation is the suspect. `0` means it never worked, the rotation is not the cause, and rolling back will not help |
+| One device fails right after you changed it | The UI truncated the community, or you removed the wrong entry | Re-enter it; check the field's max length (the APC NMC truncates at 15 on several firmwares; the card in this lab does not — it is verified at 16) |
 | `--old` reports `STILL ACCEPTED` | The device added the new community alongside the old one | Delete the old entry explicitly — some UIs need the row deleted, not blanked |
 | `--old` reports `SKIP` | The current-community check failed for that device, so a timeout proves nothing | Fix the current check first |
 | `error: ... contains whitespace or '#'` | A community was typed with a space or `#` into SOPS | `make secrets-edit`; regenerate with `make gen-secret` |

docs/security.md

@@ -59,7 +59,7 @@ repository must be treated as compromised:
 
 | What | Where | Status |
 | --- | --- | --- |
-| SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` | Replaced with per-device placeholders. **Rotate on the devices** — see [runbook](runbooks/rotate-snmp-community.md) |
+| SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` | Replaced with four distinct per-device values, SOPS-encrypted. Rotated on `morpheus`, `mjolnir` and `shiva`, each verified answering its new community and refusing the old. **Not rotated on `neo`** — see [#22](https://github.com/Gerrrt/HomeLab/issues/22) and the [runbook](runbooks/rotate-snmp-community.md) |
 | Grafana `admin` / `admin` with anonymous Admin access | compose file | Fixed: password from SOPS, anonymous auth disabled |
 | Passphrase-encrypted TLS private keys | `certificates/`, added in `efb2632`, deleted in `647d90a` but reachable at `647d90a~1` | Still in history. **Purge and regenerate** — see [runbook](runbooks/purge-git-history.md) |
 
@@ -77,12 +77,16 @@ stale and the file gets deleted.
 
 The devices are polled with SNMPv2c, which transmits the community string in
 cleartext. Anyone with a port on the management VLAN can read it off a single
-packet. Two mitigations are in place and one is not:
-
-- **Done:** each device now has a distinct community, so one captured packet
-  does not grant read access to the whole fleet.
-- **Done:** SNMP is only reachable on the management VLAN, which nothing but
-  specific trusted hosts can enter.
+packet. Two mitigations are in place, one only partly, and one is not:
+
+- **Done, with one exception:** each device has its own community, so one
+  captured packet no longer grants read access to the whole fleet. Three are
+  confirmed live on the hardware. The switch's exists only in SOPS — it has
+  never been proven on the device, because `10.7.7.2` has never answered a poll
+  ([#22](https://github.com/Gerrrt/HomeLab/issues/22)).
+- **Done:** SNMP is reachable only on the management VLAN and the
+  switch-management LAN, neither of which anything but specific trusted hosts
+  can enter.
 - **Not done:** SNMPv3 with authPriv. The MokerLink switch does not support it.
   Tracked in [roadmap](roadmap.md).
 

secrets/README.md

@@ -82,6 +82,10 @@ See [`docs/runbooks/rotate-snmp-community.md`](../docs/runbooks/rotate-snmp-comm
 `make snmp-verify` confirms a rotation landed without putting the community into
 your shell history.
 
+Which devices are actually rotated is recorded in
+[`SECURITY.md`](../SECURITY.md), not here. One place to correct when it changes
+is the only arrangement that survives the next rotation.
+
 > **Note on this repository's history.** Earlier commits contained a plaintext
 > SNMP community string shared across all four devices, and encrypted TLS
 > private keys. Both must be treated as compromised regardless of the current