diff --git a/README.md b/README.md index 57a16ea..05ab2a1 100644 --- a/README.md +++ b/README.md @@ -199,8 +199,10 @@ enforces it, and `make pin-digests` re-resolves them. ## Roadmap Open work is tracked in [`docs/roadmap.md`](docs/roadmap.md). The current top -items: rotate the SNMP communities, purge the old ones from git history, replace -the UPS battery, and get 64-bit interface counters off the switch. +items: finish the SNMP rotation on the switch, purge the old communities from +git history, replace the UPS battery, and get 64-bit interface counters off the +switch. The first and last of those share a blocker — the switch has never +answered an SNMP poll at all. ## License diff --git a/SECURITY.md b/SECURITY.md index 4e75cb0..9016e08 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -38,10 +38,17 @@ is a very different thing from an overlooked one. Full detail in | What | Status | | --- | --- | -| SNMP community committed in plaintext, shared across firewall, switch, UPS and BMC | Removed from `HEAD` and replaced with per-device SOPS-encrypted values. **Still present in git history, and not yet rotated on the devices.** Treat it as public. | +| SNMP community committed in plaintext, shared across firewall, switch, UPS and BMC | Removed from `HEAD` and replaced with four distinct per-device SOPS-encrypted values. Rotated and verified on the firewall, the UPS and the BMC: each answers to its own new community and refuses the old one. **Not rotated on the switch**, which has never answered an SNMP poll at all — so whether it still accepts the old community is unknown, not proven clean. **Still present in git history.** Treat the old string as public. | | Grafana `admin`/`admin` with anonymous Admin access enabled | Fixed — anonymous auth off, password from SOPS | | Passphrase-encrypted TLS private keys under `certificates/` | Removed from `HEAD`, still reachable in history. Purge tooling and a runbook are provided; not yet run. | +The switch is the honest gap. `10.7.7.2` has returned `up == 0` for every scrape +in the 30-day retention window, which predates the rotation — the target has +never worked, so its failure is not evidence that the rotation broke anything, +and its silence is not evidence that the old community was removed. It is +reachable at layer 3 from the monitoring host on ICMP and TCP/80; only UDP/161 +fails. Tracked separately in [#22](https://github.com/Gerrrt/HomeLab/issues/22). + Remediation is tracked in [`docs/roadmap.md`](docs/roadmap.md), with procedures in [`docs/runbooks/rotate-snmp-community.md`](docs/runbooks/rotate-snmp-community.md) and [`docs/runbooks/purge-git-history.md`](docs/runbooks/purge-git-history.md). diff --git a/docs/roadmap.md b/docs/roadmap.md index b1edb03..f59e9fc 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -5,8 +5,13 @@ inventory. Ordered roughly by how much it matters. ## Security -- [ ] **Rotate the SNMP communities on all four devices.** The previous shared - string was committed in plaintext and must be considered public. +- [ ] **Finish the SNMP rotation on the MokerLink switch (`neo`, 10.7.7.2).** + pfSense, the APC and iLO are done — each answers to its own new community + and refuses the old one. The switch is not, and cannot be verified, + because it has never answered SNMP at all: `up{job="snmp"}` for + `10.7.7.2` has been `0` for the full 30-day retention window, so the + failure predates the rotation rather than being caused by it. Blocked on + [#22](https://github.com/Gerrrt/HomeLab/issues/22). → [runbook](runbooks/rotate-snmp-community.md) - [ ] **Purge `certificates/` and the old community string from git history**, then regenerate the CA and leaf certificates. @@ -71,3 +76,6 @@ inventory. Ordered roughly by how much it matters. - [x] Add SECURITY.md with a disclosure policy and known-exposure summary - [x] Loki alerting rules (8) for auth, SSH brute force and disk/OOM events, validated in CI by booting the pinned Loki image against them +- [x] Give every SNMP device its own community, and rotate three of the four on + the hardware (pfSense, the APC NMC, iLO), confirming each refuses the old + shared string diff --git a/docs/runbooks/rotate-snmp-community.md b/docs/runbooks/rotate-snmp-community.md index a83e324..5980b57 100644 --- a/docs/runbooks/rotate-snmp-community.md +++ b/docs/runbooks/rotate-snmp-community.md @@ -90,6 +90,11 @@ make gen-secret ARGS="--snmp --length 15" `[A-Za-z0-9]` only — every other character has a specific way of going wrong quietly somewhere between `secrets-edit` and the wire. +The card in this lab accepts 16 and is verified working at that length, so the +cap is not universal to the model. Check yours rather than assuming it either +way: from the monitoring host, a silent truncation and a correct community are +indistinguishable — both present as the device refusing you. + Note that SNMPv2c sends these in cleartext on every poll. Distinct communities limit the blast radius of a captured packet; they do not make the protocol secure. Moving to SNMPv3 authPriv is tracked in [`roadmap.md`](../roadmap.md) — @@ -188,6 +193,11 @@ up{job="snmp"} The target should return within 60 seconds. The `SnmpTargetUnreachable` alert fires after 10 minutes, so a mistake here announces itself. +If a device fails here, establish whether you broke it before you roll anything +back — `max_over_time(up{job="snmp",instance=""}[30d])`. `1` means it was +working before you started and the rotation is the suspect; `0` means it never +worked, the rotation is not the cause, and reverting will not bring it back. + ### 2.5 Remove the old community, then prove it is gone Only once §2.4 is green. Delete the old entry in the device's UI — some UIs @@ -198,7 +208,9 @@ require deleting the row rather than blanking the field — then: ``` It prompts for the old community without echoing it, and asserts each device now -refuses it. +refuses it. It requires a terminal and refuses a pipe on purpose — `echo "$old" | +...` would put the old community into your shell history, which is the leak this +tooling exists to close. Run it yourself; no script or agent can. `--old` only checks devices that just passed their current-community check. SNMPv2c has no "wrong community" reply — a device that rejects you simply drops @@ -207,6 +219,12 @@ a timeout against a device that correctly refused you. Anything else is reported `SKIP`, never `PASS`: reporting it as success would be the tool agreeing with you rather than checking you. +A `SKIP` is not a pass deferred, it is an open question. If a device never +answered its current community, you do not know whether it still accepts the old +one — and if the old one leaked, that device is still exposed. Do not record the +rotation as complete while any device is `SKIP`. Fix the current-community check +first; `--old` only becomes meaningful for that device at that point. + ## 3. Commit ```bash @@ -217,11 +235,22 @@ git commit -m "chore(secrets): rotate SNMP communities" The diff shows *which* keys changed and nothing about their values — SOPS encrypts values and leaves keys in plaintext. -Then close the loop, because three files currently assert this has not happened: +Then close the loop, because these files carry the rotation's status and will +otherwise assert it never happened: + +- [`SECURITY.md`](../../SECURITY.md) — the "Known exposure" row. This is the + ledger; correct it here first. +- [`docs/security.md`](../security.md) — the historical-exposure row and the + SNMPv2c bullets, which `SECURITY.md` points at for detail. Leaving this one + stale makes the two disagree, which is worse than either being stale alone. +- [`docs/roadmap.md`](../roadmap.md) — the checkbox. + +[`secrets/README.md`](../../secrets/README.md) points at `SECURITY.md` rather +than restating the status, and needs no edit. Keep it that way. -- [`SECURITY.md`](../../SECURITY.md) — the "not yet rotated on the devices" note -- [`docs/roadmap.md`](../roadmap.md) — the unchecked box -- [`secrets/README.md`](../../secrets/README.md) — the rotation pointer +Record what `snmp-verify` actually printed — which devices passed, which refused +the old community, which were `SKIP` — on the tracking issue. §2.5's output is +the only evidence the rotation happened, and it lives in a terminal that closes. ## If something goes wrong @@ -266,8 +295,9 @@ care how many times you write it. | Symptom | Cause | Fix | | --- | --- | --- | | All four `FAIL` | Not the devices — you are not on `10.0.99.0/24`, or sops decrypted a stale file | `ping 10.0.99.1`; re-run `make render` | -| Only `neo` (`10.7.7.2`) fails | `10.7.7.0/24` is a separate segment; the pfSense rule permitting `10.0.99.20 → 10.7.7.2:161` is missing or was reset | Check pfSense **Firewall → Rules**; test from a host on that segment | -| One device fails right after you changed it | The UI truncated the community, or you removed the wrong entry | Re-enter it; check the field's max length (the APC NMC truncates at 15 on several firmwares) | +| Only `neo` (`10.7.7.2`) fails | `10.7.7.0/24` is a separate segment reached through pfSense; the rule permitting `10.0.99.20 → 10.7.7.2` either does not exist, was reset, or does not cover UDP/161 | `ping 10.7.7.2` and `nc -vz 10.7.7.2 80` from the monitoring host. If both succeed while SNMP times out, the return path is fine and the fault is protocol-specific — check the *protocol and port* on the pfSense rule (**Firewall → Rules**) before you touch the switch | +| A device `FAIL`s and you cannot tell whether you broke it | An SNMPv2c timeout looks identical for a wrong community, a filtered path, and a device that never worked | Ask Prometheus before rolling anything back: `max_over_time(up{job="snmp",instance=""}[30d])`. `1` means it worked before you started, so the rotation is the suspect. `0` means it never worked, the rotation is not the cause, and rolling back will not help | +| One device fails right after you changed it | The UI truncated the community, or you removed the wrong entry | Re-enter it; check the field's max length (the APC NMC truncates at 15 on several firmwares; the card in this lab does not — it is verified at 16) | | `--old` reports `STILL ACCEPTED` | The device added the new community alongside the old one | Delete the old entry explicitly — some UIs need the row deleted, not blanked | | `--old` reports `SKIP` | The current-community check failed for that device, so a timeout proves nothing | Fix the current check first | | `error: ... contains whitespace or '#'` | A community was typed with a space or `#` into SOPS | `make secrets-edit`; regenerate with `make gen-secret` | diff --git a/docs/security.md b/docs/security.md index 405eec0..e24be13 100644 --- a/docs/security.md +++ b/docs/security.md @@ -59,7 +59,7 @@ repository must be treated as compromised: | What | Where | Status | | --- | --- | --- | -| SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` | Replaced with per-device placeholders. **Rotate on the devices** — see [runbook](runbooks/rotate-snmp-community.md) | +| SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` | Replaced with four distinct per-device values, SOPS-encrypted. Rotated on `morpheus`, `mjolnir` and `shiva`, each verified answering its new community and refusing the old. **Not rotated on `neo`** — see [#22](https://github.com/Gerrrt/HomeLab/issues/22) and the [runbook](runbooks/rotate-snmp-community.md) | | Grafana `admin` / `admin` with anonymous Admin access | compose file | Fixed: password from SOPS, anonymous auth disabled | | Passphrase-encrypted TLS private keys | `certificates/`, added in `efb2632`, deleted in `647d90a` but reachable at `647d90a~1` | Still in history. **Purge and regenerate** — see [runbook](runbooks/purge-git-history.md) | @@ -77,12 +77,16 @@ stale and the file gets deleted. The devices are polled with SNMPv2c, which transmits the community string in cleartext. Anyone with a port on the management VLAN can read it off a single -packet. Two mitigations are in place and one is not: - -- **Done:** each device now has a distinct community, so one captured packet - does not grant read access to the whole fleet. -- **Done:** SNMP is only reachable on the management VLAN, which nothing but - specific trusted hosts can enter. +packet. Two mitigations are in place, one only partly, and one is not: + +- **Done, with one exception:** each device has its own community, so one + captured packet no longer grants read access to the whole fleet. Three are + confirmed live on the hardware. The switch's exists only in SOPS — it has + never been proven on the device, because `10.7.7.2` has never answered a poll + ([#22](https://github.com/Gerrrt/HomeLab/issues/22)). +- **Done:** SNMP is reachable only on the management VLAN and the + switch-management LAN, neither of which anything but specific trusted hosts + can enter. - **Not done:** SNMPv3 with authPriv. The MokerLink switch does not support it. Tracked in [roadmap](roadmap.md). diff --git a/secrets/README.md b/secrets/README.md index 78a1c70..41a2bd9 100644 --- a/secrets/README.md +++ b/secrets/README.md @@ -82,6 +82,10 @@ See [`docs/runbooks/rotate-snmp-community.md`](../docs/runbooks/rotate-snmp-comm `make snmp-verify` confirms a rotation landed without putting the community into your shell history. +Which devices are actually rotated is recorded in +[`SECURITY.md`](../SECURITY.md), not here. One place to correct when it changes +is the only arrangement that survives the next rotation. + > **Note on this repository's history.** Earlier commits contained a plaintext > SNMP community string shared across all four devices, and encrypted TLS > private keys. Both must be treated as compromised regardless of the current