Skip to content

docs: record which SNMP communities are actually rotated - #23

Merged
Gerrrt merged 2 commits into
mainfrom
docs/snmp-rotation-status
Aug 18, 2026
Merged

docs: record which SNMP communities are actually rotated#23
Gerrrt merged 2 commits into
mainfrom
docs/snmp-rotation-status

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Aug 18, 2026

Copy link
Copy Markdown
Owner

What changed

Documentation only. Records which SNMP communities are actually rotated on the
hardware — three of the four were, and nothing in the repository said so — and
folds what the rotation taught us back into its runbook.

Why

Closing the loop on #9, which the rotation runbook's §3 requires and which was
never done. SECURITY.md still asserted the communities were "not yet rotated
on the devices". That is materially false for pfSense, the APC NMC and iLO:
each answers to its own distinct new community and refuses the old shared one.

It is still true for the MokerLink switch, but not for the reason the issue
assumes. neo has never answered an SNMP poll at all —
max_over_time(up{job="snmp",instance="10.7.7.2"}[30d]) is 0 across 23,070
samples, so the failure predates the rotation. That is now tracked separately
in #22, and #9 stays open behind it: until the switch answers,
snmp-verify.sh --old returns SKIP for it, so whether it still accepts the
leaked community is unknown rather than proven clean.

A published SECURITY.md that understates what is fixed is as misleading as
one that overstates it.

Two things found en route:

  • docs/security.md claimed "SNMP is only reachable on the management VLAN".
    It is not — neo is polled at 10.7.7.2 on the switch-management LAN.
  • The runbook's close-the-loop list named three files and missed
    docs/security.md, which is the file SECURITY.md delegates detail to.

Blast radius

None. No rendered config, no secret, no compose service, no Prometheus target
changes. neo is deliberately left in prometheus/targets/snmp.yaml — removing
it to silence SnmpTargetUnreachable would be hiding the problem.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

./scripts/validate.sh — all checks pass (yamllint, markdownlint-cli2 and
gitleaks SKIP as not installed on this host; markdownlint-cli2 run
separately via npx, 0 issues).

Live state, re-established immediately before this PR, since the merged wording
asserts it:

$ ./scripts/snmp-verify.sh

Current community
  PASS morpheus   10.0.99.1    "pfSense morpheus.matrix.elysium 2.8.1-RELEASE ...
  PASS mjolnir    10.0.99.10   "APC Web/SNMP Management Card (MB:v4.2.9 PF:v2.0...
  FAIL neo        10.7.7.2     no response (wrong community, filtered, or down)
  PASS shiva      10.0.30.10   "Integrated Lights-Out 4 2.82 Feb 06 2023"

./scripts/snmp-verify.sh --old has since been run interactively (it needs a
terminal by design) and confirms it — rejected for morpheus, mjolnir and
shiva, SKIP for neo. Full output attached to #9:

Old community (must be refused)
  PASS morpheus   10.0.99.1    rejected
  PASS mjolnir    10.0.99.10   rejected
  SKIP neo        10.7.7.2     current-community check failed; a timeout here would prove nothing
  PASS shiva      10.0.30.10   rejected
  • make validate passes
  • Deployed to the lab and confirmed working — n/a, nothing rendered changes
  • Docs updated (docs/, service README, or docs/roadmap.md)

Refs #9, #22

Your Name and others added 2 commits August 18, 2026 23:34
Three of the four devices were rotated and nothing said so. pfSense, the APC
NMC and iLO each answer to their own new community and refuse the old shared
one; the MokerLink switch does not, and cannot be verified, because it has
never answered an SNMP poll at all.

SECURITY.md is the ledger and carries the per-device status. docs/security.md
carries the detail behind it — it also claimed SNMP was reachable only on the
management VLAN, which is wrong: neo is polled at 10.7.7.2 on the
switch-management LAN. secrets/README.md gets a pointer rather than a fifth
copy of the status to keep in sync.

The roadmap item stays unchecked but is renamed to name the switch. An
unchecked box reading "all four devices" reads as though nothing happened,
which is materially false.

Refs #9, #22

Co-Authored-By: Claude Opus 5 <[email protected]>
Nothing in the runbook let an operator tell "my rotation broke this" from "this
never worked", and that distinction decides whether to roll back. Adds the
max_over_time(up[30d]) check to §2.4 and to the troubleshooting table.

The neo row's hypothesis was right and is now evidence-backed, so it becomes a
test rather than a hunch: ICMP and TCP/80 to 10.7.7.2 both succeed while
UDP/161 does not, which narrows the pfSense rule from "missing" to "may not
cover this protocol and port".

§2.5 explained why --old skips a device but never what to do about it. A SKIP
is an open question, not a pass deferred — a device that never answered may
still accept the leaked community. Also records that --old refuses a pipe on
purpose, so no script or agent can run it.

The APC 15-character warning read as a property of the model. This lab's card
is verified working at 16.

The close-the-loop list was wrong by one: docs/security.md carries the detail
SECURITY.md points at, and leaving it stale makes the two disagree.

Refs #9, #22

Co-Authored-By: Claude Opus 5 <[email protected]>
@Gerrrt
Gerrrt merged commit 4d24a10 into main Aug 18, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the docs/snmp-rotation-status branch August 18, 2026 23:47
Gerrrt added a commit that referenced this pull request Aug 19, 2026
docs: record which SNMP communities are actually rotated
Gerrrt pushed a commit that referenced this pull request Aug 19, 2026
It has. #23 wrote "never answered an SNMP poll at all" into four files on the
strength of max_over_time(up{instance="10.7.7.2"}[30d]) == 0. That measured
Prometheus retention, not the device: all four SNMP series carry ~23,127
samples and begin together when the current stack was built, so the TSDB holds
no SNMP data from before that point and the window cannot speak to it.

The switch last polled successfully in November 2025, under a hand-tuned
minimal module. What is true is narrower and still enough to keep #9 open:
neo is not answering now, and has not for the whole span of available data.

Also records the sharper reading of the ICMP/TCP-pass, UDP-fail split — that
is what a wedged SNMP agent looks like, not a dead device.

Refs #9, #22

Co-Authored-By: Claude Opus 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant