neo (MokerLink, 10.7.7.2) stopped answering SNMP — ICMP and TCP/80 succeed, UDP/161 does not

[Gerrrt]

Area: SNMP / network devices

What should happen

snmp-exporter polls 10.7.7.2 every 60s with module=mokerlink, auth=auth_mokerlink, and
up{job="snmp",instance="10.7.7.2"} is 1. docs/observability.md lists the switch as a
collected source, and ./scripts/snmp-verify.sh --device neo reports PASS with a sysDescr
string.

What happens instead

The target has never been up. Not "went down during the SNMP community rotation" — never, for the
entire retention window:

max_over_time(up{job="snmp",instance="10.7.7.2"}[30d])    # 0
count_over_time(up{job="snmp",instance="10.7.7.2"}[30d])  # 23070 samples

SnmpTargetUnreachable and InstanceDown have been firing continuously as a result — alert
fatigue on a rule that is working correctly.

The switch is reachable at layer 3 from the monitoring host (10.0.99.20): ICMP succeeds and
TCP/80 (its management web UI) is open, both routed via pfSense. Only UDP/161 fails. Probes with
the current community from SOPS, and separately with public and private, all time out —
inconclusive by construction, because SNMPv2c has no "wrong community" reply and a rejecting
device is indistinguishable from a silent one.

Because ICMP and TCP replies already reach 10.0.99.20, the switch's return path is working. That
rules out a missing default gateway or route on the switch, and leaves two candidates:

1. The switch's SNMP agent is disabled, has no read community, or restricts the NMS to a host
   list that does not include 10.0.99.20.
2. pfSense passes ICMP and TCP/80 from 10.0.99.20 → 10.7.7.2 but not UDP/161.
   docs/network.md lists 10.7.7.0/24 as reaching "Nothing", so the poll depends on a rule that
   is deliberately not published in this repository. The rotation runbook already flagged that rule
   as the first suspect; the ICMP/TCP-pass, UDP-fail split narrows it from "the rule is missing" to
   "the rule may not cover this protocol and port".

Relevant output

$ ./scripts/snmp-verify.sh

Current community
  PASS morpheus   10.0.99.1    <sysDescr>
  PASS mjolnir    10.0.99.10   <sysDescr>
  FAIL neo        10.7.7.2     no response (wrong community, filtered, or down)
  PASS shiva      10.0.30.10   <sysDescr>

snmp verification failed

$ ping -c 3 10.7.7.2        # 0% packet loss, ~3ms
$ ip route get 10.7.7.2     # via 10.0.99.1
$ tcp/80                    # open

Diagnostic sequence

Steps 0–2 change nothing. The point is to reach step 3 knowing which hypothesis is being fixed —
widening a firewall rule that did not need touching is the failure mode here.

0. Baseline, from the monitoring host:

ping -c 3 10.7.7.2
nc -vz -w 3 10.7.7.2 80
./scripts/snmp-verify.sh --device neo --dry-run   # confirms IP / auth / secret key name
./scripts/snmp-verify.sh --device neo

1. Ask pfSense whether the packet even left — the discriminator, no config change.
Status → System Logs → Firewall, filtered on 10.7.7.2 / 161 / UDP, then re-run the probe.
A block entry confirms hypothesis 2 — stop touching the switch. No entry is only weakly
informative, since pass rules do not log by default, so also check Diagnostics → States for a
udp state to :161. Then read the rule under Firewall → Rules: one written for TCP, or for
any port but TCP-only, produces exactly this signature. Record only that the rule exists and what
protocol it covers — rule bodies are deliberately unpublished.

2. Prove the agent from the switch's own segment, bypassing the firewall entirely — a host on
10.7.7.0/24. Answers there but not from 10.0.99.20 → hypothesis 2. Times out there too →
hypothesis 1, and no firewall change will help.

Avoid snmpget -c '<community>' from pfSense's shell: it puts the live community into root's
history and /proc/<pid>/cmdline — the exact leak scripts/snmp-verify.sh exists to close. If
you do it anyway, treat that community as burned and rotate the switch again afterwards.

3. Switch UI — http://10.7.7.2 → SNMP, read-only inspection, in order: is the agent
globally enabled (most likely answer — switches in this class often ship with it off); is v1/v2c
enabled; is there a read-only community and does it match SNMP_COMMUNITY_MOKERLINK; is there an
NMS/manager allow-list excluding 10.0.99.20 (plausible, since it is on a different subnet), or an
agent bound to a specific management VLAN. Delete any public/private entries while there.

4. Fix. Hypothesis 1 → the runbook's §2.1 MokerLink row, then §2.2–§2.5. Hypothesis 2 → a
firewall change, which goes through the change.yml issue path with its segmentation checklist.

5. Either way, no redeploy. The community is already in SOPS and already in the rendered
snmp.yaml, so configuring the switch with it needs no repository change: the next 60-second
scrape goes green on its own. No make render, no make reload, no commit.

Related

• Blocks the last device in Rotate the SNMP communities on all four devices #9. Until the switch answers, ./scripts/snmp-verify.sh --old returns
  SKIP for it by design, so whether it still accepts the community leaked in this repository's
  history is unknown, not proven clean. That unknown is why Rotate the SNMP communities on all four devices #9 stays open.
• docs/observability.md lists the switch as a collected source. That row is aspirational today
  and will be true again once this is fixed, so it is deliberately left unedited.
• The roadmap item "Add ifXTable (64-bit counters) to the mokerlink SNMP module" is unreachable
  until this is fixed — there are no 32-bit counters to improve on yet.

[Gerrrt]

Correction: "never answered" was wrong, and the diagnosis changes with it

The opening claim of this issue — that neo has never answered SNMP — is not
supported. It rested on max_over_time(up{job="snmp",instance="10.7.7.2"}[30d])
being 0, which measures Prometheus retention, not the device:

count_over_time(up{job="snmp"}[30d])
  morpheus  10.0.99.1    23129
  mjolnir   10.0.99.10   23127
  neo       10.7.7.2     23127
  shiva     10.0.30.10   23128

All four series carry the same sample count and begin together — the TSDB holds
no SNMP data from before the current stack was built, so a zero across the whole
window cannot speak to what the switch did earlier. It last polled
successfully in November 2025, under a hand-tuned minimal module.

The defensible claim is narrower: neo is not answering now, and has not for
the entire span of available data.

Revised leading hypothesis: the SNMP agent is wedged, not filtered

This has happened before. The switch previously locked up under exporter polling
load and needed a power cycle; the config that recovered it used
max_repetitions: 5 with sequential GETs and a 7s timeout, deliberately gentle.

That tuning does not exist in the current configuration:

$ grep -nE 'max_repetitions|retries|timeout' snmp-exporter/generator.yaml
(no matches)

So prom/snmp-exporter:v0.30.1 is polling it with its default
max_repetitions: 25 via GETBULK — close to the load that wedged it before.
The mokerlink module itself is correctly minimal (1.3.6.1.2.1.2.2, ifTable
only), so the walk scope is not the problem; the request shape is.

This fits the observed signature better than the firewall theory. ICMP and
TCP/80 answer because they are handled by different processes on the switch; a
management CPU whose SNMP agent has crashed or is pinned looks exactly like
this. A pfSense rule that passes ICMP and TCP but not UDP/161 is still possible,
but it is now the second suspect, not the first.

Revised order of work

1. Power-cycle the switch. Cheapest test of the wedge hypothesis, and the
   documented recovery from last time. If SNMP answers afterwards, the firewall
   was never involved.
2. Add the tuning before it is polled again, or it will wedge a second time.
   max_repetitions: 5 (and a raised timeout) on the mokerlink module in
   generator.yaml, then make snmp-generate. Without this, step 1 buys hours,
   not a fix.
3. Only if it still does not answer after a power cycle: the pfSense
   protocol/port check described above.

The SnmpTargetUnreachable alert firing continuously is a real cost here — it
is 16 days of a correct rule being ignored.

[Gerrrt]

The wedge hypothesis is wrong. The management plane is healthy.

Gentle polling is confirmed live — Loaded config file at 01:52:36Z, and the
exporter's own errors changed from after 3 retries (default) to after 2,
which is the tuning from #25. So the switch is now being polled at
max_repetitions: 5, and still answers nothing.

More decisively, the switch's management CPU is plainly fine:

$ curl http://10.7.7.2/
HTTP/1.1 200 OK
Content-Type: text/html
<title>G2402GSM</title>

$ tcp/23
        Gigabit Management Switch
        Command Line Interface
        Code Version: V3.3

Model: MokerLink G2402GSM, code version V3.3. HTTP serves pages, the telnet
CLI presents its banner. A switch whose management processor had locked up would
not do either. That was the leading theory and it does not survive contact.

#25 is still worth keeping — max_repetitions: 25 on this class of switch is a
real hazard and it has bitten this device before — but it was not the fix here.

The community is not the blocker either

Every candidate was probed directly against 10.7.7.2 (via defCommunity in a
0600 snmp.conf under SNMPCONFPATH, so nothing reached argv or shell
history):

Community	Result
current, from SOPS	no response
the old leaked shared string	no response
public	no response
private	no response
admin	no response

The old community is the interesting one. This switch was never rotated, so if
its SNMP agent were running with its previous configuration, the old string
should work. It does not. Combined with a healthy management plane, that points
away from "wrong credential" entirely.

The repo side is internally consistent and correctly plumbed, for the record:
SNMP_COMMUNITY_MOKERLINK renders into auth_mokerlink, no whitespace or #,
and the exporter is using it (auth=auth_mokerlink in its scrape errors).

What could not be determined from the monitoring host

Whether UDP/161 reaches the switch at all. The obvious test — a UDP probe to a
closed port, where an ICMP port-unreachable would prove the packet arrived —
does not work across this firewall. Control: shiva (10.0.30.10), which is
reached through pfSense and does answer SNMP, is equally silent on udp/9999.
So neo's silence there proves nothing.

TCP scan of neo, which does show what the path permits:

tcp/22   closed/filtered
tcp/23   OPEN      (telnet CLI)
tcp/80   OPEN      (web UI)
tcp/443  closed/filtered
tcp/8080 closed/filtered

Next step: the switch's own CLI

Telnet being open makes this cheap — no laptop on 10.7.7.0/24 required.
From the monitoring host:

telnet 10.7.7.2

Then check whether the SNMP agent is enabled and what communities exist —
show snmp, or the equivalent under this firmware's menu. Two outcomes:

• SNMP disabled / no community configured → that is the answer. Enable it,
  set SNMP_COMMUNITY_MOKERLINK, and Rotate the SNMP communities on all four devices #9's last device closes with it.
• SNMP enabled with a community → the agent is listening and answering
  nobody, so the packets are not arriving, and pfSense's UDP/161 rule becomes
  the remaining suspect.

Note also that the community in SOPS is 24 characters. If the CLI or web UI
silently truncates it — the failure mode this repo already documents for the APC
NMC at 15 — that will present as the switch rejecting a string that was typed
correctly. Worth checking the field's limit before assuming the write took.

Security note, incidental

tcp/23 is open: management over telnet, in plaintext, on the segment
everything routes through. Worth disabling in favour of the web UI, or at least
recording as a known exposure.

[Gerrrt]

Root cause: there is no firewall rule permitting SNMP to the switch

pfSense publishes its rule labels and per-rule counters over the SNMP we
already collect (pfLabelsLblName, pfLabelsLblEvals, pfLabelsLblPktsIn,
from BEGEMOT-PF-MIB under 1.3.6.1.4.1.12325.1.200.1.11). Decoding them
answers the question without touching the firewall UI.

Keeping to this repository's convention of not publishing rule bodies, the
finding in the abstract:

• There is an explicit, named rule permitting the SNMP exporter to poll the
  BMC on VLAN 30. It has passed ~270k packets. It works, which is why
  shiva is up.
• There is an explicit, named rule permitting HTTP to the LAN switch. That
  is why http://10.7.7.2 answers.
• There is no rule of either kind for SNMP to the switch. No allow, and no
  explicit block either — it simply falls through to the segment's default deny.

So the poll never reaches neo. The community was never relevant, the walk
scope was never relevant, and the switch's SNMP agent may well have been
listening the whole time.

This also explains the shape that misled the earlier analysis. ICMP, telnet and
HTTP to the switch succeed because they are separately permitted; UDP/161 is the
one thing nobody ever allowed. That looked like a selectively wedged agent. It
was a missing rule.

Correction to earlier comments in this issue

Both prior hypotheses recorded here were wrong, and the record should say so:

• "Wedged SNMP agent" — disproved. The management CPU serves HTTP 200 and a
  telnet CLI banner (G2402GSM, code version V3.3). It is healthy.
• "Never answered SNMP" — already corrected in docs: stop claiming neo has never answered SNMP #24; it last polled in
  November 2025.

The tuning in #25 (max_repetitions: 5) is still worth keeping — this switch has
genuinely locked up under max_repetitions: 25 before — but it fixed nothing
here, and should not be described as the fix.

Fix

Add a rule permitting 10.0.99.20 → 10.7.7.2 on UDP/161, modelled on the
existing VLAN 30 SNMP rule. Per the repo's own convention a firewall change goes
through the change.yml issue path with its segmentation checklist, so that is
the next step rather than something to fold in here.

Once it is in place:

./scripts/snmp-verify.sh --device neo

If it answers, the switch still holds whatever community it had in November —
which is the point at which #9's last device becomes actionable, and at which
--old finally becomes meaningful for neo.

One loose end worth a look while you are in there

The "HTTP to LAN switch" rule reports pktsIn = 0 despite ~755k evaluations,
while HTTP to 10.7.7.2 demonstrably works. Either the counter does not track
that direction, or that traffic is actually being passed by a different rule and
this one never matches. Not urgent, but a named rule that never matches is
usually not what its author intended.

[Gerrrt]

Resolved: two faults, not one

The switch is answering SNMP. sysDescr is "Switch", ifNumber is 26, and a
walk of 1.3.6.1.2.1.2.2 — the exact OID the mokerlink module uses — returns
572 rows with sensible port names (port1, port2, …). The walk was run at
max_repetitions: 5 and the switch did not flinch.

There were two independent problems stacked on each other, which is why each
single-cause hypothesis kept failing:

1. No firewall rule permitted the poll. Fixed. pfSense now carries a rule
allowing the monitoring host to reach the switch on UDP/161, and its counters
prove traffic is traversing it — the rule matched and passed packets during
testing, where before there was no such rule at all. Until this existed, nothing
downstream could ever have worked, and every community test was meaningless
because no packet arrived.

2. The community in SOPS was never set on the device. This is issue #9's
open box, not a fault: the switch is the one device that was never rotated, so
it still holds its own pre-existing community. SNMP_COMMUNITY_MOKERLINK is a
value the switch has never seen. Confirmed by testing the device's actual
community directly — it answers immediately, on both v2c and v1.

For the record, the switch's existing community is not the string leaked in
this repository's history; the two differ. So the switch was not exposed by that
leak, which narrows the blast radius of #9 by one device.

What is left

Only #9's last step: set SNMP_COMMUNITY_MOKERLINK on the switch, per the
runbook's §2.1 MokerLink row, then §2.4/§2.5 to verify and remove the old entry.
No repository change is needed — the value is already in SOPS and already
rendered into the running exporter, so the next scrape goes green on its own.

One caution before doing it. The community currently on the switch is 15
characters; the one in SOPS is 24. Whether this firmware accepts 24 is unknown,
and silent truncation is the exact failure this repo already documents for the
APC NMC. Either verify the 24-character value took after setting it, or
pre-empt it:

./scripts/gen-secret.sh --length 15

then make secrets-edit, make render, make reload.

Corrections to this issue's record

Three hypotheses were recorded here and all three were wrong. Keeping them
visible rather than quietly deleting them:

• "Never answered SNMP" — false; corrected in docs: stop claiming neo has never answered SNMP #24.
• "Wedged SNMP agent" — false. The agent was healthy the whole time and
  answering its own community; nothing ever reached it.
• "The missing firewall rule is the root cause" — right about the missing
  rule, wrong to call it the cause. It was one of two, and fixing it alone
  changed nothing observable, because the community mismatch sat behind it.

The tuning in #25 remains justified on its own merits — this switch has locked
up under max_repetitions: 25 before — but it fixed nothing here.