Skip to content

fix(snmp): poll the MokerLink switch gently enough not to wedge it - #25

Merged
Gerrrt merged 1 commit into
mainfrom
fix/mokerlink-gentle-snmp
Aug 19, 2026
Merged

fix(snmp): poll the MokerLink switch gently enough not to wedge it#25
Gerrrt merged 1 commit into
mainfrom
fix/mokerlink-gentle-snmp

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Staged, not deployed. See sequencing below.

What changed

Adds max_repetitions: 5, retries: 2, timeout: 7s to the mokerlink
module in generator.yaml, and regenerates snmp.yaml.

The module previously carried no request tuning, so snmp-exporter:v0.30.1
polled the switch with its defaults — GETBULK at max_repetitions: 25,
retries: 3, timeout: 5s.

Why

This switch has locked its management CPU under exporter polling before. What
recovered it was a power cycle followed by small GETBULK batches and a longer
per-request timeout. That tuning was hand-applied at the time and never carried
into the generated config, so the defaults have been driving it continuously
since the stack was rebuilt.

A wedged SNMP agent is what the current symptom looks like: UDP/161 times out
while layer 2 keeps forwarding and the web UI answers on TCP/80 — different
processes, different fates. The walk scope was never a suspect; the module is
already 1.3.6.1.2.1.2.2 (ifTable) only. It is the request shape.

Worst case is retries * timeout = 21s, comfortably inside the job's 45s
scrape_timeout.

Blast radius

snmp-exporter only, and within it the mokerlink module only. The other
three modules are byte-identical. Nothing touches segmentation, ports or
credentials.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

$ make snmp-generate
metrics: 1792 -> 1792
ok — 4 placeholder(s) survived generation
  ilo=1598  apc_ups=59  pfsense=113  mokerlink=22

Regeneration changed exactly three lines, all inside the mokerlink module
(snmp.yaml is -diff in .gitattributes, so verified with a textual diff):

@@ -16261,6 +16261,9 @@
       indexes:
       - labelname: ifIndex
         type: gauge
+    max_repetitions: 5
+    retries: 2
+    timeout: 7s
   pfsense:

./scripts/validate.sh — all checks pass. make render run, so the file is
staged on disk and inside the container; the running exporter still holds the
previous config in memory.

  • make validate passes
  • Deployed to the lab — deliberately not. make reload is held back, see below
  • Docs updated (rationale is in generator.yaml, next to the values)

Sequencing — this matters

This change does not revive the switch on its own. It is what stops the
switch being driven straight back into the same state.

  1. make reload — apply the gentle polling first
  2. Power-cycle the switch
  3. ./scripts/snmp-verify.sh --device neo

Doing 2 before 1 means the exporter resumes hammering it at max_repetitions: 25 the moment it comes back.

Refs #22

The mokerlink module carried no request tuning, so snmp-exporter polled it with
its defaults: GETBULK at max_repetitions 25, retries 3, timeout 5s. This switch
has locked up under exporter load before, and what recovered it was a power
cycle followed by small batches and a longer per-request timeout. That tuning
was never carried into the generated config, so the defaults have been applied
to it continuously since the stack was rebuilt.

A wedged management CPU is what the current symptom looks like: UDP/161 times
out while layer 2 keeps forwarding and the web UI on TCP/80 keeps answering.
The walk scope was never the problem — the module is already ifTable only.

Pins max_repetitions 5, retries 2, timeout 7s. Worst case is 21s, inside the
job's 45s scrape_timeout. Regeneration changed exactly three lines and no
metric counts (1792 -> 1792, iLO still 1598).

This does not by itself revive the switch — it needs a power cycle. It is what
stops the switch being driven back into the same state afterwards, so it wants
to be live before that power cycle, not after.

Refs #22

Co-Authored-By: Claude Opus 5 <[email protected]>
@Gerrrt
Gerrrt merged commit df604e9 into main Aug 19, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the fix/mokerlink-gentle-snmp branch August 19, 2026 01:48
Gerrrt added a commit that referenced this pull request Aug 19, 2026
fix(snmp): poll the MokerLink switch gently enough not to wedge it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant