fix(snmp): poll the MokerLink switch gently enough not to wedge it - #25
Merged
Conversation
The mokerlink module carried no request tuning, so snmp-exporter polled it with its defaults: GETBULK at max_repetitions 25, retries 3, timeout 5s. This switch has locked up under exporter load before, and what recovered it was a power cycle followed by small batches and a longer per-request timeout. That tuning was never carried into the generated config, so the defaults have been applied to it continuously since the stack was rebuilt. A wedged management CPU is what the current symptom looks like: UDP/161 times out while layer 2 keeps forwarding and the web UI on TCP/80 keeps answering. The walk scope was never the problem — the module is already ifTable only. Pins max_repetitions 5, retries 2, timeout 7s. Worst case is 21s, inside the job's 45s scrape_timeout. Regeneration changed exactly three lines and no metric counts (1792 -> 1792, iLO still 1598). This does not by itself revive the switch — it needs a power cycle. It is what stops the switch being driven back into the same state afterwards, so it wants to be live before that power cycle, not after. Refs #22 Co-Authored-By: Claude Opus 5 <[email protected]>
Gerrrt
added a commit
that referenced
this pull request
Aug 19, 2026
fix(snmp): poll the MokerLink switch gently enough not to wedge it
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Staged, not deployed. See sequencing below.
What changed
Adds
max_repetitions: 5,retries: 2,timeout: 7sto themokerlinkmodule in
generator.yaml, and regeneratessnmp.yaml.The module previously carried no request tuning, so
snmp-exporter:v0.30.1polled the switch with its defaults — GETBULK at
max_repetitions: 25,retries: 3,timeout: 5s.Why
This switch has locked its management CPU under exporter polling before. What
recovered it was a power cycle followed by small GETBULK batches and a longer
per-request timeout. That tuning was hand-applied at the time and never carried
into the generated config, so the defaults have been driving it continuously
since the stack was rebuilt.
A wedged SNMP agent is what the current symptom looks like: UDP/161 times out
while layer 2 keeps forwarding and the web UI answers on TCP/80 — different
processes, different fates. The walk scope was never a suspect; the module is
already
1.3.6.1.2.1.2.2(ifTable) only. It is the request shape.Worst case is
retries * timeout= 21s, comfortably inside the job's 45sscrape_timeout.Blast radius
snmp-exporteronly, and within it themokerlinkmodule only. The otherthree modules are byte-identical. Nothing touches segmentation, ports or
credentials.
secrets/*.sops.yamlVerification
Regeneration changed exactly three lines, all inside the
mokerlinkmodule(
snmp.yamlis-diffin.gitattributes, so verified with a textual diff):./scripts/validate.sh— all checks pass.make renderrun, so the file isstaged on disk and inside the container; the running exporter still holds the
previous config in memory.
make validatepassesmake reloadis held back, see belowgenerator.yaml, next to the values)Sequencing — this matters
This change does not revive the switch on its own. It is what stops the
switch being driven straight back into the same state.
make reload— apply the gentle polling first./scripts/snmp-verify.sh --device neoDoing 2 before 1 means the exporter resumes hammering it at
max_repetitions: 25the moment it comes back.Refs #22