From f2af662242e69fb7eed55a7dd3ffc22db151810a Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 19 Aug 2026 01:44:45 +0000 Subject: [PATCH] fix(snmp): poll the MokerLink switch gently enough not to wedge it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mokerlink module carried no request tuning, so snmp-exporter polled it with its defaults: GETBULK at max_repetitions 25, retries 3, timeout 5s. This switch has locked up under exporter load before, and what recovered it was a power cycle followed by small batches and a longer per-request timeout. That tuning was never carried into the generated config, so the defaults have been applied to it continuously since the stack was rebuilt. A wedged management CPU is what the current symptom looks like: UDP/161 times out while layer 2 keeps forwarding and the web UI on TCP/80 keeps answering. The walk scope was never the problem — the module is already ifTable only. Pins max_repetitions 5, retries 2, timeout 7s. Worst case is 21s, inside the job's 45s scrape_timeout. Regeneration changed exactly three lines and no metric counts (1792 -> 1792, iLO still 1598). This does not by itself revive the switch — it needs a power cycle. It is what stops the switch being driven back into the same state afterwards, so it wants to be live before that power cycle, not after. Refs #22 Co-Authored-By: Claude Opus 5 --- stacks/observability/snmp-exporter/generator.yaml | 15 +++++++++++++++ stacks/observability/snmp-exporter/snmp.yaml | 3 +++ 2 files changed, 18 insertions(+) diff --git a/stacks/observability/snmp-exporter/generator.yaml b/stacks/observability/snmp-exporter/generator.yaml index db754f4..549db20 100644 --- a/stacks/observability/snmp-exporter/generator.yaml +++ b/stacks/observability/snmp-exporter/generator.yaml @@ -83,9 +83,24 @@ modules: # NOTE: 32-bit counters only. Adding ifXTable (64-bit) is tracked in # docs/roadmap.md — on gigabit links ifInOctets wraps in ~34 seconds at # line rate, which the 60s scrape interval cannot detect. + # + # The tuning below is not cargo cult. This switch's management CPU has locked + # up under exporter polling before, taking its SNMP agent down while layer 2 + # kept forwarding and the web UI kept answering — which presents as UDP/161 + # timing out while ICMP and TCP/80 succeed. Recovery was a power cycle. What + # ran afterwards without wedging it used small GETBULK batches and a longer + # per-request timeout, so those values are pinned here rather than left to + # the exporter's defaults (max_repetitions 25, retries 3, timeout 5s). + # + # 5 * ~22 ifTable columns is a handful of round trips against a switch that + # cannot absorb 25. Worst case here is retries * timeout = 21s, comfortably + # inside the job's 45s scrape_timeout. mokerlink: walk: - 1.3.6.1.2.1.2.2 # IF-MIB::ifTable + max_repetitions: 5 + retries: 2 + timeout: 7s # HPE ProLiant DL360 Gen9 iLO (shiva, 10.0.30.10) — SNMPv2-MIB system group # plus the Compaq/HP Insight enterprise tree for hardware health. diff --git a/stacks/observability/snmp-exporter/snmp.yaml b/stacks/observability/snmp-exporter/snmp.yaml index 67d2033..587f7c9 100644 --- a/stacks/observability/snmp-exporter/snmp.yaml +++ b/stacks/observability/snmp-exporter/snmp.yaml @@ -16261,6 +16261,9 @@ modules: indexes: - labelname: ifIndex type: gauge + max_repetitions: 5 + retries: 2 + timeout: 7s pfsense: walk: - 1.3.6.1.4.1.12325.1.200.1.1