feat(observability): show firing alerts on every dashboard - #37
Merged
Conversation
Forty alert rules, one routing tree, and not one dashboard showed any of them. The only ways to see what was firing were the Alertmanager UI or curl, neither of which is where anyone looks. That hid IloBatteryCondition — the ProLiant's failed system battery — for over a day. It matters more here than it looks, because delivery is fragile by design: the webhook pointed at ntfy.example.invalid for the entire life of the stack, so no alert had ever been delivered until #31. A panel is the fallback that does not depend on a receiver being right. Each dashboard gains an alertlist panel at the top, filtered to its own concern and backed by the Alertmanager datasource rather than the ALERTS metric — so a silenced alert shows as silenced instead of simply vanishing. An alert that disappears when silenced is how a silence outlives the reason for it. Filtering needed a label that maps onto dashboards, and neither existing one does: severity cuts across everything, and category splits the same dashboard across availability, capacity, correctness and saturation. So every rule gains component, one value per rule file — network, host, containers, power, logs. It is additive, and alertmanager.yaml routes on category and severity, so routing is untouched. check_dashboards.py learns that alertlist carries no targets. It does read from a datasource, just not through a query — it pulls from the Alertmanager datasource named in its own options. Requiring a target would have forced a fake one onto every alert panel, which teaches people to work around the script rather than trust it. One-time churn worth expecting: adding a label changes an alert's fingerprint, so currently-firing alerts resolve and re-fire, and their `for` timers restart. IloBatteryCondition is pending again and returns to firing 30 minutes after the reload. Closes #36
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #36. Deployed and verified.
What changed
Each of the five dashboards gains an
alertlistpanel at the top, filtered toits own concern. Every alert rule gains a
componentlabel to make thatfiltering possible.
{component="network"}{component="host"}{component="containers"}{component="power"}{component="logs"}Why a new label
Neither existing label maps onto a dashboard.
severitycuts acrosseverything;
categorysplits a single dashboard acrossavailability,capacity,correctnessandsaturation.componentis one value per rulefile, which is exactly the granularity a per-dashboard panel needs.
It is purely additive —
alertmanager.yamlroutes oncategoryandseverity,so routing is untouched.
Why the Alertmanager datasource rather than the ALERTS metric
ALERTSwould have been easier to verify and would have needed no validatorchange, but it does not know about silences. An alert that simply vanishes when
silenced is how a silence outlives the reason for it — the panel should show it
as silenced.
Validator change
check_dashboards.pynow treatsalertlistas targetless alongsiderowandtext. It does read from a datasource, just not through a query — it pulls fromthe Alertmanager datasource named in its own options, so a
targetsarray ismeaningless. The alternative was a fake target on every alert panel, which
teaches people to work around the script rather than trust it.
Blast radius
Config only — no service definition changes, applied with
make reload.secrets/*.sops.yamlOne-time churn: adding a label changes an alert’s fingerprint, so anything
currently firing resolves and re-fires, restarting its
fortimer.IloBatteryCondition(for: 30m) went back to pending at the reload andreturns to firing 30 minutes later. Expected, not a fault.
Verification
Panel options confirmed against the option paths in the shipped
alertListPanelbundle rather than guessed, then read back from Grafana afterprovisioning:
32 Prometheus rules reloaded, all labelled (
network10,host7,containers8,power7); Loki’s 8 carrycomponent=logs.The exact data path each panel uses — Alertmanager through the Grafana proxy,
label-filtered — confirmed working:
component=filters return 0 right now purely because of the timer reset above;ALERTS{component="network"}shows the label is on the pending alert and willmatch when it fires.
./scripts/validate.shpasses — 5 dashboards OK, 85 panels, 79 PromQLexpressions;
promtool check rulesclean on all four Prometheus files; Loki’s 8rules parse and evaluate.
make validatepasses