Skip to content

feat(observability): show firing alerts on every dashboard - #37

Merged
Gerrrt merged 1 commit into
mainfrom
feat/alert-panels
Aug 19, 2026
Merged

feat(observability): show firing alerts on every dashboard#37
Gerrrt merged 1 commit into
mainfrom
feat/alert-panels

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Closes #36. Deployed and verified.

What changed

Each of the five dashboards gains an alertlist panel at the top, filtered to
its own concern. Every alert rule gains a component label to make that
filtering possible.

Dashboard Filter Rules
Network & Firewall {component="network"} 10
Host Overview {component="host"} 7
Docker Containers {component="containers"} 8
UPS & Power {component="power"} 7
Logs {component="logs"} 8 (Loki)

Why a new label

Neither existing label maps onto a dashboard. severity cuts across
everything; category splits a single dashboard across availability,
capacity, correctness and saturation. component is one value per rule
file, which is exactly the granularity a per-dashboard panel needs.

It is purely additive — alertmanager.yaml routes on category and severity,
so routing is untouched.

Why the Alertmanager datasource rather than the ALERTS metric

ALERTS would have been easier to verify and would have needed no validator
change, but it does not know about silences. An alert that simply vanishes when
silenced is how a silence outlives the reason for it — the panel should show it
as silenced.

Validator change

check_dashboards.py now treats alertlist as targetless alongside row and
text. It does read from a datasource, just not through a query — it pulls from
the Alertmanager datasource named in its own options, so a targets array is
meaningless. The alternative was a fake target on every alert panel, which
teaches people to work around the script rather than trust it.

Blast radius

Config only — no service definition changes, applied with make reload.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

One-time churn: adding a label changes an alert’s fingerprint, so anything
currently firing resolves and re-fires, restarting its for timer.
IloBatteryCondition (for: 30m) went back to pending at the reload and
returns to firing 30 minutes later. Expected, not a fault.

Verification

Panel options confirmed against the option paths in the shipped
alertListPanel bundle rather than guessed, then read back from Grafana after
provisioning:

homelab-network         Network & firewall alerts firing  filter={component="network"}
homelab-host-overview   Host alerts firing                filter={component="host"}
homelab-docker          Container alerts firing           filter={component="containers"}
homelab-ups             Power alerts firing               filter={component="power"}
homelab-logs            Log-based alerts firing           filter={component="logs"}

32 Prometheus rules reloaded, all labelled (network 10, host 7,
containers 8, power 7); Loki’s 8 carry component=logs.

The exact data path each panel uses — Alertmanager through the Grafana proxy,
label-filtered — confirmed working:

category="hardware"  -> 1 alert: IloBatteryCondition
category="power"     -> 0 alerts

component= filters return 0 right now purely because of the timer reset above;
ALERTS{component="network"} shows the label is on the pending alert and will
match when it fires.

./scripts/validate.sh passes — 5 dashboards OK, 85 panels, 79 PromQL
expressions; promtool check rules clean on all four Prometheus files; Loki’s 8
rules parse and evaluate.

  • make validate passes
  • Deployed to the lab and confirmed working
  • Docs updated (panel descriptions carry the rationale)

Forty alert rules, one routing tree, and not one dashboard showed any of them.
The only ways to see what was firing were the Alertmanager UI or curl, neither
of which is where anyone looks. That hid IloBatteryCondition — the ProLiant's
failed system battery — for over a day.

It matters more here than it looks, because delivery is fragile by design: the
webhook pointed at ntfy.example.invalid for the entire life of the stack, so no
alert had ever been delivered until #31. A panel is the fallback that does not
depend on a receiver being right.

Each dashboard gains an alertlist panel at the top, filtered to its own concern
and backed by the Alertmanager datasource rather than the ALERTS metric — so a
silenced alert shows as silenced instead of simply vanishing. An alert that
disappears when silenced is how a silence outlives the reason for it.

Filtering needed a label that maps onto dashboards, and neither existing one
does: severity cuts across everything, and category splits the same dashboard
across availability, capacity, correctness and saturation. So every rule gains
component, one value per rule file — network, host, containers, power, logs.
It is additive, and alertmanager.yaml routes on category and severity, so
routing is untouched.

check_dashboards.py learns that alertlist carries no targets. It does read from
a datasource, just not through a query — it pulls from the Alertmanager
datasource named in its own options. Requiring a target would have forced a
fake one onto every alert panel, which teaches people to work around the script
rather than trust it.

One-time churn worth expecting: adding a label changes an alert's fingerprint,
so currently-firing alerts resolve and re-fire, and their `for` timers restart.
IloBatteryCondition is pending again and returns to firing 30 minutes after the
reload.

Closes #36
@Gerrrt
Gerrrt merged commit c7a7cb8 into main Aug 19, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the feat/alert-panels branch August 19, 2026 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No dashboard shows firing alerts — add an Alertmanager panel

1 participant