You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The stack has 40 alert rules — 32 Prometheus and 8 Loki — and one
Alertmanager routing tree, and no dashboard shows any of them. None of the
five provisioned dashboards references the Alertmanager datasource:
So the only ways to see what is firing today are the Alertmanager web UI or curl localhost:9090/api/v1/alerts. Neither is where anyone looks. Right now
that hides exactly one warning — IloBatteryCondition, the ProLiant's failed
system battery — which has been firing for over a day without appearing on any
screen anyone opens.
This matters more than it looks, because notification delivery is fragile here
by design: the Alertmanager webhook was pointed at ntfy.example.invalid for
the entire life of the stack, so no alert had ever been delivered until it
was fixed in #31. A dashboard panel is the fallback that does not depend on a
receiver being correctly configured.
Proposal
Add an alert list panel backed by the Alertmanager datasource
(uid: alertmanager). Two reasonable shapes, no strong preference:
A panel on each existing dashboard, filtered to that dashboard's concern — severity/category labels already exist on the rules, and network-snmp
and ups-power map cleanly onto them.
One "Alerts" dashboard listing everything firing, grouped by severity,
with the other five linking to it.
Worth including regardless:
firing alerts grouped by severity, most severe first;
how long each has been firing — a warning that has been up for a day reads
very differently from one that started a minute ago;
silences, so a silenced alert is visibly silenced rather than absent. An alert
that vanishes when silenced is how a silence outlives the reason for it.
Notes
The Alertmanager datasource plugin is frontend-only — its plugin.json
has no "backend": true, unlike prometheus. So /api/datasources/uid/alertmanager/health returns plugin unavailable, and
that is expected rather than a fault. Do not chase it; the datasource works.
scripts/check_dashboards.py runs in CI and will hold any new panel to the
24-column grid with no overlaps, a resolvable datasource UID, and a target on
every panel. Adding a panel to an existing dashboard means re-flowing the gridPos of everything below it.
Dashboards are provisioned from files with allowUiUpdates: false, so this is
an edit to the JSON in stacks/observability/grafana/dashboards/, not a
change made in the UI. docs/roadmap.md already notes that the export loop is
manual and therefore skipped under pressure.
What and why
The stack has 40 alert rules — 32 Prometheus and 8 Loki — and one
Alertmanager routing tree, and no dashboard shows any of them. None of the
five provisioned dashboards references the Alertmanager datasource:
The datasource is provisioned and working. Verified through the Grafana proxy:
{ "cluster": { "status": "ready" }, "versionInfo": { "version": "0.33.1" } }So the only ways to see what is firing today are the Alertmanager web UI or
curl localhost:9090/api/v1/alerts. Neither is where anyone looks. Right nowthat hides exactly one warning —
IloBatteryCondition, the ProLiant's failedsystem battery — which has been firing for over a day without appearing on any
screen anyone opens.
This matters more than it looks, because notification delivery is fragile here
by design: the Alertmanager webhook was pointed at
ntfy.example.invalidforthe entire life of the stack, so no alert had ever been delivered until it
was fixed in #31. A dashboard panel is the fallback that does not depend on a
receiver being correctly configured.
Proposal
Add an alert list panel backed by the Alertmanager datasource
(
uid: alertmanager). Two reasonable shapes, no strong preference:severity/categorylabels already exist on the rules, andnetwork-snmpand
ups-powermap cleanly onto them.with the other five linking to it.
Worth including regardless:
very differently from one that started a minute ago;
that vanishes when silenced is how a silence outlives the reason for it.
Notes
plugin.jsonhas no
"backend": true, unlikeprometheus. So/api/datasources/uid/alertmanager/healthreturnsplugin unavailable, andthat is expected rather than a fault. Do not chase it; the datasource works.
scripts/check_dashboards.pyruns in CI and will hold any new panel to the24-column grid with no overlaps, a resolvable datasource UID, and a target on
every panel. Adding a panel to an existing dashboard means re-flowing the
gridPosof everything below it.allowUiUpdates: false, so this isan edit to the JSON in
stacks/observability/grafana/dashboards/, not achange made in the UI.
docs/roadmap.mdalready notes that the export loop ismanual and therefore skipped under pressure.
when testing against the API.