Skip to content

No dashboard shows firing alerts — add an Alertmanager panel #36

Description

@Gerrrt

What and why

The stack has 40 alert rules — 32 Prometheus and 8 Loki — and one
Alertmanager routing tree, and no dashboard shows any of them. None of the
five provisioned dashboards references the Alertmanager datasource:

$ grep -l '"alertmanager"' stacks/observability/grafana/dashboards/*.json
(no matches)

The datasource is provisioned and working. Verified through the Grafana proxy:

{ "cluster": { "status": "ready" }, "versionInfo": { "version": "0.33.1" } }

So the only ways to see what is firing today are the Alertmanager web UI or
curl localhost:9090/api/v1/alerts. Neither is where anyone looks. Right now
that hides exactly one warning — IloBatteryCondition, the ProLiant's failed
system battery — which has been firing for over a day without appearing on any
screen anyone opens.

This matters more than it looks, because notification delivery is fragile here
by design: the Alertmanager webhook was pointed at ntfy.example.invalid for
the entire life of the stack, so no alert had ever been delivered until it
was fixed in #31. A dashboard panel is the fallback that does not depend on a
receiver being correctly configured.

Proposal

Add an alert list panel backed by the Alertmanager datasource
(uid: alertmanager). Two reasonable shapes, no strong preference:

  • A panel on each existing dashboard, filtered to that dashboard's concern —
    severity/category labels already exist on the rules, and network-snmp
    and ups-power map cleanly onto them.
  • One "Alerts" dashboard listing everything firing, grouped by severity,
    with the other five linking to it.

Worth including regardless:

  • firing alerts grouped by severity, most severe first;
  • how long each has been firing — a warning that has been up for a day reads
    very differently from one that started a minute ago;
  • silences, so a silenced alert is visibly silenced rather than absent. An alert
    that vanishes when silenced is how a silence outlives the reason for it.

Notes

  • The Alertmanager datasource plugin is frontend-only — its plugin.json
    has no "backend": true, unlike prometheus. So
    /api/datasources/uid/alertmanager/health returns plugin unavailable, and
    that is expected rather than a fault. Do not chase it; the datasource works.
  • scripts/check_dashboards.py runs in CI and will hold any new panel to the
    24-column grid with no overlaps, a resolvable datasource UID, and a target on
    every panel. Adding a panel to an existing dashboard means re-flowing the
    gridPos of everything below it.
  • Dashboards are provisioned from files with allowUiUpdates: false, so this is
    an edit to the JSON in stacks/observability/grafana/dashboards/, not a
    change made in the UI. docs/roadmap.md already notes that the export loop is
    manual and therefore skipped under pressure.
  • Grafana now serves TLS (feat(observability): serve Grafana over TLS with the lab CA #35), which changes nothing here but is worth knowing
    when testing against the API.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions