Skip to content

fix(observability): show alerts with a table, not the alert list panel - #38

Merged
Gerrrt merged 1 commit into
mainfrom
fix/alert-panel-datasource
Aug 19, 2026
Merged

fix(observability): show alerts with a table, not the alert list panel#38
Gerrrt merged 1 commit into
mainfrom
fix/alert-panel-datasource

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Fixes #37, whose panels rendered nothing. Verified on screen this time.

What was wrong

Three attempts, each wrong differently:

  1. Alertmanager datasource. Chosen so silences would show. But alertlist
    reads alert rules, and an Alertmanager datasource has none — nothing to
    enumerate.
  2. Prometheus/Loki by uid. UnifiedAlertList.tsx matches
    dataSourceName === options.datasource — the datasource name, not the
    uid. Nothing matched.
  3. Name corrected. Still No alerts matching filters — and still empty with
    the label filter removed entirely. That bisect settled it: the panel source
    says this should work, it does not, and the fault is not visible server-side.

Every one of those passed check_dashboards.py, validate.sh, and my own API
checks. None of those touch what the panel actually draws, which is the whole
lesson here.

What this does instead

Plain table panels over the ALERTS series Prometheus exposes for its own
rules:

ALERTS{component="network",alertstate=~"firing|pending"}

The point is that this is an ordinary PromQL query, so the exact expression each
panel runs was verified through Grafana’s own datasource proxy before asking
anyone to look at a screen.

Columns: Alert, State, Category, Device, Instance, Severity, VLAN — with
severity and state colour-mapped.

Two deliberate costs

Silences are not reflected. ALERTS is Prometheus’s view; silences live in
Alertmanager, so a silenced alert still shows. That was the original argument
for the Alertmanager datasource, given up for a panel that works.

The Logs dashboard loses its panel. Loki’s 8 rules are evaluated by Loki’s
ruler and go straight to Alertmanager — they never enter Prometheus ALERTS, so
component="logs" would be permanently empty. A panel that always reads clear
implies there are no log alerts, which is worse than no panel.

Both are worth revisiting if alertGroups ever ships as a dashboard panel in
Grafana OSS, or if the alertlist issue is understood.

Also reverted

The check_dashboards.py alertlist exemption from #37 — no alertlist panel
remains, and it documented behaviour that does not work here.

Blast radius

Dashboard JSON only, applied with make reload.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

The panel’s exact query, through Grafana’s proxy:

ALERTS{component="network",alertstate=~"firing|pending"}
-> Alert=IloBatteryCondition State=firing Severity=warning Device=shiva Instance=10.0.30.10

Confirmed rendering on screen — the table draws all seven columns and the
firing row. That is the check the previous three attempts lacked.

./scripts/validate.sh passes — 5 dashboards OK, 84 panels, 83 PromQL
expressions.

  • make validate passes
  • Deployed to the lab and confirmed working
  • Docs updated (panel descriptions carry the rationale and the silence caveat)

Refs #36

The alertlist panels added in #37 rendered nothing. Three attempts to make them
work, each wrong in a different way, and the third only proved the approach was
unusable here.

First the panels pointed at the Alertmanager datasource, chosen so silences
would show. alertlist reads alert *rules*; an Alertmanager datasource has none,
so there was nothing to enumerate. Then they pointed at Prometheus and Loki by
datasource uid, but UnifiedAlertList.tsx matches options.datasource against the
datasource *name* (`dataSourceName === options.datasource`), so nothing matched.
With the name corrected the panel still showed "No alerts matching filters" —
and still did with the label filter removed entirely, which is what settled it.
The panel source says this should work; it does not, and the fault is not
visible from the server side.

So these are plain table panels over the ALERTS series Prometheus exposes for
its own rules, filtered by component and state. The whole point is that this is
an ordinary PromQL query: the exact expression each panel runs was verified
through Grafana's own datasource proxy before asking anyone to look at a screen.
The previous three attempts all passed every check I could run and were still
broken, because none of those checks touched what the panel actually renders.

Two costs, both deliberate:

Silences are not reflected. ALERTS is Prometheus's view and silences live in
Alertmanager, so a silenced alert still appears. That was the original argument
for the Alertmanager datasource and it is given up for a panel that works.

The Logs dashboard loses its panel entirely. Loki's eight rules are evaluated by
Loki's ruler and go straight to Alertmanager, never entering Prometheus ALERTS,
so component="logs" would be permanently empty. A panel that always reads clear
implies there are no log alerts, which is worse than no panel.

The check_dashboards.py exemption from #37 is reverted — no alertlist panel
remains, and it documented behaviour that turned out not to work here.

Refs #36
@Gerrrt
Gerrrt merged commit 2d3ea7f into main Aug 19, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the fix/alert-panel-datasource branch August 19, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant