fix(observability): show alerts with a table, not the alert list panel - #38
Merged
Conversation
The alertlist panels added in #37 rendered nothing. Three attempts to make them work, each wrong in a different way, and the third only proved the approach was unusable here. First the panels pointed at the Alertmanager datasource, chosen so silences would show. alertlist reads alert *rules*; an Alertmanager datasource has none, so there was nothing to enumerate. Then they pointed at Prometheus and Loki by datasource uid, but UnifiedAlertList.tsx matches options.datasource against the datasource *name* (`dataSourceName === options.datasource`), so nothing matched. With the name corrected the panel still showed "No alerts matching filters" — and still did with the label filter removed entirely, which is what settled it. The panel source says this should work; it does not, and the fault is not visible from the server side. So these are plain table panels over the ALERTS series Prometheus exposes for its own rules, filtered by component and state. The whole point is that this is an ordinary PromQL query: the exact expression each panel runs was verified through Grafana's own datasource proxy before asking anyone to look at a screen. The previous three attempts all passed every check I could run and were still broken, because none of those checks touched what the panel actually renders. Two costs, both deliberate: Silences are not reflected. ALERTS is Prometheus's view and silences live in Alertmanager, so a silenced alert still appears. That was the original argument for the Alertmanager datasource and it is given up for a panel that works. The Logs dashboard loses its panel entirely. Loki's eight rules are evaluated by Loki's ruler and go straight to Alertmanager, never entering Prometheus ALERTS, so component="logs" would be permanently empty. A panel that always reads clear implies there are no log alerts, which is worse than no panel. The check_dashboards.py exemption from #37 is reverted — no alertlist panel remains, and it documented behaviour that turned out not to work here. Refs #36
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #37, whose panels rendered nothing. Verified on screen this time.
What was wrong
Three attempts, each wrong differently:
alertlistreads alert rules, and an Alertmanager datasource has none — nothing to
enumerate.
UnifiedAlertList.tsxmatchesdataSourceName === options.datasource— the datasource name, not theuid. Nothing matched.
No alerts matching filters— and still empty withthe label filter removed entirely. That bisect settled it: the panel source
says this should work, it does not, and the fault is not visible server-side.
Every one of those passed
check_dashboards.py,validate.sh, and my own APIchecks. None of those touch what the panel actually draws, which is the whole
lesson here.
What this does instead
Plain
tablepanels over theALERTSseries Prometheus exposes for its ownrules:
The point is that this is an ordinary PromQL query, so the exact expression each
panel runs was verified through Grafana’s own datasource proxy before asking
anyone to look at a screen.
Columns: Alert, State, Category, Device, Instance, Severity, VLAN — with
severity and state colour-mapped.
Two deliberate costs
Silences are not reflected.
ALERTSis Prometheus’s view; silences live inAlertmanager, so a silenced alert still shows. That was the original argument
for the Alertmanager datasource, given up for a panel that works.
The Logs dashboard loses its panel. Loki’s 8 rules are evaluated by Loki’s
ruler and go straight to Alertmanager — they never enter Prometheus
ALERTS, socomponent="logs"would be permanently empty. A panel that always reads clearimplies there are no log alerts, which is worse than no panel.
Both are worth revisiting if
alertGroupsever ships as a dashboard panel inGrafana OSS, or if the alertlist issue is understood.
Also reverted
The
check_dashboards.pyalertlist exemption from #37 — noalertlistpanelremains, and it documented behaviour that does not work here.
Blast radius
Dashboard JSON only, applied with
make reload.secrets/*.sops.yamlVerification
The panel’s exact query, through Grafana’s proxy:
Confirmed rendering on screen — the table draws all seven columns and the
firing row. That is the check the previous three attempts lacked.
./scripts/validate.shpasses — 5 dashboards OK, 84 panels, 83 PromQLexpressions.
make validatepassesRefs #36