feat(observability): serve Grafana over TLS with the lab CA

[Gerrrt]

Closes the last roadmap item from the certificate work. Deployed and verified.

What changed

Grafana published plain HTTP on the management VLAN — every login sent its
password in cleartext to anything with a port on VLAN 99. It now serves the
grafana.matrix.elysium leaf issued by the lab CA.

Four things that had to move together

Each of these silently breaks the stack on its own, which is why they are one
commit rather than four:

The cookie. GF_SECURITY_COOKIE_SECURE was false because a secure cookie
over plain HTTP is never sent — you get a login page that accepts your
password and loops forever. It flips to true in the same change that makes
TLS available, never before it.

The healthcheck. busybox wget cannot be handed a CA, and the probe is
testing that Grafana is serving rather than that a self-signed chain builds — so
it uses --no-check-certificate against its own loopback.

The scrape. Prometheus reaches Grafana as grafana:3000, not by FQDN. The
leaf therefore carries grafana as an additional SAN, and the job verifies
properly with ca_file + server_name rather than insecure_skip_verify.
Verification that is switched off is not verification.

The key permissions. Grafana runs as 472:0 and cannot read a key owned by
the operator at 0600. The container gets the operator’s gid as a supplementary
group and leaf keys are now 0640; certificates/ is 0700, so nothing else
on the host can traverse to them whatever the file mode says. The alternatives
were a world-readable private key, or running Grafana as the operator’s uid and
stranding it from its own 472-owned data volume.

gen-certs.sh gains a repeatable --dns for exactly this case.

Blast radius

grafana and prometheus are recreated. Grafana’s URL changes to https://
— an existing bookmark on http:// gets a 400, and browsers will warn until
certificates/ca.pem is trusted (see the certificates runbook).

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml

Verification

Against the lab CA with no -k:

$ curl --cacert certificates/ca.pem https://grafana.matrix.elysium:3000/api/health
{ "database": "ok", "version": "13.0.2" }

served cert SANs: DNS:grafana.matrix.elysium, DNS:grafana, IP:10.0.99.20
grafana logs:     HTTP Server Listen protocol=https, MinTLSVersion=TLS1.2
plain http:       400

All ten scrape targets up, none down:

grafana  scrapeUrl: https://grafana:3000/metrics  health: up  lastError: (none)

Authenticated API over TLS works (so the now-secure cookie is fine), Prometheus
and Loki datasources both report OK, a query through the Grafana proxy returns
12 series, and all 5 dashboards are still provisioned.

./scripts/validate.sh passes; docker compose config and promtool check config clean.

• make validate passes
• Deployed to the lab and confirmed working
• Docs updated (deploy-stack.md, images/README.md, make up banner)

Noticed, unrelated

The Alertmanager datasource health endpoint returns plugin unavailable
(errorMessageID=plugin.unavailable). It points at internal http://alertmanager:9093
and is untouched by this change — the failure is a Grafana plugin issue, not a
connection one. Worth its own issue rather than being folded in here.