[dan-petty]

The roadmap Service keeps three records on its volume: the request budget, the halt and the failure ledger. When a write fails, keep_state_record writes one error log line and the Service goes on with memory only (#2474). Nothing else shows the fault. A restart then loses what the record held: the budget window, a halt after a refused identity, or the suspended jobs (#4048). The log line stays. This change adds a gauge and an alert, so a person sees the fault without a log search.

Closes #4048

Change: openspec/changes/archive/2026-11-08-4048-state-unkept-gauge/
Elapsed: 5.36s

[dan-petty]

Test record for #4048.

fail-to-pass (devops ci fail-to-pass --base origin/main): 12 tests failed on base, 0 errored. The new file tests/server/test_service_state_unkept.py (10 tests) and the alert tests fail on base because the gauge and the alert are absent. They pass on this branch. Log: .data/agent/scratch/4048-f2p.txt.

Full devops ci (run in the verify stage, before a rebase and a dashboard fix): all rows passed, 21161 tests passed, 2 skipped. One failure was caused by this change: tests/grafana/test_grafana_dashboards.py needs a dashboard panel for each CONST_SERVICE_METRIC_ name. It is fixed at the cause with the panel "Service Unkept State Records". The test row took 12m 37s, over 5 minutes (known #3593; the promtool alert test takes 134 s). After the fix and the rebase onto origin/main, these narrow runs pass: test_service_state_unkept (10), dashboards (69), alerting with promtool (59). devops docs generate --check is clean. The full CI was not run again after the rebase.

Rollout rule. This change adds one gauge (devops_cli_service_state_unkept, label record), one alert (RoadmapServiceStateUnkept) and one dashboard panel. It removes, renames or tightens no setting. k8s/devops/configmap.example.yaml and the live ConfigMap stay as they are. tests/config/test_k8s_devops_runtime.py already loads the example ConfigMap into Settings, so no new test is needed. Nothing was applied to the cluster. The Argo CD sync of k8s/monitoring/prometheus-values.yaml after the merge adds the alert.

Note. The gauge is in process memory. A restart clears it, and the next failed write sets it again.