Hand over observability to the estate path #660

Activity

1. ExtraToast

   added

   type: choreMaintenance work without intended behavior change.Maintenance work without intended behavior change.

   area: platformPlatform, deployment, cluster, or runtime concern.Platform, deployment, cluster, or runtime concern.

   priority: P1High; important and should be handled in the current iteration.High; important and should be handled in the current iteration.

   status: blockedCannot move forward until an external dependency is resolved.Cannot move forward until an external dependency is resolved.

   on Oct 27, 2026
2. ExtraToast

   added a parent issue on Oct 27, 2026

   • Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project #373
3. ExtraToast commented on Oct 27, 2026

   

   ExtraToast

   on Oct 27, 2026

   ContributorAuthor

   More actions

   Resume here

   • State: not started. The body holds the Project's estate-file items, and Rewrite chapter 60 to rules, and write the handover and setup runbooks #381 holds the runbook steps.
   • Next step: once Republish delivery, _canary and _estate on 1.0.0 #380 is done and Rewrite chapter 60 to rules, and write the handover and setup runbooks #381's runbook is in, fill in the estate's project file from the worked example (the other session's example tickets, where the body names them). Then diff the render against fleet-infra's hand-written objects.
   • Blocked by: Republish delivery, _canary and _estate on 1.0.0 #380, Rewrite chapter 60 to rules, and write the handover and setup runbooks #381, and the handover before this one in Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project #373's order.
   • Decisions: in this ticket's body and in Rewrite chapter 60 to rules, and write the handover and setup runbooks #381's resume note, dated, including 2026-10-27's: Vault moves by raft snapshot and restore, observability starts fresh, Grafana goes in observability, and Garage's volume splits at handover.
   • Gotchas: keep every live claim name the body lists. A live handover step is the maintainer's go-ahead, never an agent's.
4. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   #653 has landed (#689): grafana is now in the observability example. The estate's observability file copies spec/v1/examples/observability/ as it stands on main.

   What changes for this handover:

   • grafana is an Application of observability, beside grafana-operator. It keeps its native Client against identity's provider, holding a secret. Its env file is observability/env/grafana/base.env. The Platform document names the monitoring Grafana observability/grafana.
   • Its credential path is secret/data/observability-system/grafana/client. The provider writes it there, and nothing writes the old identity-system path any more.
   • Observability now has a public exposure, so its Flux unit carries grafana's URL for rollout events and dependsOn: apps-edge (pins/rendered/projects/observability/source.yaml). It reconciles only after edge's unit.
5. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   #655 has landed (#699): the metrics stack scrapes Flux, and it can now reach its scrape targets.

   • The estate's Platform document should record bootstrap.flux.metrics: {port: 8080, path: /metrics}. observability then renders the flux-controllers PodMonitor in its own namespace, replacing fleet-infra's observability/flux-monitoring.
   • Prometheus's policy now admits egress to every declared scrape port and to Flux's port 8080. Before feat(model): scrape Flux where the bootstrap set records its metrics, and derive the metrics stack's egress #699 the rendered Prometheus could reach only DNS, so an observability handover before it would have scraped nothing.
   • Check at the handover: the Flux series' job becomes each controller's name (for example kustomize-controller) instead of flux-system/flux-controllers. Any live alert that filters on that job needs adjusting.
6. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   #380 is closed: delivery, _canary and _estate are on the 1.0 contract (evidence in #380 (comment)).
7. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Handover prepared: drafts only, nothing merged. The render-against-live diff is in JorisJonkers-dev/estate#260 and its runbook, docs/runbooks/observability-handover.md.

   Merge order

   1. JorisJonkers-dev/fleet-infra#563: prune guards. Before the window, and safe now.
      • The namespace observability, gatus-data, observability-backups and the grafana, gatus and faro routes get the prune mark.
      • loki's StatefulSet stops deleting its claim, and grafana's claim gets helm.sh/resource-policy: keep.
      • After it lands, by hand: the eight PVs to Retain.
   2. feat(bootstrap): pin the monitoring stack's CRDs in the bootstrap set estate#259: the monitoring stack's CRDs join the bootstrap set (monitoring.coreos.com/v1alpha1, grafana.integreatly.org/v1beta1) and cluster.sh applies them. Before the window.
   3. JorisJonkers-dev/fleet-infra#564: orphan. Every object is marked, the seven HelmReleases are suspended (not dcgm-exporter), and the three Kustomizations get prune: false. Window step 1.
   4. JorisJonkers-dev/fleet-infra#566: drop the manifests and the three Kustomizations. The backup CronJobs, observability-backups and dcgm-exporter stay under apps-observability-namespace, and the routes move unchanged to apps-edge. Window step 2.
   5. feat: hand the observability Project over to the estate path estate#260, then its release: the flip. Window step 3.
   6. JorisJonkers-dev/fleet-infra#568: the routes and both Traefiks' traces move to observability-system, and the old VSO copies go. After the flip, once Ready.

   Blockers

   • Render the Prometheus and Alertmanager the Prometheus operator runs #722: the render emits no Prometheus or Alertmanager for the operator. The estate's Prometheus would scrape only itself, evaluate no rule and notify nothing. Blocks.
   • Render the Grafana instance the Grafana operator imports dashboards into, and derive its data sources #723: no Grafana instance is rendered for the operator, and no data sources are derived. The estate's Grafana gets no dashboards. Blocks.
   • Scrape the kubelet, cAdvisor, the API server and CoreDNS where the bootstrap set records them #724: nothing scrapes the kubelet, cAdvisor, the API server or CoreDNS. PodOOMKilled, PersistentVolumeFillingUp and KubeletDown go quiet. Degrades.
   • The worked example's Gatus cannot gather any external checks #725: the worked example's Gatus cannot gather checks: no image, no shared directory. Worked around in the estate file.
   • A blue-green Process's Services carry only one of its surfaces #728: a blue-green Process's Services carry one port. otel-collector and alloy are interrupted in the estate file to get around it.
   • Hand over node to the estate path #659 first: node-exporter is a DaemonSet of metrics-stack, so feat(model): a Vault policy job per Project, in its own unit #564 orphans node's agent too.

   Until #722 and #723 are released, the runbook describes a bridge: the old Prometheus, Alertmanager and Grafana keep running under the estate path's operators, and the old operators and the old AlertmanagerConfig are deleted by hand. Without the bridge, the flip is a total alert gap.

   Decisions for the maintainer

   • History, per store. Nothing is carried, per 2026-10-27's decision, and no Move is derived.
     • Prometheus: 15d, readable through the old Prometheus while it runs.
     • Tempo: 7d.
     • Loki: no retention configured, so a deletion date has to be chosen.
     • Pyroscope: only self-profiles.
     • Grafana DB: export anything made in the UI first.
     • Alertmanager silences: re-create them.
     • Gatus: 191 days of uptime history; choose a date.
     • observability-backups is not observability's, and stays untouched.
   • Whether to take the bridge, or wait for Render the Prometheus and Alertmanager the Prometheus operator runs #722 and Render the Grafana instance the Grafana operator imports dashboards into, and derive its data sources #723.
   • Accepting the losses:
     • Gatus's in-cluster checks of datastores, Vault and Stalwart go: each Project renders only its routes' checks.
     • No Alertmanager inhibition, so a critical and its warning both notify.
   • When feat(composition): isolate the provider, not the run, when it drops a Surface a consumer still names #568's Traefik half lands: it restarts both proxies.
8. ExtraToast

   added 2 commits that reference this issue on Oct 30, 2026

   feat(bootstrap): pin the monitoring stack's CRDs in the bootstrap set (…

   ...

   dc23656

   feat: hand the observability Project over to the estate path (#260)

   ...

   dd83e6a
9. ExtraToast commented on Nov 1, 2026

   

   ExtraToast

   on Nov 1, 2026

   ContributorAuthor

   More actions

   Observability is on the estate path (estate 0.36.4) and the old stack is deleted.

   • Fixes found during the window: fix(observability): give pyroscope a writable /tmp and gatus's checks sidecar the memory it needs estate#283, fix(observability): give pyroscope time to start, move the checks sidecar off 8080, let the operator list Probes estate#285, fix(observability): let the operator read ScrapeConfigs, so the kubelet is scraped estate#288 (pyroscope's writable /tmp and startup budget, the checks sidecar's memory and health port, and the operator's Probe and ScrapeConfig permissions).
   • O4, JorisJonkers-dev/fleet-infra#568, merged; the routes and both Traefiks are on the new stack.
   • Proofs: Flux Ready; alerts reach Discord through the new Alertmanager; Prometheus scrapes kubelet, apiserver and Flux; Grafana signs in through auth-api with its dashboards and history; Loki and Tempo receive auth-api's logs and traces; the status page groups checks by Project.
   • Left by design: the backup jobs and their secrets in observability, the node-metrics and gpu-metrics monitors (until Scrape the metrics Node Agents, and let node-metrics read the hosts' textfile directory #731), the old claims (each when the runbook's table says) and the old Vault roles.

   Follow-ups, outside this handover: release-gate's scrape answers 403; auth-api's error-budget rule counts authorization denials as errors; kube-state-metrics' namespace label is overwritten by the scrape's own.
10. ExtraToast

    closed this as completedon Nov 1, 2026