Hand over node to the estate path #659

Activity

1. ExtraToast

   added

   type: choreMaintenance work without intended behavior change.Maintenance work without intended behavior change.

   area: platformPlatform, deployment, cluster, or runtime concern.Platform, deployment, cluster, or runtime concern.

   priority: P1High; important and should be handled in the current iteration.High; important and should be handled in the current iteration.

   status: blockedCannot move forward until an external dependency is resolved.Cannot move forward until an external dependency is resolved.

   on Oct 27, 2026
2. ExtraToast

   added a parent issue on Oct 27, 2026

   • Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project #373
3. ExtraToast commented on Oct 27, 2026

   

   ExtraToast

   on Oct 27, 2026

   ContributorAuthor

   More actions

   Resume here

   • State: not started. The body holds the Project's estate-file items, and Rewrite chapter 60 to rules, and write the handover and setup runbooks #381 holds the runbook steps.
   • Next step: once Republish delivery, _canary and _estate on 1.0.0 #380 is done and Rewrite chapter 60 to rules, and write the handover and setup runbooks #381's runbook is in, fill in the estate's project file from the worked example (the other session's example tickets, where the body names them). Then diff the render against fleet-infra's hand-written objects.
   • Blocked by: Republish delivery, _canary and _estate on 1.0.0 #380, Rewrite chapter 60 to rules, and write the handover and setup runbooks #381, and the handover before this one in Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project #373's order.
   • Decisions: in this ticket's body and in Rewrite chapter 60 to rules, and write the handover and setup runbooks #381's resume note, dated, including 2026-10-27's: Vault moves by raft snapshot and restore, observability starts fresh, Grafana goes in observability, and Garage's volume splits at handover.
   • Gotchas: keep every live claim name the body lists. A live handover step is the maintainer's go-ahead, never an agent's.
4. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   #380 is closed: delivery, _canary and _estate are on the 1.0 contract (evidence in #380 (comment)).
5. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Prepared, 2026-10-29: four draft PRs, two blockers in deploy-kit

   The window is in JorisJonkers-dev/estate docs/runbooks/node-handover.md (on #261's branch).

   The PRs, in merge order

   1. JorisJonkers-dev/fleet-infra#565, any time before the window. traefik-lan stops naming the pool enschede-lan and asks for its address alone. Otherwise the new controller clears the LAN ingress address, because the render's pool is named lan-edge.
   2. JorisJonkers-dev/fleet-infra#567, the orphan, before the window and after utility's. It is stacked on utility's chain (chore(main): release 0.29.0 #550 to docs: map where one Project's change reaches another, and which operators could carry each job #554), since it marks lan-dns and apps-metallb-lan-dns.
      • It marks 12 objects prune: disabled.
      • It suspends the metallb, nvidia-device-plugin and dcgm-exporter HelmReleases.
      • It sets prune: false on both MetalLB pool Kustomizations.
      • It puts helm.sh/resource-policy: keep on node-exporter's four kube-prometheus-stack objects.
   3. JorisJonkers-dev/fleet-infra#569, the removal, window step 2.
      • The manifests go, with both pool Kustomizations.
      • nodeExporter.enabled: false.
      • The nvidia-cdi RuntimeClass stays, moved under the same name.
      • Two PodMonitors scrape node-system as job="node-exporter" and job="dcgm-exporter".
   4. feat: hand the node Project over to the estate path estate#261, the flip, window step 6, then its release.
      • node moves to estate.
      • metallb.io/v1beta1 joins bootstrap.crds, and bootstrap/cluster.sh gains a stage that adopts MetalLB v0.15.3's CRDs with bgppeers' conversion webhook dropped.
      • nodeAgents memory is sized to live use: dcgm-exporter 1Gi, since it uses about 410Mi and was OOM-killed at 256Mi. The speaker and controller get 128Mi.
      • Every lock digest matches the live pods' imageID.

   Between 3 and 6, by hand:

   • stop the old controller and delete metallb-webhook-configuration, which fails closed and would refuse the render's pools;
   • run cluster.sh stage 5;
   • delete the old device plugin, so two plugins never share nvidia-gpu.sock.

   After 6, by hand:

   • delete metallb-speaker; the new speakers wait on host port 7472, so each node's gap is seconds;
   • then node-exporter (port 9100);
   • then dcgm-exporter, once gpu-metrics is Ready.

   Order against utility

   node goes after utility's window, and #261 is updated from main after JorisJonkers-dev/estate#248 merges, so the render holds lan-dns. Utility also has to have deleted adguard-ha-b and moved the router to .98 alone: the render has no pool for .97.

   The pool move needs no overlap trick. Each controller reads only its own namespace's pools, and the old webhook is deleted before the flip. The new controller finds .98 and .99 already assigned, inside its own lan-dns and lan-edge, and keeps both. Only the speakers change hands, node by node.

   Open blockers

   • The rendered l4-speaker and l4-controller cannot start on MetalLB 0.15.3 #729 (bug), blocking. The rendered l4-speaker and l4-controller do not start on MetalLB 0.15.3. I reproduced it on a throwaway local k3s, from the estate's own render. The speaker has no METALLB_POD_NAME and exits. Neither ClusterRole has the reads 0.15.3 always watches (configurationstates, secrets, configmaps, and the controller's nodes). With those added by hand, both ran. Needs a deploy-kit release and toolkit.yml moved to it.
   • The gpu-device Node Agent asks for GPU ids the hosts' CDI specification does not name #730, blocking, a maintainer decision. The rendered device plugin uses the default uuid GPU ids. The hosts' NixOS CDI specification names index, which is why live sets deviceIDStrategy: index. Either the render says index, or nix-config switches the specification to uuid.
   • The l4-speaker runs without memberlist, so a lost node's addresses fail over later than #372 measured #733, a decision. The rendered speaker runs without memberlist. A lost node's addresses then fail over on the API server's notice (about 40s), not Drill the failover gap of one pool address in front of two replicas #372's 6.6s.
   • Scrape the metrics Node Agents, and let node-metrics read the hosts' textfile directory #731, bridged by feat(model): a provider admits its consumers by a label each carries, so a new consumer re-renders only itself #569's PodMonitors. Nothing on the estate path scrapes the metrics agents. node-exporter's textfile collector is gone, so NvidiaCdiSpecMissing, NvidiaCdiStateStale and the module-mismatch alert go silent until it lands.
   • Account for the nvidia-cdi RuntimeClass: neither rendered nor in the bootstrap set #732, bridged by keeping it in apps-core. The nvidia-cdi RuntimeClass is neither rendered nor in the bootstrap set.
   • Hand over agents to the estate path #658 (agents), still before this in Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project #373's order.

   Render against live

   Every difference is explained in #261's body and the runbook. In short:

   • every agent is a new DaemonSet in node-system;
   • the speaker runs on the five Enschede nodes, not frankfurt-contabo-1, in native L2 mode with no excludel2 list;
   • pools are lan-edge and lan-dns, both autoAssign: false;
   • the device plugin has no config-manager sidecar or MPS;
   • dcgm-exporter uses the image's counters, a superset of the chart's;
   • node-exporter has no textfile collector or probes.

   Validated with deploy-kit 1.2.0, as toolkit.yml names it: CI's validate, every release-matrix entry, a negative E_NO_NODE_AGENT_POLICY and a local deploy-kit compose. Nothing is merged.
6. ExtraToast

   added a commit that references this issue on Oct 30, 2026

   feat: hand the node Project over to the estate path (#261)

   ...

   517a1a5
7. ExtraToast commented on Oct 30, 2026

   

   ExtraToast

   on Oct 30, 2026

   ContributorAuthor

   More actions

   node is on the estate path, delivered by estate 0.35.0 on deploy-kit 1.5.0.

   • fleet-infra feat(model): render the rollout events per Project, in its own unit #565, feat(delivery): apply a Project's routes from its own unit, after the edge Project #567 and feat(model): a provider admits its consumers by a label each carries, so a new consumer re-renders only itself #569 and estate#261 are merged and live; bootstrap/cluster.sh adopted MetalLB's CRDs.
   • apps-node is Ready. The new speakers joined one memberlist and announce both LAN addresses; DNS and LAN ingress answered from the LAN throughout the swap.
   • Both GPU nodes advertise their cards through gpu-device; jellyfin and immich kept running.
   • Prometheus scrapes node-metrics on all six nodes and gpu-metrics on both GPU nodes, under the old job names.
   • The old path's objects are deleted: the speakers, controller, webhook, device plugin, dcgm-exporter, node-exporter, their releases, namespaces and RBAC.

   Found in the window:

   • The new speakers started beside the old ones instead of waiting for a host port, so for about an hour both sets announced the same addresses. Nothing broke, but the runbook's assumption that they wait was wrong.
   • The speakers' memberlist runs over the nodes' Tailscale addresses, and the cordoned Pis keep suspecting each other failed. Election stays on the GPU nodes, but failover is less reliable than the local test showed.
   • On the T1000, the new gpu-metrics could not start until the old dcgm-exporter released DCGM's profiling module, so the runbook's step 10 order deadlocks there; deleting the old exporter first is the way out.
8. ExtraToast

   removed

   status: blockedCannot move forward until an external dependency is resolved.Cannot move forward until an external dependency is resolved.

   on Oct 30, 2026
9. ExtraToast

   closed this as completedon Oct 30, 2026