Rewrite chapter 60 to rules, and write the handover and setup runbooks #381

Activity

1. ExtraToast

   added

   type: docsDocumentation-only change.Documentation-only change.

   area: deployhomelab-deploy, deploy-v2, manifests, and rollout flow.homelab-deploy, deploy-v2, manifests, and rollout flow.

   priority: P2Medium; normal planned work.Medium; normal planned work.

   status: blockedCannot move forward until an external dependency is resolved.Cannot move forward until an external dependency is resolved.

   on Oct 12, 2026
2. ExtraToast

   added a parent issue on Oct 12, 2026

   • Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project #373
3. ExtraToast

   marked this as blocked on Oct 12, 2026

   • Move the spec into the frame and concept chapters, with no change of meaning #374
4. ExtraToast

   self-assigned this

   on Oct 27, 2026
5. ExtraToast commented on Oct 27, 2026

   

   ExtraToast

   on Oct 27, 2026

   ContributorAuthor

   More actions

   Resume here, with the source material (2026-10-27)

   State: not started.

   Next step: after #374, rewrite chapter 60 as rules, and write two runbooks: the handover runbook (one Project off fleet-infra onto the estate path) and the setup runbook (a cluster from nothing).

   Blocked by: #374 (ordering: the chapter layout settles first). #656–#663 are blocked by this issue.

   The local HANDOVER.md this drew on is being reduced to a pointer, so the material is copied here.

   Handover order and shape

   • The order is mail, utility, agents, node, observability, data, edge, secrets (Hand over mail to the estate path #656 → Hand over secrets to the estate path #663). The foundation goes last.
   • Each handover:
     1. completes the estate's project file from the worked example;
     2. proves it with a one-off diff of the render against fleet-infra's hand-written objects, where every difference is explained;
     3. waits until the Project's Reconcile Units report Ready;
     4. checks one value a consumer can see;
     5. then, in one change, flips the handover ledger entry and deletes the hand-written objects, the descriptor and the drift-check entry.
   • Routes are rendered per Project only once no Project is legacy. While a datastore is legacy, its catalog stays in the handover-only estate-vso-secrets.

   Per-Project steps the runbook must cover

   • Vault (secrets): snapshot and restore, never a Move. Take a raft snapshot, start Vault in secrets-system, restore, unseal, then retire the old one. It is a short planned outage in a quiet window, with the maintainer. Today a frankfurt restart seals Vault and takes down every database app; auto-unseal is estate#210.
   • Observability: start fresh. New empty claims in observability-system. The old claims stay read-only until their retention would have run out, and are deleted on the maintainer's go-ahead at idle IO priority.
   • Garage (data):
     • copy each live key from secret/data/<ns>/garage (s3.key_hex, s3.secret) to secret/data/<project>/garage/s3;
     • split the live 20Gi garage-data claim: stop Garage, copy the LMDB metadata onto garage-meta, keep the blocks on garage-data, and point metadata_snapshots_dir at garage-snapshots;
     • back it up from its own metadata snapshot plus file-backup, never the live LMDB.
   • mongo (data): the claim must be mongo-data-mongo-0 to adopt the live PVC.
   • Edge: the proxies move from ingress-system and ingress-lan-system to edge-system. They are stateless, but each load-balancer address must pass from the old Service to the new one. cert-manager retires, because each acme tier issues its own. secret/data/platform/cloudflare#CF_DNS_API_TOKEN must exist first (maintainer).
   • Utility: adguard-ha runs two instances on LAN VIPs and is gated by failover drill Drill the failover gap of one pool address in front of two replicas #372. headlamp needs only api: { reason } and apiAccess.namespaceHolders.
   • Node: the estate's Platform document needs a nodeAgents catalog first; otherwise the render raises E_NO_NODE_AGENT_POLICY.

   Version rollout steps, from 1.0.0 onward

   • The order: deploy-kit release → the Publish Fragment pin → each Project's toolkit pin and its own release → delivery → the estate.
   • A Project drops a field a release removes in the same change that moves it to that release, never before. For example, alertClass is required on 0.31 and 0.32 and gone in 1.0.
   • Composition is specified to admit a fragment whose schemaVersion has the same major and a minor no higher than the toolkit's (chapter 40). Until Raise E_SCHEMA_VERSION_MISMATCH when composing a fragment #665 lands that gate is not enforced, so a fragment for a newer minor composes and its unknown fields are silently dropped. That is why the toolkit is upgraded before any Project.

   Settled decisions the chapter-60 rules should state (do not reopen)

   • A Project owns every object it delivers (ADR 0116). Only the Stable Addresses stay estate-wide.
   • Each Project runs its own Vault policy Job, logging in as policy-admin with ServiceAccount vault-policy in its namespace, with policies templated on the login's namespace.
   • Consumers are admitted by the label consumes.deploy-kit.jorisjonkers.dev/<application>.<surface>: "true", from namespaces labelled managed-by: deploy-kit (ADR 0035). Network policy is render-only.
   • Rollout events go through one Provider per Project: reporter-<project> beside rollouts-<project>.
   • A Rollback takes no backup of its own (backups and rollback-backup are removed in 1.0, feat(model)!: drop the rollback-backup annotation and the snapshot's backups list #652).
   • Release ordering inside a Project follows ADR 0111, and unit ordering follows required edges only.
   • One-shot Jobs are recreated, not renamed.
   • An unbound but unscheduled PVC counts as healthy.
   • Never pin a node's gateway or DNS: DHCP and AdGuard own them.
   • One WireGuard key per Process.
   • Flux is the only writer, so a manual Helm install, such as the proposed mesh stage 2, needs the maintainer's approval.

   Setup runbook prerequisites (maintainer)

   • The age key pair (age-keygen), with its public key as the Platform document's offCluster.recipient, which 1.0 requires.
   • The Contabo bucket credentials at secret/data/platform/backup/off-cluster.
   • CF_DNS_API_TOKEN.
   • Bulk deletes on live nodes run at idle IO priority, and only with approval.
6. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Decision (maintainer, 2026-10-29): split it. The handover and setup runbooks land now under docs/. Only chapter 60's rewrite to rules stays blocked on #374.
7. ExtraToast

   added

   status: in-progressActively being worked.Actively being worked.

   on Oct 29, 2026
8. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   The runbook half is in progress on docs/381-runbooks: the handover runbook and the setup runbook under docs/runbooks/. Chapter 60's rewrite to rules stays blocked on #374 and is not part of that change; status: blocked stays on for it.
9. ExtraToast

   added a commit that references this issue on Oct 29, 2026

   docs(runbooks): write the handover and setup runbooks (#702)

   ...

   26c12b8
10. ExtraToast

    removed

    status: in-progressActively being worked.Actively being worked.

    on Oct 29, 2026
11. ExtraToast commented on Oct 29, 2026

    

    ExtraToast

    on Oct 29, 2026

    ContributorAuthor

    More actions

    The runbook half landed in #702.

    • docs/runbooks/handover.md covers one Project's handover:
      • the order rule;
      • the Move rule for a mismatched namespace or database name;
      • the steps before the window: the estate file, the one-off diff, the old route's port, the injected-token check, Vault's keys;
      • chapter 60's three changes, with suspend-then-delete for every HelmRelease and the CRDs passing to the bootstrap set first;
      • what proves it done, the closing changes (descriptor, drift-check entries, leftovers) and the rollback;
      • notes for each of Hand over mail to the estate path #656 to Hand over secrets to the estate path #663, including the retirements before secrets and the three checks (GPU resource, injected token, the failover gap Drill the failover gap of one pool address in front of two replicas #372 measured).
    • docs/runbooks/setup.md stands the estate path up on a cluster:
      • who provides what, during and after the handover;
      • the once-only maintainer items;
      • node facts, Flux, Vault unsealed, secrets.sh, cluster.sh, the bootstrap CRDs and the first composition;
      • what proves it, and the standing rules.

    Still open on this issue:

    • Chapter 60's rewrite to rules, after Move the spec into the frame and concept chapters, with no change of meaning #374: the first acceptance criterion, and the ledger naming grimoire, tribelt, latex, knowledge-platform, estate-dashboard and node. The runbook boxes are met by docs(runbooks): write the handover and setup runbooks #702.
    • What the rewrite should settle:
      • This issue says "one change flips the ledger and deletes the hand-written objects". Chapter 60 deletes the manifests before the flip, and the units can only be Ready after it. The runbook follows chapter 60.
      • The chapter's providers-first adoption order and its "renames nothing" claim are stale against Order the Project handovers and what the spec must settle before each #349.
    • Undecided, marked open in the runbooks:
      • No command takes the handover diff, so the runbook describes a manual way.
      • The order in which the edge's addresses pass (Hand over edge to the estate path #662).
      • How the bootstrap CRDs are applied once the old path's charts are gone.
      • Generate the backup age key pair, and write the off-cluster backup credential estate#228 and Decide Vault auto-unseal estate#210.
    • The handover issues cite the estate's HANDOVER.md, which does not exist on estate main.

    status: in-progress is off again; status: blocked stays for the chapter half.