Hand over secrets to the estate path #663

Activity

1. ExtraToast

   added

   type: choreMaintenance work without intended behavior change.Maintenance work without intended behavior change.

   area: platformPlatform, deployment, cluster, or runtime concern.Platform, deployment, cluster, or runtime concern.

   priority: P1High; important and should be handled in the current iteration.High; important and should be handled in the current iteration.

   status: blockedCannot move forward until an external dependency is resolved.Cannot move forward until an external dependency is resolved.

   on Oct 27, 2026
2. ExtraToast

   added a parent issue on Oct 27, 2026

   • Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project #373
3. ExtraToast commented on Oct 27, 2026

   

   ExtraToast

   on Oct 27, 2026

   ContributorAuthor

   More actions

   Resume here

   • State: not started. The body holds the Project's estate-file items, and Rewrite chapter 60 to rules, and write the handover and setup runbooks #381 holds the runbook steps.
   • Next step: once Republish delivery, _canary and _estate on 1.0.0 #380 is done and Rewrite chapter 60 to rules, and write the handover and setup runbooks #381's runbook is in, fill in the estate's project file from the worked example (the other session's example tickets, where the body names them). Then diff the render against fleet-infra's hand-written objects.
   • Blocked by: Republish delivery, _canary and _estate on 1.0.0 #380, Rewrite chapter 60 to rules, and write the handover and setup runbooks #381, and the handover before this one in Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project #373's order.
   • Decisions: in this ticket's body and in Rewrite chapter 60 to rules, and write the handover and setup runbooks #381's resume note, dated, including 2026-10-27's: Vault moves by raft snapshot and restore, observability starts fresh, Grafana goes in observability, and Garage's volume splits at handover.
   • Gotchas: keep every live claim name the body lists. A live handover step is the maintainer's go-ahead, never an agent's.
4. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   #380 is closed: delivery, _canary and _estate are on the 1.0 contract (evidence in #380 (comment)).
5. ExtraToast commented on Oct 29, 2026

   

   ExtraToast

   on Oct 29, 2026

   ContributorAuthor

   More actions

   Prepared: secrets handover, every change a draft

   Nothing is merged, and Vault was only read (vault status, /v1/sys/health, the cluster). The window is the maintainer's. The runbook is JorisJonkers-dev/estate#262's docs/runbooks/secrets-handover.md.

   Merge order

   #	PR	when
   R1	JorisJonkers-dev/fleet-infra#570: retire the Vault CSI provider	any time before the window. Nothing uses it (no CSIDriver, no SecretProviderClass, no csi volume)
   R2	JorisJonkers-dev/fleet-infra#571: retire the Vault Agent injector and the sample reader (stacked on #570)	before the window, once Postgres and RabbitMQ (fleet-infra#538) and agents-api (fleet-infra#531) have left. Its test fails until then
   R3	JorisJonkers-dev/fleet-infra#572: retire Reloader	before the window, once the agents handover drops hermes, its one user. Its test fails until then
   R4	JorisJonkers-dev/fleet-infra#573: retire cert-manager	after the edge handover (#662) and decision D7
   S1	JorisJonkers-dev/fleet-infra#574: orphan Vault and VSO (marks, both HelmReleases suspended)	window step 1, after R1 and R2 are live
   S2	JorisJonkers-dev/fleet-infra#575: drop Vault, VSO and bootstrap-auth.sh; VaultSealed watches secrets-system (stacked on #574)	window step 2, after D5
   B	JorisJonkers-dev/estate#262: the flip (ledger, vault serving statement out, VSO in the secrets file)	window step 3, merged before the seal
   P	the estate release PR release-please opens for B	window step 6, after the old VSO is stopped and the old Vault snapshotted and sealed

   Vault is down from the seal (step 5) to the unseal of the restored one (step 8): about 35 minutes; the window is booked at two hours. The old Vault stays sealed and untouched for seven days as the rollback.

   Validation

   deploy-kit 1.2.0 (toolkit.yml): CI's validate, every release-matrix entry (secrets with its config and env) and a local compose (composed 20 artifacts, secrets among them) all accept. The restore was rehearsed in Docker under the rendered posture: 1.21.2 snapshot into 1.21.4, temporary init, restore -force seals it, the original key unseals it, one raft voter, node-id kept across a restart. It also found that a restore needs a writable /tmp, now declared.

   Open blockers

   • Every handover before this one, and fleet-infra#531 and fix: a required: false edge orders no Reconcile Unit #538 (injector consumers).
   • Decisions D1 to D7, in the runbook: D1 run Vault 1.21.4, not the 2.0.4 the old StatefulSet already names (OnDelete); D2 VSO cache not persisted (Persist the Vault Secrets Operator's client cache, encrypted with a transit key #735); D3 Vault metrics without a token; D4 Vault's backup is files on a live raft volume until Back Vault up with a raft snapshot: an engine whose method is consistent #734, and fleet-infra's raft snapshot job breaks at the flip; D5 Vault's own configuration (bootstrap-auth.sh) has no home after S2 (Declare the Secret Store's own configuration: auth methods, engines, connections and transit keys #736); D6 the Platform document has no priority, so Vault renders without platform-data; D7 the LAN tier's wildcard (valid to 2026-12-16) has no issuer once cert-manager goes.
   • VaultSealed must read secrets-system wherever the rule lives at the window (S2 carries it), and the estate's ops/unseal-vault.sh (estate#212) must reach the new Vault.
   • Auto-unseal stays deferred (Decide Vault auto-unseal estate#210): the key holder runs the window.

   Filed

   • Back Vault up with a raft snapshot: an engine whose method is consistent #734 back Vault up with a raft snapshot (an engine)
   • Persist the Vault Secrets Operator's client cache, encrypted with a transit key #735 persist VSO's client cache
   • Declare the Secret Store's own configuration: auth methods, engines, connections and transit keys #736 declare the Secret Store's own configuration
   • An edge into Postgres that derives no database, for a client that administers it #737 an edge into Postgres that derives no database
   • The secrets worked example renders Vault in dev mode, with no volume #738 the secrets worked example renders Vault in dev mode, with no volume
6. ExtraToast

   added a commit that references this issue on Nov 1, 2026

   feat(secrets): hand the secrets Project over to the estate path (#262)

   ...

   adc2d9b
7. ExtraToast commented on Nov 1, 2026

   

   ExtraToast

   on Nov 1, 2026

   ContributorAuthor

   More actions

   Secrets is on the estate path: Vault moved into secrets-system by raft snapshot and restore, with the Vault Secrets Operator beside it. The handover ledger is removed: no Project is left on the old path.

   • Changes: JorisJonkers-dev/fleet-infra#570, docs(research): mark the open verifications the ownership epic settled #571, Drop the rollback-backup annotation and the cluster state's backups list #574, fix(model): a datastore's catalog method carries its surface's consumer label, so the datastore admits it #575 (with the raft snapshot job re-pointed, decision D4), chore(main): release 0.27.0 #531 (agents retired first); feat: hand the secrets Project over to the estate path estate#262 (0.41.0, the ledger removed and VaultSealed re-pointed) and A rollback's backup: a one-shot backup in the pinned render, and its result in the cluster state #302; fix(deploy): reach Vault in secrets-system, where the secrets handover runs it auth-api#132.
   • Before the window the share was composed locally and dry-run against the cluster; the flip restarted no running workload outside secrets-system.
   • The snapshot's committed index was recorded before the seal; the restored Vault came up at or above it, with one raft peer.
   • Proofs: apps-secrets Ready; every policy job succeeds; every dynamic credential renewed and its Process reads its database; Vault is scraped and reports unsealed; people sign in by OIDC; a Secret VSO syncs gained a value written to Vault, and lost it on the rollback.

   Found in the window, each worked around and filed: a dependency cycle between the secrets and edge units (#780), VSO lacking deletecollection on Secrets (#781) and lease access for its leader election (#782), and kube-state-metrics labels not honoured, which had silenced VaultSealed (#783). Two hand-applied grants stand until #781 and #782 land.

   Left: the seven-day hold, then change C (the old Vault, VSO and their claims), and bootstrap/retire.sh for the old path's roles.
8. ExtraToast

   closed this as completedon Nov 1, 2026
9. ExtraToast commented on Nov 1, 2026

   

   ExtraToast

   on Nov 1, 2026

   ContributorAuthor

   More actions

   After the proofs: the old path's VaultStaticSecrets that log in through vso-system/default (Garage's keys, hermes-credentials, observability/postgres-backup) did not sync under the new VSO. A VaultAuth in the operator's own namespace is open to every namespace only for that operator; to the VSO in secrets-system it is an ordinary one. The VaultAuth was patched by hand to admit the three namespaces and name its own default connection, and all ten synced. The runbook's step 10 should carry this; it lasts until those leftovers move or retire.