Repository navigation
Hand over data to the estate path #661
Description
Parent
Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project
What to build
The data Project leaves fleet-infra and is delivered from the estate repository, in the order #381 sets.
The estate's data file declares postgres, rabbitmq and valkey. Add platform-mongo and platform-garage. Everything stays in data-system. The live claims are kept: postgres-data, rabbitmq-data, valkey-data, and mongo-data-mongo-0, which mongo's declaration must name exactly. Garage's volumes split at handover (decided 2026-10-27): in the window, Garage stops, its metadata is copied onto the new garage-meta volume, garage-data keeps the data blocks, and Garage starts with metadata_snapshots_dir on garage-snapshots. The rest of Garage's handover (its releases, engines.s3, the live keys, bucket names, garage-asset-sync) is in the comment below. The platform-postgres serving statement leaves the handover ledger.
The estate-side changes are recorded in the estate repository's HANDOVER.md.
Acceptance criteria
- The render's diff against the Project's hand-written objects in fleet-infra is clean, or each difference is explained.
- The Project's Reconcile Units report Ready.
- A consumer-visible value checks out: Every consumer still reads its database, its queue and its bucket.
- One change flips the handover ledger entry and deletes the hand-written objects, the descriptor and the drift-check entry.
Activity
ExtraToast commentedon Oct 27, 2026
Missing from this issue, from the #387 work (#647, JorisJonkers-dev/delivery#94):
garage-catalogand locks it. The Platform document then gainsengines.s3: {backup: file-backup, catalog: garage-catalog, catalogSurface: admin}and the servingplatform-garageadmin{path: secret/data/platform/garage, token: admin-token}, as the worked platform has them.secret/data/<ns>/garage(s3.key_hex,s3.secret) tosecret/data/<project>/garage/s3(access-key-id,secret-access-key) before the catalog first runs. The catalog then imports them instead of minting new ones, so every consumer keeps its key.garage-dataclaim is kept, but the worked example declares three:garage-meta(reconstructible),garage-snapshotsandgarage-data(recoverable). So either split the live 20Gi claim at handover (metadata ontogarage-meta, withmetadata_snapshots_dirongarage-snapshots), or declare onerecoverablegarage-datauntil a quiet window. This is the maintainer's choice; it is open.buckets: [<name>], giving<project>-<name>). Those must match the live bucket names, or the catalog creates new, empty buckets.garage-asset-syncCronJob also runs indata-system. Before the hand-written objects are deleted, check what it does and whether a model concept covers it.ExtraToast commentedon Oct 27, 2026
Decided by the maintainer, 2026-10-27: split at handover. In the handover window, stop Garage and copy its metadata (the LMDB directory) onto the new
garage-metavolume. The livegarage-dataclaim keeps the data blocks. Start Garage withmetadata_snapshots_dirongarage-snapshots. This is one short Garage outage, and the estate matches the model from the first day.ExtraToast commentedon Oct 27, 2026
Resume here
Garage specifics are in the two comments above (keys, buckets, the split, and
garage-asset-sync). The releases it needs are #379 and JorisJonkers-dev/delivery#98, which locksgarage-catalog.ExtraToast commentedon Oct 29, 2026
#380 is closed: delivery,
_canaryand_estateare on the 1.0 contract (evidence in #380 (comment)).feat(data): declare Garage and run the data Project as the old path d…
feat: hand the data Project over to the estate path, keeping _estate'…
ExtraToast commentedon Oct 29, 2026
Data handover PRs, refreshed 2026-10-29
The other session's stale PRs are taken over: each has
mainmerged in (no force-push), and the work moved onto deploy-kit 1.3.0, the versiontoolkit.ymlpins (also checked on 1.2.0). The runbook isdocs/runbooks/data-handover.mdon JorisJonkers-dev/estate#215. Nothing is merged, and nothing live was written.Merge order
data-systemand the claimspostgres-data,rabbitmq-data,valkey-data,garage-datagetprune: disabled; nothing restartskubectl patch pv … Retain, commands in #562data-systemvolumes that reclaimDelete(Postgres, RabbitMQ, Valkey, Garage, MongoDB, and both of Vault's) reclaimRetainmainapps-datastops pruning; nothing restartsapps-databecomesapps-data-leftovers, and nothing is prunedgarage-metarabbitmq-definitions-backupretiresWhat changed from the stale versions
garage-meta,garage-snapshots, and the livegarage-data, whose blocks stay underdata/.engines.s3and Garage'sservingadmin location are in A.garage-asset-syncand the assets route stay inapps-data-leftovers.garage-asset-syncpublishes agent-kit's latest release to theassets.jorisjonkers.devbucket every 15 minutes. A scheduled Process covers it, but its logic is a shell script, never an Asset, so it waits for an image of its own.config_fileargument, and its exporter back as a sidecar mounting its queries (deploy-kit 1.2).HOME=/tmp. kubelet appliesfsGroupto theselocalvolumes at each mount, so every file on them becomes group-readable (read onimmich-postgres). In Docker, the live RabbitMQ image then exits:Cookie file … must be accessible by owner only. WithHOMEon a writable path, it starts.projects/_estate/source.yamlby hand. Composition rewrites it, droppingestate-vso-secrets.Render against live, in short
The full table is in #216 and the runbook. All four Deployments are recreated, because the fixed label set changes their immutable selectors. Same claims, Service names, ports and image digests. The Assets are byte-identical.
restricted; Garage moves from root to 1000:1000, and the split chowns its claims.frankfurt-contabo-1, 16.7Gi of 23.5Gi.priorityClassName: platform-datagoes: the Platform document has nopriorityblock yet.pg_isready.apps-dataand signs in withsecret/data/data/postgres/catalog.Open blockers
secret/data/data/postgres/catalog(user,password),data/postgres/backup(PGUSER,PGPASSWORD) anddata/rabbitmq/backup(RABBITMQ_USERNAME,RABBITMQ_PASSWORD). None exists.platform/backup/off-clusteris unverified. A person writes them.data-systemwith no mark. Server-side apply by the samekustomize-controllermanager will drop the mark D0 sets. After the flip, one prune of the namespace deletes Vault and MongoDB with it. This needs a deploy-kit change, or the maintainer's acceptance with every volume onRetain(P0). The rollback is ordered around it.mongo-backupimage, so the declaration is refused withE_NO_ENGINE_POLICY. MongoDB stays inapps-data-leftovers. Its own handover later also needs the replica set's member host moved offmongo-0.mongo.data-system.svc.cluster.local, a StatefulSet name the render does not make. The maintainer decides whether data hands over without it.priorityto the Platform document gives the datastoresplatform-databack, but also movesauthand the foundation. It is a separate decision.Validation
accepted (148 read)on 1.3.0 and 1.2.0.deploy-kit compose: A againstmainis byte-identical; A with B gives 20 artifacts, data's new.feat(data): declare Garage and the datastores as the estate path will…
feat(data): hand the data Project over to the estate path (#216)
ExtraToast commentedon Nov 1, 2026
Data is on the estate path: Postgres, RabbitMQ, Valkey and Garage, adopted in place on their own claims. MongoDB stays on the old path and follows under #748.
Left: two old Vault roles, and the old logical Postgres backup until the first derived backup has been read back.
Found: the datastores' derived priority and memory requests preempted other pods on their node; one (Karakeep's browser) has no room until capacity is added.