Add the s3 engine and its bucket catalog #387

Activity

1. ExtraToast

   added

   type: featureNew user-facing or operator-facing capability.New user-facing or operator-facing capability.

   area: deployhomelab-deploy, deploy-v2, manifests, and rollout flow.homelab-deploy, deploy-v2, manifests, and rollout flow.

   priority: P2Medium; normal planned work.Medium; normal planned work.

   on Oct 12, 2026
2. ExtraToast

   added a parent issue on Oct 12, 2026

   • Rewrite the spec and the compiler in place, release 1.0.0, and hand over every Project #373
3. ExtraToast

   added

   status: blockedCannot move forward until an external dependency is resolved.Cannot move forward until an external dependency is resolved.

   on Oct 12, 2026
4. ExtraToast

   marked this as blocked on Oct 12, 2026

   • Cut what the map cut and release package 1.0.0 #379
5. ExtraToast

   marked this as blocking on Oct 23, 2026

   • Hand mongo and garage over to the estate path estate#207
6. ExtraToast

   added

   status: in-progressActively being worked.Actively being worked.

   and removed

   status: blockedCannot move forward until an external dependency is resolved.Cannot move forward until an external dependency is resolved.

   on Oct 26, 2026
7. ExtraToast commented on Oct 26, 2026

   

   ExtraToast

   on Oct 26, 2026

   ContributorAuthor

   More actions

   Design draft (2026-10-26), for the maintainer to decide

   #351 item 17 settles the shape: s3 joins the engine vocabulary with a backup image, and an edge to an s3 Process derives a bucket catalog, one bucket and one scoped key per consuming Project, mirroring the database catalog. The live Garage disagrees with three parts of that, so three choices come first.

   What is live (2026-10-26). dxflrs/garage:v2.3.0 in data-system, db_engine = "lmdb", a 20Gi claim, and 12 buckets: outline-uploads, assets.jorisjonkers.dev (a public website bucket), knowledge-archive, grimoire-assets, five overleaf-*, karakeep-assets, karakeep-litestream and handover-dumps. Keys are pre-generated by bootstrap/secrets.sh into Vault per namespace (for example secret/data/karakeep-system/garage, keys s3.key_hex and s3.secret). A hand-written bootstrap Job imports them and creates the buckets through the admin API.

   1. How many buckets a Project gets

   option	what changes
   A. One bucket per Project, <project> (item 17 as written)	overleaf's five and karakeep's two buckets collapse into one each, with prefixes. That is an app configuration change in each, plus an object copy at handover.
   B. The edge names its buckets; each is <project>-<name>	buckets: [assets, litestream] on karakeep's edge derives karakeep-assets and karakeep-litestream, exactly the live names. One key per Project, scoped to its buckets.

   Recommendation: B. It keeps every live bucket name, as the database catalog keeps <project>_db. One key per Project still holds. assets.jorisjonkers.dev is the exception, a public website bucket whose name is its hostname; it stays the platform's own and is not a consumer's catalog entry.

   2. Where a key comes from

   option	what changes
   A. The method mints the key	The method creates the key in Garage through the admin API and writes it to the consumer's derived Vault path, secret/data/<project>/<s3-process>/s3 with keys access-key-id and secret-access-key. Where the document already exists, it imports it instead, so a rebuilt Garage gets the same key back. The method's policy gains create and update on exactly those paths.
   B. The operator pre-generates the key	As today: the operator writes each key to Vault, and the method only imports it and grants the buckets. Every new consumer is a manual step first.

   Recommendation: A. No manual step per consumer. A run applied twice changes nothing. The live keys are adopted once by importing them from their current paths at handover, which is a Move-style rename the runbook carries.

   Consumers read the key like any static secret: a VaultStaticSecret delivered as env or file, through the one secret delivery. The endpoint and region are engine facts the render hands over beside it.

   3. The engine's backup

   option	what changes
   A. The files method over a Garage metadata snapshot	Garage writes its own consistent metadata snapshot (metadata_auto_snapshot_interval). file-backup archives the data blocks, which are immutable, plus the snapshot directory, never the live LMDB. The backup claim must hold retain archives of up to 20Gi.
   B. Switch Garage to db_engine = "sqlite"	file-backup already copies a SQLite database consistently (JorisJonkers-dev/delivery#90). It is a one-off conversion with garage convert-db, plus Garage's own warning that SQLite is slower.
   C. No backup image yet, recorded	An s3 Process may only be reconstructible until one exists. Its buckets hold uploads and Litestream replicas that are not reconstructible, so this would be a recorded gap, not a solution.

   Recommendation: A, with the off-cluster copy (#470) covering an irreplaceable class. The open cost is space: a full archive per copy, so a large Garage needs a small retain or a later incremental method.

   Everything else follows the database catalog

   • The catalog is a ConfigMap, applied by the engine's catalog method in the datastore's namespace, under the <process>-catalog identity, with a Job and an hourly CronJob.
   • It names only consumers the ledger puts on estate. It never deletes a bucket, a key or a grant.
   • A new method image, garage-catalog, in JorisJonkers-dev/delivery, speaks the Garage admin API on a second surface the Platform entry names (admin, 3903). It signs in with the admin token at the derived secret/data/<project>/<process>/catalog (token).
   • grimoire is the worked example.
8. ExtraToast commented on Oct 26, 2026

   

   ExtraToast

   on Oct 26, 2026

   ContributorAuthor

   More actions

   Decided (2026-10-26, the maintainer): all three recommendations.

   1. The edge names its buckets, each derived as <project>-<name>, with one key per Project scoped to them. This refines Decide how the model meets the live needs no chapter covers #351 item 17's "one bucket" and keeps every live bucket name.
   2. The catalog method mints the key. It creates it in Garage and writes it to the consumer's derived Vault path, or imports it from there where it exists.
   3. Garage's backup is its own metadata snapshot (metadata_auto_snapshot_interval) plus file-backup of the data blocks and the snapshot, never the live LMDB.
9. github-actions

   removed

   status: in-progressActively being worked.Actively being worked.

   on Oct 26, 2026