Repository navigation
feat: file-backup copies SQLite consistently, and copies off-cluster for irreplaceable volumes #470
Description
Problem
recoverable and irreplaceable derive a backup from the Process's engine, and a volume without an engine is E_DURABILITY_WITHOUT_ENGINE. Many third-party apps keep their state as files or SQLite on a volume, with no engine to name: jellyfin (users, watch history), sonarr, radarr, bazarr, prowlarr, qbittorrent, and karakeep's data directory. Each gets one of two wrong answers:
reconstructible, which loses real data;- or it stays on the old path.
What to build
A platform backup method for file-shaped state. Either it is an engine keyed for files, or a volume without an engine falls back to it.
- It produces a consistent copy: SQLite is copied through its online backup API, or the Process is quiesced.
- It keeps the class's
retaincopies, and makes the off-cluster copy forirreplaceable.
Acceptance criteria
- An ADR records the decision, and
spec/v1chapters 10 and 14 carry it, with a worked example and refusals. - Both implementations land it against the same oracle files.
- A restore from the method's copy is rehearsed on one real volume.
Blocks
Moving jellyfin, arr, downloads and karakeep onto the estate path.
Activity
ExtraToast commentedon Oct 17, 2026
Most of this already exists, and was missed when this was filed: chapter 10's
enginevocabulary hasfiles. The Platform document maps it to thefile-backupmethod, which JorisJonkers-dev/delivery builds and the estate locks (ghcr.io/jorisjonkers-dev/delivery/file-backup). So arecoverablevolume of SQLite or plain files isengine: filestoday, and jellyfin's and the arr apps' config volumes need nothing new.What's still open:
file-backupreads the volume as files, so a copy taken mid-write can be torn. Either the method copies SQLite through its online backup API, or a Process declares it can be quiesced.irreplaceablerequires. JorisJonkers-dev/delivery's README says none exists yet for any method.karakeep moved without this, since its state streams to Garage through Litestream (JorisJonkers-dev/estate#134).
[-]feat: a backup method for file-shaped state with no engine[/-][+]feat: file-backup copies SQLite consistently, and copies off-cluster for irreplaceable volumes[/+]ExtraToast commentedon Oct 26, 2026
Design draft (2026-10-26), for the maintainer to decide
What stays open, per the comment above: a consistent copy of SQLite under
engine: files, and the off-cluster copyirreplaceableasks for. Each part below gives the options and a recommendation.1. A consistent SQLite copy
file-backupmounts the volume read-only at/data(chapter 30's backup row), on the volume's own node. That rules out the obvious fix. SQLite's online backup API, andVACUUM INTO, must write the database's-shmfile even to read a WAL-mode database, so neither works on a read-only mount. Jellyfin, sonarr, radarr, bazarr and prowlarr all run SQLite in WAL mode.engine: files, the render mounts the volume read-write, and the method finds each SQLite file by its header (SQLite format 3\0) and copies it through the online backup API. Every other file is copied as today.modernc.org/sqlite) keeps the image distroless.db,-waland-shmtogether, then runsPRAGMA integrity_checkon the copy and retries a few times on failure./config/Backups/*.zip; Jellyfin 10.11 has its own backup), and the method copies that alone.Recommendation: B now, A later if a restore ever shows lost transactions. B keeps the read-only guarantee, needs no model change, and the integrity check makes a bad copy fail loudly instead of being kept. These apps lose at most the minutes before the copy, which is acceptable for config and watch history.
2. The off-cluster copy
The model already has it.
durability.irreplaceable.offClusternames the destination (s3://jorisjonkers-dev-backups, Contabo Object Storage), the credential path (secret/data/platform/backup/off-cluster), and the egress ranges, and the render hands the planBACKUP_OFF_CLUSTERand the credential. Only the method is missing: today it writes the local generation and then fails withErrOffCluster, so the gap is never silent.To build, in JorisJonkers-dev/delivery's
internal/backup, for every method (files, postgres, rabbitmq):<bucket>/<project>/<process>/<volume>/<generation>with S3 SigV4 PUT. A small hand-written signer keeps the dependency list short;minio-gois the alternative.x-amz-checksum-sha256). A mismatch fails the run.retaincount as the local copies: list the prefix and delete beyond it. The alternative is the bucket's lifecycle rules, which move retention out of the model.ageto a public key the Platform document names. Then the provider never holds readable copies of photos and personal data, and the private key stays off the cluster, in a safe place. The alternative is relying on the provider's encryption at rest.Recommendation: all four as listed, with client-side
ageencryption. Restore then needs the private key, which is the point.3. Proof
A restore rehearsal on one real volume, as the criteria ask. Restore jellyfin's config volume from an off-cluster copy into a scratch namespace, and start Jellyfin against it read-only.
Order
agerecipient.irreplaceableorrecoverableas its data deserves.ExtraToast commentedon Oct 26, 2026
Decided (2026-10-26, the maintainer): option B for SQLite (copy with
-waland-shm,PRAGMA integrity_checkon the copy, retry on failure; the mount stays read-only), and client-sideageencryption for the off-cluster copy, to a public key the Platform document names. The rest of the draft stands: upload in every method, verify by size and SHA-256, and retain off-cluster by the class'sretain.ExtraToast commentedon Oct 26, 2026
Progress and one more decision (2026-10-26):
<prefix>/<BACKUP_OFF_CLUSTER_PATH>/<generation>.age, read back by size and SHA-256, and pruned toBACKUP_RETAIN.offCluster.recipientis required, and lands in the 1.0.0 cut, right after feat(model)!: cut the model to what 1.0 keeps #628 and before the 1.0.0 release is published. The render hands the methodBACKUP_OFF_CLUSTER_RECIPIENTandBACKUP_OFF_CLUSTER_PATH(<project>/<process>/<volume>). The estate adds its public key when it adopts 1.0.ExtraToast commentedon Oct 27, 2026
Resume here
age-encrypted and read back), and feat(model)!: encrypt every off-cluster copy to a recipient the platform names #636 (offCluster.recipientin the model, inside 1.0.0).irreplaceableorrecoverable.-wal, fold the log in,PRAGMA integrity_check, three attempts); client-sideageencryption to a public key the Platform document names; the recipient is required in 1.0.0.irreplaceabletoday, so nothing fails nightly meanwhile. The credential's keys areAWS_ENDPOINT_URL,AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYand optionallyAWS_REGION.