Repository navigation
feat(repo-sync): mirror backups + WAL between repositories - #9
Merged
Merged
Conversation
Add repo-sync (Layer 1): byte-identical mirroring of completed backups and
archived WAL from the active repository to every other configured repository,
plus an inline `--repo-sync` option on `backup`.
Repositories in a multi-repo set are configured as identical mirrors (global
compression; identical bundling / block-incremental / cipher-type; shared cipher
sub-key), so syncing is a pure raw byte copy of the stored objects at the same
repo paths -- no decode / re-encode / re-bundle / re-block / size recompute.
Consistency is validated up front: repo-sync refuses mismatched repositories and
aligns a fresh encrypted target's sub-key to the source's so the copied bytes
decrypt. Intended topology: the primary archives to repo1 only; repo1 then
synchronises to repo2..N, run where repo1 is local.
- crates/pgbr-command/src/sync/{mod,backup,archive}.rs: the engine --
assert_consistent, sync_backup_to_repo (diff/incr ancestor backfill,
manifest-sentinel idempotency, verbatim backup.info merge), sync_archive_to_repo
(idempotent WAL copy), and the `repo-sync` command (--type=all|wal|backup,
--set=LABEL).
- backup: new `repo-sync` boolean mirrors the just-completed backup to the other
repositories before expire-auto.
- config.yaml / help.xml: the `repo-sync` command + global option, and the
stanza / repo / repo-bundle / repo-block command associations they require.
- integration: new Docker scenario 12-repo-sync.sh (encrypted, bundle + block,
archive-to-repo1-then-sync, byte-identical verification, restore from mirror)
and Vagrant scenario 25; a repo-sync smoke check in 08-multi-repo.sh.
Layer 2 (automatic server-triggered fan-out on WAL receipt over the TLS server
topology) is a follow-up.
Validated: fmt, clippy --all-targets -D warnings, cargo test --workspace (28 new
sync tests); all 12 Docker scenarios pass; full Vagrant suite 211 passed / 0
failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…nfig faults Review fixes on the repo-sync engine: - copy integrity: shared size-checked copy_object (sync/mod.rs) re-copies torn objects instead of sealing them behind the manifest sentinel - ordering: destination-initialized check runs before any copy, and the sentinel path still reconciles the backup.info entry (merge_backup_info) - CLI: backup builds all repo backends only when --repo-sync is enabled - inline sync: best-effort across targets; expire-auto always runs, the first sync error is surfaced after retention - WAL: source archive.info decrypted with the src_index passphrase (not the lowest configured index); destination listed once instead of per-segment exists probes - consistency: assert_consistent enforces db identity (archive-id + system-id); dead repo-block-size entry replaced with real options - align_sub_key preserves prior info files as .presync before rewrite - reuse archive::COMPRESS_SUFFIXES (now pub(crate)); CLAUDE.md updated Co-Authored-By: Claude Fable 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
repo-sync (Layer 1): repo-to-repo mirroring of backups + WAL
What
A new
repo-synccommand and an inline--repo-syncoption onbackupthatmirror completed backups and archived WAL from the active repository to every
other configured repository.
Intended topology: the primary archives to repo1 only; repo1 then
synchronises to repo2..N — run where repo1 is local (e.g. on the repository
host), so no bytes round-trip through the primary.
How
Repos in a multi-repo set are configured as byte-identical mirrors:
compression is global (already identical), and bundling / block-incremental /
cipher-type are validated identical, with a shared cipher sub-key. Mirroring
is therefore a pure raw byte copy of the stored objects (standalone files,
bundles, block maps, manifests, WAL segments) at the same repo paths — no decode
/ re-encode / re-bundle / re-block / size recompute.
repo-syncvalidates this consistency up front (refusing mismatched repos) and,for a fresh empty encrypted target, aligns its sub-key to the source's so the
raw-copied bytes decrypt; a populated target under a divergent sub-key is a hard
error.
crates/pgbr-command/src/sync/):sync_backup_to_repo— diff/incr ancestor backfill, manifest-sentinelidempotency, verbatim
backup.infoentry merge (never overwrites thetarget's own backups), missing db-history copy.
sync_archive_to_repo/ per-segment copy — idempotent WAL mirroring(
repo_has_segmentskip across compression suffixes).repo-synccommand —--type=all|wal|backup(defaultall),--set=LABEL,best-effort across targets, holds the stanza backup lock.
backup --repo-syncmirrors the just-completed backup to the otherrepositories before expire-auto.
repo-synccommand + global boolean option, plus thestanza/repo/repo-bundle/repo-blockcommand associations thecommand needs to resolve storage, cipher, and mirror settings.
Tests
sync/{mod,backup,archive}.rs): raw copy, encrypted-mirrorsub-key alignment + restorability, consistency rejection (bundle/block/cipher),
diff ancestor backfill, WAL idempotency,
--type/--set.12-repo-sync.sh— encrypted + bundle + block,archive-to-repo1-then-sync, byte-identical object verification, restore from
the mirror. All 12 Docker scenarios pass.
cargo fmt --check,cargo clippy --workspace --all-targets -D warnings,cargo test --workspace— all green.Scope
Layer 1 only (engine + command + inline option). Layer 2 — automatic
server-triggered fan-out on WAL receipt over the TLS
servertopology — is adocumented follow-up (touches the remote/server subsystem).
🤖 Generated with Claude Code