From 43efbcd9e032245a8199b0d64e2679a9babf55ca Mon Sep 17 00:00:00 2001 From: Wenting Wu Date: Thu, 23 Jul 2026 13:16:46 -0400 Subject: [PATCH 1/4] docs: document multi-region backup behavior (#196) Multi-region backup/restore was implemented in #141 but the docs were not updated. This documents the region-aware backup behavior for users and maintainers: - Add a "Backups in multi-region deployments" section to the public backup-and-restore guide covering primary-only backups, standby-region skip, and automatic site-swap/failover handling. - Note multi-region awareness in the backup overview key characteristics. - Fix broken backup-and-restore link in the multi-region setup guide. - Mark multi-region backup as implemented in the design doc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0e57fca7-ab02-456c-aad6-6ddc4c89ba2f Signed-off-by: Wenting Wu --- docs/designs/backup-and-restore-design.md | 4 ++-- .../preview/operations/backup-and-restore.md | 15 +++++++++++++++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/docs/designs/backup-and-restore-design.md b/docs/designs/backup-and-restore-design.md index e1ad84538..de08d5fc2 100644 --- a/docs/designs/backup-and-restore-design.md +++ b/docs/designs/backup-and-restore-design.md @@ -27,8 +27,8 @@ When users create a DocumentDB Backup resource, the operator automatically creat **Why not use CNPG Backup directly?** -In this phase, our Backup resource acts as a wrapper around CNPG Backup. We maintain our own CRD to support future enhancements: -- **Next phase:** Multi-region backup support +In this phase, our Backup resource acts as a wrapper around CNPG Backup. We maintain our own CRD to support additional capabilities: +- **Multi-region backup support** (implemented) — in a multi-region deployment the operator backs up only the region that currently holds the primary role, skips requests against standby regions, and automatically follows the primary across failovers (site-swap). See the [Backups in multi-region deployments](../operator-public-documentation/preview/operations/backup-and-restore.md#backups-in-multi-region-deployments) user guide. - **Future:** Multi-node backup capabilities ### Creating On-Demand Backups diff --git a/docs/operator-public-documentation/preview/operations/backup-and-restore.md b/docs/operator-public-documentation/preview/operations/backup-and-restore.md index 8bb574810..249110f20 100644 --- a/docs/operator-public-documentation/preview/operations/backup-and-restore.md +++ b/docs/operator-public-documentation/preview/operations/backup-and-restore.md @@ -19,6 +19,7 @@ Key characteristics: - **VolumeSnapshot-based** — backups use the [CSI (Container Storage Interface)](https://kubernetes.io/docs/concepts/storage/volumes/#csi) driver's snapshot capability, so they are fast and storage-efficient. - **Primary-only** — the operator always targets the primary instance for backups. +- **Multi-region aware** — in a multi-region deployment the operator backs up only the region that is currently primary; requests against a standby region are skipped. See [Backups in multi-region deployments](#backups-in-multi-region-deployments). - **Namespace-scoped** — `Backup` and `ScheduledBackup` resources must reside in the same namespace as the `DocumentDB` cluster. - **Retention-managed** — expired backups are automatically deleted by the operator. @@ -210,6 +211,20 @@ Once the status shows `Cluster in healthy state`, connect and verify your data. For additional recovery options (including PV-based recovery), see [Restore a Deleted DocumentDB Cluster](restore-deleted-cluster.md). +## Backups in multi-region deployments + +In a [multi-region deployment](../multi-region-deployment/overview.md), a single DocumentDB cluster spans multiple regions with one region acting as the **primary** and the others running as **standby** replicas. Backups behave the same way as in a single-region cluster, with a few region-aware rules the operator enforces automatically: + +- **Only the primary region is backed up.** The operator resolves which region currently holds the primary role and takes the snapshot there. You do not need to target a specific region — `Backup` and `ScheduledBackup` resources are created against the DocumentDB cluster and the operator routes the snapshot to the primary. +- **Requests against a standby region are skipped, not failed.** If a `Backup` is reconciled while the local region is a standby, the operator marks it as skipped with the message *"Backups can only be created from the primary cluster"*. This keeps `ScheduledBackup` resources safe to define identically in every region. +- **Failover (site-swap) is handled automatically.** When the primary role moves to a different region (planned or unplanned failover), subsequent backups are taken from the newly promoted primary. If a backup is requested while a promotion is still in progress and the primary endpoint is not yet ready, the operator defers the backup and retries once promotion completes — it does not fail the backup. + +!!! tip "Scheduled backups across regions" + + Define your `ScheduledBackup` in the same namespace as the DocumentDB cluster. Because standby regions skip backups rather than erroring, the schedule keeps producing backups from whichever region is primary — including after a failover — with no manual reconfiguration. + +For planned and unplanned failover steps, see [Failover procedures](../multi-region-deployment/failover-procedures.md). + ## Backup Retention Policy Each backup receives an expiration time. After expiration, the operator deletes it automatically. You can define the retention period at multiple levels: From 43de5839729a854cc9ba8427931e89d8a8cd1192 Mon Sep 17 00:00:00 2001 From: Wenting Wu Date: Fri, 21 Aug 2026 14:01:59 -0400 Subject: [PATCH 2/4] docs: clarify multi-region backups require the resource on the primary cluster Each region runs its own operator with no cross-cluster routing; a Backup created only on a standby cluster is skipped and no snapshot is taken. Clarify that the resource must exist in the current primary cluster (applied directly or propagated via KubeFleet). Also redraft the CNPG-wrapper rationale to list retention management and multi-region support as the added capabilities. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu --- docs/designs/backup-and-restore-design.md | 6 +++--- .../preview/operations/backup-and-restore.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/designs/backup-and-restore-design.md b/docs/designs/backup-and-restore-design.md index de08d5fc2..88760b0e3 100644 --- a/docs/designs/backup-and-restore-design.md +++ b/docs/designs/backup-and-restore-design.md @@ -27,9 +27,9 @@ When users create a DocumentDB Backup resource, the operator automatically creat **Why not use CNPG Backup directly?** -In this phase, our Backup resource acts as a wrapper around CNPG Backup. We maintain our own CRD to support additional capabilities: -- **Multi-region backup support** (implemented) — in a multi-region deployment the operator backs up only the region that currently holds the primary role, skips requests against standby regions, and automatically follows the primary across failovers (site-swap). See the [Backups in multi-region deployments](../operator-public-documentation/preview/operations/backup-and-restore.md#backups-in-multi-region-deployments) user guide. -- **Future:** Multi-node backup capabilities +Our Backup resource acts as a wrapper around CNPG Backup, but we maintain our own CRD because it provides capabilities beyond CNPG's Backup: +- **Retention management** — each backup is assigned an expiration time and the operator automatically deletes it once it expires. The retention period can be set per-backup, per-schedule, or per-cluster (`retentionDays`), and defaults to 30 days. Backups are retained according to this period even after the cluster is deleted, so you can still restore from a deleted cluster. +- **Multi-region backup support** — in a multi-region deployment the operator backs up only the region that currently holds the primary role, skips requests against standby regions, and automatically follows the primary across failovers (site-swap). See the [Backups in multi-region deployments](../operator-public-documentation/preview/operations/backup-and-restore.md#backups-in-multi-region-deployments) user guide. ### Creating On-Demand Backups diff --git a/docs/operator-public-documentation/preview/operations/backup-and-restore.md b/docs/operator-public-documentation/preview/operations/backup-and-restore.md index 249110f20..cfbe9e12a 100644 --- a/docs/operator-public-documentation/preview/operations/backup-and-restore.md +++ b/docs/operator-public-documentation/preview/operations/backup-and-restore.md @@ -215,13 +215,13 @@ For additional recovery options (including PV-based recovery), see [Restore a De In a [multi-region deployment](../multi-region-deployment/overview.md), a single DocumentDB cluster spans multiple regions with one region acting as the **primary** and the others running as **standby** replicas. Backups behave the same way as in a single-region cluster, with a few region-aware rules the operator enforces automatically: -- **Only the primary region is backed up.** The operator resolves which region currently holds the primary role and takes the snapshot there. You do not need to target a specific region — `Backup` and `ScheduledBackup` resources are created against the DocumentDB cluster and the operator routes the snapshot to the primary. +- **Only the primary region is backed up.** Each region runs its own operator, and an operator only acts on `Backup` and `ScheduledBackup` resources in its own Kubernetes cluster — there is no cross-cluster routing. The operator in the primary region takes the snapshot; operators in standby regions skip the request. Because of this, the resource must exist in the cluster that currently holds the primary role. Either apply it directly to the current primary cluster, or propagate it to every member cluster (for example, via [KubeFleet](../multi-region-deployment/overview.md#managed-fleet-orchestration)) so that the primary-region operator executes it while standby regions skip it automatically. - **Requests against a standby region are skipped, not failed.** If a `Backup` is reconciled while the local region is a standby, the operator marks it as skipped with the message *"Backups can only be created from the primary cluster"*. This keeps `ScheduledBackup` resources safe to define identically in every region. - **Failover (site-swap) is handled automatically.** When the primary role moves to a different region (planned or unplanned failover), subsequent backups are taken from the newly promoted primary. If a backup is requested while a promotion is still in progress and the primary endpoint is not yet ready, the operator defers the backup and retries once promotion completes — it does not fail the backup. !!! tip "Scheduled backups across regions" - Define your `ScheduledBackup` in the same namespace as the DocumentDB cluster. Because standby regions skip backups rather than erroring, the schedule keeps producing backups from whichever region is primary — including after a failover — with no manual reconfiguration. + Define your `ScheduledBackup` in the same namespace as the DocumentDB cluster, and make sure it exists in the current primary cluster — either apply it there directly or propagate it to every member cluster (for example, via KubeFleet). Because standby regions skip backups rather than erroring, a propagated schedule keeps producing backups from whichever region is primary — including after a failover — with no manual reconfiguration. For planned and unplanned failover steps, see [Failover procedures](../multi-region-deployment/failover-procedures.md). From c5c39dd183b593ca3512608946db74c69e2ff44a Mon Sep 17 00:00:00 2001 From: Wenting Wu Date: Fri, 21 Aug 2026 14:06:17 -0400 Subject: [PATCH 3/4] docs: make multi-region backup section more scannable Restructure the multi-region backup guidance for readability: lead with the required user action, add an at-a-glance region-behavior table with the actual Backup phases (completed/skipped), and add a kubectl verification example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu --- .../preview/operations/backup-and-restore.md | 32 ++++++++++++++++--- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/docs/operator-public-documentation/preview/operations/backup-and-restore.md b/docs/operator-public-documentation/preview/operations/backup-and-restore.md index cfbe9e12a..4413f0571 100644 --- a/docs/operator-public-documentation/preview/operations/backup-and-restore.md +++ b/docs/operator-public-documentation/preview/operations/backup-and-restore.md @@ -213,11 +213,35 @@ For additional recovery options (including PV-based recovery), see [Restore a De ## Backups in multi-region deployments -In a [multi-region deployment](../multi-region-deployment/overview.md), a single DocumentDB cluster spans multiple regions with one region acting as the **primary** and the others running as **standby** replicas. Backups behave the same way as in a single-region cluster, with a few region-aware rules the operator enforces automatically: +In a [multi-region deployment](../multi-region-deployment/overview.md), a single DocumentDB cluster spans multiple regions — one region is the **primary** and the others are **standby** replicas. Each region runs its own operator, and **an operator only acts on `Backup` and `ScheduledBackup` resources in its own Kubernetes cluster — there is no cross-cluster routing.** Apart from that, backups behave exactly as in a single-region cluster. -- **Only the primary region is backed up.** Each region runs its own operator, and an operator only acts on `Backup` and `ScheduledBackup` resources in its own Kubernetes cluster — there is no cross-cluster routing. The operator in the primary region takes the snapshot; operators in standby regions skip the request. Because of this, the resource must exist in the cluster that currently holds the primary role. Either apply it directly to the current primary cluster, or propagate it to every member cluster (for example, via [KubeFleet](../multi-region-deployment/overview.md#managed-fleet-orchestration)) so that the primary-region operator executes it while standby regions skip it automatically. -- **Requests against a standby region are skipped, not failed.** If a `Backup` is reconciled while the local region is a standby, the operator marks it as skipped with the message *"Backups can only be created from the primary cluster"*. This keeps `ScheduledBackup` resources safe to define identically in every region. -- **Failover (site-swap) is handled automatically.** When the primary role moves to a different region (planned or unplanned failover), subsequent backups are taken from the newly promoted primary. If a backup is requested while a promotion is still in progress and the primary endpoint is not yet ready, the operator defers the backup and retries once promotion completes — it does not fail the backup. +### What you need to do + +Make sure the `Backup` or `ScheduledBackup` exists in the cluster that currently holds the **primary** role. You have two options: + +- **Apply it directly** to the current primary cluster, or +- **Propagate it to every member cluster** (for example, via [KubeFleet](../multi-region-deployment/overview.md#managed-fleet-orchestration)). The primary-region operator runs the backup while standby-region operators skip it, so the same resource is safe to define identically in every region — including after a failover. + +### Region behavior at a glance + +| Region role | Operator behavior | Backup `Phase` | +|-------------|-------------------|----------------| +| **Primary** | Takes the VolumeSnapshot | `completed` | +| **Standby** | Skips it, with the message *"Backups can only be created from the primary cluster"* (not an error) | `skipped` | +| **Promotion in progress** | Defers the backup and retries until the new primary endpoint is ready (not an error) | pending → `completed` | + +**Failover (site-swap) is handled automatically.** When the primary role moves to another region (planned or unplanned), subsequent backups are taken from the newly promoted primary. A backup requested while promotion is still in progress is deferred and retried once the primary endpoint is ready — it is never failed for this reason. + +### Verify which backups ran + +List the backups and check the `PHASE` column to see which region took the snapshot and which skipped: + +```bash +kubectl get backups -n +# NAME CLUSTER PHASE ... +# backup-example documentdb-preview completed # taken on the primary region +# backup-example documentdb-preview skipped # standby region, safely skipped +``` !!! tip "Scheduled backups across regions" From 8cbc89b1f0537077c918d7a75de5c0f124a99011 Mon Sep 17 00:00:00 2001 From: Wenting Wu Date: Fri, 21 Aug 2026 14:59:40 -0400 Subject: [PATCH 4/4] docs: drop CNPG wrapper implementation detail from backup design Per review feedback, describe the DocumentDB Backup by the capabilities it adds over CNPG's Backup rather than how it is implemented (wrapper/own CRD). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu --- docs/designs/backup-and-restore-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/designs/backup-and-restore-design.md b/docs/designs/backup-and-restore-design.md index 88760b0e3..6a3ea9ab1 100644 --- a/docs/designs/backup-and-restore-design.md +++ b/docs/designs/backup-and-restore-design.md @@ -27,7 +27,7 @@ When users create a DocumentDB Backup resource, the operator automatically creat **Why not use CNPG Backup directly?** -Our Backup resource acts as a wrapper around CNPG Backup, but we maintain our own CRD because it provides capabilities beyond CNPG's Backup: +Our Backup resource provides capabilities beyond CNPG's Backup: - **Retention management** — each backup is assigned an expiration time and the operator automatically deletes it once it expires. The retention period can be set per-backup, per-schedule, or per-cluster (`retentionDays`), and defaults to 30 days. Backups are retained according to this period even after the cluster is deleted, so you can still restore from a deleted cluster. - **Multi-region backup support** — in a multi-region deployment the operator backs up only the region that currently holds the primary role, skips requests against standby regions, and automatically follows the primary across failovers (site-swap). See the [Backups in multi-region deployments](../operator-public-documentation/preview/operations/backup-and-restore.md#backups-in-multi-region-deployments) user guide.