diff --git a/README.md b/README.md index bf9862d..b7357ec 100644 --- a/README.md +++ b/README.md @@ -22,14 +22,16 @@ Keep these three layers separate while reading the guide: | Restate cluster | Three stateful Restate server pods installed into EKS | | SDK service | Customer application code that uses a Restate SDK; deployed through an independent application workflow | -The sizing and runtime tuning come from Restate Cloud's -`3-node.xlarge-vqueues` profile: +The cluster shape and runtime tuning are derived from Restate Cloud's +`3-node.xlarge-vqueues` profile. This standalone reference adapts selected +infrastructure settings, including its initial storage capacity; the +[profile-fidelity record](docs/04-profile-fidelity.md) explains each difference. | Setting | Value | |---|---| | Restate nodes | 3, with hard host anti-affinity | | Per-node request | 24 vCPU, 50 GiB memory | -| Data volume | 256 GiB encrypted `restate-gp3` EBS volume; increase after measuring usage | +| Data volume | 256 GiB encrypted `restate-gp3` EBS volume; monitor utilization and expand with headroom | | Partitions | 48 | | Node replication | 2 | | Restate image | `docker.restate.dev/restatedev/restate:1.7.7` | diff --git a/docs/00-architecture.md b/docs/00-architecture.md index a9533cc..0e59a21 100644 --- a/docs/00-architecture.md +++ b/docs/00-architecture.md @@ -322,9 +322,10 @@ Each Restate pod starts with a 256 GiB PVC using the repository-owned - `WaitForFirstConsumer`, so the volume is provisioned in the pod's zone; - `Retain`, so deleting the PVC does not delete the EBS PV. -The PVC can grow after observing actual log and snapshot behavior, but it -cannot shrink. Choose a larger initial value before the first apply when the -expected workload requires it. +Monitor `/restate-data` utilization and growth on every pod, then expand PVC +capacity while enough operational headroom remains. Choose a larger initial +value before the first apply when the expected workload requires it. See +[Monitor and expand storage](05-operations.md#monitor-and-expand-storage). `Retain` is a safety net, not an automatic restore process. A Released PV keeps its former claim reference and must be handled explicitly during recovery. diff --git a/docs/03-deploying-services.md b/docs/03-deploying-services.md index acb0f7d..5748bd2 100644 --- a/docs/03-deploying-services.md +++ b/docs/03-deploying-services.md @@ -364,13 +364,28 @@ data: ``` With this in place, a controller failure or a foreign-deployment conflict shows -as Degraded within one reconcile. Do not classify `AdminCallRejected` as -terminal: the operator uses it for transient 5xx responses as well as -incompatible registrations, so the check leaves it Progressing and its message -explains the timeout. The previous revision keeps serving because the operator -never replaced it. Teams that deploy the cluster stages with Terraform and the -applications with Argo CD get the boundary this guide recommends without giving -up automated health gating. +as Degraded within one reconcile. `AdminCallRejected` is not reliably terminal: +the operator uses it for transient 5xx responses as well as incompatible +registrations, so the check leaves it Progressing and preserves the response +details in the message. The previous revision keeps serving because the +operator never replaced it. + +The health check itself has no elapsed-time rule. Bound an interactive or CI +wait explicitly so a permanent rejection cannot leave the caller waiting +indefinitely: + +```bash +argocd app sync --timeout 600 +# For a sync that another actor started: +argocd app wait --health --timeout 600 +``` + +A CLI timeout does not repair or reclassify the revision. Inspect the +`RestateDeployment` condition, correct an incompatible specification, and +configure an alert on prolonged Progressing health for automated syncs. Teams +that deploy the cluster stages with Terraform and applications with Argo CD +retain automated health gating without treating a transient server response as +a failed release. ### Flux diff --git a/docs/04-profile-fidelity.md b/docs/04-profile-fidelity.md index 5997c6e..a51b882 100644 --- a/docs/04-profile-fidelity.md +++ b/docs/04-profile-fidelity.md @@ -68,10 +68,9 @@ intentional. A customer performing a standard installation can skip it. exists, StorageClass parameters are immutable (applying over it fails), and a scoped name keeps other workloads off a class this stack owns. - Storage capacity: the source profile uses 1 TiB per node; this reference - starts with 256 GiB per node. Kubernetes PVCs can grow but not shrink, so the - lower initial allocation avoids provisioning unused EBS storage while - preserving a path to increase capacity as log and snapshot usage becomes - clear. + starts with 256 GiB per node. The lower initial allocation avoids + provisioning unused EBS storage while preserving a path to expand capacity + as local log and partition-store utilization grows. - NetworkPolicy: cloud exposes ingress **and admin** only to its own authenticating gateway namespace; this repo opens ingress to `restate-apps` and keeps the unauthenticated admin API closed to workloads entirely diff --git a/docs/05-operations.md b/docs/05-operations.md index 5157cb0..9cf1259 100644 --- a/docs/05-operations.md +++ b/docs/05-operations.md @@ -184,6 +184,20 @@ Restate needs no agent or sidecar; the work is on the platform side. policy blocks, so add its address under `spec.security.networkEgressRules` as described under [Private AWS endpoints](00-architecture.md#private-aws-endpoints). +- **Storage.** The 256 GiB initial PVC size is a starting point rather than a + capacity forecast. Monitor the filesystem mounted at `/restate-data` on all + three pods, alert on percentage used and recent growth, and expand with + enough headroom for the time your change process takes. This repository does + not install an observability stack, so connect these signals to the + platform's existing monitoring before handing the cluster over. + +For a direct point-in-time check: + +```bash +kubectl -n restate exec restate-0 -- df -h /restate-data +kubectl -n restate exec restate-1 -- df -h /restate-data +kubectl -n restate exec restate-2 -- df -h /restate-data +``` ## Verify the snapshot path @@ -468,12 +482,22 @@ own the example service in separate application state. Whichever delivery tool owns the service should apply the revision and its rollback. See [Deploying services](03-deploying-services.md#roll-out-a-new-version). -### Increase storage +### Monitor and expand storage -`spec.storage.storageRequestBytes` may only increase. Increasing it updates the -PVC request; actual expansion depends on the EBS CSI driver and the -StorageClass. Keep the value unchanged or increase it; retained PVs do not -shrink. +Use the `/restate-data` utilization and growth rate described under +[Observability](#observability) to choose a new capacity with operational +headroom. Increase `spec.storage.storageRequestBytes` in +`resources/04-restate-cluster.yaml`, review the manifest diff or Terraform +plan, and apply it through the workflow that owns the cluster. The operator +updates the PVC request; completed expansion depends on the EBS CSI driver and +the `restate-gp3` StorageClass. Confirm the new capacity on every pod: + +```bash +kubectl -n restate get pvc +kubectl -n restate exec restate-0 -- df -h /restate-data +kubectl -n restate exec restate-1 -- df -h /restate-data +kubectl -n restate exec restate-2 -- df -h /restate-data +``` ### Change runtime sizing or configuration diff --git a/misc/pdf/build_manual_reference.py b/misc/pdf/build_manual_reference.py index 85885a5..dee097f 100755 --- a/misc/pdf/build_manual_reference.py +++ b/misc/pdf/build_manual_reference.py @@ -21,9 +21,9 @@ ROOT = Path(__file__).resolve().parents[2] DEFAULT_OUTPUT = "output/pdf/restate-eks-manual-deployment-reference.pdf" -DEFAULT_SOURCE_COMMIT = "388fbec8fdb93b9e9218efaae8fc7fb63120a50b" -DEFAULT_SOURCE_DATE = "2026-09-01" -DEFAULT_PREPARED = "1 September 2026" +DEFAULT_SOURCE_COMMIT = "b31a2e8274bf56fc7493188b903e55106ac42d08" +DEFAULT_SOURCE_DATE = "2026-09-04" +DEFAULT_PREPARED = "4 September 2026" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--output", default=DEFAULT_OUTPUT) @@ -466,7 +466,7 @@ def draw_later(canvas, doc): ["Component", "Reference value", "Operational meaning"], [ ("Restate", "3 replicas; 24 vCPU and 50 GiB each", "Requires three different eligible nodes"), - ("Data", "1 TiB encrypted gp3 per pod; Retain", "EBS survives PVC deletion; reattachment is manual"), + ("Data", "256 GiB encrypted gp3 per pod; Retain", "Monitor /restate-data and expand with operational headroom"), ("Snapshots", "Dedicated S3 bucket through IRSA", "One bucket or unique prefix per Restate cluster"), ("Network", "Default-deny policies where enforced", "Admin 9070 stays private; SDK port 9080 is isolated"), ("Exposure", "ClusterIP only", "No ingress, DNS, public endpoint, or auth gateway is created"), @@ -817,7 +817,7 @@ def draw_later(canvas, doc): story.append( callout( "EXPECTED STORAGE", - "EBS CSI, encrypted XFS, 6000 IOPS, 500 MiB/s, WaitForFirstConsumer, allowVolumeExpansion, and reclaimPolicy Retain. Retain preserves EBS after PVC deletion; it does not perform automatic recovery.", + "Each pod starts with 256 GiB. The StorageClass uses EBS CSI, encrypted XFS, 6000 IOPS, 500 MiB/s, WaitForFirstConsumer, allowVolumeExpansion, and reclaimPolicy Retain. Monitor /restate-data on every pod and expand with operational headroom. Retain preserves EBS after PVC deletion; it does not perform automatic recovery.", "blue", ) ) @@ -1071,8 +1071,8 @@ def draw_later(canvas, doc): story.append(P("Safe-change pattern", "H2Custom")) story.append(bullet("Verify cluster health and a recent snapshot before changing runtime sizing, storage, image, chart, or experimental settings.")) story.append(bullet("Review one change at a time. Pod-template changes can roll all three stateful pods and move partition leadership.")) -story.append(bullet("Keep requested storage at its current size or increase it. Existing volume expansion depends on the EBS CSI driver and StorageClass.")) -story.append(bullet("For upgrades, validate release compatibility and every experimental/profile-derived setting; changing only the image is not a complete upgrade plan.")) +story.append(bullet("Monitor /restate-data on every pod. Expand with operational headroom, then confirm capacity.")) +story.append(bullet("For upgrades, revalidate release compatibility and profile-derived settings; changing only the image is incomplete.")) story.append(PageBreak()) # 12 diff --git a/output/pdf/restate-eks-manual-deployment-reference.pdf b/output/pdf/restate-eks-manual-deployment-reference.pdf index 093fceb..0a76ef8 100644 Binary files a/output/pdf/restate-eks-manual-deployment-reference.pdf and b/output/pdf/restate-eks-manual-deployment-reference.pdf differ diff --git a/resources/04-restate-cluster.yaml b/resources/04-restate-cluster.yaml index ef8bf8a..059aaf7 100644 --- a/resources/04-restate-cluster.yaml +++ b/resources/04-restate-cluster.yaml @@ -391,7 +391,7 @@ spec: storage: # restate-gp3 StorageClass from resources/03-gp3-storageclass.yaml (EBS # CSI driver required; scoped name — see that file's header). The PVC - # starts at 256 GiB per node and can grow later, but - # storageRequestBytes cannot be decreased. + # starts at 256 GiB per node; monitor /restate-data and increase + # storageRequestBytes with enough operational headroom. storageClassName: restate-gp3 storageRequestBytes: 274877906944