Avoid terminal classification of transient Admin failures

README.md

@@ -29,7 +29,7 @@ The sizing and runtime tuning come from Restate Cloud's
 |---|---|
 | Restate nodes | 3, with hard host anti-affinity |
 | Per-node request | 24 vCPU, 50 GiB memory |
-| Data volume | 1 TiB encrypted `restate-gp3` EBS volume |
+| Data volume | 256 GiB encrypted `restate-gp3` EBS volume; increase after measuring usage |
 | Partitions | 48 |
 | Node replication | 2 |
 | Restate image | `docker.restate.dev/restatedev/restate:1.7.7` |
@@ -126,7 +126,7 @@ and draining. See [Deploying services](docs/03-deploying-services.md).
                           namespace: restate-apps
                           RestateDeployment revisions
 
-  each Restate pod ──► 1 TiB EBS PV       cluster ──IRSA──► dedicated S3 bucket
+  each Restate pod ──► 256 GiB EBS PV     cluster ──IRSA──► dedicated S3 bucket
 ```
 
 The operator owns the `restate` namespace and materializes the StatefulSet,

docs/00-architecture.md

@@ -17,7 +17,7 @@ version `3.0.1`.
 - A `RestateCluster` named `restate` produces an operator-owned namespace of
   the same name.
 - Three Restate pods run as one StatefulSet with one pod per Kubernetes node.
-- Each pod has a 1 TiB EBS volume; partition snapshots are written to a
+- Each pod starts with a 256 GiB EBS volume; partition snapshots are written to a
   cluster-dedicated S3 bucket.
 - The operator, not an individual Restate node, provisions the cluster exactly
   once.
@@ -313,7 +313,7 @@ later tuning step.
 
 ## Storage and snapshots
 
-Each Restate pod receives a 1 TiB PVC using the repository-owned
+Each Restate pod starts with a 256 GiB PVC using the repository-owned
 `restate-gp3` StorageClass:
 
 - EBS CSI provisioner;
@@ -322,6 +322,10 @@ Each Restate pod receives a 1 TiB PVC using the repository-owned
 - `WaitForFirstConsumer`, so the volume is provisioned in the pod's zone;
 - `Retain`, so deleting the PVC does not delete the EBS PV.
 
+The PVC can grow after observing actual log and snapshot behavior, but it
+cannot shrink. Choose a larger initial value before the first apply when the
+expected workload requires it.
+
 `Retain` is a safety net, not an automatic restore process. A Released PV keeps
 its former claim reference and must be handled explicitly during recovery.
 

docs/03-deploying-services.md

@@ -312,7 +312,7 @@ Restate reason.
 
 The optional `terraform/03-services` root avoids this race with
 `scripts/wait-restatedeployment.sh`. The script waits for
-`status.observedGeneration` to reach `metadata.generation`, then requires
+`status.observedGeneration` to equal `metadata.generation`, then requires
 `Ready=True`; on timeout it prints the last reason and message. If you manage a
 `RestateDeployment` in another Terraform root, use the same generation-aware
 gate rather than a positive condition wait alone:
@@ -348,8 +348,8 @@ data:
           if c.status == "True" then
             hs.status = "Healthy"
             hs.message = c.message or "Deployed"
-          elseif c.status == "Unknown" or c.reason == "AdminCallRejected"
-              or c.reason == "ForeignDeployment" or c.reason == "NotLatest"
+          elseif c.status == "Unknown" or c.reason == "ForeignDeployment"
+              or c.reason == "NotLatest"
               or c.reason == "FailedReconcile" then
             hs.status = "Degraded"
             hs.message = c.message
@@ -363,11 +363,14 @@ data:
     return hs
 ```
 
-With this in place, a sync of a rejected revision fails its health check within
-one reconcile instead of waiting on a timeout, and the previous revision keeps
-serving because the operator never replaced it. Teams that deploy the cluster
-stages with Terraform and the applications with Argo CD get the boundary this
-guide recommends without giving up automated health gating.
+With this in place, a controller failure or a foreign-deployment conflict shows
+as Degraded within one reconcile. Do not classify `AdminCallRejected` as
+terminal: the operator uses it for transient 5xx responses as well as
+incompatible registrations, so the check leaves it Progressing and its message
+explains the timeout. The previous revision keeps serving because the operator
+never replaced it. Teams that deploy the cluster stages with Terraform and the
+applications with Argo CD get the boundary this guide recommends without giving
+up automated health gating.
 
 ### Flux
 
@@ -397,15 +400,15 @@ spec:
         status.conditions.exists(c,
           c.type == 'Ready' &&
           (c.status == 'Unknown' ||
-           c.reason == 'AdminCallRejected' ||
            c.reason == 'ForeignDeployment' ||
            c.reason == 'NotLatest' ||
            c.reason == 'FailedReconcile'))
 ```
 
-When neither expression is true, Flux continues waiting. Keep a bounded timeout
-for transient scaling and admin-connectivity failures, and inspect the `Ready`
-condition if it expires.
+When neither expression is true, Flux continues waiting. Do not include
+`AdminCallRejected` in `failed`: the operator also uses that reason for
+transient 5xx responses. Keep a bounded timeout for transient scaling and
+admin-connectivity failures, and inspect the `Ready` condition if it expires.
 
 ## Useful fields
 

docs/04-profile-fidelity.md

@@ -32,7 +32,6 @@ intentional. A customer performing a standard installation can skip it.
 - **Scheduling**: required hostname anti-affinity + preferred zone spread; the
   `cloud.restate.dev/interruptible` toleration (inert unless you taint nodes
   with it).
-- **Storage**: 1 TiB gp3 per node.
 
 ## Moved, same effect
 
@@ -68,6 +67,11 @@ intentional. A customer performing a standard installation can skip it.
   than cloud's `gp3`: on a shared cluster a generic `gp3` class often already
   exists, StorageClass parameters are immutable (applying over it fails), and
   a scoped name keeps other workloads off a class this stack owns.
+- Storage capacity: the source profile uses 1 TiB per node; this reference
+  starts with 256 GiB per node. Kubernetes PVCs can grow but not shrink, so the
+  lower initial allocation avoids provisioning unused EBS storage while
+  preserving a path to increase capacity as log and snapshot usage becomes
+  clear.
 - NetworkPolicy: cloud exposes ingress **and admin** only to its own
   authenticating gateway namespace; this repo opens ingress to `restate-apps`
   and keeps the unauthenticated admin API closed to workloads entirely

resources/04-restate-cluster.yaml

@@ -1,10 +1,12 @@
 # RestateCluster: 3-node replicated Restate on EKS, managed by the
 # restate-operator (https://github.com/restatedev/restate-operator).
 #
-# Sizing/tuning is a faithful translation of the Restate Cloud profile
+# Sizing/tuning derives from the Restate Cloud profile
 # `3-node.xlarge-vqueues` (restate-cloud/config/profiles.json):
-#   3 replicas x 24 CPU / 50 GiB, 48 partitions, 1 TiB gp3 volume each,
-#   high-throughput production cell tuning with vqueues enabled.
+#   3 replicas x 24 CPU / 50 GiB, 48 partitions, and high-throughput
+#   production cell tuning with vqueues enabled. The profile uses 1 TiB gp3
+#   volumes. This reference initially allocates 256 GiB per node, allowing a
+#   new installation to increase storage after observing workload usage.
 #
 # Deliberately dropped from the cloud rendering (cloud-control-plane machinery
 # that doesn't apply to a standalone deployment):
@@ -388,8 +390,8 @@ spec:
 
   storage:
     # restate-gp3 StorageClass from resources/03-gp3-storageclass.yaml (EBS
-    # CSI driver required; scoped name — see that file's header). 1 TiB per
-    # node, from the profile; can be grown later (storageRequestBytes can
-    # only ever increase).
+    # CSI driver required; scoped name — see that file's header). The PVC
+    # starts at 256 GiB per node and can grow later, but
+    # storageRequestBytes cannot be decreased.
     storageClassName: restate-gp3
-    storageRequestBytes: 1099511627776
+    storageRequestBytes: 274877906944

scripts/wait-restatedeployment.sh

@@ -60,7 +60,7 @@ while ((SECONDS < deadline)); do
 
     if [[ "$observed_generation" =~ ^[0-9]+$ ]] \
       && [[ "$generation" =~ ^[0-9]+$ ]] \
-      && ((observed_generation >= generation)) \
+      && ((observed_generation == generation)) \
       && [[ "$ready_status" == "True" ]]; then
       echo "RestateDeployment ${RSD_NAMESPACE}/${RSD_NAME} is Ready at generation ${generation}."
       exit 0