Phase 6

.agents/AGENTS.md

@@ -37,7 +37,7 @@
    - Comments on fields in `api/v1alpha1/` are parsed by `controller-gen` into CRD OpenAPI schema descriptions. Keep them user-facing and precise.
 
 4. **Update Docs on Architecture Changes**:
-   - When an architecture boundary or CRD field changes, update `docs/agentrax.md` and `.agents/skills/agentrax-context/SKILL.md` in the same commit.
+   - When an architecture boundary or CRD field changes, update `docs/ARCHITECTURE.md` and `.agents/skills/agentrax-context/SKILL.md` in the same commit.
 
 ## Tooling & Developer Environment
 
@@ -52,16 +52,17 @@ Configure your MCP client with `--project-from-cwd` (for example, `serena start-
 
 **Use Serena tools instead of text search for the following tasks:**
 
-| Task | Use instead of |
-| ---- | -------------- |
-| Find where a type, function, or constant is defined | `find_symbol` / `find_declaration` rather than `grep` |
-| Find all usages/call sites of a symbol across packages | `find_referencing_symbols` rather than `grep -r` |
-| Understand what symbols a file or package exports | `get_symbols_overview` rather than skimming the file |
-| Rename a symbol consistently across all packages | `rename_symbol` rather than manual multi-file sed |
-| Navigate to where an interface is implemented | `find_implementations` rather than text search |
-| Check diagnostics/type errors before proposing a fix | `get_diagnostics_for_file` |
+| Task                                                   | Use instead of                                        |
+| ------------------------------------------------------ | ----------------------------------------------------- |
+| Find where a type, function, or constant is defined    | `find_symbol` / `find_declaration` rather than `grep` |
+| Find all usages/call sites of a symbol across packages | `find_referencing_symbols` rather than `grep -r`      |
+| Understand what symbols a file or package exports      | `get_symbols_overview` rather than skimming the file  |
+| Rename a symbol consistently across all packages       | `rename_symbol` rather than manual multi-file sed     |
+| Navigate to where an interface is implemented          | `find_implementations` rather than text search        |
+| Check diagnostics/type errors before proposing a fix   | `get_diagnostics_for_file`                            |
 
 **When NOT to use Serena:**
+
 - Simple single-file reads — `view_file` is faster.
 - Writing or replacing file content — use the standard edit tools.
 - Searching for plain string literals (log messages, YAML values) — `grep` is fine.

.agents/skills/agentrax-context/SKILL.md

@@ -3,7 +3,9 @@ name: agentrax-context
 description: Project context and settled architecture decisions for the Agentrax Kubernetes operator (module agentrax.io/v1alpha1, repo agentrax). Always consult this before writing, reviewing, or reasoning about any code in this repository — CRD types, the reconciler, the rollout controller, the autoscaler, the quota webhook, or the MCP registry — so implementation stays consistent with the design doc instead of drifting or re-deriving decisions that are already settled. Trigger on any mention of AgentDeployment, TenantQuota, canary rollout, or this repo's controllers, even if the user doesn't name the skill directly.
 ---
 
-> When uncertain about any architecture decision, defer to `docs/agentrax.md` rather than improvising. Don't guess when the doc has the answer.
+# Agentrax Context Skill
+
+> When uncertain about any architecture decision, defer to `docs/ARCHITECTURE.md` rather than improvising. Don't guess when the doc has the answer.
 
 ## Non-negotiable terminology
 
@@ -23,22 +25,24 @@ description: Project context and settled architecture decisions for the Agentrax
 
 ## Package map
 
-| Package                | Responsibility                                                                            |
-| ---------------------- | ----------------------------------------------------------------------------------------- |
-| `api/v1alpha1/`        | CRD Go types, validation markers, defaulting. No business logic.                          |
-| `internal/controller/` | Reconcile loops. Only code that calls the Kubernetes API for core owned resources.        |
-| `internal/rollout/`    | Canary state machine and PromQL threshold evaluation.                                     |
-| `internal/scaling/`    | HPA generation and quota-capped scaling logic.                                            |
-| `internal/registry/`   | MCP registrar, registry HTTP handler, TTL sweep.                                          |
-| `internal/quota/`      | Quota arithmetic and in-flight reservation. Shared by webhook and TenantQuota reconciler. |
+| Package                | Responsibility                                                                                                                   |
+| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
+| `api/v1alpha1/`        | CRD Go types, validation markers, defaulting. No business logic.                                                                 |
+| `internal/controller/` | Reconcile loops. Only code that calls the Kubernetes API for core-owned resources.                                               |
+| `internal/rollout/`    | Canary state machine and PromQL threshold evaluation.                                                                            |
+| `internal/scaling/`    | HPA generation and quota-capped scaling logic.                                                                                   |
+| `internal/registry/`   | MCP registrar, registry HTTP handler, TTL sweep.                                                                                 |
+| `internal/quota/`      | Quota arithmetic and in-flight reservation. Shared by webhook and TenantQuota reconciler.                                        |
 | `internal/webhook/`    | Validating and mutating admission webhooks. Lives here (not `api/`) to import `internal/quota` without creating an import cycle. |
-| `internal/metrics/`    | Shared Prometheus client plumbing used by rollout and scaling.                            |
+| `internal/metrics/`    | Shared Prometheus client plumbing used by rollout and scaling.                                                                   |
 
 ## Where the hard logic lives
 
-- **`internal/rollout/`** — never evaluate `rollback` thresholds against a sample smaller than `minRequestSample`. A 10%-weight canary at low traffic produces statistically meaningless error rates; gate on sample size first.
+- **`internal/rollout/`** — never evaluate `rollback` thresholds against a sample smaller than `minRequestSample`. A 10%-weight canary at low traffic produces statistically meaningless error rates; gate on sample size first. Canary steps must include at least one terminal `setWeight: 100` step for full promotion. Range query windows must format to canonical Prometheus syntax (`5m`, `1h`, `30s`, no trailing `0s`).
 - **`internal/quota/`** — two concurrent near-limit creates can individually pass a read-then-write quota check but combined exceed it. Use an in-flight reservation (short-lived in-memory map, keyed by tenant), not a naive status read.
 - **`internal/registry/`** — registration requires a successful MCP-level `initialize` handshake, not just Kubernetes readiness. Entries carry a TTL/heartbeat; ungraceful termination (OOM-kill, node failure) skips the deletion event path entirely, so don't rely on it.
+- **`internal/metrics/`** — all Prometheus HTTP responses must be read with `io.LimitReader` (1 MiB ceiling) to protect against memory exhaustion.
+- **`internal/controller/`** — reconcilers consume MCP registry operations via the `AgentRegistrar` interface (`Register`, `Deregister`, `Heartbeat`) for test isolation without polluting production structs.
 - Finalizer ordering: deregister from MCP _before_ the `Service` is garbage collected. Controller-runtime's foreground deletion via finalizer is the enforcement mechanism, not best-effort.
 - Quota reduction: lowering `TenantQuota` below current usage sets an `OverQuota` condition and blocks new creates/scale-ups. Never forcibly delete existing resources.
 

.coderabbit.yaml

@@ -65,7 +65,7 @@ reviews:
         - Finalizer constant must be `AgentDeploymentFinalizer` = `"agentrax.io/mcp-deregister"`.
           Never hardcode the string directly; always use the constant.
         - Validation markers (`+kubebuilder:validation:*`) on spec fields must
-          match the rules in docs/agentrax.md section 6.3. Pay special attention
+          match the rules in docs/ARCHITECTURE.md. Pay special attention
           to enum values for `spec.replicas.metric` and `spec.rollout.strategy`.
 
     # Reconciler — enforce controller-runtime patterns strictly.
@@ -222,7 +222,7 @@ reviews:
       instructions: |
         - Any change to a CRD field, architecture boundary, or package
           responsibility must be reflected here in the same PR.
-        - `docs/agentrax.md` is the source of truth for architecture decisions.
+        - `docs/ARCHITECTURE.md` is the source of truth for architecture decisions.
           Flag any PR that changes architecture without updating it.
 
   # ── Custom review instructions (global) ──────────────────────────────────

.github/workflows/ci.yml

@@ -49,10 +49,27 @@ jobs:
           name: coverage
           path: cover.out
 
+  helm-lint:
+    name: Helm Lint & Dry-run
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Helm
+        uses: azure/setup-helm@v4
+        with:
+          version: v3.14.0
+
+      - name: Lint Helm Chart
+        run: helm lint charts/agentrax/
+
+      - name: Template Helm Chart
+        run: helm template test charts/agentrax/ --debug
+
   build:
     name: Docker Build
     runs-on: ubuntu-latest
-    needs: [lint, test]
+    needs: [lint, test, helm-lint]
     steps:
       - uses: actions/checkout@v4
 
@@ -71,3 +88,27 @@ jobs:
           tags: ghcr.io/gitcommitankit/agentrax:${{ github.sha }}
           cache-from: type=gha
           cache-to: type=gha,mode=max
+
+  e2e:
+    name: End-to-End Tests (kind)
+    runs-on: ubuntu-latest
+    needs: [build]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: "1.23"
+          cache: true
+
+      - name: Create kind Cluster
+        uses: helm/[email protected]
+        with:
+          cluster_name: agentrax-e2e
+
+      - name: Install cluster dependencies
+        run: make deploy-deps
+
+      - name: Run E2E Tests
+        run: make test-e2e

.github/workflows/soak.yml

@@ -0,0 +1,35 @@
+name: Autoscaling Soak Test
+
+on:
+  workflow_dispatch:
+    inputs:
+      soak_duration_minutes:
+        description: "Duration to run autoscaling soak in minutes"
+        default: "10"
+        required: false
+
+jobs:
+  soak:
+    name: Autoscaling Soak Test (kind)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: "1.23"
+          cache: true
+
+      - name: Create kind cluster
+        uses: helm/[email protected]
+        with:
+          cluster_name: agentrax-soak
+
+      - name: Install cluster dependencies
+        run: make deploy-deps
+
+      - name: Run Autoscaling Soak Suite
+        run: make test-e2e-soak
+        env:
+          SOAK_DURATION_MINUTES: ${{ github.event.inputs.soak_duration_minutes }}

Makefile

@@ -68,6 +68,10 @@ test: manifests generate fmt vet envtest ## Run tests.
 test-e2e:
 	go test ./test/e2e/ -v -ginkgo.v
 
+.PHONY: test-e2e-soak
+test-e2e-soak: ## Run the long-running autoscaling soak test suite.
+	go test ./test/e2e/... -tags e2e -v --timeout 25m -run TestE2E
+
 .PHONY: lint
 lint: golangci-lint ## Run golangci-lint linter
 	$(GOLANGCI_LINT) run