Casa testing , campaigns and governance

.forgejo/README.md

@@ -0,0 +1,130 @@
+# Forgejo Actions runbook
+
+`workflows/deploy-casa.yaml` builds `Dockerfile.local` with the dev license keyring and swaps the image on the
+`operator` release at `operator.<devhost>`. `workflows/pr.yaml` is the required check on `main`: it builds the
+same arm64 image without pushing it, then runs the `e2e/` Playwright suites against it (`scripts/ci/e2e.sh`).
+Runner egress isn't restricted (jobs reach the internet through OrbStack's NAT), but everything they pull comes
+from Forgejo mirrors anyway, for pinning and speed. These commands refresh the mirrors from a workstation.
+
+| Workflow / job | Runner label | Reaches |
+| --- | --- | --- |
+| `deploy-casa` / `build` | `docker` | Forgejo registry and mirrors; `deb.debian.org`, `registry.npmjs.org`, `index.crates.io`, `static.crates.io` at build time |
+| `deploy-casa` / `deploy` | `helmfile` | Forgejo registry, cluster API |
+| `pr` / `arm64` | `docker` | as `build`, plus `webhook.operator.untra.casa` for the license suite |
+
+Common shell setup:
+
+```sh
+FORGEJO=https://forgejo.<devhost>
+REPO=untra-operator/operator
+MIRROR=forgejo.<devhost>/untra-operator/mirror
+CHARTS=forgejo.<devhost>/untra-operator/charts
+FORGEJO_TOKEN=...   # scopes: write:package, write:repository, write:organization
+api() { curl -fsS -H "Authorization: token $FORGEJO_TOKEN" -H 'Content-Type: application/json' "$@"; }
+```
+
+## License secrets
+
+`OPERATOR_LICENSE_ROOT_KEYS` must carry the **dev** root only; the prod root belongs to GitHub `untra/operator`.
+
+```sh
+api -X PUT "$FORGEJO/api/v1/repos/$REPO/actions/secrets/OPERATOR_LICENSE_ROOT_KEYS" \
+  -d "$(jq -cn --arg k "$(cat ~/untra-keys/keys/untra-dev-2026.pub)" '{data: ({"untra-dev-2026": $k} | tojson)}')"
+api -X PUT "$FORGEJO/api/v1/repos/$REPO/actions/secrets/OPERATOR_LICENSE_ISSUER" -d '{"data":"operator-licensing"}'
+api -X PUT "$FORGEJO/api/v1/repos/$REPO/actions/secrets/OPERATOR_PURCHASE_URL" -d '{"data":"https://app.operator.untra.casa/"}'
+api "$FORGEJO/api/v1/repos/$REPO/actions/secrets" | jq -r '.[].name'   # names only; values are write-only
+```
+
+## Base image mirror
+
+Re-run whenever a `FROM` in `Dockerfile.local` or `e2e/Dockerfile` changes; the Playwright tag tracks
+`@playwright/test` in `e2e/package.json`. Digest-pinned refs copy whole, so the pin still resolves; the rest copy arm64 only. `--jobs 1` keeps large layer uploads under the ingress timeout.
+
+```sh
+: "${MIRROR:?run the common shell setup first}"
+crane auth login forgejo.<devhost> -u <registry-user> --password-stdin <<<"$FORGEJO_TOKEN"
+sed -n 's/^FROM \${BASE_REGISTRY}\([^ ]*\).*/\1/p' Dockerfile.local e2e/Dockerfile | sort -u | while read -r ref; do
+  case $ref in
+    *@*) crane copy --jobs 1 "$ref" "$MIRROR/${ref%@*}" ;;
+    *) crane copy --jobs 1 --platform linux/arm64 "$ref" "$MIRROR/$ref" ;;
+  esac
+done
+crane digest "$MIRROR/debian:trixie-slim"   # must equal the sha256 pinned in Dockerfile.local
+```
+
+## Chart mirror
+
+Mirror every chart version that helmfile deploys; `deploy` upgrades at the release's current chart version.
+
+```sh
+: "${CHARTS:?run the common shell setup first}"
+v=$(helm get metadata operator -n operator | awk '/^VERSION:/ {print $2}')
+helm pull oci://ghcr.io/untra/charts/operator --version "$v" -d /tmp
+helm registry login forgejo.<devhost> -u <registry-user> --password-stdin <<<"$FORGEJO_TOKEN"
+helm push "/tmp/operator-$v.tgz" "oci://$CHARTS"
+helm show chart "oci://$CHARTS/operator" --version "$v"
+```
+
+## Checkout action mirror
+
+A pull mirror refreshes from the Forgejo server, so only the server needs egress to `code.forgejo.org`.
+The `actions` org must be public so runners can fetch it without credentials.
+
+```sh
+api -X POST "$FORGEJO/api/v1/orgs" -d '{"username":"actions","visibility":"public"}'   # once
+api -X POST "$FORGEJO/api/v1/repos/migrate" -d '{"clone_addr":"https://code.forgejo.org/actions/checkout",
+  "repo_owner":"actions","repo_name":"checkout","mirror":true,"service":"git"}'          # once
+api -X POST "$FORGEJO/api/v1/repos/actions/checkout/mirror-sync"                        # refresh
+```
+
+## E2E suites
+
+`pr` runs both projects in `e2e/playwright.config.ts`: `workspace` (admin setup and onboarding on a fresh server)
+and `license` (a test-mode key from the platform webhook's `GET /e2e/license`, applied to that server's
+configuration). The license suite skips while `OPERATOR_E2E_LICENSE_TOKEN` is unset, so leave it unset until the
+casa webhook serves the route. Its value is the casa webhook's `E2E_LICENSE_TOKEN`.
+
+```sh
+api -X PUT "$FORGEJO/api/v1/repos/$REPO/actions/secrets/OPERATOR_E2E_LICENSE_TOKEN" \
+  -d "$(jq -cn --arg v "$E2E_LICENSE_TOKEN" '{data: $v}')"
+api -X PUT "$FORGEJO/api/v1/repos/$REPO/actions/secrets/OPERATOR_E2E_CA_CERT" \
+  -d "$(jq -cn --arg v "$(cat casa-root-ca.pem)" '{data: $v}')"   # only if the runner does not already trust the casa CA
+```
+
+Run the same thing locally with `make e2e`; pass the two `OPERATOR_E2E_*` variables to include the license suite.
+
+## Required check on main
+
+The status context is `<workflow name> / <job name> (<event>)`. Confirm it on the first PR's commit status
+before requiring it, because a misspelled context blocks every merge.
+
+```sh
+api "$FORGEJO/api/v1/repos/$REPO/commits/<pr-head-sha>/statuses" | jq -r '.[].context'   # expect: pr / arm64 (pull_request)
+api -X POST "$FORGEJO/api/v1/repos/$REPO/branch_protections" -d '{"rule_name":"main",
+  "enable_status_check":true,"status_check_contexts":["pr / arm64 (pull_request)"]}'
+# rule already exists:
+api -X PATCH "$FORGEJO/api/v1/repos/$REPO/branch_protections/main" -d '{"enable_status_check":true,
+  "status_check_contexts":["pr / arm64 (pull_request)"]}'
+```
+
+## Dispatch, status and rollback
+
+```sh
+api -X POST "$FORGEJO/api/v1/repos/$REPO/actions/workflows/deploy-casa.yaml/dispatches" -d '{"ref":"main"}'
+api "$FORGEJO/api/v1/repos/$REPO/actions/runs?limit=1" | jq '.workflow_runs[0] | {id, status, html_url}'
+helm history operator -n operator
+helm rollback operator <revision> -n operator
+```
+
+Job logs are not readable through the API; open the run in the web UI.
+
+## Housekeeping
+
+```sh
+api "$FORGEJO/api/v1/repos/$REPO/actions/runners" | jq -r '.[] | "\(.name) \(.status) \(.labels | join(","))"'
+api "$FORGEJO/api/v1/packages/untra-operator?type=container&q=operator" | jq -r '.[] | "\(.name):\(.version)"'
+api -X DELETE "$FORGEJO/api/v1/packages/untra-operator/container/operator/<old-sha>"
+docker buildx prune --filter until=168h --force   # on the docker runner host
+```
+
+`buildcache` is overwritten by every build; delete it to force a cold build.

.forgejo/workflows/deploy-casa.yaml

@@ -0,0 +1,108 @@
+name: deploy-casa
+on:
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: Branch, tag or sha to deploy (blank deploys the dispatched branch)
+        required: false
+        default: ""
+
+# Builds Dockerfile.local with the dev license keyring and rolls it out to operator.kube.untra.casa.
+#   build   runs-on: docker    native arm64 buildx push to the Forgejo registry, tagged by commit sha
+#   deploy  runs-on: helmfile  swaps the image on the helmfile-owned release, at its deployed chart version and values
+# Base images, the chart and the checkout action come from Forgejo mirrors; see .forgejo/README.md.
+#
+# Variables: REGISTRY_IMAGE
+# Secrets:   REGISTRY_USERNAME, REGISTRY_TOKEN, KUBECONFIG,
+#            OPERATOR_LICENSE_ROOT_KEYS (dev root only), OPERATOR_LICENSE_ISSUER, OPERATOR_PURCHASE_URL
+
+concurrency:
+  group: deploy-casa
+  cancel-in-progress: false
+
+jobs:
+  build:
+    runs-on: docker
+    outputs:
+      tag: ${{ steps.meta.outputs.tag }}
+      digest: ${{ steps.push.outputs.digest }}
+    steps:
+      - name: License secrets present
+        run: |
+          missing=0
+          for name in OPERATOR_LICENSE_ROOT_KEYS OPERATOR_LICENSE_ISSUER; do
+            if [ -z "$(printenv "$name")" ]; then
+              echo "::error::repository secret $name is unset or empty"
+              missing=1
+            fi
+          done
+          exit "$missing"
+        env:
+          OPERATOR_LICENSE_ROOT_KEYS: ${{ secrets.OPERATOR_LICENSE_ROOT_KEYS }}
+          OPERATOR_LICENSE_ISSUER: ${{ secrets.OPERATOR_LICENSE_ISSUER }}
+
+      - uses: https://forgejo.kube.untra.casa/actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - id: meta
+        run: echo "tag=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT"
+
+      - name: Registry login
+        run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login "${REGISTRY_IMAGE%%/*}" -u "${{ secrets.REGISTRY_USERNAME }}" --password-stdin
+        env:
+          REGISTRY_IMAGE: ${{ vars.REGISTRY_IMAGE }}
+
+      # Default docker driver: a docker-container builder would not trust the home CA.
+      # OPERATOR_RELEASE=1 makes build.rs fail on an empty keyring or issuer instead of shipping one that rejects every license.
+      - name: Build and push
+        id: push
+        run: |
+          docker buildx build \
+            --file Dockerfile.local \
+            --build-arg BASE_REGISTRY="${REGISTRY_IMAGE%/*}/mirror/" \
+            --build-arg OPERATOR_RELEASE=1 \
+            --build-arg OPERATOR_LICENSE_ISSUER="$OPERATOR_LICENSE_ISSUER" \
+            --build-arg OPERATOR_LICENSE_ROOT_KEYS="$OPERATOR_LICENSE_ROOT_KEYS" \
+            --build-arg OPERATOR_PURCHASE_URL="$OPERATOR_PURCHASE_URL" \
+            --cache-from "type=registry,ref=$REGISTRY_IMAGE:buildcache" \
+            --cache-to "type=registry,ref=$REGISTRY_IMAGE:buildcache,mode=max" \
+            --tag "$REGISTRY_IMAGE:${{ steps.meta.outputs.tag }}" \
+            --metadata-file meta.json \
+            --push \
+            .
+          echo "digest=$(node -p "require('./meta.json')['containerimage.digest']")" >> "$GITHUB_OUTPUT"
+        env:
+          REGISTRY_IMAGE: ${{ vars.REGISTRY_IMAGE }}
+          OPERATOR_LICENSE_ISSUER: ${{ secrets.OPERATOR_LICENSE_ISSUER }}
+          OPERATOR_LICENSE_ROOT_KEYS: ${{ secrets.OPERATOR_LICENSE_ROOT_KEYS }}
+          OPERATOR_PURCHASE_URL: ${{ secrets.OPERATOR_PURCHASE_URL }}
+
+  deploy:
+    needs: build
+    runs-on: helmfile
+    steps:
+      - name: Kubeconfig
+        run: |
+          mkdir -p ~/.kube
+          echo "${{ secrets.KUBECONFIG }}" > ~/.kube/config
+          chmod 600 ~/.kube/config
+
+      - name: Registry login
+        run: echo "${{ secrets.REGISTRY_TOKEN }}" | helm registry login "${REGISTRY_IMAGE%%/*}" -u "${{ secrets.REGISTRY_USERNAME }}" --password-stdin
+        env:
+          REGISTRY_IMAGE: ${{ vars.REGISTRY_IMAGE }}
+
+      # Helmfile owns the release: keep its chart version and values, and swap only the image.
+      # The tag carries the digest: a re-run on the same commit pushes a new image under the same
+      # tag, and a bare tag would leave the StatefulSet unchanged (and IfNotPresent on the old one).
+      - name: Helm upgrade
+        run: |
+          version=$(helm get metadata operator -n operator | awk '/^VERSION:/ {print $2}')
+          helm upgrade operator "oci://${REGISTRY_IMAGE%/*}/charts/operator" --version "$version" -n operator \
+            --reset-then-reuse-values \
+            --set image.repository="$REGISTRY_IMAGE" \
+            --set image.tag="${{ needs.build.outputs.tag }}@${{ needs.build.outputs.digest }}" \
+            --wait
+        env:
+          REGISTRY_IMAGE: ${{ vars.REGISTRY_IMAGE }}

.forgejo/workflows/pr.yaml

@@ -0,0 +1,80 @@
+name: pr
+on:
+  pull_request:
+    branches: [main]
+
+# Required check on main: builds the arm64 Dockerfile.local image this PR would deploy, then runs e2e/ against it.
+#   arm64  runs-on: docker  native arm64 buildx --load (nothing pushed), then scripts/ci/e2e.sh
+# Lint and unit tests stay on GitHub build.yaml; this gate proves the casa image boots and onboards.
+#
+# Variables: REGISTRY_IMAGE
+# Secrets:   REGISTRY_USERNAME, REGISTRY_TOKEN,
+#            OPERATOR_LICENSE_ROOT_KEYS (dev root only), OPERATOR_LICENSE_ISSUER, OPERATOR_PURCHASE_URL,
+#            OPERATOR_E2E_LICENSE_TOKEN (optional; the license suite skips without it), OPERATOR_E2E_CA_CERT (optional)
+
+concurrency:
+  group: pr-${{ github.head_ref }}
+  cancel-in-progress: true
+
+jobs:
+  arm64:
+    runs-on: docker
+    steps:
+      - name: License secrets present
+        run: |
+          missing=0
+          for name in OPERATOR_LICENSE_ROOT_KEYS OPERATOR_LICENSE_ISSUER; do
+            if [ -z "$(printenv "$name")" ]; then
+              echo "::error::repository secret $name is unset or empty"
+              missing=1
+            fi
+          done
+          exit "$missing"
+        env:
+          OPERATOR_LICENSE_ROOT_KEYS: ${{ secrets.OPERATOR_LICENSE_ROOT_KEYS }}
+          OPERATOR_LICENSE_ISSUER: ${{ secrets.OPERATOR_LICENSE_ISSUER }}
+
+      - uses: https://forgejo.kube.untra.casa/actions/checkout@v4
+
+      - id: meta
+        run: echo "tag=pr-$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT"
+
+      - name: Registry login
+        run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login "${REGISTRY_IMAGE%%/*}" -u "${{ secrets.REGISTRY_USERNAME }}" --password-stdin
+        env:
+          REGISTRY_IMAGE: ${{ vars.REGISTRY_IMAGE }}
+
+      # Reads main's cache but never writes it: PR layers stay off the registry and clear of the ingress upload limit.
+      - name: Build
+        run: |
+          docker buildx build \
+            --file Dockerfile.local \
+            --build-arg BASE_REGISTRY="${REGISTRY_IMAGE%/*}/mirror/" \
+            --build-arg OPERATOR_RELEASE=1 \
+            --build-arg OPERATOR_LICENSE_ISSUER="$OPERATOR_LICENSE_ISSUER" \
+            --build-arg OPERATOR_LICENSE_ROOT_KEYS="$OPERATOR_LICENSE_ROOT_KEYS" \
+            --build-arg OPERATOR_PURCHASE_URL="$OPERATOR_PURCHASE_URL" \
+            --cache-from "type=registry,ref=$REGISTRY_IMAGE:buildcache" \
+            --tag "operator:${{ steps.meta.outputs.tag }}" \
+            --load \
+            .
+        env:
+          REGISTRY_IMAGE: ${{ vars.REGISTRY_IMAGE }}
+          OPERATOR_LICENSE_ISSUER: ${{ secrets.OPERATOR_LICENSE_ISSUER }}
+          OPERATOR_LICENSE_ROOT_KEYS: ${{ secrets.OPERATOR_LICENSE_ROOT_KEYS }}
+          OPERATOR_PURCHASE_URL: ${{ secrets.OPERATOR_PURCHASE_URL }}
+
+      - name: E2E
+        run: |
+          export BASE_REGISTRY="${REGISTRY_IMAGE%/*}/mirror/"
+          scripts/ci/e2e.sh "operator:${{ steps.meta.outputs.tag }}"
+        env:
+          REGISTRY_IMAGE: ${{ vars.REGISTRY_IMAGE }}
+          E2E_RUN_ID: ${{ github.run_id }}
+          OPERATOR_E2E_ISSUER_URL: https://webhook.operator.untra.casa
+          OPERATOR_E2E_LICENSE_TOKEN: ${{ secrets.OPERATOR_E2E_LICENSE_TOKEN }}
+          OPERATOR_E2E_CA_CERT: ${{ secrets.OPERATOR_E2E_CA_CERT }}
+
+      - name: Remove image
+        if: always()
+        run: docker rmi "operator:${{ steps.meta.outputs.tag }}" || true