feat(aws): shared cluster in previews-shared scopes secrets per env

CHANGELOG.md

@@ -1,3 +1,11 @@
+# 0.2.0
+
+Modules: `aws-ec2-vault-cluster` 0.2.0, `aws-vault-access` 0.3.0. Needs `vault-admin` 0.3.0 and a fresh node AMI. Nothing changes for a cluster outside `previews-shared`.
+
+* A cluster launched in the `previews-shared` env is shared: output `shared = true`, and every tenant belongs to an env. Tenant roles are `<env>.<tenant>`, secrets live under `kv/data/envs/<env>/customers/<tenant>/`, and an app's broker can only log in as its own env's tenants. The cluster function binds each app role to its env through an identity entity; provisioning does the same for tenant roles.
+* `aws-vault-access` passes the app's env to the function on a shared cluster and injects `VAULT_ENV` (empty on an unshared cluster).
+* `vault-utils tenants create|destroy` take `--env` (required on a shared cluster, refused elsewhere). `tenants list` shows the env. New `envs list` and `envs destroy <env> --yes [--purge-secrets]`. `vault-utils env` prints `SHARED_ENVS=true` for a shared cluster.
+
 # 0.1.2
 
 * Node AMI serves the Vault web UI at `$VAULT_ADDR/ui`. Needs a fresh AMI.

README.md

@@ -161,6 +161,9 @@ Run from `local/` unless noted. Destructive commands require `--yes`.
 | `docker compose run --rm -e VAULT_TOKEN=... bootstrap tenants list` | no | List tenants; flags one missing a reader or writer role |
 | `docker compose run --rm -e VAULT_TOKEN=... bootstrap tenants destroy <id> --yes` | yes (access) | Revoke access; secrets kept |
 | `docker compose run --rm -e VAULT_TOKEN=... bootstrap tenants destroy <id> --yes --purge-secrets` | yes | Also destroy secret versions |
+| `vault-utils tenants create <id> --env <env>` | no | Onboard a tenant for one env (shared cluster; see [Shared cluster](#shared-cluster)) |
+| `vault-utils envs list` | no | Envs with tenants (shared cluster) |
+| `vault-utils envs destroy <env> --yes [--purge-secrets]` | yes | Offboard every tenant of an env; purge also destroys its secrets |
 | `docker compose run --rm bootstrap snapshot take` | no | Raft snapshot plus SHA-256 |
 | `docker compose run --rm bootstrap snapshot restore <file> --yes` | yes | Replaces all Vault state |
 | `go test -short ./...` | no | Unit tests (tenant ID, policy lint, render, compose lint) — repo root |
@@ -187,7 +190,34 @@ kv/metadata/customers/{tenant_id}/*
 | Tenant policies | `tenant-reader`, `tenant-writer`, `tenant-database` | Static, written at bootstrap. The tenant comes from the login role name (`{{identity.entity.aliases.<accessor>.metadata.role_name}}`), so nothing is written per tenant. |
 | Broker policies | `apps-reader`, `apps-writer` | Read `role-id` and mint `secret-id` for any onboarded tenant on one mount. No KV, database, or sys access. |
 
-Tenant IDs: `^[a-z0-9]([a-z0-9-]{1,30}[a-z0-9])$` (3-32 characters). Rejected: `/`, `..`, `*`, `sys`, `data`, `root`, and similar reserved names.
+Tenant IDs: `^[a-z0-9]([a-z0-9-]{1,30}[a-z0-9])$` (3-32 characters). Rejected: `/`, `..`, `*`, `sys`, `data`, `root`, `envs`, and similar reserved names.
+
+### Shared cluster
+
+A cluster launched in the Nullstone env `previews-shared` serves every preview env. Its output `shared` is true, its nodes run with `SHARED_ENVS=true`, and every tenant belongs to one env:
+
+```
+kv/data/envs/{env}/customers/{tenant_id}/*
+kv/metadata/envs/{env}/customers/{tenant_id}/*
+```
+
+| Piece | Shared cluster |
+|---|---|
+| Tenant role name | `{env}.{tenant_id}` on the same two mounts. Env names use the tenant ID character class, so the `.` is unambiguous. |
+| Tenant policies | Env and tenant come from entity metadata (`{{identity.entity.metadata.env}}`, `...tenant`). `tenants create --env` writes the roles, one entity per role with that metadata, and an alias on the role's `role_id`. Nothing per env. |
+| Broker policies | `auth/approle-<kind>/role/{{identity.entity.metadata.env}}.*` with every parameter denied: an app mints logins only for its own env and cannot change a role. The env comes from the entity the cluster function writes on the app's auth role (see [App access](#app-access)). |
+| Provisioning, apps-auth | May create entities and aliases; `policies` and `disabled` are refused, groups and OIDC are denied. |
+
+A cluster anywhere else is unshared and renders the 0.1.x policies unchanged. Converting a cluster between the two is not supported: launch a new one.
+
+```bash
+vault-utils tenants create acme-corp --env pr-123
+vault-utils tenants list
+vault-utils envs list
+vault-utils envs destroy pr-123 --yes --purge-secrets
+```
+
+`--env` is required on a shared cluster and refused on an unshared one; `vault-utils env` exports `SHARED_ENVS` so the CLI knows which it is talking to. `envs destroy` offboards every tenant of the env; `--purge-secrets` also destroys `kv/metadata/envs/<env>` and needs break-glass.
 
 ```bash
 export VAULT_ADDR=http://127.0.0.1:8200
@@ -429,20 +459,40 @@ On boot, `vault-configure.service` runs after cloud-init, writes `/etc/vault.d/c
 
 Connect `vault` to the cluster. The app module must expose `security_group_id` and its IAM role name.
 
-`role_name` is optional. If empty, the Vault role is `<app-name>-<resource-suffix>`. `access` is `reader` (default) or `writer`; the capability binds the matching broker policy (`apps-reader` or `apps-writer`) and nothing else. The app receives `VAULT_ADDR`, `VAULT_ROLE`, `VAULT_TLS_SERVER_NAME` (empty unless the NLB terminates TLS), and `VAULT_TENANT_MOUNT`. During apply the capability calls the cluster function with `method = "aws"` and the app IAM role ARN; the function binds only that principal to that role. The same function serves GCP service accounts (`method = "gcp"`) for a future GCP cluster module.
+`role_name` is optional. If empty, the Vault role is `<app-name>-<resource-suffix>`. `access` is `reader` (default) or `writer`; the capability binds the matching broker policy (`apps-reader` or `apps-writer`) and nothing else. The app receives `VAULT_ADDR`, `VAULT_ROLE`, `VAULT_TLS_SERVER_NAME` (empty unless the NLB terminates TLS), `VAULT_TENANT_MOUNT`, and `VAULT_ENV`. During apply the capability calls the cluster function with `method = "aws"` and the app IAM role ARN; the function binds only that principal to that role. The same function serves GCP service accounts (`method = "gcp"`) for a future GCP cluster module.
+
+On a shared cluster (output `shared`), the capability also sends the app's Nullstone env name. The function then gives the auth role an identity entity with that env, which the broker policies template, and `VAULT_ENV` carries it to the app. On an unshared cluster `VAULT_ENV` is empty and the function never touches identity. Leave `role_name` empty on a shared cluster: a fixed name collides across preview envs and the function refuses the second binding.
 
-The app does not receive a Vault token. At startup it logs in with its IAM role, then for every request it logs in again as one tenant:
+The app does not receive a Vault token. At startup it logs in with its IAM role, then for every request it logs in again as one tenant. The role is `$TENANT`, or `$VAULT_ENV.$TENANT` when `VAULT_ENV` is set; the KV path is `customers/$TENANT/...`, or `envs/$VAULT_ENV/customers/$TENANT/...`:
 
 ```bash
 vault login -method=aws role="$VAULT_ROLE"
 
-ROLE_ID=$(vault read -field=role_id auth/$VAULT_TENANT_MOUNT/role/$TENANT/role-id)
-SECRET_ID=$(vault write -f -field=secret_id auth/$VAULT_TENANT_MOUNT/role/$TENANT/secret-id)
+ROLE="${VAULT_ENV:+$VAULT_ENV.}$TENANT"
+ROLE_ID=$(vault read -field=role_id auth/$VAULT_TENANT_MOUNT/role/$ROLE/role-id)
+SECRET_ID=$(vault write -f -field=secret_id auth/$VAULT_TENANT_MOUNT/role/$ROLE/secret-id)
 TENANT_TOKEN=$(vault write -field=token auth/$VAULT_TENANT_MOUNT/login role_id=$ROLE_ID secret_id=$SECRET_ID)
 ```
 
 The broker token reads no secrets. The tenant token is good for that tenant only, for 15 minutes. A login as any other role is denied. The operator token is not injected.
 
+### Shared cluster in Nullstone
+
+Launch the cluster block in the `previews-shared` env. In `.nullstone/previews.yml`, point each app's capability at it; everywhere else the app uses its own env's cluster:
+
+```yaml
+version: "0.1"
+
+apps:
+  api:
+    capabilities:
+      vault:
+        connections:
+          vault: previews-shared.vault
+```
+
+Create the preview env's tenants with `vault-utils tenants create <id> --env <env>` and remove them with `vault-utils envs destroy <env> --yes --purge-secrets` when the env goes away.
+
 ## Security
 
 - Host ports bind to `127.0.0.1` only

aws/aws-ec2-vault-cluster/admin.tf

@@ -1,6 +1,6 @@
 module "vault_admin" {
   source  = "api.nullstone.io/nullstone/aws-vault-admin/aws"
-  version = "~> 0.2.2"
+  version = "~> 0.3.0"
 
   name             = "${local.resource_name}-apps-auth"
   tags             = local.tags

aws/aws-ec2-vault-cluster/nullstone.tf

@@ -28,3 +28,10 @@ locals {
   block_name    = data.ns_workspace.this.block_name
   resource_name = "${data.ns_workspace.this.block_ref}-${random_string.resource_suffix.result}"
 }
+
+# A cluster launched in the previews-shared env serves every preview env and keeps their secrets apart
+# (envs/<env>/customers/<tenant>). Anywhere else it is one env's cluster with the unscoped layout.
+locals {
+  shared_env_name = "previews-shared"
+  shared          = data.ns_workspace.this.env_name == local.shared_env_name
+}

aws/aws-ec2-vault-cluster/outputs.tf

@@ -102,3 +102,8 @@ output "public_urls" {
   value       = local.public_urls
   description = "list(string) ||| A list of URLs accessible to the public"
 }
+
+output "shared" {
+  value       = local.shared
+  description = "bool ||| True when this cluster runs in the previews-shared env and scopes every tenant's secrets under envs/<env>/."
+}

aws/aws-ec2-vault-cluster/outputs.tftest.hcl

@@ -121,3 +121,28 @@ run "uses_https_with_a_subdomain" {
     error_message = "With TLS, vault.internal needs the user-facing name for certificate verification."
   }
 }
+
+run "is_not_shared_outside_previews_shared" {
+  command = plan
+
+  assert {
+    condition     = output.shared == false && strcontains(base64decode(aws_launch_template.this.user_data), "SHARED_ENVS=false")
+    error_message = "A cluster in any env but previews-shared is unshared and tells its nodes so."
+  }
+}
+
+run "is_shared_in_previews_shared" {
+  command = plan
+
+  override_data {
+    target = data.ns_workspace.this
+    values = {
+      env_name = "previews-shared"
+    }
+  }
+
+  assert {
+    condition     = output.shared == true && strcontains(base64decode(aws_launch_template.this.user_data), "SHARED_ENVS=true")
+    error_message = "A cluster in previews-shared is shared and its nodes scope tenants per env."
+  }
+}

aws/aws-ec2-vault-cluster/templates/user-data.sh.tpl

@@ -13,6 +13,7 @@ VAULT_APPS_AUTH_SECRET_ARN=${apps_auth_secret_arn}
 SNAPSHOT_BUCKET=${snapshot_bucket_name}
 SNAPSHOT_PREFIX=${snapshot_prefix}
 BACKUP_SCHEDULE="${backup_schedule}"
+SHARED_ENVS=${shared}
 AUDIT_LOG_PATH=/opt/vault/audit/audit.log
 AWS_REGION=${aws_region}
 UNSEAL_KMS_KEY_ARN=${unseal_kms_key_arn}

aws/aws-ec2-vault-cluster/user-data.tf

@@ -11,6 +11,7 @@ locals {
     snapshot_bucket_name    = local.snapshot_bucket_name
     snapshot_prefix         = local.snapshot_prefix
     backup_schedule         = var.backup_schedule
+    shared                  = local.shared
     vault_api_addr          = "https://${trimsuffix(aws_route53_record.vault.fqdn, ".")}:${local.vault_api_port}"
   })
 }

aws/aws-vault-access/auth.tf

@@ -2,7 +2,8 @@ data "aws_iam_role" "app" {
   name = var.app_metadata["role_name"]
 }
 
-# The cluster function writes one Vault auth role for this app's IAM role on the aws auth mount.
+# The cluster function writes one Vault auth role for this app's IAM role on the aws auth mount. On a shared
+# cluster it also binds the role to this env, so the app can only log in as this env's tenants.
 resource "aws_lambda_invocation" "vault_role" {
   function_name   = local.admin_function_name
   lifecycle_scope = "CRUD"
@@ -14,6 +15,7 @@ resource "aws_lambda_invocation" "vault_role" {
       method    = "aws"
       principal = data.aws_iam_role.app.arn
       policies  = ["apps-${var.access}"]
+      env       = local.env
     }
   })
 }

aws/aws-vault-access/outputs.tf

@@ -15,6 +15,10 @@ output "env" {
     {
       name  = "VAULT_TENANT_MOUNT"
       value = local.tenant_mount
+    },
+    {
+      name  = "VAULT_ENV"
+      value = local.env
     }
   ]
 }

aws/aws-vault-access/vault.tf

@@ -11,3 +11,9 @@ locals {
   vault_addr            = data.ns_connection.vault.outputs.vault_addr
   tls_server_name       = try(data.ns_connection.vault.outputs.tls_server_name, "")
 }
+
+# shared is true for a cluster in previews-shared (0.2.0+). Older clusters have no such output: not shared.
+locals {
+  shared = try(tobool(data.ns_connection.vault.outputs.shared), false)
+  env    = local.shared ? data.ns_workspace.this.env_name : ""
+}

aws/aws-vault-access/vault.tftest.hcl

@@ -1,4 +1,14 @@
-mock_provider "ns" {}
+mock_provider "ns" {
+  mock_data "ns_workspace" {
+    defaults = {
+      stack_name = "core"
+      env_name   = "pr-12"
+      block_ref  = "api"
+      block_name = "api"
+      aws_tags   = {}
+    }
+  }
+}
 mock_provider "aws" {}
 
 variables {
@@ -33,8 +43,8 @@ run "injects_the_capability_role" {
   }
 
   assert {
-    condition     = output.env == [{ name = "VAULT_ADDR", value = "http://vault.internal:8200" }, { name = "VAULT_ROLE", value = "billing" }, { name = "VAULT_TLS_SERVER_NAME", value = "" }, { name = "VAULT_TENANT_MOUNT", value = "approle-reader" }]
-    error_message = "The app must receive VAULT_ADDR, the capability role, an empty TLS name without TLS, and its tenant mount."
+    condition     = output.env == [{ name = "VAULT_ADDR", value = "http://vault.internal:8200" }, { name = "VAULT_ROLE", value = "billing" }, { name = "VAULT_TLS_SERVER_NAME", value = "" }, { name = "VAULT_TENANT_MOUNT", value = "approle-reader" }, { name = "VAULT_ENV", value = "" }]
+    error_message = "The app must receive VAULT_ADDR, the capability role, an empty TLS name without TLS, its tenant mount, and an empty VAULT_ENV on a cluster without a shared output."
   }
 
   assert {
@@ -47,12 +57,63 @@ run "injects_the_capability_role" {
     error_message = "A reader app gets only the apps-reader broker policy."
   }
 
+  assert {
+    condition     = jsondecode(aws_lambda_invocation.vault_role.input).data.env == ""
+    error_message = "Without a shared output the function gets no env."
+  }
+
   assert {
     condition     = aws_security_group_rule.app_to_vault.from_port == 8200 && aws_security_group_rule.vault_from_app.source_security_group_id == "sg-app"
     error_message = "Security group rules must use the cluster API port and the app security group."
   }
 }
 
+run "shared_cluster_binds_the_env" {
+  command = plan
+
+  override_data {
+    target = data.ns_connection.vault
+    values = {
+      outputs = {
+        vault_fqdn            = "vault.internal"
+        vault_addr            = "http://vault.internal:8200"
+        nlb_security_group_id = "sg-nlb"
+        vault_api_port        = "8200"
+        admin_function_name   = "vault-apps-auth"
+        shared                = "true"
+      }
+    }
+  }
+
+  assert {
+    condition     = jsondecode(aws_lambda_invocation.vault_role.input).data.env == "pr-12" && output.env[4] == { name = "VAULT_ENV", value = "pr-12" }
+    error_message = "On a shared cluster the function binds the app to its env and the app receives VAULT_ENV."
+  }
+}
+
+run "unshared_cluster_passes_no_env" {
+  command = plan
+
+  override_data {
+    target = data.ns_connection.vault
+    values = {
+      outputs = {
+        vault_fqdn            = "vault.internal"
+        vault_addr            = "http://vault.internal:8200"
+        nlb_security_group_id = "sg-nlb"
+        vault_api_port        = "8200"
+        admin_function_name   = "vault-apps-auth"
+        shared                = "false"
+      }
+    }
+  }
+
+  assert {
+    condition     = jsondecode(aws_lambda_invocation.vault_role.input).data.env == "" && output.env[4] == { name = "VAULT_ENV", value = "" }
+    error_message = "An unshared cluster must behave as before: no env for the function, empty VAULT_ENV."
+  }
+}
+
 run "uses_the_cluster_port" {
   command = plan
 
@@ -174,6 +235,7 @@ run "uses_the_cluster_addr_and_tls_name" {
       { name = "VAULT_ROLE", value = "billing" },
       { name = "VAULT_TLS_SERVER_NAME", value = "vault.acme.example.com" },
       { name = "VAULT_TENANT_MOUNT", value = "approle-reader" },
+      { name = "VAULT_ENV", value = "" },
     ]
     error_message = "A TLS cluster must give the app its https address and the certificate name."
   }