From fbd7657ce4b62bc8f0014274e45971cdec402da2 Mon Sep 17 00:00:00 2001 From: Aaron Trowbridge Date: Wed, 4 Nov 2026 09:19:35 -0500 Subject: [PATCH] =?UTF-8?q?docs(adr):=200041=20Armonia=20team=20deployment?= =?UTF-8?q?=20=E2=80=94=20central=20hub,=20person=20principals,=20executio?= =?UTF-8?q?n=20sites,=20spend=20plane=20(#1873)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...041-armonia-team-deployment-central-hub.md | 214 ++++++++++++++++++ 1 file changed, 214 insertions(+) create mode 100644 docs/adr/0041-armonia-team-deployment-central-hub.md diff --git a/docs/adr/0041-armonia-team-deployment-central-hub.md b/docs/adr/0041-armonia-team-deployment-central-hub.md new file mode 100644 index 000000000..3bcf79530 --- /dev/null +++ b/docs/adr/0041-armonia-team-deployment-central-hub.md @@ -0,0 +1,214 @@ +--- +status: proposed +date: 2026-11-04 +deciders: Aaron Trowbridge +tags: [armonia, central-hub, identity, lab, spend, ec2, modal] +supersedes: [] +related: ["#1872", "#1873", "#1874", "#1875", "#1876", "#1880", "#1882", "#1887", "#1958", "ADR 0031", "ADR 0032", "ADR 0033", "ADR 0034", "ADR 0037", "ADR 0039", "aws-infra#360"] +--- + +# ADR 0041: Armonia team deployment — central hub, person principals, execution sites, spend plane + +## Context + +Amicode today is single-tenant: one person's engine, one shared password or open tunnel, +no per-person identity, no membership boundary, budgets that live in prompts, and compute +reached through per-machine paths. Ten Harmoniqs researchers cannot share one engine — see +each other's agents, delegate between them, and run unattended campaigns across HPC and lab +machines — without trusting every caller with everything. + +Parent design-of-record: #1872 ("Armonia: one shared coordination service, people as +principals, distributed execution sites"). This ADR records the *central hub* slice of that +design: how Harmoniqs moves onto one central coordination service on EC2, with Modal and +AWS as execution sites, Bedrock reached through an instance role, and spend tracked and capped +per Lab and per person. It is the model for what labs get later; the Lab is the tenant +primitive (members, budget, compute lane, data boundary). + +Ground truth (verified 2026-11-04): +- Today's hub is `erlich`, Aaron's workstation (i9-12900KS, 24 threads, 62 GB): frontdoor on + 4096 (loopback only after #1957), three engine shards on 4095/4195/4295 sharing one DB, + under user systemd. No per-user identity in frontdoor logs; no evidence anyone but Aaron + connects. +- `aws-infra` has no hub module. Closest template: `cpu_dev_box` (c6i.2xlarge, private + subnet, SSM/EIC, persistent EBS + AWS Backup, staging only, idle-stop). Bedrock + least-privilege template: `app_ai_bedrock_gateway`. `budget_alarm` auto hard-cap attaches + Deny + SCP when net spend passes $50 (prod) / $25 (staging), which would deny Bedrock for + everyone — so the hub goes in an account/tag scope whose budget is sized for it, coordinated + with the cloud-launch session's dedicated compute account. +- Compute accounting gaps (`packages/amico-run`): `scheduler.ts` is a serial in-process queue; + `run.lock` and `ReceiptRecord` carry no principal or Lab; local and Modal runs record no + cost; there is no Modal executor; the warrant has no dollar field. `session.cost` in the + chat DB already gives Bedrock spend per session. +- Host spike (#2114): one host is fine up to ~12 concurrent sessions; at 36 latency degrades + with uptime (#2115). Size and restart-policy the hub accordingly; do not promise more than + ~12–20 concurrent sessions per hub yet. +- Org access was locked on 2026-11-02 to four people (aarontrowbridge, jacktrnr, bc037, + loganwillits) for proprietary software. The hub must not widen it. + +## Decision + +### 1. Topology: EC2 hub, execution sites elsewhere + +**Chosen: a central EC2 coordination host with Modal and AWS as execution sites.** Rejected: +(1) Modal-as-hub — Modal is a compute plane, not a coordination authority; it has no durable +session store, no per-person identity, and no place for a frontdoor. (2) Federated personal +engines (extend Fleet Studio's peer model across people) — preserves local machines but makes +shared state, access enforcement and collaboration depend on every peer being reachable. +(3) One large shared server running every workload — simplest, but experiments and failures +disrupt everyone and isolation is weak. + +The coordination authority is a single process on EC2 (the "hub"). Scientific compute, HPC +and lab hardware run at **execution sites** that dial *out* to the hub and lease durable jobs; +sites never accept inbound connections. This matches ADR 0037 §1 (sites are identities by tag, +never by person) and the #1882 execution-site registry. + +### 2. Reconciling #1887 and aws-infra#360 + +These are **complementary, not competing**: +- **#1887 (this ADR's hub slice)** provisions the EC2 host, its instance profile granting + Bedrock invoke only, persistent EBS + AWS Backup, SSM access, Tailscale, a dedicated system + user, tags for budgets, no inbound from the internet. It is the *host*. +- **aws-infra#360 (Modal OIDC role)** lets Modal workers assume an AWS role via OIDC to reach + Bedrock/S3 *without long-lived keys*. It is one *execution site* reaching into AWS. + +The hub host never holds AWS keys for Bedrock; Bedrock is reached through the instance profile +(#1887) on the hub and through OIDC-assumed roles on Modal sites (#360). Both are +least-privilege and both are tag-scoped for budgets. Neither widens org access. + +### 3. Person principal + +Every request resolves to a **Principal** (ADR 0037, #1874). A Person is one kind of +Principal, alongside Site and Machine. The frontdoor authenticates each person against a +deployment-local people registry (#1874) and forwards a *signed* principal; engines reject any +principal they cannot verify, so bypassing the edge grants nothing. Unauthenticated requests +are rejected. Machine credentials never stand in for a person. + +Identity transport: Tailscale `serve` injects `Tailscale-User-Login` / `Tailscale-User-Name` +headers; the frontdoor maps them to the person principal, trusting them only on the loopback +hop from `tailscaled` (#1958). OIDC is the later, transport-independent path. + +### 4. Lab = environment membership (not editor state) + +Research Environments become a server-side membership record: environment × person × role, +with project restriction lists that can only narrow access (#1876). No editor workspace +folders or process env vars in production (ADR 0021). Membership is the knowledge and +visibility boundary; projects may restrict further, never broaden. The Lab is the tenant +primitive: members, budget, compute lane, data boundary. + +### 5. Spend plane + +- **Pool unit is dollars, one pool** (tokens + hosted solves), reusing the nano-USD + reservation ledger pattern from `app-harmoniqs-ai` (`inference_accounts`, + `inference_reservations`). Generalize `budget#{campaign}` → `budget#{submitter}` in + `solves_api`; settle from `solves_completion_ping` (cloud-launch sibling campaign). +- **Cost attribution:** add principal and Lab to `run.lock` and `ReceiptRecord`; cost every + local run from CPU-seconds; add a `modal` executor under the gate; roll Bedrock cost up from + `session.cost` per person and Lab. Label every number "list price, not invoice" until + reconciled against invoices. +- **Budgets are server-side and fail closed** (#1880): per-Lab and per-person dollar budgets + checked before every provider turn and every launch. Budgets are an input file that only a + human edits, seeded from Aaron's credit numbers (~$14k AWS, ~$5k Modal). The model never + enforces its own budget. +- **Budget scoping vs. the hard cap:** the hub lives in an account/tag scope sized for it so + the $50/$25 `budget_alarm` auto hard-cap does not deny Bedrock for everyone. Coordinate + with the cloud-launch session's dedicated compute account. + +### 6. Access (do not widen) + +Org access stays locked to the four people. Hub access starts with exactly those four plus a +documented way to add a person (onboarding runbook). No change to org membership, 2FA, or +proprietary-repo grants is made by this campaign. + +### 7. Capacity and restart policy + +Size the hub for ~12–20 concurrent sessions (host spike, #2114/#2115). Restart policy must +account for the latency-degrades-with-uptime finding: schedule periodic restarts inside +maintenance windows, never during live solves (A0 solve-aware restart guard). Do not promise +more. + +### 8. Relationship to other ADRs + +- **ADR 0037 (trust boundaries)** — this ADR assumes it: Tailscale transport, sovereign + client, sandboxed agents, `armonia init`/`doctor`. Identity (#1874/#1958) rides the + frontdoor; authorization stays in Armonia, not in Tailscale. +- **ADR 0031–0034 (fleet: studio, peer trust, per-session SSE fanout, base peer control)** — + person-scoped grants reuse Fleet Studio's observe/control grant model with person principals; + machine-scoped grants are unchanged. No new routing authority beyond #1760. +- **ADR 0039 (host)** — host spike results set the hub's size/restart policy; the v2 host + runs on the hub. +- **ADR 0007 (amicissimo)** — signed artifacts/entitlements are for paying labs later; not + needed for internal hub. + +## Consequences + +- A request without a valid person or site credential is rejected at every entry point; every + accepted request carries a resolved principal. +- Every session and admitted prompt records the authenticated person; attribution cannot be set + or changed through client-supplied metadata. +- A person sees sessions, events, work requests, artifacts only for environments they belong to. +- Campaigns stop issuing provider turns, tool calls, job submissions when budget/stopping + condition is reached, regardless of what the model requests. +- Delegated work requests draw from the originator's budget, carry no broader permissions, and + survive a coordinator restart. +- Execution sites dial out; reconnect reconciles rather than resubmits. +- Hardware authority stays lab-side (Strumento). +- Training eligibility is opt-in per environment. + +## Slices (this ADR is the design-of-record for) + +- #1873 (this ADR + glossary) — done when merged. +- #1874 identity (frontdoor auth) + #1958 (Tailscale serve + header mapping). +- #1875 attribution (owner/author). +- #1876 Lab membership store. +- #1880 campaign authorization / budgets. +- #1887 hub host Terraform module + instance-role Bedrock. +- Cost ledger (run.lock/ReceiptRecord principal+Lab, local CPU-sec cost, modal executor, + Bedrock rollup from session.cost) — filed as part of this campaign. +- Onboarding runbook (Jackson, Brad) — filed as part of this campaign. +- #1888 end-to-end acceptance test (needs #1878, #1880, #1881, #1883, #1884, #1887). + +## Glossary + +- **Person** — a human principal in a deployment's people registry. Has an id, display name, + credential. Never inferred from a machine or site credential. +- **Principal** — the resolved caller of a request: Person, Site, or Machine. Immutable per + credential. *Avoid:* using "user" or "account" as a synonym; those imply billing identity. +- **Team deployment** — one coordination authority (the hub) plus its outbound execution sites. + Exactly one coordination authority. *Avoid:* "cluster", "fleet" (fleet = machines, not people). +- **Membership** — environment × person × role record, server-side. *Avoid:* "access list", + "workspace membership" (editor state is not membership). +- **Session owner** — the person who created the session; immutable. *Avoid:* "creator" as a + free-text field. +- **Collaborator** — a person invited by the owner to control a session. *Avoid:* "shared + session" without an explicit invite. +- **Execution site** — a registered compute/lab/batch node that dials out and leases jobs. + Identity by tag. *Avoid:* "worker", "agent" (an agent runs *on* a site). +- **Job** — a durable work item leased to a site; reconnect reconciles, never resubmits. + *Avoid:* "task" (task = delegation record, #1881). +- **Work request** — a durable delegation record (deliverable, acceptance checks, budget slice, + permission set, assignee). *Avoid:* "subtask", "child session". +- **Campaign authorization** — owner, scope, budget, tool allowlist, device access, stopping + conditions, consumed totals. Enforced by the server. *Avoid:* "warrant" alone (a warrant is + the approval artifact; authorization is the live record). +- **Artifact reference** — content-addressed pointer recording custody site, provenance, access + policy, verification status. *Avoid:* "file", "result" (those lose custody/provenance). +- **Training eligibility** — per-environment opt-in for use of trajectories/verdicts in + post-training. Collaboration access ≠ training eligibility. *Avoid:* "data sharing". + +### How these differ from fleet terms +- **Machine** (fleet) — a device. A person may own many machines; a machine is never a person. +- **Peer studio** (fleet) — a machine running its own engine. In a team deployment, peers are + execution sites, not coordination authorities. +- **Observe / Control** (fleet) — machine-scoped grants; unchanged. Team deployment adds + person-scoped grants on top. +- **Canonical Server** (fleet) — the single-writer session store; in a team deployment the hub + is the canonical server. +- **Submitter** — who submitted a solve; becomes a person in team deployments. +- **Lifecycle-admin authority** — machine provisioning rights; separate from person membership. + +## Reversibility + +Reversible: revert the ADR and the slices; the hub can be decommissioned and researchers +return to local engines (onboarding runbook includes rollback). Irreversible only at the +human gates: org access changes, account creation, key minting, partner agreements. +glossary.md