K8s driver (ADR #63 slice 3) — sub-slice tracking

[brettchien]

Context

ADR #63 (agent-deployment-templates, Accepted) already scoped slice 3 = K8s driver as the second provider behind the same RuntimeDriver-shaped seam as ECS (slice 2, PR #65). This issue tracks that slice, broken into independently-shippable sub-slices, per Brett's ask (orbstack-local k8s cluster as a Studio-managed target).

Full deployment scope confirmed: compose → bundle → k8s apply, on par with ECS provision — not a read-only/observe-first MVP.

Relevant ADRs: deployment-control-plane (ADR-2, §6-state ⇄ k8s mapping), agent-lifecycle (ADR-1), runtime-identity-context (ADR-19, FleetBinding k8s fields), agent-deployment-templates (#63).

Correction after re-reading the code (self-review, 2026-08-21): the k8s schema shape is not net-new. manifest.rs has carried Runtime::Kubernetes(KubernetesRuntime { node_selector, service_account, tolerations }) since slice-0 (c17bf7d, both here and in the upstream openab/operator source) — it's currently hard-rejected by two explicit guards: apply.rs:239-244 and create.rs:56-58 (bail!("Kubernetes runtime not yet implemented")). 3b's job is to implement behind this existing enum arm and delete those two guards, not design new schema. KubernetesRuntime has no context/namespace/cluster field — confirms the original intent was for cluster targeting to live at the FleetBinding level (ADR-19), not per-manifest; carry that assumption forward unless there's a reason to override per-agent.

Sub-slices

• 3a — Provisioning driver trait extraction (prerequisite, no ADR yet). apply/scale/delete/get are inline aws_sdk_ecs calls with no trait boundary — ADR-2's "RuntimeDriver is the only layer with vendor terms" is only true for state classification (project() in agent-lifecycle) today, not CRUD. Extract a trait, EcsDriver becomes the first impl. Pure refactor, behavior unchanged, existing tests must stay green.
  • Also in scope for 3a (missed in the first pass): create.rs's interactive wizard is AWS-coupled starting at step 3 (Discord secret → Secrets Manager SmClient constructed unconditionally), before runtime is even asked at step 4 — VPC/subnet/security-group prompts (steps 6-8) are ECS-only concepts. The wizard needs restructuring so runtime selection happens early and branches: k8s path skips VPC/SG entirely and prompts kubeconfig context/namespace instead.
  • status.rs/studio_api.rs (Studio-local additions, not upstream) call ECS describe_* directly with no driver branching — the read-model needs a k8s-observe counterpart. This is separate from agent-lifecycle's RuntimeDriver::project() (that only classifies a native observation Studio already fetched; something has to fetch the k8s-native observation first).
• 3b — K8sDriver skeleton. kubeconfig context resolution (principal_source/scope/location per ADR-19), apply → Deployment, observe → Pod list → 4-discriminator → 6-state (ADR-2's k8s mapping table + traps). Removes the two not yet implemented guards. Adds kube + k8s-openapi crate deps.
• 3c — Bundle carrier. bundle_from (manifest.rs) currently only resolves to an S3 state prefix; add ConfigMap/volume (or init-container pull) carrier for k8s.
• 3d — Secrets. Corrected: not just a new URI scheme. secrets.rs::resolve_value_from's output contract is the ECS-native valueFrom shape ({arn}:{key}::) — a k8s target needs a different output (a Secret key selector), so this needs a driver-parameterized result type (e.g. SecretRef::Ecs(String) | SecretRef::K8s{ name, key }), with k8s-secret://<name>#<key> as the new input shorthand alongside the existing aws-sm://.
• 3e — Identity context panel. Wire k8s fields (context user/SA, cluster/namespace, kubectl auth whoami) into RuntimeContext per ADR-19's table.
• 3f — Console wiring. FleetBinding gains a k8s driver option + kubeconfig context selector; orbstack's local cluster is just another context, no special-casing.

Explicit non-goals / open questions

• ingress.rs (Cloud Map + VPC Link + API Gateway) stays ECS-only, permanently. manifest.validate() already enforces this: spec.ingress is rejected for non-ECS runtimes with "use native Kubernetes Ingress otherwise." Not a gap — a deliberate boundary from slice-0. k8s operators manage ingress with native k8s objects.
• bootstrap.rs k8s equivalent (namespace/RBAC/ServiceAccount one-time setup) — open question, leaning out of scope for v1. ECS bootstrap provisions cluster/IAM/SG/S3/log-group account-wide; for orbstack (operator already has cluster-admin on their own local cluster) this is plausibly unnecessary — assume namespace/RBAC pre-exist as an operator prerequisite (kubectl create namespace oab), revisit only if a shared/multi-tenant k8s target shows up later.

Starting with 3a.

[brettchien]

Self-review pass: re-read manifest.rs/apply.rs/create.rs/secrets.rs/status.rs/bootstrap.rs/ingress.rs end-to-end against the sub-slice list above. Found the Runtime::Kubernetes schema stub already exists (slice-0, both here and upstream in openab/operator) behind two hard-reject guards, found 3a's scope was missing create.rs's wizard + the Studio-local read-model files, found 3d's secrets scope was under-specified (output contract, not just input URI), and added two explicit non-goals/open-questions (ingress.rs, bootstrap). Issue body updated in place to fold these in.

[brettchien]

3a opened: #98 (write-path trait extraction, no behavior change, 85/85 tests green). Covers provision/scale/delete only — status.rs read-model and create.rs wizard reorder still open, tracked above.

[brettchien]

usercron tick (2026-08-22 ~00:00 TPE): 3b opened as #99 (stacked on #98, since it needs 3a's trait).

Also pushed a follow-up commit onto #98 itself: found while starting 3b that ProvisionDriver's signature leaked ECS-specific shapes (cluster in ApplyOptions/per-call params, AppliedService.ecs_service_name) — fixed before it became a second breaking change. No ECS behavior change, still 85/85 green at that point.

3b (#99) covers K8sDriver's write path only (apply/scale/delete against a Deployment, kubeconfig context resolution) — 92/92 tests green (7 new), clippy clean. Explicitly deferred and not silently papered over:

• bundleFrom carrier → 3c
• k8s secret refs → 3d
• observing k8s state into the 6-state (no k8s counterpart to status.rs yet) — this turned out bigger than I scoped in the original issue body, flagging as its own follow-up rather than folding into 3c/3d/3e/3f as originally listed. Will re-split when I get there.
• wiring K8sDriver into studio_api's actual dispatch — blocked on 3f (FleetBinding is the thing that tells studio_api which driver+target to use; studio_api's current signatures are aws_config-shaped and don't fit k8s at all).

Next tick: 3c (bundle carrier) unless #98/#99 need review-comment fixes first — will re-check PR state before continuing, not assume this holds.

Environment note for future ticks: this box OOM-kills on aws-sdk-ec2 even at cargo -j 1 on a cold cache — CARGO_PROFILE_DEV_DEBUG=0 (strip debug info) fixes it reliably. Also disk fills fast (21G total) with kube+k8s-openapi added to the tree — rm -rf target between runs is now load-bearing, not just tidy.

[brettchien]

usercron tick (2026-08-22 ~03:00 TPE): re-synced first — #98/#99 still open, no reviews yet, CI green on #99, no conflicting main-branch activity. Skipped 3c, did 3d instead; here's why.

Blocking finding on 3c (bundle carrier), needs your input before I build k8s parity for it:

manifest.rs's doc comment on spec.bundleFrom says "the runtime restores this prefix into ~ at first boot." I went looking for that restore mechanism in openabdev/openab (the actual consuming side, not oabctl's upload side) to make sure a k8s ConfigMap/volume carrier would replicate real behavior, not a documented-but-unimplemented aspiration. I couldn't find it:

• openab-core::pre_seed — the only "download from S3 and land in ~" code that exists — downloads zip archives listed in [hooks.pre_seed].sources, not a raw prefix of loose objects. Different shape from bundleFrom's s3://{bucket}/artifacts/{ns}/{name}/ prefix (individual keys, uploaded one put_object at a time by push_bundle).
• The image's ENTRYPOINT/CMD (Dockerfile.unified) is just tini -- openab run -c /etc/openab/config.toml — no wrapper script that syncs an S3 prefix.
• config-s3's load_config_raw_from_source only fetches the single config.toml text file when configFrom is an s3:// URI — it doesn't touch persona/skills files at all.
• Neither oabctl nor studio-compose write a [hooks.pre_seed] section referencing bundleFrom anywhere I can find.

So as far as I can verify, nothing currently downloads bundleFrom's prefix into ~ on the ECS path either — slice 2 (#65, merged) built the upload half (push_bundle) but I can't confirm the download half exists. Either I'm missing a mechanism (a newer openab feature, something image-build-time rather than runtime, etc.), or this has been an unverified gap since #65 merged.

Not guessing further — before I build a k8s ConfigMap/volume/init-container carrier for 3c, I'd rather confirm what the intended consuming mechanism actually is (or is supposed to become) than replicate an assumption that might already be wrong on ECS. Flagging for you rather than picking an answer myself.

Did 3d instead (independent of the above): #100, k8s secret refs. k8s-secret://<name>#<key> parses (pure, no API call — unlike aws-sm://) and wires into K8sDriver.build_deployment as secretKeyRef env vars. Non-k8s-secret:// values (e.g. a copy-pasted aws-sm:// ref) fail loudly instead of silently mis-deploying. 98/98 tests green, clippy clean. Stacked on #99.

Remaining: 3c (blocked, see above), 3e (identity context panel), 3f (console wiring — also blocked-ish, since it's the thing that would tell studio_api which driver to dispatch to). Next tick: 3e, since it doesn't depend on the bundleFrom question.

[brettchien]

usercron tick (2026-08-22 ~06:00 TPE): re-synced first — #98/#99/#100 still open, all green, no reviews/comments from Brett yet (including no answer yet on the 3c bundleFrom question from last tick). No conflicting main-branch activity.

3e opened as #101: `observe_k8s_identity` in studio-cp, the k8s counterpart to the existing AWS `observe_identity` — same `RuntimeContext` struct, live `SelfSubjectReview` call (the actual `kubectl auth whoami` API, stable since k8s 1.28) for principal/caller_id, kubeconfig context's cluster+namespace for scope, `location` left honestly empty (k8s has no region/zone equivalent). Not stacked on the others — pure studio-cp, doesn't touch `K8sDriver` at all, branches from main directly. 18/18 tests green.

Status: 3a/3b/3d/3e all have open PRs (#98/#99/#100/#101), none merged yet, none reviewed yet. 3c still blocked pending your read on the bundleFrom question. 3f (console wiring — FleetBinding gains k8s fields, so the panel can actually pick a driver+context) is the only remaining unstarted sub-slice; it's UI/config-schema work, haven't scoped it yet.

This tick's stopping here — four PRs open and unreviewed is enough in flight; next tick will check whether any need review-comment fixes before either starting 3f or (if you've answered) unblocking 3c.

[brettchien]

usercron tick (2026-08-22 ~09:00 TPE): 3f opened as #102 (stacked on #100 — needs oabctl::K8sDriver from the unmerged 3b chain), 8 new tests, 23/23 green in studio-cp.

5 of 6 sub-slices now have open PRs: 3a #98, 3b #99, 3d #100, 3e #101, 3f #102. Only 3c remains without a PR — still blocked on your read of the bundleFrom question (comment above), which no amount of further autonomous ticks can resolve on its own.

3f: `K8sFleetBinding`/`K8sFleetBindings`, config-schema+load/save mirroring `FleetBinding` exactly (context+namespace instead of cluster+profile, `resolve_binding_driver` → `K8sDriver` instead of `resolve_binding_config` → `SdkConfig`). Deliberately a separate file (`fleets-k8s.toml`) rather than a second table in `fleets.toml` — the console's editor does whole-file verbatim writes, so sharing one file would let a k8s-fleet edit silently clobber your actual prod `fleets.toml` bindings on save. `expected_principal`/IdentityMismatch wasn't ported (real follow-up, not required for the schema to be useful). Frontend console panel wiring is out of scope — backend schema only.

Stopping the usercron loop here. The remaining work (3c) needs your decision, not more autonomous cycles — a 3-hourly loop that just re-reports "still waiting" isn't useful and burns the shared Claude seat for nothing. Removing the cronjob entry now; when you've had a chance to look at bundleFrom (and to review/merge #98→#102 whenever), just say so in the thread and I'll pick 3c up from a normal message, no cron needed for a one-time human decision.

Final state: #98/#99/#100/#101/#102 all open, CI green (or running), none merged, none reviewed yet. Full summary posted to the Discord thread.

[brettchien]

3c resolved as #103 (stacked on #102), with Brett's go-ahead in-thread. Root-cause fix, not the originally-scoped k8s carrier:

Finding: nothing in `openab` ever downloaded `bundleFrom`'s S3 prefix back into `` at boot — slice 2 (#65, already merged) only ever built the upload half. `openab`'s only real "S3 → ``" mechanism is `hooks.pre_seed`, which consumes zip archives, a different shape from `bundleFrom`'s loose-object prefix.

Fix: `studio_compose::Bundle::zip_bytes()` (pure, in-memory zip) + `oabctl::studio_api::bundle_zip_uri`/`inject_pre_seed_hook` (append `[hooks.pre_seed]` into the composed `config.toml`, no-op if already present) + wire into `provision_from_library`. Fixes the ECS gap.

Payoff for k8s: `pre_seed` is orchestrator-agnostic (S3 GetObject + extract, doesn't care what started the process), and `K8sDriver.build_deployment` already points the container at `configFrom` same as ECS — so a k8s pod restores its bundle through the identical mechanism. Removed `K8sDriver`'s `bundle_from` bail entirely. 3c's originally-scoped ConfigMap/volume/init-container carrier turned out not to be needed at all once the actual restore path was fixed — one change covers both platforms.

145/145 tests green across oabctl/studio-compose/studio-cp, clippy clean.

---

All 6 sub-slices now have open PRs: 3a #98, 3b #99, 3d #100, 3e #101, 3f #102, 3c #103. Stack order: #98 → #99 → #100 → #102 → #103 (3e independent). None merged/reviewed yet — that's the remaining step, and it's yours, not mine (I only have commit/push/PR authorization). This closes out the autonomous work on studio#97; ping when you've had a look and I'll pick up anything that needs fixing from review, or the console UI wiring / read-model / ECS→k8s naming follow-ups noted across these PRs.