fix(bundle): actually restore a deployed agent's bundle at boot, via pre_seed (slice 3c) - #103
Open
brettchien wants to merge 2 commits into
Conversation
…pre_seed (slice 3c) Root-cause fix for the gap studio#97's slice-3c investigation found: nothing in openab downloads bundleFrom's loose-file S3 prefix back into ~ at boot. push_bundle (slice 2, already merged) only ever built the upload half; no restore mechanism was ever wired to the other end, on ECS or anywhere else. openab's only "S3 -> ~" feature is hooks.pre_seed, and it consumes zip archives, not a prefix of individual objects -- a different shape nothing bridged. Fix: reuse pre_seed instead of building a bespoke restore mechanism per platform. - studio_compose::Bundle::zip_bytes() -- a deterministic zip of the bundle's files (still pure/no-I/O: ZipWriter writes to an in-memory Cursor<Vec<u8>>). - oabctl::studio_api::bundle_zip_uri/inject_pre_seed_hook -- the zip's S3 URI, and appending a [hooks.pre_seed] section to a config.toml's bytes pointing at it. Appends rather than re-parsing+re-serializing, so an operator's existing comments/formatting survive; a no-op if hooks.pre_seed is already present (never overrides an operator's own hook). - control_plane::resolve_bucket exposed pub, so a caller can resolve the bucket before constructing objects (needed to compute the zip URI ahead of upload) instead of after, avoiding a second implicit resolution. - provision_from_library (studio-cp): resolves the bucket once, injects the hook into the composed config.toml object, uploads the zip alongside the existing per-file objects (kept for now -- harmless, and removing them is a separable decision). - K8sDriver: removed the bundle_from bail entirely. This is the actual payoff -- pre_seed is orchestrator-agnostic (S3 GetObject + extract, same binary/feature regardless of what started the process), and build_deployment already points the container's command at configFrom exactly like EcsDriver does, so a k8s pod restores its bundle through the identical mechanism with zero k8s-specific carrier code. No ConfigMap, no volume, no init-container -- slice 3c's original scope turned out not to exist once the actual restore path was fixed. Fixes both the ECS gap and the k8s driver in one change; the two platforms now share one restore mechanism instead of doubling the maintenance surface with a bespoke k8s carrier. 11 new tests across oabctl/studio-compose/studio-cp (145/145 total across the three crates), clippy clean (no new warnings; 3 pre-existing ones elsewhere untouched). Stacked on #102 (3f) -- needs K8sDriver + the driver-side test fixtures from the earlier branches in the chain. Ref: studio#97 (K8s driver — ADR #63 slice 3, sub-slice tracking)
Follow-up commit on #103, from a multi-angle review pass before merge: - inject_pre_seed_hook: rewritten on toml_edit (format-preserving) instead of raw string append. Previously, *any* existing [hooks.pre_seed] made injection a full no-op -- an operator's own hooks.pre_seed (e.g. this agent's own persistent-state restore, unrelated to the deploy bundle) meant the new bundle.zip was uploaded but never actually wired in, and provision_from_library still reported success. Now merges the zip URI into the existing `sources` array (true no-op only when that exact URI is already present), mirrors pre_seed's own 5-source cap, and bails instead of silently producing invalid TOML when `hooks`/`hooks.pre_seed` is already an inline table (which can't be reopened with a `[table]` header). - provision_from_library: mutates `bundle.files` in place before deriving artifact_objects/zip_bytes/digest from it, instead of deriving them from the pre-injection Bundle. Previously the uploaded zip's own config.toml never got the pre_seed hook, and ProvisionOutcome.digest didn't match what was actually uploaded. - secret_env_vars: attaches the per-secret error context (env var name, namespace/name) to both the "wrong scheme" and "right scheme, malformed body" cases. anyhow's Context impl for Option<T> only fires on None, so the previous `.with_context()??` silently dropped context on the latter. - EcsDriver::apply: preserves the structured ApplyError in the anyhow error's source chain (via .context() on Error::new(e)) instead of flattening it into anyhow::anyhow!(...). A caller can again `.chain().find_map(downcast_ref::<ApplyError>)` to recover `.completed`/`.failed_service` for a partial fleet-apply failure. ApplyError::reconciliation made pub(crate) so driver.rs's test can construct one without a live ECS call. 24 new/changed tests, 131/131 total across oabctl+studio-cp, clippy clean (same 3 pre-existing warnings elsewhere, untouched).
Contributor
Author
|
Pushed a follow-up commit addressing the review findings from the code-review pass (see thread) — most importantly, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root-cause fix for sub-slice 3c (studio#97, ADR #63 slice 3) — and for a gap in already-merged slice 2 that 3c's investigation surfaced.
The bug: nothing in
openabever downloadedbundleFrom's S3 prefix back into~at boot.push_bundle(slice 2, #65, merged) built the upload half; no restore mechanism was ever wired to the other end — not for ECS, and (obviously) not for k8s either, since 3c hadn't started.openab's only "S3 →~" feature ishooks.pre_seed, and it consumes zip archives, not a prefix of loose objects — a different shape, never bridged.The fix: reuse
pre_seedinstead of inventing a k8s-specific carrier (ConfigMap/volume/init-container, as originally scoped).studio_compose::Bundle::zip_bytes()— deterministic zip of the bundle's files. Still pure/no-I/O (ZipWriter→ in-memoryCursor<Vec<u8>>), matching the crate's existing "no AWS, no filesystem" contract.oabctl::studio_api::bundle_zip_uri/inject_pre_seed_hook— the zip's S3 URI, and appending a[hooks.pre_seed]section to aconfig.toml's bytes. Appends, doesn't re-parse+re-serialize the whole file, so an operator's comments/formatting survive untouched (same principlesave_bindings_textalready applies tofleets.toml). No-op if[hooks.pre_seed]is already present — an operator's own hook is never silently overridden.control_plane::resolve_bucketexposedpub— a caller now needs the bucket before constructing the upload objects (to compute the zip URI), not just insidepush_bundle/provisionas before.provision_from_library(studio-cp): resolves the bucket once, injects the hook into the composedconfig.tomlobject, uploads the zip alongside the existing per-object uploads (kept for now — harmless; removing them is a separate decision, nothing else reads them either way).K8sDriver: removed thebundle_frombail entirely. This is the actual payoff —pre_seedis orchestrator-agnostic (S3GetObject+ extract, same feature regardless of what started the process), andbuild_deploymentalready points the container's command atconfigFromexactly likeEcsDriverdoes. A k8s pod restores its bundle through the identical mechanism, zero k8s-specific carrier code. 3c's originally-scoped work (ConfigMap/volume/init-container) turns out not to be needed once the actual restore path is fixed.One fix covers both platforms — they now share a restore mechanism instead of doubling the maintenance surface with a bespoke k8s carrier.
Stacking note
Branched from #102 (3f) — needs
K8sDriverand its test fixtures from earlier in the chain.Testing
11 new tests across
oabctl/studio-compose/studio-cp— 145/145 total across the three crates.cargo clippyintroduces no new warnings (3 pre-existing ones elsewhere, untouched).Ref #97. Approved by Brett in-thread before starting (touches the already-merged ECS path, not just new k8s code).