[lmilbaum]

Summary

• Extracts package verification into airgap/scripts/verify-package.sh (shared by offline-run.sh and the new update script).
• Adds airgap/scripts/update-package.sh with apply, rollback, confirm, and status subcommands. The apply flow stages the new package, verifies it completely (signature, SBOM, digests) before touching the active archive, then cuts over atomically. The prior archive is kept as .bak until confirm. On deploy failure, the prior archive is restored automatically. Only the package archive is transactional; live cluster state is not rolled back.
• Adds airgap/tests/test-airgap-update-rollback.py with a stub mode (runs in mise run validate and CI without a real package) and a --zarf-package real mode (runs in the new nightly update-rollback job under sudo unshare --net).
• Adds the update-rollback job to .github/workflows/air-gapped.yml and to report-status needs so nightly failures open the tracking issue.
• Model from issue design the air-gap package build, signing, publication, and consumption model #138 comment (2026-10-16): airgap/archives/ is the artifact boundary; ephemeral CI artifacts; operators transfer the archive manually.

How to test

• python3 airgap/tests/test-airgap-update-rollback.py: stub mode, no zarf needed, runs in mise run validate.
• mise run validate: includes the above.
• Dispatch the air-gapped workflow on this branch (gh workflow run air-gapped.yml --ref feat/airgap-transactional-update); check that update-rollback is green and isolated-deploy is unaffected.
• Optional real-zarf local run: ZARF_VERIFY_KEY=/path/cosign.pub python3 airgap/tests/test-airgap-update-rollback.py --zarf-package airgap/archives/zarf-package-krops-airgap-arm64-0.1.0.tar.zst under Wi-Fi off or sudo unshare --net.

Reviewer summary

• First-pass HIGH finding (always-passing fnmatch in naming-invariant test): fixed. Test now parses ACTIVE_NAME from the script source.
• MEDIUM: ACTIVE_NAME in update-package.sh is manually maintained; comment added; naming-invariant test enforces sync with offline-run.sh at CI time.
• MEDIUM: pkg glob in air-gapped.yml guarded against multiple matches.
• No em dashes introduced.

🤖 Generated with Claude Code

[shrinedogg]

APPROVE

Verified:

• CI all green, including "Build, lint, and test" (runs the new stub-mode test), "Assembler tests and strict site build", "Air-gap image digest pins", "Renovate airgap digest pinning", "Zarf mise asset pin".
• "Rendered Flux diff" skip confirmed legitimate: the PR is from a fork and konflate.yml skips fork PRs by design (job guard head.repo.full_name == github.repository); the diff touches no cluster manifests (workflows, airgap scripts/tests, docs, AGENTS.md, mise.toml only).
• Transactional apply traced in airgap/scripts/update-package.sh: stage into archives/.staging/ (lines 137-140) -> airgap_verify_package signature/checksum/SBOM decode via the sourced verify-package.sh library (returns, never exits) -> hard-link backup of the active archive plus recorded sha256 (lines 151-154) -> atomic mv cutover (line 160) -> deploy -> on deploy failure restore_previous verifies the backup digest, restores it, and exits 3 (lines 163-167, 106-119); on success state moves to awaiting-confirm and a second apply is refused until confirm or rollback. A package rejected in staging never touches the active archive (the test asserts content and inode are unchanged and that deploy is never invoked).
• Offline rollback traced: rollback() (lines 179-195) is purely local file operations (mkdir lock, state grep, sha256 check, mv, rm) with no network, docker, or registry access. The nightly update-rollback job additionally runs the real-zarf rejection and restore scenarios under sudo unshare --net (air-gapped.yml), so the real-zarf path executes with the network namespace cut.
• Ran python3 airgap/tests/test-airgap-update-rollback.py locally (stub mode, exactly as CI invokes it): 11/11 scenarios PASS (first install, first-install rollback, truncated package rejected, corrupted package rejected, deploy-failure restore reusing the prior inode, full apply/refuse/rollback/re-apply/confirm cycle, stale staging discarded, cutover-crash apply refused + rollback restores, cutover-crash confirm refused, tampered backup exits 4 keeping the active archive, naming invariants).
• Probed paths the suite does not cover, locally with a stub deploy command and fake zarf: rollback with the .bak deleted entirely -> rc=4 fail-safe with the active archive untouched and state left for the operator; rollback/confirm with no state file -> rc=1 "no update in progress"; status on empty state -> clean "none" output; live lock (own pid) blocks apply with rc=1; stale lock with a dead pid is reclaimed automatically and the lock dir is removed after the run; unknown/corrupted phase value in the state file -> apply proceeds (harmless: the backup is always taken before the swap); usage errors rc=64; missing package file rc=1.
• Mirrored the "Assembler tests and strict site build" job (docs.yml) locally under mise: uv sync --locked OK, uv run pytest -q -> 13 passed, uv run python tools/assemble_docs.py (30 files) + uv run mkdocs build --strict -> clean, so the docs/airgap.md rewrite renders under the strict build.
• All 11 airgap tests from the validate job pass locally. Two local-only failures are environment, not code: test-airgap-cert-manager-digest-values.py fails on system Python 3.9.6 (PEP 604 str | None at import) and passes on the pinned 3.15; test-airgap-failure-notification.py needs PyYAML on a bare 3.15 interpreter (present in CI).
• Naming invariants hold: ACTIVE_NAME == offline-run.sh PACKAGE_INPUT default == zarf.yaml metadata.version 0.1.0, and no auxiliary name (.bak, .bak.sha256, .staging, .update-state, .update.lock) matches the *.tar docker-load glob in stage-and-create-cluster.sh.
• git check-ignore confirms airgap/archives/ (.gitignore line 21) covers every state/backup file the script writes.
• Docs consistent: the new "Updating a deployed bundle" section matches the code (exit codes 0/1/2/3/4/64, crash-recovery table, explicit scope: archive-only transactionality, cluster state is never rolled back, redeploy with offline-run.sh); AGENTS.md is updated to match; no dangling references to the old "Update drill" section; the referenced tracking issue #369 ("airgap: transactional bundle update with tested offline rollback") is open.
• Commit subject matches the PR title and the diff.

Note (non-blocking):

• The update-rollback job triggers only on schedule/workflow_dispatch, so its real-zarf mode (--zarf-package under unshare --net) first executes on the next nightly after merge; PR CI covers the script logic in stub mode.
• apply() (lines 128-135) refuses only cutover|awaiting-confirm phases; an unknown or corrupted phase value lets a new apply proceed. There is no data-loss path (the backup is always created before the swap), but an explicit allowlist of known phases would be tighter.
• Line 140 uses a global sync; syncing just the staging directory would be lighter. Cosmetic.