Skip to content

feat(cli): site archive container for export and import (#47, step 1) - #129

Merged
Madraka merged 1 commit into
mainfrom
feat/issue-47-archive-container
Oct 25, 2026
Merged

Madraka merged 1 commit into
mainfrom
feat/issue-47-archive-container

Conversation

@Madraka

@Madraka Madraka commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

What

This is step 1 of RFC-0039's rollout for #47: the .cloudhal container that cloudhal export (step 2) will write and cloudhal import (steps 3–4) will read. It adds no command yet and no user-visible change, so there is no changeset. The code lives in packages/cli/src/archive/, is Node-only, and no Worker imports it.

  • Format (RFC §1)
    • A gzip-compressed POSIX ustar. It holds regular files only, with mode 0644, uid/gid 0 and zero times, and is padded to 10240-byte records.
    • Entries come in one fixed order: manifest.json first, then project.json, the nodes|snapshots|blobs/NNNN.ndjson parts (a new part every 8 MiB), media/<sha256> sorted by hash, and ledger.ndjson.
    • The writer takes createdAt and createdBy from the caller, never from the clock. Step 2 passes the site's newest row timestamp and the CLI version.
  • Reader (untrusted input, RFC Security / ADR-0041 §7)
    • Paths come from a closed list, so .., absolute paths, backslashes and malformed names are refused. Links, devices, directories and pax/GNU extended headers are refused too.
    • It checks header checksums and strict octal fields.
    • Every file must be listed in the manifest, in order, exactly once. Its size and SHA-256 are checked, and a media file's hash must also match its name.
    • Row counts are checked.
    • There are caps on the manifest (16 MiB), each part (16 MiB), each media file (100 MiB, the same as MAX_MEDIA_PUT), total decompressed bytes and the number of entries.
    • Data after the tar end or the gzip stream is refused.
    • A newer archive format, bundle format or contracts schemaVersion is refused with a message naming the createdBy version.
    • Untrusted names are only echoed when they are safe: no control or bidi characters, and length-capped.
    • Parts reach the caller only after they are verified. A media body streams and throws at its end on a mismatch. A body read too late gives an explicit error instead of a short read.
  • Writer: media are verified while they stream. Output goes to a temp file, is fsynced, and is renamed into place only when complete, so a failed media source leaves no file behind.

RFC / ADR amendment, please review

Invariant 7 promised identical bytes for an unchanged site, but gzip output differs between zlib builds. We reproduced this: Homebrew's Node and the official Chromium-zlib Node give different .gz bytes for the same input.

The amendment in RFC-0039 (invariant 7, rollout step 5) and ADR-0041 (rule 6) says:

  • identical .cloudhal bytes hold for the same Node/zlib build;
  • the tar inside, and every SHA-256 in the manifest, are identical on every platform. A test pins the golden SHA-256 of the decompressed tar.

Avoiding the difference entirely would need a pure-JS deflate dependency. The step 5 demo archive is regenerated on one pinned Node, or compared through its tar.

Verification (local)

  • vitest run test/archive- in packages/cli: 6 files, 117 tests. They cover:
    • the round trip and deterministic bytes, with a golden tar SHA-256;
    • a damaged part, a wrong media hash, truncated gzip/tar and trailing data;
    • path traversal and every refused path form, plus link, dir and extended-header typeflags;
    • unlisted, missing, duplicate and out-of-order entries, and the manifest not coming first;
    • newer format and schema versions, the caps and a gzip bomb, and 8 MiB part splitting;
    • interop: system tar -tzf / -xzf give the same paths and bytes.
  • Each of the 7 fixes from review was checked by restoring the old behaviour: its new test failed.
  • pnpm --filter @cloudhal/cli typecheck, tsc -p tsconfig.build.json (declarations), repo-wide biome check and pnpm size:check pass.

Checklist

  • pnpm check and cargo test --workspace pass locally. Ran the pieces that cover this change (biome, size, CLI typecheck and declarations, the archive tests). There is no Rust or contract change, and CI runs the full set.
  • Contract change? No.
  • User-visible change in a published package? Not yet. Step 2 (cloudhal export) adds the changeset.
  • Commits are signed off (git commit -s, DCO)
  • Docs / AGENTS.md updated if behaviour or commands changed. These are the RFC-0039 / ADR-0041 amendments; the commands and their docs come in steps 2–5.

Refs #47

🤖 Generated with Claude Code

RFC-0039 rollout step 1: the .cloudhal container that `cloudhal export`
will write and `cloudhal import` read. A POSIX ustar stream (regular
files only, fixed modes, zero times, 10240-byte records) compressed with
gzip at a fixed level and a fixed OS byte, holding manifest.json first and
then project.json, nodes/snapshots/blobs NDJSON parts of at most 8 MiB,
media/<sha256> and ledger.ndjson in one fixed order.

The reader treats the file as untrusted input: a closed list of paths,
no links, devices or extended headers, header checksums, every file
checked against the manifest's size and SHA-256 (media also against its
name), row counts, caps on the manifest, parts, media, total
decompressed bytes and entry count, and no data after the tar or gzip
end. Parts reach the caller only once verified; media bodies stream and
fail at their end on a mismatch.

The writer streams media and verifies them on the way, and writes to a
temporary file that is renamed only when complete. RFC-0039 invariant 7
and ADR-0041 rule 6 are amended: gzip output differs between zlib
builds, so identical .cloudhal bytes hold for the same Node/zlib build,
while the tar inside and every manifest SHA-256 are identical on every
platform.

No command yet; `cloudhal export` comes next.

Refs #47

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Madraka <[email protected]>
@Madraka
Madraka enabled auto-merge October 25, 2026 20:18
@Madraka
Madraka merged commit d927fa5 into main Oct 25, 2026
4 checks passed
@Madraka
Madraka deleted the feat/issue-47-archive-container branch October 25, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant