Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/adr/0041-site-archives.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# ADR-0041: Site archives: a streaming `.cloudhal` tar of NDJSON parts and media, ids kept, graph only
- Status: Accepted
- Date: 2026-10-22 (decided by Halil)
- Amendment (2026-10-25, RFC-0039 rollout step 1): deterministic bytes hold for the same Node/zlib build; the tar stream and the manifest's SHA-256s on every platform. See rule 6.
- Related: RFC-0039, RFC-0004, RFC-0010, ADR-0002, #47
- Sources: [Sanity datasets](https://www.sanity.io/docs/cli-reference/dataset) · [Strapi data management](https://docs.strapi.io/cms/data-management/export) · [Contentful CLI export](https://github.com/contentful/contentful-cli/tree/main/docs/space/export) · [Ghost migration](https://docs.ghost.org/migration/ghost) · [WordPress export](https://wordpress.org/documentation/article/tools-export-screen/)

Expand All @@ -18,7 +19,7 @@ A site's content can only leave a project through a deployment (`push`, `pull`),

Never erase-then-import.
5. **Graph only.** No users, sessions, roles, agent keys, comments, readers, analytics, audit log, automations, secrets or webhook credentials. That is why there is no built-in encryption; people can pipe the file through `age` or `gpg`.
6. **Deterministic bytes:** a fixed order, zero timestamps and fixed modes, so an unchanged site exports identically.
6. **Deterministic bytes:** a fixed order, zero timestamps and fixed modes, so an unchanged site exports identically (amended: byte for byte with the same Node/zlib build, since gzip output differs between zlib builds; the tar stream inside, and every SHA-256 in the manifest, on every platform).
7. **Untrusted input:** the layout is a closed list of paths (no traversal, links or devices); sizes are capped; rows go through the precompiled validators and gates, as for `push`.
8. **Later, already decided:** `init --from`, `--new-ids` for templates, partial export, `--from <admin-url>`, Studio download, and signed archives.

Expand Down
5 changes: 3 additions & 2 deletions docs/rfcs/0039-site-archive-export-import.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# RFC-0039: Site archives: `cloudhal export` and `cloudhal import`

- Status: Accepted (2026-10-22, by Halil; ADR-0041). Open questions decided as recommended. Revised before acceptance after a comparison with Sanity, Strapi, Contentful, Ghost, WordPress, Directus and Webflow (see Prior art): NDJSON parts instead of one JSON file, per-file checksums, contract versions in the manifest, and a snapshot before `--replace`.
- Amendment (2026-10-25, rollout step 1): identical `.cloudhal` bytes need the same Node/zlib build; the tar stream inside, and so every SHA-256 in the manifest, is identical on every platform. See invariant 7.
- Author: Halil (Madraka), drafted with Claude · Date: 2026-10-21
- Related:
- #47 (found while building cloudhal.com: no way to keep or move a site's content without a deployment, so it was written as code).
Expand Down Expand Up @@ -88,7 +89,7 @@ cloudhal import <file> [--replace] [--dry-run]
4. An archive written by a newer format or a newer contracts `schemaVersion` is refused with the CLI version needed. Older ones are read, and migrated forward if a later schema version requires it.
5. Taken-down content stays taken down: the `purged` ledger travels and the import never restores those bytes.
6. Import writes the graph in atomic batches (no interactive transactions, AGENTS.md rule 4), guarded like `importSite` today. A failure leaves the project as it was, and `--replace` can always be undone through its `before import` snapshot.
7. Same content, same file: rows and files are written in a fixed order, with zero timestamps and fixed modes in the tar headers, so exporting an unchanged site twice gives identical bytes (good for git and for diffing).
7. Same content, same file: rows and files are written in a fixed order, with zero timestamps and fixed modes in the tar headers, so exporting an unchanged site twice gives identical bytes (good for git and for diffing). *Amended 2026-10-25:* identical bytes hold for the same Node/zlib build. gzip output differs between zlib builds (Homebrew's zlib and the Chromium zlib that official Node binaries ship compress the same input differently), and only a dependency or uncompressed output would avoid that. The tar stream inside the gzip, and so every SHA-256 in the manifest, is identical on every platform. A committed archive (step 5's demo) is regenerated on one pinned Node in CI, or compared through its decompressed tar.

### 4. What is not in an archive

Expand Down Expand Up @@ -125,7 +126,7 @@ The manifest says so, and `export` prints it. People and personal data never lea
2. `export`.
3. `import` with `--dry-run`.
4. `--replace`, the `before import` snapshot, and the site id switch.
5. Docs and a starter demo archive.
5. Docs and a starter demo archive (regenerated on one pinned Node, or compared through its decompressed tar: invariant 7).

## Security & policy impact

Expand Down
97 changes: 97 additions & 0 deletions packages/cli/src/archive/byte-source.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
/**
* Exact byte counts from an async iterable of chunks of any size: gunzip emits 16 KiB chunks that
* do not line up with tar's 512-byte blocks. Internal to the archive reader.
*/
import { ArchiveError } from "./errors.ts";

const EMPTY = new Uint8Array(0);

function truncated(): ArchiveError {
return new ArchiveError("truncated", "the archive ends in the middle of an entry (truncated)");
}

export class ByteSource {
readonly #iterator: AsyncIterator<Uint8Array>;
#chunk: Uint8Array = EMPTY;
#offset = 0;
#position = 0;
#done = false;

constructor(source: AsyncIterable<Uint8Array>) {
this.#iterator = source[Symbol.asyncIterator]();
}

/** Bytes consumed so far. */
get position(): number {
return this.#position;
}

/** Makes sure an unread byte is buffered; false at the end of the source. */
async #fill(): Promise<boolean> {
while (this.#offset >= this.#chunk.length) {
if (this.#done) return false;
const next = await this.#iterator.next();
if (next.done) {
this.#done = true;
return false;
}
this.#chunk = next.value;
this.#offset = 0;
}
return true;
}

#take(max: number): Uint8Array {
const n = Math.min(max, this.#chunk.length - this.#offset);
const view = this.#chunk.subarray(this.#offset, this.#offset + n);
this.#offset += n;
this.#position += n;
return view;
}

/**
* Exactly `n` bytes in a fresh array; undefined when the source ended before the first of them,
* `truncated` when it ends part-way.
*/
async read(n: number): Promise<Uint8Array | undefined> {
const out = new Uint8Array(n);
let filled = 0;
while (filled < n) {
if (!(await this.#fill())) {
if (filled === 0) return undefined;
throw truncated();
}
const view = this.#take(n - filled);
out.set(view, filled);
filled += view.length;
}
return out;
}

/**
* Between 1 and `max` bytes, as a view of the source's chunk (Node streams never reuse a chunk
* they emitted); `truncated` at the end of the source.
*/
async readSome(max: number): Promise<Uint8Array> {
if (!(await this.#fill())) throw truncated();
return this.#take(max);
}

/** Discards exactly `n` bytes; `truncated` when the source has fewer. */
async skip(n: number): Promise<void> {
let left = n;
while (left > 0) left -= (await this.readSome(left)).length;
}

/** True when every byte has been consumed and the source has ended. */
async atEnd(): Promise<boolean> {
return !(await this.#fill());
}

/** Stops the source (runs its `finally`); safe to call more than once. */
async close(): Promise<void> {
if (this.#done) return;
this.#done = true;
await this.#iterator.return?.();
}
}
60 changes: 60 additions & 0 deletions packages/cli/src/archive/errors.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/**
* The one error type of the site archive container (RFC-0039): a code that `cloudhal import` maps
* to its messages, the archive path when there is one, and a safe echo of untrusted names.
*/

/**
* - `not_archive`: empty, not gzip, or no Cloudhal manifest as the first entry.
* - `unsupported_version`: written by a newer Cloudhal (archive, contracts or bundle format).
* - `invalid_manifest`: `manifest.json` is malformed or disagrees with itself.
* - `unsafe_entry`: a tar header or path Cloudhal never writes (links, devices, traversal).
* - `mismatch`: the entries disagree with the manifest (unlisted, duplicate, order, size, missing).
* - `damaged`: a checksum, SHA-256 or zlib check failed, or data follows the tar or gzip end.
* - `truncated`, `too_large`: the input ends early, or is over a cap.
* - `invalid_input`: the writer was given content it must not write, or a body was misused.
*/
export type ArchiveErrorCode =
| "not_archive"
| "unsupported_version"
| "invalid_manifest"
| "unsafe_entry"
| "mismatch"
| "damaged"
| "truncated"
| "too_large"
| "invalid_input";

/** Every failure of the archive writer and reader. */
export class ArchiveError extends Error {
override readonly name = "ArchiveError";
readonly code: ArchiveErrorCode;
/** The archive path the error is about, when there is one (a path of the closed layout). */
readonly path: string | undefined;
constructor(
code: ArchiveErrorCode,
message: string,
options: { path?: string; cause?: unknown } = {},
) {
super(message, options.cause === undefined ? undefined : { cause: options.cause });
this.code = code;
this.path = options.path;
}
}

const MAX_ECHO = 120;

/**
* An untrusted name (an entry name by default) as it may appear in a message: JSON-quoted and cut
* when every character is printable ASCII, otherwise only `described`, so a name cannot carry
* terminal escapes (C0 or C1 controls), bidi overrides or megabytes into a message.
*/
export function printable(
raw: string,
described = "an entry name with a non-printable byte",
): string {
for (let i = 0; i < raw.length; i++) {
const c = raw.charCodeAt(i);
if (c < 0x21 || c > 0x7e) return described;
}
return JSON.stringify(raw.length > MAX_ECHO ? `${raw.slice(0, MAX_ECHO)}...` : raw);
}
143 changes: 143 additions & 0 deletions packages/cli/src/archive/gzip.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
/**
* gzip for site archives. The writer compresses with fixed parameters and a fixed OS byte, so the
* same tar stream gives the same bytes on every platform that runs the same zlib build (gzip
* output is not identical across zlib builds; the tar stream inside it is). The reader accepts any
* gzip and nothing after its last member, counts the decompressed bytes against a cap, and maps
* zlib's errors to archive errors.
*/
import { pipeline } from "node:stream/promises";
import { constants, createGunzip, createGzip, type ZlibOptions } from "node:zlib";
import { ArchiveError } from "./errors.ts";

/**
* Level 6 (zlib's default) writes XFL 0. Level 9 costs about twice the CPU on NDJSON for about 3%
* less; level 0 is not used because stored blocks follow the input's chunking.
*/
export const GZIP_OPTIONS: Readonly<ZlibOptions> = Object.freeze({
level: 6,
memLevel: 8,
windowBits: 15,
strategy: constants.Z_DEFAULT_STRATEGY,
});

/** RFC 1952 OS "unknown": the same on every platform (zlib writes 3 on Unix, 10 on Windows). */
export const GZIP_OS = 255;

const GZIP_HEADER = 10;

/** Checks zlib's header (no name, comment or header CRC: FLG 0) and sets the OS byte. */
function patchHeader(head: Uint8Array): Uint8Array {
if (head[0] !== 0x1f || head[1] !== 0x8b || head[2] !== 8 || head[3] !== 0)
throw new ArchiveError("invalid_input", "zlib wrote an unexpected gzip header");
head[9] = GZIP_OS;
return head;
}

/** `source` gzip-compressed with GZIP_OPTIONS and the OS byte set to GZIP_OS. */
export async function* gzipFixed(
source: AsyncIterable<Uint8Array>,
): AsyncGenerator<Uint8Array, void, undefined> {
const gzip = createGzip(GZIP_OPTIONS);
const piping = pipeline(source, gzip);
piping.catch(() => {}); // surfaced by the loop below (pipeline destroys gzip with the error)
let head = new Uint8Array(0);
let patched = false;
try {
for await (const chunk of gzip as AsyncIterable<Uint8Array>) {
if (patched) {
yield chunk;
continue;
}
head = Buffer.concat([head, chunk]);
if (head.length < GZIP_HEADER) continue;
patched = true;
yield patchHeader(head);
}
await piping;
if (!patched) throw new ArchiveError("invalid_input", "zlib wrote no gzip header");
} finally {
gzip.destroy();
}
}

function zlibCode(error: unknown): string | undefined {
const code = (error as { code?: unknown } | null)?.code;
return typeof code === "string" && code.startsWith("Z_") ? code : undefined;
}

/** What the source handed to gunzip: its first two bytes, its length, and whether it ended. */
interface Feed {
head: number[];
bytes: number;
ended: boolean;
}

/** `source` unchanged, counted into `feed`. */
async function* counted(
source: AsyncIterable<Uint8Array>,
feed: Feed,
): AsyncGenerator<Uint8Array, void, undefined> {
for await (const chunk of source) {
for (const b of chunk.subarray(0, 2 - feed.head.length)) feed.head.push(b);
feed.bytes += chunk.length;
yield chunk;
}
feed.ended = true;
}

/**
* A zlib error as an archive error. Whether the input is gzip at all is judged from its first two
* bytes (the magic 1f 8b), not from how much of it was decompressed before the error.
*/
function mapZlibError(error: unknown, feed: Feed): unknown {
const code = zlibCode(error);
if (code === undefined) return error;
if (feed.bytes === 0)
return new ArchiveError("not_archive", "not a Cloudhal site archive (it is empty)", {
cause: error,
});
if (feed.head[0] !== 0x1f || feed.head[1] !== 0x8b)
return new ArchiveError("not_archive", "not a Cloudhal site archive (not gzip)", {
cause: error,
});
if (code === "Z_BUF_ERROR")
return new ArchiveError("truncated", "the archive is truncated (gzip ends early)", {
cause: error,
});
return new ArchiveError("damaged", "the archive is damaged (gzip data is corrupt)", {
cause: error,
});
}

/**
* `source` decompressed, `too_large` beyond `maxBytes` decompressed bytes. It returns only after
* gunzip ended cleanly (its CRC and length check passed) and consumed the whole source: zlib stops
* quietly before input that starts with a zero byte, so anything left over is refused here as
* `damaged`. Other errors (of the source, such as a file read) pass through unchanged.
*/
export async function* gunzipCapped(
source: AsyncIterable<Uint8Array>,
maxBytes: number,
): AsyncGenerator<Uint8Array, void, undefined> {
const feed: Feed = { head: [], bytes: 0, ended: false };
const gunzip = createGunzip();
const piping = pipeline(counted(source, feed), gunzip);
piping.catch(() => {}); // surfaced by the loop below
let produced = 0;
try {
for await (const chunk of gunzip as AsyncIterable<Uint8Array>) {
produced += chunk.length;
if (produced > maxBytes)
throw new ArchiveError("too_large", `the archive is above ${maxBytes} bytes decompressed`);
yield chunk;
}
// Checked before `piping`, which fails (premature close, abort) when gunzip ended early.
if (!feed.ended || feed.bytes !== gunzip.bytesWritten)
throw new ArchiveError("damaged", "the archive has data after its gzip stream");
await piping;
} catch (error) {
throw mapZlibError(error, feed);
} finally {
gunzip.destroy();
}
}
17 changes: 17 additions & 0 deletions packages/cli/src/archive/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
/**
* Site archives (RFC-0039, ADR-0041): the `.cloudhal` container that `cloudhal export` writes and
* `cloudhal import` reads. Node-only; nothing a Worker runs imports it.
*/
export { ArchiveError, type ArchiveErrorCode } from "./errors.ts";
export { type ArchiveSlot, classifyPath, type FileSlot, type PartKind } from "./layout.ts";
export { ARCHIVE_LIMITS, type ArchiveLimits, PART_SPLIT_BYTES } from "./limits.ts";
export {
type ArchiveCounts,
type ArchiveFile,
SITE_ARCHIVE_FORMAT,
type SiteArchiveManifest,
} from "./manifest.ts";
export { encodeNdjson, type NdjsonSplitOptions, splitNdjson } from "./parts.ts";
export type { ArchiveContent, ArchiveMediaSource } from "./plan.ts";
export { type ArchiveItem, type ReadArchiveOptions, readArchive } from "./read.ts";
export { archiveStream, type WriteArchiveOptions, writeArchiveFile } from "./write.ts";
Loading
Loading