diff --git a/.changeset/site-archive-export.md b/.changeset/site-archive-export.md index db6c7416..97b67529 100644 --- a/.changeset/site-archive-export.md +++ b/.changeset/site-archive-export.md @@ -9,4 +9,4 @@ New `cloudhal export` (RFC-0039, #47) writes one site of the local project to a - `--no-media` writes the graph only. - `--latest-only` keeps only the published and the latest snapshot, and only the media they and the current content need. It is refused while a field operation is unfinished (`cloudhal fields continue|revert`). -A damaged or missing local media file stops the export (run `cloudhal media verify`); so do Ctrl-C and SIGTERM. Either way no file is written and an existing one is left as it was. Exporting an unchanged site again with the same CLI and Node version gives the same bytes. An archive never contains user accounts, sessions, roles, agent keys, comments, readers, analytics, the audit log, automations, schedule runs, deployment secrets or webhook credentials. Its history does name who made each change (the user or agent of each snapshot, upload, block edit, review and takedown). +A damaged or missing local media file stops the export (run `cloudhal media verify`); so do Ctrl-C and SIGTERM. Either way no file is written and an existing one is left as it was. Exporting an unchanged site again with the same CLI and Node version gives the same bytes. An archive never contains user accounts, sessions, roles, agent keys, comments, readers, analytics, the audit log, scheduled releases, automations, schedule runs, pending approvals, open edit suggestions, deployment secrets or webhook credentials, and the export prints this list. Its history does name who made each change (the user or agent of each snapshot, upload, block edit, review and takedown). diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 00000000..439af45b --- /dev/null +++ b/.gitattributes @@ -0,0 +1,2 @@ +# RFC-0039 site archives are gzip-compressed tar files: never convert line endings, diff or merge them. +*.cloudhal binary diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c26dc43e..5ab92a8d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,6 +33,8 @@ jobs: - run: pnpm size:check - run: node scripts/font-fallbacks.ts --check - run: node scripts/mcp-tools-doc.ts --check + # RFC-0039 invariant 7: the committed demo archive's content (its manifest), not its gzip bytes. + - run: node scripts/demo-archive.ts --check - run: pnpm turbo run typecheck # Each vitest starts a worker per core; more than two packages at once oversubscribes the # 4-vCPU runner and slow tests (auth, D1, workerd) hit their timeouts. diff --git a/.gitignore b/.gitignore index b5b4a696..1b41aa15 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,8 @@ packages/core-contracts/schemas/.bundled/ # The starters' own local site (a project created from a starter commits its cloudhal.json) starters/*/cloudhal.json +# and its site archives (`cloudhal export` in a starter; a user project may commit its own) +starters/**/*.cloudhal # Package build output .build/ diff --git a/AGENTS.md b/AGENTS.md index d9df3bc2..730e4d61 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,6 +16,7 @@ cargo clippy --workspace --all-targets -- -D warnings pnpm gen # JSON Schema → TypeScript + Rust (requires cargo-typify 0.8.0) pnpm drift # gen + fail if generated files changed pnpm changeset # add a release note for a published package +pnpm demo:archive # regenerate the starter demo archive (RFC-0039; --check runs in pnpm check) pnpm design:lint [files…] # Kumo design rules for apps/studio + packages/ui blocks (also a Claude Code hook) pnpm design:shots -- # headless screenshots at 1440/390 px; fails on sideways scroll / console errors pnpm kumo ls|doc |blocks|add |ai # Kumo CLI: component list, API docs, blocks, usage guide @@ -30,7 +31,7 @@ pnpm i18n:export / pnpm i18n:import [--locale xx] # translations as nested ICU - `packages/core-ts` — `SiteGraphStore` (CAS on `rev`, snapshots, diff, publish, rollback). Behavioural suite in `src/testing.ts` runs on sqlite and Miniflare D1. `./local` opens the offline store; `seedStarterSite` seeds a demo. `./surfaces` is RFC-0020's generative UI: the catalog (primitives + block types), eval-free surface checks, bindings, repairs, JSON Patch (`patchPage`), `catalogPrompt`; `./interop` converts A2UI v0.9 and json-render to and from surfaces; `./design` is RFC-0021's DESIGN.md lint and guidance. `./fields` is RFC-0041's typed custom fields (definition and value checks, `resolveFields`, `formatFieldValue`, the agents' JSON Schema, the conversion table); `src/fields-operations/` runs rename/retype/localize/remove. `search.ts` is RFC-0019's FTS5 site search (index written with the publish batch; articles, pages, terms and authors, RFC-0029); `search-provider.ts` is RFC-0029's `SearchProvider` (`fts5`, `memory`, `withFallback`), set with `SiteGraphStoreOptions.search`; `search-ai.ts` is the experimental Cloudflare AI Search provider (`hybridProvider`, `searchProviderFromEnv`, `cloudhal.json` `search`, docs/guides/search.md). - `packages/schedules` — RFC-0008: time model (only `temporal-polyfill/implementation`, `now` injected, never `Temporal.Now`), RRULE subset, release model (`rl1:`), automation/run state, taint and budgets, `runDue`; `./sql` is `SqlScheduleStore`, `./engine` the Temporal-free part core-ts and the site Worker use. Worker-safe; conformance suites in `src/testing.ts` on sqlite and Miniflare D1. - `packages/astro-renderer` — `GraphSource` (live/snapshot), `cloudhal()` integration (virtual config + dev route-cache watcher), `.astro` section components (`Surface.astro` for RFC-0020 surfaces; `./surfaces` renders them, `./kit` has the table of contents and reading time; `./fields` and `EntryFields.astro` render typed field values, RFC-0041). Routes live in starters, not here. -- `packages/cli` — the `cloudhal` CLI (`init`, `dev`, `studio`, `mcp`, `deploy`, `push`, `pull`, …; every command is in `src/cli/help.ts`, which `cloudhal --help` prints and the docs site's CLI reference is generated from); TS bin run by Node 24 type stripping. `src/cli.ts` parses flags and dispatches through `COMMANDS`, built from `src/cli/commands-{local,remote,content,ops,site}.ts` (a new command is one handler plus one line in `src/cli/help.ts`). E2E test spawns the bin (Astro disables dev middleware under `VITEST`). Cloudflare tests run both Workers on workerd via wrangler's `createTestHarness`; test files run serially (shared starter caches). +- `packages/cli` — the `cloudhal` CLI (`init`, `dev`, `studio`, `mcp`, `deploy`, `push`, `pull`, `export`, `import`, …; every command is in `src/cli/help.ts`, which `cloudhal --help` prints and the docs site's CLI reference is generated from); TS bin run by Node 24 type stripping. `src/cli.ts` parses flags and dispatches through `COMMANDS`, built from `src/cli/commands-{local,remote,content,ops,site}.ts` (a new command is one handler plus one line in `src/cli/help.ts`). `src/archive/` is RFC-0039's `.cloudhal` container (tar, gzip, manifest, limits), read and written by `src/commands/export.ts` and `import*.ts` (`archive-bundle.ts`, `archive-reassemble.ts`); the starter demo archive `test/fixtures/site-archives/starter-blog.cloudhal` is written by `pnpm demo:archive`, and `pnpm check` fails when it is stale. E2E test spawns the bin (Astro disables dev middleware under `VITEST`). Cloudflare tests run both Workers on workerd via wrangler's `createTestHarness`; test files run serially (shared starter caches). - `packages/cloudflare` — the admin Worker (`createAdminWorker`); `./playground` is RFC-0036's playground Worker (`createPlaygroundWorker`, the `PlaygroundSite` Durable Object, signed session cookies). Generated entries must import `@cloudhal/cloudflare/workerd-prelude` first (ADR-0012). - `packages/agent-runtime` / `packages/mcp-server` — policy, skills, audit; MCP tools and transports (ADR-0011). `@cloudhal/mcp-server/render` is RFC-0021's `render_page` core: a small DevTools client, the in-page checks and limits; the CLI drives local Chrome with it (`render-chrome.ts`), the admin Worker Browser Rendering (`browser-render.ts`). - `starters/blog` — the theme boundary: `src/pages/[...path].astro`, `src/layouts`, `src/theme.ts`. diff --git a/README.md b/README.md index 126d0349..f377b044 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ packages/astro-renderer Astro 7 integration, live/snapshot data sources, secti packages/analytics cookie-less site analytics: beacon handler, WAE and SQLite stores, rollups packages/comments reader comments (RFC-0007): rules, proof of work, submission pipeline, SQLite/D1 store packages/schedules schedules (RFC-0008): time model, RRULE subset, releases, automations, taint, budgets, runDue, SQLite/D1 store -packages/cli the cloudhal CLI: init, dev, studio, mcp, deploy, push, pull, … (all commands: cloudhal --help, https://cloudhal.dev/docs/reference/cli) +packages/cli the cloudhal CLI: init, dev, studio, mcp, deploy, push, pull, export, import, … (all commands: cloudhal --help, https://cloudhal.dev/docs/reference/cli) packages/api Hono HTTP API + Better Auth + /mcp endpoint + agent API keys (Node and Workers) packages/agent-runtime policy engine, skill loader, audit log (SDK-free) packages/mcp-server MCP server: over 100 tools (README tool reference), approval gate, agent modes, skills as resources; stdio + HTTP diff --git a/apps/docs/src/lib/site-pages.ts b/apps/docs/src/lib/site-pages.ts index 84f2f4c5..fee7e7e3 100644 --- a/apps/docs/src/lib/site-pages.ts +++ b/apps/docs/src/lib/site-pages.ts @@ -89,6 +89,7 @@ const docsIndex: SitePage = { - [Typed custom fields](/docs/guides/fields): definitions, operations, languages and themes - [Structured data, expiry and listings](/docs/guides/structured-data): JSON-LD presets, entries that end on a date, listings by field and collection feeds - [Several sites in one deployment](/docs/guides/multi-site) +- [Site archives](/docs/guides/site-archives): backups, moving a site, templates; \`cloudhal export\` and \`import\` - [Analytics and privacy](/docs/guides/analytics-privacy) - [Translations with Weblate](/docs/guides/translations-weblate) diff --git a/apps/docs/test/repo-docs.test.ts b/apps/docs/test/repo-docs.test.ts index 2d089148..5e8c3187 100644 --- a/apps/docs/test/repo-docs.test.ts +++ b/apps/docs/test/repo-docs.test.ts @@ -161,7 +161,8 @@ describe("the CLI reference", () => { it("has a section per command of cloudhal --help, with its flags", async () => { const { cliReference } = await import("../src/lib/reference.ts"); const { body } = cliReference(); - for (const command of ["init", "dev", "studio", "mcp", "deploy", "push", "pull", "schedules"]) { + const commands = ["init", "dev", "studio", "mcp", "deploy", "push", "pull", "export", "import"]; + for (const command of [...commands, "schedules"]) { expect(body).toContain(`\n## ${command}\n`); } expect(body).toContain("cloudhal mcp [--mode plan|manual|edits|auto|bypass]"); diff --git a/docs/adr/0041-site-archives.md b/docs/adr/0041-site-archives.md index 3e15824a..01da5b93 100644 --- a/docs/adr/0041-site-archives.md +++ b/docs/adr/0041-site-archives.md @@ -3,6 +3,7 @@ - Date: 2026-10-22 (decided by Halil) - Amendment (2026-10-25, RFC-0039 rollout step 1): deterministic bytes hold for the same Node/zlib build; the tar stream and the manifest's SHA-256s on every platform. See rule 6. - Amendment (2026-10-26, RFC-0039 rollout step 4): `--replace` takes the archive's published snapshot (a deployment keeps its own on `push`) and is undone with `cloudhal import --revert `, which restores the `before import` snapshot. See decision 4. +- Amendment (2026-10-26, RFC-0039 rollout step 5): the committed demo archive is checked through its manifest without `createdBy`. The graph rows are held in memory, and import writes the node set in one batch of at most 1000 statements, as `push` and `pull`; only the container and the media stream. This corrects the risk below that says memory stays flat. Both are in RFC-0039's Later. Decision 5's list also leaves out scheduled releases, pending approvals and open edit suggestions, which live outside the graph like automations. - Related: RFC-0039, RFC-0004, RFC-0010, ADR-0002, #47 - Sources: [Sanity datasets](https://www.sanity.io/docs/cli-reference/dataset) · [Strapi data management](https://docs.strapi.io/cms/data-management/export) · [Contentful CLI export](https://github.com/contentful/contentful-cli/tree/main/docs/space/export) · [Ghost migration](https://docs.ghost.org/migration/ghost) · [WordPress export](https://wordpress.org/documentation/article/tools-export-screen/) @@ -34,4 +35,4 @@ A site's content can only leave a project through a deployment (`push`, `pull`), - The format is versioned and must be read forever. - Risks: - A file is as trustworthy as its source until archives are signed. - - Large media sets make big files; `--no-media` and the parts limit keep memory flat, not size small. + - Large media sets make big files; `--no-media` and the parts limit keep memory flat, not size small (amended: flat for the media; the graph rows are held in memory). diff --git a/docs/guides/fields.md b/docs/guides/fields.md index 928e2509..0a724bbe 100644 --- a/docs/guides/fields.md +++ b/docs/guides/fields.md @@ -29,7 +29,7 @@ Collections without definitions keep `fields` as free-form JSON, exactly as befo - **Studio:** open the collection's settings, then **Fields**. Add fields (the key follows the label until you save), reorder them, and edit labels, help, limits and options. Only people who may change site settings can edit fields. Entries of that collection then get a **Fields** card in the editor, one control per type. - **Agents (MCP):** `list_collections` returns each collection's `fields` and `fieldsSchema` (the JSON Schema of the values, with labels and help in the caller's language). `update_collection` changes definitions, and `create_entry` / `update_entry` take `fields` (`null` removes a value). Definition changes always wait for a person who may change site settings, in every mode. - **API:** `PUT /api/sites/:siteId/collections/:key` with `fields` in the settings; entry routes take `fields` like any other property. Invalid values answer `422 invalid_fields` with one issue per problem (`fields/startsAt/at`). -- **Files:** definitions travel in the site node, values in entries, so `cloudhal push`/`pull` and archives carry them unchanged. +- **Files:** definitions travel in the site node, values in entries, so `cloudhal push`/`pull` and [site archives](site-archives.md) carry them unchanged. ### Changes that touch values: field operations diff --git a/docs/guides/site-archives.md b/docs/guides/site-archives.md new file mode 100644 index 00000000..4bc749e5 --- /dev/null +++ b/docs/guides/site-archives.md @@ -0,0 +1,428 @@ +# Site archives: a guide for site owners and operators + +`cloudhal export` writes one site of a local project to a single `.cloudhal` file: its content graph, snapshots and media. `cloudhal import` reads it back into a project, keeping the site's id. + +Use it to back up a site, to move a site to another machine without a deployment, to keep a site's content in git next to its theme, and to ship a demo site with a theme. An archive is the site's content and its edit history. It is not a copy of the database: the people who use Studio and what readers do stay where they are. + +The design and its reasons are in [RFC-0039](../rfcs/0039-site-archive-export-import.md) and [ADR-0041](../adr/0041-site-archives.md); the security analysis is in the [threat model](../security/threat-model.md). + +> **Status.** `cloudhal export`, `cloudhal import` with `--dry-run` and `--replace`, and `cloudhal import --revert` work on local projects. Exporting from a deployment, a Studio download, `cloudhal init --from`, `--new-ids` and signed archives come later ([Later](#later)). + +## What an archive holds + +A `.cloudhal` file is a gzip-compressed tar. Its entries always come in this order: + +| Entry | What it holds | +|---|---| +| `manifest.json` | The header: the archive format, the contracts' `schemaVersion`, the bundle format, the site's id and name, the CLI version that wrote it, a date, the counts, and every other file with its SHA-256 and size | +| `project.json` | The project row, when the site has one | +| `nodes/0001.ndjson` … | The site's nodes (site, pages, entries, taxonomies, terms, menus, redirects, media records), one per line, sorted by id; a new part every 8 MiB | +| `snapshots/0001.ndjson` … | The snapshots, by creation time | +| `blobs/0001.ndjson` … | The node bodies the snapshots refer to, as `{hash, body}` | +| `media/` | One file per media file, the raw bytes, named by their SHA-256 | +| `ledger.ndjson` | Taken-down content: what was taken down, why, by whom and when; never the bytes | + +The rows are those `cloudhal push` and `pull` send (`cloudhal.site-bundle@1`), so an archive adds a container, not a second data model. The starter demo archive, listed with `tar`: + +``` +$ tar -tzvf starter-blog.cloudhal +-rw-r--r-- 0 0 0 984 Jan 1 1970 manifest.json +-rw-r--r-- 0 0 0 389 Jan 1 1970 project.json +-rw-r--r-- 0 0 0 7388 Jan 1 1970 nodes/0001.ndjson +-rw-r--r-- 0 0 0 1690 Jan 1 1970 snapshots/0001.ndjson +-rw-r--r-- 0 0 0 5751 Jan 1 1970 blobs/0001.ndjson +``` + +The start of its manifest (then `files`, each with its `path`, `sha256` and `bytes`): + +```json +{ + "format": "cloudhal.site-archive@1", + "contracts": { + "schemaVersion": 1 + }, + "bundle": "cloudhal.site-bundle@1", + "siteId": "site_starter0001", + "siteName": "Starter Blog", + "createdBy": "0.6.1", + "createdAt": "2026-10-26T00:00:00.000Z", + "counts": { + "nodes": 9, + "snapshots": 1, + "blobs": 9, + "media": 0, + "mediaBytes": 0 + }, +``` + +**Never in an archive:** user accounts, sessions, roles, agent keys, comments, readers, analytics, the audit log, scheduled releases (with their pinned versions), automations, schedule runs, pending approvals, agents' open edit suggestions, deployment secrets and webhook credentials. `export` reads none of their tables or files, and prints this list every time. An archive is never `data.db` and never holds `.cloudhal/deploy/`. Reader comments move separately, with [`cloudhal comments export` and `import`](comments.md#moving-comments-cloudhal-comments). + +**What is scheduled stays behind.** Releases live outside the content graph, as with `push` and `pull` ([schedules](schedules.md#deployment-notes)). An article scheduled for Monday arrives as it is now, a draft or live, and nothing publishes it on Monday: schedule it again after importing. Embargoes and expiries are part of the content, so they travel and still apply. Decide pending approvals and open suggestions in Studio before exporting; they are not in the file. + +**What the history says about people.** The content graph names who made each change, as its contracts require. Snapshots, uploads, block edits, reviews, term proposals and takedowns carry the id of their user or agent, most also a display name. Entries name their authors, and the project row its owner's id. Treat an archive like the published site plus its edit history: give it to people you would show that history to. `--latest-only` drops older snapshots, not the names in the ones it keeps. + +## Export: `cloudhal export` + +``` +cloudhal export [--out ] [--site ] [--no-media] [--latest-only] +``` + +- **`--out `**: where to write, relative to the project. A file already there is replaced, but only once the new one is complete. A failed or interrupted export (Ctrl-C, SIGTERM) leaves no file, and an existing one stays as it was. +- **No `--out`**: the file is `-.cloudhal` in the current folder. The date is that of the newest row the archive holds, never the clock, so an unchanged site gets the same name. A deletion alone does not move the date; the next edit or snapshot does. A file already at that name is replaced only by the same archive; anything else (a later edit the same day, other flags, another CLI version) is refused, and the export asks for `--out`. +- **`--site `**: another site of the project. Only sites listed in `cloudhal.json` `sites` are accepted. One archive holds one site. +- **`--no-media`**: the graph only. The manifest says `mediaOmitted`, and an import then needs the media from elsewhere. +- **`--latest-only`**: only the published and the latest snapshot, the node bodies they refer to, and the media they and the current content need. For templates that should not carry their history. Refused while a field operation is unfinished (`cloudhal fields continue|revert`), because it needs its `before` snapshot. + +``` +$ cloudhal export --out backups/full.cloudhal +wrote backups/full.cloudhal (4.4 KiB) +site site_3717MDRN27BXNB6BX17V (demo-blog): 10 node(s), 4 snapshot(s), 15 blob(s), 1 media file(s) (80 B) +not in the archive (RFC-0039 §4): user accounts, sessions, roles, agent keys, comments, readers, analytics, the audit log, scheduled releases, automations, schedule runs, pending approvals, open edit suggestions, deployment secrets, webhook credentials +in the archive: who made each change: snapshots, uploads, block edits, reviews and takedowns name their user or agent (by id, most also by display name) +``` + +With `--latest-only` and `--no-media`, the summary says what was left out (the two last lines above follow, as always): + +``` +$ cloudhal export --latest-only --out backups/latest.cloudhal +wrote backups/latest.cloudhal (4.1 KiB) +site site_3717MDRN27BXNB6BX17V (demo-blog): 10 node(s), 2 snapshot(s), 12 blob(s), 1 media file(s) (80 B) +history: the published and the latest snapshot only (--latest-only) + +$ cloudhal export --no-media --out backups/graph.cloudhal +wrote backups/graph.cloudhal (4.3 KiB) +site site_3717MDRN27BXNB6BX17V (demo-blog): 10 node(s), 4 snapshot(s), 15 blob(s) +media left out (--no-media): an import needs the media from elsewhere +``` + +A media file whose local copy is missing, or whose bytes no longer match its record, stops the export: run `cloudhal media verify` to find them, or export with `--no-media`. The graph rows are read into memory; the media stream from the blob store. + +## Import: `cloudhal import` + +``` +cloudhal import [--replace] [--dry-run] +cloudhal import --revert [--dry-run] +``` + +The file is untrusted until it is checked, so the import works in two steps: + +1. **Verify and plan.** It reads the whole file and checks every file's size and SHA-256, every row and every node body, then compares the archive with the project. Every refusal of the archive or the project comes here, before anything is written. +2. **Write.** When media must be copied, it reads the file again and requires the same manifest, so the content is the one it planned. It copies the media the project lacks, then writes the graph. + +### Bootstrap, the default + +A bootstrap import brings a site into a project that does not hold it yet, or holds exactly the archive's nodes. The archive's site must be `cloudhal.json`'s `siteId` or listed in `sites`. Missing snapshots, node bodies and media are added; a site whose nodes differ is refused. Nothing local is ever erased first. + +Look first with `--dry-run`. It prints the manifest and what the import would add, change or remove, and writes nothing; in a project without a store it creates none. It exits 0, or 1 when the import would be refused. Here, the demo archive in an empty project whose `cloudhal.json` names `site_starter0001`: + +``` +$ cloudhal import starter-blog.cloudhal --dry-run +verified starter-blog.cloudhal: 5 file(s), 2.9 KiB +archive starter-blog.cloudhal: cloudhal.site-archive@1 (contracts 1), written by @cloudhal/cli 0.6.1 at 2026-10-26T00:00:00.000Z +site site_starter0001 "Starter Blog" (starter-blog): 9 node(s), 1 snapshot(s), 9 node bodies, 0 media file(s) (0 B) +dry run: bootstrap import into site site_starter0001, which is new to this project +project: proj_starter0001 is new to this project +nodes: 9 to add (1 site, 3 taxonomies, 2 terms, 2 pages, 1 entry), 0 to change, 0 to remove + + site "Starter Blog" + + taxonomy "Categories" + + taxonomy "Tags" + + taxonomy "Authors" + + term "News" + + term "Cloudhal" + + page "Starter Blog" (/) + + page "About" (/about) + + entry "Hello, world" (posts/hello-world) +snapshots: 1 to add (snap_starter0001), with 9 node bodies +media: none +nothing written (dry run) +``` + +Then the import, and the same import again: + +``` +$ cloudhal import starter-blog.cloudhal +verified starter-blog.cloudhal: 5 file(s), 2.9 KiB +imported starter-blog.cloudhal: site site_starter0001 (starter-blog) is in this project +nodes: 9 written; snapshots: 1 added; node bodies: 9 added + +$ cloudhal import starter-blog.cloudhal +verified starter-blog.cloudhal: 5 file(s), 2.9 KiB +nothing to import: this project already holds starter-blog.cloudhal (site site_starter0001, starter-blog) +``` + +The site keeps the archive's published snapshot, as with `cloudhal pull`. An archive with media also prints `copying 1 media file(s) (80 B)` and `media: 1 file(s) copied (80 B), 0 already here`. An archive written with `--no-media` imports only when the project already has its media. + +Ctrl-C or SIGTERM stops the import before the graph is written. Media already copied stay in the blob store, unreferenced, and the next import reuses them. If storage fails while the graph is written, the message says the import stopped partway; run it again to finish it, search index included. + +### Replace: `--replace` + +`--replace` makes the local site's nodes the archive's, its published snapshot included, as `cloudhal pull --replace` does. Snapshots are only ever added. When it changes a site the project already holds, it saves the current graph first as a snapshot labelled `before import `, and the result as `import `: + +``` +$ cloudhal import starter-blog.cloudhal --replace +verified starter-blog.cloudhal: 5 file(s), 2.9 KiB +imported starter-blog.cloudhal: site site_starter0001 (starter-blog) is in this project, its nodes replaced by the archive's +nodes: 9 written (9 replaced); snapshots: 0 added; node bodies: 0 added +saved first: snapshot snap_VKERSCJMDWW29T2J955K "before import starter-blog.cloudhal" holds the site as it was (undo: cloudhal import --revert snap_VKERSCJMDWW29T2J955K); snap_4ABNRF7ZP9VX8V769KYB "import starter-blog.cloudhal" holds the result +note: 1 node(s) took the archive's rev, no higher than theirs here; reload Studio tabs open on this site: an edit a tab or agent still holds at that rev saves over them without a conflict +``` + +- **The published snapshot** becomes the archive's. The summary says so when it moves: `published: snapshot snap_YJZYHFS2C6X3V4KDG1WM, as the archive has it (was: snap_SW31VHYJDHB0EZ5H5N0D)`. A deployment keeps its own: `cloudhal push` never moves production, so publish there in Studio. +- **Revs.** The archive's nodes keep their revs. A Studio tab or agent still holding a node at that rev could save over it without a conflict, so the summary counts those nodes. Reload open Studio tabs after a replace. +- **Refused before anything is written:** open releases of the site (cancel them in Studio; there is no `--cancel-schedules` here), an unfinished field operation, and a hostname another site of the local store serves. +- **No snapshot** is taken for a site that is new to the project or unchanged: there is nothing to undo. + +### Undo: `--revert` + +`cloudhal import --revert ` brings the site back to a `before import` snapshot, its published snapshot included. It saves the current graph first as `before revert import `, so a revert can be reverted too. Its revs only move forward: a node edited since comes back at the rev after its current one, so a stale editor gets a conflict. + +``` +$ cloudhal import --revert snap_VKERSCJMDWW29T2J955K --dry-run +revert to snapshot snap_VKERSCJMDWW29T2J955K "before import starter-blog.cloudhal" (site site_starter0001, 2026-10-26T01:58:37.835Z) +dry run: replace the site's nodes +nodes: 0 to add, 1 to change (1 page), 0 to remove + ~ page "About me" (/about), rev 0 here, 1 after the revert +saved first: this project's site (9 nodes) as snapshot "before revert import starter-blog.cloudhal", the result as "revert import starter-blog.cloudhal" (undo: cloudhal import --revert ) +nothing written (dry run) + +$ cloudhal import --revert snap_VKERSCJMDWW29T2J955K +reverted to snapshot snap_VKERSCJMDWW29T2J955K "before import starter-blog.cloudhal" (site site_starter0001, 2026-10-26T01:58:37.835Z) +nodes: 9 written (9 replaced) +saved first: snapshot snap_Q2J3YG5VQK2Z6F1004V4 "before revert import starter-blog.cloudhal" holds the site as it was (undo: cloudhal import --revert snap_Q2J3YG5VQK2Z6F1004V4); snap_4CHHT1F20DAY5FT8YH0B "revert import starter-blog.cloudhal" holds the result +``` + +Studio's rollback cannot undo a replace: it only moves the published pointer to an earlier published snapshot, and a `before import` snapshot is not one. + +| | `--dry-run` | `--replace` | `--revert ` | +|---|---|---|---| +| Writes | nothing | the archive's nodes, missing snapshots, bodies and media | the snapshot's nodes | +| Snapshot first | shows the one it would take | `before import ` | `before revert import ` | +| Published snapshot | shows the change | the archive's | the snapshot's | +| Revs | shows the nodes at risk | the archive's | moved forward | +| `cloudhal.json` | shows the switch | switched for another site | never changed | + +### Another site: the site id switch + +A bootstrap import of another site's archive is refused, and the message names `--replace`: + +``` +$ cloudhal import starter-blog.cloudhal +cloudhal: cannot import starter-blog.cloudhal: it holds site site_starter0001, and this project's site is site_H3R0HM6VPH17HVBJPQ6C (cloudhal.json siteId); `cloudhal import --replace` imports it and switches cloudhal.json siteId to site_starter0001 (site_H3R0HM6VPH17HVBJPQ6C stays in the local store, unlisted); nothing was imported +``` + +With `--replace`, the archive's site is imported and `cloudhal.json` is switched to it once the import has landed. Only `siteId` (or the old id's place in `sites`) changes; the rest of the file stays as written. Here, the demo archive in a project fresh from `cloudhal init`: + +``` +$ cloudhal import starter-blog.cloudhal --replace +verified starter-blog.cloudhal: 5 file(s), 2.9 KiB +imported starter-blog.cloudhal: site site_starter0001 (starter-blog) is in this project +nodes: 9 written; snapshots: 1 added; node bodies: 9 added +cloudhal.json: siteId site_H3R0HM6VPH17HVBJPQ6C → site_starter0001 (undo: write site_H3R0HM6VPH17HVBJPQ6C back in place of site_starter0001 by hand; --revert does not); site site_H3R0HM6VPH17HVBJPQ6C stays in the local store, unlisted and untouched (`cloudhal site delete site_H3R0HM6VPH17HVBJPQ6C --confirm my-blog` deletes it for good) +``` + +- **The old site** stays in the local store, unlisted and untouched. Writing its id back into `cloudhal.json` by hand undoes the switch; `--revert` restores nodes only. +- **`cloudhal site delete --confirm `** deletes the old site for good. +- **A deployed project** is warned: its deployment serves the old site, and `cloudhal push` would send the new one there as another site. + +## Recipes + +### Back up a site + +```bash +cloudhal export --out ~/backups/news-2026-10-26.cloudhal +``` + +Keep backups outside the project folder. To restore into a project whose `cloudhal.json` names the site, run `cloudhal import ` when the project has lost the site, or `cloudhal import --replace` to go back over newer local work (undo: `--revert`, with the id it prints). Check either with `--dry-run` first. + +A backup holds the content, not what was scheduled or waiting: releases, pending approvals and open suggestions are not in it. After a restore, schedule the releases again. + +### Move a site to another machine + +1. On the old machine: `cloudhal export --out site.cloudhal`. +2. On the new one: clone or copy the project (the theme and `cloudhal.json`), install its dependencies (`npm install`, `pnpm install`, …), then run `cloudhal import site.cloudhal`. +3. Run `cloudhal dev` or `cloudhal studio` as usual. Import first: until then the new project's store has no site. + +An imported site exports back to the same file: the same bytes with the same CLI and Node build. What does not move: users, reader comments (`cloudhal comments`), analytics, scheduled releases, pending approvals, open suggestions and `.cloudhal/deploy/`. Decide the approvals and suggestions in Studio before exporting. Schedule the releases again on the new machine. Cancel them on the old one if it keeps running `cloudhal dev` or `studio`: its scheduler still runs them. `cloudhal push` and `deploy` from the new machine need the deployment token in `.cloudhal/deploy/secrets.json`, which is never in an archive ([deploying to Cloudflare](../../packages/cli/README.md#deploying-to-cloudflare-adr-0012)). + +### Keep a site's content in the repository + +This replaces content written as code ([#47](https://github.com/cloudhal-dev/cloudhal/issues/47)). The project's `.gitignore` keeps `.cloudhal/` out of git, but not an archive: + +```bash +cloudhal export --out content/site.cloudhal # after editing; commit the file +cloudhal import content/site.cloudhal # in a fresh clone, before working +``` + +Export again after every round of edits. An unchanged site gives the same bytes on the same CLI and Node build, so git sees no change. An edited site gives a new binary file: git stores it whole and cannot diff it (look with `cloudhal import --dry-run`). + +A clone that already imported an earlier version refuses the plain import once `git pull` brings a newer one (`differs from the archive in node(s)`). There, run `cloudhal import content/site.cloudhal --replace`: `--dry-run` first shows what changes, local work is saved as a `before import` snapshot, and `--revert` undoes it. Each such replace adds two snapshots (`before import` and `import`), which the next export carries. Let one person export at a time: git cannot merge two versions of the file, so the second commit replaces the first person's edits. + +[Workers Builds](deploy-button.md) does not read archives: a build without a store seeds the starter site. Content reaches a deployment from a local project, through `cloudhal deploy` (the first one pushes the local site) or `cloudhal push`. + +### Templates and starters + +A theme can ship an archive of its demo site. Its users run `cloudhal init`, then `cloudhal import demo.cloudhal --replace`, which switches the new project to the demo's site. Export the demo with `--latest-only` so it does not carry its history; the snapshots it keeps still name their author. + +Ids are kept, so every project that imports one template holds the same site id. That is right for a backup and wrong for a starter used twice in one installation: two projects made from it and pushed to one installation are the same site there, not two. Importing as a copy with new ids (`--new-ids`) and `cloudhal init --from ` come later. + +### Try it: the starter demo archive + +The repository has a small demo archive, [starter-blog.cloudhal](../../packages/cli/test/fixtures/site-archives/starter-blog.cloudhal): the site `cloudhal init` seeds, published once (`site_starter0001`, no media). It is regenerated with `pnpm demo:archive`, and `pnpm check` fails when it no longer matches what the code exports. + +```bash +curl -LO https://github.com/cloudhal-dev/cloudhal/raw/main/packages/cli/test/fixtures/site-archives/starter-blog.cloudhal +npm create cloudhal@latest my-site +cd my-site +npm install +npx cloudhal import ../starter-blog.cloudhal --replace --dry-run # then without --dry-run +``` + +## Takedowns + +Taken-down content stays down. The ledger travels in the archive; the taken-down bytes never do. + +- **Export** leaves out the bytes and says how many: `…, 0 media file(s) (0 B), 1 taken down`. +- **Import** records the archive's takedowns and deletes local copies of that content: `media: 1 taken down, never imported; 1 of them deleted from this project`. A dry run says so first. +- **An archive carrying content this project has taken down is refused:** `the archive carries 1 media file(s) this project has taken down (first ); an import never brings taken-down content back`. +- **`--revert`** brings the media record back, not its bytes. + +## Same content, same file + +The writer uses a fixed order, zero timestamps and fixed modes in the tar headers, and dates the manifest by the newest row, never the clock. Exporting an unchanged site twice gives the same bytes with the same CLI and Node build. Another zlib build compresses the same tar differently (Homebrew's Node and the official Node binaries do), so the gzip bytes can differ between machines. The tar inside, and so every SHA-256 in the manifest, is the same everywhere. Compare archives through it. Here, an archive and the export of a project it was imported into: + +``` +$ gunzip -c re.cloudhal | shasum -a 256 +bcf08446a1e2c25f0c9e32c7e6993dbac1989b21ff22b111b4ee800b04da3e29 - +$ gunzip -c content/site.cloudhal | shasum -a 256 +bcf08446a1e2c25f0c9e32c7e6993dbac1989b21ff22b111b4ee800b04da3e29 - +$ tar -xOzf content/site.cloudhal manifest.json # the manifest alone +``` + +`createdBy`, the CLI version, is part of the manifest, so a new CLI version changes the tar of an unchanged site. The repository's demo archive is checked by its manifest without `createdBy`, for that reason. + +## Untrusted files (security notes) + +An archive from someone else is untrusted input. `import` runs nothing in it and checks, before anything is written: + +- **the layout:** only the paths in the table above, in that order. No `..`, absolute paths, links or devices; +- **every file** against the manifest's SHA-256 and size, and every media file against its own name; +- **every row** against fixed field tables, and every node body against the contracts' precompiled validators: each body must be the node its snapshot refs name, of the archive's site; +- **time gates** (embargoes and expiries) by the same rules as for `cloudhal push`; +- **the format:** a newer archive format, contracts `schemaVersion` or bundle format is refused, naming the CLI version needed; +- **the file itself:** it must not change between the two reads. + +The caps: + +| What | At most | +|---|---| +| `manifest.json` | 16 MiB | +| `project.json`, `ledger.ndjson`, each `nodes`, `snapshots` or `blobs` part | 16 MiB (the writer starts a new part at 8 MiB) | +| Each media file | 100 MiB, the upload limit | +| The whole archive, decompressed | 32 GiB | +| Entries | 100,000 | +| Parts of each kind | 9,999 | + +**Not signed.** An archive is as trustworthy as where it came from; a valid archive can still hold content you did not write. Signed archives come later. + +**Not encrypted.** An archive holds no secrets and no personal data beyond the names in its history, but it does hold unpublished drafts. To keep it confidential, encrypt the file after exporting, and decrypt it to a file before importing: + +```bash +age -p -o site.cloudhal.age site.cloudhal # or: gpg --symmetric --output site.cloudhal.gpg site.cloudhal +age -d -o site.cloudhal site.cloudhal.age # or: gpg --output site.cloudhal --decrypt site.cloudhal.gpg +``` + +`cloudhal import` takes a file, not standard input: `cloudhal import -` answers `no such file`. A pipe such as `/dev/stdin` fails as soon as the import has media to copy, since it reads the file again (`not a Cloudhal site archive (it is empty)`). + +## Limits + +- **Local projects only.** Both commands read and write the project's own store. To bring an archive into a deployment, import it locally, then `cloudhal push`. +- **One archive, one site.** A project with several sites exports them one at a time (`--site`). +- **Ids are kept.** There is no import as a copy yet. +- **Memory.** The graph rows are held in memory on both sides; only the media stream. Sites with very many nodes need memory to match. +- **One atomic batch.** The node set is written in one atomic batch of at most 1,000 statements, about one per node, the same limit `push` and `pull` have. A larger site is refused before anything is written: `site has too many nodes () for one atomic copy`. +- **No partial export.** A collection alone, or a site without its snapshots, comes later. + +## Troubleshooting + +An error starts with `cloudhal:`. An import refusal reads `cloudhal: cannot import : ; nothing was imported`, and a revert refusal `cloudhal: cannot revert to : ; nothing was reverted`. A dry run prints each refusal of the project as a line `would refuse: ` instead, and exits 1. Here, the demo archive in a project of another site: + +``` +$ cloudhal import starter-blog.cloudhal --dry-run +verified starter-blog.cloudhal: 5 file(s), 2.9 KiB +archive starter-blog.cloudhal: cloudhal.site-archive@1 (contracts 1), written by @cloudhal/cli 0.6.1 at 2026-10-26T00:00:00.000Z +site site_starter0001 "Starter Blog" (starter-blog): 9 node(s), 1 snapshot(s), 9 node bodies, 0 media file(s) (0 B) +would refuse: it holds site site_starter0001, and this project's site is site_3717MDRN27BXNB6BX17V (cloudhal.json siteId); `cloudhal import --replace` imports it and switches cloudhal.json siteId to site_starter0001 (site_3717MDRN27BXNB6BX17V stays in the local store, unlisted) +nothing written (dry run) +``` + +A damaged file is refused in a dry run too, with the `cannot import` line. The tables quote the reason. + +**Export** + +| Message | What to do | +|---|---| +| ``site is not listed in cloudhal.json `sites` (this project's sites: …)`` | `--site` takes only sites listed in `sites`. | +| `site is listed in cloudhal.json but the local store (.cloudhal) has no such site` | The id in `cloudhal.json` is wrong, or the site was never imported here. | +| ``local media is missing; run `cloudhal media verify` (or export with --no-media)`` | A media file is gone from `.cloudhal/blobs/`. Run `cloudhal media verify`; restore the file, or export with `--no-media`. | +| `media/: the media source has of bytes; the local copy is damaged: …`, or `is longer than bytes`, or `does not match its SHA-256` | The same, for a file cut short, grown or changed. | +| ` already exists and is not this archive (…); pass --out to replace it, or --out to keep both` | The default name is taken by another archive. | +| `--out is a directory; name a file` | Give `--out` a file name. | +| `field operation on has not finished, and --latest-only would drop the snapshot it needs; …` | Finish or revert it (`cloudhal fields continue\|revert `), or export the full history. | +| `cloudhal import does not take --no-media (only cloudhal export does)` | `--site`, `--no-media` and `--latest-only` belong to `export`. | + +**The file** + +| Reason | What to do | +|---|---| +| `not a Cloudhal site archive (not gzip)`, or `(manifest.json is not its first entry)` | It is not an archive `cloudhal export` wrote. | +| `the archive is truncated (gzip ends early); the file is damaged or incomplete, export it again` | The file is cut short: copy or download it again. | +| `nodes/0001.ndjson: its SHA-256 does not match manifest.json (the file is damaged or was edited); the file is damaged or incomplete, export it again` | A byte changed. Get an intact copy. | +| `…; this file was not written by cloudhal export` | It holds a path, link or device outside the layout. | +| `…; it is over this CLI's archive limits` | It exceeds a cap above. | +| `this archive was written by a newer Cloudhal (archive format cloudhal.site-archive@2); update @cloudhal/cli to or later` | Update the CLI to the version that wrote it. The same for a newer `contracts schemaVersion` or bundle format. | +| `no such file` | The path is wrong; `-` is not standard input. | +| `the file changed while it was being imported` | Something wrote to the file during the import. Import a copy that nobody writes to. | + +**The project** + +| Reason | What to do | +|---|---| +| ``it holds site , and this project's site is (cloudhal.json siteId); `cloudhal import --replace` imports it and switches …`` | It is another site's archive: use `--replace` ([the switch](#another-site-the-site-id-switch)), or import it into a project that names it. | +| `this project's site differs from the archive in node(s) (…); bootstrap imports only into a project without the site or holding exactly its nodes; …` | The local site has other content. `--replace --dry-run` shows what would change. | +| `it holds site , which cloudhal.json cannot name (site_ and 6 to 32 letters or digits)` | The archive's site id is not one a project can use. | +| `cloudhal.json cannot be switched in place: it names in more than its siteId and sites values, or writes it escaped; set cloudhal.json siteId to by hand …` | `--replace` edits only those values. Set `siteId` to the archive's site by hand, then import again. | +| `the archive has no media (written with --no-media) and this project lacks of the media file(s) it needs (first ); copy them into .cloudhal/blobs/media// or import an archive with media` | Copy the media in, or export again without `--no-media`. | +| `the archive carries media file(s) this project has taken down (first ); an import never brings taken-down content back` | The archive still holds content taken down here: it was exported before the takedown, or from a copy without it. Import a newer archive. | +| `site has open release(s) (…), and --replace replaces the graph their pinned versions point into; cancel them in Studio …` | Cancel the releases in Studio (or with the MCP tool `cancel_schedule`), then import again. | +| `field operation on has not finished, and --replace would drop its marker; …` | Run `cloudhal fields continue\|revert `, then import again. | +| `the archive's site serves , which site of this project's store already serves; …` | Remove the hostname from that site's domains in Studio, or delete that site if it is a leftover (`cloudhal site delete --confirm `). | +| `site has too many nodes () for one atomic copy` | See [Limits](#limits). | + +**Interruptions and partial writes** + +| Message | What it means | +|---|---| +| `export interrupted; no file written, an existing one left as it was` | Ctrl-C or SIGTERM during an export. | +| `import interrupted; nothing was imported` | Ctrl-C or SIGTERM before the graph was written. Media already copied stay and are reused. | +| `…; the import stopped after it began writing, so this project may hold part of it; run it again to finish` | A storage failure while writing. Run the same import again. With `--replace`, the message also names the `before import` snapshot and its `--revert`. | +| `imported , but cloudhal.json changed while importing, so it still names site ; …` | The import landed, the switch did not. Run it again, which only switches, or set `siteId` by hand. | + +**Revert** + +| Reason | What to do | +|---|---| +| `it is not a "before import" snapshot of this project (…)` | `--revert` takes only the ids `--replace` and `--revert` print. | +| `it is a snapshot of site , which cloudhal.json does not list (siteId and sites)` | A site id switch moved the project to another site. List both ids in `sites` (`"sites": ["", ""]`), or write the old one back as `siteId`, then revert. | +| `this project lacks or has a damaged copy of media file(s) the snapshot needs (first )` | A file in `.cloudhal/blobs/media//` was deleted or changed (`cloudhal media verify` lists it). Copy it back, or import a backup archive of the site as it is now, which copies only the missing media; then revert. | +| `this project has no local store` | Run it in the project folder that took the snapshot. | + +## Later + +Decided, not built yet ([RFC-0039 › Later](../rfcs/0039-site-archive-export-import.md#later-decided-not-in-v1)): +- `cloudhal init --from `: start a project from an archive; +- `--new-ids`: import as a copy with new ids, for templates; +- partial export: `--collection ` and `--no-snapshots`; +- export from a deployment (`--from `) and a Studio "Download site archive"; +- signed archives; +- a `--dir` layout (the same files, unpacked); +- graph rows that stream on both sides, and imports larger than one batch. diff --git a/docs/rfcs/0039-site-archive-export-import.md b/docs/rfcs/0039-site-archive-export-import.md index 896aa70d..86861f93 100644 --- a/docs/rfcs/0039-site-archive-export-import.md +++ b/docs/rfcs/0039-site-archive-export-import.md @@ -1,8 +1,16 @@ # RFC-0039: Site archives: `cloudhal export` and `cloudhal import` -- Status: Accepted (2026-10-22, by Halil; ADR-0041). Open questions decided as recommended. Revised before acceptance after a comparison with Sanity, Strapi, Contentful, Ghost, WordPress, Directus and Webflow (see Prior art): NDJSON parts instead of one JSON file, per-file checksums, contract versions in the manifest, and a snapshot before `--replace`. +- Status: Accepted (2026-10-22, by Halil; ADR-0041). Open questions decided as recommended. Revised before acceptance after a comparison with Sanity, Strapi, Contentful, Ghost, WordPress, Directus and Webflow (see Prior art): NDJSON parts instead of one JSON file, per-file checksums, contract versions in the manifest, and a snapshot before `--replace`. Implemented: rollout steps 1–5 (#47). - Amendment (2026-10-25, rollout step 1): identical `.cloudhal` bytes need the same Node/zlib build; the tar stream inside, and so every SHA-256 in the manifest, is identical on every platform. See invariant 7. - Amendment (2026-10-26, rollout step 4): `--replace` takes the archive's published snapshot (the site node carries it, as `pull --replace`; `push` keeps a deployment's own); it saves `before import ` and the result as `import `; the "rollback" that undoes it is `cloudhal import --revert `, since Studio's rollback moves only the published pointer, to an ancestor; and `cloudhal site delete` already deletes a local site. See §2 and invariant 6. +- Amendment (2026-10-26, rollout step 5): + - The demo archive is `packages/cli/test/fixtures/site-archives/starter-blog.cloudhal`, the site `cloudhal init` seeds with fixed ids and a fixed clock, published once. `scripts/demo-archive.ts` regenerates it, and its `--check` (in `pnpm check` and CI) compares the manifests of the committed and a fresh export without `createdBy`: the manifest pins every other file's SHA-256, so this compares the decompressed tar's content (invariant 7) without failing on every CLI version bump. + - §1's "neither export nor import has to hold the whole site in memory" holds for the container and the media; the graph rows are held in memory, since `exportSite` and `importSite` take whole bundles. + - Invariant 6's atomic batches are one batch of at most 1000 statements, the same limit as `push` and `pull`; a larger site is refused before anything is written. Every refusal comes before the first write, and the node set is one atomic batch, but node bodies, snapshots and the media ledger are written in batches of their own. A storage failure between them can leave part of an import; running the same import again completes it, as for `pull`, and the message says so. + - §4's list also names scheduled releases (with their pinned versions), pending approvals and agents' open edit suggestions (RFC-0014 proposals). Like automations they live outside the graph, so an archive never held them; `export` now prints them, and the guide says to schedule releases again after a move or restore. + - §4's "the manifest says so" is the manifest's `bundle` (`cloudhal.site-bundle@1`, the graph only) together with the list `export` prints and the guide documents; `@1`'s closed key set gets no field of its own. + - Invariant 4's forward migration applies once a contracts `schemaVersion` 2 exists; the demo archive can then serve as its `@1` test fixture. + - The guide is [docs/guides/site-archives.md](../guides/site-archives.md). - Author: Halil (Madraka), drafted with Claude · Date: 2026-10-21 - Related: - #47 (found while building cloudhal.com: no way to keep or move a site's content without a deployment, so it was written as code). @@ -108,6 +116,7 @@ The content graph only, as with push/pull. Excluded: - analytics; - the audit log; - automations and schedule runs; +- scheduled releases, pending approvals and open edit suggestions (*added 2026-10-26, rollout step 5*); - deployment secrets and webhook credentials. The manifest says so, and `export` prints it. People and personal data never leave in a file by accident, which is also why the format needs no encryption of its own (see Alternatives). @@ -151,6 +160,8 @@ The manifest says so, and `export` prints it. People and personal data never lea - **Export from a deployment:** `--from `, through the bundle route push and pull already use. All the headless CMS CLIs export from the remote service directly. - **Studio "Download site archive":** the same route, from the browser. - **Signed archives:** for example with Sigstore, like the npm provenance releases carry, so templates can be verified. +- **A `--dir` layout:** the same files, unpacked (Alternatives). *Added 2026-10-26.* +- **Streaming graph rows, and imports larger than one batch:** today the rows are held in memory and the node set is one atomic batch of at most 1000 statements (rollout step 5 amendment). *Added 2026-10-26.* ## Open questions (decided as recommended) diff --git a/docs/security/threat-model.md b/docs/security/threat-model.md index 5dbf1dfd..52b49cf8 100644 --- a/docs/security/threat-model.md +++ b/docs/security/threat-model.md @@ -105,6 +105,7 @@ Schedules add the first actions Cloudhal takes when nobody is at the keyboard: r - **Proof of work raises the cost of spam but does not stop a determined sender** ([Reader comments](#reader-comments-rfc-0007)). Every guest comment waits for a person anyway, and the daily cap bounds what reaches D1. - **IP bans end when the month's salt rotates** (at most about a month); email bans last until lifted. - **Posting without JavaScript is a weaker path**: no proof and no Turnstile, bounded by its own limits and switchable off per site. With Turnstile on it is off unless the owner allows it explicitly. +- **Site archives are neither signed nor encrypted** ([RFC-0039](../rfcs/0039-site-archive-export-import.md), [guide](../guides/site-archives.md#untrusted-files-security-notes)). `cloudhal import` treats the file as untrusted input: a closed layout, size caps, every file's SHA-256 and size, the precompiled validators and the push gates, all checked before anything is written. A valid archive from an untrusted source can still carry content you did not write, and every archive's history names who made each change. Encrypt the file (`age`, `gpg`) when it must stay confidential. - **A published page is a function of the snapshot and of live comments.** Rollback never touches comments, so a rolled-back article keeps its discussion. - **`cloudhal dev` and Node serve site pages without a CSP.** Only the Cloudflare site Worker sets it; a Node self-host should add one at its proxy. - **The cron trigger has no delivery guarantee** (RFC-0008). The request backstop, the heartbeat and the one-hour window make a missed tick visible and bounded; measured on Workers Paid (2026-10-12): 0 of 106 minutes missed, p99 delay 24.6 s. Not yet measured on Workers Free. diff --git a/docs/start/installation.md b/docs/start/installation.md index 69928f73..3424d7a0 100644 --- a/docs/start/installation.md +++ b/docs/start/installation.md @@ -50,7 +50,7 @@ The scaffolder prints these steps with the package manager it was run with. |---|---| | `cloudhal.json` | The project's Cloudhal settings: site id, deployment name, analytics, comments, schedules | | `src/` | The Astro theme: routes, layouts and `theme.ts`. Edit it like any Astro project | -| `.cloudhal/` | The local store (SQLite and media). Not committed; `cloudhal push` and `pull` move content between it and a deployment | +| `.cloudhal/` | The local store (SQLite and media). Not committed; `cloudhal push` and `pull` move content between it and a deployment, and `cloudhal export` copies one site into a single file ([site archives](/docs/guides/site-archives)) | | `package.json` | Scripts: `dev`, `studio`, `mcp`, `deploy`, `build`, `snapshot` | The site's content (pages, articles, menus, settings) lives in the store, not in files. Agents and Studio change it; every change is a snapshot you can diff and roll back. diff --git a/docs/start/quickstart.md b/docs/start/quickstart.md index 2c9ba59e..8cfc3e42 100644 --- a/docs/start/quickstart.md +++ b/docs/start/quickstart.md @@ -82,4 +82,4 @@ On a deployment, agents use the remote MCP endpoint at `https:///mc - [How agents work with Cloudhal](/docs/agents): modes, approvals, skills and the tool list. - [CLI reference](/docs/reference/cli). -- Guides: [search](/docs/guides/search), [schedules](/docs/guides/schedules), [comments](/docs/guides/comments), [several sites](/docs/guides/multi-site). +- Guides: [search](/docs/guides/search), [schedules](/docs/guides/schedules), [comments](/docs/guides/comments), [several sites](/docs/guides/multi-site), [site archives](/docs/guides/site-archives). diff --git a/package.json b/package.json index 0fec3b70..96d6c9c5 100644 --- a/package.json +++ b/package.json @@ -10,13 +10,14 @@ }, "type": "module", "scripts": { - "check": "biome check --diagnostic-level=error . && node scripts/size-check.ts && node scripts/font-fallbacks.ts --check && node scripts/mcp-tools-doc.ts --check && turbo run typecheck && turbo run test --concurrency=3", + "check": "biome check --diagnostic-level=error . && node scripts/size-check.ts && node scripts/font-fallbacks.ts --check && node scripts/mcp-tools-doc.ts --check && node scripts/demo-archive.ts --check && turbo run typecheck && turbo run test --concurrency=3", "lint": "biome check .", "lint:fix": "biome check --write .", "size:check": "node scripts/size-check.ts", "size:report": "node scripts/code-size.ts", "fonts:fallbacks": "node scripts/font-fallbacks.ts", "mcp:tools-doc": "node scripts/mcp-tools-doc.ts", + "demo:archive": "node scripts/demo-archive.ts", "design:lint": "node apps/studio/scripts/design-lint.ts", "design:shots": "node apps/studio/scripts/screenshots.ts", "i18n:export": "node scripts/i18n-exchange.ts export", diff --git a/packages/cli/README.md b/packages/cli/README.md index 2427a47d..d439005f 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -9,6 +9,7 @@ cloudhal studio API + Studio UI (+ /mcp) for this p cloudhal mcp [--mode plan|manual|edits|auto] MCP server over stdio for this project cloudhal deploy site + admin Workers on Cloudflare cloudhal push / cloudhal pull local site ↔ deployment +cloudhal export / cloudhal import one site ↔ a .cloudhal file ``` **Studio build.** The published CLI ships Studio prebuilt. Run from the monorepo, `cloudhal deploy` uploads `apps/studio/dist` and first checks that it is current: when it is missing, or its `index.html` is older than the newest file in `apps/studio/src`, `apps/studio/index.html` or `packages/ui/src`, it runs `pnpm --filter @cloudhal/studio build`. With `--no-build-studio` it never builds and fails on a missing or stale build instead. @@ -153,6 +154,18 @@ cloudhal fields revert events Each run snapshots the site first (`before rename_field events.speaker`; its id is the operation id), writes the new definition with a `migration` marker so every reader stays consistent, rewrites entries in guarded batches of 200, drops the marker and snapshots again. Translations that were up to date stay up to date. An interrupted run resumes with `cloudhal fields continue `; one operation per collection runs at a time. `--remote` runs it on the deployment with the deployment token. Only the draft graph changes: nothing goes live until you publish. +### Site archives (RFC-0039) + +Backups, moving a site without a deployment, content kept in git, templates, the file's layout and its security checks: [site archives](../../docs/guides/site-archives.md). + +```bash +cloudhal export [--out ] [--site ] [--no-media] [--latest-only] # one site as a .cloudhal file +cloudhal import [--replace] [--dry-run] # verified in full before anything is written +cloudhal import --revert [--dry-run] # undo a --replace +``` + +An archive holds the site's graph, snapshots, media and takedown ledger, never user accounts, comments, analytics or secrets; its history names who made each change. Import keeps the site's id. When `--replace` changes a site the project holds, it saves that site first as a `before import ` snapshot and prints the `cloudhal import --revert ` that brings it back. For an archive of another site it switches `cloudhal.json`'s `siteId`, leaving the old site in the store. + ### Deploying to Cloudflare (ADR-0012) `cloudhal deploy` uses **your** wrangler login (`npx wrangler login`, or `CLOUDFLARE_API_TOKEN`); Cloudhal never reads those credentials. It creates the D1 database and R2 bucket if needed (with a lifecycle rule that expires the snapshot bundles under `bundles/` after 30 days, RFC-0010), generates the deployment secrets once into `.cloudhal/deploy/secrets.json` (0600 — keep it out of git), builds and deploys the **site Worker** (Astro on demand, edge-cached) and the **admin Worker** (Studio, API, auth, MCP), and on the first deploy pushes your local site. diff --git a/packages/cli/src/commands/export.ts b/packages/cli/src/commands/export.ts index f37012fa..73f7565d 100644 --- a/packages/cli/src/commands/export.ts +++ b/packages/cli/src/commands/export.ts @@ -23,11 +23,11 @@ * field operation is unfinished: it needs its `before` snapshot. * - **`--no-media`:** the graph only; the manifest says `mediaOmitted`. * - **Never in an archive (§4):** user accounts, sessions, roles, agent keys, comments, readers, - * analytics, the audit log, automations, schedule runs, deployment secrets and webhook - * credentials. None of their tables or files are read. The graph does name who made each change, - * as its contracts require: snapshots, uploads, block edits, reviews, term proposals and - * takedowns carry user and agent ids (most with display names), entries their authors and the - * project its owner's id. + * analytics, the audit log, scheduled releases, automations, schedule runs, pending approvals, + * open edit suggestions, deployment secrets and webhook credentials. None of their tables or + * files are read. The graph does name who made each change, as its contracts require: + * snapshots, uploads, block edits, reviews, term proposals and takedowns carry user and agent + * ids (most with display names), entries their authors and the project its owner's id. * * A media file whose local bytes are missing, or differ from its asset's size and SHA-256, stops * the export (`cloudhal media verify` finds them). Known limitation: the graph rows are held in @@ -88,8 +88,11 @@ export const NEVER_IN_ARCHIVE: readonly string[] = [ "readers", "analytics", "the audit log", + "scheduled releases", "automations", "schedule runs", + "pending approvals", + "open edit suggestions", "deployment secrets", "webhook credentials", ]; diff --git a/packages/cli/src/seed-site.ts b/packages/cli/src/seed-site.ts index fe9db394..77649212 100644 --- a/packages/cli/src/seed-site.ts +++ b/packages/cli/src/seed-site.ts @@ -1,33 +1,56 @@ /** The starter site a new project starts with (`cloudhal init`, and a deploy from Workers Builds). */ import path from "node:path"; import type { Site } from "@cloudhal/core-contracts"; -import { seedStarterSite } from "@cloudhal/core-ts"; +import { type SiteGraphStore, seedStarterSite } from "@cloudhal/core-ts"; +import type { PropsSchema } from "@cloudhal/core-ts/blocks"; import { openLocalStore } from "@cloudhal/core-ts/local"; import { loadThemeSettings } from "./blocks.ts"; +/** + * Seeds the starter site into `store`, then applies the theme's motion default. The theme's + * settings are read after seeding, as `cloudhal init` always has. Also what the starter demo + * archive is made from (`scripts/demo-archive.ts`), so the demo is what `init` seeds. + */ +export async function seedStarter( + store: SiteGraphStore, + input: { + name: string; + slug: string; + siteId?: string | undefined; + themeSettings: () => Promise; + }, +): Promise { + const seeded = await seedStarterSite(store, { + name: input.name, + slug: input.slug, + ...(input.siteId ? { siteId: input.siteId } : {}), + }); + // RFC-0024 decision 4: new sites start with standard motion when the theme offers it; readers + // who ask for reduced motion still get less, and existing sites keep theirs. + const settings = await input.themeSettings(); + return settings && "motion" in settings.properties + ? await store.updateSite({ + ...seeded, + theme: { + ...seeded.theme, + config: { ...(seeded.theme.config ?? {}), motion: "standard" }, + }, + }) + : seeded; +} + export async function seedProjectSite( projectDir: string, input: { dataDir: string; name: string; slug: string; siteId?: string }, ): Promise { const local = await openLocalStore(path.join(projectDir, input.dataDir)); try { - const seeded = await seedStarterSite(local.store, { + return await seedStarter(local.store, { name: input.name, slug: input.slug, - ...(input.siteId ? { siteId: input.siteId } : {}), + siteId: input.siteId, + themeSettings: () => loadThemeSettings(projectDir), }); - // RFC-0024 decision 4: new sites start with standard motion when the theme offers it; readers - // who ask for reduced motion still get less, and existing sites keep theirs. - const settings = await loadThemeSettings(projectDir); - return settings && "motion" in settings.properties - ? await local.store.updateSite({ - ...seeded, - theme: { - ...seeded.theme, - config: { ...(seeded.theme.config ?? {}), motion: "standard" }, - }, - }) - : seeded; } finally { await local.close(); } diff --git a/packages/cli/test/demo-archive.test.ts b/packages/cli/test/demo-archive.test.ts new file mode 100644 index 00000000..16cbe1e1 --- /dev/null +++ b/packages/cli/test/demo-archive.test.ts @@ -0,0 +1,77 @@ +/** + * The committed starter demo archive (RFC-0039 rollout step 5): it reads and verifies in full, and + * imports into a fresh project as the starter site with its published snapshot, once. It is an + * archive an older CLI may have written (its `createdBy`). Regenerating it, and failing when it is + * stale, is `scripts/demo-archive.ts`'s job (`pnpm check`). + */ +import { createReadStream } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { afterAll, describe, expect, it } from "vitest"; +import { readArchive, type SiteArchiveManifest } from "../src/archive/index.ts"; +import { importArchive } from "../src/commands/import.ts"; +import { writeProjectConfig } from "../src/config.ts"; +import { removeTempDirs, tempDir, withStore } from "./site-fixture.ts"; + +const DEMO = path.join( + path.dirname(fileURLToPath(import.meta.url)), + "fixtures/site-archives/starter-blog.cloudhal", +); +const SITE = "site_starter0001"; + +afterAll(removeTempDirs); + +/** A project naming the demo's site, without a local store. */ +async function project(): Promise { + const dir = await tempDir("cloudhal-demo-"); + await writeProjectConfig(dir, { siteId: SITE, dataDir: ".cloudhal" }); + return dir; +} + +describe("the starter demo archive", () => { + it("reads and verifies in full", async () => { + let manifest: SiteArchiveManifest | undefined; + const paths: string[] = []; + for await (const item of readArchive(createReadStream(DEMO))) { + if (item.type === "manifest") manifest = item.manifest; + else paths.push(item.path); + } + expect(manifest).toMatchObject({ + format: "cloudhal.site-archive@1", + bundle: "cloudhal.site-bundle@1", + siteId: SITE, + siteName: "Starter Blog", + createdAt: "2026-10-26T00:00:00.000Z", + counts: { nodes: 9, snapshots: 1, blobs: 9, media: 0, mediaBytes: 0 }, + }); + expect(paths).toEqual([ + "project.json", + "nodes/0001.ndjson", + "snapshots/0001.ndjson", + "blobs/0001.ndjson", + ]); + }); + + it("imports into a fresh project as the published starter site, once", async () => { + const dir = await project(); + const dry = await importArchive({ projectDir: dir, file: DEMO, dryRun: true }); + expect(dry.refusals).toEqual([]); + expect(dry.plan?.nodes.added).toHaveLength(9); + expect(dry.plan?.snapshots).toEqual(["snap_starter0001"]); + expect(dry.newStore).toBe(true); + + const first = await importArchive({ projectDir: dir, file: DEMO }); + expect(first.imported).toMatchObject({ nodes: { inserted: 9 }, snapshots: 1, blobs: 9 }); + const site = await withStore(dir, (local) => local.store.getSite(SITE)); + expect(site?.publishedSnapshotId).toBe("snap_starter0001"); + expect(site?.theme.config?.motion).toBe("standard"); + + const again = await importArchive({ projectDir: dir, file: DEMO }); + expect(again.plan?.unchanged).toBe(true); + expect(again.imported).toMatchObject({ + nodes: { inserted: 0, deleted: 0, unchanged: true }, + snapshots: 0, + blobs: 0, + }); + }); +}); diff --git a/packages/cli/test/fixtures/site-archives/starter-blog.cloudhal b/packages/cli/test/fixtures/site-archives/starter-blog.cloudhal new file mode 100644 index 00000000..2a6fbbdf Binary files /dev/null and b/packages/cli/test/fixtures/site-archives/starter-blog.cloudhal differ diff --git a/scripts/demo-archive.ts b/scripts/demo-archive.ts new file mode 100644 index 00000000..d33911ba --- /dev/null +++ b/scripts/demo-archive.ts @@ -0,0 +1,138 @@ +/** + * The starter demo archive (RFC-0039 rollout step 5): the site `cloudhal init` seeds, published + * once as `cloudhal snapshot --publish` does, exported by the real `cloudhal export` code into + * packages/cli/test/fixtures/site-archives/starter-blog.cloudhal. A binary fixture, which the repo + * otherwise avoids, because invariant 7 (as amended) asks for a committed archive that is checked. + * + * node scripts/demo-archive.ts # regenerate; writes the file only when it is stale + * node scripts/demo-archive.ts --check # fail if it is stale (pnpm check, CI) + * + * Fixed ids (`_starter0001`, …) and a fixed clock make the content the same on every run. + * Node's mock Date is on while the site is seeded and published, because the starter dates its + * first post with `new Date()`. The check never compares gzip bytes, which differ between zlib + * builds: it reads both files in full (every size and SHA-256 verified) and compares their + * manifests without `createdBy`. The manifest pins every other file's SHA-256, so equal manifests + * mean the same content in the tar. + */ +import { createReadStream } from "node:fs"; +import { copyFile, mkdir, mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { mock } from "node:test"; +import { fileURLToPath } from "node:url"; +import { isDeepStrictEqual } from "node:util"; +import { readArchive, type SiteArchiveManifest } from "../packages/cli/src/archive/index.ts"; +import { loadThemeSettings } from "../packages/cli/src/blocks.ts"; +import { exportArchive } from "../packages/cli/src/commands/export.ts"; +import { CLI_ACTOR } from "../packages/cli/src/commands/import-apply.ts"; +import { writeProjectConfig } from "../packages/cli/src/config.ts"; +import { seedStarter } from "../packages/cli/src/seed-site.ts"; +import { GATES_ENFORCED, SiteGraphStore } from "../packages/core-ts/src/index.ts"; +import { openLocalStore } from "../packages/core-ts/src/local.ts"; + +const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const FIXTURE = path.join(ROOT, "packages/cli/test/fixtures/site-archives/starter-blog.cloudhal"); +const AT = "2026-10-26T00:00:00.000Z"; + +/** `site_starter0001`, `snap_starter0001`, …: valid ids, the same on every run. */ +function counterIds(): (prefix: string) => string { + const counters = new Map(); + return (prefix) => { + const n = (counters.get(prefix) ?? 0) + 1; + counters.set(prefix, n); + return `${prefix}_starter${String(n).padStart(4, "0")}`; + }; +} + +/** Seeds and publishes the starter site into a project at `dir`, as `cloudhal init` does. */ +async function seedProject(dir: string): Promise { + const local = await openLocalStore(path.join(dir, ".cloudhal")); + const store = new SiteGraphStore({ + sql: local.sql, + blobs: local.blobs, + gates: GATES_ENFORCED, + now: () => new Date(AT), + idGen: counterIds(), + }); + mock.timers.enable({ apis: ["Date"], now: Date.parse(AT) }); + try { + const site = await seedStarter(store, { + name: "Starter Blog", + slug: "starter-blog", + themeSettings: () => loadThemeSettings(path.join(ROOT, "starters/blog")), + }); + const snap = await store.createSnapshot(site.id, { actor: CLI_ACTOR, reason: "publish" }); + const seeded = await store.getSite(site.id); + if (!seeded) throw new Error(`site ${site.id} not found after seeding`); + await store.publish(seeded, snap.id); + await writeProjectConfig(dir, { siteId: site.id, dataDir: ".cloudhal" }); + } finally { + mock.timers.reset(); + await local.close(); + } +} + +/** The archive's manifest, after reading every file (which verifies each size and SHA-256). */ +async function verifiedManifest(file: string): Promise { + let manifest: SiteArchiveManifest | undefined; + for await (const item of readArchive(createReadStream(file))) { + if (item.type === "manifest") manifest = item.manifest; + if (item.type === "media") for await (const _ of item.body); + } + if (!manifest) throw new Error(`${file}: no manifest`); + return manifest; +} + +/** What differs between two manifests, `createdBy` aside: top-level fields and file paths. */ +function differences(a: SiteArchiveManifest, b: SiteArchiveManifest): string[] { + const { createdBy: _a, files: filesA, ...restA } = a; + const { createdBy: _b, files: filesB, ...restB } = b; + const out = Object.keys({ ...restA, ...restB }).filter( + (k) => !isDeepStrictEqual(restA[k as keyof typeof restA], restB[k as keyof typeof restB]), + ); + const listed = (files: SiteArchiveManifest["files"]) => + new Map(files.map((f) => [f.path, `${f.sha256} ${f.bytes}`])); + const [x, y] = [listed(filesA), listed(filesB)]; + for (const p of new Set([...x.keys(), ...y.keys()])) if (x.get(p) !== y.get(p)) out.push(p); + return out; +} + +/** The committed archive's manifest; undefined when it is missing or does not read in full. */ +async function committedManifest(): Promise { + try { + return await verifiedManifest(FIXTURE); + } catch (e) { + const missing = (e as NodeJS.ErrnoException).code === "ENOENT"; + console.error(`${path.relative(ROOT, FIXTURE)}: ${missing ? "missing" : (e as Error).message}`); + return undefined; + } +} + +async function run(check: boolean): Promise { + const tmp = await mkdtemp(path.join(tmpdir(), "cloudhal-demo-archive-")); + try { + await seedProject(tmp); + const { file } = await exportArchive({ projectDir: tmp, out: "starter-blog.cloudhal" }); + const fresh = await verifiedManifest(file); + const committed = await committedManifest(); + const diff = committed ? differences(committed, fresh) : ["the file"]; + const shown = path.relative(ROOT, FIXTURE); + if (diff.length === 0) { + console.log(`${shown} is up to date`); + } else if (check) { + console.error(`differs: ${diff.join(", ")}`); + console.error(`starter demo archive is stale: run pnpm demo:archive and commit ${shown}`); + process.exitCode = 1; + } else { + await mkdir(path.dirname(FIXTURE), { recursive: true }); + await copyFile(file, FIXTURE); + console.log(`wrote ${shown} (${diff.join(", ")})`); + } + } finally { + await rm(tmp, { recursive: true, force: true }); + } +} + +// No main-module guard: nothing imports this script, and comparing `import.meta.url` with argv[1] +// fails on paths with spaces, non-ASCII letters or symlinks, which would make --check pass silently. +await run(process.argv.includes("--check")); diff --git a/starters/blog/README.md b/starters/blog/README.md index 25f167fc..1d0f1d5a 100644 --- a/starters/blog/README.md +++ b/starters/blog/README.md @@ -13,6 +13,8 @@ The same scripts run with `pnpm`, `yarn` or `bun run`; `npx cloudhal ` ## What to commit Everything but `.gitignore`'s list. `cloudhal.json` holds the project's settings (the site id, custom domains, analytics, comments …) and goes with the project; the content itself lives in `.cloudhal/`, which stays on your machine (or in the deployment: `cloudhal push` / `pull`). Secrets never go in `cloudhal.json`: `cloudhal deploy` keeps them in `.cloudhal/deploy/`. +To keep the content with the project as well, export it to a file and commit that file: `cloudhal export --out content/site.cloudhal`. A fresh clone then runs `cloudhal import content/site.cloudhal` before working; a clone that imported an earlier version picks up a newer one with `cloudhal import content/site.cloudhal --replace` ([site archives](https://cloudhal.dev/docs/guides/site-archives)). + `cloudhal.json` → `domains.site` (set with `cloudhal domain set `) is also the address a static build (`cloudhal build`) writes into canonical links, the sitemap, feeds and `robots.txt`. ## Routes