From 7ff78668f2cfbfd5170a9592880425935da9a5eb Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Wed, 21 Oct 2026 22:10:41 +0300 Subject: [PATCH 1/2] docs(rfc): RFC-0038 collection base paths, RFC-0039 site archives RFC-0038 (#50): an optional base path per collection in the site (`Site.collections..basePath`, default `/`), one `entryPath` for the ~15 inline copies, and a prefix redirect when it changes, as taxonomy archives already do. RFC-0039 (#47): `cloudhal export` / `cloudhal import` with a `.cloudhal` file: a gzipped tar of a manifest, the existing SiteBundle JSON and the media bytes; keeps the site id; graph only, no people or personal data. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- docs/rfcs/0038-collection-base-paths.md | 110 +++++++++++++++++++ docs/rfcs/0039-site-archive-export-import.md | 103 +++++++++++++++++ 2 files changed, 213 insertions(+) create mode 100644 docs/rfcs/0038-collection-base-paths.md create mode 100644 docs/rfcs/0039-site-archive-export-import.md diff --git a/docs/rfcs/0038-collection-base-paths.md b/docs/rfcs/0038-collection-base-paths.md new file mode 100644 index 00000000..1cc38741 --- /dev/null +++ b/docs/rfcs/0038-collection-base-paths.md @@ -0,0 +1,110 @@ +# RFC-0038: Collection base paths + +- Status: Draft +- Author: Halil (Madraka), drafted with Claude · Date: 2026-10-21 +- Related: + - #50 (found while building cloudhal.com: the blog at `/blog` while the starter's collection is `posts`). + - RFC-0002 §7 (route order), RFC-0003 (menus, `collectionIndex`), RFC-0033 §4 (locale prefixes). + - Precedent: `Taxonomy.archive.basePath`, whose change emits prefix redirects through `RedirectPlan` (`packages/core-ts/src/taxonomy.ts`). + +## Summary + +A content entry's URL is `//`, so changing where a collection lives means renaming it, which rewrites every entry. This RFC adds an optional base path per collection in the site (`Site.collections..basePath`, default `/`), computes entry paths in one place, and redirects the old prefix when the base path changes, as taxonomy archives already do. The collection key stays the stable name that templates, filters and agents use. + +## Motivation + +- **Who:** anyone whose URLs are not the collection's internal name: `/blog` for `posts`, `/news` for `articles`, `/docs/guides` for `guides`, a localized `/haberler` on a Turkish site. Moving a site from WordPress or Ghost usually means keeping existing URLs. +- **Today:** cloudhal.com renamed its `posts` collection to `blog`. That worked thanks to the store's automatic 301s, but it changed the key that themes, filters (`collection:posts`), skills and agents' prompts refer to. `collectionIndex` only changes menu links, not where entries live. +- **Code:** the path is built inline in about 15 places (`routing.ts` `entryPath`, `navigation.ts`, `ContentList.astro`, `locales.ts`, `preview.ts`, `surfaces/tree.ts`, `search-rows.ts`, `site-search.ts`, `automations.ts`, `store/entries.ts`, `api` assets and menus, `cli` analytics, `mcp-server` schedules). Any change to URLs today means changing all of them, and they can drift. +- **Why now:** the first real site needed it, and the starter's README has nothing to say. + +## Proposal + +### 1. Contract (`site.schema.json`) + +```jsonc +// Site.properties +"collections": { + "type": "object", + "description": "Per-collection settings, keyed by collection (ContentEntry.collection). A collection without an entry here uses the defaults.", + "propertyNames": { "$ref": "common.schema.json#/$defs/Slug" }, + "additionalProperties": { "$ref": "#/$defs/CollectionSettings" } +} + +// $defs +"CollectionSettings": { + "type": "object", + "additionalProperties": false, + "properties": { + "basePath": { + "type": "string", + "description": "Where the collection's entries live: /. Leading slash, no trailing slash, one or more path segments. Default: /.", + "pattern": "^(/[a-z0-9]+(?:-[a-z0-9]+)*)+$" + } + } +} +``` + +Optional, no `schemaVersion` bump. Absent means `/`, so every existing site keeps its URLs. + +### 2. One function for entry paths (`@cloudhal/core-ts`) + +```ts +/** `/blog/hello-world`: the collection's base path (default `/`) and the slug. */ +export function entryPath( + site: Pick, + entry: Pick, +): string; + +/** The collection a path's prefix belongs to, and the rest, or undefined. Longest base wins. */ +export function collectionAt( + site: Pick, + path: string, +): { collection: string; slug: string } | undefined; +``` + +All the inline copies listed above call `entryPath`; the renderer's `resolvePath` uses `collectionAt`. Locale prefixes (RFC-0033 §4) still wrap the result, as they do today. + +### 3. Changing a base path + +`updateSite` compares each collection's old and new base path. When one changes, it adds a `RedirectPlan.move(old, new, { match: "prefix", reason: "slug-change" })` to the same batch, as `updateTaxonomy` does for `archive.basePath`. One redirect row covers the whole collection, not one row per entry. + +### 4. Invariants + +1. A base path never overlaps another collection's base path, a taxonomy archive's `basePath`, a page path, or a reserved path (`checkReservedPath`, `pathsOverlap`); a conflicting `updateSite` fails with `path_conflict`. +2. The default is `/`: a site without `collections` renders exactly as before. +3. The collection key is never derived from the URL; filters, templates, MCP tools and `ContentEntry.collection` keep using the key. +4. Changing a base path never breaks a published URL: the old prefix answers with a 301 to the new one. +5. Snapshots carry the setting (it is part of the site node), so a rollback restores the old base path. Its redirect stays, since redirects are not removed on rollback. + +### 5. Surfaces + +- **Studio:** Site settings gets a "Collections" section next to General, Domains and Previews: one row per collection with its base path (an `Input` with the URL preview). Saving goes through the existing site update. +- **MCP:** `get_site` returns the setting. No tool changes site settings today, so this adds `set_collection_path` (`site:write`). It moves public URLs, so it goes through approval like a publish. +- **Starter:** the `collectionIndex` hook defaults to `entryPath`'s prefix, so a menu item for a collection links to its base path without theme code. + +## Alternatives considered + +| Option | Pros | Cons | Why not | +|---|---|---|---| +| Rename the collection (today) | No new field | Changes the key that filters, skills and agents use; rewrites every entry | The key is an identity, not a URL | +| A `Collection` node kind | Room for labels, schemas, per-collection feeds | New node kind, store tables, sync, snapshots, MCP tools | Too big for one setting; `Site.collections` can move into a node later if collections grow | +| Base path in theme config (`cloudhal.json` / `theme.ts`) | No contract change | Not in snapshots or the graph; agents and Studio cannot change it; the API's paths would disagree with the renderer's | URLs are content, not theme | +| Per-entry `path` override | Full freedom | Thousands of paths to keep consistent; no single redirect on change | Solves a different problem; can come later on top | + +## Compatibility & migration + +- Optional field: old sites, old snapshots and old bundles are valid and render the same. +- Rust mirror: generated from the schema (`pnpm gen`); fixtures added under `fixtures/{valid,invalid}/site/` (overlapping pattern, trailing slash). +- Rollout: (a) contract and `entryPath`/`collectionAt` with every caller moved, with no behaviour change; (b) `updateSite` redirects and overlap checks; (c) Studio and MCP descriptions; (d) starter README. Each step is one PR. +- Rollback: removing the setting returns to `/`; the redirect from (b) keeps old links working. + +## Security & policy impact + +No new scopes: changing a base path is a site update (`site:write`). In Studio a person makes it; through MCP the new tool waits for a person's approval, because it moves public URLs. Redirects are same-site paths only (`RedirectPlan`), never external URLs. + +## Open questions + +1. Should a collection's base path be per locale (`/blog` in English, `/yazilar` in Turkish), like term slugs? **[default: not in v1; one base path for all locales, locale prefixes as today]** +2. Allow `/` as a base path (entries at the root, like pages)? **[default: no; entries and pages would compete for the same paths]** +3. Should the collection index page (`/blog` itself) be part of this RFC? **[default: no; it stays a page the site owner creates, and `collectionIndex` links to it]** diff --git a/docs/rfcs/0039-site-archive-export-import.md b/docs/rfcs/0039-site-archive-export-import.md new file mode 100644 index 00000000..0f303166 --- /dev/null +++ b/docs/rfcs/0039-site-archive-export-import.md @@ -0,0 +1,103 @@ +# RFC-0039: Site archives: `cloudhal export` and `cloudhal import` + +- Status: Draft +- Author: Halil (Madraka), drafted with Claude · Date: 2026-10-21 +- Related: + - #47 (found while building cloudhal.com: no way to keep or move a site's content without a deployment, so it was written as code). + - RFC-0004 §4.7 (media bytes travel apart from the bundle, verified by SHA-256), RFC-0010 (snapshot bundles). + - `SiteBundle` (`cloudhal.site-bundle@1`), `exportSite`, `importSite`, `copyMediaBlobs` in `packages/core-ts/src/sync.ts`: what `cloudhal push` and `pull` send over HTTP today. + +## Summary + +`cloudhal export` writes one site of a local project to a single file, its content graph, snapshots and media; `cloudhal import` reads it back into a local project, keeping the site's id. The file is a gzipped tar holding the existing `SiteBundle` JSON and the media bytes, so the format adds a container, not a second serialisation. This gives backups, starters and templates as files, moving a site between machines, and demo seeding without a deployment and without content written as code. + +## Motivation + +- **Who:** a person keeping a backup or moving a site to a new machine; a theme author shipping a demo site; a team seeding a playground or test environment; anyone migrating content out of one project into another. +- **Today:** `push` and `pull` need a deployment, and `.cloudhal/` is git-ignored. cloudhal.com worked around this with a hand-written `content/site.ts` and a seed script: content as code, which contradicts how Cloudhal sites are edited (Studio, agents) and was removed when the repo went public. +- **Identity:** `seedStarterSite` cannot take a site id, so a fresh clone always gets a new `siteId`. Anything keyed by the id (analytics, cache tags, the deployment's site row) then no longer matches. +- **Why now:** the bundle code exists and is tested by push/pull; what is missing is a file container and two commands. + +## Proposal + +### 1. The file + +`.cloudhal` (suggested; any name works): a gzip-compressed POSIX tar (ustar) with, in this order: + +``` +manifest.json the archive's own header (below), first so it can be read without unpacking the rest +bundle.json a SiteBundle, exactly as push/pull send it (cloudhal.site-bundle@1) +media/ one file per BundleMedia entry, the raw bytes +``` + +```ts +interface SiteArchiveManifest { + format: "cloudhal.site-archive@1"; + siteId: string; + siteName: string; + /** `@cloudhal/cli` version that wrote it. */ + createdBy: string; + createdAt: string; // RFC 3339 + /** What the bundle holds, so `import --dry-run` can show it without parsing bundle.json. */ + counts: { nodes: number; snapshots: number; media: number; mediaBytes: number }; + /** SHA-256 of bundle.json, so a damaged or edited file is refused. */ + bundleSha256: string; +} +``` + +### 2. Commands + +```text +cloudhal export [--out ] [--site ] [--no-media] [--latest-only] +cloudhal import [--replace] [--dry-run] +``` + +- **export:** reads the local store with `exportSite` (the project's site, or with `--site ` another site listed in `cloudhal.json` `sites`) and its media from the local blob store. Default file: `-.cloudhal`. `--no-media` writes the graph only; the manifest says so and `import` warns. `--latest-only` keeps only the published and latest snapshots, for templates that should not carry history. +- **import:** checks the manifest and `bundleSha256`, verifies every media file against its name (SHA-256) as it is written to the blob store, then calls `importSite`: + - into a project with no site, or one whose site is the archive's (same `siteId`): `mode: "bootstrap"`. It refuses diverged nodes, as `pull` does; + - with `--replace`: `mode: "replace"`; the local nodes become the archive's, and snapshots are added (they are immutable and never removed); + - into a project whose site has another id: refused unless `--replace`, which then switches `cloudhal.json`'s `siteId` to the archive's. The old site stays in the local store, unlisted; there is no command to delete a local site yet. +- `--dry-run` prints the manifest and what would change (nodes added, changed, removed; snapshots added; media to copy) and writes nothing. + +### 3. Invariants + +1. The site's id is kept: an imported site is the same site, so a later `push` to its deployment is an update, not a second site. +2. Nothing in the archive runs: it is data validated with the precompiled validators (`assertValidRows`), and gates (`assertGates`) apply as for push. +3. Every media file is verified by SHA-256 before it is used; a missing or wrong file fails the import before any node is written (the bundle's `missing_media` rule). +4. Taken-down content stays taken down: the `purged` ledger travels and the import never restores those bytes. +5. Import writes the graph in one atomic batch (no interactive transactions, AGENTS.md rule 4); a failure leaves the project as it was. +6. Same content, same file: entries are written in a fixed order with fixed timestamps in the tar headers, so exporting an unchanged site twice gives identical bytes (good for git and for diffing). + +### 4. What is not in an archive + +The content graph only, as with push/pull: no users, sessions, roles or agent keys; no comments or readers; no analytics; no audit log; no automations or schedule runs; no deployment secrets. The manifest says so, and `export` prints it. People and personal data never leave in a file by accident. + +## Alternatives considered + +| Option | Pros | Cons | Why not | +|---|---|---|---| +| One JSON file, media inlined as base64 | One format, no container | A third larger; whole file in memory; one bad byte breaks everything | Media can be hundreds of MB | +| A directory (`bundle.json` + `media/`) | No container code; diffable in git | Not one file to send or attach; easy to copy half of it | Can be a later `--dir` option; the tar is the default | +| Zip | Opens on every OS | Needs a zip library or a writer of our own; no streaming read of a gzipped stream | Node has gzip built in, and ustar is small enough to write without a dependency | +| Commit `.cloudhal/data.db` | No code | SQLite binary in git; schema-version bound; carries users and secrets | Unsafe and not portable | +| Content as code (cloudhal.com's old way) | Reviewable diffs | Not how sites are edited; drifts from the graph | The problem #47 reports | + +## Compatibility & migration + +- New commands only; push, pull and the `SiteBundle` format are unchanged. +- `cloudhal.site-archive@1`: a later `@2` reader keeps reading `@1`. `import` refuses unknown formats with a message naming the CLI version needed. +- `bundle.json` is `cloudhal.site-bundle@1`, so a future bundle format bump is read through `importSite`'s own format check. +- Rollout: (a) the tar writer and reader with their tests (round trip, damaged file, wrong hash); (b) `export`; (c) `import` with `--dry-run`; (d) `--replace` and the site id switch; (e) docs and a starter demo archive. + +## Security & policy impact + +- A local CLI command, so no new scopes. Importing is like `pull`: the file is untrusted input, so it is validated, size-capped (manifest counts against actual entries, per-entry and total byte limits), and tar paths outside `manifest.json`, `bundle.json` and `media/<64 hex>` are rejected (no path traversal, no symlinks or devices). +- Personal data: none of the stores that hold it are exported (§4). Entries may name authors, as published pages do. +- The archive is not signed in v1: it is as trustworthy as where it came from. See open question 2. + +## Open questions + +1. Should `cloudhal init --from ` start a new project from an archive (templates, the playground's demo)? **[default: yes, as a follow-up; it is `init` + `import`]** +2. Sign archives (e.g. Sigstore, like the npm provenance we publish with) so templates can be verified? **[default: not in v1]** +3. Import as a copy with new ids (`--as-new-site`) for templates used many times in one project? Rewriting ids across nodes, snapshots and redirects is a large change. **[default: not in v1; one archive, one site]** +4. Should Studio offer "Download site archive" through the admin API? **[default: later; the API already has the bundle route push/pull use]** From 203615c80f75ed1e5934b75e790a59275c125de3 Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Thu, 22 Oct 2026 09:17:13 +0300 Subject: [PATCH 2/2] docs: accept RFC-0038 (collection permalinks) and RFC-0039 (site archives) Revised before acceptance after comparing WordPress, Ghost, Hugo, Webflow, Eleventy and the headless CMSs (URLs), and Sanity, Strapi, Contentful, Ghost, WordPress, Directus and Webflow (exports): - RFC-0038: `permalink` (closed token set, v1 `//{slug}`) instead of `basePath`, so tokens can come without a rename; `entryPath`/`entryAt` both ways; redirects derived from the change; later items decided. - RFC-0039: NDJSON parts instead of one bundle.json (streaming), per-file SHA-256 and contract versions in the manifest, a `before import` snapshot for `--replace`, no built-in encryption; later items decided. - ADR-0040, ADR-0041. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- docs/adr/0040-collection-permalinks.md | 27 ++++ docs/adr/0041-site-archives.md | 35 +++++ docs/adr/README.md | 2 + docs/rfcs/0038-collection-base-paths.md | 110 --------------- docs/rfcs/0038-collection-permalinks.md | 141 +++++++++++++++++++ docs/rfcs/0039-site-archive-export-import.md | 107 ++++++++++---- 6 files changed, 282 insertions(+), 140 deletions(-) create mode 100644 docs/adr/0040-collection-permalinks.md create mode 100644 docs/adr/0041-site-archives.md delete mode 100644 docs/rfcs/0038-collection-base-paths.md create mode 100644 docs/rfcs/0038-collection-permalinks.md diff --git a/docs/adr/0040-collection-permalinks.md b/docs/adr/0040-collection-permalinks.md new file mode 100644 index 00000000..7bd65427 --- /dev/null +++ b/docs/adr/0040-collection-permalinks.md @@ -0,0 +1,27 @@ +# ADR-0040: Collection permalinks: a closed-token template per collection, one path function both ways +- Status: Accepted +- Date: 2026-10-22 (decided by Halil) +- Related: RFC-0038, RFC-0002, RFC-0003, RFC-0033, #50 +- Sources: [Ghost routes](https://docs.ghost.org/themes/routing/) · [Hugo permalinks](https://gohugo.io/configuration/permalinks/) · [WordPress `register_post_type`](https://developer.wordpress.org/reference/functions/register_post_type/) and [old-slug redirects](https://developer.wordpress.org/reference/functions/wp_old_slug_redirect/) · [Eleventy permalinks](https://www.11ty.dev/docs/permalinks/) + +## Context +An entry's URL is `//`, built inline in about 15 places. A site that wants its posts at `/blog` has to rename the collection, which changes the key filters, templates and agents use (cloudhal.com, #50). WordPress, Ghost and Hugo all ended up with URL templates. None of them reliably redirects when a template changes, and date tokens move URLs when an editable date changes. + +## Decision +1. **`Site.collections..permalink`**, optional, default `//{slug}`. Version 1 accepts only fixed lower-case segments followed by `/{slug}`, enforced by the schema pattern. +2. **A closed token set**, widened only by changing the schema pattern. There is no template language, so every template is checked when it is written. +3. **One path function each way** in `@cloudhal/core-ts`: `entryPath(site, entry)` and `entryAt(site, path)`. Every inline `/${collection}/${slug}` moves to them. +4. **Redirects come from the change.** A v1 template change adds one prefix redirect in the same batch as the site update (`RedirectPlan`, as for taxonomy archives). Once there are tokens, the store compares each published entry's path before and after and writes one redirect per entry. +5. **Collisions fail at write time** (`path_conflict`) against other collections, taxonomy archives, pages and reserved paths. +6. **Later, already decided:** date tokens from a fixed first-publication date; per-locale templates as a map; entries at the root once page writes run the same collision check; an explicit `indexPath`; `{category}` only once an entry has one primary term. + +## Consequences +- Positive: + - URLs become content: they are in snapshots, can be rolled back, and agents and Studio can change them behind approval. + - Moving a collection no longer changes its key. + - The path logic exists once. +- Negative: + - A larger site node. + - Every caller of the inline path moves in one PR. + - Templates and redirects have to stay in step on rollback (redirects are kept). +- Risks: an overlap check that misses a case would make two things answer one URL. That is why the check runs both ways (`entryAt`) and at write time. diff --git a/docs/adr/0041-site-archives.md b/docs/adr/0041-site-archives.md new file mode 100644 index 00000000..7a881381 --- /dev/null +++ b/docs/adr/0041-site-archives.md @@ -0,0 +1,35 @@ +# ADR-0041: Site archives: a streaming `.cloudhal` tar of NDJSON parts and media, ids kept, graph only +- Status: Accepted +- Date: 2026-10-22 (decided by Halil) +- Related: RFC-0039, RFC-0004, RFC-0010, ADR-0002, #47 +- Sources: [Sanity datasets](https://www.sanity.io/docs/cli-reference/dataset) · [Strapi data management](https://docs.strapi.io/cms/data-management/export) · [Contentful CLI export](https://github.com/contentful/contentful-cli/tree/main/docs/space/export) · [Ghost migration](https://docs.ghost.org/migration/ghost) · [WordPress export](https://wordpress.org/documentation/article/tools-export-screen/) + +## Context +A site's content can only leave a project through a deployment (`push`, `pull`), so cloudhal.com kept its content as code (#47). The `SiteBundle` format and `importSite` already exist and are tested. The CMSs compared stream large exports (Sanity's tar.gz of NDJSON, Strapi's JSONL parts) and check assets. Strapi erases the target on import and encrypts with AES-128-ECB. + +## Decision +1. **`cloudhal export` / `cloudhal import`** with a `.cloudhal` file: a gzip-compressed ustar holding `manifest.json`, `project.json`, `nodes|snapshots|blobs/NNNN.ndjson` parts of at most 8 MB, `media/` and `ledger.ndjson`. Read in order, the parts are exactly a `cloudhal.site-bundle@1`. +2. **The manifest** names the archive format, the contracts' `schemaVersion`, the bundle format, the counts, and every file's SHA-256 and size. A newer format or schema version is refused; an older one is migrated forward. +3. **Ids are kept.** An imported site is the same site. +4. **Import modes:** + - bootstrap (default): adds what is missing and refuses differing nodes; + - `--replace`: first saves the current graph as a `before import` snapshot, so the import can be undone; + - `--dry-run`: shows the difference and writes nothing. + + Never erase-then-import. +5. **Graph only.** No users, sessions, roles, agent keys, comments, readers, analytics, audit log, automations, secrets or webhook credentials. That is why there is no built-in encryption; people can pipe the file through `age` or `gpg`. +6. **Deterministic bytes:** a fixed order, zero timestamps and fixed modes, so an unchanged site exports identically. +7. **Untrusted input:** the layout is a closed list of paths (no traversal, links or devices); sizes are capped; rows go through the precompiled validators and gates, as for `push`. +8. **Later, already decided:** `init --from`, `--new-ids` for templates, partial export, `--from `, Studio download, and signed archives. + +## Consequences +- Positive: + - Backups, demos and starters become files. + - Moving a site needs no deployment. + - Nothing in the format needs a second data model. +- Negative: + - A tar reader and writer of our own (small, no dependency). + - The format is versioned and must be read forever. +- Risks: + - A file is as trustworthy as its source until archives are signed. + - Large media sets make big files; `--no-media` and the parts limit keep memory flat, not size small. diff --git a/docs/adr/README.md b/docs/adr/README.md index cb9e1405..f8250c47 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -56,3 +56,5 @@ One file per decision, numbered, immutable once accepted (write a new ADR to sup | [0037](0037-domain-strategy.md) | Domain strategy: `cloudhal.com` is marketing, a future hosted control plane and the playground; `cloudhal.dev` is docs, schemas, registry and the open-source home; customer sites never run on either (they deploy into the customer's own Cloudflare account), so no third domain or PSL entry | | [0038](0038-public-playground.md) | Public playground: one SQLite Durable Object per visitor holding store, API and renderer; `@cloudhal/storage/do-sqlite` and SQL/layered/fetcher blob stores; a deny-list guard in the Worker and the API with a route-walk test and a fixed Editor principal; 60-minute TTL with a 2-hour cap; private, uncached session pages (RFC-0036) | | [0039](0039-cloudhal-dev-docs-site.md) | cloudhal.dev: Starlight in `apps/docs` over the repo's `docs/`, generated reference, a link contract checked in CI (301s, permanent `/schemas/v1`, `/e/` pages from one error-code registry), docs for agents (`.md` per page, `Accept: text/markdown`, `llms.txt`, a standalone read-only docs MCP at `cloudhal.dev/mcp`, none in `cloudhal mcp` (amended)); all ADRs and RFCs published (RFC-0037) | +| [0040](0040-collection-permalinks.md) | Collection permalinks: `Site.collections..permalink` (default `//{slug}`, v1 a fixed prefix plus `{slug}`), a closed token set, `entryPath`/`entryAt` in core-ts, redirects derived from the change, collisions refused at write (RFC-0038) | +| [0041](0041-site-archives.md) | Site archives: `cloudhal export`/`import`, a deterministic `.cloudhal` tar of NDJSON parts and media with per-file SHA-256, ids kept, bootstrap/`--replace` (snapshot first)/`--dry-run`, graph only and no built-in encryption (RFC-0039) | diff --git a/docs/rfcs/0038-collection-base-paths.md b/docs/rfcs/0038-collection-base-paths.md deleted file mode 100644 index 1cc38741..00000000 --- a/docs/rfcs/0038-collection-base-paths.md +++ /dev/null @@ -1,110 +0,0 @@ -# RFC-0038: Collection base paths - -- Status: Draft -- Author: Halil (Madraka), drafted with Claude · Date: 2026-10-21 -- Related: - - #50 (found while building cloudhal.com: the blog at `/blog` while the starter's collection is `posts`). - - RFC-0002 §7 (route order), RFC-0003 (menus, `collectionIndex`), RFC-0033 §4 (locale prefixes). - - Precedent: `Taxonomy.archive.basePath`, whose change emits prefix redirects through `RedirectPlan` (`packages/core-ts/src/taxonomy.ts`). - -## Summary - -A content entry's URL is `//`, so changing where a collection lives means renaming it, which rewrites every entry. This RFC adds an optional base path per collection in the site (`Site.collections..basePath`, default `/`), computes entry paths in one place, and redirects the old prefix when the base path changes, as taxonomy archives already do. The collection key stays the stable name that templates, filters and agents use. - -## Motivation - -- **Who:** anyone whose URLs are not the collection's internal name: `/blog` for `posts`, `/news` for `articles`, `/docs/guides` for `guides`, a localized `/haberler` on a Turkish site. Moving a site from WordPress or Ghost usually means keeping existing URLs. -- **Today:** cloudhal.com renamed its `posts` collection to `blog`. That worked thanks to the store's automatic 301s, but it changed the key that themes, filters (`collection:posts`), skills and agents' prompts refer to. `collectionIndex` only changes menu links, not where entries live. -- **Code:** the path is built inline in about 15 places (`routing.ts` `entryPath`, `navigation.ts`, `ContentList.astro`, `locales.ts`, `preview.ts`, `surfaces/tree.ts`, `search-rows.ts`, `site-search.ts`, `automations.ts`, `store/entries.ts`, `api` assets and menus, `cli` analytics, `mcp-server` schedules). Any change to URLs today means changing all of them, and they can drift. -- **Why now:** the first real site needed it, and the starter's README has nothing to say. - -## Proposal - -### 1. Contract (`site.schema.json`) - -```jsonc -// Site.properties -"collections": { - "type": "object", - "description": "Per-collection settings, keyed by collection (ContentEntry.collection). A collection without an entry here uses the defaults.", - "propertyNames": { "$ref": "common.schema.json#/$defs/Slug" }, - "additionalProperties": { "$ref": "#/$defs/CollectionSettings" } -} - -// $defs -"CollectionSettings": { - "type": "object", - "additionalProperties": false, - "properties": { - "basePath": { - "type": "string", - "description": "Where the collection's entries live: /. Leading slash, no trailing slash, one or more path segments. Default: /.", - "pattern": "^(/[a-z0-9]+(?:-[a-z0-9]+)*)+$" - } - } -} -``` - -Optional, no `schemaVersion` bump. Absent means `/`, so every existing site keeps its URLs. - -### 2. One function for entry paths (`@cloudhal/core-ts`) - -```ts -/** `/blog/hello-world`: the collection's base path (default `/`) and the slug. */ -export function entryPath( - site: Pick, - entry: Pick, -): string; - -/** The collection a path's prefix belongs to, and the rest, or undefined. Longest base wins. */ -export function collectionAt( - site: Pick, - path: string, -): { collection: string; slug: string } | undefined; -``` - -All the inline copies listed above call `entryPath`; the renderer's `resolvePath` uses `collectionAt`. Locale prefixes (RFC-0033 §4) still wrap the result, as they do today. - -### 3. Changing a base path - -`updateSite` compares each collection's old and new base path. When one changes, it adds a `RedirectPlan.move(old, new, { match: "prefix", reason: "slug-change" })` to the same batch, as `updateTaxonomy` does for `archive.basePath`. One redirect row covers the whole collection, not one row per entry. - -### 4. Invariants - -1. A base path never overlaps another collection's base path, a taxonomy archive's `basePath`, a page path, or a reserved path (`checkReservedPath`, `pathsOverlap`); a conflicting `updateSite` fails with `path_conflict`. -2. The default is `/`: a site without `collections` renders exactly as before. -3. The collection key is never derived from the URL; filters, templates, MCP tools and `ContentEntry.collection` keep using the key. -4. Changing a base path never breaks a published URL: the old prefix answers with a 301 to the new one. -5. Snapshots carry the setting (it is part of the site node), so a rollback restores the old base path. Its redirect stays, since redirects are not removed on rollback. - -### 5. Surfaces - -- **Studio:** Site settings gets a "Collections" section next to General, Domains and Previews: one row per collection with its base path (an `Input` with the URL preview). Saving goes through the existing site update. -- **MCP:** `get_site` returns the setting. No tool changes site settings today, so this adds `set_collection_path` (`site:write`). It moves public URLs, so it goes through approval like a publish. -- **Starter:** the `collectionIndex` hook defaults to `entryPath`'s prefix, so a menu item for a collection links to its base path without theme code. - -## Alternatives considered - -| Option | Pros | Cons | Why not | -|---|---|---|---| -| Rename the collection (today) | No new field | Changes the key that filters, skills and agents use; rewrites every entry | The key is an identity, not a URL | -| A `Collection` node kind | Room for labels, schemas, per-collection feeds | New node kind, store tables, sync, snapshots, MCP tools | Too big for one setting; `Site.collections` can move into a node later if collections grow | -| Base path in theme config (`cloudhal.json` / `theme.ts`) | No contract change | Not in snapshots or the graph; agents and Studio cannot change it; the API's paths would disagree with the renderer's | URLs are content, not theme | -| Per-entry `path` override | Full freedom | Thousands of paths to keep consistent; no single redirect on change | Solves a different problem; can come later on top | - -## Compatibility & migration - -- Optional field: old sites, old snapshots and old bundles are valid and render the same. -- Rust mirror: generated from the schema (`pnpm gen`); fixtures added under `fixtures/{valid,invalid}/site/` (overlapping pattern, trailing slash). -- Rollout: (a) contract and `entryPath`/`collectionAt` with every caller moved, with no behaviour change; (b) `updateSite` redirects and overlap checks; (c) Studio and MCP descriptions; (d) starter README. Each step is one PR. -- Rollback: removing the setting returns to `/`; the redirect from (b) keeps old links working. - -## Security & policy impact - -No new scopes: changing a base path is a site update (`site:write`). In Studio a person makes it; through MCP the new tool waits for a person's approval, because it moves public URLs. Redirects are same-site paths only (`RedirectPlan`), never external URLs. - -## Open questions - -1. Should a collection's base path be per locale (`/blog` in English, `/yazilar` in Turkish), like term slugs? **[default: not in v1; one base path for all locales, locale prefixes as today]** -2. Allow `/` as a base path (entries at the root, like pages)? **[default: no; entries and pages would compete for the same paths]** -3. Should the collection index page (`/blog` itself) be part of this RFC? **[default: no; it stays a page the site owner creates, and `collectionIndex` links to it]** diff --git a/docs/rfcs/0038-collection-permalinks.md b/docs/rfcs/0038-collection-permalinks.md new file mode 100644 index 00000000..780603a3 --- /dev/null +++ b/docs/rfcs/0038-collection-permalinks.md @@ -0,0 +1,141 @@ +# RFC-0038: Collection permalinks + +- Status: Accepted (2026-10-22, by Halil; ADR-0040). Open questions decided as recommended. Revised before acceptance after a comparison with WordPress, Ghost, Hugo, Webflow, Eleventy and the headless CMSs (see Prior art). +- Author: Halil (Madraka), drafted with Claude · Date: 2026-10-21 +- Related: + - #50 (found while building cloudhal.com: the blog at `/blog` while the starter's collection is `posts`). + - RFC-0002 §7 (route order), RFC-0003 (menus, `collectionIndex`), RFC-0033 §4 (locale prefixes). + - Precedent: `Taxonomy.archive.basePath`, whose change emits prefix redirects through `RedirectPlan` (`packages/core-ts/src/taxonomy.ts`). + - Prior art: + - [Ghost routes](https://docs.ghost.org/themes/routing/) (`permalink: /blog/{slug}/`); + - [Hugo permalinks](https://gohugo.io/configuration/permalinks/) (per section and per language); + - [WordPress](https://developer.wordpress.org/reference/functions/register_post_type/) (`rewrite.slug`, [old-slug redirects](https://developer.wordpress.org/reference/functions/wp_old_slug_redirect/)); + - [Eleventy](https://www.11ty.dev/docs/permalinks/) (fails on duplicate URLs); + - [Sanity's resolver](https://www.sanity.io/docs/visual-editing/presentation-resolver-api) (path → document). + +## Summary + +A content entry's URL is `//`, so moving where a collection lives means renaming it, which changes the key every template, filter and agent uses. This RFC adds an optional permalink template per collection (`Site.collections..permalink`, default `//{slug}`), computes entry paths in one place both ways (entry → path, path → entry), and redirects old URLs when the template changes. Version 1 accepts only a fixed prefix followed by `{slug}`. The field and its closed token set are designed so date tokens and per-locale templates can follow without renaming anything. + +## Motivation + +- **Who:** anyone whose URLs are not the collection's internal name: `/blog` for `posts`, `/news` for `articles`, `/docs/guides` for `guides`. Moving a site from WordPress or Ghost usually means keeping its URLs. +- **Today:** cloudhal.com renamed its `posts` collection to `blog`. That worked thanks to the store's automatic 301s, but it changed the key that themes, filters (`collection:posts`), skills and agents' prompts refer to. `collectionIndex` only changes menu links, not where entries live. +- **Code:** the path is built inline in about 15 places (`routing.ts` `entryPath`, `navigation.ts`, `ContentList.astro`, `locales.ts`, `preview.ts`, `surfaces/tree.ts`, `search-rows.ts`, `site-search.ts`, `automations.ts`, `store/entries.ts`, `api` assets and menus, `cli` analytics, `mcp-server` schedules). Any change to URLs means changing all of them, and they can drift. +- **What others learned:** + - Every server-rendering platform ends up offering a template with tokens, not just a prefix. WordPress, Ghost and Hugo all do. + - None of them reliably redirects when the template itself changes. Per-item old-URL tracking (WordPress old slugs, Hugo aliases) works better than hand-written rules (Ghost `redirects.yaml`, Webflow). + - Date tokens change URLs when an editable date changes. + + Naming the field after a prefix (`basePath`) would force a rename the day tokens arrive. + +## Proposal + +### 1. Contract (`site.schema.json`) + +```jsonc +// Site.properties +"collections": { + "type": "object", + "description": "Per-collection settings, keyed by collection (ContentEntry.collection). A collection without an entry here uses the defaults.", + "propertyNames": { "$ref": "common.schema.json#/$defs/Slug" }, + "additionalProperties": { "$ref": "#/$defs/CollectionSettings" } +} + +// $defs +"CollectionSettings": { + "type": "object", + "additionalProperties": false, + "properties": { + "permalink": { + "type": "string", + "description": "Where the collection's entries live, as a template. Version 1: one or more fixed lower-case path segments followed by /{slug}, e.g. /blog/{slug}. Default: //{slug}.", + "pattern": "^(/[a-z0-9]+(?:-[a-z0-9]+)*)+/\\{slug\\}$" + } + } +} +``` + +Optional, no `schemaVersion` bump. Absent means `//{slug}`, so every existing site keeps its URLs. + +**The token set is closed.** Version 1 has only `{slug}`. Later versions widen the schema pattern; there is no expression language, so an agent cannot write a template the store cannot check. + +### 2. One function each way (`@cloudhal/core-ts`) + +```ts +/** `/blog/hello-world`: the collection's permalink with the entry's values filled in. */ +export function entryPath( + site: Pick, + entry: Pick, +): string; + +/** The collection and slug a path matches, or undefined. The longest fixed prefix wins. */ +export function entryAt( + site: Pick, + path: string, +): { collection: string; slug: string } | undefined; +``` + +- All the inline copies listed above call `entryPath`. The renderer's `resolvePath`, the overlap checks and the redirect planner use `entryAt`. +- Locale prefixes (RFC-0033 §4) still wrap the result, as they do today. + +### 3. Changing a permalink + +`updateSite` compares each collection's old and new template. When one changes, it adds a redirect to the same batch, as `updateTaxonomy` does for `archive.basePath`. While templates are a fixed prefix plus `{slug}`, one `RedirectPlan.move(oldPrefix, newPrefix, { match: "prefix", reason: "slug-change" })` covers every entry in the collection. + +When tokens other than `{slug}` arrive, a prefix no longer describes the change. The store then compares `entryPath` before and after for each published entry of the collection and writes one redirect per entry, the model of WordPress old slugs and Hugo aliases. The MCP tool's preview shows that list before anything is written. + +### 4. Invariants + +1. A template never overlaps another collection's template, a taxonomy archive's `basePath`, a page path, or a reserved path (`checkReservedPath`, `pathsOverlap`, and `entryAt` for the reverse check). A conflicting `updateSite` fails with `path_conflict`, the way Eleventy refuses duplicate URLs. +2. The default is `//{slug}`: a site without `collections` renders exactly as before. +3. The collection key is never derived from the URL. Filters, templates, MCP tools and `ContentEntry.collection` keep using the key. +4. Changing a template never breaks a published URL: the old URL answers with a 301 to the new one. +5. Snapshots carry the setting (it is part of the site node), so a rollback restores the old template. Its redirect stays, since redirects are not removed on rollback. +6. The token set is closed and every template is checked when it is written, never when a page is rendered. + +### 5. Surfaces + +- **Studio:** Site settings gets a "Collections" section next to General, Domains and Previews: one row per collection with its URL (an `Input` showing the fixed part before `/{slug}`, with a preview of a real entry's address). Saving goes through the existing site update. +- **MCP:** `get_site` returns the setting. No tool changes site settings today, so this adds `set_collection_permalink` (`site:write`). It moves public URLs, so it goes through approval like a publish, and its preview lists the old and new address of a few entries. +- **Starter:** the `collectionIndex` hook defaults to the template's fixed prefix, so a menu item for a collection links there without theme code. + +## Later (decided, not in v1) + +- **Date tokens** (`{year}`, `{month}`, `{day}`): taken from a stored first-publication date that never changes, not from an editable date, to avoid the date-change trap. Redirects per entry (§3). +- **Per-locale templates:** a map, `permalink: { en: "/blog/{slug}", tr: "/yazilar/{slug}" }`, as Hugo allows per language. +- **Entries at the root** (`/{slug}`): allowed only once the collision check of invariant 1 also runs when a page is created or moved. +- **The collection index page:** an explicit `indexPath` setting (Hugo's section page, WordPress `has_archive`), not implied by the template. +- **`{category}`:** only once an entry can name one primary term (Ghost's `primary_tag`). Without it, a post in two categories has two candidate URLs. + +## Alternatives considered + +| Option | Pros | Cons | Why not | +|---|---|---|---| +| Rename the collection (today) | No new field | Changes the key that filters, skills and agents use; rewrites every entry | The key is an identity, not a URL | +| A fixed `basePath` field (the first draft) | Simplest | Must be renamed or doubled when tokens arrive | `permalink` with v1 limited to `//{slug}` costs the same now | +| A free-form template language | Full freedom | Unchecked routes, collisions found only at render time, agents can write anything | A closed token set gives the useful templates and stays checkable | +| A `Collection` node kind | Room for labels, schemas, per-collection feeds | New node kind, store tables, sync, snapshots, MCP tools | Too big for one setting; `Site.collections` can move into a node later if collections grow | +| Templates in theme config (`cloudhal.json` / `theme.ts`) | No contract change | Not in snapshots or the graph; agents and Studio cannot change them; the API's paths would disagree with the renderer's | URLs are content, not theme | +| Hand-written redirect files (Ghost, Webflow) | Explicit | Forgotten when URLs move; drift | Redirects are derived from the change itself | + +## Compatibility & migration + +- Optional field: old sites, old snapshots and old bundles are valid and render the same. +- Rust mirror: generated from the schema (`pnpm gen`). Fixtures go under `fixtures/{valid,invalid}/site/`: a valid `/blog/{slug}`, and invalid templates with a trailing slash, an unknown token, or no `{slug}`. +- Rollout, one PR each: + 1. The contract plus `entryPath`/`entryAt`, with every caller moved and no behaviour change. + 2. `updateSite` redirects and overlap checks. + 3. Studio, the MCP tool and descriptions. + 4. The starter README. +- Rollback: removing the setting returns to `//{slug}`; the redirect from step 2 keeps old links working. + +## Security & policy impact + +No new scopes: changing a template is a site update (`site:write`). In Studio a person makes it; through MCP the new tool waits for a person's approval, because it moves public URLs. Redirects are same-site paths only (`RedirectPlan`), never external URLs. + +## Open questions (decided as recommended) + +1. **Per-locale templates:** not in v1; one template for all locales, with locale prefixes as today. They come later as a map (see Later). +2. **`/` as a base:** not in v1; allowed later once page writes also run the collision check. +3. **The collection index page:** not part of this RFC; it stays a page the site owner creates, and `collectionIndex` links to it. An explicit `indexPath` comes later. diff --git a/docs/rfcs/0039-site-archive-export-import.md b/docs/rfcs/0039-site-archive-export-import.md index 0f303166..c14a2a87 100644 --- a/docs/rfcs/0039-site-archive-export-import.md +++ b/docs/rfcs/0039-site-archive-export-import.md @@ -1,15 +1,21 @@ # RFC-0039: Site archives: `cloudhal export` and `cloudhal import` -- Status: Draft +- Status: Accepted (2026-10-22, by Halil; ADR-0041). Open questions decided as recommended. Revised before acceptance after a comparison with Sanity, Strapi, Contentful, Ghost, WordPress, Directus and Webflow (see Prior art): NDJSON parts instead of one JSON file, per-file checksums, contract versions in the manifest, and a snapshot before `--replace`. - Author: Halil (Madraka), drafted with Claude · Date: 2026-10-21 - Related: - #47 (found while building cloudhal.com: no way to keep or move a site's content without a deployment, so it was written as code). - RFC-0004 §4.7 (media bytes travel apart from the bundle, verified by SHA-256), RFC-0010 (snapshot bundles). - `SiteBundle` (`cloudhal.site-bundle@1`), `exportSite`, `importSite`, `copyMediaBlobs` in `packages/core-ts/src/sync.ts`: what `cloudhal push` and `pull` send over HTTP today. + - Prior art: + - [Sanity dataset export/import](https://www.sanity.io/docs/cli-reference/dataset): tar.gz with NDJSON and assets, asset checks, `--replace`/`--missing`. + - [Strapi data management](https://docs.strapi.io/cms/data-management/export): JSONL parts, schema match required, AES-128-ECB encryption, import erases the target. + - [Contentful CLI export](https://github.com/contentful/contentful-cli/tree/main/docs/space/export): one JSON file, credentials dropped. + - [Ghost](https://docs.ghost.org/migration/ghost): JSON without images. + - [WordPress WXR](https://wordpress.org/documentation/article/tools-export-screen/): XML, media by URL, carries users. ## Summary -`cloudhal export` writes one site of a local project to a single file, its content graph, snapshots and media; `cloudhal import` reads it back into a local project, keeping the site's id. The file is a gzipped tar holding the existing `SiteBundle` JSON and the media bytes, so the format adds a container, not a second serialisation. This gives backups, starters and templates as files, moving a site between machines, and demo seeding without a deployment and without content written as code. +`cloudhal export` writes one site of a local project to a single file, its content graph, snapshots and media; `cloudhal import` reads it back into a local project, keeping the site's id. The file is a gzipped tar of the existing `SiteBundle`'s rows as NDJSON parts plus the media bytes, so it streams for large sites and adds a container, not a second data model. This gives backups, starters and templates as files, moving a site between machines, and demo seeding without a deployment and without content written as code. ## Motivation @@ -25,23 +31,35 @@ `.cloudhal` (suggested; any name works): a gzip-compressed POSIX tar (ustar) with, in this order: ``` -manifest.json the archive's own header (below), first so it can be read without unpacking the rest -bundle.json a SiteBundle, exactly as push/pull send it (cloudhal.site-bundle@1) -media/ one file per BundleMedia entry, the raw bytes +manifest.json the archive's header (below), first so it can be read without unpacking the rest +project.json the project row, when the site has one +nodes/0001.ndjson … one node row per line, sorted by id, a new part every 8 MB +snapshots/0001.ndjson … one snapshot row per line, by creation time +blobs/0001.ndjson … {hash, body} per line: node bodies the snapshots reference +media/ one file per media entry, the raw bytes +ledger.ndjson taken-down content (the bundle's `purged` rows), when there is any ``` +The rows are the `SiteBundle`'s, one per line instead of one large JSON document, so neither export nor import has to hold the whole site in memory: Sanity and Strapi stream for the same reason. Reading the parts back in order gives exactly the `SiteBundle` that `importSite` takes. + ```ts interface SiteArchiveManifest { format: "cloudhal.site-archive@1"; + /** The contracts' schemaVersion the rows were written with (all `1` today). */ + contracts: { schemaVersion: number }; + /** The bundle format the rows reassemble into. */ + bundle: "cloudhal.site-bundle@1"; siteId: string; siteName: string; /** `@cloudhal/cli` version that wrote it. */ createdBy: string; createdAt: string; // RFC 3339 - /** What the bundle holds, so `import --dry-run` can show it without parsing bundle.json. */ - counts: { nodes: number; snapshots: number; media: number; mediaBytes: number }; - /** SHA-256 of bundle.json, so a damaged or edited file is refused. */ - bundleSha256: string; + /** What the archive holds, so `import --dry-run` can show it before reading the parts. */ + counts: { nodes: number; snapshots: number; blobs: number; media: number; mediaBytes: number }; + /** Every other file in the archive with its SHA-256 and size: a damaged or edited file is refused. */ + files: { path: string; sha256: string; bytes: number }[]; + /** True when written with --no-media: import warns that the media must come from elsewhere. */ + mediaOmitted?: true; } ``` @@ -52,33 +70,48 @@ cloudhal export [--out ] [--site ] [--no-media] [--latest-only] cloudhal import [--replace] [--dry-run] ``` -- **export:** reads the local store with `exportSite` (the project's site, or with `--site ` another site listed in `cloudhal.json` `sites`) and its media from the local blob store. Default file: `-.cloudhal`. `--no-media` writes the graph only; the manifest says so and `import` warns. `--latest-only` keeps only the published and latest snapshots, for templates that should not carry history. -- **import:** checks the manifest and `bundleSha256`, verifies every media file against its name (SHA-256) as it is written to the blob store, then calls `importSite`: - - into a project with no site, or one whose site is the archive's (same `siteId`): `mode: "bootstrap"`. It refuses diverged nodes, as `pull` does; - - with `--replace`: `mode: "replace"`; the local nodes become the archive's, and snapshots are added (they are immutable and never removed); - - into a project whose site has another id: refused unless `--replace`, which then switches `cloudhal.json`'s `siteId` to the archive's. The old site stays in the local store, unlisted; there is no command to delete a local site yet. -- `--dry-run` prints the manifest and what would change (nodes added, changed, removed; snapshots added; media to copy) and writes nothing. +- **export:** + - Streams the local store, the project's site or, with `--site `, another site listed in `cloudhal.json` `sites`, together with its media from the local blob store. + - The default file name is `-.cloudhal`. + - `--no-media` writes the graph only. + - `--latest-only` keeps only the published and latest snapshots, for templates that should not carry history. +- **import:** checks the manifest and every file's SHA-256 as it streams (media files are named by their hash as well), then calls `importSite`. Three modes: + - **bootstrap** (default), into a project with no site or one whose site is the archive's (same `siteId`). Like `pull`, it adds what is missing and refuses nodes that differ: never Strapi's "erase the target first". + - **`--replace`:** the local nodes become the archive's. Snapshots are only ever added. First, the current local graph is saved as a snapshot labelled `before import `, so the import can be undone with a rollback. Into a project whose site has another id, `--replace` switches `cloudhal.json`'s `siteId` to the archive's. The old site stays in the local store, unlisted; there is no command to delete a local site yet. + - **`--dry-run`:** prints the manifest and the difference the import would make (nodes added, changed and removed by kind and title; snapshots added; media to copy) and writes nothing. ### 3. Invariants 1. The site's id is kept: an imported site is the same site, so a later `push` to its deployment is an update, not a second site. 2. Nothing in the archive runs: it is data validated with the precompiled validators (`assertValidRows`), and gates (`assertGates`) apply as for push. -3. Every media file is verified by SHA-256 before it is used; a missing or wrong file fails the import before any node is written (the bundle's `missing_media` rule). -4. Taken-down content stays taken down: the `purged` ledger travels and the import never restores those bytes. -5. Import writes the graph in one atomic batch (no interactive transactions, AGENTS.md rule 4); a failure leaves the project as it was. -6. Same content, same file: entries are written in a fixed order with fixed timestamps in the tar headers, so exporting an unchanged site twice gives identical bytes (good for git and for diffing). +3. Every file is verified against the manifest (SHA-256 and size), and every media file against its own name, before it is used. A missing or wrong file fails the import before any node is written (the bundle's `missing_media` rule). +4. An archive written by a newer format or a newer contracts `schemaVersion` is refused with the CLI version needed. Older ones are read, and migrated forward if a later schema version requires it. +5. Taken-down content stays taken down: the `purged` ledger travels and the import never restores those bytes. +6. Import writes the graph in atomic batches (no interactive transactions, AGENTS.md rule 4), guarded like `importSite` today. A failure leaves the project as it was, and `--replace` can always be undone through its `before import` snapshot. +7. Same content, same file: rows and files are written in a fixed order, with zero timestamps and fixed modes in the tar headers, so exporting an unchanged site twice gives identical bytes (good for git and for diffing). ### 4. What is not in an archive -The content graph only, as with push/pull: no users, sessions, roles or agent keys; no comments or readers; no analytics; no audit log; no automations or schedule runs; no deployment secrets. The manifest says so, and `export` prints it. People and personal data never leave in a file by accident. +The content graph only, as with push/pull. Excluded: +- users, sessions, roles and agent keys; +- comments and readers; +- analytics; +- the audit log; +- automations and schedule runs; +- deployment secrets and webhook credentials. + +The manifest says so, and `export` prints it. People and personal data never leave in a file by accident, which is also why the format needs no encryption of its own (see Alternatives). ## Alternatives considered | Option | Pros | Cons | Why not | |---|---|---|---| | One JSON file, media inlined as base64 | One format, no container | A third larger; whole file in memory; one bad byte breaks everything | Media can be hundreds of MB | -| A directory (`bundle.json` + `media/`) | No container code; diffable in git | Not one file to send or attach; easy to copy half of it | Can be a later `--dir` option; the tar is the default | +| A directory (the same layout, unpacked) | No container code; diffable in git | Not one file to send or attach; easy to copy half of it | Can be a later `--dir` option; the tar is the default | | Zip | Opens on every OS | Needs a zip library or a writer of our own; no streaming read of a gzipped stream | Node has gzip built in, and ustar is small enough to write without a dependency | +| One `bundle.json` (the first draft) | Reuses the bundle as is | The whole site in memory on both sides; one damaged byte invalidates everything | NDJSON parts stream and are checked file by file | +| Built-in encryption (Strapi) | Protects a leaked file | Strapi's AES-128-ECB shows how easily it is done wrong; key handling in the CLI | No secrets or personal data are in the file; pipe through `age` or `gpg` when needed | +| Erase-then-import (Strapi) | Simple | Destroys local work on a wrong file | Bootstrap by default, `--replace` with a snapshot first | | Commit `.cloudhal/data.db` | No code | SQLite binary in git; schema-version bound; carries users and secrets | Unsafe and not portable | | Content as code (cloudhal.com's old way) | Reviewable diffs | Not how sites are edited; drifts from the graph | The problem #47 reports | @@ -86,18 +119,32 @@ The content graph only, as with push/pull: no users, sessions, roles or agent ke - New commands only; push, pull and the `SiteBundle` format are unchanged. - `cloudhal.site-archive@1`: a later `@2` reader keeps reading `@1`. `import` refuses unknown formats with a message naming the CLI version needed. -- `bundle.json` is `cloudhal.site-bundle@1`, so a future bundle format bump is read through `importSite`'s own format check. -- Rollout: (a) the tar writer and reader with their tests (round trip, damaged file, wrong hash); (b) `export`; (c) `import` with `--dry-run`; (d) `--replace` and the site id switch; (e) docs and a starter demo archive. +- The parts reassemble into `cloudhal.site-bundle@1`, named in the manifest, so a future bundle format bump is read through `importSite`'s own format check. +- Rollout, one PR each: + 1. The streaming tar writer and reader, with tests (round trip, damaged file, wrong hash, path traversal, deterministic bytes). + 2. `export`. + 3. `import` with `--dry-run`. + 4. `--replace`, the `before import` snapshot, and the site id switch. + 5. Docs and a starter demo archive. ## Security & policy impact -- A local CLI command, so no new scopes. Importing is like `pull`: the file is untrusted input, so it is validated, size-capped (manifest counts against actual entries, per-entry and total byte limits), and tar paths outside `manifest.json`, `bundle.json` and `media/<64 hex>` are rejected (no path traversal, no symlinks or devices). +- A local CLI command, so no new scopes. Importing is like `pull`: the file is untrusted input, so it is validated, size-capped (manifest counts against actual entries, per-entry and total byte limits), and tar paths outside the layout of §1 (`manifest.json`, `project.json`, `ledger.ndjson`, `nodes|snapshots|blobs/<4 digits>.ndjson`, `media/<64 hex>`) are rejected (no path traversal, no symlinks or devices). - Personal data: none of the stores that hold it are exported (§4). Entries may name authors, as published pages do. -- The archive is not signed in v1: it is as trustworthy as where it came from. See open question 2. +- The archive is not signed in v1: it is as trustworthy as where it came from. See Later. + +## Later (decided, not in v1) + +- **`cloudhal init --from `:** start a project from an archive (templates, the playground's demo). It is `init` plus `import`. +- **`--new-ids` for templates:** import as a copy with new ids, rewriting references (Sanity imports references weak and then strengthens them). Keeping ids is right for a backup and wrong for "use as a starter". +- **Partial export:** `--collection ` and `--no-snapshots`. +- **Export from a deployment:** `--from `, through the bundle route push and pull already use. All the headless CMS CLIs export from the remote service directly. +- **Studio "Download site archive":** the same route, from the browser. +- **Signed archives:** for example with Sigstore, like the npm provenance releases carry, so templates can be verified. -## Open questions +## Open questions (decided as recommended) -1. Should `cloudhal init --from ` start a new project from an archive (templates, the playground's demo)? **[default: yes, as a follow-up; it is `init` + `import`]** -2. Sign archives (e.g. Sigstore, like the npm provenance we publish with) so templates can be verified? **[default: not in v1]** -3. Import as a copy with new ids (`--as-new-site`) for templates used many times in one project? Rewriting ids across nodes, snapshots and redirects is a large change. **[default: not in v1; one archive, one site]** -4. Should Studio offer "Download site archive" through the admin API? **[default: later; the API already has the bundle route push/pull use]** +1. `cloudhal init --from`: yes, as a follow-up (see Later). +2. Signing: not in v1. +3. Import as a copy with new ids: not in v1; one archive, one site. +4. Studio download: later, through the existing bundle route.