Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/bundle-project-row.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@cloudhal/core-ts": patch
---

`importSite` (behind `cloudhal pull`, the admin bundle route `cloudhal push` uses, and `cloudhal import`) now checks a bundle's project row before inserting it: it must be the site node's project (`projectId`), a valid Project, with its `id` and `rev` columns matching its body. Before, a bundle could plant any projects row.
20 changes: 20 additions & 0 deletions .changeset/import-plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
"@cloudhal/core-ts": patch
---

`importSite`, used by `cloudhal pull`, by the admin bundle route `cloudhal push` uses, and by `cloudhal import`, now runs every check before it writes anything, the bootstrap conflict and the 1000-statement limit included, and plans the media ledger with them. A refused bundle leaves the target as it was; before, node bodies could land ahead of a conflict and snapshots ahead of the size limit. Only a storage failure can stop it between its writes, and importing the bundle again completes it. New checks, each before the first write:

- Every node body the target lacks must be a valid contract document of the kind of each snapshot ref that points to it, with that ref's id and rev, of the bundle's site (precompiled validators). A body no snapshot references is refused. Before, a body was only checked against its hash, so a crafted bundle could put invalid content behind the published snapshot.
- Node and snapshot ids may not belong to another site of the target, each appears once, and no two nodes share a kind and key or a translation group and locale (the node batch's unique keys).
- Every takedown entry has its fields with their types (`reason`, `legal`, `by`, `at`).
- The media ledger (`snapshot_blobs`) reads the target's own asset bodies, never the bundle's text for a body the target already holds.

New `planImport(to, bundle, options)` returns what an import would change, without writing:

- the nodes added, changed and removed;
- the snapshots and node bodies to add;
- whether the project row is new;
- the listed media that are present, missing, of another size or taken down on the target;
- the bundle's takedowns, those whose bytes the target holds (an import deletes them) and those its ledger does not hold yet.

With `report: true` it returns a bootstrap conflict and missing media instead of throwing them. A node body that is not JSON is refused as `blob <hash> is not JSON` instead of with the parser's message.
12 changes: 12 additions & 0 deletions .changeset/site-archive-export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
"@cloudhal/cli": patch
---

New `cloudhal export` (RFC-0039, #47) writes one site of the local project to a single `.cloudhal` file. The file is a gzipped tar holding the site's content graph (nodes, snapshots and the node bodies they reference, as NDJSON parts), its media files and its takedown ledger, with every file's SHA-256 listed in the manifest. The flags:

- `--out <file>` names the file, and replaces a file already there once the new one is complete. By default the file is `<site-slug>-<YYYYMMDD>.cloudhal` in the current folder, dated by the newest timestamp among the rows it holds (deleting content alone does not move that date; the next edit or snapshot does). A file already at the default name is replaced only by the same archive; any other (a later edit the same day, other flags, a deletion) is left alone and the export asks for `--out`.
- `--site <id>` exports another site listed in `cloudhal.json` `sites`. `--site`, `--no-media` and `--latest-only` are refused on every other command.
- `--no-media` writes the graph only.
- `--latest-only` keeps only the published and the latest snapshot, and only the media they and the current content need. It is refused while a field operation is unfinished (`cloudhal fields continue|revert`).

A damaged or missing local media file stops the export (run `cloudhal media verify`); so do Ctrl-C and SIGTERM. Either way no file is written and an existing one is left as it was. Exporting an unchanged site again with the same CLI and Node version gives the same bytes. An archive never contains user accounts, sessions, roles, agent keys, comments, readers, analytics, the audit log, automations, schedule runs, deployment secrets or webhook credentials. Its history does name who made each change (the user or agent of each snapshot, upload, block edit, review and takedown).
13 changes: 13 additions & 0 deletions .changeset/site-archive-import.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
"@cloudhal/cli": patch
---

New `cloudhal import <file> [--dry-run]` (RFC-0039, #47) reads a `.cloudhal` file written by `cloudhal export` into the local project, keeping the site's id.

- It reads the file twice. First it verifies every file's size and SHA-256, every row and every node body (each must be the node its snapshot refs name), and checks the archive against the project. A damaged or edited file, invalid content, an archive of another site, a site whose nodes differ, or missing media is therefore refused before anything is written. Then it reads the file again, refusing it if it changed in between, copies the media the project lacks and imports the graph, keeping the archive's published snapshot as `cloudhal pull` does.
- Only bootstrap imports for now. The archive's site must be `cloudhal.json`'s `siteId` or listed in `sites`, and the project must not hold it yet or hold exactly its nodes; missing snapshots, node bodies and media are added. A site whose nodes differ is refused; `--replace` is not available yet.
- An archive written with `--no-media` imports only when the project already has its media.
- Taken-down content stays down: the archive's takedowns are recorded and their bytes never written, a local copy of content the archive's ledger took down is deleted (the summary says how many), and an archive carrying content this project has taken down is refused.
- `--dry-run` prints the manifest and what the import would add, change or remove (nodes by kind and title, snapshots, media to copy), writes nothing, and exits 1 when the import would be refused.
- Ctrl-C and SIGTERM stop the import before the graph is written; media already copied stay in the blob store, unreferenced, and the next import reuses them. A storage failure while the graph is written says the import stopped partway; running it again completes it, search index included.
- Opening an existing local store applies pending migrations, as every command does; a project without one gets none from a dry run or a refused import.
7 changes: 6 additions & 1 deletion packages/cli/src/archive/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,9 @@ export {
export { encodeNdjson, type NdjsonSplitOptions, splitNdjson } from "./parts.ts";
export type { ArchiveContent, ArchiveMediaSource } from "./plan.ts";
export { type ArchiveItem, type ReadArchiveOptions, readArchive } from "./read.ts";
export { archiveStream, type WriteArchiveOptions, writeArchiveFile } from "./write.ts";
export {
archiveStream,
type WriteArchiveFileOptions,
type WriteArchiveOptions,
writeArchiveFile,
} from "./write.ts";
17 changes: 14 additions & 3 deletions packages/cli/src/archive/write.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,14 @@ export interface WriteArchiveOptions {
limits?: Partial<ArchiveLimits>;
}

export interface WriteArchiveFileOptions extends WriteArchiveOptions {
/**
* Stops the write (`cloudhal export` aborts it on Ctrl-C and SIGTERM): the temporary file is
* removed, an existing `file` stays as it was, and the promise rejects.
*/
signal?: AbortSignal | undefined;
}

/** A media body as it is written: no more than `bytes`, then the SHA-256. */
async function* verifiedMedia(
file: string,
Expand Down Expand Up @@ -96,13 +104,16 @@ export function archiveStream(

/**
* Writes the archive of `content` to `file`: to a temporary file next to it first, synced and then
* renamed over `file`, so a failure leaves no file behind and an existing `file` unchanged.
* renamed over `file`, so a failure or an abort leaves no file behind and an existing `file`
* unchanged. Only a killed process (SIGKILL, power loss) can leave the `.<name>.partial-…` file.
*/
export async function writeArchiveFile(
file: string,
content: ArchiveContent,
options: WriteArchiveOptions = {},
options: WriteArchiveFileOptions = {},
): Promise<SiteArchiveManifest> {
const { signal } = options;
signal?.throwIfAborted();
const { manifest, bytes } = archiveStream(content, options);
const temp = path.join(
path.dirname(file),
Expand All @@ -112,7 +123,7 @@ export async function writeArchiveFile(
try {
handle = await open(temp, "wx");
// The stream owns the handle: it fsyncs (flush) and closes it, on errors too.
await pipeline(bytes, handle.createWriteStream({ flush: true }));
await pipeline(bytes, handle.createWriteStream({ flush: true }), signal ? { signal } : {});
await rename(temp, file);
return manifest;
} catch (e) {
Expand Down
8 changes: 8 additions & 0 deletions packages/cli/src/assets.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,14 @@ async function packageRoot(): Promise<string> {
throw new Error("could not locate the @cloudhal/cli package root");
}

/** The `@cloudhal/cli` version (its package.json's), from either layout. */
export async function cliVersion(): Promise<string> {
const pkg = path.join(await packageRoot(), "package.json");
const version = (JSON.parse(await readFile(pkg, "utf8")) as { version?: unknown }).version;
if (typeof version !== "string") throw new Error(`${pkg} has no version`);
return version;
}

export type AssetKind = "studio" | "starters" | "skills";

const MONOREPO: Record<AssetKind, string> = {
Expand Down
24 changes: 24 additions & 0 deletions packages/cli/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ export const OPTIONS = {
views: { type: "string" },
from: { type: "string" },
out: { type: "string" },
site: { type: "string" },
"no-media": { type: "boolean" },
"latest-only": { type: "boolean" },
jurisdiction: { type: "string" },
in: { type: "string" },
file: { type: "string" },
Expand Down Expand Up @@ -70,6 +73,26 @@ export const OPTIONS = {
clear: { type: "boolean" },
} as const;

/**
* Flags only some commands take. `OPTIONS` is one table for every command, so without this a
* `cloudhal push --replace --site <other>` would parse and act on cloudhal.json's `siteId`.
*/
const ONLY_FOR: Readonly<
Partial<Record<keyof typeof OPTIONS, { commands: readonly string[]; note?: string }>>
> = {
site: { commands: ["export"], note: "; every other command acts on cloudhal.json's siteId" },
"no-media": { commands: ["export"] },
"latest-only": { commands: ["export"] },
};

function refuseForeignFlags(command: string, values: Record<string, unknown>): void {
for (const [flag, only] of Object.entries(ONLY_FOR))
if (values[flag] !== undefined && !only.commands.includes(command))
throw new Error(
`cloudhal ${command} does not take --${flag} (only cloudhal ${only.commands.join(", ")} does)${only.note ?? ""}`,
);
}

/** `cloudhal <command>` → its handler. */
export const COMMANDS: Record<string, CliCommand> = {
...LOCAL_COMMANDS,
Expand All @@ -95,5 +118,6 @@ export async function main(argv: string[]): Promise<number> {
process.stderr.write(`unknown command "${command}"\n\n${HELP}`);
return 1;
}
refuseForeignFlags(command, values);
return run(rest, values, { projectDir: process.cwd() });
}
72 changes: 70 additions & 2 deletions packages/cli/src/cli/commands-content.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,20 @@ import {
commentsStatusCli,
describeCommentsStatus,
} from "../commands/comments.ts";
import { describeExport, exportArchive } from "../commands/export.ts";
import { fieldsCommand } from "../commands/fields.ts";
import { ImportInterruptedError, importArchive } from "../commands/import.ts";
import { describeImport } from "../commands/import-describe.ts";
import { describeMediaImport, mediaGc, mediaImportUrls, mediaVerify } from "../commands/media.ts";
import { describePlaceholders, mediaPlaceholders } from "../commands/media-placeholders.ts";
import { describeMigration, migrateTags } from "../commands/migrate.ts";
import { describeReindex, searchReindex } from "../commands/search.ts";
import { type CliCommand, remoteFlags, subcommands } from "./types.ts";
import { abortOnSignal, type CliCommand, remoteFlags, subcommands } from "./types.ts";

/** `cloudhal migrate`, `media`, `search`, `comments`: the site's content and its stores. */
/**
* `cloudhal migrate`, `fields`, `media`, `search`, `comments`, `export`, `import`: the site's
* content and its stores.
*/

const migrateCommand: CliCommand = async (rest, values, { projectDir }) => {
const [what] = rest;
Expand Down Expand Up @@ -138,10 +144,72 @@ const commentsCommand = subcommands(
"usage: cloudhal comments status|export|import",
);

/** Ctrl-C or SIGTERM stops the export: no file is written, an existing one stays (exit 130/143). */
const exportCommand: CliCommand = async (rest, values, { projectDir }) => {
if (rest.length > 0)
throw new Error(
"usage: cloudhal export [--out <file>] [--site <id>] [--no-media] [--latest-only]",
);
const interrupt = abortOnSignal();
try {
const result = await exportArchive({
projectDir,
...(values.out !== undefined ? { out: values.out } : {}),
...(values.site !== undefined ? { siteId: values.site } : {}),
...(values["no-media"] ? { noMedia: true } : {}),
...(values["latest-only"] ? { latestOnly: true } : {}),
signal: interrupt.signal,
});
for (const line of describeExport(result, projectDir)) console.log(line);
return 0;
} catch (e) {
if (!interrupt.signal.aborted) throw e;
console.error("cloudhal: export interrupted; no file written, an existing one left as it was");
return interrupt.exitCode();
} finally {
interrupt.dispose();
}
};

/**
* `cloudhal import <file> [--dry-run]` (bootstrap only; `--replace` is RFC-0039 step 4). Ctrl-C or
* SIGTERM before the graph is written stops it (exit 130/143); a dry run that would be refused
* exits 1.
*/
const importCommand: CliCommand = async (rest, values, { projectDir }) => {
const [file] = rest;
if (file === undefined || rest.length !== 1)
throw new Error("usage: cloudhal import <file> [--dry-run]");
if (values.replace)
throw new Error(
"cloudhal import --replace is not available yet; this version imports in bootstrap mode only",
);
const interrupt = abortOnSignal();
try {
const result = await importArchive({
projectDir,
file,
dryRun: values["dry-run"] === true,
signal: interrupt.signal,
log: (l) => process.stderr.write(`${l}\n`),
});
for (const line of describeImport(result)) console.log(line);
return result.refusals.length > 0 ? 1 : 0;
} catch (e) {
if (!(e instanceof ImportInterruptedError)) throw e;
console.error(`cloudhal: ${e.message}`);
return interrupt.exitCode();
} finally {
interrupt.dispose();
}
};

export const CONTENT_COMMANDS = {
migrate: migrateCommand,
fields: fieldsCommand,
media: mediaCommand,
search: searchCommand,
comments: commentsCommand,
export: exportCommand,
import: importCommand,
} satisfies Record<string, CliCommand>;
18 changes: 18 additions & 0 deletions packages/cli/src/cli/help.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,24 @@ Usage:
local site → deployment (refused while releases are
open unless --cancel-schedules cancels them)
cloudhal pull [--replace] [--admin-url <url>] deployment → local site
cloudhal export [--out <file>] [--site <id>] [--no-media] [--latest-only]
one site as a .cloudhal file (RFC-0039): its graph,
snapshots and media; never user accounts, comments,
analytics or secrets (its history names who made each
change). Default <site-slug>-<YYYYMMDD>.cloudhal, dated
by the newest row it holds (a deletion alone does not
move it); a file there is replaced only by the same
archive, --out <file> replaces any;
--site: a site listed in cloudhal.json sites;
--no-media: the graph only; --latest-only: only the
published and the latest snapshot
cloudhal import <file> [--dry-run] a .cloudhal file into this project (RFC-0039), its
site id kept: the archive's site must be cloudhal.json's
siteId or listed in sites, and absent here or holding
exactly the archive's nodes (missing snapshots and media
are added); every file is verified before anything is
written; --dry-run prints the manifest and the difference
and writes nothing
cloudhal owner transfer --to <email> [--admin-url <url>]
make a member the deployment's Owner
(lockout recovery; uses the deployment token)
Expand Down
30 changes: 30 additions & 0 deletions packages/cli/src/cli/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,36 @@ export function untilSignal(also?: (resolve: () => void) => void): Promise<void>
});
}

/**
* An `AbortSignal` that Ctrl-C and SIGTERM abort while a command writes a file. Listening replaces
* Node's default exit, which runs no `finally`, so the writer can remove its temporary file; a
* second Ctrl-C exits at once. `exitCode` is the shell's code for the signal received (130, 143);
* `dispose` stops listening.
*/
export function abortOnSignal(): {
signal: AbortSignal;
exitCode(): number;
dispose(): void;
} {
const controller = new AbortController();
let code = 1;
const handlers = (["SIGINT", "SIGTERM"] as const).map((sig) => {
const handler = () => {
code = sig === "SIGINT" ? 130 : 143;
controller.abort();
};
process.once(sig, handler);
return { sig, handler };
});
return {
signal: controller.signal,
exitCode: () => code,
dispose: () => {
for (const { sig, handler } of handlers) process.off(sig, handler);
},
};
}

/**
* A command whose first positional picks an action (`cloudhal media gc`); an unknown or missing
* action throws `usage`. The action gets the positionals after its name.
Expand Down
Loading
Loading