Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/bundle-project-row.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@cloudhal/core-ts": patch
---

`importSite` (behind `cloudhal pull`, the admin bundle route `cloudhal push` uses, and the coming `cloudhal import`) now checks a bundle's project row before inserting it: it must be the site node's project (`projectId`), a valid Project, with its `id` and `rev` columns matching its body. Before, a bundle could plant any projects row.
12 changes: 12 additions & 0 deletions .changeset/site-archive-export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
"@cloudhal/cli": patch
---

New `cloudhal export` (RFC-0039, #47) writes one site of the local project to a single `.cloudhal` file. The file is a gzipped tar holding the site's content graph (nodes, snapshots and the node bodies they reference, as NDJSON parts), its media files and its takedown ledger, with every file's SHA-256 listed in the manifest. The flags:

- `--out <file>` names the file, and replaces a file already there once the new one is complete. By default the file is `<site-slug>-<YYYYMMDD>.cloudhal` in the current folder, dated by the newest timestamp among the rows it holds (deleting content alone does not move that date; the next edit or snapshot does). A file already at the default name is replaced only by the same archive; any other (a later edit the same day, other flags, a deletion) is left alone and the export asks for `--out`.
- `--site <id>` exports another site listed in `cloudhal.json` `sites`. `--site`, `--no-media` and `--latest-only` are refused on every other command.
- `--no-media` writes the graph only.
- `--latest-only` keeps only the published and the latest snapshot, and only the media they and the current content need. It is refused while a field operation is unfinished (`cloudhal fields continue|revert`).

A damaged or missing local media file stops the export (run `cloudhal media verify`); so do Ctrl-C and SIGTERM. Either way no file is written and an existing one is left as it was. Exporting an unchanged site again with the same CLI and Node version gives the same bytes. An archive never contains user accounts, sessions, roles, agent keys, comments, readers, analytics, the audit log, automations, schedule runs, deployment secrets or webhook credentials. Its history does name who made each change (the user or agent of each snapshot, upload, block edit, review and takedown). `cloudhal import` comes in a later release.
7 changes: 6 additions & 1 deletion packages/cli/src/archive/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,9 @@ export {
export { encodeNdjson, type NdjsonSplitOptions, splitNdjson } from "./parts.ts";
export type { ArchiveContent, ArchiveMediaSource } from "./plan.ts";
export { type ArchiveItem, type ReadArchiveOptions, readArchive } from "./read.ts";
export { archiveStream, type WriteArchiveOptions, writeArchiveFile } from "./write.ts";
export {
archiveStream,
type WriteArchiveFileOptions,
type WriteArchiveOptions,
writeArchiveFile,
} from "./write.ts";
17 changes: 14 additions & 3 deletions packages/cli/src/archive/write.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,14 @@ export interface WriteArchiveOptions {
limits?: Partial<ArchiveLimits>;
}

export interface WriteArchiveFileOptions extends WriteArchiveOptions {
/**
* Stops the write (`cloudhal export` aborts it on Ctrl-C and SIGTERM): the temporary file is
* removed, an existing `file` stays as it was, and the promise rejects.
*/
signal?: AbortSignal | undefined;
}

/** A media body as it is written: no more than `bytes`, then the SHA-256. */
async function* verifiedMedia(
file: string,
Expand Down Expand Up @@ -96,13 +104,16 @@ export function archiveStream(

/**
* Writes the archive of `content` to `file`: to a temporary file next to it first, synced and then
* renamed over `file`, so a failure leaves no file behind and an existing `file` unchanged.
* renamed over `file`, so a failure or an abort leaves no file behind and an existing `file`
* unchanged. Only a killed process (SIGKILL, power loss) can leave the `.<name>.partial-…` file.
*/
export async function writeArchiveFile(
file: string,
content: ArchiveContent,
options: WriteArchiveOptions = {},
options: WriteArchiveFileOptions = {},
): Promise<SiteArchiveManifest> {
const { signal } = options;
signal?.throwIfAborted();
const { manifest, bytes } = archiveStream(content, options);
const temp = path.join(
path.dirname(file),
Expand All @@ -112,7 +123,7 @@ export async function writeArchiveFile(
try {
handle = await open(temp, "wx");
// The stream owns the handle: it fsyncs (flush) and closes it, on errors too.
await pipeline(bytes, handle.createWriteStream({ flush: true }));
await pipeline(bytes, handle.createWriteStream({ flush: true }), signal ? { signal } : {});
await rename(temp, file);
return manifest;
} catch (e) {
Expand Down
8 changes: 8 additions & 0 deletions packages/cli/src/assets.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,14 @@ async function packageRoot(): Promise<string> {
throw new Error("could not locate the @cloudhal/cli package root");
}

/** The `@cloudhal/cli` version (its package.json's), from either layout. */
export async function cliVersion(): Promise<string> {
const pkg = path.join(await packageRoot(), "package.json");
const version = (JSON.parse(await readFile(pkg, "utf8")) as { version?: unknown }).version;
if (typeof version !== "string") throw new Error(`${pkg} has no version`);
return version;
}

export type AssetKind = "studio" | "starters" | "skills";

const MONOREPO: Record<AssetKind, string> = {
Expand Down
24 changes: 24 additions & 0 deletions packages/cli/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ export const OPTIONS = {
views: { type: "string" },
from: { type: "string" },
out: { type: "string" },
site: { type: "string" },
"no-media": { type: "boolean" },
"latest-only": { type: "boolean" },
jurisdiction: { type: "string" },
in: { type: "string" },
file: { type: "string" },
Expand Down Expand Up @@ -70,6 +73,26 @@ export const OPTIONS = {
clear: { type: "boolean" },
} as const;

/**
* Flags only some commands take. `OPTIONS` is one table for every command, so without this a
* `cloudhal push --replace --site <other>` would parse and act on cloudhal.json's `siteId`.
*/
const ONLY_FOR: Readonly<
Partial<Record<keyof typeof OPTIONS, { commands: readonly string[]; note?: string }>>
> = {
site: { commands: ["export"], note: "; every other command acts on cloudhal.json's siteId" },
"no-media": { commands: ["export"] },
"latest-only": { commands: ["export"] },
};

function refuseForeignFlags(command: string, values: Record<string, unknown>): void {
for (const [flag, only] of Object.entries(ONLY_FOR))
if (values[flag] !== undefined && !only.commands.includes(command))
throw new Error(
`cloudhal ${command} does not take --${flag} (only cloudhal ${only.commands.join(", ")} does)${only.note ?? ""}`,
);
}

/** `cloudhal <command>` → its handler. */
export const COMMANDS: Record<string, CliCommand> = {
...LOCAL_COMMANDS,
Expand All @@ -95,5 +118,6 @@ export async function main(argv: string[]): Promise<number> {
process.stderr.write(`unknown command "${command}"\n\n${HELP}`);
return 1;
}
refuseForeignFlags(command, values);
return run(rest, values, { projectDir: process.cwd() });
}
33 changes: 31 additions & 2 deletions packages/cli/src/cli/commands-content.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,15 @@ import {
commentsStatusCli,
describeCommentsStatus,
} from "../commands/comments.ts";
import { describeExport, exportArchive } from "../commands/export.ts";
import { fieldsCommand } from "../commands/fields.ts";
import { describeMediaImport, mediaGc, mediaImportUrls, mediaVerify } from "../commands/media.ts";
import { describePlaceholders, mediaPlaceholders } from "../commands/media-placeholders.ts";
import { describeMigration, migrateTags } from "../commands/migrate.ts";
import { describeReindex, searchReindex } from "../commands/search.ts";
import { type CliCommand, remoteFlags, subcommands } from "./types.ts";
import { abortOnSignal, type CliCommand, remoteFlags, subcommands } from "./types.ts";

/** `cloudhal migrate`, `media`, `search`, `comments`: the site's content and its stores. */
/** `cloudhal migrate`, `fields`, `media`, `search`, `comments`, `export`: the site's content and its stores. */

const migrateCommand: CliCommand = async (rest, values, { projectDir }) => {
const [what] = rest;
Expand Down Expand Up @@ -138,10 +139,38 @@ const commentsCommand = subcommands(
"usage: cloudhal comments status|export|import",
);

/** Ctrl-C or SIGTERM stops the export: no file is written, an existing one stays (exit 130/143). */
const exportCommand: CliCommand = async (rest, values, { projectDir }) => {
if (rest.length > 0)
throw new Error(
"usage: cloudhal export [--out <file>] [--site <id>] [--no-media] [--latest-only]",
);
const interrupt = abortOnSignal();
try {
const result = await exportArchive({
projectDir,
...(values.out !== undefined ? { out: values.out } : {}),
...(values.site !== undefined ? { siteId: values.site } : {}),
...(values["no-media"] ? { noMedia: true } : {}),
...(values["latest-only"] ? { latestOnly: true } : {}),
signal: interrupt.signal,
});
for (const line of describeExport(result, projectDir)) console.log(line);
return 0;
} catch (e) {
if (!interrupt.signal.aborted) throw e;
console.error("cloudhal: export interrupted; no file written, an existing one left as it was");
return interrupt.exitCode();
} finally {
interrupt.dispose();
}
};

export const CONTENT_COMMANDS = {
migrate: migrateCommand,
fields: fieldsCommand,
media: mediaCommand,
search: searchCommand,
comments: commentsCommand,
export: exportCommand,
} satisfies Record<string, CliCommand>;
11 changes: 11 additions & 0 deletions packages/cli/src/cli/help.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,17 @@ Usage:
local site → deployment (refused while releases are
open unless --cancel-schedules cancels them)
cloudhal pull [--replace] [--admin-url <url>] deployment → local site
cloudhal export [--out <file>] [--site <id>] [--no-media] [--latest-only]
one site as a .cloudhal file (RFC-0039): its graph,
snapshots and media; never user accounts, comments,
analytics or secrets (its history names who made each
change). Default <site-slug>-<YYYYMMDD>.cloudhal, dated
by the newest row it holds (a deletion alone does not
move it); a file there is replaced only by the same
archive, --out <file> replaces any;
--site: a site listed in cloudhal.json sites;
--no-media: the graph only; --latest-only: only the
published and the latest snapshot
cloudhal owner transfer --to <email> [--admin-url <url>]
make a member the deployment's Owner
(lockout recovery; uses the deployment token)
Expand Down
30 changes: 30 additions & 0 deletions packages/cli/src/cli/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,36 @@ export function untilSignal(also?: (resolve: () => void) => void): Promise<void>
});
}

/**
* An `AbortSignal` that Ctrl-C and SIGTERM abort while a command writes a file. Listening replaces
* Node's default exit, which runs no `finally`, so the writer can remove its temporary file; a
* second Ctrl-C exits at once. `exitCode` is the shell's code for the signal received (130, 143);
* `dispose` stops listening.
*/
export function abortOnSignal(): {
signal: AbortSignal;
exitCode(): number;
dispose(): void;
} {
const controller = new AbortController();
let code = 1;
const handlers = (["SIGINT", "SIGTERM"] as const).map((sig) => {
const handler = () => {
code = sig === "SIGINT" ? 130 : 143;
controller.abort();
};
process.once(sig, handler);
return { sig, handler };
});
return {
signal: controller.signal,
exitCode: () => code,
dispose: () => {
for (const { sig, handler } of handlers) process.off(sig, handler);
},
};
}

/**
* A command whose first positional picks an action (`cloudhal media gc`); an unknown or missing
* action throws `usage`. The action gets the positionals after its name.
Expand Down
Loading
Loading