Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/rfcs/0046-release-bundles.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,8 +206,8 @@ Tar format: ustar with PAX `path` records for long names; regular files only (no
- The route first shipped with one manual docs deploy. Since #236 the *Docs deploy* workflow deploys cloudhal.dev on every push to `main` that changes what it is built from, `security/**` included (RFC-0037).
- The signatures make the location trust-free; the location only matters for availability.
- **The root.** An Ed25519 root key signs only `keys.json`, at the yearly rotation or in an emergency.
- The owner generates it offline with `node scripts/release-keys.ts root-init`. The script writes the private key encrypted with a passphrase, never prints it, and writes the public key next to it.
- It has one holder for now, with two sealed copies kept in separate places. A YubiKey (firmware 5.7 or later, for Ed25519) is optional.
- The owner generates it offline with `node scripts/release-keys.ts root-init`. The script writes the private key encrypted with a passphrase (PKCS#8 PBES2, PBKDF2-HMAC-SHA256 at 600,000 iterations, AES-256-CBC), never prints it, and writes the public key next to it.
- It has one holder for now. The *encrypted* file has two sealed copies: one in the private repository `cloudhal-dev/key-custody` (owner only; no Actions, apps or deploy keys; no passphrase and no plaintext key in it), and one kept offline. The passphrase is kept apart, in an end-to-end encrypted password manager and on sealed paper. That repository copy is the one documented exception to keeping keys out of repositories and cloud services: the release key and the yank key stay out of them apart from their environment secrets, and no passphrase is ever in a repository, a chat, an issue, or a cloud service other than an end-to-end encrypted password manager. A YubiKey (firmware 5.7 or later, for Ed25519) is optional.
- A second holder is added when there is a second maintainer.
- `@cloudhal/release-verify` pins the root's public key, and a repository test keeps it equal to the key doc.
- **`keys.json`.**
Expand Down
25 changes: 19 additions & 6 deletions docs/security/release-signing-keys.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,22 @@ Other forms of `cloudhal-root-2026a`, for cross-checking:
The first `keys.json` was signed by it on 2026-11-15 (`issuedAt` 2026-11-15T07:00:45Z, expiring 2027-11-15T07:00:45Z) and is committed as `security/keys.json` with `security/keys.json.sig`.

**Custody.**
- The root is generated offline by `node scripts/release-keys.ts root-init`. The private key is only ever written encrypted with a passphrase, and the script never prints it.
- There is one holder for now, with two sealed copies (the encrypted file plus its passphrase) kept in separate places. A YubiKey with firmware 5.7 or later, which can hold Ed25519, is an option.
- The root is generated offline by `node scripts/release-keys.ts root-init`. The private key is only ever written encrypted with a passphrase (PKCS#8 PBES2: PBKDF2-HMAC-SHA256 at 600,000 iterations, AES-256-CBC), and the script never prints it.
- There is one holder for now. The *encrypted* file has two sealed copies:
- one in the private repository `cloudhal-dev/key-custody`, which only the owner can read. It has no Actions, no apps and no deploy keys, and it holds neither the passphrase nor a plaintext key;
- one kept offline.
- The passphrase is kept apart from both copies, in an end-to-end encrypted password manager and on sealed paper.
- That repository copy is the one exception to the rule that no key is ever in a repository or a cloud service (*Custody of the release key*). It is the encrypted root only, never its passphrase.
- A YubiKey with firmware 5.7 or later, which can hold Ed25519, is an option.
- A second holder is added when there is a second maintainer.
- The root is never a GitHub secret.
- **A root made before 600,000 iterations.** `root-init` used to write the root with Node's default of only 2,048 PBKDF2 iterations. `sign-keys` reads both forms. Re-encrypt such a file once, offline, with the same passphrase or a new one:

```bash
openssl pkcs8 -topk8 -v2 aes-256-cbc -v2prf hmacWithSHA256 -iter 600000 -in <id>.key.enc -out <id>.key.enc.new
```

Check it with `openssl asn1parse -in <id>.key.enc.new`: the PBKDF2 `INTEGER` is `0927C0` (600,000) and the prf is `hmacWithSHA256`. Then replace both sealed copies with it and delete the old file. `cloudhal-root-2026a` was re-encrypted this way on 2026-11-15.

## The yank key and the two lists

Expand All @@ -73,6 +85,7 @@ The first `keys.json` was signed by it on 2026-11-15 (`issuedAt` 2026-11-15T07:0
Raw public key, hex: `9029d1ea7755084550f84a0d0c5b0ff6d2ae49c3fac41f797a67100b366c346e`; SHA-256 of it: `a4be5c2588264f4d824705d85bf96cd4b214ab553c52e66364e71d18c0e9f390`. A test keeps this row equal to the committed `keys.json`.

- It lives as the secret `RELEASE_YANK_KEY` of the GitHub environment `release-yank`, whose branch policy allows `main` only.
- That secret is its only copy: the plaintext key was deleted once the secret was set. If it is lost, the yank key is rotated (one `keys.json`, below), never restored. The same holds for the `yank-list` deploy key (*First-time setup*, step 7).
- It can only reduce trust: it yanks versions and revokes release key ids, and it can never add or re-enable a key.
- A stolen yank key can yank good versions and revoke good keys (denial of service).
- It can also *undo* yanks and revocations for stateless consumers, such as a Worker deployer, by signing a fresh list without them.
Expand Down Expand Up @@ -100,22 +113,22 @@ The first `keys.json` was signed by it on 2026-11-15 (`issuedAt` 2026-11-15T07:0
- No other job, workflow or repository receives the key.
- Outside GitHub, the maintainer holds the key.
- The copies are the PKCS#8 file the key was generated into, kept readable by its owner only, and the maintainer's backups.
- No copy is ever in a repository, a chat, an issue or a cloud service.
- Apart from their environment secrets, no copy of the release key or the yank key is ever in a repository, a chat, an issue or a cloud service, and no passphrase is ever in a repository, a chat, an issue, or a cloud service other than an end-to-end encrypted password manager. The one exception is the root's *encrypted* file, without its passphrase, in the private `cloudhal-dev/key-custody` repository (*Root keys*, Custody).
- Once the environment secret is set, and before the first signed release, the maintainer moves the generated file to offline storage. From then on, the environment secret and the offline backups are the only copies.
- KMS custody is deferred to phase 3 ("Release signing: KMS-backed signer (RFC-0046 §9 phase 3)"). It is triggered by a second maintainer, or by any exposure of this secret.

## First-time setup (the owner's ceremony)

This runs once, offline where it says so. Nobody else generates or holds these keys.

1. **Root.** On an offline machine, run `node scripts/release-keys.ts root-init --key-id cloudhal-root-2026a --out <dir>`. It asks for a passphrase twice, then writes `<dir>/cloudhal-root-2026a.key.enc` (encrypted PKCS#8) and `<dir>/cloudhal-root-2026a.pub.json`. Seal two copies of the encrypted file and the passphrase, in separate places.
1. **Root.** On an offline machine, run `node scripts/release-keys.ts root-init --key-id cloudhal-root-2026a --out <dir>`. It asks for a passphrase twice, then writes `<dir>/cloudhal-root-2026a.key.enc` (encrypted PKCS#8, 600,000 PBKDF2 iterations) and `<dir>/cloudhal-root-2026a.pub.json`. Seal two copies of the encrypted file (*Root keys*, Custody) and keep the passphrase apart from them.
2. **Pin the root.** Add its public key to the *Root keys* table above and to `PINNED_ROOTS` in `packages/release-verify/src/trust.ts`, in one reviewed pull request.
3. **Yank key.** Run `node scripts/release-keys.ts yank-init --key-id cloudhal-yank-2026a --out <dir>`. It writes `<dir>/cloudhal-yank-2026a.pem` (mode 0600, never printed) and `<dir>/cloudhal-yank-2026a.pub.json`.
4. **Environment.** In the repository settings, create the environment `release-yank`, with deployment branches `main` only and no required reviewer. Set its secret: `gh secret set RELEASE_YANK_KEY --env release-yank < <dir>/cloudhal-yank-2026a.pem`. Then move the `.pem` file to offline storage.
4. **Environment.** In the repository settings, create the environment `release-yank`, with deployment branches `main` only and no required reviewer. Set its secret: `gh secret set RELEASE_YANK_KEY --env release-yank < <dir>/cloudhal-yank-2026a.pem`. Then delete the `.pem` file: the secret is its only copy, and a lost yank key is rotated, not restored.
5. **First `keys.json`.** List `cloudhal-release-2026a` and `cloudhal-yank-2026a` in `security/keys.source.json`, then run `node scripts/release-keys.ts sign-keys --root-key <dir>/cloudhal-root-2026a.key.enc --root-pub <dir>/cloudhal-root-2026a.pub.json`. It writes `security/keys.json` and `security/keys.json.sig`, after verifying the exact signed bytes as a consumer would.
6. **Publish the key list.** Commit `security/keys.json`, `security/keys.json.sig` and `security/keys.source.json` in a reviewed pull request. The yank list needs it on `main` (the workflow reads `security/keys.json`). **Merge it only while no release is pending, and do steps 7 to 9 straight after:** from this merge on, every release attaches the trust lists and refuses to publish until cloudhal.dev serves a `yanked.json` that verifies.
7. **Protect the `yank-list` branch, with a deploy key as its only writer.**
- Run `ssh-keygen -t ed25519 -N "" -C yank-list -f <dir>/yank-list-deploy`. Add `<dir>/yank-list-deploy.pub` as a deploy key of the repository with write access, named `yank-list`. Set the private half as a secret of `release-yank`: `gh secret set YANK_LIST_DEPLOY_KEY --env release-yank < <dir>/yank-list-deploy`. Then delete both files.
- Run `ssh-keygen -t ed25519 -N "" -C yank-list -f <dir>/yank-list-deploy`. Add `<dir>/yank-list-deploy.pub` as a deploy key of the repository with write access, named `yank-list`. Set the private half as a secret of `release-yank`: `gh secret set YANK_LIST_DEPLOY_KEY --env release-yank < <dir>/yank-list-deploy`. Then delete both files. If the secret is lost, make a new deploy key the same way and remove the old one.
- In the repository settings, add a ruleset for the branch `yank-list`: block force pushes, restrict deletions, and restrict updates, with the `yank-list` deploy key as the **only** bypass actor (not "GitHub Actions", which would let every workflow's token with `contents: write` push).
- The branch then only ever gains commits, and only from a job that holds the deploy key, which is a job in `release-yank` on `main`.
- **What remains.** Anyone who can run a job in `release-yank` (a reviewed change to a workflow on `main`) or a repository admin (who can edit the ruleset) can push to it. A deploy key with write access can also push to other branches no ruleset protects. Such a push can only deny service: a crafted list fails the next run and the cloudhal.dev pair check. A replayed older, genuinely signed pair would be served to stateless consumers until it expires (at most 30 days); consumers with memory refuse it. A rollback needs a new commit that the branch history shows.
Expand Down
49 changes: 49 additions & 0 deletions packages/cli/test/release-keys.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
* scripts/release-keys.ts (RFC-0046 §9, #226): the root and yank keys a ceremony makes, and the
* lists it signs, checked with the consumer's own verifier. Throwaway keys in temp directories only.
*/
import { createPrivateKey, createPublicKey, generateKeyPairSync } from "node:crypto";
import { mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
Expand All @@ -11,6 +12,8 @@ import {
main,
mergeYankList,
type PublicKeyFile,
ROOT_KDF_ITERATIONS,
readPbes2,
rootInit,
signKeys,
signYanked,
Expand Down Expand Up @@ -99,6 +102,26 @@ describe("root-init and yank-init", () => {
expect(said).not.toContain(pem.split("\n")[1] as string);
});

it("encrypt the root with PBKDF2-HMAC-SHA256 at 600,000 iterations and AES-256-CBC", () => {
const { dir, root } = ceremony();
const enc = rootKeyOf(dir);
expect(ROOT_KDF_ITERATIONS).toBeGreaterThanOrEqual(600_000);
expect(readPbes2(enc)).toEqual({
iterations: 600_000,
saltBytes: 16,
prf: "hmacWithSHA256",
cipher: "aes-256-cbc",
});
// Node (OpenSSL) opens it with the passphrase, and it is the root that was announced.
const key = createPrivateKey({ key: enc, passphrase: PASS });
const raw = createPublicKey(key).export({ format: "der", type: "spki" }).subarray(12);
expect(Buffer.from(raw).toString("base64")).toBe(root.publicKey);
expect(() => createPrivateKey({ key: enc, passphrase: "wrong passphrase!" })).toThrow();
// Each file has its own salt.
expect(readPbes2(rootKeyOf(ceremony().dir))).toEqual(readPbes2(enc));
expect(rootKeyOf(ceremony().dir)).not.toBe(enc);
});

it("refuse to overwrite a key, a short passphrase and a key id without its role", () => {
const { dir } = ceremony();
expect(() => yankInit({ keyId: "cloudhal-yank-2026a", out: dir, log: quiet })).toThrow(
Expand Down Expand Up @@ -131,6 +154,32 @@ describe("sign-keys", () => {
expect(readFileSync(path.join(out, "keys.json.sig")).length).toBe(64);
});

it("still opens a root written before the fix (2,048 iterations, Node's default export)", async () => {
const { source } = ceremony();
const { privateKey, publicKey } = generateKeyPairSync("ed25519");
const old = privateKey.export({
format: "pem",
type: "pkcs8",
cipher: "aes-256-cbc",
passphrase: PASS,
});
expect(readPbes2(old).iterations).toBe(2048);
const raw = publicKey.export({ format: "der", type: "spki" }).subarray(12);
const oldRoot = {
keyId: "cloudhal-root-2026a",
publicKey: Buffer.from(raw).toString("base64"),
};
const signed = await signKeys({
rootKey: old,
passphrase: PASS,
rootPub: oldRoot,
source,
out: tmp(),
now: NOW,
});
await verifyKeys({ ...signed, roots: roots(oldRoot), now: new Date(NOW) });
});

it("refuses a wrong passphrase, another root's key, and more than 13 months", async () => {
const { dir, root, source } = ceremony();
const out = tmp();
Expand Down
13 changes: 6 additions & 7 deletions scripts/release-keys.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
* document the way consumers do before it is written. No command ever prints a private key.
*
* node scripts/release-keys.ts root-init --key-id cloudhal-root-2026a --out <dir> [--passphrase-file <f>]
* An Ed25519 root: <dir>/<id>.key.enc (PKCS#8, AES-256 with the passphrase) + <id>.pub.json.
* An Ed25519 root: <dir>/<id>.key.enc (PKCS#8, AES-256-CBC under PBKDF2-HMAC-SHA256 with
* 600,000 iterations of the passphrase) + <id>.pub.json. sign-keys also reads older roots.
* node scripts/release-keys.ts yank-init --key-id cloudhal-yank-2026a --out <dir>
* The yank key: <dir>/<id>.pem (mode 0600, for the RELEASE_YANK_KEY secret) + <id>.pub.json.
* node scripts/release-keys.ts sign-keys --root-key <id>.key.enc --root-pub <id>.pub.json
Expand Down Expand Up @@ -35,6 +36,7 @@ import { createInterface } from "node:readline";
import { fileURLToPath } from "node:url";
import { parseArgs } from "node:util";
import { canonicalJson, sha256 } from "./release-bundle/manifest.ts";
import { encryptPkcs8 } from "./release-keys/pkcs8.ts";
import {
checkAppendOnly,
mergeYankList,
Expand All @@ -45,6 +47,7 @@ import {
type YankSource,
} from "./release-keys/yank-list.ts";

export { type Pbes2Params, ROOT_KDF_ITERATIONS, readPbes2 } from "./release-keys/pkcs8.ts";
export {
checkAppendOnly,
MAX_EXTRA,
Expand Down Expand Up @@ -115,12 +118,8 @@ export function rootInit(o: {
if (o.passphrase.length < 12) throw new Error("the passphrase must be at least 12 characters");
mkdirSync(o.out, { recursive: true });
const { privateKey, publicKey } = generateKeyPairSync("ed25519");
const enc = privateKey.export({
format: "pem",
type: "pkcs8",
cipher: "aes-256-cbc",
passphrase: o.passphrase,
});
// Not `privateKey.export({ cipher })`: that is 2,048 PBKDF2 iterations, with no way to ask for more.
const enc = encryptPkcs8(privateKey, o.passphrase);
writeNew(path.join(o.out, `${o.keyId}.key.enc`), enc, 0o600);
o.log(`wrote ${o.keyId}.key.enc (encrypted; seal two copies with the passphrase, apart)`);
return writePublic(o.out, o.keyId, publicKey, o.log);
Expand Down
Loading
Loading