masterjanic · Oct 11, 2026 · Oct 11, 2026 · Oct 11, 2026 · Oct 11, 2026 · Oct 11, 2026
diff --git a/README.md b/README.md
@@ -33,16 +33,18 @@ pve3  running  online   10.10.10.13  https://127.0.0.1:18003  127.0.0.1:18103
   link0/link1, Ceph public/cluster, migration), VLAN-aware and L2-only bridges, jumbo
   frames.
 - **Lifecycle:** start/stop, add and remove nodes (with Ceph draining), consistent
-  snapshots of the whole cluster, resumable creates.
+  snapshots of the whole cluster, live snapshots that resume a running cluster in
+  seconds, resumable creates.
 - **Fault injection:** power loss, hung nodes, pulled cables, network partitions,
   latency and loss, to test HA, fencing and Ceph recovery.
 - **Ready for automation:** API token and CA export, `proxbase env` for shells, JSON
   and the bpg/proxmox Terraform provider, `-o json` and `--progress json` for scripts,
-  and a [GitHub Action](docs/ci.md) for CI.
+  and a [GitHub Action](docs/ci.md) for CI that can cache a ready cluster as a
+  template.
 - **For AI agents:** `proxbase mcp` is an [MCP server](docs/mcp.md): agents create
   clusters, run commands on nodes, snapshot and inject faults through tools.
 - **Fast:** `--golden` clones nodes from a cached base image: a 3-node cluster in
-  about 1.5 minutes, with a unique identity per node.
+  under a minute (Ceph: 1.5 minutes), with a unique identity per node.
 - **Declarative:** one YAML file with a JSON schema; every flag overrides a field.
 - **Docker Compose:** `proxbase up` runs a cluster in the foreground with clean
   shutdown and a health check.

diff --git a/action.yml b/action.yml
@@ -39,6 +39,11 @@ inputs:
   cache:
     description: Cache the ISO and base image with actions/cache
     default: "true"
+  template:
+    description: >-
+      Cache the ready cluster as a live snapshot (memory included) and resume it in
+      later runs instead of creating it; needs golden and cache
+    default: "false"
   verify-signature:
     description: Also verify the release signature with cosign (cosign must be installed)
     default: "false"
@@ -131,7 +136,38 @@ runs:
         restore-keys: |
           proxbase-${{ runner.os }}-${{ runner.arch }}-
 
+    - name: Template cache key
+      id: template
+      if: inputs.template == 'true'
+      shell: bash
+      env:
+        NAME: ${{ inputs.name }}
+        CONFIG: ${{ inputs.config }}
+        INPUTS: ${{ inputs.nodes }}|${{ inputs.memory }}|${{ inputs.data-disks }}|${{ inputs.storage }}|${{ inputs.pve-version }}
+        VERSION: ${{ steps.install.outputs.version }}
+      run: |
+        set -euo pipefail
+        # A live snapshot only loads into the same QEMU on the same CPU model.
+        {
+          echo "$INPUTS"
+          [ -n "$CONFIG" ] && sha256sum < "$CONFIG"
+          qemu-system-x86_64 --version | head -n 1
+          grep -m 1 '^model name' /proc/cpuinfo
+          grep -m 1 '^flags' /proc/cpuinfo
+        } > "$RUNNER_TEMP/proxbase-template-key"
+        echo "key=proxbase-template-$NAME-$VERSION-$(sha256sum < "$RUNNER_TEMP/proxbase-template-key" | cut -c1-16)" >> "$GITHUB_OUTPUT"
+        echo "path=${XDG_DATA_HOME:-$HOME/.local/share}/proxbase/clusters/$NAME" >> "$GITHUB_OUTPUT"
+
+    - name: Restore cluster template
+      id: template-cache
+      if: inputs.template == 'true'
+      uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+      with:
+        path: ${{ steps.template.outputs.path }}
+        key: ${{ steps.template.outputs.key }}
+
     - name: Create cluster
+      id: create
       shell: bash
       env:
         NAME: ${{ inputs.name }}
@@ -142,8 +178,17 @@ runs:
         STORAGE: ${{ inputs.storage }}
         PVE_VERSION: ${{ inputs.pve-version }}
         GOLDEN: ${{ inputs.golden }}
+        TEMPLATE: ${{ inputs.template }}
+        TEMPLATE_HIT: ${{ steps.template-cache.outputs.cache-hit }}
       run: |
         set -euo pipefail
+        if [ "$TEMPLATE_HIT" = true ]; then
+          if proxbase snapshot restore "$NAME" template; then
+            exit 0
+          fi
+          echo "::warning::could not resume the cached cluster template; creating the cluster"
+          proxbase destroy "$NAME" --yes
+        fi
         args=(create "$NAME")
         [ -n "$CONFIG" ] && args+=(-f "$CONFIG")
         [ -n "$NODES" ] && args+=(--nodes "$NODES")
@@ -162,6 +207,27 @@ runs:
           done
           exit 1
         fi
+        if [ "$TEMPLATE" = true ] && [ "$TEMPLATE_HIT" != true ]; then
+          # Snapshot the ready cluster, stop it so the disk images are not written
+          # while they are cached, and resume it from the snapshot after saving.
+          proxbase snapshot save "$NAME" template --live
+          proxbase stop "$NAME" --timeout 0
+          echo "save-template=true" >> "$GITHUB_OUTPUT"
+        fi
+
+    - name: Save cluster template
+      if: steps.create.outputs.save-template == 'true'
+      uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+      with:
+        path: ${{ steps.template.outputs.path }}
+        key: ${{ steps.template.outputs.key }}
+
+    - name: Resume cluster template
+      if: steps.create.outputs.save-template == 'true'
+      shell: bash
+      env:
+        NAME: ${{ inputs.name }}
+      run: proxbase snapshot restore "$NAME" template
 
     - name: Export credentials
       id: env

diff --git a/docs/architecture.md b/docs/architecture.md
@@ -23,6 +23,10 @@ flowchart TD
     cluster --> nodesetup
     cluster --> image
     cluster --> answer
+    cluster --> firstboot["firstboot<br/>first-boot hook"]
+    cluster --> aptcache["aptcache<br/>package cache"]
+    cluster --> golden["golden<br/>base images"]
+    golden --> firstboot
     cluster --> state
     cluster --> pve
     cluster --> remote
@@ -45,8 +49,10 @@ flowchart TD
 | `state` | On-disk layout of a cluster, `state.json`, secrets, locking |
 | `image` | Download, verify and cache ISOs; extract the installer kernel and command line |
 | `answer` | Render the installer's `answer.toml` |
+| `firstboot` | The installer's first-boot hook: node identity before networking, base image baking; serves the hook to the installer |
+| `aptcache` | Caching HTTP proxy for the packages Proxbase installs on the nodes |
 | `vm` | Hypervisor-neutral node description and the runtime interfaces |
-| `qemu` | QEMU runtime: command lines with fixed PCI slots, install, start/stop via pidfile and QMP, qcow2 snapshots |
+| `qemu` | QEMU runtime: command lines with fixed PCI slots, install, start/stop via pidfile and QMP, qcow2 snapshots (live ones through QMP jobs) |
 | `serial` | Answer the installer's debug shell over the serial console and log it |
 | `netswitch` | Learning L2 switch over unix datagram sockets |
 | `network` | The per-cluster switch process, the nodes' NIC sockets and network faults (`run/faults.json`) |
@@ -55,7 +61,7 @@ flowchart TD
 | `pve` | Proxmox VE REST client with certificate pinning and task helpers |
 | `remote` | SSH to nodes through the forwarded ports |
 | `retry` | Retry with timeout, interval and permanent errors |
-| `golden` | Cached base images and the personalization of cloned nodes |
+| `golden` | Cached base images: build, bake, and the personalization of clones from older images |
 | `cluster` | Orchestration: create, lifecycle, nodes, snapshots, faults, status, exec |
 | `progress` | Progress events and sinks: text, JSON lines, MCP notifications |
 | `job` | Runs a create detached from its caller, with a JSON-lines progress log |
@@ -78,25 +84,52 @@ sequenceDiagram
     participant VM as QEMU (per node)
     participant Node as node (SSH)
     participant API as Proxmox API
-    CLI->>Img: ISO + SHA256SUMS (cached, verified)
+    CLI->>Img: ISO + SHA256SUMS (cached, verified; no network for a cached exact version)
     CLI->>SW: start proxbase _switch
     par every node
         CLI->>VM: install: -kernel/-initrd, answer disk PROXMOX-AIS
         VM-->>CLI: serial: debug shell
         CLI->>VM: proxmox-fetch-answer partition PROXMOX-AIS && exit
+        VM->>CLI: fetch the first-boot hook (HTTP on 10.0.2.2)
         VM-->>CLI: power off = installed
     end
-    CLI->>VM: boot nodes (daemonized)
-    CLI->>Node: hosts, repositories, bridges (nodesetup)
+    CLI->>VM: boot nodes (daemonized); clones get their identity as fw_cfg file
+    Note over VM: first boot: hook runs before networking
+    CLI->>Node: provisioning mode, hosts, repositories, bridges (nodesetup)
     CLI->>API: create cluster on node 1, join the others
-    CLI->>Node: wait for quorum and a writable /etc/pve
+    CLI->>Node: wait for quorum and a writable /etc/pve, repair lost node certificates
     CLI->>API: storage backends: ZFS pools / Ceph
     CLI->>API: API token, export CA
+    CLI->>Node: end provisioning mode
 ```
 
 Each step checks first and skips what is done, so a failed create resumes when run
 again. Progress (`installed` per node, phase, errors) is kept in `state.json`.
 
+### Speed
+
+- The installer's first-boot hook ([golden images](golden-images.md)) runs on each
+  node's first boot before networking: it skips the `pveupdate` Proxmox VE starts on
+  a first boot, removes the 5 s GRUB menu wait and, on clones, sets the identity,
+  so clones need no extra reboot.
+- While provisioning, nodes write without fsync (`zfs set sync=disabled` on `rpool`,
+  dpkg `force-unsafe-io`); the last step restores both, so a ready cluster has real
+  fsync semantics for fault injection.
+- Per-node work runs in parallel: installs, boots, configuration, ZFS pools, API
+  logins, Ceph managers and OSDs. Cluster joins, Ceph monitors and MDS daemons stay
+  sequential (the cluster file system and `ceph.conf` are not safe for concurrent
+  changes).
+- SSH probes time out their handshake after 3 s: QEMU's port forward accepts
+  connections before the guest is up and otherwise retries towards the guest only
+  every 12 s.
+- NIC 0 stays on QEMU user networking (slirp): host to node it measured 6-7 Gbit/s,
+  so passt would not speed up provisioning, and slirp also serves the nested guests
+  on `vmbr0` with DHCP.
+- Packages Proxbase installs go through a cache on the host (`aptcache`, reached as
+  `http://10.0.2.2:PORT` like the hook server), so each is downloaded once. It only
+  keeps `.deb` files; if apt rejects one (Hash Sum mismatch), Proxbase's apt runs
+  retry once with `Acquire::http::No-Cache`, which makes the cache fetch it again.
+
 ## Progress and the MCP server
 
 Long operations report through a `progress.Sink` (a function taking an `Event`).
@@ -130,6 +163,9 @@ runs in the server process, guarded by the same cluster lock as the CLI.
   for install and run, so interface names never change between them.
 - NIC 0 is QEMU user networking (`vmbr0`, NAT, port forwards); NIC 1..n are unix
   datagram sockets to the switch. MACs are derived from node and NIC number.
+- A virtio balloon with free page reporting returns memory the guest freed to the
+  host; disks use `cache=unsafe`, `discard=unmap` and `detect-zeroes=unmap`, and ZFS
+  pools in the nodes have `autotrim` on, so freed space shrinks the images.
 - The installer runs in the foreground (killed if proxbase dies); installed nodes run
   daemonized with a pidfile, a QMP socket and a serial console socket with a log file.
 - Stop: ACPI powerdown via QMP, `quit` after the timeout, then SIGKILL.
@@ -156,6 +192,7 @@ runs in the server process, guarded by the same cluster lock as the CLI.
 ~/.local/share/proxbase/jobs/  <cluster>.jsonl, .pid, .yaml of creates started over MCP
 ~/.cache/proxbase/iso/     ISOs, SHA256SUMS, extracted installer files
 ~/.cache/proxbase/images/  golden base images (create --golden)
+~/.cache/proxbase/apt/     packages served to the nodes (unused for 30 days: removed)
 ```
 
 Design decisions and measurements are in [research/feasibility.md](research/feasibility.md).
diff --git a/docs/ci.md b/docs/ci.md
@@ -1,7 +1,8 @@
 # CI and automation
 
 GitHub-hosted Linux runners expose `/dev/kvm`, so they can run a real cluster. A
-single node is ready in 2-3 minutes with a cached base image.
+single node is ready in well under a minute with a cached base image (21 s on a
+desktop host; runners are slower), and in seconds with a [template](#cluster-templates).
 
 ## GitHub Action
 
@@ -39,6 +40,7 @@ and creates the cluster with `--golden`. On failure it prints the node logs.
 | `nodes`, `memory`, `data-disks`, `storage`, `pve-version` | `1`, file defaults | As the `create` flags |
 | `golden` | `true` | Clone from a cached base image |
 | `cache` | `true` | Cache `~/.cache/proxbase` |
+| `template` | `false` | Cache the ready cluster as a live snapshot and resume it in later runs (see below) |
 | `verify-signature` | `false` | Verify with cosign (install it first) |
 
 | Output | |
@@ -53,6 +55,31 @@ step. A complete workflow is in [examples/ci/github-actions.yml](../examples/ci/
 the repository's own [E2E workflow](../.github/workflows/e2e.yml) uses the action
 from the checkout with a freshly built binary.
 
+## Cluster templates
+
+With `template: "true"` the first run creates the cluster, takes a
+[live snapshot](lifecycle.md#live-snapshots) of it, stops it, saves the cluster
+directory with `actions/cache/save` and resumes it from the snapshot. Later runs
+restore that cache and resume the cluster in seconds instead of creating it.
+
+```yaml
+      - id: pve
+        uses: virtbase/proxbase@v0
+        with:
+          nodes: "1"
+          template: "true"
+```
+
+- The cache key covers the inputs (and the cluster file), the proxbase and QEMU
+  versions and the CPU model: a live snapshot only loads into the same QEMU on the
+  same CPU. A runner on another CPU model creates and caches its own template.
+- If resuming fails, the action destroys the cluster and creates it as usual.
+- The cached cluster includes its secrets (root password, SSH key, API token). They
+  only reach a cluster bound to the runner's loopback address, but anyone who can
+  read the repository's caches can read them.
+- Each node needs cache space for its disk changes and its memory (about 2 GB for
+  one idle 4 GiB node), next to the ISO and base image cache.
+
 ## Other CI systems
 
 Any Linux runner with KVM works: install QEMU and the binary, then

diff --git a/docs/cli.md b/docs/cli.md
@@ -45,8 +45,8 @@ online and Ceph (if configured) is `HEALTH_OK`; use it for health checks.
 | --- | --- |
 | `node add [cluster] --count N` | Install, join and configure N new nodes |
 | `node remove [cluster] <node>` | Drain and remove a node (`--force` with guests or degraded Ceph pools) |
-| `snapshot save [cluster] <name>` | Snapshot every disk of the stopped cluster |
-| `snapshot restore [cluster] <name>` | Reset every disk to a snapshot |
+| `snapshot save [cluster] <name>` | Snapshot every disk of the stopped cluster; `--live`: the running cluster with its memory |
+| `snapshot restore [cluster] <name>` | Reset every disk to a snapshot; a live snapshot also starts the cluster in its saved state |
 | `snapshot delete [cluster] <name>` | Delete a snapshot |
 | `snapshot list [cluster]` | List snapshots |
 
@@ -78,7 +78,7 @@ See [fault-injection.md](fault-injection.md).
 | --- | --- |
 | `image list` | Cached base images and the clusters using them |
 | `image prune` | Delete base images no cluster uses |
-| `doctor` | Check KVM, nested virtualization, QEMU, memory, disk, ports (`--port-base`) |
+| `doctor` | Check KVM, nested virtualization, QEMU, memory, KSM, disk, ports (`--port-base`) |
 | `config init [file]` | Write an annotated example (`--force` to overwrite) |
 | `config validate <file>` | Strictly parse and check a cluster file |
 | `config schema` | Print the JSON schema |

diff --git a/docs/configuration.md b/docs/configuration.md
@@ -73,6 +73,7 @@ More in [examples/](../examples).
 | `sshKeys` | none | Extra public keys for root, next to the generated one |
 | `upgrade` | `false` | `apt dist-upgrade` on every node before clustering; reboots into a new kernel |
 | `goldenImage` | `false` | Clone nodes from a cached base image instead of installing each ([golden images](golden-images.md)) |
+| `aptCache` | `true` | Packages Proxbase installs (`upgrade`, Ceph) are downloaded once into `~/.cache/proxbase/apt` and served to the nodes from there |
 
 ### `nodes`
 

diff --git a/docs/golden-images.md b/docs/golden-images.md
@@ -11,33 +11,43 @@ proxbase image list                         # base images and the clusters using
 proxbase image prune                        # delete base images no cluster uses
 ```
 
-Measured on the same host (3 nodes, ISO cached):
+Measured on the same host (3 nodes, 4 GiB each, ISO cached):
 
 | | Installer | Golden image (cached) |
 | --- | --- | --- |
-| ZFS create | 3m04s | 1m36s |
-| Ceph create | 4m43s | 3m03s |
-| `node add` | 3m21s | 1m14s |
+| ZFS create, 1 node | – | 21s |
+| ZFS create | 2m59s | 49s |
+| Ceph create | – | 1m30s |
 
-The first golden create builds the base image (about 1m40s) and takes about as long
-as a normal create.
+The first golden create builds the base image (1m56s; 3m43s with Ceph, which
+downloads the Ceph packages) and takes about as long as a normal create.
 
 ## How it works
 
-- A base image is the root disk of an installed but **never booted** node, cached in
+- A base image is the root disk of an installed node, cached in
   `~/.cache/proxbase/images/<key>/`. The key covers the ISO, the root filesystem and
-  size, and keyboard, country and time zone; nodes with different root disks use
-  different base images.
+  size, keyboard, country and time zone, and the Ceph release if the cluster uses
+  Ceph; nodes with different root disks use different base images.
 - Each node's root disk is a qcow2 overlay of the base image; data disks are created
   fresh. The base image must stay while clusters use it, so `image prune` keeps those.
-- Everything Proxmox VE creates on first boot is created per node: the cluster CA, the
-  auth key, the node certificate and root's SSH key.
-- What the installer wrote is replaced before clustering, over SSH with the base image's
-  own key: host name (`/etc/hostname`, `/etc/hosts`, postfix), SSH host keys,
-  machine-id, root password and authorized keys. The node reboots, the base image's
-  node directory in `/etc/pve` is removed, and the new node certificate is checked.
-- A create interrupted during personalization resumes correctly: nodes that already
-  have their identity are only finished.
+- The installer installs a first-boot hook (package `firstboot`) that runs on the
+  system's first boot, before networking and before any Proxmox VE service starts.
+- **Baking.** After the install, the base image boots once with a bake script: it
+  removes the GRUB menu wait, marks the appliance list as fetched (no `pveupdate` on
+  first boot) and, for Ceph clusters, installs the Ceph packages and the newer
+  `libpve-storage-perl` they need. Then it powers off before any Proxmox VE service
+  has started, and the hook stays pending for the clones.
+- **Identity.** Each clone gets its identity as a QEMU fw_cfg file. The hook applies
+  it on the clone's first boot, before networking: host name (`/etc/hostname`,
+  `/etc/hosts`, search domain, postfix), new SSH host keys, a new machine-id, the root
+  password and the authorized keys. Everything Proxmox VE creates on its first start
+  (node certificate, root's SSH key; the cluster CA and auth key on the first node)
+  is therefore created per node, under the right name. No reboot is needed.
+- Base images from Proxbase versions before the first-boot hook have a different
+  key, so the first golden create after an upgrade builds a new one; the old ones
+  stay for the clusters that use them (`image prune` removes them afterwards). A
+  create interrupted under an older version resumes by personalizing the remaining
+  clones over SSH, with a reboot.
 
 Verified per node and across clusters: host keys, machine-ids, node certificates and
 cluster CAs all differ.