Repository navigation
perf: faster provisioning and lower resource use - #8
Merged
Merged
Conversation
QEMU's port forward accepts connections while the guest still boots and retries towards the guest only every 12s, so a probe without a handshake deadline could stall until then.
Nodes boot without the 5s GRUB menu. During create, rpool has sync=disabled and dpkg skips its syncs; the last step restores both so ready clusters keep real fsync semantics for fault injection. rpool gets autotrim.
API logins, managers and OSDs run on all nodes at once; monitors and MDS daemons stay sequential. OSD memory target and MDS cache are capped for small nodes unless already set.
A virtio balloon with free page reporting at a fixed PCI slot, and detect-zeroes=unmap on the disks.
Right after a join a node can reject a fresh ticket for a moment. Clients are now safe for concurrent use, and their idle connections time out.
The domain ends up in shell scripts on the nodes.
Logs in to the joining nodes in parallel, then before each join waits until the first node is quorate with the expected votes and /etc/pve is writable; the API reports quorum earlier. A join refused for lack of quorum is retried.
A joining node finishes its setup before the cluster file system has synced to it. Its key and certificate can be lost, or it writes a new cluster CA, and a failed join leaves pvedaemon and pveproxy rejecting logins. After the joins every node gets a new certificate from the current CA where needed.
vm.Spec.Files become QEMU fw_cfg files; BootOnce runs an installed node until it powers off and returns its serial console.
The installer fetches the hook from the host and the node runs it once, before networking: it skips the pveupdate of a first boot and removes the GRUB menu wait. It also applies an identity or a bake script passed as guest files, used by golden images. A ceph.sources that does not point at the enterprise repository is no longer disabled.
Clones get host name, SSH host keys, machine-id, password and keys from the first-boot hook before networking, so personalization needs no reboot. Base images are booted once to bake them (GRUB, pveupdate, and for Ceph clusters the Ceph packages) and power off before Proxmox VE services start. Base images from before the hook are still personalized over SSH.
A caching HTTP proxy, reached by the nodes as 10.0.2.2, serves the upgrade, Ceph install and base image bake. It only stores .deb files of the Debian and Proxmox repositories; apt runs retry once with no-cache so a broken package is fetched again. Configurable with proxmox.aptCache (default true).
snapshot save --live (MCP: live) pauses all nodes and saves their disks and memory through QMP snapshot-save. Restoring one starts every node paused from it, resumes them together, sets their clocks and waits for quorum.
With template: true the first run saves a live snapshot of the new cluster in the actions cache; later runs on the same QEMU and CPU model resume it.
After a join, pveproxy can keep serving a certificate that /etc/pve no longer holds, so logins pinned to the stored certificate fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Faster creates and less host memory and disk use; one commit per change.
pveupdate, Ceph packages for Ceph clusters).proxmox.aptCache) for upgrade, Ceph install and bakes.snapshot save --live) and cluster templates in the GitHub Action.detect-zeroes, autotrim, Ceph memory caps, KSM check indoctor.main), joins sent before the first node accepts them, rejected tickets after a join, unvalidatedproxmox.domain.How it was tested
go test -race ./..., golangci-lint, shellcheck; each commit builds and passes tests.proxmox.upgradeand Ceph without--golden(the test host's connection to deb.debian.org drops downloads), the action'stemplateinput on a GitHub runner.