Repository navigation
EC2 AMI support - #311
EC2 AMI support#311martinpitt wants to merge 8 commits into
Conversation
|
As far as I know AWS supports VMDK images, too, instead of plain images. They are internally already sparse and thus smaller and simpler to handle on disk and when uploading. A Some years ago I noticed some disk corruption with VMDKs and reported it to AWS but I don't know what came out of this and if this is still reproducible. |
|
Here the docs: https://docs.aws.amazon.com/vm-import/latest/userguide/prerequisites.html#vmimport-image-formats |
|
@pothos thanks for the pointer! It's more work locally, as it has to essentially copy the existing raw image again, and the upload shouldn't make much difference. I'll test if the possibly reduced upload time makes up for the "additional local disk space + IO" overhead. |
|
For those uploading their own AMIs from a published release it's nicer to not have to extract the large image to a local disk but directly download the vmdk from the release server and upload it as AMI because the extraction of the zstd disk is not sparse by default. It can be made sparse with a hack through piping it into |
|
I tried this on our EC2 dev account. Notes:
So I won't follow that route right now. Image upload is by and large an internal implementation detail, so we can tweak this later. What I can do easily is to add a |
|
@pothos |
|
SSH is still broken. There's an /usr/share/empty.sshd dir with 0775 permissions which sshd refuses. Probably from my local |
|
SSH fixed in #314 |
| command_alias( | ||
| name = "aws", | ||
| args = ["aws"], | ||
| # The box has no pager, neither aws' `less` default nor the host's $PAGER; aws breaks without it. |
There was a problem hiding this comment.
Shouldn't we just put a pager in the box? less isn't exactly the end of the world
There was a problem hiding this comment.
Could do. But we still need to hardcode it then. A host's $PAGER may already be set, to nvim or delta or bat or whatever, and the box can't grow all of them. I feel like paging rather belongs to the host, not the box.
There was a problem hiding this comment.
Thinking about it again, IMHO hardcoding less from the box is worse than just letting the host env do it.
Publishing images to EC2 needs the `aws` CLI, both to log in and to drive the EC2 and EBS APIs. Ship it in its own box, as it bundles an entire boto stack and is *huge*, and it's not always needed. Relaxed entry point so that it uses the host's AWS login. tine then never has to handle credentials itself. Signed-off-by: Martin Pitt <[email protected]>
A cloud image takes its SSH key and other credentials from the cloud's instance metadata service (IMDS), which systemd imports in the initrd. Name the packages that needs per distro, so that an initrd can ask for them symbolically like for the other sets. Signed-off-by: Martin Pitt <[email protected]>
A cloud instance gets its SSH key and other credentials from the instance metadata service, and systemd imports them in the initrd if it carries the IMDS tools and a network. Add an `imds` option to the initrd, and to a disk's conventional one: that adds the previous commit's package set and starts networkd for the IMDS network that systemd sets up. All of this would be dead weight for a real-iron deployment or local VM, so don't enable this option by default. The initrds are going to differ anyway per cloud target due to the cloud specific modules that we are going to add later. Signed-off-by: Martin Pitt <[email protected]>
Deploying a disk somewhere other than a local VM, such as registering it as an AMI, needs to know its architecture, its published name, the key firmware must trust, and whether its initrd takes credentials from IMDS. Only the composition knows these, so expose them in a provider rather than having each consumer repeat them as attributes. Signed-off-by: Martin Pitt <[email protected]>
An AMI is not file format but an EC2 object referring to a disk snapshot and its boot parameters. Its disk is specific to EC2 too: the initrd must take credentials from IMDS and needs EC2's network driver to reach it. Add an `image.ami()` declaration that declares such a disk, and gathers the information that EC2 `RegisterImage` needs: the raw disk, UEFI + IMDSv2 and optionally a TPM. Add an AMI for our boot-demo example. Signed-off-by: Martin Pitt <[email protected]>
Turn the previous commit's bundle into an actual EC2 AMI image: Upload the disk as an EBS snapshot and register it with the bundle's parameters, through the AWS CLI with the user's login. Only blocks holding data are uploaded, as disks commonly still have lots of empty space on partitions. That needs an unencrypted snapshot, so encrypted ones are not supported for now. If/when we do, we need to avoid big holes in images first. Signed-off-by: Martin Pitt <[email protected]>
An instance of a signed AMI booted in Secure Boot setup mode, so it ran without enforcement until something enrolled keys, and whatever booted first could enroll its own. Register the AMI with a UEFI variable store that enrolls the disk's signing certificate, and add a Secure Boot AMI with a TPM to the secureboot example. Signed-off-by: Martin Pitt <[email protected]>
Useful for VMWare or cloud importers such as AWS VM Import (you need an S3 bucket and the `import` role privilege). Signed-off-by: Martin Pitt <[email protected]>
|
Rebased after #312, there were some extensive conflicts after that. |
| "buildroot": ["base-devel"], | ||
| # Arch ships IMDS support and networkd inside systemd. systemd is already there, but a set | ||
| # cannot be empty. | ||
| "imds": ["systemd"], |
There was a problem hiding this comment.
Shouldn't we just add imds to the initrd by default? It can't be that big. As long as we don't enable it by default it's fine.
| secure_boot_key: str | None = None, | ||
| sign_expected_pcr_key: str | None = None, | ||
| initrd: str | Select | None = None, | ||
| imds: bool = False, |
There was a problem hiding this comment.
Assuming we can just add imds to the initrd by default. I would really prefer not to have a boolean here to enable/disable imds. Should be a runtime decision, not a build time one.
| "basename": provider_field(str), | ||
| "disk": provider_field(Artifact), | ||
| # Whether its initrd imports credentials from a cloud's instance metadata service. | ||
| "imds": provider_field(bool), |
There was a problem hiding this comment.
Dislike this, should be a runtime decision, not a build time one assuming imds isn't too big to include by default.
| # Whether its initrd imports credentials from a cloud's instance metadata service. | ||
| "imds": provider_field(bool), | ||
| # The certificate of the key its boot chain is signed with, which firmware must trust. | ||
| "secure_boot_certificate": provider_field(Artifact | None), |
There was a problem hiding this comment.
Don't we have a separate boot provider already that it would make sense to put this in?
I don't like having a combo of disk and bootable in one provider. It seems rather we should two distinct providers, one to cover the disk image aspect, one to cover the bootable aspect.
| image_id = image_id or name, | ||
| initrd = initrd, | ||
| imds = initrd == None, | ||
| initrd_modules = (initrd_modules if initrd_modules != None else _INITRD_MODULES["patterns"]) + _EC2_INITRD_MODULES, |
There was a problem hiding this comment.
Similarly, let's see how big these are, maybe include these modules by default?
This branch has not been deployed
3 participants
I tested this end-to-end with both the demo and the secureboot demo images, with getting the root login shell on the serial console. (Neither of these has sshd). I tested this with our downstream OS ✔️