[gmpinder]

This is the start of the new build engine that will be replacing the existing, what I'd like to call, legacy build system. This feature makes use of the individual layer control that buildah gives you.

Layers

The pattern I went with is a DAG structure that holds the instructions for running each step or Layer. This makes it so that you can construct a build "plan" without invoking the build immediately. There are several layers types that we will start with and can expand in the future if we want special build steps. Those types are:

• FromLayer
  • This layer is the "root" in a build graph and is similar to FROM
  • Define an image, a scratch container, or another completed build (LayerId) to start from
  • The platform for the build is defined here
• RunLayer
  • This layer is used to run instructions on the working container
  • Similar to the RUN instruction
  • Allows mounting images or directories
• CopyLayer
  • This layer is used to copy files directly into the build
  • Similar to COPY instruction
  • Allows copying from the build context or another image
• UnshareLayer
  • One of the special types of layers not found in a Containerfile
  • Allows running host commands under the buildah unshare command
  • Automatically mounts the working container and sets the path on BB_UNSHARE_MOUNT
  • Currently used for extracting the embeded build scripts into their own image

Build step state machine

When using buildah from, a new working container is produced and you are able to run any series of commands in that container like buildah run or buildah copy. A layer isn't created until buildah commit is ran on the working container. When the image is committed, a LayerId is produced that can be used to create another layer ontop. So when it comes to doing a squashed build, you are basically holding off on running buildah commit until the end. That way you create a single layer on top of your base image.

In order to track that state, I had to create a struct that would hold that information and enforce a state-machine operation so that a squashed build couldn't accidentally turn into a layered build mid-way. To do that I created the BuildStep struct. This object can contain either a Container or LayerId object as it is passed up the stack to the end of the build. In this struct I make use of the PhantomData marker type. This allows me to use a unit struct like ReadyState or RunningLayeredState to mark where we are in the build. I then created some traits that have implementations for a set combination of generics on BuildStep so that when you call .container() on a BuildStep<LayerId, ReadyState>, it takes owner ship and returns a (Container, BuildStep<(), RunningLayeredState>) tuple. This allows us to run a Layer on that Container. Once we are done with that container, we pass it back into the BuildStep with .commit(container) to produce a BuildStep<LayerId, ReadyState> to pass on to the next layer.

This struct will allow us to attach context to the build as we need and to ensure that we don't call buildah in the wrong order. One of the things that I will want to use it for before we release this is to attach the number of steps in that are in a stage, the current step index we're on, the image name, and the log header color. This would allow us to make use of indicatif to show a progress bar for each stage in the build.

Tasks to get to parity

• Decorated logging
• Secret mounts
• Akmods special layer
• Bin installs for cosign, bluebuild, and nushell
• Tagging and pushing images
• chore(next-builder): Hook the next-gen builder into the cli frontend #905
• Image chunking
• Prebuild, postbuild, and setup script layers
• Signing pushed images
• Image labeling
• Cached layer detection