Skip to content

Add support for creating the monitor's execution environment with libcontainer - #982

Open
cmainas wants to merge 11 commits into
mainfrom
feat/libcontainer
Open

Add support for creating the monitor's execution environment with libcontainer#982
cmainas wants to merge 11 commits into
mainfrom
feat/libcontainer

Conversation

@cmainas

@cmainas cmainas commented Aug 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Adds an experimental, opt-in path that uses runc's libcontainer to setup the monitor's execution environment (namespaces, cgroups, mounts, device nodes, pivot into the monitor rootfs) instead of urunc's native way. It is selected per-runtime via a new config flag and is off by default; the existing reexec/nsenter path is untouched when the flag is off.

The workflow:

  1. urunc create builds a libcontainer config from the container's OCI spec with re-rooted at the monitor rootfs, carrying the
    monitor mounts/devices gathered in InitialSetup and creates, then starts the init that will become the urunc monitor process. It also writes a .monitor_spec.json into the monitor rootfs with everything the urunc monitor process needs to finalize the setup.
  2. The init blocks on libcontainer's exec fifo.
  3. urunc start loads the container and calls Exec() (opens the exec fifo, releasing the init), then waits on the urunc's ready pipe.
  4. The released init re-execs the urunc binary with urunc monitor cli option. Then ExecMonitor reads the spec back, creates the tap device, drops to the container user, signals ready, and execves the monitor which replaces it and becomes the container PID.
  5. urunc delete tears down the monitor's libcontainer state + cgroup and removes the monitor's spec file.

There have been some new tests for the new functionality in the CI. The tests simply re-execute the vm_tests and the kind tests after changing the urunc configuration to enable the libcontainer mode.

Things to note:

  • vAccel is out of scope of this PR
  • Currently cgroups are just for placement, because we might get OOM if we set up the limits we get. We need to add the sandbox overhead.
  • For virtiofsd the --sandbox cli option must be set to off, because inside the monitor container the process that spawns virtiofsd does not have the necessary capabilities to create a namespace. Will be addressed in a follow up PR.
  • Future TODOs have been added in the code.
  • There is a split of libcontianer for CGO and non-CGO code. This is necessary because the shim is not built with CGO and due to its dependency in unikontainers, it fails to link. We need to address this in the future too.
  • The metrics have been messed up and we need to come up with a new way to use them, because the "urunc monitor" process does not have access to the metrics file.
  • The logs in the "urunc monitor" process are shown in stdout and not redirected in the log file/socket. We should resolve this in future iterations.
  • As reported by @ananos for the libcontainer case the urunc binary must be statically linked. Otherwise the libcontainer trick of /proc/self/exe will fail due to missing libraries in the case of Firecracker (note, we do not add any host libraries for Firecracker in its execution environment).
  • Static networking mode does not work with libcontainers. The "urunc monitor" misses the iptables binary inside its execution environment.

Related issues

How was this tested?

With e2e tests

LLM usage

Opus 4.8 for various rewrites, unit tests.

Checklist

  • I have read the contribution guide.
  • The linter passes locally (make lint).
  • The e2e tests of at least one tool pass locally (make test_ctr, make test_nerdctl, make test_docker, make test_crictl).
  • If LLMs were used: I have read the llm policy.

cmainas added 7 commits July 13, 2026 17:13
@cmainas
feat(libcontainer): Use runc's init
8a87a2c 
Since, we are going to use libcontainers, we need the init of runc which
intercepts the "normal flow" of starting a GO program from main and
continues with the setup of the container process ( after the namespaces
have been created).

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
feat(libcontainer): write the monitor spec on create
974e969 
The monitor process runs in its own rootfs and cannot reach the state
dir, so InitialSetup marshals everything it needs into
.monitor_spec.json inside that rootfs.

- Move MonitorSpec out of types into monitor_spec.go and extend it
  with everything the urunc monitor process needs to finalize the setup.
- Guest rootfs path rewritten to "/": the urunc monitor process has
  pivoted inside the new rootfs.
- Environment is nulled before writing because the monitor inherits it
  from libcontainer anyway, and persisting host env into the rootfs is
  exposure with no upside.

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
feat(libcontainer): create the monitor container with libcontainer
2bcb019 
Build the monitor's libcontainer config using as a base the contianer's
OCI spec. SPecify the root based on the monitor rootfs, append the
monitor mounts/devices and create+start the init that "urunc monitor"
becomes.

- Process argv is "/proc/self/exe monitor <id>": /proc is always mounted
  post-pivot, so the urunc binary re-execs without being copied
  into the rootfs.
- Seccomp stripped, since urunc applies monitor-specific filters
- Allow-all cgroup device rule, no resource limits, just placement
  for the time being.
- Caps = container's set + CAP_NET_ADMIN (tap setup) in all five sets.
- Drop Poststart/Poststop from the config, keep them on urunc's execution.
- libcontainer state nested under <root>/libcontainer to avoid the
  state.json collision.
- On failure after Create/Start, destroy and reap the init before the
  cgroup teardown, matching runc's ordering.

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
feat(libcontainer): Create a FIFO for urunc IPC in libcontainer
59ed961 
The "urunc monitor init" process can't reach the urunc-mode socket, so
signal readiness over a FIFO in the state dir. Create wires the write
end to the monitor as its single ExtraFile (fd 3 / ReadyPipeFD). Start
reads the message.

- Create opens O_RDWR so a writer is always present: start blocks
  instead of seeing a premature EOF, yet still gets EOF if the monitor
  dies without writing.
- Read side opens O_RDONLY|O_NONBLOCK (open never waits on a writer);
  the read itself blocks via the runtime poller.
- One byte: message readyOK(0)=success; any other byte, read error, or
  EOF=fail.

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
feat(libcontainer): branch start between monitor init and reexec
411c8af 
Split startContainer: in libcontainer mode Load the container and Exec()
it (opens the exec fifo, releasing the init blocked since create);
otherwise keep the reexec socket handshake. Both paths then wait on
AwaitMsg, FIFO- or socket-backed by mode

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
feat(libcontainer): add the internal monitor subcommand
90bcbf2 
This is the command that libcontainer's init will execve to inside the
monitor's container.  Then ExecMonitor loads the spec, sets up the tap, drops
to the container user, signals ready, and execve's the monitor (no
return on success).

- Skip root/XDG_RUNTIME_DIR resolution for this invocation: it never
  touches urunc's state dir, and the inherited XDG_RUNTIME_DIR names a
  path absent inside the rootfs, so preparing it would fail the monitor.
- Container ID is only for log/ps labelling; everything real comes from
  the spec file. Env is the one libcontainer set (spec carries none).
- signalReady(true) fires only after the exec command is built and
  closes the pipe so it isn't left open across the execve; failures
  signal false.

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
feat(libcontainer): tear down the monitor container on delete
a708242 
In libcontainer mode Delete() destroys the monitor's libcontainer state
+ cgroup (Load then Destroy, runc-style) and removes the
.monitor_spec.json left in the container rootfs.

- destroyLibcontainer split cgo/nocgo: the real implementation imports runc's
  libcontainer (cgo-only); the shim is built without cgo and never takes
  this path, so a no-op stub keeps Delete linking.
- Failed/absent Load = "nothing to destroy" (create that failed before
  libcontainer.Create, or an already-completed teardown).
- Missing spec file on removal is ignored (os.ErrNotExist).

Signed-off-by: Charalampos Mainas <[email protected]>
@netlify

netlify Bot commented Aug 17, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ Deploy Preview for urunc ready!

Name Link
🔨 Latest commit 3d72bea
🔍 Latest deploy log https://app.netlify.com/projects/urunc/deploys/6a85a5cb86e8bb0008ebc9f6
😎 Deploy Preview https://deploy-preview-982--urunc.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@cmainas
cmainas force-pushed the feat/libcontainer branch from d8a1d6f to a88174b Compare August 18, 2026 09:26
cmainas added 3 commits August 18, 2026 10:57
@cmainas
refactor: move all monitor resources in InitialSetup
cab0484 
Gather everything related to the guest rootfs (along with the preStart)
command in once during InitialSetup and store them in monitorResources.
Exec simply loads the struct and uses them. No rootfs decision is made
in Exec.

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
refactor: Break Exec and create in smaller reusable functions
27fc414 
Break the large Exec function and the create function into smaller
functions that cna be reused:
- split createUnikontainer into newUnikontainer (bundle parse and
  InitialSetup) and the reexec handshake
- make SetupNet a standalone function
- pull buildUnikernelCommand and execMonitor out of Exec

The rationale is to let the later port of libcontainer to use some of
this functionality directly instead of duplicating logic.

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
feat(libcontainer): Add configuration option for libcontainer
5e3a17d 
Add a configuration option in urunc's configuration to let users choose
between libcontianer's and urunc's own implementation for the setup of
the monitor's execution environment.

THe option is under the [runtime] section of the configuration which
should hold generic runtime options. For the time being it is off by
default.

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
cmainas force-pushed the feat/libcontainer branch 3 times, most recently from 0f7b035 to 4a09bfd Compare August 18, 2026 17:38
@cmainas
feat(libcontainer): Add tests for libcontainer in CI
3d72bea 
Since we currently have 2 versions of urunc, we need to test both urunc
native way and the libcontainer one for the setup of the monitor's
execution environment setup To do that, repeat the vm_tests and kind
test by simply editing the urunc configuration enabling libcontainer.

We might want to check how to improve this in the future.

Signed-off-by: Charalampos Mainas <[email protected]>
@cmainas
cmainas force-pushed the feat/libcontainer branch from 4a09bfd to 9627bfe Compare August 19, 2026 10:40
@cmainas

cmainas commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Some notes / TODOs to have them in our mind, also edited in the PR description:

  • vAccel is out of scope of this PR
  • Currently cgroups are just for placement, because we might get OOM if we set up the limits we get. We need to add the sandbox overhead.
  • For virtiofsd the --sandbox cli option must be set to off, because inside the monitor container the process that spawns virtiofsd does not have the necessary capabilities to create a namespace. Will be addressed in a follow up PR.
  • Future TODOs have been added in the code.
  • There is a split of libcontianer for CGO and non-CGO code. This is necessary because the shim is not built with CGO and due to its dependency in unikontainers, it fails to link. We need to address this in the future too.

@cmainas

cmainas commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Some more notes:

  • The metrics have been messed up and we need to come up with a new way to use them, because the "urunc monitor" process does not have access to the metrics file.
  • The logs in the "urunc monitor" process are shown in stdout and not redirected in the log file/socket. We should resolve this in future iterations.

@cmainas
cmainas force-pushed the feat/libcontainer branch 2 times, most recently from 3dddca5 to abe8288 Compare August 19, 2026 12:14
@cmainas

cmainas commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Extra notes:

  • As reported by @ananos for the libcontainer case the urunc binary must be statically linked. Otherwise the libcontainer trick of /proc/self/exe will fail due to missing libraries in the case of Firecracker (note, we do not add any host libraries for Firecracker in its execution environment).
  • Static networking mode does not work with libcontainers. The "urunc monitor" misses the iptables binary inside its execution environment.

@cmainas

cmainas commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Let me add some more TODOs:

  • We need to remove /dev/net/tun if the container does not have network
  • Rootless containers

@cmainas
cmainas force-pushed the feat/libcontainer branch from abe8288 to 3d72bea Compare August 19, 2026 12:47
@cmainas
cmainas requested a review from ananos August 19, 2026 13:17
@cmainas
cmainas marked this pull request as ready for review August 19, 2026 16:08
@cmainas cmainas mentioned this pull request Aug 22, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ananos ananos Awaiting requested review from ananos

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Migrate from manual namespaces creation/joining and mount operations to libcontianer

1 participant

@cmainas