Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 36 additions & 18 deletions docs/backends/nvx/nvx.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,10 +114,10 @@ characters are rejected. The signed runtime validates the remaining reference
syntax and pulls the selected image on the host. Host-side image
acquisition does not grant the guest network access. MXC has no separate
registry-credentials input. The runtime registers each reference once and keeps
it by its text in the [agent state directory](#agent-state-directory), so later
requests boot it without contacting the registry, and a moving tag such as
`latest` keeps naming the content first pulled. Use a version tag or a digest
(`repository@sha256:…`) to control what a sandbox boots.
it by its text in the [agent state directory](#agent-state-directory). While
it remains registered, later requests boot it without contacting the registry,
and a moving tag such as `latest` keeps naming the content first pulled. Use a
version tag or a digest (`repository@sha256:…`) to control what a sandbox boots.

## Supported operations and policy

Expand Down Expand Up @@ -272,20 +272,38 @@ runtime keeps its own state. MXC obtains that digest from the pinned
MXC makes the current user the directory's owner
and gives it an access list for only that user and Local SYSTEM, so the user's
elevated and non-elevated requests can share it. Every create-and-run request
reuses it, so an image is pulled once rather than for every request, and
concurrent requests share it. MXC coordinates the agent's stop and deprovision
operations and closes the client after each request; it does not maintain a
persistent container database.

To pull a reference again or reclaim disk space, delete the directory while no
MicroVM request or platform-owning SDK process is using it; the next request
creates it again and captures new templates. Lifecycle operations retain their
separate, unversioned root, `%LOCALAPPDATA%\Microsoft\MXC\nvx`, so existing
container IDs and records do not move. Removing a bundle-specific directory does
not remove lifecycle containers, registered images, or templates in that root.
An upgrade that changes the runtime file digests cannot start containers
provisioned with the previous files. Deprovision those containers and provision
them again with the selected bundle; do not move or edit their records.
reuses it, and concurrent requests share registered images. MXC coordinates the
agent's stop and deprovision operations and closes the client after each request;
it does not maintain a persistent container database.

Lifecycle operations retain their separate, unversioned root,
`%LOCALAPPDATA%\Microsoft\MXC\nvx`, so existing container IDs and records do not
move. Each root has a 10 GiB (10,737,418,240 bytes) limit on the combined size of
counted staged images. The limit is not a total disk-usage limit: temporary
conversion files, templates, other state, and older bundle-specific roots are
outside it. Other agent hosts sharing a root can use different limits.

When a new image needs space, the agent evicts the largest eligible unused
images first. Images used by provisioned, running, or stopped sandboxes are
retained until the sandboxes are deprovisioned. The agent also retains images
protected as host defaults or prefetched paths and images registered by a
caller-owned path. A newly admitted image larger than the limit, or one that
cannot fit without evicting protected or in-use images, fails registration. If
an eviction fails, earlier successful evictions can remain in effect.

The limit applies during admission; it does not immediately trim an existing
cache above 10 GiB, and reusing a registered image does not trigger a trim. An
evicted reference must be pulled again and cannot be used while its registry is
unreachable. There is no age-based refresh: an intact registered reference can
continue to use its original content indefinitely.

Removing a bundle-specific directory while no MicroVM request or platform-owning
SDK process is using it removes that root's cache and templates; the next request
creates it again. It does not remove lifecycle containers, registered images, or
templates in the separate lifecycle root. An upgrade that changes the runtime
file digests cannot start containers provisioned with the previous files.
Deprovision those containers and provision them again with the selected bundle;
do not move or edit their records.

If deprovisioning fails, MXC reports the failure and the directory's location.
The agent's records for that sandbox remain there rather than being deleted
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,21 @@ capture followed by reuse without recapture in later processes. Concurrent first
requests exercise both template capture and image registration; later concurrent
requests must still clone the same template with the registry unreachable.

The image-cache capacity scenario uses the signed image tool to create a
large protected image and verifies admission across the actual 10 GiB boundary
in both agent state roots. It checks root independence, concurrent create-and-run
requests, eviction and re-pull, reuse of an existing over-limit cache, dry-run,
subsequent admission, and protection of provisioned, running, and stopped
containers. It also checks oversized-image rejection through the signed agent's
local-image registration. The test needs registry access and at least 30 GiB of
free disk space available to the test process on the volume holding the staged
test executable. It checks that volume before creating its fixtures and fails
with the available and required byte counts when space is insufficient. If the
volume cannot be queried, it reports the directory and error. The check does not
reserve space; concurrent use and temporary files on other volumes can still
cause the test to fail. The limit covers counted staged images, not temporary
files or other agent state.

Explicit 256-MiB and 512-MiB requests exercise create-and-run execution and
lifecycle starts. Native diagnostics must report a clone for 256 MiB and a cold
boot with a memory-capacity reason for 512 MiB, without an initialization-fallback
Expand Down Expand Up @@ -404,7 +419,8 @@ and unpack the Rust crate package to check that its tests compile. Actual VM
scenarios run only in the scheduled Windows x64 job. Its dispatcher requires
completion markers for CLI and SDK snapshots, lifecycle adoption, fallback
warnings and lifecycle diagnostics, concurrent image reuse, and all three
interruption phases, as well as the existing execution and policy scenarios.
interruption phases, the image-cache capacity scenario, and the existing
execution and policy scenarios.
Missing markers, empty suites, skips, ignored tests, and filtered tests fail
the job.

Expand Down
1 change: 1 addition & 0 deletions scripts/ci/run_backend_validation_tests.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,7 @@ function Invoke-MicroVmTests {
'JSON image and policy' = 'VM_EXECUTED JSON selected-image filesystem read/write/deny and cleanup confirmed clones'
'executor' = 'VM_EXECUTED executor sequential '
'executor image reuse' = 'VM_EXECUTED executor registered-image reuse offline concurrent first-capture clone isolation'
'image cache capacity' = 'VM_EXECUTED image cache 10 GiB eviction repull and lifecycle protection'
'executor configs' = 'VM_EXECUTED executor configs '
'executor interruption' = 'VM_EXECUTED executor console interruption capture/startup/execution/owned-vmm/cleanup'
'network rules' = 'VM_EXECUTED reachable controls and network allow/deny/nonmatching-port enforcement confirmed clones'
Expand Down
2 changes: 2 additions & 0 deletions scripts/ci/stage-nvx-validation-tests.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ foreach ($name in @(
'windows_x64::signed_bundle_executor_configs',
'windows_x64::signed_bundle_executor_create_and_run',
'windows_x64::signed_bundle_executor_reuses_registered_images',
'windows_x64::image_cache_fixture_space_check_fails_closed',
'windows_x64::signed_bundle_image_cache_capacity',
'windows_x64::signed_bundle_executor_starts_sandboxes_from_platform_snapshots',
'windows_x64::signed_bundle_snapshot_memory_capacity',
'windows_x64::signed_bundle_executor_console_interrupt',
Expand Down
15 changes: 15 additions & 0 deletions scripts/tests/microvm-validation.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,12 @@ fn main() {
if mode != "missing-image-reuse-discovery" {
println!("windows_x64::signed_bundle_executor_reuses_registered_images: test");
}
if mode != "missing-image-space-check-discovery" {
println!("windows_x64::image_cache_fixture_space_check_fails_closed: test");
}
if mode != "missing-image-capacity-discovery" {
println!("windows_x64::signed_bundle_image_cache_capacity: test");
}
if mode != "missing-platform-snapshots-discovery" {
println!("windows_x64::signed_bundle_executor_starts_sandboxes_from_platform_snapshots: test");
}
Expand Down Expand Up @@ -293,6 +299,9 @@ fn main() {
} else {
println!("VM_EXECUTED executor registered-image reuse offline and concurrent");
}
if mode != "missing-image-capacity" {
println!("VM_EXECUTED image cache 10 GiB eviction repull and lifecycle protection");
}
if mode != "missing-network" {
let suffix = if mode == "unconfirmed-network" { "" } else { " confirmed clones" };
println!("VM_EXECUTED reachable controls and network allow/deny/nonmatching-port enforcement{suffix}");
Expand Down Expand Up @@ -696,6 +705,7 @@ test("artifact dispatcher executes prebuilt SDK tests and preserves the log", ()
assert.match(log, /VM_EXECUTED executor /);
assert.match(log, /VM_EXECUTED executor console interruption capture\/startup\/execution\/owned-vmm\/cleanup/);
assert.match(log, /VM_EXECUTED executor registered-image reuse offline concurrent first-capture clone isolation/);
assert.match(log, /VM_EXECUTED image cache 10 GiB eviction repull and lifecycle protection/);
assert.match(log, /VM_EXECUTED reachable controls and network allow\/deny\/nonmatching-port enforcement confirmed clones/);
assert.match(log, /VM_EXECUTED UDP datagrams replies delivery counts protocol separation and inclusive ranges confirmed clones/);
assert.match(log, /VM_EXECUTED host-loopback ingress TCP\/UDP allow deny ready listeners and reply controls confirmed clones/);
Expand All @@ -719,6 +729,7 @@ for (const [mode, message] of [
["missing-configs", /did not execute the executor configs VM scenario/],
["missing-interruption", /did not execute the executor interruption VM scenario/],
["missing-image-reuse", /did not execute the executor image reuse VM scenario/],
["missing-image-capacity", /did not execute the image cache capacity VM scenario/],
["missing-network", /did not execute the network rules VM scenario/],
["missing-network-udp", /did not execute the UDP network rules VM scenario/],
["missing-host-loopback", /did not execute the host-loopback ingress VM scenario/],
Expand Down Expand Up @@ -888,6 +899,8 @@ for (const [mode, message] of [
["missing-sdk-lifecycle-discovery", /did not discover windows_x64::signed_bundle_sdk_json_lifecycle/],
["missing-interruption-discovery", /did not discover windows_x64::signed_bundle_executor_console_interrupt/],
["missing-image-reuse-discovery", /did not discover windows_x64::signed_bundle_executor_reuses_registered_images/],
["missing-image-space-check-discovery", /did not discover windows_x64::image_cache_fixture_space_check_fails_closed/],
["missing-image-capacity-discovery", /did not discover windows_x64::signed_bundle_image_cache_capacity/],
["missing-platform-snapshots-discovery", /did not discover[\s\S]*windows_x64::signed_bundle_executor_starts_sandboxes_from_platform_snapshots/],
["missing-snapshot-memory-discovery", /did not discover[\s\S]*windows_x64::signed_bundle_snapshot_memory_capacity/],
["missing-network-discovery", /did not discover[\s\S]*windows_network::signed_bundle_network_rules_enforce_allow_deny_and_port_selectors/],
Expand All @@ -911,6 +924,8 @@ for (const [mode, message] of [
"missing-sdk-lifecycle-discovery",
"missing-interruption-discovery",
"missing-image-reuse-discovery",
"missing-image-space-check-discovery",
"missing-image-capacity-discovery",
"missing-platform-snapshots-discovery",
"missing-snapshot-memory-discovery",
"missing-network-discovery",
Expand Down
14 changes: 13 additions & 1 deletion src/mxc-sdk/src/backends/nvx/crate_runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -181,9 +181,12 @@ pub(super) struct NvxClient(Option<ClientInner>);
// unless a request sets their memory, so the smallest templates fit them.
const PLATFORM_SNAPSHOT_MAX_VCPUS: u32 = 1;
const PLATFORM_SNAPSHOT_MAX_MEMORY_MIB: u32 = 256;
const IMAGE_CACHE_CAPACITY_BYTES: u64 = 10 * 1024 * 1024 * 1024;

fn agent_setup(bundle: &BundledAgent, state_root: &Path) -> SetupConfig {
bundle.setup(state_root).with_cpu_profile(CpuProfile::Host)
let mut setup = bundle.setup(state_root).with_cpu_profile(CpuProfile::Host);
setup.images.max_cache_capacity_bytes = Some(IMAGE_CACHE_CAPACITY_BYTES);
setup
}

static RUNTIME_DIRECTORY: OnceLock<PathBuf> = OnceLock::new();
Expand Down Expand Up @@ -1549,11 +1552,20 @@ mod tests {
let bundle = BundledAgent::from_dir(r"C:\bundle").unwrap();
let setup = agent_setup(&bundle, Path::new(r"C:\state"));
assert_eq!(setup.cpu_profile, CpuProfile::Host);
assert_eq!(setup.images.max_cache_capacity_bytes, Some(10_737_418_240));
let mut other_image_settings = setup.images.clone();
other_image_settings.max_cache_capacity_bytes = None;
assert_eq!(
other_image_settings,
aci_edge_sandboxes::agent::ImageSettings::default()
);
let json = serde_json::to_value(&setup).unwrap();
assert_eq!(json["cpuProfile"], "host");
assert_eq!(json["images"]["maxCacheCapacityBytes"], 10_737_418_240_u64);
assert!(json.get("platformSnapshots").is_none(), "{json}");

let setup = setup.with_platform_snapshots(platform_snapshots());
assert_eq!(setup.images.max_cache_capacity_bytes, Some(10_737_418_240));
assert_eq!(
serde_json::to_value(&setup).unwrap()["platformSnapshots"],
serde_json::json!({"enabled": true, "maxVcpus": 1, "maxMemoryMib": 256})
Expand Down
Loading
Loading