net: let VMs talk to each other, but not to the host

[aljoscha]

Makes VM-to-VM reachability and host isolation contracts rather than accidents of the host's iptables policy.

Today nothing permits forwarding between two VM TAPs and nothing denies it, so whether two VMs can talk is decided by the host's FORWARD policy: it works on a box with no firewall and fails on one where docker or firewalld set FORWARD DROP. Host reachability is the same story in reverse. The guest's default gateway is a host address and there are no INPUT rules anywhere, so a guest can reach the host at every address it owns unless the host's own firewall happens to stop it.

Design and rationale: docs/NETWORK-POLICY-SPEC.md.

The contract

A VM reaches the internet and the other VMs of its own installation. It cannot reach the host.

Each installation owns two chains, entered from position 1 of the built-in chains:

-A INPUT   -j ember-<id>-input
-A FORWARD -j ember-<id>-forward

# ember-<id>-input
-i em<id>-+ -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-i em<id>-+ -j DROP

# ember-<id>-forward, install-wide
-i em<id>-+ -o em<id>-+ -j ACCEPT
-i em<id>-+ -j DROP

# ember-<id>-forward, per VM
-i <tap> -o <wan> -j ACCEPT
-i <wan> -o <tap> -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Position 1 is what makes this a contract instead of a suggestion. Appending cannot work: a pre-existing -A INPUT -s 10.0.0.0/8 -j ACCEPT, a common "trust the LAN" rule that matches the default guest range, would match before the host block. Jumping in at the top is only acceptable because every rule in both chains matches an ember TAP, so anything else falls off the end and resumes in the built-in chain right after the jump.

The established-accept in the input chain is mandatory. Replies to host-initiated connections arrive on INPUT from the TAP, so without it ember ssh, exec and cp break.

One rule carries VM-to-VM in both directions: A to B matches with in=tapA out=tapB, B's reply with in=tapB out=tapA, so no conntrack state is needed. Scoped to the install's TAP prefix on both sides, so cross-install traffic stays out.

Rule order needs no bookkeeping. Each chain holds ACCEPTs plus exactly one terminal DROP, ACCEPTs are inserted at the front and the DROP is appended, so the DROP is always last whatever subset already exists.

Notable pieces

A Rule type behind every iptables call. One definition serves the add, check and delete paths. iptables compares full rule text on -D, so the previous duplicate spelling of each rule was a standing invitation to leak rules. Every invocation also takes the xtables lock, which was missing and could fail two concurrent VM starts.

IPv6 was an open hole. The kernel gives both ends of a TAP a v6 link-local address, so a guest could reach the host over IPv6 while the v4 block held. Every TAP on the development host had such an address. Closed with disable_ipv6 per device before the link comes up, rather than a second parallel set of ip6tables rules.

Running VMs are adopted, not broken. A VM running when the chains first appear has its rules in the built-in FORWARD chain, below the jump, so the terminal DROP would cut it off the instant another VM start creates the chain. Reconcile moves such a VM's rules into the chain and records where they went, adding before deleting so the VM is never ruleless. The masquerade rule is excluded from that move: its shape is identical in both modes, so shifting the full set would delete it right after re-adding it and leave the VM without NAT.

Host-run resolvers no longer break guest DNS. A host running dnsmasq, or a pihole bound to the LAN address, hands out a nameserver the guest can no longer reach. Host addresses join loopback and IPv6 in the unreachable-from-the-guest filter, falling through to the next detection source with a warning about what was dropped.

Legacy compatibility. NetworkInfo.firewall_chain records where a VM's rules live, so a VM started by an older binary is still torn down from the built-in FORWARD chain with its comment tag. Masquerade keeps its shape and its home in the shared nat table, so rules written before and after this change stay mutually deletable. Installs with no instance_id use ember-input / ember-forward and the em-+ wildcard.

Testing

Unit tests cover chain and wildcard derivation, both rule shapes rendered as exact iptables argument vectors, the established-before-DROP ordering, and that the forwarding set excludes masquerade.

Two integration test groups are added, both #[ignore] since they need root and a hypervisor. CI does not run them, so they were run by hand:

./run-integration-tests.sh network_policy   # pass
./run-integration-tests.sh ssh              # pass
./run-integration-tests.sh vm               # pass
./run-integration-tests.sh isolation        # pass
./run-integration-tests.sh dm_thin          # pass

tests/network_policy.rs has a structural test that pins rule placement, not just presence, since a terminal DROP that drifts above the per-VM ACCEPTs leaves every rule technically present and the contract broken. Plus a connectivity test that boots two VMs and checks a sibling is reachable, the host is not at any of its addresses, and egress still works. tests/isolation.rs gains a guard that one install's deinit leaves another install's chains alone.

Decisions

• VM-to-host is not overridable. No escape hatch, by choice. If reaching a host service from a VM turns out to be needed, the shape would be an install-wide ember init --allow-host-access.
• Entering FORWARD at position 1 is accepted, with the consequence that a host admin's mid-chain DROP or REJECT no longer governs ember VM traffic. An admin who does want to restrain ember VMs still can: policy::ensure only checks that its jump exists, not where it sits, so a rule inserted above the jump survives every subsequent VM start.

Two things left open and out of scope, both recorded in the spec: whether masquerade should move into its own chain, and a per-VM network.isolated opt-out.

Consequences

• A guest pinging its default gateway now gets silence. Traffic is dropped rather than rejected, so guest connections to the host hang until timeout instead of failing fast.
• A WAN interface change now breaks a running VM's egress explicitly rather than accidentally. Its egress rule and masquerade both name the old interface, so the terminal DROP stops the traffic that previously leaked out unSNATed and failed upstream anyway.
• A host firewall flush while VMs run degrades the policy until the next VM start. Rules are asserted at start, not continuously.
• Guests can still reach the rest of the host's LAN. Only the host itself becomes unreachable.