Run a local DERP relay on the NixOS box - #191
Draft
samestep wants to merge 19 commits into
Draft
Conversation
Every VM is on the tailnet but behind per-VM NAT, and both physical machines are behind T-Mobile 5G CGNAT, so Tailscale can never connect them directly and relays all VM-to-VM traffic through a cloud DERP — up over the slow 5G upload and back, at ~1 MB/s. Run derper on the always-on NixOS machine so the relay stays on the LAN: the VMs prefer it at home (lowest latency) and fall back to the cloud DERPs when away. The host does not join the tailnet; derper only forwards already-encrypted WireGuard packets, so it can't read the traffic or reach the VMs. Self-signed and pinned by SHA256 in the DERP map, so no domain or ACME cert is needed. Also disable Wi-Fi power saving, which added ~60-110 ms to every LAN round-trip — enough that Tailscale rated the local relay no faster than the cloud one and refused to use it; off, the hop is ~5 ms and the local relay wins. The relay's IP is pinned so its self-signed cert doesn't drift. The gateway (Arcadyan TMO-G4AR) exposes no DHCP settings, so rather than a reservation the box holds a static 192.168.12.10 on its Wi-Fi connection, below the gateway's pool. README documents that plus the rest: reading the pin derper logs, adding the DERP-map region to the tailnet policy, and verifying. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_013Wku9K54eBg5KJnXDfpvkQ
derper stores its own private key in the file named by -c, and defaults that path (/var/lib/derper/derper.key) only when it runs as root. Under DynamicUser it's unprivileged, so without -c it exits at once with "-c <config path> not specified" and systemd's restart limiter latches. Point -c at the state directory so it creates and reuses the key there. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_013Wku9K54eBg5KJnXDfpvkQ
This was referenced Jul 20, 2026
Measured from both Linux VMs: the relay works, but they mostly aren't using it. `derper`'s LAN-side STUN lets them discover LAN-local mapped endpoints, the per-VM NATs hole-punch, and they hold a direct path (192.168.12.130:39424 <-> 192.168.12.10:41641). So step 4's "should report via DERP(home)" read a success as a failure. Also: the NixOS box's LAN hop is ~1 ms, not ~5 ms (p50 1.0, p90 1.5), while the macOS host shows the same Wi-Fi power-saving tail this config turns off on NixOS (p50 ~4 ms, p90 33 ms, p99 74 ms) with no equivalent knob available. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01GNWf7bQzqqVN7GUSEtbDMG
The static 192.168.12.10 lived only on the Wi-Fi connection profile, set once by hand with `nmcli con mod`. Moving the box to Ethernet therefore dropped it, and DHCP handed out 192.168.12.184 instead. That is enough to take the whole tailnet down between the VMs, not just slow it: the DERP map pins the relay at .10, so `netcheck` falls back to a cloud DERP, LAN STUN goes with it, and without LAN-side endpoints to hole-punch with the VMs lose every path to each other. Declare the address instead, as a NetworkManager profile bound to whatever Ethernet device is present. On Ethernet there is no PSK to keep out of the repo, which is what forced the imperative step before. Wi-Fi keeps its own profile and stays a DHCP fallback. `ensureProfiles` only reloads NetworkManager, which will not drop an active connection to adopt a new profile, so bring the profile up in `postStart` — otherwise a switch would not take effect until reboot. Assisted-by: Claude:opus-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.