Summary
On the infra node, dnsmasq uses itself as an upstream server. node_dns_method: add writes admin_ip at the top of /etc/resolv.conf, and the generated dnsmasq.conf has no no-resolv, so dnsmasq reads that file and forwards to its own address. On Ubuntu, systemd-resolved picks up the same file and forwards back to dnsmasq, which closes a second loop.
A single query gets through. A burst of queries does not: every apt update on the node leaves dnsmasq unable to answer for about a minute, internal *.pigsty names included.
Environment
|
|
| Pigsty |
v4.5.0 (template unchanged on main) |
| OS |
Ubuntu 24.04 LTS |
| dnsmasq |
2.91 |
| systemd |
255 (systemd-resolved active) |
| Node DNS |
defaults: node_dns_method: add, node_dns_servers: ['${admin_ip}'] |
Root cause
roles/node/tasks/dns.yml renders, on the infra node itself:
options single-request-reopen timeout:1
nameserver 10.0.0.1 # admin_ip, i.e. this node's dnsmasq
nameserver 127.0.0.53 # systemd-resolved stub
roles/infra/templates/dns/dnsmasq.conf keeps no-resolv and server= commented out, so dnsmasq takes its upstreams from that file:
dnsmasq[846]: reading /etc/resolv.conf
dnsmasq[846]: using nameserver 10.0.0.1#53
dnsmasq[846]: using nameserver 127.0.0.53#53
dnsmasq normally ignores a nameserver that matches a local address. Here it does not, because the listen address sits on an interface that comes up after dnsmasq starts (the bind-dynamic case from #761).
Because /etc/resolv.conf is now a regular file and no longer the stub symlink, systemd-resolved reads it as foreign configuration and uses 10.0.0.1 as its global DNS server (resolvectl dns shows Global: 10.0.0.1). The result is two loops: dnsmasq to dnsmasq, and dnsmasq to resolved to dnsmasq.
Symptoms
Tracing one uncached query on port 53 shows it bouncing between 127.0.0.1, the node address and the stub before it leaves the host. It still resolves in about 13 ms, which is why nothing looks broken.
Under a burst it breaks. Every morning at the apt-daily run and at any other apt update, systemd-resolved logs:
systemd-resolved: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 10.0.0.1.
A probe querying dnsmasq directly (dig @127.0.0.1) gets no answer for about a minute, for internal names as well as external ones. It recovers on its own. dnsmasq never restarts, so no failed unit shows up.
Workaround we run
The template already declares conf-dir=/etc/dnsmasq.d, and no Pigsty role cleans that directory, so we add a file there instead of patching the template:
# /etc/dnsmasq.d/10-upstream.conf
no-resolv
server=<upstream 1>
server=<upstream 2>
server= alone is not enough. The dnsmasq man page says it "does not suppress reading of /etc/resolv.conf": that takes no-resolv. Pointing resolv-file= at /run/systemd/resolve/resolv.conf does not help either, since resolved lists the node address there too.
After a restart, dnsmasq logs only the two upstreams. Internal names answer on both listen addresses. apt-get update with a parallel dig loop loses no answer. A burst of 300 uncached names, 50 at a time, gets 300 answers, and an internal name still resolves in the middle of it. systemd-resolved no longer logs "degraded feature set".
Suggestion
An infra variable for dnsmasq upstreams would fix this without touching /etc/resolv.conf. When set, the template would render no-resolv and one server= per entry. For example, dns_upstreams: [] would keep today's behaviour, and a list such as dns_upstreams: [1.1.1.1, 8.8.8.8] would switch to explicit forwarding.
Another option: skip admin_ip in node_dns_servers when the node is itself an infra node. That removes the self reference but keeps the resolved loop, so the variable looks like the more complete fix.
Happy to test a patch on this setup.
Summary
On the infra node, dnsmasq uses itself as an upstream server.
node_dns_method: addwritesadmin_ipat the top of/etc/resolv.conf, and the generateddnsmasq.confhas nono-resolv, so dnsmasq reads that file and forwards to its own address. On Ubuntu, systemd-resolved picks up the same file and forwards back to dnsmasq, which closes a second loop.A single query gets through. A burst of queries does not: every
apt updateon the node leaves dnsmasq unable to answer for about a minute, internal*.pigstynames included.Environment
main)node_dns_method: add,node_dns_servers: ['${admin_ip}']Root cause
roles/node/tasks/dns.ymlrenders, on the infra node itself:roles/infra/templates/dns/dnsmasq.confkeepsno-resolvandserver=commented out, so dnsmasq takes its upstreams from that file:dnsmasq normally ignores a nameserver that matches a local address. Here it does not, because the listen address sits on an interface that comes up after dnsmasq starts (the
bind-dynamiccase from #761).Because
/etc/resolv.confis now a regular file and no longer the stub symlink, systemd-resolved reads it as foreign configuration and uses10.0.0.1as its global DNS server (resolvectl dnsshowsGlobal: 10.0.0.1). The result is two loops: dnsmasq to dnsmasq, and dnsmasq to resolved to dnsmasq.Symptoms
Tracing one uncached query on port 53 shows it bouncing between 127.0.0.1, the node address and the stub before it leaves the host. It still resolves in about 13 ms, which is why nothing looks broken.
Under a burst it breaks. Every morning at the
apt-dailyrun and at any otherapt update, systemd-resolved logs:A probe querying dnsmasq directly (
dig @127.0.0.1) gets no answer for about a minute, for internal names as well as external ones. It recovers on its own. dnsmasq never restarts, so no failed unit shows up.Workaround we run
The template already declares
conf-dir=/etc/dnsmasq.d, and no Pigsty role cleans that directory, so we add a file there instead of patching the template:server=alone is not enough. The dnsmasq man page says it "does not suppress reading of /etc/resolv.conf": that takesno-resolv. Pointingresolv-file=at/run/systemd/resolve/resolv.confdoes not help either, since resolved lists the node address there too.After a restart, dnsmasq logs only the two upstreams. Internal names answer on both listen addresses.
apt-get updatewith a paralleldigloop loses no answer. A burst of 300 uncached names, 50 at a time, gets 300 answers, and an internal name still resolves in the middle of it. systemd-resolved no longer logs "degraded feature set".Suggestion
An infra variable for dnsmasq upstreams would fix this without touching
/etc/resolv.conf. When set, the template would renderno-resolvand oneserver=per entry. For example,dns_upstreams: []would keep today's behaviour, and a list such asdns_upstreams: [1.1.1.1, 8.8.8.8]would switch to explicit forwarding.Another option: skip
admin_ipinnode_dns_serverswhen the node is itself an infra node. That removes the self reference but keeps the resolved loop, so the variable looks like the more complete fix.Happy to test a patch on this setup.