nslookup failed but systemd-resolved works

Question

This is a strange problem I've spent all day on. It would be great if someone could shed some light on this.

The problem exhibits as a problem with name resolution, but I'm not sure that's the root cause:

# host www.google.com
;; connection timed out; no servers could be reached

so far so boring, but wait!:

#systemd-resolve www.google.com
www.google.com: 209.85.202.103
                209.85.202.106
                209.85.202.105
                209.85.202.104
                209.85.202.147
                209.85.202.99

Simple, the problem is with resolv.conf right?

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
nameserver 127.0.0.53

search xxx.uk xyz

So the system is using the systemd resolver?

#dig @127.0.0.53 www.google.com
; <<>> DiG 9.10.3-P4-Ubuntu <<>> @127.0.0.53 www.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

ok, so if the systemd resolver says it's on 127.0.0.53 why is it not responding.

#sudo netstat -lupn | grep 127
udp        0      0 127.0.0.53:53           0.0.0.0:*                           1679/systemd-resolv

If it's not listening, what is systemd-resolv doing?

Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 20 (veth10858e2)
      Current Scopes: LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 14 (vnet0)
      Current Scopes: LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 13 (virbr0-nic)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 12 (virbr0)
      Current Scopes: LLMNR/IPv4
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 11 (docker0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 10 (docker_gwbridge)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 3 (em2)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.100.1
                      192.168.100.2
                      192.168.100.3
                      192.168.100.4
          DNS Domain: cqp

Link 2 (em1)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: xxx.xxx.x.xx
                      xxx.xxx.x.xx
          DNS Domain: xxx.uk

I have several severs, they're all running up to date ubuntu artful, this problem has moved between servers in the process of trying to fix this.
They are part of a docker swarm and removing, then re-adding fix the problem at one point.

I turned on systemd-resolved debugging but there was no output when a request was made.

You might be right, but I can't see a rule which would affect it - what I don't understand is how is the system deciding what to ask, I thought that it read resolv.conf for where to send the request — Richard Collins, Apr 27 '18 at 05:36
nslookup is not "the system". systemd-resolvectl is not "the system". They both use a more direct approach than "the system" does, and they both use different APIs (systemd-resolvectl uses D-Bus, not UDP.) — u1686_grawity, Apr 27 '18 at 05:41
I know that it ultimately boils down to gethostbyname(), which is what is failing, how does that decide how to resolve the address if it's not using 127.0.0.53 as resolve.conf says? — Richard Collins, Apr 27 '18 at 05:58
My point was that **it doesn't** boil down to gethostbyname(). The `nslookup` tool still uses resolv.conf but actually speaks DNS directly. And the `systemd-resolve` tool does not speak DNS at all, it speaks D-Bus. — u1686_grawity, Apr 27 '18 at 06:00

score 10 · Answer 1 · answered May 14 '18 at 20:06

10

I found this answer on Hacker News which suggested symlinking /etc/resolv.conf to /run/systemd/resolve/resolv.conf:

sudo rm /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

That worked for me. I don't think I had to restart any services after recreating the symlink.

answered May 14 '18 at 20:06

Neil Smith

201
1
2

Thanks but this is already the case. – Richard Collins May 16 '18 at 13:19
@Richard Collins: So this is not the answer? – U. Windl Apr 29 '19 at 09:28
1

No, there might be some link with an issue with docker and firewalld - there's no hard evidence for that though – Richard Collins Apr 30 '19 at 11:00
This worked for me! – user643011 Feb 02 '23 at 14:38

score 1 · Answer 2 · edited Jul 05 '20 at 14:41

1

It sounds crazy, but you need to allow access to 127.0.0.53 to use systemd-resolved

iptables -I INPUT -s localhost -d 127.0.0.53  -j ACCEPT

edited Jul 05 '20 at 14:41

Machavity

643
2
7
19

answered Jul 02 '20 at 17:10

David Ordyan

11
1

score 0 · Answer 3 · answered Mar 25 '23 at 18:26

Check whether resolv.conf file is available in the file path: /run/systemd/resolve/resolv.conf

It is Symbolic linked with /etc/resolv.conf

If the file is not available then create it in this file location /run/systemd/resolve/resolv.conf and delete the file in /etc/resolv.conf

Now relink the file by cmd sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Now try to ping, it will fix the issue most

nslookup failed but systemd-resolved works

3 Answers3

Linked