default DNS server doesn't work

[as3ii]

I have problems with DNS resolution inside the sandbox while trying to use rootless docker and containerd+nerdctl.
I've tried to run a shell inside the sandbox to debug the problem and I found that the default DNS resolver doesn't works (10.0.2.3, reported inside /etc/resolv.conf), but its IP respond to ping.

• host: debian 11 (arm64) - raspberry pi 3
• Command used to do the tests: rootlesskit --state-dir=/run/user/1000/tmp --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --copy-up=/var/lib --propagation=rslave bash
• rootlesskit version: 0.14.6
• slirp4netns version: 1.1.12 (libslirp: 4.6.1)

[AkihiroSuda]

• If the default CIDR overlaps with your network, you might have to run rootlesskit with an alternative CIDR like --cidr=192.168.123.0/24
• Does --net=vpnkit work?

[as3ii]

the default CIDR don't overlaps with my local network, and at the moment I cannot test vpnkit because it fail to build

[olivermg]

I think I'm probably facing the same issue.

Observations

I was trying to go through the basic slirp4netns usage example and found out that the builtin DNS server does not work for me:

(host)$ unshare --user --map-root-user --net --mount bash
(ns)# echo $$
248202

Then, on the host, slirp4netns starts up fine:

(host)$ slirp4netns --configure --mtu=65520 --disable-host-loopback 248202 tap0
sent tapfd=5 for tap0
received tapfd=5
Starting slirp
* MTU:             65520
* Network:         10.0.2.0
* Netmask:         255.255.255.0
* Gateway:         10.0.2.2
* DNS:             10.0.2.3
* DHCP begin:      10.0.2.15
* DHCP end:        10.0.2.30
* Recommended IP:  10.0.2.100

Back in the namespace, both tap0 interface and routing table look fine to me:

(ns)# ip a show dev tap0
2: tap0: <BROADCAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether [redacted] brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.100/24 brd 10.0.2.255 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::[redacted]/64 scope link 
       valid_lft forever preferred_lft forever


(ns)# ip r
default via 10.0.2.2 dev tap0 
10.0.2.0/24 dev tap0 proto kernel scope link src 10.0.2.100

Then I bind mount an appropriate resolv.conf file:

(ns)# mount --bind /tmp/my-resolv.conf /etc/resolv.conf

(ns)# cat /etc/resolv.conf
nameserver 10.0.2.3

but resolving DNS won't work:

(ns)# dig amazon.com

; <<>> DiG 9.18.5 <<>> amazon.com
;; global options: +cmd
;; connection timed out; no servers could be reached

However, it's not a networking/ip/routing issue, as DNS resolving e.g. via Google nameserver works:

(ns)# dig @8.8.8.8 amazon.com

; <<>> DiG 9.18.5 <<>> @8.8.8.8 amazon.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64561
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;amazon.com.			IN	A

;; ANSWER SECTION:
amazon.com.		42	IN	A	205.251.242.103
amazon.com.		42	IN	A	176.32.103.205
amazon.com.		42	IN	A	54.239.28.85

;; Query time: 20 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Sun Aug 21 14:30:22 CEST 2022
;; MSG SIZE  rcvd: 87

When I start slirp4netns with --enable-ipv6 and use my internet provider's DNS servers (which have IPv6 addresses, see below), these also work fine.

Furthermore, 10.0.2.3 is pingable:

(ns)# ping 10.0.2.3 -c 3
PING 10.0.2.3 (10.0.2.3) 56(84) bytes of data.
64 bytes from 10.0.2.3: icmp_seq=1 ttl=255 time=0.142 ms
64 bytes from 10.0.2.3: icmp_seq=2 ttl=255 time=0.097 ms
64 bytes from 10.0.2.3: icmp_seq=3 ttl=255 time=0.123 ms

--- 10.0.2.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2018ms
rtt min/avg/max/mdev = 0.097/0.120/0.142/0.018 ms

lsns on the host imho doesn't show anything out of the ordinary:

(host)$ lsns -p 248202
        NS TYPE   NPROCS    PID USER   COMMAND
4026531834 time      140   2285 oliver /usr/lib/systemd/systemd --user
4026531835 cgroup    140   2285 oliver /usr/lib/systemd/systemd --user
4026531836 pid       106   2285 oliver /usr/lib/systemd/systemd --user
4026531838 uts       140   2285 oliver /usr/lib/systemd/systemd --user
4026531839 ipc       140   2285 oliver /usr/lib/systemd/systemd --user
4026533032 user        1 248202 oliver bash
4026533034 mnt         1 248202 oliver bash
4026533035 net         1 248202 oliver bash

Debugging

Among the things I've done to find out the cause of the issue are e.g. the following, see below. Currently, I'm somewhat out of ideas for how to proceed any further for finding out what's going wrong.

Is there any way to have slirp4netns print more debugging info regarding DNS setup? Is there any documentation of how slirp4netns implements the DNS server it provides / how the setup works? I'd be happy if someone pointed me to the right docs/info/whatever.

Change CIDR

I've also tried a different CIDR to rule out that it has anything to do with overlapping CIDRs on my host. Changing the CIDR to 172.28.28.0/24 changed nothing.

iptables

I've also explicitly injected iptables rules both on the host & namespace to make sure any traffic is allowed:

(host)$ sudo iptables -I INPUT -j ACCEPT
(host)$ sudo iptables -I FORWARD -j ACCEPT

(ns)# iptables -I OUTPUT -j ACCEPT

To no avail.

MTU

I've tried to leave the MTU at the default of 1500, but that doesn't change anything either.

Environment

I'm running the following software. If I'm missing any info, just let me know, I'll be happy to add it.

slirp4netns

$ slirp4netns --version
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4

OS

$ uname -a
Linux manjaro 5.15.60-1-MANJARO #1 SMP PREEMPT Thu Aug 11 13:14:05 UTC 2022 x86_64 GNU/Linux

Network config

I'm using NetworkManager on my host:

(host)$ NetworkManager -V
1.38.4-1

There don't seem to be any network namespaces on my host, as running sudo ip netns list returns nothing at all.

DNS config

I'm not using a local DNS resolver, the /etc/resolv.conf on my host points directly to my internet provider's DNS hosts, however, these are only available via IPv6 as it seems:

(host)$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 2a02:908:2:b::1
nameserver 2a02:908:2:a::1

There are a few instances of dnsmasq listening on my host. I think docker starts up those:

(host)$ sudo netstat -tulpen | grep dns
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      0          31965      1624/dnsmasq        
tcp        0      0 192.168.42.1:53         0.0.0.0:*               LISTEN      0          17919      1575/dnsmasq        
udp        0      0 192.168.122.1:53        0.0.0.0:*                           0          31964      1624/dnsmasq        
udp        0      0 192.168.42.1:53         0.0.0.0:*                           0          17918      1575/dnsmasq        
udp        0      0 0.0.0.0:67              0.0.0.0:*                           0          31961      1624/dnsmasq        
udp        0      0 0.0.0.0:67              0.0.0.0:*                           0          22269      1595/dnsmasq        
udp        0      0 0.0.0.0:67              0.0.0.0:*                           0          17915      1575/dnsmasq        

podman info

(host)$ podman info
host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.3-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.3, commit: ab52a597278b20173440140cd810dc9fa8785c93'
  cpuUtilization:
    idlePercent: 62.99
    systemPercent: 9.9
    userPercent: 27.1
  cpus: 12
  distribution:
    distribution: manjaro
    version: unknown
  eventLogger: journald
  hostname: manjaro
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.60-1-MANJARO
  linkmode: dynamic
  logDriver: journald
  memFree: 4954832896
  memTotal: 33556307968
  networkBackend: cni
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.5-1
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 43057717248
  swapTotal: 44015022080
  uptime: 45h 48m 36.46s (Approximately 1.88 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/oliver/.config/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 0
    stopped: 13
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/oliver/.local/share/containers/storage
  graphRootAllocated: 588591013888
  graphRootUsed: 472194166784
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 6
  runRoot: /run/user/1000/containers
  volumePath: /home/oliver/.local/share/containers/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1659559968
  BuiltTime: Wed Aug  3 22:52:48 2022
  GitCommit: f73d8f8875c2be7cd2049094c29aff90b1150241-dirty
  GoVersion: go1.19
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1

[philiprenich]

Did anyone in this issue ever resolve the problems?

[DrPyser]

Also experiencing the same issue with a rootless docker install on nixos. dns resolution doesn't work from the containers.

[AkihiroSuda]

nixos

Discussed in:

• [NixOS] Handling multi-layer and parent symlinks for /etc/resolv.conf #333