DNS error on macOS when using scoped DNS config

[meshulam]

Description

I have a local development setup that relies on a custom DNS server for subdomains of a certain domain. When I run rover subgraph introspect on my GraphQL server that's exposed on this domain, rover returns a DNS error, suggesting it's not using the OS's DNS settings:

error[E028]: error sending request for url (https://core.devo.test/graphql): error trying to connect: dns error: no record found for Query { name: Name("core.devo.test."), query_type: AAAA, query_class: IN }
        Make sure the endpoint is accepting connections and is spelled correctly

I found a similar issue but there weren't enough details for me to be sure it was the same problem.

Steps to reproduce

On MacOS,

Start a DNS server on port 5053 to resolve *.devo.test to localhost:

echo 'address=/.devo.test/127.0.0.1' > dnsmasq.conf
docker run -p 5053:53/udp -v ./dnsmasq.conf:/etc/dnsmasq.conf jpillora/dnsmasq

Configure macOS to use this server for the domain:

sudo mkdir -p /etc/resolver/
echo -e 'nameserver 127.0.0.1\nport 5053' > /etc/resolver/devo.test
sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Confirm the OS is configured to use this DNS config via ping foo.devo.test

Start a GraphQL server on localhost and run rover subgraph inspect --insecure-accept-invalid-certs https://foo.devo.test:$PORT/graphql.

Expected result

Rover should resolve the subdomain to localhost. I've tried a simple test making this request using Rust's reqwest library and confirmed that it resolves correctly. Rover also seems to follow if I put the specific subdomain in /etc/hosts, but we use scoped DNS as described in the repro steps so that we can handle wildcard resolution.

Actual result

Debug logging shows rover using the default DNS server (192.168.50.1) instead of the scoped one, which causes a DNS error:

Debug output

DEBUG rover_client::blocking::client: Request Body: {"variables":null,"query":"query SubgraphIntrospectQuery {\n # eslint-disable-next-line\n _service {\n sdl\n }\n}","operationName":"SubgraphIntrospectQuery"} at crates/rover-client/src/blocking/client.rs:89

DEBUG reqwest::connect: starting new connection: https://core.devo.test/
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.24/src/connect.rs:449

DEBUG trust_dns_proto::xfer::dns_handle: querying: core.devo.test A
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/xfer/dns_handle.rs:67

DEBUG trust_dns_resolver::name_server::name_server_pool: sending request: [Query { name: Name("core.devo.test"), query_type: A, query_class: IN }]
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-resolver-0.23.2/src/name_server/name_server_pool.rs:258

DEBUG trust_dns_resolver::name_server::name_server: reconnecting: NameServerConfig { socket_addr: 192.168.50.1:53, protocol: Udp, tls_dns_name: None, trust_negative_responses: false, bind_addr: None }
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-resolver-0.23.2/src/name_server/name_server.rs:105

DEBUG trust_dns_proto::xfer: enqueueing message:QUERY:[Query { name: Name("core.devo.test"), query_type: A, query_class: IN }]
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/xfer/mod.rs:171

DEBUG trust_dns_proto::udp::udp_client_stream: final message: ; header 51418:QUERY:RD:NoError:QUERY:0/0/0
; query
;; core.devo.test. IN A

at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/udp/udp_client_stream.rs:228

DEBUG trust_dns_proto::udp::udp_stream: created socket successfully
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/udp/udp_stream.rs:310

DEBUG trust_dns_proto::udp::udp_client_stream: received message id: 51418
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/udp/udp_client_stream.rs:348

DEBUG trust_dns_resolver::error: Response:; header 51418:RESPONSE:RD,AA,RA:NXDomain:QUERY:0/1/0
; query
;; core.devo.test. IN A
; answers 0
; nameservers 1
test. 10800 IN SOA localhost. nobody.invalid. 1 3600 1200 604800 10800
; additionals 0

at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-resolver-0.23.2/src/error.rs:152

DEBUG trust_dns_proto::xfer::dns_handle: querying: core.devo.test AAAA
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/xfer/dns_handle.rs:67

DEBUG trust_dns_resolver::name_server::name_server_pool: sending request: [Query { name: Name("core.devo.test"), query_type: AAAA, query_class: IN }]
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-resolver-0.23.2/src/name_server/name_server_pool.rs:258

DEBUG trust_dns_resolver::name_server::name_server: existing connection: NameServerConfig { socket_addr: 192.168.50.1:53, protocol: Udp, tls_dns_name: None, trust_negative_responses: false, bind_addr: None }
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-resolver-0.23.2/src/name_server/name_server.rs:119

DEBUG trust_dns_proto::xfer: enqueueing message:QUERY:[Query { name: Name("core.devo.test"), query_type: AAAA, query_class: IN }]
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/xfer/mod.rs:171

DEBUG trust_dns_proto::udp::udp_client_stream: final message: ; header 34031:QUERY:RD:NoError:QUERY:0/0/0
; query
;; core.devo.test. IN AAAA

at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/udp/udp_client_stream.rs:228

DEBUG trust_dns_proto::udp::udp_stream: created socket successfully
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/udp/udp_stream.rs:310

DEBUG trust_dns_proto::udp::udp_client_stream: received message id: 34031
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-proto-0.23.2/src/udp/udp_client_stream.rs:348

DEBUG trust_dns_resolver::error: Response:; header 34031:RESPONSE:RD,RA:NXDomain:QUERY:0/1/0
; query
;; core.devo.test. IN AAAA
; answers 0
; nameservers 1
. 2489 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2024051501 1800 900 604800 86400
; additionals 0

at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/trust-dns-resolver-0.23.2/src/error.rs:152

DEBUG reqwest::connect: starting new connection: https://core.devo.test/
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.24/src/connect.rs:449

DEBUG reqwest::connect: starting new connection: https://core.devo.test/
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.24/src/connect.rs:449

DEBUG reqwest::connect: starting new connection: https://core.devo.test/
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.24/src/connect.rs:449

DEBUG reqwest::connect: starting new connection: https://core.devo.test/
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.24/src/connect.rs:449

DEBUG reqwest::connect: starting new connection: https://core.devo.test/
at /Users/distiller/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.24/src/connect.rs:449

error[E028]: error sending request for url (https://core.devo.test/graphql): error trying to connect: dns error: no record found for Query { name: Name("core.devo.test."), query_type: AAAA, query_class: IN }
Make sure the endpoint is accepting connections and is spelled correctly

Environment

Rover Info:
Version: 0.23.0
Install Location: /Users/mattm/.rover/bin/rover
OS: Mac OS 13.6.6 [64-bit]
Shell: /opt/homebrew/bin/bash