Consul service not running in smebberson/alpine-consul-nodejs

[claflico]

Sorry to open an issue on this but I have been racking my brain and googling and can't figure out what I need to change to get the consul service working.

The documentation states: "This container has been setup to automatically connect to a Consul cluster, created with a service name of consul."

I'm trying to bring this up in a standalone vagrant vm:

docker run -d --name consul-test smebberson/alpine-consul-nodejs
...
Status: Downloaded newer image for smebberson/alpine-consul-nodejs:latest
b19f29cc949e2f65cefbb91509602d124bd5abcc3dcd0e03fb6a595cb461c287

docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b19f29cc949e smebberson/alpine-consul-nodejs "/init" 49 seconds ago Up 48 seconds 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8500/tcp, 8301-8302/udp consul-test

docker exec -t -i b19f29cc949e sh

/ # dig @localhost google.com
google.com. 299 IN A 216.58.218.206

/ # dig @localhost consul.service.consul
;consul.service.consul. IN A

The only log file with anything in it is /var/log/go-dnsmasq/go-dnsmasq.log
time="2017-01-16T22:08:14Z" level=info msg="Starting go-dnsmasq server 1.0.7"
time="2017-01-16T22:08:14Z" level=info msg="Nameservers: [10.0.2.15:53]"
time="2017-01-16T22:08:14Z" level=info msg="Setting host nameserver to 127.0.0.1"
time="2017-01-16T22:08:14Z" level=info msg="Ready for queries on tcp://127.0.0.1:53"
time="2017-01-16T22:08:14Z" level=info msg="Ready for queries on udp://127.0.0.1:53"
time="2017-01-16T22:09:45Z" level=error msg="[64612] Error looking up literal qname 'consul.service.consul.' with upstreams: read udp 172.17.0.4:39666->172.17.0.4:8600: read: connection refused"
time="2017-01-16T22:09:45Z" level=error msg="[65000] Error looking up literal qname 'consul.service.consul.' with upstreams: read udp 172.17.0.4:44608->172.17.0.4:8600: read: connection refused"

The only thing that appears to be listening is DNS:
/ # netstat -ant |grep LISTEN
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN

What i'm hoping to accomplish is that the container boots, connects to consul.service.ourdomain.consul (consul running on different servers) but every time I try to query that domain it tries to connect to the consul service that should be running locally.

Any advice would be appreciated since I'm not seeing anything in the logs.

[claflico]

I think that I just got a step closer as I now see consul mentioned in docker logs when I launch my container using:
docker run -d --name consul-test -e CONSUL_DOMAIN='ourdomain.consul' -e CONSUL_DC='loc' -e CONSUL_SERVER_NAME='consul.service.ourdomain.consul' smebberson/alpine-consul-nodejs

/ # netstat -ant |grep LISTEN
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
tcp 0 0 172.17.0.4:8301 0.0.0.0:* LISTEN
tcp 0 0 :::8500 :::* LISTEN
tcp 0 0 :::8600 :::* LISTEN
tcp 0 0 :::8400 :::* LISTEN

But still can't connect to my consul server
2017/01/16 23:05:37 [WARN] manager: No servers available
2017/01/16 23:05:37 [ERR] agent: failed to sync remote state: No known Consul servers

But I know that the container can talk to the consul server because this returns a valid json content:

curl http://10.0.2.15:8500/v1/health/service/consul?pretty

[claflico]

More headway, I added the following to the /etc/hosts file on my docker hosts: "10.0.2.15 consul.service.ourdomain.consul"

The container now connects to my consul server so this confirms my suspicion that it is a DNS issue.

[claflico]

I think that I know what the problem is.

Our consul servers are not running in Docker so the default name of "consul" won't work. We have a separate consul cluster for each environment tier so I'm trying to build the container with CONSUL_DOMAIN='ourdomain.consul' and CONSUL_SERVER_NAME='consul.service.ourdomain.consul'. consul.service.ourdomain.consul resolves properly on the docker host to the environment tier's respective consul servers.

The problem lies in this file https://github.com/smebberson/docker-alpine/blob/master/alpine-consul/root/etc/services.d/resolver/run#L7 which configures queries for CONSUL_DOMAIN to resolve to consul on the containers IP. This happens before the consul service has started and connected to the consul cluster thus causing the CONSUL_SERVER_NAME dns query to go to the consul service in the container....which isn't running yet.

Other than injecting an IP of the consul cluster nodes into the docker /etc/hosts file is there a way to tell go-dnsmasq to not send queries for consul.service.ourdomain.consul to the local consul service?

[sabrehagen]

I'm experiencing the same issue when running in Google Container Engine with Kubernetes. Did you find a solution @claflico?

[smebberson]

@claflico, sorry for not chiming in here previously.... Have you guys looked at --stubzones in go-dnsmasq?

That should allow you to configure a sub zone for which queries will be forwarded to another DNS server.

That should help with your problem right?

[claflico]

We went a different route. We are building our own containers now integrating consul-template and the run command gets the IP address of the container gateway (the host) and connects to the consul client running on the host via the gateway IP.

Feel free to close.