dial tcp 10.10.x.x:9090: connect: connection refused

[dineshspathak]

Hi Thanks for the help, I have re-downloaded docker-compose.yml, and I am now getting a TCP connection refused, do I need to change any config?

time=2023-06-19T12:29:52.945Z level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/main.go:31 msg="failed to init metrics manager" component=mii-lama error="failed to query tsdb status page: Get "[http://10.10.x.x:9090/api/v1/status/tsdb](http://10.10.x.x:9090/api/v1/status/tsdb%5C)": dial tcp 10.10.x.x:9090: connect: connection refused"

[mr-karan]

@dineshspathak Can you please show the config.toml file? 10.10.x.x:9090 should be Prometheus endpoint which is running in docker environment.

[dineshspathak]

hi @mr-karan i kept it default for testing

[dineshspathak]

mii-lama | time=2023-06-19T13:18:38.390Z level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/main.go:31 msg="failed to init metrics manager" component=mii-lama error="failed to query tsdb status page: Get "http://0.0.0.0:9090/api/v1/status/tsdb\": dial tcp 0.0.0.0:9090: connect: connection refused"

[dineshspathak]

hi @mr-karan i have not given any nse login credentials yet, but getting below, i think this should resolve once we enter nse login id, password, and member id

mii-lama | time=2023-06-20T07:13:29.397Z level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/main.go:51 msg="failed to init nse manager" component=mii-lama error="failed to login to NSE API: failed to send HTTP request: Post "https://lama.nse.internal/api/V1/auth/login\": dial tcp: lookup lama.nse.internal on 127.0.0.11:53: no such host"
mii-lama exited with code 1

[mr-karan]

@dineshspathak The sample config is dummy credentials. The prod/UAT creds for LAMA API Gateway has to be added by you.

[mr-karan]

"[http://0.0.0.0:9090/api/v1/status/tsdb](http://0.0.0.0:9090/api/v1/status/tsdb%5C)": dial tcp 0.0.0.0:9090: connect: connection refused"

0.0.0.0 inside the docker container is wrong. It has to be the Prometheus' container IP.

The sample config sets it to prometheus:9090 (https://github.com/zerodha/mii-lama/blob/main/config.sample.toml#L17). The container's DNS resolver does the dns resolution for prometheus and finds the IP for the Prometheus container. No change is required for this config.

[dineshspathak]

Hi @mr-karan i am getting below result after inspecting the container network
"Containers": {
"10bddc75f813736b7e3cf73fbd39cfb64aa5602b2c153d894e4d2f62b65594d5": {
"Name": "prometheus",
"EndpointID": "d1df5e0107a38a174b1b2cef3c49977be6f51dbf44d48b040c87c6488d93fdb5",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""

Where I can check the scrap targets?

[mr-karan]

The scrape targets are defined here: https://github.com/zerodha/mii-lama/blob/main/prometheus.sample.yml#L9

Replace them with your own server targets wherever node-exporter is running.

[dineshspathak]

hi @mr-karan can you please help, how can I check targets as the Prometheus is running inside a container with virtual IP which is not accessible in my environment.

[mr-karan]

Can you check localhost:9090 ? The IP you're checking is the container's IP which is accessible within the docker env only.

[dineshspathak]

Hi @mr-karan I am accessing the system via the terminal over the VPN, in this case, localhost will not work, can I check the same with my localip:9090? but for this also I need to change prometheus containers ip to my local server ip, kindly suggest.

[mr-karan]

In that case, you should use the private/public IPv4:9090 of the system instead of the docker container. I won't be able to help much in debugging the exact situation as this depends on your networking setup.

[shethpriyank]

I am facing the same problem. unable to access. Below is the error message.

mii-lama | time=2023-06-30T03:57:37.711Z level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:51 msg="Failed to query Prometheus" component=mii-lama host=172.168.x.x:9100 metric=cpu error="response contains no result data"
mii-lama | time=2023-06-30T04:02:37.691Z level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:119 msg="Failed to query Prometheus for database status" component=mii-lama host=192.168.x.x:9182 metric=status error="response contains no result data"

Same problem, virtual container ip is 172.18.0.2 which is not accessible from outside network. container is accessible from host machine.
Pl help....

[mr-karan]

Please ensure that node-exporter is running on the host 172.168.x.x:9100 and this IP is accessible from within the container.

[shethpriyank]

My both the node exporter is running. on browser I am getting data. My Prometheus container is also up and running. but still, I am getting same error. Is there any route we need to add in Prometheus container for host network? I am not sure on this.. I have tried from Prometheus container and got the response as mentioned below.

lama@lama:~$ sudo docker exec -it prometheus /bin/sh
/prometheus $
/prometheus $ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default lama 0.0.0.0 UG 0 0 0 eth0
172.20.0.0 * 255.255.0.0 U 0 0 0 eth0
/prometheus $
/prometheus $
/prometheus $ telnet 192.168.11.15 9182
Connected to 192.168.11.15
exit
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

400 Bad RequestConnection closed by foreign host
/prometheus $ telnet 172.168.10.90 9100
Connected to 172.168.10.90
exit
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

400 Bad RequestConnection closed by foreign host
/prometheus $ wget http://172.168.10.90:9100/
Connecting to 172.168.10.90:9100 (172.168.10.90:9100)
wget: can't open 'index.html': File exists
/prometheus $ wget http://172.168.10.90:9100/metrics
Connecting to 172.168.10.90:9100 (172.168.10.90:9100)
wget: can't open 'metrics': File exists
/prometheus $ wget http://172.168.10.90:9100/metric
Connecting to 172.168.10.90:9100 (172.168.10.90:9100)
wget: can't open 'metric': File exists
/prometheus $ wget http://192.168.11.15:9182/metrics
Connecting to 192.168.11.15:9182 (192.168.11.15:9182)
wget: can't open 'metrics': File exists
/prometheus $ wget http://192.168.11.15:9182/metric
Connecting to 192.168.11.15:9182 (192.168.11.15:9182)
wget: can't open 'metric': File exists
/prometheus $