Root task 'health reporter' is failed: OSError(99, "error while attempting to bind on address ('::1', 8080, 0, 0): cannot assign requested address")

[jicowan]

Question

After adding the health check to my Dockerfile, e.g.
kopf run --liveness=http://:8080/healthz main.py I get the following error:

Root task 'health reporter' is failed: OSError(99, "error while attempting to bind on address ('::1', 8080, 0, 0): cannot assign requested address")

I tried adding an EXPOSE declaration in the Dockerfile for port 8080 but that didn't fix the issue.

Checklist

• [x ] I have read the documentation and searched there for the problem
• [x ] I have searched in the GitHub Issues for similar questions

Keywords

[jicowan]

Adding 127.0.0.1 to the URL --liveness=http://127.0.0.1:8080/healthz seems to fix it, however, now the probe is failing with the following error:
Liveness probe failed: Get http://172.16.3.199:8080/healthz: dial tcp 172.16.3.199:8080: connect: connection refused
I confirmed that I can curl localhost:8080/healthz when I exec into the container, yet the probe continues to fail. Adjusting the timeouts have had no effect.

[Jc2k]

First thought isn't a kopf thing but the health check client runs outside the container doesn't it?, so binding on 127.0.0.1 won't work. Have you tried 0.0.0.0 instead of 127.0.0.1?

[jicowan]

@Jc2k, yes. Health checks are performed by the Kubelet. Changing the IP to 0.0.0.0 fixed the issue. Thanks! The documentation for kopf should be updated to reflect that.

[jumpojoy]

We have similar problem, in our case during kopf initialization it uses the same port when connecting to kubernetes as port in health probe. As result probing wasn't able to start

__[2020-05-19 20:23:53,740] kopf.reactor.running [ERROR ] Root task 'health reporter' is failed: [Errno 98] error while attempting to bind on address ('0.0.0.0', 44094): address already in use
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/kopf/reactor/running.py", line 453, in _root_task_checker
await coro
File "/usr/local/lib/python3.7/dist-packages/kopf/engines/probing.py", line 83, in health_reporter
await site.start()
File "/usr/local/lib/python3.7/dist-packages/aiohttp/web_runner.py", line 104, in start
reuse_port=self.reuse_port)
File "/usr/lib/python3.7/asyncio/base_events.py", line 1374, in create_server
% (sa, err.strerror.lower())) from None
OSError: [Errno 98] error while attempting to bind on address ('0.0.0.0', 44094): address already in use

We were able to fix this by specifying ip_local_port_range in security context for pod with kopf (and excluded health ports from the allowed range). We were using 32768 -32774 for health probes.

sysctls:
- name: net.ipv4.ip_local_port_range
value: 32800 60999