IP resolver not updating IPs on failure

[Excpt0r]

Hi,

I use jetcd in an application that runs on kubernetes and connects to a 3-node ETCD cluster (also k8s).
When I scale down the ETCD cluster to 0 instances and then scale up again to 3,
jetcd client is not able to reconnect again to the ETCD cluster (healthy again).
The new ETCD pods on kubernetes get a new IP, and DNS names are updated and also resolve correctly within the JVM (when networkaddress.cache.ttl=10 is set).
But jetcd client still retries with the previous outdated IPs.

Debugging the logs, I thought the problem is related to grpc-java so I opened ticket grpc/grpc-java#8574 there, including further logs.
It looks like the IPresolver implementation of jetcd is not capable to update the IPs after failures.

Tested with jetcd 0.5.10

[lburgazzoli]

do you have any time to work on a fix ?

[Excpt0r]

So far I have not enough knowledge about jetcd, the changes done in #814 and how that works with grpc-java, to be able to design a solution.
If you already have an idea what needs to be changed, let me know and I will have a look.

Update: Just saw that #814 did not change much beside the name. But the question is why jetcd needs own resolve logic.

[lburgazzoli]

Added a potential fix here: #994
Mind testing against that PR ?

[lburgazzoli]

Update: Just saw that #814 did not change much beside the name. But the question is why jetcd needs own resolve logic.

Because grpc-java does not provide a resolver that supports multiple hosts

[Excpt0r]

Thanks! I will test the PR in the next few days.

[Excpt0r]

Hi @lburgazzoli
I just tested PR #994 and results are looking good to be, jetcd now updates the DNS information on failure and can connect to the new pod IPs.
Thank you for your work!

[lburgazzoli]

cool !

I'll write some unit test and then I can cut a release

[lburgazzoli]

fixed by #994