Potential issue with jdomain - keep seeing requests sending to old upstream after DNS update

[kongdewen]

Hey Folks,

We are seeing some weird issue after we upgrade jdomain to the latest release.

We have a very dynamic upstream which DNS updated quite often. We have been using jdomain to help with the upstream IP resolving. After upgrade, we are starting to see the issue that request keeps sending to old upstream after DNS update. We have no idea what could be wrong.

Any help will be appreciated,

Our upstream nginx config setup is pretty simple:

upstream upstream-upstream {
    jdomain xxx.xxx.xxx.xxx port=xx;
    keepalive 256;
  }

we are using openresty with version 1.19.3.1

[nicholaschiasson]

Thanks for raising this, I'll be looking into it.

[nicholaschiasson]

Could you provide some additional details please?
How many workers do you have configured?
After a DNS update, for how long does nginx continue forwarded requests to the old upstreams? Does it self correct after a long period or require an nginx reload?

[kongdewen]

hey @nicholaschiasson, we have 64 worker process configured, we didn't some test, after a DNS update, we didn't see self correct even after a long period time and nginx reload resolve this issue.

[zdm]

Seems, that module not works woth the nginx 1.19.7.
Peers ip addresses resolved at startup and cached forever.

[InvisibleFunction]

I believe I've experienced this issue with nginx 1.16.1 on centos7. My upstream is an s3 bucket which returns 1 ip with a ttl of 5s but I would see my connections stick with the same backend even if I explicitly defined a short interval.

[AljoschaDembowsky2909]

@nicholaschiasson any updates on this issue? We are looking forward to use this module in combination with AWS ALB upstreams but we have to make sure that the DNS resolving is working as expected. AWS ALB's will return up to 3 IP addresses and they update alot if the traffic scales in or out and if one or more ip's are cached indefintely this will cause more harm than good. In our current setup with the default proxy pass resolving workaround using variables we experience timeouts once in a while but we want to reduce this to a bare minimum.
Unfortunately we dont have an C developer in our team who could support you but any feedback/update would be greatly appreciated.

Setup:
Nginx 1.19.X
50+ AWS ALB upstreams

[nicholaschiasson]

Hi @AljoschaDembowsky2909,
Thanks for the detailed description. Sorry, no update on this issue just yet. I am hoping to make the time to tackle it very soon.

[Hacksign]

Same issue here.
Alpine Linux with docker, nginx 1.18.0, nginx-mod-http-upstream-jdomain-1.18.0-r13.

[atomsnc]

@kongdewen @zdm @InvisibleFunction @AljoschaDembowsky2909 @Hacksign can you please try jdomain version 0.9.11 and see if you still get the issue?

[nicholaschiasson]

Hi all, @kongdewen @zdm @InvisibleFunction @AljoschaDembowsky2909 @Hacksign,

This issue has been accidentally closed as it was referenced in #75 . Thanks for that fix @atomsnc !

I am hopeful now with this bugfix, but I don't want to disregard this issue in case people are still facing it. I realize this is years old for some of you now, but if you would have some time to test your setup with the latest version of this module and try validating if this is fixed now, then that would be very appreciated.

I will keep this issue closed for now, and if it seems to continue being a problem, I will reopen it.

Thanks everyone!