NGINX Gateway Fabric Pods Failing Health Checks Under High Connection Load on GKE Autopilot

[NicolasPires777]

Hello everyone! I'm experiencing a very strange problem in my Kubernetes cluster that's especially affecting the availability of my NGINX Gateway Fabric dataplane pods.

Here's the issue: Even though my node running the pod has plenty of spare CPU and memory, the /health endpoint of the dataplane pods still fails occasionally. If this simple connection fails, my clients' connections fail as well. I can see in my application monitoring graphs that whenever the NGINX /health check fails, the usage of my apps drops significantly.

I found this issue that seems related: https://stackoverflow.com/questions/410616/increasing-the-maximum-number-of-tcp-ip-connections-in-linux
Should I apply some configuration like this on my machines? Can I set this directly in the NGINX deployment?

My current setup is a GKE 1.33 cluster with Autopilot. I've already contacted GCP support, and they recommended using C4 machine types, but that didn't solve my problem. During peak hours, we reach up to 600,000 requests per minute and I believe there are people handling even more traffic—what do they do to manage this load?

Our workaround has been to scale horizontally: for 600,000 requests, I need 18 nodes JUST for NGINX, so I don't hit the "machine connection limit" mentioned in the Stack Overflow link above.

Can anyone help or at least point me in the right direction? This issue is severely impacting our application's availability and it feels like it shouldn't be so hard to solve. We've been struggling with this for over a month and haven't found a solution, not even with GCP's own support. Could this be some NGINX configuration I'm missing?

Sometimes, even other types of connections fail, such as when I try to fetch logs from the pods. I'll attach some images below to illustrate the issue.

[nginx-bot]

Hi @NicolasPires777! Welcome to the project! 🎉

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[salonichf5]

It appears that the behavior you’re seeing is most likely related to ephemeral port exhaustion and outbound connection handling limits, rather than CPU or memory resource constraints. This is a fairly common scenario in high-throughput environments, and there are several approaches you can take to mitigate it:

Scaling guidance: NGF Scaling Recommendations

1. Tune NGINX Worker Settings

We recommend reviewing your NGINX worker configuration to ensure it’s optimized for high connection volumes:
Set worker_processes auto; so that NGINX spawns one worker per CPU core.
Increase worker_connections (for example, 65535) to allow each worker to handle a larger number of concurrent sockets.
These changes help maximize the capacity of each dataplane instance and reduce the total number of replicas required to sustain high levels of traffic.

2. Use Custom Metrics for Autoscaling

Horizontal scaling decisions based solely on CPU and memory usage are often insufficient in high-connection scenarios. Instead, consider scaling based on connection-level metrics:
Expose metrics such as active connections or socket usage via Prometheus.
Define a custom metric in the NGF AutoscalingSpec so the HPA can proactively scale dataplane replicas before port exhaustion occurs.
You can refer to the Prometheus monitoring guide
for details on exposing and collecting these metrics.

3. Kernel-Level Tuning

Finally, make sure the underlying node configuration is tuned for large-scale connection handling:
Increase net.ipv4.ip_local_port_range (e.g., 1024 65535) to expand the available ephemeral port range.
Lower net.ipv4.tcp_fin_timeout to allow ports to be recycled more quickly.
Raise the file descriptor limit (ulimit -n) to ensure the process can open a larger number of concurrent sockets.
These changes are consistent with the recommendations in the Stack Overflow thread you referenced and are essential for stable operation under heavy load.

We are tracking a related discussion around this topic and will ensure it’s prioritized. In the meantime, applying the recommendations above typically resolves intermittent /health probe failures and significantly improves dataplane stability at scale.

[salonichf5]

Which version of NGF are you using currently? @NicolasPires777

[NicolasPires777]

I'm using 2.1.4

[NicolasPires777]

Hi @salonichf5 , I’ll apply recommendations 1 and 2 you shared, but regarding the third one, I’m running on a GKE Autopilot cluster. It’s more like a “serverless” service, where I don’t think I can directly change the underlying machine configurations. Do you think recommendations 1 and 2 should be enough, or is the third one essential?

[salonichf5]

@NicolasPires777 I am not very sure of the level of impact for point 3, I guess the idea is open more port ranges so maybe there should be a different way to do so in GKE? Let me know how the other two go for you

[NicolasPires777]

I’ll reach out to some companies with auto-traffic to ask for tips on what I can do. I haven’t been able to apply methods 1 and 2 yet, but I’ll try during this week to see if I get any results.

But I think the limitation is really at the machine level.