Rate-limiting in combination with auto-scaling

[dbaumgarten]

Hi,

we have currently implemented Rate-Limiting on our Ingress-Resources using http- and location-snippets (using the limit_req directives).

That is working fine so far. Requests are limited properly per IP.

However, we also have auto-scaling configured for our ingress-controller-pods.

The issue now is, that the rate-limits applied via snippets are per-pod. Once additional pods are spawned by the autoscaling, the total request-limit rises (as requests are now distributed on more pods, which each have an individual rate-limit).

This means that when you simply send enough requests to trigger a scale-up, you can raise and therefore circumvent the rate-limit.

Any suggestions on how to solve this?
Is there a way to specify a rate-limit accross all pods? Or to adapt the per-pod-limit based on the number of pods?

[brianehlert]

Rate limiting is by nature per pod (or per-instance if you are running machines).
We are investigating the ability of using the NGINX Plus key/value store to share the state across the instances to allow these to be global.

The possibility of workarounds depends on your applications. If for example, you deploy as a daemonset and you set your loadbalancer with session persistence, it will tend to consistently steer a client to a particular Node/pod and thus result in a more consistent apply of the setting.
If, however, you deploy as a deployment the request will pass through KubeProxy before hitting the ingress controller pods and thus there is no way to predict which ingress controller pod the request will pass through and thus we absolutely hit your issue.

We also have some customers who run deployments and thus divide the rate limit by the number of pods in the deployment and thus apply a non-precise but better SWAG in that way.

Additional thoughts?

[brianehlert]

The NGINX Plus settings are described in this blog post:
https://www.nginx.com/blog/nginx-plus-r16-released/#r16-cluster-rate-limiting

The key point is the zone_sync. And we do have a method to easily set-up zone_sync with ingress controller using a headless service (and thus automatically follow HPA).
Let me see if I can get someone to describe that.

[brianehlert]

Here is general background on rate limiting: https://www.nginx.com/blog/rate-limiting-nginx/

[dbaumgarten]

Thats what I feared.

What about using virtual-servers instead of ingresses: https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/ ?

Are their rate-limit policies ( https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/ ) also per pod, or does it automatically account for the number of active pods?

One idea I had was to build some kind of kubernetes-controller which constantly monitors the amount of active controller-pods and automatically adjusts the rate-limit in the configmap. But that feels a little dirty to me.

[brianehlert]

Optimally, we want NGINX Plus to handle this using zone_sync as that will be the most accurate.
Alternately, I am interpreting that you are trying to achieve global behavior with the free edition.

There is one free edition customer I know who drives their scaling with automation (not HPA) and thus reconfigures the rate limit settings when a scaling action happens.

The controller does not 'automagically' attempt to do math here. As you state, we would need some type of Controller/Operator process to drive that from the outside. Helm is the only place where this is all templated together, otherwise it is distinct manifests.
We leave it in your hands do that math today.
If using VirtualServer rate limit Policy, then just the Policy object needs to be updated, not an injected snippet - much easier to programmatically handle.

[dbaumgarten]

Using nginx-plus with zone_sync sounds promising. We aleady thought about using nginx plus. I will discuss this inernally.

So when using nginx plus with the correct zone_sync settings cluster-wide rate-limiting should just work "out of the box"?

Thank you very much for the insights :)

[brianehlert]

There are some settings to turn it on (we have a headless service setup we use) and directives to set what is shared.
Not horrible, but not all exposed through YAML yet (needs snippets).