How to achieve Uniform distribution of load along with consistent hashing in ingress-nginx

[Darshan-j-24]

I have a GRPC client which creates single channel with 4 streams(say Stream A, B, C and D). I have grpc server pods running as statefulset pods in backend. In between ingress and nginx is present for load balancing.

When 10 clients were spawned following behaviour was observed for respective configurations.

1. Using default nginx configurations.
   Requests were distributed at stream level, that is 4 streams created by a single client did not end up in a single server. Even the streams were distributed uniformly.
   I don't want this to happen. All streams related to a channel/client should be served by a single server pod. The distribution should happen at Client level.

2. Using nginx.ingress.kubernetes.io/upstream-hash-by:"$request_uri"
   Out of 10 pods, only random 4 pods received requests. pod3 got A streams from all 10 clients. pod2 got B streams from all 10 Clients. pod6 got stream C from all 10, likewise pod7 got stream Ds.

3. Using custom header: nginx.ingress.kubernetes.io/upstream-hash-by:"$req_id"
   On client side added a unique serial number in the header using metadata key value pairs {"X-Request-ID" : serialID }. Distribution happened at client level. All four streams belonging to a client ended up in a single server pod. Distribution happened as below.

Pod-0 : -
Pod-1 : -
Pod-2 : client7.
Pod-3 : client3, client9.
Pod-4 : client5, client8.
Pod-5 : client2.
Pod-6 : -
Pod-7 : -
Pod-8 : client1, client10
Pod-9 : client4, client6

I tried with 20 and 30 clients, distribution was neither skewed nor disrtibuted equally as above. How to achieve a uniform distribution like a round robin disrtibution yet maintaining channel level distribution and not distributing streams.

[brianehlert]

Using nginx.ingress.kubernetes.io/upstream-hash-by:"$request_uri"

nginx.ingress.kubernetes.io/upstream-hash-by:"$req_id"

These annotations are not supported by this project.

4 streams created by a single client did not end up in a single server.
This is most likely because each connection is served by a unique Ingress and thus resulting unique upstream group and is driven by configuration and not NGINX behavior itself.

The way to make this work the way you want is for all paths to be accomplished using a single path or hostname resource. Otherwise the templating system will generate multiple unique configurations.

[Darshan-j-24]

Thanks @brianehlert for the response.

These annotations are not supported by this project.

May I know where or which repo is most suitable to ask regarding those annotations.

[vepatel]

@Darshan-j-24 https://github.com/kubernetes/ingress-nginx