Session affinity using client ip over multiple tcp ports

[malcolmvs]

Hi, we are looking at trying to implement session affinity using client ip over multiple tcp ports. Something similar to what is done here: https://stackoverflow.com/questions/61197263/kubernetes-nginx-ingress-session-affinity-across-ports

nginx.ingress.kubernetes.io/upstream-hash-by

Is there a way to do this using this ingress controller? We currently use 3 TCP ports that need to be connected to the same pod for a given client ip. When we used the remote_addr hash in upstream groups we unfortunately ended up with different hashes as each upstream group uses a different tcp port.

Anyone have ideas or input on this please?

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-cam-ingress
  namespace: cam
  uid: df8817a9-7dff-4e94-aa0c-d5785a538190
  resourceVersion: '270861733'
  creationTimestamp: '2020-08-11T03:09:16Z'
  labels:
    app.kubernetes.io/instance: nginx-cam-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx-cam-ingress-nginx-ingress
    helm.sh/chart: nginx-ingress-0.16.2
  annotations:
    meta.helm.sh/release-name: nginx-cam-ingress
    meta.helm.sh/release-namespace: cam
  managedFields:
    - manager: Go-http-client
      operation: Update
      apiVersion: v1
      time: '2021-04-16T04:20:40Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:data: {}
        f:metadata:
          f:annotations:
            .: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
    - manager: helm
      operation: Update
      apiVersion: v1
      time: '2023-03-15T01:59:45Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            f:helm.sh/chart: {}
    - manager: Mozilla
      operation: Update
      apiVersion: v1
      time: '2023-07-05T12:14:04Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          f:lb-method: {}
data:
  lb-method: hash $remote_addr consistent

kind: Service
apiVersion: v1
metadata:
  name: nginx-cam-ingress
  namespace: cam
  uid: f62ed1e9-e2c4-4e54-bc8f-87249be6fde6
  resourceVersion: '270403321'
  creationTimestamp: '2023-07-04T05:40:04Z'
  labels:
    app.kubernetes.io/instance: nginx-cam-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx-cam-ingress-nginx-ingress
    helm.sh/chart: nginx-ingress-0.16.2
  annotations:
    meta.helm.sh/release-name: nginx-cam-ingress
    meta.helm.sh/release-namespace: cam
    service.beta.kubernetes.io/azure-dns-label-name: <redacted>
    service.beta.kubernetes.io/azure-load-balancer-resource-group: KubeInfra
  finalizers:
    - service.kubernetes.io/load-balancer-cleanup
  managedFields:
    - manager: Mozilla
      operation: Update
      apiVersion: v1
      time: '2023-07-04T05:40:04Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
            f:service.beta.kubernetes.io/azure-dns-label-name: {}
            f:service.beta.kubernetes.io/azure-load-balancer-resource-group: {}
          f:finalizers:
            .: {}
            v:"service.kubernetes.io/load-balancer-cleanup": {}
          f:labels:
            .: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:helm.sh/chart: {}
        f:spec:
          f:allocateLoadBalancerNodePorts: {}
          f:externalTrafficPolicy: {}
          f:healthCheckNodePort: {}
          f:internalTrafficPolicy: {}
          f:ipFamilies: {}
          f:ipFamilyPolicy: {}
          f:loadBalancerIP: {}
          f:ports:
            .: {}
            k:{"port":80,"protocol":"TCP"}:
              .: {}
              f:name: {}
              f:port: {}
              f:protocol: {}
              f:targetPort: {}
            k:{"port":443,"protocol":"TCP"}:
              .: {}
              f:name: {}
              f:port: {}
              f:protocol: {}
              f:targetPort: {}
            k:{"port":56000,"protocol":"TCP"}:
              .: {}
              f:name: {}
              f:port: {}
              f:protocol: {}
              f:targetPort: {}
            k:{"port":56001,"protocol":"TCP"}:
              .: {}
              f:name: {}
              f:port: {}
              f:protocol: {}
              f:targetPort: {}
            k:{"port":56002,"protocol":"TCP"}:
              .: {}
              f:name: {}
              f:port: {}
              f:protocol: {}
              f:targetPort: {}
          f:selector: {}
          f:sessionAffinity: {}
          f:type: {}
    - manager: cloud-controller-manager
      operation: Update
      apiVersion: v1
      time: '2023-07-04T05:40:19Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:loadBalancer:
            f:ingress: {}
      subresource: status
spec:
  ports:
    - name: image-tcp
      protocol: TCP
      port: 56000
      targetPort: 56000
    - name: settings-tcp
      protocol: TCP
      port: 56001
      targetPort: 56001
    - name: cctv-tcp
      protocol: TCP
      port: 56002
      targetPort: 56002
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 31672
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
  selector:
    app: nginx-cam-ingress
  clusterIP: <redacted>
  clusterIPs:
    - <redacted>
  type: LoadBalancer
  sessionAffinity: None
  loadBalancerIP: <redacted>
  externalTrafficPolicy: Local
  healthCheckNodePort: 31386
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: true
  internalTrafficPolicy: Cluster
status:
  loadBalancer:
    ingress:
      - ip: <redacted>

ClientIP session affinity is enabled on the load balancer - I'm not sure if this is correct.

[vepatel]

@malcolmvs nginx.ingress.kubernetes.io/upstream-hash-by is only for community Ingress controller so won't work in https://github.com/nginxinc/kubernetes-ingress.
Are you looking for something like: https://github.com/nginxinc/kubernetes-ingress/tree/v3.2.0/examples/ingress-resources/session-persistence ?

[malcolmvs]

Hi @vepatel thank you for your reply. Yes I understood that the example I gave is for the community version. I'm looking for similar functionality for this ingress controller. What you recommended makes use of cookies - isn't that only for HTTP/S traffic and not for TCP?

[jasonwilliams14]

@malcolmvs Take a look at the hash load balancing algorithm.

https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#choosing-a-load-balancing-method

https://nginx.org/en/docs/http/ngx_http_upstream_module.html#hash

You can do this quite easily using our virtualServer CRD and use NGINX variables to key off ($remote_addr for example)
Here is the link to our CRD documentation:
https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#upstream

Take a look at lb-method:
Off the top of my head would be something like:

lb-method: hash $remote_addr consistent

Hope this helps.

[malcolmvs]

Thanks @jasonwilliams14 I believe we did try this, however, the hashing seems to be different between each upstream so the affinity is not maintained across the pods. Here's an example of the resulting config:

stream { 
	upstream app1 {     
		zone app1 256k;      
		hash $remote_addr consistent;      
		server 10.200.1.50:56000;     
		server 10.200.2.10:56000;     
		server 10.200.3.15:56000;     
		server 10.200.4.20:56000; 
	}  
	server {      
		listen 56000;     
		proxy_pass app1; 
	}  

	upstream app2 {     
		zone app2 256k;      
		hash $remote_addr consistent;      
		server 10.200.1.50:56002;     
		server 10.200.2.10:56002;     
		server 10.200.3.15:56002;     
		server 10.200.4.20:56002; 
	}  
	server {      
		listen 56002;     
		proxy_pass app2; 
	}
  
	upstream app3 {     
		zone app3 256k;      
		hash $remote_addr consistent;      
		server 10.200.1.50:56001;     
		server 10.200.2.10:56001;     
		server 10.200.3.15:56001;     
		server 10.200.4.20:56001; 
	}  
	server {      
		listen 56001;     
		proxy_pass app3; 
	}   
}

[jasonwilliams14]

@malcolmvs do you have your .yaml file that you have specified? I see this is Layer 4 and not Layer 7, so you would need to use transportServer.
Also, how are you testing the affinity to the backend pods to track the state of the originating connection?

[malcolmvs]

Sorry for not giving all the relevant information upfront. I added the ingress controller details above. Here is our ingress-servers.yaml template:

kind: VirtualServer
metadata:
  name: camserver-vs
spec:
  ingressClassName: {{.Values.ingressServers.ingressClassName}}
  host: {{.Values.ingressServers.domainName}}
  tls:
    secret: {{.Values.ingressServers.secretName}}
  upstreams:
  - name: cam
    service: {{.Values.service.name}}
    port: 80
    lb-method: ip_hash
    tls:
      enable: false
  routes:
  - path: /
    action:
      pass: cam
---
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: camserver-vs2
spec:
  ingressClassName: {{.Values.ingressServers.ingressClassName}}
  host: {{.Values.ingressServers.domainName2}}
  tls:
    secret: {{.Values.ingressServers.secretName2}}
  upstreams:
  - name: cam
    service: {{.Values.service.name}}
    port: 80
    lb-method: ip_hash
    tls:
      enable: false
  routes:
  - path: /
    action:
      pass: cam
---
apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
  name: camserver-ts-image
spec:
  listener:
    name: {{.Values.listeners.image}} 
    protocol: TCP
  upstreams:
  - name: image-app
    service: {{.Values.service.name}}
    port: {{.Values.ports.image}}
  action:
    pass: image-app
---
apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
  name: camserver-ts-settings
spec:
  listener:
    name: {{.Values.listeners.settings}}
    protocol: TCP
  upstreams:
  - name: settings-app
    service: {{.Values.service.name}}
    port: {{.Values.ports.settings}}
  action:
    pass: settings-app
---
apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
  name: camserver-ts-cctv
spec:
  listener:
    name: {{.Values.listeners.cctv}}
    protocol: TCP
  upstreams:
  - name: cctv-app
    service: {{.Values.service.name}}
    port: {{.Values.ports.cctv}}
  action:
    pass: cctv-app

For testing I have an external service with 4 instances that connect from the same IP (so a total of 12 connections for each of the 3 TCP ports). These connections are dropped and recreated at 30-60s intervals. We have 4 pods and I run a netstat on each pod to see the connections as well as checking the nginx logs to see how connections are being made. At the moment the connections are spread across all the pods and continue to jump around when they reconnect. For an additional test I do a telnet from my local machine to one of the ports on the external load balancer IP to see which pod it routes me to. As an aside if I remove the nginx controller and expose the service as LoadBalancer service type it works as expected. We can't run that setup though as we need the SSL termination + cert management that the nginx ingress controller provides.

[brianehlert]

Full stateful sessions need to be handled with a design all the way through the stack.
This starts at using a daemonset instead of a deployment to prevent kubeproxy from bouncing reconnects between the various ingress controller pods. That is the first place that can break persistence, which needs to start at the disaggregation (load balancer) layer.

After a client is reliably hitting the same ingress controller pod, then the directives should be consistently taking over.

[malcolmvs]

Thanks for weighing in @brianehlert

Correct me if I'm wrong but we have a single pod for the ingress controller, so there shouldn't be a problem there.

I'm starting to think this is specifically an issue with the ingress controller as I made mention in another comment that everything works perfectly if we remove the ingress controller.

[brianehlert]

I assume that if the ingress controller is not present you are exposing the services via nodeport and hitting them directly.
Which means there is no session persistence at play.
Another thing that I have wondered in looking at your application is that you have your services split among different VirtualServer and TransportServer objects.
This in turn means that part of the traffic is using the NGINX http block under the hood and part of the traffic is using the stream block under the hood.
It also means that each resource is also a unique upstream group.

I have been curious if this is a single client app and a single backend app that has different ports exposed for different functions.
Which would mean that we want to handle the backend as a single app unit.

I am wondering if part of the disconnect is the split between http and stream.
Which I am guessing is happening due to port 80 and 443 being involved.

[malcolmvs]

It's when I set the service type to be LoadBalancer that session affinity works as expected. Is that exposing it as node port?

You are correct, there are multiple upstreams - which in part might be causing the issue as I also previously mentioned regarding the resulting nginx configuration. My understanding for the HTTP block is for it to be able to have a health check. The services on the other ports don't have that type of functionality (returning a simple OK result to indicate the service is up and running). Yes, each port exposes a separate function. We use SSL termination so port 443 stops at the nginx controller and is then forwarded to port 80 on the backend application side. The same for HTTP traffic. Although I must note that we only listen on port 9000 and the other 3 ports on the backend app, but that is all routed correctly on the backend app service definition.

I do agree that it should be treated as a single backend app, but how do we go about that? I do think that's how AKS is handling it once I change the app from service to LoadBalancer.

Is there any additional information I can provide?

[malcolmvs]

Can anyone offer guidance on this? Have we configured this incorrectly? Is this only available on nginx plus? Is this not possible? Is it perhaps a bug?

[brianehlert]

Regarding backend health.
There are two parts here.
One is the NGINX view, which for a stream backend (TCP/UDP) I believe is a simple ping with no advanced healthcheck options.
The other is the K8s readiness state. Which causes NIC to add or remove the pods of a service from the upstream group.

If your readiness state is accurate, then NIC adds or removes the backend pods from the upstream group.

If you change the application Service to loadbalancer, then the service "sits next to" the ingress controller rather than "behind" it.

You could use TransportServer and TLS termination and a range of ports.
The complexity is if your application requires port 80 & 443 since we have not yet given you full control of those listening ports.
We are working on that, right now.

Which means the immediate fix would be to clone and modify the project to not make default listeners for port 80 and 443 under the http block. And then you should be able to define them using TransportServer and GlobalConfiguration

We have a doc related to this: https://docs.nginx.com/nginx-ingress-controller/tutorials/custom-listen-ports/