Can VPC-CNI with Security Group for Pods work with kube-proxy in IPVS mode? #2982

AhmadMS1988 · 2024-07-06T19:55:09Z

What happened:
We need to use kube-proxy in IPVS mode, with VPC CNI because of our increased numbers of pods, as long as to achieve between request load balancing using k8s services.

We noticed that pods that has security groups and branch network interfaces, the both ingress and egress traffic of these pods stops and never comeback until we go back to iptables mode and refresh all nodes.

We need to know if VPC CNI supports both security groups for pods and kube-proxy in IPVS mode.

Environment:

Kubernetes version (use kubectl version):
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1-eks-1de2ab1
CNI Version
v1.18.2-eksbuild.1
kube-proxy Version
v1.30.0-eksbuild.3
OS (e.g: cat /etc/os-release):
Amazon Linux 2023.5.20240624 v1.30.0-eks-036c24b
Bottlerocket OS 1.20.3 (aws-k8s-1.30) v1.30.0-eks-fff26e3
Kernel (e.g. uname -a):
Amazon Linux: 6.1.94-99.176.amzn2023.x86_64
Bottlerocket OS: 6.1.92

Tested on both arm64 and amd64

The text was updated successfully, but these errors were encountered:

orsenthil · 2024-07-10T17:42:11Z

VPC-CNI with Security Group for Pods work with kube-proxy in IPVS

Yes,there is no limitation that we know that will not work. Kube-Proxy is service proxy later; and it's mode of ipvs or iptables shouldn't interfere with SGPP functionality of pods.

I will setup a IPVS cluster and verify and share my output.

yash97 · 2024-07-10T20:13:41Z

Hi @AhmadMS1988,

I set up the cluster and switched kube-proxy to IPVS mode. I also created an Nginx service with a security group and tested in-cluster pod-to-pod connectivity to Nginx, which worked fine. Could you please provide more details on the specific networking test that is failing? TIA!

AhmadMS1988 · 2024-07-11T12:27:20Z

Hi all;
@orsenthil please inform me with your findings.
@yash97 are you sure that you have replaced the worker nodes after switching to ipvs mode in kube-proxy?
I open a ticket with AWS EKS team and they confirmed that they can replicate the situation.
Thanks all

orsenthil · 2024-07-12T23:35:33Z

@AhmadMS1988 - I was able to verify SGP working with kube-proxy in IPVS mode.

Followed these public docs.

https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html - Created my-app, my-deployment exactly as specified. Manually created the security group with egress/ingress ALL from anywhere for testing purposes. Associated the security group and ensured new pods had the security associated.
2.Created a curl client to test connectivity with my-app

apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  namespace: my-namespace
spec:
  containers:
  - name: curl-container
    image: curlimages/curl
    command: ['sh', '-c', 'while true; do sleep 30; done;']

Verified connect to the service with kube-proxy in iptables with SGP is working fine.

kubectl exec -it curl-pod -n my-namespace -- sh
~ $ curl my-app
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Switched to IPVS Mode.

https://repost.aws/knowledge-center/eks-configure-ipvs-kube-proxy
Edited kube-proxy to ipvs mode with rr scheduling - kubectl edit cm kube-proxy-config -n kube-system
Scaled up and down my node-group for ipvs changes to take effect.

Verified my service is backed by pod ips in the IPVS mode

# sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  ip-10-100-0-1.us-west-2.comp rr
  -> ip-192-168-157-183.us-west-2 Masq    1      1          0
  -> ip-192-168-172-5.us-west-2.c Masq    1      2          0
TCP  ip-10-100-0-10.us-west-2.com rr
  -> ip-192-168-8-160.us-west-2.c Masq    1      0          0
  -> ip-192-168-64-32.us-west-2.c Masq    1      0          0
TCP  ip-10-100-166-45.us-west-2.c rr
  -> ip-192-168-20-65.us-west-2.c Masq    1      0          0
  -> ip-192-168-30-168.us-west-2. Masq    1      0          0
  -> ip-192-168-72-39.us-west-2.c Masq    1      0          0
  -> ip-192-168-78-170.us-west-2. Masq    1      0          0
UDP  ip-10-100-0-10.us-west-2.com rr
  -> ip-192-168-8-160.us-west-2.c Masq    1      0          0
  -> ip-192-168-64-32.us-west-2.c Masq    1      0          0

Note the my-app service and my-deployment pods (match with the ipvs in the ipvsadm -L)

[senthilx@dev-dsk-senthilx-2a-b140a96c ~]$ kubectl get svc -o wide -A
NAMESPACE      NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE   SELECTOR
default        kubernetes   ClusterIP   10.100.0.1      <none>        443/TCP         9d    <none>
kube-system    kube-dns     ClusterIP   10.100.0.10     <none>        53/UDP,53/TCP   9d    k8s-app=kube-dns
my-namespace   my-app       ClusterIP   10.100.166.45   <none>        80/TCP          21m   app=my-app

[senthilx@dev-dsk-senthilx-2a-b140a96c ~]$ kubectl get pods -o wide -n my-namespace
NAME                             READY   STATUS    RESTARTS   AGE     IP               NODE                                          NOMINATED NODE   READINESS GATES
curl-pod                         1/1     Running   0          2m58s   192.168.64.33    ip-192-168-78-86.us-west-2.compute.internal   <none>           <none>
my-deployment-5bf4d6bfff-4q7s9   1/1     Running   0          11m     192.168.72.39    ip-192-168-78-86.us-west-2.compute.internal   <none>           <none>
my-deployment-5bf4d6bfff-f4trw   1/1     Running   0          11m     192.168.30.168   ip-192-168-6-104.us-west-2.compute.internal   <none>           <none>
my-deployment-5bf4d6bfff-s7v47   1/1     Running   0          11m     192.168.78.170   ip-192-168-78-86.us-west-2.compute.internal   <none>           <none>
my-deployment-5bf4d6bfff-x4rlx   1/1     Running   0          11m     192.168.20.65    ip-192-168-6-104.us-west-2.compute.internal   <none>           <none>

Ensured that test connectivity is successful.

$ kubectl exec -it curl-pod -n my-namespace -- sh
~ $ curl my-app
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

AhmadMS1988 · 2024-07-13T10:57:27Z

Hi @orsenthil
I confirmed the behaviour you see for the inbound requests.
Can you exec info one of my-deployment pods and do any outbound curl? for example, google.com?
or if you want try to give curl-pod a security group try to curl the service again will it work in IPVS mode?
If it did not work in IPVS, can you disable it and refresh the instances and try the same setup again?
Please also note that I have confirmed the behaviour on AL2023 and Bottlerocket and I do no know if it work on AL2 or Ubuntu.
Wanted to mention that AWS support replicated the behaviour, so if you want and based on your availability we can jump into a call with them to discuss it.
Thanks

yash97 · 2024-07-16T21:11:51Z

Hi @AhmadMS1988 I did reproduce your issue. Yes it indeed does not work. Here's Why:
When you create pod with security group it will send traffic from eth1 interface. It will also use separate routing table.
You can execute ip rule list to verfiy.
Now for ipvs to work, packet has to pass through kube-ipvs0 interface device. You can see routes for that
do ip route show table all to see all routes on node. So packet from pod with security group does go outside of node. But is will not get DNATed. So at parent eni when packet will arrive it will have dst ip as service ip instead of endpoint ip.

AhmadMS1988 · 2024-07-16T22:13:31Z

Thank you.
So can this be fixed from CNI level?
Or is this a limitation that security groups for pods can't work with ipvs?

orsenthil · 2024-07-16T23:42:07Z

@AhmadMS1988 - Confirming the observation from @yash97 . This seems problematic only with IPVS mode. That is, Pods with SGP only on IPVS does not seem connect to Service IP.
Trying to see if a fix can be made a a CNI level.

orsenthil · 2024-07-16T23:52:49Z

@AhmadMS1988 - We are going to call this out as a limitation for now. We had an known issue with Pods with Security Group not working IPVS mode for sometime now.

github-actions · 2024-09-15T00:04:19Z

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

AhmadMS1988 added needs investigation question labels Jul 6, 2024

github-actions bot added the stale Issue or PR is stale label Sep 15, 2024

orsenthil removed the stale Issue or PR is stale label Sep 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can VPC-CNI with Security Group for Pods work with kube-proxy in IPVS mode? #2982

Can VPC-CNI with Security Group for Pods work with kube-proxy in IPVS mode? #2982

AhmadMS1988 commented Jul 6, 2024

orsenthil commented Jul 10, 2024

yash97 commented Jul 10, 2024

AhmadMS1988 commented Jul 11, 2024 •

edited

Loading

orsenthil commented Jul 12, 2024

AhmadMS1988 commented Jul 13, 2024 •

edited

Loading

yash97 commented Jul 16, 2024 •

edited

Loading

AhmadMS1988 commented Jul 16, 2024

orsenthil commented Jul 16, 2024

orsenthil commented Jul 16, 2024

github-actions bot commented Sep 15, 2024

Can VPC-CNI with Security Group for Pods work with kube-proxy in IPVS mode? #2982

Can VPC-CNI with Security Group for Pods work with kube-proxy in IPVS mode? #2982

Comments

AhmadMS1988 commented Jul 6, 2024

orsenthil commented Jul 10, 2024

yash97 commented Jul 10, 2024

AhmadMS1988 commented Jul 11, 2024 • edited Loading

orsenthil commented Jul 12, 2024

Switched to IPVS Mode.

AhmadMS1988 commented Jul 13, 2024 • edited Loading

yash97 commented Jul 16, 2024 • edited Loading

AhmadMS1988 commented Jul 16, 2024

orsenthil commented Jul 16, 2024

orsenthil commented Jul 16, 2024

github-actions bot commented Sep 15, 2024

AhmadMS1988 commented Jul 11, 2024 •

edited

Loading

AhmadMS1988 commented Jul 13, 2024 •

edited

Loading

yash97 commented Jul 16, 2024 •

edited

Loading