custom vpc pod support tcp http probe with tproxy method #3024

changluyi · 2023-07-10T03:52:34Z

Make sure you have followed Kube-OVN Code Style.

What type of this PR

Examples of user facing changes:

Features
Bug fixes
Docs
Tests

Which issue(s) this PR fixes:

Fixes #2869

TPROXY 引流路线：

kubelet发出探测TCP或者HTTP报文

1）到OVN-OUTPUT链mangle表

-A OVN-OUTPUT -d 10.0.1.3/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003

2）然后策略路由到达10001 table

32765:  from all fwmark 0x90003/0x90003 lookup 10001

ip route show table 10001
local default dev lo scope host

3）把流量引入到lo 口，从而让将流量重新走OVN-PREROUTING链 mangle表，因为-j tproxy动作只能适用在prerouting链，所以不在OUTPUT 直接把流量发给tproxy，而是得这么绕一圈

-A OVN-PREROUTING -d 10.0.1.3/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.4 --tproxy-mark 0x90004/0x90004

4）策略路由

32764:  from all fwmark 0x90004/0x90004 lookup 10001
local default dev lo scope host

到达tproxy后HTTP探测和TCP probe有区别

http 探测，将报文直接经由lo 口发给 pod ns内得lo口，
tcp探测，由于tcp探测只做三次握手，所以探测报文只要能到达tproxy和tproxy的端口做了三次握手, tcpprobe就能成功，所以这里想了个办法，使用tproxy 去探测pod端口是否成功，如果不成功，删除kubelet到tproxy得引流规则。

另外这个方案有个限制：

不同VPC 同样POD IP的POD不能在同一个节点，因为TPROXY根据流量来决定是否代发给POD ns，所以没有办法去区分同样IP情况下报文属于哪个VPC

WHAT

`🤖 Generated by Copilot at 95c2a80`

This pull request adds a new feature to enable transparent proxying of TCP traffic from kubelet to custom VPC pods using iptables and a tproxy listener. It also improves the CNI server's port binding logic and adds a new test case for pod probes in a custom VPC. It modifies several files in the cmd, pkg, test, and go.mod directories to implement and test this feature.

`🤖 Generated by Copilot at 95c2a80`

We are the masters of the tproxy
We bind the ports and route the packets
We use the iptables and the marks
We probe the pods with nginx

HOW

`🤖 Generated by Copilot at 95c2a80`

Add transparent proxying functionality for custom VPC pod liveness and readiness probes (link,link,link,link,link,link,link,link,link,link,link,link,link,link,link)

Add a new configuration option EnableTProxy to the daemon package in cmd/daemon/cniserver.go and pkg/daemon/config.go (link,link,link,link)
Add a new dependency on the github.com/Asphaltt/go-tproxy package in go.mod (link)
Add a new package tproxy in pkg/tproxy/tproxy_tcp.go that provides a Listener type and a Conn type for transparent proxying of TCP connections (link)
Add a new function GetDefaultListenPort to the daemon package in pkg/daemon/listen.go that returns the address to listen on for the CNI server based on the environment variables (link,link)
Add a new function GetInterfacePodNs to the ovs package in pkg/ovs/ovs-vsctl.go that returns the pod network namespace associated with an interface (link,link)
Add new constants to the daemon package in pkg/daemon/gateway_linux.go and the util package in pkg/util/const.go that represent the iptables marks, masks, tables, chains, and ports used for transparent proxying (link,link,link)
Add a new function reconcileTProxyIPTableRules to the daemon package in pkg/daemon/gateway_linux.go that updates the iptables rules for transparent proxying according to the pod IP, host IP, and probe ports (link)
Add a new helper function formatProbePorts to the daemon package in pkg/daemon/gateway_linux.go that deduplicates and sorts the probe ports of a pod (link)
Add a call to the reconcileTProxyIPTableRules function to the setIptables function of the Controller type in pkg/daemon/gateway_linux.go (link)
Add a conditional block to the Run function of the Controller type in pkg/daemon/controller.go that starts three goroutines to handle the transparent proxying logic if the EnableTProxy configuration option is true, or calls the cleanTProxyConfig function to remove any existing iptables rules for transparent proxying if the option is false (link)

Add a new test case to the pod package in test/e2e/kube-ovn/pod/pod.go that tests the transparent proxying functionality for custom VPC pod liveness and readiness probes (link,link,link,link,link)
- Add new imports to the pod package in test/e2e/kube-ovn/pod/pod.go that provide core Kubernetes API types, integer and string utilities, and time-related functionality (link)
- Add new variables to the pod package in test/e2e/kube-ovn/pod/pod.go that hold a VpcClient object and a custom VPC name (link)
- Assign a value to the vpcClient variable in the pod package in test/e2e/kube-ovn/pod/pod.go by calling the VpcClient method of the framework type (link)
- Add a conditional block to the pod package in test/e2e/kube-ovn/pod/pod.go that deletes the custom VPC with the vpcName variable by calling the DeleteSync method of the vpcClient object (link)
- Add a new test case to the pod package in test/e2e/kube-ovn/pod/pod.go that creates a custom VPC and a subnet in that VPC, and then creates four pods with different combinations of HTTP and TCP liveness and readiness probes, checks the pod readiness status, and deletes the pods, the subnet, and the VPC after testing (link)
- Add a new constant to the framework package in test/e2e/framework/image.go that represents the image name of the nginx:alpine container image, which is used for testing pod liveness and readiness probes (link)
- Add a new function to the PodClient type in the framework package in test/e2e/framework/pod.go that returns a pointer to a corev1.Pod object by calling the PodInterface (link)
Replace the hard-coded address "0.0.0.0" with a call to the GetDefaultListenPort function in cmd/daemon/cniserver.go to optionally bind to the local pod IP if the environment variable ENABLE_BIND_LOCAL_IP is set to true (link)
Add a new dependency on the k8s.io/klog package in go.mod to provide a logging framework for Kubernetes components (link)

oilbeater · 2023-07-12T07:43:38Z

charts/values.yaml

@@ -61,6 +61,7 @@ func:
  LOGICAL_GATEWAY: false
  ENABLE_BIND_LOCAL_IP: true
  U2O_INTERCONNECTION: false
+  ENABLE_TPROXY: true


默认需要关闭，避免未知问题影响其他功能

oilbeater · 2023-07-12T07:47:49Z

dist/images/install.sh

@@ -32,6 +32,7 @@ IFACE=${IFACE:-}
 # Note that the dpdk tunnel iface and tunnel ip cidr should be diffierent with Kubernetes api cidr, otherwise the route will be a problem.
 DPDK_TUNNEL_IFACE=${DPDK_TUNNEL_IFACE:-br-phy}
 ENABLE_BIND_LOCAL_IP=${ENABLE_BIND_LOCAL_IP:-true}
+ENABLE_TPROXY=${ENABLE_TPROXY:-true}


oilbeater · 2023-07-12T07:47:58Z

pkg/daemon/config.go

@@ -100,6 +101,7 @@ func ParseFlags() *Configuration {
 		argEnableVerboseConnCheck    = pflag.Bool("enable-verbose-conn-check", false, "enable TCP/UDP connectivity check listen port")
 		argTCPConnectivityCheckPort  = pflag.Int("tcp-conn-check-port", 8100, "TCP connectivity Check Port")
 		argUDPConnectivityCheckPort  = pflag.Int("udp-conn-check-port", 8101, "UDP connectivity Check Port")
+		argEnableTProxy              = pflag.Bool("enable-tproxy", true, "enable tproxy for vpc pod liveness or readiness probe")


oilbeater · 2023-07-12T08:05:10Z

pkg/tproxy/tproxy_tcp_linux.go

代码参考哪里要写明来源

oilbeater · 2023-07-12T08:10:38Z

pkg/ovs/ovs-vsctl.go

+	}
+
+	podNetNs := ""
+	re := regexp.MustCompile(`pod_netns="([^"]+)"`)


可以用一个全局变量，避免频繁初始化 re 对象

oilbeater · 2023-07-12T08:27:11Z

pkg/daemon/tproxy_linux.go

+	}
+
+	_ = ns.WithNetNSPath(podNS.Path(), func(_ ns.NetNS) error {
+		// Packet 's src and dst IP are both PodIP in netns


Suggested change

// Packet 's src and dst IP are both PodIP in netns

// Packet's src and dst IP are both PodIP in netns

test/e2e/kube-ovn/pod/pod.go

dist/images/uninstall.sh

pkg/daemon/tproxy_linux.go

pkg/daemon/gateway_linux.go

Makefile

pkg/util/net.go

pkg/daemon/config.go

pkg/daemon/gateway_linux.go

pkg/daemon/tproxy_linux.go

changluyi force-pushed the custom_vpc_pod_probe_with_tproxy branch 5 times, most recently from 38184a8 to b8eae39 Compare July 10, 2023 07:19

changluyi added 8 commits July 11, 2023 20:31

custom vpc pod support tcp http probe with tproxy method

d139877

fix golang lint

fbfb30c

remove unused import

8420e13

fix security

b7d99eb

fix e2e err

d18177b

fix e2e

7871098

remove unused map

6857648

refactor

69d0725

changluyi force-pushed the custom_vpc_pod_probe_with_tproxy branch from 05e8aa5 to 69d0725 Compare July 11, 2023 12:36

changluyi added 7 commits July 11, 2023 22:47

fix

488447a

fix

6ca6f0e

move func

99c7f85

remove no need code

fc71cb5

refactor

dcfe86b

conflict with submariner table id 100

4d55af3

add install param

d452b40

changluyi requested a review from oilbeater July 12, 2023 07:04

oilbeater requested changes Jul 12, 2023

View reviewed changes

refactor comment

7db36d1

zhangzujian reviewed Jul 13, 2023

View reviewed changes

test/e2e/kube-ovn/pod/pod.go Outdated Show resolved Hide resolved

zhangzujian reviewed Jul 13, 2023

View reviewed changes

test/e2e/kube-ovn/pod/pod.go Outdated Show resolved Hide resolved

zhangzujian reviewed Jul 13, 2023

View reviewed changes

dist/images/uninstall.sh Show resolved Hide resolved

zhangzujian reviewed Jul 13, 2023

View reviewed changes

pkg/daemon/tproxy_linux.go Outdated Show resolved Hide resolved

zhangzujian reviewed Jul 13, 2023

View reviewed changes

pkg/daemon/tproxy_linux.go Outdated Show resolved Hide resolved

zhangzujian reviewed Jul 13, 2023

View reviewed changes

pkg/daemon/gateway_linux.go Outdated Show resolved Hide resolved

changluyi added 2 commits July 13, 2023 15:18

refactor

c5ac116

fix

704d4ad

zhangzujian reviewed Jul 13, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

zhangzujian reviewed Jul 13, 2023

View reviewed changes

pkg/util/net.go Outdated Show resolved Hide resolved

changluyi added 2 commits July 14, 2023 09:16

fix

659e78c

fix

b9c3618