default vpc eip and snat conflict with kube-ovn multus nic #3126

wenwenxiong · 2023-08-09T04:34:40Z

Feature request

使用kube-ovn的默认vpc下的多子网，给pod设置多个子网的网卡。同时想支持指定网卡的 eip地址设置。

Use case

在实际使用过程中，我给pod默认vpc下的默认子网，以及按照kube-ovn多网卡配置，创建一个子网kubeovnnet。创建pod同时使用2个子网的网卡，并且设置其中一个子网的eip，注解如下
annotations:
ovn.kubernetes.io/ip_address: 10.233.64.59
ovn.kubernetes.io/mac_address: 00:00:00:53:6B:B7
ovn.kubernetes.io/default_route: 'true'
kubeovnnet.default.ovn.kubernetes.io/logical_switch: kubeovnnet
kubeovnnet.default.ovn.kubernetes.io/allow_live_migration: 'true'
kubeovnnet.default.ovn.kubernetes.io/ip_address: 172.57.0.19
kubeovnnet.default.ovn.kubernetes.io/mac_address: '00:00:11:53:6B:B7'
ovn.kubernetes.io/eip: 192.168.122.51
在pod的创建过程中，报错如下
I0809 11:41:46.793874 6 pod.go:476] handle update pod default/virt-launcher-vm-u1804-q95rd
I0809 11:41:46.794048 6 pod.go:935] update pod default/virt-launcher-vm-u1804-q95rd
I0809 11:41:46.794068 6 network_attachment.go:66] parsePodNetworkAnnotation: [{"interface":"podfff8ab36656","mac":"00:00:00:53:6B:B6","name":"kubeovnnet","namespace":"default"}], default
W0809 11:41:46.857073 6 ovn-nbctl-legacy.go:53] ovn-nbctl command error: ovn-nbctl --timeout=60 --no-wait --may-exist lr-nat-add ovn-cluster dnat_and_snat 192.168.122.51 10.233.64.59 in 4ms
E0809 11:41:46.857150 6 pod.go:1040] failed to add nat rules, ovn-nbctl: a NAT with this type (dnat_and_snat) and external_ip (192.168.122.51) already exists
, "exit status 1"
E0809 11:41:46.857220 6 pod.go:488] error syncing 'default/virt-launcher-vm-u1804-q95rd': ovn-nbctl: a NAT with this type (dnat_and_snat) and external_ip (192.168.122.51) already exists
, "exit status 1", requeuing
查看源码pkg/controller/pod.go发现handleUpdatePod方法中有以下循环

for _, podNet := range podNets {
if !isOvnSubnet(podNet.Subnet) {
continue
}
.......
if c.config.EnableEipSnat {
for _, ipStr := range strings.Split(podIP, ",") {
if err := c.ovnLegacyClient.UpdateNatRule("dnat_and_snat", ipStr, pod.Annotations[util.EipAnnotation], c.config.ClusterRouter, pod.Annotations[util.MacAddressAnnotation], fmt.Sprintf("%s.%s", podName, pod.Namespace)); err != nil {
klog.Errorf("failed to add nat rules, %v", err)
return err
}

				if err := c.ovnLegacyClient.UpdateNatRule("snat", ipStr, pod.Annotations[util.SnatAnnotation], c.config.ClusterRouter, "", ""); err != nil {
					klog.Errorf("failed to add nat rules, %v", err)
					return err
				}
			}
		}
	}

......
会针对pod每一kube-ovn管理的网路，处理eip和snat规则。因为eip/snat注解名称固定的只能分配一个IP，到解析第二个网路就会报错，因为外部 IP以及被第一个网络使用了。希望扩展eip snat 注解名称，让它与指定的kube-ovn网络绑定。由名称去判断哪个网络做eip 和snat。也支持配置多个网络的eip和snat。

The text was updated successfully, but these errors were encountered:

bobz965 · 2023-08-09T04:41:27Z

你是期望一定要使用 annotation 这种方式来管理多网卡pod的不同vpc subnet 网卡都能使用 eip 或者 snat 么？
或许可以使用下 ovn-fip ovn-snat crd 来控制呢？

wenwenxiong · 2023-08-09T05:00:39Z

ovn-fip 这种怎么配置，文档中没看到这块了

wenwenxiong · 2023-08-09T05:55:58Z

看到了，这个要kube-ovn v1.12版本才支持吧，看到这个版本的doc有文档记录

bobz965 · 2023-08-09T06:44:24Z

看到了，这个要kube-ovn v1.12版本才支持吧，看到这个版本的doc有文档记录

是的

wenwenxiong · 2023-08-11T01:56:10Z

我搭环境如下：
os: ubuntu20.04.6
k8s: v1.24.8
kube-ovn: v1.12.0
参考文档https://kubeovn.github.io/docs/v1.12.x/advance/ovn-eip-fip-snat/ 配置默认vpc的fip功能
root@master1:/home/mep/ovneip# cat eip-network.yaml

准备 provider-network， vlan， subnet

cat 01-provider-network.yaml

apiVersion: kubeovn.io/v1
kind: ProviderNetwork
metadata:
name: eipnet
spec:
defaultInterface: ens10

cat 02-vlan.yaml

apiVersion: kubeovn.io/v1
kind: Vlan
metadata:
name: vlan0
spec:
id: 0
provider: eipnet

cat 03-vlan-subnet.yaml

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
name: eipnet
spec:
protocol: IPv4
cidrBlock: 172.56.0.0/24
gateway: 172.56.0.1
vlan: vlan0
excludeIps:

172.56.0.1..172.56.0.200
root@master1:/home/mep/ovneip#
root@master1:/home/mep/ovneip# cat external-eip-cm.yaml

启用默认 vpc 和上述 underlay 公网 provider subnet 互联

#cat 00-centralized-external-gw-no-ip.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: ovn-external-gw-config
namespace: kube-system
data:
enable-external-gw: "true"
external-gw-nodes: "master1,master2,master3"
type: "centralized"
external-gw-nic: "ens10" # 用于接入 ovs 公网网桥的网卡
external-gw-addr: "172.56.0.1/24" # underlay 物理网关的 ip

root@master1:/home/mep/ovneip#
root@master1:/home/mep/ovneip# cat fip01.yaml

kind: OvnEip
apiVersion: kubeovn.io/v1
metadata:
name: eip-static
spec:
externalSubnet: eipnet
type: nat

kind: OvnFip
apiVersion: kubeovn.io/v1
metadata:
name: eip-static
spec:
ovnEip: eip-static
ipName: vm2-u1804.default # 注意这里是 ip crd 的名字，具有唯一性
root@master1:/home/mep/ovneip#
在kube-ovn-controller 和kube-ovn-cni 加上了启动参数 - --external-gateway-switch=eipnet

发现ip无法通信
kubectl-ko nbctl show 输出master1的信息如下
root@master1:/home/mep/ovneip# kubectl-ko nbctl show
switch 61118865-c5ef-4c8a-9961-54dc9bfc0d32 (join)
port node-master2
addresses: ["00:00:00:0B:97:70 100.64.0.3"]
port join-ovn-cluster
type: router
router-port: ovn-cluster-join
port node-master3
addresses: ["00:00:00:5C:8A:2D 100.64.0.4"]
port node-master1
addresses: ["00:00:00:E4:1A:12 100.64.0.2"]
switch 9d7be2fd-a6ca-4216-a3ec-1334b8946f09 (ovn-default)
port virt-handler-28wzn.ns-mec-computing
addresses: ["00:00:00:1B:64:5D 10.233.64.34"]
port minio-56ff975766-x8zsp.ns-mec-middleware
addresses: ["00:00:00:44:68:5F 10.233.64.41"]
port topolvm-node-4d69k.ns-mec-storage
addresses: ["00:00:00:0B:DE:DB 10.233.64.16"]
port virt-handler-g9cvj.ns-mec-computing
addresses: ["00:00:00:85:53:6F 10.233.64.36"]
port vm2-u1804.default
addresses: ["00:00:00:53:6B:B7 10.233.64.59"]
port virt-operator-6d8dc5bb77-dlwlg.ns-mec-computing
addresses: ["00:00:00:67:3D:8C 10.233.64.25"]
port nginx-errors-784975c95c-p2sph.ns-mec-network
addresses: ["00:00:00:80:D6:D6 10.233.64.20"]
port cdi-operator-6ff7bc58dc-knp9b.ns-mec-computing
addresses: ["00:00:00:D1:67:2B 10.233.64.37"]
port topolvm-lvmd-0-qt9sm.ns-mec-storage
addresses: ["00:00:00:BB:2E:91 10.233.64.13"]
port topolvm-lvmd-0-9vf44.ns-mec-storage
addresses: ["00:00:00:7B:F1:A5 10.233.64.14"]
port virt-api-5bf754c46c-lc5bb.ns-mec-computing
addresses: ["00:00:00:D4:34:87 10.233.64.29"]
port coredns-8bf58c9b8-75qml.kube-system
addresses: ["00:00:00:C1:18:47 10.233.64.44"]
port coredns-8bf58c9b8-d9x44.kube-system
addresses: ["00:00:00:75:0C:C2 10.233.64.43"]
port kube-ovn-pinger-snd9m.kube-system
addresses: ["00:00:00:D4:6C:AA 10.233.64.6"]
port topolvm-controller-69b6f559fc-8nv9s.ns-mec-storage
addresses: ["00:00:00:1B:43:63 10.233.64.15"]
port virt-api-5bf754c46c-4n77h.ns-mec-computing
addresses: ["00:00:00:67:1C:38 10.233.64.28"]
port cdi-deployment-6779f8669b-n59nq.ns-mec-computing
addresses: ["00:00:00:4A:C2:E4 10.233.64.39"]
port cdi-uploadproxy-859dcb8f7f-rtmwx.ns-mec-computing
addresses: ["00:00:00:C7:FE:7E 10.233.64.40"]
port ingress-nginx-controller-pl97t.ns-mec-network
addresses: ["00:00:00:FA:3A:B3 10.233.64.22"]
port ovn-default-ovn-cluster
type: router
router-port: ovn-cluster-ovn-default
port topolvm-lvmd-0-7wsvj.ns-mec-storage
addresses: ["00:00:00:F6:85:ED 10.233.64.11"]
port virt-controller-5b7bf9f4f4-7htnm.ns-mec-computing
addresses: ["00:00:00:AB:76:BD 10.233.64.30"]
port ingress-nginx-controller-t944f.ns-mec-network
addresses: ["00:00:00:B2:E8:1D 10.233.64.21"]
port topolvm-node-xz6rk.ns-mec-storage
addresses: ["00:00:00:05:2C:97 10.233.64.17"]
port kube-ovn-pinger-x5np9.kube-system
addresses: ["00:00:00:EF:37:E7 10.233.64.7"]
port vm-u1804.default
addresses: ["00:00:00:85:0D:DF 10.233.64.45"]
port cdi-apiserver-54d6fb8cbf-pv59n.ns-mec-computing
addresses: ["00:00:00:CD:11:FB 10.233.64.38"]
port virt-exportproxy-658f4b4-6hljj.ns-mec-computing
addresses: ["00:00:00:FA:F2:BD 10.233.64.32"]
port kube-ovn-pinger-5ql7g.kube-system
addresses: ["00:00:00:65:79:B1 10.233.64.8"]
port virt-handler-k9bqn.ns-mec-computing
addresses: ["00:00:00:0A:ED:2A 10.233.64.35"]
port virt-controller-5b7bf9f4f4-c2fxz.ns-mec-computing
addresses: ["00:00:00:0B:67:71 10.233.64.31"]
port topolvm-node-scht5.ns-mec-storage
addresses: ["00:00:00:22:4C:D6 10.233.64.12"]
port virt-operator-6d8dc5bb77-d5x4g.ns-mec-computing
addresses: ["00:00:00:F0:19:B0 10.233.64.24"]
port virt-exportproxy-658f4b4-vxldm.ns-mec-computing
addresses: ["00:00:00:99:58:5C 10.233.64.33"]
port ingress-nginx-controller-sq5ww.ns-mec-network
addresses: ["00:00:00:6E:63:E5 10.233.64.19"]
switch e61363e3-4eba-43fd-84cd-c1a091d86ed0 (kubeovnnet)
port kubeovnnet-ovn-cluster
type: router
router-port: ovn-cluster-kubeovnnet
port vm-u1804.default.kubeovnnet.default.ovn
addresses: ["00:00:00:53:6B:B6 172.57.0.18"]
port vm2-u1804.default.kubeovnnet.default.ovn
addresses: ["00:00:11:53:6B:B7 172.57.0.19"]
switch b5856453-94a1-42f8-ad09-3aea14817467 (eipnet)
port localnet.eipnet
type: localnet
addresses: ["unknown"]
port eipnet-ovn-cluster
type: router
router-port: ovn-cluster-eipnet
router a87c14ee-448f-44d8-acd0-ccb7bc893bb2 (ovn-cluster)
port ovn-cluster-join
mac: "00:00:00:A7:D9:D8"
networks: ["100.64.0.1/16"]
port ovn-cluster-kubeovnnet
mac: "00:00:00:5E:5D:46"
networks: ["172.57.0.1/24"]
port ovn-cluster-eipnet
mac: "00:00:00:14:79:1E"
networks: ["172.56.0.201/24"]
gateway chassis: [5daf786c-21a0-4770-9f58-db2b4f4e6ac9 c77d5a37-ff3e-44c7-988e-a13dde0832da 2544bbb4-d1fe-41d1-ad25-6344769c7a2d]
port ovn-cluster-ovn-default
mac: "00:00:00:99:67:FE"
networks: ["10.233.64.1/18"]
nat 67608024-8378-4bf9-bd05-521836752051
external ip: "172.56.0.201"
logical ip: "172.57.0.19"
type: "dnat_and_snat"
nat 9fc22f30-5d88-48ce-bdf7-95c04ccf0012
external ip: "172.56.0.202"
logical ip: "10.233.64.59"
type: "dnat_and_snat"
root@master1:/home/mep/ovneip#
root@master1:/home/mep/ovneip# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
virt-launcher-vm-u1804-qll8b 1/1 Running 0 9m48s 10.233.64.45 master2 1/1
virt-launcher-vm2-u1804-mk9dg 1/1 Running 0 69m 10.233.64.59 master3 1/1
root@master1:/home/mep/ovneip#
root@master1:/home/mep/ovneip# kubectl get vmi -o wide
NAME AGE PHASE IP NODENAME READY LIVE-MIGRATABLE PAUSED
vm-u1804 9m53s Running 172.57.0.18 master2 True False
vm2-u1804 73m Running 10.233.64.59 master3 True False
root@master1:/home/mep/ovneip#
root@master1:/home/mep/ovneip# ping 10.233.64.59
PING 10.233.64.59 (10.233.64.59) 56(84) bytes of data.
64 bytes from 10.233.64.59: icmp_seq=1 ttl=63 time=2.91 ms
^C
--- 10.233.64.59 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.912/2.912/2.912/0.000 ms
root@master1:/home/mep/ovneip# ping 172.56.0.1
PING 172.56.0.1 (172.56.0.1) 56(84) bytes of data.
64 bytes from 172.56.0.1: icmp_seq=1 ttl=64 time=0.864 ms

64 bytes from 172.56.0.1: icmp_seq=2 ttl=64 time=0.283 ms
^C
--- 172.56.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1033ms
rtt min/avg/max/mdev = 0.283/0.573/0.864/0.290 ms
root@master1:/home/mep/ovneip# ping 172.56.0.202
PING 172.56.0.202 (172.56.0.202) 56(84) bytes of data.
From 172.56.0.11 icmp_seq=1 Destination Host Unreachable
From 172.56.0.11 icmp_seq=2 Destination Host Unreachable
From 172.56.0.11 icmp_seq=3 Destination Host Unreachable
^C
--- 172.56.0.202 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3059ms
pipe 4
root@master1:/home/mep/ovneip#

在master1上有网卡ens10作为172.56.0.0/24网段的ip 172.56.0.11，能访问gateway 172.56.0.1 和pod ip 10.233.64.59 ，看ovn的nbctl信息中 dnat_and_snat规则也创建了，但是ping 172.56.0.202 不通。
还有个问题，就是删除ofip了，ovn中 dnat_and_snat规则还是存在的。麻烦帮忙看一下，谢谢了！

bobz965 · 2023-08-14T10:08:23Z

ovn nat 残留问题基于该PR修复: #3139

在默认vpc下，我没有复现ovn-fip 创建后不通的情况，而且环境需要准备两张网卡，一张给vpc网络，另一张给provider-network。建议再对照下文档。

已回合 release1.12

wenwenxiong · 2023-08-15T00:39:29Z

我是2张网卡了，在三台kvm虚拟机上。宿主机上创建了2个nat网络 192.168.122.0/24,和172.56.0.0/24，给3台虚拟机每台2张网卡，分别属于2个nat网络。这样的环境有问题吗？

你先测试下单张网卡配 ovn-fip 是否能通，或者主网卡配 ovn-fip是否能通，一张网卡单独测试是否能通

wenwenxiong · 2023-08-15T08:02:03Z

我这环境的网络表现的越来越奇怪了，对默认vpc下的pod IP设置ovn fip，只有pod所在节点才能ping通 ovn fip，其他节点不能ping通。tcpdump抓icmp包，有request和reply 报文，但是ping命令就是卡住，不显示

bobz965 · 2023-08-15T09:05:16Z

我这环境的网络表现的越来越奇怪了，对默认vpc下的pod IP设置ovn fip，只有pod所在节点才能ping通 ovn fip，其他节点不能ping通。tcpdump抓icmp包，有request和reply 报文，但是ping命令就是卡住，不显示

你可以先测试下自定义vpc下的单网卡 pod 的 ovn fip 跨节点是否可以 ping 通

github-actions · 2023-11-01T00:00:59Z

Issues go stale after 60d of inactivity. Please comment or re-open the issue if you are still interested in getting this issue fixed.

github-actions bot added the no-issue-activity label Nov 1, 2023

bobz965 closed this as completed Nov 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

default vpc eip and snat conflict with kube-ovn multus nic #3126

default vpc eip and snat conflict with kube-ovn multus nic #3126

wenwenxiong commented Aug 9, 2023 •

edited

Loading

bobz965 commented Aug 9, 2023

wenwenxiong commented Aug 9, 2023

wenwenxiong commented Aug 9, 2023

bobz965 commented Aug 9, 2023

wenwenxiong commented Aug 11, 2023

bobz965 commented Aug 14, 2023 •

edited

Loading

wenwenxiong commented Aug 15, 2023 •

edited by bobz965

Loading

wenwenxiong commented Aug 15, 2023

bobz965 commented Aug 15, 2023

github-actions bot commented Nov 1, 2023

default vpc eip and snat conflict with kube-ovn multus nic #3126

default vpc eip and snat conflict with kube-ovn multus nic #3126

Comments

wenwenxiong commented Aug 9, 2023 • edited Loading

Feature request

Use case

bobz965 commented Aug 9, 2023

wenwenxiong commented Aug 9, 2023

wenwenxiong commented Aug 9, 2023

bobz965 commented Aug 9, 2023

wenwenxiong commented Aug 11, 2023

准备 provider-network， vlan， subnet

cat 01-provider-network.yaml

cat 02-vlan.yaml

cat 03-vlan-subnet.yaml

启用默认 vpc 和上述 underlay 公网 provider subnet 互联

#cat 00-centralized-external-gw-no-ip.yaml

root@master1:/home/mep/ovneip# root@master1:/home/mep/ovneip# cat fip01.yaml

bobz965 commented Aug 14, 2023 • edited Loading

wenwenxiong commented Aug 15, 2023 • edited by bobz965 Loading

wenwenxiong commented Aug 15, 2023

bobz965 commented Aug 15, 2023

github-actions bot commented Nov 1, 2023

wenwenxiong commented Aug 9, 2023 •

edited

Loading

root@master1:/home/mep/ovneip#
root@master1:/home/mep/ovneip# cat fip01.yaml

bobz965 commented Aug 14, 2023 •

edited

Loading

wenwenxiong commented Aug 15, 2023 •

edited by bobz965

Loading