创建基于ECMP BFD的分布式网关-按照模板创建无法使用。

[zhoulw]

创建完underlay公网网络、vpc、subnet之后，

[root@hci-ctl-1 installAndUpdate]# kubectl get vpcs | grep vpc
NAME          ENABLEEXTERNAL   ENABLEBFD   STANDBY   SUBNETS                             NAMESPACES
vpc2          true             true        true      ["vpc2-subnet1"]                    ["vpc2"]

[root@hci-ctl-1 installAndUpdate]# k ko nbctl list bfd
_uuid               : ddbad301-1784-45de-80de-7c754d014bdf
detect_mult         : 3
dst_ip              : "10.120.31.12"
external_ids        : {}
logical_port        : vpc2-external
min_rx              : 100
min_tx              : 100
options             : {}
status              : admin_down

·················

vpc的静态路在配置项中是有的：

spec:
  enableBfd: true
  enableExternal: true
  namespaces:
  - vpc2
  staticRoutes:
  - bfdId: ddbad301-1784-45de-80de-7c754d014bdf
    cidr: 10.155.165.0/24
    ecmpMode: ecmp-symmetric-reply
    nextHopIP: 10.120.31.12
    policy: policySrc
    routeTable: ""
  - bfdId: 4cb0e1e7-cd8c-4f19-8f22-2ee3c892bcbf
    cidr: 10.155.165.0/24
    ecmpMode: ecmp-symmetric-reply
    nextHopIP: 10.120.31.13
    policy: policySrc
    routeTable: ""
  - bfdId: 1a939a03-2200-4c93-917d-5a41a30c1b78
    cidr: 10.155.165.0/24
    ecmpMode: ecmp-symmetric-reply
    nextHopIP: 10.120.31.14
    policy: policySrc
    routeTable: ""

但是没有配置到ovs中：

[root@hci-ctl-1 installAndUpdate]# k ko nbctl lr-route-list vpc2
没有值

目前已有的而报错是kube-ovn-cni中：无法get bfd status，没有创建起来ovnext0 这个port。

0802 17:51:17.119517   36409 ovs_linux.go:369] network 10.120.31.13/24 with gateway 10.120.31.254 is ready for interface ovnext0 after 1 checks
E0802 17:51:17.124709   36409 ovs_linux.go:463] failed to get bfdd status, exit status 1
I0802 17:51:17.124771   36409 ovs_linux.go:696] setup nic ovnext0 ip 10.120.31.13/24, mac 00:00:00:5b:ee:fd, mtu 1400
I0802 17:51:17.137682   36409 init.go:125] config provider nic enp3s0f1 on bridge br-external
I0802 17:51:17.206672   36409 ovs_linux.go:837] delete ip address 2010:120:31:1:200:ff:fe5b:eefd/64 on ovnext0
I0802 17:51:17.227478   36409 ovs_linux.go:368] MAC addresses of gateway 10.120.31.254 is dc:ef:80:5a:44:12
I0802 17:51:17.227511   36409 ovs_linux.go:369] network 10.120.31.13/24 with gateway 10.120.31.254 is ready for interface ovnext0 after 1 checks

然后就没有其他地方报错了，日志一片祥和，好迷茫，请大佬看一下这个是什么问题。

[bobz965]

我抽空看下

[zhoulw]

版本是前几天天出的版本，没有加上你最近的代码”Ovn nat 1“的这个commit。underlay公网的vlan id 0，name是external。

[root@hci-ctl-1 ecmp-bfd]# kubectl get subnets.kubeovn.io 
NAME           PROVIDER   VPC           PROTOCOL   CIDR              PRIVATE   NAT     DEFAULT   GATEWAYTYPE   V4USED   V4AVAILABLE   V6USED   V6AVAILABLE   EXCLUDEIPS                                                     U2OINTERCONNECTIONIP
external       ovn        ovn-cluster   IPv4       10.120.31.0/24    false     false   false     distributed   0        90            0        0             ["10.120.31.1..10.120.31.10","10.120.31.101..10.120.31.254"] 

[root@hci-ctl-1 ecmp-bfd]# kubectl get provider-networks.kubeovn.io 
NAME       DEFAULTINTERFACE   READY
external   enp3s0f1           true

[bobz965]

版本是前几天天出的版本，没有加上你最近的代码”Ovn nat 1“的这个commit。underlay公网的vlan id 0，name是external。

[root@hci-ctl-1 ecmp-bfd]# kubectl get subnets.kubeovn.io 
NAME           PROVIDER   VPC           PROTOCOL   CIDR              PRIVATE   NAT     DEFAULT   GATEWAYTYPE   V4USED   V4AVAILABLE   V6USED   V6AVAILABLE   EXCLUDEIPS                                                     U2OINTERCONNECTIONIP
external       ovn        ovn-cluster   IPv4       10.120.31.0/24    false     false   false     distributed   0        90            0        0             ["10.120.31.1..10.120.31.10","10.120.31.101..10.120.31.254"] 

[root@hci-ctl-1 ecmp-bfd]# kubectl get provider-networks.kubeovn.io 
NAME       DEFAULTINTERFACE   READY
external   enp3s0f1           true

最近还有一些小优化，我再看看

[bobz965]

子网需要启用ecmp

[zhoulw]

我的vpc下的子网是：
[root@hci-ctl-1 ecmp-bfd]#kubectl get subnets.kubeovn.io vpc2-subnet1 -oyaml

spec:
  cidrBlock: 10.155.165.0/24
  default: false
  disableInterConnection: true
  enableEcmp: true
  enableLb: true
  excludeIps:
  - 10.155.165.1
  gateway: 10.155.165.1
  gatewayNode: ""
  gatewayType: distributed
  namespaces:
  - vpc2
  natOutgoing: false
  private: false
  protocol: IPv4
  provider: ovn
  vpc: vpc2

有启用ECMP。

[zhoulw]

请问一下我的br-int 如下：

 Bridge br-int
        fail_mode: secure
        Port "e7f39e1a652d_h"
            Interface "e7f39e1a652d_h"
        Port "d6e7bf49c802_h"
            Interface "d6e7bf49c802_h"
        Port "ovn0"
            Interface "ovn0"
                type: internal
        Port patch-br-int-to-localnet.external
            Interface patch-br-int-to-localnet.external
                type: patch
                options: {peer=patch-localnet.external-to-br-int}
        Port br-int
            Interface br-int
                type: internal
        Port "mirror0"
            Interface "mirror0"
                type: internal
        Port "ovn-3832c9-0"
            Interface "ovn-3832c9-0"
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.120.12.103"}
        Port "ovnext0"
            Interface "ovnext0"
                type: internal
        Port "ovn-0682bf-0"
            Interface "ovn-0682bf-0"
                type: geneve
                options: {csum="true", key=flow, remote_ip="10.120.12.102"}

请问一下，ovnext0是br-int的internal port吗？😢😢这个地方没太懂。

[bobz965]

等这个 PR 合入 #3107

[zhoulw]

Thanks a lot!!!

[bobz965]

Thanks a lot!!!

文档也已更新提交: https://github.com/kubeovn/docs/pull/111/files

[zhoulw]

多谢，目前已可以使用基于ECMP&BFD的分布式网关。