chore(bgp): documentation on new bgp nat gw feature (#186)

* chore(bgp): documentation on new bgp nat gw feature * fix(doc): linter * Update with-bgp.en.md
kubeovn · Aug 12, 2024 · 4af6094 · 4af6094
1 parent bbcc961
commit 4af6094
Showing 1 changed file with 108 additions and 7 deletions.
diff --git a/docs/advance/with-bgp.en.md b/docs/advance/with-bgp.en.md
@@ -1,12 +1,14 @@
 # BGP Support
 
-Kube-OVN supports broadcasting the IP address of Pods/Subnets to the outside world via the BGP protocol,
-so that the outside world can access the Pods directly through their Pod IP.
-To use this feature, you need to install `kube-ovn-speaker` on specific nodes and
-add the corresponding annotation to the Pod or Subnet that needs to be exposed to the outside world.
+Kube-OVN supports broadcasting the IP address of Pods/Subnets/Services/EIPs to the outside world via the BGP protocol.  
 
+To use this feature on Pods/Subnets/Services, you need to install `kube-ovn-speaker` on specific (or all) nodes and
+add the corresponding annotation to the Pod or Subnet that needs to be exposed to the outside world.  
 Kube-OVN also supports broadcasting the IP address of services of type `ClusterIP` via the same annotation.
 
+To use this feature on EIPs, you need to create your NAT Gateway with special parameters to enable the BGP speaker sidecar.
+See [Publishing EIPs](#publishing-eips) for more information.
+
 ## Installing `kube-ovn-speaker`
 
 `kube-ovn-speaker` uses [GoBGP](https://osrg.github.io/gobgp/) to publish routing information to the outside world and to
@@ -48,7 +50,7 @@ kubectl apply -f speaker.yaml
 
 ## Publishing Pod/Subnet Routes
 
-To use BGP for external routing, first set `natOutgoing` to `false` for the corresponding Subnet to allow the Pod IP to enter the underlying network directly.
+To use BGP for external routing on subnets, first set `natOutgoing` to `false` for the corresponding Subnet to allow the Pod IP to enter the underlying network directly.
 
 Add annotation to publish routes:
 
@@ -64,6 +66,8 @@ kubectl annotate pod sample ovn.kubernetes.io/bgp-
 kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp-
 ```
 
+See [Announcement Policies](#announcement-policies) for the announcement behaviour depending on the policy set in the annotation.
+
 ## Publishing Services of type `ClusterIP`
 
 To announce the ClusterIP of services to the outside world, the `kube-ovn-speaker` option `announce-cluster-ip` needs to be set to `true`.
@@ -81,20 +85,117 @@ Delete annotation to disable the publishing:
 kubectl annotate service sample ovn.kubernetes.io/bgp-
 ```
 
+The speakers will all start announcing the `ClusterIP` of that service to the outside world.
+
+## Publishing EIPs
+
+EIPs can be announced by the NAT gateways to which they are attached. When BGP is enabled on a `VpcNatGateway`  
+a new BGP speaker sidecar gets injected to it.  
+
+To add BGP capabilities to NAT gateways, we first need to create a new `NetworkAttachmentDefinition` that can be  
+attached to our BGP speaker sidecars. This NAD will reference a provider shared by a `Subnet` in the default VPC (in which the Kubernetes API is running).  
+This will enable the sidecar to reach the K8S API, automatically detecting new EIPs added to the gateway. This operation only needs to be done once.  All the NAT gateways will use this provider from now on. This is the same principle used for the CoreDNS in a custom VPC, which means you can reuse that NAD if you've already done that setup before.
+
+Create a `NetworkAttachmentDefintion` and a `Subnet` with the same `provider`.
+The name of the provider needs to be of the form `nadName.nadNamespace.ovn`:
+
+```yaml
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: api-ovn-nad
+  namespace: default
+spec:
+  config: '{
+      "cniVersion": "0.3.0",
+      "type": "kube-ovn",
+      "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
+      "provider": "api-ovn-nad.default.ovn"
+    }'
+---
+apiVersion: kubeovn.io/v1
+kind: Subnet
+metadata:
+  name: vpc-apiserver-subnet
+spec:
+  protocol: IPv4
+  cidrBlock: 100.100.100.0/24
+  provider: api-ovn-nad.default.ovn
+```
+
+The `ovn-vpc-nat-config` needs to be modified to reference our new provider and the image used by the BGP speaker:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ovn-vpc-nat-config
+  namespace: kube-system
+data:
+  apiNadProvider: api-ovn-nad.default.ovn              # What NetworkAttachmentDefinition provider to use so that the sidecar
+                                                       # can access the K8S API, as it can't by default due to VPC segmentation
+  bgpSpeakerImage: docker.io/kubeovn/kube-ovn:v1.13.0  # Sets the BGP speaker image used
+  image: docker.io/kubeovn/vpc-nat-gateway:v1.13.0
+```
+
+We need to make sure the default Subnet `ovn-default` uses the same NAD so both Subnets can see each others.  
+Edit the Subnet and add a `provider` field with your custom provider:
+
+```yaml
+provider: api-ovn-nad.default.ovn
+```
+
+The NAT gateway(s) now needs to be created with BGP enabled so that the speaker sidecar gets created along it:
+
+```yaml
+kind: VpcNatGateway
+apiVersion: kubeovn.io/v1
+metadata:
+  name: vpc-natgw
+spec:
+  vpc: vpc1
+  subnet: net1
+  lanIp: 10.0.1.10
+  bgpSpeaker:
+    enabled: true
+    asn: 65500
+    remoteAsn: 65000
+    neighbors:
+      - 100.127.4.161
+      - fd:01::1
+    enableGracefulRestart: true # Optional
+    routerId: 1.1.1.1           # Optional
+    holdTime: 1m                # Optional
+    password: "password123"     # Optional
+    extraArgs:                  # Optional, passed directly to the BGP speaker
+      - -v5                     # Enables verbose debugging of the BGP speaker sidecar
+  selector:
+    - "kubernetes.io/os: linux"
+  externalSubnets:
+  - ovn-vpc-external-network # Network on which we'll speak BGP and receive/send traffic to the outside world
+                             # BGP neighbors need to be on that network
+```
+
+This gateway is now capable of announcing any EIP that gets attached to it as long as it has the BGP annotation:
+
+```yaml
+kubectl annotate eip sample ovn.kubernetes.io/bgp=true
+```
+
 ## Announcement policies
 
 There are 2 policies used by `kube-ovn-speaker` to announce the routes:
 
 - **Cluster**: this policy makes the Pod IPs/Subnet CIDRs be announced from every speaker, whether there's Pods
 that have that specific IP or that are part of the Subnet CIDR on that node. In other words, traffic may enter from
 any node hosting a speaker, and then be internally routed in the cluster to the actual Pod. In this configuration
-extra hops might be used. This is the default policy to Pods and Subnets.
+extra hops might be used. This is the default policy for Pods and Subnets.
 - **Local**: this policy makes the Pod IPs be announced only from speakers on nodes that are actively hosting
 them. In other words, traffic will only enter from the node hosting the Pod marked as needing BGP advertisement,
 or from the node hosting a Pod with an IP belonging to a Subnet marked as needing BGP advertisement.
 This makes the network path shorter as external traffic arrives directly to the physical host of the Pod.
 
-NOTE: You'll probably need to run `kube-ovn-speaker` on every node for the`Local` policy to work.
+**NOTE**: You'll probably need to run `kube-ovn-speaker` on every node for the`Local` policy to work.
 If a Pod you're trying to announce lands on a node with no speaker on it, its IP will simply not be announced.
 
 The default policy used is `Cluster`. Policies can be overridden for each Pod/Subnet using the `ovn.kubernetes.io/bgp` annotation: