-
Notifications
You must be signed in to change notification settings - Fork 36
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
refactor ovn ipsec and remove old ipsec.sh (#187)
Signed-off-by: clyi <[email protected]>
- Loading branch information
Showing
2 changed files
with
11 additions
and
176 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,94 +1,12 @@ | ||
| # Encrypt inter-node communication using IPsec | ||
|
|
||
| This function is supported after v1.10.11 and v1.11.4, the kernel version is at least 3.10.0 or above, and UDP ports 500 and 4500 are available. | ||
| # Use IPsec to encrypt communication between nodes | ||
|
|
||
| ## Start IPsec | ||
| This function is supported from v1.13.0 onwards, and the host UDP 500 and 4500 ports need to be available. | ||
|
|
||
| Copy the script from the Kube-OVN source code [ipsec.sh](https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/ipsec.sh), execute the command as follows, the script will call ovs-pki to generate and distribute the certificate required for encryption: | ||
| ## Encryption process | ||
|
|
||
| ```bash | ||
| bash ipsec.sh init | ||
| ``` | ||
| kube-ovn-cni is responsible for applying for certificates and will create a certificate signing request to kube-ovn-controller. kube-ovn-controller will automatically approve the certificate application, and then kube-ovn-cni will generate an ipsec configuration file based on the certificate and finally start the ipsec process. | ||
|
|
||
| After the execution is completed, the nodes will negotiate for a period of time to establish an IPsec tunnel. The experience value is between ten seconds and one minute.You can check the IPsec status with the following command: | ||
| ## Configure IPsec | ||
|
|
||
| ```bash | ||
| # bash ipsec.sh status | ||
| Pod {ovs-ovn-d7hdt} ipsec status... | ||
| Interface name: ovn-a4718e-0 v1 (CONFIGURED) | ||
| Tunnel Type: geneve | ||
| Local IP: 172.18.0.2 | ||
| Remote IP: 172.18.0.4 | ||
| Address Family: IPv4 | ||
| SKB mark: None | ||
| Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem | ||
| Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d | ||
| Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem | ||
| Remote cert: None | ||
| Remote name: a4718e55-5b85-4f46-90e6-63527d080590 | ||
| CA cert: /etc/ipsec.d/cacerts/cacert.pem | ||
| PSK: None | ||
| Custom Options: {} | ||
| Ofport: 2 | ||
| CFM state: Disabled | ||
| Kernel policies installed: | ||
| src 172.18.0.2/32 dst 172.18.0.4/32 proto udp sport 6081 | ||
| src 172.18.0.2/32 dst 172.18.0.4/32 proto udp sport 6081 | ||
| src 172.18.0.2/32 dst 172.18.0.4/32 proto udp dport 6081 | ||
| src 172.18.0.2/32 dst 172.18.0.4/32 proto udp dport 6081 | ||
| Kernel security associations installed: | ||
| sel src 172.18.0.2/32 dst 172.18.0.4/32 proto udp sport 6081 | ||
| sel src 172.18.0.4/32 dst 172.18.0.2/32 proto udp dport 6081 | ||
| sel src 172.18.0.2/32 dst 172.18.0.4/32 proto udp dport 6081 | ||
| sel src 172.18.0.4/32 dst 172.18.0.2/32 proto udp sport 6081 | ||
| IPsec connections that are active: | ||
|
|
||
| Pod {ovs-ovn-fvbbj} ipsec status... | ||
| Interface name: ovn-8aebd9-0 v1 (CONFIGURED) | ||
| Tunnel Type: geneve | ||
| Local IP: 172.18.0.4 | ||
| Remote IP: 172.18.0.2 | ||
| Address Family: IPv4 | ||
| SKB mark: None | ||
| Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem | ||
| Local name: a4718e55-5b85-4f46-90e6-63527d080590 | ||
| Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem | ||
| Remote cert: None | ||
| Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d | ||
| CA cert: /etc/ipsec.d/cacerts/cacert.pem | ||
| PSK: None | ||
| Custom Options: {} | ||
| Ofport: 1 | ||
| CFM state: Disabled | ||
| Kernel policies installed: | ||
| src 172.18.0.4/32 dst 172.18.0.2/32 proto udp dport 6081 | ||
| src 172.18.0.4/32 dst 172.18.0.2/32 proto udp dport 6081 | ||
| src 172.18.0.4/32 dst 172.18.0.2/32 proto udp sport 6081 | ||
| src 172.18.0.4/32 dst 172.18.0.2/32 proto udp sport 6081 | ||
| Kernel security associations installed: | ||
| sel src 172.18.0.4/32 dst 172.18.0.2/32 proto udp dport 6081 | ||
| sel src 172.18.0.2/32 dst 172.18.0.4/32 proto udp sport 6081 | ||
| sel src 172.18.0.4/32 dst 172.18.0.2/32 proto udp sport 6081 | ||
| sel src 172.18.0.2/32 dst 172.18.0.4/32 proto udp dport 6081 | ||
| IPsec connections that are active: | ||
| ``` | ||
|
|
||
| After the establishment is complete, you can capture packets and observe that the packets have been encrypted: | ||
|
|
||
| ```bash | ||
| # tcpdump -i eth0 -nel esp | ||
| 10:01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP(spi=0xcc91322a,seq=0x13d0), length 156 | ||
| 10:01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP(spi=0xc8df4221,seq=0x1d37), length 156 | ||
| ``` | ||
|
|
||
| After executing the script, you can turn off IPsec by executing the command: | ||
|
|
||
| ```bash | ||
| # bash ipsec.sh stop | ||
| ``` | ||
|
|
||
| Or execute the command to open it again: | ||
|
|
||
| ```bash | ||
| # bash ipsec.sh start | ||
| ``` | ||
| Change the args `--enable-ovn-ipsec=false` in kube-ovn-controller and kube-ovn-cni to `--enable-ovn-ipsec=true`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,94 +1,11 @@ | ||
| # 使用 IPsec 加密节点间通信 | ||
|
|
||
| 该功能从 v1.10.11 和 v1.11.4 后开始支持，kernel 版本至少是 3.10.0 以上，同时需要保证主机 UDP 500 和 4500 端口可用。 | ||
| 该功能从 v1.13.0 后支持，同时需要保证主机 UDP 500 和 4500 端口可用。 | ||
|
|
||
| ## 启动 IPsec | ||
| ## 加密流程 | ||
|
|
||
| 从 Kube-OVN 源码拷贝脚本 [ipsec.sh](https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/ipsec.sh)，执行命令如下，该脚本会调用 ovs-pki 生成和分配加密需要的证书： | ||
| kube-ovn-cni 负责将证书申请，会创建一个 certificatesigningrequest 给 kube-ovn-controller，kube-ovn-controller 会自动 approve 证书申请，然后 kube-ovn-cni 会根据证书生成 ipsec 配置文件，最后启动 ipsec 进程。 | ||
|
|
||
| ```bash | ||
| bash ipsec.sh init | ||
| ``` | ||
| ## 配置 IPsec | ||
|
|
||
| 执行完毕后，节点之间会协商一段时间建立 IPsec 隧道，经验值是十几秒到一分钟之间，可以通过如下命令来查看 IPsec 状态： | ||
|
|
||
| ```bash | ||
| # bash ipsec.sh status | ||
| Pod {ovs-ovn-d7hdt} ipsec status... | ||
| Interface name: ovn-a4718e-0 v1 (CONFIGURED) | ||
| Tunnel Type: geneve | ||
| Local IP: 172.18.0.2 | ||
| Remote IP: 172.18.0.4 | ||
| Address Family: IPv4 | ||
| SKB mark: None | ||
| Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem | ||
| Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d | ||
| Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem | ||
| Remote cert: None | ||
| Remote name: a4718e55-5b85-4f46-90e6-63527d080590 | ||
| CA cert: /etc/ipsec.d/cacerts/cacert.pem | ||
| PSK: None | ||
| Custom Options: {} | ||
| Ofport: 2 | ||
| CFM state: Disabled | ||
| Kernel policies installed: | ||
| src 172.18.0.2/32 dst 172.18.0.4/32 proto udp sport 6081 | ||
| src 172.18.0.2/32 dst 172.18.0.4/32 proto udp sport 6081 | ||
| src 172.18.0.2/32 dst 172.18.0.4/32 proto udp dport 6081 | ||
| src 172.18.0.2/32 dst 172.18.0.4/32 proto udp dport 6081 | ||
| Kernel security associations installed: | ||
| sel src 172.18.0.2/32 dst 172.18.0.4/32 proto udp sport 6081 | ||
| sel src 172.18.0.4/32 dst 172.18.0.2/32 proto udp dport 6081 | ||
| sel src 172.18.0.2/32 dst 172.18.0.4/32 proto udp dport 6081 | ||
| sel src 172.18.0.4/32 dst 172.18.0.2/32 proto udp sport 6081 | ||
| IPsec connections that are active: | ||
|
|
||
| Pod {ovs-ovn-fvbbj} ipsec status... | ||
| Interface name: ovn-8aebd9-0 v1 (CONFIGURED) | ||
| Tunnel Type: geneve | ||
| Local IP: 172.18.0.4 | ||
| Remote IP: 172.18.0.2 | ||
| Address Family: IPv4 | ||
| SKB mark: None | ||
| Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem | ||
| Local name: a4718e55-5b85-4f46-90e6-63527d080590 | ||
| Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem | ||
| Remote cert: None | ||
| Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d | ||
| CA cert: /etc/ipsec.d/cacerts/cacert.pem | ||
| PSK: None | ||
| Custom Options: {} | ||
| Ofport: 1 | ||
| CFM state: Disabled | ||
| Kernel policies installed: | ||
| src 172.18.0.4/32 dst 172.18.0.2/32 proto udp dport 6081 | ||
| src 172.18.0.4/32 dst 172.18.0.2/32 proto udp dport 6081 | ||
| src 172.18.0.4/32 dst 172.18.0.2/32 proto udp sport 6081 | ||
| src 172.18.0.4/32 dst 172.18.0.2/32 proto udp sport 6081 | ||
| Kernel security associations installed: | ||
| sel src 172.18.0.4/32 dst 172.18.0.2/32 proto udp dport 6081 | ||
| sel src 172.18.0.2/32 dst 172.18.0.4/32 proto udp sport 6081 | ||
| sel src 172.18.0.4/32 dst 172.18.0.2/32 proto udp sport 6081 | ||
| sel src 172.18.0.2/32 dst 172.18.0.4/32 proto udp dport 6081 | ||
| IPsec connections that are active: | ||
| ``` | ||
|
|
||
| 建立完成后可以抓包观察报文已经被加密： | ||
|
|
||
| ```bash | ||
| # tcpdump -i eth0 -nel esp | ||
| 10:01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP(spi=0xcc91322a,seq=0x13d0), length 156 | ||
| 10:01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP(spi=0xc8df4221,seq=0x1d37), length 156 | ||
| ``` | ||
|
|
||
| 当执行完脚本后，可以通过执行命令关闭 IPsec： | ||
|
|
||
| ```bash | ||
| # bash ipsec.sh stop | ||
| ``` | ||
|
|
||
| 或者执行命令再次打开： | ||
|
|
||
| ```bash | ||
| # bash ipsec.sh start | ||
| ``` | ||
| 将 kube-ovn-controller 和 kube-ovn-cni 中的 args `--enable-ovn-ipsec=false` 修改为 `--enable-ovn-ipsec=true`。 |