diff --git a/v1.12.x/advance/ovn-eip-fip-snat/index.html b/v1.12.x/advance/ovn-eip-fip-snat/index.html
index 3c7801f4e..7a86440fb 100644
--- a/v1.12.x/advance/ovn-eip-fip-snat/index.html
+++ b/v1.12.x/advance/ovn-eip-fip-snat/index.html
@@ -21,7 +21,7 @@
 <span class=w>          </span>-<span class=w> </span>--external-gateway-switch<span class=o>=</span>external204<span class=w> </span>
 
 <span class=c1>### 以上配置都和下面的公网网络配置 vlan id 和资源名保持一致，目前仅支持指定一个 underlay 公网作为默认外部公网。</span>
-</code></pre></div> <p>该配置项的设计和使用主要考虑了如下因素：</p> <ul> <li>基于该配置项可以对接到 provider network，vlan，subnet 的资源。</li> <li>基于该配置项可以将默认 vpc enable_eip_snat 功能对接到已有的 vlan，subnet 资源，同时支持公网 ip 的 ipam。</li> <li>如果仅使用默认 vpc 的 enable_eip_snat 模式, 且仅使用旧的基于 pod annotaion 的 eip fip snat，那么这个配置无需配置。</li> <li>基于该配置可以不使用默认 vpc enable_eip_snat 流程，仅通过对应到 vlan，subnet 流程，可以兼容仅自定义 vpc 使用 eip snat 的使用场景。</li> </ul> <h3 id=11-underlay>1.1 准备 underlay 公网网络<a class=headerlink href=#11-underlay title="Permanent link">&para;</a></h3> <div class=highlight><pre><span></span><code><span class=c1># 准备 provider-network， vlan， subnet</span>
+</code></pre></div> <p>该配置项的设计和使用主要考虑了如下因素：</p> <ul> <li>基于该配置项可以对接到 provider network，vlan，subnet 的资源。</li> <li>基于该配置项可以将默认 vpc enable_eip_snat 功能对接到已有的 vlan，subnet 资源，同时支持公网 ip 的 ipam。</li> <li>如果仅使用默认 vpc 的 enable_eip_snat 模式, 且仅使用旧的基于 pod annotaion 的 fip snat，那么这个配置无需配置。</li> <li>基于该配置可以不使用默认 vpc enable_eip_snat 流程，仅通过对应到 vlan，subnet 流程，可以兼容仅自定义 vpc 使用 eip snat 的使用场景。</li> </ul> <h3 id=11-underlay>1.1 准备 underlay 公网网络<a class=headerlink href=#11-underlay title="Permanent link">&para;</a></h3> <div class=highlight><pre><span></span><code><span class=c1># 准备 provider-network， vlan， subnet</span>
 <span class=c1># cat 01-provider-network.yaml</span>
 apiVersion:<span class=w> </span>kubeovn.io/v1
 kind:<span class=w> </span>ProviderNetwork
@@ -120,14 +120,14 @@
 Route<span class=w> </span>Table<span class=w> </span>&lt;main&gt;:
 <span class=w>                </span><span class=m>0</span>.0.0.0/0<span class=w>              </span><span class=m>10</span>.5.204.254<span class=w> </span>dst-ip
 <span class=c1># 目前该路由已自动维护</span>
-</code></pre></div> <h2 id=2-ovn-eip>2. ovn-eip<a class=headerlink href=#2-ovn-eip title="Permanent link">&para;</a></h2> <p>该功能和 iptables-eip 设计和使用方式基本一致，ovn-eip 目前有三种 type</p> <ul> <li>nat: 用于 ovn dnat，fip, snat, 这些 nat 类型会记录在 status 中</li> <li>lrp: 用于 vpc 和公网相连的资源</li> <li>node-ext-gw: 用于 ovn 基于 bfd 的 ecmp 静态路由场景</li> </ul> <div class=highlight><pre><span></span><code>---
+</code></pre></div> <h2 id=2-ovn-eip>2. ovn-eip<a class=headerlink href=#2-ovn-eip title="Permanent link">&para;</a></h2> <p>该功能和 iptables-eip 设计和使用方式基本一致，ovn-eip 目前有三种 type</p> <ul> <li>nat: 用于 ovn dnat，fip, snat, 这些 nat 类型会记录在 status 中</li> <li>lrp: Resources connected to the public network from a vpc can be used by nat</li> <li>lsp: 用于 ovn 基于 bfd 的 ecmp 静态路由场景，在网关节点上提供一个 ovs internal port 作为 ecmp 路由的下一跳</li> </ul> <div class=highlight><pre><span></span><code>---
 kind:<span class=w> </span>OvnEip
 apiVersion:<span class=w> </span>kubeovn.io/v1
 metadata:
 <span class=w>  </span>name:<span class=w> </span>eip-static
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>fip
+<span class=w>  </span>type:<span class=w> </span>nat
 
 <span class=c1># 动态分配一个 eip 资源，该资源预留用于 fip 场景</span>
 </code></pre></div> <h3 id=21-ovn-fip-pod-fip>2.1 ovn-fip 为 pod 绑定一个 fip<a class=headerlink href=#21-ovn-fip-pod-fip title="Permanent link">&para;</a></h3> <div class=highlight><pre><span></span><code><span class=c1># k get po -o wide -n vpc1 vpc-1-busybox01</span>
@@ -146,7 +146,7 @@
 <span class=w>  </span>name:<span class=w> </span>eip-static
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>fip
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnFip
@@ -207,7 +207,7 @@
 <span class=w>  </span>name:<span class=w> </span>eip-for-vip
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>fip
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnFip
@@ -268,7 +268,7 @@
 <span class=w>  </span>name:<span class=w> </span>snat-for-subnet-in-vpc
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>snat
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnSnatRule
@@ -286,7 +286,7 @@
 <span class=w>  </span>name:<span class=w> </span>snat-for-pod-vpc-ip
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>snat
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnSnatRule
@@ -417,7 +417,7 @@
 <span class=c1># kubectl get odnat eip-dnat </span>
 NAME<span class=w>       </span>EIP<span class=w>        </span>PROTOCOL<span class=w>   </span>V4EIP<span class=w>       </span>V4IP<span class=w>          </span>INTERNALPORT<span class=w>   </span>EXTERNALPORT<span class=w>   </span>IPNAME<span class=w>          </span>READY
 eip-dnat<span class=w>   </span>eip-dnat<span class=w>   </span>tcp<span class=w>        </span><span class=m>10</span>.5.49.4<span class=w>   </span><span class=m>192</span>.168.0.4<span class=w>   </span><span class=m>22</span><span class=w>             </span><span class=m>22</span><span class=w>             </span>test-dnat-vip<span class=w>   </span><span class=nb>true</span>
-</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> 最后更新: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年8月3日</span> <br> 创建日期: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年3月3日</span> </small> </div> <h2 id=__comments>评论</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=zh-CN crossorigin=anonymous async>
+</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> 最后更新: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年8月8日</span> <br> 创建日期: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年3月3日</span> </small> </div> <h2 id=__comments>评论</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=zh-CN crossorigin=anonymous async>
 </script> <script>
     var palette = __md_get("__palette")
     if (palette && typeof palette.color === "object")
diff --git a/v1.12.x/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html b/v1.12.x/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html
index 238243cb6..50b9713cb 100644
--- a/v1.12.x/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html
+++ b/v1.12.x/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html
@@ -10,7 +10,7 @@
 
 pod--&gt;vpc-subnet--&gt;vpc--&gt;snat--&gt;ecmp--&gt;external-subnet--&gt;gw-node1-ovnext0--&gt; node1-external-switch
 external-subnet--&gt;gw-node2-ovnext0--&gt; node2-external-switch
-external-subnet--&gt;gw-node3-ovnext0--&gt; node3-external-switch</code></pre> <p>该功能的使用方式和<a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> 基本一致，一致的部分包括 install.sh 的部署部分，provider-network，vlan，subnet 的准备部分。</p> <p>至于不相同的部分，会在以下部分具体阐述，主要包括 node-ext-gw 类型的 ovn-eip 的创建，以及基于 vpc enable_bfd 自动维护 bfd 以及 ecmp 静态路由。</p> <h2 id=1>1. 部署<a class=headerlink href=#1 title="Permanent link">&para;</a></h2> <h3 id=11-underlay>1.1 准备 underlay 公网网络<a class=headerlink href=#11-underlay title="Permanent link">&para;</a></h3> <h3 id=12-vpc-eip_snat>1.2 默认 vpc 启用 eip_snat<a class=headerlink href=#12-vpc-eip_snat title="Permanent link">&para;</a></h3> <h3 id=13-vpc-eip-snat-fip>1.3 自定义 vpc 启用 eip snat fip 功能<a class=headerlink href=#13-vpc-eip-snat-fip title="Permanent link">&para;</a></h3> <p>以上部分和 <a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> 完全一致，这些功能验证通过后，可以直接基于如下方式，将 vpc 切换到基于 ecmp 的 bfd 静态路由，当然也可以切回。</p> <p>自定义 vpc 使用该功能之前，需要先提供好网关节点，至少需要提供 2 个以上网关节点，注意当前实现 ovn-eip 的名字必须和网关节点名保持一致，目前没有做该资源的自动化维护。</p> <div class=highlight><pre><span></span><code><span class=c1># cat gw-node-eip.yaml</span>
+external-subnet--&gt;gw-node3-ovnext0--&gt; node3-external-switch</code></pre> <p>该功能的使用方式和<a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> 基本一致，一致的部分包括 install.sh 的部署部分，provider-network，vlan，subnet 的准备部分。</p> <p>至于不相同的部分，会在以下部分具体阐述，主要包括 lsp 类型的 ovn-eip 的创建，以及基于 vpc enable_bfd 自动维护 bfd 以及 ecmp 静态路由。</p> <h2 id=1>1. 部署<a class=headerlink href=#1 title="Permanent link">&para;</a></h2> <h3 id=11-underlay>1.1 准备 underlay 公网网络<a class=headerlink href=#11-underlay title="Permanent link">&para;</a></h3> <h3 id=12-vpc-eip_snat>1.2 默认 vpc 启用 eip_snat<a class=headerlink href=#12-vpc-eip_snat title="Permanent link">&para;</a></h3> <h3 id=13-vpc-eip-snat-fip>1.3 自定义 vpc 启用 eip snat fip 功能<a class=headerlink href=#13-vpc-eip-snat-fip title="Permanent link">&para;</a></h3> <p>以上部分和 <a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> 完全一致，这些功能验证通过后，可以直接基于如下方式，将 vpc 切换到基于 ecmp 的 bfd 静态路由，当然也可以切回。</p> <p>自定义 vpc 使用该功能之前，需要先提供好网关节点，至少需要提供 2 个以上网关节点，注意当前实现 ovn-eip 的名字必须和网关节点名保持一致，目前没有做该资源的自动化维护。</p> <div class=highlight><pre><span></span><code><span class=c1># cat gw-node-eip.yaml</span>
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnEip</span>
 <span class=nt>apiVersion</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">kubeovn.io/v1</span>
@@ -18,7 +18,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">pc-node-1</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">external204</span>
-<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">node-ext-gw</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">lsp</span>
 
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnEip</span>
@@ -27,7 +27,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">pc-node-2</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">external204</span>
-<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">node-ext-gw</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">lsp</span>
 
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnEip</span>
@@ -36,7 +36,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">pc-node-3</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">external204</span>
-<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">node-ext-gw</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">lsp</span>
 </code></pre></div> <p>由于这个场景目前设计上是供 vpc ecmp 出公网使用，所以以上在没有 vpc 启用 bfd 的时候，即不存在带有 enable bfd 标签的 lrp 的 ovn eip 的时候，网关节点不会触发创建网关网卡，也无法成功启动对端 bfd 会话的监听。</p> <h2 id=2-vpc-ecmp-bfd-l3-ha>2. 自定义 vpc 启用 ecmp bfd L3 HA 公网功能<a class=headerlink href=#2-vpc-ecmp-bfd-l3-ha title="Permanent link">&para;</a></h2> <div class=highlight><pre><span></span><code><span class=c1># cat 01-vpc-ecmp-enable-external-bfd.yml</span>
 kind:<span class=w> </span>Vpc
 apiVersion:<span class=w> </span>kubeovn.io/v1
@@ -258,7 +258,7 @@
 <span class=c1># 而且 ovnext ns 中的对端 bfd 会话也自动移除</span>
 <span class=c1># 该切换过程保持 vpc subnet 内保持 ping 未看到(秒级)丢包</span>
 <span class=c1># 再切换回去 也未看到(秒级)丢包</span>
-</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> 最后更新: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年5月9日</span> <br> 创建日期: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年3月3日</span> </small> </div> <h2 id=__comments>评论</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=zh-CN crossorigin=anonymous async>
+</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> 最后更新: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年8月8日</span> <br> 创建日期: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年3月3日</span> </small> </div> <h2 id=__comments>评论</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=zh-CN crossorigin=anonymous async>
 </script> <script>
     var palette = __md_get("__palette")
     if (palette && typeof palette.color === "object")
diff --git a/v1.12.x/en/advance/ovn-eip-fip-snat/index.html b/v1.12.x/en/advance/ovn-eip-fip-snat/index.html
index d2bc9d95c..cf57ce4f1 100644
--- a/v1.12.x/en/advance/ovn-eip-fip-snat/index.html
+++ b/v1.12.x/en/advance/ovn-eip-fip-snat/index.html
@@ -121,16 +121,16 @@
 Route<span class=w> </span>Table<span class=w> </span>&lt;main&gt;:
 <span class=w>                </span><span class=m>0</span>.0.0.0/0<span class=w>              </span><span class=m>10</span>.5.204.254<span class=w> </span>dst-ip
 <span class=c1># The route currently supports automatic maintenance</span>
-</code></pre></div> <h2 id=2-ovn-eip>2. ovn-eip<a class=headerlink href=#2-ovn-eip title="Permanent link">&para;</a></h2> <p>This function is designed and used in the same way as iptables-eip, ovn-eip currently has three types</p> <ul> <li>nat: indicates ovn dnat, fip, and snat. These nat types are recorded in status</li> <li>lrp: indicates the resource used to connect a vpc to the public network</li> <li>node-ext-gw: applies to ovn BFD-based ecmp static route scenario</li> </ul> <div class=highlight><pre><span></span><code>---
+</code></pre></div> <h2 id=2-ovn-eip>2. ovn-eip<a class=headerlink href=#2-ovn-eip title="Permanent link">&para;</a></h2> <p>This function is designed and used in the same way as iptables-eip, ovn-eip currently has three types</p> <ul> <li>nat: indicates ovn dnat, fip, and snat. These nat types are recorded in status</li> <li>lrp: indicates the resource used to connect a vpc to the public network</li> <li>lsp: In the ovn BFD-based ecmp static route scenario, an ovs internal port is provided on the gateway node as the next hop of the ecmp route</li> </ul> <div class=highlight><pre><span></span><code>---
 kind:<span class=w> </span>OvnEip
 apiVersion:<span class=w> </span>kubeovn.io/v1
 metadata:
 <span class=w>  </span>name:<span class=w> </span>eip-static
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>fip
+<span class=w>  </span>type:<span class=w> </span>nat
 
-<span class=c1># Dynamically allocate an eip resource that is reserved for fip scenarios</span>
+<span class=c1># Dynamically allocate an eip resource that is reserved for fip dnat_and_snat scenarios</span>
 </code></pre></div> <h3 id=21-create-an-fip-for-pod>2.1 Create an fip for pod<a class=headerlink href=#21-create-an-fip-for-pod title="Permanent link">&para;</a></h3> <div class=highlight><pre><span></span><code><span class=c1># k get po -o wide -n vpc1 vpc-1-busybox01</span>
 NAME<span class=w>              </span>READY<span class=w>   </span>STATUS<span class=w>    </span>RESTARTS<span class=w>   </span>AGE<span class=w>     </span>IP<span class=w>            </span>NODE
 vpc-1-busybox01<span class=w>   </span><span class=m>1</span>/1<span class=w>     </span>Running<span class=w>   </span><span class=m>0</span><span class=w>          </span>3d15h<span class=w>   </span><span class=m>192</span>.168.0.2<span class=w>   </span>pc-node-2
@@ -147,7 +147,7 @@
 <span class=w>  </span>name:<span class=w> </span>eip-static
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>fip
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnFip
@@ -209,7 +209,7 @@
 <span class=w>  </span>name:<span class=w> </span>eip-for-vip
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>fip
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnFip
@@ -269,7 +269,7 @@
 <span class=w>  </span>name:<span class=w> </span>snat-for-subnet-in-vpc
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>snat
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnSnatRule
@@ -287,7 +287,7 @@
 <span class=w>  </span>name:<span class=w> </span>snat-for-pod-vpc-ip
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>snat
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnSnatRule
@@ -378,6 +378,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">eip-static</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">underlay</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">nat</span>
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnDnatRule</span>
 <span class=nt>apiVersion</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">kubeovn.io/v1</span>
@@ -418,7 +419,7 @@
 <span class=c1># kubectl get odnat eip-dnat </span>
 NAME<span class=w>       </span>EIP<span class=w>        </span>PROTOCOL<span class=w>   </span>V4EIP<span class=w>       </span>V4IP<span class=w>          </span>INTERNALPORT<span class=w>   </span>EXTERNALPORT<span class=w>   </span>IPNAME<span class=w>          </span>READY
 eip-dnat<span class=w>   </span>eip-dnat<span class=w>   </span>tcp<span class=w>        </span><span class=m>10</span>.5.49.4<span class=w>   </span><span class=m>192</span>.168.0.4<span class=w>   </span><span class=m>22</span><span class=w>             </span><span class=m>22</span><span class=w>             </span>test-dnat-vip<span class=w>   </span><span class=nb>true</span>
-</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">August 3, 2023</span> <br> Created: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">March 3, 2023</span> </small> </div> <h2 id=__comments>Comments</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=en crossorigin=anonymous async>
+</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">August 8, 2023</span> <br> Created: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">March 3, 2023</span> </small> </div> <h2 id=__comments>Comments</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=en crossorigin=anonymous async>
 </script> <script>
     var palette = __md_get("__palette")
     if (palette && typeof palette.color === "object")
diff --git a/v1.12.x/en/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html b/v1.12.x/en/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html
index f7bb44162..682dc5c35 100644
--- a/v1.12.x/en/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html
+++ b/v1.12.x/en/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html
@@ -10,7 +10,7 @@
 
 pod--&gt;vpc-subnet--&gt;vpc--&gt;snat--&gt;ecmp--&gt;external-subnet--&gt;gw-node1-ovnext0--&gt; node1-external-switch
 external-subnet--&gt;gw-node2-ovnext0--&gt; node2-external-switch
-external-subnet--&gt;gw-node3-ovnext0--&gt; node3-external-switch</code></pre> <p>This functions basically the same as <a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> .</p> <p>As for the different parts, which will be specified in the following sections, mainly including the creation of ovn-eip of node-ext-gw type and the automatic maintenance of bfd as well as ecmp static routes based on vpc enable_bfd.</p> <h2 id=1-deployment>1. Deployment<a class=headerlink href=#1-deployment title="Permanent link">&para;</a></h2> <h3 id=11-create-the-underlay-public-network>1.1 Create the underlay public network<a class=headerlink href=#11-create-the-underlay-public-network title="Permanent link">&para;</a></h3> <h3 id=12-default-vpc-enable-eip_snat>1.2 Default vpc enable eip_snat<a class=headerlink href=#12-default-vpc-enable-eip_snat title="Permanent link">&para;</a></h3> <h3 id=13-custom-vpc-enable-eip-snat-fip-function>1.3 Custom vpc enable eip snat fip function<a class=headerlink href=#13-custom-vpc-enable-eip-snat-fip-function title="Permanent link">&para;</a></h3> <p>The above section is exactly the same with <a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a>.</p> <p>After these functions are verified, the vpc can be switched directly to the ecmp-based bfd static route based on the following way, or of course, switched directly back.</p> <p>Before customizing vpc to use this feature, you need to provide some gateway nodes, at least 2. Note that the name of the current implementation of ovn-eip must be consistent with the gateway node name, no automated maintenance is currently done for this resource.</p> <div class=highlight><pre><span></span><code><span class=c1># cat gw-node-eip.yaml</span>
+external-subnet--&gt;gw-node3-ovnext0--&gt; node3-external-switch</code></pre> <p>This functions basically the same as <a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> .</p> <p>As for the different parts, which will be specified in the following sections, mainly including the creation of ovn-eip of lsp type and the automatic maintenance of bfd as well as ecmp static routes based on vpc enable_bfd.</p> <h2 id=1-deployment>1. Deployment<a class=headerlink href=#1-deployment title="Permanent link">&para;</a></h2> <h3 id=11-create-the-underlay-public-network>1.1 Create the underlay public network<a class=headerlink href=#11-create-the-underlay-public-network title="Permanent link">&para;</a></h3> <h3 id=12-default-vpc-enable-eip_snat>1.2 Default vpc enable eip_snat<a class=headerlink href=#12-default-vpc-enable-eip_snat title="Permanent link">&para;</a></h3> <h3 id=13-custom-vpc-enable-eip-snat-fip-function>1.3 Custom vpc enable eip snat fip function<a class=headerlink href=#13-custom-vpc-enable-eip-snat-fip-function title="Permanent link">&para;</a></h3> <p>The above section is exactly the same with <a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a>.</p> <p>After these functions are verified, the vpc can be switched directly to the ecmp-based bfd static route based on the following way, or of course, switched directly back.</p> <p>Before customizing vpc to use this feature, you need to provide some gateway nodes, at least 2. Note that the name of the current implementation of ovn-eip must be consistent with the gateway node name, no automated maintenance is currently done for this resource.</p> <div class=highlight><pre><span></span><code><span class=c1># cat gw-node-eip.yaml</span>
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnEip</span>
 <span class=nt>apiVersion</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">kubeovn.io/v1</span>
@@ -18,7 +18,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">pc-node-1</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">external204</span>
-<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">node-ext-gw</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">lsp</span>
 
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnEip</span>
@@ -27,7 +27,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">pc-node-2</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">external204</span>
-<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">node-ext-gw</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">lsp</span>
 
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnEip</span>
@@ -36,7 +36,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">pc-node-3</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">external204</span>
-<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">node-ext-gw</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">lsp</span>
 </code></pre></div> <p>Since this scenario is currently designed for vpc ecmp out of the public network, the gateway node above will not trigger the creation of a gateway NIC when there is no vpc enabled bfd, i.e. when there is no ovn eip (lrp) with enable bfd labeled, and will not be able to successfully start listening to the bfd session on the other side.</p> <h2 id=2-custom-vpc-enable-ecmp-bfd-l3-ha-public-network-function>2. Custom vpc enable ecmp bfd L3 HA public network function<a class=headerlink href=#2-custom-vpc-enable-ecmp-bfd-l3-ha-public-network-function title="Permanent link">&para;</a></h2> <div class=highlight><pre><span></span><code><span class=c1># cat 01-vpc-ecmp-enable-external-bfd.yml</span>
 kind:<span class=w> </span>Vpc
 apiVersion:<span class=w> </span>kubeovn.io/v1
@@ -250,7 +250,7 @@
 <span class=c1># After application the route will switch back to the normal default static route</span>
 <span class=c1># nbctl list bfd, the bfd session associated with lrp has been removed</span>
 <span class=c1># And the opposite side of the bfd session in ovnext ns is automatically removed</span>
-</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">May 9, 2023</span> <br> Created: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">March 3, 2023</span> </small> </div> <h2 id=__comments>Comments</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=en crossorigin=anonymous async>
+</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">August 8, 2023</span> <br> Created: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">March 3, 2023</span> </small> </div> <h2 id=__comments>Comments</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=en crossorigin=anonymous async>
 </script> <script>
     var palette = __md_get("__palette")
     if (palette && typeof palette.color === "object")
diff --git a/v1.12.x/search/search_index.json b/v1.12.x/search/search_index.json
index 64502a31b..782694c66 100644
--- a/v1.12.x/search/search_index.json
+++ b/v1.12.x/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\u200b\\-]"},"docs":[{"location":"","text":"Kube-OVN \u00b6 Kube-OVN \u662f\u4e00\u6b3e CNCF \u65d7\u4e0b\u7684\u4f01\u4e1a\u7ea7\u4e91\u539f\u751f\u7f51\u7edc\u7f16\u6392\u7cfb\u7edf\uff0c\u5c06 SDN \u7684\u80fd\u529b\u548c\u4e91\u539f\u751f\u7ed3\u5408\uff0c \u63d0\u4f9b\u4e30\u5bcc\u7684\u529f\u80fd\uff0c\u6781\u81f4\u7684\u6027\u80fd\u4ee5\u53ca\u826f\u597d\u7684\u53ef\u8fd0\u7ef4\u6027\u3002 \u4e30\u5bcc\u7684\u529f\u80fd\uff1a \u5982\u679c\u4f60\u6000\u5ff5 SDN \u9886\u57df\u4e30\u5bcc\u7684\u7f51\u7edc\u80fd\u529b\u5374\u5728\u4e91\u539f\u751f\u9886\u57df\u82e6\u82e6\u8ffd\u5bfb\u800c\u4e0d\u5f97\uff0c\u90a3\u4e48 Kube-OVN \u5c06\u662f\u4f60\u7684\u6700\u4f73\u9009\u62e9\u3002 \u501f\u52a9 OVS/OVN \u5728 SDN \u9886\u57df\u6210\u719f\u7684\u80fd\u529b\uff0cKube-OVN \u5c06\u7f51\u7edc\u865a\u62df\u5316\u7684\u4e30\u5bcc\u529f\u80fd\u5e26\u5165\u4e91\u539f\u751f\u9886\u57df\u3002\u76ee\u524d\u5df2\u652f\u6301 \u5b50\u7f51\u7ba1\u7406 \uff0c \u9759\u6001 IP \u5206\u914d \uff0c \u5206\u5e03\u5f0f/\u96c6\u4e2d\u5f0f\u7f51\u5173 \uff0c Underlay/Overlay \u6df7\u5408\u7f51\u7edc \uff0c VPC \u591a\u79df\u6237\u7f51\u7edc \uff0c \u8de8\u96c6\u7fa4\u4e92\u8054\u7f51\u7edc \uff0c QoS \u7ba1\u7406 \uff0c \u591a\u7f51\u5361\u7ba1\u7406 \uff0c ACL \u7f51\u7edc\u63a7\u5236 \uff0c \u6d41\u91cf\u955c\u50cf \uff0cARM \u652f\u6301\uff0c Windows \u652f\u6301 \u7b49\u8bf8\u591a\u529f\u80fd\u3002 \u6781\u81f4\u7684\u6027\u80fd\uff1a \u5982\u679c\u4f60\u62c5\u5fc3\u5bb9\u5668\u7f51\u7edc\u4f1a\u5e26\u6765\u989d\u5916\u7684\u6027\u80fd\u635f\u8017\uff0c\u90a3\u4e48\u6765\u770b\u4e00\u4e0b Kube-OVN \u662f\u5982\u4f55\u6781\u81f4\u7684 \u4f18\u5316\u6027\u80fd \u3002 \u5728\u6570\u636e\u5e73\u9762\uff0c\u901a\u8fc7\u4e00\u7cfb\u5217\u5bf9\u6d41\u8868\u548c\u5185\u6838\u7684\u7cbe\u5fc3\u4f18\u5316\uff0c\u5e76\u501f\u52a9 eBPF \u3001 DPDK \u3001 \u667a\u80fd\u7f51\u5361\u5378\u8f7d \u7b49\u65b0\u5174\u6280\u672f\uff0c Kube-OVN \u53ef\u4ee5\u5728\u5ef6\u8fdf\u548c\u541e\u5410\u91cf\u7b49\u65b9\u9762\u7684\u6307\u6807\u8fbe\u5230\u8fd1\u4f3c\u6216\u8d85\u51fa\u5bbf\u4e3b\u673a\u7f51\u7edc\u6027\u80fd\u7684\u6c34\u5e73\u3002\u5728\u63a7\u5236\u5e73\u9762\uff0c\u901a\u8fc7\u5bf9 OVN \u4e0a\u6e38\u6d41\u8868\u7684\u88c1\u526a \uff0c \u5404\u79cd\u7f13\u5b58\u6280\u672f\u7684\u4f7f\u7528\u548c\u8c03\u4f18\uff0cKube-OVN \u53ef\u4ee5\u652f\u6301\u5927\u89c4\u6a21\u4e0a\u5343\u8282\u70b9\u548c\u4e0a\u4e07 Pod \u7684\u96c6\u7fa4\u3002 \u6b64\u5916 Kube-OVN \u8fd8\u5728\u4e0d\u65ad\u4f18\u5316 CPU \u548c\u5185\u5b58\u7b49\u8d44\u6e90\u7684\u4f7f\u7528\u91cf\uff0c\u4ee5\u9002\u5e94\u8fb9\u7f18\u7b49\u8d44\u6e90\u6709\u9650\u573a\u666f\u3002 \u826f\u597d\u7684\u53ef\u8fd0\u7ef4\u6027\uff1a \u5982\u679c\u4f60\u5bf9\u5bb9\u5668\u7f51\u7edc\u7684\u8fd0\u7ef4\u5fc3\u5b58\u5fe7\u8651\uff0cKube-OVN \u5185\u7f6e\u4e86\u5927\u91cf\u7684\u5de5\u5177\u6765\u5e2e\u52a9\u4f60\u7b80\u5316\u8fd0\u7ef4\u64cd\u4f5c\u3002 Kube-OVN \u63d0\u4f9b\u4e86 \u4e00\u952e\u5b89\u88c5\u811a\u672c \uff0c\u5e2e\u52a9\u7528\u6237\u8fc5\u901f\u642d\u5efa\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002\u540c\u65f6\u5185\u7f6e\u7684\u4e30\u5bcc\u7684 \u76d1\u63a7\u6307\u6807 \u548c Grafana \u9762\u677f \uff0c \u53ef\u5e2e\u52a9\u7528\u6237\u5efa\u7acb\u5b8c\u5584\u7684\u76d1\u63a7\u4f53\u7cfb\u3002\u5f3a\u5927\u7684 \u547d\u4ee4\u884c\u5de5\u5177 \u53ef\u4ee5\u7b80\u5316\u7528\u6237\u7684\u65e5\u5e38\u8fd0\u7ef4\u64cd\u4f5c\u3002\u901a\u8fc7 \u548c Cilium \u7ed3\u5408 \uff0c\u5229\u7528 eBPF \u80fd\u529b\u7528\u6237\u53ef\u4ee5 \u589e\u5f3a\u5bf9\u7f51\u7edc\u7684\u53ef\u89c2\u6d4b\u6027\u3002 \u6b64\u5916 \u6d41\u91cf\u955c\u50cf \u7684\u80fd\u529b\u53ef\u4ee5\u65b9\u4fbf\u7528\u6237\u81ea\u5b9a\u4e49\u6d41\u91cf\u76d1\u63a7\uff0c\u5e76\u548c\u4f20\u7edf\u7684 NPM \u7cfb\u7edf\u5bf9\u63a5\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4e3b\u9875"},{"location":"#kube-ovn","text":"Kube-OVN \u662f\u4e00\u6b3e CNCF \u65d7\u4e0b\u7684\u4f01\u4e1a\u7ea7\u4e91\u539f\u751f\u7f51\u7edc\u7f16\u6392\u7cfb\u7edf\uff0c\u5c06 SDN \u7684\u80fd\u529b\u548c\u4e91\u539f\u751f\u7ed3\u5408\uff0c \u63d0\u4f9b\u4e30\u5bcc\u7684\u529f\u80fd\uff0c\u6781\u81f4\u7684\u6027\u80fd\u4ee5\u53ca\u826f\u597d\u7684\u53ef\u8fd0\u7ef4\u6027\u3002 \u4e30\u5bcc\u7684\u529f\u80fd\uff1a \u5982\u679c\u4f60\u6000\u5ff5 SDN \u9886\u57df\u4e30\u5bcc\u7684\u7f51\u7edc\u80fd\u529b\u5374\u5728\u4e91\u539f\u751f\u9886\u57df\u82e6\u82e6\u8ffd\u5bfb\u800c\u4e0d\u5f97\uff0c\u90a3\u4e48 Kube-OVN \u5c06\u662f\u4f60\u7684\u6700\u4f73\u9009\u62e9\u3002 \u501f\u52a9 OVS/OVN \u5728 SDN \u9886\u57df\u6210\u719f\u7684\u80fd\u529b\uff0cKube-OVN \u5c06\u7f51\u7edc\u865a\u62df\u5316\u7684\u4e30\u5bcc\u529f\u80fd\u5e26\u5165\u4e91\u539f\u751f\u9886\u57df\u3002\u76ee\u524d\u5df2\u652f\u6301 \u5b50\u7f51\u7ba1\u7406 \uff0c \u9759\u6001 IP \u5206\u914d \uff0c \u5206\u5e03\u5f0f/\u96c6\u4e2d\u5f0f\u7f51\u5173 \uff0c Underlay/Overlay \u6df7\u5408\u7f51\u7edc \uff0c VPC \u591a\u79df\u6237\u7f51\u7edc \uff0c \u8de8\u96c6\u7fa4\u4e92\u8054\u7f51\u7edc \uff0c QoS \u7ba1\u7406 \uff0c \u591a\u7f51\u5361\u7ba1\u7406 \uff0c ACL \u7f51\u7edc\u63a7\u5236 \uff0c \u6d41\u91cf\u955c\u50cf \uff0cARM \u652f\u6301\uff0c Windows \u652f\u6301 \u7b49\u8bf8\u591a\u529f\u80fd\u3002 \u6781\u81f4\u7684\u6027\u80fd\uff1a \u5982\u679c\u4f60\u62c5\u5fc3\u5bb9\u5668\u7f51\u7edc\u4f1a\u5e26\u6765\u989d\u5916\u7684\u6027\u80fd\u635f\u8017\uff0c\u90a3\u4e48\u6765\u770b\u4e00\u4e0b Kube-OVN \u662f\u5982\u4f55\u6781\u81f4\u7684 \u4f18\u5316\u6027\u80fd \u3002 \u5728\u6570\u636e\u5e73\u9762\uff0c\u901a\u8fc7\u4e00\u7cfb\u5217\u5bf9\u6d41\u8868\u548c\u5185\u6838\u7684\u7cbe\u5fc3\u4f18\u5316\uff0c\u5e76\u501f\u52a9 eBPF \u3001 DPDK \u3001 \u667a\u80fd\u7f51\u5361\u5378\u8f7d \u7b49\u65b0\u5174\u6280\u672f\uff0c Kube-OVN \u53ef\u4ee5\u5728\u5ef6\u8fdf\u548c\u541e\u5410\u91cf\u7b49\u65b9\u9762\u7684\u6307\u6807\u8fbe\u5230\u8fd1\u4f3c\u6216\u8d85\u51fa\u5bbf\u4e3b\u673a\u7f51\u7edc\u6027\u80fd\u7684\u6c34\u5e73\u3002\u5728\u63a7\u5236\u5e73\u9762\uff0c\u901a\u8fc7\u5bf9 OVN \u4e0a\u6e38\u6d41\u8868\u7684\u88c1\u526a \uff0c \u5404\u79cd\u7f13\u5b58\u6280\u672f\u7684\u4f7f\u7528\u548c\u8c03\u4f18\uff0cKube-OVN \u53ef\u4ee5\u652f\u6301\u5927\u89c4\u6a21\u4e0a\u5343\u8282\u70b9\u548c\u4e0a\u4e07 Pod \u7684\u96c6\u7fa4\u3002 \u6b64\u5916 Kube-OVN \u8fd8\u5728\u4e0d\u65ad\u4f18\u5316 CPU \u548c\u5185\u5b58\u7b49\u8d44\u6e90\u7684\u4f7f\u7528\u91cf\uff0c\u4ee5\u9002\u5e94\u8fb9\u7f18\u7b49\u8d44\u6e90\u6709\u9650\u573a\u666f\u3002 \u826f\u597d\u7684\u53ef\u8fd0\u7ef4\u6027\uff1a \u5982\u679c\u4f60\u5bf9\u5bb9\u5668\u7f51\u7edc\u7684\u8fd0\u7ef4\u5fc3\u5b58\u5fe7\u8651\uff0cKube-OVN \u5185\u7f6e\u4e86\u5927\u91cf\u7684\u5de5\u5177\u6765\u5e2e\u52a9\u4f60\u7b80\u5316\u8fd0\u7ef4\u64cd\u4f5c\u3002 Kube-OVN \u63d0\u4f9b\u4e86 \u4e00\u952e\u5b89\u88c5\u811a\u672c \uff0c\u5e2e\u52a9\u7528\u6237\u8fc5\u901f\u642d\u5efa\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002\u540c\u65f6\u5185\u7f6e\u7684\u4e30\u5bcc\u7684 \u76d1\u63a7\u6307\u6807 \u548c Grafana \u9762\u677f \uff0c \u53ef\u5e2e\u52a9\u7528\u6237\u5efa\u7acb\u5b8c\u5584\u7684\u76d1\u63a7\u4f53\u7cfb\u3002\u5f3a\u5927\u7684 \u547d\u4ee4\u884c\u5de5\u5177 \u53ef\u4ee5\u7b80\u5316\u7528\u6237\u7684\u65e5\u5e38\u8fd0\u7ef4\u64cd\u4f5c\u3002\u901a\u8fc7 \u548c Cilium \u7ed3\u5408 \uff0c\u5229\u7528 eBPF \u80fd\u529b\u7528\u6237\u53ef\u4ee5 \u589e\u5f3a\u5bf9\u7f51\u7edc\u7684\u53ef\u89c2\u6d4b\u6027\u3002 \u6b64\u5916 \u6d41\u91cf\u955c\u50cf \u7684\u80fd\u529b\u53ef\u4ee5\u65b9\u4fbf\u7528\u6237\u81ea\u5b9a\u4e49\u6d41\u91cf\u76d1\u63a7\uff0c\u5e76\u548c\u4f20\u7edf\u7684 NPM \u7cfb\u7edf\u5bf9\u63a5\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN"},{"location":"contact/","text":"\u8054\u7cfb\u65b9\u5f0f \u00b6 \u5173\u6ce8\u516c\u4f17\u53f7\u83b7\u5f97\u66f4\u591a\u6700\u65b0\u4fe1\u606f\uff0c\u8bf7\u626b\u63cf\u4e0b\u65b9\u4e8c\u7ef4\u7801: \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8054\u7cfb\u65b9\u5f0f"},{"location":"contact/#_1","text":"\u5173\u6ce8\u516c\u4f17\u53f7\u83b7\u5f97\u66f4\u591a\u6700\u65b0\u4fe1\u606f\uff0c\u8bf7\u626b\u63cf\u4e0b\u65b9\u4e8c\u7ef4\u7801: \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8054\u7cfb\u65b9\u5f0f"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/","text":"\u4f7f\u7528 eBPF \u52a0\u901f\u8282\u70b9\u5185 TCP \u901a\u4fe1 \u00b6 \u5728\u4e00\u4e9b\u8fb9\u7f18\u548c 5G \u7684\u573a\u666f\u4e0b\uff0c\u540c\u8282\u70b9\u5185\u7684 Pod \u4e4b\u95f4\u4f1a\u8fdb\u884c\u5927\u91cf\u7684 TCP \u901a\u4fe1\uff0c\u901a\u8fc7\u4f7f\u7528 Intel \u5f00\u6e90\u7684 istio-tcpip-bypass \u9879\u76ee\uff0cPod \u53ef\u4ee5\u501f\u52a9 eBPF \u7684\u80fd\u529b\u7ed5\u8fc7\u4e3b\u673a\u7684 TCP/IP \u534f\u8bae\u6808\uff0c\u76f4\u63a5\u8fdb\u884c socket \u901a\u4fe1\uff0c\u4ece\u800c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u63d0\u5347\u541e\u5410\u91cf\u3002 \u57fa\u672c\u539f\u7406 \u00b6 \u5728\u5f53\u524d\u7684\u5b9e\u73b0\u4e0b\uff0c\u540c\u4e3b\u673a\u7684\u4e24\u4e2a Pod \u8fdb\u884c TCP \u8fdb\u884c\u901a\u4fe1\u9700\u8981\u7ecf\u8fc7\u5927\u91cf\u7684\u7f51\u7edc\u6808\uff0c\u5305\u62ec TCP/IP, netfilter\uff0cOVS \u7b49\u5982\u4e0b\u56fe\u6240\u793a\uff1a istio-tcpip-bypass \u63d2\u4ef6\u53ef\u4ee5\u81ea\u52a8\u5206\u6790\u5e76\u8bc6\u522b\u51fa\u540c\u4e3b\u673a\u5185\u7684 TCP \u901a\u4fe1\uff0c\u5e76\u7ed5\u8fc7\u590d\u6742\u7684\u5185\u6838\u6808\u4ece\u800c\u53ef\u4ee5\u76f4\u63a5\u8fdb\u884c socket \u95f4\u7684\u6570\u636e\u4f20\u8f93\uff0c \u6765\u964d\u4f4e\u7f51\u7edc\u6808\u5904\u7406\u5f00\u9500\uff0c\u5982\u4e0b\u56fe\u6240\u793a\uff1a \u7531\u4e8e\u8be5\u7ec4\u4ef6\u53ef\u4ee5\u81ea\u52a8\u8bc6\u522b\u540c\u4e3b\u673a\u5185\u7684 TCP \u901a\u4fe1\uff0c\u5e76\u8fdb\u884c\u4f18\u5316\u3002\u5728\u57fa\u4e8e\u4ee3\u7406\u6a21\u5f0f\u7684 Service Mesh \u73af\u5883\u4e0b\uff0c\u8be5\u7ec4\u4ef6\u4e5f\u53ef\u4ee5\u589e\u5f3a Service Mesh \u7684\u6027\u80fd\u8868\u73b0\u3002 \u66f4\u591a\u6280\u672f\u5b9e\u73b0\u7ec6\u8282\u53ef\u4ee5\u53c2\u8003 Tanzu Service Mesh Acceleration using eBPF \u3002 \u73af\u5883\u51c6\u5907 \u00b6 eBPF \u5bf9\u5185\u6838\u7248\u672c\u6709\u4e00\u5b9a\u8981\u6c42\uff0c\u63a8\u8350\u4f7f\u7528 Ubuntu 20.04 \u548c Linux 5.4.0-74-generic \u7248\u672c\u5185\u6838\u8fdb\u884c\u5b9e\u9a8c\u3002 \u5b9e\u9a8c\u6b65\u9aa4 \u00b6 \u5728\u540c\u4e00\u4e2a\u8282\u70b9\u4e0a\u90e8\u7f72\u4e24\u4e2a\u6027\u80fd\u6d4b\u8bd5 Pod\uff0c\u82e5\u96c6\u7fa4\u5185\u5b58\u5728\u591a\u53f0\u673a\u5668\u9700\u8981\u6307\u5b9a nodeSelector \uff1a # kubectl create deployment perf --image=kubeovn/perf:dev --replicas=2 deployment.apps/perf created # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES perf-7697bc6ddf-b2cpv 1 /1 Running 0 28s 100 .64.0.3 sealos <none> <none> perf-7697bc6ddf-p2xpt 1 /1 Running 0 28s 100 .64.0.2 sealos <none> <none> \u8fdb\u5165\u5176\u4e2d\u4e00\u4e2a Pod \u5f00\u542f qperf server\uff0c\u5728\u53e6\u4e00\u4e2a Pod \u4e2d\u542f\u52a8 qperf client \u8fdb\u884c\u6027\u80fd\u6d4b\u8bd5\uff1a # kubectl exec -it perf-7697bc6ddf-b2cpv sh / # qperf # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw \u90e8\u7f72 istio-tcpip-bypass \u63d2\u4ef6\uff1a kubectl apply -f https://raw.githubusercontent.com/intel/istio-tcpip-bypass/main/bypass-tcpip-daemonset.yaml \u518d\u6b21\u8fdb\u5165 perf client \u5bb9\u5668\u8fdb\u884c\u6027\u80fd\u6d4b\u8bd5\uff1a # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw \u6d4b\u8bd5\u7ed3\u679c \u00b6 \u6839\u636e\u6d4b\u8bd5\u7ed3\u679c TCP \u5ef6\u8fdf\u5728\u4e0d\u540c\u6570\u636e\u5305\u5927\u5c0f\u7684\u60c5\u51b5\u4e0b\u4f1a\u6709 40% ~ 60% \u7684\u5ef6\u8fdf\u4e0b\u964d\uff0c\u5728\u6570\u636e\u5305\u5927\u4e8e 1024 \u5b57\u8282\u65f6\u541e\u5410\u91cf\u4f1a\u6709 40% ~ 80% \u63d0\u5347\u3002 Packet Size (byte) eBPF tcp_lat (us) Default tcp_lat (us) eBPF tcp_bw (Mb/s) Default tcp_bw(Mb/s) 1 20.2 44.5 1.36 4.27 4 20.2 48.7 5.48 16.7 16 19.6 41.6 21.7 63.5 64 18.8 41.3 96.8 201 256 19.2 36 395 539 1024 18.3 42.4 1360 846 4096 16.5 62.6 4460 2430 16384 20.2 58.8 9600 6900 \u5728\u6d4b\u8bd5\u7684\u786c\u4ef6\u73af\u5883\u4e0b\uff0c\u6570\u636e\u5305\u5c0f\u4e8e 512 \u5b57\u8282\u65f6\uff0c\u4f7f\u7528 eBPF \u4f18\u5316\u541e\u5410\u91cf\u6307\u6807\u4f1a\u4f4e\u4e8e\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7684\u541e\u5410\u91cf\u3002 \u8be5\u60c5\u51b5\u53ef\u80fd\u548c\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7f51\u5361\u5f00\u542f TCP \u805a\u5408\u4f18\u5316\u76f8\u5173\u3002\u5982\u679c\u5e94\u7528\u573a\u666f\u5bf9\u5c0f\u5305\u541e\u5410\u91cf\u654f\u611f\uff0c\u9700\u8981\u5728\u76f8\u5e94\u73af\u5883\u4e0b \u8fdb\u884c\u6d4b\u8bd5\u5224\u65ad\u662f\u5426\u5f00\u542f eBPF \u4f18\u5316\u3002\u6211\u4eec\u4e5f\u4f1a\u540e\u7eed\u5bf9 eBPF TCP \u5c0f\u5305\u573a\u666f\u7684\u541e\u5410\u91cf\u8fdb\u884c\u4f18\u5316\u3002 \u53c2\u8003\u8d44\u6599 \u00b6 istio-tcpip-bypass Deep Dive TCP/IP Bypass with eBPF in Service Mesh Tanzu Service Mesh Acceleration using eBPF \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 eBPF \u52a0\u901f\u8282\u70b9\u5185 TCP \u901a\u4fe1"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#ebpf-tcp","text":"\u5728\u4e00\u4e9b\u8fb9\u7f18\u548c 5G \u7684\u573a\u666f\u4e0b\uff0c\u540c\u8282\u70b9\u5185\u7684 Pod \u4e4b\u95f4\u4f1a\u8fdb\u884c\u5927\u91cf\u7684 TCP \u901a\u4fe1\uff0c\u901a\u8fc7\u4f7f\u7528 Intel \u5f00\u6e90\u7684 istio-tcpip-bypass \u9879\u76ee\uff0cPod \u53ef\u4ee5\u501f\u52a9 eBPF \u7684\u80fd\u529b\u7ed5\u8fc7\u4e3b\u673a\u7684 TCP/IP \u534f\u8bae\u6808\uff0c\u76f4\u63a5\u8fdb\u884c socket \u901a\u4fe1\uff0c\u4ece\u800c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u63d0\u5347\u541e\u5410\u91cf\u3002","title":"\u4f7f\u7528 eBPF \u52a0\u901f\u8282\u70b9\u5185 TCP \u901a\u4fe1"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_1","text":"\u5728\u5f53\u524d\u7684\u5b9e\u73b0\u4e0b\uff0c\u540c\u4e3b\u673a\u7684\u4e24\u4e2a Pod \u8fdb\u884c TCP \u8fdb\u884c\u901a\u4fe1\u9700\u8981\u7ecf\u8fc7\u5927\u91cf\u7684\u7f51\u7edc\u6808\uff0c\u5305\u62ec TCP/IP, netfilter\uff0cOVS \u7b49\u5982\u4e0b\u56fe\u6240\u793a\uff1a istio-tcpip-bypass \u63d2\u4ef6\u53ef\u4ee5\u81ea\u52a8\u5206\u6790\u5e76\u8bc6\u522b\u51fa\u540c\u4e3b\u673a\u5185\u7684 TCP \u901a\u4fe1\uff0c\u5e76\u7ed5\u8fc7\u590d\u6742\u7684\u5185\u6838\u6808\u4ece\u800c\u53ef\u4ee5\u76f4\u63a5\u8fdb\u884c socket \u95f4\u7684\u6570\u636e\u4f20\u8f93\uff0c \u6765\u964d\u4f4e\u7f51\u7edc\u6808\u5904\u7406\u5f00\u9500\uff0c\u5982\u4e0b\u56fe\u6240\u793a\uff1a \u7531\u4e8e\u8be5\u7ec4\u4ef6\u53ef\u4ee5\u81ea\u52a8\u8bc6\u522b\u540c\u4e3b\u673a\u5185\u7684 TCP \u901a\u4fe1\uff0c\u5e76\u8fdb\u884c\u4f18\u5316\u3002\u5728\u57fa\u4e8e\u4ee3\u7406\u6a21\u5f0f\u7684 Service Mesh \u73af\u5883\u4e0b\uff0c\u8be5\u7ec4\u4ef6\u4e5f\u53ef\u4ee5\u589e\u5f3a Service Mesh \u7684\u6027\u80fd\u8868\u73b0\u3002 \u66f4\u591a\u6280\u672f\u5b9e\u73b0\u7ec6\u8282\u53ef\u4ee5\u53c2\u8003 Tanzu Service Mesh Acceleration using eBPF \u3002","title":"\u57fa\u672c\u539f\u7406"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_2","text":"eBPF \u5bf9\u5185\u6838\u7248\u672c\u6709\u4e00\u5b9a\u8981\u6c42\uff0c\u63a8\u8350\u4f7f\u7528 Ubuntu 20.04 \u548c Linux 5.4.0-74-generic \u7248\u672c\u5185\u6838\u8fdb\u884c\u5b9e\u9a8c\u3002","title":"\u73af\u5883\u51c6\u5907"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_3","text":"\u5728\u540c\u4e00\u4e2a\u8282\u70b9\u4e0a\u90e8\u7f72\u4e24\u4e2a\u6027\u80fd\u6d4b\u8bd5 Pod\uff0c\u82e5\u96c6\u7fa4\u5185\u5b58\u5728\u591a\u53f0\u673a\u5668\u9700\u8981\u6307\u5b9a nodeSelector \uff1a # kubectl create deployment perf --image=kubeovn/perf:dev --replicas=2 deployment.apps/perf created # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES perf-7697bc6ddf-b2cpv 1 /1 Running 0 28s 100 .64.0.3 sealos <none> <none> perf-7697bc6ddf-p2xpt 1 /1 Running 0 28s 100 .64.0.2 sealos <none> <none> \u8fdb\u5165\u5176\u4e2d\u4e00\u4e2a Pod \u5f00\u542f qperf server\uff0c\u5728\u53e6\u4e00\u4e2a Pod \u4e2d\u542f\u52a8 qperf client \u8fdb\u884c\u6027\u80fd\u6d4b\u8bd5\uff1a # kubectl exec -it perf-7697bc6ddf-b2cpv sh / # qperf # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw \u90e8\u7f72 istio-tcpip-bypass \u63d2\u4ef6\uff1a kubectl apply -f https://raw.githubusercontent.com/intel/istio-tcpip-bypass/main/bypass-tcpip-daemonset.yaml \u518d\u6b21\u8fdb\u5165 perf client \u5bb9\u5668\u8fdb\u884c\u6027\u80fd\u6d4b\u8bd5\uff1a # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw","title":"\u5b9e\u9a8c\u6b65\u9aa4"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_4","text":"\u6839\u636e\u6d4b\u8bd5\u7ed3\u679c TCP \u5ef6\u8fdf\u5728\u4e0d\u540c\u6570\u636e\u5305\u5927\u5c0f\u7684\u60c5\u51b5\u4e0b\u4f1a\u6709 40% ~ 60% \u7684\u5ef6\u8fdf\u4e0b\u964d\uff0c\u5728\u6570\u636e\u5305\u5927\u4e8e 1024 \u5b57\u8282\u65f6\u541e\u5410\u91cf\u4f1a\u6709 40% ~ 80% \u63d0\u5347\u3002 Packet Size (byte) eBPF tcp_lat (us) Default tcp_lat (us) eBPF tcp_bw (Mb/s) Default tcp_bw(Mb/s) 1 20.2 44.5 1.36 4.27 4 20.2 48.7 5.48 16.7 16 19.6 41.6 21.7 63.5 64 18.8 41.3 96.8 201 256 19.2 36 395 539 1024 18.3 42.4 1360 846 4096 16.5 62.6 4460 2430 16384 20.2 58.8 9600 6900 \u5728\u6d4b\u8bd5\u7684\u786c\u4ef6\u73af\u5883\u4e0b\uff0c\u6570\u636e\u5305\u5c0f\u4e8e 512 \u5b57\u8282\u65f6\uff0c\u4f7f\u7528 eBPF \u4f18\u5316\u541e\u5410\u91cf\u6307\u6807\u4f1a\u4f4e\u4e8e\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7684\u541e\u5410\u91cf\u3002 \u8be5\u60c5\u51b5\u53ef\u80fd\u548c\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7f51\u5361\u5f00\u542f TCP \u805a\u5408\u4f18\u5316\u76f8\u5173\u3002\u5982\u679c\u5e94\u7528\u573a\u666f\u5bf9\u5c0f\u5305\u541e\u5410\u91cf\u654f\u611f\uff0c\u9700\u8981\u5728\u76f8\u5e94\u73af\u5883\u4e0b \u8fdb\u884c\u6d4b\u8bd5\u5224\u65ad\u662f\u5426\u5f00\u542f eBPF \u4f18\u5316\u3002\u6211\u4eec\u4e5f\u4f1a\u540e\u7eed\u5bf9 eBPF TCP \u5c0f\u5305\u573a\u666f\u7684\u541e\u5410\u91cf\u8fdb\u884c\u4f18\u5316\u3002","title":"\u6d4b\u8bd5\u7ed3\u679c"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_5","text":"istio-tcpip-bypass Deep Dive TCP/IP Bypass with eBPF in Service Mesh Tanzu Service Mesh Acceleration using eBPF \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u53c2\u8003\u8d44\u6599"},{"location":"advance/cilium-hubble-observe/","text":"Cilium \u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b \u00b6 Kube-OVN \u5f53\u524d\u5df2\u7ecf\u652f\u6301\u4e0e Cilium \u96c6\u6210\uff0c\u5177\u4f53\u64cd\u4f5c\u53ef\u4ee5\u53c2\u8003 Cilium \u96c6\u6210 \u3002 Cilium \u63d0\u4f9b\u4e86\u4e30\u5bcc\u7684\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u80fd\u529b\uff0c\u6d41\u91cf\u53ef\u89c2\u6d4b\u6027\u662f\u7531 Hubble \u63d0\u4f9b\u7684\u3002Hubble \u53ef\u4ee5\u89c2\u5bdf\u8282\u70b9\u3001\u96c6\u7fa4\u751a\u81f3\u591a\u96c6\u7fa4\u573a\u666f\u4e0b\u8de8\u96c6\u7fa4\u7684\u6d41\u91cf\u3002 \u5b89\u88c5 Hubble \u00b6 \u9ed8\u8ba4\u7684 Cilium \u96c6\u6210\u5b89\u88c5\u4e2d\uff0c\u5e76\u6ca1\u6709\u5b89\u88c5 Hubble \u76f8\u5173\u7ec4\u4ef6\uff0c\u56e0\u6b64\u8981\u652f\u6301\u6d41\u91cf\u89c2\u6d4b\uff0c\u9700\u8981\u5148\u5728\u73af\u5883\u4e0a\u8865\u5145\u5b89\u88c5 Hubble\u3002 \u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u4f7f\u7528 helm \u5b89\u88c5 Hubble\uff1a helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \u8865\u5145\u5b89\u88c5 Hubble \u4e4b\u540e\uff0c\u6267\u884c cilium status \u67e5\u770b\u7ec4\u4ef6\u72b6\u6001\uff0c\u786e\u8ba4 Hubble \u5b89\u88c5\u6210\u529f\u3002 # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: OK \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ Deployment hubble-relay Desired: 1 , Ready: 1 /1, Available: 1 /1 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment hubble-ui Desired: 1 , Ready: 1 /1, Available: 1 /1 Containers: cilium Running: 2 hubble-ui Running: 1 hubble-relay Running: 1 cilium-operator Running: 2 Cluster Pods: 16 /17 managed by Cilium Image versions hubble-relay quay.io/cilium/hubble-relay:v1.11.6@sha256:fd9034a2d04d5b973f1e8ed44f230ea195b89c37955ff32e34e5aa68f3ed675a: 1 cilium-operator quay.io/cilium/operator-generic:v1.11.6@sha256:9f6063c7bcaede801a39315ec7c166309f6a6783e98665f6693939cf1701bc17: 2 cilium quay.io/cilium/cilium:v1.11.6@sha256:f7f93c26739b6641a3fa3d76b1e1605b15989f25d06625260099e01c8243f54c: 2 hubble-ui quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1 apple@bogon cilium % \u5b89\u88c5 Hubble \u7ec4\u4ef6\u4e4b\u540e\uff0c\u9700\u8981\u5b89\u88c5\u547d\u4ee4\u884c\uff0c\u7528\u4e8e\u5728\u73af\u5883\u4e0a\u67e5\u770b\u6d41\u91cf\u4fe1\u606f\u3002 \u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u5b89\u88c5 Hubble CLI : curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/v0.10.0/hubble-linux-amd64.tar.gz sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin \u90e8\u7f72\u6d4b\u8bd5\u4e1a\u52a1 \u00b6 Cilium \u5b98\u65b9\u63d0\u4f9b\u4e86\u4e00\u4e2a\u6d41\u91cf\u6d4b\u8bd5\u7684\u90e8\u7f72\u65b9\u6848\uff0c\u53ef\u4ee5\u76f4\u63a5\u4f7f\u7528\u5b98\u65b9\u90e8\u7f72\u7684\u4e1a\u52a1\u8fdb\u884c\u6d4b\u8bd5\u3002 \u6267\u884c\u547d\u4ee4 cilium connectivity test \uff0cCilium \u4f1a\u81ea\u52a8\u521b\u5efa cilium-test \u7684 Namespace\uff0c\u540c\u65f6\u5728 cilium-test \u4e0b\u90e8\u7f72\u6d4b\u8bd5\u4e1a\u52a1\u3002 \u6b63\u5e38\u90e8\u7f72\u5b8c\u540e\uff0c\u53ef\u4ee5\u67e5\u770b cilium-test namespace \u4e0b\u7684\u8d44\u6e90\u4fe1\u606f\uff0c\u53c2\u8003\u5982\u4e0b\uff1a # kubectl get all -n cilium-test NAME READY STATUS RESTARTS AGE pod/client-7df6cfbf7b-z5t2j 1 /1 Running 0 21s pod/client2-547996d7d8-nvgxg 1 /1 Running 0 21s pod/echo-other-node-d79544ccf-hl4gg 2 /2 Running 0 21s pod/echo-same-node-5d466d5444-ml7tc 2 /2 Running 0 21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE service/echo-other-node NodePort 10 .109.58.126 <none> 8080 :32269/TCP 21s service/echo-same-node NodePort 10 .108.70.32 <none> 8080 :32490/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/client 1 /1 1 1 21s deployment.apps/client2 1 /1 1 1 21s deployment.apps/echo-other-node 1 /1 1 1 21s deployment.apps/echo-same-node 1 /1 1 1 21s NAME DESIRED CURRENT READY AGE replicaset.apps/client-7df6cfbf7b 1 1 1 21s replicaset.apps/client2-547996d7d8 1 1 1 21s replicaset.apps/echo-other-node-d79544ccf 1 1 1 21s replicaset.apps/echo-same-node-5d466d5444 1 1 1 21s \u4f7f\u7528\u547d\u4ee4\u884c\u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b \u00b6 \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u4ec5\u63d0\u4f9b\u6bcf\u4e2a\u8282\u70b9 Cilium \u4ee3\u7406\u89c2\u5bdf\u5230\u7684\u6d41\u91cf\u3002 \u53ef\u4ee5\u5728 kube-system namespace \u4e0b\u7684 Cilium \u4ee3\u7406 pod \u4e2d\u6267\u884c hubble observe \u547d\u4ee4\uff0c\u67e5\u770b\u8be5\u8282\u70b9\u4e0a\u7684\u6d41\u91cf\u4fe1\u606f\u3002 # kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cilium-d6h56 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-c7sb2 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-wj8gt 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> cilium-tq5xb 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> kube-ovn-pinger-7lgk8 1 /1 Running 0 21h 10 .16.0.19 kube-ovn-control-plane <none> <none> kube-ovn-pinger-msvcn 1 /1 Running 0 21h 10 .16.0.18 kube-ovn-worker <none> <none> # kubectl exec -it -n kube-system cilium-d6h56 -- bash root@kube-ovn-worker:/home/cilium# hubble observe --from-namespace kube-system Jul 29 03 :24:25.551: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: SYN ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.651: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.761: kube-system/kube-ovn-pinger-msvcn:52004 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn -> kube-system/kube-ovn-pinger-7lgk8 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn <- kube-system/kube-ovn-pinger-7lgk8 to-endpoint FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:80 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 -> kube-system/kube-ovn-pinger-msvcn to-endpoint FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 <- kube-system/kube-ovn-pinger-msvcn to-stack FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.979: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.3 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:26.037: kube-system/coredns-6d4b75cb6d-lbgjg:36430 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:26.282: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.2 to-stack FORWARDED ( ICMPv4 EchoRequest ) \u90e8\u7f72 Hubble Relay \u540e\uff0cHubble \u53ef\u4ee5\u63d0\u4f9b\u5b8c\u6574\u7684\u96c6\u7fa4\u8303\u56f4\u7684\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u3002 \u914d\u7f6e\u7aef\u53e3\u8f6c\u53d1 \u00b6 \u4e3a\u4e86\u80fd\u6b63\u5e38\u8bbf\u95ee Hubble API\uff0c\u9700\u8981\u521b\u5efa\u7aef\u53e3\u8f6c\u53d1\uff0c\u5c06\u672c\u5730\u8bf7\u6c42\u8f6c\u53d1\u5230 Hubble Service\u3002\u53ef\u4ee5\u6267\u884c kubectl port-forward deployment/hubble-relay -n kube-system 4245:4245 \u547d\u4ee4\uff0c\u5728\u5f53\u524d\u7ec8\u7aef\u5f00\u542f\u7aef\u53e3\u8f6c\u53d1\u3002 \u7aef\u53e3\u8f6c\u53d1\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 \u7aef\u53e3\u8f6c\u53d1 \u3002 kubectl port-forward \u547d\u4ee4\u4e0d\u4f1a\u8fd4\u56de\uff0c\u9700\u8981\u6253\u5f00\u53e6\u4e00\u4e2a\u7ec8\u7aef\u6765\u7ee7\u7eed\u6d4b\u8bd5\u3002 \u914d\u7f6e\u5b8c\u7aef\u53e3\u8f6c\u53d1\u4e4b\u540e\uff0c\u5728\u7ec8\u7aef\u6267\u884c hubble status \u547d\u4ee4\uff0c\u5982\u679c\u6709\u7c7b\u4f3c\u5982\u4e0b\u8f93\u51fa\uff0c\u5219\u7aef\u53e3\u8f6c\u53d1\u914d\u7f6e\u6b63\u786e\uff0c\u53ef\u4ee5\u4f7f\u7528\u547d\u4ee4\u884c\u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b\u3002 # hubble status Healthcheck ( via localhost:4245 ) : Ok Current/Max Flows: 8 ,190/8,190 ( 100 .00% ) Flows/s: 22 .86 Connected Nodes: 2 /2 \u547d\u4ee4\u884c\u89c2\u6d4b \u00b6 \u5728\u7ec8\u7aef\u4e0a\u6267\u884c hubble observe \u547d\u4ee4\uff0c\u67e5\u770b\u96c6\u7fa4\u7684\u6d41\u91cf\u4fe1\u606f\u3002 \u89c2\u6d4b\u5230\u7684 cilium-test \u76f8\u5173\u7684\u6d4b\u8bd5\u6d41\u91cf\u53c2\u8003\u5982\u4e0b\uff1a \u9700\u8981\u6ce8\u610f\u7684\u662f\uff0c hubble observe \u547d\u4ee4\u7684\u663e\u793a\u7ed3\u679c\uff0c\u662f\u5f53\u524d\u547d\u4ee4\u884c\u6267\u884c\u65f6\u67e5\u8be2\u5230\u7684\u6d41\u91cf\u4fe1\u606f\u3002\u591a\u6b21\u6267\u884c\u547d\u4ee4\u884c\uff0c\u53ef\u4ee5\u67e5\u770b\u5230\u4e0d\u540c\u7684\u6d41\u91cf\u4fe1\u606f\u3002 \u66f4\u591a\u8be6\u7ec6\u7684\u89c2\u6d4b\u4fe1\u606f\uff0c\u53ef\u4ee5\u6267\u884c hubble help observe \u547d\u4ee4\u67e5\u770b Hubble CLI \u7684\u8be6\u7ec6\u4f7f\u7528\u65b9\u5f0f\u3002 \u4f7f\u7528 UI \u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b \u00b6 \u6267\u884c cilium status \u547d\u4ee4\uff0c\u786e\u8ba4 Hubble UI \u5df2\u7ecf\u5b89\u88c5\u6210\u529f\u3002\u5728\u7b2c\u4e8c\u6b65\u7684 Hubble \u5b89\u88c5\u4e2d\uff0c\u5df2\u7ecf\u8865\u5145\u4e86 UI \u7684\u5b89\u88c5\u3002 \u6267\u884c\u547d\u4ee4 cilium hubble ui \u53ef\u4ee5\u81ea\u52a8\u521b\u5efa\u7aef\u53e3\u8f6c\u53d1\uff0c\u5c06 hubble-ui service \u6620\u5c04\u5230\u672c\u5730\u7aef\u53e3\u3002 \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u6267\u884c\u5b8c\u547d\u4ee4\u540e\uff0c\u4f1a\u81ea\u52a8\u6253\u5f00\u672c\u5730\u7684\u6d4f\u89c8\u5668\uff0c\u8df3\u8f6c\u5230 Hubble UI \u754c\u9762\u3002\u5982\u679c\u6ca1\u6709\u81ea\u52a8\u8df3\u8f6c\uff0c\u5728\u6d4f\u89c8\u5668\u4e2d\u8f93\u5165 http://localhost:12000 \u6253\u5f00 UI \u89c2\u5bdf\u754c\u9762\u3002 \u5728\u754c\u9762\u5de6\u4e0a\u89d2\uff0c\u9009\u62e9 cilium-test namespace\uff0c\u67e5\u770b Cilium \u63d0\u4f9b\u7684\u6d4b\u8bd5\u6d41\u91cf\u4fe1\u606f\u3002 Hubble \u6d41\u91cf\u76d1\u63a7 \u00b6 Hubble \u7ec4\u4ef6\u63d0\u4f9b\u4e86\u96c6\u7fa4\u4e2d Pod \u7f51\u7edc\u884c\u4e3a\u7684\u76d1\u63a7\uff0c\u4e3a\u4e86\u652f\u6301\u67e5\u770b Hubble \u63d0\u4f9b\u7684\u76d1\u63a7\u6570\u636e\uff0c\u9700\u8981\u4f7f\u80fd\u76d1\u63a7\u7edf\u8ba1\u3002 \u53c2\u8003\u4ee5\u4e0b\u547d\u4ee4\uff0c\u8865\u5145 hubble.metrics.enabled \u914d\u7f6e\u9879: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \\ --set hubble.metrics.enabled = \"{dns,drop,tcp,flow,icmp,http}\" \u90e8\u7f72\u4e4b\u540e\uff0c\u4f1a\u5728 kube-system namespace \u751f\u6210\u540d\u79f0\u4e3a hubble-metrics \u7684\u670d\u52a1\u3002\u901a\u8fc7\u8bbf\u95ee Endpoints \u67e5\u8be2 Hubble \u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\uff0c\u53c2\u8003\u5982\u4e0b: # curl 172.18.0.2:9091/metrics # HELP hubble_drop_total Number of drops # TYPE hubble_drop_total counter hubble_drop_total { protocol = \"ICMPv6\" ,reason = \"Unsupported L3 protocol\" } 2 # HELP hubble_flows_processed_total Total number of flows processed # TYPE hubble_flows_processed_total counter hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv6\" ,subtype = \"\" ,type = \"Drop\" ,verdict = \"DROPPED\" } 2 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 8282 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 6767 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 # HELP hubble_icmp_total Number of ICMP messages # TYPE hubble_icmp_total counter hubble_icmp_total { family = \"IPv4\" ,type = \"EchoReply\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"EchoRequest\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"RouterSolicitation\" } 2 # HELP hubble_tcp_flags_total TCP flag occurrences # TYPE hubble_tcp_flags_total counter hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"FIN\" } 2043 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"RST\" } 301 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN\" } 1169 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN-ACK\" } 1169 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium \u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#cilium","text":"Kube-OVN \u5f53\u524d\u5df2\u7ecf\u652f\u6301\u4e0e Cilium \u96c6\u6210\uff0c\u5177\u4f53\u64cd\u4f5c\u53ef\u4ee5\u53c2\u8003 Cilium \u96c6\u6210 \u3002 Cilium \u63d0\u4f9b\u4e86\u4e30\u5bcc\u7684\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u80fd\u529b\uff0c\u6d41\u91cf\u53ef\u89c2\u6d4b\u6027\u662f\u7531 Hubble \u63d0\u4f9b\u7684\u3002Hubble \u53ef\u4ee5\u89c2\u5bdf\u8282\u70b9\u3001\u96c6\u7fa4\u751a\u81f3\u591a\u96c6\u7fa4\u573a\u666f\u4e0b\u8de8\u96c6\u7fa4\u7684\u6d41\u91cf\u3002","title":"Cilium \u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#hubble","text":"\u9ed8\u8ba4\u7684 Cilium \u96c6\u6210\u5b89\u88c5\u4e2d\uff0c\u5e76\u6ca1\u6709\u5b89\u88c5 Hubble \u76f8\u5173\u7ec4\u4ef6\uff0c\u56e0\u6b64\u8981\u652f\u6301\u6d41\u91cf\u89c2\u6d4b\uff0c\u9700\u8981\u5148\u5728\u73af\u5883\u4e0a\u8865\u5145\u5b89\u88c5 Hubble\u3002 \u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u4f7f\u7528 helm \u5b89\u88c5 Hubble\uff1a helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \u8865\u5145\u5b89\u88c5 Hubble \u4e4b\u540e\uff0c\u6267\u884c cilium status \u67e5\u770b\u7ec4\u4ef6\u72b6\u6001\uff0c\u786e\u8ba4 Hubble \u5b89\u88c5\u6210\u529f\u3002 # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: OK \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ Deployment hubble-relay Desired: 1 , Ready: 1 /1, Available: 1 /1 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment hubble-ui Desired: 1 , Ready: 1 /1, Available: 1 /1 Containers: cilium Running: 2 hubble-ui Running: 1 hubble-relay Running: 1 cilium-operator Running: 2 Cluster Pods: 16 /17 managed by Cilium Image versions hubble-relay quay.io/cilium/hubble-relay:v1.11.6@sha256:fd9034a2d04d5b973f1e8ed44f230ea195b89c37955ff32e34e5aa68f3ed675a: 1 cilium-operator quay.io/cilium/operator-generic:v1.11.6@sha256:9f6063c7bcaede801a39315ec7c166309f6a6783e98665f6693939cf1701bc17: 2 cilium quay.io/cilium/cilium:v1.11.6@sha256:f7f93c26739b6641a3fa3d76b1e1605b15989f25d06625260099e01c8243f54c: 2 hubble-ui quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1 apple@bogon cilium % \u5b89\u88c5 Hubble \u7ec4\u4ef6\u4e4b\u540e\uff0c\u9700\u8981\u5b89\u88c5\u547d\u4ee4\u884c\uff0c\u7528\u4e8e\u5728\u73af\u5883\u4e0a\u67e5\u770b\u6d41\u91cf\u4fe1\u606f\u3002 \u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u5b89\u88c5 Hubble CLI : curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/v0.10.0/hubble-linux-amd64.tar.gz sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin","title":"\u5b89\u88c5 Hubble"},{"location":"advance/cilium-hubble-observe/#_1","text":"Cilium \u5b98\u65b9\u63d0\u4f9b\u4e86\u4e00\u4e2a\u6d41\u91cf\u6d4b\u8bd5\u7684\u90e8\u7f72\u65b9\u6848\uff0c\u53ef\u4ee5\u76f4\u63a5\u4f7f\u7528\u5b98\u65b9\u90e8\u7f72\u7684\u4e1a\u52a1\u8fdb\u884c\u6d4b\u8bd5\u3002 \u6267\u884c\u547d\u4ee4 cilium connectivity test \uff0cCilium \u4f1a\u81ea\u52a8\u521b\u5efa cilium-test \u7684 Namespace\uff0c\u540c\u65f6\u5728 cilium-test \u4e0b\u90e8\u7f72\u6d4b\u8bd5\u4e1a\u52a1\u3002 \u6b63\u5e38\u90e8\u7f72\u5b8c\u540e\uff0c\u53ef\u4ee5\u67e5\u770b cilium-test namespace \u4e0b\u7684\u8d44\u6e90\u4fe1\u606f\uff0c\u53c2\u8003\u5982\u4e0b\uff1a # kubectl get all -n cilium-test NAME READY STATUS RESTARTS AGE pod/client-7df6cfbf7b-z5t2j 1 /1 Running 0 21s pod/client2-547996d7d8-nvgxg 1 /1 Running 0 21s pod/echo-other-node-d79544ccf-hl4gg 2 /2 Running 0 21s pod/echo-same-node-5d466d5444-ml7tc 2 /2 Running 0 21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE service/echo-other-node NodePort 10 .109.58.126 <none> 8080 :32269/TCP 21s service/echo-same-node NodePort 10 .108.70.32 <none> 8080 :32490/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/client 1 /1 1 1 21s deployment.apps/client2 1 /1 1 1 21s deployment.apps/echo-other-node 1 /1 1 1 21s deployment.apps/echo-same-node 1 /1 1 1 21s NAME DESIRED CURRENT READY AGE replicaset.apps/client-7df6cfbf7b 1 1 1 21s replicaset.apps/client2-547996d7d8 1 1 1 21s replicaset.apps/echo-other-node-d79544ccf 1 1 1 21s replicaset.apps/echo-same-node-5d466d5444 1 1 1 21s","title":"\u90e8\u7f72\u6d4b\u8bd5\u4e1a\u52a1"},{"location":"advance/cilium-hubble-observe/#_2","text":"\u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u4ec5\u63d0\u4f9b\u6bcf\u4e2a\u8282\u70b9 Cilium \u4ee3\u7406\u89c2\u5bdf\u5230\u7684\u6d41\u91cf\u3002 \u53ef\u4ee5\u5728 kube-system namespace \u4e0b\u7684 Cilium \u4ee3\u7406 pod \u4e2d\u6267\u884c hubble observe \u547d\u4ee4\uff0c\u67e5\u770b\u8be5\u8282\u70b9\u4e0a\u7684\u6d41\u91cf\u4fe1\u606f\u3002 # kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cilium-d6h56 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-c7sb2 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-wj8gt 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> cilium-tq5xb 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> kube-ovn-pinger-7lgk8 1 /1 Running 0 21h 10 .16.0.19 kube-ovn-control-plane <none> <none> kube-ovn-pinger-msvcn 1 /1 Running 0 21h 10 .16.0.18 kube-ovn-worker <none> <none> # kubectl exec -it -n kube-system cilium-d6h56 -- bash root@kube-ovn-worker:/home/cilium# hubble observe --from-namespace kube-system Jul 29 03 :24:25.551: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: SYN ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.651: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.761: kube-system/kube-ovn-pinger-msvcn:52004 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn -> kube-system/kube-ovn-pinger-7lgk8 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn <- kube-system/kube-ovn-pinger-7lgk8 to-endpoint FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:80 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 -> kube-system/kube-ovn-pinger-msvcn to-endpoint FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 <- kube-system/kube-ovn-pinger-msvcn to-stack FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.979: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.3 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:26.037: kube-system/coredns-6d4b75cb6d-lbgjg:36430 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:26.282: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.2 to-stack FORWARDED ( ICMPv4 EchoRequest ) \u90e8\u7f72 Hubble Relay \u540e\uff0cHubble \u53ef\u4ee5\u63d0\u4f9b\u5b8c\u6574\u7684\u96c6\u7fa4\u8303\u56f4\u7684\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u3002","title":"\u4f7f\u7528\u547d\u4ee4\u884c\u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#_3","text":"\u4e3a\u4e86\u80fd\u6b63\u5e38\u8bbf\u95ee Hubble API\uff0c\u9700\u8981\u521b\u5efa\u7aef\u53e3\u8f6c\u53d1\uff0c\u5c06\u672c\u5730\u8bf7\u6c42\u8f6c\u53d1\u5230 Hubble Service\u3002\u53ef\u4ee5\u6267\u884c kubectl port-forward deployment/hubble-relay -n kube-system 4245:4245 \u547d\u4ee4\uff0c\u5728\u5f53\u524d\u7ec8\u7aef\u5f00\u542f\u7aef\u53e3\u8f6c\u53d1\u3002 \u7aef\u53e3\u8f6c\u53d1\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 \u7aef\u53e3\u8f6c\u53d1 \u3002 kubectl port-forward \u547d\u4ee4\u4e0d\u4f1a\u8fd4\u56de\uff0c\u9700\u8981\u6253\u5f00\u53e6\u4e00\u4e2a\u7ec8\u7aef\u6765\u7ee7\u7eed\u6d4b\u8bd5\u3002 \u914d\u7f6e\u5b8c\u7aef\u53e3\u8f6c\u53d1\u4e4b\u540e\uff0c\u5728\u7ec8\u7aef\u6267\u884c hubble status \u547d\u4ee4\uff0c\u5982\u679c\u6709\u7c7b\u4f3c\u5982\u4e0b\u8f93\u51fa\uff0c\u5219\u7aef\u53e3\u8f6c\u53d1\u914d\u7f6e\u6b63\u786e\uff0c\u53ef\u4ee5\u4f7f\u7528\u547d\u4ee4\u884c\u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b\u3002 # hubble status Healthcheck ( via localhost:4245 ) : Ok Current/Max Flows: 8 ,190/8,190 ( 100 .00% ) Flows/s: 22 .86 Connected Nodes: 2 /2","title":"\u914d\u7f6e\u7aef\u53e3\u8f6c\u53d1"},{"location":"advance/cilium-hubble-observe/#_4","text":"\u5728\u7ec8\u7aef\u4e0a\u6267\u884c hubble observe \u547d\u4ee4\uff0c\u67e5\u770b\u96c6\u7fa4\u7684\u6d41\u91cf\u4fe1\u606f\u3002 \u89c2\u6d4b\u5230\u7684 cilium-test \u76f8\u5173\u7684\u6d4b\u8bd5\u6d41\u91cf\u53c2\u8003\u5982\u4e0b\uff1a \u9700\u8981\u6ce8\u610f\u7684\u662f\uff0c hubble observe \u547d\u4ee4\u7684\u663e\u793a\u7ed3\u679c\uff0c\u662f\u5f53\u524d\u547d\u4ee4\u884c\u6267\u884c\u65f6\u67e5\u8be2\u5230\u7684\u6d41\u91cf\u4fe1\u606f\u3002\u591a\u6b21\u6267\u884c\u547d\u4ee4\u884c\uff0c\u53ef\u4ee5\u67e5\u770b\u5230\u4e0d\u540c\u7684\u6d41\u91cf\u4fe1\u606f\u3002 \u66f4\u591a\u8be6\u7ec6\u7684\u89c2\u6d4b\u4fe1\u606f\uff0c\u53ef\u4ee5\u6267\u884c hubble help observe \u547d\u4ee4\u67e5\u770b Hubble CLI \u7684\u8be6\u7ec6\u4f7f\u7528\u65b9\u5f0f\u3002","title":"\u547d\u4ee4\u884c\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#ui","text":"\u6267\u884c cilium status \u547d\u4ee4\uff0c\u786e\u8ba4 Hubble UI \u5df2\u7ecf\u5b89\u88c5\u6210\u529f\u3002\u5728\u7b2c\u4e8c\u6b65\u7684 Hubble \u5b89\u88c5\u4e2d\uff0c\u5df2\u7ecf\u8865\u5145\u4e86 UI \u7684\u5b89\u88c5\u3002 \u6267\u884c\u547d\u4ee4 cilium hubble ui \u53ef\u4ee5\u81ea\u52a8\u521b\u5efa\u7aef\u53e3\u8f6c\u53d1\uff0c\u5c06 hubble-ui service \u6620\u5c04\u5230\u672c\u5730\u7aef\u53e3\u3002 \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u6267\u884c\u5b8c\u547d\u4ee4\u540e\uff0c\u4f1a\u81ea\u52a8\u6253\u5f00\u672c\u5730\u7684\u6d4f\u89c8\u5668\uff0c\u8df3\u8f6c\u5230 Hubble UI \u754c\u9762\u3002\u5982\u679c\u6ca1\u6709\u81ea\u52a8\u8df3\u8f6c\uff0c\u5728\u6d4f\u89c8\u5668\u4e2d\u8f93\u5165 http://localhost:12000 \u6253\u5f00 UI \u89c2\u5bdf\u754c\u9762\u3002 \u5728\u754c\u9762\u5de6\u4e0a\u89d2\uff0c\u9009\u62e9 cilium-test namespace\uff0c\u67e5\u770b Cilium \u63d0\u4f9b\u7684\u6d4b\u8bd5\u6d41\u91cf\u4fe1\u606f\u3002","title":"\u4f7f\u7528 UI \u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#hubble_1","text":"Hubble \u7ec4\u4ef6\u63d0\u4f9b\u4e86\u96c6\u7fa4\u4e2d Pod \u7f51\u7edc\u884c\u4e3a\u7684\u76d1\u63a7\uff0c\u4e3a\u4e86\u652f\u6301\u67e5\u770b Hubble \u63d0\u4f9b\u7684\u76d1\u63a7\u6570\u636e\uff0c\u9700\u8981\u4f7f\u80fd\u76d1\u63a7\u7edf\u8ba1\u3002 \u53c2\u8003\u4ee5\u4e0b\u547d\u4ee4\uff0c\u8865\u5145 hubble.metrics.enabled \u914d\u7f6e\u9879: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \\ --set hubble.metrics.enabled = \"{dns,drop,tcp,flow,icmp,http}\" \u90e8\u7f72\u4e4b\u540e\uff0c\u4f1a\u5728 kube-system namespace \u751f\u6210\u540d\u79f0\u4e3a hubble-metrics \u7684\u670d\u52a1\u3002\u901a\u8fc7\u8bbf\u95ee Endpoints \u67e5\u8be2 Hubble \u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\uff0c\u53c2\u8003\u5982\u4e0b: # curl 172.18.0.2:9091/metrics # HELP hubble_drop_total Number of drops # TYPE hubble_drop_total counter hubble_drop_total { protocol = \"ICMPv6\" ,reason = \"Unsupported L3 protocol\" } 2 # HELP hubble_flows_processed_total Total number of flows processed # TYPE hubble_flows_processed_total counter hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv6\" ,subtype = \"\" ,type = \"Drop\" ,verdict = \"DROPPED\" } 2 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 8282 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 6767 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 # HELP hubble_icmp_total Number of ICMP messages # TYPE hubble_icmp_total counter hubble_icmp_total { family = \"IPv4\" ,type = \"EchoReply\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"EchoRequest\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"RouterSolicitation\" } 2 # HELP hubble_tcp_flags_total TCP flag occurrences # TYPE hubble_tcp_flags_total counter hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"FIN\" } 2043 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"RST\" } 301 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN\" } 1169 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN-ACK\" } 1169 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Hubble \u6d41\u91cf\u76d1\u63a7"},{"location":"advance/cilium-networkpolicy/","text":"Cilium NetworkPolicy \u652f\u6301 \u00b6 Kube-OVN \u5f53\u524d\u5df2\u7ecf\u652f\u6301\u4e0e Cilium \u96c6\u6210\uff0c\u5177\u4f53\u64cd\u4f5c\u53ef\u4ee5\u53c2\u8003 Cilium \u96c6\u6210 \u3002 \u5728\u96c6\u6210 Cilium \u4e4b\u540e\uff0c\u5c31\u53ef\u4ee5\u4f7f\u7528 Cilium \u4f18\u79c0\u7684\u7f51\u7edc\u7b56\u7565\u80fd\u529b\uff0c\u5b9e\u73b0\u5bf9\u6d41\u91cf\u8bbf\u95ee\u7684\u63a7\u5236\u3002\u4ee5\u4e0b\u6587\u6863\u63d0\u4f9b\u4e86\u5bf9 Cilium L3 \u548c L4 \u7f51\u7edc\u7b56\u7565\u80fd\u529b\u7684\u96c6\u6210\u9a8c\u8bc1\u3002 \u9a8c\u8bc1\u6b65\u9aa4 \u00b6 \u521b\u5efa\u6d4b\u8bd5 Pod \u00b6 \u521b\u5efa namespace test \u3002\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u5728 test namespace \u4e2d\u521b\u5efa\u6307\u5b9a label app=test \u7684 Pod\uff0c\u4f5c\u4e3a\u6d4b\u8bd5\u8bbf\u95ee\u7684\u76ee\u7684 Pod\u3002 apiVersion : apps/v1 kind : Deployment metadata : labels : app : test name : test namespace : test spec : replicas : 1 selector : matchLabels : app : test strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : test spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx \u540c\u6837\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u5728 default namespace \u4e0b\u521b\u5efa\u6307\u5b9a label app=dynamic \u7684 Pod \u4e3a\u53d1\u8d77\u8bbf\u95ee\u6d4b\u8bd5\u7684 Pod\u3002 apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx \u67e5\u770b\u6d4b\u8bd5 Pod \u4ee5\u53ca Label \u4fe1\u606f: # kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-9v5c4 1 /1 Running 0 28h 10 .16.0.35 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-s8z2n 1 /1 Running 0 28h 10 .16.0.36 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 # kubectl get pod -o wide -n test --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-6dsg6 1 /1 Running 0 7h20m 10 .16.0.2 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-tjgtp 1 /1 Running 0 7h46m 10 .16.0.42 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 label-test1-77b6764857-swq4k 1 /1 Running 0 3h43m 10 .16.0.12 kube-ovn-worker <none> <none> app = test1,pod-template-hash = 77b6764857 // \u4ee5\u4e0b\u4e3a\u6d4b\u8bd5\u8bbf\u95ee\u76ee\u7684 Pod test-54c98bc466-mft5s 1 /1 Running 0 8h 10 .16.0.41 kube-ovn-worker <none> <none> app = test,pod-template-hash = 54c98bc466 L3 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa CiliumNetworkPolicy \u8d44\u6e90: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l3-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic \u5728 default namespace \u4e0b\u7684\u6d4b\u8bd5 Pod \u4e2d\uff0c\u53d1\u8d77\u5bf9\u76ee\u7684 Pod \u7684\u8bbf\u95ee\uff0c\u7ed3\u679c\u8bbf\u95ee\u4e0d\u901a\u3002 \u4f46\u662f\u5728 test namespace \u4e0b\uff0c\u6d4b\u8bd5\u5230\u76ee\u7684 Pod \u7684\u8bbf\u95ee\uff0c\u6d4b\u8bd5\u6b63\u5e38\u3002 default namespace \u4e0b\u6d4b\u8bd5\u7ed3\u679c: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss test namepsace \u4e0b Pod \u7684\u6d4b\u8bd5\uff0c\u8bbf\u95ee\u6b63\u5e38: # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .558 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .223 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .304 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .223/1.028/2.558 ms \u67e5\u770b Cilium \u5b98\u65b9\u6587\u6863\u89e3\u91ca\uff0c CiliumNetworkPolicy \u8d44\u6e90\u5c06\u9650\u5236\u63a7\u5236\u5728\u4e86 Namespace \u7ea7\u522b\u3002\u8be6\u7ec6\u4fe1\u606f\u53ef\u4ee5\u67e5\u770b Cilium \u9650\u5236 \u3002 \u5728\u6709\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u5339\u914d\u7684\u60c5\u51b5\u4e0b\uff0c\u53ea\u6709 \u540c\u4e00\u4e2a Namespace \u7684 Pod \uff0c\u624d\u53ef\u4ee5\u6309\u7167\u89c4\u5219\u8fdb\u884c\u8bbf\u95ee\uff0c\u9ed8\u8ba4\u62d2\u7edd \u5176\u4ed6 Namespace \u7684 Pod \u8fdb\u884c\u8bbf\u95ee\u3002 \u5982\u679c\u60f3\u5b9e\u73b0\u8de8 Namespace \u7684\u8bbf\u95ee\uff0c\u9700\u8981\u5728\u89c4\u5219\u4e2d\u660e\u786e\u6307\u5b9a Namespace \u4fe1\u606f\u3002 \u53c2\u8003\u6587\u6863\uff0c\u4fee\u6539 CiliumNetworkPolicy \u8d44\u6e90\uff0c\u589e\u52a0 namespace \u4fe1\u606f: ingress : - fromEndpoints : - matchLabels : app : dynamic k8s:io.kubernetes.pod.namespace : default // \u63a7\u5236\u5176\u4ed6 Namespace \u4e0b\u7684 Pod \u8bbf\u95ee \u67e5\u770b\u4fee\u6539\u540e\u7684 CiliumNetworkPolicy \u8d44\u6e90\u4fe1\u606f: # kubectl get cnp -n test -o yaml l3-rule apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: l3-rule namespace: test spec: endpointSelector: matchLabels: app: test ingress: - fromEndpoints: - matchLabels: app: dynamic - matchLabels: app: dynamic k8s:io.kubernetes.pod.namespace: default \u518d\u6b21\u6d4b\u8bd5 default namespace \u4e0b\u7684 Pod \u8bbf\u95ee\uff0c\u76ee\u7684 Pod \u8bbf\u95ee\u6b63\u5e38: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -n test -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .383 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .115 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .142 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .115/0.880/2.383 ms \u4f7f\u7528\u6807\u51c6\u7684 Kubernetes \u7f51\u7edc\u7b56\u7565 networkpolicy \uff0c\u6d4b\u8bd5\u7ed3\u679c\u663e\u793a Cilium \u540c\u6837\u5c06\u8bbf\u95ee\u9650\u5236\u5728\u540c\u4e00\u4e2a Namespace \u5185\uff0c\u8de8 Namespace \u7684\u8bbf\u95ee\u662f\u7981\u6b62\u7684\u3002 \u8fd9\u70b9\u4e0e Kube-OVN \u5b9e\u73b0\u662f\u4e0d\u540c\u7684\u3002Kube-OVN \u652f\u6301\u6807\u51c6\u7684 k8s \u7f51\u7edc\u7b56\u7565\uff0c\u9650\u5236\u4e86\u5177\u4f53 Namespace \u4e0b\u7684 \u76ee\u7684 Pod \uff0c\u4f46\u662f\u5bf9\u6e90\u5730\u5740 Pod\uff0c\u662f\u6ca1\u6709 Namespace \u9650\u5236\u7684\uff0c\u4efb\u4f55 Namespace \u4e0b\u7b26\u5408\u9650\u5236\u89c4\u5219\u7684 Pod\uff0c\u90fd\u53ef\u4ee5\u5b9e\u73b0\u5bf9\u76ee\u7684 Pod \u7684\u8bbf\u95ee\u3002 L4 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa L4 \u5c42\u7684\u7f51\u7edc\u7b56\u7565\u8d44\u6e90: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l4-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic toPorts : - ports : - port : \"80\" protocol : TCP \u6d4b\u8bd5\u76f8\u540c Namespace \u4e0b\uff0c\u7b26\u5408\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u7684 Pod \u7684\u8bbf\u95ee # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl 10 .16.0.41:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on test-54c98bc466-mft5s</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u76f8\u540c Namespace \u4e0b\uff0c\u4e0d\u7b26\u5408\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u7684 Pod \u8bbf\u95ee\u6d4b\u8bd5 # kubectl exec -it -n test label-test1-77b6764857-swq4k -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl -v 10 .16.0.41:80 --connect-timeout 10 * Trying 10 .16.0.41:80... * After 10000ms connect time, move on! * connect to 10 .16.0.41 port 80 failed: Operation timed out * Connection timeout after 10001 ms * Closing connection 0 curl: ( 28 ) Connection timeout after 10001 ms \u7f51\u7edc\u7b56\u7565\u751f\u6548\u540e\uff0c\u8de8 Namespace \u7684\u8bbf\u95ee\uff0c\u4f9d\u7136\u662f\u88ab\u7981\u6b62\u7684\uff0c\u8ddf L3 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5\u7ed3\u679c\u4e00\u81f4\u3002 \u5728 L4 \u7f51\u7edc\u89c4\u5219\u751f\u6548\u540e\uff0cping \u65e0\u6cd5\u4f7f\u7528\uff0c\u4f46\u662f\u7b26\u5408\u7b56\u7565\u89c4\u5219\u7684 TCP \u8bbf\u95ee\uff0c\u662f\u53ef\u4ee5\u6b63\u5e38\u6267\u884c\u7684\u3002 \u5173\u4e8e ICMP \u7684\u9650\u5236\uff0c\u53ef\u4ee5\u53c2\u8003\u5b98\u65b9\u8bf4\u660e L4 \u9650\u5236\u8bf4\u660e \u3002 L7 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5 \u00b6 chaining \u6a21\u5f0f\u4e0b\uff0cL7 \u7f51\u7edc\u7b56\u7565\u76ee\u524d\u662f\u5b58\u5728\u95ee\u9898\u7684\u3002\u5728 Cilium \u5b98\u65b9\u6587\u6863\u4e2d\uff0c\u5bf9\u8fd9\u79cd\u60c5\u51b5\u7ed9\u51fa\u4e86\u8bf4\u660e\uff0c\u53c2\u8003 Generic Veth Chaining \u3002 \u8fd9\u4e2a\u95ee\u9898\u4f7f\u7528 issue 12454 \u8ddf\u8e2a\uff0c\u76ee\u524d\u8fd8\u6ca1\u6709\u89e3\u51b3\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium NetworkPolicy \u652f\u6301"},{"location":"advance/cilium-networkpolicy/#cilium-networkpolicy","text":"Kube-OVN \u5f53\u524d\u5df2\u7ecf\u652f\u6301\u4e0e Cilium \u96c6\u6210\uff0c\u5177\u4f53\u64cd\u4f5c\u53ef\u4ee5\u53c2\u8003 Cilium \u96c6\u6210 \u3002 \u5728\u96c6\u6210 Cilium \u4e4b\u540e\uff0c\u5c31\u53ef\u4ee5\u4f7f\u7528 Cilium \u4f18\u79c0\u7684\u7f51\u7edc\u7b56\u7565\u80fd\u529b\uff0c\u5b9e\u73b0\u5bf9\u6d41\u91cf\u8bbf\u95ee\u7684\u63a7\u5236\u3002\u4ee5\u4e0b\u6587\u6863\u63d0\u4f9b\u4e86\u5bf9 Cilium L3 \u548c L4 \u7f51\u7edc\u7b56\u7565\u80fd\u529b\u7684\u96c6\u6210\u9a8c\u8bc1\u3002","title":"Cilium NetworkPolicy \u652f\u6301"},{"location":"advance/cilium-networkpolicy/#_1","text":"","title":"\u9a8c\u8bc1\u6b65\u9aa4"},{"location":"advance/cilium-networkpolicy/#pod","text":"\u521b\u5efa namespace test \u3002\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u5728 test namespace \u4e2d\u521b\u5efa\u6307\u5b9a label app=test \u7684 Pod\uff0c\u4f5c\u4e3a\u6d4b\u8bd5\u8bbf\u95ee\u7684\u76ee\u7684 Pod\u3002 apiVersion : apps/v1 kind : Deployment metadata : labels : app : test name : test namespace : test spec : replicas : 1 selector : matchLabels : app : test strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : test spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx \u540c\u6837\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u5728 default namespace \u4e0b\u521b\u5efa\u6307\u5b9a label app=dynamic \u7684 Pod \u4e3a\u53d1\u8d77\u8bbf\u95ee\u6d4b\u8bd5\u7684 Pod\u3002 apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx \u67e5\u770b\u6d4b\u8bd5 Pod \u4ee5\u53ca Label \u4fe1\u606f: # kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-9v5c4 1 /1 Running 0 28h 10 .16.0.35 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-s8z2n 1 /1 Running 0 28h 10 .16.0.36 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 # kubectl get pod -o wide -n test --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-6dsg6 1 /1 Running 0 7h20m 10 .16.0.2 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-tjgtp 1 /1 Running 0 7h46m 10 .16.0.42 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 label-test1-77b6764857-swq4k 1 /1 Running 0 3h43m 10 .16.0.12 kube-ovn-worker <none> <none> app = test1,pod-template-hash = 77b6764857 // \u4ee5\u4e0b\u4e3a\u6d4b\u8bd5\u8bbf\u95ee\u76ee\u7684 Pod test-54c98bc466-mft5s 1 /1 Running 0 8h 10 .16.0.41 kube-ovn-worker <none> <none> app = test,pod-template-hash = 54c98bc466","title":"\u521b\u5efa\u6d4b\u8bd5 Pod"},{"location":"advance/cilium-networkpolicy/#l3","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa CiliumNetworkPolicy \u8d44\u6e90: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l3-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic \u5728 default namespace \u4e0b\u7684\u6d4b\u8bd5 Pod \u4e2d\uff0c\u53d1\u8d77\u5bf9\u76ee\u7684 Pod \u7684\u8bbf\u95ee\uff0c\u7ed3\u679c\u8bbf\u95ee\u4e0d\u901a\u3002 \u4f46\u662f\u5728 test namespace \u4e0b\uff0c\u6d4b\u8bd5\u5230\u76ee\u7684 Pod \u7684\u8bbf\u95ee\uff0c\u6d4b\u8bd5\u6b63\u5e38\u3002 default namespace \u4e0b\u6d4b\u8bd5\u7ed3\u679c: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss test namepsace \u4e0b Pod \u7684\u6d4b\u8bd5\uff0c\u8bbf\u95ee\u6b63\u5e38: # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .558 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .223 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .304 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .223/1.028/2.558 ms \u67e5\u770b Cilium \u5b98\u65b9\u6587\u6863\u89e3\u91ca\uff0c CiliumNetworkPolicy \u8d44\u6e90\u5c06\u9650\u5236\u63a7\u5236\u5728\u4e86 Namespace \u7ea7\u522b\u3002\u8be6\u7ec6\u4fe1\u606f\u53ef\u4ee5\u67e5\u770b Cilium \u9650\u5236 \u3002 \u5728\u6709\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u5339\u914d\u7684\u60c5\u51b5\u4e0b\uff0c\u53ea\u6709 \u540c\u4e00\u4e2a Namespace \u7684 Pod \uff0c\u624d\u53ef\u4ee5\u6309\u7167\u89c4\u5219\u8fdb\u884c\u8bbf\u95ee\uff0c\u9ed8\u8ba4\u62d2\u7edd \u5176\u4ed6 Namespace \u7684 Pod \u8fdb\u884c\u8bbf\u95ee\u3002 \u5982\u679c\u60f3\u5b9e\u73b0\u8de8 Namespace \u7684\u8bbf\u95ee\uff0c\u9700\u8981\u5728\u89c4\u5219\u4e2d\u660e\u786e\u6307\u5b9a Namespace \u4fe1\u606f\u3002 \u53c2\u8003\u6587\u6863\uff0c\u4fee\u6539 CiliumNetworkPolicy \u8d44\u6e90\uff0c\u589e\u52a0 namespace \u4fe1\u606f: ingress : - fromEndpoints : - matchLabels : app : dynamic k8s:io.kubernetes.pod.namespace : default // \u63a7\u5236\u5176\u4ed6 Namespace \u4e0b\u7684 Pod \u8bbf\u95ee \u67e5\u770b\u4fee\u6539\u540e\u7684 CiliumNetworkPolicy \u8d44\u6e90\u4fe1\u606f: # kubectl get cnp -n test -o yaml l3-rule apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: l3-rule namespace: test spec: endpointSelector: matchLabels: app: test ingress: - fromEndpoints: - matchLabels: app: dynamic - matchLabels: app: dynamic k8s:io.kubernetes.pod.namespace: default \u518d\u6b21\u6d4b\u8bd5 default namespace \u4e0b\u7684 Pod \u8bbf\u95ee\uff0c\u76ee\u7684 Pod \u8bbf\u95ee\u6b63\u5e38: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -n test -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .383 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .115 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .142 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .115/0.880/2.383 ms \u4f7f\u7528\u6807\u51c6\u7684 Kubernetes \u7f51\u7edc\u7b56\u7565 networkpolicy \uff0c\u6d4b\u8bd5\u7ed3\u679c\u663e\u793a Cilium \u540c\u6837\u5c06\u8bbf\u95ee\u9650\u5236\u5728\u540c\u4e00\u4e2a Namespace \u5185\uff0c\u8de8 Namespace \u7684\u8bbf\u95ee\u662f\u7981\u6b62\u7684\u3002 \u8fd9\u70b9\u4e0e Kube-OVN \u5b9e\u73b0\u662f\u4e0d\u540c\u7684\u3002Kube-OVN \u652f\u6301\u6807\u51c6\u7684 k8s \u7f51\u7edc\u7b56\u7565\uff0c\u9650\u5236\u4e86\u5177\u4f53 Namespace \u4e0b\u7684 \u76ee\u7684 Pod \uff0c\u4f46\u662f\u5bf9\u6e90\u5730\u5740 Pod\uff0c\u662f\u6ca1\u6709 Namespace \u9650\u5236\u7684\uff0c\u4efb\u4f55 Namespace \u4e0b\u7b26\u5408\u9650\u5236\u89c4\u5219\u7684 Pod\uff0c\u90fd\u53ef\u4ee5\u5b9e\u73b0\u5bf9\u76ee\u7684 Pod \u7684\u8bbf\u95ee\u3002","title":"L3 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5"},{"location":"advance/cilium-networkpolicy/#l4","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa L4 \u5c42\u7684\u7f51\u7edc\u7b56\u7565\u8d44\u6e90: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l4-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic toPorts : - ports : - port : \"80\" protocol : TCP \u6d4b\u8bd5\u76f8\u540c Namespace \u4e0b\uff0c\u7b26\u5408\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u7684 Pod \u7684\u8bbf\u95ee # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl 10 .16.0.41:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on test-54c98bc466-mft5s</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u76f8\u540c Namespace \u4e0b\uff0c\u4e0d\u7b26\u5408\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u7684 Pod \u8bbf\u95ee\u6d4b\u8bd5 # kubectl exec -it -n test label-test1-77b6764857-swq4k -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl -v 10 .16.0.41:80 --connect-timeout 10 * Trying 10 .16.0.41:80... * After 10000ms connect time, move on! * connect to 10 .16.0.41 port 80 failed: Operation timed out * Connection timeout after 10001 ms * Closing connection 0 curl: ( 28 ) Connection timeout after 10001 ms \u7f51\u7edc\u7b56\u7565\u751f\u6548\u540e\uff0c\u8de8 Namespace \u7684\u8bbf\u95ee\uff0c\u4f9d\u7136\u662f\u88ab\u7981\u6b62\u7684\uff0c\u8ddf L3 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5\u7ed3\u679c\u4e00\u81f4\u3002 \u5728 L4 \u7f51\u7edc\u89c4\u5219\u751f\u6548\u540e\uff0cping \u65e0\u6cd5\u4f7f\u7528\uff0c\u4f46\u662f\u7b26\u5408\u7b56\u7565\u89c4\u5219\u7684 TCP \u8bbf\u95ee\uff0c\u662f\u53ef\u4ee5\u6b63\u5e38\u6267\u884c\u7684\u3002 \u5173\u4e8e ICMP \u7684\u9650\u5236\uff0c\u53ef\u4ee5\u53c2\u8003\u5b98\u65b9\u8bf4\u660e L4 \u9650\u5236\u8bf4\u660e \u3002","title":"L4 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5"},{"location":"advance/cilium-networkpolicy/#l7","text":"chaining \u6a21\u5f0f\u4e0b\uff0cL7 \u7f51\u7edc\u7b56\u7565\u76ee\u524d\u662f\u5b58\u5728\u95ee\u9898\u7684\u3002\u5728 Cilium \u5b98\u65b9\u6587\u6863\u4e2d\uff0c\u5bf9\u8fd9\u79cd\u60c5\u51b5\u7ed9\u51fa\u4e86\u8bf4\u660e\uff0c\u53c2\u8003 Generic Veth Chaining \u3002 \u8fd9\u4e2a\u95ee\u9898\u4f7f\u7528 issue 12454 \u8ddf\u8e2a\uff0c\u76ee\u524d\u8fd8\u6ca1\u6709\u89e3\u51b3\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"L7 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5"},{"location":"advance/dhcp/","text":"DHCP \u8bbe\u7f6e \u00b6 \u5728\u4f7f\u7528 SR-IOV \u6216 DPDK \u7c7b\u578b\u7f51\u7edc\u65f6\uff0cKubeVirt \u5185\u7f6e\u7684 DHCP \u65e0\u6cd5\u5728\u8be5\u7f51\u7edc\u6a21\u5f0f\u4e0b\u5de5\u4f5c\u3002Kube-OVN \u53ef\u4ee5\u5229\u7528 OVN \u7684 DHCP \u80fd\u529b\u5728\u5b50\u7f51\u7ea7\u522b\u8bbe\u7f6e DHCP \u9009\u9879\uff0c\u4ece\u800c\u5e2e\u52a9\u8be5\u7f51\u7edc\u7c7b\u578b\u7684 KubeVirt \u865a\u673a\u6b63\u5e38\u4f7f\u7528 DHCP \u83b7\u5f97\u5206\u914d\u7684 IP \u5730\u5740\u3002Kube-OVN \u540c\u65f6\u652f\u6301 DHCPv4 \u548c DHCPv6\u3002 \u5b50\u7f51 DHCP \u7684\u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : sn-dual spec : cidrBlock : \"10.0.0.0/24,240e::a00/120\" default : false disableGatewayCheck : true disableInterConnection : false excludeIps : - 10.0.0.1 - 240e::a01 gateway : 10.0.0.1,240e::a01 gatewayNode : '' gatewayType : distributed natOutgoing : false private : false protocol : Dual provider : ovn vpc : vpc-test enableDHCP : true dhcpV4Options : \"lease_time=3600,router=10.0.0.1,server_id=169.254.0.254,server_mac=00:00:00:2E:2F:B8\" dhcpV6Options : \"server_id=00:00:00:2E:2F:C5\" enableIPv6RA : true ipv6RAConfigs : \"address_mode=dhcpv6_stateful,max_interval=30,min_interval=5,send_periodic=true\" enableDHCP : \u662f\u5426\u5f00\u542f\u5b50\u7f51\u7684 DHCP \u529f\u80fd\u3002 dhcpV4Options , dhcpV6Options : \u8be5\u5b57\u6bb5\u76f4\u63a5\u66b4\u9732 ovn-nb \u5185 DHCP \u76f8\u5173\u9009\u9879\uff0c\u8bf7\u53c2\u8003 DHCP Options \u3002 \u9ed8\u8ba4\u503c\u5206\u522b\u4e3a \"lease_time=3600, router=$ipv4_gateway, server_id=169.254.0.254, server_mac=$random_mac\" \u548c server_id=$random_mac \u3002 enableIPv6RA : \u662f\u5426\u5f00\u542f DHCPv6 \u7684\u8def\u7531\u5e7f\u64ad\u529f\u80fd\u3002 ipv6RAConfigs \uff1a\u8be5\u5b57\u6bb5\u76f4\u63a5\u66b4\u9732 ovn-nb \u5185 Logical_Router_Port \u76f8\u5173\u9009\u9879\uff0c\u8bf7\u53c2\u8003 Logical Router Port \u9ed8\u8ba4\u503c\u4e3a address_mode=dhcpv6_stateful, max_interval=30, min_interval=5, send_periodic=true \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DHCP \u8bbe\u7f6e"},{"location":"advance/dhcp/#dhcp","text":"\u5728\u4f7f\u7528 SR-IOV \u6216 DPDK \u7c7b\u578b\u7f51\u7edc\u65f6\uff0cKubeVirt \u5185\u7f6e\u7684 DHCP \u65e0\u6cd5\u5728\u8be5\u7f51\u7edc\u6a21\u5f0f\u4e0b\u5de5\u4f5c\u3002Kube-OVN \u53ef\u4ee5\u5229\u7528 OVN \u7684 DHCP \u80fd\u529b\u5728\u5b50\u7f51\u7ea7\u522b\u8bbe\u7f6e DHCP \u9009\u9879\uff0c\u4ece\u800c\u5e2e\u52a9\u8be5\u7f51\u7edc\u7c7b\u578b\u7684 KubeVirt \u865a\u673a\u6b63\u5e38\u4f7f\u7528 DHCP \u83b7\u5f97\u5206\u914d\u7684 IP \u5730\u5740\u3002Kube-OVN \u540c\u65f6\u652f\u6301 DHCPv4 \u548c DHCPv6\u3002 \u5b50\u7f51 DHCP \u7684\u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : sn-dual spec : cidrBlock : \"10.0.0.0/24,240e::a00/120\" default : false disableGatewayCheck : true disableInterConnection : false excludeIps : - 10.0.0.1 - 240e::a01 gateway : 10.0.0.1,240e::a01 gatewayNode : '' gatewayType : distributed natOutgoing : false private : false protocol : Dual provider : ovn vpc : vpc-test enableDHCP : true dhcpV4Options : \"lease_time=3600,router=10.0.0.1,server_id=169.254.0.254,server_mac=00:00:00:2E:2F:B8\" dhcpV6Options : \"server_id=00:00:00:2E:2F:C5\" enableIPv6RA : true ipv6RAConfigs : \"address_mode=dhcpv6_stateful,max_interval=30,min_interval=5,send_periodic=true\" enableDHCP : \u662f\u5426\u5f00\u542f\u5b50\u7f51\u7684 DHCP \u529f\u80fd\u3002 dhcpV4Options , dhcpV6Options : \u8be5\u5b57\u6bb5\u76f4\u63a5\u66b4\u9732 ovn-nb \u5185 DHCP \u76f8\u5173\u9009\u9879\uff0c\u8bf7\u53c2\u8003 DHCP Options \u3002 \u9ed8\u8ba4\u503c\u5206\u522b\u4e3a \"lease_time=3600, router=$ipv4_gateway, server_id=169.254.0.254, server_mac=$random_mac\" \u548c server_id=$random_mac \u3002 enableIPv6RA : \u662f\u5426\u5f00\u542f DHCPv6 \u7684\u8def\u7531\u5e7f\u64ad\u529f\u80fd\u3002 ipv6RAConfigs \uff1a\u8be5\u5b57\u6bb5\u76f4\u63a5\u66b4\u9732 ovn-nb \u5185 Logical_Router_Port \u76f8\u5173\u9009\u9879\uff0c\u8bf7\u53c2\u8003 Logical Router Port \u9ed8\u8ba4\u503c\u4e3a address_mode=dhcpv6_stateful, max_interval=30, min_interval=5, send_periodic=true \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DHCP \u8bbe\u7f6e"},{"location":"advance/dpdk/","text":"DPDK \u652f\u6301 \u00b6 \u8be5\u6587\u6863\u4ecb\u7ecd Kube-OVN \u5982\u4f55\u548c OVS-DPDK \u7ed3\u5408\uff0c\u7ed9 KubeVirt \u7684\u865a\u673a\u63d0\u4f9b DPDK \u7c7b\u578b\u7684\u7f51\u7edc\u63a5\u53e3\u3002 \u4e0a\u6e38\u7684 KubeVirt \u76ee\u524d\u8fd8\u672a\u652f\u6301 OVS-DPDK\uff0c\u7528\u6237\u9700\u8981\u81ea\u5df1\u901a\u8fc7\u76f8\u5173 patch Vhostuser implementation \u6784\u5efa KubeVirt \u6216 KVM Device Plugin \u6765\u4f7f\u7528 OVS-DPDK\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u8282\u70b9\u9700\u63d0\u4f9b\u4e13\u95e8\u7ed9 DPDK \u9a71\u52a8\u8fd0\u884c\u7684\u7f51\u5361\u3002 \u8282\u70b9\u9700\u5f00\u542f Hugepages\u3002 \u7f51\u5361\u8bbe\u7f6e DPDK \u9a71\u52a8 \u00b6 \u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 driverctl \u4e3a\u4f8b\u8fdb\u884c\u64cd\u4f5c\uff0c\u5177\u4f53\u53c2\u6570\u548c\u5176\u4ed6\u9a71\u52a8\u4f7f\u7528\u8bf7\u53c2\u8003 DPDK \u6587\u6863 \u8fdb\u884c\u64cd\u4f5c\u3002 driverctl set-override 0000 :00:0b.0 uio_pci_generic \u8282\u70b9\u914d\u7f6e \u00b6 \u5bf9\u652f\u6301 OVS-DPDK \u7684\u8282\u70b9\u6253\u6807\u7b7e\uff0c\u4ee5\u4fbf Kube-OVN \u8fdb\u884c\u8bc6\u522b\u5904\u7406\uff1a kubectl label nodes <node> ovn.kubernetes.io/ovs_dp_type = \"userspace\" \u5728\u652f\u6301 OVS-DPDK \u8282\u70b9\u7684 /opt/ovs-config \u76ee\u5f55\u4e0b\u521b\u5efa\u914d\u7f6e\u6587\u4ef6 ovs-dpdk-config \uff1a ENCAP_IP = 192 .168.122.193/24 DPDK_DEV = 0000 :00:0b.0 ENCAP_IP : \u96a7\u9053\u7aef\u70b9\u5730\u5740\u3002 DPDK_DEV : \u8bbe\u5907\u7684 PCI ID\u3002 \u5b89\u88c5 Kube-OVN \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u542f\u7528 DPDK \u5b89\u88c5\u9009\u9879\u8fdb\u884c\u5b89\u88c5\uff1a bash install.sh --with-hybrid-dpdk \u4f7f\u7528\u65b9\u5f0f \u00b6 \u8fd9\u91cc\u6211\u4eec\u901a\u8fc7\u521b\u5efa\u4e00\u4e2a\u4f7f\u7528 vhostuser \u7c7b\u578b\u7f51\u5361\u7684\u865a\u673a\u6765\u9a8c\u8bc1 OVS-DPDK \u529f\u80fd\u3002 \u5b89\u88c5 KVM Device Plugin \u6765\u521b\u5efa\u865a\u673a\uff0c\u66f4\u591a\u4f7f\u7528\u65b9\u5f0f\u8bf7\u53c2\u8003 KVM Device Plugin \u3002 kubectl apply -f https://raw.githubusercontent.com/kubevirt/kubernetes-device-plugins/master/manifests/kvm-ds.yml \u521b\u5efa NetworkAttachmentDefinition\uff1a apiVersion : k8s.cni.cncf.io/v1 kind : NetworkAttachmentDefinition metadata : name : ovn-dpdk namespace : default spec : config : >- { \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-dpdk.default.ovn\", \"vhost_user_socket_volume_name\": \"vhostuser-sockets\", \"vhost_user_socket_name\": \"sock\" } \u4f7f\u7528\u4e0b\u9762\u7684 Dockerfile \u521b\u5efa VM \u955c\u50cf\uff1a FROM quay.io/kubevirt/virt-launcher:v0.46.1 # wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 COPY CentOS-7-x86_64-GenericCloud.qcow2 /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2 \u521b\u5efa\u865a\u62df\u673a\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vm-config data : start.sh : | chmod u+w /etc/libvirt/qemu.conf echo \"hugetlbfs_mount = \\\"/dev/hugepages\\\"\" >> /etc/libvirt/qemu.conf virtlogd & libvirtd & mkdir /var/lock sleep 5 virsh define /root/vm/vm.xml virsh start vm tail -f /dev/null vm.xml : | <domain type='kvm'> <name>vm</name> <uuid>4a9b3f53-fa2a-47f3-a757-dd87720d9d1d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <hugepages> <page size='2' unit='M' nodeset='0'/> </hugepages> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <emulatorpin cpuset='1,3'/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='2' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='2097152' unit='KiB' memAccess='shared'/> </numa> </cpu> <on_reboot>restart</on_reboot> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <interface type='vhostuser'> <mac address='00:00:00:0A:30:89'/> <source type='unix' path='/var/run/vm/sock' mode='server'/> <model type='virtio'/> <driver queues='2'> <host mrg_rxbuf='off'/> </driver> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-vm/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> </devices> </domain> --- apiVersion : apps/v1 kind : Deployment metadata : name : vm-deployment labels : app : vm spec : replicas : 1 selector : matchLabels : app : vm template : metadata : labels : app : vm annotations : k8s.v1.cni.cncf.io/networks : default/ovn-dpdk ovn-dpdk.default.ovn.kubernetes.io/ip_address : 10.16.0.96 ovn-dpdk.default.ovn.kubernetes.io/mac_address : 00:00:00:0A:30:89 spec : nodeSelector : ovn.kubernetes.io/ovs_dp_type : userspace securityContext : runAsUser : 0 volumes : - name : vhostuser-sockets emptyDir : {} - name : xml configMap : name : vm-config - name : hugepage emptyDir : medium : HugePages-2Mi - name : libvirt-runtime emptyDir : {} containers : - name : vm image : vm-vhostuser:latest command : [ \"bash\" , \"/root/vm/start.sh\" ] securityContext : capabilities : add : - NET_BIND_SERVICE - SYS_NICE - NET_RAW - NET_ADMIN privileged : false runAsUser : 0 resources : limits : cpu : '2' devices.kubevirt.io/kvm : '1' memory : '8784969729' hugepages-2Mi : 2Gi requests : cpu : 666m devices.kubevirt.io/kvm : '1' ephemeral-storage : 50M memory : '4490002433' volumeMounts : - name : vhostuser-sockets mountPath : /var/run/vm - name : xml mountPath : /root/vm/ - mountPath : /dev/hugepages name : hugepage - name : libvirt-runtime mountPath : /var/run/libvirt \u7b49\u5f85\u865a\u62df\u673a\u521b\u5efa\u6210\u529f\u540e\u8fdb\u5165 Pod \u8fdb\u884c\u865a\u673a\u914d\u7f6e\uff1a # virsh set-user-password vm root 12345 Password set successfully for root in vm # virsh console vm Connected to domain 'vm' Escape character is ^ ] ( Ctrl + ]) CentOS Linux 7 ( Core ) Kernel 3 .10.0-1127.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Fri Feb 25 09 :52:54 on ttyS0 \u63a5\u4e0b\u6765\u53ef\u4ee5\u767b\u5f55\u865a\u673a\u8fdb\u884c\u7f51\u7edc\u914d\u7f6e\u5e76\u6d4b\u8bd5\uff1a ip link set eth0 mtu 1400 ip addr add 10 .16.0.96/16 dev eth0 ip ro add default via 10 .16.0.1 ping 114 .114.114.114 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DPDK \u652f\u6301"},{"location":"advance/dpdk/#dpdk","text":"\u8be5\u6587\u6863\u4ecb\u7ecd Kube-OVN \u5982\u4f55\u548c OVS-DPDK \u7ed3\u5408\uff0c\u7ed9 KubeVirt \u7684\u865a\u673a\u63d0\u4f9b DPDK \u7c7b\u578b\u7684\u7f51\u7edc\u63a5\u53e3\u3002 \u4e0a\u6e38\u7684 KubeVirt \u76ee\u524d\u8fd8\u672a\u652f\u6301 OVS-DPDK\uff0c\u7528\u6237\u9700\u8981\u81ea\u5df1\u901a\u8fc7\u76f8\u5173 patch Vhostuser implementation \u6784\u5efa KubeVirt \u6216 KVM Device Plugin \u6765\u4f7f\u7528 OVS-DPDK\u3002","title":"DPDK \u652f\u6301"},{"location":"advance/dpdk/#_1","text":"\u8282\u70b9\u9700\u63d0\u4f9b\u4e13\u95e8\u7ed9 DPDK \u9a71\u52a8\u8fd0\u884c\u7684\u7f51\u5361\u3002 \u8282\u70b9\u9700\u5f00\u542f Hugepages\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/dpdk/#dpdk_1","text":"\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 driverctl \u4e3a\u4f8b\u8fdb\u884c\u64cd\u4f5c\uff0c\u5177\u4f53\u53c2\u6570\u548c\u5176\u4ed6\u9a71\u52a8\u4f7f\u7528\u8bf7\u53c2\u8003 DPDK \u6587\u6863 \u8fdb\u884c\u64cd\u4f5c\u3002 driverctl set-override 0000 :00:0b.0 uio_pci_generic","title":"\u7f51\u5361\u8bbe\u7f6e DPDK \u9a71\u52a8"},{"location":"advance/dpdk/#_2","text":"\u5bf9\u652f\u6301 OVS-DPDK \u7684\u8282\u70b9\u6253\u6807\u7b7e\uff0c\u4ee5\u4fbf Kube-OVN \u8fdb\u884c\u8bc6\u522b\u5904\u7406\uff1a kubectl label nodes <node> ovn.kubernetes.io/ovs_dp_type = \"userspace\" \u5728\u652f\u6301 OVS-DPDK \u8282\u70b9\u7684 /opt/ovs-config \u76ee\u5f55\u4e0b\u521b\u5efa\u914d\u7f6e\u6587\u4ef6 ovs-dpdk-config \uff1a ENCAP_IP = 192 .168.122.193/24 DPDK_DEV = 0000 :00:0b.0 ENCAP_IP : \u96a7\u9053\u7aef\u70b9\u5730\u5740\u3002 DPDK_DEV : \u8bbe\u5907\u7684 PCI ID\u3002","title":"\u8282\u70b9\u914d\u7f6e"},{"location":"advance/dpdk/#kube-ovn","text":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u542f\u7528 DPDK \u5b89\u88c5\u9009\u9879\u8fdb\u884c\u5b89\u88c5\uff1a bash install.sh --with-hybrid-dpdk","title":"\u5b89\u88c5 Kube-OVN"},{"location":"advance/dpdk/#_3","text":"\u8fd9\u91cc\u6211\u4eec\u901a\u8fc7\u521b\u5efa\u4e00\u4e2a\u4f7f\u7528 vhostuser \u7c7b\u578b\u7f51\u5361\u7684\u865a\u673a\u6765\u9a8c\u8bc1 OVS-DPDK \u529f\u80fd\u3002 \u5b89\u88c5 KVM Device Plugin \u6765\u521b\u5efa\u865a\u673a\uff0c\u66f4\u591a\u4f7f\u7528\u65b9\u5f0f\u8bf7\u53c2\u8003 KVM Device Plugin \u3002 kubectl apply -f https://raw.githubusercontent.com/kubevirt/kubernetes-device-plugins/master/manifests/kvm-ds.yml \u521b\u5efa NetworkAttachmentDefinition\uff1a apiVersion : k8s.cni.cncf.io/v1 kind : NetworkAttachmentDefinition metadata : name : ovn-dpdk namespace : default spec : config : >- { \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-dpdk.default.ovn\", \"vhost_user_socket_volume_name\": \"vhostuser-sockets\", \"vhost_user_socket_name\": \"sock\" } \u4f7f\u7528\u4e0b\u9762\u7684 Dockerfile \u521b\u5efa VM \u955c\u50cf\uff1a FROM quay.io/kubevirt/virt-launcher:v0.46.1 # wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 COPY CentOS-7-x86_64-GenericCloud.qcow2 /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2 \u521b\u5efa\u865a\u62df\u673a\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vm-config data : start.sh : | chmod u+w /etc/libvirt/qemu.conf echo \"hugetlbfs_mount = \\\"/dev/hugepages\\\"\" >> /etc/libvirt/qemu.conf virtlogd & libvirtd & mkdir /var/lock sleep 5 virsh define /root/vm/vm.xml virsh start vm tail -f /dev/null vm.xml : | <domain type='kvm'> <name>vm</name> <uuid>4a9b3f53-fa2a-47f3-a757-dd87720d9d1d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <hugepages> <page size='2' unit='M' nodeset='0'/> </hugepages> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <emulatorpin cpuset='1,3'/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='2' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='2097152' unit='KiB' memAccess='shared'/> </numa> </cpu> <on_reboot>restart</on_reboot> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <interface type='vhostuser'> <mac address='00:00:00:0A:30:89'/> <source type='unix' path='/var/run/vm/sock' mode='server'/> <model type='virtio'/> <driver queues='2'> <host mrg_rxbuf='off'/> </driver> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-vm/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> </devices> </domain> --- apiVersion : apps/v1 kind : Deployment metadata : name : vm-deployment labels : app : vm spec : replicas : 1 selector : matchLabels : app : vm template : metadata : labels : app : vm annotations : k8s.v1.cni.cncf.io/networks : default/ovn-dpdk ovn-dpdk.default.ovn.kubernetes.io/ip_address : 10.16.0.96 ovn-dpdk.default.ovn.kubernetes.io/mac_address : 00:00:00:0A:30:89 spec : nodeSelector : ovn.kubernetes.io/ovs_dp_type : userspace securityContext : runAsUser : 0 volumes : - name : vhostuser-sockets emptyDir : {} - name : xml configMap : name : vm-config - name : hugepage emptyDir : medium : HugePages-2Mi - name : libvirt-runtime emptyDir : {} containers : - name : vm image : vm-vhostuser:latest command : [ \"bash\" , \"/root/vm/start.sh\" ] securityContext : capabilities : add : - NET_BIND_SERVICE - SYS_NICE - NET_RAW - NET_ADMIN privileged : false runAsUser : 0 resources : limits : cpu : '2' devices.kubevirt.io/kvm : '1' memory : '8784969729' hugepages-2Mi : 2Gi requests : cpu : 666m devices.kubevirt.io/kvm : '1' ephemeral-storage : 50M memory : '4490002433' volumeMounts : - name : vhostuser-sockets mountPath : /var/run/vm - name : xml mountPath : /root/vm/ - mountPath : /dev/hugepages name : hugepage - name : libvirt-runtime mountPath : /var/run/libvirt \u7b49\u5f85\u865a\u62df\u673a\u521b\u5efa\u6210\u529f\u540e\u8fdb\u5165 Pod \u8fdb\u884c\u865a\u673a\u914d\u7f6e\uff1a # virsh set-user-password vm root 12345 Password set successfully for root in vm # virsh console vm Connected to domain 'vm' Escape character is ^ ] ( Ctrl + ]) CentOS Linux 7 ( Core ) Kernel 3 .10.0-1127.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Fri Feb 25 09 :52:54 on ttyS0 \u63a5\u4e0b\u6765\u53ef\u4ee5\u767b\u5f55\u865a\u673a\u8fdb\u884c\u7f51\u7edc\u914d\u7f6e\u5e76\u6d4b\u8bd5\uff1a ip link set eth0 mtu 1400 ip addr add 10 .16.0.96/16 dev eth0 ip ro add default via 10 .16.0.1 ping 114 .114.114.114 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528\u65b9\u5f0f"},{"location":"advance/external-gateway/","text":"\u5916\u90e8\u7f51\u5173\u8bbe\u7f6e \u00b6 \u5728\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u5bf9\u6240\u6709\u5bb9\u5668\u8bbf\u95ee\u5916\u90e8\u7684\u6d41\u91cf\u9700\u8981\u901a\u8fc7\u4e00\u4e2a\u5916\u90e8\u7684\u7f51\u5173\u8fdb\u884c\u7edf\u4e00\u7684\u7ba1\u7406\u548c\u5ba1\u8ba1\u3002 Kube-OVN \u53ef\u4ee5\u901a\u8fc7\u5728\u5b50\u7f51\u4e2d\u8fdb\u884c\u76f8\u5e94\u7684\u8def\u7531\u914d\u7f6e\uff0c\u5c06\u51fa\u7f51\u6d41\u91cf\u8f6c\u53d1\u81f3\u5bf9\u5e94\u7684\u5916\u90e8\u7f51\u5173\u3002 \u4f7f\u7528\u65b9\u5f0f \u00b6 kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : external spec : cidrBlock : 172.31.0.0/16 gatewayType : centralized natOutgoing : false externalEgressGateway : 192.168.0.1 policyRoutingTableID : 1000 policyRoutingPriority : 1500 natOutgoing : \u9700\u8981\u8bbe\u7f6e\u4e3a false \u3002 externalEgressGateway \uff1a\u8bbe\u7f6e\u4e3a\u5916\u90e8\u7f51\u5173\u7684\u5730\u5740\uff0c\u9700\u8981\u548c\u7f51\u5173\u8282\u70b9\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u53ef\u8fbe\u57df\u3002 policyRoutingTableID \uff1a\u4f7f\u7528\u7684\u672c\u5730\u7b56\u7565\u8def\u7531\u8868\u7684 TableID \u6bcf\u4e2a\u5b50\u7f51\u5747\u9700\u4e0d\u540c\u4ee5\u907f\u514d\u51b2\u7a81\u3002 policyRoutingPriority \uff1a\u8def\u7531\u4f18\u5148\u7ea7\uff0c\u4e3a\u907f\u514d\u540e\u7eed\u7528\u6237\u5b9a\u5236\u5316\u7684\u5176\u4ed6\u8def\u7531\u64cd\u4f5c\u51b2\u7a81\uff0c\u8fd9\u91cc\u53ef\u4ee5\u6307\u5b9a\u8def\u7531\u4f18\u5148\u7ea7\uff0c\u82e5\u65e0\u7279\u6b8a\u9700\u6c42\u586b\u5165\u4efb\u610f\u503c\u5373\u53ef\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5916\u90e8\u7f51\u5173\u8bbe\u7f6e"},{"location":"advance/external-gateway/#_1","text":"\u5728\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u5bf9\u6240\u6709\u5bb9\u5668\u8bbf\u95ee\u5916\u90e8\u7684\u6d41\u91cf\u9700\u8981\u901a\u8fc7\u4e00\u4e2a\u5916\u90e8\u7684\u7f51\u5173\u8fdb\u884c\u7edf\u4e00\u7684\u7ba1\u7406\u548c\u5ba1\u8ba1\u3002 Kube-OVN \u53ef\u4ee5\u901a\u8fc7\u5728\u5b50\u7f51\u4e2d\u8fdb\u884c\u76f8\u5e94\u7684\u8def\u7531\u914d\u7f6e\uff0c\u5c06\u51fa\u7f51\u6d41\u91cf\u8f6c\u53d1\u81f3\u5bf9\u5e94\u7684\u5916\u90e8\u7f51\u5173\u3002","title":"\u5916\u90e8\u7f51\u5173\u8bbe\u7f6e"},{"location":"advance/external-gateway/#_2","text":"kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : external spec : cidrBlock : 172.31.0.0/16 gatewayType : centralized natOutgoing : false externalEgressGateway : 192.168.0.1 policyRoutingTableID : 1000 policyRoutingPriority : 1500 natOutgoing : \u9700\u8981\u8bbe\u7f6e\u4e3a false \u3002 externalEgressGateway \uff1a\u8bbe\u7f6e\u4e3a\u5916\u90e8\u7f51\u5173\u7684\u5730\u5740\uff0c\u9700\u8981\u548c\u7f51\u5173\u8282\u70b9\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u53ef\u8fbe\u57df\u3002 policyRoutingTableID \uff1a\u4f7f\u7528\u7684\u672c\u5730\u7b56\u7565\u8def\u7531\u8868\u7684 TableID \u6bcf\u4e2a\u5b50\u7f51\u5747\u9700\u4e0d\u540c\u4ee5\u907f\u514d\u51b2\u7a81\u3002 policyRoutingPriority \uff1a\u8def\u7531\u4f18\u5148\u7ea7\uff0c\u4e3a\u907f\u514d\u540e\u7eed\u7528\u6237\u5b9a\u5236\u5316\u7684\u5176\u4ed6\u8def\u7531\u64cd\u4f5c\u51b2\u7a81\uff0c\u8fd9\u91cc\u53ef\u4ee5\u6307\u5b9a\u8def\u7531\u4f18\u5148\u7ea7\uff0c\u82e5\u65e0\u7279\u6b8a\u9700\u6c42\u586b\u5165\u4efb\u610f\u503c\u5373\u53ef\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528\u65b9\u5f0f"},{"location":"advance/fastpath/","text":"\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757 \u00b6 \u7ecf\u8fc7\u6570\u636e\u5e73\u9762\u7684\u6027\u80fd Profile\uff0c Netfilter \u5728\u5bb9\u5668\u5185\u548c\u5bbf\u4e3b\u673a\u4e0a\u7684\u76f8\u5173\u5904\u7406\u6d88\u8017\u4e86 20% \u5de6\u53f3\u7684 CPU \u8d44\u6e90\uff0cFastPath \u6a21\u5757\u53ef\u4ee5\u7ed5\u8fc7 Netfilter \u4ece\u800c \u964d\u4f4e CPU \u7684\u6d88\u8017\u548c\u5ef6\u8fdf\uff0c\u5e76\u63d0\u5347\u541e\u5410\u91cf\u3002\u672c\u6587\u6863\u5c06\u4ecb\u7ecd\u5982\u4f55\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757\u3002 \u4e0b\u8f7d\u76f8\u5173\u5185\u6838\u6a21\u5757\u4ee3\u7801 \u00b6 git clone --depth = 1 https://github.com/kubeovn/kube-ovn.git \u5b89\u88c5\u4f9d\u8d56 \u00b6 \u8fd9\u91cc\u4ee5 CentOS \u4e3a\u4f8b\u4e0b\u8f7d\u76f8\u5173\u4f9d\u8d56\uff1a yum install -y kernel-devel- $( uname -r ) gcc elfutils-libelf-devel \u7f16\u8bd1\u76f8\u5173\u6a21\u5757 \u00b6 \u9488\u5bf9 3.x \u7684\u5185\u6838\uff1a cd kube-ovn/fastpath make all \u9488\u5bf9 4.x \u7684\u5185\u6838\uff1a cd kube-ovn/fastpath/4.18 cp ../Makefile . make all \u5b89\u88c5\u5185\u6838\u6a21\u5757 \u00b6 \u5c06 kube_ovn_fastpath.ko \u590d\u5236\u5230\u6bcf\u4e2a\u9700\u8981\u6027\u80fd\u4f18\u5316\u7684\u8282\u70b9 /tmp \u76ee\u5f55\u4e0b\uff0c kube-ovn-cni \u4f1a\u81ea\u52a8\u52a0\u8f7d\u8be5\u6a21\u5757\u3002 \u4f7f\u7528 dmesg \u786e\u8ba4\u5b89\u88c5\u6210\u529f\uff1a # dmesg [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in \u5982\u9700\u5378\u8f7d\u6a21\u5757\uff0c\u53ef\u5c06\u8be5\u6a21\u5757\u4ece /tmp \u76ee\u5f55\u4e0b\u79fb\u9664\uff0c kube-ovn-cni \u4f1a\u81ea\u52a8\u5378\u8f7d\u8be5\u6a21\u5757\u3002 \u8be5\u6a21\u5757\u5728\u673a\u5668\u91cd\u542f\u540e\u4e0d\u4f1a\u81ea\u52a8\u52a0\u8f7d\uff0c\u5982\u9700\u81ea\u52a8\u52a0\u8f7d\u8bf7\u6839\u636e\u7cfb\u7edf\u5f04\u914d\u7f6e\u7f16\u5199\u76f8\u5e94\u81ea\u542f\u52a8\u811a\u672c\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757"},{"location":"advance/fastpath/#fastpath","text":"\u7ecf\u8fc7\u6570\u636e\u5e73\u9762\u7684\u6027\u80fd Profile\uff0c Netfilter \u5728\u5bb9\u5668\u5185\u548c\u5bbf\u4e3b\u673a\u4e0a\u7684\u76f8\u5173\u5904\u7406\u6d88\u8017\u4e86 20% \u5de6\u53f3\u7684 CPU \u8d44\u6e90\uff0cFastPath \u6a21\u5757\u53ef\u4ee5\u7ed5\u8fc7 Netfilter \u4ece\u800c \u964d\u4f4e CPU \u7684\u6d88\u8017\u548c\u5ef6\u8fdf\uff0c\u5e76\u63d0\u5347\u541e\u5410\u91cf\u3002\u672c\u6587\u6863\u5c06\u4ecb\u7ecd\u5982\u4f55\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757\u3002","title":"\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757"},{"location":"advance/fastpath/#_1","text":"git clone --depth = 1 https://github.com/kubeovn/kube-ovn.git","title":"\u4e0b\u8f7d\u76f8\u5173\u5185\u6838\u6a21\u5757\u4ee3\u7801"},{"location":"advance/fastpath/#_2","text":"\u8fd9\u91cc\u4ee5 CentOS \u4e3a\u4f8b\u4e0b\u8f7d\u76f8\u5173\u4f9d\u8d56\uff1a yum install -y kernel-devel- $( uname -r ) gcc elfutils-libelf-devel","title":"\u5b89\u88c5\u4f9d\u8d56"},{"location":"advance/fastpath/#_3","text":"\u9488\u5bf9 3.x \u7684\u5185\u6838\uff1a cd kube-ovn/fastpath make all \u9488\u5bf9 4.x \u7684\u5185\u6838\uff1a cd kube-ovn/fastpath/4.18 cp ../Makefile . make all","title":"\u7f16\u8bd1\u76f8\u5173\u6a21\u5757"},{"location":"advance/fastpath/#_4","text":"\u5c06 kube_ovn_fastpath.ko \u590d\u5236\u5230\u6bcf\u4e2a\u9700\u8981\u6027\u80fd\u4f18\u5316\u7684\u8282\u70b9 /tmp \u76ee\u5f55\u4e0b\uff0c kube-ovn-cni \u4f1a\u81ea\u52a8\u52a0\u8f7d\u8be5\u6a21\u5757\u3002 \u4f7f\u7528 dmesg \u786e\u8ba4\u5b89\u88c5\u6210\u529f\uff1a # dmesg [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in \u5982\u9700\u5378\u8f7d\u6a21\u5757\uff0c\u53ef\u5c06\u8be5\u6a21\u5757\u4ece /tmp \u76ee\u5f55\u4e0b\u79fb\u9664\uff0c kube-ovn-cni \u4f1a\u81ea\u52a8\u5378\u8f7d\u8be5\u6a21\u5757\u3002 \u8be5\u6a21\u5757\u5728\u673a\u5668\u91cd\u542f\u540e\u4e0d\u4f1a\u81ea\u52a8\u52a0\u8f7d\uff0c\u5982\u9700\u81ea\u52a8\u52a0\u8f7d\u8bf7\u6839\u636e\u7cfb\u7edf\u5f04\u914d\u7f6e\u7f16\u5199\u76f8\u5e94\u81ea\u542f\u52a8\u811a\u672c\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b89\u88c5\u5185\u6838\u6a21\u5757"},{"location":"advance/multi-nic/","text":"\u591a\u7f51\u5361\u7ba1\u7406 \u00b6 Kube-OVN \u53ef\u4ee5\u4e3a\u5176\u4ed6 CNI \u7f51\u7edc\u63d2\u4ef6\uff0c\u4f8b\u5982 macvlan\u3001vlan\u3001host-device \u7b49\u63d2\u4ef6\u63d0\u4f9b\u96c6\u7fa4\u7ea7\u522b\u7684 IPAM \u80fd\u529b\uff0c \u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4e5f\u53ef\u4ee5\u4f7f\u7528\u5230 Kube-OVN \u4e2d\u5b50\u7f51\u4ee5\u53ca\u56fa\u5b9a IP \u529f\u80fd\u3002 \u540c\u65f6 Kube-OVN \u4e5f\u652f\u6301\u591a\u5757\u7f51\u5361\u5747\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\u60c5\u51b5\u4e0b\u7684\u5730\u5740\u7ba1\u7406\u3002 \u5de5\u4f5c\u539f\u7406 \u00b6 \u901a\u8fc7\u4f7f\u7528 Multus CNI , \u6211\u4eec\u53ef\u4ee5\u7ed9\u4e00\u4e2a Pod \u6dfb\u52a0\u591a\u5757\u4e0d\u540c\u7f51\u7edc\u7684\u7f51\u5361\u3002 \u7136\u800c\u6211\u4eec\u4ecd\u7136\u7f3a\u4e4f\u5bf9\u96c6\u7fa4\u8303\u56f4\u5185\u4e0d\u540c\u7f51\u7edc\u7684 IP \u5730\u5740\u8fdb\u884c\u7ba1\u7406\u7684\u80fd\u529b\u3002\u5728 Kube-OVN \u4e2d\uff0c\u6211\u4eec\u5df2\u7ecf\u80fd\u591f\u901a\u8fc7 Subnet \u548c IP \u7684 CRD \u6765\u8fdb\u884c IP \u7684\u9ad8\u7ea7\u7ba1\u7406\uff0c \u4f8b\u5982\u5b50\u7f51\u7ba1\u7406\uff0cIP \u9884\u7559\uff0c\u968f\u673a\u5206\u914d\uff0c\u56fa\u5b9a\u5206\u914d\u7b49\u3002\u73b0\u5728\u6211\u4eec\u5bf9\u5b50\u7f51\u8fdb\u884c\u6269\u5c55\uff0c\u6765\u63a5\u5165\u5176\u4ed6\u4e0d\u540c\u7684\u7f51\u7edc\u63d2\u4ef6\uff0c\u4f7f\u5f97\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4e5f\u53ef\u4ee5\u4f7f\u7528 Kube-OVN \u7684 IPAM \u529f\u80fd\u3002 \u5de5\u4f5c\u6d41\u7a0b \u00b6 \u4e0a\u56fe\u5c55\u793a\u4e86\u5982\u4f55\u901a\u8fc7 Kube-OVN \u6765\u7ba1\u7406\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u7684 IP \u5730\u5740\u3002\u5176\u4e2d\u5bb9\u5668\u7684 eth0 \u7f51\u5361\u63a5\u5165 OVN \u7f51\u7edc\uff0cnet1 \u7f51\u5361\u63a5\u5165\u5176\u4ed6 CNI \u7f51\u7edc\u3002 net1 \u7f51\u7edc\u7684\u7f51\u7edc\u5b9a\u4e49\u6765\u81ea\u4e8e multus-cni \u4e2d\u7684 NetworkAttachmentDefinition \u8d44\u6e90\u5b9a\u4e49\u3002 \u5f53 Pod \u521b\u5efa\u65f6\uff0c kube-ovn-controller \u4f1a\u76d1\u542c\u5230 Pod \u6dfb\u52a0\u4e8b\u4ef6\uff0c\u5e76\u6839\u636e Pod \u4e2d\u7684 annotation \u53bb\u5bfb\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u5e76\u4ece\u4e2d\u8fdb\u884c IP \u7684\u5206\u914d\u548c\u7ba1\u7406\uff0c \u5e76\u5c06 Pod \u6240\u5206\u914d\u5230\u7684\u5730\u5740\u4fe1\u606f\u5199\u56de\u5230 Pod annotation \u4e2d\u3002 \u5728\u5bb9\u5668\u6240\u5728\u673a\u5668\u7684 CNI \u53ef\u4ee5\u901a\u8fc7\u5728\u914d\u7f6e\u4e2d\u914d\u7f6e kube-ovn-cni \u4f5c\u4e3a ipam \u63d2\u4ef6, kube-ovn-cni \u5c06\u4f1a\u8bfb\u53d6 Pod annotation \u5e76\u5c06\u5730\u5740\u4fe1\u606f\u901a\u8fc7 CNI \u534f\u8bae\u7684\u6807\u51c6\u683c\u5f0f\u8fd4\u56de\u7ed9\u76f8\u5e94\u7684 CNI \u63d2\u4ef6\u3002 \u4f7f\u7528\u65b9\u6cd5 \u00b6 \u5b89\u88c5 Kube-OVN \u548c Multus \u00b6 \u8bf7\u53c2\u8003 Kube-OVN \u4e00\u952e\u5b89\u88c5 \u548c Multus how to use \u6765\u5b89\u88c5 Kube-OVN \u548c Multus-CNI\u3002 \u4e3a\u5176\u4ed6 CNI \u63d0\u4f9b IPAM \u00b6 \u6b64\u65f6\u4e3b\u7f51\u5361\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\uff0c\u9644\u5c5e\u7f51\u5361\u4e3a\u5176\u4ed6\u7c7b\u578b CNI\u3002 \u521b\u5efa NetworkAttachmentDefinition \u00b6 \u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 macvlan \u4f5c\u4e3a\u5bb9\u5668\u7f51\u7edc\u7684\u7b2c\u4e8c\u4e2a\u7f51\u7edc\uff0c\u5e76\u5c06\u5176 ipam \u8bbe\u7f6e\u4e3a kube-ovn \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : macvlan namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"macvlan.default\" } }' spec.config.ipam.type : \u9700\u8981\u4e3a kube-ovn \u6765\u8c03\u7528 kube-ovn \u7684\u63d2\u4ef6\u6765\u83b7\u53d6\u5730\u5740\u4fe1\u606f\u3002 server_socket : Kube-OVN \u901a\u4fe1\u4f7f\u7528\u7684 socket \u6587\u4ef6\u3002 \u9ed8\u8ba4\u4f4d\u7f6e\u4e3a /run/openvswitch/kube-ovn-daemon.sock \u3002 provider : \u5f53\u524d NetworkAttachmentDefinition \u7684 <name>.<namespace> , Kube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90\u3002 \u9644\u5c5e\u7f51\u5361\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361 \u00b6 \u6b64\u65f6\u591a\u5757\u7f51\u5361\u5747\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\u3002 \u521b\u5efa NetworkAttachmentDefinition \u00b6 \u5c06 provider \u7684\u540e\u7f00\u8bbe\u7f6e\u4e3a ovn \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" }' spec.config.type : \u8bbe\u7f6e\u4e3a kube-ovn \u6765\u89e6\u53d1 CNI \u63d2\u4ef6\u4f7f\u7528 Kube-OVN \u5b50\u7f51\u3002 server_socket : Kube-OVN \u901a\u4fe1\u4f7f\u7528\u7684 socket \u6587\u4ef6\u3002 \u9ed8\u8ba4\u4f4d\u7f6e\u4e3a /run/openvswitch/kube-ovn-daemon.sock \u3002 provider : \u5f53\u524d NetworkAttachmentDefinition \u7684 <name>.<namespace>.ovn , Kube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90\uff0c\u6ce8\u610f\u540e\u7f00\u9700\u8981\u8bbe\u7f6e\u4e3a ovn\u3002 \u521b\u5efa\u4e00\u4e2a Kube-OVN Subnet \u00b6 \u521b\u5efa\u4e00\u4e2a Kube-OVN Subnet,\u8bbe\u7f6e\u5bf9\u5e94\u7684 cidrBlock \u548c exclude_ips , provider \u5e94\u8be5\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <name>.<namespace> , \u4f8b\u5982\u7528 macvlan \u63d0\u4f9b\u9644\u52a0\u7f51\u5361\uff0c\u521b\u5efa Subnet \u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : macvlan spec : protocol : IPv4 provider : macvlan.default cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 gateway , private , nat \u53ea\u5bf9 provider \u7c7b\u578b\u4e3a ovn \u7684\u7f51\u7edc\u751f\u6548\uff0c\u4e0d\u9002\u7528\u4e8e attachment network\u3002 \u5982\u679c\u4ee5 Kube-OVN \u4f5c\u4e3a\u9644\u52a0\u7f51\u5361\uff0c\u5219 provider \u5e94\u8be5\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <name>.<namespace>.ovn \uff0c\u5e76\u8981\u4ee5 ovn \u4f5c\u4e3a\u540e\u7f00\u7ed3\u675f\u3002 \u7528 Kube-OVN \u63d0\u4f9b\u9644\u52a0\u7f51\u5361\uff0c\u521b\u5efa Subnet \u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attachnet spec : protocol : IPv4 provider : attachnet.default.ovn cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 \u521b\u5efa\u4e00\u4e2a\u591a\u7f51\u7edc\u7684 Pod \u00b6 \u5bf9\u4e8e\u5730\u5740\u968f\u673a\u5206\u914d\u7684 Pod\uff0c\u53ea\u9700\u8981\u6dfb\u52a0\u5982\u4e0b annotation k8s.v1.cni.cncf.io/networks ,\u53d6\u503c\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <namespace>/<name> \uff1a apiVersion : v1 kind : Pod metadata : name : samplepod namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan spec : containers : - name : samplepod command : [ \"/bin/ash\" , \"-c\" , \"trap : TERM INT; sleep infinity & wait\" ] image : docker.io/library/alpine:edge \u521b\u5efa\u56fa\u5b9a IP \u7684 Pod \u00b6 \u5bf9\u4e8e\u56fa\u5b9a IP \u7684 Pod\uff0c\u6dfb\u52a0 <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_address annotation\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 macvlan.default.kubernetes.io/ip_address : 172.17.0.100 macvlan.default.kubernetes.io/mac_address : 00:00:00:53:6B:BB spec : containers : - name : static-ip image : docker.io/library/nginx:alpine \u521b\u5efa\u4f7f\u7528\u56fa\u5b9a IP \u7684\u5de5\u4f5c\u8d1f\u8f7d \u00b6 \u5bf9\u4e8e\u4f7f\u7528 ippool \u7684\u5de5\u4f5c\u8d1f\u8f7d, \u6dfb\u52a0 <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_pool annotations: apiVersion : apps/v1 kind : Deployment metadata : namespace : default name : static-workload labels : app : static-workload spec : replicas : 2 selector : matchLabels : app : static-workload template : metadata : labels : app : static-workload annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 macvlan.default.kubernetes.io/ip_pool : 172.17.0.200,172.17.0.201,172.17.0.202 spec : containers : - name : static-workload image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u591a\u7f51\u5361\u7ba1\u7406"},{"location":"advance/multi-nic/#_1","text":"Kube-OVN \u53ef\u4ee5\u4e3a\u5176\u4ed6 CNI \u7f51\u7edc\u63d2\u4ef6\uff0c\u4f8b\u5982 macvlan\u3001vlan\u3001host-device \u7b49\u63d2\u4ef6\u63d0\u4f9b\u96c6\u7fa4\u7ea7\u522b\u7684 IPAM \u80fd\u529b\uff0c \u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4e5f\u53ef\u4ee5\u4f7f\u7528\u5230 Kube-OVN \u4e2d\u5b50\u7f51\u4ee5\u53ca\u56fa\u5b9a IP \u529f\u80fd\u3002 \u540c\u65f6 Kube-OVN \u4e5f\u652f\u6301\u591a\u5757\u7f51\u5361\u5747\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\u60c5\u51b5\u4e0b\u7684\u5730\u5740\u7ba1\u7406\u3002","title":"\u591a\u7f51\u5361\u7ba1\u7406"},{"location":"advance/multi-nic/#_2","text":"\u901a\u8fc7\u4f7f\u7528 Multus CNI , \u6211\u4eec\u53ef\u4ee5\u7ed9\u4e00\u4e2a Pod \u6dfb\u52a0\u591a\u5757\u4e0d\u540c\u7f51\u7edc\u7684\u7f51\u5361\u3002 \u7136\u800c\u6211\u4eec\u4ecd\u7136\u7f3a\u4e4f\u5bf9\u96c6\u7fa4\u8303\u56f4\u5185\u4e0d\u540c\u7f51\u7edc\u7684 IP \u5730\u5740\u8fdb\u884c\u7ba1\u7406\u7684\u80fd\u529b\u3002\u5728 Kube-OVN \u4e2d\uff0c\u6211\u4eec\u5df2\u7ecf\u80fd\u591f\u901a\u8fc7 Subnet \u548c IP \u7684 CRD \u6765\u8fdb\u884c IP \u7684\u9ad8\u7ea7\u7ba1\u7406\uff0c \u4f8b\u5982\u5b50\u7f51\u7ba1\u7406\uff0cIP \u9884\u7559\uff0c\u968f\u673a\u5206\u914d\uff0c\u56fa\u5b9a\u5206\u914d\u7b49\u3002\u73b0\u5728\u6211\u4eec\u5bf9\u5b50\u7f51\u8fdb\u884c\u6269\u5c55\uff0c\u6765\u63a5\u5165\u5176\u4ed6\u4e0d\u540c\u7684\u7f51\u7edc\u63d2\u4ef6\uff0c\u4f7f\u5f97\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4e5f\u53ef\u4ee5\u4f7f\u7528 Kube-OVN \u7684 IPAM \u529f\u80fd\u3002","title":"\u5de5\u4f5c\u539f\u7406"},{"location":"advance/multi-nic/#_3","text":"\u4e0a\u56fe\u5c55\u793a\u4e86\u5982\u4f55\u901a\u8fc7 Kube-OVN \u6765\u7ba1\u7406\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u7684 IP \u5730\u5740\u3002\u5176\u4e2d\u5bb9\u5668\u7684 eth0 \u7f51\u5361\u63a5\u5165 OVN \u7f51\u7edc\uff0cnet1 \u7f51\u5361\u63a5\u5165\u5176\u4ed6 CNI \u7f51\u7edc\u3002 net1 \u7f51\u7edc\u7684\u7f51\u7edc\u5b9a\u4e49\u6765\u81ea\u4e8e multus-cni \u4e2d\u7684 NetworkAttachmentDefinition \u8d44\u6e90\u5b9a\u4e49\u3002 \u5f53 Pod \u521b\u5efa\u65f6\uff0c kube-ovn-controller \u4f1a\u76d1\u542c\u5230 Pod \u6dfb\u52a0\u4e8b\u4ef6\uff0c\u5e76\u6839\u636e Pod \u4e2d\u7684 annotation \u53bb\u5bfb\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u5e76\u4ece\u4e2d\u8fdb\u884c IP \u7684\u5206\u914d\u548c\u7ba1\u7406\uff0c \u5e76\u5c06 Pod \u6240\u5206\u914d\u5230\u7684\u5730\u5740\u4fe1\u606f\u5199\u56de\u5230 Pod annotation \u4e2d\u3002 \u5728\u5bb9\u5668\u6240\u5728\u673a\u5668\u7684 CNI \u53ef\u4ee5\u901a\u8fc7\u5728\u914d\u7f6e\u4e2d\u914d\u7f6e kube-ovn-cni \u4f5c\u4e3a ipam \u63d2\u4ef6, kube-ovn-cni \u5c06\u4f1a\u8bfb\u53d6 Pod annotation \u5e76\u5c06\u5730\u5740\u4fe1\u606f\u901a\u8fc7 CNI \u534f\u8bae\u7684\u6807\u51c6\u683c\u5f0f\u8fd4\u56de\u7ed9\u76f8\u5e94\u7684 CNI \u63d2\u4ef6\u3002","title":"\u5de5\u4f5c\u6d41\u7a0b"},{"location":"advance/multi-nic/#_4","text":"","title":"\u4f7f\u7528\u65b9\u6cd5"},{"location":"advance/multi-nic/#kube-ovn-multus","text":"\u8bf7\u53c2\u8003 Kube-OVN \u4e00\u952e\u5b89\u88c5 \u548c Multus how to use \u6765\u5b89\u88c5 Kube-OVN \u548c Multus-CNI\u3002","title":"\u5b89\u88c5 Kube-OVN \u548c Multus"},{"location":"advance/multi-nic/#cni-ipam","text":"\u6b64\u65f6\u4e3b\u7f51\u5361\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\uff0c\u9644\u5c5e\u7f51\u5361\u4e3a\u5176\u4ed6\u7c7b\u578b CNI\u3002","title":"\u4e3a\u5176\u4ed6 CNI \u63d0\u4f9b IPAM"},{"location":"advance/multi-nic/#networkattachmentdefinition","text":"\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 macvlan \u4f5c\u4e3a\u5bb9\u5668\u7f51\u7edc\u7684\u7b2c\u4e8c\u4e2a\u7f51\u7edc\uff0c\u5e76\u5c06\u5176 ipam \u8bbe\u7f6e\u4e3a kube-ovn \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : macvlan namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"macvlan.default\" } }' spec.config.ipam.type : \u9700\u8981\u4e3a kube-ovn \u6765\u8c03\u7528 kube-ovn \u7684\u63d2\u4ef6\u6765\u83b7\u53d6\u5730\u5740\u4fe1\u606f\u3002 server_socket : Kube-OVN \u901a\u4fe1\u4f7f\u7528\u7684 socket \u6587\u4ef6\u3002 \u9ed8\u8ba4\u4f4d\u7f6e\u4e3a /run/openvswitch/kube-ovn-daemon.sock \u3002 provider : \u5f53\u524d NetworkAttachmentDefinition \u7684 <name>.<namespace> , Kube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90\u3002","title":"\u521b\u5efa NetworkAttachmentDefinition"},{"location":"advance/multi-nic/#kube-ovn","text":"\u6b64\u65f6\u591a\u5757\u7f51\u5361\u5747\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\u3002","title":"\u9644\u5c5e\u7f51\u5361\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361"},{"location":"advance/multi-nic/#networkattachmentdefinition_1","text":"\u5c06 provider \u7684\u540e\u7f00\u8bbe\u7f6e\u4e3a ovn \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" }' spec.config.type : \u8bbe\u7f6e\u4e3a kube-ovn \u6765\u89e6\u53d1 CNI \u63d2\u4ef6\u4f7f\u7528 Kube-OVN \u5b50\u7f51\u3002 server_socket : Kube-OVN \u901a\u4fe1\u4f7f\u7528\u7684 socket \u6587\u4ef6\u3002 \u9ed8\u8ba4\u4f4d\u7f6e\u4e3a /run/openvswitch/kube-ovn-daemon.sock \u3002 provider : \u5f53\u524d NetworkAttachmentDefinition \u7684 <name>.<namespace>.ovn , Kube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90\uff0c\u6ce8\u610f\u540e\u7f00\u9700\u8981\u8bbe\u7f6e\u4e3a ovn\u3002","title":"\u521b\u5efa NetworkAttachmentDefinition"},{"location":"advance/multi-nic/#kube-ovn-subnet","text":"\u521b\u5efa\u4e00\u4e2a Kube-OVN Subnet,\u8bbe\u7f6e\u5bf9\u5e94\u7684 cidrBlock \u548c exclude_ips , provider \u5e94\u8be5\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <name>.<namespace> , \u4f8b\u5982\u7528 macvlan \u63d0\u4f9b\u9644\u52a0\u7f51\u5361\uff0c\u521b\u5efa Subnet \u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : macvlan spec : protocol : IPv4 provider : macvlan.default cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 gateway , private , nat \u53ea\u5bf9 provider \u7c7b\u578b\u4e3a ovn \u7684\u7f51\u7edc\u751f\u6548\uff0c\u4e0d\u9002\u7528\u4e8e attachment network\u3002 \u5982\u679c\u4ee5 Kube-OVN \u4f5c\u4e3a\u9644\u52a0\u7f51\u5361\uff0c\u5219 provider \u5e94\u8be5\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <name>.<namespace>.ovn \uff0c\u5e76\u8981\u4ee5 ovn \u4f5c\u4e3a\u540e\u7f00\u7ed3\u675f\u3002 \u7528 Kube-OVN \u63d0\u4f9b\u9644\u52a0\u7f51\u5361\uff0c\u521b\u5efa Subnet \u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attachnet spec : protocol : IPv4 provider : attachnet.default.ovn cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10","title":"\u521b\u5efa\u4e00\u4e2a Kube-OVN Subnet"},{"location":"advance/multi-nic/#pod","text":"\u5bf9\u4e8e\u5730\u5740\u968f\u673a\u5206\u914d\u7684 Pod\uff0c\u53ea\u9700\u8981\u6dfb\u52a0\u5982\u4e0b annotation k8s.v1.cni.cncf.io/networks ,\u53d6\u503c\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <namespace>/<name> \uff1a apiVersion : v1 kind : Pod metadata : name : samplepod namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan spec : containers : - name : samplepod command : [ \"/bin/ash\" , \"-c\" , \"trap : TERM INT; sleep infinity & wait\" ] image : docker.io/library/alpine:edge","title":"\u521b\u5efa\u4e00\u4e2a\u591a\u7f51\u7edc\u7684 Pod"},{"location":"advance/multi-nic/#ip-pod","text":"\u5bf9\u4e8e\u56fa\u5b9a IP \u7684 Pod\uff0c\u6dfb\u52a0 <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_address annotation\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 macvlan.default.kubernetes.io/ip_address : 172.17.0.100 macvlan.default.kubernetes.io/mac_address : 00:00:00:53:6B:BB spec : containers : - name : static-ip image : docker.io/library/nginx:alpine","title":"\u521b\u5efa\u56fa\u5b9a IP \u7684 Pod"},{"location":"advance/multi-nic/#ip","text":"\u5bf9\u4e8e\u4f7f\u7528 ippool \u7684\u5de5\u4f5c\u8d1f\u8f7d, \u6dfb\u52a0 <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_pool annotations: apiVersion : apps/v1 kind : Deployment metadata : namespace : default name : static-workload labels : app : static-workload spec : replicas : 2 selector : matchLabels : app : static-workload template : metadata : labels : app : static-workload annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 macvlan.default.kubernetes.io/ip_pool : 172.17.0.200,172.17.0.201,172.17.0.202 spec : containers : - name : static-workload image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u521b\u5efa\u4f7f\u7528\u56fa\u5b9a IP \u7684\u5de5\u4f5c\u8d1f\u8f7d"},{"location":"advance/nat-policy-rule/","text":"NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd \u00b6 NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd\u7528\u9014 \u00b6 \u9ed8\u8ba4 VPC \u4e0b\u7684 Overlay \u5b50\u7f51\uff0c\u6253\u5f00 natOutgoing \u5f00\u5173\u65f6\uff0csubnet \u4e0b\u7684\u6240\u6709 Pod \u8bbf\u95ee\u5916\u7f51\u90fd\u9700\u8981\u505a SNAT \u6210\u5f53\u524d\u8282\u70b9\u7684 IP\uff0c\u4f46\u662f\u6709\u4e9b\u573a\u666f\u6211\u4eec\u5e76\u4e0d\u5e0c\u671b\u5b50\u7f51\u5185\u6240\u6709 Pod \u8bbf\u95ee\u5916\u7f51\u90fd\u505a SNAT\u3002 \u6240\u4ee5 NAT \u7b56\u7565\u5c31\u662f\u4e3a\u4e86\u63d0\u4f9b\u4e00\u4e2a\u63a5\u53e3\u8ba9\u7528\u6237\u51b3\u5b9a\u5b50\u7f51\u5185\u7684\u54ea\u4e9b CIDR \u6216\u8005 IP \u8bbf\u95ee\u5916\u7f51\u505a SNAT\u3002 NAT \u7b56\u7565\u89c4\u5219\u4f7f\u7528\u65b9\u6cd5 \u00b6 \u5728 subnet.Spec \u4e2d\u5f00\u542f natOutgoing \u5f00\u5173\uff0c \u5e76\u4e14\u6dfb\u52a0\u5b57\u6bb5 natOutgoingPolicyRules \u5982\u4e0b\uff1a spec : natOutgoing : true natOutgoingPolicyRules : - action : forward match : srcIPs : 10.0.11.0/30,10.0.11.254 - action : nat match : srcIPs : 10.0.11.128/26 dstIPs : 114.114.114.114,8.8.8.8 \u4ee5\u4e0a\u6848\u4f8b\u8868\u793a\u6709\u4e24\u6761 NAT \u7b56\u7565\u89c4\u5219\uff1a \u6e90 IP \u662f 10.0.11.0/30 \u6216\u8005 10.0.11.254 \u7684\u62a5\u6587\u8bbf\u95ee\u5916\u7f51\u65f6\u4e0d\u4f1a\u505a SNAT\u3002 \u6e90 IP \u662f 10.0.11.128/26 \u5e76\u4e14\u76ee\u7684 IP \u662f 114.114.114.114 \u6216\u8005 8.8.8.8 \u7684\u62a5\u6587\u8bbf\u95ee\u5916\u7f51\u65f6\u4f1a\u505a SNAT\u3002 \u5b57\u6bb5\u63cf\u8ff0\uff1a action\uff1a\u8868\u793a\u6ee1\u8db3 match \u5bf9\u5e94\u6761\u4ef6\u7684\u62a5\u6587\uff0c\u4f1a\u6267\u884c\u7684 action, action \u5206\u4e3a\u4e24\u79cd forward \u548c nat \uff0c forward \u8868\u793a\u62a5\u6587\u51fa\u5916\u7f51\u4e0d\u505a SNAT, nat \u8868\u793a\u62a5\u6587\u51fa\u5916\u7f51\u505a SNAT\u3002 \u6ca1\u6709\u914d\u7f6e natOutgoingPolicyRules \u65f6\uff0c\u9ed8\u8ba4\u60c5\u51b5\u62a5\u6587\u4ecd\u7136\u662f\u505a SNAT\u3002 match\uff1a\u8868\u793a\u62a5\u6587\u7684\u5339\u914d\u6bb5\uff0c\u5339\u914d\u6bb5\u6709 srcIPs \u548c dstIPs\uff0c \u8fd9\u91cc\u8868\u793a\u4ece\u5b50\u7f51\u5185\u5230\u5916\u7f51\u65b9\u5411\u4e0a\u7684\u62a5\u6587\u7684\u6e90 IP \u548c \u76ee\u7684 IP\u3002 match.srcIPs \u548c match.dstIPs \u652f\u6301\u591a\u4e2a cidr \u548c ip\uff0c\u4e4b\u95f4\u7528\u9017\u53f7\u9694\u5f00\u5373\u53ef\u3002 \u5982\u679c\u51fa\u73b0\u51e0\u4e2a match \u91cd\u590d\u4f46\u662f action \u4e0d\u4e00\u6837\uff0c\u6309\u7167 natOutgoingPolicyRules \u7684\u6570\u7ec4\u4f4d\u7f6e\u4e3a\u51c6\uff0c\u6570\u7ec4\u7d22\u5f15\u8d8a\u4f4e\u7684\u4f18\u5148\u7ea7\u8d8a\u9ad8\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd"},{"location":"advance/nat-policy-rule/#nat","text":"","title":"NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd"},{"location":"advance/nat-policy-rule/#nat_1","text":"\u9ed8\u8ba4 VPC \u4e0b\u7684 Overlay \u5b50\u7f51\uff0c\u6253\u5f00 natOutgoing \u5f00\u5173\u65f6\uff0csubnet \u4e0b\u7684\u6240\u6709 Pod \u8bbf\u95ee\u5916\u7f51\u90fd\u9700\u8981\u505a SNAT \u6210\u5f53\u524d\u8282\u70b9\u7684 IP\uff0c\u4f46\u662f\u6709\u4e9b\u573a\u666f\u6211\u4eec\u5e76\u4e0d\u5e0c\u671b\u5b50\u7f51\u5185\u6240\u6709 Pod \u8bbf\u95ee\u5916\u7f51\u90fd\u505a SNAT\u3002 \u6240\u4ee5 NAT \u7b56\u7565\u5c31\u662f\u4e3a\u4e86\u63d0\u4f9b\u4e00\u4e2a\u63a5\u53e3\u8ba9\u7528\u6237\u51b3\u5b9a\u5b50\u7f51\u5185\u7684\u54ea\u4e9b CIDR \u6216\u8005 IP \u8bbf\u95ee\u5916\u7f51\u505a SNAT\u3002","title":"NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd\u7528\u9014"},{"location":"advance/nat-policy-rule/#nat_2","text":"\u5728 subnet.Spec \u4e2d\u5f00\u542f natOutgoing \u5f00\u5173\uff0c \u5e76\u4e14\u6dfb\u52a0\u5b57\u6bb5 natOutgoingPolicyRules \u5982\u4e0b\uff1a spec : natOutgoing : true natOutgoingPolicyRules : - action : forward match : srcIPs : 10.0.11.0/30,10.0.11.254 - action : nat match : srcIPs : 10.0.11.128/26 dstIPs : 114.114.114.114,8.8.8.8 \u4ee5\u4e0a\u6848\u4f8b\u8868\u793a\u6709\u4e24\u6761 NAT \u7b56\u7565\u89c4\u5219\uff1a \u6e90 IP \u662f 10.0.11.0/30 \u6216\u8005 10.0.11.254 \u7684\u62a5\u6587\u8bbf\u95ee\u5916\u7f51\u65f6\u4e0d\u4f1a\u505a SNAT\u3002 \u6e90 IP \u662f 10.0.11.128/26 \u5e76\u4e14\u76ee\u7684 IP \u662f 114.114.114.114 \u6216\u8005 8.8.8.8 \u7684\u62a5\u6587\u8bbf\u95ee\u5916\u7f51\u65f6\u4f1a\u505a SNAT\u3002 \u5b57\u6bb5\u63cf\u8ff0\uff1a action\uff1a\u8868\u793a\u6ee1\u8db3 match \u5bf9\u5e94\u6761\u4ef6\u7684\u62a5\u6587\uff0c\u4f1a\u6267\u884c\u7684 action, action \u5206\u4e3a\u4e24\u79cd forward \u548c nat \uff0c forward \u8868\u793a\u62a5\u6587\u51fa\u5916\u7f51\u4e0d\u505a SNAT, nat \u8868\u793a\u62a5\u6587\u51fa\u5916\u7f51\u505a SNAT\u3002 \u6ca1\u6709\u914d\u7f6e natOutgoingPolicyRules \u65f6\uff0c\u9ed8\u8ba4\u60c5\u51b5\u62a5\u6587\u4ecd\u7136\u662f\u505a SNAT\u3002 match\uff1a\u8868\u793a\u62a5\u6587\u7684\u5339\u914d\u6bb5\uff0c\u5339\u914d\u6bb5\u6709 srcIPs \u548c dstIPs\uff0c \u8fd9\u91cc\u8868\u793a\u4ece\u5b50\u7f51\u5185\u5230\u5916\u7f51\u65b9\u5411\u4e0a\u7684\u62a5\u6587\u7684\u6e90 IP \u548c \u76ee\u7684 IP\u3002 match.srcIPs \u548c match.dstIPs \u652f\u6301\u591a\u4e2a cidr \u548c ip\uff0c\u4e4b\u95f4\u7528\u9017\u53f7\u9694\u5f00\u5373\u53ef\u3002 \u5982\u679c\u51fa\u73b0\u51e0\u4e2a match \u91cd\u590d\u4f46\u662f action \u4e0d\u4e00\u6837\uff0c\u6309\u7167 natOutgoingPolicyRules \u7684\u6570\u7ec4\u4f4d\u7f6e\u4e3a\u51c6\uff0c\u6570\u7ec4\u7d22\u5f15\u8d8a\u4f4e\u7684\u4f18\u5148\u7ea7\u8d8a\u9ad8\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NAT \u7b56\u7565\u89c4\u5219\u4f7f\u7528\u65b9\u6cd5"},{"location":"advance/node-local-dns/","text":"\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u548c Kube-OVN \u9002\u914d \u00b6 NodeLocal DNSCache \u662f\u901a\u8fc7\u96c6\u7fa4\u8282\u70b9\u4e0a\u4f5c\u4e3a DaemonSet \u8fd0\u884c DNS \u7f13\u5b58\u6765\u63d0\u9ad8\u96c6\u7fa4 DNS \u6027\u80fd\uff0c\u8be5\u529f\u80fd\u4e5f\u53ef\u4ee5\u548c Kube-OVN \u9002\u914d\u3002 \u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u90e8\u7f72 \u00b6 \u90e8\u7f72 Kubernetes \u7684\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58 \u00b6 \u8be5\u6b65\u9aa4\u53c2\u8003 Kubernetes \u5b98\u7f51\u914d\u7f6e Nodelocaldnscache \u3002 \u4f7f\u7528\u4ee5\u4e0b\u811a\u672c\u90e8\u7f72\uff1a #!bin/bash localdns = 169 .254.20.10 domain = cluster.local kubedns = 10 .96.0.10 wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml sed -i \"s/__PILLAR__LOCAL__DNS__/ $localdns /g; s/__PILLAR__DNS__DOMAIN__/ $domain /g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/ $kubedns /g\" nodelocaldns.yaml kubectl apply -f nodelocaldns.yaml \u4fee\u6539\u6bcf\u4e2a\u8282\u70b9\u4e0a\u7684 kubelet \u914d\u7f6e\u6587\u4ef6\uff0c\u5c06 /var/lib/kubelet/config.yaml \u4e2d\u7684 clusterDNS \u5b57\u6bb5\u4fee\u6539\u4e3a\u672c\u5730 DNS IP 169.254.20.10\uff0c\u7136\u540e\u91cd\u542f kubelet \u670d\u52a1\u3002 Kube-OVN \u76f8\u5e94 DNS \u914d\u7f6e \u00b6 \u90e8\u7f72\u597d Kubernetes \u7684 Nodelocal DNScache \u7ec4\u4ef6\u540e\uff0c Kube-OVN \u9700\u8981\u505a\u51fa\u4e0b\u9762\u4fee\u6539\uff1a Underlay Subnet \u5f00\u542f U2O \u5f00\u5173 \u00b6 \u5982\u679c\u662f Underlay Subnet \u9700\u8981\u4f7f\u7528\u672c\u5730 DNS \u529f\u80fd\uff0c\u9700\u8981\u5f00\u542f U2O \u529f\u80fd\uff0c\u5373\u5728 kubectl edit subnet {your subnet} \u4e2d\u914d\u7f6e spec.u2oInterconnection = true , \u5982\u679c\u662f Overlay Subnet \u5219\u4e0d\u9700\u8981\u8fd9\u6b65\u64cd\u4f5c\u3002 \u7ed9 Kube-ovn-controller \u6307\u5b9a\u5bf9\u5e94\u7684\u672c\u5730 DNS IP \u00b6 kubectl edit deployment kube-ovn-controller -n kube-system \u7ed9 spec.template.spec.containers.args \u589e\u52a0\u5b57\u6bb5 --node-local-dns-ip=169.254.20.10 \u91cd\u5efa\u5df2\u7ecf\u521b\u5efa\u7684 Pod \u00b6 \u8fd9\u6b65\u539f\u56e0\u662f\u8ba9 Pod \u91cd\u65b0\u751f\u6210 /etc/resolv.conf \u8ba9 nameserver \u6307\u5411\u672c\u5730 DNS IP\uff0c\u5982\u679c\u6ca1\u6709\u91cd\u5efa Pod \u7684 nameserver \u5c06\u4ecd\u7136\u4f7f\u7528\u96c6\u7fa4\u7684 DNS ClusterIP\u3002\u540c\u65f6 u2o \u5f00\u5173\u5982\u679c\u5f00\u542f\u4e5f\u9700\u8981\u91cd\u5efa Pod \u6765\u91cd\u65b0\u751f\u6210 Pod \u7f51\u5173\u3002 \u9a8c\u8bc1\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u529f\u80fd \u00b6 \u4ee5\u4e0a\u914d\u7f6e\u5b8c\u6210\u540e\u53ef\u4ee5\u627e\u5230 Pod \u9a8c\u8bc1\u5982\u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u7684 DNS \u670d\u52a1\u5668\u662f\u6307\u5411\u672c\u5730 169.254.20.10 \uff0c\u5e76\u6210\u529f\u89e3\u6790\u57df\u540d\uff1a # kubectl exec -it pod1 -- nslookup github.com Server: 169 .254.20.10 Address: 169 .254.20.10:53 Name: github.com Address: 20 .205.243.166 \u4e5f\u53ef\u4ee5\u5728\u8282\u70b9\u6293\u5305\u9a8c\u8bc1\u5982\u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 DNS \u67e5\u8be2\u62a5\u6587\u901a\u8fc7 ovn0 \u7f51\u5361\u5230\u8fbe\u672c\u5730\u7684 DNS \u670d\u52a1\uff0cDNS \u54cd\u5e94\u62a5\u6587\u539f\u8def\u8fd4\u56de: # tcpdump -i any port 53 06 :20:00.441889 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441889 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441950 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.441950 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.442203 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442219 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442273 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) 06 :20:00.442278 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u548c Kube-OVN \u9002\u914d"},{"location":"advance/node-local-dns/#dns-kube-ovn","text":"NodeLocal DNSCache \u662f\u901a\u8fc7\u96c6\u7fa4\u8282\u70b9\u4e0a\u4f5c\u4e3a DaemonSet \u8fd0\u884c DNS \u7f13\u5b58\u6765\u63d0\u9ad8\u96c6\u7fa4 DNS \u6027\u80fd\uff0c\u8be5\u529f\u80fd\u4e5f\u53ef\u4ee5\u548c Kube-OVN \u9002\u914d\u3002","title":"\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u548c Kube-OVN \u9002\u914d"},{"location":"advance/node-local-dns/#dns","text":"","title":"\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u90e8\u7f72"},{"location":"advance/node-local-dns/#kubernetes-dns","text":"\u8be5\u6b65\u9aa4\u53c2\u8003 Kubernetes \u5b98\u7f51\u914d\u7f6e Nodelocaldnscache \u3002 \u4f7f\u7528\u4ee5\u4e0b\u811a\u672c\u90e8\u7f72\uff1a #!bin/bash localdns = 169 .254.20.10 domain = cluster.local kubedns = 10 .96.0.10 wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml sed -i \"s/__PILLAR__LOCAL__DNS__/ $localdns /g; s/__PILLAR__DNS__DOMAIN__/ $domain /g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/ $kubedns /g\" nodelocaldns.yaml kubectl apply -f nodelocaldns.yaml \u4fee\u6539\u6bcf\u4e2a\u8282\u70b9\u4e0a\u7684 kubelet \u914d\u7f6e\u6587\u4ef6\uff0c\u5c06 /var/lib/kubelet/config.yaml \u4e2d\u7684 clusterDNS \u5b57\u6bb5\u4fee\u6539\u4e3a\u672c\u5730 DNS IP 169.254.20.10\uff0c\u7136\u540e\u91cd\u542f kubelet \u670d\u52a1\u3002","title":"\u90e8\u7f72 Kubernetes \u7684\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58"},{"location":"advance/node-local-dns/#kube-ovn-dns","text":"\u90e8\u7f72\u597d Kubernetes \u7684 Nodelocal DNScache \u7ec4\u4ef6\u540e\uff0c Kube-OVN \u9700\u8981\u505a\u51fa\u4e0b\u9762\u4fee\u6539\uff1a","title":"Kube-OVN \u76f8\u5e94 DNS \u914d\u7f6e"},{"location":"advance/node-local-dns/#underlay-subnet-u2o","text":"\u5982\u679c\u662f Underlay Subnet \u9700\u8981\u4f7f\u7528\u672c\u5730 DNS \u529f\u80fd\uff0c\u9700\u8981\u5f00\u542f U2O \u529f\u80fd\uff0c\u5373\u5728 kubectl edit subnet {your subnet} \u4e2d\u914d\u7f6e spec.u2oInterconnection = true , \u5982\u679c\u662f Overlay Subnet \u5219\u4e0d\u9700\u8981\u8fd9\u6b65\u64cd\u4f5c\u3002","title":"Underlay Subnet \u5f00\u542f U2O \u5f00\u5173"},{"location":"advance/node-local-dns/#kube-ovn-controller-dns-ip","text":"kubectl edit deployment kube-ovn-controller -n kube-system \u7ed9 spec.template.spec.containers.args \u589e\u52a0\u5b57\u6bb5 --node-local-dns-ip=169.254.20.10","title":"\u7ed9 Kube-ovn-controller \u6307\u5b9a\u5bf9\u5e94\u7684\u672c\u5730 DNS IP"},{"location":"advance/node-local-dns/#pod","text":"\u8fd9\u6b65\u539f\u56e0\u662f\u8ba9 Pod \u91cd\u65b0\u751f\u6210 /etc/resolv.conf \u8ba9 nameserver \u6307\u5411\u672c\u5730 DNS IP\uff0c\u5982\u679c\u6ca1\u6709\u91cd\u5efa Pod \u7684 nameserver \u5c06\u4ecd\u7136\u4f7f\u7528\u96c6\u7fa4\u7684 DNS ClusterIP\u3002\u540c\u65f6 u2o \u5f00\u5173\u5982\u679c\u5f00\u542f\u4e5f\u9700\u8981\u91cd\u5efa Pod \u6765\u91cd\u65b0\u751f\u6210 Pod \u7f51\u5173\u3002","title":"\u91cd\u5efa\u5df2\u7ecf\u521b\u5efa\u7684 Pod"},{"location":"advance/node-local-dns/#dns_1","text":"\u4ee5\u4e0a\u914d\u7f6e\u5b8c\u6210\u540e\u53ef\u4ee5\u627e\u5230 Pod \u9a8c\u8bc1\u5982\u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u7684 DNS \u670d\u52a1\u5668\u662f\u6307\u5411\u672c\u5730 169.254.20.10 \uff0c\u5e76\u6210\u529f\u89e3\u6790\u57df\u540d\uff1a # kubectl exec -it pod1 -- nslookup github.com Server: 169 .254.20.10 Address: 169 .254.20.10:53 Name: github.com Address: 20 .205.243.166 \u4e5f\u53ef\u4ee5\u5728\u8282\u70b9\u6293\u5305\u9a8c\u8bc1\u5982\u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 DNS \u67e5\u8be2\u62a5\u6587\u901a\u8fc7 ovn0 \u7f51\u5361\u5230\u8fbe\u672c\u5730\u7684 DNS \u670d\u52a1\uff0cDNS \u54cd\u5e94\u62a5\u6587\u539f\u8def\u8fd4\u56de: # tcpdump -i any port 53 06 :20:00.441889 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441889 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441950 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.441950 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.442203 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442219 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442273 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) 06 :20:00.442278 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9a8c\u8bc1\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u529f\u80fd"},{"location":"advance/offload-corigine/","text":"\u82af\u542f\u6e90\u7f51\u5361 Offload \u652f\u6301 \u00b6 Kube-OVN \u5728\u6700\u7ec8\u7684\u6570\u636e\u5e73\u9762\u4f7f\u7528 OVS \u6765\u5b8c\u6210\u6d41\u91cf\u8f6c\u53d1\uff0c\u76f8\u5173\u7684\u6d41\u8868\u5339\u914d\uff0c\u96a7\u9053\u5c01\u88c5\u7b49\u529f\u80fd\u4e3a CPU \u5bc6\u96c6\u578b\uff0c\u5728\u5927\u6d41\u91cf\u4e0b\u4f1a\u6d88\u8017\u5927\u91cf CPU \u8d44\u6e90\u5e76\u5bfc\u81f4 \u5ef6\u8fdf\u4e0a\u5347\u548c\u541e\u5410\u91cf\u4e0b\u964d\u3002\u82af\u542f\u6e90\u7684 Agilio CX \u7cfb\u5217\u667a\u80fd\u7f51\u5361\u53ef\u4ee5\u5c06 OVS \u76f8\u5173\u7684\u64cd\u4f5c\u5378\u8f7d\u5230\u786c\u4ef6\u7f51\u5361\u4e2d\u6267\u884c\u3002 \u8be5\u6280\u672f\u53ef\u4ee5\u5728\u65e0\u9700\u5bf9 OVS \u63a7\u5236\u5e73\u9762\u8fdb\u884c\u4fee\u6539\u7684\u60c5\u51b5\u4e0b\uff0c\u7f29\u77ed\u6570\u636e\u8def\u5f84\uff0c\u907f\u514d\u5bf9\u4e3b\u673a CPU \u8d44\u6e90\u7684\u4f7f\u7528\uff0c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u663e\u8457\u63d0\u5347\u541e\u5410\u91cf\u3002 \u524d\u7f6e\u6761\u4ef6 \u00b6 \u82af\u542f\u6e90 Agilio CX \u7cfb\u5217\u7684\u786c\u4ef6\u7f51\u5361\u3002 CentOS 8 Stream \u6216\u4e0a\u6e38 Linux 5.7 \u4ee5\u4e0a\u5185\u6838\u652f\u6301\u3002 \u7531\u4e8e\u5f53\u524d\u7f51\u5361\u4e0d\u652f\u6301 dp_hash \u548c hash \u64cd\u4f5c\u5378\u8f7d\uff0c\u9700\u5173\u95ed OVN LB \u529f\u80fd\u3002 \u8bbe\u7f6e\u7f51\u5361 SR-IOV \u6a21\u5f0f \u00b6 \u7528\u6237\u53ef\u53c2\u8003 Agilio Open vSwitch TC User Guide \u83b7\u5f97\u8be5\u7f51\u5361\u4f7f\u7528\u7684\u66f4\u591a\u8be6\u7ec6\u4fe1\u606f\u3002 \u4fdd\u5b58\u4e0b\u5217\u811a\u672c\u7528\u4e8e\u540e\u7eed\u6267\u884c\u56fa\u4ef6\u76f8\u5173\u64cd\u4f5c\uff1a #!/bin/bash DEVICE = ${ 1 } DEFAULT_ASSY = scan ASSY = ${ 2 :- ${ DEFAULT_ASSY }} APP = ${ 3 :- flower } if [ \"x ${ DEVICE } \" = \"x\" -o ! -e /sys/class/net/ ${ DEVICE } ] ; then echo Syntax: ${ 0 } device [ ASSY ] [ APP ] echo echo This script associates the TC Offload firmware echo with a Netronome SmartNIC. echo echo device: is the network device associated with the SmartNIC echo ASSY: defaults to ${ DEFAULT_ASSY } echo APP: defaults to flower. flower-next is supported if updated echo firmware has been installed. exit 1 fi # It is recommended that the assembly be determined by inspection # The following code determines the value via the debug interface if [ \" ${ ASSY } x\" = \"scanx\" ] ; then ethtool -W ${ DEVICE } 0 DEBUG = $( ethtool -w ${ DEVICE } data /dev/stdout | strings ) SERIAL = $( echo \" ${ DEBUG } \" | grep \"^SN:\" ) ASSY = $( echo ${ SERIAL } | grep -oE AMDA [ 0 -9 ]{ 4 } ) fi PCIADDR = $( basename $( readlink -e /sys/class/net/ ${ DEVICE } /device )) FWDIR = \"/lib/firmware/netronome\" # AMDA0081 and AMDA0097 uses the same firmware if [ \" ${ ASSY } \" = \"AMDA0081\" ] ; then if [ ! -e ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw ] ; then ln -sf nic_AMDA0097.nffw ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw fi fi FW = \" ${ FWDIR } /pci- ${ PCIADDR } .nffw\" ln -sf \" ${ APP } /nic_ ${ ASSY } .nffw\" \" ${ FW } \" # insert distro-specific initramfs section here... \u5207\u6362\u56fa\u4ef6\u9009\u9879\u5e76\u91cd\u8f7d\u9a71\u52a8\uff1a ./agilio-tc-fw-select.sh ens47np0 scan rmmod nfp modprobe nfp \u68c0\u67e5\u53ef\u7528 VF \u6570\u91cf\uff0c\u5e76\u521b\u5efa VF\uff1a # cat /sys/class/net/ens3/device/sriov_totalvfs 65 # echo 4 > /sys/class/net/ens47/device/sriov_numvfs \u5b89\u88c5 SR-IOV Device Plugin \u00b6 \u7531\u4e8e\u6bcf\u4e2a\u673a\u5668\u7684 VF \u6570\u91cf\u6709\u9650\uff0c\u6bcf\u4e2a\u4f7f\u7528\u52a0\u901f\u7684 Pod \u4f1a\u5360\u7528 VF \u8d44\u6e90\uff0c\u6211\u4eec\u9700\u8981\u4f7f\u7528 SR-IOV Device Plugin \u7ba1\u7406\u76f8\u5e94\u8d44\u6e90\uff0c\u4f7f\u5f97\u8c03\u5ea6\u5668\u77e5\u9053\u5982\u4f55\u6839\u636e \u8d44\u6e90\u8fdb\u884c\u8c03\u5ea6\u3002 \u521b\u5efa SR-IOV \u76f8\u5173 Configmap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"corigine.com\", \"resourceName\": \"agilio_sriov\", \"selectors\": { \"vendors\": [\"19ee\"], \"devices\": [\"6003\"], \"drivers\": [\"nfp_netvf\"] } } ] } \u53c2\u8003 SR-IOV \u6587\u6863 \u8fdb\u884c\u90e8\u7f72: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml \u68c0\u67e5 SR-IOV \u8d44\u6e90\u662f\u5426\u5df2\u7ecf\u6ce8\u518c\u5230 Kubernetes Node \u4e2d\uff1a kubectl describe no containerserver | grep corigine corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov 0 0 \u5b89\u88c5 Multus-CNI \u00b6 SR-IOV Device Plugin \u8c03\u5ea6\u65f6\u83b7\u5f97\u7684\u8bbe\u5907 ID \u9700\u8981\u901a\u8fc7 Multus-CNI \u4f20\u9012\u7ed9 Kube-OVN\uff0c\u56e0\u6b64\u9700\u8981\u914d\u7f6e Multus-CNI \u914d\u5408\u5b8c\u6210\u76f8\u5173\u4efb\u52a1\u3002 \u53c2\u8003 Multus-CNI \u6587\u6863 \u8fdb\u884c\u90e8\u7f72\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml \u521b\u5efa NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : corigine.com/agilio_sriov spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : \u683c\u5f0f\u4e3a\u5f53\u524d NetworkAttachmentDefinition \u7684 {name}.{namespace}.ovn\u3002 Kube-OVN \u4e2d\u5f00\u542f\u5378\u8f7d\u6a21\u5f0f \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u76f8\u5173\u53c2\u6570\uff0c IFACE \u9700\u8981\u4e3a\u7269\u7406\u7f51\u5361\u540d\uff0c\u8be5\u7f51\u5361\u9700\u8981\u6709\u53ef\u8def\u7531 IP\uff1a ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" \u5b89\u88c5 Kube-OVN\uff1a bash install.sh \u521b\u5efa\u4f7f\u7528 VF \u7f51\u5361\u7684 Pod \u00b6 \u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b yaml \u683c\u5f0f\u521b\u5efa\u4f7f\u7528 VF \u8fdb\u884c\u7f51\u7edc\u5378\u8f7d\u52a0\u901f\u7684 Pod: apiVersion : v1 kind : Pod metadata : name : nginx namespace : default annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : corigine.com/agilio_sriov : '1' limits : corigine.com/agilio_sriov : '1' v1.multus-cni.io/default-network : \u4e3a\u4e0a\u4e00\u6b65\u9aa4\u4e2d NetworkAttachmentDefinition \u7684 {namespace}/{name}\u3002 \u53ef\u901a\u8fc7\u5728 Pod \u8fd0\u884c\u8282\u70b9\u7684 ovs-ovn \u5bb9\u5668\u4e2d\u8fd0\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u89c2\u5bdf\u5378\u8f7d\u662f\u5426\u6210\u529f\uff1a # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h \u5982\u679c\u6709 offloaded:yes, dp:tc \u5185\u5bb9\u8bc1\u660e\u5378\u8f7d\u6210\u529f\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u82af\u542f\u6e90\u7f51\u5361 Offload \u652f\u6301"},{"location":"advance/offload-corigine/#offload","text":"Kube-OVN \u5728\u6700\u7ec8\u7684\u6570\u636e\u5e73\u9762\u4f7f\u7528 OVS \u6765\u5b8c\u6210\u6d41\u91cf\u8f6c\u53d1\uff0c\u76f8\u5173\u7684\u6d41\u8868\u5339\u914d\uff0c\u96a7\u9053\u5c01\u88c5\u7b49\u529f\u80fd\u4e3a CPU \u5bc6\u96c6\u578b\uff0c\u5728\u5927\u6d41\u91cf\u4e0b\u4f1a\u6d88\u8017\u5927\u91cf CPU \u8d44\u6e90\u5e76\u5bfc\u81f4 \u5ef6\u8fdf\u4e0a\u5347\u548c\u541e\u5410\u91cf\u4e0b\u964d\u3002\u82af\u542f\u6e90\u7684 Agilio CX \u7cfb\u5217\u667a\u80fd\u7f51\u5361\u53ef\u4ee5\u5c06 OVS \u76f8\u5173\u7684\u64cd\u4f5c\u5378\u8f7d\u5230\u786c\u4ef6\u7f51\u5361\u4e2d\u6267\u884c\u3002 \u8be5\u6280\u672f\u53ef\u4ee5\u5728\u65e0\u9700\u5bf9 OVS \u63a7\u5236\u5e73\u9762\u8fdb\u884c\u4fee\u6539\u7684\u60c5\u51b5\u4e0b\uff0c\u7f29\u77ed\u6570\u636e\u8def\u5f84\uff0c\u907f\u514d\u5bf9\u4e3b\u673a CPU \u8d44\u6e90\u7684\u4f7f\u7528\uff0c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u663e\u8457\u63d0\u5347\u541e\u5410\u91cf\u3002","title":"\u82af\u542f\u6e90\u7f51\u5361 Offload \u652f\u6301"},{"location":"advance/offload-corigine/#_1","text":"\u82af\u542f\u6e90 Agilio CX \u7cfb\u5217\u7684\u786c\u4ef6\u7f51\u5361\u3002 CentOS 8 Stream \u6216\u4e0a\u6e38 Linux 5.7 \u4ee5\u4e0a\u5185\u6838\u652f\u6301\u3002 \u7531\u4e8e\u5f53\u524d\u7f51\u5361\u4e0d\u652f\u6301 dp_hash \u548c hash \u64cd\u4f5c\u5378\u8f7d\uff0c\u9700\u5173\u95ed OVN LB \u529f\u80fd\u3002","title":"\u524d\u7f6e\u6761\u4ef6"},{"location":"advance/offload-corigine/#sr-iov","text":"\u7528\u6237\u53ef\u53c2\u8003 Agilio Open vSwitch TC User Guide \u83b7\u5f97\u8be5\u7f51\u5361\u4f7f\u7528\u7684\u66f4\u591a\u8be6\u7ec6\u4fe1\u606f\u3002 \u4fdd\u5b58\u4e0b\u5217\u811a\u672c\u7528\u4e8e\u540e\u7eed\u6267\u884c\u56fa\u4ef6\u76f8\u5173\u64cd\u4f5c\uff1a #!/bin/bash DEVICE = ${ 1 } DEFAULT_ASSY = scan ASSY = ${ 2 :- ${ DEFAULT_ASSY }} APP = ${ 3 :- flower } if [ \"x ${ DEVICE } \" = \"x\" -o ! -e /sys/class/net/ ${ DEVICE } ] ; then echo Syntax: ${ 0 } device [ ASSY ] [ APP ] echo echo This script associates the TC Offload firmware echo with a Netronome SmartNIC. echo echo device: is the network device associated with the SmartNIC echo ASSY: defaults to ${ DEFAULT_ASSY } echo APP: defaults to flower. flower-next is supported if updated echo firmware has been installed. exit 1 fi # It is recommended that the assembly be determined by inspection # The following code determines the value via the debug interface if [ \" ${ ASSY } x\" = \"scanx\" ] ; then ethtool -W ${ DEVICE } 0 DEBUG = $( ethtool -w ${ DEVICE } data /dev/stdout | strings ) SERIAL = $( echo \" ${ DEBUG } \" | grep \"^SN:\" ) ASSY = $( echo ${ SERIAL } | grep -oE AMDA [ 0 -9 ]{ 4 } ) fi PCIADDR = $( basename $( readlink -e /sys/class/net/ ${ DEVICE } /device )) FWDIR = \"/lib/firmware/netronome\" # AMDA0081 and AMDA0097 uses the same firmware if [ \" ${ ASSY } \" = \"AMDA0081\" ] ; then if [ ! -e ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw ] ; then ln -sf nic_AMDA0097.nffw ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw fi fi FW = \" ${ FWDIR } /pci- ${ PCIADDR } .nffw\" ln -sf \" ${ APP } /nic_ ${ ASSY } .nffw\" \" ${ FW } \" # insert distro-specific initramfs section here... \u5207\u6362\u56fa\u4ef6\u9009\u9879\u5e76\u91cd\u8f7d\u9a71\u52a8\uff1a ./agilio-tc-fw-select.sh ens47np0 scan rmmod nfp modprobe nfp \u68c0\u67e5\u53ef\u7528 VF \u6570\u91cf\uff0c\u5e76\u521b\u5efa VF\uff1a # cat /sys/class/net/ens3/device/sriov_totalvfs 65 # echo 4 > /sys/class/net/ens47/device/sriov_numvfs","title":"\u8bbe\u7f6e\u7f51\u5361 SR-IOV \u6a21\u5f0f"},{"location":"advance/offload-corigine/#sr-iov-device-plugin","text":"\u7531\u4e8e\u6bcf\u4e2a\u673a\u5668\u7684 VF \u6570\u91cf\u6709\u9650\uff0c\u6bcf\u4e2a\u4f7f\u7528\u52a0\u901f\u7684 Pod \u4f1a\u5360\u7528 VF \u8d44\u6e90\uff0c\u6211\u4eec\u9700\u8981\u4f7f\u7528 SR-IOV Device Plugin \u7ba1\u7406\u76f8\u5e94\u8d44\u6e90\uff0c\u4f7f\u5f97\u8c03\u5ea6\u5668\u77e5\u9053\u5982\u4f55\u6839\u636e \u8d44\u6e90\u8fdb\u884c\u8c03\u5ea6\u3002 \u521b\u5efa SR-IOV \u76f8\u5173 Configmap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"corigine.com\", \"resourceName\": \"agilio_sriov\", \"selectors\": { \"vendors\": [\"19ee\"], \"devices\": [\"6003\"], \"drivers\": [\"nfp_netvf\"] } } ] } \u53c2\u8003 SR-IOV \u6587\u6863 \u8fdb\u884c\u90e8\u7f72: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml \u68c0\u67e5 SR-IOV \u8d44\u6e90\u662f\u5426\u5df2\u7ecf\u6ce8\u518c\u5230 Kubernetes Node \u4e2d\uff1a kubectl describe no containerserver | grep corigine corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov 0 0","title":"\u5b89\u88c5 SR-IOV Device Plugin"},{"location":"advance/offload-corigine/#multus-cni","text":"SR-IOV Device Plugin \u8c03\u5ea6\u65f6\u83b7\u5f97\u7684\u8bbe\u5907 ID \u9700\u8981\u901a\u8fc7 Multus-CNI \u4f20\u9012\u7ed9 Kube-OVN\uff0c\u56e0\u6b64\u9700\u8981\u914d\u7f6e Multus-CNI \u914d\u5408\u5b8c\u6210\u76f8\u5173\u4efb\u52a1\u3002 \u53c2\u8003 Multus-CNI \u6587\u6863 \u8fdb\u884c\u90e8\u7f72\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml \u521b\u5efa NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : corigine.com/agilio_sriov spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : \u683c\u5f0f\u4e3a\u5f53\u524d NetworkAttachmentDefinition \u7684 {name}.{namespace}.ovn\u3002","title":"\u5b89\u88c5 Multus-CNI"},{"location":"advance/offload-corigine/#kube-ovn","text":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u76f8\u5173\u53c2\u6570\uff0c IFACE \u9700\u8981\u4e3a\u7269\u7406\u7f51\u5361\u540d\uff0c\u8be5\u7f51\u5361\u9700\u8981\u6709\u53ef\u8def\u7531 IP\uff1a ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" \u5b89\u88c5 Kube-OVN\uff1a bash install.sh","title":"Kube-OVN \u4e2d\u5f00\u542f\u5378\u8f7d\u6a21\u5f0f"},{"location":"advance/offload-corigine/#vf-pod","text":"\u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b yaml \u683c\u5f0f\u521b\u5efa\u4f7f\u7528 VF \u8fdb\u884c\u7f51\u7edc\u5378\u8f7d\u52a0\u901f\u7684 Pod: apiVersion : v1 kind : Pod metadata : name : nginx namespace : default annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : corigine.com/agilio_sriov : '1' limits : corigine.com/agilio_sriov : '1' v1.multus-cni.io/default-network : \u4e3a\u4e0a\u4e00\u6b65\u9aa4\u4e2d NetworkAttachmentDefinition \u7684 {namespace}/{name}\u3002 \u53ef\u901a\u8fc7\u5728 Pod \u8fd0\u884c\u8282\u70b9\u7684 ovs-ovn \u5bb9\u5668\u4e2d\u8fd0\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u89c2\u5bdf\u5378\u8f7d\u662f\u5426\u6210\u529f\uff1a # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h \u5982\u679c\u6709 offloaded:yes, dp:tc \u5185\u5bb9\u8bc1\u660e\u5378\u8f7d\u6210\u529f\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u521b\u5efa\u4f7f\u7528 VF \u7f51\u5361\u7684 Pod"},{"location":"advance/offload-mellanox/","text":"Mellanox \u7f51\u5361 Offload \u652f\u6301 \u00b6 Kube-OVN \u5728\u6700\u7ec8\u7684\u6570\u636e\u5e73\u9762\u4f7f\u7528 OVS \u6765\u5b8c\u6210\u6d41\u91cf\u8f6c\u53d1\uff0c\u76f8\u5173\u7684\u6d41\u8868\u5339\u914d\uff0c\u96a7\u9053\u5c01\u88c5\u7b49\u529f\u80fd\u4e3a CPU \u5bc6\u96c6\u578b\uff0c\u5728\u5927\u6d41\u91cf\u4e0b\u4f1a\u6d88\u8017\u5927\u91cf CPU \u8d44\u6e90\u5e76\u5bfc\u81f4 \u5ef6\u8fdf\u4e0a\u5347\u548c\u541e\u5410\u91cf\u4e0b\u964d\u3002Mellanox \u7684 Accelerated Switching And Packet Processing (ASAP\u00b2) \u6280\u672f\u53ef\u4ee5\u5c06 OVS \u76f8\u5173\u7684\u64cd\u4f5c\u5378\u8f7d\u5230\u786c\u4ef6\u7f51\u5361\u5185\u7684 eSwitch \u4e0a\u6267\u884c\u3002\u8be5\u6280\u672f\u53ef\u4ee5\u5728\u65e0\u9700\u5bf9 OVS \u63a7\u5236\u5e73\u9762\u8fdb\u884c\u4fee\u6539\u7684\u60c5\u51b5\u4e0b\uff0c\u7f29\u77ed\u6570\u636e\u8def\u5f84\uff0c\u907f\u514d\u5bf9\u4e3b\u673a CPU \u8d44\u6e90\u7684\u4f7f\u7528\uff0c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u663e\u8457\u63d0\u5347\u541e\u5410\u91cf\u3002 \u524d\u7f6e\u6761\u4ef6 \u00b6 Mellanox CX5/CX6/CX7/BlueField \u7b49\u652f\u6301 ASAP\u00b2 \u7684\u786c\u4ef6\u7f51\u5361\u3002 CentOS 8 Stream \u6216\u4e0a\u6e38 Linux 5.7 \u4ee5\u4e0a\u5185\u6838\u652f\u6301\u3002 \u7531\u4e8e\u5f53\u524d\u7f51\u5361\u4e0d\u652f\u6301 dp_hash \u548c hash \u64cd\u4f5c\u5378\u8f7d\uff0c\u9700\u5173\u95ed OVN LB \u529f\u80fd\u3002 \u4e3a\u4e86\u652f\u6301\u5378\u8f7d\u6a21\u5f0f\uff0c\u7f51\u5361\u4e0d\u80fd\u505a bond\u3002 \u914d\u7f6e SR-IOV \u548c Device Plugin \u00b6 Mellanox \u7f51\u5361\u652f\u6301\u4e24\u79cd\u914d\u7f6e offload \u7684\u65b9\u5f0f\uff0c\u4e00\u79cd\u624b\u52a8\u914d\u7f6e\u7f51\u5361 SR-IOV \u548c Device Plugin\uff0c\u53e6\u4e00\u79cd\u901a\u8fc7 sriov-network-operator \u8fdb\u884c\u81ea\u52a8\u914d\u7f6e\u3002 \u624b\u52a8\u914d\u7f6e SR-IOV \u548c Device Plugin \u00b6 \u67e5\u8be2\u7f51\u5361\u7684\u8bbe\u5907 ID\uff0c\u4e0b\u9762\u7684\u4f8b\u5b50\u4e2d\u4e3a 42:00.0 \uff1a # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] \u6839\u636e\u8bbe\u5907 ID \u627e\u5230\u5bf9\u5e94\u7f51\u5361\uff1a # ls -l /sys/class/net/ | grep 42:00.0 lrwxrwxrwx. 1 root root 0 Jul 22 23 :16 p4p1 -> ../../devices/pci0000:40/0000:40:02.0/0000:42:00.0/net/p4p1 \u68c0\u67e5\u53ef\u7528 VF \u6570\u91cf\uff1a # cat /sys/class/net/p4p1/device/sriov_totalvfs 8 \u521b\u5efa VF\uff0c\u603b\u6570\u4e0d\u8981\u8d85\u8fc7\u4e0a\u9762\u67e5\u8be2\u51fa\u7684\u6570\u91cf\uff1a # echo '4' > /sys/class/net/p4p1/device/sriov_numvfs # ip link show p4p1 10 : p4p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:59:9f:c1:ec:12 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 1 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 2 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 3 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off # ip link set p4p1 up \u627e\u5230\u4e0a\u8ff0 VF \u5bf9\u5e94\u7684\u8bbe\u5907 ID\uff1a # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] \u5c06 VF \u4ece\u9a71\u52a8\u4e2d\u89e3\u7ed1\uff1a echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind \u5f00\u542f eSwitch \u6a21\u5f0f\uff0c\u5e76\u8bbe\u7f6e\u786c\u4ef6\u5378\u8f7d\uff1a devlink dev eswitch set pci/0000:42:00.0 mode switchdev ethtool -K enp66s0f0 hw-tc-offload on \u91cd\u65b0\u7ed1\u5b9a\u9a71\u52a8\uff0c\u5b8c\u6210 VF \u8bbe\u7f6e\uff1a echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/bind NetworkManager \u7684\u4e00\u4e9b\u884c\u4e3a\u53ef\u80fd\u4f1a\u5bfc\u81f4\u9a71\u52a8\u5f02\u5e38\uff0c\u5982\u679c\u5378\u8f7d\u51fa\u73b0\u95ee\u9898\u5efa\u8bae\u5173\u95ed NetworkManager \u518d\u8fdb\u884c\u5c1d\u8bd5\uff1a systemctl stop NetworkManager systemctl disable NetworkManager \u7531\u4e8e\u6bcf\u4e2a\u673a\u5668\u7684 VF \u6570\u91cf\u4f18\u5148\uff0c\u6bcf\u4e2a\u4f7f\u7528\u52a0\u901f\u7684 Pod \u4f1a\u5360\u7528 VF \u8d44\u6e90\uff0c\u6211\u4eec\u9700\u8981\u4f7f\u7528 SR-IOV Device Plugin \u7ba1\u7406\u76f8\u5e94\u8d44\u6e90\uff0c\u4f7f\u5f97\u8c03\u5ea6\u5668\u77e5\u9053\u5982\u4f55\u6839\u636e \u8d44\u6e90\u8fdb\u884c\u8c03\u5ea6\u3002 \u521b\u5efa SR-IOV \u76f8\u5173 Configmap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"mellanox.com\", \"resourceName\": \"cx5_sriov_switchdev\", \"selectors\": { \"vendors\": [\"15b3\"], \"devices\": [\"1018\"], \"drivers\": [\"mlx5_core\"] } } ] } \u53c2\u8003 SR-IOV \u6587\u6863 \u8fdb\u884c\u90e8\u7f72: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml \u68c0\u67e5 SR-IOV \u8d44\u6e90\u662f\u5426\u5df2\u7ecf\u6ce8\u518c\u5230 Kubernetes Node \u4e2d\uff1a kubectl describe node kube-ovn-01 | grep mellanox mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev 0 0 \u4f7f\u7528 sriov-network-operator \u914d\u7f6e SR-IOV \u548c Device Plugin \u00b6 \u5b89\u88c5 node-feature-discovery \u81ea\u52a8\u68c0\u6d4b\u786c\u4ef6\u7684\u529f\u80fd\u548c\u7cfb\u7edf\u914d\u7f6e: kubectl apply -k https://github.com/kubernetes-sigs/node-feature-discovery/deployment/overlays/default?ref = v0.11.3 \u6216\u8005\u901a\u8fc7\u4e0b\u9762\u7684\u547d\u4ee4\uff0c\u624b\u52a8\u7ed9\u6709 offload \u80fd\u529b\u7684\u7f51\u5361\u589e\u52a0 annotation: kubectl label nodes [ offloadNicNode ] feature.node.kubernetes.io/network-sriov.capable = true \u514b\u9686\u4ee3\u7801\u4ed3\u5e93\u5e76\u5b89\u88c5 Operator\uff1a git clone --depth = 1 https://github.com/kubeovn/sriov-network-operator.git kubectl apply -k sriov-network-operator/deploy \u68c0\u67e5 Operator \u7ec4\u4ef6\u662f\u5426\u5de5\u4f5c\u6b63\u5e38\uff1a # kubectl get -n kube-system all | grep sriov NAME READY STATUS RESTARTS AGE pod/sriov-network-config-daemon-bf9nt 1 /1 Running 0 8s pod/sriov-network-operator-54d7545f65-296gb 1 /1 Running 0 10s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/sriov-network-config-daemon 1 1 1 1 1 beta.kubernetes.io/os = linux,feature.node.kubernetes.io/network-sriov.capable = true 8s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/sriov-network-operator 1 /1 1 1 10s NAME DESIRED CURRENT READY AGE replicaset.apps/sriov-network-operator-54d7545f65 1 1 1 10s \u68c0\u67e5 SriovNetworkNodeState \uff0c\u4e0b\u9762\u4ee5 node1 \u8282\u70b9\u4e3a\u4f8b\uff0c\u8be5\u8282\u70b9\u4e0a\u6709\u4e24\u4e2a Mellanox \u7f51\u5361\uff1a # kubectl get sriovnetworknodestates.sriovnetwork.openshift.io -n kube-system node1 -o yaml apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodeState spec: ... status: interfaces: - deviceID: \"1017\" driver: mlx5_core mtu: 1500 pciAddress: \"0000:5f:00.0\" totalvfs: 8 vendor: \"15b3\" linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb name: ens41f0np0 - deviceID: \"1017\" driver: mlx5_core mtu: 1500 pciAddress: \"0000:5f:00.1\" totalvfs: 8 vendor: \"15b3\" linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb name: ens41f1np1 \u521b\u5efa SriovNetworkNodePolicy \u8d44\u6e90\uff0c\u5e76\u901a\u8fc7 nicSelector \u9009\u62e9\u8981\u7ba1\u7406\u7684\u7f51\u5361\uff1a apiVersion : sriovnetwork.openshift.io/v1 kind : SriovNetworkNodePolicy metadata : name : policy namespace : kube-system spec : nodeSelector : feature.node.kubernetes.io/network-sriov.capable : \"true\" eSwitchMode : switchdev numVfs : 3 nicSelector : pfNames : - ens41f0np0 - ens41f1np1 resourceName : cx_sriov_switchdev \u518d\u6b21\u68c0\u67e5 SriovNetworkNodeState \u7684 status \u5b57\u6bb5\uff1a # kubectl get sriovnetworknodestates.sriovnetwork.openshift.io -n kube-system node1 -o yaml ... spec: interfaces: - eSwitchMode: switchdev name: ens41f0np0 numVfs: 3 pciAddress: 0000 :5f:00.0 vfGroups: - policyName: policy vfRange: 0 -2 resourceName: cx_sriov_switchdev - eSwitchMode: switchdev name: ens41f1np1 numVfs: 3 pciAddress: 0000 :5f:00.1 vfGroups: - policyName: policy vfRange: 0 -2 resourceName: cx_sriov_switchdev status: interfaces - Vfs: - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.2 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.3 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.4 vendor: \"15b3\" deviceID: \"1017\" driver: mlx5_core linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:ab mtu: 1500 name: ens41f0np0 numVfs: 3 pciAddress: 0000 :5f:00.0 totalvfs: 3 vendor: \"15b3\" - Vfs: - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.5 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.6 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.7 vendor: \"15b3\" deviceID: \"1017\" driver: mlx5_core linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb mtu: 1500 name: ens41f1np1 numVfs: 3 pciAddress: 0000 :5f:00.1 totalvfs: 3 vendor: \"15b3\" \u68c0\u67e5 VF \u7684\u72b6\u6001\uff1a # lspci -nn | grep ConnectX 5f:00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 5f:00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 5f:00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.6 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.7 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] \u68c0\u67e5 PF \u5de5\u4f5c\u6a21\u5f0f\uff1a # cat /sys/class/net/ens41f0np0/compat/devlink/mode switchdev \u5b89\u88c5 Multus-CNI \u00b6 SR-IOV Device Plugin \u8c03\u5ea6\u65f6\u83b7\u5f97\u7684\u8bbe\u5907 ID \u9700\u8981\u901a\u8fc7 Multus-CNI \u4f20\u9012\u7ed9 Kube-OVN\uff0c\u56e0\u6b64\u9700\u8981\u914d\u7f6e Multus-CNI \u914d\u5408\u5b8c\u6210\u76f8\u5173\u4efb\u52a1\u3002 \u53c2\u8003 Multus-CNI \u6587\u6863 \u8fdb\u884c\u90e8\u7f72\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml \u521b\u5efa NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : mellanox.com/cx5_sriov_switchdev spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : \u683c\u5f0f\u4e3a\u5f53\u524d NetworkAttachmentDefinition \u7684 {name}.{namespace}.ovn\u3002 Kube-OVN \u4e2d\u5f00\u542f\u5378\u8f7d\u6a21\u5f0f \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u76f8\u5173\u53c2\u6570\uff0c IFACE \u9700\u8981\u4e3a\u7269\u7406\u7f51\u5361\u540d\uff0c\u8be5\u7f51\u5361\u9700\u8981\u6709\u53ef\u8def\u7531 IP\uff1a ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" \u5b89\u88c5 Kube-OVN\uff1a bash install.sh \u521b\u5efa\u4f7f\u7528 VF \u7f51\u5361\u7684 Pod \u00b6 \u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b yaml \u683c\u5f0f\u521b\u5efa\u4f7f\u7528 VF \u8fdb\u884c\u7f51\u7edc\u5378\u8f7d\u52a0\u901f\u7684 Pod: apiVersion : v1 kind : Pod metadata : name : nginx annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : mellanox.com/cx5_sriov_switchdev : '1' limits : mellanox.com/cx5_sriov_switchdev : '1' v1.multus-cni.io/default-network : \u4e3a\u4e0a\u4e00\u6b65\u9aa4\u4e2d NetworkAttachmentDefinition \u7684 {namespace}/{name}\u3002 \u53ef\u901a\u8fc7\u5728 Pod \u8fd0\u884c\u8282\u70b9\u7684 ovs-ovn \u5bb9\u5668\u4e2d\u8fd0\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u89c2\u5bdf\u5378\u8f7d\u662f\u5426\u6210\u529f\uff1a # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h \u5982\u679c\u6709 offloaded:yes, dp:tc \u5185\u5bb9\u8bc1\u660e\u5378\u8f7d\u6210\u529f\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Mellanox \u7f51\u5361 Offload \u652f\u6301"},{"location":"advance/offload-mellanox/#mellanox-offload","text":"Kube-OVN \u5728\u6700\u7ec8\u7684\u6570\u636e\u5e73\u9762\u4f7f\u7528 OVS \u6765\u5b8c\u6210\u6d41\u91cf\u8f6c\u53d1\uff0c\u76f8\u5173\u7684\u6d41\u8868\u5339\u914d\uff0c\u96a7\u9053\u5c01\u88c5\u7b49\u529f\u80fd\u4e3a CPU \u5bc6\u96c6\u578b\uff0c\u5728\u5927\u6d41\u91cf\u4e0b\u4f1a\u6d88\u8017\u5927\u91cf CPU \u8d44\u6e90\u5e76\u5bfc\u81f4 \u5ef6\u8fdf\u4e0a\u5347\u548c\u541e\u5410\u91cf\u4e0b\u964d\u3002Mellanox \u7684 Accelerated Switching And Packet Processing (ASAP\u00b2) \u6280\u672f\u53ef\u4ee5\u5c06 OVS \u76f8\u5173\u7684\u64cd\u4f5c\u5378\u8f7d\u5230\u786c\u4ef6\u7f51\u5361\u5185\u7684 eSwitch \u4e0a\u6267\u884c\u3002\u8be5\u6280\u672f\u53ef\u4ee5\u5728\u65e0\u9700\u5bf9 OVS \u63a7\u5236\u5e73\u9762\u8fdb\u884c\u4fee\u6539\u7684\u60c5\u51b5\u4e0b\uff0c\u7f29\u77ed\u6570\u636e\u8def\u5f84\uff0c\u907f\u514d\u5bf9\u4e3b\u673a CPU \u8d44\u6e90\u7684\u4f7f\u7528\uff0c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u663e\u8457\u63d0\u5347\u541e\u5410\u91cf\u3002","title":"Mellanox \u7f51\u5361 Offload \u652f\u6301"},{"location":"advance/offload-mellanox/#_1","text":"Mellanox CX5/CX6/CX7/BlueField \u7b49\u652f\u6301 ASAP\u00b2 \u7684\u786c\u4ef6\u7f51\u5361\u3002 CentOS 8 Stream \u6216\u4e0a\u6e38 Linux 5.7 \u4ee5\u4e0a\u5185\u6838\u652f\u6301\u3002 \u7531\u4e8e\u5f53\u524d\u7f51\u5361\u4e0d\u652f\u6301 dp_hash \u548c hash \u64cd\u4f5c\u5378\u8f7d\uff0c\u9700\u5173\u95ed OVN LB \u529f\u80fd\u3002 \u4e3a\u4e86\u652f\u6301\u5378\u8f7d\u6a21\u5f0f\uff0c\u7f51\u5361\u4e0d\u80fd\u505a bond\u3002","title":"\u524d\u7f6e\u6761\u4ef6"},{"location":"advance/offload-mellanox/#sr-iov-device-plugin","text":"Mellanox \u7f51\u5361\u652f\u6301\u4e24\u79cd\u914d\u7f6e offload \u7684\u65b9\u5f0f\uff0c\u4e00\u79cd\u624b\u52a8\u914d\u7f6e\u7f51\u5361 SR-IOV \u548c Device Plugin\uff0c\u53e6\u4e00\u79cd\u901a\u8fc7 sriov-network-operator \u8fdb\u884c\u81ea\u52a8\u914d\u7f6e\u3002","title":"\u914d\u7f6e SR-IOV \u548c Device Plugin"},{"location":"advance/offload-mellanox/#sr-iov-device-plugin_1","text":"\u67e5\u8be2\u7f51\u5361\u7684\u8bbe\u5907 ID\uff0c\u4e0b\u9762\u7684\u4f8b\u5b50\u4e2d\u4e3a 42:00.0 \uff1a # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] \u6839\u636e\u8bbe\u5907 ID \u627e\u5230\u5bf9\u5e94\u7f51\u5361\uff1a # ls -l /sys/class/net/ | grep 42:00.0 lrwxrwxrwx. 1 root root 0 Jul 22 23 :16 p4p1 -> ../../devices/pci0000:40/0000:40:02.0/0000:42:00.0/net/p4p1 \u68c0\u67e5\u53ef\u7528 VF \u6570\u91cf\uff1a # cat /sys/class/net/p4p1/device/sriov_totalvfs 8 \u521b\u5efa VF\uff0c\u603b\u6570\u4e0d\u8981\u8d85\u8fc7\u4e0a\u9762\u67e5\u8be2\u51fa\u7684\u6570\u91cf\uff1a # echo '4' > /sys/class/net/p4p1/device/sriov_numvfs # ip link show p4p1 10 : p4p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:59:9f:c1:ec:12 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 1 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 2 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 3 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off # ip link set p4p1 up \u627e\u5230\u4e0a\u8ff0 VF \u5bf9\u5e94\u7684\u8bbe\u5907 ID\uff1a # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] \u5c06 VF \u4ece\u9a71\u52a8\u4e2d\u89e3\u7ed1\uff1a echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind \u5f00\u542f eSwitch \u6a21\u5f0f\uff0c\u5e76\u8bbe\u7f6e\u786c\u4ef6\u5378\u8f7d\uff1a devlink dev eswitch set pci/0000:42:00.0 mode switchdev ethtool -K enp66s0f0 hw-tc-offload on \u91cd\u65b0\u7ed1\u5b9a\u9a71\u52a8\uff0c\u5b8c\u6210 VF \u8bbe\u7f6e\uff1a echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/bind NetworkManager \u7684\u4e00\u4e9b\u884c\u4e3a\u53ef\u80fd\u4f1a\u5bfc\u81f4\u9a71\u52a8\u5f02\u5e38\uff0c\u5982\u679c\u5378\u8f7d\u51fa\u73b0\u95ee\u9898\u5efa\u8bae\u5173\u95ed NetworkManager \u518d\u8fdb\u884c\u5c1d\u8bd5\uff1a systemctl stop NetworkManager systemctl disable NetworkManager \u7531\u4e8e\u6bcf\u4e2a\u673a\u5668\u7684 VF \u6570\u91cf\u4f18\u5148\uff0c\u6bcf\u4e2a\u4f7f\u7528\u52a0\u901f\u7684 Pod \u4f1a\u5360\u7528 VF \u8d44\u6e90\uff0c\u6211\u4eec\u9700\u8981\u4f7f\u7528 SR-IOV Device Plugin \u7ba1\u7406\u76f8\u5e94\u8d44\u6e90\uff0c\u4f7f\u5f97\u8c03\u5ea6\u5668\u77e5\u9053\u5982\u4f55\u6839\u636e \u8d44\u6e90\u8fdb\u884c\u8c03\u5ea6\u3002 \u521b\u5efa SR-IOV \u76f8\u5173 Configmap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"mellanox.com\", \"resourceName\": \"cx5_sriov_switchdev\", \"selectors\": { \"vendors\": [\"15b3\"], \"devices\": [\"1018\"], \"drivers\": [\"mlx5_core\"] } } ] } \u53c2\u8003 SR-IOV \u6587\u6863 \u8fdb\u884c\u90e8\u7f72: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml \u68c0\u67e5 SR-IOV \u8d44\u6e90\u662f\u5426\u5df2\u7ecf\u6ce8\u518c\u5230 Kubernetes Node \u4e2d\uff1a kubectl describe node kube-ovn-01 | grep mellanox mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev 0 0","title":"\u624b\u52a8\u914d\u7f6e SR-IOV \u548c Device Plugin"},{"location":"advance/offload-mellanox/#sriov-network-operator-sr-iov-device-plugin","text":"\u5b89\u88c5 node-feature-discovery \u81ea\u52a8\u68c0\u6d4b\u786c\u4ef6\u7684\u529f\u80fd\u548c\u7cfb\u7edf\u914d\u7f6e: kubectl apply -k https://github.com/kubernetes-sigs/node-feature-discovery/deployment/overlays/default?ref = v0.11.3 \u6216\u8005\u901a\u8fc7\u4e0b\u9762\u7684\u547d\u4ee4\uff0c\u624b\u52a8\u7ed9\u6709 offload \u80fd\u529b\u7684\u7f51\u5361\u589e\u52a0 annotation: kubectl label nodes [ offloadNicNode ] feature.node.kubernetes.io/network-sriov.capable = true \u514b\u9686\u4ee3\u7801\u4ed3\u5e93\u5e76\u5b89\u88c5 Operator\uff1a git clone --depth = 1 https://github.com/kubeovn/sriov-network-operator.git kubectl apply -k sriov-network-operator/deploy \u68c0\u67e5 Operator \u7ec4\u4ef6\u662f\u5426\u5de5\u4f5c\u6b63\u5e38\uff1a # kubectl get -n kube-system all | grep sriov NAME READY STATUS RESTARTS AGE pod/sriov-network-config-daemon-bf9nt 1 /1 Running 0 8s pod/sriov-network-operator-54d7545f65-296gb 1 /1 Running 0 10s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/sriov-network-config-daemon 1 1 1 1 1 beta.kubernetes.io/os = linux,feature.node.kubernetes.io/network-sriov.capable = true 8s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/sriov-network-operator 1 /1 1 1 10s NAME DESIRED CURRENT READY AGE replicaset.apps/sriov-network-operator-54d7545f65 1 1 1 10s \u68c0\u67e5 SriovNetworkNodeState \uff0c\u4e0b\u9762\u4ee5 node1 \u8282\u70b9\u4e3a\u4f8b\uff0c\u8be5\u8282\u70b9\u4e0a\u6709\u4e24\u4e2a Mellanox \u7f51\u5361\uff1a # kubectl get sriovnetworknodestates.sriovnetwork.openshift.io -n kube-system node1 -o yaml apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodeState spec: ... status: interfaces: - deviceID: \"1017\" driver: mlx5_core mtu: 1500 pciAddress: \"0000:5f:00.0\" totalvfs: 8 vendor: \"15b3\" linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb name: ens41f0np0 - deviceID: \"1017\" driver: mlx5_core mtu: 1500 pciAddress: \"0000:5f:00.1\" totalvfs: 8 vendor: \"15b3\" linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb name: ens41f1np1 \u521b\u5efa SriovNetworkNodePolicy \u8d44\u6e90\uff0c\u5e76\u901a\u8fc7 nicSelector \u9009\u62e9\u8981\u7ba1\u7406\u7684\u7f51\u5361\uff1a apiVersion : sriovnetwork.openshift.io/v1 kind : SriovNetworkNodePolicy metadata : name : policy namespace : kube-system spec : nodeSelector : feature.node.kubernetes.io/network-sriov.capable : \"true\" eSwitchMode : switchdev numVfs : 3 nicSelector : pfNames : - ens41f0np0 - ens41f1np1 resourceName : cx_sriov_switchdev \u518d\u6b21\u68c0\u67e5 SriovNetworkNodeState \u7684 status \u5b57\u6bb5\uff1a # kubectl get sriovnetworknodestates.sriovnetwork.openshift.io -n kube-system node1 -o yaml ... spec: interfaces: - eSwitchMode: switchdev name: ens41f0np0 numVfs: 3 pciAddress: 0000 :5f:00.0 vfGroups: - policyName: policy vfRange: 0 -2 resourceName: cx_sriov_switchdev - eSwitchMode: switchdev name: ens41f1np1 numVfs: 3 pciAddress: 0000 :5f:00.1 vfGroups: - policyName: policy vfRange: 0 -2 resourceName: cx_sriov_switchdev status: interfaces - Vfs: - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.2 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.3 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.4 vendor: \"15b3\" deviceID: \"1017\" driver: mlx5_core linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:ab mtu: 1500 name: ens41f0np0 numVfs: 3 pciAddress: 0000 :5f:00.0 totalvfs: 3 vendor: \"15b3\" - Vfs: - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.5 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.6 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.7 vendor: \"15b3\" deviceID: \"1017\" driver: mlx5_core linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb mtu: 1500 name: ens41f1np1 numVfs: 3 pciAddress: 0000 :5f:00.1 totalvfs: 3 vendor: \"15b3\" \u68c0\u67e5 VF \u7684\u72b6\u6001\uff1a # lspci -nn | grep ConnectX 5f:00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 5f:00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 5f:00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.6 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.7 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] \u68c0\u67e5 PF \u5de5\u4f5c\u6a21\u5f0f\uff1a # cat /sys/class/net/ens41f0np0/compat/devlink/mode switchdev","title":"\u4f7f\u7528 sriov-network-operator \u914d\u7f6e SR-IOV \u548c Device Plugin"},{"location":"advance/offload-mellanox/#multus-cni","text":"SR-IOV Device Plugin \u8c03\u5ea6\u65f6\u83b7\u5f97\u7684\u8bbe\u5907 ID \u9700\u8981\u901a\u8fc7 Multus-CNI \u4f20\u9012\u7ed9 Kube-OVN\uff0c\u56e0\u6b64\u9700\u8981\u914d\u7f6e Multus-CNI \u914d\u5408\u5b8c\u6210\u76f8\u5173\u4efb\u52a1\u3002 \u53c2\u8003 Multus-CNI \u6587\u6863 \u8fdb\u884c\u90e8\u7f72\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml \u521b\u5efa NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : mellanox.com/cx5_sriov_switchdev spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : \u683c\u5f0f\u4e3a\u5f53\u524d NetworkAttachmentDefinition \u7684 {name}.{namespace}.ovn\u3002","title":"\u5b89\u88c5 Multus-CNI"},{"location":"advance/offload-mellanox/#kube-ovn","text":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u76f8\u5173\u53c2\u6570\uff0c IFACE \u9700\u8981\u4e3a\u7269\u7406\u7f51\u5361\u540d\uff0c\u8be5\u7f51\u5361\u9700\u8981\u6709\u53ef\u8def\u7531 IP\uff1a ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" \u5b89\u88c5 Kube-OVN\uff1a bash install.sh","title":"Kube-OVN \u4e2d\u5f00\u542f\u5378\u8f7d\u6a21\u5f0f"},{"location":"advance/offload-mellanox/#vf-pod","text":"\u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b yaml \u683c\u5f0f\u521b\u5efa\u4f7f\u7528 VF \u8fdb\u884c\u7f51\u7edc\u5378\u8f7d\u52a0\u901f\u7684 Pod: apiVersion : v1 kind : Pod metadata : name : nginx annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : mellanox.com/cx5_sriov_switchdev : '1' limits : mellanox.com/cx5_sriov_switchdev : '1' v1.multus-cni.io/default-network : \u4e3a\u4e0a\u4e00\u6b65\u9aa4\u4e2d NetworkAttachmentDefinition \u7684 {namespace}/{name}\u3002 \u53ef\u901a\u8fc7\u5728 Pod \u8fd0\u884c\u8282\u70b9\u7684 ovs-ovn \u5bb9\u5668\u4e2d\u8fd0\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u89c2\u5bdf\u5378\u8f7d\u662f\u5426\u6210\u529f\uff1a # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h \u5982\u679c\u6709 offloaded:yes, dp:tc \u5185\u5bb9\u8bc1\u660e\u5378\u8f7d\u6210\u529f\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u521b\u5efa\u4f7f\u7528 VF \u7f51\u5361\u7684 Pod"},{"location":"advance/overlay-with-route/","text":"Overlay \u4e0b\u8def\u7531\u65b9\u5f0f\u7f51\u7edc\u6253\u901a \u00b6 \u5728\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u7f51\u7edc\u73af\u5883\u4e0d\u652f\u6301 Underlay \u6a21\u5f0f\uff0c\u4f46\u662f\u4f9d\u7136\u9700\u8981 Pod \u80fd\u548c\u5916\u90e8\u8bbe\u65bd\u76f4\u63a5\u901a\u8fc7 IP \u8fdb\u884c\u4e92\u8bbf\uff0c \u8fd9\u65f6\u5019\u53ef\u4ee5\u4f7f\u7528\u8def\u7531\u65b9\u5f0f\u5c06\u5bb9\u5668\u7f51\u7edc\u548c\u5916\u90e8\u8054\u901a\u3002 \u8def\u7531\u6a21\u5f0f\u53ea\u652f\u6301\u9ed8\u8ba4 VPC \u4e0b\u7684 Overlay \u7f51\u7edc\u548c\u5916\u90e8\u6253\u901a\uff0c\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\uff0cPod IP \u4f1a\u76f4\u63a5\u8fdb\u5165\u5e95\u5c42\u7f51\u7edc\uff0c\u5e95\u5c42\u7f51\u7edc\u9700\u8981\u653e\u5f00\u5173\u4e8e\u6e90\u5730\u5740\u548c\u76ee\u5730\u5740\u7684 IP \u68c0\u67e5\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u6b64\u6a21\u5f0f\u4e0b\uff0c\u4e3b\u673a\u9700\u8981\u5f00\u653e ip_forward \u3002 \u68c0\u67e5\u4e3b\u673a iptables \u89c4\u5219\u4e2d\u662f\u5426\u5728 forward \u94fe\u4e2d\u662f\u5426\u6709 Drop \u89c4\u5219\uff0c\u9700\u8981\u653e\u884c\u5bb9\u5668\u76f8\u5173\u6d41\u91cf\u3002 \u7531\u4e8e\u53ef\u80fd\u5b58\u5728\u975e\u5bf9\u79f0\u8def\u7531\u7684\u60c5\u51b5\uff0c\u4e3b\u673a\u9700\u653e\u884c ct \u72b6\u6001\u4e3a INVALID \u7684\u6570\u636e\u5305\u3002 \u8bbe\u7f6e\u6b65\u9aa4 \u00b6 \u5bf9\u4e8e\u9700\u8981\u5bf9\u5916\u76f4\u63a5\u8def\u7531\u7684\u5b50\u7f51\uff0c\u9700\u8981\u5c06\u5b50\u7f51\u7684 natOutgoing \u8bbe\u7f6e\u4e3a false \uff0c\u5173\u95ed nat \u6620\u5c04\uff0c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : false \u6b64\u65f6\uff0cPod \u7684\u6570\u636e\u5305\u53ef\u4ee5\u901a\u8fc7\u4e3b\u673a\u8def\u7531\u5230\u8fbe\u5bf9\u7aef\u8282\u70b9\uff0c\u4f46\u662f\u5bf9\u7aef\u8282\u70b9\u8fd8\u4e0d\u77e5\u9053\u56de\u7a0b\u6570\u636e\u5305\u5e94\u8be5\u53d1\u9001\u5230\u54ea\u91cc\uff0c\u9700\u8981\u6dfb\u52a0\u56de\u7a0b\u8def\u7531\u3002 \u5982\u679c\u5bf9\u7aef\u4e3b\u673a\u548c\u5bb9\u5668\u6240\u5728\u5bbf\u4e3b\u673a\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u7f51\u7edc\uff0c\u6211\u4eec\u53ef\u4ee5\u76f4\u63a5\u5728\u5bf9\u7aef\u4e3b\u673a\u6dfb\u52a0\u9759\u6001\u8def\u7531\u5c06\u5bb9\u5668\u7f51\u7edc\u7684\u4e0b\u4e00\u8df3\u6307\u5411 Kubernetes \u96c6\u7fa4\u5185\u7684\u4efb\u610f\u4e00\u53f0\u673a\u5668\u3002 ip route add 10 .166.0.0/16 via 192 .168.2.10 dev eth0 10.166.0.0/16 \u4e3a\u5bb9\u5668\u5b50\u7f51\u7f51\u6bb5\uff0c 192.168.2.10 \u4e3a Kubernetes \u96c6\u7fa4\u5185\u4efb\u610f\u4e00\u4e2a\u8282\u70b9\u3002 \u82e5\u5bf9\u7aef\u4e3b\u673a\u548c\u5bb9\u5668\u6240\u5728\u5bbf\u4e3b\u673a\u4e0d\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u7f51\u7edc\uff0c\u5219\u9700\u8981\u5728\u8def\u7531\u5668\u4e0a\u914d\u7f6e\u76f8\u5e94\u7684\u89c4\u5219\uff0c\u901a\u8fc7\u8def\u7531\u5668\u8fdb\u884c\u6253\u901a\u3002 \u5728\u4e00\u4e9b\u865a\u62df\u5316\u73af\u5883\u4e2d\uff0c\u865a\u62df\u7f51\u7edc\u4f1a\u5c06\u975e\u5bf9\u79f0\u6d41\u91cf\u8bc6\u522b\u4e3a\u975e\u6cd5\u6d41\u91cf\u5e76\u4e22\u5f03\u3002 \u6b64\u65f6\u9700\u8981\u5c06 Subnet \u7684 gatewayType \u8c03\u6574\u4e3a centralized \uff0c\u5e76\u5728\u8def\u7531\u8bbe\u7f6e\u65f6\u5c06\u4e0b\u4e00\u8df3\u8bbe\u7f6e\u4e3a gatewayNode \u8282\u70b9\u7684 IP\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1\" natOutgoing : false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Overlay \u4e0b\u8def\u7531\u65b9\u5f0f\u7f51\u7edc\u6253\u901a"},{"location":"advance/overlay-with-route/#overlay","text":"\u5728\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u7f51\u7edc\u73af\u5883\u4e0d\u652f\u6301 Underlay \u6a21\u5f0f\uff0c\u4f46\u662f\u4f9d\u7136\u9700\u8981 Pod \u80fd\u548c\u5916\u90e8\u8bbe\u65bd\u76f4\u63a5\u901a\u8fc7 IP \u8fdb\u884c\u4e92\u8bbf\uff0c \u8fd9\u65f6\u5019\u53ef\u4ee5\u4f7f\u7528\u8def\u7531\u65b9\u5f0f\u5c06\u5bb9\u5668\u7f51\u7edc\u548c\u5916\u90e8\u8054\u901a\u3002 \u8def\u7531\u6a21\u5f0f\u53ea\u652f\u6301\u9ed8\u8ba4 VPC \u4e0b\u7684 Overlay \u7f51\u7edc\u548c\u5916\u90e8\u6253\u901a\uff0c\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\uff0cPod IP \u4f1a\u76f4\u63a5\u8fdb\u5165\u5e95\u5c42\u7f51\u7edc\uff0c\u5e95\u5c42\u7f51\u7edc\u9700\u8981\u653e\u5f00\u5173\u4e8e\u6e90\u5730\u5740\u548c\u76ee\u5730\u5740\u7684 IP \u68c0\u67e5\u3002","title":"Overlay \u4e0b\u8def\u7531\u65b9\u5f0f\u7f51\u7edc\u6253\u901a"},{"location":"advance/overlay-with-route/#_1","text":"\u6b64\u6a21\u5f0f\u4e0b\uff0c\u4e3b\u673a\u9700\u8981\u5f00\u653e ip_forward \u3002 \u68c0\u67e5\u4e3b\u673a iptables \u89c4\u5219\u4e2d\u662f\u5426\u5728 forward \u94fe\u4e2d\u662f\u5426\u6709 Drop \u89c4\u5219\uff0c\u9700\u8981\u653e\u884c\u5bb9\u5668\u76f8\u5173\u6d41\u91cf\u3002 \u7531\u4e8e\u53ef\u80fd\u5b58\u5728\u975e\u5bf9\u79f0\u8def\u7531\u7684\u60c5\u51b5\uff0c\u4e3b\u673a\u9700\u653e\u884c ct \u72b6\u6001\u4e3a INVALID \u7684\u6570\u636e\u5305\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/overlay-with-route/#_2","text":"\u5bf9\u4e8e\u9700\u8981\u5bf9\u5916\u76f4\u63a5\u8def\u7531\u7684\u5b50\u7f51\uff0c\u9700\u8981\u5c06\u5b50\u7f51\u7684 natOutgoing \u8bbe\u7f6e\u4e3a false \uff0c\u5173\u95ed nat \u6620\u5c04\uff0c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : false \u6b64\u65f6\uff0cPod \u7684\u6570\u636e\u5305\u53ef\u4ee5\u901a\u8fc7\u4e3b\u673a\u8def\u7531\u5230\u8fbe\u5bf9\u7aef\u8282\u70b9\uff0c\u4f46\u662f\u5bf9\u7aef\u8282\u70b9\u8fd8\u4e0d\u77e5\u9053\u56de\u7a0b\u6570\u636e\u5305\u5e94\u8be5\u53d1\u9001\u5230\u54ea\u91cc\uff0c\u9700\u8981\u6dfb\u52a0\u56de\u7a0b\u8def\u7531\u3002 \u5982\u679c\u5bf9\u7aef\u4e3b\u673a\u548c\u5bb9\u5668\u6240\u5728\u5bbf\u4e3b\u673a\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u7f51\u7edc\uff0c\u6211\u4eec\u53ef\u4ee5\u76f4\u63a5\u5728\u5bf9\u7aef\u4e3b\u673a\u6dfb\u52a0\u9759\u6001\u8def\u7531\u5c06\u5bb9\u5668\u7f51\u7edc\u7684\u4e0b\u4e00\u8df3\u6307\u5411 Kubernetes \u96c6\u7fa4\u5185\u7684\u4efb\u610f\u4e00\u53f0\u673a\u5668\u3002 ip route add 10 .166.0.0/16 via 192 .168.2.10 dev eth0 10.166.0.0/16 \u4e3a\u5bb9\u5668\u5b50\u7f51\u7f51\u6bb5\uff0c 192.168.2.10 \u4e3a Kubernetes \u96c6\u7fa4\u5185\u4efb\u610f\u4e00\u4e2a\u8282\u70b9\u3002 \u82e5\u5bf9\u7aef\u4e3b\u673a\u548c\u5bb9\u5668\u6240\u5728\u5bbf\u4e3b\u673a\u4e0d\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u7f51\u7edc\uff0c\u5219\u9700\u8981\u5728\u8def\u7531\u5668\u4e0a\u914d\u7f6e\u76f8\u5e94\u7684\u89c4\u5219\uff0c\u901a\u8fc7\u8def\u7531\u5668\u8fdb\u884c\u6253\u901a\u3002 \u5728\u4e00\u4e9b\u865a\u62df\u5316\u73af\u5883\u4e2d\uff0c\u865a\u62df\u7f51\u7edc\u4f1a\u5c06\u975e\u5bf9\u79f0\u6d41\u91cf\u8bc6\u522b\u4e3a\u975e\u6cd5\u6d41\u91cf\u5e76\u4e22\u5f03\u3002 \u6b64\u65f6\u9700\u8981\u5c06 Subnet \u7684 gatewayType \u8c03\u6574\u4e3a centralized \uff0c\u5e76\u5728\u8def\u7531\u8bbe\u7f6e\u65f6\u5c06\u4e0b\u4e00\u8df3\u8bbe\u7f6e\u4e3a gatewayNode \u8282\u70b9\u7684 IP\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1\" natOutgoing : false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8bbe\u7f6e\u6b65\u9aa4"},{"location":"advance/ovn-eip-fip-snat/","text":"OVN EIP FIP SNAT DNAT \u652f\u6301 \u00b6 graph LR pod-->vpc1-subnet-->vpc1-->snat-->lrp-->external-subnet-->gw-node-external-nic Pod \u57fa\u4e8e SNAT \u51fa\u516c\u7f51\u7684\u5927\u81f4\u6d41\u7a0b\uff0c\u6700\u540e\u662f\u7ecf\u8fc7\u7f51\u5173\u8282\u70b9\u7684\u516c\u7f51\u7f51\u5361\u3002 graph LR pod-->vpc1-subnet-->vpc1-->fip-->lrp-->external-subnet-->local-node-external-nic Pod \u57fa\u4e8e FIP \u51fa\u516c\u7f51\u7684\u5927\u81f4\u6d41\u7a0b\uff0c\u6700\u540e\u53ef\u4ee5\u57fa\u4e8e\u672c\u5730\u8282\u70b9\u7684\u516c\u7f51\u7f51\u5361\u51fa\u516c\u7f51\u3002 \u8be5\u529f\u80fd\u6240\u652f\u6301\u7684 CRD \u5728\u4f7f\u7528\u4e0a\u5c06\u548c iptable nat gw \u516c\u7f51\u65b9\u6848\u4fdd\u6301\u57fa\u672c\u4e00\u81f4\u3002 ovn eip: \u7528\u4e8e\u516c\u7f51 ip \u5360\u4f4d\uff0c\u4ece underlay provider network vlan subnet \u4e2d\u5206\u914d ovn fip\uff1a \u4e00\u5bf9\u4e00 dnat snat\uff0c\u4e3a vpc \u5185\u7684 ip \u6216\u8005 vip \u63d0\u4f9b\u516c\u7f51\u76f4\u63a5\u8bbf\u95ee\u80fd\u529b ovn snat\uff1a\u6574\u4e2a\u5b50\u7f51\u6216\u8005\u5355\u4e2a vpc \u5185 ip \u53ef\u4ee5\u57fa\u4e8e snat \u8bbf\u95ee\u516c\u7f51 ovn dnat\uff1a\u57fa\u4e8e router lb \u5b9e\u73b0, \u57fa\u4e8e\u516c\u7f51 ip + \u7aef\u53e3 \u76f4\u63a5\u8bbf\u95ee vpc \u5185\u7684 \u4e00\u7ec4 endpoints 1. \u90e8\u7f72 \u00b6 \u76ee\u524d\u5141\u8bb8\u6240\u6709\uff08\u9ed8\u8ba4\u4ee5\u53ca\u81ea\u5b9a\u4e49\uff09vpc \u4f7f\u7528\u540c\u4e00\u4e2a provider vlan subnet \u8d44\u6e90\uff0c\u540c\u65f6\u517c\u5bb9 \u9ed8\u8ba4 VPC EIP/SNAT \u7684\u573a\u666f\u3002 \u7c7b\u4f3c neutron ovn\uff0c\u670d\u52a1\u542f\u52a8\u914d\u7f6e\u4e2d\u9700\u8981\u6307\u5b9a provider network \u76f8\u5173\u7684\u914d\u7f6e\uff0c\u4e0b\u8ff0\u7684\u542f\u52a8\u53c2\u6570\u4e5f\u662f\u4e3a\u4e86\u517c\u5bb9 VPC EIP/SNAT \u7684\u5b9e\u73b0\u3002 \u90e8\u7f72\u9636\u6bb5\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\uff0c\u53ef\u80fd\u9700\u8981\u6307\u5b9a\u9ed8\u8ba4\u516c\u7f51\u903b\u8f91\u4ea4\u6362\u673a\u3002 \u5982\u679c\u5b9e\u9645\u4f7f\u7528\u4e2d\u6ca1\u6709 vlan\uff08\u4f7f\u7528 vlan 0\uff09\uff0c\u90a3\u4e48\u4e0b\u8ff0\u542f\u52a8\u53c2\u6570\u65e0\u9700\u914d\u7f6e\u3002 # \u90e8\u7f72\u7684\u65f6\u5019\u4f60\u9700\u8981\u53c2\u8003\u4ee5\u4e0a\u573a\u666f\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\uff0c\u6309\u9700\u6307\u5b9a\u5982\u4e0b\u53c2\u6570 # 1. kube-ovn-controller \u542f\u52a8\u53c2\u6570\u9700\u8981\u914d\u7f6e\uff1a - --external-gateway-vlanid = 204 - --external-gateway-switch = external204 # 2. kube-ovn-cni \u542f\u52a8\u53c2\u6570\u9700\u8981\u914d\u7f6e: - --external-gateway-switch = external204 ### \u4ee5\u4e0a\u914d\u7f6e\u90fd\u548c\u4e0b\u9762\u7684\u516c\u7f51\u7f51\u7edc\u914d\u7f6e vlan id \u548c\u8d44\u6e90\u540d\u4fdd\u6301\u4e00\u81f4\uff0c\u76ee\u524d\u4ec5\u652f\u6301\u6307\u5b9a\u4e00\u4e2a underlay \u516c\u7f51\u4f5c\u4e3a\u9ed8\u8ba4\u5916\u90e8\u516c\u7f51\u3002 \u8be5\u914d\u7f6e\u9879\u7684\u8bbe\u8ba1\u548c\u4f7f\u7528\u4e3b\u8981\u8003\u8651\u4e86\u5982\u4e0b\u56e0\u7d20\uff1a \u57fa\u4e8e\u8be5\u914d\u7f6e\u9879\u53ef\u4ee5\u5bf9\u63a5\u5230 provider network\uff0cvlan\uff0csubnet \u7684\u8d44\u6e90\u3002 \u57fa\u4e8e\u8be5\u914d\u7f6e\u9879\u53ef\u4ee5\u5c06\u9ed8\u8ba4 vpc enable_eip_snat \u529f\u80fd\u5bf9\u63a5\u5230\u5df2\u6709\u7684 vlan\uff0csubnet \u8d44\u6e90\uff0c\u540c\u65f6\u652f\u6301\u516c\u7f51 ip \u7684 ipam\u3002 \u5982\u679c\u4ec5\u4f7f\u7528\u9ed8\u8ba4 vpc \u7684 enable_eip_snat \u6a21\u5f0f, \u4e14\u4ec5\u4f7f\u7528\u65e7\u7684\u57fa\u4e8e pod annotaion \u7684 eip fip snat\uff0c\u90a3\u4e48\u8fd9\u4e2a\u914d\u7f6e\u65e0\u9700\u914d\u7f6e\u3002 \u57fa\u4e8e\u8be5\u914d\u7f6e\u53ef\u4ee5\u4e0d\u4f7f\u7528\u9ed8\u8ba4 vpc enable_eip_snat \u6d41\u7a0b\uff0c\u4ec5\u901a\u8fc7\u5bf9\u5e94\u5230 vlan\uff0csubnet \u6d41\u7a0b\uff0c\u53ef\u4ee5\u517c\u5bb9\u4ec5\u81ea\u5b9a\u4e49 vpc \u4f7f\u7528 eip snat \u7684\u4f7f\u7528\u573a\u666f\u3002 1.1 \u51c6\u5907 underlay \u516c\u7f51\u7f51\u7edc \u00b6 # \u51c6\u5907 provider-network\uff0c vlan\uff0c subnet # cat 01-provider-network.yaml apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: external204 spec: defaultInterface: vlan # cat 02-vlan.yaml apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan204 spec: id: 204 provider: external204 # cat 03-vlan-subnet.yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: external204 spec: protocol: IPv4 cidrBlock: 10 .5.204.0/24 gateway: 10 .5.204.254 vlan: vlan204 excludeIps: - 10 .5.204.1..10.5.204.100 1.2 \u9ed8\u8ba4 vpc \u542f\u7528 eip_snat \u00b6 # \u542f\u7528\u9ed8\u8ba4 vpc \u548c\u4e0a\u8ff0 underlay \u516c\u7f51 provider subnet \u4e92\u8054 cat 00 -centralized-external-gw-no-ip.yaml apiVersion: v1 kind: ConfigMap metadata: name: ovn-external-gw-config namespace: kube-system data: enable-external-gw: \"true\" external-gw-nodes: \"pc-node-1,pc-node-2,pc-node-3\" type: \"centralized\" external-gw-nic: \"vlan\" # \u7528\u4e8e\u63a5\u5165 ovs \u516c\u7f51\u7f51\u6865\u7684\u7f51\u5361 external-gw-addr: \"10.5.204.254/24\" # underlay \u7269\u7406\u7f51\u5173\u7684 ip \u76ee\u524d\u8be5\u529f\u80fd\u5df2\u652f\u6301\u53ef\u4ee5\u4e0d\u6307\u5b9a lrp ip \u548c mac\uff0c\u5df2\u652f\u6301\u81ea\u52a8\u83b7\u53d6\uff0c\u521b\u5efa lrp \u7c7b\u578b\u7684 ovn eip \u8d44\u6e90\u3002 \u5982\u679c\u6307\u5b9a\u4e86\uff0c\u5219\u76f8\u5f53\u4e8e\u6307\u5b9a ip \u521b\u5efa lrp \u7c7b\u578b\u7684 ovn-eip\u3002 \u5f53\u7136\u4e5f\u53ef\u4ee5\u63d0\u524d\u624b\u52a8\u521b\u5efa lrp \u7c7b\u578b\u7684 ovn eip\u3002 1.3 \u81ea\u5b9a\u4e49 vpc \u542f\u7528 eip snat fip \u529f\u80fd \u00b6 # cat 00-ns.yml apiVersion: v1 kind: Namespace metadata: name: vpc1 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true # vpc \u542f\u7528 enableExternal \u4f1a\u81ea\u52a8\u521b\u5efa lrp \u5173\u8054\u5230\u4e0a\u8ff0\u6307\u5b9a\u7684\u516c\u7f51 # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 # \u8fd9\u91cc\u5b50\u7f51\u548c\u4e4b\u524d\u4f7f\u7528\u5b50\u7f51\u4e00\u6837\uff0c\u8be5\u529f\u80fd\u5728 subnet \u4e0a\u6ca1\u6709\u65b0\u589e\u5c5e\u6027\uff0c\u6ca1\u6709\u4efb\u4f55\u53d8\u66f4 \u4ee5\u4e0a\u6a21\u677f\u5e94\u7528\u540e\uff0c\u5e94\u8be5\u53ef\u4ee5\u770b\u5230\u5982\u4e0b\u8d44\u6e90\u5b58\u5728 # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # \u76ee\u524d\u8be5\u8def\u7531\u5df2\u81ea\u52a8\u7ef4\u62a4 2. ovn-eip \u00b6 \u8be5\u529f\u80fd\u548c iptables-eip \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4\uff0covn-eip \u76ee\u524d\u6709\u4e09\u79cd type nat: \u7528\u4e8e ovn dnat\uff0cfip, snat, \u8fd9\u4e9b nat \u7c7b\u578b\u4f1a\u8bb0\u5f55\u5728 status \u4e2d lrp: \u7528\u4e8e vpc \u548c\u516c\u7f51\u76f8\u8fde\u7684\u8d44\u6e90 node-ext-gw: \u7528\u4e8e ovn \u57fa\u4e8e bfd \u7684 ecmp \u9759\u6001\u8def\u7531\u573a\u666f --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: fip # \u52a8\u6001\u5206\u914d\u4e00\u4e2a eip \u8d44\u6e90\uff0c\u8be5\u8d44\u6e90\u9884\u7559\u7528\u4e8e fip \u573a\u666f 2.1 ovn-fip \u4e3a pod \u7ed1\u5b9a\u4e00\u4e2a fip \u00b6 # k get po -o wide -n vpc1 vpc-1-busybox01 NAME READY STATUS RESTARTS AGE IP NODE vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 # k get ip vpc-1-busybox01.vpc1 NAME V4IP V6IP MAC NODE SUBNET vpc-1-busybox01.vpc1 192 .168.0.2 00 :00:00:0A:DD:27 pc-node-2 vpc1-subnet1 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: fip --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: ovnEip: eip-static ipName: vpc-1-busybox01.vpc1 # \u6ce8\u610f\u8fd9\u91cc\u662f ip crd \u7684\u540d\u5b57\uff0c\u5177\u6709\u552f\u4e00\u6027 # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 # k get ofip eip-static NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 [ root@pc-node-1 03 -cust-vpc ] # ping 10.5.204.101 PING 10 .5.204.101 ( 10 .5.204.101 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.101: icmp_seq = 2 ttl = 62 time = 1 .21 ms 64 bytes from 10 .5.204.101: icmp_seq = 3 ttl = 62 time = 0 .624 ms 64 bytes from 10 .5.204.101: icmp_seq = 4 ttl = 62 time = 0 .368 ms ^C --- 10 .5.204.101 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3049ms rtt min/avg/max/mdev = 0 .368/0.734/1.210/0.352 ms [ root@pc-node-1 03 -cust-vpc ] # # \u53ef\u4ee5\u770b\u5230\u5728 node ping \u9ed8\u8ba4 vpc \u4e0b\u7684 pod \u7684\u516c\u7f51 ip \u662f\u80fd\u901a\u7684 # \u8be5\u516c\u7f51 ip \u80fd\u901a\u7684\u5173\u952e\u8d44\u6e90\u4e3b\u8981\u5305\u62ec\u4ee5\u4e0b\u90e8\u5206 # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 813523e7-c68c-408f-bd8c-cba30cb2e4f4 external ip: \"10.5.204.101\" logical ip: \"192.168.0.2\" type: \"dnat_and_snat\" 2.2 ovn-fip \u4e3a vip \u7ed1\u5b9a\u4e00\u4e2a fip \u00b6 \u4e3a\u4e86\u4fbf\u4e8e\u4e00\u4e9b vip \u573a\u666f\u7684\u4f7f\u7528\uff0c\u6bd4\u5982 kubevirt \u865a\u62df\u673a\u5185\u90e8\u6211\u53ef\u80fd\u4f1a\u4f7f\u7528\u4e00\u4e9b vip \u63d0\u4f9b\u7ed9 keepalived\uff0ckube-vip \u7b49\u573a\u666f\u6765\u4f7f\u7528\uff0c\u540c\u65f6\u652f\u6301\u516c\u7f51\u8bbf\u95ee\u3002 \u90a3\u4e48\u53ef\u4ee5\u57fa\u4e8e fip \u7ed1\u5b9a vpc \u5185\u90e8\u7684 vip \u7684\u65b9\u5f0f\u6765\u63d0\u4f9b vip \u7684\u516c\u7f51\u80fd\u529b\u3002 # \u5148\u521b\u5efa vip\uff0ceip\uff0c\u518d\u5c06 eip \u7ed1\u5b9a\u5230 vip # cat vip.yaml apiVersion: kubeovn.io/v1 kind: Vip metadata: name: test-fip-vip spec: subnet: vpc1-subnet1 # cat 04-fip.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: externalSubnet: external204 type: fip --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: ovnEip: eip-for-vip ipType: vip # \u9ed8\u8ba4\u60c5\u51b5\u4e0b fip \u662f\u9762\u5411 pod ip \u7684\uff0c\u8fd9\u91cc\u9700\u8981\u6807\u6ce8\u6307\u5b9a\u5bf9\u63a5\u5230 vip \u8d44\u6e90 ipName: test-fip-vip # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip [ root@pc-node-1 fip-vip ] # ping 10.5.204.106 PING 10 .5.204.106 ( 10 .5.204.106 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.106: icmp_seq = 1 ttl = 62 time = 0 .694 ms 64 bytes from 10 .5.204.106: icmp_seq = 2 ttl = 62 time = 0 .436 ms # \u5728 node \u4e0a\u662f ping \u5f97\u901a\u7684 # pod \u5185\u90e8\u7684 ip \u4f7f\u7528\u65b9\u5f0f\u5927\u81f4\u5c31\u662f\u5982\u4e0b\u8fd9\u79cd\u60c5\u51b5 [ root@pc-node-1 fip-vip ] # k -n vpc1 exec -it vpc-1-busybox03 -- bash [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1568 : eth0@if1569: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:56:40:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.5/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 192 .168.0.3/24 scope global secondary eth0 # \u53ef\u4ee5\u770b\u5230 vip \u7684\u914d\u7f6e valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe56:40e5/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox03 / ] # tcpdump -i eth0 host 192.168.0.3 -netvv tcpdump: listening on eth0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:ed:8e:c7 > 00 :00:00:56:40:e5, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 44830 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.32.51 > 192 .168.0.3: ICMP echo request, id 177 , seq 1 , length 64 00 :00:00:56:40:e5 > 00 :00:00:ed:8e:c7, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 64 , id 43962 , offset 0 , flags [ none ] , proto ICMP ( 1 ) , length 84 ) 192 .168.0.3 > 10 .5.32.51: ICMP echo reply, id 177 , seq 1 , length 64 # pod \u5185\u90e8\u53ef\u4ee5\u6293\u5230 fip \u76f8\u5173\u7684 icmp \u5305 3. ovn-snat \u00b6 3.1 ovn-snat \u5bf9\u5e94\u4e00\u4e2a subnet \u7684 cidr \u00b6 \u8be5\u529f\u80fd\u548c iptables-snat \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4 # cat 03-subnet-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: externalSubnet: external204 type: snat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: ovnEip: snat-for-subnet-in-vpc vpcSubnet: vpc1-subnet1 # eip \u5bf9\u5e94\u6574\u4e2a\u7f51\u6bb5 3.2 ovn-fip \u5bf9\u5e94\u5230\u4e00\u4e2a pod ip \u00b6 \u8be5\u529f\u80fd\u548c iptables-fip \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4 # cat 03-pod-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-pod-vpc-ip spec: externalSubnet: external204 type: snat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat01 spec: ovnEip: snat-for-pod-vpc-ip ipName: vpc-1-busybox02.vpc1 # eip \u5bf9\u5e94\u5355\u4e2a pod ip \u4ee5\u4e0a\u8d44\u6e90\u521b\u5efa\u540e\uff0c\u53ef\u4ee5\u770b\u5230 snat \u516c\u7f51\u529f\u80fd\u4f9d\u8d56\u7684\u5982\u4e0b\u8d44\u6e90\u3002 # kubectl ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" nat da77a11f-c523-439c-b1d1-72c664196a0f external ip: \"10.5.204.116\" logical ip: \"192.168.0.4\" type: \"snat\" [ root@pc-node-1 03 -cust-vpc ] # k get po -A -o wide | grep busy vpc1 vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 <none> <none> vpc1 vpc-1-busybox02 1 /1 Running 0 17h 192 .168.0.4 pc-node-1 <none> <none> vpc1 vpc-1-busybox03 1 /1 Running 0 17h 192 .168.0.5 pc-node-1 <none> <none> vpc1 vpc-1-busybox04 1 /1 Running 0 17h 192 .168.0.6 pc-node-3 <none> <none> vpc1 vpc-1-busybox05 1 /1 Running 0 17h 192 .168.0.7 pc-node-1 <none> <none> # k exec -it -n vpc1 vpc-1-busybox04 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17095 : eth0@if17096: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:76:94:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.6/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe76:9455/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox04 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 114 time = 22 .2 ms 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 21 .8 ms [ root@pc-node-1 03 -cust-vpc ] # k exec -it -n vpc1 vpc-1-busybox02 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1566 : eth0@if1567: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:0b:e9:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.4/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe0b:e9d0/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox02 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 22 .7 ms 64 bytes from 223 .5.5.5: icmp_seq = 3 ttl = 114 time = 22 .6 ms 64 bytes from 223 .5.5.5: icmp_seq = 4 ttl = 114 time = 22 .1 ms ^C --- 223 .5.5.5 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3064ms rtt min/avg/max/mdev = 22 .126/22.518/22.741/0.278 ms # \u53ef\u4ee5\u770b\u5230\u4e24\u4e2a pod \u53ef\u4ee5\u5206\u522b\u57fa\u4e8e\u8fd9\u4e24\u79cd snat \u8d44\u6e90\u4e0a\u5916\u7f51 4. ovn-dnat \u00b6 4.1 ovn-dnat \u4e3a pod \u7ed1\u5b9a\u4e00\u4e2a dnat \u00b6 kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : externalSubnet : underlay --- kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ovnEip : eip-dnat ipName : vpc-1-busybox01.vpc1 # \u6ce8\u610f\u8fd9\u91cc\u662f pod ip crd \u7684\u540d\u5b57\uff0c\u5177\u6709\u552f\u4e00\u6027 protocol : tcp internalPort : \"22\" externalPort : \"22\" OvnDnatRule \u7684\u914d\u7f6e\u4e0e IptablesDnatRule \u7c7b\u4f3c # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.3 22 22 vpc-1-busybox01.vpc1 true 4.2 ovn-dnat \u4e3a vip \u7ed1\u5b9a\u4e00\u4e2a dnat \u00b6 kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ipType : vip # \u9ed8\u8ba4\u60c5\u51b5\u4e0b dnat \u662f\u9762\u5411 pod ip \u7684\uff0c\u8fd9\u91cc\u9700\u8981\u6807\u6ce8\u6307\u5b9a\u5bf9\u63a5\u5230 vip \u8d44\u6e90 ovnEip : eip-dnat ipName : test-dnat-vip protocol : tcp internalPort : \"22\" externalPort : \"22\" OvnDnatRule \u7684\u914d\u7f6e\u4e0e IptablesDnatRule \u7c7b\u4f3c # kubectl get vip test-dnat-vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY test-dnat-vip 192 .168.0.4 00 :00:00:D0:C0:B5 vpc1-subnet1 true # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat eip-dnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.4 22 22 test-dnat-vip true \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN EIP FIP SNAT \u652f\u6301"},{"location":"advance/ovn-eip-fip-snat/#ovn-eip-fip-snat-dnat","text":"graph LR pod-->vpc1-subnet-->vpc1-->snat-->lrp-->external-subnet-->gw-node-external-nic Pod \u57fa\u4e8e SNAT \u51fa\u516c\u7f51\u7684\u5927\u81f4\u6d41\u7a0b\uff0c\u6700\u540e\u662f\u7ecf\u8fc7\u7f51\u5173\u8282\u70b9\u7684\u516c\u7f51\u7f51\u5361\u3002 graph LR pod-->vpc1-subnet-->vpc1-->fip-->lrp-->external-subnet-->local-node-external-nic Pod \u57fa\u4e8e FIP \u51fa\u516c\u7f51\u7684\u5927\u81f4\u6d41\u7a0b\uff0c\u6700\u540e\u53ef\u4ee5\u57fa\u4e8e\u672c\u5730\u8282\u70b9\u7684\u516c\u7f51\u7f51\u5361\u51fa\u516c\u7f51\u3002 \u8be5\u529f\u80fd\u6240\u652f\u6301\u7684 CRD \u5728\u4f7f\u7528\u4e0a\u5c06\u548c iptable nat gw \u516c\u7f51\u65b9\u6848\u4fdd\u6301\u57fa\u672c\u4e00\u81f4\u3002 ovn eip: \u7528\u4e8e\u516c\u7f51 ip \u5360\u4f4d\uff0c\u4ece underlay provider network vlan subnet \u4e2d\u5206\u914d ovn fip\uff1a \u4e00\u5bf9\u4e00 dnat snat\uff0c\u4e3a vpc \u5185\u7684 ip \u6216\u8005 vip \u63d0\u4f9b\u516c\u7f51\u76f4\u63a5\u8bbf\u95ee\u80fd\u529b ovn snat\uff1a\u6574\u4e2a\u5b50\u7f51\u6216\u8005\u5355\u4e2a vpc \u5185 ip \u53ef\u4ee5\u57fa\u4e8e snat \u8bbf\u95ee\u516c\u7f51 ovn dnat\uff1a\u57fa\u4e8e router lb \u5b9e\u73b0, \u57fa\u4e8e\u516c\u7f51 ip + \u7aef\u53e3 \u76f4\u63a5\u8bbf\u95ee vpc \u5185\u7684 \u4e00\u7ec4 endpoints","title":"OVN EIP FIP SNAT DNAT \u652f\u6301"},{"location":"advance/ovn-eip-fip-snat/#1","text":"\u76ee\u524d\u5141\u8bb8\u6240\u6709\uff08\u9ed8\u8ba4\u4ee5\u53ca\u81ea\u5b9a\u4e49\uff09vpc \u4f7f\u7528\u540c\u4e00\u4e2a provider vlan subnet \u8d44\u6e90\uff0c\u540c\u65f6\u517c\u5bb9 \u9ed8\u8ba4 VPC EIP/SNAT \u7684\u573a\u666f\u3002 \u7c7b\u4f3c neutron ovn\uff0c\u670d\u52a1\u542f\u52a8\u914d\u7f6e\u4e2d\u9700\u8981\u6307\u5b9a provider network \u76f8\u5173\u7684\u914d\u7f6e\uff0c\u4e0b\u8ff0\u7684\u542f\u52a8\u53c2\u6570\u4e5f\u662f\u4e3a\u4e86\u517c\u5bb9 VPC EIP/SNAT \u7684\u5b9e\u73b0\u3002 \u90e8\u7f72\u9636\u6bb5\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\uff0c\u53ef\u80fd\u9700\u8981\u6307\u5b9a\u9ed8\u8ba4\u516c\u7f51\u903b\u8f91\u4ea4\u6362\u673a\u3002 \u5982\u679c\u5b9e\u9645\u4f7f\u7528\u4e2d\u6ca1\u6709 vlan\uff08\u4f7f\u7528 vlan 0\uff09\uff0c\u90a3\u4e48\u4e0b\u8ff0\u542f\u52a8\u53c2\u6570\u65e0\u9700\u914d\u7f6e\u3002 # \u90e8\u7f72\u7684\u65f6\u5019\u4f60\u9700\u8981\u53c2\u8003\u4ee5\u4e0a\u573a\u666f\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\uff0c\u6309\u9700\u6307\u5b9a\u5982\u4e0b\u53c2\u6570 # 1. kube-ovn-controller \u542f\u52a8\u53c2\u6570\u9700\u8981\u914d\u7f6e\uff1a - --external-gateway-vlanid = 204 - --external-gateway-switch = external204 # 2. kube-ovn-cni \u542f\u52a8\u53c2\u6570\u9700\u8981\u914d\u7f6e: - --external-gateway-switch = external204 ### \u4ee5\u4e0a\u914d\u7f6e\u90fd\u548c\u4e0b\u9762\u7684\u516c\u7f51\u7f51\u7edc\u914d\u7f6e vlan id \u548c\u8d44\u6e90\u540d\u4fdd\u6301\u4e00\u81f4\uff0c\u76ee\u524d\u4ec5\u652f\u6301\u6307\u5b9a\u4e00\u4e2a underlay \u516c\u7f51\u4f5c\u4e3a\u9ed8\u8ba4\u5916\u90e8\u516c\u7f51\u3002 \u8be5\u914d\u7f6e\u9879\u7684\u8bbe\u8ba1\u548c\u4f7f\u7528\u4e3b\u8981\u8003\u8651\u4e86\u5982\u4e0b\u56e0\u7d20\uff1a \u57fa\u4e8e\u8be5\u914d\u7f6e\u9879\u53ef\u4ee5\u5bf9\u63a5\u5230 provider network\uff0cvlan\uff0csubnet \u7684\u8d44\u6e90\u3002 \u57fa\u4e8e\u8be5\u914d\u7f6e\u9879\u53ef\u4ee5\u5c06\u9ed8\u8ba4 vpc enable_eip_snat \u529f\u80fd\u5bf9\u63a5\u5230\u5df2\u6709\u7684 vlan\uff0csubnet \u8d44\u6e90\uff0c\u540c\u65f6\u652f\u6301\u516c\u7f51 ip \u7684 ipam\u3002 \u5982\u679c\u4ec5\u4f7f\u7528\u9ed8\u8ba4 vpc \u7684 enable_eip_snat \u6a21\u5f0f, \u4e14\u4ec5\u4f7f\u7528\u65e7\u7684\u57fa\u4e8e pod annotaion \u7684 eip fip snat\uff0c\u90a3\u4e48\u8fd9\u4e2a\u914d\u7f6e\u65e0\u9700\u914d\u7f6e\u3002 \u57fa\u4e8e\u8be5\u914d\u7f6e\u53ef\u4ee5\u4e0d\u4f7f\u7528\u9ed8\u8ba4 vpc enable_eip_snat \u6d41\u7a0b\uff0c\u4ec5\u901a\u8fc7\u5bf9\u5e94\u5230 vlan\uff0csubnet \u6d41\u7a0b\uff0c\u53ef\u4ee5\u517c\u5bb9\u4ec5\u81ea\u5b9a\u4e49 vpc \u4f7f\u7528 eip snat \u7684\u4f7f\u7528\u573a\u666f\u3002","title":"1. \u90e8\u7f72"},{"location":"advance/ovn-eip-fip-snat/#11-underlay","text":"# \u51c6\u5907 provider-network\uff0c vlan\uff0c subnet # cat 01-provider-network.yaml apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: external204 spec: defaultInterface: vlan # cat 02-vlan.yaml apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan204 spec: id: 204 provider: external204 # cat 03-vlan-subnet.yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: external204 spec: protocol: IPv4 cidrBlock: 10 .5.204.0/24 gateway: 10 .5.204.254 vlan: vlan204 excludeIps: - 10 .5.204.1..10.5.204.100","title":"1.1 \u51c6\u5907 underlay \u516c\u7f51\u7f51\u7edc"},{"location":"advance/ovn-eip-fip-snat/#12-vpc-eip_snat","text":"# \u542f\u7528\u9ed8\u8ba4 vpc \u548c\u4e0a\u8ff0 underlay \u516c\u7f51 provider subnet \u4e92\u8054 cat 00 -centralized-external-gw-no-ip.yaml apiVersion: v1 kind: ConfigMap metadata: name: ovn-external-gw-config namespace: kube-system data: enable-external-gw: \"true\" external-gw-nodes: \"pc-node-1,pc-node-2,pc-node-3\" type: \"centralized\" external-gw-nic: \"vlan\" # \u7528\u4e8e\u63a5\u5165 ovs \u516c\u7f51\u7f51\u6865\u7684\u7f51\u5361 external-gw-addr: \"10.5.204.254/24\" # underlay \u7269\u7406\u7f51\u5173\u7684 ip \u76ee\u524d\u8be5\u529f\u80fd\u5df2\u652f\u6301\u53ef\u4ee5\u4e0d\u6307\u5b9a lrp ip \u548c mac\uff0c\u5df2\u652f\u6301\u81ea\u52a8\u83b7\u53d6\uff0c\u521b\u5efa lrp \u7c7b\u578b\u7684 ovn eip \u8d44\u6e90\u3002 \u5982\u679c\u6307\u5b9a\u4e86\uff0c\u5219\u76f8\u5f53\u4e8e\u6307\u5b9a ip \u521b\u5efa lrp \u7c7b\u578b\u7684 ovn-eip\u3002 \u5f53\u7136\u4e5f\u53ef\u4ee5\u63d0\u524d\u624b\u52a8\u521b\u5efa lrp \u7c7b\u578b\u7684 ovn eip\u3002","title":"1.2 \u9ed8\u8ba4 vpc \u542f\u7528 eip_snat"},{"location":"advance/ovn-eip-fip-snat/#13-vpc-eip-snat-fip","text":"# cat 00-ns.yml apiVersion: v1 kind: Namespace metadata: name: vpc1 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true # vpc \u542f\u7528 enableExternal \u4f1a\u81ea\u52a8\u521b\u5efa lrp \u5173\u8054\u5230\u4e0a\u8ff0\u6307\u5b9a\u7684\u516c\u7f51 # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 # \u8fd9\u91cc\u5b50\u7f51\u548c\u4e4b\u524d\u4f7f\u7528\u5b50\u7f51\u4e00\u6837\uff0c\u8be5\u529f\u80fd\u5728 subnet \u4e0a\u6ca1\u6709\u65b0\u589e\u5c5e\u6027\uff0c\u6ca1\u6709\u4efb\u4f55\u53d8\u66f4 \u4ee5\u4e0a\u6a21\u677f\u5e94\u7528\u540e\uff0c\u5e94\u8be5\u53ef\u4ee5\u770b\u5230\u5982\u4e0b\u8d44\u6e90\u5b58\u5728 # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # \u76ee\u524d\u8be5\u8def\u7531\u5df2\u81ea\u52a8\u7ef4\u62a4","title":"1.3 \u81ea\u5b9a\u4e49 vpc \u542f\u7528 eip snat fip \u529f\u80fd"},{"location":"advance/ovn-eip-fip-snat/#2-ovn-eip","text":"\u8be5\u529f\u80fd\u548c iptables-eip \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4\uff0covn-eip \u76ee\u524d\u6709\u4e09\u79cd type nat: \u7528\u4e8e ovn dnat\uff0cfip, snat, \u8fd9\u4e9b nat \u7c7b\u578b\u4f1a\u8bb0\u5f55\u5728 status \u4e2d lrp: \u7528\u4e8e vpc \u548c\u516c\u7f51\u76f8\u8fde\u7684\u8d44\u6e90 node-ext-gw: \u7528\u4e8e ovn \u57fa\u4e8e bfd \u7684 ecmp \u9759\u6001\u8def\u7531\u573a\u666f --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: fip # \u52a8\u6001\u5206\u914d\u4e00\u4e2a eip \u8d44\u6e90\uff0c\u8be5\u8d44\u6e90\u9884\u7559\u7528\u4e8e fip \u573a\u666f","title":"2. ovn-eip"},{"location":"advance/ovn-eip-fip-snat/#21-ovn-fip-pod-fip","text":"# k get po -o wide -n vpc1 vpc-1-busybox01 NAME READY STATUS RESTARTS AGE IP NODE vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 # k get ip vpc-1-busybox01.vpc1 NAME V4IP V6IP MAC NODE SUBNET vpc-1-busybox01.vpc1 192 .168.0.2 00 :00:00:0A:DD:27 pc-node-2 vpc1-subnet1 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: fip --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: ovnEip: eip-static ipName: vpc-1-busybox01.vpc1 # \u6ce8\u610f\u8fd9\u91cc\u662f ip crd \u7684\u540d\u5b57\uff0c\u5177\u6709\u552f\u4e00\u6027 # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 # k get ofip eip-static NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 [ root@pc-node-1 03 -cust-vpc ] # ping 10.5.204.101 PING 10 .5.204.101 ( 10 .5.204.101 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.101: icmp_seq = 2 ttl = 62 time = 1 .21 ms 64 bytes from 10 .5.204.101: icmp_seq = 3 ttl = 62 time = 0 .624 ms 64 bytes from 10 .5.204.101: icmp_seq = 4 ttl = 62 time = 0 .368 ms ^C --- 10 .5.204.101 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3049ms rtt min/avg/max/mdev = 0 .368/0.734/1.210/0.352 ms [ root@pc-node-1 03 -cust-vpc ] # # \u53ef\u4ee5\u770b\u5230\u5728 node ping \u9ed8\u8ba4 vpc \u4e0b\u7684 pod \u7684\u516c\u7f51 ip \u662f\u80fd\u901a\u7684 # \u8be5\u516c\u7f51 ip \u80fd\u901a\u7684\u5173\u952e\u8d44\u6e90\u4e3b\u8981\u5305\u62ec\u4ee5\u4e0b\u90e8\u5206 # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 813523e7-c68c-408f-bd8c-cba30cb2e4f4 external ip: \"10.5.204.101\" logical ip: \"192.168.0.2\" type: \"dnat_and_snat\"","title":"2.1 ovn-fip \u4e3a pod \u7ed1\u5b9a\u4e00\u4e2a fip"},{"location":"advance/ovn-eip-fip-snat/#22-ovn-fip-vip-fip","text":"\u4e3a\u4e86\u4fbf\u4e8e\u4e00\u4e9b vip \u573a\u666f\u7684\u4f7f\u7528\uff0c\u6bd4\u5982 kubevirt \u865a\u62df\u673a\u5185\u90e8\u6211\u53ef\u80fd\u4f1a\u4f7f\u7528\u4e00\u4e9b vip \u63d0\u4f9b\u7ed9 keepalived\uff0ckube-vip \u7b49\u573a\u666f\u6765\u4f7f\u7528\uff0c\u540c\u65f6\u652f\u6301\u516c\u7f51\u8bbf\u95ee\u3002 \u90a3\u4e48\u53ef\u4ee5\u57fa\u4e8e fip \u7ed1\u5b9a vpc \u5185\u90e8\u7684 vip \u7684\u65b9\u5f0f\u6765\u63d0\u4f9b vip \u7684\u516c\u7f51\u80fd\u529b\u3002 # \u5148\u521b\u5efa vip\uff0ceip\uff0c\u518d\u5c06 eip \u7ed1\u5b9a\u5230 vip # cat vip.yaml apiVersion: kubeovn.io/v1 kind: Vip metadata: name: test-fip-vip spec: subnet: vpc1-subnet1 # cat 04-fip.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: externalSubnet: external204 type: fip --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: ovnEip: eip-for-vip ipType: vip # \u9ed8\u8ba4\u60c5\u51b5\u4e0b fip \u662f\u9762\u5411 pod ip \u7684\uff0c\u8fd9\u91cc\u9700\u8981\u6807\u6ce8\u6307\u5b9a\u5bf9\u63a5\u5230 vip \u8d44\u6e90 ipName: test-fip-vip # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip [ root@pc-node-1 fip-vip ] # ping 10.5.204.106 PING 10 .5.204.106 ( 10 .5.204.106 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.106: icmp_seq = 1 ttl = 62 time = 0 .694 ms 64 bytes from 10 .5.204.106: icmp_seq = 2 ttl = 62 time = 0 .436 ms # \u5728 node \u4e0a\u662f ping \u5f97\u901a\u7684 # pod \u5185\u90e8\u7684 ip \u4f7f\u7528\u65b9\u5f0f\u5927\u81f4\u5c31\u662f\u5982\u4e0b\u8fd9\u79cd\u60c5\u51b5 [ root@pc-node-1 fip-vip ] # k -n vpc1 exec -it vpc-1-busybox03 -- bash [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1568 : eth0@if1569: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:56:40:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.5/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 192 .168.0.3/24 scope global secondary eth0 # \u53ef\u4ee5\u770b\u5230 vip \u7684\u914d\u7f6e valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe56:40e5/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox03 / ] # tcpdump -i eth0 host 192.168.0.3 -netvv tcpdump: listening on eth0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:ed:8e:c7 > 00 :00:00:56:40:e5, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 44830 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.32.51 > 192 .168.0.3: ICMP echo request, id 177 , seq 1 , length 64 00 :00:00:56:40:e5 > 00 :00:00:ed:8e:c7, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 64 , id 43962 , offset 0 , flags [ none ] , proto ICMP ( 1 ) , length 84 ) 192 .168.0.3 > 10 .5.32.51: ICMP echo reply, id 177 , seq 1 , length 64 # pod \u5185\u90e8\u53ef\u4ee5\u6293\u5230 fip \u76f8\u5173\u7684 icmp \u5305","title":"2.2 ovn-fip \u4e3a vip \u7ed1\u5b9a\u4e00\u4e2a fip"},{"location":"advance/ovn-eip-fip-snat/#3-ovn-snat","text":"","title":"3. ovn-snat"},{"location":"advance/ovn-eip-fip-snat/#31-ovn-snat-subnet-cidr","text":"\u8be5\u529f\u80fd\u548c iptables-snat \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4 # cat 03-subnet-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: externalSubnet: external204 type: snat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: ovnEip: snat-for-subnet-in-vpc vpcSubnet: vpc1-subnet1 # eip \u5bf9\u5e94\u6574\u4e2a\u7f51\u6bb5","title":"3.1 ovn-snat \u5bf9\u5e94\u4e00\u4e2a subnet \u7684 cidr"},{"location":"advance/ovn-eip-fip-snat/#32-ovn-fip-pod-ip","text":"\u8be5\u529f\u80fd\u548c iptables-fip \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4 # cat 03-pod-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-pod-vpc-ip spec: externalSubnet: external204 type: snat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat01 spec: ovnEip: snat-for-pod-vpc-ip ipName: vpc-1-busybox02.vpc1 # eip \u5bf9\u5e94\u5355\u4e2a pod ip \u4ee5\u4e0a\u8d44\u6e90\u521b\u5efa\u540e\uff0c\u53ef\u4ee5\u770b\u5230 snat \u516c\u7f51\u529f\u80fd\u4f9d\u8d56\u7684\u5982\u4e0b\u8d44\u6e90\u3002 # kubectl ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" nat da77a11f-c523-439c-b1d1-72c664196a0f external ip: \"10.5.204.116\" logical ip: \"192.168.0.4\" type: \"snat\" [ root@pc-node-1 03 -cust-vpc ] # k get po -A -o wide | grep busy vpc1 vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 <none> <none> vpc1 vpc-1-busybox02 1 /1 Running 0 17h 192 .168.0.4 pc-node-1 <none> <none> vpc1 vpc-1-busybox03 1 /1 Running 0 17h 192 .168.0.5 pc-node-1 <none> <none> vpc1 vpc-1-busybox04 1 /1 Running 0 17h 192 .168.0.6 pc-node-3 <none> <none> vpc1 vpc-1-busybox05 1 /1 Running 0 17h 192 .168.0.7 pc-node-1 <none> <none> # k exec -it -n vpc1 vpc-1-busybox04 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17095 : eth0@if17096: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:76:94:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.6/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe76:9455/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox04 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 114 time = 22 .2 ms 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 21 .8 ms [ root@pc-node-1 03 -cust-vpc ] # k exec -it -n vpc1 vpc-1-busybox02 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1566 : eth0@if1567: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:0b:e9:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.4/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe0b:e9d0/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox02 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 22 .7 ms 64 bytes from 223 .5.5.5: icmp_seq = 3 ttl = 114 time = 22 .6 ms 64 bytes from 223 .5.5.5: icmp_seq = 4 ttl = 114 time = 22 .1 ms ^C --- 223 .5.5.5 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3064ms rtt min/avg/max/mdev = 22 .126/22.518/22.741/0.278 ms # \u53ef\u4ee5\u770b\u5230\u4e24\u4e2a pod \u53ef\u4ee5\u5206\u522b\u57fa\u4e8e\u8fd9\u4e24\u79cd snat \u8d44\u6e90\u4e0a\u5916\u7f51","title":"3.2 ovn-fip \u5bf9\u5e94\u5230\u4e00\u4e2a pod ip"},{"location":"advance/ovn-eip-fip-snat/#4-ovn-dnat","text":"","title":"4. ovn-dnat"},{"location":"advance/ovn-eip-fip-snat/#41-ovn-dnat-pod-dnat","text":"kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : externalSubnet : underlay --- kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ovnEip : eip-dnat ipName : vpc-1-busybox01.vpc1 # \u6ce8\u610f\u8fd9\u91cc\u662f pod ip crd \u7684\u540d\u5b57\uff0c\u5177\u6709\u552f\u4e00\u6027 protocol : tcp internalPort : \"22\" externalPort : \"22\" OvnDnatRule \u7684\u914d\u7f6e\u4e0e IptablesDnatRule \u7c7b\u4f3c # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.3 22 22 vpc-1-busybox01.vpc1 true","title":"4.1 ovn-dnat \u4e3a pod \u7ed1\u5b9a\u4e00\u4e2a dnat"},{"location":"advance/ovn-eip-fip-snat/#42-ovn-dnat-vip-dnat","text":"kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ipType : vip # \u9ed8\u8ba4\u60c5\u51b5\u4e0b dnat \u662f\u9762\u5411 pod ip \u7684\uff0c\u8fd9\u91cc\u9700\u8981\u6807\u6ce8\u6307\u5b9a\u5bf9\u63a5\u5230 vip \u8d44\u6e90 ovnEip : eip-dnat ipName : test-dnat-vip protocol : tcp internalPort : \"22\" externalPort : \"22\" OvnDnatRule \u7684\u914d\u7f6e\u4e0e IptablesDnatRule \u7c7b\u4f3c # kubectl get vip test-dnat-vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY test-dnat-vip 192 .168.0.4 00 :00:00:D0:C0:B5 vpc1-subnet1 true # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat eip-dnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.4 22 22 test-dnat-vip true \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"4.2 ovn-dnat \u4e3a vip \u7ed1\u5b9a\u4e00\u4e2a dnat"},{"location":"advance/ovn-ipsec/","text":"\u4f7f\u7528 IPsec \u52a0\u5bc6\u8282\u70b9\u95f4\u901a\u4fe1 \u00b6 \u8be5\u529f\u80fd\u4ece v1.10.11 \u548c v1.11.4 \u540e\u5f00\u59cb\u652f\u6301\uff0ckernel \u7248\u672c\u81f3\u5c11\u662f 3.10.0 \u4ee5\u4e0a\u3002 \u542f\u52a8 IPsec \u00b6 \u4ece Kube-OVN \u6e90\u7801\u62f7\u8d1d\u811a\u672c ipsec.sh \uff0c\u6267\u884c\u547d\u4ee4\u5982\u4e0b\uff0c\u8be5\u811a\u672c\u4f1a\u8c03\u7528 ovs-pki \u751f\u6210\u548c\u5206\u914d\u52a0\u5bc6\u9700\u8981\u7684\u8bc1\u4e66\uff1a bash ipsec.sh init \u6267\u884c\u5b8c\u6bd5\u540e\uff0c\u8282\u70b9\u4e4b\u95f4\u4f1a\u534f\u5546\u4e00\u6bb5\u65f6\u95f4\u5efa\u7acb IPsec \u96a7\u9053\uff0c\u7ecf\u9a8c\u503c\u662f\u5341\u51e0\u79d2\u5230\u4e00\u5206\u949f\u4e4b\u95f4\uff0c\u53ef\u4ee5\u901a\u8fc7\u5982\u4e0b\u547d\u4ee4\u6765\u67e5\u770b IPsec \u72b6\u6001\uff1a # sh ipsec.sh status Pod { ovs-ovn-d7hdt } ipsec status... Interface name: ovn-a4718e-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.2 Remote IP: 172 .18.0.4 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem Remote cert: None Remote name: a4718e55-5b85-4f46-90e6-63527d080590 CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 2 CFM state: Disabled Kernel policies installed: src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 Kernel security associations installed: sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 IPsec connections that are active: Pod { ovs-ovn-fvbbj } ipsec status... Interface name: ovn-8aebd9-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.4 Remote IP: 172 .18.0.2 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem Local name: a4718e55-5b85-4f46-90e6-63527d080590 Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem Remote cert: None Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 1 CFM state: Disabled Kernel policies installed: src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 Kernel security associations installed: sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 IPsec connections that are active: \u5efa\u7acb\u5b8c\u6210\u540e\u53ef\u4ee5\u6293\u5305\u89c2\u5bdf\u62a5\u6587\u5df2\u7ecf\u88ab\u52a0\u5bc6\uff1a # tcpdump -i eth0 -nel esp 10 :01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP ( spi = 0xcc91322a,seq = 0x13d0 ) , length 156 10 :01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP ( spi = 0xc8df4221,seq = 0x1d37 ) , length 156 \u5f53\u6267\u884c\u5b8c\u811a\u672c\u540e\uff0c\u53ef\u4ee5\u901a\u8fc7\u6267\u884c\u547d\u4ee4\u5173\u95ed IPsec\uff1a # bash ipsec.sh stop \u6216\u8005\u6267\u884c\u547d\u4ee4\u518d\u6b21\u6253\u5f00\uff1a # bash ipsec.sh start \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN IPsec \u652f\u6301"},{"location":"advance/ovn-ipsec/#ipsec","text":"\u8be5\u529f\u80fd\u4ece v1.10.11 \u548c v1.11.4 \u540e\u5f00\u59cb\u652f\u6301\uff0ckernel \u7248\u672c\u81f3\u5c11\u662f 3.10.0 \u4ee5\u4e0a\u3002","title":"\u4f7f\u7528 IPsec \u52a0\u5bc6\u8282\u70b9\u95f4\u901a\u4fe1"},{"location":"advance/ovn-ipsec/#ipsec_1","text":"\u4ece Kube-OVN \u6e90\u7801\u62f7\u8d1d\u811a\u672c ipsec.sh \uff0c\u6267\u884c\u547d\u4ee4\u5982\u4e0b\uff0c\u8be5\u811a\u672c\u4f1a\u8c03\u7528 ovs-pki \u751f\u6210\u548c\u5206\u914d\u52a0\u5bc6\u9700\u8981\u7684\u8bc1\u4e66\uff1a bash ipsec.sh init \u6267\u884c\u5b8c\u6bd5\u540e\uff0c\u8282\u70b9\u4e4b\u95f4\u4f1a\u534f\u5546\u4e00\u6bb5\u65f6\u95f4\u5efa\u7acb IPsec \u96a7\u9053\uff0c\u7ecf\u9a8c\u503c\u662f\u5341\u51e0\u79d2\u5230\u4e00\u5206\u949f\u4e4b\u95f4\uff0c\u53ef\u4ee5\u901a\u8fc7\u5982\u4e0b\u547d\u4ee4\u6765\u67e5\u770b IPsec \u72b6\u6001\uff1a # sh ipsec.sh status Pod { ovs-ovn-d7hdt } ipsec status... Interface name: ovn-a4718e-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.2 Remote IP: 172 .18.0.4 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem Remote cert: None Remote name: a4718e55-5b85-4f46-90e6-63527d080590 CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 2 CFM state: Disabled Kernel policies installed: src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 Kernel security associations installed: sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 IPsec connections that are active: Pod { ovs-ovn-fvbbj } ipsec status... Interface name: ovn-8aebd9-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.4 Remote IP: 172 .18.0.2 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem Local name: a4718e55-5b85-4f46-90e6-63527d080590 Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem Remote cert: None Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 1 CFM state: Disabled Kernel policies installed: src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 Kernel security associations installed: sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 IPsec connections that are active: \u5efa\u7acb\u5b8c\u6210\u540e\u53ef\u4ee5\u6293\u5305\u89c2\u5bdf\u62a5\u6587\u5df2\u7ecf\u88ab\u52a0\u5bc6\uff1a # tcpdump -i eth0 -nel esp 10 :01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP ( spi = 0xcc91322a,seq = 0x13d0 ) , length 156 10 :01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP ( spi = 0xc8df4221,seq = 0x1d37 ) , length 156 \u5f53\u6267\u884c\u5b8c\u811a\u672c\u540e\uff0c\u53ef\u4ee5\u901a\u8fc7\u6267\u884c\u547d\u4ee4\u5173\u95ed IPsec\uff1a # bash ipsec.sh stop \u6216\u8005\u6267\u884c\u547d\u4ee4\u518d\u6b21\u6253\u5f00\uff1a # bash ipsec.sh start \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u542f\u52a8 IPsec"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/","text":"OVN SNAT \u57fa\u4e8e ECMP BFD \u9759\u6001\u8def\u7531\u7684 L3 HA \u652f\u6301 \u00b6 \u81ea\u5b9a\u4e49 vpc \u57fa\u4e8e ovn snat \u540e\u57fa\u4e8e ecmp \u9759\u6001\u8def\u7531\u54c8\u5e0c\u5230\u591a\u4e2a gw node ovnext0 \u7f51\u5361\u51fa\u516c\u7f51 \u652f\u6301\u57fa\u4e8e bfd \u7684\u9ad8\u53ef\u7528 \u4ec5\u652f\u6301 hash \u8d1f\u8f7d\u5747\u8861 graph LR pod-->vpc-subnet-->vpc-->snat-->ecmp-->external-subnet-->gw-node1-ovnext0--> node1-external-switch external-subnet-->gw-node2-ovnext0--> node2-external-switch external-subnet-->gw-node3-ovnext0--> node3-external-switch \u8be5\u529f\u80fd\u7684\u4f7f\u7528\u65b9\u5f0f\u548c ovn-eip-fip-snat.md \u57fa\u672c\u4e00\u81f4\uff0c\u4e00\u81f4\u7684\u90e8\u5206\u5305\u62ec install.sh \u7684\u90e8\u7f72\u90e8\u5206\uff0cprovider-network\uff0cvlan\uff0csubnet \u7684\u51c6\u5907\u90e8\u5206\u3002 \u81f3\u4e8e\u4e0d\u76f8\u540c\u7684\u90e8\u5206\uff0c\u4f1a\u5728\u4ee5\u4e0b\u90e8\u5206\u5177\u4f53\u9610\u8ff0\uff0c\u4e3b\u8981\u5305\u62ec node-ext-gw \u7c7b\u578b\u7684 ovn-eip \u7684\u521b\u5efa\uff0c\u4ee5\u53ca\u57fa\u4e8e vpc enable_bfd \u81ea\u52a8\u7ef4\u62a4 bfd \u4ee5\u53ca ecmp \u9759\u6001\u8def\u7531\u3002 1. \u90e8\u7f72 \u00b6 1.1 \u51c6\u5907 underlay \u516c\u7f51\u7f51\u7edc \u00b6 1.2 \u9ed8\u8ba4 vpc \u542f\u7528 eip_snat \u00b6 1.3 \u81ea\u5b9a\u4e49 vpc \u542f\u7528 eip snat fip \u529f\u80fd \u00b6 \u4ee5\u4e0a\u90e8\u5206\u548c ovn-eip-fip-snat.md \u5b8c\u5168\u4e00\u81f4\uff0c\u8fd9\u4e9b\u529f\u80fd\u9a8c\u8bc1\u901a\u8fc7\u540e\uff0c\u53ef\u4ee5\u76f4\u63a5\u57fa\u4e8e\u5982\u4e0b\u65b9\u5f0f\uff0c\u5c06 vpc \u5207\u6362\u5230\u57fa\u4e8e ecmp \u7684 bfd \u9759\u6001\u8def\u7531\uff0c\u5f53\u7136\u4e5f\u53ef\u4ee5\u5207\u56de\u3002 \u81ea\u5b9a\u4e49 vpc \u4f7f\u7528\u8be5\u529f\u80fd\u4e4b\u524d\uff0c\u9700\u8981\u5148\u63d0\u4f9b\u597d\u7f51\u5173\u8282\u70b9\uff0c\u81f3\u5c11\u9700\u8981\u63d0\u4f9b 2 \u4e2a\u4ee5\u4e0a\u7f51\u5173\u8282\u70b9\uff0c\u6ce8\u610f\u5f53\u524d\u5b9e\u73b0 ovn-eip \u7684\u540d\u5b57\u5fc5\u987b\u548c\u7f51\u5173\u8282\u70b9\u540d\u4fdd\u6301\u4e00\u81f4\uff0c\u76ee\u524d\u6ca1\u6709\u505a\u8be5\u8d44\u6e90\u7684\u81ea\u52a8\u5316\u7ef4\u62a4\u3002 # cat gw-node-eip.yaml --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-1 spec : externalSubnet : external204 type : node-ext-gw --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-2 spec : externalSubnet : external204 type : node-ext-gw --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-3 spec : externalSubnet : external204 type : node-ext-gw \u7531\u4e8e\u8fd9\u4e2a\u573a\u666f\u76ee\u524d\u8bbe\u8ba1\u4e0a\u662f\u4f9b vpc ecmp \u51fa\u516c\u7f51\u4f7f\u7528\uff0c\u6240\u4ee5\u4ee5\u4e0a\u5728\u6ca1\u6709 vpc \u542f\u7528 bfd \u7684\u65f6\u5019\uff0c\u5373\u4e0d\u5b58\u5728\u5e26\u6709 enable bfd \u6807\u7b7e\u7684 lrp \u7684 ovn eip \u7684\u65f6\u5019\uff0c\u7f51\u5173\u8282\u70b9\u4e0d\u4f1a\u89e6\u53d1\u521b\u5efa\u7f51\u5173\u7f51\u5361\uff0c\u4e5f\u65e0\u6cd5\u6210\u529f\u542f\u52a8\u5bf9\u7aef bfd \u4f1a\u8bdd\u7684\u76d1\u542c\u3002 2. \u81ea\u5b9a\u4e49 vpc \u542f\u7528 ecmp bfd L3 HA \u516c\u7f51\u529f\u80fd \u00b6 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true enableBfd: true # bfd \u5f00\u5173\u53ef\u4ee5\u968f\u610f\u5207\u6362\uff0c\u5f00\u8868\u793a\u542f\u7528 bfd ecmp \u8def\u7531 #enableBfd: false # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true # \u53ea\u9700\u5f00\u542f ecmp gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 \u4f7f\u7528\u4e0a\u7684\u6ce8\u610f\u70b9: \u81ea\u5b9a\u4e49 vpc \u4e0b\u7684 ecmp \u53ea\u7528\u9759\u6001 ecmp bfd \u8def\u7531\uff0cvpc enableBfd \u548c subnet enableEcmp \u540c\u65f6\u5f00\u542f\u7684\u60c5\u51b5\u4e0b\u624d\u4f1a\u751f\u6548\uff0c\u624d\u4f1a\u81ea\u52a8\u7ba1\u7406\u9759\u6001 ecmp bfd \u8def\u7531\u3002 \u4e0a\u8ff0\u914d\u7f6e\u5173\u95ed\u7684\u60c5\u51b5\u4e0b\uff0c\u4f1a\u81ea\u52a8\u5207\u56de\u5e38\u89c4\u9ed8\u8ba4\u9759\u6001\u8def\u7531\u3002 \u9ed8\u8ba4 vpc \u65e0\u6cd5\u4f7f\u7528\u8be5\u529f\u80fd\uff0c\u4ec5\u652f\u6301\u81ea\u5b9a\u4e49 vpc\uff0c\u9ed8\u8ba4 vpc \u6709\u66f4\u590d\u6742\u7684\u7b56\u7565\u8def\u7531\u4ee5\u53ca snat \u8bbe\u8ba1\u3002 \u81ea\u5b9a\u4e49 vpc \u7684 subnet \u7684 enableEcmp \u4ec5\u4f7f\u7528\u9759\u6001\u8def\u7531\uff0c\u7f51\u5173\u7c7b\u578b gatewayType \u6ca1\u6709\u4f5c\u7528\u3002 \u5f53\u5173\u95ed EnableExternal \u65f6\uff0cvpc \u5185\u65e0\u6cd5\u901a\u5916\u7f51\u3002 \u5f53\u5f00\u542f EnableExternal \u65f6\uff0c\u5173\u95ed EnableBfd \u65f6\uff0c\u4f1a\u57fa\u4e8e\u666e\u901a\u9ed8\u8ba4\u8def\u7531\u4e0a\u5916\u7f51\uff0c\u4e0d\u5177\u5907\u9ad8\u53ef\u7528\u3002 # \u4e0a\u8ff0\u6a21\u677f\u5e94\u7528\u540e ovn \u903b\u8f91\u5c42\u5e94\u8be5\u53ef\u4ee5\u770b\u5230\u5982\u4e0b\u8d44\u6e90 # \u67e5\u770b vpc # k get vpc NAME ENABLEEXTERNAL ENABLEBFD STANDBY SUBNETS NAMESPACES ovn-cluster true true [ \"external204\" , \"join\" , \"ovn-default\" ] vpc1 true true true [ \"vpc1-subnet1\" ] [ \"vpc1\" ] # \u9ed8\u8ba4 vpc \u672a\u652f\u6301 ENABLEBFD # \u81ea\u5b9a\u4e49 vpc \u5df2\u652f\u6301\u4e14\u5df2\u542f\u7528 # 1. \u521b\u5efa\u4e86 bfd \u4f1a\u8bdd # k ko nbctl list bfd _uuid : be7df545-2c4c-4751-878f-b3507987f050 detect_mult : 3 dst_ip : \"10.5.204.121\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : 684c4489-5b59-4693-8d8c-3beab93f8093 detect_mult : 3 dst_ip : \"10.5.204.109\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : f0f62077-2ae9-4e79-b4f8-a446ec6e784c detect_mult : 3 dst_ip : \"10.5.204.108\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up ### \u6ce8\u610f\u6240\u6709 status \u6b63\u5e38\u90fd\u5e94\u8be5\u662f up \u7684 # 2. \u521b\u5efa\u4e86\u57fa\u4e8e bfd \u7684\u9759\u6001\u8def\u7531 # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 192 .168.0.0/24 10 .5.204.108 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.109 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.121 src-ip ecmp ecmp-symmetric-reply bfd # 3. \u9759\u6001\u8def\u7531\u8be6\u60c5 # k ko nbctl find Logical_Router_Static_Route policy=src-ip options=ecmp_symmetric_reply=\"true\" _uuid : 3aacb384-d5ee-4b14-aebf-59e8c11717ba bfd : 684c4489-5b59-4693-8d8c-3beab93f8093 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.109\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 18bcc585-bc05-430b-925b-ef673c8e1aef bfd : f0f62077-2ae9-4e79-b4f8-a446ec6e784c external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.108\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 7d0a4e6b-cde0-4110-8176-fbaf19738498 bfd : be7df545-2c4c-4751-878f-b3507987f050 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.121\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" # \u540c\u65f6\u5728\u7f51\u5173\u8282\u70b9\u90fd\u5e94\u8be5\u5177\u5907\u4ee5\u4e0b\u8d44\u6e90 [ root@pc-node-1 ~ ] # ip netns exec ovnext bash ip a /usr/sbin/ip: /usr/sbin/ip: cannot execute binary file [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1541 : ovnext0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 00 :00:00:ab:bd:87 brd ff:ff:ff:ff:ff:ff inet 10 .5.204.108/24 brd 10 .5.204.255 scope global ovnext0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:feab:bd87/64 scope link valid_lft forever preferred_lft forever [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0 .0.0.0 10 .5.204.254 0 .0.0.0 UG 0 0 0 ovnext0 10 .5.204.0 0 .0.0.0 255 .255.255.0 U 0 0 0 ovnext0 ## \u6ce8\u610f\u4ee5\u4e0a\u5185\u5bb9\u548c\u4e00\u4e2a internal port unerlay \u516c\u7f51 pod \u5185\u90e8\u7684 ns \u5927\u81f4\u662f\u4e00\u81f4\u7684\uff0c\u8fd9\u91cc\u53ea\u662f\u5728\u7f51\u5173\u8282\u70b9\u4e0a\u5355\u72ec\u7ef4\u62a4\u4e86\u4e00\u4e2a ns [ root@pc-node-1 ~ ] # ip netns exec ovnext bfdd-control status There are 1 sessions: Session 1 id = 1 local = 10 .5.204.108 ( p ) remote = 10 .5.204.122 state = Up ## \u8fd9\u91cc\u5373\u662f lrp bfd \u4f1a\u8bdd\u7684\u53e6\u4e00\u7aef\uff0c\u4e5f\u662f lrp ecmp \u7684\u4e0b\u4e00\u8df3\u7684\u5176\u4e2d\u4e00\u4e2a [ root@pc-node-1 ~ ] # ip netns exec ovnext ping -c1 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 115 time = 21 .6 ms # \u5230\u516c\u7f51\u6ca1\u95ee\u9898 \u53ef\u4ee5\u5728\u67d0\u4e00\u4e2a\u7f51\u5173\u8282\u70b9\u7684 ovnext ns \u5185\u6293\u5230\u51fa\u53bb\u7684\u5305 # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-1 ~ ] # exit [ root@pc-node-1 ~ ] # ssh pc-node-2 Last login: Thu Feb 23 09 :21:08 2023 from 10 .5.32.51 [ root@pc-node-2 ~ ] # ip netns exec ovnext bash [ root@pc-node-2 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-2 ~ ] # exit [ root@pc-node-2 ~ ] # logout Connection to pc-node-2 closed. [ root@pc-node-1 ~ ] # ssh pc-node-3 Last login: Thu Feb 23 08 :32:41 2023 from 10 .5.32.51 [ root@pc-node-3 ~ ] # ip netns exec ovnext bash [ root@pc-node-3 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:2d:f8:ce > 00 :00:00:fd:b2:a4, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 63 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 00 :00:00:fd:b2:a4 > dc:ef:80:5a:44:1a, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [ root@pc-node-3 ~ ] # # \u53ef\u4ee5\u5728\u8be5\u8282\u70b9 down \u6389\u51fa\u53bb\u7684\u7f51\u5361\uff0c\u7136\u540e\u770b pod \u51fa\u53bb\u7684\u5305\u5728\u7f51\u7edc\u4e2d\u65ad\u4e2d\u4f1a\u51fa\u73b0\u51e0\u4e2a\u5305 # \u4e00\u822c\u90fd\u4f1a\u770b\u5230\u4e22 3 \u4e2a\u5305 3. \u5173\u95ed bfd \u6a21\u5f0f \u00b6 \u5728\u67d0\u4e9b\u573a\u666f\u4e0b\uff0c\u53ef\u80fd\u60f3\u76f4\u63a5\u4f7f\u7528\uff08\u96c6\u4e2d\u5f0f\uff09\u5355\u4e2a\u7f51\u5173\u76f4\u63a5\u51fa\u516c\u7f51\uff0c\u8fd9\u4e2a\u65f6\u5019\u548c\u9ed8\u8ba4 vpc enable_eip_snat \u7684\u4f7f\u7528\u6a21\u5f0f\u662f\u4e00\u81f4\u7684 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc2 spec: namespaces: - vpc2 enableExternal: true #enableBfd: true enableBfd: false ## \u5c06 bfd \u529f\u80fd\u76f4\u63a5\u7981\u7528\u5373\u53ef # k ko nbctl lr-route-list vpc2 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # \u5e94\u7528\u540e\u8def\u7531\u4f1a\u5207\u6362\u56de\u6b63\u5e38\u7684\u9ed8\u8ba4\u9759\u6001\u8def\u7531 # \u540c\u65f6 nbctl list bfd \u53ef\u4ee5\u770b\u5230 lrp \u5173\u8054\u7684 bfd \u4f1a\u8bdd\u5df2\u7ecf\u79fb\u9664 # \u800c\u4e14 ovnext ns \u4e2d\u7684\u5bf9\u7aef bfd \u4f1a\u8bdd\u4e5f\u81ea\u52a8\u79fb\u9664 # \u8be5\u5207\u6362\u8fc7\u7a0b\u4fdd\u6301 vpc subnet \u5185\u4fdd\u6301 ping \u672a\u770b\u5230(\u79d2\u7ea7)\u4e22\u5305 # \u518d\u5207\u6362\u56de\u53bb \u4e5f\u672a\u770b\u5230(\u79d2\u7ea7)\u4e22\u5305 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN SNAT \u57fa\u4e8e ECMP BFD \u9759\u6001\u8def\u7531\u7684 L3 HA \u652f\u6301"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#ovn-snat-ecmp-bfd-l3-ha","text":"\u81ea\u5b9a\u4e49 vpc \u57fa\u4e8e ovn snat \u540e\u57fa\u4e8e ecmp \u9759\u6001\u8def\u7531\u54c8\u5e0c\u5230\u591a\u4e2a gw node ovnext0 \u7f51\u5361\u51fa\u516c\u7f51 \u652f\u6301\u57fa\u4e8e bfd \u7684\u9ad8\u53ef\u7528 \u4ec5\u652f\u6301 hash \u8d1f\u8f7d\u5747\u8861 graph LR pod-->vpc-subnet-->vpc-->snat-->ecmp-->external-subnet-->gw-node1-ovnext0--> node1-external-switch external-subnet-->gw-node2-ovnext0--> node2-external-switch external-subnet-->gw-node3-ovnext0--> node3-external-switch \u8be5\u529f\u80fd\u7684\u4f7f\u7528\u65b9\u5f0f\u548c ovn-eip-fip-snat.md \u57fa\u672c\u4e00\u81f4\uff0c\u4e00\u81f4\u7684\u90e8\u5206\u5305\u62ec install.sh \u7684\u90e8\u7f72\u90e8\u5206\uff0cprovider-network\uff0cvlan\uff0csubnet \u7684\u51c6\u5907\u90e8\u5206\u3002 \u81f3\u4e8e\u4e0d\u76f8\u540c\u7684\u90e8\u5206\uff0c\u4f1a\u5728\u4ee5\u4e0b\u90e8\u5206\u5177\u4f53\u9610\u8ff0\uff0c\u4e3b\u8981\u5305\u62ec node-ext-gw \u7c7b\u578b\u7684 ovn-eip \u7684\u521b\u5efa\uff0c\u4ee5\u53ca\u57fa\u4e8e vpc enable_bfd \u81ea\u52a8\u7ef4\u62a4 bfd \u4ee5\u53ca ecmp \u9759\u6001\u8def\u7531\u3002","title":"OVN SNAT \u57fa\u4e8e ECMP BFD \u9759\u6001\u8def\u7531\u7684 L3 HA \u652f\u6301"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#1","text":"","title":"1. \u90e8\u7f72"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#11-underlay","text":"","title":"1.1 \u51c6\u5907 underlay \u516c\u7f51\u7f51\u7edc"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#12-vpc-eip_snat","text":"","title":"1.2 \u9ed8\u8ba4 vpc \u542f\u7528 eip_snat"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#13-vpc-eip-snat-fip","text":"\u4ee5\u4e0a\u90e8\u5206\u548c ovn-eip-fip-snat.md \u5b8c\u5168\u4e00\u81f4\uff0c\u8fd9\u4e9b\u529f\u80fd\u9a8c\u8bc1\u901a\u8fc7\u540e\uff0c\u53ef\u4ee5\u76f4\u63a5\u57fa\u4e8e\u5982\u4e0b\u65b9\u5f0f\uff0c\u5c06 vpc \u5207\u6362\u5230\u57fa\u4e8e ecmp \u7684 bfd \u9759\u6001\u8def\u7531\uff0c\u5f53\u7136\u4e5f\u53ef\u4ee5\u5207\u56de\u3002 \u81ea\u5b9a\u4e49 vpc \u4f7f\u7528\u8be5\u529f\u80fd\u4e4b\u524d\uff0c\u9700\u8981\u5148\u63d0\u4f9b\u597d\u7f51\u5173\u8282\u70b9\uff0c\u81f3\u5c11\u9700\u8981\u63d0\u4f9b 2 \u4e2a\u4ee5\u4e0a\u7f51\u5173\u8282\u70b9\uff0c\u6ce8\u610f\u5f53\u524d\u5b9e\u73b0 ovn-eip \u7684\u540d\u5b57\u5fc5\u987b\u548c\u7f51\u5173\u8282\u70b9\u540d\u4fdd\u6301\u4e00\u81f4\uff0c\u76ee\u524d\u6ca1\u6709\u505a\u8be5\u8d44\u6e90\u7684\u81ea\u52a8\u5316\u7ef4\u62a4\u3002 # cat gw-node-eip.yaml --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-1 spec : externalSubnet : external204 type : node-ext-gw --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-2 spec : externalSubnet : external204 type : node-ext-gw --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-3 spec : externalSubnet : external204 type : node-ext-gw \u7531\u4e8e\u8fd9\u4e2a\u573a\u666f\u76ee\u524d\u8bbe\u8ba1\u4e0a\u662f\u4f9b vpc ecmp \u51fa\u516c\u7f51\u4f7f\u7528\uff0c\u6240\u4ee5\u4ee5\u4e0a\u5728\u6ca1\u6709 vpc \u542f\u7528 bfd \u7684\u65f6\u5019\uff0c\u5373\u4e0d\u5b58\u5728\u5e26\u6709 enable bfd \u6807\u7b7e\u7684 lrp \u7684 ovn eip \u7684\u65f6\u5019\uff0c\u7f51\u5173\u8282\u70b9\u4e0d\u4f1a\u89e6\u53d1\u521b\u5efa\u7f51\u5173\u7f51\u5361\uff0c\u4e5f\u65e0\u6cd5\u6210\u529f\u542f\u52a8\u5bf9\u7aef bfd \u4f1a\u8bdd\u7684\u76d1\u542c\u3002","title":"1.3 \u81ea\u5b9a\u4e49 vpc \u542f\u7528 eip snat fip \u529f\u80fd"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#2-vpc-ecmp-bfd-l3-ha","text":"# cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true enableBfd: true # bfd \u5f00\u5173\u53ef\u4ee5\u968f\u610f\u5207\u6362\uff0c\u5f00\u8868\u793a\u542f\u7528 bfd ecmp \u8def\u7531 #enableBfd: false # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true # \u53ea\u9700\u5f00\u542f ecmp gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 \u4f7f\u7528\u4e0a\u7684\u6ce8\u610f\u70b9: \u81ea\u5b9a\u4e49 vpc \u4e0b\u7684 ecmp \u53ea\u7528\u9759\u6001 ecmp bfd \u8def\u7531\uff0cvpc enableBfd \u548c subnet enableEcmp \u540c\u65f6\u5f00\u542f\u7684\u60c5\u51b5\u4e0b\u624d\u4f1a\u751f\u6548\uff0c\u624d\u4f1a\u81ea\u52a8\u7ba1\u7406\u9759\u6001 ecmp bfd \u8def\u7531\u3002 \u4e0a\u8ff0\u914d\u7f6e\u5173\u95ed\u7684\u60c5\u51b5\u4e0b\uff0c\u4f1a\u81ea\u52a8\u5207\u56de\u5e38\u89c4\u9ed8\u8ba4\u9759\u6001\u8def\u7531\u3002 \u9ed8\u8ba4 vpc \u65e0\u6cd5\u4f7f\u7528\u8be5\u529f\u80fd\uff0c\u4ec5\u652f\u6301\u81ea\u5b9a\u4e49 vpc\uff0c\u9ed8\u8ba4 vpc \u6709\u66f4\u590d\u6742\u7684\u7b56\u7565\u8def\u7531\u4ee5\u53ca snat \u8bbe\u8ba1\u3002 \u81ea\u5b9a\u4e49 vpc \u7684 subnet \u7684 enableEcmp \u4ec5\u4f7f\u7528\u9759\u6001\u8def\u7531\uff0c\u7f51\u5173\u7c7b\u578b gatewayType \u6ca1\u6709\u4f5c\u7528\u3002 \u5f53\u5173\u95ed EnableExternal \u65f6\uff0cvpc \u5185\u65e0\u6cd5\u901a\u5916\u7f51\u3002 \u5f53\u5f00\u542f EnableExternal \u65f6\uff0c\u5173\u95ed EnableBfd \u65f6\uff0c\u4f1a\u57fa\u4e8e\u666e\u901a\u9ed8\u8ba4\u8def\u7531\u4e0a\u5916\u7f51\uff0c\u4e0d\u5177\u5907\u9ad8\u53ef\u7528\u3002 # \u4e0a\u8ff0\u6a21\u677f\u5e94\u7528\u540e ovn \u903b\u8f91\u5c42\u5e94\u8be5\u53ef\u4ee5\u770b\u5230\u5982\u4e0b\u8d44\u6e90 # \u67e5\u770b vpc # k get vpc NAME ENABLEEXTERNAL ENABLEBFD STANDBY SUBNETS NAMESPACES ovn-cluster true true [ \"external204\" , \"join\" , \"ovn-default\" ] vpc1 true true true [ \"vpc1-subnet1\" ] [ \"vpc1\" ] # \u9ed8\u8ba4 vpc \u672a\u652f\u6301 ENABLEBFD # \u81ea\u5b9a\u4e49 vpc \u5df2\u652f\u6301\u4e14\u5df2\u542f\u7528 # 1. \u521b\u5efa\u4e86 bfd \u4f1a\u8bdd # k ko nbctl list bfd _uuid : be7df545-2c4c-4751-878f-b3507987f050 detect_mult : 3 dst_ip : \"10.5.204.121\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : 684c4489-5b59-4693-8d8c-3beab93f8093 detect_mult : 3 dst_ip : \"10.5.204.109\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : f0f62077-2ae9-4e79-b4f8-a446ec6e784c detect_mult : 3 dst_ip : \"10.5.204.108\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up ### \u6ce8\u610f\u6240\u6709 status \u6b63\u5e38\u90fd\u5e94\u8be5\u662f up \u7684 # 2. \u521b\u5efa\u4e86\u57fa\u4e8e bfd \u7684\u9759\u6001\u8def\u7531 # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 192 .168.0.0/24 10 .5.204.108 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.109 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.121 src-ip ecmp ecmp-symmetric-reply bfd # 3. \u9759\u6001\u8def\u7531\u8be6\u60c5 # k ko nbctl find Logical_Router_Static_Route policy=src-ip options=ecmp_symmetric_reply=\"true\" _uuid : 3aacb384-d5ee-4b14-aebf-59e8c11717ba bfd : 684c4489-5b59-4693-8d8c-3beab93f8093 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.109\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 18bcc585-bc05-430b-925b-ef673c8e1aef bfd : f0f62077-2ae9-4e79-b4f8-a446ec6e784c external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.108\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 7d0a4e6b-cde0-4110-8176-fbaf19738498 bfd : be7df545-2c4c-4751-878f-b3507987f050 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.121\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" # \u540c\u65f6\u5728\u7f51\u5173\u8282\u70b9\u90fd\u5e94\u8be5\u5177\u5907\u4ee5\u4e0b\u8d44\u6e90 [ root@pc-node-1 ~ ] # ip netns exec ovnext bash ip a /usr/sbin/ip: /usr/sbin/ip: cannot execute binary file [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1541 : ovnext0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 00 :00:00:ab:bd:87 brd ff:ff:ff:ff:ff:ff inet 10 .5.204.108/24 brd 10 .5.204.255 scope global ovnext0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:feab:bd87/64 scope link valid_lft forever preferred_lft forever [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0 .0.0.0 10 .5.204.254 0 .0.0.0 UG 0 0 0 ovnext0 10 .5.204.0 0 .0.0.0 255 .255.255.0 U 0 0 0 ovnext0 ## \u6ce8\u610f\u4ee5\u4e0a\u5185\u5bb9\u548c\u4e00\u4e2a internal port unerlay \u516c\u7f51 pod \u5185\u90e8\u7684 ns \u5927\u81f4\u662f\u4e00\u81f4\u7684\uff0c\u8fd9\u91cc\u53ea\u662f\u5728\u7f51\u5173\u8282\u70b9\u4e0a\u5355\u72ec\u7ef4\u62a4\u4e86\u4e00\u4e2a ns [ root@pc-node-1 ~ ] # ip netns exec ovnext bfdd-control status There are 1 sessions: Session 1 id = 1 local = 10 .5.204.108 ( p ) remote = 10 .5.204.122 state = Up ## \u8fd9\u91cc\u5373\u662f lrp bfd \u4f1a\u8bdd\u7684\u53e6\u4e00\u7aef\uff0c\u4e5f\u662f lrp ecmp \u7684\u4e0b\u4e00\u8df3\u7684\u5176\u4e2d\u4e00\u4e2a [ root@pc-node-1 ~ ] # ip netns exec ovnext ping -c1 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 115 time = 21 .6 ms # \u5230\u516c\u7f51\u6ca1\u95ee\u9898 \u53ef\u4ee5\u5728\u67d0\u4e00\u4e2a\u7f51\u5173\u8282\u70b9\u7684 ovnext ns \u5185\u6293\u5230\u51fa\u53bb\u7684\u5305 # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-1 ~ ] # exit [ root@pc-node-1 ~ ] # ssh pc-node-2 Last login: Thu Feb 23 09 :21:08 2023 from 10 .5.32.51 [ root@pc-node-2 ~ ] # ip netns exec ovnext bash [ root@pc-node-2 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-2 ~ ] # exit [ root@pc-node-2 ~ ] # logout Connection to pc-node-2 closed. [ root@pc-node-1 ~ ] # ssh pc-node-3 Last login: Thu Feb 23 08 :32:41 2023 from 10 .5.32.51 [ root@pc-node-3 ~ ] # ip netns exec ovnext bash [ root@pc-node-3 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:2d:f8:ce > 00 :00:00:fd:b2:a4, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 63 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 00 :00:00:fd:b2:a4 > dc:ef:80:5a:44:1a, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [ root@pc-node-3 ~ ] # # \u53ef\u4ee5\u5728\u8be5\u8282\u70b9 down \u6389\u51fa\u53bb\u7684\u7f51\u5361\uff0c\u7136\u540e\u770b pod \u51fa\u53bb\u7684\u5305\u5728\u7f51\u7edc\u4e2d\u65ad\u4e2d\u4f1a\u51fa\u73b0\u51e0\u4e2a\u5305 # \u4e00\u822c\u90fd\u4f1a\u770b\u5230\u4e22 3 \u4e2a\u5305","title":"2. \u81ea\u5b9a\u4e49 vpc \u542f\u7528 ecmp bfd L3 HA \u516c\u7f51\u529f\u80fd"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#3-bfd","text":"\u5728\u67d0\u4e9b\u573a\u666f\u4e0b\uff0c\u53ef\u80fd\u60f3\u76f4\u63a5\u4f7f\u7528\uff08\u96c6\u4e2d\u5f0f\uff09\u5355\u4e2a\u7f51\u5173\u76f4\u63a5\u51fa\u516c\u7f51\uff0c\u8fd9\u4e2a\u65f6\u5019\u548c\u9ed8\u8ba4 vpc enable_eip_snat \u7684\u4f7f\u7528\u6a21\u5f0f\u662f\u4e00\u81f4\u7684 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc2 spec: namespaces: - vpc2 enableExternal: true #enableBfd: true enableBfd: false ## \u5c06 bfd \u529f\u80fd\u76f4\u63a5\u7981\u7528\u5373\u53ef # k ko nbctl lr-route-list vpc2 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # \u5e94\u7528\u540e\u8def\u7531\u4f1a\u5207\u6362\u56de\u6b63\u5e38\u7684\u9ed8\u8ba4\u9759\u6001\u8def\u7531 # \u540c\u65f6 nbctl list bfd \u53ef\u4ee5\u770b\u5230 lrp \u5173\u8054\u7684 bfd \u4f1a\u8bdd\u5df2\u7ecf\u79fb\u9664 # \u800c\u4e14 ovnext ns \u4e2d\u7684\u5bf9\u7aef bfd \u4f1a\u8bdd\u4e5f\u81ea\u52a8\u79fb\u9664 # \u8be5\u5207\u6362\u8fc7\u7a0b\u4fdd\u6301 vpc subnet \u5185\u4fdd\u6301 ping \u672a\u770b\u5230(\u79d2\u7ea7)\u4e22\u5305 # \u518d\u5207\u6362\u56de\u53bb \u4e5f\u672a\u770b\u5230(\u79d2\u7ea7)\u4e22\u5305 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"3. \u5173\u95ed bfd \u6a21\u5f0f"},{"location":"advance/ovn-remote-port-mirroring/","text":"OVN \u6d41\u91cf\u955c\u50cf \u00b6 \u6b64\u529f\u80fd\u53ef\u4ee5\u5c06\u6307\u5b9a Pod\u3001\u6307\u5b9a\u65b9\u5411\u7684\u6d41\u91cf\uff0c\u901a\u8fc7 GRE/ERSPAN \u5c01\u88c5\u540e\uff0c\u4f20\u8f93\u5230\u8fdc\u7aef\u3002 \u6b64\u529f\u80fd\u8981\u6c42 Kube-OVN \u7248\u672c\u4e0d\u4f4e\u4e8e v1.12\u3002 \u90e8\u7f72 Multus-CNI \u00b6 \u53c2\u8003 Multus-CNI \u6587\u6863 \u90e8\u7f72 Multus\u3002 \u521b\u5efa\u9644\u5c5e\u7f51\u7edc \u00b6 \u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa\u9644\u5c5e\u7f51\u7edc\uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : | { \"cniVersion\": \"0.3.1\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" } \u5176\u4e2d provider \u5b57\u6bb5\u683c\u5f0f\u4e3a <NAME>.<NAMESPACE>.ovn \u3002 \u521b\u5efa Underlay \u7f51\u7edc \u00b6 \u955c\u50cf\u6d41\u91cf\u662f\u5c01\u88c5\u540e\u8fdb\u884c\u4f20\u8f93\u7684\uff0c\u56e0\u6b64\u7528\u4e8e\u4f20\u8f93\u7684\u7f51\u7edc\uff0cMTU \u9700\u8981\u5927\u4e8e\u88ab\u955c\u50cf\u7684 LSP/Pod\u3002\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 Underlay \u4f5c\u4e3a\u4f20\u8f93\u7f51\u7edc\u3002 \u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa Underlay \u7f51\u7edc\uff1a apiVersion : kubeovn.io/v1 kind : ProviderNetwork metadata : name : net1 spec : defaultInterface : eth1 --- apiVersion : kubeovn.io/v1 kind : Vlan metadata : name : vlan1 spec : id : 0 provider : net1 --- apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.19.0.0/16 excludeIps : - 172.19.0.2..172.19.0.20 gateway : 172.19.0.1 vlan : vlan1 provider : attachnet.default.ovn \u5176\u4e2d\uff0c\u5b50\u7f51\u7684 provider \u5fc5\u987b\u4e0e\u9644\u5c5e\u7f51\u7edc\u7684 provider \u76f8\u540c\u3002 \u521b\u5efa\u6d41\u91cf\u63a5\u6536 Pod \u00b6 \u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa\u7528\u4e8e\u63a5\u6536\u955c\u50cf\u6d41\u91cf\u7684 Pod\uff1a apiVersion : v1 kind : Pod metadata : name : pod1 annotations : k8s.v1.cni.cncf.io/networks : default/attachnet spec : containers : - name : bash image : docker.io/kubeovn/kube-ovn:v1.12.0 args : - bash - -c - sleep infinity securityContext : privileged : true \u521b\u5efa\u5b8c\u6210\u540e\uff0c\u67e5\u770b Pod \u7684 IP \u5730\u5740\uff1a $ kubectl get ips | grep pod1 pod1.default 10 .16.0.12 00 :00:00:FF:34:24 kube-ovn-worker ovn-default pod1.default.attachnet.default.ovn 172 .19.0.21 00 :00:00:A0:30:68 kube-ovn-worker subnet1 \u8bb0\u4f4f\u7b2c\u4e8c\u7f51\u5361\u7684 IP \u5730\u5740 172.19.0.21 \u3002 \u521b\u5efa OVN \u6d41\u91cf\u955c\u50cf \u00b6 \u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u521b\u5efa OVN \u6d41\u91cf\u955c\u50cf\uff1a kubectl ko nbctl mirror-add mirror1 gre 99 from-lport 172 .19.0.21 kubectl ko nbctl lsp-attach-mirror coredns-787d4945fb-gpnkb.kube-system mirror1 \u5176\u4e2d coredns-787d4945fb-gpnkb.kube-system \u662f OVN LSP \u7684\u540d\u79f0\uff0c\u683c\u5f0f\u901a\u5e38\u4e3a <POD_NAME>.<POD_NAMESPACE> \u3002 \u76f8\u5173\u7684 OVN \u547d\u4ee4\u4f7f\u7528\u65b9\u6cd5\u5982\u4e0b\uff1a ovn-nbctl mirror-add <NAME> <TYPE> <INDEX> <FILTER> <IP> NAME - add a mirror with given name TYPE - specify TYPE 'gre' or 'erspan' INDEX - specify the tunnel INDEX value (indicates key if GRE, erpsan_idx if ERSPAN) FILTER - specify FILTER for mirroring selection ('to-lport' / 'from-lport') IP - specify Sink / Destination i.e. Remote IP ovn-nbctl mirror-del [NAME] remove mirrors ovn-nbctl mirror-list print mirrors ovn-nbctl lsp-attach-mirror PORT MIRROR attach source PORT to MIRROR ovn-nbctl lsp-detach-mirror PORT MIRROR detach source PORT from MIRROR \u914d\u7f6e\u6d41\u91cf\u63a5\u6536 Pod \u00b6 \u5728\u524d\u9762\u521b\u5efa\u7684 Pod \u4e2d\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff1a root@pod1:/kube-ovn# ip link add mirror1 type gretap local 172 .19.0.21 key 99 dev net1 root@pod1:/kube-ovn# ip link set mirror1 up \u63a5\u4e0b\u6765\u5c31\u53ef\u4ee5\u5728\u63a5\u6536\u6d41\u91cf\u7684 Pod \u4e2d\u8fdb\u884c\u6293\u5305\u9a8c\u8bc1\uff1a root@pod1:/kube-ovn# tcpdump -i mirror1 -nnve tcpdump: listening on mirror1, link-type EN10MB ( Ethernet ) , snapshot length 262144 bytes 05 :13:30.328808 00 :00:00:a3:f5:e2 > 00 :00:00:97:0f:6e, ethertype ARP ( 0x0806 ) , length 42 : Ethernet ( len 6 ) , IPv4 ( len 4 ) , Request who-has 10 .16.0.7 tell 10 .16.0.4, length 28 05 :13:30.559167 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57364 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.50472: 34511 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.559343 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57365 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.45177: 1659 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.560625 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 200 : ( tos 0x0, ttl 64 , id 57367 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 186 ) 10 .16.0.4.53 > 10 .16.0.6.43848: 2636 *- 0 /1/1 ( 158 ) 05 :13:30.562774 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 191 : ( tos 0x0, ttl 64 , id 57368 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 177 ) 10 .16.0.4.53 > 10 .16.0.6.37755: 48737 NXDomain*- 0 /1/1 ( 149 ) 05 :13:30.563523 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 187 : ( tos 0x0, ttl 64 , id 57369 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 173 ) 10 .16.0.4.53 > 10 .16.0.6.53887: 45519 NXDomain*- 0 /1/1 ( 145 ) 05 :13:30.564940 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57370 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.40846: 25745 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.565140 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57371 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.45214: 61875 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.566023 00 :00:00:a3:f5:e2 > 00 :00:00:55:e4:4e, ethertype IPv4 ( 0x0800 ) , length 80 : ( tos 0x0, ttl 64 , id 45937 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 66 ) 10 .16.0.4.44116 > 172 .18.0.1.53: 16025 + [ 1au ] AAAA? alauda.cn. ( 38 ) \u6ce8\u610f\u4e8b\u9879 \u00b6 \u5982\u679c\u4f7f\u7528 ERSPAN \u4f5c\u4e3a\u5c01\u88c5\u534f\u8bae\uff0cOVN \u8282\u70b9\u53ca\u8fdc\u7aef\u8bbe\u5907\u7684 Linux \u5185\u6838\u7248\u672c\u4e0d\u5f97\u4f4e\u4e8e 4.14\u3002\u82e5\u4f7f\u7528 ERSPAN \u4f5c\u4e3a\u5c01\u88c5\u534f\u8bae\u4e14\u4f7f\u7528 IPv6 \u4f5c\u4e3a\u4f20\u8f93\u7f51\u7edc\uff0cLinux \u5185\u6838\u7248\u672c\u4e0d\u5f97\u4f4e\u4e8e 4.16\u3002 \u955c\u50cf\u6d41\u91cf\u7684\u4f20\u8f93\u662f\u5355\u5411\u7684\uff0c\u53ea\u9700\u4fdd\u8bc1 OVN \u8282\u70b9\u80fd\u591f\u8bbf\u95ee\u8fdc\u7aef\u8bbe\u5907\u5373\u53ef\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN \u6d41\u91cf\u955c\u50cf"},{"location":"advance/ovn-remote-port-mirroring/#ovn","text":"\u6b64\u529f\u80fd\u53ef\u4ee5\u5c06\u6307\u5b9a Pod\u3001\u6307\u5b9a\u65b9\u5411\u7684\u6d41\u91cf\uff0c\u901a\u8fc7 GRE/ERSPAN \u5c01\u88c5\u540e\uff0c\u4f20\u8f93\u5230\u8fdc\u7aef\u3002 \u6b64\u529f\u80fd\u8981\u6c42 Kube-OVN \u7248\u672c\u4e0d\u4f4e\u4e8e v1.12\u3002","title":"OVN \u6d41\u91cf\u955c\u50cf"},{"location":"advance/ovn-remote-port-mirroring/#multus-cni","text":"\u53c2\u8003 Multus-CNI \u6587\u6863 \u90e8\u7f72 Multus\u3002","title":"\u90e8\u7f72 Multus-CNI"},{"location":"advance/ovn-remote-port-mirroring/#_1","text":"\u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa\u9644\u5c5e\u7f51\u7edc\uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : | { \"cniVersion\": \"0.3.1\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" } \u5176\u4e2d provider \u5b57\u6bb5\u683c\u5f0f\u4e3a <NAME>.<NAMESPACE>.ovn \u3002","title":"\u521b\u5efa\u9644\u5c5e\u7f51\u7edc"},{"location":"advance/ovn-remote-port-mirroring/#underlay","text":"\u955c\u50cf\u6d41\u91cf\u662f\u5c01\u88c5\u540e\u8fdb\u884c\u4f20\u8f93\u7684\uff0c\u56e0\u6b64\u7528\u4e8e\u4f20\u8f93\u7684\u7f51\u7edc\uff0cMTU \u9700\u8981\u5927\u4e8e\u88ab\u955c\u50cf\u7684 LSP/Pod\u3002\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 Underlay \u4f5c\u4e3a\u4f20\u8f93\u7f51\u7edc\u3002 \u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa Underlay \u7f51\u7edc\uff1a apiVersion : kubeovn.io/v1 kind : ProviderNetwork metadata : name : net1 spec : defaultInterface : eth1 --- apiVersion : kubeovn.io/v1 kind : Vlan metadata : name : vlan1 spec : id : 0 provider : net1 --- apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.19.0.0/16 excludeIps : - 172.19.0.2..172.19.0.20 gateway : 172.19.0.1 vlan : vlan1 provider : attachnet.default.ovn \u5176\u4e2d\uff0c\u5b50\u7f51\u7684 provider \u5fc5\u987b\u4e0e\u9644\u5c5e\u7f51\u7edc\u7684 provider \u76f8\u540c\u3002","title":"\u521b\u5efa Underlay \u7f51\u7edc"},{"location":"advance/ovn-remote-port-mirroring/#pod","text":"\u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa\u7528\u4e8e\u63a5\u6536\u955c\u50cf\u6d41\u91cf\u7684 Pod\uff1a apiVersion : v1 kind : Pod metadata : name : pod1 annotations : k8s.v1.cni.cncf.io/networks : default/attachnet spec : containers : - name : bash image : docker.io/kubeovn/kube-ovn:v1.12.0 args : - bash - -c - sleep infinity securityContext : privileged : true \u521b\u5efa\u5b8c\u6210\u540e\uff0c\u67e5\u770b Pod \u7684 IP \u5730\u5740\uff1a $ kubectl get ips | grep pod1 pod1.default 10 .16.0.12 00 :00:00:FF:34:24 kube-ovn-worker ovn-default pod1.default.attachnet.default.ovn 172 .19.0.21 00 :00:00:A0:30:68 kube-ovn-worker subnet1 \u8bb0\u4f4f\u7b2c\u4e8c\u7f51\u5361\u7684 IP \u5730\u5740 172.19.0.21 \u3002","title":"\u521b\u5efa\u6d41\u91cf\u63a5\u6536 Pod"},{"location":"advance/ovn-remote-port-mirroring/#ovn_1","text":"\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u521b\u5efa OVN \u6d41\u91cf\u955c\u50cf\uff1a kubectl ko nbctl mirror-add mirror1 gre 99 from-lport 172 .19.0.21 kubectl ko nbctl lsp-attach-mirror coredns-787d4945fb-gpnkb.kube-system mirror1 \u5176\u4e2d coredns-787d4945fb-gpnkb.kube-system \u662f OVN LSP \u7684\u540d\u79f0\uff0c\u683c\u5f0f\u901a\u5e38\u4e3a <POD_NAME>.<POD_NAMESPACE> \u3002 \u76f8\u5173\u7684 OVN \u547d\u4ee4\u4f7f\u7528\u65b9\u6cd5\u5982\u4e0b\uff1a ovn-nbctl mirror-add <NAME> <TYPE> <INDEX> <FILTER> <IP> NAME - add a mirror with given name TYPE - specify TYPE 'gre' or 'erspan' INDEX - specify the tunnel INDEX value (indicates key if GRE, erpsan_idx if ERSPAN) FILTER - specify FILTER for mirroring selection ('to-lport' / 'from-lport') IP - specify Sink / Destination i.e. Remote IP ovn-nbctl mirror-del [NAME] remove mirrors ovn-nbctl mirror-list print mirrors ovn-nbctl lsp-attach-mirror PORT MIRROR attach source PORT to MIRROR ovn-nbctl lsp-detach-mirror PORT MIRROR detach source PORT from MIRROR","title":"\u521b\u5efa OVN \u6d41\u91cf\u955c\u50cf"},{"location":"advance/ovn-remote-port-mirroring/#pod_1","text":"\u5728\u524d\u9762\u521b\u5efa\u7684 Pod \u4e2d\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff1a root@pod1:/kube-ovn# ip link add mirror1 type gretap local 172 .19.0.21 key 99 dev net1 root@pod1:/kube-ovn# ip link set mirror1 up \u63a5\u4e0b\u6765\u5c31\u53ef\u4ee5\u5728\u63a5\u6536\u6d41\u91cf\u7684 Pod \u4e2d\u8fdb\u884c\u6293\u5305\u9a8c\u8bc1\uff1a root@pod1:/kube-ovn# tcpdump -i mirror1 -nnve tcpdump: listening on mirror1, link-type EN10MB ( Ethernet ) , snapshot length 262144 bytes 05 :13:30.328808 00 :00:00:a3:f5:e2 > 00 :00:00:97:0f:6e, ethertype ARP ( 0x0806 ) , length 42 : Ethernet ( len 6 ) , IPv4 ( len 4 ) , Request who-has 10 .16.0.7 tell 10 .16.0.4, length 28 05 :13:30.559167 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57364 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.50472: 34511 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.559343 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57365 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.45177: 1659 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.560625 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 200 : ( tos 0x0, ttl 64 , id 57367 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 186 ) 10 .16.0.4.53 > 10 .16.0.6.43848: 2636 *- 0 /1/1 ( 158 ) 05 :13:30.562774 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 191 : ( tos 0x0, ttl 64 , id 57368 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 177 ) 10 .16.0.4.53 > 10 .16.0.6.37755: 48737 NXDomain*- 0 /1/1 ( 149 ) 05 :13:30.563523 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 187 : ( tos 0x0, ttl 64 , id 57369 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 173 ) 10 .16.0.4.53 > 10 .16.0.6.53887: 45519 NXDomain*- 0 /1/1 ( 145 ) 05 :13:30.564940 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57370 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.40846: 25745 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.565140 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57371 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.45214: 61875 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.566023 00 :00:00:a3:f5:e2 > 00 :00:00:55:e4:4e, ethertype IPv4 ( 0x0800 ) , length 80 : ( tos 0x0, ttl 64 , id 45937 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 66 ) 10 .16.0.4.44116 > 172 .18.0.1.53: 16025 + [ 1au ] AAAA? alauda.cn. ( 38 )","title":"\u914d\u7f6e\u6d41\u91cf\u63a5\u6536 Pod"},{"location":"advance/ovn-remote-port-mirroring/#_2","text":"\u5982\u679c\u4f7f\u7528 ERSPAN \u4f5c\u4e3a\u5c01\u88c5\u534f\u8bae\uff0cOVN \u8282\u70b9\u53ca\u8fdc\u7aef\u8bbe\u5907\u7684 Linux \u5185\u6838\u7248\u672c\u4e0d\u5f97\u4f4e\u4e8e 4.14\u3002\u82e5\u4f7f\u7528 ERSPAN \u4f5c\u4e3a\u5c01\u88c5\u534f\u8bae\u4e14\u4f7f\u7528 IPv6 \u4f5c\u4e3a\u4f20\u8f93\u7f51\u7edc\uff0cLinux \u5185\u6838\u7248\u672c\u4e0d\u5f97\u4f4e\u4e8e 4.16\u3002 \u955c\u50cf\u6d41\u91cf\u7684\u4f20\u8f93\u662f\u5355\u5411\u7684\uff0c\u53ea\u9700\u4fdd\u8bc1 OVN \u8282\u70b9\u80fd\u591f\u8bbf\u95ee\u8fdc\u7aef\u8bbe\u5907\u5373\u53ef\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"advance/performance-tuning/","text":"\u6027\u80fd\u8c03\u4f18 \u00b6 \u4e3a\u4e86\u4fdd\u6301\u5b89\u88c5\u7684\u7b80\u5355\u548c\u529f\u80fd\u7684\u5b8c\u5907\uff0cKube-OVN \u7684\u9ed8\u8ba4\u5b89\u88c5\u811a\u672c\u5e76\u6ca1\u6709\u5bf9\u6027\u80fd\u9488\u5bf9\u6027\u7684\u4f18\u5316\u3002\u5982\u679c\u5e94\u7528\u5bf9\u5ef6\u8fdf\u548c\u541e\u5410\u91cf\u654f\u611f\uff0c \u7ba1\u7406\u5458\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u5bf9\u6027\u80fd\u8fdb\u884c\u9488\u5bf9\u6027\u4f18\u5316\u3002 \u793e\u533a\u4f1a\u4e0d\u65ad\u8fed\u4ee3\u63a7\u5236\u9762\u677f\u548c\u4f18\u5316\u9762\u7684\u6027\u80fd\uff0c\u90e8\u5206\u901a\u7528\u6027\u80fd\u4f18\u5316\u5df2\u7ecf\u96c6\u6210\u5230\u6700\u65b0\u7248\u672c\uff0c\u5efa\u8bae\u4f7f\u7528\u6700\u65b0\u7248\u672c\u83b7\u5f97\u66f4\u597d\u7684\u9ed8\u8ba4\u6027\u80fd\u3002 \u66f4\u591a\u5173\u4e8e\u6027\u80fd\u4f18\u5316\u7684\u8fc7\u7a0b\u548c\u65b9\u6cd5\u8bba\uff0c\u53ef\u4ee5\u89c2\u770b\u89c6\u9891\u5206\u4eab\uff1a Kube-OVN \u5bb9\u5668\u6027\u80fd\u4f18\u5316\u4e4b\u65c5 \u3002 \u57fa\u51c6\u6d4b\u8bd5 \u00b6 \u7531\u4e8e\u8f6f\u786c\u4ef6\u73af\u5883\u7684\u5dee\u5f02\u6781\u5927\uff0c\u8fd9\u91cc\u63d0\u4f9b\u7684\u6027\u80fd\u6d4b\u8bd5\u6570\u636e\u53ea\u80fd\u4f5c\u4e3a\u53c2\u8003\uff0c\u5b9e\u9645\u6d4b\u8bd5\u7ed3\u679c\u4f1a\u548c\u672c\u6587\u6863\u4e2d\u7684\u7ed3\u679c\u5b58\u5728\u8f83\u5927\u5dee\u5f02\u3002 \u5efa\u8bae\u6bd4\u8f83\u4f18\u5316\u524d\u540e\u7684\u6027\u80fd\u6d4b\u8bd5\u7ed3\u679c\uff0c\u548c\u5bbf\u4e3b\u673a\u7f51\u7edc\u548c\u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u6bd4\u8f83\u3002 Overlay \u4f18\u5316\u524d\u540e\u6027\u80fd\u5bf9\u6bd4 \u00b6 \u73af\u5883\u4fe1\u606f\uff1a Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 Overlay \u6a21\u5f0f CPU: Intel(R) Xeon(R) E-2278G Network: 2*10Gbps, xmit_hash_policy=layer3+4 \u6211\u4eec\u4f7f\u7528 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw \u6d4b\u8bd5 1 \u5b57\u8282\u5c0f\u5305\u4e0b tcp/udp \u7684\u5e26\u5bbd\u548c\u5ef6\u8fdf\uff0c\u5206\u522b\u6d4b\u8bd5\u4f18\u5316\u524d\uff0c\u4f18\u5316\u540e\u4ee5\u53ca\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u6027\u80fd\uff1a Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Default 25.7 22.9 27.1 1.59 Kube-OVN Optimized 13.9 12.9 27.6 5.57 HOST Network 13.1 12.4 28.2 6.02 Overlay\uff0c Underlay \u4ee5\u53ca Calico \u4e0d\u540c\u6a21\u5f0f\u6027\u80fd\u5bf9\u6bd4 \u00b6 \u4e0b\u9762\u6211\u4eec\u4f1a\u6bd4\u8f83\u4f18\u5316\u540e Kube-OVN \u5728\u4e0d\u540c\u5305\u5927\u5c0f\u4e0b\u7684 Overlay \u548c Underlay \u6027\u80fd\uff0c\u5e76\u548c Calico \u7684 IPIP Always , IPIP never \u4ee5\u53ca\u5bbf\u4e3b\u673a\u7f51\u7edc\u505a\u6bd4\u8f83\u3002 Environment : Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 CPU: AMD EPYC 7402P 24-Core Processor Network: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Overlay 15.2 14.6 23.6 2.65 Kube-OVN Underlay 14.3 13.8 24.2 3.46 Calico IPIP 21.4 20.2 23.6 1.18 Calico NoEncap 19.3 16.9 23.6 1.76 HOST Network 16.6 15.4 24.8 2.64 qperf -t 60 <server ip> -ub -oo msg_size:1K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 16.5 15.8 10.2 2.77 Kube-OVN Underlay 15.9 14.5 9.6 3.22 Calico IPIP 22.5 21.5 1.45 1.14 Calico NoEncap 19.4 18.3 3.76 1.63 HOST Network 18.1 16.6 9.32 2.66 qperf -t 60 <server ip> -ub -oo msg_size:4K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 34.7 41.6 16.0 9.23 Kube-OVN Underlay 32.6 44 15.1 6.71 Calico IPIP 44.8 52.9 2.94 3.26 Calico NoEncap 40 49.6 6.56 4.19 HOST Network 35.9 45.9 14.6 5.59 \u5728\u90e8\u5206\u60c5\u51b5\u4e0b\u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u4f1a\u4f18\u4e8e\u5bbf\u4e3b\u673a\u7f51\u7edc\uff0c\u8fd9\u662f\u4f18\u4e8e\u7ecf\u8fc7\u4f18\u5316\u540e\u5bb9\u5668\u7f51\u7edc\u8def\u5f84\u5b8c\u5168\u7ed5\u8fc7\u4e86 netfilter\uff0c \u800c\u5bbf\u4e3b\u673a\u7f51\u7edc\u7531\u4e8e kube-proxy \u7684\u5b58\u5728\u6240\u6709\u6570\u636e\u5305\u5747\u9700\u7ecf\u8fc7 netfilter\uff0c\u4f1a\u5bfc\u81f4\u5728\u4e00\u4e9b\u73af\u5883\u4e0b\u5bb9\u5668\u7f51\u7edc \u7684\u6d88\u8017\u76f8\u5bf9\u66f4\u5c0f\uff0c\u56e0\u6b64\u4f1a\u6709\u66f4\u597d\u7684\u6027\u80fd\u8868\u73b0\u3002 \u6570\u636e\u5e73\u9762\u6027\u80fd\u4f18\u5316\u65b9\u6cd5 \u00b6 \u8fd9\u91cc\u4ecb\u7ecd\u7684\u4f18\u5316\u65b9\u6cd5\u548c\u8f6f\u786c\u4ef6\u73af\u5883\u4ee5\u53ca\u6240\u9700\u8981\u7684\u529f\u80fd\u76f8\u5173\uff0c\u8bf7\u4ed4\u7ec6\u4e86\u89e3\u4f18\u5316\u7684\u524d\u63d0\u6761\u4ef6\u518d\u8fdb\u884c\u5c1d\u8bd5\u3002 CPU \u6027\u80fd\u6a21\u5f0f\u8c03\u6574 \u00b6 \u90e8\u5206\u73af\u5883\u4e0b CPU \u8fd0\u884c\u5728\u8282\u80fd\u6a21\u5f0f\uff0c\u8be5\u6a21\u5f0f\u4e0b\u6027\u80fd\u8868\u73b0\u5c06\u4f1a\u4e0d\u7a33\u5b9a\uff0c\u5ef6\u8fdf\u4f1a\u51fa\u73b0\u660e\u663e\u589e\u52a0\uff0c\u5efa\u8bae\u4f7f\u7528 CPU \u7684\u6027\u80fd\u6a21\u5f0f\u83b7\u5f97\u66f4\u7a33\u5b9a\u7684\u6027\u80fd\u8868\u73b0\uff1a cpupower frequency-set -g performance \u7f51\u5361\u786c\u4ef6\u961f\u5217\u8c03\u6574 \u00b6 \u5728\u6d41\u91cf\u589e\u5927\u7684\u60c5\u51b5\u4e0b\uff0c\u7f13\u51b2\u961f\u5217\u8fc7\u77ed\u53ef\u80fd\u5bfc\u81f4\u8f83\u9ad8\u7684\u4e22\u5305\u7387\u5bfc\u81f4\u6027\u80fd\u663e\u8457\u4e0b\u964d\uff0c\u9700\u8981\u8fdb\u884c\u8c03\u6574 \u68c0\u67e5\u5f53\u524d\u7f51\u5361\u961f\u5217\u957f\u5ea6\uff1a # ethtool -g eno1 Ring parameters for eno1: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 255 RX Mini: 0 RX Jumbo: 0 TX: 255 \u589e\u52a0\u961f\u5217\u957f\u5ea6\u81f3\u6700\u5927\u503c\uff1a ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096 \u4f7f\u7528 tuned \u4f18\u5316\u7cfb\u7edf\u53c2\u6570 \u00b6 tuned \u53ef\u4ee5\u4f7f\u7528\u4e00\u7cfb\u5217\u9884\u7f6e\u7684 profile \u6587\u4ef6\u4fdd\u5b58\u4e86\u9488\u5bf9\u7279\u5b9a\u573a\u666f\u7684\u4e00\u7cfb\u5217\u7cfb\u7edf\u4f18\u5316\u914d\u7f6e\u3002 \u9488\u5bf9\u5ef6\u8fdf\u4f18\u5148\u573a\u666f\uff1a tuned-adm profile network-latency \u9488\u5bf9\u541e\u5410\u91cf\u4f18\u5148\u573a\u666f\uff1a tuned-adm profile network-throughput \u4e2d\u65ad\u7ed1\u5b9a \u00b6 \u6211\u4eec\u63a8\u8350\u7981\u7528 irqbalance \u5e76\u5c06\u7f51\u5361\u4e2d\u65ad\u548c\u7279\u5b9a CPU \u8fdb\u884c\u7ed1\u5b9a\uff0c\u6765\u907f\u514d\u5728\u591a\u4e2a CPU \u4e4b\u95f4\u5207\u6362\u5bfc\u81f4\u7684\u6027\u80fd\u6ce2\u52a8\u3002 \u5173\u95ed OVN LB \u00b6 OVN \u7684 L2 LB \u5b9e\u73b0\u8fc7\u7a0b\u4e2d\u9700\u8981\u8c03\u7528\u5185\u6838\u7684 conntrack \u6a21\u5757\u5e76\u8fdb\u884c recirculate \u5bfc\u81f4\u5927\u91cf\u7684 CPU \u5f00\u9500\uff0c\u7ecf\u6d4b\u8bd5\u8be5\u529f\u80fd\u4f1a\u5e26\u6765 20% \u5de6\u53f3\u7684 CPU \u5f00\u9500\uff0c \u5728 Overlay \u7f51\u7edc\u6a21\u5f0f\u4e0b\u53ef\u4ee5\u4f7f\u7528 kube-proxy \u5b8c\u6210 Service \u8f6c\u53d1\u529f\u80fd\uff0c\u83b7\u5f97\u66f4\u597d\u7684 Pod-to-Pod \u6027\u80fd\u3002\u53ef\u4ee5\u5728 kube-ovn-controller \u4e2d\u5173\u95ed\u8be5\u529f\u80fd\uff1a command : - /kube-ovn/start-controller.sh args : ... - --enable-lb=false ... Underlay \u6a21\u5f0f\u4e0b kube-proxy \u65e0\u6cd5\u4f7f\u7528 iptables \u6216 ipvs \u63a7\u5236\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\uff0c\u5982\u9700\u5173\u95ed LB \u529f\u80fd\u9700\u8981\u786e\u8ba4\u662f\u5426\u4e0d\u9700\u8981 Service \u529f\u80fd\u3002 \u5185\u6838 FastPath \u6a21\u5757 \u00b6 \u7531\u4e8e\u5bb9\u5668\u7f51\u7edc\u548c\u5bbf\u4e3b\u673a\u7f51\u7edc\u5728\u4e0d\u540c\u7684 network ns\uff0c\u6570\u636e\u5305\u5728\u8de8\u5bbf\u4e3b\u673a\u4f20\u8f93\u65f6\u4f1a\u591a\u6b21\u7ecf\u8fc7 netfilter \u6a21\u5757\uff0c\u4f1a\u5e26\u6765\u8fd1 20% \u7684 CPU \u5f00\u9500\u3002\u7531\u4e8e\u5927\u90e8\u5206\u60c5\u51b5\u4e0b \u5bb9\u5668\u7f51\u7edc\u5185\u5e94\u7528\u65e0\u987b\u4f7f\u7528 netfilter \u6a21\u5757\u7684\u529f\u80fd\uff0c FastPath \u6a21\u5757\u53ef\u4ee5\u7ed5\u8fc7 netfilter \u964d\u4f4e CPU \u5f00\u9500\u3002 \u5982\u5bb9\u5668\u7f51\u7edc\u5185\u9700\u8981\u4f7f\u7528 netfilter \u63d0\u4f9b\u7684\u529f\u80fd\u5982 iptables\uff0cipvs\uff0cnftables \u7b49\uff0c\u8be5\u6a21\u5757\u4f1a\u4f7f\u76f8\u5173\u529f\u80fd\u5931\u6548\u3002 \u7531\u4e8e\u5185\u6838\u6a21\u5757\u548c\u5185\u6838\u7248\u672c\u76f8\u5173\uff0c\u65e0\u6cd5\u63d0\u4f9b\u4e00\u4e2a\u5355\u4e00\u9002\u5e94\u6240\u6709\u5185\u6838\u7684\u5185\u6838\u6a21\u5757\u5236\u54c1\u3002\u6211\u4eec\u9884\u5148\u7f16\u8bd1\u4e86\u90e8\u5206\u5185\u6838\u7684 FastPath \u6a21\u5757\uff0c \u53ef\u4ee5\u524d\u5f80 tunning-package \u8fdb\u884c\u4e0b\u8f7d\u3002 \u4e5f\u53ef\u4ee5\u624b\u52a8\u8fdb\u884c\u7f16\u8bd1\uff0c\u65b9\u6cd5\u53c2\u8003 \u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757 \u83b7\u5f97\u5185\u6838\u6a21\u5757\u540e\u53ef\u5728\u6bcf\u4e2a\u8282\u70b9\u4f7f\u7528 insmod kube_ovn_fastpath.ko \u52a0\u8f7d FastPath \u6a21\u5757\uff0c\u5e76\u4f7f\u7528 dmesg \u9a8c\u8bc1\u6a21\u5757\u52a0\u8f7d\u6210\u529f\uff1a # dmesg ... [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in ... OVS \u5185\u6838\u6a21\u5757\u4f18\u5316 \u00b6 OVS \u7684 flow \u5904\u7406\u5305\u62ec\u54c8\u5e0c\u8ba1\u7b97\uff0c\u5339\u914d\u7b49\u64cd\u4f5c\u4f1a\u6d88\u8017\u5927\u7ea6 10% \u5de6\u53f3\u7684 CPU \u8d44\u6e90\u3002\u73b0\u4ee3 x86 CPU \u4e0a\u7684\u4e00\u4e9b\u6307\u4ee4\u96c6\u4f8b\u5982 popcnt \u548c sse4.2 \u53ef\u4ee5 \u52a0\u901f\u76f8\u5173\u8ba1\u7b97\u8fc7\u7a0b\uff0c\u4f46\u5185\u6838\u9ed8\u8ba4\u7f16\u8bd1\u672a\u5f00\u542f\u76f8\u5173\u9009\u9879\u3002\u7ecf\u6d4b\u8bd5\u5728\u5f00\u542f\u76f8\u5e94\u6307\u4ee4\u96c6\u4f18\u5316\u540e\uff0cflow \u76f8\u5173\u64cd\u4f5c CPU \u6d88\u8017\u5c06\u4f1a\u964d\u81f3 5% \u5de6\u53f3\u3002 \u548c FastPath \u6a21\u5757\u7684\u7f16\u8bd1\u7c7b\u4f3c\uff0c\u7531\u4e8e\u5185\u6838\u6a21\u5757\u548c\u5185\u6838\u7248\u672c\u76f8\u5173\uff0c\u65e0\u6cd5\u63d0\u4f9b\u4e00\u4e2a\u5355\u4e00\u9002\u5e94\u6240\u6709\u5185\u6838\u7684\u5185\u6838\u6a21\u5757\u5236\u54c1\u3002\u7528\u6237\u9700\u8981\u624b\u52a8\u7f16\u8bd1\u6216\u8005 \u524d\u5f80 tunning-package \u67e5\u770b\u662f\u5426\u6709\u5df2\u7f16\u8bd1\u597d\u7684\u5236\u54c1\u8fdb\u884c\u4e0b\u8f7d\u3002 \u4f7f\u7528\u8be5\u5185\u6838\u6a21\u5757\u524d\u8bf7\u5148\u786e\u8ba4 CPU \u662f\u5426\u652f\u6301\u76f8\u5173\u6307\u4ee4\u96c6\uff1a cat /proc/cpuinfo | grep popcnt cat /proc/cpuinfo | grep sse4_2 CentOS \u4e0b\u7f16\u8bd1\u5b89\u88c5 \u00b6 \u5b89\u88c5\u76f8\u5173\u7f16\u8bd1\u4f9d\u8d56\u548c\u5185\u6838\u5934\u6587\u4ef6\uff1a yum install -y gcc kernel-devel- $( uname -r ) python3 autoconf automake libtool rpm-build openssl-devel \u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u5e76\u751f\u6210\u5bf9\u5e94 RPM \u6587\u4ef6: git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --with-linux = /lib/modules/ $( uname -r ) /build CFLAGS = \"-g -O2 -mpopcnt -msse4.2\" make rpm-fedora-kmod cd rpm/rpmbuild/RPMS/x86_64/ \u590d\u5236 RPM \u5230\u6bcf\u4e2a\u8282\u70b9\u5e76\u8fdb\u884c\u5b89\u88c5\uff1a rpm -i openvswitch-kmod-2.15.2-1.el7.x86_64.rpm \u82e5\u4e4b\u524d\u5df2\u7ecf\u542f\u52a8\u8fc7 Kube-OVN\uff0c\u65e7\u7248\u672c OVS \u6a21\u5757\u5df2\u52a0\u8f7d\u81f3\u5185\u6838\uff0c\u5efa\u8bae\u91cd\u542f\u673a\u5668\u91cd\u65b0\u52a0\u8f7d\u65b0\u7248\u5185\u6838\u6a21\u5757\u3002 Ubuntu \u4e0b\u7f16\u8bd1\u5b89\u88c5 \u00b6 \u5b89\u88c5\u76f8\u5173\u7f16\u8bd1\u4f9d\u8d56\u548c\u5185\u6838\u5934\u6587\u4ef6\uff1a apt install -y autoconf automake libtool gcc build-essential libssl-dev \u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u5e76\u5b89\u88c5\uff1a apt install -y autoconf automake libtool gcc build-essential libssl-dev git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --prefix = /usr/ --localstatedir = /var --enable-ssl --with-linux = /lib/modules/ $( uname -r ) /build make -j ` nproc ` make install make modules_install cat > /etc/depmod.d/openvswitch.conf << EOF override openvswitch * extra override vport-* * extra EOF depmod -a cp debian/openvswitch-switch.init /etc/init.d/openvswitch-switch /etc/init.d/openvswitch-switch force-reload-kmod \u82e5\u4e4b\u524d\u5df2\u7ecf\u542f\u52a8\u8fc7 Kube-OVN\uff0c\u65e7\u7248\u672c OVS \u6a21\u5757\u5df2\u52a0\u8f7d\u81f3\u5185\u6838\uff0c\u5efa\u8bae\u91cd\u542f\u673a\u5668\u91cd\u65b0\u52a0\u8f7d\u65b0\u7248\u5185\u6838\u6a21\u5757\u3002 \u4f7f\u7528 STT \u7c7b\u578b\u96a7\u9053 \u00b6 \u5e38\u89c1\u7684\u96a7\u9053\u5c01\u88c5\u534f\u8bae\u4f8b\u5982 Geneve \u548c Vxlan \u4f7f\u7528 UDP \u534f\u8bae\u5bf9\u6570\u636e\u5305\u8fdb\u884c\u5c01\u88c5\uff0c\u5728\u5185\u6838\u4e2d\u6709\u826f\u597d\u7684\u652f\u6301\u3002\u4f46\u662f\u5f53\u4f7f\u7528 UDP \u5c01\u88c5 TCP \u6570\u636e\u5305\u65f6\uff0c \u73b0\u4ee3\u64cd\u4f5c\u7cfb\u7edf\u548c\u7f51\u5361\u9488\u5bf9 TCP \u534f\u8bae\u7684\u4f18\u5316\u548c offload \u529f\u80fd\u5c06\u65e0\u6cd5\u987a\u5229\u5de5\u4f5c\uff0c\u5bfc\u81f4 TCP \u7684\u541e\u5410\u91cf\u51fa\u73b0\u663e\u8457\u4e0b\u964d\u3002\u5728\u865a\u62df\u5316\u573a\u666f\u4e0b\u7531\u4e8e CPU \u7684\u9650\u5236\uff0c TCP \u5927\u5305\u7684\u541e\u5410\u91cf\u751a\u81f3\u53ef\u80fd\u53ea\u6709\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u5341\u5206\u4e4b\u4e00\u3002 STT \u63d0\u4f9b\u4e86\u4e00\u79cd\u521b\u65b0\u5f0f\u7684\u4f7f\u7528 TCP \u683c\u5f0f\u6570\u636e\u5305\u8fdb\u884c\u5c01\u88c5\u7684\u96a7\u9053\u534f\u8bae\uff0c\u8be5\u5c01\u88c5\u53ea\u662f\u6a21\u62df\u4e86 TCP \u534f\u8bae\u7684\u5934\u90e8\u683c\u5f0f\uff0c\u5e76\u6ca1\u6709\u771f\u6b63\u5efa\u7acb TCP \u8fde\u63a5\uff0c\u4f46\u662f\u53ef\u4ee5 \u5145\u5206\u5229\u7528\u73b0\u4ee3\u64cd\u4f5c\u7cfb\u7edf\u548c\u7f51\u5361\u7684 TCP \u4f18\u5316\u80fd\u529b\u3002\u5728\u6211\u4eec\u7684\u6d4b\u8bd5\u4e2d TCP \u5927\u5305\u7684\u541e\u5410\u91cf\u80fd\u6709\u6570\u500d\u7684\u63d0\u5347\uff0c\u8fbe\u5230\u63a5\u8fd1\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u6027\u80fd\u6c34\u5e73\u3002 STT \u96a7\u9053\u5e76\u6ca1\u6709\u9884\u5b89\u88c5\u5728\u5185\u6838\u5185\uff0c\u9700\u8981\u901a\u8fc7\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u6765\u5b89\u88c5\uff0cOVS \u5185\u6838\u6a21\u5757\u7684\u7f16\u8bd1\u65b9\u6cd5\u53ef\u4ee5\u53c2\u8003\u4e0a\u4e00\u8282\u3002 STT \u96a7\u9053\u5f00\u542f\uff1a kubectl set env daemonset/ovs-ovn -n kube-system TUNNEL_TYPE = stt kubectl delete pod -n kube-system -lapp = ovs \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6027\u80fd\u8c03\u4f18"},{"location":"advance/performance-tuning/#_1","text":"\u4e3a\u4e86\u4fdd\u6301\u5b89\u88c5\u7684\u7b80\u5355\u548c\u529f\u80fd\u7684\u5b8c\u5907\uff0cKube-OVN \u7684\u9ed8\u8ba4\u5b89\u88c5\u811a\u672c\u5e76\u6ca1\u6709\u5bf9\u6027\u80fd\u9488\u5bf9\u6027\u7684\u4f18\u5316\u3002\u5982\u679c\u5e94\u7528\u5bf9\u5ef6\u8fdf\u548c\u541e\u5410\u91cf\u654f\u611f\uff0c \u7ba1\u7406\u5458\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u5bf9\u6027\u80fd\u8fdb\u884c\u9488\u5bf9\u6027\u4f18\u5316\u3002 \u793e\u533a\u4f1a\u4e0d\u65ad\u8fed\u4ee3\u63a7\u5236\u9762\u677f\u548c\u4f18\u5316\u9762\u7684\u6027\u80fd\uff0c\u90e8\u5206\u901a\u7528\u6027\u80fd\u4f18\u5316\u5df2\u7ecf\u96c6\u6210\u5230\u6700\u65b0\u7248\u672c\uff0c\u5efa\u8bae\u4f7f\u7528\u6700\u65b0\u7248\u672c\u83b7\u5f97\u66f4\u597d\u7684\u9ed8\u8ba4\u6027\u80fd\u3002 \u66f4\u591a\u5173\u4e8e\u6027\u80fd\u4f18\u5316\u7684\u8fc7\u7a0b\u548c\u65b9\u6cd5\u8bba\uff0c\u53ef\u4ee5\u89c2\u770b\u89c6\u9891\u5206\u4eab\uff1a Kube-OVN \u5bb9\u5668\u6027\u80fd\u4f18\u5316\u4e4b\u65c5 \u3002","title":"\u6027\u80fd\u8c03\u4f18"},{"location":"advance/performance-tuning/#_2","text":"\u7531\u4e8e\u8f6f\u786c\u4ef6\u73af\u5883\u7684\u5dee\u5f02\u6781\u5927\uff0c\u8fd9\u91cc\u63d0\u4f9b\u7684\u6027\u80fd\u6d4b\u8bd5\u6570\u636e\u53ea\u80fd\u4f5c\u4e3a\u53c2\u8003\uff0c\u5b9e\u9645\u6d4b\u8bd5\u7ed3\u679c\u4f1a\u548c\u672c\u6587\u6863\u4e2d\u7684\u7ed3\u679c\u5b58\u5728\u8f83\u5927\u5dee\u5f02\u3002 \u5efa\u8bae\u6bd4\u8f83\u4f18\u5316\u524d\u540e\u7684\u6027\u80fd\u6d4b\u8bd5\u7ed3\u679c\uff0c\u548c\u5bbf\u4e3b\u673a\u7f51\u7edc\u548c\u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u6bd4\u8f83\u3002","title":"\u57fa\u51c6\u6d4b\u8bd5"},{"location":"advance/performance-tuning/#overlay","text":"\u73af\u5883\u4fe1\u606f\uff1a Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 Overlay \u6a21\u5f0f CPU: Intel(R) Xeon(R) E-2278G Network: 2*10Gbps, xmit_hash_policy=layer3+4 \u6211\u4eec\u4f7f\u7528 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw \u6d4b\u8bd5 1 \u5b57\u8282\u5c0f\u5305\u4e0b tcp/udp \u7684\u5e26\u5bbd\u548c\u5ef6\u8fdf\uff0c\u5206\u522b\u6d4b\u8bd5\u4f18\u5316\u524d\uff0c\u4f18\u5316\u540e\u4ee5\u53ca\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u6027\u80fd\uff1a Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Default 25.7 22.9 27.1 1.59 Kube-OVN Optimized 13.9 12.9 27.6 5.57 HOST Network 13.1 12.4 28.2 6.02","title":"Overlay \u4f18\u5316\u524d\u540e\u6027\u80fd\u5bf9\u6bd4"},{"location":"advance/performance-tuning/#overlay-underlay-calico","text":"\u4e0b\u9762\u6211\u4eec\u4f1a\u6bd4\u8f83\u4f18\u5316\u540e Kube-OVN \u5728\u4e0d\u540c\u5305\u5927\u5c0f\u4e0b\u7684 Overlay \u548c Underlay \u6027\u80fd\uff0c\u5e76\u548c Calico \u7684 IPIP Always , IPIP never \u4ee5\u53ca\u5bbf\u4e3b\u673a\u7f51\u7edc\u505a\u6bd4\u8f83\u3002 Environment : Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 CPU: AMD EPYC 7402P 24-Core Processor Network: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Overlay 15.2 14.6 23.6 2.65 Kube-OVN Underlay 14.3 13.8 24.2 3.46 Calico IPIP 21.4 20.2 23.6 1.18 Calico NoEncap 19.3 16.9 23.6 1.76 HOST Network 16.6 15.4 24.8 2.64 qperf -t 60 <server ip> -ub -oo msg_size:1K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 16.5 15.8 10.2 2.77 Kube-OVN Underlay 15.9 14.5 9.6 3.22 Calico IPIP 22.5 21.5 1.45 1.14 Calico NoEncap 19.4 18.3 3.76 1.63 HOST Network 18.1 16.6 9.32 2.66 qperf -t 60 <server ip> -ub -oo msg_size:4K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 34.7 41.6 16.0 9.23 Kube-OVN Underlay 32.6 44 15.1 6.71 Calico IPIP 44.8 52.9 2.94 3.26 Calico NoEncap 40 49.6 6.56 4.19 HOST Network 35.9 45.9 14.6 5.59 \u5728\u90e8\u5206\u60c5\u51b5\u4e0b\u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u4f1a\u4f18\u4e8e\u5bbf\u4e3b\u673a\u7f51\u7edc\uff0c\u8fd9\u662f\u4f18\u4e8e\u7ecf\u8fc7\u4f18\u5316\u540e\u5bb9\u5668\u7f51\u7edc\u8def\u5f84\u5b8c\u5168\u7ed5\u8fc7\u4e86 netfilter\uff0c \u800c\u5bbf\u4e3b\u673a\u7f51\u7edc\u7531\u4e8e kube-proxy \u7684\u5b58\u5728\u6240\u6709\u6570\u636e\u5305\u5747\u9700\u7ecf\u8fc7 netfilter\uff0c\u4f1a\u5bfc\u81f4\u5728\u4e00\u4e9b\u73af\u5883\u4e0b\u5bb9\u5668\u7f51\u7edc \u7684\u6d88\u8017\u76f8\u5bf9\u66f4\u5c0f\uff0c\u56e0\u6b64\u4f1a\u6709\u66f4\u597d\u7684\u6027\u80fd\u8868\u73b0\u3002","title":"Overlay\uff0c Underlay \u4ee5\u53ca Calico \u4e0d\u540c\u6a21\u5f0f\u6027\u80fd\u5bf9\u6bd4"},{"location":"advance/performance-tuning/#_3","text":"\u8fd9\u91cc\u4ecb\u7ecd\u7684\u4f18\u5316\u65b9\u6cd5\u548c\u8f6f\u786c\u4ef6\u73af\u5883\u4ee5\u53ca\u6240\u9700\u8981\u7684\u529f\u80fd\u76f8\u5173\uff0c\u8bf7\u4ed4\u7ec6\u4e86\u89e3\u4f18\u5316\u7684\u524d\u63d0\u6761\u4ef6\u518d\u8fdb\u884c\u5c1d\u8bd5\u3002","title":"\u6570\u636e\u5e73\u9762\u6027\u80fd\u4f18\u5316\u65b9\u6cd5"},{"location":"advance/performance-tuning/#cpu","text":"\u90e8\u5206\u73af\u5883\u4e0b CPU \u8fd0\u884c\u5728\u8282\u80fd\u6a21\u5f0f\uff0c\u8be5\u6a21\u5f0f\u4e0b\u6027\u80fd\u8868\u73b0\u5c06\u4f1a\u4e0d\u7a33\u5b9a\uff0c\u5ef6\u8fdf\u4f1a\u51fa\u73b0\u660e\u663e\u589e\u52a0\uff0c\u5efa\u8bae\u4f7f\u7528 CPU \u7684\u6027\u80fd\u6a21\u5f0f\u83b7\u5f97\u66f4\u7a33\u5b9a\u7684\u6027\u80fd\u8868\u73b0\uff1a cpupower frequency-set -g performance","title":"CPU \u6027\u80fd\u6a21\u5f0f\u8c03\u6574"},{"location":"advance/performance-tuning/#_4","text":"\u5728\u6d41\u91cf\u589e\u5927\u7684\u60c5\u51b5\u4e0b\uff0c\u7f13\u51b2\u961f\u5217\u8fc7\u77ed\u53ef\u80fd\u5bfc\u81f4\u8f83\u9ad8\u7684\u4e22\u5305\u7387\u5bfc\u81f4\u6027\u80fd\u663e\u8457\u4e0b\u964d\uff0c\u9700\u8981\u8fdb\u884c\u8c03\u6574 \u68c0\u67e5\u5f53\u524d\u7f51\u5361\u961f\u5217\u957f\u5ea6\uff1a # ethtool -g eno1 Ring parameters for eno1: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 255 RX Mini: 0 RX Jumbo: 0 TX: 255 \u589e\u52a0\u961f\u5217\u957f\u5ea6\u81f3\u6700\u5927\u503c\uff1a ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096","title":"\u7f51\u5361\u786c\u4ef6\u961f\u5217\u8c03\u6574"},{"location":"advance/performance-tuning/#tuned","text":"tuned \u53ef\u4ee5\u4f7f\u7528\u4e00\u7cfb\u5217\u9884\u7f6e\u7684 profile \u6587\u4ef6\u4fdd\u5b58\u4e86\u9488\u5bf9\u7279\u5b9a\u573a\u666f\u7684\u4e00\u7cfb\u5217\u7cfb\u7edf\u4f18\u5316\u914d\u7f6e\u3002 \u9488\u5bf9\u5ef6\u8fdf\u4f18\u5148\u573a\u666f\uff1a tuned-adm profile network-latency \u9488\u5bf9\u541e\u5410\u91cf\u4f18\u5148\u573a\u666f\uff1a tuned-adm profile network-throughput","title":"\u4f7f\u7528 tuned \u4f18\u5316\u7cfb\u7edf\u53c2\u6570"},{"location":"advance/performance-tuning/#_5","text":"\u6211\u4eec\u63a8\u8350\u7981\u7528 irqbalance \u5e76\u5c06\u7f51\u5361\u4e2d\u65ad\u548c\u7279\u5b9a CPU \u8fdb\u884c\u7ed1\u5b9a\uff0c\u6765\u907f\u514d\u5728\u591a\u4e2a CPU \u4e4b\u95f4\u5207\u6362\u5bfc\u81f4\u7684\u6027\u80fd\u6ce2\u52a8\u3002","title":"\u4e2d\u65ad\u7ed1\u5b9a"},{"location":"advance/performance-tuning/#ovn-lb","text":"OVN \u7684 L2 LB \u5b9e\u73b0\u8fc7\u7a0b\u4e2d\u9700\u8981\u8c03\u7528\u5185\u6838\u7684 conntrack \u6a21\u5757\u5e76\u8fdb\u884c recirculate \u5bfc\u81f4\u5927\u91cf\u7684 CPU \u5f00\u9500\uff0c\u7ecf\u6d4b\u8bd5\u8be5\u529f\u80fd\u4f1a\u5e26\u6765 20% \u5de6\u53f3\u7684 CPU \u5f00\u9500\uff0c \u5728 Overlay \u7f51\u7edc\u6a21\u5f0f\u4e0b\u53ef\u4ee5\u4f7f\u7528 kube-proxy \u5b8c\u6210 Service \u8f6c\u53d1\u529f\u80fd\uff0c\u83b7\u5f97\u66f4\u597d\u7684 Pod-to-Pod \u6027\u80fd\u3002\u53ef\u4ee5\u5728 kube-ovn-controller \u4e2d\u5173\u95ed\u8be5\u529f\u80fd\uff1a command : - /kube-ovn/start-controller.sh args : ... - --enable-lb=false ... Underlay \u6a21\u5f0f\u4e0b kube-proxy \u65e0\u6cd5\u4f7f\u7528 iptables \u6216 ipvs \u63a7\u5236\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\uff0c\u5982\u9700\u5173\u95ed LB \u529f\u80fd\u9700\u8981\u786e\u8ba4\u662f\u5426\u4e0d\u9700\u8981 Service \u529f\u80fd\u3002","title":"\u5173\u95ed OVN LB"},{"location":"advance/performance-tuning/#fastpath","text":"\u7531\u4e8e\u5bb9\u5668\u7f51\u7edc\u548c\u5bbf\u4e3b\u673a\u7f51\u7edc\u5728\u4e0d\u540c\u7684 network ns\uff0c\u6570\u636e\u5305\u5728\u8de8\u5bbf\u4e3b\u673a\u4f20\u8f93\u65f6\u4f1a\u591a\u6b21\u7ecf\u8fc7 netfilter \u6a21\u5757\uff0c\u4f1a\u5e26\u6765\u8fd1 20% \u7684 CPU \u5f00\u9500\u3002\u7531\u4e8e\u5927\u90e8\u5206\u60c5\u51b5\u4e0b \u5bb9\u5668\u7f51\u7edc\u5185\u5e94\u7528\u65e0\u987b\u4f7f\u7528 netfilter \u6a21\u5757\u7684\u529f\u80fd\uff0c FastPath \u6a21\u5757\u53ef\u4ee5\u7ed5\u8fc7 netfilter \u964d\u4f4e CPU \u5f00\u9500\u3002 \u5982\u5bb9\u5668\u7f51\u7edc\u5185\u9700\u8981\u4f7f\u7528 netfilter \u63d0\u4f9b\u7684\u529f\u80fd\u5982 iptables\uff0cipvs\uff0cnftables \u7b49\uff0c\u8be5\u6a21\u5757\u4f1a\u4f7f\u76f8\u5173\u529f\u80fd\u5931\u6548\u3002 \u7531\u4e8e\u5185\u6838\u6a21\u5757\u548c\u5185\u6838\u7248\u672c\u76f8\u5173\uff0c\u65e0\u6cd5\u63d0\u4f9b\u4e00\u4e2a\u5355\u4e00\u9002\u5e94\u6240\u6709\u5185\u6838\u7684\u5185\u6838\u6a21\u5757\u5236\u54c1\u3002\u6211\u4eec\u9884\u5148\u7f16\u8bd1\u4e86\u90e8\u5206\u5185\u6838\u7684 FastPath \u6a21\u5757\uff0c \u53ef\u4ee5\u524d\u5f80 tunning-package \u8fdb\u884c\u4e0b\u8f7d\u3002 \u4e5f\u53ef\u4ee5\u624b\u52a8\u8fdb\u884c\u7f16\u8bd1\uff0c\u65b9\u6cd5\u53c2\u8003 \u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757 \u83b7\u5f97\u5185\u6838\u6a21\u5757\u540e\u53ef\u5728\u6bcf\u4e2a\u8282\u70b9\u4f7f\u7528 insmod kube_ovn_fastpath.ko \u52a0\u8f7d FastPath \u6a21\u5757\uff0c\u5e76\u4f7f\u7528 dmesg \u9a8c\u8bc1\u6a21\u5757\u52a0\u8f7d\u6210\u529f\uff1a # dmesg ... [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in ...","title":"\u5185\u6838 FastPath \u6a21\u5757"},{"location":"advance/performance-tuning/#ovs","text":"OVS \u7684 flow \u5904\u7406\u5305\u62ec\u54c8\u5e0c\u8ba1\u7b97\uff0c\u5339\u914d\u7b49\u64cd\u4f5c\u4f1a\u6d88\u8017\u5927\u7ea6 10% \u5de6\u53f3\u7684 CPU \u8d44\u6e90\u3002\u73b0\u4ee3 x86 CPU \u4e0a\u7684\u4e00\u4e9b\u6307\u4ee4\u96c6\u4f8b\u5982 popcnt \u548c sse4.2 \u53ef\u4ee5 \u52a0\u901f\u76f8\u5173\u8ba1\u7b97\u8fc7\u7a0b\uff0c\u4f46\u5185\u6838\u9ed8\u8ba4\u7f16\u8bd1\u672a\u5f00\u542f\u76f8\u5173\u9009\u9879\u3002\u7ecf\u6d4b\u8bd5\u5728\u5f00\u542f\u76f8\u5e94\u6307\u4ee4\u96c6\u4f18\u5316\u540e\uff0cflow \u76f8\u5173\u64cd\u4f5c CPU \u6d88\u8017\u5c06\u4f1a\u964d\u81f3 5% \u5de6\u53f3\u3002 \u548c FastPath \u6a21\u5757\u7684\u7f16\u8bd1\u7c7b\u4f3c\uff0c\u7531\u4e8e\u5185\u6838\u6a21\u5757\u548c\u5185\u6838\u7248\u672c\u76f8\u5173\uff0c\u65e0\u6cd5\u63d0\u4f9b\u4e00\u4e2a\u5355\u4e00\u9002\u5e94\u6240\u6709\u5185\u6838\u7684\u5185\u6838\u6a21\u5757\u5236\u54c1\u3002\u7528\u6237\u9700\u8981\u624b\u52a8\u7f16\u8bd1\u6216\u8005 \u524d\u5f80 tunning-package \u67e5\u770b\u662f\u5426\u6709\u5df2\u7f16\u8bd1\u597d\u7684\u5236\u54c1\u8fdb\u884c\u4e0b\u8f7d\u3002 \u4f7f\u7528\u8be5\u5185\u6838\u6a21\u5757\u524d\u8bf7\u5148\u786e\u8ba4 CPU \u662f\u5426\u652f\u6301\u76f8\u5173\u6307\u4ee4\u96c6\uff1a cat /proc/cpuinfo | grep popcnt cat /proc/cpuinfo | grep sse4_2","title":"OVS \u5185\u6838\u6a21\u5757\u4f18\u5316"},{"location":"advance/performance-tuning/#centos","text":"\u5b89\u88c5\u76f8\u5173\u7f16\u8bd1\u4f9d\u8d56\u548c\u5185\u6838\u5934\u6587\u4ef6\uff1a yum install -y gcc kernel-devel- $( uname -r ) python3 autoconf automake libtool rpm-build openssl-devel \u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u5e76\u751f\u6210\u5bf9\u5e94 RPM \u6587\u4ef6: git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --with-linux = /lib/modules/ $( uname -r ) /build CFLAGS = \"-g -O2 -mpopcnt -msse4.2\" make rpm-fedora-kmod cd rpm/rpmbuild/RPMS/x86_64/ \u590d\u5236 RPM \u5230\u6bcf\u4e2a\u8282\u70b9\u5e76\u8fdb\u884c\u5b89\u88c5\uff1a rpm -i openvswitch-kmod-2.15.2-1.el7.x86_64.rpm \u82e5\u4e4b\u524d\u5df2\u7ecf\u542f\u52a8\u8fc7 Kube-OVN\uff0c\u65e7\u7248\u672c OVS \u6a21\u5757\u5df2\u52a0\u8f7d\u81f3\u5185\u6838\uff0c\u5efa\u8bae\u91cd\u542f\u673a\u5668\u91cd\u65b0\u52a0\u8f7d\u65b0\u7248\u5185\u6838\u6a21\u5757\u3002","title":"CentOS \u4e0b\u7f16\u8bd1\u5b89\u88c5"},{"location":"advance/performance-tuning/#ubuntu","text":"\u5b89\u88c5\u76f8\u5173\u7f16\u8bd1\u4f9d\u8d56\u548c\u5185\u6838\u5934\u6587\u4ef6\uff1a apt install -y autoconf automake libtool gcc build-essential libssl-dev \u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u5e76\u5b89\u88c5\uff1a apt install -y autoconf automake libtool gcc build-essential libssl-dev git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --prefix = /usr/ --localstatedir = /var --enable-ssl --with-linux = /lib/modules/ $( uname -r ) /build make -j ` nproc ` make install make modules_install cat > /etc/depmod.d/openvswitch.conf << EOF override openvswitch * extra override vport-* * extra EOF depmod -a cp debian/openvswitch-switch.init /etc/init.d/openvswitch-switch /etc/init.d/openvswitch-switch force-reload-kmod \u82e5\u4e4b\u524d\u5df2\u7ecf\u542f\u52a8\u8fc7 Kube-OVN\uff0c\u65e7\u7248\u672c OVS \u6a21\u5757\u5df2\u52a0\u8f7d\u81f3\u5185\u6838\uff0c\u5efa\u8bae\u91cd\u542f\u673a\u5668\u91cd\u65b0\u52a0\u8f7d\u65b0\u7248\u5185\u6838\u6a21\u5757\u3002","title":"Ubuntu \u4e0b\u7f16\u8bd1\u5b89\u88c5"},{"location":"advance/performance-tuning/#stt","text":"\u5e38\u89c1\u7684\u96a7\u9053\u5c01\u88c5\u534f\u8bae\u4f8b\u5982 Geneve \u548c Vxlan \u4f7f\u7528 UDP \u534f\u8bae\u5bf9\u6570\u636e\u5305\u8fdb\u884c\u5c01\u88c5\uff0c\u5728\u5185\u6838\u4e2d\u6709\u826f\u597d\u7684\u652f\u6301\u3002\u4f46\u662f\u5f53\u4f7f\u7528 UDP \u5c01\u88c5 TCP \u6570\u636e\u5305\u65f6\uff0c \u73b0\u4ee3\u64cd\u4f5c\u7cfb\u7edf\u548c\u7f51\u5361\u9488\u5bf9 TCP \u534f\u8bae\u7684\u4f18\u5316\u548c offload \u529f\u80fd\u5c06\u65e0\u6cd5\u987a\u5229\u5de5\u4f5c\uff0c\u5bfc\u81f4 TCP \u7684\u541e\u5410\u91cf\u51fa\u73b0\u663e\u8457\u4e0b\u964d\u3002\u5728\u865a\u62df\u5316\u573a\u666f\u4e0b\u7531\u4e8e CPU \u7684\u9650\u5236\uff0c TCP \u5927\u5305\u7684\u541e\u5410\u91cf\u751a\u81f3\u53ef\u80fd\u53ea\u6709\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u5341\u5206\u4e4b\u4e00\u3002 STT \u63d0\u4f9b\u4e86\u4e00\u79cd\u521b\u65b0\u5f0f\u7684\u4f7f\u7528 TCP \u683c\u5f0f\u6570\u636e\u5305\u8fdb\u884c\u5c01\u88c5\u7684\u96a7\u9053\u534f\u8bae\uff0c\u8be5\u5c01\u88c5\u53ea\u662f\u6a21\u62df\u4e86 TCP \u534f\u8bae\u7684\u5934\u90e8\u683c\u5f0f\uff0c\u5e76\u6ca1\u6709\u771f\u6b63\u5efa\u7acb TCP \u8fde\u63a5\uff0c\u4f46\u662f\u53ef\u4ee5 \u5145\u5206\u5229\u7528\u73b0\u4ee3\u64cd\u4f5c\u7cfb\u7edf\u548c\u7f51\u5361\u7684 TCP \u4f18\u5316\u80fd\u529b\u3002\u5728\u6211\u4eec\u7684\u6d4b\u8bd5\u4e2d TCP \u5927\u5305\u7684\u541e\u5410\u91cf\u80fd\u6709\u6570\u500d\u7684\u63d0\u5347\uff0c\u8fbe\u5230\u63a5\u8fd1\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u6027\u80fd\u6c34\u5e73\u3002 STT \u96a7\u9053\u5e76\u6ca1\u6709\u9884\u5b89\u88c5\u5728\u5185\u6838\u5185\uff0c\u9700\u8981\u901a\u8fc7\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u6765\u5b89\u88c5\uff0cOVS \u5185\u6838\u6a21\u5757\u7684\u7f16\u8bd1\u65b9\u6cd5\u53ef\u4ee5\u53c2\u8003\u4e0a\u4e00\u8282\u3002 STT \u96a7\u9053\u5f00\u542f\uff1a kubectl set env daemonset/ovs-ovn -n kube-system TUNNEL_TYPE = stt kubectl delete pod -n kube-system -lapp = ovs \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 STT \u7c7b\u578b\u96a7\u9053"},{"location":"advance/security-group/","text":"SecurityGroup \u4f7f\u7528 \u00b6 Kube-OVN \u652f\u6301\u4e86\u5b89\u5168\u7ec4\u7684\u914d\u7f6e\uff0c\u914d\u7f6e\u5b89\u5168\u7ec4\u4f7f\u7528\u7684 CRD \u4e3a SecurityGroup\u3002 \u5b89\u5168\u7ec4\u793a\u4f8b \u00b6 apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-example spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 1 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address \u5b89\u5168\u7ec4\u5404\u5b57\u6bb5\u7684\u5177\u4f53\u542b\u4e49\uff0c\u53ef\u4ee5\u53c2\u8003 Kube-OVN \u63a5\u53e3\u89c4\u8303 \u3002 Pod \u901a\u8fc7\u6dfb\u52a0 annotation \u6765\u7ed1\u5b9a\u5b89\u5168\u7ec4\uff0c\u4f7f\u7528\u7684 annotation \u6709\u4e24\u4e2a\uff1a ovn.kubernetes.io/port_security : \"true\" ovn.kubernetes.io/security_groups : sg-example \u6ce8\u610f\u4e8b\u9879 \u00b6 \u5b89\u5168\u7ec4\u6700\u540e\u662f\u901a\u8fc7\u8bbe\u7f6e ACL \u89c4\u5219\u6765\u9650\u5236\u8bbf\u95ee\u7684\uff0cOVN \u6587\u6863\u4e2d\u63d0\u5230\uff0c\u5982\u679c\u5339\u914d\u5230\u7684\u4e24\u4e2a ACL \u89c4\u5219\u62e5\u6709\u76f8\u540c\u7684\u4f18\u5148\u7ea7\uff0c\u5b9e\u9645\u8d77\u4f5c\u7528\u7684\u662f\u54ea\u4e2a ACL \u662f\u4e0d\u786e\u5b9a\u7684\u3002\u56e0\u6b64\u8bbe\u7f6e\u5b89\u5168\u7ec4\u89c4\u5219\u7684\u65f6\u5019\uff0c\u9700\u8981\u6ce8\u610f\u533a\u5206\u4f18\u5148\u7ea7\u3002 \u5f53\u6dfb\u52a0\u5b89\u5168\u7ec4\u7684\u65f6\u5019\uff0c\u8981\u6e05\u695a\u7684\u77e5\u9053\u662f\u5728\u6dfb\u52a0\u4ec0\u4e48\u9650\u5236\u3002Kube-OVN \u4f5c\u4e3a CNI\uff0c\u521b\u5efa Pod \u540e\u4f1a\u8fdb\u884c Pod \u5230\u7f51\u5173\u7684\u8fde\u901a\u6027\u6d4b\u8bd5\uff0c\u5982\u679c\u8bbf\u95ee\u4e0d\u901a\u7f51\u5173\uff0c\u5c31\u4f1a\u5bfc\u81f4 Pod \u4e00\u76f4\u5904\u4e8e ContainerCreating \u72b6\u6001\uff0c\u65e0\u6cd5\u987a\u5229\u5207\u6362\u5230 Running \u72b6\u6001\u3002 \u5b9e\u9645\u6d4b\u8bd5 \u00b6 \u5229\u7528\u4ee5\u4e0b yaml \u521b\u5efa Pod\uff0c\u5728 annotation \u4e2d\u6307\u5b9a\u7ed1\u5b9a\u793a\u4f8b\u4e2d\u7684\u5b89\u5168\u7ec4\uff1a apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-example' name : sg-test-pod namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u5b9e\u9645\u6d4b\u8bd5\u7ed3\u679c\u663e\u793a\u5982\u4e0b\uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h32m <none> kube-ovn-worker <none> <none> test-99fff7f86-52h9r 1 /1 Running 0 5h41m 10 .16.0.14 kube-ovn-control-plane <none> <none> test-99fff7f86-qcgjw 1 /1 Running 0 5h43m 10 .16.0.13 kube-ovn-worker <none> <none> \u6267\u884c kubectl describe pod \u67e5\u770b Pod \u7684\u4fe1\u606f\uff0c\u53ef\u4ee5\u770b\u5230\u62a5\u9519\u63d0\u793a\uff1a # kubectl describe pod sg-test-pod Name: sg-test-pod Namespace: default Priority: 0 Node: kube-ovn-worker/172.18.0.2 Start Time: Tue, 28 Feb 2023 10 :29:36 +0800 Labels: app = static Annotations: ovn.kubernetes.io/allocated: true ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.15 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:FA:17:97 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/port_security: true ovn.kubernetes.io/routed: true ovn.kubernetes.io/security_groups: sg-allow-reject Status: Pending IP: IPs: <none> \u00b7 \u00b7 \u00b7 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 5m3s ( x70 over 4h59m ) kubelet ( combined from similar events ) : Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"40636e0c7f1ade5500fa958486163d74f2e2300051a71522a9afd7ba0538afb6\" : plugin type = \"kube-ovn\" failed ( add ) : RPC failed ; request ip return 500 configure nic failed 10 .16.0.15 network not ready after 200 ping 10 .16.0.1 \u4fee\u6539\u5b89\u5168\u7ec4\u7684\u89c4\u5219\uff0c\u6dfb\u52a0\u5230\u7f51\u5173\u7684\u8bbf\u95ee\u89c4\u5219\uff0c\u53c2\u8003\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-gw-both spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 2 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.1/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 2 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : icmp remoteAddress : 10.16.0.1/16 remoteType : address \u5206\u522b\u5728\u5165\u65b9\u5411\u548c\u51fa\u65b9\u5411\u89c4\u5219\u4e2d\uff0c\u6dfb\u52a0\u5141\u8bb8\u5230\u7f51\u5173\u7684\u8bbf\u95ee\u89c4\u5219\uff0c\u5e76\u4e14\u8bbe\u7f6e\u8be5\u89c4\u5219\u7684\u4f18\u5148\u7ea7\u6700\u9ad8\u3002 \u5229\u7528\u4ee5\u4e0b yaml \u7ed1\u5b9a\u5b89\u5168\u7ec4\uff0c\u90e8\u7f72 Pod \u540e\uff0c\u786e\u8ba4 Pod \u53ef\u4ee5\u6b63\u5e38\u8fd0\u884c\uff1a apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-gw-both' name : sg-gw-both namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u90e8\u7f72\u540e\u67e5\u770b Pod \u4fe1\u606f\uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h41m <none> kube-ovn-worker <none> <none> sg-gw-both 1 /1 Running 0 5h37m 10 .16.0.19 kube-ovn-worker <none> <none> \u56e0\u6b64\u5bf9\u4e8e\u5b89\u5168\u7ec4\u7684\u4f7f\u7528\uff0c\u8981\u7279\u522b\u660e\u786e\u6dfb\u52a0\u7684\u9650\u5236\u89c4\u5219\u7684\u4f5c\u7528\u3002\u5982\u679c\u5355\u7eaf\u662f\u9650\u5236\u6d41\u91cf\u8bbf\u95ee\uff0c\u53ef\u4ee5\u8003\u8651\u4f7f\u7528\u7f51\u7edc\u7b56\u7565\u5b9e\u73b0\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"SecurityGroup \u4f7f\u7528"},{"location":"advance/security-group/#securitygroup","text":"Kube-OVN \u652f\u6301\u4e86\u5b89\u5168\u7ec4\u7684\u914d\u7f6e\uff0c\u914d\u7f6e\u5b89\u5168\u7ec4\u4f7f\u7528\u7684 CRD \u4e3a SecurityGroup\u3002","title":"SecurityGroup \u4f7f\u7528"},{"location":"advance/security-group/#_1","text":"apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-example spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 1 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address \u5b89\u5168\u7ec4\u5404\u5b57\u6bb5\u7684\u5177\u4f53\u542b\u4e49\uff0c\u53ef\u4ee5\u53c2\u8003 Kube-OVN \u63a5\u53e3\u89c4\u8303 \u3002 Pod \u901a\u8fc7\u6dfb\u52a0 annotation \u6765\u7ed1\u5b9a\u5b89\u5168\u7ec4\uff0c\u4f7f\u7528\u7684 annotation \u6709\u4e24\u4e2a\uff1a ovn.kubernetes.io/port_security : \"true\" ovn.kubernetes.io/security_groups : sg-example","title":"\u5b89\u5168\u7ec4\u793a\u4f8b"},{"location":"advance/security-group/#_2","text":"\u5b89\u5168\u7ec4\u6700\u540e\u662f\u901a\u8fc7\u8bbe\u7f6e ACL \u89c4\u5219\u6765\u9650\u5236\u8bbf\u95ee\u7684\uff0cOVN \u6587\u6863\u4e2d\u63d0\u5230\uff0c\u5982\u679c\u5339\u914d\u5230\u7684\u4e24\u4e2a ACL \u89c4\u5219\u62e5\u6709\u76f8\u540c\u7684\u4f18\u5148\u7ea7\uff0c\u5b9e\u9645\u8d77\u4f5c\u7528\u7684\u662f\u54ea\u4e2a ACL \u662f\u4e0d\u786e\u5b9a\u7684\u3002\u56e0\u6b64\u8bbe\u7f6e\u5b89\u5168\u7ec4\u89c4\u5219\u7684\u65f6\u5019\uff0c\u9700\u8981\u6ce8\u610f\u533a\u5206\u4f18\u5148\u7ea7\u3002 \u5f53\u6dfb\u52a0\u5b89\u5168\u7ec4\u7684\u65f6\u5019\uff0c\u8981\u6e05\u695a\u7684\u77e5\u9053\u662f\u5728\u6dfb\u52a0\u4ec0\u4e48\u9650\u5236\u3002Kube-OVN \u4f5c\u4e3a CNI\uff0c\u521b\u5efa Pod \u540e\u4f1a\u8fdb\u884c Pod \u5230\u7f51\u5173\u7684\u8fde\u901a\u6027\u6d4b\u8bd5\uff0c\u5982\u679c\u8bbf\u95ee\u4e0d\u901a\u7f51\u5173\uff0c\u5c31\u4f1a\u5bfc\u81f4 Pod \u4e00\u76f4\u5904\u4e8e ContainerCreating \u72b6\u6001\uff0c\u65e0\u6cd5\u987a\u5229\u5207\u6362\u5230 Running \u72b6\u6001\u3002","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"advance/security-group/#_3","text":"\u5229\u7528\u4ee5\u4e0b yaml \u521b\u5efa Pod\uff0c\u5728 annotation \u4e2d\u6307\u5b9a\u7ed1\u5b9a\u793a\u4f8b\u4e2d\u7684\u5b89\u5168\u7ec4\uff1a apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-example' name : sg-test-pod namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u5b9e\u9645\u6d4b\u8bd5\u7ed3\u679c\u663e\u793a\u5982\u4e0b\uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h32m <none> kube-ovn-worker <none> <none> test-99fff7f86-52h9r 1 /1 Running 0 5h41m 10 .16.0.14 kube-ovn-control-plane <none> <none> test-99fff7f86-qcgjw 1 /1 Running 0 5h43m 10 .16.0.13 kube-ovn-worker <none> <none> \u6267\u884c kubectl describe pod \u67e5\u770b Pod \u7684\u4fe1\u606f\uff0c\u53ef\u4ee5\u770b\u5230\u62a5\u9519\u63d0\u793a\uff1a # kubectl describe pod sg-test-pod Name: sg-test-pod Namespace: default Priority: 0 Node: kube-ovn-worker/172.18.0.2 Start Time: Tue, 28 Feb 2023 10 :29:36 +0800 Labels: app = static Annotations: ovn.kubernetes.io/allocated: true ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.15 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:FA:17:97 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/port_security: true ovn.kubernetes.io/routed: true ovn.kubernetes.io/security_groups: sg-allow-reject Status: Pending IP: IPs: <none> \u00b7 \u00b7 \u00b7 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 5m3s ( x70 over 4h59m ) kubelet ( combined from similar events ) : Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"40636e0c7f1ade5500fa958486163d74f2e2300051a71522a9afd7ba0538afb6\" : plugin type = \"kube-ovn\" failed ( add ) : RPC failed ; request ip return 500 configure nic failed 10 .16.0.15 network not ready after 200 ping 10 .16.0.1 \u4fee\u6539\u5b89\u5168\u7ec4\u7684\u89c4\u5219\uff0c\u6dfb\u52a0\u5230\u7f51\u5173\u7684\u8bbf\u95ee\u89c4\u5219\uff0c\u53c2\u8003\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-gw-both spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 2 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.1/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 2 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : icmp remoteAddress : 10.16.0.1/16 remoteType : address \u5206\u522b\u5728\u5165\u65b9\u5411\u548c\u51fa\u65b9\u5411\u89c4\u5219\u4e2d\uff0c\u6dfb\u52a0\u5141\u8bb8\u5230\u7f51\u5173\u7684\u8bbf\u95ee\u89c4\u5219\uff0c\u5e76\u4e14\u8bbe\u7f6e\u8be5\u89c4\u5219\u7684\u4f18\u5148\u7ea7\u6700\u9ad8\u3002 \u5229\u7528\u4ee5\u4e0b yaml \u7ed1\u5b9a\u5b89\u5168\u7ec4\uff0c\u90e8\u7f72 Pod \u540e\uff0c\u786e\u8ba4 Pod \u53ef\u4ee5\u6b63\u5e38\u8fd0\u884c\uff1a apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-gw-both' name : sg-gw-both namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u90e8\u7f72\u540e\u67e5\u770b Pod \u4fe1\u606f\uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h41m <none> kube-ovn-worker <none> <none> sg-gw-both 1 /1 Running 0 5h37m 10 .16.0.19 kube-ovn-worker <none> <none> \u56e0\u6b64\u5bf9\u4e8e\u5b89\u5168\u7ec4\u7684\u4f7f\u7528\uff0c\u8981\u7279\u522b\u660e\u786e\u6dfb\u52a0\u7684\u9650\u5236\u89c4\u5219\u7684\u4f5c\u7528\u3002\u5982\u679c\u5355\u7eaf\u662f\u9650\u5236\u6d41\u91cf\u8bbf\u95ee\uff0c\u53ef\u4ee5\u8003\u8651\u4f7f\u7528\u7f51\u7edc\u7b56\u7565\u5b9e\u73b0\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b9e\u9645\u6d4b\u8bd5"},{"location":"advance/vip/","text":"VIP \u9884\u7559\u8bbe\u7f6e \u00b6 \u5728\u4e00\u4e9b\u573a\u666f\u4e0b\u6211\u4eec\u5e0c\u671b\u52a8\u6001\u7684\u9884\u7559\u4e00\u90e8\u5206 IP \u4f46\u662f\u5e76\u4e0d\u5206\u914d\u7ed9 Pod \u800c\u662f\u5206\u914d\u7ed9\u5176\u4ed6\u7684\u57fa\u7840\u8bbe\u65bd\u542f\u7528\uff0c\u4f8b\u5982\uff1a Kubernetes \u5d4c\u5957 Kubernetes \u7684\u573a\u666f\u4e2d\u4e0a\u5c42 Kubernetes \u4f7f\u7528 Underlay \u7f51\u7edc\u4f1a\u5360\u7528\u5e95\u5c42 Subnet \u53ef\u7528\u5730\u5740\u3002 LB \u6216\u5176\u4ed6\u7f51\u7edc\u57fa\u7840\u8bbe\u65bd\u9700\u8981\u4f7f\u7528\u4e00\u4e2a Subnet \u5185\u7684 IP\uff0c\u4f46\u4e0d\u4f1a\u5355\u72ec\u8d77 Pod\u3002 \u521b\u5efa\u968f\u673a\u5730\u5740 VIP \u00b6 \u5982\u679c\u53ea\u662f\u4e3a\u4e86\u9884\u7559\u82e5\u5e72 IP \u800c\u5bf9 IP \u5730\u5740\u672c\u8eab\u6ca1\u6709\u8981\u6c42\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684 yaml \u8fdb\u884c\u521b\u5efa\uff1a apiVersion : kubeovn.io/v1 kind : Vip metadata : name : vip-dynamic-01 spec : subnet : ovn-default type : \"\" subnet : \u5c06\u4ece\u8be5 Subnet \u4e2d\u9884\u7559 IP\u3002 type : \u76ee\u524d\u652f\u6301\u4e24\u79cd\u7c7b\u578b\uff0c\u4e3a\u7a7a\u8868\u793a\u4ec5\u7528\u4e8e ipam ip \u5360\u4f4d\uff0c switch_lb_vip \u8868\u793a\u8be5 vip \u4ec5\u7528\u4e8e switch lb \u524d\u7aef vip \u548c\u540e\u7aef ip \u9700\u5904\u4e8e\u540c\u4e00\u5b50\u7f51\u3002 \u521b\u5efa\u6210\u529f\u540e\u67e5\u8be2\u8be5 VIP\uff1a # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY vip-dynamic-01 10 .16.0.12 00 :00:00:F0:DB:25 ovn-default true \u53ef\u89c1\u8be5 VIP \u88ab\u5206\u914d\u4e86 10.16.0.12 \u7684 IP \u5730\u5740\uff0c\u8be5\u5730\u5740\u53ef\u4ee5\u4e4b\u540e\u4f9b\u5176\u4ed6\u7f51\u7edc\u57fa\u7840\u8bbe\u65bd\u4f7f\u7528\u3002 \u521b\u5efa\u56fa\u5b9a\u5730\u5740 VIP \u00b6 \u5982\u5bf9\u9884\u7559\u7684 VIP \u7684 IP \u5730\u5740\u6709\u9700\u6c42\u53ef\u4f7f\u7528\u4e0b\u9762\u7684 yaml \u8fdb\u884c\u56fa\u5b9a\u5206\u914d\uff1a apiVersion : kubeovn.io/v1 kind : Vip metadata : name : static-vip01 spec : subnet : ovn-default v4ip : \"10.16.0.121\" subnet : \u5c06\u4ece\u8be5 Subnet \u4e2d\u9884\u7559 IP\u3002 v4ip : \u56fa\u5b9a\u5206\u914d\u7684 IP \u5730\u5740\uff0c\u8be5\u5730\u5740\u9700\u5728 subnet \u7684 CIDR \u8303\u56f4\u5185\u3002 \u521b\u5efa\u6210\u529f\u540e\u67e5\u8be2\u8be5 VIP\uff1a # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY static-vip01 10 .16.0.121 00 :00:00:F0:DB:26 ovn-default true \u53ef\u89c1\u8be5 VIP \u88ab\u5206\u914d\u4e86\u6240\u9884\u671f\u7684 IP \u5730\u5740\u3002 Pod \u4f7f\u7528 VIP \u6765\u56fa\u5b9a IP \u00b6 \u8be5\u529f\u80fd\u76ee\u524d\u53ea\u5728 master \u5206\u652f\u652f\u6301\u3002 \u53ef\u4ee5\u4f7f\u7528 annotation \u5c06\u67d0\u4e2a VIP \u5206\u914d\u7ed9\u4e00\u4e2a Pod\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip annotations : ovn.kubernetes.io/vip : vip-dynamic-01 # \u6307\u5b9a vip namespace : default spec : containers : - name : static-ip image : docker.io/library/nginx:alpine StatefulSet \u548c Kubevirt VM \u4fdd\u7559 VIP \u00b6 \u9488\u5bf9 StatefulSet \u548c VM \u7684\u7279\u6b8a\u6027\uff0c\u5728\u4ed6\u4eec\u7684 Pod \u9500\u6bc1\u518d\u62c9\u8d77\u8d77\u540e\u4f1a\u91cd\u65b0\u4f7f\u7528\u4e4b\u524d\u8bbe\u7f6e\u7684 VIP\u3002 VM \u4fdd\u7559 VIP \u9700\u8981\u786e\u4fdd kube-ovn-controller \u7684 keep-vm-ip \u53c2\u6570\u4e3a true \u3002\u8bf7\u53c2\u8003 Kubevirt VM \u56fa\u5b9a\u5730\u5740\u5f00\u542f\u8bbe\u7f6e \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VIP \u9884\u7559\u8bbe\u7f6e"},{"location":"advance/vip/#vip","text":"\u5728\u4e00\u4e9b\u573a\u666f\u4e0b\u6211\u4eec\u5e0c\u671b\u52a8\u6001\u7684\u9884\u7559\u4e00\u90e8\u5206 IP \u4f46\u662f\u5e76\u4e0d\u5206\u914d\u7ed9 Pod \u800c\u662f\u5206\u914d\u7ed9\u5176\u4ed6\u7684\u57fa\u7840\u8bbe\u65bd\u542f\u7528\uff0c\u4f8b\u5982\uff1a Kubernetes \u5d4c\u5957 Kubernetes \u7684\u573a\u666f\u4e2d\u4e0a\u5c42 Kubernetes \u4f7f\u7528 Underlay \u7f51\u7edc\u4f1a\u5360\u7528\u5e95\u5c42 Subnet \u53ef\u7528\u5730\u5740\u3002 LB \u6216\u5176\u4ed6\u7f51\u7edc\u57fa\u7840\u8bbe\u65bd\u9700\u8981\u4f7f\u7528\u4e00\u4e2a Subnet \u5185\u7684 IP\uff0c\u4f46\u4e0d\u4f1a\u5355\u72ec\u8d77 Pod\u3002","title":"VIP \u9884\u7559\u8bbe\u7f6e"},{"location":"advance/vip/#vip_1","text":"\u5982\u679c\u53ea\u662f\u4e3a\u4e86\u9884\u7559\u82e5\u5e72 IP \u800c\u5bf9 IP \u5730\u5740\u672c\u8eab\u6ca1\u6709\u8981\u6c42\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684 yaml \u8fdb\u884c\u521b\u5efa\uff1a apiVersion : kubeovn.io/v1 kind : Vip metadata : name : vip-dynamic-01 spec : subnet : ovn-default type : \"\" subnet : \u5c06\u4ece\u8be5 Subnet \u4e2d\u9884\u7559 IP\u3002 type : \u76ee\u524d\u652f\u6301\u4e24\u79cd\u7c7b\u578b\uff0c\u4e3a\u7a7a\u8868\u793a\u4ec5\u7528\u4e8e ipam ip \u5360\u4f4d\uff0c switch_lb_vip \u8868\u793a\u8be5 vip \u4ec5\u7528\u4e8e switch lb \u524d\u7aef vip \u548c\u540e\u7aef ip \u9700\u5904\u4e8e\u540c\u4e00\u5b50\u7f51\u3002 \u521b\u5efa\u6210\u529f\u540e\u67e5\u8be2\u8be5 VIP\uff1a # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY vip-dynamic-01 10 .16.0.12 00 :00:00:F0:DB:25 ovn-default true \u53ef\u89c1\u8be5 VIP \u88ab\u5206\u914d\u4e86 10.16.0.12 \u7684 IP \u5730\u5740\uff0c\u8be5\u5730\u5740\u53ef\u4ee5\u4e4b\u540e\u4f9b\u5176\u4ed6\u7f51\u7edc\u57fa\u7840\u8bbe\u65bd\u4f7f\u7528\u3002","title":"\u521b\u5efa\u968f\u673a\u5730\u5740 VIP"},{"location":"advance/vip/#vip_2","text":"\u5982\u5bf9\u9884\u7559\u7684 VIP \u7684 IP \u5730\u5740\u6709\u9700\u6c42\u53ef\u4f7f\u7528\u4e0b\u9762\u7684 yaml \u8fdb\u884c\u56fa\u5b9a\u5206\u914d\uff1a apiVersion : kubeovn.io/v1 kind : Vip metadata : name : static-vip01 spec : subnet : ovn-default v4ip : \"10.16.0.121\" subnet : \u5c06\u4ece\u8be5 Subnet \u4e2d\u9884\u7559 IP\u3002 v4ip : \u56fa\u5b9a\u5206\u914d\u7684 IP \u5730\u5740\uff0c\u8be5\u5730\u5740\u9700\u5728 subnet \u7684 CIDR \u8303\u56f4\u5185\u3002 \u521b\u5efa\u6210\u529f\u540e\u67e5\u8be2\u8be5 VIP\uff1a # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY static-vip01 10 .16.0.121 00 :00:00:F0:DB:26 ovn-default true \u53ef\u89c1\u8be5 VIP \u88ab\u5206\u914d\u4e86\u6240\u9884\u671f\u7684 IP \u5730\u5740\u3002","title":"\u521b\u5efa\u56fa\u5b9a\u5730\u5740 VIP"},{"location":"advance/vip/#pod-vip-ip","text":"\u8be5\u529f\u80fd\u76ee\u524d\u53ea\u5728 master \u5206\u652f\u652f\u6301\u3002 \u53ef\u4ee5\u4f7f\u7528 annotation \u5c06\u67d0\u4e2a VIP \u5206\u914d\u7ed9\u4e00\u4e2a Pod\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip annotations : ovn.kubernetes.io/vip : vip-dynamic-01 # \u6307\u5b9a vip namespace : default spec : containers : - name : static-ip image : docker.io/library/nginx:alpine","title":"Pod \u4f7f\u7528 VIP \u6765\u56fa\u5b9a IP"},{"location":"advance/vip/#statefulset-kubevirt-vm-vip","text":"\u9488\u5bf9 StatefulSet \u548c VM \u7684\u7279\u6b8a\u6027\uff0c\u5728\u4ed6\u4eec\u7684 Pod \u9500\u6bc1\u518d\u62c9\u8d77\u8d77\u540e\u4f1a\u91cd\u65b0\u4f7f\u7528\u4e4b\u524d\u8bbe\u7f6e\u7684 VIP\u3002 VM \u4fdd\u7559 VIP \u9700\u8981\u786e\u4fdd kube-ovn-controller \u7684 keep-vm-ip \u53c2\u6570\u4e3a true \u3002\u8bf7\u53c2\u8003 Kubevirt VM \u56fa\u5b9a\u5730\u5740\u5f00\u542f\u8bbe\u7f6e \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"StatefulSet \u548c Kubevirt VM \u4fdd\u7559 VIP"},{"location":"advance/vpc-dns/","text":"\u81ea\u5b9a\u4e49 VPC DNS \u00b6 \u7531\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u548c \u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0c\u81ea\u5b9a VPC \u5185\u65e0\u6cd5\u8bbf\u95ee\u5230\u90e8\u7f72\u5728\u9ed8\u8ba4 VPC \u5185\u7684 coredns\u3002 \u5982\u679c\u7528\u6237\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 Kubernetes \u63d0\u4f9b\u7684\u96c6\u7fa4\u5185\u57df\u540d\u89e3\u6790\u80fd\u529b\uff0c\u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\uff0c\u5229\u7528 vpc-dns CRD \u6765\u5b9e\u73b0\u3002 \u8be5 CRD \u6700\u7ec8\u4f1a\u90e8\u7f72\u4e00\u4e2a coredns\uff0c\u8be5 Pod \u6709\u4e24\u4e2a\u7f51\u5361\uff0c\u4e00\u4e2a\u7f51\u5361\u5728\u7528\u6237\u81ea\u5b9a\u4e49 VPC\uff0c\u53e6\u4e00\u4e2a\u7f51\u5361\u5728\u9ed8\u8ba4 VPC \u4ece\u800c\u5b9e\u73b0\u7f51\u7edc\u4e92\u901a\uff0c\u540c\u65f6\u901a\u8fc7 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u63d0\u4f9b\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u4e00\u4e2a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u3002 \u90e8\u7f72 vpc-dns \u6240\u4f9d\u8d56\u7684\u8d44\u6e90 \u00b6 apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u9664\u4e86\u4ee5\u4e0a\u8d44\u6e90\uff0c\u8be5\u529f\u80fd\u8fd8\u4f9d\u8d56 nat-gw-pod \u955c\u50cf\u8fdb\u884c\u8def\u7531\u914d\u7f6e\u3002 \u914d\u7f6e\u9644\u52a0\u7f51\u5361 \u00b6 apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }' \u4fee\u6539 ovn-default \u5b50\u7f51\u7684 provider \u00b6 \u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn # \u53ea\u9700\u4fee\u6539\u8be5\u5b57\u6bb5 vpc : ovn-cluster \u914d\u7f6e vpc-dns \u7684 Configmap \u00b6 \u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : \"true\" nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\u662f\u5426\u542f\u7528\u529f\u80fd\uff0c\u9ed8\u8ba4 true \u3002 coredns-image \uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 coredns-template \uff1acoredns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\u83b7\u53d6\u5f53\u524d\u7248\u672c ovn \u76ee\u5f55\u4e0b coredns-template.yaml \u9ed8\u8ba4\u4e3a https://raw.githubusercontent.com/kubeovn/kube-ovn/\u5f53\u524d\u7248\u672c/yamls/coredns-template.yaml \u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u5730\u5740\u3002 k8s-service-port \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u7aef\u53e3\u3002 \u90e8\u7f72 vpc-dns \u00b6 \u914d\u7f6e vpc-dns yaml\uff1a kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u90e8\u7f72\u8d44\u6e90\u7684\u4fe1\u606f\uff1a # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72\u3002 \u9650\u5236\uff1a\u4e00\u4e2a VPC \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a VPC \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a VPC \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002 \u9a8c\u8bc1\u90e8\u7f72\u7ed3\u679c \u00b6 \u67e5\u770b vpc-dns Pod \u72b6\u6001\uff0c\u4f7f\u7528 label app=vpc-dns \uff0c\u53ef\u4ee5\u67e5\u770b\u6240\u6709 vpc-dns pod \u72b6\u6001\uff1a # kubectl -n kube-system get pods -l app=vpc-dns NAME READY STATUS RESTARTS AGE vpc-dns-test-cjh1-7b878d96b4-g5979 1 /1 Running 0 28s vpc-dns-test-cjh1-7b878d96b4-ltmf9 1 /1 Running 0 28s \u67e5\u770b slr \u72b6\u6001\u4fe1\u606f\uff1a # kubectl -n kube-system get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh1 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh1 113s \u8fdb\u5165\u8be5 VPC \u4e0b\u7684 Pod\uff0c\u6d4b\u8bd5 dns \u89e3\u6790: nslookup kubernetes.default.svc.cluster.local 10 .96.0.3 \u8be5 VPC \u4e0b\u7684 switch lb rule \u6240\u5728\u7684\u5b50\u7f51\u4ee5\u53ca\u540c\u4e00 VPC \u4e0b\u7684\u5176\u4ed6\u5b50\u7f51\u4e0b\u7684 pod \u90fd\u53ef\u4ee5\u89e3\u6790\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u81ea\u5b9a\u4e49 VPC DNS"},{"location":"advance/vpc-dns/#vpc-dns","text":"\u7531\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u548c \u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0c\u81ea\u5b9a VPC \u5185\u65e0\u6cd5\u8bbf\u95ee\u5230\u90e8\u7f72\u5728\u9ed8\u8ba4 VPC \u5185\u7684 coredns\u3002 \u5982\u679c\u7528\u6237\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 Kubernetes \u63d0\u4f9b\u7684\u96c6\u7fa4\u5185\u57df\u540d\u89e3\u6790\u80fd\u529b\uff0c\u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\uff0c\u5229\u7528 vpc-dns CRD \u6765\u5b9e\u73b0\u3002 \u8be5 CRD \u6700\u7ec8\u4f1a\u90e8\u7f72\u4e00\u4e2a coredns\uff0c\u8be5 Pod \u6709\u4e24\u4e2a\u7f51\u5361\uff0c\u4e00\u4e2a\u7f51\u5361\u5728\u7528\u6237\u81ea\u5b9a\u4e49 VPC\uff0c\u53e6\u4e00\u4e2a\u7f51\u5361\u5728\u9ed8\u8ba4 VPC \u4ece\u800c\u5b9e\u73b0\u7f51\u7edc\u4e92\u901a\uff0c\u540c\u65f6\u901a\u8fc7 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u63d0\u4f9b\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u4e00\u4e2a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u3002","title":"\u81ea\u5b9a\u4e49 VPC DNS"},{"location":"advance/vpc-dns/#vpc-dns_1","text":"apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u9664\u4e86\u4ee5\u4e0a\u8d44\u6e90\uff0c\u8be5\u529f\u80fd\u8fd8\u4f9d\u8d56 nat-gw-pod \u955c\u50cf\u8fdb\u884c\u8def\u7531\u914d\u7f6e\u3002","title":"\u90e8\u7f72 vpc-dns \u6240\u4f9d\u8d56\u7684\u8d44\u6e90"},{"location":"advance/vpc-dns/#_1","text":"apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }'","title":"\u914d\u7f6e\u9644\u52a0\u7f51\u5361"},{"location":"advance/vpc-dns/#ovn-default-provider","text":"\u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn # \u53ea\u9700\u4fee\u6539\u8be5\u5b57\u6bb5 vpc : ovn-cluster","title":"\u4fee\u6539 ovn-default \u5b50\u7f51\u7684 provider"},{"location":"advance/vpc-dns/#vpc-dns-configmap","text":"\u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : \"true\" nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\u662f\u5426\u542f\u7528\u529f\u80fd\uff0c\u9ed8\u8ba4 true \u3002 coredns-image \uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 coredns-template \uff1acoredns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\u83b7\u53d6\u5f53\u524d\u7248\u672c ovn \u76ee\u5f55\u4e0b coredns-template.yaml \u9ed8\u8ba4\u4e3a https://raw.githubusercontent.com/kubeovn/kube-ovn/\u5f53\u524d\u7248\u672c/yamls/coredns-template.yaml \u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u5730\u5740\u3002 k8s-service-port \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u7aef\u53e3\u3002","title":"\u914d\u7f6e vpc-dns \u7684 Configmap"},{"location":"advance/vpc-dns/#vpc-dns_2","text":"\u914d\u7f6e vpc-dns yaml\uff1a kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u90e8\u7f72\u8d44\u6e90\u7684\u4fe1\u606f\uff1a # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72\u3002 \u9650\u5236\uff1a\u4e00\u4e2a VPC \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a VPC \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a VPC \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002","title":"\u90e8\u7f72 vpc-dns"},{"location":"advance/vpc-dns/#_2","text":"\u67e5\u770b vpc-dns Pod \u72b6\u6001\uff0c\u4f7f\u7528 label app=vpc-dns \uff0c\u53ef\u4ee5\u67e5\u770b\u6240\u6709 vpc-dns pod \u72b6\u6001\uff1a # kubectl -n kube-system get pods -l app=vpc-dns NAME READY STATUS RESTARTS AGE vpc-dns-test-cjh1-7b878d96b4-g5979 1 /1 Running 0 28s vpc-dns-test-cjh1-7b878d96b4-ltmf9 1 /1 Running 0 28s \u67e5\u770b slr \u72b6\u6001\u4fe1\u606f\uff1a # kubectl -n kube-system get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh1 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh1 113s \u8fdb\u5165\u8be5 VPC \u4e0b\u7684 Pod\uff0c\u6d4b\u8bd5 dns \u89e3\u6790: nslookup kubernetes.default.svc.cluster.local 10 .96.0.3 \u8be5 VPC \u4e0b\u7684 switch lb rule \u6240\u5728\u7684\u5b50\u7f51\u4ee5\u53ca\u540c\u4e00 VPC \u4e0b\u7684\u5176\u4ed6\u5b50\u7f51\u4e0b\u7684 pod \u90fd\u53ef\u4ee5\u89e3\u6790\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9a8c\u8bc1\u90e8\u7f72\u7ed3\u679c"},{"location":"advance/vpc-internal-lb/","text":"\u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u00b6 Kubernetes \u63d0\u4f9b\u7684 Service \u53ef\u4ee5\u7528\u4f5c\u96c6\u7fa4\u5185\u7684\u8d1f\u8f7d\u5747\u8861\uff0c \u4f46\u662f\u5728\u81ea\u5b9a\u4e49 VPC \u6a21\u5f0f\u4e0b\uff0c \u4f7f\u7528 Service \u4f5c\u4e3a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u5b58\u5728\u5982\u4e0b\u51e0\u4e2a\u95ee\u9898\uff1a Service IP \u8303\u56f4\u4e3a\u96c6\u7fa4\u8d44\u6e90\uff0c\u6240\u6709\u81ea\u5b9a\u4e49 VPC \u5171\u4eab\uff0c\u65e0\u6cd5\u91cd\u53e0\u3002 \u7528\u6237\u65e0\u6cd5\u6309\u7167\u81ea\u5df1\u610f\u613f\u8bbe\u7f6e\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 \u4e3a\u4e86\u89e3\u51b3\u4e0a\u8ff0\u95ee\u9898\uff0cKube-OVN \u5728 1.11 \u5f15\u5165 SwitchLBRule CRD\uff0c\u7528\u6237\u53ef\u4ee5\u8bbe\u7f6e\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 SwitchLBRule \u652f\u6301\u4ee5\u4e0b\u4e24\u79cd\u65b9\u5f0f\u8bbe\u7f6e\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 Selector \u81ea\u52a8\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219 \u00b6 \u901a\u8fc7 selector \u53ef\u4ee5\u901a\u8fc7 label \u81ea\u52a8\u5173\u8054 pod \u914d\u7f6e\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 SwitchLBRule \u6837\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP selector , sessionAffinity \u548c port \u4f7f\u7528\u65b9\u5f0f\u540c Kubernetes Service\u3002 vip \uff1a\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 namespace \uff1a selector \u6240\u9009\u62e9 Pod \u6240\u5728\u547d\u540d\u7a7a\u95f4\u3002 Kube-OVN \u4f1a\u6839\u636e SwitchLBRule \u5b9a\u4e49\u9009\u62e9\u7684 Pod \u5f97\u51fa Pod \u6240\u5728 VPC \u5e76\u8bbe\u7f6e\u5bf9\u5e94\u7684 L2 LB\u3002 Endpoints \u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u89c4\u5219 \u00b6 \u901a\u8fc7 endpoints \u53ef\u4ee5\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\uff0c\u7528\u4ee5\u652f\u6301\u65e0\u6cd5\u901a\u8fc7 selector \u81ea\u52a8\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u7684\u573a\u666f\uff0c\u6bd4\u5982\u8d1f\u8f7d\u5747\u8861\u540e\u7aef\u662f kubevirt \u521b\u5efa\u7684 vm \u3002 SwitchLBRule \u6837\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default endpoints : - 192.168.0.101 - 192.168.0.102 - 192.168.0.103 ports : - name : dns port : 8888 targetPort : 80 protocol : TCP sessionAffinity \u548c port \u4f7f\u7528\u65b9\u5f0f\u540c Kubernetes Service\u3002 vip \uff1a\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 namespace \uff1a selector \u6240\u9009\u62e9 Pod \u6240\u5728\u547d\u540d\u7a7a\u95f4\u3002 endpoints \uff1a\u8d1f\u8f7d\u5747\u8861\u540e\u7aef IP \u5217\u8868\u3002 \u6ce8\uff1a \u5982\u679c\u540c\u65f6\u914d\u7f6e\u4e86 selector \u548c endpoints ,\u4f1a\u81ea\u52a8\u5ffd\u7565 selector \u914d\u7f6e\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861"},{"location":"advance/vpc-internal-lb/#vpc","text":"Kubernetes \u63d0\u4f9b\u7684 Service \u53ef\u4ee5\u7528\u4f5c\u96c6\u7fa4\u5185\u7684\u8d1f\u8f7d\u5747\u8861\uff0c \u4f46\u662f\u5728\u81ea\u5b9a\u4e49 VPC \u6a21\u5f0f\u4e0b\uff0c \u4f7f\u7528 Service \u4f5c\u4e3a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u5b58\u5728\u5982\u4e0b\u51e0\u4e2a\u95ee\u9898\uff1a Service IP \u8303\u56f4\u4e3a\u96c6\u7fa4\u8d44\u6e90\uff0c\u6240\u6709\u81ea\u5b9a\u4e49 VPC \u5171\u4eab\uff0c\u65e0\u6cd5\u91cd\u53e0\u3002 \u7528\u6237\u65e0\u6cd5\u6309\u7167\u81ea\u5df1\u610f\u613f\u8bbe\u7f6e\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 \u4e3a\u4e86\u89e3\u51b3\u4e0a\u8ff0\u95ee\u9898\uff0cKube-OVN \u5728 1.11 \u5f15\u5165 SwitchLBRule CRD\uff0c\u7528\u6237\u53ef\u4ee5\u8bbe\u7f6e\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 SwitchLBRule \u652f\u6301\u4ee5\u4e0b\u4e24\u79cd\u65b9\u5f0f\u8bbe\u7f6e\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002","title":"\u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861"},{"location":"advance/vpc-internal-lb/#selector","text":"\u901a\u8fc7 selector \u53ef\u4ee5\u901a\u8fc7 label \u81ea\u52a8\u5173\u8054 pod \u914d\u7f6e\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 SwitchLBRule \u6837\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP selector , sessionAffinity \u548c port \u4f7f\u7528\u65b9\u5f0f\u540c Kubernetes Service\u3002 vip \uff1a\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 namespace \uff1a selector \u6240\u9009\u62e9 Pod \u6240\u5728\u547d\u540d\u7a7a\u95f4\u3002 Kube-OVN \u4f1a\u6839\u636e SwitchLBRule \u5b9a\u4e49\u9009\u62e9\u7684 Pod \u5f97\u51fa Pod \u6240\u5728 VPC \u5e76\u8bbe\u7f6e\u5bf9\u5e94\u7684 L2 LB\u3002","title":"Selector \u81ea\u52a8\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219"},{"location":"advance/vpc-internal-lb/#endpoints","text":"\u901a\u8fc7 endpoints \u53ef\u4ee5\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\uff0c\u7528\u4ee5\u652f\u6301\u65e0\u6cd5\u901a\u8fc7 selector \u81ea\u52a8\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u7684\u573a\u666f\uff0c\u6bd4\u5982\u8d1f\u8f7d\u5747\u8861\u540e\u7aef\u662f kubevirt \u521b\u5efa\u7684 vm \u3002 SwitchLBRule \u6837\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default endpoints : - 192.168.0.101 - 192.168.0.102 - 192.168.0.103 ports : - name : dns port : 8888 targetPort : 80 protocol : TCP sessionAffinity \u548c port \u4f7f\u7528\u65b9\u5f0f\u540c Kubernetes Service\u3002 vip \uff1a\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 namespace \uff1a selector \u6240\u9009\u62e9 Pod \u6240\u5728\u547d\u540d\u7a7a\u95f4\u3002 endpoints \uff1a\u8d1f\u8f7d\u5747\u8861\u540e\u7aef IP \u5217\u8868\u3002 \u6ce8\uff1a \u5982\u679c\u540c\u65f6\u914d\u7f6e\u4e86 selector \u548c endpoints ,\u4f1a\u81ea\u52a8\u5ffd\u7565 selector \u914d\u7f6e\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Endpoints \u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u89c4\u5219"},{"location":"advance/vpc-peering/","text":"VPC \u4e92\u8054 \u00b6 VPC \u4e92\u8054\u63d0\u4f9b\u4e86\u4e00\u79cd\u5c06\u4e24\u4e2a VPC \u7f51\u7edc\u901a\u8fc7\u903b\u8f91\u8def\u7531\u6253\u901a\u7684\u673a\u5236\uff0c\u4ece\u800c\u4f7f\u4e24\u4e2a VPC \u5185\u7684\u5de5\u4f5c\u8d1f\u8f7d\u53ef\u4ee5\u50cf\u5728\u540c\u4e00\u4e2a\u79c1\u6709\u7f51\u7edc\u4e00\u6837\uff0c \u901a\u8fc7\u79c1\u6709\u5730\u5740\u76f8\u4e92\u8bbf\u95ee\uff0c\u65e0\u9700\u901a\u8fc7\u5916\u90e8\u7f51\u5173\u8fdb\u884c NAT \u8f6c\u53d1\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u8be5\u529f\u80fd\u53ea\u9002\u7528\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC\u3002 \u4e3a\u4e86\u907f\u514d\u8def\u7531\u91cd\u53e0\u4e24\u4e2a VPC \u5185\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 \u76ee\u524d\u53ea\u652f\u6301\u4e24\u4e2a VPC \u7684\u4e92\u8054\uff0c\u66f4\u591a\u7ec4 VPC \u4e4b\u95f4\u7684\u4e92\u8054\u6682\u4e0d\u652f\u6301\u3002 \u4f7f\u7528\u65b9\u5f0f \u00b6 \u9996\u5148\u521b\u5efa\u4e24\u4e2a\u4e0d\u4e92\u8054\u7684 VPC\uff0c\u6bcf\u4e2a VPC \u4e0b\u5404\u6709\u4e00\u4e2a Subnet\uff0cSubnet \u7684 CIDR \u4e92\u4e0d\u91cd\u53e0\u3002 kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : vpc-1 cidrBlock : 10.0.0.0/16 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : vpc-2 cidrBlock : 172.31.0.0/16 \u5728\u6bcf\u4e2a VPC \u5185\u5206\u522b\u589e\u52a0 vpcPeerings \u548c\u5bf9\u5e94\u7684\u9759\u6001\u8def\u7531\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : vpcPeerings : - remoteVpc : vpc-2 localConnectIP : 169.254.0.1/30 staticRoutes : - cidr : 172.31.0.0/16 nextHopIP : 169.254.0.2 policy : policyDst --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : vpcPeerings : - remoteVpc : vpc-1 localConnectIP : 169.254.0.2/30 staticRoutes : - cidr : 10.0.0.0/16 nextHopIP : 169.254.0.1 policy : policyDst remoteVpc : \u4e92\u8054\u7684\u53e6\u4e00\u4e2a VPC \u7684\u540d\u5b57\u3002 localConnectIP : \u4f5c\u4e3a\u4e92\u8054\u7aef\u70b9\u7684 IP \u5730\u5740\u548c CIDR\uff0c\u6ce8\u610f\u4e24\u7aef IP \u5e94\u5c5e\u4e8e\u540c\u4e00 CIDR\uff0c\u4e14\u4e0d\u80fd\u548c\u5df2\u6709\u5b50\u7f51\u51b2\u7a81\u3002 cidr \uff1a\u53e6\u4e00\u7aef Subnet \u7684 CIDR\u3002 nextHopIP \uff1a\u4e92\u8054 VPC \u53e6\u4e00\u7aef\u7684 localConnectIP \u3002 \u5206\u522b\u5728\u4e24\u4e2a Subnet \u4e0b\u521b\u5efa Pod apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 name : vpc-1-pod spec : containers : - name : vpc-1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 name : vpc-2-pod spec : containers : - name : vpc-2-pod image : docker.io/library/nginx:alpine \u6d4b\u8bd5\u7f51\u7edc\u8fde\u901a\u6027 # kubectl exec -it vpc-1-pod -- ping $(kubectl get pod vpc-2-pod -o jsonpath='{.status.podIP}') PING 172 .31.0.2 ( 172 .31.0.2 ) : 56 data bytes 64 bytes from 172 .31.0.2: seq = 0 ttl = 62 time = 0 .655 ms 64 bytes from 172 .31.0.2: seq = 1 ttl = 62 time = 0 .086 ms 64 bytes from 172 .31.0.2: seq = 2 ttl = 62 time = 0 .098 ms ^C --- 172 .31.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .086/0.279/0.655 ms # kubectl exec -it vpc-2-pod -- ping $(kubectl get pod vpc-1-pod -o jsonpath='{.status.podIP}') PING 10 .0.0.2 ( 10 .0.0.2 ) : 56 data bytes 64 bytes from 10 .0.0.2: seq = 0 ttl = 62 time = 0 .594 ms 64 bytes from 10 .0.0.2: seq = 1 ttl = 62 time = 0 .093 ms 64 bytes from 10 .0.0.2: seq = 2 ttl = 62 time = 0 .088 ms ^C --- 10 .0.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .088/0.258/0.594 ms \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC \u4e92\u8054"},{"location":"advance/vpc-peering/#vpc","text":"VPC \u4e92\u8054\u63d0\u4f9b\u4e86\u4e00\u79cd\u5c06\u4e24\u4e2a VPC \u7f51\u7edc\u901a\u8fc7\u903b\u8f91\u8def\u7531\u6253\u901a\u7684\u673a\u5236\uff0c\u4ece\u800c\u4f7f\u4e24\u4e2a VPC \u5185\u7684\u5de5\u4f5c\u8d1f\u8f7d\u53ef\u4ee5\u50cf\u5728\u540c\u4e00\u4e2a\u79c1\u6709\u7f51\u7edc\u4e00\u6837\uff0c \u901a\u8fc7\u79c1\u6709\u5730\u5740\u76f8\u4e92\u8bbf\u95ee\uff0c\u65e0\u9700\u901a\u8fc7\u5916\u90e8\u7f51\u5173\u8fdb\u884c NAT \u8f6c\u53d1\u3002","title":"VPC \u4e92\u8054"},{"location":"advance/vpc-peering/#_1","text":"\u8be5\u529f\u80fd\u53ea\u9002\u7528\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC\u3002 \u4e3a\u4e86\u907f\u514d\u8def\u7531\u91cd\u53e0\u4e24\u4e2a VPC \u5185\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 \u76ee\u524d\u53ea\u652f\u6301\u4e24\u4e2a VPC \u7684\u4e92\u8054\uff0c\u66f4\u591a\u7ec4 VPC \u4e4b\u95f4\u7684\u4e92\u8054\u6682\u4e0d\u652f\u6301\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/vpc-peering/#_2","text":"\u9996\u5148\u521b\u5efa\u4e24\u4e2a\u4e0d\u4e92\u8054\u7684 VPC\uff0c\u6bcf\u4e2a VPC \u4e0b\u5404\u6709\u4e00\u4e2a Subnet\uff0cSubnet \u7684 CIDR \u4e92\u4e0d\u91cd\u53e0\u3002 kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : vpc-1 cidrBlock : 10.0.0.0/16 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : vpc-2 cidrBlock : 172.31.0.0/16 \u5728\u6bcf\u4e2a VPC \u5185\u5206\u522b\u589e\u52a0 vpcPeerings \u548c\u5bf9\u5e94\u7684\u9759\u6001\u8def\u7531\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : vpcPeerings : - remoteVpc : vpc-2 localConnectIP : 169.254.0.1/30 staticRoutes : - cidr : 172.31.0.0/16 nextHopIP : 169.254.0.2 policy : policyDst --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : vpcPeerings : - remoteVpc : vpc-1 localConnectIP : 169.254.0.2/30 staticRoutes : - cidr : 10.0.0.0/16 nextHopIP : 169.254.0.1 policy : policyDst remoteVpc : \u4e92\u8054\u7684\u53e6\u4e00\u4e2a VPC \u7684\u540d\u5b57\u3002 localConnectIP : \u4f5c\u4e3a\u4e92\u8054\u7aef\u70b9\u7684 IP \u5730\u5740\u548c CIDR\uff0c\u6ce8\u610f\u4e24\u7aef IP \u5e94\u5c5e\u4e8e\u540c\u4e00 CIDR\uff0c\u4e14\u4e0d\u80fd\u548c\u5df2\u6709\u5b50\u7f51\u51b2\u7a81\u3002 cidr \uff1a\u53e6\u4e00\u7aef Subnet \u7684 CIDR\u3002 nextHopIP \uff1a\u4e92\u8054 VPC \u53e6\u4e00\u7aef\u7684 localConnectIP \u3002 \u5206\u522b\u5728\u4e24\u4e2a Subnet \u4e0b\u521b\u5efa Pod apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 name : vpc-1-pod spec : containers : - name : vpc-1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 name : vpc-2-pod spec : containers : - name : vpc-2-pod image : docker.io/library/nginx:alpine \u6d4b\u8bd5\u7f51\u7edc\u8fde\u901a\u6027 # kubectl exec -it vpc-1-pod -- ping $(kubectl get pod vpc-2-pod -o jsonpath='{.status.podIP}') PING 172 .31.0.2 ( 172 .31.0.2 ) : 56 data bytes 64 bytes from 172 .31.0.2: seq = 0 ttl = 62 time = 0 .655 ms 64 bytes from 172 .31.0.2: seq = 1 ttl = 62 time = 0 .086 ms 64 bytes from 172 .31.0.2: seq = 2 ttl = 62 time = 0 .098 ms ^C --- 172 .31.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .086/0.279/0.655 ms # kubectl exec -it vpc-2-pod -- ping $(kubectl get pod vpc-1-pod -o jsonpath='{.status.podIP}') PING 10 .0.0.2 ( 10 .0.0.2 ) : 56 data bytes 64 bytes from 10 .0.0.2: seq = 0 ttl = 62 time = 0 .594 ms 64 bytes from 10 .0.0.2: seq = 1 ttl = 62 time = 0 .093 ms 64 bytes from 10 .0.0.2: seq = 2 ttl = 62 time = 0 .088 ms ^C --- 10 .0.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .088/0.258/0.594 ms \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528\u65b9\u5f0f"},{"location":"advance/windows/","text":"Windows \u652f\u6301 \u00b6 Kube-OVN \u652f\u6301\u5305\u542b Windows \u7cfb\u7edf\u8282\u70b9\u7684 Kubernetes \u96c6\u7fa4\u7f51\u7edc\uff0c\u53ef\u4ee5\u5c06 Windows \u5bb9\u5668\u7684\u7f51\u7edc\u7edf\u4e00\u63a5\u5165\u8fdb\u884c\u7ba1\u7406\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u53c2\u8003 Adding Windows nodes \u589e\u52a0 Windows \u8282\u70b9\u3002 Windows \u8282\u70b9\u5fc5\u987b\u5b89\u88c5 KB4489899 \u8865\u4e01\u4ee5\u4f7f Overlay/VXLAN \u7f51\u7edc\u6b63\u5e38\u5de5\u4f5c\uff0c\u5efa\u8bae\u66f4\u65b0\u7cfb\u7edf\u81f3\u6700\u65b0\u7248\u672c\u3002 Windows \u8282\u70b9\u5fc5\u987b\u5b89\u88c5 Hyper-V \u53ca\u7ba1\u7406\u5de5\u5177\u3002 \u7531\u4e8e Windows \u9650\u5236\u96a7\u9053\u5c01\u88c5\u53ea\u80fd\u4f7f\u7528 Vxlan \u6a21\u5f0f\u3002 \u6682\u4e0d\u652f\u6301 SSL\uff0cIPv6\uff0c\u53cc\u6808\uff0cQoS \u529f\u80fd\u3002 \u6682\u4e0d\u652f\u6301\u52a8\u6001\u5b50\u7f51\uff0c\u52a8\u6001\u96a7\u9053\u63a5\u53e3\u529f\u80fd\uff0c\u9700\u5728\u5b89\u88c5 Windows \u8282\u70b9\u524d\u5b8c\u6210\u5b50\u7f51\u521b\u5efa\uff0c\u5e76\u56fa\u5b9a\u7f51\u7edc\u63a5\u53e3\u3002 \u4e0d\u652f\u6301\u591a\u4e2a ProviderNetwork \uff0c\u4e14\u65e0\u6cd5\u52a8\u6001\u8c03\u6574\u6865\u63a5\u63a5\u53e3\u914d\u7f6e\u3002 \u5b89\u88c5 OVS \u00b6 \u7531\u4e8e\u4e0a\u6e38 OVN \u548c OVS \u5bf9 Windows \u5bb9\u5668\u652f\u6301\u5b58\u5728\u4e00\u4e9b\u95ee\u9898\uff0c\u9700\u8981\u4f7f\u7528 Kube-OVN \u63d0\u4f9b\u7684\u7ecf\u8fc7\u4fee\u6539\u7684\u5b89\u88c5\u5305\u8fdb\u884c\u5b89\u88c5\u3002 \u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u6253\u5f00 Windows \u8282\u70b9\u7684 TESTSIGNING \u542f\u52a8\u9879\uff0c\u6267\u884c\u6210\u529f\u540e\u9700\u8981\u91cd\u542f\u7cfb\u7edf\u751f\u6548\uff1a bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS bcdedit /set TESTSIGNING ON bcdedit /set nointegritychecks ON \u5728 Windows \u8282\u70b9\u4e0b\u8f7d Windows \u5b89\u88c5\u5305 \u5e76\u89e3\u538b\u5b89\u88c5\u3002 \u5b89\u88c5\u5b8c\u6210\u540e\u786e\u8ba4\u670d\u52a1\u6b63\u5e38\u8fd0\u884c\uff1a PS > Get-Service | findstr ovs Running ovsdb-server Open vSwitch DB Service Running ovs-vswitchd Open vSwitch Service \u5b89\u88c5 Kube-OVN \u00b6 \u5728 Windows \u8282\u70b9\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c install.ps1 \u3002 \u8865\u5145\u76f8\u5173\u53c2\u6570\u5e76\u6267\u884c\uff1a . \\i nstall.ps1 -KubeConfig C: \\k\\a dmin.conf -ApiServer https://192.168.140.180:6443 -ServiceCIDR 10 .96.0.0/12 \u9ed8\u8ba4\u60c5\u51b5\u4e0b, Kube-OVN \u4f7f\u7528\u8282\u70b9 IP \u6240\u5728\u7684\u7f51\u5361\u4f5c\u4e3a\u96a7\u9053\u63a5\u53e3\u3002 \u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u5b83\u7f51\u5361\uff0c\u9700\u8981\u5728\u5b89\u88c5\u524d\u7ed9\u8282\u70b9\u6dfb\u52a0\u6307\u5b9a\u7684 Annotation\uff0c\u5982 ovn.kubernetes.io/tunnel_interface=Ethernet1 \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Windows \u652f\u6301"},{"location":"advance/windows/#windows","text":"Kube-OVN \u652f\u6301\u5305\u542b Windows \u7cfb\u7edf\u8282\u70b9\u7684 Kubernetes \u96c6\u7fa4\u7f51\u7edc\uff0c\u53ef\u4ee5\u5c06 Windows \u5bb9\u5668\u7684\u7f51\u7edc\u7edf\u4e00\u63a5\u5165\u8fdb\u884c\u7ba1\u7406\u3002","title":"Windows \u652f\u6301"},{"location":"advance/windows/#_1","text":"\u53c2\u8003 Adding Windows nodes \u589e\u52a0 Windows \u8282\u70b9\u3002 Windows \u8282\u70b9\u5fc5\u987b\u5b89\u88c5 KB4489899 \u8865\u4e01\u4ee5\u4f7f Overlay/VXLAN \u7f51\u7edc\u6b63\u5e38\u5de5\u4f5c\uff0c\u5efa\u8bae\u66f4\u65b0\u7cfb\u7edf\u81f3\u6700\u65b0\u7248\u672c\u3002 Windows \u8282\u70b9\u5fc5\u987b\u5b89\u88c5 Hyper-V \u53ca\u7ba1\u7406\u5de5\u5177\u3002 \u7531\u4e8e Windows \u9650\u5236\u96a7\u9053\u5c01\u88c5\u53ea\u80fd\u4f7f\u7528 Vxlan \u6a21\u5f0f\u3002 \u6682\u4e0d\u652f\u6301 SSL\uff0cIPv6\uff0c\u53cc\u6808\uff0cQoS \u529f\u80fd\u3002 \u6682\u4e0d\u652f\u6301\u52a8\u6001\u5b50\u7f51\uff0c\u52a8\u6001\u96a7\u9053\u63a5\u53e3\u529f\u80fd\uff0c\u9700\u5728\u5b89\u88c5 Windows \u8282\u70b9\u524d\u5b8c\u6210\u5b50\u7f51\u521b\u5efa\uff0c\u5e76\u56fa\u5b9a\u7f51\u7edc\u63a5\u53e3\u3002 \u4e0d\u652f\u6301\u591a\u4e2a ProviderNetwork \uff0c\u4e14\u65e0\u6cd5\u52a8\u6001\u8c03\u6574\u6865\u63a5\u63a5\u53e3\u914d\u7f6e\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/windows/#ovs","text":"\u7531\u4e8e\u4e0a\u6e38 OVN \u548c OVS \u5bf9 Windows \u5bb9\u5668\u652f\u6301\u5b58\u5728\u4e00\u4e9b\u95ee\u9898\uff0c\u9700\u8981\u4f7f\u7528 Kube-OVN \u63d0\u4f9b\u7684\u7ecf\u8fc7\u4fee\u6539\u7684\u5b89\u88c5\u5305\u8fdb\u884c\u5b89\u88c5\u3002 \u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u6253\u5f00 Windows \u8282\u70b9\u7684 TESTSIGNING \u542f\u52a8\u9879\uff0c\u6267\u884c\u6210\u529f\u540e\u9700\u8981\u91cd\u542f\u7cfb\u7edf\u751f\u6548\uff1a bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS bcdedit /set TESTSIGNING ON bcdedit /set nointegritychecks ON \u5728 Windows \u8282\u70b9\u4e0b\u8f7d Windows \u5b89\u88c5\u5305 \u5e76\u89e3\u538b\u5b89\u88c5\u3002 \u5b89\u88c5\u5b8c\u6210\u540e\u786e\u8ba4\u670d\u52a1\u6b63\u5e38\u8fd0\u884c\uff1a PS > Get-Service | findstr ovs Running ovsdb-server Open vSwitch DB Service Running ovs-vswitchd Open vSwitch Service","title":"\u5b89\u88c5 OVS"},{"location":"advance/windows/#kube-ovn","text":"\u5728 Windows \u8282\u70b9\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c install.ps1 \u3002 \u8865\u5145\u76f8\u5173\u53c2\u6570\u5e76\u6267\u884c\uff1a . \\i nstall.ps1 -KubeConfig C: \\k\\a dmin.conf -ApiServer https://192.168.140.180:6443 -ServiceCIDR 10 .96.0.0/12 \u9ed8\u8ba4\u60c5\u51b5\u4e0b, Kube-OVN \u4f7f\u7528\u8282\u70b9 IP \u6240\u5728\u7684\u7f51\u5361\u4f5c\u4e3a\u96a7\u9053\u63a5\u53e3\u3002 \u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u5b83\u7f51\u5361\uff0c\u9700\u8981\u5728\u5b89\u88c5\u524d\u7ed9\u8282\u70b9\u6dfb\u52a0\u6307\u5b9a\u7684 Annotation\uff0c\u5982 ovn.kubernetes.io/tunnel_interface=Ethernet1 \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b89\u88c5 Kube-OVN"},{"location":"advance/with-bgp/","text":"BGP \u652f\u6301 \u00b6 Kube-OVN \u652f\u6301\u5c06 Pod \u6216 Subnet \u7684 IP \u5730\u5740\u901a\u8fc7 BGP \u534f\u8bae\u5411\u5916\u90e8\u8fdb\u884c\u8def\u7531\u5e7f\u64ad\uff0c\u4ece\u800c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u5bf9\u5916\u66b4\u9732\u3002 \u5982\u679c\u9700\u8981\u4f7f\u7528\u8be5\u529f\u80fd\uff0c\u9700\u8981\u5728\u7279\u5b9a\u8282\u70b9\u5b89\u88c5 kube-ovn-speaker \u5e76\u5bf9\u9700\u8981\u5bf9\u5916\u66b4\u9732\u7684 Pod \u6216 Subnet \u589e\u52a0\u5bf9\u5e94\u7684 annotation\u3002 \u5b89\u88c5 kube-ovn-speaker \u00b6 kube-ovn-speaker \u5185\u4f7f\u7528 GoBGP \u5bf9\u5916\u53d1\u5e03\u8def\u7531\u4fe1\u606f\uff0c\u5e76\u5c06\u8bbf\u95ee\u66b4\u9732\u5730\u5740\u7684\u4e0b\u4e00\u8df3\u8def\u7531\u6307\u5411\u81ea\u8eab\u3002 \u7531\u4e8e\u90e8\u7f72 kube-ovn-speaker \u7684\u8282\u70b9\u9700\u8981\u627f\u62c5\u56de\u7a0b\u6d41\u91cf\uff0c\u56e0\u6b64\u9700\u8981\u9009\u62e9\u7279\u5b9a\u8282\u70b9\u8fdb\u884c\u90e8\u7f72\uff1a kubectl label nodes speaker-node-1 ovn.kubernetes.io/bgp = true kubectl label nodes speaker-node-2 ovn.kubernetes.io/bgp = true \u5f53\u5b58\u5728\u591a\u4e2a kube-ovn-speaker \u5b9e\u4f8b\u65f6\uff0c\u6bcf\u4e2a\u5b9e\u4f8b\u90fd\u4f1a\u5bf9\u5916\u53d1\u5e03\u8def\u7531\uff0c\u4e0a\u6e38\u8def\u7531\u5668\u9700\u8981\u652f\u6301\u591a\u8def\u5f84 ECMP\u3002 \u4e0b\u8f7d\u5bf9\u5e94 yaml: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/speaker.yaml \u4fee\u6539 yaml \u5185\u76f8\u5e94\u914d\u7f6e\uff1a --neighbor-address=10.32.32.1 --neighbor-as=65030 --cluster-as=65000 neighbor-address : BGP Peer \u7684\u5730\u5740\uff0c\u901a\u5e38\u4e3a\u8def\u7531\u5668\u7f51\u5173\u5730\u5740\u3002 neighbor-as : BGP Peer \u7684 AS \u53f7\u3002 cluster-as : \u5bb9\u5668\u7f51\u7edc\u7684 AS \u53f7\u3002 \u90e8\u7f72 yaml: kubectl apply -f speaker.yaml \u53d1\u5e03 Pod/Subnet \u8def\u7531 \u00b6 \u5982\u9700\u4f7f\u7528 BGP \u5bf9\u5916\u53d1\u5e03\u8def\u7531\uff0c\u9996\u5148\u9700\u8981\u5c06\u5bf9\u5e94 Subnet \u7684 natOutgoing \u8bbe\u7f6e\u4e3a false \uff0c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u8fdb\u5165\u5e95\u5c42\u7f51\u7edc\u3002 \u589e\u52a0 annotation \u5bf9\u5916\u53d1\u5e03\uff1a kubectl annotate pod sample ovn.kubernetes.io/bgp = true kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp = true \u5220\u9664 annotation \u53d6\u6d88\u53d1\u5e03\uff1a kubectl annotate pod perf-ovn-xzvd4 ovn.kubernetes.io/bgp- kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp- BGP \u9ad8\u7ea7\u9009\u9879 \u00b6 kube-ovn-speaker \u652f\u6301\u66f4\u591a BGP \u53c2\u6570\u8fdb\u884c\u9ad8\u7ea7\u914d\u7f6e\uff0c\u7528\u6237\u53ef\u6839\u636e\u81ea\u5df1\u7f51\u7edc\u73af\u5883\u8fdb\u884c\u8c03\u6574\uff1a announce-cluster-ip : \u662f\u5426\u5bf9\u5916\u53d1\u5e03 Service \u8def\u7531\uff0c\u9ed8\u8ba4\u4e3a false \u3002 auth-password : BGP peer \u7684\u8bbf\u95ee\u5bc6\u7801\u3002 holdtime : BGP \u90bb\u5c45\u95f4\u7684\u5fc3\u8df3\u63a2\u6d4b\u65f6\u95f4\uff0c\u8d85\u8fc7\u6539\u65f6\u95f4\u6ca1\u6709\u6d88\u606f\u7684\u90bb\u5c45\u5c06\u4f1a\u88ab\u79fb\u9664\uff0c\u9ed8\u8ba4\u4e3a 90 \u79d2\u3002 graceful-restart : \u662f\u5426\u542f\u7528 BGP Graceful Restart\u3002 graceful-restart-time : BGP Graceful restart time \u53ef\u53c2\u8003 RFC4724 3\u3002 graceful-restart-deferral-time : BGP Graceful restart deferral time \u53ef\u53c2\u8003 RFC4724 4.1\u3002 passivemode : Speaker \u8fd0\u884c\u5728 passive \u6a21\u5f0f\uff0c\u4e0d\u4e3b\u52a8\u8fde\u63a5 peer\u3002 ebgp-multihop : ebgp ttl \u9ed8\u8ba4\u503c\u4e3a 1\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"BGP \u652f\u6301"},{"location":"advance/with-bgp/#bgp","text":"Kube-OVN \u652f\u6301\u5c06 Pod \u6216 Subnet \u7684 IP \u5730\u5740\u901a\u8fc7 BGP \u534f\u8bae\u5411\u5916\u90e8\u8fdb\u884c\u8def\u7531\u5e7f\u64ad\uff0c\u4ece\u800c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u5bf9\u5916\u66b4\u9732\u3002 \u5982\u679c\u9700\u8981\u4f7f\u7528\u8be5\u529f\u80fd\uff0c\u9700\u8981\u5728\u7279\u5b9a\u8282\u70b9\u5b89\u88c5 kube-ovn-speaker \u5e76\u5bf9\u9700\u8981\u5bf9\u5916\u66b4\u9732\u7684 Pod \u6216 Subnet \u589e\u52a0\u5bf9\u5e94\u7684 annotation\u3002","title":"BGP \u652f\u6301"},{"location":"advance/with-bgp/#kube-ovn-speaker","text":"kube-ovn-speaker \u5185\u4f7f\u7528 GoBGP \u5bf9\u5916\u53d1\u5e03\u8def\u7531\u4fe1\u606f\uff0c\u5e76\u5c06\u8bbf\u95ee\u66b4\u9732\u5730\u5740\u7684\u4e0b\u4e00\u8df3\u8def\u7531\u6307\u5411\u81ea\u8eab\u3002 \u7531\u4e8e\u90e8\u7f72 kube-ovn-speaker \u7684\u8282\u70b9\u9700\u8981\u627f\u62c5\u56de\u7a0b\u6d41\u91cf\uff0c\u56e0\u6b64\u9700\u8981\u9009\u62e9\u7279\u5b9a\u8282\u70b9\u8fdb\u884c\u90e8\u7f72\uff1a kubectl label nodes speaker-node-1 ovn.kubernetes.io/bgp = true kubectl label nodes speaker-node-2 ovn.kubernetes.io/bgp = true \u5f53\u5b58\u5728\u591a\u4e2a kube-ovn-speaker \u5b9e\u4f8b\u65f6\uff0c\u6bcf\u4e2a\u5b9e\u4f8b\u90fd\u4f1a\u5bf9\u5916\u53d1\u5e03\u8def\u7531\uff0c\u4e0a\u6e38\u8def\u7531\u5668\u9700\u8981\u652f\u6301\u591a\u8def\u5f84 ECMP\u3002 \u4e0b\u8f7d\u5bf9\u5e94 yaml: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/speaker.yaml \u4fee\u6539 yaml \u5185\u76f8\u5e94\u914d\u7f6e\uff1a --neighbor-address=10.32.32.1 --neighbor-as=65030 --cluster-as=65000 neighbor-address : BGP Peer \u7684\u5730\u5740\uff0c\u901a\u5e38\u4e3a\u8def\u7531\u5668\u7f51\u5173\u5730\u5740\u3002 neighbor-as : BGP Peer \u7684 AS \u53f7\u3002 cluster-as : \u5bb9\u5668\u7f51\u7edc\u7684 AS \u53f7\u3002 \u90e8\u7f72 yaml: kubectl apply -f speaker.yaml","title":"\u5b89\u88c5 kube-ovn-speaker"},{"location":"advance/with-bgp/#podsubnet","text":"\u5982\u9700\u4f7f\u7528 BGP \u5bf9\u5916\u53d1\u5e03\u8def\u7531\uff0c\u9996\u5148\u9700\u8981\u5c06\u5bf9\u5e94 Subnet \u7684 natOutgoing \u8bbe\u7f6e\u4e3a false \uff0c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u8fdb\u5165\u5e95\u5c42\u7f51\u7edc\u3002 \u589e\u52a0 annotation \u5bf9\u5916\u53d1\u5e03\uff1a kubectl annotate pod sample ovn.kubernetes.io/bgp = true kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp = true \u5220\u9664 annotation \u53d6\u6d88\u53d1\u5e03\uff1a kubectl annotate pod perf-ovn-xzvd4 ovn.kubernetes.io/bgp- kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp-","title":"\u53d1\u5e03 Pod/Subnet \u8def\u7531"},{"location":"advance/with-bgp/#bgp_1","text":"kube-ovn-speaker \u652f\u6301\u66f4\u591a BGP \u53c2\u6570\u8fdb\u884c\u9ad8\u7ea7\u914d\u7f6e\uff0c\u7528\u6237\u53ef\u6839\u636e\u81ea\u5df1\u7f51\u7edc\u73af\u5883\u8fdb\u884c\u8c03\u6574\uff1a announce-cluster-ip : \u662f\u5426\u5bf9\u5916\u53d1\u5e03 Service \u8def\u7531\uff0c\u9ed8\u8ba4\u4e3a false \u3002 auth-password : BGP peer \u7684\u8bbf\u95ee\u5bc6\u7801\u3002 holdtime : BGP \u90bb\u5c45\u95f4\u7684\u5fc3\u8df3\u63a2\u6d4b\u65f6\u95f4\uff0c\u8d85\u8fc7\u6539\u65f6\u95f4\u6ca1\u6709\u6d88\u606f\u7684\u90bb\u5c45\u5c06\u4f1a\u88ab\u79fb\u9664\uff0c\u9ed8\u8ba4\u4e3a 90 \u79d2\u3002 graceful-restart : \u662f\u5426\u542f\u7528 BGP Graceful Restart\u3002 graceful-restart-time : BGP Graceful restart time \u53ef\u53c2\u8003 RFC4724 3\u3002 graceful-restart-deferral-time : BGP Graceful restart deferral time \u53ef\u53c2\u8003 RFC4724 4.1\u3002 passivemode : Speaker \u8fd0\u884c\u5728 passive \u6a21\u5f0f\uff0c\u4e0d\u4e3b\u52a8\u8fde\u63a5 peer\u3002 ebgp-multihop : ebgp ttl \u9ed8\u8ba4\u503c\u4e3a 1\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"BGP \u9ad8\u7ea7\u9009\u9879"},{"location":"advance/with-cilium/","text":"Cilium \u96c6\u6210 \u00b6 Cilium \u662f\u4e00\u6b3e\u57fa\u4e8e eBPF \u7684\u7f51\u7edc\u548c\u5b89\u5168\u7ec4\u4ef6\uff0cKube-OVN \u5229\u7528\u5176\u4e2d\u7684 CNI Chaining \u6a21\u5f0f\u6765\u5bf9\u5df2\u6709\u529f\u80fd\u8fdb\u884c\u589e\u5f3a\u3002 \u7528\u6237\u53ef\u4ee5\u540c\u65f6\u4f7f\u7528 Kube-OVN \u4e30\u5bcc\u7684\u7f51\u7edc\u62bd\u8c61\u80fd\u529b\u548c eBPF \u5e26\u6765\u7684\u76d1\u63a7\u548c\u5b89\u5168\u80fd\u529b\u3002 \u901a\u8fc7\u96c6\u6210 Cilium\uff0cKube-OVN \u7528\u6237\u53ef\u4ee5\u83b7\u5f97\u5982\u4e0b\u589e\u76ca\uff1a \u66f4\u4e30\u5bcc\u9ad8\u6548\u7684\u5b89\u5168\u7b56\u7565\u3002 \u57fa\u4e8e Hubble \u7684\u76d1\u63a7\u89c6\u56fe\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 Linux \u5185\u6838\u7248\u672c\u9ad8\u4e8e 4.19 \u6216\u5176\u4ed6\u517c\u5bb9\u5185\u6838\u4ee5\u83b7\u5f97\u5b8c\u6574 eBPF \u80fd\u529b\u652f\u6301\u3002 \u63d0\u524d\u90e8\u7f72 Helm \u4e3a\u5b89\u88c5 Cilium \u505a\u51c6\u5907\uff0c\u90e8\u7f72 Helm \u8bf7\u53c2\u8003 Installing Helm \u3002 \u914d\u7f6e Kube-OVN \u00b6 \u4e3a\u4e86\u5145\u5206\u4f7f\u7528 Cilium \u7684\u5b89\u5168\u80fd\u529b\uff0c\u9700\u8981\u5173\u95ed Kube-OVN \u5185\u7684 networkpolicy \u529f\u80fd\uff0c\u5e76\u8c03\u6574 CNI \u914d\u7f6e\u4f18\u5148\u7ea7\u3002 \u5728 install.sh \u811a\u672c\u91cc\u4fee\u6539\u4e0b\u5217\u53d8\u91cf\uff1a ENABLE_NP = false CNI_CONFIG_PRIORITY = 10 \u82e5\u5df2\u90e8\u7f72\u5b8c\u6210\uff0c\u53ef\u901a\u8fc7\u4fee\u6539 kube-ovn-controller \u7684\u542f\u52a8\u53c2\u6570\u8fdb\u884c\u8c03\u6574 networkpolicy \uff1a args : - --enable-np=false \u4fee\u6539 kube-ovn-cni \u542f\u52a8\u53c2\u6570\u8c03\u6574 CNI \u914d\u7f6e\u4f18\u5148\u7ea7\uff1a args : - --cni-conf-name=10-kube-ovn.conflist \u5728\u6bcf\u4e2a\u8282\u70b9\u8c03\u6574 Kube-OVN \u914d\u7f6e\u6587\u4ef6\u540d\u79f0\uff0c\u4ee5\u4fbf\u4f18\u5148\u4f7f\u7528 Cilium \u8fdb\u884c\u64cd\u4f5c\uff1a mv /etc/cni/net.d/01-kube-ovn.conflist /etc/cni/net.d/10-kube-ovn.conflist \u90e8\u7f72 Cilium \u00b6 \u521b\u5efa chaining.yaml \u914d\u7f6e\u6587\u4ef6\uff0c\u4f7f\u7528 Cilium \u7684 generic-veth \u6a21\u5f0f\uff1a apiVersion : v1 kind : ConfigMap metadata : name : cni-configuration namespace : kube-system data : cni-config : |- { \"name\": \"generic-veth\", \"cniVersion\": \"0.3.1\", \"plugins\": [ { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\" } }, { \"type\": \"portmap\", \"snat\": true, \"capabilities\": {\"portMappings\": true} }, { \"type\": \"cilium-cni\" } ] } \u5b89\u88c5\u914d\u7f6e\u6587\u4ef6\uff1a kubectl apply -f chaining.yaml \u4f7f\u7528 Helm \u90e8\u7f72 Cilium\uff1a helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --set cni.chainingMode = generic-veth \\ --set cni.customConf = true \\ --set cni.configMap = cni-configuration \\ --set tunnel = disabled \\ --set enableIPv4Masquerade = false \\ --set enableIdentityMark = false \u786e\u8ba4 Cilium \u5b89\u88c5\u6210\u529f\uff1a # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: disabled \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 Containers: cilium Running: 2 cilium-operator Running: 2 Cluster Pods: 8 /11 managed by Cilium Image versions cilium quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb: 2 cilium-operator quay.io/cilium/operator-generic:v1.10.5@sha256:2d2f730f219d489ff0702923bf24c0002cd93eb4b47ba344375566202f56d972: 2 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium \u96c6\u6210"},{"location":"advance/with-cilium/#cilium","text":"Cilium \u662f\u4e00\u6b3e\u57fa\u4e8e eBPF \u7684\u7f51\u7edc\u548c\u5b89\u5168\u7ec4\u4ef6\uff0cKube-OVN \u5229\u7528\u5176\u4e2d\u7684 CNI Chaining \u6a21\u5f0f\u6765\u5bf9\u5df2\u6709\u529f\u80fd\u8fdb\u884c\u589e\u5f3a\u3002 \u7528\u6237\u53ef\u4ee5\u540c\u65f6\u4f7f\u7528 Kube-OVN \u4e30\u5bcc\u7684\u7f51\u7edc\u62bd\u8c61\u80fd\u529b\u548c eBPF \u5e26\u6765\u7684\u76d1\u63a7\u548c\u5b89\u5168\u80fd\u529b\u3002 \u901a\u8fc7\u96c6\u6210 Cilium\uff0cKube-OVN \u7528\u6237\u53ef\u4ee5\u83b7\u5f97\u5982\u4e0b\u589e\u76ca\uff1a \u66f4\u4e30\u5bcc\u9ad8\u6548\u7684\u5b89\u5168\u7b56\u7565\u3002 \u57fa\u4e8e Hubble \u7684\u76d1\u63a7\u89c6\u56fe\u3002","title":"Cilium \u96c6\u6210"},{"location":"advance/with-cilium/#_1","text":"Linux \u5185\u6838\u7248\u672c\u9ad8\u4e8e 4.19 \u6216\u5176\u4ed6\u517c\u5bb9\u5185\u6838\u4ee5\u83b7\u5f97\u5b8c\u6574 eBPF \u80fd\u529b\u652f\u6301\u3002 \u63d0\u524d\u90e8\u7f72 Helm \u4e3a\u5b89\u88c5 Cilium \u505a\u51c6\u5907\uff0c\u90e8\u7f72 Helm \u8bf7\u53c2\u8003 Installing Helm \u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/with-cilium/#kube-ovn","text":"\u4e3a\u4e86\u5145\u5206\u4f7f\u7528 Cilium \u7684\u5b89\u5168\u80fd\u529b\uff0c\u9700\u8981\u5173\u95ed Kube-OVN \u5185\u7684 networkpolicy \u529f\u80fd\uff0c\u5e76\u8c03\u6574 CNI \u914d\u7f6e\u4f18\u5148\u7ea7\u3002 \u5728 install.sh \u811a\u672c\u91cc\u4fee\u6539\u4e0b\u5217\u53d8\u91cf\uff1a ENABLE_NP = false CNI_CONFIG_PRIORITY = 10 \u82e5\u5df2\u90e8\u7f72\u5b8c\u6210\uff0c\u53ef\u901a\u8fc7\u4fee\u6539 kube-ovn-controller \u7684\u542f\u52a8\u53c2\u6570\u8fdb\u884c\u8c03\u6574 networkpolicy \uff1a args : - --enable-np=false \u4fee\u6539 kube-ovn-cni \u542f\u52a8\u53c2\u6570\u8c03\u6574 CNI \u914d\u7f6e\u4f18\u5148\u7ea7\uff1a args : - --cni-conf-name=10-kube-ovn.conflist \u5728\u6bcf\u4e2a\u8282\u70b9\u8c03\u6574 Kube-OVN \u914d\u7f6e\u6587\u4ef6\u540d\u79f0\uff0c\u4ee5\u4fbf\u4f18\u5148\u4f7f\u7528 Cilium \u8fdb\u884c\u64cd\u4f5c\uff1a mv /etc/cni/net.d/01-kube-ovn.conflist /etc/cni/net.d/10-kube-ovn.conflist","title":"\u914d\u7f6e Kube-OVN"},{"location":"advance/with-cilium/#cilium_1","text":"\u521b\u5efa chaining.yaml \u914d\u7f6e\u6587\u4ef6\uff0c\u4f7f\u7528 Cilium \u7684 generic-veth \u6a21\u5f0f\uff1a apiVersion : v1 kind : ConfigMap metadata : name : cni-configuration namespace : kube-system data : cni-config : |- { \"name\": \"generic-veth\", \"cniVersion\": \"0.3.1\", \"plugins\": [ { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\" } }, { \"type\": \"portmap\", \"snat\": true, \"capabilities\": {\"portMappings\": true} }, { \"type\": \"cilium-cni\" } ] } \u5b89\u88c5\u914d\u7f6e\u6587\u4ef6\uff1a kubectl apply -f chaining.yaml \u4f7f\u7528 Helm \u90e8\u7f72 Cilium\uff1a helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --set cni.chainingMode = generic-veth \\ --set cni.customConf = true \\ --set cni.configMap = cni-configuration \\ --set tunnel = disabled \\ --set enableIPv4Masquerade = false \\ --set enableIdentityMark = false \u786e\u8ba4 Cilium \u5b89\u88c5\u6210\u529f\uff1a # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: disabled \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 Containers: cilium Running: 2 cilium-operator Running: 2 Cluster Pods: 8 /11 managed by Cilium Image versions cilium quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb: 2 cilium-operator quay.io/cilium/operator-generic:v1.10.5@sha256:2d2f730f219d489ff0702923bf24c0002cd93eb4b47ba344375566202f56d972: 2 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u90e8\u7f72 Cilium"},{"location":"advance/with-openstack/","text":"OpenStack \u96c6\u6210 \u00b6 \u5728\u4e00\u4e9b\u60c5\u51b5\u4e0b\uff0c\u7528\u6237\u9700\u8981\u4f7f\u7528 OpenStack \u8fd0\u884c\u865a\u62df\u673a\uff0c\u4f7f\u7528 Kubernetes \u8fd0\u884c\u5bb9\u5668\uff0c\u5e76\u9700\u8981\u5bb9\u5668\u548c\u865a\u673a\u4e4b\u95f4\u7f51\u7edc\u4e92\u901a\u5e76\u5904\u4e8e\u7edf\u4e00\u63a7\u5236\u5e73\u9762\u4e0b\u3002\u5982\u679c OpenStack Neutron \u4fa7\u540c\u6837\u4f7f\u7528 OVN \u4f5c\u4e3a\u5e95\u5c42\u7f51\u7edc\u63a7\u5236\uff0c\u90a3\u4e48 Kube-OVN \u53ef\u4ee5\u4f7f\u7528\u96c6\u7fa4\u4e92\u8054\u548c\u5171\u4eab\u5e95\u5c42 OVN \u4e24\u79cd\u65b9\u5f0f\u6253\u901a OpenStack \u548c Kubernetes \u7684\u7f51\u7edc\u3002 \u96c6\u7fa4\u4e92\u8054 \u00b6 \u8be5\u6a21\u5f0f\u548c \u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054 \u6253\u901a\u4e24\u4e2a Kubernetes \u96c6\u7fa4\u7f51\u7edc\u65b9\u5f0f\u7c7b\u4f3c\uff0c\u53ea\u4e0d\u8fc7\u5c06\u96c6\u7fa4\u4e24\u7aef\u6362\u6210 OpenStack \u548c Kubernetes\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u81ea\u52a8\u4e92\u8054\u6a21\u5f0f\u4e0b OpenStack \u548c Kubernetes \u5185\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u76f8\u4e92\u91cd\u53e0\u3002 \u9700\u8981\u5b58\u5728\u4e00\u7ec4\u673a\u5668\u53ef\u4ee5\u88ab\u6bcf\u4e2a\u96c6\u7fa4\u901a\u8fc7\u7f51\u7edc\u8bbf\u95ee\uff0c\u7528\u6765\u90e8\u7f72\u8de8\u96c6\u7fa4\u4e92\u8054\u7684\u63a7\u5236\u5668\u3002 \u6bcf\u4e2a\u96c6\u7fa4\u9700\u8981\u6709\u4e00\u7ec4\u53ef\u4ee5\u901a\u8fc7 IP \u8fdb\u884c\u8de8\u96c6\u7fa4\u4e92\u8bbf\u7684\u673a\u5668\u4f5c\u4e3a\u4e4b\u540e\u7684\u7f51\u5173\u8282\u70b9\u3002 \u8be5\u65b9\u6848\u53ea\u6253\u901a Kubernetes \u9ed8\u8ba4\u5b50\u7f51\u548c OpenStack \u7684\u9009\u5b9a VPC\u3002 \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93 \u00b6 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\uff1a docker run --name = ovn-ic-db -d --network = host -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh Kubernetes \u4fa7\u64cd\u4f5c \u00b6 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : \u662f\u5426\u5f00\u542f\u96c6\u7fa4\u4e92\u8054\u3002 az-name : \u533a\u5206\u4e0d\u540c\u96c6\u7fa4\u7684\u96c6\u7fa4\u540d\u79f0\uff0c\u6bcf\u4e2a\u4e92\u8054\u96c6\u7fa4\u9700\u4e0d\u540c\u3002 ic-db-host : \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u5730\u5740\u3002 ic-nb-port : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6645\u3002 ic-sb-port : OVN-IC \u5357\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6646\u3002 gw-nodes : \u96c6\u7fa4\u4e92\u8054\u4e2d\u627f\u62c5\u7f51\u5173\u5de5\u4f5c\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 auto-route : \u662f\u5426\u81ea\u52a8\u5bf9\u5916\u53d1\u5e03\u548c\u5b66\u4e60\u8def\u7531\u3002 OpenStack \u4fa7\u64cd\u4f5c \u00b6 \u521b\u5efa\u548c Kubernetes \u4e92\u8054\u7684\u903b\u8f91\u8def\u7531\u5668\uff1a # openstack router create router0 # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | d5b38655-249a-4192-8046-71aa4d2b4af1 | router0 | ACTIVE | UP | 98a29ab7388347e7b5ff8bdd181ba4f9 | +--------------------------------------+---------+--------+-------+----------------------------------+ \u5728 OpenStack \u5185\u7684 OVN \u5317\u5411\u6570\u636e\u5e93\u4e2d\u8bbe\u7f6e\u53ef\u7528\u533a\u540d\u5b57\uff0c\u8be5\u540d\u79f0\u9700\u548c\u5176\u4ed6\u4e92\u8054\u96c6\u7fa4\u4e0d\u540c\uff1a ovn-nbctl set NB_Global . name = op-az \u5728\u53ef\u8bbf\u95ee OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u542f\u52a8 OVN-IC \u63a7\u5236\u5668\uff1a /usr/share/ovn/scripts/ovn-ctl --ovn-ic-nb-db = tcp:192.168.65.3:6645 \\ --ovn-ic-sb-db = tcp:192.168.65.3:6646 \\ --ovn-northd-nb-db = unix:/run/ovn/ovnnb_db.sock \\ --ovn-northd-sb-db = unix:/run/ovn/ovnsb_db.sock \\ start_ic ovn-ic-nb-db \uff0c ovn-ic-sb-db : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u548c\u5357\u5411\u6570\u636e\u5e93\u5730\u5740\u3002 ovn-northd-nb-db \uff0c ovn-northd-sb-db : \u5f53\u524d\u96c6\u7fa4 OVN \u5317\u5411\u6570\u636e\u5e93\u548c\u5357\u5411\u6570\u636e\u5730\u5740\u3002 \u914d\u7f6e\u4e92\u8054\u7f51\u5173\u8282\u70b9\uff1a ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn = true \u63a5\u4e0b\u6765\u9700\u8981\u5728 OpenStack \u7684 OVN \u5185\u8fdb\u884c\u64cd\u4f5c\u521b\u5efa\u903b\u8f91\u62d3\u6251\u3002 \u8fde\u63a5 ts \u4e92\u8054\u4ea4\u6362\u673a\u548c router0 \u903b\u8f91\u8def\u7531\u5668\uff0c\u5e76\u8bbe\u7f6e\u76f8\u5173\u89c4\u5219\uff1a ovn-nbctl lrp-add router0 lrp-router0-ts 00 :02:ef:11:39:4f 169 .254.100.73/24 ovn-nbctl lsp-add ts lsp-ts-router0 -- lsp-set-addresses lsp-ts-router0 router \\ -- lsp-set-type lsp-ts-router0 router \\ -- lsp-set-options lsp-ts-router0 router-port = lrp-router0-ts ovn-nbctl lrp-set-gateway-chassis lrp-router0-ts { gateway chassis } 1000 ovn-nbctl set NB_Global . options:ic-route-adv = true options:ic-route-learn = true \u9a8c\u8bc1\u5df2\u5b66\u4e60\u5230 Kubernetes \u8def\u7531\u89c4\u5219\uff1a # ovn-nbctl lr-route-list router0 IPv4 Routes 10 .0.0.22 169 .254.100.34 dst-ip ( learned ) 10 .16.0.0/16 169 .254.100.34 dst-ip ( learned ) \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728 router0 \u7f51\u7edc\u4e0b\u521b\u5efa\u865a\u673a\u9a8c\u8bc1\u662f\u5426\u53ef\u4ee5\u548c Kubernetes \u4e0b Pod \u4e92\u901a\u3002 \u5171\u4eab\u5e95\u5c42 OVN \u00b6 \u5728\u8be5\u65b9\u6848\u4e0b\uff0cOpenStack \u548c Kubernetes \u5171\u4eab\u4f7f\u7528\u540c\u4e00\u4e2a OVN\uff0c\u56e0\u6b64\u53ef\u4ee5\u5c06\u4e24\u8005\u7684 VPC \u548c Subnet \u7b49\u6982\u5ff5\u62c9\u9f50\uff0c\u5b9e\u73b0\u66f4\u597d\u7684\u63a7\u5236\u548c\u4e92\u8054\u3002 \u5728\u8be5\u6a21\u5f0f\u4e0b\u6211\u4eec\u6b63\u5e38\u4f7f\u7528 Kube-OVN \u90e8\u7f72 OVN\uff0cOpenStack \u4fee\u6539 Neutron \u914d\u7f6e\u5b9e\u73b0\u8fde\u63a5\u540c\u4e00\u4e2a OVN \u6570\u636e\u5e93\u3002OpenStack \u9700\u4f7f\u7528 networking-ovn \u4f5c\u4e3a Neutron \u540e\u7aef\u5b9e\u73b0\u3002 Neutron \u914d\u7f6e\u4fee\u6539 \u00b6 \u4fee\u6539 Neutron \u914d\u7f6e\u6587\u4ef6 /etc/neutron/plugins/ml2/ml2_conf.ini \uff1a [ ovn ] ... ovn_nb_connection = tcp: [ 192 .168.137.176 ] :6641,tcp: [ 192 .168.137.177 ] :6641,tcp: [ 192 .168.137.178 ] :6641 ovn_sb_connection = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovn_l3_scheduler = OVN_L3_SCHEDULER ovn_nb_connection \uff0c ovn_sb_connection : \u5730\u5740\u9700\u4fee\u6539\u4e3a Kube-OVN \u90e8\u7f72 ovn-central \u8282\u70b9\u7684\u5730\u5740\u3002 \u4fee\u6539\u6bcf\u4e2a\u8282\u70b9\u7684 OVS \u914d\u7f6e\uff1a ovs-vsctl set open . external-ids:ovn-remote = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovs-vsctl set open . external-ids:ovn-encap-type = geneve ovs-vsctl set open . external-ids:ovn-encap-ip = 192 .168.137.200 external-ids:ovn-remote : \u5730\u5740\u9700\u4fee\u6539\u4e3a Kube-OVN \u90e8\u7f72 ovn-central \u8282\u70b9\u7684\u5730\u5740\u3002 ovn-encap-ip : \u4fee\u6539\u4e3a\u5f53\u524d\u8282\u70b9\u7684 IP \u5730\u5740\u3002 \u5728 Kubernetes \u4e2d\u4f7f\u7528 OpenStack \u5185\u8d44\u6e90 \u00b6 \u63a5\u4e0b\u6765\u4ecb\u7ecd\u5982\u4f55\u5728 Kubernetes \u4e2d\u67e5\u8be2 OpenStack \u7684\u7f51\u7edc\u8d44\u6e90\u5e76\u5728 OpenStack \u7684\u5b50\u7f51\u4e2d\u521b\u5efa Pod\u3002 \u67e5\u8be2 OpenStack \u4e2d\u5df2\u6709\u7684\u7f51\u7edc\u8d44\u6e90\uff0c\u5982\u4e0b\u8d44\u6e90\u5df2\u7ecf\u9884\u5148\u521b\u5efa\u5b8c\u6210\uff1a # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | 22040ed5-0598-4f77-bffd-e7fd4db47e93 | router0 | ACTIVE | UP | 62381a21d569404aa236a5dd8712449c | +--------------------------------------+---------+--------+-------+----------------------------------+ # openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | cd59e36a-37db-4c27-b709-d35379a7920f | provider | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | +--------------------------------------+----------+--------------------------------------+ # openstack subnet list +--------------------------------------+-------------+--------------------------------------+----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------------+--------------------------------------+----------------+ | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | provider-v4 | cd59e36a-37db-4c27-b709-d35379a7920f | 192 .168.1.0/24 | +--------------------------------------+-------------+--------------------------------------+----------------+ # openstack server list +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | 8433d622-a8d6-41a7-8b31-49abfd64f639 | provider-instance | ACTIVE | provider = 192 .168.1.61 | ubuntu | m1 | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ \u5728 Kubernetes \u4fa7\uff0c\u67e5\u8be2 VPC \u8d44\u6e90\uff1a # kubectl get vpc NAME STANDBY SUBNETS neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 true [ \"neutron-cd59e36a-37db-4c27-b709-d35379a7920f\" ] ovn-cluster true [ \"join\" , \"ovn-default\" ] neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 \u4e3a\u4ece OpenStack \u540c\u6b65\u8fc7\u6765\u7684 VPC \u8d44\u6e90\u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u6309\u7167 Kube-OVN \u539f\u751f\u7684 VPC \u548c Subnet \u64cd\u4f5c\u521b\u5efa Pod \u5e76\u8fd0\u884c\u3002 VPC, Subnet \u7ed1\u5b9a Namespace net2 \uff0c\u5e76\u521b\u5efa Pod: apiVersion : v1 kind : Namespace metadata : name : net2 --- apiVersion : kubeovn.io/v1 kind : Vpc metadata : creationTimestamp : \"2021-06-20T13:34:11Z\" generation : 2 labels : ovn.kubernetes.io/vpc_external : \"true\" name : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 resourceVersion : \"583728\" uid : 18d4c654-f511-4def-a3a0-a6434d237c1e spec : namespaces : - net2 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 namespaces : - net2 cidrBlock : 12.0.1.0/24 natOutgoing : false --- apiVersion : v1 kind : Pod metadata : name : ubuntu namespace : net2 spec : containers : - image : docker.io/kubeovn/kube-ovn:v1.8.0 command : - \"sleep\" - \"604800\" imagePullPolicy : IfNotPresent name : ubuntu restartPolicy : Always \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OpenStack \u96c6\u6210"},{"location":"advance/with-openstack/#openstack","text":"\u5728\u4e00\u4e9b\u60c5\u51b5\u4e0b\uff0c\u7528\u6237\u9700\u8981\u4f7f\u7528 OpenStack \u8fd0\u884c\u865a\u62df\u673a\uff0c\u4f7f\u7528 Kubernetes \u8fd0\u884c\u5bb9\u5668\uff0c\u5e76\u9700\u8981\u5bb9\u5668\u548c\u865a\u673a\u4e4b\u95f4\u7f51\u7edc\u4e92\u901a\u5e76\u5904\u4e8e\u7edf\u4e00\u63a7\u5236\u5e73\u9762\u4e0b\u3002\u5982\u679c OpenStack Neutron \u4fa7\u540c\u6837\u4f7f\u7528 OVN \u4f5c\u4e3a\u5e95\u5c42\u7f51\u7edc\u63a7\u5236\uff0c\u90a3\u4e48 Kube-OVN \u53ef\u4ee5\u4f7f\u7528\u96c6\u7fa4\u4e92\u8054\u548c\u5171\u4eab\u5e95\u5c42 OVN \u4e24\u79cd\u65b9\u5f0f\u6253\u901a OpenStack \u548c Kubernetes \u7684\u7f51\u7edc\u3002","title":"OpenStack \u96c6\u6210"},{"location":"advance/with-openstack/#_1","text":"\u8be5\u6a21\u5f0f\u548c \u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054 \u6253\u901a\u4e24\u4e2a Kubernetes \u96c6\u7fa4\u7f51\u7edc\u65b9\u5f0f\u7c7b\u4f3c\uff0c\u53ea\u4e0d\u8fc7\u5c06\u96c6\u7fa4\u4e24\u7aef\u6362\u6210 OpenStack \u548c Kubernetes\u3002","title":"\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-openstack/#_2","text":"\u81ea\u52a8\u4e92\u8054\u6a21\u5f0f\u4e0b OpenStack \u548c Kubernetes \u5185\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u76f8\u4e92\u91cd\u53e0\u3002 \u9700\u8981\u5b58\u5728\u4e00\u7ec4\u673a\u5668\u53ef\u4ee5\u88ab\u6bcf\u4e2a\u96c6\u7fa4\u901a\u8fc7\u7f51\u7edc\u8bbf\u95ee\uff0c\u7528\u6765\u90e8\u7f72\u8de8\u96c6\u7fa4\u4e92\u8054\u7684\u63a7\u5236\u5668\u3002 \u6bcf\u4e2a\u96c6\u7fa4\u9700\u8981\u6709\u4e00\u7ec4\u53ef\u4ee5\u901a\u8fc7 IP \u8fdb\u884c\u8de8\u96c6\u7fa4\u4e92\u8bbf\u7684\u673a\u5668\u4f5c\u4e3a\u4e4b\u540e\u7684\u7f51\u5173\u8282\u70b9\u3002 \u8be5\u65b9\u6848\u53ea\u6253\u901a Kubernetes \u9ed8\u8ba4\u5b50\u7f51\u548c OpenStack \u7684\u9009\u5b9a VPC\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/with-openstack/#ovn-ic","text":"\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\uff1a docker run --name = ovn-ic-db -d --network = host -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh","title":"\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93"},{"location":"advance/with-openstack/#kubernetes","text":"\u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : \u662f\u5426\u5f00\u542f\u96c6\u7fa4\u4e92\u8054\u3002 az-name : \u533a\u5206\u4e0d\u540c\u96c6\u7fa4\u7684\u96c6\u7fa4\u540d\u79f0\uff0c\u6bcf\u4e2a\u4e92\u8054\u96c6\u7fa4\u9700\u4e0d\u540c\u3002 ic-db-host : \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u5730\u5740\u3002 ic-nb-port : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6645\u3002 ic-sb-port : OVN-IC \u5357\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6646\u3002 gw-nodes : \u96c6\u7fa4\u4e92\u8054\u4e2d\u627f\u62c5\u7f51\u5173\u5de5\u4f5c\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 auto-route : \u662f\u5426\u81ea\u52a8\u5bf9\u5916\u53d1\u5e03\u548c\u5b66\u4e60\u8def\u7531\u3002","title":"Kubernetes \u4fa7\u64cd\u4f5c"},{"location":"advance/with-openstack/#openstack_1","text":"\u521b\u5efa\u548c Kubernetes \u4e92\u8054\u7684\u903b\u8f91\u8def\u7531\u5668\uff1a # openstack router create router0 # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | d5b38655-249a-4192-8046-71aa4d2b4af1 | router0 | ACTIVE | UP | 98a29ab7388347e7b5ff8bdd181ba4f9 | +--------------------------------------+---------+--------+-------+----------------------------------+ \u5728 OpenStack \u5185\u7684 OVN \u5317\u5411\u6570\u636e\u5e93\u4e2d\u8bbe\u7f6e\u53ef\u7528\u533a\u540d\u5b57\uff0c\u8be5\u540d\u79f0\u9700\u548c\u5176\u4ed6\u4e92\u8054\u96c6\u7fa4\u4e0d\u540c\uff1a ovn-nbctl set NB_Global . name = op-az \u5728\u53ef\u8bbf\u95ee OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u542f\u52a8 OVN-IC \u63a7\u5236\u5668\uff1a /usr/share/ovn/scripts/ovn-ctl --ovn-ic-nb-db = tcp:192.168.65.3:6645 \\ --ovn-ic-sb-db = tcp:192.168.65.3:6646 \\ --ovn-northd-nb-db = unix:/run/ovn/ovnnb_db.sock \\ --ovn-northd-sb-db = unix:/run/ovn/ovnsb_db.sock \\ start_ic ovn-ic-nb-db \uff0c ovn-ic-sb-db : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u548c\u5357\u5411\u6570\u636e\u5e93\u5730\u5740\u3002 ovn-northd-nb-db \uff0c ovn-northd-sb-db : \u5f53\u524d\u96c6\u7fa4 OVN \u5317\u5411\u6570\u636e\u5e93\u548c\u5357\u5411\u6570\u636e\u5730\u5740\u3002 \u914d\u7f6e\u4e92\u8054\u7f51\u5173\u8282\u70b9\uff1a ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn = true \u63a5\u4e0b\u6765\u9700\u8981\u5728 OpenStack \u7684 OVN \u5185\u8fdb\u884c\u64cd\u4f5c\u521b\u5efa\u903b\u8f91\u62d3\u6251\u3002 \u8fde\u63a5 ts \u4e92\u8054\u4ea4\u6362\u673a\u548c router0 \u903b\u8f91\u8def\u7531\u5668\uff0c\u5e76\u8bbe\u7f6e\u76f8\u5173\u89c4\u5219\uff1a ovn-nbctl lrp-add router0 lrp-router0-ts 00 :02:ef:11:39:4f 169 .254.100.73/24 ovn-nbctl lsp-add ts lsp-ts-router0 -- lsp-set-addresses lsp-ts-router0 router \\ -- lsp-set-type lsp-ts-router0 router \\ -- lsp-set-options lsp-ts-router0 router-port = lrp-router0-ts ovn-nbctl lrp-set-gateway-chassis lrp-router0-ts { gateway chassis } 1000 ovn-nbctl set NB_Global . options:ic-route-adv = true options:ic-route-learn = true \u9a8c\u8bc1\u5df2\u5b66\u4e60\u5230 Kubernetes \u8def\u7531\u89c4\u5219\uff1a # ovn-nbctl lr-route-list router0 IPv4 Routes 10 .0.0.22 169 .254.100.34 dst-ip ( learned ) 10 .16.0.0/16 169 .254.100.34 dst-ip ( learned ) \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728 router0 \u7f51\u7edc\u4e0b\u521b\u5efa\u865a\u673a\u9a8c\u8bc1\u662f\u5426\u53ef\u4ee5\u548c Kubernetes \u4e0b Pod \u4e92\u901a\u3002","title":"OpenStack \u4fa7\u64cd\u4f5c"},{"location":"advance/with-openstack/#ovn","text":"\u5728\u8be5\u65b9\u6848\u4e0b\uff0cOpenStack \u548c Kubernetes \u5171\u4eab\u4f7f\u7528\u540c\u4e00\u4e2a OVN\uff0c\u56e0\u6b64\u53ef\u4ee5\u5c06\u4e24\u8005\u7684 VPC \u548c Subnet \u7b49\u6982\u5ff5\u62c9\u9f50\uff0c\u5b9e\u73b0\u66f4\u597d\u7684\u63a7\u5236\u548c\u4e92\u8054\u3002 \u5728\u8be5\u6a21\u5f0f\u4e0b\u6211\u4eec\u6b63\u5e38\u4f7f\u7528 Kube-OVN \u90e8\u7f72 OVN\uff0cOpenStack \u4fee\u6539 Neutron \u914d\u7f6e\u5b9e\u73b0\u8fde\u63a5\u540c\u4e00\u4e2a OVN \u6570\u636e\u5e93\u3002OpenStack \u9700\u4f7f\u7528 networking-ovn \u4f5c\u4e3a Neutron \u540e\u7aef\u5b9e\u73b0\u3002","title":"\u5171\u4eab\u5e95\u5c42 OVN"},{"location":"advance/with-openstack/#neutron","text":"\u4fee\u6539 Neutron \u914d\u7f6e\u6587\u4ef6 /etc/neutron/plugins/ml2/ml2_conf.ini \uff1a [ ovn ] ... ovn_nb_connection = tcp: [ 192 .168.137.176 ] :6641,tcp: [ 192 .168.137.177 ] :6641,tcp: [ 192 .168.137.178 ] :6641 ovn_sb_connection = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovn_l3_scheduler = OVN_L3_SCHEDULER ovn_nb_connection \uff0c ovn_sb_connection : \u5730\u5740\u9700\u4fee\u6539\u4e3a Kube-OVN \u90e8\u7f72 ovn-central \u8282\u70b9\u7684\u5730\u5740\u3002 \u4fee\u6539\u6bcf\u4e2a\u8282\u70b9\u7684 OVS \u914d\u7f6e\uff1a ovs-vsctl set open . external-ids:ovn-remote = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovs-vsctl set open . external-ids:ovn-encap-type = geneve ovs-vsctl set open . external-ids:ovn-encap-ip = 192 .168.137.200 external-ids:ovn-remote : \u5730\u5740\u9700\u4fee\u6539\u4e3a Kube-OVN \u90e8\u7f72 ovn-central \u8282\u70b9\u7684\u5730\u5740\u3002 ovn-encap-ip : \u4fee\u6539\u4e3a\u5f53\u524d\u8282\u70b9\u7684 IP \u5730\u5740\u3002","title":"Neutron \u914d\u7f6e\u4fee\u6539"},{"location":"advance/with-openstack/#kubernetes-openstack","text":"\u63a5\u4e0b\u6765\u4ecb\u7ecd\u5982\u4f55\u5728 Kubernetes \u4e2d\u67e5\u8be2 OpenStack \u7684\u7f51\u7edc\u8d44\u6e90\u5e76\u5728 OpenStack \u7684\u5b50\u7f51\u4e2d\u521b\u5efa Pod\u3002 \u67e5\u8be2 OpenStack \u4e2d\u5df2\u6709\u7684\u7f51\u7edc\u8d44\u6e90\uff0c\u5982\u4e0b\u8d44\u6e90\u5df2\u7ecf\u9884\u5148\u521b\u5efa\u5b8c\u6210\uff1a # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | 22040ed5-0598-4f77-bffd-e7fd4db47e93 | router0 | ACTIVE | UP | 62381a21d569404aa236a5dd8712449c | +--------------------------------------+---------+--------+-------+----------------------------------+ # openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | cd59e36a-37db-4c27-b709-d35379a7920f | provider | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | +--------------------------------------+----------+--------------------------------------+ # openstack subnet list +--------------------------------------+-------------+--------------------------------------+----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------------+--------------------------------------+----------------+ | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | provider-v4 | cd59e36a-37db-4c27-b709-d35379a7920f | 192 .168.1.0/24 | +--------------------------------------+-------------+--------------------------------------+----------------+ # openstack server list +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | 8433d622-a8d6-41a7-8b31-49abfd64f639 | provider-instance | ACTIVE | provider = 192 .168.1.61 | ubuntu | m1 | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ \u5728 Kubernetes \u4fa7\uff0c\u67e5\u8be2 VPC \u8d44\u6e90\uff1a # kubectl get vpc NAME STANDBY SUBNETS neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 true [ \"neutron-cd59e36a-37db-4c27-b709-d35379a7920f\" ] ovn-cluster true [ \"join\" , \"ovn-default\" ] neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 \u4e3a\u4ece OpenStack \u540c\u6b65\u8fc7\u6765\u7684 VPC \u8d44\u6e90\u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u6309\u7167 Kube-OVN \u539f\u751f\u7684 VPC \u548c Subnet \u64cd\u4f5c\u521b\u5efa Pod \u5e76\u8fd0\u884c\u3002 VPC, Subnet \u7ed1\u5b9a Namespace net2 \uff0c\u5e76\u521b\u5efa Pod: apiVersion : v1 kind : Namespace metadata : name : net2 --- apiVersion : kubeovn.io/v1 kind : Vpc metadata : creationTimestamp : \"2021-06-20T13:34:11Z\" generation : 2 labels : ovn.kubernetes.io/vpc_external : \"true\" name : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 resourceVersion : \"583728\" uid : 18d4c654-f511-4def-a3a0-a6434d237c1e spec : namespaces : - net2 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 namespaces : - net2 cidrBlock : 12.0.1.0/24 natOutgoing : false --- apiVersion : v1 kind : Pod metadata : name : ubuntu namespace : net2 spec : containers : - image : docker.io/kubeovn/kube-ovn:v1.8.0 command : - \"sleep\" - \"604800\" imagePullPolicy : IfNotPresent name : ubuntu restartPolicy : Always \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5728 Kubernetes \u4e2d\u4f7f\u7528 OpenStack \u5185\u8d44\u6e90"},{"location":"advance/with-ovn-ic/","text":"\u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054 \u00b6 Kube-OVN \u652f\u6301\u901a\u8fc7 OVN-IC \u5c06\u4e24\u4e2a Kubernetes \u96c6\u7fa4 Pod \u7f51\u7edc\u6253\u901a\uff0c\u6253\u901a\u540e\u7684\u4e24\u4e2a\u96c6\u7fa4\u5185\u7684 Pod \u53ef\u4ee5\u901a\u8fc7 Pod IP \u8fdb\u884c\u76f4\u63a5\u901a\u4fe1\u3002 Kube-OVN \u4f7f\u7528\u96a7\u9053\u5bf9\u8de8\u96c6\u7fa4\u6d41\u91cf\u8fdb\u884c\u5c01\u88c5\uff0c\u4e24\u4e2a\u96c6\u7fa4\u4e4b\u95f4\u53ea\u8981\u5b58\u5728\u4e00\u7ec4 IP \u53ef\u8fbe\u7684\u673a\u5668\u5373\u53ef\u5b8c\u6210\u5bb9\u5668\u7f51\u7edc\u7684\u4e92\u901a\u3002 \u8be5\u6a21\u5f0f\u7684\u591a\u96c6\u7fa4\u4e92\u8054\u4e3a Overlay \u7f51\u7edc\u529f\u80fd\uff0cUnderlay \u7f51\u7edc\u5982\u679c\u60f3\u8981\u5b9e\u73b0\u96c6\u7fa4\u4e92\u8054\u9700\u8981\u5e95\u5c42\u57fa\u7840\u8bbe\u65bd\u505a\u7f51\u7edc\u6253\u901a\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u81ea\u52a8\u4e92\u8054\u6a21\u5f0f\u4e0b\u4e0d\u540c\u96c6\u7fa4\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u76f8\u4e92\u91cd\u53e0\uff0c\u9ed8\u8ba4\u5b50\u7f51\u9700\u5728\u5b89\u88c5\u65f6\u914d\u7f6e\u4e3a\u4e0d\u91cd\u53e0\u7684\u7f51\u6bb5\u3002\u82e5\u5b58\u5728\u91cd\u53e0\u9700\u53c2\u8003\u540e\u7eed\u624b\u52a8\u4e92\u8054\u8fc7\u7a0b\uff0c\u53ea\u80fd\u5c06\u4e0d\u91cd\u53e0\u7f51\u6bb5\u6253\u901a\u3002 \u9700\u8981\u5b58\u5728\u4e00\u7ec4\u673a\u5668\u53ef\u4ee5\u88ab\u6bcf\u4e2a\u96c6\u7fa4\u7684 kube-ovn-controller \u901a\u8fc7 IP \u8bbf\u95ee\uff0c\u7528\u6765\u90e8\u7f72\u8de8\u96c6\u7fa4\u4e92\u8054\u7684\u63a7\u5236\u5668\u3002 \u6bcf\u4e2a\u96c6\u7fa4\u9700\u8981\u6709\u4e00\u7ec4\u53ef\u4ee5\u901a\u8fc7 IP \u8fdb\u884c\u8de8\u96c6\u7fa4\u4e92\u8bbf\u7684\u673a\u5668\u4f5c\u4e3a\u4e4b\u540e\u7684\u7f51\u5173\u8282\u70b9\u3002 \u8be5\u529f\u80fd\u53ea\u5bf9\u9ed8\u8ba4 VPC \u751f\u6548\uff0c\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u65e0\u6cd5\u4f7f\u7528\u4e92\u8054\u529f\u80fd\u3002 \u90e8\u7f72\u5355\u8282\u70b9 OVN-IC \u6570\u636e\u5e93 \u00b6 \u5728\u6bcf\u4e2a\u96c6\u7fa4 kube-ovn-controller \u53ef\u901a\u8fc7 IP \u8bbf\u95ee\u7684\u673a\u5668\u4e0a\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\uff0c\u8be5\u8282\u70b9\u5c06\u4fdd\u5b58\u5404\u4e2a\u96c6\u7fa4\u540c\u6b65\u4e0a\u6765\u7684\u7f51\u7edc\u914d\u7f6e\u4fe1\u606f\u3002 \u90e8\u7f72 docker \u7684\u73af\u5883\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5bf9\u4e8e\u90e8\u7f72 containerd \u53d6\u4ee3 docker \u7684\u73af\u5883\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh \u81ea\u52a8\u8def\u7531\u8bbe\u7f6e \u00b6 \u5728\u81ea\u52a8\u8def\u7531\u8bbe\u7f6e\u4e0b\uff0c\u6bcf\u4e2a\u96c6\u7fa4\u4f1a\u5c06\u81ea\u5df1\u9ed8\u8ba4 VPC \u4e0b Subnet \u7684 CIDR \u4fe1\u606f\u540c\u6b65\u7ed9 OVN-IC \uff0c\u56e0\u6b64\u8981\u786e\u4fdd\u4e24\u4e2a\u96c6\u7fa4\u7684 Subnet CIDR \u4e0d\u5b58\u5728\u91cd\u53e0\u3002 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : \u662f\u5426\u5f00\u542f\u96c6\u7fa4\u4e92\u8054\u3002 az-name : \u533a\u5206\u4e0d\u540c\u96c6\u7fa4\u7684\u96c6\u7fa4\u540d\u79f0\uff0c\u6bcf\u4e2a\u4e92\u8054\u96c6\u7fa4\u9700\u4e0d\u540c\u3002 ic-db-host : \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u5730\u5740\u3002 ic-nb-port : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6645\u3002 ic-sb-port : OVN-IC \u5357\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6646\u3002 gw-nodes : \u96c6\u7fa4\u4e92\u8054\u4e2d\u627f\u62c5\u7f51\u5173\u5de5\u4f5c\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 auto-route : \u662f\u5426\u81ea\u52a8\u5bf9\u5916\u53d1\u5e03\u548c\u5b66\u4e60\u8def\u7531\u3002 \u6ce8\u610f\uff1a \u4e3a\u4e86\u4fdd\u8bc1\u64cd\u4f5c\u7684\u6b63\u786e\u6027\uff0c ovn-ic-config \u8fd9\u4e2a ConfigMap \u4e0d\u5141\u8bb8\u4fee\u6539\u3002\u5982\u6709\u53c2\u6570\u9700\u8981\u53d8\u66f4\uff0c\u8bf7\u5220\u9664\u8be5 ConfigMap\uff0c\u4fee\u6539\u540e\u518d\u5e94\u7528\u6b64 ConfigMap\u3002 \u5728 ovn-ic \u5bb9\u5668\u5185\u901a\u8fc7\u4e0b\u9762\u7684\u547d\u4ee4\u67e5\u770b\u662f\u5426\u5df2\u5efa\u7acb\u4e92\u8054\u903b\u8f91\u4ea4\u6362\u673a ts \uff1a # ovn-ic-sbctl show availability-zone az1 gateway deee03e0-af16-4f45-91e9-b50c3960f809 hostname: az1-gw type: geneve ip: 192 .168.42.145 port ts-az1 transit switch: ts address: [ \"00:00:00:50:AC:8C 169.254.100.45/24\" ] availability-zone az2 gateway e94cc831-8143-40e3-a478-90352773327b hostname: az2-gw type: geneve ip: 192 .168.42.149 port ts-az2 transit switch: ts address: [ \"00:00:00:07:4A:59 169.254.100.63/24\" ] \u5728\u6bcf\u4e2a\u96c6\u7fa4\u89c2\u5bdf\u903b\u8f91\u8def\u7531\u662f\u5426\u6709\u5b66\u4e60\u5230\u7684\u5bf9\u7aef\u8def\u7531\uff1a # kubectl ko nbctl lr-route-list ovn-cluster IPv4 Routes 10 .42.1.1 169 .254.100.45 dst-ip ( learned ) 10 .42.1.3 100 .64.0.2 dst-ip 10 .16.0.2 100 .64.0.2 src-ip 10 .16.0.3 100 .64.0.2 src-ip 10 .16.0.4 100 .64.0.2 src-ip 10 .16.0.6 100 .64.0.2 src-ip 10 .17.0.0/16 169 .254.100.45 dst-ip ( learned ) 100 .65.0.0/16 169 .254.100.45 dst-ip ( learned ) \u63a5\u4e0b\u6765\u53ef\u4ee5\u5c1d\u8bd5\u5728\u96c6\u7fa4 1 \u5185\u7684\u4e00\u4e2a Pod \u5185\u76f4\u63a5 ping \u96c6\u7fa4 2 \u5185\u7684\u4e00\u4e2a Pod IP \u89c2\u5bdf\u662f\u5426\u53ef\u4ee5\u8054\u901a\u3002 \u5bf9\u4e8e\u67d0\u4e2a\u4e0d\u60f3\u5bf9\u5916\u81ea\u52a8\u53d1\u5e03\u8def\u7531\u7684\u5b50\u7f51\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 Subnet \u91cc\u7684 disableInterConnection \u6765\u7981\u6b62\u8def\u7531\u5e7f\u64ad\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : no-advertise spec : cidrBlock : 10.199.0.0/16 disableInterConnection : true \u624b\u52a8\u8def\u7531\u8bbe\u7f6e \u00b6 \u5bf9\u4e8e\u96c6\u7fa4\u95f4\u5b58\u5728\u91cd\u53e0 CIDR \u53ea\u5e0c\u671b\u505a\u90e8\u5206\u5b50\u7f51\u6253\u901a\u7684\u60c5\u51b5\uff0c\u53ef\u4ee5\u901a\u8fc7\u4e0b\u9762\u7684\u6b65\u9aa4\u624b\u52a8\u53d1\u5e03\u5b50\u7f51\u8def\u7531\u3002 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff0c\u5e76\u5c06 auto-route \u8bbe\u7f6e\u4e3a false \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"false\" \u5728\u6bcf\u4e2a\u96c6\u7fa4\u5206\u522b\u67e5\u770b\u8fdc\u7aef\u903b\u8f91\u7aef\u53e3\u7684\u5730\u5740\uff0c\u7528\u4e8e\u4e4b\u540e\u624b\u52a8\u914d\u7f6e\u8def\u7531\uff1a [ root@az1 ~ ] # kubectl ko nbctl show switch a391d3a1-14a0-4841-9836-4bd930c447fb ( ts ) port ts-az1 type: router router-port: az1-ts port ts-az2 type: remote addresses: [ \"00:00:00:4B:E2:9F 169.254.100.31/24\" ] [ root@az2 ~ ] # kubectl ko nbctl show switch da6138b8-de81-4908-abf9-b2224ec4edf3 ( ts ) port ts-az2 type: router router-port: az2-ts port ts-az1 type: remote addresses: [ \"00:00:00:FB:2A:F7 169.254.100.79/24\" ] \u7531\u4e0a\u8f93\u51fa\u53ef\u77e5\uff0c\u96c6\u7fa4 az1 \u5230 \u96c6\u7fa4 az2 \u7684\u8fdc\u7aef\u5730\u5740\u4e3a 169.254.100.31 \uff0c az2 \u5230 az1 \u7684\u8fdc\u7aef\u5730\u5740\u4e3a 169.254.100.79 \u3002 \u4e0b\u9762\u624b\u52a8\u8bbe\u7f6e\u8def\u7531\uff0c\u5728\u8be5\u4f8b\u5b50\u4e2d\uff0c\u96c6\u7fa4 az1 \u5185\u7684\u5b50\u7f51 CIDR \u4e3a 10.16.0.0/24 \uff0c\u96c6\u7fa4 az2 \u5185\u7684\u5b50\u7f51 CIDR \u4e3a 10.17.0.0/24 \u3002 \u5728\u96c6\u7fa4 az1 \u8bbe\u7f6e\u5230\u96c6\u7fa4 az2 \u7684\u8def\u7531: kubectl ko nbctl lr-route-add ovn-cluster 10 .17.0.0/24 169 .254.100.31 \u5728\u96c6\u7fa4 az2 \u8bbe\u7f6e\u5230\u96c6\u7fa4 az1 \u7684\u8def\u7531: kubectl ko nbctl lr-route-add ovn-cluster 10 .16.0.0/24 169 .254.100.79 \u9ad8\u53ef\u7528 OVN-IC \u6570\u636e\u5e93\u90e8\u7f72 \u00b6 OVN-IC \u6570\u636e\u5e93\u4e4b\u95f4\u53ef\u4ee5\u901a\u8fc7 Raft \u534f\u8bae\u7ec4\u6210\u4e00\u4e2a\u9ad8\u53ef\u7528\u96c6\u7fa4\uff0c\u8be5\u90e8\u7f72\u6a21\u5f0f\u9700\u8981\u81f3\u5c11 3 \u4e2a\u8282\u70b9\u3002 \u9996\u5148\u5728\u7b2c\u4e00\u4e2a\u8282\u70b9\u4e0a\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\u7684 leader\u3002 \u90e8\u7f72 docker \u73af\u5883\u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.3\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5982\u679c\u662f\u90e8\u7f72 containerd \u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a \u5f53\u524d\u5bb9\u5668\u6240\u5728\u8282\u70b9 IP \u5730\u5740\u3002 NODE_IPS \uff1a \u8fd0\u884c OVN-IC \u6570\u636e\u5e93\u7684\u4e09\u4e2a\u8282\u70b9 IP \u5730\u5740\uff0c\u4f7f\u7528\u9017\u53f7\u8fdb\u884c\u5206\u9694\u3002 \u63a5\u4e0b\u6765\uff0c\u5728\u53e6\u5916\u4e24\u4e2a\u8282\u70b9\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684 follower\u3002 \u90e8\u7f72 docker \u73af\u5883\u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.2\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" -e LEADER_IP = \"192.168.65.3\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5982\u679c\u662f\u90e8\u7f72 containerd \u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.2 \"\" --env = \"LEADER_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a \u5f53\u524d\u5bb9\u5668\u6240\u5728\u8282\u70b9 IP \u5730\u5740\u3002 NODE_IPS \uff1a \u8fd0\u884c OVN-IC \u6570\u636e\u5e93\u7684\u4e09\u4e2a\u8282\u70b9 IP \u5730\u5740\uff0c\u4f7f\u7528\u9017\u53f7\u8fdb\u884c\u5206\u9694\u3002 LEADER_IP : \u8fd0\u884c OVN-IC \u6570\u636e\u5e93 leader \u8282\u70b9\u7684 IP \u5730\u5740\u3002 \u5728\u6bcf\u4e2a\u96c6\u7fa4\u521b\u5efa ovn-ic-config \u65f6\u6307\u5b9a\u591a\u4e2a OVN-IC \u6570\u636e\u5e93\u8282\u70b9\u5730\u5740\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3,192.168.65.2,192.168.65.1\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" \u624b\u52a8\u91cd\u7f6e \u00b6 \u5728\u4e00\u4e9b\u60c5\u51b5\u4e0b\uff0c\u7531\u4e8e\u914d\u7f6e\u9519\u8bef\u9700\u8981\u5bf9\u6574\u4e2a\u4e92\u8054\u914d\u7f6e\u8fdb\u884c\u6e05\u7406\uff0c\u53ef\u4ee5\u53c2\u8003\u4e0b\u9762\u7684\u6b65\u9aa4\u6e05\u7406\u73af\u5883\u3002 \u5220\u9664\u5f53\u524d\u7684 ovn-ic-config Configmap\uff1a kubectl -n kube-system delete cm ovn-ic-config \u5220\u9664 ts \u903b\u8f91\u4ea4\u6362\u673a\uff1a kubectl-ko nbctl ls-del ts \u5728\u5bf9\u7aef\u96c6\u7fa4\u91cd\u590d\u540c\u6837\u7684\u6b65\u9aa4\u3002 \u4fee\u6539 az-name \u00b6 \u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7 kubectl edit \u7684\u65b9\u5f0f\u5bf9 ovn-ic-config \u8fd9\u4e2a configmap \u4e2d\u7684 az-name \u5b57\u6bb5\u8fdb\u884c\u4fee\u6539\u3002 \u4f46\u662f\u9700\u8981\u5728\u6bcf\u4e2a ovn-cni pod \u4e0a\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u5426\u5219\u53ef\u80fd\u51fa\u73b0\u6700\u957f 10 \u5206\u949f\u7684\u8de8\u96c6\u7fa4\u7f51\u7edc\u4e2d\u65ad\u3002 ovn-appctl -t ovn-controller inc-engine/recompute \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-ovn-ic/#ovn-ic","text":"Kube-OVN \u652f\u6301\u901a\u8fc7 OVN-IC \u5c06\u4e24\u4e2a Kubernetes \u96c6\u7fa4 Pod \u7f51\u7edc\u6253\u901a\uff0c\u6253\u901a\u540e\u7684\u4e24\u4e2a\u96c6\u7fa4\u5185\u7684 Pod \u53ef\u4ee5\u901a\u8fc7 Pod IP \u8fdb\u884c\u76f4\u63a5\u901a\u4fe1\u3002 Kube-OVN \u4f7f\u7528\u96a7\u9053\u5bf9\u8de8\u96c6\u7fa4\u6d41\u91cf\u8fdb\u884c\u5c01\u88c5\uff0c\u4e24\u4e2a\u96c6\u7fa4\u4e4b\u95f4\u53ea\u8981\u5b58\u5728\u4e00\u7ec4 IP \u53ef\u8fbe\u7684\u673a\u5668\u5373\u53ef\u5b8c\u6210\u5bb9\u5668\u7f51\u7edc\u7684\u4e92\u901a\u3002 \u8be5\u6a21\u5f0f\u7684\u591a\u96c6\u7fa4\u4e92\u8054\u4e3a Overlay \u7f51\u7edc\u529f\u80fd\uff0cUnderlay \u7f51\u7edc\u5982\u679c\u60f3\u8981\u5b9e\u73b0\u96c6\u7fa4\u4e92\u8054\u9700\u8981\u5e95\u5c42\u57fa\u7840\u8bbe\u65bd\u505a\u7f51\u7edc\u6253\u901a\u3002","title":"\u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-ovn-ic/#_1","text":"\u81ea\u52a8\u4e92\u8054\u6a21\u5f0f\u4e0b\u4e0d\u540c\u96c6\u7fa4\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u76f8\u4e92\u91cd\u53e0\uff0c\u9ed8\u8ba4\u5b50\u7f51\u9700\u5728\u5b89\u88c5\u65f6\u914d\u7f6e\u4e3a\u4e0d\u91cd\u53e0\u7684\u7f51\u6bb5\u3002\u82e5\u5b58\u5728\u91cd\u53e0\u9700\u53c2\u8003\u540e\u7eed\u624b\u52a8\u4e92\u8054\u8fc7\u7a0b\uff0c\u53ea\u80fd\u5c06\u4e0d\u91cd\u53e0\u7f51\u6bb5\u6253\u901a\u3002 \u9700\u8981\u5b58\u5728\u4e00\u7ec4\u673a\u5668\u53ef\u4ee5\u88ab\u6bcf\u4e2a\u96c6\u7fa4\u7684 kube-ovn-controller \u901a\u8fc7 IP \u8bbf\u95ee\uff0c\u7528\u6765\u90e8\u7f72\u8de8\u96c6\u7fa4\u4e92\u8054\u7684\u63a7\u5236\u5668\u3002 \u6bcf\u4e2a\u96c6\u7fa4\u9700\u8981\u6709\u4e00\u7ec4\u53ef\u4ee5\u901a\u8fc7 IP \u8fdb\u884c\u8de8\u96c6\u7fa4\u4e92\u8bbf\u7684\u673a\u5668\u4f5c\u4e3a\u4e4b\u540e\u7684\u7f51\u5173\u8282\u70b9\u3002 \u8be5\u529f\u80fd\u53ea\u5bf9\u9ed8\u8ba4 VPC \u751f\u6548\uff0c\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u65e0\u6cd5\u4f7f\u7528\u4e92\u8054\u529f\u80fd\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/with-ovn-ic/#ovn-ic_1","text":"\u5728\u6bcf\u4e2a\u96c6\u7fa4 kube-ovn-controller \u53ef\u901a\u8fc7 IP \u8bbf\u95ee\u7684\u673a\u5668\u4e0a\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\uff0c\u8be5\u8282\u70b9\u5c06\u4fdd\u5b58\u5404\u4e2a\u96c6\u7fa4\u540c\u6b65\u4e0a\u6765\u7684\u7f51\u7edc\u914d\u7f6e\u4fe1\u606f\u3002 \u90e8\u7f72 docker \u7684\u73af\u5883\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5bf9\u4e8e\u90e8\u7f72 containerd \u53d6\u4ee3 docker \u7684\u73af\u5883\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh","title":"\u90e8\u7f72\u5355\u8282\u70b9 OVN-IC \u6570\u636e\u5e93"},{"location":"advance/with-ovn-ic/#_2","text":"\u5728\u81ea\u52a8\u8def\u7531\u8bbe\u7f6e\u4e0b\uff0c\u6bcf\u4e2a\u96c6\u7fa4\u4f1a\u5c06\u81ea\u5df1\u9ed8\u8ba4 VPC \u4e0b Subnet \u7684 CIDR \u4fe1\u606f\u540c\u6b65\u7ed9 OVN-IC \uff0c\u56e0\u6b64\u8981\u786e\u4fdd\u4e24\u4e2a\u96c6\u7fa4\u7684 Subnet CIDR \u4e0d\u5b58\u5728\u91cd\u53e0\u3002 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : \u662f\u5426\u5f00\u542f\u96c6\u7fa4\u4e92\u8054\u3002 az-name : \u533a\u5206\u4e0d\u540c\u96c6\u7fa4\u7684\u96c6\u7fa4\u540d\u79f0\uff0c\u6bcf\u4e2a\u4e92\u8054\u96c6\u7fa4\u9700\u4e0d\u540c\u3002 ic-db-host : \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u5730\u5740\u3002 ic-nb-port : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6645\u3002 ic-sb-port : OVN-IC \u5357\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6646\u3002 gw-nodes : \u96c6\u7fa4\u4e92\u8054\u4e2d\u627f\u62c5\u7f51\u5173\u5de5\u4f5c\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 auto-route : \u662f\u5426\u81ea\u52a8\u5bf9\u5916\u53d1\u5e03\u548c\u5b66\u4e60\u8def\u7531\u3002 \u6ce8\u610f\uff1a \u4e3a\u4e86\u4fdd\u8bc1\u64cd\u4f5c\u7684\u6b63\u786e\u6027\uff0c ovn-ic-config \u8fd9\u4e2a ConfigMap \u4e0d\u5141\u8bb8\u4fee\u6539\u3002\u5982\u6709\u53c2\u6570\u9700\u8981\u53d8\u66f4\uff0c\u8bf7\u5220\u9664\u8be5 ConfigMap\uff0c\u4fee\u6539\u540e\u518d\u5e94\u7528\u6b64 ConfigMap\u3002 \u5728 ovn-ic \u5bb9\u5668\u5185\u901a\u8fc7\u4e0b\u9762\u7684\u547d\u4ee4\u67e5\u770b\u662f\u5426\u5df2\u5efa\u7acb\u4e92\u8054\u903b\u8f91\u4ea4\u6362\u673a ts \uff1a # ovn-ic-sbctl show availability-zone az1 gateway deee03e0-af16-4f45-91e9-b50c3960f809 hostname: az1-gw type: geneve ip: 192 .168.42.145 port ts-az1 transit switch: ts address: [ \"00:00:00:50:AC:8C 169.254.100.45/24\" ] availability-zone az2 gateway e94cc831-8143-40e3-a478-90352773327b hostname: az2-gw type: geneve ip: 192 .168.42.149 port ts-az2 transit switch: ts address: [ \"00:00:00:07:4A:59 169.254.100.63/24\" ] \u5728\u6bcf\u4e2a\u96c6\u7fa4\u89c2\u5bdf\u903b\u8f91\u8def\u7531\u662f\u5426\u6709\u5b66\u4e60\u5230\u7684\u5bf9\u7aef\u8def\u7531\uff1a # kubectl ko nbctl lr-route-list ovn-cluster IPv4 Routes 10 .42.1.1 169 .254.100.45 dst-ip ( learned ) 10 .42.1.3 100 .64.0.2 dst-ip 10 .16.0.2 100 .64.0.2 src-ip 10 .16.0.3 100 .64.0.2 src-ip 10 .16.0.4 100 .64.0.2 src-ip 10 .16.0.6 100 .64.0.2 src-ip 10 .17.0.0/16 169 .254.100.45 dst-ip ( learned ) 100 .65.0.0/16 169 .254.100.45 dst-ip ( learned ) \u63a5\u4e0b\u6765\u53ef\u4ee5\u5c1d\u8bd5\u5728\u96c6\u7fa4 1 \u5185\u7684\u4e00\u4e2a Pod \u5185\u76f4\u63a5 ping \u96c6\u7fa4 2 \u5185\u7684\u4e00\u4e2a Pod IP \u89c2\u5bdf\u662f\u5426\u53ef\u4ee5\u8054\u901a\u3002 \u5bf9\u4e8e\u67d0\u4e2a\u4e0d\u60f3\u5bf9\u5916\u81ea\u52a8\u53d1\u5e03\u8def\u7531\u7684\u5b50\u7f51\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 Subnet \u91cc\u7684 disableInterConnection \u6765\u7981\u6b62\u8def\u7531\u5e7f\u64ad\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : no-advertise spec : cidrBlock : 10.199.0.0/16 disableInterConnection : true","title":"\u81ea\u52a8\u8def\u7531\u8bbe\u7f6e"},{"location":"advance/with-ovn-ic/#_3","text":"\u5bf9\u4e8e\u96c6\u7fa4\u95f4\u5b58\u5728\u91cd\u53e0 CIDR \u53ea\u5e0c\u671b\u505a\u90e8\u5206\u5b50\u7f51\u6253\u901a\u7684\u60c5\u51b5\uff0c\u53ef\u4ee5\u901a\u8fc7\u4e0b\u9762\u7684\u6b65\u9aa4\u624b\u52a8\u53d1\u5e03\u5b50\u7f51\u8def\u7531\u3002 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff0c\u5e76\u5c06 auto-route \u8bbe\u7f6e\u4e3a false \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"false\" \u5728\u6bcf\u4e2a\u96c6\u7fa4\u5206\u522b\u67e5\u770b\u8fdc\u7aef\u903b\u8f91\u7aef\u53e3\u7684\u5730\u5740\uff0c\u7528\u4e8e\u4e4b\u540e\u624b\u52a8\u914d\u7f6e\u8def\u7531\uff1a [ root@az1 ~ ] # kubectl ko nbctl show switch a391d3a1-14a0-4841-9836-4bd930c447fb ( ts ) port ts-az1 type: router router-port: az1-ts port ts-az2 type: remote addresses: [ \"00:00:00:4B:E2:9F 169.254.100.31/24\" ] [ root@az2 ~ ] # kubectl ko nbctl show switch da6138b8-de81-4908-abf9-b2224ec4edf3 ( ts ) port ts-az2 type: router router-port: az2-ts port ts-az1 type: remote addresses: [ \"00:00:00:FB:2A:F7 169.254.100.79/24\" ] \u7531\u4e0a\u8f93\u51fa\u53ef\u77e5\uff0c\u96c6\u7fa4 az1 \u5230 \u96c6\u7fa4 az2 \u7684\u8fdc\u7aef\u5730\u5740\u4e3a 169.254.100.31 \uff0c az2 \u5230 az1 \u7684\u8fdc\u7aef\u5730\u5740\u4e3a 169.254.100.79 \u3002 \u4e0b\u9762\u624b\u52a8\u8bbe\u7f6e\u8def\u7531\uff0c\u5728\u8be5\u4f8b\u5b50\u4e2d\uff0c\u96c6\u7fa4 az1 \u5185\u7684\u5b50\u7f51 CIDR \u4e3a 10.16.0.0/24 \uff0c\u96c6\u7fa4 az2 \u5185\u7684\u5b50\u7f51 CIDR \u4e3a 10.17.0.0/24 \u3002 \u5728\u96c6\u7fa4 az1 \u8bbe\u7f6e\u5230\u96c6\u7fa4 az2 \u7684\u8def\u7531: kubectl ko nbctl lr-route-add ovn-cluster 10 .17.0.0/24 169 .254.100.31 \u5728\u96c6\u7fa4 az2 \u8bbe\u7f6e\u5230\u96c6\u7fa4 az1 \u7684\u8def\u7531: kubectl ko nbctl lr-route-add ovn-cluster 10 .16.0.0/24 169 .254.100.79","title":"\u624b\u52a8\u8def\u7531\u8bbe\u7f6e"},{"location":"advance/with-ovn-ic/#ovn-ic_2","text":"OVN-IC \u6570\u636e\u5e93\u4e4b\u95f4\u53ef\u4ee5\u901a\u8fc7 Raft \u534f\u8bae\u7ec4\u6210\u4e00\u4e2a\u9ad8\u53ef\u7528\u96c6\u7fa4\uff0c\u8be5\u90e8\u7f72\u6a21\u5f0f\u9700\u8981\u81f3\u5c11 3 \u4e2a\u8282\u70b9\u3002 \u9996\u5148\u5728\u7b2c\u4e00\u4e2a\u8282\u70b9\u4e0a\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\u7684 leader\u3002 \u90e8\u7f72 docker \u73af\u5883\u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.3\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5982\u679c\u662f\u90e8\u7f72 containerd \u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a \u5f53\u524d\u5bb9\u5668\u6240\u5728\u8282\u70b9 IP \u5730\u5740\u3002 NODE_IPS \uff1a \u8fd0\u884c OVN-IC \u6570\u636e\u5e93\u7684\u4e09\u4e2a\u8282\u70b9 IP \u5730\u5740\uff0c\u4f7f\u7528\u9017\u53f7\u8fdb\u884c\u5206\u9694\u3002 \u63a5\u4e0b\u6765\uff0c\u5728\u53e6\u5916\u4e24\u4e2a\u8282\u70b9\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684 follower\u3002 \u90e8\u7f72 docker \u73af\u5883\u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.2\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" -e LEADER_IP = \"192.168.65.3\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5982\u679c\u662f\u90e8\u7f72 containerd \u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.2 \"\" --env = \"LEADER_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a \u5f53\u524d\u5bb9\u5668\u6240\u5728\u8282\u70b9 IP \u5730\u5740\u3002 NODE_IPS \uff1a \u8fd0\u884c OVN-IC \u6570\u636e\u5e93\u7684\u4e09\u4e2a\u8282\u70b9 IP \u5730\u5740\uff0c\u4f7f\u7528\u9017\u53f7\u8fdb\u884c\u5206\u9694\u3002 LEADER_IP : \u8fd0\u884c OVN-IC \u6570\u636e\u5e93 leader \u8282\u70b9\u7684 IP \u5730\u5740\u3002 \u5728\u6bcf\u4e2a\u96c6\u7fa4\u521b\u5efa ovn-ic-config \u65f6\u6307\u5b9a\u591a\u4e2a OVN-IC \u6570\u636e\u5e93\u8282\u70b9\u5730\u5740\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3,192.168.65.2,192.168.65.1\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\"","title":"\u9ad8\u53ef\u7528 OVN-IC \u6570\u636e\u5e93\u90e8\u7f72"},{"location":"advance/with-ovn-ic/#_4","text":"\u5728\u4e00\u4e9b\u60c5\u51b5\u4e0b\uff0c\u7531\u4e8e\u914d\u7f6e\u9519\u8bef\u9700\u8981\u5bf9\u6574\u4e2a\u4e92\u8054\u914d\u7f6e\u8fdb\u884c\u6e05\u7406\uff0c\u53ef\u4ee5\u53c2\u8003\u4e0b\u9762\u7684\u6b65\u9aa4\u6e05\u7406\u73af\u5883\u3002 \u5220\u9664\u5f53\u524d\u7684 ovn-ic-config Configmap\uff1a kubectl -n kube-system delete cm ovn-ic-config \u5220\u9664 ts \u903b\u8f91\u4ea4\u6362\u673a\uff1a kubectl-ko nbctl ls-del ts \u5728\u5bf9\u7aef\u96c6\u7fa4\u91cd\u590d\u540c\u6837\u7684\u6b65\u9aa4\u3002","title":"\u624b\u52a8\u91cd\u7f6e"},{"location":"advance/with-ovn-ic/#az-name","text":"\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7 kubectl edit \u7684\u65b9\u5f0f\u5bf9 ovn-ic-config \u8fd9\u4e2a configmap \u4e2d\u7684 az-name \u5b57\u6bb5\u8fdb\u884c\u4fee\u6539\u3002 \u4f46\u662f\u9700\u8981\u5728\u6bcf\u4e2a ovn-cni pod \u4e0a\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u5426\u5219\u53ef\u80fd\u51fa\u73b0\u6700\u957f 10 \u5206\u949f\u7684\u8de8\u96c6\u7fa4\u7f51\u7edc\u4e2d\u65ad\u3002 ovn-appctl -t ovn-controller inc-engine/recompute \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4fee\u6539 az-name"},{"location":"advance/with-submariner/","text":"\u4f7f\u7528 Submariner \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054 \u00b6 Submariner \u4f5c\u4e3a\u53ef\u4ee5\u6253\u901a\u591a\u4e2a Kubernetes \u96c6\u7fa4 Pod \u548c Service \u7f51\u7edc\u7684\u5f00\u6e90\u7f51\u7edc\u7ec4\u4ef6\uff0c\u80fd\u591f\u5e2e\u52a9 Kube-OVN \u5b9e\u73b0\u591a\u96c6\u7fa4\u4e92\u8054\u3002 \u76f8\u6bd4\u901a\u8fc7 OVN-IC \u6253\u901a\u591a\u96c6\u7fa4\u7f51\u7edc\u7684\u65b9\u5f0f\uff0cSubmariner \u53ef\u4ee5\u6253\u901a Kube-OVN \u548c\u975e Kube-OVN \u7684\u96c6\u7fa4\u7f51\u7edc\uff0c\u5e76 \u80fd\u63d0\u4f9b Service \u7684\u8de8\u96c6\u7fa4\u80fd\u529b\u3002\u4f46\u662f Submariner \u76ee\u524d\u53ea\u80fd\u5b9e\u73b0\u9ed8\u8ba4\u5b50\u7f51\u7684\u6253\u901a\uff0c\u65e0\u6cd5\u5b9e\u73b0\u591a\u5b50\u7f51\u9009\u62e9\u6027\u6253\u901a\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u4e24\u4e2a\u96c6\u7fa4\u7684 Service CIDR \u548c\u9ed8\u8ba4\u5b50\u7f51\u7684 CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 \u90e8\u7f72 Submariner \u00b6 \u4e0b\u8f7d subctl \u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u5e76\u90e8\u7f72\u5230\u76f8\u5e94\u8def\u5f84\uff1a curl -Ls https://get.submariner.io | bash export PATH = $PATH :~/.local/bin echo export PATH = \\$ PATH:~/.local/bin >> ~/.profile \u5207\u6362 kubeconfig \u81f3\u5e0c\u671b\u90e8\u7f72 submariner-broker \u7684\u96c6\u7fa4\u8fdb\u884c\u90e8\u7f72\uff1a subctl deploy-broker \u5728\u672c\u6587\u6863\u4e2d cluster0 \u7684\u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e3a 10.16.0.0/16 \uff0cjoin \u5b50\u7f51 CIDR \u4e3a 100.64.0.0/16 \uff0c cluster1 \u7684\u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e3a 11.16.0.0/16 \uff0cjoin \u5b50\u7f51 CIDR \u4e3a '100.68.0.0/16'\u3002 \u5207\u6362 kubeconfig \u81f3 cluster0 \u6ce8\u518c\u96c6\u7fa4\u81f3 broker\uff0c\u5e76\u6ce8\u518c\u7f51\u5173\u8282\u70b9: subctl join broker-info.subm --clusterid cluster0 --clustercidr 100 .64.0.0/16,10.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster0 submariner.io/gateway = true \u5207\u6362 kubeconfig \u81f3 cluster1 \u6ce8\u518c\u96c6\u7fa4\u81f3 broker\uff0c\u5e76\u6ce8\u518c\u7f51\u5173\u8282\u70b9: subctl join broker-info.subm --clusterid cluster1 --clustercidr 100 .68.0.0/16,11.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster1 submariner.io/gateway = true \u5982\u679c\u6267\u884c join \u547d\u4ee4\u4e4b\u540e\u6ca1\u6709\u65b0\u7684 gateway, routeagent pod \u51fa\u73b0\u7684\u8bdd, \u8bf7\u4e3a submariner-operator \u8fd9\u4e2a clusterrole \u589e\u52a0\u4ee5\u4e0b\u6743\u9650: - apiGroups : - \"apps\" resources : - daemonsets verbs : - create - get - list - watch - update \u5bf9\u4e8e\u591a\u8282\u70b9\u7684\u96c6\u7fa4\uff0c\u9700\u8981\u5c06\u9ed8\u8ba4\u7684 subnet ovn-default \u7684\u7f51\u5173\u914d\u7f6e\u6539\u4e3a centralized \u3002\u4e3a submariner \u914d\u7f6e\u7684 gateway \u8282\u70b9\u9700\u8981\u548c subnet \u8282\u70b9\u5b8c\u5168\u76f8\u540c\u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728\u4e24\u4e2a\u96c6\u7fa4\u5185\u5206\u522b\u542f\u52a8 Pod \u5e76\u5c1d\u8bd5\u4f7f\u7528 IP \u8fdb\u884c\u76f8\u4e92\u8bbf\u95ee\u3002 \u5982\u679c\u51fa\u73b0\u7f51\u7edc\u4e92\u901a\u95ee\u9898\u53ef\u901a\u8fc7 subctl \u547d\u4ee4\u8fdb\u884c\u8bca\u65ad\uff1a subctl show all subctl diagnose all \u66f4\u591a Submariner \u76f8\u5173\u64cd\u4f5c\u8bf7\u67e5\u770b Submariner \u7528\u6237\u624b\u518c \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 Submariner \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-submariner/#submariner","text":"Submariner \u4f5c\u4e3a\u53ef\u4ee5\u6253\u901a\u591a\u4e2a Kubernetes \u96c6\u7fa4 Pod \u548c Service \u7f51\u7edc\u7684\u5f00\u6e90\u7f51\u7edc\u7ec4\u4ef6\uff0c\u80fd\u591f\u5e2e\u52a9 Kube-OVN \u5b9e\u73b0\u591a\u96c6\u7fa4\u4e92\u8054\u3002 \u76f8\u6bd4\u901a\u8fc7 OVN-IC \u6253\u901a\u591a\u96c6\u7fa4\u7f51\u7edc\u7684\u65b9\u5f0f\uff0cSubmariner \u53ef\u4ee5\u6253\u901a Kube-OVN \u548c\u975e Kube-OVN \u7684\u96c6\u7fa4\u7f51\u7edc\uff0c\u5e76 \u80fd\u63d0\u4f9b Service \u7684\u8de8\u96c6\u7fa4\u80fd\u529b\u3002\u4f46\u662f Submariner \u76ee\u524d\u53ea\u80fd\u5b9e\u73b0\u9ed8\u8ba4\u5b50\u7f51\u7684\u6253\u901a\uff0c\u65e0\u6cd5\u5b9e\u73b0\u591a\u5b50\u7f51\u9009\u62e9\u6027\u6253\u901a\u3002","title":"\u4f7f\u7528 Submariner \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-submariner/#_1","text":"\u4e24\u4e2a\u96c6\u7fa4\u7684 Service CIDR \u548c\u9ed8\u8ba4\u5b50\u7f51\u7684 CIDR \u4e0d\u80fd\u91cd\u53e0\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/with-submariner/#submariner_1","text":"\u4e0b\u8f7d subctl \u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u5e76\u90e8\u7f72\u5230\u76f8\u5e94\u8def\u5f84\uff1a curl -Ls https://get.submariner.io | bash export PATH = $PATH :~/.local/bin echo export PATH = \\$ PATH:~/.local/bin >> ~/.profile \u5207\u6362 kubeconfig \u81f3\u5e0c\u671b\u90e8\u7f72 submariner-broker \u7684\u96c6\u7fa4\u8fdb\u884c\u90e8\u7f72\uff1a subctl deploy-broker \u5728\u672c\u6587\u6863\u4e2d cluster0 \u7684\u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e3a 10.16.0.0/16 \uff0cjoin \u5b50\u7f51 CIDR \u4e3a 100.64.0.0/16 \uff0c cluster1 \u7684\u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e3a 11.16.0.0/16 \uff0cjoin \u5b50\u7f51 CIDR \u4e3a '100.68.0.0/16'\u3002 \u5207\u6362 kubeconfig \u81f3 cluster0 \u6ce8\u518c\u96c6\u7fa4\u81f3 broker\uff0c\u5e76\u6ce8\u518c\u7f51\u5173\u8282\u70b9: subctl join broker-info.subm --clusterid cluster0 --clustercidr 100 .64.0.0/16,10.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster0 submariner.io/gateway = true \u5207\u6362 kubeconfig \u81f3 cluster1 \u6ce8\u518c\u96c6\u7fa4\u81f3 broker\uff0c\u5e76\u6ce8\u518c\u7f51\u5173\u8282\u70b9: subctl join broker-info.subm --clusterid cluster1 --clustercidr 100 .68.0.0/16,11.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster1 submariner.io/gateway = true \u5982\u679c\u6267\u884c join \u547d\u4ee4\u4e4b\u540e\u6ca1\u6709\u65b0\u7684 gateway, routeagent pod \u51fa\u73b0\u7684\u8bdd, \u8bf7\u4e3a submariner-operator \u8fd9\u4e2a clusterrole \u589e\u52a0\u4ee5\u4e0b\u6743\u9650: - apiGroups : - \"apps\" resources : - daemonsets verbs : - create - get - list - watch - update \u5bf9\u4e8e\u591a\u8282\u70b9\u7684\u96c6\u7fa4\uff0c\u9700\u8981\u5c06\u9ed8\u8ba4\u7684 subnet ovn-default \u7684\u7f51\u5173\u914d\u7f6e\u6539\u4e3a centralized \u3002\u4e3a submariner \u914d\u7f6e\u7684 gateway \u8282\u70b9\u9700\u8981\u548c subnet \u8282\u70b9\u5b8c\u5168\u76f8\u540c\u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728\u4e24\u4e2a\u96c6\u7fa4\u5185\u5206\u522b\u542f\u52a8 Pod \u5e76\u5c1d\u8bd5\u4f7f\u7528 IP \u8fdb\u884c\u76f8\u4e92\u8bbf\u95ee\u3002 \u5982\u679c\u51fa\u73b0\u7f51\u7edc\u4e92\u901a\u95ee\u9898\u53ef\u901a\u8fc7 subctl \u547d\u4ee4\u8fdb\u884c\u8bca\u65ad\uff1a subctl show all subctl diagnose all \u66f4\u591a Submariner \u76f8\u5173\u64cd\u4f5c\u8bf7\u67e5\u770b Submariner \u7528\u6237\u624b\u518c \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u90e8\u7f72 Submariner"},{"location":"guide/custom-routes/","text":"\u81ea\u5b9a\u4e49\u8def\u7531 \u00b6 \u53ef\u4ee5\u5728\u521b\u5efa Pod \u65f6\u901a\u8fc7 Annotations \u6765\u6307\u5b9a\u9700\u8981\u914d\u7f6e\u7684\u8def\u7531\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : v1 kind : Pod metadata : name : custom-routes annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine dst \u5b57\u6bb5\u4e3a\u7a7a\u8868\u793a\u4fee\u6539\u9ed8\u8ba4\u8def\u7531\u3002 \u5982\u679c\u5de5\u4f5c\u8d1f\u8f7d\u4e3a Deployment\u3001DaemonSet \u6216 StatefulSet\uff0c\u5bf9\u5e94\u7684 Annotation \u9700\u8981\u914d\u7f6e\u5728\u8d44\u6e90\u7684 .spec.template.metadata.annotations \u4e2d\uff0c\u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : apps/v1 kind : Deployment metadata : name : custom-routes labels : app : nginx spec : replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u81ea\u5b9a\u4e49\u8def\u7531"},{"location":"guide/custom-routes/#_1","text":"\u53ef\u4ee5\u5728\u521b\u5efa Pod \u65f6\u901a\u8fc7 Annotations \u6765\u6307\u5b9a\u9700\u8981\u914d\u7f6e\u7684\u8def\u7531\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : v1 kind : Pod metadata : name : custom-routes annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine dst \u5b57\u6bb5\u4e3a\u7a7a\u8868\u793a\u4fee\u6539\u9ed8\u8ba4\u8def\u7531\u3002 \u5982\u679c\u5de5\u4f5c\u8d1f\u8f7d\u4e3a Deployment\u3001DaemonSet \u6216 StatefulSet\uff0c\u5bf9\u5e94\u7684 Annotation \u9700\u8981\u914d\u7f6e\u5728\u8d44\u6e90\u7684 .spec.template.metadata.annotations \u4e2d\uff0c\u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : apps/v1 kind : Deployment metadata : name : custom-routes labels : app : nginx spec : replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u81ea\u5b9a\u4e49\u8def\u7531"},{"location":"guide/dual-stack/","text":"\u53cc\u6808\u4f7f\u7528 \u00b6 Kube-OVN \u4e2d\u4e0d\u540c\u7684\u5b50\u7f51\u53ef\u4ee5\u652f\u6301\u4e0d\u540c\u7684\u534f\u8bae\uff0c\u4e00\u4e2a\u96c6\u7fa4\u5185\u53ef\u4ee5\u540c\u65f6\u5b58\u5728 IPv4\uff0cIPv6 \u548c\u53cc\u6808\u7c7b\u578b\u7684\u5b50\u7f51\u3002 \u6211\u4eec\u63a8\u8350\u4e00\u4e2a\u96c6\u7fa4\u5185\u4f7f\u7528\u7edf\u4e00\u7684\u534f\u8bae\u7c7b\u578b\u4ee5\u7b80\u5316\u4f7f\u7528\u548c\u7ef4\u62a4\u3002 \u4e3a\u4e86\u652f\u6301\u53cc\u6808\uff0c\u9700\u8981\u4e3b\u673a\u7f51\u7edc\u6ee1\u8db3\u53cc\u6808\u8981\u6c42\uff0c\u540c\u65f6\u9700\u8981\u5bf9 Kubernetes \u76f8\u5173\u53c2\u6570\u505a\u8c03\u6574\uff0c \u8bf7\u53c2\u8003 Kubernetes \u7684 \u53cc\u6808\u5b98\u65b9\u6307\u5bfc \u3002 \u521b\u5efa\u53cc\u6808\u5b50\u7f51 \u00b6 \u5728\u914d\u7f6e\u53cc\u6808\u65f6\uff0c\u53ea\u9700\u8981\u8bbe\u7f6e\u5bf9\u5e94\u5b50\u7f51 CIDR \u683c\u5f0f\u4e3a cidr=<IPv4 CIDR>,<IPv6 CIDR> \u5373\u53ef\u3002 CIDR \u987a\u5e8f\u8981\u6c42 IPv4 \u5728\u524d\uff0cIPv6 \u5728\u540e\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-test spec : cidrBlock : 10.16.0.0/16,fd00:10:16::/64 excludeIps : - 10.16.0.1 - fd00:10:16::1 gateway : 10.16.0.1,fd00:10:16::1 \u5982\u679c\u9700\u8981\u5728\u5b89\u88c5\u65f6\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u53cc\u6808\uff0c\u9700\u8981\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u4fee\u6539\u5982\u4e0b\u53c2\u6570\uff1a POD_CIDR = \"10.16.0.0/16,fd00:10:16::/64\" JOIN_CIDR = \"100.64.0.0/16,fd00:100:64::/64\" \u67e5\u770b Pod \u5730\u5740 \u00b6 \u914d\u7f6e\u53cc\u6808\u7f51\u7edc\u7684 Pod \u5c06\u4f1a\u4ece\u8be5\u5b50\u7f51\u540c\u65f6\u5206\u914d IPv4 \u548c IPv6 \u7684\u5730\u5740\uff0c\u5206\u914d\u7ed3\u679c\u4f1a\u663e\u793a\u5728 Pod \u7684 annotation \u4e2d: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/allocated : \"true\" ovn.kubernetes.io/cidr : 10.16.0.0/16,fd00:10:16::/64 ovn.kubernetes.io/gateway : 10.16.0.1,fd00:10:16::1 ovn.kubernetes.io/ip_address : 10.16.0.9,fd00:10:16::9 ovn.kubernetes.io/logical_switch : ovn-default ovn.kubernetes.io/mac_address : 00:00:00:14:88:09 ovn.kubernetes.io/network_types : geneve ovn.kubernetes.io/routed : \"true\" ... podIP : 10.16.0.9 podIPs : - ip : 10.16.0.9 - ip : fd00:10:16::9 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u53cc\u6808\u4f7f\u7528"},{"location":"guide/dual-stack/#_1","text":"Kube-OVN \u4e2d\u4e0d\u540c\u7684\u5b50\u7f51\u53ef\u4ee5\u652f\u6301\u4e0d\u540c\u7684\u534f\u8bae\uff0c\u4e00\u4e2a\u96c6\u7fa4\u5185\u53ef\u4ee5\u540c\u65f6\u5b58\u5728 IPv4\uff0cIPv6 \u548c\u53cc\u6808\u7c7b\u578b\u7684\u5b50\u7f51\u3002 \u6211\u4eec\u63a8\u8350\u4e00\u4e2a\u96c6\u7fa4\u5185\u4f7f\u7528\u7edf\u4e00\u7684\u534f\u8bae\u7c7b\u578b\u4ee5\u7b80\u5316\u4f7f\u7528\u548c\u7ef4\u62a4\u3002 \u4e3a\u4e86\u652f\u6301\u53cc\u6808\uff0c\u9700\u8981\u4e3b\u673a\u7f51\u7edc\u6ee1\u8db3\u53cc\u6808\u8981\u6c42\uff0c\u540c\u65f6\u9700\u8981\u5bf9 Kubernetes \u76f8\u5173\u53c2\u6570\u505a\u8c03\u6574\uff0c \u8bf7\u53c2\u8003 Kubernetes \u7684 \u53cc\u6808\u5b98\u65b9\u6307\u5bfc \u3002","title":"\u53cc\u6808\u4f7f\u7528"},{"location":"guide/dual-stack/#_2","text":"\u5728\u914d\u7f6e\u53cc\u6808\u65f6\uff0c\u53ea\u9700\u8981\u8bbe\u7f6e\u5bf9\u5e94\u5b50\u7f51 CIDR \u683c\u5f0f\u4e3a cidr=<IPv4 CIDR>,<IPv6 CIDR> \u5373\u53ef\u3002 CIDR \u987a\u5e8f\u8981\u6c42 IPv4 \u5728\u524d\uff0cIPv6 \u5728\u540e\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-test spec : cidrBlock : 10.16.0.0/16,fd00:10:16::/64 excludeIps : - 10.16.0.1 - fd00:10:16::1 gateway : 10.16.0.1,fd00:10:16::1 \u5982\u679c\u9700\u8981\u5728\u5b89\u88c5\u65f6\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u53cc\u6808\uff0c\u9700\u8981\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u4fee\u6539\u5982\u4e0b\u53c2\u6570\uff1a POD_CIDR = \"10.16.0.0/16,fd00:10:16::/64\" JOIN_CIDR = \"100.64.0.0/16,fd00:100:64::/64\"","title":"\u521b\u5efa\u53cc\u6808\u5b50\u7f51"},{"location":"guide/dual-stack/#pod","text":"\u914d\u7f6e\u53cc\u6808\u7f51\u7edc\u7684 Pod \u5c06\u4f1a\u4ece\u8be5\u5b50\u7f51\u540c\u65f6\u5206\u914d IPv4 \u548c IPv6 \u7684\u5730\u5740\uff0c\u5206\u914d\u7ed3\u679c\u4f1a\u663e\u793a\u5728 Pod \u7684 annotation \u4e2d: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/allocated : \"true\" ovn.kubernetes.io/cidr : 10.16.0.0/16,fd00:10:16::/64 ovn.kubernetes.io/gateway : 10.16.0.1,fd00:10:16::1 ovn.kubernetes.io/ip_address : 10.16.0.9,fd00:10:16::9 ovn.kubernetes.io/logical_switch : ovn-default ovn.kubernetes.io/mac_address : 00:00:00:14:88:09 ovn.kubernetes.io/network_types : geneve ovn.kubernetes.io/routed : \"true\" ... podIP : 10.16.0.9 podIPs : - ip : 10.16.0.9 - ip : fd00:10:16::9 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u67e5\u770b Pod \u5730\u5740"},{"location":"guide/eip-snat/","text":"EIP \u548c SNAT \u914d\u7f6e \u00b6 \u8be5\u914d\u7f6e\u9488\u5bf9\u9ed8\u8ba4 VPC \u4e0b\u7684\u7f51\u7edc\uff0c\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u8bf7\u53c2\u8003 VPC \u7f51\u5173 Kube-OVN \u652f\u6301\u5229\u7528 OVN \u4e2d\u7684 L3 Gateway \u529f\u80fd\u6765\u5b9e\u73b0 Pod \u7ea7\u522b\u7684 SNAT \u548c EIP \u529f\u80fd\u3002 \u901a\u8fc7\u4f7f\u7528 SNAT\uff0c\u4e00\u7ec4 Pod \u53ef\u4ee5\u5171\u4eab\u4e00\u4e2a IP \u5730\u5740\u5bf9\u5916\u8fdb\u884c\u8bbf\u95ee\u3002 \u901a\u8fc7 EIP \u7684\u529f\u80fd\uff0c\u4e00\u4e2a Pod \u53ef\u4ee5\u76f4\u63a5\u548c\u4e00\u4e2a\u5916\u90e8 IP \u5173\u8054\uff0c \u5916\u90e8\u670d\u52a1\u53ef\u4ee5\u901a\u8fc7 EIP \u76f4\u63a5\u8bbf\u95ee Pod\uff0cPod \u4e5f\u5c06\u901a\u8fc7\u8fd9\u4e2a EIP \u8bbf\u95ee\u5916\u90e8\u670d\u52a1\u3002 \u51c6\u5907\u5de5\u4f5c \u00b6 \u4e3a\u4e86\u4f7f\u7528 OVN \u7684 L3 Gateway \u80fd\u529b\uff0c\u5fc5\u987b\u5c06\u4e00\u4e2a\u5355\u72ec\u7684\u7f51\u5361\u63a5\u5165 OVS \u7f51\u6865\u4e2d\u8fdb\u884c Overlay \u548c Underlay \u7f51\u7edc\u7684\u6253\u901a\uff0c \u4e3b\u673a\u5fc5\u987b\u6709\u5176\u4ed6\u7684\u7f51\u5361\u7528\u4e8e\u8fd0\u7ef4\u7ba1\u7406\u3002 \u7531\u4e8e\u7ecf\u8fc7 NAT \u540e\u7684\u6570\u636e\u5305\u4f1a\u76f4\u63a5\u8fdb\u5165 Underlay \u7f51\u7edc\uff0c\u5fc5\u987b\u786e\u8ba4\u5f53\u524d\u7684\u7f51\u7edc\u67b6\u6784\u4e0b\u6b64\u7c7b\u6570\u636e\u5305\u53ef\u4ee5\u5b89\u5168\u901a\u8fc7\u3002 \u76ee\u524d EIP \u548c SNAT \u5730\u5740\u6ca1\u6709\u51b2\u7a81\u68c0\u6d4b\uff0c\u9700\u8981\u7ba1\u7406\u5458\u624b\u52a8\u5206\u914d\u907f\u514d\u5730\u5740\u51b2\u7a81\u3002 \u521b\u5efa\u914d\u7f6e\u6587\u4ef6 \u00b6 \u5728 kube-system \u4e0b\u521b\u5efa ConfigMap ovn-external-gw-config \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-external-gw-config namespace : kube-system data : enable-external-gw : \"true\" external-gw-nodes : \"kube-ovn-worker\" external-gw-nic : \"eth1\" external-gw-addr : \"172.56.0.1/16\" nic-ip : \"172.56.0.254/16\" nic-mac : \"16:52:f3:13:6a:25\" enable-external-gw : \u662f\u5426\u5f00\u542f SNAT \u548c EIP \u529f\u80fd\u3002 type : centrailized \u6216 distributed \uff0c \u9ed8\u8ba4\u4e3a centralized \u5982\u679c\u4f7f\u7528 distributed \uff0c\u5219\u96c6\u7fa4\u6240\u6709\u8282\u70b9\u90fd\u9700\u8981\u6709\u540c\u540d\u7f51\u5361\u6765\u627f\u62c5\u7f51\u5173\u529f\u80fd\u3002 external-gw-nodes : centralized \u6a21\u5f0f\u4e0b\uff0c\u627f\u62c5\u7f51\u5173\u4f5c\u7528\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 external-gw-nic : \u8282\u70b9\u4e0a\u627f\u62c5\u7f51\u5173\u4f5c\u7528\u7684\u7f51\u5361\u540d\u3002 external-gw-addr : \u7269\u7406\u7f51\u7edc\u7f51\u5173\u7684 IP \u548c\u63a9\u7801\u3002 nic-ip , nic-mac : \u5206\u914d\u7ed9\u903b\u8f91\u7f51\u5173\u7aef\u53e3\u7684 IP \u548c Mac\uff0c\u9700\u4e3a\u7269\u7406\u6bb5\u672a\u88ab\u5360\u7528\u7684 IP \u548c Mac\u3002 \u89c2\u5bdf OVN \u548c OVS \u72b6\u6001\u786e\u8ba4\u914d\u7f6e\u751f\u6548 \u00b6 \u68c0\u67e5 OVN-NB \u72b6\u6001, \u786e\u8ba4 ovn-external \u903b\u8f91\u4ea4\u6362\u673a\u5b58\u5728\uff0c\u5e76\u4e14 ovn-cluster-ovn-external \u903b\u8f91\u8def\u7531\u5668\u7aef\u53e3\u4e0a \u7ed1\u5b9a\u4e86\u6b63\u786e\u7684\u5730\u5740\u548c chassis\u3002 # kubectl ko nbctl show switch 3de4cea7-1a71-43f3-8b62-435a57ef16a6 ( ovn-external ) port ln-ovn-external type: localnet addresses: [ \"unknown\" ] port ovn-external-ovn-cluster type: router router-port: ovn-cluster-ovn-external router e1eb83ad-34be-4ed5-9a02-fcc8b1d357c4 ( ovn-cluster ) port ovn-cluster-ovn-external mac: \"ac:1f:6b:2d:33:f1\" networks: [ \"172.56.0.100/16\" ] gateway chassis: [ a5682814-2e2c-46dd-9c1c-6803ef0dab66 ] \u68c0\u67e5 OVS \u72b6\u6001\uff0c\u786e\u8ba4\u76f8\u5e94\u7684\u7f51\u5361\u5df2\u7ecf\u6865\u63a5\u8fdb br-external \u7f51\u6865\uff1a # kubectl ko vsctl ${gateway node name} show e7d81150-7743-4d6e-9e6f-5c688232e130 Bridge br-external Port br-external Interface br-external type: internal Port eno2 Interface eno2 Port patch-ln-ovn-external-to-br-int Interface patch-ln-ovn-external-to-br-int type: patch options: { peer = patch-br-int-to-ln-ovn-external } Pod \u914d\u7f6e EIP \u548c SNAT \u00b6 \u53ef\u901a\u8fc7\u5728 Pod \u4e0a\u589e\u52a0 ovn.kubernetes.io/snat \u6216 ovn.kubernetes.io/eip annotation \u6765\u5206\u522b\u914d\u7f6e SNAT \u548c EIP\uff1a apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/snat : 172.56.0.200 spec : containers : - name : snat-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/eip : 172.56.0.233 spec : containers : - name : eip-pod image : docker.io/library/nginx:alpine \u53ef\u901a\u8fc7 kubectl \u6216\u5176\u4ed6\u5de5\u5177\u52a8\u6001\u8c03\u6574 Pod \u6240\u914d\u7f6e\u7684 EIP \u6216 SNAT \u89c4\u5219\uff0c\u66f4\u6539\u65f6\u8bf7\u6ce8\u610f\u8981\u540c\u65f6\u5220\u9664 ovn.kubernetes.io/routed annotation \u89e6\u53d1\u8def\u7531\u7684\u53d8\u66f4\uff1a kubectl annotate pod pod-gw ovn.kubernetes.io/eip = 172 .56.0.221 --overwrite kubectl annotate pod pod-gw ovn.kubernetes.io/routed- \u5f53 EIP \u6216 SNAT \u89c4\u5219\u751f\u6548\u540e\uff0c ovn.kubernetes.io/routed annotation \u4f1a\u88ab\u91cd\u65b0\u6dfb\u52a0\u3002 \u9ad8\u7ea7\u914d\u7f6e \u00b6 kube-ovn-controller \u7684\u90e8\u5206\u542f\u52a8\u53c2\u6570\u53ef\u5bf9 SNAT \u548c EIP \u529f\u80fd\u8fdb\u884c\u9ad8\u9636\u914d\u7f6e\uff1a --external-gateway-config-ns : Configmap ovn-external-gw-config \u6240\u5c5e Namespace\uff0c \u9ed8\u8ba4\u4e3a kube-system \u3002 --external-gateway-net : \u7269\u7406\u7f51\u5361\u6240\u6865\u63a5\u7684\u7f51\u6865\u540d\uff0c\u9ed8\u8ba4\u4e3a external \u3002 --external-gateway-vlanid : \u7269\u7406\u7f51\u7edc Vlan Tag \u53f7\uff0c\u9ed8\u8ba4\u4e3a 0\uff0c \u5373\u4e0d\u4f7f\u7528 Vlan\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"EIP \u548c SNAT \u914d\u7f6e"},{"location":"guide/eip-snat/#eip-snat","text":"\u8be5\u914d\u7f6e\u9488\u5bf9\u9ed8\u8ba4 VPC \u4e0b\u7684\u7f51\u7edc\uff0c\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u8bf7\u53c2\u8003 VPC \u7f51\u5173 Kube-OVN \u652f\u6301\u5229\u7528 OVN \u4e2d\u7684 L3 Gateway \u529f\u80fd\u6765\u5b9e\u73b0 Pod \u7ea7\u522b\u7684 SNAT \u548c EIP \u529f\u80fd\u3002 \u901a\u8fc7\u4f7f\u7528 SNAT\uff0c\u4e00\u7ec4 Pod \u53ef\u4ee5\u5171\u4eab\u4e00\u4e2a IP \u5730\u5740\u5bf9\u5916\u8fdb\u884c\u8bbf\u95ee\u3002 \u901a\u8fc7 EIP \u7684\u529f\u80fd\uff0c\u4e00\u4e2a Pod \u53ef\u4ee5\u76f4\u63a5\u548c\u4e00\u4e2a\u5916\u90e8 IP \u5173\u8054\uff0c \u5916\u90e8\u670d\u52a1\u53ef\u4ee5\u901a\u8fc7 EIP \u76f4\u63a5\u8bbf\u95ee Pod\uff0cPod \u4e5f\u5c06\u901a\u8fc7\u8fd9\u4e2a EIP \u8bbf\u95ee\u5916\u90e8\u670d\u52a1\u3002","title":"EIP \u548c SNAT \u914d\u7f6e"},{"location":"guide/eip-snat/#_1","text":"\u4e3a\u4e86\u4f7f\u7528 OVN \u7684 L3 Gateway \u80fd\u529b\uff0c\u5fc5\u987b\u5c06\u4e00\u4e2a\u5355\u72ec\u7684\u7f51\u5361\u63a5\u5165 OVS \u7f51\u6865\u4e2d\u8fdb\u884c Overlay \u548c Underlay \u7f51\u7edc\u7684\u6253\u901a\uff0c \u4e3b\u673a\u5fc5\u987b\u6709\u5176\u4ed6\u7684\u7f51\u5361\u7528\u4e8e\u8fd0\u7ef4\u7ba1\u7406\u3002 \u7531\u4e8e\u7ecf\u8fc7 NAT \u540e\u7684\u6570\u636e\u5305\u4f1a\u76f4\u63a5\u8fdb\u5165 Underlay \u7f51\u7edc\uff0c\u5fc5\u987b\u786e\u8ba4\u5f53\u524d\u7684\u7f51\u7edc\u67b6\u6784\u4e0b\u6b64\u7c7b\u6570\u636e\u5305\u53ef\u4ee5\u5b89\u5168\u901a\u8fc7\u3002 \u76ee\u524d EIP \u548c SNAT \u5730\u5740\u6ca1\u6709\u51b2\u7a81\u68c0\u6d4b\uff0c\u9700\u8981\u7ba1\u7406\u5458\u624b\u52a8\u5206\u914d\u907f\u514d\u5730\u5740\u51b2\u7a81\u3002","title":"\u51c6\u5907\u5de5\u4f5c"},{"location":"guide/eip-snat/#_2","text":"\u5728 kube-system \u4e0b\u521b\u5efa ConfigMap ovn-external-gw-config \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-external-gw-config namespace : kube-system data : enable-external-gw : \"true\" external-gw-nodes : \"kube-ovn-worker\" external-gw-nic : \"eth1\" external-gw-addr : \"172.56.0.1/16\" nic-ip : \"172.56.0.254/16\" nic-mac : \"16:52:f3:13:6a:25\" enable-external-gw : \u662f\u5426\u5f00\u542f SNAT \u548c EIP \u529f\u80fd\u3002 type : centrailized \u6216 distributed \uff0c \u9ed8\u8ba4\u4e3a centralized \u5982\u679c\u4f7f\u7528 distributed \uff0c\u5219\u96c6\u7fa4\u6240\u6709\u8282\u70b9\u90fd\u9700\u8981\u6709\u540c\u540d\u7f51\u5361\u6765\u627f\u62c5\u7f51\u5173\u529f\u80fd\u3002 external-gw-nodes : centralized \u6a21\u5f0f\u4e0b\uff0c\u627f\u62c5\u7f51\u5173\u4f5c\u7528\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 external-gw-nic : \u8282\u70b9\u4e0a\u627f\u62c5\u7f51\u5173\u4f5c\u7528\u7684\u7f51\u5361\u540d\u3002 external-gw-addr : \u7269\u7406\u7f51\u7edc\u7f51\u5173\u7684 IP \u548c\u63a9\u7801\u3002 nic-ip , nic-mac : \u5206\u914d\u7ed9\u903b\u8f91\u7f51\u5173\u7aef\u53e3\u7684 IP \u548c Mac\uff0c\u9700\u4e3a\u7269\u7406\u6bb5\u672a\u88ab\u5360\u7528\u7684 IP \u548c Mac\u3002","title":"\u521b\u5efa\u914d\u7f6e\u6587\u4ef6"},{"location":"guide/eip-snat/#ovn-ovs","text":"\u68c0\u67e5 OVN-NB \u72b6\u6001, \u786e\u8ba4 ovn-external \u903b\u8f91\u4ea4\u6362\u673a\u5b58\u5728\uff0c\u5e76\u4e14 ovn-cluster-ovn-external \u903b\u8f91\u8def\u7531\u5668\u7aef\u53e3\u4e0a \u7ed1\u5b9a\u4e86\u6b63\u786e\u7684\u5730\u5740\u548c chassis\u3002 # kubectl ko nbctl show switch 3de4cea7-1a71-43f3-8b62-435a57ef16a6 ( ovn-external ) port ln-ovn-external type: localnet addresses: [ \"unknown\" ] port ovn-external-ovn-cluster type: router router-port: ovn-cluster-ovn-external router e1eb83ad-34be-4ed5-9a02-fcc8b1d357c4 ( ovn-cluster ) port ovn-cluster-ovn-external mac: \"ac:1f:6b:2d:33:f1\" networks: [ \"172.56.0.100/16\" ] gateway chassis: [ a5682814-2e2c-46dd-9c1c-6803ef0dab66 ] \u68c0\u67e5 OVS \u72b6\u6001\uff0c\u786e\u8ba4\u76f8\u5e94\u7684\u7f51\u5361\u5df2\u7ecf\u6865\u63a5\u8fdb br-external \u7f51\u6865\uff1a # kubectl ko vsctl ${gateway node name} show e7d81150-7743-4d6e-9e6f-5c688232e130 Bridge br-external Port br-external Interface br-external type: internal Port eno2 Interface eno2 Port patch-ln-ovn-external-to-br-int Interface patch-ln-ovn-external-to-br-int type: patch options: { peer = patch-br-int-to-ln-ovn-external }","title":"\u89c2\u5bdf OVN \u548c OVS \u72b6\u6001\u786e\u8ba4\u914d\u7f6e\u751f\u6548"},{"location":"guide/eip-snat/#pod-eip-snat","text":"\u53ef\u901a\u8fc7\u5728 Pod \u4e0a\u589e\u52a0 ovn.kubernetes.io/snat \u6216 ovn.kubernetes.io/eip annotation \u6765\u5206\u522b\u914d\u7f6e SNAT \u548c EIP\uff1a apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/snat : 172.56.0.200 spec : containers : - name : snat-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/eip : 172.56.0.233 spec : containers : - name : eip-pod image : docker.io/library/nginx:alpine \u53ef\u901a\u8fc7 kubectl \u6216\u5176\u4ed6\u5de5\u5177\u52a8\u6001\u8c03\u6574 Pod \u6240\u914d\u7f6e\u7684 EIP \u6216 SNAT \u89c4\u5219\uff0c\u66f4\u6539\u65f6\u8bf7\u6ce8\u610f\u8981\u540c\u65f6\u5220\u9664 ovn.kubernetes.io/routed annotation \u89e6\u53d1\u8def\u7531\u7684\u53d8\u66f4\uff1a kubectl annotate pod pod-gw ovn.kubernetes.io/eip = 172 .56.0.221 --overwrite kubectl annotate pod pod-gw ovn.kubernetes.io/routed- \u5f53 EIP \u6216 SNAT \u89c4\u5219\u751f\u6548\u540e\uff0c ovn.kubernetes.io/routed annotation \u4f1a\u88ab\u91cd\u65b0\u6dfb\u52a0\u3002","title":"Pod \u914d\u7f6e EIP \u548c SNAT"},{"location":"guide/eip-snat/#_3","text":"kube-ovn-controller \u7684\u90e8\u5206\u542f\u52a8\u53c2\u6570\u53ef\u5bf9 SNAT \u548c EIP \u529f\u80fd\u8fdb\u884c\u9ad8\u9636\u914d\u7f6e\uff1a --external-gateway-config-ns : Configmap ovn-external-gw-config \u6240\u5c5e Namespace\uff0c \u9ed8\u8ba4\u4e3a kube-system \u3002 --external-gateway-net : \u7269\u7406\u7f51\u5361\u6240\u6865\u63a5\u7684\u7f51\u6865\u540d\uff0c\u9ed8\u8ba4\u4e3a external \u3002 --external-gateway-vlanid : \u7269\u7406\u7f51\u7edc Vlan Tag \u53f7\uff0c\u9ed8\u8ba4\u4e3a 0\uff0c \u5373\u4e0d\u4f7f\u7528 Vlan\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9ad8\u7ea7\u914d\u7f6e"},{"location":"guide/ippool/","text":"IP \u6c60\u4f7f\u7528 \u00b6 IP \u6c60\uff08IPPool\uff09\u662f\u6bd4\u5b50\u7f51\uff08Subnet\uff09\u66f4\u7ec6\u529b\u5ea6\u7684 IPAM \u7ba1\u7406\u5355\u5143\u3002\u4f60\u53ef\u4ee5\u901a\u8fc7 IP \u6c60\u5c06\u5b50\u7f51\u7f51\u6bb5\u7ec6\u5206\u4e3a\u591a\u4e2a\u5355\u5143\uff0c\u6bcf\u4e2a\u5355\u5143\u7ed1\u5b9a\u4e00\u4e2a\u6216\u591a\u4e2a\u547d\u540d\u7a7a\u95f4\uff08Namespace\uff09\u3002 \u4f7f\u7528\u65b9\u6cd5 \u00b6 \u4f7f\u7528\u793a\u4f8b\uff1a apiVersion : kubeovn.io/v1 kind : IPPool metadata : name : pool-1 spec : subnet : ovn-default ips : - \"10.16.0.201\" - \"10.16.0.210/30\" - \"10.16.0.220..10.16.0.230\" namespaces : - ns-1 \u5b57\u6bb5\u8bf4\u660e\uff1a \u540d\u79f0 \u7528\u9014 \u5907\u6ce8 subnet \u6307\u5b9a\u6240\u5c5e\u5b50\u7f51 \u5fc5\u586b ips \u6307\u5b9a\u5305\u542b\u7684 IP \u8303\u56f4 \u652f\u6301 \u3001 \u4ee5\u53ca .. \u4e09\u79cd\u683c\u5f0f\uff0c\u652f\u6301 IPv6\u3002 namespaces \u7ed1\u5b9a\u547d\u540d\u7a7a\u95f4 \u53ef\u9009 \u6ce8\u610f\u4e8b\u9879 \u00b6 \u4e3a\u4fdd\u8bc1\u4e0e Workload \u901a\u7528 IP Pool \u56fa\u5b9a\u5730\u5740 \u7684\u517c\u5bb9\u6027\uff0cIP \u6c60\u7684\u540d\u79f0\u4e0d\u80fd\u662f\u4e00\u4e2a IP \u5730\u5740\uff1b IP \u6c60\u7684 .spec.ips \u53ef\u6307\u5b9a\u8d85\u51fa\u5b50\u7f51\u8303\u56f4\u7684 IP \u5730\u5740\uff0c\u4f46\u5b9e\u9645\u6709\u6548\u7684 IP \u5730\u5740\u662f .spec.ips \u4e0e\u5b50\u7f51 CIDR \u7684\u4ea4\u96c6\uff1b \u540c\u4e00\u4e2a\u5b50\u7f51\u7684\u4e0d\u540c IP \u6c60\uff0c\u4e0d\u80fd\u5305\u542b\u76f8\u540c\u7684\uff08\u6709\u6548\uff09IP \u5730\u5740\uff1b IP \u6c60\u7684 .spec.ips \u53ef\u52a8\u6001\u4fee\u6539\uff1b IP \u6c60\u4f1a\u7ee7\u627f\u5b50\u7f51\u7684\u4fdd\u7559 IP\uff0c\u4ece IP \u6c60\u968f\u673a\u5206\u914d IP \u5730\u5740\u65f6\uff0c\u4f1a\u8df3\u8fc7\u5305\u542b\u5728 IP \u6c60\u4e2d\u7684\u4fdd\u7559 IP\uff1b \u4ece\u5b50\u7f51\u968f\u673a\u5206\u914d IP \u5730\u5740\u65f6\uff0c\u53ea\u4f1a\u4ece\u5b50\u7f51\u6240\u6709 IP \u6c60\u4ee5\u5916\u7684\u8303\u56f4\u5206\u914d\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"IP\u6c60\u4f7f\u7528"},{"location":"guide/ippool/#ip","text":"IP \u6c60\uff08IPPool\uff09\u662f\u6bd4\u5b50\u7f51\uff08Subnet\uff09\u66f4\u7ec6\u529b\u5ea6\u7684 IPAM \u7ba1\u7406\u5355\u5143\u3002\u4f60\u53ef\u4ee5\u901a\u8fc7 IP \u6c60\u5c06\u5b50\u7f51\u7f51\u6bb5\u7ec6\u5206\u4e3a\u591a\u4e2a\u5355\u5143\uff0c\u6bcf\u4e2a\u5355\u5143\u7ed1\u5b9a\u4e00\u4e2a\u6216\u591a\u4e2a\u547d\u540d\u7a7a\u95f4\uff08Namespace\uff09\u3002","title":"IP \u6c60\u4f7f\u7528"},{"location":"guide/ippool/#_1","text":"\u4f7f\u7528\u793a\u4f8b\uff1a apiVersion : kubeovn.io/v1 kind : IPPool metadata : name : pool-1 spec : subnet : ovn-default ips : - \"10.16.0.201\" - \"10.16.0.210/30\" - \"10.16.0.220..10.16.0.230\" namespaces : - ns-1 \u5b57\u6bb5\u8bf4\u660e\uff1a \u540d\u79f0 \u7528\u9014 \u5907\u6ce8 subnet \u6307\u5b9a\u6240\u5c5e\u5b50\u7f51 \u5fc5\u586b ips \u6307\u5b9a\u5305\u542b\u7684 IP \u8303\u56f4 \u652f\u6301 \u3001 \u4ee5\u53ca .. \u4e09\u79cd\u683c\u5f0f\uff0c\u652f\u6301 IPv6\u3002 namespaces \u7ed1\u5b9a\u547d\u540d\u7a7a\u95f4 \u53ef\u9009","title":"\u4f7f\u7528\u65b9\u6cd5"},{"location":"guide/ippool/#_2","text":"\u4e3a\u4fdd\u8bc1\u4e0e Workload \u901a\u7528 IP Pool \u56fa\u5b9a\u5730\u5740 \u7684\u517c\u5bb9\u6027\uff0cIP \u6c60\u7684\u540d\u79f0\u4e0d\u80fd\u662f\u4e00\u4e2a IP \u5730\u5740\uff1b IP \u6c60\u7684 .spec.ips \u53ef\u6307\u5b9a\u8d85\u51fa\u5b50\u7f51\u8303\u56f4\u7684 IP \u5730\u5740\uff0c\u4f46\u5b9e\u9645\u6709\u6548\u7684 IP \u5730\u5740\u662f .spec.ips \u4e0e\u5b50\u7f51 CIDR \u7684\u4ea4\u96c6\uff1b \u540c\u4e00\u4e2a\u5b50\u7f51\u7684\u4e0d\u540c IP \u6c60\uff0c\u4e0d\u80fd\u5305\u542b\u76f8\u540c\u7684\uff08\u6709\u6548\uff09IP \u5730\u5740\uff1b IP \u6c60\u7684 .spec.ips \u53ef\u52a8\u6001\u4fee\u6539\uff1b IP \u6c60\u4f1a\u7ee7\u627f\u5b50\u7f51\u7684\u4fdd\u7559 IP\uff0c\u4ece IP \u6c60\u968f\u673a\u5206\u914d IP \u5730\u5740\u65f6\uff0c\u4f1a\u8df3\u8fc7\u5305\u542b\u5728 IP \u6c60\u4e2d\u7684\u4fdd\u7559 IP\uff1b \u4ece\u5b50\u7f51\u968f\u673a\u5206\u914d IP \u5730\u5740\u65f6\uff0c\u53ea\u4f1a\u4ece\u5b50\u7f51\u6240\u6709 IP \u6c60\u4ee5\u5916\u7684\u8303\u56f4\u5206\u914d\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"guide/loadbalancer-service/","text":"LoadBalancer \u7c7b\u578b Service \u00b6 Kube-OVN \u5df2\u7ecf\u652f\u6301\u4e86 VPC \u548c VPC \u7f51\u5173\u7684\u5b9e\u73b0\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 VPC \u914d\u7f6e \u3002 \u7531\u4e8e VPC \u7f51\u5173\u7684\u4f7f\u7528\u6bd4\u8f83\u590d\u6742\uff0c\u57fa\u4e8e VPC \u7f51\u5173\u7684\u5b9e\u73b0\u505a\u4e86\u7b80\u5316\uff0c\u652f\u6301\u5728\u9ed8\u8ba4 VPC \u4e0b\u521b\u5efa LoadBalancer \u7c7b\u578b\u7684 Service\uff0c\u5b9e\u73b0\u901a\u8fc7 LoadBalancerIP \u6765\u8bbf\u95ee\u9ed8\u8ba4 VPC \u4e0b\u7684 Service\u3002 \u9996\u5148\u786e\u8ba4\u73af\u5883\u4e0a\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff1a \u5b89\u88c5\u4e86 multus-cni \u548c macvlan cni \u3002 LoadBalancer Service \u7684\u652f\u6301\uff0c\u662f\u5bf9 VPC \u7f51\u5173\u4ee3\u7801\u8fdb\u884c\u7b80\u5316\u5b9e\u73b0\u7684\uff0c\u4ecd\u7136\u4f7f\u7528 vpc-nat-gw \u7684\u955c\u50cf\uff0c\u4f9d\u8d56 macvlan \u63d0\u4f9b\u591a\u7f51\u5361\u529f\u80fd\u652f\u6301\u3002 \u76ee\u524d\u53ea\u652f\u6301\u5728 \u9ed8\u8ba4 VPC \u914d\u7f6e\uff0c\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 LoadBalancer \u652f\u6301\u53ef\u4ee5\u53c2\u8003 VPC \u7684\u6587\u6863 VPC \u914d\u7f6e \u3002 \u9ed8\u8ba4 VPC LoadBalancer Service \u914d\u7f6e\u6b65\u9aa4 \u00b6 \u5f00\u542f\u7279\u6027\u5f00\u5173 \u00b6 \u4fee\u6539 kube-system namespace \u4e0b\u7684 deployment kube-ovn-controller \uff0c\u5728 args \u4e2d\u589e\u52a0\u53c2\u6570 --enable-lb-svc=true \uff0c\u5f00\u542f\u529f\u80fd\u5f00\u5173\uff0c\u8be5\u53c2\u6570\u9ed8\u8ba4\u4e3a false\u3002 containers : - args : - /kube-ovn/start-controller.sh - --default-cidr=10.16.0.0/16 - --default-gateway=10.16.0.1 - --default-gateway-check=true - --enable-lb-svc=true // \u53c2\u6570\u8bbe\u7f6e\u4e3a true \u521b\u5efa NetworkAttachmentDefinition CRD \u8d44\u6e90 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa net-attach-def \u8d44\u6e90: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : lb-svc-attachment namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", //\u7269\u7406\u7f51\u5361\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u914d\u7f6e \"mode\": \"bridge\" }' \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u901a\u8fc7\u7269\u7406\u7f51\u5361 eth0 \u6765\u5b9e\u73b0\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u4ed6\u7269\u7406\u7f51\u5361\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 master \u53d6\u503c\uff0c\u6307\u5b9a\u4f7f\u7528\u7684\u7269\u7406\u7f51\u5361\u540d\u79f0\u3002 \u521b\u5efa Subnet \u00b6 \u521b\u5efa\u7684 Subnet\uff0c\u7528\u4e8e\u7ed9 LoadBalancer Service \u5206\u914d LoadBalancerIP\uff0c\u8be5\u5730\u5740\u6b63\u5e38\u60c5\u51b5\u4e0b\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u53ef\u4ee5\u914d\u7f6e Underlay Subnet \u7528\u4e8e\u5730\u5740\u5206\u914d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa\u65b0\u5b50\u7f51\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attach-subnet spec : protocol : IPv4 provider : lb-svc-attachment.kube-system # provider \u683c\u5f0f\u56fa\u5b9a\uff0c\u7531\u4e0a\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210 cidrBlock : 172.18.0.0/16 gateway : 172.18.0.1 excludeIps : - 172.18.0.0..172.18.0.10 Subnet \u4e2d provider \u53c2\u6570\u4ee5 ovn \u6216\u8005\u4ee5 .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u8868\u793a\u8be5\u5b50\u7f51\u662f\u7531 Kube-OVN \u7ba1\u7406\u4f7f\u7528\uff0c\u9700\u8981\u5bf9\u5e94\u521b\u5efa logical switch \u8bb0\u5f55\u3002 provider \u975e ovn \u6216\u8005\u975e .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u5219 Kube-OVN \u53ea\u63d0\u4f9b IPAM \u529f\u80fd\uff0c\u8bb0\u5f55 IP \u5730\u5740\u5206\u914d\u60c5\u51b5\uff0c\u4e0d\u5bf9\u5b50\u7f51\u505a\u4e1a\u52a1\u903b\u8f91\u5904\u7406\u3002 \u521b\u5efa LoadBalancer Service \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa LoadBalancer Service\uff1a apiVersion : v1 kind : Service metadata : annotations : lb-svc-attachment.kube-system.kubernetes.io/logical_switch : attach-subnet #\u53ef\u9009 ovn.kubernetes.io/attachmentprovider : lb-svc-attachment.kube-system #\u5fc5\u987b labels : app : dynamic name : test-service namespace : default spec : loadBalancerIP : 172.18.0.18 #\u53ef\u9009 ports : - name : test protocol : TCP port : 80 targetPort : 80 selector : app : dynamic sessionAffinity : None type : LoadBalancer \u5728 yaml \u4e2d\uff0cannotation ovn.kubernetes.io/attachmentprovider \u4e3a\u5fc5\u586b\u9879\uff0c\u53d6\u503c\u7531\u7b2c\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210\u3002\u8be5 annotation \u7528\u4e8e\u5728\u521b\u5efa Pod \u65f6\uff0c\u67e5\u627e net-attach-def \u8d44\u6e90\u3002 \u53ef\u4ee5\u901a\u8fc7 annotation \u6307\u5b9a\u591a\u7f51\u5361\u5730\u5740\u5206\u914d\u4f7f\u7528\u7684\u5b50\u7f51\u3002annotation key \u683c\u5f0f\u4e3a net-attach-def \u8d44\u6e90\u7684 Name.Namespace.kubernetes.io/logical_switch \u3002\u8be5\u914d\u7f6e\u4e3a \u53ef\u9009 \u9009\u9879\uff0c\u5728\u6ca1\u6709\u6307\u5b9a LoadBalancerIP \u5730\u5740\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u4ece\u8be5\u5b50\u7f51\u52a8\u6001\u5206\u914d\u5730\u5740\uff0c\u586b\u5145\u5230 LoadBalancerIP \u5b57\u6bb5\u3002 \u5982\u679c\u9700\u8981\u9759\u6001\u914d\u7f6e LoadBalancerIP \u5730\u5740\uff0c\u53ef\u4ee5\u914d\u7f6e spec.loadBalancerIP \u5b57\u6bb5\uff0c\u8be5\u5730\u5740\u9700\u8981\u5728\u6307\u5b9a\u5b50\u7f51\u7684\u5730\u5740\u8303\u56f4\u5185\u3002 \u5728\u6267\u884c yaml \u521b\u5efa Service \u540e\uff0c\u5728 Service \u540c Namespace \u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u542f\u52a8\u4fe1\u606f\uff1a # kubectl get pod NAME READY STATUS RESTARTS AGE lb-svc-test-service-6869d98dd8-cjvll 1 /1 Running 0 107m # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE test-service LoadBalancer 10 .109.201.193 172 .18.0.18 80 :30056/TCP 107m \u6307\u5b9a service.spec.loadBalancerIP \u53c2\u6570\u65f6\uff0c\u6700\u7ec8\u5c06\u8be5\u53c2\u6570\u8d4b\u503c\u7ed9 service external-ip \u5b57\u6bb5\u3002\u4e0d\u6307\u5b9a\u7684\u60c5\u51b5\u4e0b\uff0c\u8be5\u53c2\u6570\u4e3a\u968f\u673a\u5206\u914d\u503c\u3002 \u67e5\u770b\u6d4b\u8bd5 Pod \u7684 yaml \u8f93\u51fa\uff0c\u5b58\u5728\u591a\u7f51\u5361\u5206\u914d\u7684\u5730\u5740\u4fe1\u606f\uff1a # kubectl get pod -o yaml lb-svc-test-service-6869d98dd8-cjvll apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/network-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] k8s.v1.cni.cncf.io/networks: default/test-service k8s.v1.cni.cncf.io/networks-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] ovn.kubernetes.io/allocated: \"true\" ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.2 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:45:F4:29 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/routed: \"true\" test-service.default.kubernetes.io/allocated: \"true\" test-service.default.kubernetes.io/cidr: 172 .18.0.0/16 test-service.default.kubernetes.io/gateway: 172 .18.0.1 test-service.default.kubernetes.io/ip_address: 172 .18.0.18 test-service.default.kubernetes.io/logical_switch: attach-subnet test-service.default.kubernetes.io/mac_address: 00 :00:00:AF:AA:BF test-service.default.kubernetes.io/pod_nic_type: veth-pair \u67e5\u770b Service \u7684\u4fe1\u606f\uff1a # kubectl get svc -o yaml test-service apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { \"apiVersion\" : \"v1\" , \"kind\" : \"Service\" , \"metadata\" : { \"annotations\" : { \"test-service.default.kubernetes.io/logical_switch\" : \"attach-subnet\" } , \"labels\" : { \"app\" : \"dynamic\" } , \"name\" : \"test-service\" , \"namespace\" : \"default\" } , \"spec\" : { \"ports\" : [{ \"name\" : \"test\" , \"port\" :80, \"protocol\" : \"TCP\" , \"targetPort\" :80 }] , \"selector\" : { \"app\" : \"dynamic\" } , \"sessionAffinity\" : \"None\" , \"type\" : \"LoadBalancer\" }} ovn.kubernetes.io/vpc: ovn-cluster test-service.default.kubernetes.io/logical_switch: attach-subnet creationTimestamp: \"2022-06-15T09:01:58Z\" labels: app: dynamic name: test-service namespace: default resourceVersion: \"38485\" uid: 161edee1-7f6e-40f5-9e09-5a52c44267d0 spec: allocateLoadBalancerNodePorts: true clusterIP: 10 .109.201.193 clusterIPs: - 10 .109.201.193 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: test nodePort: 30056 port: 80 protocol: TCP targetPort: 80 selector: app: dynamic sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - ip: 172 .18.0.18 \u6d4b\u8bd5 LoadBalancerIP \u8bbf\u95ee \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml, \u521b\u5efa\u6d4b\u8bd5 Pod\uff0c\u4f5c\u4e3a Service \u7684 Endpoints \u63d0\u4f9b\u670d\u52a1: apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx dnsPolicy : ClusterFirst restartPolicy : Always \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u63d0\u4f9b\u7684\u5b50\u7f51\u5730\u5740\uff0c\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u4e3a\u4e86\u7b80\u5355\u9a8c\u8bc1\uff0c\u5728\u96c6\u7fa4\u5185\u8bbf\u95ee Service \u7684 LoadBalancerIP:Port \uff0c\u67e5\u770b\u662f\u5426\u6b63\u5e38\u8bbf\u95ee\u6210\u529f\u3002 # curl 172.18.0.11:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on dynamic-7d8d7874f5-hsgc4</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u8fdb\u5165 Service \u521b\u5efa\u7684 Pod\uff0c\u67e5\u770b\u7f51\u7edc\u7684\u4fe1\u606f # ip a 4 : net1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ba:85:f7:02:9f:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172 .18.0.18/16 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::b885:f7ff:fe02:9f42/64 scope link valid_lft forever preferred_lft forever 36 : eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 00 :00:00:45:f4:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10 .16.0.2/16 brd 10 .16.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe45:f429/64 scope link valid_lft forever preferred_lft forever # ip rule 0 : from all lookup local 32764 : from all iif eth0 lookup 100 32765 : from all iif net1 lookup 100 32766 : from all lookup main 32767 : from all lookup default # ip route show table 100 default via 172 .18.0.1 dev net1 10 .109.201.193 via 10 .16.0.1 dev eth0 172 .18.0.0/16 dev net1 scope link # iptables -t nat -L -n -v Chain PREROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- * * 0 .0.0.0/0 172 .18.0.18 tcp dpt:80 to:10.109.201.193:80 Chain INPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain OUTPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain POSTROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0 .0.0.0/0 10 .109.201.193 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"LoadBalancer \u7c7b\u578b Service"},{"location":"guide/loadbalancer-service/#loadbalancer-service","text":"Kube-OVN \u5df2\u7ecf\u652f\u6301\u4e86 VPC \u548c VPC \u7f51\u5173\u7684\u5b9e\u73b0\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 VPC \u914d\u7f6e \u3002 \u7531\u4e8e VPC \u7f51\u5173\u7684\u4f7f\u7528\u6bd4\u8f83\u590d\u6742\uff0c\u57fa\u4e8e VPC \u7f51\u5173\u7684\u5b9e\u73b0\u505a\u4e86\u7b80\u5316\uff0c\u652f\u6301\u5728\u9ed8\u8ba4 VPC \u4e0b\u521b\u5efa LoadBalancer \u7c7b\u578b\u7684 Service\uff0c\u5b9e\u73b0\u901a\u8fc7 LoadBalancerIP \u6765\u8bbf\u95ee\u9ed8\u8ba4 VPC \u4e0b\u7684 Service\u3002 \u9996\u5148\u786e\u8ba4\u73af\u5883\u4e0a\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff1a \u5b89\u88c5\u4e86 multus-cni \u548c macvlan cni \u3002 LoadBalancer Service \u7684\u652f\u6301\uff0c\u662f\u5bf9 VPC \u7f51\u5173\u4ee3\u7801\u8fdb\u884c\u7b80\u5316\u5b9e\u73b0\u7684\uff0c\u4ecd\u7136\u4f7f\u7528 vpc-nat-gw \u7684\u955c\u50cf\uff0c\u4f9d\u8d56 macvlan \u63d0\u4f9b\u591a\u7f51\u5361\u529f\u80fd\u652f\u6301\u3002 \u76ee\u524d\u53ea\u652f\u6301\u5728 \u9ed8\u8ba4 VPC \u914d\u7f6e\uff0c\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 LoadBalancer \u652f\u6301\u53ef\u4ee5\u53c2\u8003 VPC \u7684\u6587\u6863 VPC \u914d\u7f6e \u3002","title":"LoadBalancer \u7c7b\u578b Service"},{"location":"guide/loadbalancer-service/#vpc-loadbalancer-service","text":"","title":"\u9ed8\u8ba4 VPC LoadBalancer Service \u914d\u7f6e\u6b65\u9aa4"},{"location":"guide/loadbalancer-service/#_1","text":"\u4fee\u6539 kube-system namespace \u4e0b\u7684 deployment kube-ovn-controller \uff0c\u5728 args \u4e2d\u589e\u52a0\u53c2\u6570 --enable-lb-svc=true \uff0c\u5f00\u542f\u529f\u80fd\u5f00\u5173\uff0c\u8be5\u53c2\u6570\u9ed8\u8ba4\u4e3a false\u3002 containers : - args : - /kube-ovn/start-controller.sh - --default-cidr=10.16.0.0/16 - --default-gateway=10.16.0.1 - --default-gateway-check=true - --enable-lb-svc=true // \u53c2\u6570\u8bbe\u7f6e\u4e3a true","title":"\u5f00\u542f\u7279\u6027\u5f00\u5173"},{"location":"guide/loadbalancer-service/#networkattachmentdefinition-crd","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa net-attach-def \u8d44\u6e90: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : lb-svc-attachment namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", //\u7269\u7406\u7f51\u5361\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u914d\u7f6e \"mode\": \"bridge\" }' \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u901a\u8fc7\u7269\u7406\u7f51\u5361 eth0 \u6765\u5b9e\u73b0\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u4ed6\u7269\u7406\u7f51\u5361\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 master \u53d6\u503c\uff0c\u6307\u5b9a\u4f7f\u7528\u7684\u7269\u7406\u7f51\u5361\u540d\u79f0\u3002","title":"\u521b\u5efa NetworkAttachmentDefinition CRD \u8d44\u6e90"},{"location":"guide/loadbalancer-service/#subnet","text":"\u521b\u5efa\u7684 Subnet\uff0c\u7528\u4e8e\u7ed9 LoadBalancer Service \u5206\u914d LoadBalancerIP\uff0c\u8be5\u5730\u5740\u6b63\u5e38\u60c5\u51b5\u4e0b\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u53ef\u4ee5\u914d\u7f6e Underlay Subnet \u7528\u4e8e\u5730\u5740\u5206\u914d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa\u65b0\u5b50\u7f51\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attach-subnet spec : protocol : IPv4 provider : lb-svc-attachment.kube-system # provider \u683c\u5f0f\u56fa\u5b9a\uff0c\u7531\u4e0a\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210 cidrBlock : 172.18.0.0/16 gateway : 172.18.0.1 excludeIps : - 172.18.0.0..172.18.0.10 Subnet \u4e2d provider \u53c2\u6570\u4ee5 ovn \u6216\u8005\u4ee5 .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u8868\u793a\u8be5\u5b50\u7f51\u662f\u7531 Kube-OVN \u7ba1\u7406\u4f7f\u7528\uff0c\u9700\u8981\u5bf9\u5e94\u521b\u5efa logical switch \u8bb0\u5f55\u3002 provider \u975e ovn \u6216\u8005\u975e .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u5219 Kube-OVN \u53ea\u63d0\u4f9b IPAM \u529f\u80fd\uff0c\u8bb0\u5f55 IP \u5730\u5740\u5206\u914d\u60c5\u51b5\uff0c\u4e0d\u5bf9\u5b50\u7f51\u505a\u4e1a\u52a1\u903b\u8f91\u5904\u7406\u3002","title":"\u521b\u5efa Subnet"},{"location":"guide/loadbalancer-service/#loadbalancer-service_1","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa LoadBalancer Service\uff1a apiVersion : v1 kind : Service metadata : annotations : lb-svc-attachment.kube-system.kubernetes.io/logical_switch : attach-subnet #\u53ef\u9009 ovn.kubernetes.io/attachmentprovider : lb-svc-attachment.kube-system #\u5fc5\u987b labels : app : dynamic name : test-service namespace : default spec : loadBalancerIP : 172.18.0.18 #\u53ef\u9009 ports : - name : test protocol : TCP port : 80 targetPort : 80 selector : app : dynamic sessionAffinity : None type : LoadBalancer \u5728 yaml \u4e2d\uff0cannotation ovn.kubernetes.io/attachmentprovider \u4e3a\u5fc5\u586b\u9879\uff0c\u53d6\u503c\u7531\u7b2c\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210\u3002\u8be5 annotation \u7528\u4e8e\u5728\u521b\u5efa Pod \u65f6\uff0c\u67e5\u627e net-attach-def \u8d44\u6e90\u3002 \u53ef\u4ee5\u901a\u8fc7 annotation \u6307\u5b9a\u591a\u7f51\u5361\u5730\u5740\u5206\u914d\u4f7f\u7528\u7684\u5b50\u7f51\u3002annotation key \u683c\u5f0f\u4e3a net-attach-def \u8d44\u6e90\u7684 Name.Namespace.kubernetes.io/logical_switch \u3002\u8be5\u914d\u7f6e\u4e3a \u53ef\u9009 \u9009\u9879\uff0c\u5728\u6ca1\u6709\u6307\u5b9a LoadBalancerIP \u5730\u5740\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u4ece\u8be5\u5b50\u7f51\u52a8\u6001\u5206\u914d\u5730\u5740\uff0c\u586b\u5145\u5230 LoadBalancerIP \u5b57\u6bb5\u3002 \u5982\u679c\u9700\u8981\u9759\u6001\u914d\u7f6e LoadBalancerIP \u5730\u5740\uff0c\u53ef\u4ee5\u914d\u7f6e spec.loadBalancerIP \u5b57\u6bb5\uff0c\u8be5\u5730\u5740\u9700\u8981\u5728\u6307\u5b9a\u5b50\u7f51\u7684\u5730\u5740\u8303\u56f4\u5185\u3002 \u5728\u6267\u884c yaml \u521b\u5efa Service \u540e\uff0c\u5728 Service \u540c Namespace \u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u542f\u52a8\u4fe1\u606f\uff1a # kubectl get pod NAME READY STATUS RESTARTS AGE lb-svc-test-service-6869d98dd8-cjvll 1 /1 Running 0 107m # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE test-service LoadBalancer 10 .109.201.193 172 .18.0.18 80 :30056/TCP 107m \u6307\u5b9a service.spec.loadBalancerIP \u53c2\u6570\u65f6\uff0c\u6700\u7ec8\u5c06\u8be5\u53c2\u6570\u8d4b\u503c\u7ed9 service external-ip \u5b57\u6bb5\u3002\u4e0d\u6307\u5b9a\u7684\u60c5\u51b5\u4e0b\uff0c\u8be5\u53c2\u6570\u4e3a\u968f\u673a\u5206\u914d\u503c\u3002 \u67e5\u770b\u6d4b\u8bd5 Pod \u7684 yaml \u8f93\u51fa\uff0c\u5b58\u5728\u591a\u7f51\u5361\u5206\u914d\u7684\u5730\u5740\u4fe1\u606f\uff1a # kubectl get pod -o yaml lb-svc-test-service-6869d98dd8-cjvll apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/network-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] k8s.v1.cni.cncf.io/networks: default/test-service k8s.v1.cni.cncf.io/networks-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] ovn.kubernetes.io/allocated: \"true\" ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.2 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:45:F4:29 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/routed: \"true\" test-service.default.kubernetes.io/allocated: \"true\" test-service.default.kubernetes.io/cidr: 172 .18.0.0/16 test-service.default.kubernetes.io/gateway: 172 .18.0.1 test-service.default.kubernetes.io/ip_address: 172 .18.0.18 test-service.default.kubernetes.io/logical_switch: attach-subnet test-service.default.kubernetes.io/mac_address: 00 :00:00:AF:AA:BF test-service.default.kubernetes.io/pod_nic_type: veth-pair \u67e5\u770b Service \u7684\u4fe1\u606f\uff1a # kubectl get svc -o yaml test-service apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { \"apiVersion\" : \"v1\" , \"kind\" : \"Service\" , \"metadata\" : { \"annotations\" : { \"test-service.default.kubernetes.io/logical_switch\" : \"attach-subnet\" } , \"labels\" : { \"app\" : \"dynamic\" } , \"name\" : \"test-service\" , \"namespace\" : \"default\" } , \"spec\" : { \"ports\" : [{ \"name\" : \"test\" , \"port\" :80, \"protocol\" : \"TCP\" , \"targetPort\" :80 }] , \"selector\" : { \"app\" : \"dynamic\" } , \"sessionAffinity\" : \"None\" , \"type\" : \"LoadBalancer\" }} ovn.kubernetes.io/vpc: ovn-cluster test-service.default.kubernetes.io/logical_switch: attach-subnet creationTimestamp: \"2022-06-15T09:01:58Z\" labels: app: dynamic name: test-service namespace: default resourceVersion: \"38485\" uid: 161edee1-7f6e-40f5-9e09-5a52c44267d0 spec: allocateLoadBalancerNodePorts: true clusterIP: 10 .109.201.193 clusterIPs: - 10 .109.201.193 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: test nodePort: 30056 port: 80 protocol: TCP targetPort: 80 selector: app: dynamic sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - ip: 172 .18.0.18","title":"\u521b\u5efa LoadBalancer Service"},{"location":"guide/loadbalancer-service/#loadbalancerip","text":"\u53c2\u8003\u4ee5\u4e0b yaml, \u521b\u5efa\u6d4b\u8bd5 Pod\uff0c\u4f5c\u4e3a Service \u7684 Endpoints \u63d0\u4f9b\u670d\u52a1: apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx dnsPolicy : ClusterFirst restartPolicy : Always \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u63d0\u4f9b\u7684\u5b50\u7f51\u5730\u5740\uff0c\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u4e3a\u4e86\u7b80\u5355\u9a8c\u8bc1\uff0c\u5728\u96c6\u7fa4\u5185\u8bbf\u95ee Service \u7684 LoadBalancerIP:Port \uff0c\u67e5\u770b\u662f\u5426\u6b63\u5e38\u8bbf\u95ee\u6210\u529f\u3002 # curl 172.18.0.11:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on dynamic-7d8d7874f5-hsgc4</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u8fdb\u5165 Service \u521b\u5efa\u7684 Pod\uff0c\u67e5\u770b\u7f51\u7edc\u7684\u4fe1\u606f # ip a 4 : net1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ba:85:f7:02:9f:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172 .18.0.18/16 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::b885:f7ff:fe02:9f42/64 scope link valid_lft forever preferred_lft forever 36 : eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 00 :00:00:45:f4:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10 .16.0.2/16 brd 10 .16.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe45:f429/64 scope link valid_lft forever preferred_lft forever # ip rule 0 : from all lookup local 32764 : from all iif eth0 lookup 100 32765 : from all iif net1 lookup 100 32766 : from all lookup main 32767 : from all lookup default # ip route show table 100 default via 172 .18.0.1 dev net1 10 .109.201.193 via 10 .16.0.1 dev eth0 172 .18.0.0/16 dev net1 scope link # iptables -t nat -L -n -v Chain PREROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- * * 0 .0.0.0/0 172 .18.0.18 tcp dpt:80 to:10.109.201.193:80 Chain INPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain OUTPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain POSTROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0 .0.0.0/0 10 .109.201.193 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6d4b\u8bd5 LoadBalancerIP \u8bbf\u95ee"},{"location":"guide/mirror/","text":"\u6d41\u91cf\u955c\u50cf \u00b6 \u6d41\u91cf\u955c\u50cf\u529f\u80fd\u53ef\u4ee5\u5c06\u8fdb\u51fa\u5bb9\u5668\u7f51\u7edc\u7684\u6570\u636e\u5305\u8fdb\u884c\u590d\u5236\u5230\u4e3b\u673a\u7684\u7279\u5b9a\u7f51\u5361\u3002\u7ba1\u7406\u5458\u6216\u5f00\u53d1\u8005 \u53ef\u4ee5\u901a\u8fc7\u76d1\u542c\u8fd9\u5757\u7f51\u5361\u83b7\u5f97\u5b8c\u6574\u7684\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u6765\u8fdb\u4e00\u6b65\u8fdb\u884c\u5206\u6790\uff0c\u76d1\u63a7\uff0c\u5b89\u5168\u5ba1\u8ba1\u7b49\u64cd\u4f5c\u3002 \u4e5f\u53ef\u548c\u4f20\u7edf\u7684 NPM \u5bf9\u63a5\u83b7\u53d6\u66f4\u7ec6\u7c92\u5ea6\u7684\u6d41\u91cf\u76d1\u63a7\u3002 \u6d41\u91cf\u955c\u50cf\u529f\u80fd\u4f1a\u5e26\u6765\u4e00\u5b9a\u7684\u6027\u80fd\u635f\u5931\uff0c\u6839\u636e CPU \u6027\u80fd\u4ee5\u53ca\u6d41\u91cf\u7684\u7279\u5f81\uff0c\u4f1a\u6709 5%~10% \u7684 \u989d\u5916 CPU \u6d88\u8017\u3002 \u5168\u5c40\u6d41\u91cf\u955c\u50cf\u914d\u7f6e \u00b6 \u6d41\u91cf\u955c\u50cf\u529f\u80fd\u9ed8\u8ba4\u4e3a\u5173\u95ed\u72b6\u6001\uff0c\u5982\u679c\u9700\u8981\u5f00\u542f\u8bf7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u542f\u52a8\u53c2\u6570\uff1a --enable-mirror=true \uff1a \u662f\u5426\u5f00\u542f\u6d41\u91cf\u955c\u50cf\u3002 --mirror-iface=mirror0 : \u6d41\u91cf\u955c\u50cf\u6240\u590d\u5236\u5230\u7684\u7f51\u5361\u540d\u3002\u8be5\u7f51\u5361\u53ef\u4e3a\u4e3b\u673a\u4e0a\u5df2\u5b58\u5728\u7684\u4e00\u5757\u7269\u7406\u7f51\u5361\uff0c \u6b64\u65f6\u8be5\u7f51\u5361\u4f1a\u88ab\u6865\u63a5\u8fdb br-int \u7f51\u6865\uff0c\u955c\u50cf\u6d41\u91cf\u4f1a\u76f4\u63a5\u63a5\u5165\u5e95\u5c42\u4ea4\u6362\u673a\u3002\u82e5\u7f51\u5361\u540d\u4e0d\u5b58\u5728\uff0cKube-OVN \u4f1a\u81ea\u52a8 \u521b\u5efa\u4e00\u5757\u540c\u540d\u7684\u865a\u62df\u7f51\u5361\uff0c\u7ba1\u7406\u5458\u6216\u5f00\u53d1\u8005\u53ef\u4ee5\u5728\u5bbf\u4e3b\u673a\u4e0a\u901a\u8fc7\u8be5\u7f51\u5361\u83b7\u53d6\u5f53\u524d\u8282\u70b9\u6240\u6709\u6d41\u91cf\u3002\u9ed8\u8ba4\u4e3a mirror0 \u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u7528 tcpdump \u6216\u5176\u4ed6\u6d41\u91cf\u5206\u6790\u5de5\u5177\u76d1\u542c mirror0 \u4e0a\u7684\u6d41\u91cf\uff1a tcpdump -ni mirror0 Pod \u7ea7\u522b\u6d41\u91cf\u955c\u50cf\u914d\u7f6e \u00b6 \u5982\u679c\u53ea\u9700\u5bf9\u90e8\u5206 Pod \u6d41\u91cf\u8fdb\u884c\u955c\u50cf\uff0c\u5219\u9700\u8981\u5173\u95ed\u5168\u5c40\u7684\u6d41\u91cf\u955c\u50cf\u529f\u80fd\uff0c\u7136\u540e\u5728\u7279\u5b9a Pod \u4e0a\u589e\u52a0 ovn.kubernetes.io/mirror annotation \u6765\u5f00\u542f Pod \u7ea7\u522b\u6d41\u91cf\u955c\u50cf\u3002 apiVersion : v1 kind : Pod metadata : name : mirror-pod namespace : ls1 annotations : ovn.kubernetes.io/mirror : \"true\" spec : containers : - name : mirror-pod image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6d41\u91cf\u955c\u50cf"},{"location":"guide/mirror/#_1","text":"\u6d41\u91cf\u955c\u50cf\u529f\u80fd\u53ef\u4ee5\u5c06\u8fdb\u51fa\u5bb9\u5668\u7f51\u7edc\u7684\u6570\u636e\u5305\u8fdb\u884c\u590d\u5236\u5230\u4e3b\u673a\u7684\u7279\u5b9a\u7f51\u5361\u3002\u7ba1\u7406\u5458\u6216\u5f00\u53d1\u8005 \u53ef\u4ee5\u901a\u8fc7\u76d1\u542c\u8fd9\u5757\u7f51\u5361\u83b7\u5f97\u5b8c\u6574\u7684\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u6765\u8fdb\u4e00\u6b65\u8fdb\u884c\u5206\u6790\uff0c\u76d1\u63a7\uff0c\u5b89\u5168\u5ba1\u8ba1\u7b49\u64cd\u4f5c\u3002 \u4e5f\u53ef\u548c\u4f20\u7edf\u7684 NPM \u5bf9\u63a5\u83b7\u53d6\u66f4\u7ec6\u7c92\u5ea6\u7684\u6d41\u91cf\u76d1\u63a7\u3002 \u6d41\u91cf\u955c\u50cf\u529f\u80fd\u4f1a\u5e26\u6765\u4e00\u5b9a\u7684\u6027\u80fd\u635f\u5931\uff0c\u6839\u636e CPU \u6027\u80fd\u4ee5\u53ca\u6d41\u91cf\u7684\u7279\u5f81\uff0c\u4f1a\u6709 5%~10% \u7684 \u989d\u5916 CPU \u6d88\u8017\u3002","title":"\u6d41\u91cf\u955c\u50cf"},{"location":"guide/mirror/#_2","text":"\u6d41\u91cf\u955c\u50cf\u529f\u80fd\u9ed8\u8ba4\u4e3a\u5173\u95ed\u72b6\u6001\uff0c\u5982\u679c\u9700\u8981\u5f00\u542f\u8bf7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u542f\u52a8\u53c2\u6570\uff1a --enable-mirror=true \uff1a \u662f\u5426\u5f00\u542f\u6d41\u91cf\u955c\u50cf\u3002 --mirror-iface=mirror0 : \u6d41\u91cf\u955c\u50cf\u6240\u590d\u5236\u5230\u7684\u7f51\u5361\u540d\u3002\u8be5\u7f51\u5361\u53ef\u4e3a\u4e3b\u673a\u4e0a\u5df2\u5b58\u5728\u7684\u4e00\u5757\u7269\u7406\u7f51\u5361\uff0c \u6b64\u65f6\u8be5\u7f51\u5361\u4f1a\u88ab\u6865\u63a5\u8fdb br-int \u7f51\u6865\uff0c\u955c\u50cf\u6d41\u91cf\u4f1a\u76f4\u63a5\u63a5\u5165\u5e95\u5c42\u4ea4\u6362\u673a\u3002\u82e5\u7f51\u5361\u540d\u4e0d\u5b58\u5728\uff0cKube-OVN \u4f1a\u81ea\u52a8 \u521b\u5efa\u4e00\u5757\u540c\u540d\u7684\u865a\u62df\u7f51\u5361\uff0c\u7ba1\u7406\u5458\u6216\u5f00\u53d1\u8005\u53ef\u4ee5\u5728\u5bbf\u4e3b\u673a\u4e0a\u901a\u8fc7\u8be5\u7f51\u5361\u83b7\u53d6\u5f53\u524d\u8282\u70b9\u6240\u6709\u6d41\u91cf\u3002\u9ed8\u8ba4\u4e3a mirror0 \u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u7528 tcpdump \u6216\u5176\u4ed6\u6d41\u91cf\u5206\u6790\u5de5\u5177\u76d1\u542c mirror0 \u4e0a\u7684\u6d41\u91cf\uff1a tcpdump -ni mirror0","title":"\u5168\u5c40\u6d41\u91cf\u955c\u50cf\u914d\u7f6e"},{"location":"guide/mirror/#pod","text":"\u5982\u679c\u53ea\u9700\u5bf9\u90e8\u5206 Pod \u6d41\u91cf\u8fdb\u884c\u955c\u50cf\uff0c\u5219\u9700\u8981\u5173\u95ed\u5168\u5c40\u7684\u6d41\u91cf\u955c\u50cf\u529f\u80fd\uff0c\u7136\u540e\u5728\u7279\u5b9a Pod \u4e0a\u589e\u52a0 ovn.kubernetes.io/mirror annotation \u6765\u5f00\u542f Pod \u7ea7\u522b\u6d41\u91cf\u955c\u50cf\u3002 apiVersion : v1 kind : Pod metadata : name : mirror-pod namespace : ls1 annotations : ovn.kubernetes.io/mirror : \"true\" spec : containers : - name : mirror-pod image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Pod \u7ea7\u522b\u6d41\u91cf\u955c\u50cf\u914d\u7f6e"},{"location":"guide/networkpolicy-log/","text":"NetworkPolicy \u65e5\u5fd7 \u00b6 NetworkPolicy \u4e3a Kubernetes \u63d0\u4f9b\u7684\u7f51\u7edc\u7b56\u7565\u63a5\u53e3\uff0cKube-OVN \u901a\u8fc7 OVN \u7684 ACL \u8fdb\u884c\u4e86\u5b9e\u73b0\u3002 \u4f7f\u7528\u4e86 NetworkPolicy \u540e\u5982\u679c\u51fa\u73b0\u7f51\u7edc\u4e0d\u901a\u7684\u60c5\u51b5\uff0c\u96be\u4ee5\u5224\u65ad\u662f\u7f51\u7edc\u6545\u969c\u95ee\u9898\u8fd8\u662f NetworkPolicy \u89c4\u5219\u8bbe\u7f6e\u95ee\u9898\u5bfc\u81f4\u7684\u7f51\u7edc\u4e2d\u65ad\u3002 Kube-OVN \u63d0\u4f9b\u4e86 NetworkPolicy \u65e5\u5fd7\u529f\u80fd\uff0c\u5e2e\u52a9\u7ba1\u7406\u5458\u5feb\u901f\u5b9a\u4f4d NetworkPolicy Drop \u89c4\u5219\u662f\u5426\u547d\u4e2d\uff0c\u5e76\u8bb0\u5f55\u6709\u54ea\u4e9b\u975e\u6cd5\u8bbf\u95ee\u3002 NetworkPolicy \u65e5\u5fd7\u529f\u80fd\u4e00\u65e6\u5f00\u542f\uff0c\u5bf9\u6bcf\u4e2a\u547d\u4e2d Drop \u89c4\u5219\u7684\u6570\u636e\u5305\u90fd\u9700\u8981\u6253\u5370\u65e5\u5fd7\uff0c\u4f1a\u5e26\u6765\u989d\u5916\u6027\u80fd\u5f00\u9500\u3002 \u5728\u6076\u610f\u653b\u51fb\u4e0b\uff0c\u77ed\u65f6\u95f4\u5927\u91cf\u65e5\u5fd7\u53ef\u80fd\u4f1a\u8017\u5c3d CPU\u3002\u6211\u4eec\u5efa\u8bae\u5728\u751f\u4ea7\u73af\u5883\u9ed8\u8ba4\u5173\u95ed\u65e5\u5fd7\u529f\u80fd\uff0c\u5728\u9700\u8981\u6392\u67e5\u95ee\u9898\u65f6\uff0c\u52a8\u6001\u5f00\u542f\u65e5\u5fd7\u3002 \u5f00\u542f NetworkPolicy \u65e5\u5fd7 \u00b6 \u5728\u9700\u8981\u5f00\u542f\u65e5\u5fd7\u8bb0\u5f55\u7684 NetworkPolicy \u4e2d\u589e\u52a0 annotation ovn.kubernetes.io/enable_log \uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : networking.k8s.io/v1 kind : NetworkPolicy metadata : name : default-deny-ingress namespace : kube-system annotations : ovn.kubernetes.io/enable_log : \"true\" spec : podSelector : {} policyTypes : - Ingress \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728\u5bf9\u5e94 Pod \u6240\u5728\u4e3b\u673a\u7684 /var/log/ovn/ovn-controller.log \u4e2d\u89c2\u5bdf\u5230\u88ab\u4e22\u5f03\u6570\u636e\u5305\u7684\u65e5\u5fd7\uff1a # tail -f /var/log/ovn/ovn-controller.log 2022 -07-20T05:55:03.229Z | 00394 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54343 ,tp_dst = 53 2022 -07-20T05:55:06.229Z | 00395 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 44187 ,tp_dst = 53 2022 -07-20T05:55:08.230Z | 00396 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54274 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00397 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 32778 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00398 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 34188 ,tp_dst = 53 2022 -07-20T05:55:13.231Z | 00399 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 43290 ,tp_dst = 53 2022 -07-20T05:55:22.096Z | 00400 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.097Z | 00401 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.098Z | 00402 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 \u5173\u95ed NetworkPolicy \u65e5\u5fd7 \u00b6 \u5c06\u5bf9\u5e94 NetworkPolicy \u4e2d\u7684 annotation ovn.kubernetes.io/enable_log \u8bbe\u7f6e\u4e3a false \u5373\u53ef\u5173\u95ed NetworkPolicy \u65e5\u5fd7\uff1a kubectl annotate networkpolicy -n kube-system default-deny-ingress ovn.kubernetes.io/enable_log = false --overwrite \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NetworkPolicy \u65e5\u5fd7"},{"location":"guide/networkpolicy-log/#networkpolicy","text":"NetworkPolicy \u4e3a Kubernetes \u63d0\u4f9b\u7684\u7f51\u7edc\u7b56\u7565\u63a5\u53e3\uff0cKube-OVN \u901a\u8fc7 OVN \u7684 ACL \u8fdb\u884c\u4e86\u5b9e\u73b0\u3002 \u4f7f\u7528\u4e86 NetworkPolicy \u540e\u5982\u679c\u51fa\u73b0\u7f51\u7edc\u4e0d\u901a\u7684\u60c5\u51b5\uff0c\u96be\u4ee5\u5224\u65ad\u662f\u7f51\u7edc\u6545\u969c\u95ee\u9898\u8fd8\u662f NetworkPolicy \u89c4\u5219\u8bbe\u7f6e\u95ee\u9898\u5bfc\u81f4\u7684\u7f51\u7edc\u4e2d\u65ad\u3002 Kube-OVN \u63d0\u4f9b\u4e86 NetworkPolicy \u65e5\u5fd7\u529f\u80fd\uff0c\u5e2e\u52a9\u7ba1\u7406\u5458\u5feb\u901f\u5b9a\u4f4d NetworkPolicy Drop \u89c4\u5219\u662f\u5426\u547d\u4e2d\uff0c\u5e76\u8bb0\u5f55\u6709\u54ea\u4e9b\u975e\u6cd5\u8bbf\u95ee\u3002 NetworkPolicy \u65e5\u5fd7\u529f\u80fd\u4e00\u65e6\u5f00\u542f\uff0c\u5bf9\u6bcf\u4e2a\u547d\u4e2d Drop \u89c4\u5219\u7684\u6570\u636e\u5305\u90fd\u9700\u8981\u6253\u5370\u65e5\u5fd7\uff0c\u4f1a\u5e26\u6765\u989d\u5916\u6027\u80fd\u5f00\u9500\u3002 \u5728\u6076\u610f\u653b\u51fb\u4e0b\uff0c\u77ed\u65f6\u95f4\u5927\u91cf\u65e5\u5fd7\u53ef\u80fd\u4f1a\u8017\u5c3d CPU\u3002\u6211\u4eec\u5efa\u8bae\u5728\u751f\u4ea7\u73af\u5883\u9ed8\u8ba4\u5173\u95ed\u65e5\u5fd7\u529f\u80fd\uff0c\u5728\u9700\u8981\u6392\u67e5\u95ee\u9898\u65f6\uff0c\u52a8\u6001\u5f00\u542f\u65e5\u5fd7\u3002","title":"NetworkPolicy \u65e5\u5fd7"},{"location":"guide/networkpolicy-log/#networkpolicy_1","text":"\u5728\u9700\u8981\u5f00\u542f\u65e5\u5fd7\u8bb0\u5f55\u7684 NetworkPolicy \u4e2d\u589e\u52a0 annotation ovn.kubernetes.io/enable_log \uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : networking.k8s.io/v1 kind : NetworkPolicy metadata : name : default-deny-ingress namespace : kube-system annotations : ovn.kubernetes.io/enable_log : \"true\" spec : podSelector : {} policyTypes : - Ingress \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728\u5bf9\u5e94 Pod \u6240\u5728\u4e3b\u673a\u7684 /var/log/ovn/ovn-controller.log \u4e2d\u89c2\u5bdf\u5230\u88ab\u4e22\u5f03\u6570\u636e\u5305\u7684\u65e5\u5fd7\uff1a # tail -f /var/log/ovn/ovn-controller.log 2022 -07-20T05:55:03.229Z | 00394 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54343 ,tp_dst = 53 2022 -07-20T05:55:06.229Z | 00395 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 44187 ,tp_dst = 53 2022 -07-20T05:55:08.230Z | 00396 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54274 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00397 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 32778 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00398 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 34188 ,tp_dst = 53 2022 -07-20T05:55:13.231Z | 00399 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 43290 ,tp_dst = 53 2022 -07-20T05:55:22.096Z | 00400 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.097Z | 00401 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.098Z | 00402 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0","title":"\u5f00\u542f NetworkPolicy \u65e5\u5fd7"},{"location":"guide/networkpolicy-log/#networkpolicy_2","text":"\u5c06\u5bf9\u5e94 NetworkPolicy \u4e2d\u7684 annotation ovn.kubernetes.io/enable_log \u8bbe\u7f6e\u4e3a false \u5373\u53ef\u5173\u95ed NetworkPolicy \u65e5\u5fd7\uff1a kubectl annotate networkpolicy -n kube-system default-deny-ingress ovn.kubernetes.io/enable_log = false --overwrite \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5173\u95ed NetworkPolicy \u65e5\u5fd7"},{"location":"guide/prometheus-grafana/","text":"\u914d\u7f6e\u76d1\u63a7\u548c\u9762\u677f \u00b6 Kube-OVN \u53ef\u4ee5\u5c06\u7f51\u7edc\u63a7\u5236\u5e73\u9762\u4fe1\u606f\u4ee5\u53ca\u7f51\u7edc\u6570\u636e\u5e73\u9762\u8d28\u91cf\u4fe1\u606f\u6307\u6807\u4ee5 Prometheus \u6240\u652f\u6301\u7684\u683c\u5f0f\u5bf9\u5916\u8f93\u51fa\u3002 \u6211\u4eec\u4f7f\u7528 kube-prometheus \u6240\u63d0\u4f9b\u7684 CRD \u6765\u5b9a\u4e49\u76f8\u5e94\u7684 Prometheus \u76d1\u63a7\u89c4\u5219\u3002 \u7528\u6237\u9700\u8981\u9884\u5148\u5b89\u88c5 kube-prometheus \u6765\u542f\u7528\u76f8\u5173\u7684 CRD\u3002Kube-OVN \u6240\u652f\u6301\u7684\u5168\u90e8\u76d1\u63a7\u6307\u6807\u8bf7\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002 \u5982\u679c\u4f7f\u7528\u539f\u751f Prometheus \u8bf7\u53c2\u8003 \u914d\u7f6e\u539f\u751f Prometheus \u83b7\u53d6\u76d1\u63a7\u6570\u636e \u8fdb\u884c\u914d\u7f6e\u3002 \u5b89\u88c5 Prometheus Monitor \u00b6 Kube-OVN \u4f7f\u7528 Prometheus Monitor CRD \u6765\u7ba1\u7406\u76d1\u63a7\u8f93\u51fa\uff1a # \u7f51\u54af\u8d28\u91cf\u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-monitor.yaml # kube-ovn-controller \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-monitor.yaml # kube-ovn-cni \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-monitor.yaml # ovn \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-monitor.yaml Prometheus \u62c9\u53d6\u76d1\u63a7\u65f6\u95f4\u95f4\u9694\u9ed8\u8ba4\u4e3a 15s\uff0c\u5982\u679c\u9700\u8981\u8c03\u6574\u9700\u8981\u4fee\u6539 yaml \u4e2d\u7684 interval \u5b57\u6bb5\u3002 \u52a0\u8f7d Grafana \u9762\u677f \u00b6 Kube-OVN \u8fd8\u63d0\u4f9b\u4e86\u9884\u5148\u5b9a\u4e49\u597d\u7684 Grafana Dashboard \u5c55\u793a\u63a7\u5236\u5e73\u9762\u548c\u6570\u636e\u5e73\u9762\u76f8\u5173\u4fe1\u606f\u3002 \u4e0b\u8f7d\u5bf9\u5e94 Dashboard \u6a21\u677f\uff1a # \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-grafana.json # kube-ovn-controller \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-grafana.json # kube-ovn-cni \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-grafana.json # ovn \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-grafana.json # ovs \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovs-grafana.json \u5728 Grafana \u4e2d\u5bfc\u5165\u6a21\u677f\uff0c\u5e76\u5c06\u6570\u636e\u6e90\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 Prometheus \u5373\u53ef\u770b\u5230\u5982\u4e0b Dashboard\uff1a kube-ovn-controller \u8fd0\u884c\u72b6\u51b5\u76f8\u5173\u9762\u677f\uff1a kube-ovn-pinger \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u9762\u677f\uff1a kube-ovn-cni \u8fd0\u884c\u72b6\u51b5\u76f8\u5173\u9762\u677f\uff1a \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u914d\u7f6e\u76d1\u63a7\u548c\u9762\u677f"},{"location":"guide/prometheus-grafana/#_1","text":"Kube-OVN \u53ef\u4ee5\u5c06\u7f51\u7edc\u63a7\u5236\u5e73\u9762\u4fe1\u606f\u4ee5\u53ca\u7f51\u7edc\u6570\u636e\u5e73\u9762\u8d28\u91cf\u4fe1\u606f\u6307\u6807\u4ee5 Prometheus \u6240\u652f\u6301\u7684\u683c\u5f0f\u5bf9\u5916\u8f93\u51fa\u3002 \u6211\u4eec\u4f7f\u7528 kube-prometheus \u6240\u63d0\u4f9b\u7684 CRD \u6765\u5b9a\u4e49\u76f8\u5e94\u7684 Prometheus \u76d1\u63a7\u89c4\u5219\u3002 \u7528\u6237\u9700\u8981\u9884\u5148\u5b89\u88c5 kube-prometheus \u6765\u542f\u7528\u76f8\u5173\u7684 CRD\u3002Kube-OVN \u6240\u652f\u6301\u7684\u5168\u90e8\u76d1\u63a7\u6307\u6807\u8bf7\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002 \u5982\u679c\u4f7f\u7528\u539f\u751f Prometheus \u8bf7\u53c2\u8003 \u914d\u7f6e\u539f\u751f Prometheus \u83b7\u53d6\u76d1\u63a7\u6570\u636e \u8fdb\u884c\u914d\u7f6e\u3002","title":"\u914d\u7f6e\u76d1\u63a7\u548c\u9762\u677f"},{"location":"guide/prometheus-grafana/#prometheus-monitor","text":"Kube-OVN \u4f7f\u7528 Prometheus Monitor CRD \u6765\u7ba1\u7406\u76d1\u63a7\u8f93\u51fa\uff1a # \u7f51\u54af\u8d28\u91cf\u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-monitor.yaml # kube-ovn-controller \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-monitor.yaml # kube-ovn-cni \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-monitor.yaml # ovn \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-monitor.yaml Prometheus \u62c9\u53d6\u76d1\u63a7\u65f6\u95f4\u95f4\u9694\u9ed8\u8ba4\u4e3a 15s\uff0c\u5982\u679c\u9700\u8981\u8c03\u6574\u9700\u8981\u4fee\u6539 yaml \u4e2d\u7684 interval \u5b57\u6bb5\u3002","title":"\u5b89\u88c5 Prometheus Monitor"},{"location":"guide/prometheus-grafana/#grafana","text":"Kube-OVN \u8fd8\u63d0\u4f9b\u4e86\u9884\u5148\u5b9a\u4e49\u597d\u7684 Grafana Dashboard \u5c55\u793a\u63a7\u5236\u5e73\u9762\u548c\u6570\u636e\u5e73\u9762\u76f8\u5173\u4fe1\u606f\u3002 \u4e0b\u8f7d\u5bf9\u5e94 Dashboard \u6a21\u677f\uff1a # \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-grafana.json # kube-ovn-controller \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-grafana.json # kube-ovn-cni \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-grafana.json # ovn \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-grafana.json # ovs \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovs-grafana.json \u5728 Grafana \u4e2d\u5bfc\u5165\u6a21\u677f\uff0c\u5e76\u5c06\u6570\u636e\u6e90\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 Prometheus \u5373\u53ef\u770b\u5230\u5982\u4e0b Dashboard\uff1a kube-ovn-controller \u8fd0\u884c\u72b6\u51b5\u76f8\u5173\u9762\u677f\uff1a kube-ovn-pinger \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u9762\u677f\uff1a kube-ovn-cni \u8fd0\u884c\u72b6\u51b5\u76f8\u5173\u9762\u677f\uff1a \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u52a0\u8f7d Grafana \u9762\u677f"},{"location":"guide/prometheus/","text":"\u914d\u7f6e\u539f\u751f Prometheus \u83b7\u53d6\u76d1\u63a7\u6570\u636e \u00b6 Kube-OVN \u63d0\u4f9b\u4e86\u4e30\u5bcc\u7684\u76d1\u63a7\u6570\u636e\uff0c\u7528\u4e8e OVN/OVS \u5065\u5eb7\u72b6\u6001\u68c0\u67e5\uff0c\u4ee5\u53ca\u5bb9\u5668\u7f51\u7edc\u548c\u4e3b\u673a\u7f51\u7edc\u7684\u8fde\u901a\u6027\u68c0\u67e5\u3002Kube-OVN \u914d\u7f6e\u4e86 ServiceMonitor\uff0c\u53ef\u4ee5\u7528\u4e8e Prometheus \u52a8\u6001\u83b7\u53d6\u76d1\u63a7\u6307\u6807\u3002 \u5728\u67d0\u4e9b\u60c5\u51b5\u4e0b\uff0c\u53ea\u5b89\u88c5\u4e86 Prometheus Server\uff0c\u6ca1\u6709\u5b89\u88c5\u5176\u4ed6\u7684\u7ec4\u4ef6\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 Prometheus \u7684\u914d\u7f6e\uff0c\u52a8\u6001\u83b7\u53d6\u96c6\u7fa4\u73af\u5883\u7684\u76d1\u63a7\u6570\u636e\u3002 Prometheus \u914d\u7f6e \u00b6 \u4ee5\u4e0b\u7684\u914d\u7f6e\u6587\u6863\uff0c\u53c2\u8003\u81ea Prometheus \u670d\u52a1\u53d1\u73b0 \u3002 \u6743\u9650\u914d\u7f6e \u00b6 Prometheus \u90e8\u7f72\u5728\u96c6\u7fa4\u5185\uff0c\u9700\u8981\u901a\u8fc7 k8s apiserver \u6765\u8bbf\u95ee\u96c6\u7fa4\u5185\u7684\u8d44\u6e90\uff0c\u4ece\u800c\u5b9e\u73b0\u67e5\u8be2\u4e1a\u52a1\u7684\u76d1\u63a7\u6570\u636e\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u914d\u7f6e Prometheus \u9700\u8981\u7684\u6743\u9650\uff1a apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : prometheus rules : - apiGroups : [ \"\" ] resources : - nodes - nodes/proxy - services - endpoints - pods verbs : [ \"get\" , \"list\" , \"watch\" ] - apiGroups : - extensions resources : - ingresses verbs : [ \"get\" , \"list\" , \"watch\" ] - nonResourceURLs : [ \"/metrics\" ] verbs : [ \"get\" ] --- apiVersion : v1 kind : ServiceAccount metadata : name : prometheus namespace : default --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : prometheus roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : prometheus subjects : - kind : ServiceAccount name : prometheus namespace : default Prometheus \u914d\u7f6e\u6587\u4ef6 \u00b6 Prometheus \u7684\u542f\u52a8\uff0c\u4f9d\u8d56\u4e8e\u914d\u7f6e\u6587\u4ef6 prometheus.yml\uff0c\u53ef\u4ee5\u5c06\u8be5\u6587\u4ef6\u5185\u5bb9\u914d\u7f6e\u5728 ConfigMap \u5185\uff0c\u52a8\u6001\u6302\u8f7d\u5230 Pod \u4e2d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa Prometheus \u4f7f\u7528\u7684 ConfigMap \u6587\u4ef6\uff1a apiVersion : v1 kind : ConfigMap metadata : name : prometheus-config data : prometheus.yml : |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'kubernetes-nodes' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node - job_name: 'kubernetes-service' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: service - job_name: 'kubernetes-endpoints' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints - job_name: 'kubernetes-ingress' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: ingress - job_name: 'kubernetes-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: pod Prometheus \u63d0\u4f9b\u4e86\u57fa\u4e8e\u89d2\u8272\u67e5\u8be2 Kubernetes \u8d44\u6e90\u76d1\u63a7\u7684\u64cd\u4f5c\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u67e5\u770b\u5b98\u65b9\u6587\u6863 kubernetes_sd_config \u3002 \u5728 Kubernetes \u96c6\u7fa4\u4e2d\uff0cPrometheus \u652f\u6301\u67e5\u8be2\u76d1\u63a7\u6307\u6807\u7684\u89d2\u8272\u5305\u542b node\u3001service\u3001pod\u3001endpoints \u548c ingress\u3002\u5728 ConfigMap \u914d\u7f6e\u6587\u4ef6\u4e2d\u7ed9\u51fa\u4e86\u4ee5\u4e0a\u5168\u90e8\u8d44\u6e90\u7684\u76d1\u63a7\u67e5\u8be2\u914d\u7f6e\u793a\u4f8b\uff0c\u53ef\u4ee5\u6839\u636e\u9700\u8981\u9009\u62e9\u914d\u7f6e\u3002 Prometheus \u90e8\u7f72 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml \u6587\u4ef6\uff0c\u90e8\u7f72 Prometheus Server\uff1a apiVersion : apps/v1 kind : Deployment metadata : labels : app : prometheus name : prometheus namespace : default spec : replicas : 1 selector : matchLabels : app : prometheus strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : prometheus spec : serviceAccountName : prometheus serviceAccount : prometheus containers : - image : docker.io/prom/prometheus:latest imagePullPolicy : IfNotPresent name : prometheus command : - \"/bin/prometheus\" args : - \"--config.file=/etc/prometheus/prometheus.yml\" ports : - containerPort : 9090 protocol : TCP volumeMounts : - mountPath : \"/etc/prometheus\" name : prometheus-config volumes : - name : prometheus-config configMap : name : prometheus-config \u5728\u90e8\u7f72\u5b8c Prometheus \u4e4b\u540e\uff0c\u53c2\u8003\u4ee5\u4e0b yaml \u6587\u4ef6\uff0c\u90e8\u7f72 Prometheus Service\uff1a kind : Service apiVersion : v1 metadata : name : prometheus namespace : default labels : name : prometheus spec : ports : - name : test protocol : TCP port : 9090 targetPort : 9090 type : NodePort selector : app : prometheus sessionAffinity : None \u5c06 Prometheus \u901a\u8fc7 NodePort \u66b4\u9732\u540e\uff0c\u5373\u53ef\u901a\u8fc7\u8282\u70b9\u6765\u8bbf\u95ee Prometheus\u3002 Prometheus \u76d1\u63a7\u6570\u636e\u9a8c\u8bc1 \u00b6 \u67e5\u770b\u73af\u5883\u4e0a Prometheus \u76f8\u5173\u7684\u4fe1\u606f\uff1a # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kubernetes ClusterIP 10 .4.0.1 <none> 443 /TCP 8d prometheus NodePort 10 .4.102.222 <none> 9090 :32611/TCP 8d # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-7544b6b84d-v9m8s 1 /1 Running 0 3d5h 10 .3.0.7 192 .168.137.219 <none> <none> # kubectl get endpoints -o wide NAME ENDPOINTS AGE kubernetes 192 .168.136.228:6443,192.168.136.232:6443,192.168.137.219:6443 8d prometheus 10 .3.0.7:9090 8d \u901a\u8fc7 NodePort \u8bbf\u95ee Prometheus\uff0c\u67e5\u770b Status/Service Discovery \u52a8\u6001\u67e5\u8be2\u5230\u7684\u6570\u636e\uff1a \u53ef\u4ee5\u770b\u5230\u5f53\u524d\u53ef\u4ee5\u67e5\u8be2\u5230\u96c6\u7fa4\u4e0a\u5168\u90e8\u7684 Service \u6570\u636e\u4fe1\u606f\u3002 \u914d\u7f6e\u67e5\u8be2\u6307\u5b9a\u7684\u8d44\u6e90 \u00b6 \u4ee5\u4e0a\u7684 ConfigMap \u914d\u7f6e\u4e2d\uff0c\u6ca1\u6709\u6dfb\u52a0\u8fc7\u6ee4\u6761\u4ef6\uff0c\u67e5\u8be2\u4e86\u6240\u6709\u7684\u8d44\u6e90\u6570\u636e\u3002\u5982\u679c\u53ea\u9700\u8981\u67d0\u4e2a\u89d2\u8272\u7684\u8d44\u6e90\u6570\u636e\uff0c\u5219\u53ef\u4ee5\u6dfb\u52a0\u8fc7\u6ee4\u6761\u4ef6\u3002 \u4ee5 Service \u4e3a\u4f8b\uff0c\u4fee\u6539 ConfigMap \u5185\u5bb9\uff0c\u53ea\u67e5\u8be2\u5173\u5fc3\u7684 Service \u76d1\u63a7\u6570\u636e\u3002 - job_name : 'kubernetes-service' tls_config : ca_file : /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file : /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs : - role : service relabel_configs : - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_scrape ] action : \"keep\" regex : \"true\" - action : labelmap regex : __meta_kubernetes_service_label_(.+) - source_labels : [ __meta_kubernetes_namespace ] target_label : kubernetes_namespace - source_labels : [ __meta_kubernetes_service_name ] target_label : kubernetes_service_name - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_path ] action : replace target_label : __metrics_path__ regex : \"(.+)\" Service \u9ed8\u8ba4\u76d1\u63a7\u8def\u5f84\u4e3a /metrics\u3002\u5982\u679c Service \u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\u662f\u5176\u4ed6\u7684\u8def\u5f84\uff0c\u53ef\u4ee5\u901a\u8fc7\u7ed9 Service \u6dfb\u52a0 annotation prometheus.io/path \u6765\u6307\u5b9a\u91c7\u96c6\u8def\u5f84\u3002 \u5e94\u7528\u4ee5\u4e0a yaml\uff0c\u66f4\u65b0 ConfigMap \u4fe1\u606f\uff0c\u91cd\u5efa Prometheus Pod\uff0c\u4f7f\u914d\u7f6e\u751f\u6548\u3002 \u67e5\u770b kube-system Namespace \u4e0b\u7684 Service \u4fe1\u606f\uff1a # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kube-dns ClusterIP 10 .4.0.10 <none> 53 /UDP,53/TCP,9153/TCP 13d kube-ovn-cni ClusterIP 10 .4.228.60 <none> 10665 /TCP 13d kube-ovn-controller ClusterIP 10 .4.172.213 <none> 10660 /TCP 13d kube-ovn-monitor ClusterIP 10 .4.242.9 <none> 10661 /TCP 13d kube-ovn-pinger ClusterIP 10 .4.122.52 <none> 8080 /TCP 13d ovn-nb ClusterIP 10 .4.80.213 <none> 6641 /TCP 13d ovn-northd ClusterIP 10 .4.126.234 <none> 6643 /TCP 13d ovn-sb ClusterIP 10 .4.216.249 <none> 6642 /TCP 13d \u7ed9 Service \u6dfb\u52a0 annotation prometheus.io/scrape=\"true\" \uff1a # kubectl annotate svc -n kube-system kube-ovn-cni prometheus.io/scrape=true service/kube-ovn-cni annotated # kubectl annotate svc -n kube-system kube-ovn-controller prometheus.io/scrape=true service/kube-ovn-controller annotated # kubectl annotate svc -n kube-system kube-ovn-monitor prometheus.io/scrape=true service/kube-ovn-monitor annotated # kubectl annotate svc -n kube-system kube-ovn-pinger prometheus.io/scrape=true service/kube-ovn-pinger annotated \u67e5\u770b\u914d\u7f6e\u540e\u7684 Service \u4fe1\u606f\uff1a # kubectl get svc -o yaml -n kube-system kube-ovn-controller apiVersion: v1 kind: Service metadata: annotations: helm.sh/chart-version: v3.10.0-alpha.55 helm.sh/original-name: kube-ovn-controller ovn.kubernetes.io/vpc: ovn-cluster prometheus.io/scrape: \"true\" // \u6dfb\u52a0\u7684 annotation labels: app: kube-ovn-controller name: kube-ovn-controller namespace: kube-system spec: clusterIP: 10 .4.172.213 clusterIPs: - 10 .4.172.213 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: metrics port: 10660 protocol: TCP targetPort: 10660 selector: app: kube-ovn-controller sessionAffinity: None type: ClusterIP status: loadBalancer: {} \u67e5\u770b Prometheus Status Targets \u4fe1\u606f\uff0c\u53ef\u4ee5\u770b\u5230\u53ea\u6709\u6dfb\u52a0\u4e86 annotation \u7684 Service \u88ab\u8fc7\u6ee4\u51fa\u6765\uff1a \u66f4\u591a\u5173\u4e8e relabel \u6dfb\u52a0\u8fc7\u6ee4\u53c2\u6570\u7684\u4fe1\u606f\uff0c\u53ef\u4ee5\u53c2\u8003 Prometheus-Relabel \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u914d\u7f6e\u539f\u751f Prometheus"},{"location":"guide/prometheus/#prometheus","text":"Kube-OVN \u63d0\u4f9b\u4e86\u4e30\u5bcc\u7684\u76d1\u63a7\u6570\u636e\uff0c\u7528\u4e8e OVN/OVS \u5065\u5eb7\u72b6\u6001\u68c0\u67e5\uff0c\u4ee5\u53ca\u5bb9\u5668\u7f51\u7edc\u548c\u4e3b\u673a\u7f51\u7edc\u7684\u8fde\u901a\u6027\u68c0\u67e5\u3002Kube-OVN \u914d\u7f6e\u4e86 ServiceMonitor\uff0c\u53ef\u4ee5\u7528\u4e8e Prometheus \u52a8\u6001\u83b7\u53d6\u76d1\u63a7\u6307\u6807\u3002 \u5728\u67d0\u4e9b\u60c5\u51b5\u4e0b\uff0c\u53ea\u5b89\u88c5\u4e86 Prometheus Server\uff0c\u6ca1\u6709\u5b89\u88c5\u5176\u4ed6\u7684\u7ec4\u4ef6\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 Prometheus \u7684\u914d\u7f6e\uff0c\u52a8\u6001\u83b7\u53d6\u96c6\u7fa4\u73af\u5883\u7684\u76d1\u63a7\u6570\u636e\u3002","title":"\u914d\u7f6e\u539f\u751f Prometheus \u83b7\u53d6\u76d1\u63a7\u6570\u636e"},{"location":"guide/prometheus/#prometheus_1","text":"\u4ee5\u4e0b\u7684\u914d\u7f6e\u6587\u6863\uff0c\u53c2\u8003\u81ea Prometheus \u670d\u52a1\u53d1\u73b0 \u3002","title":"Prometheus \u914d\u7f6e"},{"location":"guide/prometheus/#_1","text":"Prometheus \u90e8\u7f72\u5728\u96c6\u7fa4\u5185\uff0c\u9700\u8981\u901a\u8fc7 k8s apiserver \u6765\u8bbf\u95ee\u96c6\u7fa4\u5185\u7684\u8d44\u6e90\uff0c\u4ece\u800c\u5b9e\u73b0\u67e5\u8be2\u4e1a\u52a1\u7684\u76d1\u63a7\u6570\u636e\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u914d\u7f6e Prometheus \u9700\u8981\u7684\u6743\u9650\uff1a apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : prometheus rules : - apiGroups : [ \"\" ] resources : - nodes - nodes/proxy - services - endpoints - pods verbs : [ \"get\" , \"list\" , \"watch\" ] - apiGroups : - extensions resources : - ingresses verbs : [ \"get\" , \"list\" , \"watch\" ] - nonResourceURLs : [ \"/metrics\" ] verbs : [ \"get\" ] --- apiVersion : v1 kind : ServiceAccount metadata : name : prometheus namespace : default --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : prometheus roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : prometheus subjects : - kind : ServiceAccount name : prometheus namespace : default","title":"\u6743\u9650\u914d\u7f6e"},{"location":"guide/prometheus/#prometheus_2","text":"Prometheus \u7684\u542f\u52a8\uff0c\u4f9d\u8d56\u4e8e\u914d\u7f6e\u6587\u4ef6 prometheus.yml\uff0c\u53ef\u4ee5\u5c06\u8be5\u6587\u4ef6\u5185\u5bb9\u914d\u7f6e\u5728 ConfigMap \u5185\uff0c\u52a8\u6001\u6302\u8f7d\u5230 Pod \u4e2d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa Prometheus \u4f7f\u7528\u7684 ConfigMap \u6587\u4ef6\uff1a apiVersion : v1 kind : ConfigMap metadata : name : prometheus-config data : prometheus.yml : |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'kubernetes-nodes' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node - job_name: 'kubernetes-service' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: service - job_name: 'kubernetes-endpoints' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints - job_name: 'kubernetes-ingress' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: ingress - job_name: 'kubernetes-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: pod Prometheus \u63d0\u4f9b\u4e86\u57fa\u4e8e\u89d2\u8272\u67e5\u8be2 Kubernetes \u8d44\u6e90\u76d1\u63a7\u7684\u64cd\u4f5c\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u67e5\u770b\u5b98\u65b9\u6587\u6863 kubernetes_sd_config \u3002 \u5728 Kubernetes \u96c6\u7fa4\u4e2d\uff0cPrometheus \u652f\u6301\u67e5\u8be2\u76d1\u63a7\u6307\u6807\u7684\u89d2\u8272\u5305\u542b node\u3001service\u3001pod\u3001endpoints \u548c ingress\u3002\u5728 ConfigMap \u914d\u7f6e\u6587\u4ef6\u4e2d\u7ed9\u51fa\u4e86\u4ee5\u4e0a\u5168\u90e8\u8d44\u6e90\u7684\u76d1\u63a7\u67e5\u8be2\u914d\u7f6e\u793a\u4f8b\uff0c\u53ef\u4ee5\u6839\u636e\u9700\u8981\u9009\u62e9\u914d\u7f6e\u3002","title":"Prometheus \u914d\u7f6e\u6587\u4ef6"},{"location":"guide/prometheus/#prometheus_3","text":"\u53c2\u8003\u4ee5\u4e0b yaml \u6587\u4ef6\uff0c\u90e8\u7f72 Prometheus Server\uff1a apiVersion : apps/v1 kind : Deployment metadata : labels : app : prometheus name : prometheus namespace : default spec : replicas : 1 selector : matchLabels : app : prometheus strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : prometheus spec : serviceAccountName : prometheus serviceAccount : prometheus containers : - image : docker.io/prom/prometheus:latest imagePullPolicy : IfNotPresent name : prometheus command : - \"/bin/prometheus\" args : - \"--config.file=/etc/prometheus/prometheus.yml\" ports : - containerPort : 9090 protocol : TCP volumeMounts : - mountPath : \"/etc/prometheus\" name : prometheus-config volumes : - name : prometheus-config configMap : name : prometheus-config \u5728\u90e8\u7f72\u5b8c Prometheus \u4e4b\u540e\uff0c\u53c2\u8003\u4ee5\u4e0b yaml \u6587\u4ef6\uff0c\u90e8\u7f72 Prometheus Service\uff1a kind : Service apiVersion : v1 metadata : name : prometheus namespace : default labels : name : prometheus spec : ports : - name : test protocol : TCP port : 9090 targetPort : 9090 type : NodePort selector : app : prometheus sessionAffinity : None \u5c06 Prometheus \u901a\u8fc7 NodePort \u66b4\u9732\u540e\uff0c\u5373\u53ef\u901a\u8fc7\u8282\u70b9\u6765\u8bbf\u95ee Prometheus\u3002","title":"Prometheus \u90e8\u7f72"},{"location":"guide/prometheus/#prometheus_4","text":"\u67e5\u770b\u73af\u5883\u4e0a Prometheus \u76f8\u5173\u7684\u4fe1\u606f\uff1a # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kubernetes ClusterIP 10 .4.0.1 <none> 443 /TCP 8d prometheus NodePort 10 .4.102.222 <none> 9090 :32611/TCP 8d # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-7544b6b84d-v9m8s 1 /1 Running 0 3d5h 10 .3.0.7 192 .168.137.219 <none> <none> # kubectl get endpoints -o wide NAME ENDPOINTS AGE kubernetes 192 .168.136.228:6443,192.168.136.232:6443,192.168.137.219:6443 8d prometheus 10 .3.0.7:9090 8d \u901a\u8fc7 NodePort \u8bbf\u95ee Prometheus\uff0c\u67e5\u770b Status/Service Discovery \u52a8\u6001\u67e5\u8be2\u5230\u7684\u6570\u636e\uff1a \u53ef\u4ee5\u770b\u5230\u5f53\u524d\u53ef\u4ee5\u67e5\u8be2\u5230\u96c6\u7fa4\u4e0a\u5168\u90e8\u7684 Service \u6570\u636e\u4fe1\u606f\u3002","title":"Prometheus \u76d1\u63a7\u6570\u636e\u9a8c\u8bc1"},{"location":"guide/prometheus/#_2","text":"\u4ee5\u4e0a\u7684 ConfigMap \u914d\u7f6e\u4e2d\uff0c\u6ca1\u6709\u6dfb\u52a0\u8fc7\u6ee4\u6761\u4ef6\uff0c\u67e5\u8be2\u4e86\u6240\u6709\u7684\u8d44\u6e90\u6570\u636e\u3002\u5982\u679c\u53ea\u9700\u8981\u67d0\u4e2a\u89d2\u8272\u7684\u8d44\u6e90\u6570\u636e\uff0c\u5219\u53ef\u4ee5\u6dfb\u52a0\u8fc7\u6ee4\u6761\u4ef6\u3002 \u4ee5 Service \u4e3a\u4f8b\uff0c\u4fee\u6539 ConfigMap \u5185\u5bb9\uff0c\u53ea\u67e5\u8be2\u5173\u5fc3\u7684 Service \u76d1\u63a7\u6570\u636e\u3002 - job_name : 'kubernetes-service' tls_config : ca_file : /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file : /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs : - role : service relabel_configs : - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_scrape ] action : \"keep\" regex : \"true\" - action : labelmap regex : __meta_kubernetes_service_label_(.+) - source_labels : [ __meta_kubernetes_namespace ] target_label : kubernetes_namespace - source_labels : [ __meta_kubernetes_service_name ] target_label : kubernetes_service_name - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_path ] action : replace target_label : __metrics_path__ regex : \"(.+)\" Service \u9ed8\u8ba4\u76d1\u63a7\u8def\u5f84\u4e3a /metrics\u3002\u5982\u679c Service \u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\u662f\u5176\u4ed6\u7684\u8def\u5f84\uff0c\u53ef\u4ee5\u901a\u8fc7\u7ed9 Service \u6dfb\u52a0 annotation prometheus.io/path \u6765\u6307\u5b9a\u91c7\u96c6\u8def\u5f84\u3002 \u5e94\u7528\u4ee5\u4e0a yaml\uff0c\u66f4\u65b0 ConfigMap \u4fe1\u606f\uff0c\u91cd\u5efa Prometheus Pod\uff0c\u4f7f\u914d\u7f6e\u751f\u6548\u3002 \u67e5\u770b kube-system Namespace \u4e0b\u7684 Service \u4fe1\u606f\uff1a # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kube-dns ClusterIP 10 .4.0.10 <none> 53 /UDP,53/TCP,9153/TCP 13d kube-ovn-cni ClusterIP 10 .4.228.60 <none> 10665 /TCP 13d kube-ovn-controller ClusterIP 10 .4.172.213 <none> 10660 /TCP 13d kube-ovn-monitor ClusterIP 10 .4.242.9 <none> 10661 /TCP 13d kube-ovn-pinger ClusterIP 10 .4.122.52 <none> 8080 /TCP 13d ovn-nb ClusterIP 10 .4.80.213 <none> 6641 /TCP 13d ovn-northd ClusterIP 10 .4.126.234 <none> 6643 /TCP 13d ovn-sb ClusterIP 10 .4.216.249 <none> 6642 /TCP 13d \u7ed9 Service \u6dfb\u52a0 annotation prometheus.io/scrape=\"true\" \uff1a # kubectl annotate svc -n kube-system kube-ovn-cni prometheus.io/scrape=true service/kube-ovn-cni annotated # kubectl annotate svc -n kube-system kube-ovn-controller prometheus.io/scrape=true service/kube-ovn-controller annotated # kubectl annotate svc -n kube-system kube-ovn-monitor prometheus.io/scrape=true service/kube-ovn-monitor annotated # kubectl annotate svc -n kube-system kube-ovn-pinger prometheus.io/scrape=true service/kube-ovn-pinger annotated \u67e5\u770b\u914d\u7f6e\u540e\u7684 Service \u4fe1\u606f\uff1a # kubectl get svc -o yaml -n kube-system kube-ovn-controller apiVersion: v1 kind: Service metadata: annotations: helm.sh/chart-version: v3.10.0-alpha.55 helm.sh/original-name: kube-ovn-controller ovn.kubernetes.io/vpc: ovn-cluster prometheus.io/scrape: \"true\" // \u6dfb\u52a0\u7684 annotation labels: app: kube-ovn-controller name: kube-ovn-controller namespace: kube-system spec: clusterIP: 10 .4.172.213 clusterIPs: - 10 .4.172.213 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: metrics port: 10660 protocol: TCP targetPort: 10660 selector: app: kube-ovn-controller sessionAffinity: None type: ClusterIP status: loadBalancer: {} \u67e5\u770b Prometheus Status Targets \u4fe1\u606f\uff0c\u53ef\u4ee5\u770b\u5230\u53ea\u6709\u6dfb\u52a0\u4e86 annotation \u7684 Service \u88ab\u8fc7\u6ee4\u51fa\u6765\uff1a \u66f4\u591a\u5173\u4e8e relabel \u6dfb\u52a0\u8fc7\u6ee4\u53c2\u6570\u7684\u4fe1\u606f\uff0c\u53ef\u4ee5\u53c2\u8003 Prometheus-Relabel \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u914d\u7f6e\u67e5\u8be2\u6307\u5b9a\u7684\u8d44\u6e90"},{"location":"guide/qos/","text":"\u5bb9\u5668\u7f51\u7edc QoS \u914d\u7f6e \u00b6 Kube-OVN \u652f\u6301\u57fa\u4e8e\u5355\u4e2a Pod \u7684\u4e24\u79cd\u4e0d\u540c\u7c7b\u578b\u7684 QoS\uff1a \u6700\u5927\u5e26\u5bbd\u9650\u5236 QoS\u3002 linux-netem \uff0c\u6a21\u62df\u8bbe\u5907\u5e72\u6270\u4e22\u5305\u7b49\u7684 QoS\uff0c\u53ef\u7528\u4e8e\u6a21\u62df\u6d4b\u8bd5\u3002 \u76ee\u524d\u53ea\u652f\u6301 Pod \u7ea7\u522b QoS \u4e0d\u652f\u6301 Namespace \u6216 Subnet \u7ea7\u522b\u7684 QoS \u9650\u5236\u3002 \u57fa\u4e8e\u6700\u5927\u5e26\u5bbd\u9650\u5236\u7684 QoS \u00b6 \u8be5\u7c7b\u578b\u7684 QoS \u53ef\u4ee5\u901a\u8fc7 Pod annotation \u52a8\u6001\u8fdb\u884c\u914d\u7f6e\uff0c\u53ef\u4ee5\u5728\u4e0d\u4e2d\u65ad Pod \u8fd0\u884c\u7684\u60c5\u51b5\u4e0b\u8fdb\u884c\u8c03\u6574\u3002 \u5e26\u5bbd\u9650\u901f\u7684\u5355\u4f4d\u4e3a Mbit/s \u3002 apiVersion : v1 kind : Pod metadata : name : qos namespace : ls1 annotations : ovn.kubernetes.io/ingress_rate : \"3\" ovn.kubernetes.io/egress_rate : \"1\" spec : containers : - name : qos image : docker.io/library/nginx:alpine \u4f7f\u7528 annotation \u52a8\u6001\u8c03\u6574 QoS\uff1a kubectl annotate --overwrite pod nginx-74d5899f46-d7qkn ovn.kubernetes.io/ingress_rate = 3 \u6d4b\u8bd5 QoS \u8c03\u6574 \u00b6 \u90e8\u7f72\u6027\u80fd\u6d4b\u8bd5\u9700\u8981\u7684\u5bb9\u5668\uff1a kind : DaemonSet apiVersion : apps/v1 metadata : name : perf namespace : ls1 labels : app : perf spec : selector : matchLabels : app : perf template : metadata : labels : app : perf spec : containers : - name : nginx image : docker.io/kubeovn/perf \u8fdb\u5165\u5176\u4e2d\u4e00\u4e2a Pod \u5e76\u5f00\u542f iperf3 server\uff1a # kubectl exec -it perf-4n4gt -n ls1 sh # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- \u8fdb\u5165\u53e6\u4e00\u4e2a Pod \u8bf7\u6c42\u4e4b\u524d\u7684 Pod\uff1a # kubectl exec -it perf-d4mqc -n ls1 sh # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 51544 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 86 .4 MBytes 725 Mbits/sec 3 350 KBytes [ 4 ] 1 .00-2.00 sec 89 .9 MBytes 754 Mbits/sec 118 473 KBytes [ 4 ] 2 .00-3.00 sec 101 MBytes 848 Mbits/sec 184 586 KBytes [ 4 ] 3 .00-4.00 sec 104 MBytes 875 Mbits/sec 217 671 KBytes [ 4 ] 4 .00-5.00 sec 111 MBytes 935 Mbits/sec 175 772 KBytes [ 4 ] 5 .00-6.00 sec 100 MBytes 840 Mbits/sec 658 598 KBytes [ 4 ] 6 .00-7.00 sec 106 MBytes 890 Mbits/sec 742 668 KBytes [ 4 ] 7 .00-8.00 sec 102 MBytes 857 Mbits/sec 764 724 KBytes [ 4 ] 8 .00-9.00 sec 97 .4 MBytes 817 Mbits/sec 1175 764 KBytes [ 4 ] 9 .00-10.00 sec 111 MBytes 934 Mbits/sec 1083 838 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 1010 MBytes 848 Mbits/sec 5119 sender [ 4 ] 0 .00-10.00 sec 1008 MBytes 846 Mbits/sec receiver iperf Done. \u4fee\u6539\u7b2c\u4e00\u4e2a Pod \u7684\u5165\u53e3\u5e26\u5bbd QoS\uff1a kubectl annotate --overwrite pod perf-4n4gt -n ls1 ovn.kubernetes.io/ingress_rate = 30 \u518d\u6b21\u4ece\u7b2c\u4e8c\u4e2a Pod \u6d4b\u8bd5\u7b2c\u4e00\u4e2a Pod \u5e26\u5bbd\uff1a # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 52372 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 3 .66 MBytes 30 .7 Mbits/sec 2 76 .1 KBytes [ 4 ] 1 .00-2.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 104 KBytes [ 4 ] 2 .00-3.00 sec 3 .50 MBytes 29 .4 Mbits/sec 0 126 KBytes [ 4 ] 3 .00-4.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 144 KBytes [ 4 ] 4 .00-5.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 160 KBytes [ 4 ] 5 .00-6.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 175 KBytes [ 4 ] 6 .00-7.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 212 KBytes [ 4 ] 7 .00-8.00 sec 3 .68 MBytes 30 .9 Mbits/sec 0 294 KBytes [ 4 ] 8 .00-9.00 sec 3 .74 MBytes 31 .4 Mbits/sec 0 398 KBytes [ 4 ] 9 .00-10.00 sec 3 .80 MBytes 31 .9 Mbits/sec 0 526 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 35 .7 MBytes 29 .9 Mbits/sec 2 sender [ 4 ] 0 .00-10.00 sec 34 .5 MBytes 29 .0 Mbits/sec receiver iperf Done. linux-netem QoS \u00b6 Pod \u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b annotation \u914d\u7f6e linux-netem \u7c7b\u578b QoS\uff1a ovn.kubernetes.io/latency \u3001 ovn.kubernetes.io/limit \u548c ovn.kubernetes.io/loss \u3002 ovn.kubernetes.io/latency \uff1a\u8bbe\u7f6e Pod \u6d41\u91cf\u5ef6\u8fdf\uff0c\u53d6\u503c\u4e3a\u6574\u6570\uff0c\u5355\u4f4d\u4e3a ms\u3002 ovn.kubernetes.io/limit \uff1a \u4e3a qdisc \u961f\u5217\u53ef\u5bb9\u7eb3\u7684\u6700\u5927\u6570\u636e\u5305\u6570\uff0c\u53d6\u503c\u4e3a\u6574\u5f62\u6570\u503c\uff0c\u4f8b\u5982 1000\u3002 ovn.kubernetes.io/loss \uff1a \u4e3a\u8bbe\u7f6e\u7684\u62a5\u6587\u4e22\u5305\u6982\u7387\uff0c\u53d6\u503c\u4e3a float \u7c7b\u578b\uff0c\u4f8b\u5982\u53d6\u503c\u4e3a 20\uff0c\u5219\u4e3a\u8bbe\u7f6e 20% \u7684\u4e22\u5305\u6982\u7387\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5bb9\u5668\u7f51\u7edc QoS \u914d\u7f6e"},{"location":"guide/qos/#qos","text":"Kube-OVN \u652f\u6301\u57fa\u4e8e\u5355\u4e2a Pod \u7684\u4e24\u79cd\u4e0d\u540c\u7c7b\u578b\u7684 QoS\uff1a \u6700\u5927\u5e26\u5bbd\u9650\u5236 QoS\u3002 linux-netem \uff0c\u6a21\u62df\u8bbe\u5907\u5e72\u6270\u4e22\u5305\u7b49\u7684 QoS\uff0c\u53ef\u7528\u4e8e\u6a21\u62df\u6d4b\u8bd5\u3002 \u76ee\u524d\u53ea\u652f\u6301 Pod \u7ea7\u522b QoS \u4e0d\u652f\u6301 Namespace \u6216 Subnet \u7ea7\u522b\u7684 QoS \u9650\u5236\u3002","title":"\u5bb9\u5668\u7f51\u7edc QoS \u914d\u7f6e"},{"location":"guide/qos/#qos_1","text":"\u8be5\u7c7b\u578b\u7684 QoS \u53ef\u4ee5\u901a\u8fc7 Pod annotation \u52a8\u6001\u8fdb\u884c\u914d\u7f6e\uff0c\u53ef\u4ee5\u5728\u4e0d\u4e2d\u65ad Pod \u8fd0\u884c\u7684\u60c5\u51b5\u4e0b\u8fdb\u884c\u8c03\u6574\u3002 \u5e26\u5bbd\u9650\u901f\u7684\u5355\u4f4d\u4e3a Mbit/s \u3002 apiVersion : v1 kind : Pod metadata : name : qos namespace : ls1 annotations : ovn.kubernetes.io/ingress_rate : \"3\" ovn.kubernetes.io/egress_rate : \"1\" spec : containers : - name : qos image : docker.io/library/nginx:alpine \u4f7f\u7528 annotation \u52a8\u6001\u8c03\u6574 QoS\uff1a kubectl annotate --overwrite pod nginx-74d5899f46-d7qkn ovn.kubernetes.io/ingress_rate = 3","title":"\u57fa\u4e8e\u6700\u5927\u5e26\u5bbd\u9650\u5236\u7684 QoS"},{"location":"guide/qos/#qos_2","text":"\u90e8\u7f72\u6027\u80fd\u6d4b\u8bd5\u9700\u8981\u7684\u5bb9\u5668\uff1a kind : DaemonSet apiVersion : apps/v1 metadata : name : perf namespace : ls1 labels : app : perf spec : selector : matchLabels : app : perf template : metadata : labels : app : perf spec : containers : - name : nginx image : docker.io/kubeovn/perf \u8fdb\u5165\u5176\u4e2d\u4e00\u4e2a Pod \u5e76\u5f00\u542f iperf3 server\uff1a # kubectl exec -it perf-4n4gt -n ls1 sh # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- \u8fdb\u5165\u53e6\u4e00\u4e2a Pod \u8bf7\u6c42\u4e4b\u524d\u7684 Pod\uff1a # kubectl exec -it perf-d4mqc -n ls1 sh # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 51544 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 86 .4 MBytes 725 Mbits/sec 3 350 KBytes [ 4 ] 1 .00-2.00 sec 89 .9 MBytes 754 Mbits/sec 118 473 KBytes [ 4 ] 2 .00-3.00 sec 101 MBytes 848 Mbits/sec 184 586 KBytes [ 4 ] 3 .00-4.00 sec 104 MBytes 875 Mbits/sec 217 671 KBytes [ 4 ] 4 .00-5.00 sec 111 MBytes 935 Mbits/sec 175 772 KBytes [ 4 ] 5 .00-6.00 sec 100 MBytes 840 Mbits/sec 658 598 KBytes [ 4 ] 6 .00-7.00 sec 106 MBytes 890 Mbits/sec 742 668 KBytes [ 4 ] 7 .00-8.00 sec 102 MBytes 857 Mbits/sec 764 724 KBytes [ 4 ] 8 .00-9.00 sec 97 .4 MBytes 817 Mbits/sec 1175 764 KBytes [ 4 ] 9 .00-10.00 sec 111 MBytes 934 Mbits/sec 1083 838 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 1010 MBytes 848 Mbits/sec 5119 sender [ 4 ] 0 .00-10.00 sec 1008 MBytes 846 Mbits/sec receiver iperf Done. \u4fee\u6539\u7b2c\u4e00\u4e2a Pod \u7684\u5165\u53e3\u5e26\u5bbd QoS\uff1a kubectl annotate --overwrite pod perf-4n4gt -n ls1 ovn.kubernetes.io/ingress_rate = 30 \u518d\u6b21\u4ece\u7b2c\u4e8c\u4e2a Pod \u6d4b\u8bd5\u7b2c\u4e00\u4e2a Pod \u5e26\u5bbd\uff1a # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 52372 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 3 .66 MBytes 30 .7 Mbits/sec 2 76 .1 KBytes [ 4 ] 1 .00-2.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 104 KBytes [ 4 ] 2 .00-3.00 sec 3 .50 MBytes 29 .4 Mbits/sec 0 126 KBytes [ 4 ] 3 .00-4.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 144 KBytes [ 4 ] 4 .00-5.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 160 KBytes [ 4 ] 5 .00-6.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 175 KBytes [ 4 ] 6 .00-7.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 212 KBytes [ 4 ] 7 .00-8.00 sec 3 .68 MBytes 30 .9 Mbits/sec 0 294 KBytes [ 4 ] 8 .00-9.00 sec 3 .74 MBytes 31 .4 Mbits/sec 0 398 KBytes [ 4 ] 9 .00-10.00 sec 3 .80 MBytes 31 .9 Mbits/sec 0 526 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 35 .7 MBytes 29 .9 Mbits/sec 2 sender [ 4 ] 0 .00-10.00 sec 34 .5 MBytes 29 .0 Mbits/sec receiver iperf Done.","title":"\u6d4b\u8bd5 QoS \u8c03\u6574"},{"location":"guide/qos/#linux-netem-qos","text":"Pod \u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b annotation \u914d\u7f6e linux-netem \u7c7b\u578b QoS\uff1a ovn.kubernetes.io/latency \u3001 ovn.kubernetes.io/limit \u548c ovn.kubernetes.io/loss \u3002 ovn.kubernetes.io/latency \uff1a\u8bbe\u7f6e Pod \u6d41\u91cf\u5ef6\u8fdf\uff0c\u53d6\u503c\u4e3a\u6574\u6570\uff0c\u5355\u4f4d\u4e3a ms\u3002 ovn.kubernetes.io/limit \uff1a \u4e3a qdisc \u961f\u5217\u53ef\u5bb9\u7eb3\u7684\u6700\u5927\u6570\u636e\u5305\u6570\uff0c\u53d6\u503c\u4e3a\u6574\u5f62\u6570\u503c\uff0c\u4f8b\u5982 1000\u3002 ovn.kubernetes.io/loss \uff1a \u4e3a\u8bbe\u7f6e\u7684\u62a5\u6587\u4e22\u5305\u6982\u7387\uff0c\u53d6\u503c\u4e3a float \u7c7b\u578b\uff0c\u4f8b\u5982\u53d6\u503c\u4e3a 20\uff0c\u5219\u4e3a\u8bbe\u7f6e 20% \u7684\u4e22\u5305\u6982\u7387\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"linux-netem QoS"},{"location":"guide/setup-options/","text":"\u5b89\u88c5\u548c\u914d\u7f6e\u9009\u9879 \u00b6 \u5728 \u4e00\u952e\u5b89\u88c5\u4e2d \u6211\u4eec\u4f7f\u7528\u9ed8\u8ba4\u914d\u7f6e\u8fdb\u884c\u5b89\u88c5\uff0cKube-OVN \u8fd8\u652f\u6301\u66f4\u591a \u81ea\u5b9a\u4e49\u914d\u7f6e\uff0c\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff0c\u6216\u8005\u4e4b\u540e\u66f4\u6539\u5404\u4e2a\u7ec4\u4ef6\u7684\u53c2\u6570\u6765\u8fdb\u884c\u914d\u7f6e\u3002\u672c\u6587\u6863\u5c06\u4f1a\u4ecb\u7ecd\u8fd9\u4e9b\u81ea\u5b9a\u4e49\u9009\u9879 \u7684\u4f5c\u7528\uff0c\u4ee5\u53ca\u5982\u4f55\u8fdb\u884c\u914d\u7f6e\u3002 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \u00b6 Kube-OVN \u5728\u5b89\u88c5\u65f6\u4f1a\u914d\u7f6e\u4e24\u4e2a\u5185\u7f6e\u5b50\u7f51\uff1a default \u5b50\u7f51\uff0c\u4f5c\u4e3a Pod \u5206\u914d IP \u4f7f\u7528\u7684\u9ed8\u8ba4\u5b50\u7f51\uff0c\u9ed8\u8ba4 CIDR \u4e3a 10.16.0.0/16 \uff0c\u7f51\u5173\u4e3a 10.16.0.1 \u3002 join \u5b50\u7f51\uff0c\u4f5c\u4e3a Node \u548c Pod \u4e4b\u95f4\u8fdb\u884c\u7f51\u7edc\u901a\u4fe1\u7684\u7279\u6b8a\u5b50\u7f51, \u9ed8\u8ba4 CIDR \u4e3a 100.64.0.0/16 \uff0c\u7f51\u5173\u4e3a 100.64.0.1 \u3002 \u5728\u5b89\u88c5\u65f6\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u5185\u7684\u914d\u7f6e\u8fdb\u884c\u66f4\u6539\uff1a POD_CIDR = \"10.16.0.0/16\" POD_GATEWAY = \"10.16.0.1\" JOIN_CIDR = \"100.64.0.0/16\" EXCLUDE_IPS = \"\" EXCLUDE_IP \u53ef\u8bbe\u7f6e POD_CIDR \u4e0d\u8fdb\u884c\u5206\u914d\u7684\u5730\u5740\u8303\u56f4\uff0c\u683c\u5f0f\u4e3a\uff1a 192.168.10.20..192.168.10.30 \u3002 \u9700\u8981\u6ce8\u610f Overlay \u60c5\u51b5\u4e0b\u8fd9\u4e24\u4e2a\u7f51\u7edc\u4e0d\u80fd\u548c\u5df2\u6709\u7684\u4e3b\u673a\u7f51\u7edc\u548c Service CIDR \u51b2\u7a81\u3002 \u5728\u5b89\u88c5\u540e\u53ef\u4ee5\u5bf9\u8fd9\u4e24\u4e2a\u7f51\u7edc\u7684\u5730\u5740\u8303\u56f4\u8fdb\u884c\u4fee\u6539\u8bf7\u53c2\u8003 \u4fee\u6539\u9ed8\u8ba4\u5b50\u7f51 \u548c \u4fee\u6539 Join \u5b50\u7f51 \u3002 Service \u7f51\u6bb5\u914d\u7f6e \u00b6 \u7531\u4e8e\u90e8\u5206 kube-proxy \u8bbe\u7f6e\u7684 iptables \u548c\u8def\u7531\u89c4\u5219\u4f1a\u548c Kube-OVN \u8bbe\u7f6e\u7684\u89c4\u5219\u4ea7\u751f\u4ea4\u96c6\uff0c\u56e0\u6b64 Kube-OVN \u9700\u8981\u77e5\u9053 Service \u7684 CIDR \u6765\u6b63\u786e\u8bbe\u7f6e\u5bf9\u5e94\u7684\u89c4\u5219\u3002 \u5728\u5b89\u88c5\u811a\u672c\u4e2d\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539\uff1a SVC_CIDR = \"10.96.0.0/12\" \u6765\u8fdb\u884c\u914d\u7f6e\u3002 \u4e5f\u53ef\u4ee5\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u4fee\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\uff1a args : - --service-cluster-ip-range=10.96.0.0/12 \u6765\u8fdb\u884c\u4fee\u6539\u3002 Overlay \u7f51\u5361\u9009\u62e9 \u00b6 \u5728\u8282\u70b9\u5b58\u5728\u591a\u5757\u7f51\u5361\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u9ed8\u8ba4\u4f1a\u9009\u62e9 Kubernetes Node IP \u5bf9\u5e94\u7684\u7f51\u5361\u4f5c\u4e3a\u5bb9\u5668\u95f4\u8de8\u8282\u70b9\u901a\u4fe1\u7684\u7f51\u5361\u5e76\u5efa\u7acb\u5bf9\u5e94\u7684\u96a7\u9053\u3002 \u5982\u679c\u9700\u8981\u9009\u62e9\u5176\u4ed6\u7684\u7f51\u5361\u5efa\u7acb\u5bb9\u5668\u96a7\u9053\uff0c\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u4fee\u6539\uff1a IFACE = eth1 \u8be5\u9009\u9879\u652f\u6301\u4ee5\u9017\u53f7\u6240\u5206\u9694\u6b63\u5219\u8868\u8fbe\u5f0f,\u4f8b\u5982 ens[a-z0-9]*,eth[a-z0-9]* \u3002 \u5b89\u88c5\u540e\u4e5f\u53ef\u901a\u8fc7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a args : - --iface=eth1 \u5982\u679c\u6bcf\u53f0\u673a\u5668\u7684\u7f51\u5361\u540d\u5747\u4e0d\u540c\uff0c\u4e14\u6ca1\u6709\u56fa\u5b9a\u89c4\u5f8b\uff0c\u53ef\u4ee5\u4f7f\u7528\u8282\u70b9 annotation ovn.kubernetes.io/tunnel_interface \u8fdb\u884c\u6bcf\u4e2a\u8282\u70b9\u7684\u9010\u4e00\u914d\u7f6e\uff0c\u62e5\u6709\u8be5 annotation \u8282\u70b9\u4f1a\u8986\u76d6 iface \u7684\u914d\u7f6e\uff0c\u4f18\u5148\u4f7f\u7528 annotation\u3002 kubectl annotate node no1 ovn.kubernetes.io/tunnel_interface = ethx MTU \u8bbe\u7f6e \u00b6 \u7531\u4e8e Overlay \u5c01\u88c5\u9700\u8981\u5360\u636e\u989d\u5916\u7684\u7a7a\u95f4\uff0cKube-OVN \u5728\u521b\u5efa\u5bb9\u5668\u7f51\u5361\u65f6\u4f1a\u6839\u636e\u9009\u62e9\u7f51\u5361\u7684 MTU \u8fdb\u884c\u5bb9\u5668\u7f51\u5361\u7684 MTU \u8c03\u6574\uff0c \u9ed8\u8ba4\u60c5\u51b5\u4e0b Overlay \u5b50\u7f51\u4e0b Pod \u7f51\u5361 MTU \u4e3a\u4e3b\u673a\u7f51\u5361 MTU - 100\uff0cUnderlay \u5b50\u7f51\u4e0b\uff0cPod \u7f51\u5361\u548c\u4e3b\u673a\u7f51\u5361\u6709\u76f8\u540c MTU\u3002 \u5982\u679c\u9700\u8981\u8c03\u6574 Overlay \u5b50\u7f51\u4e0b MTU \u7684\u5927\u5c0f\uff0c\u53ef\u4ee5\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\uff1a args : - --mtu=1333 \u5168\u5c40\u6d41\u91cf\u955c\u50cf\u5f00\u542f\u8bbe\u7f6e \u00b6 \u5728\u5f00\u542f\u5168\u5c40\u6d41\u91cf\u955c\u50cf\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u4f1a\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\u521b\u5efa\u4e00\u5757 mirror0 \u7684\u865a\u62df\u7f51\u5361\uff0c\u590d\u5236\u5f53\u524d\u673a\u5668\u6240\u6709\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u5230\u8be5\u7f51\u5361\u4e0a\uff0c \u7528\u6237\u53ef\u4ee5\u901a\u8fc7 tcpdump \u53ca\u5176\u4ed6\u5de5\u5177\u8fdb\u884c\u6d41\u91cf\u5206\u6790\uff0c\u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u901a\u8fc7\u4e0b\u9762\u7684\u914d\u7f6e\u5f00\u542f\uff1a ENABLE_MIRROR = true \u4e5f\u53ef\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\u65b9\u5f0f\u8fdb\u884c\u8c03\u6574: args : - --enable-mirror=true \u6d41\u91cf\u955c\u50cf\u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5173\u95ed\uff0c\u5982\u679c\u9700\u8981\u7ec6\u7c92\u5ea6\u7684\u6d41\u91cf\u955c\u50cf\u6216\u9700\u8981\u5c06\u6d41\u91cf\u955c\u50cf\u5230\u989d\u5916\u7684\u7f51\u5361\u8bf7\u53c2\u8003 \u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u955c\u50cf \u3002 LB \u5f00\u542f\u8bbe\u7f6e \u00b6 Kube-OVN \u4f7f\u7528 OVN \u4e2d\u7684 L2 LB \u6765\u5b9e\u73b0 Service \u8f6c\u53d1\uff0c\u5728 Overlay \u573a\u666f\u4e2d\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9\u4f7f\u7528 kube-proxy \u6765\u5b8c\u6210 Service \u6d41\u91cf\u8f6c\u53d1, \u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u5173\u95ed Kube-OVN \u7684 LB \u529f\u80fd\u4ee5\u8fbe\u5230\u63a7\u5236\u9762\u548c\u6570\u636e\u9762\u66f4\u597d\u7684\u6027\u80fd\u3002 \u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_LB = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-lb=false LB \u7684\u529f\u80fd\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableLb \uff0c\u5c06 Kube-OVN \u7684 LB \u529f\u80fd\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f LB \u529f\u80fd\u3002 kube-ovn-controller Deployment \u4e2d\u7684 enable-lb \u53c2\u6570\u4f5c\u4e3a\u5168\u5c40\u53c2\u6570\uff0c\u63a7\u5236\u662f\u5426\u521b\u5efa load-balancer \u8bb0\u5f55\uff0c\u5b50\u7f51\u4e2d\u65b0\u589e\u7684 enableLb \u53c2\u6570\u7528\u4e8e\u63a7\u5236\u5b50\u7f51\u662f\u5426\u5173\u8054 load-balancer \u8bb0\u5f55\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51 enableLb \u53c2\u6570\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002 NetworkPolicy \u5f00\u542f\u8bbe\u7f6e \u00b6 Kube-OVN \u4f7f\u7528 OVN \u4e2d\u7684 ACL \u6765\u5b9e\u73b0 NetworkPolicy\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9\u5173\u95ed NetworkPolicy \u529f\u80fd \u6216\u8005\u4f7f\u7528 Cilium Chain \u7684\u65b9\u5f0f\u5229\u7528 eBPF \u5b9e\u73b0 NetworkPolicy\uff0c \u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u5173\u95ed Kube-OVN \u7684 NetworkPolicy \u529f\u80fd\u4ee5\u8fbe\u5230\u63a7\u5236\u9762\u548c\u6570\u636e\u9762\u66f4\u597d\u7684\u6027\u80fd\u3002 \u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_NP = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-np=false NetworkPolicy \u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002 EIP \u548c SNAT \u5f00\u542f\u8bbe\u7f6e \u00b6 \u9ed8\u8ba4\u7f51\u7edc\u4e0b\u5982\u679c\u65e0\u9700\u4f7f\u7528 EIP \u548c SNAT \u7684\u80fd\u529b\uff0c\u53ef\u4ee5\u9009\u62e9\u5173\u95ed\u76f8\u5173\u529f\u80fd\uff0c\u4ee5\u51cf\u5c11 kube-ovn-controller \u5728\u521b\u5efa\u548c\u66f4\u65b0 \u7f51\u7edc\u65f6\u7684\u68c0\u67e5\u6d88\u8017\uff0c\u5728\u5927\u89c4\u6a21\u96c6\u7fa4\u73af\u5883\u4e0b\u53ef\u4ee5\u63d0\u5347\u5904\u7406\u901f\u5ea6\u3002 \u8be5\u529f\u80fd\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_EIP_SNAT = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-eip-snat=false EIP \u548c SNAT \u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002\u8be5\u529f\u80fd\u7684\u76f8\u5173\u4f7f\u7528\u548c\u5176\u4ed6\u53ef\u914d\u53c2\u6570\u8bf7\u53c2\u8003 EIP \u548c SNAT \u914d\u7f6e \u3002 Load Balancer \u7c7b\u578b Service \u652f\u6301\u5f00\u542f\u8bbe\u7f6e \u00b6 \u9ed8\u8ba4 VPC \u4e0b\u53ef\u901a\u8fc7\u5f00\u542f\u8be5\u9009\u9879\u6765\u652f\u6301 Load Balancer \u7c7b\u578b Service\u3002\u8be5\u529f\u80fd\u7684\u76f8\u5173\u4f7f\u7528\u548c\u5176\u4ed6\u53ef\u914d\u53c2\u6570\u8bf7\u53c2\u8003 LoadBalancer \u7c7b\u578b Service \u3002 \u8be5\u529f\u80fd\u9ed8\u8ba4\u5173\u95ed\uff0c\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_LB_SVC = true \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-lb-svc=true \u96c6\u4e2d\u5f0f\u7f51\u5173 ECMP \u5f00\u542f\u8bbe\u7f6e \u00b6 \u96c6\u4e2d\u5f0f\u7f51\u5173\u652f\u6301\u4e3b\u5907\u548c ECMP \u4e24\u79cd\u9ad8\u53ef\u7528\u6a21\u5f0f\uff0c\u5982\u679c\u9700\u8981\u542f\u7528 ECMP \u6a21\u5f0f\uff0c \u9700\u8981\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e: args : - --enable-ecmp=true \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableEcmp \uff0c\u5c06\u96c6\u4e2d\u5f0f\u5b50\u7f51 ECMP \u5f00\u5173\u63a7\u5236\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f ECMP \u6a21\u5f0f\u3002\u539f\u6709\u7684 kube-ovn-controller Deployment \u4e2d\u7684 enable-ecmp \u53c2\u6570\u4e0d\u518d\u4f7f\u7528\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51\u5f00\u5173\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002 \u96c6\u4e2d\u5f0f\u7f51\u5173\u9ed8\u8ba4\u5b89\u88c5\u4e0b\u4e3a\u4e3b\u5907\u6a21\u5f0f\uff0c\u66f4\u591a\u7f51\u5173\u76f8\u5173\u5185\u5bb9\u8bf7\u53c2\u8003 \u5b50\u7f51\u4f7f\u7528 \u3002 Kubevirt VM \u56fa\u5b9a\u5730\u5740\u5f00\u542f\u8bbe\u7f6e \u00b6 \u9488\u5bf9 Kubevirt \u521b\u5efa\u7684 VM \u5b9e\u4f8b\uff0c kube-ovn-controller \u53ef\u4ee5\u6309\u7167\u7c7b\u4f3c StatefulSet Pod \u7684\u65b9\u5f0f\u8fdb\u884c IP \u5730\u5740\u5206\u914d\u548c\u7ba1\u7406\u3002 \u4ee5\u8fbe\u5230 VM \u5b9e\u4f8b\u5728\u751f\u547d\u5468\u671f\u5185\u542f\u505c\uff0c\u5347\u7ea7\uff0c\u8fc1\u79fb\u7b49\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\u5730\u5740\u56fa\u5b9a\u4e0d\u53d8\uff0c\u66f4\u7b26\u865a\u62df\u5316\u5408\u7528\u6237\u7684\u5b9e\u9645\u4f7f\u7528\u4f53\u9a8c\u3002 \u8be5\u529f\u80fd\u5728 1.10.6 \u540e\u9ed8\u8ba4\u5f00\u542f\uff0c\u82e5\u8981\u5173\u95ed\u6b64\u529f\u80fd\uff0c\u9700\u8981\u5728 kube-ovn-controller Deployment \u7684\u542f\u52a8\u547d\u4ee4\u4e2d\u8bbe\u7f6e\u5982\u4e0b\u53c2\u6570\uff1a args : - --keep-vm-ip=false CNI \u914d\u7f6e\u76f8\u5173\u8bbe\u7f6e \u00b6 Kube-OVN \u9ed8\u8ba4\u4f1a\u5728 /opt/cni/bin \u76ee\u5f55\u4e0b\u5b89\u88c5 CNI \u6267\u884c\u6587\u4ef6\uff0c\u5728 /etc/cni/net.d \u76ee\u5f55\u4e0b\u5b89\u88c5 CNI \u914d\u7f6e\u6587\u4ef6 01-kube-ovn.conflist \u3002 \u5982\u679c\u9700\u8981\u66f4\u6539\u5b89\u88c5\u4f4d\u7f6e\u548c CNI \u914d\u7f6e\u6587\u4ef6\u7684\u4f18\u5148\u7ea7\uff0c\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u7684\u4e0b\u5217\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a CNI_CONF_DIR = \"/etc/cni/net.d\" CNI_BIN_DIR = \"/opt/cni/bin\" CNI_CONFIG_PRIORITY = \"01\" \u6216\u8005\u5728\u5b89\u88c5\u540e\u66f4\u6539 kube-ovn-cni DaemonSet \u7684 Volume \u6302\u8f7d\u548c\u542f\u52a8\u53c2\u6570\uff1a volumes : - name : cni-conf hostPath : path : \"/etc/cni/net.d\" - name : cni-bin hostPath : path:\"/opt/cni/bin\" ... args : - --cni-conf-name=01-kube-ovn.conflist \u96a7\u9053\u7c7b\u578b\u8bbe\u7f6e \u00b6 Kube-OVN \u9ed8\u8ba4 Overlay \u7684\u5c01\u88c5\u6a21\u5f0f\u4e3a Geneve\uff0c\u5982\u679c\u60f3\u66f4\u6362\u4e3a Vxlan \u6216 STT\uff0c\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u7684\u4e0b\u5217\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a TUNNEL_TYPE = \"vxlan\" \u6216\u8005\u5728\u5b89\u88c5\u540e\u66f4\u6539 ovs-ovn DaemonSet \u7684\u73af\u5883\u53d8\u91cf\uff1a env : - name : TUNNEL_TYPE value : \"vxlan\" \u5982\u679c\u9700\u8981\u4f7f\u7528 STT \u96a7\u9053\u9700\u8981\u989d\u5916\u7f16\u8bd1 ovs \u7684\u5185\u6838\u6a21\u5757\uff0c\u8bf7\u53c2\u8003 \u6027\u80fd\u8c03\u4f18 \u3002 \u4e0d\u540c\u534f\u8bae\u5728\u5b9e\u9645\u4f7f\u7528\u4e2d\u7684\u533a\u522b\u8bf7\u53c2\u8003 \u96a7\u9053\u534f\u8bae\u8bf4\u660e \u3002 SSL \u8bbe\u7f6e \u00b6 OVN DB \u7684 API \u63a5\u53e3\u652f\u6301 SSL \u52a0\u5bc6\u6765\u4fdd\u8bc1\u8fde\u63a5\u5b89\u5168\uff0c\u5982\u8981\u5f00\u542f\u53ef\u8c03\u6574\u5b89\u88c5\u811a\u672c\u4e2d\u7684\u5982\u4e0b\u53c2\u6570: ENABLE_SSL = true SSL \u529f\u80fd\u9ed8\u8ba4\u5b89\u88c5\u4e0b\u4e3a\u5173\u95ed\u6a21\u5f0f\u3002 \u7ed1\u5b9a\u672c\u5730 ip \u00b6 kube-ovn-controller/kube-ovn-cni/kube-ovn-monitor \u8fd9\u4e9b\u670d\u52a1\u652f\u6301\u7ed1\u5b9a\u672c\u5730 ip\uff0c\u8be5\u529f\u80fd\u8bbe\u8ba1\u539f\u56e0\u4e3b\u8981\u662f\u56e0\u4e3a\u67d0\u4e9b\u573a\u666f\u4e0b\u51fa\u4e8e\u5b89\u5168\u8003\u8651\u4e0d\u5141\u8bb8\u670d\u52a1\u7ed1\u5b9a 0.0.0.0 \uff08\u6bd4\u5982\u8be5\u670d\u52a1\u90e8\u7f72\u5728\u67d0\u4e2a\u5bf9\u5916\u7f51\u5173\u4e0a\uff0c\u5916\u90e8\u7528\u6237\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7\u516c\u7f51 ip \u5e76\u6307\u5b9a\u7aef\u53e3\u53bb\u8bbf\u95ee\u5230\u8be5\u670d\u52a1\uff09\uff0c\u8be5\u529f\u80fd\u9ed8\u8ba4\u662f\u6253\u5f00\u7684\uff0c\u7531\u5b89\u88c5\u811a\u672c\u4e2d\u5982\u4e0b\u53c2\u6570\u63a7\u5236\uff1a ENABLE_BIND_LOCAL_IP = true \u4ee5 kube-ovn-monitor \u4e3a\u4f8b\uff0c\u5f00\u542f\u529f\u80fd\u540e\u4f1a\u628a\u670d\u52a1\u7ed1\u5b9a\u672c\u5730\u7684 pod ip \u5982\u4e0b\uff1a # netstat -tunlp |grep kube-ovn tcp 0 0 172 .18.0.5:10661 0 .0.0.0:* LISTEN 2612 /./kube-ovn-mon \u5b89\u88c5\u540e\u4e5f\u53ef\u901a\u8fc7\u4fee\u6539\u670d\u52a1\u7684 deployment \u6216\u8005 daemonSet \u7684\u73af\u5883\u53d8\u91cf\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a env : - name : ENABLE_BIND_LOCAL_IP value : \"false\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b89\u88c5\u548c\u914d\u7f6e\u9009\u9879"},{"location":"guide/setup-options/#_1","text":"\u5728 \u4e00\u952e\u5b89\u88c5\u4e2d \u6211\u4eec\u4f7f\u7528\u9ed8\u8ba4\u914d\u7f6e\u8fdb\u884c\u5b89\u88c5\uff0cKube-OVN \u8fd8\u652f\u6301\u66f4\u591a \u81ea\u5b9a\u4e49\u914d\u7f6e\uff0c\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff0c\u6216\u8005\u4e4b\u540e\u66f4\u6539\u5404\u4e2a\u7ec4\u4ef6\u7684\u53c2\u6570\u6765\u8fdb\u884c\u914d\u7f6e\u3002\u672c\u6587\u6863\u5c06\u4f1a\u4ecb\u7ecd\u8fd9\u4e9b\u81ea\u5b9a\u4e49\u9009\u9879 \u7684\u4f5c\u7528\uff0c\u4ee5\u53ca\u5982\u4f55\u8fdb\u884c\u914d\u7f6e\u3002","title":"\u5b89\u88c5\u548c\u914d\u7f6e\u9009\u9879"},{"location":"guide/setup-options/#_2","text":"Kube-OVN \u5728\u5b89\u88c5\u65f6\u4f1a\u914d\u7f6e\u4e24\u4e2a\u5185\u7f6e\u5b50\u7f51\uff1a default \u5b50\u7f51\uff0c\u4f5c\u4e3a Pod \u5206\u914d IP \u4f7f\u7528\u7684\u9ed8\u8ba4\u5b50\u7f51\uff0c\u9ed8\u8ba4 CIDR \u4e3a 10.16.0.0/16 \uff0c\u7f51\u5173\u4e3a 10.16.0.1 \u3002 join \u5b50\u7f51\uff0c\u4f5c\u4e3a Node \u548c Pod \u4e4b\u95f4\u8fdb\u884c\u7f51\u7edc\u901a\u4fe1\u7684\u7279\u6b8a\u5b50\u7f51, \u9ed8\u8ba4 CIDR \u4e3a 100.64.0.0/16 \uff0c\u7f51\u5173\u4e3a 100.64.0.1 \u3002 \u5728\u5b89\u88c5\u65f6\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u5185\u7684\u914d\u7f6e\u8fdb\u884c\u66f4\u6539\uff1a POD_CIDR = \"10.16.0.0/16\" POD_GATEWAY = \"10.16.0.1\" JOIN_CIDR = \"100.64.0.0/16\" EXCLUDE_IPS = \"\" EXCLUDE_IP \u53ef\u8bbe\u7f6e POD_CIDR \u4e0d\u8fdb\u884c\u5206\u914d\u7684\u5730\u5740\u8303\u56f4\uff0c\u683c\u5f0f\u4e3a\uff1a 192.168.10.20..192.168.10.30 \u3002 \u9700\u8981\u6ce8\u610f Overlay \u60c5\u51b5\u4e0b\u8fd9\u4e24\u4e2a\u7f51\u7edc\u4e0d\u80fd\u548c\u5df2\u6709\u7684\u4e3b\u673a\u7f51\u7edc\u548c Service CIDR \u51b2\u7a81\u3002 \u5728\u5b89\u88c5\u540e\u53ef\u4ee5\u5bf9\u8fd9\u4e24\u4e2a\u7f51\u7edc\u7684\u5730\u5740\u8303\u56f4\u8fdb\u884c\u4fee\u6539\u8bf7\u53c2\u8003 \u4fee\u6539\u9ed8\u8ba4\u5b50\u7f51 \u548c \u4fee\u6539 Join \u5b50\u7f51 \u3002","title":"\u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e"},{"location":"guide/setup-options/#service","text":"\u7531\u4e8e\u90e8\u5206 kube-proxy \u8bbe\u7f6e\u7684 iptables \u548c\u8def\u7531\u89c4\u5219\u4f1a\u548c Kube-OVN \u8bbe\u7f6e\u7684\u89c4\u5219\u4ea7\u751f\u4ea4\u96c6\uff0c\u56e0\u6b64 Kube-OVN \u9700\u8981\u77e5\u9053 Service \u7684 CIDR \u6765\u6b63\u786e\u8bbe\u7f6e\u5bf9\u5e94\u7684\u89c4\u5219\u3002 \u5728\u5b89\u88c5\u811a\u672c\u4e2d\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539\uff1a SVC_CIDR = \"10.96.0.0/12\" \u6765\u8fdb\u884c\u914d\u7f6e\u3002 \u4e5f\u53ef\u4ee5\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u4fee\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\uff1a args : - --service-cluster-ip-range=10.96.0.0/12 \u6765\u8fdb\u884c\u4fee\u6539\u3002","title":"Service \u7f51\u6bb5\u914d\u7f6e"},{"location":"guide/setup-options/#overlay","text":"\u5728\u8282\u70b9\u5b58\u5728\u591a\u5757\u7f51\u5361\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u9ed8\u8ba4\u4f1a\u9009\u62e9 Kubernetes Node IP \u5bf9\u5e94\u7684\u7f51\u5361\u4f5c\u4e3a\u5bb9\u5668\u95f4\u8de8\u8282\u70b9\u901a\u4fe1\u7684\u7f51\u5361\u5e76\u5efa\u7acb\u5bf9\u5e94\u7684\u96a7\u9053\u3002 \u5982\u679c\u9700\u8981\u9009\u62e9\u5176\u4ed6\u7684\u7f51\u5361\u5efa\u7acb\u5bb9\u5668\u96a7\u9053\uff0c\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u4fee\u6539\uff1a IFACE = eth1 \u8be5\u9009\u9879\u652f\u6301\u4ee5\u9017\u53f7\u6240\u5206\u9694\u6b63\u5219\u8868\u8fbe\u5f0f,\u4f8b\u5982 ens[a-z0-9]*,eth[a-z0-9]* \u3002 \u5b89\u88c5\u540e\u4e5f\u53ef\u901a\u8fc7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a args : - --iface=eth1 \u5982\u679c\u6bcf\u53f0\u673a\u5668\u7684\u7f51\u5361\u540d\u5747\u4e0d\u540c\uff0c\u4e14\u6ca1\u6709\u56fa\u5b9a\u89c4\u5f8b\uff0c\u53ef\u4ee5\u4f7f\u7528\u8282\u70b9 annotation ovn.kubernetes.io/tunnel_interface \u8fdb\u884c\u6bcf\u4e2a\u8282\u70b9\u7684\u9010\u4e00\u914d\u7f6e\uff0c\u62e5\u6709\u8be5 annotation \u8282\u70b9\u4f1a\u8986\u76d6 iface \u7684\u914d\u7f6e\uff0c\u4f18\u5148\u4f7f\u7528 annotation\u3002 kubectl annotate node no1 ovn.kubernetes.io/tunnel_interface = ethx","title":"Overlay \u7f51\u5361\u9009\u62e9"},{"location":"guide/setup-options/#mtu","text":"\u7531\u4e8e Overlay \u5c01\u88c5\u9700\u8981\u5360\u636e\u989d\u5916\u7684\u7a7a\u95f4\uff0cKube-OVN \u5728\u521b\u5efa\u5bb9\u5668\u7f51\u5361\u65f6\u4f1a\u6839\u636e\u9009\u62e9\u7f51\u5361\u7684 MTU \u8fdb\u884c\u5bb9\u5668\u7f51\u5361\u7684 MTU \u8c03\u6574\uff0c \u9ed8\u8ba4\u60c5\u51b5\u4e0b Overlay \u5b50\u7f51\u4e0b Pod \u7f51\u5361 MTU \u4e3a\u4e3b\u673a\u7f51\u5361 MTU - 100\uff0cUnderlay \u5b50\u7f51\u4e0b\uff0cPod \u7f51\u5361\u548c\u4e3b\u673a\u7f51\u5361\u6709\u76f8\u540c MTU\u3002 \u5982\u679c\u9700\u8981\u8c03\u6574 Overlay \u5b50\u7f51\u4e0b MTU \u7684\u5927\u5c0f\uff0c\u53ef\u4ee5\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\uff1a args : - --mtu=1333","title":"MTU \u8bbe\u7f6e"},{"location":"guide/setup-options/#_3","text":"\u5728\u5f00\u542f\u5168\u5c40\u6d41\u91cf\u955c\u50cf\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u4f1a\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\u521b\u5efa\u4e00\u5757 mirror0 \u7684\u865a\u62df\u7f51\u5361\uff0c\u590d\u5236\u5f53\u524d\u673a\u5668\u6240\u6709\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u5230\u8be5\u7f51\u5361\u4e0a\uff0c \u7528\u6237\u53ef\u4ee5\u901a\u8fc7 tcpdump \u53ca\u5176\u4ed6\u5de5\u5177\u8fdb\u884c\u6d41\u91cf\u5206\u6790\uff0c\u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u901a\u8fc7\u4e0b\u9762\u7684\u914d\u7f6e\u5f00\u542f\uff1a ENABLE_MIRROR = true \u4e5f\u53ef\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\u65b9\u5f0f\u8fdb\u884c\u8c03\u6574: args : - --enable-mirror=true \u6d41\u91cf\u955c\u50cf\u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5173\u95ed\uff0c\u5982\u679c\u9700\u8981\u7ec6\u7c92\u5ea6\u7684\u6d41\u91cf\u955c\u50cf\u6216\u9700\u8981\u5c06\u6d41\u91cf\u955c\u50cf\u5230\u989d\u5916\u7684\u7f51\u5361\u8bf7\u53c2\u8003 \u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u955c\u50cf \u3002","title":"\u5168\u5c40\u6d41\u91cf\u955c\u50cf\u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#lb","text":"Kube-OVN \u4f7f\u7528 OVN \u4e2d\u7684 L2 LB \u6765\u5b9e\u73b0 Service \u8f6c\u53d1\uff0c\u5728 Overlay \u573a\u666f\u4e2d\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9\u4f7f\u7528 kube-proxy \u6765\u5b8c\u6210 Service \u6d41\u91cf\u8f6c\u53d1, \u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u5173\u95ed Kube-OVN \u7684 LB \u529f\u80fd\u4ee5\u8fbe\u5230\u63a7\u5236\u9762\u548c\u6570\u636e\u9762\u66f4\u597d\u7684\u6027\u80fd\u3002 \u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_LB = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-lb=false LB \u7684\u529f\u80fd\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableLb \uff0c\u5c06 Kube-OVN \u7684 LB \u529f\u80fd\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f LB \u529f\u80fd\u3002 kube-ovn-controller Deployment \u4e2d\u7684 enable-lb \u53c2\u6570\u4f5c\u4e3a\u5168\u5c40\u53c2\u6570\uff0c\u63a7\u5236\u662f\u5426\u521b\u5efa load-balancer \u8bb0\u5f55\uff0c\u5b50\u7f51\u4e2d\u65b0\u589e\u7684 enableLb \u53c2\u6570\u7528\u4e8e\u63a7\u5236\u5b50\u7f51\u662f\u5426\u5173\u8054 load-balancer \u8bb0\u5f55\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51 enableLb \u53c2\u6570\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002","title":"LB \u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#networkpolicy","text":"Kube-OVN \u4f7f\u7528 OVN \u4e2d\u7684 ACL \u6765\u5b9e\u73b0 NetworkPolicy\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9\u5173\u95ed NetworkPolicy \u529f\u80fd \u6216\u8005\u4f7f\u7528 Cilium Chain \u7684\u65b9\u5f0f\u5229\u7528 eBPF \u5b9e\u73b0 NetworkPolicy\uff0c \u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u5173\u95ed Kube-OVN \u7684 NetworkPolicy \u529f\u80fd\u4ee5\u8fbe\u5230\u63a7\u5236\u9762\u548c\u6570\u636e\u9762\u66f4\u597d\u7684\u6027\u80fd\u3002 \u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_NP = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-np=false NetworkPolicy \u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002","title":"NetworkPolicy \u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#eip-snat","text":"\u9ed8\u8ba4\u7f51\u7edc\u4e0b\u5982\u679c\u65e0\u9700\u4f7f\u7528 EIP \u548c SNAT \u7684\u80fd\u529b\uff0c\u53ef\u4ee5\u9009\u62e9\u5173\u95ed\u76f8\u5173\u529f\u80fd\uff0c\u4ee5\u51cf\u5c11 kube-ovn-controller \u5728\u521b\u5efa\u548c\u66f4\u65b0 \u7f51\u7edc\u65f6\u7684\u68c0\u67e5\u6d88\u8017\uff0c\u5728\u5927\u89c4\u6a21\u96c6\u7fa4\u73af\u5883\u4e0b\u53ef\u4ee5\u63d0\u5347\u5904\u7406\u901f\u5ea6\u3002 \u8be5\u529f\u80fd\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_EIP_SNAT = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-eip-snat=false EIP \u548c SNAT \u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002\u8be5\u529f\u80fd\u7684\u76f8\u5173\u4f7f\u7528\u548c\u5176\u4ed6\u53ef\u914d\u53c2\u6570\u8bf7\u53c2\u8003 EIP \u548c SNAT \u914d\u7f6e \u3002","title":"EIP \u548c SNAT \u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#load-balancer-service","text":"\u9ed8\u8ba4 VPC \u4e0b\u53ef\u901a\u8fc7\u5f00\u542f\u8be5\u9009\u9879\u6765\u652f\u6301 Load Balancer \u7c7b\u578b Service\u3002\u8be5\u529f\u80fd\u7684\u76f8\u5173\u4f7f\u7528\u548c\u5176\u4ed6\u53ef\u914d\u53c2\u6570\u8bf7\u53c2\u8003 LoadBalancer \u7c7b\u578b Service \u3002 \u8be5\u529f\u80fd\u9ed8\u8ba4\u5173\u95ed\uff0c\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_LB_SVC = true \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-lb-svc=true","title":"Load Balancer \u7c7b\u578b Service \u652f\u6301\u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#ecmp","text":"\u96c6\u4e2d\u5f0f\u7f51\u5173\u652f\u6301\u4e3b\u5907\u548c ECMP \u4e24\u79cd\u9ad8\u53ef\u7528\u6a21\u5f0f\uff0c\u5982\u679c\u9700\u8981\u542f\u7528 ECMP \u6a21\u5f0f\uff0c \u9700\u8981\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e: args : - --enable-ecmp=true \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableEcmp \uff0c\u5c06\u96c6\u4e2d\u5f0f\u5b50\u7f51 ECMP \u5f00\u5173\u63a7\u5236\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f ECMP \u6a21\u5f0f\u3002\u539f\u6709\u7684 kube-ovn-controller Deployment \u4e2d\u7684 enable-ecmp \u53c2\u6570\u4e0d\u518d\u4f7f\u7528\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51\u5f00\u5173\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002 \u96c6\u4e2d\u5f0f\u7f51\u5173\u9ed8\u8ba4\u5b89\u88c5\u4e0b\u4e3a\u4e3b\u5907\u6a21\u5f0f\uff0c\u66f4\u591a\u7f51\u5173\u76f8\u5173\u5185\u5bb9\u8bf7\u53c2\u8003 \u5b50\u7f51\u4f7f\u7528 \u3002","title":"\u96c6\u4e2d\u5f0f\u7f51\u5173 ECMP \u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#kubevirt-vm","text":"\u9488\u5bf9 Kubevirt \u521b\u5efa\u7684 VM \u5b9e\u4f8b\uff0c kube-ovn-controller \u53ef\u4ee5\u6309\u7167\u7c7b\u4f3c StatefulSet Pod \u7684\u65b9\u5f0f\u8fdb\u884c IP \u5730\u5740\u5206\u914d\u548c\u7ba1\u7406\u3002 \u4ee5\u8fbe\u5230 VM \u5b9e\u4f8b\u5728\u751f\u547d\u5468\u671f\u5185\u542f\u505c\uff0c\u5347\u7ea7\uff0c\u8fc1\u79fb\u7b49\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\u5730\u5740\u56fa\u5b9a\u4e0d\u53d8\uff0c\u66f4\u7b26\u865a\u62df\u5316\u5408\u7528\u6237\u7684\u5b9e\u9645\u4f7f\u7528\u4f53\u9a8c\u3002 \u8be5\u529f\u80fd\u5728 1.10.6 \u540e\u9ed8\u8ba4\u5f00\u542f\uff0c\u82e5\u8981\u5173\u95ed\u6b64\u529f\u80fd\uff0c\u9700\u8981\u5728 kube-ovn-controller Deployment \u7684\u542f\u52a8\u547d\u4ee4\u4e2d\u8bbe\u7f6e\u5982\u4e0b\u53c2\u6570\uff1a args : - --keep-vm-ip=false","title":"Kubevirt VM \u56fa\u5b9a\u5730\u5740\u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#cni","text":"Kube-OVN \u9ed8\u8ba4\u4f1a\u5728 /opt/cni/bin \u76ee\u5f55\u4e0b\u5b89\u88c5 CNI \u6267\u884c\u6587\u4ef6\uff0c\u5728 /etc/cni/net.d \u76ee\u5f55\u4e0b\u5b89\u88c5 CNI \u914d\u7f6e\u6587\u4ef6 01-kube-ovn.conflist \u3002 \u5982\u679c\u9700\u8981\u66f4\u6539\u5b89\u88c5\u4f4d\u7f6e\u548c CNI \u914d\u7f6e\u6587\u4ef6\u7684\u4f18\u5148\u7ea7\uff0c\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u7684\u4e0b\u5217\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a CNI_CONF_DIR = \"/etc/cni/net.d\" CNI_BIN_DIR = \"/opt/cni/bin\" CNI_CONFIG_PRIORITY = \"01\" \u6216\u8005\u5728\u5b89\u88c5\u540e\u66f4\u6539 kube-ovn-cni DaemonSet \u7684 Volume \u6302\u8f7d\u548c\u542f\u52a8\u53c2\u6570\uff1a volumes : - name : cni-conf hostPath : path : \"/etc/cni/net.d\" - name : cni-bin hostPath : path:\"/opt/cni/bin\" ... args : - --cni-conf-name=01-kube-ovn.conflist","title":"CNI \u914d\u7f6e\u76f8\u5173\u8bbe\u7f6e"},{"location":"guide/setup-options/#_4","text":"Kube-OVN \u9ed8\u8ba4 Overlay \u7684\u5c01\u88c5\u6a21\u5f0f\u4e3a Geneve\uff0c\u5982\u679c\u60f3\u66f4\u6362\u4e3a Vxlan \u6216 STT\uff0c\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u7684\u4e0b\u5217\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a TUNNEL_TYPE = \"vxlan\" \u6216\u8005\u5728\u5b89\u88c5\u540e\u66f4\u6539 ovs-ovn DaemonSet \u7684\u73af\u5883\u53d8\u91cf\uff1a env : - name : TUNNEL_TYPE value : \"vxlan\" \u5982\u679c\u9700\u8981\u4f7f\u7528 STT \u96a7\u9053\u9700\u8981\u989d\u5916\u7f16\u8bd1 ovs \u7684\u5185\u6838\u6a21\u5757\uff0c\u8bf7\u53c2\u8003 \u6027\u80fd\u8c03\u4f18 \u3002 \u4e0d\u540c\u534f\u8bae\u5728\u5b9e\u9645\u4f7f\u7528\u4e2d\u7684\u533a\u522b\u8bf7\u53c2\u8003 \u96a7\u9053\u534f\u8bae\u8bf4\u660e \u3002","title":"\u96a7\u9053\u7c7b\u578b\u8bbe\u7f6e"},{"location":"guide/setup-options/#ssl","text":"OVN DB \u7684 API \u63a5\u53e3\u652f\u6301 SSL \u52a0\u5bc6\u6765\u4fdd\u8bc1\u8fde\u63a5\u5b89\u5168\uff0c\u5982\u8981\u5f00\u542f\u53ef\u8c03\u6574\u5b89\u88c5\u811a\u672c\u4e2d\u7684\u5982\u4e0b\u53c2\u6570: ENABLE_SSL = true SSL \u529f\u80fd\u9ed8\u8ba4\u5b89\u88c5\u4e0b\u4e3a\u5173\u95ed\u6a21\u5f0f\u3002","title":"SSL \u8bbe\u7f6e"},{"location":"guide/setup-options/#ip","text":"kube-ovn-controller/kube-ovn-cni/kube-ovn-monitor \u8fd9\u4e9b\u670d\u52a1\u652f\u6301\u7ed1\u5b9a\u672c\u5730 ip\uff0c\u8be5\u529f\u80fd\u8bbe\u8ba1\u539f\u56e0\u4e3b\u8981\u662f\u56e0\u4e3a\u67d0\u4e9b\u573a\u666f\u4e0b\u51fa\u4e8e\u5b89\u5168\u8003\u8651\u4e0d\u5141\u8bb8\u670d\u52a1\u7ed1\u5b9a 0.0.0.0 \uff08\u6bd4\u5982\u8be5\u670d\u52a1\u90e8\u7f72\u5728\u67d0\u4e2a\u5bf9\u5916\u7f51\u5173\u4e0a\uff0c\u5916\u90e8\u7528\u6237\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7\u516c\u7f51 ip \u5e76\u6307\u5b9a\u7aef\u53e3\u53bb\u8bbf\u95ee\u5230\u8be5\u670d\u52a1\uff09\uff0c\u8be5\u529f\u80fd\u9ed8\u8ba4\u662f\u6253\u5f00\u7684\uff0c\u7531\u5b89\u88c5\u811a\u672c\u4e2d\u5982\u4e0b\u53c2\u6570\u63a7\u5236\uff1a ENABLE_BIND_LOCAL_IP = true \u4ee5 kube-ovn-monitor \u4e3a\u4f8b\uff0c\u5f00\u542f\u529f\u80fd\u540e\u4f1a\u628a\u670d\u52a1\u7ed1\u5b9a\u672c\u5730\u7684 pod ip \u5982\u4e0b\uff1a # netstat -tunlp |grep kube-ovn tcp 0 0 172 .18.0.5:10661 0 .0.0.0:* LISTEN 2612 /./kube-ovn-mon \u5b89\u88c5\u540e\u4e5f\u53ef\u901a\u8fc7\u4fee\u6539\u670d\u52a1\u7684 deployment \u6216\u8005 daemonSet \u7684\u73af\u5883\u53d8\u91cf\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a env : - name : ENABLE_BIND_LOCAL_IP value : \"false\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7ed1\u5b9a\u672c\u5730 ip"},{"location":"guide/static-ip-mac/","text":"\u56fa\u5b9a\u5730\u5740 \u00b6 Kube-OVN \u9ed8\u8ba4\u4f1a\u6839\u636e Pod \u6240\u5728 Namespace \u6240\u5c5e\u7684\u5b50\u7f51\u4e2d\u968f\u673a\u5206\u914d IP \u548c Mac\u3002 \u9488\u5bf9\u5de5\u4f5c\u8d1f\u8f7d\u9700\u8981\u56fa\u5b9a\u5730\u5740\u7684\u60c5\u51b5\uff0cKube-OVN \u6839\u636e\u4e0d\u540c\u7684\u573a\u666f\uff0c\u63d0\u4f9b\u4e86\u591a\u79cd\u56fa\u5b9a\u5730\u5740\u7684\u65b9\u6cd5\uff1a \u5355\u4e2a Pod \u56fa\u5b9a IP/Mac\u3002 Workload \u901a\u7528 IP Pool \u65b9\u5f0f\u6307\u5b9a\u56fa\u5b9a\u5730\u5740\u8303\u56f4\u3002 StatefulSet \u56fa\u5b9a\u5730\u5740\u3002 KubeVirt VM \u56fa\u5b9a\u5730\u5740\u3002 \u5355\u4e2a Pod \u56fa\u5b9a IP \u548c Mac \u00b6 \u53ef\u4ee5\u5728\u521b\u5efa Pod \u65f6\u901a\u8fc7 annotation \u6765\u6307\u5b9a Pod \u8fd0\u884c\u65f6\u6240\u9700\u7684 IP/Mac, kube-ovn-controller \u8fd0\u884c\u65f6\u5c06\u4f1a\u8df3\u8fc7\u5730\u5740\u968f\u673a\u5206\u914d\u9636\u6bb5\uff0c\u7ecf\u8fc7\u51b2\u7a81\u68c0\u6d4b\u540e\u76f4\u63a5\u4f7f\u7528\u6307\u5b9a\u5730\u5740\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u9017\u53f7\u5206\u9694 10.16.0.15,fd00:10:16::15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 spec : containers : - name : static-ip image : docker.io/library/nginx:alpine \u5728\u4f7f\u7528 annotation \u5b9a\u4e49\u5355\u4e2a Pod IP/Mac \u65f6\u9700\u8981\u6ce8\u610f\u4ee5\u4e0b\u51e0\u70b9\uff1a \u6240\u4f7f\u7528\u7684 IP/Mac \u4e0d\u80fd\u548c\u5df2\u6709\u7684 IP/Mac \u51b2\u7a81\u3002 IP \u5fc5\u987b\u5728\u6240\u5c5e\u5b50\u7f51\u7684 CIDR \u5185\u3002 \u53ef\u4ee5\u53ea\u6307\u5b9a IP \u6216 Mac\uff0c\u53ea\u6307\u5b9a\u4e00\u4e2a\u65f6\uff0c\u53e6\u4e00\u4e2a\u4f1a\u968f\u673a\u5206\u914d\u3002 Workload \u901a\u7528 IP Pool \u56fa\u5b9a\u5730\u5740 \u00b6 Kube-OVN \u652f\u6301\u901a\u8fc7 annotation ovn.kubernetes.io/ip_pool \u7ed9 Workload\uff08Deployment/StatefulSet/DaemonSet/Job/CronJob\uff09\u8bbe\u7f6e\u56fa\u5b9a IP\u3002 kube-ovn-controllerr \u4f1a\u81ea\u52a8\u9009\u62e9 ovn.kubernetes.io/ip_pool \u4e2d\u6307\u5b9a\u7684 IP \u5e76\u8fdb\u884c\u51b2\u7a81\u68c0\u6d4b\u3002 IP Pool \u7684 Annotation \u9700\u8981\u52a0\u5728 template \u5185\u7684 annotation \u5b57\u6bb5\uff0c\u9664\u4e86 Kubernetes \u5185\u7f6e\u7684 Workload \u7c7b\u578b\uff0c \u5176\u4ed6\u7528\u6237\u81ea\u5b9a\u4e49\u7684 Workload \u4e5f\u53ef\u4ee5\u4f7f\u7528\u540c\u6837\u7684\u65b9\u5f0f\u8fdb\u884c\u56fa\u5b9a\u5730\u5740\u5206\u914d\u3002 Deployment \u56fa\u5b9a IP \u793a\u4f8b \u00b6 apiVersion : apps/v1 kind : Deployment metadata : name : ippool labels : app : ippool spec : replicas : 2 selector : matchLabels : app : ippool template : metadata : labels : app : ippool annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u5206\u53f7\u8fdb\u884c\u5206\u9694 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::000F;10.16.0.17,fd00:10:16::0010 spec : containers : - name : ippool image : docker.io/library/nginx:alpine \u5bf9 Workload \u4f7f\u7528\u56fa\u5b9a IP \u9700\u8981\u6ce8\u610f\u4ee5\u4e0b\u51e0\u70b9\uff1a ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u5e94\u8be5\u5c5e\u4e8e\u6240\u5728\u5b50\u7f51\u7684 CIDR \u5185\u3002 ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u4e0d\u80fd\u548c\u5df2\u4f7f\u7528\u7684 IP \u51b2\u7a81\u3002 \u5f53 ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u6570\u91cf\u5c0f\u4e8e replicas \u6570\u91cf\u65f6\uff0c\u591a\u51fa\u7684 Pod \u5c06\u65e0\u6cd5\u521b\u5efa\u3002\u4f60\u9700\u8981\u6839\u636e Workload \u7684\u66f4\u65b0\u7b56\u7565\u4ee5\u53ca\u6269\u5bb9\u89c4\u5212\u8c03\u6574 ovn.kubernetes.io/ip_pool \u4e2d IP \u7684\u6570\u91cf\u3002 StatefulSet \u56fa\u5b9a\u5730\u5740 \u00b6 StatefulSet \u548c\u5176\u4ed6 Workload \u76f8\u540c\u53ef\u4ee5\u4f7f\u7528 ovn.kubernetes.io/ip_pool \u6765\u6307\u5b9a Pod \u4f7f\u7528\u7684 IP\u3002 \u7531\u4e8e StatefulSet \u591a\u7528\u4e8e\u6709\u72b6\u6001\u670d\u52a1\uff0c\u5bf9\u7f51\u7edc\u6807\u793a\u7684\u56fa\u5b9a\u6709\u66f4\u9ad8\u7684\u8981\u6c42\uff0cKube-OVN \u505a\u4e86\u7279\u6b8a\u7684\u5f3a\u5316\uff1a Pod \u4f1a\u6309\u987a\u5e8f\u5206\u914d ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP\u3002\u4f8b\u5982 StatefulSet \u7684\u540d\u5b57\u4e3a web\uff0c\u5219 web-0 \u4f1a\u4f7f\u7528 ovn.kubernetes.io/ip_pool \u4e2d\u7684\u7b2c\u4e00\u4e2a IP\uff0c web-1 \u4f1a\u4f7f\u7528\u7b2c\u4e8c\u4e2a IP\uff0c\u4ee5\u6b64\u7c7b\u63a8\u3002 StatefulSet Pod \u5728\u66f4\u65b0\u6216\u5220\u9664\u7684\u8fc7\u7a0b\u4e2d OVN \u4e2d\u7684 logical_switch_port \u4e0d\u4f1a\u5220\u9664\uff0c\u65b0\u751f\u6210\u7684 Pod \u76f4\u63a5\u590d\u7528\u65e7\u7684 interface \u4fe1\u606f\u3002\u56e0\u6b64 Pod \u53ef\u4ee5\u590d\u7528 IP/Mac \u53ca\u5176\u4ed6\u7f51\u7edc\u4fe1\u606f\uff0c\u8fbe\u5230\u548c StatefulSet Volume \u7c7b\u4f3c\u7684\u72b6\u6001\u4fdd\u7559\u529f\u80fd\u3002 \u57fa\u4e8e 2 \u7684\u80fd\u529b\uff0c\u5bf9\u4e8e\u6ca1\u6709 ovn.kubernetes.io/ip_pool \u6ce8\u89e3\u7684 StatefulSet\uff0cPod \u7b2c\u4e00\u6b21\u751f\u6210\u65f6\u4f1a\u968f\u673a\u5206\u914d IP/Mac\uff0c\u4e4b\u540e\u5728\u6574\u4e2a StatefulSet \u7684\u751f\u547d\u5468\u671f\u5185\uff0c\u7f51\u7edc\u4fe1\u606f\u90fd\u4f1a\u4fdd\u6301\u56fa\u5b9a\u3002 StatefulSet \u793a\u4f8b \u00b6 apiVersion : apps/v1 kind : StatefulSet metadata : name : web spec : serviceName : \"nginx\" replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx spec : containers : - name : nginx image : docker.io/library/nginx:alpine ports : - containerPort : 80 name : web \u53ef\u4ee5\u5c1d\u8bd5\u5220\u9664 StatefulSet \u4e0b Pod \u89c2\u5bdf Pod IP \u53d8\u5316\u4fe1\u606f\u3002 KubeVirt VM \u56fa\u5b9a\u5730\u5740 \u00b6 \u9488\u5bf9 KubeVirt \u521b\u5efa\u7684 VM \u5b9e\u4f8b\uff0c kube-ovn-controller \u53ef\u4ee5\u6309\u7167\u7c7b\u4f3c StatefulSet Pod \u7684\u65b9\u5f0f\u8fdb\u884c IP \u5730\u5740\u5206\u914d\u548c\u7ba1\u7406\u3002 \u4ee5\u8fbe\u5230 VM \u5b9e\u4f8b\u5728\u751f\u547d\u5468\u671f\u5185\u542f\u505c\uff0c\u5347\u7ea7\uff0c\u8fc1\u79fb\u7b49\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\u5730\u5740\u56fa\u5b9a\u4e0d\u53d8\uff0c\u66f4\u7b26\u865a\u62df\u5316\u5408\u7528\u6237\u7684\u5b9e\u9645\u4f7f\u7528\u4f53\u9a8c\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u56fa\u5b9a\u5730\u5740"},{"location":"guide/static-ip-mac/#_1","text":"Kube-OVN \u9ed8\u8ba4\u4f1a\u6839\u636e Pod \u6240\u5728 Namespace \u6240\u5c5e\u7684\u5b50\u7f51\u4e2d\u968f\u673a\u5206\u914d IP \u548c Mac\u3002 \u9488\u5bf9\u5de5\u4f5c\u8d1f\u8f7d\u9700\u8981\u56fa\u5b9a\u5730\u5740\u7684\u60c5\u51b5\uff0cKube-OVN \u6839\u636e\u4e0d\u540c\u7684\u573a\u666f\uff0c\u63d0\u4f9b\u4e86\u591a\u79cd\u56fa\u5b9a\u5730\u5740\u7684\u65b9\u6cd5\uff1a \u5355\u4e2a Pod \u56fa\u5b9a IP/Mac\u3002 Workload \u901a\u7528 IP Pool \u65b9\u5f0f\u6307\u5b9a\u56fa\u5b9a\u5730\u5740\u8303\u56f4\u3002 StatefulSet \u56fa\u5b9a\u5730\u5740\u3002 KubeVirt VM \u56fa\u5b9a\u5730\u5740\u3002","title":"\u56fa\u5b9a\u5730\u5740"},{"location":"guide/static-ip-mac/#pod-ip-mac","text":"\u53ef\u4ee5\u5728\u521b\u5efa Pod \u65f6\u901a\u8fc7 annotation \u6765\u6307\u5b9a Pod \u8fd0\u884c\u65f6\u6240\u9700\u7684 IP/Mac, kube-ovn-controller \u8fd0\u884c\u65f6\u5c06\u4f1a\u8df3\u8fc7\u5730\u5740\u968f\u673a\u5206\u914d\u9636\u6bb5\uff0c\u7ecf\u8fc7\u51b2\u7a81\u68c0\u6d4b\u540e\u76f4\u63a5\u4f7f\u7528\u6307\u5b9a\u5730\u5740\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u9017\u53f7\u5206\u9694 10.16.0.15,fd00:10:16::15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 spec : containers : - name : static-ip image : docker.io/library/nginx:alpine \u5728\u4f7f\u7528 annotation \u5b9a\u4e49\u5355\u4e2a Pod IP/Mac \u65f6\u9700\u8981\u6ce8\u610f\u4ee5\u4e0b\u51e0\u70b9\uff1a \u6240\u4f7f\u7528\u7684 IP/Mac \u4e0d\u80fd\u548c\u5df2\u6709\u7684 IP/Mac \u51b2\u7a81\u3002 IP \u5fc5\u987b\u5728\u6240\u5c5e\u5b50\u7f51\u7684 CIDR \u5185\u3002 \u53ef\u4ee5\u53ea\u6307\u5b9a IP \u6216 Mac\uff0c\u53ea\u6307\u5b9a\u4e00\u4e2a\u65f6\uff0c\u53e6\u4e00\u4e2a\u4f1a\u968f\u673a\u5206\u914d\u3002","title":"\u5355\u4e2a Pod \u56fa\u5b9a IP \u548c Mac"},{"location":"guide/static-ip-mac/#workload-ip-pool","text":"Kube-OVN \u652f\u6301\u901a\u8fc7 annotation ovn.kubernetes.io/ip_pool \u7ed9 Workload\uff08Deployment/StatefulSet/DaemonSet/Job/CronJob\uff09\u8bbe\u7f6e\u56fa\u5b9a IP\u3002 kube-ovn-controllerr \u4f1a\u81ea\u52a8\u9009\u62e9 ovn.kubernetes.io/ip_pool \u4e2d\u6307\u5b9a\u7684 IP \u5e76\u8fdb\u884c\u51b2\u7a81\u68c0\u6d4b\u3002 IP Pool \u7684 Annotation \u9700\u8981\u52a0\u5728 template \u5185\u7684 annotation \u5b57\u6bb5\uff0c\u9664\u4e86 Kubernetes \u5185\u7f6e\u7684 Workload \u7c7b\u578b\uff0c \u5176\u4ed6\u7528\u6237\u81ea\u5b9a\u4e49\u7684 Workload \u4e5f\u53ef\u4ee5\u4f7f\u7528\u540c\u6837\u7684\u65b9\u5f0f\u8fdb\u884c\u56fa\u5b9a\u5730\u5740\u5206\u914d\u3002","title":"Workload \u901a\u7528 IP Pool \u56fa\u5b9a\u5730\u5740"},{"location":"guide/static-ip-mac/#deployment-ip","text":"apiVersion : apps/v1 kind : Deployment metadata : name : ippool labels : app : ippool spec : replicas : 2 selector : matchLabels : app : ippool template : metadata : labels : app : ippool annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u5206\u53f7\u8fdb\u884c\u5206\u9694 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::000F;10.16.0.17,fd00:10:16::0010 spec : containers : - name : ippool image : docker.io/library/nginx:alpine \u5bf9 Workload \u4f7f\u7528\u56fa\u5b9a IP \u9700\u8981\u6ce8\u610f\u4ee5\u4e0b\u51e0\u70b9\uff1a ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u5e94\u8be5\u5c5e\u4e8e\u6240\u5728\u5b50\u7f51\u7684 CIDR \u5185\u3002 ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u4e0d\u80fd\u548c\u5df2\u4f7f\u7528\u7684 IP \u51b2\u7a81\u3002 \u5f53 ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u6570\u91cf\u5c0f\u4e8e replicas \u6570\u91cf\u65f6\uff0c\u591a\u51fa\u7684 Pod \u5c06\u65e0\u6cd5\u521b\u5efa\u3002\u4f60\u9700\u8981\u6839\u636e Workload \u7684\u66f4\u65b0\u7b56\u7565\u4ee5\u53ca\u6269\u5bb9\u89c4\u5212\u8c03\u6574 ovn.kubernetes.io/ip_pool \u4e2d IP \u7684\u6570\u91cf\u3002","title":"Deployment \u56fa\u5b9a IP \u793a\u4f8b"},{"location":"guide/static-ip-mac/#statefulset","text":"StatefulSet \u548c\u5176\u4ed6 Workload \u76f8\u540c\u53ef\u4ee5\u4f7f\u7528 ovn.kubernetes.io/ip_pool \u6765\u6307\u5b9a Pod \u4f7f\u7528\u7684 IP\u3002 \u7531\u4e8e StatefulSet \u591a\u7528\u4e8e\u6709\u72b6\u6001\u670d\u52a1\uff0c\u5bf9\u7f51\u7edc\u6807\u793a\u7684\u56fa\u5b9a\u6709\u66f4\u9ad8\u7684\u8981\u6c42\uff0cKube-OVN \u505a\u4e86\u7279\u6b8a\u7684\u5f3a\u5316\uff1a Pod \u4f1a\u6309\u987a\u5e8f\u5206\u914d ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP\u3002\u4f8b\u5982 StatefulSet \u7684\u540d\u5b57\u4e3a web\uff0c\u5219 web-0 \u4f1a\u4f7f\u7528 ovn.kubernetes.io/ip_pool \u4e2d\u7684\u7b2c\u4e00\u4e2a IP\uff0c web-1 \u4f1a\u4f7f\u7528\u7b2c\u4e8c\u4e2a IP\uff0c\u4ee5\u6b64\u7c7b\u63a8\u3002 StatefulSet Pod \u5728\u66f4\u65b0\u6216\u5220\u9664\u7684\u8fc7\u7a0b\u4e2d OVN \u4e2d\u7684 logical_switch_port \u4e0d\u4f1a\u5220\u9664\uff0c\u65b0\u751f\u6210\u7684 Pod \u76f4\u63a5\u590d\u7528\u65e7\u7684 interface \u4fe1\u606f\u3002\u56e0\u6b64 Pod \u53ef\u4ee5\u590d\u7528 IP/Mac \u53ca\u5176\u4ed6\u7f51\u7edc\u4fe1\u606f\uff0c\u8fbe\u5230\u548c StatefulSet Volume \u7c7b\u4f3c\u7684\u72b6\u6001\u4fdd\u7559\u529f\u80fd\u3002 \u57fa\u4e8e 2 \u7684\u80fd\u529b\uff0c\u5bf9\u4e8e\u6ca1\u6709 ovn.kubernetes.io/ip_pool \u6ce8\u89e3\u7684 StatefulSet\uff0cPod \u7b2c\u4e00\u6b21\u751f\u6210\u65f6\u4f1a\u968f\u673a\u5206\u914d IP/Mac\uff0c\u4e4b\u540e\u5728\u6574\u4e2a StatefulSet \u7684\u751f\u547d\u5468\u671f\u5185\uff0c\u7f51\u7edc\u4fe1\u606f\u90fd\u4f1a\u4fdd\u6301\u56fa\u5b9a\u3002","title":"StatefulSet \u56fa\u5b9a\u5730\u5740"},{"location":"guide/static-ip-mac/#statefulset_1","text":"apiVersion : apps/v1 kind : StatefulSet metadata : name : web spec : serviceName : \"nginx\" replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx spec : containers : - name : nginx image : docker.io/library/nginx:alpine ports : - containerPort : 80 name : web \u53ef\u4ee5\u5c1d\u8bd5\u5220\u9664 StatefulSet \u4e0b Pod \u89c2\u5bdf Pod IP \u53d8\u5316\u4fe1\u606f\u3002","title":"StatefulSet \u793a\u4f8b"},{"location":"guide/static-ip-mac/#kubevirt-vm","text":"\u9488\u5bf9 KubeVirt \u521b\u5efa\u7684 VM \u5b9e\u4f8b\uff0c kube-ovn-controller \u53ef\u4ee5\u6309\u7167\u7c7b\u4f3c StatefulSet Pod \u7684\u65b9\u5f0f\u8fdb\u884c IP \u5730\u5740\u5206\u914d\u548c\u7ba1\u7406\u3002 \u4ee5\u8fbe\u5230 VM \u5b9e\u4f8b\u5728\u751f\u547d\u5468\u671f\u5185\u542f\u505c\uff0c\u5347\u7ea7\uff0c\u8fc1\u79fb\u7b49\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\u5730\u5740\u56fa\u5b9a\u4e0d\u53d8\uff0c\u66f4\u7b26\u865a\u62df\u5316\u5408\u7528\u6237\u7684\u5b9e\u9645\u4f7f\u7528\u4f53\u9a8c\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"KubeVirt VM \u56fa\u5b9a\u5730\u5740"},{"location":"guide/subnet/","text":"\u5b50\u7f51\u4f7f\u7528 \u00b6 \u5b50\u7f51\u662f Kube-OVN \u4e2d\u7684\u4e00\u4e2a\u6838\u5fc3\u6982\u5ff5\u548c\u57fa\u672c\u4f7f\u7528\u5355\u5143\uff0cKube-OVN \u4f1a\u4ee5\u5b50\u7f51\u6765\u7ec4\u7ec7 IP \u548c\u7f51\u7edc\u914d\u7f6e\uff0c\u6bcf\u4e2a Namespace \u53ef\u4ee5\u5f52\u5c5e\u4e8e\u7279\u5b9a\u7684\u5b50\u7f51\uff0c Namespace \u4e0b\u7684 Pod \u4f1a\u81ea\u52a8\u4ece\u6240\u5c5e\u7684\u5b50\u7f51\u4e2d\u83b7\u53d6 IP \u5e76\u5171\u4eab\u5b50\u7f51\u7684\u7f51\u7edc\u914d\u7f6e\uff08CIDR\uff0c\u7f51\u5173\u7c7b\u578b\uff0c\u8bbf\u95ee\u63a7\u5236\uff0cNAT \u63a7\u5236\u7b49\uff09\u3002 \u548c\u5176\u4ed6 CNI \u7684\u6bcf\u4e2a\u8282\u70b9\u7ed1\u5b9a\u4e00\u4e2a\u5b50\u7f51\u7684\u5b9e\u73b0\u4e0d\u540c\uff0c\u5728 Kube-OVN \u4e2d\u5b50\u7f51\u4e3a\u4e00\u4e2a\u5168\u5c40\u7684\u865a\u62df\u7f51\u7edc\u914d\u7f6e\uff0c\u540c\u4e00\u4e2a\u5b50\u7f51\u7684\u5730\u5740\u53ef\u4ee5\u5206\u5e03\u5728\u4efb\u610f\u4e00\u4e2a\u8282\u70b9\u4e0a\u3002 Overlay \u548c Underlay \u7684\u5b50\u7f51\u5728\u4f7f\u7528\u548c\u914d\u7f6e\u4e0a\u5b58\u5728\u4e00\u4e9b\u5dee\u5f02\uff0c\u672c\u6587\u6863\u5c06\u4f1a\u4ecb\u7ecd\u4e0d\u540c\u7c7b\u578b\u5b50\u7f51\u7684\u4e00\u4e9b\u5171\u540c\u914d\u7f6e\u548c\u5dee\u5f02\u5316\u529f\u80fd\u3002 \u9ed8\u8ba4\u5b50\u7f51 \u00b6 \u4e3a\u4e86\u65b9\u4fbf\u7528\u6237\u7684\u5feb\u901f\u4e0a\u624b\u4f7f\u7528\uff0cKube-OVN \u5185\u7f6e\u4e86\u4e00\u4e2a\u9ed8\u8ba4\u5b50\u7f51\uff0c\u6240\u6709\u672a\u663e\u5f0f\u58f0\u660e\u5b50\u7f51\u5f52\u5c5e\u7684 Namespace \u4f1a\u81ea\u52a8\u4ece\u9ed8\u8ba4\u5b50\u7f51\u4e2d\u5206\u914d IP\uff0c \u5e76\u4f7f\u7528\u9ed8\u8ba4\u5b50\u7f51\u7684\u7f51\u7edc\u4fe1\u606f\u3002\u8be5\u5b50\u7f51\u7684\u914d\u7f6e\u4e3a\u5b89\u88c5\u65f6\u6307\u5b9a\uff0c\u53ef\u4ee5\u53c2\u8003 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \uff0c \u5982\u679c\u8981\u5728\u5b89\u88c5\u540e\u4fee\u6539\u9ed8\u8ba4\u7f51\u7edc\u7684 CIDR \u8bf7\u53c2\u8003 \u4fee\u6539\u9ed8\u8ba4\u7f51\u7edc \u3002 \u5728 Overlay \u6a21\u5f0f\u4e0b\uff0c\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u4e86\u5206\u5e03\u5f0f\u7f51\u5173\u5e76\u5bf9\u51fa\u7f51\u6d41\u91cf\u8fdb\u884c NAT \u8f6c\u6362\uff0c\u5176\u884c\u4e3a\u548c Flannel \u7684\u9ed8\u8ba4\u884c\u4e3a\u57fa\u672c\u4e00\u81f4\uff0c \u7528\u6237\u65e0\u9700\u989d\u5916\u7684\u914d\u7f6e\u5373\u53ef\u4f7f\u7528\u5230\u5927\u90e8\u5206\u7684\u7f51\u7edc\u529f\u80fd\u3002 \u5728 Underlay \u6a21\u5f0f\u4e0b\uff0c\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u7269\u7406\u7f51\u5173\u4f5c\u4e3a\u51fa\u7f51\u7f51\u5173\uff0c\u5e76\u5f00\u542f arping \u68c0\u67e5\u7f51\u7edc\u8fde\u901a\u6027\u3002 \u67e5\u770b\u9ed8\u8ba4\u5b50\u7f51 \u00b6 \u9ed8\u8ba4\u5b50\u7f51 spec \u4e2d\u7684 default \u5b57\u6bb5\u4e3a true\uff0c\u4e00\u4e2a\u96c6\u7fa4\u4e0b\u53ea\u6709\u4e00\u4e2a\u9ed8\u8ba4\u5b50\u7f51\uff0c\u9ed8\u8ba4\u540d\u4e3a ovn-default \u3002 \u67e5\u770b\u9ed8\u8ba4\u5b50\u7f51\uff1a # kubectl get subnet ovn-default -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: ovn-default resourceVersion: \"1571334\" selfLink: /apis/kubeovn.io/v1/subnets/ovn-default uid: 7e2451f8-fb44-4f7f-b3e0-cfd27f6fd5d6 spec: cidrBlock: 10 .16.0.0/16 default: true excludeIps: - 10 .16.0.1 gateway: 10 .16.0.1 gatewayType: distributed natOutgoing: true private: false protocol: IPv4 Join \u5b50\u7f51 \u00b6 \u5728 Kubernetes \u7684\u7f51\u7edc\u89c4\u8303\u4e2d\uff0c\u8981\u6c42 Node \u53ef\u4ee5\u548c\u6240\u6709\u7684 Pod \u76f4\u63a5\u901a\u4fe1\u3002 \u4e3a\u4e86\u5728 Overlay \u7f51\u7edc\u6a21\u5f0f\u4e0b\u8fbe\u5230\u8fd9\u4e2a\u76ee\u7684\uff0c Kube-OVN \u521b\u5efa\u4e86\u4e00\u4e2a join \u5b50\u7f51\uff0c \u5e76\u5728\u6bcf\u4e2a Node \u8282\u70b9\u521b\u5efa\u4e86\u4e00\u5757\u865a\u62df\u7f51\u5361 ovn0 \u63a5\u5165 join \u5b50\u7f51\uff0c\u901a\u8fc7\u8be5\u7f51\u7edc\u5b8c\u6210\u8282\u70b9\u548c Pod \u4e4b\u95f4\u7684\u7f51\u7edc\u4e92\u901a\u3002 \u8be5\u5b50\u7f51\u7684\u914d\u7f6e\u4e3a\u5b89\u88c5\u65f6\u6307\u5b9a\uff0c\u53ef\u4ee5\u53c2\u8003 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \uff0c\u5982\u679c\u8981\u5728\u5b89\u88c5\u540e\u4fee\u6539\u3002 join \u5b50\u7f51\u7684 CIDR \u8bf7\u53c2\u8003 \u4fee\u6539 Join \u5b50\u7f51 \u67e5\u770b Join \u5b50\u7f51 \u00b6 \u8be5\u5b50\u7f51\u9ed8\u8ba4\u540d\u4e3a join \u4e00\u822c\u65e0\u9700\u5bf9\u8be5\u5b50\u7f51 CIDR \u5916\u7684\u5176\u4ed6\u7f51\u7edc\u914d\u7f6e\u8fdb\u884c\u4fee\u6539\u3002 # kubectl get subnet join -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: join resourceVersion: \"1571333\" selfLink: /apis/kubeovn.io/v1/subnets/join uid: 9c744810-c678-4d50-8a7d-b8ec12ef91b8 spec: cidrBlock: 100 .64.0.0/16 default: false excludeIps: - 100 .64.0.1 gateway: 100 .64.0.1 gatewayNode: \"\" gatewayType: \"\" natOutgoing: false private: false protocol: IPv4 \u5728 node \u8282\u70b9\u67e5\u770b ovn0 \u7f51\u5361\uff1a # ifconfig ovn0 ovn0: flags = 4163 <UP,BROADCAST,RUNNING,MULTICAST> mtu 1420 inet 100 .64.0.4 netmask 255 .255.0.0 broadcast 100 .64.255.255 inet6 fe80::800:ff:fe40:5 prefixlen 64 scopeid 0x20<link> ether 0a:00:00:40:00:05 txqueuelen 1000 ( Ethernet ) RX packets 18 bytes 1428 ( 1 .3 KiB ) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 19 bytes 1810 ( 1 .7 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 \u521b\u5efa\u81ea\u5b9a\u4e49\u5b50\u7f51 \u00b6 \u8fd9\u91cc\u6211\u4eec\u4ecb\u7ecd\u521b\u5efa\u4e00\u4e2a\u5b50\u7f51\uff0c\u5e76\u5c06\u5176\u548c\u67d0\u4e2a Namespace \u505a\u5173\u8054\u7684\u57fa\u672c\u64cd\u4f5c\uff0c\u66f4\u591a\u9ad8\u7ea7\u914d\u7f6e\u8bf7\u53c2\u8003\u540e\u7eed\u5185\u5bb9\u3002 \u521b\u5efa\u5b50\u7f51 \u00b6 cat <<EOF | kubectl create -f - apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 10.66.0.0/16 excludeIps: - 10.66.0.1..10.66.0.10 - 10.66.0.101..10.66.0.151 gateway: 10.66.0.1 gatewayType: distributed natOutgoing: true routeTable: \"\" namespaces: - ns1 - ns2 EOF cidrBlock : \u5b50\u7f51 CIDR \u8303\u56f4\uff0c\u540c\u4e00\u4e2a VPC \u4e0b\u7684\u4e0d\u540c Subnet CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 excludeIps : \u4fdd\u7559\u5730\u5740\u5217\u8868\uff0c\u5bb9\u5668\u7f51\u7edc\u5c06\u4e0d\u4f1a\u81ea\u52a8\u5206\u914d\u5217\u8868\u5185\u7684\u5730\u5740\uff0c\u53ef\u7528\u505a\u56fa\u5b9a IP \u5730\u5740\u5206\u914d\u6bb5\uff0c\u4e5f\u53ef\u5728 Underlay \u6a21\u5f0f\u4e0b\u907f\u514d\u548c\u7269\u7406\u7f51\u7edc\u4e2d\u5df2\u6709\u8bbe\u5907\u51b2\u7a81\u3002 gateway \uff1a\u8be5\u5b50\u7f51\u7f51\u5173\u5730\u5740\uff0cOverlay \u6a21\u5f0f\u4e0b Kube-OVN \u4f1a\u81ea\u52a8\u5206\u914d\u5bf9\u5e94\u7684\u903b\u8f91\u7f51\u5173\uff0cUnderlay \u6a21\u5f0f\u4e0b\u8be5\u5730\u5740\u9700\u4e3a\u5e95\u5c42\u7269\u7406\u7f51\u5173\u5730\u5740\u3002 namespaces : \u7ed1\u5b9a\u8be5\u5b50\u7f51\u7684 Namespace \u5217\u8868\uff0c\u7ed1\u5b9a\u540e Namespace \u4e0b\u7684 Pod \u5c06\u4f1a\u4ece\u5f53\u524d\u5b50\u7f51\u5206\u914d\u5730\u5740\u3002 routeTable : \u5173\u8054\u7684\u8def\u7531\u8868\uff0c\u9ed8\u8ba4\u5173\u8054\u4e3b\u8def\u7531\u8868\uff0c\u8def\u7531\u8868\u5b9a\u4e49\u8bf7\u53c2\u8003 \u9759\u6001\u8def\u7531 \u9a8c\u8bc1\u5b50\u7f51\u7ed1\u5b9a\u751f\u6548 \u00b6 # kubectl create ns ns1 namespace/ns1 created # kubectl run nginx --image=docker.io/library/nginx:alpine -n ns1 deployment.apps/nginx created # kubectl get pod -n ns1 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-74d5899f46-n8wtg 1 /1 Running 0 10s 10 .66.0.11 node1 <none> <none> Overlay \u5b50\u7f51\u7f51\u5173\u914d\u7f6e \u00b6 \u8be5\u529f\u80fd\u53ea\u5bf9 Overlay \u6a21\u5f0f\u5b50\u7f51\u751f\u6548\uff0cUnderlay \u7c7b\u578b\u5b50\u7f51\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u9700\u8981\u501f\u52a9\u5e95\u5c42\u7269\u7406\u7f51\u5173\u3002 Overlay \u5b50\u7f51\u4e0b\u7684 Pod \u9700\u8981\u901a\u8fc7\u7f51\u5173\u6765\u8bbf\u95ee\u96c6\u7fa4\u5916\u90e8\u7f51\u7edc\uff0cKube-OVN \u76ee\u524d\u652f\u6301\u4e24\u79cd\u7c7b\u578b\u7684\u7f51\u5173\uff1a \u5206\u5e03\u5f0f\u7f51\u5173\u548c\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u7528\u6237\u53ef\u4ee5\u5728\u5b50\u7f51\u4e2d\u5bf9\u7f51\u5173\u7684\u7c7b\u578b\u8fdb\u884c\u8c03\u6574\u3002 \u4e24\u79cd\u7c7b\u578b\u7f51\u5173\u5747\u652f\u6301 natOutgoing \u8bbe\u7f6e\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9 Pod \u8bbf\u95ee\u5916\u7f51\u65f6\u662f\u5426\u9700\u8981\u8fdb\u884c snat\u3002 \u5206\u5e03\u5f0f\u7f51\u5173 \u00b6 \u5b50\u7f51\u7684\u9ed8\u8ba4\u7c7b\u578b\u7f51\u5173\uff0c\u6bcf\u4e2a node \u4f1a\u4f5c\u4e3a\u5f53\u524d node \u4e0a pod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u7684\u7f51\u5173\u3002 \u6570\u636e\u5305\u4f1a\u901a\u8fc7\u672c\u673a\u7684 ovn0 \u7f51\u5361\u6d41\u5165\u4e3b\u673a\u7f51\u7edc\u6808\uff0c\u518d\u6839\u636e\u4e3b\u673a\u7684\u8def\u7531\u89c4\u5219\u8fdb\u884c\u51fa\u7f51\u3002 \u5f53 natOutgoing \u4e3a true \u65f6\uff0cPod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u5c06\u4f1a\u4f7f\u7528\u5f53\u524d\u6240\u5728\u5bbf\u4e3b\u673a\u7684 IP\u3002 \u5b50\u7f51\u793a\u4f8b\uff0c\u5176\u4e2d gatewayType \u5b57\u6bb5\u4e3a distributed \uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : distributed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : true \u96c6\u4e2d\u5f0f\u7f51\u5173 \u00b6 \u5982\u679c\u5e0c\u671b\u5b50\u7f51\u5185\u6d41\u91cf\u8bbf\u95ee\u5916\u7f51\u4f7f\u7528\u56fa\u5b9a\u7684 IP\uff0c\u4ee5\u4fbf\u5ba1\u8ba1\u548c\u767d\u540d\u5355\u7b49\u5b89\u5168\u64cd\u4f5c\uff0c\u53ef\u4ee5\u5728\u5b50\u7f51\u4e2d\u8bbe\u7f6e\u7f51\u5173\u7c7b\u578b\u4e3a\u96c6\u4e2d\u5f0f\u7f51\u5173\u3002 \u5728\u96c6\u4e2d\u5f0f\u7f51\u5173\u6a21\u5f0f\u4e0b\uff0cPod \u8bbf\u95ee\u5916\u7f51\u7684\u6570\u636e\u5305\u4f1a\u9996\u5148\u88ab\u8def\u7531\u5230\u7279\u5b9a\u8282\u70b9\u7684 ovn0 \u7f51\u5361\uff0c\u518d\u901a\u8fc7\u4e3b\u673a\u7684\u8def\u7531\u89c4\u5219\u8fdb\u884c\u51fa\u7f51\u3002 \u5f53 natOutgoing \u4e3a true \u65f6\uff0cPod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u5c06\u4f1a\u4f7f\u7528\u7279\u5b9a\u5bbf\u4e3b\u673a\u7684 IP\u3002 \u5b50\u7f51\u793a\u4f8b\uff0c\u5176\u4e2d gatewayType \u5b57\u6bb5\u4e3a centralized \uff0c gatewayNode \u4e3a\u7279\u5b9a\u673a\u5668\u5728 Kubernetes \u4e2d\u7684 NodeName\u3002 \u5176\u4e2d gatewayNode \u5b57\u6bb5\u53ef\u4ee5\u4e3a\u9017\u53f7\u5206\u9694\u7684\u591a\u53f0\u4e3b\u673a\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : centralized spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1,node2\" natOutgoing : true \u96c6\u4e2d\u5f0f\u7f51\u5173\u5982\u679c\u5e0c\u671b\u6307\u5b9a\u673a\u5668\u7684\u7279\u5b9a\u7f51\u5361\u8fdb\u884c\u51fa\u7f51\uff0c gatewayNode \u53ef\u66f4\u6539\u4e3a kube-ovn-worker:172.18.0.2, kube-ovn-control-plane:172.18.0.3 \u683c\u5f0f\u3002 \u96c6\u4e2d\u5f0f\u7f51\u5173\u9ed8\u8ba4\u4e3a\u4e3b\u5907\u6a21\u5f0f\uff0c\u53ea\u6709\u4e3b\u8282\u70b9\u8fdb\u884c\u6d41\u91cf\u8f6c\u53d1\uff0c \u5982\u679c\u9700\u8981\u5207\u6362\u4e3a ECMP \u6a21\u5f0f\uff0c\u8bf7\u53c2\u8003 \u96c6\u4e2d\u5f0f\u7f51\u5173 ECMP \u5f00\u542f\u8bbe\u7f6e \u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableEcmp \uff0c\u5c06\u96c6\u4e2d\u5f0f\u5b50\u7f51 ECMP \u5f00\u5173\u63a7\u5236\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f ECMP \u6a21\u5f0f\u3002\u539f\u6709\u7684 kube-ovn-controller Deployment \u4e2d\u7684 enable-ecmp \u53c2\u6570\u4e0d\u518d\u4f7f\u7528\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51\u5f00\u5173\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002 \u5b50\u7f51 ACL \u8bbe\u7f6e \u00b6 \u5bf9\u4e8e\u6709\u7ec6\u7c92\u5ea6 ACL \u63a7\u5236\u7684\u573a\u666f\uff0cKube-OVN \u7684 Subnet \u63d0\u4f9b\u4e86 ACL \u89c4\u5219\u7684\u8bbe\u7f6e\uff0c\u53ef\u4ee5\u5b9e\u73b0\u7f51\u7edc\u89c4\u5219\u7684\u7cbe\u7ec6\u63a7\u5236\u3002 Subnet \u4e2d\u7684 ACL \u89c4\u5219\u548c OVN \u7684 ACL \u89c4\u5219\u4e00\u81f4\uff0c\u76f8\u5173\u5b57\u6bb5\u5185\u5bb9\u53ef\u4ee5\u53c2\u8003 ovn-nb ACL Table \uff0c match \u5b57\u6bb5\u652f\u6301\u7684\u5b57\u6bb5\u53ef\u53c2\u8003 ovn-sb Logical Flow Table \u3002 \u5141\u8bb8 IP \u5730\u5740\u4e3a 10.10.0.2 \u7684 Pod \u8bbf\u95ee\u6240\u6709\u5730\u5740\uff0c\u4f46\u4e0d\u5141\u8bb8\u5176\u4ed6\u5730\u5740\u4e3b\u52a8\u8bbf\u95ee\u81ea\u5df1\u7684 ACL \u89c4\u5219\u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : acl spec : acls : - action : drop direction : to-lport match : ip4.dst == 10.10.0.2 && ip priority : 1002 - action : allow-related direction : from-lport match : ip4.src == 10.10.0.2 && ip priority : 1002 cidrBlock : 10.10.0.0/24 \u5b50\u7f51\u9694\u79bb\u8bbe\u7f6e \u00b6 \u5b50\u7f51 ACL \u7684\u529f\u80fd\u53ef\u4ee5\u8986\u76d6\u5b50\u7f51\u9694\u79bb\u7684\u529f\u80fd\uff0c\u5e76\u6709\u66f4\u597d\u7684\u7075\u6d3b\u6027\uff0c\u6211\u4eec\u63a8\u8350\u4f7f\u7528\u5b50\u7f51 ACL \u6765\u505a\u76f8\u5e94\u7684\u914d\u7f6e\u3002 \u9ed8\u8ba4\u60c5\u51b5\u4e0b Kube-OVN \u521b\u5efa\u7684\u5b50\u7f51\u4e4b\u95f4\u53ef\u4ee5\u76f8\u4e92\u901a\u4fe1\uff0cPod \u4e5f\u53ef\u4ee5\u901a\u8fc7\u7f51\u5173\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u3002 \u5982\u9700\u5bf9\u5b50\u7f51\u95f4\u7684\u8bbf\u95ee\u8fdb\u884c\u63a7\u5236\uff0c\u53ef\u4ee5\u5728\u5b50\u7f51 CRD \u4e2d\u5c06 private \u8bbe\u7f6e\u4e3a true\uff0c\u5219\u8be5\u5b50\u7f51\u5c06\u548c\u5176\u4ed6\u5b50\u7f51\u4ee5\u53ca\u5916\u90e8\u7f51\u7edc\u9694\u79bb\uff0c \u53ea\u80fd\u8fdb\u884c\u5b50\u7f51\u5185\u90e8\u7684\u901a\u4fe1\u3002\u5982\u9700\u5f00\u767d\u540d\u5355\uff0c\u53ef\u4ee5\u901a\u8fc7 allowSubnets \u8fdb\u884c\u8bbe\u7f6e\u3002 allowSubnets \u5185\u7684\u7f51\u6bb5\u548c\u8be5\u5b50\u7f51\u53ef\u4ee5\u53cc\u5411\u4e92\u8bbf\u3002 \u5f00\u542f\u8bbf\u95ee\u63a7\u5236\u7684\u5b50\u7f51\u793a\u4f8b \u00b6 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : private spec : protocol : IPv4 default : false namespaces : - ns1 - ns2 cidrBlock : 10.69.0.0/16 private : true allowSubnets : - 10.16.0.0/16 - 10.18.0.0/16 Underlay \u76f8\u5173\u9009\u9879 \u00b6 \u8be5\u90e8\u5206\u529f\u80fd\u53ea\u5bf9 Underlay \u7c7b\u578b\u5b50\u7f51\u751f\u6548\u3002 vlan : \u5982\u679c\u4f7f\u7528 Underlay \u7f51\u7edc\uff0c\u8be5\u5b57\u6bb5\u7528\u6765\u63a7\u5236\u8be5 Subnet \u548c\u54ea\u4e2a Vlan CR \u8fdb\u884c\u7ed1\u5b9a\u3002\u8be5\u9009\u9879\u9ed8\u8ba4\u4e3a\u7a7a\u5b57\u7b26\u4e32\uff0c\u5373\u4e0d\u4f7f\u7528 Underlay \u7f51\u7edc\u3002 logicalGateway : \u4e00\u4e9b Underlay \u73af\u5883\u4e3a\u7eaf\u4e8c\u5c42\u7f51\u7edc\uff0c\u4e0d\u5b58\u5728\u7269\u7406\u7684\u4e09\u5c42\u7f51\u5173\u3002\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u501f\u52a9 OVN \u672c\u8eab\u7684\u80fd\u529b\u8bbe\u7f6e\u4e00\u4e2a\u865a\u62df\u7f51\u5173\uff0c\u5c06 Underlay \u548c Overlay \u7f51\u7edc\u6253\u901a\u3002\u9ed8\u8ba4\u503c\u4e3a\uff1a false \u3002 \u7f51\u5173\u68c0\u67e5\u8bbe\u7f6e \u00b6 \u9ed8\u8ba4\u60c5\u51b5\u4e0b kube-ovn-cni \u5728\u542f\u52a8 Pod \u540e\u4f1a\u4f7f\u7528 ICMP \u6216 ARP \u534f\u8bae\u8bf7\u6c42\u7f51\u5173\u5e76\u7b49\u5f85\u8fd4\u56de\uff0c \u4ee5\u9a8c\u8bc1\u7f51\u7edc\u5de5\u4f5c\u6b63\u5e38\uff0c\u5728\u90e8\u5206 Underlay \u73af\u5883\u7f51\u5173\u65e0\u6cd5\u54cd\u5e94 ARP \u8bf7\u6c42\uff0c\u6216\u65e0\u9700\u7f51\u7edc\u5916\u90e8\u8054\u901a\u7684\u573a\u666f \u53ef\u4ee5\u5173\u95ed\u7f51\u5173\u68c0\u67e5\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : disable-gw-check spec : disableGatewayCheck : true \u5176\u4ed6\u9ad8\u7ea7\u8bbe\u7f6e \u00b6 IP \u6c60\u4f7f\u7528 QoS \u8bbe\u7f6e \u591a\u7f51\u5361\u7ba1\u7406 DHCP \u9009\u9879 \u5916\u90e8\u7f51\u5173\u8bbe\u7f6e \u96c6\u7fa4\u4e92\u8054\u8bbe\u7f6e \u865a\u62df IP \u8bbe\u7f6e \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b50\u7f51\u4f7f\u7528"},{"location":"guide/subnet/#_1","text":"\u5b50\u7f51\u662f Kube-OVN \u4e2d\u7684\u4e00\u4e2a\u6838\u5fc3\u6982\u5ff5\u548c\u57fa\u672c\u4f7f\u7528\u5355\u5143\uff0cKube-OVN \u4f1a\u4ee5\u5b50\u7f51\u6765\u7ec4\u7ec7 IP \u548c\u7f51\u7edc\u914d\u7f6e\uff0c\u6bcf\u4e2a Namespace \u53ef\u4ee5\u5f52\u5c5e\u4e8e\u7279\u5b9a\u7684\u5b50\u7f51\uff0c Namespace \u4e0b\u7684 Pod \u4f1a\u81ea\u52a8\u4ece\u6240\u5c5e\u7684\u5b50\u7f51\u4e2d\u83b7\u53d6 IP \u5e76\u5171\u4eab\u5b50\u7f51\u7684\u7f51\u7edc\u914d\u7f6e\uff08CIDR\uff0c\u7f51\u5173\u7c7b\u578b\uff0c\u8bbf\u95ee\u63a7\u5236\uff0cNAT \u63a7\u5236\u7b49\uff09\u3002 \u548c\u5176\u4ed6 CNI \u7684\u6bcf\u4e2a\u8282\u70b9\u7ed1\u5b9a\u4e00\u4e2a\u5b50\u7f51\u7684\u5b9e\u73b0\u4e0d\u540c\uff0c\u5728 Kube-OVN \u4e2d\u5b50\u7f51\u4e3a\u4e00\u4e2a\u5168\u5c40\u7684\u865a\u62df\u7f51\u7edc\u914d\u7f6e\uff0c\u540c\u4e00\u4e2a\u5b50\u7f51\u7684\u5730\u5740\u53ef\u4ee5\u5206\u5e03\u5728\u4efb\u610f\u4e00\u4e2a\u8282\u70b9\u4e0a\u3002 Overlay \u548c Underlay \u7684\u5b50\u7f51\u5728\u4f7f\u7528\u548c\u914d\u7f6e\u4e0a\u5b58\u5728\u4e00\u4e9b\u5dee\u5f02\uff0c\u672c\u6587\u6863\u5c06\u4f1a\u4ecb\u7ecd\u4e0d\u540c\u7c7b\u578b\u5b50\u7f51\u7684\u4e00\u4e9b\u5171\u540c\u914d\u7f6e\u548c\u5dee\u5f02\u5316\u529f\u80fd\u3002","title":"\u5b50\u7f51\u4f7f\u7528"},{"location":"guide/subnet/#_2","text":"\u4e3a\u4e86\u65b9\u4fbf\u7528\u6237\u7684\u5feb\u901f\u4e0a\u624b\u4f7f\u7528\uff0cKube-OVN \u5185\u7f6e\u4e86\u4e00\u4e2a\u9ed8\u8ba4\u5b50\u7f51\uff0c\u6240\u6709\u672a\u663e\u5f0f\u58f0\u660e\u5b50\u7f51\u5f52\u5c5e\u7684 Namespace \u4f1a\u81ea\u52a8\u4ece\u9ed8\u8ba4\u5b50\u7f51\u4e2d\u5206\u914d IP\uff0c \u5e76\u4f7f\u7528\u9ed8\u8ba4\u5b50\u7f51\u7684\u7f51\u7edc\u4fe1\u606f\u3002\u8be5\u5b50\u7f51\u7684\u914d\u7f6e\u4e3a\u5b89\u88c5\u65f6\u6307\u5b9a\uff0c\u53ef\u4ee5\u53c2\u8003 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \uff0c \u5982\u679c\u8981\u5728\u5b89\u88c5\u540e\u4fee\u6539\u9ed8\u8ba4\u7f51\u7edc\u7684 CIDR \u8bf7\u53c2\u8003 \u4fee\u6539\u9ed8\u8ba4\u7f51\u7edc \u3002 \u5728 Overlay \u6a21\u5f0f\u4e0b\uff0c\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u4e86\u5206\u5e03\u5f0f\u7f51\u5173\u5e76\u5bf9\u51fa\u7f51\u6d41\u91cf\u8fdb\u884c NAT \u8f6c\u6362\uff0c\u5176\u884c\u4e3a\u548c Flannel \u7684\u9ed8\u8ba4\u884c\u4e3a\u57fa\u672c\u4e00\u81f4\uff0c \u7528\u6237\u65e0\u9700\u989d\u5916\u7684\u914d\u7f6e\u5373\u53ef\u4f7f\u7528\u5230\u5927\u90e8\u5206\u7684\u7f51\u7edc\u529f\u80fd\u3002 \u5728 Underlay \u6a21\u5f0f\u4e0b\uff0c\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u7269\u7406\u7f51\u5173\u4f5c\u4e3a\u51fa\u7f51\u7f51\u5173\uff0c\u5e76\u5f00\u542f arping \u68c0\u67e5\u7f51\u7edc\u8fde\u901a\u6027\u3002","title":"\u9ed8\u8ba4\u5b50\u7f51"},{"location":"guide/subnet/#_3","text":"\u9ed8\u8ba4\u5b50\u7f51 spec \u4e2d\u7684 default \u5b57\u6bb5\u4e3a true\uff0c\u4e00\u4e2a\u96c6\u7fa4\u4e0b\u53ea\u6709\u4e00\u4e2a\u9ed8\u8ba4\u5b50\u7f51\uff0c\u9ed8\u8ba4\u540d\u4e3a ovn-default \u3002 \u67e5\u770b\u9ed8\u8ba4\u5b50\u7f51\uff1a # kubectl get subnet ovn-default -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: ovn-default resourceVersion: \"1571334\" selfLink: /apis/kubeovn.io/v1/subnets/ovn-default uid: 7e2451f8-fb44-4f7f-b3e0-cfd27f6fd5d6 spec: cidrBlock: 10 .16.0.0/16 default: true excludeIps: - 10 .16.0.1 gateway: 10 .16.0.1 gatewayType: distributed natOutgoing: true private: false protocol: IPv4","title":"\u67e5\u770b\u9ed8\u8ba4\u5b50\u7f51"},{"location":"guide/subnet/#join","text":"\u5728 Kubernetes \u7684\u7f51\u7edc\u89c4\u8303\u4e2d\uff0c\u8981\u6c42 Node \u53ef\u4ee5\u548c\u6240\u6709\u7684 Pod \u76f4\u63a5\u901a\u4fe1\u3002 \u4e3a\u4e86\u5728 Overlay \u7f51\u7edc\u6a21\u5f0f\u4e0b\u8fbe\u5230\u8fd9\u4e2a\u76ee\u7684\uff0c Kube-OVN \u521b\u5efa\u4e86\u4e00\u4e2a join \u5b50\u7f51\uff0c \u5e76\u5728\u6bcf\u4e2a Node \u8282\u70b9\u521b\u5efa\u4e86\u4e00\u5757\u865a\u62df\u7f51\u5361 ovn0 \u63a5\u5165 join \u5b50\u7f51\uff0c\u901a\u8fc7\u8be5\u7f51\u7edc\u5b8c\u6210\u8282\u70b9\u548c Pod \u4e4b\u95f4\u7684\u7f51\u7edc\u4e92\u901a\u3002 \u8be5\u5b50\u7f51\u7684\u914d\u7f6e\u4e3a\u5b89\u88c5\u65f6\u6307\u5b9a\uff0c\u53ef\u4ee5\u53c2\u8003 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \uff0c\u5982\u679c\u8981\u5728\u5b89\u88c5\u540e\u4fee\u6539\u3002 join \u5b50\u7f51\u7684 CIDR \u8bf7\u53c2\u8003 \u4fee\u6539 Join \u5b50\u7f51","title":"Join \u5b50\u7f51"},{"location":"guide/subnet/#join_1","text":"\u8be5\u5b50\u7f51\u9ed8\u8ba4\u540d\u4e3a join \u4e00\u822c\u65e0\u9700\u5bf9\u8be5\u5b50\u7f51 CIDR \u5916\u7684\u5176\u4ed6\u7f51\u7edc\u914d\u7f6e\u8fdb\u884c\u4fee\u6539\u3002 # kubectl get subnet join -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: join resourceVersion: \"1571333\" selfLink: /apis/kubeovn.io/v1/subnets/join uid: 9c744810-c678-4d50-8a7d-b8ec12ef91b8 spec: cidrBlock: 100 .64.0.0/16 default: false excludeIps: - 100 .64.0.1 gateway: 100 .64.0.1 gatewayNode: \"\" gatewayType: \"\" natOutgoing: false private: false protocol: IPv4 \u5728 node \u8282\u70b9\u67e5\u770b ovn0 \u7f51\u5361\uff1a # ifconfig ovn0 ovn0: flags = 4163 <UP,BROADCAST,RUNNING,MULTICAST> mtu 1420 inet 100 .64.0.4 netmask 255 .255.0.0 broadcast 100 .64.255.255 inet6 fe80::800:ff:fe40:5 prefixlen 64 scopeid 0x20<link> ether 0a:00:00:40:00:05 txqueuelen 1000 ( Ethernet ) RX packets 18 bytes 1428 ( 1 .3 KiB ) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 19 bytes 1810 ( 1 .7 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0","title":"\u67e5\u770b Join \u5b50\u7f51"},{"location":"guide/subnet/#_4","text":"\u8fd9\u91cc\u6211\u4eec\u4ecb\u7ecd\u521b\u5efa\u4e00\u4e2a\u5b50\u7f51\uff0c\u5e76\u5c06\u5176\u548c\u67d0\u4e2a Namespace \u505a\u5173\u8054\u7684\u57fa\u672c\u64cd\u4f5c\uff0c\u66f4\u591a\u9ad8\u7ea7\u914d\u7f6e\u8bf7\u53c2\u8003\u540e\u7eed\u5185\u5bb9\u3002","title":"\u521b\u5efa\u81ea\u5b9a\u4e49\u5b50\u7f51"},{"location":"guide/subnet/#_5","text":"cat <<EOF | kubectl create -f - apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 10.66.0.0/16 excludeIps: - 10.66.0.1..10.66.0.10 - 10.66.0.101..10.66.0.151 gateway: 10.66.0.1 gatewayType: distributed natOutgoing: true routeTable: \"\" namespaces: - ns1 - ns2 EOF cidrBlock : \u5b50\u7f51 CIDR \u8303\u56f4\uff0c\u540c\u4e00\u4e2a VPC \u4e0b\u7684\u4e0d\u540c Subnet CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 excludeIps : \u4fdd\u7559\u5730\u5740\u5217\u8868\uff0c\u5bb9\u5668\u7f51\u7edc\u5c06\u4e0d\u4f1a\u81ea\u52a8\u5206\u914d\u5217\u8868\u5185\u7684\u5730\u5740\uff0c\u53ef\u7528\u505a\u56fa\u5b9a IP \u5730\u5740\u5206\u914d\u6bb5\uff0c\u4e5f\u53ef\u5728 Underlay \u6a21\u5f0f\u4e0b\u907f\u514d\u548c\u7269\u7406\u7f51\u7edc\u4e2d\u5df2\u6709\u8bbe\u5907\u51b2\u7a81\u3002 gateway \uff1a\u8be5\u5b50\u7f51\u7f51\u5173\u5730\u5740\uff0cOverlay \u6a21\u5f0f\u4e0b Kube-OVN \u4f1a\u81ea\u52a8\u5206\u914d\u5bf9\u5e94\u7684\u903b\u8f91\u7f51\u5173\uff0cUnderlay \u6a21\u5f0f\u4e0b\u8be5\u5730\u5740\u9700\u4e3a\u5e95\u5c42\u7269\u7406\u7f51\u5173\u5730\u5740\u3002 namespaces : \u7ed1\u5b9a\u8be5\u5b50\u7f51\u7684 Namespace \u5217\u8868\uff0c\u7ed1\u5b9a\u540e Namespace \u4e0b\u7684 Pod \u5c06\u4f1a\u4ece\u5f53\u524d\u5b50\u7f51\u5206\u914d\u5730\u5740\u3002 routeTable : \u5173\u8054\u7684\u8def\u7531\u8868\uff0c\u9ed8\u8ba4\u5173\u8054\u4e3b\u8def\u7531\u8868\uff0c\u8def\u7531\u8868\u5b9a\u4e49\u8bf7\u53c2\u8003 \u9759\u6001\u8def\u7531","title":"\u521b\u5efa\u5b50\u7f51"},{"location":"guide/subnet/#_6","text":"# kubectl create ns ns1 namespace/ns1 created # kubectl run nginx --image=docker.io/library/nginx:alpine -n ns1 deployment.apps/nginx created # kubectl get pod -n ns1 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-74d5899f46-n8wtg 1 /1 Running 0 10s 10 .66.0.11 node1 <none> <none>","title":"\u9a8c\u8bc1\u5b50\u7f51\u7ed1\u5b9a\u751f\u6548"},{"location":"guide/subnet/#overlay","text":"\u8be5\u529f\u80fd\u53ea\u5bf9 Overlay \u6a21\u5f0f\u5b50\u7f51\u751f\u6548\uff0cUnderlay \u7c7b\u578b\u5b50\u7f51\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u9700\u8981\u501f\u52a9\u5e95\u5c42\u7269\u7406\u7f51\u5173\u3002 Overlay \u5b50\u7f51\u4e0b\u7684 Pod \u9700\u8981\u901a\u8fc7\u7f51\u5173\u6765\u8bbf\u95ee\u96c6\u7fa4\u5916\u90e8\u7f51\u7edc\uff0cKube-OVN \u76ee\u524d\u652f\u6301\u4e24\u79cd\u7c7b\u578b\u7684\u7f51\u5173\uff1a \u5206\u5e03\u5f0f\u7f51\u5173\u548c\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u7528\u6237\u53ef\u4ee5\u5728\u5b50\u7f51\u4e2d\u5bf9\u7f51\u5173\u7684\u7c7b\u578b\u8fdb\u884c\u8c03\u6574\u3002 \u4e24\u79cd\u7c7b\u578b\u7f51\u5173\u5747\u652f\u6301 natOutgoing \u8bbe\u7f6e\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9 Pod \u8bbf\u95ee\u5916\u7f51\u65f6\u662f\u5426\u9700\u8981\u8fdb\u884c snat\u3002","title":"Overlay \u5b50\u7f51\u7f51\u5173\u914d\u7f6e"},{"location":"guide/subnet/#_7","text":"\u5b50\u7f51\u7684\u9ed8\u8ba4\u7c7b\u578b\u7f51\u5173\uff0c\u6bcf\u4e2a node \u4f1a\u4f5c\u4e3a\u5f53\u524d node \u4e0a pod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u7684\u7f51\u5173\u3002 \u6570\u636e\u5305\u4f1a\u901a\u8fc7\u672c\u673a\u7684 ovn0 \u7f51\u5361\u6d41\u5165\u4e3b\u673a\u7f51\u7edc\u6808\uff0c\u518d\u6839\u636e\u4e3b\u673a\u7684\u8def\u7531\u89c4\u5219\u8fdb\u884c\u51fa\u7f51\u3002 \u5f53 natOutgoing \u4e3a true \u65f6\uff0cPod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u5c06\u4f1a\u4f7f\u7528\u5f53\u524d\u6240\u5728\u5bbf\u4e3b\u673a\u7684 IP\u3002 \u5b50\u7f51\u793a\u4f8b\uff0c\u5176\u4e2d gatewayType \u5b57\u6bb5\u4e3a distributed \uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : distributed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : true","title":"\u5206\u5e03\u5f0f\u7f51\u5173"},{"location":"guide/subnet/#_8","text":"\u5982\u679c\u5e0c\u671b\u5b50\u7f51\u5185\u6d41\u91cf\u8bbf\u95ee\u5916\u7f51\u4f7f\u7528\u56fa\u5b9a\u7684 IP\uff0c\u4ee5\u4fbf\u5ba1\u8ba1\u548c\u767d\u540d\u5355\u7b49\u5b89\u5168\u64cd\u4f5c\uff0c\u53ef\u4ee5\u5728\u5b50\u7f51\u4e2d\u8bbe\u7f6e\u7f51\u5173\u7c7b\u578b\u4e3a\u96c6\u4e2d\u5f0f\u7f51\u5173\u3002 \u5728\u96c6\u4e2d\u5f0f\u7f51\u5173\u6a21\u5f0f\u4e0b\uff0cPod \u8bbf\u95ee\u5916\u7f51\u7684\u6570\u636e\u5305\u4f1a\u9996\u5148\u88ab\u8def\u7531\u5230\u7279\u5b9a\u8282\u70b9\u7684 ovn0 \u7f51\u5361\uff0c\u518d\u901a\u8fc7\u4e3b\u673a\u7684\u8def\u7531\u89c4\u5219\u8fdb\u884c\u51fa\u7f51\u3002 \u5f53 natOutgoing \u4e3a true \u65f6\uff0cPod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u5c06\u4f1a\u4f7f\u7528\u7279\u5b9a\u5bbf\u4e3b\u673a\u7684 IP\u3002 \u5b50\u7f51\u793a\u4f8b\uff0c\u5176\u4e2d gatewayType \u5b57\u6bb5\u4e3a centralized \uff0c gatewayNode \u4e3a\u7279\u5b9a\u673a\u5668\u5728 Kubernetes \u4e2d\u7684 NodeName\u3002 \u5176\u4e2d gatewayNode \u5b57\u6bb5\u53ef\u4ee5\u4e3a\u9017\u53f7\u5206\u9694\u7684\u591a\u53f0\u4e3b\u673a\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : centralized spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1,node2\" natOutgoing : true \u96c6\u4e2d\u5f0f\u7f51\u5173\u5982\u679c\u5e0c\u671b\u6307\u5b9a\u673a\u5668\u7684\u7279\u5b9a\u7f51\u5361\u8fdb\u884c\u51fa\u7f51\uff0c gatewayNode \u53ef\u66f4\u6539\u4e3a kube-ovn-worker:172.18.0.2, kube-ovn-control-plane:172.18.0.3 \u683c\u5f0f\u3002 \u96c6\u4e2d\u5f0f\u7f51\u5173\u9ed8\u8ba4\u4e3a\u4e3b\u5907\u6a21\u5f0f\uff0c\u53ea\u6709\u4e3b\u8282\u70b9\u8fdb\u884c\u6d41\u91cf\u8f6c\u53d1\uff0c \u5982\u679c\u9700\u8981\u5207\u6362\u4e3a ECMP \u6a21\u5f0f\uff0c\u8bf7\u53c2\u8003 \u96c6\u4e2d\u5f0f\u7f51\u5173 ECMP \u5f00\u542f\u8bbe\u7f6e \u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableEcmp \uff0c\u5c06\u96c6\u4e2d\u5f0f\u5b50\u7f51 ECMP \u5f00\u5173\u63a7\u5236\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f ECMP \u6a21\u5f0f\u3002\u539f\u6709\u7684 kube-ovn-controller Deployment \u4e2d\u7684 enable-ecmp \u53c2\u6570\u4e0d\u518d\u4f7f\u7528\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51\u5f00\u5173\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002","title":"\u96c6\u4e2d\u5f0f\u7f51\u5173"},{"location":"guide/subnet/#acl","text":"\u5bf9\u4e8e\u6709\u7ec6\u7c92\u5ea6 ACL \u63a7\u5236\u7684\u573a\u666f\uff0cKube-OVN \u7684 Subnet \u63d0\u4f9b\u4e86 ACL \u89c4\u5219\u7684\u8bbe\u7f6e\uff0c\u53ef\u4ee5\u5b9e\u73b0\u7f51\u7edc\u89c4\u5219\u7684\u7cbe\u7ec6\u63a7\u5236\u3002 Subnet \u4e2d\u7684 ACL \u89c4\u5219\u548c OVN \u7684 ACL \u89c4\u5219\u4e00\u81f4\uff0c\u76f8\u5173\u5b57\u6bb5\u5185\u5bb9\u53ef\u4ee5\u53c2\u8003 ovn-nb ACL Table \uff0c match \u5b57\u6bb5\u652f\u6301\u7684\u5b57\u6bb5\u53ef\u53c2\u8003 ovn-sb Logical Flow Table \u3002 \u5141\u8bb8 IP \u5730\u5740\u4e3a 10.10.0.2 \u7684 Pod \u8bbf\u95ee\u6240\u6709\u5730\u5740\uff0c\u4f46\u4e0d\u5141\u8bb8\u5176\u4ed6\u5730\u5740\u4e3b\u52a8\u8bbf\u95ee\u81ea\u5df1\u7684 ACL \u89c4\u5219\u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : acl spec : acls : - action : drop direction : to-lport match : ip4.dst == 10.10.0.2 && ip priority : 1002 - action : allow-related direction : from-lport match : ip4.src == 10.10.0.2 && ip priority : 1002 cidrBlock : 10.10.0.0/24","title":"\u5b50\u7f51 ACL \u8bbe\u7f6e"},{"location":"guide/subnet/#_9","text":"\u5b50\u7f51 ACL \u7684\u529f\u80fd\u53ef\u4ee5\u8986\u76d6\u5b50\u7f51\u9694\u79bb\u7684\u529f\u80fd\uff0c\u5e76\u6709\u66f4\u597d\u7684\u7075\u6d3b\u6027\uff0c\u6211\u4eec\u63a8\u8350\u4f7f\u7528\u5b50\u7f51 ACL \u6765\u505a\u76f8\u5e94\u7684\u914d\u7f6e\u3002 \u9ed8\u8ba4\u60c5\u51b5\u4e0b Kube-OVN \u521b\u5efa\u7684\u5b50\u7f51\u4e4b\u95f4\u53ef\u4ee5\u76f8\u4e92\u901a\u4fe1\uff0cPod \u4e5f\u53ef\u4ee5\u901a\u8fc7\u7f51\u5173\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u3002 \u5982\u9700\u5bf9\u5b50\u7f51\u95f4\u7684\u8bbf\u95ee\u8fdb\u884c\u63a7\u5236\uff0c\u53ef\u4ee5\u5728\u5b50\u7f51 CRD \u4e2d\u5c06 private \u8bbe\u7f6e\u4e3a true\uff0c\u5219\u8be5\u5b50\u7f51\u5c06\u548c\u5176\u4ed6\u5b50\u7f51\u4ee5\u53ca\u5916\u90e8\u7f51\u7edc\u9694\u79bb\uff0c \u53ea\u80fd\u8fdb\u884c\u5b50\u7f51\u5185\u90e8\u7684\u901a\u4fe1\u3002\u5982\u9700\u5f00\u767d\u540d\u5355\uff0c\u53ef\u4ee5\u901a\u8fc7 allowSubnets \u8fdb\u884c\u8bbe\u7f6e\u3002 allowSubnets \u5185\u7684\u7f51\u6bb5\u548c\u8be5\u5b50\u7f51\u53ef\u4ee5\u53cc\u5411\u4e92\u8bbf\u3002","title":"\u5b50\u7f51\u9694\u79bb\u8bbe\u7f6e"},{"location":"guide/subnet/#_10","text":"apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : private spec : protocol : IPv4 default : false namespaces : - ns1 - ns2 cidrBlock : 10.69.0.0/16 private : true allowSubnets : - 10.16.0.0/16 - 10.18.0.0/16","title":"\u5f00\u542f\u8bbf\u95ee\u63a7\u5236\u7684\u5b50\u7f51\u793a\u4f8b"},{"location":"guide/subnet/#underlay","text":"\u8be5\u90e8\u5206\u529f\u80fd\u53ea\u5bf9 Underlay \u7c7b\u578b\u5b50\u7f51\u751f\u6548\u3002 vlan : \u5982\u679c\u4f7f\u7528 Underlay \u7f51\u7edc\uff0c\u8be5\u5b57\u6bb5\u7528\u6765\u63a7\u5236\u8be5 Subnet \u548c\u54ea\u4e2a Vlan CR \u8fdb\u884c\u7ed1\u5b9a\u3002\u8be5\u9009\u9879\u9ed8\u8ba4\u4e3a\u7a7a\u5b57\u7b26\u4e32\uff0c\u5373\u4e0d\u4f7f\u7528 Underlay \u7f51\u7edc\u3002 logicalGateway : \u4e00\u4e9b Underlay \u73af\u5883\u4e3a\u7eaf\u4e8c\u5c42\u7f51\u7edc\uff0c\u4e0d\u5b58\u5728\u7269\u7406\u7684\u4e09\u5c42\u7f51\u5173\u3002\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u501f\u52a9 OVN \u672c\u8eab\u7684\u80fd\u529b\u8bbe\u7f6e\u4e00\u4e2a\u865a\u62df\u7f51\u5173\uff0c\u5c06 Underlay \u548c Overlay \u7f51\u7edc\u6253\u901a\u3002\u9ed8\u8ba4\u503c\u4e3a\uff1a false \u3002","title":"Underlay \u76f8\u5173\u9009\u9879"},{"location":"guide/subnet/#_11","text":"\u9ed8\u8ba4\u60c5\u51b5\u4e0b kube-ovn-cni \u5728\u542f\u52a8 Pod \u540e\u4f1a\u4f7f\u7528 ICMP \u6216 ARP \u534f\u8bae\u8bf7\u6c42\u7f51\u5173\u5e76\u7b49\u5f85\u8fd4\u56de\uff0c \u4ee5\u9a8c\u8bc1\u7f51\u7edc\u5de5\u4f5c\u6b63\u5e38\uff0c\u5728\u90e8\u5206 Underlay \u73af\u5883\u7f51\u5173\u65e0\u6cd5\u54cd\u5e94 ARP \u8bf7\u6c42\uff0c\u6216\u65e0\u9700\u7f51\u7edc\u5916\u90e8\u8054\u901a\u7684\u573a\u666f \u53ef\u4ee5\u5173\u95ed\u7f51\u5173\u68c0\u67e5\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : disable-gw-check spec : disableGatewayCheck : true","title":"\u7f51\u5173\u68c0\u67e5\u8bbe\u7f6e"},{"location":"guide/subnet/#_12","text":"IP \u6c60\u4f7f\u7528 QoS \u8bbe\u7f6e \u591a\u7f51\u5361\u7ba1\u7406 DHCP \u9009\u9879 \u5916\u90e8\u7f51\u5173\u8bbe\u7f6e \u96c6\u7fa4\u4e92\u8054\u8bbe\u7f6e \u865a\u62df IP \u8bbe\u7f6e \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5176\u4ed6\u9ad8\u7ea7\u8bbe\u7f6e"},{"location":"guide/vpc-qos/","text":"VPC QoS \u00b6 Kube-OVN \u652f\u6301\u4f7f\u7528 QoSPolicy CRD \u5bf9\u81ea\u5b9a\u4e49 VPC \u7684\u6d41\u91cf\u901f\u7387\u8fdb\u884c\u9650\u5236\u3002 EIP QoS \u00b6 \u5bf9 EIP \u8fdb\u884c\u9650\u901f\uff0c\u9650\u901f\u503c\u4e3a 1Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 1\uff0c\u8fd9\u91cc shared=false \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ea\u80fd\u7ed9\u8fd9\u4e2a EIP \u4f7f\u7528\u4e14\u652f\u6301\u52a8\u6001\u4fee\u6539 QoSPolicy \u53bb\u53d8\u66f4 QoS \u89c4\u5219\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-eip-example spec : shared : false bindingType : EIP bandwidthLimitRules : - name : eip-ingress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : ingress - name : eip-egress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : egress IptablesEIP \u914d\u7f6e\u5982\u4e0b\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-1 spec : natGwDp : gw1 qosPolicy : qos-eip-example .spec.qosPolicy \u7684\u503c\u652f\u6301\u521b\u5efa\u65f6\u4f20\u5165\uff0c\u4e5f\u652f\u6301\u521b\u5efa\u540e\u4fee\u6539\u3002 \u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 EIP \u00b6 \u901a\u8fc7 label \u67e5\u770b\u5df2\u7ecf\u8bbe\u7f6e\u5bf9\u5e94 qos \u7684 eip\uff1a # kubectl get eip -l ovn.kubernetes.io/qos=qos-eip-example NAME IP MAC NAT NATGWDP READY eip-1 172 .18.11.24 00 :00:00:34:41:0B fip gw1 true VPC NATGW net1 \u7f51\u5361 QoS \u00b6 \u5bf9 VPC NATGW \u7684 net1 \u7f51\u5361\u901f\u7387\u8fdb\u884c\u9650\u5236\uff0c\u9650\u901f\u503c\u4e3a 10Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 3\uff0c\u8fd9\u91cc shared=true \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ef\u4ee5\u540c\u65f6\u7ed9\u591a\u4e2a\u8d44\u6e90\u4f7f\u7528\uff0c\u8fd9\u79cd\u573a\u666f\u4e0b\u4e0d\u5141\u8bb8\u4fee\u6539 QoSPolicy \u7684\u5185\u5bb9\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-ingress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : ingress - name : net1-egress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : egress VpcNatGateway \u914d\u7f6e\u5982\u4e0b\uff1a kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" .spec.qosPolicy \u7684\u503c\u652f\u6301\u521b\u5efa\u4f20\u5165\uff0c\u4e5f\u652f\u6301\u540e\u7eed\u4fee\u6539\u3002 net1 \u7f51\u5361\u7279\u5b9a\u6d41\u91cf QoS \u00b6 \u5bf9 net1 \u7f51\u5361\u4e0a\u7279\u5b9a\u6d41\u91cf\u8fdb\u884c\u9650\u901f\uff0c\u9650\u901f\u503c\u4e3a 5Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 2\uff0c\u8fd9\u91cc shared=true \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ef\u4ee5\u540c\u65f6\u7ed9\u591a\u4e2a\u8d44\u6e90\u4f7f\u7528\uff0c\u6b64\u65f6\u4e0d\u5141\u8bb8\u4fee\u6539 QoSPolicy \u7684\u5185\u5bb9\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-extip-ingress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : ingress matchType : ip matchValue : src 172.18.11.22/32 - name : net1-extip-egress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : egress matchType : ip matchValue : dst 172.18.11.23/32 VpcNatGateway \u914d\u7f6e\u5982\u4e0b\uff1a kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" \u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 NATGW \u00b6 \u901a\u8fc7 label \u67e5\u770b\u5df2\u7ecf\u8bbe\u7f6e\u5bf9\u5e94 qos \u7684 eip\uff1a # kubectl get vpc-nat-gw -l ovn.kubernetes.io/qos=qos-natgw-example NAME VPC SUBNET LANIP gw1 test-vpc-1 net1 10 .0.1.254 \u67e5\u770b qos \u89c4\u5219 \u00b6 # kubectl get qos -A NAME SHARED BINDINGTYPE qos-eip-example false EIP qos-natgw-example true NATGW \u9650\u5236 \u00b6 \u53ea\u6709\u5728\u672a\u4f7f\u7528\u65f6\u624d\u80fd\u5220\u9664 QoS \u7b56\u7565\u3002\u56e0\u6b64\uff0c\u5728\u5220\u9664 QoS \u7b56\u7565\u4e4b\u524d\uff0c\u8bf7\u5148\u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 EIP \u548c NATGW\uff0c\u53bb\u6389\u5b83\u4eec\u7684 spec.qosPolicy \u914d\u7f6e\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC QoS \u914d\u7f6e"},{"location":"guide/vpc-qos/#vpc-qos","text":"Kube-OVN \u652f\u6301\u4f7f\u7528 QoSPolicy CRD \u5bf9\u81ea\u5b9a\u4e49 VPC \u7684\u6d41\u91cf\u901f\u7387\u8fdb\u884c\u9650\u5236\u3002","title":"VPC QoS"},{"location":"guide/vpc-qos/#eip-qos","text":"\u5bf9 EIP \u8fdb\u884c\u9650\u901f\uff0c\u9650\u901f\u503c\u4e3a 1Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 1\uff0c\u8fd9\u91cc shared=false \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ea\u80fd\u7ed9\u8fd9\u4e2a EIP \u4f7f\u7528\u4e14\u652f\u6301\u52a8\u6001\u4fee\u6539 QoSPolicy \u53bb\u53d8\u66f4 QoS \u89c4\u5219\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-eip-example spec : shared : false bindingType : EIP bandwidthLimitRules : - name : eip-ingress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : ingress - name : eip-egress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : egress IptablesEIP \u914d\u7f6e\u5982\u4e0b\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-1 spec : natGwDp : gw1 qosPolicy : qos-eip-example .spec.qosPolicy \u7684\u503c\u652f\u6301\u521b\u5efa\u65f6\u4f20\u5165\uff0c\u4e5f\u652f\u6301\u521b\u5efa\u540e\u4fee\u6539\u3002","title":"EIP QoS"},{"location":"guide/vpc-qos/#qos-eip","text":"\u901a\u8fc7 label \u67e5\u770b\u5df2\u7ecf\u8bbe\u7f6e\u5bf9\u5e94 qos \u7684 eip\uff1a # kubectl get eip -l ovn.kubernetes.io/qos=qos-eip-example NAME IP MAC NAT NATGWDP READY eip-1 172 .18.11.24 00 :00:00:34:41:0B fip gw1 true","title":"\u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 EIP"},{"location":"guide/vpc-qos/#vpc-natgw-net1-qos","text":"\u5bf9 VPC NATGW \u7684 net1 \u7f51\u5361\u901f\u7387\u8fdb\u884c\u9650\u5236\uff0c\u9650\u901f\u503c\u4e3a 10Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 3\uff0c\u8fd9\u91cc shared=true \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ef\u4ee5\u540c\u65f6\u7ed9\u591a\u4e2a\u8d44\u6e90\u4f7f\u7528\uff0c\u8fd9\u79cd\u573a\u666f\u4e0b\u4e0d\u5141\u8bb8\u4fee\u6539 QoSPolicy \u7684\u5185\u5bb9\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-ingress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : ingress - name : net1-egress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : egress VpcNatGateway \u914d\u7f6e\u5982\u4e0b\uff1a kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" .spec.qosPolicy \u7684\u503c\u652f\u6301\u521b\u5efa\u4f20\u5165\uff0c\u4e5f\u652f\u6301\u540e\u7eed\u4fee\u6539\u3002","title":"VPC NATGW net1 \u7f51\u5361 QoS"},{"location":"guide/vpc-qos/#net1-qos","text":"\u5bf9 net1 \u7f51\u5361\u4e0a\u7279\u5b9a\u6d41\u91cf\u8fdb\u884c\u9650\u901f\uff0c\u9650\u901f\u503c\u4e3a 5Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 2\uff0c\u8fd9\u91cc shared=true \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ef\u4ee5\u540c\u65f6\u7ed9\u591a\u4e2a\u8d44\u6e90\u4f7f\u7528\uff0c\u6b64\u65f6\u4e0d\u5141\u8bb8\u4fee\u6539 QoSPolicy \u7684\u5185\u5bb9\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-extip-ingress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : ingress matchType : ip matchValue : src 172.18.11.22/32 - name : net1-extip-egress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : egress matchType : ip matchValue : dst 172.18.11.23/32 VpcNatGateway \u914d\u7f6e\u5982\u4e0b\uff1a kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\"","title":"net1 \u7f51\u5361\u7279\u5b9a\u6d41\u91cf QoS"},{"location":"guide/vpc-qos/#qos-natgw","text":"\u901a\u8fc7 label \u67e5\u770b\u5df2\u7ecf\u8bbe\u7f6e\u5bf9\u5e94 qos \u7684 eip\uff1a # kubectl get vpc-nat-gw -l ovn.kubernetes.io/qos=qos-natgw-example NAME VPC SUBNET LANIP gw1 test-vpc-1 net1 10 .0.1.254","title":"\u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 NATGW"},{"location":"guide/vpc-qos/#qos","text":"# kubectl get qos -A NAME SHARED BINDINGTYPE qos-eip-example false EIP qos-natgw-example true NATGW","title":"\u67e5\u770b qos \u89c4\u5219"},{"location":"guide/vpc-qos/#_1","text":"\u53ea\u6709\u5728\u672a\u4f7f\u7528\u65f6\u624d\u80fd\u5220\u9664 QoS \u7b56\u7565\u3002\u56e0\u6b64\uff0c\u5728\u5220\u9664 QoS \u7b56\u7565\u4e4b\u524d\uff0c\u8bf7\u5148\u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 EIP \u548c NATGW\uff0c\u53bb\u6389\u5b83\u4eec\u7684 spec.qosPolicy \u914d\u7f6e\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9650\u5236"},{"location":"guide/vpc/","text":"VPC \u4f7f\u7528 \u00b6 Kube-OVN \u652f\u6301\u591a\u79df\u6237\u9694\u79bb\u7ea7\u522b\u7684 VPC \u7f51\u7edc\u3002\u4e0d\u540c VPC \u7f51\u7edc\u76f8\u4e92\u72ec\u7acb\uff0c\u53ef\u4ee5\u5206\u522b\u914d\u7f6e Subnet \u7f51\u6bb5\uff0c \u8def\u7531\u7b56\u7565\uff0c\u5b89\u5168\u7b56\u7565\uff0c\u51fa\u7f51\u7f51\u5173\uff0cEIP \u7b49\u914d\u7f6e\u3002 VPC \u4e3b\u8981\u7528\u4e8e\u6709\u591a\u79df\u6237\u7f51\u7edc\u5f3a\u9694\u79bb\u7684\u573a\u666f\uff0c\u90e8\u5206 Kubernetes \u7f51\u7edc\u529f\u80fd\u5728\u591a\u79df\u6237\u7f51\u7edc\u4e0b\u5b58\u5728\u51b2\u7a81\u3002 \u4f8b\u5982\u8282\u70b9\u548c Pod \u4e92\u8bbf\uff0cNodePort \u529f\u80fd\uff0c\u57fa\u4e8e\u7f51\u7edc\u8bbf\u95ee\u7684\u5065\u5eb7\u68c0\u67e5\u548c DNS \u80fd\u529b\u5728\u591a\u79df\u6237\u7f51\u7edc\u573a\u666f\u6682\u4e0d\u652f\u6301\u3002 \u4e3a\u4e86\u65b9\u4fbf\u5e38\u89c1 Kubernetes \u7684\u4f7f\u7528\u573a\u666f\uff0cKube-OVN \u9ed8\u8ba4 VPC \u505a\u4e86\u7279\u6b8a\u8bbe\u8ba1\uff0c\u8be5 VPC \u4e0b\u7684 Subnet \u53ef\u4ee5\u6ee1\u8db3 Kubernetes \u89c4\u8303\u3002\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u652f\u6301\u672c\u6587\u6863\u4ecb\u7ecd\u7684\u9759\u6001\u8def\u7531\uff0cEIP \u548c NAT \u7f51\u5173\u7b49\u529f\u80fd\u3002 \u5e38\u89c1\u9694\u79bb\u9700\u6c42\u53ef\u901a\u8fc7\u9ed8\u8ba4 VPC \u4e0b\u7684\u7f51\u7edc\u7b56\u7565\u548c\u5b50\u7f51 ACL \u5b9e\u73b0\uff0c\u5728\u4f7f\u7528\u81ea\u5b9a\u4e49 VPC \u524d\u8bf7\u660e\u786e\u662f\u5426\u9700\u8981 VPC \u7ea7\u522b\u7684\u9694\u79bb\uff0c\u5e76\u4e86\u89e3\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684\u9650\u5236\u3002 \u5728 Underlay \u7f51\u7edc\u4e0b\uff0c\u7269\u7406\u4ea4\u6362\u673a\u8d1f\u8d23\u6570\u636e\u9762\u8f6c\u53d1\uff0cVPC \u65e0\u6cd5\u5bf9 Underlay \u5b50\u7f51\u8fdb\u884c\u9694\u79bb\u3002 \u521b\u5efa\u81ea\u5b9a\u4e49 VPC \u00b6 \u521b\u5efa\u4e24\u4e2a VPC\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : namespaces : - ns1 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-2 spec : namespaces : - ns2 namespaces \u53ef\u4ee5\u9650\u5b9a\u53ea\u6709\u54ea\u4e9b Namespace \u53ef\u4ee5\u4f7f\u7528\u5f53\u524d VPC\uff0c\u82e5\u4e3a\u7a7a\u5219\u4e0d\u9650\u5b9a\u3002 \u521b\u5efa\u4e24\u4e2a\u5b50\u7f51\uff0c\u5206\u5c5e\u4e24\u4e2a\u4e0d\u540c\u7684 VPC \u5e76\u6709\u76f8\u540c\u7684 CIDR: kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : test-vpc-1 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns1 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : test-vpc-2 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns2 \u5206\u522b\u5728\u4e24\u4e2a Namespace \u4e0b\u521b\u5efa Pod: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 namespace : ns1 name : vpc1-pod spec : containers : - name : vpc1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 namespace : ns2 name : vpc2-pod spec : containers : - name : vpc2-pod image : docker.io/library/nginx:alpine \u8fd0\u884c\u6210\u529f\u540e\u53ef\u89c2\u5bdf\u4e24\u4e2a Pod \u5730\u5740\u5c5e\u4e8e\u540c\u4e00\u4e2a CIDR\uff0c\u4f46\u7531\u4e8e\u8fd0\u884c\u5728\u4e0d\u540c\u7684\u79df\u6237 VPC\uff0c\u4e24\u4e2a Pod \u65e0\u6cd5\u76f8\u4e92\u8bbf\u95ee\u3002 \u81ea\u5b9a\u4e49 VPC Pod \u652f\u6301 livenessProbe \u548c readinessProbe \u00b6 \u7531\u4e8e\u5e38\u89c4\u914d\u7f6e\u4e0b\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 Pod \u548c\u8282\u70b9\u7684\u7f51\u7edc\u4e4b\u95f4\u5e76\u4e0d\u4e92\u901a\uff0c\u6240\u4ee5 kubelet \u53d1\u9001\u7684\u63a2\u6d4b\u62a5\u6587\u65e0\u6cd5\u5230\u8fbe\u81ea\u5b9a VPC \u5185\u7684 Pod\u3002Kube-OVN \u901a\u8fc7 TProxy \u5c06 kubelet \u53d1\u9001\u7684\u63a2\u6d4b\u62a5\u6587\u91cd\u5b9a\u5411\u5230\u81ea\u5b9a\u4e49 VPC \u5185\u7684 Pod\uff0c\u4ece\u800c\u5b9e\u73b0\u8fd9\u4e00\u529f\u80fd\u3002 \u914d\u7f6e\u65b9\u6cd5\u5982\u4e0b\uff0c\u5728 Daemonset kube-ovn-cni \u4e2d\u589e\u52a0\u53c2\u6570 --enable-tproxy=true \uff1a spec : template : spec : containers : - args : - --enable-tproxy=true \u8be5\u529f\u80fd\u9650\u5236\u6761\u4ef6\uff1a \u5f53\u540c\u4e00\u4e2a\u8282\u70b9\u4e0b\u51fa\u73b0\u4e0d\u540c VPC \u4e0b\u7684 Pod \u5177\u6709\u76f8\u540c\u7684 IP\uff0c\u63a2\u6d4b\u529f\u80fd\u5931\u6548\u3002 \u76ee\u524d\u6682\u65f6\u53ea\u652f\u6301 tcpSocket \u548c httpGet \u4e24\u79cd\u63a2\u6d4b\u65b9\u5f0f\u3002 \u521b\u5efa VPC \u7f51\u5173 \u00b6 \u81ea\u5b9a\u4e49 VPC \u4e0b\u7684\u5b50\u7f51\u4e0d\u652f\u6301\u9ed8\u8ba4 VPC \u4e0b\u7684\u5206\u5e03\u5f0f\u7f51\u5173\u548c\u96c6\u4e2d\u5f0f\u7f51\u5173\u3002 VPC \u5185\u5bb9\u5668\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u9700\u8981\u901a\u8fc7 VPC \u7f51\u5173\uff0cVPC \u7f51\u5173\u53ef\u4ee5\u6253\u901a\u7269\u7406\u7f51\u7edc\u548c\u79df\u6237\u7f51\u7edc\uff0c\u5e76\u63d0\u4f9b \u6d6e\u52a8 IP\uff0cSNAT \u548c DNAT \u529f\u80fd\u3002 VPC \u7f51\u5173\u529f\u80fd\u4f9d\u8d56 Multus-CNI \u7684\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5b89\u88c5\u8bf7\u53c2\u8003 multus-cni \u3002 \u914d\u7f6e\u5916\u90e8\u7f51\u7edc \u00b6 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-vpc-external-network spec : protocol : IPv4 provider : ovn-vpc-external-network.kube-system cidrBlock : 192.168.0.0/24 gateway : 192.168.0.1 # IP address of the physical gateway excludeIps : - 192.168.0.1..192.168.0.10 --- apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-vpc-external-network namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth1\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-vpc-external-network.kube-system\" } }' \u8be5 Subnet \u7528\u6765\u7ba1\u7406\u53ef\u7528\u7684\u5916\u90e8\u5730\u5740\uff0c\u7f51\u6bb5\u5185\u7684\u5730\u5740\u5c06\u4f1a\u901a\u8fc7 Macvlan \u5206\u914d\u7ed9 VPC \u7f51\u5173\uff0c\u8bf7\u548c\u7f51\u7edc\u7ba1\u7406\u6c9f\u901a\u7ed9\u51fa\u53ef\u7528\u7684\u7269\u7406\u6bb5 IP\u3002 VPC \u7f51\u5173\u4f7f\u7528 Macvlan \u505a\u7269\u7406\u7f51\u7edc\u914d\u7f6e\uff0c NetworkAttachmentDefinition \u7684 master \u9700\u4e3a\u5bf9\u5e94\u7269\u7406\u7f51\u8def\u7f51\u5361\u7684\u7f51\u5361\u540d\u3002 name \u5916\u90e8\u7f51\u7edc\u540d\u79f0\u3002 \u5728 Macvlan \u6a21\u5f0f\u4e0b\uff0c\u9644\u5c5e\u7f51\u5361\u4f1a\u5c06\u6570\u636e\u5305\u76f4\u63a5\u901a\u8fc7\u8be5\u8282\u70b9\u7f51\u5361\u5bf9\u5916\u53d1\u9001\uff0cL2/L3 \u5c42\u9762\u7684\u8f6c\u53d1\u80fd\u529b\u9700\u8981\u4f9d\u8d56\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u3002 \u9700\u8981\u9884\u5148\u5728\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u914d\u7f6e\u5bf9\u5e94\u7684\u7f51\u5173\u3001Vlan \u548c\u5b89\u5168\u7b56\u7565\u7b49\u914d\u7f6e\u3002 \u5bf9\u4e8e OpenStack \u7684 VM \u73af\u5883\uff0c\u9700\u8981\u5c06\u5bf9\u5e94\u7f51\u7edc\u7aef\u53e3\u7684 PortSecurity \u5173\u95ed\u3002 \u5bf9\u4e8e VMware \u7684 vSwitch \u7f51\u7edc\uff0c\u9700\u8981\u5c06 MAC Address Changes , Forged Transmits \u548c Promiscuous Mode Operation \u8bbe\u7f6e\u4e3a allow \u3002 \u5bf9\u4e8e Hyper-V \u865a\u62df\u5316\uff0c\u9700\u8981\u5f00\u542f\u865a\u62df\u673a\u7f51\u5361\u9ad8\u7ea7\u529f\u80fd\u4e2d\u7684 MAC Address Spoofing \u3002 \u516c\u6709\u4e91\uff0c\u4f8b\u5982 AWS\u3001GCE\u3001\u963f\u91cc\u4e91\u7b49\u7531\u4e8e\u4e0d\u652f\u6301\u7528\u6237\u81ea\u5b9a\u4e49 Mac \u65e0\u6cd5\u652f\u6301 Macvlan \u6a21\u5f0f\u7f51\u7edc\u3002 \u7531\u4e8e Macvlan \u672c\u8eab\u7684\u9650\u5236\uff0cMacvlan \u5b50\u63a5\u53e3\u65e0\u6cd5\u8bbf\u95ee\u7236\u63a5\u53e3\u5730\u5740\u3002 \u5982\u679c\u7269\u7406\u7f51\u5361\u5bf9\u5e94\u4ea4\u6362\u673a\u63a5\u53e3\u4e3a Trunk \u6a21\u5f0f\uff0c\u9700\u8981\u5728\u8be5\u7f51\u5361\u4e0a\u521b\u5efa\u5b50\u63a5\u53e3\u518d\u63d0\u4f9b\u7ed9 Macvlan \u4f7f\u7528\u3002 \u5f00\u542f VPC \u7f51\u5173\u529f\u80fd \u00b6 VPC \u7f51\u5173\u529f\u80fd\u9700\u8981\u901a\u8fc7 kube-system \u4e0b\u7684 ovn-vpc-nat-gw-config \u5f00\u542f\uff1a --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-config namespace : kube-system data : image : 'docker.io/kubeovn/vpc-nat-gateway:v1.12.0' --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-gw-config namespace : kube-system data : enable-vpc-nat-gw : 'true' image : \u7f51\u5173 Pod \u6240\u4f7f\u7528\u7684\u955c\u50cf\u3002 enable-vpc-nat-gw \uff1a \u63a7\u5236\u662f\u5426\u542f\u7528 VPC \u7f51\u5173\u529f\u80fd\u3002 \u521b\u5efa VPC \u7f51\u5173\u5e76\u914d\u7f6e\u9ed8\u8ba4\u8def\u7531 \u00b6 kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" externalSubnets : - ovn-vpc-external-network vpc \uff1a\u8be5 VpcNatGateway \u6240\u5c5e\u7684 VPC\u3002 subnet \uff1a\u4e3a VPC \u5185\u67d0\u4e2a Subnet \u540d\uff0cVPC \u7f51\u5173 Pod \u4f1a\u5728\u8be5\u5b50\u7f51\u4e0b\u7528 lanIp \u6765\u8fde\u63a5\u79df\u6237\u7f51\u7edc\u3002 lanIp \uff1a subnet \u5185\u67d0\u4e2a\u672a\u88ab\u4f7f\u7528\u7684 IP\uff0cVPC \u7f51\u5173 Pod \u6700\u7ec8\u4f1a\u4f7f\u7528\u8be5 Pod\u3002\u5f53 VPC \u914d\u7f6e\u8def\u7531\u9700\u8981\u6307\u5411\u5f53\u524d VpcNatGateway \u65f6 nextHopIP \u9700\u8981\u8bbe\u7f6e\u4e3a\u8fd9\u4e2a lanIp \u3002 selector \uff1aVpcNatGateway Pod \u7684\u8282\u70b9\u9009\u62e9\u5668\uff0c\u683c\u5f0f\u548c Kubernetes \u4e2d\u7684 NodeSelector \u683c\u5f0f\u76f8\u540c\u3002 externalSubnets \uff1a VPC \u7f51\u5173\u4f7f\u7528\u7684\u5916\u90e8\u7f51\u7edc\uff0c\u5982\u679c\u4e0d\u914d\u7f6e\u5219\u9ed8\u8ba4\u4f7f\u7528 ovn-vpc-external-network \uff0c\u5f53\u524d\u7248\u672c\u53ea\u652f\u6301\u914d\u7f6e\u4e00\u4e2a\u5916\u90e8\u7f51\u7edc\u3002 \u5176\u4ed6\u53ef\u914d\u53c2\u6570\uff1a tolerations : \u4e3a VPC \u7f51\u5173\u914d\u7f6e\u5bb9\u5fcd\u5ea6\uff0c\u5177\u4f53\u914d\u7f6e\u53c2\u8003 \u6c61\u70b9\u548c\u5bb9\u5fcd\u5ea6 \u3002 affinity : \u4e3a VPC \u7f51\u5173 Pod \u6216\u8282\u70b9\u914d\u7f6e\u4eb2\u548c\u6027\uff0c\u5177\u4f53\u8bbe\u7f6e\u53c2\u8003 \u4eb2\u548c\u6027\u4e0e\u53cd\u4eb2\u548c\u6027 \u3002 \u521b\u5efa EIP \u00b6 EIP \u4e3a\u5916\u90e8\u7f51\u7edc\u6bb5\u7684\u67d0\u4e2a IP \u5206\u914d\u7ed9 VPC \u7f51\u5173\u540e\u53ef\u8fdb\u884c\u6d6e\u52a8 IP\uff0cSNAT \u548c DNAT \u64cd\u4f5c\u3002 \u968f\u673a\u5206\u914d\u4e00\u4e2a\u5730\u5740\u7ed9 EIP\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 \u56fa\u5b9a EIP \u5730\u5740\u5206\u914d\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : natGwDp : gw1 v4ip : 10.0.1.111 \u6307\u5b9a EIP \u6240\u5728\u7684\u5916\u90e8\u7f51\u7edc\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 externalSubnet : ovn-vpc-external-network externalSubnet \uff1a EIP \u6240\u5728\u5916\u90e8\u7f51\u7edc\u540d\u79f0\uff0c\u5982\u679c\u4e0d\u6307\u5b9a\u5219\u9ed8\u8ba4\u4e3a ovn-vpc-external-network \uff0c\u5982\u679c\u6307\u5b9a\u5219\u5fc5\u987b\u4e3a\u6240\u5728 VPC \u7f51\u5173\u7684 externalSubnets \u4e2d\u7684\u4e00\u4e2a\u3002 \u521b\u5efa DNAT \u89c4\u5219 \u00b6 kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipd01 spec : natGwDp : gw1 --- kind : IptablesDnatRule apiVersion : kubeovn.io/v1 metadata : name : dnat01 spec : eip : eipd01 externalPort : '8888' internalIp : 10.0.1.10 internalPort : '80' protocol : tcp \u521b\u5efa SNAT \u89c4\u5219 \u00b6 --- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eips01 spec : natGwDp : gw1 --- kind : IptablesSnatRule apiVersion : kubeovn.io/v1 metadata : name : snat01 spec : eip : eips01 internalCIDR : 10.0.1.0/24 \u521b\u5efa\u6d6e\u52a8 IP \u00b6 --- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipf01 spec : natGwDp : gw1 --- kind : IptablesFIPRule apiVersion : kubeovn.io/v1 metadata : name : fip01 spec : eip : eipf01 internalIp : 10.0.1.5 \u81ea\u5b9a\u4e49\u8def\u7531 \u00b6 \u5728\u81ea\u5b9a\u4e49 VPC \u5185\uff0c\u7528\u6237\u53ef\u4ee5\u81ea\u5b9a\u4e49\u7f51\u7edc\u5185\u90e8\u7684\u8def\u7531\u89c4\u5219\uff0c\u7ed3\u5408\u7f51\u5173\u5b9e\u73b0\u66f4\u7075\u6d3b\u7684\u8f6c\u53d1\u3002 Kube-OVN \u652f\u6301\u9759\u6001\u8def\u7531\u548c\u66f4\u4e3a\u7075\u6d3b\u7684\u7b56\u7565\u8def\u7531\u3002 \u9759\u6001\u8def\u7531 \u00b6 kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : staticRoutes : - cidr : 0.0.0.0/0 nextHopIP : 10.0.1.254 policy : policyDst - cidr : 172.31.0.0/24 nextHopIP : 10.0.1.253 policy : policySrc routeTable : \"rtb1\" policy : \u652f\u6301\u76ee\u7684\u5730\u5740\u8def\u7531 policyDst \u548c\u6e90\u5730\u5740\u8def\u7531 policySrc \u3002 \u5f53\u8def\u7531\u89c4\u5219\u5b58\u5728\u91cd\u53e0\u65f6\uff0cCIDR \u63a9\u7801\u8f83\u957f\u7684\u89c4\u5219\u4f18\u5148\u7ea7\u66f4\u9ad8\uff0c\u82e5\u63a9\u7801\u957f\u5ea6\u76f8\u540c\u5219\u76ee\u7684\u5730\u5740\u8def\u7531\u4f18\u5148\u4e8e\u6e90\u5730\u5740\u8def\u7531\u3002 routeTable : \u53ef\u6307\u5b9a\u9759\u6001\u8def\u7531\u6240\u5728\u7684\u8def\u7531\u8868\uff0c\u9ed8\u8ba4\u5728\u4e3b\u8def\u7531\u8868\u3002\u5b50\u7f51\u5173\u8054\u8def\u7531\u8868\u8bf7\u53c2\u8003 \u521b\u5efa\u5b50\u7f51 \u7b56\u7565\u8def\u7531 \u00b6 \u9488\u5bf9\u9759\u6001\u8def\u7531\u5339\u914d\u7684\u6d41\u91cf\uff0c\u53ef\u901a\u8fc7\u7b56\u7565\u8def\u7531\u8fdb\u884c\u66f4\u7ec6\u7c92\u5ea6\u7684\u63a7\u5236\u3002\u7b56\u7565\u8def\u7531\u63d0\u4f9b\u4e86\u66f4\u7cbe\u786e\u7684\u5339\u914d\u89c4\u5219\uff0c\u4f18\u5148\u7ea7\u63a7\u5236 \u548c\u66f4\u591a\u7684\u8f6c\u53d1\u52a8\u4f5c\u3002\u8be5\u529f\u80fd\u4e3a OVN \u5185\u90e8\u903b\u8f91\u8def\u7531\u5668\u7b56\u7565\u529f\u80fd\u7684\u4e00\u4e2a\u5bf9\u5916\u66b4\u9732\uff0c\u66f4\u591a\u4f7f\u7528\u4fe1\u606f\u8bf7\u53c2\u8003 Logical Router Policy \u3002 \u7b80\u5355\u793a\u4f8b\u5982\u4e0b\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : policyRoutes : - action : drop match : ip4.src==10.0.1.0/24 && ip4.dst==10.0.1.250 priority : 11 - action : reroute match : ip4.src==10.0.1.0/24 nextHopIP : 10.0.1.252 priority : 10 \u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219 \u00b6 Kubernetes \u672c\u8eab\u63d0\u4f9b\u7684 Service \u80fd\u529b\u53ef\u4ee5\u5b8c\u6210\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u529f\u80fd\uff0c\u4f46\u662f\u53d7\u9650\u4e8e Kubernetes \u5b9e\u73b0\uff0c Service \u7684 IP \u5730\u5740\u662f\u5168\u5c40\u5206\u914d\u4e14\u4e0d\u80fd\u91cd\u590d\u3002\u5bf9\u4e8e VPC \u7684\u4f7f\u7528\u573a\u666f\uff0c\u7528\u6237\u5e0c\u671b\u80fd\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740 \u8303\u56f4\uff0c\u4e0d\u540c VPC \u4e0b\u7684\u8d1f\u8f7d\u5747\u8861\u5730\u5740\u53ef\u80fd\u91cd\u53e0\uff0cKubernetes \u5185\u7f6e\u7684 Service \u529f\u80fd\u65e0\u6cd5\u5b8c\u5168\u6ee1\u8db3\u3002 \u9488\u5bf9\u8fd9\u7c7b\u573a\u666f\uff0cKube-OVN \u63d0\u4f9b\u4e86 SwitchLBRule \u8d44\u6e90\uff0c\u7528\u6237\u53ef\u4ee5\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740\u8303\u56f4\u3002 \u4e00\u4e2a `SwitchLBRule`` \u4f8b\u5b50\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP vip \uff1a\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740\u3002 namespace \uff1a\u8d1f\u8f7d\u5747\u8861\u5668\u540e\u7aef Pod \u6240\u5728\u7684 Namespace\u3002 sessionAffinity \uff1a\u548c Service \u7684 sessionAffinity \u529f\u80fd\u76f8\u540c\u3002 selector \uff1a\u548c Service \u7684 selector \u529f\u80fd\u76f8\u540c\u3002 ports \uff1a\u548c Service \u7684 port \u529f\u80fd\u76f8\u540c\u3002 \u67e5\u770b\u90e8\u7f72\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u5668\u89c4\u5219\uff1a # kubectl get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh2 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh2 88m \u81ea\u5b9a\u4e49 vpc-dns \u00b6 \u7531\u4e8e\u81ea\u5b9a\u4e49 VPC \u548c\u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0cVPC \u5185 Pod \u65e0\u6cd5\u4f7f\u7528\u9ed8\u8ba4\u7684 coredns \u670d\u52a1\u8fdb\u884c\u57df\u540d\u89e3\u6790\u3002 \u5982\u679c\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 coredns \u89e3\u6790\u96c6\u7fa4\u5185 Service \u57df\u540d\uff0c\u53ef\u4ee5\u901a\u8fc7 Kube-OVN \u63d0\u4f9b\u7684 vpc-dns \u8d44\u6e90\u6765\u5b9e\u73b0\u3002 \u521b\u5efa\u9644\u52a0\u7f51\u5361 \u00b6 apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }' \u4fee\u6539 ovn-default \u903b\u8f91\u4ea4\u6362\u673a\u7684 provider \u00b6 \u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn \uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn vpc : ovn-cluster \u914d\u7f6e vpc-dns \u7684 ConfigMap \u00b6 \u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : true nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\uff08\u53ef\u7f3a\u7701\uff09 true \u542f\u7528\u529f\u80fd\uff0c false \u5173\u95ed\u529f\u80fd\u3002\u9ed8\u8ba4 true \u3002 coredns-image \uff1a\uff08\u53ef\u7701\u7565\uff09\uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-template \uff1a\uff08\u53ef\u7701\u7565\uff09\uff1adns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\uff1a\u5f53\u524d\u7248\u672c\u4ed3\u5e93\u91cc\u7684 yamls/coredns-template.yaml \u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\uff08\u53ef\u7f3a\u7701\uff09 \u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\u3002 k8s-service-port \uff1a\uff08\u53ef\u7f3a\u7701\uff09\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\u3002 \u90e8\u7f72 vpc-dns \u4f9d\u8d56\u8d44\u6e90 \u00b6 apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u90e8\u7f72 vpc-dns \u00b6 kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u8d44\u6e90\u4fe1\u606f\uff1a [ root@hci-dev-mst-1 kubeovn ] # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u6210\u529f\u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72 \u9650\u5236 \u00b6 \u4e00\u4e2a vpc \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a vpc \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a vpc \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC \u4f7f\u7528"},{"location":"guide/vpc/#vpc","text":"Kube-OVN \u652f\u6301\u591a\u79df\u6237\u9694\u79bb\u7ea7\u522b\u7684 VPC \u7f51\u7edc\u3002\u4e0d\u540c VPC \u7f51\u7edc\u76f8\u4e92\u72ec\u7acb\uff0c\u53ef\u4ee5\u5206\u522b\u914d\u7f6e Subnet \u7f51\u6bb5\uff0c \u8def\u7531\u7b56\u7565\uff0c\u5b89\u5168\u7b56\u7565\uff0c\u51fa\u7f51\u7f51\u5173\uff0cEIP \u7b49\u914d\u7f6e\u3002 VPC \u4e3b\u8981\u7528\u4e8e\u6709\u591a\u79df\u6237\u7f51\u7edc\u5f3a\u9694\u79bb\u7684\u573a\u666f\uff0c\u90e8\u5206 Kubernetes \u7f51\u7edc\u529f\u80fd\u5728\u591a\u79df\u6237\u7f51\u7edc\u4e0b\u5b58\u5728\u51b2\u7a81\u3002 \u4f8b\u5982\u8282\u70b9\u548c Pod \u4e92\u8bbf\uff0cNodePort \u529f\u80fd\uff0c\u57fa\u4e8e\u7f51\u7edc\u8bbf\u95ee\u7684\u5065\u5eb7\u68c0\u67e5\u548c DNS \u80fd\u529b\u5728\u591a\u79df\u6237\u7f51\u7edc\u573a\u666f\u6682\u4e0d\u652f\u6301\u3002 \u4e3a\u4e86\u65b9\u4fbf\u5e38\u89c1 Kubernetes \u7684\u4f7f\u7528\u573a\u666f\uff0cKube-OVN \u9ed8\u8ba4 VPC \u505a\u4e86\u7279\u6b8a\u8bbe\u8ba1\uff0c\u8be5 VPC \u4e0b\u7684 Subnet \u53ef\u4ee5\u6ee1\u8db3 Kubernetes \u89c4\u8303\u3002\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u652f\u6301\u672c\u6587\u6863\u4ecb\u7ecd\u7684\u9759\u6001\u8def\u7531\uff0cEIP \u548c NAT \u7f51\u5173\u7b49\u529f\u80fd\u3002 \u5e38\u89c1\u9694\u79bb\u9700\u6c42\u53ef\u901a\u8fc7\u9ed8\u8ba4 VPC \u4e0b\u7684\u7f51\u7edc\u7b56\u7565\u548c\u5b50\u7f51 ACL \u5b9e\u73b0\uff0c\u5728\u4f7f\u7528\u81ea\u5b9a\u4e49 VPC \u524d\u8bf7\u660e\u786e\u662f\u5426\u9700\u8981 VPC \u7ea7\u522b\u7684\u9694\u79bb\uff0c\u5e76\u4e86\u89e3\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684\u9650\u5236\u3002 \u5728 Underlay \u7f51\u7edc\u4e0b\uff0c\u7269\u7406\u4ea4\u6362\u673a\u8d1f\u8d23\u6570\u636e\u9762\u8f6c\u53d1\uff0cVPC \u65e0\u6cd5\u5bf9 Underlay \u5b50\u7f51\u8fdb\u884c\u9694\u79bb\u3002","title":"VPC \u4f7f\u7528"},{"location":"guide/vpc/#vpc_1","text":"\u521b\u5efa\u4e24\u4e2a VPC\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : namespaces : - ns1 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-2 spec : namespaces : - ns2 namespaces \u53ef\u4ee5\u9650\u5b9a\u53ea\u6709\u54ea\u4e9b Namespace \u53ef\u4ee5\u4f7f\u7528\u5f53\u524d VPC\uff0c\u82e5\u4e3a\u7a7a\u5219\u4e0d\u9650\u5b9a\u3002 \u521b\u5efa\u4e24\u4e2a\u5b50\u7f51\uff0c\u5206\u5c5e\u4e24\u4e2a\u4e0d\u540c\u7684 VPC \u5e76\u6709\u76f8\u540c\u7684 CIDR: kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : test-vpc-1 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns1 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : test-vpc-2 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns2 \u5206\u522b\u5728\u4e24\u4e2a Namespace \u4e0b\u521b\u5efa Pod: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 namespace : ns1 name : vpc1-pod spec : containers : - name : vpc1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 namespace : ns2 name : vpc2-pod spec : containers : - name : vpc2-pod image : docker.io/library/nginx:alpine \u8fd0\u884c\u6210\u529f\u540e\u53ef\u89c2\u5bdf\u4e24\u4e2a Pod \u5730\u5740\u5c5e\u4e8e\u540c\u4e00\u4e2a CIDR\uff0c\u4f46\u7531\u4e8e\u8fd0\u884c\u5728\u4e0d\u540c\u7684\u79df\u6237 VPC\uff0c\u4e24\u4e2a Pod \u65e0\u6cd5\u76f8\u4e92\u8bbf\u95ee\u3002","title":"\u521b\u5efa\u81ea\u5b9a\u4e49 VPC"},{"location":"guide/vpc/#vpc-pod-livenessprobe-readinessprobe","text":"\u7531\u4e8e\u5e38\u89c4\u914d\u7f6e\u4e0b\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 Pod \u548c\u8282\u70b9\u7684\u7f51\u7edc\u4e4b\u95f4\u5e76\u4e0d\u4e92\u901a\uff0c\u6240\u4ee5 kubelet \u53d1\u9001\u7684\u63a2\u6d4b\u62a5\u6587\u65e0\u6cd5\u5230\u8fbe\u81ea\u5b9a VPC \u5185\u7684 Pod\u3002Kube-OVN \u901a\u8fc7 TProxy \u5c06 kubelet \u53d1\u9001\u7684\u63a2\u6d4b\u62a5\u6587\u91cd\u5b9a\u5411\u5230\u81ea\u5b9a\u4e49 VPC \u5185\u7684 Pod\uff0c\u4ece\u800c\u5b9e\u73b0\u8fd9\u4e00\u529f\u80fd\u3002 \u914d\u7f6e\u65b9\u6cd5\u5982\u4e0b\uff0c\u5728 Daemonset kube-ovn-cni \u4e2d\u589e\u52a0\u53c2\u6570 --enable-tproxy=true \uff1a spec : template : spec : containers : - args : - --enable-tproxy=true \u8be5\u529f\u80fd\u9650\u5236\u6761\u4ef6\uff1a \u5f53\u540c\u4e00\u4e2a\u8282\u70b9\u4e0b\u51fa\u73b0\u4e0d\u540c VPC \u4e0b\u7684 Pod \u5177\u6709\u76f8\u540c\u7684 IP\uff0c\u63a2\u6d4b\u529f\u80fd\u5931\u6548\u3002 \u76ee\u524d\u6682\u65f6\u53ea\u652f\u6301 tcpSocket \u548c httpGet \u4e24\u79cd\u63a2\u6d4b\u65b9\u5f0f\u3002","title":"\u81ea\u5b9a\u4e49 VPC Pod \u652f\u6301 livenessProbe \u548c readinessProbe"},{"location":"guide/vpc/#vpc_2","text":"\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684\u5b50\u7f51\u4e0d\u652f\u6301\u9ed8\u8ba4 VPC \u4e0b\u7684\u5206\u5e03\u5f0f\u7f51\u5173\u548c\u96c6\u4e2d\u5f0f\u7f51\u5173\u3002 VPC \u5185\u5bb9\u5668\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u9700\u8981\u901a\u8fc7 VPC \u7f51\u5173\uff0cVPC \u7f51\u5173\u53ef\u4ee5\u6253\u901a\u7269\u7406\u7f51\u7edc\u548c\u79df\u6237\u7f51\u7edc\uff0c\u5e76\u63d0\u4f9b \u6d6e\u52a8 IP\uff0cSNAT \u548c DNAT \u529f\u80fd\u3002 VPC \u7f51\u5173\u529f\u80fd\u4f9d\u8d56 Multus-CNI \u7684\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5b89\u88c5\u8bf7\u53c2\u8003 multus-cni \u3002","title":"\u521b\u5efa VPC \u7f51\u5173"},{"location":"guide/vpc/#_1","text":"apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-vpc-external-network spec : protocol : IPv4 provider : ovn-vpc-external-network.kube-system cidrBlock : 192.168.0.0/24 gateway : 192.168.0.1 # IP address of the physical gateway excludeIps : - 192.168.0.1..192.168.0.10 --- apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-vpc-external-network namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth1\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-vpc-external-network.kube-system\" } }' \u8be5 Subnet \u7528\u6765\u7ba1\u7406\u53ef\u7528\u7684\u5916\u90e8\u5730\u5740\uff0c\u7f51\u6bb5\u5185\u7684\u5730\u5740\u5c06\u4f1a\u901a\u8fc7 Macvlan \u5206\u914d\u7ed9 VPC \u7f51\u5173\uff0c\u8bf7\u548c\u7f51\u7edc\u7ba1\u7406\u6c9f\u901a\u7ed9\u51fa\u53ef\u7528\u7684\u7269\u7406\u6bb5 IP\u3002 VPC \u7f51\u5173\u4f7f\u7528 Macvlan \u505a\u7269\u7406\u7f51\u7edc\u914d\u7f6e\uff0c NetworkAttachmentDefinition \u7684 master \u9700\u4e3a\u5bf9\u5e94\u7269\u7406\u7f51\u8def\u7f51\u5361\u7684\u7f51\u5361\u540d\u3002 name \u5916\u90e8\u7f51\u7edc\u540d\u79f0\u3002 \u5728 Macvlan \u6a21\u5f0f\u4e0b\uff0c\u9644\u5c5e\u7f51\u5361\u4f1a\u5c06\u6570\u636e\u5305\u76f4\u63a5\u901a\u8fc7\u8be5\u8282\u70b9\u7f51\u5361\u5bf9\u5916\u53d1\u9001\uff0cL2/L3 \u5c42\u9762\u7684\u8f6c\u53d1\u80fd\u529b\u9700\u8981\u4f9d\u8d56\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u3002 \u9700\u8981\u9884\u5148\u5728\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u914d\u7f6e\u5bf9\u5e94\u7684\u7f51\u5173\u3001Vlan \u548c\u5b89\u5168\u7b56\u7565\u7b49\u914d\u7f6e\u3002 \u5bf9\u4e8e OpenStack \u7684 VM \u73af\u5883\uff0c\u9700\u8981\u5c06\u5bf9\u5e94\u7f51\u7edc\u7aef\u53e3\u7684 PortSecurity \u5173\u95ed\u3002 \u5bf9\u4e8e VMware \u7684 vSwitch \u7f51\u7edc\uff0c\u9700\u8981\u5c06 MAC Address Changes , Forged Transmits \u548c Promiscuous Mode Operation \u8bbe\u7f6e\u4e3a allow \u3002 \u5bf9\u4e8e Hyper-V \u865a\u62df\u5316\uff0c\u9700\u8981\u5f00\u542f\u865a\u62df\u673a\u7f51\u5361\u9ad8\u7ea7\u529f\u80fd\u4e2d\u7684 MAC Address Spoofing \u3002 \u516c\u6709\u4e91\uff0c\u4f8b\u5982 AWS\u3001GCE\u3001\u963f\u91cc\u4e91\u7b49\u7531\u4e8e\u4e0d\u652f\u6301\u7528\u6237\u81ea\u5b9a\u4e49 Mac \u65e0\u6cd5\u652f\u6301 Macvlan \u6a21\u5f0f\u7f51\u7edc\u3002 \u7531\u4e8e Macvlan \u672c\u8eab\u7684\u9650\u5236\uff0cMacvlan \u5b50\u63a5\u53e3\u65e0\u6cd5\u8bbf\u95ee\u7236\u63a5\u53e3\u5730\u5740\u3002 \u5982\u679c\u7269\u7406\u7f51\u5361\u5bf9\u5e94\u4ea4\u6362\u673a\u63a5\u53e3\u4e3a Trunk \u6a21\u5f0f\uff0c\u9700\u8981\u5728\u8be5\u7f51\u5361\u4e0a\u521b\u5efa\u5b50\u63a5\u53e3\u518d\u63d0\u4f9b\u7ed9 Macvlan \u4f7f\u7528\u3002","title":"\u914d\u7f6e\u5916\u90e8\u7f51\u7edc"},{"location":"guide/vpc/#vpc_3","text":"VPC \u7f51\u5173\u529f\u80fd\u9700\u8981\u901a\u8fc7 kube-system \u4e0b\u7684 ovn-vpc-nat-gw-config \u5f00\u542f\uff1a --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-config namespace : kube-system data : image : 'docker.io/kubeovn/vpc-nat-gateway:v1.12.0' --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-gw-config namespace : kube-system data : enable-vpc-nat-gw : 'true' image : \u7f51\u5173 Pod \u6240\u4f7f\u7528\u7684\u955c\u50cf\u3002 enable-vpc-nat-gw \uff1a \u63a7\u5236\u662f\u5426\u542f\u7528 VPC \u7f51\u5173\u529f\u80fd\u3002","title":"\u5f00\u542f VPC \u7f51\u5173\u529f\u80fd"},{"location":"guide/vpc/#vpc_4","text":"kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" externalSubnets : - ovn-vpc-external-network vpc \uff1a\u8be5 VpcNatGateway \u6240\u5c5e\u7684 VPC\u3002 subnet \uff1a\u4e3a VPC \u5185\u67d0\u4e2a Subnet \u540d\uff0cVPC \u7f51\u5173 Pod \u4f1a\u5728\u8be5\u5b50\u7f51\u4e0b\u7528 lanIp \u6765\u8fde\u63a5\u79df\u6237\u7f51\u7edc\u3002 lanIp \uff1a subnet \u5185\u67d0\u4e2a\u672a\u88ab\u4f7f\u7528\u7684 IP\uff0cVPC \u7f51\u5173 Pod \u6700\u7ec8\u4f1a\u4f7f\u7528\u8be5 Pod\u3002\u5f53 VPC \u914d\u7f6e\u8def\u7531\u9700\u8981\u6307\u5411\u5f53\u524d VpcNatGateway \u65f6 nextHopIP \u9700\u8981\u8bbe\u7f6e\u4e3a\u8fd9\u4e2a lanIp \u3002 selector \uff1aVpcNatGateway Pod \u7684\u8282\u70b9\u9009\u62e9\u5668\uff0c\u683c\u5f0f\u548c Kubernetes \u4e2d\u7684 NodeSelector \u683c\u5f0f\u76f8\u540c\u3002 externalSubnets \uff1a VPC \u7f51\u5173\u4f7f\u7528\u7684\u5916\u90e8\u7f51\u7edc\uff0c\u5982\u679c\u4e0d\u914d\u7f6e\u5219\u9ed8\u8ba4\u4f7f\u7528 ovn-vpc-external-network \uff0c\u5f53\u524d\u7248\u672c\u53ea\u652f\u6301\u914d\u7f6e\u4e00\u4e2a\u5916\u90e8\u7f51\u7edc\u3002 \u5176\u4ed6\u53ef\u914d\u53c2\u6570\uff1a tolerations : \u4e3a VPC \u7f51\u5173\u914d\u7f6e\u5bb9\u5fcd\u5ea6\uff0c\u5177\u4f53\u914d\u7f6e\u53c2\u8003 \u6c61\u70b9\u548c\u5bb9\u5fcd\u5ea6 \u3002 affinity : \u4e3a VPC \u7f51\u5173 Pod \u6216\u8282\u70b9\u914d\u7f6e\u4eb2\u548c\u6027\uff0c\u5177\u4f53\u8bbe\u7f6e\u53c2\u8003 \u4eb2\u548c\u6027\u4e0e\u53cd\u4eb2\u548c\u6027 \u3002","title":"\u521b\u5efa VPC \u7f51\u5173\u5e76\u914d\u7f6e\u9ed8\u8ba4\u8def\u7531"},{"location":"guide/vpc/#eip","text":"EIP \u4e3a\u5916\u90e8\u7f51\u7edc\u6bb5\u7684\u67d0\u4e2a IP \u5206\u914d\u7ed9 VPC \u7f51\u5173\u540e\u53ef\u8fdb\u884c\u6d6e\u52a8 IP\uff0cSNAT \u548c DNAT \u64cd\u4f5c\u3002 \u968f\u673a\u5206\u914d\u4e00\u4e2a\u5730\u5740\u7ed9 EIP\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 \u56fa\u5b9a EIP \u5730\u5740\u5206\u914d\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : natGwDp : gw1 v4ip : 10.0.1.111 \u6307\u5b9a EIP \u6240\u5728\u7684\u5916\u90e8\u7f51\u7edc\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 externalSubnet : ovn-vpc-external-network externalSubnet \uff1a EIP \u6240\u5728\u5916\u90e8\u7f51\u7edc\u540d\u79f0\uff0c\u5982\u679c\u4e0d\u6307\u5b9a\u5219\u9ed8\u8ba4\u4e3a ovn-vpc-external-network \uff0c\u5982\u679c\u6307\u5b9a\u5219\u5fc5\u987b\u4e3a\u6240\u5728 VPC \u7f51\u5173\u7684 externalSubnets \u4e2d\u7684\u4e00\u4e2a\u3002","title":"\u521b\u5efa EIP"},{"location":"guide/vpc/#dnat","text":"kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipd01 spec : natGwDp : gw1 --- kind : IptablesDnatRule apiVersion : kubeovn.io/v1 metadata : name : dnat01 spec : eip : eipd01 externalPort : '8888' internalIp : 10.0.1.10 internalPort : '80' protocol : tcp","title":"\u521b\u5efa DNAT \u89c4\u5219"},{"location":"guide/vpc/#snat","text":"--- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eips01 spec : natGwDp : gw1 --- kind : IptablesSnatRule apiVersion : kubeovn.io/v1 metadata : name : snat01 spec : eip : eips01 internalCIDR : 10.0.1.0/24","title":"\u521b\u5efa SNAT \u89c4\u5219"},{"location":"guide/vpc/#ip","text":"--- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipf01 spec : natGwDp : gw1 --- kind : IptablesFIPRule apiVersion : kubeovn.io/v1 metadata : name : fip01 spec : eip : eipf01 internalIp : 10.0.1.5","title":"\u521b\u5efa\u6d6e\u52a8 IP"},{"location":"guide/vpc/#_2","text":"\u5728\u81ea\u5b9a\u4e49 VPC \u5185\uff0c\u7528\u6237\u53ef\u4ee5\u81ea\u5b9a\u4e49\u7f51\u7edc\u5185\u90e8\u7684\u8def\u7531\u89c4\u5219\uff0c\u7ed3\u5408\u7f51\u5173\u5b9e\u73b0\u66f4\u7075\u6d3b\u7684\u8f6c\u53d1\u3002 Kube-OVN \u652f\u6301\u9759\u6001\u8def\u7531\u548c\u66f4\u4e3a\u7075\u6d3b\u7684\u7b56\u7565\u8def\u7531\u3002","title":"\u81ea\u5b9a\u4e49\u8def\u7531"},{"location":"guide/vpc/#_3","text":"kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : staticRoutes : - cidr : 0.0.0.0/0 nextHopIP : 10.0.1.254 policy : policyDst - cidr : 172.31.0.0/24 nextHopIP : 10.0.1.253 policy : policySrc routeTable : \"rtb1\" policy : \u652f\u6301\u76ee\u7684\u5730\u5740\u8def\u7531 policyDst \u548c\u6e90\u5730\u5740\u8def\u7531 policySrc \u3002 \u5f53\u8def\u7531\u89c4\u5219\u5b58\u5728\u91cd\u53e0\u65f6\uff0cCIDR \u63a9\u7801\u8f83\u957f\u7684\u89c4\u5219\u4f18\u5148\u7ea7\u66f4\u9ad8\uff0c\u82e5\u63a9\u7801\u957f\u5ea6\u76f8\u540c\u5219\u76ee\u7684\u5730\u5740\u8def\u7531\u4f18\u5148\u4e8e\u6e90\u5730\u5740\u8def\u7531\u3002 routeTable : \u53ef\u6307\u5b9a\u9759\u6001\u8def\u7531\u6240\u5728\u7684\u8def\u7531\u8868\uff0c\u9ed8\u8ba4\u5728\u4e3b\u8def\u7531\u8868\u3002\u5b50\u7f51\u5173\u8054\u8def\u7531\u8868\u8bf7\u53c2\u8003 \u521b\u5efa\u5b50\u7f51","title":"\u9759\u6001\u8def\u7531"},{"location":"guide/vpc/#_4","text":"\u9488\u5bf9\u9759\u6001\u8def\u7531\u5339\u914d\u7684\u6d41\u91cf\uff0c\u53ef\u901a\u8fc7\u7b56\u7565\u8def\u7531\u8fdb\u884c\u66f4\u7ec6\u7c92\u5ea6\u7684\u63a7\u5236\u3002\u7b56\u7565\u8def\u7531\u63d0\u4f9b\u4e86\u66f4\u7cbe\u786e\u7684\u5339\u914d\u89c4\u5219\uff0c\u4f18\u5148\u7ea7\u63a7\u5236 \u548c\u66f4\u591a\u7684\u8f6c\u53d1\u52a8\u4f5c\u3002\u8be5\u529f\u80fd\u4e3a OVN \u5185\u90e8\u903b\u8f91\u8def\u7531\u5668\u7b56\u7565\u529f\u80fd\u7684\u4e00\u4e2a\u5bf9\u5916\u66b4\u9732\uff0c\u66f4\u591a\u4f7f\u7528\u4fe1\u606f\u8bf7\u53c2\u8003 Logical Router Policy \u3002 \u7b80\u5355\u793a\u4f8b\u5982\u4e0b\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : policyRoutes : - action : drop match : ip4.src==10.0.1.0/24 && ip4.dst==10.0.1.250 priority : 11 - action : reroute match : ip4.src==10.0.1.0/24 nextHopIP : 10.0.1.252 priority : 10","title":"\u7b56\u7565\u8def\u7531"},{"location":"guide/vpc/#_5","text":"Kubernetes \u672c\u8eab\u63d0\u4f9b\u7684 Service \u80fd\u529b\u53ef\u4ee5\u5b8c\u6210\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u529f\u80fd\uff0c\u4f46\u662f\u53d7\u9650\u4e8e Kubernetes \u5b9e\u73b0\uff0c Service \u7684 IP \u5730\u5740\u662f\u5168\u5c40\u5206\u914d\u4e14\u4e0d\u80fd\u91cd\u590d\u3002\u5bf9\u4e8e VPC \u7684\u4f7f\u7528\u573a\u666f\uff0c\u7528\u6237\u5e0c\u671b\u80fd\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740 \u8303\u56f4\uff0c\u4e0d\u540c VPC \u4e0b\u7684\u8d1f\u8f7d\u5747\u8861\u5730\u5740\u53ef\u80fd\u91cd\u53e0\uff0cKubernetes \u5185\u7f6e\u7684 Service \u529f\u80fd\u65e0\u6cd5\u5b8c\u5168\u6ee1\u8db3\u3002 \u9488\u5bf9\u8fd9\u7c7b\u573a\u666f\uff0cKube-OVN \u63d0\u4f9b\u4e86 SwitchLBRule \u8d44\u6e90\uff0c\u7528\u6237\u53ef\u4ee5\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740\u8303\u56f4\u3002 \u4e00\u4e2a `SwitchLBRule`` \u4f8b\u5b50\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP vip \uff1a\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740\u3002 namespace \uff1a\u8d1f\u8f7d\u5747\u8861\u5668\u540e\u7aef Pod \u6240\u5728\u7684 Namespace\u3002 sessionAffinity \uff1a\u548c Service \u7684 sessionAffinity \u529f\u80fd\u76f8\u540c\u3002 selector \uff1a\u548c Service \u7684 selector \u529f\u80fd\u76f8\u540c\u3002 ports \uff1a\u548c Service \u7684 port \u529f\u80fd\u76f8\u540c\u3002 \u67e5\u770b\u90e8\u7f72\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u5668\u89c4\u5219\uff1a # kubectl get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh2 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh2 88m","title":"\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219"},{"location":"guide/vpc/#vpc-dns","text":"\u7531\u4e8e\u81ea\u5b9a\u4e49 VPC \u548c\u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0cVPC \u5185 Pod \u65e0\u6cd5\u4f7f\u7528\u9ed8\u8ba4\u7684 coredns \u670d\u52a1\u8fdb\u884c\u57df\u540d\u89e3\u6790\u3002 \u5982\u679c\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 coredns \u89e3\u6790\u96c6\u7fa4\u5185 Service \u57df\u540d\uff0c\u53ef\u4ee5\u901a\u8fc7 Kube-OVN \u63d0\u4f9b\u7684 vpc-dns \u8d44\u6e90\u6765\u5b9e\u73b0\u3002","title":"\u81ea\u5b9a\u4e49 vpc-dns"},{"location":"guide/vpc/#_6","text":"apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }'","title":"\u521b\u5efa\u9644\u52a0\u7f51\u5361"},{"location":"guide/vpc/#ovn-default-provider","text":"\u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn \uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn vpc : ovn-cluster","title":"\u4fee\u6539 ovn-default \u903b\u8f91\u4ea4\u6362\u673a\u7684 provider"},{"location":"guide/vpc/#vpc-dns-configmap","text":"\u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : true nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\uff08\u53ef\u7f3a\u7701\uff09 true \u542f\u7528\u529f\u80fd\uff0c false \u5173\u95ed\u529f\u80fd\u3002\u9ed8\u8ba4 true \u3002 coredns-image \uff1a\uff08\u53ef\u7701\u7565\uff09\uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-template \uff1a\uff08\u53ef\u7701\u7565\uff09\uff1adns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\uff1a\u5f53\u524d\u7248\u672c\u4ed3\u5e93\u91cc\u7684 yamls/coredns-template.yaml \u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\uff08\u53ef\u7f3a\u7701\uff09 \u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\u3002 k8s-service-port \uff1a\uff08\u53ef\u7f3a\u7701\uff09\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\u3002","title":"\u914d\u7f6e vpc-dns \u7684 ConfigMap"},{"location":"guide/vpc/#vpc-dns_1","text":"apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance }","title":"\u90e8\u7f72 vpc-dns \u4f9d\u8d56\u8d44\u6e90"},{"location":"guide/vpc/#vpc-dns_2","text":"kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u8d44\u6e90\u4fe1\u606f\uff1a [ root@hci-dev-mst-1 kubeovn ] # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u6210\u529f\u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72","title":"\u90e8\u7f72 vpc-dns"},{"location":"guide/vpc/#_7","text":"\u4e00\u4e2a vpc \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a vpc \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a vpc \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9650\u5236"},{"location":"guide/webhook/","text":"Webhook \u4f7f\u7528 \u00b6 \u4f7f\u7528 Webhook \u53ef\u4ee5\u5bf9 Kube-OVN \u5185\u7684 CRD \u8d44\u6e90\u8fdb\u884c\u6821\u9a8c\uff0c\u76ee\u524d Webhook \u4e3b\u8981\u5b8c\u6210 \u56fa\u5b9a IP \u5730\u5740\u51b2\u7a81\u68c0\u6d4b\u548c Subnet CIDR \u7684\u51b2\u7a81\u68c0\u6d4b\uff0c\u5e76\u5728\u8fd9\u7c7b\u8d44\u6e90\u521b\u5efa\u51b2\u7a81\u65f6\u63d0\u793a\u9519\u8bef\u3002 \u7531\u4e8e Webhook \u4f1a\u62e6\u622a\u6240\u6709\u7684 Subnet \u548c Pod \u521b\u5efa\u7684\u8bf7\u6c42\uff0c\u56e0\u6b64\u9700\u8981\u5148\u90e8\u7f72 Kube-OVN \u540e\u90e8\u7f72 Webhook \u907f\u514d\u65e0\u6cd5\u521b\u5efa Pod\u3002 Cert-Manager \u5b89\u88c5 \u00b6 Webhook \u90e8\u7f72\u9700\u8981\u76f8\u5173\u8bc1\u4e66\u52a0\u5bc6\uff0c\u6211\u4eec\u4f7f\u7528 cert-manager \u751f\u6210\u76f8\u5173\u8bc1\u4e66\uff0c\u6211\u4eec\u9700\u8981\u5728\u90e8\u7f72 Webhook \u524d\u5148\u90e8\u7f72 cert-manager\u3002 \u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u6765\u90e8\u7f72 cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml \u66f4\u591a cert-manager \u4f7f\u7528\u8bf7\u53c2\u8003 cert-manager \u6587\u6863 \u3002 \u5b89\u88c5 Webhook \u00b6 \u4e0b\u8f7d Webhook \u5bf9\u5e94\u7684 yaml \u8fdb\u884c\u5b89\u88c5: # kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/webhook.yaml deployment.apps/kube-ovn-webhook created service/kube-ovn-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/kube-ovn-webhook created certificate.cert-manager.io/kube-ovn-webhook-serving-cert created issuer.cert-manager.io/kube-ovn-webhook-selfsigned-issuer created \u9a8c\u8bc1 Webhook \u751f\u6548 \u00b6 \u67e5\u770b\u5df2\u8fd0\u884c Pod\uff0c\u5f97\u5230 Pod IP 10.16.0.15 \uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES static-7584848b74-fw9dm 1 /1 Running 0 2d13h 10 .16.0.15 kube-ovn-worker <none> \u7f16\u5199 yaml \u521b\u5efa\u76f8\u540c IP \u7684 Pod\uff1a apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 labels : app : static managedFields : name : staticip-pod namespace : default spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u4f7f\u7528\u4ee5\u4e0a yaml \u521b\u5efa\u9759\u6001\u5730\u5740 Pod \u7684\u65f6\u5019\uff0c\u63d0\u793a IP \u5730\u5740\u51b2\u7a81\uff1a # kubectl apply -f pod-static.yaml Error from server ( annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 ) : error when creating \"pod-static.yaml\" : admission webhook \"pod-ip-validaing.kube-ovn.io\" denied the request: annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Webhook \u4f7f\u7528"},{"location":"guide/webhook/#webhook","text":"\u4f7f\u7528 Webhook \u53ef\u4ee5\u5bf9 Kube-OVN \u5185\u7684 CRD \u8d44\u6e90\u8fdb\u884c\u6821\u9a8c\uff0c\u76ee\u524d Webhook \u4e3b\u8981\u5b8c\u6210 \u56fa\u5b9a IP \u5730\u5740\u51b2\u7a81\u68c0\u6d4b\u548c Subnet CIDR \u7684\u51b2\u7a81\u68c0\u6d4b\uff0c\u5e76\u5728\u8fd9\u7c7b\u8d44\u6e90\u521b\u5efa\u51b2\u7a81\u65f6\u63d0\u793a\u9519\u8bef\u3002 \u7531\u4e8e Webhook \u4f1a\u62e6\u622a\u6240\u6709\u7684 Subnet \u548c Pod \u521b\u5efa\u7684\u8bf7\u6c42\uff0c\u56e0\u6b64\u9700\u8981\u5148\u90e8\u7f72 Kube-OVN \u540e\u90e8\u7f72 Webhook \u907f\u514d\u65e0\u6cd5\u521b\u5efa Pod\u3002","title":"Webhook \u4f7f\u7528"},{"location":"guide/webhook/#cert-manager","text":"Webhook \u90e8\u7f72\u9700\u8981\u76f8\u5173\u8bc1\u4e66\u52a0\u5bc6\uff0c\u6211\u4eec\u4f7f\u7528 cert-manager \u751f\u6210\u76f8\u5173\u8bc1\u4e66\uff0c\u6211\u4eec\u9700\u8981\u5728\u90e8\u7f72 Webhook \u524d\u5148\u90e8\u7f72 cert-manager\u3002 \u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u6765\u90e8\u7f72 cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml \u66f4\u591a cert-manager \u4f7f\u7528\u8bf7\u53c2\u8003 cert-manager \u6587\u6863 \u3002","title":"Cert-Manager \u5b89\u88c5"},{"location":"guide/webhook/#webhook_1","text":"\u4e0b\u8f7d Webhook \u5bf9\u5e94\u7684 yaml \u8fdb\u884c\u5b89\u88c5: # kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/webhook.yaml deployment.apps/kube-ovn-webhook created service/kube-ovn-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/kube-ovn-webhook created certificate.cert-manager.io/kube-ovn-webhook-serving-cert created issuer.cert-manager.io/kube-ovn-webhook-selfsigned-issuer created","title":"\u5b89\u88c5 Webhook"},{"location":"guide/webhook/#webhook_2","text":"\u67e5\u770b\u5df2\u8fd0\u884c Pod\uff0c\u5f97\u5230 Pod IP 10.16.0.15 \uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES static-7584848b74-fw9dm 1 /1 Running 0 2d13h 10 .16.0.15 kube-ovn-worker <none> \u7f16\u5199 yaml \u521b\u5efa\u76f8\u540c IP \u7684 Pod\uff1a apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 labels : app : static managedFields : name : staticip-pod namespace : default spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u4f7f\u7528\u4ee5\u4e0a yaml \u521b\u5efa\u9759\u6001\u5730\u5740 Pod \u7684\u65f6\u5019\uff0c\u63d0\u793a IP \u5730\u5740\u51b2\u7a81\uff1a # kubectl apply -f pod-static.yaml Error from server ( annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 ) : error when creating \"pod-static.yaml\" : admission webhook \"pod-ip-validaing.kube-ovn.io\" denied the request: annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9a8c\u8bc1 Webhook \u751f\u6548"},{"location":"ops/change-default-subnet/","text":"\u4fee\u6539\u5b50\u7f51 CIDR \u00b6 \u5982\u679c\u521b\u5efa\u7684\u5b50\u7f51 CIDR \u51b2\u7a81\u6216\u4e0d\u7b26\u5408\u9884\u671f\uff0c\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u7684\u6b65\u9aa4\u8fdb\u884c\u4fee\u6539\u3002 \u4fee\u6539\u5b50\u7f51 CIDR \u540e\u4e4b\u524d\u521b\u5efa\u7684 Pod \u5c06\u65e0\u6cd5\u6b63\u5e38\u8bbf\u95ee\u7f51\u7edc\u9700\u8981\u8fdb\u884c\u91cd\u5efa\u3002 \u5efa\u8bae\u64cd\u4f5c\u524d\u614e\u91cd\u8003\u8651\u3002\u672c\u6587\u53ea\u9488\u5bf9\u4e1a\u52a1\u5b50\u7f51 CIDR \u66f4\u6539\u8fdb\u884c\u64cd\u4f5c\uff0c\u5982\u9700 \u66f4\u6539 Join \u5b50\u7f51 CIDR \u8bf7\u53c2\u8003 \u66f4\u6539 Join \u5b50\u7f51 CIDR \u3002 \u7f16\u8f91\u5b50\u7f51 \u00b6 \u4f7f\u7528 kubectl edit \u4fee\u6539\u5b50\u7f51 cidrBlock \uff0c gateway \u548c excludeIps \u3002 kubectl edit subnet test-subnet \u91cd\u5efa\u8be5\u5b50\u7f51\u7ed1\u5b9a\u7684 Namespace \u4e0b\u6240\u6709 Pod \u00b6 \u4ee5\u5b50\u7f51\u7ed1\u5b9a test Namespace \u4e3a\u4f8b\uff1a for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n test --ignore-not-found done \u82e5\u53ea\u4f7f\u7528\u4e86\u9ed8\u8ba4\u5b50\u7f51\uff0c\u53ef\u4ee5\u4f7f\u7528\u4e0b\u5217\u547d\u4ee4\u5220\u9664\u6240\u6709\u975e host \u7f51\u7edc\u6a21\u5f0f\u7684 Pod\uff1a for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done \u66f4\u6539\u9ed8\u8ba4\u5b50\u7f51\u914d\u7f6e \u00b6 \u82e5\u4fee\u6539\u7684\u4e3a\u9ed8\u8ba4\u5b50\u7f51\u7684 CIDR \u8fd8\u9700\u8981\u66f4\u6539 kube-ovn-controller Deployment \u7684\u542f\u52a8\u53c2\u6570\uff1a args : - --default-cidr=10.17.0.0/16 - --default-gateway=10.17.0.1 - --default-exclude-ips=10.17.0.1 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4fee\u6539\u5b50\u7f51 CIDR"},{"location":"ops/change-default-subnet/#cidr","text":"\u5982\u679c\u521b\u5efa\u7684\u5b50\u7f51 CIDR \u51b2\u7a81\u6216\u4e0d\u7b26\u5408\u9884\u671f\uff0c\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u7684\u6b65\u9aa4\u8fdb\u884c\u4fee\u6539\u3002 \u4fee\u6539\u5b50\u7f51 CIDR \u540e\u4e4b\u524d\u521b\u5efa\u7684 Pod \u5c06\u65e0\u6cd5\u6b63\u5e38\u8bbf\u95ee\u7f51\u7edc\u9700\u8981\u8fdb\u884c\u91cd\u5efa\u3002 \u5efa\u8bae\u64cd\u4f5c\u524d\u614e\u91cd\u8003\u8651\u3002\u672c\u6587\u53ea\u9488\u5bf9\u4e1a\u52a1\u5b50\u7f51 CIDR \u66f4\u6539\u8fdb\u884c\u64cd\u4f5c\uff0c\u5982\u9700 \u66f4\u6539 Join \u5b50\u7f51 CIDR \u8bf7\u53c2\u8003 \u66f4\u6539 Join \u5b50\u7f51 CIDR \u3002","title":"\u4fee\u6539\u5b50\u7f51 CIDR"},{"location":"ops/change-default-subnet/#_1","text":"\u4f7f\u7528 kubectl edit \u4fee\u6539\u5b50\u7f51 cidrBlock \uff0c gateway \u548c excludeIps \u3002 kubectl edit subnet test-subnet","title":"\u7f16\u8f91\u5b50\u7f51"},{"location":"ops/change-default-subnet/#namespace-pod","text":"\u4ee5\u5b50\u7f51\u7ed1\u5b9a test Namespace \u4e3a\u4f8b\uff1a for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n test --ignore-not-found done \u82e5\u53ea\u4f7f\u7528\u4e86\u9ed8\u8ba4\u5b50\u7f51\uff0c\u53ef\u4ee5\u4f7f\u7528\u4e0b\u5217\u547d\u4ee4\u5220\u9664\u6240\u6709\u975e host \u7f51\u7edc\u6a21\u5f0f\u7684 Pod\uff1a for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done","title":"\u91cd\u5efa\u8be5\u5b50\u7f51\u7ed1\u5b9a\u7684 Namespace \u4e0b\u6240\u6709 Pod"},{"location":"ops/change-default-subnet/#_2","text":"\u82e5\u4fee\u6539\u7684\u4e3a\u9ed8\u8ba4\u5b50\u7f51\u7684 CIDR \u8fd8\u9700\u8981\u66f4\u6539 kube-ovn-controller Deployment \u7684\u542f\u52a8\u53c2\u6570\uff1a args : - --default-cidr=10.17.0.0/16 - --default-gateway=10.17.0.1 - --default-exclude-ips=10.17.0.1 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u66f4\u6539\u9ed8\u8ba4\u5b50\u7f51\u914d\u7f6e"},{"location":"ops/change-join-subnet/","text":"\u4fee\u6539 Join \u5b50\u7f51 CIDR \u00b6 \u82e5\u53d1\u73b0\u521b\u5efa\u7684 Join \u5b50\u7f51 CIDR \u51b2\u7a81\u6216\u4e0d\u7b26\u5408\u9884\u671f\uff0c\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u8fdb\u884c\u4fee\u6539\u3002 \u4fee\u6539 Join \u5b50\u7f51 CIDR \u540e\u4e4b\u524d\u521b\u5efa\u7684 Pod \u5c06\u65e0\u6cd5\u6b63\u5e38\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\uff0c\u9700\u8981\u7b49\u91cd\u5efa\u5b8c\u6210, \u5efa\u8bae\u524d\u64cd\u4f5c\u65f6\u614e\u91cd\u8003\u8651\u3002 \u5220\u9664 Join \u5b50\u7f51 \u00b6 kubectl patch subnet join --type = 'json' -p '[{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\": []}]' kubectl delete subnet join \u6e05\u7406\u76f8\u5173\u5206\u914d\u4fe1\u606f \u00b6 kubectl annotate node ovn.kubernetes.io/allocated = false --all --overwrite \u4fee\u6539 Join \u5b50\u7f51\u76f8\u5173\u4fe1\u606f \u00b6 \u4fee\u6539 kube-ovn-controller \u5185 Join \u5b50\u7f51\u76f8\u5173\u4fe1\u606f\uff1a kubectl edit deployment -n kube-system kube-ovn-controller \u4fee\u6539\u4e0b\u5217\u53c2\u6570\uff1a args : - --node-switch-cidr=100.51.0.0/16 \u91cd\u542f kube-ovn-controller \u91cd\u5efa join \u5b50\u7f51\uff1a kubectl delete pod -n kube-system -lapp = kube-ovn-controller \u67e5\u770b\u65b0\u7684 Join \u5b50\u7f51\u4fe1\u606f\uff1a # kubectl get subnet NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS join ovn ovn-cluster IPv4 100 .51.0.0/16 false false false distributed 2 65531 0 0 [ \"100.51.0.1\" ] ovn-default ovn ovn-cluster IPv4 10 .17.0.0/16 false true true distributed 5 65528 0 0 [ \"10.17.0.1\" ] \u91cd\u65b0\u914d\u7f6e ovn0 \u7f51\u5361\u5730\u5740 \u00b6 \u6bcf\u4e2a\u8282\u70b9\u7684 ovn0 \u7f51\u5361\u4fe1\u606f\u9700\u8981\u91cd\u65b0\u66f4\u65b0\uff0c\u53ef\u901a\u8fc7\u91cd\u542f kube-ovn-cni \u6765\u5b8c\u6210\uff1a kubectl delete pod -n kube-system -l app = kube-ovn-cni \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4fee\u6539 Join \u5b50\u7f51 CIDR"},{"location":"ops/change-join-subnet/#join-cidr","text":"\u82e5\u53d1\u73b0\u521b\u5efa\u7684 Join \u5b50\u7f51 CIDR \u51b2\u7a81\u6216\u4e0d\u7b26\u5408\u9884\u671f\uff0c\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u8fdb\u884c\u4fee\u6539\u3002 \u4fee\u6539 Join \u5b50\u7f51 CIDR \u540e\u4e4b\u524d\u521b\u5efa\u7684 Pod \u5c06\u65e0\u6cd5\u6b63\u5e38\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\uff0c\u9700\u8981\u7b49\u91cd\u5efa\u5b8c\u6210, \u5efa\u8bae\u524d\u64cd\u4f5c\u65f6\u614e\u91cd\u8003\u8651\u3002","title":"\u4fee\u6539 Join \u5b50\u7f51 CIDR"},{"location":"ops/change-join-subnet/#join","text":"kubectl patch subnet join --type = 'json' -p '[{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\": []}]' kubectl delete subnet join","title":"\u5220\u9664 Join \u5b50\u7f51"},{"location":"ops/change-join-subnet/#_1","text":"kubectl annotate node ovn.kubernetes.io/allocated = false --all --overwrite","title":"\u6e05\u7406\u76f8\u5173\u5206\u914d\u4fe1\u606f"},{"location":"ops/change-join-subnet/#join_1","text":"\u4fee\u6539 kube-ovn-controller \u5185 Join \u5b50\u7f51\u76f8\u5173\u4fe1\u606f\uff1a kubectl edit deployment -n kube-system kube-ovn-controller \u4fee\u6539\u4e0b\u5217\u53c2\u6570\uff1a args : - --node-switch-cidr=100.51.0.0/16 \u91cd\u542f kube-ovn-controller \u91cd\u5efa join \u5b50\u7f51\uff1a kubectl delete pod -n kube-system -lapp = kube-ovn-controller \u67e5\u770b\u65b0\u7684 Join \u5b50\u7f51\u4fe1\u606f\uff1a # kubectl get subnet NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS join ovn ovn-cluster IPv4 100 .51.0.0/16 false false false distributed 2 65531 0 0 [ \"100.51.0.1\" ] ovn-default ovn ovn-cluster IPv4 10 .17.0.0/16 false true true distributed 5 65528 0 0 [ \"10.17.0.1\" ]","title":"\u4fee\u6539 Join \u5b50\u7f51\u76f8\u5173\u4fe1\u606f"},{"location":"ops/change-join-subnet/#ovn0","text":"\u6bcf\u4e2a\u8282\u70b9\u7684 ovn0 \u7f51\u5361\u4fe1\u606f\u9700\u8981\u91cd\u65b0\u66f4\u65b0\uff0c\u53ef\u901a\u8fc7\u91cd\u542f kube-ovn-cni \u6765\u5b8c\u6210\uff1a kubectl delete pod -n kube-system -l app = kube-ovn-cni \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u91cd\u65b0\u914d\u7f6e ovn0 \u7f51\u5361\u5730\u5740"},{"location":"ops/change-log-level/","text":"\u8c03\u6574\u65e5\u5fd7\u7b49\u7ea7 \u00b6 \u6253\u5f00 kube-ovn.yaml \uff0c\u5728\u670d\u52a1\u542f\u52a8\u811a\u672c\u7684\u53c2\u6570\u5217\u8868\u4e2d\u8bbe\u7f6e log \u7b49\u7ea7\uff0c\u6bd4\u5982\uff1a vi kube-ovn.yaml # ... - name: kube-ovn-controller image: \"docker.io/kubeovn/kube-ovn:v1.12.0\" imagePullPolicy: IfNotPresent args: - /kube-ovn/start-controller.sh - --v = 3 # ... # log \u7b49\u7ea7\u8d8a\u9ad8\uff0clog \u5c31\u8d8a\u8be6\u7ec6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8c03\u6574\u65e5\u5fd7\u7b49\u7ea7"},{"location":"ops/change-log-level/#_1","text":"\u6253\u5f00 kube-ovn.yaml \uff0c\u5728\u670d\u52a1\u542f\u52a8\u811a\u672c\u7684\u53c2\u6570\u5217\u8868\u4e2d\u8bbe\u7f6e log \u7b49\u7ea7\uff0c\u6bd4\u5982\uff1a vi kube-ovn.yaml # ... - name: kube-ovn-controller image: \"docker.io/kubeovn/kube-ovn:v1.12.0\" imagePullPolicy: IfNotPresent args: - /kube-ovn/start-controller.sh - --v = 3 # ... # log \u7b49\u7ea7\u8d8a\u9ad8\uff0clog \u5c31\u8d8a\u8be6\u7ec6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8c03\u6574\u65e5\u5fd7\u7b49\u7ea7"},{"location":"ops/change-ovn-central-node/","text":"\u66f4\u6362 ovn-central \u8282\u70b9 \u00b6 \u7531\u4e8e ovn-central \u5185\u7684 ovn-nb \u548c ovn-sb \u5206\u522b\u5efa\u7acb\u4e86\u7c7b\u4f3c etcd \u7684 raft \u96c6\u7fa4\uff0c\u56e0\u6b64\u66f4\u6362 ovn-central \u8282\u70b9\u9700\u8981\u989d\u5916\u7684\u64cd\u4f5c\uff0c\u4fdd\u8bc1\u96c6\u7fa4\u72b6\u6001\u7684\u6b63\u786e\u548c\u6570\u636e\u7684\u4e00\u81f4\u3002\u5efa\u8bae\u6bcf\u6b21\u53ea\u5bf9\u4e00\u4e2a\u8282\u70b9\u8fdb\u884c\u4e0a\u4e0b\u7ebf\u5904\u7406\uff0c\u4ee5\u907f\u514d\u96c6\u7fa4\u8fdb\u5165\u4e0d\u53ef\u7528 \u72b6\u6001\uff0c\u5f71\u54cd\u96c6\u7fa4\u6574\u4f53\u7f51\u7edc\u3002 ovn-central \u8282\u70b9\u4e0b\u7ebf \u00b6 \u672c\u6587\u6863\u9488\u5bf9\u5982\u4e0b\u7684\u96c6\u7fa4\u60c5\u51b5\uff0c\u4ee5\u4e0b\u7ebf kube-ovn-control-plane2 \u8282\u70b9\u4e3a\u4f8b\uff0c\u4ecb\u7ecd\u5982\u4f55\u5c06\u5176\u4ece ovn-central \u96c6\u7fa4\u4e2d\u79fb\u9664\u3002 # kubectl -n kube-system get pod -o wide | grep central ovn-central-6bf58cbc97-2cdhg 1 /1 Running 0 21m 172 .18.0.3 kube-ovn-control-plane <none> <none> ovn-central-6bf58cbc97-crmfp 1 /1 Running 0 21m 172 .18.0.5 kube-ovn-control-plane2 <none> <none> ovn-central-6bf58cbc97-lxmpl 1 /1 Running 0 21m 172 .18.0.4 kube-ovn-control-plane3 <none> <none> \u4e0b\u7ebf ovn-nb \u96c6\u7fa4\u5185\u5bf9\u5e94\u8282\u70b9 \u00b6 \u9996\u5148\u67e5\u770b\u8282\u70b9\u5728\u96c6\u7fa4\u5185\u7684 ID\uff0c\u4ee5\u4fbf\u540e\u7eed\u64cd\u4f5c\u3002 # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2135194 ms ago, reason: timeout Last Election won: 2135188 ms ago Election timer: 5000 Log: [ 135 , 135 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-d64b ->d64b <-4984 ->4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 134 d64b ( d64b at tcp: [ 172 .18.0.5 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago status: ok kube-ovn-control-plane2 \u5bf9\u5e94\u8282\u70b9 IP \u4e3a 172.18.0.5 \uff0c\u96c6\u7fa4\u5185\u5bf9\u5e94\u7684 ID \u4e3a d64b \u3002\u63a5\u4e0b\u6765\u4ece ovn-nb \u96c6\u7fa4\u4e2d\u8e22\u51fa\u8be5\u8282\u70b9\uff1a # kubectl ko nb kick d64b started removal \u786e\u8ba4\u8282\u70b9\u8e22\u51fa\u6210\u529f\uff1a # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2297649 ms ago, reason: timeout Last Election won: 2297643 ms ago Election timer: 5000 Log: [ 136 , 136 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-4984 ->4984 Disconnections: 2 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 136 match_index = 135 last msg 1270 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 135 status: ok \u4e0b\u7ebf ovn-sb \u96c6\u7fa4\u5185\u5bf9\u5e94\u8282\u70b9 \u00b6 \u63a5\u4e0b\u6765\u9700\u8981\u64cd\u4f5c ovn-sb \u96c6\u7fa4\uff0c\u9996\u5148\u67e5\u770b\u8282\u70b9\u5728\u96c6\u7fa4\u5185\u7684 ID\uff0c\u4ee5\u4fbf\u540e\u7eed\u64cd\u4f5c\uff1a kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2395317 ms ago, reason: timeout Last Election won: 2395316 ms ago Election timer: 5000 Log: [ 130 , 130 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-e9f7 ->e9f7 <-6e84 ->6e84 Disconnections: 0 Servers: e9f7 ( e9f7 at tcp: [ 172 .18.0.5 ] :6644 ) next_index = 130 match_index = 129 last msg 1006 ms ago 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 130 match_index = 129 last msg 1004 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 129 status: ok kube-ovn-control-plane2 \u5bf9\u5e94\u8282\u70b9 IP \u4e3a 172.18.0.5 \uff0c\u96c6\u7fa4\u5185\u5bf9\u5e94\u7684 ID \u4e3a e9f7 \u3002\u63a5\u4e0b\u6765\u4ece ovn-sb \u96c6\u7fa4\u4e2d\u8e22\u51fa\u8be5\u8282\u70b9\uff1a # kubectl ko sb kick e9f7 started removal \u786e\u8ba4\u8282\u70b9\u8e22\u51fa\u6210\u529f\uff1a # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2481636 ms ago, reason: timeout Last Election won: 2481635 ms ago Election timer: 5000 Log: [ 131 , 131 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-6e84 ->6e84 Disconnections: 2 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 131 match_index = 130 last msg 642 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 130 status: ok \u5220\u9664\u8282\u70b9\u6807\u7b7e\uff0c\u5e76\u7f29\u5bb9 ovn-central \u00b6 \u6ce8\u610f\u9700\u5728 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u7684\u8282\u70b9\u5730\u5740\u4e2d\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u3002 kubectl label node kube-ovn-control-plane2 kube-ovn/role- kubectl scale deployment -n kube-system ovn-central --replicas = 2 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4\" kubectl rollout status deployment/ovn-central -n kube-system \u4fee\u6539\u5176\u4ed6\u7ec4\u4ef6\u8fde\u63a5 ovn-central \u5730\u5740 \u00b6 \u4fee\u6539 ovs-ovn \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u5730\u5740\u3002 # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted \u4fee\u6539 kube-ovn-controller \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u5730\u5740\u3002 # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u6e05\u7406\u8282\u70b9 \u00b6 \u5220\u9664 kube-ovn-control-plane2 \u8282\u70b9\u5185\u7684\u6570\u636e\u5e93\u6587\u4ef6\uff0c\u907f\u514d\u91cd\u590d\u6dfb\u52a0\u8282\u70b9\u65f6\u53d1\u751f\u5f02\u5e38\uff1a rm -rf /etc/origin/ovn \u5982\u9700\u5c06\u8282\u70b9\u4ece\u6574\u4e2a Kubernetes \u96c6\u7fa4\u4e0b\u7ebf\uff0c\u8fd8\u9700\u7ee7\u7eed\u53c2\u8003 \u5220\u9664\u5de5\u4f5c\u8282\u70b9 \u8fdb\u884c\u64cd\u4f5c\u3002 ovn-central \u8282\u70b9\u4e0a\u7ebf \u00b6 \u4e0b\u5217\u6b65\u9aa4\u4f1a\u5c06\u4e00\u4e2a\u65b0\u7684 Kubernetes \u8282\u70b9\u52a0\u5165 ovn-central \u96c6\u7fa4\u3002 \u76ee\u5f55\u68c0\u67e5 \u00b6 \u68c0\u67e5\u65b0\u589e\u8282\u70b9\u7684 /etc/origin/ovn \u76ee\u5f55\u4e2d\u662f\u5426\u5b58\u5728 ovnnb_db.db \u6216 ovnsb_db.db \u6587\u4ef6\uff0c\u82e5\u5b58\u5728\u9700\u63d0\u524d\u5220\u9664\uff1a rm -rf /etc/origin/ovn \u786e\u8ba4\u5f53\u524d ovn-central \u96c6\u7fa4\u72b6\u6001\u6b63\u5e38 \u00b6 \u82e5\u5f53\u524d ovn-central \u96c6\u7fa4\u72b6\u6001\u5df2\u7ecf\u5f02\u5e38\uff0c\u65b0\u589e\u8282\u70b9\u53ef\u80fd\u5bfc\u81f4\u6295\u7968\u9009\u4e3e\u65e0\u6cd5\u8fc7\u534a\u6570\uff0c\u5f71\u54cd\u540e\u7eed\u64cd\u4f5c\u3002 # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 44 Leader: self Vote: self Last Election started 1855739 ms ago, reason: timeout Last Election won: 1855729 ms ago Election timer: 5000 Log: [ 147 , 147 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->4984 <-4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 147 match_index = 146 last msg 367 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 140 match_index = 146 status: ok # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 33 Leader: self Vote: self Last Election started 1868589 ms ago, reason: timeout Last Election won: 1868579 ms ago Election timer: 5000 Log: [ 142 , 142 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->6e84 <-6e84 Disconnections: 0 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 142 match_index = 141 last msg 728 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 134 match_index = 141 status: ok \u7ed9\u8282\u70b9\u589e\u52a0\u6807\u7b7e\u5e76\u6269\u5bb9 \u00b6 \u6ce8\u610f\u9700\u5728 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u7684\u8282\u70b9\u5730\u5740\u4e2d\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\u3002 kubectl label node kube-ovn-control-plane2 kube-ovn/role = master kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4,172.18.0.5\" kubectl rollout status deployment/ovn-central -n kube-system \u4fee\u6539\u5176\u4ed6\u7ec4\u4ef6\u8fde\u63a5 ovn-central \u5730\u5740 \u00b6 \u4fee\u6539 ovs-ovn \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\uff1a # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted \u4fee\u6539 kube-ovn-controller \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\uff1a # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u66f4\u6362 ovn-central \u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-central","text":"\u7531\u4e8e ovn-central \u5185\u7684 ovn-nb \u548c ovn-sb \u5206\u522b\u5efa\u7acb\u4e86\u7c7b\u4f3c etcd \u7684 raft \u96c6\u7fa4\uff0c\u56e0\u6b64\u66f4\u6362 ovn-central \u8282\u70b9\u9700\u8981\u989d\u5916\u7684\u64cd\u4f5c\uff0c\u4fdd\u8bc1\u96c6\u7fa4\u72b6\u6001\u7684\u6b63\u786e\u548c\u6570\u636e\u7684\u4e00\u81f4\u3002\u5efa\u8bae\u6bcf\u6b21\u53ea\u5bf9\u4e00\u4e2a\u8282\u70b9\u8fdb\u884c\u4e0a\u4e0b\u7ebf\u5904\u7406\uff0c\u4ee5\u907f\u514d\u96c6\u7fa4\u8fdb\u5165\u4e0d\u53ef\u7528 \u72b6\u6001\uff0c\u5f71\u54cd\u96c6\u7fa4\u6574\u4f53\u7f51\u7edc\u3002","title":"\u66f4\u6362 ovn-central \u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-central_1","text":"\u672c\u6587\u6863\u9488\u5bf9\u5982\u4e0b\u7684\u96c6\u7fa4\u60c5\u51b5\uff0c\u4ee5\u4e0b\u7ebf kube-ovn-control-plane2 \u8282\u70b9\u4e3a\u4f8b\uff0c\u4ecb\u7ecd\u5982\u4f55\u5c06\u5176\u4ece ovn-central \u96c6\u7fa4\u4e2d\u79fb\u9664\u3002 # kubectl -n kube-system get pod -o wide | grep central ovn-central-6bf58cbc97-2cdhg 1 /1 Running 0 21m 172 .18.0.3 kube-ovn-control-plane <none> <none> ovn-central-6bf58cbc97-crmfp 1 /1 Running 0 21m 172 .18.0.5 kube-ovn-control-plane2 <none> <none> ovn-central-6bf58cbc97-lxmpl 1 /1 Running 0 21m 172 .18.0.4 kube-ovn-control-plane3 <none> <none>","title":"ovn-central \u8282\u70b9\u4e0b\u7ebf"},{"location":"ops/change-ovn-central-node/#ovn-nb","text":"\u9996\u5148\u67e5\u770b\u8282\u70b9\u5728\u96c6\u7fa4\u5185\u7684 ID\uff0c\u4ee5\u4fbf\u540e\u7eed\u64cd\u4f5c\u3002 # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2135194 ms ago, reason: timeout Last Election won: 2135188 ms ago Election timer: 5000 Log: [ 135 , 135 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-d64b ->d64b <-4984 ->4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 134 d64b ( d64b at tcp: [ 172 .18.0.5 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago status: ok kube-ovn-control-plane2 \u5bf9\u5e94\u8282\u70b9 IP \u4e3a 172.18.0.5 \uff0c\u96c6\u7fa4\u5185\u5bf9\u5e94\u7684 ID \u4e3a d64b \u3002\u63a5\u4e0b\u6765\u4ece ovn-nb \u96c6\u7fa4\u4e2d\u8e22\u51fa\u8be5\u8282\u70b9\uff1a # kubectl ko nb kick d64b started removal \u786e\u8ba4\u8282\u70b9\u8e22\u51fa\u6210\u529f\uff1a # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2297649 ms ago, reason: timeout Last Election won: 2297643 ms ago Election timer: 5000 Log: [ 136 , 136 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-4984 ->4984 Disconnections: 2 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 136 match_index = 135 last msg 1270 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 135 status: ok","title":"\u4e0b\u7ebf ovn-nb \u96c6\u7fa4\u5185\u5bf9\u5e94\u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-sb","text":"\u63a5\u4e0b\u6765\u9700\u8981\u64cd\u4f5c ovn-sb \u96c6\u7fa4\uff0c\u9996\u5148\u67e5\u770b\u8282\u70b9\u5728\u96c6\u7fa4\u5185\u7684 ID\uff0c\u4ee5\u4fbf\u540e\u7eed\u64cd\u4f5c\uff1a kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2395317 ms ago, reason: timeout Last Election won: 2395316 ms ago Election timer: 5000 Log: [ 130 , 130 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-e9f7 ->e9f7 <-6e84 ->6e84 Disconnections: 0 Servers: e9f7 ( e9f7 at tcp: [ 172 .18.0.5 ] :6644 ) next_index = 130 match_index = 129 last msg 1006 ms ago 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 130 match_index = 129 last msg 1004 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 129 status: ok kube-ovn-control-plane2 \u5bf9\u5e94\u8282\u70b9 IP \u4e3a 172.18.0.5 \uff0c\u96c6\u7fa4\u5185\u5bf9\u5e94\u7684 ID \u4e3a e9f7 \u3002\u63a5\u4e0b\u6765\u4ece ovn-sb \u96c6\u7fa4\u4e2d\u8e22\u51fa\u8be5\u8282\u70b9\uff1a # kubectl ko sb kick e9f7 started removal \u786e\u8ba4\u8282\u70b9\u8e22\u51fa\u6210\u529f\uff1a # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2481636 ms ago, reason: timeout Last Election won: 2481635 ms ago Election timer: 5000 Log: [ 131 , 131 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-6e84 ->6e84 Disconnections: 2 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 131 match_index = 130 last msg 642 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 130 status: ok","title":"\u4e0b\u7ebf ovn-sb \u96c6\u7fa4\u5185\u5bf9\u5e94\u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-central_2","text":"\u6ce8\u610f\u9700\u5728 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u7684\u8282\u70b9\u5730\u5740\u4e2d\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u3002 kubectl label node kube-ovn-control-plane2 kube-ovn/role- kubectl scale deployment -n kube-system ovn-central --replicas = 2 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4\" kubectl rollout status deployment/ovn-central -n kube-system","title":"\u5220\u9664\u8282\u70b9\u6807\u7b7e\uff0c\u5e76\u7f29\u5bb9 ovn-central"},{"location":"ops/change-ovn-central-node/#ovn-central_3","text":"\u4fee\u6539 ovs-ovn \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u5730\u5740\u3002 # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted \u4fee\u6539 kube-ovn-controller \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u5730\u5740\u3002 # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out","title":"\u4fee\u6539\u5176\u4ed6\u7ec4\u4ef6\u8fde\u63a5 ovn-central \u5730\u5740"},{"location":"ops/change-ovn-central-node/#_1","text":"\u5220\u9664 kube-ovn-control-plane2 \u8282\u70b9\u5185\u7684\u6570\u636e\u5e93\u6587\u4ef6\uff0c\u907f\u514d\u91cd\u590d\u6dfb\u52a0\u8282\u70b9\u65f6\u53d1\u751f\u5f02\u5e38\uff1a rm -rf /etc/origin/ovn \u5982\u9700\u5c06\u8282\u70b9\u4ece\u6574\u4e2a Kubernetes \u96c6\u7fa4\u4e0b\u7ebf\uff0c\u8fd8\u9700\u7ee7\u7eed\u53c2\u8003 \u5220\u9664\u5de5\u4f5c\u8282\u70b9 \u8fdb\u884c\u64cd\u4f5c\u3002","title":"\u6e05\u7406\u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-central_4","text":"\u4e0b\u5217\u6b65\u9aa4\u4f1a\u5c06\u4e00\u4e2a\u65b0\u7684 Kubernetes \u8282\u70b9\u52a0\u5165 ovn-central \u96c6\u7fa4\u3002","title":"ovn-central \u8282\u70b9\u4e0a\u7ebf"},{"location":"ops/change-ovn-central-node/#_2","text":"\u68c0\u67e5\u65b0\u589e\u8282\u70b9\u7684 /etc/origin/ovn \u76ee\u5f55\u4e2d\u662f\u5426\u5b58\u5728 ovnnb_db.db \u6216 ovnsb_db.db \u6587\u4ef6\uff0c\u82e5\u5b58\u5728\u9700\u63d0\u524d\u5220\u9664\uff1a rm -rf /etc/origin/ovn","title":"\u76ee\u5f55\u68c0\u67e5"},{"location":"ops/change-ovn-central-node/#ovn-central_5","text":"\u82e5\u5f53\u524d ovn-central \u96c6\u7fa4\u72b6\u6001\u5df2\u7ecf\u5f02\u5e38\uff0c\u65b0\u589e\u8282\u70b9\u53ef\u80fd\u5bfc\u81f4\u6295\u7968\u9009\u4e3e\u65e0\u6cd5\u8fc7\u534a\u6570\uff0c\u5f71\u54cd\u540e\u7eed\u64cd\u4f5c\u3002 # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 44 Leader: self Vote: self Last Election started 1855739 ms ago, reason: timeout Last Election won: 1855729 ms ago Election timer: 5000 Log: [ 147 , 147 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->4984 <-4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 147 match_index = 146 last msg 367 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 140 match_index = 146 status: ok # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 33 Leader: self Vote: self Last Election started 1868589 ms ago, reason: timeout Last Election won: 1868579 ms ago Election timer: 5000 Log: [ 142 , 142 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->6e84 <-6e84 Disconnections: 0 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 142 match_index = 141 last msg 728 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 134 match_index = 141 status: ok","title":"\u786e\u8ba4\u5f53\u524d ovn-central \u96c6\u7fa4\u72b6\u6001\u6b63\u5e38"},{"location":"ops/change-ovn-central-node/#_3","text":"\u6ce8\u610f\u9700\u5728 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u7684\u8282\u70b9\u5730\u5740\u4e2d\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\u3002 kubectl label node kube-ovn-control-plane2 kube-ovn/role = master kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4,172.18.0.5\" kubectl rollout status deployment/ovn-central -n kube-system","title":"\u7ed9\u8282\u70b9\u589e\u52a0\u6807\u7b7e\u5e76\u6269\u5bb9"},{"location":"ops/change-ovn-central-node/#ovn-central_6","text":"\u4fee\u6539 ovs-ovn \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\uff1a # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted \u4fee\u6539 kube-ovn-controller \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\uff1a # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4fee\u6539\u5176\u4ed6\u7ec4\u4ef6\u8fde\u63a5 ovn-central \u5730\u5740"},{"location":"ops/delete-worker-node/","text":"\u5220\u9664\u5de5\u4f5c\u8282\u70b9 \u00b6 \u5982\u679c\u53ea\u662f\u7b80\u5355\u4ece Kubernetes \u4e2d\u5220\u9664\u8282\u70b9\uff0c\u7531\u4e8e\u8282\u70b9\u4e0a ovs-ovn \u4e2d\u8fd0\u884c\u7684 ovn-controller \u8fdb\u7a0b\u4ecd\u5728\u8fd0\u884c\u4f1a\u5b9a\u671f\u8fde\u63a5 ovn-central \u6ce8\u518c\u76f8\u5173\u7f51\u7edc\u4fe1\u606f\uff0c \u4f1a\u5bfc\u81f4\u989d\u5916\u8d44\u6e90\u6d6a\u8d39\u5e76\u6709\u6f5c\u5728\u7684\u89c4\u5219\u51b2\u7a81\u98ce\u9669\u3002 \u56e0\u6b64\u5728\u4ece Kubernetes \u5185\u5220\u9664\u8282\u70b9\u65f6\uff0c\u8bf7\u6309\u7167\u4e0b\u9762\u7684\u6b65\u9aa4\u6765\u4fdd\u8bc1\u7f51\u7edc\u4fe1\u606f\u53ef\u4ee5\u6b63\u5e38\u88ab\u6e05\u7406\u3002 \u8be5\u6587\u6863\u4ecb\u7ecd\u5220\u9664\u5de5\u4f5c\u8282\u70b9\u7684\u6b65\u9aa4\uff0c\u5982\u9700\u66f4\u6362 ovn-central \u6240\u5728\u8282\u70b9\uff0c\u8bf7\u53c2\u8003 \u66f4\u6362 ovn-central \u8282\u70b9 \u3002 \u9a71\u9010\u8282\u70b9\u4e0a\u6240\u6709\u5bb9\u5668 \u00b6 # kubectl drain kube-ovn-worker --ignore-daemonsets --force node/kube-ovn-worker cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-ovn-cni-zt74b, kube-system/kube-ovn-pinger-5rxfs, kube-system/kube-proxy-jpmnm, kube-system/ovs-ovn-v2kll evicting pod kube-system/coredns-64897985d-qsgpt evicting pod local-path-storage/local-path-provisioner-5ddd94ff66-llss6 evicting pod kube-system/kube-ovn-controller-8459db5ff4-94lxb pod/kube-ovn-controller-8459db5ff4-94lxb evicted pod/coredns-64897985d-qsgpt evicted pod/local-path-provisioner-5ddd94ff66-llss6 evicted node/kube-ovn-worker drained \u505c\u6b62 kubelet \u548c docker \u00b6 \u8be5\u6b65\u9aa4\u4f1a\u505c\u6b62 ovs-ovn \u5bb9\u5668\uff0c\u4ee5\u907f\u514d\u5411 ovn-central \u8fdb\u884c\u4fe1\u606f\u6ce8\u518c\uff0c\u767b\u5f55\u5230\u5bf9\u5e94\u8282\u70b9\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a systemctl stop kubelet systemctl stop docker \u5982\u679c\u4f7f\u7528\u7684 CRI \u4e3a containerd\uff0c\u9700\u8981\u6267\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u6765\u505c\u6b62 ovs-ovn \u5bb9\u5668\uff1a crictl rm -f $( crictl ps | grep openvswitch | awk '{print $1}' ) \u6e05\u7406 Node \u4e0a\u7684\u6b8b\u7559\u6570\u636e \u00b6 rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn \u4f7f\u7528 kubectl \u5220\u9664\u8282\u70b9 \u00b6 kubectl delete no kube-ovn-01 \u68c0\u67e5\u5bf9\u5e94\u8282\u70b9\u662f\u5426\u4ece ovn-sb \u4e2d\u5220\u9664 \u00b6 \u4e0b\u9762\u7684\u793a\u4f8b\u4e3a kube-ovn-worker \u4f9d\u7136\u672a\u88ab\u5220\u9664\uff1a # kubectl ko sbctl show Chassis \"b0564934-5a0d-4804-a4c0-476c93596a17\" hostname: kube-ovn-worker Encap geneve ip: \"172.18.0.2\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-5rxfs.kube-system Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u82e5\u8282\u70b9\u5bf9\u5e94\u7684 chassis \u4f9d\u7136\u5b58\u5728\uff0c\u624b\u52a8\u8fdb\u884c\u5220\u9664 \u00b6 uuid \u4e3a\u4e4b\u524d\u547d\u4ee4\u6240\u67e5\u51fa\u7684 Chassis \u5bf9\u5e94 id\uff1a # kubectl ko sbctl chassis-del b0564934-5a0d-4804-a4c0-476c93596a17 # kubectl ko sbctl show Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5220\u9664\u5de5\u4f5c\u8282\u70b9"},{"location":"ops/delete-worker-node/#_1","text":"\u5982\u679c\u53ea\u662f\u7b80\u5355\u4ece Kubernetes \u4e2d\u5220\u9664\u8282\u70b9\uff0c\u7531\u4e8e\u8282\u70b9\u4e0a ovs-ovn \u4e2d\u8fd0\u884c\u7684 ovn-controller \u8fdb\u7a0b\u4ecd\u5728\u8fd0\u884c\u4f1a\u5b9a\u671f\u8fde\u63a5 ovn-central \u6ce8\u518c\u76f8\u5173\u7f51\u7edc\u4fe1\u606f\uff0c \u4f1a\u5bfc\u81f4\u989d\u5916\u8d44\u6e90\u6d6a\u8d39\u5e76\u6709\u6f5c\u5728\u7684\u89c4\u5219\u51b2\u7a81\u98ce\u9669\u3002 \u56e0\u6b64\u5728\u4ece Kubernetes \u5185\u5220\u9664\u8282\u70b9\u65f6\uff0c\u8bf7\u6309\u7167\u4e0b\u9762\u7684\u6b65\u9aa4\u6765\u4fdd\u8bc1\u7f51\u7edc\u4fe1\u606f\u53ef\u4ee5\u6b63\u5e38\u88ab\u6e05\u7406\u3002 \u8be5\u6587\u6863\u4ecb\u7ecd\u5220\u9664\u5de5\u4f5c\u8282\u70b9\u7684\u6b65\u9aa4\uff0c\u5982\u9700\u66f4\u6362 ovn-central \u6240\u5728\u8282\u70b9\uff0c\u8bf7\u53c2\u8003 \u66f4\u6362 ovn-central \u8282\u70b9 \u3002","title":"\u5220\u9664\u5de5\u4f5c\u8282\u70b9"},{"location":"ops/delete-worker-node/#_2","text":"# kubectl drain kube-ovn-worker --ignore-daemonsets --force node/kube-ovn-worker cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-ovn-cni-zt74b, kube-system/kube-ovn-pinger-5rxfs, kube-system/kube-proxy-jpmnm, kube-system/ovs-ovn-v2kll evicting pod kube-system/coredns-64897985d-qsgpt evicting pod local-path-storage/local-path-provisioner-5ddd94ff66-llss6 evicting pod kube-system/kube-ovn-controller-8459db5ff4-94lxb pod/kube-ovn-controller-8459db5ff4-94lxb evicted pod/coredns-64897985d-qsgpt evicted pod/local-path-provisioner-5ddd94ff66-llss6 evicted node/kube-ovn-worker drained","title":"\u9a71\u9010\u8282\u70b9\u4e0a\u6240\u6709\u5bb9\u5668"},{"location":"ops/delete-worker-node/#kubelet-docker","text":"\u8be5\u6b65\u9aa4\u4f1a\u505c\u6b62 ovs-ovn \u5bb9\u5668\uff0c\u4ee5\u907f\u514d\u5411 ovn-central \u8fdb\u884c\u4fe1\u606f\u6ce8\u518c\uff0c\u767b\u5f55\u5230\u5bf9\u5e94\u8282\u70b9\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a systemctl stop kubelet systemctl stop docker \u5982\u679c\u4f7f\u7528\u7684 CRI \u4e3a containerd\uff0c\u9700\u8981\u6267\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u6765\u505c\u6b62 ovs-ovn \u5bb9\u5668\uff1a crictl rm -f $( crictl ps | grep openvswitch | awk '{print $1}' )","title":"\u505c\u6b62 kubelet \u548c docker"},{"location":"ops/delete-worker-node/#node","text":"rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn","title":"\u6e05\u7406 Node \u4e0a\u7684\u6b8b\u7559\u6570\u636e"},{"location":"ops/delete-worker-node/#kubectl","text":"kubectl delete no kube-ovn-01","title":"\u4f7f\u7528 kubectl \u5220\u9664\u8282\u70b9"},{"location":"ops/delete-worker-node/#ovn-sb","text":"\u4e0b\u9762\u7684\u793a\u4f8b\u4e3a kube-ovn-worker \u4f9d\u7136\u672a\u88ab\u5220\u9664\uff1a # kubectl ko sbctl show Chassis \"b0564934-5a0d-4804-a4c0-476c93596a17\" hostname: kube-ovn-worker Encap geneve ip: \"172.18.0.2\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-5rxfs.kube-system Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system","title":"\u68c0\u67e5\u5bf9\u5e94\u8282\u70b9\u662f\u5426\u4ece ovn-sb \u4e2d\u5220\u9664"},{"location":"ops/delete-worker-node/#chassis","text":"uuid \u4e3a\u4e4b\u524d\u547d\u4ee4\u6240\u67e5\u51fa\u7684 Chassis \u5bf9\u5e94 id\uff1a # kubectl ko sbctl chassis-del b0564934-5a0d-4804-a4c0-476c93596a17 # kubectl ko sbctl show Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u82e5\u8282\u70b9\u5bf9\u5e94\u7684 chassis \u4f9d\u7136\u5b58\u5728\uff0c\u624b\u52a8\u8fdb\u884c\u5220\u9664"},{"location":"ops/faq/","text":"\u5176\u4ed6\u5e38\u89c1\u95ee\u9898 \u00b6 \u9e92\u9e9f ARM \u7cfb\u7edf\u8de8\u4e3b\u673a\u5bb9\u5668\u8bbf\u95ee\u95f4\u6b47\u5931\u8d25 \u00b6 \u73b0\u8c61 \u00b6 \u9e92\u9e9f ARM \u7cfb\u7edf\u548c\u90e8\u5206\u56fd\u4ea7\u5316\u7f51\u5361 offload \u914d\u5408\u5b58\u5728\u95ee\u9898\uff0c\u4f1a\u5bfc\u81f4\u5bb9\u5668\u7f51\u7edc\u95f4\u6b47\u6545\u969c\u3002 \u4f7f\u7528 netstat \u786e\u8ba4\u95ee\u9898\uff1a # netstat -us IcmpMsg: InType0: 22 InType3: 24 InType8: 117852 OutType0: 117852 OutType3: 29 OutType8: 22 Udp: 3040636 packets received 0 packets to unknown port received. 4 packet receive errors 602 packets sent 0 receive buffer errors 0 send buffer errors InCsumErrors: 4 UdpLite: IpExt: InBcastPkts: 10244 InOctets: 4446320361 OutOctets: 1496815600 InBcastOctets: 3095950 InNoECTPkts: 7683903 \u82e5\u5b58\u5728 InCsumErrors \uff0c\u4e14\u968f\u7740\u8bbf\u95ee\u5931\u8d25\u589e\u52a0\uff0c\u53ef\u786e\u8ba4\u662f\u8be5\u95ee\u9898\u3002 \u89e3\u51b3\u65b9\u6cd5 \u00b6 \u6839\u672c\u89e3\u51b3\u9700\u8981\u548c\u9e92\u9e9f\u4ee5\u53ca\u5bf9\u5e94\u7f51\u5361\u5382\u5546\u6c9f\u901a\uff0c\u66f4\u65b0\u7cfb\u7edf\u548c\u9a71\u52a8\u3002\u4e34\u65f6\u89e3\u51b3\u53ef\u5148\u5173\u95ed\u7269\u7406 \u7f51\u5361\u7684 tx offload \u4f46\u662f\u4f1a\u5bfc\u81f4 tcp \u6027\u80fd\u6709\u8f83\u660e\u663e\u4e0b\u964d\u3002 ethtool -K eth0 tx off \u7ecf\u793e\u533a\u53cd\u9988\u4f7f\u7528 4.19.90-25.16.v2101 \u5185\u6838\u540e\u53ef\u4ee5\u89e3\u51b3\u8be5\u95ee\u9898\u3002 Pod \u8bbf\u95ee Service \u4e0d\u901a \u00b6 \u73b0\u8c61 \u00b6 Pod \u5185\u65e0\u6cd5\u8bbf\u95ee Service \u5bf9\u5e94\u7684\u670d\u52a1\uff0c dmesg \u663e\u793a\u5f02\u5e38\uff1a netlink: Unknown conntrack attr ( type = 6 , max = 5 ) openvswitch: netlink: Flow actions may not be safe on all matching packets. \u8be5\u65e5\u5fd7\u8bf4\u660e\u5185\u6838\u5185 OVS \u7248\u672c\u8fc7\u4f4e\u4e0d\u652f\u6301\u5bf9\u5e94 NAT \u64cd\u4f5c\u3002 \u89e3\u51b3\u65b9\u6cd5 \u00b6 \u5347\u7ea7\u5185\u6838\u6a21\u5757\u6216\u624b\u52a8\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u3002 \u82e5\u53ea\u4f7f\u7528 Overlay \u7f51\u7edc\u53ef\u4ee5\u66f4\u6539 kube-ovn-controller \u542f\u52a8\u53c2\u6570\u8bbe\u7f6e --enable-lb=false \u5173\u95ed OVN LB \u4f7f\u7528 kube-proxy \u8fdb\u884c Service \u8f6c\u53d1\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5176\u4ed6\u5e38\u89c1\u95ee\u9898"},{"location":"ops/faq/#_1","text":"","title":"\u5176\u4ed6\u5e38\u89c1\u95ee\u9898"},{"location":"ops/faq/#arm","text":"","title":"\u9e92\u9e9f ARM \u7cfb\u7edf\u8de8\u4e3b\u673a\u5bb9\u5668\u8bbf\u95ee\u95f4\u6b47\u5931\u8d25"},{"location":"ops/faq/#_2","text":"\u9e92\u9e9f ARM \u7cfb\u7edf\u548c\u90e8\u5206\u56fd\u4ea7\u5316\u7f51\u5361 offload \u914d\u5408\u5b58\u5728\u95ee\u9898\uff0c\u4f1a\u5bfc\u81f4\u5bb9\u5668\u7f51\u7edc\u95f4\u6b47\u6545\u969c\u3002 \u4f7f\u7528 netstat \u786e\u8ba4\u95ee\u9898\uff1a # netstat -us IcmpMsg: InType0: 22 InType3: 24 InType8: 117852 OutType0: 117852 OutType3: 29 OutType8: 22 Udp: 3040636 packets received 0 packets to unknown port received. 4 packet receive errors 602 packets sent 0 receive buffer errors 0 send buffer errors InCsumErrors: 4 UdpLite: IpExt: InBcastPkts: 10244 InOctets: 4446320361 OutOctets: 1496815600 InBcastOctets: 3095950 InNoECTPkts: 7683903 \u82e5\u5b58\u5728 InCsumErrors \uff0c\u4e14\u968f\u7740\u8bbf\u95ee\u5931\u8d25\u589e\u52a0\uff0c\u53ef\u786e\u8ba4\u662f\u8be5\u95ee\u9898\u3002","title":"\u73b0\u8c61"},{"location":"ops/faq/#_3","text":"\u6839\u672c\u89e3\u51b3\u9700\u8981\u548c\u9e92\u9e9f\u4ee5\u53ca\u5bf9\u5e94\u7f51\u5361\u5382\u5546\u6c9f\u901a\uff0c\u66f4\u65b0\u7cfb\u7edf\u548c\u9a71\u52a8\u3002\u4e34\u65f6\u89e3\u51b3\u53ef\u5148\u5173\u95ed\u7269\u7406 \u7f51\u5361\u7684 tx offload \u4f46\u662f\u4f1a\u5bfc\u81f4 tcp \u6027\u80fd\u6709\u8f83\u660e\u663e\u4e0b\u964d\u3002 ethtool -K eth0 tx off \u7ecf\u793e\u533a\u53cd\u9988\u4f7f\u7528 4.19.90-25.16.v2101 \u5185\u6838\u540e\u53ef\u4ee5\u89e3\u51b3\u8be5\u95ee\u9898\u3002","title":"\u89e3\u51b3\u65b9\u6cd5"},{"location":"ops/faq/#pod-service","text":"","title":"Pod \u8bbf\u95ee Service \u4e0d\u901a"},{"location":"ops/faq/#_4","text":"Pod \u5185\u65e0\u6cd5\u8bbf\u95ee Service \u5bf9\u5e94\u7684\u670d\u52a1\uff0c dmesg \u663e\u793a\u5f02\u5e38\uff1a netlink: Unknown conntrack attr ( type = 6 , max = 5 ) openvswitch: netlink: Flow actions may not be safe on all matching packets. \u8be5\u65e5\u5fd7\u8bf4\u660e\u5185\u6838\u5185 OVS \u7248\u672c\u8fc7\u4f4e\u4e0d\u652f\u6301\u5bf9\u5e94 NAT \u64cd\u4f5c\u3002","title":"\u73b0\u8c61"},{"location":"ops/faq/#_5","text":"\u5347\u7ea7\u5185\u6838\u6a21\u5757\u6216\u624b\u52a8\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u3002 \u82e5\u53ea\u4f7f\u7528 Overlay \u7f51\u7edc\u53ef\u4ee5\u66f4\u6539 kube-ovn-controller \u542f\u52a8\u53c2\u6570\u8bbe\u7f6e --enable-lb=false \u5173\u95ed OVN LB \u4f7f\u7528 kube-proxy \u8fdb\u884c Service \u8f6c\u53d1\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u89e3\u51b3\u65b9\u6cd5"},{"location":"ops/from-calico/","text":"\u5378\u8f7d Calico \u5b89\u88c5 Kube-OVN \u00b6 \u82e5 Kubernetes \u96c6\u7fa4\u5df2\u5b89\u88c5 Calico \u9700\u8981\u53d8\u66f4\u4e3a Kube-OVN \u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\u3002 \u672c\u6587\u4ee5 Calico v3.24.1 \u4e3a\u4f8b\uff0c\u5176\u5b83 Calico \u7248\u672c\u9700\u8981\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u8fdb\u884c\u8c03\u6574\u3002 \u51c6\u5907\u5de5\u4f5c \u00b6 \u4e3a\u4e86\u4fdd\u8bc1\u5207\u6362 CNI \u8fc7\u7a0b\u4e2d\u96c6\u7fa4\u7f51\u7edc\u4fdd\u6301\u7545\u901a\uff0cCalico ippool \u9700\u8981\u5f00\u542f nat outgoing\uff0c \u6216 \u5728\u6240\u6709\u8282\u70b9\u4e0a\u5173\u95ed rp_filter\uff1a sysctl net.ipv4.conf.all.rp_filter = 0 sysctl net.ipv4.conf.default.rp_filter = 0 # IPIP \u6a21\u5f0f sysctl net.ipv4.conf.tunl0.rp_filter = 0 # VXLAN \u6a21\u5f0f sysctl net.ipv4.conf.vxlan/calico.rp_filter = 0 # \u8def\u7531\u6a21\u5f0f\uff0ceth0 \u9700\u8981\u4fee\u6539\u4e3a\u5b9e\u9645\u4f7f\u7528\u7684\u7f51\u5361 sysctl net.ipv4.conf.eth0.rp_filter = 0 \u90e8\u7f72 Kube-OVN \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c \u00b6 wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u5b89\u88c5\u811a\u672c \u00b6 \u5c06\u5b89\u88c5\u811a\u672c\u4e2d\u91cd\u5efa Pod \u7684\u90e8\u5206\u5220\u9664\uff1a echo \"[Step 4/6] Delete pod that not in host network mode\" for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done \u6309\u9700\u4fee\u6539\u4ee5\u4e0b\u914d\u7f6e\uff1a REGISTRY = \"kubeovn\" # \u955c\u50cf\u4ed3\u5e93\u5730\u5740 VERSION = \"v1.12.0\" # \u955c\u50cf\u7248\u672c/Tag POD_CIDR = \"10.16.0.0/16\" # \u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e0d\u8981\u548c SVC/NODE/JOIN CIDR \u91cd\u53e0 SVC_CIDR = \"10.96.0.0/12\" # \u9700\u8981\u548c apiserver \u7684 service-cluster-ip-range \u4fdd\u6301\u4e00\u81f4 JOIN_CIDR = \"100.64.0.0/16\" # Pod \u548c\u4e3b\u673a\u901a\u4fe1\u7f51\u7edc CIDR\uff0c\u4e0d\u8981\u548c SVC/NODE/POD CIDR \u91cd\u53e0 LABEL = \"node-role.kubernetes.io/master\" # \u90e8\u7f72 OVN DB \u8282\u70b9\u7684\u6807\u7b7e IFACE = \"\" # \u5bb9\u5668\u7f51\u7edc\u6240\u4f7f\u7528\u7684\u7684\u5bbf\u4e3b\u673a\u7f51\u5361\u540d\uff0c\u5982\u679c\u4e3a\u7a7a\u5219\u4f7f\u7528 Kubernetes \u4e2d\u7684 Node IP \u6240\u5728\u7f51\u5361 TUNNEL_TYPE = \"geneve\" # \u96a7\u9053\u5c01\u88c5\u534f\u8bae\uff0c\u53ef\u9009 geneve, vxlan \u6216 stt\uff0cstt \u9700\u8981\u5355\u72ec\u7f16\u8bd1 ovs \u5185\u6838\u6a21\u5757 \u6ce8\u610f \uff1aPOD_CIDR \u53ca JOIN_CIDR \u4e0d\u53ef\u4e0e Calico ippool \u7684 CIDR \u51b2\u7a81\uff0c\u4e14 POD_CIDR \u9700\u8981\u5305\u542b\u8db3\u591f\u591a\u7684 IP \u6765\u5bb9\u7eb3\u96c6\u7fa4\u4e2d\u5df2\u6709\u7684 Pod\u3002 \u6267\u884c\u5b89\u88c5\u811a\u672c \u00b6 bash install.sh \u9010\u4e2a\u8282\u70b9\u8fc1\u79fb \u00b6 \u6309\u7167\u4ee5\u4e0b\u65b9\u6cd5\u4e3a\u6bcf\u4e2a\u8282\u70b9\u9010\u4e2a\u8fdb\u884c\u8fc1\u79fb\u3002 \u6ce8\u610f \uff1a\u547d\u4ee4\u4e2d\u7684 \\<NODE> \u9700\u8981\u66ff\u6362\u4e3a\u8282\u70b9\u540d\u79f0\u3002 \u9a71\u9010\u8282\u70b9 \u00b6 kubectl drain --ignore-daemonsets <NODE> \u82e5\u6b64\u547d\u4ee4\u4e00\u76f4\u7b49\u5f85 Pod \u88ab\u9a71\u9010\uff0c\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\u5f3a\u5236\u5220\u9664\u88ab\u9a71\u9010\u7684 Pod\uff1a kubectl get pod -A --field-selector = spec.nodeName = <NODE> --no-headers | \\ awk '$4==\"Terminating\" {print $1\" \"$2}' | \\ while read s ; do kubectl delete pod --force -n $s ; done \u91cd\u542f\u8282\u70b9 \u00b6 \u5728\u8282\u70b9\u4e2d\u6267\u884c\uff1a shutdown -r 0 \u6062\u590d\u8282\u70b9 \u00b6 kubectl uncordon <NODE> \u5378\u8f7d Calico \u00b6 \u5220\u9664 k8s \u8d44\u6e90 \u00b6 kubectl -n kube-system delete deploy calico-kube-controllers kubectl -n kube-system delete ds calico-node kubectl -n kube-system delete cm calico-config # \u5220\u9664 CRD \u53ca\u76f8\u5173\u8d44\u6e90 kubectl get crd -o jsonpath = '{range .items[*]}{.metadata.name}{\"\\n\"}{end}' | while read crd ; do if ! echo $crd | grep '.crd.projectcalico.org$' >/dev/null ; then continue fi for name in $( kubectl get $crd -o jsonpath = '{.items[*].metadata.name}' ) ; do kubectl delete $crd $name done kubectl delete crd $crd done # \u5176\u5b83\u8d44\u6e90 kubectl delete --ignore-not-found clusterrolebinding calico-node calico-kube-controllers kubectl delete --ignore-not-found clusterrole calico-node calico-kube-controllers kubectl delete --ignore-not-found sa -n kube-system calico-kube-controllers calico-node kubectl delete --ignore-not-found pdb -n kube-system calico-kube-controllers \u6e05\u7406\u8282\u70b9\u6587\u4ef6 \u00b6 \u5728\u6bcf\u4e2a\u8282\u70b9\u4e2d\u6267\u884c\uff1a rm -f /etc/cni/net.d/10-calico.conflist /etc/cni/net.d/calico-kubeconfig rm -f /opt/cni/bin/calico /opt/cni/bin/calico-ipam \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5378\u8f7d Calico \u5b89\u88c5 Kube-OVN"},{"location":"ops/from-calico/#calico-kube-ovn","text":"\u82e5 Kubernetes \u96c6\u7fa4\u5df2\u5b89\u88c5 Calico \u9700\u8981\u53d8\u66f4\u4e3a Kube-OVN \u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\u3002 \u672c\u6587\u4ee5 Calico v3.24.1 \u4e3a\u4f8b\uff0c\u5176\u5b83 Calico \u7248\u672c\u9700\u8981\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u8fdb\u884c\u8c03\u6574\u3002","title":"\u5378\u8f7d Calico \u5b89\u88c5 Kube-OVN"},{"location":"ops/from-calico/#_1","text":"\u4e3a\u4e86\u4fdd\u8bc1\u5207\u6362 CNI \u8fc7\u7a0b\u4e2d\u96c6\u7fa4\u7f51\u7edc\u4fdd\u6301\u7545\u901a\uff0cCalico ippool \u9700\u8981\u5f00\u542f nat outgoing\uff0c \u6216 \u5728\u6240\u6709\u8282\u70b9\u4e0a\u5173\u95ed rp_filter\uff1a sysctl net.ipv4.conf.all.rp_filter = 0 sysctl net.ipv4.conf.default.rp_filter = 0 # IPIP \u6a21\u5f0f sysctl net.ipv4.conf.tunl0.rp_filter = 0 # VXLAN \u6a21\u5f0f sysctl net.ipv4.conf.vxlan/calico.rp_filter = 0 # \u8def\u7531\u6a21\u5f0f\uff0ceth0 \u9700\u8981\u4fee\u6539\u4e3a\u5b9e\u9645\u4f7f\u7528\u7684\u7f51\u5361 sysctl net.ipv4.conf.eth0.rp_filter = 0","title":"\u51c6\u5907\u5de5\u4f5c"},{"location":"ops/from-calico/#kube-ovn","text":"","title":"\u90e8\u7f72 Kube-OVN"},{"location":"ops/from-calico/#_2","text":"wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c"},{"location":"ops/from-calico/#_3","text":"\u5c06\u5b89\u88c5\u811a\u672c\u4e2d\u91cd\u5efa Pod \u7684\u90e8\u5206\u5220\u9664\uff1a echo \"[Step 4/6] Delete pod that not in host network mode\" for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done \u6309\u9700\u4fee\u6539\u4ee5\u4e0b\u914d\u7f6e\uff1a REGISTRY = \"kubeovn\" # \u955c\u50cf\u4ed3\u5e93\u5730\u5740 VERSION = \"v1.12.0\" # \u955c\u50cf\u7248\u672c/Tag POD_CIDR = \"10.16.0.0/16\" # \u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e0d\u8981\u548c SVC/NODE/JOIN CIDR \u91cd\u53e0 SVC_CIDR = \"10.96.0.0/12\" # \u9700\u8981\u548c apiserver \u7684 service-cluster-ip-range \u4fdd\u6301\u4e00\u81f4 JOIN_CIDR = \"100.64.0.0/16\" # Pod \u548c\u4e3b\u673a\u901a\u4fe1\u7f51\u7edc CIDR\uff0c\u4e0d\u8981\u548c SVC/NODE/POD CIDR \u91cd\u53e0 LABEL = \"node-role.kubernetes.io/master\" # \u90e8\u7f72 OVN DB \u8282\u70b9\u7684\u6807\u7b7e IFACE = \"\" # \u5bb9\u5668\u7f51\u7edc\u6240\u4f7f\u7528\u7684\u7684\u5bbf\u4e3b\u673a\u7f51\u5361\u540d\uff0c\u5982\u679c\u4e3a\u7a7a\u5219\u4f7f\u7528 Kubernetes \u4e2d\u7684 Node IP \u6240\u5728\u7f51\u5361 TUNNEL_TYPE = \"geneve\" # \u96a7\u9053\u5c01\u88c5\u534f\u8bae\uff0c\u53ef\u9009 geneve, vxlan \u6216 stt\uff0cstt \u9700\u8981\u5355\u72ec\u7f16\u8bd1 ovs \u5185\u6838\u6a21\u5757 \u6ce8\u610f \uff1aPOD_CIDR \u53ca JOIN_CIDR \u4e0d\u53ef\u4e0e Calico ippool \u7684 CIDR \u51b2\u7a81\uff0c\u4e14 POD_CIDR \u9700\u8981\u5305\u542b\u8db3\u591f\u591a\u7684 IP \u6765\u5bb9\u7eb3\u96c6\u7fa4\u4e2d\u5df2\u6709\u7684 Pod\u3002","title":"\u4fee\u6539\u5b89\u88c5\u811a\u672c"},{"location":"ops/from-calico/#_4","text":"bash install.sh","title":"\u6267\u884c\u5b89\u88c5\u811a\u672c"},{"location":"ops/from-calico/#_5","text":"\u6309\u7167\u4ee5\u4e0b\u65b9\u6cd5\u4e3a\u6bcf\u4e2a\u8282\u70b9\u9010\u4e2a\u8fdb\u884c\u8fc1\u79fb\u3002 \u6ce8\u610f \uff1a\u547d\u4ee4\u4e2d\u7684 \\<NODE> \u9700\u8981\u66ff\u6362\u4e3a\u8282\u70b9\u540d\u79f0\u3002","title":"\u9010\u4e2a\u8282\u70b9\u8fc1\u79fb"},{"location":"ops/from-calico/#_6","text":"kubectl drain --ignore-daemonsets <NODE> \u82e5\u6b64\u547d\u4ee4\u4e00\u76f4\u7b49\u5f85 Pod \u88ab\u9a71\u9010\uff0c\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\u5f3a\u5236\u5220\u9664\u88ab\u9a71\u9010\u7684 Pod\uff1a kubectl get pod -A --field-selector = spec.nodeName = <NODE> --no-headers | \\ awk '$4==\"Terminating\" {print $1\" \"$2}' | \\ while read s ; do kubectl delete pod --force -n $s ; done","title":"\u9a71\u9010\u8282\u70b9"},{"location":"ops/from-calico/#_7","text":"\u5728\u8282\u70b9\u4e2d\u6267\u884c\uff1a shutdown -r 0","title":"\u91cd\u542f\u8282\u70b9"},{"location":"ops/from-calico/#_8","text":"kubectl uncordon <NODE>","title":"\u6062\u590d\u8282\u70b9"},{"location":"ops/from-calico/#calico","text":"","title":"\u5378\u8f7d Calico"},{"location":"ops/from-calico/#k8s","text":"kubectl -n kube-system delete deploy calico-kube-controllers kubectl -n kube-system delete ds calico-node kubectl -n kube-system delete cm calico-config # \u5220\u9664 CRD \u53ca\u76f8\u5173\u8d44\u6e90 kubectl get crd -o jsonpath = '{range .items[*]}{.metadata.name}{\"\\n\"}{end}' | while read crd ; do if ! echo $crd | grep '.crd.projectcalico.org$' >/dev/null ; then continue fi for name in $( kubectl get $crd -o jsonpath = '{.items[*].metadata.name}' ) ; do kubectl delete $crd $name done kubectl delete crd $crd done # \u5176\u5b83\u8d44\u6e90 kubectl delete --ignore-not-found clusterrolebinding calico-node calico-kube-controllers kubectl delete --ignore-not-found clusterrole calico-node calico-kube-controllers kubectl delete --ignore-not-found sa -n kube-system calico-kube-controllers calico-node kubectl delete --ignore-not-found pdb -n kube-system calico-kube-controllers","title":"\u5220\u9664 k8s \u8d44\u6e90"},{"location":"ops/from-calico/#_9","text":"\u5728\u6bcf\u4e2a\u8282\u70b9\u4e2d\u6267\u884c\uff1a rm -f /etc/cni/net.d/10-calico.conflist /etc/cni/net.d/calico-kubeconfig rm -f /opt/cni/bin/calico /opt/cni/bin/calico-ipam \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6e05\u7406\u8282\u70b9\u6587\u4ef6"},{"location":"ops/kubectl-ko/","text":"kubectl \u63d2\u4ef6\u4f7f\u7528 \u00b6 \u4e3a\u4e86\u65b9\u4fbf\u65e5\u5e38\u7684\u8fd0\u7ef4\u64cd\u4f5c\uff0cKube-OVN \u63d0\u4f9b\u4e86 kubectl \u63d2\u4ef6\u5de5\u5177\uff0c\u7f51\u7edc\u7ba1\u7406\u5458 \u53ef\u4ee5\u901a\u8fc7\u8be5\u547d\u4ee4\u8fdb\u884c\u65e5\u5e38\u64cd\u4f5c\uff0c\u4f8b\u5982\uff1a\u67e5\u770b OVN \u6570\u636e\u5e93\u4fe1\u606f\u548c\u72b6\u6001\uff0cOVN \u6570\u636e\u5e93 \u5907\u4efd\u548c\u6062\u590d\uff0cOVS \u76f8\u5173\u4fe1\u606f\u67e5\u770b\uff0ctcpdump \u7279\u5b9a\u5bb9\u5668\uff0c\u7279\u5b9a\u94fe\u8def\u903b\u8f91\u62d3\u6251\u5c55\u793a\uff0c \u7f51\u7edc\u95ee\u9898\u8bca\u65ad\u548c\u6027\u80fd\u4f18\u5316\u3002 \u63d2\u4ef6\u5b89\u88c5 \u00b6 Kube-OVN \u5b89\u88c5\u65f6\u9ed8\u8ba4\u4f1a\u90e8\u7f72\u63d2\u4ef6\u5230\u6bcf\u4e2a\u8282\u70b9\uff0c\u82e5\u6267\u884c kubectl \u7684\u673a\u5668\u4e0d\u5728\u96c6\u7fa4\u5185\uff0c \u6216\u9700\u8981\u91cd\u88c5\u63d2\u4ef6\uff0c\u53ef\u53c2\u8003\u4e0b\u9762\u7684\u6b65\u9aa4\uff1a \u4e0b\u8f7d kubectl-ko \u6587\u4ef6\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/kubectl-ko \u5c06\u8be5\u6587\u4ef6\u79fb\u52a8\u81f3 $PATH \u76ee\u5f55\u4e0b\uff1a mv kubectl-ko /usr/local/bin/kubectl-ko \u589e\u52a0\u53ef\u6267\u884c\u6743\u9650\uff1a chmod +x /usr/local/bin/kubectl-ko \u68c0\u67e5\u63d2\u4ef6\u662f\u5426\u53ef\u4ee5\u6b63\u5e38\u4f7f\u7528\uff1a # kubectl plugin list The following compatible plugins are available: /usr/local/bin/kubectl-ko \u63d2\u4ef6\u4f7f\u7528 \u00b6 \u8fd0\u884c kubectl ko \u4f1a\u5c55\u793a\u8be5\u63d2\u4ef6\u6240\u6709\u53ef\u7528\u7684\u547d\u4ee4\u548c\u7528\u6cd5\u63cf\u8ff0\uff0c\u5982\u4e0b\u6240\u793a\uff1a # kubectl ko kubectl ko { subcommand } [ option... ] Available Subcommands: [ nb | sb ] [ status | kick | backup | dbstatus | restore ] ovn-db operations show cluster status, kick stale server, backup database, get db consistency status or restore ovn nb db when met 'inconsistent data' error nbctl [ ovn-nbctl options ... ] invoke ovn-nbctl sbctl [ ovn-sbctl options ... ] invoke ovn-sbctl vsctl { nodeName } [ ovs-vsctl options ... ] invoke ovs-vsctl on the specified node ofctl { nodeName } [ ovs-ofctl options ... ] invoke ovs-ofctl on the specified node dpctl { nodeName } [ ovs-dpctl options ... ] invoke ovs-dpctl on the specified node appctl { nodeName } [ ovs-appctl options ... ] invoke ovs-appctl on the specified node tcpdump { namespace/podname } [ tcpdump options ... ] capture pod traffic trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply echo \" diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] diagnose connectivity of all nodes or a specific node or specify subnet's ds pod or IPPorts like 'tcp-172.18.0.2-53,udp-172.18.0.3-53'\" tuning { install-fastpath | local-install-fastpath | remove-fastpath | install-stt | local-install-stt | remove-stt } { centos7 | centos8 }} [ kernel-devel-version ] deploy kernel optimisation components to the system reload restart all kube-ovn components log { kube-ovn | ovn | ovs | linux | all } save log to ./kubectl-ko-log/ perf [ image ] performance test default image is kubeovn/test:v1.12.0 \u4e0b\u9762\u5c06\u4ecb\u7ecd\u6bcf\u4e2a\u547d\u4ee4\u7684\u5177\u4f53\u529f\u80fd\u548c\u4f7f\u7528\u3002 [nb | sb] [status | kick | backup | dbstatus | restore] \u00b6 \u8be5\u5b50\u547d\u4ee4\u4e3b\u8981\u5bf9 OVN \u5317\u5411\u6216\u5357\u5411\u6570\u636e\u5e93\u8fdb\u884c\u64cd\u4f5c\uff0c\u5305\u62ec\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u67e5\u770b\uff0c\u6570\u636e\u5e93\u8282\u70b9\u4e0b\u7ebf\uff0c \u6570\u636e\u5e93\u5907\u4efd\uff0c\u6570\u636e\u5e93\u5b58\u50a8\u72b6\u6001\u67e5\u770b\u548c\u6570\u636e\u5e93\u4fee\u590d\u3002 \u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u67e5\u770b \u00b6 \u8be5\u547d\u4ee4\u4f1a\u5728\u5bf9\u5e94 OVN \u6570\u636e\u5e93\u7684 leader \u8282\u70b9\u6267\u884c ovs-appctl cluster/status \u5c55\u793a\u96c6\u7fa4\u72b6\u6001: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 280309 ms ago, reason: timeout Last Election won: 280309 ms ago Election timer: 5000 Log: [ 139 , 139 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-8723 ->8723 <-85d6 ->85d6 Disconnections: 0 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 8723 ( 8723 at tcp: [ 172 .18.0.3 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 138 status: ok \u82e5 Server \u4e0b\u7684 match_index \u51fa\u73b0\u8f83\u5927\u5dee\u522b\uff0c\u4e14 last msg \u65f6\u95f4\u8f83\u957f\u5219\u5bf9\u5e94 Server \u53ef\u80fd\u957f\u65f6\u95f4\u6ca1\u6709\u54cd\u5e94\uff0c \u9700\u8981\u8fdb\u4e00\u6b65\u67e5\u770b\u3002 \u6570\u636e\u5e93\u8282\u70b9\u4e0b\u7ebf \u00b6 \u8be5\u547d\u4ee4\u4f1a\u5c06\u67d0\u4e2a\u8282\u70b9\u4ece OVN \u6570\u636e\u5e93\u4e2d\u79fb\u9664\uff0c\u5728\u8282\u70b9\u4e0b\u7ebf\u6216\u66f4\u6362\u8282\u70b9\u65f6\u9700\u8981\u7528\u5230\u3002 \u4e0b\u9762\u5c06\u4ee5\u4e0a\u4e00\u6761\u547d\u4ee4\u6240\u67e5\u770b\u5230\u7684\u96c6\u7fa4\u72b6\u6001\u4e3a\u4f8b\uff0c\u4e0b\u7ebf 172.18.0.3 \u8282\u70b9: # kubectl ko nb kick 8723 started removal \u518d\u6b21\u67e5\u770b\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u786e\u8ba4\u8282\u70b9\u5df2\u79fb\u9664\uff1a # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 324356 ms ago, reason: timeout Last Election won: 324356 ms ago Election timer: 5000 Log: [ 140 , 140 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-85d6 ->85d6 Disconnections: 2 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 140 match_index = 139 last msg 848 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 139 status: ok \u6570\u636e\u5e93\u5907\u4efd \u00b6 \u8be5\u5b50\u547d\u4ee4\u4f1a\u5907\u4efd\u5f53\u524d OVN \u6570\u636e\u5e93\u81f3\u672c\u5730\uff0c\u53ef\u7528\u4e8e\u707e\u5907\u548c\u6062\u590d\uff1a # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup \u6570\u636e\u5e93\u5b58\u50a8\u72b6\u6001\u67e5\u770b \u00b6 \u8be5\u547d\u4ee4\u7528\u6765\u67e5\u770b\u6570\u636e\u5e93\u6587\u4ef6\u662f\u5426\u5b58\u5728\u635f\u574f\uff1a # kubectl ko nb dbstatus status: ok \u82e5\u5f02\u5e38\u5219\u663e\u793a inconsistent data \u9700\u8981\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u8fdb\u884c\u4fee\u590d\u3002 \u6570\u636e\u5e93\u4fee\u590d \u00b6 \u82e5\u6570\u636e\u5e93\u72b6\u6001\u8fdb\u5165 inconsistent data \u53ef\u4f7f\u7528\u8be5\u547d\u4ee4\u8fdb\u884c\u4fee\u590d\uff1a # kubectl ko nb restore deployment.apps/ovn-central scaled ovn-central original replicas is 3 first nodeIP is 172 .18.0.5 ovs-ovn pod on node 172 .18.0.5 is ovs-ovn-8jxv9 ovs-ovn pod on node 172 .18.0.3 is ovs-ovn-sjzb6 ovs-ovn pod on node 172 .18.0.4 is ovs-ovn-t87zk backup nb db file restore nb db file, operate in pod ovs-ovn-8jxv9 deployment.apps/ovn-central scaled finish restore nb db file and ovn-central replicas recreate ovs-ovn pods pod \"ovs-ovn-8jxv9\" deleted pod \"ovs-ovn-sjzb6\" deleted pod \"ovs-ovn-t87zk\" deleted [nbctl | sbctl] [options ...] \u00b6 \u8be5\u5b50\u547d\u4ee4\u4f1a\u76f4\u63a5\u8fdb\u5165 OVN \u5317\u5411\u6570\u636e\u5e93\u6216\u5357\u5411\u6570\u636e\u5e93 \u7684 leader \u8282\u70b9\u5206\u522b\u6267\u884c ovn-nbctl \u548c ovn-sbctl \u547d\u4ee4\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVN \u7684\u5b98\u65b9\u6587\u6863 ovn-nbctl(8) \u548c ovn-sbctl(8) \u3002 # kubectl ko nbctl show switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port node-liumengxin-ovn2-192.168.137.177 addresses: [ \"00:00:00:D9:58:B8 100.64.0.3\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port alertmanager-main-0.monitoring addresses: [ \"00:00:00:6C:DF:A3 10.16.0.19\" ] port kube-state-metrics-5d6885d89-4nf8h.monitoring addresses: [ \"00:00:00:6F:02:1C 10.16.0.15\" ] port fake-kubelet-67c55dfd89-pv86k.kube-system addresses: [ \"00:00:00:5C:12:E8 10.16.19.177\" ] port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] vsctl {nodeName} [options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-vsctl \u547d\u4ee4\uff0c\u67e5\u8be2\u5e76\u914d\u7f6e vswitchd \u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-vsctl(8) \u3002 # kubectl ko vsctl kube-ovn-01 show 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port mirror0 Interface mirror0 type: internal Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" ofctl {nodeName} [options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-ofctl \u547d\u4ee4\uff0c\u67e5\u8be2\u6216\u7ba1\u7406 OpenFlow\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-ofctl(8) \u3002 # kubectl ko ofctl kube-ovn-01 dump-flows br-int NXST_FLOW reply ( xid = 0x4 ) : flags =[ more ] cookie = 0xcf3429e6, duration = 671791 .432s, table = 0 , n_packets = 0 , n_bytes = 0 , idle_age = 65534 , hard_age = 65534 , priority = 100 ,in_port = 2 actions = load:0x4->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x1->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xc91413c6, duration = 671791 .431s, table = 0 , n_packets = 907489 , n_bytes = 99978275 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 7 actions = load:0x1->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x4->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xf180459, duration = 671791 .431s, table = 0 , n_packets = 17348582 , n_bytes = 2667811214 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 6317 actions = load:0xa->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x9->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0x7806dd90, duration = 671791 .431s, table = 0 , n_packets = 3235428 , n_bytes = 833821312 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 1 actions = load:0xd->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x3->NXM_NX_REG14 [] ,resubmit ( ,8 ) ... dpctl {nodeName} [options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-dpctl \u547d\u4ee4\uff0c\u67e5\u8be2\u6216\u7ba1\u7406 OVS datapath\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-dpctl(8) \u3002 # kubectl ko dpctl kube-ovn-01 show system@ovs-system: lookups: hit:350805055 missed:21983648 lost:73 flows: 105 masks: hit:1970748791 total:22 hit/pkt:5.29 port 0 : ovs-system ( internal ) port 1 : ovn0 ( internal ) port 2 : mirror0 ( internal ) port 3 : br-int ( internal ) port 4 : stt_sys_7471 ( stt: packet_type = ptap ) port 5 : eeb4d9e51b5d_h port 6 : a2c1a8a8b83a_h port 7 : 4fa5c4cbb1a5_h appctl {nodeName} [options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-appctl \u547d\u4ee4\uff0c\u6765\u64cd\u4f5c\u76f8\u5173 daemon \u8fdb\u7a0b\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-appctl(8) \u3002 # kubectl ko appctl kube-ovn-01 vlog/list console syslog file ------- ------ ------ backtrace OFF ERR INFO bfd OFF ERR INFO bond OFF ERR INFO bridge OFF ERR INFO bundle OFF ERR INFO bundles OFF ERR INFO ... tcpdump {namespace/podname} [tcpdump options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165 namespace/podname \u6240\u5728\u673a\u5668\u7684 kube-ovn-cni \u5bb9\u5668\uff0c\u5e76\u6267\u884c tcpdump \u6293\u53d6\u5bf9\u5e94\u5bb9\u5668 veth \u7f51\u5361 \u7aef\u7684\u6d41\u91cf\uff0c\u53ef\u4ee5\u65b9\u4fbf\u6392\u67e5\u7f51\u7edc\u76f8\u5173\u95ee\u9898\uff0c\u5982\u4e0b\u6240\u793a\uff1a # kubectl ko tcpdump default/ds1-l6n7p icmp + kubectl exec -it kube-ovn-cni-wlg4s -n kube-ovn -- tcpdump -nn -i d7176fe7b4e0_h icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on d7176fe7b4e0_h, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 06 :52:36.619688 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 1 , length 64 06 :52:36.619746 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 1 , length 64 06 :52:37.619588 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 2 , length 64 06 :52:37.619630 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 2 , length 64 06 :52:38.619933 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 3 , length 64 06 :52:38.619973 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 3 , length 64 trace [arguments ...] \u00b6 \u8be5\u547d\u4ee4\u5c06\u4f1a\u6253\u5370 Pod \u6216\u8282\u70b9\u901a\u8fc7\u7279\u5b9a\u534f\u8bae\u8bbf\u95ee\u67d0\u5730\u5740\u65f6\u5bf9\u5e94\u7684 OVN \u903b\u8f91\u6d41\u8868\u548c\u6700\u7ec8\u7684 Openflow \u6d41\u8868\uff0c \u65b9\u4fbf\u5f00\u53d1\u6216\u8fd0\u7ef4\u65f6\u5b9a\u4f4d\u6d41\u8868\u76f8\u5173\u95ee\u9898\u3002 \u652f\u6301\u7684\u547d\u4ee4\uff1a kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } kubectl ko trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } \u793a\u4f8b\uff1a # kubectl ko trace default/ds1-l6n7p 8.8.8.8 icmp + kubectl exec ovn-central-5bc494cb5-np9hm -n kube-ovn -- ovn-trace --ct = new ovn-default 'inport == \"ds1-l6n7p.default\" && ip.ttl == 64 && icmp && eth.src == 0a:00:00:10:00:05 && ip4.src == 10.16.0.4 && eth.dst == 00:00:00:B8:CA:43 && ip4.dst == 8.8.8.8' # icmp,reg14=0xf,vlan_tci=0x0000,dl_src=0a:00:00:10:00:05,dl_dst=00:00:00:b8:ca:43,nw_src=10.16.0.4,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 ingress ( dp = \"ovn-default\" , inport = \"ds1-l6n7p.default\" ) ----------------------------------------------------- 0 . ls_in_port_sec_l2 ( ovn-northd.c:4143 ) : inport == \"ds1-l6n7p.default\" && eth.src == { 0a:00:00:10:00:05 } , priority 50 , uuid 39453393 next ; 1 . ls_in_port_sec_ip ( ovn-northd.c:2898 ) : inport == \"ds1-l6n7p.default\" && eth.src == 0a:00:00:10:00:05 && ip4.src == { 10 .16.0.4 } , priority 90 , uuid 81bcd485 next ; 3 . ls_in_pre_acl ( ovn-northd.c:3269 ) : ip, priority 100 , uuid 7b4f4971 reg0 [ 0 ] = 1 ; next ; 5 . ls_in_pre_stateful ( ovn-northd.c:3396 ) : reg0 [ 0 ] == 1 , priority 100 , uuid 36cdd577 ct_next ; ct_next ( ct_state = new | trk ) ------------------------- 6 . ls_in_acl ( ovn-northd.c:3759 ) : ip && ( !ct.est || ( ct.est && ct_label.blocked == 1 )) , priority 1 , uuid 7608af5b reg0 [ 1 ] = 1 ; next ; 10 . ls_in_stateful ( ovn-northd.c:3995 ) : reg0 [ 1 ] == 1 , priority 100 , uuid 2aba1b90 ct_commit ( ct_label = 0 /0x1 ) ; next ; 16 . ls_in_l2_lkup ( ovn-northd.c:4470 ) : eth.dst == 00 :00:00:b8:ca:43, priority 50 , uuid 5c9c3c9f outport = \"ovn-default-ovn-cluster\" ; output ; ... \u82e5 trace \u5bf9\u8c61\u4e3a\u8fd0\u884c\u4e8e Underlay \u7f51\u7edc\u4e0b\u7684\u865a\u62df\u673a\uff0c\u9700\u8981\u6dfb\u52a0\u989d\u5916\u53c2\u6570\u6765\u6307\u5b9a\u76ee\u7684 Mac \u5730\u5740\uff1a kubectl ko trace default/virt-handler-7lvml 8 .8.8.8 82 :7c:9f:83:8c:01 icmp diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] \u00b6 \u8bca\u65ad\u96c6\u7fa4\u7f51\u7edc\u7ec4\u4ef6\u72b6\u6001\uff0c\u5e76\u53bb\u5bf9\u5e94\u8282\u70b9\u7684 kube-ovn-pinger \u68c0\u6d4b\u5f53\u524d\u8282\u70b9\u5230\u5176\u4ed6\u8282\u70b9\u548c\u5173\u952e\u670d\u52a1\u7684\u8fde\u901a\u6027\u548c\u7f51\u7edc\u5ef6\u8fdf\uff1a # kubectl ko diagnose all switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default port prometheus-k8s-1.monitoring addresses: [ \"00:00:00:AA:37:DF 10.16.0.23\" ] router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] Routing Policies 31000 ip4.dst == 10 .16.0.0/16 allow 31000 ip4.dst == 100 .64.0.0/16 allow 30000 ip4.dst == 192 .168.137.177 reroute 100 .64.0.3 30000 ip4.dst == 192 .168.137.178 reroute 100 .64.0.4 29000 ip4.src == $ovn .default.fake.6_ip4 reroute 100 .64.0.22 29000 ip4.src == $ovn .default.fake.7_ip4 reroute 100 .64.0.21 29000 ip4.src == $ovn .default.fake.8_ip4 reroute 100 .64.0.23 29000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 reroute 100 .64.0.4 20000 ip4.src == $ovn .default.liumengxin.ovn1.192.168.137.176_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.2 20000 ip4.src == $ovn .default.liumengxin.ovn2.192.168.137.177_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.3 20000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.4 IPv4 Routes Route Table <main>: 0 .0.0.0/0 100 .64.0.1 dst-ip UUID LB PROTO VIP IPs e9bcfd9d-793e-4431-9073-6dec96b75d71 cluster-tcp-load tcp 10 .100.209.132:10660 192 .168.137.176:10660 tcp 10 .101.239.192:6641 192 .168.137.177:6641 tcp 10 .101.240.101:3000 10 .16.0.7:3000 tcp 10 .103.184.186:6642 192 .168.137.177:6642 35d2b7a5-e3a7-485a-a4b7-b4970eb0e63b cluster-tcp-sess tcp 10 .100.158.128:8080 10 .16.0.10:8080,10.16.0.5:8080,10.16.63.30:8080 tcp 10 .107.26.215:8080 10 .16.0.19:8080,10.16.0.20:8080,10.16.0.21:8080 tcp 10 .107.26.215:9093 10 .16.0.19:9093,10.16.0.20:9093,10.16.0.21:9093 tcp 10 .98.187.99:8080 10 .16.0.22:8080,10.16.0.23:8080 tcp 10 .98.187.99:9090 10 .16.0.22:9090,10.16.0.23:9090 f43303e4-89aa-4d3e-a3dc-278a552fe27b cluster-udp-load udp 10 .96.0.10:53 10 .16.0.4:53,10.16.0.9:53 _uuid : 06776304 -5a96-43ed-90c4-c4854c251699 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn2_192.168.137.177_underlay_v6 _uuid : 62690625 -87d5-491c-8675-9fd83b1f433c addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn1_192.168.137.176_underlay_v6 _uuid : b03a9bae-94d5-4562-b34c-b5f6198e180b addresses : [ \"10.16.0.0/16\" , \"100.64.0.0/16\" , \"172.22.0.0/16\" , \"192.101.0.0/16\" ] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv4 _uuid : e1056f3a-24cc-4666-8a91-75ee6c3c2426 addresses : [] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv6 _uuid : 3e5d5fff-e670-47b2-a2f5-a39f4698a8c5 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn3_192.168.137.178_underlay_v6 _uuid : 2d85dbdc-d0db-4abe-b19e-cc806d32b492 action : drop direction : from-lport external_ids : {} label : 0 log : false match : \"inport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] _uuid : de790cc8-f155-405f-bb32-5a51f30c545f action : drop direction : to-lport external_ids : {} label : 0 log : false match : \"outport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] Chassis \"e15ed4d4-1780-4d50-b09e-ea8372ed48b8\" hostname: liumengxin-ovn1-192.168.137.176 Encap stt ip: \"192.168.137.176\" options: { csum = \"true\" } Port_Binding node-liumengxin-ovn1-192.168.137.176 Port_Binding perf-6vxkn.default Port_Binding kube-state-metrics-5d6885d89-4nf8h.monitoring Port_Binding alertmanager-main-0.monitoring Port_Binding kube-ovn-pinger-6ftdf.kube-system Port_Binding fake-kubelet-67c55dfd89-pv86k.kube-system Port_Binding prometheus-k8s-0.monitoring Chassis \"eef07da1-f8ad-4775-b14d-bd6a3b4eb0d5\" hostname: liumengxin-ovn3-192.168.137.178 Encap stt ip: \"192.168.137.178\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-7twb4.kube-system Port_Binding prometheus-adapter-86df476d87-rl88g.monitoring Port_Binding prometheus-k8s-1.monitoring Port_Binding node-liumengxin-ovn3-192.168.137.178 Port_Binding perf-ff475.default Port_Binding alertmanager-main-1.monitoring Port_Binding blackbox-exporter-676d976865-tvsjd.monitoring Chassis \"efa253c9-494d-4719-83ae-b48ab0f11c03\" hostname: liumengxin-ovn2-192.168.137.177 Encap stt ip: \"192.168.137.177\" options: { csum = \"true\" } Port_Binding grafana-6c4c6b8fb7-pzd2c.monitoring Port_Binding node-liumengxin-ovn2-192.168.137.177 Port_Binding alertmanager-main-2.monitoring Port_Binding coredns-6789c94dd8-9jqsz.kube-system Port_Binding coredns-6789c94dd8-25d4r.kube-system Port_Binding prometheus-operator-7bbc99fc8b-wgjm4.monitoring Port_Binding prometheus-adapter-86df476d87-gdxmc.monitoring Port_Binding perf-fjnws.default Port_Binding kube-ovn-pinger-vh2xg.kube-system ds kube-proxy ready kube-proxy ready deployment ovn-central ready deployment kube-ovn-controller ready ds kube-ovn-cni ready ds ovs-ovn ready deployment coredns ready ovn-nb leader check ok ovn-sb leader check ok ovn-northd leader check ok ### kube-ovn-controller recent log ### start to diagnose node liumengxin-ovn1-192.168.137.176 #### ovn-controller log: 2022 -06-03T00:56:44.897Z | 16722 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:06:44.912Z | 16723 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:16:44.925Z | 16724 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:26:44.936Z | 16725 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:36:44.959Z | 16726 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:46:44.974Z | 16727 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:56:44.988Z | 16728 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:06:45.001Z | 16729 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:16:45.025Z | 16730 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:26:45.040Z | 16731 | inc_proc_eng | INFO | User triggered force recompute. #### ovs-vswitchd log: 2022 -06-02T23:03:00.137Z | 00079 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:f9d1 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-02T23:23:31.840Z | 00080 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:15b2 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:09:15.659Z | 00081 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:dc:e3:63,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.63.30,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:e5a5 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x150017000004,src = 192 .168.137.178,dst = 192 .168.137.176,ttl = 64 ,tp_src = 9239 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.63.30,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:30:13.409Z | 00064 | dpif ( handler2 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:6b4a with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T02:02:33.832Z | 00082 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:a819 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 #### ovs-vsctl show results: 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port \"04d03360e9a0_h\" Interface \"04d03360e9a0_h\" Port eeb4d9e51b5d_h Interface eeb4d9e51b5d_h Port mirror0 Interface mirror0 type: internal Port \"8e5d887ccd80_h\" Interface \"8e5d887ccd80_h\" Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port \"17512d5be1f1_h\" Interface \"17512d5be1f1_h\" Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" #### pinger diagnose results: I0603 10 :35:04.349404 17619 pinger.go:19 ] ------------------------------------------------------------------------------- Kube-OVN: Version: v1.12.0 Build: 2022 -04-24_08:02:50 Commit: git-73f9d15 Go Version: go1.17.8 Arch: amd64 ------------------------------------------------------------------------------- I0603 10 :35:04.376797 17619 config.go:166 ] pinger config is & { KubeConfigFile: KubeClient:0xc000493380 Port:8080 DaemonSetNamespace:kube-system DaemonSetName:kube-ovn-pinger Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:liumengxin-ovn1-192.168.137.176 HostIP:192.168.137.176 PodName:kube-ovn-pinger-6ftdf PodIP:10.16.0.10 PodProtocols: [ IPv4 ] ExternalAddress: NetworkMode:kube-ovn PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid } I0603 10 :35:04.449166 17619 exporter.go:75 ] liumengxin-ovn1-192.168.137.176: exporter connect successfully I0603 10 :35:04.554011 17619 ovn.go:21 ] ovs-vswitchd and ovsdb are up I0603 10 :35:04.651293 17619 ovn.go:33 ] ovn_controller is up I0603 10 :35:04.651342 17619 ovn.go:39 ] start to check port binding I0603 10 :35:04.749613 17619 ovn.go:135 ] chassis id is 1d7f3d6c-eec5-4b3c-adca-2969d9cdfd80 I0603 10 :35:04.763487 17619 ovn.go:49 ] port in sb is [ node-liumengxin-ovn1-192.168.137.176 perf-6vxkn.default kube-state-metrics-5d6885d89-4nf8h.monitoring alertmanager-main-0.monitoring kube-ovn-pinger-6ftdf.kube-system fake-kubelet-67c55dfd89-pv86k.kube-system prometheus-k8s-0.monitoring ] I0603 10 :35:04.763583 17619 ovn.go:61 ] ovs and ovn-sb binding check passed I0603 10 :35:05.049309 17619 ping.go:259 ] start to check apiserver connectivity I0603 10 :35:05.053666 17619 ping.go:268 ] connect to apiserver success in 4 .27ms I0603 10 :35:05.053786 17619 ping.go:129 ] start to check pod connectivity I0603 10 :35:05.249590 17619 ping.go:159 ] ping pod: kube-ovn-pinger-6ftdf 10 .16.0.10, count: 3 , loss count 0 , average rtt 16 .30ms I0603 10 :35:05.354135 17619 ping.go:159 ] ping pod: kube-ovn-pinger-7twb4 10 .16.63.30, count: 3 , loss count 0 , average rtt 1 .81ms I0603 10 :35:05.458460 17619 ping.go:159 ] ping pod: kube-ovn-pinger-vh2xg 10 .16.0.5, count: 3 , loss count 0 , average rtt 1 .92ms I0603 10 :35:05.458523 17619 ping.go:83 ] start to check node connectivity \u5982\u679c diagnose \u7684\u76ee\u6807\u6307\u5b9a\u4e3a subnet \u8be5\u811a\u672c\u4f1a\u5728 subnet \u4e0a\u5efa\u7acb daemonset\uff0c\u7531 kube-ovn-pinger \u53bb\u63a2\u6d4b\u8fd9\u4e2a daemonset \u7684\u6240\u6709 pod \u7684\u8fde\u901a\u6027\u548c\u7f51\u7edc\u5ef6\u65f6\uff0c\u6d4b\u8bd5\u5b8c\u540e\u81ea\u52a8\u9500\u6bc1\u8be5 daemonset\u3002 \u5982\u679c diagnose \u7684\u76ee\u6807\u6307\u5b9a\u4e3a IPPorts \u8be5\u811a\u672c\u4f1a\u8ba9\u6bcf\u4e2a kube-ovn-pinger pod \u53bb\u63a2\u6d4b\u76ee\u6807\u534f\u8bae\uff0cIP\uff0cPort \u662f\u5426\u53ef\u8fbe\u3002 tuning {install-fastpath|local-install-fastpath|remove-fastpath|install-stt|local-install-stt|remove-stt} {centos7|centos8}} [kernel-devel-version] \u00b6 \u8be5\u547d\u4ee4\u6267\u884c\u6027\u80fd\u8c03\u4f18\u76f8\u5173\u64cd\u4f5c\uff0c\u5177\u4f53\u4f7f\u7528\u8bf7\u53c2\u8003 \u6027\u80fd\u8c03\u4f18 \u3002 reload \u00b6 \u8be5\u547d\u4ee4\u91cd\u542f\u6240\u6709 Kube-OVN \u76f8\u5173\u7ec4\u4ef6\uff1a # kubectl ko reload pod \"ovn-central-8684dd94bd-vzgcr\" deleted Waiting for deployment \"ovn-central\" rollout to finish: 0 of 1 updated replicas are available... deployment \"ovn-central\" successfully rolled out pod \"ovs-ovn-bsnvz\" deleted pod \"ovs-ovn-m9b98\" deleted pod \"kube-ovn-controller-8459db5ff4-64c62\" deleted Waiting for deployment \"kube-ovn-controller\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out pod \"kube-ovn-cni-2klnh\" deleted pod \"kube-ovn-cni-t2jz4\" deleted Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-cni\" successfully rolled out pod \"kube-ovn-pinger-ln72z\" deleted pod \"kube-ovn-pinger-w8lrk\" deleted Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-pinger\" successfully rolled out pod \"kube-ovn-monitor-7fb67d5488-7q6zb\" deleted Waiting for deployment \"kube-ovn-monitor\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-monitor\" successfully rolled out log \u00b6 \u4f7f\u7528\u8be5\u547d\u4ee4\u4f1a\u6293\u53d6 kube-ovn \u6240\u6709\u8282\u70b9\u4e0a\u7684 Kube-OVN\uff0cOVN\uff0cOpenvswitch \u7684 log \u4ee5\u53ca linux \u5e38\u7528\u7684\u4e00\u4e9b debug \u4fe1\u606f\u3002 # kubectl ko log all Collecting kube-ovn logging files Collecting ovn logging files Collecting openvswitch logging files Collecting linux dmesg files Collecting linux iptables-legacy files Collecting linux iptables-nft files Collecting linux route files Collecting linux link files Collecting linux neigh files Collecting linux memory files Collecting linux top files Collecting linux sysctl files Collecting linux netstat files Collecting linux addr files Collecting linux ipset files Collecting linux tcp files Collected files have been saved in the directory /root/kubectl-ko-log \u76ee\u5f55\u5982\u4e0b\uff1a # tree kubectl-ko-log/ kubectl-ko-log/ | -- kube-ovn-control-plane | | -- kube-ovn | | | -- kube-ovn-cni.log | | | -- kube-ovn-monitor.log | | ` -- kube-ovn-pinger.log | | -- linux | | | -- addr.log | | | -- dmesg.log | | | -- ipset.log | | | -- iptables-legacy.log | | | -- iptables-nft.log | | | -- link.log | | | -- memory.log | | | -- neigh.log | | | -- netstat.log | | | -- route.log | | | -- sysctl.log | | | -- tcp.log | | ` -- top.log | | -- openvswitch | | | -- ovs-vswitchd.log | | ` -- ovsdb-server.log | ` -- ovn | | -- ovn-controller.log | | -- ovn-northd.log | | -- ovsdb-server-nb.log | ` -- ovsdb-server-sb.log perf [image] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u53bb\u6d4b\u8bd5 Kube-OVN \u7684\u4e00\u4e9b\u6027\u80fd\u6307\u6807\u5982\u4e0b\uff1a \u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u6307\u6807\uff1b Hostnetwork \u7f51\u7edc\u6027\u80fd\u6307\u6807\uff1b \u5bb9\u5668\u7f51\u7edc\u7ec4\u64ad\u62a5\u6587\u6027\u80fd\u6307\u6807\uff1b OVN-NB, OVN-SB, OVN-Northd leader \u5220\u9664\u6062\u590d\u6240\u9700\u65f6\u95f4\u3002 \u53c2\u6570 image \u7528\u4e8e\u6307\u5b9a\u6027\u80fd\u6d4b\u8bd5 pod \u6240\u7528\u7684\u955c\u50cf\uff0c\u9ed8\u8ba4\u60c5\u51b5\u4e0b\u662f kubeovn/test:v1.12.0 , \u8bbe\u7f6e\u8be5\u53c2\u6570\u4e3b\u8981\u662f\u4e3a\u4e86\u79bb\u7ebf\u573a\u666f\uff0c\u5c06\u955c\u50cf\u62c9\u5230\u5185\u7f51\u73af\u5883\u53ef\u80fd\u4f1a\u6709\u955c\u50cf\u540d\u53d8\u5316\u3002 # kubectl ko perf ============================== Prepareing Performance Test Resources =============================== pod/test-client created pod/test-host-client created pod/test-server created pod/test-host-server created service/test-server created pod/test-client condition met pod/test-host-client condition met pod/test-host-server condition met pod/test-server condition met ==================================================================================================== ============================ Start Pod Network Unicast Performance Test ============================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 82 .8 us 97 .7 Mbits/sec 67 .6 us ( 0 % ) 8 .42 Mbits/sec 128 85 .4 us 167 Mbits/sec 67 .2 us ( 0 % ) 17 .2 Mbits/sec 512 85 .8 us 440 Mbits/sec 68 .7 us ( 0 % ) 68 .4 Mbits/sec 1k 85 .1 us 567 Mbits/sec 68 .7 us ( 0 % ) 134 Mbits/sec 4k 138 us 826 Mbits/sec 78 .1 us ( 1 .4% ) 503 Mbits/sec ==================================================================================================== =============================== Start Host Network Performance Test ================================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 49 .7 us 120 Mbits/sec 37 .9 us ( 0 % ) 18 .6 Mbits/sec 128 49 .7 us 200 Mbits/sec 38 .1 us ( 0 % ) 35 .5 Mbits/sec 512 51 .9 us 588 Mbits/sec 38 .9 us ( 0 % ) 142 Mbits/sec 1k 51 .7 us 944 Mbits/sec 37 .2 us ( 0 % ) 279 Mbits/sec 4k 74 .9 us 1 .66 Gbits/sec 39 .9 us ( 0 % ) 1 .20 Gbits/sec ==================================================================================================== ============================== Start Service Network Performance Test ============================== Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 111 us 96 .3 Mbits/sec 88 .4 us ( 0 % ) 7 .59 Mbits/sec 128 83 .7 us 150 Mbits/sec 69 .2 us ( 0 % ) 16 .9 Mbits/sec 512 87 .4 us 374 Mbits/sec 75 .8 us ( 0 % ) 60 .9 Mbits/sec 1k 88 .2 us 521 Mbits/sec 73 .1 us ( 0 % ) 123 Mbits/sec 4k 148 us 813 Mbits/sec 77 .6 us ( 0 .0044% ) 451 Mbits/sec ==================================================================================================== =========================== Start Pod Multicast Network Performance Test =========================== Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .014 ms ( 0 .17% ) 5 .80 Mbits/sec 128 0 .012 ms ( 0 % ) 11 .4 Mbits/sec 512 0 .016 ms ( 0 % ) 46 .1 Mbits/sec 1k 0 .023 ms ( 0 .073% ) 89 .8 Mbits/sec 4k 0 .035 ms ( 1 .3% ) 126 Mbits/sec ==================================================================================================== ============================= Start Host Multicast Network Performance ============================= Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .007 ms ( 0 % ) 9 .95 Mbits/sec 128 0 .005 ms ( 0 % ) 21 .8 Mbits/sec 512 0 .008 ms ( 0 % ) 86 .8 Mbits/sec 1k 0 .013 ms ( 0 .045% ) 168 Mbits/sec 4k 0 .010 ms ( 0 .31% ) 242 Mbits/sec ==================================================================================================== ================================== Start Leader Recover Time Test ================================== Delete ovn central nb pod pod \"ovn-central-5cb9c67d75-tlz9w\" deleted Waiting for ovn central nb pod running =============================== OVN nb Recovery takes 3 .305236803 s ================================ Delete ovn central sb pod pod \"ovn-central-5cb9c67d75-szx4c\" deleted Waiting for ovn central sb pod running =============================== OVN sb Recovery takes 3 .462698535 s ================================ Delete ovn central northd pod pod \"ovn-central-5cb9c67d75-zqmqv\" deleted Waiting for ovn central northd pod running ============================= OVN northd Recovery takes 2 .691291403 s ============================== ==================================================================================================== ================================= Remove Performance Test Resource ================================= rm -f unicast-test-client.log rm -f unicast-test-host-client.log rm -f unicast-test-client.log kubectl ko nbctl lb-del test-server rm -f multicast-test-server.log kubectl exec ovs-ovn-gxdrf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 kubectl exec ovs-ovn-h57bf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 rm -f multicast-test-host-server.log pod \"test-client\" deleted pod \"test-host-client\" deleted pod \"test-host-server\" deleted pod \"test-server\" deleted service \"test-server\" deleted ==================================================================================================== \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kubectl \u63d2\u4ef6\u4f7f\u7528"},{"location":"ops/kubectl-ko/#kubectl","text":"\u4e3a\u4e86\u65b9\u4fbf\u65e5\u5e38\u7684\u8fd0\u7ef4\u64cd\u4f5c\uff0cKube-OVN \u63d0\u4f9b\u4e86 kubectl \u63d2\u4ef6\u5de5\u5177\uff0c\u7f51\u7edc\u7ba1\u7406\u5458 \u53ef\u4ee5\u901a\u8fc7\u8be5\u547d\u4ee4\u8fdb\u884c\u65e5\u5e38\u64cd\u4f5c\uff0c\u4f8b\u5982\uff1a\u67e5\u770b OVN \u6570\u636e\u5e93\u4fe1\u606f\u548c\u72b6\u6001\uff0cOVN \u6570\u636e\u5e93 \u5907\u4efd\u548c\u6062\u590d\uff0cOVS \u76f8\u5173\u4fe1\u606f\u67e5\u770b\uff0ctcpdump \u7279\u5b9a\u5bb9\u5668\uff0c\u7279\u5b9a\u94fe\u8def\u903b\u8f91\u62d3\u6251\u5c55\u793a\uff0c \u7f51\u7edc\u95ee\u9898\u8bca\u65ad\u548c\u6027\u80fd\u4f18\u5316\u3002","title":"kubectl \u63d2\u4ef6\u4f7f\u7528"},{"location":"ops/kubectl-ko/#_1","text":"Kube-OVN \u5b89\u88c5\u65f6\u9ed8\u8ba4\u4f1a\u90e8\u7f72\u63d2\u4ef6\u5230\u6bcf\u4e2a\u8282\u70b9\uff0c\u82e5\u6267\u884c kubectl \u7684\u673a\u5668\u4e0d\u5728\u96c6\u7fa4\u5185\uff0c \u6216\u9700\u8981\u91cd\u88c5\u63d2\u4ef6\uff0c\u53ef\u53c2\u8003\u4e0b\u9762\u7684\u6b65\u9aa4\uff1a \u4e0b\u8f7d kubectl-ko \u6587\u4ef6\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/kubectl-ko \u5c06\u8be5\u6587\u4ef6\u79fb\u52a8\u81f3 $PATH \u76ee\u5f55\u4e0b\uff1a mv kubectl-ko /usr/local/bin/kubectl-ko \u589e\u52a0\u53ef\u6267\u884c\u6743\u9650\uff1a chmod +x /usr/local/bin/kubectl-ko \u68c0\u67e5\u63d2\u4ef6\u662f\u5426\u53ef\u4ee5\u6b63\u5e38\u4f7f\u7528\uff1a # kubectl plugin list The following compatible plugins are available: /usr/local/bin/kubectl-ko","title":"\u63d2\u4ef6\u5b89\u88c5"},{"location":"ops/kubectl-ko/#_2","text":"\u8fd0\u884c kubectl ko \u4f1a\u5c55\u793a\u8be5\u63d2\u4ef6\u6240\u6709\u53ef\u7528\u7684\u547d\u4ee4\u548c\u7528\u6cd5\u63cf\u8ff0\uff0c\u5982\u4e0b\u6240\u793a\uff1a # kubectl ko kubectl ko { subcommand } [ option... ] Available Subcommands: [ nb | sb ] [ status | kick | backup | dbstatus | restore ] ovn-db operations show cluster status, kick stale server, backup database, get db consistency status or restore ovn nb db when met 'inconsistent data' error nbctl [ ovn-nbctl options ... ] invoke ovn-nbctl sbctl [ ovn-sbctl options ... ] invoke ovn-sbctl vsctl { nodeName } [ ovs-vsctl options ... ] invoke ovs-vsctl on the specified node ofctl { nodeName } [ ovs-ofctl options ... ] invoke ovs-ofctl on the specified node dpctl { nodeName } [ ovs-dpctl options ... ] invoke ovs-dpctl on the specified node appctl { nodeName } [ ovs-appctl options ... ] invoke ovs-appctl on the specified node tcpdump { namespace/podname } [ tcpdump options ... ] capture pod traffic trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply echo \" diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] diagnose connectivity of all nodes or a specific node or specify subnet's ds pod or IPPorts like 'tcp-172.18.0.2-53,udp-172.18.0.3-53'\" tuning { install-fastpath | local-install-fastpath | remove-fastpath | install-stt | local-install-stt | remove-stt } { centos7 | centos8 }} [ kernel-devel-version ] deploy kernel optimisation components to the system reload restart all kube-ovn components log { kube-ovn | ovn | ovs | linux | all } save log to ./kubectl-ko-log/ perf [ image ] performance test default image is kubeovn/test:v1.12.0 \u4e0b\u9762\u5c06\u4ecb\u7ecd\u6bcf\u4e2a\u547d\u4ee4\u7684\u5177\u4f53\u529f\u80fd\u548c\u4f7f\u7528\u3002","title":"\u63d2\u4ef6\u4f7f\u7528"},{"location":"ops/kubectl-ko/#nb-sb-status-kick-backup-dbstatus-restore","text":"\u8be5\u5b50\u547d\u4ee4\u4e3b\u8981\u5bf9 OVN \u5317\u5411\u6216\u5357\u5411\u6570\u636e\u5e93\u8fdb\u884c\u64cd\u4f5c\uff0c\u5305\u62ec\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u67e5\u770b\uff0c\u6570\u636e\u5e93\u8282\u70b9\u4e0b\u7ebf\uff0c \u6570\u636e\u5e93\u5907\u4efd\uff0c\u6570\u636e\u5e93\u5b58\u50a8\u72b6\u6001\u67e5\u770b\u548c\u6570\u636e\u5e93\u4fee\u590d\u3002","title":"[nb | sb] [status | kick | backup | dbstatus | restore]"},{"location":"ops/kubectl-ko/#_3","text":"\u8be5\u547d\u4ee4\u4f1a\u5728\u5bf9\u5e94 OVN \u6570\u636e\u5e93\u7684 leader \u8282\u70b9\u6267\u884c ovs-appctl cluster/status \u5c55\u793a\u96c6\u7fa4\u72b6\u6001: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 280309 ms ago, reason: timeout Last Election won: 280309 ms ago Election timer: 5000 Log: [ 139 , 139 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-8723 ->8723 <-85d6 ->85d6 Disconnections: 0 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 8723 ( 8723 at tcp: [ 172 .18.0.3 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 138 status: ok \u82e5 Server \u4e0b\u7684 match_index \u51fa\u73b0\u8f83\u5927\u5dee\u522b\uff0c\u4e14 last msg \u65f6\u95f4\u8f83\u957f\u5219\u5bf9\u5e94 Server \u53ef\u80fd\u957f\u65f6\u95f4\u6ca1\u6709\u54cd\u5e94\uff0c \u9700\u8981\u8fdb\u4e00\u6b65\u67e5\u770b\u3002","title":"\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u67e5\u770b"},{"location":"ops/kubectl-ko/#_4","text":"\u8be5\u547d\u4ee4\u4f1a\u5c06\u67d0\u4e2a\u8282\u70b9\u4ece OVN \u6570\u636e\u5e93\u4e2d\u79fb\u9664\uff0c\u5728\u8282\u70b9\u4e0b\u7ebf\u6216\u66f4\u6362\u8282\u70b9\u65f6\u9700\u8981\u7528\u5230\u3002 \u4e0b\u9762\u5c06\u4ee5\u4e0a\u4e00\u6761\u547d\u4ee4\u6240\u67e5\u770b\u5230\u7684\u96c6\u7fa4\u72b6\u6001\u4e3a\u4f8b\uff0c\u4e0b\u7ebf 172.18.0.3 \u8282\u70b9: # kubectl ko nb kick 8723 started removal \u518d\u6b21\u67e5\u770b\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u786e\u8ba4\u8282\u70b9\u5df2\u79fb\u9664\uff1a # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 324356 ms ago, reason: timeout Last Election won: 324356 ms ago Election timer: 5000 Log: [ 140 , 140 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-85d6 ->85d6 Disconnections: 2 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 140 match_index = 139 last msg 848 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 139 status: ok","title":"\u6570\u636e\u5e93\u8282\u70b9\u4e0b\u7ebf"},{"location":"ops/kubectl-ko/#_5","text":"\u8be5\u5b50\u547d\u4ee4\u4f1a\u5907\u4efd\u5f53\u524d OVN \u6570\u636e\u5e93\u81f3\u672c\u5730\uff0c\u53ef\u7528\u4e8e\u707e\u5907\u548c\u6062\u590d\uff1a # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup","title":"\u6570\u636e\u5e93\u5907\u4efd"},{"location":"ops/kubectl-ko/#_6","text":"\u8be5\u547d\u4ee4\u7528\u6765\u67e5\u770b\u6570\u636e\u5e93\u6587\u4ef6\u662f\u5426\u5b58\u5728\u635f\u574f\uff1a # kubectl ko nb dbstatus status: ok \u82e5\u5f02\u5e38\u5219\u663e\u793a inconsistent data \u9700\u8981\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u8fdb\u884c\u4fee\u590d\u3002","title":"\u6570\u636e\u5e93\u5b58\u50a8\u72b6\u6001\u67e5\u770b"},{"location":"ops/kubectl-ko/#_7","text":"\u82e5\u6570\u636e\u5e93\u72b6\u6001\u8fdb\u5165 inconsistent data \u53ef\u4f7f\u7528\u8be5\u547d\u4ee4\u8fdb\u884c\u4fee\u590d\uff1a # kubectl ko nb restore deployment.apps/ovn-central scaled ovn-central original replicas is 3 first nodeIP is 172 .18.0.5 ovs-ovn pod on node 172 .18.0.5 is ovs-ovn-8jxv9 ovs-ovn pod on node 172 .18.0.3 is ovs-ovn-sjzb6 ovs-ovn pod on node 172 .18.0.4 is ovs-ovn-t87zk backup nb db file restore nb db file, operate in pod ovs-ovn-8jxv9 deployment.apps/ovn-central scaled finish restore nb db file and ovn-central replicas recreate ovs-ovn pods pod \"ovs-ovn-8jxv9\" deleted pod \"ovs-ovn-sjzb6\" deleted pod \"ovs-ovn-t87zk\" deleted","title":"\u6570\u636e\u5e93\u4fee\u590d"},{"location":"ops/kubectl-ko/#nbctl-sbctl-options","text":"\u8be5\u5b50\u547d\u4ee4\u4f1a\u76f4\u63a5\u8fdb\u5165 OVN \u5317\u5411\u6570\u636e\u5e93\u6216\u5357\u5411\u6570\u636e\u5e93 \u7684 leader \u8282\u70b9\u5206\u522b\u6267\u884c ovn-nbctl \u548c ovn-sbctl \u547d\u4ee4\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVN \u7684\u5b98\u65b9\u6587\u6863 ovn-nbctl(8) \u548c ovn-sbctl(8) \u3002 # kubectl ko nbctl show switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port node-liumengxin-ovn2-192.168.137.177 addresses: [ \"00:00:00:D9:58:B8 100.64.0.3\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port alertmanager-main-0.monitoring addresses: [ \"00:00:00:6C:DF:A3 10.16.0.19\" ] port kube-state-metrics-5d6885d89-4nf8h.monitoring addresses: [ \"00:00:00:6F:02:1C 10.16.0.15\" ] port fake-kubelet-67c55dfd89-pv86k.kube-system addresses: [ \"00:00:00:5C:12:E8 10.16.19.177\" ] port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ]","title":"[nbctl | sbctl] [options ...]"},{"location":"ops/kubectl-ko/#vsctl-nodename-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-vsctl \u547d\u4ee4\uff0c\u67e5\u8be2\u5e76\u914d\u7f6e vswitchd \u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-vsctl(8) \u3002 # kubectl ko vsctl kube-ovn-01 show 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port mirror0 Interface mirror0 type: internal Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port br-int Interface br-int type: internal ovs_version: \"2.17.2\"","title":"vsctl {nodeName} [options ...]"},{"location":"ops/kubectl-ko/#ofctl-nodename-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-ofctl \u547d\u4ee4\uff0c\u67e5\u8be2\u6216\u7ba1\u7406 OpenFlow\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-ofctl(8) \u3002 # kubectl ko ofctl kube-ovn-01 dump-flows br-int NXST_FLOW reply ( xid = 0x4 ) : flags =[ more ] cookie = 0xcf3429e6, duration = 671791 .432s, table = 0 , n_packets = 0 , n_bytes = 0 , idle_age = 65534 , hard_age = 65534 , priority = 100 ,in_port = 2 actions = load:0x4->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x1->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xc91413c6, duration = 671791 .431s, table = 0 , n_packets = 907489 , n_bytes = 99978275 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 7 actions = load:0x1->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x4->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xf180459, duration = 671791 .431s, table = 0 , n_packets = 17348582 , n_bytes = 2667811214 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 6317 actions = load:0xa->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x9->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0x7806dd90, duration = 671791 .431s, table = 0 , n_packets = 3235428 , n_bytes = 833821312 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 1 actions = load:0xd->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x3->NXM_NX_REG14 [] ,resubmit ( ,8 ) ...","title":"ofctl {nodeName} [options ...]"},{"location":"ops/kubectl-ko/#dpctl-nodename-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-dpctl \u547d\u4ee4\uff0c\u67e5\u8be2\u6216\u7ba1\u7406 OVS datapath\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-dpctl(8) \u3002 # kubectl ko dpctl kube-ovn-01 show system@ovs-system: lookups: hit:350805055 missed:21983648 lost:73 flows: 105 masks: hit:1970748791 total:22 hit/pkt:5.29 port 0 : ovs-system ( internal ) port 1 : ovn0 ( internal ) port 2 : mirror0 ( internal ) port 3 : br-int ( internal ) port 4 : stt_sys_7471 ( stt: packet_type = ptap ) port 5 : eeb4d9e51b5d_h port 6 : a2c1a8a8b83a_h port 7 : 4fa5c4cbb1a5_h","title":"dpctl {nodeName} [options ...]"},{"location":"ops/kubectl-ko/#appctl-nodename-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-appctl \u547d\u4ee4\uff0c\u6765\u64cd\u4f5c\u76f8\u5173 daemon \u8fdb\u7a0b\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-appctl(8) \u3002 # kubectl ko appctl kube-ovn-01 vlog/list console syslog file ------- ------ ------ backtrace OFF ERR INFO bfd OFF ERR INFO bond OFF ERR INFO bridge OFF ERR INFO bundle OFF ERR INFO bundles OFF ERR INFO ...","title":"appctl {nodeName} [options ...]"},{"location":"ops/kubectl-ko/#tcpdump-namespacepodname-tcpdump-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165 namespace/podname \u6240\u5728\u673a\u5668\u7684 kube-ovn-cni \u5bb9\u5668\uff0c\u5e76\u6267\u884c tcpdump \u6293\u53d6\u5bf9\u5e94\u5bb9\u5668 veth \u7f51\u5361 \u7aef\u7684\u6d41\u91cf\uff0c\u53ef\u4ee5\u65b9\u4fbf\u6392\u67e5\u7f51\u7edc\u76f8\u5173\u95ee\u9898\uff0c\u5982\u4e0b\u6240\u793a\uff1a # kubectl ko tcpdump default/ds1-l6n7p icmp + kubectl exec -it kube-ovn-cni-wlg4s -n kube-ovn -- tcpdump -nn -i d7176fe7b4e0_h icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on d7176fe7b4e0_h, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 06 :52:36.619688 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 1 , length 64 06 :52:36.619746 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 1 , length 64 06 :52:37.619588 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 2 , length 64 06 :52:37.619630 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 2 , length 64 06 :52:38.619933 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 3 , length 64 06 :52:38.619973 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 3 , length 64","title":"tcpdump {namespace/podname} [tcpdump options ...]"},{"location":"ops/kubectl-ko/#trace-arguments","text":"\u8be5\u547d\u4ee4\u5c06\u4f1a\u6253\u5370 Pod \u6216\u8282\u70b9\u901a\u8fc7\u7279\u5b9a\u534f\u8bae\u8bbf\u95ee\u67d0\u5730\u5740\u65f6\u5bf9\u5e94\u7684 OVN \u903b\u8f91\u6d41\u8868\u548c\u6700\u7ec8\u7684 Openflow \u6d41\u8868\uff0c \u65b9\u4fbf\u5f00\u53d1\u6216\u8fd0\u7ef4\u65f6\u5b9a\u4f4d\u6d41\u8868\u76f8\u5173\u95ee\u9898\u3002 \u652f\u6301\u7684\u547d\u4ee4\uff1a kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } kubectl ko trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } \u793a\u4f8b\uff1a # kubectl ko trace default/ds1-l6n7p 8.8.8.8 icmp + kubectl exec ovn-central-5bc494cb5-np9hm -n kube-ovn -- ovn-trace --ct = new ovn-default 'inport == \"ds1-l6n7p.default\" && ip.ttl == 64 && icmp && eth.src == 0a:00:00:10:00:05 && ip4.src == 10.16.0.4 && eth.dst == 00:00:00:B8:CA:43 && ip4.dst == 8.8.8.8' # icmp,reg14=0xf,vlan_tci=0x0000,dl_src=0a:00:00:10:00:05,dl_dst=00:00:00:b8:ca:43,nw_src=10.16.0.4,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 ingress ( dp = \"ovn-default\" , inport = \"ds1-l6n7p.default\" ) ----------------------------------------------------- 0 . ls_in_port_sec_l2 ( ovn-northd.c:4143 ) : inport == \"ds1-l6n7p.default\" && eth.src == { 0a:00:00:10:00:05 } , priority 50 , uuid 39453393 next ; 1 . ls_in_port_sec_ip ( ovn-northd.c:2898 ) : inport == \"ds1-l6n7p.default\" && eth.src == 0a:00:00:10:00:05 && ip4.src == { 10 .16.0.4 } , priority 90 , uuid 81bcd485 next ; 3 . ls_in_pre_acl ( ovn-northd.c:3269 ) : ip, priority 100 , uuid 7b4f4971 reg0 [ 0 ] = 1 ; next ; 5 . ls_in_pre_stateful ( ovn-northd.c:3396 ) : reg0 [ 0 ] == 1 , priority 100 , uuid 36cdd577 ct_next ; ct_next ( ct_state = new | trk ) ------------------------- 6 . ls_in_acl ( ovn-northd.c:3759 ) : ip && ( !ct.est || ( ct.est && ct_label.blocked == 1 )) , priority 1 , uuid 7608af5b reg0 [ 1 ] = 1 ; next ; 10 . ls_in_stateful ( ovn-northd.c:3995 ) : reg0 [ 1 ] == 1 , priority 100 , uuid 2aba1b90 ct_commit ( ct_label = 0 /0x1 ) ; next ; 16 . ls_in_l2_lkup ( ovn-northd.c:4470 ) : eth.dst == 00 :00:00:b8:ca:43, priority 50 , uuid 5c9c3c9f outport = \"ovn-default-ovn-cluster\" ; output ; ... \u82e5 trace \u5bf9\u8c61\u4e3a\u8fd0\u884c\u4e8e Underlay \u7f51\u7edc\u4e0b\u7684\u865a\u62df\u673a\uff0c\u9700\u8981\u6dfb\u52a0\u989d\u5916\u53c2\u6570\u6765\u6307\u5b9a\u76ee\u7684 Mac \u5730\u5740\uff1a kubectl ko trace default/virt-handler-7lvml 8 .8.8.8 82 :7c:9f:83:8c:01 icmp","title":"trace [arguments ...]"},{"location":"ops/kubectl-ko/#diagnose-allnodesubnetipports-nodenamesubnetnameproto1-ip1-port1proto2-ip2-port2","text":"\u8bca\u65ad\u96c6\u7fa4\u7f51\u7edc\u7ec4\u4ef6\u72b6\u6001\uff0c\u5e76\u53bb\u5bf9\u5e94\u8282\u70b9\u7684 kube-ovn-pinger \u68c0\u6d4b\u5f53\u524d\u8282\u70b9\u5230\u5176\u4ed6\u8282\u70b9\u548c\u5173\u952e\u670d\u52a1\u7684\u8fde\u901a\u6027\u548c\u7f51\u7edc\u5ef6\u8fdf\uff1a # kubectl ko diagnose all switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default port prometheus-k8s-1.monitoring addresses: [ \"00:00:00:AA:37:DF 10.16.0.23\" ] router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] Routing Policies 31000 ip4.dst == 10 .16.0.0/16 allow 31000 ip4.dst == 100 .64.0.0/16 allow 30000 ip4.dst == 192 .168.137.177 reroute 100 .64.0.3 30000 ip4.dst == 192 .168.137.178 reroute 100 .64.0.4 29000 ip4.src == $ovn .default.fake.6_ip4 reroute 100 .64.0.22 29000 ip4.src == $ovn .default.fake.7_ip4 reroute 100 .64.0.21 29000 ip4.src == $ovn .default.fake.8_ip4 reroute 100 .64.0.23 29000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 reroute 100 .64.0.4 20000 ip4.src == $ovn .default.liumengxin.ovn1.192.168.137.176_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.2 20000 ip4.src == $ovn .default.liumengxin.ovn2.192.168.137.177_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.3 20000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.4 IPv4 Routes Route Table <main>: 0 .0.0.0/0 100 .64.0.1 dst-ip UUID LB PROTO VIP IPs e9bcfd9d-793e-4431-9073-6dec96b75d71 cluster-tcp-load tcp 10 .100.209.132:10660 192 .168.137.176:10660 tcp 10 .101.239.192:6641 192 .168.137.177:6641 tcp 10 .101.240.101:3000 10 .16.0.7:3000 tcp 10 .103.184.186:6642 192 .168.137.177:6642 35d2b7a5-e3a7-485a-a4b7-b4970eb0e63b cluster-tcp-sess tcp 10 .100.158.128:8080 10 .16.0.10:8080,10.16.0.5:8080,10.16.63.30:8080 tcp 10 .107.26.215:8080 10 .16.0.19:8080,10.16.0.20:8080,10.16.0.21:8080 tcp 10 .107.26.215:9093 10 .16.0.19:9093,10.16.0.20:9093,10.16.0.21:9093 tcp 10 .98.187.99:8080 10 .16.0.22:8080,10.16.0.23:8080 tcp 10 .98.187.99:9090 10 .16.0.22:9090,10.16.0.23:9090 f43303e4-89aa-4d3e-a3dc-278a552fe27b cluster-udp-load udp 10 .96.0.10:53 10 .16.0.4:53,10.16.0.9:53 _uuid : 06776304 -5a96-43ed-90c4-c4854c251699 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn2_192.168.137.177_underlay_v6 _uuid : 62690625 -87d5-491c-8675-9fd83b1f433c addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn1_192.168.137.176_underlay_v6 _uuid : b03a9bae-94d5-4562-b34c-b5f6198e180b addresses : [ \"10.16.0.0/16\" , \"100.64.0.0/16\" , \"172.22.0.0/16\" , \"192.101.0.0/16\" ] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv4 _uuid : e1056f3a-24cc-4666-8a91-75ee6c3c2426 addresses : [] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv6 _uuid : 3e5d5fff-e670-47b2-a2f5-a39f4698a8c5 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn3_192.168.137.178_underlay_v6 _uuid : 2d85dbdc-d0db-4abe-b19e-cc806d32b492 action : drop direction : from-lport external_ids : {} label : 0 log : false match : \"inport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] _uuid : de790cc8-f155-405f-bb32-5a51f30c545f action : drop direction : to-lport external_ids : {} label : 0 log : false match : \"outport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] Chassis \"e15ed4d4-1780-4d50-b09e-ea8372ed48b8\" hostname: liumengxin-ovn1-192.168.137.176 Encap stt ip: \"192.168.137.176\" options: { csum = \"true\" } Port_Binding node-liumengxin-ovn1-192.168.137.176 Port_Binding perf-6vxkn.default Port_Binding kube-state-metrics-5d6885d89-4nf8h.monitoring Port_Binding alertmanager-main-0.monitoring Port_Binding kube-ovn-pinger-6ftdf.kube-system Port_Binding fake-kubelet-67c55dfd89-pv86k.kube-system Port_Binding prometheus-k8s-0.monitoring Chassis \"eef07da1-f8ad-4775-b14d-bd6a3b4eb0d5\" hostname: liumengxin-ovn3-192.168.137.178 Encap stt ip: \"192.168.137.178\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-7twb4.kube-system Port_Binding prometheus-adapter-86df476d87-rl88g.monitoring Port_Binding prometheus-k8s-1.monitoring Port_Binding node-liumengxin-ovn3-192.168.137.178 Port_Binding perf-ff475.default Port_Binding alertmanager-main-1.monitoring Port_Binding blackbox-exporter-676d976865-tvsjd.monitoring Chassis \"efa253c9-494d-4719-83ae-b48ab0f11c03\" hostname: liumengxin-ovn2-192.168.137.177 Encap stt ip: \"192.168.137.177\" options: { csum = \"true\" } Port_Binding grafana-6c4c6b8fb7-pzd2c.monitoring Port_Binding node-liumengxin-ovn2-192.168.137.177 Port_Binding alertmanager-main-2.monitoring Port_Binding coredns-6789c94dd8-9jqsz.kube-system Port_Binding coredns-6789c94dd8-25d4r.kube-system Port_Binding prometheus-operator-7bbc99fc8b-wgjm4.monitoring Port_Binding prometheus-adapter-86df476d87-gdxmc.monitoring Port_Binding perf-fjnws.default Port_Binding kube-ovn-pinger-vh2xg.kube-system ds kube-proxy ready kube-proxy ready deployment ovn-central ready deployment kube-ovn-controller ready ds kube-ovn-cni ready ds ovs-ovn ready deployment coredns ready ovn-nb leader check ok ovn-sb leader check ok ovn-northd leader check ok ### kube-ovn-controller recent log ### start to diagnose node liumengxin-ovn1-192.168.137.176 #### ovn-controller log: 2022 -06-03T00:56:44.897Z | 16722 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:06:44.912Z | 16723 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:16:44.925Z | 16724 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:26:44.936Z | 16725 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:36:44.959Z | 16726 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:46:44.974Z | 16727 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:56:44.988Z | 16728 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:06:45.001Z | 16729 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:16:45.025Z | 16730 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:26:45.040Z | 16731 | inc_proc_eng | INFO | User triggered force recompute. #### ovs-vswitchd log: 2022 -06-02T23:03:00.137Z | 00079 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:f9d1 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-02T23:23:31.840Z | 00080 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:15b2 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:09:15.659Z | 00081 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:dc:e3:63,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.63.30,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:e5a5 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x150017000004,src = 192 .168.137.178,dst = 192 .168.137.176,ttl = 64 ,tp_src = 9239 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.63.30,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:30:13.409Z | 00064 | dpif ( handler2 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:6b4a with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T02:02:33.832Z | 00082 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:a819 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 #### ovs-vsctl show results: 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port \"04d03360e9a0_h\" Interface \"04d03360e9a0_h\" Port eeb4d9e51b5d_h Interface eeb4d9e51b5d_h Port mirror0 Interface mirror0 type: internal Port \"8e5d887ccd80_h\" Interface \"8e5d887ccd80_h\" Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port \"17512d5be1f1_h\" Interface \"17512d5be1f1_h\" Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" #### pinger diagnose results: I0603 10 :35:04.349404 17619 pinger.go:19 ] ------------------------------------------------------------------------------- Kube-OVN: Version: v1.12.0 Build: 2022 -04-24_08:02:50 Commit: git-73f9d15 Go Version: go1.17.8 Arch: amd64 ------------------------------------------------------------------------------- I0603 10 :35:04.376797 17619 config.go:166 ] pinger config is & { KubeConfigFile: KubeClient:0xc000493380 Port:8080 DaemonSetNamespace:kube-system DaemonSetName:kube-ovn-pinger Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:liumengxin-ovn1-192.168.137.176 HostIP:192.168.137.176 PodName:kube-ovn-pinger-6ftdf PodIP:10.16.0.10 PodProtocols: [ IPv4 ] ExternalAddress: NetworkMode:kube-ovn PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid } I0603 10 :35:04.449166 17619 exporter.go:75 ] liumengxin-ovn1-192.168.137.176: exporter connect successfully I0603 10 :35:04.554011 17619 ovn.go:21 ] ovs-vswitchd and ovsdb are up I0603 10 :35:04.651293 17619 ovn.go:33 ] ovn_controller is up I0603 10 :35:04.651342 17619 ovn.go:39 ] start to check port binding I0603 10 :35:04.749613 17619 ovn.go:135 ] chassis id is 1d7f3d6c-eec5-4b3c-adca-2969d9cdfd80 I0603 10 :35:04.763487 17619 ovn.go:49 ] port in sb is [ node-liumengxin-ovn1-192.168.137.176 perf-6vxkn.default kube-state-metrics-5d6885d89-4nf8h.monitoring alertmanager-main-0.monitoring kube-ovn-pinger-6ftdf.kube-system fake-kubelet-67c55dfd89-pv86k.kube-system prometheus-k8s-0.monitoring ] I0603 10 :35:04.763583 17619 ovn.go:61 ] ovs and ovn-sb binding check passed I0603 10 :35:05.049309 17619 ping.go:259 ] start to check apiserver connectivity I0603 10 :35:05.053666 17619 ping.go:268 ] connect to apiserver success in 4 .27ms I0603 10 :35:05.053786 17619 ping.go:129 ] start to check pod connectivity I0603 10 :35:05.249590 17619 ping.go:159 ] ping pod: kube-ovn-pinger-6ftdf 10 .16.0.10, count: 3 , loss count 0 , average rtt 16 .30ms I0603 10 :35:05.354135 17619 ping.go:159 ] ping pod: kube-ovn-pinger-7twb4 10 .16.63.30, count: 3 , loss count 0 , average rtt 1 .81ms I0603 10 :35:05.458460 17619 ping.go:159 ] ping pod: kube-ovn-pinger-vh2xg 10 .16.0.5, count: 3 , loss count 0 , average rtt 1 .92ms I0603 10 :35:05.458523 17619 ping.go:83 ] start to check node connectivity \u5982\u679c diagnose \u7684\u76ee\u6807\u6307\u5b9a\u4e3a subnet \u8be5\u811a\u672c\u4f1a\u5728 subnet \u4e0a\u5efa\u7acb daemonset\uff0c\u7531 kube-ovn-pinger \u53bb\u63a2\u6d4b\u8fd9\u4e2a daemonset \u7684\u6240\u6709 pod \u7684\u8fde\u901a\u6027\u548c\u7f51\u7edc\u5ef6\u65f6\uff0c\u6d4b\u8bd5\u5b8c\u540e\u81ea\u52a8\u9500\u6bc1\u8be5 daemonset\u3002 \u5982\u679c diagnose \u7684\u76ee\u6807\u6307\u5b9a\u4e3a IPPorts \u8be5\u811a\u672c\u4f1a\u8ba9\u6bcf\u4e2a kube-ovn-pinger pod \u53bb\u63a2\u6d4b\u76ee\u6807\u534f\u8bae\uff0cIP\uff0cPort \u662f\u5426\u53ef\u8fbe\u3002","title":"diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}]"},{"location":"ops/kubectl-ko/#tuning-install-fastpathlocal-install-fastpathremove-fastpathinstall-sttlocal-install-sttremove-stt-centos7centos8-kernel-devel-version","text":"\u8be5\u547d\u4ee4\u6267\u884c\u6027\u80fd\u8c03\u4f18\u76f8\u5173\u64cd\u4f5c\uff0c\u5177\u4f53\u4f7f\u7528\u8bf7\u53c2\u8003 \u6027\u80fd\u8c03\u4f18 \u3002","title":"tuning {install-fastpath|local-install-fastpath|remove-fastpath|install-stt|local-install-stt|remove-stt} {centos7|centos8}} [kernel-devel-version]"},{"location":"ops/kubectl-ko/#reload","text":"\u8be5\u547d\u4ee4\u91cd\u542f\u6240\u6709 Kube-OVN \u76f8\u5173\u7ec4\u4ef6\uff1a # kubectl ko reload pod \"ovn-central-8684dd94bd-vzgcr\" deleted Waiting for deployment \"ovn-central\" rollout to finish: 0 of 1 updated replicas are available... deployment \"ovn-central\" successfully rolled out pod \"ovs-ovn-bsnvz\" deleted pod \"ovs-ovn-m9b98\" deleted pod \"kube-ovn-controller-8459db5ff4-64c62\" deleted Waiting for deployment \"kube-ovn-controller\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out pod \"kube-ovn-cni-2klnh\" deleted pod \"kube-ovn-cni-t2jz4\" deleted Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-cni\" successfully rolled out pod \"kube-ovn-pinger-ln72z\" deleted pod \"kube-ovn-pinger-w8lrk\" deleted Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-pinger\" successfully rolled out pod \"kube-ovn-monitor-7fb67d5488-7q6zb\" deleted Waiting for deployment \"kube-ovn-monitor\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-monitor\" successfully rolled out","title":"reload"},{"location":"ops/kubectl-ko/#log","text":"\u4f7f\u7528\u8be5\u547d\u4ee4\u4f1a\u6293\u53d6 kube-ovn \u6240\u6709\u8282\u70b9\u4e0a\u7684 Kube-OVN\uff0cOVN\uff0cOpenvswitch \u7684 log \u4ee5\u53ca linux \u5e38\u7528\u7684\u4e00\u4e9b debug \u4fe1\u606f\u3002 # kubectl ko log all Collecting kube-ovn logging files Collecting ovn logging files Collecting openvswitch logging files Collecting linux dmesg files Collecting linux iptables-legacy files Collecting linux iptables-nft files Collecting linux route files Collecting linux link files Collecting linux neigh files Collecting linux memory files Collecting linux top files Collecting linux sysctl files Collecting linux netstat files Collecting linux addr files Collecting linux ipset files Collecting linux tcp files Collected files have been saved in the directory /root/kubectl-ko-log \u76ee\u5f55\u5982\u4e0b\uff1a # tree kubectl-ko-log/ kubectl-ko-log/ | -- kube-ovn-control-plane | | -- kube-ovn | | | -- kube-ovn-cni.log | | | -- kube-ovn-monitor.log | | ` -- kube-ovn-pinger.log | | -- linux | | | -- addr.log | | | -- dmesg.log | | | -- ipset.log | | | -- iptables-legacy.log | | | -- iptables-nft.log | | | -- link.log | | | -- memory.log | | | -- neigh.log | | | -- netstat.log | | | -- route.log | | | -- sysctl.log | | | -- tcp.log | | ` -- top.log | | -- openvswitch | | | -- ovs-vswitchd.log | | ` -- ovsdb-server.log | ` -- ovn | | -- ovn-controller.log | | -- ovn-northd.log | | -- ovsdb-server-nb.log | ` -- ovsdb-server-sb.log","title":"log"},{"location":"ops/kubectl-ko/#perf-image","text":"\u8be5\u547d\u4ee4\u4f1a\u53bb\u6d4b\u8bd5 Kube-OVN \u7684\u4e00\u4e9b\u6027\u80fd\u6307\u6807\u5982\u4e0b\uff1a \u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u6307\u6807\uff1b Hostnetwork \u7f51\u7edc\u6027\u80fd\u6307\u6807\uff1b \u5bb9\u5668\u7f51\u7edc\u7ec4\u64ad\u62a5\u6587\u6027\u80fd\u6307\u6807\uff1b OVN-NB, OVN-SB, OVN-Northd leader \u5220\u9664\u6062\u590d\u6240\u9700\u65f6\u95f4\u3002 \u53c2\u6570 image \u7528\u4e8e\u6307\u5b9a\u6027\u80fd\u6d4b\u8bd5 pod \u6240\u7528\u7684\u955c\u50cf\uff0c\u9ed8\u8ba4\u60c5\u51b5\u4e0b\u662f kubeovn/test:v1.12.0 , \u8bbe\u7f6e\u8be5\u53c2\u6570\u4e3b\u8981\u662f\u4e3a\u4e86\u79bb\u7ebf\u573a\u666f\uff0c\u5c06\u955c\u50cf\u62c9\u5230\u5185\u7f51\u73af\u5883\u53ef\u80fd\u4f1a\u6709\u955c\u50cf\u540d\u53d8\u5316\u3002 # kubectl ko perf ============================== Prepareing Performance Test Resources =============================== pod/test-client created pod/test-host-client created pod/test-server created pod/test-host-server created service/test-server created pod/test-client condition met pod/test-host-client condition met pod/test-host-server condition met pod/test-server condition met ==================================================================================================== ============================ Start Pod Network Unicast Performance Test ============================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 82 .8 us 97 .7 Mbits/sec 67 .6 us ( 0 % ) 8 .42 Mbits/sec 128 85 .4 us 167 Mbits/sec 67 .2 us ( 0 % ) 17 .2 Mbits/sec 512 85 .8 us 440 Mbits/sec 68 .7 us ( 0 % ) 68 .4 Mbits/sec 1k 85 .1 us 567 Mbits/sec 68 .7 us ( 0 % ) 134 Mbits/sec 4k 138 us 826 Mbits/sec 78 .1 us ( 1 .4% ) 503 Mbits/sec ==================================================================================================== =============================== Start Host Network Performance Test ================================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 49 .7 us 120 Mbits/sec 37 .9 us ( 0 % ) 18 .6 Mbits/sec 128 49 .7 us 200 Mbits/sec 38 .1 us ( 0 % ) 35 .5 Mbits/sec 512 51 .9 us 588 Mbits/sec 38 .9 us ( 0 % ) 142 Mbits/sec 1k 51 .7 us 944 Mbits/sec 37 .2 us ( 0 % ) 279 Mbits/sec 4k 74 .9 us 1 .66 Gbits/sec 39 .9 us ( 0 % ) 1 .20 Gbits/sec ==================================================================================================== ============================== Start Service Network Performance Test ============================== Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 111 us 96 .3 Mbits/sec 88 .4 us ( 0 % ) 7 .59 Mbits/sec 128 83 .7 us 150 Mbits/sec 69 .2 us ( 0 % ) 16 .9 Mbits/sec 512 87 .4 us 374 Mbits/sec 75 .8 us ( 0 % ) 60 .9 Mbits/sec 1k 88 .2 us 521 Mbits/sec 73 .1 us ( 0 % ) 123 Mbits/sec 4k 148 us 813 Mbits/sec 77 .6 us ( 0 .0044% ) 451 Mbits/sec ==================================================================================================== =========================== Start Pod Multicast Network Performance Test =========================== Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .014 ms ( 0 .17% ) 5 .80 Mbits/sec 128 0 .012 ms ( 0 % ) 11 .4 Mbits/sec 512 0 .016 ms ( 0 % ) 46 .1 Mbits/sec 1k 0 .023 ms ( 0 .073% ) 89 .8 Mbits/sec 4k 0 .035 ms ( 1 .3% ) 126 Mbits/sec ==================================================================================================== ============================= Start Host Multicast Network Performance ============================= Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .007 ms ( 0 % ) 9 .95 Mbits/sec 128 0 .005 ms ( 0 % ) 21 .8 Mbits/sec 512 0 .008 ms ( 0 % ) 86 .8 Mbits/sec 1k 0 .013 ms ( 0 .045% ) 168 Mbits/sec 4k 0 .010 ms ( 0 .31% ) 242 Mbits/sec ==================================================================================================== ================================== Start Leader Recover Time Test ================================== Delete ovn central nb pod pod \"ovn-central-5cb9c67d75-tlz9w\" deleted Waiting for ovn central nb pod running =============================== OVN nb Recovery takes 3 .305236803 s ================================ Delete ovn central sb pod pod \"ovn-central-5cb9c67d75-szx4c\" deleted Waiting for ovn central sb pod running =============================== OVN sb Recovery takes 3 .462698535 s ================================ Delete ovn central northd pod pod \"ovn-central-5cb9c67d75-zqmqv\" deleted Waiting for ovn central northd pod running ============================= OVN northd Recovery takes 2 .691291403 s ============================== ==================================================================================================== ================================= Remove Performance Test Resource ================================= rm -f unicast-test-client.log rm -f unicast-test-host-client.log rm -f unicast-test-client.log kubectl ko nbctl lb-del test-server rm -f multicast-test-server.log kubectl exec ovs-ovn-gxdrf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 kubectl exec ovs-ovn-h57bf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 rm -f multicast-test-host-server.log pod \"test-client\" deleted pod \"test-host-client\" deleted pod \"test-host-server\" deleted pod \"test-server\" deleted service \"test-server\" deleted ==================================================================================================== \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"perf [image]"},{"location":"ops/recover-db/","text":"OVN \u6570\u636e\u5e93\u5907\u4efd\u548c\u6062\u590d \u00b6 \u672c\u6587\u6863\u4ecb\u7ecd\u5982\u4f55\u8fdb\u884c\u6570\u636e\u5e93\u5907\u4efd\uff0c\u4ee5\u53ca\u5728\u4e0d\u540c\u60c5\u51b5\u4e0b\u5982\u4f55\u901a\u8fc7\u5df2\u6709\u7684\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u96c6\u7fa4\u6062\u590d\u3002 \u6570\u636e\u5e93\u5907\u4efd \u00b6 \u5229\u7528 kubectl \u63d2\u4ef6\u7684 backup \u547d\u4ee4\u53ef\u4ee5\u5bf9\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u5907\u4efd\uff0c\u4ee5\u7528\u4e8e\u6545\u969c\u65f6\u6062\u590d\uff1a # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup # kubectl ko sb backup tar: Removing leading `/' from member names backup ovn-nb db to /root/ovnsb_db.060223191654183154.backup \u96c6\u7fa4\u90e8\u5206\u6545\u969c\u6062\u590d \u00b6 \u82e5\u96c6\u7fa4\u4e2d\u5b58\u5728\u90e8\u5206\u8282\u70b9\u56e0\u4e3a\u65ad\u7535\uff0c\u6587\u4ef6\u7cfb\u7edf\u6545\u969c\u6216\u78c1\u76d8\u7a7a\u95f4\u4e0d\u8db3\u5bfc\u81f4\u5de5\u4f5c\u5f02\u5e38\uff0c \u4f46\u662f\u96c6\u7fa4\u4ecd\u53ef\u6b63\u5e38\u5de5\u4f5c\u53ef\u4ee5\u901a\u8fc7\u5982\u4e0b\u6b65\u9aa4\u8fdb\u884c\u6062\u590d\u3002 \u67e5\u770b\u65e5\u5fd7\u786e\u8ba4\u72b6\u6001\u5f02\u5e38 \u00b6 \u67e5\u770b\u5bf9\u5e94\u8282\u70b9 /var/log/ovn/ovn-northd.log \uff0c\u82e5\u63d0\u793a\u7c7b\u4f3c\u9519\u8bef\u5219\u53ef\u5224\u65ad\u6570\u636e\u5e93\u5b58\u5728\u5f02\u5e38 * ovn-northd is not running ovsdb-server: ovsdb error: error reading record 2739 from OVN_Northbound log: record 2739 advances commit index to 6308 but last log index is 6307 * Starting ovsdb-nb \u4ece\u96c6\u7fa4\u4e2d\u8e22\u51fa\u5bf9\u5e94\u8282\u70b9 \u00b6 \u6839\u636e\u65e5\u5fd7\u63d0\u793a\u662f OVN_Northbound \u8fd8\u662f OVN_Southbound \u9009\u62e9\u5bf9\u5e94\u7684\u6570\u636e\u5e93\u8fdb\u884c\u64cd\u4f5c\u3002 \u4e0a\u8ff0\u65e5\u5fd7\u63d0\u793a\u4e3a OVN_Northbound \u5219\u5bf9 ovn-nb \u8fdb\u884c\u64cd\u4f5c\uff1a # kubectl ko nb status 9182 Name: OVN_Northbound Cluster ID: e75f ( e75fa340-49ed-45ab-990e-26cb865ebc85 ) Server ID: 9182 ( 9182e8dd-b5b0-4dd8-8518-598cc1e374f3 ) Address: tcp: [ 10 .0.128.61 ] :6643 Status: cluster member Role: leader Term: 1454 Leader: self Vote: self Last Election started 1732603 ms ago, reason: timeout Last Election won: 1732587 ms ago Election timer: 1000 Log: [ 7332 , 12512 ] Entries not yet committed: 1 Entries not yet applied: 1 Connections: ->f080 <-f080 <-e631 ->e631 Disconnections: 1 Servers: f080 ( f080 at tcp: [ 10 .0.129.139 ] :6643 ) next_index = 12512 match_index = 12510 last msg 63 ms ago 9182 ( 9182 at tcp: [ 10 .0.128.61 ] :6643 ) ( self ) next_index = 10394 match_index = 12510 e631 ( e631 at tcp: [ 10 .0.131.173 ] :6643 ) next_index = 12512 match_index = 0 \u4ece\u96c6\u7fa4\u4e2d\u8e22\u51fa\u72b6\u6001\u5f02\u5e38\u8282\u70b9\uff1a kubectl ko nb kick e631 \u767b\u5f55\u5f02\u5e38\u8282\u70b9\uff0c\u5220\u9664\u5bf9\u5e94\u7684\u6570\u636e\u5e93\u6587\u4ef6\uff1a mv /etc/origin/ovn/ovnnb_db.db /tmp \u5220\u9664\u5bf9\u5e94\u8282\u70b9\u7684 ovn-central Pod\uff0c\u7b49\u5f85\u96c6\u7fa4\u81ea\u52a8\u6062\u590d\uff1a kubectl delete pod -n kube-system ovn-central-xxxx \u96c6\u7fa4\u4e0d\u80fd\u6b63\u5e38\u5de5\u4f5c\u4e0b\u7684\u6062\u590d \u00b6 \u82e5\u96c6\u7fa4\u591a\u6570\u8282\u70b9\u53d7\u635f\u65e0\u6cd5\u9009\u4e3e\u51fa leader\uff0c\u8bf7\u53c2\u7167\u4e0b\u9762\u7684\u6b65\u9aa4\u8fdb\u884c\u6062\u590d\u3002 \u505c\u6b62 ovn-central \u00b6 \u8bb0\u5f55\u5f53\u524d ovn-central \u526f\u672c\u6570\u91cf\uff0c\u5e76\u505c\u6b62 ovn-central \u907f\u514d\u65b0\u7684\u6570\u636e\u5e93\u53d8\u66f4\u5f71\u54cd\u6062\u590d\uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 0 \u9009\u62e9\u5907\u4efd \u00b6 \u7531\u4e8e\u591a\u6570\u8282\u70b9\u53d7\u635f\uff0c\u9700\u8981\u4ece\u67d0\u4e2a\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u6062\u590d\u91cd\u5efa\u96c6\u7fa4\u3002\u5982\u679c\u4e4b\u524d\u5907\u4efd\u8fc7\u6570\u636e\u5e93 \u53ef\u4f7f\u7528\u4e4b\u524d\u7684\u5907\u4efd\u6587\u4ef6\u8fdb\u884c\u6062\u590d\u3002\u5982\u679c\u6ca1\u6709\u8fdb\u884c\u8fc7\u5907\u4efd\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u6b65\u9aa4\u4ece\u5df2\u6709\u7684\u6570\u636e\u5e93\u6587\u4ef6 \u4e2d\u751f\u6210\u4e00\u4e2a\u5907\u4efd\u3002 \u7531\u4e8e\u9ed8\u8ba4\u6587\u4ef6\u5939\u4e0b\u7684\u6570\u636e\u5e93\u6587\u4ef6\u4e3a\u96c6\u7fa4\u683c\u5f0f\u6570\u636e\u5e93\u6587\u4ef6\uff0c\u5305\u542b\u5f53\u524d\u96c6\u7fa4\u7684\u4fe1\u606f\uff0c\u65e0\u6cd5\u76f4\u63a5 \u7528\u8be5\u6587\u4ef6\u91cd\u5efa\u6570\u636e\u5e93\uff0c\u9700\u8981\u4f7f\u7528 ovsdb-tool cluster-to-standalone \u8fdb\u884c\u683c\u5f0f\u8f6c\u6362\u3002 \u9009\u62e9 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u4e2d\u6392\u7b2c\u4e00\u7684\u8282\u70b9\u6062\u590d\u6570\u636e\u5e93\u6587\u4ef6\uff0c \u5982\u679c\u7b2c\u4e00\u4e2a\u8282\u70b9\u6570\u636e\u5e93\u6587\u4ef6\u5df2\u635f\u574f\uff0c\u4ece\u5176\u4ed6\u673a\u5668 /etc/origin/ovn \u4e0b\u590d\u5236\u6587\u4ef6\u5230\u7b2c\u4e00\u53f0\u673a\u5668 \uff0c \u6267\u884c\u4e0b\u5217\u547d\u4ee4\u751f\u6210\u6570\u636e\u5e93\u6587\u4ef6\u5907\u4efd\u3002 docker run -it -v /etc/origin/ovn:/etc/ovn kubeovn/kube-ovn:v1.12.0 bash cd /etc/ovn/ ovsdb-tool cluster-to-standalone ovnnb_db_standalone.db ovnnb_db.db ovsdb-tool cluster-to-standalone ovnsb_db_standalone.db ovnsb_db.db \u5220\u9664\u6bcf\u4e2a ovn-central \u8282\u70b9\u4e0a\u7684\u6570\u636e\u5e93\u6587\u4ef6 \u00b6 \u4e3a\u4e86\u907f\u514d\u91cd\u5efa\u96c6\u7fa4\u65f6\u4f7f\u7528\u5230\u9519\u8bef\u7684\u6570\u636e\uff0c\u9700\u8981\u5bf9\u5df2\u6709\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u6e05\u7406\uff1a mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp \u6062\u590d\u6570\u636e\u5e93\u96c6\u7fa4 \u00b6 \u5c06\u5907\u4efd\u6570\u636e\u5e93\u5206\u522b\u91cd\u547d\u540d\u4e3a ovnnb_db.db \u548c ovnsb_db.db \uff0c\u5e76\u590d\u5236\u5230 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u4e2d\u6392\u7b2c\u4e00\u673a\u5668\u7684 /etc/origin/ovn/ \u76ee\u5f55\u4e0b\uff1a mv /etc/origin/ovn/ovnnb_db_standalone.db /etc/origin/ovn/ovnnb_db.db mv /etc/origin/ovn/ovnsb_db_standalone.db /etc/origin/ovn/ovnsb_db.db \u6062\u590d ovn-central \u7684\u526f\u672c\u6570\uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl rollout status deployment/ovn-central -n kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN \u6570\u636e\u5e93\u5907\u4efd\u548c\u6062\u590d"},{"location":"ops/recover-db/#ovn","text":"\u672c\u6587\u6863\u4ecb\u7ecd\u5982\u4f55\u8fdb\u884c\u6570\u636e\u5e93\u5907\u4efd\uff0c\u4ee5\u53ca\u5728\u4e0d\u540c\u60c5\u51b5\u4e0b\u5982\u4f55\u901a\u8fc7\u5df2\u6709\u7684\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u96c6\u7fa4\u6062\u590d\u3002","title":"OVN \u6570\u636e\u5e93\u5907\u4efd\u548c\u6062\u590d"},{"location":"ops/recover-db/#_1","text":"\u5229\u7528 kubectl \u63d2\u4ef6\u7684 backup \u547d\u4ee4\u53ef\u4ee5\u5bf9\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u5907\u4efd\uff0c\u4ee5\u7528\u4e8e\u6545\u969c\u65f6\u6062\u590d\uff1a # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup # kubectl ko sb backup tar: Removing leading `/' from member names backup ovn-nb db to /root/ovnsb_db.060223191654183154.backup","title":"\u6570\u636e\u5e93\u5907\u4efd"},{"location":"ops/recover-db/#_2","text":"\u82e5\u96c6\u7fa4\u4e2d\u5b58\u5728\u90e8\u5206\u8282\u70b9\u56e0\u4e3a\u65ad\u7535\uff0c\u6587\u4ef6\u7cfb\u7edf\u6545\u969c\u6216\u78c1\u76d8\u7a7a\u95f4\u4e0d\u8db3\u5bfc\u81f4\u5de5\u4f5c\u5f02\u5e38\uff0c \u4f46\u662f\u96c6\u7fa4\u4ecd\u53ef\u6b63\u5e38\u5de5\u4f5c\u53ef\u4ee5\u901a\u8fc7\u5982\u4e0b\u6b65\u9aa4\u8fdb\u884c\u6062\u590d\u3002","title":"\u96c6\u7fa4\u90e8\u5206\u6545\u969c\u6062\u590d"},{"location":"ops/recover-db/#_3","text":"\u67e5\u770b\u5bf9\u5e94\u8282\u70b9 /var/log/ovn/ovn-northd.log \uff0c\u82e5\u63d0\u793a\u7c7b\u4f3c\u9519\u8bef\u5219\u53ef\u5224\u65ad\u6570\u636e\u5e93\u5b58\u5728\u5f02\u5e38 * ovn-northd is not running ovsdb-server: ovsdb error: error reading record 2739 from OVN_Northbound log: record 2739 advances commit index to 6308 but last log index is 6307 * Starting ovsdb-nb","title":"\u67e5\u770b\u65e5\u5fd7\u786e\u8ba4\u72b6\u6001\u5f02\u5e38"},{"location":"ops/recover-db/#_4","text":"\u6839\u636e\u65e5\u5fd7\u63d0\u793a\u662f OVN_Northbound \u8fd8\u662f OVN_Southbound \u9009\u62e9\u5bf9\u5e94\u7684\u6570\u636e\u5e93\u8fdb\u884c\u64cd\u4f5c\u3002 \u4e0a\u8ff0\u65e5\u5fd7\u63d0\u793a\u4e3a OVN_Northbound \u5219\u5bf9 ovn-nb \u8fdb\u884c\u64cd\u4f5c\uff1a # kubectl ko nb status 9182 Name: OVN_Northbound Cluster ID: e75f ( e75fa340-49ed-45ab-990e-26cb865ebc85 ) Server ID: 9182 ( 9182e8dd-b5b0-4dd8-8518-598cc1e374f3 ) Address: tcp: [ 10 .0.128.61 ] :6643 Status: cluster member Role: leader Term: 1454 Leader: self Vote: self Last Election started 1732603 ms ago, reason: timeout Last Election won: 1732587 ms ago Election timer: 1000 Log: [ 7332 , 12512 ] Entries not yet committed: 1 Entries not yet applied: 1 Connections: ->f080 <-f080 <-e631 ->e631 Disconnections: 1 Servers: f080 ( f080 at tcp: [ 10 .0.129.139 ] :6643 ) next_index = 12512 match_index = 12510 last msg 63 ms ago 9182 ( 9182 at tcp: [ 10 .0.128.61 ] :6643 ) ( self ) next_index = 10394 match_index = 12510 e631 ( e631 at tcp: [ 10 .0.131.173 ] :6643 ) next_index = 12512 match_index = 0 \u4ece\u96c6\u7fa4\u4e2d\u8e22\u51fa\u72b6\u6001\u5f02\u5e38\u8282\u70b9\uff1a kubectl ko nb kick e631 \u767b\u5f55\u5f02\u5e38\u8282\u70b9\uff0c\u5220\u9664\u5bf9\u5e94\u7684\u6570\u636e\u5e93\u6587\u4ef6\uff1a mv /etc/origin/ovn/ovnnb_db.db /tmp \u5220\u9664\u5bf9\u5e94\u8282\u70b9\u7684 ovn-central Pod\uff0c\u7b49\u5f85\u96c6\u7fa4\u81ea\u52a8\u6062\u590d\uff1a kubectl delete pod -n kube-system ovn-central-xxxx","title":"\u4ece\u96c6\u7fa4\u4e2d\u8e22\u51fa\u5bf9\u5e94\u8282\u70b9"},{"location":"ops/recover-db/#_5","text":"\u82e5\u96c6\u7fa4\u591a\u6570\u8282\u70b9\u53d7\u635f\u65e0\u6cd5\u9009\u4e3e\u51fa leader\uff0c\u8bf7\u53c2\u7167\u4e0b\u9762\u7684\u6b65\u9aa4\u8fdb\u884c\u6062\u590d\u3002","title":"\u96c6\u7fa4\u4e0d\u80fd\u6b63\u5e38\u5de5\u4f5c\u4e0b\u7684\u6062\u590d"},{"location":"ops/recover-db/#ovn-central","text":"\u8bb0\u5f55\u5f53\u524d ovn-central \u526f\u672c\u6570\u91cf\uff0c\u5e76\u505c\u6b62 ovn-central \u907f\u514d\u65b0\u7684\u6570\u636e\u5e93\u53d8\u66f4\u5f71\u54cd\u6062\u590d\uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 0","title":"\u505c\u6b62 ovn-central"},{"location":"ops/recover-db/#_6","text":"\u7531\u4e8e\u591a\u6570\u8282\u70b9\u53d7\u635f\uff0c\u9700\u8981\u4ece\u67d0\u4e2a\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u6062\u590d\u91cd\u5efa\u96c6\u7fa4\u3002\u5982\u679c\u4e4b\u524d\u5907\u4efd\u8fc7\u6570\u636e\u5e93 \u53ef\u4f7f\u7528\u4e4b\u524d\u7684\u5907\u4efd\u6587\u4ef6\u8fdb\u884c\u6062\u590d\u3002\u5982\u679c\u6ca1\u6709\u8fdb\u884c\u8fc7\u5907\u4efd\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u6b65\u9aa4\u4ece\u5df2\u6709\u7684\u6570\u636e\u5e93\u6587\u4ef6 \u4e2d\u751f\u6210\u4e00\u4e2a\u5907\u4efd\u3002 \u7531\u4e8e\u9ed8\u8ba4\u6587\u4ef6\u5939\u4e0b\u7684\u6570\u636e\u5e93\u6587\u4ef6\u4e3a\u96c6\u7fa4\u683c\u5f0f\u6570\u636e\u5e93\u6587\u4ef6\uff0c\u5305\u542b\u5f53\u524d\u96c6\u7fa4\u7684\u4fe1\u606f\uff0c\u65e0\u6cd5\u76f4\u63a5 \u7528\u8be5\u6587\u4ef6\u91cd\u5efa\u6570\u636e\u5e93\uff0c\u9700\u8981\u4f7f\u7528 ovsdb-tool cluster-to-standalone \u8fdb\u884c\u683c\u5f0f\u8f6c\u6362\u3002 \u9009\u62e9 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u4e2d\u6392\u7b2c\u4e00\u7684\u8282\u70b9\u6062\u590d\u6570\u636e\u5e93\u6587\u4ef6\uff0c \u5982\u679c\u7b2c\u4e00\u4e2a\u8282\u70b9\u6570\u636e\u5e93\u6587\u4ef6\u5df2\u635f\u574f\uff0c\u4ece\u5176\u4ed6\u673a\u5668 /etc/origin/ovn \u4e0b\u590d\u5236\u6587\u4ef6\u5230\u7b2c\u4e00\u53f0\u673a\u5668 \uff0c \u6267\u884c\u4e0b\u5217\u547d\u4ee4\u751f\u6210\u6570\u636e\u5e93\u6587\u4ef6\u5907\u4efd\u3002 docker run -it -v /etc/origin/ovn:/etc/ovn kubeovn/kube-ovn:v1.12.0 bash cd /etc/ovn/ ovsdb-tool cluster-to-standalone ovnnb_db_standalone.db ovnnb_db.db ovsdb-tool cluster-to-standalone ovnsb_db_standalone.db ovnsb_db.db","title":"\u9009\u62e9\u5907\u4efd"},{"location":"ops/recover-db/#ovn-central_1","text":"\u4e3a\u4e86\u907f\u514d\u91cd\u5efa\u96c6\u7fa4\u65f6\u4f7f\u7528\u5230\u9519\u8bef\u7684\u6570\u636e\uff0c\u9700\u8981\u5bf9\u5df2\u6709\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u6e05\u7406\uff1a mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp","title":"\u5220\u9664\u6bcf\u4e2a ovn-central \u8282\u70b9\u4e0a\u7684\u6570\u636e\u5e93\u6587\u4ef6"},{"location":"ops/recover-db/#_7","text":"\u5c06\u5907\u4efd\u6570\u636e\u5e93\u5206\u522b\u91cd\u547d\u540d\u4e3a ovnnb_db.db \u548c ovnsb_db.db \uff0c\u5e76\u590d\u5236\u5230 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u4e2d\u6392\u7b2c\u4e00\u673a\u5668\u7684 /etc/origin/ovn/ \u76ee\u5f55\u4e0b\uff1a mv /etc/origin/ovn/ovnnb_db_standalone.db /etc/origin/ovn/ovnnb_db.db mv /etc/origin/ovn/ovnsb_db_standalone.db /etc/origin/ovn/ovnsb_db.db \u6062\u590d ovn-central \u7684\u526f\u672c\u6570\uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl rollout status deployment/ovn-central -n kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6062\u590d\u6570\u636e\u5e93\u96c6\u7fa4"},{"location":"reference/architecture/","text":"\u603b\u4f53\u67b6\u6784 \u00b6 \u672c\u6587\u6863\u5c06\u4ecb\u7ecd Kube-OVN \u7684\u603b\u4f53\u67b6\u6784\uff0c\u548c\u5404\u4e2a\u7ec4\u4ef6\u7684\u529f\u80fd\u4ee5\u53ca\u5176\u4e4b\u95f4\u7684\u4ea4\u4e92\u3002 \u603b\u4f53\u6765\u770b\uff0cKube-OVN \u4f5c\u4e3a Kubernetes \u548c OVN \u4e4b\u95f4\u7684\u4e00\u4e2a\u6865\u6881\uff0c\u5c06\u6210\u719f\u7684 SDN \u548c\u4e91\u539f\u751f\u76f8\u7ed3\u5408\u3002 \u8fd9\u610f\u5473\u7740 Kube-OVN \u4e0d\u4ec5\u901a\u8fc7 OVN \u5b9e\u73b0\u4e86 Kubernetes \u4e0b\u7684\u7f51\u7edc\u89c4\u8303\uff0c\u4f8b\u5982 CNI\uff0cService \u548c Networkpolicy\uff0c\u8fd8\u5c06\u5927\u91cf\u7684 SDN \u9886\u57df\u80fd\u529b\u5e26\u5165\u4e91\u539f\u751f\uff0c\u4f8b\u5982\u903b\u8f91\u4ea4\u6362\u673a\uff0c\u903b\u8f91\u8def\u7531\u5668\uff0cVPC\uff0c\u7f51\u5173\uff0cQoS\uff0cACL \u548c\u6d41\u91cf\u955c\u50cf\u3002 \u540c\u65f6 Kube-OVN \u8fd8\u4fdd\u6301\u4e86\u826f\u597d\u7684\u5f00\u653e\u6027\u53ef\u4ee5\u548c\u8bf8\u591a\u6280\u672f\u65b9\u6848\u96c6\u6210\uff0c\u4f8b\u5982 Cilium\uff0cSubmariner\uff0cPrometheus\uff0cKubeVirt \u7b49\u7b49\u3002 \u7ec4\u4ef6\u4ecb\u7ecd \u00b6 Kube-OVN \u7684\u7ec4\u4ef6\u53ef\u4ee5\u5927\u81f4\u5206\u4e3a\u4e09\u7c7b\uff1a \u4e0a\u6e38 OVN/OVS \u7ec4\u4ef6\u3002 \u6838\u5fc3\u63a7\u5236\u5668\u548c Agent\u3002 \u76d1\u63a7\uff0c\u8fd0\u7ef4\u5de5\u5177\u548c\u6269\u5c55\u7ec4\u4ef6\u3002 \u4e0a\u6e38 OVN/OVS \u7ec4\u4ef6 \u00b6 \u8be5\u7c7b\u578b\u7ec4\u4ef6\u6765\u81ea OVN/OVS \u793e\u533a\uff0c\u5e76\u9488\u5bf9 Kube-OVN \u7684\u4f7f\u7528\u573a\u666f\u505a\u4e86\u7279\u5b9a\u4fee\u6539\u3002 OVN/OVS \u672c\u8eab\u662f\u4e00\u5957\u6210\u719f\u7684\u7ba1\u7406\u865a\u673a\u548c\u5bb9\u5668\u7684 SDN \u7cfb\u7edf\uff0c\u6211\u4eec\u5f3a\u70c8\u5efa\u8bae \u5bf9 Kube-OVN \u5b9e\u73b0\u611f\u5174\u8da3\u7684\u7528\u6237\u5148\u53bb\u8bfb\u4e00\u4e0b ovn-architecture(7) \u6765\u4e86\u89e3\u4ec0\u4e48\u662f OVN \u4ee5\u53ca \u5982\u4f55\u548c\u5b83\u8fdb\u884c\u96c6\u6210\u3002Kube-OVN \u4f7f\u7528 OVN \u7684\u5317\u5411\u63a5\u53e3\u521b\u5efa\u548c\u8c03\u6574\u865a\u62df\u7f51\u7edc\uff0c\u5e76\u5c06\u5176\u4e2d\u7684\u7f51\u7edc\u6982\u5ff5\u6620\u5c04\u5230 Kubernetes \u4e4b\u5185\u3002 \u6240\u6709 OVN/OVS \u76f8\u5173\u7ec4\u4ef6\u90fd\u5df2\u6253\u5305\u6210\u5bf9\u5e94\u955c\u50cf\uff0c\u5e76\u53ef\u5728 Kubernetes \u4e2d\u8fd0\u884c\u3002 ovn-central \u00b6 ovn-central Deployment \u8fd0\u884c OVN \u7684\u7ba1\u7406\u5e73\u9762\u7ec4\u4ef6\uff0c\u5305\u62ec ovn-nb , ovn-sb , \u548c ovn-northd \u3002 ovn-nb \uff1a \u4fdd\u5b58\u865a\u62df\u7f51\u7edc\u914d\u7f6e\uff0c\u5e76\u63d0\u4f9b API \u8fdb\u884c\u865a\u62df\u7f51\u7edc\u7ba1\u7406\u3002 kube-ovn-controller \u5c06\u4f1a\u4e3b\u8981\u548c ovn-nb \u8fdb\u884c\u4ea4\u4e92\u914d\u7f6e\u865a\u62df\u7f51\u7edc\u3002 ovn-sb \uff1a \u4fdd\u5b58\u4ece ovn-nb \u7684\u903b\u8f91\u7f51\u7edc\u751f\u6210\u7684\u903b\u8f91\u6d41\u8868\uff0c\u4ee5\u53ca\u5404\u4e2a\u8282\u70b9\u7684\u5b9e\u9645\u7269\u7406\u7f51\u7edc\u72b6\u6001\u3002 ovn-northd \uff1a\u5c06 ovn-nb \u7684\u865a\u62df\u7f51\u7edc\u7ffb\u8bd1\u6210 ovn-sb \u4e2d\u7684\u903b\u8f91\u6d41\u8868\u3002 \u591a\u4e2a ovn-central \u5b9e\u4f8b\u4f1a\u901a\u8fc7 Raft \u534f\u8bae\u540c\u6b65\u6570\u636e\u4fdd\u8bc1\u9ad8\u53ef\u7528\u3002 ovs-ovn \u00b6 ovs-ovn \u4ee5 DaemonSet \u5f62\u5f0f\u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\uff0c\u5728 Pod \u5185\u8fd0\u884c\u4e86 openvswitch , ovsdb , \u548c ovn-controller \u3002\u8fd9\u4e9b\u7ec4\u4ef6\u4f5c\u4e3a ovn-central \u7684 Agent \u5c06\u903b\u8f91\u6d41\u8868\u7ffb\u8bd1\u6210\u771f\u5b9e\u7684\u7f51\u7edc\u914d\u7f6e\u3002 \u6838\u5fc3\u63a7\u5236\u5668\u548c Agent \u00b6 \u8be5\u90e8\u5206\u4e3a Kube-OVN \u7684\u6838\u5fc3\u7ec4\u4ef6\uff0c\u4f5c\u4e3a OVN \u548c Kubernetes \u4e4b\u95f4\u7684\u4e00\u4e2a\u6865\u6881\uff0c\u5c06\u4e24\u4e2a\u7cfb\u7edf\u6253\u901a\u5e76\u5c06\u7f51\u7edc\u6982\u5ff5\u8fdb\u884c\u76f8\u4e92\u8f6c\u6362\u3002 \u5927\u90e8\u5206\u7684\u6838\u5fc3\u529f\u80fd\u90fd\u5728\u8be5\u90e8\u5206\u7ec4\u4ef6\u4e2d\u5b9e\u73b0\u3002 kube-ovn-controller \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a Deployment \u6267\u884c\u6240\u6709 Kubernetes \u5185\u8d44\u6e90\u5230 OVN \u8d44\u6e90\u7684\u7ffb\u8bd1\u5de5\u4f5c\uff0c\u5176\u4f5c\u7528\u76f8\u5f53\u4e8e\u6574\u4e2a Kube-OVN \u7cfb\u7edf\u7684\u63a7\u5236\u5e73\u9762\u3002 kube-ovn-controller \u76d1\u542c\u4e86\u6240\u6709\u548c\u7f51\u7edc\u529f\u80fd\u76f8\u5173\u8d44\u6e90\u7684\u4e8b\u4ef6\uff0c\u5e76\u6839\u636e\u8d44\u6e90\u53d8\u5316\u60c5\u51b5\u66f4\u65b0 OVN \u5185\u7684\u903b\u8f91\u7f51\u7edc\u3002\u4e3b\u8981\u76d1\u542c\u7684\u8d44\u6e90\u5305\u62ec\uff1a Pod\uff0cService\uff0cEndpoint\uff0cNode\uff0cNetworkPolicy\uff0cVPC\uff0cSubnet\uff0cVlan\uff0cProviderNetwork\u3002 \u4ee5 Pod \u4e8b\u4ef6\u4e3a\u4f8b\uff0c kube-ovn-controller \u76d1\u542c\u5230 Pod \u521b\u5efa\u4e8b\u4ef6\u540e\uff0c\u901a\u8fc7\u5185\u7f6e\u7684\u5185\u5b58 IPAM \u529f\u80fd\u5206\u914d\u5730\u5740\uff0c\u5e76\u8c03\u7528 ovn-central \u521b\u5efa \u903b\u8f91\u7aef\u53e3\uff0c\u9759\u6001\u8def\u7531\u548c\u53ef\u80fd\u7684 ACL \u89c4\u5219\u3002\u63a5\u4e0b\u6765 kube-ovn-controller \u5c06\u5206\u914d\u5230\u7684\u5730\u5740\uff0c\u548c\u5b50\u7f51\u4fe1\u606f\u4f8b\u5982 CIDR\uff0c\u7f51\u5173\uff0c\u8def\u7531\u7b49\u4fe1\u606f\u5199\u4f1a\u5230 Pod \u7684 annotation \u4e2d\u3002\u8be5 annotation \u540e\u7eed\u4f1a\u88ab kube-ovn-cni \u8bfb\u53d6\u7528\u6765\u914d\u7f6e\u672c\u5730\u7f51\u7edc\u3002 kube-ovn-cni \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\uff0c\u5b9e\u73b0 CNI \u63a5\u53e3\uff0c\u5e76\u64cd\u4f5c\u672c\u5730\u7684 OVS \u914d\u7f6e\u5355\u673a\u7f51\u7edc\u3002 \u8be5 DaemonSet \u4f1a\u590d\u5236 kube-ovn \u4e8c\u8fdb\u5236\u6587\u4ef6\u5230\u6bcf\u53f0\u673a\u5668\uff0c\u4f5c\u4e3a kubelet \u548c kube-ovn-cni \u4e4b\u95f4\u7684\u4ea4\u4e92\u5de5\u5177\uff0c\u5c06\u76f8\u5e94 CNI \u8bf7\u6c42 \u53d1\u9001\u7ed9 kube-ovn-cni \u6267\u884c\u3002\u8be5\u4e8c\u8fdb\u5236\u6587\u4ef6\u9ed8\u8ba4\u4f1a\u88ab\u590d\u5236\u5230 /opt/cni/bin \u76ee\u5f55\u4e0b\u3002 kube-ovn-cni \u4f1a\u914d\u7f6e\u5177\u4f53\u7684\u7f51\u7edc\u6765\u6267\u884c\u76f8\u5e94\u6d41\u91cf\u64cd\u4f5c\uff0c\u4e3b\u8981\u5de5\u4f5c\u5305\u62ec\uff1a \u914d\u7f6e ovn-controller \u548c vswitchd \u3002 \u5904\u7406 CNI add/del \u8bf7\u6c42\uff1a \u521b\u5efa\u5220\u9664 veth \u5e76\u548c OVS \u7aef\u53e3\u7ed1\u5b9a\u3002 \u914d\u7f6e OVS \u7aef\u53e3\u4fe1\u606f\u3002 \u66f4\u65b0\u5bbf\u4e3b\u673a\u7684 iptables/ipset/route \u7b49\u89c4\u5219\u3002 \u52a8\u6001\u66f4\u65b0\u5bb9\u5668 QoS. \u521b\u5efa\u5e76\u914d\u7f6e ovn0 \u7f51\u5361\u8054\u901a\u5bb9\u5668\u7f51\u7edc\u548c\u4e3b\u673a\u7f51\u7edc\u3002 \u914d\u7f6e\u4e3b\u673a\u7f51\u5361\u6765\u5b9e\u73b0 Vlan/Underlay/EIP \u7b49\u529f\u80fd\u3002 \u52a8\u6001\u914d\u7f6e\u96c6\u7fa4\u4e92\u8054\u7f51\u5173\u3002 \u76d1\u63a7\uff0c\u8fd0\u7ef4\u5de5\u5177\u548c\u6269\u5c55\u7ec4\u4ef6 \u00b6 \u8be5\u90e8\u5206\u7ec4\u4ef6\u4e3b\u8981\u63d0\u4f9b\u76d1\u63a7\uff0c\u8bca\u65ad\uff0c\u8fd0\u7ef4\u64cd\u4f5c\u4ee5\u53ca\u548c\u5916\u90e8\u8fdb\u884c\u5bf9\u63a5\uff0c\u5bf9 Kube-OVN \u7684\u6838\u5fc3\u7f51\u7edc\u80fd\u529b\u8fdb\u884c\u6269\u5c55\uff0c\u5e76\u7b80\u5316\u65e5\u5e38\u8fd0\u7ef4\u64cd\u4f5c\u3002 kube-ovn-speaker \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u7279\u5b9a\u6807\u7b7e\u7684\u8282\u70b9\u4e0a\uff0c\u5bf9\u5916\u53d1\u5e03\u5bb9\u5668\u7f51\u7edc\u7684\u8def\u7531\uff0c\u4f7f\u5f97\u5916\u90e8\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7 Pod IP \u8bbf\u95ee\u5bb9\u5668\u3002 \u66f4\u591a\u76f8\u5173\u4f7f\u7528\u65b9\u5f0f\u8bf7\u53c2\u8003 BGP \u652f\u6301 \u3002 kube-ovn-pinger \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\u6536\u96c6 OVS \u8fd0\u884c\u4fe1\u606f\uff0c\u8282\u70b9\u7f51\u7edc\u8d28\u91cf\uff0c\u7f51\u7edc\u5ef6\u8fdf\u7b49\u4fe1\u606f\uff0c\u6536\u96c6\u7684\u76d1\u63a7\u6307\u6807\u53ef\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002 kube-ovn-monitor \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a Deployment \u6536\u96c6 OVN \u7684\u8fd0\u884c\u4fe1\u606f\uff0c\u6536\u96c6\u7684\u76d1\u63a7\u6307\u6807\u53ef\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002 kubectl-ko \u00b6 \u8be5\u7ec4\u4ef6\u4e3a kubectl \u63d2\u4ef6\uff0c\u53ef\u4ee5\u5feb\u901f\u8fd0\u884c\u5e38\u89c1\u8fd0\u7ef4\u64cd\u4f5c\uff0c\u66f4\u591a\u4f7f\u7528\u8bf7\u53c2\u8003 kubectl \u63d2\u4ef6\u4f7f\u7528 \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u603b\u4f53\u67b6\u6784"},{"location":"reference/architecture/#_1","text":"\u672c\u6587\u6863\u5c06\u4ecb\u7ecd Kube-OVN \u7684\u603b\u4f53\u67b6\u6784\uff0c\u548c\u5404\u4e2a\u7ec4\u4ef6\u7684\u529f\u80fd\u4ee5\u53ca\u5176\u4e4b\u95f4\u7684\u4ea4\u4e92\u3002 \u603b\u4f53\u6765\u770b\uff0cKube-OVN \u4f5c\u4e3a Kubernetes \u548c OVN \u4e4b\u95f4\u7684\u4e00\u4e2a\u6865\u6881\uff0c\u5c06\u6210\u719f\u7684 SDN \u548c\u4e91\u539f\u751f\u76f8\u7ed3\u5408\u3002 \u8fd9\u610f\u5473\u7740 Kube-OVN \u4e0d\u4ec5\u901a\u8fc7 OVN \u5b9e\u73b0\u4e86 Kubernetes \u4e0b\u7684\u7f51\u7edc\u89c4\u8303\uff0c\u4f8b\u5982 CNI\uff0cService \u548c Networkpolicy\uff0c\u8fd8\u5c06\u5927\u91cf\u7684 SDN \u9886\u57df\u80fd\u529b\u5e26\u5165\u4e91\u539f\u751f\uff0c\u4f8b\u5982\u903b\u8f91\u4ea4\u6362\u673a\uff0c\u903b\u8f91\u8def\u7531\u5668\uff0cVPC\uff0c\u7f51\u5173\uff0cQoS\uff0cACL \u548c\u6d41\u91cf\u955c\u50cf\u3002 \u540c\u65f6 Kube-OVN \u8fd8\u4fdd\u6301\u4e86\u826f\u597d\u7684\u5f00\u653e\u6027\u53ef\u4ee5\u548c\u8bf8\u591a\u6280\u672f\u65b9\u6848\u96c6\u6210\uff0c\u4f8b\u5982 Cilium\uff0cSubmariner\uff0cPrometheus\uff0cKubeVirt \u7b49\u7b49\u3002","title":"\u603b\u4f53\u67b6\u6784"},{"location":"reference/architecture/#_2","text":"Kube-OVN \u7684\u7ec4\u4ef6\u53ef\u4ee5\u5927\u81f4\u5206\u4e3a\u4e09\u7c7b\uff1a \u4e0a\u6e38 OVN/OVS \u7ec4\u4ef6\u3002 \u6838\u5fc3\u63a7\u5236\u5668\u548c Agent\u3002 \u76d1\u63a7\uff0c\u8fd0\u7ef4\u5de5\u5177\u548c\u6269\u5c55\u7ec4\u4ef6\u3002","title":"\u7ec4\u4ef6\u4ecb\u7ecd"},{"location":"reference/architecture/#ovnovs","text":"\u8be5\u7c7b\u578b\u7ec4\u4ef6\u6765\u81ea OVN/OVS \u793e\u533a\uff0c\u5e76\u9488\u5bf9 Kube-OVN \u7684\u4f7f\u7528\u573a\u666f\u505a\u4e86\u7279\u5b9a\u4fee\u6539\u3002 OVN/OVS \u672c\u8eab\u662f\u4e00\u5957\u6210\u719f\u7684\u7ba1\u7406\u865a\u673a\u548c\u5bb9\u5668\u7684 SDN \u7cfb\u7edf\uff0c\u6211\u4eec\u5f3a\u70c8\u5efa\u8bae \u5bf9 Kube-OVN \u5b9e\u73b0\u611f\u5174\u8da3\u7684\u7528\u6237\u5148\u53bb\u8bfb\u4e00\u4e0b ovn-architecture(7) \u6765\u4e86\u89e3\u4ec0\u4e48\u662f OVN \u4ee5\u53ca \u5982\u4f55\u548c\u5b83\u8fdb\u884c\u96c6\u6210\u3002Kube-OVN \u4f7f\u7528 OVN \u7684\u5317\u5411\u63a5\u53e3\u521b\u5efa\u548c\u8c03\u6574\u865a\u62df\u7f51\u7edc\uff0c\u5e76\u5c06\u5176\u4e2d\u7684\u7f51\u7edc\u6982\u5ff5\u6620\u5c04\u5230 Kubernetes \u4e4b\u5185\u3002 \u6240\u6709 OVN/OVS \u76f8\u5173\u7ec4\u4ef6\u90fd\u5df2\u6253\u5305\u6210\u5bf9\u5e94\u955c\u50cf\uff0c\u5e76\u53ef\u5728 Kubernetes \u4e2d\u8fd0\u884c\u3002","title":"\u4e0a\u6e38 OVN/OVS \u7ec4\u4ef6"},{"location":"reference/architecture/#ovn-central","text":"ovn-central Deployment \u8fd0\u884c OVN \u7684\u7ba1\u7406\u5e73\u9762\u7ec4\u4ef6\uff0c\u5305\u62ec ovn-nb , ovn-sb , \u548c ovn-northd \u3002 ovn-nb \uff1a \u4fdd\u5b58\u865a\u62df\u7f51\u7edc\u914d\u7f6e\uff0c\u5e76\u63d0\u4f9b API \u8fdb\u884c\u865a\u62df\u7f51\u7edc\u7ba1\u7406\u3002 kube-ovn-controller \u5c06\u4f1a\u4e3b\u8981\u548c ovn-nb \u8fdb\u884c\u4ea4\u4e92\u914d\u7f6e\u865a\u62df\u7f51\u7edc\u3002 ovn-sb \uff1a \u4fdd\u5b58\u4ece ovn-nb \u7684\u903b\u8f91\u7f51\u7edc\u751f\u6210\u7684\u903b\u8f91\u6d41\u8868\uff0c\u4ee5\u53ca\u5404\u4e2a\u8282\u70b9\u7684\u5b9e\u9645\u7269\u7406\u7f51\u7edc\u72b6\u6001\u3002 ovn-northd \uff1a\u5c06 ovn-nb \u7684\u865a\u62df\u7f51\u7edc\u7ffb\u8bd1\u6210 ovn-sb \u4e2d\u7684\u903b\u8f91\u6d41\u8868\u3002 \u591a\u4e2a ovn-central \u5b9e\u4f8b\u4f1a\u901a\u8fc7 Raft \u534f\u8bae\u540c\u6b65\u6570\u636e\u4fdd\u8bc1\u9ad8\u53ef\u7528\u3002","title":"ovn-central"},{"location":"reference/architecture/#ovs-ovn","text":"ovs-ovn \u4ee5 DaemonSet \u5f62\u5f0f\u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\uff0c\u5728 Pod \u5185\u8fd0\u884c\u4e86 openvswitch , ovsdb , \u548c ovn-controller \u3002\u8fd9\u4e9b\u7ec4\u4ef6\u4f5c\u4e3a ovn-central \u7684 Agent \u5c06\u903b\u8f91\u6d41\u8868\u7ffb\u8bd1\u6210\u771f\u5b9e\u7684\u7f51\u7edc\u914d\u7f6e\u3002","title":"ovs-ovn"},{"location":"reference/architecture/#agent","text":"\u8be5\u90e8\u5206\u4e3a Kube-OVN \u7684\u6838\u5fc3\u7ec4\u4ef6\uff0c\u4f5c\u4e3a OVN \u548c Kubernetes \u4e4b\u95f4\u7684\u4e00\u4e2a\u6865\u6881\uff0c\u5c06\u4e24\u4e2a\u7cfb\u7edf\u6253\u901a\u5e76\u5c06\u7f51\u7edc\u6982\u5ff5\u8fdb\u884c\u76f8\u4e92\u8f6c\u6362\u3002 \u5927\u90e8\u5206\u7684\u6838\u5fc3\u529f\u80fd\u90fd\u5728\u8be5\u90e8\u5206\u7ec4\u4ef6\u4e2d\u5b9e\u73b0\u3002","title":"\u6838\u5fc3\u63a7\u5236\u5668\u548c Agent"},{"location":"reference/architecture/#kube-ovn-controller","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a Deployment \u6267\u884c\u6240\u6709 Kubernetes \u5185\u8d44\u6e90\u5230 OVN \u8d44\u6e90\u7684\u7ffb\u8bd1\u5de5\u4f5c\uff0c\u5176\u4f5c\u7528\u76f8\u5f53\u4e8e\u6574\u4e2a Kube-OVN \u7cfb\u7edf\u7684\u63a7\u5236\u5e73\u9762\u3002 kube-ovn-controller \u76d1\u542c\u4e86\u6240\u6709\u548c\u7f51\u7edc\u529f\u80fd\u76f8\u5173\u8d44\u6e90\u7684\u4e8b\u4ef6\uff0c\u5e76\u6839\u636e\u8d44\u6e90\u53d8\u5316\u60c5\u51b5\u66f4\u65b0 OVN \u5185\u7684\u903b\u8f91\u7f51\u7edc\u3002\u4e3b\u8981\u76d1\u542c\u7684\u8d44\u6e90\u5305\u62ec\uff1a Pod\uff0cService\uff0cEndpoint\uff0cNode\uff0cNetworkPolicy\uff0cVPC\uff0cSubnet\uff0cVlan\uff0cProviderNetwork\u3002 \u4ee5 Pod \u4e8b\u4ef6\u4e3a\u4f8b\uff0c kube-ovn-controller \u76d1\u542c\u5230 Pod \u521b\u5efa\u4e8b\u4ef6\u540e\uff0c\u901a\u8fc7\u5185\u7f6e\u7684\u5185\u5b58 IPAM \u529f\u80fd\u5206\u914d\u5730\u5740\uff0c\u5e76\u8c03\u7528 ovn-central \u521b\u5efa \u903b\u8f91\u7aef\u53e3\uff0c\u9759\u6001\u8def\u7531\u548c\u53ef\u80fd\u7684 ACL \u89c4\u5219\u3002\u63a5\u4e0b\u6765 kube-ovn-controller \u5c06\u5206\u914d\u5230\u7684\u5730\u5740\uff0c\u548c\u5b50\u7f51\u4fe1\u606f\u4f8b\u5982 CIDR\uff0c\u7f51\u5173\uff0c\u8def\u7531\u7b49\u4fe1\u606f\u5199\u4f1a\u5230 Pod \u7684 annotation \u4e2d\u3002\u8be5 annotation \u540e\u7eed\u4f1a\u88ab kube-ovn-cni \u8bfb\u53d6\u7528\u6765\u914d\u7f6e\u672c\u5730\u7f51\u7edc\u3002","title":"kube-ovn-controller"},{"location":"reference/architecture/#kube-ovn-cni","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\uff0c\u5b9e\u73b0 CNI \u63a5\u53e3\uff0c\u5e76\u64cd\u4f5c\u672c\u5730\u7684 OVS \u914d\u7f6e\u5355\u673a\u7f51\u7edc\u3002 \u8be5 DaemonSet \u4f1a\u590d\u5236 kube-ovn \u4e8c\u8fdb\u5236\u6587\u4ef6\u5230\u6bcf\u53f0\u673a\u5668\uff0c\u4f5c\u4e3a kubelet \u548c kube-ovn-cni \u4e4b\u95f4\u7684\u4ea4\u4e92\u5de5\u5177\uff0c\u5c06\u76f8\u5e94 CNI \u8bf7\u6c42 \u53d1\u9001\u7ed9 kube-ovn-cni \u6267\u884c\u3002\u8be5\u4e8c\u8fdb\u5236\u6587\u4ef6\u9ed8\u8ba4\u4f1a\u88ab\u590d\u5236\u5230 /opt/cni/bin \u76ee\u5f55\u4e0b\u3002 kube-ovn-cni \u4f1a\u914d\u7f6e\u5177\u4f53\u7684\u7f51\u7edc\u6765\u6267\u884c\u76f8\u5e94\u6d41\u91cf\u64cd\u4f5c\uff0c\u4e3b\u8981\u5de5\u4f5c\u5305\u62ec\uff1a \u914d\u7f6e ovn-controller \u548c vswitchd \u3002 \u5904\u7406 CNI add/del \u8bf7\u6c42\uff1a \u521b\u5efa\u5220\u9664 veth \u5e76\u548c OVS \u7aef\u53e3\u7ed1\u5b9a\u3002 \u914d\u7f6e OVS \u7aef\u53e3\u4fe1\u606f\u3002 \u66f4\u65b0\u5bbf\u4e3b\u673a\u7684 iptables/ipset/route \u7b49\u89c4\u5219\u3002 \u52a8\u6001\u66f4\u65b0\u5bb9\u5668 QoS. \u521b\u5efa\u5e76\u914d\u7f6e ovn0 \u7f51\u5361\u8054\u901a\u5bb9\u5668\u7f51\u7edc\u548c\u4e3b\u673a\u7f51\u7edc\u3002 \u914d\u7f6e\u4e3b\u673a\u7f51\u5361\u6765\u5b9e\u73b0 Vlan/Underlay/EIP \u7b49\u529f\u80fd\u3002 \u52a8\u6001\u914d\u7f6e\u96c6\u7fa4\u4e92\u8054\u7f51\u5173\u3002","title":"kube-ovn-cni"},{"location":"reference/architecture/#_3","text":"\u8be5\u90e8\u5206\u7ec4\u4ef6\u4e3b\u8981\u63d0\u4f9b\u76d1\u63a7\uff0c\u8bca\u65ad\uff0c\u8fd0\u7ef4\u64cd\u4f5c\u4ee5\u53ca\u548c\u5916\u90e8\u8fdb\u884c\u5bf9\u63a5\uff0c\u5bf9 Kube-OVN \u7684\u6838\u5fc3\u7f51\u7edc\u80fd\u529b\u8fdb\u884c\u6269\u5c55\uff0c\u5e76\u7b80\u5316\u65e5\u5e38\u8fd0\u7ef4\u64cd\u4f5c\u3002","title":"\u76d1\u63a7\uff0c\u8fd0\u7ef4\u5de5\u5177\u548c\u6269\u5c55\u7ec4\u4ef6"},{"location":"reference/architecture/#kube-ovn-speaker","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u7279\u5b9a\u6807\u7b7e\u7684\u8282\u70b9\u4e0a\uff0c\u5bf9\u5916\u53d1\u5e03\u5bb9\u5668\u7f51\u7edc\u7684\u8def\u7531\uff0c\u4f7f\u5f97\u5916\u90e8\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7 Pod IP \u8bbf\u95ee\u5bb9\u5668\u3002 \u66f4\u591a\u76f8\u5173\u4f7f\u7528\u65b9\u5f0f\u8bf7\u53c2\u8003 BGP \u652f\u6301 \u3002","title":"kube-ovn-speaker"},{"location":"reference/architecture/#kube-ovn-pinger","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\u6536\u96c6 OVS \u8fd0\u884c\u4fe1\u606f\uff0c\u8282\u70b9\u7f51\u7edc\u8d28\u91cf\uff0c\u7f51\u7edc\u5ef6\u8fdf\u7b49\u4fe1\u606f\uff0c\u6536\u96c6\u7684\u76d1\u63a7\u6307\u6807\u53ef\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002","title":"kube-ovn-pinger"},{"location":"reference/architecture/#kube-ovn-monitor","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a Deployment \u6536\u96c6 OVN \u7684\u8fd0\u884c\u4fe1\u606f\uff0c\u6536\u96c6\u7684\u76d1\u63a7\u6307\u6807\u53ef\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002","title":"kube-ovn-monitor"},{"location":"reference/architecture/#kubectl-ko","text":"\u8be5\u7ec4\u4ef6\u4e3a kubectl \u63d2\u4ef6\uff0c\u53ef\u4ee5\u5feb\u901f\u8fd0\u884c\u5e38\u89c1\u8fd0\u7ef4\u64cd\u4f5c\uff0c\u66f4\u591a\u4f7f\u7528\u8bf7\u53c2\u8003 kubectl \u63d2\u4ef6\u4f7f\u7528 \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kubectl-ko"},{"location":"reference/dev-env/","text":"\u5f00\u53d1\u73af\u5883\u6784\u5efa \u00b6 \u73af\u5883\u51c6\u5907 \u00b6 Kube-OVN \u4f7f\u7528 Go 1.20 \u5f00\u53d1\u5e76\u4f7f\u7528 Go Modules \u7ba1\u7406\u4f9d\u8d56\uff0c \u8bf7\u786e\u8ba4\u73af\u5883\u53d8\u91cf GO111MODULE=\"on\" \u3002 gosec \u88ab\u7528\u6765\u626b\u63cf\u4ee3\u7801\u5b89\u5168\u76f8\u5173\u95ee\u9898\uff0c\u9700\u8981\u5728\u5f00\u53d1\u73af\u5883\u5b89\u88c5\uff1a go get github.com/securego/gosec/v2/cmd/gosec \u4e3a\u4e86\u964d\u4f4e\u6700\u7ec8\u751f\u6210\u955c\u50cf\u5927\u5c0f\uff0cKube-OVN \u4f7f\u7528\u4e86\u90e8\u5206 Docker buildx \u8bd5\u9a8c\u7279\u6027\uff0c\u8bf7\u66f4\u65b0 Docker \u81f3\u6700\u65b0\u7248\u672c \u5e76\u5f00\u542f buildx: docker buildx create --use \u6784\u5efa\u955c\u50cf \u00b6 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u4ee3\u7801\uff0c\u5e76\u751f\u6210\u8fd0\u884c Kube-OVN \u6240\u9700\u955c\u50cf\uff1a git clone https://github.com/kubeovn/kube-ovn.git cd kube-ovn make release \u5982\u9700\u6784\u5efa\u5728 ARM \u73af\u5883\u4e0b\u8fd0\u884c\u7684\u955c\u50cf\uff0c\u8bf7\u6267\u884c\u4e0b\u9762\u7684\u547d\u4ee4\uff1a make release-arm \u6784\u5efa base \u955c\u50cf \u00b6 \u5982\u9700\u8981\u66f4\u6539\u64cd\u4f5c\u7cfb\u7edf\u7248\u672c\uff0c\u4f9d\u8d56\u5e93\uff0cOVS/OVN \u4ee3\u7801\u7b49\uff0c\u9700\u8981\u5bf9 base \u955c\u50cf\u8fdb\u884c\u91cd\u65b0\u6784\u5efa\u3002 base \u955c\u50cf\u4f7f\u7528\u7684 Dockerfile \u4e3a dist/images/Dockerfile.base \u3002 \u6784\u5efa\u65b9\u6cd5\uff1a # build x86 base image make base-amd64 # build arm base image make base-arm64 \u8fd0\u884c E2E \u00b6 Kube-OVN \u4f7f\u7528 KIND \u6784\u5efa\u672c\u5730 Kubernetes \u96c6\u7fa4\uff0c j2cli \u6e32\u67d3\u6a21\u677f\uff0c Ginkgo \u6765\u8fd0\u884c\u6d4b\u8bd5\u4ee3\u7801\u3002\u8bf7\u53c2\u8003\u76f8\u5173\u6587\u6863\u8fdb\u884c\u4f9d\u8d56\u5b89\u88c5\u3002 \u672c\u5730\u6267\u884c E2E \u6d4b\u8bd5\uff1a make kind-init make kind-install make e2e \u5982\u9700\u8fd0\u884c Underlay E2E \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install-underlay make e2e-underlay-single-nic \u5982\u9700\u8fd0\u884c ovn vpc nat gw eip, fip, snat, dnat \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make ovn-vpc-nat-gw-conformance-e2e \u5982\u9700\u8fd0\u884c iptables vpc nat gw eip, fip, snat, dnat \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make kind-install-vpc-nat-gw make iptables-vpc-nat-gw-conformance-e2e \u5982\u9700\u8fd0\u884c loadbalancer service \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make kind-install-lb-svc make kube-ovn-lb-svc-conformance-e2e \u5982\u9700\u6e05\u7406\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-clean \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5f00\u53d1\u73af\u5883\u6784\u5efa"},{"location":"reference/dev-env/#_1","text":"","title":"\u5f00\u53d1\u73af\u5883\u6784\u5efa"},{"location":"reference/dev-env/#_2","text":"Kube-OVN \u4f7f\u7528 Go 1.20 \u5f00\u53d1\u5e76\u4f7f\u7528 Go Modules \u7ba1\u7406\u4f9d\u8d56\uff0c \u8bf7\u786e\u8ba4\u73af\u5883\u53d8\u91cf GO111MODULE=\"on\" \u3002 gosec \u88ab\u7528\u6765\u626b\u63cf\u4ee3\u7801\u5b89\u5168\u76f8\u5173\u95ee\u9898\uff0c\u9700\u8981\u5728\u5f00\u53d1\u73af\u5883\u5b89\u88c5\uff1a go get github.com/securego/gosec/v2/cmd/gosec \u4e3a\u4e86\u964d\u4f4e\u6700\u7ec8\u751f\u6210\u955c\u50cf\u5927\u5c0f\uff0cKube-OVN \u4f7f\u7528\u4e86\u90e8\u5206 Docker buildx \u8bd5\u9a8c\u7279\u6027\uff0c\u8bf7\u66f4\u65b0 Docker \u81f3\u6700\u65b0\u7248\u672c \u5e76\u5f00\u542f buildx: docker buildx create --use","title":"\u73af\u5883\u51c6\u5907"},{"location":"reference/dev-env/#_3","text":"\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u4ee3\u7801\uff0c\u5e76\u751f\u6210\u8fd0\u884c Kube-OVN \u6240\u9700\u955c\u50cf\uff1a git clone https://github.com/kubeovn/kube-ovn.git cd kube-ovn make release \u5982\u9700\u6784\u5efa\u5728 ARM \u73af\u5883\u4e0b\u8fd0\u884c\u7684\u955c\u50cf\uff0c\u8bf7\u6267\u884c\u4e0b\u9762\u7684\u547d\u4ee4\uff1a make release-arm","title":"\u6784\u5efa\u955c\u50cf"},{"location":"reference/dev-env/#base","text":"\u5982\u9700\u8981\u66f4\u6539\u64cd\u4f5c\u7cfb\u7edf\u7248\u672c\uff0c\u4f9d\u8d56\u5e93\uff0cOVS/OVN \u4ee3\u7801\u7b49\uff0c\u9700\u8981\u5bf9 base \u955c\u50cf\u8fdb\u884c\u91cd\u65b0\u6784\u5efa\u3002 base \u955c\u50cf\u4f7f\u7528\u7684 Dockerfile \u4e3a dist/images/Dockerfile.base \u3002 \u6784\u5efa\u65b9\u6cd5\uff1a # build x86 base image make base-amd64 # build arm base image make base-arm64","title":"\u6784\u5efa base \u955c\u50cf"},{"location":"reference/dev-env/#e2e","text":"Kube-OVN \u4f7f\u7528 KIND \u6784\u5efa\u672c\u5730 Kubernetes \u96c6\u7fa4\uff0c j2cli \u6e32\u67d3\u6a21\u677f\uff0c Ginkgo \u6765\u8fd0\u884c\u6d4b\u8bd5\u4ee3\u7801\u3002\u8bf7\u53c2\u8003\u76f8\u5173\u6587\u6863\u8fdb\u884c\u4f9d\u8d56\u5b89\u88c5\u3002 \u672c\u5730\u6267\u884c E2E \u6d4b\u8bd5\uff1a make kind-init make kind-install make e2e \u5982\u9700\u8fd0\u884c Underlay E2E \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install-underlay make e2e-underlay-single-nic \u5982\u9700\u8fd0\u884c ovn vpc nat gw eip, fip, snat, dnat \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make ovn-vpc-nat-gw-conformance-e2e \u5982\u9700\u8fd0\u884c iptables vpc nat gw eip, fip, snat, dnat \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make kind-install-vpc-nat-gw make iptables-vpc-nat-gw-conformance-e2e \u5982\u9700\u8fd0\u884c loadbalancer service \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make kind-install-lb-svc make kube-ovn-lb-svc-conformance-e2e \u5982\u9700\u6e05\u7406\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-clean \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8fd0\u884c E2E"},{"location":"reference/document-convention/","text":"\u6587\u6863\u89c4\u8303 \u00b6 \u4e3a\u4e86\u4fdd\u8bc1\u6587\u6863\u98ce\u683c\u4e00\u81f4\uff0c\u8bf7\u5728\u63d0\u4ea4\u6587\u6863\u65f6\u9075\u5faa\u4e0b\u5217\u7684\u98ce\u683c\u89c4\u8303\u3002 \u6807\u70b9 \u00b6 \u4e2d\u6587\u6587\u6863\u4e2d\u6587\u672c\u5185\u5bb9\u6240\u6709\u6807\u70b9\u5e94\u4f7f\u7528\u4e2d\u6587\u683c\u5f0f\u6807\u70b9\uff0c\u82f1\u6587\u6587\u6863\u4e2d\u6240\u6709\u6587\u672c\u5185\u5bb9\u4e2d\u5e94\u4f7f\u7528\u82f1\u6587\u6807\u70b9\u3002 Bad Good \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c,\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528,\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc. \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002 \u82f1\u6587\u6570\u5b57\u548c\u4e2d\u6587\u5e94\u8be5\u7528\u7a7a\u683c\u8fdb\u884c\u5206\u9694\u3002 Bad Good Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 \u793a\u4f8b\u5185\u5bb9\u5e94\u8be5\u4ee5 \uff1a \u5f00\u542f\uff0c\u5176\u4ed6\u53e5\u5c3e\u9700\u8981\u7528 \u3002 \u7ed3\u675f\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\u3002 wget 127 .0.0.1 \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget 127 .0.0.1 \u4ee3\u7801\u5757 \u00b6 yaml \u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a yaml\u3002 Bad Good ```` apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` ````yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` \u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a bash\u3002 Bad Good ```` wget 127.0.0.1 ```` ````bash wget 127.0.0.1 ```` \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u5305\u542b\u8f93\u51fa\u5185\u5bb9\uff0c\u5219\u6240\u6267\u884c\u547d\u4ee4\u9700\u8981\u4ee5 # \u5f00\u59cb\uff0c\u4ee5\u533a\u5206\u8f93\u5165\u4e0e\u8f93\u51fa\u3002 Bad Good oilbeater@macdeMac-3 ~ ping 114 .114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms # ping 114.114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u53ea\u5305\u542b\u6267\u884c\u547d\u4ee4\uff0c\u6ca1\u6709\u8f93\u51fa\u7ed3\u679c\uff0c\u5219\u591a\u6761\u547d\u4ee4\u65e0\u9700 # \u5f00\u59cb\u3002 Bad Good # mv /etc/origin/ovn/ovnnb_db.db /tmp # mv /etc/origin/ovn/ovnsb_db.db /tmp mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp \u94fe\u63a5 \u00b6 \u7ad9\u5185\u94fe\u63a5\u4f7f\u7528\u5bf9\u5e94 md \u6587\u4ef6\u8def\u5f84\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](http://kubeovn.github.io/prepare)\u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](./prepare.md)\u3002 Bad Good \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io )\u3002 \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io ){: target=\"_blank\" }\u3002 \u7a7a\u884c \u00b6 \u4e0d\u540c\u903b\u8f91\u5757\uff0c\u4f8b\u5982\u6807\u9898\u548c\u6587\u672c\uff0c\u6587\u672c\u548c\u4ee3\u7801\uff0c\u6587\u672c\u548c\u7f16\u53f7\u4e4b\u95f4\u9700\u8981\u7528\u7a7a\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0d\u540c\u903b\u8f91\u5757\u4e4b\u95f4\u53ea\u4f7f\u7528 \u4e00\u4e2a \u7a7a\u884c\u8fdb\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6587\u6863\u89c4\u8303"},{"location":"reference/document-convention/#_1","text":"\u4e3a\u4e86\u4fdd\u8bc1\u6587\u6863\u98ce\u683c\u4e00\u81f4\uff0c\u8bf7\u5728\u63d0\u4ea4\u6587\u6863\u65f6\u9075\u5faa\u4e0b\u5217\u7684\u98ce\u683c\u89c4\u8303\u3002","title":"\u6587\u6863\u89c4\u8303"},{"location":"reference/document-convention/#_2","text":"\u4e2d\u6587\u6587\u6863\u4e2d\u6587\u672c\u5185\u5bb9\u6240\u6709\u6807\u70b9\u5e94\u4f7f\u7528\u4e2d\u6587\u683c\u5f0f\u6807\u70b9\uff0c\u82f1\u6587\u6587\u6863\u4e2d\u6240\u6709\u6587\u672c\u5185\u5bb9\u4e2d\u5e94\u4f7f\u7528\u82f1\u6587\u6807\u70b9\u3002 Bad Good \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c,\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528,\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc. \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002 \u82f1\u6587\u6570\u5b57\u548c\u4e2d\u6587\u5e94\u8be5\u7528\u7a7a\u683c\u8fdb\u884c\u5206\u9694\u3002 Bad Good Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 \u793a\u4f8b\u5185\u5bb9\u5e94\u8be5\u4ee5 \uff1a \u5f00\u542f\uff0c\u5176\u4ed6\u53e5\u5c3e\u9700\u8981\u7528 \u3002 \u7ed3\u675f\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\u3002 wget 127 .0.0.1 \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget 127 .0.0.1","title":"\u6807\u70b9"},{"location":"reference/document-convention/#_3","text":"yaml \u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a yaml\u3002 Bad Good ```` apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` ````yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` \u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a bash\u3002 Bad Good ```` wget 127.0.0.1 ```` ````bash wget 127.0.0.1 ```` \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u5305\u542b\u8f93\u51fa\u5185\u5bb9\uff0c\u5219\u6240\u6267\u884c\u547d\u4ee4\u9700\u8981\u4ee5 # \u5f00\u59cb\uff0c\u4ee5\u533a\u5206\u8f93\u5165\u4e0e\u8f93\u51fa\u3002 Bad Good oilbeater@macdeMac-3 ~ ping 114 .114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms # ping 114.114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u53ea\u5305\u542b\u6267\u884c\u547d\u4ee4\uff0c\u6ca1\u6709\u8f93\u51fa\u7ed3\u679c\uff0c\u5219\u591a\u6761\u547d\u4ee4\u65e0\u9700 # \u5f00\u59cb\u3002 Bad Good # mv /etc/origin/ovn/ovnnb_db.db /tmp # mv /etc/origin/ovn/ovnsb_db.db /tmp mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp","title":"\u4ee3\u7801\u5757"},{"location":"reference/document-convention/#_4","text":"\u7ad9\u5185\u94fe\u63a5\u4f7f\u7528\u5bf9\u5e94 md \u6587\u4ef6\u8def\u5f84\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](http://kubeovn.github.io/prepare)\u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](./prepare.md)\u3002 Bad Good \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io )\u3002 \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io ){: target=\"_blank\" }\u3002","title":"\u94fe\u63a5"},{"location":"reference/document-convention/#_5","text":"\u4e0d\u540c\u903b\u8f91\u5757\uff0c\u4f8b\u5982\u6807\u9898\u548c\u6587\u672c\uff0c\u6587\u672c\u548c\u4ee3\u7801\uff0c\u6587\u672c\u548c\u7f16\u53f7\u4e4b\u95f4\u9700\u8981\u7528\u7a7a\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0d\u540c\u903b\u8f91\u5757\u4e4b\u95f4\u53ea\u4f7f\u7528 \u4e00\u4e2a \u7a7a\u884c\u8fdb\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7a7a\u884c"},{"location":"reference/feature-stage/","text":"\u529f\u80fd\u6210\u719f\u5ea6 \u00b6 \u5728 Kube-OVN \u4e2d\u6839\u636e\u529f\u80fd\u4f7f\u7528\u5ea6\uff0c\u6587\u6863\u5b8c\u5584\u7a0b\u5ea6\u548c\u6d4b\u8bd5\u8986\u76d6\u7a0b\u5ea6\u5c06\u529f\u80fd\u6210\u719f\u5ea6\u5206\u4e3a Alpha \uff0c Beta \u548c GA \u4e09\u4e2a\u9636\u6bb5\u3002 \u6210\u719f\u5ea6\u5b9a\u4e49 \u00b6 \u5bf9\u4e8e Alpha \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6ca1\u6709\u5b8c\u6574\u7684\u6587\u6863\u548c\u5b8c\u5584\u7684\u6d4b\u8bd5\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u672a\u6765\u53ef\u80fd\u4f1a\u53d1\u751f\u53d8\u5316\u751a\u81f3\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd API \u4e0d\u4fdd\u8bc1\u7a33\u5b9a\uff0c\u53ef\u80fd\u4f1a\u88ab\u79fb\u9664\u3002 \u8be5\u529f\u80fd\u7684\u793e\u533a\u652f\u6301\u4f18\u5148\u7ea7\u8f83\u4f4e\uff0c\u4e14\u65e0\u6cd5\u4fdd\u8bc1\u957f\u671f\u652f\u6301\u3002 \u7531\u4e8e\u529f\u80fd\u7a33\u5b9a\u6027\u548c\u957f\u671f\u652f\u6301\u65e0\u6cd5\u4fdd\u8bc1\uff0c\u53ef\u4ee5\u8fdb\u884c\u6d4b\u8bd5\u9a8c\u8bc1\uff0c\u4f46\u4e0d\u63a8\u8350\u751f\u4ea7\u4f7f\u7528\u3002 \u5bf9\u4e8e Beta \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6709\u90e8\u5206\u6587\u6863\u548c\u6d4b\u8bd5\uff0c\u4f46\u662f\u4e0d\u4fdd\u8bc1\u5b8c\u6574\u7684\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u672a\u6765\u53ef\u80fd\u53d1\u751f\u53d8\u5316\uff0c\u5347\u7ea7\u53ef\u80fd\u4f1a\u5f71\u54cd\u7f51\u7edc\uff0c\u4f46\u4e0d\u4f1a\u88ab\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd API \u672a\u6765\u53ef\u80fd\u4f1a\u53d1\u751f\u53d8\u5316\uff0c\u5b57\u6bb5\u53ef\u80fd\u4f1a\u8fdb\u884c\u8c03\u6574\uff0c\u4f46\u4e0d\u4f1a\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd\u4f1a\u5f97\u5230\u793e\u533a\u7684\u957f\u671f\u652f\u6301\u3002 \u7531\u4e8e\u529f\u80fd\u4f1a\u5f97\u5230\u957f\u671f\u652f\u6301\uff0c\u53ef\u4ee5\u5728\u975e\u5173\u952e\u4e1a\u52a1\u4e0a\u8fdb\u884c\u4f7f\u7528\uff0c\u4f46\u662f\u7531\u4e8e\u529f\u80fd\u548c API \u5b58\u5728\u53d8\u5316\u7684\u53ef\u80fd\uff0c\u53ef\u80fd\u4f1a\u5728\u5347\u7ea7\u4e2d\u51fa\u73b0\u4e2d\u65ad\uff0c\u4e0d\u63a8\u8350\u5728\u5173\u952e\u751f\u4ea7\u4e1a\u52a1\u4e0a\u4f7f\u7528\u3002 \u5bf9\u4e8e GA \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6709\u5b8c\u6574\u7684\u6587\u6863\u548c\u6d4b\u8bd5\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u4f1a\u4fdd\u6301\u7a33\u5b9a\uff0c\u5347\u7ea7\u4f1a\u4fdd\u8bc1\u5e73\u6ed1\u3002 \u8be5\u529f\u80fd API \u4e0d\u4f1a\u53d1\u751f\u7834\u574f\u6027\u53d8\u5316\u3002 \u8be5\u529f\u80fd\u4f1a\u5f97\u5230\u793e\u533a\u9ad8\u4f18\u5148\u7ea7\u652f\u6301\uff0c\u5e76\u4f1a\u4fdd\u8bc1\u957f\u671f\u652f\u6301\u3002 \u6210\u719f\u5ea6\u5217\u8868 \u00b6 \u672c\u5217\u8868\u7edf\u8ba1\u4ece v1.8 \u7248\u672c\u4e2d\u5305\u542b\u7684\u529f\u80fd\u5bf9\u5e94\u6210\u719f\u5ea6\u3002 \u529f\u80fd \u9ed8\u8ba4\u5f00\u542f \u72b6\u6001 \u5f00\u59cb\uff08Since\uff09 \u7ed3\u675f\uff08Until\uff09 Namespaced Subnet true GA 1.8 \u5206\u5e03\u5f0f\u7f51\u5173 true GA 1.8 \u4e3b\u4ece\u6a21\u5f0f\u96c6\u4e2d\u5f0f\u7f51\u5173 true GA 1.8 ECMP \u6a21\u5f0f\u96c6\u4e2d\u5f0f\u7f51\u5173 false Beta 1.8 \u5b50\u7f51 ACL true Alpha 1.9 \u5b50\u7f51\u9694\u79bb (\u672a\u6765\u4f1a\u548c\u5b50\u7f51 ACL \u5408\u5e76) true Beta 1.8 Underlay \u5b50\u7f51 true GA 1.8 \u591a\u7f51\u5361\u7ba1\u7406 true Beta 1.8 \u5b50\u7f51 DHCP false Alpha 1.10 \u5b50\u7f51\u8bbe\u7f6e\u5916\u90e8\u7f51\u5173 false Alpha 1.8 \u4f7f\u7528 OVN-IC \u8fdb\u884c\u96c6\u7fa4\u4e92\u8054 false Beta 1.8 \u4f7f\u7528 Submariner \u8fdb\u884c\u96c6\u7fa4\u4e92\u8054 false Alpha 1.9 \u5b50\u7f51 VIP \u9884\u7559 true Alpha 1.10 \u521b\u5efa\u81ea\u5b9a\u4e49 VPC true Beta 1.8 \u81ea\u5b9a\u4e49 VPC \u6d6e\u52a8 IP/SNAT/DNAT true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u9759\u6001\u8def\u7531 true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u7b56\u7565\u8def\u7531 true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u5b89\u5168\u7ec4 true Alpha 1.10 \u5bb9\u5668\u6700\u5927\u5e26\u5bbd QoS true GA 1.8 linux-netem QoS true Alpha 1.9 Prometheus \u96c6\u6210 false GA 1.8 Grafana \u96c6\u6210 false GA 1.8 \u53cc\u6808\u7f51\u7edc false GA 1.8 \u9ed8\u8ba4 VPC EIP/SNAT false Beta 1.8 \u6d41\u91cf\u955c\u50cf false GA 1.8 NetworkPolicy true Beta 1.8 Webhook false Alpha 1.10 \u6027\u80fd\u8c03\u4f18 false Beta 1.8 Overlay \u5b50\u7f51\u9759\u6001\u8def\u7531\u5bf9\u5916\u66b4\u9732 false Alpha 1.8 Overlay \u5b50\u7f51 BGP \u5bf9\u5916\u66b4\u9732 false Alpha 1.9 Cilium \u96c6\u6210 false Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u4e92\u8054 false Alpha 1.10 Mellanox Offload false Alpha 1.8 \u82af\u542f\u6e90 Offload false Alpha 1.10 Windows \u652f\u6301 false Alpha 1.10 DPDK \u652f\u6301 false Alpha 1.10 OpenStack \u96c6\u6210 false Alpha 1.9 \u5355\u4e2a Pod \u56fa\u5b9a IP/Mac true GA 1.8 Workload \u56fa\u5b9a IP true GA 1.8 StatefulSet \u56fa\u5b9a IP true GA 1.8 VM \u56fa\u5b9a IP false Beta 1.9 \u9ed8\u8ba4 VPC Load Balancer \u7c7b\u578b Service false Alpha 1.11 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 false Alpha 1.11 \u81ea\u5b9a\u4e49 VPC DNS false Alpha 1.11 Underlay \u548c Overlay \u4e92\u901a false Alpha 1.11 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u529f\u80fd\u6210\u719f\u5ea6"},{"location":"reference/feature-stage/#_1","text":"\u5728 Kube-OVN \u4e2d\u6839\u636e\u529f\u80fd\u4f7f\u7528\u5ea6\uff0c\u6587\u6863\u5b8c\u5584\u7a0b\u5ea6\u548c\u6d4b\u8bd5\u8986\u76d6\u7a0b\u5ea6\u5c06\u529f\u80fd\u6210\u719f\u5ea6\u5206\u4e3a Alpha \uff0c Beta \u548c GA \u4e09\u4e2a\u9636\u6bb5\u3002","title":"\u529f\u80fd\u6210\u719f\u5ea6"},{"location":"reference/feature-stage/#_2","text":"\u5bf9\u4e8e Alpha \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6ca1\u6709\u5b8c\u6574\u7684\u6587\u6863\u548c\u5b8c\u5584\u7684\u6d4b\u8bd5\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u672a\u6765\u53ef\u80fd\u4f1a\u53d1\u751f\u53d8\u5316\u751a\u81f3\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd API \u4e0d\u4fdd\u8bc1\u7a33\u5b9a\uff0c\u53ef\u80fd\u4f1a\u88ab\u79fb\u9664\u3002 \u8be5\u529f\u80fd\u7684\u793e\u533a\u652f\u6301\u4f18\u5148\u7ea7\u8f83\u4f4e\uff0c\u4e14\u65e0\u6cd5\u4fdd\u8bc1\u957f\u671f\u652f\u6301\u3002 \u7531\u4e8e\u529f\u80fd\u7a33\u5b9a\u6027\u548c\u957f\u671f\u652f\u6301\u65e0\u6cd5\u4fdd\u8bc1\uff0c\u53ef\u4ee5\u8fdb\u884c\u6d4b\u8bd5\u9a8c\u8bc1\uff0c\u4f46\u4e0d\u63a8\u8350\u751f\u4ea7\u4f7f\u7528\u3002 \u5bf9\u4e8e Beta \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6709\u90e8\u5206\u6587\u6863\u548c\u6d4b\u8bd5\uff0c\u4f46\u662f\u4e0d\u4fdd\u8bc1\u5b8c\u6574\u7684\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u672a\u6765\u53ef\u80fd\u53d1\u751f\u53d8\u5316\uff0c\u5347\u7ea7\u53ef\u80fd\u4f1a\u5f71\u54cd\u7f51\u7edc\uff0c\u4f46\u4e0d\u4f1a\u88ab\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd API \u672a\u6765\u53ef\u80fd\u4f1a\u53d1\u751f\u53d8\u5316\uff0c\u5b57\u6bb5\u53ef\u80fd\u4f1a\u8fdb\u884c\u8c03\u6574\uff0c\u4f46\u4e0d\u4f1a\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd\u4f1a\u5f97\u5230\u793e\u533a\u7684\u957f\u671f\u652f\u6301\u3002 \u7531\u4e8e\u529f\u80fd\u4f1a\u5f97\u5230\u957f\u671f\u652f\u6301\uff0c\u53ef\u4ee5\u5728\u975e\u5173\u952e\u4e1a\u52a1\u4e0a\u8fdb\u884c\u4f7f\u7528\uff0c\u4f46\u662f\u7531\u4e8e\u529f\u80fd\u548c API \u5b58\u5728\u53d8\u5316\u7684\u53ef\u80fd\uff0c\u53ef\u80fd\u4f1a\u5728\u5347\u7ea7\u4e2d\u51fa\u73b0\u4e2d\u65ad\uff0c\u4e0d\u63a8\u8350\u5728\u5173\u952e\u751f\u4ea7\u4e1a\u52a1\u4e0a\u4f7f\u7528\u3002 \u5bf9\u4e8e GA \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6709\u5b8c\u6574\u7684\u6587\u6863\u548c\u6d4b\u8bd5\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u4f1a\u4fdd\u6301\u7a33\u5b9a\uff0c\u5347\u7ea7\u4f1a\u4fdd\u8bc1\u5e73\u6ed1\u3002 \u8be5\u529f\u80fd API \u4e0d\u4f1a\u53d1\u751f\u7834\u574f\u6027\u53d8\u5316\u3002 \u8be5\u529f\u80fd\u4f1a\u5f97\u5230\u793e\u533a\u9ad8\u4f18\u5148\u7ea7\u652f\u6301\uff0c\u5e76\u4f1a\u4fdd\u8bc1\u957f\u671f\u652f\u6301\u3002","title":"\u6210\u719f\u5ea6\u5b9a\u4e49"},{"location":"reference/feature-stage/#_3","text":"\u672c\u5217\u8868\u7edf\u8ba1\u4ece v1.8 \u7248\u672c\u4e2d\u5305\u542b\u7684\u529f\u80fd\u5bf9\u5e94\u6210\u719f\u5ea6\u3002 \u529f\u80fd \u9ed8\u8ba4\u5f00\u542f \u72b6\u6001 \u5f00\u59cb\uff08Since\uff09 \u7ed3\u675f\uff08Until\uff09 Namespaced Subnet true GA 1.8 \u5206\u5e03\u5f0f\u7f51\u5173 true GA 1.8 \u4e3b\u4ece\u6a21\u5f0f\u96c6\u4e2d\u5f0f\u7f51\u5173 true GA 1.8 ECMP \u6a21\u5f0f\u96c6\u4e2d\u5f0f\u7f51\u5173 false Beta 1.8 \u5b50\u7f51 ACL true Alpha 1.9 \u5b50\u7f51\u9694\u79bb (\u672a\u6765\u4f1a\u548c\u5b50\u7f51 ACL \u5408\u5e76) true Beta 1.8 Underlay \u5b50\u7f51 true GA 1.8 \u591a\u7f51\u5361\u7ba1\u7406 true Beta 1.8 \u5b50\u7f51 DHCP false Alpha 1.10 \u5b50\u7f51\u8bbe\u7f6e\u5916\u90e8\u7f51\u5173 false Alpha 1.8 \u4f7f\u7528 OVN-IC \u8fdb\u884c\u96c6\u7fa4\u4e92\u8054 false Beta 1.8 \u4f7f\u7528 Submariner \u8fdb\u884c\u96c6\u7fa4\u4e92\u8054 false Alpha 1.9 \u5b50\u7f51 VIP \u9884\u7559 true Alpha 1.10 \u521b\u5efa\u81ea\u5b9a\u4e49 VPC true Beta 1.8 \u81ea\u5b9a\u4e49 VPC \u6d6e\u52a8 IP/SNAT/DNAT true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u9759\u6001\u8def\u7531 true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u7b56\u7565\u8def\u7531 true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u5b89\u5168\u7ec4 true Alpha 1.10 \u5bb9\u5668\u6700\u5927\u5e26\u5bbd QoS true GA 1.8 linux-netem QoS true Alpha 1.9 Prometheus \u96c6\u6210 false GA 1.8 Grafana \u96c6\u6210 false GA 1.8 \u53cc\u6808\u7f51\u7edc false GA 1.8 \u9ed8\u8ba4 VPC EIP/SNAT false Beta 1.8 \u6d41\u91cf\u955c\u50cf false GA 1.8 NetworkPolicy true Beta 1.8 Webhook false Alpha 1.10 \u6027\u80fd\u8c03\u4f18 false Beta 1.8 Overlay \u5b50\u7f51\u9759\u6001\u8def\u7531\u5bf9\u5916\u66b4\u9732 false Alpha 1.8 Overlay \u5b50\u7f51 BGP \u5bf9\u5916\u66b4\u9732 false Alpha 1.9 Cilium \u96c6\u6210 false Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u4e92\u8054 false Alpha 1.10 Mellanox Offload false Alpha 1.8 \u82af\u542f\u6e90 Offload false Alpha 1.10 Windows \u652f\u6301 false Alpha 1.10 DPDK \u652f\u6301 false Alpha 1.10 OpenStack \u96c6\u6210 false Alpha 1.9 \u5355\u4e2a Pod \u56fa\u5b9a IP/Mac true GA 1.8 Workload \u56fa\u5b9a IP true GA 1.8 StatefulSet \u56fa\u5b9a IP true GA 1.8 VM \u56fa\u5b9a IP false Beta 1.9 \u9ed8\u8ba4 VPC Load Balancer \u7c7b\u578b Service false Alpha 1.11 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 false Alpha 1.11 \u81ea\u5b9a\u4e49 VPC DNS false Alpha 1.11 Underlay \u548c Overlay \u4e92\u901a false Alpha 1.11 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6210\u719f\u5ea6\u5217\u8868"},{"location":"reference/iptables-rules/","text":"Iptables \u89c4\u5219 \u00b6 Kube-OVN \u4f7f\u7528 ipset \u53ca iptables \u8f85\u52a9\u5b9e\u73b0\u9ed8\u8ba4 VPC \u4e0b\u5bb9\u5668\u7f51\u7edc\uff08Overlay\uff09\u7f51\u5173 NAT \u7684\u529f\u80fd\u3002 \u4f7f\u7528\u7684 ipset \u5982\u4e0b\u8868\u6240\u793a\uff1a \u540d\u79f0\uff08IPv4/IPv6\uff09 \u7c7b\u578b \u5b58\u50a8\u5bf9\u8c61 ovn40services/ovn60services hash:net Service \u7f51\u6bb5 ovn40subnets/ovn60subnets hash:net Overlay \u5b50\u7f51\u7f51\u6bb5\u4ee5\u53ca NodeLocal DNS IP \u5730\u5740 ovn40subnets-nat/ovn60subnets-nat hash:net \u5f00\u542f NatOutgoing \u7684 Overlay \u5b50\u7f51\u7f51\u6bb5 ovn40subnets-distributed-gw/ovn60subnets-distributed-gw hash:net \u5f00\u542f\u5206\u5e03\u5f0f\u7f51\u5173\u7684 Overlay \u5b50\u7f51\u7f51\u6bb5 ovn40other-node/ovn60other-node hash:net \u5176\u5b83\u8282\u70b9\u7684\u5185\u90e8 IP \u5730\u5740 ovn40local-pod-ip-nat/ovn60local-pod-ip-nat hash:ip \u5df2\u5f03\u7528 ovn40subnets-nat-policy hash:net \u914d\u7f6e\u4e86 natOutgoingPolicyRules \u7684\u6240\u6709\u5b50\u7f51\u7f51\u6bb5 ovn40natpr-418e79269dc5-dst hash:net natOutgoingPolicyRules \u4e2d rule \u5bf9\u5e94\u7684 dstIPs ovn40natpr-418e79269dc5-src hash:net natOutgoingPolicyRules \u4e2d rule \u5bf9\u5e94\u7684 srcIPs \u4f7f\u7528\u7684 iptables \u89c4\u5219\uff08IPv4\uff09\u5982\u4e0b\u8868\u6240\u793a\uff1a \u8868 \u94fe \u89c4\u5219 \u7528\u9014 \u5907\u6ce8 filter INPUT -m set --match-set ovn40services src -j ACCEPT \u5141\u8bb8 k8s Service \u548c Pod \u76f8\u5173\u6d41\u91cf\u901a\u8fc7 -- filter INPUT -m set --match-set ovn40services dst -j ACCEPT \u540c\u4e0a -- filter INPUT -m set --match-set ovn40subnets src -j ACCEPT \u540c\u4e0a -- filter INPUT -m set --match-set ovn40subnets dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40services src -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40services dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40subnets src -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40subnets dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -s 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" \u7528\u4e8e\u8ba1\u6570\u4ece subnet \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u7684\u62a5\u6587 10.16.0.0/16 \u4e3a subnet \u7684 cidr \uff0ccomment \u4e2d\u9017\u53f7\u524d\u9762\u7684 ovn-subnet-gateway \u7528\u4e8e\u6807\u8bc6\u8be5 iptables \u89c4\u5219\u7528\u4e8e subnet \u51fa\u5165\u7f51\u5173\u62a5\u6587\u8ba1\u6570\uff0c\u9017\u53f7\u540e\u9762 ovn-default \u662f\u8be5 subnet \u7684\u540d\u5b57 filter FORWARD -d 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" \u7528\u4e8e\u8ba1\u6570\u4ece\u5916\u90e8\u7f51\u7edc\u8bbf\u95ee subnet \u7684\u62a5\u6587 \u540c\u4e0a filter OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0 \u6e05\u9664\u6d41\u91cf\u6807\u8bb0\uff0c\u907f\u514d\u6267\u884c SNAT UDP: bad checksum on VXLAN interface nat PREROUTING -m comment --comment \"kube-ovn prerouting rules\" -j OVN-PREROUTING \u8fdb\u5165 OVN-PREROUTING \u94fe\u5904\u7406 -- nat POSTROUTING -m comment --comment \"kube-ovn postrouting rules\" -j OVN-POSTROUTING \u8fdb\u5165 OVN-POSTROUTING \u94fe\u5904\u7406 -- nat OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set ovn40services dst -j MARK --set-xmark 0x4000/0x4000 \u4e3a Pod \u8bbf\u95ee Service \u6d41\u91cf\u6dfb\u52a0 masquerade \u6807\u8bb0 \u4f5c\u7528\u4e8e\u5173\u95ed\u5185\u7f6e LB \u7684\u573a\u666f nat OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000 \u4e3a ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff08TCP\uff09\u6dfb\u52a0\u7279\u5b9a\u6807\u8bb0 \u4ec5 kube-proxy \u4f7f\u7528 ipvs \u6a21\u5f0f\u65f6\u5b58\u5728 nat OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000 \u4e3a ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff08UDP\uff09\u6dfb\u52a0\u7279\u5b9a\u6807\u8bb0 \u540c\u4e0a nat OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source \u5f53\u8282\u70b9\u901a\u8fc7 Service IP \u8bbf\u95ee Overlay Pod \u65f6\uff0c\u4fdd\u6301\u6e90 IP \u4e3a\u8282\u70b9 IP\u3002 \u4ec5 kube-proxy \u4f7f\u7528 ipvs \u6a21\u5f0f\u65f6\u751f\u6548 nat OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE \u4e3a\u7279\u5b9a\u6807\u8bb0\u7684\u6d41\u91cf\u6267\u884c SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j MASQUERADE \u4e3a\u901a\u8fc7\u8282\u70b9\u7684 Pod \u4e4b\u95f4\u7684 Service \u6d41\u91cf\u6267\u884c SNAT -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN \u5bf9\u4e8e ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff0c\u82e5 Endpoint \u4f7f\u7528\u5206\u5e03\u5f0f\u7f51\u5173\uff0c\u65e0\u9700\u6267\u884c SNAT -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j MASQUERADE \u5bf9\u4e8e ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff0c\u82e5 Endpoint \u4f7f\u7528\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u6267\u884c SNAT -- nat OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN Pod IP \u5bf9\u5916\u66b4\u9732\u65f6\uff0c\u4e0d\u6267\u884c SNAT -- nat OVN-POSTROUTING -s 10.16.0.0/16 -m set ! --match-set ovn40subnets dst -j SNAT --to-source 192.168.0.101 Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f NatOutgoing \u4e14\u4f7f\u7528\u6307\u5b9a IP \u7684\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u6267\u884c SNAT 10.16.0.0/16 \u4e3a\u5b50\u7f51\u7f51\u6bb5\uff0c192.168.0.101 \u4e3a\u6307\u5b9a\u7684\u7f51\u5173\u8282\u70b9 IP nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j MASQUERADE Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f NatOutgoing\uff0c\u6267\u884c SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f natOutgoingPolicyRules\uff0c\u6307\u5b9a\u7b56\u7565\u7684\u62a5\u6587\u6267\u884c SNAT \u914d\u7f6e\u4e86 natOutgoingPolicyRules \u5b50\u7f51\u7684\u51fa\u5916\u7f51\u62a5\u6587\u7684\u8fdb\u5165\u94fe OVN-NAT-POLICY nat OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j MASQUERADE --random-fully \u540c\u4e0a \u4ece OVN-NAT-POLICY \u51fa\u6765\u540e\uff0c\u5982\u679c\u88ab\u6253\u4e0a tag 0x90001/0x90001 \u5c31\u4f1a\u505a SNAT nat OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN \u540c\u4e0a \u4ece OVN-NAT-POLICY \u51fa\u6765\u540e, \u5982\u679c\u88ab\u6253\u4e0a tag 0x90002/0x90002 \u4e0d\u4f1a\u505a SNAT nat OVN-NAT-POLICY -s 10.0.11.0/24 -m comment --comment natPolicySubnet-net1 -j OVN-NAT-PSUBNET-aa98851157c5 \u540c\u4e0a 10.0.11.0/24 \u8868\u793a\u5b50\u7f51 net1 \u7684 CIDR\uff0c OVN-NAT-PSUBNET-aa98851157c5 \u8fd9\u6761\u94fe\u4e0b\u7684\u89c4\u5219\u5c31\u5bf9\u5e94\u8fd9\u4e2a\u5b50\u7f51\u7684 natOutgoingPolicyRules \u914d\u7f6e nat OVN-NAT-PSUBNET-xxxxxxxxxxxx -m set --match-set ovn40natpr-418e79269dc5-src src -m set --match-set ovn40natpr-418e79269dc5-dst dst -j MARK --set-xmark 0x90002/0x90002 \u540c\u4e0a 418e79269dc5 \u8868\u793a natOutgoingPolicyRules \u4e2d\u7684\u4e00\u6761\u89c4\u5219\u7684 ID\uff0c\u53ef\u4ee5\u901a\u8fc7 status.natOutgoingPolicyRules[index].RuleID \u67e5\u770b\u5230\uff0c \u8868\u793a srcIPs \u6ee1\u8db3 ovn40natpr-418e79269dc5-src\uff0c dstIPS \u6ee1\u8db3 ovn40natpr-418e79269dc5-dst \u4f1a\u6253\u4e0a tag 0x90002 mangle OVN-OUTPUT -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003 \u5c06 kubelet \u7684\u63a2\u6d4b\u6d41\u91cf\u52a0\u4e0a\u7279\u5b9a\u6807\u8bb0\u4ece\u800c\u5f15\u5165\u5230 tproxy mangle OVN-PREROUTING -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.3 --tproxy-mark 0x90004/0x90004 \u5c06 kubelet \u7684\u63a2\u6d4b\u6d41\u91cf\u52a0\u4e0a\u7279\u5b9a\u6807\u8bb0\u4ece\u800c\u5f15\u5165\u5230 tproxy \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Iptables \u89c4\u5219"},{"location":"reference/iptables-rules/#iptables","text":"Kube-OVN \u4f7f\u7528 ipset \u53ca iptables \u8f85\u52a9\u5b9e\u73b0\u9ed8\u8ba4 VPC \u4e0b\u5bb9\u5668\u7f51\u7edc\uff08Overlay\uff09\u7f51\u5173 NAT \u7684\u529f\u80fd\u3002 \u4f7f\u7528\u7684 ipset \u5982\u4e0b\u8868\u6240\u793a\uff1a \u540d\u79f0\uff08IPv4/IPv6\uff09 \u7c7b\u578b \u5b58\u50a8\u5bf9\u8c61 ovn40services/ovn60services hash:net Service \u7f51\u6bb5 ovn40subnets/ovn60subnets hash:net Overlay \u5b50\u7f51\u7f51\u6bb5\u4ee5\u53ca NodeLocal DNS IP \u5730\u5740 ovn40subnets-nat/ovn60subnets-nat hash:net \u5f00\u542f NatOutgoing \u7684 Overlay \u5b50\u7f51\u7f51\u6bb5 ovn40subnets-distributed-gw/ovn60subnets-distributed-gw hash:net \u5f00\u542f\u5206\u5e03\u5f0f\u7f51\u5173\u7684 Overlay \u5b50\u7f51\u7f51\u6bb5 ovn40other-node/ovn60other-node hash:net \u5176\u5b83\u8282\u70b9\u7684\u5185\u90e8 IP \u5730\u5740 ovn40local-pod-ip-nat/ovn60local-pod-ip-nat hash:ip \u5df2\u5f03\u7528 ovn40subnets-nat-policy hash:net \u914d\u7f6e\u4e86 natOutgoingPolicyRules \u7684\u6240\u6709\u5b50\u7f51\u7f51\u6bb5 ovn40natpr-418e79269dc5-dst hash:net natOutgoingPolicyRules \u4e2d rule \u5bf9\u5e94\u7684 dstIPs ovn40natpr-418e79269dc5-src hash:net natOutgoingPolicyRules \u4e2d rule \u5bf9\u5e94\u7684 srcIPs \u4f7f\u7528\u7684 iptables \u89c4\u5219\uff08IPv4\uff09\u5982\u4e0b\u8868\u6240\u793a\uff1a \u8868 \u94fe \u89c4\u5219 \u7528\u9014 \u5907\u6ce8 filter INPUT -m set --match-set ovn40services src -j ACCEPT \u5141\u8bb8 k8s Service \u548c Pod \u76f8\u5173\u6d41\u91cf\u901a\u8fc7 -- filter INPUT -m set --match-set ovn40services dst -j ACCEPT \u540c\u4e0a -- filter INPUT -m set --match-set ovn40subnets src -j ACCEPT \u540c\u4e0a -- filter INPUT -m set --match-set ovn40subnets dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40services src -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40services dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40subnets src -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40subnets dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -s 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" \u7528\u4e8e\u8ba1\u6570\u4ece subnet \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u7684\u62a5\u6587 10.16.0.0/16 \u4e3a subnet \u7684 cidr \uff0ccomment \u4e2d\u9017\u53f7\u524d\u9762\u7684 ovn-subnet-gateway \u7528\u4e8e\u6807\u8bc6\u8be5 iptables \u89c4\u5219\u7528\u4e8e subnet \u51fa\u5165\u7f51\u5173\u62a5\u6587\u8ba1\u6570\uff0c\u9017\u53f7\u540e\u9762 ovn-default \u662f\u8be5 subnet \u7684\u540d\u5b57 filter FORWARD -d 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" \u7528\u4e8e\u8ba1\u6570\u4ece\u5916\u90e8\u7f51\u7edc\u8bbf\u95ee subnet \u7684\u62a5\u6587 \u540c\u4e0a filter OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0 \u6e05\u9664\u6d41\u91cf\u6807\u8bb0\uff0c\u907f\u514d\u6267\u884c SNAT UDP: bad checksum on VXLAN interface nat PREROUTING -m comment --comment \"kube-ovn prerouting rules\" -j OVN-PREROUTING \u8fdb\u5165 OVN-PREROUTING \u94fe\u5904\u7406 -- nat POSTROUTING -m comment --comment \"kube-ovn postrouting rules\" -j OVN-POSTROUTING \u8fdb\u5165 OVN-POSTROUTING \u94fe\u5904\u7406 -- nat OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set ovn40services dst -j MARK --set-xmark 0x4000/0x4000 \u4e3a Pod \u8bbf\u95ee Service \u6d41\u91cf\u6dfb\u52a0 masquerade \u6807\u8bb0 \u4f5c\u7528\u4e8e\u5173\u95ed\u5185\u7f6e LB \u7684\u573a\u666f nat OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000 \u4e3a ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff08TCP\uff09\u6dfb\u52a0\u7279\u5b9a\u6807\u8bb0 \u4ec5 kube-proxy \u4f7f\u7528 ipvs \u6a21\u5f0f\u65f6\u5b58\u5728 nat OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000 \u4e3a ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff08UDP\uff09\u6dfb\u52a0\u7279\u5b9a\u6807\u8bb0 \u540c\u4e0a nat OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source \u5f53\u8282\u70b9\u901a\u8fc7 Service IP \u8bbf\u95ee Overlay Pod \u65f6\uff0c\u4fdd\u6301\u6e90 IP \u4e3a\u8282\u70b9 IP\u3002 \u4ec5 kube-proxy \u4f7f\u7528 ipvs \u6a21\u5f0f\u65f6\u751f\u6548 nat OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE \u4e3a\u7279\u5b9a\u6807\u8bb0\u7684\u6d41\u91cf\u6267\u884c SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j MASQUERADE \u4e3a\u901a\u8fc7\u8282\u70b9\u7684 Pod \u4e4b\u95f4\u7684 Service \u6d41\u91cf\u6267\u884c SNAT -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN \u5bf9\u4e8e ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff0c\u82e5 Endpoint \u4f7f\u7528\u5206\u5e03\u5f0f\u7f51\u5173\uff0c\u65e0\u9700\u6267\u884c SNAT -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j MASQUERADE \u5bf9\u4e8e ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff0c\u82e5 Endpoint \u4f7f\u7528\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u6267\u884c SNAT -- nat OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN Pod IP \u5bf9\u5916\u66b4\u9732\u65f6\uff0c\u4e0d\u6267\u884c SNAT -- nat OVN-POSTROUTING -s 10.16.0.0/16 -m set ! --match-set ovn40subnets dst -j SNAT --to-source 192.168.0.101 Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f NatOutgoing \u4e14\u4f7f\u7528\u6307\u5b9a IP \u7684\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u6267\u884c SNAT 10.16.0.0/16 \u4e3a\u5b50\u7f51\u7f51\u6bb5\uff0c192.168.0.101 \u4e3a\u6307\u5b9a\u7684\u7f51\u5173\u8282\u70b9 IP nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j MASQUERADE Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f NatOutgoing\uff0c\u6267\u884c SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f natOutgoingPolicyRules\uff0c\u6307\u5b9a\u7b56\u7565\u7684\u62a5\u6587\u6267\u884c SNAT \u914d\u7f6e\u4e86 natOutgoingPolicyRules \u5b50\u7f51\u7684\u51fa\u5916\u7f51\u62a5\u6587\u7684\u8fdb\u5165\u94fe OVN-NAT-POLICY nat OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j MASQUERADE --random-fully \u540c\u4e0a \u4ece OVN-NAT-POLICY \u51fa\u6765\u540e\uff0c\u5982\u679c\u88ab\u6253\u4e0a tag 0x90001/0x90001 \u5c31\u4f1a\u505a SNAT nat OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN \u540c\u4e0a \u4ece OVN-NAT-POLICY \u51fa\u6765\u540e, \u5982\u679c\u88ab\u6253\u4e0a tag 0x90002/0x90002 \u4e0d\u4f1a\u505a SNAT nat OVN-NAT-POLICY -s 10.0.11.0/24 -m comment --comment natPolicySubnet-net1 -j OVN-NAT-PSUBNET-aa98851157c5 \u540c\u4e0a 10.0.11.0/24 \u8868\u793a\u5b50\u7f51 net1 \u7684 CIDR\uff0c OVN-NAT-PSUBNET-aa98851157c5 \u8fd9\u6761\u94fe\u4e0b\u7684\u89c4\u5219\u5c31\u5bf9\u5e94\u8fd9\u4e2a\u5b50\u7f51\u7684 natOutgoingPolicyRules \u914d\u7f6e nat OVN-NAT-PSUBNET-xxxxxxxxxxxx -m set --match-set ovn40natpr-418e79269dc5-src src -m set --match-set ovn40natpr-418e79269dc5-dst dst -j MARK --set-xmark 0x90002/0x90002 \u540c\u4e0a 418e79269dc5 \u8868\u793a natOutgoingPolicyRules \u4e2d\u7684\u4e00\u6761\u89c4\u5219\u7684 ID\uff0c\u53ef\u4ee5\u901a\u8fc7 status.natOutgoingPolicyRules[index].RuleID \u67e5\u770b\u5230\uff0c \u8868\u793a srcIPs \u6ee1\u8db3 ovn40natpr-418e79269dc5-src\uff0c dstIPS \u6ee1\u8db3 ovn40natpr-418e79269dc5-dst \u4f1a\u6253\u4e0a tag 0x90002 mangle OVN-OUTPUT -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003 \u5c06 kubelet \u7684\u63a2\u6d4b\u6d41\u91cf\u52a0\u4e0a\u7279\u5b9a\u6807\u8bb0\u4ece\u800c\u5f15\u5165\u5230 tproxy mangle OVN-PREROUTING -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.3 --tproxy-mark 0x90004/0x90004 \u5c06 kubelet \u7684\u63a2\u6d4b\u6d41\u91cf\u52a0\u4e0a\u7279\u5b9a\u6807\u8bb0\u4ece\u800c\u5f15\u5165\u5230 tproxy \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Iptables \u89c4\u5219"},{"location":"reference/kube-ovn-api/","text":"Kube-OVN \u63a5\u53e3\u89c4\u8303 \u00b6 \u57fa\u4e8e Kube-OVN v1.12.0 \u7248\u672c\uff0c\u6574\u7406\u4e86 Kube-OVN \u652f\u6301\u7684 CRD \u8d44\u6e90\u5217\u8868\uff0c\u5217\u51fa CRD \u5b9a\u4e49\u5404\u5b57\u6bb5\u7684\u53d6\u503c\u7c7b\u578b\u548c\u542b\u4e49\uff0c\u4ee5\u4f9b\u53c2\u8003\u3002 \u901a\u7528\u7684 Condition \u5b9a\u4e49 \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 type String \u72b6\u6001\u7c7b\u578b status String \u72b6\u6001\u503c\uff0c\u53d6\u503c\u4e3a True \uff0c False \u6216 Unknown reason String \u72b6\u6001\u53d8\u5316\u7684\u539f\u56e0 message String \u72b6\u6001\u53d8\u5316\u7684\u5177\u4f53\u4fe1\u606f lastUpdateTime Time \u4e0a\u6b21\u72b6\u6001\u66f4\u65b0\u65f6\u95f4 lastTransitionTime Time \u4e0a\u6b21\u72b6\u6001\u7c7b\u578b\u53d1\u751f\u53d8\u5316\u7684\u65f6\u95f4 \u5728\u5404 CRD \u7684\u5b9a\u4e49\u4e2d\uff0cStatus \u4e2d\u7684 Condition \u5b57\u6bb5\uff0c\u90fd\u9075\u5faa\u4e0a\u8ff0\u683c\u5f0f\uff0c\u56e0\u6b64\u63d0\u524d\u8fdb\u884c\u8bf4\u660e\u3002 Subnet \u5b9a\u4e49 \u00b6 Subnet \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Subnet metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SubnetSpec Subnet \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SubnetStatus Subnet \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5 SubnetSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 default Bool \u8be5\u5b50\u7f51\u662f\u5426\u4e3a\u9ed8\u8ba4\u5b50\u7f51 vpc String \u5b50\u7f51\u6240\u5c5e Vpc\uff0c\u9ed8\u8ba4\u4e3a ovn-cluster protocol String IP \u534f\u8bae\uff0c\u53d6\u503c\u53ef\u4ee5\u4e3a IPv4 \uff0c IPv6 \u6216 Dual namespaces []String \u8be5\u5b50\u7f51\u6240\u7ed1\u5b9a\u7684 namespace \u5217\u8868 cidrBlock String \u5b50\u7f51\u7684\u7f51\u6bb5\u8303\u56f4\uff0c\u5982 10.16.0.0/16 gateway String \u5b50\u7f51\u7f51\u5173\u5730\u5740\uff0c\u9ed8\u8ba4\u4e3a\u8be5\u5b50\u7f51 CIDRBlock \u4e0b\u7684\u7b2c\u4e00\u4e2a\u53ef\u7528\u5730\u5740 excludeIps []String \u8be5\u5b50\u7f51\u4e0b\u4e0d\u4f1a\u88ab\u81ea\u52a8\u5206\u914d\u7684\u5730\u5740\u8303\u56f4 provider String \u9ed8\u8ba4\u4e3a ovn\u3002\u591a\u7f51\u5361\u60c5\u51b5\u4e0b\u53ef\u4ee5\u914d\u7f6e\u53d6\u503c\u4e3a NetworkAttachmentDefinition \u7684 . \uff0cKube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90 gatewayType String Overlay \u6a21\u5f0f\u4e0b\u7684\u7f51\u5173\u7c7b\u578b\uff0c\u53d6\u503c\u53ef\u4ee5\u4e3a distributed \u6216 centralized gatewayNode String \u5f53\u7f51\u5173\u6a21\u5f0f\u4e3a centralized \u65f6\u7684\u7f51\u5173\u8282\u70b9\uff0c\u53ef\u4ee5\u4e3a\u9017\u53f7\u5206\u9694\u7684\u591a\u4e2a\u8282\u70b9 natOutgoing Bool \u51fa\u7f51\u6d41\u91cf\u662f\u5426\u8fdb\u884c NAT\u3002\u8be5\u53c2\u6570\u548c externalEgressGateway \u53c2\u6570\u4e0d\u80fd\u540c\u65f6\u8bbe\u7f6e\u3002 externalEgressGateway String \u5916\u90e8\u7f51\u5173\u5730\u5740\u3002\u9700\u8981\u548c\u5b50\u7f51\u7f51\u5173\u8282\u70b9\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u53ef\u8fbe\u57df\uff0c\u8be5\u53c2\u6570\u548c natOutgoing \u53c2\u6570\u4e0d\u80fd\u540c\u65f6\u8bbe\u7f6e policyRoutingPriority Uint32 \u7b56\u7565\u8def\u7531\u4f18\u5148\u7ea7\u3002\u6dfb\u52a0\u7b56\u7565\u8def\u7531\u4f7f\u7528\u53c2\u6570\uff0c\u63a7\u5236\u6d41\u91cf\u7ecf\u5b50\u7f51\u7f51\u5173\u4e4b\u540e\uff0c\u8f6c\u53d1\u5230\u5916\u90e8\u7f51\u5173\u5730\u5740 policyRoutingTableID Uint32 \u4f7f\u7528\u7684\u672c\u5730\u7b56\u7565\u8def\u7531\u8868\u7684 TableID\uff0c\u6bcf\u4e2a\u5b50\u7f51\u5747\u9700\u4e0d\u540c\u4ee5\u907f\u514d\u51b2\u7a81 private Bool \u6807\u8bc6\u8be5\u5b50\u7f51\u662f\u5426\u4e3a\u79c1\u6709\u5b50\u7f51\uff0c\u79c1\u6709\u5b50\u7f51\u9ed8\u8ba4\u62d2\u7edd\u5b50\u7f51\u5916\u7684\u5730\u5740\u8bbf\u95ee allowSubnets []String \u5b50\u7f51\u4e3a\u79c1\u6709\u5b50\u7f51\u7684\u60c5\u51b5\u4e0b\uff0c\u5141\u8bb8\u8bbf\u95ee\u8be5\u5b50\u7f51\u5730\u5740\u7684\u96c6\u5408 vlan String \u5b50\u7f51\u7ed1\u5b9a\u7684 Vlan \u540d\u79f0 vips []String \u5b50\u7f51\u4e0b virtual \u7c7b\u578b lsp \u7684 virtual-ip \u53c2\u6570\u4fe1\u606f logicalGateway Bool \u662f\u5426\u542f\u7528\u903b\u8f91\u7f51\u5173 disableGatewayCheck Bool \u521b\u5efa Pod \u65f6\u662f\u5426\u8df3\u8fc7\u7f51\u5173\u8054\u901a\u6027\u68c0\u67e5 disableInterConnection Bool \u63a7\u5236\u662f\u5426\u5f00\u542f\u5b50\u7f51\u8de8\u96c6\u7fa4\u4e92\u8054 enableDHCP Bool \u63a7\u5236\u662f\u5426\u914d\u7f6e\u5b50\u7f51\u4e0b lsp \u7684 dhcp \u914d\u7f6e\u9009\u9879 dhcpV4Options String \u5b50\u7f51\u4e0b lsp dhcpv4_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55 dhcpV6Options String \u5b50\u7f51\u4e0b lsp dhcpv6_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55 enableIPv6RA Bool \u63a7\u5236\u5b50\u7f51\u8fde\u63a5\u8def\u7531\u5668\u7684 lrp \u7aef\u53e3\uff0c\u662f\u5426\u914d\u7f6e ipv6_ra_configs \u53c2\u6570 ipv6RAConfigs String \u5b50\u7f51\u8fde\u63a5\u8def\u7531\u5668\u7684 lrp \u7aef\u53e3\uff0cipv6_ra_configs \u53c2\u6570\u914d\u7f6e\u4fe1\u606f acls []Acl \u5b50\u7f51\u5bf9\u5e94 logical-switch \u5173\u8054\u7684 acls \u8bb0\u5f55 u2oInterconnection Bool \u662f\u5426\u5f00\u542f Overlay/Underlay \u7684\u4e92\u8054\u6a21\u5f0f enableLb *Bool \u63a7\u5236\u5b50\u7f51\u5bf9\u5e94\u7684 logical-switch \u662f\u5426\u5173\u8054 load-balancer \u8bb0\u5f55 enableEcmp Bool \u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u662f\u5426\u5f00\u542f ECMP \u8def\u7531 Acl \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 direction String Acl \u9650\u5236\u65b9\u5411\uff0c\u53d6\u503c\u4e3a from-lport \u6216\u8005 to-lport priority Int Acl \u4f18\u5148\u7ea7\uff0c\u53d6\u503c\u8303\u56f4 0 \u5230 32767 match String Acl \u89c4\u5219\u5339\u914d\u8868\u8fbe\u5f0f action String Acl \u89c4\u5219\u52a8\u4f5c\uff0c\u53d6\u503c\u4e3a allow-related , allow-stateless , allow , drop , reject \u5176\u4e2d\u4e00\u4e2a SubnetStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []SubnetCondition \u5b50\u7f51\u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 v4AvailableIPs Float64 \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv4 IP \u5730\u5740\u6570\u91cf v4availableIPrange String \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv4 \u5730\u5740\u8303\u56f4 v4UsingIPs Float64 \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv4 IP \u5730\u5740\u6570\u91cf v4usingIPrange String \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv4 \u5730\u5740\u8303\u56f4 v6AvailableIPs Float64 \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv6 IP \u5730\u5740\u6570\u91cf v6availableIPrange String \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv6 \u5730\u5740\u8303\u56f4 v6UsingIPs Float64 \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv6 IP \u5730\u5740\u6570\u91cf v6usingIPrange String \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv6 \u5730\u5740\u8303\u56f4 sctivateGateway String \u96c6\u4e2d\u5f0f\u5b50\u7f51\uff0c\u4e3b\u5907\u6a21\u5f0f\u4e0b\u5f53\u524d\u6b63\u5728\u5de5\u4f5c\u7684\u7f51\u5173\u8282\u70b9 dhcpV4OptionsUUID String \u5b50\u7f51\u4e0b lsp dhcpv4_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55\u6807\u8bc6 dhcpV6OptionsUUID String \u5b50\u7f51\u4e0b lsp dhcpv6_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55\u6807\u8bc6 u2oInterconnectionIP String \u5f00\u542f Overlay/Underlay \u4e92\u8054\u6a21\u5f0f\u540e\uff0c\u6240\u5360\u7528\u7684\u7528\u4e8e\u4e92\u8054\u7684 IP \u5730\u5740 IP \u5b9a\u4e49 \u00b6 IP \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IP metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IPSpec IP \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 IPSepc \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 podName String \u7ed1\u5b9a Pod \u540d\u79f0 namespace String \u7ed1\u5b9a Pod \u6240\u5728 Namespace \u540d\u79f0 subnet String IP \u6240\u5c5e Subnet attachSubnets []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e\u5b50\u7f51\u540d\u79f0\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 nodeName String \u7ed1\u5b9a Pod \u6240\u5728\u7684\u8282\u70b9\u540d\u79f0 ipAddress String IP \u5730\u5740\uff0c\u53cc\u6808\u60c5\u51b5\u4e0b\u4e3a v4IP,v6IP \u683c\u5f0f v4IPAddress String IPv4 IP \u5730\u5740 v6IPAddress String IPv6 IP \u5730\u5740 attachIPs []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e IP \u5730\u5740\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 macAddress String \u7ed1\u5b9a Pod \u7684 Mac \u5730\u5740 attachMacs []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e Mac \u5730\u5740\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 containerID String \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 Container ID podType String \u7279\u6b8a\u5de5\u4f5c\u8d1f\u8f7d Pod\uff0c\u53ef\u4e3a StatefulSet \uff0c VirtualMachine \u6216\u7a7a Underlay \u914d\u7f6e \u00b6 Vlan \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vlan metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VlanSpec Vlan \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VlanStatus Vlan \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5 VlanSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 id Int Vlan tag \u53f7\uff0c\u53d6\u503c\u8303\u56f4\u4e3a 0~4096 provider String Vlan \u7ed1\u5b9a\u7684 ProviderNetwork \u540d\u79f0 VlanStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 subnets []String Vlan \u7ed1\u5b9a\u7684\u5b50\u7f51\u5217\u8868 conditions []VlanCondition Vlan \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ProviderNetwork \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a ProviderNetwork metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec ProviderNetworkSpec ProviderNetwork \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status ProviderNetworkStatus ProviderNetwork \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5 ProviderNetworkSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 defaultInterface String \u8be5\u6865\u63a5\u7f51\u7edc\u9ed8\u8ba4\u4f7f\u7528\u7684\u7f51\u5361\u63a5\u53e3\u540d\u79f0 customInterfaces []CustomInterface \u8be5\u6865\u63a5\u7f51\u7edc\u7279\u6b8a\u4f7f\u7528\u7684\u7f51\u5361\u914d\u7f6e excludeNodes []String \u8be5\u6865\u63a5\u7f51\u7edc\u4e0d\u4f1a\u7ed1\u5b9a\u7684\u8282\u70b9\u540d\u79f0 exchangeLinkName Bool \u662f\u5426\u4ea4\u6362\u6865\u63a5\u7f51\u5361\u548c\u5bf9\u5e94 OVS \u7f51\u6865\u540d\u79f0 CustomInterface \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 interface String Underlay \u4f7f\u7528\u7f51\u5361\u63a5\u53e3\u540d\u79f0 nodes []String \u4f7f\u7528\u81ea\u5b9a\u4e49\u7f51\u5361\u63a5\u53e3\u7684\u8282\u70b9\u5217\u8868 ProviderNetworkStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool \u5f53\u524d\u6865\u63a5\u7f51\u7edc\u662f\u5426\u8fdb\u5165\u5c31\u7eea\u72b6\u6001 readyNodes []String \u6865\u63a5\u7f51\u7edc\u8fdb\u5165\u5c31\u7eea\u72b6\u6001\u7684\u8282\u70b9\u540d\u79f0 notReadyNodes []String \u6865\u63a5\u7f51\u7edc\u672a\u8fdb\u5165\u5c31\u7eea\u72b6\u6001\u7684\u8282\u70b9\u540d\u79f0 vlans []String \u6865\u63a5\u7f51\u7edc\u7ed1\u5b9a\u7684 Vlan \u540d\u79f0 conditions []ProviderNetworkCondition ProviderNetwork \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 Vpc \u5b9a\u4e49 \u00b6 Vpc \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vpc metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcSpec Vpc \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VpcStatus Vpc \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5 VpcSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 namespaces []String Vpc \u7ed1\u5b9a\u7684\u547d\u540d\u7a7a\u95f4\u5217\u8868 staticRoutes []*StaticRoute Vpc \u4e0b\u914d\u7f6e\u7684\u9759\u6001\u8def\u7531\u4fe1\u606f policyRoutes []*PolicyRoute Vpc \u4e0b\u914d\u7f6e\u7684\u7b56\u7565\u8def\u7531\u4fe1\u606f vpcPeerings []*VpcPeering Vpc \u4e92\u8054\u4fe1\u606f enableExternal Bool Vpc \u662f\u5426\u8fde\u63a5\u5230\u5916\u90e8\u4ea4\u6362\u673a StaticRoute \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 policy String \u8def\u7531\u7b56\u7565\uff0c\u53d6\u503c\u4e3a policySrc \u6216\u8005 policyDst cidr String \u8def\u7531 Cidr \u7f51\u6bb5 nextHopIP String \u8def\u7531\u4e0b\u4e00\u8df3\u4fe1\u606f PolicyRoute \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 priority Int32 \u7b56\u7565\u8def\u7531\u4f18\u5148\u7ea7 match String \u7b56\u7565\u8def\u7531\u5339\u914d\u6761\u4ef6 action String \u7b56\u7565\u8def\u7531\u52a8\u4f5c\uff0c\u53d6\u503c\u4e3a allow \u3001 drop \u6216\u8005 reroute nextHopIP String \u7b56\u7565\u8def\u7531\u4e0b\u4e00\u8df3\u4fe1\u606f\uff0cECMP \u8def\u7531\u60c5\u51b5\u4e0b\u4e0b\u4e00\u8df3\u5730\u5740\u4f7f\u7528\u9017\u53f7\u9694\u5f00 VpcPeering \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 remoteVpc String Vpc \u4e92\u8054\u5bf9\u7aef Vpc \u540d\u79f0 localConnectIP String Vpc \u4e92\u8054\u672c\u7aef IP \u5730\u5740 VpcStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VpcCondition Vpc \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 standby Bool \u6807\u8bc6 Vpc \u662f\u5426\u521b\u5efa\u5b8c\u6210\uff0cVpc \u4e0b\u7684 Subnet \u9700\u8981\u7b49 Vpc \u521b\u5efa\u5b8c\u6210\u8f6c\u6362\u518d\u7ee7\u7eed\u5904\u7406 default Bool \u662f\u5426\u662f\u9ed8\u8ba4 Vpc defaultLogicalSwitch String Vpc \u4e0b\u7684\u9ed8\u8ba4\u5b50\u7f51 router String Vpc \u5bf9\u5e94\u7684 logical-router \u540d\u79f0 tcpLoadBalancer String Vpc \u4e0b\u7684 TCP LB \u4fe1\u606f udpLoadBalancer String Vpc \u4e0b\u7684 UDP LB \u4fe1\u606f tcpSessionLoadBalancer String Vpc \u4e0b\u7684 TCP \u4f1a\u8bdd\u4fdd\u6301 LB \u4fe1\u606f udpSessionLoadBalancer String Vpc \u4e0b\u7684 UDP \u4f1a\u8bdd\u4fdd\u6301 LB \u4fe1\u606f subnets []String Vpc \u4e0b\u7684\u5b50\u7f51\u5217\u8868 vpcPeerings []String Vpc \u4e92\u8054\u7684\u5bf9\u7aef Vpc \u5217\u8868 enableExternal Bool Vpc \u662f\u5426\u8fde\u63a5\u5230\u5916\u90e8\u4ea4\u6362\u673a VpcNatGateway \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a VpcNatGateway metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcNatSpec Vpc \u7f51\u5173\u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 VpcNatSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vpc String Vpc \u7f51\u5173 Pod \u6240\u5728\u7684 Vpc \u540d\u79f0 subnet String Vpc \u7f51\u5173 Pod \u6240\u5c5e\u7684\u5b50\u7f51\u540d\u79f0 lanIp String Vpc \u7f51\u5173 Pod \u6307\u5b9a\u5206\u914d\u7684 IP \u5730\u5740 selector []String \u6807\u51c6 Kubernetes Selector \u5339\u914d\u4fe1\u606f tolerations []VpcNatToleration \u6807\u51c6 Kubernetes \u5bb9\u5fcd\u4fe1\u606f VpcNatToleration \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 key String \u5bb9\u5fcd\u6c61\u70b9\u7684 key \u4fe1\u606f operator String \u53d6\u503c\u4e3a Exists \u6216\u8005 Equal value String \u5bb9\u5fcd\u6c61\u70b9\u7684 value \u4fe1\u606f effect String \u5bb9\u5fcd\u6c61\u70b9\u7684\u4f5c\u7528\u6548\u679c\uff0c\u53d6\u503c\u4e3a NoExecute \u3001 NoSchedule \u6216\u8005 PreferNoSchedule tolerationSeconds Int64 \u6dfb\u52a0\u6c61\u70b9\u540e\uff0cPod \u8fd8\u80fd\u7ee7\u7eed\u5728\u8282\u70b9\u4e0a\u8fd0\u884c\u7684\u65f6\u95f4 \u4ee5\u4e0a\u5bb9\u5fcd\u5b57\u6bb5\u7684\u542b\u4e49\uff0c\u53ef\u4ee5\u53c2\u8003 Kubernetes \u5b98\u65b9\u6587\u6863 \u6c61\u70b9\u548c\u5bb9\u5fcd\u5ea6 \u3002 IptablesEIP \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesEIP metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesEipSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesEIP \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesEipStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesEIP \u72b6\u6001\u4fe1\u606f IptablesEipSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 v4ip String IptablesEIP v4 \u5730\u5740 v6ip String IptablesEIP v6 \u5730\u5740 macAddress String IptablesEIP crd \u8bb0\u5f55\u5206\u914d\u7684 mac \u5730\u5740\uff0c\u6ca1\u6709\u5b9e\u9645\u4f7f\u7528 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 IptablesEipStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesEIP \u662f\u5426\u914d\u7f6e\u5b8c\u6210 ip String IptablesEIP \u4f7f\u7528\u7684 IP \u5730\u5740\uff0c\u76ee\u524d\u53ea\u652f\u6301\u4e86 IPv4 \u5730\u5740 redo String IptablesEIP crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 nat String IptablesEIP \u7684\u4f7f\u7528\u7c7b\u578b\uff0c\u53d6\u503c\u4e3a fip \u3001 snat \u6216\u8005 dnat conditions []IptablesEIPCondition IptablesEIP \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 IptablesFIPRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesFIPRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesFIPRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesFIPRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesFIPRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesFIPRule \u72b6\u6001\u4fe1\u606f IptablesFIPRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip String IptablesFIPRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 internalIp String IptablesFIPRule \u5bf9\u5e94\u7684\u5185\u90e8\u7684 IP \u5730\u5740 IptablesFIPRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesFIPRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesEIP \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesEIP \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesFIPRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesFIPRuleCondition IptablesFIPRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 IptablesSnatRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesSnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesSnatRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesSnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesSnatRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesSnatRule \u72b6\u6001\u4fe1\u606f IptablesSnatRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip String IptablesSnatRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 internalIp String IptablesSnatRule \u5bf9\u5e94\u7684\u5185\u90e8\u7684 IP \u5730\u5740 IptablesSnatRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesSnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesSnatRule \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesSnatRule \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesSnatRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesSnatRuleCondition IptablesSnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 IptablesDnatRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesDnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesDnatRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesDnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesDnatRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesDnatRule \u72b6\u6001\u4fe1\u606f IptablesDnatRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 externalPort Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5916\u90e8\u7aef\u53e3 protocol Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u7684\u534f\u8bae\u7c7b\u578b internalIp Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5185\u90e8 IP \u5730\u5740 internalPort Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5185\u90e8\u7aef\u53e3 IptablesDnatRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesDnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesDnatRule \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesDnatRule \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesDnatRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesDnatRuleCondition IptablesDnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 VpcDns \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a VpcDns metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcDnsSpec VpcDns \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VpcDnsStatus VpcDns \u72b6\u6001\u4fe1\u606f VpcDnsSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vpc String VpcDns \u6240\u5728\u7684 Vpc \u540d\u79f0 subnet String VpcDns Pod \u5206\u914d\u5730\u5740\u7684 Subnet \u540d\u79f0 VpcDnsStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VpcDnsCondition VpcDns \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 active Bool VpcDns \u662f\u5426\u6b63\u5728\u4f7f\u7528 VpcDns \u7684\u8be6\u7ec6\u4f7f\u7528\u6587\u6863\uff0c\u53ef\u4ee5\u53c2\u8003 \u81ea\u5b9a\u4e49 VPC DNS \u3002 SwitchLBRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a SwitchLBRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SwitchLBRuleSpec SwitchLBRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SwitchLBRuleStatus SwitchLBRule \u72b6\u6001\u4fe1\u606f SwitchLBRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vip String SwitchLBRule \u914d\u7f6e\u7684 vip \u5730\u5740 namespace String SwitchLBRule \u7684\u547d\u540d\u7a7a\u95f4 selector []String \u6807\u51c6 Kubernetes Selector \u5339\u914d\u4fe1\u606f sessionAffinity String \u6807\u51c6 Kubernetes Service \u4e2d sessionAffinity \u53d6\u503c ports []SlrPort SwitchLBRule \u7aef\u53e3\u5217\u8868 SwitchLBRule \u7684\u8be6\u7ec6\u914d\u7f6e\u4fe1\u606f\uff0c\u53ef\u4ee5\u53c2\u8003 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u3002 SlrPort \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 name String \u7aef\u53e3\u540d\u79f0 port Int32 \u7aef\u53e3\u53f7 targetPort Int32 \u76ee\u6807\u7aef\u53e3\u53f7 protocol String \u534f\u8bae\u7c7b\u578b SwitchLBRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []SwitchLBRuleCondition SwitchLBRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ports String SwitchLBRule \u7aef\u53e3\u4fe1\u606f service String SwitchLBRule \u63d0\u4f9b\u670d\u52a1\u7684 service \u540d\u79f0 \u5b89\u5168\u7ec4\u4e0e Vip \u00b6 SecurityGroup \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a SecurityGroup metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SecurityGroupSpec \u5b89\u5168\u7ec4\u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SecurityGroupStatus \u5b89\u5168\u7ec4\u72b6\u6001\u4fe1\u606f SecurityGroupSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ingressRules []*SgRule \u5165\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 egressRules []*SgRule \u51fa\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 allowSameGroupTraffic Bool \u540c\u4e00\u5b89\u5168\u7ec4\u5185\u7684 lsp \u662f\u5426\u53ef\u4ee5\u4e92\u901a\uff0c\u4ee5\u53ca\u6d41\u91cf\u89c4\u5219\u662f\u5426\u9700\u8981\u66f4\u65b0 SgRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ipVersion String IP \u7248\u672c\u53f7\uff0c\u53d6\u503c\u4e3a ipv4 \u6216\u8005 ipv6 protocol String \u53d6\u503c\u4e3a all \u3001 icmp \u3001 tcp \u6216\u8005 udp priority Int Acl \u4f18\u5148\u7ea7\uff0c\u53d6\u503c\u8303\u56f4\u4e3a 1-200\uff0c\u6570\u503c\u8d8a\u5c0f\uff0c\u4f18\u5148\u7ea7\u8d8a\u9ad8 remoteType String \u53d6\u503c\u4e3a address \u6216\u8005 securityGroup remoteAddress String \u5bf9\u7aef\u5730\u5740 remoteSecurityGroup String \u5bf9\u7aef\u5b89\u5168\u7ec4 portRangeMin Int \u7aef\u53e3\u8303\u56f4\u8d77\u59cb\u503c\uff0c\u6700\u5c0f\u53d6\u503c\u4e3a 1 portRangeMax Int \u7aef\u53e3\u8303\u56f4\u6700\u5927\u503c\uff0c\u6700\u5927\u53d6\u503c\u4e3a 65535 policy String \u53d6\u503c\u4e3a allow \u6216\u8005 drop SecurityGroupStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 portGroup String \u5b89\u5168\u7ec4\u5bf9\u5e94\u7684 port-group \u540d\u79f0 allowSameGroupTraffic Bool \u540c\u4e00\u5b89\u5168\u7ec4\u5185\u7684 lsp \u662f\u5426\u53ef\u4ee5\u4e92\u901a\uff0c\u4ee5\u53ca\u5b89\u5168\u7ec4\u7684\u6d41\u91cf\u89c4\u5219\u662f\u5426\u9700\u8981\u66f4\u65b0 ingressMd5 String \u5165\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 MD5 \u53d6\u503c egressMd5 String \u51fa\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 MD5 \u53d6\u503c ingressLastSyncSuccess Bool \u5165\u65b9\u5411\u89c4\u5219\u4e0a\u4e00\u6b21\u540c\u6b65\u662f\u5426\u6210\u529f egressLastSyncSuccess Bool \u51fa\u65b9\u5411\u89c4\u5219\u4e0a\u4e00\u6b21\u540c\u6b65\u662f\u5426\u6210\u529f Vip \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VipSpec Vip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VipStatus Vip \u72b6\u6001\u4fe1\u606f VipSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 namespace String Vip \u6240\u5728\u547d\u540d\u7a7a\u95f4 subnet String Vip \u6240\u5c5e\u5b50\u7f51 v4ip String Vip v4 IP \u5730\u5740 v6ip String Vip v6 IP \u5730\u5740 macAddress String Vip mac \u5730\u5740 parentV4ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 parentV6ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 parentMac String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 attachSubnets []String \u8be5\u5b57\u6bb5\u5e9f\u5f03\uff0c\u4e0d\u518d\u4f7f\u7528 VipStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VipCondition Vip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ready Bool Vip \u662f\u5426\u51c6\u5907\u597d v4ip String Vip v4 IP \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 v6ip String Vip v6 IP \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 mac String Vip mac \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 pv4ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 pv6ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 pmac String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 OvnEip \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnEip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnEipSpec \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnEip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnEipStatus \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnEip \u72b6\u6001\u4fe1\u606f OvnEipSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 externalSubnet String OvnEip \u6240\u5728\u7684\u5b50\u7f51\u540d\u79f0 v4ip String OvnEip IP \u5730\u5740 macAddress String OvnEip Mac \u5730\u5740 type String OvnEip \u4f7f\u7528\u7c7b\u578b\uff0c\u53d6\u503c\u6709 fip \u3001 snat \u6216\u8005 lrp OvnEipStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []OvnEipCondition \u9ed8\u8ba4 Vpc OvnEip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 v4ip String OvnEip \u4f7f\u7528\u7684 v4 IP \u5730\u5740 macAddress String OvnEip \u4f7f\u7528\u7684 Mac \u5730\u5740 OvnFip \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnFip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnFipSpec \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnFip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnFipStatus \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnFip \u72b6\u6001\u4fe1\u606f OvnFipSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ovnEip String OvnFip \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 ipName String OvnFip \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 IP crd \u540d\u79f0 OvnFipStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool OvnFip \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4Eip String OvnFip \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 v4Ip String OvnFip \u5f53\u524d\u4f7f\u7528\u7684 OvnEip \u5730\u5740 macAddress String OvnFip \u914d\u7f6e\u7684 Mac \u5730\u5740 vpc String OvnFip \u6240\u5728\u7684 Vpc \u540d\u79f0 conditions []OvnFipCondition OvnFip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 OvnSnatRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnSnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnSnatRuleSpec \u9ed8\u8ba4 Vpc OvnSnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnSnatRuleStatus \u9ed8\u8ba4 Vpc OvnSnatRule \u72b6\u6001\u4fe1\u606f OvnSnatRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ovnEip String OvnSnatRule \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 vpcSubnet String OvnSnatRule \u914d\u7f6e\u7684\u5b50\u7f51\u540d\u79f0 ipName String OvnSnatRule \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 IP crd \u540d\u79f0 OvnSnatRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool OvnSnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4Eip String OvnSnatRule \u7ed1\u5b9a\u7684 OvnEip \u5730\u5740 v4IpCidr String \u5728 logical-router \u4e2d\u914d\u7f6e snat \u8f6c\u6362\u4f7f\u7528\u7684 cidr \u5730\u5740 vpc String OvnSnatRule \u6240\u5728\u7684 Vpc \u540d\u79f0 conditions []OvnSnatRuleCondition OvnSnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN \u63a5\u53e3\u89c4\u8303"},{"location":"reference/kube-ovn-api/#kube-ovn","text":"\u57fa\u4e8e Kube-OVN v1.12.0 \u7248\u672c\uff0c\u6574\u7406\u4e86 Kube-OVN \u652f\u6301\u7684 CRD \u8d44\u6e90\u5217\u8868\uff0c\u5217\u51fa CRD \u5b9a\u4e49\u5404\u5b57\u6bb5\u7684\u53d6\u503c\u7c7b\u578b\u548c\u542b\u4e49\uff0c\u4ee5\u4f9b\u53c2\u8003\u3002","title":"Kube-OVN \u63a5\u53e3\u89c4\u8303"},{"location":"reference/kube-ovn-api/#condition","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 type String \u72b6\u6001\u7c7b\u578b status String \u72b6\u6001\u503c\uff0c\u53d6\u503c\u4e3a True \uff0c False \u6216 Unknown reason String \u72b6\u6001\u53d8\u5316\u7684\u539f\u56e0 message String \u72b6\u6001\u53d8\u5316\u7684\u5177\u4f53\u4fe1\u606f lastUpdateTime Time \u4e0a\u6b21\u72b6\u6001\u66f4\u65b0\u65f6\u95f4 lastTransitionTime Time \u4e0a\u6b21\u72b6\u6001\u7c7b\u578b\u53d1\u751f\u53d8\u5316\u7684\u65f6\u95f4 \u5728\u5404 CRD \u7684\u5b9a\u4e49\u4e2d\uff0cStatus \u4e2d\u7684 Condition \u5b57\u6bb5\uff0c\u90fd\u9075\u5faa\u4e0a\u8ff0\u683c\u5f0f\uff0c\u56e0\u6b64\u63d0\u524d\u8fdb\u884c\u8bf4\u660e\u3002","title":"\u901a\u7528\u7684 Condition \u5b9a\u4e49"},{"location":"reference/kube-ovn-api/#subnet","text":"","title":"Subnet \u5b9a\u4e49"},{"location":"reference/kube-ovn-api/#subnet_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Subnet metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SubnetSpec Subnet \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SubnetStatus Subnet \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5","title":"Subnet"},{"location":"reference/kube-ovn-api/#subnetspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 default Bool \u8be5\u5b50\u7f51\u662f\u5426\u4e3a\u9ed8\u8ba4\u5b50\u7f51 vpc String \u5b50\u7f51\u6240\u5c5e Vpc\uff0c\u9ed8\u8ba4\u4e3a ovn-cluster protocol String IP \u534f\u8bae\uff0c\u53d6\u503c\u53ef\u4ee5\u4e3a IPv4 \uff0c IPv6 \u6216 Dual namespaces []String \u8be5\u5b50\u7f51\u6240\u7ed1\u5b9a\u7684 namespace \u5217\u8868 cidrBlock String \u5b50\u7f51\u7684\u7f51\u6bb5\u8303\u56f4\uff0c\u5982 10.16.0.0/16 gateway String \u5b50\u7f51\u7f51\u5173\u5730\u5740\uff0c\u9ed8\u8ba4\u4e3a\u8be5\u5b50\u7f51 CIDRBlock \u4e0b\u7684\u7b2c\u4e00\u4e2a\u53ef\u7528\u5730\u5740 excludeIps []String \u8be5\u5b50\u7f51\u4e0b\u4e0d\u4f1a\u88ab\u81ea\u52a8\u5206\u914d\u7684\u5730\u5740\u8303\u56f4 provider String \u9ed8\u8ba4\u4e3a ovn\u3002\u591a\u7f51\u5361\u60c5\u51b5\u4e0b\u53ef\u4ee5\u914d\u7f6e\u53d6\u503c\u4e3a NetworkAttachmentDefinition \u7684 . \uff0cKube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90 gatewayType String Overlay \u6a21\u5f0f\u4e0b\u7684\u7f51\u5173\u7c7b\u578b\uff0c\u53d6\u503c\u53ef\u4ee5\u4e3a distributed \u6216 centralized gatewayNode String \u5f53\u7f51\u5173\u6a21\u5f0f\u4e3a centralized \u65f6\u7684\u7f51\u5173\u8282\u70b9\uff0c\u53ef\u4ee5\u4e3a\u9017\u53f7\u5206\u9694\u7684\u591a\u4e2a\u8282\u70b9 natOutgoing Bool \u51fa\u7f51\u6d41\u91cf\u662f\u5426\u8fdb\u884c NAT\u3002\u8be5\u53c2\u6570\u548c externalEgressGateway \u53c2\u6570\u4e0d\u80fd\u540c\u65f6\u8bbe\u7f6e\u3002 externalEgressGateway String \u5916\u90e8\u7f51\u5173\u5730\u5740\u3002\u9700\u8981\u548c\u5b50\u7f51\u7f51\u5173\u8282\u70b9\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u53ef\u8fbe\u57df\uff0c\u8be5\u53c2\u6570\u548c natOutgoing \u53c2\u6570\u4e0d\u80fd\u540c\u65f6\u8bbe\u7f6e policyRoutingPriority Uint32 \u7b56\u7565\u8def\u7531\u4f18\u5148\u7ea7\u3002\u6dfb\u52a0\u7b56\u7565\u8def\u7531\u4f7f\u7528\u53c2\u6570\uff0c\u63a7\u5236\u6d41\u91cf\u7ecf\u5b50\u7f51\u7f51\u5173\u4e4b\u540e\uff0c\u8f6c\u53d1\u5230\u5916\u90e8\u7f51\u5173\u5730\u5740 policyRoutingTableID Uint32 \u4f7f\u7528\u7684\u672c\u5730\u7b56\u7565\u8def\u7531\u8868\u7684 TableID\uff0c\u6bcf\u4e2a\u5b50\u7f51\u5747\u9700\u4e0d\u540c\u4ee5\u907f\u514d\u51b2\u7a81 private Bool \u6807\u8bc6\u8be5\u5b50\u7f51\u662f\u5426\u4e3a\u79c1\u6709\u5b50\u7f51\uff0c\u79c1\u6709\u5b50\u7f51\u9ed8\u8ba4\u62d2\u7edd\u5b50\u7f51\u5916\u7684\u5730\u5740\u8bbf\u95ee allowSubnets []String \u5b50\u7f51\u4e3a\u79c1\u6709\u5b50\u7f51\u7684\u60c5\u51b5\u4e0b\uff0c\u5141\u8bb8\u8bbf\u95ee\u8be5\u5b50\u7f51\u5730\u5740\u7684\u96c6\u5408 vlan String \u5b50\u7f51\u7ed1\u5b9a\u7684 Vlan \u540d\u79f0 vips []String \u5b50\u7f51\u4e0b virtual \u7c7b\u578b lsp \u7684 virtual-ip \u53c2\u6570\u4fe1\u606f logicalGateway Bool \u662f\u5426\u542f\u7528\u903b\u8f91\u7f51\u5173 disableGatewayCheck Bool \u521b\u5efa Pod \u65f6\u662f\u5426\u8df3\u8fc7\u7f51\u5173\u8054\u901a\u6027\u68c0\u67e5 disableInterConnection Bool \u63a7\u5236\u662f\u5426\u5f00\u542f\u5b50\u7f51\u8de8\u96c6\u7fa4\u4e92\u8054 enableDHCP Bool \u63a7\u5236\u662f\u5426\u914d\u7f6e\u5b50\u7f51\u4e0b lsp \u7684 dhcp \u914d\u7f6e\u9009\u9879 dhcpV4Options String \u5b50\u7f51\u4e0b lsp dhcpv4_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55 dhcpV6Options String \u5b50\u7f51\u4e0b lsp dhcpv6_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55 enableIPv6RA Bool \u63a7\u5236\u5b50\u7f51\u8fde\u63a5\u8def\u7531\u5668\u7684 lrp \u7aef\u53e3\uff0c\u662f\u5426\u914d\u7f6e ipv6_ra_configs \u53c2\u6570 ipv6RAConfigs String \u5b50\u7f51\u8fde\u63a5\u8def\u7531\u5668\u7684 lrp \u7aef\u53e3\uff0cipv6_ra_configs \u53c2\u6570\u914d\u7f6e\u4fe1\u606f acls []Acl \u5b50\u7f51\u5bf9\u5e94 logical-switch \u5173\u8054\u7684 acls \u8bb0\u5f55 u2oInterconnection Bool \u662f\u5426\u5f00\u542f Overlay/Underlay \u7684\u4e92\u8054\u6a21\u5f0f enableLb *Bool \u63a7\u5236\u5b50\u7f51\u5bf9\u5e94\u7684 logical-switch \u662f\u5426\u5173\u8054 load-balancer \u8bb0\u5f55 enableEcmp Bool \u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u662f\u5426\u5f00\u542f ECMP \u8def\u7531","title":"SubnetSpec"},{"location":"reference/kube-ovn-api/#acl","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 direction String Acl \u9650\u5236\u65b9\u5411\uff0c\u53d6\u503c\u4e3a from-lport \u6216\u8005 to-lport priority Int Acl \u4f18\u5148\u7ea7\uff0c\u53d6\u503c\u8303\u56f4 0 \u5230 32767 match String Acl \u89c4\u5219\u5339\u914d\u8868\u8fbe\u5f0f action String Acl \u89c4\u5219\u52a8\u4f5c\uff0c\u53d6\u503c\u4e3a allow-related , allow-stateless , allow , drop , reject \u5176\u4e2d\u4e00\u4e2a","title":"Acl"},{"location":"reference/kube-ovn-api/#subnetstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []SubnetCondition \u5b50\u7f51\u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 v4AvailableIPs Float64 \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv4 IP \u5730\u5740\u6570\u91cf v4availableIPrange String \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv4 \u5730\u5740\u8303\u56f4 v4UsingIPs Float64 \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv4 IP \u5730\u5740\u6570\u91cf v4usingIPrange String \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv4 \u5730\u5740\u8303\u56f4 v6AvailableIPs Float64 \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv6 IP \u5730\u5740\u6570\u91cf v6availableIPrange String \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv6 \u5730\u5740\u8303\u56f4 v6UsingIPs Float64 \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv6 IP \u5730\u5740\u6570\u91cf v6usingIPrange String \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv6 \u5730\u5740\u8303\u56f4 sctivateGateway String \u96c6\u4e2d\u5f0f\u5b50\u7f51\uff0c\u4e3b\u5907\u6a21\u5f0f\u4e0b\u5f53\u524d\u6b63\u5728\u5de5\u4f5c\u7684\u7f51\u5173\u8282\u70b9 dhcpV4OptionsUUID String \u5b50\u7f51\u4e0b lsp dhcpv4_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55\u6807\u8bc6 dhcpV6OptionsUUID String \u5b50\u7f51\u4e0b lsp dhcpv6_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55\u6807\u8bc6 u2oInterconnectionIP String \u5f00\u542f Overlay/Underlay \u4e92\u8054\u6a21\u5f0f\u540e\uff0c\u6240\u5360\u7528\u7684\u7528\u4e8e\u4e92\u8054\u7684 IP \u5730\u5740","title":"SubnetStatus"},{"location":"reference/kube-ovn-api/#ip","text":"","title":"IP \u5b9a\u4e49"},{"location":"reference/kube-ovn-api/#ip_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IP metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IPSpec IP \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5","title":"IP"},{"location":"reference/kube-ovn-api/#ipsepc","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 podName String \u7ed1\u5b9a Pod \u540d\u79f0 namespace String \u7ed1\u5b9a Pod \u6240\u5728 Namespace \u540d\u79f0 subnet String IP \u6240\u5c5e Subnet attachSubnets []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e\u5b50\u7f51\u540d\u79f0\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 nodeName String \u7ed1\u5b9a Pod \u6240\u5728\u7684\u8282\u70b9\u540d\u79f0 ipAddress String IP \u5730\u5740\uff0c\u53cc\u6808\u60c5\u51b5\u4e0b\u4e3a v4IP,v6IP \u683c\u5f0f v4IPAddress String IPv4 IP \u5730\u5740 v6IPAddress String IPv6 IP \u5730\u5740 attachIPs []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e IP \u5730\u5740\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 macAddress String \u7ed1\u5b9a Pod \u7684 Mac \u5730\u5740 attachMacs []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e Mac \u5730\u5740\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 containerID String \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 Container ID podType String \u7279\u6b8a\u5de5\u4f5c\u8d1f\u8f7d Pod\uff0c\u53ef\u4e3a StatefulSet \uff0c VirtualMachine \u6216\u7a7a","title":"IPSepc"},{"location":"reference/kube-ovn-api/#underlay","text":"","title":"Underlay \u914d\u7f6e"},{"location":"reference/kube-ovn-api/#vlan","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vlan metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VlanSpec Vlan \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VlanStatus Vlan \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5","title":"Vlan"},{"location":"reference/kube-ovn-api/#vlanspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 id Int Vlan tag \u53f7\uff0c\u53d6\u503c\u8303\u56f4\u4e3a 0~4096 provider String Vlan \u7ed1\u5b9a\u7684 ProviderNetwork \u540d\u79f0","title":"VlanSpec"},{"location":"reference/kube-ovn-api/#vlanstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 subnets []String Vlan \u7ed1\u5b9a\u7684\u5b50\u7f51\u5217\u8868 conditions []VlanCondition Vlan \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"VlanStatus"},{"location":"reference/kube-ovn-api/#providernetwork","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a ProviderNetwork metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec ProviderNetworkSpec ProviderNetwork \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status ProviderNetworkStatus ProviderNetwork \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5","title":"ProviderNetwork"},{"location":"reference/kube-ovn-api/#providernetworkspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 defaultInterface String \u8be5\u6865\u63a5\u7f51\u7edc\u9ed8\u8ba4\u4f7f\u7528\u7684\u7f51\u5361\u63a5\u53e3\u540d\u79f0 customInterfaces []CustomInterface \u8be5\u6865\u63a5\u7f51\u7edc\u7279\u6b8a\u4f7f\u7528\u7684\u7f51\u5361\u914d\u7f6e excludeNodes []String \u8be5\u6865\u63a5\u7f51\u7edc\u4e0d\u4f1a\u7ed1\u5b9a\u7684\u8282\u70b9\u540d\u79f0 exchangeLinkName Bool \u662f\u5426\u4ea4\u6362\u6865\u63a5\u7f51\u5361\u548c\u5bf9\u5e94 OVS \u7f51\u6865\u540d\u79f0","title":"ProviderNetworkSpec"},{"location":"reference/kube-ovn-api/#custominterface","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 interface String Underlay \u4f7f\u7528\u7f51\u5361\u63a5\u53e3\u540d\u79f0 nodes []String \u4f7f\u7528\u81ea\u5b9a\u4e49\u7f51\u5361\u63a5\u53e3\u7684\u8282\u70b9\u5217\u8868","title":"CustomInterface"},{"location":"reference/kube-ovn-api/#providernetworkstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool \u5f53\u524d\u6865\u63a5\u7f51\u7edc\u662f\u5426\u8fdb\u5165\u5c31\u7eea\u72b6\u6001 readyNodes []String \u6865\u63a5\u7f51\u7edc\u8fdb\u5165\u5c31\u7eea\u72b6\u6001\u7684\u8282\u70b9\u540d\u79f0 notReadyNodes []String \u6865\u63a5\u7f51\u7edc\u672a\u8fdb\u5165\u5c31\u7eea\u72b6\u6001\u7684\u8282\u70b9\u540d\u79f0 vlans []String \u6865\u63a5\u7f51\u7edc\u7ed1\u5b9a\u7684 Vlan \u540d\u79f0 conditions []ProviderNetworkCondition ProviderNetwork \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"ProviderNetworkStatus"},{"location":"reference/kube-ovn-api/#vpc","text":"","title":"Vpc \u5b9a\u4e49"},{"location":"reference/kube-ovn-api/#vpc_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vpc metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcSpec Vpc \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VpcStatus Vpc \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5","title":"Vpc"},{"location":"reference/kube-ovn-api/#vpcspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 namespaces []String Vpc \u7ed1\u5b9a\u7684\u547d\u540d\u7a7a\u95f4\u5217\u8868 staticRoutes []*StaticRoute Vpc \u4e0b\u914d\u7f6e\u7684\u9759\u6001\u8def\u7531\u4fe1\u606f policyRoutes []*PolicyRoute Vpc \u4e0b\u914d\u7f6e\u7684\u7b56\u7565\u8def\u7531\u4fe1\u606f vpcPeerings []*VpcPeering Vpc \u4e92\u8054\u4fe1\u606f enableExternal Bool Vpc \u662f\u5426\u8fde\u63a5\u5230\u5916\u90e8\u4ea4\u6362\u673a","title":"VpcSpec"},{"location":"reference/kube-ovn-api/#staticroute","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 policy String \u8def\u7531\u7b56\u7565\uff0c\u53d6\u503c\u4e3a policySrc \u6216\u8005 policyDst cidr String \u8def\u7531 Cidr \u7f51\u6bb5 nextHopIP String \u8def\u7531\u4e0b\u4e00\u8df3\u4fe1\u606f","title":"StaticRoute"},{"location":"reference/kube-ovn-api/#policyroute","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 priority Int32 \u7b56\u7565\u8def\u7531\u4f18\u5148\u7ea7 match String \u7b56\u7565\u8def\u7531\u5339\u914d\u6761\u4ef6 action String \u7b56\u7565\u8def\u7531\u52a8\u4f5c\uff0c\u53d6\u503c\u4e3a allow \u3001 drop \u6216\u8005 reroute nextHopIP String \u7b56\u7565\u8def\u7531\u4e0b\u4e00\u8df3\u4fe1\u606f\uff0cECMP \u8def\u7531\u60c5\u51b5\u4e0b\u4e0b\u4e00\u8df3\u5730\u5740\u4f7f\u7528\u9017\u53f7\u9694\u5f00","title":"PolicyRoute"},{"location":"reference/kube-ovn-api/#vpcpeering","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 remoteVpc String Vpc \u4e92\u8054\u5bf9\u7aef Vpc \u540d\u79f0 localConnectIP String Vpc \u4e92\u8054\u672c\u7aef IP \u5730\u5740","title":"VpcPeering"},{"location":"reference/kube-ovn-api/#vpcstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VpcCondition Vpc \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 standby Bool \u6807\u8bc6 Vpc \u662f\u5426\u521b\u5efa\u5b8c\u6210\uff0cVpc \u4e0b\u7684 Subnet \u9700\u8981\u7b49 Vpc \u521b\u5efa\u5b8c\u6210\u8f6c\u6362\u518d\u7ee7\u7eed\u5904\u7406 default Bool \u662f\u5426\u662f\u9ed8\u8ba4 Vpc defaultLogicalSwitch String Vpc \u4e0b\u7684\u9ed8\u8ba4\u5b50\u7f51 router String Vpc \u5bf9\u5e94\u7684 logical-router \u540d\u79f0 tcpLoadBalancer String Vpc \u4e0b\u7684 TCP LB \u4fe1\u606f udpLoadBalancer String Vpc \u4e0b\u7684 UDP LB \u4fe1\u606f tcpSessionLoadBalancer String Vpc \u4e0b\u7684 TCP \u4f1a\u8bdd\u4fdd\u6301 LB \u4fe1\u606f udpSessionLoadBalancer String Vpc \u4e0b\u7684 UDP \u4f1a\u8bdd\u4fdd\u6301 LB \u4fe1\u606f subnets []String Vpc \u4e0b\u7684\u5b50\u7f51\u5217\u8868 vpcPeerings []String Vpc \u4e92\u8054\u7684\u5bf9\u7aef Vpc \u5217\u8868 enableExternal Bool Vpc \u662f\u5426\u8fde\u63a5\u5230\u5916\u90e8\u4ea4\u6362\u673a","title":"VpcStatus"},{"location":"reference/kube-ovn-api/#vpcnatgateway","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a VpcNatGateway metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcNatSpec Vpc \u7f51\u5173\u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5","title":"VpcNatGateway"},{"location":"reference/kube-ovn-api/#vpcnatspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vpc String Vpc \u7f51\u5173 Pod \u6240\u5728\u7684 Vpc \u540d\u79f0 subnet String Vpc \u7f51\u5173 Pod \u6240\u5c5e\u7684\u5b50\u7f51\u540d\u79f0 lanIp String Vpc \u7f51\u5173 Pod \u6307\u5b9a\u5206\u914d\u7684 IP \u5730\u5740 selector []String \u6807\u51c6 Kubernetes Selector \u5339\u914d\u4fe1\u606f tolerations []VpcNatToleration \u6807\u51c6 Kubernetes \u5bb9\u5fcd\u4fe1\u606f","title":"VpcNatSpec"},{"location":"reference/kube-ovn-api/#vpcnattoleration","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 key String \u5bb9\u5fcd\u6c61\u70b9\u7684 key \u4fe1\u606f operator String \u53d6\u503c\u4e3a Exists \u6216\u8005 Equal value String \u5bb9\u5fcd\u6c61\u70b9\u7684 value \u4fe1\u606f effect String \u5bb9\u5fcd\u6c61\u70b9\u7684\u4f5c\u7528\u6548\u679c\uff0c\u53d6\u503c\u4e3a NoExecute \u3001 NoSchedule \u6216\u8005 PreferNoSchedule tolerationSeconds Int64 \u6dfb\u52a0\u6c61\u70b9\u540e\uff0cPod \u8fd8\u80fd\u7ee7\u7eed\u5728\u8282\u70b9\u4e0a\u8fd0\u884c\u7684\u65f6\u95f4 \u4ee5\u4e0a\u5bb9\u5fcd\u5b57\u6bb5\u7684\u542b\u4e49\uff0c\u53ef\u4ee5\u53c2\u8003 Kubernetes \u5b98\u65b9\u6587\u6863 \u6c61\u70b9\u548c\u5bb9\u5fcd\u5ea6 \u3002","title":"VpcNatToleration"},{"location":"reference/kube-ovn-api/#iptableseip","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesEIP metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesEipSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesEIP \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesEipStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesEIP \u72b6\u6001\u4fe1\u606f","title":"IptablesEIP"},{"location":"reference/kube-ovn-api/#iptableseipspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 v4ip String IptablesEIP v4 \u5730\u5740 v6ip String IptablesEIP v6 \u5730\u5740 macAddress String IptablesEIP crd \u8bb0\u5f55\u5206\u914d\u7684 mac \u5730\u5740\uff0c\u6ca1\u6709\u5b9e\u9645\u4f7f\u7528 natGwDp String Vpc \u7f51\u5173\u540d\u79f0","title":"IptablesEipSpec"},{"location":"reference/kube-ovn-api/#iptableseipstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesEIP \u662f\u5426\u914d\u7f6e\u5b8c\u6210 ip String IptablesEIP \u4f7f\u7528\u7684 IP \u5730\u5740\uff0c\u76ee\u524d\u53ea\u652f\u6301\u4e86 IPv4 \u5730\u5740 redo String IptablesEIP crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 nat String IptablesEIP \u7684\u4f7f\u7528\u7c7b\u578b\uff0c\u53d6\u503c\u4e3a fip \u3001 snat \u6216\u8005 dnat conditions []IptablesEIPCondition IptablesEIP \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"IptablesEipStatus"},{"location":"reference/kube-ovn-api/#iptablesfiprule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesFIPRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesFIPRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesFIPRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesFIPRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesFIPRule \u72b6\u6001\u4fe1\u606f","title":"IptablesFIPRule"},{"location":"reference/kube-ovn-api/#iptablesfiprulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip String IptablesFIPRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 internalIp String IptablesFIPRule \u5bf9\u5e94\u7684\u5185\u90e8\u7684 IP \u5730\u5740","title":"IptablesFIPRuleSpec"},{"location":"reference/kube-ovn-api/#iptablesfiprulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesFIPRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesEIP \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesEIP \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesFIPRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesFIPRuleCondition IptablesFIPRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"IptablesFIPRuleStatus"},{"location":"reference/kube-ovn-api/#iptablessnatrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesSnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesSnatRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesSnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesSnatRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesSnatRule \u72b6\u6001\u4fe1\u606f","title":"IptablesSnatRule"},{"location":"reference/kube-ovn-api/#iptablessnatrulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip String IptablesSnatRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 internalIp String IptablesSnatRule \u5bf9\u5e94\u7684\u5185\u90e8\u7684 IP \u5730\u5740","title":"IptablesSnatRuleSpec"},{"location":"reference/kube-ovn-api/#iptablessnatrulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesSnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesSnatRule \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesSnatRule \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesSnatRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesSnatRuleCondition IptablesSnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"IptablesSnatRuleStatus"},{"location":"reference/kube-ovn-api/#iptablesdnatrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesDnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesDnatRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesDnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesDnatRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesDnatRule \u72b6\u6001\u4fe1\u606f","title":"IptablesDnatRule"},{"location":"reference/kube-ovn-api/#iptablesdnatrulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 externalPort Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5916\u90e8\u7aef\u53e3 protocol Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u7684\u534f\u8bae\u7c7b\u578b internalIp Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5185\u90e8 IP \u5730\u5740 internalPort Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5185\u90e8\u7aef\u53e3","title":"IptablesDnatRuleSpec"},{"location":"reference/kube-ovn-api/#iptablesdnatrulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesDnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesDnatRule \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesDnatRule \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesDnatRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesDnatRuleCondition IptablesDnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"IptablesDnatRuleStatus"},{"location":"reference/kube-ovn-api/#vpcdns","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a VpcDns metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcDnsSpec VpcDns \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VpcDnsStatus VpcDns \u72b6\u6001\u4fe1\u606f","title":"VpcDns"},{"location":"reference/kube-ovn-api/#vpcdnsspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vpc String VpcDns \u6240\u5728\u7684 Vpc \u540d\u79f0 subnet String VpcDns Pod \u5206\u914d\u5730\u5740\u7684 Subnet \u540d\u79f0","title":"VpcDnsSpec"},{"location":"reference/kube-ovn-api/#vpcdnsstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VpcDnsCondition VpcDns \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 active Bool VpcDns \u662f\u5426\u6b63\u5728\u4f7f\u7528 VpcDns \u7684\u8be6\u7ec6\u4f7f\u7528\u6587\u6863\uff0c\u53ef\u4ee5\u53c2\u8003 \u81ea\u5b9a\u4e49 VPC DNS \u3002","title":"VpcDnsStatus"},{"location":"reference/kube-ovn-api/#switchlbrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a SwitchLBRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SwitchLBRuleSpec SwitchLBRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SwitchLBRuleStatus SwitchLBRule \u72b6\u6001\u4fe1\u606f","title":"SwitchLBRule"},{"location":"reference/kube-ovn-api/#switchlbrulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vip String SwitchLBRule \u914d\u7f6e\u7684 vip \u5730\u5740 namespace String SwitchLBRule \u7684\u547d\u540d\u7a7a\u95f4 selector []String \u6807\u51c6 Kubernetes Selector \u5339\u914d\u4fe1\u606f sessionAffinity String \u6807\u51c6 Kubernetes Service \u4e2d sessionAffinity \u53d6\u503c ports []SlrPort SwitchLBRule \u7aef\u53e3\u5217\u8868 SwitchLBRule \u7684\u8be6\u7ec6\u914d\u7f6e\u4fe1\u606f\uff0c\u53ef\u4ee5\u53c2\u8003 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u3002","title":"SwitchLBRuleSpec"},{"location":"reference/kube-ovn-api/#slrport","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 name String \u7aef\u53e3\u540d\u79f0 port Int32 \u7aef\u53e3\u53f7 targetPort Int32 \u76ee\u6807\u7aef\u53e3\u53f7 protocol String \u534f\u8bae\u7c7b\u578b","title":"SlrPort"},{"location":"reference/kube-ovn-api/#switchlbrulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []SwitchLBRuleCondition SwitchLBRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ports String SwitchLBRule \u7aef\u53e3\u4fe1\u606f service String SwitchLBRule \u63d0\u4f9b\u670d\u52a1\u7684 service \u540d\u79f0","title":"SwitchLBRuleStatus"},{"location":"reference/kube-ovn-api/#vip","text":"","title":"\u5b89\u5168\u7ec4\u4e0e Vip"},{"location":"reference/kube-ovn-api/#securitygroup","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a SecurityGroup metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SecurityGroupSpec \u5b89\u5168\u7ec4\u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SecurityGroupStatus \u5b89\u5168\u7ec4\u72b6\u6001\u4fe1\u606f","title":"SecurityGroup"},{"location":"reference/kube-ovn-api/#securitygroupspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ingressRules []*SgRule \u5165\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 egressRules []*SgRule \u51fa\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 allowSameGroupTraffic Bool \u540c\u4e00\u5b89\u5168\u7ec4\u5185\u7684 lsp \u662f\u5426\u53ef\u4ee5\u4e92\u901a\uff0c\u4ee5\u53ca\u6d41\u91cf\u89c4\u5219\u662f\u5426\u9700\u8981\u66f4\u65b0","title":"SecurityGroupSpec"},{"location":"reference/kube-ovn-api/#sgrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ipVersion String IP \u7248\u672c\u53f7\uff0c\u53d6\u503c\u4e3a ipv4 \u6216\u8005 ipv6 protocol String \u53d6\u503c\u4e3a all \u3001 icmp \u3001 tcp \u6216\u8005 udp priority Int Acl \u4f18\u5148\u7ea7\uff0c\u53d6\u503c\u8303\u56f4\u4e3a 1-200\uff0c\u6570\u503c\u8d8a\u5c0f\uff0c\u4f18\u5148\u7ea7\u8d8a\u9ad8 remoteType String \u53d6\u503c\u4e3a address \u6216\u8005 securityGroup remoteAddress String \u5bf9\u7aef\u5730\u5740 remoteSecurityGroup String \u5bf9\u7aef\u5b89\u5168\u7ec4 portRangeMin Int \u7aef\u53e3\u8303\u56f4\u8d77\u59cb\u503c\uff0c\u6700\u5c0f\u53d6\u503c\u4e3a 1 portRangeMax Int \u7aef\u53e3\u8303\u56f4\u6700\u5927\u503c\uff0c\u6700\u5927\u53d6\u503c\u4e3a 65535 policy String \u53d6\u503c\u4e3a allow \u6216\u8005 drop","title":"SgRule"},{"location":"reference/kube-ovn-api/#securitygroupstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 portGroup String \u5b89\u5168\u7ec4\u5bf9\u5e94\u7684 port-group \u540d\u79f0 allowSameGroupTraffic Bool \u540c\u4e00\u5b89\u5168\u7ec4\u5185\u7684 lsp \u662f\u5426\u53ef\u4ee5\u4e92\u901a\uff0c\u4ee5\u53ca\u5b89\u5168\u7ec4\u7684\u6d41\u91cf\u89c4\u5219\u662f\u5426\u9700\u8981\u66f4\u65b0 ingressMd5 String \u5165\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 MD5 \u53d6\u503c egressMd5 String \u51fa\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 MD5 \u53d6\u503c ingressLastSyncSuccess Bool \u5165\u65b9\u5411\u89c4\u5219\u4e0a\u4e00\u6b21\u540c\u6b65\u662f\u5426\u6210\u529f egressLastSyncSuccess Bool \u51fa\u65b9\u5411\u89c4\u5219\u4e0a\u4e00\u6b21\u540c\u6b65\u662f\u5426\u6210\u529f","title":"SecurityGroupStatus"},{"location":"reference/kube-ovn-api/#vip_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VipSpec Vip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VipStatus Vip \u72b6\u6001\u4fe1\u606f","title":"Vip"},{"location":"reference/kube-ovn-api/#vipspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 namespace String Vip \u6240\u5728\u547d\u540d\u7a7a\u95f4 subnet String Vip \u6240\u5c5e\u5b50\u7f51 v4ip String Vip v4 IP \u5730\u5740 v6ip String Vip v6 IP \u5730\u5740 macAddress String Vip mac \u5730\u5740 parentV4ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 parentV6ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 parentMac String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 attachSubnets []String \u8be5\u5b57\u6bb5\u5e9f\u5f03\uff0c\u4e0d\u518d\u4f7f\u7528","title":"VipSpec"},{"location":"reference/kube-ovn-api/#vipstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VipCondition Vip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ready Bool Vip \u662f\u5426\u51c6\u5907\u597d v4ip String Vip v4 IP \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 v6ip String Vip v6 IP \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 mac String Vip mac \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 pv4ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 pv6ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 pmac String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528","title":"VipStatus"},{"location":"reference/kube-ovn-api/#ovneip","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnEip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnEipSpec \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnEip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnEipStatus \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnEip \u72b6\u6001\u4fe1\u606f","title":"OvnEip"},{"location":"reference/kube-ovn-api/#ovneipspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 externalSubnet String OvnEip \u6240\u5728\u7684\u5b50\u7f51\u540d\u79f0 v4ip String OvnEip IP \u5730\u5740 macAddress String OvnEip Mac \u5730\u5740 type String OvnEip \u4f7f\u7528\u7c7b\u578b\uff0c\u53d6\u503c\u6709 fip \u3001 snat \u6216\u8005 lrp","title":"OvnEipSpec"},{"location":"reference/kube-ovn-api/#ovneipstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []OvnEipCondition \u9ed8\u8ba4 Vpc OvnEip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 v4ip String OvnEip \u4f7f\u7528\u7684 v4 IP \u5730\u5740 macAddress String OvnEip \u4f7f\u7528\u7684 Mac \u5730\u5740","title":"OvnEipStatus"},{"location":"reference/kube-ovn-api/#ovnfip","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnFip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnFipSpec \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnFip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnFipStatus \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnFip \u72b6\u6001\u4fe1\u606f","title":"OvnFip"},{"location":"reference/kube-ovn-api/#ovnfipspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ovnEip String OvnFip \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 ipName String OvnFip \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 IP crd \u540d\u79f0","title":"OvnFipSpec"},{"location":"reference/kube-ovn-api/#ovnfipstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool OvnFip \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4Eip String OvnFip \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 v4Ip String OvnFip \u5f53\u524d\u4f7f\u7528\u7684 OvnEip \u5730\u5740 macAddress String OvnFip \u914d\u7f6e\u7684 Mac \u5730\u5740 vpc String OvnFip \u6240\u5728\u7684 Vpc \u540d\u79f0 conditions []OvnFipCondition OvnFip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"OvnFipStatus"},{"location":"reference/kube-ovn-api/#ovnsnatrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnSnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnSnatRuleSpec \u9ed8\u8ba4 Vpc OvnSnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnSnatRuleStatus \u9ed8\u8ba4 Vpc OvnSnatRule \u72b6\u6001\u4fe1\u606f","title":"OvnSnatRule"},{"location":"reference/kube-ovn-api/#ovnsnatrulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ovnEip String OvnSnatRule \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 vpcSubnet String OvnSnatRule \u914d\u7f6e\u7684\u5b50\u7f51\u540d\u79f0 ipName String OvnSnatRule \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 IP crd \u540d\u79f0","title":"OvnSnatRuleSpec"},{"location":"reference/kube-ovn-api/#ovnsnatrulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool OvnSnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4Eip String OvnSnatRule \u7ed1\u5b9a\u7684 OvnEip \u5730\u5740 v4IpCidr String \u5728 logical-router \u4e2d\u914d\u7f6e snat \u8f6c\u6362\u4f7f\u7528\u7684 cidr \u5730\u5740 vpc String OvnSnatRule \u6240\u5728\u7684 Vpc \u540d\u79f0 conditions []OvnSnatRuleCondition OvnSnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OvnSnatRuleStatus"},{"location":"reference/kube-ovn-pinger-args/","text":"Kube-OVN-Pinger \u53c2\u6570\u53c2\u8003 \u00b6 \u57fa\u4e8e Kube-OVN v1.12.0 \u7248\u672c\uff0c\u6574\u7406\u4e86 Kube-ovn-pinger \u652f\u6301\u7684\u53c2\u6570\uff0c\u5217\u51fa\u53c2\u6570\u5b9a\u4e49\u5404\u5b57\u6bb5\u7684\u53d6\u503c\u7c7b\u578b\uff0c\u542b\u4e49\u548c\u9ed8\u8ba4\u503c\uff0c\u4ee5\u4f9b\u53c2\u8003 \u53c2\u6570\u63cf\u8ff0 \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 \u9ed8\u8ba4\u503c port Int metrics \u7aef\u53e3 8080 kubeconfig String \u5177\u6709\u8ba4\u8bc1\u4fe1\u606f\u7684 kubeconfig \u6587\u4ef6\u8def\u5f84\uff0c \u5982\u679c\u672a\u8bbe\u7f6e\uff0c\u4f7f\u7528 inCluster \u4ee4\u724c\u3002 \"\" ds-namespace String kube-ovn-pinger \u5b88\u62a4\u8fdb\u7a0b\u547d\u540d\u7a7a\u95f4 \"kube-system\" ds-name String kube-ovn-pinger \u5b88\u62a4\u8fdb\u7a0b\u540d\u5b57 \"kube-ovn-pinger\" interval Int \u8fde\u7eed ping \u4e4b\u95f4\u7684\u95f4\u9694\u79d2\u6570 5 mode String \u670d\u52a1\u5668\u6216\u5de5\u4f5c\u6a21\u5f0f \"server\" exit-code Int \u5931\u8d25\u65f6\u9000\u51fa\u4ee3\u7801 0 internal-dns String \u4ece pod \u5185\u89e3\u6790\u5185\u90e8 dns \"kubernetes.default\" external-dns String \u4ece pod \u5185\u89e3\u6790\u5916\u90e8 dns \"\" external-address String \u68c0\u67e5\u4e0e\u5916\u90e8\u5730\u5740\u7684 ping \u8fde\u901a \"114.114.114.114\" network-mode String \u5f53\u524d\u96c6\u7fa4\u4f7f\u7528\u7684 cni \u63d2\u4ef6 \"kube-ovn\" enable-metrics Bool \u662f\u5426\u652f\u6301 metrics \u67e5\u8be2 true ovs.timeout Int \u5bf9 OVS \u7684 JSON-RPC \u8bf7\u6c42\u8d85\u65f6\u3002 2 system.run.dir String OVS \u9ed8\u8ba4\u8fd0\u884c\u76ee\u5f55\u3002 \"/var/run/openvswitch\" database.vswitch.name String OVS \u6570\u636e\u5e93\u7684\u540d\u79f0\u3002 \"Open_vSwitch\" database.vswitch.socket.remote String JSON-RPC unix \u5957\u63a5\u5b57\u5230 OVS \u6570\u636e\u5e93\u3002 \"unix:/var/run/openvswitch/db.sock\" database.vswitch.file.data.path String OVS \u6570\u636e\u5e93\u6587\u4ef6\u3002 \"/etc/openvswitch/conf.db\" database.vswitch.file.log.path String OVS \u6570\u636e\u5e93\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/openvswitch/ovsdb-server.log\" database.vswitch.file.pid.path String OVS \u6570\u636e\u5e93\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/openvswitch/ovsdb-server.pid\" database.vswitch.file.system.id.path String OVS \u7cfb\u7edf\u6807\u8bc6\u6587\u4ef6\u3002 \"/etc/openvswitch/system-id.conf\" service.vswitchd.file.log.path String OVS vswitchd \u5b88\u62a4\u8fdb\u7a0b\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/openvswitch/ovs-vswitchd.log\" service.vswitchd.file.pid.path String OVS vswitchd \u5b88\u62a4\u8fdb\u7a0b\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/openvswitch/ovs-vswitchd.pid\" service.ovncontroller.file.log.path String OVN \u63a7\u5236\u5668\u5b88\u62a4\u8fdb\u7a0b\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/ovn/ovn-controller.log\" service.ovncontroller.file.pid.path String OVN \u63a7\u5236\u5668\u5b88\u62a4\u8fdb\u7a0b\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/ovn/ovn-controller.pid\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kube-ovn-pinger \u53c2\u6570\u63cf\u8ff0"},{"location":"reference/kube-ovn-pinger-args/#kube-ovn-pinger","text":"\u57fa\u4e8e Kube-OVN v1.12.0 \u7248\u672c\uff0c\u6574\u7406\u4e86 Kube-ovn-pinger \u652f\u6301\u7684\u53c2\u6570\uff0c\u5217\u51fa\u53c2\u6570\u5b9a\u4e49\u5404\u5b57\u6bb5\u7684\u53d6\u503c\u7c7b\u578b\uff0c\u542b\u4e49\u548c\u9ed8\u8ba4\u503c\uff0c\u4ee5\u4f9b\u53c2\u8003","title":"Kube-OVN-Pinger \u53c2\u6570\u53c2\u8003"},{"location":"reference/kube-ovn-pinger-args/#_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 \u9ed8\u8ba4\u503c port Int metrics \u7aef\u53e3 8080 kubeconfig String \u5177\u6709\u8ba4\u8bc1\u4fe1\u606f\u7684 kubeconfig \u6587\u4ef6\u8def\u5f84\uff0c \u5982\u679c\u672a\u8bbe\u7f6e\uff0c\u4f7f\u7528 inCluster \u4ee4\u724c\u3002 \"\" ds-namespace String kube-ovn-pinger \u5b88\u62a4\u8fdb\u7a0b\u547d\u540d\u7a7a\u95f4 \"kube-system\" ds-name String kube-ovn-pinger \u5b88\u62a4\u8fdb\u7a0b\u540d\u5b57 \"kube-ovn-pinger\" interval Int \u8fde\u7eed ping \u4e4b\u95f4\u7684\u95f4\u9694\u79d2\u6570 5 mode String \u670d\u52a1\u5668\u6216\u5de5\u4f5c\u6a21\u5f0f \"server\" exit-code Int \u5931\u8d25\u65f6\u9000\u51fa\u4ee3\u7801 0 internal-dns String \u4ece pod \u5185\u89e3\u6790\u5185\u90e8 dns \"kubernetes.default\" external-dns String \u4ece pod \u5185\u89e3\u6790\u5916\u90e8 dns \"\" external-address String \u68c0\u67e5\u4e0e\u5916\u90e8\u5730\u5740\u7684 ping \u8fde\u901a \"114.114.114.114\" network-mode String \u5f53\u524d\u96c6\u7fa4\u4f7f\u7528\u7684 cni \u63d2\u4ef6 \"kube-ovn\" enable-metrics Bool \u662f\u5426\u652f\u6301 metrics \u67e5\u8be2 true ovs.timeout Int \u5bf9 OVS \u7684 JSON-RPC \u8bf7\u6c42\u8d85\u65f6\u3002 2 system.run.dir String OVS \u9ed8\u8ba4\u8fd0\u884c\u76ee\u5f55\u3002 \"/var/run/openvswitch\" database.vswitch.name String OVS \u6570\u636e\u5e93\u7684\u540d\u79f0\u3002 \"Open_vSwitch\" database.vswitch.socket.remote String JSON-RPC unix \u5957\u63a5\u5b57\u5230 OVS \u6570\u636e\u5e93\u3002 \"unix:/var/run/openvswitch/db.sock\" database.vswitch.file.data.path String OVS \u6570\u636e\u5e93\u6587\u4ef6\u3002 \"/etc/openvswitch/conf.db\" database.vswitch.file.log.path String OVS \u6570\u636e\u5e93\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/openvswitch/ovsdb-server.log\" database.vswitch.file.pid.path String OVS \u6570\u636e\u5e93\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/openvswitch/ovsdb-server.pid\" database.vswitch.file.system.id.path String OVS \u7cfb\u7edf\u6807\u8bc6\u6587\u4ef6\u3002 \"/etc/openvswitch/system-id.conf\" service.vswitchd.file.log.path String OVS vswitchd \u5b88\u62a4\u8fdb\u7a0b\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/openvswitch/ovs-vswitchd.log\" service.vswitchd.file.pid.path String OVS vswitchd \u5b88\u62a4\u8fdb\u7a0b\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/openvswitch/ovs-vswitchd.pid\" service.ovncontroller.file.log.path String OVN \u63a7\u5236\u5668\u5b88\u62a4\u8fdb\u7a0b\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/ovn/ovn-controller.log\" service.ovncontroller.file.pid.path String OVN \u63a7\u5236\u5668\u5b88\u62a4\u8fdb\u7a0b\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/ovn/ovn-controller.pid\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u53c2\u6570\u63cf\u8ff0"},{"location":"reference/metrics/","text":"Kube-OVN \u76d1\u63a7\u6307\u6807 \u00b6 \u672c\u6587\u6863\u5217\u4e3e Kube-OVN \u6240\u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\u3002 ovn-monitor \u00b6 OVN \u81ea\u8eab\u72b6\u6001\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge kube_ovn_ovn_status OVN \u89d2\u8272\u72b6\u6001\uff0c (2) \u4e3a follower\uff1b (1) \u4e3a leader, (0) \u4e3a\u5f02\u5e38\u72b6\u6001\u3002 Gauge kube_ovn_failed_req_count OVN \u5931\u8d25\u8bf7\u6c42\u6570\u91cf\u3002 Gauge kube_ovn_log_file_size_bytes OVN \u7ec4\u4ef6\u65e5\u5fd7\u6587\u4ef6\u5927\u5c0f\u3002 Gauge kube_ovn_db_file_size_bytes OVN \u7ec4\u4ef6\u6570\u636e\u5e93\u6587\u4ef6\u5927\u5c0f\u3002 Gauge kube_ovn_chassis_info OVN chassis \u72b6\u6001 (1) \u8fd0\u884c\u4e2d\uff0c(0) \u505c\u6b62\u3002 Gauge kube_ovn_db_status OVN \u6570\u636e\u5e93\u72b6\u6001, (1) \u4e3a\u6b63\u5e38\uff1b (0) \u4e3a\u5f02\u5e38\u3002 Gauge kube_ovn_logical_switch_info OVN logical switch \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b logical switch \u540d\u5b57\u3002 Gauge kube_ovn_logical_switch_external_id OVN logical switch external_id \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b external-id \u5185\u5bb9\u3002 Gauge kube_ovn_logical_switch_port_binding OVN logical switch \u548c logical switch port \u5173\u8054\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u901a\u8fc7\u6807\u7b7e\u8fdb\u884c\u5173\u8054\u3002 Gauge kube_ovn_logical_switch_tunnel_key \u548c OVN logical switch \u5173\u8054\u7684 tunnel key \u4fe1\u606f\u3002 Gauge kube_ovn_logical_switch_ports_num OVN logical switch \u4e0a logical port \u7684\u6570\u91cf\u3002 Gauge kube_ovn_logical_switch_port_info OVN logical switch port \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5177\u4f53\u4fe1\u606f\u3002 Gauge kube_ovn_logical_switch_port_tunnel_key \u548c OVN logical switch port \u5173\u8054\u7684 tunnel key \u4fe1\u606f\u3002 Gauge kube_ovn_cluster_enabled (1) OVN \u6570\u636e\u5e93\u4e3a\u96c6\u7fa4\u6a21\u5f0f\uff1b (0) OVN \u6570\u636e\u5e93\u4e3a\u975e\u96c6\u7fa4\u6a21\u5f0f\u3002 Gauge kube_ovn_cluster_role \u6bcf\u4e2a\u6570\u636e\u5e93\u5b9e\u4f8b\u7684\u89d2\u8272\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u89d2\u8272\u4fe1\u606f\u3002 Gauge kube_ovn_cluster_status \u6bcf\u4e2a\u6570\u636e\u5e93\u5b9e\u4f8b\u7684\u72b6\u6001\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u72b6\u6001\u4fe1\u606f\u3002 Gauge kube_ovn_cluster_term RAFT term \u4fe1\u606f\u3002 Gauge kube_ovn_cluster_leader_self \u5f53\u524d\u6570\u636e\u5e93\u5b9e\u4f8b\u662f\u5426\u4e3a leader (1) \u662f\uff0c (0) \u4e0d\u662f\u3002 Gauge kube_ovn_cluster_vote_self \u5f53\u524d\u6570\u636e\u5e93\u5b9e\u4f8b\u662f\u5426\u9009\u4e3e\u81ea\u5df1\u4e3a leader (1) \u662f\uff0c (0) \u4e0d\u662f\u3002 Gauge kube_ovn_cluster_election_timer \u5f53\u524d election timer \u503c\u3002 Gauge kube_ovn_cluster_log_not_committed \u672a commit \u7684 RAFT \u65e5\u5fd7\u6570\u91cf\u3002 Gauge kube_ovn_cluster_log_not_applied \u672a apply \u7684 RAFT \u65e5\u5fd7\u6570\u91cf\u3002 Gauge kube_ovn_cluster_log_index_start \u5f53\u524d RAFT \u65e5\u5fd7\u6761\u76ee\u7684\u8d77\u59cb\u503c\u3002 Gauge kube_ovn_cluster_log_index_next RAFT \u65e5\u5fd7\u6761\u76ee\u7684\u4e0b\u4e00\u4e2a\u503c\u3002 Gauge kube_ovn_cluster_inbound_connections_total \u5f53\u524d\u5b9e\u4f8b\u7684\u5165\u5411\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_outbound_connections_total \u5f53\u524d\u5b9e\u4f8b\u7684\u51fa\u5411\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_inbound_connections_error_total \u5f53\u524d\u5b9e\u4f8b\u7684\u5165\u5411\u9519\u8bef\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_outbound_connections_error_total \u5f53\u524d\u5b9e\u4f8b\u7684\u51fa\u5411\u9519\u8bef\u8fde\u63a5\u6570\u91cf\u3002 ovs-monitor \u00b6 ovsdb \u548c vswitchd \u81ea\u8eab\u72b6\u6001\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge ovs_status OVS \u5065\u5eb7\u72b6\u6001\uff0c (1) \u4e3a\u6b63\u5e38\uff0c(0) \u4e3a\u5f02\u5e38\u3002 Gauge ovs_info OVS \u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge failed_req_count OVS \u5931\u8d25\u8bf7\u6c42\u6570\u91cf\u3002 Gauge log_file_size OVS \u7ec4\u4ef6\u65e5\u5fd7\u6587\u4ef6\u5927\u5c0f\u3002 Gauge db_file_size OVS \u7ec4\u4ef6\u6570\u636e\u5e93\u6587\u4ef6\u5927\u5c0f\u3002 Gauge datapath Datapath \u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge dp_total \u5f53\u524d OVS \u4e2d datapath \u6570\u91cf\u3002 Gauge dp_if Datapath \u63a5\u53e3\u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge dp_if_total \u5f53\u524d datapath \u4e2d port \u6570\u91cf\u3002 Gauge dp_flows_total Datapath \u4e2d flow \u6570\u91cf\u3002 Gauge dp_flows_lookup_hit Datapath \u4e2d\u547d\u4e2d\u5f53\u524d flow \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_flows_lookup_missed Datapath \u4e2d\u672a\u547d\u4e2d\u5f53\u524d flow \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_flows_lookup_lost Datapath \u4e2d\u9700\u8981\u53d1\u9001\u7ed9 userspace \u5904\u7406\u7684\u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_masks_hit Datapath \u4e2d\u547d\u4e2d\u5f53\u524d mask \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_masks_total Datapath \u4e2d mask \u7684\u6570\u91cf\u3002 Gauge dp_masks_hit_ratio Datapath \u4e2d \u6570\u636e\u5305\u547d\u4e2d mask \u7684\u6bd4\u7387\u3002 Gauge interface OVS \u63a5\u53e3\u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge interface_admin_state \u63a5\u53e3\u7ba1\u7406\u72b6\u6001\u4fe1\u606f (0) \u4e3a down, (1) \u4e3a up, (2) \u4e3a\u5176\u4ed6\u72b6\u6001\u3002 Gauge interface_link_state \u63a5\u53e3\u94fe\u8def\u72b6\u6001\u4fe1\u606f (0) \u4e3a down, (1) \u4e3a up, (2) \u4e3a\u5176\u4ed6\u72b6\u6001\u3002 Gauge interface_mac_in_use OVS Interface \u4f7f\u7528\u7684 MAC \u5730\u5740 Gauge interface_mtu OVS Interface \u4f7f\u7528\u7684 MTU\u3002 Gauge interface_of_port OVS Interface \u5173\u8054\u7684 OpenFlow Port ID\u3002 Gauge interface_if_index OVS Interface \u5173\u8054\u7684 Index\u3002 Gauge interface_tx_packets OVS Interface \u53d1\u9001\u5305\u6570\u91cf\u3002 Gauge interface_tx_bytes OVS Interface \u53d1\u9001\u5305\u5927\u5c0f\u3002 Gauge interface_rx_packets OVS Interface \u63a5\u6536\u5305\u6570\u91cf\u3002 Gauge interface_rx_bytes OVS Interface \u63a5\u6536\u5305\u5927\u5c0f\u3002 Gauge interface_rx_crc_err OVS Interface \u63a5\u6536\u5305\u6821\u9a8c\u548c\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_dropped OVS Interface \u63a5\u6536\u5305\u4e22\u5f03\u6570\u91cf\u3002 Gauge interface_rx_errors OVS Interface \u63a5\u6536\u5305\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_frame_err OVS Interface \u63a5\u6536\u5e27\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_missed_err OVS Interface \u63a5\u6536\u5305 miss \u6570\u91cf\u3002 Gauge interface_rx_over_err OVS Interface \u63a5\u6536\u5305 overrun \u6570\u91cf\u3002 Gauge interface_tx_dropped OVS Interface \u53d1\u9001\u5305\u4e22\u5f03\u6570\u91cf\u3002 Gauge interface_tx_errors OVS Interface \u53d1\u9001\u5305\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_collisions OVS interface \u51b2\u7a81\u6570\u91cf\u3002 kube-ovn-pinger \u00b6 \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge pinger_ovs_up \u8282\u70b9 OVS \u8fd0\u884c\u3002 Gauge pinger_ovs_down \u8282\u70b9 OVS \u505c\u6b62\u3002 Gauge pinger_ovn_controller_up \u8282\u70b9 ovn-controller \u8fd0\u884c\u3002 Gauge pinger_ovn_controller_down \u8282\u70b9 ovn-controller \u505c\u6b62\u3002 Gauge pinger_inconsistent_port_binding OVN-SB \u91cc portbinding \u6570\u91cf\u548c\u4e3b\u673a OVS interface \u4e0d\u4e00\u81f4\u7684\u6570\u91cf\u3002 Gauge pinger_apiserver_healthy kube-ovn-pinger \u53ef\u4ee5\u8054\u901a apiserver\u3002 Gauge pinger_apiserver_unhealthy kube-ovn-pinger \u65e0\u6cd5\u8054\u901a apiserver\u3002 Histogram pinger_apiserver_latency_ms kube-ovn-pinger \u8bbf\u95ee apiserver \u5ef6\u8fdf\u3002 Gauge pinger_internal_dns_healthy kube-ovn-pinger \u53ef\u4ee5\u89e3\u6790\u5185\u90e8\u57df\u540d\u3002 Gauge pinger_internal_dns_unhealthy kube-ovn-pinger \u65e0\u6cd5\u89e3\u6790\u5185\u90e8\u57df\u540d\u3002 Histogram pinger_internal_dns_latency_ms kube-ovn-pinger \u89e3\u6790\u5185\u90e8\u57df\u540d\u5ef6\u8fdf\u3002 Gauge pinger_external_dns_health kube-ovn-pinger \u53ef\u4ee5\u89e3\u6790\u5916\u90e8\u57df\u540d\u3002 Gauge pinger_external_dns_unhealthy kube-ovn-pinger \u65e0\u6cd5\u89e3\u6790\u5916\u90e8\u57df\u540d\u3002 Histogram pinger_external_dns_latency_ms kube-ovn-pinger \u89e3\u6790\u5916\u90e8\u57df\u540d\u5ef6\u8fdf\u3002 Histogram pinger_pod_ping_latency_ms kube-ovn-pinger ping Pod \u5ef6\u8fdf\u3002 Gauge pinger_pod_ping_lost_total kube-ovn-pinger ping Pod \u4e22\u5305\u6570\u91cf\u3002 Gauge pinger_pod_ping_count_total kube-ovn-pinger ping Pod \u6570\u91cf\u3002 Histogram pinger_node_ping_latency_ms kube-ovn-pinger ping Node \u5ef6\u8fdf\u3002 Gauge pinger_node_ping_lost_total kube-ovn-pinger ping Node \u4e22\u5305\u3002 Gauge pinger_node_ping_count_total kube-ovn-pinger ping Node \u6570\u91cf\u3002 Histogram pinger_external_ping_latency_ms kube-ovn-pinger ping \u5916\u90e8\u5730\u5740 \u5ef6\u8fdf\u3002 Gauge pinger_external_lost_total kube-ovn-pinger ping \u5916\u90e8\u4e22\u5305\u6570\u91cf\u3002 kube-ovn-controller \u00b6 kube-ovn-controller \u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Histogram rest_client_request_latency_seconds \u8bf7\u6c42 apiserver \u5ef6\u8fdf\u3002 Counter rest_client_requests_total \u8bf7\u6c42 apiserver \u6570\u91cf\u3002 Counter lists_total API list \u8bf7\u6c42\u6570\u91cf\u3002 Summary list_duration_seconds API list \u8bf7\u6c42\u5ef6\u8fdf\u3002 Summary items_per_list API list \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Counter watches_total API watch \u8bf7\u6c42\u6570\u91cf\u3002 Counter short_watches_total \u77ed\u65f6\u95f4 API watch \u8bf7\u6c42\u6570\u91cf\u3002 Summary watch_duration_seconds API watch \u6301\u7eed\u65f6\u95f4\u3002 Summary items_per_watch API watch \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Gauge last_resource_version \u6700\u65b0\u7684 resource version\u3002 Histogram ovs_client_request_latency_milliseconds \u8bf7\u6c42 OVN \u7ec4\u4ef6\u5ef6\u8fdf\u3002 Gauge subnet_available_ip_count \u5b50\u7f51\u53ef\u7528 IP \u6570\u91cf\u3002 Gauge subnet_used_ip_count \u5b50\u7f51\u5df2\u7528 IP \u6570\u91cf\u3002 kube-ovn-cni \u00b6 kube-ovn-cni \u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Histogram cni_op_latency_seconds CNI \u64cd\u4f5c\u5ef6\u8fdf\u3002 Counter cni_wait_address_seconds_total CNI \u7b49\u5f85\u5730\u5740\u5c31\u7eea\u65f6\u95f4\u3002 Counter cni_wait_connectivity_seconds_total CNI \u7b49\u5f85\u8fde\u63a5\u5c31\u7eea\u65f6\u95f4\u3002 Counter cni_wait_route_seconds_total CNI \u7b49\u5f85\u8def\u7531\u5c31\u7eea\u65f6\u95f4\u3002 Histogram rest_client_request_latency_seconds \u8bf7\u6c42 apiserver \u5ef6\u8fdf\u3002 Counter rest_client_requests_total \u8bf7\u6c42 apiserver \u6570\u91cf\u3002 Counter lists_total API list \u8bf7\u6c42\u6570\u91cf\u3002 Summary list_duration_seconds API list \u8bf7\u6c42\u5ef6\u8fdf\u3002 Summary items_per_list API list \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Counter watches_total API watch \u8bf7\u6c42\u6570\u91cf\u3002 Counter short_watches_total \u77ed\u65f6\u95f4 API watch \u8bf7\u6c42\u6570\u91cf\u3002 Summary watch_duration_seconds API watch \u6301\u7eed\u65f6\u95f4\u3002 Summary items_per_watch API watch \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Gauge last_resource_version \u6700\u65b0\u7684 resource version\u3002 Histogram ovs_client_request_latency_milliseconds \u8bf7\u6c42 OVN \u7ec4\u4ef6\u5ef6\u8fdf\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN \u76d1\u63a7\u6307\u6807"},{"location":"reference/metrics/#kube-ovn","text":"\u672c\u6587\u6863\u5217\u4e3e Kube-OVN \u6240\u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\u3002","title":"Kube-OVN \u76d1\u63a7\u6307\u6807"},{"location":"reference/metrics/#ovn-monitor","text":"OVN \u81ea\u8eab\u72b6\u6001\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge kube_ovn_ovn_status OVN \u89d2\u8272\u72b6\u6001\uff0c (2) \u4e3a follower\uff1b (1) \u4e3a leader, (0) \u4e3a\u5f02\u5e38\u72b6\u6001\u3002 Gauge kube_ovn_failed_req_count OVN \u5931\u8d25\u8bf7\u6c42\u6570\u91cf\u3002 Gauge kube_ovn_log_file_size_bytes OVN \u7ec4\u4ef6\u65e5\u5fd7\u6587\u4ef6\u5927\u5c0f\u3002 Gauge kube_ovn_db_file_size_bytes OVN \u7ec4\u4ef6\u6570\u636e\u5e93\u6587\u4ef6\u5927\u5c0f\u3002 Gauge kube_ovn_chassis_info OVN chassis \u72b6\u6001 (1) \u8fd0\u884c\u4e2d\uff0c(0) \u505c\u6b62\u3002 Gauge kube_ovn_db_status OVN \u6570\u636e\u5e93\u72b6\u6001, (1) \u4e3a\u6b63\u5e38\uff1b (0) \u4e3a\u5f02\u5e38\u3002 Gauge kube_ovn_logical_switch_info OVN logical switch \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b logical switch \u540d\u5b57\u3002 Gauge kube_ovn_logical_switch_external_id OVN logical switch external_id \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b external-id \u5185\u5bb9\u3002 Gauge kube_ovn_logical_switch_port_binding OVN logical switch \u548c logical switch port \u5173\u8054\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u901a\u8fc7\u6807\u7b7e\u8fdb\u884c\u5173\u8054\u3002 Gauge kube_ovn_logical_switch_tunnel_key \u548c OVN logical switch \u5173\u8054\u7684 tunnel key \u4fe1\u606f\u3002 Gauge kube_ovn_logical_switch_ports_num OVN logical switch \u4e0a logical port \u7684\u6570\u91cf\u3002 Gauge kube_ovn_logical_switch_port_info OVN logical switch port \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5177\u4f53\u4fe1\u606f\u3002 Gauge kube_ovn_logical_switch_port_tunnel_key \u548c OVN logical switch port \u5173\u8054\u7684 tunnel key \u4fe1\u606f\u3002 Gauge kube_ovn_cluster_enabled (1) OVN \u6570\u636e\u5e93\u4e3a\u96c6\u7fa4\u6a21\u5f0f\uff1b (0) OVN \u6570\u636e\u5e93\u4e3a\u975e\u96c6\u7fa4\u6a21\u5f0f\u3002 Gauge kube_ovn_cluster_role \u6bcf\u4e2a\u6570\u636e\u5e93\u5b9e\u4f8b\u7684\u89d2\u8272\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u89d2\u8272\u4fe1\u606f\u3002 Gauge kube_ovn_cluster_status \u6bcf\u4e2a\u6570\u636e\u5e93\u5b9e\u4f8b\u7684\u72b6\u6001\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u72b6\u6001\u4fe1\u606f\u3002 Gauge kube_ovn_cluster_term RAFT term \u4fe1\u606f\u3002 Gauge kube_ovn_cluster_leader_self \u5f53\u524d\u6570\u636e\u5e93\u5b9e\u4f8b\u662f\u5426\u4e3a leader (1) \u662f\uff0c (0) \u4e0d\u662f\u3002 Gauge kube_ovn_cluster_vote_self \u5f53\u524d\u6570\u636e\u5e93\u5b9e\u4f8b\u662f\u5426\u9009\u4e3e\u81ea\u5df1\u4e3a leader (1) \u662f\uff0c (0) \u4e0d\u662f\u3002 Gauge kube_ovn_cluster_election_timer \u5f53\u524d election timer \u503c\u3002 Gauge kube_ovn_cluster_log_not_committed \u672a commit \u7684 RAFT \u65e5\u5fd7\u6570\u91cf\u3002 Gauge kube_ovn_cluster_log_not_applied \u672a apply \u7684 RAFT \u65e5\u5fd7\u6570\u91cf\u3002 Gauge kube_ovn_cluster_log_index_start \u5f53\u524d RAFT \u65e5\u5fd7\u6761\u76ee\u7684\u8d77\u59cb\u503c\u3002 Gauge kube_ovn_cluster_log_index_next RAFT \u65e5\u5fd7\u6761\u76ee\u7684\u4e0b\u4e00\u4e2a\u503c\u3002 Gauge kube_ovn_cluster_inbound_connections_total \u5f53\u524d\u5b9e\u4f8b\u7684\u5165\u5411\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_outbound_connections_total \u5f53\u524d\u5b9e\u4f8b\u7684\u51fa\u5411\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_inbound_connections_error_total \u5f53\u524d\u5b9e\u4f8b\u7684\u5165\u5411\u9519\u8bef\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_outbound_connections_error_total \u5f53\u524d\u5b9e\u4f8b\u7684\u51fa\u5411\u9519\u8bef\u8fde\u63a5\u6570\u91cf\u3002","title":"ovn-monitor"},{"location":"reference/metrics/#ovs-monitor","text":"ovsdb \u548c vswitchd \u81ea\u8eab\u72b6\u6001\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge ovs_status OVS \u5065\u5eb7\u72b6\u6001\uff0c (1) \u4e3a\u6b63\u5e38\uff0c(0) \u4e3a\u5f02\u5e38\u3002 Gauge ovs_info OVS \u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge failed_req_count OVS \u5931\u8d25\u8bf7\u6c42\u6570\u91cf\u3002 Gauge log_file_size OVS \u7ec4\u4ef6\u65e5\u5fd7\u6587\u4ef6\u5927\u5c0f\u3002 Gauge db_file_size OVS \u7ec4\u4ef6\u6570\u636e\u5e93\u6587\u4ef6\u5927\u5c0f\u3002 Gauge datapath Datapath \u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge dp_total \u5f53\u524d OVS \u4e2d datapath \u6570\u91cf\u3002 Gauge dp_if Datapath \u63a5\u53e3\u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge dp_if_total \u5f53\u524d datapath \u4e2d port \u6570\u91cf\u3002 Gauge dp_flows_total Datapath \u4e2d flow \u6570\u91cf\u3002 Gauge dp_flows_lookup_hit Datapath \u4e2d\u547d\u4e2d\u5f53\u524d flow \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_flows_lookup_missed Datapath \u4e2d\u672a\u547d\u4e2d\u5f53\u524d flow \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_flows_lookup_lost Datapath \u4e2d\u9700\u8981\u53d1\u9001\u7ed9 userspace \u5904\u7406\u7684\u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_masks_hit Datapath \u4e2d\u547d\u4e2d\u5f53\u524d mask \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_masks_total Datapath \u4e2d mask \u7684\u6570\u91cf\u3002 Gauge dp_masks_hit_ratio Datapath \u4e2d \u6570\u636e\u5305\u547d\u4e2d mask \u7684\u6bd4\u7387\u3002 Gauge interface OVS \u63a5\u53e3\u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge interface_admin_state \u63a5\u53e3\u7ba1\u7406\u72b6\u6001\u4fe1\u606f (0) \u4e3a down, (1) \u4e3a up, (2) \u4e3a\u5176\u4ed6\u72b6\u6001\u3002 Gauge interface_link_state \u63a5\u53e3\u94fe\u8def\u72b6\u6001\u4fe1\u606f (0) \u4e3a down, (1) \u4e3a up, (2) \u4e3a\u5176\u4ed6\u72b6\u6001\u3002 Gauge interface_mac_in_use OVS Interface \u4f7f\u7528\u7684 MAC \u5730\u5740 Gauge interface_mtu OVS Interface \u4f7f\u7528\u7684 MTU\u3002 Gauge interface_of_port OVS Interface \u5173\u8054\u7684 OpenFlow Port ID\u3002 Gauge interface_if_index OVS Interface \u5173\u8054\u7684 Index\u3002 Gauge interface_tx_packets OVS Interface \u53d1\u9001\u5305\u6570\u91cf\u3002 Gauge interface_tx_bytes OVS Interface \u53d1\u9001\u5305\u5927\u5c0f\u3002 Gauge interface_rx_packets OVS Interface \u63a5\u6536\u5305\u6570\u91cf\u3002 Gauge interface_rx_bytes OVS Interface \u63a5\u6536\u5305\u5927\u5c0f\u3002 Gauge interface_rx_crc_err OVS Interface \u63a5\u6536\u5305\u6821\u9a8c\u548c\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_dropped OVS Interface \u63a5\u6536\u5305\u4e22\u5f03\u6570\u91cf\u3002 Gauge interface_rx_errors OVS Interface \u63a5\u6536\u5305\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_frame_err OVS Interface \u63a5\u6536\u5e27\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_missed_err OVS Interface \u63a5\u6536\u5305 miss \u6570\u91cf\u3002 Gauge interface_rx_over_err OVS Interface \u63a5\u6536\u5305 overrun \u6570\u91cf\u3002 Gauge interface_tx_dropped OVS Interface \u53d1\u9001\u5305\u4e22\u5f03\u6570\u91cf\u3002 Gauge interface_tx_errors OVS Interface \u53d1\u9001\u5305\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_collisions OVS interface \u51b2\u7a81\u6570\u91cf\u3002","title":"ovs-monitor"},{"location":"reference/metrics/#kube-ovn-pinger","text":"\u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge pinger_ovs_up \u8282\u70b9 OVS \u8fd0\u884c\u3002 Gauge pinger_ovs_down \u8282\u70b9 OVS \u505c\u6b62\u3002 Gauge pinger_ovn_controller_up \u8282\u70b9 ovn-controller \u8fd0\u884c\u3002 Gauge pinger_ovn_controller_down \u8282\u70b9 ovn-controller \u505c\u6b62\u3002 Gauge pinger_inconsistent_port_binding OVN-SB \u91cc portbinding \u6570\u91cf\u548c\u4e3b\u673a OVS interface \u4e0d\u4e00\u81f4\u7684\u6570\u91cf\u3002 Gauge pinger_apiserver_healthy kube-ovn-pinger \u53ef\u4ee5\u8054\u901a apiserver\u3002 Gauge pinger_apiserver_unhealthy kube-ovn-pinger \u65e0\u6cd5\u8054\u901a apiserver\u3002 Histogram pinger_apiserver_latency_ms kube-ovn-pinger \u8bbf\u95ee apiserver \u5ef6\u8fdf\u3002 Gauge pinger_internal_dns_healthy kube-ovn-pinger \u53ef\u4ee5\u89e3\u6790\u5185\u90e8\u57df\u540d\u3002 Gauge pinger_internal_dns_unhealthy kube-ovn-pinger \u65e0\u6cd5\u89e3\u6790\u5185\u90e8\u57df\u540d\u3002 Histogram pinger_internal_dns_latency_ms kube-ovn-pinger \u89e3\u6790\u5185\u90e8\u57df\u540d\u5ef6\u8fdf\u3002 Gauge pinger_external_dns_health kube-ovn-pinger \u53ef\u4ee5\u89e3\u6790\u5916\u90e8\u57df\u540d\u3002 Gauge pinger_external_dns_unhealthy kube-ovn-pinger \u65e0\u6cd5\u89e3\u6790\u5916\u90e8\u57df\u540d\u3002 Histogram pinger_external_dns_latency_ms kube-ovn-pinger \u89e3\u6790\u5916\u90e8\u57df\u540d\u5ef6\u8fdf\u3002 Histogram pinger_pod_ping_latency_ms kube-ovn-pinger ping Pod \u5ef6\u8fdf\u3002 Gauge pinger_pod_ping_lost_total kube-ovn-pinger ping Pod \u4e22\u5305\u6570\u91cf\u3002 Gauge pinger_pod_ping_count_total kube-ovn-pinger ping Pod \u6570\u91cf\u3002 Histogram pinger_node_ping_latency_ms kube-ovn-pinger ping Node \u5ef6\u8fdf\u3002 Gauge pinger_node_ping_lost_total kube-ovn-pinger ping Node \u4e22\u5305\u3002 Gauge pinger_node_ping_count_total kube-ovn-pinger ping Node \u6570\u91cf\u3002 Histogram pinger_external_ping_latency_ms kube-ovn-pinger ping \u5916\u90e8\u5730\u5740 \u5ef6\u8fdf\u3002 Gauge pinger_external_lost_total kube-ovn-pinger ping \u5916\u90e8\u4e22\u5305\u6570\u91cf\u3002","title":"kube-ovn-pinger"},{"location":"reference/metrics/#kube-ovn-controller","text":"kube-ovn-controller \u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Histogram rest_client_request_latency_seconds \u8bf7\u6c42 apiserver \u5ef6\u8fdf\u3002 Counter rest_client_requests_total \u8bf7\u6c42 apiserver \u6570\u91cf\u3002 Counter lists_total API list \u8bf7\u6c42\u6570\u91cf\u3002 Summary list_duration_seconds API list \u8bf7\u6c42\u5ef6\u8fdf\u3002 Summary items_per_list API list \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Counter watches_total API watch \u8bf7\u6c42\u6570\u91cf\u3002 Counter short_watches_total \u77ed\u65f6\u95f4 API watch \u8bf7\u6c42\u6570\u91cf\u3002 Summary watch_duration_seconds API watch \u6301\u7eed\u65f6\u95f4\u3002 Summary items_per_watch API watch \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Gauge last_resource_version \u6700\u65b0\u7684 resource version\u3002 Histogram ovs_client_request_latency_milliseconds \u8bf7\u6c42 OVN \u7ec4\u4ef6\u5ef6\u8fdf\u3002 Gauge subnet_available_ip_count \u5b50\u7f51\u53ef\u7528 IP \u6570\u91cf\u3002 Gauge subnet_used_ip_count \u5b50\u7f51\u5df2\u7528 IP \u6570\u91cf\u3002","title":"kube-ovn-controller"},{"location":"reference/metrics/#kube-ovn-cni","text":"kube-ovn-cni \u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Histogram cni_op_latency_seconds CNI \u64cd\u4f5c\u5ef6\u8fdf\u3002 Counter cni_wait_address_seconds_total CNI \u7b49\u5f85\u5730\u5740\u5c31\u7eea\u65f6\u95f4\u3002 Counter cni_wait_connectivity_seconds_total CNI \u7b49\u5f85\u8fde\u63a5\u5c31\u7eea\u65f6\u95f4\u3002 Counter cni_wait_route_seconds_total CNI \u7b49\u5f85\u8def\u7531\u5c31\u7eea\u65f6\u95f4\u3002 Histogram rest_client_request_latency_seconds \u8bf7\u6c42 apiserver \u5ef6\u8fdf\u3002 Counter rest_client_requests_total \u8bf7\u6c42 apiserver \u6570\u91cf\u3002 Counter lists_total API list \u8bf7\u6c42\u6570\u91cf\u3002 Summary list_duration_seconds API list \u8bf7\u6c42\u5ef6\u8fdf\u3002 Summary items_per_list API list \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Counter watches_total API watch \u8bf7\u6c42\u6570\u91cf\u3002 Counter short_watches_total \u77ed\u65f6\u95f4 API watch \u8bf7\u6c42\u6570\u91cf\u3002 Summary watch_duration_seconds API watch \u6301\u7eed\u65f6\u95f4\u3002 Summary items_per_watch API watch \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Gauge last_resource_version \u6700\u65b0\u7684 resource version\u3002 Histogram ovs_client_request_latency_milliseconds \u8bf7\u6c42 OVN \u7ec4\u4ef6\u5ef6\u8fdf\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kube-ovn-cni"},{"location":"reference/ovs-ovn-customized/","text":"\u5bf9\u4e0a\u6e38 OVS/OVN \u4fee\u6539 \u00b6 \u4e0a\u6e38 OVN/OVS \u6700\u521d\u8bbe\u8ba1\u76ee\u6807\u4e3a\u901a\u7528 SDN \u63a7\u5236\u5668\u548c\u6570\u636e\u5e73\u9762\u3002\u7531\u4e8e Kubernetes \u7f51\u7edc\u5b58\u5728\u4e00\u4e9b\u7279\u6b8a\u7684\u7528\u6cd5\uff0c \u5e76\u4e14 Kube-OVN \u53ea\u91cd\u70b9\u4f7f\u7528\u4e86\u90e8\u5206\u529f\u80fd\uff0c\u4e3a\u4e86 \u8fbe\u5230\u66f4\u597d\u7684\u6027\u80fd\u3001\u7a33\u5b9a\u6027\u548c\u7279\u5b9a\u7684\u529f\u80fd\uff0cKube-OVN \u5bf9\u4e0a\u6e38 OVN/OVS \u505a\u4e86\u90e8\u5206\u4fee\u6539\u3002\u7528\u6237\u5982\u679c\u4f7f\u7528\u81ea\u5df1\u7684 OVN/OVS \u914d\u5408 Kube-OVN \u7684\u63a7\u5236\u5668\u8fdb\u884c\u5de5\u4f5c\u65f6\u9700\u8981\u6ce8\u610f \u4e0b\u8ff0\u7684\u6539\u52a8\u53ef\u80fd\u9020\u6210\u7684\u5f71\u54cd\u3002 \u672a\u5408\u5165\u4e0a\u6e38\u4fee\u6539\uff1a 38df6fa3f7 \u8c03\u6574\u9009\u4e3e timer\uff0c\u907f\u514d\u5927\u89c4\u6a21\u96c6\u7fa4\u9009\u4e3e\u6296\u52a8\u3002 d4888c4e75 \u6dfb\u52a0 fdb \u66f4\u65b0\u65e5\u5fd7\u3002 d4888c4e75 \u4fee\u590d hairpin \u73af\u5883\u4e0b fdb \u5b66\u4e60\u9519\u8bef\u7684\u95ee\u9898\u3002 9a81b91368 \u4e3a ovsdb-tool \u7684 join-cluster \u5b50\u547d\u4ee4\u6dfb\u52a0 Server ID \u53c2\u6570\u3002 62d4969877 \u4fee\u590d\u5f00\u542f SSL \u540e OVSDB \u76d1\u542c\u5730\u5740\u9519\u8bef\u7684\u95ee\u9898\u3002 0700cb90f9 \u76ee\u7684\u5730\u5740\u975e Service \u6d41\u91cf\u7ed5\u8fc7 conntrack \u4ee5\u63d0\u9ad8\u7279\u5b9a\u6570\u636e\u94fe\u8def\u6027\u80fd\u3002 c48049a64f ECMP \u7b97\u6cd5\u7531 dp_hash \u8c03\u6574\u4e3a hash\uff0c\u907f\u514d\u90e8\u5206\u5185\u6838\u51fa\u73b0\u7684\u54c8\u5e0c\u9519\u8bef\u95ee\u9898\u3002 64383c14a9 \u4fee\u590d Windows \u4e0b\u5185\u6838 Crash \u95ee\u9898\u3002 08a95db2ca \u652f\u6301 Windows \u4e0b\u7684 github action \u6784\u5efa\u3002 680e77a190 Windows \u4e0b\u9ed8\u8ba4\u4f7f\u7528 tcp \u76d1\u542c\u3002 05e57b3227 \u652f\u6301 Windows \u7f16\u8bd1\u3002 b3801ecb73 \u4fee\u6539\u6e90\u8def\u7531\u7684\u4f18\u5148\u7ea7\u3002 977e569539 \u4fee\u590d Underlay \u6a21\u5f0f\u4e0b Pod \u6570\u91cf\u8fc7\u591a\u5bfc\u81f4 OVS \u6d41\u8868 resubmit \u6b21\u6570\u8d85\u8fc7\u4e0a\u9650\u7684\u95ee\u9898\u3002 45a4a22161 ovn-nbctl\uff1avips \u4e3a\u7a7a\u65f6\u4e0d\u5220\u9664 Load Balancer\u3002 540592b9ff DNAT \u540e\u66ff\u6362 Mac \u5730\u5740\u4e3a\u76ee\u6807\u5730\u5740\uff0c\u51cf\u5c11\u989d\u5916\u6027\u80fd\u5f00\u9500\u3002 10972d9632 \u4fee\u590d vswitchd ofport_usage \u5185\u5b58\u6cc4\u9732\u3002 \u5df2\u5408\u5165\u4e0a\u6e38\u4fee\u6539\uff1a 20626ea909 \u7ec4\u64ad\u6d41\u91cf\u7ed5\u8fc7 LB \u548c ACL \u5904\u7406\u9636\u6bb5\uff0c\u4ee5\u63d0\u9ad8\u7279\u5b9a\u6570\u636e\u94fe\u8def\u6027\u80fd\u3002 a2d9ff3ccd Deb \u6784\u5efa\u589e\u52a0\u7f16\u8bd1\u4f18\u5316\u9009\u9879\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5bf9\u4e0a\u6e38 OVS/OVN \u4fee\u6539"},{"location":"reference/ovs-ovn-customized/#ovsovn","text":"\u4e0a\u6e38 OVN/OVS \u6700\u521d\u8bbe\u8ba1\u76ee\u6807\u4e3a\u901a\u7528 SDN \u63a7\u5236\u5668\u548c\u6570\u636e\u5e73\u9762\u3002\u7531\u4e8e Kubernetes \u7f51\u7edc\u5b58\u5728\u4e00\u4e9b\u7279\u6b8a\u7684\u7528\u6cd5\uff0c \u5e76\u4e14 Kube-OVN \u53ea\u91cd\u70b9\u4f7f\u7528\u4e86\u90e8\u5206\u529f\u80fd\uff0c\u4e3a\u4e86 \u8fbe\u5230\u66f4\u597d\u7684\u6027\u80fd\u3001\u7a33\u5b9a\u6027\u548c\u7279\u5b9a\u7684\u529f\u80fd\uff0cKube-OVN \u5bf9\u4e0a\u6e38 OVN/OVS \u505a\u4e86\u90e8\u5206\u4fee\u6539\u3002\u7528\u6237\u5982\u679c\u4f7f\u7528\u81ea\u5df1\u7684 OVN/OVS \u914d\u5408 Kube-OVN \u7684\u63a7\u5236\u5668\u8fdb\u884c\u5de5\u4f5c\u65f6\u9700\u8981\u6ce8\u610f \u4e0b\u8ff0\u7684\u6539\u52a8\u53ef\u80fd\u9020\u6210\u7684\u5f71\u54cd\u3002 \u672a\u5408\u5165\u4e0a\u6e38\u4fee\u6539\uff1a 38df6fa3f7 \u8c03\u6574\u9009\u4e3e timer\uff0c\u907f\u514d\u5927\u89c4\u6a21\u96c6\u7fa4\u9009\u4e3e\u6296\u52a8\u3002 d4888c4e75 \u6dfb\u52a0 fdb \u66f4\u65b0\u65e5\u5fd7\u3002 d4888c4e75 \u4fee\u590d hairpin \u73af\u5883\u4e0b fdb \u5b66\u4e60\u9519\u8bef\u7684\u95ee\u9898\u3002 9a81b91368 \u4e3a ovsdb-tool \u7684 join-cluster \u5b50\u547d\u4ee4\u6dfb\u52a0 Server ID \u53c2\u6570\u3002 62d4969877 \u4fee\u590d\u5f00\u542f SSL \u540e OVSDB \u76d1\u542c\u5730\u5740\u9519\u8bef\u7684\u95ee\u9898\u3002 0700cb90f9 \u76ee\u7684\u5730\u5740\u975e Service \u6d41\u91cf\u7ed5\u8fc7 conntrack \u4ee5\u63d0\u9ad8\u7279\u5b9a\u6570\u636e\u94fe\u8def\u6027\u80fd\u3002 c48049a64f ECMP \u7b97\u6cd5\u7531 dp_hash \u8c03\u6574\u4e3a hash\uff0c\u907f\u514d\u90e8\u5206\u5185\u6838\u51fa\u73b0\u7684\u54c8\u5e0c\u9519\u8bef\u95ee\u9898\u3002 64383c14a9 \u4fee\u590d Windows \u4e0b\u5185\u6838 Crash \u95ee\u9898\u3002 08a95db2ca \u652f\u6301 Windows \u4e0b\u7684 github action \u6784\u5efa\u3002 680e77a190 Windows \u4e0b\u9ed8\u8ba4\u4f7f\u7528 tcp \u76d1\u542c\u3002 05e57b3227 \u652f\u6301 Windows \u7f16\u8bd1\u3002 b3801ecb73 \u4fee\u6539\u6e90\u8def\u7531\u7684\u4f18\u5148\u7ea7\u3002 977e569539 \u4fee\u590d Underlay \u6a21\u5f0f\u4e0b Pod \u6570\u91cf\u8fc7\u591a\u5bfc\u81f4 OVS \u6d41\u8868 resubmit \u6b21\u6570\u8d85\u8fc7\u4e0a\u9650\u7684\u95ee\u9898\u3002 45a4a22161 ovn-nbctl\uff1avips \u4e3a\u7a7a\u65f6\u4e0d\u5220\u9664 Load Balancer\u3002 540592b9ff DNAT \u540e\u66ff\u6362 Mac \u5730\u5740\u4e3a\u76ee\u6807\u5730\u5740\uff0c\u51cf\u5c11\u989d\u5916\u6027\u80fd\u5f00\u9500\u3002 10972d9632 \u4fee\u590d vswitchd ofport_usage \u5185\u5b58\u6cc4\u9732\u3002 \u5df2\u5408\u5165\u4e0a\u6e38\u4fee\u6539\uff1a 20626ea909 \u7ec4\u64ad\u6d41\u91cf\u7ed5\u8fc7 LB \u548c ACL \u5904\u7406\u9636\u6bb5\uff0c\u4ee5\u63d0\u9ad8\u7279\u5b9a\u6570\u636e\u94fe\u8def\u6027\u80fd\u3002 a2d9ff3ccd Deb \u6784\u5efa\u589e\u52a0\u7f16\u8bd1\u4f18\u5316\u9009\u9879\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5bf9\u4e0a\u6e38 OVS/OVN \u4fee\u6539"},{"location":"reference/tunnel-protocol/","text":"\u96a7\u9053\u534f\u8bae\u8bf4\u660e \u00b6 Kube-OVN \u4f7f\u7528 OVN/OVS \u4f5c\u4e3a\u6570\u636e\u5e73\u9762\u5b9e\u73b0\uff0c\u76ee\u524d\u652f\u6301 Geneve \uff0c Vxlan \u548c STT \u4e09\u79cd\u96a7\u9053\u5c01\u88c5\u534f\u8bae\u3002 \u8fd9\u4e09\u79cd\u534f\u8bae\u5728\u529f\u80fd\uff0c\u6027\u80fd\u548c\u6613\u7528\u6027\u4e0a\u5b58\u5728\u7740\u533a\u522b\uff0c\u672c\u6587\u6863\u5c06\u4ecb\u7ecd\u4e09\u79cd\u534f\u8bae\u5728\u4f7f\u7528\u4e2d\u7684\u5dee\u5f02\uff0c\u7528\u6237\u53ef\u6839\u636e\u81ea\u5df1\u7684\u60c5\u51b5\u8fdb\u884c\u9009\u62e9\u3002 Geneve \u00b6 Geneve \u534f\u8bae\u4e3a Kube-OVN \u90e8\u7f72\u65f6\u9009\u62e9\u7684\u9ed8\u8ba4\u96a7\u9053\u534f\u8bae\uff0c\u4e5f\u662f OVN \u9ed8\u8ba4\u63a8\u8350\u7684\u96a7\u9053\u534f\u8bae\u3002\u8be5\u534f\u8bae\u5728\u5185\u6838\u4e2d\u5f97\u5230\u4e86\u5e7f\u6cdb\u7684\u652f\u6301\uff0c \u5e76\u53ef\u4ee5\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684\u901a\u7528 Offload \u80fd\u529b\u8fdb\u884c\u52a0\u901f\u3002\u7531\u4e8e Geneve \u6709\u7740\u53ef\u53d8\u957f\u7684\u5934\u90e8\uff0c\u53ef\u4ee5\u4f7f\u7528 24bit \u7a7a\u95f4\u6765\u6807\u5fd7\u4e0d\u540c\u7684 datapath \u7528\u6237\u53ef\u4ee5\u521b\u5efa\u66f4\u591a\u6570\u91cf\u7684\u865a\u62df\u7f51\u7edc\u3002 \u5982\u679c\u4f7f\u7528 Mellanox \u6216\u82af\u542f\u6e90\u7684\u667a\u80fd\u7f51\u5361 OVS \u5378\u8f7d\uff0c Geneve \u9700\u8981\u8f83\u9ad8\u7248\u672c\u7684\u5185\u6838\u652f\u6301\uff0c\u9700\u8981\u9009\u62e9 5.4 \u4ee5\u4e0a\u7684\u4e0a\u6e38\u5185\u6838\uff0c \u6216 backport \u4e86\u8be5\u529f\u80fd\u7684\u5176\u4ed6\u517c\u5bb9\u5185\u6838\u3002 \u7531\u4e8e\u4f7f\u7528 UDP \u8fdb\u884c\u5c01\u88c5\uff0c\u8be5\u534f\u8bae\u5728\u5904\u7406 TCP over UDP \u65f6\u4e0d\u80fd\u5f88\u597d\u7684\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684 TCP \u76f8\u5173\u5378\u8f7d\uff0c\u5728\u5904\u7406\u5927\u5305\u65f6\u4f1a\u6d88\u8017\u8f83\u591a CPU \u8d44\u6e90\u3002 Vxlan \u00b6 Vxlan \u4e3a\u4e0a\u6e38 OVN \u8fd1\u671f\u652f\u6301\u7684\u534f\u8bae\uff0c\u8be5\u534f\u8bae\u5728\u5185\u6838\u4e2d\u5f97\u5230\u4e86\u5e7f\u6cdb\u7684\u652f\u6301\uff0c \u5e76\u53ef\u4ee5\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684\u901a\u7528 Offload \u80fd\u529b\u8fdb\u884c\u52a0\u901f\u3002 \u7531\u4e8e\u8be5\u534f\u8bae\u5934\u90e8\u957f\u5ea6\u6709\u9650\uff0c\u5e76\u4e14 OVN \u9700\u8981\u4f7f\u7528\u989d\u5916\u7684\u7a7a\u95f4\u8fdb\u884c\u7f16\u6392\uff0cdatapath \u7684\u6570\u91cf\u5b58\u5728\u9650\u5236\uff0c\u6700\u591a\u53ea\u80fd\u521b\u5efa 4096 \u4e2a datapath\uff0c \u6bcf\u4e2a datapath \u4e0b\u6700\u591a 4096 \u4e2a\u7aef\u53e3\u3002\u540c\u65f6\u7531\u4e8e\u7a7a\u95f4\u6709\u9650\uff0c\u57fa\u4e8e inport \u7684 ACL \u6ca1\u6709\u8fdb\u884c\u652f\u6301\u3002 \u5982\u679c\u4f7f\u7528 Mellanox \u6216\u82af\u542f\u6e90\u7684\u667a\u80fd\u7f51\u5361 OVS \u5378\u8f7d\uff0c Vxlan \u7684\u5378\u8f7d\u5728\u5e38\u89c1\u5185\u6838\u4e2d\u5df2\u83b7\u5f97\u652f\u6301\u3002 \u7531\u4e8e\u4f7f\u7528 UDP \u8fdb\u884c\u5c01\u88c5\uff0c\u8be5\u534f\u8bae\u5728\u5904\u7406 TCP over UDP \u65f6\u4e0d\u80fd\u5f88\u597d\u7684\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684 TCP \u76f8\u5173\u5378\u8f7d\uff0c\u5728\u5904\u7406\u5927\u5305\u65f6\u4f1a\u6d88\u8017\u8f83\u591a CPU \u8d44\u6e90\u3002 STT \u00b6 STT \u534f\u8bae\u4e3a OVN \u8f83\u65e9\u652f\u6301\u7684\u96a7\u9053\u534f\u8bae\uff0c\u8be5\u534f\u8bae\u4f7f\u7528\u7c7b TCP \u7684\u5934\u90e8\uff0c\u53ef\u4ee5\u5145\u5206\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u901a\u7528\u7684 TCP \u5378\u8f7d\u80fd\u529b\uff0c\u5927\u5e45\u63d0\u5347 TCP \u7684\u541e\u5410\u91cf\u3002\u540c\u65f6\u8be5\u534f\u8bae\u5934\u90e8\u8f83\u957f\u53ef\u652f\u6301\u5b8c\u6574\u7684 OVN \u80fd\u529b\u548c\u5927\u89c4\u6a21\u7684 datapath\u3002 \u8be5\u534f\u8bae\u672a\u5728\u5185\u6838\u4e2d\u652f\u6301\uff0c\u82e5\u8981\u4f7f\u7528\u9700\u8981\u989d\u5916\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\uff0c\u5e76\u5728\u5347\u7ea7\u5185\u6838\u65f6\u5bf9\u5e94\u518d\u6b21\u7f16\u8bd1\u65b0\u7248\u672c\u5185\u6838\u6a21\u5757\u3002 \u8be5\u534f\u8bae\u76ee\u524d\u672a\u88ab\u667a\u80fd\u7f51\u5361\u652f\u6301\uff0c\u65e0\u6cd5\u4f7f\u7528 OVS \u7684\u5378\u8f7d\u80fd\u529b\u3002 \u53c2\u8003\u8d44\u6599 \u00b6 https://ipwithease.com/vxlan-vs-geneve-understand-the-difference/ OVN FAQ What is Geneve \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u96a7\u9053\u534f\u8bae\u8bf4\u660e"},{"location":"reference/tunnel-protocol/#_1","text":"Kube-OVN \u4f7f\u7528 OVN/OVS \u4f5c\u4e3a\u6570\u636e\u5e73\u9762\u5b9e\u73b0\uff0c\u76ee\u524d\u652f\u6301 Geneve \uff0c Vxlan \u548c STT \u4e09\u79cd\u96a7\u9053\u5c01\u88c5\u534f\u8bae\u3002 \u8fd9\u4e09\u79cd\u534f\u8bae\u5728\u529f\u80fd\uff0c\u6027\u80fd\u548c\u6613\u7528\u6027\u4e0a\u5b58\u5728\u7740\u533a\u522b\uff0c\u672c\u6587\u6863\u5c06\u4ecb\u7ecd\u4e09\u79cd\u534f\u8bae\u5728\u4f7f\u7528\u4e2d\u7684\u5dee\u5f02\uff0c\u7528\u6237\u53ef\u6839\u636e\u81ea\u5df1\u7684\u60c5\u51b5\u8fdb\u884c\u9009\u62e9\u3002","title":"\u96a7\u9053\u534f\u8bae\u8bf4\u660e"},{"location":"reference/tunnel-protocol/#geneve","text":"Geneve \u534f\u8bae\u4e3a Kube-OVN \u90e8\u7f72\u65f6\u9009\u62e9\u7684\u9ed8\u8ba4\u96a7\u9053\u534f\u8bae\uff0c\u4e5f\u662f OVN \u9ed8\u8ba4\u63a8\u8350\u7684\u96a7\u9053\u534f\u8bae\u3002\u8be5\u534f\u8bae\u5728\u5185\u6838\u4e2d\u5f97\u5230\u4e86\u5e7f\u6cdb\u7684\u652f\u6301\uff0c \u5e76\u53ef\u4ee5\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684\u901a\u7528 Offload \u80fd\u529b\u8fdb\u884c\u52a0\u901f\u3002\u7531\u4e8e Geneve \u6709\u7740\u53ef\u53d8\u957f\u7684\u5934\u90e8\uff0c\u53ef\u4ee5\u4f7f\u7528 24bit \u7a7a\u95f4\u6765\u6807\u5fd7\u4e0d\u540c\u7684 datapath \u7528\u6237\u53ef\u4ee5\u521b\u5efa\u66f4\u591a\u6570\u91cf\u7684\u865a\u62df\u7f51\u7edc\u3002 \u5982\u679c\u4f7f\u7528 Mellanox \u6216\u82af\u542f\u6e90\u7684\u667a\u80fd\u7f51\u5361 OVS \u5378\u8f7d\uff0c Geneve \u9700\u8981\u8f83\u9ad8\u7248\u672c\u7684\u5185\u6838\u652f\u6301\uff0c\u9700\u8981\u9009\u62e9 5.4 \u4ee5\u4e0a\u7684\u4e0a\u6e38\u5185\u6838\uff0c \u6216 backport \u4e86\u8be5\u529f\u80fd\u7684\u5176\u4ed6\u517c\u5bb9\u5185\u6838\u3002 \u7531\u4e8e\u4f7f\u7528 UDP \u8fdb\u884c\u5c01\u88c5\uff0c\u8be5\u534f\u8bae\u5728\u5904\u7406 TCP over UDP \u65f6\u4e0d\u80fd\u5f88\u597d\u7684\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684 TCP \u76f8\u5173\u5378\u8f7d\uff0c\u5728\u5904\u7406\u5927\u5305\u65f6\u4f1a\u6d88\u8017\u8f83\u591a CPU \u8d44\u6e90\u3002","title":"Geneve"},{"location":"reference/tunnel-protocol/#vxlan","text":"Vxlan \u4e3a\u4e0a\u6e38 OVN \u8fd1\u671f\u652f\u6301\u7684\u534f\u8bae\uff0c\u8be5\u534f\u8bae\u5728\u5185\u6838\u4e2d\u5f97\u5230\u4e86\u5e7f\u6cdb\u7684\u652f\u6301\uff0c \u5e76\u53ef\u4ee5\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684\u901a\u7528 Offload \u80fd\u529b\u8fdb\u884c\u52a0\u901f\u3002 \u7531\u4e8e\u8be5\u534f\u8bae\u5934\u90e8\u957f\u5ea6\u6709\u9650\uff0c\u5e76\u4e14 OVN \u9700\u8981\u4f7f\u7528\u989d\u5916\u7684\u7a7a\u95f4\u8fdb\u884c\u7f16\u6392\uff0cdatapath \u7684\u6570\u91cf\u5b58\u5728\u9650\u5236\uff0c\u6700\u591a\u53ea\u80fd\u521b\u5efa 4096 \u4e2a datapath\uff0c \u6bcf\u4e2a datapath \u4e0b\u6700\u591a 4096 \u4e2a\u7aef\u53e3\u3002\u540c\u65f6\u7531\u4e8e\u7a7a\u95f4\u6709\u9650\uff0c\u57fa\u4e8e inport \u7684 ACL \u6ca1\u6709\u8fdb\u884c\u652f\u6301\u3002 \u5982\u679c\u4f7f\u7528 Mellanox \u6216\u82af\u542f\u6e90\u7684\u667a\u80fd\u7f51\u5361 OVS \u5378\u8f7d\uff0c Vxlan \u7684\u5378\u8f7d\u5728\u5e38\u89c1\u5185\u6838\u4e2d\u5df2\u83b7\u5f97\u652f\u6301\u3002 \u7531\u4e8e\u4f7f\u7528 UDP \u8fdb\u884c\u5c01\u88c5\uff0c\u8be5\u534f\u8bae\u5728\u5904\u7406 TCP over UDP \u65f6\u4e0d\u80fd\u5f88\u597d\u7684\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684 TCP \u76f8\u5173\u5378\u8f7d\uff0c\u5728\u5904\u7406\u5927\u5305\u65f6\u4f1a\u6d88\u8017\u8f83\u591a CPU \u8d44\u6e90\u3002","title":"Vxlan"},{"location":"reference/tunnel-protocol/#stt","text":"STT \u534f\u8bae\u4e3a OVN \u8f83\u65e9\u652f\u6301\u7684\u96a7\u9053\u534f\u8bae\uff0c\u8be5\u534f\u8bae\u4f7f\u7528\u7c7b TCP \u7684\u5934\u90e8\uff0c\u53ef\u4ee5\u5145\u5206\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u901a\u7528\u7684 TCP \u5378\u8f7d\u80fd\u529b\uff0c\u5927\u5e45\u63d0\u5347 TCP \u7684\u541e\u5410\u91cf\u3002\u540c\u65f6\u8be5\u534f\u8bae\u5934\u90e8\u8f83\u957f\u53ef\u652f\u6301\u5b8c\u6574\u7684 OVN \u80fd\u529b\u548c\u5927\u89c4\u6a21\u7684 datapath\u3002 \u8be5\u534f\u8bae\u672a\u5728\u5185\u6838\u4e2d\u652f\u6301\uff0c\u82e5\u8981\u4f7f\u7528\u9700\u8981\u989d\u5916\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\uff0c\u5e76\u5728\u5347\u7ea7\u5185\u6838\u65f6\u5bf9\u5e94\u518d\u6b21\u7f16\u8bd1\u65b0\u7248\u672c\u5185\u6838\u6a21\u5757\u3002 \u8be5\u534f\u8bae\u76ee\u524d\u672a\u88ab\u667a\u80fd\u7f51\u5361\u652f\u6301\uff0c\u65e0\u6cd5\u4f7f\u7528 OVS \u7684\u5378\u8f7d\u80fd\u529b\u3002","title":"STT"},{"location":"reference/tunnel-protocol/#_2","text":"https://ipwithease.com/vxlan-vs-geneve-understand-the-difference/ OVN FAQ What is Geneve \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u53c2\u8003\u8d44\u6599"},{"location":"reference/underlay-topology/","text":"Underlay \u6d41\u91cf\u62d3\u6251 \u00b6 \u672c\u6587\u6863\u4ecb\u7ecd Underlay \u6a21\u5f0f\u4e0b\u6d41\u91cf\u5728\u4e0d\u540c\u60c5\u51b5\u4e0b\u7684\u8f6c\u53d1\u8def\u5f84\u3002 \u540c\u8282\u70b9\u540c\u5b50\u7f51 \u00b6 \u5185\u90e8\u903b\u8f91\u4ea4\u6362\u673a\u76f4\u63a5\u4ea4\u6362\u6570\u636e\u5305\uff0c\u4e0d\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\u3002 \u8de8\u8282\u70b9\u540c\u5b50\u7f51 \u00b6 \u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u4ea4\u6362\u673a\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u8fdb\u884c\u4ea4\u6362\u3002 \u540c\u8282\u70b9\u4e0d\u540c\u5b50\u7f51 \u00b6 \u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u6b64\u5904 br-provider-1 \u548c br-provider-2 \u53ef\u4ee5\u662f\u540c\u4e00\u4e2a OVS \u7f51\u6865\uff0c\u5373\u591a\u4e2a\u4e0d\u540c\u5b50\u7f51\u53ef\u4ee5\u4f7f\u7528\u540c\u4e00\u4e2a Provider Network\u3002 \u8de8\u8282\u70b9\u4e0d\u540c\u5b50\u7f51 \u00b6 \u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u8bbf\u95ee\u5916\u90e8 \u00b6 \u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u8282\u70b9\u4e0e Pod \u4e4b\u95f4\u7684\u901a\u4fe1\u5927\u4f53\u4e0a\u4e5f\u9075\u5faa\u6b64\u903b\u8f91\u3002 \u65e0 Vlan Tag \u4e0b\u603b\u89c8 \u00b6 \u591a VLAN \u603b\u89c8 \u00b6 Pod \u8bbf\u95ee Service IP \u00b6 Kube-OVN \u4e3a\u6bcf\u4e2a Kubernetes Service \u5728\u6bcf\u4e2a\u5b50\u7f51\u7684\u903b\u8f91\u4ea4\u6362\u673a\u4e0a\u914d\u7f6e\u4e86\u8d1f\u8f7d\u5747\u8861\u3002 \u5f53 Pod \u901a\u8fc7\u8bbf\u95ee Service IP \u8bbf\u95ee\u5176\u5b83 Pod \u65f6\uff0c\u4f1a\u6784\u9020\u4e00\u4e2a\u76ee\u7684\u5730\u5740\u4e3a Service IP\u3001\u76ee\u7684 MAC \u5730\u5740\u4e3a\u7f51\u5173 MAC \u5730\u5740\u7684\u7f51\u7edc\u5305\u3002 \u7f51\u7edc\u5305\u8fdb\u5165\u903b\u8f91\u4ea4\u6362\u673a\u540e\uff0c\u8d1f\u8f7d\u5747\u8861\u4f1a\u5bf9\u7f51\u7edc\u5305\u8fdb\u884c\u62e6\u622a\u548c DNAT \u5904\u7406\uff0c\u5c06\u76ee\u7684 IP \u548c\u7aef\u53e3\u4fee\u6539\u4e3a Service \u5bf9\u5e94\u7684\u67d0\u4e2a Endpoint \u7684 IP \u548c\u7aef\u53e3\u3002 \u7531\u4e8e\u903b\u8f91\u4ea4\u6362\u673a\u5e76\u672a\u4fee\u6539\u7f51\u7edc\u5305\u7684\u4e8c\u5c42\u76ee\u7684 MAC \u5730\u5740\uff0c\u7f51\u7edc\u5305\u5728\u8fdb\u5165\u5916\u90e8\u4ea4\u6362\u673a\u540e\u4ecd\u7136\u4f1a\u9001\u5230\u5916\u90e8\u7f51\u5173\uff0c\u6b64\u65f6\u9700\u8981\u5916\u90e8\u7f51\u5173\u5bf9\u7f51\u7edc\u5305\u8fdb\u884c\u8f6c\u53d1\u3002 Service \u540e\u7aef\u4e3a\u540c\u8282\u70b9\u540c\u5b50\u7f51 Pod \u00b6 Service \u540e\u7aef\u4e3a\u540c\u8282\u70b9\u4e0d\u540c\u5b50\u7f51 Pod \u00b6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Underlay \u6d41\u91cf\u62d3\u6251"},{"location":"reference/underlay-topology/#underlay","text":"\u672c\u6587\u6863\u4ecb\u7ecd Underlay \u6a21\u5f0f\u4e0b\u6d41\u91cf\u5728\u4e0d\u540c\u60c5\u51b5\u4e0b\u7684\u8f6c\u53d1\u8def\u5f84\u3002","title":"Underlay \u6d41\u91cf\u62d3\u6251"},{"location":"reference/underlay-topology/#_1","text":"\u5185\u90e8\u903b\u8f91\u4ea4\u6362\u673a\u76f4\u63a5\u4ea4\u6362\u6570\u636e\u5305\uff0c\u4e0d\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\u3002","title":"\u540c\u8282\u70b9\u540c\u5b50\u7f51"},{"location":"reference/underlay-topology/#_2","text":"\u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u4ea4\u6362\u673a\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u8fdb\u884c\u4ea4\u6362\u3002","title":"\u8de8\u8282\u70b9\u540c\u5b50\u7f51"},{"location":"reference/underlay-topology/#_3","text":"\u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u6b64\u5904 br-provider-1 \u548c br-provider-2 \u53ef\u4ee5\u662f\u540c\u4e00\u4e2a OVS \u7f51\u6865\uff0c\u5373\u591a\u4e2a\u4e0d\u540c\u5b50\u7f51\u53ef\u4ee5\u4f7f\u7528\u540c\u4e00\u4e2a Provider Network\u3002","title":"\u540c\u8282\u70b9\u4e0d\u540c\u5b50\u7f51"},{"location":"reference/underlay-topology/#_4","text":"\u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002","title":"\u8de8\u8282\u70b9\u4e0d\u540c\u5b50\u7f51"},{"location":"reference/underlay-topology/#_5","text":"\u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u8282\u70b9\u4e0e Pod \u4e4b\u95f4\u7684\u901a\u4fe1\u5927\u4f53\u4e0a\u4e5f\u9075\u5faa\u6b64\u903b\u8f91\u3002","title":"\u8bbf\u95ee\u5916\u90e8"},{"location":"reference/underlay-topology/#vlan-tag","text":"","title":"\u65e0 Vlan Tag \u4e0b\u603b\u89c8"},{"location":"reference/underlay-topology/#vlan","text":"","title":"\u591a VLAN \u603b\u89c8"},{"location":"reference/underlay-topology/#pod-service-ip","text":"Kube-OVN \u4e3a\u6bcf\u4e2a Kubernetes Service \u5728\u6bcf\u4e2a\u5b50\u7f51\u7684\u903b\u8f91\u4ea4\u6362\u673a\u4e0a\u914d\u7f6e\u4e86\u8d1f\u8f7d\u5747\u8861\u3002 \u5f53 Pod \u901a\u8fc7\u8bbf\u95ee Service IP \u8bbf\u95ee\u5176\u5b83 Pod \u65f6\uff0c\u4f1a\u6784\u9020\u4e00\u4e2a\u76ee\u7684\u5730\u5740\u4e3a Service IP\u3001\u76ee\u7684 MAC \u5730\u5740\u4e3a\u7f51\u5173 MAC \u5730\u5740\u7684\u7f51\u7edc\u5305\u3002 \u7f51\u7edc\u5305\u8fdb\u5165\u903b\u8f91\u4ea4\u6362\u673a\u540e\uff0c\u8d1f\u8f7d\u5747\u8861\u4f1a\u5bf9\u7f51\u7edc\u5305\u8fdb\u884c\u62e6\u622a\u548c DNAT \u5904\u7406\uff0c\u5c06\u76ee\u7684 IP \u548c\u7aef\u53e3\u4fee\u6539\u4e3a Service \u5bf9\u5e94\u7684\u67d0\u4e2a Endpoint \u7684 IP \u548c\u7aef\u53e3\u3002 \u7531\u4e8e\u903b\u8f91\u4ea4\u6362\u673a\u5e76\u672a\u4fee\u6539\u7f51\u7edc\u5305\u7684\u4e8c\u5c42\u76ee\u7684 MAC \u5730\u5740\uff0c\u7f51\u7edc\u5305\u5728\u8fdb\u5165\u5916\u90e8\u4ea4\u6362\u673a\u540e\u4ecd\u7136\u4f1a\u9001\u5230\u5916\u90e8\u7f51\u5173\uff0c\u6b64\u65f6\u9700\u8981\u5916\u90e8\u7f51\u5173\u5bf9\u7f51\u7edc\u5305\u8fdb\u884c\u8f6c\u53d1\u3002","title":"Pod \u8bbf\u95ee Service IP"},{"location":"reference/underlay-topology/#service-pod","text":"","title":"Service \u540e\u7aef\u4e3a\u540c\u8282\u70b9\u540c\u5b50\u7f51 Pod"},{"location":"reference/underlay-topology/#service-pod_1","text":"\u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Service \u540e\u7aef\u4e3a\u540c\u8282\u70b9\u4e0d\u540c\u5b50\u7f51 Pod"},{"location":"start/one-step-install/","text":"\u4e00\u952e\u5b89\u88c5 \u00b6 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684 Kube-OVN \u5bb9\u5668\u7f51\u7edc\uff0c\u9ed8\u8ba4\u90e8\u7f72\u4e3a Overlay \u7c7b\u578b\u7f51\u7edc\u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u652f\u6301 Helm Chart \u5b89\u88c5\uff0c\u9ed8\u8ba4\u90e8\u7f72\u4e3a Overlay \u7c7b\u578b\u7f51\u7edc\u3002 \u5982\u679c\u9ed8\u8ba4\u7f51\u7edc\u9700\u8981\u642d\u5efa Underlay/Vlan \u7f51\u7edc\uff0c\u8bf7\u53c2\u8003 Underlay \u7f51\u7edc\u652f\u6301 \u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003 \u51c6\u5907\u5de5\u4f5c \u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u811a\u672c\u5b89\u88c5 \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c \u00b6 \u6211\u4eec\u63a8\u8350\u5728\u751f\u4ea7\u73af\u5883\u4f7f\u7528\u7a33\u5b9a\u7684 release \u7248\u672c\uff0c\u8bf7\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u7a33\u5b9a\u7248\u672c\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u5982\u679c\u5bf9 master \u5206\u652f\u7684\u6700\u65b0\u529f\u80fd\u611f\u5174\u8da3\uff0c\u60f3\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5f00\u53d1\u7248\u672c\u90e8\u7f72\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u914d\u7f6e\u53c2\u6570 \u00b6 \u4f7f\u7528\u7f16\u8f91\u5668\u6253\u5f00\u811a\u672c\uff0c\u5e76\u4fee\u6539\u4e0b\u5217\u53d8\u91cf\u4e3a\u9884\u671f\u503c\uff1a REGISTRY = \"kubeovn\" # \u955c\u50cf\u4ed3\u5e93\u5730\u5740 VERSION = \"v1.12.0\" # \u955c\u50cf\u7248\u672c/Tag POD_CIDR = \"10.16.0.0/16\" # \u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e0d\u8981\u548c SVC/NODE/JOIN CIDR \u91cd\u53e0 SVC_CIDR = \"10.96.0.0/12\" # \u9700\u8981\u548c apiserver \u7684 service-cluster-ip-range \u4fdd\u6301\u4e00\u81f4 JOIN_CIDR = \"100.64.0.0/16\" # Pod \u548c\u4e3b\u673a\u901a\u4fe1\u7f51\u7edc CIDR\uff0c\u4e0d\u8981\u548c SVC/NODE/POD CIDR \u91cd\u53e0 LABEL = \"node-role.kubernetes.io/master\" # \u90e8\u7f72 OVN DB \u8282\u70b9\u7684\u6807\u7b7e IFACE = \"\" # \u5bb9\u5668\u7f51\u7edc\u6240\u4f7f\u7528\u7684\u7684\u5bbf\u4e3b\u673a\u7f51\u5361\u540d\uff0c\u5982\u679c\u4e3a\u7a7a\u5219\u4f7f\u7528 Kubernetes \u4e2d\u7684 Node IP \u6240\u5728\u7f51\u5361 TUNNEL_TYPE = \"geneve\" # \u96a7\u9053\u5c01\u88c5\u534f\u8bae\uff0c\u53ef\u9009 geneve, vxlan \u6216 stt\uff0cstt \u9700\u8981\u5355\u72ec\u7f16\u8bd1 ovs \u5185\u6838\u6a21\u5757 \u53ef\u4f7f\u7528\u6b63\u5219\u8868\u8fbe\u5f0f\u6765\u5339\u914d\u7f51\u5361\u540d\uff0c\u4f8b\u5982 IFACE=enp6s0f0,eth.* \u3002 \u6267\u884c\u5b89\u88c5\u811a\u672c \u00b6 bash install.sh \u7b49\u5f85\u5b89\u88c5\u5b8c\u6210\u3002 Helm Chart \u5b89\u88c5 \u00b6 \u7531\u4e8e Kube-OVN \u7684\u5b89\u88c5\uff0c\u9700\u8981\u8bbe\u7f6e\u4e00\u4e9b\u53c2\u6570\uff0c\u56e0\u6b64\u4f7f\u7528 Helm \u5b89\u88c5 Kube-OVN\uff0c\u9700\u8981\u6309\u7167\u4ee5\u4e0b\u6b65\u9aa4\u6267\u884c\u3002 \u67e5\u770b\u8282\u70b9 IP \u5730\u5740 \u00b6 $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-ovn-control-plane NotReady control-plane 20h v1.26.0 172 .18.0.3 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 kube-ovn-worker NotReady <none> 20h v1.26.0 172 .18.0.2 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 \u53bb\u6389\u96c6\u7fa4 master \u8282\u70b9\u6c61\u70b9 \u00b6 $ kubectl taint node kube-ovn-control-plane node-role.kubernetes.io/control-plane:NoSchedule- node/kube-ovn-control-plane untainted \u5982\u679c\u786e\u5b9a\u4e0d\u9700\u8981\u5728 master \u8282\u70b9\u8c03\u5ea6\u4e1a\u52a1 Pod\uff0c\u8fd9\u4e00\u6b65\u53ef\u4ee5\u8df3\u8fc7\u3002 \u7ed9\u8282\u70b9\u6dfb\u52a0 label \u00b6 $ kubectl label node -lbeta.kubernetes.io/os = linux kubernetes.io/os = linux --overwrite node/kube-ovn-control-plane not labeled node/kube-ovn-worker not labeled $ kubectl label node -lnode-role.kubernetes.io/control-plane kube-ovn/role = master --overwrite node/kube-ovn-control-plane labeled # \u4ee5\u4e0b label \u7528\u4e8e dpdk \u955c\u50cf\u7684\u5b89\u88c5\uff0c\u975e dpdk \u60c5\u51b5\uff0c\u53ef\u4ee5\u5ffd\u7565 $ kubectl label node -lovn.kubernetes.io/ovs_dp_type! = userspace ovn.kubernetes.io/ovs_dp_type = kernel --overwrite node/kube-ovn-control-plane labeled node/kube-ovn-worker labeled \u6dfb\u52a0 Helm Repo \u4fe1\u606f \u00b6 $ helm repo add kubeovn https://kubeovn.github.io/kube-ovn/ \"kubeovn\" has been added to your repositories $ helm repo list NAME URL kubeovn https://kubeovn.github.io/kube-ovn/ $ helm search repo kubeovn NAME CHART VERSION APP VERSION DESCRIPTION kubeovn/kube-ovn 0 .1.0 1 .12.0 Helm chart for Kube-OVN \u6267\u884c helm install \u5b89\u88c5 Kube-OVN \u00b6 Node0IP\u3001Node1IP\u3001Node2IP \u53c2\u6570\u5206\u522b\u4e3a\u96c6\u7fa4 master \u8282\u70b9\u7684 IP \u5730\u5740\u3002\u5176\u4ed6\u53c2\u6570\u7684\u8bbe\u7f6e\uff0c\u53ef\u4ee5\u53c2\u8003 values.yaml \u6587\u4ef6\u4e2d\u53d8\u91cf\u5b9a\u4e49\u3002 # \u5355 master \u8282\u70b9\u73af\u5883\u5b89\u88c5 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } # \u4ee5\u4e0a\u8fb9\u7684 node \u4fe1\u606f\u4e3a\u4f8b\uff0c\u6267\u884c\u5b89\u88c5\u547d\u4ee4 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = 172 .18.0.3 NAME: kube-ovn LAST DEPLOYED: Fri Mar 31 12 :43:43 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None # \u9ad8\u53ef\u7528\u96c6\u7fa4\u5b89\u88c5 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } , ${ Node1IP } , ${ Node2IP } , --set replicaCount = 3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4e00\u952e\u5b89\u88c5"},{"location":"start/one-step-install/#_1","text":"Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684 Kube-OVN \u5bb9\u5668\u7f51\u7edc\uff0c\u9ed8\u8ba4\u90e8\u7f72\u4e3a Overlay \u7c7b\u578b\u7f51\u7edc\u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u652f\u6301 Helm Chart \u5b89\u88c5\uff0c\u9ed8\u8ba4\u90e8\u7f72\u4e3a Overlay \u7c7b\u578b\u7f51\u7edc\u3002 \u5982\u679c\u9ed8\u8ba4\u7f51\u7edc\u9700\u8981\u642d\u5efa Underlay/Vlan \u7f51\u7edc\uff0c\u8bf7\u53c2\u8003 Underlay \u7f51\u7edc\u652f\u6301 \u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003 \u51c6\u5907\u5de5\u4f5c \u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002","title":"\u4e00\u952e\u5b89\u88c5"},{"location":"start/one-step-install/#_2","text":"","title":"\u811a\u672c\u5b89\u88c5"},{"location":"start/one-step-install/#_3","text":"\u6211\u4eec\u63a8\u8350\u5728\u751f\u4ea7\u73af\u5883\u4f7f\u7528\u7a33\u5b9a\u7684 release \u7248\u672c\uff0c\u8bf7\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u7a33\u5b9a\u7248\u672c\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u5982\u679c\u5bf9 master \u5206\u652f\u7684\u6700\u65b0\u529f\u80fd\u611f\u5174\u8da3\uff0c\u60f3\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5f00\u53d1\u7248\u672c\u90e8\u7f72\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c"},{"location":"start/one-step-install/#_4","text":"\u4f7f\u7528\u7f16\u8f91\u5668\u6253\u5f00\u811a\u672c\uff0c\u5e76\u4fee\u6539\u4e0b\u5217\u53d8\u91cf\u4e3a\u9884\u671f\u503c\uff1a REGISTRY = \"kubeovn\" # \u955c\u50cf\u4ed3\u5e93\u5730\u5740 VERSION = \"v1.12.0\" # \u955c\u50cf\u7248\u672c/Tag POD_CIDR = \"10.16.0.0/16\" # \u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e0d\u8981\u548c SVC/NODE/JOIN CIDR \u91cd\u53e0 SVC_CIDR = \"10.96.0.0/12\" # \u9700\u8981\u548c apiserver \u7684 service-cluster-ip-range \u4fdd\u6301\u4e00\u81f4 JOIN_CIDR = \"100.64.0.0/16\" # Pod \u548c\u4e3b\u673a\u901a\u4fe1\u7f51\u7edc CIDR\uff0c\u4e0d\u8981\u548c SVC/NODE/POD CIDR \u91cd\u53e0 LABEL = \"node-role.kubernetes.io/master\" # \u90e8\u7f72 OVN DB \u8282\u70b9\u7684\u6807\u7b7e IFACE = \"\" # \u5bb9\u5668\u7f51\u7edc\u6240\u4f7f\u7528\u7684\u7684\u5bbf\u4e3b\u673a\u7f51\u5361\u540d\uff0c\u5982\u679c\u4e3a\u7a7a\u5219\u4f7f\u7528 Kubernetes \u4e2d\u7684 Node IP \u6240\u5728\u7f51\u5361 TUNNEL_TYPE = \"geneve\" # \u96a7\u9053\u5c01\u88c5\u534f\u8bae\uff0c\u53ef\u9009 geneve, vxlan \u6216 stt\uff0cstt \u9700\u8981\u5355\u72ec\u7f16\u8bd1 ovs \u5185\u6838\u6a21\u5757 \u53ef\u4f7f\u7528\u6b63\u5219\u8868\u8fbe\u5f0f\u6765\u5339\u914d\u7f51\u5361\u540d\uff0c\u4f8b\u5982 IFACE=enp6s0f0,eth.* \u3002","title":"\u4fee\u6539\u914d\u7f6e\u53c2\u6570"},{"location":"start/one-step-install/#_5","text":"bash install.sh \u7b49\u5f85\u5b89\u88c5\u5b8c\u6210\u3002","title":"\u6267\u884c\u5b89\u88c5\u811a\u672c"},{"location":"start/one-step-install/#helm-chart","text":"\u7531\u4e8e Kube-OVN \u7684\u5b89\u88c5\uff0c\u9700\u8981\u8bbe\u7f6e\u4e00\u4e9b\u53c2\u6570\uff0c\u56e0\u6b64\u4f7f\u7528 Helm \u5b89\u88c5 Kube-OVN\uff0c\u9700\u8981\u6309\u7167\u4ee5\u4e0b\u6b65\u9aa4\u6267\u884c\u3002","title":"Helm Chart \u5b89\u88c5"},{"location":"start/one-step-install/#ip","text":"$ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-ovn-control-plane NotReady control-plane 20h v1.26.0 172 .18.0.3 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 kube-ovn-worker NotReady <none> 20h v1.26.0 172 .18.0.2 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9","title":"\u67e5\u770b\u8282\u70b9 IP \u5730\u5740"},{"location":"start/one-step-install/#master","text":"$ kubectl taint node kube-ovn-control-plane node-role.kubernetes.io/control-plane:NoSchedule- node/kube-ovn-control-plane untainted \u5982\u679c\u786e\u5b9a\u4e0d\u9700\u8981\u5728 master \u8282\u70b9\u8c03\u5ea6\u4e1a\u52a1 Pod\uff0c\u8fd9\u4e00\u6b65\u53ef\u4ee5\u8df3\u8fc7\u3002","title":"\u53bb\u6389\u96c6\u7fa4 master \u8282\u70b9\u6c61\u70b9"},{"location":"start/one-step-install/#label","text":"$ kubectl label node -lbeta.kubernetes.io/os = linux kubernetes.io/os = linux --overwrite node/kube-ovn-control-plane not labeled node/kube-ovn-worker not labeled $ kubectl label node -lnode-role.kubernetes.io/control-plane kube-ovn/role = master --overwrite node/kube-ovn-control-plane labeled # \u4ee5\u4e0b label \u7528\u4e8e dpdk \u955c\u50cf\u7684\u5b89\u88c5\uff0c\u975e dpdk \u60c5\u51b5\uff0c\u53ef\u4ee5\u5ffd\u7565 $ kubectl label node -lovn.kubernetes.io/ovs_dp_type! = userspace ovn.kubernetes.io/ovs_dp_type = kernel --overwrite node/kube-ovn-control-plane labeled node/kube-ovn-worker labeled","title":"\u7ed9\u8282\u70b9\u6dfb\u52a0 label"},{"location":"start/one-step-install/#helm-repo","text":"$ helm repo add kubeovn https://kubeovn.github.io/kube-ovn/ \"kubeovn\" has been added to your repositories $ helm repo list NAME URL kubeovn https://kubeovn.github.io/kube-ovn/ $ helm search repo kubeovn NAME CHART VERSION APP VERSION DESCRIPTION kubeovn/kube-ovn 0 .1.0 1 .12.0 Helm chart for Kube-OVN","title":"\u6dfb\u52a0 Helm Repo \u4fe1\u606f"},{"location":"start/one-step-install/#helm-install-kube-ovn","text":"Node0IP\u3001Node1IP\u3001Node2IP \u53c2\u6570\u5206\u522b\u4e3a\u96c6\u7fa4 master \u8282\u70b9\u7684 IP \u5730\u5740\u3002\u5176\u4ed6\u53c2\u6570\u7684\u8bbe\u7f6e\uff0c\u53ef\u4ee5\u53c2\u8003 values.yaml \u6587\u4ef6\u4e2d\u53d8\u91cf\u5b9a\u4e49\u3002 # \u5355 master \u8282\u70b9\u73af\u5883\u5b89\u88c5 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } # \u4ee5\u4e0a\u8fb9\u7684 node \u4fe1\u606f\u4e3a\u4f8b\uff0c\u6267\u884c\u5b89\u88c5\u547d\u4ee4 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = 172 .18.0.3 NAME: kube-ovn LAST DEPLOYED: Fri Mar 31 12 :43:43 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None # \u9ad8\u53ef\u7528\u96c6\u7fa4\u5b89\u88c5 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } , ${ Node1IP } , ${ Node2IP } , --set replicaCount = 3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6267\u884c helm install \u5b89\u88c5 Kube-OVN"},{"location":"start/prepare/","text":"\u51c6\u5907\u5de5\u4f5c \u00b6 Kube-OVN \u662f\u4e00\u4e2a\u7b26\u5408 CNI \u89c4\u8303\u7684\u7f51\u7edc\u7ec4\u4ef6\uff0c\u5176\u8fd0\u884c\u9700\u8981\u4f9d\u8d56 Kubernetes \u73af\u5883\u53ca\u5bf9\u5e94\u7684\u5185\u6838\u7f51\u7edc\u6a21\u5757\u3002 \u4ee5\u4e0b\u662f\u901a\u8fc7\u6d4b\u8bd5\u7684\u64cd\u4f5c\u7cfb\u7edf\u548c\u8f6f\u4ef6\u7248\u672c\uff0c\u73af\u5883\u914d\u7f6e\u548c\u6240\u9700\u8981\u5f00\u653e\u7684\u7aef\u53e3\u4fe1\u606f\u3002 \u8f6f\u4ef6\u7248\u672c \u00b6 Kubernetes >= 1.23\u3002 Docker >= 1.12.6, Containerd >= 1.3.4\u3002 \u64cd\u4f5c\u7cfb\u7edf: CentOS 7/8, Ubuntu 16.04/18.04/20.04\u3002 \u5176\u4ed6 Linux \u53d1\u884c\u7248\uff0c\u9700\u8981\u68c0\u67e5\u4e00\u4e0b\u5185\u6838\u6a21\u5757\u662f\u5426\u5b58\u5728 geneve , openvswitch , ip_tables \u548c iptable_nat \uff0cKube-OVN \u6b63\u5e38\u5de5\u4f5c\u4f9d\u8d56\u4e0a\u8ff0\u6a21\u5757\u3002 \u6ce8\u610f\u4e8b\u9879 \uff1a \u5982\u679c\u5185\u6838\u7248\u672c\u4e3a 3.10.0-862 \u5185\u6838 netfilter \u6a21\u5757\u5b58\u5728 bug \u4f1a\u5bfc\u81f4 Kube-OVN \u5185\u7f6e\u8d1f\u8f7d\u5747\u8861\u5668\u65e0\u6cd5\u5de5\u4f5c\uff0c\u9700\u8981\u5bf9\u5185\u6838\u5347\u7ea7\uff0c\u5efa\u8bae\u4f7f\u7528 CentOS \u5b98\u65b9\u5bf9\u5e94\u7248\u672c\u6700\u65b0\u5185\u6838\u4fdd\u8bc1\u7cfb\u7edf\u7684\u5b89\u5168\u3002\u76f8\u5173\u5185\u6838 bug \u53c2\u8003 Floating IPs broken after kernel upgrade to Centos/RHEL 7.5 - DNAT not working \u3002 Rocky Linux 8.6 \u7684\u5185\u6838 4.18.0-372.9.1.el8.x86_64 \u5b58\u5728 TCP \u901a\u4fe1\u95ee\u9898 TCP connection failed in Rocky Linux 8.6 \uff0c\u8bf7\u5347\u7ea7\u5185\u6838\u81f3 4.18.0-372.13.1.el8_6.x86_64 \u6216\u66f4\u9ad8\u7248\u672c\u3002 \u5982\u679c\u5185\u6838\u7248\u672c\u4e3a 4.4 \u5219\u5bf9\u5e94\u7684\u5185\u6838 openvswitch \u6a21\u5757\u5b58\u5728\u95ee\u9898\uff0c\u5efa\u8bae\u5347\u7ea7\u6216\u624b\u52a8\u7f16\u8bd1 openvswitch \u65b0\u7248\u672c\u6a21\u5757\u8fdb\u884c\u66f4\u65b0 Geneve \u96a7\u9053\u5efa\u7acb\u9700\u8981\u68c0\u67e5 IPv6\uff0c\u53ef\u901a\u8fc7 cat /proc/cmdline \u68c0\u67e5\u5185\u6838\u542f\u52a8\u53c2\u6570\uff0c \u76f8\u5173\u5185\u6838 bug \u8bf7\u53c2\u8003 Geneve tunnels don't work when ipv6 is disabled \u3002 \u73af\u5883\u914d\u7f6e \u00b6 Kernel \u542f\u52a8\u9700\u8981\u5f00\u542f ipv6, \u5982\u679c kernel \u542f\u52a8\u53c2\u6570\u5305\u542b ipv6.disable=1 \u9700\u8981\u5c06\u5176\u8bbe\u7f6e\u4e3a 0\u3002 kube-proxy \u6b63\u5e38\u5de5\u4f5c\uff0cKube-OVN \u53ef\u4ee5\u901a\u8fc7 SVC IP \u8bbf\u95ee\u5230 kube-apiserver \u3002 \u786e\u8ba4 kubelet \u914d\u7f6e\u53c2\u6570\u5f00\u542f\u4e86 CNI\uff0c\u5e76\u4e14\u914d\u7f6e\u5728\u6807\u51c6\u8def\u5f84\u4e0b, kubelet \u542f\u52a8\u65f6\u5e94\u5305\u542b\u5982\u4e0b\u53c2\u6570 --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d \u3002 \u786e\u8ba4\u672a\u5b89\u88c5\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u6216\u8005\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u5df2\u7ecf\u88ab\u6e05\u9664\uff0c\u68c0\u67e5 /etc/cni/net.d/ \u8def\u5f84\u4e0b\u65e0\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u914d\u7f6e\u6587\u4ef6\u3002\u5982\u679c\u4e4b\u524d\u5b89\u88c5\u8fc7\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u5efa\u8bae\u5220\u9664\u540e\u91cd\u542f\u673a\u5668\u6e05\u7406\u6b8b\u7559\u7f51\u7edc\u8d44\u6e90\u3002 \u7aef\u53e3\u4fe1\u606f \u00b6 \u7ec4\u4ef6 \u7aef\u53e3 \u7528\u9014 ovn-central 6641/tcp, 6642/tcp, 6643/tcp, 6644/tcp ovn-db \u548c raft server \u76d1\u542c\u7aef\u53e3 ovs-ovn Geneve 6081/udp, STT 7471/tcp, Vxlan 4789/udp \u96a7\u9053\u7aef\u53e3 kube-ovn-controller 10660/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 kube-ovn-daemon 10665/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 kube-ovn-monitor 10661/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u51c6\u5907\u5de5\u4f5c"},{"location":"start/prepare/#_1","text":"Kube-OVN \u662f\u4e00\u4e2a\u7b26\u5408 CNI \u89c4\u8303\u7684\u7f51\u7edc\u7ec4\u4ef6\uff0c\u5176\u8fd0\u884c\u9700\u8981\u4f9d\u8d56 Kubernetes \u73af\u5883\u53ca\u5bf9\u5e94\u7684\u5185\u6838\u7f51\u7edc\u6a21\u5757\u3002 \u4ee5\u4e0b\u662f\u901a\u8fc7\u6d4b\u8bd5\u7684\u64cd\u4f5c\u7cfb\u7edf\u548c\u8f6f\u4ef6\u7248\u672c\uff0c\u73af\u5883\u914d\u7f6e\u548c\u6240\u9700\u8981\u5f00\u653e\u7684\u7aef\u53e3\u4fe1\u606f\u3002","title":"\u51c6\u5907\u5de5\u4f5c"},{"location":"start/prepare/#_2","text":"Kubernetes >= 1.23\u3002 Docker >= 1.12.6, Containerd >= 1.3.4\u3002 \u64cd\u4f5c\u7cfb\u7edf: CentOS 7/8, Ubuntu 16.04/18.04/20.04\u3002 \u5176\u4ed6 Linux \u53d1\u884c\u7248\uff0c\u9700\u8981\u68c0\u67e5\u4e00\u4e0b\u5185\u6838\u6a21\u5757\u662f\u5426\u5b58\u5728 geneve , openvswitch , ip_tables \u548c iptable_nat \uff0cKube-OVN \u6b63\u5e38\u5de5\u4f5c\u4f9d\u8d56\u4e0a\u8ff0\u6a21\u5757\u3002 \u6ce8\u610f\u4e8b\u9879 \uff1a \u5982\u679c\u5185\u6838\u7248\u672c\u4e3a 3.10.0-862 \u5185\u6838 netfilter \u6a21\u5757\u5b58\u5728 bug \u4f1a\u5bfc\u81f4 Kube-OVN \u5185\u7f6e\u8d1f\u8f7d\u5747\u8861\u5668\u65e0\u6cd5\u5de5\u4f5c\uff0c\u9700\u8981\u5bf9\u5185\u6838\u5347\u7ea7\uff0c\u5efa\u8bae\u4f7f\u7528 CentOS \u5b98\u65b9\u5bf9\u5e94\u7248\u672c\u6700\u65b0\u5185\u6838\u4fdd\u8bc1\u7cfb\u7edf\u7684\u5b89\u5168\u3002\u76f8\u5173\u5185\u6838 bug \u53c2\u8003 Floating IPs broken after kernel upgrade to Centos/RHEL 7.5 - DNAT not working \u3002 Rocky Linux 8.6 \u7684\u5185\u6838 4.18.0-372.9.1.el8.x86_64 \u5b58\u5728 TCP \u901a\u4fe1\u95ee\u9898 TCP connection failed in Rocky Linux 8.6 \uff0c\u8bf7\u5347\u7ea7\u5185\u6838\u81f3 4.18.0-372.13.1.el8_6.x86_64 \u6216\u66f4\u9ad8\u7248\u672c\u3002 \u5982\u679c\u5185\u6838\u7248\u672c\u4e3a 4.4 \u5219\u5bf9\u5e94\u7684\u5185\u6838 openvswitch \u6a21\u5757\u5b58\u5728\u95ee\u9898\uff0c\u5efa\u8bae\u5347\u7ea7\u6216\u624b\u52a8\u7f16\u8bd1 openvswitch \u65b0\u7248\u672c\u6a21\u5757\u8fdb\u884c\u66f4\u65b0 Geneve \u96a7\u9053\u5efa\u7acb\u9700\u8981\u68c0\u67e5 IPv6\uff0c\u53ef\u901a\u8fc7 cat /proc/cmdline \u68c0\u67e5\u5185\u6838\u542f\u52a8\u53c2\u6570\uff0c \u76f8\u5173\u5185\u6838 bug \u8bf7\u53c2\u8003 Geneve tunnels don't work when ipv6 is disabled \u3002","title":"\u8f6f\u4ef6\u7248\u672c"},{"location":"start/prepare/#_3","text":"Kernel \u542f\u52a8\u9700\u8981\u5f00\u542f ipv6, \u5982\u679c kernel \u542f\u52a8\u53c2\u6570\u5305\u542b ipv6.disable=1 \u9700\u8981\u5c06\u5176\u8bbe\u7f6e\u4e3a 0\u3002 kube-proxy \u6b63\u5e38\u5de5\u4f5c\uff0cKube-OVN \u53ef\u4ee5\u901a\u8fc7 SVC IP \u8bbf\u95ee\u5230 kube-apiserver \u3002 \u786e\u8ba4 kubelet \u914d\u7f6e\u53c2\u6570\u5f00\u542f\u4e86 CNI\uff0c\u5e76\u4e14\u914d\u7f6e\u5728\u6807\u51c6\u8def\u5f84\u4e0b, kubelet \u542f\u52a8\u65f6\u5e94\u5305\u542b\u5982\u4e0b\u53c2\u6570 --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d \u3002 \u786e\u8ba4\u672a\u5b89\u88c5\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u6216\u8005\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u5df2\u7ecf\u88ab\u6e05\u9664\uff0c\u68c0\u67e5 /etc/cni/net.d/ \u8def\u5f84\u4e0b\u65e0\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u914d\u7f6e\u6587\u4ef6\u3002\u5982\u679c\u4e4b\u524d\u5b89\u88c5\u8fc7\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u5efa\u8bae\u5220\u9664\u540e\u91cd\u542f\u673a\u5668\u6e05\u7406\u6b8b\u7559\u7f51\u7edc\u8d44\u6e90\u3002","title":"\u73af\u5883\u914d\u7f6e"},{"location":"start/prepare/#_4","text":"\u7ec4\u4ef6 \u7aef\u53e3 \u7528\u9014 ovn-central 6641/tcp, 6642/tcp, 6643/tcp, 6644/tcp ovn-db \u548c raft server \u76d1\u542c\u7aef\u53e3 ovs-ovn Geneve 6081/udp, STT 7471/tcp, Vxlan 4789/udp \u96a7\u9053\u7aef\u53e3 kube-ovn-controller 10660/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 kube-ovn-daemon 10665/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 kube-ovn-monitor 10661/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7aef\u53e3\u4fe1\u606f"},{"location":"start/sealos-install/","text":"\u4f7f\u7528 sealos \u4e00\u952e\u90e8\u7f72 Kubernetes \u548c Kube-OVN \u00b6 sealos \u4f5c\u4e3a Kubernetes \u7684\u4e00\u4e2a\u53d1\u884c\u7248\uff0c\u901a\u8fc7\u6781\u7b80\u7684\u4f7f\u7528\u65b9\u5f0f\u548c\u56fd\u5185\u7684\u955c\u50cf\u4ed3\u5e93\uff0c\u53ef\u4ee5\u5e2e\u52a9\u7528\u6237\u5feb\u901f\u4ece\u96f6\u521d\u59cb\u5316\u4e00\u4e2a\u5bb9\u5668\u96c6\u7fa4\u3002 \u901a\u8fc7\u4f7f\u7528 sealos \u7528\u6237\u53ef\u4ee5\u901a\u8fc7\u4e00\u6761\u547d\u4ee4\u5728\u51e0\u5206\u949f\u5185\u90e8\u7f72\u51fa\u4e00\u4e2a\u5b89\u88c5\u597d Kube-OVN \u7684 Kubernetes \u96c6\u7fa4\u3002 \u4e0b\u8f7d\u5b89\u88c5 sealos \u00b6 AMD64 ARM64 wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_amd64.tar.gz && \\ tar -zxvf sealos_4.1.4_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_arm64.tar.gz && \\ tar -zxvf sealos_4.1.4_linux_arm64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin \u90e8\u7f72 Kubernetes \u548c Kube-OVN \u00b6 ```bash sealos run labring/kubernetes:v1.24.3 labring/kube-ovn:v1.10.5 \\ --masters [masters ips seperated by comma] \\ --nodes [nodes ips seperated by comma] -p [your-ssh-passwd] ``` \u7b49\u5f85\u90e8\u7f72\u5b8c\u6210 \u00b6 ```bash [Step 6/6] Finish ,,,, ,::, ,,::,,,, ,,,,,::::::::::::,,,,, ,,,::::::::::::::::::::::,,, ,,::::::::::::::::::::::::::::,, ,,::::::::::::::::::::::::::::::::,, ,::::::::::::::::::::::::::::::::::::, ,:::::::::::::,, ,,:::::,,,::::::::::, ,,:::::::::::::, ,::, ,:::::::::, ,:::::::::::::, :x, ,:: :, ,:::::::::, ,:::::::::::::::, ,,, ,::, ,, ,::::::::::, ,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x ,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x ,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx ,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx ,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,:: ,::::, ,::::, ,::::, ,::::, ,::::, ,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,, ,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::, ,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::, ,::::, ,::::, ,::::, ,::::::::::::,,,,, ,,,, ,::::, ,,,, ,,,::::,,,, ,::::, ,,::, Thanks for choosing Kube-OVN! For more advanced features, please read https://github.com/kubeovn/kube-ovn#documents If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose 2022-08-10T16:31:34 info succeeded in creating a new cluster, enjoy it! 2022-08-10T16:31:34 info ___ ___ ___ ___ ___ ___ /\\ \\ /\\ \\ /\\ \\ /\\__\\ /\\ \\ /\\ \\ /::\\ \\ /::\\ \\ /::\\ \\ /:/ / /::\\ \\ /::\\ \\ /:/\\ \\ \\ /:/\\:\\ \\ /:/\\:\\ \\ /:/ / /:/\\:\\ \\ /:/\\ \\ \\ _\\:\\~\\ \\ \\ /::\\~\\:\\ \\ /::\\~\\:\\ \\ /:/ / /:/ \\:\\ \\ _\\:\\~\\ \\ \\ /\\ \\:\\ \\ \\__\\ /:/\\:\\ \\:\\__\\ /:/\\:\\ \\:\\__\\ /:/__/ /:/__/ \\:\\__\\ /\\ \\:\\ \\ \\__\\ \\:\\ \\:\\ \\/__/ \\:\\~\\:\\ \\/__/ \\/__\\:\\/:/ / \\:\\ \\ \\:\\ \\ /:/ / \\:\\ \\:\\ \\/__/ \\:\\ \\:\\__\\ \\:\\ \\:\\__\\ \\::/ / \\:\\ \\ \\:\\ /:/ / \\:\\ \\:\\__\\ \\:\\/:/ / \\:\\ \\/__/ /:/ / \\:\\ \\ \\:\\/:/ / \\:\\/:/ / \\::/ / \\:\\__\\ /:/ / \\:\\__\\ \\::/ / \\::/ / \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ Website :https://www.sealos.io/ Address :github.com/labring/sealos ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 Sealos \u4e00\u952e\u90e8\u7f72 Kubernetes \u548c Kube-OVN"},{"location":"start/sealos-install/#sealos-kubernetes-kube-ovn","text":"sealos \u4f5c\u4e3a Kubernetes \u7684\u4e00\u4e2a\u53d1\u884c\u7248\uff0c\u901a\u8fc7\u6781\u7b80\u7684\u4f7f\u7528\u65b9\u5f0f\u548c\u56fd\u5185\u7684\u955c\u50cf\u4ed3\u5e93\uff0c\u53ef\u4ee5\u5e2e\u52a9\u7528\u6237\u5feb\u901f\u4ece\u96f6\u521d\u59cb\u5316\u4e00\u4e2a\u5bb9\u5668\u96c6\u7fa4\u3002 \u901a\u8fc7\u4f7f\u7528 sealos \u7528\u6237\u53ef\u4ee5\u901a\u8fc7\u4e00\u6761\u547d\u4ee4\u5728\u51e0\u5206\u949f\u5185\u90e8\u7f72\u51fa\u4e00\u4e2a\u5b89\u88c5\u597d Kube-OVN \u7684 Kubernetes \u96c6\u7fa4\u3002","title":"\u4f7f\u7528 sealos \u4e00\u952e\u90e8\u7f72 Kubernetes \u548c Kube-OVN"},{"location":"start/sealos-install/#sealos","text":"AMD64 ARM64 wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_amd64.tar.gz && \\ tar -zxvf sealos_4.1.4_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_arm64.tar.gz && \\ tar -zxvf sealos_4.1.4_linux_arm64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin","title":"\u4e0b\u8f7d\u5b89\u88c5 sealos"},{"location":"start/sealos-install/#kubernetes-kube-ovn","text":"```bash sealos run labring/kubernetes:v1.24.3 labring/kube-ovn:v1.10.5 \\ --masters [masters ips seperated by comma] \\ --nodes [nodes ips seperated by comma] -p [your-ssh-passwd] ```","title":"\u90e8\u7f72 Kubernetes \u548c Kube-OVN"},{"location":"start/sealos-install/#_1","text":"```bash [Step 6/6] Finish ,,,, ,::, ,,::,,,, ,,,,,::::::::::::,,,,, ,,,::::::::::::::::::::::,,, ,,::::::::::::::::::::::::::::,, ,,::::::::::::::::::::::::::::::::,, ,::::::::::::::::::::::::::::::::::::, ,:::::::::::::,, ,,:::::,,,::::::::::, ,,:::::::::::::, ,::, ,:::::::::, ,:::::::::::::, :x, ,:: :, ,:::::::::, ,:::::::::::::::, ,,, ,::, ,, ,::::::::::, ,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x ,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x ,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx ,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx ,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,:: ,::::, ,::::, ,::::, ,::::, ,::::, ,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,, ,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::, ,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::, ,::::, ,::::, ,::::, ,::::::::::::,,,,, ,,,, ,::::, ,,,, ,,,::::,,,, ,::::, ,,::, Thanks for choosing Kube-OVN! For more advanced features, please read https://github.com/kubeovn/kube-ovn#documents If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose 2022-08-10T16:31:34 info succeeded in creating a new cluster, enjoy it! 2022-08-10T16:31:34 info ___ ___ ___ ___ ___ ___ /\\ \\ /\\ \\ /\\ \\ /\\__\\ /\\ \\ /\\ \\ /::\\ \\ /::\\ \\ /::\\ \\ /:/ / /::\\ \\ /::\\ \\ /:/\\ \\ \\ /:/\\:\\ \\ /:/\\:\\ \\ /:/ / /:/\\:\\ \\ /:/\\ \\ \\ _\\:\\~\\ \\ \\ /::\\~\\:\\ \\ /::\\~\\:\\ \\ /:/ / /:/ \\:\\ \\ _\\:\\~\\ \\ \\ /\\ \\:\\ \\ \\__\\ /:/\\:\\ \\:\\__\\ /:/\\:\\ \\:\\__\\ /:/__/ /:/__/ \\:\\__\\ /\\ \\:\\ \\ \\__\\ \\:\\ \\:\\ \\/__/ \\:\\~\\:\\ \\/__/ \\/__\\:\\/:/ / \\:\\ \\ \\:\\ \\ /:/ / \\:\\ \\:\\ \\/__/ \\:\\ \\:\\__\\ \\:\\ \\:\\__\\ \\::/ / \\:\\ \\ \\:\\ /:/ / \\:\\ \\:\\__\\ \\:\\/:/ / \\:\\ \\/__/ /:/ / \\:\\ \\ \\:\\/:/ / \\:\\/:/ / \\::/ / \\:\\__\\ /:/ / \\:\\__\\ \\::/ / \\::/ / \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ Website :https://www.sealos.io/ Address :github.com/labring/sealos ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7b49\u5f85\u90e8\u7f72\u5b8c\u6210"},{"location":"start/underlay/","text":"Underlay \u7f51\u7edc\u5b89\u88c5 \u00b6 \u9ed8\u8ba4\u60c5\u51b5\u4e0b Kube-OVN \u7684\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528 Geneve \u5bf9\u8de8\u4e3b\u673a\u6d41\u91cf\u8fdb\u884c\u5c01\u88c5\uff0c\u5728\u57fa\u7840\u8bbe\u65bd\u4e4b\u4e0a\u62bd\u8c61\u51fa\u4e00\u5c42\u865a\u62df\u7684 Overlay \u7f51\u7edc\u3002 \u5bf9\u4e8e\u5e0c\u671b\u5bb9\u5668\u7f51\u7edc\u76f4\u63a5\u4f7f\u7528\u7269\u7406\u7f51\u7edc\u5730\u5740\u6bb5\u60c5\u51b5\uff0c\u53ef\u4ee5\u5c06 Kube-OVN \u7684\u9ed8\u8ba4\u5b50\u7f51\u5de5\u4f5c\u5728 Underlay \u6a21\u5f0f\uff0c\u53ef\u4ee5\u76f4\u63a5\u7ed9\u5bb9\u5668\u5206\u914d\u7269\u7406\u7f51\u7edc\u4e2d\u7684\u5730\u5740\u8d44\u6e90\uff0c\u8fbe\u5230\u66f4\u597d\u7684\u6027\u80fd\u4ee5\u53ca\u548c\u7269\u7406\u7f51\u7edc\u7684\u8fde\u901a\u6027\u3002 \u529f\u80fd\u9650\u5236 \u00b6 \u7531\u4e8e\u8be5\u6a21\u5f0f\u4e0b\u5bb9\u5668\u7f51\u7edc\u76f4\u63a5\u4f7f\u7528\u7269\u7406\u7f51\u7edc\u8fdb\u884c\u4e8c\u5c42\u5305\u8f6c\u53d1\uff0cOverlay \u6a21\u5f0f\u4e0b\u7684 SNAT/EIP\uff0c \u5206\u5e03\u5f0f\u7f51\u5173/\u96c6\u4e2d\u5f0f\u7f51\u5173\u7b49 L3 \u529f\u80fd\u65e0\u6cd5\u4f7f\u7528\uff0cVPC \u7ea7\u522b\u7684\u9694\u79bb\u4e5f\u65e0\u6cd5\u5bf9 Underlay \u5b50\u7f51\u751f\u6548\u3002 \u548c Macvlan \u6bd4\u8f83 \u00b6 Kube-OVN \u7684 Underlay \u6a21\u5f0f\u548c Macvlan \u5de5\u4f5c\u6a21\u5f0f\u5341\u5206\u7c7b\u4f3c\uff0c\u5728\u529f\u80fd\u548c\u6027\u80fd\u4e0a\u4e3b\u8981\u6709\u4ee5\u4e0b\u51e0\u4e2a\u533a\u522b\uff1a \u7531\u4e8e Macvlan \u7684\u5185\u6838\u8def\u5f84\u66f4\u77ed\uff0c\u5e76\u4e14\u4e0d\u9700\u8981 OVS \u5bf9\u6570\u636e\u5305\u8fdb\u884c\u5904\u7406\uff0cMacvlan \u5728\u541e\u5410\u91cf\u548c\u5ef6\u8fdf\u6027\u80fd\u6307\u6807\u4e0a\u8868\u73b0\u4f1a\u66f4\u597d\u3002 Kube-OVN \u901a\u8fc7\u6d41\u8868\u63d0\u4f9b\u4e86 arp-proxy \u529f\u80fd\uff0c\u53ef\u4ee5\u7f13\u89e3\u5927\u89c4\u6a21\u7f51\u7edc\u4e0b\u7684 arp \u5e7f\u64ad\u98ce\u66b4\u98ce\u9669\u3002 \u7531\u4e8e Macvlan \u5de5\u4f5c\u5728\u5185\u6838\u5e95\u5c42\uff0c\u4f1a\u7ed5\u8fc7\u5bbf\u4e3b\u673a\u7684 netfilter\uff0cService \u548c NetworkPolicy \u529f\u80fd\u9700\u8981\u989d\u5916\u5f00\u53d1\u3002Kube-OVN \u901a\u8fc7 OVS \u6d41\u8868\u63d0\u4f9b\u4e86 Service \u548c NetworkPolicy \u7684\u80fd\u529b\u3002 Kube-OVN \u7684 Underlay \u6a21\u5f0f\u76f8\u6bd4 Macvlan \u989d\u5916\u63d0\u4f9b\u4e86\u5730\u5740\u7ba1\u7406\uff0c\u56fa\u5b9a IP \u548c QoS \u7b49\u529f\u80fd\u3002 \u73af\u5883\u8981\u6c42 \u00b6 \u5728 Underlay \u6a21\u5f0f\u4e0b\uff0cOVS \u5c06\u4f1a\u6865\u63a5\u4e00\u4e2a\u8282\u70b9\u7f51\u5361\u5230 OVS \u7f51\u6865\uff0c\u5e76\u5c06\u6570\u636e\u5305\u76f4\u63a5\u901a\u8fc7\u8be5\u8282\u70b9\u7f51\u5361\u5bf9\u5916\u53d1\u9001\uff0cL2/L3 \u5c42\u9762\u7684\u8f6c\u53d1\u80fd\u529b\u9700\u8981\u4f9d\u8d56\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u3002 \u9700\u8981\u9884\u5148\u5728\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u914d\u7f6e\u5bf9\u5e94\u7684\u7f51\u5173\u3001Vlan \u548c\u5b89\u5168\u7b56\u7565\u7b49\u914d\u7f6e\u3002 \u5bf9\u4e8e OpenStack \u7684 VM \u73af\u5883\uff0c\u9700\u8981\u5c06\u5bf9\u5e94\u7f51\u7edc\u7aef\u53e3\u7684 PortSecurity \u5173\u95ed\u3002 \u5bf9\u4e8e VMware \u7684 vSwitch \u7f51\u7edc\uff0c\u9700\u8981\u5c06 MAC Address Changes , Forged Transmits \u548c Promiscuous Mode Operation \u8bbe\u7f6e\u4e3a allow \u3002 \u5bf9\u4e8e Hyper-V \u865a\u62df\u5316\uff0c\u9700\u8981\u5f00\u542f\u865a\u62df\u673a\u7f51\u5361\u9ad8\u7ea7\u529f\u80fd\u4e2d\u7684 MAC Address Spoofing \u3002 \u516c\u6709\u4e91\uff0c\u4f8b\u5982 AWS\u3001GCE\u3001\u963f\u91cc\u4e91\u7b49\u7531\u4e8e\u4e0d\u652f\u6301\u7528\u6237\u81ea\u5b9a\u4e49 Mac \u65e0\u6cd5\u652f\u6301 Underlay \u6a21\u5f0f\u7f51\u7edc\uff0c\u5728\u8fd9\u79cd\u573a\u666f\u4e0b\u5982\u679c\u60f3\u4f7f\u7528 Underlay \u63a8\u8350\u4f7f\u7528\u5bf9\u5e94\u516c\u6709\u4e91\u5382\u5546\u63d0\u4f9b\u7684 VPC-CNI\u3002 \u6865\u63a5\u7f51\u5361\u4e0d\u80fd\u4e3a Linux Bridge\u3002 \u5bf9\u4e8e\u7ba1\u7406\u7f51\u548c\u5bb9\u5668\u7f51\u4f7f\u7528\u540c\u4e00\u4e2a\u7f51\u5361\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u7684 Mac \u5730\u5740\u3001IP \u5730\u5740\u3001\u8def\u7531\u4ee5\u53ca MTU \u5c06\u8f6c\u79fb\u6216\u590d\u5236\u81f3\u5bf9\u5e94\u7684 OVS Bridge\uff0c \u4ee5\u652f\u6301\u5355\u7f51\u5361\u90e8\u7f72 Underlay \u7f51\u7edc\u3002OVS Bridge \u540d\u79f0\u683c\u5f0f\u4e3a br-PROVIDER_NAME \uff0c PROVIDER_NAME \u4e3a Provider \u7f51\u7edc\u540d\u79f0\uff08\u9ed8\u8ba4\u4e3a provider\uff09\u3002 \u90e8\u7f72\u65f6\u6307\u5b9a\u7f51\u7edc\u6a21\u5f0f \u00b6 \u8be5\u90e8\u7f72\u6a21\u5f0f\u5c06\u9ed8\u8ba4\u5b50\u7f51\u8bbe\u7f6e\u4e3a Underlay \u6a21\u5f0f\uff0c\u6240\u6709\u672a\u6307\u5b9a\u5b50\u7f51\u7684 Pod \u5747\u4f1a\u9ed8\u8ba4\u8fd0\u884c\u5728 Underlay \u7f51\u7edc\u4e2d\u3002 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c \u00b6 wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u811a\u672c\u4e2d\u76f8\u5e94\u914d\u7f6e \u00b6 ENABLE_ARP_DETECT_IP_CONFLICT # \u5982\u6709\u9700\u8981\uff0c\u53ef\u4ee5\u9009\u62e9\u5173\u95ed vlan \u7f51\u7edc arp \u51b2\u7a81\u68c0\u6d4b NETWORK_TYPE # \u8bbe\u7f6e\u4e3a vlan VLAN_INTERFACE_NAME # \u8bbe\u7f6e\u4e3a\u5bbf\u4e3b\u673a\u4e0a\u627f\u62c5\u5bb9\u5668\u6d41\u91cf\u7684\u7f51\u5361\uff0c\u4f8b\u5982 eth1 VLAN_ID # \u4ea4\u6362\u673a\u6240\u63a5\u53d7\u7684 VLAN Tag\uff0c\u82e5\u8bbe\u7f6e\u4e3a 0 \u5219\u4e0d\u505a VLAN \u5c01\u88c5 POD_CIDR # \u8bbe\u7f6e\u4e3a\u7269\u7406\u7f51\u7edc CIDR\uff0c \u4f8b\u5982 192.168.1.0/24 POD_GATEWAY # \u8bbe\u7f6e\u4e3a\u7269\u7406\u7f51\u7edc\u7f51\u5173\uff0c\u4f8b\u5982 192.168.1.1 EXCLUDE_IPS # \u6392\u9664\u8303\u56f4\uff0c\u907f\u514d\u5bb9\u5668\u7f51\u6bb5\u548c\u7269\u7406\u7f51\u7edc\u5df2\u7528 IP \u51b2\u7a81\uff0c\u4f8b\u5982 192.168.1.1..192.168.1.100 ENABLE_LB # \u5982\u679c Underlay \u5b50\u7f51\u9700\u8981\u4f7f\u7528 Service \u9700\u8981\u8bbe\u7f6e\u4e3a true EXCHANGE_LINK_NAME # \u662f\u5426\u4ea4\u6362\u9ed8\u8ba4 provider-network \u4e0b OVS \u7f51\u6865\u548c\u6865\u63a5\u7f51\u5361\u7684\u540d\u5b57\uff0c\u9ed8\u8ba4\u4e3a false LS_DNAT_MOD_DL_DST # DNAT \u65f6\u662f\u5426\u5bf9 MAC \u5730\u5740\u8fdb\u884c\u8f6c\u6362\uff0c\u53ef\u52a0\u901f Service \u7684\u8bbf\u95ee\uff0c\u9ed8\u8ba4\u4e3a true \u8fd0\u884c\u5b89\u88c5\u811a\u672c \u00b6 bash install.sh \u901a\u8fc7 CRD \u52a8\u6001\u521b\u5efa Underlay \u7f51\u7edc \u00b6 \u8be5\u65b9\u5f0f\u53ef\u5728\u5b89\u88c5\u540e\u52a8\u6001\u7684\u521b\u5efa\u67d0\u4e2a Underlay \u5b50\u7f51\u4f9b Pod \u4f7f\u7528\u3002\u9700\u8981\u914d\u7f6e ProviderNetwork \uff0c Vlan \u548c Subnet \u4e09\u79cd\u81ea\u5b9a\u4e49\u8d44\u6e90\u3002 \u521b\u5efa ProviderNetwork \u00b6 ProviderNetwork \u63d0\u4f9b\u4e86\u4e3b\u673a\u7f51\u5361\u5230\u7269\u7406\u7f51\u7edc\u6620\u5c04\u7684\u62bd\u8c61\uff0c\u5c06\u540c\u5c5e\u4e00\u4e2a\u7f51\u7edc\u7684\u7f51\u5361\u8fdb\u884c\u7edf\u4e00\u7ba1\u7406\uff0c \u5e76\u89e3\u51b3\u5728\u590d\u6742\u73af\u5883\u4e0b\u540c\u673a\u5668\u591a\u7f51\u5361\u3001\u7f51\u5361\u540d\u4e0d\u4e00\u81f4\u3001\u5bf9\u5e94 Underlay \u7f51\u7edc\u4e0d\u4e00\u81f4\u7b49\u60c5\u51b5\u4e0b\u7684\u914d\u7f6e\u95ee\u9898\u3002 \u521b\u5efa\u5982\u4e0b ProviderNetwork \u5e76\u5e94\u7528: apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: net1 spec: defaultInterface: eth1 customInterfaces: - interface: eth2 nodes: - node1 excludeNodes: - node2 \u6ce8\u610f\uff1aProviderNetwork \u8d44\u6e90\u540d\u79f0\u7684\u957f\u5ea6\u4e0d\u5f97\u8d85\u8fc7 12\u3002 defaultInterface : \u4e3a\u9ed8\u8ba4\u4f7f\u7528\u7684\u8282\u70b9\u7f51\u5361\u540d\u79f0\u3002 ProviderNetwork \u521b\u5efa\u6210\u529f\u540e\uff0c\u5404\u8282\u70b9\uff08\u9664 excludeNodes \u5916\uff09\u4e2d\u4f1a\u521b\u5efa\u540d\u4e3a br-net1\uff08\u683c\u5f0f\u4e3a br-NAME \uff09\u7684 OVS \u7f51\u6865\uff0c\u5e76\u5c06\u6307\u5b9a\u7684\u8282\u70b9\u7f51\u5361\u6865\u63a5\u81f3\u6b64\u7f51\u6865\u3002 customInterfaces : \u4e3a\u53ef\u9009\u9879\uff0c\u53ef\u9488\u5bf9\u7279\u5b9a\u8282\u70b9\u6307\u5b9a\u9700\u8981\u4f7f\u7528\u7684\u7f51\u5361\u3002 excludeNodes : \u53ef\u9009\u9879\uff0c\u7528\u4e8e\u6307\u5b9a\u4e0d\u6865\u63a5\u7f51\u5361\u7684\u8282\u70b9\u3002\u8be5\u5217\u8868\u4e2d\u7684\u8282\u70b9\u4f1a\u88ab\u6dfb\u52a0 net1.provider-network.ovn.kubernetes.io/exclude=true \u6807\u7b7e\u3002 \u5176\u5b83\u8282\u70b9\u4f1a\u88ab\u6dfb\u52a0\u5982\u4e0b\u6807\u7b7e\uff1a Key Value \u63cf\u8ff0 net1.provider-network.ovn.kubernetes.io/ready true \u8282\u70b9\u4e2d\u7684\u6865\u63a5\u5de5\u4f5c\u5df2\u5b8c\u6210\uff0cProviderNetwork \u5728\u8282\u70b9\u4e2d\u53ef\u7528 net1.provider-network.ovn.kubernetes.io/interface eth1 \u8282\u70b9\u4e2d\u88ab\u6865\u63a5\u7684\u7f51\u5361\u7684\u540d\u79f0 net1.provider-network.ovn.kubernetes.io/mtu 1500 \u8282\u70b9\u4e2d\u88ab\u6865\u63a5\u7684\u7f51\u5361\u7684 MTU \u5982\u679c\u8282\u70b9\u7f51\u5361\u4e0a\u5df2\u7ecf\u914d\u7f6e\u4e86 IP\uff0c\u5219 IP \u5730\u5740\u548c\u7f51\u5361\u4e0a\u7684\u8def\u7531\u4f1a\u88ab\u8f6c\u79fb\u81f3\u5bf9\u5e94\u7684 OVS \u7f51\u6865\u3002 \u521b\u5efa VLAN \u00b6 Vlan \u63d0\u4f9b\u4e86\u5c06 Vlan Tag \u548c ProviderNetwork \u8fdb\u884c\u7ed1\u5b9a\u7684\u80fd\u529b\u3002 \u521b\u5efa\u5982\u4e0b VLAN \u5e76\u5e94\u7528\uff1a apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan1 spec: id: 0 provider: net1 id : \u4e3a VLAN ID/Tag\uff0cKube-OVN \u4f1a\u5bf9\u5bf9\u8be5 Vlan \u4e0b\u7684\u6d41\u91cf\u589e\u52a0 Vlan \u6807\u7b7e\uff0c\u4e3a 0 \u65f6\u4e0d\u589e\u52a0\u4efb\u4f55\u6807\u7b7e\u3002 provider : \u4e3a\u9700\u8981\u4f7f\u7528\u7684 ProviderNetwork \u8d44\u6e90\u7684\u540d\u79f0\u3002\u591a\u4e2a VLAN \u53ef\u4ee5\u5f15\u7528\u540c\u4e00\u4e2a ProviderNetwork\u3002 \u521b\u5efa Subnet \u00b6 \u5c06 Vlan \u548c\u4e00\u4e2a\u5b50\u7f51\u7ed1\u5b9a\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 \u5c06 vlan \u7684\u503c\u6307\u5b9a\u4e3a\u9700\u8981\u4f7f\u7528\u7684 VLAN \u540d\u79f0\u5373\u53ef\u3002\u591a\u4e2a Subnet \u53ef\u4ee5\u5f15\u7528\u540c\u4e00\u4e2a VLAN\u3002 \u5bb9\u5668\u521b\u5efa \u00b6 \u53ef\u6309\u6b63\u5e38\u5bb9\u5668\u521b\u5efa\u65b9\u5f0f\u8fdb\u884c\u521b\u5efa\uff0c\u67e5\u770b\u5bb9\u5668 IP \u662f\u5426\u5728\u89c4\u5b9a\u8303\u56f4\u5185\uff0c\u4ee5\u53ca\u5bb9\u5668\u662f\u5426\u53ef\u4ee5\u548c\u7269\u7406\u7f51\u7edc\u4e92\u901a\u3002 \u5982\u6709\u56fa\u5b9a IP \u9700\u6c42\uff0c\u53ef\u53c2\u8003 Pod \u56fa\u5b9a IP \u548c Mac \u4f7f\u7528\u903b\u8f91\u7f51\u5173 \u00b6 \u5bf9\u4e8e\u7269\u7406\u7f51\u7edc\u4e0d\u5b58\u5728\u7f51\u5173\u7684\u60c5\u51b5\uff0cKube-OVN \u652f\u6301\u5728 Underlay \u6a21\u5f0f\u7684\u5b50\u7f51\u4e2d\u914d\u7f6e\u4f7f\u7528\u903b\u8f91\u7f51\u5173\u3002 \u82e5\u8981\u4f7f\u7528\u6b64\u529f\u80fd\uff0c\u8bbe\u7f6e\u5b50\u7f51\u7684 spec.logicalGateway \u4e3a true \u5373\u53ef\uff1a apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 172.17.0.0/16 gateway: 172.17.0.1 vlan: vlan1 logicalGateway: true \u5f00\u542f\u6b64\u529f\u80fd\u540e\uff0cPod \u4e0d\u4f7f\u7528\u5916\u90e8\u7f51\u5173\uff0c\u800c\u662f\u4f7f\u7528 Kube-OVN \u521b\u5efa\u7684\u903b\u8f91\u8def\u7531\u5668\uff08Logical Router\uff09\u5bf9\u4e8e\u8de8\u7f51\u6bb5\u901a\u4fe1\u8fdb\u884c\u8f6c\u53d1\u3002 Underlay \u548c Overlay \u7f51\u7edc\u4e92\u901a \u00b6 \u5982\u679c\u4e00\u4e2a\u96c6\u7fa4\u540c\u65f6\u5b58\u5728 Underlay \u548c Overlay \u5b50\u7f51\uff0c\u9ed8\u8ba4\u60c5\u51b5\u4e0b Overlay \u5b50\u7f51\u4e0b\u7684 Pod \u53ef\u4ee5\u901a\u8fc7\u7f51\u5173\u4ee5 NAT \u7684\u65b9\u5f0f\u8bbf\u95ee Underlay \u5b50\u7f51\u4e0b\u7684 Pod IP\u3002 \u5728 Underlay \u5b50\u7f51\u7684 Pod \u770b\u6765 Overlay \u5b50\u7f51\u7684\u5730\u5740\u662f\u4e00\u4e2a\u5916\u90e8\u7684\u5730\u5740\uff0c\u9700\u8981\u901a\u8fc7\u5e95\u5c42\u7269\u7406\u8bbe\u5907\u53bb\u8f6c\u53d1\uff0c\u4f46\u5e95\u5c42\u7269\u7406\u8bbe\u5907\u5e76\u4e0d\u6e05\u695a Overlay \u5b50\u7f51\u7684\u5730\u5740\u65e0\u6cd5\u8fdb\u884c\u8f6c\u53d1\u3002 \u56e0\u6b64 Underlay \u5b50\u7f51\u4e0b\u7684 Pod \u65e0\u6cd5\u901a\u8fc7 Pod IP \u76f4\u63a5\u8bbf\u95ee Overlay \u5b50\u7f51\u7684 Pod\u3002 \u5982\u679c\u9700\u8981 Underlay \u548c Overlay \u4e92\u901a\u9700\u8981\u5c06\u5b50\u7f51\u7684 u2oInterconnection \u8bbe\u7f6e\u4e3a true \uff0c\u5728\u8fd9\u4e2a\u60c5\u51b5\u4e0b Kube-OVN \u4f1a\u989d\u5916\u4f7f\u7528\u4e00\u4e2a Underlay IP \u5c06 Underlay \u5b50\u7f51 \u548c ovn-cluster \u903b\u8f91\u8def\u7531\u5668\u8fde\u63a5\uff0c\u5e76\u8bbe\u7f6e\u5bf9\u5e94\u7684\u8def\u7531\u89c4\u5219\u5b9e\u73b0\u4e92\u901a\u3002 \u548c\u903b\u8f91\u7f51\u5173\u4e0d\u540c\uff0c\u8be5\u65b9\u6848\u53ea\u4f1a\u8fde\u63a5 Kube-OVN \u5185\u90e8\u7684 Underlay \u548c Overlay \u5b50\u7f51\uff0c\u5176\u4ed6\u8bbf\u95ee\u5916\u7f51\u7684\u6d41\u91cf\u8fd8\u662f\u4f1a\u901a\u8fc7\u7269\u7406\u7f51\u5173\u8fdb\u884c\u8f6c\u53d1\u3002 \u6307\u5b9a\u903b\u8f91\u7f51\u5173 IP \u00b6 \u5f00\u542f\u4e92\u901a\u529f\u80fd\u540e\uff0c\u4f1a\u968f\u673a\u4ece subnet \u5185\u7684\u53d6\u4e00\u4e2a IP \u4f5c\u4e3a\u903b\u8f91\u7f51\u5173\uff0c\u5982\u679c\u9700\u8981\u6307\u5b9a Underlay Subnet \u7684\u903b\u8f91\u7f51\u5173\u53ef\u4ee5\u6307\u5b9a\u5b57\u6bb5 u2oInterconnectionIP \u3002 \u6307\u5b9a Underlay Subnet \u8fde\u63a5\u7684\u81ea\u5b9a\u4e49 VPC \u00b6 \u9ed8\u8ba4\u60c5\u51b5\u4e0b Underlay Subnet \u4f1a\u548c\u9ed8\u8ba4 VPC \u4e0a\u7684 Overlay Subnet \u4e92\u901a\uff0c\u5982\u679c\u8981\u6307\u5b9a\u548c\u67d0\u4e2a VPC \u4e92\u901a\uff0c\u5728 u2oInterconnection \u8bbe\u7f6e\u4e3a true \u540e\uff0c\u6307\u5b9a subnet.spec.vpc \u5b57\u6bb5\u4e3a\u8be5 VPC \u540d\u5b57\u5373\u53ef\u3002 \u6ce8\u610f\u4e8b\u9879 \u00b6 \u5982\u679c\u60a8\u4f7f\u7528\u7684\u8282\u70b9\u7f51\u5361\u4e0a\u914d\u7f6e\u6709 IP \u5730\u5740\uff0c\u4e14\u64cd\u4f5c\u7cfb\u7edf\u662f Ubuntu \u5e76\u901a\u8fc7 Netplan \u914d\u7f6e\u7f51\u7edc\uff0c\u5efa\u8bae\u60a8\u5c06 Netplan \u7684 renderer \u8bbe\u7f6e\u4e3a NetworkManager\uff0c\u5e76\u4e3a\u8282\u70b9\u7f51\u5361\u914d\u7f6e\u9759\u6001 IP \u5730\u5740\uff08\u5173\u95ed DHCP\uff09\uff1a network : renderer : NetworkManager ethernets : eth0 : dhcp4 : no addresses : - 172.16.143.129/24 version : 2 \u82e5\u8282\u70b9\u7f51\u7edc\u7ba1\u7406\u670d\u52a1\u4e3a NetworkManager\uff0c\u5728\u4f7f\u7528\u8282\u70b9\u7f51\u5361\u521b\u5efa ProviderNetwork \u540e\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u4ece NetworkManager \u7ba1\u7406\u5217\u8868\u4e2d\u79fb\u9664\uff08managed \u5c5e\u6027\u4e3a no\uff09\uff1a root@ubuntu:~# nmcli device status DEVICE TYPE STATE CONNECTION eth0 ethernet unmanaged netplan-eth0 \u5982\u679c\u60a8\u8981\u4fee\u6539\u7f51\u5361\u7684 IP \u6216\u8def\u7531\u914d\u7f6e\uff0c\u9700\u8981\u624b\u52a8\u5c06\u7f51\u5361\u91cd\u65b0\u52a0\u5165 NetworkManager \u7ba1\u7406\u5217\u8868\uff1a nmcli device set eth0 managed yes \u6267\u884c\u4ee5\u4e0a\u547d\u4ee4\u540e\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u4e0a\u7684 IP \u53ca\u8def\u7531\u91cd\u65b0\u8f6c\u79fb\u81f3 OVS \u7f51\u6865\uff0c\u5e76\u518d\u6b21\u5c06\u7f51\u5361\u4ece NetworkManager \u7ba1\u7406\u5217\u8868\u4e2d\u79fb\u9664\u3002 \u6ce8\u610f \uff1a\u8282\u70b9\u7f51\u5361\u914d\u7f6e\u7684\u52a8\u6001\u4fee\u6539\u4ec5\u652f\u6301 IP \u548c\u8def\u7531\uff0c\u4e0d\u652f\u6301 MAC \u5730\u5740\u7684\u4fee\u6539\u3002 \u5df2\u77e5\u95ee\u9898 \u00b6 \u7269\u7406\u7f51\u7edc\u5f00\u542f hairpin \u65f6 Pod \u7f51\u7edc\u5f02\u5e38 \u00b6 \u5f53\u7269\u7406\u7f51\u7edc\u5f00\u542f hairpin \u6216\u7c7b\u4f3c\u884c\u4e3a\u65f6\uff0c\u53ef\u80fd\u51fa\u73b0\u521b\u5efa Pod \u65f6\u7f51\u5173\u68c0\u67e5\u5931\u8d25\u3001Pod \u7f51\u7edc\u901a\u4fe1\u5f02\u5e38\u7b49\u95ee\u9898\u3002\u8fd9\u662f\u56e0\u4e3a OVS \u7f51\u6865\u9ed8\u8ba4\u7684 MAC \u5b66\u4e60\u529f\u80fd\u4e0d\u652f\u6301\u8fd9\u79cd\u7f51\u7edc\u73af\u5883\u3002 \u8981\u89e3\u51b3\u6b64\u95ee\u9898\uff0c\u9700\u8981\u5173\u95ed hairpin\uff08\u6216\u4fee\u6539\u7269\u7406\u7f51\u7edc\u7684\u76f8\u5173\u914d\u7f6e\uff09\uff0c\u6216\u66f4\u65b0 Kube-OVN \u7248\u672c\u3002 Pod \u6570\u91cf\u8f83\u591a\u65f6\u65b0\u5efa Pod \u7f51\u5173\u68c0\u67e5\u5931\u8d25 \u00b6 \u82e5\u540c\u4e00\u4e2a\u8282\u70b9\u4e0a\u8fd0\u884c\u7684 Pod \u6570\u91cf\u8f83\u591a\uff08\u5927\u4e8e 300\uff09\uff0c\u53ef\u80fd\u4f1a\u51fa\u73b0 ARP \u5e7f\u64ad\u5305\u7684 OVS \u6d41\u8868 resubmit \u6b21\u6570\u8d85\u8fc7\u4e0a\u9650\u5bfc\u81f4\u4e22\u5305\u7684\u73b0\u8c61\uff1a 2022-11-13T08:43:46.782Z|00222|ofproto_dpif_upcall(handler5)|WARN|Flow: arp,in_port=331,vlan_tci=0x0000,dl_src=00:00:00:25:eb:39,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.131.240,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:25:eb:39,arp_tha=ff:ff:ff:ff:ff:ff bridge(\"br-int\") ---------------- 0. No match. >>>> received packet on unknown port 331 <<<< drop Final flow: unchanged Megaflow: recirc_id=0,eth,arp,in_port=331,dl_src=00:00:00:25:eb:39 Datapath actions: drop 2022-11-13T08:44:34.077Z|00224|ofproto_dpif_xlate(handler5)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,in_port=13483,vlan_tci=0x0000,dl_src=00:00:00:59:ef:13,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.152.3,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:59:ef:13,arp_tha=ff:ff:ff:ff:ff:ff \u8981\u89e3\u51b3\u6b64\u95ee\u9898\uff0c\u53ef\u4fee\u6539 OVN NB \u9009\u9879 bcast_arp_req_flood \u4e3a false \uff1a kubectl ko nbctl set NB_Global . options:bcast_arp_req_flood = false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Underlay \u7f51\u7edc\u5b89\u88c5"},{"location":"start/underlay/#underlay","text":"\u9ed8\u8ba4\u60c5\u51b5\u4e0b Kube-OVN \u7684\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528 Geneve \u5bf9\u8de8\u4e3b\u673a\u6d41\u91cf\u8fdb\u884c\u5c01\u88c5\uff0c\u5728\u57fa\u7840\u8bbe\u65bd\u4e4b\u4e0a\u62bd\u8c61\u51fa\u4e00\u5c42\u865a\u62df\u7684 Overlay \u7f51\u7edc\u3002 \u5bf9\u4e8e\u5e0c\u671b\u5bb9\u5668\u7f51\u7edc\u76f4\u63a5\u4f7f\u7528\u7269\u7406\u7f51\u7edc\u5730\u5740\u6bb5\u60c5\u51b5\uff0c\u53ef\u4ee5\u5c06 Kube-OVN \u7684\u9ed8\u8ba4\u5b50\u7f51\u5de5\u4f5c\u5728 Underlay \u6a21\u5f0f\uff0c\u53ef\u4ee5\u76f4\u63a5\u7ed9\u5bb9\u5668\u5206\u914d\u7269\u7406\u7f51\u7edc\u4e2d\u7684\u5730\u5740\u8d44\u6e90\uff0c\u8fbe\u5230\u66f4\u597d\u7684\u6027\u80fd\u4ee5\u53ca\u548c\u7269\u7406\u7f51\u7edc\u7684\u8fde\u901a\u6027\u3002","title":"Underlay \u7f51\u7edc\u5b89\u88c5"},{"location":"start/underlay/#_1","text":"\u7531\u4e8e\u8be5\u6a21\u5f0f\u4e0b\u5bb9\u5668\u7f51\u7edc\u76f4\u63a5\u4f7f\u7528\u7269\u7406\u7f51\u7edc\u8fdb\u884c\u4e8c\u5c42\u5305\u8f6c\u53d1\uff0cOverlay \u6a21\u5f0f\u4e0b\u7684 SNAT/EIP\uff0c \u5206\u5e03\u5f0f\u7f51\u5173/\u96c6\u4e2d\u5f0f\u7f51\u5173\u7b49 L3 \u529f\u80fd\u65e0\u6cd5\u4f7f\u7528\uff0cVPC \u7ea7\u522b\u7684\u9694\u79bb\u4e5f\u65e0\u6cd5\u5bf9 Underlay \u5b50\u7f51\u751f\u6548\u3002","title":"\u529f\u80fd\u9650\u5236"},{"location":"start/underlay/#macvlan","text":"Kube-OVN \u7684 Underlay \u6a21\u5f0f\u548c Macvlan \u5de5\u4f5c\u6a21\u5f0f\u5341\u5206\u7c7b\u4f3c\uff0c\u5728\u529f\u80fd\u548c\u6027\u80fd\u4e0a\u4e3b\u8981\u6709\u4ee5\u4e0b\u51e0\u4e2a\u533a\u522b\uff1a \u7531\u4e8e Macvlan \u7684\u5185\u6838\u8def\u5f84\u66f4\u77ed\uff0c\u5e76\u4e14\u4e0d\u9700\u8981 OVS \u5bf9\u6570\u636e\u5305\u8fdb\u884c\u5904\u7406\uff0cMacvlan \u5728\u541e\u5410\u91cf\u548c\u5ef6\u8fdf\u6027\u80fd\u6307\u6807\u4e0a\u8868\u73b0\u4f1a\u66f4\u597d\u3002 Kube-OVN \u901a\u8fc7\u6d41\u8868\u63d0\u4f9b\u4e86 arp-proxy \u529f\u80fd\uff0c\u53ef\u4ee5\u7f13\u89e3\u5927\u89c4\u6a21\u7f51\u7edc\u4e0b\u7684 arp \u5e7f\u64ad\u98ce\u66b4\u98ce\u9669\u3002 \u7531\u4e8e Macvlan \u5de5\u4f5c\u5728\u5185\u6838\u5e95\u5c42\uff0c\u4f1a\u7ed5\u8fc7\u5bbf\u4e3b\u673a\u7684 netfilter\uff0cService \u548c NetworkPolicy \u529f\u80fd\u9700\u8981\u989d\u5916\u5f00\u53d1\u3002Kube-OVN \u901a\u8fc7 OVS \u6d41\u8868\u63d0\u4f9b\u4e86 Service \u548c NetworkPolicy \u7684\u80fd\u529b\u3002 Kube-OVN \u7684 Underlay \u6a21\u5f0f\u76f8\u6bd4 Macvlan \u989d\u5916\u63d0\u4f9b\u4e86\u5730\u5740\u7ba1\u7406\uff0c\u56fa\u5b9a IP \u548c QoS \u7b49\u529f\u80fd\u3002","title":"\u548c Macvlan \u6bd4\u8f83"},{"location":"start/underlay/#_2","text":"\u5728 Underlay \u6a21\u5f0f\u4e0b\uff0cOVS \u5c06\u4f1a\u6865\u63a5\u4e00\u4e2a\u8282\u70b9\u7f51\u5361\u5230 OVS \u7f51\u6865\uff0c\u5e76\u5c06\u6570\u636e\u5305\u76f4\u63a5\u901a\u8fc7\u8be5\u8282\u70b9\u7f51\u5361\u5bf9\u5916\u53d1\u9001\uff0cL2/L3 \u5c42\u9762\u7684\u8f6c\u53d1\u80fd\u529b\u9700\u8981\u4f9d\u8d56\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u3002 \u9700\u8981\u9884\u5148\u5728\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u914d\u7f6e\u5bf9\u5e94\u7684\u7f51\u5173\u3001Vlan \u548c\u5b89\u5168\u7b56\u7565\u7b49\u914d\u7f6e\u3002 \u5bf9\u4e8e OpenStack \u7684 VM \u73af\u5883\uff0c\u9700\u8981\u5c06\u5bf9\u5e94\u7f51\u7edc\u7aef\u53e3\u7684 PortSecurity \u5173\u95ed\u3002 \u5bf9\u4e8e VMware \u7684 vSwitch \u7f51\u7edc\uff0c\u9700\u8981\u5c06 MAC Address Changes , Forged Transmits \u548c Promiscuous Mode Operation \u8bbe\u7f6e\u4e3a allow \u3002 \u5bf9\u4e8e Hyper-V \u865a\u62df\u5316\uff0c\u9700\u8981\u5f00\u542f\u865a\u62df\u673a\u7f51\u5361\u9ad8\u7ea7\u529f\u80fd\u4e2d\u7684 MAC Address Spoofing \u3002 \u516c\u6709\u4e91\uff0c\u4f8b\u5982 AWS\u3001GCE\u3001\u963f\u91cc\u4e91\u7b49\u7531\u4e8e\u4e0d\u652f\u6301\u7528\u6237\u81ea\u5b9a\u4e49 Mac \u65e0\u6cd5\u652f\u6301 Underlay \u6a21\u5f0f\u7f51\u7edc\uff0c\u5728\u8fd9\u79cd\u573a\u666f\u4e0b\u5982\u679c\u60f3\u4f7f\u7528 Underlay \u63a8\u8350\u4f7f\u7528\u5bf9\u5e94\u516c\u6709\u4e91\u5382\u5546\u63d0\u4f9b\u7684 VPC-CNI\u3002 \u6865\u63a5\u7f51\u5361\u4e0d\u80fd\u4e3a Linux Bridge\u3002 \u5bf9\u4e8e\u7ba1\u7406\u7f51\u548c\u5bb9\u5668\u7f51\u4f7f\u7528\u540c\u4e00\u4e2a\u7f51\u5361\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u7684 Mac \u5730\u5740\u3001IP \u5730\u5740\u3001\u8def\u7531\u4ee5\u53ca MTU \u5c06\u8f6c\u79fb\u6216\u590d\u5236\u81f3\u5bf9\u5e94\u7684 OVS Bridge\uff0c \u4ee5\u652f\u6301\u5355\u7f51\u5361\u90e8\u7f72 Underlay \u7f51\u7edc\u3002OVS Bridge \u540d\u79f0\u683c\u5f0f\u4e3a br-PROVIDER_NAME \uff0c PROVIDER_NAME \u4e3a Provider \u7f51\u7edc\u540d\u79f0\uff08\u9ed8\u8ba4\u4e3a provider\uff09\u3002","title":"\u73af\u5883\u8981\u6c42"},{"location":"start/underlay/#_3","text":"\u8be5\u90e8\u7f72\u6a21\u5f0f\u5c06\u9ed8\u8ba4\u5b50\u7f51\u8bbe\u7f6e\u4e3a Underlay \u6a21\u5f0f\uff0c\u6240\u6709\u672a\u6307\u5b9a\u5b50\u7f51\u7684 Pod \u5747\u4f1a\u9ed8\u8ba4\u8fd0\u884c\u5728 Underlay \u7f51\u7edc\u4e2d\u3002","title":"\u90e8\u7f72\u65f6\u6307\u5b9a\u7f51\u7edc\u6a21\u5f0f"},{"location":"start/underlay/#_4","text":"wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c"},{"location":"start/underlay/#_5","text":"ENABLE_ARP_DETECT_IP_CONFLICT # \u5982\u6709\u9700\u8981\uff0c\u53ef\u4ee5\u9009\u62e9\u5173\u95ed vlan \u7f51\u7edc arp \u51b2\u7a81\u68c0\u6d4b NETWORK_TYPE # \u8bbe\u7f6e\u4e3a vlan VLAN_INTERFACE_NAME # \u8bbe\u7f6e\u4e3a\u5bbf\u4e3b\u673a\u4e0a\u627f\u62c5\u5bb9\u5668\u6d41\u91cf\u7684\u7f51\u5361\uff0c\u4f8b\u5982 eth1 VLAN_ID # \u4ea4\u6362\u673a\u6240\u63a5\u53d7\u7684 VLAN Tag\uff0c\u82e5\u8bbe\u7f6e\u4e3a 0 \u5219\u4e0d\u505a VLAN \u5c01\u88c5 POD_CIDR # \u8bbe\u7f6e\u4e3a\u7269\u7406\u7f51\u7edc CIDR\uff0c \u4f8b\u5982 192.168.1.0/24 POD_GATEWAY # \u8bbe\u7f6e\u4e3a\u7269\u7406\u7f51\u7edc\u7f51\u5173\uff0c\u4f8b\u5982 192.168.1.1 EXCLUDE_IPS # \u6392\u9664\u8303\u56f4\uff0c\u907f\u514d\u5bb9\u5668\u7f51\u6bb5\u548c\u7269\u7406\u7f51\u7edc\u5df2\u7528 IP \u51b2\u7a81\uff0c\u4f8b\u5982 192.168.1.1..192.168.1.100 ENABLE_LB # \u5982\u679c Underlay \u5b50\u7f51\u9700\u8981\u4f7f\u7528 Service \u9700\u8981\u8bbe\u7f6e\u4e3a true EXCHANGE_LINK_NAME # \u662f\u5426\u4ea4\u6362\u9ed8\u8ba4 provider-network \u4e0b OVS \u7f51\u6865\u548c\u6865\u63a5\u7f51\u5361\u7684\u540d\u5b57\uff0c\u9ed8\u8ba4\u4e3a false LS_DNAT_MOD_DL_DST # DNAT \u65f6\u662f\u5426\u5bf9 MAC \u5730\u5740\u8fdb\u884c\u8f6c\u6362\uff0c\u53ef\u52a0\u901f Service \u7684\u8bbf\u95ee\uff0c\u9ed8\u8ba4\u4e3a true","title":"\u4fee\u6539\u811a\u672c\u4e2d\u76f8\u5e94\u914d\u7f6e"},{"location":"start/underlay/#_6","text":"bash install.sh","title":"\u8fd0\u884c\u5b89\u88c5\u811a\u672c"},{"location":"start/underlay/#crd-underlay","text":"\u8be5\u65b9\u5f0f\u53ef\u5728\u5b89\u88c5\u540e\u52a8\u6001\u7684\u521b\u5efa\u67d0\u4e2a Underlay \u5b50\u7f51\u4f9b Pod \u4f7f\u7528\u3002\u9700\u8981\u914d\u7f6e ProviderNetwork \uff0c Vlan \u548c Subnet \u4e09\u79cd\u81ea\u5b9a\u4e49\u8d44\u6e90\u3002","title":"\u901a\u8fc7 CRD \u52a8\u6001\u521b\u5efa Underlay \u7f51\u7edc"},{"location":"start/underlay/#providernetwork","text":"ProviderNetwork \u63d0\u4f9b\u4e86\u4e3b\u673a\u7f51\u5361\u5230\u7269\u7406\u7f51\u7edc\u6620\u5c04\u7684\u62bd\u8c61\uff0c\u5c06\u540c\u5c5e\u4e00\u4e2a\u7f51\u7edc\u7684\u7f51\u5361\u8fdb\u884c\u7edf\u4e00\u7ba1\u7406\uff0c \u5e76\u89e3\u51b3\u5728\u590d\u6742\u73af\u5883\u4e0b\u540c\u673a\u5668\u591a\u7f51\u5361\u3001\u7f51\u5361\u540d\u4e0d\u4e00\u81f4\u3001\u5bf9\u5e94 Underlay \u7f51\u7edc\u4e0d\u4e00\u81f4\u7b49\u60c5\u51b5\u4e0b\u7684\u914d\u7f6e\u95ee\u9898\u3002 \u521b\u5efa\u5982\u4e0b ProviderNetwork \u5e76\u5e94\u7528: apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: net1 spec: defaultInterface: eth1 customInterfaces: - interface: eth2 nodes: - node1 excludeNodes: - node2 \u6ce8\u610f\uff1aProviderNetwork \u8d44\u6e90\u540d\u79f0\u7684\u957f\u5ea6\u4e0d\u5f97\u8d85\u8fc7 12\u3002 defaultInterface : \u4e3a\u9ed8\u8ba4\u4f7f\u7528\u7684\u8282\u70b9\u7f51\u5361\u540d\u79f0\u3002 ProviderNetwork \u521b\u5efa\u6210\u529f\u540e\uff0c\u5404\u8282\u70b9\uff08\u9664 excludeNodes \u5916\uff09\u4e2d\u4f1a\u521b\u5efa\u540d\u4e3a br-net1\uff08\u683c\u5f0f\u4e3a br-NAME \uff09\u7684 OVS \u7f51\u6865\uff0c\u5e76\u5c06\u6307\u5b9a\u7684\u8282\u70b9\u7f51\u5361\u6865\u63a5\u81f3\u6b64\u7f51\u6865\u3002 customInterfaces : \u4e3a\u53ef\u9009\u9879\uff0c\u53ef\u9488\u5bf9\u7279\u5b9a\u8282\u70b9\u6307\u5b9a\u9700\u8981\u4f7f\u7528\u7684\u7f51\u5361\u3002 excludeNodes : \u53ef\u9009\u9879\uff0c\u7528\u4e8e\u6307\u5b9a\u4e0d\u6865\u63a5\u7f51\u5361\u7684\u8282\u70b9\u3002\u8be5\u5217\u8868\u4e2d\u7684\u8282\u70b9\u4f1a\u88ab\u6dfb\u52a0 net1.provider-network.ovn.kubernetes.io/exclude=true \u6807\u7b7e\u3002 \u5176\u5b83\u8282\u70b9\u4f1a\u88ab\u6dfb\u52a0\u5982\u4e0b\u6807\u7b7e\uff1a Key Value \u63cf\u8ff0 net1.provider-network.ovn.kubernetes.io/ready true \u8282\u70b9\u4e2d\u7684\u6865\u63a5\u5de5\u4f5c\u5df2\u5b8c\u6210\uff0cProviderNetwork \u5728\u8282\u70b9\u4e2d\u53ef\u7528 net1.provider-network.ovn.kubernetes.io/interface eth1 \u8282\u70b9\u4e2d\u88ab\u6865\u63a5\u7684\u7f51\u5361\u7684\u540d\u79f0 net1.provider-network.ovn.kubernetes.io/mtu 1500 \u8282\u70b9\u4e2d\u88ab\u6865\u63a5\u7684\u7f51\u5361\u7684 MTU \u5982\u679c\u8282\u70b9\u7f51\u5361\u4e0a\u5df2\u7ecf\u914d\u7f6e\u4e86 IP\uff0c\u5219 IP \u5730\u5740\u548c\u7f51\u5361\u4e0a\u7684\u8def\u7531\u4f1a\u88ab\u8f6c\u79fb\u81f3\u5bf9\u5e94\u7684 OVS \u7f51\u6865\u3002","title":"\u521b\u5efa ProviderNetwork"},{"location":"start/underlay/#vlan","text":"Vlan \u63d0\u4f9b\u4e86\u5c06 Vlan Tag \u548c ProviderNetwork \u8fdb\u884c\u7ed1\u5b9a\u7684\u80fd\u529b\u3002 \u521b\u5efa\u5982\u4e0b VLAN \u5e76\u5e94\u7528\uff1a apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan1 spec: id: 0 provider: net1 id : \u4e3a VLAN ID/Tag\uff0cKube-OVN \u4f1a\u5bf9\u5bf9\u8be5 Vlan \u4e0b\u7684\u6d41\u91cf\u589e\u52a0 Vlan \u6807\u7b7e\uff0c\u4e3a 0 \u65f6\u4e0d\u589e\u52a0\u4efb\u4f55\u6807\u7b7e\u3002 provider : \u4e3a\u9700\u8981\u4f7f\u7528\u7684 ProviderNetwork \u8d44\u6e90\u7684\u540d\u79f0\u3002\u591a\u4e2a VLAN \u53ef\u4ee5\u5f15\u7528\u540c\u4e00\u4e2a ProviderNetwork\u3002","title":"\u521b\u5efa VLAN"},{"location":"start/underlay/#subnet","text":"\u5c06 Vlan \u548c\u4e00\u4e2a\u5b50\u7f51\u7ed1\u5b9a\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 \u5c06 vlan \u7684\u503c\u6307\u5b9a\u4e3a\u9700\u8981\u4f7f\u7528\u7684 VLAN \u540d\u79f0\u5373\u53ef\u3002\u591a\u4e2a Subnet \u53ef\u4ee5\u5f15\u7528\u540c\u4e00\u4e2a VLAN\u3002","title":"\u521b\u5efa Subnet"},{"location":"start/underlay/#_7","text":"\u53ef\u6309\u6b63\u5e38\u5bb9\u5668\u521b\u5efa\u65b9\u5f0f\u8fdb\u884c\u521b\u5efa\uff0c\u67e5\u770b\u5bb9\u5668 IP \u662f\u5426\u5728\u89c4\u5b9a\u8303\u56f4\u5185\uff0c\u4ee5\u53ca\u5bb9\u5668\u662f\u5426\u53ef\u4ee5\u548c\u7269\u7406\u7f51\u7edc\u4e92\u901a\u3002 \u5982\u6709\u56fa\u5b9a IP \u9700\u6c42\uff0c\u53ef\u53c2\u8003 Pod \u56fa\u5b9a IP \u548c Mac","title":"\u5bb9\u5668\u521b\u5efa"},{"location":"start/underlay/#_8","text":"\u5bf9\u4e8e\u7269\u7406\u7f51\u7edc\u4e0d\u5b58\u5728\u7f51\u5173\u7684\u60c5\u51b5\uff0cKube-OVN \u652f\u6301\u5728 Underlay \u6a21\u5f0f\u7684\u5b50\u7f51\u4e2d\u914d\u7f6e\u4f7f\u7528\u903b\u8f91\u7f51\u5173\u3002 \u82e5\u8981\u4f7f\u7528\u6b64\u529f\u80fd\uff0c\u8bbe\u7f6e\u5b50\u7f51\u7684 spec.logicalGateway \u4e3a true \u5373\u53ef\uff1a apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 172.17.0.0/16 gateway: 172.17.0.1 vlan: vlan1 logicalGateway: true \u5f00\u542f\u6b64\u529f\u80fd\u540e\uff0cPod \u4e0d\u4f7f\u7528\u5916\u90e8\u7f51\u5173\uff0c\u800c\u662f\u4f7f\u7528 Kube-OVN \u521b\u5efa\u7684\u903b\u8f91\u8def\u7531\u5668\uff08Logical Router\uff09\u5bf9\u4e8e\u8de8\u7f51\u6bb5\u901a\u4fe1\u8fdb\u884c\u8f6c\u53d1\u3002","title":"\u4f7f\u7528\u903b\u8f91\u7f51\u5173"},{"location":"start/underlay/#underlay-overlay","text":"\u5982\u679c\u4e00\u4e2a\u96c6\u7fa4\u540c\u65f6\u5b58\u5728 Underlay \u548c Overlay \u5b50\u7f51\uff0c\u9ed8\u8ba4\u60c5\u51b5\u4e0b Overlay \u5b50\u7f51\u4e0b\u7684 Pod \u53ef\u4ee5\u901a\u8fc7\u7f51\u5173\u4ee5 NAT \u7684\u65b9\u5f0f\u8bbf\u95ee Underlay \u5b50\u7f51\u4e0b\u7684 Pod IP\u3002 \u5728 Underlay \u5b50\u7f51\u7684 Pod \u770b\u6765 Overlay \u5b50\u7f51\u7684\u5730\u5740\u662f\u4e00\u4e2a\u5916\u90e8\u7684\u5730\u5740\uff0c\u9700\u8981\u901a\u8fc7\u5e95\u5c42\u7269\u7406\u8bbe\u5907\u53bb\u8f6c\u53d1\uff0c\u4f46\u5e95\u5c42\u7269\u7406\u8bbe\u5907\u5e76\u4e0d\u6e05\u695a Overlay \u5b50\u7f51\u7684\u5730\u5740\u65e0\u6cd5\u8fdb\u884c\u8f6c\u53d1\u3002 \u56e0\u6b64 Underlay \u5b50\u7f51\u4e0b\u7684 Pod \u65e0\u6cd5\u901a\u8fc7 Pod IP \u76f4\u63a5\u8bbf\u95ee Overlay \u5b50\u7f51\u7684 Pod\u3002 \u5982\u679c\u9700\u8981 Underlay \u548c Overlay \u4e92\u901a\u9700\u8981\u5c06\u5b50\u7f51\u7684 u2oInterconnection \u8bbe\u7f6e\u4e3a true \uff0c\u5728\u8fd9\u4e2a\u60c5\u51b5\u4e0b Kube-OVN \u4f1a\u989d\u5916\u4f7f\u7528\u4e00\u4e2a Underlay IP \u5c06 Underlay \u5b50\u7f51 \u548c ovn-cluster \u903b\u8f91\u8def\u7531\u5668\u8fde\u63a5\uff0c\u5e76\u8bbe\u7f6e\u5bf9\u5e94\u7684\u8def\u7531\u89c4\u5219\u5b9e\u73b0\u4e92\u901a\u3002 \u548c\u903b\u8f91\u7f51\u5173\u4e0d\u540c\uff0c\u8be5\u65b9\u6848\u53ea\u4f1a\u8fde\u63a5 Kube-OVN \u5185\u90e8\u7684 Underlay \u548c Overlay \u5b50\u7f51\uff0c\u5176\u4ed6\u8bbf\u95ee\u5916\u7f51\u7684\u6d41\u91cf\u8fd8\u662f\u4f1a\u901a\u8fc7\u7269\u7406\u7f51\u5173\u8fdb\u884c\u8f6c\u53d1\u3002","title":"Underlay \u548c Overlay \u7f51\u7edc\u4e92\u901a"},{"location":"start/underlay/#ip","text":"\u5f00\u542f\u4e92\u901a\u529f\u80fd\u540e\uff0c\u4f1a\u968f\u673a\u4ece subnet \u5185\u7684\u53d6\u4e00\u4e2a IP \u4f5c\u4e3a\u903b\u8f91\u7f51\u5173\uff0c\u5982\u679c\u9700\u8981\u6307\u5b9a Underlay Subnet \u7684\u903b\u8f91\u7f51\u5173\u53ef\u4ee5\u6307\u5b9a\u5b57\u6bb5 u2oInterconnectionIP \u3002","title":"\u6307\u5b9a\u903b\u8f91\u7f51\u5173 IP"},{"location":"start/underlay/#underlay-subnet-vpc","text":"\u9ed8\u8ba4\u60c5\u51b5\u4e0b Underlay Subnet \u4f1a\u548c\u9ed8\u8ba4 VPC \u4e0a\u7684 Overlay Subnet \u4e92\u901a\uff0c\u5982\u679c\u8981\u6307\u5b9a\u548c\u67d0\u4e2a VPC \u4e92\u901a\uff0c\u5728 u2oInterconnection \u8bbe\u7f6e\u4e3a true \u540e\uff0c\u6307\u5b9a subnet.spec.vpc \u5b57\u6bb5\u4e3a\u8be5 VPC \u540d\u5b57\u5373\u53ef\u3002","title":"\u6307\u5b9a Underlay Subnet \u8fde\u63a5\u7684\u81ea\u5b9a\u4e49 VPC"},{"location":"start/underlay/#_9","text":"\u5982\u679c\u60a8\u4f7f\u7528\u7684\u8282\u70b9\u7f51\u5361\u4e0a\u914d\u7f6e\u6709 IP \u5730\u5740\uff0c\u4e14\u64cd\u4f5c\u7cfb\u7edf\u662f Ubuntu \u5e76\u901a\u8fc7 Netplan \u914d\u7f6e\u7f51\u7edc\uff0c\u5efa\u8bae\u60a8\u5c06 Netplan \u7684 renderer \u8bbe\u7f6e\u4e3a NetworkManager\uff0c\u5e76\u4e3a\u8282\u70b9\u7f51\u5361\u914d\u7f6e\u9759\u6001 IP \u5730\u5740\uff08\u5173\u95ed DHCP\uff09\uff1a network : renderer : NetworkManager ethernets : eth0 : dhcp4 : no addresses : - 172.16.143.129/24 version : 2 \u82e5\u8282\u70b9\u7f51\u7edc\u7ba1\u7406\u670d\u52a1\u4e3a NetworkManager\uff0c\u5728\u4f7f\u7528\u8282\u70b9\u7f51\u5361\u521b\u5efa ProviderNetwork \u540e\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u4ece NetworkManager \u7ba1\u7406\u5217\u8868\u4e2d\u79fb\u9664\uff08managed \u5c5e\u6027\u4e3a no\uff09\uff1a root@ubuntu:~# nmcli device status DEVICE TYPE STATE CONNECTION eth0 ethernet unmanaged netplan-eth0 \u5982\u679c\u60a8\u8981\u4fee\u6539\u7f51\u5361\u7684 IP \u6216\u8def\u7531\u914d\u7f6e\uff0c\u9700\u8981\u624b\u52a8\u5c06\u7f51\u5361\u91cd\u65b0\u52a0\u5165 NetworkManager \u7ba1\u7406\u5217\u8868\uff1a nmcli device set eth0 managed yes \u6267\u884c\u4ee5\u4e0a\u547d\u4ee4\u540e\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u4e0a\u7684 IP \u53ca\u8def\u7531\u91cd\u65b0\u8f6c\u79fb\u81f3 OVS \u7f51\u6865\uff0c\u5e76\u518d\u6b21\u5c06\u7f51\u5361\u4ece NetworkManager \u7ba1\u7406\u5217\u8868\u4e2d\u79fb\u9664\u3002 \u6ce8\u610f \uff1a\u8282\u70b9\u7f51\u5361\u914d\u7f6e\u7684\u52a8\u6001\u4fee\u6539\u4ec5\u652f\u6301 IP \u548c\u8def\u7531\uff0c\u4e0d\u652f\u6301 MAC \u5730\u5740\u7684\u4fee\u6539\u3002","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"start/underlay/#_10","text":"","title":"\u5df2\u77e5\u95ee\u9898"},{"location":"start/underlay/#hairpin-pod","text":"\u5f53\u7269\u7406\u7f51\u7edc\u5f00\u542f hairpin \u6216\u7c7b\u4f3c\u884c\u4e3a\u65f6\uff0c\u53ef\u80fd\u51fa\u73b0\u521b\u5efa Pod \u65f6\u7f51\u5173\u68c0\u67e5\u5931\u8d25\u3001Pod \u7f51\u7edc\u901a\u4fe1\u5f02\u5e38\u7b49\u95ee\u9898\u3002\u8fd9\u662f\u56e0\u4e3a OVS \u7f51\u6865\u9ed8\u8ba4\u7684 MAC \u5b66\u4e60\u529f\u80fd\u4e0d\u652f\u6301\u8fd9\u79cd\u7f51\u7edc\u73af\u5883\u3002 \u8981\u89e3\u51b3\u6b64\u95ee\u9898\uff0c\u9700\u8981\u5173\u95ed hairpin\uff08\u6216\u4fee\u6539\u7269\u7406\u7f51\u7edc\u7684\u76f8\u5173\u914d\u7f6e\uff09\uff0c\u6216\u66f4\u65b0 Kube-OVN \u7248\u672c\u3002","title":"\u7269\u7406\u7f51\u7edc\u5f00\u542f hairpin \u65f6 Pod \u7f51\u7edc\u5f02\u5e38"},{"location":"start/underlay/#pod-pod","text":"\u82e5\u540c\u4e00\u4e2a\u8282\u70b9\u4e0a\u8fd0\u884c\u7684 Pod \u6570\u91cf\u8f83\u591a\uff08\u5927\u4e8e 300\uff09\uff0c\u53ef\u80fd\u4f1a\u51fa\u73b0 ARP \u5e7f\u64ad\u5305\u7684 OVS \u6d41\u8868 resubmit \u6b21\u6570\u8d85\u8fc7\u4e0a\u9650\u5bfc\u81f4\u4e22\u5305\u7684\u73b0\u8c61\uff1a 2022-11-13T08:43:46.782Z|00222|ofproto_dpif_upcall(handler5)|WARN|Flow: arp,in_port=331,vlan_tci=0x0000,dl_src=00:00:00:25:eb:39,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.131.240,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:25:eb:39,arp_tha=ff:ff:ff:ff:ff:ff bridge(\"br-int\") ---------------- 0. No match. >>>> received packet on unknown port 331 <<<< drop Final flow: unchanged Megaflow: recirc_id=0,eth,arp,in_port=331,dl_src=00:00:00:25:eb:39 Datapath actions: drop 2022-11-13T08:44:34.077Z|00224|ofproto_dpif_xlate(handler5)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,in_port=13483,vlan_tci=0x0000,dl_src=00:00:00:59:ef:13,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.152.3,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:59:ef:13,arp_tha=ff:ff:ff:ff:ff:ff \u8981\u89e3\u51b3\u6b64\u95ee\u9898\uff0c\u53ef\u4fee\u6539 OVN NB \u9009\u9879 bcast_arp_req_flood \u4e3a false \uff1a kubectl ko nbctl set NB_Global . options:bcast_arp_req_flood = false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Pod \u6570\u91cf\u8f83\u591a\u65f6\u65b0\u5efa Pod \u7f51\u5173\u68c0\u67e5\u5931\u8d25"},{"location":"start/uninstall/","text":"\u5378\u8f7d \u00b6 \u5982\u679c\u9700\u8981\u5220\u9664 Kube-OVN \u5e76\u66f4\u6362\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u8bf7\u6309\u7167\u4e0b\u5217\u7684\u6b65\u9aa4\u5220\u9664\u5bf9\u5e94\u7684 Kube-OVN \u7ec4\u4ef6\u4ee5\u53ca OVS \u914d\u7f6e\uff0c\u4ee5\u907f\u514d\u5bf9\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4ea7\u751f\u5e72\u6270\u3002 \u4e5f\u6b22\u8fce\u63d0 issue \u8054\u7cfb\u6211\u4eec\u53cd\u9988\u4e0d\u4f7f\u7528 Kube-OVN \u7684\u539f\u56e0\u5e2e\u52a9\u6211\u4eec\u6539\u8fdb\u3002 \u5220\u9664\u5728 Kubernetes \u4e2d\u521b\u5efa\u7684\u8d44\u6e90 \u00b6 \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u6267\u884c\u811a\u672c\u5220\u9664\u5728 Kubernetes \u4e2d\u521b\u5efa\u7684\u8d44\u6e90\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/cleanup.sh bash cleanup.sh \u6e05\u7406\u4e3b\u673a\u4e0a\u7684\u65e5\u5fd7\u548c\u914d\u7f6e\u6587\u4ef6 \u00b6 \u5728\u6bcf\u53f0\u673a\u5668\u4e0a\u6267\u884c\u4e0b\u5217\u64cd\u4f5c\uff0c\u6e05\u7406 ovsdb \u4ee5\u53ca openvswitch \u4fdd\u5b58\u7684\u914d\u7f6e\uff1a rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn rm -fr /var/log/kube-ovn \u91cd\u542f\u8282\u70b9 \u00b6 \u91cd\u542f\u673a\u5668\u786e\u4fdd\u5bf9\u5e94\u7684\u7f51\u5361\u4fe1\u606f\uff0ciptable/ipset \u89c4\u5219\u5f97\u4ee5\u6e05\u9664\uff0c\u907f\u514d\u5bf9\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u7684\u5f71\u54cd\uff1a reboot \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5378\u8f7d"},{"location":"start/uninstall/#_1","text":"\u5982\u679c\u9700\u8981\u5220\u9664 Kube-OVN \u5e76\u66f4\u6362\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u8bf7\u6309\u7167\u4e0b\u5217\u7684\u6b65\u9aa4\u5220\u9664\u5bf9\u5e94\u7684 Kube-OVN \u7ec4\u4ef6\u4ee5\u53ca OVS \u914d\u7f6e\uff0c\u4ee5\u907f\u514d\u5bf9\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4ea7\u751f\u5e72\u6270\u3002 \u4e5f\u6b22\u8fce\u63d0 issue \u8054\u7cfb\u6211\u4eec\u53cd\u9988\u4e0d\u4f7f\u7528 Kube-OVN \u7684\u539f\u56e0\u5e2e\u52a9\u6211\u4eec\u6539\u8fdb\u3002","title":"\u5378\u8f7d"},{"location":"start/uninstall/#kubernetes","text":"\u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u6267\u884c\u811a\u672c\u5220\u9664\u5728 Kubernetes \u4e2d\u521b\u5efa\u7684\u8d44\u6e90\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/cleanup.sh bash cleanup.sh","title":"\u5220\u9664\u5728 Kubernetes \u4e2d\u521b\u5efa\u7684\u8d44\u6e90"},{"location":"start/uninstall/#_2","text":"\u5728\u6bcf\u53f0\u673a\u5668\u4e0a\u6267\u884c\u4e0b\u5217\u64cd\u4f5c\uff0c\u6e05\u7406 ovsdb \u4ee5\u53ca openvswitch \u4fdd\u5b58\u7684\u914d\u7f6e\uff1a rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn rm -fr /var/log/kube-ovn","title":"\u6e05\u7406\u4e3b\u673a\u4e0a\u7684\u65e5\u5fd7\u548c\u914d\u7f6e\u6587\u4ef6"},{"location":"start/uninstall/#_3","text":"\u91cd\u542f\u673a\u5668\u786e\u4fdd\u5bf9\u5e94\u7684\u7f51\u5361\u4fe1\u606f\uff0ciptable/ipset \u89c4\u5219\u5f97\u4ee5\u6e05\u9664\uff0c\u907f\u514d\u5bf9\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u7684\u5f71\u54cd\uff1a reboot \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u91cd\u542f\u8282\u70b9"},{"location":"en/","text":"Kube-OVN \u00b6 Kube-OVN, a CNCF Sandbox Project, bridges the SDN into Cloud Native. It offers an advanced Container Network Fabric for Enterprises with the most functions, extreme performance and the easiest operation. Most Functions: If you miss the rich networking capabilities of the SDN age but are struggling to find them in the cloud-native age, Kube-OVN should be your best choice. Leveraging the proven capabilities of OVS/OVN in the SDN, Kube-OVN brings the rich capabilities of network virtualization to the cloud-native space. It currently supports Subnet Management , Static IP Allocation , Distributed/Centralized Gateways , Underlay/Overlay Hybrid Networks , VPC Multi-Tenant Networks , Cross-Cluster Interconnect , QoS Management , Multi-NIC Management , ACL , Traffic Mirroring , ARM Support, Windows Support , and many more. Extreme Performance: If you're concerned about the additional performance loss associated with container networks, then take a look at How Kube-OVN is doing everything it can to optimize performance . In the data plane, through a series of carefully optimized flow and kernel optimizations, and with emerging technologies such as eBPF , DPDK and SmartNIC Offload , Kube-OVN can approximate or exceed host network performance in terms of latency and throughput. In the control plane, Kube-OVN can support large-scale clusters of thousands of nodes and tens of thousands of Pods through the tailoring of OVN upstream flow tables and the use and tuning of various caching techniques. In addition, Kube-OVN is continuously optimizing the usage of resources such as CPU and memory to accommodate resource-limited scenarios such as the edge. Easiest Operation: If you're worried about container network operations, Kube-OVN has a number of built-in tools to help you simplify your operations. Kube-OVN provides one-click installation scripts to help users quickly build production-ready container networks. Also built-in rich monitoring metrics and Grafana dashboard help users to quickly set up monitoring system. Powerful command line tools simplify daily operations and maintenance for users. By combining with Cilium , users can enhance the observability of their networks with eBPF capabilities. In addition, the ability to mirror traffic makes it easy to customize traffic monitoring and interface with traditional NPM systems. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVERVIEW"},{"location":"en/#kube-ovn","text":"Kube-OVN, a CNCF Sandbox Project, bridges the SDN into Cloud Native. It offers an advanced Container Network Fabric for Enterprises with the most functions, extreme performance and the easiest operation. Most Functions: If you miss the rich networking capabilities of the SDN age but are struggling to find them in the cloud-native age, Kube-OVN should be your best choice. Leveraging the proven capabilities of OVS/OVN in the SDN, Kube-OVN brings the rich capabilities of network virtualization to the cloud-native space. It currently supports Subnet Management , Static IP Allocation , Distributed/Centralized Gateways , Underlay/Overlay Hybrid Networks , VPC Multi-Tenant Networks , Cross-Cluster Interconnect , QoS Management , Multi-NIC Management , ACL , Traffic Mirroring , ARM Support, Windows Support , and many more. Extreme Performance: If you're concerned about the additional performance loss associated with container networks, then take a look at How Kube-OVN is doing everything it can to optimize performance . In the data plane, through a series of carefully optimized flow and kernel optimizations, and with emerging technologies such as eBPF , DPDK and SmartNIC Offload , Kube-OVN can approximate or exceed host network performance in terms of latency and throughput. In the control plane, Kube-OVN can support large-scale clusters of thousands of nodes and tens of thousands of Pods through the tailoring of OVN upstream flow tables and the use and tuning of various caching techniques. In addition, Kube-OVN is continuously optimizing the usage of resources such as CPU and memory to accommodate resource-limited scenarios such as the edge. Easiest Operation: If you're worried about container network operations, Kube-OVN has a number of built-in tools to help you simplify your operations. Kube-OVN provides one-click installation scripts to help users quickly build production-ready container networks. Also built-in rich monitoring metrics and Grafana dashboard help users to quickly set up monitoring system. Powerful command line tools simplify daily operations and maintenance for users. By combining with Cilium , users can enhance the observability of their networks with eBPF capabilities. In addition, the ability to mirror traffic makes it easy to customize traffic monitoring and interface with traditional NPM systems. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN"},{"location":"en/contact/","text":"Contact US \u00b6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"CONTACT US"},{"location":"en/contact/#contact-us","text":"\u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Contact US"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/","text":"Accelerate TCP Communication in Node with eBPF \u00b6 At some edge and 5G scenarios, there will be a lot of TCP communication between Pods on the same node. By using the open source istio-tcpip-bypass project from Intel, Pods can use the ability of eBPF to bypass the host's TCP/IP protocol stack and communicate directly through sockets, thereby greatly reducing latency and improving throughput. Basic Principle \u00b6 At present, two Pods on the same host need to go through a lot of network stacks, including TCP/IP, netfilter, OVS, etc., as shown in the following figure: istio-tcpip-bypass plugin can automatically analyze and identify TCP communication within the same host, and bypass the complex kernel stack so that socket data transmission can be performed directly to reduce network stack processing overhead, as shown in the following figure: Due to the fact that this component can automatically identify TCP communication within the same host and optimize it. In the Service Mesh environment based on the proxy mode, this component can also enhance the performance of Service Mesh. For more technical implementation details, please refer to Tanzu Service Mesh Acceleration using eBPF . Prerequisites \u00b6 eBPF requires a kernel version of at least 5.4.0-74-generic. It is recommended to use Ubuntu 20.04 and Linux 5.4.0-74-generic kernel version for testing. Experimental Steps \u00b6 Deploy two performance test Pods on the same node. If there are multiple machines in the cluster, you need to specify nodeSelector : # kubectl create deployment perf --image=kubeovn/perf:dev --replicas=2 deployment.apps/perf created # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES perf-7697bc6ddf-b2cpv 1 /1 Running 0 28s 100 .64.0.3 sealos <none> <none> perf-7697bc6ddf-p2xpt 1 /1 Running 0 28s 100 .64.0.2 sealos <none> <none> Enter one of the Pods to start the qperf server, and start the qperf client in another Pod for performance testing: # kubectl exec -it perf-7697bc6ddf-b2cpv sh / # qperf # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw Deploy the istio-tcpip-bypass plugin: kubectl apply -f https://raw.githubusercontent.com/intel/istio-tcpip-bypass/main/bypass-tcpip-daemonset.yaml Enter the perf client container again for performance testing: # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw Test Results \u00b6 According to the test results, the TCP latency will decrease by 40% ~ 60% under different packet sizes, and the throughput will increase by 40% ~ 80% when the packet size is greater than 1024 bytes. Packet Size (byte) eBPF tcp_lat (us) Default tcp_lat (us) eBPF tcp_bw (Mb/s) Default tcp_bw(Mb/s) 1 20.2 44.5 1.36 4.27 4 20.2 48.7 5.48 16.7 16 19.6 41.6 21.7 63.5 64 18.8 41.3 96.8 201 256 19.2 36 395 539 1024 18.3 42.4 1360 846 4096 16.5 62.6 4460 2430 16384 20.2 58.8 9600 6900 In the hardware environment under test, when the packet size is less than 512 bytes, the throughput indicator optimized by eBPF is lower than the throughput under the default configuration. This situation may be related to the TCP aggregation optimization of the network card under the default configuration. If the application scenario is sensitive to small packet throughput, you need to test in the corresponding environment Determine whether to enable eBPF optimization. We will also optimize the throughput of eBPF TCP small packet scenarios in the future. References \u00b6 istio-tcpip-bypass Deep Dive TCP/IP Bypass with eBPF in Service Mesh Tanzu Service Mesh Acceleration using eBPF \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Accelerate TCP Communication in Node with eBPF"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#accelerate-tcp-communication-in-node-with-ebpf","text":"At some edge and 5G scenarios, there will be a lot of TCP communication between Pods on the same node. By using the open source istio-tcpip-bypass project from Intel, Pods can use the ability of eBPF to bypass the host's TCP/IP protocol stack and communicate directly through sockets, thereby greatly reducing latency and improving throughput.","title":"Accelerate TCP Communication in Node with eBPF"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#basic-principle","text":"At present, two Pods on the same host need to go through a lot of network stacks, including TCP/IP, netfilter, OVS, etc., as shown in the following figure: istio-tcpip-bypass plugin can automatically analyze and identify TCP communication within the same host, and bypass the complex kernel stack so that socket data transmission can be performed directly to reduce network stack processing overhead, as shown in the following figure: Due to the fact that this component can automatically identify TCP communication within the same host and optimize it. In the Service Mesh environment based on the proxy mode, this component can also enhance the performance of Service Mesh. For more technical implementation details, please refer to Tanzu Service Mesh Acceleration using eBPF .","title":"Basic Principle"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#prerequisites","text":"eBPF requires a kernel version of at least 5.4.0-74-generic. It is recommended to use Ubuntu 20.04 and Linux 5.4.0-74-generic kernel version for testing.","title":"Prerequisites"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#experimental-steps","text":"Deploy two performance test Pods on the same node. If there are multiple machines in the cluster, you need to specify nodeSelector : # kubectl create deployment perf --image=kubeovn/perf:dev --replicas=2 deployment.apps/perf created # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES perf-7697bc6ddf-b2cpv 1 /1 Running 0 28s 100 .64.0.3 sealos <none> <none> perf-7697bc6ddf-p2xpt 1 /1 Running 0 28s 100 .64.0.2 sealos <none> <none> Enter one of the Pods to start the qperf server, and start the qperf client in another Pod for performance testing: # kubectl exec -it perf-7697bc6ddf-b2cpv sh / # qperf # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw Deploy the istio-tcpip-bypass plugin: kubectl apply -f https://raw.githubusercontent.com/intel/istio-tcpip-bypass/main/bypass-tcpip-daemonset.yaml Enter the perf client container again for performance testing: # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw","title":"Experimental Steps"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#test-results","text":"According to the test results, the TCP latency will decrease by 40% ~ 60% under different packet sizes, and the throughput will increase by 40% ~ 80% when the packet size is greater than 1024 bytes. Packet Size (byte) eBPF tcp_lat (us) Default tcp_lat (us) eBPF tcp_bw (Mb/s) Default tcp_bw(Mb/s) 1 20.2 44.5 1.36 4.27 4 20.2 48.7 5.48 16.7 16 19.6 41.6 21.7 63.5 64 18.8 41.3 96.8 201 256 19.2 36 395 539 1024 18.3 42.4 1360 846 4096 16.5 62.6 4460 2430 16384 20.2 58.8 9600 6900 In the hardware environment under test, when the packet size is less than 512 bytes, the throughput indicator optimized by eBPF is lower than the throughput under the default configuration. This situation may be related to the TCP aggregation optimization of the network card under the default configuration. If the application scenario is sensitive to small packet throughput, you need to test in the corresponding environment Determine whether to enable eBPF optimization. We will also optimize the throughput of eBPF TCP small packet scenarios in the future.","title":"Test Results"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#references","text":"istio-tcpip-bypass Deep Dive TCP/IP Bypass with eBPF in Service Mesh Tanzu Service Mesh Acceleration using eBPF \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"References"},{"location":"en/advance/cilium-hubble-observe/","text":"Cilium Network Traffic Observation \u00b6 Kube-OVN supports Cilium integration, please refer to Cilium integration for details. Cilium provides rich network traffic observation capabilities, and the flow observability is provided by Hubble. Hubble can observe the traffic across nodes, clusters, and even multi-cluster scenarios. Install Hubble \u00b6 In the default Cilium integration installation, the Hubble related components are not installed, so to support traffic observation, you need to supplement the installation of Hubble on the environment. Execute the following command to install Hubble using helm: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true After installing Hubble, execute cilium status to check the status of the component and confirm that the installation is successful. # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: OK \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ Deployment hubble-relay Desired: 1 , Ready: 1 /1, Available: 1 /1 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment hubble-ui Desired: 1 , Ready: 1 /1, Available: 1 /1 Containers: cilium Running: 2 hubble-ui Running: 1 hubble-relay Running: 1 cilium-operator Running: 2 Cluster Pods: 16 /17 managed by Cilium Image versions hubble-relay quay.io/cilium/hubble-relay:v1.11.6@sha256:fd9034a2d04d5b973f1e8ed44f230ea195b89c37955ff32e34e5aa68f3ed675a: 1 cilium-operator quay.io/cilium/operator-generic:v1.11.6@sha256:9f6063c7bcaede801a39315ec7c166309f6a6783e98665f6693939cf1701bc17: 2 cilium quay.io/cilium/cilium:v1.11.6@sha256:f7f93c26739b6641a3fa3d76b1e1605b15989f25d06625260099e01c8243f54c: 2 hubble-ui quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1 apple@bogon cilium % After installing the Hubble component, you need to install the command line to view the traffic information in the environment. Execute the following command to install Hubble CLI: curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/v0.10.0/hubble-linux-amd64.tar.gz sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin Deploy and test \u00b6 Cilium offers a traffic test deployment solution, you can directly use the official deployment solution to deploy the test. Execute the command cilium connectivity test , Cilium will automatically create the cilium-test namespace, and deploy the test under cilium-test. After the normal deployment, you can view the resource information under the cilium-test namespace, as follows: # kubectl get all -n cilium-test NAME READY STATUS RESTARTS AGE pod/client-7df6cfbf7b-z5t2j 1 /1 Running 0 21s pod/client2-547996d7d8-nvgxg 1 /1 Running 0 21s pod/echo-other-node-d79544ccf-hl4gg 2 /2 Running 0 21s pod/echo-same-node-5d466d5444-ml7tc 2 /2 Running 0 21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE service/echo-other-node NodePort 10 .109.58.126 <none> 8080 :32269/TCP 21s service/echo-same-node NodePort 10 .108.70.32 <none> 8080 :32490/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/client 1 /1 1 1 21s deployment.apps/client2 1 /1 1 1 21s deployment.apps/echo-other-node 1 /1 1 1 21s deployment.apps/echo-same-node 1 /1 1 1 21s NAME DESIRED CURRENT READY AGE replicaset.apps/client-7df6cfbf7b 1 1 1 21s replicaset.apps/client2-547996d7d8 1 1 1 21s replicaset.apps/echo-other-node-d79544ccf 1 1 1 21s replicaset.apps/echo-same-node-5d466d5444 1 1 1 21s Use the command line to observe traffic \u00b6 By default, the network traffic observation only provides the traffic observed by the Cilium agent on each node. Execute the hubble observe command in the Cilium agent pod under the kube-system namespace to view the traffic information on the node. # kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cilium-d6h56 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-c7sb2 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-wj8gt 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> cilium-tq5xb 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> kube-ovn-pinger-7lgk8 1 /1 Running 0 21h 10 .16.0.19 kube-ovn-control-plane <none> <none> kube-ovn-pinger-msvcn 1 /1 Running 0 21h 10 .16.0.18 kube-ovn-worker <none> <none> # kubectl exec -it -n kube-system cilium-d6h56 -- bash root@kube-ovn-worker:/home/cilium# hubble observe --from-namespace kube-system Jul 29 03 :24:25.551: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: SYN ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.651: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.761: kube-system/kube-ovn-pinger-msvcn:52004 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn -> kube-system/kube-ovn-pinger-7lgk8 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn <- kube-system/kube-ovn-pinger-7lgk8 to-endpoint FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:80 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 -> kube-system/kube-ovn-pinger-msvcn to-endpoint FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 <- kube-system/kube-ovn-pinger-msvcn to-stack FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.979: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.3 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:26.037: kube-system/coredns-6d4b75cb6d-lbgjg:36430 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:26.282: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.2 to-stack FORWARDED ( ICMPv4 EchoRequest ) After deploying Hubble Relay, Hubble can provide complete cluster-wide network traffic observation. Configure port forwarding \u00b6 In order to access the Hubble API normally, you need to create a port forwarding to forward the local request to the Hubble Service. You can execute the kubectl port-forward deployment/hubble-relay -n kube-system 4245:4245 command to open the port forwarding in the current terminal. The port forwarding configuration can refer to Port Forwarding . kubectl port-forward is a blocking command, you can open a new terminal to execute the following command to observe the traffic information. After configuring the port forwarding, execute the hubble status command in the terminal. If there is an output similar to the following, the port forwarding configuration is correct, and you can use the command line to observe the traffic. # hubble status Healthcheck ( via localhost:4245 ) : Ok Current/Max Flows: 8 ,190/8,190 ( 100 .00% ) Flows/s: 22 .86 Connected Nodes: 2 /2 Use the command line to observe traffic \u00b6 Execute the hubble observe command in the terminal to view the traffic information of the cluster. The traffic observed by the cilium-test namespace is as follows: Pay attention to the hubble observe command display result, which is the traffic information queried when the current command line is executed. Executing the command line multiple times can view different traffic information. For more detailed observation information, you can execute the hubble help observe command to view the detailed usage of Hubble CLI. Use UI to observe traffic \u00b6 Execute the cilium status command to confirm that the Hubble UI has been successfully installed. In the second step of the Hubble installation, the installation of the UI has been supplemented. Execute the command cilium hubble ui to automatically create port forwarding and map the hubble-ui service to the local port. When the command is executed normally, the local browser will be automatically opened and jump to the Hubble UI interface. If it does not jump automatically, enter http://localhost:12000 in the browser to open the UI observation interface. On the top left of the UI, select the cilium-test namespace to view the test traffic information provided by Cilium. Hubble Traffic Monitoring \u00b6 Hubble component provides monitoring of Pod network behavior in the cluster. In order to support viewing the monitoring data provided by Hubble, you need to enable monitoring statistics. Refer to the following command to supplement the hubble.metrics.enabled configuration item: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \\ --set hubble.metrics.enabled = \"{dns,drop,tcp,flow,icmp,http}\" After the deployment is completed, you can view the monitoring data provided by Hubble through the hubble-metrics service. Execute the following command to view the monitoring data: # curl 172.18.0.2:9091/metrics # HELP hubble_drop_total Number of drops # TYPE hubble_drop_total counter hubble_drop_total { protocol = \"ICMPv6\" ,reason = \"Unsupported L3 protocol\" } 2 # HELP hubble_flows_processed_total Total number of flows processed # TYPE hubble_flows_processed_total counter hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv6\" ,subtype = \"\" ,type = \"Drop\" ,verdict = \"DROPPED\" } 2 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 8282 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 6767 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 # HELP hubble_icmp_total Number of ICMP messages # TYPE hubble_icmp_total counter hubble_icmp_total { family = \"IPv4\" ,type = \"EchoReply\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"EchoRequest\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"RouterSolicitation\" } 2 # HELP hubble_tcp_flags_total TCP flag occurrences # TYPE hubble_tcp_flags_total counter hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"FIN\" } 2043 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"RST\" } 301 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN\" } 1169 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN-ACK\" } 1169 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium Network Traffic Observation"},{"location":"en/advance/cilium-hubble-observe/#cilium-network-traffic-observation","text":"Kube-OVN supports Cilium integration, please refer to Cilium integration for details. Cilium provides rich network traffic observation capabilities, and the flow observability is provided by Hubble. Hubble can observe the traffic across nodes, clusters, and even multi-cluster scenarios.","title":"Cilium Network Traffic Observation"},{"location":"en/advance/cilium-hubble-observe/#install-hubble","text":"In the default Cilium integration installation, the Hubble related components are not installed, so to support traffic observation, you need to supplement the installation of Hubble on the environment. Execute the following command to install Hubble using helm: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true After installing Hubble, execute cilium status to check the status of the component and confirm that the installation is successful. # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: OK \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ Deployment hubble-relay Desired: 1 , Ready: 1 /1, Available: 1 /1 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment hubble-ui Desired: 1 , Ready: 1 /1, Available: 1 /1 Containers: cilium Running: 2 hubble-ui Running: 1 hubble-relay Running: 1 cilium-operator Running: 2 Cluster Pods: 16 /17 managed by Cilium Image versions hubble-relay quay.io/cilium/hubble-relay:v1.11.6@sha256:fd9034a2d04d5b973f1e8ed44f230ea195b89c37955ff32e34e5aa68f3ed675a: 1 cilium-operator quay.io/cilium/operator-generic:v1.11.6@sha256:9f6063c7bcaede801a39315ec7c166309f6a6783e98665f6693939cf1701bc17: 2 cilium quay.io/cilium/cilium:v1.11.6@sha256:f7f93c26739b6641a3fa3d76b1e1605b15989f25d06625260099e01c8243f54c: 2 hubble-ui quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1 apple@bogon cilium % After installing the Hubble component, you need to install the command line to view the traffic information in the environment. Execute the following command to install Hubble CLI: curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/v0.10.0/hubble-linux-amd64.tar.gz sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin","title":"Install Hubble"},{"location":"en/advance/cilium-hubble-observe/#deploy-and-test","text":"Cilium offers a traffic test deployment solution, you can directly use the official deployment solution to deploy the test. Execute the command cilium connectivity test , Cilium will automatically create the cilium-test namespace, and deploy the test under cilium-test. After the normal deployment, you can view the resource information under the cilium-test namespace, as follows: # kubectl get all -n cilium-test NAME READY STATUS RESTARTS AGE pod/client-7df6cfbf7b-z5t2j 1 /1 Running 0 21s pod/client2-547996d7d8-nvgxg 1 /1 Running 0 21s pod/echo-other-node-d79544ccf-hl4gg 2 /2 Running 0 21s pod/echo-same-node-5d466d5444-ml7tc 2 /2 Running 0 21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE service/echo-other-node NodePort 10 .109.58.126 <none> 8080 :32269/TCP 21s service/echo-same-node NodePort 10 .108.70.32 <none> 8080 :32490/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/client 1 /1 1 1 21s deployment.apps/client2 1 /1 1 1 21s deployment.apps/echo-other-node 1 /1 1 1 21s deployment.apps/echo-same-node 1 /1 1 1 21s NAME DESIRED CURRENT READY AGE replicaset.apps/client-7df6cfbf7b 1 1 1 21s replicaset.apps/client2-547996d7d8 1 1 1 21s replicaset.apps/echo-other-node-d79544ccf 1 1 1 21s replicaset.apps/echo-same-node-5d466d5444 1 1 1 21s","title":"Deploy and test"},{"location":"en/advance/cilium-hubble-observe/#use-the-command-line-to-observe-traffic","text":"By default, the network traffic observation only provides the traffic observed by the Cilium agent on each node. Execute the hubble observe command in the Cilium agent pod under the kube-system namespace to view the traffic information on the node. # kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cilium-d6h56 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-c7sb2 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-wj8gt 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> cilium-tq5xb 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> kube-ovn-pinger-7lgk8 1 /1 Running 0 21h 10 .16.0.19 kube-ovn-control-plane <none> <none> kube-ovn-pinger-msvcn 1 /1 Running 0 21h 10 .16.0.18 kube-ovn-worker <none> <none> # kubectl exec -it -n kube-system cilium-d6h56 -- bash root@kube-ovn-worker:/home/cilium# hubble observe --from-namespace kube-system Jul 29 03 :24:25.551: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: SYN ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.651: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.761: kube-system/kube-ovn-pinger-msvcn:52004 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn -> kube-system/kube-ovn-pinger-7lgk8 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn <- kube-system/kube-ovn-pinger-7lgk8 to-endpoint FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:80 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 -> kube-system/kube-ovn-pinger-msvcn to-endpoint FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 <- kube-system/kube-ovn-pinger-msvcn to-stack FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.979: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.3 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:26.037: kube-system/coredns-6d4b75cb6d-lbgjg:36430 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:26.282: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.2 to-stack FORWARDED ( ICMPv4 EchoRequest ) After deploying Hubble Relay, Hubble can provide complete cluster-wide network traffic observation.","title":"Use the command line to observe traffic"},{"location":"en/advance/cilium-hubble-observe/#configure-port-forwarding","text":"In order to access the Hubble API normally, you need to create a port forwarding to forward the local request to the Hubble Service. You can execute the kubectl port-forward deployment/hubble-relay -n kube-system 4245:4245 command to open the port forwarding in the current terminal. The port forwarding configuration can refer to Port Forwarding . kubectl port-forward is a blocking command, you can open a new terminal to execute the following command to observe the traffic information. After configuring the port forwarding, execute the hubble status command in the terminal. If there is an output similar to the following, the port forwarding configuration is correct, and you can use the command line to observe the traffic. # hubble status Healthcheck ( via localhost:4245 ) : Ok Current/Max Flows: 8 ,190/8,190 ( 100 .00% ) Flows/s: 22 .86 Connected Nodes: 2 /2","title":"Configure port forwarding"},{"location":"en/advance/cilium-hubble-observe/#use-the-command-line-to-observe-traffic_1","text":"Execute the hubble observe command in the terminal to view the traffic information of the cluster. The traffic observed by the cilium-test namespace is as follows: Pay attention to the hubble observe command display result, which is the traffic information queried when the current command line is executed. Executing the command line multiple times can view different traffic information. For more detailed observation information, you can execute the hubble help observe command to view the detailed usage of Hubble CLI.","title":"Use the command line to observe traffic"},{"location":"en/advance/cilium-hubble-observe/#use-ui-to-observe-traffic","text":"Execute the cilium status command to confirm that the Hubble UI has been successfully installed. In the second step of the Hubble installation, the installation of the UI has been supplemented. Execute the command cilium hubble ui to automatically create port forwarding and map the hubble-ui service to the local port. When the command is executed normally, the local browser will be automatically opened and jump to the Hubble UI interface. If it does not jump automatically, enter http://localhost:12000 in the browser to open the UI observation interface. On the top left of the UI, select the cilium-test namespace to view the test traffic information provided by Cilium.","title":"Use UI to observe traffic"},{"location":"en/advance/cilium-hubble-observe/#hubble-traffic-monitoring","text":"Hubble component provides monitoring of Pod network behavior in the cluster. In order to support viewing the monitoring data provided by Hubble, you need to enable monitoring statistics. Refer to the following command to supplement the hubble.metrics.enabled configuration item: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \\ --set hubble.metrics.enabled = \"{dns,drop,tcp,flow,icmp,http}\" After the deployment is completed, you can view the monitoring data provided by Hubble through the hubble-metrics service. Execute the following command to view the monitoring data: # curl 172.18.0.2:9091/metrics # HELP hubble_drop_total Number of drops # TYPE hubble_drop_total counter hubble_drop_total { protocol = \"ICMPv6\" ,reason = \"Unsupported L3 protocol\" } 2 # HELP hubble_flows_processed_total Total number of flows processed # TYPE hubble_flows_processed_total counter hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv6\" ,subtype = \"\" ,type = \"Drop\" ,verdict = \"DROPPED\" } 2 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 8282 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 6767 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 # HELP hubble_icmp_total Number of ICMP messages # TYPE hubble_icmp_total counter hubble_icmp_total { family = \"IPv4\" ,type = \"EchoReply\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"EchoRequest\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"RouterSolicitation\" } 2 # HELP hubble_tcp_flags_total TCP flag occurrences # TYPE hubble_tcp_flags_total counter hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"FIN\" } 2043 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"RST\" } 301 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN\" } 1169 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN-ACK\" } 1169 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Hubble Traffic Monitoring"},{"location":"en/advance/cilium-networkpolicy/","text":"Cilium NetworkPolicy Support \u00b6 Kube-OVN currently supports integration with Cilium, and the specific operation can refer to Cilium integration . After integrating Cilium, you can use Cilium's excellent network policy capabilities to control the access of Pods in the cluster.The following documents provide integration verification of Cilium L3 and L4 network policy capabilities. Verification Steps \u00b6 Create test Pod \u00b6 Create namespace test . Refer to the following yaml, create Pod with label app=test in namespace test as the destination Pod for testing access. apiVersion : apps/v1 kind : Deployment metadata : labels : app : test name : test namespace : test spec : replicas : 1 selector : matchLabels : app : test strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : test spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx Similarly, refer to the following yaml, create Pod with label app=dynamic in namespace default as the Pod for testing access. apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx View the test Pod and Label information: # kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-9v5c4 1 /1 Running 0 28h 10 .16.0.35 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-s8z2n 1 /1 Running 0 28h 10 .16.0.36 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 # kubectl get pod -o wide -n test --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-6dsg6 1 /1 Running 0 7h20m 10 .16.0.2 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-tjgtp 1 /1 Running 0 7h46m 10 .16.0.42 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 label-test1-77b6764857-swq4k 1 /1 Running 0 3h43m 10 .16.0.12 kube-ovn-worker <none> <none> app = test1,pod-template-hash = 77b6764857 // As the destination Pod for testing access. test-54c98bc466-mft5s 1 /1 Running 0 8h 10 .16.0.41 kube-ovn-worker <none> <none> app = test,pod-template-hash = 54c98bc466 L3 Network Policy Test \u00b6 Refer to the following yaml, create CiliumNetworkPolicy resource: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l3-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic At this point, the test Pod in the default namespace cannot access the destination Pod, but the test Pod to the destination Pod in the test namespace is accessible. Test results in the default namespace: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss Test results in the test namespace: # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .558 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .223 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .304 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .223/1.028/2.558 ms Look at the Cilium official document explanation, the CiliumNetworkPolicy resource limits the control at the namespace level. For more information, please refer to Cilium Limitations . If there is a network policy rule match, only the Pod in the same namespace can access according to the rule, and the Pod in the other namespace is denied access by default. If you want to implement cross-namespace access, you need to specify the namespace information in the rule. Refer to the document, modify the CiliumNetworkPolicy resource, and add namespace information: ingress : - fromEndpoints : - matchLabels : app : dynamic k8s:io.kubernetes.pod.namespace : default // control the Pod access in other namespace Look at the modified CiliumNetworkPolicy resource information: # kubectl get cnp -n test -o yaml l3-rule apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: l3-rule namespace: test spec: endpointSelector: matchLabels: app: test ingress: - fromEndpoints: - matchLabels: app: dynamic - matchLabels: app: dynamic k8s:io.kubernetes.pod.namespace: default Test the Pod access in the default namespace again, and the destination Pod access is normal: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -n test -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .383 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .115 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .142 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .115/0.880/2.383 ms Using the standard Kubernetes network policy networkpolicy , the test results show that Cilium also restricts access within the same namespace, and cross-namespace access is prohibited. It is different from Kube-OVN implementation. Kube-OVN supports standard k8s network policy, which restricts the destination Pod in a specific namespace, but there is no namespace restriction on the source Pod. Any Pod that meets the restriction rules in any namespace can access the destination Pod. L4 Network Policy Test \u00b6 Refer to the following yaml, create CiliumNetworkPolicy resource: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l4-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic toPorts : - ports : - port : \"80\" protocol : TCP Test the access of the Pod that meets the network policy rules in the same namespace # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl 10 .16.0.41:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on test-54c98bc466-mft5s</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> The Pod that does not meet the network policy rules in the same namespace cannot access # kubectl exec -it -n test label-test1-77b6764857-swq4k -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl -v 10 .16.0.41:80 --connect-timeout 10 * Trying 10 .16.0.41:80... * After 10000ms connect time, move on! * connect to 10 .16.0.41 port 80 failed: Operation timed out * Connection timeout after 10001 ms * Closing connection 0 curl: ( 28 ) Connection timeout after 10001 ms After the network policy takes effect, cross-namespace access is still prohibited, which is consistent with the L3 network policy test results. After the L4 network policy takes effect, ping cannot be used, but TCP access that meets the policy rules can be executed normally. About the restriction of ICMP, please refer to the official description L4 Limitation Description . L7 Network Policy Test \u00b6 chaining mode, L7 network policy currently has problems. In the Cilium official document, there is an explanation for this situation, please refer to Generic Veth Chaining . This problem is tracked using issue 12454 , and it has not been resolved yet. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium NetworkPolicy Support"},{"location":"en/advance/cilium-networkpolicy/#cilium-networkpolicy-support","text":"Kube-OVN currently supports integration with Cilium, and the specific operation can refer to Cilium integration . After integrating Cilium, you can use Cilium's excellent network policy capabilities to control the access of Pods in the cluster.The following documents provide integration verification of Cilium L3 and L4 network policy capabilities.","title":"Cilium NetworkPolicy Support"},{"location":"en/advance/cilium-networkpolicy/#verification-steps","text":"","title":"Verification Steps"},{"location":"en/advance/cilium-networkpolicy/#create-test-pod","text":"Create namespace test . Refer to the following yaml, create Pod with label app=test in namespace test as the destination Pod for testing access. apiVersion : apps/v1 kind : Deployment metadata : labels : app : test name : test namespace : test spec : replicas : 1 selector : matchLabels : app : test strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : test spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx Similarly, refer to the following yaml, create Pod with label app=dynamic in namespace default as the Pod for testing access. apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx View the test Pod and Label information: # kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-9v5c4 1 /1 Running 0 28h 10 .16.0.35 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-s8z2n 1 /1 Running 0 28h 10 .16.0.36 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 # kubectl get pod -o wide -n test --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-6dsg6 1 /1 Running 0 7h20m 10 .16.0.2 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-tjgtp 1 /1 Running 0 7h46m 10 .16.0.42 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 label-test1-77b6764857-swq4k 1 /1 Running 0 3h43m 10 .16.0.12 kube-ovn-worker <none> <none> app = test1,pod-template-hash = 77b6764857 // As the destination Pod for testing access. test-54c98bc466-mft5s 1 /1 Running 0 8h 10 .16.0.41 kube-ovn-worker <none> <none> app = test,pod-template-hash = 54c98bc466","title":"Create test Pod"},{"location":"en/advance/cilium-networkpolicy/#l3-network-policy-test","text":"Refer to the following yaml, create CiliumNetworkPolicy resource: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l3-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic At this point, the test Pod in the default namespace cannot access the destination Pod, but the test Pod to the destination Pod in the test namespace is accessible. Test results in the default namespace: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss Test results in the test namespace: # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .558 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .223 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .304 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .223/1.028/2.558 ms Look at the Cilium official document explanation, the CiliumNetworkPolicy resource limits the control at the namespace level. For more information, please refer to Cilium Limitations . If there is a network policy rule match, only the Pod in the same namespace can access according to the rule, and the Pod in the other namespace is denied access by default. If you want to implement cross-namespace access, you need to specify the namespace information in the rule. Refer to the document, modify the CiliumNetworkPolicy resource, and add namespace information: ingress : - fromEndpoints : - matchLabels : app : dynamic k8s:io.kubernetes.pod.namespace : default // control the Pod access in other namespace Look at the modified CiliumNetworkPolicy resource information: # kubectl get cnp -n test -o yaml l3-rule apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: l3-rule namespace: test spec: endpointSelector: matchLabels: app: test ingress: - fromEndpoints: - matchLabels: app: dynamic - matchLabels: app: dynamic k8s:io.kubernetes.pod.namespace: default Test the Pod access in the default namespace again, and the destination Pod access is normal: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -n test -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .383 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .115 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .142 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .115/0.880/2.383 ms Using the standard Kubernetes network policy networkpolicy , the test results show that Cilium also restricts access within the same namespace, and cross-namespace access is prohibited. It is different from Kube-OVN implementation. Kube-OVN supports standard k8s network policy, which restricts the destination Pod in a specific namespace, but there is no namespace restriction on the source Pod. Any Pod that meets the restriction rules in any namespace can access the destination Pod.","title":"L3 Network Policy Test"},{"location":"en/advance/cilium-networkpolicy/#l4-network-policy-test","text":"Refer to the following yaml, create CiliumNetworkPolicy resource: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l4-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic toPorts : - ports : - port : \"80\" protocol : TCP Test the access of the Pod that meets the network policy rules in the same namespace # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl 10 .16.0.41:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on test-54c98bc466-mft5s</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> The Pod that does not meet the network policy rules in the same namespace cannot access # kubectl exec -it -n test label-test1-77b6764857-swq4k -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl -v 10 .16.0.41:80 --connect-timeout 10 * Trying 10 .16.0.41:80... * After 10000ms connect time, move on! * connect to 10 .16.0.41 port 80 failed: Operation timed out * Connection timeout after 10001 ms * Closing connection 0 curl: ( 28 ) Connection timeout after 10001 ms After the network policy takes effect, cross-namespace access is still prohibited, which is consistent with the L3 network policy test results. After the L4 network policy takes effect, ping cannot be used, but TCP access that meets the policy rules can be executed normally. About the restriction of ICMP, please refer to the official description L4 Limitation Description .","title":"L4 Network Policy Test"},{"location":"en/advance/cilium-networkpolicy/#l7-network-policy-test","text":"chaining mode, L7 network policy currently has problems. In the Cilium official document, there is an explanation for this situation, please refer to Generic Veth Chaining . This problem is tracked using issue 12454 , and it has not been resolved yet. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"L7 Network Policy Test"},{"location":"en/advance/dhcp/","text":"DHCP \u00b6 When using SR-IOV or DPDK type networks, KubeVirt's built-in DHCP does not work in this network mode. Kube-OVN can use the DHCP capabilities of OVN to set DHCP options at the subnet level to help KubeVirt VMs of these network types to properly use DHCP to obtain assigned IP addresses. Kube-OVN supports both DHCPv4 and DHCPv6. The subnet DHCP is configured as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : sn-dual spec : cidrBlock : \"10.0.0.0/24,240e::a00/120\" default : false disableGatewayCheck : true disableInterConnection : false excludeIps : - 10.0.0.1 - 240e::a01 gateway : 10.0.0.1,240e::a01 gatewayNode : '' gatewayType : distributed natOutgoing : false private : false protocol : Dual provider : ovn vpc : vpc-test enableDHCP : true dhcpV4Options : \"lease_time=3600,router=10.0.0.1,server_id=169.254.0.254,server_mac=00:00:00:2E:2F:B8\" dhcpV6Options : \"server_id=00:00:00:2E:2F:C5\" enableIPv6RA : true ipv6RAConfigs : \"address_mode=dhcpv6_stateful,max_interval=30,min_interval=5,send_periodic=true\" enableDHCP : Whether to enable the DHCP function for the subnet. dhcpV4Options , dhcpV6Options : This field directly exposes DHCP-related options within ovn-nb, please reade DHCP Options for more detail. The default value is \"lease_time=3600, router=$ipv4_gateway, server_id=169.254.0.254, server_mac=$random_mac\" and server_id=$random_mac \u3002 enableIPv6RA : Whether to enable the route broadcast function of DHCPv6. ipv6RAConfigs \uff1aThis field directly exposes DHCP-related options within ovn-nb Logical_Router_Port, please read Logical Router Port for more detail. The default value is address_mode=dhcpv6_stateful, max_interval=30, min_interval=5, send_periodic=true \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DHCP"},{"location":"en/advance/dhcp/#dhcp","text":"When using SR-IOV or DPDK type networks, KubeVirt's built-in DHCP does not work in this network mode. Kube-OVN can use the DHCP capabilities of OVN to set DHCP options at the subnet level to help KubeVirt VMs of these network types to properly use DHCP to obtain assigned IP addresses. Kube-OVN supports both DHCPv4 and DHCPv6. The subnet DHCP is configured as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : sn-dual spec : cidrBlock : \"10.0.0.0/24,240e::a00/120\" default : false disableGatewayCheck : true disableInterConnection : false excludeIps : - 10.0.0.1 - 240e::a01 gateway : 10.0.0.1,240e::a01 gatewayNode : '' gatewayType : distributed natOutgoing : false private : false protocol : Dual provider : ovn vpc : vpc-test enableDHCP : true dhcpV4Options : \"lease_time=3600,router=10.0.0.1,server_id=169.254.0.254,server_mac=00:00:00:2E:2F:B8\" dhcpV6Options : \"server_id=00:00:00:2E:2F:C5\" enableIPv6RA : true ipv6RAConfigs : \"address_mode=dhcpv6_stateful,max_interval=30,min_interval=5,send_periodic=true\" enableDHCP : Whether to enable the DHCP function for the subnet. dhcpV4Options , dhcpV6Options : This field directly exposes DHCP-related options within ovn-nb, please reade DHCP Options for more detail. The default value is \"lease_time=3600, router=$ipv4_gateway, server_id=169.254.0.254, server_mac=$random_mac\" and server_id=$random_mac \u3002 enableIPv6RA : Whether to enable the route broadcast function of DHCPv6. ipv6RAConfigs \uff1aThis field directly exposes DHCP-related options within ovn-nb Logical_Router_Port, please read Logical Router Port for more detail. The default value is address_mode=dhcpv6_stateful, max_interval=30, min_interval=5, send_periodic=true \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DHCP"},{"location":"en/advance/dpdk/","text":"DPDK Support \u00b6 This document describes how Kube-OVN combines with OVS-DPDK to provide a DPDK-type network interface to KubeVirt's virtual machines. Upstream KubeVirt does not currently support OVS-DPDK, users need to use the downstream patch Vhostuser implementation to build KubeVirt by themselves or KVM Device Plugin to use OVS-DPDK. Prerequisites \u00b6 The node needs to provide a dedicated NIC for the DPDK driver to run. The node needs to have Hugepages enabled. Set DPDK driver \u00b6 Here we use driverctl for example, please refer to the DPDK documentation for specific parameters and other driver usage: driverctl set-override 0000 :00:0b.0 uio_pci_generic Configure Nodes \u00b6 Labeling OVS-DPDK-enabled nodes for Kube-OVN to recognize: kubectl label nodes <node> ovn.kubernetes.io/ovs_dp_type = \"userspace\" Create the configuration file ovs-dpdk-config in the /opt/ovs-config directory on nodes that support DPDK. ENCAP_IP = 192 .168.122.193/24 DPDK_DEV = 0000 :00:0b.0 ENCAP_IP : The tunnel endpoint address. DPDK_DEV : The PCI ID of the device. Install Kube-OVN \u00b6 Download scripts: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh Enable the DPDK installation option: bash install.sh --with-hybrid-dpdk Usage \u00b6 Here we verify the OVS-DPDK functionality by creating a virtual machine with a vhostuser type NIC. Here we use the KVM Device Plugin to create virtual machines. For more information on how to use it, please refer to [KVM Device Plugin].( https://github.com/kubevirt/kubernetes-device-plugins/blob/master/docs/README.kvm.md ). kubectl apply -f https://raw.githubusercontent.com/kubevirt/kubernetes-device-plugins/master/manifests/kvm-ds.yml Create NetworkAttachmentDefinition: apiVersion : k8s.cni.cncf.io/v1 kind : NetworkAttachmentDefinition metadata : name : ovn-dpdk namespace : default spec : config : >- { \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-dpdk.default.ovn\", \"vhost_user_socket_volume_name\": \"vhostuser-sockets\", \"vhost_user_socket_name\": \"sock\" } Create a VM image using the following Dockerfile: FROM quay.io/kubevirt/virt-launcher:v0.46.1 # wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 COPY CentOS-7-x86_64-GenericCloud.qcow2 /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2 Create a virtual machine: apiVersion : v1 kind : ConfigMap metadata : name : vm-config data : start.sh : | chmod u+w /etc/libvirt/qemu.conf echo \"hugetlbfs_mount = \\\"/dev/hugepages\\\"\" >> /etc/libvirt/qemu.conf virtlogd & libvirtd & mkdir /var/lock sleep 5 virsh define /root/vm/vm.xml virsh start vm tail -f /dev/null vm.xml : | <domain type='kvm'> <name>vm</name> <uuid>4a9b3f53-fa2a-47f3-a757-dd87720d9d1d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <hugepages> <page size='2' unit='M' nodeset='0'/> </hugepages> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <emulatorpin cpuset='1,3'/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='2' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='2097152' unit='KiB' memAccess='shared'/> </numa> </cpu> <on_reboot>restart</on_reboot> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <interface type='vhostuser'> <mac address='00:00:00:0A:30:89'/> <source type='unix' path='/var/run/vm/sock' mode='server'/> <model type='virtio'/> <driver queues='2'> <host mrg_rxbuf='off'/> </driver> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-vm/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> </devices> </domain> --- apiVersion : apps/v1 kind : Deployment metadata : name : vm-deployment labels : app : vm spec : replicas : 1 selector : matchLabels : app : vm template : metadata : labels : app : vm annotations : k8s.v1.cni.cncf.io/networks : default/ovn-dpdk ovn-dpdk.default.ovn.kubernetes.io/ip_address : 10.16.0.96 ovn-dpdk.default.ovn.kubernetes.io/mac_address : 00:00:00:0A:30:89 spec : nodeSelector : ovn.kubernetes.io/ovs_dp_type : userspace securityContext : runAsUser : 0 volumes : - name : vhostuser-sockets emptyDir : {} - name : xml configMap : name : vm-config - name : hugepage emptyDir : medium : HugePages-2Mi - name : libvirt-runtime emptyDir : {} containers : - name : vm image : vm-vhostuser:latest command : [ \"bash\" , \"/root/vm/start.sh\" ] securityContext : capabilities : add : - NET_BIND_SERVICE - SYS_NICE - NET_RAW - NET_ADMIN privileged : false runAsUser : 0 resources : limits : cpu : '2' devices.kubevirt.io/kvm : '1' memory : '8784969729' hugepages-2Mi : 2Gi requests : cpu : 666m devices.kubevirt.io/kvm : '1' ephemeral-storage : 50M memory : '4490002433' volumeMounts : - name : vhostuser-sockets mountPath : /var/run/vm - name : xml mountPath : /root/vm/ - mountPath : /dev/hugepages name : hugepage - name : libvirt-runtime mountPath : /var/run/libvirt Wait for the virtual machine to be created successfully and then go to the Pod to configure the virtual machine: # virsh set-user-password vm root 12345 Password set successfully for root in vm # virsh console vm Connected to domain 'vm' Escape character is ^ ] ( Ctrl + ]) CentOS Linux 7 ( Core ) Kernel 3 .10.0-1127.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Fri Feb 25 09 :52:54 on ttyS0 Next, you can log into the virtual machine for network configuration and test: ip link set eth0 mtu 1400 ip addr add 10 .16.0.96/16 dev eth0 ip ro add default via 10 .16.0.1 ping 114 .114.114.114 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DPDK Support"},{"location":"en/advance/dpdk/#dpdk-support","text":"This document describes how Kube-OVN combines with OVS-DPDK to provide a DPDK-type network interface to KubeVirt's virtual machines. Upstream KubeVirt does not currently support OVS-DPDK, users need to use the downstream patch Vhostuser implementation to build KubeVirt by themselves or KVM Device Plugin to use OVS-DPDK.","title":"DPDK Support"},{"location":"en/advance/dpdk/#prerequisites","text":"The node needs to provide a dedicated NIC for the DPDK driver to run. The node needs to have Hugepages enabled.","title":"Prerequisites"},{"location":"en/advance/dpdk/#set-dpdk-driver","text":"Here we use driverctl for example, please refer to the DPDK documentation for specific parameters and other driver usage: driverctl set-override 0000 :00:0b.0 uio_pci_generic","title":"Set DPDK driver"},{"location":"en/advance/dpdk/#configure-nodes","text":"Labeling OVS-DPDK-enabled nodes for Kube-OVN to recognize: kubectl label nodes <node> ovn.kubernetes.io/ovs_dp_type = \"userspace\" Create the configuration file ovs-dpdk-config in the /opt/ovs-config directory on nodes that support DPDK. ENCAP_IP = 192 .168.122.193/24 DPDK_DEV = 0000 :00:0b.0 ENCAP_IP : The tunnel endpoint address. DPDK_DEV : The PCI ID of the device.","title":"Configure Nodes"},{"location":"en/advance/dpdk/#install-kube-ovn","text":"Download scripts: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh Enable the DPDK installation option: bash install.sh --with-hybrid-dpdk","title":"Install Kube-OVN"},{"location":"en/advance/dpdk/#usage","text":"Here we verify the OVS-DPDK functionality by creating a virtual machine with a vhostuser type NIC. Here we use the KVM Device Plugin to create virtual machines. For more information on how to use it, please refer to [KVM Device Plugin].( https://github.com/kubevirt/kubernetes-device-plugins/blob/master/docs/README.kvm.md ). kubectl apply -f https://raw.githubusercontent.com/kubevirt/kubernetes-device-plugins/master/manifests/kvm-ds.yml Create NetworkAttachmentDefinition: apiVersion : k8s.cni.cncf.io/v1 kind : NetworkAttachmentDefinition metadata : name : ovn-dpdk namespace : default spec : config : >- { \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-dpdk.default.ovn\", \"vhost_user_socket_volume_name\": \"vhostuser-sockets\", \"vhost_user_socket_name\": \"sock\" } Create a VM image using the following Dockerfile: FROM quay.io/kubevirt/virt-launcher:v0.46.1 # wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 COPY CentOS-7-x86_64-GenericCloud.qcow2 /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2 Create a virtual machine: apiVersion : v1 kind : ConfigMap metadata : name : vm-config data : start.sh : | chmod u+w /etc/libvirt/qemu.conf echo \"hugetlbfs_mount = \\\"/dev/hugepages\\\"\" >> /etc/libvirt/qemu.conf virtlogd & libvirtd & mkdir /var/lock sleep 5 virsh define /root/vm/vm.xml virsh start vm tail -f /dev/null vm.xml : | <domain type='kvm'> <name>vm</name> <uuid>4a9b3f53-fa2a-47f3-a757-dd87720d9d1d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <hugepages> <page size='2' unit='M' nodeset='0'/> </hugepages> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <emulatorpin cpuset='1,3'/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='2' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='2097152' unit='KiB' memAccess='shared'/> </numa> </cpu> <on_reboot>restart</on_reboot> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <interface type='vhostuser'> <mac address='00:00:00:0A:30:89'/> <source type='unix' path='/var/run/vm/sock' mode='server'/> <model type='virtio'/> <driver queues='2'> <host mrg_rxbuf='off'/> </driver> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-vm/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> </devices> </domain> --- apiVersion : apps/v1 kind : Deployment metadata : name : vm-deployment labels : app : vm spec : replicas : 1 selector : matchLabels : app : vm template : metadata : labels : app : vm annotations : k8s.v1.cni.cncf.io/networks : default/ovn-dpdk ovn-dpdk.default.ovn.kubernetes.io/ip_address : 10.16.0.96 ovn-dpdk.default.ovn.kubernetes.io/mac_address : 00:00:00:0A:30:89 spec : nodeSelector : ovn.kubernetes.io/ovs_dp_type : userspace securityContext : runAsUser : 0 volumes : - name : vhostuser-sockets emptyDir : {} - name : xml configMap : name : vm-config - name : hugepage emptyDir : medium : HugePages-2Mi - name : libvirt-runtime emptyDir : {} containers : - name : vm image : vm-vhostuser:latest command : [ \"bash\" , \"/root/vm/start.sh\" ] securityContext : capabilities : add : - NET_BIND_SERVICE - SYS_NICE - NET_RAW - NET_ADMIN privileged : false runAsUser : 0 resources : limits : cpu : '2' devices.kubevirt.io/kvm : '1' memory : '8784969729' hugepages-2Mi : 2Gi requests : cpu : 666m devices.kubevirt.io/kvm : '1' ephemeral-storage : 50M memory : '4490002433' volumeMounts : - name : vhostuser-sockets mountPath : /var/run/vm - name : xml mountPath : /root/vm/ - mountPath : /dev/hugepages name : hugepage - name : libvirt-runtime mountPath : /var/run/libvirt Wait for the virtual machine to be created successfully and then go to the Pod to configure the virtual machine: # virsh set-user-password vm root 12345 Password set successfully for root in vm # virsh console vm Connected to domain 'vm' Escape character is ^ ] ( Ctrl + ]) CentOS Linux 7 ( Core ) Kernel 3 .10.0-1127.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Fri Feb 25 09 :52:54 on ttyS0 Next, you can log into the virtual machine for network configuration and test: ip link set eth0 mtu 1400 ip addr add 10 .16.0.96/16 dev eth0 ip ro add default via 10 .16.0.1 ping 114 .114.114.114 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Usage"},{"location":"en/advance/external-gateway/","text":"External Gateway \u00b6 In some scenarios, all container traffic access to the outside needs to be managed and audited through an external gateway. Kube-OVN can forward outbound traffic to the corresponding external gateway by configuring the appropriate routes in the subnet. Usage \u00b6 kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : external spec : cidrBlock : 172.31.0.0/16 gatewayType : centralized natOutgoing : false externalEgressGateway : 192.168.0.1 policyRoutingTableID : 1000 policyRoutingPriority : 1500 natOutgoing : needs to be set to false . externalEgressGateway : Set to the address of the external gateway, which needs to be in the same Layer 2 reachable domain as the gateway node. policyRoutingTableID : The TableID of the local policy routing table used needs to be different for each subnet to avoid conflicts. policyRoutingPriority : Route priority, in order to avoid subsequent user customization of other routing operations conflict, here you can specify the route priority. If no special needs, you can fill in any value. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"External Gateway"},{"location":"en/advance/external-gateway/#external-gateway","text":"In some scenarios, all container traffic access to the outside needs to be managed and audited through an external gateway. Kube-OVN can forward outbound traffic to the corresponding external gateway by configuring the appropriate routes in the subnet.","title":"External Gateway"},{"location":"en/advance/external-gateway/#usage","text":"kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : external spec : cidrBlock : 172.31.0.0/16 gatewayType : centralized natOutgoing : false externalEgressGateway : 192.168.0.1 policyRoutingTableID : 1000 policyRoutingPriority : 1500 natOutgoing : needs to be set to false . externalEgressGateway : Set to the address of the external gateway, which needs to be in the same Layer 2 reachable domain as the gateway node. policyRoutingTableID : The TableID of the local policy routing table used needs to be different for each subnet to avoid conflicts. policyRoutingPriority : Route priority, in order to avoid subsequent user customization of other routing operations conflict, here you can specify the route priority. If no special needs, you can fill in any value. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Usage"},{"location":"en/advance/fastpath/","text":"Compile FastPath Module \u00b6 After a data plane performance profile, netfilter consumes about 20% of CPU resources for related processing within the container and on the host. The FastPath module can bypass netfilter to reduce CPU consumption and latency, and increase throughput. This document will describe how to compile the FastPath module manually. Download Related Code \u00b6 git clone --depth = 1 https://github.com/kubeovn/kube-ovn.git Install Dependencies \u00b6 Here is an example of CentOS dependencies to download: yum install -y kernel-devel- $( uname -r ) gcc elfutils-libelf-devel Compile the Module \u00b6 For the 3.x kernel: cd kube-ovn/fastpath make all For the 4.x kernel: cd kube-ovn/fastpath/4.18 cp ../Makefile . make all Instal the Kernel Module \u00b6 Copy kube_ovn_fastpath.ko to each node that needs performance optimization, and run the following command: insmod kube_ovn_fastpath.ko Use dmesg to confirm successful installation: # dmesg [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in To uninstall a module, use the following command. rmmod kube_ovn_fastpath.ko This module will not be loaded automatically after machine reboot. If you want to load it automatically, please write the corresponding autostart script according to the system configuration. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Compile FastPath Module"},{"location":"en/advance/fastpath/#compile-fastpath-module","text":"After a data plane performance profile, netfilter consumes about 20% of CPU resources for related processing within the container and on the host. The FastPath module can bypass netfilter to reduce CPU consumption and latency, and increase throughput. This document will describe how to compile the FastPath module manually.","title":"Compile FastPath Module"},{"location":"en/advance/fastpath/#download-related-code","text":"git clone --depth = 1 https://github.com/kubeovn/kube-ovn.git","title":"Download Related Code"},{"location":"en/advance/fastpath/#install-dependencies","text":"Here is an example of CentOS dependencies to download: yum install -y kernel-devel- $( uname -r ) gcc elfutils-libelf-devel","title":"Install Dependencies"},{"location":"en/advance/fastpath/#compile-the-module","text":"For the 3.x kernel: cd kube-ovn/fastpath make all For the 4.x kernel: cd kube-ovn/fastpath/4.18 cp ../Makefile . make all","title":"Compile the Module"},{"location":"en/advance/fastpath/#instal-the-kernel-module","text":"Copy kube_ovn_fastpath.ko to each node that needs performance optimization, and run the following command: insmod kube_ovn_fastpath.ko Use dmesg to confirm successful installation: # dmesg [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in To uninstall a module, use the following command. rmmod kube_ovn_fastpath.ko This module will not be loaded automatically after machine reboot. If you want to load it automatically, please write the corresponding autostart script according to the system configuration. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Instal the Kernel Module"},{"location":"en/advance/multi-nic/","text":"Manage Multiple Interface \u00b6 Kube-OVN can provide cluster-level IPAM capabilities for other CNI network plugins such as macvlan, vlan, host-device, etc. Other network plugins can then use the subnet and fixed IP capabilities in Kube-OVN. Kube-OVN also supports address management when multiple NICs are all of Kube-OVN type. Working Principle \u00b6 By using Multus CNI , we can add multiple NICs of different networks to a Pod. However, we still lack the ability to manage the IP addresses of different networks within a cluster. In Kube-OVN, we have been able to perform advanced IP management such as subnet management, IP reservation, random assignment, fixed assignment, etc. through CRD of Subnet and IP. Now Kube-OVN extend the subnet to integrate with other different network plugins, so that other network plugins can also use the IPAM functionality of Kube-OVN. Workflow \u00b6 The above diagram shows how to manage the IP addresses of other network plugins via Kube-OVN. The eth0 NIC of the container is connected to the OVN network and the net1 NIC is connected to other CNI networks. The network definition for the net1 network is taken from the NetworkAttachmentDefinition resource definition in multus-cni. When a Pod is created, kube-ovn-controller will get the Pod add event, find the corresponding Subnet according to the annotation in the Pod, then manage the address from it, and write the address information assigned to the Pod back to the Pod annotation. The CNI on the container machine can configure kube-ovn-cni as the ipam plugin. kube-ovn-cni will read the Pod annotation and return the address information to the corresponding CNI plugin using the standard format of the CNI protocol. Usage \u00b6 Install Kube-OVN and Multus \u00b6 Please refer One-Click Installation and Multus how to use to install Kube-OVN and Multus-CNI. Provide IPAM for other types of CNI \u00b6 Create NetworkAttachmentDefinition \u00b6 Here we use macvlan as the second network of the container network and set its ipam to kube-ovn : apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : macvlan namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"macvlan.default\" } }' spec.config.ipam.type : Need to be set to kube-ovn to call the kube-ovn plugin to get the address information. server_socket : The socket file used for communication to Kube-OVN. The default location is /run/openvswitch/kube-ovn-daemon.sock . provider : The current NetworkAttachmentDefinition's <name>. <namespace> , Kube-OVN will use this information to find the corresponding Subnet resource. The attached NIC is a Kube-OVN type NIC \u00b6 At this point, the multiple NICs are all Kube-OVN type NICs. Create NetworkAttachmentDefinition \u00b6 Set the provider suffix to ovn : apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" }' spec.config.ipam.type : Need to be set to kube-ovn to call the kube-ovn plugin to get the address information. server_socket : The socket file used for communication to Kube-OVN. The default location is /run/openvswitch/kube-ovn-daemon.sock . provider : The current NetworkAttachmentDefinition's <name>. <namespace> , Kube-OVN will use this information to find the corresponding Subnet resource. It should have the suffix ovn here. Create a Kube-OVN Subnet \u00b6 Create a Kube-OVN Subnet, set the corresponding cidrBlock and exclude_ips , the provider should be set to the <name>. <namespace> of corresponding NetworkAttachmentDefinition. For example, to provide additional NICs with macvlan, create a Subnet as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : macvlan spec : protocol : IPv4 provider : macvlan.default cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 gateway , private , nat are only valid for networks with provider type ovn, not for attachment networks. If you are using Kube-OVN as an attached NIC, provider should be set to the <name>. <namespace>.ovn of the corresponding NetworkAttachmentDefinition, and should end with ovn as a suffix. An example of creating a Subnet with an additional NIC provided by Kube-OVN is as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attachnet spec : protocol : IPv4 provider : attachnet.default.ovn cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 Create a Pod with Multiple NIC \u00b6 For Pods with randomly assigned addresses, simply add the following annotation k8s.v1.cni.cncf.io/networks , taking the value <namespace>/<name> of the corresponding NetworkAttachmentDefinition.\uff1a apiVersion : v1 kind : Pod metadata : name : samplepod namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan spec : containers : - name : samplepod command : [ \"/bin/ash\" , \"-c\" , \"trap : TERM INT; sleep infinity & wait\" ] image : docker.io/library/alpine:edge Create Pod with a Fixed IP \u00b6 For Pods with fixed IPs, add <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_address annotation\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 macvlan.default.kubernetes.io/ip_address : 172.17.0.100 macvlan.default.kubernetes.io/mac_address : 00:00:00:53:6B:BB spec : containers : - name : static-ip image : docker.io/library/nginx:alpine Create Workloads with Fixed IPs \u00b6 For workloads that use ippool, add <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_pool annotations: apiVersion : apps/v1 kind : Deployment metadata : namespace : default name : static-workload labels : app : static-workload spec : replicas : 2 selector : matchLabels : app : static-workload template : metadata : labels : app : static-workload annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 macvlan.default.kubernetes.io/ip_pool : 172.17.0.200,172.17.0.201,172.17.0.202 spec : containers : - name : static-workload image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Manage Multiple Interface"},{"location":"en/advance/multi-nic/#manage-multiple-interface","text":"Kube-OVN can provide cluster-level IPAM capabilities for other CNI network plugins such as macvlan, vlan, host-device, etc. Other network plugins can then use the subnet and fixed IP capabilities in Kube-OVN. Kube-OVN also supports address management when multiple NICs are all of Kube-OVN type.","title":"Manage Multiple Interface"},{"location":"en/advance/multi-nic/#working-principle","text":"By using Multus CNI , we can add multiple NICs of different networks to a Pod. However, we still lack the ability to manage the IP addresses of different networks within a cluster. In Kube-OVN, we have been able to perform advanced IP management such as subnet management, IP reservation, random assignment, fixed assignment, etc. through CRD of Subnet and IP. Now Kube-OVN extend the subnet to integrate with other different network plugins, so that other network plugins can also use the IPAM functionality of Kube-OVN.","title":"Working Principle"},{"location":"en/advance/multi-nic/#workflow","text":"The above diagram shows how to manage the IP addresses of other network plugins via Kube-OVN. The eth0 NIC of the container is connected to the OVN network and the net1 NIC is connected to other CNI networks. The network definition for the net1 network is taken from the NetworkAttachmentDefinition resource definition in multus-cni. When a Pod is created, kube-ovn-controller will get the Pod add event, find the corresponding Subnet according to the annotation in the Pod, then manage the address from it, and write the address information assigned to the Pod back to the Pod annotation. The CNI on the container machine can configure kube-ovn-cni as the ipam plugin. kube-ovn-cni will read the Pod annotation and return the address information to the corresponding CNI plugin using the standard format of the CNI protocol.","title":"Workflow"},{"location":"en/advance/multi-nic/#usage","text":"","title":"Usage"},{"location":"en/advance/multi-nic/#install-kube-ovn-and-multus","text":"Please refer One-Click Installation and Multus how to use to install Kube-OVN and Multus-CNI.","title":"Install Kube-OVN and Multus"},{"location":"en/advance/multi-nic/#provide-ipam-for-other-types-of-cni","text":"","title":"Provide IPAM for other types of CNI"},{"location":"en/advance/multi-nic/#create-networkattachmentdefinition","text":"Here we use macvlan as the second network of the container network and set its ipam to kube-ovn : apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : macvlan namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"macvlan.default\" } }' spec.config.ipam.type : Need to be set to kube-ovn to call the kube-ovn plugin to get the address information. server_socket : The socket file used for communication to Kube-OVN. The default location is /run/openvswitch/kube-ovn-daemon.sock . provider : The current NetworkAttachmentDefinition's <name>. <namespace> , Kube-OVN will use this information to find the corresponding Subnet resource.","title":"Create NetworkAttachmentDefinition"},{"location":"en/advance/multi-nic/#the-attached-nic-is-a-kube-ovn-type-nic","text":"At this point, the multiple NICs are all Kube-OVN type NICs.","title":"The attached NIC is a Kube-OVN type NIC"},{"location":"en/advance/multi-nic/#create-networkattachmentdefinition_1","text":"Set the provider suffix to ovn : apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" }' spec.config.ipam.type : Need to be set to kube-ovn to call the kube-ovn plugin to get the address information. server_socket : The socket file used for communication to Kube-OVN. The default location is /run/openvswitch/kube-ovn-daemon.sock . provider : The current NetworkAttachmentDefinition's <name>. <namespace> , Kube-OVN will use this information to find the corresponding Subnet resource. It should have the suffix ovn here.","title":"Create NetworkAttachmentDefinition"},{"location":"en/advance/multi-nic/#create-a-kube-ovn-subnet","text":"Create a Kube-OVN Subnet, set the corresponding cidrBlock and exclude_ips , the provider should be set to the <name>. <namespace> of corresponding NetworkAttachmentDefinition. For example, to provide additional NICs with macvlan, create a Subnet as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : macvlan spec : protocol : IPv4 provider : macvlan.default cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 gateway , private , nat are only valid for networks with provider type ovn, not for attachment networks. If you are using Kube-OVN as an attached NIC, provider should be set to the <name>. <namespace>.ovn of the corresponding NetworkAttachmentDefinition, and should end with ovn as a suffix. An example of creating a Subnet with an additional NIC provided by Kube-OVN is as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attachnet spec : protocol : IPv4 provider : attachnet.default.ovn cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10","title":"Create a Kube-OVN Subnet"},{"location":"en/advance/multi-nic/#create-a-pod-with-multiple-nic","text":"For Pods with randomly assigned addresses, simply add the following annotation k8s.v1.cni.cncf.io/networks , taking the value <namespace>/<name> of the corresponding NetworkAttachmentDefinition.\uff1a apiVersion : v1 kind : Pod metadata : name : samplepod namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan spec : containers : - name : samplepod command : [ \"/bin/ash\" , \"-c\" , \"trap : TERM INT; sleep infinity & wait\" ] image : docker.io/library/alpine:edge","title":"Create a Pod with Multiple NIC"},{"location":"en/advance/multi-nic/#create-pod-with-a-fixed-ip","text":"For Pods with fixed IPs, add <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_address annotation\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 macvlan.default.kubernetes.io/ip_address : 172.17.0.100 macvlan.default.kubernetes.io/mac_address : 00:00:00:53:6B:BB spec : containers : - name : static-ip image : docker.io/library/nginx:alpine","title":"Create Pod with a Fixed IP"},{"location":"en/advance/multi-nic/#create-workloads-with-fixed-ips","text":"For workloads that use ippool, add <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_pool annotations: apiVersion : apps/v1 kind : Deployment metadata : namespace : default name : static-workload labels : app : static-workload spec : replicas : 2 selector : matchLabels : app : static-workload template : metadata : labels : app : static-workload annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 macvlan.default.kubernetes.io/ip_pool : 172.17.0.200,172.17.0.201,172.17.0.202 spec : containers : - name : static-workload image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Create Workloads with Fixed IPs"},{"location":"en/advance/nat-policy-rule/","text":"NAT Policy Rule Function \u00b6 NAT Policy Rule Function Purpose \u00b6 In the Overlay subnet under the default VPC, when the natOutgoing switch is turned on, all Pods in the subnet need to do SNAT to access the external network to the IP of the current node, but in some scenarios we do not want all Pods in the subnet to access the external network by SNAT. So the NAT Policy Rule is to provide an interface for users to decide which CIDRs or IPs in the subnet to access the external network for SNAT. How to use NAT Policy Rules \u00b6 Enable the natOutgoing switch in subnet.Spec , and add the field natOutgoingPolicyRules as follows: spec : natOutgoing : true natOutgoingPolicyRules : - action : forward match : srcIPs : 10.0.11.0/30,10.0.11.254 - action : nat match : srcIPs : 10.0.11.128/26 dstIPs : 114.114.114.114,8.8.8.8 The above case shows that there are two NAT policy rules: Packets with source IP 10.0.11.0/30 or 10.0.11.254 will not perform SNAT when accessing the external network. When a packet with source IP 10.0.11.128/26 and destination IP 114.114.114.114 or 8.8.8.8 accesses the external network, SNAT will be performed. Field description: Action: Indicates the action that will be executed for the message that meets the corresponding conditions of the \"match\". The action is divided into two types: forward and nat . SNAT. When natOutgoingPolicyRules is not configured, packets are still SNAT by default. match: Indicates the matching segment of the message, the matching segment includes srcIPs and dstIPs, here indicates the source IP and destination IP of the message from the subnet to the external network. match.srcIPs and match.dstIPs support multiple cidr and ip, separated by commas. If several matches are repeated but the actions are different, the array position of natOutgoingPolicyRules shall prevail, and the lower the array index, the higher the priority. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NAT Policy Rule Function"},{"location":"en/advance/nat-policy-rule/#nat-policy-rule-function","text":"","title":"NAT Policy Rule Function"},{"location":"en/advance/nat-policy-rule/#nat-policy-rule-function-purpose","text":"In the Overlay subnet under the default VPC, when the natOutgoing switch is turned on, all Pods in the subnet need to do SNAT to access the external network to the IP of the current node, but in some scenarios we do not want all Pods in the subnet to access the external network by SNAT. So the NAT Policy Rule is to provide an interface for users to decide which CIDRs or IPs in the subnet to access the external network for SNAT.","title":"NAT Policy Rule Function Purpose"},{"location":"en/advance/nat-policy-rule/#how-to-use-nat-policy-rules","text":"Enable the natOutgoing switch in subnet.Spec , and add the field natOutgoingPolicyRules as follows: spec : natOutgoing : true natOutgoingPolicyRules : - action : forward match : srcIPs : 10.0.11.0/30,10.0.11.254 - action : nat match : srcIPs : 10.0.11.128/26 dstIPs : 114.114.114.114,8.8.8.8 The above case shows that there are two NAT policy rules: Packets with source IP 10.0.11.0/30 or 10.0.11.254 will not perform SNAT when accessing the external network. When a packet with source IP 10.0.11.128/26 and destination IP 114.114.114.114 or 8.8.8.8 accesses the external network, SNAT will be performed. Field description: Action: Indicates the action that will be executed for the message that meets the corresponding conditions of the \"match\". The action is divided into two types: forward and nat . SNAT. When natOutgoingPolicyRules is not configured, packets are still SNAT by default. match: Indicates the matching segment of the message, the matching segment includes srcIPs and dstIPs, here indicates the source IP and destination IP of the message from the subnet to the external network. match.srcIPs and match.dstIPs support multiple cidr and ip, separated by commas. If several matches are repeated but the actions are different, the array position of natOutgoingPolicyRules shall prevail, and the lower the array index, the higher the priority. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"How to use NAT Policy Rules"},{"location":"en/advance/node-local-dns/","text":"NodeLocal DNSCache and Kube-OVN adaptation \u00b6 NodeLocal DNSCache improves cluster DNS performance by running DNS cache as a DaemonSet on cluster nodes. This function can also be adapted to Kube-OVN. Nodelocal DNSCache deployment \u00b6 Deploy Kubernetes NodeLocal DNScache \u00b6 This step refers to Kubernetes official website configuration nodelocaldnscache . Deploy with the following script: #!bin/bash localdns = 169 .254.20.10 domain = cluster.local kubedns = 10 .96.0.10 wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml sed -i \"s/__PILLAR__LOCAL__DNS__/ $localdns /g; s/__PILLAR__DNS__DOMAIN__/ $domain /g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/ $kubedns /g\" nodelocaldns.yaml kubectl apply -f nodelocaldns.yaml Modify the kubelet configuration file on each node, modify the clusterDNS field in /var/lib/kubelet/config.yaml to the local DNS IP 169.254.20.10, and then restart the kubelet service. Kube-OVN corresponding DNS configuration \u00b6 After deploying the Nodelocal DNScache component of Kubernetes, Kube-OVN needs to make the following modifications: Underlay subnet enable U2O switch \u00b6 If the underlay subnet needs to use the local DNS function, you need to enable the U2O function, that is, configure spec.u2oInterconnection = true in kubectl edit subnet {your subnet} . If it is an overlay subnet, this step is not required. Specify the corresponding local DNS IP for kube-ovn-controller \u00b6 kubectl edit deployment kube-ovn-controller -n kube-system Add field to spec.template.spec.containers.args --node-local-dns-ip=169.254.20.10 Rebuild the created Pods \u00b6 The reason for this step is to let the Pod regenerate /etc/resolv.conf so that the nameserver points to the local DNS IP. If the nameserver of the Pod is not rebuilt, it will still use the DNS ClusterIP of the cluster. At the same time, if the u2o switch is turned on, the Pod needs to be rebuilt to regenerate the Pod gateway. Validator local DNS cache function \u00b6 After the above configuration is completed, you can find the Pod verification as follows. You can see that the Pod's DNS server points to the local 169.254.20.10 and successfully resolves the domain name: # kubectl exec -it pod1 -- nslookup github.com Server: 169 .254.20.10 Address: 169 .254.20.10:53 Name: github.com Address: 20 .205.243.166 You can also capture packets at the node and verify as follows. You can see that the DNS query message reaches the local DNS service through the ovn0 network card, and the DNS response message returns in the same way: # tcpdump -i any port 53 06 :20:00.441889 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441889 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441950 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.441950 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.442203 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442219 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442273 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) 06 :20:00.442278 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Node Local DNS Cache and Kube-OVN Adaptation"},{"location":"en/advance/node-local-dns/#nodelocal-dnscache-and-kube-ovn-adaptation","text":"NodeLocal DNSCache improves cluster DNS performance by running DNS cache as a DaemonSet on cluster nodes. This function can also be adapted to Kube-OVN.","title":"NodeLocal DNSCache and Kube-OVN adaptation"},{"location":"en/advance/node-local-dns/#nodelocal-dnscache-deployment","text":"","title":"Nodelocal DNSCache deployment"},{"location":"en/advance/node-local-dns/#deploy-kubernetes-nodelocal-dnscache","text":"This step refers to Kubernetes official website configuration nodelocaldnscache . Deploy with the following script: #!bin/bash localdns = 169 .254.20.10 domain = cluster.local kubedns = 10 .96.0.10 wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml sed -i \"s/__PILLAR__LOCAL__DNS__/ $localdns /g; s/__PILLAR__DNS__DOMAIN__/ $domain /g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/ $kubedns /g\" nodelocaldns.yaml kubectl apply -f nodelocaldns.yaml Modify the kubelet configuration file on each node, modify the clusterDNS field in /var/lib/kubelet/config.yaml to the local DNS IP 169.254.20.10, and then restart the kubelet service.","title":"Deploy Kubernetes NodeLocal DNScache"},{"location":"en/advance/node-local-dns/#kube-ovn-corresponding-dns-configuration","text":"After deploying the Nodelocal DNScache component of Kubernetes, Kube-OVN needs to make the following modifications:","title":"Kube-OVN corresponding DNS configuration"},{"location":"en/advance/node-local-dns/#underlay-subnet-enable-u2o-switch","text":"If the underlay subnet needs to use the local DNS function, you need to enable the U2O function, that is, configure spec.u2oInterconnection = true in kubectl edit subnet {your subnet} . If it is an overlay subnet, this step is not required.","title":"Underlay subnet enable U2O switch"},{"location":"en/advance/node-local-dns/#specify-the-corresponding-local-dns-ip-for-kube-ovn-controller","text":"kubectl edit deployment kube-ovn-controller -n kube-system Add field to spec.template.spec.containers.args --node-local-dns-ip=169.254.20.10","title":"Specify the corresponding local DNS IP for kube-ovn-controller"},{"location":"en/advance/node-local-dns/#rebuild-the-created-pods","text":"The reason for this step is to let the Pod regenerate /etc/resolv.conf so that the nameserver points to the local DNS IP. If the nameserver of the Pod is not rebuilt, it will still use the DNS ClusterIP of the cluster. At the same time, if the u2o switch is turned on, the Pod needs to be rebuilt to regenerate the Pod gateway.","title":"Rebuild the created Pods"},{"location":"en/advance/node-local-dns/#validator-local-dns-cache-function","text":"After the above configuration is completed, you can find the Pod verification as follows. You can see that the Pod's DNS server points to the local 169.254.20.10 and successfully resolves the domain name: # kubectl exec -it pod1 -- nslookup github.com Server: 169 .254.20.10 Address: 169 .254.20.10:53 Name: github.com Address: 20 .205.243.166 You can also capture packets at the node and verify as follows. You can see that the DNS query message reaches the local DNS service through the ovn0 network card, and the DNS response message returns in the same way: # tcpdump -i any port 53 06 :20:00.441889 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441889 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441950 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.441950 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.442203 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442219 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442273 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) 06 :20:00.442278 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Validator local DNS cache function"},{"location":"en/advance/offload-corigine/","text":"Offload with Corigine \u00b6 Kube-OVN uses OVS for traffic forwarding in the final data plane, and the associated flow table matching, tunnel encapsulation and other functions are CPU-intensive, which consumes a lot of CPU resources and leads to higher latency and lower throughput under heavy traffic. Corigine Agilio CX series SmartNIC can offload OVS-related operations to the hardware. This technology can shorten the data path without modifying the OVS control plane, avoiding the use of host CPU resources, which dramatically reduce latency and significantly increase the throughput. Prerequisites \u00b6 Corigine Agilio CX series SmartNIC. CentOS 8 Stream or Linux 5.7 above. Since the current NIC does not support dp_hash and hash operation offload, OVN LB function should be disabled. Setup SR-IOV \u00b6 Please read Agilio Open vSwitch TC User Guide for the detail usage of this SmartNIC. The following scripts are saved for subsequent execution of firmware-related operations: #!/bin/bash DEVICE = ${ 1 } DEFAULT_ASSY = scan ASSY = ${ 2 :- ${ DEFAULT_ASSY }} APP = ${ 3 :- flower } if [ \"x ${ DEVICE } \" = \"x\" -o ! -e /sys/class/net/ ${ DEVICE } ] ; then echo Syntax: ${ 0 } device [ ASSY ] [ APP ] echo echo This script associates the TC Offload firmware echo with a Netronome SmartNIC. echo echo device: is the network device associated with the SmartNIC echo ASSY: defaults to ${ DEFAULT_ASSY } echo APP: defaults to flower. flower-next is supported if updated echo firmware has been installed. exit 1 fi # It is recommended that the assembly be determined by inspection # The following code determines the value via the debug interface if [ \" ${ ASSY } x\" = \"scanx\" ] ; then ethtool -W ${ DEVICE } 0 DEBUG = $( ethtool -w ${ DEVICE } data /dev/stdout | strings ) SERIAL = $( echo \" ${ DEBUG } \" | grep \"^SN:\" ) ASSY = $( echo ${ SERIAL } | grep -oE AMDA [ 0 -9 ]{ 4 } ) fi PCIADDR = $( basename $( readlink -e /sys/class/net/ ${ DEVICE } /device )) FWDIR = \"/lib/firmware/netronome\" # AMDA0081 and AMDA0097 uses the same firmware if [ \" ${ ASSY } \" = \"AMDA0081\" ] ; then if [ ! -e ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw ] ; then ln -sf nic_AMDA0097.nffw ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw fi fi FW = \" ${ FWDIR } /pci- ${ PCIADDR } .nffw\" ln -sf \" ${ APP } /nic_ ${ ASSY } .nffw\" \" ${ FW } \" # insert distro-specific initramfs section here... Switching firmware options and reloading the driver: ./agilio-tc-fw-select.sh ens47np0 scan rmmod nfp modprobe nfp Check the number of available VFs and create VFs. # cat /sys/class/net/ens3/device/sriov_totalvfs 65 # echo 4 > /sys/class/net/ens47/device/sriov_numvfs Install SR-IOV Device Plugin \u00b6 Since each machine has a limited number of VFs and each Pod that uses acceleration will take up VF resources, we need to use the SR-IOV Device Plugin to manage the corresponding resources so that the scheduler knows how to schedule. Create SR-IOV Configmap: apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"corigine.com\", \"resourceName\": \"agilio_sriov\", \"selectors\": { \"vendors\": [\"19ee\"], \"devices\": [\"6003\"], \"drivers\": [\"nfp_netvf\"] } } ] } Please read the SR-IOV device plugin to deploy: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml Check if SR-IOV resources have been registered to Kubernetes Node: kubectl describe no containerserver | grep corigine corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov 0 0 Install Multus-CNI \u00b6 The device IDs obtained during SR-IOV Device Plugin scheduling need to be passed to Kube-OVN via Multus-CNI, so Multus-CNI needs to be configured to perform the related tasks. Please read Multus-CNI Document to deploy\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml Create NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : corigine.com/agilio_sriov spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : the format should be {name}.{namespace}.ovn of related NetworkAttachmentDefinition . Enable Offload in Kube-OVN \u00b6 Download the scripts: wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh Change the related options\uff0c IFACE should be the physic NIC and has an IP: ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" Install Kube-OVN\uff1a bash install.sh Create Pods with VF NICs \u00b6 Pods that use VF for network offload acceleration can be created using the following yaml: apiVersion : v1 kind : Pod metadata : name : nginx namespace : default annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : corigine.com/agilio_sriov : '1' limits : corigine.com/agilio_sriov : '1' v1.multus-cni.io/default-network : should be the {namespace}/{name} of related NetworkAttachmentDefinition . Running the following command in the ovs-ovn container of the Pod run node to observe if offload success. # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h If there is offloaded:yes, dp:tc content, the offloading is successful. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Offload with Corigine"},{"location":"en/advance/offload-corigine/#offload-with-corigine","text":"Kube-OVN uses OVS for traffic forwarding in the final data plane, and the associated flow table matching, tunnel encapsulation and other functions are CPU-intensive, which consumes a lot of CPU resources and leads to higher latency and lower throughput under heavy traffic. Corigine Agilio CX series SmartNIC can offload OVS-related operations to the hardware. This technology can shorten the data path without modifying the OVS control plane, avoiding the use of host CPU resources, which dramatically reduce latency and significantly increase the throughput.","title":"Offload with Corigine"},{"location":"en/advance/offload-corigine/#prerequisites","text":"Corigine Agilio CX series SmartNIC. CentOS 8 Stream or Linux 5.7 above. Since the current NIC does not support dp_hash and hash operation offload, OVN LB function should be disabled.","title":"Prerequisites"},{"location":"en/advance/offload-corigine/#setup-sr-iov","text":"Please read Agilio Open vSwitch TC User Guide for the detail usage of this SmartNIC. The following scripts are saved for subsequent execution of firmware-related operations: #!/bin/bash DEVICE = ${ 1 } DEFAULT_ASSY = scan ASSY = ${ 2 :- ${ DEFAULT_ASSY }} APP = ${ 3 :- flower } if [ \"x ${ DEVICE } \" = \"x\" -o ! -e /sys/class/net/ ${ DEVICE } ] ; then echo Syntax: ${ 0 } device [ ASSY ] [ APP ] echo echo This script associates the TC Offload firmware echo with a Netronome SmartNIC. echo echo device: is the network device associated with the SmartNIC echo ASSY: defaults to ${ DEFAULT_ASSY } echo APP: defaults to flower. flower-next is supported if updated echo firmware has been installed. exit 1 fi # It is recommended that the assembly be determined by inspection # The following code determines the value via the debug interface if [ \" ${ ASSY } x\" = \"scanx\" ] ; then ethtool -W ${ DEVICE } 0 DEBUG = $( ethtool -w ${ DEVICE } data /dev/stdout | strings ) SERIAL = $( echo \" ${ DEBUG } \" | grep \"^SN:\" ) ASSY = $( echo ${ SERIAL } | grep -oE AMDA [ 0 -9 ]{ 4 } ) fi PCIADDR = $( basename $( readlink -e /sys/class/net/ ${ DEVICE } /device )) FWDIR = \"/lib/firmware/netronome\" # AMDA0081 and AMDA0097 uses the same firmware if [ \" ${ ASSY } \" = \"AMDA0081\" ] ; then if [ ! -e ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw ] ; then ln -sf nic_AMDA0097.nffw ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw fi fi FW = \" ${ FWDIR } /pci- ${ PCIADDR } .nffw\" ln -sf \" ${ APP } /nic_ ${ ASSY } .nffw\" \" ${ FW } \" # insert distro-specific initramfs section here... Switching firmware options and reloading the driver: ./agilio-tc-fw-select.sh ens47np0 scan rmmod nfp modprobe nfp Check the number of available VFs and create VFs. # cat /sys/class/net/ens3/device/sriov_totalvfs 65 # echo 4 > /sys/class/net/ens47/device/sriov_numvfs","title":"Setup SR-IOV"},{"location":"en/advance/offload-corigine/#install-sr-iov-device-plugin","text":"Since each machine has a limited number of VFs and each Pod that uses acceleration will take up VF resources, we need to use the SR-IOV Device Plugin to manage the corresponding resources so that the scheduler knows how to schedule. Create SR-IOV Configmap: apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"corigine.com\", \"resourceName\": \"agilio_sriov\", \"selectors\": { \"vendors\": [\"19ee\"], \"devices\": [\"6003\"], \"drivers\": [\"nfp_netvf\"] } } ] } Please read the SR-IOV device plugin to deploy: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml Check if SR-IOV resources have been registered to Kubernetes Node: kubectl describe no containerserver | grep corigine corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov 0 0","title":"Install SR-IOV Device Plugin"},{"location":"en/advance/offload-corigine/#install-multus-cni","text":"The device IDs obtained during SR-IOV Device Plugin scheduling need to be passed to Kube-OVN via Multus-CNI, so Multus-CNI needs to be configured to perform the related tasks. Please read Multus-CNI Document to deploy\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml Create NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : corigine.com/agilio_sriov spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : the format should be {name}.{namespace}.ovn of related NetworkAttachmentDefinition .","title":"Install Multus-CNI"},{"location":"en/advance/offload-corigine/#enable-offload-in-kube-ovn","text":"Download the scripts: wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh Change the related options\uff0c IFACE should be the physic NIC and has an IP: ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" Install Kube-OVN\uff1a bash install.sh","title":"Enable Offload in Kube-OVN"},{"location":"en/advance/offload-corigine/#create-pods-with-vf-nics","text":"Pods that use VF for network offload acceleration can be created using the following yaml: apiVersion : v1 kind : Pod metadata : name : nginx namespace : default annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : corigine.com/agilio_sriov : '1' limits : corigine.com/agilio_sriov : '1' v1.multus-cni.io/default-network : should be the {namespace}/{name} of related NetworkAttachmentDefinition . Running the following command in the ovs-ovn container of the Pod run node to observe if offload success. # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h If there is offloaded:yes, dp:tc content, the offloading is successful. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Create Pods with VF NICs"},{"location":"en/advance/offload-mellanox/","text":"Offload with Mellanox \u00b6 Kube-OVN uses OVS for traffic forwarding in the final data plane, and the associated flow table matching, tunnel encapsulation and other functions are CPU-intensive, which consumes a lot of CPU resources and leads to higher latency and lower throughput under heavy traffic. Mellanox Accelerated Switching And Packet Processing (ASAP\u00b2) technology offloads OVS-related operations to an eSwitch within the eSwitch in the hardware. This technology can shorten the data path without modifying the OVS control plane, avoiding the use of host CPU resources, which dramatically reduce latency and significantly increase the throughput. Prerequisites \u00b6 Mellanox CX5/CX6/BlueField that support ASAP\u00b2. CentOS 8 Stream or Linux 5.7 above. Since the current NIC does not support dp_hash and hash operation offload, OVN LB function should be disabled. In order to support offload mode, the NIC cannot do bond. Setup SR-IOV \u00b6 Check the device ID of the NIC, in the following example it is 42:00.0 : # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] Find the corresponding NIC by its device ID: # ls -l /sys/class/net/ | grep 42:00.0 lrwxrwxrwx. 1 root root 0 Jul 22 23 :16 p4p1 -> ../../devices/pci0000:40/0000:40:02.0/0000:42:00.0/net/p4p1 Check the number of available VFs: # cat /sys/class/net/p4p1/device/sriov_totalvfs 8 Create VFs and do not exceeding the number found above: # echo '4' > /sys/class/net/p4p1/device/sriov_numvfs # ip link show p4p1 10 : p4p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:59:9f:c1:ec:12 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 1 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 2 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 3 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off # ip link set p4p1 up Find the device IDs corresponding to the above VFs: # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] Unbound the VFs from the driver: echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind Enable eSwitch mode and set up hardware offload: devlink dev eswitch set pci/0000:42:00.0 mode switchdev ethtool -K enp66s0f0 hw-tc-offload on Rebind the driver and complete the VF setup: echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/bind Some behaviors of NetworkManager may cause driver exceptions, if offloading problems occur we recommended to close NetworkManager and try again. systemctl stop NetworkManager systemctl disable NetworkManager Install SR-IOV Device Plugin \u00b6 Since each machine has a limited number of VFs and each Pod that uses acceleration will take up VF resources, we need to use the SR-IOV Device Plugin to manage the corresponding resources so that the scheduler knows how to schedule. Create SR-IOV Configmap: apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"mellanox.com\", \"resourceName\": \"cx5_sriov_switchdev\", \"selectors\": { \"vendors\": [\"15b3\"], \"devices\": [\"1018\"], \"drivers\": [\"mlx5_core\"] } } ] } Please read the SR-IOV device plugin to deploy: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml Check if SR-IOV resources have been registered to Kubernetes Node: kubectl describe node kube-ovn-01 | grep mellanox mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev 0 0 Install Multus-CNI \u00b6 The device IDs obtained during SR-IOV Device Plugin scheduling need to be passed to Kube-OVN via Multus-CNI, so Multus-CNI needs to be configured to perform the related tasks. Please read Multus-CNI Document to deploy\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml Create NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : mellanox.com/cx5_sriov_switchdev spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : the format should be {name}.{namespace}.ovn of related NetworkAttachmentDefinition . Enable Offload in Kube-OVN \u00b6 Download the scripts: wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh Change the related options\uff0c IFACE should be the physic NIC and has an IP: ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" Install Kube-OVN\uff1a bash install.sh Create Pods with VF NICs \u00b6 Pods that use VF for network offload acceleration can be created using the following yaml: apiVersion : v1 kind : Pod metadata : name : nginx annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : mellanox.com/cx5_sriov_switchdev : '1' limits : mellanox.com/cx5_sriov_switchdev : '1' v1.multus-cni.io/default-network : should be the {namespace}/{name} of related NetworkAttachmentDefinition . Running the following command in the ovs-ovn container of the Pod run node to observe if offload success. # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h If there is offloaded:yes, dp:tc content, the offloading is successful. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Offload with Mellanox"},{"location":"en/advance/offload-mellanox/#offload-with-mellanox","text":"Kube-OVN uses OVS for traffic forwarding in the final data plane, and the associated flow table matching, tunnel encapsulation and other functions are CPU-intensive, which consumes a lot of CPU resources and leads to higher latency and lower throughput under heavy traffic. Mellanox Accelerated Switching And Packet Processing (ASAP\u00b2) technology offloads OVS-related operations to an eSwitch within the eSwitch in the hardware. This technology can shorten the data path without modifying the OVS control plane, avoiding the use of host CPU resources, which dramatically reduce latency and significantly increase the throughput.","title":"Offload with Mellanox"},{"location":"en/advance/offload-mellanox/#prerequisites","text":"Mellanox CX5/CX6/BlueField that support ASAP\u00b2. CentOS 8 Stream or Linux 5.7 above. Since the current NIC does not support dp_hash and hash operation offload, OVN LB function should be disabled. In order to support offload mode, the NIC cannot do bond.","title":"Prerequisites"},{"location":"en/advance/offload-mellanox/#setup-sr-iov","text":"Check the device ID of the NIC, in the following example it is 42:00.0 : # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] Find the corresponding NIC by its device ID: # ls -l /sys/class/net/ | grep 42:00.0 lrwxrwxrwx. 1 root root 0 Jul 22 23 :16 p4p1 -> ../../devices/pci0000:40/0000:40:02.0/0000:42:00.0/net/p4p1 Check the number of available VFs: # cat /sys/class/net/p4p1/device/sriov_totalvfs 8 Create VFs and do not exceeding the number found above: # echo '4' > /sys/class/net/p4p1/device/sriov_numvfs # ip link show p4p1 10 : p4p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:59:9f:c1:ec:12 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 1 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 2 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 3 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off # ip link set p4p1 up Find the device IDs corresponding to the above VFs: # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] Unbound the VFs from the driver: echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind Enable eSwitch mode and set up hardware offload: devlink dev eswitch set pci/0000:42:00.0 mode switchdev ethtool -K enp66s0f0 hw-tc-offload on Rebind the driver and complete the VF setup: echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/bind Some behaviors of NetworkManager may cause driver exceptions, if offloading problems occur we recommended to close NetworkManager and try again. systemctl stop NetworkManager systemctl disable NetworkManager","title":"Setup SR-IOV"},{"location":"en/advance/offload-mellanox/#install-sr-iov-device-plugin","text":"Since each machine has a limited number of VFs and each Pod that uses acceleration will take up VF resources, we need to use the SR-IOV Device Plugin to manage the corresponding resources so that the scheduler knows how to schedule. Create SR-IOV Configmap: apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"mellanox.com\", \"resourceName\": \"cx5_sriov_switchdev\", \"selectors\": { \"vendors\": [\"15b3\"], \"devices\": [\"1018\"], \"drivers\": [\"mlx5_core\"] } } ] } Please read the SR-IOV device plugin to deploy: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml Check if SR-IOV resources have been registered to Kubernetes Node: kubectl describe node kube-ovn-01 | grep mellanox mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev 0 0","title":"Install SR-IOV Device Plugin"},{"location":"en/advance/offload-mellanox/#install-multus-cni","text":"The device IDs obtained during SR-IOV Device Plugin scheduling need to be passed to Kube-OVN via Multus-CNI, so Multus-CNI needs to be configured to perform the related tasks. Please read Multus-CNI Document to deploy\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml Create NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : mellanox.com/cx5_sriov_switchdev spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : the format should be {name}.{namespace}.ovn of related NetworkAttachmentDefinition .","title":"Install Multus-CNI"},{"location":"en/advance/offload-mellanox/#enable-offload-in-kube-ovn","text":"Download the scripts: wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh Change the related options\uff0c IFACE should be the physic NIC and has an IP: ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" Install Kube-OVN\uff1a bash install.sh","title":"Enable Offload in Kube-OVN"},{"location":"en/advance/offload-mellanox/#create-pods-with-vf-nics","text":"Pods that use VF for network offload acceleration can be created using the following yaml: apiVersion : v1 kind : Pod metadata : name : nginx annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : mellanox.com/cx5_sriov_switchdev : '1' limits : mellanox.com/cx5_sriov_switchdev : '1' v1.multus-cni.io/default-network : should be the {namespace}/{name} of related NetworkAttachmentDefinition . Running the following command in the ovs-ovn container of the Pod run node to observe if offload success. # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h If there is offloaded:yes, dp:tc content, the offloading is successful. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Create Pods with VF NICs"},{"location":"en/advance/overlay-with-route/","text":"Interconnection with Routes in Overlay Mode \u00b6 In some scenarios, the network environment does not support Underlay mode, but still need Pods and external devices directly access through IP, then you can use the routing method to connect the container network and the external. Only Overlay Subnets in default VPC support this method. In this case, the Pod IP goes directly to the underlying network, which needs to disable IP checks for source and destination addresses. Prerequisites \u00b6 In this mode, the host needs to open the ip_forward . Check if there is a Drop rule in the forward chain in the host iptables that should be modified for container-related traffic. Due to the possibility of asymmetric routing, the host needs to allow packets with a ct status of INVALID . Steps \u00b6 For subnets that require direct external routing, you need to set natOutgoing of the subnet to false to turn off nat mapping and make the Pod IP directly accessible to the external network. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : false At this point, the Pod's packets can reach the peer node via the host route, but the peer node does not yet know where the return packets should be sent to and needs to add a return route. If the peer host and the container host are on the same Layer 2 network, we can add a static route directly to the peer host to point the next hop of the container network to any machine in the Kubernetes cluster. ip route add 10 .166.0.0/16 via 192 .168.2.10 dev eth0 10.166.0.0/16 is the container subnet CIDR, and 192.168.2.10 is one node in the Kubernetes cluster. If the peer host and the container host are not in the same layer 2 network, you need to configure the corresponding rules on the router. In some virtualized environments, the virtual network identifies asymmetric traffic as illegal traffic and drops it. In this case, you need to adjust the gatewayType of the Subnet to centralized and set the next hop to the IP of the gatewayNode node during route setup. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1\" natOutgoing : false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Interconnection with Routes in Overlay Mode"},{"location":"en/advance/overlay-with-route/#interconnection-with-routes-in-overlay-mode","text":"In some scenarios, the network environment does not support Underlay mode, but still need Pods and external devices directly access through IP, then you can use the routing method to connect the container network and the external. Only Overlay Subnets in default VPC support this method. In this case, the Pod IP goes directly to the underlying network, which needs to disable IP checks for source and destination addresses.","title":"Interconnection with Routes in Overlay Mode"},{"location":"en/advance/overlay-with-route/#prerequisites","text":"In this mode, the host needs to open the ip_forward . Check if there is a Drop rule in the forward chain in the host iptables that should be modified for container-related traffic. Due to the possibility of asymmetric routing, the host needs to allow packets with a ct status of INVALID .","title":"Prerequisites"},{"location":"en/advance/overlay-with-route/#steps","text":"For subnets that require direct external routing, you need to set natOutgoing of the subnet to false to turn off nat mapping and make the Pod IP directly accessible to the external network. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : false At this point, the Pod's packets can reach the peer node via the host route, but the peer node does not yet know where the return packets should be sent to and needs to add a return route. If the peer host and the container host are on the same Layer 2 network, we can add a static route directly to the peer host to point the next hop of the container network to any machine in the Kubernetes cluster. ip route add 10 .166.0.0/16 via 192 .168.2.10 dev eth0 10.166.0.0/16 is the container subnet CIDR, and 192.168.2.10 is one node in the Kubernetes cluster. If the peer host and the container host are not in the same layer 2 network, you need to configure the corresponding rules on the router. In some virtualized environments, the virtual network identifies asymmetric traffic as illegal traffic and drops it. In this case, you need to adjust the gatewayType of the Subnet to centralized and set the next hop to the IP of the gatewayNode node during route setup. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1\" natOutgoing : false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Steps"},{"location":"en/advance/ovn-eip-fip-snat/","text":"Support OVN EIP,FIP and SNAT \u00b6 graph LR pod-->vpc1-subnet-->vpc1-->snat-->lrp-->external-subnet-->gw-node-external-nic The pod access the public network based on the snat graph LR pod-->vpc1-subnet-->vpc1-->fip-->lrp-->external-subnet-->local-node-external-nic The pod access the public network based on the fip The CRD supported by this function is basically the same as the iptable nat gw public network solution. ovn eip: occupies a public ip address and is allocated from the underlay provider network vlan subnet ovn fip: one-to-one dnat snat, which provides direct public network access for ip addresses and vip in a vpc ovn snat: a subnet cidr or a single vpc ip or vip can access public networks based on snat ovn dnat: based router lb, which enables direct access to a group of endpoints in a vpc based on a public endpoint 1. Deployment \u00b6 Currently allows all vpcs to share the same provider vlan subnet resources, similar to neutron ovn mode. Compatible with previous scenarios default VPC EIP/SNAT . During the deployment phase, you may need to specify a default public network logical switch based on actual conditions. If no vlan is in use (vlan 0 is used), the following startup parameters do not need to be configured. # When deploying you need to refer to the above scenario and specify the following parameters as needed according to the actual situation # 1. kube-ovn-controller Startup parameters to be configured\uff1a - --external-gateway-vlanid = 204 - --external-gateway-switch = external204 # 2. kube-ovn-cni Startup parameters to be configured: - --external-gateway-switch = external204 # The above configuration is consistent with the following public network configuration vlan id and resource name, # currently only support to specify one underlay public network as the default external public network. The design and use of this configuration item takes into account the following factors\uff1a Based on this configuration item can be docked to the provider network, vlan, subnet resources. Based on this configuration item, the default vpc enable_eip_snat function can be docked to the existing vlan, subnet resources, while supporting the ipam If only the default vpc's enable_eip_snat mode is used with the old pod annotaion based eip fip snat, then the following configuration is not required. Based on this configuration you can not use the default vpc enable_eip_snat process, only by corresponding to vlan, subnet process, can be compatible with only custom vpc use eip snat usage scenarios. The neutron ovn mode also has a certain static file configuration designation that is, for now, generally consistent. 1.1 Create the underlay public network \u00b6 # provider-network\uff0c vlan\uff0c subnet # cat 01-provider-network.yaml apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: external204 spec: defaultInterface: vlan # cat 02-vlan.yaml apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan204 spec: id: 204 provider: external204 # cat 03-vlan-subnet.yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: external204 spec: protocol: IPv4 cidrBlock: 10 .5.204.0/24 gateway: 10 .5.204.254 vlan: vlan204 excludeIps: - 10 .5.204.1..10.5.204.100 1.2 Default vpc enable eip_snat \u00b6 # Enable the default vpc and the above underlay public provider subnet interconnection cat 00 -centralized-external-gw-no-ip.yaml apiVersion: v1 kind: ConfigMap metadata: name: ovn-external-gw-config namespace: kube-system data: enable-external-gw: \"true\" external-gw-nodes: \"pc-node-1,pc-node-2,pc-node-3\" type: \"centralized\" external-gw-nic: \"vlan\" external-gw-addr: \"10.5.204.254/24\" This feature currently supports the ability to create lrp type ovn eip resources without specifying the lrp ip and mac, which is already supported for automatic acquisition. If specified, it is equivalent to specifying the ip to create an ovn-eip of type lrp. Of course, you can also manually create the lrp type ovn eip in advance. 1.3 Custom vpc enable eip snat fip function \u00b6 # cat 00-ns.yml apiVersion: v1 kind: Namespace metadata: name: vpc1 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true # vpc enableExternal will automatically create an lrp association to the public network specified above # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 After the above template is applied, you should see the following resources exist # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # The route currently supports automatic maintenance 2. ovn-eip \u00b6 This function is designed and used in the same way as iptables-eip, ovn-eip currently has three types nat: indicates ovn dnat, fip, and snat. These nat types are recorded in status lrp: indicates the resource used to connect a vpc to the public network node-ext-gw: applies to ovn BFD-based ecmp static route scenario --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: fip # Dynamically allocate an eip resource that is reserved for fip scenarios 2.1 Create an fip for pod \u00b6 # k get po -o wide -n vpc1 vpc-1-busybox01 NAME READY STATUS RESTARTS AGE IP NODE vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 # k get ip vpc-1-busybox01.vpc1 NAME V4IP V6IP MAC NODE SUBNET vpc-1-busybox01.vpc1 192 .168.0.2 00 :00:00:0A:DD:27 pc-node-2 vpc1-subnet1 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: fip --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: ovnEip: eip-static ipName: vpc-1-busybox01.vpc1 # the name of the ip crd, which is unique # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 # k get ofip eip-static NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 [ root@pc-node-1 03 -cust-vpc ] # ping 10.5.204.101 PING 10 .5.204.101 ( 10 .5.204.101 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.101: icmp_seq = 2 ttl = 62 time = 1 .21 ms 64 bytes from 10 .5.204.101: icmp_seq = 3 ttl = 62 time = 0 .624 ms 64 bytes from 10 .5.204.101: icmp_seq = 4 ttl = 62 time = 0 .368 ms ^C --- 10 .5.204.101 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3049ms rtt min/avg/max/mdev = 0 .368/0.734/1.210/0.352 ms [ root@pc-node-1 03 -cust-vpc ] # # pod <--> node ping is working # The key resources that this public ip can pass include the following ovn nb resources # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 813523e7-c68c-408f-bd8c-cba30cb2e4f4 external ip: \"10.5.204.101\" logical ip: \"192.168.0.2\" type: \"dnat_and_snat\" 2.2 Create an fip for vip \u00b6 In order to facilitate the use of some vip scenarios, such as inside kubevirt VM, keepalived use vip, kube-vip use vip, etc. the vip need public network access. # First create vip, eip, then bind eip to vip # cat vip.yaml apiVersion: kubeovn.io/v1 kind: Vip metadata: name: test-fip-vip spec: subnet: vpc1-subnet1 # cat 04-fip.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: externalSubnet: external204 type: fip --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: ovnEip: eip-for-vip ipType: vip # By default fip is for pod ip, here you need to specify the docking to vip resources ipName: test-fip-vip # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip [ root@pc-node-1 fip-vip ] # ping 10.5.204.106 PING 10 .5.204.106 ( 10 .5.204.106 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.106: icmp_seq = 1 ttl = 62 time = 0 .694 ms 64 bytes from 10 .5.204.106: icmp_seq = 2 ttl = 62 time = 0 .436 ms # node <--> pod fip is working # The way ip is used inside the pod is roughly as follows [ root@pc-node-1 fip-vip ] # k -n vpc1 exec -it vpc-1-busybox03 -- bash [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1568 : eth0@if1569: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:56:40:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.5/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 192 .168.0.3/24 scope global secondary eth0 # vip here valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe56:40e5/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox03 / ] # tcpdump -i eth0 host 192.168.0.3 -netvv tcpdump: listening on eth0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:ed:8e:c7 > 00 :00:00:56:40:e5, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 44830 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.32.51 > 192 .168.0.3: ICMP echo request, id 177 , seq 1 , length 64 00 :00:00:56:40:e5 > 00 :00:00:ed:8e:c7, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 64 , id 43962 , offset 0 , flags [ none ] , proto ICMP ( 1 ) , length 84 ) 192 .168.0.3 > 10 .5.32.51: ICMP echo reply, id 177 , seq 1 , length 64 # pod internal can catch fip related icmp packets 3. ovn-snat \u00b6 3.1 ovn-snat corresponds to the CIDR of a subnet \u00b6 This feature is designed and used in much the same way as iptables-snat # cat 03-subnet-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: externalSubnet: external204 type: snat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: ovnEip: snat-for-subnet-in-vpc vpcSubnet: vpc1-subnet1 # eip corresponds to the entire network segment 3.2 ovn-fip corresponds to a pod IP \u00b6 This feature is designed and used in much the same way as iptables-fip # cat 03-pod-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-pod-vpc-ip spec: externalSubnet: external204 type: snat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat01 spec: ovnEip: snat-for-pod-vpc-ip ipName: vpc-1-busybox02.vpc1 # eip corresponds to a single pod ip After the above resources are created, you can see the following resources that the snat public network feature depends on. # kubectl ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" nat da77a11f-c523-439c-b1d1-72c664196a0f external ip: \"10.5.204.116\" logical ip: \"192.168.0.4\" type: \"snat\" [ root@pc-node-1 03 -cust-vpc ] # k get po -A -o wide | grep busy vpc1 vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 <none> <none> vpc1 vpc-1-busybox02 1 /1 Running 0 17h 192 .168.0.4 pc-node-1 <none> <none> vpc1 vpc-1-busybox03 1 /1 Running 0 17h 192 .168.0.5 pc-node-1 <none> <none> vpc1 vpc-1-busybox04 1 /1 Running 0 17h 192 .168.0.6 pc-node-3 <none> <none> vpc1 vpc-1-busybox05 1 /1 Running 0 17h 192 .168.0.7 pc-node-1 <none> <none> # k exec -it -n vpc1 vpc-1-busybox04 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17095 : eth0@if17096: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:76:94:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.6/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe76:9455/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox04 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 114 time = 22 .2 ms 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 21 .8 ms [ root@pc-node-1 03 -cust-vpc ] # k exec -it -n vpc1 vpc-1-busybox02 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1566 : eth0@if1567: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:0b:e9:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.4/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe0b:e9d0/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox02 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 22 .7 ms 64 bytes from 223 .5.5.5: icmp_seq = 3 ttl = 114 time = 22 .6 ms 64 bytes from 223 .5.5.5: icmp_seq = 4 ttl = 114 time = 22 .1 ms ^C --- 223 .5.5.5 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3064ms rtt min/avg/max/mdev = 22 .126/22.518/22.741/0.278 ms # the two pods can access the external network based on these two type snat resources respectively 4. ovn-dnat \u00b6 4.1 ovn-dnat binds a DNAT to a pod \u00b6 kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : externalSubnet : underlay --- kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ovnEip : eip-dnat ipName : vpc-1-busybox01.vpc1 # Note that this is the name of the pod IP CRD and it is unique protocol : tcp internalPort : \"22\" externalPort : \"22\" The configuration of OvnDnatRule is similar to that of IptablesDnatRule. # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.3 22 22 vpc-1-busybox01.vpc1 true 4.2 ovn-dnat binds a DNAT to a VIP \u00b6 kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ipType : vip # By default, Dnat is oriented towards pod IPs. Here, it is necessary to specify that it is connected to VIP resources ovnEip : eip-dnat ipName : test-dnat-vip protocol : tcp internalPort : \"22\" externalPort : \"22\" The configuration of OvnDnatRule is similar to that of IptablesDnatRule. # kubectl get vip test-dnat-vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY test-dnat-vip 192 .168.0.4 00 :00:00:D0:C0:B5 vpc1-subnet1 true # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat eip-dnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.4 22 22 test-dnat-vip true \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Support OVN EIP,FIP and SNAT"},{"location":"en/advance/ovn-eip-fip-snat/#support-ovn-eipfip-and-snat","text":"graph LR pod-->vpc1-subnet-->vpc1-->snat-->lrp-->external-subnet-->gw-node-external-nic The pod access the public network based on the snat graph LR pod-->vpc1-subnet-->vpc1-->fip-->lrp-->external-subnet-->local-node-external-nic The pod access the public network based on the fip The CRD supported by this function is basically the same as the iptable nat gw public network solution. ovn eip: occupies a public ip address and is allocated from the underlay provider network vlan subnet ovn fip: one-to-one dnat snat, which provides direct public network access for ip addresses and vip in a vpc ovn snat: a subnet cidr or a single vpc ip or vip can access public networks based on snat ovn dnat: based router lb, which enables direct access to a group of endpoints in a vpc based on a public endpoint","title":"Support OVN EIP,FIP and SNAT"},{"location":"en/advance/ovn-eip-fip-snat/#1-deployment","text":"Currently allows all vpcs to share the same provider vlan subnet resources, similar to neutron ovn mode. Compatible with previous scenarios default VPC EIP/SNAT . During the deployment phase, you may need to specify a default public network logical switch based on actual conditions. If no vlan is in use (vlan 0 is used), the following startup parameters do not need to be configured. # When deploying you need to refer to the above scenario and specify the following parameters as needed according to the actual situation # 1. kube-ovn-controller Startup parameters to be configured\uff1a - --external-gateway-vlanid = 204 - --external-gateway-switch = external204 # 2. kube-ovn-cni Startup parameters to be configured: - --external-gateway-switch = external204 # The above configuration is consistent with the following public network configuration vlan id and resource name, # currently only support to specify one underlay public network as the default external public network. The design and use of this configuration item takes into account the following factors\uff1a Based on this configuration item can be docked to the provider network, vlan, subnet resources. Based on this configuration item, the default vpc enable_eip_snat function can be docked to the existing vlan, subnet resources, while supporting the ipam If only the default vpc's enable_eip_snat mode is used with the old pod annotaion based eip fip snat, then the following configuration is not required. Based on this configuration you can not use the default vpc enable_eip_snat process, only by corresponding to vlan, subnet process, can be compatible with only custom vpc use eip snat usage scenarios. The neutron ovn mode also has a certain static file configuration designation that is, for now, generally consistent.","title":"1. Deployment"},{"location":"en/advance/ovn-eip-fip-snat/#11-create-the-underlay-public-network","text":"# provider-network\uff0c vlan\uff0c subnet # cat 01-provider-network.yaml apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: external204 spec: defaultInterface: vlan # cat 02-vlan.yaml apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan204 spec: id: 204 provider: external204 # cat 03-vlan-subnet.yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: external204 spec: protocol: IPv4 cidrBlock: 10 .5.204.0/24 gateway: 10 .5.204.254 vlan: vlan204 excludeIps: - 10 .5.204.1..10.5.204.100","title":"1.1 Create the underlay public network"},{"location":"en/advance/ovn-eip-fip-snat/#12-default-vpc-enable-eip_snat","text":"# Enable the default vpc and the above underlay public provider subnet interconnection cat 00 -centralized-external-gw-no-ip.yaml apiVersion: v1 kind: ConfigMap metadata: name: ovn-external-gw-config namespace: kube-system data: enable-external-gw: \"true\" external-gw-nodes: \"pc-node-1,pc-node-2,pc-node-3\" type: \"centralized\" external-gw-nic: \"vlan\" external-gw-addr: \"10.5.204.254/24\" This feature currently supports the ability to create lrp type ovn eip resources without specifying the lrp ip and mac, which is already supported for automatic acquisition. If specified, it is equivalent to specifying the ip to create an ovn-eip of type lrp. Of course, you can also manually create the lrp type ovn eip in advance.","title":"1.2 Default vpc enable eip_snat"},{"location":"en/advance/ovn-eip-fip-snat/#13-custom-vpc-enable-eip-snat-fip-function","text":"# cat 00-ns.yml apiVersion: v1 kind: Namespace metadata: name: vpc1 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true # vpc enableExternal will automatically create an lrp association to the public network specified above # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 After the above template is applied, you should see the following resources exist # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # The route currently supports automatic maintenance","title":"1.3 Custom vpc enable eip snat fip function"},{"location":"en/advance/ovn-eip-fip-snat/#2-ovn-eip","text":"This function is designed and used in the same way as iptables-eip, ovn-eip currently has three types nat: indicates ovn dnat, fip, and snat. These nat types are recorded in status lrp: indicates the resource used to connect a vpc to the public network node-ext-gw: applies to ovn BFD-based ecmp static route scenario --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: fip # Dynamically allocate an eip resource that is reserved for fip scenarios","title":"2. ovn-eip"},{"location":"en/advance/ovn-eip-fip-snat/#21-create-an-fip-for-pod","text":"# k get po -o wide -n vpc1 vpc-1-busybox01 NAME READY STATUS RESTARTS AGE IP NODE vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 # k get ip vpc-1-busybox01.vpc1 NAME V4IP V6IP MAC NODE SUBNET vpc-1-busybox01.vpc1 192 .168.0.2 00 :00:00:0A:DD:27 pc-node-2 vpc1-subnet1 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: fip --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: ovnEip: eip-static ipName: vpc-1-busybox01.vpc1 # the name of the ip crd, which is unique # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 # k get ofip eip-static NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 [ root@pc-node-1 03 -cust-vpc ] # ping 10.5.204.101 PING 10 .5.204.101 ( 10 .5.204.101 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.101: icmp_seq = 2 ttl = 62 time = 1 .21 ms 64 bytes from 10 .5.204.101: icmp_seq = 3 ttl = 62 time = 0 .624 ms 64 bytes from 10 .5.204.101: icmp_seq = 4 ttl = 62 time = 0 .368 ms ^C --- 10 .5.204.101 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3049ms rtt min/avg/max/mdev = 0 .368/0.734/1.210/0.352 ms [ root@pc-node-1 03 -cust-vpc ] # # pod <--> node ping is working # The key resources that this public ip can pass include the following ovn nb resources # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 813523e7-c68c-408f-bd8c-cba30cb2e4f4 external ip: \"10.5.204.101\" logical ip: \"192.168.0.2\" type: \"dnat_and_snat\"","title":"2.1 Create an fip for pod"},{"location":"en/advance/ovn-eip-fip-snat/#22-create-an-fip-for-vip","text":"In order to facilitate the use of some vip scenarios, such as inside kubevirt VM, keepalived use vip, kube-vip use vip, etc. the vip need public network access. # First create vip, eip, then bind eip to vip # cat vip.yaml apiVersion: kubeovn.io/v1 kind: Vip metadata: name: test-fip-vip spec: subnet: vpc1-subnet1 # cat 04-fip.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: externalSubnet: external204 type: fip --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: ovnEip: eip-for-vip ipType: vip # By default fip is for pod ip, here you need to specify the docking to vip resources ipName: test-fip-vip # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip [ root@pc-node-1 fip-vip ] # ping 10.5.204.106 PING 10 .5.204.106 ( 10 .5.204.106 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.106: icmp_seq = 1 ttl = 62 time = 0 .694 ms 64 bytes from 10 .5.204.106: icmp_seq = 2 ttl = 62 time = 0 .436 ms # node <--> pod fip is working # The way ip is used inside the pod is roughly as follows [ root@pc-node-1 fip-vip ] # k -n vpc1 exec -it vpc-1-busybox03 -- bash [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1568 : eth0@if1569: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:56:40:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.5/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 192 .168.0.3/24 scope global secondary eth0 # vip here valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe56:40e5/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox03 / ] # tcpdump -i eth0 host 192.168.0.3 -netvv tcpdump: listening on eth0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:ed:8e:c7 > 00 :00:00:56:40:e5, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 44830 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.32.51 > 192 .168.0.3: ICMP echo request, id 177 , seq 1 , length 64 00 :00:00:56:40:e5 > 00 :00:00:ed:8e:c7, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 64 , id 43962 , offset 0 , flags [ none ] , proto ICMP ( 1 ) , length 84 ) 192 .168.0.3 > 10 .5.32.51: ICMP echo reply, id 177 , seq 1 , length 64 # pod internal can catch fip related icmp packets","title":"2.2 Create an fip for vip"},{"location":"en/advance/ovn-eip-fip-snat/#3-ovn-snat","text":"","title":"3. ovn-snat"},{"location":"en/advance/ovn-eip-fip-snat/#31-ovn-snat-corresponds-to-the-cidr-of-a-subnet","text":"This feature is designed and used in much the same way as iptables-snat # cat 03-subnet-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: externalSubnet: external204 type: snat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: ovnEip: snat-for-subnet-in-vpc vpcSubnet: vpc1-subnet1 # eip corresponds to the entire network segment","title":"3.1 ovn-snat corresponds to the CIDR of a subnet"},{"location":"en/advance/ovn-eip-fip-snat/#32-ovn-fip-corresponds-to-a-pod-ip","text":"This feature is designed and used in much the same way as iptables-fip # cat 03-pod-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-pod-vpc-ip spec: externalSubnet: external204 type: snat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat01 spec: ovnEip: snat-for-pod-vpc-ip ipName: vpc-1-busybox02.vpc1 # eip corresponds to a single pod ip After the above resources are created, you can see the following resources that the snat public network feature depends on. # kubectl ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" nat da77a11f-c523-439c-b1d1-72c664196a0f external ip: \"10.5.204.116\" logical ip: \"192.168.0.4\" type: \"snat\" [ root@pc-node-1 03 -cust-vpc ] # k get po -A -o wide | grep busy vpc1 vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 <none> <none> vpc1 vpc-1-busybox02 1 /1 Running 0 17h 192 .168.0.4 pc-node-1 <none> <none> vpc1 vpc-1-busybox03 1 /1 Running 0 17h 192 .168.0.5 pc-node-1 <none> <none> vpc1 vpc-1-busybox04 1 /1 Running 0 17h 192 .168.0.6 pc-node-3 <none> <none> vpc1 vpc-1-busybox05 1 /1 Running 0 17h 192 .168.0.7 pc-node-1 <none> <none> # k exec -it -n vpc1 vpc-1-busybox04 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17095 : eth0@if17096: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:76:94:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.6/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe76:9455/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox04 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 114 time = 22 .2 ms 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 21 .8 ms [ root@pc-node-1 03 -cust-vpc ] # k exec -it -n vpc1 vpc-1-busybox02 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1566 : eth0@if1567: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:0b:e9:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.4/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe0b:e9d0/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox02 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 22 .7 ms 64 bytes from 223 .5.5.5: icmp_seq = 3 ttl = 114 time = 22 .6 ms 64 bytes from 223 .5.5.5: icmp_seq = 4 ttl = 114 time = 22 .1 ms ^C --- 223 .5.5.5 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3064ms rtt min/avg/max/mdev = 22 .126/22.518/22.741/0.278 ms # the two pods can access the external network based on these two type snat resources respectively","title":"3.2 ovn-fip corresponds to a pod IP"},{"location":"en/advance/ovn-eip-fip-snat/#4-ovn-dnat","text":"","title":"4. ovn-dnat"},{"location":"en/advance/ovn-eip-fip-snat/#41-ovn-dnat-binds-a-dnat-to-a-pod","text":"kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : externalSubnet : underlay --- kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ovnEip : eip-dnat ipName : vpc-1-busybox01.vpc1 # Note that this is the name of the pod IP CRD and it is unique protocol : tcp internalPort : \"22\" externalPort : \"22\" The configuration of OvnDnatRule is similar to that of IptablesDnatRule. # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.3 22 22 vpc-1-busybox01.vpc1 true","title":"4.1 ovn-dnat binds a DNAT to a pod"},{"location":"en/advance/ovn-eip-fip-snat/#42-ovn-dnat-binds-a-dnat-to-a-vip","text":"kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ipType : vip # By default, Dnat is oriented towards pod IPs. Here, it is necessary to specify that it is connected to VIP resources ovnEip : eip-dnat ipName : test-dnat-vip protocol : tcp internalPort : \"22\" externalPort : \"22\" The configuration of OvnDnatRule is similar to that of IptablesDnatRule. # kubectl get vip test-dnat-vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY test-dnat-vip 192 .168.0.4 00 :00:00:D0:C0:B5 vpc1-subnet1 true # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat eip-dnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.4 22 22 test-dnat-vip true \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"4.2 ovn-dnat binds a DNAT to a VIP"},{"location":"en/advance/ovn-ipsec/","text":"Encrypt inter-node communication using IPsec \u00b6 This function is supported after v1.10.11 and v1.11.4, and the kernel version is at least 3.10.0 or above. start IPsec \u00b6 Copy the script from the Kube-OVN source code ipsec.sh , execute the command as follows, the script will call ovs-pki to generate and distribute the certificate required for encryption: bash ipsec.sh init After the execution is completed, the nodes will negotiate for a period of time to establish an IPsec tunnel. The experience value is between ten seconds and one minute.You can check the IPsec status with the following command: # bash ipsec.sh status Pod { ovs-ovn-d7hdt } ipsec status... Interface name: ovn-a4718e-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.2 Remote IP: 172 .18.0.4 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem Remote cert: None Remote name: a4718e55-5b85-4f46-90e6-63527d080590 CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 2 CFM state: Disabled Kernel policies installed: src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 Kernel security associations installed: sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 IPsec connections that are active: Pod { ovs-ovn-fvbbj } ipsec status... Interface name: ovn-8aebd9-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.4 Remote IP: 172 .18.0.2 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem Local name: a4718e55-5b85-4f46-90e6-63527d080590 Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem Remote cert: None Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 1 CFM state: Disabled Kernel policies installed: src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 Kernel security associations installed: sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 IPsec connections that are active: After the establishment is complete, you can capture packets and observe that the packets have been encrypted: # tcpdump -i eth0 -nel esp 10 :01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP ( spi = 0xcc91322a,seq = 0x13d0 ) , length 156 10 :01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP ( spi = 0xc8df4221,seq = 0x1d37 ) , length 156 After executing the script, you can turn off IPsec by executing the command: # bash ipsec.sh stop Or execute the command to open it again: # bash ipsec.sh start \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN IPsec Support"},{"location":"en/advance/ovn-ipsec/#encrypt-inter-node-communication-using-ipsec","text":"This function is supported after v1.10.11 and v1.11.4, and the kernel version is at least 3.10.0 or above.","title":"Encrypt inter-node communication using IPsec"},{"location":"en/advance/ovn-ipsec/#start-ipsec","text":"Copy the script from the Kube-OVN source code ipsec.sh , execute the command as follows, the script will call ovs-pki to generate and distribute the certificate required for encryption: bash ipsec.sh init After the execution is completed, the nodes will negotiate for a period of time to establish an IPsec tunnel. The experience value is between ten seconds and one minute.You can check the IPsec status with the following command: # bash ipsec.sh status Pod { ovs-ovn-d7hdt } ipsec status... Interface name: ovn-a4718e-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.2 Remote IP: 172 .18.0.4 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem Remote cert: None Remote name: a4718e55-5b85-4f46-90e6-63527d080590 CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 2 CFM state: Disabled Kernel policies installed: src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 Kernel security associations installed: sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 IPsec connections that are active: Pod { ovs-ovn-fvbbj } ipsec status... Interface name: ovn-8aebd9-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.4 Remote IP: 172 .18.0.2 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem Local name: a4718e55-5b85-4f46-90e6-63527d080590 Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem Remote cert: None Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 1 CFM state: Disabled Kernel policies installed: src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 Kernel security associations installed: sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 IPsec connections that are active: After the establishment is complete, you can capture packets and observe that the packets have been encrypted: # tcpdump -i eth0 -nel esp 10 :01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP ( spi = 0xcc91322a,seq = 0x13d0 ) , length 156 10 :01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP ( spi = 0xc8df4221,seq = 0x1d37 ) , length 156 After executing the script, you can turn off IPsec by executing the command: # bash ipsec.sh stop Or execute the command to open it again: # bash ipsec.sh start \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"start IPsec"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/","text":"Support OVN SNAT L3 HA Based ECMP and BFD Static Route \u00b6 Custom vpc based on ovn snat after ecmp based static route hash to multiple gw node ovnext0 NICs out of the public network Supports bfd-based high availability Only supports hash load balancing graph LR pod-->vpc-subnet-->vpc-->snat-->ecmp-->external-subnet-->gw-node1-ovnext0--> node1-external-switch external-subnet-->gw-node2-ovnext0--> node2-external-switch external-subnet-->gw-node3-ovnext0--> node3-external-switch This functions basically the same as ovn-eip-fip-snat.md . As for the different parts, which will be specified in the following sections, mainly including the creation of ovn-eip of node-ext-gw type and the automatic maintenance of bfd as well as ecmp static routes based on vpc enable_bfd. 1. Deployment \u00b6 1.1 Create the underlay public network \u00b6 1.2 Default vpc enable eip_snat \u00b6 1.3 Custom vpc enable eip snat fip function \u00b6 The above section is exactly the same with ovn-eip-fip-snat.md . After these functions are verified, the vpc can be switched directly to the ecmp-based bfd static route based on the following way, or of course, switched directly back. Before customizing vpc to use this feature, you need to provide some gateway nodes, at least 2. Note that the name of the current implementation of ovn-eip must be consistent with the gateway node name, no automated maintenance is currently done for this resource. # cat gw-node-eip.yaml --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-1 spec : externalSubnet : external204 type : node-ext-gw --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-2 spec : externalSubnet : external204 type : node-ext-gw --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-3 spec : externalSubnet : external204 type : node-ext-gw Since this scenario is currently designed for vpc ecmp out of the public network, the gateway node above will not trigger the creation of a gateway NIC when there is no vpc enabled bfd, i.e. when there is no ovn eip (lrp) with enable bfd labeled, and will not be able to successfully start listening to the bfd session on the other side. 2. Custom vpc enable ecmp bfd L3 HA public network function \u00b6 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true enableBfd: true # bfd switch can be switched at will #enableBfd: false # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true # enable ecmp gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 note: Customize ecmp under vpc to use only static ecmp bfd routes. vpc enableBfd and subnet enableEcmp will only take effect if they are enabled at the same time, before static ecmp bfd routes are automatically managed. If the above configuration is turned off, it will automatically switch back to the regular default static route. This feature is not available for the default vpc, only custom vpc is supported, the default vpc has more complex policy routing. The enableEcmp of the subnet of the custom vpc uses only static routes, the gateway type gatewayType has no effect. When EnableExternal is turned off in vpc, the external network cannot be passed inside vpc. When EnableExternal is enabled on vpc, when EnableBfd is turned off, it will be based on the normal default route to the external network and will not have high availability. # After the above template is applied the ovn logic layer should see the following resources # k get vpc NAME ENABLEEXTERNAL ENABLEBFD STANDBY SUBNETS NAMESPACES ovn-cluster true true [ \"external204\" , \"join\" , \"ovn-default\" ] vpc1 true true true [ \"vpc1-subnet1\" ] [ \"vpc1\" ] # Default vpc does not support ENABLEBFD # Custom vpc is supported and enabled # 1. bfd table created # k ko nbctl list bfd _uuid : be7df545-2c4c-4751-878f-b3507987f050 detect_mult : 3 dst_ip : \"10.5.204.121\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : 684c4489-5b59-4693-8d8c-3beab93f8093 detect_mult : 3 dst_ip : \"10.5.204.109\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : f0f62077-2ae9-4e79-b4f8-a446ec6e784c detect_mult : 3 dst_ip : \"10.5.204.108\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up ### Note that all statuses should normally be up # 2. bfd ecmp static routes table created # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 192 .168.0.0/24 10 .5.204.108 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.109 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.121 src-ip ecmp ecmp-symmetric-reply bfd # 3. Static Route Details # k ko nbctl find Logical_Router_Static_Route policy=src-ip options=ecmp_symmetric_reply=\"true\" _uuid : 3aacb384-d5ee-4b14-aebf-59e8c11717ba bfd : 684c4489-5b59-4693-8d8c-3beab93f8093 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.109\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 18bcc585-bc05-430b-925b-ef673c8e1aef bfd : f0f62077-2ae9-4e79-b4f8-a446ec6e784c external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.108\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 7d0a4e6b-cde0-4110-8176-fbaf19738498 bfd : be7df545-2c4c-4751-878f-b3507987f050 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.121\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" # Also, the following resources should be available at all gateway nodes [ root@pc-node-1 ~ ] # ip netns exec ovnext bash ip a /usr/sbin/ip: /usr/sbin/ip: cannot execute binary file [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1541 : ovnext0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 00 :00:00:ab:bd:87 brd ff:ff:ff:ff:ff:ff inet 10 .5.204.108/24 brd 10 .5.204.255 scope global ovnext0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:feab:bd87/64 scope link valid_lft forever preferred_lft forever [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0 .0.0.0 10 .5.204.254 0 .0.0.0 UG 0 0 0 ovnext0 10 .5.204.0 0 .0.0.0 255 .255.255.0 U 0 0 0 ovnext0 [ root@pc-node-1 ~ ] # ip netns exec ovnext bfdd-control status There are 1 sessions: Session 1 id = 1 local = 10 .5.204.108 ( p ) remote = 10 .5.204.122 state = Up ## This is the other end of the lrp bfd session and one of the next hops of the lrp ecmp [ root@pc-node-1 ~ ] # ip netns exec ovnext ping -c1 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 115 time = 21 .6 ms # No problem to the public network catch outgoing packets within the ovnext ns of a gateway node # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-1 ~ ] # exit [ root@pc-node-1 ~ ] # ssh pc-node-2 Last login: Thu Feb 23 09 :21:08 2023 from 10 .5.32.51 [ root@pc-node-2 ~ ] # ip netns exec ovnext bash [ root@pc-node-2 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-2 ~ ] # exit [ root@pc-node-2 ~ ] # logout Connection to pc-node-2 closed. [ root@pc-node-1 ~ ] # ssh pc-node-3 Last login: Thu Feb 23 08 :32:41 2023 from 10 .5.32.51 [ root@pc-node-3 ~ ] # ip netns exec ovnext bash [ root@pc-node-3 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:2d:f8:ce > 00 :00:00:fd:b2:a4, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 63 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 00 :00:00:fd:b2:a4 > dc:ef:80:5a:44:1a, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [ root@pc-node-3 ~ ] # 3. Turn off bfd mode \u00b6 In some scenarios, you may want to use a (centralized) single gateway directly out of the public network, which is the same as the default vpc enable_eip_snat usage pattern # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc2 spec: namespaces: - vpc2 enableExternal: true #enableBfd: true enableBfd: false ## set it false add apply # k ko nbctl lr-route-list vpc2 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # After application the route will switch back to the normal default static route # nbctl list bfd, the bfd session associated with lrp has been removed # And the opposite side of the bfd session in ovnext ns is automatically removed \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Support OVN SNAT L3 HA Based ECMP and BFD Static Route"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#support-ovn-snat-l3-ha-based-ecmp-and-bfd-static-route","text":"Custom vpc based on ovn snat after ecmp based static route hash to multiple gw node ovnext0 NICs out of the public network Supports bfd-based high availability Only supports hash load balancing graph LR pod-->vpc-subnet-->vpc-->snat-->ecmp-->external-subnet-->gw-node1-ovnext0--> node1-external-switch external-subnet-->gw-node2-ovnext0--> node2-external-switch external-subnet-->gw-node3-ovnext0--> node3-external-switch This functions basically the same as ovn-eip-fip-snat.md . As for the different parts, which will be specified in the following sections, mainly including the creation of ovn-eip of node-ext-gw type and the automatic maintenance of bfd as well as ecmp static routes based on vpc enable_bfd.","title":"Support OVN SNAT L3 HA Based ECMP and BFD Static Route"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#1-deployment","text":"","title":"1. Deployment"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#11-create-the-underlay-public-network","text":"","title":"1.1 Create the underlay public network"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#12-default-vpc-enable-eip_snat","text":"","title":"1.2 Default vpc enable eip_snat"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#13-custom-vpc-enable-eip-snat-fip-function","text":"The above section is exactly the same with ovn-eip-fip-snat.md . After these functions are verified, the vpc can be switched directly to the ecmp-based bfd static route based on the following way, or of course, switched directly back. Before customizing vpc to use this feature, you need to provide some gateway nodes, at least 2. Note that the name of the current implementation of ovn-eip must be consistent with the gateway node name, no automated maintenance is currently done for this resource. # cat gw-node-eip.yaml --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-1 spec : externalSubnet : external204 type : node-ext-gw --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-2 spec : externalSubnet : external204 type : node-ext-gw --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-3 spec : externalSubnet : external204 type : node-ext-gw Since this scenario is currently designed for vpc ecmp out of the public network, the gateway node above will not trigger the creation of a gateway NIC when there is no vpc enabled bfd, i.e. when there is no ovn eip (lrp) with enable bfd labeled, and will not be able to successfully start listening to the bfd session on the other side.","title":"1.3 Custom vpc enable eip snat fip function"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#2-custom-vpc-enable-ecmp-bfd-l3-ha-public-network-function","text":"# cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true enableBfd: true # bfd switch can be switched at will #enableBfd: false # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true # enable ecmp gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 note: Customize ecmp under vpc to use only static ecmp bfd routes. vpc enableBfd and subnet enableEcmp will only take effect if they are enabled at the same time, before static ecmp bfd routes are automatically managed. If the above configuration is turned off, it will automatically switch back to the regular default static route. This feature is not available for the default vpc, only custom vpc is supported, the default vpc has more complex policy routing. The enableEcmp of the subnet of the custom vpc uses only static routes, the gateway type gatewayType has no effect. When EnableExternal is turned off in vpc, the external network cannot be passed inside vpc. When EnableExternal is enabled on vpc, when EnableBfd is turned off, it will be based on the normal default route to the external network and will not have high availability. # After the above template is applied the ovn logic layer should see the following resources # k get vpc NAME ENABLEEXTERNAL ENABLEBFD STANDBY SUBNETS NAMESPACES ovn-cluster true true [ \"external204\" , \"join\" , \"ovn-default\" ] vpc1 true true true [ \"vpc1-subnet1\" ] [ \"vpc1\" ] # Default vpc does not support ENABLEBFD # Custom vpc is supported and enabled # 1. bfd table created # k ko nbctl list bfd _uuid : be7df545-2c4c-4751-878f-b3507987f050 detect_mult : 3 dst_ip : \"10.5.204.121\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : 684c4489-5b59-4693-8d8c-3beab93f8093 detect_mult : 3 dst_ip : \"10.5.204.109\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : f0f62077-2ae9-4e79-b4f8-a446ec6e784c detect_mult : 3 dst_ip : \"10.5.204.108\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up ### Note that all statuses should normally be up # 2. bfd ecmp static routes table created # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 192 .168.0.0/24 10 .5.204.108 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.109 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.121 src-ip ecmp ecmp-symmetric-reply bfd # 3. Static Route Details # k ko nbctl find Logical_Router_Static_Route policy=src-ip options=ecmp_symmetric_reply=\"true\" _uuid : 3aacb384-d5ee-4b14-aebf-59e8c11717ba bfd : 684c4489-5b59-4693-8d8c-3beab93f8093 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.109\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 18bcc585-bc05-430b-925b-ef673c8e1aef bfd : f0f62077-2ae9-4e79-b4f8-a446ec6e784c external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.108\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 7d0a4e6b-cde0-4110-8176-fbaf19738498 bfd : be7df545-2c4c-4751-878f-b3507987f050 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.121\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" # Also, the following resources should be available at all gateway nodes [ root@pc-node-1 ~ ] # ip netns exec ovnext bash ip a /usr/sbin/ip: /usr/sbin/ip: cannot execute binary file [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1541 : ovnext0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 00 :00:00:ab:bd:87 brd ff:ff:ff:ff:ff:ff inet 10 .5.204.108/24 brd 10 .5.204.255 scope global ovnext0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:feab:bd87/64 scope link valid_lft forever preferred_lft forever [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0 .0.0.0 10 .5.204.254 0 .0.0.0 UG 0 0 0 ovnext0 10 .5.204.0 0 .0.0.0 255 .255.255.0 U 0 0 0 ovnext0 [ root@pc-node-1 ~ ] # ip netns exec ovnext bfdd-control status There are 1 sessions: Session 1 id = 1 local = 10 .5.204.108 ( p ) remote = 10 .5.204.122 state = Up ## This is the other end of the lrp bfd session and one of the next hops of the lrp ecmp [ root@pc-node-1 ~ ] # ip netns exec ovnext ping -c1 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 115 time = 21 .6 ms # No problem to the public network catch outgoing packets within the ovnext ns of a gateway node # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-1 ~ ] # exit [ root@pc-node-1 ~ ] # ssh pc-node-2 Last login: Thu Feb 23 09 :21:08 2023 from 10 .5.32.51 [ root@pc-node-2 ~ ] # ip netns exec ovnext bash [ root@pc-node-2 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-2 ~ ] # exit [ root@pc-node-2 ~ ] # logout Connection to pc-node-2 closed. [ root@pc-node-1 ~ ] # ssh pc-node-3 Last login: Thu Feb 23 08 :32:41 2023 from 10 .5.32.51 [ root@pc-node-3 ~ ] # ip netns exec ovnext bash [ root@pc-node-3 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:2d:f8:ce > 00 :00:00:fd:b2:a4, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 63 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 00 :00:00:fd:b2:a4 > dc:ef:80:5a:44:1a, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [ root@pc-node-3 ~ ] #","title":"2. Custom vpc enable ecmp bfd L3 HA public network function"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#3-turn-off-bfd-mode","text":"In some scenarios, you may want to use a (centralized) single gateway directly out of the public network, which is the same as the default vpc enable_eip_snat usage pattern # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc2 spec: namespaces: - vpc2 enableExternal: true #enableBfd: true enableBfd: false ## set it false add apply # k ko nbctl lr-route-list vpc2 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # After application the route will switch back to the normal default static route # nbctl list bfd, the bfd session associated with lrp has been removed # And the opposite side of the bfd session in ovnext ns is automatically removed \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"3. Turn off bfd mode"},{"location":"en/advance/ovn-remote-port-mirroring/","text":"OVN Remote Port Mirroring \u00b6 This feature provides ability to mirror the traffic of the specified Pod and direction, and to send the mirrored traffic to a remote destination. This feature requires Kube-OVN version not lower than v1.12. Install Multus-CNI \u00b6 Install Multus-CNI by referring the Multus-CNI Document . Create NetworkAttachmentDefinition \u00b6 Create the following NetworkAttachmentDefinition: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : | { \"cniVersion\": \"0.3.1\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" } Format of the provider field is <NAME>.<NAMESPACE>.ovn . Create Underlay Network \u00b6 The mirrored traffic is encapsulated before transmition, so MTU of the network used to transmit the traffic should be greater than the mirrored LSP/Pod. Here we are using an underlay network. Create the following underlay network: apiVersion : kubeovn.io/v1 kind : ProviderNetwork metadata : name : net1 spec : defaultInterface : eth1 --- apiVersion : kubeovn.io/v1 kind : Vlan metadata : name : vlan1 spec : id : 0 provider : net1 --- apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.19.0.0/16 excludeIps : - 172.19.0.2..172.19.0.20 gateway : 172.19.0.1 vlan : vlan1 provider : attachnet.default.ovn The subnet's provider MUST be the same as the provider of the NetworkAttachmentDefinition created above. Create Receiving Pod \u00b6 Create the following Pod\uff1a apiVersion : v1 kind : Pod metadata : name : pod1 annotations : k8s.v1.cni.cncf.io/networks : default/attachnet spec : containers : - name : bash image : docker.io/kubeovn/kube-ovn:v1.12.0 args : - bash - -c - sleep infinity securityContext : privileged : true After the Pod has been created, checkout the IP addresses: $ kubectl get ips | grep pod1 pod1.default 10 .16.0.12 00 :00:00:FF:34:24 kube-ovn-worker ovn-default pod1.default.attachnet.default.ovn 172 .19.0.21 00 :00:00:A0:30:68 kube-ovn-worker subnet1 The IP address 172.19.0.21 will be used later. Create OVN Remote Port Mirroring \u00b6 Create the following OVN remote port mirroring\uff1a kubectl ko nbctl mirror-add mirror1 gre 99 from-lport 172 .19.0.21 kubectl ko nbctl lsp-attach-mirror coredns-787d4945fb-gpnkb.kube-system mirror1 coredns-787d4945fb-gpnkb.kube-system is the OVN LSP name with a format <POD_NAME>.<POD_NAMESPACE> . Here is the OVN command usage: ovn-nbctl mirror-add <NAME> <TYPE> <INDEX> <FILTER> <IP> NAME - add a mirror with given name TYPE - specify TYPE 'gre' or 'erspan' INDEX - specify the tunnel INDEX value (indicates key if GRE, erpsan_idx if ERSPAN) FILTER - specify FILTER for mirroring selection ('to-lport' / 'from-lport') IP - specify Sink / Destination i.e. Remote IP ovn-nbctl mirror-del [NAME] remove mirrors ovn-nbctl mirror-list print mirrors ovn-nbctl lsp-attach-mirror PORT MIRROR attach source PORT to MIRROR ovn-nbctl lsp-detach-mirror PORT MIRROR detach source PORT from MIRROR Configure Receiving Pod \u00b6 Execute the following commands in the Pod: root@pod1:/kube-ovn# ip link add mirror1 type gretap local 172 .19.0.21 key 99 dev net1 root@pod1:/kube-ovn# ip link set mirror1 up Now you can capture the mirrored packets: root@pod1:/kube-ovn# tcpdump -i mirror1 -nnve tcpdump: listening on mirror1, link-type EN10MB ( Ethernet ) , snapshot length 262144 bytes 05 :13:30.328808 00 :00:00:a3:f5:e2 > 00 :00:00:97:0f:6e, ethertype ARP ( 0x0806 ) , length 42 : Ethernet ( len 6 ) , IPv4 ( len 4 ) , Request who-has 10 .16.0.7 tell 10 .16.0.4, length 28 05 :13:30.559167 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57364 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.50472: 34511 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.559343 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57365 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.45177: 1659 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.560625 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 200 : ( tos 0x0, ttl 64 , id 57367 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 186 ) 10 .16.0.4.53 > 10 .16.0.6.43848: 2636 *- 0 /1/1 ( 158 ) 05 :13:30.562774 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 191 : ( tos 0x0, ttl 64 , id 57368 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 177 ) 10 .16.0.4.53 > 10 .16.0.6.37755: 48737 NXDomain*- 0 /1/1 ( 149 ) 05 :13:30.563523 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 187 : ( tos 0x0, ttl 64 , id 57369 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 173 ) 10 .16.0.4.53 > 10 .16.0.6.53887: 45519 NXDomain*- 0 /1/1 ( 145 ) 05 :13:30.564940 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57370 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.40846: 25745 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.565140 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57371 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.45214: 61875 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.566023 00 :00:00:a3:f5:e2 > 00 :00:00:55:e4:4e, ethertype IPv4 ( 0x0800 ) , length 80 : ( tos 0x0, ttl 64 , id 45937 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 66 ) 10 .16.0.4.44116 > 172 .18.0.1.53: 16025 + [ 1au ] AAAA? alauda.cn. ( 38 ) Notice \u00b6 If you are using ERSPAN as the encapsulation protocol, the Linux kernel version of the OVN nodes and remote devices must not be lower than 4.14. If you are using ERSPAN as the encapsulation protocol and using IPv6 as the transport network, the Linux kernel version must not be lower than 4.16. The transmission of mirrored traffic is unidirectional, so you only need to ensure that the OVN node can access the remote device. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN Remote Port Mirroring"},{"location":"en/advance/ovn-remote-port-mirroring/#ovn-remote-port-mirroring","text":"This feature provides ability to mirror the traffic of the specified Pod and direction, and to send the mirrored traffic to a remote destination. This feature requires Kube-OVN version not lower than v1.12.","title":"OVN Remote Port Mirroring"},{"location":"en/advance/ovn-remote-port-mirroring/#install-multus-cni","text":"Install Multus-CNI by referring the Multus-CNI Document .","title":"Install Multus-CNI"},{"location":"en/advance/ovn-remote-port-mirroring/#create-networkattachmentdefinition","text":"Create the following NetworkAttachmentDefinition: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : | { \"cniVersion\": \"0.3.1\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" } Format of the provider field is <NAME>.<NAMESPACE>.ovn .","title":"Create NetworkAttachmentDefinition"},{"location":"en/advance/ovn-remote-port-mirroring/#create-underlay-network","text":"The mirrored traffic is encapsulated before transmition, so MTU of the network used to transmit the traffic should be greater than the mirrored LSP/Pod. Here we are using an underlay network. Create the following underlay network: apiVersion : kubeovn.io/v1 kind : ProviderNetwork metadata : name : net1 spec : defaultInterface : eth1 --- apiVersion : kubeovn.io/v1 kind : Vlan metadata : name : vlan1 spec : id : 0 provider : net1 --- apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.19.0.0/16 excludeIps : - 172.19.0.2..172.19.0.20 gateway : 172.19.0.1 vlan : vlan1 provider : attachnet.default.ovn The subnet's provider MUST be the same as the provider of the NetworkAttachmentDefinition created above.","title":"Create Underlay Network"},{"location":"en/advance/ovn-remote-port-mirroring/#create-receiving-pod","text":"Create the following Pod\uff1a apiVersion : v1 kind : Pod metadata : name : pod1 annotations : k8s.v1.cni.cncf.io/networks : default/attachnet spec : containers : - name : bash image : docker.io/kubeovn/kube-ovn:v1.12.0 args : - bash - -c - sleep infinity securityContext : privileged : true After the Pod has been created, checkout the IP addresses: $ kubectl get ips | grep pod1 pod1.default 10 .16.0.12 00 :00:00:FF:34:24 kube-ovn-worker ovn-default pod1.default.attachnet.default.ovn 172 .19.0.21 00 :00:00:A0:30:68 kube-ovn-worker subnet1 The IP address 172.19.0.21 will be used later.","title":"Create Receiving Pod"},{"location":"en/advance/ovn-remote-port-mirroring/#create-ovn-remote-port-mirroring","text":"Create the following OVN remote port mirroring\uff1a kubectl ko nbctl mirror-add mirror1 gre 99 from-lport 172 .19.0.21 kubectl ko nbctl lsp-attach-mirror coredns-787d4945fb-gpnkb.kube-system mirror1 coredns-787d4945fb-gpnkb.kube-system is the OVN LSP name with a format <POD_NAME>.<POD_NAMESPACE> . Here is the OVN command usage: ovn-nbctl mirror-add <NAME> <TYPE> <INDEX> <FILTER> <IP> NAME - add a mirror with given name TYPE - specify TYPE 'gre' or 'erspan' INDEX - specify the tunnel INDEX value (indicates key if GRE, erpsan_idx if ERSPAN) FILTER - specify FILTER for mirroring selection ('to-lport' / 'from-lport') IP - specify Sink / Destination i.e. Remote IP ovn-nbctl mirror-del [NAME] remove mirrors ovn-nbctl mirror-list print mirrors ovn-nbctl lsp-attach-mirror PORT MIRROR attach source PORT to MIRROR ovn-nbctl lsp-detach-mirror PORT MIRROR detach source PORT from MIRROR","title":"Create OVN Remote Port Mirroring"},{"location":"en/advance/ovn-remote-port-mirroring/#configure-receiving-pod","text":"Execute the following commands in the Pod: root@pod1:/kube-ovn# ip link add mirror1 type gretap local 172 .19.0.21 key 99 dev net1 root@pod1:/kube-ovn# ip link set mirror1 up Now you can capture the mirrored packets: root@pod1:/kube-ovn# tcpdump -i mirror1 -nnve tcpdump: listening on mirror1, link-type EN10MB ( Ethernet ) , snapshot length 262144 bytes 05 :13:30.328808 00 :00:00:a3:f5:e2 > 00 :00:00:97:0f:6e, ethertype ARP ( 0x0806 ) , length 42 : Ethernet ( len 6 ) , IPv4 ( len 4 ) , Request who-has 10 .16.0.7 tell 10 .16.0.4, length 28 05 :13:30.559167 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57364 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.50472: 34511 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.559343 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57365 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.45177: 1659 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.560625 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 200 : ( tos 0x0, ttl 64 , id 57367 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 186 ) 10 .16.0.4.53 > 10 .16.0.6.43848: 2636 *- 0 /1/1 ( 158 ) 05 :13:30.562774 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 191 : ( tos 0x0, ttl 64 , id 57368 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 177 ) 10 .16.0.4.53 > 10 .16.0.6.37755: 48737 NXDomain*- 0 /1/1 ( 149 ) 05 :13:30.563523 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 187 : ( tos 0x0, ttl 64 , id 57369 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 173 ) 10 .16.0.4.53 > 10 .16.0.6.53887: 45519 NXDomain*- 0 /1/1 ( 145 ) 05 :13:30.564940 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57370 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.40846: 25745 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.565140 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57371 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.45214: 61875 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.566023 00 :00:00:a3:f5:e2 > 00 :00:00:55:e4:4e, ethertype IPv4 ( 0x0800 ) , length 80 : ( tos 0x0, ttl 64 , id 45937 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 66 ) 10 .16.0.4.44116 > 172 .18.0.1.53: 16025 + [ 1au ] AAAA? alauda.cn. ( 38 )","title":"Configure Receiving Pod"},{"location":"en/advance/ovn-remote-port-mirroring/#notice","text":"If you are using ERSPAN as the encapsulation protocol, the Linux kernel version of the OVN nodes and remote devices must not be lower than 4.14. If you are using ERSPAN as the encapsulation protocol and using IPv6 as the transport network, the Linux kernel version must not be lower than 4.16. The transmission of mirrored traffic is unidirectional, so you only need to ensure that the OVN node can access the remote device. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Notice"},{"location":"en/advance/performance-tuning/","text":"Performance Tuning \u00b6 To keep the installation simple and feature-complete, the default installation script for Kube-OVN does not have performance-specific optimizations. If the applications are sensitive to latency and throughput, administrators can use this document to make specific performance optimizations. The community will continue to iterate on the performance. Some general performance optimizations have been integrated into the latest version, so it is recommended to use the latest version to get better default performance. For more on the process and methodology of performance optimization, please watch the video Kube-OVN \u5bb9\u5668\u6027\u80fd\u4f18\u5316\u4e4b\u65c5 \u3002 Benchmarking \u00b6 Because the hardware and software environments vary greatly, the performance test data provided here can only be used as a reference, and the actual test results may differ significantly from the results in this document. It is recommended to compare the performance test results before and after optimization, and the performance comparison between the host network and the container network. Overlay Performance Comparison before and after Optimization \u00b6 Environment: Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 Overlay Mode CPU: Intel(R) Xeon(R) E-2278G Network: 2*10Gbps, xmit_hash_policy=layer3+4 We use qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw to test bandwidth and latency of tcp/udp in 1-byte packets and the host network, respectively. Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Default 25.7 22.9 27.1 1.59 Kube-OVN Optimized 13.9 12.9 27.6 5.57 HOST Network 13.1 12.4 28.2 6.02 Overlay\uff0c Underlay and Calico Comparison \u00b6 Next, we compare the overlay and underlay performance of the optimized Kube-OVN at different packet sizes with Calico's IPIP Always , IPIP never and the host network. Environment : Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 CPU: AMD EPYC 7402P 24-Core Processor Network: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Overlay 15.2 14.6 23.6 2.65 Kube-OVN Underlay 14.3 13.8 24.2 3.46 Calico IPIP 21.4 20.2 23.6 1.18 Calico NoEncap 19.3 16.9 23.6 1.76 HOST Network 16.6 15.4 24.8 2.64 qperf -t 60 <server ip> -ub -oo msg_size:1K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 16.5 15.8 10.2 2.77 Kube-OVN Underlay 15.9 14.5 9.6 3.22 Calico IPIP 22.5 21.5 1.45 1.14 Calico NoEncap 19.4 18.3 3.76 1.63 HOST Network 18.1 16.6 9.32 2.66 qperf -t 60 <server ip> -ub -oo msg_size:4K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 34.7 41.6 16.0 9.23 Kube-OVN Underlay 32.6 44 15.1 6.71 Calico IPIP 44.8 52.9 2.94 3.26 Calico NoEncap 40 49.6 6.56 4.19 HOST Network 35.9 45.9 14.6 5.59 In some cases the container network outperforms the host network, this is because the container network path is optimized to completely bypass netfilter. Due to the existence of kube-proxy , all packets in host network have to go through netfilter, which will lead to more CPU consumption, so that container network in some environments has better performance. Dataplane performance optimization methods \u00b6 The optimization methods described here are related to the hardware and software environment and the desired functionality, so please carefully understand the prerequisites for optimization before attempting it. CPU Performance Mode Tuning \u00b6 In some environments the CPU is running in power saving mode, performance in this mode will be unstable and latency will increase significantly, it is recommended to use the CPU's performance mode for more stable performance. cpupower frequency-set -g performance NIC Hardware Queue Adjustment \u00b6 In the case of increased traffic, a small buffer queue may lead to significant performance degradation due to a high packet loss rate and needs to be tuned. Check the current NIC queue length: # ethtool -g eno1 Ring parameters for eno1: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 255 RX Mini: 0 RX Jumbo: 0 TX: 255 Increase the queue length to the maximum: ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096 Optimize with tuned \u00b6 tuned can use a series of preconfigured profile files to perform system optimizations for a specific scenario. For latency-first scenarios: tuned-adm profile network-latency For throughput-first scenarios: tuned-adm profile network-throughput Interrupt Binding \u00b6 We recommend disabling irqbalance and binding NIC interrupts to specific CPUs to avoid performance fluctuations caused by switching between multiple CPUs. Disable OVN LB \u00b6 The L2 LB implementation of OVN requires calling the kernel's conntrack module and recirculate, resulting in a significant CPU overhead, which is tested to be around 20%. For Overlay networks you can use kube-proxy to complete the service forwarding function for better Pod-to-Pod performance. This can be turned off in kube-ovn-controller args: command : - /kube-ovn/start-controller.sh args : ... - --enable-lb=false ... In Underlay mode kube-proxy cannot use iptables or ipvs to control container network traffic, if you want to disable the LB function, you need to confirm whether you do not need the Service function. FastPath Kernel Module \u00b6 Since the container network and the host network are on different network ns, the packets will pass through the netfilter module several times when they are transmitted across the host, which results in a CPU overhead of nearly 20%. The FastPath module can reduce CPU overhead by bypassing netfilter, since in most cases applications within a container network do not need to use the functionality of the netfilter module. If you need to use the functions provided by netfilter such as iptables, ipvs, nftables, etc. in the container network, this module will disable the related functions. Since kernel modules are kernel version dependent, it is not possible to provide a single kernel module artifact that adapts to all kernels. We pre-compiled the FastPath module for part of the kernels, which can be accessed by tunning-package . You can also compile it manually, see Compiling FastPath Module After obtaining the kernel module, you can load the FastPath module on each node using insmod kube_ovn_fastpath.ko and verify that the module was loaded successfully using dmesg : # dmesg ... [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in ... OVS Kernel Module Optimization \u00b6 OVS flow processing including hashing, matching, etc. consumes about 10% of the CPU resources. Some instruction sets on modern x86 CPUs such as popcnt and sse4.2 can speed up the computation process, but the kernel is not compiled with these options enabled. It has been tested that the CPU consumption of flow-related operations is reduced to about 5% when the corresponding instruction set optimizations are enabled. Similar to the compilation of the FastPath module, it is not possible to provide a single kernel module artifact for all kernels. Users need to compile manually or go to tunning-package to see if a compiled package is available for download. Before using this kernel module, please check if the CPU supports the following instruction set: cat /proc/cpuinfo | grep popcnt cat /proc/cpuinfo | grep sse4_2 Compile and Install in CentOS \u00b6 Install the relevant compilation dependencies and kernel headers: yum install -y gcc kernel-devel- $( uname -r ) python3 autoconf automake libtool rpm-build openssl-devel Compile the OVS kernel module and generate the corresponding RPM: git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --with-linux = /lib/modules/ $( uname -r ) /build CFLAGS = \"-g -O2 -mpopcnt -msse4.2\" make rpm-fedora-kmod cd rpm/rpmbuild/RPMS/x86_64/ Copy the RPM to each node and install: rpm -i openvswitch-kmod-2.15.2-1.el7.x86_64.rpm If you have previously started Kube-OVN and the older version of the OVS module has been loaded into the kernel. It is recommended to reboot the machine to reload the new version of the kernel module. Compile and Install in Ubuntu \u00b6 Install the relevant compilation dependencies and kernel headers: apt install -y autoconf automake libtool gcc build-essential libssl-dev Compile the OVS kernel module and install: apt install -y autoconf automake libtool gcc build-essential libssl-dev git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --prefix = /usr/ --localstatedir = /var --enable-ssl --with-linux = /lib/modules/ $( uname -r ) /build make -j ` nproc ` make install make modules_install cat > /etc/depmod.d/openvswitch.conf << EOF override openvswitch * extra override vport-* * extra EOF depmod -a cp debian/openvswitch-switch.init /etc/init.d/openvswitch-switch /etc/init.d/openvswitch-switch force-reload-kmod If you have previously started Kube-OVN and the older version of the OVS module has been loaded into the kernel. It is recommended to reboot the machine to reload the new version of the kernel module. Using STT Type Tunnel \u00b6 Common tunnel encapsulation protocols such as Geneve and Vxlan use the UDP protocol to encapsulate packets and are well supported in the kernel. However, when TCP packets are encapsulated using UDP, the optimization and offload features of modern operating systems and network cards for the TCP protocol do not work well, resulting in a significant drop in TCP throughput. In some virtualization scenarios, due to CPU limitations, TCP packet throughput may even be a tenth of that of the host network. STT provides an innovative tunneling protocol that uses TCP formatted header for encapsulation. This encapsulation only emulates the TCP protocol header format without actually establishing a TCP connection, but can take full advantage of the TCP optimization capabilities of modern operating systems and network cards. In our tests TCP packet throughput can be improved several times, reaching performance levels close to those of the host network. The STT tunnel is not pre-installed in the kernel and needs to be installed by compiling the OVS kernel module, which can be found in the previous section. Enable STT tunnel: kubectl set env daemonset/ovs-ovn -n kube-system TUNNEL_TYPE = stt kubectl delete pod -n kube-system -lapp = ovs \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Performance Tuning"},{"location":"en/advance/performance-tuning/#performance-tuning","text":"To keep the installation simple and feature-complete, the default installation script for Kube-OVN does not have performance-specific optimizations. If the applications are sensitive to latency and throughput, administrators can use this document to make specific performance optimizations. The community will continue to iterate on the performance. Some general performance optimizations have been integrated into the latest version, so it is recommended to use the latest version to get better default performance. For more on the process and methodology of performance optimization, please watch the video Kube-OVN \u5bb9\u5668\u6027\u80fd\u4f18\u5316\u4e4b\u65c5 \u3002","title":"Performance Tuning"},{"location":"en/advance/performance-tuning/#benchmarking","text":"Because the hardware and software environments vary greatly, the performance test data provided here can only be used as a reference, and the actual test results may differ significantly from the results in this document. It is recommended to compare the performance test results before and after optimization, and the performance comparison between the host network and the container network.","title":"Benchmarking"},{"location":"en/advance/performance-tuning/#overlay-performance-comparison-before-and-after-optimization","text":"Environment: Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 Overlay Mode CPU: Intel(R) Xeon(R) E-2278G Network: 2*10Gbps, xmit_hash_policy=layer3+4 We use qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw to test bandwidth and latency of tcp/udp in 1-byte packets and the host network, respectively. Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Default 25.7 22.9 27.1 1.59 Kube-OVN Optimized 13.9 12.9 27.6 5.57 HOST Network 13.1 12.4 28.2 6.02","title":"Overlay Performance Comparison before and after Optimization"},{"location":"en/advance/performance-tuning/#overlay-underlay-and-calico-comparison","text":"Next, we compare the overlay and underlay performance of the optimized Kube-OVN at different packet sizes with Calico's IPIP Always , IPIP never and the host network. Environment : Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 CPU: AMD EPYC 7402P 24-Core Processor Network: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Overlay 15.2 14.6 23.6 2.65 Kube-OVN Underlay 14.3 13.8 24.2 3.46 Calico IPIP 21.4 20.2 23.6 1.18 Calico NoEncap 19.3 16.9 23.6 1.76 HOST Network 16.6 15.4 24.8 2.64 qperf -t 60 <server ip> -ub -oo msg_size:1K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 16.5 15.8 10.2 2.77 Kube-OVN Underlay 15.9 14.5 9.6 3.22 Calico IPIP 22.5 21.5 1.45 1.14 Calico NoEncap 19.4 18.3 3.76 1.63 HOST Network 18.1 16.6 9.32 2.66 qperf -t 60 <server ip> -ub -oo msg_size:4K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 34.7 41.6 16.0 9.23 Kube-OVN Underlay 32.6 44 15.1 6.71 Calico IPIP 44.8 52.9 2.94 3.26 Calico NoEncap 40 49.6 6.56 4.19 HOST Network 35.9 45.9 14.6 5.59 In some cases the container network outperforms the host network, this is because the container network path is optimized to completely bypass netfilter. Due to the existence of kube-proxy , all packets in host network have to go through netfilter, which will lead to more CPU consumption, so that container network in some environments has better performance.","title":"Overlay\uff0c Underlay and Calico Comparison"},{"location":"en/advance/performance-tuning/#dataplane-performance-optimization-methods","text":"The optimization methods described here are related to the hardware and software environment and the desired functionality, so please carefully understand the prerequisites for optimization before attempting it.","title":"Dataplane performance optimization methods"},{"location":"en/advance/performance-tuning/#cpu-performance-mode-tuning","text":"In some environments the CPU is running in power saving mode, performance in this mode will be unstable and latency will increase significantly, it is recommended to use the CPU's performance mode for more stable performance. cpupower frequency-set -g performance","title":"CPU Performance Mode Tuning"},{"location":"en/advance/performance-tuning/#nic-hardware-queue-adjustment","text":"In the case of increased traffic, a small buffer queue may lead to significant performance degradation due to a high packet loss rate and needs to be tuned. Check the current NIC queue length: # ethtool -g eno1 Ring parameters for eno1: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 255 RX Mini: 0 RX Jumbo: 0 TX: 255 Increase the queue length to the maximum: ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096","title":"NIC Hardware Queue Adjustment"},{"location":"en/advance/performance-tuning/#optimize-with-tuned","text":"tuned can use a series of preconfigured profile files to perform system optimizations for a specific scenario. For latency-first scenarios: tuned-adm profile network-latency For throughput-first scenarios: tuned-adm profile network-throughput","title":"Optimize with tuned"},{"location":"en/advance/performance-tuning/#interrupt-binding","text":"We recommend disabling irqbalance and binding NIC interrupts to specific CPUs to avoid performance fluctuations caused by switching between multiple CPUs.","title":"Interrupt Binding"},{"location":"en/advance/performance-tuning/#disable-ovn-lb","text":"The L2 LB implementation of OVN requires calling the kernel's conntrack module and recirculate, resulting in a significant CPU overhead, which is tested to be around 20%. For Overlay networks you can use kube-proxy to complete the service forwarding function for better Pod-to-Pod performance. This can be turned off in kube-ovn-controller args: command : - /kube-ovn/start-controller.sh args : ... - --enable-lb=false ... In Underlay mode kube-proxy cannot use iptables or ipvs to control container network traffic, if you want to disable the LB function, you need to confirm whether you do not need the Service function.","title":"Disable OVN LB"},{"location":"en/advance/performance-tuning/#fastpath-kernel-module","text":"Since the container network and the host network are on different network ns, the packets will pass through the netfilter module several times when they are transmitted across the host, which results in a CPU overhead of nearly 20%. The FastPath module can reduce CPU overhead by bypassing netfilter, since in most cases applications within a container network do not need to use the functionality of the netfilter module. If you need to use the functions provided by netfilter such as iptables, ipvs, nftables, etc. in the container network, this module will disable the related functions. Since kernel modules are kernel version dependent, it is not possible to provide a single kernel module artifact that adapts to all kernels. We pre-compiled the FastPath module for part of the kernels, which can be accessed by tunning-package . You can also compile it manually, see Compiling FastPath Module After obtaining the kernel module, you can load the FastPath module on each node using insmod kube_ovn_fastpath.ko and verify that the module was loaded successfully using dmesg : # dmesg ... [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in ...","title":"FastPath Kernel Module"},{"location":"en/advance/performance-tuning/#ovs-kernel-module-optimization","text":"OVS flow processing including hashing, matching, etc. consumes about 10% of the CPU resources. Some instruction sets on modern x86 CPUs such as popcnt and sse4.2 can speed up the computation process, but the kernel is not compiled with these options enabled. It has been tested that the CPU consumption of flow-related operations is reduced to about 5% when the corresponding instruction set optimizations are enabled. Similar to the compilation of the FastPath module, it is not possible to provide a single kernel module artifact for all kernels. Users need to compile manually or go to tunning-package to see if a compiled package is available for download. Before using this kernel module, please check if the CPU supports the following instruction set: cat /proc/cpuinfo | grep popcnt cat /proc/cpuinfo | grep sse4_2","title":"OVS Kernel Module Optimization"},{"location":"en/advance/performance-tuning/#compile-and-install-in-centos","text":"Install the relevant compilation dependencies and kernel headers: yum install -y gcc kernel-devel- $( uname -r ) python3 autoconf automake libtool rpm-build openssl-devel Compile the OVS kernel module and generate the corresponding RPM: git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --with-linux = /lib/modules/ $( uname -r ) /build CFLAGS = \"-g -O2 -mpopcnt -msse4.2\" make rpm-fedora-kmod cd rpm/rpmbuild/RPMS/x86_64/ Copy the RPM to each node and install: rpm -i openvswitch-kmod-2.15.2-1.el7.x86_64.rpm If you have previously started Kube-OVN and the older version of the OVS module has been loaded into the kernel. It is recommended to reboot the machine to reload the new version of the kernel module.","title":"Compile and Install in CentOS"},{"location":"en/advance/performance-tuning/#compile-and-install-in-ubuntu","text":"Install the relevant compilation dependencies and kernel headers: apt install -y autoconf automake libtool gcc build-essential libssl-dev Compile the OVS kernel module and install: apt install -y autoconf automake libtool gcc build-essential libssl-dev git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --prefix = /usr/ --localstatedir = /var --enable-ssl --with-linux = /lib/modules/ $( uname -r ) /build make -j ` nproc ` make install make modules_install cat > /etc/depmod.d/openvswitch.conf << EOF override openvswitch * extra override vport-* * extra EOF depmod -a cp debian/openvswitch-switch.init /etc/init.d/openvswitch-switch /etc/init.d/openvswitch-switch force-reload-kmod If you have previously started Kube-OVN and the older version of the OVS module has been loaded into the kernel. It is recommended to reboot the machine to reload the new version of the kernel module.","title":"Compile and Install in Ubuntu"},{"location":"en/advance/performance-tuning/#using-stt-type-tunnel","text":"Common tunnel encapsulation protocols such as Geneve and Vxlan use the UDP protocol to encapsulate packets and are well supported in the kernel. However, when TCP packets are encapsulated using UDP, the optimization and offload features of modern operating systems and network cards for the TCP protocol do not work well, resulting in a significant drop in TCP throughput. In some virtualization scenarios, due to CPU limitations, TCP packet throughput may even be a tenth of that of the host network. STT provides an innovative tunneling protocol that uses TCP formatted header for encapsulation. This encapsulation only emulates the TCP protocol header format without actually establishing a TCP connection, but can take full advantage of the TCP optimization capabilities of modern operating systems and network cards. In our tests TCP packet throughput can be improved several times, reaching performance levels close to those of the host network. The STT tunnel is not pre-installed in the kernel and needs to be installed by compiling the OVS kernel module, which can be found in the previous section. Enable STT tunnel: kubectl set env daemonset/ovs-ovn -n kube-system TUNNEL_TYPE = stt kubectl delete pod -n kube-system -lapp = ovs \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Using STT Type Tunnel"},{"location":"en/advance/security-group/","text":"SecurityGroup Usage \u00b6 Kube-OVN has supported the configuration of security-groups, and the CRD used to configure security-groups is SecurityGroup. SecurityGroup Example \u00b6 apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-example spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 1 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address The specific meaning of each field of the SecurityGroup can be found in the Kube-OVN API Reference . Pods bind security-groups by adding annotations, two annotations are used. ovn.kubernetes.io/port_security : \"true\" ovn.kubernetes.io/security_groups : sg-example Caution \u00b6 Security-groups are finally restricted by setting ACL rules, and as mentioned in the OVN documentation, if two ACL rules match with the same priority, it is uncertain which ACL will actually work. Therefore, when setting up security-group rules, you need to be careful to differentiate the priority. When adding a security-group, it is important to know what restrictions are being added. As a CNI, Kube-OVN will perform a Pod-to-Gateway connectivity test after creating a Pod. Actual test \u00b6 Create a Pod using the following yaml, and specify the security-group in the annotation for the pod. apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-example' name : sg-test-pod namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest The actual test results show as follows: # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h32m <none> kube-ovn-worker <none> <none> test-99fff7f86-52h9r 1 /1 Running 0 5h41m 10 .16.0.14 kube-ovn-control-plane <none> <none> test-99fff7f86-qcgjw 1 /1 Running 0 5h43m 10 .16.0.13 kube-ovn-worker <none> <none> Execute kubectl describe pod to see information about the pod, and you can see the error message: # kubectl describe pod sg-test-pod Name: sg-test-pod Namespace: default Priority: 0 Node: kube-ovn-worker/172.18.0.2 Start Time: Tue, 28 Feb 2023 10 :29:36 +0800 Labels: app = static Annotations: ovn.kubernetes.io/allocated: true ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.15 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:FA:17:97 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/port_security: true ovn.kubernetes.io/routed: true ovn.kubernetes.io/security_groups: sg-allow-reject Status: Pending IP: IPs: <none> - - - - - Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 5m3s ( x70 over 4h59m ) kubelet ( combined from similar events ) : Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"40636e0c7f1ade5500fa958486163d74f2e2300051a71522a9afd7ba0538afb6\" : plugin type = \"kube-ovn\" failed ( add ) : RPC failed ; request ip return 500 configure nic failed 10 .16.0.15 network not ready after 200 ping 10 .16.0.1 Modify the rules for the security group to add access rules to the gateway, refer to the following: apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-gw-both spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 2 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.1/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 2 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : icmp remoteAddress : 10.16.0.1/16 remoteType : address In the inbound and outbound rules respectively, add a rule to allow access to the gateway, and set the rule to have the highest priority. Deploying with the following yaml to bind security group, confirm that the Pod is operational: apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-gw-both' name : sg-gw-both namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest To view Pod information after deployment: # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h41m <none> kube-ovn-worker <none> <none> sg-gw-both 1 /1 Running 0 5h37m 10 .16.0.19 kube-ovn-worker <none> <none> So for the use of security groups, be particularly clear about the effect of the added restriction rules. If it is simply to restrict traffic access, consider using a network policy instead. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"SecurityGroup Usage"},{"location":"en/advance/security-group/#securitygroup-usage","text":"Kube-OVN has supported the configuration of security-groups, and the CRD used to configure security-groups is SecurityGroup.","title":"SecurityGroup Usage"},{"location":"en/advance/security-group/#securitygroup-example","text":"apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-example spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 1 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address The specific meaning of each field of the SecurityGroup can be found in the Kube-OVN API Reference . Pods bind security-groups by adding annotations, two annotations are used. ovn.kubernetes.io/port_security : \"true\" ovn.kubernetes.io/security_groups : sg-example","title":"SecurityGroup Example"},{"location":"en/advance/security-group/#caution","text":"Security-groups are finally restricted by setting ACL rules, and as mentioned in the OVN documentation, if two ACL rules match with the same priority, it is uncertain which ACL will actually work. Therefore, when setting up security-group rules, you need to be careful to differentiate the priority. When adding a security-group, it is important to know what restrictions are being added. As a CNI, Kube-OVN will perform a Pod-to-Gateway connectivity test after creating a Pod.","title":"Caution"},{"location":"en/advance/security-group/#actual-test","text":"Create a Pod using the following yaml, and specify the security-group in the annotation for the pod. apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-example' name : sg-test-pod namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest The actual test results show as follows: # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h32m <none> kube-ovn-worker <none> <none> test-99fff7f86-52h9r 1 /1 Running 0 5h41m 10 .16.0.14 kube-ovn-control-plane <none> <none> test-99fff7f86-qcgjw 1 /1 Running 0 5h43m 10 .16.0.13 kube-ovn-worker <none> <none> Execute kubectl describe pod to see information about the pod, and you can see the error message: # kubectl describe pod sg-test-pod Name: sg-test-pod Namespace: default Priority: 0 Node: kube-ovn-worker/172.18.0.2 Start Time: Tue, 28 Feb 2023 10 :29:36 +0800 Labels: app = static Annotations: ovn.kubernetes.io/allocated: true ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.15 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:FA:17:97 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/port_security: true ovn.kubernetes.io/routed: true ovn.kubernetes.io/security_groups: sg-allow-reject Status: Pending IP: IPs: <none> - - - - - Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 5m3s ( x70 over 4h59m ) kubelet ( combined from similar events ) : Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"40636e0c7f1ade5500fa958486163d74f2e2300051a71522a9afd7ba0538afb6\" : plugin type = \"kube-ovn\" failed ( add ) : RPC failed ; request ip return 500 configure nic failed 10 .16.0.15 network not ready after 200 ping 10 .16.0.1 Modify the rules for the security group to add access rules to the gateway, refer to the following: apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-gw-both spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 2 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.1/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 2 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : icmp remoteAddress : 10.16.0.1/16 remoteType : address In the inbound and outbound rules respectively, add a rule to allow access to the gateway, and set the rule to have the highest priority. Deploying with the following yaml to bind security group, confirm that the Pod is operational: apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-gw-both' name : sg-gw-both namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest To view Pod information after deployment: # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h41m <none> kube-ovn-worker <none> <none> sg-gw-both 1 /1 Running 0 5h37m 10 .16.0.19 kube-ovn-worker <none> <none> So for the use of security groups, be particularly clear about the effect of the added restriction rules. If it is simply to restrict traffic access, consider using a network policy instead. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Actual test"},{"location":"en/advance/vip/","text":"VIP Reservation \u00b6 In some scenarios we want to dynamically reserve part of the IP but not assign it to Pods but to other infrastructure e.g: Kubernetes nested Kubernetes scenarios where the upper Kubernetes uses the Underlay network take up the available addresses of the underlying Subnet. LB or other network infrastructure requires the use of an IP within a Subnet. Create Random Address VIP \u00b6 If you just want to set aside a number of IPs and have no requirement for the IP addresses themselves, you can use the following yaml to create them: apiVersion : kubeovn.io/v1 kind : Vip metadata : name : vip-dynamic-01 spec : subnet : ovn-default type : \"\" subnet : reserve the IP from this Subnet. type : Currently, two types are supported. If the value is empty, it indicates that it is only used for occupying ip addresses of ipam. switch_lb_vip The front-end vip address and back-end ip address of the switch lb must be on the same subnet. Query the VIP after creation. # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY vip-dynamic-01 10 .16.0.12 00 :00:00:F0:DB:25 ovn-default true It can be seen that the VIP is assigned the IP address 10.16.0.12 , which can later be used by other network infrastructures. Create a fixed address VIP \u00b6 The IP address of the reserved VIP can be fixed using the following yaml: apiVersion : kubeovn.io/v1 kind : Vip metadata : name : static-vip01 spec : subnet : ovn-default V4ip : \"10.16.0.121\" subnet : reserve the IP from this Subnet. V4ip : A fixed-assigned IP address that should within the CIDR range of subnet . Query the VIP after creation: # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY static-vip01 10 .16.0.121 00 :00:00:F0:DB:26 ovn-default true It can be seen that the VIP has been assigned the expected IP address. StatefulSet & Kubevirt VM keep VIP \u00b6 Specify for StatefulSet and VM resources, these Pods their owned will reuse the VIP when these Pods recreating. VM keep VIP must be enable the keep-vm-ip param in kube-ovn-controller . Refer Kubevirt VM Fixed Address Settings \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VIP Reservation"},{"location":"en/advance/vip/#vip-reservation","text":"In some scenarios we want to dynamically reserve part of the IP but not assign it to Pods but to other infrastructure e.g: Kubernetes nested Kubernetes scenarios where the upper Kubernetes uses the Underlay network take up the available addresses of the underlying Subnet. LB or other network infrastructure requires the use of an IP within a Subnet.","title":"VIP Reservation"},{"location":"en/advance/vip/#create-random-address-vip","text":"If you just want to set aside a number of IPs and have no requirement for the IP addresses themselves, you can use the following yaml to create them: apiVersion : kubeovn.io/v1 kind : Vip metadata : name : vip-dynamic-01 spec : subnet : ovn-default type : \"\" subnet : reserve the IP from this Subnet. type : Currently, two types are supported. If the value is empty, it indicates that it is only used for occupying ip addresses of ipam. switch_lb_vip The front-end vip address and back-end ip address of the switch lb must be on the same subnet. Query the VIP after creation. # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY vip-dynamic-01 10 .16.0.12 00 :00:00:F0:DB:25 ovn-default true It can be seen that the VIP is assigned the IP address 10.16.0.12 , which can later be used by other network infrastructures.","title":"Create Random Address VIP"},{"location":"en/advance/vip/#create-a-fixed-address-vip","text":"The IP address of the reserved VIP can be fixed using the following yaml: apiVersion : kubeovn.io/v1 kind : Vip metadata : name : static-vip01 spec : subnet : ovn-default V4ip : \"10.16.0.121\" subnet : reserve the IP from this Subnet. V4ip : A fixed-assigned IP address that should within the CIDR range of subnet . Query the VIP after creation: # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY static-vip01 10 .16.0.121 00 :00:00:F0:DB:26 ovn-default true It can be seen that the VIP has been assigned the expected IP address.","title":"Create a fixed address VIP"},{"location":"en/advance/vip/#statefulset-kubevirt-vm-keep-vip","text":"Specify for StatefulSet and VM resources, these Pods their owned will reuse the VIP when these Pods recreating. VM keep VIP must be enable the keep-vm-ip param in kube-ovn-controller . Refer Kubevirt VM Fixed Address Settings \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"StatefulSet &amp; Kubevirt VM keep VIP"},{"location":"en/advance/vpc-dns/","text":"\u81ea\u5b9a\u4e49 VPC DNS \u00b6 \u7531\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u548c \u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0c\u81ea\u5b9a VPC \u5185\u65e0\u6cd5\u8bbf\u95ee\u5230\u90e8\u7f72\u5728\u9ed8\u8ba4 VPC \u5185\u7684 coredns\u3002 \u5982\u679c\u7528\u6237\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 Kubernetes \u63d0\u4f9b\u7684\u96c6\u7fa4\u5185\u57df\u540d\u89e3\u6790\u80fd\u529b\uff0c\u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\uff0c\u5229\u7528 vpc-dns CRD \u6765\u5b9e\u73b0\u3002 \u8be5 CRD \u6700\u7ec8\u4f1a\u90e8\u7f72\u4e00\u4e2a coredns\uff0c\u8be5 Pod \u6709\u4e24\u4e2a\u7f51\u5361\uff0c\u4e00\u4e2a\u7f51\u5361\u5728\u7528\u6237\u81ea\u5b9a\u4e49 VPC\uff0c\u53e6\u4e00\u4e2a\u7f51\u5361\u5728\u9ed8\u8ba4 VPC \u4ece\u800c\u5b9e\u73b0\u7f51\u7edc\u4e92\u901a\uff0c\u540c\u65f6\u901a\u8fc7 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u63d0\u4f9b\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u4e00\u4e2a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u3002 \u90e8\u7f72 vpc-dns \u6240\u4f9d\u8d56\u7684\u8d44\u6e90 \u00b6 apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u9664\u4e86\u4ee5\u4e0a\u8d44\u6e90\uff0c\u8be5\u529f\u80fd\u8fd8\u4f9d\u8d56 nat-gw-pod \u955c\u50cf\u8fdb\u884c\u8def\u7531\u914d\u7f6e\u3002 \u914d\u7f6e\u9644\u52a0\u7f51\u5361 \u00b6 apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }' \u4fee\u6539 ovn-default \u5b50\u7f51\u7684 provider \u00b6 \u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn # \u53ea\u9700\u4fee\u6539\u8be5\u5b57\u6bb5 vpc : ovn-cluster \u914d\u7f6e vpc-dns \u7684 Configmap \u00b6 \u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : \"true\" nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\u662f\u5426\u542f\u7528\u529f\u80fd\uff0c\u9ed8\u8ba4 true \u3002 coredns-image \uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 coredns-template \uff1acoredns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\u83b7\u53d6\u5f53\u524d\u7248\u672c ovn \u76ee\u5f55\u4e0b coredns-template.yaml \u9ed8\u8ba4\u4e3a https://raw.githubusercontent.com/kubeovn/kube-ovn/\u5f53\u524d\u7248\u672c/yamls/coredns-template.yaml \u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u5730\u5740\u3002 k8s-service-port \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u7aef\u53e3\u3002 \u90e8\u7f72 vpc-dns \u00b6 \u914d\u7f6e vpc-dns yaml\uff1a kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u90e8\u7f72\u8d44\u6e90\u7684\u4fe1\u606f\uff1a # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72\u3002 \u9650\u5236\uff1a\u4e00\u4e2a VPC \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a VPC \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a VPC \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002 \u9a8c\u8bc1\u90e8\u7f72\u7ed3\u679c \u00b6 \u67e5\u770b vpc-dns Pod \u72b6\u6001\uff0c\u4f7f\u7528 label app=vpc-dns \uff0c\u53ef\u4ee5\u67e5\u770b\u6240\u6709 vpc-dns pod \u72b6\u6001\uff1a # kubectl -n kube-system get pods -l app=vpc-dns NAME READY STATUS RESTARTS AGE vpc-dns-test-cjh1-7b878d96b4-g5979 1 /1 Running 0 28s vpc-dns-test-cjh1-7b878d96b4-ltmf9 1 /1 Running 0 28s \u67e5\u770b slr \u72b6\u6001\u4fe1\u606f\uff1a # kubectl -n kube-system get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh1 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh1 113s \u8fdb\u5165\u8be5 VPC \u4e0b\u7684 Pod\uff0c\u6d4b\u8bd5 dns \u89e3\u6790: nslookup kubernetes.default.svc.cluster.local 10 .96.0.3 \u8be5 VPC \u4e0b\u7684 switch lb rule \u6240\u5728\u7684\u5b50\u7f51\u4ee5\u53ca\u540c\u4e00 VPC \u4e0b\u7684\u5176\u4ed6\u5b50\u7f51\u4e0b\u7684 pod \u90fd\u53ef\u4ee5\u89e3\u6790\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC DNS"},{"location":"en/advance/vpc-dns/#vpc-dns","text":"\u7531\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u548c \u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0c\u81ea\u5b9a VPC \u5185\u65e0\u6cd5\u8bbf\u95ee\u5230\u90e8\u7f72\u5728\u9ed8\u8ba4 VPC \u5185\u7684 coredns\u3002 \u5982\u679c\u7528\u6237\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 Kubernetes \u63d0\u4f9b\u7684\u96c6\u7fa4\u5185\u57df\u540d\u89e3\u6790\u80fd\u529b\uff0c\u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\uff0c\u5229\u7528 vpc-dns CRD \u6765\u5b9e\u73b0\u3002 \u8be5 CRD \u6700\u7ec8\u4f1a\u90e8\u7f72\u4e00\u4e2a coredns\uff0c\u8be5 Pod \u6709\u4e24\u4e2a\u7f51\u5361\uff0c\u4e00\u4e2a\u7f51\u5361\u5728\u7528\u6237\u81ea\u5b9a\u4e49 VPC\uff0c\u53e6\u4e00\u4e2a\u7f51\u5361\u5728\u9ed8\u8ba4 VPC \u4ece\u800c\u5b9e\u73b0\u7f51\u7edc\u4e92\u901a\uff0c\u540c\u65f6\u901a\u8fc7 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u63d0\u4f9b\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u4e00\u4e2a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u3002","title":"\u81ea\u5b9a\u4e49 VPC DNS"},{"location":"en/advance/vpc-dns/#vpc-dns_1","text":"apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u9664\u4e86\u4ee5\u4e0a\u8d44\u6e90\uff0c\u8be5\u529f\u80fd\u8fd8\u4f9d\u8d56 nat-gw-pod \u955c\u50cf\u8fdb\u884c\u8def\u7531\u914d\u7f6e\u3002","title":"\u90e8\u7f72 vpc-dns \u6240\u4f9d\u8d56\u7684\u8d44\u6e90"},{"location":"en/advance/vpc-dns/#_1","text":"apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }'","title":"\u914d\u7f6e\u9644\u52a0\u7f51\u5361"},{"location":"en/advance/vpc-dns/#ovn-default-provider","text":"\u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn # \u53ea\u9700\u4fee\u6539\u8be5\u5b57\u6bb5 vpc : ovn-cluster","title":"\u4fee\u6539 ovn-default \u5b50\u7f51\u7684 provider"},{"location":"en/advance/vpc-dns/#vpc-dns-configmap","text":"\u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : \"true\" nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\u662f\u5426\u542f\u7528\u529f\u80fd\uff0c\u9ed8\u8ba4 true \u3002 coredns-image \uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 coredns-template \uff1acoredns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\u83b7\u53d6\u5f53\u524d\u7248\u672c ovn \u76ee\u5f55\u4e0b coredns-template.yaml \u9ed8\u8ba4\u4e3a https://raw.githubusercontent.com/kubeovn/kube-ovn/\u5f53\u524d\u7248\u672c/yamls/coredns-template.yaml \u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u5730\u5740\u3002 k8s-service-port \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u7aef\u53e3\u3002","title":"\u914d\u7f6e vpc-dns \u7684 Configmap"},{"location":"en/advance/vpc-dns/#vpc-dns_2","text":"\u914d\u7f6e vpc-dns yaml\uff1a kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u90e8\u7f72\u8d44\u6e90\u7684\u4fe1\u606f\uff1a # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72\u3002 \u9650\u5236\uff1a\u4e00\u4e2a VPC \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a VPC \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a VPC \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002","title":"\u90e8\u7f72 vpc-dns"},{"location":"en/advance/vpc-dns/#_2","text":"\u67e5\u770b vpc-dns Pod \u72b6\u6001\uff0c\u4f7f\u7528 label app=vpc-dns \uff0c\u53ef\u4ee5\u67e5\u770b\u6240\u6709 vpc-dns pod \u72b6\u6001\uff1a # kubectl -n kube-system get pods -l app=vpc-dns NAME READY STATUS RESTARTS AGE vpc-dns-test-cjh1-7b878d96b4-g5979 1 /1 Running 0 28s vpc-dns-test-cjh1-7b878d96b4-ltmf9 1 /1 Running 0 28s \u67e5\u770b slr \u72b6\u6001\u4fe1\u606f\uff1a # kubectl -n kube-system get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh1 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh1 113s \u8fdb\u5165\u8be5 VPC \u4e0b\u7684 Pod\uff0c\u6d4b\u8bd5 dns \u89e3\u6790: nslookup kubernetes.default.svc.cluster.local 10 .96.0.3 \u8be5 VPC \u4e0b\u7684 switch lb rule \u6240\u5728\u7684\u5b50\u7f51\u4ee5\u53ca\u540c\u4e00 VPC \u4e0b\u7684\u5176\u4ed6\u5b50\u7f51\u4e0b\u7684 pod \u90fd\u53ef\u4ee5\u89e3\u6790\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9a8c\u8bc1\u90e8\u7f72\u7ed3\u679c"},{"location":"en/advance/vpc-internal-lb/","text":"Customize VPC Internal Load Balancing \u00b6 The Service provided by Kubernetes can be used for load balancing within the cluster. However, there are several issues with using Service as internal load balancing in customize VPC mode: The Service IP range is a cluster resource, shared by all customize VPCs, and cannot overlap. Users cannot set internal load balancing IP addresses according to their own preferences. To address the above issues, Kube OVN introduced the SwitchLBRule CRD in 1.11, allowing users to set internal load balancing rules within customize VPCs. SwitchLBRule support the following two ways to set internal load balancing rules within a customize VPC. Automatically Generate Load Balancing Rules by Selector \u00b6 Load balancing rules can be generated by selector automatic association with pod configuration through label . example of SwitchLBRule is as follows: apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP usage of selector , sessionAffinity , and port is the same as Kubernetes Service. vip \uff1acustomize load balancing IP address. namespace \uff1anamespace of the pod selected by selector . Kube OVN will determine the VPC of the selected pod based on the SwitchLBRule definition and set the corresponding L2 LB. Manually Defined Load Balancing Rules by Endpoints \u00b6 Load balancing rules can be customized configured by endpoints , to support scenarios where load balancing rules cannot be automatically generated through selector . For example, the load balancing backend is vm created by kubevirt . example of SwitchLBRule is as follows: apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default endpoints : - 192.168.0.101 - 192.168.0.102 - 192.168.0.103 ports : - name : dns port : 8888 targetPort : 80 protocol : TCP usage of sessionAffinity , and port is the same as Kubernetes Service. vip \uff1acustomize load balancing IP address. namespace \uff1anamespace of the pod selected by selector . endpoints \uff1aload balancing backend IP list. attention\uff1a If both selector and endpoints are configured, the selector configuration will be automatically ignored. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC Internal Load Balancer"},{"location":"en/advance/vpc-internal-lb/#customize-vpc-internal-load-balancing","text":"The Service provided by Kubernetes can be used for load balancing within the cluster. However, there are several issues with using Service as internal load balancing in customize VPC mode: The Service IP range is a cluster resource, shared by all customize VPCs, and cannot overlap. Users cannot set internal load balancing IP addresses according to their own preferences. To address the above issues, Kube OVN introduced the SwitchLBRule CRD in 1.11, allowing users to set internal load balancing rules within customize VPCs. SwitchLBRule support the following two ways to set internal load balancing rules within a customize VPC.","title":"Customize VPC Internal Load Balancing"},{"location":"en/advance/vpc-internal-lb/#automatically-generate-load-balancing-rules-by-selector","text":"Load balancing rules can be generated by selector automatic association with pod configuration through label . example of SwitchLBRule is as follows: apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP usage of selector , sessionAffinity , and port is the same as Kubernetes Service. vip \uff1acustomize load balancing IP address. namespace \uff1anamespace of the pod selected by selector . Kube OVN will determine the VPC of the selected pod based on the SwitchLBRule definition and set the corresponding L2 LB.","title":"Automatically Generate Load Balancing Rules by Selector"},{"location":"en/advance/vpc-internal-lb/#manually-defined-load-balancing-rules-by-endpoints","text":"Load balancing rules can be customized configured by endpoints , to support scenarios where load balancing rules cannot be automatically generated through selector . For example, the load balancing backend is vm created by kubevirt . example of SwitchLBRule is as follows: apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default endpoints : - 192.168.0.101 - 192.168.0.102 - 192.168.0.103 ports : - name : dns port : 8888 targetPort : 80 protocol : TCP usage of sessionAffinity , and port is the same as Kubernetes Service. vip \uff1acustomize load balancing IP address. namespace \uff1anamespace of the pod selected by selector . endpoints \uff1aload balancing backend IP list. attention\uff1a If both selector and endpoints are configured, the selector configuration will be automatically ignored. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Manually Defined Load Balancing Rules by Endpoints"},{"location":"en/advance/vpc-peering/","text":"VPC Peering \u00b6 VPC peering provides a mechanism for bridging two VPC networks through logical routes so that workloads within two VPCs can access each other through private addresses as if they were on the same private network, without the need for NAT forwarding through a gateway. Prerequisites \u00b6 This feature is only available for customized VPCs. To avoid route overlap the subnet CIDRs within the two VPCs cannot overlap. Currently, only interconnection of two VPCs is supported. Usage \u00b6 First create two non-interconnected VPCs with one Subnet under each VPC, and the CIDRs of the Subnets do not overlap with each other. kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : vpc-1 cidrBlock : 10.0.0.0/16 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : vpc-2 cidrBlock : 172.31.0.0/16 Add vpcPeerings and the corresponding static routes within each VPC: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : vpcPeerings : - remoteVpc : vpc-2 localConnectIP : 169.254.0.1/30 staticRoutes : - cidr : 172.31.0.0/16 nextHopIP : 169.254.0.2 policy : policyDst --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : vpcPeerings : - remoteVpc : vpc-1 localConnectIP : 169.254.0.2/30 staticRoutes : - cidr : 10.0.0.0/16 nextHopIP : 169.254.0.1 policy : policyDst remoteVpc : The name of another peering VPC. localConnectIP : As the IP address and CIDR of the interconnection endpoint. Note that both IPs should belong to the same CIDR and should not conflict with existing subnets. cidr \uff1aCIDR of the peering Subnet. nextHopIP \uff1aThe localConnectIP on the other end of the peering VPC. Create Pods under the two Subnets apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 name : vpc-1-pod spec : containers : - name : vpc-1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 name : vpc-2-pod spec : containers : - name : vpc-2-pod image : docker.io/library/nginx:alpine Test the network connectivity # kubectl exec -it vpc-1-pod -- ping $(kubectl get pod vpc-2-pod -o jsonpath='{.status.podIP}') PING 172 .31.0.2 ( 172 .31.0.2 ) : 56 data bytes 64 bytes from 172 .31.0.2: seq = 0 ttl = 62 time = 0 .655 ms 64 bytes from 172 .31.0.2: seq = 1 ttl = 62 time = 0 .086 ms 64 bytes from 172 .31.0.2: seq = 2 ttl = 62 time = 0 .098 ms ^C --- 172 .31.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .086/0.279/0.655 ms # kubectl exec -it vpc-2-pod -- ping $(kubectl get pod vpc-1-pod -o jsonpath='{.status.podIP}') PING 10 .0.0.2 ( 10 .0.0.2 ) : 56 data bytes 64 bytes from 10 .0.0.2: seq = 0 ttl = 62 time = 0 .594 ms 64 bytes from 10 .0.0.2: seq = 1 ttl = 62 time = 0 .093 ms 64 bytes from 10 .0.0.2: seq = 2 ttl = 62 time = 0 .088 ms ^C --- 10 .0.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .088/0.258/0.594 ms \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC Peering"},{"location":"en/advance/vpc-peering/#vpc-peering","text":"VPC peering provides a mechanism for bridging two VPC networks through logical routes so that workloads within two VPCs can access each other through private addresses as if they were on the same private network, without the need for NAT forwarding through a gateway.","title":"VPC Peering"},{"location":"en/advance/vpc-peering/#prerequisites","text":"This feature is only available for customized VPCs. To avoid route overlap the subnet CIDRs within the two VPCs cannot overlap. Currently, only interconnection of two VPCs is supported.","title":"Prerequisites"},{"location":"en/advance/vpc-peering/#usage","text":"First create two non-interconnected VPCs with one Subnet under each VPC, and the CIDRs of the Subnets do not overlap with each other. kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : vpc-1 cidrBlock : 10.0.0.0/16 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : vpc-2 cidrBlock : 172.31.0.0/16 Add vpcPeerings and the corresponding static routes within each VPC: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : vpcPeerings : - remoteVpc : vpc-2 localConnectIP : 169.254.0.1/30 staticRoutes : - cidr : 172.31.0.0/16 nextHopIP : 169.254.0.2 policy : policyDst --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : vpcPeerings : - remoteVpc : vpc-1 localConnectIP : 169.254.0.2/30 staticRoutes : - cidr : 10.0.0.0/16 nextHopIP : 169.254.0.1 policy : policyDst remoteVpc : The name of another peering VPC. localConnectIP : As the IP address and CIDR of the interconnection endpoint. Note that both IPs should belong to the same CIDR and should not conflict with existing subnets. cidr \uff1aCIDR of the peering Subnet. nextHopIP \uff1aThe localConnectIP on the other end of the peering VPC. Create Pods under the two Subnets apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 name : vpc-1-pod spec : containers : - name : vpc-1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 name : vpc-2-pod spec : containers : - name : vpc-2-pod image : docker.io/library/nginx:alpine Test the network connectivity # kubectl exec -it vpc-1-pod -- ping $(kubectl get pod vpc-2-pod -o jsonpath='{.status.podIP}') PING 172 .31.0.2 ( 172 .31.0.2 ) : 56 data bytes 64 bytes from 172 .31.0.2: seq = 0 ttl = 62 time = 0 .655 ms 64 bytes from 172 .31.0.2: seq = 1 ttl = 62 time = 0 .086 ms 64 bytes from 172 .31.0.2: seq = 2 ttl = 62 time = 0 .098 ms ^C --- 172 .31.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .086/0.279/0.655 ms # kubectl exec -it vpc-2-pod -- ping $(kubectl get pod vpc-1-pod -o jsonpath='{.status.podIP}') PING 10 .0.0.2 ( 10 .0.0.2 ) : 56 data bytes 64 bytes from 10 .0.0.2: seq = 0 ttl = 62 time = 0 .594 ms 64 bytes from 10 .0.0.2: seq = 1 ttl = 62 time = 0 .093 ms 64 bytes from 10 .0.0.2: seq = 2 ttl = 62 time = 0 .088 ms ^C --- 10 .0.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .088/0.258/0.594 ms \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Usage"},{"location":"en/advance/windows/","text":"Windows Support \u00b6 Kube-OVN supports Kubernetes cluster networks that include Windows system nodes, allowing unified containers network management. Prerequisites \u00b6 Read Adding Windows nodes to add Windows nodes. Windows nodes must have the KB4489899 patch installed for Overlay/VXLAN networks to work properly, and it is recommended to update your system to the latest version. Hyper-V and management tools must be installed on the Windows node. Due to Windows restrictions tunnel encapsulation can only be used in Vxlan mode. SSL, IPv6, dual-stack, QoS features are not supported at this time. Dynamic subnet and dynamic tunnel interface are not supported at this time. You need to create the subnet and select the network interface before installing the Windows node. Multiple ProviderNetwork s are not supported, and the bridge interface configuration cannot be dynamically adjusted. Install OVS on Windows \u00b6 Due to some issues with upstream OVN and OVS support for Windows containers, a modified installation package provided by Kube-OVN is required. Use the following command to enable the TESTSIGNING startup item on the Windows node, which requires a system reboot to take effect. bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS bcdedit /set TESTSIGNING ON bcdedit /set nointegritychecks ON Download Windows package on Windows node and install. Confirm that the service is running properly after installation: PS > Get-Service | findstr ovs Running ovsdb-server Open vSwitch DB Service Running ovs-vswitchd Open vSwitch Service Install Kube-OVN \u00b6 Download the installation script in the Windows node install.ps1 . Add relevant parameters and run: . \\i nstall.ps1 -KubeConfig C: \\k\\a dmin.conf -ApiServer https://192.168.140.180:6443 -ServiceCIDR 10 .96.0.0/12 By default, Kube-OVN uses the NIC where the node IP is located as the tunnel interface. If you need to use another NIC, you need to add the specified annotation to the Node before installation, e.g. ovn.kubernetes.io/tunnel_interface=Ethernet1 . \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Windows Support"},{"location":"en/advance/windows/#windows-support","text":"Kube-OVN supports Kubernetes cluster networks that include Windows system nodes, allowing unified containers network management.","title":"Windows Support"},{"location":"en/advance/windows/#prerequisites","text":"Read Adding Windows nodes to add Windows nodes. Windows nodes must have the KB4489899 patch installed for Overlay/VXLAN networks to work properly, and it is recommended to update your system to the latest version. Hyper-V and management tools must be installed on the Windows node. Due to Windows restrictions tunnel encapsulation can only be used in Vxlan mode. SSL, IPv6, dual-stack, QoS features are not supported at this time. Dynamic subnet and dynamic tunnel interface are not supported at this time. You need to create the subnet and select the network interface before installing the Windows node. Multiple ProviderNetwork s are not supported, and the bridge interface configuration cannot be dynamically adjusted.","title":"Prerequisites"},{"location":"en/advance/windows/#install-ovs-on-windows","text":"Due to some issues with upstream OVN and OVS support for Windows containers, a modified installation package provided by Kube-OVN is required. Use the following command to enable the TESTSIGNING startup item on the Windows node, which requires a system reboot to take effect. bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS bcdedit /set TESTSIGNING ON bcdedit /set nointegritychecks ON Download Windows package on Windows node and install. Confirm that the service is running properly after installation: PS > Get-Service | findstr ovs Running ovsdb-server Open vSwitch DB Service Running ovs-vswitchd Open vSwitch Service","title":"Install OVS on Windows"},{"location":"en/advance/windows/#install-kube-ovn","text":"Download the installation script in the Windows node install.ps1 . Add relevant parameters and run: . \\i nstall.ps1 -KubeConfig C: \\k\\a dmin.conf -ApiServer https://192.168.140.180:6443 -ServiceCIDR 10 .96.0.0/12 By default, Kube-OVN uses the NIC where the node IP is located as the tunnel interface. If you need to use another NIC, you need to add the specified annotation to the Node before installation, e.g. ovn.kubernetes.io/tunnel_interface=Ethernet1 . \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Install Kube-OVN"},{"location":"en/advance/with-bgp/","text":"BGP Support \u00b6 Kube-OVN supports broadcasting the IP address of the Pod or Subnet to the outside world via BGP protocol, so that the outside world can access the Pod directly through the Pod IP. To use this feature, you need to install kube-ovn-speaker on specific nodes and add the corresponding annotation to the Pod or Subnet that needs to be exposed to the outside world. Install kube-ovn-speaker \u00b6 kube-ovn-speaker use GoBGP to publish routing information to the outside world and set the next-hop route to itself. Since the node where kube-ovn-speaker is deployed needs to carry return traffic, specific labeled nodes need to be selected for deployment: kubectl label nodes speaker-node-1 ovn.kubernetes.io/bgp = true kubectl label nodes speaker-node-2 ovn.kubernetes.io/bgp = true When there are multiple instances of kube-ovn-speaker, each of them will publish routes to the outside world, the upstream router needs to support multi-path ECMP. Download the corresponding yaml: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/speaker.yaml Modify the corresponding configuration in yaml: --neighbor-address=10.32.32.1 --neighbor-as=65030 --cluster-as=65000 neighbor-address : The address of the BGP Peer, usually the router gateway address. neighbor-as : The AS number of the BGP Peer. cluster-as : The AS number of the container network. Deploy yaml: kubectl apply -f speaker.yaml Publish Pod/Subnet Routes \u00b6 To use BGP for external routing, first set natOutgoing to false for the corresponding Subnet to allow the Pod IP to enter the underlying network directly. Add annotation to publish routes: kubectl annotate pod sample ovn.kubernetes.io/bgp = true kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp = true Delete annotation to disable the publishing: kubectl annotate pod perf-ovn-xzvd4 ovn.kubernetes.io/bgp- kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp- BGP Advance Options \u00b6 kube-ovn-speaker supports more BGP parameters for advanced configuration, which can be adjusted by users according to their network environment: announce-cluster-ip : Whether to publish Service routes to the public, default is false . auth-password : The access password for the BGP peer. holdtime : The heartbeat detection time between BGP neighbors. Neighbors with no messages after the change time will be removed, the default is 90 seconds. graceful-restart : Whether to enable BGP Graceful Restart. graceful-restart-time : BGP Graceful restart time refer to RFC4724 3. graceful-restart-deferral-time : BGP Graceful restart deferral time refer to RFC4724 4.1. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"BGP Support"},{"location":"en/advance/with-bgp/#bgp-support","text":"Kube-OVN supports broadcasting the IP address of the Pod or Subnet to the outside world via BGP protocol, so that the outside world can access the Pod directly through the Pod IP. To use this feature, you need to install kube-ovn-speaker on specific nodes and add the corresponding annotation to the Pod or Subnet that needs to be exposed to the outside world.","title":"BGP Support"},{"location":"en/advance/with-bgp/#install-kube-ovn-speaker","text":"kube-ovn-speaker use GoBGP to publish routing information to the outside world and set the next-hop route to itself. Since the node where kube-ovn-speaker is deployed needs to carry return traffic, specific labeled nodes need to be selected for deployment: kubectl label nodes speaker-node-1 ovn.kubernetes.io/bgp = true kubectl label nodes speaker-node-2 ovn.kubernetes.io/bgp = true When there are multiple instances of kube-ovn-speaker, each of them will publish routes to the outside world, the upstream router needs to support multi-path ECMP. Download the corresponding yaml: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/speaker.yaml Modify the corresponding configuration in yaml: --neighbor-address=10.32.32.1 --neighbor-as=65030 --cluster-as=65000 neighbor-address : The address of the BGP Peer, usually the router gateway address. neighbor-as : The AS number of the BGP Peer. cluster-as : The AS number of the container network. Deploy yaml: kubectl apply -f speaker.yaml","title":"Install kube-ovn-speaker"},{"location":"en/advance/with-bgp/#publish-podsubnet-routes","text":"To use BGP for external routing, first set natOutgoing to false for the corresponding Subnet to allow the Pod IP to enter the underlying network directly. Add annotation to publish routes: kubectl annotate pod sample ovn.kubernetes.io/bgp = true kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp = true Delete annotation to disable the publishing: kubectl annotate pod perf-ovn-xzvd4 ovn.kubernetes.io/bgp- kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp-","title":"Publish Pod/Subnet Routes"},{"location":"en/advance/with-bgp/#bgp-advance-options","text":"kube-ovn-speaker supports more BGP parameters for advanced configuration, which can be adjusted by users according to their network environment: announce-cluster-ip : Whether to publish Service routes to the public, default is false . auth-password : The access password for the BGP peer. holdtime : The heartbeat detection time between BGP neighbors. Neighbors with no messages after the change time will be removed, the default is 90 seconds. graceful-restart : Whether to enable BGP Graceful Restart. graceful-restart-time : BGP Graceful restart time refer to RFC4724 3. graceful-restart-deferral-time : BGP Graceful restart deferral time refer to RFC4724 4.1. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"BGP Advance Options"},{"location":"en/advance/with-cilium/","text":"Integration with Cilium \u00b6 Cilium is an eBPF-based networking and security component. Kube-OVN uses the CNI Chaining mode to enhance existing features. Users can use both the rich network abstraction capabilities of Kube-OVN and the monitoring and security capabilities that come with eBPF. By integrating Cilium, Kube-OVN users can have the following gains: Richer and more efficient security policies. Hubble-based monitoring and UI. Prerequisites \u00b6 Linux kernel version above 4.19 or other compatible kernel for full eBPF capability support. Install Helm in advance to prepare for the installation of Cilium, please refer to Installing Helm to deploy Helm. Configure Kube-OVN \u00b6 In order to fully utilize the security capabilities of Cilium, you need to disable the networkpolicy feature within Kube-OVN and adjust the CNI configuration priority. Change the following variables in the install.sh script: ENABLE_NP = false CNI_CONFIG_PRIORITY = 10 If the deployment is complete, you can adjust the args of kube-ovn-controller : args : - --enable-np=false Modify the kube-ovn-cni args to adjust the CNI configuration priority: args : - --cni-conf-name=10-kube-ovn.conflist Adjust the Kube-OVN cni configuration name on each node: mv /etc/cni/net.d/01-kube-ovn.conflist /etc/cni/net.d/10-kube-ovn.conflist Deploy Cilium \u00b6 Create the chaining.yaml configuration file to use Cilium's generic-veth mode: apiVersion : v1 kind : ConfigMap metadata : name : cni-configuration namespace : kube-system data : cni-config : |- { \"name\": \"generic-veth\", \"cniVersion\": \"0.3.1\", \"plugins\": [ { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\" } }, { \"type\": \"portmap\", \"snat\": true, \"capabilities\": {\"portMappings\": true} }, { \"type\": \"cilium-cni\" } ] } Installation the chaining config: kubectl apply -f chaining.yaml Deploying Cilium with Helm: helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --set cni.chainingMode = generic-veth \\ --set cni.customConf = true \\ --set cni.configMap = cni-configuration \\ --set tunnel = disabled \\ --set enableIPv4Masquerade = false \\ --set enableIdentityMark = false Confirm that the Cilium installation was successful: # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: disabled \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 Containers: cilium Running: 2 cilium-operator Running: 2 Cluster Pods: 8 /11 managed by Cilium Image versions cilium quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb: 2 cilium-operator quay.io/cilium/operator-generic:v1.10.5@sha256:2d2f730f219d489ff0702923bf24c0002cd93eb4b47ba344375566202f56d972: 2 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Integration with Cilium"},{"location":"en/advance/with-cilium/#integration-with-cilium","text":"Cilium is an eBPF-based networking and security component. Kube-OVN uses the CNI Chaining mode to enhance existing features. Users can use both the rich network abstraction capabilities of Kube-OVN and the monitoring and security capabilities that come with eBPF. By integrating Cilium, Kube-OVN users can have the following gains: Richer and more efficient security policies. Hubble-based monitoring and UI.","title":"Integration with Cilium"},{"location":"en/advance/with-cilium/#prerequisites","text":"Linux kernel version above 4.19 or other compatible kernel for full eBPF capability support. Install Helm in advance to prepare for the installation of Cilium, please refer to Installing Helm to deploy Helm.","title":"Prerequisites"},{"location":"en/advance/with-cilium/#configure-kube-ovn","text":"In order to fully utilize the security capabilities of Cilium, you need to disable the networkpolicy feature within Kube-OVN and adjust the CNI configuration priority. Change the following variables in the install.sh script: ENABLE_NP = false CNI_CONFIG_PRIORITY = 10 If the deployment is complete, you can adjust the args of kube-ovn-controller : args : - --enable-np=false Modify the kube-ovn-cni args to adjust the CNI configuration priority: args : - --cni-conf-name=10-kube-ovn.conflist Adjust the Kube-OVN cni configuration name on each node: mv /etc/cni/net.d/01-kube-ovn.conflist /etc/cni/net.d/10-kube-ovn.conflist","title":"Configure Kube-OVN"},{"location":"en/advance/with-cilium/#deploy-cilium","text":"Create the chaining.yaml configuration file to use Cilium's generic-veth mode: apiVersion : v1 kind : ConfigMap metadata : name : cni-configuration namespace : kube-system data : cni-config : |- { \"name\": \"generic-veth\", \"cniVersion\": \"0.3.1\", \"plugins\": [ { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\" } }, { \"type\": \"portmap\", \"snat\": true, \"capabilities\": {\"portMappings\": true} }, { \"type\": \"cilium-cni\" } ] } Installation the chaining config: kubectl apply -f chaining.yaml Deploying Cilium with Helm: helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --set cni.chainingMode = generic-veth \\ --set cni.customConf = true \\ --set cni.configMap = cni-configuration \\ --set tunnel = disabled \\ --set enableIPv4Masquerade = false \\ --set enableIdentityMark = false Confirm that the Cilium installation was successful: # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: disabled \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 Containers: cilium Running: 2 cilium-operator Running: 2 Cluster Pods: 8 /11 managed by Cilium Image versions cilium quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb: 2 cilium-operator quay.io/cilium/operator-generic:v1.10.5@sha256:2d2f730f219d489ff0702923bf24c0002cd93eb4b47ba344375566202f56d972: 2 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Deploy Cilium"},{"location":"en/advance/with-openstack/","text":"Integration with OpenStack \u00b6 In some cases, users need to run virtual machines with OpenStack and containers with Kubernetes, and need the network to interoperate between containers and virtual machines and be under a unified control plane. If the OpenStack Neutron side also uses OVN as the underlying network, then Kube-OVN can use either cluster interconnection or shared underlying OVN to connect the OpenStack and Kubernetes networks. Cluster Interconnection \u00b6 This pattern is similar to Cluster Inter-Connection with OVN-IC to connect two Kubernetes cluster networks, except that the two ends of the cluster are replaced with OpenStack and Kubernetes\u3002 Prerequisites \u00b6 The subnet CIDRs within OpenStack and Kubernetes cannot overlap with each other in auto-route mode. A set of machines needs to exist that can be accessed by each cluster over the network and used to deploy controllers that interconnect across clusters. Each cluster needs to have a set of machines that can access each other across clusters via IP as the gateway nodes. This solution only connects to the Kubernetes default subnet with selected VPC in OpenStack. Deploy OVN-IC DB \u00b6 Start the OVN-IC DB with the following command: docker run --name = ovn-ic-db -d --network = host -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh Kubernetes Side Operations \u00b6 Create ovn-ic-config ConfigMap in kube-system Namespace \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : Whether to enable cluster interconnection. az-name : Distinguish the cluster names of different clusters, each interconnected cluster needs to be different. ic-db-host : Address of the node where the OVN-IC DB is deployed. ic-nb-port : OVN-IC Northbound Database port, default 6645. ic-sb-port : OVN-IC Southbound Database port, default 6645. gw-nodes : The name of the nodes in the cluster interconnection that takes on the work of the gateways, separated by commas. auto-route : Whether to automatically publish and learn routes. OpenStack Side Operations \u00b6 Create logical routers that interconnect with Kubernetes: # openstack router create router0 # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | d5b38655-249a-4192-8046-71aa4d2b4af1 | router0 | ACTIVE | UP | 98a29ab7388347e7b5ff8bdd181ba4f9 | +--------------------------------------+---------+--------+-------+----------------------------------+ Set the availability zone name in the OVN northbound database within OpenStack, which needs to be different from the other interconnected clusters: ovn-nbctl set NB_Global . name = op-az Start the OVN-IC controller at a node that has access to the OVN-IC DB: /usr/share/ovn/scripts/ovn-ctl --ovn-ic-nb-db = tcp:192.168.65.3:6645 \\ --ovn-ic-sb-db = tcp:192.168.65.3:6646 \\ --ovn-northd-nb-db = unix:/run/ovn/ovnnb_db.sock \\ --ovn-northd-sb-db = unix:/run/ovn/ovnsb_db.sock \\ start_ic ovn-ic-nb-db \uff0c ovn-ic-sb-db : OVN-IC Northbound database and southbound database addresses. ovn-northd-nb-db \uff0c ovn-northd-sb-db : Current cluster OVN northbound database and southbound data address. Configuration gateway nodes: ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn = true The next step is to create a logical topology by operating the OVN in OpenStack. Connect the ts interconnect switch and the router0 logical router, and set the relevant rules: ovn-nbctl lrp-add router0 lrp-router0-ts 00 :02:ef:11:39:4f 169 .254.100.73/24 ovn-nbctl lsp-add ts lsp-ts-router0 -- lsp-set-addresses lsp-ts-router0 router \\ -- lsp-set-type lsp-ts-router0 router \\ -- lsp-set-options lsp-ts-router0 router-port = lrp-router0-ts ovn-nbctl lrp-set-gateway-chassis lrp-router0-ts { gateway chassis } 1000 ovn-nbctl set NB_Global . options:ic-route-adv = true options:ic-route-learn = true Verify that OpenStack has learned the Kubernetes routing rules: # ovn-nbctl lr-route-list router0 IPv4 Routes 10 .0.0.22 169 .254.100.34 dst-ip ( learned ) 10 .16.0.0/16 169 .254.100.34 dst-ip ( learned ) Next, you can create a virtual machine under the router0 network to verify that it can interconnect with Pods under Kubernetes. Shared Underlay OVN \u00b6 In this scenario, OpenStack and Kubernetes share the same OVN, so concepts such as VPC and Subnet can be pulled together for better control and interconnection. In this mode we deploy the OVN normally using Kube-OVN, and OpenStack modifies the Neutron configuration to connect to the same OVN DB. OpenStack requires networking-ovn as a Neutron backend implementation. Neutron Modification \u00b6 Modify the Neutron configuration file /etc/neutron/plugins/ml2/ml2_conf.ini \uff1a [ ovn ] ... ovn_nb_connection = tcp: [ 192 .168.137.176 ] :6641,tcp: [ 192 .168.137.177 ] :6641,tcp: [ 192 .168.137.178 ] :6641 ovn_sb_connection = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovn_l3_scheduler = OVN_L3_SCHEDULER ovn_nb_connection \uff0c ovn_sb_connection : The address needs to be changed to the address of the ovn-central nodes deployed by Kube-OVN. Modify the OVS configuration for each node: ovs-vsctl set open . external-ids:ovn-remote = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovs-vsctl set open . external-ids:ovn-encap-type = geneve ovs-vsctl set open . external-ids:ovn-encap-ip = 192 .168.137.200 external-ids:ovn-remote : The address needs to be changed to the address of the ovn-central nodes deployed by Kube-OVN. ovn-encap-ip : Change to the IP address of the current node. Using OpenStack Internal Resources in Kubernetes \u00b6 The next section describes how to query OpenStack's network resources in Kubernetes and create Pods in the subnet from OpenStack. Query the existing network resources in OpenStack for the following resources that have been pre-created. # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | 22040ed5-0598-4f77-bffd-e7fd4db47e93 | router0 | ACTIVE | UP | 62381a21d569404aa236a5dd8712449c | +--------------------------------------+---------+--------+-------+----------------------------------+ # openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | cd59e36a-37db-4c27-b709-d35379a7920f | provider | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | +--------------------------------------+----------+--------------------------------------+ # openstack subnet list +--------------------------------------+-------------+--------------------------------------+----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------------+--------------------------------------+----------------+ | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | provider-v4 | cd59e36a-37db-4c27-b709-d35379a7920f | 192 .168.1.0/24 | +--------------------------------------+-------------+--------------------------------------+----------------+ # openstack server list +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | 8433d622-a8d6-41a7-8b31-49abfd64f639 | provider-instance | ACTIVE | provider = 192 .168.1.61 | ubuntu | m1 | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ On the Kubernetes side, query the VPC resources from OpenStack: # kubectl get vpc NAME STANDBY SUBNETS neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 true [ \"neutron-cd59e36a-37db-4c27-b709-d35379a7920f\" ] ovn-cluster true [ \"join\" , \"ovn-default\" ] neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 is the VPC resources synchronized from OpenStack. Next, you can create Pods and run them according to Kube-OVN's native VPC and Subnet operations. Bind VPC, Subnet to Namespace net2 and create Pod: apiVersion : v1 kind : Namespace metadata : name : net2 --- apiVersion : kubeovn.io/v1 kind : Vpc metadata : creationTimestamp : \"2021-06-20T13:34:11Z\" generation : 2 labels : ovn.kubernetes.io/vpc_external : \"true\" name : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 resourceVersion : \"583728\" uid : 18d4c654-f511-4def-a3a0-a6434d237c1e spec : namespaces : - net2 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 namespaces : - net2 cidrBlock : 12.0.1.0/24 natOutgoing : false --- apiVersion : v1 kind : Pod metadata : name : ubuntu namespace : net2 spec : containers : - image : docker.io/kubeovn/kube-ovn:v1.8.0 command : - \"sleep\" - \"604800\" imagePullPolicy : IfNotPresent name : ubuntu restartPolicy : Always \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Integration with OpenStack"},{"location":"en/advance/with-openstack/#integration-with-openstack","text":"In some cases, users need to run virtual machines with OpenStack and containers with Kubernetes, and need the network to interoperate between containers and virtual machines and be under a unified control plane. If the OpenStack Neutron side also uses OVN as the underlying network, then Kube-OVN can use either cluster interconnection or shared underlying OVN to connect the OpenStack and Kubernetes networks.","title":"Integration with OpenStack"},{"location":"en/advance/with-openstack/#cluster-interconnection","text":"This pattern is similar to Cluster Inter-Connection with OVN-IC to connect two Kubernetes cluster networks, except that the two ends of the cluster are replaced with OpenStack and Kubernetes\u3002","title":"Cluster Interconnection"},{"location":"en/advance/with-openstack/#prerequisites","text":"The subnet CIDRs within OpenStack and Kubernetes cannot overlap with each other in auto-route mode. A set of machines needs to exist that can be accessed by each cluster over the network and used to deploy controllers that interconnect across clusters. Each cluster needs to have a set of machines that can access each other across clusters via IP as the gateway nodes. This solution only connects to the Kubernetes default subnet with selected VPC in OpenStack.","title":"Prerequisites"},{"location":"en/advance/with-openstack/#deploy-ovn-ic-db","text":"Start the OVN-IC DB with the following command: docker run --name = ovn-ic-db -d --network = host -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh","title":"Deploy OVN-IC DB"},{"location":"en/advance/with-openstack/#kubernetes-side-operations","text":"Create ovn-ic-config ConfigMap in kube-system Namespace \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : Whether to enable cluster interconnection. az-name : Distinguish the cluster names of different clusters, each interconnected cluster needs to be different. ic-db-host : Address of the node where the OVN-IC DB is deployed. ic-nb-port : OVN-IC Northbound Database port, default 6645. ic-sb-port : OVN-IC Southbound Database port, default 6645. gw-nodes : The name of the nodes in the cluster interconnection that takes on the work of the gateways, separated by commas. auto-route : Whether to automatically publish and learn routes.","title":"Kubernetes Side Operations"},{"location":"en/advance/with-openstack/#openstack-side-operations","text":"Create logical routers that interconnect with Kubernetes: # openstack router create router0 # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | d5b38655-249a-4192-8046-71aa4d2b4af1 | router0 | ACTIVE | UP | 98a29ab7388347e7b5ff8bdd181ba4f9 | +--------------------------------------+---------+--------+-------+----------------------------------+ Set the availability zone name in the OVN northbound database within OpenStack, which needs to be different from the other interconnected clusters: ovn-nbctl set NB_Global . name = op-az Start the OVN-IC controller at a node that has access to the OVN-IC DB: /usr/share/ovn/scripts/ovn-ctl --ovn-ic-nb-db = tcp:192.168.65.3:6645 \\ --ovn-ic-sb-db = tcp:192.168.65.3:6646 \\ --ovn-northd-nb-db = unix:/run/ovn/ovnnb_db.sock \\ --ovn-northd-sb-db = unix:/run/ovn/ovnsb_db.sock \\ start_ic ovn-ic-nb-db \uff0c ovn-ic-sb-db : OVN-IC Northbound database and southbound database addresses. ovn-northd-nb-db \uff0c ovn-northd-sb-db : Current cluster OVN northbound database and southbound data address. Configuration gateway nodes: ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn = true The next step is to create a logical topology by operating the OVN in OpenStack. Connect the ts interconnect switch and the router0 logical router, and set the relevant rules: ovn-nbctl lrp-add router0 lrp-router0-ts 00 :02:ef:11:39:4f 169 .254.100.73/24 ovn-nbctl lsp-add ts lsp-ts-router0 -- lsp-set-addresses lsp-ts-router0 router \\ -- lsp-set-type lsp-ts-router0 router \\ -- lsp-set-options lsp-ts-router0 router-port = lrp-router0-ts ovn-nbctl lrp-set-gateway-chassis lrp-router0-ts { gateway chassis } 1000 ovn-nbctl set NB_Global . options:ic-route-adv = true options:ic-route-learn = true Verify that OpenStack has learned the Kubernetes routing rules: # ovn-nbctl lr-route-list router0 IPv4 Routes 10 .0.0.22 169 .254.100.34 dst-ip ( learned ) 10 .16.0.0/16 169 .254.100.34 dst-ip ( learned ) Next, you can create a virtual machine under the router0 network to verify that it can interconnect with Pods under Kubernetes.","title":"OpenStack Side Operations"},{"location":"en/advance/with-openstack/#shared-underlay-ovn","text":"In this scenario, OpenStack and Kubernetes share the same OVN, so concepts such as VPC and Subnet can be pulled together for better control and interconnection. In this mode we deploy the OVN normally using Kube-OVN, and OpenStack modifies the Neutron configuration to connect to the same OVN DB. OpenStack requires networking-ovn as a Neutron backend implementation.","title":"Shared Underlay OVN"},{"location":"en/advance/with-openstack/#neutron-modification","text":"Modify the Neutron configuration file /etc/neutron/plugins/ml2/ml2_conf.ini \uff1a [ ovn ] ... ovn_nb_connection = tcp: [ 192 .168.137.176 ] :6641,tcp: [ 192 .168.137.177 ] :6641,tcp: [ 192 .168.137.178 ] :6641 ovn_sb_connection = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovn_l3_scheduler = OVN_L3_SCHEDULER ovn_nb_connection \uff0c ovn_sb_connection : The address needs to be changed to the address of the ovn-central nodes deployed by Kube-OVN. Modify the OVS configuration for each node: ovs-vsctl set open . external-ids:ovn-remote = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovs-vsctl set open . external-ids:ovn-encap-type = geneve ovs-vsctl set open . external-ids:ovn-encap-ip = 192 .168.137.200 external-ids:ovn-remote : The address needs to be changed to the address of the ovn-central nodes deployed by Kube-OVN. ovn-encap-ip : Change to the IP address of the current node.","title":"Neutron Modification"},{"location":"en/advance/with-openstack/#using-openstack-internal-resources-in-kubernetes","text":"The next section describes how to query OpenStack's network resources in Kubernetes and create Pods in the subnet from OpenStack. Query the existing network resources in OpenStack for the following resources that have been pre-created. # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | 22040ed5-0598-4f77-bffd-e7fd4db47e93 | router0 | ACTIVE | UP | 62381a21d569404aa236a5dd8712449c | +--------------------------------------+---------+--------+-------+----------------------------------+ # openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | cd59e36a-37db-4c27-b709-d35379a7920f | provider | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | +--------------------------------------+----------+--------------------------------------+ # openstack subnet list +--------------------------------------+-------------+--------------------------------------+----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------------+--------------------------------------+----------------+ | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | provider-v4 | cd59e36a-37db-4c27-b709-d35379a7920f | 192 .168.1.0/24 | +--------------------------------------+-------------+--------------------------------------+----------------+ # openstack server list +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | 8433d622-a8d6-41a7-8b31-49abfd64f639 | provider-instance | ACTIVE | provider = 192 .168.1.61 | ubuntu | m1 | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ On the Kubernetes side, query the VPC resources from OpenStack: # kubectl get vpc NAME STANDBY SUBNETS neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 true [ \"neutron-cd59e36a-37db-4c27-b709-d35379a7920f\" ] ovn-cluster true [ \"join\" , \"ovn-default\" ] neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 is the VPC resources synchronized from OpenStack. Next, you can create Pods and run them according to Kube-OVN's native VPC and Subnet operations. Bind VPC, Subnet to Namespace net2 and create Pod: apiVersion : v1 kind : Namespace metadata : name : net2 --- apiVersion : kubeovn.io/v1 kind : Vpc metadata : creationTimestamp : \"2021-06-20T13:34:11Z\" generation : 2 labels : ovn.kubernetes.io/vpc_external : \"true\" name : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 resourceVersion : \"583728\" uid : 18d4c654-f511-4def-a3a0-a6434d237c1e spec : namespaces : - net2 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 namespaces : - net2 cidrBlock : 12.0.1.0/24 natOutgoing : false --- apiVersion : v1 kind : Pod metadata : name : ubuntu namespace : net2 spec : containers : - image : docker.io/kubeovn/kube-ovn:v1.8.0 command : - \"sleep\" - \"604800\" imagePullPolicy : IfNotPresent name : ubuntu restartPolicy : Always \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Using OpenStack Internal Resources in Kubernetes"},{"location":"en/advance/with-ovn-ic/","text":"Cluster Inter-Connection with OVN-IC \u00b6 Kube-OVN supports interconnecting two Kubernetes cluster Pod networks via OVN-IC , and the Pods in the two clusters can communicate directly via Pod IPs . Kube-OVN uses tunnels to encapsulate cross-cluster traffic, allowing container networks to interconnect between two clusters as long as there is a set of IP reachable machines. This mode of multi-cluster interconnection is for Overlay network. For Underlay network, it needs the underlying infrastructure to do the inter-connection work. Prerequisites \u00b6 The subnet CIDRs within OpenStack and Kubernetes cannot overlap with each other in auto-interconnect mode. If there is overlap, you need to refer to the subsequent manual interconnection process, which can only connect non-overlapping Subnets. A set of machines needs to exist that can be accessed by each cluster over the network and used to deploy controllers that interconnect across clusters. Each cluster needs to have a set of machines that can access each other across clusters via IP as the gateway nodes. This solution only connects to the Kubernetes default VPCs. Deploy a single-node OVN-IC DB \u00b6 Deploy the OVN-IC DB on a machine accessible by kube-ovn-controller , This DB will hold the network configuration information synchronized up from each cluster. An environment deploying docker can start the OVN-IC DB with the following command. docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh For deploying a containerd environment instead of docker you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh Automatic Routing Mode \u00b6 In auto-routing mode, each cluster synchronizes the CIDR information of the Subnet under its own default VPC to OVN-IC , so make sure there is no overlap between the Subnet CIDRs of the two clusters. Create ovn-ic-config ConfigMap in kube-system Namespace: apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : Whether to enable cluster interconnection. az-name : Distinguish the cluster names of different clusters, each interconnected cluster needs to be different. ic-db-host : Address of the node where the OVN-IC DB is deployed. ic-nb-port : OVN-IC Northbound Database port, default 6645. ic-sb-port : OVN-IC Southbound Database port, default 6645. gw-nodes : The name of the nodes in the cluster interconnection that takes on the work of the gateways, separated by commas. auto-route : Whether to automatically publish and learn routes. Note: To ensure the correct operation, the ConfigMap ovn-ic-config is not allowed to be modified. If any parameter needs to be changed, please delete this ConfigMap, modify it and then apply it again. Check if the interconnected logical switch ts has been established in the ovn-ic container with the following command\uff1a # ovn-ic-sbctl show availability-zone az1 gateway deee03e0-af16-4f45-91e9-b50c3960f809 hostname: az1-gw type: geneve ip: 192 .168.42.145 port ts-az1 transit switch: ts address: [ \"00:00:00:50:AC:8C 169.254.100.45/24\" ] availability-zone az2 gateway e94cc831-8143-40e3-a478-90352773327b hostname: az2-gw type: geneve ip: 192 .168.42.149 port ts-az2 transit switch: ts address: [ \"00:00:00:07:4A:59 169.254.100.63/24\" ] At each cluster observe if logical routes have learned peer routes: # kubectl ko nbctl lr-route-list ovn-cluster IPv4 Routes 10 .42.1.1 169 .254.100.45 dst-ip ( learned ) 10 .42.1.3 100 .64.0.2 dst-ip 10 .16.0.2 100 .64.0.2 src-ip 10 .16.0.3 100 .64.0.2 src-ip 10 .16.0.4 100 .64.0.2 src-ip 10 .16.0.6 100 .64.0.2 src-ip 10 .17.0.0/16 169 .254.100.45 dst-ip ( learned ) 100 .65.0.0/16 169 .254.100.45 dst-ip ( learned ) Next, you can try ping a Pod IP in Cluster 1 directly from a Pod in Cluster 2 to see if you can work. For a subnet that does not want to automatically publish routes to the other end, you can disable route broadcasting by modifying disableInterConnection in the Subnet spec. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : no-advertise spec : cidrBlock : 10.199.0.0/16 disableInterConnection : true Manual Routing Mode \u00b6 For cases where there are overlapping CIDRs between clusters, and you only want to do partial subnet interconnection, you can manually publish subnet routing by following the steps below. Create ovn-ic-config ConfigMap in kube-system Namespace, and set auto-route to false : apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"false\" Find the address of the remote logical ports in each cluster separately, for later manual configuration of the route: [ root@az1 ~ ] # kubectl ko nbctl show switch a391d3a1-14a0-4841-9836-4bd930c447fb ( ts ) port ts-az1 type: router router-port: az1-ts port ts-az2 type: remote addresses: [ \"00:00:00:4B:E2:9F 169.254.100.31/24\" ] [ root@az2 ~ ] # kubectl ko nbctl show switch da6138b8-de81-4908-abf9-b2224ec4edf3 ( ts ) port ts-az2 type: router router-port: az2-ts port ts-az1 type: remote addresses: [ \"00:00:00:FB:2A:F7 169.254.100.79/24\" ] The output above shows that the remote address from cluster az1 to cluster az2 is 169.254.100.31 and the remote address from az2 to az1 is 169.254.100.79 . In this example, the subnet CIDR within cluster az1 is 10.16.0.0/24 and the subnet CIDR within cluster az2 is 10.17.0.0/24 . Set up a route from cluster az1 to cluster az2 in cluster az1 : kubectl ko nbctl lr-route-add ovn-cluster 10 .17.0.0/24 169 .254.100.31 Set up a route to cluster az1 in cluster az2 : kubectl ko nbctl lr-route-add ovn-cluster 10 .16.0.0/24 169 .254.100.79 Highly Available OVN-IC DB Installation \u00b6 A highly available cluster can be formed between OVN-IC DB via the Raft protocol, which requires a minimum of 3 nodes for this deployment model. First start the leader of the OVN-IC DB on the first node. Users deploying a docker environment can use the following command: docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.3\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh If you are using containerd you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a The IP address of the node where the current container is located. NODE_IPS \uff1a The IP addresses of the three nodes running the OVN-IC database, separated by commas. Next, deploy the follower of the OVN-IC DB on the other two nodes. docker environment can use the following command. docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.2\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" -e LEADER_IP = \"192.168.65.3\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh If using containerd you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.2 \"\" --env = \"LEADER_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a The IP address of the node where the current container is located. NODE_IPS \uff1a The IP addresses of the three nodes running the OVN-IC database, separated by commas. LEADER_IP : The IP address of the OVN-IC DB leader node. Specify multiple OVN-IC database node addresses when creating ovn-ic-config for each cluster: apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3,192.168.65.2,192.168.65.1\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" Manual Reset \u00b6 In some cases, the entire interconnection configuration needs to be cleaned up due to configuration errors, you can refer to the following steps to clean up your environment. Delete the current ovn-ic-config Configmap: kubectl -n kube-system delete cm ovn-ic-config Delete ts logical switch: kubectl-ko nbctl ls-del ts Repeat the same steps at the peer cluster. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cluster Inter-Connection with OVN-IC"},{"location":"en/advance/with-ovn-ic/#cluster-inter-connection-with-ovn-ic","text":"Kube-OVN supports interconnecting two Kubernetes cluster Pod networks via OVN-IC , and the Pods in the two clusters can communicate directly via Pod IPs . Kube-OVN uses tunnels to encapsulate cross-cluster traffic, allowing container networks to interconnect between two clusters as long as there is a set of IP reachable machines. This mode of multi-cluster interconnection is for Overlay network. For Underlay network, it needs the underlying infrastructure to do the inter-connection work.","title":"Cluster Inter-Connection with OVN-IC"},{"location":"en/advance/with-ovn-ic/#prerequisites","text":"The subnet CIDRs within OpenStack and Kubernetes cannot overlap with each other in auto-interconnect mode. If there is overlap, you need to refer to the subsequent manual interconnection process, which can only connect non-overlapping Subnets. A set of machines needs to exist that can be accessed by each cluster over the network and used to deploy controllers that interconnect across clusters. Each cluster needs to have a set of machines that can access each other across clusters via IP as the gateway nodes. This solution only connects to the Kubernetes default VPCs.","title":"Prerequisites"},{"location":"en/advance/with-ovn-ic/#deploy-a-single-node-ovn-ic-db","text":"Deploy the OVN-IC DB on a machine accessible by kube-ovn-controller , This DB will hold the network configuration information synchronized up from each cluster. An environment deploying docker can start the OVN-IC DB with the following command. docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh For deploying a containerd environment instead of docker you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh","title":"Deploy a single-node OVN-IC DB"},{"location":"en/advance/with-ovn-ic/#automatic-routing-mode","text":"In auto-routing mode, each cluster synchronizes the CIDR information of the Subnet under its own default VPC to OVN-IC , so make sure there is no overlap between the Subnet CIDRs of the two clusters. Create ovn-ic-config ConfigMap in kube-system Namespace: apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : Whether to enable cluster interconnection. az-name : Distinguish the cluster names of different clusters, each interconnected cluster needs to be different. ic-db-host : Address of the node where the OVN-IC DB is deployed. ic-nb-port : OVN-IC Northbound Database port, default 6645. ic-sb-port : OVN-IC Southbound Database port, default 6645. gw-nodes : The name of the nodes in the cluster interconnection that takes on the work of the gateways, separated by commas. auto-route : Whether to automatically publish and learn routes. Note: To ensure the correct operation, the ConfigMap ovn-ic-config is not allowed to be modified. If any parameter needs to be changed, please delete this ConfigMap, modify it and then apply it again. Check if the interconnected logical switch ts has been established in the ovn-ic container with the following command\uff1a # ovn-ic-sbctl show availability-zone az1 gateway deee03e0-af16-4f45-91e9-b50c3960f809 hostname: az1-gw type: geneve ip: 192 .168.42.145 port ts-az1 transit switch: ts address: [ \"00:00:00:50:AC:8C 169.254.100.45/24\" ] availability-zone az2 gateway e94cc831-8143-40e3-a478-90352773327b hostname: az2-gw type: geneve ip: 192 .168.42.149 port ts-az2 transit switch: ts address: [ \"00:00:00:07:4A:59 169.254.100.63/24\" ] At each cluster observe if logical routes have learned peer routes: # kubectl ko nbctl lr-route-list ovn-cluster IPv4 Routes 10 .42.1.1 169 .254.100.45 dst-ip ( learned ) 10 .42.1.3 100 .64.0.2 dst-ip 10 .16.0.2 100 .64.0.2 src-ip 10 .16.0.3 100 .64.0.2 src-ip 10 .16.0.4 100 .64.0.2 src-ip 10 .16.0.6 100 .64.0.2 src-ip 10 .17.0.0/16 169 .254.100.45 dst-ip ( learned ) 100 .65.0.0/16 169 .254.100.45 dst-ip ( learned ) Next, you can try ping a Pod IP in Cluster 1 directly from a Pod in Cluster 2 to see if you can work. For a subnet that does not want to automatically publish routes to the other end, you can disable route broadcasting by modifying disableInterConnection in the Subnet spec. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : no-advertise spec : cidrBlock : 10.199.0.0/16 disableInterConnection : true","title":"Automatic Routing Mode"},{"location":"en/advance/with-ovn-ic/#manual-routing-mode","text":"For cases where there are overlapping CIDRs between clusters, and you only want to do partial subnet interconnection, you can manually publish subnet routing by following the steps below. Create ovn-ic-config ConfigMap in kube-system Namespace, and set auto-route to false : apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"false\" Find the address of the remote logical ports in each cluster separately, for later manual configuration of the route: [ root@az1 ~ ] # kubectl ko nbctl show switch a391d3a1-14a0-4841-9836-4bd930c447fb ( ts ) port ts-az1 type: router router-port: az1-ts port ts-az2 type: remote addresses: [ \"00:00:00:4B:E2:9F 169.254.100.31/24\" ] [ root@az2 ~ ] # kubectl ko nbctl show switch da6138b8-de81-4908-abf9-b2224ec4edf3 ( ts ) port ts-az2 type: router router-port: az2-ts port ts-az1 type: remote addresses: [ \"00:00:00:FB:2A:F7 169.254.100.79/24\" ] The output above shows that the remote address from cluster az1 to cluster az2 is 169.254.100.31 and the remote address from az2 to az1 is 169.254.100.79 . In this example, the subnet CIDR within cluster az1 is 10.16.0.0/24 and the subnet CIDR within cluster az2 is 10.17.0.0/24 . Set up a route from cluster az1 to cluster az2 in cluster az1 : kubectl ko nbctl lr-route-add ovn-cluster 10 .17.0.0/24 169 .254.100.31 Set up a route to cluster az1 in cluster az2 : kubectl ko nbctl lr-route-add ovn-cluster 10 .16.0.0/24 169 .254.100.79","title":"Manual Routing Mode"},{"location":"en/advance/with-ovn-ic/#highly-available-ovn-ic-db-installation","text":"A highly available cluster can be formed between OVN-IC DB via the Raft protocol, which requires a minimum of 3 nodes for this deployment model. First start the leader of the OVN-IC DB on the first node. Users deploying a docker environment can use the following command: docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.3\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh If you are using containerd you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a The IP address of the node where the current container is located. NODE_IPS \uff1a The IP addresses of the three nodes running the OVN-IC database, separated by commas. Next, deploy the follower of the OVN-IC DB on the other two nodes. docker environment can use the following command. docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.2\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" -e LEADER_IP = \"192.168.65.3\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh If using containerd you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.2 \"\" --env = \"LEADER_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a The IP address of the node where the current container is located. NODE_IPS \uff1a The IP addresses of the three nodes running the OVN-IC database, separated by commas. LEADER_IP : The IP address of the OVN-IC DB leader node. Specify multiple OVN-IC database node addresses when creating ovn-ic-config for each cluster: apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3,192.168.65.2,192.168.65.1\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\"","title":"Highly Available OVN-IC DB Installation"},{"location":"en/advance/with-ovn-ic/#manual-reset","text":"In some cases, the entire interconnection configuration needs to be cleaned up due to configuration errors, you can refer to the following steps to clean up your environment. Delete the current ovn-ic-config Configmap: kubectl -n kube-system delete cm ovn-ic-config Delete ts logical switch: kubectl-ko nbctl ls-del ts Repeat the same steps at the peer cluster. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Manual Reset"},{"location":"en/advance/with-submariner/","text":"Cluster Inter-Connection with Submariner \u00b6 Submariner is an open source networking component that connects multiple Kubernetes cluster Pod and Service networks which can help Kube-OVN interconnect multiple clusters. Compared to OVN-IC , Submariner can connect Kube-OVN and non-Kube-OVN cluster networks, and Submariner can provide cross-cluster capability for services. However, Submariner currently only enables the default subnets to be connected, and cannot selectively connect multiple subnets. Prerequisites \u00b6 The Service CIDRs of the two clusters and the CIDR of the default Subnet cannot overlap. Install Submariner \u00b6 Download the subctl binary and deploy it to the appropriate path: curl -Ls https://get.submariner.io | bash export PATH = $PATH :~/.local/bin echo export PATH = \\$ PATH:~/.local/bin >> ~/.profile Change kubeconfig context to the cluster that need to deploy submariner-broker : subctl deploy-broker In this document the default subnet CIDR for cluster0 is 10.16.0.0/16 and the join subnet CIDR for cluster0 is 100.64.0.0/16 , the default subnet CIDR for cluster1 is 11.16.0.0/16 and the join subnet CIDR for cluster1 is 100.68.0.0/16 . Switch kubeconfig to cluster0 to register the cluster to the broker, and register the gateway node: subctl join broker-info.subm --clusterid cluster0 --clustercidr 100 .64.0.0/16,10.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster0 submariner.io/gateway = true Switch kubeconfig to cluster1 to register the cluster to the broker, and register the gateway node: subctl join broker-info.subm --clusterid cluster1 --clustercidr 100 .68.0.0/16,11.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster1 submariner.io/gateway = true Next, you can start Pods in each of the two clusters and try to access each other using IPs. Network communication problems can be diagnosed by using the subctl command: subctl show all subctl diagnose all For more Submariner operations please read Submariner Usage . \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cluster Inter-Connection with Submariner"},{"location":"en/advance/with-submariner/#cluster-inter-connection-with-submariner","text":"Submariner is an open source networking component that connects multiple Kubernetes cluster Pod and Service networks which can help Kube-OVN interconnect multiple clusters. Compared to OVN-IC , Submariner can connect Kube-OVN and non-Kube-OVN cluster networks, and Submariner can provide cross-cluster capability for services. However, Submariner currently only enables the default subnets to be connected, and cannot selectively connect multiple subnets.","title":"Cluster Inter-Connection with Submariner"},{"location":"en/advance/with-submariner/#prerequisites","text":"The Service CIDRs of the two clusters and the CIDR of the default Subnet cannot overlap.","title":"Prerequisites"},{"location":"en/advance/with-submariner/#install-submariner","text":"Download the subctl binary and deploy it to the appropriate path: curl -Ls https://get.submariner.io | bash export PATH = $PATH :~/.local/bin echo export PATH = \\$ PATH:~/.local/bin >> ~/.profile Change kubeconfig context to the cluster that need to deploy submariner-broker : subctl deploy-broker In this document the default subnet CIDR for cluster0 is 10.16.0.0/16 and the join subnet CIDR for cluster0 is 100.64.0.0/16 , the default subnet CIDR for cluster1 is 11.16.0.0/16 and the join subnet CIDR for cluster1 is 100.68.0.0/16 . Switch kubeconfig to cluster0 to register the cluster to the broker, and register the gateway node: subctl join broker-info.subm --clusterid cluster0 --clustercidr 100 .64.0.0/16,10.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster0 submariner.io/gateway = true Switch kubeconfig to cluster1 to register the cluster to the broker, and register the gateway node: subctl join broker-info.subm --clusterid cluster1 --clustercidr 100 .68.0.0/16,11.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster1 submariner.io/gateway = true Next, you can start Pods in each of the two clusters and try to access each other using IPs. Network communication problems can be diagnosed by using the subctl command: subctl show all subctl diagnose all For more Submariner operations please read Submariner Usage . \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Install Submariner"},{"location":"en/guide/custom-routes/","text":"Custom Routes \u00b6 Custom routes can be configured via Pod's annotations. Here is an example: apiVersion : v1 kind : Pod metadata : name : custom-routes annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine Do not set the dst field if you want to configure the default route. For workloads such as Deployment, DaemonSet and StatefulSet, custom routes must be configured via .spec.template.metadata.annotations : apiVersion : apps/v1 kind : Deployment metadata : name : custom-routes labels : app : nginx spec : replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Custom Routes"},{"location":"en/guide/custom-routes/#custom-routes","text":"Custom routes can be configured via Pod's annotations. Here is an example: apiVersion : v1 kind : Pod metadata : name : custom-routes annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine Do not set the dst field if you want to configure the default route. For workloads such as Deployment, DaemonSet and StatefulSet, custom routes must be configured via .spec.template.metadata.annotations : apiVersion : apps/v1 kind : Deployment metadata : name : custom-routes labels : app : nginx spec : replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Custom Routes"},{"location":"en/guide/dual-stack/","text":"DualStack \u00b6 Different subnets in Kube-OVN can support different IP protocols. IPv4, IPv6 and dual-stack types of subnets can exist within one cluster. However, it is recommended to use a uniform protocol type within a cluster to simplify usage and maintenance. In order to support dual-stack, the host network needs to meet the dual-stack requirements, and the Kubernetes-related parameters need to be adjusted, please refer to official guide to dual-stack . Create dual-stack Subnet \u00b6 When configuring a dual stack Subnet, you only need to set the corresponding subnet CIDR format as cidr=<IPv4 CIDR>,<IPv6 CIDR> . The CIDR order requires IPv4 to come first and IPv6 to come second, as follows. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-test spec : cidrBlock : 10.16.0.0/16,fd00:10:16::/64 excludeIps : - 10.16.0.1 - fd00:10:16::1 gateway : 10.16.0.1,fd00:10:16::1 If you need to use a dual stack for the default subnet during installation, you need to change the following parameters in the installation script: POD_CIDR = \"10.16.0.0/16,fd00:10:16::/64\" JOIN_CIDR = \"100.64.0.0/16,fd00:100:64::/64\" Check Pod Address \u00b6 Pods configured for dual-stack networks will be assigned both IPv4 and IPv6 addresses from that subnet, and the results will be displayed in the annotation of the Pod: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/allocated : \"true\" ovn.kubernetes.io/cidr : 10.16.0.0/16,fd00:10:16::/64 ovn.kubernetes.io/gateway : 10.16.0.1,fd00:10:16::1 ovn.kubernetes.io/ip_address : 10.16.0.9,fd00:10:16::9 ovn.kubernetes.io/logical_switch : ovn-default ovn.kubernetes.io/mac_address : 00:00:00:14:88:09 ovn.kubernetes.io/network_types : geneve ovn.kubernetes.io/routed : \"true\" ... podIP : 10.16.0.9 podIPs : - ip : 10.16.0.9 - ip : fd00:10:16::9 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DualStack"},{"location":"en/guide/dual-stack/#dualstack","text":"Different subnets in Kube-OVN can support different IP protocols. IPv4, IPv6 and dual-stack types of subnets can exist within one cluster. However, it is recommended to use a uniform protocol type within a cluster to simplify usage and maintenance. In order to support dual-stack, the host network needs to meet the dual-stack requirements, and the Kubernetes-related parameters need to be adjusted, please refer to official guide to dual-stack .","title":"DualStack"},{"location":"en/guide/dual-stack/#create-dual-stack-subnet","text":"When configuring a dual stack Subnet, you only need to set the corresponding subnet CIDR format as cidr=<IPv4 CIDR>,<IPv6 CIDR> . The CIDR order requires IPv4 to come first and IPv6 to come second, as follows. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-test spec : cidrBlock : 10.16.0.0/16,fd00:10:16::/64 excludeIps : - 10.16.0.1 - fd00:10:16::1 gateway : 10.16.0.1,fd00:10:16::1 If you need to use a dual stack for the default subnet during installation, you need to change the following parameters in the installation script: POD_CIDR = \"10.16.0.0/16,fd00:10:16::/64\" JOIN_CIDR = \"100.64.0.0/16,fd00:100:64::/64\"","title":"Create dual-stack Subnet"},{"location":"en/guide/dual-stack/#check-pod-address","text":"Pods configured for dual-stack networks will be assigned both IPv4 and IPv6 addresses from that subnet, and the results will be displayed in the annotation of the Pod: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/allocated : \"true\" ovn.kubernetes.io/cidr : 10.16.0.0/16,fd00:10:16::/64 ovn.kubernetes.io/gateway : 10.16.0.1,fd00:10:16::1 ovn.kubernetes.io/ip_address : 10.16.0.9,fd00:10:16::9 ovn.kubernetes.io/logical_switch : ovn-default ovn.kubernetes.io/mac_address : 00:00:00:14:88:09 ovn.kubernetes.io/network_types : geneve ovn.kubernetes.io/routed : \"true\" ... podIP : 10.16.0.9 podIPs : - ip : 10.16.0.9 - ip : fd00:10:16::9 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Check Pod Address"},{"location":"en/guide/eip-snat/","text":"EIP and SNAT \u00b6 This configuration is for the network under default VPC, for custom VPC please refer to VPC Gateway Kube-OVN supports SNAT and EIP functionality at the Pod level using the L3 Gateway feature in OVN. By using SNAT, a group of Pods can share an IP address for external access. With the EIP feature, a Pod can be directly associated with an external IP. External services can access the Pod directly through the EIP, and the Pod will also access external services through this EIP. Preparation \u00b6 In order to use the OVN's L3 Gateway capability, a separate NIC must be bridged into the OVS bridge for overlay and underlay network communication. The host must have other NICs for management. Since packets passing through NAT will go directly to the Underlay network, it is important to confirm that such packets can pass safely on the current network architecture. Currently, there is no conflict detection for EIP and SNAT addresses, and an administrator needs to manually assign them to avoid address conflicts. Create Config \u00b6 Create ConfigMap ovn-external-gw-config in kube-system Namespace: apiVersion : v1 kind : ConfigMap metadata : name : ovn-external-gw-config namespace : kube-system data : enable-external-gw : \"true\" external-gw-nodes : \"kube-ovn-worker\" external-gw-nic : \"eth1\" external-gw-addr : \"172.56.0.1/16\" nic-ip : \"172.56.0.254/16\" nic-mac : \"16:52:f3:13:6a:25\" enable-external-gw : Whether to enable SNAT and EIP functions. type : centrailized or distributed \uff0c Default is centralized If distributed is used, all nodes of the cluster need to have the same name NIC to perform the gateway function. external-gw-nodes : In centralized mode\uff0cThe names of the node performing the gateway role, comma separated. external-gw-nic : The name of the NIC that performs the role of a gateway on the node. external-gw-addr : The IP and mask of the physical network gateway. nic-ip , nic-mac : The IP and Mac assigned to the logical gateway port needs to be an unoccupied IP and Mac for the physical subnet. Confirm the Configuration Take Effect \u00b6 Check the OVN-NB status to confirm that the ovn-external logical switch exists and that the correct address and chassis are bound to the ovn-cluster-ovn-external logical router port. # kubectl ko nbctl show switch 3de4cea7-1a71-43f3-8b62-435a57ef16a6 ( ovn-external ) port ln-ovn-external type: localnet addresses: [ \"unknown\" ] port ovn-external-ovn-cluster type: router router-port: ovn-cluster-ovn-external router e1eb83ad-34be-4ed5-9a02-fcc8b1d357c4 ( ovn-cluster ) port ovn-cluster-ovn-external mac: \"ac:1f:6b:2d:33:f1\" networks: [ \"172.56.0.100/16\" ] gateway chassis: [ a5682814-2e2c-46dd-9c1c-6803ef0dab66 ] Check the OVS status to confirm that the corresponding NIC is bridged into the br-external bridge: # kubectl ko vsctl ${gateway node name} show e7d81150-7743-4d6e-9e6f-5c688232e130 Bridge br-external Port br-external Interface br-external type: internal Port eno2 Interface eno2 Port patch-ln-ovn-external-to-br-int Interface patch-ln-ovn-external-to-br-int type: patch options: { peer = patch-br-int-to-ln-ovn-external } Config EIP amd SNAT on Pod \u00b6 SNAT and EIP can be configured by adding the ovn.kubernetes.io/snat or ovn.kubernetes.io/eip annotation to the Pod, respectively: apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/snat : 172.56.0.200 spec : containers : - name : snat-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/eip : 172.56.0.233 spec : containers : - name : eip-pod image : docker.io/library/nginx:alpine The EIP or SNAT rules configured by the Pod can be dynamically adjusted via kubectl or other tools, remember to remove the ovn.kubernetes.io/routed annotation to trigger the routing change. kubectl annotate pod pod-gw ovn.kubernetes.io/eip = 172 .56.0.221 --overwrite kubectl annotate pod pod-gw ovn.kubernetes.io/routed- When the EIP or SNAT takes into effect, the ovn.kubernetes.io/routed annotation will be added back. Advanced Configuration \u00b6 Some args of kube-ovn-controller allow for advanced configuration of SNAT and EIP: --external-gateway-config-ns : The Namespace of Configmap ovn-external-gw-config , default is kube-system \u3002 --external-gateway-net : The name of the bridge to which the physical NIC is bridged, default is external . --external-gateway-vlanid : Physical network Vlan Tag number, default is 0, i.e. no Vlan is used. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"EIP and SNAT"},{"location":"en/guide/eip-snat/#eip-and-snat","text":"This configuration is for the network under default VPC, for custom VPC please refer to VPC Gateway Kube-OVN supports SNAT and EIP functionality at the Pod level using the L3 Gateway feature in OVN. By using SNAT, a group of Pods can share an IP address for external access. With the EIP feature, a Pod can be directly associated with an external IP. External services can access the Pod directly through the EIP, and the Pod will also access external services through this EIP.","title":"EIP and SNAT"},{"location":"en/guide/eip-snat/#preparation","text":"In order to use the OVN's L3 Gateway capability, a separate NIC must be bridged into the OVS bridge for overlay and underlay network communication. The host must have other NICs for management. Since packets passing through NAT will go directly to the Underlay network, it is important to confirm that such packets can pass safely on the current network architecture. Currently, there is no conflict detection for EIP and SNAT addresses, and an administrator needs to manually assign them to avoid address conflicts.","title":"Preparation"},{"location":"en/guide/eip-snat/#create-config","text":"Create ConfigMap ovn-external-gw-config in kube-system Namespace: apiVersion : v1 kind : ConfigMap metadata : name : ovn-external-gw-config namespace : kube-system data : enable-external-gw : \"true\" external-gw-nodes : \"kube-ovn-worker\" external-gw-nic : \"eth1\" external-gw-addr : \"172.56.0.1/16\" nic-ip : \"172.56.0.254/16\" nic-mac : \"16:52:f3:13:6a:25\" enable-external-gw : Whether to enable SNAT and EIP functions. type : centrailized or distributed \uff0c Default is centralized If distributed is used, all nodes of the cluster need to have the same name NIC to perform the gateway function. external-gw-nodes : In centralized mode\uff0cThe names of the node performing the gateway role, comma separated. external-gw-nic : The name of the NIC that performs the role of a gateway on the node. external-gw-addr : The IP and mask of the physical network gateway. nic-ip , nic-mac : The IP and Mac assigned to the logical gateway port needs to be an unoccupied IP and Mac for the physical subnet.","title":"Create Config"},{"location":"en/guide/eip-snat/#confirm-the-configuration-take-effect","text":"Check the OVN-NB status to confirm that the ovn-external logical switch exists and that the correct address and chassis are bound to the ovn-cluster-ovn-external logical router port. # kubectl ko nbctl show switch 3de4cea7-1a71-43f3-8b62-435a57ef16a6 ( ovn-external ) port ln-ovn-external type: localnet addresses: [ \"unknown\" ] port ovn-external-ovn-cluster type: router router-port: ovn-cluster-ovn-external router e1eb83ad-34be-4ed5-9a02-fcc8b1d357c4 ( ovn-cluster ) port ovn-cluster-ovn-external mac: \"ac:1f:6b:2d:33:f1\" networks: [ \"172.56.0.100/16\" ] gateway chassis: [ a5682814-2e2c-46dd-9c1c-6803ef0dab66 ] Check the OVS status to confirm that the corresponding NIC is bridged into the br-external bridge: # kubectl ko vsctl ${gateway node name} show e7d81150-7743-4d6e-9e6f-5c688232e130 Bridge br-external Port br-external Interface br-external type: internal Port eno2 Interface eno2 Port patch-ln-ovn-external-to-br-int Interface patch-ln-ovn-external-to-br-int type: patch options: { peer = patch-br-int-to-ln-ovn-external }","title":"Confirm the Configuration Take Effect"},{"location":"en/guide/eip-snat/#config-eip-amd-snat-on-pod","text":"SNAT and EIP can be configured by adding the ovn.kubernetes.io/snat or ovn.kubernetes.io/eip annotation to the Pod, respectively: apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/snat : 172.56.0.200 spec : containers : - name : snat-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/eip : 172.56.0.233 spec : containers : - name : eip-pod image : docker.io/library/nginx:alpine The EIP or SNAT rules configured by the Pod can be dynamically adjusted via kubectl or other tools, remember to remove the ovn.kubernetes.io/routed annotation to trigger the routing change. kubectl annotate pod pod-gw ovn.kubernetes.io/eip = 172 .56.0.221 --overwrite kubectl annotate pod pod-gw ovn.kubernetes.io/routed- When the EIP or SNAT takes into effect, the ovn.kubernetes.io/routed annotation will be added back.","title":"Config EIP amd SNAT on Pod"},{"location":"en/guide/eip-snat/#advanced-configuration","text":"Some args of kube-ovn-controller allow for advanced configuration of SNAT and EIP: --external-gateway-config-ns : The Namespace of Configmap ovn-external-gw-config , default is kube-system \u3002 --external-gateway-net : The name of the bridge to which the physical NIC is bridged, default is external . --external-gateway-vlanid : Physical network Vlan Tag number, default is 0, i.e. no Vlan is used. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Advanced Configuration"},{"location":"en/guide/ippool/","text":"Configure IPPool \u00b6 IPPool is a more granular IPAM management unit than Subnet. You can subdivide the subnet segment into multiple units through IPPool, and each unit is bound to one or more namespaces. Instructions \u00b6 Below is an example\uff1a apiVersion : kubeovn.io/v1 kind : IPPool metadata : name : pool-1 spec : subnet : ovn-default ips : - \"10.16.0.201\" - \"10.16.0.210/30\" - \"10.16.0.220..10.16.0.230\" namespaces : - ns-1 Field description: Field Usage Comment subnet Specify the subnet to which it belongs Required ips Specify IP ranges Support three formats: , and .. . Support IPv6. namespaces Specifies the bound namespaces Optional \u6ce8\u610f\u4e8b\u9879 \u00b6 To ensure compatibility with Workload Universal IP Pool Fixed Address , the name of the IP pool cannot be an IP address; The .spec.ips of the IP pool can specify an IP address beyond the scope of the subnet, but the actual effective IP address is the intersection of .spec.ips and the CIDR of the subnet; Different IP pools of the same subnet cannot contain the same (effective) IP address; The .spec.ips of the IP pool can be modified dynamically; The IP pool will inherit the reserved IP of the subnet. When randomly assigning an IP address from the IP pool, the reserved IP included in the IP pool will be skipped; When randomly assigning an IP address from a subnet, it will only be assigned from a range other than all IP pools in the subnet. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"IP\u6c60\u4f7f\u7528"},{"location":"en/guide/ippool/#configure-ippool","text":"IPPool is a more granular IPAM management unit than Subnet. You can subdivide the subnet segment into multiple units through IPPool, and each unit is bound to one or more namespaces.","title":"Configure IPPool"},{"location":"en/guide/ippool/#instructions","text":"Below is an example\uff1a apiVersion : kubeovn.io/v1 kind : IPPool metadata : name : pool-1 spec : subnet : ovn-default ips : - \"10.16.0.201\" - \"10.16.0.210/30\" - \"10.16.0.220..10.16.0.230\" namespaces : - ns-1 Field description: Field Usage Comment subnet Specify the subnet to which it belongs Required ips Specify IP ranges Support three formats: , and .. . Support IPv6. namespaces Specifies the bound namespaces Optional","title":"Instructions"},{"location":"en/guide/ippool/#_1","text":"To ensure compatibility with Workload Universal IP Pool Fixed Address , the name of the IP pool cannot be an IP address; The .spec.ips of the IP pool can specify an IP address beyond the scope of the subnet, but the actual effective IP address is the intersection of .spec.ips and the CIDR of the subnet; Different IP pools of the same subnet cannot contain the same (effective) IP address; The .spec.ips of the IP pool can be modified dynamically; The IP pool will inherit the reserved IP of the subnet. When randomly assigning an IP address from the IP pool, the reserved IP included in the IP pool will be skipped; When randomly assigning an IP address from a subnet, it will only be assigned from a range other than all IP pools in the subnet. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"en/guide/loadbalancer-service/","text":"LoadBalancer \u7c7b\u578b Service \u00b6 Kube-OVN \u5df2\u7ecf\u652f\u6301\u4e86 VPC \u548c VPC \u7f51\u5173\u7684\u5b9e\u73b0\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 VPC \u914d\u7f6e \u3002 \u7531\u4e8e VPC \u7f51\u5173\u7684\u4f7f\u7528\u6bd4\u8f83\u590d\u6742\uff0c\u57fa\u4e8e VPC \u7f51\u5173\u7684\u5b9e\u73b0\u505a\u4e86\u7b80\u5316\uff0c\u652f\u6301\u5728\u9ed8\u8ba4 VPC \u4e0b\u521b\u5efa LoadBalancer \u7c7b\u578b\u7684 Service\uff0c\u5b9e\u73b0\u901a\u8fc7 LoadBalancerIP \u6765\u8bbf\u95ee\u9ed8\u8ba4 VPC \u4e0b\u7684 Service\u3002 \u9996\u5148\u786e\u8ba4\u73af\u5883\u4e0a\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff1a \u5b89\u88c5\u4e86 multus-cni \u548c macvlan cni \u3002 LoadBalancer Service \u7684\u652f\u6301\uff0c\u662f\u5bf9 VPC \u7f51\u5173\u4ee3\u7801\u8fdb\u884c\u7b80\u5316\u5b9e\u73b0\u7684\uff0c\u4ecd\u7136\u4f7f\u7528 vpc-nat-gw \u7684\u955c\u50cf\uff0c\u4f9d\u8d56 macvlan \u63d0\u4f9b\u591a\u7f51\u5361\u529f\u80fd\u652f\u6301\u3002 \u76ee\u524d\u53ea\u652f\u6301\u5728 \u9ed8\u8ba4 VPC \u914d\u7f6e\uff0c\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 LoadBalancer \u652f\u6301\u53ef\u4ee5\u53c2\u8003 VPC \u7684\u6587\u6863 VPC \u914d\u7f6e \u3002 \u9ed8\u8ba4 VPC LoadBalancer Service \u914d\u7f6e\u6b65\u9aa4 \u00b6 \u5f00\u542f\u7279\u6027\u5f00\u5173 \u00b6 \u4fee\u6539 kube-system namespace \u4e0b\u7684 deployment kube-ovn-controller \uff0c\u5728 args \u4e2d\u589e\u52a0\u53c2\u6570 --enable-lb-svc=true \uff0c\u5f00\u542f\u529f\u80fd\u5f00\u5173\uff0c\u8be5\u53c2\u6570\u9ed8\u8ba4\u4e3a false\u3002 containers : - args : - /kube-ovn/start-controller.sh - --default-cidr=10.16.0.0/16 - --default-gateway=10.16.0.1 - --default-gateway-check=true - --enable-lb-svc=true // \u53c2\u6570\u8bbe\u7f6e\u4e3a true \u521b\u5efa NetworkAttachmentDefinition CRD \u8d44\u6e90 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa net-attach-def \u8d44\u6e90: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : lb-svc-attachment namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", //\u7269\u7406\u7f51\u5361\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u914d\u7f6e \"mode\": \"bridge\" }' \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u901a\u8fc7\u7269\u7406\u7f51\u5361 eth0 \u6765\u5b9e\u73b0\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u4ed6\u7269\u7406\u7f51\u5361\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 master \u53d6\u503c\uff0c\u6307\u5b9a\u4f7f\u7528\u7684\u7269\u7406\u7f51\u5361\u540d\u79f0\u3002 \u521b\u5efa Subnet \u00b6 \u521b\u5efa\u7684 Subnet\uff0c\u7528\u4e8e\u7ed9 LoadBalancer Service \u5206\u914d LoadBalancerIP\uff0c\u8be5\u5730\u5740\u6b63\u5e38\u60c5\u51b5\u4e0b\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u53ef\u4ee5\u914d\u7f6e Underlay Subnet \u7528\u4e8e\u5730\u5740\u5206\u914d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa\u65b0\u5b50\u7f51\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attach-subnet spec : protocol : IPv4 provider : lb-svc-attachment.kube-system # provider \u683c\u5f0f\u56fa\u5b9a\uff0c\u7531\u4e0a\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210 cidrBlock : 172.18.0.0/16 gateway : 172.18.0.1 excludeIps : - 172.18.0.0..172.18.0.10 Subnet \u4e2d provider \u53c2\u6570\u4ee5 ovn \u6216\u8005\u4ee5 .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u8868\u793a\u8be5\u5b50\u7f51\u662f\u7531 Kube-OVN \u7ba1\u7406\u4f7f\u7528\uff0c\u9700\u8981\u5bf9\u5e94\u521b\u5efa logical switch \u8bb0\u5f55\u3002 provider \u975e ovn \u6216\u8005\u975e .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u5219 Kube-OVN \u53ea\u63d0\u4f9b IPAM \u529f\u80fd\uff0c\u8bb0\u5f55 IP \u5730\u5740\u5206\u914d\u60c5\u51b5\uff0c\u4e0d\u5bf9\u5b50\u7f51\u505a\u4e1a\u52a1\u903b\u8f91\u5904\u7406\u3002 \u521b\u5efa LoadBalancer Service \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa LoadBalancer Service\uff1a apiVersion : v1 kind : Service metadata : annotations : lb-svc-attachment.kube-system.kubernetes.io/logical_switch : attach-subnet #\u53ef\u9009 ovn.kubernetes.io/attachmentprovider : lb-svc-attachment.kube-system #\u5fc5\u987b labels : app : dynamic name : test-service namespace : default spec : loadBalancerIP : 172.18.0.18 #\u53ef\u9009 ports : - name : test protocol : TCP port : 80 targetPort : 80 selector : app : dynamic sessionAffinity : None type : LoadBalancer \u5728 yaml \u4e2d\uff0cannotation ovn.kubernetes.io/attachmentprovider \u4e3a\u5fc5\u586b\u9879\uff0c\u53d6\u503c\u7531\u7b2c\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210\u3002\u8be5 annotation \u7528\u4e8e\u5728\u521b\u5efa Pod \u65f6\uff0c\u67e5\u627e net-attach-def \u8d44\u6e90\u3002 \u53ef\u4ee5\u901a\u8fc7 annotation \u6307\u5b9a\u591a\u7f51\u5361\u5730\u5740\u5206\u914d\u4f7f\u7528\u7684\u5b50\u7f51\u3002annotation key \u683c\u5f0f\u4e3a net-attach-def \u8d44\u6e90\u7684 Name.Namespace.kubernetes.io/logical_switch \u3002\u8be5\u914d\u7f6e\u4e3a \u53ef\u9009 \u9009\u9879\uff0c\u5728\u6ca1\u6709\u6307\u5b9a LoadBalancerIP \u5730\u5740\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u4ece\u8be5\u5b50\u7f51\u52a8\u6001\u5206\u914d\u5730\u5740\uff0c\u586b\u5145\u5230 LoadBalancerIP \u5b57\u6bb5\u3002 \u5982\u679c\u9700\u8981\u9759\u6001\u914d\u7f6e LoadBalancerIP \u5730\u5740\uff0c\u53ef\u4ee5\u914d\u7f6e spec.loadBalancerIP \u5b57\u6bb5\uff0c\u8be5\u5730\u5740\u9700\u8981\u5728\u6307\u5b9a\u5b50\u7f51\u7684\u5730\u5740\u8303\u56f4\u5185\u3002 \u5728\u6267\u884c yaml \u521b\u5efa Service \u540e\uff0c\u5728 Service \u540c Namespace \u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u542f\u52a8\u4fe1\u606f\uff1a # kubectl get pod NAME READY STATUS RESTARTS AGE lb-svc-test-service-6869d98dd8-cjvll 1 /1 Running 0 107m # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE test-service LoadBalancer 10 .109.201.193 172 .18.0.18 80 :30056/TCP 107m \u6307\u5b9a service.spec.loadBalancerIP \u53c2\u6570\u65f6\uff0c\u6700\u7ec8\u5c06\u8be5\u53c2\u6570\u8d4b\u503c\u7ed9 service external-ip \u5b57\u6bb5\u3002\u4e0d\u6307\u5b9a\u7684\u60c5\u51b5\u4e0b\uff0c\u8be5\u53c2\u6570\u4e3a\u968f\u673a\u5206\u914d\u503c\u3002 \u67e5\u770b\u6d4b\u8bd5 Pod \u7684 yaml \u8f93\u51fa\uff0c\u5b58\u5728\u591a\u7f51\u5361\u5206\u914d\u7684\u5730\u5740\u4fe1\u606f\uff1a # kubectl get pod -o yaml lb-svc-test-service-6869d98dd8-cjvll apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/network-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] k8s.v1.cni.cncf.io/networks: default/test-service k8s.v1.cni.cncf.io/networks-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] ovn.kubernetes.io/allocated: \"true\" ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.2 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:45:F4:29 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/routed: \"true\" test-service.default.kubernetes.io/allocated: \"true\" test-service.default.kubernetes.io/cidr: 172 .18.0.0/16 test-service.default.kubernetes.io/gateway: 172 .18.0.1 test-service.default.kubernetes.io/ip_address: 172 .18.0.18 test-service.default.kubernetes.io/logical_switch: attach-subnet test-service.default.kubernetes.io/mac_address: 00 :00:00:AF:AA:BF test-service.default.kubernetes.io/pod_nic_type: veth-pair \u67e5\u770b Service \u7684\u4fe1\u606f\uff1a # kubectl get svc -o yaml test-service apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { \"apiVersion\" : \"v1\" , \"kind\" : \"Service\" , \"metadata\" : { \"annotations\" : { \"test-service.default.kubernetes.io/logical_switch\" : \"attach-subnet\" } , \"labels\" : { \"app\" : \"dynamic\" } , \"name\" : \"test-service\" , \"namespace\" : \"default\" } , \"spec\" : { \"ports\" : [{ \"name\" : \"test\" , \"port\" :80, \"protocol\" : \"TCP\" , \"targetPort\" :80 }] , \"selector\" : { \"app\" : \"dynamic\" } , \"sessionAffinity\" : \"None\" , \"type\" : \"LoadBalancer\" }} ovn.kubernetes.io/vpc: ovn-cluster test-service.default.kubernetes.io/logical_switch: attach-subnet creationTimestamp: \"2022-06-15T09:01:58Z\" labels: app: dynamic name: test-service namespace: default resourceVersion: \"38485\" uid: 161edee1-7f6e-40f5-9e09-5a52c44267d0 spec: allocateLoadBalancerNodePorts: true clusterIP: 10 .109.201.193 clusterIPs: - 10 .109.201.193 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: test nodePort: 30056 port: 80 protocol: TCP targetPort: 80 selector: app: dynamic sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - ip: 172 .18.0.18 \u6d4b\u8bd5 LoadBalancerIP \u8bbf\u95ee \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml, \u521b\u5efa\u6d4b\u8bd5 Pod\uff0c\u4f5c\u4e3a Service \u7684 Endpoints \u63d0\u4f9b\u670d\u52a1: apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx dnsPolicy : ClusterFirst restartPolicy : Always \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u63d0\u4f9b\u7684\u5b50\u7f51\u5730\u5740\uff0c\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u4e3a\u4e86\u7b80\u5355\u9a8c\u8bc1\uff0c\u5728\u96c6\u7fa4\u5185\u8bbf\u95ee Service \u7684 LoadBalancerIP:Port \uff0c\u67e5\u770b\u662f\u5426\u6b63\u5e38\u8bbf\u95ee\u6210\u529f\u3002 # curl 172.18.0.11:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on dynamic-7d8d7874f5-hsgc4</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u8fdb\u5165 Service \u521b\u5efa\u7684 Pod\uff0c\u67e5\u770b\u7f51\u7edc\u7684\u4fe1\u606f # ip a 4 : net1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ba:85:f7:02:9f:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172 .18.0.18/16 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::b885:f7ff:fe02:9f42/64 scope link valid_lft forever preferred_lft forever 36 : eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 00 :00:00:45:f4:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10 .16.0.2/16 brd 10 .16.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe45:f429/64 scope link valid_lft forever preferred_lft forever # ip rule 0 : from all lookup local 32764 : from all iif eth0 lookup 100 32765 : from all iif net1 lookup 100 32766 : from all lookup main 32767 : from all lookup default # ip route show table 100 default via 172 .18.0.1 dev net1 10 .109.201.193 via 10 .16.0.1 dev eth0 172 .18.0.0/16 dev net1 scope link # iptables -t nat -L -n -v Chain PREROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- * * 0 .0.0.0/0 172 .18.0.18 tcp dpt:80 to:10.109.201.193:80 Chain INPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain OUTPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain POSTROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0 .0.0.0/0 10 .109.201.193 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"LoadBalancer Type Service"},{"location":"en/guide/loadbalancer-service/#loadbalancer-service","text":"Kube-OVN \u5df2\u7ecf\u652f\u6301\u4e86 VPC \u548c VPC \u7f51\u5173\u7684\u5b9e\u73b0\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 VPC \u914d\u7f6e \u3002 \u7531\u4e8e VPC \u7f51\u5173\u7684\u4f7f\u7528\u6bd4\u8f83\u590d\u6742\uff0c\u57fa\u4e8e VPC \u7f51\u5173\u7684\u5b9e\u73b0\u505a\u4e86\u7b80\u5316\uff0c\u652f\u6301\u5728\u9ed8\u8ba4 VPC \u4e0b\u521b\u5efa LoadBalancer \u7c7b\u578b\u7684 Service\uff0c\u5b9e\u73b0\u901a\u8fc7 LoadBalancerIP \u6765\u8bbf\u95ee\u9ed8\u8ba4 VPC \u4e0b\u7684 Service\u3002 \u9996\u5148\u786e\u8ba4\u73af\u5883\u4e0a\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff1a \u5b89\u88c5\u4e86 multus-cni \u548c macvlan cni \u3002 LoadBalancer Service \u7684\u652f\u6301\uff0c\u662f\u5bf9 VPC \u7f51\u5173\u4ee3\u7801\u8fdb\u884c\u7b80\u5316\u5b9e\u73b0\u7684\uff0c\u4ecd\u7136\u4f7f\u7528 vpc-nat-gw \u7684\u955c\u50cf\uff0c\u4f9d\u8d56 macvlan \u63d0\u4f9b\u591a\u7f51\u5361\u529f\u80fd\u652f\u6301\u3002 \u76ee\u524d\u53ea\u652f\u6301\u5728 \u9ed8\u8ba4 VPC \u914d\u7f6e\uff0c\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 LoadBalancer \u652f\u6301\u53ef\u4ee5\u53c2\u8003 VPC \u7684\u6587\u6863 VPC \u914d\u7f6e \u3002","title":"LoadBalancer \u7c7b\u578b Service"},{"location":"en/guide/loadbalancer-service/#vpc-loadbalancer-service","text":"","title":"\u9ed8\u8ba4 VPC LoadBalancer Service \u914d\u7f6e\u6b65\u9aa4"},{"location":"en/guide/loadbalancer-service/#_1","text":"\u4fee\u6539 kube-system namespace \u4e0b\u7684 deployment kube-ovn-controller \uff0c\u5728 args \u4e2d\u589e\u52a0\u53c2\u6570 --enable-lb-svc=true \uff0c\u5f00\u542f\u529f\u80fd\u5f00\u5173\uff0c\u8be5\u53c2\u6570\u9ed8\u8ba4\u4e3a false\u3002 containers : - args : - /kube-ovn/start-controller.sh - --default-cidr=10.16.0.0/16 - --default-gateway=10.16.0.1 - --default-gateway-check=true - --enable-lb-svc=true // \u53c2\u6570\u8bbe\u7f6e\u4e3a true","title":"\u5f00\u542f\u7279\u6027\u5f00\u5173"},{"location":"en/guide/loadbalancer-service/#networkattachmentdefinition-crd","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa net-attach-def \u8d44\u6e90: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : lb-svc-attachment namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", //\u7269\u7406\u7f51\u5361\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u914d\u7f6e \"mode\": \"bridge\" }' \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u901a\u8fc7\u7269\u7406\u7f51\u5361 eth0 \u6765\u5b9e\u73b0\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u4ed6\u7269\u7406\u7f51\u5361\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 master \u53d6\u503c\uff0c\u6307\u5b9a\u4f7f\u7528\u7684\u7269\u7406\u7f51\u5361\u540d\u79f0\u3002","title":"\u521b\u5efa NetworkAttachmentDefinition CRD \u8d44\u6e90"},{"location":"en/guide/loadbalancer-service/#subnet","text":"\u521b\u5efa\u7684 Subnet\uff0c\u7528\u4e8e\u7ed9 LoadBalancer Service \u5206\u914d LoadBalancerIP\uff0c\u8be5\u5730\u5740\u6b63\u5e38\u60c5\u51b5\u4e0b\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u53ef\u4ee5\u914d\u7f6e Underlay Subnet \u7528\u4e8e\u5730\u5740\u5206\u914d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa\u65b0\u5b50\u7f51\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attach-subnet spec : protocol : IPv4 provider : lb-svc-attachment.kube-system # provider \u683c\u5f0f\u56fa\u5b9a\uff0c\u7531\u4e0a\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210 cidrBlock : 172.18.0.0/16 gateway : 172.18.0.1 excludeIps : - 172.18.0.0..172.18.0.10 Subnet \u4e2d provider \u53c2\u6570\u4ee5 ovn \u6216\u8005\u4ee5 .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u8868\u793a\u8be5\u5b50\u7f51\u662f\u7531 Kube-OVN \u7ba1\u7406\u4f7f\u7528\uff0c\u9700\u8981\u5bf9\u5e94\u521b\u5efa logical switch \u8bb0\u5f55\u3002 provider \u975e ovn \u6216\u8005\u975e .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u5219 Kube-OVN \u53ea\u63d0\u4f9b IPAM \u529f\u80fd\uff0c\u8bb0\u5f55 IP \u5730\u5740\u5206\u914d\u60c5\u51b5\uff0c\u4e0d\u5bf9\u5b50\u7f51\u505a\u4e1a\u52a1\u903b\u8f91\u5904\u7406\u3002","title":"\u521b\u5efa Subnet"},{"location":"en/guide/loadbalancer-service/#loadbalancer-service_1","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa LoadBalancer Service\uff1a apiVersion : v1 kind : Service metadata : annotations : lb-svc-attachment.kube-system.kubernetes.io/logical_switch : attach-subnet #\u53ef\u9009 ovn.kubernetes.io/attachmentprovider : lb-svc-attachment.kube-system #\u5fc5\u987b labels : app : dynamic name : test-service namespace : default spec : loadBalancerIP : 172.18.0.18 #\u53ef\u9009 ports : - name : test protocol : TCP port : 80 targetPort : 80 selector : app : dynamic sessionAffinity : None type : LoadBalancer \u5728 yaml \u4e2d\uff0cannotation ovn.kubernetes.io/attachmentprovider \u4e3a\u5fc5\u586b\u9879\uff0c\u53d6\u503c\u7531\u7b2c\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210\u3002\u8be5 annotation \u7528\u4e8e\u5728\u521b\u5efa Pod \u65f6\uff0c\u67e5\u627e net-attach-def \u8d44\u6e90\u3002 \u53ef\u4ee5\u901a\u8fc7 annotation \u6307\u5b9a\u591a\u7f51\u5361\u5730\u5740\u5206\u914d\u4f7f\u7528\u7684\u5b50\u7f51\u3002annotation key \u683c\u5f0f\u4e3a net-attach-def \u8d44\u6e90\u7684 Name.Namespace.kubernetes.io/logical_switch \u3002\u8be5\u914d\u7f6e\u4e3a \u53ef\u9009 \u9009\u9879\uff0c\u5728\u6ca1\u6709\u6307\u5b9a LoadBalancerIP \u5730\u5740\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u4ece\u8be5\u5b50\u7f51\u52a8\u6001\u5206\u914d\u5730\u5740\uff0c\u586b\u5145\u5230 LoadBalancerIP \u5b57\u6bb5\u3002 \u5982\u679c\u9700\u8981\u9759\u6001\u914d\u7f6e LoadBalancerIP \u5730\u5740\uff0c\u53ef\u4ee5\u914d\u7f6e spec.loadBalancerIP \u5b57\u6bb5\uff0c\u8be5\u5730\u5740\u9700\u8981\u5728\u6307\u5b9a\u5b50\u7f51\u7684\u5730\u5740\u8303\u56f4\u5185\u3002 \u5728\u6267\u884c yaml \u521b\u5efa Service \u540e\uff0c\u5728 Service \u540c Namespace \u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u542f\u52a8\u4fe1\u606f\uff1a # kubectl get pod NAME READY STATUS RESTARTS AGE lb-svc-test-service-6869d98dd8-cjvll 1 /1 Running 0 107m # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE test-service LoadBalancer 10 .109.201.193 172 .18.0.18 80 :30056/TCP 107m \u6307\u5b9a service.spec.loadBalancerIP \u53c2\u6570\u65f6\uff0c\u6700\u7ec8\u5c06\u8be5\u53c2\u6570\u8d4b\u503c\u7ed9 service external-ip \u5b57\u6bb5\u3002\u4e0d\u6307\u5b9a\u7684\u60c5\u51b5\u4e0b\uff0c\u8be5\u53c2\u6570\u4e3a\u968f\u673a\u5206\u914d\u503c\u3002 \u67e5\u770b\u6d4b\u8bd5 Pod \u7684 yaml \u8f93\u51fa\uff0c\u5b58\u5728\u591a\u7f51\u5361\u5206\u914d\u7684\u5730\u5740\u4fe1\u606f\uff1a # kubectl get pod -o yaml lb-svc-test-service-6869d98dd8-cjvll apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/network-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] k8s.v1.cni.cncf.io/networks: default/test-service k8s.v1.cni.cncf.io/networks-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] ovn.kubernetes.io/allocated: \"true\" ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.2 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:45:F4:29 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/routed: \"true\" test-service.default.kubernetes.io/allocated: \"true\" test-service.default.kubernetes.io/cidr: 172 .18.0.0/16 test-service.default.kubernetes.io/gateway: 172 .18.0.1 test-service.default.kubernetes.io/ip_address: 172 .18.0.18 test-service.default.kubernetes.io/logical_switch: attach-subnet test-service.default.kubernetes.io/mac_address: 00 :00:00:AF:AA:BF test-service.default.kubernetes.io/pod_nic_type: veth-pair \u67e5\u770b Service \u7684\u4fe1\u606f\uff1a # kubectl get svc -o yaml test-service apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { \"apiVersion\" : \"v1\" , \"kind\" : \"Service\" , \"metadata\" : { \"annotations\" : { \"test-service.default.kubernetes.io/logical_switch\" : \"attach-subnet\" } , \"labels\" : { \"app\" : \"dynamic\" } , \"name\" : \"test-service\" , \"namespace\" : \"default\" } , \"spec\" : { \"ports\" : [{ \"name\" : \"test\" , \"port\" :80, \"protocol\" : \"TCP\" , \"targetPort\" :80 }] , \"selector\" : { \"app\" : \"dynamic\" } , \"sessionAffinity\" : \"None\" , \"type\" : \"LoadBalancer\" }} ovn.kubernetes.io/vpc: ovn-cluster test-service.default.kubernetes.io/logical_switch: attach-subnet creationTimestamp: \"2022-06-15T09:01:58Z\" labels: app: dynamic name: test-service namespace: default resourceVersion: \"38485\" uid: 161edee1-7f6e-40f5-9e09-5a52c44267d0 spec: allocateLoadBalancerNodePorts: true clusterIP: 10 .109.201.193 clusterIPs: - 10 .109.201.193 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: test nodePort: 30056 port: 80 protocol: TCP targetPort: 80 selector: app: dynamic sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - ip: 172 .18.0.18","title":"\u521b\u5efa LoadBalancer Service"},{"location":"en/guide/loadbalancer-service/#loadbalancerip","text":"\u53c2\u8003\u4ee5\u4e0b yaml, \u521b\u5efa\u6d4b\u8bd5 Pod\uff0c\u4f5c\u4e3a Service \u7684 Endpoints \u63d0\u4f9b\u670d\u52a1: apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx dnsPolicy : ClusterFirst restartPolicy : Always \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u63d0\u4f9b\u7684\u5b50\u7f51\u5730\u5740\uff0c\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u4e3a\u4e86\u7b80\u5355\u9a8c\u8bc1\uff0c\u5728\u96c6\u7fa4\u5185\u8bbf\u95ee Service \u7684 LoadBalancerIP:Port \uff0c\u67e5\u770b\u662f\u5426\u6b63\u5e38\u8bbf\u95ee\u6210\u529f\u3002 # curl 172.18.0.11:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on dynamic-7d8d7874f5-hsgc4</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u8fdb\u5165 Service \u521b\u5efa\u7684 Pod\uff0c\u67e5\u770b\u7f51\u7edc\u7684\u4fe1\u606f # ip a 4 : net1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ba:85:f7:02:9f:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172 .18.0.18/16 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::b885:f7ff:fe02:9f42/64 scope link valid_lft forever preferred_lft forever 36 : eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 00 :00:00:45:f4:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10 .16.0.2/16 brd 10 .16.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe45:f429/64 scope link valid_lft forever preferred_lft forever # ip rule 0 : from all lookup local 32764 : from all iif eth0 lookup 100 32765 : from all iif net1 lookup 100 32766 : from all lookup main 32767 : from all lookup default # ip route show table 100 default via 172 .18.0.1 dev net1 10 .109.201.193 via 10 .16.0.1 dev eth0 172 .18.0.0/16 dev net1 scope link # iptables -t nat -L -n -v Chain PREROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- * * 0 .0.0.0/0 172 .18.0.18 tcp dpt:80 to:10.109.201.193:80 Chain INPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain OUTPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain POSTROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0 .0.0.0/0 10 .109.201.193 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6d4b\u8bd5 LoadBalancerIP \u8bbf\u95ee"},{"location":"en/guide/mirror/","text":"Traffic Mirror \u00b6 The traffic mirroring feature allows packets to and from the container network to be copied to a specific NIC of the host. Administrators or developers can listen to this NIC to get the complete container network traffic for further analysis, monitoring, security auditing and other operations. It can also be integrated with traditional NPM for more fine-grained traffic visibility. The traffic mirroring feature introduces some performance loss, with an additional CPU consumption of 5% to 10% depending on CPU performance and traffic characteristics. Global Traffic Mirroring Settings \u00b6 The traffic mirroring is disabled by default, please modify the args of kube-ovn-cni DaemonSet to enable it: --enable-mirror=true : Whether to enable traffic mirroring. --mirror-iface=mirror0 : The name of the NIC that the traffic mirror is copied to. This NIC can be a physical NIC that already exists on the host machine. At this point the NIC will be bridged into the br-int bridge and the mirrored traffic will go directly to the underlying switch. If the NIC name does not exist, Kube-OVN will automatically create a virtual NIC with the same name, through which the administrator or developer can access all traffic on the current node on the host. The default is mirror0 . Next, you can listen to the traffic on mirror0 with tcpdump or other traffic analysis tools. tcpdump -ni mirror0 Pod Level Mirroring Settings \u00b6 If you only need to mirror some Pod traffic, you need to disable the global traffic mirroring and then add the ovn.kubernetes.io/mirror annotation on a specific Pod to enable Pod-level traffic mirroring. apiVersion : v1 kind : Pod metadata : name : mirror-pod namespace : ls1 annotations : ovn.kubernetes.io/mirror : \"true\" spec : containers : - name : mirror-pod image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Traffic Mirror"},{"location":"en/guide/mirror/#traffic-mirror","text":"The traffic mirroring feature allows packets to and from the container network to be copied to a specific NIC of the host. Administrators or developers can listen to this NIC to get the complete container network traffic for further analysis, monitoring, security auditing and other operations. It can also be integrated with traditional NPM for more fine-grained traffic visibility. The traffic mirroring feature introduces some performance loss, with an additional CPU consumption of 5% to 10% depending on CPU performance and traffic characteristics.","title":"Traffic Mirror"},{"location":"en/guide/mirror/#global-traffic-mirroring-settings","text":"The traffic mirroring is disabled by default, please modify the args of kube-ovn-cni DaemonSet to enable it: --enable-mirror=true : Whether to enable traffic mirroring. --mirror-iface=mirror0 : The name of the NIC that the traffic mirror is copied to. This NIC can be a physical NIC that already exists on the host machine. At this point the NIC will be bridged into the br-int bridge and the mirrored traffic will go directly to the underlying switch. If the NIC name does not exist, Kube-OVN will automatically create a virtual NIC with the same name, through which the administrator or developer can access all traffic on the current node on the host. The default is mirror0 . Next, you can listen to the traffic on mirror0 with tcpdump or other traffic analysis tools. tcpdump -ni mirror0","title":"Global Traffic Mirroring Settings"},{"location":"en/guide/mirror/#pod-level-mirroring-settings","text":"If you only need to mirror some Pod traffic, you need to disable the global traffic mirroring and then add the ovn.kubernetes.io/mirror annotation on a specific Pod to enable Pod-level traffic mirroring. apiVersion : v1 kind : Pod metadata : name : mirror-pod namespace : ls1 annotations : ovn.kubernetes.io/mirror : \"true\" spec : containers : - name : mirror-pod image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Pod Level Mirroring Settings"},{"location":"en/guide/networkpolicy-log/","text":"NetworkPolicy Logging \u00b6 NetworkPolicy is a interface provided by Kubernetes and implemented by Kube-OVN through OVN's ACLs. With NetworkPolicy, if the networks are down, it is difficult to determine whether it is caused by a network failure or a NetworkPolicy rule problem. Kube-OVN provides NetworkPolicy logging to help administrators quickly locate whether a NetworkPolicy drop rule has been hit, and to record the illegal accesses. Once NetworkPolicy logging is turned on, logs need to be printed for every packet that hits a Drop rule, which introduces additional performance overhead. Under a malicious attack, a large number of logs in a short period of time may exhaust the CPU. We recommend turning off logging by default in production environments and dynamically turning it on when you need to troubleshoot problems. Enable NetworkPolicy Logging \u00b6 Add the annotation ovn.kubernetes.io/enable_log to the NetworkPolicy where logging needs to be enabled, as follows: apiVersion : networking.k8s.io/v1 kind : NetworkPolicy metadata : name : default-deny-ingress namespace : kube-system annotations : ovn.kubernetes.io/enable_log : \"true\" spec : podSelector : {} policyTypes : - Ingress Next, you can observe the log of dropped packets in /var/log/ovn/ovn-controller.log on the host of the corresponding Pod: # tail -f /var/log/ovn/ovn-controller.log 2022 -07-20T05:55:03.229Z | 00394 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54343 ,tp_dst = 53 2022 -07-20T05:55:06.229Z | 00395 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 44187 ,tp_dst = 53 2022 -07-20T05:55:08.230Z | 00396 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54274 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00397 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 32778 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00398 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 34188 ,tp_dst = 53 2022 -07-20T05:55:13.231Z | 00399 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 43290 ,tp_dst = 53 2022 -07-20T05:55:22.096Z | 00400 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.097Z | 00401 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.098Z | 00402 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 Disable NetworkPolicy Logging \u00b6 Set annotation ovn.kubernetes.io/enable_log in the corresponding NetworkPolicy to false to disable NetworkPolicy logging: kubectl annotate networkpolicy -n kube-system default-deny-ingress ovn.kubernetes.io/enable_log = false --overwrite \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NetworkPolicy Logging"},{"location":"en/guide/networkpolicy-log/#networkpolicy-logging","text":"NetworkPolicy is a interface provided by Kubernetes and implemented by Kube-OVN through OVN's ACLs. With NetworkPolicy, if the networks are down, it is difficult to determine whether it is caused by a network failure or a NetworkPolicy rule problem. Kube-OVN provides NetworkPolicy logging to help administrators quickly locate whether a NetworkPolicy drop rule has been hit, and to record the illegal accesses. Once NetworkPolicy logging is turned on, logs need to be printed for every packet that hits a Drop rule, which introduces additional performance overhead. Under a malicious attack, a large number of logs in a short period of time may exhaust the CPU. We recommend turning off logging by default in production environments and dynamically turning it on when you need to troubleshoot problems.","title":"NetworkPolicy Logging"},{"location":"en/guide/networkpolicy-log/#enable-networkpolicy-logging","text":"Add the annotation ovn.kubernetes.io/enable_log to the NetworkPolicy where logging needs to be enabled, as follows: apiVersion : networking.k8s.io/v1 kind : NetworkPolicy metadata : name : default-deny-ingress namespace : kube-system annotations : ovn.kubernetes.io/enable_log : \"true\" spec : podSelector : {} policyTypes : - Ingress Next, you can observe the log of dropped packets in /var/log/ovn/ovn-controller.log on the host of the corresponding Pod: # tail -f /var/log/ovn/ovn-controller.log 2022 -07-20T05:55:03.229Z | 00394 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54343 ,tp_dst = 53 2022 -07-20T05:55:06.229Z | 00395 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 44187 ,tp_dst = 53 2022 -07-20T05:55:08.230Z | 00396 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54274 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00397 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 32778 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00398 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 34188 ,tp_dst = 53 2022 -07-20T05:55:13.231Z | 00399 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 43290 ,tp_dst = 53 2022 -07-20T05:55:22.096Z | 00400 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.097Z | 00401 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.098Z | 00402 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0","title":"Enable NetworkPolicy Logging"},{"location":"en/guide/networkpolicy-log/#disable-networkpolicy-logging","text":"Set annotation ovn.kubernetes.io/enable_log in the corresponding NetworkPolicy to false to disable NetworkPolicy logging: kubectl annotate networkpolicy -n kube-system default-deny-ingress ovn.kubernetes.io/enable_log = false --overwrite \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Disable NetworkPolicy Logging"},{"location":"en/guide/prometheus-grafana/","text":"Monitor and Dashboard \u00b6 Kube-OVN can export network control plane information and network data plane quality information metrics to the external in formats supported by Prometheus. We use the CRD provided by kube-prometheus to define the corresponding Prometheus monitoring rules. For all monitoring metrics supported by Kube-OVN, please refer to Kube-OVN Monitoring Metrics . If you are using native Prometheus, please refer to Configuring Native Prometheus for configuration. Install Prometheus Monitor \u00b6 Kube-OVN uses Prometheus Monitor CRD to manage the monitoring output. # network quality related monitoring metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-monitor.yaml # kube-ovn-controller metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-monitor.yaml # kube-ovn-cni metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-monitor.yaml # ovn metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-monitor.yaml The default interval for Prometheus pull is 15s, if you need to adjust it, modify the interval value in yaml. Import Grafana Dashboard \u00b6 Kube-OVN provides a predefined Grafana Dashboard to display control plane and data plane related metrics. Download the corresponding Dashboard template: # network quality related monitoring dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-grafana.json # kube-ovn-controller dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-grafana.json # kube-ovn-cni dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-grafana.json # ovn dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-grafana.json # ovs dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovs-grafana.json Import these templates into Grafana and set the data source to the corresponding Prometheus to see the following Dashboards. kube-ovn-controller dashboard: kube-ovn-pinger dashboard: kube-ovn-cni dashboard: \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Monitor and Dashboard"},{"location":"en/guide/prometheus-grafana/#monitor-and-dashboard","text":"Kube-OVN can export network control plane information and network data plane quality information metrics to the external in formats supported by Prometheus. We use the CRD provided by kube-prometheus to define the corresponding Prometheus monitoring rules. For all monitoring metrics supported by Kube-OVN, please refer to Kube-OVN Monitoring Metrics . If you are using native Prometheus, please refer to Configuring Native Prometheus for configuration.","title":"Monitor and Dashboard"},{"location":"en/guide/prometheus-grafana/#install-prometheus-monitor","text":"Kube-OVN uses Prometheus Monitor CRD to manage the monitoring output. # network quality related monitoring metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-monitor.yaml # kube-ovn-controller metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-monitor.yaml # kube-ovn-cni metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-monitor.yaml # ovn metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-monitor.yaml The default interval for Prometheus pull is 15s, if you need to adjust it, modify the interval value in yaml.","title":"Install Prometheus Monitor"},{"location":"en/guide/prometheus-grafana/#import-grafana-dashboard","text":"Kube-OVN provides a predefined Grafana Dashboard to display control plane and data plane related metrics. Download the corresponding Dashboard template: # network quality related monitoring dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-grafana.json # kube-ovn-controller dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-grafana.json # kube-ovn-cni dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-grafana.json # ovn dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-grafana.json # ovs dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovs-grafana.json Import these templates into Grafana and set the data source to the corresponding Prometheus to see the following Dashboards. kube-ovn-controller dashboard: kube-ovn-pinger dashboard: kube-ovn-cni dashboard: \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Import Grafana Dashboard"},{"location":"en/guide/prometheus/","text":"Config Native Prometheus \u00b6 Kube-OVN provides rich monitoring data for OVN/OVS health status checks and connectivity checks of container and host networks, and Kube-OVN is configured with ServiceMonitor for Prometheus to dynamically obtain monitoring metrics. In some cases, where only Prometheus Server is installed and no other components are installed, you can dynamically obtain monitoring data for the cluster environment by modifying the configuration of Prometheus. Config Prometheus \u00b6 The following configuration documentation, referenced from Prometheus Service Discovery . Permission Configuration \u00b6 Prometheus is deployed in the cluster and needs to access the k8s apiserver to query the monitoring data of the containers. Refer to the following yaml to configure the permissions required by Prometheus: apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : prometheus rules : - apiGroups : [ \"\" ] resources : - nodes - nodes/proxy - services - endpoints - pods verbs : [ \"get\" , \"list\" , \"watch\" ] - apiGroups : - extensions resources : - ingresses verbs : [ \"get\" , \"list\" , \"watch\" ] - nonResourceURLs : [ \"/metrics\" ] verbs : [ \"get\" ] --- apiVersion : v1 kind : ServiceAccount metadata : name : prometheus namespace : default --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : prometheus roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : prometheus subjects : - kind : ServiceAccount name : prometheus namespace : default Prometheus ConfigMap \u00b6 The startup of Prometheus relies on the configuration file prometheus.yml, the contents of which can be configured in ConfigMap and dynamically mounted to the Pod. Create the ConfigMap file used by Prometheus by referring to the following yaml: apiVersion : v1 kind : ConfigMap metadata : name : prometheus-config data : prometheus.yml : |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'kubernetes-nodes' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node - job_name: 'kubernetes-service' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: service - job_name: 'kubernetes-endpoints' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints - job_name: 'kubernetes-ingress' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: ingress - job_name: 'kubernetes-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: pod Prometheus provides role-based querying of Kubernetes resource monitoring operations, which can be configured in the official documentation kubernetes_sd_config \u3002 Deploy Prometheus \u00b6 Deploy Prometheus Server by referring to the following yaml: apiVersion : apps/v1 kind : Deployment metadata : labels : app : prometheus name : prometheus namespace : default spec : replicas : 1 selector : matchLabels : app : prometheus strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : prometheus spec : serviceAccountName : prometheus serviceAccount : prometheus containers : - image : docker.io/prom/prometheus:latest imagePullPolicy : IfNotPresent name : prometheus command : - \"/bin/prometheus\" args : - \"--config.file=/etc/prometheus/prometheus.yml\" ports : - containerPort : 9090 protocol : TCP volumeMounts : - mountPath : \"/etc/prometheus\" name : prometheus-config volumes : - name : prometheus-config configMap : name : prometheus-config Deploy Prometheus Service by referring to the following yaml: kind : Service apiVersion : v1 metadata : name : prometheus namespace : default labels : name : prometheus spec : ports : - name : test protocol : TCP port : 9090 targetPort : 9090 type : NodePort selector : app : prometheus sessionAffinity : None After exposing Prometheus through NodePort, Prometheus can be accessed through the node address. Prometheus Metrics Config \u00b6 View information about Prometheus on the environment: # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kubernetes ClusterIP 10 .4.0.1 <none> 443 /TCP 8d prometheus NodePort 10 .4.102.222 <none> 9090 :32611/TCP 8d # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-7544b6b84d-v9m8s 1 /1 Running 0 3d5h 10 .3.0.7 192 .168.137.219 <none> <none> # kubectl get endpoints -o wide NAME ENDPOINTS AGE kubernetes 192 .168.136.228:6443,192.168.136.232:6443,192.168.137.219:6443 8d prometheus 10 .3.0.7:9090 8d Access Prometheus via NodePort to see the data dynamically queried by Status/Service Discovery: You can see that you can currently query all the service data information on the cluster. Configure to Query Specified Resource \u00b6 The ConfigMap configuration above queries all resource data. If you only need resource data for a certain role, you can add filter conditions. Take Service as an example, modify the ConfigMap content to query only the service monitoring data: - job_name : 'kubernetes-service' tls_config : ca_file : /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file : /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs : - role : service relabel_configs : - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_scrape ] action : \"keep\" regex : \"true\" - action : labelmap regex : __meta_kubernetes_service_label_(.+) - source_labels : [ __meta_kubernetes_namespace ] target_label : kubernetes_namespace - source_labels : [ __meta_kubernetes_service_name ] target_label : kubernetes_service_name - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_path ] action : replace target_label : __metrics_path__ regex : \"(.+)\" Check the Kube-OVN Service in kube-system Namespace: # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kube-dns ClusterIP 10 .4.0.10 <none> 53 /UDP,53/TCP,9153/TCP 13d kube-ovn-cni ClusterIP 10 .4.228.60 <none> 10665 /TCP 13d kube-ovn-controller ClusterIP 10 .4.172.213 <none> 10660 /TCP 13d kube-ovn-monitor ClusterIP 10 .4.242.9 <none> 10661 /TCP 13d kube-ovn-pinger ClusterIP 10 .4.122.52 <none> 8080 /TCP 13d ovn-nb ClusterIP 10 .4.80.213 <none> 6641 /TCP 13d ovn-northd ClusterIP 10 .4.126.234 <none> 6643 /TCP 13d ovn-sb ClusterIP 10 .4.216.249 <none> 6642 /TCP 13d Add annotation prometheus.io/scrape=\"true\" to Service\uff1a # kubectl annotate svc -n kube-system kube-ovn-cni prometheus.io/scrape=true service/kube-ovn-cni annotated # kubectl annotate svc -n kube-system kube-ovn-controller prometheus.io/scrape=true service/kube-ovn-controller annotated # kubectl annotate svc -n kube-system kube-ovn-monitor prometheus.io/scrape=true service/kube-ovn-monitor annotated # kubectl annotate svc -n kube-system kube-ovn-pinger prometheus.io/scrape=true service/kube-ovn-pinger annotated Check the configured Service information: # kubectl get svc -o yaml -n kube-system kube-ovn-controller apiVersion: v1 kind: Service metadata: annotations: helm.sh/chart-version: v3.10.0-alpha.55 helm.sh/original-name: kube-ovn-controller ovn.kubernetes.io/vpc: ovn-cluster prometheus.io/scrape: \"true\" // added annotation labels: app: kube-ovn-controller name: kube-ovn-controller namespace: kube-system spec: clusterIP: 10 .4.172.213 clusterIPs: - 10 .4.172.213 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: metrics port: 10660 protocol: TCP targetPort: 10660 selector: app: kube-ovn-controller sessionAffinity: None type: ClusterIP status: loadBalancer: {} Looking at the Prometheus Status Targets information, you can only see the Services with annotation: For more information about adding filter parameters to relabel, please check Prometheus-Relabel \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Config Native Prometheus"},{"location":"en/guide/prometheus/#config-native-prometheus","text":"Kube-OVN provides rich monitoring data for OVN/OVS health status checks and connectivity checks of container and host networks, and Kube-OVN is configured with ServiceMonitor for Prometheus to dynamically obtain monitoring metrics. In some cases, where only Prometheus Server is installed and no other components are installed, you can dynamically obtain monitoring data for the cluster environment by modifying the configuration of Prometheus.","title":"Config Native Prometheus"},{"location":"en/guide/prometheus/#config-prometheus","text":"The following configuration documentation, referenced from Prometheus Service Discovery .","title":"Config Prometheus"},{"location":"en/guide/prometheus/#permission-configuration","text":"Prometheus is deployed in the cluster and needs to access the k8s apiserver to query the monitoring data of the containers. Refer to the following yaml to configure the permissions required by Prometheus: apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : prometheus rules : - apiGroups : [ \"\" ] resources : - nodes - nodes/proxy - services - endpoints - pods verbs : [ \"get\" , \"list\" , \"watch\" ] - apiGroups : - extensions resources : - ingresses verbs : [ \"get\" , \"list\" , \"watch\" ] - nonResourceURLs : [ \"/metrics\" ] verbs : [ \"get\" ] --- apiVersion : v1 kind : ServiceAccount metadata : name : prometheus namespace : default --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : prometheus roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : prometheus subjects : - kind : ServiceAccount name : prometheus namespace : default","title":"Permission Configuration"},{"location":"en/guide/prometheus/#prometheus-configmap","text":"The startup of Prometheus relies on the configuration file prometheus.yml, the contents of which can be configured in ConfigMap and dynamically mounted to the Pod. Create the ConfigMap file used by Prometheus by referring to the following yaml: apiVersion : v1 kind : ConfigMap metadata : name : prometheus-config data : prometheus.yml : |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'kubernetes-nodes' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node - job_name: 'kubernetes-service' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: service - job_name: 'kubernetes-endpoints' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints - job_name: 'kubernetes-ingress' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: ingress - job_name: 'kubernetes-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: pod Prometheus provides role-based querying of Kubernetes resource monitoring operations, which can be configured in the official documentation kubernetes_sd_config \u3002","title":"Prometheus ConfigMap"},{"location":"en/guide/prometheus/#deploy-prometheus","text":"Deploy Prometheus Server by referring to the following yaml: apiVersion : apps/v1 kind : Deployment metadata : labels : app : prometheus name : prometheus namespace : default spec : replicas : 1 selector : matchLabels : app : prometheus strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : prometheus spec : serviceAccountName : prometheus serviceAccount : prometheus containers : - image : docker.io/prom/prometheus:latest imagePullPolicy : IfNotPresent name : prometheus command : - \"/bin/prometheus\" args : - \"--config.file=/etc/prometheus/prometheus.yml\" ports : - containerPort : 9090 protocol : TCP volumeMounts : - mountPath : \"/etc/prometheus\" name : prometheus-config volumes : - name : prometheus-config configMap : name : prometheus-config Deploy Prometheus Service by referring to the following yaml: kind : Service apiVersion : v1 metadata : name : prometheus namespace : default labels : name : prometheus spec : ports : - name : test protocol : TCP port : 9090 targetPort : 9090 type : NodePort selector : app : prometheus sessionAffinity : None After exposing Prometheus through NodePort, Prometheus can be accessed through the node address.","title":"Deploy Prometheus"},{"location":"en/guide/prometheus/#prometheus-metrics-config","text":"View information about Prometheus on the environment: # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kubernetes ClusterIP 10 .4.0.1 <none> 443 /TCP 8d prometheus NodePort 10 .4.102.222 <none> 9090 :32611/TCP 8d # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-7544b6b84d-v9m8s 1 /1 Running 0 3d5h 10 .3.0.7 192 .168.137.219 <none> <none> # kubectl get endpoints -o wide NAME ENDPOINTS AGE kubernetes 192 .168.136.228:6443,192.168.136.232:6443,192.168.137.219:6443 8d prometheus 10 .3.0.7:9090 8d Access Prometheus via NodePort to see the data dynamically queried by Status/Service Discovery: You can see that you can currently query all the service data information on the cluster.","title":"Prometheus Metrics Config"},{"location":"en/guide/prometheus/#configure-to-query-specified-resource","text":"The ConfigMap configuration above queries all resource data. If you only need resource data for a certain role, you can add filter conditions. Take Service as an example, modify the ConfigMap content to query only the service monitoring data: - job_name : 'kubernetes-service' tls_config : ca_file : /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file : /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs : - role : service relabel_configs : - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_scrape ] action : \"keep\" regex : \"true\" - action : labelmap regex : __meta_kubernetes_service_label_(.+) - source_labels : [ __meta_kubernetes_namespace ] target_label : kubernetes_namespace - source_labels : [ __meta_kubernetes_service_name ] target_label : kubernetes_service_name - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_path ] action : replace target_label : __metrics_path__ regex : \"(.+)\" Check the Kube-OVN Service in kube-system Namespace: # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kube-dns ClusterIP 10 .4.0.10 <none> 53 /UDP,53/TCP,9153/TCP 13d kube-ovn-cni ClusterIP 10 .4.228.60 <none> 10665 /TCP 13d kube-ovn-controller ClusterIP 10 .4.172.213 <none> 10660 /TCP 13d kube-ovn-monitor ClusterIP 10 .4.242.9 <none> 10661 /TCP 13d kube-ovn-pinger ClusterIP 10 .4.122.52 <none> 8080 /TCP 13d ovn-nb ClusterIP 10 .4.80.213 <none> 6641 /TCP 13d ovn-northd ClusterIP 10 .4.126.234 <none> 6643 /TCP 13d ovn-sb ClusterIP 10 .4.216.249 <none> 6642 /TCP 13d Add annotation prometheus.io/scrape=\"true\" to Service\uff1a # kubectl annotate svc -n kube-system kube-ovn-cni prometheus.io/scrape=true service/kube-ovn-cni annotated # kubectl annotate svc -n kube-system kube-ovn-controller prometheus.io/scrape=true service/kube-ovn-controller annotated # kubectl annotate svc -n kube-system kube-ovn-monitor prometheus.io/scrape=true service/kube-ovn-monitor annotated # kubectl annotate svc -n kube-system kube-ovn-pinger prometheus.io/scrape=true service/kube-ovn-pinger annotated Check the configured Service information: # kubectl get svc -o yaml -n kube-system kube-ovn-controller apiVersion: v1 kind: Service metadata: annotations: helm.sh/chart-version: v3.10.0-alpha.55 helm.sh/original-name: kube-ovn-controller ovn.kubernetes.io/vpc: ovn-cluster prometheus.io/scrape: \"true\" // added annotation labels: app: kube-ovn-controller name: kube-ovn-controller namespace: kube-system spec: clusterIP: 10 .4.172.213 clusterIPs: - 10 .4.172.213 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: metrics port: 10660 protocol: TCP targetPort: 10660 selector: app: kube-ovn-controller sessionAffinity: None type: ClusterIP status: loadBalancer: {} Looking at the Prometheus Status Targets information, you can only see the Services with annotation: For more information about adding filter parameters to relabel, please check Prometheus-Relabel \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Configure to Query Specified Resource"},{"location":"en/guide/qos/","text":"Manage QoS \u00b6 Kube-OVN supports two types of Pod level QoS: Maximum bandwidth limit QoS. linux-netem , QoS for simulating latency and packet loss that can be used for simulation testing. Currently, only Pod level QoS is supported, and QoS restrictions at the Namespace or Subnet level are not supported. Maximum Bandwidth Limit QoS \u00b6 This type of QoS can be dynamically configured via Pod annotation and can be adjusted without restarting running Pod. Bandwidth speed limit unit is Mbit/s . apiVersion : v1 kind : Pod metadata : name : qos namespace : ls1 annotations : ovn.kubernetes.io/ingress_rate : \"3\" ovn.kubernetes.io/egress_rate : \"1\" spec : containers : - name : qos image : docker.io/library/nginx:alpine Use annotation to dynamically adjust QoS: kubectl annotate --overwrite pod nginx-74d5899f46-d7qkn ovn.kubernetes.io/ingress_rate = 3 Test QoS \u00b6 Deploy the containers needed for performance testing: kind : DaemonSet apiVersion : apps/v1 metadata : name : perf namespace : ls1 labels : app : perf spec : selector : matchLabels : app : perf template : metadata : labels : app : perf spec : containers : - name : nginx image : docker.io/kubeovn/perf Exec into one Pod and run iperf3 server: # kubectl exec -it perf-4n4gt -n ls1 sh # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Exec into the other Pod and run iperf3 client to connect above server address: # kubectl exec -it perf-d4mqc -n ls1 sh # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 51544 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 86 .4 MBytes 725 Mbits/sec 3 350 KBytes [ 4 ] 1 .00-2.00 sec 89 .9 MBytes 754 Mbits/sec 118 473 KBytes [ 4 ] 2 .00-3.00 sec 101 MBytes 848 Mbits/sec 184 586 KBytes [ 4 ] 3 .00-4.00 sec 104 MBytes 875 Mbits/sec 217 671 KBytes [ 4 ] 4 .00-5.00 sec 111 MBytes 935 Mbits/sec 175 772 KBytes [ 4 ] 5 .00-6.00 sec 100 MBytes 840 Mbits/sec 658 598 KBytes [ 4 ] 6 .00-7.00 sec 106 MBytes 890 Mbits/sec 742 668 KBytes [ 4 ] 7 .00-8.00 sec 102 MBytes 857 Mbits/sec 764 724 KBytes [ 4 ] 8 .00-9.00 sec 97 .4 MBytes 817 Mbits/sec 1175 764 KBytes [ 4 ] 9 .00-10.00 sec 111 MBytes 934 Mbits/sec 1083 838 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 1010 MBytes 848 Mbits/sec 5119 sender [ 4 ] 0 .00-10.00 sec 1008 MBytes 846 Mbits/sec receiver iperf Done. Modify the ingress bandwidth QoS for the first Pod: kubectl annotate --overwrite pod perf-4n4gt -n ls1 ovn.kubernetes.io/ingress_rate = 30 Test the Pod bandwidth again from the second Pod: # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 52372 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 3 .66 MBytes 30 .7 Mbits/sec 2 76 .1 KBytes [ 4 ] 1 .00-2.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 104 KBytes [ 4 ] 2 .00-3.00 sec 3 .50 MBytes 29 .4 Mbits/sec 0 126 KBytes [ 4 ] 3 .00-4.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 144 KBytes [ 4 ] 4 .00-5.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 160 KBytes [ 4 ] 5 .00-6.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 175 KBytes [ 4 ] 6 .00-7.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 212 KBytes [ 4 ] 7 .00-8.00 sec 3 .68 MBytes 30 .9 Mbits/sec 0 294 KBytes [ 4 ] 8 .00-9.00 sec 3 .74 MBytes 31 .4 Mbits/sec 0 398 KBytes [ 4 ] 9 .00-10.00 sec 3 .80 MBytes 31 .9 Mbits/sec 0 526 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 35 .7 MBytes 29 .9 Mbits/sec 2 sender [ 4 ] 0 .00-10.00 sec 34 .5 MBytes 29 .0 Mbits/sec receiver iperf Done. linux-netem QoS \u00b6 Pod can use annotation below to config linux-netem type QoS\uff1a ovn.kubernetes.io/latency \u3001 ovn.kubernetes.io/limit and ovn.kubernetes.io/loss \u3002 ovn.kubernetes.io/latency : Set the Pod traffic delay to an integer value in ms. ovn.kubernetes.io/limit \uff1a Set the maximum number of packets that the qdisc queue can hold, and takes an integer value, such as 1000. ovn.kubernetes.io/loss \uff1a Set packet loss probability, the value is float type, for example, the value is 20, then it is set 20% packet loss probability. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Manage QoS"},{"location":"en/guide/qos/#manage-qos","text":"Kube-OVN supports two types of Pod level QoS: Maximum bandwidth limit QoS. linux-netem , QoS for simulating latency and packet loss that can be used for simulation testing. Currently, only Pod level QoS is supported, and QoS restrictions at the Namespace or Subnet level are not supported.","title":"Manage QoS"},{"location":"en/guide/qos/#maximum-bandwidth-limit-qos","text":"This type of QoS can be dynamically configured via Pod annotation and can be adjusted without restarting running Pod. Bandwidth speed limit unit is Mbit/s . apiVersion : v1 kind : Pod metadata : name : qos namespace : ls1 annotations : ovn.kubernetes.io/ingress_rate : \"3\" ovn.kubernetes.io/egress_rate : \"1\" spec : containers : - name : qos image : docker.io/library/nginx:alpine Use annotation to dynamically adjust QoS: kubectl annotate --overwrite pod nginx-74d5899f46-d7qkn ovn.kubernetes.io/ingress_rate = 3","title":"Maximum Bandwidth Limit QoS"},{"location":"en/guide/qos/#test-qos","text":"Deploy the containers needed for performance testing: kind : DaemonSet apiVersion : apps/v1 metadata : name : perf namespace : ls1 labels : app : perf spec : selector : matchLabels : app : perf template : metadata : labels : app : perf spec : containers : - name : nginx image : docker.io/kubeovn/perf Exec into one Pod and run iperf3 server: # kubectl exec -it perf-4n4gt -n ls1 sh # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Exec into the other Pod and run iperf3 client to connect above server address: # kubectl exec -it perf-d4mqc -n ls1 sh # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 51544 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 86 .4 MBytes 725 Mbits/sec 3 350 KBytes [ 4 ] 1 .00-2.00 sec 89 .9 MBytes 754 Mbits/sec 118 473 KBytes [ 4 ] 2 .00-3.00 sec 101 MBytes 848 Mbits/sec 184 586 KBytes [ 4 ] 3 .00-4.00 sec 104 MBytes 875 Mbits/sec 217 671 KBytes [ 4 ] 4 .00-5.00 sec 111 MBytes 935 Mbits/sec 175 772 KBytes [ 4 ] 5 .00-6.00 sec 100 MBytes 840 Mbits/sec 658 598 KBytes [ 4 ] 6 .00-7.00 sec 106 MBytes 890 Mbits/sec 742 668 KBytes [ 4 ] 7 .00-8.00 sec 102 MBytes 857 Mbits/sec 764 724 KBytes [ 4 ] 8 .00-9.00 sec 97 .4 MBytes 817 Mbits/sec 1175 764 KBytes [ 4 ] 9 .00-10.00 sec 111 MBytes 934 Mbits/sec 1083 838 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 1010 MBytes 848 Mbits/sec 5119 sender [ 4 ] 0 .00-10.00 sec 1008 MBytes 846 Mbits/sec receiver iperf Done. Modify the ingress bandwidth QoS for the first Pod: kubectl annotate --overwrite pod perf-4n4gt -n ls1 ovn.kubernetes.io/ingress_rate = 30 Test the Pod bandwidth again from the second Pod: # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 52372 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 3 .66 MBytes 30 .7 Mbits/sec 2 76 .1 KBytes [ 4 ] 1 .00-2.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 104 KBytes [ 4 ] 2 .00-3.00 sec 3 .50 MBytes 29 .4 Mbits/sec 0 126 KBytes [ 4 ] 3 .00-4.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 144 KBytes [ 4 ] 4 .00-5.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 160 KBytes [ 4 ] 5 .00-6.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 175 KBytes [ 4 ] 6 .00-7.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 212 KBytes [ 4 ] 7 .00-8.00 sec 3 .68 MBytes 30 .9 Mbits/sec 0 294 KBytes [ 4 ] 8 .00-9.00 sec 3 .74 MBytes 31 .4 Mbits/sec 0 398 KBytes [ 4 ] 9 .00-10.00 sec 3 .80 MBytes 31 .9 Mbits/sec 0 526 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 35 .7 MBytes 29 .9 Mbits/sec 2 sender [ 4 ] 0 .00-10.00 sec 34 .5 MBytes 29 .0 Mbits/sec receiver iperf Done.","title":"Test QoS"},{"location":"en/guide/qos/#linux-netem-qos","text":"Pod can use annotation below to config linux-netem type QoS\uff1a ovn.kubernetes.io/latency \u3001 ovn.kubernetes.io/limit and ovn.kubernetes.io/loss \u3002 ovn.kubernetes.io/latency : Set the Pod traffic delay to an integer value in ms. ovn.kubernetes.io/limit \uff1a Set the maximum number of packets that the qdisc queue can hold, and takes an integer value, such as 1000. ovn.kubernetes.io/loss \uff1a Set packet loss probability, the value is float type, for example, the value is 20, then it is set 20% packet loss probability. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"linux-netem QoS"},{"location":"en/guide/setup-options/","text":"Installation and Configuration Options \u00b6 In One-Click Installation we use the default configuration for installation. Kube-OVN also supports more custom configurations, which can be configured in the installation script, or later by changing the parameters of individual components. This document will describe what these customization options do, and how to configure them. Built-in Network Settings \u00b6 Kube-OVN will configure two built-in Subnets during installation: default Subnet, as the default subnet used by the Pod to assign IPs, with a default CIDR of 10.16.0.0/16 and a gateway of 10.16.0.1 . The join subnet, as a special subnet for network communication between the Node and Pod, has a default CIDR of 100.64.0.0/16 and a gateway of 100.64.0.1 . The configuration of these two subnets can be changed during installation via the installation scripts variables: POD_CIDR = \"10.16.0.0/16\" POD_GATEWAY = \"10.16.0.1\" JOIN_CIDR = \"100.64.0.0/16\" EXCLUDE_IPS = \"\" EXCLUDE_IP sets the address range for which kube-ovn-controller will not automatically assign from it, the format is: 192.168.10.20..192.168.10.30 . Note that in the Overlay case these two Subnets CIDRs cannot conflict with existing host networks and Service CIDRs. You can change the address range of both Subnets after installation by referring to Change Subnet CIDR and Change Join Subnet CIDR . Config Service CIDR \u00b6 Since some of the iptables and routing rules set by kube-proxy will conflict with the rules set by Kube-OVN, Kube-OVN needs to know the CIDR of the service to set the corresponding rules correctly. This can be done by modifying the installation script: SVC_CIDR = \"10.96.0.0/12\" You can also modify the args of the kube-ovn-controller Deployment after installation: args : - --service-cluster-ip-range=10.96.0.0/12 Overlay NIC Selection \u00b6 In the case of multiple NICs on a node, Kube-OVN will select the NIC corresponding to the Kubernetes Node IP as the NIC for cross-node communication between containers and establish the corresponding tunnel. If you need to select another NIC to create a container tunnel, you can change it in the installation script: IFACE = eth1 This option supports regular expressions separated by commas, e.g. 'ens[a-z0-9] ,eth[a-z0-9] '. It can also be adjusted after installation by modifying the args of the kube-ovn-cni DaemonSet: args : - --iface=eth1 If each machine has a different NIC name and there is no fixed pattern, you can use the node annotation ovn.kubernetes.io/tunnel_interface to configure each node one by one. This annotation will override the configuration of iface . kubectl annotate node no1 ovn.kubernetes.io/tunnel_interface = ethx Config MTU \u00b6 Since Overlay encapsulation requires additional space, Kube-OVN will adjust the MTU of the container NIC based on the MTU of the selected NIC when creating the container NIC. By default, the Pod NIC MTU is the host NIC MTU - 100 on the Overlay Subnet, and the Pod NIC and host NIC have the same MTU on the Underlay Subnet. If you need to adjust the size of the MTU under the Overlay subnet, you can modify the parameters of the kube-ovn-cni DaemonSet: args : - --mtu=1333 Global Traffic Mirroring Setting \u00b6 When global traffic mirroring is enabled, Kube-OVN will create a mirror0 virtual NIC on each node and copy all container network traffic from the current machine to that NIC\uff0c Users can perform traffic analysis with tcpdump and other tools. This function can be enabled in the installation script: ENABLE_MIRROR = true It can also be adjusted after installation by modifying the args of the kube-ovn-cni DaemonSet: args : - --enable-mirror=true The ability to mirror traffic is disabled in the default installation, if you need fine-grained traffic mirroring or need to mirror traffic to additional NICs please refer to Traffic Mirror . LB Settings \u00b6 Kube-OVN uses L2 LB in OVN to implement service forwarding. In Overlay scenarios, users can choose to use kube-proxy for service traffic forwarding, in which case the LB function of Kube-OVN can be disabled to achieve better performance on the control plane and data plane. This feature can be configured in the installation script: ENABLE_LB = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-lb=false The LB feature is enabled in the default installation. The spec field enableLb has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the LB function of Kube-OVN to the subnet level. You can set whether to enable the LB function based on different subnets. The enable-lb parameter in the kube-ovn-controller deployment is used as a global switch to control whether to create a load-balancer record. The enableLb parameter added in the subnet is used to control whether the subnet is associated with a load-balancer record. After the previous version is upgraded to v1.12.0, the enableLb parameter of the subnet will automatically inherit the value of the original global switch parameter. NetworkPolicy Settings \u00b6 Kube-OVN uses ACLs in OVN to implement NetworkPolicy. Users can choose to disable the NetworkPolicy feature or use the Cilium Chain approach to implement NetworkPolicy using eBPF. In this case, the NetworkPolicy feature of Kube-OVN can be disabled to achieve better performance on the control plane and data plane. This feature can be configured in the installation script: ENABLE_NP = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-np=false NetworkPolicy is enabled by default. EIP and SNAT Settings \u00b6 If the EIP and SNAT capabilities are not required on the default VPC, users can choose to disable them to reduce the performance overhead of kube-ovn-controller in large scale cluster environments and improve processing speed. This feature can be configured in the installation script: ENABLE_EIP_SNAT = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-eip-snat=false EIP and SNAT is enabled by default. More information can refer to EIP and SNAT \u3002 Centralized Gateway ECMP Settings \u00b6 The centralized gateway supports two mode of high availability, primary-backup and ECMP. If you want to enable ECMP mode, you need to change the args of kube-ovn-controller Deployment: args : - --enable-ecmp=true Centralized gateway default installation under the primary-backup mode, more gateway-related content please refer to Config Subnet . The spec field enableEcmp has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the ECMP switch to the subnet level. You can set whether to enable ECMP mode based on different subnets. The enable-ecmp parameter in the kube-ovn-controller deployment is no longer used. After the previous version is upgraded to v1.12.0, the subnet switch will automatically inherit the value of the original global switch parameter. Kubevirt VM Fixed Address Settings \u00b6 For VM instances created by Kubevirt, kube-ovn-controller can assign and manage IP addresses in a similar way to the StatefulSet Pod. This allows VM instances address fixed during start-up, shutdown, upgrade, migration, and other operations throughout their lifecycle, making them more compatible with the actual virtualization user experience. This feature is enable by default after v1.10.6. To disable this feature, you need to change the following args in the kube-ovn-controller Deployment: args : - --keep-vm-ip=false CNI Settings \u00b6 KBy default, Kube-OVN installs the CNI binary in the /opt/cni/bin directory and the CNI configuration file 01-kube-ovn.conflist in the /etc/cni/net.d directory. If you need to change the installation location and the priority of the CNI configuration file, you can modify the following parameters of the installation script. CNI_CONF_DIR = \"/etc/cni/net.d\" CNI_BIN_DIR = \"/opt/cni/bin\" CNI_CONFIG_PRIORITY = \"01\" Or change the Volume mount and args of the kube-ovn-cni DaemonSet after installation: volumes : - name : cni-conf hostPath : path : \"/etc/cni/net.d\" - name : cni-bin hostPath : path:\"/opt/cni/bin\" ... args : - --cni-conf-name=01-kube-ovn.conflist Tunnel Type Settings \u00b6 The default encapsulation mode of Kube-OVN Overlay is Geneve, if you want to change it to Vxlan or STT, please adjust the following parameters in the installation script: TUNNEL_TYPE = \"vxlan\" Or change the environment variables of ovs-ovn DaemonSet after installation: env : - name : TUNNEL_TYPE value : \"vxlan\" If you need to use the STT tunnel and need to compile additional kernel modules for ovs, please refer to Performance Tunning \u3002 Please refer to Tunneling Protocol Selection for the differences between the different protocols in practice. SSL Settings \u00b6 The OVN DB API interface supports SSL encryption to secure the connection. To enable it, adjust the following parameters in the installation script: ENABLE_SSL = true The SSL is disabled by default. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Installation and Configuration Options"},{"location":"en/guide/setup-options/#installation-and-configuration-options","text":"In One-Click Installation we use the default configuration for installation. Kube-OVN also supports more custom configurations, which can be configured in the installation script, or later by changing the parameters of individual components. This document will describe what these customization options do, and how to configure them.","title":"Installation and Configuration Options"},{"location":"en/guide/setup-options/#built-in-network-settings","text":"Kube-OVN will configure two built-in Subnets during installation: default Subnet, as the default subnet used by the Pod to assign IPs, with a default CIDR of 10.16.0.0/16 and a gateway of 10.16.0.1 . The join subnet, as a special subnet for network communication between the Node and Pod, has a default CIDR of 100.64.0.0/16 and a gateway of 100.64.0.1 . The configuration of these two subnets can be changed during installation via the installation scripts variables: POD_CIDR = \"10.16.0.0/16\" POD_GATEWAY = \"10.16.0.1\" JOIN_CIDR = \"100.64.0.0/16\" EXCLUDE_IPS = \"\" EXCLUDE_IP sets the address range for which kube-ovn-controller will not automatically assign from it, the format is: 192.168.10.20..192.168.10.30 . Note that in the Overlay case these two Subnets CIDRs cannot conflict with existing host networks and Service CIDRs. You can change the address range of both Subnets after installation by referring to Change Subnet CIDR and Change Join Subnet CIDR .","title":"Built-in Network Settings"},{"location":"en/guide/setup-options/#config-service-cidr","text":"Since some of the iptables and routing rules set by kube-proxy will conflict with the rules set by Kube-OVN, Kube-OVN needs to know the CIDR of the service to set the corresponding rules correctly. This can be done by modifying the installation script: SVC_CIDR = \"10.96.0.0/12\" You can also modify the args of the kube-ovn-controller Deployment after installation: args : - --service-cluster-ip-range=10.96.0.0/12","title":"Config Service CIDR"},{"location":"en/guide/setup-options/#overlay-nic-selection","text":"In the case of multiple NICs on a node, Kube-OVN will select the NIC corresponding to the Kubernetes Node IP as the NIC for cross-node communication between containers and establish the corresponding tunnel. If you need to select another NIC to create a container tunnel, you can change it in the installation script: IFACE = eth1 This option supports regular expressions separated by commas, e.g. 'ens[a-z0-9] ,eth[a-z0-9] '. It can also be adjusted after installation by modifying the args of the kube-ovn-cni DaemonSet: args : - --iface=eth1 If each machine has a different NIC name and there is no fixed pattern, you can use the node annotation ovn.kubernetes.io/tunnel_interface to configure each node one by one. This annotation will override the configuration of iface . kubectl annotate node no1 ovn.kubernetes.io/tunnel_interface = ethx","title":"Overlay NIC Selection"},{"location":"en/guide/setup-options/#config-mtu","text":"Since Overlay encapsulation requires additional space, Kube-OVN will adjust the MTU of the container NIC based on the MTU of the selected NIC when creating the container NIC. By default, the Pod NIC MTU is the host NIC MTU - 100 on the Overlay Subnet, and the Pod NIC and host NIC have the same MTU on the Underlay Subnet. If you need to adjust the size of the MTU under the Overlay subnet, you can modify the parameters of the kube-ovn-cni DaemonSet: args : - --mtu=1333","title":"Config MTU"},{"location":"en/guide/setup-options/#global-traffic-mirroring-setting","text":"When global traffic mirroring is enabled, Kube-OVN will create a mirror0 virtual NIC on each node and copy all container network traffic from the current machine to that NIC\uff0c Users can perform traffic analysis with tcpdump and other tools. This function can be enabled in the installation script: ENABLE_MIRROR = true It can also be adjusted after installation by modifying the args of the kube-ovn-cni DaemonSet: args : - --enable-mirror=true The ability to mirror traffic is disabled in the default installation, if you need fine-grained traffic mirroring or need to mirror traffic to additional NICs please refer to Traffic Mirror .","title":"Global Traffic Mirroring Setting"},{"location":"en/guide/setup-options/#lb-settings","text":"Kube-OVN uses L2 LB in OVN to implement service forwarding. In Overlay scenarios, users can choose to use kube-proxy for service traffic forwarding, in which case the LB function of Kube-OVN can be disabled to achieve better performance on the control plane and data plane. This feature can be configured in the installation script: ENABLE_LB = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-lb=false The LB feature is enabled in the default installation. The spec field enableLb has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the LB function of Kube-OVN to the subnet level. You can set whether to enable the LB function based on different subnets. The enable-lb parameter in the kube-ovn-controller deployment is used as a global switch to control whether to create a load-balancer record. The enableLb parameter added in the subnet is used to control whether the subnet is associated with a load-balancer record. After the previous version is upgraded to v1.12.0, the enableLb parameter of the subnet will automatically inherit the value of the original global switch parameter.","title":"LB Settings"},{"location":"en/guide/setup-options/#networkpolicy-settings","text":"Kube-OVN uses ACLs in OVN to implement NetworkPolicy. Users can choose to disable the NetworkPolicy feature or use the Cilium Chain approach to implement NetworkPolicy using eBPF. In this case, the NetworkPolicy feature of Kube-OVN can be disabled to achieve better performance on the control plane and data plane. This feature can be configured in the installation script: ENABLE_NP = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-np=false NetworkPolicy is enabled by default.","title":"NetworkPolicy Settings"},{"location":"en/guide/setup-options/#eip-and-snat-settings","text":"If the EIP and SNAT capabilities are not required on the default VPC, users can choose to disable them to reduce the performance overhead of kube-ovn-controller in large scale cluster environments and improve processing speed. This feature can be configured in the installation script: ENABLE_EIP_SNAT = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-eip-snat=false EIP and SNAT is enabled by default. More information can refer to EIP and SNAT \u3002","title":"EIP and SNAT Settings"},{"location":"en/guide/setup-options/#centralized-gateway-ecmp-settings","text":"The centralized gateway supports two mode of high availability, primary-backup and ECMP. If you want to enable ECMP mode, you need to change the args of kube-ovn-controller Deployment: args : - --enable-ecmp=true Centralized gateway default installation under the primary-backup mode, more gateway-related content please refer to Config Subnet . The spec field enableEcmp has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the ECMP switch to the subnet level. You can set whether to enable ECMP mode based on different subnets. The enable-ecmp parameter in the kube-ovn-controller deployment is no longer used. After the previous version is upgraded to v1.12.0, the subnet switch will automatically inherit the value of the original global switch parameter.","title":"Centralized Gateway ECMP Settings"},{"location":"en/guide/setup-options/#kubevirt-vm-fixed-address-settings","text":"For VM instances created by Kubevirt, kube-ovn-controller can assign and manage IP addresses in a similar way to the StatefulSet Pod. This allows VM instances address fixed during start-up, shutdown, upgrade, migration, and other operations throughout their lifecycle, making them more compatible with the actual virtualization user experience. This feature is enable by default after v1.10.6. To disable this feature, you need to change the following args in the kube-ovn-controller Deployment: args : - --keep-vm-ip=false","title":"Kubevirt VM Fixed Address Settings"},{"location":"en/guide/setup-options/#cni-settings","text":"KBy default, Kube-OVN installs the CNI binary in the /opt/cni/bin directory and the CNI configuration file 01-kube-ovn.conflist in the /etc/cni/net.d directory. If you need to change the installation location and the priority of the CNI configuration file, you can modify the following parameters of the installation script. CNI_CONF_DIR = \"/etc/cni/net.d\" CNI_BIN_DIR = \"/opt/cni/bin\" CNI_CONFIG_PRIORITY = \"01\" Or change the Volume mount and args of the kube-ovn-cni DaemonSet after installation: volumes : - name : cni-conf hostPath : path : \"/etc/cni/net.d\" - name : cni-bin hostPath : path:\"/opt/cni/bin\" ... args : - --cni-conf-name=01-kube-ovn.conflist","title":"CNI Settings"},{"location":"en/guide/setup-options/#tunnel-type-settings","text":"The default encapsulation mode of Kube-OVN Overlay is Geneve, if you want to change it to Vxlan or STT, please adjust the following parameters in the installation script: TUNNEL_TYPE = \"vxlan\" Or change the environment variables of ovs-ovn DaemonSet after installation: env : - name : TUNNEL_TYPE value : \"vxlan\" If you need to use the STT tunnel and need to compile additional kernel modules for ovs, please refer to Performance Tunning \u3002 Please refer to Tunneling Protocol Selection for the differences between the different protocols in practice.","title":"Tunnel Type Settings"},{"location":"en/guide/setup-options/#ssl-settings","text":"The OVN DB API interface supports SSL encryption to secure the connection. To enable it, adjust the following parameters in the installation script: ENABLE_SSL = true The SSL is disabled by default. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"SSL Settings"},{"location":"en/guide/static-ip-mac/","text":"Fixed Addresses \u00b6 By default, Kube-OVN randomly assigns IPs and Macs based on the Subnet to which the Pod's Namespace belongs. For workloads that require fixed addresses, Kube-OVN provides multiple methods of fixing addresses depending on the scenario. Single Pod fixed IP/Mac. Workload IP Pool to specify fixed addresses. StatefulSet fixed address. KubeVirt VM fixed address. Single Pod Fixed IP/Mac \u00b6 You can specify the IP/Mac required for the Pod by annotation when creating the Pod. The kube-ovn-controller will skip the address random assignment phase and use the specified address directly after conflict detection, as follows: apiVersion : apps/v1 kind : Deployment metadata : name : ippool labels : app : ippool spec : replicas : 2 selector : matchLabels : app : ippool template : metadata : labels : app : ippool annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u5206\u53f7\u8fdb\u884c\u5206\u9694 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::0 spec : containers : - name : ippool image : docker.io/library/nginx:alpine The following points need to be noted when using annotation. The IP/Mac used cannot conflict with an existing IP/Mac. The IP must be in the CIDR range of the Subnet it belongs to. You can specify only IP or Mac. When you specify only one, the other one will be assigned randomly. Workload IP Pool \u00b6 Kube-OVN supports setting fixed IPs for Workloads (Deployment/StatefulSet/DaemonSet/Job/CronJob) via annotation ovn.kubernetes.io/ip_pool . kube-ovn-controllerr will automatically select the IP specified in ovn.kubernetes.io/ip_pool and perform conflict detection. The Annotation of the IP Pool needs to be added to the annotation field in the template . In addition to Kubernetes built-in workload types, other user-defined workloads can also be assigned fixed addresses using the same approach. Deployment With Fixed IPs \u00b6 apiVersion : apps/v1 kind : Deployment metadata : namespace : ls1 name : starter-backend labels : app : starter-backend spec : replicas : 2 selector : matchLabels : app : starter-backend template : metadata : labels : app : starter-backend annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // for dualstack ippool use semicolon to separate addresses 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::000F;10.16.0.17,fd00:10:16::0010 spec : containers : - name : backend image : docker.io/library/nginx:alpine Using a fixed IP for Workload requires the following: The IP in ovn.kubernetes.io/ip_pool should belong to the CIDR of the Subnet. The IP in ovn.kubernetes.io/ip_pool cannot conflict with an IP already in use. When the number of IPs in ovn.kubernetes.io/ip_pool is less than the number of replicas, the extra Pods will not be created. You need to adjust the number of IPs in ovn.kubernetes.io/ip_pool according to the update policy of the workload and the scaling plan. StatefulSet Fixed Address \u00b6 StatefulSet, like other workloads, can use ovn.kubernetes.io/ip_pool to specify the IP used by the Pod. Since StatefulSet is mostly used for stateful services, which have higher requirements for fixed addresses, Kube-OVN has made special enhancements: Pods are assigned IPs in ovn.kubernetes.io/ip_pool in order. For example, if the name of the StatefulSet is web, web-0 will use the first IP in ovn.kubernetes.io/ip_pool , web-1 will use the second IP, and so on. The logical_switch_port in the OVN is not deleted during update or deletion of the StatefulSet Pod, and the newly generated Pod directly reuses the old logical port information. Pods can therefore reuse IP/Mac and other network information to achieve similar state retention as StatefulSet Volumes. Based on the capabilities of 2, for StatefulSet without the ovn.kubernetes.io/ip_pool annotation, a Pod is randomly assigned an IP/Mac when it is first generated, and then the network information remains fixed for the lifetime of the StatefulSet. StatefulSet Example \u00b6 apiVersion : apps/v1 kind : StatefulSet metadata : name : web spec : serviceName : \"nginx\" replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx spec : containers : - name : nginx image : docker.io/library/nginx:alpine ports : - containerPort : 80 name : web You can try to delete the Pod under StatefulSet to observe if the Pod IP changes. KubeVirt VM Fixed Address \u00b6 For VM instances created by KubeVirt, kube-ovn-controller can assign and manage IP addresses in a similar way to the StatefulSet Pod. This allows VM instances address fixed during start-up, shutdown, upgrade, migration, and other operations throughout their lifecycle, making them more compatible with the actual virtualization user experience. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Fixed Addresses"},{"location":"en/guide/static-ip-mac/#fixed-addresses","text":"By default, Kube-OVN randomly assigns IPs and Macs based on the Subnet to which the Pod's Namespace belongs. For workloads that require fixed addresses, Kube-OVN provides multiple methods of fixing addresses depending on the scenario. Single Pod fixed IP/Mac. Workload IP Pool to specify fixed addresses. StatefulSet fixed address. KubeVirt VM fixed address.","title":"Fixed Addresses"},{"location":"en/guide/static-ip-mac/#single-pod-fixed-ipmac","text":"You can specify the IP/Mac required for the Pod by annotation when creating the Pod. The kube-ovn-controller will skip the address random assignment phase and use the specified address directly after conflict detection, as follows: apiVersion : apps/v1 kind : Deployment metadata : name : ippool labels : app : ippool spec : replicas : 2 selector : matchLabels : app : ippool template : metadata : labels : app : ippool annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u5206\u53f7\u8fdb\u884c\u5206\u9694 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::0 spec : containers : - name : ippool image : docker.io/library/nginx:alpine The following points need to be noted when using annotation. The IP/Mac used cannot conflict with an existing IP/Mac. The IP must be in the CIDR range of the Subnet it belongs to. You can specify only IP or Mac. When you specify only one, the other one will be assigned randomly.","title":"Single Pod Fixed IP/Mac"},{"location":"en/guide/static-ip-mac/#workload-ip-pool","text":"Kube-OVN supports setting fixed IPs for Workloads (Deployment/StatefulSet/DaemonSet/Job/CronJob) via annotation ovn.kubernetes.io/ip_pool . kube-ovn-controllerr will automatically select the IP specified in ovn.kubernetes.io/ip_pool and perform conflict detection. The Annotation of the IP Pool needs to be added to the annotation field in the template . In addition to Kubernetes built-in workload types, other user-defined workloads can also be assigned fixed addresses using the same approach.","title":"Workload IP Pool"},{"location":"en/guide/static-ip-mac/#deployment-with-fixed-ips","text":"apiVersion : apps/v1 kind : Deployment metadata : namespace : ls1 name : starter-backend labels : app : starter-backend spec : replicas : 2 selector : matchLabels : app : starter-backend template : metadata : labels : app : starter-backend annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // for dualstack ippool use semicolon to separate addresses 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::000F;10.16.0.17,fd00:10:16::0010 spec : containers : - name : backend image : docker.io/library/nginx:alpine Using a fixed IP for Workload requires the following: The IP in ovn.kubernetes.io/ip_pool should belong to the CIDR of the Subnet. The IP in ovn.kubernetes.io/ip_pool cannot conflict with an IP already in use. When the number of IPs in ovn.kubernetes.io/ip_pool is less than the number of replicas, the extra Pods will not be created. You need to adjust the number of IPs in ovn.kubernetes.io/ip_pool according to the update policy of the workload and the scaling plan.","title":"Deployment With Fixed IPs"},{"location":"en/guide/static-ip-mac/#statefulset-fixed-address","text":"StatefulSet, like other workloads, can use ovn.kubernetes.io/ip_pool to specify the IP used by the Pod. Since StatefulSet is mostly used for stateful services, which have higher requirements for fixed addresses, Kube-OVN has made special enhancements: Pods are assigned IPs in ovn.kubernetes.io/ip_pool in order. For example, if the name of the StatefulSet is web, web-0 will use the first IP in ovn.kubernetes.io/ip_pool , web-1 will use the second IP, and so on. The logical_switch_port in the OVN is not deleted during update or deletion of the StatefulSet Pod, and the newly generated Pod directly reuses the old logical port information. Pods can therefore reuse IP/Mac and other network information to achieve similar state retention as StatefulSet Volumes. Based on the capabilities of 2, for StatefulSet without the ovn.kubernetes.io/ip_pool annotation, a Pod is randomly assigned an IP/Mac when it is first generated, and then the network information remains fixed for the lifetime of the StatefulSet.","title":"StatefulSet Fixed Address"},{"location":"en/guide/static-ip-mac/#statefulset-example","text":"apiVersion : apps/v1 kind : StatefulSet metadata : name : web spec : serviceName : \"nginx\" replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx spec : containers : - name : nginx image : docker.io/library/nginx:alpine ports : - containerPort : 80 name : web You can try to delete the Pod under StatefulSet to observe if the Pod IP changes.","title":"StatefulSet Example"},{"location":"en/guide/static-ip-mac/#kubevirt-vm-fixed-address","text":"For VM instances created by KubeVirt, kube-ovn-controller can assign and manage IP addresses in a similar way to the StatefulSet Pod. This allows VM instances address fixed during start-up, shutdown, upgrade, migration, and other operations throughout their lifecycle, making them more compatible with the actual virtualization user experience. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"KubeVirt VM Fixed Address"},{"location":"en/guide/subnet/","text":"Config Subnet \u00b6 Subnet is a core concept and basic unit of use in Kube-OVN, and Kube-OVN organizes IP and network configuration in terms of Subnet. Each Namespace can belong to a specific Subnet, and Pods under the Namespace automatically obtain IPs from the Subnet they belong to and share the network configuration (CIDR, gateway type, access control, NAT control, etc.). Unlike other CNI implementations where each node is bound to a subnet, in Kube-OVN the Subnet is a global level virtual network configuration, and the addresses of one Subnet can be distributed on any node. There are some differences in the usage and configuration of Overlay and Underlay Subnets, and this document will describe the common configurations and differentiated features of the different types of Subnets. Default Subnet \u00b6 To make it easier for users to get started quickly, Kube-OVN has a built-in default Subnet, all Namespaces that do not explicitly declare subnet affiliation are automatically assigned IPs from the default subnet and the network information. The configuration of this Subnet is specified at installation time, you can refer to Built-in Network Settings for more details. To change the CIDR of the default Subnet after installation please refer to Change Subnet CIDR . In Overlay mode, the default Subnet uses a distributed gateway and NAT translation for outbound traffic, which behaves much the same as the Flannel's default behavior, allowing users to use most of the network features without additional configuration. In Underlay mode, the default Subnet uses the physical gateway as the outgoing gateway and enables arping to check network connectivity. Check the Default Subnet \u00b6 The default field in the default Subnet spec is set to true , and there is only one default Subnet in a cluster, named ovn-default . # kubectl get subnet ovn-default -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: ovn-default resourceVersion: \"1571334\" selfLink: /apis/kubeovn.io/v1/subnets/ovn-default uid: 7e2451f8-fb44-4f7f-b3e0-cfd27f6fd5d6 spec: cidrBlock: 10 .16.0.0/16 default: true excludeIps: - 10 .16.0.1 gateway: 10 .16.0.1 gatewayType: distributed natOutgoing: true private: false protocol: IPv4 Join Subnet \u00b6 In the Kubernetes network specification, it is required that Nodes can communicate directly with all Pods. To achieve this in Overlay network mode, Kube-OVN creates a join Subnet and creates a virtual NIC ovn0 at each node that connect to the join subnet, through which the nodes and Pods can communicate with each other. The configuration of this Subnet is specified at installation time, you can refer to Built-in Network Settings for more details. To change the CIDR of the Join Subnet after installation please refer to Change Join CIDR . Check the Join Subnet \u00b6 The default name of this subnet is join . There is generally no need to make changes to the network configuration except the CIDR. # kubectl get subnet join -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: join resourceVersion: \"1571333\" selfLink: /apis/kubeovn.io/v1/subnets/join uid: 9c744810-c678-4d50-8a7d-b8ec12ef91b8 spec: cidrBlock: 100 .64.0.0/16 default: false excludeIps: - 100 .64.0.1 gateway: 100 .64.0.1 gatewayNode: \"\" gatewayType: \"\" natOutgoing: false private: false protocol: IPv4 Check the ovn0 NIC at the node: # ifconfig ovn0 ovn0: flags = 4163 <UP,BROADCAST,RUNNING,MULTICAST> mtu 1420 inet 100 .64.0.4 netmask 255 .255.0.0 broadcast 100 .64.255.255 inet6 fe80::800:ff:fe40:5 prefixlen 64 scopeid 0x20<link> ether 0a:00:00:40:00:05 txqueuelen 1000 ( Ethernet ) RX packets 18 bytes 1428 ( 1 .3 KiB ) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 19 bytes 1810 ( 1 .7 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Create Custom Subnets \u00b6 Here we describe the basic operation of how to create a Subnet and associate it with a Namespace, for more advanced configuration, please refer to the subsequent content. Create Subnet \u00b6 cat <<EOF | kubectl create -f - apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 10.66.0.0/16 excludeIps: - 10.66.0.1..10.66.0.10 - 10.66.0.101..10.66.0.151 gateway: 10.66.0.1 gatewayType: distributed natOutgoing: true routeTable: \"\" namespaces: - ns1 - ns2 EOF cidrBlock : Subnet CIDR range, different Subnet CIDRs under the same VPC cannot overlap. excludeIps : The address list is reserved so that the container network will not automatically assign addresses in the list, which can be used as a fixed IP address assignment segment or to avoid conflicts with existing devices in the physical network in Underlay mode. gateway \uff1aFor this subnet gateway address, Kube-OVN will automatically assign the corresponding logical gateway in Overlay mode, and the address should be the underlying physical gateway address in Underlay mode. namespaces : Bind the list of Namespace for this Subnet. Pods under the Namespace will be assigned addresses from the current Subnet after binding. routeTable : Associate the route table, default is main table, route table definition please defer to Static Routes Create Pod in the Subnet \u00b6 # kubectl create ns ns1 namespace/ns1 created # kubectl run nginx --image=docker.io/library/nginx:alpine -n ns1 deployment.apps/nginx created # kubectl get pod -n ns1 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-74d5899f46-n8wtg 1 /1 Running 0 10s 10 .66.0.11 node1 <none> <none> Overlay Subnet Gateway Settings \u00b6 This feature only works for Overlay mode Subnets, Underlay type Subnets need to use the underlying physical gateway to access the external network. Pods under the Overlay Subnet need to access the external network through a gateway, and Kube-OVN currently supports two types of gateways: distributed gateway and centralized gateway which can be changed in the Subnet spec. Both types of gateways support the natOutgoing setting, which allows the user to choose whether snat is required when the Pod accesses the external network. Distributed Gateway \u00b6 The default type of gateway for the Subnet, each node will act as a gateway for the pod on the current node to access the external network. The packets from container will flow into the host network stack from the local ovn0 NIC, and then forwarding the network according to the host's routing rules. When natOutgoing is true , the Pod will use the IP of the current host when accessing the external network. Example of a Subnet, where the gatewayType field is distributed : apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : distributed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : true Centralized Gateway \u00b6 If you want traffic within the Subnet to access the external network using a fixed IP for security operations such as auditing and whitelisting, you can set the gateway type in the Subnet to centralized. In centralized gateway mode, packets from Pods accessing the external network are first routed to the ovn0 NIC of a specific nodes, and then outbound through the host's routing rules. When natOutgoing is true , the Pod will use the IP of a specific nodes when accessing the external network. The centralized gateway example is as follows, where the gatewayType field is centralized and gatewayNode is the NodeName of the particular machine in Kubernetes. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : centralized spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1,node2\" natOutgoing : true If a centralized gateway wants to specify a specific NIC of a machine for outbound networking, gatewayNode format can be changed to kube-ovn-worker:172.18.0.2, kube-ovn-control-plane:172.18.0.3 . The centralized gateway defaults to primary-backup mode, with only the primary node performing traffic forwarding. If you need to switch to ECMP mode, please refer to ECMP Settings . The spec field enableEcmp has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the ECMP switch to the subnet level. You can set whether to enable ECMP mode based on different subnets. The enable-ecmp parameter in the kube-ovn-controller deployment is no longer used. After the previous version is upgraded to v1.12.0, the subnet switch will automatically inherit the value of the original global switch parameter. Subnet ACL \u00b6 For scenarios with fine-grained ACL control, Subnet of Kube-OVN provides ACL to enable fine-grained rules. The ACL rules in Subnet are the same as the ACL rules in OVN, and you can refer to ovn-nb ACL Table for more details. The supported filed in match can refer to ovn-sb Logical Flow Table . Example of an ACL rule that allows Pods with IP address 10.10.0.2 to access all addresses, but does not allow other addresses to access itself, is as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : acl spec : acls : - action : drop direction : to-lport match : ip4.dst == 10.10.0.2 && ip priority : 1002 - action : allow-related direction : from-lport match : ip4.src == 10.10.0.2 && ip priority : 1002 cidrBlock : 10.10.0.0/24 Subnet Isolation \u00b6 The function of Subnet ACL can cover the function of Subnet isolation with better flexibility, we recommend using Subnet ACL to do the corresponding configuration. By default the Subnets created by Kube-OVN can communicate with each other, and Pods can also access external networks through the gateway. To control access between Subnets, set private to true in the subnet spec, and the Subnet will be isolated from other Subnets and external networks and can only communicate within the Subnet. If you want to open a whitelist, you can set it by allowSubnets . The CIDRs in allowSubnets can access the Subnet bidirectionally. Enable Subnet Isolation Examples \u00b6 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : private spec : protocol : IPv4 default : false namespaces : - ns1 - ns2 cidrBlock : 10.69.0.0/16 private : true allowSubnets : - 10.16.0.0/16 - 10.18.0.0/16 Underlay Settings \u00b6 This part of the feature is only available for Underlay type Subnets. vlan : If an Underlay network is used, this field is used to control which Vlan CR the Subnet is bound to. This option defaults to the empty string, meaning that the Underlay network is not used. logicalGateway : Some Underlay environments are pure Layer 2 networks, with no physical Layer 3 gateway. In this case a virtual gateway can be set up with the OVN to connect the Underlay and Overlay networks. The default value is: false . Gateway Check Settings \u00b6 By default kube-ovn-cni will request the gateway using ICMP or ARP protocol after starting the Pod and wait for the return to verify that the network is working properly. Some Underlay environment gateways cannot respond to ARP requests, or scenarios that do not require external connectivity, the checking can be disabled . apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : disable-gw-check spec : disableGatewayCheck : true Other Advanced Settings \u00b6 Configure IPPool Manage QoS Manage Multiple Interface DHCP External Gateway Cluster Inter-Connection with OVN-IC VIP Reservation \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Config Subnet"},{"location":"en/guide/subnet/#config-subnet","text":"Subnet is a core concept and basic unit of use in Kube-OVN, and Kube-OVN organizes IP and network configuration in terms of Subnet. Each Namespace can belong to a specific Subnet, and Pods under the Namespace automatically obtain IPs from the Subnet they belong to and share the network configuration (CIDR, gateway type, access control, NAT control, etc.). Unlike other CNI implementations where each node is bound to a subnet, in Kube-OVN the Subnet is a global level virtual network configuration, and the addresses of one Subnet can be distributed on any node. There are some differences in the usage and configuration of Overlay and Underlay Subnets, and this document will describe the common configurations and differentiated features of the different types of Subnets.","title":"Config Subnet"},{"location":"en/guide/subnet/#default-subnet","text":"To make it easier for users to get started quickly, Kube-OVN has a built-in default Subnet, all Namespaces that do not explicitly declare subnet affiliation are automatically assigned IPs from the default subnet and the network information. The configuration of this Subnet is specified at installation time, you can refer to Built-in Network Settings for more details. To change the CIDR of the default Subnet after installation please refer to Change Subnet CIDR . In Overlay mode, the default Subnet uses a distributed gateway and NAT translation for outbound traffic, which behaves much the same as the Flannel's default behavior, allowing users to use most of the network features without additional configuration. In Underlay mode, the default Subnet uses the physical gateway as the outgoing gateway and enables arping to check network connectivity.","title":"Default Subnet"},{"location":"en/guide/subnet/#check-the-default-subnet","text":"The default field in the default Subnet spec is set to true , and there is only one default Subnet in a cluster, named ovn-default . # kubectl get subnet ovn-default -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: ovn-default resourceVersion: \"1571334\" selfLink: /apis/kubeovn.io/v1/subnets/ovn-default uid: 7e2451f8-fb44-4f7f-b3e0-cfd27f6fd5d6 spec: cidrBlock: 10 .16.0.0/16 default: true excludeIps: - 10 .16.0.1 gateway: 10 .16.0.1 gatewayType: distributed natOutgoing: true private: false protocol: IPv4","title":"Check the Default Subnet"},{"location":"en/guide/subnet/#join-subnet","text":"In the Kubernetes network specification, it is required that Nodes can communicate directly with all Pods. To achieve this in Overlay network mode, Kube-OVN creates a join Subnet and creates a virtual NIC ovn0 at each node that connect to the join subnet, through which the nodes and Pods can communicate with each other. The configuration of this Subnet is specified at installation time, you can refer to Built-in Network Settings for more details. To change the CIDR of the Join Subnet after installation please refer to Change Join CIDR .","title":"Join Subnet"},{"location":"en/guide/subnet/#check-the-join-subnet","text":"The default name of this subnet is join . There is generally no need to make changes to the network configuration except the CIDR. # kubectl get subnet join -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: join resourceVersion: \"1571333\" selfLink: /apis/kubeovn.io/v1/subnets/join uid: 9c744810-c678-4d50-8a7d-b8ec12ef91b8 spec: cidrBlock: 100 .64.0.0/16 default: false excludeIps: - 100 .64.0.1 gateway: 100 .64.0.1 gatewayNode: \"\" gatewayType: \"\" natOutgoing: false private: false protocol: IPv4 Check the ovn0 NIC at the node: # ifconfig ovn0 ovn0: flags = 4163 <UP,BROADCAST,RUNNING,MULTICAST> mtu 1420 inet 100 .64.0.4 netmask 255 .255.0.0 broadcast 100 .64.255.255 inet6 fe80::800:ff:fe40:5 prefixlen 64 scopeid 0x20<link> ether 0a:00:00:40:00:05 txqueuelen 1000 ( Ethernet ) RX packets 18 bytes 1428 ( 1 .3 KiB ) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 19 bytes 1810 ( 1 .7 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0","title":"Check the Join Subnet"},{"location":"en/guide/subnet/#create-custom-subnets","text":"Here we describe the basic operation of how to create a Subnet and associate it with a Namespace, for more advanced configuration, please refer to the subsequent content.","title":"Create Custom Subnets"},{"location":"en/guide/subnet/#create-subnet","text":"cat <<EOF | kubectl create -f - apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 10.66.0.0/16 excludeIps: - 10.66.0.1..10.66.0.10 - 10.66.0.101..10.66.0.151 gateway: 10.66.0.1 gatewayType: distributed natOutgoing: true routeTable: \"\" namespaces: - ns1 - ns2 EOF cidrBlock : Subnet CIDR range, different Subnet CIDRs under the same VPC cannot overlap. excludeIps : The address list is reserved so that the container network will not automatically assign addresses in the list, which can be used as a fixed IP address assignment segment or to avoid conflicts with existing devices in the physical network in Underlay mode. gateway \uff1aFor this subnet gateway address, Kube-OVN will automatically assign the corresponding logical gateway in Overlay mode, and the address should be the underlying physical gateway address in Underlay mode. namespaces : Bind the list of Namespace for this Subnet. Pods under the Namespace will be assigned addresses from the current Subnet after binding. routeTable : Associate the route table, default is main table, route table definition please defer to Static Routes","title":"Create Subnet"},{"location":"en/guide/subnet/#create-pod-in-the-subnet","text":"# kubectl create ns ns1 namespace/ns1 created # kubectl run nginx --image=docker.io/library/nginx:alpine -n ns1 deployment.apps/nginx created # kubectl get pod -n ns1 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-74d5899f46-n8wtg 1 /1 Running 0 10s 10 .66.0.11 node1 <none> <none>","title":"Create Pod in the Subnet"},{"location":"en/guide/subnet/#overlay-subnet-gateway-settings","text":"This feature only works for Overlay mode Subnets, Underlay type Subnets need to use the underlying physical gateway to access the external network. Pods under the Overlay Subnet need to access the external network through a gateway, and Kube-OVN currently supports two types of gateways: distributed gateway and centralized gateway which can be changed in the Subnet spec. Both types of gateways support the natOutgoing setting, which allows the user to choose whether snat is required when the Pod accesses the external network.","title":"Overlay Subnet Gateway Settings"},{"location":"en/guide/subnet/#distributed-gateway","text":"The default type of gateway for the Subnet, each node will act as a gateway for the pod on the current node to access the external network. The packets from container will flow into the host network stack from the local ovn0 NIC, and then forwarding the network according to the host's routing rules. When natOutgoing is true , the Pod will use the IP of the current host when accessing the external network. Example of a Subnet, where the gatewayType field is distributed : apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : distributed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : true","title":"Distributed Gateway"},{"location":"en/guide/subnet/#centralized-gateway","text":"If you want traffic within the Subnet to access the external network using a fixed IP for security operations such as auditing and whitelisting, you can set the gateway type in the Subnet to centralized. In centralized gateway mode, packets from Pods accessing the external network are first routed to the ovn0 NIC of a specific nodes, and then outbound through the host's routing rules. When natOutgoing is true , the Pod will use the IP of a specific nodes when accessing the external network. The centralized gateway example is as follows, where the gatewayType field is centralized and gatewayNode is the NodeName of the particular machine in Kubernetes. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : centralized spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1,node2\" natOutgoing : true If a centralized gateway wants to specify a specific NIC of a machine for outbound networking, gatewayNode format can be changed to kube-ovn-worker:172.18.0.2, kube-ovn-control-plane:172.18.0.3 . The centralized gateway defaults to primary-backup mode, with only the primary node performing traffic forwarding. If you need to switch to ECMP mode, please refer to ECMP Settings . The spec field enableEcmp has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the ECMP switch to the subnet level. You can set whether to enable ECMP mode based on different subnets. The enable-ecmp parameter in the kube-ovn-controller deployment is no longer used. After the previous version is upgraded to v1.12.0, the subnet switch will automatically inherit the value of the original global switch parameter.","title":"Centralized Gateway"},{"location":"en/guide/subnet/#subnet-acl","text":"For scenarios with fine-grained ACL control, Subnet of Kube-OVN provides ACL to enable fine-grained rules. The ACL rules in Subnet are the same as the ACL rules in OVN, and you can refer to ovn-nb ACL Table for more details. The supported filed in match can refer to ovn-sb Logical Flow Table . Example of an ACL rule that allows Pods with IP address 10.10.0.2 to access all addresses, but does not allow other addresses to access itself, is as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : acl spec : acls : - action : drop direction : to-lport match : ip4.dst == 10.10.0.2 && ip priority : 1002 - action : allow-related direction : from-lport match : ip4.src == 10.10.0.2 && ip priority : 1002 cidrBlock : 10.10.0.0/24","title":"Subnet ACL"},{"location":"en/guide/subnet/#subnet-isolation","text":"The function of Subnet ACL can cover the function of Subnet isolation with better flexibility, we recommend using Subnet ACL to do the corresponding configuration. By default the Subnets created by Kube-OVN can communicate with each other, and Pods can also access external networks through the gateway. To control access between Subnets, set private to true in the subnet spec, and the Subnet will be isolated from other Subnets and external networks and can only communicate within the Subnet. If you want to open a whitelist, you can set it by allowSubnets . The CIDRs in allowSubnets can access the Subnet bidirectionally.","title":"Subnet Isolation"},{"location":"en/guide/subnet/#enable-subnet-isolation-examples","text":"apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : private spec : protocol : IPv4 default : false namespaces : - ns1 - ns2 cidrBlock : 10.69.0.0/16 private : true allowSubnets : - 10.16.0.0/16 - 10.18.0.0/16","title":"Enable Subnet Isolation Examples"},{"location":"en/guide/subnet/#underlay-settings","text":"This part of the feature is only available for Underlay type Subnets. vlan : If an Underlay network is used, this field is used to control which Vlan CR the Subnet is bound to. This option defaults to the empty string, meaning that the Underlay network is not used. logicalGateway : Some Underlay environments are pure Layer 2 networks, with no physical Layer 3 gateway. In this case a virtual gateway can be set up with the OVN to connect the Underlay and Overlay networks. The default value is: false .","title":"Underlay Settings"},{"location":"en/guide/subnet/#gateway-check-settings","text":"By default kube-ovn-cni will request the gateway using ICMP or ARP protocol after starting the Pod and wait for the return to verify that the network is working properly. Some Underlay environment gateways cannot respond to ARP requests, or scenarios that do not require external connectivity, the checking can be disabled . apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : disable-gw-check spec : disableGatewayCheck : true","title":"Gateway Check Settings"},{"location":"en/guide/subnet/#other-advanced-settings","text":"Configure IPPool Manage QoS Manage Multiple Interface DHCP External Gateway Cluster Inter-Connection with OVN-IC VIP Reservation \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Other Advanced Settings"},{"location":"en/guide/vpc-qos/","text":"VPC QoS \u00b6 Kube-OVN supports using QoSPolicy CRD to limit the traffic rate of custom VPC. EIP QoS \u00b6 Limit the speed of EIP to 1Mbps and the priority to 1, and shared=false here means that this QoSPolicy can only be used for this EIP and support dynamically modifying QoSPolicy to change QoS rules. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-eip-example spec : shared : false bindingType : EIP bandwidthLimitRules : - name : eip-ingress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : ingress - name : eip-egress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : egress The IptablesEIP configuration is as follows: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-1 spec : natGwDp : gw1 qosPolicy : qos-eip-example The value of .spec.qosPolicy supports being specified during creation and also supports modification after creation. View EIPs with QoS enabled \u00b6 View the corresponding EIPs that have been set up using label : # kubectl get eip -l ovn.kubernetes.io/qos=qos-eip-example NAME IP MAC NAT NATGWDP READY eip-1 172 .18.11.24 00 :00:00:34:41:0B fip gw1 true QoS for VPC NATGW net1 NIC \u00b6 Limit the speed of the net1 NIC on VPC NATGW to 10Mbps and set the priority to 3. Here shared=true , which means that this QoSPolicy can be used by multiple resources at the same time, and does not allow the modification of the contents of the QoSPolicy in this scenario. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-ingress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : ingress - name : net1-egress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : egress The VpcNatGateway configuration is as follows: kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" The value of .spec.qosPolicy supports both creation and subsequent modification. QoS for specific traffic on net1 NIC \u00b6 Limit the specific traffic on net1 NIC to 5Mbps and set the priority to 2. Here shared=true , which means that this QoSPolicy can be used by multiple resources at the same time, and does not allow the modification of the contents of the QoSPolicy in this scenario. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-extip-ingress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : ingress matchType : ip matchValue : src 172.18.11.22/32 - name : net1-extip-egress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : egress matchType : ip matchValue : dst 172.18.11.23/32 The VpcNatGateway configuration is as follows: kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" View NATGWs with QoS enabled \u00b6 View the corresponding NATGWs that have been set up using label : # kubectl get vpc-nat-gw -l ovn.kubernetes.io/qos=qos-natgw-example NAME VPC SUBNET LANIP gw1 test-vpc-1 net1 10 .0.1.254 View QoS rules \u00b6 # kubectl get qos -A NAME SHARED BINDINGTYPE qos-eip-example false EIP qos-natgw-example true NATGW Limitations \u00b6 QoSPolicy can only be deleted when it is not in use. Therefore, before deleting the QoSPolicy, please check the EIP and NATGW that have enabled QoS, and remove their spec.qosPolicy configuration. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC QoS"},{"location":"en/guide/vpc-qos/#vpc-qos","text":"Kube-OVN supports using QoSPolicy CRD to limit the traffic rate of custom VPC.","title":"VPC QoS"},{"location":"en/guide/vpc-qos/#eip-qos","text":"Limit the speed of EIP to 1Mbps and the priority to 1, and shared=false here means that this QoSPolicy can only be used for this EIP and support dynamically modifying QoSPolicy to change QoS rules. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-eip-example spec : shared : false bindingType : EIP bandwidthLimitRules : - name : eip-ingress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : ingress - name : eip-egress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : egress The IptablesEIP configuration is as follows: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-1 spec : natGwDp : gw1 qosPolicy : qos-eip-example The value of .spec.qosPolicy supports being specified during creation and also supports modification after creation.","title":"EIP QoS"},{"location":"en/guide/vpc-qos/#view-eips-with-qos-enabled","text":"View the corresponding EIPs that have been set up using label : # kubectl get eip -l ovn.kubernetes.io/qos=qos-eip-example NAME IP MAC NAT NATGWDP READY eip-1 172 .18.11.24 00 :00:00:34:41:0B fip gw1 true","title":"View EIPs with QoS enabled"},{"location":"en/guide/vpc-qos/#qos-for-vpc-natgw-net1-nic","text":"Limit the speed of the net1 NIC on VPC NATGW to 10Mbps and set the priority to 3. Here shared=true , which means that this QoSPolicy can be used by multiple resources at the same time, and does not allow the modification of the contents of the QoSPolicy in this scenario. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-ingress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : ingress - name : net1-egress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : egress The VpcNatGateway configuration is as follows: kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" The value of .spec.qosPolicy supports both creation and subsequent modification.","title":"QoS for VPC NATGW net1 NIC"},{"location":"en/guide/vpc-qos/#qos-for-specific-traffic-on-net1-nic","text":"Limit the specific traffic on net1 NIC to 5Mbps and set the priority to 2. Here shared=true , which means that this QoSPolicy can be used by multiple resources at the same time, and does not allow the modification of the contents of the QoSPolicy in this scenario. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-extip-ingress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : ingress matchType : ip matchValue : src 172.18.11.22/32 - name : net1-extip-egress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : egress matchType : ip matchValue : dst 172.18.11.23/32 The VpcNatGateway configuration is as follows: kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\"","title":"QoS for specific traffic on net1 NIC"},{"location":"en/guide/vpc-qos/#view-natgws-with-qos-enabled","text":"View the corresponding NATGWs that have been set up using label : # kubectl get vpc-nat-gw -l ovn.kubernetes.io/qos=qos-natgw-example NAME VPC SUBNET LANIP gw1 test-vpc-1 net1 10 .0.1.254","title":"View NATGWs with QoS enabled"},{"location":"en/guide/vpc-qos/#view-qos-rules","text":"# kubectl get qos -A NAME SHARED BINDINGTYPE qos-eip-example false EIP qos-natgw-example true NATGW","title":"View QoS rules"},{"location":"en/guide/vpc-qos/#limitations","text":"QoSPolicy can only be deleted when it is not in use. Therefore, before deleting the QoSPolicy, please check the EIP and NATGW that have enabled QoS, and remove their spec.qosPolicy configuration. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Limitations"},{"location":"en/guide/vpc/","text":"Config VPC \u00b6 Kube-OVN supports multi-tenant isolation level VPC networks. Different VPC networks are independent of each other and can be configured separately with Subnet CIDRs, routing policies, security policies, outbound gateways, EIP, etc. VPC is mainly used in scenarios where there requires strong isolation of multi-tenant networks and some Kubernetes networking features conflict under multi-tenant networks. For example, node and pod access, NodePort functionality, network access-based health checks, and DNS capabilities are not supported in multi-tenant network scenarios at this time. In order to facilitate common Kubernetes usage scenarios, Kube-OVN has a special design for the default VPC where the Subnet under the VPC can meet the Kubernetes specification. The custom VPC supports static routing, EIP and NAT gateways as described in this document. Common isolation requirements can be achieved through network policies and Subnet ACLs under the default VPC, so before using a custom VPC, please make sure whether you need VPC-level isolation and understand the limitations under the custom VPC. For Underlay subnets, physical switches are responsible for data-plane forwarding, so VPCs cannot isolate Underlay subnets. Creating Custom VPCs \u00b6 Create two VPCs: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : namespaces : - ns1 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-2 spec : namespaces : - ns2 namespaces : Limit which namespaces can use this VPC. If empty, all namespaces can use this VPC. Create two Subnets, belonging to two different VPCs and having the same CIDR: kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : test-vpc-1 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns1 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : test-vpc-2 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns2 Create Pods under two separate Namespaces: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 namespace : ns1 name : vpc1-pod spec : containers : - name : vpc1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 namespace : ns2 name : vpc2-pod spec : containers : - name : vpc2-pod image : docker.io/library/nginx:alpine After running successfully, you can observe that the two Pod addresses belong to the same CIDR, but the two Pods cannot access each other because they are running on different tenant VPCs. Custom VPC Pod supports livenessProbe and readinessProbe \u00b6 Since the Pods under the custom VPC do not communicate with the network of the node, the probe packets sent by the kubelet cannot reach the Pods in the custom VPC. Kube-OVN uses TProxy to redirect the detection packets sent by kubelet to Pods in the custom VPC to achieve this function. The configuration method is as follows, add the parameter --enable-tproxy=true in Daemonset kube-ovn-cni : spec : template : spec : containers : - args : - --enable-tproxy=true Restrictions for this feature: When Pods under different VPCs have the same IP under the same node, the detection function fails. Currently, only tcpSocket and httpGet are supported. Create VPC NAT Gateway \u00b6 Subnets under custom VPCs do not support distributed gateways and centralized gateways under default VPCs. Pod access to the external network within the VPC requires a VPC gateway, which bridges the physical and tenant networks and provides floating IP, SNAT and DNAT capabilities. The VPC gateway function relies on Multus-CNI function, please refer to multus-cni . Configuring the External Network \u00b6 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-vpc-external-network spec : protocol : IPv4 provider : ovn-vpc-external-network.kube-system cidrBlock : 192.168.0.0/24 gateway : 192.168.0.1 # IP address of the physical gateway excludeIps : - 192.168.0.1..192.168.0.10 --- apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-vpc-external-network namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth1\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-vpc-external-network.kube-system\" } }' This Subnet is used to manage the available external addresses and the address will be allocated to VPC NAT Gateway through Macvlan, so please communicate with your network management to give you the available physical segment IPs. The VPC gateway uses Macvlan for physical network configuration, and master of NetworkAttachmentDefinition should be the NIC name of the corresponding physical network NIC. name : External network name. For macvlan mode, the nic will send packets directly through that node NIC, relying on the underlying network devices for L2/L3 level forwarding capabilities. You need to configure the corresponding gateway, Vlan and security policy in the underlying network device in advance. For OpenStack VM environments, you need to turn off PortSecurity on the corresponding network port. For VMware vSwitch networks, MAC Address Changes , Forged Transmits and Promiscuous Mode Operation should be set to allow . For Hyper-V virtualization, MAC Address Spoofing should be enabled in VM nic advanced features. Public clouds, such as AWS, GCE, AliCloud, etc., do not support user-defined Mac, so they cannot support Macvlan mode network. Due to the limitations of Macvlan, the Macvlan sub-interface cannot access the parent interface address. If the physical network card corresponds to a switch interface in Trunk mode, a sub-interface needs to be created on the network card and provided to Macvlan for use. Enabling the VPC Gateway \u00b6 VPC gateway functionality needs to be enabled via ovn-vpc-nat-gw-config under kube-system : --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-config namespace : kube-system data : image : docker.io/kubeovn/vpc-nat-gateway:v1.12.0 --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-gw-config namespace : kube-system data : enable-vpc-nat-gw : 'true' image : The image used by the Gateway Pod. enable-vpc-nat-gw : Controls whether the VPC Gateway feature is enabled. Create VPC Gateway and Set the Default Route \u00b6 kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" externalSubnets : - ovn-vpc-external-network vpc : The VPC to which this VpcNatGateway belongs. subnet : A Subnet within the VPC, the VPC Gateway Pod will use lanIp to connect to the tenant network under that subnet. lanIp : An unused IP within the subnet that the VPC Gateway Pod will eventually use for the Pod. When configuring routing for a VPC, the nextHopIP needs to be set to the lanIp of the current VpcNatGateway. selector : The node selector for VpcNatGateway Pod has the same format as NodeSelector in Kubernetes. externalSubnets : External network used by the VPC gateway, if not configured, ovn-vpc-external-network is used by default, and only one external network is supported in the current version. Other configurable parameters: tolerations : Configure tolerance for the VPC gateway. For details, see Taints and Tolerations affinity : Configure affinity for the Pod or node of the VPC gateway. For details, see Assigning Pods to Nodes Create EIP \u00b6 EIP allows for floating IP, SNAT, and DNAT operations after assigning an IP from an external network segment to a VPC gateway. Randomly assign an address to the EIP: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 Fixed EIP address assignment: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : natGwDp : gw1 v4ip : 10.0.1.111 Specify the external network on which the EIP is located: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 externalSubnet : ovn-vpc-external-network externalSubnet : The name of the external network on which the EIP is located. If not specified, it defaults to ovn-vpc-external-network . If specified, it must be one of the externalSubnets of the VPC gateway. Create DNAT Rules \u00b6 kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipd01 spec : natGwDp : gw1 --- kind : IptablesDnatRule apiVersion : kubeovn.io/v1 metadata : name : dnat01 spec : eip : eipd01 externalPort : '8888' internalIp : 10.0.1.10 internalPort : '80' protocol : tcp Create SNAT Rules \u00b6 --- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eips01 spec : natGwDp : gw1 --- kind : IptablesSnatRule apiVersion : kubeovn.io/v1 metadata : name : snat01 spec : eip : eips01 internalCIDR : 10.0.1.0/24 Create Floating IP \u00b6 --- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipf01 spec : natGwDp : gw1 --- kind : IptablesFIPRule apiVersion : kubeovn.io/v1 metadata : name : fip01 spec : eip : eipf01 internalIp : 10.0.1.5 Custom Routing \u00b6 Within the custom VPC, users can customize the routing rules within the VPC and combine it with the gateway for more flexible forwarding. Kube-OVN supports static routes and more flexible policy routes. Static Routes \u00b6 kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : staticRoutes : - cidr : 0.0.0.0/0 nextHopIP : 10.0.1.254 policy : policyDst - cidr : 172.31.0.0/24 nextHopIP : 10.0.1.253 policy : policySrc routeTable : \"rtb1\" policy : Supports destination routing policyDst and source routing policySrc . When there are overlapping routing rules, the rule with the longer CIDR mask has higher priority, and if the mask length is the same, the destination route has a higher priority over the source route. routeTable : You can store the route in specific table, default is main table. Associate with subnet please defer to Create Custom Subnets Policy Routes \u00b6 Traffic matched by static routes can be controlled at a finer granularity by policy routing. Policy routing provides more precise matching rules, priority control and more forwarding actions. This feature brings the OVN internal logical router policy function directly to the outside world, for more information on its use, please refer to Logical Router Policy . An example of policy routes: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : policyRoutes : - action : drop match : ip4.src==10.0.1.0/24 && ip4.dst==10.0.1.250 priority : 11 - action : reroute match : ip4.src==10.0.1.0/24 nextHopIP : 10.0.1.252 priority : 10 Custom vpc-dns \u00b6 Due to the isolation between custom VPCs and default VPC networks, Pods in VPCs cannot use the default coredns service for domain name resolution. If you want to use coredns to resolve Service domain names within the custom VPC, you can use the vpc-dns resource provided by Kube-OVN. Create an Additional Network \u00b6 apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }' Modify the Provider of the ovn-default Logical Switch \u00b6 Modify the provider of ovn-default to the provider ovn-nad.default.ovn configured above in nad\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn vpc : ovn-cluster Modify the vpc-dns ConfigMap \u00b6 Create a ConfigMap in the kube-system namespace, configure the vpc-dns parameters to be used for the subsequent vpc-dns feature activation: apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : true nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns : (optional) true to enable the feature, false to disable the feature. Default true . coredns-image : (optional): DNS deployment image. Default is the cluster coredns deployment version. coredns-template : (optional): URL of the DNS deployment template. Default: yamls/coredns-template.yaml in the current version repository. coredns-vip : VIP providing LB service for coredns. nad-name : Name of the configured network-attachment-definitions resource. nad-provider : Name of the used provider. k8s-service-host : (optional) IP used by coredns to access the k8s apiserver service. k8s-service-port : (optional) Port used by coredns to access the k8s apiserver service. Deploying VPC-DNS Dependent Resources \u00b6 apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } Deploy vpc-dns \u00b6 kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc : The VPC name used to deploy the DNS component. subnet : The subnet name used to deploy the DNS component. View resource information: [ root@hci-dev-mst-1 kubeovn ] # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : if the custom vpc-dns is ready. Restrictions \u00b6 Only one custom DNS component will be deployed in one VPC; When multiple VPC-DNS resources (i.e. different subnets in the same VPC) are configured in one VPC, only one VPC-DNS resource with status true will be active, while the others will be false ; When the true VPC-DNS is deleted, another false VPC-DNS will be deployed. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Config VPC"},{"location":"en/guide/vpc/#config-vpc","text":"Kube-OVN supports multi-tenant isolation level VPC networks. Different VPC networks are independent of each other and can be configured separately with Subnet CIDRs, routing policies, security policies, outbound gateways, EIP, etc. VPC is mainly used in scenarios where there requires strong isolation of multi-tenant networks and some Kubernetes networking features conflict under multi-tenant networks. For example, node and pod access, NodePort functionality, network access-based health checks, and DNS capabilities are not supported in multi-tenant network scenarios at this time. In order to facilitate common Kubernetes usage scenarios, Kube-OVN has a special design for the default VPC where the Subnet under the VPC can meet the Kubernetes specification. The custom VPC supports static routing, EIP and NAT gateways as described in this document. Common isolation requirements can be achieved through network policies and Subnet ACLs under the default VPC, so before using a custom VPC, please make sure whether you need VPC-level isolation and understand the limitations under the custom VPC. For Underlay subnets, physical switches are responsible for data-plane forwarding, so VPCs cannot isolate Underlay subnets.","title":"Config VPC"},{"location":"en/guide/vpc/#creating-custom-vpcs","text":"Create two VPCs: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : namespaces : - ns1 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-2 spec : namespaces : - ns2 namespaces : Limit which namespaces can use this VPC. If empty, all namespaces can use this VPC. Create two Subnets, belonging to two different VPCs and having the same CIDR: kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : test-vpc-1 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns1 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : test-vpc-2 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns2 Create Pods under two separate Namespaces: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 namespace : ns1 name : vpc1-pod spec : containers : - name : vpc1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 namespace : ns2 name : vpc2-pod spec : containers : - name : vpc2-pod image : docker.io/library/nginx:alpine After running successfully, you can observe that the two Pod addresses belong to the same CIDR, but the two Pods cannot access each other because they are running on different tenant VPCs.","title":"Creating Custom VPCs"},{"location":"en/guide/vpc/#custom-vpc-pod-supports-livenessprobe-and-readinessprobe","text":"Since the Pods under the custom VPC do not communicate with the network of the node, the probe packets sent by the kubelet cannot reach the Pods in the custom VPC. Kube-OVN uses TProxy to redirect the detection packets sent by kubelet to Pods in the custom VPC to achieve this function. The configuration method is as follows, add the parameter --enable-tproxy=true in Daemonset kube-ovn-cni : spec : template : spec : containers : - args : - --enable-tproxy=true Restrictions for this feature: When Pods under different VPCs have the same IP under the same node, the detection function fails. Currently, only tcpSocket and httpGet are supported.","title":"Custom VPC Pod supports livenessProbe and readinessProbe"},{"location":"en/guide/vpc/#create-vpc-nat-gateway","text":"Subnets under custom VPCs do not support distributed gateways and centralized gateways under default VPCs. Pod access to the external network within the VPC requires a VPC gateway, which bridges the physical and tenant networks and provides floating IP, SNAT and DNAT capabilities. The VPC gateway function relies on Multus-CNI function, please refer to multus-cni .","title":"Create VPC NAT Gateway"},{"location":"en/guide/vpc/#configuring-the-external-network","text":"apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-vpc-external-network spec : protocol : IPv4 provider : ovn-vpc-external-network.kube-system cidrBlock : 192.168.0.0/24 gateway : 192.168.0.1 # IP address of the physical gateway excludeIps : - 192.168.0.1..192.168.0.10 --- apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-vpc-external-network namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth1\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-vpc-external-network.kube-system\" } }' This Subnet is used to manage the available external addresses and the address will be allocated to VPC NAT Gateway through Macvlan, so please communicate with your network management to give you the available physical segment IPs. The VPC gateway uses Macvlan for physical network configuration, and master of NetworkAttachmentDefinition should be the NIC name of the corresponding physical network NIC. name : External network name. For macvlan mode, the nic will send packets directly through that node NIC, relying on the underlying network devices for L2/L3 level forwarding capabilities. You need to configure the corresponding gateway, Vlan and security policy in the underlying network device in advance. For OpenStack VM environments, you need to turn off PortSecurity on the corresponding network port. For VMware vSwitch networks, MAC Address Changes , Forged Transmits and Promiscuous Mode Operation should be set to allow . For Hyper-V virtualization, MAC Address Spoofing should be enabled in VM nic advanced features. Public clouds, such as AWS, GCE, AliCloud, etc., do not support user-defined Mac, so they cannot support Macvlan mode network. Due to the limitations of Macvlan, the Macvlan sub-interface cannot access the parent interface address. If the physical network card corresponds to a switch interface in Trunk mode, a sub-interface needs to be created on the network card and provided to Macvlan for use.","title":"Configuring the External Network"},{"location":"en/guide/vpc/#enabling-the-vpc-gateway","text":"VPC gateway functionality needs to be enabled via ovn-vpc-nat-gw-config under kube-system : --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-config namespace : kube-system data : image : docker.io/kubeovn/vpc-nat-gateway:v1.12.0 --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-gw-config namespace : kube-system data : enable-vpc-nat-gw : 'true' image : The image used by the Gateway Pod. enable-vpc-nat-gw : Controls whether the VPC Gateway feature is enabled.","title":"Enabling the VPC Gateway"},{"location":"en/guide/vpc/#create-vpc-gateway-and-set-the-default-route","text":"kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" externalSubnets : - ovn-vpc-external-network vpc : The VPC to which this VpcNatGateway belongs. subnet : A Subnet within the VPC, the VPC Gateway Pod will use lanIp to connect to the tenant network under that subnet. lanIp : An unused IP within the subnet that the VPC Gateway Pod will eventually use for the Pod. When configuring routing for a VPC, the nextHopIP needs to be set to the lanIp of the current VpcNatGateway. selector : The node selector for VpcNatGateway Pod has the same format as NodeSelector in Kubernetes. externalSubnets : External network used by the VPC gateway, if not configured, ovn-vpc-external-network is used by default, and only one external network is supported in the current version. Other configurable parameters: tolerations : Configure tolerance for the VPC gateway. For details, see Taints and Tolerations affinity : Configure affinity for the Pod or node of the VPC gateway. For details, see Assigning Pods to Nodes","title":"Create VPC Gateway and Set the Default Route"},{"location":"en/guide/vpc/#create-eip","text":"EIP allows for floating IP, SNAT, and DNAT operations after assigning an IP from an external network segment to a VPC gateway. Randomly assign an address to the EIP: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 Fixed EIP address assignment: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : natGwDp : gw1 v4ip : 10.0.1.111 Specify the external network on which the EIP is located: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 externalSubnet : ovn-vpc-external-network externalSubnet : The name of the external network on which the EIP is located. If not specified, it defaults to ovn-vpc-external-network . If specified, it must be one of the externalSubnets of the VPC gateway.","title":"Create EIP"},{"location":"en/guide/vpc/#create-dnat-rules","text":"kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipd01 spec : natGwDp : gw1 --- kind : IptablesDnatRule apiVersion : kubeovn.io/v1 metadata : name : dnat01 spec : eip : eipd01 externalPort : '8888' internalIp : 10.0.1.10 internalPort : '80' protocol : tcp","title":"Create DNAT Rules"},{"location":"en/guide/vpc/#create-snat-rules","text":"--- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eips01 spec : natGwDp : gw1 --- kind : IptablesSnatRule apiVersion : kubeovn.io/v1 metadata : name : snat01 spec : eip : eips01 internalCIDR : 10.0.1.0/24","title":"Create SNAT Rules"},{"location":"en/guide/vpc/#create-floating-ip","text":"--- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipf01 spec : natGwDp : gw1 --- kind : IptablesFIPRule apiVersion : kubeovn.io/v1 metadata : name : fip01 spec : eip : eipf01 internalIp : 10.0.1.5","title":"Create Floating IP"},{"location":"en/guide/vpc/#custom-routing","text":"Within the custom VPC, users can customize the routing rules within the VPC and combine it with the gateway for more flexible forwarding. Kube-OVN supports static routes and more flexible policy routes.","title":"Custom Routing"},{"location":"en/guide/vpc/#static-routes","text":"kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : staticRoutes : - cidr : 0.0.0.0/0 nextHopIP : 10.0.1.254 policy : policyDst - cidr : 172.31.0.0/24 nextHopIP : 10.0.1.253 policy : policySrc routeTable : \"rtb1\" policy : Supports destination routing policyDst and source routing policySrc . When there are overlapping routing rules, the rule with the longer CIDR mask has higher priority, and if the mask length is the same, the destination route has a higher priority over the source route. routeTable : You can store the route in specific table, default is main table. Associate with subnet please defer to Create Custom Subnets","title":"Static Routes"},{"location":"en/guide/vpc/#policy-routes","text":"Traffic matched by static routes can be controlled at a finer granularity by policy routing. Policy routing provides more precise matching rules, priority control and more forwarding actions. This feature brings the OVN internal logical router policy function directly to the outside world, for more information on its use, please refer to Logical Router Policy . An example of policy routes: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : policyRoutes : - action : drop match : ip4.src==10.0.1.0/24 && ip4.dst==10.0.1.250 priority : 11 - action : reroute match : ip4.src==10.0.1.0/24 nextHopIP : 10.0.1.252 priority : 10","title":"Policy Routes"},{"location":"en/guide/vpc/#custom-vpc-dns","text":"Due to the isolation between custom VPCs and default VPC networks, Pods in VPCs cannot use the default coredns service for domain name resolution. If you want to use coredns to resolve Service domain names within the custom VPC, you can use the vpc-dns resource provided by Kube-OVN.","title":"Custom vpc-dns"},{"location":"en/guide/vpc/#create-an-additional-network","text":"apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }'","title":"Create an Additional Network"},{"location":"en/guide/vpc/#modify-the-provider-of-the-ovn-default-logical-switch","text":"Modify the provider of ovn-default to the provider ovn-nad.default.ovn configured above in nad\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn vpc : ovn-cluster","title":"Modify the Provider of the ovn-default Logical Switch"},{"location":"en/guide/vpc/#modify-the-vpc-dns-configmap","text":"Create a ConfigMap in the kube-system namespace, configure the vpc-dns parameters to be used for the subsequent vpc-dns feature activation: apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : true nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns : (optional) true to enable the feature, false to disable the feature. Default true . coredns-image : (optional): DNS deployment image. Default is the cluster coredns deployment version. coredns-template : (optional): URL of the DNS deployment template. Default: yamls/coredns-template.yaml in the current version repository. coredns-vip : VIP providing LB service for coredns. nad-name : Name of the configured network-attachment-definitions resource. nad-provider : Name of the used provider. k8s-service-host : (optional) IP used by coredns to access the k8s apiserver service. k8s-service-port : (optional) Port used by coredns to access the k8s apiserver service.","title":"Modify the vpc-dns ConfigMap"},{"location":"en/guide/vpc/#deploying-vpc-dns-dependent-resources","text":"apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance }","title":"Deploying VPC-DNS Dependent Resources"},{"location":"en/guide/vpc/#deploy-vpc-dns","text":"kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc : The VPC name used to deploy the DNS component. subnet : The subnet name used to deploy the DNS component. View resource information: [ root@hci-dev-mst-1 kubeovn ] # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : if the custom vpc-dns is ready.","title":"Deploy vpc-dns"},{"location":"en/guide/vpc/#restrictions","text":"Only one custom DNS component will be deployed in one VPC; When multiple VPC-DNS resources (i.e. different subnets in the same VPC) are configured in one VPC, only one VPC-DNS resource with status true will be active, while the others will be false ; When the true VPC-DNS is deleted, another false VPC-DNS will be deployed. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Restrictions"},{"location":"en/guide/webhook/","text":"Webhook \u00b6 Using Webhook, you can verify CRD resources within Kube-OVN. Currently, Webhook mainly performs fixed IP address conflict detection and Subnet CIDR conflict detection, and prompts errors when such conflicts happen. Since Webhook intercepts all Subnet and Pod creation requests, you need to deploy Kube-OVN first and Webhook later. Install Cert-Manager \u00b6 Webhook deployment requires certificate, we use cert-manager to generate the associated certificate, we need to deploy cert-manager before deploying Webhook. You can use the following command to deploy cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml More cert-manager usage please refer to cert-manager document \u3002 Install Webhook \u00b6 Download Webhook yaml and install: # kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/webhook.yaml deployment.apps/kube-ovn-webhook created service/kube-ovn-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/kube-ovn-webhook created certificate.cert-manager.io/kube-ovn-webhook-serving-cert created issuer.cert-manager.io/kube-ovn-webhook-selfsigned-issuer created Verify Webhook Take Effect \u00b6 Check the running Pod and get the Pod IP 10.16.0.15 : # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES static-7584848b74-fw9dm 1 /1 Running 0 2d13h 10 .16.0.15 kube-ovn-worker <none> Write yaml to create a Pod with the same IP: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 labels : app : static managedFields : name : staticip-pod namespace : default spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest When using the above yaml to create a fixed address Pod, it prompts an IP address conflict: # kubectl apply -f pod-static.yaml Error from server ( annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 ) : error when creating \"pod-static.yaml\" : admission webhook \"pod-ip-validaing.kube-ovn.io\" denied the request: annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Webhook"},{"location":"en/guide/webhook/#webhook","text":"Using Webhook, you can verify CRD resources within Kube-OVN. Currently, Webhook mainly performs fixed IP address conflict detection and Subnet CIDR conflict detection, and prompts errors when such conflicts happen. Since Webhook intercepts all Subnet and Pod creation requests, you need to deploy Kube-OVN first and Webhook later.","title":"Webhook"},{"location":"en/guide/webhook/#install-cert-manager","text":"Webhook deployment requires certificate, we use cert-manager to generate the associated certificate, we need to deploy cert-manager before deploying Webhook. You can use the following command to deploy cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml More cert-manager usage please refer to cert-manager document \u3002","title":"Install Cert-Manager"},{"location":"en/guide/webhook/#install-webhook","text":"Download Webhook yaml and install: # kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/webhook.yaml deployment.apps/kube-ovn-webhook created service/kube-ovn-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/kube-ovn-webhook created certificate.cert-manager.io/kube-ovn-webhook-serving-cert created issuer.cert-manager.io/kube-ovn-webhook-selfsigned-issuer created","title":"Install Webhook"},{"location":"en/guide/webhook/#verify-webhook-take-effect","text":"Check the running Pod and get the Pod IP 10.16.0.15 : # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES static-7584848b74-fw9dm 1 /1 Running 0 2d13h 10 .16.0.15 kube-ovn-worker <none> Write yaml to create a Pod with the same IP: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 labels : app : static managedFields : name : staticip-pod namespace : default spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest When using the above yaml to create a fixed address Pod, it prompts an IP address conflict: # kubectl apply -f pod-static.yaml Error from server ( annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 ) : error when creating \"pod-static.yaml\" : admission webhook \"pod-ip-validaing.kube-ovn.io\" denied the request: annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Verify Webhook Take Effect"},{"location":"en/ops/change-default-subnet/","text":"Change Subnet CIDR \u00b6 If a subnet CIDR is created that conflicts or does not meet expectations, it can be modified by following the steps in this document. After modifying the subnet CIDR, the previously created Pods will not be able to access the network properly and need to be rebuilt. Careful consideration is recommended before operating\u3002This document is only for business subnet CIDR changes, if you need to Change the Join subnet CIDR, please refer to Change Join CIDR . Edit Subnet \u00b6 Use kubectl edit to modify cidrBlock \uff0c gateway and excludeIps . kubectl edit subnet test-subnet Rebuild all Pods under this Subnet \u00b6 Take the subnet binding test Namespace as example: for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n test --ignore-not-found done If only the default subnet is used, you can delete all Pods that are not in host network mode using the following command: for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done Change Default Subnet Settings \u00b6 If you are modifying the CIDR for the default Subnet, you also need to change the args of the kube-ovn-controller Deployment: args : - --default-cidr=10.17.0.0/16 - --default-gateway=10.17.0.1 - --default-exclude-ips=10.17.0.1 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Subnet CIDR"},{"location":"en/ops/change-default-subnet/#change-subnet-cidr","text":"If a subnet CIDR is created that conflicts or does not meet expectations, it can be modified by following the steps in this document. After modifying the subnet CIDR, the previously created Pods will not be able to access the network properly and need to be rebuilt. Careful consideration is recommended before operating\u3002This document is only for business subnet CIDR changes, if you need to Change the Join subnet CIDR, please refer to Change Join CIDR .","title":"Change Subnet CIDR"},{"location":"en/ops/change-default-subnet/#edit-subnet","text":"Use kubectl edit to modify cidrBlock \uff0c gateway and excludeIps . kubectl edit subnet test-subnet","title":"Edit Subnet"},{"location":"en/ops/change-default-subnet/#rebuild-all-pods-under-this-subnet","text":"Take the subnet binding test Namespace as example: for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n test --ignore-not-found done If only the default subnet is used, you can delete all Pods that are not in host network mode using the following command: for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done","title":"Rebuild all Pods under this Subnet"},{"location":"en/ops/change-default-subnet/#change-default-subnet-settings","text":"If you are modifying the CIDR for the default Subnet, you also need to change the args of the kube-ovn-controller Deployment: args : - --default-cidr=10.17.0.0/16 - --default-gateway=10.17.0.1 - --default-exclude-ips=10.17.0.1 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Default Subnet Settings"},{"location":"en/ops/change-join-subnet/","text":"Change Join Subnet CIDR \u00b6 If the Join subnet CIDR created conflicts or does not meet expectations, you can use this document to modify. After modifying the Join Subnet CIDR, the previously created Pods will not be able to access the external network normally and need to wait for the rebuild completed. Delete Join Subnet \u00b6 kubectl patch subnet join --type = 'json' -p '[{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\": []}]' kubectl delete subnet join Cleanup Allocated Config \u00b6 kubectl annotate node ovn.kubernetes.io/allocated = false --all --overwrite Modify Join Subnet \u00b6 Change Join Subnet args in kube-ovn-controller : kubectl edit deployment -n kube-system kube-ovn-controller Change the CIDR below: args : - --node-switch-cidr=100.51.0.0/16 Reboot the kube-ovn-controller and rebuild join Subnet: kubectl delete pod -n kube-system -lapp = kube-ovn-controller Check the new Join Subnet information: # kubectl get subnet NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS join ovn ovn-cluster IPv4 100 .51.0.0/16 false false false distributed 2 65531 0 0 [ \"100.51.0.1\" ] ovn-default ovn ovn-cluster IPv4 10 .17.0.0/16 false true true distributed 5 65528 0 0 [ \"10.17.0.1\" ] Reconfigure ovn0 NIC Address \u00b6 The ovn0 NIC information for each node needs to be re-updated, which can be done by restarting kube-ovn-cni : kubectl delete pod -n kube-system -l app = kube-ovn-cni \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Join Subnet CIDR"},{"location":"en/ops/change-join-subnet/#change-join-subnet-cidr","text":"If the Join subnet CIDR created conflicts or does not meet expectations, you can use this document to modify. After modifying the Join Subnet CIDR, the previously created Pods will not be able to access the external network normally and need to wait for the rebuild completed.","title":"Change Join Subnet CIDR"},{"location":"en/ops/change-join-subnet/#delete-join-subnet","text":"kubectl patch subnet join --type = 'json' -p '[{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\": []}]' kubectl delete subnet join","title":"Delete Join Subnet"},{"location":"en/ops/change-join-subnet/#cleanup-allocated-config","text":"kubectl annotate node ovn.kubernetes.io/allocated = false --all --overwrite","title":"Cleanup Allocated Config"},{"location":"en/ops/change-join-subnet/#modify-join-subnet","text":"Change Join Subnet args in kube-ovn-controller : kubectl edit deployment -n kube-system kube-ovn-controller Change the CIDR below: args : - --node-switch-cidr=100.51.0.0/16 Reboot the kube-ovn-controller and rebuild join Subnet: kubectl delete pod -n kube-system -lapp = kube-ovn-controller Check the new Join Subnet information: # kubectl get subnet NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS join ovn ovn-cluster IPv4 100 .51.0.0/16 false false false distributed 2 65531 0 0 [ \"100.51.0.1\" ] ovn-default ovn ovn-cluster IPv4 10 .17.0.0/16 false true true distributed 5 65528 0 0 [ \"10.17.0.1\" ]","title":"Modify Join Subnet"},{"location":"en/ops/change-join-subnet/#reconfigure-ovn0-nic-address","text":"The ovn0 NIC information for each node needs to be re-updated, which can be done by restarting kube-ovn-cni : kubectl delete pod -n kube-system -l app = kube-ovn-cni \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Reconfigure ovn0 NIC Address"},{"location":"en/ops/change-log-level/","text":"Change Log Level \u00b6 Open kube-ovn.yaml and set the log level in the parameter list of the service startup script, such as: vi kube-ovn.yaml # ... - name: kube-ovn-controller image: \"docker.io/kubeovn/kube-ovn:v1.12.0\" imagePullPolicy: IfNotPresent args: - /kube-ovn/start-controller.sh - --v = 3 # ... # The higher the log level, the more detailed the log \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Log Level"},{"location":"en/ops/change-log-level/#change-log-level","text":"Open kube-ovn.yaml and set the log level in the parameter list of the service startup script, such as: vi kube-ovn.yaml # ... - name: kube-ovn-controller image: \"docker.io/kubeovn/kube-ovn:v1.12.0\" imagePullPolicy: IfNotPresent args: - /kube-ovn/start-controller.sh - --v = 3 # ... # The higher the log level, the more detailed the log \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Log Level"},{"location":"en/ops/change-ovn-central-node/","text":"Replace ovn-central Node \u00b6 Since ovn-nb and ovn-sb within ovn-central create separate etcd-like raft clusters, replacing the ovn-central node requires additional operations to ensure correct cluster state and consistent data. It is recommended that only one node be up and down at a time to avoid the cluster going into an unavailable state and affecting the overall cluster network. ovn-central Nodes Offline \u00b6 This document use the cluster below to describes how to remove the kube-ovn-control-plane2 node from the ovn-central as an example. # kubectl -n kube-system get pod -o wide | grep central ovn-central-6bf58cbc97-2cdhg 1 /1 Running 0 21m 172 .18.0.3 kube-ovn-control-plane <none> <none> ovn-central-6bf58cbc97-crmfp 1 /1 Running 0 21m 172 .18.0.5 kube-ovn-control-plane2 <none> <none> ovn-central-6bf58cbc97-lxmpl 1 /1 Running 0 21m 172 .18.0.4 kube-ovn-control-plane3 <none> <none> Kick Node in ovn-nb \u00b6 First check the ID of the node within the cluster for subsequent operations. # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2135194 ms ago, reason: timeout Last Election won: 2135188 ms ago Election timer: 5000 Log: [ 135 , 135 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-d64b ->d64b <-4984 ->4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 134 d64b ( d64b at tcp: [ 172 .18.0.5 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago status: ok kube-ovn-control-plane2 corresponds to a node IP of 172.18.0.5 and the corresponding ID within the cluster is d64b . Next, kick the node out of the ovn-nb cluster. # kubectl ko nb kick d64b started removal Check if the node has been kicked: # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2297649 ms ago, reason: timeout Last Election won: 2297643 ms ago Election timer: 5000 Log: [ 136 , 136 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-4984 ->4984 Disconnections: 2 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 136 match_index = 135 last msg 1270 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 135 status: ok Kick Node in ovn-sb \u00b6 Next, for the ovn-sb cluster, you need to first check the ID of the node within the cluster for subsequent operations. kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2395317 ms ago, reason: timeout Last Election won: 2395316 ms ago Election timer: 5000 Log: [ 130 , 130 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-e9f7 ->e9f7 <-6e84 ->6e84 Disconnections: 0 Servers: e9f7 ( e9f7 at tcp: [ 172 .18.0.5 ] :6644 ) next_index = 130 match_index = 129 last msg 1006 ms ago 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 130 match_index = 129 last msg 1004 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 129 status: ok kube-ovn-control-plane2 corresponds to node IP 172.18.0.5 and the corresponding ID within the cluster is e9f7 . Next, kick the node out of the ovn-sb cluster. # kubectl ko sb kick e9f7 started removal Check if the node has been kicked: # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2481636 ms ago, reason: timeout Last Election won: 2481635 ms ago Election timer: 5000 Log: [ 131 , 131 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-6e84 ->6e84 Disconnections: 2 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 131 match_index = 130 last msg 642 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 130 status: ok Delete Node Label and Downscale ovn-central \u00b6 Note that you need to remove the offline node from the node address of the ovn-central environment variable NODE_IPS . kubectl label node kube-ovn-control-plane2 kube-ovn/role- kubectl scale deployment -n kube-system ovn-central --replicas = 2 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4\" kubectl rollout status deployment/ovn-central -n kube-system Modify Components Address to ovn-central \u00b6 Modify ovs-ovn to remove the offline Node address: # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted Modify kube-ovn-controller to remove the offline Node address: # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out Clean Node \u00b6 Delete the database files in the kube-ovn-control-plane2 node to avoid errors when adding the node again: rm -rf /etc/origin/ovn To take a node offline from a Kubernetes cluster entirely, please continue with Delete Work Node . ovn-central Online \u00b6 The following steps will add a new Kubernetes node to the ovn-central cluster. Directory Check \u00b6 Check if the ovnnb_db.db or ovnsb_db.db file exists in the /etc/origin/ovn directory of the new node, and if so, delete it: rm -rf /etc/origin/ovn Check Current ovn-central Status \u00b6 If the current ovn-central cluster state is already abnormal, adding new nodes may cause the voting election to fail to pass the majority, affecting subsequent operations. # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 44 Leader: self Vote: self Last Election started 1855739 ms ago, reason: timeout Last Election won: 1855729 ms ago Election timer: 5000 Log: [ 147 , 147 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->4984 <-4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 147 match_index = 146 last msg 367 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 140 match_index = 146 status: ok # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 33 Leader: self Vote: self Last Election started 1868589 ms ago, reason: timeout Last Election won: 1868579 ms ago Election timer: 5000 Log: [ 142 , 142 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->6e84 <-6e84 Disconnections: 0 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 142 match_index = 141 last msg 728 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 134 match_index = 141 status: ok Label Node and Scale ovn-central \u00b6 Note that you need to add the online node address to the node address of the ovn-central environment variable NODE_IPS . kubectl label node kube-ovn-control-plane2 kube-ovn/role = master kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4,172.18.0.5\" kubectl rollout status deployment/ovn-central -n kube-system Modify Components Address to ovn-central \u00b6 Modify ovs-ovn to add the online Node address: # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted Modify kube-ovn-controller to add the online Node address: # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Replace ovn-central Node"},{"location":"en/ops/change-ovn-central-node/#replace-ovn-central-node","text":"Since ovn-nb and ovn-sb within ovn-central create separate etcd-like raft clusters, replacing the ovn-central node requires additional operations to ensure correct cluster state and consistent data. It is recommended that only one node be up and down at a time to avoid the cluster going into an unavailable state and affecting the overall cluster network.","title":"Replace ovn-central Node"},{"location":"en/ops/change-ovn-central-node/#ovn-central-nodes-offline","text":"This document use the cluster below to describes how to remove the kube-ovn-control-plane2 node from the ovn-central as an example. # kubectl -n kube-system get pod -o wide | grep central ovn-central-6bf58cbc97-2cdhg 1 /1 Running 0 21m 172 .18.0.3 kube-ovn-control-plane <none> <none> ovn-central-6bf58cbc97-crmfp 1 /1 Running 0 21m 172 .18.0.5 kube-ovn-control-plane2 <none> <none> ovn-central-6bf58cbc97-lxmpl 1 /1 Running 0 21m 172 .18.0.4 kube-ovn-control-plane3 <none> <none>","title":"ovn-central Nodes Offline"},{"location":"en/ops/change-ovn-central-node/#kick-node-in-ovn-nb","text":"First check the ID of the node within the cluster for subsequent operations. # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2135194 ms ago, reason: timeout Last Election won: 2135188 ms ago Election timer: 5000 Log: [ 135 , 135 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-d64b ->d64b <-4984 ->4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 134 d64b ( d64b at tcp: [ 172 .18.0.5 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago status: ok kube-ovn-control-plane2 corresponds to a node IP of 172.18.0.5 and the corresponding ID within the cluster is d64b . Next, kick the node out of the ovn-nb cluster. # kubectl ko nb kick d64b started removal Check if the node has been kicked: # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2297649 ms ago, reason: timeout Last Election won: 2297643 ms ago Election timer: 5000 Log: [ 136 , 136 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-4984 ->4984 Disconnections: 2 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 136 match_index = 135 last msg 1270 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 135 status: ok","title":"Kick Node in ovn-nb"},{"location":"en/ops/change-ovn-central-node/#kick-node-in-ovn-sb","text":"Next, for the ovn-sb cluster, you need to first check the ID of the node within the cluster for subsequent operations. kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2395317 ms ago, reason: timeout Last Election won: 2395316 ms ago Election timer: 5000 Log: [ 130 , 130 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-e9f7 ->e9f7 <-6e84 ->6e84 Disconnections: 0 Servers: e9f7 ( e9f7 at tcp: [ 172 .18.0.5 ] :6644 ) next_index = 130 match_index = 129 last msg 1006 ms ago 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 130 match_index = 129 last msg 1004 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 129 status: ok kube-ovn-control-plane2 corresponds to node IP 172.18.0.5 and the corresponding ID within the cluster is e9f7 . Next, kick the node out of the ovn-sb cluster. # kubectl ko sb kick e9f7 started removal Check if the node has been kicked: # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2481636 ms ago, reason: timeout Last Election won: 2481635 ms ago Election timer: 5000 Log: [ 131 , 131 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-6e84 ->6e84 Disconnections: 2 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 131 match_index = 130 last msg 642 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 130 status: ok","title":"Kick Node in ovn-sb"},{"location":"en/ops/change-ovn-central-node/#delete-node-label-and-downscale-ovn-central","text":"Note that you need to remove the offline node from the node address of the ovn-central environment variable NODE_IPS . kubectl label node kube-ovn-control-plane2 kube-ovn/role- kubectl scale deployment -n kube-system ovn-central --replicas = 2 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4\" kubectl rollout status deployment/ovn-central -n kube-system","title":"Delete Node Label and Downscale ovn-central"},{"location":"en/ops/change-ovn-central-node/#modify-components-address-to-ovn-central","text":"Modify ovs-ovn to remove the offline Node address: # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted Modify kube-ovn-controller to remove the offline Node address: # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out","title":"Modify Components Address to ovn-central"},{"location":"en/ops/change-ovn-central-node/#clean-node","text":"Delete the database files in the kube-ovn-control-plane2 node to avoid errors when adding the node again: rm -rf /etc/origin/ovn To take a node offline from a Kubernetes cluster entirely, please continue with Delete Work Node .","title":"Clean Node"},{"location":"en/ops/change-ovn-central-node/#ovn-central-online","text":"The following steps will add a new Kubernetes node to the ovn-central cluster.","title":"ovn-central Online"},{"location":"en/ops/change-ovn-central-node/#directory-check","text":"Check if the ovnnb_db.db or ovnsb_db.db file exists in the /etc/origin/ovn directory of the new node, and if so, delete it: rm -rf /etc/origin/ovn","title":"Directory Check"},{"location":"en/ops/change-ovn-central-node/#check-current-ovn-central-status","text":"If the current ovn-central cluster state is already abnormal, adding new nodes may cause the voting election to fail to pass the majority, affecting subsequent operations. # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 44 Leader: self Vote: self Last Election started 1855739 ms ago, reason: timeout Last Election won: 1855729 ms ago Election timer: 5000 Log: [ 147 , 147 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->4984 <-4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 147 match_index = 146 last msg 367 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 140 match_index = 146 status: ok # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 33 Leader: self Vote: self Last Election started 1868589 ms ago, reason: timeout Last Election won: 1868579 ms ago Election timer: 5000 Log: [ 142 , 142 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->6e84 <-6e84 Disconnections: 0 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 142 match_index = 141 last msg 728 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 134 match_index = 141 status: ok","title":"Check Current ovn-central Status"},{"location":"en/ops/change-ovn-central-node/#label-node-and-scale-ovn-central","text":"Note that you need to add the online node address to the node address of the ovn-central environment variable NODE_IPS . kubectl label node kube-ovn-control-plane2 kube-ovn/role = master kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4,172.18.0.5\" kubectl rollout status deployment/ovn-central -n kube-system","title":"Label Node and Scale ovn-central"},{"location":"en/ops/change-ovn-central-node/#modify-components-address-to-ovn-central_1","text":"Modify ovs-ovn to add the online Node address: # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted Modify kube-ovn-controller to add the online Node address: # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Modify Components Address to ovn-central"},{"location":"en/ops/delete-worker-node/","text":"Delete Work Node \u00b6 If the node is simply removed from Kubernetes, the ovn-controller process running in ovs-ovn on the node will periodically connect to ovn-central to register relevant network information. This leads to additional resource waste and potential rule conflict risk\u3002 Therefore, when removing nodes from within Kubernetes, follow the steps below to ensure that related resources are cleaned up properly. This document describes the steps to delete a worker node, if you want to change the node where ovn-central is located, please refer to Replace ovn-central Node . Evict Pods on the Node \u00b6 # kubectl drain kube-ovn-worker --ignore-daemonsets --force node/kube-ovn-worker cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-ovn-cni-zt74b, kube-system/kube-ovn-pinger-5rxfs, kube-system/kube-proxy-jpmnm, kube-system/ovs-ovn-v2kll evicting pod kube-system/coredns-64897985d-qsgpt evicting pod local-path-storage/local-path-provisioner-5ddd94ff66-llss6 evicting pod kube-system/kube-ovn-controller-8459db5ff4-94lxb pod/kube-ovn-controller-8459db5ff4-94lxb evicted pod/coredns-64897985d-qsgpt evicted pod/local-path-provisioner-5ddd94ff66-llss6 evicted node/kube-ovn-worker drained Stop kubelet and docker \u00b6 This step stops the ovs-ovn container to avoid registering information to ovn-central . Log into to the corresponding node and ruu the following commands: systemctl stop kubelet systemctl stop docker If using containerd as the CRI, the following command needs to be executed to stop the ovs-ovn container: crictl rm -f $( crictl ps | grep openvswitch | awk '{print $1}' ) Cleanup Files on Node \u00b6 rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn Delete the Node \u00b6 kubectl delete no kube-ovn-01 Check If Node Removed from OVN-SB \u00b6 In the example below, the node kube-ovn-worker is not removed: # kubectl ko sbctl show Chassis \"b0564934-5a0d-4804-a4c0-476c93596a17\" hostname: kube-ovn-worker Encap geneve ip: \"172.18.0.2\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-5rxfs.kube-system Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system Delete the Chassis Manually \u00b6 Use the uuid find above to delete the chassis: # kubectl ko sbctl chassis-del b0564934-5a0d-4804-a4c0-476c93596a17 # kubectl ko sbctl show Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Deleting Work Node"},{"location":"en/ops/delete-worker-node/#delete-work-node","text":"If the node is simply removed from Kubernetes, the ovn-controller process running in ovs-ovn on the node will periodically connect to ovn-central to register relevant network information. This leads to additional resource waste and potential rule conflict risk\u3002 Therefore, when removing nodes from within Kubernetes, follow the steps below to ensure that related resources are cleaned up properly. This document describes the steps to delete a worker node, if you want to change the node where ovn-central is located, please refer to Replace ovn-central Node .","title":"Delete Work Node"},{"location":"en/ops/delete-worker-node/#evict-pods-on-the-node","text":"# kubectl drain kube-ovn-worker --ignore-daemonsets --force node/kube-ovn-worker cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-ovn-cni-zt74b, kube-system/kube-ovn-pinger-5rxfs, kube-system/kube-proxy-jpmnm, kube-system/ovs-ovn-v2kll evicting pod kube-system/coredns-64897985d-qsgpt evicting pod local-path-storage/local-path-provisioner-5ddd94ff66-llss6 evicting pod kube-system/kube-ovn-controller-8459db5ff4-94lxb pod/kube-ovn-controller-8459db5ff4-94lxb evicted pod/coredns-64897985d-qsgpt evicted pod/local-path-provisioner-5ddd94ff66-llss6 evicted node/kube-ovn-worker drained","title":"Evict Pods on the Node"},{"location":"en/ops/delete-worker-node/#stop-kubelet-and-docker","text":"This step stops the ovs-ovn container to avoid registering information to ovn-central . Log into to the corresponding node and ruu the following commands: systemctl stop kubelet systemctl stop docker If using containerd as the CRI, the following command needs to be executed to stop the ovs-ovn container: crictl rm -f $( crictl ps | grep openvswitch | awk '{print $1}' )","title":"Stop kubelet and docker"},{"location":"en/ops/delete-worker-node/#cleanup-files-on-node","text":"rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn","title":"Cleanup Files on Node"},{"location":"en/ops/delete-worker-node/#delete-the-node","text":"kubectl delete no kube-ovn-01","title":"Delete the Node"},{"location":"en/ops/delete-worker-node/#check-if-node-removed-from-ovn-sb","text":"In the example below, the node kube-ovn-worker is not removed: # kubectl ko sbctl show Chassis \"b0564934-5a0d-4804-a4c0-476c93596a17\" hostname: kube-ovn-worker Encap geneve ip: \"172.18.0.2\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-5rxfs.kube-system Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system","title":"Check If Node Removed from OVN-SB"},{"location":"en/ops/delete-worker-node/#delete-the-chassis-manually","text":"Use the uuid find above to delete the chassis: # kubectl ko sbctl chassis-del b0564934-5a0d-4804-a4c0-476c93596a17 # kubectl ko sbctl show Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Delete the Chassis Manually"},{"location":"en/ops/faq/","text":"FAQ \u00b6 Kylin ARM system cross-host container access intermittently fails \u00b6 Behavior \u00b6 There is a problem with Kylin ARM system and some NIC offload, which can cause intermittent container network failure. Use netstat to identify the problem: # netstat -us IcmpMsg: InType0: 22 InType3: 24 InType8: 117852 OutType0: 117852 OutType3: 29 OutType8: 22 Udp: 3040636 packets received 0 packets to unknown port received. 4 packet receive errors 602 packets sent 0 receive buffer errors 0 send buffer errors InCsumErrors: 4 UdpLite: IpExt: InBcastPkts: 10244 InOctets: 4446320361 OutOctets: 1496815600 InBcastOctets: 3095950 InNoECTPkts: 7683903 If InCsumErrors is present and increases with netwoork failures, you can confirm that this is the problem. Solution \u00b6 The fundamental solution requires communication with Kylin and the corresponding network card manufacturer to update the system and drivers. A temporary solution would be to turn off tx offload on the physical NIC, but this would cause a significant degradation in tcp performance. ethtool -K eth0 tx off From the community feedback, the problem can be solved by the 4.19.90-25.16.v2101 kernel. Pod can not Access Service \u00b6 Behavior \u00b6 Pod can not access Service, and dmesg show errors: netlink\uff1aUnknown conntrack attr ( type = 6 , max = 5 ) openvswitch: netlink: Flow actions may not be safe on all matching packets. This log indicates that the in-kernel OVS version is too low to support the corresponding NAT operation. Solution \u00b6 Upgrade the kernel module or compile the OVS kernel module manually. If you are using an Overlay network you can change the kube-ovn-controller args, setting --enable-lb=false to disable the OVN LB to use kube-proxy for service forwarding. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"FAQ"},{"location":"en/ops/faq/#faq","text":"","title":"FAQ"},{"location":"en/ops/faq/#kylin-arm-system-cross-host-container-access-intermittently-fails","text":"","title":"Kylin ARM system cross-host container access intermittently fails"},{"location":"en/ops/faq/#behavior","text":"There is a problem with Kylin ARM system and some NIC offload, which can cause intermittent container network failure. Use netstat to identify the problem: # netstat -us IcmpMsg: InType0: 22 InType3: 24 InType8: 117852 OutType0: 117852 OutType3: 29 OutType8: 22 Udp: 3040636 packets received 0 packets to unknown port received. 4 packet receive errors 602 packets sent 0 receive buffer errors 0 send buffer errors InCsumErrors: 4 UdpLite: IpExt: InBcastPkts: 10244 InOctets: 4446320361 OutOctets: 1496815600 InBcastOctets: 3095950 InNoECTPkts: 7683903 If InCsumErrors is present and increases with netwoork failures, you can confirm that this is the problem.","title":"Behavior"},{"location":"en/ops/faq/#solution","text":"The fundamental solution requires communication with Kylin and the corresponding network card manufacturer to update the system and drivers. A temporary solution would be to turn off tx offload on the physical NIC, but this would cause a significant degradation in tcp performance. ethtool -K eth0 tx off From the community feedback, the problem can be solved by the 4.19.90-25.16.v2101 kernel.","title":"Solution"},{"location":"en/ops/faq/#pod-can-not-access-service","text":"","title":"Pod can not Access Service"},{"location":"en/ops/faq/#behavior_1","text":"Pod can not access Service, and dmesg show errors: netlink\uff1aUnknown conntrack attr ( type = 6 , max = 5 ) openvswitch: netlink: Flow actions may not be safe on all matching packets. This log indicates that the in-kernel OVS version is too low to support the corresponding NAT operation.","title":"Behavior"},{"location":"en/ops/faq/#solution_1","text":"Upgrade the kernel module or compile the OVS kernel module manually. If you are using an Overlay network you can change the kube-ovn-controller args, setting --enable-lb=false to disable the OVN LB to use kube-proxy for service forwarding. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Solution"},{"location":"en/ops/from-calico/","text":"Install Kube-OVN From Calico \u00b6 If a Kubernetes cluster already has Calico installed and needs to change to Kube-OVN you can refer to this document. Since the installation of Calico may vary from version to version and the existing Pod network may be disrupted during the replacement process, it is recommended that you plan ahead and compare the differences in Calico installation from version to version. Uninstall Calico \u00b6 For Calico installed from an Operator: kubectl delete -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml kubectl delete -f https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml For Calico installed from manifests: kubectl delete -f https://projectcalico.docs.tigera.io/manifests/calico.yaml Cleanup Config Files \u00b6 Delete the CNI-related configuration files on each machine, depending on the environment: rm -f /etc/cni/net.d/10-calico.conflist rm -f /etc/cni/net.d/calico-kubeconfig Calico still leaves routing rules, iptables rules, veth network interfaces and other configuration information on the node, so it is recommended to reboot the node to clean up the relevant configuration to avoid problems that are difficult to troubleshoot. Install Kube-OVN \u00b6 You can refer to One Click Installation to install Kube-OVN as usual. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Install Kube-OVN From Calico"},{"location":"en/ops/from-calico/#install-kube-ovn-from-calico","text":"If a Kubernetes cluster already has Calico installed and needs to change to Kube-OVN you can refer to this document. Since the installation of Calico may vary from version to version and the existing Pod network may be disrupted during the replacement process, it is recommended that you plan ahead and compare the differences in Calico installation from version to version.","title":"Install Kube-OVN From Calico"},{"location":"en/ops/from-calico/#uninstall-calico","text":"For Calico installed from an Operator: kubectl delete -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml kubectl delete -f https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml For Calico installed from manifests: kubectl delete -f https://projectcalico.docs.tigera.io/manifests/calico.yaml","title":"Uninstall Calico"},{"location":"en/ops/from-calico/#cleanup-config-files","text":"Delete the CNI-related configuration files on each machine, depending on the environment: rm -f /etc/cni/net.d/10-calico.conflist rm -f /etc/cni/net.d/calico-kubeconfig Calico still leaves routing rules, iptables rules, veth network interfaces and other configuration information on the node, so it is recommended to reboot the node to clean up the relevant configuration to avoid problems that are difficult to troubleshoot.","title":"Cleanup Config Files"},{"location":"en/ops/from-calico/#install-kube-ovn","text":"You can refer to One Click Installation to install Kube-OVN as usual. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Install Kube-OVN"},{"location":"en/ops/kubectl-ko/","text":"Kubectl Plugin \u00b6 To facilitate daily operations and maintenance, Kube-OVN provides the kubectl plug-in tool, which allows administrators to perform daily operations through this command. For examples: Check OVN database information and status, OVN database backup and restore, OVS related information, tcpdump specific containers, specific link logical topology, network problem diagnosis and performance optimization. Plugin Installation \u00b6 Kube-OVN installation will deploy the plugin to each node by default. If the machine that runs kubectl is not in the cluster, or if you need to reinstall the plugin, please refer to the following steps: Download kubectl-ko file: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/kubectl-ko Move file to $PATH : mv kubectl-ko /usr/local/bin/kubectl-ko Add executable permissions: chmod +x /usr/local/bin/kubectl-ko Check if the plugin works properly: # kubectl plugin list The following compatible plugins are available: /usr/local/bin/kubectl-ko Plugin Usage \u00b6 Running kubectl ko will show all the available commands and usage descriptions, as follows: # kubectl ko kubectl ko { subcommand } [ option... ] Available Subcommands: [ nb | sb ] [ status | kick | backup | dbstatus | restore ] ovn-db operations show cluster status, kick stale server, backup database, get db consistency status or restore ovn nb db when met 'inconsistent data' error nbctl [ ovn-nbctl options ... ] invoke ovn-nbctl sbctl [ ovn-sbctl options ... ] invoke ovn-sbctl vsctl { nodeName } [ ovs-vsctl options ... ] invoke ovs-vsctl on the specified node ofctl { nodeName } [ ovs-ofctl options ... ] invoke ovs-ofctl on the specified node dpctl { nodeName } [ ovs-dpctl options ... ] invoke ovs-dpctl on the specified node appctl { nodeName } [ ovs-appctl options ... ] invoke ovs-appctl on the specified node tcpdump { namespace/podname } [ tcpdump options ... ] capture pod traffic trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply echo \" diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] diagnose connectivity of all nodes or a specific node or specify subnet's ds pod or IPPorts like 'tcp-172.18.0.2-53,udp-172.18.0.3-53'\" tuning { install-fastpath | local-install-fastpath | remove-fastpath | install-stt | local-install-stt | remove-stt } { centos7 | centos8 }} [ kernel-devel-version ] deploy kernel optimisation components to the system reload restart all kube-ovn components log { kube-ovn | ovn | ovs | linux | all } save log to ./kubectl-ko-log/ perf [ image ] performance test default image is kubeovn/test:v1.12.0 The specific functions and usage of each command are described below. [nb | sb] [status | kick | backup | dbstatus | restore] \u00b6 This subcommand mainly operates on OVN northbound or southbound databases, including database cluster status check, database node offline, database backup, database storage status check and database repair. DB Cluster Status Check \u00b6 This command executes ovs-appctl cluster/status on the leader node of the corresponding OVN database to show the cluster status: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 280309 ms ago, reason: timeout Last Election won: 280309 ms ago Election timer: 5000 Log: [ 139 , 139 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-8723 ->8723 <-85d6 ->85d6 Disconnections: 0 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 8723 ( 8723 at tcp: [ 172 .18.0.3 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 138 status: ok If the match_index under Server has a large difference and the last msg time is long, the corresponding Server may not respond for a long time and needs to be checked further. DB Nodes Offline \u00b6 This command removes a node from the OVN database and is required when a node is taken offline or replaced. The following is an example of the cluster status from the previous command, to offline the 172.18.0.3 node: # kubectl ko nb kick 8723 started removal Check the database cluster status again to confirm that the node has been removed: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 324356 ms ago, reason: timeout Last Election won: 324356 ms ago Election timer: 5000 Log: [ 140 , 140 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-85d6 ->85d6 Disconnections: 2 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 140 match_index = 139 last msg 848 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 139 status: ok DB Backup \u00b6 This subcommand backs up the current OVN database locally and can be used for disaster recovery: # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup Database Storage Status Check \u00b6 This command is used to check if the database file is corrupt: # kubectl ko nb dbstatus status: ok If error happens, inconsistent data is displayed and needs to be fixed with the following command. Database Repair \u00b6 If the database status goes to inconsistent data , this command can be used to repair: # kubectl ko nb restore deployment.apps/ovn-central scaled ovn-central original replicas is 3 first nodeIP is 172 .18.0.5 ovs-ovn pod on node 172 .18.0.5 is ovs-ovn-8jxv9 ovs-ovn pod on node 172 .18.0.3 is ovs-ovn-sjzb6 ovs-ovn pod on node 172 .18.0.4 is ovs-ovn-t87zk backup nb db file restore nb db file, operate in pod ovs-ovn-8jxv9 deployment.apps/ovn-central scaled finish restore nb db file and ovn-central replicas recreate ovs-ovn pods pod \"ovs-ovn-8jxv9\" deleted pod \"ovs-ovn-sjzb6\" deleted pod \"ovs-ovn-t87zk\" deleted [nbctl | sbctl] [options ...] \u00b6 This subcommand executes the ovn-nbctl and ovn-sbctl commands directly into the leader node of the OVN northbound or southbound database. For more detailed usage of this command, please refer to the official documentation of the upstream OVN ovn-nbctl(8) \u548c ovn-sbctl(8) \u3002 # kubectl ko nbctl show switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port node-liumengxin-ovn2-192.168.137.177 addresses: [ \"00:00:00:D9:58:B8 100.64.0.3\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port alertmanager-main-0.monitoring addresses: [ \"00:00:00:6C:DF:A3 10.16.0.19\" ] port kube-state-metrics-5d6885d89-4nf8h.monitoring addresses: [ \"00:00:00:6F:02:1C 10.16.0.15\" ] port fake-kubelet-67c55dfd89-pv86k.kube-system addresses: [ \"00:00:00:5C:12:E8 10.16.19.177\" ] port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] vsctl {nodeName} [options ...] \u00b6 This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-vsctl command to query and configure vswitchd . For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-vsctl(8) \u3002 # kubectl ko vsctl kube-ovn-01 show 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port mirror0 Interface mirror0 type: internal Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" ofctl {nodeName} [options ...] \u00b6 This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-ofctl command to query or manage OpenFlow. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-ofctl(8) \u3002 # kubectl ko ofctl kube-ovn-01 dump-flows br-int NXST_FLOW reply ( xid = 0x4 ) : flags =[ more ] cookie = 0xcf3429e6, duration = 671791 .432s, table = 0 , n_packets = 0 , n_bytes = 0 , idle_age = 65534 , hard_age = 65534 , priority = 100 ,in_port = 2 actions = load:0x4->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x1->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xc91413c6, duration = 671791 .431s, table = 0 , n_packets = 907489 , n_bytes = 99978275 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 7 actions = load:0x1->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x4->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xf180459, duration = 671791 .431s, table = 0 , n_packets = 17348582 , n_bytes = 2667811214 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 6317 actions = load:0xa->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x9->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0x7806dd90, duration = 671791 .431s, table = 0 , n_packets = 3235428 , n_bytes = 833821312 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 1 actions = load:0xd->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x3->NXM_NX_REG14 [] ,resubmit ( ,8 ) ... dpctl {nodeName} [options ...] \u00b6 This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-dpctl command to query or manage the OVS datapath. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-dpctl(8) \u3002 # kubectl ko dpctl kube-ovn-01 show system@ovs-system: lookups: hit:350805055 missed:21983648 lost:73 flows: 105 masks: hit:1970748791 total:22 hit/pkt:5.29 port 0 : ovs-system ( internal ) port 1 : ovn0 ( internal ) port 2 : mirror0 ( internal ) port 3 : br-int ( internal ) port 4 : stt_sys_7471 ( stt: packet_type = ptap ) port 5 : eeb4d9e51b5d_h port 6 : a2c1a8a8b83a_h port 7 : 4fa5c4cbb1a5_h appctl {nodeName} [options ...] \u00b6 This command will enter the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-appctl command to operate the associated daemon process. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-appctl(8) \u3002 # kubectl ko appctl kube-ovn-01 vlog/list console syslog file ------- ------ ------ backtrace OFF ERR INFO bfd OFF ERR INFO bond OFF ERR INFO bridge OFF ERR INFO bundle OFF ERR INFO bundles OFF ERR INFO ... tcpdump {namespace/podname} [tcpdump options ...] \u00b6 This command will enter the kube-ovn-cni container on the machine where namespace/podname is located, and run tcpdump to capture the traffic on the veth NIC of the corresponding container, which can be used to troubleshoot network-related problems. # kubectl ko tcpdump default/ds1-l6n7p icmp + kubectl exec -it kube-ovn-cni-wlg4s -n kube-ovn -- tcpdump -nn -i d7176fe7b4e0_h icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on d7176fe7b4e0_h, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 06 :52:36.619688 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 1 , length 64 06 :52:36.619746 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 1 , length 64 06 :52:37.619588 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 2 , length 64 06 :52:37.619630 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 2 , length 64 06 :52:38.619933 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 3 , length 64 06 :52:38.619973 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 3 , length 64 trace [arguments ...] \u00b6 This command will print the OVN logical flow table and the final Openflow flow table when the Pod/node accesses an address through a specific protocol, so that it make locate flow table related problems during development or troubleshooting much easy. Supported commands: kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } kubectl ko trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } Example: # kubectl ko trace default/ds1-l6n7p 8.8.8.8 icmp + kubectl exec ovn-central-5bc494cb5-np9hm -n kube-ovn -- ovn-trace --ct = new ovn-default 'inport == \"ds1-l6n7p.default\" && ip.ttl == 64 && icmp && eth.src == 0a:00:00:10:00:05 && ip4.src == 10.16.0.4 && eth.dst == 00:00:00:B8:CA:43 && ip4.dst == 8.8.8.8' # icmp,reg14=0xf,vlan_tci=0x0000,dl_src=0a:00:00:10:00:05,dl_dst=00:00:00:b8:ca:43,nw_src=10.16.0.4,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 ingress ( dp = \"ovn-default\" , inport = \"ds1-l6n7p.default\" ) ----------------------------------------------------- 0 . ls_in_port_sec_l2 ( ovn-northd.c:4143 ) : inport == \"ds1-l6n7p.default\" && eth.src == { 0a:00:00:10:00:05 } , priority 50 , uuid 39453393 next ; 1 . ls_in_port_sec_ip ( ovn-northd.c:2898 ) : inport == \"ds1-l6n7p.default\" && eth.src == 0a:00:00:10:00:05 && ip4.src == { 10 .16.0.4 } , priority 90 , uuid 81bcd485 next ; 3 . ls_in_pre_acl ( ovn-northd.c:3269 ) : ip, priority 100 , uuid 7b4f4971 reg0 [ 0 ] = 1 ; next ; 5 . ls_in_pre_stateful ( ovn-northd.c:3396 ) : reg0 [ 0 ] == 1 , priority 100 , uuid 36cdd577 ct_next ; ct_next ( ct_state = new | trk ) ------------------------- 6 . ls_in_acl ( ovn-northd.c:3759 ) : ip && ( !ct.est || ( ct.est && ct_label.blocked == 1 )) , priority 1 , uuid 7608af5b reg0 [ 1 ] = 1 ; next ; 10 . ls_in_stateful ( ovn-northd.c:3995 ) : reg0 [ 1 ] == 1 , priority 100 , uuid 2aba1b90 ct_commit ( ct_label = 0 /0x1 ) ; next ; 16 . ls_in_l2_lkup ( ovn-northd.c:4470 ) : eth.dst == 00 :00:00:b8:ca:43, priority 50 , uuid 5c9c3c9f outport = \"ovn-default-ovn-cluster\" ; output ; ... If the trace object is a virtual machine running in Underlay network, additional parameters is needed to specify the destination Mac address. kubectl ko trace default/virt-handler-7lvml 8 .8.8.8 82 :7c:9f:83:8c:01 icmp diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] \u00b6 Diagnose the status of cluster network components and go to the corresponding node's kube-ovn-pinger to detect connectivity and network latency from the current node to other nodes and critical services. # kubectl ko diagnose all switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default port prometheus-k8s-1.monitoring addresses: [ \"00:00:00:AA:37:DF 10.16.0.23\" ] router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] Routing Policies 31000 ip4.dst == 10 .16.0.0/16 allow 31000 ip4.dst == 100 .64.0.0/16 allow 30000 ip4.dst == 192 .168.137.177 reroute 100 .64.0.3 30000 ip4.dst == 192 .168.137.178 reroute 100 .64.0.4 29000 ip4.src == $ovn .default.fake.6_ip4 reroute 100 .64.0.22 29000 ip4.src == $ovn .default.fake.7_ip4 reroute 100 .64.0.21 29000 ip4.src == $ovn .default.fake.8_ip4 reroute 100 .64.0.23 29000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 reroute 100 .64.0.4 20000 ip4.src == $ovn .default.liumengxin.ovn1.192.168.137.176_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.2 20000 ip4.src == $ovn .default.liumengxin.ovn2.192.168.137.177_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.3 20000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.4 IPv4 Routes Route Table <main>: 0 .0.0.0/0 100 .64.0.1 dst-ip UUID LB PROTO VIP IPs e9bcfd9d-793e-4431-9073-6dec96b75d71 cluster-tcp-load tcp 10 .100.209.132:10660 192 .168.137.176:10660 tcp 10 .101.239.192:6641 192 .168.137.177:6641 tcp 10 .101.240.101:3000 10 .16.0.7:3000 tcp 10 .103.184.186:6642 192 .168.137.177:6642 35d2b7a5-e3a7-485a-a4b7-b4970eb0e63b cluster-tcp-sess tcp 10 .100.158.128:8080 10 .16.0.10:8080,10.16.0.5:8080,10.16.63.30:8080 tcp 10 .107.26.215:8080 10 .16.0.19:8080,10.16.0.20:8080,10.16.0.21:8080 tcp 10 .107.26.215:9093 10 .16.0.19:9093,10.16.0.20:9093,10.16.0.21:9093 tcp 10 .98.187.99:8080 10 .16.0.22:8080,10.16.0.23:8080 tcp 10 .98.187.99:9090 10 .16.0.22:9090,10.16.0.23:9090 f43303e4-89aa-4d3e-a3dc-278a552fe27b cluster-udp-load udp 10 .96.0.10:53 10 .16.0.4:53,10.16.0.9:53 _uuid : 06776304 -5a96-43ed-90c4-c4854c251699 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn2_192.168.137.177_underlay_v6 _uuid : 62690625 -87d5-491c-8675-9fd83b1f433c addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn1_192.168.137.176_underlay_v6 _uuid : b03a9bae-94d5-4562-b34c-b5f6198e180b addresses : [ \"10.16.0.0/16\" , \"100.64.0.0/16\" , \"172.22.0.0/16\" , \"192.101.0.0/16\" ] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv4 _uuid : e1056f3a-24cc-4666-8a91-75ee6c3c2426 addresses : [] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv6 _uuid : 3e5d5fff-e670-47b2-a2f5-a39f4698a8c5 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn3_192.168.137.178_underlay_v6 _uuid : 2d85dbdc-d0db-4abe-b19e-cc806d32b492 action : drop direction : from-lport external_ids : {} label : 0 log : false match : \"inport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] _uuid : de790cc8-f155-405f-bb32-5a51f30c545f action : drop direction : to-lport external_ids : {} label : 0 log : false match : \"outport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] Chassis \"e15ed4d4-1780-4d50-b09e-ea8372ed48b8\" hostname: liumengxin-ovn1-192.168.137.176 Encap stt ip: \"192.168.137.176\" options: { csum = \"true\" } Port_Binding node-liumengxin-ovn1-192.168.137.176 Port_Binding perf-6vxkn.default Port_Binding kube-state-metrics-5d6885d89-4nf8h.monitoring Port_Binding alertmanager-main-0.monitoring Port_Binding kube-ovn-pinger-6ftdf.kube-system Port_Binding fake-kubelet-67c55dfd89-pv86k.kube-system Port_Binding prometheus-k8s-0.monitoring Chassis \"eef07da1-f8ad-4775-b14d-bd6a3b4eb0d5\" hostname: liumengxin-ovn3-192.168.137.178 Encap stt ip: \"192.168.137.178\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-7twb4.kube-system Port_Binding prometheus-adapter-86df476d87-rl88g.monitoring Port_Binding prometheus-k8s-1.monitoring Port_Binding node-liumengxin-ovn3-192.168.137.178 Port_Binding perf-ff475.default Port_Binding alertmanager-main-1.monitoring Port_Binding blackbox-exporter-676d976865-tvsjd.monitoring Chassis \"efa253c9-494d-4719-83ae-b48ab0f11c03\" hostname: liumengxin-ovn2-192.168.137.177 Encap stt ip: \"192.168.137.177\" options: { csum = \"true\" } Port_Binding grafana-6c4c6b8fb7-pzd2c.monitoring Port_Binding node-liumengxin-ovn2-192.168.137.177 Port_Binding alertmanager-main-2.monitoring Port_Binding coredns-6789c94dd8-9jqsz.kube-system Port_Binding coredns-6789c94dd8-25d4r.kube-system Port_Binding prometheus-operator-7bbc99fc8b-wgjm4.monitoring Port_Binding prometheus-adapter-86df476d87-gdxmc.monitoring Port_Binding perf-fjnws.default Port_Binding kube-ovn-pinger-vh2xg.kube-system ds kube-proxy ready kube-proxy ready deployment ovn-central ready deployment kube-ovn-controller ready ds kube-ovn-cni ready ds ovs-ovn ready deployment coredns ready ovn-nb leader check ok ovn-sb leader check ok ovn-northd leader check ok ### kube-ovn-controller recent log ### start to diagnose node liumengxin-ovn1-192.168.137.176 #### ovn-controller log: 2022 -06-03T00:56:44.897Z | 16722 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:06:44.912Z | 16723 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:16:44.925Z | 16724 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:26:44.936Z | 16725 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:36:44.959Z | 16726 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:46:44.974Z | 16727 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:56:44.988Z | 16728 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:06:45.001Z | 16729 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:16:45.025Z | 16730 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:26:45.040Z | 16731 | inc_proc_eng | INFO | User triggered force recompute. #### ovs-vswitchd log: 2022 -06-02T23:03:00.137Z | 00079 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:f9d1 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-02T23:23:31.840Z | 00080 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:15b2 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:09:15.659Z | 00081 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:dc:e3:63,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.63.30,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:e5a5 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x150017000004,src = 192 .168.137.178,dst = 192 .168.137.176,ttl = 64 ,tp_src = 9239 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.63.30,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:30:13.409Z | 00064 | dpif ( handler2 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:6b4a with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T02:02:33.832Z | 00082 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:a819 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 #### ovs-vsctl show results: 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port \"04d03360e9a0_h\" Interface \"04d03360e9a0_h\" Port eeb4d9e51b5d_h Interface eeb4d9e51b5d_h Port mirror0 Interface mirror0 type: internal Port \"8e5d887ccd80_h\" Interface \"8e5d887ccd80_h\" Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port \"17512d5be1f1_h\" Interface \"17512d5be1f1_h\" Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" #### pinger diagnose results: I0603 10 :35:04.349404 17619 pinger.go:19 ] ------------------------------------------------------------------------------- Kube-OVN: Version: v1.12.0 Build: 2022 -04-24_08:02:50 Commit: git-73f9d15 Go Version: go1.17.8 Arch: amd64 ------------------------------------------------------------------------------- I0603 10 :35:04.376797 17619 config.go:166 ] pinger config is & { KubeConfigFile: KubeClient:0xc000493380 Port:8080 DaemonSetNamespace:kube-system DaemonSetName:kube-ovn-pinger Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:liumengxin-ovn1-192.168.137.176 HostIP:192.168.137.176 PodName:kube-ovn-pinger-6ftdf PodIP:10.16.0.10 PodProtocols: [ IPv4 ] ExternalAddress: NetworkMode:kube-ovn PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid } I0603 10 :35:04.449166 17619 exporter.go:75 ] liumengxin-ovn1-192.168.137.176: exporter connect successfully I0603 10 :35:04.554011 17619 ovn.go:21 ] ovs-vswitchd and ovsdb are up I0603 10 :35:04.651293 17619 ovn.go:33 ] ovn_controller is up I0603 10 :35:04.651342 17619 ovn.go:39 ] start to check port binding I0603 10 :35:04.749613 17619 ovn.go:135 ] chassis id is 1d7f3d6c-eec5-4b3c-adca-2969d9cdfd80 I0603 10 :35:04.763487 17619 ovn.go:49 ] port in sb is [ node-liumengxin-ovn1-192.168.137.176 perf-6vxkn.default kube-state-metrics-5d6885d89-4nf8h.monitoring alertmanager-main-0.monitoring kube-ovn-pinger-6ftdf.kube-system fake-kubelet-67c55dfd89-pv86k.kube-system prometheus-k8s-0.monitoring ] I0603 10 :35:04.763583 17619 ovn.go:61 ] ovs and ovn-sb binding check passed I0603 10 :35:05.049309 17619 ping.go:259 ] start to check apiserver connectivity I0603 10 :35:05.053666 17619 ping.go:268 ] connect to apiserver success in 4 .27ms I0603 10 :35:05.053786 17619 ping.go:129 ] start to check pod connectivity I0603 10 :35:05.249590 17619 ping.go:159 ] ping pod: kube-ovn-pinger-6ftdf 10 .16.0.10, count: 3 , loss count 0 , average rtt 16 .30ms I0603 10 :35:05.354135 17619 ping.go:159 ] ping pod: kube-ovn-pinger-7twb4 10 .16.63.30, count: 3 , loss count 0 , average rtt 1 .81ms I0603 10 :35:05.458460 17619 ping.go:159 ] ping pod: kube-ovn-pinger-vh2xg 10 .16.0.5, count: 3 , loss count 0 , average rtt 1 .92ms I0603 10 :35:05.458523 17619 ping.go:83 ] start to check node connectivity If the target of diagnose is specified as subnet, the script will create a daemonset on the subnet, and kube-ovn-pinger will detect the connectivity and network delay of all pods in this daemonset, and automatically destroy the daemonset after the test. If the target of diagnose is specified as IPPorts, the script will let each kube-ovn-pinger pod detect whether the target protocol, IP, and Port are reachable. tuning {install-fastpath|local-install-fastpath|remove-fastpath|install-stt|local-install-stt|remove-stt} {centos7|centos8}} [kernel-devel-version] \u00b6 This command performs performance tuning related operations, please refer to Performance Tunning . reload \u00b6 This command restarts all Kube-OVN related components: # kubectl ko reload pod \"ovn-central-8684dd94bd-vzgcr\" deleted Waiting for deployment \"ovn-central\" rollout to finish: 0 of 1 updated replicas are available... deployment \"ovn-central\" successfully rolled out pod \"ovs-ovn-bsnvz\" deleted pod \"ovs-ovn-m9b98\" deleted pod \"kube-ovn-controller-8459db5ff4-64c62\" deleted Waiting for deployment \"kube-ovn-controller\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out pod \"kube-ovn-cni-2klnh\" deleted pod \"kube-ovn-cni-t2jz4\" deleted Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-cni\" successfully rolled out pod \"kube-ovn-pinger-ln72z\" deleted pod \"kube-ovn-pinger-w8lrk\" deleted Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-pinger\" successfully rolled out pod \"kube-ovn-monitor-7fb67d5488-7q6zb\" deleted Waiting for deployment \"kube-ovn-monitor\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-monitor\" successfully rolled out log \u00b6 Using this command will capture the logs of Kube-OVN, OVN, Openvswitch on all nodes of kube-ovn and some debug information commonly used in linux. # kubectl ko log all Collecting kube-ovn logging files Collecting ovn logging files Collecting openvswitch logging files Collecting linux dmesg files Collecting linux iptables-legacy files Collecting linux iptables-nft files Collecting linux route files Collecting linux link files Collecting linux neigh files Collecting linux memory files Collecting linux top files Collecting linux sysctl files Collecting linux netstat files Collecting linux addr files Collecting linux ipset files Collecting linux tcp files Collected files have been saved in the directory /root/kubectl-ko-log The directory is as follows: # tree kubectl-ko-log/ kubectl-ko-log/ | -- kube-ovn-control-plane | | -- kube-ovn | | | -- kube-ovn-cni.log | | | -- kube-ovn-monitor.log | | ` -- kube-ovn-pinger.log | | -- linux | | | -- addr.log | | | -- dmesg.log | | | -- ipset.log | | | -- iptables-legacy.log | | | -- iptables-nft.log | | | -- link.log | | | -- memory.log | | | -- neigh.log | | | -- netstat.log | | | -- route.log | | | -- sysctl.log | | | -- tcp.log | | ` -- top.log | | -- openvswitch | | | -- ovs-vswitchd.log | | ` -- ovsdb-server.log | ` -- ovn | | -- ovn-controller.log | | -- ovn-northd.log | | -- ovsdb-server-nb.log | ` -- ovsdb-server-sb.log perf [image] \u00b6 This command will test some performance indicators of Kube-OVN as follows: The performance indicators of the container network; Hostnetwork network performance indicators; Container network multicast packet performance indicators; Time required for OVN-NB, OVN-SB, and OVN-Northd leader deletion recovery. The parameter image is used to specify the image used by the performance test pod. By default, it is kubeovn/test:v1.12.0 . This parameter is mainly set for offline scenarios, and the image name may change when the image is pulled to the intranet environment. # kubectl ko perf ============================== Prepareing Performance Test Resources =============================== pod/test-client created pod/test-host-client created pod/test-server created pod/test-host-server created service/test-server created pod/test-client condition met pod/test-host-client condition met pod/test-host-server condition met pod/test-server condition met ==================================================================================================== ============================ Start Pod Network Unicast Performance Test ============================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 82 .8 us 97 .7 Mbits/sec 67 .6 us ( 0 % ) 8 .42 Mbits/sec 128 85 .4 us 167 Mbits/sec 67 .2 us ( 0 % ) 17 .2 Mbits/sec 512 85 .8 us 440 Mbits/sec 68 .7 us ( 0 % ) 68 .4 Mbits/sec 1k 85 .1 us 567 Mbits/sec 68 .7 us ( 0 % ) 134 Mbits/sec 4k 138 us 826 Mbits/sec 78 .1 us ( 1 .4% ) 503 Mbits/sec ==================================================================================================== =============================== Start Host Network Performance Test ================================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 49 .7 us 120 Mbits/sec 37 .9 us ( 0 % ) 18 .6 Mbits/sec 128 49 .7 us 200 Mbits/sec 38 .1 us ( 0 % ) 35 .5 Mbits/sec 512 51 .9 us 588 Mbits/sec 38 .9 us ( 0 % ) 142 Mbits/sec 1k 51 .7 us 944 Mbits/sec 37 .2 us ( 0 % ) 279 Mbits/sec 4k 74 .9 us 1 .66 Gbits/sec 39 .9 us ( 0 % ) 1 .20 Gbits/sec ==================================================================================================== ============================== Start Service Network Performance Test ============================== Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 111 us 96 .3 Mbits/sec 88 .4 us ( 0 % ) 7 .59 Mbits/sec 128 83 .7 us 150 Mbits/sec 69 .2 us ( 0 % ) 16 .9 Mbits/sec 512 87 .4 us 374 Mbits/sec 75 .8 us ( 0 % ) 60 .9 Mbits/sec 1k 88 .2 us 521 Mbits/sec 73 .1 us ( 0 % ) 123 Mbits/sec 4k 148 us 813 Mbits/sec 77 .6 us ( 0 .0044% ) 451 Mbits/sec ==================================================================================================== =========================== Start Pod Multicast Network Performance Test =========================== Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .014 ms ( 0 .17% ) 5 .80 Mbits/sec 128 0 .012 ms ( 0 % ) 11 .4 Mbits/sec 512 0 .016 ms ( 0 % ) 46 .1 Mbits/sec 1k 0 .023 ms ( 0 .073% ) 89 .8 Mbits/sec 4k 0 .035 ms ( 1 .3% ) 126 Mbits/sec ==================================================================================================== ============================= Start Host Multicast Network Performance ============================= Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .007 ms ( 0 % ) 9 .95 Mbits/sec 128 0 .005 ms ( 0 % ) 21 .8 Mbits/sec 512 0 .008 ms ( 0 % ) 86 .8 Mbits/sec 1k 0 .013 ms ( 0 .045% ) 168 Mbits/sec 4k 0 .010 ms ( 0 .31% ) 242 Mbits/sec ==================================================================================================== ================================== Start Leader Recover Time Test ================================== Delete ovn central nb pod pod \"ovn-central-5cb9c67d75-tlz9w\" deleted Waiting for ovn central nb pod running =============================== OVN nb Recovery takes 3 .305236803 s ================================ Delete ovn central sb pod pod \"ovn-central-5cb9c67d75-szx4c\" deleted Waiting for ovn central sb pod running =============================== OVN sb Recovery takes 3 .462698535 s ================================ Delete ovn central northd pod pod \"ovn-central-5cb9c67d75-zqmqv\" deleted Waiting for ovn central northd pod running ============================= OVN northd Recovery takes 2 .691291403 s ============================== ==================================================================================================== ================================= Remove Performance Test Resource ================================= rm -f unicast-test-client.log rm -f unicast-test-host-client.log rm -f unicast-test-client.log kubectl ko nbctl lb-del test-server rm -f multicast-test-server.log kubectl exec ovs-ovn-gxdrf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 kubectl exec ovs-ovn-h57bf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 rm -f multicast-test-host-server.log pod \"test-client\" deleted pod \"test-host-client\" deleted pod \"test-host-server\" deleted pod \"test-server\" deleted service \"test-server\" deleted ==================================================================================================== \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kubectl Plugin"},{"location":"en/ops/kubectl-ko/#kubectl-plugin","text":"To facilitate daily operations and maintenance, Kube-OVN provides the kubectl plug-in tool, which allows administrators to perform daily operations through this command. For examples: Check OVN database information and status, OVN database backup and restore, OVS related information, tcpdump specific containers, specific link logical topology, network problem diagnosis and performance optimization.","title":"Kubectl Plugin"},{"location":"en/ops/kubectl-ko/#plugin-installation","text":"Kube-OVN installation will deploy the plugin to each node by default. If the machine that runs kubectl is not in the cluster, or if you need to reinstall the plugin, please refer to the following steps: Download kubectl-ko file: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/kubectl-ko Move file to $PATH : mv kubectl-ko /usr/local/bin/kubectl-ko Add executable permissions: chmod +x /usr/local/bin/kubectl-ko Check if the plugin works properly: # kubectl plugin list The following compatible plugins are available: /usr/local/bin/kubectl-ko","title":"Plugin Installation"},{"location":"en/ops/kubectl-ko/#plugin-usage","text":"Running kubectl ko will show all the available commands and usage descriptions, as follows: # kubectl ko kubectl ko { subcommand } [ option... ] Available Subcommands: [ nb | sb ] [ status | kick | backup | dbstatus | restore ] ovn-db operations show cluster status, kick stale server, backup database, get db consistency status or restore ovn nb db when met 'inconsistent data' error nbctl [ ovn-nbctl options ... ] invoke ovn-nbctl sbctl [ ovn-sbctl options ... ] invoke ovn-sbctl vsctl { nodeName } [ ovs-vsctl options ... ] invoke ovs-vsctl on the specified node ofctl { nodeName } [ ovs-ofctl options ... ] invoke ovs-ofctl on the specified node dpctl { nodeName } [ ovs-dpctl options ... ] invoke ovs-dpctl on the specified node appctl { nodeName } [ ovs-appctl options ... ] invoke ovs-appctl on the specified node tcpdump { namespace/podname } [ tcpdump options ... ] capture pod traffic trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply echo \" diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] diagnose connectivity of all nodes or a specific node or specify subnet's ds pod or IPPorts like 'tcp-172.18.0.2-53,udp-172.18.0.3-53'\" tuning { install-fastpath | local-install-fastpath | remove-fastpath | install-stt | local-install-stt | remove-stt } { centos7 | centos8 }} [ kernel-devel-version ] deploy kernel optimisation components to the system reload restart all kube-ovn components log { kube-ovn | ovn | ovs | linux | all } save log to ./kubectl-ko-log/ perf [ image ] performance test default image is kubeovn/test:v1.12.0 The specific functions and usage of each command are described below.","title":"Plugin Usage"},{"location":"en/ops/kubectl-ko/#nb-sb-status-kick-backup-dbstatus-restore","text":"This subcommand mainly operates on OVN northbound or southbound databases, including database cluster status check, database node offline, database backup, database storage status check and database repair.","title":"[nb | sb] [status | kick | backup | dbstatus | restore]"},{"location":"en/ops/kubectl-ko/#db-cluster-status-check","text":"This command executes ovs-appctl cluster/status on the leader node of the corresponding OVN database to show the cluster status: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 280309 ms ago, reason: timeout Last Election won: 280309 ms ago Election timer: 5000 Log: [ 139 , 139 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-8723 ->8723 <-85d6 ->85d6 Disconnections: 0 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 8723 ( 8723 at tcp: [ 172 .18.0.3 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 138 status: ok If the match_index under Server has a large difference and the last msg time is long, the corresponding Server may not respond for a long time and needs to be checked further.","title":"DB Cluster Status Check"},{"location":"en/ops/kubectl-ko/#db-nodes-offline","text":"This command removes a node from the OVN database and is required when a node is taken offline or replaced. The following is an example of the cluster status from the previous command, to offline the 172.18.0.3 node: # kubectl ko nb kick 8723 started removal Check the database cluster status again to confirm that the node has been removed: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 324356 ms ago, reason: timeout Last Election won: 324356 ms ago Election timer: 5000 Log: [ 140 , 140 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-85d6 ->85d6 Disconnections: 2 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 140 match_index = 139 last msg 848 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 139 status: ok","title":"DB Nodes Offline"},{"location":"en/ops/kubectl-ko/#db-backup","text":"This subcommand backs up the current OVN database locally and can be used for disaster recovery: # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup","title":"DB Backup"},{"location":"en/ops/kubectl-ko/#database-storage-status-check","text":"This command is used to check if the database file is corrupt: # kubectl ko nb dbstatus status: ok If error happens, inconsistent data is displayed and needs to be fixed with the following command.","title":"Database Storage Status Check"},{"location":"en/ops/kubectl-ko/#database-repair","text":"If the database status goes to inconsistent data , this command can be used to repair: # kubectl ko nb restore deployment.apps/ovn-central scaled ovn-central original replicas is 3 first nodeIP is 172 .18.0.5 ovs-ovn pod on node 172 .18.0.5 is ovs-ovn-8jxv9 ovs-ovn pod on node 172 .18.0.3 is ovs-ovn-sjzb6 ovs-ovn pod on node 172 .18.0.4 is ovs-ovn-t87zk backup nb db file restore nb db file, operate in pod ovs-ovn-8jxv9 deployment.apps/ovn-central scaled finish restore nb db file and ovn-central replicas recreate ovs-ovn pods pod \"ovs-ovn-8jxv9\" deleted pod \"ovs-ovn-sjzb6\" deleted pod \"ovs-ovn-t87zk\" deleted","title":"Database Repair"},{"location":"en/ops/kubectl-ko/#nbctl-sbctl-options","text":"This subcommand executes the ovn-nbctl and ovn-sbctl commands directly into the leader node of the OVN northbound or southbound database. For more detailed usage of this command, please refer to the official documentation of the upstream OVN ovn-nbctl(8) \u548c ovn-sbctl(8) \u3002 # kubectl ko nbctl show switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port node-liumengxin-ovn2-192.168.137.177 addresses: [ \"00:00:00:D9:58:B8 100.64.0.3\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port alertmanager-main-0.monitoring addresses: [ \"00:00:00:6C:DF:A3 10.16.0.19\" ] port kube-state-metrics-5d6885d89-4nf8h.monitoring addresses: [ \"00:00:00:6F:02:1C 10.16.0.15\" ] port fake-kubelet-67c55dfd89-pv86k.kube-system addresses: [ \"00:00:00:5C:12:E8 10.16.19.177\" ] port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ]","title":"[nbctl | sbctl] [options ...]"},{"location":"en/ops/kubectl-ko/#vsctl-nodename-options","text":"This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-vsctl command to query and configure vswitchd . For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-vsctl(8) \u3002 # kubectl ko vsctl kube-ovn-01 show 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port mirror0 Interface mirror0 type: internal Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port br-int Interface br-int type: internal ovs_version: \"2.17.2\"","title":"vsctl {nodeName} [options ...]"},{"location":"en/ops/kubectl-ko/#ofctl-nodename-options","text":"This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-ofctl command to query or manage OpenFlow. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-ofctl(8) \u3002 # kubectl ko ofctl kube-ovn-01 dump-flows br-int NXST_FLOW reply ( xid = 0x4 ) : flags =[ more ] cookie = 0xcf3429e6, duration = 671791 .432s, table = 0 , n_packets = 0 , n_bytes = 0 , idle_age = 65534 , hard_age = 65534 , priority = 100 ,in_port = 2 actions = load:0x4->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x1->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xc91413c6, duration = 671791 .431s, table = 0 , n_packets = 907489 , n_bytes = 99978275 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 7 actions = load:0x1->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x4->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xf180459, duration = 671791 .431s, table = 0 , n_packets = 17348582 , n_bytes = 2667811214 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 6317 actions = load:0xa->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x9->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0x7806dd90, duration = 671791 .431s, table = 0 , n_packets = 3235428 , n_bytes = 833821312 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 1 actions = load:0xd->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x3->NXM_NX_REG14 [] ,resubmit ( ,8 ) ...","title":"ofctl {nodeName} [options ...]"},{"location":"en/ops/kubectl-ko/#dpctl-nodename-options","text":"This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-dpctl command to query or manage the OVS datapath. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-dpctl(8) \u3002 # kubectl ko dpctl kube-ovn-01 show system@ovs-system: lookups: hit:350805055 missed:21983648 lost:73 flows: 105 masks: hit:1970748791 total:22 hit/pkt:5.29 port 0 : ovs-system ( internal ) port 1 : ovn0 ( internal ) port 2 : mirror0 ( internal ) port 3 : br-int ( internal ) port 4 : stt_sys_7471 ( stt: packet_type = ptap ) port 5 : eeb4d9e51b5d_h port 6 : a2c1a8a8b83a_h port 7 : 4fa5c4cbb1a5_h","title":"dpctl {nodeName} [options ...]"},{"location":"en/ops/kubectl-ko/#appctl-nodename-options","text":"This command will enter the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-appctl command to operate the associated daemon process. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-appctl(8) \u3002 # kubectl ko appctl kube-ovn-01 vlog/list console syslog file ------- ------ ------ backtrace OFF ERR INFO bfd OFF ERR INFO bond OFF ERR INFO bridge OFF ERR INFO bundle OFF ERR INFO bundles OFF ERR INFO ...","title":"appctl {nodeName} [options ...]"},{"location":"en/ops/kubectl-ko/#tcpdump-namespacepodname-tcpdump-options","text":"This command will enter the kube-ovn-cni container on the machine where namespace/podname is located, and run tcpdump to capture the traffic on the veth NIC of the corresponding container, which can be used to troubleshoot network-related problems. # kubectl ko tcpdump default/ds1-l6n7p icmp + kubectl exec -it kube-ovn-cni-wlg4s -n kube-ovn -- tcpdump -nn -i d7176fe7b4e0_h icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on d7176fe7b4e0_h, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 06 :52:36.619688 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 1 , length 64 06 :52:36.619746 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 1 , length 64 06 :52:37.619588 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 2 , length 64 06 :52:37.619630 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 2 , length 64 06 :52:38.619933 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 3 , length 64 06 :52:38.619973 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 3 , length 64","title":"tcpdump {namespace/podname} [tcpdump options ...]"},{"location":"en/ops/kubectl-ko/#trace-arguments","text":"This command will print the OVN logical flow table and the final Openflow flow table when the Pod/node accesses an address through a specific protocol, so that it make locate flow table related problems during development or troubleshooting much easy. Supported commands: kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } kubectl ko trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } Example: # kubectl ko trace default/ds1-l6n7p 8.8.8.8 icmp + kubectl exec ovn-central-5bc494cb5-np9hm -n kube-ovn -- ovn-trace --ct = new ovn-default 'inport == \"ds1-l6n7p.default\" && ip.ttl == 64 && icmp && eth.src == 0a:00:00:10:00:05 && ip4.src == 10.16.0.4 && eth.dst == 00:00:00:B8:CA:43 && ip4.dst == 8.8.8.8' # icmp,reg14=0xf,vlan_tci=0x0000,dl_src=0a:00:00:10:00:05,dl_dst=00:00:00:b8:ca:43,nw_src=10.16.0.4,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 ingress ( dp = \"ovn-default\" , inport = \"ds1-l6n7p.default\" ) ----------------------------------------------------- 0 . ls_in_port_sec_l2 ( ovn-northd.c:4143 ) : inport == \"ds1-l6n7p.default\" && eth.src == { 0a:00:00:10:00:05 } , priority 50 , uuid 39453393 next ; 1 . ls_in_port_sec_ip ( ovn-northd.c:2898 ) : inport == \"ds1-l6n7p.default\" && eth.src == 0a:00:00:10:00:05 && ip4.src == { 10 .16.0.4 } , priority 90 , uuid 81bcd485 next ; 3 . ls_in_pre_acl ( ovn-northd.c:3269 ) : ip, priority 100 , uuid 7b4f4971 reg0 [ 0 ] = 1 ; next ; 5 . ls_in_pre_stateful ( ovn-northd.c:3396 ) : reg0 [ 0 ] == 1 , priority 100 , uuid 36cdd577 ct_next ; ct_next ( ct_state = new | trk ) ------------------------- 6 . ls_in_acl ( ovn-northd.c:3759 ) : ip && ( !ct.est || ( ct.est && ct_label.blocked == 1 )) , priority 1 , uuid 7608af5b reg0 [ 1 ] = 1 ; next ; 10 . ls_in_stateful ( ovn-northd.c:3995 ) : reg0 [ 1 ] == 1 , priority 100 , uuid 2aba1b90 ct_commit ( ct_label = 0 /0x1 ) ; next ; 16 . ls_in_l2_lkup ( ovn-northd.c:4470 ) : eth.dst == 00 :00:00:b8:ca:43, priority 50 , uuid 5c9c3c9f outport = \"ovn-default-ovn-cluster\" ; output ; ... If the trace object is a virtual machine running in Underlay network, additional parameters is needed to specify the destination Mac address. kubectl ko trace default/virt-handler-7lvml 8 .8.8.8 82 :7c:9f:83:8c:01 icmp","title":"trace [arguments ...]"},{"location":"en/ops/kubectl-ko/#diagnose-allnodesubnetipports-nodenamesubnetnameproto1-ip1-port1proto2-ip2-port2","text":"Diagnose the status of cluster network components and go to the corresponding node's kube-ovn-pinger to detect connectivity and network latency from the current node to other nodes and critical services. # kubectl ko diagnose all switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default port prometheus-k8s-1.monitoring addresses: [ \"00:00:00:AA:37:DF 10.16.0.23\" ] router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] Routing Policies 31000 ip4.dst == 10 .16.0.0/16 allow 31000 ip4.dst == 100 .64.0.0/16 allow 30000 ip4.dst == 192 .168.137.177 reroute 100 .64.0.3 30000 ip4.dst == 192 .168.137.178 reroute 100 .64.0.4 29000 ip4.src == $ovn .default.fake.6_ip4 reroute 100 .64.0.22 29000 ip4.src == $ovn .default.fake.7_ip4 reroute 100 .64.0.21 29000 ip4.src == $ovn .default.fake.8_ip4 reroute 100 .64.0.23 29000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 reroute 100 .64.0.4 20000 ip4.src == $ovn .default.liumengxin.ovn1.192.168.137.176_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.2 20000 ip4.src == $ovn .default.liumengxin.ovn2.192.168.137.177_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.3 20000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.4 IPv4 Routes Route Table <main>: 0 .0.0.0/0 100 .64.0.1 dst-ip UUID LB PROTO VIP IPs e9bcfd9d-793e-4431-9073-6dec96b75d71 cluster-tcp-load tcp 10 .100.209.132:10660 192 .168.137.176:10660 tcp 10 .101.239.192:6641 192 .168.137.177:6641 tcp 10 .101.240.101:3000 10 .16.0.7:3000 tcp 10 .103.184.186:6642 192 .168.137.177:6642 35d2b7a5-e3a7-485a-a4b7-b4970eb0e63b cluster-tcp-sess tcp 10 .100.158.128:8080 10 .16.0.10:8080,10.16.0.5:8080,10.16.63.30:8080 tcp 10 .107.26.215:8080 10 .16.0.19:8080,10.16.0.20:8080,10.16.0.21:8080 tcp 10 .107.26.215:9093 10 .16.0.19:9093,10.16.0.20:9093,10.16.0.21:9093 tcp 10 .98.187.99:8080 10 .16.0.22:8080,10.16.0.23:8080 tcp 10 .98.187.99:9090 10 .16.0.22:9090,10.16.0.23:9090 f43303e4-89aa-4d3e-a3dc-278a552fe27b cluster-udp-load udp 10 .96.0.10:53 10 .16.0.4:53,10.16.0.9:53 _uuid : 06776304 -5a96-43ed-90c4-c4854c251699 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn2_192.168.137.177_underlay_v6 _uuid : 62690625 -87d5-491c-8675-9fd83b1f433c addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn1_192.168.137.176_underlay_v6 _uuid : b03a9bae-94d5-4562-b34c-b5f6198e180b addresses : [ \"10.16.0.0/16\" , \"100.64.0.0/16\" , \"172.22.0.0/16\" , \"192.101.0.0/16\" ] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv4 _uuid : e1056f3a-24cc-4666-8a91-75ee6c3c2426 addresses : [] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv6 _uuid : 3e5d5fff-e670-47b2-a2f5-a39f4698a8c5 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn3_192.168.137.178_underlay_v6 _uuid : 2d85dbdc-d0db-4abe-b19e-cc806d32b492 action : drop direction : from-lport external_ids : {} label : 0 log : false match : \"inport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] _uuid : de790cc8-f155-405f-bb32-5a51f30c545f action : drop direction : to-lport external_ids : {} label : 0 log : false match : \"outport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] Chassis \"e15ed4d4-1780-4d50-b09e-ea8372ed48b8\" hostname: liumengxin-ovn1-192.168.137.176 Encap stt ip: \"192.168.137.176\" options: { csum = \"true\" } Port_Binding node-liumengxin-ovn1-192.168.137.176 Port_Binding perf-6vxkn.default Port_Binding kube-state-metrics-5d6885d89-4nf8h.monitoring Port_Binding alertmanager-main-0.monitoring Port_Binding kube-ovn-pinger-6ftdf.kube-system Port_Binding fake-kubelet-67c55dfd89-pv86k.kube-system Port_Binding prometheus-k8s-0.monitoring Chassis \"eef07da1-f8ad-4775-b14d-bd6a3b4eb0d5\" hostname: liumengxin-ovn3-192.168.137.178 Encap stt ip: \"192.168.137.178\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-7twb4.kube-system Port_Binding prometheus-adapter-86df476d87-rl88g.monitoring Port_Binding prometheus-k8s-1.monitoring Port_Binding node-liumengxin-ovn3-192.168.137.178 Port_Binding perf-ff475.default Port_Binding alertmanager-main-1.monitoring Port_Binding blackbox-exporter-676d976865-tvsjd.monitoring Chassis \"efa253c9-494d-4719-83ae-b48ab0f11c03\" hostname: liumengxin-ovn2-192.168.137.177 Encap stt ip: \"192.168.137.177\" options: { csum = \"true\" } Port_Binding grafana-6c4c6b8fb7-pzd2c.monitoring Port_Binding node-liumengxin-ovn2-192.168.137.177 Port_Binding alertmanager-main-2.monitoring Port_Binding coredns-6789c94dd8-9jqsz.kube-system Port_Binding coredns-6789c94dd8-25d4r.kube-system Port_Binding prometheus-operator-7bbc99fc8b-wgjm4.monitoring Port_Binding prometheus-adapter-86df476d87-gdxmc.monitoring Port_Binding perf-fjnws.default Port_Binding kube-ovn-pinger-vh2xg.kube-system ds kube-proxy ready kube-proxy ready deployment ovn-central ready deployment kube-ovn-controller ready ds kube-ovn-cni ready ds ovs-ovn ready deployment coredns ready ovn-nb leader check ok ovn-sb leader check ok ovn-northd leader check ok ### kube-ovn-controller recent log ### start to diagnose node liumengxin-ovn1-192.168.137.176 #### ovn-controller log: 2022 -06-03T00:56:44.897Z | 16722 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:06:44.912Z | 16723 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:16:44.925Z | 16724 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:26:44.936Z | 16725 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:36:44.959Z | 16726 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:46:44.974Z | 16727 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:56:44.988Z | 16728 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:06:45.001Z | 16729 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:16:45.025Z | 16730 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:26:45.040Z | 16731 | inc_proc_eng | INFO | User triggered force recompute. #### ovs-vswitchd log: 2022 -06-02T23:03:00.137Z | 00079 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:f9d1 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-02T23:23:31.840Z | 00080 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:15b2 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:09:15.659Z | 00081 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:dc:e3:63,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.63.30,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:e5a5 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x150017000004,src = 192 .168.137.178,dst = 192 .168.137.176,ttl = 64 ,tp_src = 9239 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.63.30,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:30:13.409Z | 00064 | dpif ( handler2 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:6b4a with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T02:02:33.832Z | 00082 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:a819 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 #### ovs-vsctl show results: 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port \"04d03360e9a0_h\" Interface \"04d03360e9a0_h\" Port eeb4d9e51b5d_h Interface eeb4d9e51b5d_h Port mirror0 Interface mirror0 type: internal Port \"8e5d887ccd80_h\" Interface \"8e5d887ccd80_h\" Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port \"17512d5be1f1_h\" Interface \"17512d5be1f1_h\" Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" #### pinger diagnose results: I0603 10 :35:04.349404 17619 pinger.go:19 ] ------------------------------------------------------------------------------- Kube-OVN: Version: v1.12.0 Build: 2022 -04-24_08:02:50 Commit: git-73f9d15 Go Version: go1.17.8 Arch: amd64 ------------------------------------------------------------------------------- I0603 10 :35:04.376797 17619 config.go:166 ] pinger config is & { KubeConfigFile: KubeClient:0xc000493380 Port:8080 DaemonSetNamespace:kube-system DaemonSetName:kube-ovn-pinger Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:liumengxin-ovn1-192.168.137.176 HostIP:192.168.137.176 PodName:kube-ovn-pinger-6ftdf PodIP:10.16.0.10 PodProtocols: [ IPv4 ] ExternalAddress: NetworkMode:kube-ovn PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid } I0603 10 :35:04.449166 17619 exporter.go:75 ] liumengxin-ovn1-192.168.137.176: exporter connect successfully I0603 10 :35:04.554011 17619 ovn.go:21 ] ovs-vswitchd and ovsdb are up I0603 10 :35:04.651293 17619 ovn.go:33 ] ovn_controller is up I0603 10 :35:04.651342 17619 ovn.go:39 ] start to check port binding I0603 10 :35:04.749613 17619 ovn.go:135 ] chassis id is 1d7f3d6c-eec5-4b3c-adca-2969d9cdfd80 I0603 10 :35:04.763487 17619 ovn.go:49 ] port in sb is [ node-liumengxin-ovn1-192.168.137.176 perf-6vxkn.default kube-state-metrics-5d6885d89-4nf8h.monitoring alertmanager-main-0.monitoring kube-ovn-pinger-6ftdf.kube-system fake-kubelet-67c55dfd89-pv86k.kube-system prometheus-k8s-0.monitoring ] I0603 10 :35:04.763583 17619 ovn.go:61 ] ovs and ovn-sb binding check passed I0603 10 :35:05.049309 17619 ping.go:259 ] start to check apiserver connectivity I0603 10 :35:05.053666 17619 ping.go:268 ] connect to apiserver success in 4 .27ms I0603 10 :35:05.053786 17619 ping.go:129 ] start to check pod connectivity I0603 10 :35:05.249590 17619 ping.go:159 ] ping pod: kube-ovn-pinger-6ftdf 10 .16.0.10, count: 3 , loss count 0 , average rtt 16 .30ms I0603 10 :35:05.354135 17619 ping.go:159 ] ping pod: kube-ovn-pinger-7twb4 10 .16.63.30, count: 3 , loss count 0 , average rtt 1 .81ms I0603 10 :35:05.458460 17619 ping.go:159 ] ping pod: kube-ovn-pinger-vh2xg 10 .16.0.5, count: 3 , loss count 0 , average rtt 1 .92ms I0603 10 :35:05.458523 17619 ping.go:83 ] start to check node connectivity If the target of diagnose is specified as subnet, the script will create a daemonset on the subnet, and kube-ovn-pinger will detect the connectivity and network delay of all pods in this daemonset, and automatically destroy the daemonset after the test. If the target of diagnose is specified as IPPorts, the script will let each kube-ovn-pinger pod detect whether the target protocol, IP, and Port are reachable.","title":"diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}]"},{"location":"en/ops/kubectl-ko/#tuning-install-fastpathlocal-install-fastpathremove-fastpathinstall-sttlocal-install-sttremove-stt-centos7centos8-kernel-devel-version","text":"This command performs performance tuning related operations, please refer to Performance Tunning .","title":"tuning {install-fastpath|local-install-fastpath|remove-fastpath|install-stt|local-install-stt|remove-stt} {centos7|centos8}} [kernel-devel-version]"},{"location":"en/ops/kubectl-ko/#reload","text":"This command restarts all Kube-OVN related components: # kubectl ko reload pod \"ovn-central-8684dd94bd-vzgcr\" deleted Waiting for deployment \"ovn-central\" rollout to finish: 0 of 1 updated replicas are available... deployment \"ovn-central\" successfully rolled out pod \"ovs-ovn-bsnvz\" deleted pod \"ovs-ovn-m9b98\" deleted pod \"kube-ovn-controller-8459db5ff4-64c62\" deleted Waiting for deployment \"kube-ovn-controller\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out pod \"kube-ovn-cni-2klnh\" deleted pod \"kube-ovn-cni-t2jz4\" deleted Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-cni\" successfully rolled out pod \"kube-ovn-pinger-ln72z\" deleted pod \"kube-ovn-pinger-w8lrk\" deleted Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-pinger\" successfully rolled out pod \"kube-ovn-monitor-7fb67d5488-7q6zb\" deleted Waiting for deployment \"kube-ovn-monitor\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-monitor\" successfully rolled out","title":"reload"},{"location":"en/ops/kubectl-ko/#log","text":"Using this command will capture the logs of Kube-OVN, OVN, Openvswitch on all nodes of kube-ovn and some debug information commonly used in linux. # kubectl ko log all Collecting kube-ovn logging files Collecting ovn logging files Collecting openvswitch logging files Collecting linux dmesg files Collecting linux iptables-legacy files Collecting linux iptables-nft files Collecting linux route files Collecting linux link files Collecting linux neigh files Collecting linux memory files Collecting linux top files Collecting linux sysctl files Collecting linux netstat files Collecting linux addr files Collecting linux ipset files Collecting linux tcp files Collected files have been saved in the directory /root/kubectl-ko-log The directory is as follows: # tree kubectl-ko-log/ kubectl-ko-log/ | -- kube-ovn-control-plane | | -- kube-ovn | | | -- kube-ovn-cni.log | | | -- kube-ovn-monitor.log | | ` -- kube-ovn-pinger.log | | -- linux | | | -- addr.log | | | -- dmesg.log | | | -- ipset.log | | | -- iptables-legacy.log | | | -- iptables-nft.log | | | -- link.log | | | -- memory.log | | | -- neigh.log | | | -- netstat.log | | | -- route.log | | | -- sysctl.log | | | -- tcp.log | | ` -- top.log | | -- openvswitch | | | -- ovs-vswitchd.log | | ` -- ovsdb-server.log | ` -- ovn | | -- ovn-controller.log | | -- ovn-northd.log | | -- ovsdb-server-nb.log | ` -- ovsdb-server-sb.log","title":"log"},{"location":"en/ops/kubectl-ko/#perf-image","text":"This command will test some performance indicators of Kube-OVN as follows: The performance indicators of the container network; Hostnetwork network performance indicators; Container network multicast packet performance indicators; Time required for OVN-NB, OVN-SB, and OVN-Northd leader deletion recovery. The parameter image is used to specify the image used by the performance test pod. By default, it is kubeovn/test:v1.12.0 . This parameter is mainly set for offline scenarios, and the image name may change when the image is pulled to the intranet environment. # kubectl ko perf ============================== Prepareing Performance Test Resources =============================== pod/test-client created pod/test-host-client created pod/test-server created pod/test-host-server created service/test-server created pod/test-client condition met pod/test-host-client condition met pod/test-host-server condition met pod/test-server condition met ==================================================================================================== ============================ Start Pod Network Unicast Performance Test ============================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 82 .8 us 97 .7 Mbits/sec 67 .6 us ( 0 % ) 8 .42 Mbits/sec 128 85 .4 us 167 Mbits/sec 67 .2 us ( 0 % ) 17 .2 Mbits/sec 512 85 .8 us 440 Mbits/sec 68 .7 us ( 0 % ) 68 .4 Mbits/sec 1k 85 .1 us 567 Mbits/sec 68 .7 us ( 0 % ) 134 Mbits/sec 4k 138 us 826 Mbits/sec 78 .1 us ( 1 .4% ) 503 Mbits/sec ==================================================================================================== =============================== Start Host Network Performance Test ================================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 49 .7 us 120 Mbits/sec 37 .9 us ( 0 % ) 18 .6 Mbits/sec 128 49 .7 us 200 Mbits/sec 38 .1 us ( 0 % ) 35 .5 Mbits/sec 512 51 .9 us 588 Mbits/sec 38 .9 us ( 0 % ) 142 Mbits/sec 1k 51 .7 us 944 Mbits/sec 37 .2 us ( 0 % ) 279 Mbits/sec 4k 74 .9 us 1 .66 Gbits/sec 39 .9 us ( 0 % ) 1 .20 Gbits/sec ==================================================================================================== ============================== Start Service Network Performance Test ============================== Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 111 us 96 .3 Mbits/sec 88 .4 us ( 0 % ) 7 .59 Mbits/sec 128 83 .7 us 150 Mbits/sec 69 .2 us ( 0 % ) 16 .9 Mbits/sec 512 87 .4 us 374 Mbits/sec 75 .8 us ( 0 % ) 60 .9 Mbits/sec 1k 88 .2 us 521 Mbits/sec 73 .1 us ( 0 % ) 123 Mbits/sec 4k 148 us 813 Mbits/sec 77 .6 us ( 0 .0044% ) 451 Mbits/sec ==================================================================================================== =========================== Start Pod Multicast Network Performance Test =========================== Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .014 ms ( 0 .17% ) 5 .80 Mbits/sec 128 0 .012 ms ( 0 % ) 11 .4 Mbits/sec 512 0 .016 ms ( 0 % ) 46 .1 Mbits/sec 1k 0 .023 ms ( 0 .073% ) 89 .8 Mbits/sec 4k 0 .035 ms ( 1 .3% ) 126 Mbits/sec ==================================================================================================== ============================= Start Host Multicast Network Performance ============================= Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .007 ms ( 0 % ) 9 .95 Mbits/sec 128 0 .005 ms ( 0 % ) 21 .8 Mbits/sec 512 0 .008 ms ( 0 % ) 86 .8 Mbits/sec 1k 0 .013 ms ( 0 .045% ) 168 Mbits/sec 4k 0 .010 ms ( 0 .31% ) 242 Mbits/sec ==================================================================================================== ================================== Start Leader Recover Time Test ================================== Delete ovn central nb pod pod \"ovn-central-5cb9c67d75-tlz9w\" deleted Waiting for ovn central nb pod running =============================== OVN nb Recovery takes 3 .305236803 s ================================ Delete ovn central sb pod pod \"ovn-central-5cb9c67d75-szx4c\" deleted Waiting for ovn central sb pod running =============================== OVN sb Recovery takes 3 .462698535 s ================================ Delete ovn central northd pod pod \"ovn-central-5cb9c67d75-zqmqv\" deleted Waiting for ovn central northd pod running ============================= OVN northd Recovery takes 2 .691291403 s ============================== ==================================================================================================== ================================= Remove Performance Test Resource ================================= rm -f unicast-test-client.log rm -f unicast-test-host-client.log rm -f unicast-test-client.log kubectl ko nbctl lb-del test-server rm -f multicast-test-server.log kubectl exec ovs-ovn-gxdrf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 kubectl exec ovs-ovn-h57bf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 rm -f multicast-test-host-server.log pod \"test-client\" deleted pod \"test-host-client\" deleted pod \"test-host-server\" deleted pod \"test-server\" deleted service \"test-server\" deleted ==================================================================================================== \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"perf [image]"},{"location":"en/ops/recover-db/","text":"OVN DB Backup and Recovery \u00b6 This document describes how to perform database backups and how to perform cluster recovery from existing database files in different situations. Database Backup \u00b6 The database files can be backed up for recovery in case of failure. Use the backup command of the kubectl plugin: # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup # kubectl ko sb backup tar: Removing leading `/' from member names backup ovn-nb db to /root/ovnsb_db.060223191654183154.backup Cluster Partial Nodes Failure Recovery \u00b6 If some nodes in the cluster are working abnormally due to power failure, file system failure or lack of disk space, but the cluster is still working normally, you can recover it by following the steps below. Check the Logs to Confirm Status \u00b6 Check the log in /var/log/ovn/ovn-northd.log , if it shows similar error as follows, you can make sue that there is an exception in the database: * ovn-northd is not running ovsdb-server: ovsdb error: error reading record 2739 from OVN_Northbound log: record 2739 advances commit index to 6308 but last log index is 6307 * Starting ovsdb-nb Kick Node from Cluster \u00b6 Select the corresponding database for the operation based on whether the log prompt is OVN_Northbound or OVN_Southbound . The above log prompt is OVN_Northbound then for ovn-nb do the following: # kubectl ko nb status 9182 Name: OVN_Northbound Cluster ID: e75f ( e75fa340-49ed-45ab-990e-26cb865ebc85 ) Server ID: 9182 ( 9182e8dd-b5b0-4dd8-8518-598cc1e374f3 ) Address: tcp: [ 10 .0.128.61 ] :6643 Status: cluster member Role: leader Term: 1454 Leader: self Vote: self Last Election started 1732603 ms ago, reason: timeout Last Election won: 1732587 ms ago Election timer: 1000 Log: [ 7332 , 12512 ] Entries not yet committed: 1 Entries not yet applied: 1 Connections: ->f080 <-f080 <-e631 ->e631 Disconnections: 1 Servers: f080 ( f080 at tcp: [ 10 .0.129.139 ] :6643 ) next_index = 12512 match_index = 12510 last msg 63 ms ago 9182 ( 9182 at tcp: [ 10 .0.128.61 ] :6643 ) ( self ) next_index = 10394 match_index = 12510 e631 ( e631 at tcp: [ 10 .0.131.173 ] :6643 ) next_index = 12512 match_index = 0 Kick abnormal nodes from the cluster: kubectl ko nb kick e631 Log in to the abnormal node and delete the database file: mv /etc/origin/ovn/ovnnb_db.db /tmp Delete the ovn-central pod of the corresponding node and wait for the cluster to recover\uff1a kubectl delete pod -n kube-system ovn-central-xxxx Recover when Total Cluster Failed \u00b6 If the majority of the cluster nodes are broken and the leader cannot be elected, please refer to the following steps to recover. Stop ovn-central \u00b6 Record the current replicas of ovn-central and stop ovn-central to avoid new database changes that affect recovery: kubectl scale deployment -n kube-system ovn-central --replicas = 0 Select a Backup \u00b6 As most of the nodes are damaged, the cluster needs to be rebuilt by recovering from one of the database files. If you have previously backed up the database you can use the previous backup file to restore it. If not you can use the following steps to generate a backup from an existing file. Since the database file in the default folder is a cluster format database file containing information about the current cluster, you can't rebuild the database directly with this file, you need to use ovsdb-tool cluster-to-standalone to convert the format. Select the first node in the ovn-central environment variable NODE_IPS to restore the database files. If the database file of the first node is corrupted, copy the file from the other machine /etc/origin/ovn to the first machine. Run the following command to generate a database file backup. docker run -it -v /etc/origin/ovn:/etc/ovn kubeovn/kube-ovn:v1.12.0 bash cd /etc/ovn/ ovsdb-tool cluster-to-standalone ovnnb_db_standalone.db ovnnb_db.db ovsdb-tool cluster-to-standalone ovnsb_db_standalone.db ovnsb_db.db Delete the Database Files on All ovn-central Nodes \u00b6 In order to avoid rebuilding the cluster with the wrong data, the existing database files need to be cleaned up: mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp Recovering Database Cluster \u00b6 Rename the backup databases to ovnnb_db.db and ovnsb_db.db respectively, and copy them to the /etc/origin/ovn/ directory of the first machine in the ovn-central environment variable NODE_IPS \uff1a mv /etc/origin/ovn/ovnnb_db_standalone.db /etc/origin/ovn/ovnnb_db.db mv /etc/origin/ovn/ovnsb_db_standalone.db /etc/origin/ovn/ovnsb_db.db Restore the number of replicas of ovn-central \uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl rollout status deployment/ovn-central -n kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN DB Backup and Recovery"},{"location":"en/ops/recover-db/#ovn-db-backup-and-recovery","text":"This document describes how to perform database backups and how to perform cluster recovery from existing database files in different situations.","title":"OVN DB Backup and Recovery"},{"location":"en/ops/recover-db/#database-backup","text":"The database files can be backed up for recovery in case of failure. Use the backup command of the kubectl plugin: # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup # kubectl ko sb backup tar: Removing leading `/' from member names backup ovn-nb db to /root/ovnsb_db.060223191654183154.backup","title":"Database Backup"},{"location":"en/ops/recover-db/#cluster-partial-nodes-failure-recovery","text":"If some nodes in the cluster are working abnormally due to power failure, file system failure or lack of disk space, but the cluster is still working normally, you can recover it by following the steps below.","title":"Cluster Partial Nodes Failure Recovery"},{"location":"en/ops/recover-db/#check-the-logs-to-confirm-status","text":"Check the log in /var/log/ovn/ovn-northd.log , if it shows similar error as follows, you can make sue that there is an exception in the database: * ovn-northd is not running ovsdb-server: ovsdb error: error reading record 2739 from OVN_Northbound log: record 2739 advances commit index to 6308 but last log index is 6307 * Starting ovsdb-nb","title":"Check the Logs to Confirm Status"},{"location":"en/ops/recover-db/#kick-node-from-cluster","text":"Select the corresponding database for the operation based on whether the log prompt is OVN_Northbound or OVN_Southbound . The above log prompt is OVN_Northbound then for ovn-nb do the following: # kubectl ko nb status 9182 Name: OVN_Northbound Cluster ID: e75f ( e75fa340-49ed-45ab-990e-26cb865ebc85 ) Server ID: 9182 ( 9182e8dd-b5b0-4dd8-8518-598cc1e374f3 ) Address: tcp: [ 10 .0.128.61 ] :6643 Status: cluster member Role: leader Term: 1454 Leader: self Vote: self Last Election started 1732603 ms ago, reason: timeout Last Election won: 1732587 ms ago Election timer: 1000 Log: [ 7332 , 12512 ] Entries not yet committed: 1 Entries not yet applied: 1 Connections: ->f080 <-f080 <-e631 ->e631 Disconnections: 1 Servers: f080 ( f080 at tcp: [ 10 .0.129.139 ] :6643 ) next_index = 12512 match_index = 12510 last msg 63 ms ago 9182 ( 9182 at tcp: [ 10 .0.128.61 ] :6643 ) ( self ) next_index = 10394 match_index = 12510 e631 ( e631 at tcp: [ 10 .0.131.173 ] :6643 ) next_index = 12512 match_index = 0 Kick abnormal nodes from the cluster: kubectl ko nb kick e631 Log in to the abnormal node and delete the database file: mv /etc/origin/ovn/ovnnb_db.db /tmp Delete the ovn-central pod of the corresponding node and wait for the cluster to recover\uff1a kubectl delete pod -n kube-system ovn-central-xxxx","title":"Kick Node from Cluster"},{"location":"en/ops/recover-db/#recover-when-total-cluster-failed","text":"If the majority of the cluster nodes are broken and the leader cannot be elected, please refer to the following steps to recover.","title":"Recover when Total Cluster Failed"},{"location":"en/ops/recover-db/#stop-ovn-central","text":"Record the current replicas of ovn-central and stop ovn-central to avoid new database changes that affect recovery: kubectl scale deployment -n kube-system ovn-central --replicas = 0","title":"Stop ovn-central"},{"location":"en/ops/recover-db/#select-a-backup","text":"As most of the nodes are damaged, the cluster needs to be rebuilt by recovering from one of the database files. If you have previously backed up the database you can use the previous backup file to restore it. If not you can use the following steps to generate a backup from an existing file. Since the database file in the default folder is a cluster format database file containing information about the current cluster, you can't rebuild the database directly with this file, you need to use ovsdb-tool cluster-to-standalone to convert the format. Select the first node in the ovn-central environment variable NODE_IPS to restore the database files. If the database file of the first node is corrupted, copy the file from the other machine /etc/origin/ovn to the first machine. Run the following command to generate a database file backup. docker run -it -v /etc/origin/ovn:/etc/ovn kubeovn/kube-ovn:v1.12.0 bash cd /etc/ovn/ ovsdb-tool cluster-to-standalone ovnnb_db_standalone.db ovnnb_db.db ovsdb-tool cluster-to-standalone ovnsb_db_standalone.db ovnsb_db.db","title":"Select a Backup"},{"location":"en/ops/recover-db/#delete-the-database-files-on-all-ovn-central-nodes","text":"In order to avoid rebuilding the cluster with the wrong data, the existing database files need to be cleaned up: mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp","title":"Delete the Database Files on All ovn-central Nodes"},{"location":"en/ops/recover-db/#recovering-database-cluster","text":"Rename the backup databases to ovnnb_db.db and ovnsb_db.db respectively, and copy them to the /etc/origin/ovn/ directory of the first machine in the ovn-central environment variable NODE_IPS \uff1a mv /etc/origin/ovn/ovnnb_db_standalone.db /etc/origin/ovn/ovnnb_db.db mv /etc/origin/ovn/ovnsb_db_standalone.db /etc/origin/ovn/ovnsb_db.db Restore the number of replicas of ovn-central \uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl rollout status deployment/ovn-central -n kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Recovering Database Cluster"},{"location":"en/reference/architecture/","text":"Architecture \u00b6 This document describes the general architecture of Kube-OVN, the functionality of each component and how they interact with each other. Overall, Kube-OVN serves as a bridge between Kubernetes and OVN, combining proven SDN with Cloud Native. This means that Kube-OVN not only implements network specifications under Kubernetes, such as CNI, Service and Networkpolicy, but also brings a large number of SDN domain capabilities to cloud-native, such as logical switches, logical routers, VPCs, gateways, QoS, ACLs and traffic mirroring. Kube-OVN also maintains a good openness to integrate with many technology solutions, such as Cilium, Submariner, Prometheus, KubeVirt, etc. Component Introduction \u00b6 The components of Kube-OVN can be broadly divided into three categories. Upstream OVN/OVS components. Core Controller and Agent. Monitoring, operation and maintenance tools and extension components. Upstream OVN/OVS Components \u00b6 This type of component comes from the OVN/OVS community with specific modifications for Kube-OVN usage scenarios. OVN/OVS itself is a mature SDN system for managing virtual machines and containers, and we strongly recommend that users interested in the Kube-OVN implementation read ovn-architecture(7) first to understand what OVN is and how to integrate with it. Kube-OVN uses the northbound interface of OVN to create and coordinate virtual networks and map the network concepts into Kubernetes. All OVN/OVS-related components have been packaged into images and are ready to run in Kubernetes. ovn-central \u00b6 The ovn-central Deployment runs the control plane components of OVN, including ovn-nb , ovn-sb , and ovn-northd . ovn-nb : Saves the virtual network configuration and provides an API for virtual network management. kube-ovn-controller will mainly interact with ovn-nb to configure the virtual network. ovn-sb : Holds the logical flow table generated from the logical network of ovn-nb , as well as the actual physical network state of each node. ovn-northd : translates the virtual network of ovn-nb into a logical flow table in ovn-sb . Multiple instances of ovn-central will synchronize data via the Raft protocol to ensure high availability. ovs-ovn \u00b6 ovs-ovn runs as a DaemonSet on each node, with openvswitch , ovsdb , and ovn-controller running inside the Pod. These components act as agents for ovn-central to translate logical flow tables into real network configurations. Core Controller and Agent \u00b6 This part is the core component of Kube-OVN, serving as a bridge between OVN and Kubernetes, bridging the two systems and translating network concepts between them. Most of the core functions are implemented in these components. kube-ovn-controller \u00b6 This component performs the translation of all resources within Kubernetes to OVN resources and acts as the control plane for the entire Kube-OVN system. The kube-ovn-controller listens for events on all resources related to network functionality and updates the logical network within the OVN based on resource changes. The main resources listened including: Pod\uff0cService\uff0cEndpoint\uff0cNode\uff0cNetworkPolicy\uff0cVPC\uff0cSubnet\uff0cVlan\uff0cProviderNetwork\u3002 Taking the Pod event as an example, kube-ovn-controller listens to the Pod creation event, allocates the address via the built-in in-memory IPAM function, and calls ovn-central to create logical ports, static routes and possible ACL rules. Next, kube-ovn-controller writes the assigned address and subnet information such as CIDR, gateway, route, etc. to the annotation of the Pod. This annotation is then read by kube-ovn-cni and used to configure the local network. kube-ovn-cni \u00b6 This component runs on each node as a DaemonSet, implements the CNI interface, and operates the local OVS to configure the local network. This DaemonSet copies the kube-ovn binary to each machine as a tool for interaction between kubelet and kube-ovn-cni . This binary sends the corresponding CNI request to kube-ovn-cni for further operation. The binary will be copied to the /opt/cni/bin directory by default. kube-ovn-cni will configure the specific network to perform the appropriate traffic operations, and the main tasks including: Config ovn-controller and vswitchd . Handle CNI Add/Del requests: Create or delete veth pair and bind or unbind to OVS ports. Configure OVS ports Update host iptables/ipset/route rules. Dynamically update the network QoS. Create and configure the ovn0 NIC to connect the container network and the host network. Configure the host NIC to implement Vlan/Underlay/EIP. Dynamically config inter-cluster gateways. Monitoring, Operation and Maintenance Tools and Extension Components \u00b6 These components provide monitoring, diagnostics, operations tools, and external interface to extend the core network capabilities of Kube-OVN and simplify daily operations and maintenance. kube-ovn-speaker \u00b6 This component is a DaemonSet running on a specific labeled nodes that publish routes to the external, allowing external access to the container directly through the Pod IP. For more information on how to use it, please refer to BGP Support . kube-ovn-pinger \u00b6 This component is a DaemonSet running on each node to collect OVS status information, node network quality, network latency, etc. The monitoring metrics collected can be found in Metrics . kube-ovn-monitor \u00b6 This component collects OVN status information and the monitoring metrics, all metrics can be found in Metrics . kubectl-ko \u00b6 This component is a kubectl plugin, which can quickly run common operations, for more usage, please refer to [kubectl plugin].(../ops/kubectl-ko.en.md)\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Architecture"},{"location":"en/reference/architecture/#architecture","text":"This document describes the general architecture of Kube-OVN, the functionality of each component and how they interact with each other. Overall, Kube-OVN serves as a bridge between Kubernetes and OVN, combining proven SDN with Cloud Native. This means that Kube-OVN not only implements network specifications under Kubernetes, such as CNI, Service and Networkpolicy, but also brings a large number of SDN domain capabilities to cloud-native, such as logical switches, logical routers, VPCs, gateways, QoS, ACLs and traffic mirroring. Kube-OVN also maintains a good openness to integrate with many technology solutions, such as Cilium, Submariner, Prometheus, KubeVirt, etc.","title":"Architecture"},{"location":"en/reference/architecture/#component-introduction","text":"The components of Kube-OVN can be broadly divided into three categories. Upstream OVN/OVS components. Core Controller and Agent. Monitoring, operation and maintenance tools and extension components.","title":"Component Introduction"},{"location":"en/reference/architecture/#upstream-ovnovs-components","text":"This type of component comes from the OVN/OVS community with specific modifications for Kube-OVN usage scenarios. OVN/OVS itself is a mature SDN system for managing virtual machines and containers, and we strongly recommend that users interested in the Kube-OVN implementation read ovn-architecture(7) first to understand what OVN is and how to integrate with it. Kube-OVN uses the northbound interface of OVN to create and coordinate virtual networks and map the network concepts into Kubernetes. All OVN/OVS-related components have been packaged into images and are ready to run in Kubernetes.","title":"Upstream OVN/OVS Components"},{"location":"en/reference/architecture/#ovn-central","text":"The ovn-central Deployment runs the control plane components of OVN, including ovn-nb , ovn-sb , and ovn-northd . ovn-nb : Saves the virtual network configuration and provides an API for virtual network management. kube-ovn-controller will mainly interact with ovn-nb to configure the virtual network. ovn-sb : Holds the logical flow table generated from the logical network of ovn-nb , as well as the actual physical network state of each node. ovn-northd : translates the virtual network of ovn-nb into a logical flow table in ovn-sb . Multiple instances of ovn-central will synchronize data via the Raft protocol to ensure high availability.","title":"ovn-central"},{"location":"en/reference/architecture/#ovs-ovn","text":"ovs-ovn runs as a DaemonSet on each node, with openvswitch , ovsdb , and ovn-controller running inside the Pod. These components act as agents for ovn-central to translate logical flow tables into real network configurations.","title":"ovs-ovn"},{"location":"en/reference/architecture/#core-controller-and-agent","text":"This part is the core component of Kube-OVN, serving as a bridge between OVN and Kubernetes, bridging the two systems and translating network concepts between them. Most of the core functions are implemented in these components.","title":"Core Controller and Agent"},{"location":"en/reference/architecture/#kube-ovn-controller","text":"This component performs the translation of all resources within Kubernetes to OVN resources and acts as the control plane for the entire Kube-OVN system. The kube-ovn-controller listens for events on all resources related to network functionality and updates the logical network within the OVN based on resource changes. The main resources listened including: Pod\uff0cService\uff0cEndpoint\uff0cNode\uff0cNetworkPolicy\uff0cVPC\uff0cSubnet\uff0cVlan\uff0cProviderNetwork\u3002 Taking the Pod event as an example, kube-ovn-controller listens to the Pod creation event, allocates the address via the built-in in-memory IPAM function, and calls ovn-central to create logical ports, static routes and possible ACL rules. Next, kube-ovn-controller writes the assigned address and subnet information such as CIDR, gateway, route, etc. to the annotation of the Pod. This annotation is then read by kube-ovn-cni and used to configure the local network.","title":"kube-ovn-controller"},{"location":"en/reference/architecture/#kube-ovn-cni","text":"This component runs on each node as a DaemonSet, implements the CNI interface, and operates the local OVS to configure the local network. This DaemonSet copies the kube-ovn binary to each machine as a tool for interaction between kubelet and kube-ovn-cni . This binary sends the corresponding CNI request to kube-ovn-cni for further operation. The binary will be copied to the /opt/cni/bin directory by default. kube-ovn-cni will configure the specific network to perform the appropriate traffic operations, and the main tasks including: Config ovn-controller and vswitchd . Handle CNI Add/Del requests: Create or delete veth pair and bind or unbind to OVS ports. Configure OVS ports Update host iptables/ipset/route rules. Dynamically update the network QoS. Create and configure the ovn0 NIC to connect the container network and the host network. Configure the host NIC to implement Vlan/Underlay/EIP. Dynamically config inter-cluster gateways.","title":"kube-ovn-cni"},{"location":"en/reference/architecture/#monitoring-operation-and-maintenance-tools-and-extension-components","text":"These components provide monitoring, diagnostics, operations tools, and external interface to extend the core network capabilities of Kube-OVN and simplify daily operations and maintenance.","title":"Monitoring, Operation and Maintenance Tools and Extension Components"},{"location":"en/reference/architecture/#kube-ovn-speaker","text":"This component is a DaemonSet running on a specific labeled nodes that publish routes to the external, allowing external access to the container directly through the Pod IP. For more information on how to use it, please refer to BGP Support .","title":"kube-ovn-speaker"},{"location":"en/reference/architecture/#kube-ovn-pinger","text":"This component is a DaemonSet running on each node to collect OVS status information, node network quality, network latency, etc. The monitoring metrics collected can be found in Metrics .","title":"kube-ovn-pinger"},{"location":"en/reference/architecture/#kube-ovn-monitor","text":"This component collects OVN status information and the monitoring metrics, all metrics can be found in Metrics .","title":"kube-ovn-monitor"},{"location":"en/reference/architecture/#kubectl-ko","text":"This component is a kubectl plugin, which can quickly run common operations, for more usage, please refer to [kubectl plugin].(../ops/kubectl-ko.en.md)\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kubectl-ko"},{"location":"en/reference/dev-env/","text":"Development Setup \u00b6 Environmental Preparation \u00b6 Kube-OVN uses Golang 1.20 to develop and Go Modules to manage dependency, please check env GO111MODULE=\"on\" \u3002 gosec is used to scan for code security related issues and requires to be installed in the development environment: go get github.com/securego/gosec/v2/cmd/gosec To reduce the size of the final generated image, Kube-OVN uses some of the Docker buildx experimental features, please update Docker to the latest version and enable buildx: docker buildx create --use Build Image \u00b6 Use the following command to download the code and generate the image required to run Kube-OVN: git clone https://github.com/kubeovn/kube-ovn.git cd kube-ovn make release To build an image to run in an ARM environment, run the following command: make release-arm Building the Base Image \u00b6 If you need to change the operating system version, dependencies, OVS/OVN code, etc., you need to rebuild the base image. The Dockerfile used for the base image is dist/images/Dockerfile.base . Build instructions: # build x86 base image make base-amd64 # build arm base image make base-arm64 Run E2E \u00b6 Kube-OVN uses KIND to build local Kubernetes cluster, j2cli to render templates, and Ginkgo to run test cases. Please refer to the relevant documentation for dependency installation. Run E2E locally: make kind-init make kind-install make e2e To run the Underlay E2E test, run the following commands: make kind-init make kind-install-underlay make e2e-underlay-single-nic To run the ovn vpc nat gw eip, fip, snat, dnat E2E test, run the following commands: make kind-init make kind-install make ovn-vpc-nat-gw-conformance-e2e To run the iptables vpc nat gw eip, fip, snat, dnat E2E test, run the following commands: make kind-init make kind-install make kind-install-vpc-nat-gw make iptables-vpc-nat-gw-conformance-e2e To run the loadbalancer service E2E test, run the following commands: make kind-init make kind-install make kind-install-lb-svc make kube-ovn-lb-svc-conformance-e2e To clean, run the following commands: make kind-clean \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Development Setup"},{"location":"en/reference/dev-env/#development-setup","text":"","title":"Development Setup"},{"location":"en/reference/dev-env/#environmental-preparation","text":"Kube-OVN uses Golang 1.20 to develop and Go Modules to manage dependency, please check env GO111MODULE=\"on\" \u3002 gosec is used to scan for code security related issues and requires to be installed in the development environment: go get github.com/securego/gosec/v2/cmd/gosec To reduce the size of the final generated image, Kube-OVN uses some of the Docker buildx experimental features, please update Docker to the latest version and enable buildx: docker buildx create --use","title":"Environmental Preparation"},{"location":"en/reference/dev-env/#build-image","text":"Use the following command to download the code and generate the image required to run Kube-OVN: git clone https://github.com/kubeovn/kube-ovn.git cd kube-ovn make release To build an image to run in an ARM environment, run the following command: make release-arm","title":"Build Image"},{"location":"en/reference/dev-env/#building-the-base-image","text":"If you need to change the operating system version, dependencies, OVS/OVN code, etc., you need to rebuild the base image. The Dockerfile used for the base image is dist/images/Dockerfile.base . Build instructions: # build x86 base image make base-amd64 # build arm base image make base-arm64","title":"Building the Base Image"},{"location":"en/reference/dev-env/#run-e2e","text":"Kube-OVN uses KIND to build local Kubernetes cluster, j2cli to render templates, and Ginkgo to run test cases. Please refer to the relevant documentation for dependency installation. Run E2E locally: make kind-init make kind-install make e2e To run the Underlay E2E test, run the following commands: make kind-init make kind-install-underlay make e2e-underlay-single-nic To run the ovn vpc nat gw eip, fip, snat, dnat E2E test, run the following commands: make kind-init make kind-install make ovn-vpc-nat-gw-conformance-e2e To run the iptables vpc nat gw eip, fip, snat, dnat E2E test, run the following commands: make kind-init make kind-install make kind-install-vpc-nat-gw make iptables-vpc-nat-gw-conformance-e2e To run the loadbalancer service E2E test, run the following commands: make kind-init make kind-install make kind-install-lb-svc make kube-ovn-lb-svc-conformance-e2e To clean, run the following commands: make kind-clean \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Run E2E"},{"location":"en/reference/document-convention/","text":"\u6587\u6863\u89c4\u8303 \u00b6 \u4e3a\u4e86\u4fdd\u8bc1\u6587\u6863\u98ce\u683c\u4e00\u81f4\uff0c\u8bf7\u5728\u63d0\u4ea4\u6587\u6863\u65f6\u9075\u5faa\u4e0b\u5217\u7684\u98ce\u683c\u89c4\u8303\u3002 \u6807\u70b9 \u00b6 \u4e2d\u6587\u6587\u6863\u4e2d\u6587\u672c\u5185\u5bb9\u6240\u6709\u6807\u70b9\u5e94\u4f7f\u7528\u4e2d\u6587\u683c\u5f0f\u6807\u70b9\uff0c\u82f1\u6587\u6587\u6863\u4e2d\u6240\u6709\u6587\u672c\u5185\u5bb9\u4e2d\u5e94\u4f7f\u7528\u82f1\u6587\u6807\u70b9\u3002 Bad Good \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c,\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528,\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc. \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002 \u82f1\u6587\u6570\u5b57\u548c\u4e2d\u6587\u5e94\u8be5\u7528\u7a7a\u683c\u8fdb\u884c\u5206\u9694\u3002 Bad Good Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 \u793a\u4f8b\u5185\u5bb9\u5e94\u8be5\u4ee5 \uff1a \u5f00\u542f\uff0c\u5176\u4ed6\u53e5\u5c3e\u9700\u8981\u7528 \u3002 \u7ed3\u675f\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\u3002 wget 127 .0.0.1 \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget 127 .0.0.1 \u4ee3\u7801\u5757 \u00b6 yaml \u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a yaml\u3002 Bad Good ```` apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` ````yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` \u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a bash\u3002 Bad Good ```` wget 127.0.0.1 ```` ````bash wget 127.0.0.1 ```` \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u5305\u542b\u8f93\u51fa\u5185\u5bb9\uff0c\u5219\u6240\u6267\u884c\u547d\u4ee4\u9700\u8981\u4ee5 # \u5f00\u59cb\uff0c\u4ee5\u533a\u5206\u8f93\u5165\u4e0e\u8f93\u51fa\u3002 Bad Good oilbeater@macdeMac-3 ~ ping 114 .114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms # ping 114.114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u53ea\u5305\u542b\u6267\u884c\u547d\u4ee4\uff0c\u6ca1\u6709\u8f93\u51fa\u7ed3\u679c\uff0c\u5219\u591a\u6761\u547d\u4ee4\u65e0\u9700 # \u5f00\u59cb\u3002 Bad Good # mv /etc/origin/ovn/ovnnb_db.db /tmp # mv /etc/origin/ovn/ovnsb_db.db /tmp mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp \u94fe\u63a5 \u00b6 \u7ad9\u5185\u94fe\u63a5\u4f7f\u7528\u5bf9\u5e94 md \u6587\u4ef6\u8def\u5f84\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](http://kubeovn.github.io/prepare)\u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](./prepare.md)\u3002 Bad Good \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io )\u3002 \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io ){: target=\"_blank\" }\u3002 \u7a7a\u884c \u00b6 \u4e0d\u540c\u903b\u8f91\u5757\uff0c\u4f8b\u5982\u6807\u9898\u548c\u6587\u672c\uff0c\u6587\u672c\u548c\u4ee3\u7801\uff0c\u6587\u672c\u548c\u7f16\u53f7\u4e4b\u95f4\u9700\u8981\u7528\u7a7a\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0d\u540c\u903b\u8f91\u5757\u4e4b\u95f4\u53ea\u4f7f\u7528 \u4e00\u4e2a \u7a7a\u884c\u8fdb\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Document Convention"},{"location":"en/reference/document-convention/#_1","text":"\u4e3a\u4e86\u4fdd\u8bc1\u6587\u6863\u98ce\u683c\u4e00\u81f4\uff0c\u8bf7\u5728\u63d0\u4ea4\u6587\u6863\u65f6\u9075\u5faa\u4e0b\u5217\u7684\u98ce\u683c\u89c4\u8303\u3002","title":"\u6587\u6863\u89c4\u8303"},{"location":"en/reference/document-convention/#_2","text":"\u4e2d\u6587\u6587\u6863\u4e2d\u6587\u672c\u5185\u5bb9\u6240\u6709\u6807\u70b9\u5e94\u4f7f\u7528\u4e2d\u6587\u683c\u5f0f\u6807\u70b9\uff0c\u82f1\u6587\u6587\u6863\u4e2d\u6240\u6709\u6587\u672c\u5185\u5bb9\u4e2d\u5e94\u4f7f\u7528\u82f1\u6587\u6807\u70b9\u3002 Bad Good \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c,\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528,\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc. \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002 \u82f1\u6587\u6570\u5b57\u548c\u4e2d\u6587\u5e94\u8be5\u7528\u7a7a\u683c\u8fdb\u884c\u5206\u9694\u3002 Bad Good Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 \u793a\u4f8b\u5185\u5bb9\u5e94\u8be5\u4ee5 \uff1a \u5f00\u542f\uff0c\u5176\u4ed6\u53e5\u5c3e\u9700\u8981\u7528 \u3002 \u7ed3\u675f\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\u3002 wget 127 .0.0.1 \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget 127 .0.0.1","title":"\u6807\u70b9"},{"location":"en/reference/document-convention/#_3","text":"yaml \u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a yaml\u3002 Bad Good ```` apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` ````yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` \u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a bash\u3002 Bad Good ```` wget 127.0.0.1 ```` ````bash wget 127.0.0.1 ```` \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u5305\u542b\u8f93\u51fa\u5185\u5bb9\uff0c\u5219\u6240\u6267\u884c\u547d\u4ee4\u9700\u8981\u4ee5 # \u5f00\u59cb\uff0c\u4ee5\u533a\u5206\u8f93\u5165\u4e0e\u8f93\u51fa\u3002 Bad Good oilbeater@macdeMac-3 ~ ping 114 .114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms # ping 114.114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u53ea\u5305\u542b\u6267\u884c\u547d\u4ee4\uff0c\u6ca1\u6709\u8f93\u51fa\u7ed3\u679c\uff0c\u5219\u591a\u6761\u547d\u4ee4\u65e0\u9700 # \u5f00\u59cb\u3002 Bad Good # mv /etc/origin/ovn/ovnnb_db.db /tmp # mv /etc/origin/ovn/ovnsb_db.db /tmp mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp","title":"\u4ee3\u7801\u5757"},{"location":"en/reference/document-convention/#_4","text":"\u7ad9\u5185\u94fe\u63a5\u4f7f\u7528\u5bf9\u5e94 md \u6587\u4ef6\u8def\u5f84\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](http://kubeovn.github.io/prepare)\u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](./prepare.md)\u3002 Bad Good \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io )\u3002 \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io ){: target=\"_blank\" }\u3002","title":"\u94fe\u63a5"},{"location":"en/reference/document-convention/#_5","text":"\u4e0d\u540c\u903b\u8f91\u5757\uff0c\u4f8b\u5982\u6807\u9898\u548c\u6587\u672c\uff0c\u6587\u672c\u548c\u4ee3\u7801\uff0c\u6587\u672c\u548c\u7f16\u53f7\u4e4b\u95f4\u9700\u8981\u7528\u7a7a\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0d\u540c\u903b\u8f91\u5757\u4e4b\u95f4\u53ea\u4f7f\u7528 \u4e00\u4e2a \u7a7a\u884c\u8fdb\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7a7a\u884c"},{"location":"en/reference/feature-stage/","text":"Feature Stage \u00b6 In Kube-OVN, feature stage is classified into Alpha , Beta and GA , based on the degree of feature usage, documentation and test coverage. Definition of Stage \u00b6 For Alpha stage functions: The feature is not fully documented and well tested. This feature may change or even be removed in the future. This feature API is not guaranteed to be stable and may be removed. Community provides low priority support for this feature and long-term support cannot be guaranteed. Since feature stability and long-term support cannot be guaranteed, it can be tested and verified, but is not recommended for production use. For Beta stage functions: This feature is partially documented and tested, but complete coverage is not guaranteed. This feature may change in the future and the upgrade may affect the network, but it will not be removed as a whole. This feature API may change in the future and the fields may be adjusted, but not removed as a whole. This feature will be supported by the community in the long term. It can be used on non-critical services as the functionality will be supported for a long time, but it is not recommended for critical production service as there is a possibility of changes in functionality and APIs that may break the network. For GA stage functions: The feature has full documentation and test coverage. The feature will remain stable and upgrades will be guaranteed to be smooth. This feature API is not subject to disruptive changes. This feature will be supported with high priority by the community and long-term support will be guaranteed. Feature Stage List \u00b6 This list records the feature stages from the 1.8 release. Feature Default Stage Since Until Namespaced Subnet true GA 1.8 Distributed Gateway true GA 1.8 Active-backup Centralized Gateway true GA 1.8 ECMP Centralized Gateway false Beta 1.8 Subnet ACL true Alpha 1.9 Subnet Isolation (Will be replaced by ACL later) true Beta 1.8 Underlay Subnet true GA 1.8 Multiple Pod Interface true Beta 1.8 Subnet DHCP false Alpha 1.10 Subnet with External Gateway false Alpha 1.8 Cluster Inter-Connection with OVN-IC false Beta 1.8 Cluster Inter-Connection with Submariner false Alpha 1.9 VIP Reservation true Alpha 1.10 Create Custom VPC true Beta 1.8 Custom VPC Floating IP/SNAT/DNAT true Alpha 1.10 Custom VPC Static Route true Alpha 1.10 Custom VPC Policy Route true Alpha 1.10 Custom VPC Security Group true Alpha 1.10 Container Bandwidth QoS true GA 1.8 linux-netem QoS true Alpha 1.9 Prometheus Integration false GA 1.8 Grafana Integration false GA 1.8 IPv4/v6 DualStack false GA 1.8 Default VPC EIP/SNAT false Beta 1.8 Traffic Mirroring false GA 1.8 NetworkPolicy true Beta 1.8 Webhook false Alpha 1.10 Performance Tunning false Beta 1.8 Interconnection with Routes in Overlay Mode false Alpha 1.8 BGP Support false Alpha 1.9 Cilium Integration false Alpha 1.10 Custom VPC Peering false Alpha 1.10 Mellanox Offload false Alpha 1.8 Corigine Offload false Alpha 1.10 Windows Support false Alpha 1.10 DPDK Support false Alpha 1.10 OpenStack Integration false Alpha 1.9 Single Pod Fixed IP/Mac true GA 1.8 Workload with Fixed IP true GA 1.8 StatefulSet with Fixed IP true GA 1.8 VM with Fixed IP false Beta 1.9 Load Balancer Type Service in Default VPC false Alpha 1.11 Load Balance in Custom VPC false Alpha 1.11 DNS in Custom VPC false Alpha 1.11 Underlay and Overlay Interconnection false Alpha 1.11 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Feature Stage"},{"location":"en/reference/feature-stage/#feature-stage","text":"In Kube-OVN, feature stage is classified into Alpha , Beta and GA , based on the degree of feature usage, documentation and test coverage.","title":"Feature Stage"},{"location":"en/reference/feature-stage/#definition-of-stage","text":"For Alpha stage functions: The feature is not fully documented and well tested. This feature may change or even be removed in the future. This feature API is not guaranteed to be stable and may be removed. Community provides low priority support for this feature and long-term support cannot be guaranteed. Since feature stability and long-term support cannot be guaranteed, it can be tested and verified, but is not recommended for production use. For Beta stage functions: This feature is partially documented and tested, but complete coverage is not guaranteed. This feature may change in the future and the upgrade may affect the network, but it will not be removed as a whole. This feature API may change in the future and the fields may be adjusted, but not removed as a whole. This feature will be supported by the community in the long term. It can be used on non-critical services as the functionality will be supported for a long time, but it is not recommended for critical production service as there is a possibility of changes in functionality and APIs that may break the network. For GA stage functions: The feature has full documentation and test coverage. The feature will remain stable and upgrades will be guaranteed to be smooth. This feature API is not subject to disruptive changes. This feature will be supported with high priority by the community and long-term support will be guaranteed.","title":"Definition of Stage"},{"location":"en/reference/feature-stage/#feature-stage-list","text":"This list records the feature stages from the 1.8 release. Feature Default Stage Since Until Namespaced Subnet true GA 1.8 Distributed Gateway true GA 1.8 Active-backup Centralized Gateway true GA 1.8 ECMP Centralized Gateway false Beta 1.8 Subnet ACL true Alpha 1.9 Subnet Isolation (Will be replaced by ACL later) true Beta 1.8 Underlay Subnet true GA 1.8 Multiple Pod Interface true Beta 1.8 Subnet DHCP false Alpha 1.10 Subnet with External Gateway false Alpha 1.8 Cluster Inter-Connection with OVN-IC false Beta 1.8 Cluster Inter-Connection with Submariner false Alpha 1.9 VIP Reservation true Alpha 1.10 Create Custom VPC true Beta 1.8 Custom VPC Floating IP/SNAT/DNAT true Alpha 1.10 Custom VPC Static Route true Alpha 1.10 Custom VPC Policy Route true Alpha 1.10 Custom VPC Security Group true Alpha 1.10 Container Bandwidth QoS true GA 1.8 linux-netem QoS true Alpha 1.9 Prometheus Integration false GA 1.8 Grafana Integration false GA 1.8 IPv4/v6 DualStack false GA 1.8 Default VPC EIP/SNAT false Beta 1.8 Traffic Mirroring false GA 1.8 NetworkPolicy true Beta 1.8 Webhook false Alpha 1.10 Performance Tunning false Beta 1.8 Interconnection with Routes in Overlay Mode false Alpha 1.8 BGP Support false Alpha 1.9 Cilium Integration false Alpha 1.10 Custom VPC Peering false Alpha 1.10 Mellanox Offload false Alpha 1.8 Corigine Offload false Alpha 1.10 Windows Support false Alpha 1.10 DPDK Support false Alpha 1.10 OpenStack Integration false Alpha 1.9 Single Pod Fixed IP/Mac true GA 1.8 Workload with Fixed IP true GA 1.8 StatefulSet with Fixed IP true GA 1.8 VM with Fixed IP false Beta 1.9 Load Balancer Type Service in Default VPC false Alpha 1.11 Load Balance in Custom VPC false Alpha 1.11 DNS in Custom VPC false Alpha 1.11 Underlay and Overlay Interconnection false Alpha 1.11 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Feature Stage List"},{"location":"en/reference/iptables-rules/","text":"Iptables Rules \u00b6 Kube-OVN uses ipset and iptables to implement gateway NAT functionality in the default VPC overlay Subnets. The ipset used is shown in the following table: Name\uff08IPv4/IPv6\uff09 Type Usage ovn40services/ovn60services hash:net Service CIDR ovn40subnets/ovn60subnets hash:net Overlay Subnet CIDR and NodeLocal DNS IP address ovn40subnets-nat/ovn60subnets-nat hash:net Overlay Subnet CIDRs that enable NatOutgoing ovn40subnets-distributed-gw/ovn60subnets-distributed-gw hash:net Overlay Subnet CIDRs that use distributed gateway ovn40other-node/ovn60other-node hash:net Internal IP addresses for other Nodes ovn40local-pod-ip-nat/ovn60local-pod-ip-nat hash:ip Deprecated ovn40subnets-nat-policy hash:net All subnet cidrs configured with natOutgoingPolicyRules ovn40natpr-418e79269dc5-dst hash:net The dstIPs corresponding to the rule in natOutgoingPolicyRules ovn40natpr-418e79269dc5-src hash:net The srcIPs corresponding to the rule in natOutgoingPolicyRules The iptables rules (IPv4) used are shown in the following table: Table Chain Rule Usage Note filter INPUT -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -s 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" Used to count packets from the subnet to the external network \"10.16.0.0/16\" is the cidr of the subnet, the \"ovn-subnet-gateway\" before the \",\" in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the \"ovn-default\" after the \",\" is the name of the subnet filter FORWARD -d 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" Used to count packets from the external network accessing the subnet \"10.16.0.0/16\" is the cidr of the subnet, the \"ovn-subnet-gateway\" before the \",\" in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the \"ovn-default\" after the \",\" is the name of the subnet filter OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0 Clear traffic tag to prevent SNAT UDP: bad checksum on VXLAN interface nat PREROUTING -m comment --comment \"kube-ovn prerouting rules\" -j OVN-PREROUTING Enter OVN-PREROUTING chain processing -- nat POSTROUTING -m comment --comment \"kube-ovn postrouting rules\" -j OVN-POSTROUTING Enter OVN-POSTROUTING chain processing -- nat OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set ovn40services dst -j MARK --set-xmark 0x4000/0x4000 Adding masquerade tags to Pod access service traffic Used when the built-in LB is turned off nat OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (TCP) Only used when kube-proxy is using ipvs mode nat OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (UDP) Only used when kube-proxy is using ipvs mode nat OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source Use node IP as the source address for access from node to overlay Pods via service IP\u3002 Works only when kube-proxy is using ipvs mode nat OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE Perform SNAT for specific tagged traffic -- nat OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j MASQUERADE Perform SNAT for Service traffic between Pods passing through the node -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a distributed gateway, SNAT is not required. -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j MASQUERADE For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a centralized gateway, SNAT is required. -- nat OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN No SNAT is performed when the Pod IP is exposed to the outside world -- nat OVN-POSTROUTING -s 10.16.0.0/16 -m set ! --match-set ovn40subnets dst -j SNAT --to-source 192.168.0.101 When the Pod accesses the network outside the cluster, if the subnet is NatOutgoing and a centralized gateway with the specified IP is used, perform SNAT 10.16.0.0/16 is the Subnet CIDR\uff0c192.168.0.101 is the specified IP of gateway node nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j MASQUERADE When the Pod accesses the network outside the cluster, if NatOutgoing is enabled on the subnet, perform SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT ovn40subnets-nat-policy is all subnet segments configured with natOutgoingPolicyRules nat OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j MASQUERADE --random-fully When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90001/0x90001, it will do SNAT nat OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90002/0x90002, it will not do SNAT nat OVN-NAT-POLICY -s 10.0.11.0/24 -m comment --comment natPolicySubnet-net1 -j OVN-NAT-PSUBNET-aa98851157c5 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 10.0.11.0/24 represents the CIDR of the subnet net1, and the rules under the OVN-NAT-PSUBNET-aa98851157c5 chain correspond to the natOutgoingPolicyRules configuration of this subnet nat OVN-NAT-PSUBNET-xxxxxxxxxxxx -m set --match-set ovn40natpr-418e79269dc5-src src -m set --match-set ovn40natpr-418e79269dc5-dst dst -j MARK --set-xmark 0x90002/0x90002 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 418e79269dc5 indicates the ID of a rule in natOutgoingPolicyRules, which can be viewed through status.natOutgoingPolicyRules[index].RuleID, indicating that srcIPs meets ovn40natpr-418e79269dc5-src, and dstIPS meets ovn40natpr-418e79269dc5- dst will be marked with tag 0x90002 mangle OVN-OUTPUT -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003 Introduce kubelet's detection traffic to tproxy with a specific mark mangle OVN-PREROUTING -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.3 --tproxy-mark 0x90004/0x90004 Introduce kubelet's detection traffic to tproxy with a specific mark \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Iptables Rules"},{"location":"en/reference/iptables-rules/#iptables-rules","text":"Kube-OVN uses ipset and iptables to implement gateway NAT functionality in the default VPC overlay Subnets. The ipset used is shown in the following table: Name\uff08IPv4/IPv6\uff09 Type Usage ovn40services/ovn60services hash:net Service CIDR ovn40subnets/ovn60subnets hash:net Overlay Subnet CIDR and NodeLocal DNS IP address ovn40subnets-nat/ovn60subnets-nat hash:net Overlay Subnet CIDRs that enable NatOutgoing ovn40subnets-distributed-gw/ovn60subnets-distributed-gw hash:net Overlay Subnet CIDRs that use distributed gateway ovn40other-node/ovn60other-node hash:net Internal IP addresses for other Nodes ovn40local-pod-ip-nat/ovn60local-pod-ip-nat hash:ip Deprecated ovn40subnets-nat-policy hash:net All subnet cidrs configured with natOutgoingPolicyRules ovn40natpr-418e79269dc5-dst hash:net The dstIPs corresponding to the rule in natOutgoingPolicyRules ovn40natpr-418e79269dc5-src hash:net The srcIPs corresponding to the rule in natOutgoingPolicyRules The iptables rules (IPv4) used are shown in the following table: Table Chain Rule Usage Note filter INPUT -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -s 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" Used to count packets from the subnet to the external network \"10.16.0.0/16\" is the cidr of the subnet, the \"ovn-subnet-gateway\" before the \",\" in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the \"ovn-default\" after the \",\" is the name of the subnet filter FORWARD -d 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" Used to count packets from the external network accessing the subnet \"10.16.0.0/16\" is the cidr of the subnet, the \"ovn-subnet-gateway\" before the \",\" in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the \"ovn-default\" after the \",\" is the name of the subnet filter OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0 Clear traffic tag to prevent SNAT UDP: bad checksum on VXLAN interface nat PREROUTING -m comment --comment \"kube-ovn prerouting rules\" -j OVN-PREROUTING Enter OVN-PREROUTING chain processing -- nat POSTROUTING -m comment --comment \"kube-ovn postrouting rules\" -j OVN-POSTROUTING Enter OVN-POSTROUTING chain processing -- nat OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set ovn40services dst -j MARK --set-xmark 0x4000/0x4000 Adding masquerade tags to Pod access service traffic Used when the built-in LB is turned off nat OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (TCP) Only used when kube-proxy is using ipvs mode nat OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (UDP) Only used when kube-proxy is using ipvs mode nat OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source Use node IP as the source address for access from node to overlay Pods via service IP\u3002 Works only when kube-proxy is using ipvs mode nat OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE Perform SNAT for specific tagged traffic -- nat OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j MASQUERADE Perform SNAT for Service traffic between Pods passing through the node -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a distributed gateway, SNAT is not required. -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j MASQUERADE For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a centralized gateway, SNAT is required. -- nat OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN No SNAT is performed when the Pod IP is exposed to the outside world -- nat OVN-POSTROUTING -s 10.16.0.0/16 -m set ! --match-set ovn40subnets dst -j SNAT --to-source 192.168.0.101 When the Pod accesses the network outside the cluster, if the subnet is NatOutgoing and a centralized gateway with the specified IP is used, perform SNAT 10.16.0.0/16 is the Subnet CIDR\uff0c192.168.0.101 is the specified IP of gateway node nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j MASQUERADE When the Pod accesses the network outside the cluster, if NatOutgoing is enabled on the subnet, perform SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT ovn40subnets-nat-policy is all subnet segments configured with natOutgoingPolicyRules nat OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j MASQUERADE --random-fully When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90001/0x90001, it will do SNAT nat OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90002/0x90002, it will not do SNAT nat OVN-NAT-POLICY -s 10.0.11.0/24 -m comment --comment natPolicySubnet-net1 -j OVN-NAT-PSUBNET-aa98851157c5 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 10.0.11.0/24 represents the CIDR of the subnet net1, and the rules under the OVN-NAT-PSUBNET-aa98851157c5 chain correspond to the natOutgoingPolicyRules configuration of this subnet nat OVN-NAT-PSUBNET-xxxxxxxxxxxx -m set --match-set ovn40natpr-418e79269dc5-src src -m set --match-set ovn40natpr-418e79269dc5-dst dst -j MARK --set-xmark 0x90002/0x90002 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 418e79269dc5 indicates the ID of a rule in natOutgoingPolicyRules, which can be viewed through status.natOutgoingPolicyRules[index].RuleID, indicating that srcIPs meets ovn40natpr-418e79269dc5-src, and dstIPS meets ovn40natpr-418e79269dc5- dst will be marked with tag 0x90002 mangle OVN-OUTPUT -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003 Introduce kubelet's detection traffic to tproxy with a specific mark mangle OVN-PREROUTING -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.3 --tproxy-mark 0x90004/0x90004 Introduce kubelet's detection traffic to tproxy with a specific mark \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Iptables Rules"},{"location":"en/reference/kube-ovn-api/","text":"Kube-OVN API Reference \u00b6 Based on Kube-OVN v1.12.0, we have compiled a list of CRD resources supported by Kube-OVN, listing the types and meanings of each field of CRD definition for reference. Generic Condition Definition \u00b6 Property Name Type Description type String Type of status status String The value of status, in the range of True , False or Unknown reason String The reason for the status change message String The specific message of the status change lastUpdateTime Time The last time the status was updated lastTransitionTime Time Time of last status type change In each CRD definition, the Condition field in Status follows the above format, so we explain it in advance. Subnet Definition \u00b6 Subnet \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Subnet metadata ObjectMeta Standard Kubernetes resource metadata information spec SubnetSpec Subnet specific configuration information status SubnetStatus Subnet status information SubnetSpec \u00b6 Property Name Type Description default Bool Whether this subnet is the default subnet vpc String The vpc which the subnet belongs to, default is ovn-cluster protocol String IP protocol, the value is in the range of IPv4 , IPv6 or Dual namespaces []String The list of namespaces bound to this subnet cidrBlock String The range of the subnet, e.g. 10.16.0.0/16 gateway String The gateway address of the subnet, the default value is the first available address under the CIDRBlock of the subnet excludeIps []String The range of addresses under this subnet that will not be automatically assigned provider String Default value is ovn . In the case of multiple NICs, the value is <name>.<namespace> of the NetworkAttachmentDefinition, Kube-OVN will use this information to find the corresponding subnet resource gatewayType String The gateway type in overlay mode, either distributed or centralized gatewayNode String The gateway node when the gateway mode is centralized, node names can be comma-separated natOutgoing Bool Whether the outgoing traffic is NAT externalEgressGateway String The address of the external gateway. This parameter and the natOutgoing parameter cannot be set at the same time policyRoutingPriority Uint32 Policy route priority. Used to control the forwarding of traffic to the external gateway address after the subnet gateway policyRoutingTableID Uint32 The TableID of the local policy routing table, should be different for each subnet to avoid conflicts private Bool Whether the subnet is a private subnet, which denies access to addresses inside the subnet if the subnet is private allowSubnets []String If the subnet is a private subnet, the set of addresses that are allowed to access the subnet vlan String The name of vlan to which the subnet is bound vips []String The virtual-ip parameter information for virtual type lsp on the subnet logicalGateway Bool Whether to enable logical gateway disableGatewayCheck Bool Whether to skip the gateway connectivity check when creating a pod disableInterConnection Bool Whether to enable subnet interconnection across clusters enableDHCP Bool Whether to configure dhcp configuration options for lsps belong this subnet dhcpV4Options String The DHCP_Options record associated with lsp dhcpv4_options on the subnet dhcpV6Options String The DHCP_Options record associated with lsp dhcpv6_options on the subnet enableIPv6RA Bool Whether to configure the ipv6_ra_configs parameter for the lrp port of the router connected to the subnet ipv6RAConfigs String The ipv6_ra_configs parameter configuration for the lrp port of the router connected to the subnet acls []Acl The acls record associated with the logical-switch of the subnet u2oInterconnection Bool Whether to enable interconnection mode for Overlay/Underlay enableLb *Bool Whether the logical-switch of the subnet is associated with load-balancer records enableEcmp Bool Centralized subnet, whether to enable ECMP routing Acl \u00b6 Property Name Type Description direction String Restrict the direction of acl, which value is from-lport or to-lport priority Int Acl priority, in the range 0 to 32767 match String Acl rule match expression action String The action of the rule, which value is in the range of allow-related , allow-stateless , allow , drop , reject SubnetStatus \u00b6 Property Name Type Description conditions []SubnetCondition Subnet status change information, refer to the beginning of the document for the definition of Condition v4AvailableIPs Float64 Number of available IPv4 IPs v4availableIPrange String The available range of IPv4 addresses on the subnet v4UsingIPs Float64 Number of used IPv4 IPs v4usingIPrange String Used IPv4 address ranges on the subnet v6AvailableIPs Float64 Number of available IPv6 IPs v6availableIPrange String The available range of IPv6 addresses on the subnet v6UsingIPs Float64 Number of used IPv6 IPs v6usingIPrange String Used IPv6 address ranges on the subnet sctivateGateway String The currently working gateway node in centralized subnet of master-backup mode dhcpV4OptionsUUID String The DHCP_Options record identifier associated with the lsp dhcpv4_options on the subnet dhcpV6OptionsUUID String The DHCP_Options record identifier associated with the lsp dhcpv6_options on the subnet u2oInterconnectionIP String The IP address used for interconnection when Overlay/Underlay interconnection mode is enabled IP Definition \u00b6 IP \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IP metadata ObjectMeta Standard Kubernetes resource metadata information spec IPSpec IP specific configuration information IPSepc \u00b6 Property Name Type Description podName String Pod name which assigned with this IP namespace String The name of the namespace where the pod is bound subnet String The subnet which the ip belongs to attachSubnets []String The name of the other subnets attached to this primary IP (field deprecated) nodeName String The name of the node where the pod is bound ipAddress String IP address, in v4IP,v6IP format for dual-stack cases v4IPAddress String IPv4 IP address v6IPAddress String IPv6 IP address attachIPs []String Other IP addresses attached to this primary IP (field is deprecated) macAddress String The Mac address of the bound pod attachMacs []String Other Mac addresses attached to this primary IP (field deprecated) containerID String The Container ID corresponding to the bound pod podType String Special workload pod, can be StatefulSet , VirtualMachine or empty Underlay configuration \u00b6 Vlan \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all instances of this resource will be kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vlan metadata ObjectMeta Standard Kubernetes resource metadata information spec VlanSpec Vlan specific configuration information status VlanStatus Vlan status information VlanSpec \u00b6 Property Name Type Description id Int Vlan tag number, in the range of 0~4096 provider String The name of the ProviderNetwork to which the vlan is bound VlanStatus \u00b6 Property Name Type Description subnets []String The list of subnets to which the vlan is bound conditions []VlanCondition Vlan status change information, refer to the beginning of the document for the definition of Condition ProviderNetwork \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value ProviderNetwork metadata ObjectMeta Standard Kubernetes resource metadata information spec ProviderNetworkSpec ProviderNetwork specific configuration information status ProviderNetworkStatus ProviderNetwork status information ProviderNetworkSpec \u00b6 Property Name Type Description defaultInterface String The name of the NIC interface used by default for this bridge network customInterfaces []CustomInterface The special NIC configuration used by this bridge network excludeNodes []String The names of the nodes that will not be bound to this bridge network exchangeLinkName Bool Whether to exchange the bridge NIC and the corresponding OVS bridge name CustomInterface \u00b6 Property Name Type Description interface String NIC interface name used for underlay nodes []String List of nodes using the custom NIC interface ProviderNetworkStatus \u00b6 Property Name Type Description ready Bool Whether the current bridge network is in the ready state readyNodes []String The name of the node whose bridge network is ready notReadyNodes []String The name of the node whose bridge network is not ready vlans []String The name of the vlan to which the bridge network is bound conditions []ProviderNetworkCondition ProviderNetwork status change information, refer to the beginning of the document for the definition of Condition Vpc Definition \u00b6 Vpc \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vpc metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcSpec Vpc specific configuration information status VpcStatus Vpc status information VpcSpec \u00b6 Property Name Type Description namespaces []String List of namespaces bound by Vpc staticRoutes []*StaticRoute The static route information configured under Vpc policyRoutes []*PolicyRoute The policy route information configured under Vpc vpcPeerings []*VpcPeering Vpc interconnection information enableExternal Bool Whether vpc is connected to an external switch StaticRoute \u00b6 Property Name Type Description policy String Routing policy, takes the value of policySrc or policyDst cidr String Routing cidr value nextHopIP String The next hop information of the route PolicyRoute \u00b6 Property Name Type Description priority Int32 Priority for policy route match String Match expression for policy route action String Action for policy route, the value is in the range of allow , drop , reroute nextHopIP String The next hop of the policy route, separated by commas in the case of ECMP routing VpcPeering \u00b6 Property Name Type Description remoteVpc String Name of the interconnected peering vpc localConnectIP String The local ip for vpc used to connect to peer vpc VpcStatus \u00b6 Property Name Type Description conditions []VpcCondition Vpc status change information, refer to the beginning of the documentation for the definition of Condition standby Bool Whether the vpc creation is complete, the subnet under the vpc needs to wait for the vpc creation to complete other proceeding default Bool Whether it is the default vpc defaultLogicalSwitch String The default subnet under vpc router String The logical-router name for the vpc tcpLoadBalancer String TCP LB information for vpc udpLoadBalancer String UDP LB information for vpc tcpSessionLoadBalancer String TCP Session Hold LB Information for Vpc udpSessionLoadBalancer String UDP session hold LB information for Vpc subnets []String List of subnets for vpc vpcPeerings []String List of peer vpcs for vpc interconnection enableExternal Bool Whether the vpc is connected to an external switch VpcNatGateway \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value VpcNatGateway metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcNatSpec Vpc gateway specific configuration information VpcNatSpec \u00b6 Property Name Type Description vpc String Vpc name which the vpc gateway belongs to subnet String The name of the subnet to which the gateway pod belongs lanIp String The IP address assigned to the gateway pod selector []String Standard Kubernetes selector match information tolerations []VpcNatToleration Standard Kubernetes tolerance information VpcNatToleration \u00b6 Property Name Type Description key String The key information of the taint tolerance operator String Takes the value of Exists or Equal value String The value information of the taint tolerance effect String The effect of the taint tolerance, takes the value of NoExecute , NoSchedule , or PreferNoSchedule tolerationSeconds Int64 The amount of time the pod can continue to run on the node after the taint is added The meaning of the above tolerance fields can be found in the official Kubernetes documentation Taint and Tolerance . IptablesEIP \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesEIP metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesEipSpec IptablesEIP specific configuration information used by vpc gateway status IptablesEipStatus IptablesEIP status information used by vpc gateway IptablesEipSpec \u00b6 Property Name Type Description v4ip String IptablesEIP v4 address v6ip String IptablesEIP v6 address macAddress String The assigned mac address, not actually used natGwDp String Vpc gateway name IptablesEipStatus \u00b6 Property Name Type Description ready Bool Whether IptablesEIP is configured complete ip String The IP address used by IptablesEIP, currently only IPv4 addresses are supported redo String IptablesEIP crd creation or update time nat String The type of IptablesEIP, either fip , snat , or dnat conditions []IptablesEIPCondition IptablesEIP status change information, refer to the beginning of the documentation for the definition of Condition IptablesFIPRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesFIPRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesFIPRuleSpec The IptablesFIPRule specific configuration information used by vpc gateway status IptablesFIPRuleStatus IptablesFIPRule status information used by vpc gateway IptablesFIPRuleSpec \u00b6 Property Name Type Description eip String Name of the IptablesEIP used for IptablesFIPRule internalIp String The corresponding internal IP address IptablesFIPRuleStatus \u00b6 Property Name Type Description ready Bool Whether IptablesFIPRule is configured or not v4ip String The v4 IP address used by IptablesEIP v6ip String The v6 IP address used by IptablesEIP natGwDp String Vpc gateway name redo String IptablesFIPRule crd creation or update time conditions []IptablesFIPRuleCondition IptablesFIPRule status change information, refer to the beginning of the documentation for the definition of Condition IptablesSnatRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesSnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesSnatRuleSpec The IptablesSnatRule specific configuration information used by the vpc gateway status IptablesSnatRuleStatus IptablesSnatRule status information used by vpc gateway IptablesSnatRuleSpec \u00b6 Property Name Type Description eip String Name of the IptablesEIP used by IptablesSnatRule internalIp String IptablesSnatRule's corresponding internal IP address IptablesSnatRuleStatus \u00b6 Property Name Type Description ready Bool Whether the configuration is complete v4ip String The v4 IP address used by IptablesSnatRule v6ip String The v6 IP address used by IptablesSnatRule natGwDp String Vpc gateway name redo String IptablesSnatRule crd creation or update time conditions []IptablesSnatRuleCondition IptablesSnatRule status change information, refer to the beginning of the documentation for the definition of Condition IptablesDnatRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesDnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesDnatRuleSpec The IptablesDnatRule specific configuration information used by vpc gateway status IptablesDnatRuleStatus IptablesDnatRule status information used by vpc gateway IptablesDnatRuleSpec \u00b6 Property Name Type Description eip Sting Name of IptablesEIP used by IptablesDnatRule externalPort Sting External port used by IptablesDnatRule protocol Sting Vpc gateway dnat protocol type internalIp Sting Internal IP address used by IptablesDnatRule internalPort Sting Internal port used by IptablesDnatRule IptablesDnatRuleStatus \u00b6 Property Name Type Description ready Bool Whether the configuration is complete v4ip String The v4 IP address used by IptablesDnatRule v6ip String The v6 IP address used by IptablesDnatRule natGwDp String Vpc gateway name redo String IptablesDnatRule crd creation or update time conditions []IptablesDnatRuleCondition IptablesDnatRule Status change information, refer to the beginning of the documentation for the definition of Condition VpcDns \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value VpcDns metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcDnsSpec VpcDns specific configuration information status VpcDnsStatus VpcDns status information VpcDnsSpec \u00b6 Property Name Type Description vpc String Name of the vpc where VpcDns is located subnet String The subnet name of the address assigned to the VpcDns pod VpcDnsStatus \u00b6 Property Name Type Description conditions []VpcDnsCondition VpcDns status change information, refer to the beginning of the document for the definition of Condition active Bool Whether VpcDns is in use For detailed documentation on the use of VpcDns, see Customizing VPC DNS . SwitchLBRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value SwitchLBRule metadata ObjectMeta Standard Kubernetes resource metadata information spec SwitchLBRuleSpec SwitchLBRule specific configuration information status SwitchLBRuleStatus SwitchLBRule status information SwitchLBRuleSpec \u00b6 Property Name Type Description vip String Vip address of SwitchLBRule namespace String SwitchLBRule's namespace selector []String Standard Kubernetes selector match information sessionAffinity String Standard Kubernetes service sessionAffinity value ports []SlrPort List of SwitchLBRule ports For detailed configuration information of SwitchLBRule, you can refer to Customizing VPC Internal Load Balancing . SlrPort \u00b6 Property Name Type Description name String Port name port Int32 Port number targetPort Int32 Target port of SwitchLBRule protocol String Protocol type SwitchLBRuleStatus \u00b6 Property Name Type Description conditions []SwitchLBRuleCondition SwitchLBRule status change information, refer to the beginning of the document for the definition of Condition ports String Port information service String Name of the service Security Group and Vip \u00b6 SecurityGroup \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have a value of SecurityGroup metadata ObjectMeta Standard Kubernetes resource metadata information spec SecurityGroupSpec Security Group specific configuration information status SecurityGroupStatus Security group status information SecurityGroupSpec \u00b6 Property Name Type Description ingressRules []*SgRule Inbound security group rules egressRules []*SgRule Outbound security group rules allowSameGroupTraffic Bool Whether lsps in the same security group can interoperate and whether traffic rules need to be updated SgRule \u00b6 Property Name Type Description ipVersion String IP version number, ipv4 or ipv6 protocol String The value of icmp , tcp , or udp priority Int Acl priority. The value range is 1-200, the smaller the value, the higher the priority. remoteType String The value is either address or securityGroup remoteAddress String The address of the other side remoteSecurityGroup String The name of security group on the other side portRangeMin Int The starting value of the port range, the minimum value is 1. portRangeMax Int The ending value of the port range, the maximum value is 65535. policy String The value is allow or drop SecurityGroupStatus \u00b6 Property Name Type Description portGroup String The name of the port-group for the security group allowSameGroupTraffic Bool Whether lsps in the same security group can interoperate, and whether the security group traffic rules need to be updated ingressMd5 String The MD5 value of the inbound security group rule egressMd5 String The MD5 value of the outbound security group rule ingressLastSyncSuccess Bool Whether the last synchronization of the inbound rule was successful egressLastSyncSuccess Bool Whether the last synchronization of the outbound rule was successful Vip \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vip metadata ObjectMeta Standard Kubernetes resource metadata information spec VipSpec Vip specific configuration information status VipStatus Vip status information VipSpec \u00b6 Property Name Type Description namespace String Vip's namespace subnet String Vip's subnet v4ip String Vip IPv4 ip address v6ip String Vip IPv6 ip address macAddress String Vip mac address parentV4ip String Not currently in use parentV6ip String Not currently in use parentMac String Not currently in use attachSubnets []String This field is deprecated and no longer used VipStatus \u00b6 Property Name Type Description conditions []VipCondition Vip status change information, refer to the beginning of the documentation for the definition of Condition ready Bool Vip is ready or not v4ip String Vip IPv4 ip address, should be the same as the spec field v6ip String Vip IPv6 ip address, should be the same as the spec field mac String The vip mac address, which should be the same as the spec field pv4ip String Not currently used pv6ip String Not currently used pmac String Not currently used OvnEip \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnEip metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnEipSpec OvnEip specific configuration information for default vpc status OvnEipStatus OvnEip status information for default vpc OvnEipSpec \u00b6 Property Name Type Description externalSubnet String OvnEip's subnet name v4ip String OvnEip IP address macAddress String OvnEip Mac address type String OvnEip use type, the value can be fip , snat or lrp OvnEipStatus \u00b6 Property Name Type Description conditions []OvnEipCondition OvnEip status change information, refer to the beginning of the documentation for the definition of Condition v4ip String The IPv4 ip address used by ovnEip macAddress String Mac address used by ovnEip OvnFip \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnFip metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnFipSpec OvnFip specific configuration information in default vpc status OvnFipStatus OvnFip status information in default vpc OvnFipSpec \u00b6 Property Name Type Description ovnEip String Name of the bound ovnEip ipName String The IP crd name corresponding to the bound Pod OvnFipStatus \u00b6 Property Name Type Description ready Bool OvnFip is ready or not v4Eip String Name of the ovnEip to which ovnFip is bound v4Ip String The ovnEip address currently in use macAddress String OvnFip's configured mac address vpc String The name of the vpc where ovnFip is located conditions []OvnFipCondition OvnFip status change information, refer to the beginning of the document for the definition of Condition OvnSnatRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnSnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnSnatRuleSpec OvnSnatRule specific configuration information in default vpc status OvnSnatRuleStatus OvnSnatRule status information in default vpc OvnSnatRuleSpec \u00b6 Property Name Type Description ovnEip String Name of the ovnEip to which ovnSnatRule is bound vpcSubnet String The name of the subnet configured by ovnSnatRule ipName String The IP crd name corresponding to the ovnSnatRule bound Pod OvnSnatRuleStatus \u00b6 Property Name Type Description ready Bool OvnSnatRule is ready or not v4Eip String The ovnEip address to which ovnSnatRule is bound v4IpCidr String The cidr address used to configure snat in the logical-router vpc String The name of the vpc where ovnSnatRule is located conditions []OvnSnatRuleCondition OvnSnatRule status change information, refer to the beginning of the document for the definition of Condition \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN API Reference"},{"location":"en/reference/kube-ovn-api/#kube-ovn-api-reference","text":"Based on Kube-OVN v1.12.0, we have compiled a list of CRD resources supported by Kube-OVN, listing the types and meanings of each field of CRD definition for reference.","title":"Kube-OVN API Reference"},{"location":"en/reference/kube-ovn-api/#generic-condition-definition","text":"Property Name Type Description type String Type of status status String The value of status, in the range of True , False or Unknown reason String The reason for the status change message String The specific message of the status change lastUpdateTime Time The last time the status was updated lastTransitionTime Time Time of last status type change In each CRD definition, the Condition field in Status follows the above format, so we explain it in advance.","title":"Generic Condition Definition"},{"location":"en/reference/kube-ovn-api/#subnet-definition","text":"","title":"Subnet Definition"},{"location":"en/reference/kube-ovn-api/#subnet","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Subnet metadata ObjectMeta Standard Kubernetes resource metadata information spec SubnetSpec Subnet specific configuration information status SubnetStatus Subnet status information","title":"Subnet"},{"location":"en/reference/kube-ovn-api/#subnetspec","text":"Property Name Type Description default Bool Whether this subnet is the default subnet vpc String The vpc which the subnet belongs to, default is ovn-cluster protocol String IP protocol, the value is in the range of IPv4 , IPv6 or Dual namespaces []String The list of namespaces bound to this subnet cidrBlock String The range of the subnet, e.g. 10.16.0.0/16 gateway String The gateway address of the subnet, the default value is the first available address under the CIDRBlock of the subnet excludeIps []String The range of addresses under this subnet that will not be automatically assigned provider String Default value is ovn . In the case of multiple NICs, the value is <name>.<namespace> of the NetworkAttachmentDefinition, Kube-OVN will use this information to find the corresponding subnet resource gatewayType String The gateway type in overlay mode, either distributed or centralized gatewayNode String The gateway node when the gateway mode is centralized, node names can be comma-separated natOutgoing Bool Whether the outgoing traffic is NAT externalEgressGateway String The address of the external gateway. This parameter and the natOutgoing parameter cannot be set at the same time policyRoutingPriority Uint32 Policy route priority. Used to control the forwarding of traffic to the external gateway address after the subnet gateway policyRoutingTableID Uint32 The TableID of the local policy routing table, should be different for each subnet to avoid conflicts private Bool Whether the subnet is a private subnet, which denies access to addresses inside the subnet if the subnet is private allowSubnets []String If the subnet is a private subnet, the set of addresses that are allowed to access the subnet vlan String The name of vlan to which the subnet is bound vips []String The virtual-ip parameter information for virtual type lsp on the subnet logicalGateway Bool Whether to enable logical gateway disableGatewayCheck Bool Whether to skip the gateway connectivity check when creating a pod disableInterConnection Bool Whether to enable subnet interconnection across clusters enableDHCP Bool Whether to configure dhcp configuration options for lsps belong this subnet dhcpV4Options String The DHCP_Options record associated with lsp dhcpv4_options on the subnet dhcpV6Options String The DHCP_Options record associated with lsp dhcpv6_options on the subnet enableIPv6RA Bool Whether to configure the ipv6_ra_configs parameter for the lrp port of the router connected to the subnet ipv6RAConfigs String The ipv6_ra_configs parameter configuration for the lrp port of the router connected to the subnet acls []Acl The acls record associated with the logical-switch of the subnet u2oInterconnection Bool Whether to enable interconnection mode for Overlay/Underlay enableLb *Bool Whether the logical-switch of the subnet is associated with load-balancer records enableEcmp Bool Centralized subnet, whether to enable ECMP routing","title":"SubnetSpec"},{"location":"en/reference/kube-ovn-api/#acl","text":"Property Name Type Description direction String Restrict the direction of acl, which value is from-lport or to-lport priority Int Acl priority, in the range 0 to 32767 match String Acl rule match expression action String The action of the rule, which value is in the range of allow-related , allow-stateless , allow , drop , reject","title":"Acl"},{"location":"en/reference/kube-ovn-api/#subnetstatus","text":"Property Name Type Description conditions []SubnetCondition Subnet status change information, refer to the beginning of the document for the definition of Condition v4AvailableIPs Float64 Number of available IPv4 IPs v4availableIPrange String The available range of IPv4 addresses on the subnet v4UsingIPs Float64 Number of used IPv4 IPs v4usingIPrange String Used IPv4 address ranges on the subnet v6AvailableIPs Float64 Number of available IPv6 IPs v6availableIPrange String The available range of IPv6 addresses on the subnet v6UsingIPs Float64 Number of used IPv6 IPs v6usingIPrange String Used IPv6 address ranges on the subnet sctivateGateway String The currently working gateway node in centralized subnet of master-backup mode dhcpV4OptionsUUID String The DHCP_Options record identifier associated with the lsp dhcpv4_options on the subnet dhcpV6OptionsUUID String The DHCP_Options record identifier associated with the lsp dhcpv6_options on the subnet u2oInterconnectionIP String The IP address used for interconnection when Overlay/Underlay interconnection mode is enabled","title":"SubnetStatus"},{"location":"en/reference/kube-ovn-api/#ip-definition","text":"","title":"IP Definition"},{"location":"en/reference/kube-ovn-api/#ip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IP metadata ObjectMeta Standard Kubernetes resource metadata information spec IPSpec IP specific configuration information","title":"IP"},{"location":"en/reference/kube-ovn-api/#ipsepc","text":"Property Name Type Description podName String Pod name which assigned with this IP namespace String The name of the namespace where the pod is bound subnet String The subnet which the ip belongs to attachSubnets []String The name of the other subnets attached to this primary IP (field deprecated) nodeName String The name of the node where the pod is bound ipAddress String IP address, in v4IP,v6IP format for dual-stack cases v4IPAddress String IPv4 IP address v6IPAddress String IPv6 IP address attachIPs []String Other IP addresses attached to this primary IP (field is deprecated) macAddress String The Mac address of the bound pod attachMacs []String Other Mac addresses attached to this primary IP (field deprecated) containerID String The Container ID corresponding to the bound pod podType String Special workload pod, can be StatefulSet , VirtualMachine or empty","title":"IPSepc"},{"location":"en/reference/kube-ovn-api/#underlay-configuration","text":"","title":"Underlay configuration"},{"location":"en/reference/kube-ovn-api/#vlan","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all instances of this resource will be kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vlan metadata ObjectMeta Standard Kubernetes resource metadata information spec VlanSpec Vlan specific configuration information status VlanStatus Vlan status information","title":"Vlan"},{"location":"en/reference/kube-ovn-api/#vlanspec","text":"Property Name Type Description id Int Vlan tag number, in the range of 0~4096 provider String The name of the ProviderNetwork to which the vlan is bound","title":"VlanSpec"},{"location":"en/reference/kube-ovn-api/#vlanstatus","text":"Property Name Type Description subnets []String The list of subnets to which the vlan is bound conditions []VlanCondition Vlan status change information, refer to the beginning of the document for the definition of Condition","title":"VlanStatus"},{"location":"en/reference/kube-ovn-api/#providernetwork","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value ProviderNetwork metadata ObjectMeta Standard Kubernetes resource metadata information spec ProviderNetworkSpec ProviderNetwork specific configuration information status ProviderNetworkStatus ProviderNetwork status information","title":"ProviderNetwork"},{"location":"en/reference/kube-ovn-api/#providernetworkspec","text":"Property Name Type Description defaultInterface String The name of the NIC interface used by default for this bridge network customInterfaces []CustomInterface The special NIC configuration used by this bridge network excludeNodes []String The names of the nodes that will not be bound to this bridge network exchangeLinkName Bool Whether to exchange the bridge NIC and the corresponding OVS bridge name","title":"ProviderNetworkSpec"},{"location":"en/reference/kube-ovn-api/#custominterface","text":"Property Name Type Description interface String NIC interface name used for underlay nodes []String List of nodes using the custom NIC interface","title":"CustomInterface"},{"location":"en/reference/kube-ovn-api/#providernetworkstatus","text":"Property Name Type Description ready Bool Whether the current bridge network is in the ready state readyNodes []String The name of the node whose bridge network is ready notReadyNodes []String The name of the node whose bridge network is not ready vlans []String The name of the vlan to which the bridge network is bound conditions []ProviderNetworkCondition ProviderNetwork status change information, refer to the beginning of the document for the definition of Condition","title":"ProviderNetworkStatus"},{"location":"en/reference/kube-ovn-api/#vpc-definition","text":"","title":"Vpc Definition"},{"location":"en/reference/kube-ovn-api/#vpc","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vpc metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcSpec Vpc specific configuration information status VpcStatus Vpc status information","title":"Vpc"},{"location":"en/reference/kube-ovn-api/#vpcspec","text":"Property Name Type Description namespaces []String List of namespaces bound by Vpc staticRoutes []*StaticRoute The static route information configured under Vpc policyRoutes []*PolicyRoute The policy route information configured under Vpc vpcPeerings []*VpcPeering Vpc interconnection information enableExternal Bool Whether vpc is connected to an external switch","title":"VpcSpec"},{"location":"en/reference/kube-ovn-api/#staticroute","text":"Property Name Type Description policy String Routing policy, takes the value of policySrc or policyDst cidr String Routing cidr value nextHopIP String The next hop information of the route","title":"StaticRoute"},{"location":"en/reference/kube-ovn-api/#policyroute","text":"Property Name Type Description priority Int32 Priority for policy route match String Match expression for policy route action String Action for policy route, the value is in the range of allow , drop , reroute nextHopIP String The next hop of the policy route, separated by commas in the case of ECMP routing","title":"PolicyRoute"},{"location":"en/reference/kube-ovn-api/#vpcpeering","text":"Property Name Type Description remoteVpc String Name of the interconnected peering vpc localConnectIP String The local ip for vpc used to connect to peer vpc","title":"VpcPeering"},{"location":"en/reference/kube-ovn-api/#vpcstatus","text":"Property Name Type Description conditions []VpcCondition Vpc status change information, refer to the beginning of the documentation for the definition of Condition standby Bool Whether the vpc creation is complete, the subnet under the vpc needs to wait for the vpc creation to complete other proceeding default Bool Whether it is the default vpc defaultLogicalSwitch String The default subnet under vpc router String The logical-router name for the vpc tcpLoadBalancer String TCP LB information for vpc udpLoadBalancer String UDP LB information for vpc tcpSessionLoadBalancer String TCP Session Hold LB Information for Vpc udpSessionLoadBalancer String UDP session hold LB information for Vpc subnets []String List of subnets for vpc vpcPeerings []String List of peer vpcs for vpc interconnection enableExternal Bool Whether the vpc is connected to an external switch","title":"VpcStatus"},{"location":"en/reference/kube-ovn-api/#vpcnatgateway","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value VpcNatGateway metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcNatSpec Vpc gateway specific configuration information","title":"VpcNatGateway"},{"location":"en/reference/kube-ovn-api/#vpcnatspec","text":"Property Name Type Description vpc String Vpc name which the vpc gateway belongs to subnet String The name of the subnet to which the gateway pod belongs lanIp String The IP address assigned to the gateway pod selector []String Standard Kubernetes selector match information tolerations []VpcNatToleration Standard Kubernetes tolerance information","title":"VpcNatSpec"},{"location":"en/reference/kube-ovn-api/#vpcnattoleration","text":"Property Name Type Description key String The key information of the taint tolerance operator String Takes the value of Exists or Equal value String The value information of the taint tolerance effect String The effect of the taint tolerance, takes the value of NoExecute , NoSchedule , or PreferNoSchedule tolerationSeconds Int64 The amount of time the pod can continue to run on the node after the taint is added The meaning of the above tolerance fields can be found in the official Kubernetes documentation Taint and Tolerance .","title":"VpcNatToleration"},{"location":"en/reference/kube-ovn-api/#iptableseip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesEIP metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesEipSpec IptablesEIP specific configuration information used by vpc gateway status IptablesEipStatus IptablesEIP status information used by vpc gateway","title":"IptablesEIP"},{"location":"en/reference/kube-ovn-api/#iptableseipspec","text":"Property Name Type Description v4ip String IptablesEIP v4 address v6ip String IptablesEIP v6 address macAddress String The assigned mac address, not actually used natGwDp String Vpc gateway name","title":"IptablesEipSpec"},{"location":"en/reference/kube-ovn-api/#iptableseipstatus","text":"Property Name Type Description ready Bool Whether IptablesEIP is configured complete ip String The IP address used by IptablesEIP, currently only IPv4 addresses are supported redo String IptablesEIP crd creation or update time nat String The type of IptablesEIP, either fip , snat , or dnat conditions []IptablesEIPCondition IptablesEIP status change information, refer to the beginning of the documentation for the definition of Condition","title":"IptablesEipStatus"},{"location":"en/reference/kube-ovn-api/#iptablesfiprule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesFIPRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesFIPRuleSpec The IptablesFIPRule specific configuration information used by vpc gateway status IptablesFIPRuleStatus IptablesFIPRule status information used by vpc gateway","title":"IptablesFIPRule"},{"location":"en/reference/kube-ovn-api/#iptablesfiprulespec","text":"Property Name Type Description eip String Name of the IptablesEIP used for IptablesFIPRule internalIp String The corresponding internal IP address","title":"IptablesFIPRuleSpec"},{"location":"en/reference/kube-ovn-api/#iptablesfiprulestatus","text":"Property Name Type Description ready Bool Whether IptablesFIPRule is configured or not v4ip String The v4 IP address used by IptablesEIP v6ip String The v6 IP address used by IptablesEIP natGwDp String Vpc gateway name redo String IptablesFIPRule crd creation or update time conditions []IptablesFIPRuleCondition IptablesFIPRule status change information, refer to the beginning of the documentation for the definition of Condition","title":"IptablesFIPRuleStatus"},{"location":"en/reference/kube-ovn-api/#iptablessnatrule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesSnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesSnatRuleSpec The IptablesSnatRule specific configuration information used by the vpc gateway status IptablesSnatRuleStatus IptablesSnatRule status information used by vpc gateway","title":"IptablesSnatRule"},{"location":"en/reference/kube-ovn-api/#iptablessnatrulespec","text":"Property Name Type Description eip String Name of the IptablesEIP used by IptablesSnatRule internalIp String IptablesSnatRule's corresponding internal IP address","title":"IptablesSnatRuleSpec"},{"location":"en/reference/kube-ovn-api/#iptablessnatrulestatus","text":"Property Name Type Description ready Bool Whether the configuration is complete v4ip String The v4 IP address used by IptablesSnatRule v6ip String The v6 IP address used by IptablesSnatRule natGwDp String Vpc gateway name redo String IptablesSnatRule crd creation or update time conditions []IptablesSnatRuleCondition IptablesSnatRule status change information, refer to the beginning of the documentation for the definition of Condition","title":"IptablesSnatRuleStatus"},{"location":"en/reference/kube-ovn-api/#iptablesdnatrule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesDnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesDnatRuleSpec The IptablesDnatRule specific configuration information used by vpc gateway status IptablesDnatRuleStatus IptablesDnatRule status information used by vpc gateway","title":"IptablesDnatRule"},{"location":"en/reference/kube-ovn-api/#iptablesdnatrulespec","text":"Property Name Type Description eip Sting Name of IptablesEIP used by IptablesDnatRule externalPort Sting External port used by IptablesDnatRule protocol Sting Vpc gateway dnat protocol type internalIp Sting Internal IP address used by IptablesDnatRule internalPort Sting Internal port used by IptablesDnatRule","title":"IptablesDnatRuleSpec"},{"location":"en/reference/kube-ovn-api/#iptablesdnatrulestatus","text":"Property Name Type Description ready Bool Whether the configuration is complete v4ip String The v4 IP address used by IptablesDnatRule v6ip String The v6 IP address used by IptablesDnatRule natGwDp String Vpc gateway name redo String IptablesDnatRule crd creation or update time conditions []IptablesDnatRuleCondition IptablesDnatRule Status change information, refer to the beginning of the documentation for the definition of Condition","title":"IptablesDnatRuleStatus"},{"location":"en/reference/kube-ovn-api/#vpcdns","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value VpcDns metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcDnsSpec VpcDns specific configuration information status VpcDnsStatus VpcDns status information","title":"VpcDns"},{"location":"en/reference/kube-ovn-api/#vpcdnsspec","text":"Property Name Type Description vpc String Name of the vpc where VpcDns is located subnet String The subnet name of the address assigned to the VpcDns pod","title":"VpcDnsSpec"},{"location":"en/reference/kube-ovn-api/#vpcdnsstatus","text":"Property Name Type Description conditions []VpcDnsCondition VpcDns status change information, refer to the beginning of the document for the definition of Condition active Bool Whether VpcDns is in use For detailed documentation on the use of VpcDns, see Customizing VPC DNS .","title":"VpcDnsStatus"},{"location":"en/reference/kube-ovn-api/#switchlbrule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value SwitchLBRule metadata ObjectMeta Standard Kubernetes resource metadata information spec SwitchLBRuleSpec SwitchLBRule specific configuration information status SwitchLBRuleStatus SwitchLBRule status information","title":"SwitchLBRule"},{"location":"en/reference/kube-ovn-api/#switchlbrulespec","text":"Property Name Type Description vip String Vip address of SwitchLBRule namespace String SwitchLBRule's namespace selector []String Standard Kubernetes selector match information sessionAffinity String Standard Kubernetes service sessionAffinity value ports []SlrPort List of SwitchLBRule ports For detailed configuration information of SwitchLBRule, you can refer to Customizing VPC Internal Load Balancing .","title":"SwitchLBRuleSpec"},{"location":"en/reference/kube-ovn-api/#slrport","text":"Property Name Type Description name String Port name port Int32 Port number targetPort Int32 Target port of SwitchLBRule protocol String Protocol type","title":"SlrPort"},{"location":"en/reference/kube-ovn-api/#switchlbrulestatus","text":"Property Name Type Description conditions []SwitchLBRuleCondition SwitchLBRule status change information, refer to the beginning of the document for the definition of Condition ports String Port information service String Name of the service","title":"SwitchLBRuleStatus"},{"location":"en/reference/kube-ovn-api/#security-group-and-vip","text":"","title":"Security Group and Vip"},{"location":"en/reference/kube-ovn-api/#securitygroup","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have a value of SecurityGroup metadata ObjectMeta Standard Kubernetes resource metadata information spec SecurityGroupSpec Security Group specific configuration information status SecurityGroupStatus Security group status information","title":"SecurityGroup"},{"location":"en/reference/kube-ovn-api/#securitygroupspec","text":"Property Name Type Description ingressRules []*SgRule Inbound security group rules egressRules []*SgRule Outbound security group rules allowSameGroupTraffic Bool Whether lsps in the same security group can interoperate and whether traffic rules need to be updated","title":"SecurityGroupSpec"},{"location":"en/reference/kube-ovn-api/#sgrule","text":"Property Name Type Description ipVersion String IP version number, ipv4 or ipv6 protocol String The value of icmp , tcp , or udp priority Int Acl priority. The value range is 1-200, the smaller the value, the higher the priority. remoteType String The value is either address or securityGroup remoteAddress String The address of the other side remoteSecurityGroup String The name of security group on the other side portRangeMin Int The starting value of the port range, the minimum value is 1. portRangeMax Int The ending value of the port range, the maximum value is 65535. policy String The value is allow or drop","title":"SgRule"},{"location":"en/reference/kube-ovn-api/#securitygroupstatus","text":"Property Name Type Description portGroup String The name of the port-group for the security group allowSameGroupTraffic Bool Whether lsps in the same security group can interoperate, and whether the security group traffic rules need to be updated ingressMd5 String The MD5 value of the inbound security group rule egressMd5 String The MD5 value of the outbound security group rule ingressLastSyncSuccess Bool Whether the last synchronization of the inbound rule was successful egressLastSyncSuccess Bool Whether the last synchronization of the outbound rule was successful","title":"SecurityGroupStatus"},{"location":"en/reference/kube-ovn-api/#vip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vip metadata ObjectMeta Standard Kubernetes resource metadata information spec VipSpec Vip specific configuration information status VipStatus Vip status information","title":"Vip"},{"location":"en/reference/kube-ovn-api/#vipspec","text":"Property Name Type Description namespace String Vip's namespace subnet String Vip's subnet v4ip String Vip IPv4 ip address v6ip String Vip IPv6 ip address macAddress String Vip mac address parentV4ip String Not currently in use parentV6ip String Not currently in use parentMac String Not currently in use attachSubnets []String This field is deprecated and no longer used","title":"VipSpec"},{"location":"en/reference/kube-ovn-api/#vipstatus","text":"Property Name Type Description conditions []VipCondition Vip status change information, refer to the beginning of the documentation for the definition of Condition ready Bool Vip is ready or not v4ip String Vip IPv4 ip address, should be the same as the spec field v6ip String Vip IPv6 ip address, should be the same as the spec field mac String The vip mac address, which should be the same as the spec field pv4ip String Not currently used pv6ip String Not currently used pmac String Not currently used","title":"VipStatus"},{"location":"en/reference/kube-ovn-api/#ovneip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnEip metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnEipSpec OvnEip specific configuration information for default vpc status OvnEipStatus OvnEip status information for default vpc","title":"OvnEip"},{"location":"en/reference/kube-ovn-api/#ovneipspec","text":"Property Name Type Description externalSubnet String OvnEip's subnet name v4ip String OvnEip IP address macAddress String OvnEip Mac address type String OvnEip use type, the value can be fip , snat or lrp","title":"OvnEipSpec"},{"location":"en/reference/kube-ovn-api/#ovneipstatus","text":"Property Name Type Description conditions []OvnEipCondition OvnEip status change information, refer to the beginning of the documentation for the definition of Condition v4ip String The IPv4 ip address used by ovnEip macAddress String Mac address used by ovnEip","title":"OvnEipStatus"},{"location":"en/reference/kube-ovn-api/#ovnfip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnFip metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnFipSpec OvnFip specific configuration information in default vpc status OvnFipStatus OvnFip status information in default vpc","title":"OvnFip"},{"location":"en/reference/kube-ovn-api/#ovnfipspec","text":"Property Name Type Description ovnEip String Name of the bound ovnEip ipName String The IP crd name corresponding to the bound Pod","title":"OvnFipSpec"},{"location":"en/reference/kube-ovn-api/#ovnfipstatus","text":"Property Name Type Description ready Bool OvnFip is ready or not v4Eip String Name of the ovnEip to which ovnFip is bound v4Ip String The ovnEip address currently in use macAddress String OvnFip's configured mac address vpc String The name of the vpc where ovnFip is located conditions []OvnFipCondition OvnFip status change information, refer to the beginning of the document for the definition of Condition","title":"OvnFipStatus"},{"location":"en/reference/kube-ovn-api/#ovnsnatrule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnSnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnSnatRuleSpec OvnSnatRule specific configuration information in default vpc status OvnSnatRuleStatus OvnSnatRule status information in default vpc","title":"OvnSnatRule"},{"location":"en/reference/kube-ovn-api/#ovnsnatrulespec","text":"Property Name Type Description ovnEip String Name of the ovnEip to which ovnSnatRule is bound vpcSubnet String The name of the subnet configured by ovnSnatRule ipName String The IP crd name corresponding to the ovnSnatRule bound Pod","title":"OvnSnatRuleSpec"},{"location":"en/reference/kube-ovn-api/#ovnsnatrulestatus","text":"Property Name Type Description ready Bool OvnSnatRule is ready or not v4Eip String The ovnEip address to which ovnSnatRule is bound v4IpCidr String The cidr address used to configure snat in the logical-router vpc String The name of the vpc where ovnSnatRule is located conditions []OvnSnatRuleCondition OvnSnatRule status change information, refer to the beginning of the document for the definition of Condition \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OvnSnatRuleStatus"},{"location":"en/reference/kube-ovn-pinger-args/","text":"Kube-OVN-Pinger args Reference \u00b6 Based on the Kube-OVN v1.12.0 version, We have compiled the parameters supported by Kube-ovn-pinger, and listed the value types, meanings, and default values of each field defined by the parameters for reference Args Describeption \u00b6 Arg Name Type Description Default Value port Int metrics port 8080 kubeconfig String Path to kubeconfig file with authorization and master location information. If not set use the inCluster token. \"\" ds-namespace String kube-ovn-pinger daemonset namespace \"kube-system\" ds-name String kube-ovn-pinger daemonset name \"kube-ovn-pinger\" interval Int interval seconds between consecutive pings 5 mode String server or job Mode \"server\" exit-code Int exit code when failure happens 0 internal-dns String check dns from pod \"kubernetes.default\" external-dns String check external dns resolve from pod \"\" external-address String check ping connection to an external address \"114.114.114.114\" network-mode String The cni plugin current cluster used \"kube-ovn\" enable-metrics Bool Whether to support metrics query true ovs.timeout Int Timeout on JSON-RPC requests to OVS. 2 system.run.dir String OVS default run directory. \"/var/run/openvswitch\" database.vswitch.name String The name of OVS db. \"Open_vSwitch\" database.vswitch.socket.remote String JSON-RPC unix socket to OVS db. \"unix:/var/run/openvswitch/db.sock\" database.vswitch.file.data.path String OVS db file. \"/etc/openvswitch/conf.db\" database.vswitch.file.log.path String OVS db log file. \"/var/log/openvswitch/ovsdb-server.log\" database.vswitch.file.pid.path String OVS db process id file. \"/var/run/openvswitch/ovsdb-server.pid\" database.vswitch.file.system.id.path String OVS system id file. \"/etc/openvswitch/system-id.conf\" service.vswitchd.file.log.path String OVS vswitchd daemon log file. \"/var/log/openvswitch/ovs-vswitchd.log\" service.vswitchd.file.pid.path String OVS vswitchd daemon process id file. \"/var/run/openvswitch/ovs-vswitchd.pid\" service.ovncontroller.file.log.path String OVN controller daemon log file. \"/var/log/ovn/ovn-controller.log\" service.ovncontroller.file.pid.path String OVN controller daemon process id file. \"/var/run/ovn/ovn-controller.pid\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kube-ovn-pinger args description"},{"location":"en/reference/kube-ovn-pinger-args/#kube-ovn-pinger-args-reference","text":"Based on the Kube-OVN v1.12.0 version, We have compiled the parameters supported by Kube-ovn-pinger, and listed the value types, meanings, and default values of each field defined by the parameters for reference","title":"Kube-OVN-Pinger args Reference"},{"location":"en/reference/kube-ovn-pinger-args/#args-describeption","text":"Arg Name Type Description Default Value port Int metrics port 8080 kubeconfig String Path to kubeconfig file with authorization and master location information. If not set use the inCluster token. \"\" ds-namespace String kube-ovn-pinger daemonset namespace \"kube-system\" ds-name String kube-ovn-pinger daemonset name \"kube-ovn-pinger\" interval Int interval seconds between consecutive pings 5 mode String server or job Mode \"server\" exit-code Int exit code when failure happens 0 internal-dns String check dns from pod \"kubernetes.default\" external-dns String check external dns resolve from pod \"\" external-address String check ping connection to an external address \"114.114.114.114\" network-mode String The cni plugin current cluster used \"kube-ovn\" enable-metrics Bool Whether to support metrics query true ovs.timeout Int Timeout on JSON-RPC requests to OVS. 2 system.run.dir String OVS default run directory. \"/var/run/openvswitch\" database.vswitch.name String The name of OVS db. \"Open_vSwitch\" database.vswitch.socket.remote String JSON-RPC unix socket to OVS db. \"unix:/var/run/openvswitch/db.sock\" database.vswitch.file.data.path String OVS db file. \"/etc/openvswitch/conf.db\" database.vswitch.file.log.path String OVS db log file. \"/var/log/openvswitch/ovsdb-server.log\" database.vswitch.file.pid.path String OVS db process id file. \"/var/run/openvswitch/ovsdb-server.pid\" database.vswitch.file.system.id.path String OVS system id file. \"/etc/openvswitch/system-id.conf\" service.vswitchd.file.log.path String OVS vswitchd daemon log file. \"/var/log/openvswitch/ovs-vswitchd.log\" service.vswitchd.file.pid.path String OVS vswitchd daemon process id file. \"/var/run/openvswitch/ovs-vswitchd.pid\" service.ovncontroller.file.log.path String OVN controller daemon log file. \"/var/log/ovn/ovn-controller.log\" service.ovncontroller.file.pid.path String OVN controller daemon process id file. \"/var/run/ovn/ovn-controller.pid\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Args Describeption"},{"location":"en/reference/metrics/","text":"Metrics \u00b6 This document lists all the monitoring metrics provided by Kube-OVN. ovn-monitor \u00b6 OVN status metrics: Type Metric Description Gauge kube_ovn_ovn_status OVN Health Status. The values are: (2) for standby or follower, (1) for active or leader, (0) for unhealthy. Gauge kube_ovn_failed_req_count The number of failed requests to OVN stack. Gauge kube_ovn_log_file_size The size of a log file associated with an OVN component. Gauge kube_ovn_db_file_size The size of a database file associated with an OVN component. Gauge kube_ovn_chassis_info Whether the OVN chassis is up (1) or down (0), together with additional information about the chassis. Gauge kube_ovn_db_status The status of OVN NB/SB DB, (1) for healthy, (0) for unhealthy. Gauge kube_ovn_logical_switch_info The information about OVN logical switch. This metric is always up (1). Gauge kube_ovn_logical_switch_external_id Provides the external IDs and values associated with OVN logical switches. This metric is always up (1). Gauge kube_ovn_logical_switch_port_binding Provides the association between a logical switch and a logical switch port. This metric is always up (1). Gauge kube_ovn_logical_switch_tunnel_key The value of the tunnel key associated with the logical switch. Gauge kube_ovn_logical_switch_ports_num The number of logical switch ports connected to the OVN logical switch. Gauge kube_ovn_logical_switch_port_info The information about OVN logical switch port. This metric is always up (1). Gauge kube_ovn_logical_switch_port_tunnel_key The value of the tunnel key associated with the logical switch port. Gauge kube_ovn_cluster_enabled Is OVN clustering enabled (1) or not (0). Gauge kube_ovn_cluster_role A metric with a constant '1' value labeled by server role. Gauge kube_ovn_cluster_status A metric with a constant '1' value labeled by server status. Gauge kube_ovn_cluster_term The current raft term known by this server. Gauge kube_ovn_cluster_leader_self Is this server consider itself a leader (1) or not (0). Gauge kube_ovn_cluster_vote_self Is this server voted itself as a leader (1) or not (0). Gauge kube_ovn_cluster_election_timer The current election timer value. Gauge kube_ovn_cluster_log_not_committed The number of log entries not yet committed by this server. Gauge kube_ovn_cluster_log_not_applied The number of log entries not yet applied by this server. Gauge kube_ovn_cluster_log_index_start The log entry index start value associated with this server. Gauge kube_ovn_cluster_log_index_next The log entry index next value associated with this server. Gauge kube_ovn_cluster_inbound_connections_total The total number of inbound connections to the server. Gauge kube_ovn_cluster_outbound_connections_total The total number of outbound connections from the server. Gauge kube_ovn_cluster_inbound_connections_error_total The total number of failed inbound connections to the server. Gauge kube_ovn_cluster_outbound_connections_error_total The total number of failed outbound connections from the server. ovs-monitor \u00b6 ovsdb and vswitchd status metrics: Type Metric Description Gauge ovs_status OVS Health Status. The values are: health(1), unhealthy(0). Gauge ovs_info This metric provides basic information about OVS. It is always set to 1. Gauge failed_req_count The number of failed requests to OVS stack. Gauge log_file_size The size of a log file associated with an OVS component. Gauge db_file_size The size of a database file associated with an OVS component. Gauge datapath Represents an existing datapath. This metrics is always 1. Gauge dp_total Represents total number of datapaths on the system. Gauge dp_if Represents an existing datapath interface. This metrics is always 1. Gauge dp_if_total Represents the number of ports connected to the datapath. Gauge dp_flows_total The number of flows in a datapath. Gauge dp_flows_lookup_hit The number of incoming packets in a datapath matching existing flows in the datapath. Gauge dp_flows_lookup_missed The number of incoming packets in a datapath not matching any existing flow in the datapath. Gauge dp_flows_lookup_lost The number of incoming packets in a datapath destined for userspace process but subsequently dropped before reaching userspace. Gauge dp_masks_hit The total number of masks visited for matching incoming packets. Gauge dp_masks_total The number of masks in a datapath. Gauge dp_masks_hit_ratio The average number of masks visited per packet. It is the ration between hit and total number of packets processed by a datapath. Gauge interface Represents OVS interface. This is the primary metric for all other interface metrics. This metrics is always 1. Gauge interface_admin_state The administrative state of the physical network link of OVS interface. The values are: down(0), up(1), other(2). Gauge interface_link_state The state of the physical network link of OVS interface. The values are: down(0), up(1), other(2). Gauge interface_mac_in_use The MAC address in use by OVS interface. Gauge interface_mtu The currently configured MTU for OVS interface. Gauge interface_of_port Represents the OpenFlow port ID associated with OVS interface. Gauge interface_if_index Represents the interface index associated with OVS interface. Gauge interface_tx_packets Represents the number of transmitted packets by OVS interface. Gauge interface_tx_bytes Represents the number of transmitted bytes by OVS interface. Gauge interface_rx_packets Represents the number of received packets by OVS interface. Gauge interface_rx_bytes Represents the number of received bytes by OVS interface. Gauge interface_rx_crc_err Represents the number of CRC errors for the packets received by OVS interface. Gauge interface_rx_dropped Represents the number of input packets dropped by OVS interface. Gauge interface_rx_errors Represents the total number of packets with errors received by OVS interface. Gauge interface_rx_frame_err Represents the number of frame alignment errors on the packets received by OVS interface. Gauge interface_rx_missed_err Represents the number of packets with RX missed received by OVS interface. Gauge interface_rx_over_err Represents the number of packets with RX overrun received by OVS interface. Gauge interface_tx_dropped Represents the number of output packets dropped by OVS interface. Gauge interface_tx_errors Represents the total number of transmit errors by OVS interface. Gauge interface_collisions Represents the number of collisions on OVS interface. kube-ovn-pinger \u00b6 Network quality related metrics: Type Metric Description Gauge pinger_ovs_up If the ovs on the node is up Gauge pinger_ovs_down If the ovs on the node is down Gauge pinger_ovn_controller_up If the ovn_controller on the node is up Gauge pinger_ovn_controller_down If the ovn_controller on the node is down Gauge pinger_inconsistent_port_binding The number of mismatch port bindings between ovs and ovn-sb Gauge pinger_apiserver_healthy If the apiserver request is healthy on this node Gauge pinger_apiserver_unhealthy If the apiserver request is unhealthy on this node Histogram pinger_apiserver_latency_ms The latency ms histogram the node request apiserver Gauge pinger_internal_dns_healthy If the internal dns request is unhealthy on this node Gauge pinger_internal_dns_unhealthy If the internal dns request is unhealthy on this node Histogram pinger_internal_dns_latency_ms The latency ms histogram the node request internal dns Gauge pinger_external_dns_health If the external dns request is healthy on this node Gauge pinger_external_dns_unhealthy If the external dns request is unhealthy on this node Histogram pinger_external_dns_latency_ms The latency ms histogram the node request external dns Histogram pinger_pod_ping_latency_ms The latency ms histogram for pod peer ping Gauge pinger_pod_ping_lost_total The lost count for pod peer ping Gauge pinger_pod_ping_count_total The total count for pod peer ping Histogram pinger_node_ping_latency_ms The latency ms histogram for pod ping node Gauge pinger_node_ping_lost_total The lost count for pod ping node Gauge pinger_node_ping_count_total The total count for pod ping node Histogram pinger_external_ping_latency_ms The latency ms histogram for pod ping external address Gauge pinger_external_lost_total The lost count for pod ping external address kube-ovn-controller \u00b6 kube-ovn-controller status metrics\uff1a Type Metric Description Histogram rest_client_request_latency_seconds Request latency in seconds. Broken down by verb and URL Counter rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host Counter lists_total Total number of API lists done by the reflectors Summary list_duration_seconds How long an API list takes to return and decode for the reflectors Summary items_per_list How many items an API list returns to the reflectors Counter watches_total Total number of API watches done by the reflectors Counter short_watches_total Total number of short API watches done by the reflectors Summary watch_duration_seconds How long an API watch takes to return and decode for the reflectors Summary items_per_watch How many items an API watch returns to the reflectors Gauge last_resource_version Last resource version seen for the reflectors Histogram ovs_client_request_latency_milliseconds The latency histogram for ovs request Gauge subnet_available_ip_count The available num of ip address in subnet Gauge subnet_used_ip_count The used num of ip address in subnet kube-ovn-cni \u00b6 kube-ovn-cni status metrics: Type Metric Description Histogram cni_op_latency_seconds The latency seconds for cni operations Counter cni_wait_address_seconds_total Latency that cni wait controller to assign an address Counter cni_wait_connectivity_seconds_total Latency that cni wait address ready in overlay network Counter cni_wait_route_seconds_total Latency that cni wait controller to add routed annotation to pod Histogram rest_client_request_latency_seconds Request latency in seconds. Broken down by verb and URL Counter rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host Counter lists_total Total number of API lists done by the reflectors Summary list_duration_seconds How long an API list takes to return and decode for the reflectors Summary items_per_list How many items an API list returns to the reflectors Counter watches_total Total number of API watches done by the reflectors Counter short_watches_total Total number of short API watches done by the reflectors Summary watch_duration_seconds How long an API watch takes to return and decode for the reflectors Summary items_per_watch How many items an API watch returns to the reflectors Gauge last_resource_version Last resource version seen for the reflectors Histogram ovs_client_request_latency_milliseconds The latency histogram for ovs request \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Metrics"},{"location":"en/reference/metrics/#metrics","text":"This document lists all the monitoring metrics provided by Kube-OVN.","title":"Metrics"},{"location":"en/reference/metrics/#ovn-monitor","text":"OVN status metrics: Type Metric Description Gauge kube_ovn_ovn_status OVN Health Status. The values are: (2) for standby or follower, (1) for active or leader, (0) for unhealthy. Gauge kube_ovn_failed_req_count The number of failed requests to OVN stack. Gauge kube_ovn_log_file_size The size of a log file associated with an OVN component. Gauge kube_ovn_db_file_size The size of a database file associated with an OVN component. Gauge kube_ovn_chassis_info Whether the OVN chassis is up (1) or down (0), together with additional information about the chassis. Gauge kube_ovn_db_status The status of OVN NB/SB DB, (1) for healthy, (0) for unhealthy. Gauge kube_ovn_logical_switch_info The information about OVN logical switch. This metric is always up (1). Gauge kube_ovn_logical_switch_external_id Provides the external IDs and values associated with OVN logical switches. This metric is always up (1). Gauge kube_ovn_logical_switch_port_binding Provides the association between a logical switch and a logical switch port. This metric is always up (1). Gauge kube_ovn_logical_switch_tunnel_key The value of the tunnel key associated with the logical switch. Gauge kube_ovn_logical_switch_ports_num The number of logical switch ports connected to the OVN logical switch. Gauge kube_ovn_logical_switch_port_info The information about OVN logical switch port. This metric is always up (1). Gauge kube_ovn_logical_switch_port_tunnel_key The value of the tunnel key associated with the logical switch port. Gauge kube_ovn_cluster_enabled Is OVN clustering enabled (1) or not (0). Gauge kube_ovn_cluster_role A metric with a constant '1' value labeled by server role. Gauge kube_ovn_cluster_status A metric with a constant '1' value labeled by server status. Gauge kube_ovn_cluster_term The current raft term known by this server. Gauge kube_ovn_cluster_leader_self Is this server consider itself a leader (1) or not (0). Gauge kube_ovn_cluster_vote_self Is this server voted itself as a leader (1) or not (0). Gauge kube_ovn_cluster_election_timer The current election timer value. Gauge kube_ovn_cluster_log_not_committed The number of log entries not yet committed by this server. Gauge kube_ovn_cluster_log_not_applied The number of log entries not yet applied by this server. Gauge kube_ovn_cluster_log_index_start The log entry index start value associated with this server. Gauge kube_ovn_cluster_log_index_next The log entry index next value associated with this server. Gauge kube_ovn_cluster_inbound_connections_total The total number of inbound connections to the server. Gauge kube_ovn_cluster_outbound_connections_total The total number of outbound connections from the server. Gauge kube_ovn_cluster_inbound_connections_error_total The total number of failed inbound connections to the server. Gauge kube_ovn_cluster_outbound_connections_error_total The total number of failed outbound connections from the server.","title":"ovn-monitor"},{"location":"en/reference/metrics/#ovs-monitor","text":"ovsdb and vswitchd status metrics: Type Metric Description Gauge ovs_status OVS Health Status. The values are: health(1), unhealthy(0). Gauge ovs_info This metric provides basic information about OVS. It is always set to 1. Gauge failed_req_count The number of failed requests to OVS stack. Gauge log_file_size The size of a log file associated with an OVS component. Gauge db_file_size The size of a database file associated with an OVS component. Gauge datapath Represents an existing datapath. This metrics is always 1. Gauge dp_total Represents total number of datapaths on the system. Gauge dp_if Represents an existing datapath interface. This metrics is always 1. Gauge dp_if_total Represents the number of ports connected to the datapath. Gauge dp_flows_total The number of flows in a datapath. Gauge dp_flows_lookup_hit The number of incoming packets in a datapath matching existing flows in the datapath. Gauge dp_flows_lookup_missed The number of incoming packets in a datapath not matching any existing flow in the datapath. Gauge dp_flows_lookup_lost The number of incoming packets in a datapath destined for userspace process but subsequently dropped before reaching userspace. Gauge dp_masks_hit The total number of masks visited for matching incoming packets. Gauge dp_masks_total The number of masks in a datapath. Gauge dp_masks_hit_ratio The average number of masks visited per packet. It is the ration between hit and total number of packets processed by a datapath. Gauge interface Represents OVS interface. This is the primary metric for all other interface metrics. This metrics is always 1. Gauge interface_admin_state The administrative state of the physical network link of OVS interface. The values are: down(0), up(1), other(2). Gauge interface_link_state The state of the physical network link of OVS interface. The values are: down(0), up(1), other(2). Gauge interface_mac_in_use The MAC address in use by OVS interface. Gauge interface_mtu The currently configured MTU for OVS interface. Gauge interface_of_port Represents the OpenFlow port ID associated with OVS interface. Gauge interface_if_index Represents the interface index associated with OVS interface. Gauge interface_tx_packets Represents the number of transmitted packets by OVS interface. Gauge interface_tx_bytes Represents the number of transmitted bytes by OVS interface. Gauge interface_rx_packets Represents the number of received packets by OVS interface. Gauge interface_rx_bytes Represents the number of received bytes by OVS interface. Gauge interface_rx_crc_err Represents the number of CRC errors for the packets received by OVS interface. Gauge interface_rx_dropped Represents the number of input packets dropped by OVS interface. Gauge interface_rx_errors Represents the total number of packets with errors received by OVS interface. Gauge interface_rx_frame_err Represents the number of frame alignment errors on the packets received by OVS interface. Gauge interface_rx_missed_err Represents the number of packets with RX missed received by OVS interface. Gauge interface_rx_over_err Represents the number of packets with RX overrun received by OVS interface. Gauge interface_tx_dropped Represents the number of output packets dropped by OVS interface. Gauge interface_tx_errors Represents the total number of transmit errors by OVS interface. Gauge interface_collisions Represents the number of collisions on OVS interface.","title":"ovs-monitor"},{"location":"en/reference/metrics/#kube-ovn-pinger","text":"Network quality related metrics: Type Metric Description Gauge pinger_ovs_up If the ovs on the node is up Gauge pinger_ovs_down If the ovs on the node is down Gauge pinger_ovn_controller_up If the ovn_controller on the node is up Gauge pinger_ovn_controller_down If the ovn_controller on the node is down Gauge pinger_inconsistent_port_binding The number of mismatch port bindings between ovs and ovn-sb Gauge pinger_apiserver_healthy If the apiserver request is healthy on this node Gauge pinger_apiserver_unhealthy If the apiserver request is unhealthy on this node Histogram pinger_apiserver_latency_ms The latency ms histogram the node request apiserver Gauge pinger_internal_dns_healthy If the internal dns request is unhealthy on this node Gauge pinger_internal_dns_unhealthy If the internal dns request is unhealthy on this node Histogram pinger_internal_dns_latency_ms The latency ms histogram the node request internal dns Gauge pinger_external_dns_health If the external dns request is healthy on this node Gauge pinger_external_dns_unhealthy If the external dns request is unhealthy on this node Histogram pinger_external_dns_latency_ms The latency ms histogram the node request external dns Histogram pinger_pod_ping_latency_ms The latency ms histogram for pod peer ping Gauge pinger_pod_ping_lost_total The lost count for pod peer ping Gauge pinger_pod_ping_count_total The total count for pod peer ping Histogram pinger_node_ping_latency_ms The latency ms histogram for pod ping node Gauge pinger_node_ping_lost_total The lost count for pod ping node Gauge pinger_node_ping_count_total The total count for pod ping node Histogram pinger_external_ping_latency_ms The latency ms histogram for pod ping external address Gauge pinger_external_lost_total The lost count for pod ping external address","title":"kube-ovn-pinger"},{"location":"en/reference/metrics/#kube-ovn-controller","text":"kube-ovn-controller status metrics\uff1a Type Metric Description Histogram rest_client_request_latency_seconds Request latency in seconds. Broken down by verb and URL Counter rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host Counter lists_total Total number of API lists done by the reflectors Summary list_duration_seconds How long an API list takes to return and decode for the reflectors Summary items_per_list How many items an API list returns to the reflectors Counter watches_total Total number of API watches done by the reflectors Counter short_watches_total Total number of short API watches done by the reflectors Summary watch_duration_seconds How long an API watch takes to return and decode for the reflectors Summary items_per_watch How many items an API watch returns to the reflectors Gauge last_resource_version Last resource version seen for the reflectors Histogram ovs_client_request_latency_milliseconds The latency histogram for ovs request Gauge subnet_available_ip_count The available num of ip address in subnet Gauge subnet_used_ip_count The used num of ip address in subnet","title":"kube-ovn-controller"},{"location":"en/reference/metrics/#kube-ovn-cni","text":"kube-ovn-cni status metrics: Type Metric Description Histogram cni_op_latency_seconds The latency seconds for cni operations Counter cni_wait_address_seconds_total Latency that cni wait controller to assign an address Counter cni_wait_connectivity_seconds_total Latency that cni wait address ready in overlay network Counter cni_wait_route_seconds_total Latency that cni wait controller to add routed annotation to pod Histogram rest_client_request_latency_seconds Request latency in seconds. Broken down by verb and URL Counter rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host Counter lists_total Total number of API lists done by the reflectors Summary list_duration_seconds How long an API list takes to return and decode for the reflectors Summary items_per_list How many items an API list returns to the reflectors Counter watches_total Total number of API watches done by the reflectors Counter short_watches_total Total number of short API watches done by the reflectors Summary watch_duration_seconds How long an API watch takes to return and decode for the reflectors Summary items_per_watch How many items an API watch returns to the reflectors Gauge last_resource_version Last resource version seen for the reflectors Histogram ovs_client_request_latency_milliseconds The latency histogram for ovs request \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kube-ovn-cni"},{"location":"en/reference/ovs-ovn-customized/","text":"OVS/OVN Customization \u00b6 Upstream OVN/OVS was originally designed with the goal of a general purpose SDN controller and data plane. Due to some specific usage of the Kubernetes network,Kube-OVN only focused on part of the features. In order to achieve better performance, stability and specific features, Kube-OVN has made some modifications to the upstream OVN/OVS. Users using their own OVN/OVS with Kube-OVN controllers need to be aware of the possible impact of the following changes: Did not merge into the upstream modification. 38df6fa3f7 Adjust the election timer to avoid large-scale cluster election jitter. d4888c4e75 add fdb update logging. d4888c4e75 fdb: fix mac learning in environments with hairpin enabled. 9a81b91368 ovsdb-tool: add optional server id parameter for \"join-cluster\" command. 0700cb90f9 Destination non-service traffic bypasses conntrack to improve performance on a particular data path. c48049a64f ECMP algorithm is adjusted from dp_hash to hash to avoid the hash error problem in some kernels. 64383c14a9 Fix kernel Crash issue under Windows. 08a95db2ca Support for github action builds on Windows. 680e77a190 Windows uses tcp listening by default. 05e57b3227 add support for windows. 0181b68be1 br-int controller: listen on 127.0.0.1:6653 by default. b3801ecb73 modify src route priority. 977e569539 fix reaching resubmit limit in underlay. 45a4a22161 ovn-nbctl: do not remove LB if vips is empty. 540592b9ff Replaces the Mac address as the destination address after DNAT to reduce additional performance overhead. 10972d9632 Fix vswitchd ofport_usage memory leak. Merged into upstream modification: 20626ea909 Multicast traffic bypasses LB and ACL processing stages to improve specific data path performance. a2d9ff3ccd Deb build adds compile optimization options. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVS/OVN Customization"},{"location":"en/reference/ovs-ovn-customized/#ovsovn-customization","text":"Upstream OVN/OVS was originally designed with the goal of a general purpose SDN controller and data plane. Due to some specific usage of the Kubernetes network,Kube-OVN only focused on part of the features. In order to achieve better performance, stability and specific features, Kube-OVN has made some modifications to the upstream OVN/OVS. Users using their own OVN/OVS with Kube-OVN controllers need to be aware of the possible impact of the following changes: Did not merge into the upstream modification. 38df6fa3f7 Adjust the election timer to avoid large-scale cluster election jitter. d4888c4e75 add fdb update logging. d4888c4e75 fdb: fix mac learning in environments with hairpin enabled. 9a81b91368 ovsdb-tool: add optional server id parameter for \"join-cluster\" command. 0700cb90f9 Destination non-service traffic bypasses conntrack to improve performance on a particular data path. c48049a64f ECMP algorithm is adjusted from dp_hash to hash to avoid the hash error problem in some kernels. 64383c14a9 Fix kernel Crash issue under Windows. 08a95db2ca Support for github action builds on Windows. 680e77a190 Windows uses tcp listening by default. 05e57b3227 add support for windows. 0181b68be1 br-int controller: listen on 127.0.0.1:6653 by default. b3801ecb73 modify src route priority. 977e569539 fix reaching resubmit limit in underlay. 45a4a22161 ovn-nbctl: do not remove LB if vips is empty. 540592b9ff Replaces the Mac address as the destination address after DNAT to reduce additional performance overhead. 10972d9632 Fix vswitchd ofport_usage memory leak. Merged into upstream modification: 20626ea909 Multicast traffic bypasses LB and ACL processing stages to improve specific data path performance. a2d9ff3ccd Deb build adds compile optimization options. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVS/OVN Customization"},{"location":"en/reference/tunnel-protocol/","text":"Tunnel Protocol Selection \u00b6 Kube-OVN uses OVN/OVS as the data plane implementation and currently supports Geneve , Vxlan and STT tunnel encapsulation protocols. These three protocols differ in terms of functionality, performance and ease of use. This document will describe the differences in the use of the three protocols so that users can choose according to their situation. Geneve \u00b6 The Geneve protocol is the default tunneling protocol selected during Kube-OVN deployment and is also the default recommended tunneling protocol for OVN. This protocol is widely supported in the kernel and can be accelerated using the generic offload capability of modern NICs. Since Geneve has a variable header, it is possible to use 24bit space to mark different datapaths users can create a larger number of virtual networks. If you are using Mellanox or Corigine SmartNIC OVS offload, Geneve requires a higher kernel version. Upstream kernel of 5.4 or higher, or other compatible kernels that backports this feature. Due to the use of UDP encapsulation, this protocol does not make good use of the TCP-related offloads of modern NICs when handling TCP over UDP, and consumes more CPU resources when handling large packets. Vxlan \u00b6 Vxlan is a recently supported protocol in the upstream OVN, which is widely supported in the kernel and can be accelerated using the common offload capabilities of modern NICs. Due to the limited length of the protocol header and the additional space required for OVN orchestration, there is a limit to the number of datapaths that can be created, with a maximum of 4096 datapaths and a maximum of 4096 ports under each datapath. Also, inport -based ACLs are not supported due to header length limitations. Vxlan offloading is supported in common kernels if using Mellanox or Corigine SmartNIC. Due to the use of UDP encapsulation, this protocol does not make good use of the TCP-related offloads of modern NICs when handling TCP over UDP, and consumes more CPU resources when handling large packets. STT \u00b6 The STT protocol is an early tunneling protocol supported by the OVN that uses TCP-like headers to take advantage of the TCP offload capabilities common to modern NICs and significantly increase TCP throughput. The protocol also has a long header to support full OVN capabilities and large-scale datapaths. This protocol is not supported in the kernel. To use it, you need to compile an additional OVS kernel module and recompile the new version of the kernel module when upgrading the kernel. This protocol is not currently supported by the SmartNic and cannot use the offloading capability of OVS offloading. References \u00b6 https://ipwithease.com/vxlan-vs-geneve-understand-the-difference/ OVN FAQ What is Geneve \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Tunnel Protocol Selection"},{"location":"en/reference/tunnel-protocol/#tunnel-protocol-selection","text":"Kube-OVN uses OVN/OVS as the data plane implementation and currently supports Geneve , Vxlan and STT tunnel encapsulation protocols. These three protocols differ in terms of functionality, performance and ease of use. This document will describe the differences in the use of the three protocols so that users can choose according to their situation.","title":"Tunnel Protocol Selection"},{"location":"en/reference/tunnel-protocol/#geneve","text":"The Geneve protocol is the default tunneling protocol selected during Kube-OVN deployment and is also the default recommended tunneling protocol for OVN. This protocol is widely supported in the kernel and can be accelerated using the generic offload capability of modern NICs. Since Geneve has a variable header, it is possible to use 24bit space to mark different datapaths users can create a larger number of virtual networks. If you are using Mellanox or Corigine SmartNIC OVS offload, Geneve requires a higher kernel version. Upstream kernel of 5.4 or higher, or other compatible kernels that backports this feature. Due to the use of UDP encapsulation, this protocol does not make good use of the TCP-related offloads of modern NICs when handling TCP over UDP, and consumes more CPU resources when handling large packets.","title":"Geneve"},{"location":"en/reference/tunnel-protocol/#vxlan","text":"Vxlan is a recently supported protocol in the upstream OVN, which is widely supported in the kernel and can be accelerated using the common offload capabilities of modern NICs. Due to the limited length of the protocol header and the additional space required for OVN orchestration, there is a limit to the number of datapaths that can be created, with a maximum of 4096 datapaths and a maximum of 4096 ports under each datapath. Also, inport -based ACLs are not supported due to header length limitations. Vxlan offloading is supported in common kernels if using Mellanox or Corigine SmartNIC. Due to the use of UDP encapsulation, this protocol does not make good use of the TCP-related offloads of modern NICs when handling TCP over UDP, and consumes more CPU resources when handling large packets.","title":"Vxlan"},{"location":"en/reference/tunnel-protocol/#stt","text":"The STT protocol is an early tunneling protocol supported by the OVN that uses TCP-like headers to take advantage of the TCP offload capabilities common to modern NICs and significantly increase TCP throughput. The protocol also has a long header to support full OVN capabilities and large-scale datapaths. This protocol is not supported in the kernel. To use it, you need to compile an additional OVS kernel module and recompile the new version of the kernel module when upgrading the kernel. This protocol is not currently supported by the SmartNic and cannot use the offloading capability of OVS offloading.","title":"STT"},{"location":"en/reference/tunnel-protocol/#references","text":"https://ipwithease.com/vxlan-vs-geneve-understand-the-difference/ OVN FAQ What is Geneve \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"References"},{"location":"en/reference/underlay-topology/","text":"Underlay Traffic Topology \u00b6 This document describes the forwarding path of traffic in Underlay mode under different scenarios. Pods in Same Node and Same Subnet \u00b6 Internal logical switches exchange packets directly, without access to the external network. Pods in Different Nodes and Same Subnet \u00b6 Packets enter the physic switch via the node NIC and are exchanged by the physic switch. Pods in Same Node and Different Subnets \u00b6 Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. Here br-provider-1 and br-provider-2 can be the same OVS bridge\uff0cmultiple subnet can share a Provider Network\u3002 Pods in Different Nodes and Different Subnets \u00b6 Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. Access to External \u00b6 Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. The communication between nodes and Pods follows the same logic. Overview without Vlan Tag \u00b6 Overview with Vlan Tag \u00b6 Pod visit Service IP \u00b6 Kube-OVN configures load balancing for each Kubernetes Service on a logical switch on each subnet. When a Pod accesses other Pods by accessing the Service IP, a network packet is constructed with the Service IP as the destination address and the MAC address of the gateway as the destination MAC address. After the network packet enters the logical switch, load balancing will intercept and DNAT the network packet to modify the destination IP and port to the IP and port of one of the Endpoint corresponding to the Service. Since the logical switch does not modify the Layer 2 destination MAC address of the network packet, the network packet will still be delivered to the physic gateway after entering the physic switch, and the physic gateway will be required to forward the network packet. Service Backend is the Same Node and Same Subnet Pod \u00b6 Service Backend is the Same Node and Different Subnets Pod \u00b6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Underlay Traffic Topology"},{"location":"en/reference/underlay-topology/#underlay-traffic-topology","text":"This document describes the forwarding path of traffic in Underlay mode under different scenarios.","title":"Underlay Traffic Topology"},{"location":"en/reference/underlay-topology/#pods-in-same-node-and-same-subnet","text":"Internal logical switches exchange packets directly, without access to the external network.","title":"Pods in Same Node and Same Subnet"},{"location":"en/reference/underlay-topology/#pods-in-different-nodes-and-same-subnet","text":"Packets enter the physic switch via the node NIC and are exchanged by the physic switch.","title":"Pods in Different Nodes and Same Subnet"},{"location":"en/reference/underlay-topology/#pods-in-same-node-and-different-subnets","text":"Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. Here br-provider-1 and br-provider-2 can be the same OVS bridge\uff0cmultiple subnet can share a Provider Network\u3002","title":"Pods in Same Node and Different Subnets"},{"location":"en/reference/underlay-topology/#pods-in-different-nodes-and-different-subnets","text":"Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers.","title":"Pods in Different Nodes and Different Subnets"},{"location":"en/reference/underlay-topology/#access-to-external","text":"Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. The communication between nodes and Pods follows the same logic.","title":"Access to External"},{"location":"en/reference/underlay-topology/#overview-without-vlan-tag","text":"","title":"Overview without Vlan Tag"},{"location":"en/reference/underlay-topology/#overview-with-vlan-tag","text":"","title":"Overview with Vlan Tag"},{"location":"en/reference/underlay-topology/#pod-visit-service-ip","text":"Kube-OVN configures load balancing for each Kubernetes Service on a logical switch on each subnet. When a Pod accesses other Pods by accessing the Service IP, a network packet is constructed with the Service IP as the destination address and the MAC address of the gateway as the destination MAC address. After the network packet enters the logical switch, load balancing will intercept and DNAT the network packet to modify the destination IP and port to the IP and port of one of the Endpoint corresponding to the Service. Since the logical switch does not modify the Layer 2 destination MAC address of the network packet, the network packet will still be delivered to the physic gateway after entering the physic switch, and the physic gateway will be required to forward the network packet.","title":"Pod visit Service IP"},{"location":"en/reference/underlay-topology/#service-backend-is-the-same-node-and-same-subnet-pod","text":"","title":"Service Backend is the Same Node and Same Subnet Pod"},{"location":"en/reference/underlay-topology/#service-backend-is-the-same-node-and-different-subnets-pod","text":"\u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Service Backend is the Same Node and Different Subnets Pod"},{"location":"en/start/one-step-install/","text":"One-Click Installation \u00b6 Kube-OVN provides a one-click installation script to help you quickly install a highly available, production-ready Kube-OVN container network with Overlay networking by default. Helm Chart installation is supported since Kube-OVN v1.12.0, and the default deployment is Overlay networking. If you need Underlay/Vlan networking as the default container network\uff0cplease read Underlay Installation Before installation please read Prerequisites first to make sure the environment is ready. Script Installation \u00b6 Download the installation script \u00b6 We recommend using the stable release version for production environments, please use the following command to download: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh If you are interested in the latest features of the master branch, please use the following command to download: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh Modify Configuration Options \u00b6 Open the script using the editor and change the following variables to the expected: REGISTRY = \"kubeovn\" # Image Repo VERSION = \"v1.12.0\" # Image Tag POD_CIDR = \"10.16.0.0/16\" # Default subnet CIDR don't overlay with SVC/NODE/JOIN CIDR SVC_CIDR = \"10.96.0.0/12\" # Be consistent with apiserver's service-cluster-ip-range JOIN_CIDR = \"100.64.0.0/16\" # Pod/Host communication Subnet CIDR, don't overlay with SVC/NODE/POD CIDR LABEL = \"node-role.kubernetes.io/master\" # The node label to deploy OVN DB IFACE = \"\" # The name of the host NIC used by the container network, or if empty use the NIC that host Node IP in Kubernetes TUNNEL_TYPE = \"geneve\" # Tunnel protocol\uff0cavailable options: geneve, vxlan or stt. stt requires compilation of ovs kernel module You can also use regular expression to math NIC names\uff0csuch as IFACE=enp6s0f0,eth.* . Run the Script \u00b6 bash install.sh Wait Kube-OVN ready. Helm Chart Installation \u00b6 Since the installation of Kube-OVN requires setting some parameters, to install Kube-OVN using Helm, you need to follow the steps below. View the node IP address \u00b6 $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-ovn-control-plane NotReady control-plane 20h v1.26.0 172 .18.0.3 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 kube-ovn-worker NotReady <none> 20h v1.26.0 172 .18.0.2 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 Remove cluster master node taint \u00b6 $ kubectl taint node kube-ovn-control-plane node-role.kubernetes.io/control-plane:NoSchedule- node/kube-ovn-control-plane untainted This step can be skipped if you are sure that you do not need to schedule the pod at the master node. Add label to node \u00b6 $ kubectl label node -lbeta.kubernetes.io/os = linux kubernetes.io/os = linux --overwrite node/kube-ovn-control-plane not labeled node/kube-ovn-worker not labeled $ kubectl label node -lnode-role.kubernetes.io/control-plane kube-ovn/role = master --overwrite node/kube-ovn-control-plane labeled # The following labels are used for the installation of dpdk images and can be ignored in non-dpdk cases $ kubectl label node -lovn.kubernetes.io/ovs_dp_type! = userspace ovn.kubernetes.io/ovs_dp_type = kernel --overwrite node/kube-ovn-control-plane labeled node/kube-ovn-worker labeled Add Helm Repo information \u00b6 $ helm repo add kubeovn https://kubeovn.github.io/kube-ovn/ \"kubeovn\" has been added to your repositories $ helm repo list NAME URL kubeovn https://kubeovn.github.io/kube-ovn/ $ helm search repo kubeovn NAME CHART VERSION APP VERSION DESCRIPTION kubeovn/kube-ovn 0 .1.0 1 .12.0 Helm chart for Kube-OVN Run helm install to install Kube-OVN \u00b6 The Node0IP, Node1IP, and Node2IP parameters are the IP addresses of the cluster master nodes, respectively. For other parameters, you can refer to the variable definitions in the values.yaml file. # Single master node environment install $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } # Using the node information above as an example, execute the install command $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = 172 .18.0.3 NAME: kube-ovn LAST DEPLOYED: Fri Mar 31 12 :43:43 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None # Highly Available Cluster Installation $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } , ${ Node1IP } , ${ Node2IP } , --set replicaCount = 3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"One-Click Installation"},{"location":"en/start/one-step-install/#one-click-installation","text":"Kube-OVN provides a one-click installation script to help you quickly install a highly available, production-ready Kube-OVN container network with Overlay networking by default. Helm Chart installation is supported since Kube-OVN v1.12.0, and the default deployment is Overlay networking. If you need Underlay/Vlan networking as the default container network\uff0cplease read Underlay Installation Before installation please read Prerequisites first to make sure the environment is ready.","title":"One-Click Installation"},{"location":"en/start/one-step-install/#script-installation","text":"","title":"Script Installation"},{"location":"en/start/one-step-install/#download-the-installation-script","text":"We recommend using the stable release version for production environments, please use the following command to download: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh If you are interested in the latest features of the master branch, please use the following command to download: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"Download the installation script"},{"location":"en/start/one-step-install/#modify-configuration-options","text":"Open the script using the editor and change the following variables to the expected: REGISTRY = \"kubeovn\" # Image Repo VERSION = \"v1.12.0\" # Image Tag POD_CIDR = \"10.16.0.0/16\" # Default subnet CIDR don't overlay with SVC/NODE/JOIN CIDR SVC_CIDR = \"10.96.0.0/12\" # Be consistent with apiserver's service-cluster-ip-range JOIN_CIDR = \"100.64.0.0/16\" # Pod/Host communication Subnet CIDR, don't overlay with SVC/NODE/POD CIDR LABEL = \"node-role.kubernetes.io/master\" # The node label to deploy OVN DB IFACE = \"\" # The name of the host NIC used by the container network, or if empty use the NIC that host Node IP in Kubernetes TUNNEL_TYPE = \"geneve\" # Tunnel protocol\uff0cavailable options: geneve, vxlan or stt. stt requires compilation of ovs kernel module You can also use regular expression to math NIC names\uff0csuch as IFACE=enp6s0f0,eth.* .","title":"Modify Configuration Options"},{"location":"en/start/one-step-install/#run-the-script","text":"bash install.sh Wait Kube-OVN ready.","title":"Run the Script"},{"location":"en/start/one-step-install/#helm-chart-installation","text":"Since the installation of Kube-OVN requires setting some parameters, to install Kube-OVN using Helm, you need to follow the steps below.","title":"Helm Chart Installation"},{"location":"en/start/one-step-install/#view-the-node-ip-address","text":"$ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-ovn-control-plane NotReady control-plane 20h v1.26.0 172 .18.0.3 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 kube-ovn-worker NotReady <none> 20h v1.26.0 172 .18.0.2 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9","title":"View the node IP address"},{"location":"en/start/one-step-install/#remove-cluster-master-node-taint","text":"$ kubectl taint node kube-ovn-control-plane node-role.kubernetes.io/control-plane:NoSchedule- node/kube-ovn-control-plane untainted This step can be skipped if you are sure that you do not need to schedule the pod at the master node.","title":"Remove cluster master node taint"},{"location":"en/start/one-step-install/#add-label-to-node","text":"$ kubectl label node -lbeta.kubernetes.io/os = linux kubernetes.io/os = linux --overwrite node/kube-ovn-control-plane not labeled node/kube-ovn-worker not labeled $ kubectl label node -lnode-role.kubernetes.io/control-plane kube-ovn/role = master --overwrite node/kube-ovn-control-plane labeled # The following labels are used for the installation of dpdk images and can be ignored in non-dpdk cases $ kubectl label node -lovn.kubernetes.io/ovs_dp_type! = userspace ovn.kubernetes.io/ovs_dp_type = kernel --overwrite node/kube-ovn-control-plane labeled node/kube-ovn-worker labeled","title":"Add label to node"},{"location":"en/start/one-step-install/#add-helm-repo-information","text":"$ helm repo add kubeovn https://kubeovn.github.io/kube-ovn/ \"kubeovn\" has been added to your repositories $ helm repo list NAME URL kubeovn https://kubeovn.github.io/kube-ovn/ $ helm search repo kubeovn NAME CHART VERSION APP VERSION DESCRIPTION kubeovn/kube-ovn 0 .1.0 1 .12.0 Helm chart for Kube-OVN","title":"Add Helm Repo information"},{"location":"en/start/one-step-install/#run-helm-install-to-install-kube-ovn","text":"The Node0IP, Node1IP, and Node2IP parameters are the IP addresses of the cluster master nodes, respectively. For other parameters, you can refer to the variable definitions in the values.yaml file. # Single master node environment install $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } # Using the node information above as an example, execute the install command $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = 172 .18.0.3 NAME: kube-ovn LAST DEPLOYED: Fri Mar 31 12 :43:43 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None # Highly Available Cluster Installation $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } , ${ Node1IP } , ${ Node2IP } , --set replicaCount = 3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Run helm install to install Kube-OVN"},{"location":"en/start/prepare/","text":"Prerequisites \u00b6 Kube-OVN is a CNI-compliant network system that depends on the Kubernetes environment and the corresponding kernel network module for its operation. Below are the operating system and software versions tested, the environment configuration and the ports that need to be opened. Software Version \u00b6 Kubernetes >= 1.23. Docker >= 1.12.6, Containerd >= 1.3.4. OS: CentOS 7/8, Ubuntu 16.04/18.04/20.04. For other Linux distributions, please make sure geneve , openvswitch , ip_tables and iptable_nat kernel modules exist. Attention \uff1a For CentOS kernel version 3.10.0-862 bug exists in netfilter modules that lead Kube-OVN embed nat and lb failure.Please update kernel and check Floating IPs broken after kernel upgrade to Centos/RHEL 7.5 - DNAT not working . Kernel version 4.18.0-372.9.1.el8.x86_64 in Rocky Linux 8.6 has a TCP connection problem TCP connection failed in Rocky Linux 8.6 \uff0cplease update kernel to 4.18.0-372.13.1.el8_6.x86_64 or later\u3002 For kernel version 4.4, the related openvswitch module has some issues for ct\uff0cplease update kernel version or manually compile openvswitch kernel module. When building Geneve tunnel IPv6 in kernel should be enabled\uff0ccheck the kernel bootstrap options with cat /proc/cmdline .Check Geneve tunnels don't work when ipv6 is disabled for the detail bug info. Environment Setup \u00b6 Kernel should enable IPv6, if kernel bootstrap options contain ipv6.disable=1 , it should be set to 0 . kube-proxy works, Kube-OVN can visit kube-apiserver from Service ClusterIP. Make sure kubelet enabled CNI and find cni-bin and cni-conf in default directories, kubelet bootstrap options should contain --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d . Make sure no other CNI installed or has been removed\uff0ccheck if any config files still exist in /etc/cni/net.d/ . Ports Need Open \u00b6 Component Port Usage ovn-central 6641/tcp, 6642/tcp, 6643/tcp, 6644/tcp ovn-db and raft server listen ports ovs-ovn Geneve 6081/udp, STT 7471/tcp, Vxlan 4789/udp tunnel ports kube-ovn-controller 10660/tcp metrics port kube-ovn-daemon 10665/tcp metrics port kube-ovn-monitor 10661/tcp metrics port \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Prerequisites"},{"location":"en/start/prepare/#prerequisites","text":"Kube-OVN is a CNI-compliant network system that depends on the Kubernetes environment and the corresponding kernel network module for its operation. Below are the operating system and software versions tested, the environment configuration and the ports that need to be opened.","title":"Prerequisites"},{"location":"en/start/prepare/#software-version","text":"Kubernetes >= 1.23. Docker >= 1.12.6, Containerd >= 1.3.4. OS: CentOS 7/8, Ubuntu 16.04/18.04/20.04. For other Linux distributions, please make sure geneve , openvswitch , ip_tables and iptable_nat kernel modules exist. Attention \uff1a For CentOS kernel version 3.10.0-862 bug exists in netfilter modules that lead Kube-OVN embed nat and lb failure.Please update kernel and check Floating IPs broken after kernel upgrade to Centos/RHEL 7.5 - DNAT not working . Kernel version 4.18.0-372.9.1.el8.x86_64 in Rocky Linux 8.6 has a TCP connection problem TCP connection failed in Rocky Linux 8.6 \uff0cplease update kernel to 4.18.0-372.13.1.el8_6.x86_64 or later\u3002 For kernel version 4.4, the related openvswitch module has some issues for ct\uff0cplease update kernel version or manually compile openvswitch kernel module. When building Geneve tunnel IPv6 in kernel should be enabled\uff0ccheck the kernel bootstrap options with cat /proc/cmdline .Check Geneve tunnels don't work when ipv6 is disabled for the detail bug info.","title":"Software Version"},{"location":"en/start/prepare/#environment-setup","text":"Kernel should enable IPv6, if kernel bootstrap options contain ipv6.disable=1 , it should be set to 0 . kube-proxy works, Kube-OVN can visit kube-apiserver from Service ClusterIP. Make sure kubelet enabled CNI and find cni-bin and cni-conf in default directories, kubelet bootstrap options should contain --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d . Make sure no other CNI installed or has been removed\uff0ccheck if any config files still exist in /etc/cni/net.d/ .","title":"Environment Setup"},{"location":"en/start/prepare/#ports-need-open","text":"Component Port Usage ovn-central 6641/tcp, 6642/tcp, 6643/tcp, 6644/tcp ovn-db and raft server listen ports ovs-ovn Geneve 6081/udp, STT 7471/tcp, Vxlan 4789/udp tunnel ports kube-ovn-controller 10660/tcp metrics port kube-ovn-daemon 10665/tcp metrics port kube-ovn-monitor 10661/tcp metrics port \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Ports Need Open"},{"location":"en/start/sealos-install/","text":"One-Click Deployment of Kubernetes and Kube-OVN with sealos \u00b6 sealos , a distribution of Kubernetes, helps users quickly initialize a container cluster from scratch. By using sealos, users can deploy a Kubernetes cluster with Kube-OVN installed in minutes with a single command. Download sealos \u00b6 AMD64 ARM64 wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_amd64.tar.gz \\ && tar zxvf sealos_4.0.0_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_arm64.tar.gz \\ && tar zxvf sealos_4.0.0_linux_arm64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin Deploy Kubernetes and Kube-OVN \u00b6 ```bash sealos run labring/kubernetes:v1.24.3 labring/kube-ovn:v1.10.5 \\ --masters [masters ips seperated by comma] \\ --nodes [nodes ips seperated by comma] -p [your-ssh-passwd] ``` Wait to finish \u00b6 ```bash [Step 6/6] Finish ,,,, ,::, ,,::,,,, ,,,,,::::::::::::,,,,, ,,,::::::::::::::::::::::,,, ,,::::::::::::::::::::::::::::,, ,,::::::::::::::::::::::::::::::::,, ,::::::::::::::::::::::::::::::::::::, ,:::::::::::::,, ,,:::::,,,::::::::::, ,,:::::::::::::, ,::, ,:::::::::, ,:::::::::::::, :x, ,:: :, ,:::::::::, ,:::::::::::::::, ,,, ,::, ,, ,::::::::::, ,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x ,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x ,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx ,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx ,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,:: ,::::, ,::::, ,::::, ,::::, ,::::, ,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,, ,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::, ,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::, ,::::, ,::::, ,::::, ,::::::::::::,,,,, ,,,, ,::::, ,,,, ,,,::::,,,, ,::::, ,,::, Thanks for choosing Kube-OVN! For more advanced features, please read https://github.com/kubeovn/kube-ovn#documents If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose 2022-08-10T16:31:34 info succeeded in creating a new cluster, enjoy it! 2022-08-10T16:31:34 info ___ ___ ___ ___ ___ ___ /\\ \\ /\\ \\ /\\ \\ /\\__\\ /\\ \\ /\\ \\ /::\\ \\ /::\\ \\ /::\\ \\ /:/ / /::\\ \\ /::\\ \\ /:/\\ \\ \\ /:/\\:\\ \\ /:/\\:\\ \\ /:/ / /:/\\:\\ \\ /:/\\ \\ \\ _\\:\\~\\ \\ \\ /::\\~\\:\\ \\ /::\\~\\:\\ \\ /:/ / /:/ \\:\\ \\ _\\:\\~\\ \\ \\ /\\ \\:\\ \\ \\__\\ /:/\\:\\ \\:\\__\\ /:/\\:\\ \\:\\__\\ /:/__/ /:/__/ \\:\\__\\ /\\ \\:\\ \\ \\__\\ \\:\\ \\:\\ \\/__/ \\:\\~\\:\\ \\/__/ \\/__\\:\\/:/ / \\:\\ \\ \\:\\ \\ /:/ / \\:\\ \\:\\ \\/__/ \\:\\ \\:\\__\\ \\:\\ \\:\\__\\ \\::/ / \\:\\ \\ \\:\\ /:/ / \\:\\ \\:\\__\\ \\:\\/:/ / \\:\\ \\/__/ /:/ / \\:\\ \\ \\:\\/:/ / \\:\\/:/ / \\::/ / \\:\\__\\ /:/ / \\:\\__\\ \\::/ / \\::/ / \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ Website :https://www.sealos.io/ Address :github.com/labring/sealos ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Use Sealos to Deploy Kubernetes and Kube-OVN"},{"location":"en/start/sealos-install/#one-click-deployment-of-kubernetes-and-kube-ovn-with-sealos","text":"sealos , a distribution of Kubernetes, helps users quickly initialize a container cluster from scratch. By using sealos, users can deploy a Kubernetes cluster with Kube-OVN installed in minutes with a single command.","title":"One-Click Deployment of Kubernetes and Kube-OVN with sealos"},{"location":"en/start/sealos-install/#download-sealos","text":"AMD64 ARM64 wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_amd64.tar.gz \\ && tar zxvf sealos_4.0.0_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_arm64.tar.gz \\ && tar zxvf sealos_4.0.0_linux_arm64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin","title":"Download sealos"},{"location":"en/start/sealos-install/#deploy-kubernetes-and-kube-ovn","text":"```bash sealos run labring/kubernetes:v1.24.3 labring/kube-ovn:v1.10.5 \\ --masters [masters ips seperated by comma] \\ --nodes [nodes ips seperated by comma] -p [your-ssh-passwd] ```","title":"Deploy Kubernetes and Kube-OVN"},{"location":"en/start/sealos-install/#wait-to-finish","text":"```bash [Step 6/6] Finish ,,,, ,::, ,,::,,,, ,,,,,::::::::::::,,,,, ,,,::::::::::::::::::::::,,, ,,::::::::::::::::::::::::::::,, ,,::::::::::::::::::::::::::::::::,, ,::::::::::::::::::::::::::::::::::::, ,:::::::::::::,, ,,:::::,,,::::::::::, ,,:::::::::::::, ,::, ,:::::::::, ,:::::::::::::, :x, ,:: :, ,:::::::::, ,:::::::::::::::, ,,, ,::, ,, ,::::::::::, ,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x ,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x ,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx ,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx ,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,:: ,::::, ,::::, ,::::, ,::::, ,::::, ,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,, ,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::, ,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::, ,::::, ,::::, ,::::, ,::::::::::::,,,,, ,,,, ,::::, ,,,, ,,,::::,,,, ,::::, ,,::, Thanks for choosing Kube-OVN! For more advanced features, please read https://github.com/kubeovn/kube-ovn#documents If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose 2022-08-10T16:31:34 info succeeded in creating a new cluster, enjoy it! 2022-08-10T16:31:34 info ___ ___ ___ ___ ___ ___ /\\ \\ /\\ \\ /\\ \\ /\\__\\ /\\ \\ /\\ \\ /::\\ \\ /::\\ \\ /::\\ \\ /:/ / /::\\ \\ /::\\ \\ /:/\\ \\ \\ /:/\\:\\ \\ /:/\\:\\ \\ /:/ / /:/\\:\\ \\ /:/\\ \\ \\ _\\:\\~\\ \\ \\ /::\\~\\:\\ \\ /::\\~\\:\\ \\ /:/ / /:/ \\:\\ \\ _\\:\\~\\ \\ \\ /\\ \\:\\ \\ \\__\\ /:/\\:\\ \\:\\__\\ /:/\\:\\ \\:\\__\\ /:/__/ /:/__/ \\:\\__\\ /\\ \\:\\ \\ \\__\\ \\:\\ \\:\\ \\/__/ \\:\\~\\:\\ \\/__/ \\/__\\:\\/:/ / \\:\\ \\ \\:\\ \\ /:/ / \\:\\ \\:\\ \\/__/ \\:\\ \\:\\__\\ \\:\\ \\:\\__\\ \\::/ / \\:\\ \\ \\:\\ /:/ / \\:\\ \\:\\__\\ \\:\\/:/ / \\:\\ \\/__/ /:/ / \\:\\ \\ \\:\\/:/ / \\:\\/:/ / \\::/ / \\:\\__\\ /:/ / \\:\\__\\ \\::/ / \\::/ / \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ Website :https://www.sealos.io/ Address :github.com/labring/sealos ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Wait to finish"},{"location":"en/start/underlay/","text":"Underlay Installation \u00b6 By default, the default subnet uses Geneve to encapsulate cross-host traffic, and build an overlay network on top of the infrastructure. For the case that you want the container network to use the physical network address directly, you can set the default subnet of Kube-OVN to work in Underlay mode, which can directly assign the address resources in the physical network to the containers, achieving better performance and connectivity with the physical network. Limitation \u00b6 Since the container network in this mode uses physical network directly for L2 packet forwarding, L3 functions such as SNAT/EIP, distributed gateway/centralized gateway in Overlay mode cannot be used. VPC level isolation is also not available for underlay subnet. Comparison with Macvlan \u00b6 The Underlay mode of Kube-OVN is very similar to the Macvlan, with the following major differences in functionality and performance: Macvlan performs better in terms of throughput and latency performance metrics due to its shorter kernel path and the fact that it does not require OVS for packet processing. Kube-OVN provides arp-proxy functionality through flow tables to mitigate the risk of arp broadcast storms on large-scale networks. Since Macvlan works at the bottom of the kernel and bypasses the host netfilter, Service and NetworkPolicy functionality requires additional development. Kube-OVN provides Service and NetworkPolicy capabilities through the OVS flow table. Kube-OVN Underlay mode provides additional features such as address management, fixed IP and QoS compared to Macvlan. Environment Requirements \u00b6 In Underlay mode, the OVS will bridge a node NIC to the OVS bridge and send packets directly through that node NIC, relying on the underlying network devices for L2/L3 level forwarding capabilities. You need to configure the corresponding gateway, Vlan and security policy in the underlying network device in advance. For OpenStack VM environments, you need to turn off PortSecurity on the corresponding network port. For VMware vSwitch networks, MAC Address Changes , Forged Transmits and Promiscuous Mode Operation should be set to allow . For Hyper-V virtualization, MAC Address Spoofing should be enabled in VM nic advanced features. Public clouds, such as AWS, GCE, AliCloud, etc., do not support user-defined Mac, so they cannot support Underlay mode network. In this scenario, if you want to use Underlay, it is recommended to use the VPC-CNI provided by the corresponding public cloud vendor.. The network interface that is bridged into ovs can not be type of Linux Bridge. For management and container networks using the same NIC, Kube-OVN will transfer the NIC's Mac address, IP address, route, and MTU to the corresponding OVS Bridge to support single NIC deployment of Underlay networks. OVS Bridge name format is br-PROVIDER_NAME \uff0c PROVIDER_NAME is the name of ProviderNetwork (Default: provider). Specify Network Mode When Deploying \u00b6 This deployment mode sets the default subnet to Underlay mode, and all Pods with no subnet specified will run in the Underlay network by default. Download Script \u00b6 wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh Modify Configuration Options \u00b6 ENABLE_ARP_DETECT_IP_CONFLICT # disable vlan arp conflict detection if necessary NETWORK_TYPE # set to vlan VLAN_INTERFACE_NAME # set to the NIC that carries the Underlay traffic, e.g. eth1 VLAN_ID # The VLAN Tag need to be added\uff0cif set 0 no vlan tag will be added POD_CIDR # The Underlay network CIDR\uff0c e.g. 192.168.1.0/24 POD_GATEWAY # Underlay physic gateway address, e.g. 192.168.1.1 EXCLUDE_IPS # Exclude ranges to avoid conflicts between container network and IPs already in use on the physical network, e.g. 192.168.1.1..192.168.1.100 ENABLE_LB # If Underlay Subnet needs to visit Service set it to true EXCHANGE_LINK_NAME # If swap the names of the OVS bridge and the bridge interface under the default provider-network. Default to false. LS_DNAT_MOD_DL_DST # If DNAT translate MAC addresses to accelerate service access. Default to true. Run the Script \u00b6 bash install.sh Dynamically Create Underlay Networks via CRD \u00b6 This approach dynamically creates an Underlay subnet that Pod can use after installation. Create ProviderNetwork \u00b6 ProviderNetwork provides the abstraction of host NIC to physical network mapping, unifies the management of NICs belonging to the same network, and solves the configuration problems in complex environments with multiple NICs on the same machine, inconsistent NIC names and inconsistent corresponding Underlay networks. Create ProviderNetwork as below: apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: net1 spec: defaultInterface: eth1 customInterfaces: - interface: eth2 nodes: - node1 excludeNodes: - node2 Note: The length of the ProviderNetwork resource name must not exceed 12. defaultInterface : The default node NIC name. When the ProviderNetwork is successfully created, an OVS bridge named br-net1 (in the format br-NAME ) is created in each node (except excludeNodes) and the specified node NIC is bridged to this bridge. customInterfaces : Optionally, you can specify the NIC to be used for a specific node. excludeNodes : Optional, to specify nodes that do not bridge the NIC. Nodes in this list will be added with the net1.provider-network.ovn.kubernetes.io/exclude=true tag. Other nodes will be added with the following tags: Key Value Description net1.provider-network.ovn.kubernetes.io/ready true bridge work finished, ProviderNetwork is ready on this node net1.provider-network.ovn.kubernetes.io/interface eth1 The name of the bridged NIC in the node. net1.provider-network.ovn.kubernetes.io/mtu 1500 MTU of bridged NIC in node If an IP has been configured on the node NIC, the IP address and the route on the NIC are transferred to the corresponding OVS bridge. Create VLAN \u00b6 Vlan provides an abstraction to bind Vlan Tag and ProviderNetwork. Create a VLAN as below: apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan1 spec: id: 0 provider: net1 id : VLAN ID/Tag\uff0cKube-OVN will add this Vlan tag to traffic, if set 0, no tag is added. provider : The name of ProviderNetwork. Multiple VLAN can use a same ProviderNetwork. Create Subnet \u00b6 Bind Vlan to a Subnet as below\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 Simply specify the value of vlan as the name of the VLAN to be used. Multiple subnets can refer to the same VLAN. Create Pod \u00b6 You can create containers in the normal way, check whether the container IP is in the specified range and whether the container can interoperate with the physical network. For fixed IP requirements, please refer to Fixed Addresses Logical Gateway \u00b6 For cases where no gateway exists in the physical network, Kube-OVN supports the use of logical gateways configured in the subnet in Underlay mode. To use this feature, set spec.logicalGateway to true for the subnet: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 logicalGateway : true When this feature is turned on, the Pod does not use an external gateway, but a Logical Router created by Kube-OVN to forward cross-subnet communication. Interconnection of Underlay and Overlay Networks \u00b6 If a cluster has both Underlay and Overlay subnets, by default, Pods in the Overlay subnet can access the Pod IPs in the Underlay subnet via a gateway using NAT. From the perspective of Pods in the Underlay subnet, the addresses in the Overlay subnet are external, and require the underlying physical device to forward, but the underlying physical device does not know the addresses in the Overlay subnet and cannot forward. Therefore, Pods in the Underlay subnet cannot access Pods in the Overlay subnet directly via Pod IPs. If you need to enable communication between Underlay and Overlay networks, you need to set the u2oInterconnection of the subnet to true . In this case, Kube-OVN will use an additional Underlay IP to connect the Underlay subnet and the ovn-cluster logical router, and set the corresponding routing rules to enable communication. Unlike the logical gateway, this solution only connects the Underlay and Overlay subnets within Kube-OVN, and other traffic accessing the Internet will still be forwarded through the physical gateway. Specify logical gateway IP \u00b6 After the interworking function is enabled, an IP from the subnet will be randomly selected as the logical gateway. If you need to specify the logical gateway of the Underlay Subnet, you can specify the field u2oInterconnectionIP . Specify custom VPC for Underlay Subnet connection \u00b6 By default, the Underlay Subnet will communicate with the Overlay Subnet on the default VPC. If you want to specify to communicate with a certain VPC, after setting u2oInterconnection to true , specify the subnet.spec.vpc field as the name of the VPC. Notice \u00b6 If there are IP addresses on the host nic, the OS you are using is Ubuntu, and the networking is configured with Netplan, it's recomanded to set NetworkManager as the renderer and to set static IP addresses for the nic (disable DHCP): network : renderer : NetworkManager ethernets : eth0 : dhcp4 : no addresses : - 172.16.143.129/24 version : 2 If the host networking service is NetworkManager, Kube-OVN will remove the host nic from the managed devices (managed by NetworkManager is no) after creating ProviderNetwork: root@ubuntu:~# nmcli device status DEVICE TYPE STATE CONNECTION eth0 ethernet unmanaged netplan-eth0 If you want to change the host nic's IP/route configuration, you need to set the nic managed by NetworkManager manually: nmcli device set eth0 managed yes After setting managed to yes\uff0cKube-OVN will transfer IP and routes on the nic to the OVS bridge, and remove the nic from the managed devices again. Notice \uff1aIf the host nic's MAC is changed, Kube-OVN will not change the OVS bridge's MAC unless kube-ovn-cni is restarted. Known Issues \u00b6 When the physical network is enabled with hairpin, Pod network is abnormal \u00b6 When physical networks enable hairpin or similar behaviors, problems such as gateway check failure when creating Pods and abnormal network communication of Pods may occur. This is because the default MAC learning function of OVS bridge does not support this kind of network environment. To solve this problem, it is necessary to turn off hairpin (or modify the relevant configuration of physical network), or update the Kube-OVN version. When there are a large number of Pods, gateway check for new Pods fails \u00b6 If there are a large number of Pods running on the same node (more than 300), it may cause packet loss due to the OVS flow table resubmit times exceeding the upper limit of ARP broadcast packets. 2022-11-13T08:43:46.782Z|00222|ofproto_dpif_upcall(handler5)|WARN|Flow: arp,in_port=331,vlan_tci=0x0000,dl_src=00:00:00:25:eb:39,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.131.240,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:25:eb:39,arp_tha=ff:ff:ff:ff:ff:ff bridge(\"br-int\") ---------------- 0. No match. >>>> received packet on unknown port 331 <<<< drop Final flow: unchanged Megaflow: recirc_id=0,eth,arp,in_port=331,dl_src=00:00:00:25:eb:39 Datapath actions: drop 2022-11-13T08:44:34.077Z|00224|ofproto_dpif_xlate(handler5)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,in_port=13483,vlan_tci=0x0000,dl_src=00:00:00:59:ef:13,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.152.3,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:59:ef:13,arp_tha=ff:ff:ff:ff:ff:ff To solve this issue, modify the OVN NB option bcast_arp_req_flood to false : kubectl ko nbctl set NB_Global . options:bcast_arp_req_flood = false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Underlay Installation"},{"location":"en/start/underlay/#underlay-installation","text":"By default, the default subnet uses Geneve to encapsulate cross-host traffic, and build an overlay network on top of the infrastructure. For the case that you want the container network to use the physical network address directly, you can set the default subnet of Kube-OVN to work in Underlay mode, which can directly assign the address resources in the physical network to the containers, achieving better performance and connectivity with the physical network.","title":"Underlay Installation"},{"location":"en/start/underlay/#limitation","text":"Since the container network in this mode uses physical network directly for L2 packet forwarding, L3 functions such as SNAT/EIP, distributed gateway/centralized gateway in Overlay mode cannot be used. VPC level isolation is also not available for underlay subnet.","title":"Limitation"},{"location":"en/start/underlay/#comparison-with-macvlan","text":"The Underlay mode of Kube-OVN is very similar to the Macvlan, with the following major differences in functionality and performance: Macvlan performs better in terms of throughput and latency performance metrics due to its shorter kernel path and the fact that it does not require OVS for packet processing. Kube-OVN provides arp-proxy functionality through flow tables to mitigate the risk of arp broadcast storms on large-scale networks. Since Macvlan works at the bottom of the kernel and bypasses the host netfilter, Service and NetworkPolicy functionality requires additional development. Kube-OVN provides Service and NetworkPolicy capabilities through the OVS flow table. Kube-OVN Underlay mode provides additional features such as address management, fixed IP and QoS compared to Macvlan.","title":"Comparison with Macvlan"},{"location":"en/start/underlay/#environment-requirements","text":"In Underlay mode, the OVS will bridge a node NIC to the OVS bridge and send packets directly through that node NIC, relying on the underlying network devices for L2/L3 level forwarding capabilities. You need to configure the corresponding gateway, Vlan and security policy in the underlying network device in advance. For OpenStack VM environments, you need to turn off PortSecurity on the corresponding network port. For VMware vSwitch networks, MAC Address Changes , Forged Transmits and Promiscuous Mode Operation should be set to allow . For Hyper-V virtualization, MAC Address Spoofing should be enabled in VM nic advanced features. Public clouds, such as AWS, GCE, AliCloud, etc., do not support user-defined Mac, so they cannot support Underlay mode network. In this scenario, if you want to use Underlay, it is recommended to use the VPC-CNI provided by the corresponding public cloud vendor.. The network interface that is bridged into ovs can not be type of Linux Bridge. For management and container networks using the same NIC, Kube-OVN will transfer the NIC's Mac address, IP address, route, and MTU to the corresponding OVS Bridge to support single NIC deployment of Underlay networks. OVS Bridge name format is br-PROVIDER_NAME \uff0c PROVIDER_NAME is the name of ProviderNetwork (Default: provider).","title":"Environment Requirements"},{"location":"en/start/underlay/#specify-network-mode-when-deploying","text":"This deployment mode sets the default subnet to Underlay mode, and all Pods with no subnet specified will run in the Underlay network by default.","title":"Specify Network Mode When Deploying"},{"location":"en/start/underlay/#download-script","text":"wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"Download Script"},{"location":"en/start/underlay/#modify-configuration-options","text":"ENABLE_ARP_DETECT_IP_CONFLICT # disable vlan arp conflict detection if necessary NETWORK_TYPE # set to vlan VLAN_INTERFACE_NAME # set to the NIC that carries the Underlay traffic, e.g. eth1 VLAN_ID # The VLAN Tag need to be added\uff0cif set 0 no vlan tag will be added POD_CIDR # The Underlay network CIDR\uff0c e.g. 192.168.1.0/24 POD_GATEWAY # Underlay physic gateway address, e.g. 192.168.1.1 EXCLUDE_IPS # Exclude ranges to avoid conflicts between container network and IPs already in use on the physical network, e.g. 192.168.1.1..192.168.1.100 ENABLE_LB # If Underlay Subnet needs to visit Service set it to true EXCHANGE_LINK_NAME # If swap the names of the OVS bridge and the bridge interface under the default provider-network. Default to false. LS_DNAT_MOD_DL_DST # If DNAT translate MAC addresses to accelerate service access. Default to true.","title":"Modify Configuration Options"},{"location":"en/start/underlay/#run-the-script","text":"bash install.sh","title":"Run the Script"},{"location":"en/start/underlay/#dynamically-create-underlay-networks-via-crd","text":"This approach dynamically creates an Underlay subnet that Pod can use after installation.","title":"Dynamically Create Underlay Networks via CRD"},{"location":"en/start/underlay/#create-providernetwork","text":"ProviderNetwork provides the abstraction of host NIC to physical network mapping, unifies the management of NICs belonging to the same network, and solves the configuration problems in complex environments with multiple NICs on the same machine, inconsistent NIC names and inconsistent corresponding Underlay networks. Create ProviderNetwork as below: apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: net1 spec: defaultInterface: eth1 customInterfaces: - interface: eth2 nodes: - node1 excludeNodes: - node2 Note: The length of the ProviderNetwork resource name must not exceed 12. defaultInterface : The default node NIC name. When the ProviderNetwork is successfully created, an OVS bridge named br-net1 (in the format br-NAME ) is created in each node (except excludeNodes) and the specified node NIC is bridged to this bridge. customInterfaces : Optionally, you can specify the NIC to be used for a specific node. excludeNodes : Optional, to specify nodes that do not bridge the NIC. Nodes in this list will be added with the net1.provider-network.ovn.kubernetes.io/exclude=true tag. Other nodes will be added with the following tags: Key Value Description net1.provider-network.ovn.kubernetes.io/ready true bridge work finished, ProviderNetwork is ready on this node net1.provider-network.ovn.kubernetes.io/interface eth1 The name of the bridged NIC in the node. net1.provider-network.ovn.kubernetes.io/mtu 1500 MTU of bridged NIC in node If an IP has been configured on the node NIC, the IP address and the route on the NIC are transferred to the corresponding OVS bridge.","title":"Create ProviderNetwork"},{"location":"en/start/underlay/#create-vlan","text":"Vlan provides an abstraction to bind Vlan Tag and ProviderNetwork. Create a VLAN as below: apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan1 spec: id: 0 provider: net1 id : VLAN ID/Tag\uff0cKube-OVN will add this Vlan tag to traffic, if set 0, no tag is added. provider : The name of ProviderNetwork. Multiple VLAN can use a same ProviderNetwork.","title":"Create VLAN"},{"location":"en/start/underlay/#create-subnet","text":"Bind Vlan to a Subnet as below\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 Simply specify the value of vlan as the name of the VLAN to be used. Multiple subnets can refer to the same VLAN.","title":"Create Subnet"},{"location":"en/start/underlay/#create-pod","text":"You can create containers in the normal way, check whether the container IP is in the specified range and whether the container can interoperate with the physical network. For fixed IP requirements, please refer to Fixed Addresses","title":"Create Pod"},{"location":"en/start/underlay/#logical-gateway","text":"For cases where no gateway exists in the physical network, Kube-OVN supports the use of logical gateways configured in the subnet in Underlay mode. To use this feature, set spec.logicalGateway to true for the subnet: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 logicalGateway : true When this feature is turned on, the Pod does not use an external gateway, but a Logical Router created by Kube-OVN to forward cross-subnet communication.","title":"Logical Gateway"},{"location":"en/start/underlay/#interconnection-of-underlay-and-overlay-networks","text":"If a cluster has both Underlay and Overlay subnets, by default, Pods in the Overlay subnet can access the Pod IPs in the Underlay subnet via a gateway using NAT. From the perspective of Pods in the Underlay subnet, the addresses in the Overlay subnet are external, and require the underlying physical device to forward, but the underlying physical device does not know the addresses in the Overlay subnet and cannot forward. Therefore, Pods in the Underlay subnet cannot access Pods in the Overlay subnet directly via Pod IPs. If you need to enable communication between Underlay and Overlay networks, you need to set the u2oInterconnection of the subnet to true . In this case, Kube-OVN will use an additional Underlay IP to connect the Underlay subnet and the ovn-cluster logical router, and set the corresponding routing rules to enable communication. Unlike the logical gateway, this solution only connects the Underlay and Overlay subnets within Kube-OVN, and other traffic accessing the Internet will still be forwarded through the physical gateway.","title":"Interconnection of Underlay and Overlay Networks"},{"location":"en/start/underlay/#specify-logical-gateway-ip","text":"After the interworking function is enabled, an IP from the subnet will be randomly selected as the logical gateway. If you need to specify the logical gateway of the Underlay Subnet, you can specify the field u2oInterconnectionIP .","title":"Specify logical gateway IP"},{"location":"en/start/underlay/#specify-custom-vpc-for-underlay-subnet-connection","text":"By default, the Underlay Subnet will communicate with the Overlay Subnet on the default VPC. If you want to specify to communicate with a certain VPC, after setting u2oInterconnection to true , specify the subnet.spec.vpc field as the name of the VPC.","title":"Specify custom VPC for Underlay Subnet connection"},{"location":"en/start/underlay/#notice","text":"If there are IP addresses on the host nic, the OS you are using is Ubuntu, and the networking is configured with Netplan, it's recomanded to set NetworkManager as the renderer and to set static IP addresses for the nic (disable DHCP): network : renderer : NetworkManager ethernets : eth0 : dhcp4 : no addresses : - 172.16.143.129/24 version : 2 If the host networking service is NetworkManager, Kube-OVN will remove the host nic from the managed devices (managed by NetworkManager is no) after creating ProviderNetwork: root@ubuntu:~# nmcli device status DEVICE TYPE STATE CONNECTION eth0 ethernet unmanaged netplan-eth0 If you want to change the host nic's IP/route configuration, you need to set the nic managed by NetworkManager manually: nmcli device set eth0 managed yes After setting managed to yes\uff0cKube-OVN will transfer IP and routes on the nic to the OVS bridge, and remove the nic from the managed devices again. Notice \uff1aIf the host nic's MAC is changed, Kube-OVN will not change the OVS bridge's MAC unless kube-ovn-cni is restarted.","title":"Notice"},{"location":"en/start/underlay/#known-issues","text":"","title":"Known Issues"},{"location":"en/start/underlay/#when-the-physical-network-is-enabled-with-hairpin-pod-network-is-abnormal","text":"When physical networks enable hairpin or similar behaviors, problems such as gateway check failure when creating Pods and abnormal network communication of Pods may occur. This is because the default MAC learning function of OVS bridge does not support this kind of network environment. To solve this problem, it is necessary to turn off hairpin (or modify the relevant configuration of physical network), or update the Kube-OVN version.","title":"When the physical network is enabled with hairpin, Pod network is abnormal"},{"location":"en/start/underlay/#when-there-are-a-large-number-of-pods-gateway-check-for-new-pods-fails","text":"If there are a large number of Pods running on the same node (more than 300), it may cause packet loss due to the OVS flow table resubmit times exceeding the upper limit of ARP broadcast packets. 2022-11-13T08:43:46.782Z|00222|ofproto_dpif_upcall(handler5)|WARN|Flow: arp,in_port=331,vlan_tci=0x0000,dl_src=00:00:00:25:eb:39,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.131.240,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:25:eb:39,arp_tha=ff:ff:ff:ff:ff:ff bridge(\"br-int\") ---------------- 0. No match. >>>> received packet on unknown port 331 <<<< drop Final flow: unchanged Megaflow: recirc_id=0,eth,arp,in_port=331,dl_src=00:00:00:25:eb:39 Datapath actions: drop 2022-11-13T08:44:34.077Z|00224|ofproto_dpif_xlate(handler5)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,in_port=13483,vlan_tci=0x0000,dl_src=00:00:00:59:ef:13,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.152.3,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:59:ef:13,arp_tha=ff:ff:ff:ff:ff:ff To solve this issue, modify the OVN NB option bcast_arp_req_flood to false : kubectl ko nbctl set NB_Global . options:bcast_arp_req_flood = false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"When there are a large number of Pods, gateway check for new Pods fails"},{"location":"en/start/uninstall/","text":"Uninstall \u00b6 If you need to remove the Kube-OVN and replace it with another network plugin, please follow the steps below to remove all the corresponding Kube-OVN component and OVS configuration to avoid interference with other network plugins. Feel free to contact us with an Issue to give us feedback on why you don't use Kube-OVN to help us improve it. Delete Resource in Kubernetes \u00b6 Download and run the script below to delete resource created in Kubernetes: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/cleanup.sh bash cleanup.sh Cleanup Config and Logs on Every Node \u00b6 Run the following commands on each node to clean up the configuration retained by ovsdb and openvswitch: rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn rm -fr /var/log/kube-ovn Reboot Node \u00b6 Reboot the machine to ensure that the corresponding NIC information and iptable/ipset rules are cleared to avoid the interference with other network plugins: reboot \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Uninstall"},{"location":"en/start/uninstall/#uninstall","text":"If you need to remove the Kube-OVN and replace it with another network plugin, please follow the steps below to remove all the corresponding Kube-OVN component and OVS configuration to avoid interference with other network plugins. Feel free to contact us with an Issue to give us feedback on why you don't use Kube-OVN to help us improve it.","title":"Uninstall"},{"location":"en/start/uninstall/#delete-resource-in-kubernetes","text":"Download and run the script below to delete resource created in Kubernetes: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/cleanup.sh bash cleanup.sh","title":"Delete Resource in Kubernetes"},{"location":"en/start/uninstall/#cleanup-config-and-logs-on-every-node","text":"Run the following commands on each node to clean up the configuration retained by ovsdb and openvswitch: rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn rm -fr /var/log/kube-ovn","title":"Cleanup Config and Logs on Every Node"},{"location":"en/start/uninstall/#reboot-node","text":"Reboot the machine to ensure that the corresponding NIC information and iptable/ipset rules are cleared to avoid the interference with other network plugins: reboot \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Reboot Node"}]}
\ No newline at end of file
+{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\u200b\\-]"},"docs":[{"location":"","text":"Kube-OVN \u00b6 Kube-OVN \u662f\u4e00\u6b3e CNCF \u65d7\u4e0b\u7684\u4f01\u4e1a\u7ea7\u4e91\u539f\u751f\u7f51\u7edc\u7f16\u6392\u7cfb\u7edf\uff0c\u5c06 SDN \u7684\u80fd\u529b\u548c\u4e91\u539f\u751f\u7ed3\u5408\uff0c \u63d0\u4f9b\u4e30\u5bcc\u7684\u529f\u80fd\uff0c\u6781\u81f4\u7684\u6027\u80fd\u4ee5\u53ca\u826f\u597d\u7684\u53ef\u8fd0\u7ef4\u6027\u3002 \u4e30\u5bcc\u7684\u529f\u80fd\uff1a \u5982\u679c\u4f60\u6000\u5ff5 SDN \u9886\u57df\u4e30\u5bcc\u7684\u7f51\u7edc\u80fd\u529b\u5374\u5728\u4e91\u539f\u751f\u9886\u57df\u82e6\u82e6\u8ffd\u5bfb\u800c\u4e0d\u5f97\uff0c\u90a3\u4e48 Kube-OVN \u5c06\u662f\u4f60\u7684\u6700\u4f73\u9009\u62e9\u3002 \u501f\u52a9 OVS/OVN \u5728 SDN \u9886\u57df\u6210\u719f\u7684\u80fd\u529b\uff0cKube-OVN \u5c06\u7f51\u7edc\u865a\u62df\u5316\u7684\u4e30\u5bcc\u529f\u80fd\u5e26\u5165\u4e91\u539f\u751f\u9886\u57df\u3002\u76ee\u524d\u5df2\u652f\u6301 \u5b50\u7f51\u7ba1\u7406 \uff0c \u9759\u6001 IP \u5206\u914d \uff0c \u5206\u5e03\u5f0f/\u96c6\u4e2d\u5f0f\u7f51\u5173 \uff0c Underlay/Overlay \u6df7\u5408\u7f51\u7edc \uff0c VPC \u591a\u79df\u6237\u7f51\u7edc \uff0c \u8de8\u96c6\u7fa4\u4e92\u8054\u7f51\u7edc \uff0c QoS \u7ba1\u7406 \uff0c \u591a\u7f51\u5361\u7ba1\u7406 \uff0c ACL \u7f51\u7edc\u63a7\u5236 \uff0c \u6d41\u91cf\u955c\u50cf \uff0cARM \u652f\u6301\uff0c Windows \u652f\u6301 \u7b49\u8bf8\u591a\u529f\u80fd\u3002 \u6781\u81f4\u7684\u6027\u80fd\uff1a \u5982\u679c\u4f60\u62c5\u5fc3\u5bb9\u5668\u7f51\u7edc\u4f1a\u5e26\u6765\u989d\u5916\u7684\u6027\u80fd\u635f\u8017\uff0c\u90a3\u4e48\u6765\u770b\u4e00\u4e0b Kube-OVN \u662f\u5982\u4f55\u6781\u81f4\u7684 \u4f18\u5316\u6027\u80fd \u3002 \u5728\u6570\u636e\u5e73\u9762\uff0c\u901a\u8fc7\u4e00\u7cfb\u5217\u5bf9\u6d41\u8868\u548c\u5185\u6838\u7684\u7cbe\u5fc3\u4f18\u5316\uff0c\u5e76\u501f\u52a9 eBPF \u3001 DPDK \u3001 \u667a\u80fd\u7f51\u5361\u5378\u8f7d \u7b49\u65b0\u5174\u6280\u672f\uff0c Kube-OVN \u53ef\u4ee5\u5728\u5ef6\u8fdf\u548c\u541e\u5410\u91cf\u7b49\u65b9\u9762\u7684\u6307\u6807\u8fbe\u5230\u8fd1\u4f3c\u6216\u8d85\u51fa\u5bbf\u4e3b\u673a\u7f51\u7edc\u6027\u80fd\u7684\u6c34\u5e73\u3002\u5728\u63a7\u5236\u5e73\u9762\uff0c\u901a\u8fc7\u5bf9 OVN \u4e0a\u6e38\u6d41\u8868\u7684\u88c1\u526a \uff0c \u5404\u79cd\u7f13\u5b58\u6280\u672f\u7684\u4f7f\u7528\u548c\u8c03\u4f18\uff0cKube-OVN \u53ef\u4ee5\u652f\u6301\u5927\u89c4\u6a21\u4e0a\u5343\u8282\u70b9\u548c\u4e0a\u4e07 Pod \u7684\u96c6\u7fa4\u3002 \u6b64\u5916 Kube-OVN \u8fd8\u5728\u4e0d\u65ad\u4f18\u5316 CPU \u548c\u5185\u5b58\u7b49\u8d44\u6e90\u7684\u4f7f\u7528\u91cf\uff0c\u4ee5\u9002\u5e94\u8fb9\u7f18\u7b49\u8d44\u6e90\u6709\u9650\u573a\u666f\u3002 \u826f\u597d\u7684\u53ef\u8fd0\u7ef4\u6027\uff1a \u5982\u679c\u4f60\u5bf9\u5bb9\u5668\u7f51\u7edc\u7684\u8fd0\u7ef4\u5fc3\u5b58\u5fe7\u8651\uff0cKube-OVN \u5185\u7f6e\u4e86\u5927\u91cf\u7684\u5de5\u5177\u6765\u5e2e\u52a9\u4f60\u7b80\u5316\u8fd0\u7ef4\u64cd\u4f5c\u3002 Kube-OVN \u63d0\u4f9b\u4e86 \u4e00\u952e\u5b89\u88c5\u811a\u672c \uff0c\u5e2e\u52a9\u7528\u6237\u8fc5\u901f\u642d\u5efa\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002\u540c\u65f6\u5185\u7f6e\u7684\u4e30\u5bcc\u7684 \u76d1\u63a7\u6307\u6807 \u548c Grafana \u9762\u677f \uff0c \u53ef\u5e2e\u52a9\u7528\u6237\u5efa\u7acb\u5b8c\u5584\u7684\u76d1\u63a7\u4f53\u7cfb\u3002\u5f3a\u5927\u7684 \u547d\u4ee4\u884c\u5de5\u5177 \u53ef\u4ee5\u7b80\u5316\u7528\u6237\u7684\u65e5\u5e38\u8fd0\u7ef4\u64cd\u4f5c\u3002\u901a\u8fc7 \u548c Cilium \u7ed3\u5408 \uff0c\u5229\u7528 eBPF \u80fd\u529b\u7528\u6237\u53ef\u4ee5 \u589e\u5f3a\u5bf9\u7f51\u7edc\u7684\u53ef\u89c2\u6d4b\u6027\u3002 \u6b64\u5916 \u6d41\u91cf\u955c\u50cf \u7684\u80fd\u529b\u53ef\u4ee5\u65b9\u4fbf\u7528\u6237\u81ea\u5b9a\u4e49\u6d41\u91cf\u76d1\u63a7\uff0c\u5e76\u548c\u4f20\u7edf\u7684 NPM \u7cfb\u7edf\u5bf9\u63a5\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4e3b\u9875"},{"location":"#kube-ovn","text":"Kube-OVN \u662f\u4e00\u6b3e CNCF \u65d7\u4e0b\u7684\u4f01\u4e1a\u7ea7\u4e91\u539f\u751f\u7f51\u7edc\u7f16\u6392\u7cfb\u7edf\uff0c\u5c06 SDN \u7684\u80fd\u529b\u548c\u4e91\u539f\u751f\u7ed3\u5408\uff0c \u63d0\u4f9b\u4e30\u5bcc\u7684\u529f\u80fd\uff0c\u6781\u81f4\u7684\u6027\u80fd\u4ee5\u53ca\u826f\u597d\u7684\u53ef\u8fd0\u7ef4\u6027\u3002 \u4e30\u5bcc\u7684\u529f\u80fd\uff1a \u5982\u679c\u4f60\u6000\u5ff5 SDN \u9886\u57df\u4e30\u5bcc\u7684\u7f51\u7edc\u80fd\u529b\u5374\u5728\u4e91\u539f\u751f\u9886\u57df\u82e6\u82e6\u8ffd\u5bfb\u800c\u4e0d\u5f97\uff0c\u90a3\u4e48 Kube-OVN \u5c06\u662f\u4f60\u7684\u6700\u4f73\u9009\u62e9\u3002 \u501f\u52a9 OVS/OVN \u5728 SDN \u9886\u57df\u6210\u719f\u7684\u80fd\u529b\uff0cKube-OVN \u5c06\u7f51\u7edc\u865a\u62df\u5316\u7684\u4e30\u5bcc\u529f\u80fd\u5e26\u5165\u4e91\u539f\u751f\u9886\u57df\u3002\u76ee\u524d\u5df2\u652f\u6301 \u5b50\u7f51\u7ba1\u7406 \uff0c \u9759\u6001 IP \u5206\u914d \uff0c \u5206\u5e03\u5f0f/\u96c6\u4e2d\u5f0f\u7f51\u5173 \uff0c Underlay/Overlay \u6df7\u5408\u7f51\u7edc \uff0c VPC \u591a\u79df\u6237\u7f51\u7edc \uff0c \u8de8\u96c6\u7fa4\u4e92\u8054\u7f51\u7edc \uff0c QoS \u7ba1\u7406 \uff0c \u591a\u7f51\u5361\u7ba1\u7406 \uff0c ACL \u7f51\u7edc\u63a7\u5236 \uff0c \u6d41\u91cf\u955c\u50cf \uff0cARM \u652f\u6301\uff0c Windows \u652f\u6301 \u7b49\u8bf8\u591a\u529f\u80fd\u3002 \u6781\u81f4\u7684\u6027\u80fd\uff1a \u5982\u679c\u4f60\u62c5\u5fc3\u5bb9\u5668\u7f51\u7edc\u4f1a\u5e26\u6765\u989d\u5916\u7684\u6027\u80fd\u635f\u8017\uff0c\u90a3\u4e48\u6765\u770b\u4e00\u4e0b Kube-OVN \u662f\u5982\u4f55\u6781\u81f4\u7684 \u4f18\u5316\u6027\u80fd \u3002 \u5728\u6570\u636e\u5e73\u9762\uff0c\u901a\u8fc7\u4e00\u7cfb\u5217\u5bf9\u6d41\u8868\u548c\u5185\u6838\u7684\u7cbe\u5fc3\u4f18\u5316\uff0c\u5e76\u501f\u52a9 eBPF \u3001 DPDK \u3001 \u667a\u80fd\u7f51\u5361\u5378\u8f7d \u7b49\u65b0\u5174\u6280\u672f\uff0c Kube-OVN \u53ef\u4ee5\u5728\u5ef6\u8fdf\u548c\u541e\u5410\u91cf\u7b49\u65b9\u9762\u7684\u6307\u6807\u8fbe\u5230\u8fd1\u4f3c\u6216\u8d85\u51fa\u5bbf\u4e3b\u673a\u7f51\u7edc\u6027\u80fd\u7684\u6c34\u5e73\u3002\u5728\u63a7\u5236\u5e73\u9762\uff0c\u901a\u8fc7\u5bf9 OVN \u4e0a\u6e38\u6d41\u8868\u7684\u88c1\u526a \uff0c \u5404\u79cd\u7f13\u5b58\u6280\u672f\u7684\u4f7f\u7528\u548c\u8c03\u4f18\uff0cKube-OVN \u53ef\u4ee5\u652f\u6301\u5927\u89c4\u6a21\u4e0a\u5343\u8282\u70b9\u548c\u4e0a\u4e07 Pod \u7684\u96c6\u7fa4\u3002 \u6b64\u5916 Kube-OVN \u8fd8\u5728\u4e0d\u65ad\u4f18\u5316 CPU \u548c\u5185\u5b58\u7b49\u8d44\u6e90\u7684\u4f7f\u7528\u91cf\uff0c\u4ee5\u9002\u5e94\u8fb9\u7f18\u7b49\u8d44\u6e90\u6709\u9650\u573a\u666f\u3002 \u826f\u597d\u7684\u53ef\u8fd0\u7ef4\u6027\uff1a \u5982\u679c\u4f60\u5bf9\u5bb9\u5668\u7f51\u7edc\u7684\u8fd0\u7ef4\u5fc3\u5b58\u5fe7\u8651\uff0cKube-OVN \u5185\u7f6e\u4e86\u5927\u91cf\u7684\u5de5\u5177\u6765\u5e2e\u52a9\u4f60\u7b80\u5316\u8fd0\u7ef4\u64cd\u4f5c\u3002 Kube-OVN \u63d0\u4f9b\u4e86 \u4e00\u952e\u5b89\u88c5\u811a\u672c \uff0c\u5e2e\u52a9\u7528\u6237\u8fc5\u901f\u642d\u5efa\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002\u540c\u65f6\u5185\u7f6e\u7684\u4e30\u5bcc\u7684 \u76d1\u63a7\u6307\u6807 \u548c Grafana \u9762\u677f \uff0c \u53ef\u5e2e\u52a9\u7528\u6237\u5efa\u7acb\u5b8c\u5584\u7684\u76d1\u63a7\u4f53\u7cfb\u3002\u5f3a\u5927\u7684 \u547d\u4ee4\u884c\u5de5\u5177 \u53ef\u4ee5\u7b80\u5316\u7528\u6237\u7684\u65e5\u5e38\u8fd0\u7ef4\u64cd\u4f5c\u3002\u901a\u8fc7 \u548c Cilium \u7ed3\u5408 \uff0c\u5229\u7528 eBPF \u80fd\u529b\u7528\u6237\u53ef\u4ee5 \u589e\u5f3a\u5bf9\u7f51\u7edc\u7684\u53ef\u89c2\u6d4b\u6027\u3002 \u6b64\u5916 \u6d41\u91cf\u955c\u50cf \u7684\u80fd\u529b\u53ef\u4ee5\u65b9\u4fbf\u7528\u6237\u81ea\u5b9a\u4e49\u6d41\u91cf\u76d1\u63a7\uff0c\u5e76\u548c\u4f20\u7edf\u7684 NPM \u7cfb\u7edf\u5bf9\u63a5\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN"},{"location":"contact/","text":"\u8054\u7cfb\u65b9\u5f0f \u00b6 \u5173\u6ce8\u516c\u4f17\u53f7\u83b7\u5f97\u66f4\u591a\u6700\u65b0\u4fe1\u606f\uff0c\u8bf7\u626b\u63cf\u4e0b\u65b9\u4e8c\u7ef4\u7801: \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8054\u7cfb\u65b9\u5f0f"},{"location":"contact/#_1","text":"\u5173\u6ce8\u516c\u4f17\u53f7\u83b7\u5f97\u66f4\u591a\u6700\u65b0\u4fe1\u606f\uff0c\u8bf7\u626b\u63cf\u4e0b\u65b9\u4e8c\u7ef4\u7801: \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8054\u7cfb\u65b9\u5f0f"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/","text":"\u4f7f\u7528 eBPF \u52a0\u901f\u8282\u70b9\u5185 TCP \u901a\u4fe1 \u00b6 \u5728\u4e00\u4e9b\u8fb9\u7f18\u548c 5G \u7684\u573a\u666f\u4e0b\uff0c\u540c\u8282\u70b9\u5185\u7684 Pod \u4e4b\u95f4\u4f1a\u8fdb\u884c\u5927\u91cf\u7684 TCP \u901a\u4fe1\uff0c\u901a\u8fc7\u4f7f\u7528 Intel \u5f00\u6e90\u7684 istio-tcpip-bypass \u9879\u76ee\uff0cPod \u53ef\u4ee5\u501f\u52a9 eBPF \u7684\u80fd\u529b\u7ed5\u8fc7\u4e3b\u673a\u7684 TCP/IP \u534f\u8bae\u6808\uff0c\u76f4\u63a5\u8fdb\u884c socket \u901a\u4fe1\uff0c\u4ece\u800c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u63d0\u5347\u541e\u5410\u91cf\u3002 \u57fa\u672c\u539f\u7406 \u00b6 \u5728\u5f53\u524d\u7684\u5b9e\u73b0\u4e0b\uff0c\u540c\u4e3b\u673a\u7684\u4e24\u4e2a Pod \u8fdb\u884c TCP \u8fdb\u884c\u901a\u4fe1\u9700\u8981\u7ecf\u8fc7\u5927\u91cf\u7684\u7f51\u7edc\u6808\uff0c\u5305\u62ec TCP/IP, netfilter\uff0cOVS \u7b49\u5982\u4e0b\u56fe\u6240\u793a\uff1a istio-tcpip-bypass \u63d2\u4ef6\u53ef\u4ee5\u81ea\u52a8\u5206\u6790\u5e76\u8bc6\u522b\u51fa\u540c\u4e3b\u673a\u5185\u7684 TCP \u901a\u4fe1\uff0c\u5e76\u7ed5\u8fc7\u590d\u6742\u7684\u5185\u6838\u6808\u4ece\u800c\u53ef\u4ee5\u76f4\u63a5\u8fdb\u884c socket \u95f4\u7684\u6570\u636e\u4f20\u8f93\uff0c \u6765\u964d\u4f4e\u7f51\u7edc\u6808\u5904\u7406\u5f00\u9500\uff0c\u5982\u4e0b\u56fe\u6240\u793a\uff1a \u7531\u4e8e\u8be5\u7ec4\u4ef6\u53ef\u4ee5\u81ea\u52a8\u8bc6\u522b\u540c\u4e3b\u673a\u5185\u7684 TCP \u901a\u4fe1\uff0c\u5e76\u8fdb\u884c\u4f18\u5316\u3002\u5728\u57fa\u4e8e\u4ee3\u7406\u6a21\u5f0f\u7684 Service Mesh \u73af\u5883\u4e0b\uff0c\u8be5\u7ec4\u4ef6\u4e5f\u53ef\u4ee5\u589e\u5f3a Service Mesh \u7684\u6027\u80fd\u8868\u73b0\u3002 \u66f4\u591a\u6280\u672f\u5b9e\u73b0\u7ec6\u8282\u53ef\u4ee5\u53c2\u8003 Tanzu Service Mesh Acceleration using eBPF \u3002 \u73af\u5883\u51c6\u5907 \u00b6 eBPF \u5bf9\u5185\u6838\u7248\u672c\u6709\u4e00\u5b9a\u8981\u6c42\uff0c\u63a8\u8350\u4f7f\u7528 Ubuntu 20.04 \u548c Linux 5.4.0-74-generic \u7248\u672c\u5185\u6838\u8fdb\u884c\u5b9e\u9a8c\u3002 \u5b9e\u9a8c\u6b65\u9aa4 \u00b6 \u5728\u540c\u4e00\u4e2a\u8282\u70b9\u4e0a\u90e8\u7f72\u4e24\u4e2a\u6027\u80fd\u6d4b\u8bd5 Pod\uff0c\u82e5\u96c6\u7fa4\u5185\u5b58\u5728\u591a\u53f0\u673a\u5668\u9700\u8981\u6307\u5b9a nodeSelector \uff1a # kubectl create deployment perf --image=kubeovn/perf:dev --replicas=2 deployment.apps/perf created # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES perf-7697bc6ddf-b2cpv 1 /1 Running 0 28s 100 .64.0.3 sealos <none> <none> perf-7697bc6ddf-p2xpt 1 /1 Running 0 28s 100 .64.0.2 sealos <none> <none> \u8fdb\u5165\u5176\u4e2d\u4e00\u4e2a Pod \u5f00\u542f qperf server\uff0c\u5728\u53e6\u4e00\u4e2a Pod \u4e2d\u542f\u52a8 qperf client \u8fdb\u884c\u6027\u80fd\u6d4b\u8bd5\uff1a # kubectl exec -it perf-7697bc6ddf-b2cpv sh / # qperf # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw \u90e8\u7f72 istio-tcpip-bypass \u63d2\u4ef6\uff1a kubectl apply -f https://raw.githubusercontent.com/intel/istio-tcpip-bypass/main/bypass-tcpip-daemonset.yaml \u518d\u6b21\u8fdb\u5165 perf client \u5bb9\u5668\u8fdb\u884c\u6027\u80fd\u6d4b\u8bd5\uff1a # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw \u6d4b\u8bd5\u7ed3\u679c \u00b6 \u6839\u636e\u6d4b\u8bd5\u7ed3\u679c TCP \u5ef6\u8fdf\u5728\u4e0d\u540c\u6570\u636e\u5305\u5927\u5c0f\u7684\u60c5\u51b5\u4e0b\u4f1a\u6709 40% ~ 60% \u7684\u5ef6\u8fdf\u4e0b\u964d\uff0c\u5728\u6570\u636e\u5305\u5927\u4e8e 1024 \u5b57\u8282\u65f6\u541e\u5410\u91cf\u4f1a\u6709 40% ~ 80% \u63d0\u5347\u3002 Packet Size (byte) eBPF tcp_lat (us) Default tcp_lat (us) eBPF tcp_bw (Mb/s) Default tcp_bw(Mb/s) 1 20.2 44.5 1.36 4.27 4 20.2 48.7 5.48 16.7 16 19.6 41.6 21.7 63.5 64 18.8 41.3 96.8 201 256 19.2 36 395 539 1024 18.3 42.4 1360 846 4096 16.5 62.6 4460 2430 16384 20.2 58.8 9600 6900 \u5728\u6d4b\u8bd5\u7684\u786c\u4ef6\u73af\u5883\u4e0b\uff0c\u6570\u636e\u5305\u5c0f\u4e8e 512 \u5b57\u8282\u65f6\uff0c\u4f7f\u7528 eBPF \u4f18\u5316\u541e\u5410\u91cf\u6307\u6807\u4f1a\u4f4e\u4e8e\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7684\u541e\u5410\u91cf\u3002 \u8be5\u60c5\u51b5\u53ef\u80fd\u548c\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7f51\u5361\u5f00\u542f TCP \u805a\u5408\u4f18\u5316\u76f8\u5173\u3002\u5982\u679c\u5e94\u7528\u573a\u666f\u5bf9\u5c0f\u5305\u541e\u5410\u91cf\u654f\u611f\uff0c\u9700\u8981\u5728\u76f8\u5e94\u73af\u5883\u4e0b \u8fdb\u884c\u6d4b\u8bd5\u5224\u65ad\u662f\u5426\u5f00\u542f eBPF \u4f18\u5316\u3002\u6211\u4eec\u4e5f\u4f1a\u540e\u7eed\u5bf9 eBPF TCP \u5c0f\u5305\u573a\u666f\u7684\u541e\u5410\u91cf\u8fdb\u884c\u4f18\u5316\u3002 \u53c2\u8003\u8d44\u6599 \u00b6 istio-tcpip-bypass Deep Dive TCP/IP Bypass with eBPF in Service Mesh Tanzu Service Mesh Acceleration using eBPF \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 eBPF \u52a0\u901f\u8282\u70b9\u5185 TCP \u901a\u4fe1"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#ebpf-tcp","text":"\u5728\u4e00\u4e9b\u8fb9\u7f18\u548c 5G \u7684\u573a\u666f\u4e0b\uff0c\u540c\u8282\u70b9\u5185\u7684 Pod \u4e4b\u95f4\u4f1a\u8fdb\u884c\u5927\u91cf\u7684 TCP \u901a\u4fe1\uff0c\u901a\u8fc7\u4f7f\u7528 Intel \u5f00\u6e90\u7684 istio-tcpip-bypass \u9879\u76ee\uff0cPod \u53ef\u4ee5\u501f\u52a9 eBPF \u7684\u80fd\u529b\u7ed5\u8fc7\u4e3b\u673a\u7684 TCP/IP \u534f\u8bae\u6808\uff0c\u76f4\u63a5\u8fdb\u884c socket \u901a\u4fe1\uff0c\u4ece\u800c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u63d0\u5347\u541e\u5410\u91cf\u3002","title":"\u4f7f\u7528 eBPF \u52a0\u901f\u8282\u70b9\u5185 TCP \u901a\u4fe1"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_1","text":"\u5728\u5f53\u524d\u7684\u5b9e\u73b0\u4e0b\uff0c\u540c\u4e3b\u673a\u7684\u4e24\u4e2a Pod \u8fdb\u884c TCP \u8fdb\u884c\u901a\u4fe1\u9700\u8981\u7ecf\u8fc7\u5927\u91cf\u7684\u7f51\u7edc\u6808\uff0c\u5305\u62ec TCP/IP, netfilter\uff0cOVS \u7b49\u5982\u4e0b\u56fe\u6240\u793a\uff1a istio-tcpip-bypass \u63d2\u4ef6\u53ef\u4ee5\u81ea\u52a8\u5206\u6790\u5e76\u8bc6\u522b\u51fa\u540c\u4e3b\u673a\u5185\u7684 TCP \u901a\u4fe1\uff0c\u5e76\u7ed5\u8fc7\u590d\u6742\u7684\u5185\u6838\u6808\u4ece\u800c\u53ef\u4ee5\u76f4\u63a5\u8fdb\u884c socket \u95f4\u7684\u6570\u636e\u4f20\u8f93\uff0c \u6765\u964d\u4f4e\u7f51\u7edc\u6808\u5904\u7406\u5f00\u9500\uff0c\u5982\u4e0b\u56fe\u6240\u793a\uff1a \u7531\u4e8e\u8be5\u7ec4\u4ef6\u53ef\u4ee5\u81ea\u52a8\u8bc6\u522b\u540c\u4e3b\u673a\u5185\u7684 TCP \u901a\u4fe1\uff0c\u5e76\u8fdb\u884c\u4f18\u5316\u3002\u5728\u57fa\u4e8e\u4ee3\u7406\u6a21\u5f0f\u7684 Service Mesh \u73af\u5883\u4e0b\uff0c\u8be5\u7ec4\u4ef6\u4e5f\u53ef\u4ee5\u589e\u5f3a Service Mesh \u7684\u6027\u80fd\u8868\u73b0\u3002 \u66f4\u591a\u6280\u672f\u5b9e\u73b0\u7ec6\u8282\u53ef\u4ee5\u53c2\u8003 Tanzu Service Mesh Acceleration using eBPF \u3002","title":"\u57fa\u672c\u539f\u7406"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_2","text":"eBPF \u5bf9\u5185\u6838\u7248\u672c\u6709\u4e00\u5b9a\u8981\u6c42\uff0c\u63a8\u8350\u4f7f\u7528 Ubuntu 20.04 \u548c Linux 5.4.0-74-generic \u7248\u672c\u5185\u6838\u8fdb\u884c\u5b9e\u9a8c\u3002","title":"\u73af\u5883\u51c6\u5907"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_3","text":"\u5728\u540c\u4e00\u4e2a\u8282\u70b9\u4e0a\u90e8\u7f72\u4e24\u4e2a\u6027\u80fd\u6d4b\u8bd5 Pod\uff0c\u82e5\u96c6\u7fa4\u5185\u5b58\u5728\u591a\u53f0\u673a\u5668\u9700\u8981\u6307\u5b9a nodeSelector \uff1a # kubectl create deployment perf --image=kubeovn/perf:dev --replicas=2 deployment.apps/perf created # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES perf-7697bc6ddf-b2cpv 1 /1 Running 0 28s 100 .64.0.3 sealos <none> <none> perf-7697bc6ddf-p2xpt 1 /1 Running 0 28s 100 .64.0.2 sealos <none> <none> \u8fdb\u5165\u5176\u4e2d\u4e00\u4e2a Pod \u5f00\u542f qperf server\uff0c\u5728\u53e6\u4e00\u4e2a Pod \u4e2d\u542f\u52a8 qperf client \u8fdb\u884c\u6027\u80fd\u6d4b\u8bd5\uff1a # kubectl exec -it perf-7697bc6ddf-b2cpv sh / # qperf # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw \u90e8\u7f72 istio-tcpip-bypass \u63d2\u4ef6\uff1a kubectl apply -f https://raw.githubusercontent.com/intel/istio-tcpip-bypass/main/bypass-tcpip-daemonset.yaml \u518d\u6b21\u8fdb\u5165 perf client \u5bb9\u5668\u8fdb\u884c\u6027\u80fd\u6d4b\u8bd5\uff1a # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw","title":"\u5b9e\u9a8c\u6b65\u9aa4"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_4","text":"\u6839\u636e\u6d4b\u8bd5\u7ed3\u679c TCP \u5ef6\u8fdf\u5728\u4e0d\u540c\u6570\u636e\u5305\u5927\u5c0f\u7684\u60c5\u51b5\u4e0b\u4f1a\u6709 40% ~ 60% \u7684\u5ef6\u8fdf\u4e0b\u964d\uff0c\u5728\u6570\u636e\u5305\u5927\u4e8e 1024 \u5b57\u8282\u65f6\u541e\u5410\u91cf\u4f1a\u6709 40% ~ 80% \u63d0\u5347\u3002 Packet Size (byte) eBPF tcp_lat (us) Default tcp_lat (us) eBPF tcp_bw (Mb/s) Default tcp_bw(Mb/s) 1 20.2 44.5 1.36 4.27 4 20.2 48.7 5.48 16.7 16 19.6 41.6 21.7 63.5 64 18.8 41.3 96.8 201 256 19.2 36 395 539 1024 18.3 42.4 1360 846 4096 16.5 62.6 4460 2430 16384 20.2 58.8 9600 6900 \u5728\u6d4b\u8bd5\u7684\u786c\u4ef6\u73af\u5883\u4e0b\uff0c\u6570\u636e\u5305\u5c0f\u4e8e 512 \u5b57\u8282\u65f6\uff0c\u4f7f\u7528 eBPF \u4f18\u5316\u541e\u5410\u91cf\u6307\u6807\u4f1a\u4f4e\u4e8e\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7684\u541e\u5410\u91cf\u3002 \u8be5\u60c5\u51b5\u53ef\u80fd\u548c\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7f51\u5361\u5f00\u542f TCP \u805a\u5408\u4f18\u5316\u76f8\u5173\u3002\u5982\u679c\u5e94\u7528\u573a\u666f\u5bf9\u5c0f\u5305\u541e\u5410\u91cf\u654f\u611f\uff0c\u9700\u8981\u5728\u76f8\u5e94\u73af\u5883\u4e0b \u8fdb\u884c\u6d4b\u8bd5\u5224\u65ad\u662f\u5426\u5f00\u542f eBPF \u4f18\u5316\u3002\u6211\u4eec\u4e5f\u4f1a\u540e\u7eed\u5bf9 eBPF TCP \u5c0f\u5305\u573a\u666f\u7684\u541e\u5410\u91cf\u8fdb\u884c\u4f18\u5316\u3002","title":"\u6d4b\u8bd5\u7ed3\u679c"},{"location":"advance/accelerate-intra-node-tcp-with-ebpf/#_5","text":"istio-tcpip-bypass Deep Dive TCP/IP Bypass with eBPF in Service Mesh Tanzu Service Mesh Acceleration using eBPF \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u53c2\u8003\u8d44\u6599"},{"location":"advance/cilium-hubble-observe/","text":"Cilium \u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b \u00b6 Kube-OVN \u5f53\u524d\u5df2\u7ecf\u652f\u6301\u4e0e Cilium \u96c6\u6210\uff0c\u5177\u4f53\u64cd\u4f5c\u53ef\u4ee5\u53c2\u8003 Cilium \u96c6\u6210 \u3002 Cilium \u63d0\u4f9b\u4e86\u4e30\u5bcc\u7684\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u80fd\u529b\uff0c\u6d41\u91cf\u53ef\u89c2\u6d4b\u6027\u662f\u7531 Hubble \u63d0\u4f9b\u7684\u3002Hubble \u53ef\u4ee5\u89c2\u5bdf\u8282\u70b9\u3001\u96c6\u7fa4\u751a\u81f3\u591a\u96c6\u7fa4\u573a\u666f\u4e0b\u8de8\u96c6\u7fa4\u7684\u6d41\u91cf\u3002 \u5b89\u88c5 Hubble \u00b6 \u9ed8\u8ba4\u7684 Cilium \u96c6\u6210\u5b89\u88c5\u4e2d\uff0c\u5e76\u6ca1\u6709\u5b89\u88c5 Hubble \u76f8\u5173\u7ec4\u4ef6\uff0c\u56e0\u6b64\u8981\u652f\u6301\u6d41\u91cf\u89c2\u6d4b\uff0c\u9700\u8981\u5148\u5728\u73af\u5883\u4e0a\u8865\u5145\u5b89\u88c5 Hubble\u3002 \u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u4f7f\u7528 helm \u5b89\u88c5 Hubble\uff1a helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \u8865\u5145\u5b89\u88c5 Hubble \u4e4b\u540e\uff0c\u6267\u884c cilium status \u67e5\u770b\u7ec4\u4ef6\u72b6\u6001\uff0c\u786e\u8ba4 Hubble \u5b89\u88c5\u6210\u529f\u3002 # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: OK \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ Deployment hubble-relay Desired: 1 , Ready: 1 /1, Available: 1 /1 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment hubble-ui Desired: 1 , Ready: 1 /1, Available: 1 /1 Containers: cilium Running: 2 hubble-ui Running: 1 hubble-relay Running: 1 cilium-operator Running: 2 Cluster Pods: 16 /17 managed by Cilium Image versions hubble-relay quay.io/cilium/hubble-relay:v1.11.6@sha256:fd9034a2d04d5b973f1e8ed44f230ea195b89c37955ff32e34e5aa68f3ed675a: 1 cilium-operator quay.io/cilium/operator-generic:v1.11.6@sha256:9f6063c7bcaede801a39315ec7c166309f6a6783e98665f6693939cf1701bc17: 2 cilium quay.io/cilium/cilium:v1.11.6@sha256:f7f93c26739b6641a3fa3d76b1e1605b15989f25d06625260099e01c8243f54c: 2 hubble-ui quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1 apple@bogon cilium % \u5b89\u88c5 Hubble \u7ec4\u4ef6\u4e4b\u540e\uff0c\u9700\u8981\u5b89\u88c5\u547d\u4ee4\u884c\uff0c\u7528\u4e8e\u5728\u73af\u5883\u4e0a\u67e5\u770b\u6d41\u91cf\u4fe1\u606f\u3002 \u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u5b89\u88c5 Hubble CLI : curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/v0.10.0/hubble-linux-amd64.tar.gz sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin \u90e8\u7f72\u6d4b\u8bd5\u4e1a\u52a1 \u00b6 Cilium \u5b98\u65b9\u63d0\u4f9b\u4e86\u4e00\u4e2a\u6d41\u91cf\u6d4b\u8bd5\u7684\u90e8\u7f72\u65b9\u6848\uff0c\u53ef\u4ee5\u76f4\u63a5\u4f7f\u7528\u5b98\u65b9\u90e8\u7f72\u7684\u4e1a\u52a1\u8fdb\u884c\u6d4b\u8bd5\u3002 \u6267\u884c\u547d\u4ee4 cilium connectivity test \uff0cCilium \u4f1a\u81ea\u52a8\u521b\u5efa cilium-test \u7684 Namespace\uff0c\u540c\u65f6\u5728 cilium-test \u4e0b\u90e8\u7f72\u6d4b\u8bd5\u4e1a\u52a1\u3002 \u6b63\u5e38\u90e8\u7f72\u5b8c\u540e\uff0c\u53ef\u4ee5\u67e5\u770b cilium-test namespace \u4e0b\u7684\u8d44\u6e90\u4fe1\u606f\uff0c\u53c2\u8003\u5982\u4e0b\uff1a # kubectl get all -n cilium-test NAME READY STATUS RESTARTS AGE pod/client-7df6cfbf7b-z5t2j 1 /1 Running 0 21s pod/client2-547996d7d8-nvgxg 1 /1 Running 0 21s pod/echo-other-node-d79544ccf-hl4gg 2 /2 Running 0 21s pod/echo-same-node-5d466d5444-ml7tc 2 /2 Running 0 21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE service/echo-other-node NodePort 10 .109.58.126 <none> 8080 :32269/TCP 21s service/echo-same-node NodePort 10 .108.70.32 <none> 8080 :32490/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/client 1 /1 1 1 21s deployment.apps/client2 1 /1 1 1 21s deployment.apps/echo-other-node 1 /1 1 1 21s deployment.apps/echo-same-node 1 /1 1 1 21s NAME DESIRED CURRENT READY AGE replicaset.apps/client-7df6cfbf7b 1 1 1 21s replicaset.apps/client2-547996d7d8 1 1 1 21s replicaset.apps/echo-other-node-d79544ccf 1 1 1 21s replicaset.apps/echo-same-node-5d466d5444 1 1 1 21s \u4f7f\u7528\u547d\u4ee4\u884c\u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b \u00b6 \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u4ec5\u63d0\u4f9b\u6bcf\u4e2a\u8282\u70b9 Cilium \u4ee3\u7406\u89c2\u5bdf\u5230\u7684\u6d41\u91cf\u3002 \u53ef\u4ee5\u5728 kube-system namespace \u4e0b\u7684 Cilium \u4ee3\u7406 pod \u4e2d\u6267\u884c hubble observe \u547d\u4ee4\uff0c\u67e5\u770b\u8be5\u8282\u70b9\u4e0a\u7684\u6d41\u91cf\u4fe1\u606f\u3002 # kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cilium-d6h56 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-c7sb2 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-wj8gt 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> cilium-tq5xb 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> kube-ovn-pinger-7lgk8 1 /1 Running 0 21h 10 .16.0.19 kube-ovn-control-plane <none> <none> kube-ovn-pinger-msvcn 1 /1 Running 0 21h 10 .16.0.18 kube-ovn-worker <none> <none> # kubectl exec -it -n kube-system cilium-d6h56 -- bash root@kube-ovn-worker:/home/cilium# hubble observe --from-namespace kube-system Jul 29 03 :24:25.551: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: SYN ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.651: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.761: kube-system/kube-ovn-pinger-msvcn:52004 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn -> kube-system/kube-ovn-pinger-7lgk8 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn <- kube-system/kube-ovn-pinger-7lgk8 to-endpoint FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:80 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 -> kube-system/kube-ovn-pinger-msvcn to-endpoint FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 <- kube-system/kube-ovn-pinger-msvcn to-stack FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.979: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.3 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:26.037: kube-system/coredns-6d4b75cb6d-lbgjg:36430 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:26.282: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.2 to-stack FORWARDED ( ICMPv4 EchoRequest ) \u90e8\u7f72 Hubble Relay \u540e\uff0cHubble \u53ef\u4ee5\u63d0\u4f9b\u5b8c\u6574\u7684\u96c6\u7fa4\u8303\u56f4\u7684\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u3002 \u914d\u7f6e\u7aef\u53e3\u8f6c\u53d1 \u00b6 \u4e3a\u4e86\u80fd\u6b63\u5e38\u8bbf\u95ee Hubble API\uff0c\u9700\u8981\u521b\u5efa\u7aef\u53e3\u8f6c\u53d1\uff0c\u5c06\u672c\u5730\u8bf7\u6c42\u8f6c\u53d1\u5230 Hubble Service\u3002\u53ef\u4ee5\u6267\u884c kubectl port-forward deployment/hubble-relay -n kube-system 4245:4245 \u547d\u4ee4\uff0c\u5728\u5f53\u524d\u7ec8\u7aef\u5f00\u542f\u7aef\u53e3\u8f6c\u53d1\u3002 \u7aef\u53e3\u8f6c\u53d1\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 \u7aef\u53e3\u8f6c\u53d1 \u3002 kubectl port-forward \u547d\u4ee4\u4e0d\u4f1a\u8fd4\u56de\uff0c\u9700\u8981\u6253\u5f00\u53e6\u4e00\u4e2a\u7ec8\u7aef\u6765\u7ee7\u7eed\u6d4b\u8bd5\u3002 \u914d\u7f6e\u5b8c\u7aef\u53e3\u8f6c\u53d1\u4e4b\u540e\uff0c\u5728\u7ec8\u7aef\u6267\u884c hubble status \u547d\u4ee4\uff0c\u5982\u679c\u6709\u7c7b\u4f3c\u5982\u4e0b\u8f93\u51fa\uff0c\u5219\u7aef\u53e3\u8f6c\u53d1\u914d\u7f6e\u6b63\u786e\uff0c\u53ef\u4ee5\u4f7f\u7528\u547d\u4ee4\u884c\u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b\u3002 # hubble status Healthcheck ( via localhost:4245 ) : Ok Current/Max Flows: 8 ,190/8,190 ( 100 .00% ) Flows/s: 22 .86 Connected Nodes: 2 /2 \u547d\u4ee4\u884c\u89c2\u6d4b \u00b6 \u5728\u7ec8\u7aef\u4e0a\u6267\u884c hubble observe \u547d\u4ee4\uff0c\u67e5\u770b\u96c6\u7fa4\u7684\u6d41\u91cf\u4fe1\u606f\u3002 \u89c2\u6d4b\u5230\u7684 cilium-test \u76f8\u5173\u7684\u6d4b\u8bd5\u6d41\u91cf\u53c2\u8003\u5982\u4e0b\uff1a \u9700\u8981\u6ce8\u610f\u7684\u662f\uff0c hubble observe \u547d\u4ee4\u7684\u663e\u793a\u7ed3\u679c\uff0c\u662f\u5f53\u524d\u547d\u4ee4\u884c\u6267\u884c\u65f6\u67e5\u8be2\u5230\u7684\u6d41\u91cf\u4fe1\u606f\u3002\u591a\u6b21\u6267\u884c\u547d\u4ee4\u884c\uff0c\u53ef\u4ee5\u67e5\u770b\u5230\u4e0d\u540c\u7684\u6d41\u91cf\u4fe1\u606f\u3002 \u66f4\u591a\u8be6\u7ec6\u7684\u89c2\u6d4b\u4fe1\u606f\uff0c\u53ef\u4ee5\u6267\u884c hubble help observe \u547d\u4ee4\u67e5\u770b Hubble CLI \u7684\u8be6\u7ec6\u4f7f\u7528\u65b9\u5f0f\u3002 \u4f7f\u7528 UI \u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b \u00b6 \u6267\u884c cilium status \u547d\u4ee4\uff0c\u786e\u8ba4 Hubble UI \u5df2\u7ecf\u5b89\u88c5\u6210\u529f\u3002\u5728\u7b2c\u4e8c\u6b65\u7684 Hubble \u5b89\u88c5\u4e2d\uff0c\u5df2\u7ecf\u8865\u5145\u4e86 UI \u7684\u5b89\u88c5\u3002 \u6267\u884c\u547d\u4ee4 cilium hubble ui \u53ef\u4ee5\u81ea\u52a8\u521b\u5efa\u7aef\u53e3\u8f6c\u53d1\uff0c\u5c06 hubble-ui service \u6620\u5c04\u5230\u672c\u5730\u7aef\u53e3\u3002 \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u6267\u884c\u5b8c\u547d\u4ee4\u540e\uff0c\u4f1a\u81ea\u52a8\u6253\u5f00\u672c\u5730\u7684\u6d4f\u89c8\u5668\uff0c\u8df3\u8f6c\u5230 Hubble UI \u754c\u9762\u3002\u5982\u679c\u6ca1\u6709\u81ea\u52a8\u8df3\u8f6c\uff0c\u5728\u6d4f\u89c8\u5668\u4e2d\u8f93\u5165 http://localhost:12000 \u6253\u5f00 UI \u89c2\u5bdf\u754c\u9762\u3002 \u5728\u754c\u9762\u5de6\u4e0a\u89d2\uff0c\u9009\u62e9 cilium-test namespace\uff0c\u67e5\u770b Cilium \u63d0\u4f9b\u7684\u6d4b\u8bd5\u6d41\u91cf\u4fe1\u606f\u3002 Hubble \u6d41\u91cf\u76d1\u63a7 \u00b6 Hubble \u7ec4\u4ef6\u63d0\u4f9b\u4e86\u96c6\u7fa4\u4e2d Pod \u7f51\u7edc\u884c\u4e3a\u7684\u76d1\u63a7\uff0c\u4e3a\u4e86\u652f\u6301\u67e5\u770b Hubble \u63d0\u4f9b\u7684\u76d1\u63a7\u6570\u636e\uff0c\u9700\u8981\u4f7f\u80fd\u76d1\u63a7\u7edf\u8ba1\u3002 \u53c2\u8003\u4ee5\u4e0b\u547d\u4ee4\uff0c\u8865\u5145 hubble.metrics.enabled \u914d\u7f6e\u9879: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \\ --set hubble.metrics.enabled = \"{dns,drop,tcp,flow,icmp,http}\" \u90e8\u7f72\u4e4b\u540e\uff0c\u4f1a\u5728 kube-system namespace \u751f\u6210\u540d\u79f0\u4e3a hubble-metrics \u7684\u670d\u52a1\u3002\u901a\u8fc7\u8bbf\u95ee Endpoints \u67e5\u8be2 Hubble \u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\uff0c\u53c2\u8003\u5982\u4e0b: # curl 172.18.0.2:9091/metrics # HELP hubble_drop_total Number of drops # TYPE hubble_drop_total counter hubble_drop_total { protocol = \"ICMPv6\" ,reason = \"Unsupported L3 protocol\" } 2 # HELP hubble_flows_processed_total Total number of flows processed # TYPE hubble_flows_processed_total counter hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv6\" ,subtype = \"\" ,type = \"Drop\" ,verdict = \"DROPPED\" } 2 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 8282 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 6767 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 # HELP hubble_icmp_total Number of ICMP messages # TYPE hubble_icmp_total counter hubble_icmp_total { family = \"IPv4\" ,type = \"EchoReply\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"EchoRequest\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"RouterSolicitation\" } 2 # HELP hubble_tcp_flags_total TCP flag occurrences # TYPE hubble_tcp_flags_total counter hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"FIN\" } 2043 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"RST\" } 301 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN\" } 1169 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN-ACK\" } 1169 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium \u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#cilium","text":"Kube-OVN \u5f53\u524d\u5df2\u7ecf\u652f\u6301\u4e0e Cilium \u96c6\u6210\uff0c\u5177\u4f53\u64cd\u4f5c\u53ef\u4ee5\u53c2\u8003 Cilium \u96c6\u6210 \u3002 Cilium \u63d0\u4f9b\u4e86\u4e30\u5bcc\u7684\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u80fd\u529b\uff0c\u6d41\u91cf\u53ef\u89c2\u6d4b\u6027\u662f\u7531 Hubble \u63d0\u4f9b\u7684\u3002Hubble \u53ef\u4ee5\u89c2\u5bdf\u8282\u70b9\u3001\u96c6\u7fa4\u751a\u81f3\u591a\u96c6\u7fa4\u573a\u666f\u4e0b\u8de8\u96c6\u7fa4\u7684\u6d41\u91cf\u3002","title":"Cilium \u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#hubble","text":"\u9ed8\u8ba4\u7684 Cilium \u96c6\u6210\u5b89\u88c5\u4e2d\uff0c\u5e76\u6ca1\u6709\u5b89\u88c5 Hubble \u76f8\u5173\u7ec4\u4ef6\uff0c\u56e0\u6b64\u8981\u652f\u6301\u6d41\u91cf\u89c2\u6d4b\uff0c\u9700\u8981\u5148\u5728\u73af\u5883\u4e0a\u8865\u5145\u5b89\u88c5 Hubble\u3002 \u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u4f7f\u7528 helm \u5b89\u88c5 Hubble\uff1a helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \u8865\u5145\u5b89\u88c5 Hubble \u4e4b\u540e\uff0c\u6267\u884c cilium status \u67e5\u770b\u7ec4\u4ef6\u72b6\u6001\uff0c\u786e\u8ba4 Hubble \u5b89\u88c5\u6210\u529f\u3002 # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: OK \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ Deployment hubble-relay Desired: 1 , Ready: 1 /1, Available: 1 /1 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment hubble-ui Desired: 1 , Ready: 1 /1, Available: 1 /1 Containers: cilium Running: 2 hubble-ui Running: 1 hubble-relay Running: 1 cilium-operator Running: 2 Cluster Pods: 16 /17 managed by Cilium Image versions hubble-relay quay.io/cilium/hubble-relay:v1.11.6@sha256:fd9034a2d04d5b973f1e8ed44f230ea195b89c37955ff32e34e5aa68f3ed675a: 1 cilium-operator quay.io/cilium/operator-generic:v1.11.6@sha256:9f6063c7bcaede801a39315ec7c166309f6a6783e98665f6693939cf1701bc17: 2 cilium quay.io/cilium/cilium:v1.11.6@sha256:f7f93c26739b6641a3fa3d76b1e1605b15989f25d06625260099e01c8243f54c: 2 hubble-ui quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1 apple@bogon cilium % \u5b89\u88c5 Hubble \u7ec4\u4ef6\u4e4b\u540e\uff0c\u9700\u8981\u5b89\u88c5\u547d\u4ee4\u884c\uff0c\u7528\u4e8e\u5728\u73af\u5883\u4e0a\u67e5\u770b\u6d41\u91cf\u4fe1\u606f\u3002 \u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u5b89\u88c5 Hubble CLI : curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/v0.10.0/hubble-linux-amd64.tar.gz sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin","title":"\u5b89\u88c5 Hubble"},{"location":"advance/cilium-hubble-observe/#_1","text":"Cilium \u5b98\u65b9\u63d0\u4f9b\u4e86\u4e00\u4e2a\u6d41\u91cf\u6d4b\u8bd5\u7684\u90e8\u7f72\u65b9\u6848\uff0c\u53ef\u4ee5\u76f4\u63a5\u4f7f\u7528\u5b98\u65b9\u90e8\u7f72\u7684\u4e1a\u52a1\u8fdb\u884c\u6d4b\u8bd5\u3002 \u6267\u884c\u547d\u4ee4 cilium connectivity test \uff0cCilium \u4f1a\u81ea\u52a8\u521b\u5efa cilium-test \u7684 Namespace\uff0c\u540c\u65f6\u5728 cilium-test \u4e0b\u90e8\u7f72\u6d4b\u8bd5\u4e1a\u52a1\u3002 \u6b63\u5e38\u90e8\u7f72\u5b8c\u540e\uff0c\u53ef\u4ee5\u67e5\u770b cilium-test namespace \u4e0b\u7684\u8d44\u6e90\u4fe1\u606f\uff0c\u53c2\u8003\u5982\u4e0b\uff1a # kubectl get all -n cilium-test NAME READY STATUS RESTARTS AGE pod/client-7df6cfbf7b-z5t2j 1 /1 Running 0 21s pod/client2-547996d7d8-nvgxg 1 /1 Running 0 21s pod/echo-other-node-d79544ccf-hl4gg 2 /2 Running 0 21s pod/echo-same-node-5d466d5444-ml7tc 2 /2 Running 0 21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE service/echo-other-node NodePort 10 .109.58.126 <none> 8080 :32269/TCP 21s service/echo-same-node NodePort 10 .108.70.32 <none> 8080 :32490/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/client 1 /1 1 1 21s deployment.apps/client2 1 /1 1 1 21s deployment.apps/echo-other-node 1 /1 1 1 21s deployment.apps/echo-same-node 1 /1 1 1 21s NAME DESIRED CURRENT READY AGE replicaset.apps/client-7df6cfbf7b 1 1 1 21s replicaset.apps/client2-547996d7d8 1 1 1 21s replicaset.apps/echo-other-node-d79544ccf 1 1 1 21s replicaset.apps/echo-same-node-5d466d5444 1 1 1 21s","title":"\u90e8\u7f72\u6d4b\u8bd5\u4e1a\u52a1"},{"location":"advance/cilium-hubble-observe/#_2","text":"\u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u4ec5\u63d0\u4f9b\u6bcf\u4e2a\u8282\u70b9 Cilium \u4ee3\u7406\u89c2\u5bdf\u5230\u7684\u6d41\u91cf\u3002 \u53ef\u4ee5\u5728 kube-system namespace \u4e0b\u7684 Cilium \u4ee3\u7406 pod \u4e2d\u6267\u884c hubble observe \u547d\u4ee4\uff0c\u67e5\u770b\u8be5\u8282\u70b9\u4e0a\u7684\u6d41\u91cf\u4fe1\u606f\u3002 # kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cilium-d6h56 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-c7sb2 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-wj8gt 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> cilium-tq5xb 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> kube-ovn-pinger-7lgk8 1 /1 Running 0 21h 10 .16.0.19 kube-ovn-control-plane <none> <none> kube-ovn-pinger-msvcn 1 /1 Running 0 21h 10 .16.0.18 kube-ovn-worker <none> <none> # kubectl exec -it -n kube-system cilium-d6h56 -- bash root@kube-ovn-worker:/home/cilium# hubble observe --from-namespace kube-system Jul 29 03 :24:25.551: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: SYN ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.651: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.761: kube-system/kube-ovn-pinger-msvcn:52004 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn -> kube-system/kube-ovn-pinger-7lgk8 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn <- kube-system/kube-ovn-pinger-7lgk8 to-endpoint FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:80 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 -> kube-system/kube-ovn-pinger-msvcn to-endpoint FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 <- kube-system/kube-ovn-pinger-msvcn to-stack FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.979: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.3 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:26.037: kube-system/coredns-6d4b75cb6d-lbgjg:36430 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:26.282: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.2 to-stack FORWARDED ( ICMPv4 EchoRequest ) \u90e8\u7f72 Hubble Relay \u540e\uff0cHubble \u53ef\u4ee5\u63d0\u4f9b\u5b8c\u6574\u7684\u96c6\u7fa4\u8303\u56f4\u7684\u7f51\u7edc\u6d41\u91cf\u89c2\u6d4b\u3002","title":"\u4f7f\u7528\u547d\u4ee4\u884c\u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#_3","text":"\u4e3a\u4e86\u80fd\u6b63\u5e38\u8bbf\u95ee Hubble API\uff0c\u9700\u8981\u521b\u5efa\u7aef\u53e3\u8f6c\u53d1\uff0c\u5c06\u672c\u5730\u8bf7\u6c42\u8f6c\u53d1\u5230 Hubble Service\u3002\u53ef\u4ee5\u6267\u884c kubectl port-forward deployment/hubble-relay -n kube-system 4245:4245 \u547d\u4ee4\uff0c\u5728\u5f53\u524d\u7ec8\u7aef\u5f00\u542f\u7aef\u53e3\u8f6c\u53d1\u3002 \u7aef\u53e3\u8f6c\u53d1\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 \u7aef\u53e3\u8f6c\u53d1 \u3002 kubectl port-forward \u547d\u4ee4\u4e0d\u4f1a\u8fd4\u56de\uff0c\u9700\u8981\u6253\u5f00\u53e6\u4e00\u4e2a\u7ec8\u7aef\u6765\u7ee7\u7eed\u6d4b\u8bd5\u3002 \u914d\u7f6e\u5b8c\u7aef\u53e3\u8f6c\u53d1\u4e4b\u540e\uff0c\u5728\u7ec8\u7aef\u6267\u884c hubble status \u547d\u4ee4\uff0c\u5982\u679c\u6709\u7c7b\u4f3c\u5982\u4e0b\u8f93\u51fa\uff0c\u5219\u7aef\u53e3\u8f6c\u53d1\u914d\u7f6e\u6b63\u786e\uff0c\u53ef\u4ee5\u4f7f\u7528\u547d\u4ee4\u884c\u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b\u3002 # hubble status Healthcheck ( via localhost:4245 ) : Ok Current/Max Flows: 8 ,190/8,190 ( 100 .00% ) Flows/s: 22 .86 Connected Nodes: 2 /2","title":"\u914d\u7f6e\u7aef\u53e3\u8f6c\u53d1"},{"location":"advance/cilium-hubble-observe/#_4","text":"\u5728\u7ec8\u7aef\u4e0a\u6267\u884c hubble observe \u547d\u4ee4\uff0c\u67e5\u770b\u96c6\u7fa4\u7684\u6d41\u91cf\u4fe1\u606f\u3002 \u89c2\u6d4b\u5230\u7684 cilium-test \u76f8\u5173\u7684\u6d4b\u8bd5\u6d41\u91cf\u53c2\u8003\u5982\u4e0b\uff1a \u9700\u8981\u6ce8\u610f\u7684\u662f\uff0c hubble observe \u547d\u4ee4\u7684\u663e\u793a\u7ed3\u679c\uff0c\u662f\u5f53\u524d\u547d\u4ee4\u884c\u6267\u884c\u65f6\u67e5\u8be2\u5230\u7684\u6d41\u91cf\u4fe1\u606f\u3002\u591a\u6b21\u6267\u884c\u547d\u4ee4\u884c\uff0c\u53ef\u4ee5\u67e5\u770b\u5230\u4e0d\u540c\u7684\u6d41\u91cf\u4fe1\u606f\u3002 \u66f4\u591a\u8be6\u7ec6\u7684\u89c2\u6d4b\u4fe1\u606f\uff0c\u53ef\u4ee5\u6267\u884c hubble help observe \u547d\u4ee4\u67e5\u770b Hubble CLI \u7684\u8be6\u7ec6\u4f7f\u7528\u65b9\u5f0f\u3002","title":"\u547d\u4ee4\u884c\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#ui","text":"\u6267\u884c cilium status \u547d\u4ee4\uff0c\u786e\u8ba4 Hubble UI \u5df2\u7ecf\u5b89\u88c5\u6210\u529f\u3002\u5728\u7b2c\u4e8c\u6b65\u7684 Hubble \u5b89\u88c5\u4e2d\uff0c\u5df2\u7ecf\u8865\u5145\u4e86 UI \u7684\u5b89\u88c5\u3002 \u6267\u884c\u547d\u4ee4 cilium hubble ui \u53ef\u4ee5\u81ea\u52a8\u521b\u5efa\u7aef\u53e3\u8f6c\u53d1\uff0c\u5c06 hubble-ui service \u6620\u5c04\u5230\u672c\u5730\u7aef\u53e3\u3002 \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u6267\u884c\u5b8c\u547d\u4ee4\u540e\uff0c\u4f1a\u81ea\u52a8\u6253\u5f00\u672c\u5730\u7684\u6d4f\u89c8\u5668\uff0c\u8df3\u8f6c\u5230 Hubble UI \u754c\u9762\u3002\u5982\u679c\u6ca1\u6709\u81ea\u52a8\u8df3\u8f6c\uff0c\u5728\u6d4f\u89c8\u5668\u4e2d\u8f93\u5165 http://localhost:12000 \u6253\u5f00 UI \u89c2\u5bdf\u754c\u9762\u3002 \u5728\u754c\u9762\u5de6\u4e0a\u89d2\uff0c\u9009\u62e9 cilium-test namespace\uff0c\u67e5\u770b Cilium \u63d0\u4f9b\u7684\u6d4b\u8bd5\u6d41\u91cf\u4fe1\u606f\u3002","title":"\u4f7f\u7528 UI \u8fdb\u884c\u6d41\u91cf\u89c2\u6d4b"},{"location":"advance/cilium-hubble-observe/#hubble_1","text":"Hubble \u7ec4\u4ef6\u63d0\u4f9b\u4e86\u96c6\u7fa4\u4e2d Pod \u7f51\u7edc\u884c\u4e3a\u7684\u76d1\u63a7\uff0c\u4e3a\u4e86\u652f\u6301\u67e5\u770b Hubble \u63d0\u4f9b\u7684\u76d1\u63a7\u6570\u636e\uff0c\u9700\u8981\u4f7f\u80fd\u76d1\u63a7\u7edf\u8ba1\u3002 \u53c2\u8003\u4ee5\u4e0b\u547d\u4ee4\uff0c\u8865\u5145 hubble.metrics.enabled \u914d\u7f6e\u9879: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \\ --set hubble.metrics.enabled = \"{dns,drop,tcp,flow,icmp,http}\" \u90e8\u7f72\u4e4b\u540e\uff0c\u4f1a\u5728 kube-system namespace \u751f\u6210\u540d\u79f0\u4e3a hubble-metrics \u7684\u670d\u52a1\u3002\u901a\u8fc7\u8bbf\u95ee Endpoints \u67e5\u8be2 Hubble \u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\uff0c\u53c2\u8003\u5982\u4e0b: # curl 172.18.0.2:9091/metrics # HELP hubble_drop_total Number of drops # TYPE hubble_drop_total counter hubble_drop_total { protocol = \"ICMPv6\" ,reason = \"Unsupported L3 protocol\" } 2 # HELP hubble_flows_processed_total Total number of flows processed # TYPE hubble_flows_processed_total counter hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv6\" ,subtype = \"\" ,type = \"Drop\" ,verdict = \"DROPPED\" } 2 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 8282 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 6767 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 # HELP hubble_icmp_total Number of ICMP messages # TYPE hubble_icmp_total counter hubble_icmp_total { family = \"IPv4\" ,type = \"EchoReply\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"EchoRequest\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"RouterSolicitation\" } 2 # HELP hubble_tcp_flags_total TCP flag occurrences # TYPE hubble_tcp_flags_total counter hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"FIN\" } 2043 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"RST\" } 301 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN\" } 1169 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN-ACK\" } 1169 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Hubble \u6d41\u91cf\u76d1\u63a7"},{"location":"advance/cilium-networkpolicy/","text":"Cilium NetworkPolicy \u652f\u6301 \u00b6 Kube-OVN \u5f53\u524d\u5df2\u7ecf\u652f\u6301\u4e0e Cilium \u96c6\u6210\uff0c\u5177\u4f53\u64cd\u4f5c\u53ef\u4ee5\u53c2\u8003 Cilium \u96c6\u6210 \u3002 \u5728\u96c6\u6210 Cilium \u4e4b\u540e\uff0c\u5c31\u53ef\u4ee5\u4f7f\u7528 Cilium \u4f18\u79c0\u7684\u7f51\u7edc\u7b56\u7565\u80fd\u529b\uff0c\u5b9e\u73b0\u5bf9\u6d41\u91cf\u8bbf\u95ee\u7684\u63a7\u5236\u3002\u4ee5\u4e0b\u6587\u6863\u63d0\u4f9b\u4e86\u5bf9 Cilium L3 \u548c L4 \u7f51\u7edc\u7b56\u7565\u80fd\u529b\u7684\u96c6\u6210\u9a8c\u8bc1\u3002 \u9a8c\u8bc1\u6b65\u9aa4 \u00b6 \u521b\u5efa\u6d4b\u8bd5 Pod \u00b6 \u521b\u5efa namespace test \u3002\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u5728 test namespace \u4e2d\u521b\u5efa\u6307\u5b9a label app=test \u7684 Pod\uff0c\u4f5c\u4e3a\u6d4b\u8bd5\u8bbf\u95ee\u7684\u76ee\u7684 Pod\u3002 apiVersion : apps/v1 kind : Deployment metadata : labels : app : test name : test namespace : test spec : replicas : 1 selector : matchLabels : app : test strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : test spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx \u540c\u6837\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u5728 default namespace \u4e0b\u521b\u5efa\u6307\u5b9a label app=dynamic \u7684 Pod \u4e3a\u53d1\u8d77\u8bbf\u95ee\u6d4b\u8bd5\u7684 Pod\u3002 apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx \u67e5\u770b\u6d4b\u8bd5 Pod \u4ee5\u53ca Label \u4fe1\u606f: # kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-9v5c4 1 /1 Running 0 28h 10 .16.0.35 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-s8z2n 1 /1 Running 0 28h 10 .16.0.36 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 # kubectl get pod -o wide -n test --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-6dsg6 1 /1 Running 0 7h20m 10 .16.0.2 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-tjgtp 1 /1 Running 0 7h46m 10 .16.0.42 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 label-test1-77b6764857-swq4k 1 /1 Running 0 3h43m 10 .16.0.12 kube-ovn-worker <none> <none> app = test1,pod-template-hash = 77b6764857 // \u4ee5\u4e0b\u4e3a\u6d4b\u8bd5\u8bbf\u95ee\u76ee\u7684 Pod test-54c98bc466-mft5s 1 /1 Running 0 8h 10 .16.0.41 kube-ovn-worker <none> <none> app = test,pod-template-hash = 54c98bc466 L3 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa CiliumNetworkPolicy \u8d44\u6e90: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l3-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic \u5728 default namespace \u4e0b\u7684\u6d4b\u8bd5 Pod \u4e2d\uff0c\u53d1\u8d77\u5bf9\u76ee\u7684 Pod \u7684\u8bbf\u95ee\uff0c\u7ed3\u679c\u8bbf\u95ee\u4e0d\u901a\u3002 \u4f46\u662f\u5728 test namespace \u4e0b\uff0c\u6d4b\u8bd5\u5230\u76ee\u7684 Pod \u7684\u8bbf\u95ee\uff0c\u6d4b\u8bd5\u6b63\u5e38\u3002 default namespace \u4e0b\u6d4b\u8bd5\u7ed3\u679c: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss test namepsace \u4e0b Pod \u7684\u6d4b\u8bd5\uff0c\u8bbf\u95ee\u6b63\u5e38: # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .558 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .223 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .304 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .223/1.028/2.558 ms \u67e5\u770b Cilium \u5b98\u65b9\u6587\u6863\u89e3\u91ca\uff0c CiliumNetworkPolicy \u8d44\u6e90\u5c06\u9650\u5236\u63a7\u5236\u5728\u4e86 Namespace \u7ea7\u522b\u3002\u8be6\u7ec6\u4fe1\u606f\u53ef\u4ee5\u67e5\u770b Cilium \u9650\u5236 \u3002 \u5728\u6709\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u5339\u914d\u7684\u60c5\u51b5\u4e0b\uff0c\u53ea\u6709 \u540c\u4e00\u4e2a Namespace \u7684 Pod \uff0c\u624d\u53ef\u4ee5\u6309\u7167\u89c4\u5219\u8fdb\u884c\u8bbf\u95ee\uff0c\u9ed8\u8ba4\u62d2\u7edd \u5176\u4ed6 Namespace \u7684 Pod \u8fdb\u884c\u8bbf\u95ee\u3002 \u5982\u679c\u60f3\u5b9e\u73b0\u8de8 Namespace \u7684\u8bbf\u95ee\uff0c\u9700\u8981\u5728\u89c4\u5219\u4e2d\u660e\u786e\u6307\u5b9a Namespace \u4fe1\u606f\u3002 \u53c2\u8003\u6587\u6863\uff0c\u4fee\u6539 CiliumNetworkPolicy \u8d44\u6e90\uff0c\u589e\u52a0 namespace \u4fe1\u606f: ingress : - fromEndpoints : - matchLabels : app : dynamic k8s:io.kubernetes.pod.namespace : default // \u63a7\u5236\u5176\u4ed6 Namespace \u4e0b\u7684 Pod \u8bbf\u95ee \u67e5\u770b\u4fee\u6539\u540e\u7684 CiliumNetworkPolicy \u8d44\u6e90\u4fe1\u606f: # kubectl get cnp -n test -o yaml l3-rule apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: l3-rule namespace: test spec: endpointSelector: matchLabels: app: test ingress: - fromEndpoints: - matchLabels: app: dynamic - matchLabels: app: dynamic k8s:io.kubernetes.pod.namespace: default \u518d\u6b21\u6d4b\u8bd5 default namespace \u4e0b\u7684 Pod \u8bbf\u95ee\uff0c\u76ee\u7684 Pod \u8bbf\u95ee\u6b63\u5e38: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -n test -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .383 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .115 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .142 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .115/0.880/2.383 ms \u4f7f\u7528\u6807\u51c6\u7684 Kubernetes \u7f51\u7edc\u7b56\u7565 networkpolicy \uff0c\u6d4b\u8bd5\u7ed3\u679c\u663e\u793a Cilium \u540c\u6837\u5c06\u8bbf\u95ee\u9650\u5236\u5728\u540c\u4e00\u4e2a Namespace \u5185\uff0c\u8de8 Namespace \u7684\u8bbf\u95ee\u662f\u7981\u6b62\u7684\u3002 \u8fd9\u70b9\u4e0e Kube-OVN \u5b9e\u73b0\u662f\u4e0d\u540c\u7684\u3002Kube-OVN \u652f\u6301\u6807\u51c6\u7684 k8s \u7f51\u7edc\u7b56\u7565\uff0c\u9650\u5236\u4e86\u5177\u4f53 Namespace \u4e0b\u7684 \u76ee\u7684 Pod \uff0c\u4f46\u662f\u5bf9\u6e90\u5730\u5740 Pod\uff0c\u662f\u6ca1\u6709 Namespace \u9650\u5236\u7684\uff0c\u4efb\u4f55 Namespace \u4e0b\u7b26\u5408\u9650\u5236\u89c4\u5219\u7684 Pod\uff0c\u90fd\u53ef\u4ee5\u5b9e\u73b0\u5bf9\u76ee\u7684 Pod \u7684\u8bbf\u95ee\u3002 L4 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa L4 \u5c42\u7684\u7f51\u7edc\u7b56\u7565\u8d44\u6e90: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l4-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic toPorts : - ports : - port : \"80\" protocol : TCP \u6d4b\u8bd5\u76f8\u540c Namespace \u4e0b\uff0c\u7b26\u5408\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u7684 Pod \u7684\u8bbf\u95ee # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl 10 .16.0.41:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on test-54c98bc466-mft5s</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u76f8\u540c Namespace \u4e0b\uff0c\u4e0d\u7b26\u5408\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u7684 Pod \u8bbf\u95ee\u6d4b\u8bd5 # kubectl exec -it -n test label-test1-77b6764857-swq4k -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl -v 10 .16.0.41:80 --connect-timeout 10 * Trying 10 .16.0.41:80... * After 10000ms connect time, move on! * connect to 10 .16.0.41 port 80 failed: Operation timed out * Connection timeout after 10001 ms * Closing connection 0 curl: ( 28 ) Connection timeout after 10001 ms \u7f51\u7edc\u7b56\u7565\u751f\u6548\u540e\uff0c\u8de8 Namespace \u7684\u8bbf\u95ee\uff0c\u4f9d\u7136\u662f\u88ab\u7981\u6b62\u7684\uff0c\u8ddf L3 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5\u7ed3\u679c\u4e00\u81f4\u3002 \u5728 L4 \u7f51\u7edc\u89c4\u5219\u751f\u6548\u540e\uff0cping \u65e0\u6cd5\u4f7f\u7528\uff0c\u4f46\u662f\u7b26\u5408\u7b56\u7565\u89c4\u5219\u7684 TCP \u8bbf\u95ee\uff0c\u662f\u53ef\u4ee5\u6b63\u5e38\u6267\u884c\u7684\u3002 \u5173\u4e8e ICMP \u7684\u9650\u5236\uff0c\u53ef\u4ee5\u53c2\u8003\u5b98\u65b9\u8bf4\u660e L4 \u9650\u5236\u8bf4\u660e \u3002 L7 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5 \u00b6 chaining \u6a21\u5f0f\u4e0b\uff0cL7 \u7f51\u7edc\u7b56\u7565\u76ee\u524d\u662f\u5b58\u5728\u95ee\u9898\u7684\u3002\u5728 Cilium \u5b98\u65b9\u6587\u6863\u4e2d\uff0c\u5bf9\u8fd9\u79cd\u60c5\u51b5\u7ed9\u51fa\u4e86\u8bf4\u660e\uff0c\u53c2\u8003 Generic Veth Chaining \u3002 \u8fd9\u4e2a\u95ee\u9898\u4f7f\u7528 issue 12454 \u8ddf\u8e2a\uff0c\u76ee\u524d\u8fd8\u6ca1\u6709\u89e3\u51b3\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium NetworkPolicy \u652f\u6301"},{"location":"advance/cilium-networkpolicy/#cilium-networkpolicy","text":"Kube-OVN \u5f53\u524d\u5df2\u7ecf\u652f\u6301\u4e0e Cilium \u96c6\u6210\uff0c\u5177\u4f53\u64cd\u4f5c\u53ef\u4ee5\u53c2\u8003 Cilium \u96c6\u6210 \u3002 \u5728\u96c6\u6210 Cilium \u4e4b\u540e\uff0c\u5c31\u53ef\u4ee5\u4f7f\u7528 Cilium \u4f18\u79c0\u7684\u7f51\u7edc\u7b56\u7565\u80fd\u529b\uff0c\u5b9e\u73b0\u5bf9\u6d41\u91cf\u8bbf\u95ee\u7684\u63a7\u5236\u3002\u4ee5\u4e0b\u6587\u6863\u63d0\u4f9b\u4e86\u5bf9 Cilium L3 \u548c L4 \u7f51\u7edc\u7b56\u7565\u80fd\u529b\u7684\u96c6\u6210\u9a8c\u8bc1\u3002","title":"Cilium NetworkPolicy \u652f\u6301"},{"location":"advance/cilium-networkpolicy/#_1","text":"","title":"\u9a8c\u8bc1\u6b65\u9aa4"},{"location":"advance/cilium-networkpolicy/#pod","text":"\u521b\u5efa namespace test \u3002\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u5728 test namespace \u4e2d\u521b\u5efa\u6307\u5b9a label app=test \u7684 Pod\uff0c\u4f5c\u4e3a\u6d4b\u8bd5\u8bbf\u95ee\u7684\u76ee\u7684 Pod\u3002 apiVersion : apps/v1 kind : Deployment metadata : labels : app : test name : test namespace : test spec : replicas : 1 selector : matchLabels : app : test strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : test spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx \u540c\u6837\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u5728 default namespace \u4e0b\u521b\u5efa\u6307\u5b9a label app=dynamic \u7684 Pod \u4e3a\u53d1\u8d77\u8bbf\u95ee\u6d4b\u8bd5\u7684 Pod\u3002 apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx \u67e5\u770b\u6d4b\u8bd5 Pod \u4ee5\u53ca Label \u4fe1\u606f: # kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-9v5c4 1 /1 Running 0 28h 10 .16.0.35 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-s8z2n 1 /1 Running 0 28h 10 .16.0.36 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 # kubectl get pod -o wide -n test --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-6dsg6 1 /1 Running 0 7h20m 10 .16.0.2 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-tjgtp 1 /1 Running 0 7h46m 10 .16.0.42 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 label-test1-77b6764857-swq4k 1 /1 Running 0 3h43m 10 .16.0.12 kube-ovn-worker <none> <none> app = test1,pod-template-hash = 77b6764857 // \u4ee5\u4e0b\u4e3a\u6d4b\u8bd5\u8bbf\u95ee\u76ee\u7684 Pod test-54c98bc466-mft5s 1 /1 Running 0 8h 10 .16.0.41 kube-ovn-worker <none> <none> app = test,pod-template-hash = 54c98bc466","title":"\u521b\u5efa\u6d4b\u8bd5 Pod"},{"location":"advance/cilium-networkpolicy/#l3","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa CiliumNetworkPolicy \u8d44\u6e90: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l3-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic \u5728 default namespace \u4e0b\u7684\u6d4b\u8bd5 Pod \u4e2d\uff0c\u53d1\u8d77\u5bf9\u76ee\u7684 Pod \u7684\u8bbf\u95ee\uff0c\u7ed3\u679c\u8bbf\u95ee\u4e0d\u901a\u3002 \u4f46\u662f\u5728 test namespace \u4e0b\uff0c\u6d4b\u8bd5\u5230\u76ee\u7684 Pod \u7684\u8bbf\u95ee\uff0c\u6d4b\u8bd5\u6b63\u5e38\u3002 default namespace \u4e0b\u6d4b\u8bd5\u7ed3\u679c: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss test namepsace \u4e0b Pod \u7684\u6d4b\u8bd5\uff0c\u8bbf\u95ee\u6b63\u5e38: # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .558 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .223 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .304 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .223/1.028/2.558 ms \u67e5\u770b Cilium \u5b98\u65b9\u6587\u6863\u89e3\u91ca\uff0c CiliumNetworkPolicy \u8d44\u6e90\u5c06\u9650\u5236\u63a7\u5236\u5728\u4e86 Namespace \u7ea7\u522b\u3002\u8be6\u7ec6\u4fe1\u606f\u53ef\u4ee5\u67e5\u770b Cilium \u9650\u5236 \u3002 \u5728\u6709\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u5339\u914d\u7684\u60c5\u51b5\u4e0b\uff0c\u53ea\u6709 \u540c\u4e00\u4e2a Namespace \u7684 Pod \uff0c\u624d\u53ef\u4ee5\u6309\u7167\u89c4\u5219\u8fdb\u884c\u8bbf\u95ee\uff0c\u9ed8\u8ba4\u62d2\u7edd \u5176\u4ed6 Namespace \u7684 Pod \u8fdb\u884c\u8bbf\u95ee\u3002 \u5982\u679c\u60f3\u5b9e\u73b0\u8de8 Namespace \u7684\u8bbf\u95ee\uff0c\u9700\u8981\u5728\u89c4\u5219\u4e2d\u660e\u786e\u6307\u5b9a Namespace \u4fe1\u606f\u3002 \u53c2\u8003\u6587\u6863\uff0c\u4fee\u6539 CiliumNetworkPolicy \u8d44\u6e90\uff0c\u589e\u52a0 namespace \u4fe1\u606f: ingress : - fromEndpoints : - matchLabels : app : dynamic k8s:io.kubernetes.pod.namespace : default // \u63a7\u5236\u5176\u4ed6 Namespace \u4e0b\u7684 Pod \u8bbf\u95ee \u67e5\u770b\u4fee\u6539\u540e\u7684 CiliumNetworkPolicy \u8d44\u6e90\u4fe1\u606f: # kubectl get cnp -n test -o yaml l3-rule apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: l3-rule namespace: test spec: endpointSelector: matchLabels: app: test ingress: - fromEndpoints: - matchLabels: app: dynamic - matchLabels: app: dynamic k8s:io.kubernetes.pod.namespace: default \u518d\u6b21\u6d4b\u8bd5 default namespace \u4e0b\u7684 Pod \u8bbf\u95ee\uff0c\u76ee\u7684 Pod \u8bbf\u95ee\u6b63\u5e38: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -n test -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .383 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .115 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .142 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .115/0.880/2.383 ms \u4f7f\u7528\u6807\u51c6\u7684 Kubernetes \u7f51\u7edc\u7b56\u7565 networkpolicy \uff0c\u6d4b\u8bd5\u7ed3\u679c\u663e\u793a Cilium \u540c\u6837\u5c06\u8bbf\u95ee\u9650\u5236\u5728\u540c\u4e00\u4e2a Namespace \u5185\uff0c\u8de8 Namespace \u7684\u8bbf\u95ee\u662f\u7981\u6b62\u7684\u3002 \u8fd9\u70b9\u4e0e Kube-OVN \u5b9e\u73b0\u662f\u4e0d\u540c\u7684\u3002Kube-OVN \u652f\u6301\u6807\u51c6\u7684 k8s \u7f51\u7edc\u7b56\u7565\uff0c\u9650\u5236\u4e86\u5177\u4f53 Namespace \u4e0b\u7684 \u76ee\u7684 Pod \uff0c\u4f46\u662f\u5bf9\u6e90\u5730\u5740 Pod\uff0c\u662f\u6ca1\u6709 Namespace \u9650\u5236\u7684\uff0c\u4efb\u4f55 Namespace \u4e0b\u7b26\u5408\u9650\u5236\u89c4\u5219\u7684 Pod\uff0c\u90fd\u53ef\u4ee5\u5b9e\u73b0\u5bf9\u76ee\u7684 Pod \u7684\u8bbf\u95ee\u3002","title":"L3 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5"},{"location":"advance/cilium-networkpolicy/#l4","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa L4 \u5c42\u7684\u7f51\u7edc\u7b56\u7565\u8d44\u6e90: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l4-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic toPorts : - ports : - port : \"80\" protocol : TCP \u6d4b\u8bd5\u76f8\u540c Namespace \u4e0b\uff0c\u7b26\u5408\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u7684 Pod \u7684\u8bbf\u95ee # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl 10 .16.0.41:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on test-54c98bc466-mft5s</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u76f8\u540c Namespace \u4e0b\uff0c\u4e0d\u7b26\u5408\u7f51\u7edc\u7b56\u7565\u89c4\u5219\u7684 Pod \u8bbf\u95ee\u6d4b\u8bd5 # kubectl exec -it -n test label-test1-77b6764857-swq4k -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl -v 10 .16.0.41:80 --connect-timeout 10 * Trying 10 .16.0.41:80... * After 10000ms connect time, move on! * connect to 10 .16.0.41 port 80 failed: Operation timed out * Connection timeout after 10001 ms * Closing connection 0 curl: ( 28 ) Connection timeout after 10001 ms \u7f51\u7edc\u7b56\u7565\u751f\u6548\u540e\uff0c\u8de8 Namespace \u7684\u8bbf\u95ee\uff0c\u4f9d\u7136\u662f\u88ab\u7981\u6b62\u7684\uff0c\u8ddf L3 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5\u7ed3\u679c\u4e00\u81f4\u3002 \u5728 L4 \u7f51\u7edc\u89c4\u5219\u751f\u6548\u540e\uff0cping \u65e0\u6cd5\u4f7f\u7528\uff0c\u4f46\u662f\u7b26\u5408\u7b56\u7565\u89c4\u5219\u7684 TCP \u8bbf\u95ee\uff0c\u662f\u53ef\u4ee5\u6b63\u5e38\u6267\u884c\u7684\u3002 \u5173\u4e8e ICMP \u7684\u9650\u5236\uff0c\u53ef\u4ee5\u53c2\u8003\u5b98\u65b9\u8bf4\u660e L4 \u9650\u5236\u8bf4\u660e \u3002","title":"L4 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5"},{"location":"advance/cilium-networkpolicy/#l7","text":"chaining \u6a21\u5f0f\u4e0b\uff0cL7 \u7f51\u7edc\u7b56\u7565\u76ee\u524d\u662f\u5b58\u5728\u95ee\u9898\u7684\u3002\u5728 Cilium \u5b98\u65b9\u6587\u6863\u4e2d\uff0c\u5bf9\u8fd9\u79cd\u60c5\u51b5\u7ed9\u51fa\u4e86\u8bf4\u660e\uff0c\u53c2\u8003 Generic Veth Chaining \u3002 \u8fd9\u4e2a\u95ee\u9898\u4f7f\u7528 issue 12454 \u8ddf\u8e2a\uff0c\u76ee\u524d\u8fd8\u6ca1\u6709\u89e3\u51b3\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"L7 \u7f51\u7edc\u7b56\u7565\u6d4b\u8bd5"},{"location":"advance/dhcp/","text":"DHCP \u8bbe\u7f6e \u00b6 \u5728\u4f7f\u7528 SR-IOV \u6216 DPDK \u7c7b\u578b\u7f51\u7edc\u65f6\uff0cKubeVirt \u5185\u7f6e\u7684 DHCP \u65e0\u6cd5\u5728\u8be5\u7f51\u7edc\u6a21\u5f0f\u4e0b\u5de5\u4f5c\u3002Kube-OVN \u53ef\u4ee5\u5229\u7528 OVN \u7684 DHCP \u80fd\u529b\u5728\u5b50\u7f51\u7ea7\u522b\u8bbe\u7f6e DHCP \u9009\u9879\uff0c\u4ece\u800c\u5e2e\u52a9\u8be5\u7f51\u7edc\u7c7b\u578b\u7684 KubeVirt \u865a\u673a\u6b63\u5e38\u4f7f\u7528 DHCP \u83b7\u5f97\u5206\u914d\u7684 IP \u5730\u5740\u3002Kube-OVN \u540c\u65f6\u652f\u6301 DHCPv4 \u548c DHCPv6\u3002 \u5b50\u7f51 DHCP \u7684\u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : sn-dual spec : cidrBlock : \"10.0.0.0/24,240e::a00/120\" default : false disableGatewayCheck : true disableInterConnection : false excludeIps : - 10.0.0.1 - 240e::a01 gateway : 10.0.0.1,240e::a01 gatewayNode : '' gatewayType : distributed natOutgoing : false private : false protocol : Dual provider : ovn vpc : vpc-test enableDHCP : true dhcpV4Options : \"lease_time=3600,router=10.0.0.1,server_id=169.254.0.254,server_mac=00:00:00:2E:2F:B8\" dhcpV6Options : \"server_id=00:00:00:2E:2F:C5\" enableIPv6RA : true ipv6RAConfigs : \"address_mode=dhcpv6_stateful,max_interval=30,min_interval=5,send_periodic=true\" enableDHCP : \u662f\u5426\u5f00\u542f\u5b50\u7f51\u7684 DHCP \u529f\u80fd\u3002 dhcpV4Options , dhcpV6Options : \u8be5\u5b57\u6bb5\u76f4\u63a5\u66b4\u9732 ovn-nb \u5185 DHCP \u76f8\u5173\u9009\u9879\uff0c\u8bf7\u53c2\u8003 DHCP Options \u3002 \u9ed8\u8ba4\u503c\u5206\u522b\u4e3a \"lease_time=3600, router=$ipv4_gateway, server_id=169.254.0.254, server_mac=$random_mac\" \u548c server_id=$random_mac \u3002 enableIPv6RA : \u662f\u5426\u5f00\u542f DHCPv6 \u7684\u8def\u7531\u5e7f\u64ad\u529f\u80fd\u3002 ipv6RAConfigs \uff1a\u8be5\u5b57\u6bb5\u76f4\u63a5\u66b4\u9732 ovn-nb \u5185 Logical_Router_Port \u76f8\u5173\u9009\u9879\uff0c\u8bf7\u53c2\u8003 Logical Router Port \u9ed8\u8ba4\u503c\u4e3a address_mode=dhcpv6_stateful, max_interval=30, min_interval=5, send_periodic=true \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DHCP \u8bbe\u7f6e"},{"location":"advance/dhcp/#dhcp","text":"\u5728\u4f7f\u7528 SR-IOV \u6216 DPDK \u7c7b\u578b\u7f51\u7edc\u65f6\uff0cKubeVirt \u5185\u7f6e\u7684 DHCP \u65e0\u6cd5\u5728\u8be5\u7f51\u7edc\u6a21\u5f0f\u4e0b\u5de5\u4f5c\u3002Kube-OVN \u53ef\u4ee5\u5229\u7528 OVN \u7684 DHCP \u80fd\u529b\u5728\u5b50\u7f51\u7ea7\u522b\u8bbe\u7f6e DHCP \u9009\u9879\uff0c\u4ece\u800c\u5e2e\u52a9\u8be5\u7f51\u7edc\u7c7b\u578b\u7684 KubeVirt \u865a\u673a\u6b63\u5e38\u4f7f\u7528 DHCP \u83b7\u5f97\u5206\u914d\u7684 IP \u5730\u5740\u3002Kube-OVN \u540c\u65f6\u652f\u6301 DHCPv4 \u548c DHCPv6\u3002 \u5b50\u7f51 DHCP \u7684\u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : sn-dual spec : cidrBlock : \"10.0.0.0/24,240e::a00/120\" default : false disableGatewayCheck : true disableInterConnection : false excludeIps : - 10.0.0.1 - 240e::a01 gateway : 10.0.0.1,240e::a01 gatewayNode : '' gatewayType : distributed natOutgoing : false private : false protocol : Dual provider : ovn vpc : vpc-test enableDHCP : true dhcpV4Options : \"lease_time=3600,router=10.0.0.1,server_id=169.254.0.254,server_mac=00:00:00:2E:2F:B8\" dhcpV6Options : \"server_id=00:00:00:2E:2F:C5\" enableIPv6RA : true ipv6RAConfigs : \"address_mode=dhcpv6_stateful,max_interval=30,min_interval=5,send_periodic=true\" enableDHCP : \u662f\u5426\u5f00\u542f\u5b50\u7f51\u7684 DHCP \u529f\u80fd\u3002 dhcpV4Options , dhcpV6Options : \u8be5\u5b57\u6bb5\u76f4\u63a5\u66b4\u9732 ovn-nb \u5185 DHCP \u76f8\u5173\u9009\u9879\uff0c\u8bf7\u53c2\u8003 DHCP Options \u3002 \u9ed8\u8ba4\u503c\u5206\u522b\u4e3a \"lease_time=3600, router=$ipv4_gateway, server_id=169.254.0.254, server_mac=$random_mac\" \u548c server_id=$random_mac \u3002 enableIPv6RA : \u662f\u5426\u5f00\u542f DHCPv6 \u7684\u8def\u7531\u5e7f\u64ad\u529f\u80fd\u3002 ipv6RAConfigs \uff1a\u8be5\u5b57\u6bb5\u76f4\u63a5\u66b4\u9732 ovn-nb \u5185 Logical_Router_Port \u76f8\u5173\u9009\u9879\uff0c\u8bf7\u53c2\u8003 Logical Router Port \u9ed8\u8ba4\u503c\u4e3a address_mode=dhcpv6_stateful, max_interval=30, min_interval=5, send_periodic=true \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DHCP \u8bbe\u7f6e"},{"location":"advance/dpdk/","text":"DPDK \u652f\u6301 \u00b6 \u8be5\u6587\u6863\u4ecb\u7ecd Kube-OVN \u5982\u4f55\u548c OVS-DPDK \u7ed3\u5408\uff0c\u7ed9 KubeVirt \u7684\u865a\u673a\u63d0\u4f9b DPDK \u7c7b\u578b\u7684\u7f51\u7edc\u63a5\u53e3\u3002 \u4e0a\u6e38\u7684 KubeVirt \u76ee\u524d\u8fd8\u672a\u652f\u6301 OVS-DPDK\uff0c\u7528\u6237\u9700\u8981\u81ea\u5df1\u901a\u8fc7\u76f8\u5173 patch Vhostuser implementation \u6784\u5efa KubeVirt \u6216 KVM Device Plugin \u6765\u4f7f\u7528 OVS-DPDK\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u8282\u70b9\u9700\u63d0\u4f9b\u4e13\u95e8\u7ed9 DPDK \u9a71\u52a8\u8fd0\u884c\u7684\u7f51\u5361\u3002 \u8282\u70b9\u9700\u5f00\u542f Hugepages\u3002 \u7f51\u5361\u8bbe\u7f6e DPDK \u9a71\u52a8 \u00b6 \u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 driverctl \u4e3a\u4f8b\u8fdb\u884c\u64cd\u4f5c\uff0c\u5177\u4f53\u53c2\u6570\u548c\u5176\u4ed6\u9a71\u52a8\u4f7f\u7528\u8bf7\u53c2\u8003 DPDK \u6587\u6863 \u8fdb\u884c\u64cd\u4f5c\u3002 driverctl set-override 0000 :00:0b.0 uio_pci_generic \u8282\u70b9\u914d\u7f6e \u00b6 \u5bf9\u652f\u6301 OVS-DPDK \u7684\u8282\u70b9\u6253\u6807\u7b7e\uff0c\u4ee5\u4fbf Kube-OVN \u8fdb\u884c\u8bc6\u522b\u5904\u7406\uff1a kubectl label nodes <node> ovn.kubernetes.io/ovs_dp_type = \"userspace\" \u5728\u652f\u6301 OVS-DPDK \u8282\u70b9\u7684 /opt/ovs-config \u76ee\u5f55\u4e0b\u521b\u5efa\u914d\u7f6e\u6587\u4ef6 ovs-dpdk-config \uff1a ENCAP_IP = 192 .168.122.193/24 DPDK_DEV = 0000 :00:0b.0 ENCAP_IP : \u96a7\u9053\u7aef\u70b9\u5730\u5740\u3002 DPDK_DEV : \u8bbe\u5907\u7684 PCI ID\u3002 \u5b89\u88c5 Kube-OVN \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u542f\u7528 DPDK \u5b89\u88c5\u9009\u9879\u8fdb\u884c\u5b89\u88c5\uff1a bash install.sh --with-hybrid-dpdk \u4f7f\u7528\u65b9\u5f0f \u00b6 \u8fd9\u91cc\u6211\u4eec\u901a\u8fc7\u521b\u5efa\u4e00\u4e2a\u4f7f\u7528 vhostuser \u7c7b\u578b\u7f51\u5361\u7684\u865a\u673a\u6765\u9a8c\u8bc1 OVS-DPDK \u529f\u80fd\u3002 \u5b89\u88c5 KVM Device Plugin \u6765\u521b\u5efa\u865a\u673a\uff0c\u66f4\u591a\u4f7f\u7528\u65b9\u5f0f\u8bf7\u53c2\u8003 KVM Device Plugin \u3002 kubectl apply -f https://raw.githubusercontent.com/kubevirt/kubernetes-device-plugins/master/manifests/kvm-ds.yml \u521b\u5efa NetworkAttachmentDefinition\uff1a apiVersion : k8s.cni.cncf.io/v1 kind : NetworkAttachmentDefinition metadata : name : ovn-dpdk namespace : default spec : config : >- { \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-dpdk.default.ovn\", \"vhost_user_socket_volume_name\": \"vhostuser-sockets\", \"vhost_user_socket_name\": \"sock\" } \u4f7f\u7528\u4e0b\u9762\u7684 Dockerfile \u521b\u5efa VM \u955c\u50cf\uff1a FROM quay.io/kubevirt/virt-launcher:v0.46.1 # wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 COPY CentOS-7-x86_64-GenericCloud.qcow2 /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2 \u521b\u5efa\u865a\u62df\u673a\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vm-config data : start.sh : | chmod u+w /etc/libvirt/qemu.conf echo \"hugetlbfs_mount = \\\"/dev/hugepages\\\"\" >> /etc/libvirt/qemu.conf virtlogd & libvirtd & mkdir /var/lock sleep 5 virsh define /root/vm/vm.xml virsh start vm tail -f /dev/null vm.xml : | <domain type='kvm'> <name>vm</name> <uuid>4a9b3f53-fa2a-47f3-a757-dd87720d9d1d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <hugepages> <page size='2' unit='M' nodeset='0'/> </hugepages> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <emulatorpin cpuset='1,3'/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='2' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='2097152' unit='KiB' memAccess='shared'/> </numa> </cpu> <on_reboot>restart</on_reboot> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <interface type='vhostuser'> <mac address='00:00:00:0A:30:89'/> <source type='unix' path='/var/run/vm/sock' mode='server'/> <model type='virtio'/> <driver queues='2'> <host mrg_rxbuf='off'/> </driver> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-vm/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> </devices> </domain> --- apiVersion : apps/v1 kind : Deployment metadata : name : vm-deployment labels : app : vm spec : replicas : 1 selector : matchLabels : app : vm template : metadata : labels : app : vm annotations : k8s.v1.cni.cncf.io/networks : default/ovn-dpdk ovn-dpdk.default.ovn.kubernetes.io/ip_address : 10.16.0.96 ovn-dpdk.default.ovn.kubernetes.io/mac_address : 00:00:00:0A:30:89 spec : nodeSelector : ovn.kubernetes.io/ovs_dp_type : userspace securityContext : runAsUser : 0 volumes : - name : vhostuser-sockets emptyDir : {} - name : xml configMap : name : vm-config - name : hugepage emptyDir : medium : HugePages-2Mi - name : libvirt-runtime emptyDir : {} containers : - name : vm image : vm-vhostuser:latest command : [ \"bash\" , \"/root/vm/start.sh\" ] securityContext : capabilities : add : - NET_BIND_SERVICE - SYS_NICE - NET_RAW - NET_ADMIN privileged : false runAsUser : 0 resources : limits : cpu : '2' devices.kubevirt.io/kvm : '1' memory : '8784969729' hugepages-2Mi : 2Gi requests : cpu : 666m devices.kubevirt.io/kvm : '1' ephemeral-storage : 50M memory : '4490002433' volumeMounts : - name : vhostuser-sockets mountPath : /var/run/vm - name : xml mountPath : /root/vm/ - mountPath : /dev/hugepages name : hugepage - name : libvirt-runtime mountPath : /var/run/libvirt \u7b49\u5f85\u865a\u62df\u673a\u521b\u5efa\u6210\u529f\u540e\u8fdb\u5165 Pod \u8fdb\u884c\u865a\u673a\u914d\u7f6e\uff1a # virsh set-user-password vm root 12345 Password set successfully for root in vm # virsh console vm Connected to domain 'vm' Escape character is ^ ] ( Ctrl + ]) CentOS Linux 7 ( Core ) Kernel 3 .10.0-1127.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Fri Feb 25 09 :52:54 on ttyS0 \u63a5\u4e0b\u6765\u53ef\u4ee5\u767b\u5f55\u865a\u673a\u8fdb\u884c\u7f51\u7edc\u914d\u7f6e\u5e76\u6d4b\u8bd5\uff1a ip link set eth0 mtu 1400 ip addr add 10 .16.0.96/16 dev eth0 ip ro add default via 10 .16.0.1 ping 114 .114.114.114 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DPDK \u652f\u6301"},{"location":"advance/dpdk/#dpdk","text":"\u8be5\u6587\u6863\u4ecb\u7ecd Kube-OVN \u5982\u4f55\u548c OVS-DPDK \u7ed3\u5408\uff0c\u7ed9 KubeVirt \u7684\u865a\u673a\u63d0\u4f9b DPDK \u7c7b\u578b\u7684\u7f51\u7edc\u63a5\u53e3\u3002 \u4e0a\u6e38\u7684 KubeVirt \u76ee\u524d\u8fd8\u672a\u652f\u6301 OVS-DPDK\uff0c\u7528\u6237\u9700\u8981\u81ea\u5df1\u901a\u8fc7\u76f8\u5173 patch Vhostuser implementation \u6784\u5efa KubeVirt \u6216 KVM Device Plugin \u6765\u4f7f\u7528 OVS-DPDK\u3002","title":"DPDK \u652f\u6301"},{"location":"advance/dpdk/#_1","text":"\u8282\u70b9\u9700\u63d0\u4f9b\u4e13\u95e8\u7ed9 DPDK \u9a71\u52a8\u8fd0\u884c\u7684\u7f51\u5361\u3002 \u8282\u70b9\u9700\u5f00\u542f Hugepages\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/dpdk/#dpdk_1","text":"\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 driverctl \u4e3a\u4f8b\u8fdb\u884c\u64cd\u4f5c\uff0c\u5177\u4f53\u53c2\u6570\u548c\u5176\u4ed6\u9a71\u52a8\u4f7f\u7528\u8bf7\u53c2\u8003 DPDK \u6587\u6863 \u8fdb\u884c\u64cd\u4f5c\u3002 driverctl set-override 0000 :00:0b.0 uio_pci_generic","title":"\u7f51\u5361\u8bbe\u7f6e DPDK \u9a71\u52a8"},{"location":"advance/dpdk/#_2","text":"\u5bf9\u652f\u6301 OVS-DPDK \u7684\u8282\u70b9\u6253\u6807\u7b7e\uff0c\u4ee5\u4fbf Kube-OVN \u8fdb\u884c\u8bc6\u522b\u5904\u7406\uff1a kubectl label nodes <node> ovn.kubernetes.io/ovs_dp_type = \"userspace\" \u5728\u652f\u6301 OVS-DPDK \u8282\u70b9\u7684 /opt/ovs-config \u76ee\u5f55\u4e0b\u521b\u5efa\u914d\u7f6e\u6587\u4ef6 ovs-dpdk-config \uff1a ENCAP_IP = 192 .168.122.193/24 DPDK_DEV = 0000 :00:0b.0 ENCAP_IP : \u96a7\u9053\u7aef\u70b9\u5730\u5740\u3002 DPDK_DEV : \u8bbe\u5907\u7684 PCI ID\u3002","title":"\u8282\u70b9\u914d\u7f6e"},{"location":"advance/dpdk/#kube-ovn","text":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u542f\u7528 DPDK \u5b89\u88c5\u9009\u9879\u8fdb\u884c\u5b89\u88c5\uff1a bash install.sh --with-hybrid-dpdk","title":"\u5b89\u88c5 Kube-OVN"},{"location":"advance/dpdk/#_3","text":"\u8fd9\u91cc\u6211\u4eec\u901a\u8fc7\u521b\u5efa\u4e00\u4e2a\u4f7f\u7528 vhostuser \u7c7b\u578b\u7f51\u5361\u7684\u865a\u673a\u6765\u9a8c\u8bc1 OVS-DPDK \u529f\u80fd\u3002 \u5b89\u88c5 KVM Device Plugin \u6765\u521b\u5efa\u865a\u673a\uff0c\u66f4\u591a\u4f7f\u7528\u65b9\u5f0f\u8bf7\u53c2\u8003 KVM Device Plugin \u3002 kubectl apply -f https://raw.githubusercontent.com/kubevirt/kubernetes-device-plugins/master/manifests/kvm-ds.yml \u521b\u5efa NetworkAttachmentDefinition\uff1a apiVersion : k8s.cni.cncf.io/v1 kind : NetworkAttachmentDefinition metadata : name : ovn-dpdk namespace : default spec : config : >- { \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-dpdk.default.ovn\", \"vhost_user_socket_volume_name\": \"vhostuser-sockets\", \"vhost_user_socket_name\": \"sock\" } \u4f7f\u7528\u4e0b\u9762\u7684 Dockerfile \u521b\u5efa VM \u955c\u50cf\uff1a FROM quay.io/kubevirt/virt-launcher:v0.46.1 # wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 COPY CentOS-7-x86_64-GenericCloud.qcow2 /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2 \u521b\u5efa\u865a\u62df\u673a\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vm-config data : start.sh : | chmod u+w /etc/libvirt/qemu.conf echo \"hugetlbfs_mount = \\\"/dev/hugepages\\\"\" >> /etc/libvirt/qemu.conf virtlogd & libvirtd & mkdir /var/lock sleep 5 virsh define /root/vm/vm.xml virsh start vm tail -f /dev/null vm.xml : | <domain type='kvm'> <name>vm</name> <uuid>4a9b3f53-fa2a-47f3-a757-dd87720d9d1d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <hugepages> <page size='2' unit='M' nodeset='0'/> </hugepages> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <emulatorpin cpuset='1,3'/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='2' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='2097152' unit='KiB' memAccess='shared'/> </numa> </cpu> <on_reboot>restart</on_reboot> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <interface type='vhostuser'> <mac address='00:00:00:0A:30:89'/> <source type='unix' path='/var/run/vm/sock' mode='server'/> <model type='virtio'/> <driver queues='2'> <host mrg_rxbuf='off'/> </driver> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-vm/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> </devices> </domain> --- apiVersion : apps/v1 kind : Deployment metadata : name : vm-deployment labels : app : vm spec : replicas : 1 selector : matchLabels : app : vm template : metadata : labels : app : vm annotations : k8s.v1.cni.cncf.io/networks : default/ovn-dpdk ovn-dpdk.default.ovn.kubernetes.io/ip_address : 10.16.0.96 ovn-dpdk.default.ovn.kubernetes.io/mac_address : 00:00:00:0A:30:89 spec : nodeSelector : ovn.kubernetes.io/ovs_dp_type : userspace securityContext : runAsUser : 0 volumes : - name : vhostuser-sockets emptyDir : {} - name : xml configMap : name : vm-config - name : hugepage emptyDir : medium : HugePages-2Mi - name : libvirt-runtime emptyDir : {} containers : - name : vm image : vm-vhostuser:latest command : [ \"bash\" , \"/root/vm/start.sh\" ] securityContext : capabilities : add : - NET_BIND_SERVICE - SYS_NICE - NET_RAW - NET_ADMIN privileged : false runAsUser : 0 resources : limits : cpu : '2' devices.kubevirt.io/kvm : '1' memory : '8784969729' hugepages-2Mi : 2Gi requests : cpu : 666m devices.kubevirt.io/kvm : '1' ephemeral-storage : 50M memory : '4490002433' volumeMounts : - name : vhostuser-sockets mountPath : /var/run/vm - name : xml mountPath : /root/vm/ - mountPath : /dev/hugepages name : hugepage - name : libvirt-runtime mountPath : /var/run/libvirt \u7b49\u5f85\u865a\u62df\u673a\u521b\u5efa\u6210\u529f\u540e\u8fdb\u5165 Pod \u8fdb\u884c\u865a\u673a\u914d\u7f6e\uff1a # virsh set-user-password vm root 12345 Password set successfully for root in vm # virsh console vm Connected to domain 'vm' Escape character is ^ ] ( Ctrl + ]) CentOS Linux 7 ( Core ) Kernel 3 .10.0-1127.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Fri Feb 25 09 :52:54 on ttyS0 \u63a5\u4e0b\u6765\u53ef\u4ee5\u767b\u5f55\u865a\u673a\u8fdb\u884c\u7f51\u7edc\u914d\u7f6e\u5e76\u6d4b\u8bd5\uff1a ip link set eth0 mtu 1400 ip addr add 10 .16.0.96/16 dev eth0 ip ro add default via 10 .16.0.1 ping 114 .114.114.114 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528\u65b9\u5f0f"},{"location":"advance/external-gateway/","text":"\u5916\u90e8\u7f51\u5173\u8bbe\u7f6e \u00b6 \u5728\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u5bf9\u6240\u6709\u5bb9\u5668\u8bbf\u95ee\u5916\u90e8\u7684\u6d41\u91cf\u9700\u8981\u901a\u8fc7\u4e00\u4e2a\u5916\u90e8\u7684\u7f51\u5173\u8fdb\u884c\u7edf\u4e00\u7684\u7ba1\u7406\u548c\u5ba1\u8ba1\u3002 Kube-OVN \u53ef\u4ee5\u901a\u8fc7\u5728\u5b50\u7f51\u4e2d\u8fdb\u884c\u76f8\u5e94\u7684\u8def\u7531\u914d\u7f6e\uff0c\u5c06\u51fa\u7f51\u6d41\u91cf\u8f6c\u53d1\u81f3\u5bf9\u5e94\u7684\u5916\u90e8\u7f51\u5173\u3002 \u4f7f\u7528\u65b9\u5f0f \u00b6 kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : external spec : cidrBlock : 172.31.0.0/16 gatewayType : centralized natOutgoing : false externalEgressGateway : 192.168.0.1 policyRoutingTableID : 1000 policyRoutingPriority : 1500 natOutgoing : \u9700\u8981\u8bbe\u7f6e\u4e3a false \u3002 externalEgressGateway \uff1a\u8bbe\u7f6e\u4e3a\u5916\u90e8\u7f51\u5173\u7684\u5730\u5740\uff0c\u9700\u8981\u548c\u7f51\u5173\u8282\u70b9\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u53ef\u8fbe\u57df\u3002 policyRoutingTableID \uff1a\u4f7f\u7528\u7684\u672c\u5730\u7b56\u7565\u8def\u7531\u8868\u7684 TableID \u6bcf\u4e2a\u5b50\u7f51\u5747\u9700\u4e0d\u540c\u4ee5\u907f\u514d\u51b2\u7a81\u3002 policyRoutingPriority \uff1a\u8def\u7531\u4f18\u5148\u7ea7\uff0c\u4e3a\u907f\u514d\u540e\u7eed\u7528\u6237\u5b9a\u5236\u5316\u7684\u5176\u4ed6\u8def\u7531\u64cd\u4f5c\u51b2\u7a81\uff0c\u8fd9\u91cc\u53ef\u4ee5\u6307\u5b9a\u8def\u7531\u4f18\u5148\u7ea7\uff0c\u82e5\u65e0\u7279\u6b8a\u9700\u6c42\u586b\u5165\u4efb\u610f\u503c\u5373\u53ef\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5916\u90e8\u7f51\u5173\u8bbe\u7f6e"},{"location":"advance/external-gateway/#_1","text":"\u5728\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u5bf9\u6240\u6709\u5bb9\u5668\u8bbf\u95ee\u5916\u90e8\u7684\u6d41\u91cf\u9700\u8981\u901a\u8fc7\u4e00\u4e2a\u5916\u90e8\u7684\u7f51\u5173\u8fdb\u884c\u7edf\u4e00\u7684\u7ba1\u7406\u548c\u5ba1\u8ba1\u3002 Kube-OVN \u53ef\u4ee5\u901a\u8fc7\u5728\u5b50\u7f51\u4e2d\u8fdb\u884c\u76f8\u5e94\u7684\u8def\u7531\u914d\u7f6e\uff0c\u5c06\u51fa\u7f51\u6d41\u91cf\u8f6c\u53d1\u81f3\u5bf9\u5e94\u7684\u5916\u90e8\u7f51\u5173\u3002","title":"\u5916\u90e8\u7f51\u5173\u8bbe\u7f6e"},{"location":"advance/external-gateway/#_2","text":"kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : external spec : cidrBlock : 172.31.0.0/16 gatewayType : centralized natOutgoing : false externalEgressGateway : 192.168.0.1 policyRoutingTableID : 1000 policyRoutingPriority : 1500 natOutgoing : \u9700\u8981\u8bbe\u7f6e\u4e3a false \u3002 externalEgressGateway \uff1a\u8bbe\u7f6e\u4e3a\u5916\u90e8\u7f51\u5173\u7684\u5730\u5740\uff0c\u9700\u8981\u548c\u7f51\u5173\u8282\u70b9\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u53ef\u8fbe\u57df\u3002 policyRoutingTableID \uff1a\u4f7f\u7528\u7684\u672c\u5730\u7b56\u7565\u8def\u7531\u8868\u7684 TableID \u6bcf\u4e2a\u5b50\u7f51\u5747\u9700\u4e0d\u540c\u4ee5\u907f\u514d\u51b2\u7a81\u3002 policyRoutingPriority \uff1a\u8def\u7531\u4f18\u5148\u7ea7\uff0c\u4e3a\u907f\u514d\u540e\u7eed\u7528\u6237\u5b9a\u5236\u5316\u7684\u5176\u4ed6\u8def\u7531\u64cd\u4f5c\u51b2\u7a81\uff0c\u8fd9\u91cc\u53ef\u4ee5\u6307\u5b9a\u8def\u7531\u4f18\u5148\u7ea7\uff0c\u82e5\u65e0\u7279\u6b8a\u9700\u6c42\u586b\u5165\u4efb\u610f\u503c\u5373\u53ef\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528\u65b9\u5f0f"},{"location":"advance/fastpath/","text":"\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757 \u00b6 \u7ecf\u8fc7\u6570\u636e\u5e73\u9762\u7684\u6027\u80fd Profile\uff0c Netfilter \u5728\u5bb9\u5668\u5185\u548c\u5bbf\u4e3b\u673a\u4e0a\u7684\u76f8\u5173\u5904\u7406\u6d88\u8017\u4e86 20% \u5de6\u53f3\u7684 CPU \u8d44\u6e90\uff0cFastPath \u6a21\u5757\u53ef\u4ee5\u7ed5\u8fc7 Netfilter \u4ece\u800c \u964d\u4f4e CPU \u7684\u6d88\u8017\u548c\u5ef6\u8fdf\uff0c\u5e76\u63d0\u5347\u541e\u5410\u91cf\u3002\u672c\u6587\u6863\u5c06\u4ecb\u7ecd\u5982\u4f55\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757\u3002 \u4e0b\u8f7d\u76f8\u5173\u5185\u6838\u6a21\u5757\u4ee3\u7801 \u00b6 git clone --depth = 1 https://github.com/kubeovn/kube-ovn.git \u5b89\u88c5\u4f9d\u8d56 \u00b6 \u8fd9\u91cc\u4ee5 CentOS \u4e3a\u4f8b\u4e0b\u8f7d\u76f8\u5173\u4f9d\u8d56\uff1a yum install -y kernel-devel- $( uname -r ) gcc elfutils-libelf-devel \u7f16\u8bd1\u76f8\u5173\u6a21\u5757 \u00b6 \u9488\u5bf9 3.x \u7684\u5185\u6838\uff1a cd kube-ovn/fastpath make all \u9488\u5bf9 4.x \u7684\u5185\u6838\uff1a cd kube-ovn/fastpath/4.18 cp ../Makefile . make all \u5b89\u88c5\u5185\u6838\u6a21\u5757 \u00b6 \u5c06 kube_ovn_fastpath.ko \u590d\u5236\u5230\u6bcf\u4e2a\u9700\u8981\u6027\u80fd\u4f18\u5316\u7684\u8282\u70b9 /tmp \u76ee\u5f55\u4e0b\uff0c kube-ovn-cni \u4f1a\u81ea\u52a8\u52a0\u8f7d\u8be5\u6a21\u5757\u3002 \u4f7f\u7528 dmesg \u786e\u8ba4\u5b89\u88c5\u6210\u529f\uff1a # dmesg [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in \u5982\u9700\u5378\u8f7d\u6a21\u5757\uff0c\u53ef\u5c06\u8be5\u6a21\u5757\u4ece /tmp \u76ee\u5f55\u4e0b\u79fb\u9664\uff0c kube-ovn-cni \u4f1a\u81ea\u52a8\u5378\u8f7d\u8be5\u6a21\u5757\u3002 \u8be5\u6a21\u5757\u5728\u673a\u5668\u91cd\u542f\u540e\u4e0d\u4f1a\u81ea\u52a8\u52a0\u8f7d\uff0c\u5982\u9700\u81ea\u52a8\u52a0\u8f7d\u8bf7\u6839\u636e\u7cfb\u7edf\u5f04\u914d\u7f6e\u7f16\u5199\u76f8\u5e94\u81ea\u542f\u52a8\u811a\u672c\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757"},{"location":"advance/fastpath/#fastpath","text":"\u7ecf\u8fc7\u6570\u636e\u5e73\u9762\u7684\u6027\u80fd Profile\uff0c Netfilter \u5728\u5bb9\u5668\u5185\u548c\u5bbf\u4e3b\u673a\u4e0a\u7684\u76f8\u5173\u5904\u7406\u6d88\u8017\u4e86 20% \u5de6\u53f3\u7684 CPU \u8d44\u6e90\uff0cFastPath \u6a21\u5757\u53ef\u4ee5\u7ed5\u8fc7 Netfilter \u4ece\u800c \u964d\u4f4e CPU \u7684\u6d88\u8017\u548c\u5ef6\u8fdf\uff0c\u5e76\u63d0\u5347\u541e\u5410\u91cf\u3002\u672c\u6587\u6863\u5c06\u4ecb\u7ecd\u5982\u4f55\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757\u3002","title":"\u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757"},{"location":"advance/fastpath/#_1","text":"git clone --depth = 1 https://github.com/kubeovn/kube-ovn.git","title":"\u4e0b\u8f7d\u76f8\u5173\u5185\u6838\u6a21\u5757\u4ee3\u7801"},{"location":"advance/fastpath/#_2","text":"\u8fd9\u91cc\u4ee5 CentOS \u4e3a\u4f8b\u4e0b\u8f7d\u76f8\u5173\u4f9d\u8d56\uff1a yum install -y kernel-devel- $( uname -r ) gcc elfutils-libelf-devel","title":"\u5b89\u88c5\u4f9d\u8d56"},{"location":"advance/fastpath/#_3","text":"\u9488\u5bf9 3.x \u7684\u5185\u6838\uff1a cd kube-ovn/fastpath make all \u9488\u5bf9 4.x \u7684\u5185\u6838\uff1a cd kube-ovn/fastpath/4.18 cp ../Makefile . make all","title":"\u7f16\u8bd1\u76f8\u5173\u6a21\u5757"},{"location":"advance/fastpath/#_4","text":"\u5c06 kube_ovn_fastpath.ko \u590d\u5236\u5230\u6bcf\u4e2a\u9700\u8981\u6027\u80fd\u4f18\u5316\u7684\u8282\u70b9 /tmp \u76ee\u5f55\u4e0b\uff0c kube-ovn-cni \u4f1a\u81ea\u52a8\u52a0\u8f7d\u8be5\u6a21\u5757\u3002 \u4f7f\u7528 dmesg \u786e\u8ba4\u5b89\u88c5\u6210\u529f\uff1a # dmesg [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in \u5982\u9700\u5378\u8f7d\u6a21\u5757\uff0c\u53ef\u5c06\u8be5\u6a21\u5757\u4ece /tmp \u76ee\u5f55\u4e0b\u79fb\u9664\uff0c kube-ovn-cni \u4f1a\u81ea\u52a8\u5378\u8f7d\u8be5\u6a21\u5757\u3002 \u8be5\u6a21\u5757\u5728\u673a\u5668\u91cd\u542f\u540e\u4e0d\u4f1a\u81ea\u52a8\u52a0\u8f7d\uff0c\u5982\u9700\u81ea\u52a8\u52a0\u8f7d\u8bf7\u6839\u636e\u7cfb\u7edf\u5f04\u914d\u7f6e\u7f16\u5199\u76f8\u5e94\u81ea\u542f\u52a8\u811a\u672c\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b89\u88c5\u5185\u6838\u6a21\u5757"},{"location":"advance/multi-nic/","text":"\u591a\u7f51\u5361\u7ba1\u7406 \u00b6 Kube-OVN \u53ef\u4ee5\u4e3a\u5176\u4ed6 CNI \u7f51\u7edc\u63d2\u4ef6\uff0c\u4f8b\u5982 macvlan\u3001vlan\u3001host-device \u7b49\u63d2\u4ef6\u63d0\u4f9b\u96c6\u7fa4\u7ea7\u522b\u7684 IPAM \u80fd\u529b\uff0c \u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4e5f\u53ef\u4ee5\u4f7f\u7528\u5230 Kube-OVN \u4e2d\u5b50\u7f51\u4ee5\u53ca\u56fa\u5b9a IP \u529f\u80fd\u3002 \u540c\u65f6 Kube-OVN \u4e5f\u652f\u6301\u591a\u5757\u7f51\u5361\u5747\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\u60c5\u51b5\u4e0b\u7684\u5730\u5740\u7ba1\u7406\u3002 \u5de5\u4f5c\u539f\u7406 \u00b6 \u901a\u8fc7\u4f7f\u7528 Multus CNI , \u6211\u4eec\u53ef\u4ee5\u7ed9\u4e00\u4e2a Pod \u6dfb\u52a0\u591a\u5757\u4e0d\u540c\u7f51\u7edc\u7684\u7f51\u5361\u3002 \u7136\u800c\u6211\u4eec\u4ecd\u7136\u7f3a\u4e4f\u5bf9\u96c6\u7fa4\u8303\u56f4\u5185\u4e0d\u540c\u7f51\u7edc\u7684 IP \u5730\u5740\u8fdb\u884c\u7ba1\u7406\u7684\u80fd\u529b\u3002\u5728 Kube-OVN \u4e2d\uff0c\u6211\u4eec\u5df2\u7ecf\u80fd\u591f\u901a\u8fc7 Subnet \u548c IP \u7684 CRD \u6765\u8fdb\u884c IP \u7684\u9ad8\u7ea7\u7ba1\u7406\uff0c \u4f8b\u5982\u5b50\u7f51\u7ba1\u7406\uff0cIP \u9884\u7559\uff0c\u968f\u673a\u5206\u914d\uff0c\u56fa\u5b9a\u5206\u914d\u7b49\u3002\u73b0\u5728\u6211\u4eec\u5bf9\u5b50\u7f51\u8fdb\u884c\u6269\u5c55\uff0c\u6765\u63a5\u5165\u5176\u4ed6\u4e0d\u540c\u7684\u7f51\u7edc\u63d2\u4ef6\uff0c\u4f7f\u5f97\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4e5f\u53ef\u4ee5\u4f7f\u7528 Kube-OVN \u7684 IPAM \u529f\u80fd\u3002 \u5de5\u4f5c\u6d41\u7a0b \u00b6 \u4e0a\u56fe\u5c55\u793a\u4e86\u5982\u4f55\u901a\u8fc7 Kube-OVN \u6765\u7ba1\u7406\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u7684 IP \u5730\u5740\u3002\u5176\u4e2d\u5bb9\u5668\u7684 eth0 \u7f51\u5361\u63a5\u5165 OVN \u7f51\u7edc\uff0cnet1 \u7f51\u5361\u63a5\u5165\u5176\u4ed6 CNI \u7f51\u7edc\u3002 net1 \u7f51\u7edc\u7684\u7f51\u7edc\u5b9a\u4e49\u6765\u81ea\u4e8e multus-cni \u4e2d\u7684 NetworkAttachmentDefinition \u8d44\u6e90\u5b9a\u4e49\u3002 \u5f53 Pod \u521b\u5efa\u65f6\uff0c kube-ovn-controller \u4f1a\u76d1\u542c\u5230 Pod \u6dfb\u52a0\u4e8b\u4ef6\uff0c\u5e76\u6839\u636e Pod \u4e2d\u7684 annotation \u53bb\u5bfb\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u5e76\u4ece\u4e2d\u8fdb\u884c IP \u7684\u5206\u914d\u548c\u7ba1\u7406\uff0c \u5e76\u5c06 Pod \u6240\u5206\u914d\u5230\u7684\u5730\u5740\u4fe1\u606f\u5199\u56de\u5230 Pod annotation \u4e2d\u3002 \u5728\u5bb9\u5668\u6240\u5728\u673a\u5668\u7684 CNI \u53ef\u4ee5\u901a\u8fc7\u5728\u914d\u7f6e\u4e2d\u914d\u7f6e kube-ovn-cni \u4f5c\u4e3a ipam \u63d2\u4ef6, kube-ovn-cni \u5c06\u4f1a\u8bfb\u53d6 Pod annotation \u5e76\u5c06\u5730\u5740\u4fe1\u606f\u901a\u8fc7 CNI \u534f\u8bae\u7684\u6807\u51c6\u683c\u5f0f\u8fd4\u56de\u7ed9\u76f8\u5e94\u7684 CNI \u63d2\u4ef6\u3002 \u4f7f\u7528\u65b9\u6cd5 \u00b6 \u5b89\u88c5 Kube-OVN \u548c Multus \u00b6 \u8bf7\u53c2\u8003 Kube-OVN \u4e00\u952e\u5b89\u88c5 \u548c Multus how to use \u6765\u5b89\u88c5 Kube-OVN \u548c Multus-CNI\u3002 \u4e3a\u5176\u4ed6 CNI \u63d0\u4f9b IPAM \u00b6 \u6b64\u65f6\u4e3b\u7f51\u5361\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\uff0c\u9644\u5c5e\u7f51\u5361\u4e3a\u5176\u4ed6\u7c7b\u578b CNI\u3002 \u521b\u5efa NetworkAttachmentDefinition \u00b6 \u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 macvlan \u4f5c\u4e3a\u5bb9\u5668\u7f51\u7edc\u7684\u7b2c\u4e8c\u4e2a\u7f51\u7edc\uff0c\u5e76\u5c06\u5176 ipam \u8bbe\u7f6e\u4e3a kube-ovn \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : macvlan namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"macvlan.default\" } }' spec.config.ipam.type : \u9700\u8981\u4e3a kube-ovn \u6765\u8c03\u7528 kube-ovn \u7684\u63d2\u4ef6\u6765\u83b7\u53d6\u5730\u5740\u4fe1\u606f\u3002 server_socket : Kube-OVN \u901a\u4fe1\u4f7f\u7528\u7684 socket \u6587\u4ef6\u3002 \u9ed8\u8ba4\u4f4d\u7f6e\u4e3a /run/openvswitch/kube-ovn-daemon.sock \u3002 provider : \u5f53\u524d NetworkAttachmentDefinition \u7684 <name>.<namespace> , Kube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90\u3002 \u9644\u5c5e\u7f51\u5361\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361 \u00b6 \u6b64\u65f6\u591a\u5757\u7f51\u5361\u5747\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\u3002 \u521b\u5efa NetworkAttachmentDefinition \u00b6 \u5c06 provider \u7684\u540e\u7f00\u8bbe\u7f6e\u4e3a ovn \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" }' spec.config.type : \u8bbe\u7f6e\u4e3a kube-ovn \u6765\u89e6\u53d1 CNI \u63d2\u4ef6\u4f7f\u7528 Kube-OVN \u5b50\u7f51\u3002 server_socket : Kube-OVN \u901a\u4fe1\u4f7f\u7528\u7684 socket \u6587\u4ef6\u3002 \u9ed8\u8ba4\u4f4d\u7f6e\u4e3a /run/openvswitch/kube-ovn-daemon.sock \u3002 provider : \u5f53\u524d NetworkAttachmentDefinition \u7684 <name>.<namespace>.ovn , Kube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90\uff0c\u6ce8\u610f\u540e\u7f00\u9700\u8981\u8bbe\u7f6e\u4e3a ovn\u3002 \u521b\u5efa\u4e00\u4e2a Kube-OVN Subnet \u00b6 \u521b\u5efa\u4e00\u4e2a Kube-OVN Subnet,\u8bbe\u7f6e\u5bf9\u5e94\u7684 cidrBlock \u548c exclude_ips , provider \u5e94\u8be5\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <name>.<namespace> , \u4f8b\u5982\u7528 macvlan \u63d0\u4f9b\u9644\u52a0\u7f51\u5361\uff0c\u521b\u5efa Subnet \u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : macvlan spec : protocol : IPv4 provider : macvlan.default cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 gateway , private , nat \u53ea\u5bf9 provider \u7c7b\u578b\u4e3a ovn \u7684\u7f51\u7edc\u751f\u6548\uff0c\u4e0d\u9002\u7528\u4e8e attachment network\u3002 \u5982\u679c\u4ee5 Kube-OVN \u4f5c\u4e3a\u9644\u52a0\u7f51\u5361\uff0c\u5219 provider \u5e94\u8be5\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <name>.<namespace>.ovn \uff0c\u5e76\u8981\u4ee5 ovn \u4f5c\u4e3a\u540e\u7f00\u7ed3\u675f\u3002 \u7528 Kube-OVN \u63d0\u4f9b\u9644\u52a0\u7f51\u5361\uff0c\u521b\u5efa Subnet \u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attachnet spec : protocol : IPv4 provider : attachnet.default.ovn cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 \u521b\u5efa\u4e00\u4e2a\u591a\u7f51\u7edc\u7684 Pod \u00b6 \u5bf9\u4e8e\u5730\u5740\u968f\u673a\u5206\u914d\u7684 Pod\uff0c\u53ea\u9700\u8981\u6dfb\u52a0\u5982\u4e0b annotation k8s.v1.cni.cncf.io/networks ,\u53d6\u503c\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <namespace>/<name> \uff1a apiVersion : v1 kind : Pod metadata : name : samplepod namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan spec : containers : - name : samplepod command : [ \"/bin/ash\" , \"-c\" , \"trap : TERM INT; sleep infinity & wait\" ] image : docker.io/library/alpine:edge \u521b\u5efa\u56fa\u5b9a IP \u7684 Pod \u00b6 \u5bf9\u4e8e\u56fa\u5b9a IP \u7684 Pod\uff0c\u6dfb\u52a0 <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_address annotation\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 macvlan.default.kubernetes.io/ip_address : 172.17.0.100 macvlan.default.kubernetes.io/mac_address : 00:00:00:53:6B:BB spec : containers : - name : static-ip image : docker.io/library/nginx:alpine \u521b\u5efa\u4f7f\u7528\u56fa\u5b9a IP \u7684\u5de5\u4f5c\u8d1f\u8f7d \u00b6 \u5bf9\u4e8e\u4f7f\u7528 ippool \u7684\u5de5\u4f5c\u8d1f\u8f7d, \u6dfb\u52a0 <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_pool annotations: apiVersion : apps/v1 kind : Deployment metadata : namespace : default name : static-workload labels : app : static-workload spec : replicas : 2 selector : matchLabels : app : static-workload template : metadata : labels : app : static-workload annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 macvlan.default.kubernetes.io/ip_pool : 172.17.0.200,172.17.0.201,172.17.0.202 spec : containers : - name : static-workload image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u591a\u7f51\u5361\u7ba1\u7406"},{"location":"advance/multi-nic/#_1","text":"Kube-OVN \u53ef\u4ee5\u4e3a\u5176\u4ed6 CNI \u7f51\u7edc\u63d2\u4ef6\uff0c\u4f8b\u5982 macvlan\u3001vlan\u3001host-device \u7b49\u63d2\u4ef6\u63d0\u4f9b\u96c6\u7fa4\u7ea7\u522b\u7684 IPAM \u80fd\u529b\uff0c \u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4e5f\u53ef\u4ee5\u4f7f\u7528\u5230 Kube-OVN \u4e2d\u5b50\u7f51\u4ee5\u53ca\u56fa\u5b9a IP \u529f\u80fd\u3002 \u540c\u65f6 Kube-OVN \u4e5f\u652f\u6301\u591a\u5757\u7f51\u5361\u5747\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\u60c5\u51b5\u4e0b\u7684\u5730\u5740\u7ba1\u7406\u3002","title":"\u591a\u7f51\u5361\u7ba1\u7406"},{"location":"advance/multi-nic/#_2","text":"\u901a\u8fc7\u4f7f\u7528 Multus CNI , \u6211\u4eec\u53ef\u4ee5\u7ed9\u4e00\u4e2a Pod \u6dfb\u52a0\u591a\u5757\u4e0d\u540c\u7f51\u7edc\u7684\u7f51\u5361\u3002 \u7136\u800c\u6211\u4eec\u4ecd\u7136\u7f3a\u4e4f\u5bf9\u96c6\u7fa4\u8303\u56f4\u5185\u4e0d\u540c\u7f51\u7edc\u7684 IP \u5730\u5740\u8fdb\u884c\u7ba1\u7406\u7684\u80fd\u529b\u3002\u5728 Kube-OVN \u4e2d\uff0c\u6211\u4eec\u5df2\u7ecf\u80fd\u591f\u901a\u8fc7 Subnet \u548c IP \u7684 CRD \u6765\u8fdb\u884c IP \u7684\u9ad8\u7ea7\u7ba1\u7406\uff0c \u4f8b\u5982\u5b50\u7f51\u7ba1\u7406\uff0cIP \u9884\u7559\uff0c\u968f\u673a\u5206\u914d\uff0c\u56fa\u5b9a\u5206\u914d\u7b49\u3002\u73b0\u5728\u6211\u4eec\u5bf9\u5b50\u7f51\u8fdb\u884c\u6269\u5c55\uff0c\u6765\u63a5\u5165\u5176\u4ed6\u4e0d\u540c\u7684\u7f51\u7edc\u63d2\u4ef6\uff0c\u4f7f\u5f97\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4e5f\u53ef\u4ee5\u4f7f\u7528 Kube-OVN \u7684 IPAM \u529f\u80fd\u3002","title":"\u5de5\u4f5c\u539f\u7406"},{"location":"advance/multi-nic/#_3","text":"\u4e0a\u56fe\u5c55\u793a\u4e86\u5982\u4f55\u901a\u8fc7 Kube-OVN \u6765\u7ba1\u7406\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u7684 IP \u5730\u5740\u3002\u5176\u4e2d\u5bb9\u5668\u7684 eth0 \u7f51\u5361\u63a5\u5165 OVN \u7f51\u7edc\uff0cnet1 \u7f51\u5361\u63a5\u5165\u5176\u4ed6 CNI \u7f51\u7edc\u3002 net1 \u7f51\u7edc\u7684\u7f51\u7edc\u5b9a\u4e49\u6765\u81ea\u4e8e multus-cni \u4e2d\u7684 NetworkAttachmentDefinition \u8d44\u6e90\u5b9a\u4e49\u3002 \u5f53 Pod \u521b\u5efa\u65f6\uff0c kube-ovn-controller \u4f1a\u76d1\u542c\u5230 Pod \u6dfb\u52a0\u4e8b\u4ef6\uff0c\u5e76\u6839\u636e Pod \u4e2d\u7684 annotation \u53bb\u5bfb\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u5e76\u4ece\u4e2d\u8fdb\u884c IP \u7684\u5206\u914d\u548c\u7ba1\u7406\uff0c \u5e76\u5c06 Pod \u6240\u5206\u914d\u5230\u7684\u5730\u5740\u4fe1\u606f\u5199\u56de\u5230 Pod annotation \u4e2d\u3002 \u5728\u5bb9\u5668\u6240\u5728\u673a\u5668\u7684 CNI \u53ef\u4ee5\u901a\u8fc7\u5728\u914d\u7f6e\u4e2d\u914d\u7f6e kube-ovn-cni \u4f5c\u4e3a ipam \u63d2\u4ef6, kube-ovn-cni \u5c06\u4f1a\u8bfb\u53d6 Pod annotation \u5e76\u5c06\u5730\u5740\u4fe1\u606f\u901a\u8fc7 CNI \u534f\u8bae\u7684\u6807\u51c6\u683c\u5f0f\u8fd4\u56de\u7ed9\u76f8\u5e94\u7684 CNI \u63d2\u4ef6\u3002","title":"\u5de5\u4f5c\u6d41\u7a0b"},{"location":"advance/multi-nic/#_4","text":"","title":"\u4f7f\u7528\u65b9\u6cd5"},{"location":"advance/multi-nic/#kube-ovn-multus","text":"\u8bf7\u53c2\u8003 Kube-OVN \u4e00\u952e\u5b89\u88c5 \u548c Multus how to use \u6765\u5b89\u88c5 Kube-OVN \u548c Multus-CNI\u3002","title":"\u5b89\u88c5 Kube-OVN \u548c Multus"},{"location":"advance/multi-nic/#cni-ipam","text":"\u6b64\u65f6\u4e3b\u7f51\u5361\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\uff0c\u9644\u5c5e\u7f51\u5361\u4e3a\u5176\u4ed6\u7c7b\u578b CNI\u3002","title":"\u4e3a\u5176\u4ed6 CNI \u63d0\u4f9b IPAM"},{"location":"advance/multi-nic/#networkattachmentdefinition","text":"\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 macvlan \u4f5c\u4e3a\u5bb9\u5668\u7f51\u7edc\u7684\u7b2c\u4e8c\u4e2a\u7f51\u7edc\uff0c\u5e76\u5c06\u5176 ipam \u8bbe\u7f6e\u4e3a kube-ovn \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : macvlan namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"macvlan.default\" } }' spec.config.ipam.type : \u9700\u8981\u4e3a kube-ovn \u6765\u8c03\u7528 kube-ovn \u7684\u63d2\u4ef6\u6765\u83b7\u53d6\u5730\u5740\u4fe1\u606f\u3002 server_socket : Kube-OVN \u901a\u4fe1\u4f7f\u7528\u7684 socket \u6587\u4ef6\u3002 \u9ed8\u8ba4\u4f4d\u7f6e\u4e3a /run/openvswitch/kube-ovn-daemon.sock \u3002 provider : \u5f53\u524d NetworkAttachmentDefinition \u7684 <name>.<namespace> , Kube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90\u3002","title":"\u521b\u5efa NetworkAttachmentDefinition"},{"location":"advance/multi-nic/#kube-ovn","text":"\u6b64\u65f6\u591a\u5757\u7f51\u5361\u5747\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361\u3002","title":"\u9644\u5c5e\u7f51\u5361\u4e3a Kube-OVN \u7c7b\u578b\u7f51\u5361"},{"location":"advance/multi-nic/#networkattachmentdefinition_1","text":"\u5c06 provider \u7684\u540e\u7f00\u8bbe\u7f6e\u4e3a ovn \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" }' spec.config.type : \u8bbe\u7f6e\u4e3a kube-ovn \u6765\u89e6\u53d1 CNI \u63d2\u4ef6\u4f7f\u7528 Kube-OVN \u5b50\u7f51\u3002 server_socket : Kube-OVN \u901a\u4fe1\u4f7f\u7528\u7684 socket \u6587\u4ef6\u3002 \u9ed8\u8ba4\u4f4d\u7f6e\u4e3a /run/openvswitch/kube-ovn-daemon.sock \u3002 provider : \u5f53\u524d NetworkAttachmentDefinition \u7684 <name>.<namespace>.ovn , Kube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90\uff0c\u6ce8\u610f\u540e\u7f00\u9700\u8981\u8bbe\u7f6e\u4e3a ovn\u3002","title":"\u521b\u5efa NetworkAttachmentDefinition"},{"location":"advance/multi-nic/#kube-ovn-subnet","text":"\u521b\u5efa\u4e00\u4e2a Kube-OVN Subnet,\u8bbe\u7f6e\u5bf9\u5e94\u7684 cidrBlock \u548c exclude_ips , provider \u5e94\u8be5\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <name>.<namespace> , \u4f8b\u5982\u7528 macvlan \u63d0\u4f9b\u9644\u52a0\u7f51\u5361\uff0c\u521b\u5efa Subnet \u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : macvlan spec : protocol : IPv4 provider : macvlan.default cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 gateway , private , nat \u53ea\u5bf9 provider \u7c7b\u578b\u4e3a ovn \u7684\u7f51\u7edc\u751f\u6548\uff0c\u4e0d\u9002\u7528\u4e8e attachment network\u3002 \u5982\u679c\u4ee5 Kube-OVN \u4f5c\u4e3a\u9644\u52a0\u7f51\u5361\uff0c\u5219 provider \u5e94\u8be5\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <name>.<namespace>.ovn \uff0c\u5e76\u8981\u4ee5 ovn \u4f5c\u4e3a\u540e\u7f00\u7ed3\u675f\u3002 \u7528 Kube-OVN \u63d0\u4f9b\u9644\u52a0\u7f51\u5361\uff0c\u521b\u5efa Subnet \u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attachnet spec : protocol : IPv4 provider : attachnet.default.ovn cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10","title":"\u521b\u5efa\u4e00\u4e2a Kube-OVN Subnet"},{"location":"advance/multi-nic/#pod","text":"\u5bf9\u4e8e\u5730\u5740\u968f\u673a\u5206\u914d\u7684 Pod\uff0c\u53ea\u9700\u8981\u6dfb\u52a0\u5982\u4e0b annotation k8s.v1.cni.cncf.io/networks ,\u53d6\u503c\u4e3a\u5bf9\u5e94\u7684 NetworkAttachmentDefinition \u7684 <namespace>/<name> \uff1a apiVersion : v1 kind : Pod metadata : name : samplepod namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan spec : containers : - name : samplepod command : [ \"/bin/ash\" , \"-c\" , \"trap : TERM INT; sleep infinity & wait\" ] image : docker.io/library/alpine:edge","title":"\u521b\u5efa\u4e00\u4e2a\u591a\u7f51\u7edc\u7684 Pod"},{"location":"advance/multi-nic/#ip-pod","text":"\u5bf9\u4e8e\u56fa\u5b9a IP \u7684 Pod\uff0c\u6dfb\u52a0 <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_address annotation\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 macvlan.default.kubernetes.io/ip_address : 172.17.0.100 macvlan.default.kubernetes.io/mac_address : 00:00:00:53:6B:BB spec : containers : - name : static-ip image : docker.io/library/nginx:alpine","title":"\u521b\u5efa\u56fa\u5b9a IP \u7684 Pod"},{"location":"advance/multi-nic/#ip","text":"\u5bf9\u4e8e\u4f7f\u7528 ippool \u7684\u5de5\u4f5c\u8d1f\u8f7d, \u6dfb\u52a0 <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_pool annotations: apiVersion : apps/v1 kind : Deployment metadata : namespace : default name : static-workload labels : app : static-workload spec : replicas : 2 selector : matchLabels : app : static-workload template : metadata : labels : app : static-workload annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 macvlan.default.kubernetes.io/ip_pool : 172.17.0.200,172.17.0.201,172.17.0.202 spec : containers : - name : static-workload image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u521b\u5efa\u4f7f\u7528\u56fa\u5b9a IP \u7684\u5de5\u4f5c\u8d1f\u8f7d"},{"location":"advance/nat-policy-rule/","text":"NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd \u00b6 NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd\u7528\u9014 \u00b6 \u9ed8\u8ba4 VPC \u4e0b\u7684 Overlay \u5b50\u7f51\uff0c\u6253\u5f00 natOutgoing \u5f00\u5173\u65f6\uff0csubnet \u4e0b\u7684\u6240\u6709 Pod \u8bbf\u95ee\u5916\u7f51\u90fd\u9700\u8981\u505a SNAT \u6210\u5f53\u524d\u8282\u70b9\u7684 IP\uff0c\u4f46\u662f\u6709\u4e9b\u573a\u666f\u6211\u4eec\u5e76\u4e0d\u5e0c\u671b\u5b50\u7f51\u5185\u6240\u6709 Pod \u8bbf\u95ee\u5916\u7f51\u90fd\u505a SNAT\u3002 \u6240\u4ee5 NAT \u7b56\u7565\u5c31\u662f\u4e3a\u4e86\u63d0\u4f9b\u4e00\u4e2a\u63a5\u53e3\u8ba9\u7528\u6237\u51b3\u5b9a\u5b50\u7f51\u5185\u7684\u54ea\u4e9b CIDR \u6216\u8005 IP \u8bbf\u95ee\u5916\u7f51\u505a SNAT\u3002 NAT \u7b56\u7565\u89c4\u5219\u4f7f\u7528\u65b9\u6cd5 \u00b6 \u5728 subnet.Spec \u4e2d\u5f00\u542f natOutgoing \u5f00\u5173\uff0c \u5e76\u4e14\u6dfb\u52a0\u5b57\u6bb5 natOutgoingPolicyRules \u5982\u4e0b\uff1a spec : natOutgoing : true natOutgoingPolicyRules : - action : forward match : srcIPs : 10.0.11.0/30,10.0.11.254 - action : nat match : srcIPs : 10.0.11.128/26 dstIPs : 114.114.114.114,8.8.8.8 \u4ee5\u4e0a\u6848\u4f8b\u8868\u793a\u6709\u4e24\u6761 NAT \u7b56\u7565\u89c4\u5219\uff1a \u6e90 IP \u662f 10.0.11.0/30 \u6216\u8005 10.0.11.254 \u7684\u62a5\u6587\u8bbf\u95ee\u5916\u7f51\u65f6\u4e0d\u4f1a\u505a SNAT\u3002 \u6e90 IP \u662f 10.0.11.128/26 \u5e76\u4e14\u76ee\u7684 IP \u662f 114.114.114.114 \u6216\u8005 8.8.8.8 \u7684\u62a5\u6587\u8bbf\u95ee\u5916\u7f51\u65f6\u4f1a\u505a SNAT\u3002 \u5b57\u6bb5\u63cf\u8ff0\uff1a action\uff1a\u8868\u793a\u6ee1\u8db3 match \u5bf9\u5e94\u6761\u4ef6\u7684\u62a5\u6587\uff0c\u4f1a\u6267\u884c\u7684 action, action \u5206\u4e3a\u4e24\u79cd forward \u548c nat \uff0c forward \u8868\u793a\u62a5\u6587\u51fa\u5916\u7f51\u4e0d\u505a SNAT, nat \u8868\u793a\u62a5\u6587\u51fa\u5916\u7f51\u505a SNAT\u3002 \u6ca1\u6709\u914d\u7f6e natOutgoingPolicyRules \u65f6\uff0c\u9ed8\u8ba4\u60c5\u51b5\u62a5\u6587\u4ecd\u7136\u662f\u505a SNAT\u3002 match\uff1a\u8868\u793a\u62a5\u6587\u7684\u5339\u914d\u6bb5\uff0c\u5339\u914d\u6bb5\u6709 srcIPs \u548c dstIPs\uff0c \u8fd9\u91cc\u8868\u793a\u4ece\u5b50\u7f51\u5185\u5230\u5916\u7f51\u65b9\u5411\u4e0a\u7684\u62a5\u6587\u7684\u6e90 IP \u548c \u76ee\u7684 IP\u3002 match.srcIPs \u548c match.dstIPs \u652f\u6301\u591a\u4e2a cidr \u548c ip\uff0c\u4e4b\u95f4\u7528\u9017\u53f7\u9694\u5f00\u5373\u53ef\u3002 \u5982\u679c\u51fa\u73b0\u51e0\u4e2a match \u91cd\u590d\u4f46\u662f action \u4e0d\u4e00\u6837\uff0c\u6309\u7167 natOutgoingPolicyRules \u7684\u6570\u7ec4\u4f4d\u7f6e\u4e3a\u51c6\uff0c\u6570\u7ec4\u7d22\u5f15\u8d8a\u4f4e\u7684\u4f18\u5148\u7ea7\u8d8a\u9ad8\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd"},{"location":"advance/nat-policy-rule/#nat","text":"","title":"NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd"},{"location":"advance/nat-policy-rule/#nat_1","text":"\u9ed8\u8ba4 VPC \u4e0b\u7684 Overlay \u5b50\u7f51\uff0c\u6253\u5f00 natOutgoing \u5f00\u5173\u65f6\uff0csubnet \u4e0b\u7684\u6240\u6709 Pod \u8bbf\u95ee\u5916\u7f51\u90fd\u9700\u8981\u505a SNAT \u6210\u5f53\u524d\u8282\u70b9\u7684 IP\uff0c\u4f46\u662f\u6709\u4e9b\u573a\u666f\u6211\u4eec\u5e76\u4e0d\u5e0c\u671b\u5b50\u7f51\u5185\u6240\u6709 Pod \u8bbf\u95ee\u5916\u7f51\u90fd\u505a SNAT\u3002 \u6240\u4ee5 NAT \u7b56\u7565\u5c31\u662f\u4e3a\u4e86\u63d0\u4f9b\u4e00\u4e2a\u63a5\u53e3\u8ba9\u7528\u6237\u51b3\u5b9a\u5b50\u7f51\u5185\u7684\u54ea\u4e9b CIDR \u6216\u8005 IP \u8bbf\u95ee\u5916\u7f51\u505a SNAT\u3002","title":"NAT \u7b56\u7565\u89c4\u5219\u529f\u80fd\u7528\u9014"},{"location":"advance/nat-policy-rule/#nat_2","text":"\u5728 subnet.Spec \u4e2d\u5f00\u542f natOutgoing \u5f00\u5173\uff0c \u5e76\u4e14\u6dfb\u52a0\u5b57\u6bb5 natOutgoingPolicyRules \u5982\u4e0b\uff1a spec : natOutgoing : true natOutgoingPolicyRules : - action : forward match : srcIPs : 10.0.11.0/30,10.0.11.254 - action : nat match : srcIPs : 10.0.11.128/26 dstIPs : 114.114.114.114,8.8.8.8 \u4ee5\u4e0a\u6848\u4f8b\u8868\u793a\u6709\u4e24\u6761 NAT \u7b56\u7565\u89c4\u5219\uff1a \u6e90 IP \u662f 10.0.11.0/30 \u6216\u8005 10.0.11.254 \u7684\u62a5\u6587\u8bbf\u95ee\u5916\u7f51\u65f6\u4e0d\u4f1a\u505a SNAT\u3002 \u6e90 IP \u662f 10.0.11.128/26 \u5e76\u4e14\u76ee\u7684 IP \u662f 114.114.114.114 \u6216\u8005 8.8.8.8 \u7684\u62a5\u6587\u8bbf\u95ee\u5916\u7f51\u65f6\u4f1a\u505a SNAT\u3002 \u5b57\u6bb5\u63cf\u8ff0\uff1a action\uff1a\u8868\u793a\u6ee1\u8db3 match \u5bf9\u5e94\u6761\u4ef6\u7684\u62a5\u6587\uff0c\u4f1a\u6267\u884c\u7684 action, action \u5206\u4e3a\u4e24\u79cd forward \u548c nat \uff0c forward \u8868\u793a\u62a5\u6587\u51fa\u5916\u7f51\u4e0d\u505a SNAT, nat \u8868\u793a\u62a5\u6587\u51fa\u5916\u7f51\u505a SNAT\u3002 \u6ca1\u6709\u914d\u7f6e natOutgoingPolicyRules \u65f6\uff0c\u9ed8\u8ba4\u60c5\u51b5\u62a5\u6587\u4ecd\u7136\u662f\u505a SNAT\u3002 match\uff1a\u8868\u793a\u62a5\u6587\u7684\u5339\u914d\u6bb5\uff0c\u5339\u914d\u6bb5\u6709 srcIPs \u548c dstIPs\uff0c \u8fd9\u91cc\u8868\u793a\u4ece\u5b50\u7f51\u5185\u5230\u5916\u7f51\u65b9\u5411\u4e0a\u7684\u62a5\u6587\u7684\u6e90 IP \u548c \u76ee\u7684 IP\u3002 match.srcIPs \u548c match.dstIPs \u652f\u6301\u591a\u4e2a cidr \u548c ip\uff0c\u4e4b\u95f4\u7528\u9017\u53f7\u9694\u5f00\u5373\u53ef\u3002 \u5982\u679c\u51fa\u73b0\u51e0\u4e2a match \u91cd\u590d\u4f46\u662f action \u4e0d\u4e00\u6837\uff0c\u6309\u7167 natOutgoingPolicyRules \u7684\u6570\u7ec4\u4f4d\u7f6e\u4e3a\u51c6\uff0c\u6570\u7ec4\u7d22\u5f15\u8d8a\u4f4e\u7684\u4f18\u5148\u7ea7\u8d8a\u9ad8\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NAT \u7b56\u7565\u89c4\u5219\u4f7f\u7528\u65b9\u6cd5"},{"location":"advance/node-local-dns/","text":"\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u548c Kube-OVN \u9002\u914d \u00b6 NodeLocal DNSCache \u662f\u901a\u8fc7\u96c6\u7fa4\u8282\u70b9\u4e0a\u4f5c\u4e3a DaemonSet \u8fd0\u884c DNS \u7f13\u5b58\u6765\u63d0\u9ad8\u96c6\u7fa4 DNS \u6027\u80fd\uff0c\u8be5\u529f\u80fd\u4e5f\u53ef\u4ee5\u548c Kube-OVN \u9002\u914d\u3002 \u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u90e8\u7f72 \u00b6 \u90e8\u7f72 Kubernetes \u7684\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58 \u00b6 \u8be5\u6b65\u9aa4\u53c2\u8003 Kubernetes \u5b98\u7f51\u914d\u7f6e Nodelocaldnscache \u3002 \u4f7f\u7528\u4ee5\u4e0b\u811a\u672c\u90e8\u7f72\uff1a #!bin/bash localdns = 169 .254.20.10 domain = cluster.local kubedns = 10 .96.0.10 wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml sed -i \"s/__PILLAR__LOCAL__DNS__/ $localdns /g; s/__PILLAR__DNS__DOMAIN__/ $domain /g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/ $kubedns /g\" nodelocaldns.yaml kubectl apply -f nodelocaldns.yaml \u4fee\u6539\u6bcf\u4e2a\u8282\u70b9\u4e0a\u7684 kubelet \u914d\u7f6e\u6587\u4ef6\uff0c\u5c06 /var/lib/kubelet/config.yaml \u4e2d\u7684 clusterDNS \u5b57\u6bb5\u4fee\u6539\u4e3a\u672c\u5730 DNS IP 169.254.20.10\uff0c\u7136\u540e\u91cd\u542f kubelet \u670d\u52a1\u3002 Kube-OVN \u76f8\u5e94 DNS \u914d\u7f6e \u00b6 \u90e8\u7f72\u597d Kubernetes \u7684 Nodelocal DNScache \u7ec4\u4ef6\u540e\uff0c Kube-OVN \u9700\u8981\u505a\u51fa\u4e0b\u9762\u4fee\u6539\uff1a Underlay Subnet \u5f00\u542f U2O \u5f00\u5173 \u00b6 \u5982\u679c\u662f Underlay Subnet \u9700\u8981\u4f7f\u7528\u672c\u5730 DNS \u529f\u80fd\uff0c\u9700\u8981\u5f00\u542f U2O \u529f\u80fd\uff0c\u5373\u5728 kubectl edit subnet {your subnet} \u4e2d\u914d\u7f6e spec.u2oInterconnection = true , \u5982\u679c\u662f Overlay Subnet \u5219\u4e0d\u9700\u8981\u8fd9\u6b65\u64cd\u4f5c\u3002 \u7ed9 Kube-ovn-controller \u6307\u5b9a\u5bf9\u5e94\u7684\u672c\u5730 DNS IP \u00b6 kubectl edit deployment kube-ovn-controller -n kube-system \u7ed9 spec.template.spec.containers.args \u589e\u52a0\u5b57\u6bb5 --node-local-dns-ip=169.254.20.10 \u91cd\u5efa\u5df2\u7ecf\u521b\u5efa\u7684 Pod \u00b6 \u8fd9\u6b65\u539f\u56e0\u662f\u8ba9 Pod \u91cd\u65b0\u751f\u6210 /etc/resolv.conf \u8ba9 nameserver \u6307\u5411\u672c\u5730 DNS IP\uff0c\u5982\u679c\u6ca1\u6709\u91cd\u5efa Pod \u7684 nameserver \u5c06\u4ecd\u7136\u4f7f\u7528\u96c6\u7fa4\u7684 DNS ClusterIP\u3002\u540c\u65f6 u2o \u5f00\u5173\u5982\u679c\u5f00\u542f\u4e5f\u9700\u8981\u91cd\u5efa Pod \u6765\u91cd\u65b0\u751f\u6210 Pod \u7f51\u5173\u3002 \u9a8c\u8bc1\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u529f\u80fd \u00b6 \u4ee5\u4e0a\u914d\u7f6e\u5b8c\u6210\u540e\u53ef\u4ee5\u627e\u5230 Pod \u9a8c\u8bc1\u5982\u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u7684 DNS \u670d\u52a1\u5668\u662f\u6307\u5411\u672c\u5730 169.254.20.10 \uff0c\u5e76\u6210\u529f\u89e3\u6790\u57df\u540d\uff1a # kubectl exec -it pod1 -- nslookup github.com Server: 169 .254.20.10 Address: 169 .254.20.10:53 Name: github.com Address: 20 .205.243.166 \u4e5f\u53ef\u4ee5\u5728\u8282\u70b9\u6293\u5305\u9a8c\u8bc1\u5982\u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 DNS \u67e5\u8be2\u62a5\u6587\u901a\u8fc7 ovn0 \u7f51\u5361\u5230\u8fbe\u672c\u5730\u7684 DNS \u670d\u52a1\uff0cDNS \u54cd\u5e94\u62a5\u6587\u539f\u8def\u8fd4\u56de: # tcpdump -i any port 53 06 :20:00.441889 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441889 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441950 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.441950 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.442203 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442219 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442273 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) 06 :20:00.442278 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u548c Kube-OVN \u9002\u914d"},{"location":"advance/node-local-dns/#dns-kube-ovn","text":"NodeLocal DNSCache \u662f\u901a\u8fc7\u96c6\u7fa4\u8282\u70b9\u4e0a\u4f5c\u4e3a DaemonSet \u8fd0\u884c DNS \u7f13\u5b58\u6765\u63d0\u9ad8\u96c6\u7fa4 DNS \u6027\u80fd\uff0c\u8be5\u529f\u80fd\u4e5f\u53ef\u4ee5\u548c Kube-OVN \u9002\u914d\u3002","title":"\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u548c Kube-OVN \u9002\u914d"},{"location":"advance/node-local-dns/#dns","text":"","title":"\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u90e8\u7f72"},{"location":"advance/node-local-dns/#kubernetes-dns","text":"\u8be5\u6b65\u9aa4\u53c2\u8003 Kubernetes \u5b98\u7f51\u914d\u7f6e Nodelocaldnscache \u3002 \u4f7f\u7528\u4ee5\u4e0b\u811a\u672c\u90e8\u7f72\uff1a #!bin/bash localdns = 169 .254.20.10 domain = cluster.local kubedns = 10 .96.0.10 wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml sed -i \"s/__PILLAR__LOCAL__DNS__/ $localdns /g; s/__PILLAR__DNS__DOMAIN__/ $domain /g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/ $kubedns /g\" nodelocaldns.yaml kubectl apply -f nodelocaldns.yaml \u4fee\u6539\u6bcf\u4e2a\u8282\u70b9\u4e0a\u7684 kubelet \u914d\u7f6e\u6587\u4ef6\uff0c\u5c06 /var/lib/kubelet/config.yaml \u4e2d\u7684 clusterDNS \u5b57\u6bb5\u4fee\u6539\u4e3a\u672c\u5730 DNS IP 169.254.20.10\uff0c\u7136\u540e\u91cd\u542f kubelet \u670d\u52a1\u3002","title":"\u90e8\u7f72 Kubernetes \u7684\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58"},{"location":"advance/node-local-dns/#kube-ovn-dns","text":"\u90e8\u7f72\u597d Kubernetes \u7684 Nodelocal DNScache \u7ec4\u4ef6\u540e\uff0c Kube-OVN \u9700\u8981\u505a\u51fa\u4e0b\u9762\u4fee\u6539\uff1a","title":"Kube-OVN \u76f8\u5e94 DNS \u914d\u7f6e"},{"location":"advance/node-local-dns/#underlay-subnet-u2o","text":"\u5982\u679c\u662f Underlay Subnet \u9700\u8981\u4f7f\u7528\u672c\u5730 DNS \u529f\u80fd\uff0c\u9700\u8981\u5f00\u542f U2O \u529f\u80fd\uff0c\u5373\u5728 kubectl edit subnet {your subnet} \u4e2d\u914d\u7f6e spec.u2oInterconnection = true , \u5982\u679c\u662f Overlay Subnet \u5219\u4e0d\u9700\u8981\u8fd9\u6b65\u64cd\u4f5c\u3002","title":"Underlay Subnet \u5f00\u542f U2O \u5f00\u5173"},{"location":"advance/node-local-dns/#kube-ovn-controller-dns-ip","text":"kubectl edit deployment kube-ovn-controller -n kube-system \u7ed9 spec.template.spec.containers.args \u589e\u52a0\u5b57\u6bb5 --node-local-dns-ip=169.254.20.10","title":"\u7ed9 Kube-ovn-controller \u6307\u5b9a\u5bf9\u5e94\u7684\u672c\u5730 DNS IP"},{"location":"advance/node-local-dns/#pod","text":"\u8fd9\u6b65\u539f\u56e0\u662f\u8ba9 Pod \u91cd\u65b0\u751f\u6210 /etc/resolv.conf \u8ba9 nameserver \u6307\u5411\u672c\u5730 DNS IP\uff0c\u5982\u679c\u6ca1\u6709\u91cd\u5efa Pod \u7684 nameserver \u5c06\u4ecd\u7136\u4f7f\u7528\u96c6\u7fa4\u7684 DNS ClusterIP\u3002\u540c\u65f6 u2o \u5f00\u5173\u5982\u679c\u5f00\u542f\u4e5f\u9700\u8981\u91cd\u5efa Pod \u6765\u91cd\u65b0\u751f\u6210 Pod \u7f51\u5173\u3002","title":"\u91cd\u5efa\u5df2\u7ecf\u521b\u5efa\u7684 Pod"},{"location":"advance/node-local-dns/#dns_1","text":"\u4ee5\u4e0a\u914d\u7f6e\u5b8c\u6210\u540e\u53ef\u4ee5\u627e\u5230 Pod \u9a8c\u8bc1\u5982\u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u7684 DNS \u670d\u52a1\u5668\u662f\u6307\u5411\u672c\u5730 169.254.20.10 \uff0c\u5e76\u6210\u529f\u89e3\u6790\u57df\u540d\uff1a # kubectl exec -it pod1 -- nslookup github.com Server: 169 .254.20.10 Address: 169 .254.20.10:53 Name: github.com Address: 20 .205.243.166 \u4e5f\u53ef\u4ee5\u5728\u8282\u70b9\u6293\u5305\u9a8c\u8bc1\u5982\u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 DNS \u67e5\u8be2\u62a5\u6587\u901a\u8fc7 ovn0 \u7f51\u5361\u5230\u8fbe\u672c\u5730\u7684 DNS \u670d\u52a1\uff0cDNS \u54cd\u5e94\u62a5\u6587\u539f\u8def\u8fd4\u56de: # tcpdump -i any port 53 06 :20:00.441889 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441889 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441950 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.441950 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.442203 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442219 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442273 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) 06 :20:00.442278 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9a8c\u8bc1\u8282\u70b9\u672c\u5730 DNS \u7f13\u5b58\u529f\u80fd"},{"location":"advance/offload-corigine/","text":"\u82af\u542f\u6e90\u7f51\u5361 Offload \u652f\u6301 \u00b6 Kube-OVN \u5728\u6700\u7ec8\u7684\u6570\u636e\u5e73\u9762\u4f7f\u7528 OVS \u6765\u5b8c\u6210\u6d41\u91cf\u8f6c\u53d1\uff0c\u76f8\u5173\u7684\u6d41\u8868\u5339\u914d\uff0c\u96a7\u9053\u5c01\u88c5\u7b49\u529f\u80fd\u4e3a CPU \u5bc6\u96c6\u578b\uff0c\u5728\u5927\u6d41\u91cf\u4e0b\u4f1a\u6d88\u8017\u5927\u91cf CPU \u8d44\u6e90\u5e76\u5bfc\u81f4 \u5ef6\u8fdf\u4e0a\u5347\u548c\u541e\u5410\u91cf\u4e0b\u964d\u3002\u82af\u542f\u6e90\u7684 Agilio CX \u7cfb\u5217\u667a\u80fd\u7f51\u5361\u53ef\u4ee5\u5c06 OVS \u76f8\u5173\u7684\u64cd\u4f5c\u5378\u8f7d\u5230\u786c\u4ef6\u7f51\u5361\u4e2d\u6267\u884c\u3002 \u8be5\u6280\u672f\u53ef\u4ee5\u5728\u65e0\u9700\u5bf9 OVS \u63a7\u5236\u5e73\u9762\u8fdb\u884c\u4fee\u6539\u7684\u60c5\u51b5\u4e0b\uff0c\u7f29\u77ed\u6570\u636e\u8def\u5f84\uff0c\u907f\u514d\u5bf9\u4e3b\u673a CPU \u8d44\u6e90\u7684\u4f7f\u7528\uff0c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u663e\u8457\u63d0\u5347\u541e\u5410\u91cf\u3002 \u524d\u7f6e\u6761\u4ef6 \u00b6 \u82af\u542f\u6e90 Agilio CX \u7cfb\u5217\u7684\u786c\u4ef6\u7f51\u5361\u3002 CentOS 8 Stream \u6216\u4e0a\u6e38 Linux 5.7 \u4ee5\u4e0a\u5185\u6838\u652f\u6301\u3002 \u7531\u4e8e\u5f53\u524d\u7f51\u5361\u4e0d\u652f\u6301 dp_hash \u548c hash \u64cd\u4f5c\u5378\u8f7d\uff0c\u9700\u5173\u95ed OVN LB \u529f\u80fd\u3002 \u8bbe\u7f6e\u7f51\u5361 SR-IOV \u6a21\u5f0f \u00b6 \u7528\u6237\u53ef\u53c2\u8003 Agilio Open vSwitch TC User Guide \u83b7\u5f97\u8be5\u7f51\u5361\u4f7f\u7528\u7684\u66f4\u591a\u8be6\u7ec6\u4fe1\u606f\u3002 \u4fdd\u5b58\u4e0b\u5217\u811a\u672c\u7528\u4e8e\u540e\u7eed\u6267\u884c\u56fa\u4ef6\u76f8\u5173\u64cd\u4f5c\uff1a #!/bin/bash DEVICE = ${ 1 } DEFAULT_ASSY = scan ASSY = ${ 2 :- ${ DEFAULT_ASSY }} APP = ${ 3 :- flower } if [ \"x ${ DEVICE } \" = \"x\" -o ! -e /sys/class/net/ ${ DEVICE } ] ; then echo Syntax: ${ 0 } device [ ASSY ] [ APP ] echo echo This script associates the TC Offload firmware echo with a Netronome SmartNIC. echo echo device: is the network device associated with the SmartNIC echo ASSY: defaults to ${ DEFAULT_ASSY } echo APP: defaults to flower. flower-next is supported if updated echo firmware has been installed. exit 1 fi # It is recommended that the assembly be determined by inspection # The following code determines the value via the debug interface if [ \" ${ ASSY } x\" = \"scanx\" ] ; then ethtool -W ${ DEVICE } 0 DEBUG = $( ethtool -w ${ DEVICE } data /dev/stdout | strings ) SERIAL = $( echo \" ${ DEBUG } \" | grep \"^SN:\" ) ASSY = $( echo ${ SERIAL } | grep -oE AMDA [ 0 -9 ]{ 4 } ) fi PCIADDR = $( basename $( readlink -e /sys/class/net/ ${ DEVICE } /device )) FWDIR = \"/lib/firmware/netronome\" # AMDA0081 and AMDA0097 uses the same firmware if [ \" ${ ASSY } \" = \"AMDA0081\" ] ; then if [ ! -e ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw ] ; then ln -sf nic_AMDA0097.nffw ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw fi fi FW = \" ${ FWDIR } /pci- ${ PCIADDR } .nffw\" ln -sf \" ${ APP } /nic_ ${ ASSY } .nffw\" \" ${ FW } \" # insert distro-specific initramfs section here... \u5207\u6362\u56fa\u4ef6\u9009\u9879\u5e76\u91cd\u8f7d\u9a71\u52a8\uff1a ./agilio-tc-fw-select.sh ens47np0 scan rmmod nfp modprobe nfp \u68c0\u67e5\u53ef\u7528 VF \u6570\u91cf\uff0c\u5e76\u521b\u5efa VF\uff1a # cat /sys/class/net/ens3/device/sriov_totalvfs 65 # echo 4 > /sys/class/net/ens47/device/sriov_numvfs \u5b89\u88c5 SR-IOV Device Plugin \u00b6 \u7531\u4e8e\u6bcf\u4e2a\u673a\u5668\u7684 VF \u6570\u91cf\u6709\u9650\uff0c\u6bcf\u4e2a\u4f7f\u7528\u52a0\u901f\u7684 Pod \u4f1a\u5360\u7528 VF \u8d44\u6e90\uff0c\u6211\u4eec\u9700\u8981\u4f7f\u7528 SR-IOV Device Plugin \u7ba1\u7406\u76f8\u5e94\u8d44\u6e90\uff0c\u4f7f\u5f97\u8c03\u5ea6\u5668\u77e5\u9053\u5982\u4f55\u6839\u636e \u8d44\u6e90\u8fdb\u884c\u8c03\u5ea6\u3002 \u521b\u5efa SR-IOV \u76f8\u5173 Configmap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"corigine.com\", \"resourceName\": \"agilio_sriov\", \"selectors\": { \"vendors\": [\"19ee\"], \"devices\": [\"6003\"], \"drivers\": [\"nfp_netvf\"] } } ] } \u53c2\u8003 SR-IOV \u6587\u6863 \u8fdb\u884c\u90e8\u7f72: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml \u68c0\u67e5 SR-IOV \u8d44\u6e90\u662f\u5426\u5df2\u7ecf\u6ce8\u518c\u5230 Kubernetes Node \u4e2d\uff1a kubectl describe no containerserver | grep corigine corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov 0 0 \u5b89\u88c5 Multus-CNI \u00b6 SR-IOV Device Plugin \u8c03\u5ea6\u65f6\u83b7\u5f97\u7684\u8bbe\u5907 ID \u9700\u8981\u901a\u8fc7 Multus-CNI \u4f20\u9012\u7ed9 Kube-OVN\uff0c\u56e0\u6b64\u9700\u8981\u914d\u7f6e Multus-CNI \u914d\u5408\u5b8c\u6210\u76f8\u5173\u4efb\u52a1\u3002 \u53c2\u8003 Multus-CNI \u6587\u6863 \u8fdb\u884c\u90e8\u7f72\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml \u521b\u5efa NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : corigine.com/agilio_sriov spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : \u683c\u5f0f\u4e3a\u5f53\u524d NetworkAttachmentDefinition \u7684 {name}.{namespace}.ovn\u3002 Kube-OVN \u4e2d\u5f00\u542f\u5378\u8f7d\u6a21\u5f0f \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u76f8\u5173\u53c2\u6570\uff0c IFACE \u9700\u8981\u4e3a\u7269\u7406\u7f51\u5361\u540d\uff0c\u8be5\u7f51\u5361\u9700\u8981\u6709\u53ef\u8def\u7531 IP\uff1a ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" \u5b89\u88c5 Kube-OVN\uff1a bash install.sh \u521b\u5efa\u4f7f\u7528 VF \u7f51\u5361\u7684 Pod \u00b6 \u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b yaml \u683c\u5f0f\u521b\u5efa\u4f7f\u7528 VF \u8fdb\u884c\u7f51\u7edc\u5378\u8f7d\u52a0\u901f\u7684 Pod: apiVersion : v1 kind : Pod metadata : name : nginx namespace : default annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : corigine.com/agilio_sriov : '1' limits : corigine.com/agilio_sriov : '1' v1.multus-cni.io/default-network : \u4e3a\u4e0a\u4e00\u6b65\u9aa4\u4e2d NetworkAttachmentDefinition \u7684 {namespace}/{name}\u3002 \u53ef\u901a\u8fc7\u5728 Pod \u8fd0\u884c\u8282\u70b9\u7684 ovs-ovn \u5bb9\u5668\u4e2d\u8fd0\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u89c2\u5bdf\u5378\u8f7d\u662f\u5426\u6210\u529f\uff1a # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h \u5982\u679c\u6709 offloaded:yes, dp:tc \u5185\u5bb9\u8bc1\u660e\u5378\u8f7d\u6210\u529f\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u82af\u542f\u6e90\u7f51\u5361 Offload \u652f\u6301"},{"location":"advance/offload-corigine/#offload","text":"Kube-OVN \u5728\u6700\u7ec8\u7684\u6570\u636e\u5e73\u9762\u4f7f\u7528 OVS \u6765\u5b8c\u6210\u6d41\u91cf\u8f6c\u53d1\uff0c\u76f8\u5173\u7684\u6d41\u8868\u5339\u914d\uff0c\u96a7\u9053\u5c01\u88c5\u7b49\u529f\u80fd\u4e3a CPU \u5bc6\u96c6\u578b\uff0c\u5728\u5927\u6d41\u91cf\u4e0b\u4f1a\u6d88\u8017\u5927\u91cf CPU \u8d44\u6e90\u5e76\u5bfc\u81f4 \u5ef6\u8fdf\u4e0a\u5347\u548c\u541e\u5410\u91cf\u4e0b\u964d\u3002\u82af\u542f\u6e90\u7684 Agilio CX \u7cfb\u5217\u667a\u80fd\u7f51\u5361\u53ef\u4ee5\u5c06 OVS \u76f8\u5173\u7684\u64cd\u4f5c\u5378\u8f7d\u5230\u786c\u4ef6\u7f51\u5361\u4e2d\u6267\u884c\u3002 \u8be5\u6280\u672f\u53ef\u4ee5\u5728\u65e0\u9700\u5bf9 OVS \u63a7\u5236\u5e73\u9762\u8fdb\u884c\u4fee\u6539\u7684\u60c5\u51b5\u4e0b\uff0c\u7f29\u77ed\u6570\u636e\u8def\u5f84\uff0c\u907f\u514d\u5bf9\u4e3b\u673a CPU \u8d44\u6e90\u7684\u4f7f\u7528\uff0c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u663e\u8457\u63d0\u5347\u541e\u5410\u91cf\u3002","title":"\u82af\u542f\u6e90\u7f51\u5361 Offload \u652f\u6301"},{"location":"advance/offload-corigine/#_1","text":"\u82af\u542f\u6e90 Agilio CX \u7cfb\u5217\u7684\u786c\u4ef6\u7f51\u5361\u3002 CentOS 8 Stream \u6216\u4e0a\u6e38 Linux 5.7 \u4ee5\u4e0a\u5185\u6838\u652f\u6301\u3002 \u7531\u4e8e\u5f53\u524d\u7f51\u5361\u4e0d\u652f\u6301 dp_hash \u548c hash \u64cd\u4f5c\u5378\u8f7d\uff0c\u9700\u5173\u95ed OVN LB \u529f\u80fd\u3002","title":"\u524d\u7f6e\u6761\u4ef6"},{"location":"advance/offload-corigine/#sr-iov","text":"\u7528\u6237\u53ef\u53c2\u8003 Agilio Open vSwitch TC User Guide \u83b7\u5f97\u8be5\u7f51\u5361\u4f7f\u7528\u7684\u66f4\u591a\u8be6\u7ec6\u4fe1\u606f\u3002 \u4fdd\u5b58\u4e0b\u5217\u811a\u672c\u7528\u4e8e\u540e\u7eed\u6267\u884c\u56fa\u4ef6\u76f8\u5173\u64cd\u4f5c\uff1a #!/bin/bash DEVICE = ${ 1 } DEFAULT_ASSY = scan ASSY = ${ 2 :- ${ DEFAULT_ASSY }} APP = ${ 3 :- flower } if [ \"x ${ DEVICE } \" = \"x\" -o ! -e /sys/class/net/ ${ DEVICE } ] ; then echo Syntax: ${ 0 } device [ ASSY ] [ APP ] echo echo This script associates the TC Offload firmware echo with a Netronome SmartNIC. echo echo device: is the network device associated with the SmartNIC echo ASSY: defaults to ${ DEFAULT_ASSY } echo APP: defaults to flower. flower-next is supported if updated echo firmware has been installed. exit 1 fi # It is recommended that the assembly be determined by inspection # The following code determines the value via the debug interface if [ \" ${ ASSY } x\" = \"scanx\" ] ; then ethtool -W ${ DEVICE } 0 DEBUG = $( ethtool -w ${ DEVICE } data /dev/stdout | strings ) SERIAL = $( echo \" ${ DEBUG } \" | grep \"^SN:\" ) ASSY = $( echo ${ SERIAL } | grep -oE AMDA [ 0 -9 ]{ 4 } ) fi PCIADDR = $( basename $( readlink -e /sys/class/net/ ${ DEVICE } /device )) FWDIR = \"/lib/firmware/netronome\" # AMDA0081 and AMDA0097 uses the same firmware if [ \" ${ ASSY } \" = \"AMDA0081\" ] ; then if [ ! -e ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw ] ; then ln -sf nic_AMDA0097.nffw ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw fi fi FW = \" ${ FWDIR } /pci- ${ PCIADDR } .nffw\" ln -sf \" ${ APP } /nic_ ${ ASSY } .nffw\" \" ${ FW } \" # insert distro-specific initramfs section here... \u5207\u6362\u56fa\u4ef6\u9009\u9879\u5e76\u91cd\u8f7d\u9a71\u52a8\uff1a ./agilio-tc-fw-select.sh ens47np0 scan rmmod nfp modprobe nfp \u68c0\u67e5\u53ef\u7528 VF \u6570\u91cf\uff0c\u5e76\u521b\u5efa VF\uff1a # cat /sys/class/net/ens3/device/sriov_totalvfs 65 # echo 4 > /sys/class/net/ens47/device/sriov_numvfs","title":"\u8bbe\u7f6e\u7f51\u5361 SR-IOV \u6a21\u5f0f"},{"location":"advance/offload-corigine/#sr-iov-device-plugin","text":"\u7531\u4e8e\u6bcf\u4e2a\u673a\u5668\u7684 VF \u6570\u91cf\u6709\u9650\uff0c\u6bcf\u4e2a\u4f7f\u7528\u52a0\u901f\u7684 Pod \u4f1a\u5360\u7528 VF \u8d44\u6e90\uff0c\u6211\u4eec\u9700\u8981\u4f7f\u7528 SR-IOV Device Plugin \u7ba1\u7406\u76f8\u5e94\u8d44\u6e90\uff0c\u4f7f\u5f97\u8c03\u5ea6\u5668\u77e5\u9053\u5982\u4f55\u6839\u636e \u8d44\u6e90\u8fdb\u884c\u8c03\u5ea6\u3002 \u521b\u5efa SR-IOV \u76f8\u5173 Configmap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"corigine.com\", \"resourceName\": \"agilio_sriov\", \"selectors\": { \"vendors\": [\"19ee\"], \"devices\": [\"6003\"], \"drivers\": [\"nfp_netvf\"] } } ] } \u53c2\u8003 SR-IOV \u6587\u6863 \u8fdb\u884c\u90e8\u7f72: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml \u68c0\u67e5 SR-IOV \u8d44\u6e90\u662f\u5426\u5df2\u7ecf\u6ce8\u518c\u5230 Kubernetes Node \u4e2d\uff1a kubectl describe no containerserver | grep corigine corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov 0 0","title":"\u5b89\u88c5 SR-IOV Device Plugin"},{"location":"advance/offload-corigine/#multus-cni","text":"SR-IOV Device Plugin \u8c03\u5ea6\u65f6\u83b7\u5f97\u7684\u8bbe\u5907 ID \u9700\u8981\u901a\u8fc7 Multus-CNI \u4f20\u9012\u7ed9 Kube-OVN\uff0c\u56e0\u6b64\u9700\u8981\u914d\u7f6e Multus-CNI \u914d\u5408\u5b8c\u6210\u76f8\u5173\u4efb\u52a1\u3002 \u53c2\u8003 Multus-CNI \u6587\u6863 \u8fdb\u884c\u90e8\u7f72\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml \u521b\u5efa NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : corigine.com/agilio_sriov spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : \u683c\u5f0f\u4e3a\u5f53\u524d NetworkAttachmentDefinition \u7684 {name}.{namespace}.ovn\u3002","title":"\u5b89\u88c5 Multus-CNI"},{"location":"advance/offload-corigine/#kube-ovn","text":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u76f8\u5173\u53c2\u6570\uff0c IFACE \u9700\u8981\u4e3a\u7269\u7406\u7f51\u5361\u540d\uff0c\u8be5\u7f51\u5361\u9700\u8981\u6709\u53ef\u8def\u7531 IP\uff1a ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" \u5b89\u88c5 Kube-OVN\uff1a bash install.sh","title":"Kube-OVN \u4e2d\u5f00\u542f\u5378\u8f7d\u6a21\u5f0f"},{"location":"advance/offload-corigine/#vf-pod","text":"\u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b yaml \u683c\u5f0f\u521b\u5efa\u4f7f\u7528 VF \u8fdb\u884c\u7f51\u7edc\u5378\u8f7d\u52a0\u901f\u7684 Pod: apiVersion : v1 kind : Pod metadata : name : nginx namespace : default annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : corigine.com/agilio_sriov : '1' limits : corigine.com/agilio_sriov : '1' v1.multus-cni.io/default-network : \u4e3a\u4e0a\u4e00\u6b65\u9aa4\u4e2d NetworkAttachmentDefinition \u7684 {namespace}/{name}\u3002 \u53ef\u901a\u8fc7\u5728 Pod \u8fd0\u884c\u8282\u70b9\u7684 ovs-ovn \u5bb9\u5668\u4e2d\u8fd0\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u89c2\u5bdf\u5378\u8f7d\u662f\u5426\u6210\u529f\uff1a # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h \u5982\u679c\u6709 offloaded:yes, dp:tc \u5185\u5bb9\u8bc1\u660e\u5378\u8f7d\u6210\u529f\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u521b\u5efa\u4f7f\u7528 VF \u7f51\u5361\u7684 Pod"},{"location":"advance/offload-mellanox/","text":"Mellanox \u7f51\u5361 Offload \u652f\u6301 \u00b6 Kube-OVN \u5728\u6700\u7ec8\u7684\u6570\u636e\u5e73\u9762\u4f7f\u7528 OVS \u6765\u5b8c\u6210\u6d41\u91cf\u8f6c\u53d1\uff0c\u76f8\u5173\u7684\u6d41\u8868\u5339\u914d\uff0c\u96a7\u9053\u5c01\u88c5\u7b49\u529f\u80fd\u4e3a CPU \u5bc6\u96c6\u578b\uff0c\u5728\u5927\u6d41\u91cf\u4e0b\u4f1a\u6d88\u8017\u5927\u91cf CPU \u8d44\u6e90\u5e76\u5bfc\u81f4 \u5ef6\u8fdf\u4e0a\u5347\u548c\u541e\u5410\u91cf\u4e0b\u964d\u3002Mellanox \u7684 Accelerated Switching And Packet Processing (ASAP\u00b2) \u6280\u672f\u53ef\u4ee5\u5c06 OVS \u76f8\u5173\u7684\u64cd\u4f5c\u5378\u8f7d\u5230\u786c\u4ef6\u7f51\u5361\u5185\u7684 eSwitch \u4e0a\u6267\u884c\u3002\u8be5\u6280\u672f\u53ef\u4ee5\u5728\u65e0\u9700\u5bf9 OVS \u63a7\u5236\u5e73\u9762\u8fdb\u884c\u4fee\u6539\u7684\u60c5\u51b5\u4e0b\uff0c\u7f29\u77ed\u6570\u636e\u8def\u5f84\uff0c\u907f\u514d\u5bf9\u4e3b\u673a CPU \u8d44\u6e90\u7684\u4f7f\u7528\uff0c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u663e\u8457\u63d0\u5347\u541e\u5410\u91cf\u3002 \u524d\u7f6e\u6761\u4ef6 \u00b6 Mellanox CX5/CX6/CX7/BlueField \u7b49\u652f\u6301 ASAP\u00b2 \u7684\u786c\u4ef6\u7f51\u5361\u3002 CentOS 8 Stream \u6216\u4e0a\u6e38 Linux 5.7 \u4ee5\u4e0a\u5185\u6838\u652f\u6301\u3002 \u7531\u4e8e\u5f53\u524d\u7f51\u5361\u4e0d\u652f\u6301 dp_hash \u548c hash \u64cd\u4f5c\u5378\u8f7d\uff0c\u9700\u5173\u95ed OVN LB \u529f\u80fd\u3002 \u4e3a\u4e86\u652f\u6301\u5378\u8f7d\u6a21\u5f0f\uff0c\u7f51\u5361\u4e0d\u80fd\u505a bond\u3002 \u914d\u7f6e SR-IOV \u548c Device Plugin \u00b6 Mellanox \u7f51\u5361\u652f\u6301\u4e24\u79cd\u914d\u7f6e offload \u7684\u65b9\u5f0f\uff0c\u4e00\u79cd\u624b\u52a8\u914d\u7f6e\u7f51\u5361 SR-IOV \u548c Device Plugin\uff0c\u53e6\u4e00\u79cd\u901a\u8fc7 sriov-network-operator \u8fdb\u884c\u81ea\u52a8\u914d\u7f6e\u3002 \u624b\u52a8\u914d\u7f6e SR-IOV \u548c Device Plugin \u00b6 \u67e5\u8be2\u7f51\u5361\u7684\u8bbe\u5907 ID\uff0c\u4e0b\u9762\u7684\u4f8b\u5b50\u4e2d\u4e3a 42:00.0 \uff1a # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] \u6839\u636e\u8bbe\u5907 ID \u627e\u5230\u5bf9\u5e94\u7f51\u5361\uff1a # ls -l /sys/class/net/ | grep 42:00.0 lrwxrwxrwx. 1 root root 0 Jul 22 23 :16 p4p1 -> ../../devices/pci0000:40/0000:40:02.0/0000:42:00.0/net/p4p1 \u68c0\u67e5\u53ef\u7528 VF \u6570\u91cf\uff1a # cat /sys/class/net/p4p1/device/sriov_totalvfs 8 \u521b\u5efa VF\uff0c\u603b\u6570\u4e0d\u8981\u8d85\u8fc7\u4e0a\u9762\u67e5\u8be2\u51fa\u7684\u6570\u91cf\uff1a # echo '4' > /sys/class/net/p4p1/device/sriov_numvfs # ip link show p4p1 10 : p4p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:59:9f:c1:ec:12 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 1 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 2 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 3 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off # ip link set p4p1 up \u627e\u5230\u4e0a\u8ff0 VF \u5bf9\u5e94\u7684\u8bbe\u5907 ID\uff1a # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] \u5c06 VF \u4ece\u9a71\u52a8\u4e2d\u89e3\u7ed1\uff1a echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind \u5f00\u542f eSwitch \u6a21\u5f0f\uff0c\u5e76\u8bbe\u7f6e\u786c\u4ef6\u5378\u8f7d\uff1a devlink dev eswitch set pci/0000:42:00.0 mode switchdev ethtool -K enp66s0f0 hw-tc-offload on \u91cd\u65b0\u7ed1\u5b9a\u9a71\u52a8\uff0c\u5b8c\u6210 VF \u8bbe\u7f6e\uff1a echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/bind NetworkManager \u7684\u4e00\u4e9b\u884c\u4e3a\u53ef\u80fd\u4f1a\u5bfc\u81f4\u9a71\u52a8\u5f02\u5e38\uff0c\u5982\u679c\u5378\u8f7d\u51fa\u73b0\u95ee\u9898\u5efa\u8bae\u5173\u95ed NetworkManager \u518d\u8fdb\u884c\u5c1d\u8bd5\uff1a systemctl stop NetworkManager systemctl disable NetworkManager \u7531\u4e8e\u6bcf\u4e2a\u673a\u5668\u7684 VF \u6570\u91cf\u4f18\u5148\uff0c\u6bcf\u4e2a\u4f7f\u7528\u52a0\u901f\u7684 Pod \u4f1a\u5360\u7528 VF \u8d44\u6e90\uff0c\u6211\u4eec\u9700\u8981\u4f7f\u7528 SR-IOV Device Plugin \u7ba1\u7406\u76f8\u5e94\u8d44\u6e90\uff0c\u4f7f\u5f97\u8c03\u5ea6\u5668\u77e5\u9053\u5982\u4f55\u6839\u636e \u8d44\u6e90\u8fdb\u884c\u8c03\u5ea6\u3002 \u521b\u5efa SR-IOV \u76f8\u5173 Configmap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"mellanox.com\", \"resourceName\": \"cx5_sriov_switchdev\", \"selectors\": { \"vendors\": [\"15b3\"], \"devices\": [\"1018\"], \"drivers\": [\"mlx5_core\"] } } ] } \u53c2\u8003 SR-IOV \u6587\u6863 \u8fdb\u884c\u90e8\u7f72: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml \u68c0\u67e5 SR-IOV \u8d44\u6e90\u662f\u5426\u5df2\u7ecf\u6ce8\u518c\u5230 Kubernetes Node \u4e2d\uff1a kubectl describe node kube-ovn-01 | grep mellanox mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev 0 0 \u4f7f\u7528 sriov-network-operator \u914d\u7f6e SR-IOV \u548c Device Plugin \u00b6 \u5b89\u88c5 node-feature-discovery \u81ea\u52a8\u68c0\u6d4b\u786c\u4ef6\u7684\u529f\u80fd\u548c\u7cfb\u7edf\u914d\u7f6e: kubectl apply -k https://github.com/kubernetes-sigs/node-feature-discovery/deployment/overlays/default?ref = v0.11.3 \u6216\u8005\u901a\u8fc7\u4e0b\u9762\u7684\u547d\u4ee4\uff0c\u624b\u52a8\u7ed9\u6709 offload \u80fd\u529b\u7684\u7f51\u5361\u589e\u52a0 annotation: kubectl label nodes [ offloadNicNode ] feature.node.kubernetes.io/network-sriov.capable = true \u514b\u9686\u4ee3\u7801\u4ed3\u5e93\u5e76\u5b89\u88c5 Operator\uff1a git clone --depth = 1 https://github.com/kubeovn/sriov-network-operator.git kubectl apply -k sriov-network-operator/deploy \u68c0\u67e5 Operator \u7ec4\u4ef6\u662f\u5426\u5de5\u4f5c\u6b63\u5e38\uff1a # kubectl get -n kube-system all | grep sriov NAME READY STATUS RESTARTS AGE pod/sriov-network-config-daemon-bf9nt 1 /1 Running 0 8s pod/sriov-network-operator-54d7545f65-296gb 1 /1 Running 0 10s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/sriov-network-config-daemon 1 1 1 1 1 beta.kubernetes.io/os = linux,feature.node.kubernetes.io/network-sriov.capable = true 8s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/sriov-network-operator 1 /1 1 1 10s NAME DESIRED CURRENT READY AGE replicaset.apps/sriov-network-operator-54d7545f65 1 1 1 10s \u68c0\u67e5 SriovNetworkNodeState \uff0c\u4e0b\u9762\u4ee5 node1 \u8282\u70b9\u4e3a\u4f8b\uff0c\u8be5\u8282\u70b9\u4e0a\u6709\u4e24\u4e2a Mellanox \u7f51\u5361\uff1a # kubectl get sriovnetworknodestates.sriovnetwork.openshift.io -n kube-system node1 -o yaml apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodeState spec: ... status: interfaces: - deviceID: \"1017\" driver: mlx5_core mtu: 1500 pciAddress: \"0000:5f:00.0\" totalvfs: 8 vendor: \"15b3\" linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb name: ens41f0np0 - deviceID: \"1017\" driver: mlx5_core mtu: 1500 pciAddress: \"0000:5f:00.1\" totalvfs: 8 vendor: \"15b3\" linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb name: ens41f1np1 \u521b\u5efa SriovNetworkNodePolicy \u8d44\u6e90\uff0c\u5e76\u901a\u8fc7 nicSelector \u9009\u62e9\u8981\u7ba1\u7406\u7684\u7f51\u5361\uff1a apiVersion : sriovnetwork.openshift.io/v1 kind : SriovNetworkNodePolicy metadata : name : policy namespace : kube-system spec : nodeSelector : feature.node.kubernetes.io/network-sriov.capable : \"true\" eSwitchMode : switchdev numVfs : 3 nicSelector : pfNames : - ens41f0np0 - ens41f1np1 resourceName : cx_sriov_switchdev \u518d\u6b21\u68c0\u67e5 SriovNetworkNodeState \u7684 status \u5b57\u6bb5\uff1a # kubectl get sriovnetworknodestates.sriovnetwork.openshift.io -n kube-system node1 -o yaml ... spec: interfaces: - eSwitchMode: switchdev name: ens41f0np0 numVfs: 3 pciAddress: 0000 :5f:00.0 vfGroups: - policyName: policy vfRange: 0 -2 resourceName: cx_sriov_switchdev - eSwitchMode: switchdev name: ens41f1np1 numVfs: 3 pciAddress: 0000 :5f:00.1 vfGroups: - policyName: policy vfRange: 0 -2 resourceName: cx_sriov_switchdev status: interfaces - Vfs: - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.2 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.3 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.4 vendor: \"15b3\" deviceID: \"1017\" driver: mlx5_core linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:ab mtu: 1500 name: ens41f0np0 numVfs: 3 pciAddress: 0000 :5f:00.0 totalvfs: 3 vendor: \"15b3\" - Vfs: - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.5 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.6 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.7 vendor: \"15b3\" deviceID: \"1017\" driver: mlx5_core linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb mtu: 1500 name: ens41f1np1 numVfs: 3 pciAddress: 0000 :5f:00.1 totalvfs: 3 vendor: \"15b3\" \u68c0\u67e5 VF \u7684\u72b6\u6001\uff1a # lspci -nn | grep ConnectX 5f:00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 5f:00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 5f:00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.6 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.7 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] \u68c0\u67e5 PF \u5de5\u4f5c\u6a21\u5f0f\uff1a # cat /sys/class/net/ens41f0np0/compat/devlink/mode switchdev \u5b89\u88c5 Multus-CNI \u00b6 SR-IOV Device Plugin \u8c03\u5ea6\u65f6\u83b7\u5f97\u7684\u8bbe\u5907 ID \u9700\u8981\u901a\u8fc7 Multus-CNI \u4f20\u9012\u7ed9 Kube-OVN\uff0c\u56e0\u6b64\u9700\u8981\u914d\u7f6e Multus-CNI \u914d\u5408\u5b8c\u6210\u76f8\u5173\u4efb\u52a1\u3002 \u53c2\u8003 Multus-CNI \u6587\u6863 \u8fdb\u884c\u90e8\u7f72\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml \u521b\u5efa NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : mellanox.com/cx5_sriov_switchdev spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : \u683c\u5f0f\u4e3a\u5f53\u524d NetworkAttachmentDefinition \u7684 {name}.{namespace}.ovn\u3002 Kube-OVN \u4e2d\u5f00\u542f\u5378\u8f7d\u6a21\u5f0f \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u76f8\u5173\u53c2\u6570\uff0c IFACE \u9700\u8981\u4e3a\u7269\u7406\u7f51\u5361\u540d\uff0c\u8be5\u7f51\u5361\u9700\u8981\u6709\u53ef\u8def\u7531 IP\uff1a ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" \u5b89\u88c5 Kube-OVN\uff1a bash install.sh \u521b\u5efa\u4f7f\u7528 VF \u7f51\u5361\u7684 Pod \u00b6 \u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b yaml \u683c\u5f0f\u521b\u5efa\u4f7f\u7528 VF \u8fdb\u884c\u7f51\u7edc\u5378\u8f7d\u52a0\u901f\u7684 Pod: apiVersion : v1 kind : Pod metadata : name : nginx annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : mellanox.com/cx5_sriov_switchdev : '1' limits : mellanox.com/cx5_sriov_switchdev : '1' v1.multus-cni.io/default-network : \u4e3a\u4e0a\u4e00\u6b65\u9aa4\u4e2d NetworkAttachmentDefinition \u7684 {namespace}/{name}\u3002 \u53ef\u901a\u8fc7\u5728 Pod \u8fd0\u884c\u8282\u70b9\u7684 ovs-ovn \u5bb9\u5668\u4e2d\u8fd0\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u89c2\u5bdf\u5378\u8f7d\u662f\u5426\u6210\u529f\uff1a # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h \u5982\u679c\u6709 offloaded:yes, dp:tc \u5185\u5bb9\u8bc1\u660e\u5378\u8f7d\u6210\u529f\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Mellanox \u7f51\u5361 Offload \u652f\u6301"},{"location":"advance/offload-mellanox/#mellanox-offload","text":"Kube-OVN \u5728\u6700\u7ec8\u7684\u6570\u636e\u5e73\u9762\u4f7f\u7528 OVS \u6765\u5b8c\u6210\u6d41\u91cf\u8f6c\u53d1\uff0c\u76f8\u5173\u7684\u6d41\u8868\u5339\u914d\uff0c\u96a7\u9053\u5c01\u88c5\u7b49\u529f\u80fd\u4e3a CPU \u5bc6\u96c6\u578b\uff0c\u5728\u5927\u6d41\u91cf\u4e0b\u4f1a\u6d88\u8017\u5927\u91cf CPU \u8d44\u6e90\u5e76\u5bfc\u81f4 \u5ef6\u8fdf\u4e0a\u5347\u548c\u541e\u5410\u91cf\u4e0b\u964d\u3002Mellanox \u7684 Accelerated Switching And Packet Processing (ASAP\u00b2) \u6280\u672f\u53ef\u4ee5\u5c06 OVS \u76f8\u5173\u7684\u64cd\u4f5c\u5378\u8f7d\u5230\u786c\u4ef6\u7f51\u5361\u5185\u7684 eSwitch \u4e0a\u6267\u884c\u3002\u8be5\u6280\u672f\u53ef\u4ee5\u5728\u65e0\u9700\u5bf9 OVS \u63a7\u5236\u5e73\u9762\u8fdb\u884c\u4fee\u6539\u7684\u60c5\u51b5\u4e0b\uff0c\u7f29\u77ed\u6570\u636e\u8def\u5f84\uff0c\u907f\u514d\u5bf9\u4e3b\u673a CPU \u8d44\u6e90\u7684\u4f7f\u7528\uff0c\u5927\u5e45\u964d\u4f4e\u5ef6\u8fdf\u5e76\u663e\u8457\u63d0\u5347\u541e\u5410\u91cf\u3002","title":"Mellanox \u7f51\u5361 Offload \u652f\u6301"},{"location":"advance/offload-mellanox/#_1","text":"Mellanox CX5/CX6/CX7/BlueField \u7b49\u652f\u6301 ASAP\u00b2 \u7684\u786c\u4ef6\u7f51\u5361\u3002 CentOS 8 Stream \u6216\u4e0a\u6e38 Linux 5.7 \u4ee5\u4e0a\u5185\u6838\u652f\u6301\u3002 \u7531\u4e8e\u5f53\u524d\u7f51\u5361\u4e0d\u652f\u6301 dp_hash \u548c hash \u64cd\u4f5c\u5378\u8f7d\uff0c\u9700\u5173\u95ed OVN LB \u529f\u80fd\u3002 \u4e3a\u4e86\u652f\u6301\u5378\u8f7d\u6a21\u5f0f\uff0c\u7f51\u5361\u4e0d\u80fd\u505a bond\u3002","title":"\u524d\u7f6e\u6761\u4ef6"},{"location":"advance/offload-mellanox/#sr-iov-device-plugin","text":"Mellanox \u7f51\u5361\u652f\u6301\u4e24\u79cd\u914d\u7f6e offload \u7684\u65b9\u5f0f\uff0c\u4e00\u79cd\u624b\u52a8\u914d\u7f6e\u7f51\u5361 SR-IOV \u548c Device Plugin\uff0c\u53e6\u4e00\u79cd\u901a\u8fc7 sriov-network-operator \u8fdb\u884c\u81ea\u52a8\u914d\u7f6e\u3002","title":"\u914d\u7f6e SR-IOV \u548c Device Plugin"},{"location":"advance/offload-mellanox/#sr-iov-device-plugin_1","text":"\u67e5\u8be2\u7f51\u5361\u7684\u8bbe\u5907 ID\uff0c\u4e0b\u9762\u7684\u4f8b\u5b50\u4e2d\u4e3a 42:00.0 \uff1a # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] \u6839\u636e\u8bbe\u5907 ID \u627e\u5230\u5bf9\u5e94\u7f51\u5361\uff1a # ls -l /sys/class/net/ | grep 42:00.0 lrwxrwxrwx. 1 root root 0 Jul 22 23 :16 p4p1 -> ../../devices/pci0000:40/0000:40:02.0/0000:42:00.0/net/p4p1 \u68c0\u67e5\u53ef\u7528 VF \u6570\u91cf\uff1a # cat /sys/class/net/p4p1/device/sriov_totalvfs 8 \u521b\u5efa VF\uff0c\u603b\u6570\u4e0d\u8981\u8d85\u8fc7\u4e0a\u9762\u67e5\u8be2\u51fa\u7684\u6570\u91cf\uff1a # echo '4' > /sys/class/net/p4p1/device/sriov_numvfs # ip link show p4p1 10 : p4p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:59:9f:c1:ec:12 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 1 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 2 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 3 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off # ip link set p4p1 up \u627e\u5230\u4e0a\u8ff0 VF \u5bf9\u5e94\u7684\u8bbe\u5907 ID\uff1a # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] \u5c06 VF \u4ece\u9a71\u52a8\u4e2d\u89e3\u7ed1\uff1a echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind \u5f00\u542f eSwitch \u6a21\u5f0f\uff0c\u5e76\u8bbe\u7f6e\u786c\u4ef6\u5378\u8f7d\uff1a devlink dev eswitch set pci/0000:42:00.0 mode switchdev ethtool -K enp66s0f0 hw-tc-offload on \u91cd\u65b0\u7ed1\u5b9a\u9a71\u52a8\uff0c\u5b8c\u6210 VF \u8bbe\u7f6e\uff1a echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/bind NetworkManager \u7684\u4e00\u4e9b\u884c\u4e3a\u53ef\u80fd\u4f1a\u5bfc\u81f4\u9a71\u52a8\u5f02\u5e38\uff0c\u5982\u679c\u5378\u8f7d\u51fa\u73b0\u95ee\u9898\u5efa\u8bae\u5173\u95ed NetworkManager \u518d\u8fdb\u884c\u5c1d\u8bd5\uff1a systemctl stop NetworkManager systemctl disable NetworkManager \u7531\u4e8e\u6bcf\u4e2a\u673a\u5668\u7684 VF \u6570\u91cf\u4f18\u5148\uff0c\u6bcf\u4e2a\u4f7f\u7528\u52a0\u901f\u7684 Pod \u4f1a\u5360\u7528 VF \u8d44\u6e90\uff0c\u6211\u4eec\u9700\u8981\u4f7f\u7528 SR-IOV Device Plugin \u7ba1\u7406\u76f8\u5e94\u8d44\u6e90\uff0c\u4f7f\u5f97\u8c03\u5ea6\u5668\u77e5\u9053\u5982\u4f55\u6839\u636e \u8d44\u6e90\u8fdb\u884c\u8c03\u5ea6\u3002 \u521b\u5efa SR-IOV \u76f8\u5173 Configmap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"mellanox.com\", \"resourceName\": \"cx5_sriov_switchdev\", \"selectors\": { \"vendors\": [\"15b3\"], \"devices\": [\"1018\"], \"drivers\": [\"mlx5_core\"] } } ] } \u53c2\u8003 SR-IOV \u6587\u6863 \u8fdb\u884c\u90e8\u7f72: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml \u68c0\u67e5 SR-IOV \u8d44\u6e90\u662f\u5426\u5df2\u7ecf\u6ce8\u518c\u5230 Kubernetes Node \u4e2d\uff1a kubectl describe node kube-ovn-01 | grep mellanox mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev 0 0","title":"\u624b\u52a8\u914d\u7f6e SR-IOV \u548c Device Plugin"},{"location":"advance/offload-mellanox/#sriov-network-operator-sr-iov-device-plugin","text":"\u5b89\u88c5 node-feature-discovery \u81ea\u52a8\u68c0\u6d4b\u786c\u4ef6\u7684\u529f\u80fd\u548c\u7cfb\u7edf\u914d\u7f6e: kubectl apply -k https://github.com/kubernetes-sigs/node-feature-discovery/deployment/overlays/default?ref = v0.11.3 \u6216\u8005\u901a\u8fc7\u4e0b\u9762\u7684\u547d\u4ee4\uff0c\u624b\u52a8\u7ed9\u6709 offload \u80fd\u529b\u7684\u7f51\u5361\u589e\u52a0 annotation: kubectl label nodes [ offloadNicNode ] feature.node.kubernetes.io/network-sriov.capable = true \u514b\u9686\u4ee3\u7801\u4ed3\u5e93\u5e76\u5b89\u88c5 Operator\uff1a git clone --depth = 1 https://github.com/kubeovn/sriov-network-operator.git kubectl apply -k sriov-network-operator/deploy \u68c0\u67e5 Operator \u7ec4\u4ef6\u662f\u5426\u5de5\u4f5c\u6b63\u5e38\uff1a # kubectl get -n kube-system all | grep sriov NAME READY STATUS RESTARTS AGE pod/sriov-network-config-daemon-bf9nt 1 /1 Running 0 8s pod/sriov-network-operator-54d7545f65-296gb 1 /1 Running 0 10s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/sriov-network-config-daemon 1 1 1 1 1 beta.kubernetes.io/os = linux,feature.node.kubernetes.io/network-sriov.capable = true 8s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/sriov-network-operator 1 /1 1 1 10s NAME DESIRED CURRENT READY AGE replicaset.apps/sriov-network-operator-54d7545f65 1 1 1 10s \u68c0\u67e5 SriovNetworkNodeState \uff0c\u4e0b\u9762\u4ee5 node1 \u8282\u70b9\u4e3a\u4f8b\uff0c\u8be5\u8282\u70b9\u4e0a\u6709\u4e24\u4e2a Mellanox \u7f51\u5361\uff1a # kubectl get sriovnetworknodestates.sriovnetwork.openshift.io -n kube-system node1 -o yaml apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodeState spec: ... status: interfaces: - deviceID: \"1017\" driver: mlx5_core mtu: 1500 pciAddress: \"0000:5f:00.0\" totalvfs: 8 vendor: \"15b3\" linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb name: ens41f0np0 - deviceID: \"1017\" driver: mlx5_core mtu: 1500 pciAddress: \"0000:5f:00.1\" totalvfs: 8 vendor: \"15b3\" linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb name: ens41f1np1 \u521b\u5efa SriovNetworkNodePolicy \u8d44\u6e90\uff0c\u5e76\u901a\u8fc7 nicSelector \u9009\u62e9\u8981\u7ba1\u7406\u7684\u7f51\u5361\uff1a apiVersion : sriovnetwork.openshift.io/v1 kind : SriovNetworkNodePolicy metadata : name : policy namespace : kube-system spec : nodeSelector : feature.node.kubernetes.io/network-sriov.capable : \"true\" eSwitchMode : switchdev numVfs : 3 nicSelector : pfNames : - ens41f0np0 - ens41f1np1 resourceName : cx_sriov_switchdev \u518d\u6b21\u68c0\u67e5 SriovNetworkNodeState \u7684 status \u5b57\u6bb5\uff1a # kubectl get sriovnetworknodestates.sriovnetwork.openshift.io -n kube-system node1 -o yaml ... spec: interfaces: - eSwitchMode: switchdev name: ens41f0np0 numVfs: 3 pciAddress: 0000 :5f:00.0 vfGroups: - policyName: policy vfRange: 0 -2 resourceName: cx_sriov_switchdev - eSwitchMode: switchdev name: ens41f1np1 numVfs: 3 pciAddress: 0000 :5f:00.1 vfGroups: - policyName: policy vfRange: 0 -2 resourceName: cx_sriov_switchdev status: interfaces - Vfs: - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.2 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.3 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.4 vendor: \"15b3\" deviceID: \"1017\" driver: mlx5_core linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:ab mtu: 1500 name: ens41f0np0 numVfs: 3 pciAddress: 0000 :5f:00.0 totalvfs: 3 vendor: \"15b3\" - Vfs: - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.5 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.6 vendor: \"15b3\" - deviceID: 1018 driver: mlx5_core pciAddress: 0000 :5f:00.7 vendor: \"15b3\" deviceID: \"1017\" driver: mlx5_core linkSeed: 25000Mb/s linkType: ETH mac: 08 :c0:eb:f4:85:bb mtu: 1500 name: ens41f1np1 numVfs: 3 pciAddress: 0000 :5f:00.1 totalvfs: 3 vendor: \"15b3\" \u68c0\u67e5 VF \u7684\u72b6\u6001\uff1a # lspci -nn | grep ConnectX 5f:00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 5f:00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 5f:00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.6 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 5f:00.7 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] \u68c0\u67e5 PF \u5de5\u4f5c\u6a21\u5f0f\uff1a # cat /sys/class/net/ens41f0np0/compat/devlink/mode switchdev","title":"\u4f7f\u7528 sriov-network-operator \u914d\u7f6e SR-IOV \u548c Device Plugin"},{"location":"advance/offload-mellanox/#multus-cni","text":"SR-IOV Device Plugin \u8c03\u5ea6\u65f6\u83b7\u5f97\u7684\u8bbe\u5907 ID \u9700\u8981\u901a\u8fc7 Multus-CNI \u4f20\u9012\u7ed9 Kube-OVN\uff0c\u56e0\u6b64\u9700\u8981\u914d\u7f6e Multus-CNI \u914d\u5408\u5b8c\u6210\u76f8\u5173\u4efb\u52a1\u3002 \u53c2\u8003 Multus-CNI \u6587\u6863 \u8fdb\u884c\u90e8\u7f72\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml \u521b\u5efa NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : mellanox.com/cx5_sriov_switchdev spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : \u683c\u5f0f\u4e3a\u5f53\u524d NetworkAttachmentDefinition \u7684 {name}.{namespace}.ovn\u3002","title":"\u5b89\u88c5 Multus-CNI"},{"location":"advance/offload-mellanox/#kube-ovn","text":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u76f8\u5173\u53c2\u6570\uff0c IFACE \u9700\u8981\u4e3a\u7269\u7406\u7f51\u5361\u540d\uff0c\u8be5\u7f51\u5361\u9700\u8981\u6709\u53ef\u8def\u7531 IP\uff1a ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" \u5b89\u88c5 Kube-OVN\uff1a bash install.sh","title":"Kube-OVN \u4e2d\u5f00\u542f\u5378\u8f7d\u6a21\u5f0f"},{"location":"advance/offload-mellanox/#vf-pod","text":"\u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b yaml \u683c\u5f0f\u521b\u5efa\u4f7f\u7528 VF \u8fdb\u884c\u7f51\u7edc\u5378\u8f7d\u52a0\u901f\u7684 Pod: apiVersion : v1 kind : Pod metadata : name : nginx annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : mellanox.com/cx5_sriov_switchdev : '1' limits : mellanox.com/cx5_sriov_switchdev : '1' v1.multus-cni.io/default-network : \u4e3a\u4e0a\u4e00\u6b65\u9aa4\u4e2d NetworkAttachmentDefinition \u7684 {namespace}/{name}\u3002 \u53ef\u901a\u8fc7\u5728 Pod \u8fd0\u884c\u8282\u70b9\u7684 ovs-ovn \u5bb9\u5668\u4e2d\u8fd0\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u89c2\u5bdf\u5378\u8f7d\u662f\u5426\u6210\u529f\uff1a # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h \u5982\u679c\u6709 offloaded:yes, dp:tc \u5185\u5bb9\u8bc1\u660e\u5378\u8f7d\u6210\u529f\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u521b\u5efa\u4f7f\u7528 VF \u7f51\u5361\u7684 Pod"},{"location":"advance/overlay-with-route/","text":"Overlay \u4e0b\u8def\u7531\u65b9\u5f0f\u7f51\u7edc\u6253\u901a \u00b6 \u5728\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u7f51\u7edc\u73af\u5883\u4e0d\u652f\u6301 Underlay \u6a21\u5f0f\uff0c\u4f46\u662f\u4f9d\u7136\u9700\u8981 Pod \u80fd\u548c\u5916\u90e8\u8bbe\u65bd\u76f4\u63a5\u901a\u8fc7 IP \u8fdb\u884c\u4e92\u8bbf\uff0c \u8fd9\u65f6\u5019\u53ef\u4ee5\u4f7f\u7528\u8def\u7531\u65b9\u5f0f\u5c06\u5bb9\u5668\u7f51\u7edc\u548c\u5916\u90e8\u8054\u901a\u3002 \u8def\u7531\u6a21\u5f0f\u53ea\u652f\u6301\u9ed8\u8ba4 VPC \u4e0b\u7684 Overlay \u7f51\u7edc\u548c\u5916\u90e8\u6253\u901a\uff0c\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\uff0cPod IP \u4f1a\u76f4\u63a5\u8fdb\u5165\u5e95\u5c42\u7f51\u7edc\uff0c\u5e95\u5c42\u7f51\u7edc\u9700\u8981\u653e\u5f00\u5173\u4e8e\u6e90\u5730\u5740\u548c\u76ee\u5730\u5740\u7684 IP \u68c0\u67e5\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u6b64\u6a21\u5f0f\u4e0b\uff0c\u4e3b\u673a\u9700\u8981\u5f00\u653e ip_forward \u3002 \u68c0\u67e5\u4e3b\u673a iptables \u89c4\u5219\u4e2d\u662f\u5426\u5728 forward \u94fe\u4e2d\u662f\u5426\u6709 Drop \u89c4\u5219\uff0c\u9700\u8981\u653e\u884c\u5bb9\u5668\u76f8\u5173\u6d41\u91cf\u3002 \u7531\u4e8e\u53ef\u80fd\u5b58\u5728\u975e\u5bf9\u79f0\u8def\u7531\u7684\u60c5\u51b5\uff0c\u4e3b\u673a\u9700\u653e\u884c ct \u72b6\u6001\u4e3a INVALID \u7684\u6570\u636e\u5305\u3002 \u8bbe\u7f6e\u6b65\u9aa4 \u00b6 \u5bf9\u4e8e\u9700\u8981\u5bf9\u5916\u76f4\u63a5\u8def\u7531\u7684\u5b50\u7f51\uff0c\u9700\u8981\u5c06\u5b50\u7f51\u7684 natOutgoing \u8bbe\u7f6e\u4e3a false \uff0c\u5173\u95ed nat \u6620\u5c04\uff0c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : false \u6b64\u65f6\uff0cPod \u7684\u6570\u636e\u5305\u53ef\u4ee5\u901a\u8fc7\u4e3b\u673a\u8def\u7531\u5230\u8fbe\u5bf9\u7aef\u8282\u70b9\uff0c\u4f46\u662f\u5bf9\u7aef\u8282\u70b9\u8fd8\u4e0d\u77e5\u9053\u56de\u7a0b\u6570\u636e\u5305\u5e94\u8be5\u53d1\u9001\u5230\u54ea\u91cc\uff0c\u9700\u8981\u6dfb\u52a0\u56de\u7a0b\u8def\u7531\u3002 \u5982\u679c\u5bf9\u7aef\u4e3b\u673a\u548c\u5bb9\u5668\u6240\u5728\u5bbf\u4e3b\u673a\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u7f51\u7edc\uff0c\u6211\u4eec\u53ef\u4ee5\u76f4\u63a5\u5728\u5bf9\u7aef\u4e3b\u673a\u6dfb\u52a0\u9759\u6001\u8def\u7531\u5c06\u5bb9\u5668\u7f51\u7edc\u7684\u4e0b\u4e00\u8df3\u6307\u5411 Kubernetes \u96c6\u7fa4\u5185\u7684\u4efb\u610f\u4e00\u53f0\u673a\u5668\u3002 ip route add 10 .166.0.0/16 via 192 .168.2.10 dev eth0 10.166.0.0/16 \u4e3a\u5bb9\u5668\u5b50\u7f51\u7f51\u6bb5\uff0c 192.168.2.10 \u4e3a Kubernetes \u96c6\u7fa4\u5185\u4efb\u610f\u4e00\u4e2a\u8282\u70b9\u3002 \u82e5\u5bf9\u7aef\u4e3b\u673a\u548c\u5bb9\u5668\u6240\u5728\u5bbf\u4e3b\u673a\u4e0d\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u7f51\u7edc\uff0c\u5219\u9700\u8981\u5728\u8def\u7531\u5668\u4e0a\u914d\u7f6e\u76f8\u5e94\u7684\u89c4\u5219\uff0c\u901a\u8fc7\u8def\u7531\u5668\u8fdb\u884c\u6253\u901a\u3002 \u5728\u4e00\u4e9b\u865a\u62df\u5316\u73af\u5883\u4e2d\uff0c\u865a\u62df\u7f51\u7edc\u4f1a\u5c06\u975e\u5bf9\u79f0\u6d41\u91cf\u8bc6\u522b\u4e3a\u975e\u6cd5\u6d41\u91cf\u5e76\u4e22\u5f03\u3002 \u6b64\u65f6\u9700\u8981\u5c06 Subnet \u7684 gatewayType \u8c03\u6574\u4e3a centralized \uff0c\u5e76\u5728\u8def\u7531\u8bbe\u7f6e\u65f6\u5c06\u4e0b\u4e00\u8df3\u8bbe\u7f6e\u4e3a gatewayNode \u8282\u70b9\u7684 IP\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1\" natOutgoing : false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Overlay \u4e0b\u8def\u7531\u65b9\u5f0f\u7f51\u7edc\u6253\u901a"},{"location":"advance/overlay-with-route/#overlay","text":"\u5728\u4e00\u4e9b\u573a\u666f\u4e0b\uff0c\u7f51\u7edc\u73af\u5883\u4e0d\u652f\u6301 Underlay \u6a21\u5f0f\uff0c\u4f46\u662f\u4f9d\u7136\u9700\u8981 Pod \u80fd\u548c\u5916\u90e8\u8bbe\u65bd\u76f4\u63a5\u901a\u8fc7 IP \u8fdb\u884c\u4e92\u8bbf\uff0c \u8fd9\u65f6\u5019\u53ef\u4ee5\u4f7f\u7528\u8def\u7531\u65b9\u5f0f\u5c06\u5bb9\u5668\u7f51\u7edc\u548c\u5916\u90e8\u8054\u901a\u3002 \u8def\u7531\u6a21\u5f0f\u53ea\u652f\u6301\u9ed8\u8ba4 VPC \u4e0b\u7684 Overlay \u7f51\u7edc\u548c\u5916\u90e8\u6253\u901a\uff0c\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\uff0cPod IP \u4f1a\u76f4\u63a5\u8fdb\u5165\u5e95\u5c42\u7f51\u7edc\uff0c\u5e95\u5c42\u7f51\u7edc\u9700\u8981\u653e\u5f00\u5173\u4e8e\u6e90\u5730\u5740\u548c\u76ee\u5730\u5740\u7684 IP \u68c0\u67e5\u3002","title":"Overlay \u4e0b\u8def\u7531\u65b9\u5f0f\u7f51\u7edc\u6253\u901a"},{"location":"advance/overlay-with-route/#_1","text":"\u6b64\u6a21\u5f0f\u4e0b\uff0c\u4e3b\u673a\u9700\u8981\u5f00\u653e ip_forward \u3002 \u68c0\u67e5\u4e3b\u673a iptables \u89c4\u5219\u4e2d\u662f\u5426\u5728 forward \u94fe\u4e2d\u662f\u5426\u6709 Drop \u89c4\u5219\uff0c\u9700\u8981\u653e\u884c\u5bb9\u5668\u76f8\u5173\u6d41\u91cf\u3002 \u7531\u4e8e\u53ef\u80fd\u5b58\u5728\u975e\u5bf9\u79f0\u8def\u7531\u7684\u60c5\u51b5\uff0c\u4e3b\u673a\u9700\u653e\u884c ct \u72b6\u6001\u4e3a INVALID \u7684\u6570\u636e\u5305\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/overlay-with-route/#_2","text":"\u5bf9\u4e8e\u9700\u8981\u5bf9\u5916\u76f4\u63a5\u8def\u7531\u7684\u5b50\u7f51\uff0c\u9700\u8981\u5c06\u5b50\u7f51\u7684 natOutgoing \u8bbe\u7f6e\u4e3a false \uff0c\u5173\u95ed nat \u6620\u5c04\uff0c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : false \u6b64\u65f6\uff0cPod \u7684\u6570\u636e\u5305\u53ef\u4ee5\u901a\u8fc7\u4e3b\u673a\u8def\u7531\u5230\u8fbe\u5bf9\u7aef\u8282\u70b9\uff0c\u4f46\u662f\u5bf9\u7aef\u8282\u70b9\u8fd8\u4e0d\u77e5\u9053\u56de\u7a0b\u6570\u636e\u5305\u5e94\u8be5\u53d1\u9001\u5230\u54ea\u91cc\uff0c\u9700\u8981\u6dfb\u52a0\u56de\u7a0b\u8def\u7531\u3002 \u5982\u679c\u5bf9\u7aef\u4e3b\u673a\u548c\u5bb9\u5668\u6240\u5728\u5bbf\u4e3b\u673a\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u7f51\u7edc\uff0c\u6211\u4eec\u53ef\u4ee5\u76f4\u63a5\u5728\u5bf9\u7aef\u4e3b\u673a\u6dfb\u52a0\u9759\u6001\u8def\u7531\u5c06\u5bb9\u5668\u7f51\u7edc\u7684\u4e0b\u4e00\u8df3\u6307\u5411 Kubernetes \u96c6\u7fa4\u5185\u7684\u4efb\u610f\u4e00\u53f0\u673a\u5668\u3002 ip route add 10 .166.0.0/16 via 192 .168.2.10 dev eth0 10.166.0.0/16 \u4e3a\u5bb9\u5668\u5b50\u7f51\u7f51\u6bb5\uff0c 192.168.2.10 \u4e3a Kubernetes \u96c6\u7fa4\u5185\u4efb\u610f\u4e00\u4e2a\u8282\u70b9\u3002 \u82e5\u5bf9\u7aef\u4e3b\u673a\u548c\u5bb9\u5668\u6240\u5728\u5bbf\u4e3b\u673a\u4e0d\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u7f51\u7edc\uff0c\u5219\u9700\u8981\u5728\u8def\u7531\u5668\u4e0a\u914d\u7f6e\u76f8\u5e94\u7684\u89c4\u5219\uff0c\u901a\u8fc7\u8def\u7531\u5668\u8fdb\u884c\u6253\u901a\u3002 \u5728\u4e00\u4e9b\u865a\u62df\u5316\u73af\u5883\u4e2d\uff0c\u865a\u62df\u7f51\u7edc\u4f1a\u5c06\u975e\u5bf9\u79f0\u6d41\u91cf\u8bc6\u522b\u4e3a\u975e\u6cd5\u6d41\u91cf\u5e76\u4e22\u5f03\u3002 \u6b64\u65f6\u9700\u8981\u5c06 Subnet \u7684 gatewayType \u8c03\u6574\u4e3a centralized \uff0c\u5e76\u5728\u8def\u7531\u8bbe\u7f6e\u65f6\u5c06\u4e0b\u4e00\u8df3\u8bbe\u7f6e\u4e3a gatewayNode \u8282\u70b9\u7684 IP\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1\" natOutgoing : false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8bbe\u7f6e\u6b65\u9aa4"},{"location":"advance/ovn-eip-fip-snat/","text":"OVN EIP FIP SNAT DNAT \u652f\u6301 \u00b6 graph LR pod-->vpc1-subnet-->vpc1-->snat-->lrp-->external-subnet-->gw-node-external-nic Pod \u57fa\u4e8e SNAT \u51fa\u516c\u7f51\u7684\u5927\u81f4\u6d41\u7a0b\uff0c\u6700\u540e\u662f\u7ecf\u8fc7\u7f51\u5173\u8282\u70b9\u7684\u516c\u7f51\u7f51\u5361\u3002 graph LR pod-->vpc1-subnet-->vpc1-->fip-->lrp-->external-subnet-->local-node-external-nic Pod \u57fa\u4e8e FIP \u51fa\u516c\u7f51\u7684\u5927\u81f4\u6d41\u7a0b\uff0c\u6700\u540e\u53ef\u4ee5\u57fa\u4e8e\u672c\u5730\u8282\u70b9\u7684\u516c\u7f51\u7f51\u5361\u51fa\u516c\u7f51\u3002 \u8be5\u529f\u80fd\u6240\u652f\u6301\u7684 CRD \u5728\u4f7f\u7528\u4e0a\u5c06\u548c iptable nat gw \u516c\u7f51\u65b9\u6848\u4fdd\u6301\u57fa\u672c\u4e00\u81f4\u3002 ovn eip: \u7528\u4e8e\u516c\u7f51 ip \u5360\u4f4d\uff0c\u4ece underlay provider network vlan subnet \u4e2d\u5206\u914d ovn fip\uff1a \u4e00\u5bf9\u4e00 dnat snat\uff0c\u4e3a vpc \u5185\u7684 ip \u6216\u8005 vip \u63d0\u4f9b\u516c\u7f51\u76f4\u63a5\u8bbf\u95ee\u80fd\u529b ovn snat\uff1a\u6574\u4e2a\u5b50\u7f51\u6216\u8005\u5355\u4e2a vpc \u5185 ip \u53ef\u4ee5\u57fa\u4e8e snat \u8bbf\u95ee\u516c\u7f51 ovn dnat\uff1a\u57fa\u4e8e router lb \u5b9e\u73b0, \u57fa\u4e8e\u516c\u7f51 ip + \u7aef\u53e3 \u76f4\u63a5\u8bbf\u95ee vpc \u5185\u7684 \u4e00\u7ec4 endpoints 1. \u90e8\u7f72 \u00b6 \u76ee\u524d\u5141\u8bb8\u6240\u6709\uff08\u9ed8\u8ba4\u4ee5\u53ca\u81ea\u5b9a\u4e49\uff09vpc \u4f7f\u7528\u540c\u4e00\u4e2a provider vlan subnet \u8d44\u6e90\uff0c\u540c\u65f6\u517c\u5bb9 \u9ed8\u8ba4 VPC EIP/SNAT \u7684\u573a\u666f\u3002 \u7c7b\u4f3c neutron ovn\uff0c\u670d\u52a1\u542f\u52a8\u914d\u7f6e\u4e2d\u9700\u8981\u6307\u5b9a provider network \u76f8\u5173\u7684\u914d\u7f6e\uff0c\u4e0b\u8ff0\u7684\u542f\u52a8\u53c2\u6570\u4e5f\u662f\u4e3a\u4e86\u517c\u5bb9 VPC EIP/SNAT \u7684\u5b9e\u73b0\u3002 \u90e8\u7f72\u9636\u6bb5\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\uff0c\u53ef\u80fd\u9700\u8981\u6307\u5b9a\u9ed8\u8ba4\u516c\u7f51\u903b\u8f91\u4ea4\u6362\u673a\u3002 \u5982\u679c\u5b9e\u9645\u4f7f\u7528\u4e2d\u6ca1\u6709 vlan\uff08\u4f7f\u7528 vlan 0\uff09\uff0c\u90a3\u4e48\u4e0b\u8ff0\u542f\u52a8\u53c2\u6570\u65e0\u9700\u914d\u7f6e\u3002 # \u90e8\u7f72\u7684\u65f6\u5019\u4f60\u9700\u8981\u53c2\u8003\u4ee5\u4e0a\u573a\u666f\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\uff0c\u6309\u9700\u6307\u5b9a\u5982\u4e0b\u53c2\u6570 # 1. kube-ovn-controller \u542f\u52a8\u53c2\u6570\u9700\u8981\u914d\u7f6e\uff1a - --external-gateway-vlanid = 204 - --external-gateway-switch = external204 # 2. kube-ovn-cni \u542f\u52a8\u53c2\u6570\u9700\u8981\u914d\u7f6e: - --external-gateway-switch = external204 ### \u4ee5\u4e0a\u914d\u7f6e\u90fd\u548c\u4e0b\u9762\u7684\u516c\u7f51\u7f51\u7edc\u914d\u7f6e vlan id \u548c\u8d44\u6e90\u540d\u4fdd\u6301\u4e00\u81f4\uff0c\u76ee\u524d\u4ec5\u652f\u6301\u6307\u5b9a\u4e00\u4e2a underlay \u516c\u7f51\u4f5c\u4e3a\u9ed8\u8ba4\u5916\u90e8\u516c\u7f51\u3002 \u8be5\u914d\u7f6e\u9879\u7684\u8bbe\u8ba1\u548c\u4f7f\u7528\u4e3b\u8981\u8003\u8651\u4e86\u5982\u4e0b\u56e0\u7d20\uff1a \u57fa\u4e8e\u8be5\u914d\u7f6e\u9879\u53ef\u4ee5\u5bf9\u63a5\u5230 provider network\uff0cvlan\uff0csubnet \u7684\u8d44\u6e90\u3002 \u57fa\u4e8e\u8be5\u914d\u7f6e\u9879\u53ef\u4ee5\u5c06\u9ed8\u8ba4 vpc enable_eip_snat \u529f\u80fd\u5bf9\u63a5\u5230\u5df2\u6709\u7684 vlan\uff0csubnet \u8d44\u6e90\uff0c\u540c\u65f6\u652f\u6301\u516c\u7f51 ip \u7684 ipam\u3002 \u5982\u679c\u4ec5\u4f7f\u7528\u9ed8\u8ba4 vpc \u7684 enable_eip_snat \u6a21\u5f0f, \u4e14\u4ec5\u4f7f\u7528\u65e7\u7684\u57fa\u4e8e pod annotaion \u7684 fip snat\uff0c\u90a3\u4e48\u8fd9\u4e2a\u914d\u7f6e\u65e0\u9700\u914d\u7f6e\u3002 \u57fa\u4e8e\u8be5\u914d\u7f6e\u53ef\u4ee5\u4e0d\u4f7f\u7528\u9ed8\u8ba4 vpc enable_eip_snat \u6d41\u7a0b\uff0c\u4ec5\u901a\u8fc7\u5bf9\u5e94\u5230 vlan\uff0csubnet \u6d41\u7a0b\uff0c\u53ef\u4ee5\u517c\u5bb9\u4ec5\u81ea\u5b9a\u4e49 vpc \u4f7f\u7528 eip snat \u7684\u4f7f\u7528\u573a\u666f\u3002 1.1 \u51c6\u5907 underlay \u516c\u7f51\u7f51\u7edc \u00b6 # \u51c6\u5907 provider-network\uff0c vlan\uff0c subnet # cat 01-provider-network.yaml apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: external204 spec: defaultInterface: vlan # cat 02-vlan.yaml apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan204 spec: id: 204 provider: external204 # cat 03-vlan-subnet.yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: external204 spec: protocol: IPv4 cidrBlock: 10 .5.204.0/24 gateway: 10 .5.204.254 vlan: vlan204 excludeIps: - 10 .5.204.1..10.5.204.100 1.2 \u9ed8\u8ba4 vpc \u542f\u7528 eip_snat \u00b6 # \u542f\u7528\u9ed8\u8ba4 vpc \u548c\u4e0a\u8ff0 underlay \u516c\u7f51 provider subnet \u4e92\u8054 cat 00 -centralized-external-gw-no-ip.yaml apiVersion: v1 kind: ConfigMap metadata: name: ovn-external-gw-config namespace: kube-system data: enable-external-gw: \"true\" external-gw-nodes: \"pc-node-1,pc-node-2,pc-node-3\" type: \"centralized\" external-gw-nic: \"vlan\" # \u7528\u4e8e\u63a5\u5165 ovs \u516c\u7f51\u7f51\u6865\u7684\u7f51\u5361 external-gw-addr: \"10.5.204.254/24\" # underlay \u7269\u7406\u7f51\u5173\u7684 ip \u76ee\u524d\u8be5\u529f\u80fd\u5df2\u652f\u6301\u53ef\u4ee5\u4e0d\u6307\u5b9a lrp ip \u548c mac\uff0c\u5df2\u652f\u6301\u81ea\u52a8\u83b7\u53d6\uff0c\u521b\u5efa lrp \u7c7b\u578b\u7684 ovn eip \u8d44\u6e90\u3002 \u5982\u679c\u6307\u5b9a\u4e86\uff0c\u5219\u76f8\u5f53\u4e8e\u6307\u5b9a ip \u521b\u5efa lrp \u7c7b\u578b\u7684 ovn-eip\u3002 \u5f53\u7136\u4e5f\u53ef\u4ee5\u63d0\u524d\u624b\u52a8\u521b\u5efa lrp \u7c7b\u578b\u7684 ovn eip\u3002 1.3 \u81ea\u5b9a\u4e49 vpc \u542f\u7528 eip snat fip \u529f\u80fd \u00b6 # cat 00-ns.yml apiVersion: v1 kind: Namespace metadata: name: vpc1 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true # vpc \u542f\u7528 enableExternal \u4f1a\u81ea\u52a8\u521b\u5efa lrp \u5173\u8054\u5230\u4e0a\u8ff0\u6307\u5b9a\u7684\u516c\u7f51 # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 # \u8fd9\u91cc\u5b50\u7f51\u548c\u4e4b\u524d\u4f7f\u7528\u5b50\u7f51\u4e00\u6837\uff0c\u8be5\u529f\u80fd\u5728 subnet \u4e0a\u6ca1\u6709\u65b0\u589e\u5c5e\u6027\uff0c\u6ca1\u6709\u4efb\u4f55\u53d8\u66f4 \u4ee5\u4e0a\u6a21\u677f\u5e94\u7528\u540e\uff0c\u5e94\u8be5\u53ef\u4ee5\u770b\u5230\u5982\u4e0b\u8d44\u6e90\u5b58\u5728 # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # \u76ee\u524d\u8be5\u8def\u7531\u5df2\u81ea\u52a8\u7ef4\u62a4 2. ovn-eip \u00b6 \u8be5\u529f\u80fd\u548c iptables-eip \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4\uff0covn-eip \u76ee\u524d\u6709\u4e09\u79cd type nat: \u7528\u4e8e ovn dnat\uff0cfip, snat, \u8fd9\u4e9b nat \u7c7b\u578b\u4f1a\u8bb0\u5f55\u5728 status \u4e2d lrp: Resources connected to the public network from a vpc can be used by nat lsp: \u7528\u4e8e ovn \u57fa\u4e8e bfd \u7684 ecmp \u9759\u6001\u8def\u7531\u573a\u666f\uff0c\u5728\u7f51\u5173\u8282\u70b9\u4e0a\u63d0\u4f9b\u4e00\u4e2a ovs internal port \u4f5c\u4e3a ecmp \u8def\u7531\u7684\u4e0b\u4e00\u8df3 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: nat # \u52a8\u6001\u5206\u914d\u4e00\u4e2a eip \u8d44\u6e90\uff0c\u8be5\u8d44\u6e90\u9884\u7559\u7528\u4e8e fip \u573a\u666f 2.1 ovn-fip \u4e3a pod \u7ed1\u5b9a\u4e00\u4e2a fip \u00b6 # k get po -o wide -n vpc1 vpc-1-busybox01 NAME READY STATUS RESTARTS AGE IP NODE vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 # k get ip vpc-1-busybox01.vpc1 NAME V4IP V6IP MAC NODE SUBNET vpc-1-busybox01.vpc1 192 .168.0.2 00 :00:00:0A:DD:27 pc-node-2 vpc1-subnet1 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: nat --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: ovnEip: eip-static ipName: vpc-1-busybox01.vpc1 # \u6ce8\u610f\u8fd9\u91cc\u662f ip crd \u7684\u540d\u5b57\uff0c\u5177\u6709\u552f\u4e00\u6027 # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 # k get ofip eip-static NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 [ root@pc-node-1 03 -cust-vpc ] # ping 10.5.204.101 PING 10 .5.204.101 ( 10 .5.204.101 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.101: icmp_seq = 2 ttl = 62 time = 1 .21 ms 64 bytes from 10 .5.204.101: icmp_seq = 3 ttl = 62 time = 0 .624 ms 64 bytes from 10 .5.204.101: icmp_seq = 4 ttl = 62 time = 0 .368 ms ^C --- 10 .5.204.101 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3049ms rtt min/avg/max/mdev = 0 .368/0.734/1.210/0.352 ms [ root@pc-node-1 03 -cust-vpc ] # # \u53ef\u4ee5\u770b\u5230\u5728 node ping \u9ed8\u8ba4 vpc \u4e0b\u7684 pod \u7684\u516c\u7f51 ip \u662f\u80fd\u901a\u7684 # \u8be5\u516c\u7f51 ip \u80fd\u901a\u7684\u5173\u952e\u8d44\u6e90\u4e3b\u8981\u5305\u62ec\u4ee5\u4e0b\u90e8\u5206 # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 813523e7-c68c-408f-bd8c-cba30cb2e4f4 external ip: \"10.5.204.101\" logical ip: \"192.168.0.2\" type: \"dnat_and_snat\" 2.2 ovn-fip \u4e3a vip \u7ed1\u5b9a\u4e00\u4e2a fip \u00b6 \u4e3a\u4e86\u4fbf\u4e8e\u4e00\u4e9b vip \u573a\u666f\u7684\u4f7f\u7528\uff0c\u6bd4\u5982 kubevirt \u865a\u62df\u673a\u5185\u90e8\u6211\u53ef\u80fd\u4f1a\u4f7f\u7528\u4e00\u4e9b vip \u63d0\u4f9b\u7ed9 keepalived\uff0ckube-vip \u7b49\u573a\u666f\u6765\u4f7f\u7528\uff0c\u540c\u65f6\u652f\u6301\u516c\u7f51\u8bbf\u95ee\u3002 \u90a3\u4e48\u53ef\u4ee5\u57fa\u4e8e fip \u7ed1\u5b9a vpc \u5185\u90e8\u7684 vip \u7684\u65b9\u5f0f\u6765\u63d0\u4f9b vip \u7684\u516c\u7f51\u80fd\u529b\u3002 # \u5148\u521b\u5efa vip\uff0ceip\uff0c\u518d\u5c06 eip \u7ed1\u5b9a\u5230 vip # cat vip.yaml apiVersion: kubeovn.io/v1 kind: Vip metadata: name: test-fip-vip spec: subnet: vpc1-subnet1 # cat 04-fip.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: externalSubnet: external204 type: nat --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: ovnEip: eip-for-vip ipType: vip # \u9ed8\u8ba4\u60c5\u51b5\u4e0b fip \u662f\u9762\u5411 pod ip \u7684\uff0c\u8fd9\u91cc\u9700\u8981\u6807\u6ce8\u6307\u5b9a\u5bf9\u63a5\u5230 vip \u8d44\u6e90 ipName: test-fip-vip # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip [ root@pc-node-1 fip-vip ] # ping 10.5.204.106 PING 10 .5.204.106 ( 10 .5.204.106 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.106: icmp_seq = 1 ttl = 62 time = 0 .694 ms 64 bytes from 10 .5.204.106: icmp_seq = 2 ttl = 62 time = 0 .436 ms # \u5728 node \u4e0a\u662f ping \u5f97\u901a\u7684 # pod \u5185\u90e8\u7684 ip \u4f7f\u7528\u65b9\u5f0f\u5927\u81f4\u5c31\u662f\u5982\u4e0b\u8fd9\u79cd\u60c5\u51b5 [ root@pc-node-1 fip-vip ] # k -n vpc1 exec -it vpc-1-busybox03 -- bash [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1568 : eth0@if1569: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:56:40:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.5/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 192 .168.0.3/24 scope global secondary eth0 # \u53ef\u4ee5\u770b\u5230 vip \u7684\u914d\u7f6e valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe56:40e5/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox03 / ] # tcpdump -i eth0 host 192.168.0.3 -netvv tcpdump: listening on eth0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:ed:8e:c7 > 00 :00:00:56:40:e5, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 44830 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.32.51 > 192 .168.0.3: ICMP echo request, id 177 , seq 1 , length 64 00 :00:00:56:40:e5 > 00 :00:00:ed:8e:c7, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 64 , id 43962 , offset 0 , flags [ none ] , proto ICMP ( 1 ) , length 84 ) 192 .168.0.3 > 10 .5.32.51: ICMP echo reply, id 177 , seq 1 , length 64 # pod \u5185\u90e8\u53ef\u4ee5\u6293\u5230 fip \u76f8\u5173\u7684 icmp \u5305 3. ovn-snat \u00b6 3.1 ovn-snat \u5bf9\u5e94\u4e00\u4e2a subnet \u7684 cidr \u00b6 \u8be5\u529f\u80fd\u548c iptables-snat \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4 # cat 03-subnet-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: externalSubnet: external204 type: nat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: ovnEip: snat-for-subnet-in-vpc vpcSubnet: vpc1-subnet1 # eip \u5bf9\u5e94\u6574\u4e2a\u7f51\u6bb5 3.2 ovn-fip \u5bf9\u5e94\u5230\u4e00\u4e2a pod ip \u00b6 \u8be5\u529f\u80fd\u548c iptables-fip \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4 # cat 03-pod-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-pod-vpc-ip spec: externalSubnet: external204 type: nat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat01 spec: ovnEip: snat-for-pod-vpc-ip ipName: vpc-1-busybox02.vpc1 # eip \u5bf9\u5e94\u5355\u4e2a pod ip \u4ee5\u4e0a\u8d44\u6e90\u521b\u5efa\u540e\uff0c\u53ef\u4ee5\u770b\u5230 snat \u516c\u7f51\u529f\u80fd\u4f9d\u8d56\u7684\u5982\u4e0b\u8d44\u6e90\u3002 # kubectl ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" nat da77a11f-c523-439c-b1d1-72c664196a0f external ip: \"10.5.204.116\" logical ip: \"192.168.0.4\" type: \"snat\" [ root@pc-node-1 03 -cust-vpc ] # k get po -A -o wide | grep busy vpc1 vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 <none> <none> vpc1 vpc-1-busybox02 1 /1 Running 0 17h 192 .168.0.4 pc-node-1 <none> <none> vpc1 vpc-1-busybox03 1 /1 Running 0 17h 192 .168.0.5 pc-node-1 <none> <none> vpc1 vpc-1-busybox04 1 /1 Running 0 17h 192 .168.0.6 pc-node-3 <none> <none> vpc1 vpc-1-busybox05 1 /1 Running 0 17h 192 .168.0.7 pc-node-1 <none> <none> # k exec -it -n vpc1 vpc-1-busybox04 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17095 : eth0@if17096: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:76:94:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.6/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe76:9455/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox04 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 114 time = 22 .2 ms 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 21 .8 ms [ root@pc-node-1 03 -cust-vpc ] # k exec -it -n vpc1 vpc-1-busybox02 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1566 : eth0@if1567: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:0b:e9:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.4/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe0b:e9d0/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox02 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 22 .7 ms 64 bytes from 223 .5.5.5: icmp_seq = 3 ttl = 114 time = 22 .6 ms 64 bytes from 223 .5.5.5: icmp_seq = 4 ttl = 114 time = 22 .1 ms ^C --- 223 .5.5.5 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3064ms rtt min/avg/max/mdev = 22 .126/22.518/22.741/0.278 ms # \u53ef\u4ee5\u770b\u5230\u4e24\u4e2a pod \u53ef\u4ee5\u5206\u522b\u57fa\u4e8e\u8fd9\u4e24\u79cd snat \u8d44\u6e90\u4e0a\u5916\u7f51 4. ovn-dnat \u00b6 4.1 ovn-dnat \u4e3a pod \u7ed1\u5b9a\u4e00\u4e2a dnat \u00b6 kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : externalSubnet : underlay --- kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ovnEip : eip-dnat ipName : vpc-1-busybox01.vpc1 # \u6ce8\u610f\u8fd9\u91cc\u662f pod ip crd \u7684\u540d\u5b57\uff0c\u5177\u6709\u552f\u4e00\u6027 protocol : tcp internalPort : \"22\" externalPort : \"22\" OvnDnatRule \u7684\u914d\u7f6e\u4e0e IptablesDnatRule \u7c7b\u4f3c # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.3 22 22 vpc-1-busybox01.vpc1 true 4.2 ovn-dnat \u4e3a vip \u7ed1\u5b9a\u4e00\u4e2a dnat \u00b6 kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ipType : vip # \u9ed8\u8ba4\u60c5\u51b5\u4e0b dnat \u662f\u9762\u5411 pod ip \u7684\uff0c\u8fd9\u91cc\u9700\u8981\u6807\u6ce8\u6307\u5b9a\u5bf9\u63a5\u5230 vip \u8d44\u6e90 ovnEip : eip-dnat ipName : test-dnat-vip protocol : tcp internalPort : \"22\" externalPort : \"22\" OvnDnatRule \u7684\u914d\u7f6e\u4e0e IptablesDnatRule \u7c7b\u4f3c # kubectl get vip test-dnat-vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY test-dnat-vip 192 .168.0.4 00 :00:00:D0:C0:B5 vpc1-subnet1 true # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat eip-dnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.4 22 22 test-dnat-vip true \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN EIP FIP SNAT \u652f\u6301"},{"location":"advance/ovn-eip-fip-snat/#ovn-eip-fip-snat-dnat","text":"graph LR pod-->vpc1-subnet-->vpc1-->snat-->lrp-->external-subnet-->gw-node-external-nic Pod \u57fa\u4e8e SNAT \u51fa\u516c\u7f51\u7684\u5927\u81f4\u6d41\u7a0b\uff0c\u6700\u540e\u662f\u7ecf\u8fc7\u7f51\u5173\u8282\u70b9\u7684\u516c\u7f51\u7f51\u5361\u3002 graph LR pod-->vpc1-subnet-->vpc1-->fip-->lrp-->external-subnet-->local-node-external-nic Pod \u57fa\u4e8e FIP \u51fa\u516c\u7f51\u7684\u5927\u81f4\u6d41\u7a0b\uff0c\u6700\u540e\u53ef\u4ee5\u57fa\u4e8e\u672c\u5730\u8282\u70b9\u7684\u516c\u7f51\u7f51\u5361\u51fa\u516c\u7f51\u3002 \u8be5\u529f\u80fd\u6240\u652f\u6301\u7684 CRD \u5728\u4f7f\u7528\u4e0a\u5c06\u548c iptable nat gw \u516c\u7f51\u65b9\u6848\u4fdd\u6301\u57fa\u672c\u4e00\u81f4\u3002 ovn eip: \u7528\u4e8e\u516c\u7f51 ip \u5360\u4f4d\uff0c\u4ece underlay provider network vlan subnet \u4e2d\u5206\u914d ovn fip\uff1a \u4e00\u5bf9\u4e00 dnat snat\uff0c\u4e3a vpc \u5185\u7684 ip \u6216\u8005 vip \u63d0\u4f9b\u516c\u7f51\u76f4\u63a5\u8bbf\u95ee\u80fd\u529b ovn snat\uff1a\u6574\u4e2a\u5b50\u7f51\u6216\u8005\u5355\u4e2a vpc \u5185 ip \u53ef\u4ee5\u57fa\u4e8e snat \u8bbf\u95ee\u516c\u7f51 ovn dnat\uff1a\u57fa\u4e8e router lb \u5b9e\u73b0, \u57fa\u4e8e\u516c\u7f51 ip + \u7aef\u53e3 \u76f4\u63a5\u8bbf\u95ee vpc \u5185\u7684 \u4e00\u7ec4 endpoints","title":"OVN EIP FIP SNAT DNAT \u652f\u6301"},{"location":"advance/ovn-eip-fip-snat/#1","text":"\u76ee\u524d\u5141\u8bb8\u6240\u6709\uff08\u9ed8\u8ba4\u4ee5\u53ca\u81ea\u5b9a\u4e49\uff09vpc \u4f7f\u7528\u540c\u4e00\u4e2a provider vlan subnet \u8d44\u6e90\uff0c\u540c\u65f6\u517c\u5bb9 \u9ed8\u8ba4 VPC EIP/SNAT \u7684\u573a\u666f\u3002 \u7c7b\u4f3c neutron ovn\uff0c\u670d\u52a1\u542f\u52a8\u914d\u7f6e\u4e2d\u9700\u8981\u6307\u5b9a provider network \u76f8\u5173\u7684\u914d\u7f6e\uff0c\u4e0b\u8ff0\u7684\u542f\u52a8\u53c2\u6570\u4e5f\u662f\u4e3a\u4e86\u517c\u5bb9 VPC EIP/SNAT \u7684\u5b9e\u73b0\u3002 \u90e8\u7f72\u9636\u6bb5\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\uff0c\u53ef\u80fd\u9700\u8981\u6307\u5b9a\u9ed8\u8ba4\u516c\u7f51\u903b\u8f91\u4ea4\u6362\u673a\u3002 \u5982\u679c\u5b9e\u9645\u4f7f\u7528\u4e2d\u6ca1\u6709 vlan\uff08\u4f7f\u7528 vlan 0\uff09\uff0c\u90a3\u4e48\u4e0b\u8ff0\u542f\u52a8\u53c2\u6570\u65e0\u9700\u914d\u7f6e\u3002 # \u90e8\u7f72\u7684\u65f6\u5019\u4f60\u9700\u8981\u53c2\u8003\u4ee5\u4e0a\u573a\u666f\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\uff0c\u6309\u9700\u6307\u5b9a\u5982\u4e0b\u53c2\u6570 # 1. kube-ovn-controller \u542f\u52a8\u53c2\u6570\u9700\u8981\u914d\u7f6e\uff1a - --external-gateway-vlanid = 204 - --external-gateway-switch = external204 # 2. kube-ovn-cni \u542f\u52a8\u53c2\u6570\u9700\u8981\u914d\u7f6e: - --external-gateway-switch = external204 ### \u4ee5\u4e0a\u914d\u7f6e\u90fd\u548c\u4e0b\u9762\u7684\u516c\u7f51\u7f51\u7edc\u914d\u7f6e vlan id \u548c\u8d44\u6e90\u540d\u4fdd\u6301\u4e00\u81f4\uff0c\u76ee\u524d\u4ec5\u652f\u6301\u6307\u5b9a\u4e00\u4e2a underlay \u516c\u7f51\u4f5c\u4e3a\u9ed8\u8ba4\u5916\u90e8\u516c\u7f51\u3002 \u8be5\u914d\u7f6e\u9879\u7684\u8bbe\u8ba1\u548c\u4f7f\u7528\u4e3b\u8981\u8003\u8651\u4e86\u5982\u4e0b\u56e0\u7d20\uff1a \u57fa\u4e8e\u8be5\u914d\u7f6e\u9879\u53ef\u4ee5\u5bf9\u63a5\u5230 provider network\uff0cvlan\uff0csubnet \u7684\u8d44\u6e90\u3002 \u57fa\u4e8e\u8be5\u914d\u7f6e\u9879\u53ef\u4ee5\u5c06\u9ed8\u8ba4 vpc enable_eip_snat \u529f\u80fd\u5bf9\u63a5\u5230\u5df2\u6709\u7684 vlan\uff0csubnet \u8d44\u6e90\uff0c\u540c\u65f6\u652f\u6301\u516c\u7f51 ip \u7684 ipam\u3002 \u5982\u679c\u4ec5\u4f7f\u7528\u9ed8\u8ba4 vpc \u7684 enable_eip_snat \u6a21\u5f0f, \u4e14\u4ec5\u4f7f\u7528\u65e7\u7684\u57fa\u4e8e pod annotaion \u7684 fip snat\uff0c\u90a3\u4e48\u8fd9\u4e2a\u914d\u7f6e\u65e0\u9700\u914d\u7f6e\u3002 \u57fa\u4e8e\u8be5\u914d\u7f6e\u53ef\u4ee5\u4e0d\u4f7f\u7528\u9ed8\u8ba4 vpc enable_eip_snat \u6d41\u7a0b\uff0c\u4ec5\u901a\u8fc7\u5bf9\u5e94\u5230 vlan\uff0csubnet \u6d41\u7a0b\uff0c\u53ef\u4ee5\u517c\u5bb9\u4ec5\u81ea\u5b9a\u4e49 vpc \u4f7f\u7528 eip snat \u7684\u4f7f\u7528\u573a\u666f\u3002","title":"1. \u90e8\u7f72"},{"location":"advance/ovn-eip-fip-snat/#11-underlay","text":"# \u51c6\u5907 provider-network\uff0c vlan\uff0c subnet # cat 01-provider-network.yaml apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: external204 spec: defaultInterface: vlan # cat 02-vlan.yaml apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan204 spec: id: 204 provider: external204 # cat 03-vlan-subnet.yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: external204 spec: protocol: IPv4 cidrBlock: 10 .5.204.0/24 gateway: 10 .5.204.254 vlan: vlan204 excludeIps: - 10 .5.204.1..10.5.204.100","title":"1.1 \u51c6\u5907 underlay \u516c\u7f51\u7f51\u7edc"},{"location":"advance/ovn-eip-fip-snat/#12-vpc-eip_snat","text":"# \u542f\u7528\u9ed8\u8ba4 vpc \u548c\u4e0a\u8ff0 underlay \u516c\u7f51 provider subnet \u4e92\u8054 cat 00 -centralized-external-gw-no-ip.yaml apiVersion: v1 kind: ConfigMap metadata: name: ovn-external-gw-config namespace: kube-system data: enable-external-gw: \"true\" external-gw-nodes: \"pc-node-1,pc-node-2,pc-node-3\" type: \"centralized\" external-gw-nic: \"vlan\" # \u7528\u4e8e\u63a5\u5165 ovs \u516c\u7f51\u7f51\u6865\u7684\u7f51\u5361 external-gw-addr: \"10.5.204.254/24\" # underlay \u7269\u7406\u7f51\u5173\u7684 ip \u76ee\u524d\u8be5\u529f\u80fd\u5df2\u652f\u6301\u53ef\u4ee5\u4e0d\u6307\u5b9a lrp ip \u548c mac\uff0c\u5df2\u652f\u6301\u81ea\u52a8\u83b7\u53d6\uff0c\u521b\u5efa lrp \u7c7b\u578b\u7684 ovn eip \u8d44\u6e90\u3002 \u5982\u679c\u6307\u5b9a\u4e86\uff0c\u5219\u76f8\u5f53\u4e8e\u6307\u5b9a ip \u521b\u5efa lrp \u7c7b\u578b\u7684 ovn-eip\u3002 \u5f53\u7136\u4e5f\u53ef\u4ee5\u63d0\u524d\u624b\u52a8\u521b\u5efa lrp \u7c7b\u578b\u7684 ovn eip\u3002","title":"1.2 \u9ed8\u8ba4 vpc \u542f\u7528 eip_snat"},{"location":"advance/ovn-eip-fip-snat/#13-vpc-eip-snat-fip","text":"# cat 00-ns.yml apiVersion: v1 kind: Namespace metadata: name: vpc1 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true # vpc \u542f\u7528 enableExternal \u4f1a\u81ea\u52a8\u521b\u5efa lrp \u5173\u8054\u5230\u4e0a\u8ff0\u6307\u5b9a\u7684\u516c\u7f51 # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 # \u8fd9\u91cc\u5b50\u7f51\u548c\u4e4b\u524d\u4f7f\u7528\u5b50\u7f51\u4e00\u6837\uff0c\u8be5\u529f\u80fd\u5728 subnet \u4e0a\u6ca1\u6709\u65b0\u589e\u5c5e\u6027\uff0c\u6ca1\u6709\u4efb\u4f55\u53d8\u66f4 \u4ee5\u4e0a\u6a21\u677f\u5e94\u7528\u540e\uff0c\u5e94\u8be5\u53ef\u4ee5\u770b\u5230\u5982\u4e0b\u8d44\u6e90\u5b58\u5728 # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # \u76ee\u524d\u8be5\u8def\u7531\u5df2\u81ea\u52a8\u7ef4\u62a4","title":"1.3 \u81ea\u5b9a\u4e49 vpc \u542f\u7528 eip snat fip \u529f\u80fd"},{"location":"advance/ovn-eip-fip-snat/#2-ovn-eip","text":"\u8be5\u529f\u80fd\u548c iptables-eip \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4\uff0covn-eip \u76ee\u524d\u6709\u4e09\u79cd type nat: \u7528\u4e8e ovn dnat\uff0cfip, snat, \u8fd9\u4e9b nat \u7c7b\u578b\u4f1a\u8bb0\u5f55\u5728 status \u4e2d lrp: Resources connected to the public network from a vpc can be used by nat lsp: \u7528\u4e8e ovn \u57fa\u4e8e bfd \u7684 ecmp \u9759\u6001\u8def\u7531\u573a\u666f\uff0c\u5728\u7f51\u5173\u8282\u70b9\u4e0a\u63d0\u4f9b\u4e00\u4e2a ovs internal port \u4f5c\u4e3a ecmp \u8def\u7531\u7684\u4e0b\u4e00\u8df3 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: nat # \u52a8\u6001\u5206\u914d\u4e00\u4e2a eip \u8d44\u6e90\uff0c\u8be5\u8d44\u6e90\u9884\u7559\u7528\u4e8e fip \u573a\u666f","title":"2. ovn-eip"},{"location":"advance/ovn-eip-fip-snat/#21-ovn-fip-pod-fip","text":"# k get po -o wide -n vpc1 vpc-1-busybox01 NAME READY STATUS RESTARTS AGE IP NODE vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 # k get ip vpc-1-busybox01.vpc1 NAME V4IP V6IP MAC NODE SUBNET vpc-1-busybox01.vpc1 192 .168.0.2 00 :00:00:0A:DD:27 pc-node-2 vpc1-subnet1 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: nat --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: ovnEip: eip-static ipName: vpc-1-busybox01.vpc1 # \u6ce8\u610f\u8fd9\u91cc\u662f ip crd \u7684\u540d\u5b57\uff0c\u5177\u6709\u552f\u4e00\u6027 # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 # k get ofip eip-static NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 [ root@pc-node-1 03 -cust-vpc ] # ping 10.5.204.101 PING 10 .5.204.101 ( 10 .5.204.101 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.101: icmp_seq = 2 ttl = 62 time = 1 .21 ms 64 bytes from 10 .5.204.101: icmp_seq = 3 ttl = 62 time = 0 .624 ms 64 bytes from 10 .5.204.101: icmp_seq = 4 ttl = 62 time = 0 .368 ms ^C --- 10 .5.204.101 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3049ms rtt min/avg/max/mdev = 0 .368/0.734/1.210/0.352 ms [ root@pc-node-1 03 -cust-vpc ] # # \u53ef\u4ee5\u770b\u5230\u5728 node ping \u9ed8\u8ba4 vpc \u4e0b\u7684 pod \u7684\u516c\u7f51 ip \u662f\u80fd\u901a\u7684 # \u8be5\u516c\u7f51 ip \u80fd\u901a\u7684\u5173\u952e\u8d44\u6e90\u4e3b\u8981\u5305\u62ec\u4ee5\u4e0b\u90e8\u5206 # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 813523e7-c68c-408f-bd8c-cba30cb2e4f4 external ip: \"10.5.204.101\" logical ip: \"192.168.0.2\" type: \"dnat_and_snat\"","title":"2.1 ovn-fip \u4e3a pod \u7ed1\u5b9a\u4e00\u4e2a fip"},{"location":"advance/ovn-eip-fip-snat/#22-ovn-fip-vip-fip","text":"\u4e3a\u4e86\u4fbf\u4e8e\u4e00\u4e9b vip \u573a\u666f\u7684\u4f7f\u7528\uff0c\u6bd4\u5982 kubevirt \u865a\u62df\u673a\u5185\u90e8\u6211\u53ef\u80fd\u4f1a\u4f7f\u7528\u4e00\u4e9b vip \u63d0\u4f9b\u7ed9 keepalived\uff0ckube-vip \u7b49\u573a\u666f\u6765\u4f7f\u7528\uff0c\u540c\u65f6\u652f\u6301\u516c\u7f51\u8bbf\u95ee\u3002 \u90a3\u4e48\u53ef\u4ee5\u57fa\u4e8e fip \u7ed1\u5b9a vpc \u5185\u90e8\u7684 vip \u7684\u65b9\u5f0f\u6765\u63d0\u4f9b vip \u7684\u516c\u7f51\u80fd\u529b\u3002 # \u5148\u521b\u5efa vip\uff0ceip\uff0c\u518d\u5c06 eip \u7ed1\u5b9a\u5230 vip # cat vip.yaml apiVersion: kubeovn.io/v1 kind: Vip metadata: name: test-fip-vip spec: subnet: vpc1-subnet1 # cat 04-fip.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: externalSubnet: external204 type: nat --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: ovnEip: eip-for-vip ipType: vip # \u9ed8\u8ba4\u60c5\u51b5\u4e0b fip \u662f\u9762\u5411 pod ip \u7684\uff0c\u8fd9\u91cc\u9700\u8981\u6807\u6ce8\u6307\u5b9a\u5bf9\u63a5\u5230 vip \u8d44\u6e90 ipName: test-fip-vip # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip [ root@pc-node-1 fip-vip ] # ping 10.5.204.106 PING 10 .5.204.106 ( 10 .5.204.106 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.106: icmp_seq = 1 ttl = 62 time = 0 .694 ms 64 bytes from 10 .5.204.106: icmp_seq = 2 ttl = 62 time = 0 .436 ms # \u5728 node \u4e0a\u662f ping \u5f97\u901a\u7684 # pod \u5185\u90e8\u7684 ip \u4f7f\u7528\u65b9\u5f0f\u5927\u81f4\u5c31\u662f\u5982\u4e0b\u8fd9\u79cd\u60c5\u51b5 [ root@pc-node-1 fip-vip ] # k -n vpc1 exec -it vpc-1-busybox03 -- bash [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1568 : eth0@if1569: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:56:40:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.5/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 192 .168.0.3/24 scope global secondary eth0 # \u53ef\u4ee5\u770b\u5230 vip \u7684\u914d\u7f6e valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe56:40e5/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox03 / ] # tcpdump -i eth0 host 192.168.0.3 -netvv tcpdump: listening on eth0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:ed:8e:c7 > 00 :00:00:56:40:e5, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 44830 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.32.51 > 192 .168.0.3: ICMP echo request, id 177 , seq 1 , length 64 00 :00:00:56:40:e5 > 00 :00:00:ed:8e:c7, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 64 , id 43962 , offset 0 , flags [ none ] , proto ICMP ( 1 ) , length 84 ) 192 .168.0.3 > 10 .5.32.51: ICMP echo reply, id 177 , seq 1 , length 64 # pod \u5185\u90e8\u53ef\u4ee5\u6293\u5230 fip \u76f8\u5173\u7684 icmp \u5305","title":"2.2 ovn-fip \u4e3a vip \u7ed1\u5b9a\u4e00\u4e2a fip"},{"location":"advance/ovn-eip-fip-snat/#3-ovn-snat","text":"","title":"3. ovn-snat"},{"location":"advance/ovn-eip-fip-snat/#31-ovn-snat-subnet-cidr","text":"\u8be5\u529f\u80fd\u548c iptables-snat \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4 # cat 03-subnet-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: externalSubnet: external204 type: nat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: ovnEip: snat-for-subnet-in-vpc vpcSubnet: vpc1-subnet1 # eip \u5bf9\u5e94\u6574\u4e2a\u7f51\u6bb5","title":"3.1 ovn-snat \u5bf9\u5e94\u4e00\u4e2a subnet \u7684 cidr"},{"location":"advance/ovn-eip-fip-snat/#32-ovn-fip-pod-ip","text":"\u8be5\u529f\u80fd\u548c iptables-fip \u8bbe\u8ba1\u548c\u4f7f\u7528\u65b9\u5f0f\u57fa\u672c\u4e00\u81f4 # cat 03-pod-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-pod-vpc-ip spec: externalSubnet: external204 type: nat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat01 spec: ovnEip: snat-for-pod-vpc-ip ipName: vpc-1-busybox02.vpc1 # eip \u5bf9\u5e94\u5355\u4e2a pod ip \u4ee5\u4e0a\u8d44\u6e90\u521b\u5efa\u540e\uff0c\u53ef\u4ee5\u770b\u5230 snat \u516c\u7f51\u529f\u80fd\u4f9d\u8d56\u7684\u5982\u4e0b\u8d44\u6e90\u3002 # kubectl ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" nat da77a11f-c523-439c-b1d1-72c664196a0f external ip: \"10.5.204.116\" logical ip: \"192.168.0.4\" type: \"snat\" [ root@pc-node-1 03 -cust-vpc ] # k get po -A -o wide | grep busy vpc1 vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 <none> <none> vpc1 vpc-1-busybox02 1 /1 Running 0 17h 192 .168.0.4 pc-node-1 <none> <none> vpc1 vpc-1-busybox03 1 /1 Running 0 17h 192 .168.0.5 pc-node-1 <none> <none> vpc1 vpc-1-busybox04 1 /1 Running 0 17h 192 .168.0.6 pc-node-3 <none> <none> vpc1 vpc-1-busybox05 1 /1 Running 0 17h 192 .168.0.7 pc-node-1 <none> <none> # k exec -it -n vpc1 vpc-1-busybox04 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17095 : eth0@if17096: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:76:94:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.6/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe76:9455/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox04 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 114 time = 22 .2 ms 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 21 .8 ms [ root@pc-node-1 03 -cust-vpc ] # k exec -it -n vpc1 vpc-1-busybox02 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1566 : eth0@if1567: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:0b:e9:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.4/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe0b:e9d0/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox02 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 22 .7 ms 64 bytes from 223 .5.5.5: icmp_seq = 3 ttl = 114 time = 22 .6 ms 64 bytes from 223 .5.5.5: icmp_seq = 4 ttl = 114 time = 22 .1 ms ^C --- 223 .5.5.5 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3064ms rtt min/avg/max/mdev = 22 .126/22.518/22.741/0.278 ms # \u53ef\u4ee5\u770b\u5230\u4e24\u4e2a pod \u53ef\u4ee5\u5206\u522b\u57fa\u4e8e\u8fd9\u4e24\u79cd snat \u8d44\u6e90\u4e0a\u5916\u7f51","title":"3.2 ovn-fip \u5bf9\u5e94\u5230\u4e00\u4e2a pod ip"},{"location":"advance/ovn-eip-fip-snat/#4-ovn-dnat","text":"","title":"4. ovn-dnat"},{"location":"advance/ovn-eip-fip-snat/#41-ovn-dnat-pod-dnat","text":"kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : externalSubnet : underlay --- kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ovnEip : eip-dnat ipName : vpc-1-busybox01.vpc1 # \u6ce8\u610f\u8fd9\u91cc\u662f pod ip crd \u7684\u540d\u5b57\uff0c\u5177\u6709\u552f\u4e00\u6027 protocol : tcp internalPort : \"22\" externalPort : \"22\" OvnDnatRule \u7684\u914d\u7f6e\u4e0e IptablesDnatRule \u7c7b\u4f3c # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.3 22 22 vpc-1-busybox01.vpc1 true","title":"4.1 ovn-dnat \u4e3a pod \u7ed1\u5b9a\u4e00\u4e2a dnat"},{"location":"advance/ovn-eip-fip-snat/#42-ovn-dnat-vip-dnat","text":"kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ipType : vip # \u9ed8\u8ba4\u60c5\u51b5\u4e0b dnat \u662f\u9762\u5411 pod ip \u7684\uff0c\u8fd9\u91cc\u9700\u8981\u6807\u6ce8\u6307\u5b9a\u5bf9\u63a5\u5230 vip \u8d44\u6e90 ovnEip : eip-dnat ipName : test-dnat-vip protocol : tcp internalPort : \"22\" externalPort : \"22\" OvnDnatRule \u7684\u914d\u7f6e\u4e0e IptablesDnatRule \u7c7b\u4f3c # kubectl get vip test-dnat-vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY test-dnat-vip 192 .168.0.4 00 :00:00:D0:C0:B5 vpc1-subnet1 true # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat eip-dnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.4 22 22 test-dnat-vip true \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"4.2 ovn-dnat \u4e3a vip \u7ed1\u5b9a\u4e00\u4e2a dnat"},{"location":"advance/ovn-ipsec/","text":"\u4f7f\u7528 IPsec \u52a0\u5bc6\u8282\u70b9\u95f4\u901a\u4fe1 \u00b6 \u8be5\u529f\u80fd\u4ece v1.10.11 \u548c v1.11.4 \u540e\u5f00\u59cb\u652f\u6301\uff0ckernel \u7248\u672c\u81f3\u5c11\u662f 3.10.0 \u4ee5\u4e0a\u3002 \u542f\u52a8 IPsec \u00b6 \u4ece Kube-OVN \u6e90\u7801\u62f7\u8d1d\u811a\u672c ipsec.sh \uff0c\u6267\u884c\u547d\u4ee4\u5982\u4e0b\uff0c\u8be5\u811a\u672c\u4f1a\u8c03\u7528 ovs-pki \u751f\u6210\u548c\u5206\u914d\u52a0\u5bc6\u9700\u8981\u7684\u8bc1\u4e66\uff1a bash ipsec.sh init \u6267\u884c\u5b8c\u6bd5\u540e\uff0c\u8282\u70b9\u4e4b\u95f4\u4f1a\u534f\u5546\u4e00\u6bb5\u65f6\u95f4\u5efa\u7acb IPsec \u96a7\u9053\uff0c\u7ecf\u9a8c\u503c\u662f\u5341\u51e0\u79d2\u5230\u4e00\u5206\u949f\u4e4b\u95f4\uff0c\u53ef\u4ee5\u901a\u8fc7\u5982\u4e0b\u547d\u4ee4\u6765\u67e5\u770b IPsec \u72b6\u6001\uff1a # sh ipsec.sh status Pod { ovs-ovn-d7hdt } ipsec status... Interface name: ovn-a4718e-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.2 Remote IP: 172 .18.0.4 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem Remote cert: None Remote name: a4718e55-5b85-4f46-90e6-63527d080590 CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 2 CFM state: Disabled Kernel policies installed: src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 Kernel security associations installed: sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 IPsec connections that are active: Pod { ovs-ovn-fvbbj } ipsec status... Interface name: ovn-8aebd9-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.4 Remote IP: 172 .18.0.2 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem Local name: a4718e55-5b85-4f46-90e6-63527d080590 Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem Remote cert: None Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 1 CFM state: Disabled Kernel policies installed: src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 Kernel security associations installed: sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 IPsec connections that are active: \u5efa\u7acb\u5b8c\u6210\u540e\u53ef\u4ee5\u6293\u5305\u89c2\u5bdf\u62a5\u6587\u5df2\u7ecf\u88ab\u52a0\u5bc6\uff1a # tcpdump -i eth0 -nel esp 10 :01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP ( spi = 0xcc91322a,seq = 0x13d0 ) , length 156 10 :01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP ( spi = 0xc8df4221,seq = 0x1d37 ) , length 156 \u5f53\u6267\u884c\u5b8c\u811a\u672c\u540e\uff0c\u53ef\u4ee5\u901a\u8fc7\u6267\u884c\u547d\u4ee4\u5173\u95ed IPsec\uff1a # bash ipsec.sh stop \u6216\u8005\u6267\u884c\u547d\u4ee4\u518d\u6b21\u6253\u5f00\uff1a # bash ipsec.sh start \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN IPsec \u652f\u6301"},{"location":"advance/ovn-ipsec/#ipsec","text":"\u8be5\u529f\u80fd\u4ece v1.10.11 \u548c v1.11.4 \u540e\u5f00\u59cb\u652f\u6301\uff0ckernel \u7248\u672c\u81f3\u5c11\u662f 3.10.0 \u4ee5\u4e0a\u3002","title":"\u4f7f\u7528 IPsec \u52a0\u5bc6\u8282\u70b9\u95f4\u901a\u4fe1"},{"location":"advance/ovn-ipsec/#ipsec_1","text":"\u4ece Kube-OVN \u6e90\u7801\u62f7\u8d1d\u811a\u672c ipsec.sh \uff0c\u6267\u884c\u547d\u4ee4\u5982\u4e0b\uff0c\u8be5\u811a\u672c\u4f1a\u8c03\u7528 ovs-pki \u751f\u6210\u548c\u5206\u914d\u52a0\u5bc6\u9700\u8981\u7684\u8bc1\u4e66\uff1a bash ipsec.sh init \u6267\u884c\u5b8c\u6bd5\u540e\uff0c\u8282\u70b9\u4e4b\u95f4\u4f1a\u534f\u5546\u4e00\u6bb5\u65f6\u95f4\u5efa\u7acb IPsec \u96a7\u9053\uff0c\u7ecf\u9a8c\u503c\u662f\u5341\u51e0\u79d2\u5230\u4e00\u5206\u949f\u4e4b\u95f4\uff0c\u53ef\u4ee5\u901a\u8fc7\u5982\u4e0b\u547d\u4ee4\u6765\u67e5\u770b IPsec \u72b6\u6001\uff1a # sh ipsec.sh status Pod { ovs-ovn-d7hdt } ipsec status... Interface name: ovn-a4718e-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.2 Remote IP: 172 .18.0.4 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem Remote cert: None Remote name: a4718e55-5b85-4f46-90e6-63527d080590 CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 2 CFM state: Disabled Kernel policies installed: src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 Kernel security associations installed: sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 IPsec connections that are active: Pod { ovs-ovn-fvbbj } ipsec status... Interface name: ovn-8aebd9-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.4 Remote IP: 172 .18.0.2 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem Local name: a4718e55-5b85-4f46-90e6-63527d080590 Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem Remote cert: None Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 1 CFM state: Disabled Kernel policies installed: src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 Kernel security associations installed: sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 IPsec connections that are active: \u5efa\u7acb\u5b8c\u6210\u540e\u53ef\u4ee5\u6293\u5305\u89c2\u5bdf\u62a5\u6587\u5df2\u7ecf\u88ab\u52a0\u5bc6\uff1a # tcpdump -i eth0 -nel esp 10 :01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP ( spi = 0xcc91322a,seq = 0x13d0 ) , length 156 10 :01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP ( spi = 0xc8df4221,seq = 0x1d37 ) , length 156 \u5f53\u6267\u884c\u5b8c\u811a\u672c\u540e\uff0c\u53ef\u4ee5\u901a\u8fc7\u6267\u884c\u547d\u4ee4\u5173\u95ed IPsec\uff1a # bash ipsec.sh stop \u6216\u8005\u6267\u884c\u547d\u4ee4\u518d\u6b21\u6253\u5f00\uff1a # bash ipsec.sh start \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u542f\u52a8 IPsec"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/","text":"OVN SNAT \u57fa\u4e8e ECMP BFD \u9759\u6001\u8def\u7531\u7684 L3 HA \u652f\u6301 \u00b6 \u81ea\u5b9a\u4e49 vpc \u57fa\u4e8e ovn snat \u540e\u57fa\u4e8e ecmp \u9759\u6001\u8def\u7531\u54c8\u5e0c\u5230\u591a\u4e2a gw node ovnext0 \u7f51\u5361\u51fa\u516c\u7f51 \u652f\u6301\u57fa\u4e8e bfd \u7684\u9ad8\u53ef\u7528 \u4ec5\u652f\u6301 hash \u8d1f\u8f7d\u5747\u8861 graph LR pod-->vpc-subnet-->vpc-->snat-->ecmp-->external-subnet-->gw-node1-ovnext0--> node1-external-switch external-subnet-->gw-node2-ovnext0--> node2-external-switch external-subnet-->gw-node3-ovnext0--> node3-external-switch \u8be5\u529f\u80fd\u7684\u4f7f\u7528\u65b9\u5f0f\u548c ovn-eip-fip-snat.md \u57fa\u672c\u4e00\u81f4\uff0c\u4e00\u81f4\u7684\u90e8\u5206\u5305\u62ec install.sh \u7684\u90e8\u7f72\u90e8\u5206\uff0cprovider-network\uff0cvlan\uff0csubnet \u7684\u51c6\u5907\u90e8\u5206\u3002 \u81f3\u4e8e\u4e0d\u76f8\u540c\u7684\u90e8\u5206\uff0c\u4f1a\u5728\u4ee5\u4e0b\u90e8\u5206\u5177\u4f53\u9610\u8ff0\uff0c\u4e3b\u8981\u5305\u62ec lsp \u7c7b\u578b\u7684 ovn-eip \u7684\u521b\u5efa\uff0c\u4ee5\u53ca\u57fa\u4e8e vpc enable_bfd \u81ea\u52a8\u7ef4\u62a4 bfd \u4ee5\u53ca ecmp \u9759\u6001\u8def\u7531\u3002 1. \u90e8\u7f72 \u00b6 1.1 \u51c6\u5907 underlay \u516c\u7f51\u7f51\u7edc \u00b6 1.2 \u9ed8\u8ba4 vpc \u542f\u7528 eip_snat \u00b6 1.3 \u81ea\u5b9a\u4e49 vpc \u542f\u7528 eip snat fip \u529f\u80fd \u00b6 \u4ee5\u4e0a\u90e8\u5206\u548c ovn-eip-fip-snat.md \u5b8c\u5168\u4e00\u81f4\uff0c\u8fd9\u4e9b\u529f\u80fd\u9a8c\u8bc1\u901a\u8fc7\u540e\uff0c\u53ef\u4ee5\u76f4\u63a5\u57fa\u4e8e\u5982\u4e0b\u65b9\u5f0f\uff0c\u5c06 vpc \u5207\u6362\u5230\u57fa\u4e8e ecmp \u7684 bfd \u9759\u6001\u8def\u7531\uff0c\u5f53\u7136\u4e5f\u53ef\u4ee5\u5207\u56de\u3002 \u81ea\u5b9a\u4e49 vpc \u4f7f\u7528\u8be5\u529f\u80fd\u4e4b\u524d\uff0c\u9700\u8981\u5148\u63d0\u4f9b\u597d\u7f51\u5173\u8282\u70b9\uff0c\u81f3\u5c11\u9700\u8981\u63d0\u4f9b 2 \u4e2a\u4ee5\u4e0a\u7f51\u5173\u8282\u70b9\uff0c\u6ce8\u610f\u5f53\u524d\u5b9e\u73b0 ovn-eip \u7684\u540d\u5b57\u5fc5\u987b\u548c\u7f51\u5173\u8282\u70b9\u540d\u4fdd\u6301\u4e00\u81f4\uff0c\u76ee\u524d\u6ca1\u6709\u505a\u8be5\u8d44\u6e90\u7684\u81ea\u52a8\u5316\u7ef4\u62a4\u3002 # cat gw-node-eip.yaml --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-1 spec : externalSubnet : external204 type : lsp --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-2 spec : externalSubnet : external204 type : lsp --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-3 spec : externalSubnet : external204 type : lsp \u7531\u4e8e\u8fd9\u4e2a\u573a\u666f\u76ee\u524d\u8bbe\u8ba1\u4e0a\u662f\u4f9b vpc ecmp \u51fa\u516c\u7f51\u4f7f\u7528\uff0c\u6240\u4ee5\u4ee5\u4e0a\u5728\u6ca1\u6709 vpc \u542f\u7528 bfd \u7684\u65f6\u5019\uff0c\u5373\u4e0d\u5b58\u5728\u5e26\u6709 enable bfd \u6807\u7b7e\u7684 lrp \u7684 ovn eip \u7684\u65f6\u5019\uff0c\u7f51\u5173\u8282\u70b9\u4e0d\u4f1a\u89e6\u53d1\u521b\u5efa\u7f51\u5173\u7f51\u5361\uff0c\u4e5f\u65e0\u6cd5\u6210\u529f\u542f\u52a8\u5bf9\u7aef bfd \u4f1a\u8bdd\u7684\u76d1\u542c\u3002 2. \u81ea\u5b9a\u4e49 vpc \u542f\u7528 ecmp bfd L3 HA \u516c\u7f51\u529f\u80fd \u00b6 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true enableBfd: true # bfd \u5f00\u5173\u53ef\u4ee5\u968f\u610f\u5207\u6362\uff0c\u5f00\u8868\u793a\u542f\u7528 bfd ecmp \u8def\u7531 #enableBfd: false # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true # \u53ea\u9700\u5f00\u542f ecmp gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 \u4f7f\u7528\u4e0a\u7684\u6ce8\u610f\u70b9: \u81ea\u5b9a\u4e49 vpc \u4e0b\u7684 ecmp \u53ea\u7528\u9759\u6001 ecmp bfd \u8def\u7531\uff0cvpc enableBfd \u548c subnet enableEcmp \u540c\u65f6\u5f00\u542f\u7684\u60c5\u51b5\u4e0b\u624d\u4f1a\u751f\u6548\uff0c\u624d\u4f1a\u81ea\u52a8\u7ba1\u7406\u9759\u6001 ecmp bfd \u8def\u7531\u3002 \u4e0a\u8ff0\u914d\u7f6e\u5173\u95ed\u7684\u60c5\u51b5\u4e0b\uff0c\u4f1a\u81ea\u52a8\u5207\u56de\u5e38\u89c4\u9ed8\u8ba4\u9759\u6001\u8def\u7531\u3002 \u9ed8\u8ba4 vpc \u65e0\u6cd5\u4f7f\u7528\u8be5\u529f\u80fd\uff0c\u4ec5\u652f\u6301\u81ea\u5b9a\u4e49 vpc\uff0c\u9ed8\u8ba4 vpc \u6709\u66f4\u590d\u6742\u7684\u7b56\u7565\u8def\u7531\u4ee5\u53ca snat \u8bbe\u8ba1\u3002 \u81ea\u5b9a\u4e49 vpc \u7684 subnet \u7684 enableEcmp \u4ec5\u4f7f\u7528\u9759\u6001\u8def\u7531\uff0c\u7f51\u5173\u7c7b\u578b gatewayType \u6ca1\u6709\u4f5c\u7528\u3002 \u5f53\u5173\u95ed EnableExternal \u65f6\uff0cvpc \u5185\u65e0\u6cd5\u901a\u5916\u7f51\u3002 \u5f53\u5f00\u542f EnableExternal \u65f6\uff0c\u5173\u95ed EnableBfd \u65f6\uff0c\u4f1a\u57fa\u4e8e\u666e\u901a\u9ed8\u8ba4\u8def\u7531\u4e0a\u5916\u7f51\uff0c\u4e0d\u5177\u5907\u9ad8\u53ef\u7528\u3002 # \u4e0a\u8ff0\u6a21\u677f\u5e94\u7528\u540e ovn \u903b\u8f91\u5c42\u5e94\u8be5\u53ef\u4ee5\u770b\u5230\u5982\u4e0b\u8d44\u6e90 # \u67e5\u770b vpc # k get vpc NAME ENABLEEXTERNAL ENABLEBFD STANDBY SUBNETS NAMESPACES ovn-cluster true true [ \"external204\" , \"join\" , \"ovn-default\" ] vpc1 true true true [ \"vpc1-subnet1\" ] [ \"vpc1\" ] # \u9ed8\u8ba4 vpc \u672a\u652f\u6301 ENABLEBFD # \u81ea\u5b9a\u4e49 vpc \u5df2\u652f\u6301\u4e14\u5df2\u542f\u7528 # 1. \u521b\u5efa\u4e86 bfd \u4f1a\u8bdd # k ko nbctl list bfd _uuid : be7df545-2c4c-4751-878f-b3507987f050 detect_mult : 3 dst_ip : \"10.5.204.121\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : 684c4489-5b59-4693-8d8c-3beab93f8093 detect_mult : 3 dst_ip : \"10.5.204.109\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : f0f62077-2ae9-4e79-b4f8-a446ec6e784c detect_mult : 3 dst_ip : \"10.5.204.108\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up ### \u6ce8\u610f\u6240\u6709 status \u6b63\u5e38\u90fd\u5e94\u8be5\u662f up \u7684 # 2. \u521b\u5efa\u4e86\u57fa\u4e8e bfd \u7684\u9759\u6001\u8def\u7531 # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 192 .168.0.0/24 10 .5.204.108 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.109 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.121 src-ip ecmp ecmp-symmetric-reply bfd # 3. \u9759\u6001\u8def\u7531\u8be6\u60c5 # k ko nbctl find Logical_Router_Static_Route policy=src-ip options=ecmp_symmetric_reply=\"true\" _uuid : 3aacb384-d5ee-4b14-aebf-59e8c11717ba bfd : 684c4489-5b59-4693-8d8c-3beab93f8093 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.109\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 18bcc585-bc05-430b-925b-ef673c8e1aef bfd : f0f62077-2ae9-4e79-b4f8-a446ec6e784c external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.108\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 7d0a4e6b-cde0-4110-8176-fbaf19738498 bfd : be7df545-2c4c-4751-878f-b3507987f050 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.121\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" # \u540c\u65f6\u5728\u7f51\u5173\u8282\u70b9\u90fd\u5e94\u8be5\u5177\u5907\u4ee5\u4e0b\u8d44\u6e90 [ root@pc-node-1 ~ ] # ip netns exec ovnext bash ip a /usr/sbin/ip: /usr/sbin/ip: cannot execute binary file [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1541 : ovnext0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 00 :00:00:ab:bd:87 brd ff:ff:ff:ff:ff:ff inet 10 .5.204.108/24 brd 10 .5.204.255 scope global ovnext0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:feab:bd87/64 scope link valid_lft forever preferred_lft forever [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0 .0.0.0 10 .5.204.254 0 .0.0.0 UG 0 0 0 ovnext0 10 .5.204.0 0 .0.0.0 255 .255.255.0 U 0 0 0 ovnext0 ## \u6ce8\u610f\u4ee5\u4e0a\u5185\u5bb9\u548c\u4e00\u4e2a internal port unerlay \u516c\u7f51 pod \u5185\u90e8\u7684 ns \u5927\u81f4\u662f\u4e00\u81f4\u7684\uff0c\u8fd9\u91cc\u53ea\u662f\u5728\u7f51\u5173\u8282\u70b9\u4e0a\u5355\u72ec\u7ef4\u62a4\u4e86\u4e00\u4e2a ns [ root@pc-node-1 ~ ] # ip netns exec ovnext bfdd-control status There are 1 sessions: Session 1 id = 1 local = 10 .5.204.108 ( p ) remote = 10 .5.204.122 state = Up ## \u8fd9\u91cc\u5373\u662f lrp bfd \u4f1a\u8bdd\u7684\u53e6\u4e00\u7aef\uff0c\u4e5f\u662f lrp ecmp \u7684\u4e0b\u4e00\u8df3\u7684\u5176\u4e2d\u4e00\u4e2a [ root@pc-node-1 ~ ] # ip netns exec ovnext ping -c1 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 115 time = 21 .6 ms # \u5230\u516c\u7f51\u6ca1\u95ee\u9898 \u53ef\u4ee5\u5728\u67d0\u4e00\u4e2a\u7f51\u5173\u8282\u70b9\u7684 ovnext ns \u5185\u6293\u5230\u51fa\u53bb\u7684\u5305 # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-1 ~ ] # exit [ root@pc-node-1 ~ ] # ssh pc-node-2 Last login: Thu Feb 23 09 :21:08 2023 from 10 .5.32.51 [ root@pc-node-2 ~ ] # ip netns exec ovnext bash [ root@pc-node-2 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-2 ~ ] # exit [ root@pc-node-2 ~ ] # logout Connection to pc-node-2 closed. [ root@pc-node-1 ~ ] # ssh pc-node-3 Last login: Thu Feb 23 08 :32:41 2023 from 10 .5.32.51 [ root@pc-node-3 ~ ] # ip netns exec ovnext bash [ root@pc-node-3 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:2d:f8:ce > 00 :00:00:fd:b2:a4, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 63 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 00 :00:00:fd:b2:a4 > dc:ef:80:5a:44:1a, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [ root@pc-node-3 ~ ] # # \u53ef\u4ee5\u5728\u8be5\u8282\u70b9 down \u6389\u51fa\u53bb\u7684\u7f51\u5361\uff0c\u7136\u540e\u770b pod \u51fa\u53bb\u7684\u5305\u5728\u7f51\u7edc\u4e2d\u65ad\u4e2d\u4f1a\u51fa\u73b0\u51e0\u4e2a\u5305 # \u4e00\u822c\u90fd\u4f1a\u770b\u5230\u4e22 3 \u4e2a\u5305 3. \u5173\u95ed bfd \u6a21\u5f0f \u00b6 \u5728\u67d0\u4e9b\u573a\u666f\u4e0b\uff0c\u53ef\u80fd\u60f3\u76f4\u63a5\u4f7f\u7528\uff08\u96c6\u4e2d\u5f0f\uff09\u5355\u4e2a\u7f51\u5173\u76f4\u63a5\u51fa\u516c\u7f51\uff0c\u8fd9\u4e2a\u65f6\u5019\u548c\u9ed8\u8ba4 vpc enable_eip_snat \u7684\u4f7f\u7528\u6a21\u5f0f\u662f\u4e00\u81f4\u7684 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc2 spec: namespaces: - vpc2 enableExternal: true #enableBfd: true enableBfd: false ## \u5c06 bfd \u529f\u80fd\u76f4\u63a5\u7981\u7528\u5373\u53ef # k ko nbctl lr-route-list vpc2 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # \u5e94\u7528\u540e\u8def\u7531\u4f1a\u5207\u6362\u56de\u6b63\u5e38\u7684\u9ed8\u8ba4\u9759\u6001\u8def\u7531 # \u540c\u65f6 nbctl list bfd \u53ef\u4ee5\u770b\u5230 lrp \u5173\u8054\u7684 bfd \u4f1a\u8bdd\u5df2\u7ecf\u79fb\u9664 # \u800c\u4e14 ovnext ns \u4e2d\u7684\u5bf9\u7aef bfd \u4f1a\u8bdd\u4e5f\u81ea\u52a8\u79fb\u9664 # \u8be5\u5207\u6362\u8fc7\u7a0b\u4fdd\u6301 vpc subnet \u5185\u4fdd\u6301 ping \u672a\u770b\u5230(\u79d2\u7ea7)\u4e22\u5305 # \u518d\u5207\u6362\u56de\u53bb \u4e5f\u672a\u770b\u5230(\u79d2\u7ea7)\u4e22\u5305 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN SNAT \u57fa\u4e8e ECMP BFD \u9759\u6001\u8def\u7531\u7684 L3 HA \u652f\u6301"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#ovn-snat-ecmp-bfd-l3-ha","text":"\u81ea\u5b9a\u4e49 vpc \u57fa\u4e8e ovn snat \u540e\u57fa\u4e8e ecmp \u9759\u6001\u8def\u7531\u54c8\u5e0c\u5230\u591a\u4e2a gw node ovnext0 \u7f51\u5361\u51fa\u516c\u7f51 \u652f\u6301\u57fa\u4e8e bfd \u7684\u9ad8\u53ef\u7528 \u4ec5\u652f\u6301 hash \u8d1f\u8f7d\u5747\u8861 graph LR pod-->vpc-subnet-->vpc-->snat-->ecmp-->external-subnet-->gw-node1-ovnext0--> node1-external-switch external-subnet-->gw-node2-ovnext0--> node2-external-switch external-subnet-->gw-node3-ovnext0--> node3-external-switch \u8be5\u529f\u80fd\u7684\u4f7f\u7528\u65b9\u5f0f\u548c ovn-eip-fip-snat.md \u57fa\u672c\u4e00\u81f4\uff0c\u4e00\u81f4\u7684\u90e8\u5206\u5305\u62ec install.sh \u7684\u90e8\u7f72\u90e8\u5206\uff0cprovider-network\uff0cvlan\uff0csubnet \u7684\u51c6\u5907\u90e8\u5206\u3002 \u81f3\u4e8e\u4e0d\u76f8\u540c\u7684\u90e8\u5206\uff0c\u4f1a\u5728\u4ee5\u4e0b\u90e8\u5206\u5177\u4f53\u9610\u8ff0\uff0c\u4e3b\u8981\u5305\u62ec lsp \u7c7b\u578b\u7684 ovn-eip \u7684\u521b\u5efa\uff0c\u4ee5\u53ca\u57fa\u4e8e vpc enable_bfd \u81ea\u52a8\u7ef4\u62a4 bfd \u4ee5\u53ca ecmp \u9759\u6001\u8def\u7531\u3002","title":"OVN SNAT \u57fa\u4e8e ECMP BFD \u9759\u6001\u8def\u7531\u7684 L3 HA \u652f\u6301"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#1","text":"","title":"1. \u90e8\u7f72"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#11-underlay","text":"","title":"1.1 \u51c6\u5907 underlay \u516c\u7f51\u7f51\u7edc"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#12-vpc-eip_snat","text":"","title":"1.2 \u9ed8\u8ba4 vpc \u542f\u7528 eip_snat"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#13-vpc-eip-snat-fip","text":"\u4ee5\u4e0a\u90e8\u5206\u548c ovn-eip-fip-snat.md \u5b8c\u5168\u4e00\u81f4\uff0c\u8fd9\u4e9b\u529f\u80fd\u9a8c\u8bc1\u901a\u8fc7\u540e\uff0c\u53ef\u4ee5\u76f4\u63a5\u57fa\u4e8e\u5982\u4e0b\u65b9\u5f0f\uff0c\u5c06 vpc \u5207\u6362\u5230\u57fa\u4e8e ecmp \u7684 bfd \u9759\u6001\u8def\u7531\uff0c\u5f53\u7136\u4e5f\u53ef\u4ee5\u5207\u56de\u3002 \u81ea\u5b9a\u4e49 vpc \u4f7f\u7528\u8be5\u529f\u80fd\u4e4b\u524d\uff0c\u9700\u8981\u5148\u63d0\u4f9b\u597d\u7f51\u5173\u8282\u70b9\uff0c\u81f3\u5c11\u9700\u8981\u63d0\u4f9b 2 \u4e2a\u4ee5\u4e0a\u7f51\u5173\u8282\u70b9\uff0c\u6ce8\u610f\u5f53\u524d\u5b9e\u73b0 ovn-eip \u7684\u540d\u5b57\u5fc5\u987b\u548c\u7f51\u5173\u8282\u70b9\u540d\u4fdd\u6301\u4e00\u81f4\uff0c\u76ee\u524d\u6ca1\u6709\u505a\u8be5\u8d44\u6e90\u7684\u81ea\u52a8\u5316\u7ef4\u62a4\u3002 # cat gw-node-eip.yaml --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-1 spec : externalSubnet : external204 type : lsp --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-2 spec : externalSubnet : external204 type : lsp --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-3 spec : externalSubnet : external204 type : lsp \u7531\u4e8e\u8fd9\u4e2a\u573a\u666f\u76ee\u524d\u8bbe\u8ba1\u4e0a\u662f\u4f9b vpc ecmp \u51fa\u516c\u7f51\u4f7f\u7528\uff0c\u6240\u4ee5\u4ee5\u4e0a\u5728\u6ca1\u6709 vpc \u542f\u7528 bfd \u7684\u65f6\u5019\uff0c\u5373\u4e0d\u5b58\u5728\u5e26\u6709 enable bfd \u6807\u7b7e\u7684 lrp \u7684 ovn eip \u7684\u65f6\u5019\uff0c\u7f51\u5173\u8282\u70b9\u4e0d\u4f1a\u89e6\u53d1\u521b\u5efa\u7f51\u5173\u7f51\u5361\uff0c\u4e5f\u65e0\u6cd5\u6210\u529f\u542f\u52a8\u5bf9\u7aef bfd \u4f1a\u8bdd\u7684\u76d1\u542c\u3002","title":"1.3 \u81ea\u5b9a\u4e49 vpc \u542f\u7528 eip snat fip \u529f\u80fd"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#2-vpc-ecmp-bfd-l3-ha","text":"# cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true enableBfd: true # bfd \u5f00\u5173\u53ef\u4ee5\u968f\u610f\u5207\u6362\uff0c\u5f00\u8868\u793a\u542f\u7528 bfd ecmp \u8def\u7531 #enableBfd: false # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true # \u53ea\u9700\u5f00\u542f ecmp gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 \u4f7f\u7528\u4e0a\u7684\u6ce8\u610f\u70b9: \u81ea\u5b9a\u4e49 vpc \u4e0b\u7684 ecmp \u53ea\u7528\u9759\u6001 ecmp bfd \u8def\u7531\uff0cvpc enableBfd \u548c subnet enableEcmp \u540c\u65f6\u5f00\u542f\u7684\u60c5\u51b5\u4e0b\u624d\u4f1a\u751f\u6548\uff0c\u624d\u4f1a\u81ea\u52a8\u7ba1\u7406\u9759\u6001 ecmp bfd \u8def\u7531\u3002 \u4e0a\u8ff0\u914d\u7f6e\u5173\u95ed\u7684\u60c5\u51b5\u4e0b\uff0c\u4f1a\u81ea\u52a8\u5207\u56de\u5e38\u89c4\u9ed8\u8ba4\u9759\u6001\u8def\u7531\u3002 \u9ed8\u8ba4 vpc \u65e0\u6cd5\u4f7f\u7528\u8be5\u529f\u80fd\uff0c\u4ec5\u652f\u6301\u81ea\u5b9a\u4e49 vpc\uff0c\u9ed8\u8ba4 vpc \u6709\u66f4\u590d\u6742\u7684\u7b56\u7565\u8def\u7531\u4ee5\u53ca snat \u8bbe\u8ba1\u3002 \u81ea\u5b9a\u4e49 vpc \u7684 subnet \u7684 enableEcmp \u4ec5\u4f7f\u7528\u9759\u6001\u8def\u7531\uff0c\u7f51\u5173\u7c7b\u578b gatewayType \u6ca1\u6709\u4f5c\u7528\u3002 \u5f53\u5173\u95ed EnableExternal \u65f6\uff0cvpc \u5185\u65e0\u6cd5\u901a\u5916\u7f51\u3002 \u5f53\u5f00\u542f EnableExternal \u65f6\uff0c\u5173\u95ed EnableBfd \u65f6\uff0c\u4f1a\u57fa\u4e8e\u666e\u901a\u9ed8\u8ba4\u8def\u7531\u4e0a\u5916\u7f51\uff0c\u4e0d\u5177\u5907\u9ad8\u53ef\u7528\u3002 # \u4e0a\u8ff0\u6a21\u677f\u5e94\u7528\u540e ovn \u903b\u8f91\u5c42\u5e94\u8be5\u53ef\u4ee5\u770b\u5230\u5982\u4e0b\u8d44\u6e90 # \u67e5\u770b vpc # k get vpc NAME ENABLEEXTERNAL ENABLEBFD STANDBY SUBNETS NAMESPACES ovn-cluster true true [ \"external204\" , \"join\" , \"ovn-default\" ] vpc1 true true true [ \"vpc1-subnet1\" ] [ \"vpc1\" ] # \u9ed8\u8ba4 vpc \u672a\u652f\u6301 ENABLEBFD # \u81ea\u5b9a\u4e49 vpc \u5df2\u652f\u6301\u4e14\u5df2\u542f\u7528 # 1. \u521b\u5efa\u4e86 bfd \u4f1a\u8bdd # k ko nbctl list bfd _uuid : be7df545-2c4c-4751-878f-b3507987f050 detect_mult : 3 dst_ip : \"10.5.204.121\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : 684c4489-5b59-4693-8d8c-3beab93f8093 detect_mult : 3 dst_ip : \"10.5.204.109\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : f0f62077-2ae9-4e79-b4f8-a446ec6e784c detect_mult : 3 dst_ip : \"10.5.204.108\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up ### \u6ce8\u610f\u6240\u6709 status \u6b63\u5e38\u90fd\u5e94\u8be5\u662f up \u7684 # 2. \u521b\u5efa\u4e86\u57fa\u4e8e bfd \u7684\u9759\u6001\u8def\u7531 # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 192 .168.0.0/24 10 .5.204.108 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.109 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.121 src-ip ecmp ecmp-symmetric-reply bfd # 3. \u9759\u6001\u8def\u7531\u8be6\u60c5 # k ko nbctl find Logical_Router_Static_Route policy=src-ip options=ecmp_symmetric_reply=\"true\" _uuid : 3aacb384-d5ee-4b14-aebf-59e8c11717ba bfd : 684c4489-5b59-4693-8d8c-3beab93f8093 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.109\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 18bcc585-bc05-430b-925b-ef673c8e1aef bfd : f0f62077-2ae9-4e79-b4f8-a446ec6e784c external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.108\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 7d0a4e6b-cde0-4110-8176-fbaf19738498 bfd : be7df545-2c4c-4751-878f-b3507987f050 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.121\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" # \u540c\u65f6\u5728\u7f51\u5173\u8282\u70b9\u90fd\u5e94\u8be5\u5177\u5907\u4ee5\u4e0b\u8d44\u6e90 [ root@pc-node-1 ~ ] # ip netns exec ovnext bash ip a /usr/sbin/ip: /usr/sbin/ip: cannot execute binary file [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1541 : ovnext0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 00 :00:00:ab:bd:87 brd ff:ff:ff:ff:ff:ff inet 10 .5.204.108/24 brd 10 .5.204.255 scope global ovnext0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:feab:bd87/64 scope link valid_lft forever preferred_lft forever [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0 .0.0.0 10 .5.204.254 0 .0.0.0 UG 0 0 0 ovnext0 10 .5.204.0 0 .0.0.0 255 .255.255.0 U 0 0 0 ovnext0 ## \u6ce8\u610f\u4ee5\u4e0a\u5185\u5bb9\u548c\u4e00\u4e2a internal port unerlay \u516c\u7f51 pod \u5185\u90e8\u7684 ns \u5927\u81f4\u662f\u4e00\u81f4\u7684\uff0c\u8fd9\u91cc\u53ea\u662f\u5728\u7f51\u5173\u8282\u70b9\u4e0a\u5355\u72ec\u7ef4\u62a4\u4e86\u4e00\u4e2a ns [ root@pc-node-1 ~ ] # ip netns exec ovnext bfdd-control status There are 1 sessions: Session 1 id = 1 local = 10 .5.204.108 ( p ) remote = 10 .5.204.122 state = Up ## \u8fd9\u91cc\u5373\u662f lrp bfd \u4f1a\u8bdd\u7684\u53e6\u4e00\u7aef\uff0c\u4e5f\u662f lrp ecmp \u7684\u4e0b\u4e00\u8df3\u7684\u5176\u4e2d\u4e00\u4e2a [ root@pc-node-1 ~ ] # ip netns exec ovnext ping -c1 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 115 time = 21 .6 ms # \u5230\u516c\u7f51\u6ca1\u95ee\u9898 \u53ef\u4ee5\u5728\u67d0\u4e00\u4e2a\u7f51\u5173\u8282\u70b9\u7684 ovnext ns \u5185\u6293\u5230\u51fa\u53bb\u7684\u5305 # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-1 ~ ] # exit [ root@pc-node-1 ~ ] # ssh pc-node-2 Last login: Thu Feb 23 09 :21:08 2023 from 10 .5.32.51 [ root@pc-node-2 ~ ] # ip netns exec ovnext bash [ root@pc-node-2 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-2 ~ ] # exit [ root@pc-node-2 ~ ] # logout Connection to pc-node-2 closed. [ root@pc-node-1 ~ ] # ssh pc-node-3 Last login: Thu Feb 23 08 :32:41 2023 from 10 .5.32.51 [ root@pc-node-3 ~ ] # ip netns exec ovnext bash [ root@pc-node-3 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:2d:f8:ce > 00 :00:00:fd:b2:a4, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 63 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 00 :00:00:fd:b2:a4 > dc:ef:80:5a:44:1a, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [ root@pc-node-3 ~ ] # # \u53ef\u4ee5\u5728\u8be5\u8282\u70b9 down \u6389\u51fa\u53bb\u7684\u7f51\u5361\uff0c\u7136\u540e\u770b pod \u51fa\u53bb\u7684\u5305\u5728\u7f51\u7edc\u4e2d\u65ad\u4e2d\u4f1a\u51fa\u73b0\u51e0\u4e2a\u5305 # \u4e00\u822c\u90fd\u4f1a\u770b\u5230\u4e22 3 \u4e2a\u5305","title":"2. \u81ea\u5b9a\u4e49 vpc \u542f\u7528 ecmp bfd L3 HA \u516c\u7f51\u529f\u80fd"},{"location":"advance/ovn-l3-ha-based-ecmp-with-bfd/#3-bfd","text":"\u5728\u67d0\u4e9b\u573a\u666f\u4e0b\uff0c\u53ef\u80fd\u60f3\u76f4\u63a5\u4f7f\u7528\uff08\u96c6\u4e2d\u5f0f\uff09\u5355\u4e2a\u7f51\u5173\u76f4\u63a5\u51fa\u516c\u7f51\uff0c\u8fd9\u4e2a\u65f6\u5019\u548c\u9ed8\u8ba4 vpc enable_eip_snat \u7684\u4f7f\u7528\u6a21\u5f0f\u662f\u4e00\u81f4\u7684 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc2 spec: namespaces: - vpc2 enableExternal: true #enableBfd: true enableBfd: false ## \u5c06 bfd \u529f\u80fd\u76f4\u63a5\u7981\u7528\u5373\u53ef # k ko nbctl lr-route-list vpc2 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # \u5e94\u7528\u540e\u8def\u7531\u4f1a\u5207\u6362\u56de\u6b63\u5e38\u7684\u9ed8\u8ba4\u9759\u6001\u8def\u7531 # \u540c\u65f6 nbctl list bfd \u53ef\u4ee5\u770b\u5230 lrp \u5173\u8054\u7684 bfd \u4f1a\u8bdd\u5df2\u7ecf\u79fb\u9664 # \u800c\u4e14 ovnext ns \u4e2d\u7684\u5bf9\u7aef bfd \u4f1a\u8bdd\u4e5f\u81ea\u52a8\u79fb\u9664 # \u8be5\u5207\u6362\u8fc7\u7a0b\u4fdd\u6301 vpc subnet \u5185\u4fdd\u6301 ping \u672a\u770b\u5230(\u79d2\u7ea7)\u4e22\u5305 # \u518d\u5207\u6362\u56de\u53bb \u4e5f\u672a\u770b\u5230(\u79d2\u7ea7)\u4e22\u5305 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"3. \u5173\u95ed bfd \u6a21\u5f0f"},{"location":"advance/ovn-remote-port-mirroring/","text":"OVN \u6d41\u91cf\u955c\u50cf \u00b6 \u6b64\u529f\u80fd\u53ef\u4ee5\u5c06\u6307\u5b9a Pod\u3001\u6307\u5b9a\u65b9\u5411\u7684\u6d41\u91cf\uff0c\u901a\u8fc7 GRE/ERSPAN \u5c01\u88c5\u540e\uff0c\u4f20\u8f93\u5230\u8fdc\u7aef\u3002 \u6b64\u529f\u80fd\u8981\u6c42 Kube-OVN \u7248\u672c\u4e0d\u4f4e\u4e8e v1.12\u3002 \u90e8\u7f72 Multus-CNI \u00b6 \u53c2\u8003 Multus-CNI \u6587\u6863 \u90e8\u7f72 Multus\u3002 \u521b\u5efa\u9644\u5c5e\u7f51\u7edc \u00b6 \u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa\u9644\u5c5e\u7f51\u7edc\uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : | { \"cniVersion\": \"0.3.1\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" } \u5176\u4e2d provider \u5b57\u6bb5\u683c\u5f0f\u4e3a <NAME>.<NAMESPACE>.ovn \u3002 \u521b\u5efa Underlay \u7f51\u7edc \u00b6 \u955c\u50cf\u6d41\u91cf\u662f\u5c01\u88c5\u540e\u8fdb\u884c\u4f20\u8f93\u7684\uff0c\u56e0\u6b64\u7528\u4e8e\u4f20\u8f93\u7684\u7f51\u7edc\uff0cMTU \u9700\u8981\u5927\u4e8e\u88ab\u955c\u50cf\u7684 LSP/Pod\u3002\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 Underlay \u4f5c\u4e3a\u4f20\u8f93\u7f51\u7edc\u3002 \u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa Underlay \u7f51\u7edc\uff1a apiVersion : kubeovn.io/v1 kind : ProviderNetwork metadata : name : net1 spec : defaultInterface : eth1 --- apiVersion : kubeovn.io/v1 kind : Vlan metadata : name : vlan1 spec : id : 0 provider : net1 --- apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.19.0.0/16 excludeIps : - 172.19.0.2..172.19.0.20 gateway : 172.19.0.1 vlan : vlan1 provider : attachnet.default.ovn \u5176\u4e2d\uff0c\u5b50\u7f51\u7684 provider \u5fc5\u987b\u4e0e\u9644\u5c5e\u7f51\u7edc\u7684 provider \u76f8\u540c\u3002 \u521b\u5efa\u6d41\u91cf\u63a5\u6536 Pod \u00b6 \u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa\u7528\u4e8e\u63a5\u6536\u955c\u50cf\u6d41\u91cf\u7684 Pod\uff1a apiVersion : v1 kind : Pod metadata : name : pod1 annotations : k8s.v1.cni.cncf.io/networks : default/attachnet spec : containers : - name : bash image : docker.io/kubeovn/kube-ovn:v1.12.0 args : - bash - -c - sleep infinity securityContext : privileged : true \u521b\u5efa\u5b8c\u6210\u540e\uff0c\u67e5\u770b Pod \u7684 IP \u5730\u5740\uff1a $ kubectl get ips | grep pod1 pod1.default 10 .16.0.12 00 :00:00:FF:34:24 kube-ovn-worker ovn-default pod1.default.attachnet.default.ovn 172 .19.0.21 00 :00:00:A0:30:68 kube-ovn-worker subnet1 \u8bb0\u4f4f\u7b2c\u4e8c\u7f51\u5361\u7684 IP \u5730\u5740 172.19.0.21 \u3002 \u521b\u5efa OVN \u6d41\u91cf\u955c\u50cf \u00b6 \u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u521b\u5efa OVN \u6d41\u91cf\u955c\u50cf\uff1a kubectl ko nbctl mirror-add mirror1 gre 99 from-lport 172 .19.0.21 kubectl ko nbctl lsp-attach-mirror coredns-787d4945fb-gpnkb.kube-system mirror1 \u5176\u4e2d coredns-787d4945fb-gpnkb.kube-system \u662f OVN LSP \u7684\u540d\u79f0\uff0c\u683c\u5f0f\u901a\u5e38\u4e3a <POD_NAME>.<POD_NAMESPACE> \u3002 \u76f8\u5173\u7684 OVN \u547d\u4ee4\u4f7f\u7528\u65b9\u6cd5\u5982\u4e0b\uff1a ovn-nbctl mirror-add <NAME> <TYPE> <INDEX> <FILTER> <IP> NAME - add a mirror with given name TYPE - specify TYPE 'gre' or 'erspan' INDEX - specify the tunnel INDEX value (indicates key if GRE, erpsan_idx if ERSPAN) FILTER - specify FILTER for mirroring selection ('to-lport' / 'from-lport') IP - specify Sink / Destination i.e. Remote IP ovn-nbctl mirror-del [NAME] remove mirrors ovn-nbctl mirror-list print mirrors ovn-nbctl lsp-attach-mirror PORT MIRROR attach source PORT to MIRROR ovn-nbctl lsp-detach-mirror PORT MIRROR detach source PORT from MIRROR \u914d\u7f6e\u6d41\u91cf\u63a5\u6536 Pod \u00b6 \u5728\u524d\u9762\u521b\u5efa\u7684 Pod \u4e2d\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff1a root@pod1:/kube-ovn# ip link add mirror1 type gretap local 172 .19.0.21 key 99 dev net1 root@pod1:/kube-ovn# ip link set mirror1 up \u63a5\u4e0b\u6765\u5c31\u53ef\u4ee5\u5728\u63a5\u6536\u6d41\u91cf\u7684 Pod \u4e2d\u8fdb\u884c\u6293\u5305\u9a8c\u8bc1\uff1a root@pod1:/kube-ovn# tcpdump -i mirror1 -nnve tcpdump: listening on mirror1, link-type EN10MB ( Ethernet ) , snapshot length 262144 bytes 05 :13:30.328808 00 :00:00:a3:f5:e2 > 00 :00:00:97:0f:6e, ethertype ARP ( 0x0806 ) , length 42 : Ethernet ( len 6 ) , IPv4 ( len 4 ) , Request who-has 10 .16.0.7 tell 10 .16.0.4, length 28 05 :13:30.559167 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57364 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.50472: 34511 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.559343 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57365 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.45177: 1659 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.560625 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 200 : ( tos 0x0, ttl 64 , id 57367 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 186 ) 10 .16.0.4.53 > 10 .16.0.6.43848: 2636 *- 0 /1/1 ( 158 ) 05 :13:30.562774 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 191 : ( tos 0x0, ttl 64 , id 57368 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 177 ) 10 .16.0.4.53 > 10 .16.0.6.37755: 48737 NXDomain*- 0 /1/1 ( 149 ) 05 :13:30.563523 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 187 : ( tos 0x0, ttl 64 , id 57369 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 173 ) 10 .16.0.4.53 > 10 .16.0.6.53887: 45519 NXDomain*- 0 /1/1 ( 145 ) 05 :13:30.564940 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57370 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.40846: 25745 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.565140 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57371 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.45214: 61875 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.566023 00 :00:00:a3:f5:e2 > 00 :00:00:55:e4:4e, ethertype IPv4 ( 0x0800 ) , length 80 : ( tos 0x0, ttl 64 , id 45937 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 66 ) 10 .16.0.4.44116 > 172 .18.0.1.53: 16025 + [ 1au ] AAAA? alauda.cn. ( 38 ) \u6ce8\u610f\u4e8b\u9879 \u00b6 \u5982\u679c\u4f7f\u7528 ERSPAN \u4f5c\u4e3a\u5c01\u88c5\u534f\u8bae\uff0cOVN \u8282\u70b9\u53ca\u8fdc\u7aef\u8bbe\u5907\u7684 Linux \u5185\u6838\u7248\u672c\u4e0d\u5f97\u4f4e\u4e8e 4.14\u3002\u82e5\u4f7f\u7528 ERSPAN \u4f5c\u4e3a\u5c01\u88c5\u534f\u8bae\u4e14\u4f7f\u7528 IPv6 \u4f5c\u4e3a\u4f20\u8f93\u7f51\u7edc\uff0cLinux \u5185\u6838\u7248\u672c\u4e0d\u5f97\u4f4e\u4e8e 4.16\u3002 \u955c\u50cf\u6d41\u91cf\u7684\u4f20\u8f93\u662f\u5355\u5411\u7684\uff0c\u53ea\u9700\u4fdd\u8bc1 OVN \u8282\u70b9\u80fd\u591f\u8bbf\u95ee\u8fdc\u7aef\u8bbe\u5907\u5373\u53ef\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN \u6d41\u91cf\u955c\u50cf"},{"location":"advance/ovn-remote-port-mirroring/#ovn","text":"\u6b64\u529f\u80fd\u53ef\u4ee5\u5c06\u6307\u5b9a Pod\u3001\u6307\u5b9a\u65b9\u5411\u7684\u6d41\u91cf\uff0c\u901a\u8fc7 GRE/ERSPAN \u5c01\u88c5\u540e\uff0c\u4f20\u8f93\u5230\u8fdc\u7aef\u3002 \u6b64\u529f\u80fd\u8981\u6c42 Kube-OVN \u7248\u672c\u4e0d\u4f4e\u4e8e v1.12\u3002","title":"OVN \u6d41\u91cf\u955c\u50cf"},{"location":"advance/ovn-remote-port-mirroring/#multus-cni","text":"\u53c2\u8003 Multus-CNI \u6587\u6863 \u90e8\u7f72 Multus\u3002","title":"\u90e8\u7f72 Multus-CNI"},{"location":"advance/ovn-remote-port-mirroring/#_1","text":"\u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa\u9644\u5c5e\u7f51\u7edc\uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : | { \"cniVersion\": \"0.3.1\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" } \u5176\u4e2d provider \u5b57\u6bb5\u683c\u5f0f\u4e3a <NAME>.<NAMESPACE>.ovn \u3002","title":"\u521b\u5efa\u9644\u5c5e\u7f51\u7edc"},{"location":"advance/ovn-remote-port-mirroring/#underlay","text":"\u955c\u50cf\u6d41\u91cf\u662f\u5c01\u88c5\u540e\u8fdb\u884c\u4f20\u8f93\u7684\uff0c\u56e0\u6b64\u7528\u4e8e\u4f20\u8f93\u7684\u7f51\u7edc\uff0cMTU \u9700\u8981\u5927\u4e8e\u88ab\u955c\u50cf\u7684 LSP/Pod\u3002\u8fd9\u91cc\u6211\u4eec\u4f7f\u7528 Underlay \u4f5c\u4e3a\u4f20\u8f93\u7f51\u7edc\u3002 \u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa Underlay \u7f51\u7edc\uff1a apiVersion : kubeovn.io/v1 kind : ProviderNetwork metadata : name : net1 spec : defaultInterface : eth1 --- apiVersion : kubeovn.io/v1 kind : Vlan metadata : name : vlan1 spec : id : 0 provider : net1 --- apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.19.0.0/16 excludeIps : - 172.19.0.2..172.19.0.20 gateway : 172.19.0.1 vlan : vlan1 provider : attachnet.default.ovn \u5176\u4e2d\uff0c\u5b50\u7f51\u7684 provider \u5fc5\u987b\u4e0e\u9644\u5c5e\u7f51\u7edc\u7684 provider \u76f8\u540c\u3002","title":"\u521b\u5efa Underlay \u7f51\u7edc"},{"location":"advance/ovn-remote-port-mirroring/#pod","text":"\u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\u521b\u5efa\u7528\u4e8e\u63a5\u6536\u955c\u50cf\u6d41\u91cf\u7684 Pod\uff1a apiVersion : v1 kind : Pod metadata : name : pod1 annotations : k8s.v1.cni.cncf.io/networks : default/attachnet spec : containers : - name : bash image : docker.io/kubeovn/kube-ovn:v1.12.0 args : - bash - -c - sleep infinity securityContext : privileged : true \u521b\u5efa\u5b8c\u6210\u540e\uff0c\u67e5\u770b Pod \u7684 IP \u5730\u5740\uff1a $ kubectl get ips | grep pod1 pod1.default 10 .16.0.12 00 :00:00:FF:34:24 kube-ovn-worker ovn-default pod1.default.attachnet.default.ovn 172 .19.0.21 00 :00:00:A0:30:68 kube-ovn-worker subnet1 \u8bb0\u4f4f\u7b2c\u4e8c\u7f51\u5361\u7684 IP \u5730\u5740 172.19.0.21 \u3002","title":"\u521b\u5efa\u6d41\u91cf\u63a5\u6536 Pod"},{"location":"advance/ovn-remote-port-mirroring/#ovn_1","text":"\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u521b\u5efa OVN \u6d41\u91cf\u955c\u50cf\uff1a kubectl ko nbctl mirror-add mirror1 gre 99 from-lport 172 .19.0.21 kubectl ko nbctl lsp-attach-mirror coredns-787d4945fb-gpnkb.kube-system mirror1 \u5176\u4e2d coredns-787d4945fb-gpnkb.kube-system \u662f OVN LSP \u7684\u540d\u79f0\uff0c\u683c\u5f0f\u901a\u5e38\u4e3a <POD_NAME>.<POD_NAMESPACE> \u3002 \u76f8\u5173\u7684 OVN \u547d\u4ee4\u4f7f\u7528\u65b9\u6cd5\u5982\u4e0b\uff1a ovn-nbctl mirror-add <NAME> <TYPE> <INDEX> <FILTER> <IP> NAME - add a mirror with given name TYPE - specify TYPE 'gre' or 'erspan' INDEX - specify the tunnel INDEX value (indicates key if GRE, erpsan_idx if ERSPAN) FILTER - specify FILTER for mirroring selection ('to-lport' / 'from-lport') IP - specify Sink / Destination i.e. Remote IP ovn-nbctl mirror-del [NAME] remove mirrors ovn-nbctl mirror-list print mirrors ovn-nbctl lsp-attach-mirror PORT MIRROR attach source PORT to MIRROR ovn-nbctl lsp-detach-mirror PORT MIRROR detach source PORT from MIRROR","title":"\u521b\u5efa OVN \u6d41\u91cf\u955c\u50cf"},{"location":"advance/ovn-remote-port-mirroring/#pod_1","text":"\u5728\u524d\u9762\u521b\u5efa\u7684 Pod \u4e2d\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff1a root@pod1:/kube-ovn# ip link add mirror1 type gretap local 172 .19.0.21 key 99 dev net1 root@pod1:/kube-ovn# ip link set mirror1 up \u63a5\u4e0b\u6765\u5c31\u53ef\u4ee5\u5728\u63a5\u6536\u6d41\u91cf\u7684 Pod \u4e2d\u8fdb\u884c\u6293\u5305\u9a8c\u8bc1\uff1a root@pod1:/kube-ovn# tcpdump -i mirror1 -nnve tcpdump: listening on mirror1, link-type EN10MB ( Ethernet ) , snapshot length 262144 bytes 05 :13:30.328808 00 :00:00:a3:f5:e2 > 00 :00:00:97:0f:6e, ethertype ARP ( 0x0806 ) , length 42 : Ethernet ( len 6 ) , IPv4 ( len 4 ) , Request who-has 10 .16.0.7 tell 10 .16.0.4, length 28 05 :13:30.559167 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57364 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.50472: 34511 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.559343 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57365 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.45177: 1659 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.560625 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 200 : ( tos 0x0, ttl 64 , id 57367 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 186 ) 10 .16.0.4.53 > 10 .16.0.6.43848: 2636 *- 0 /1/1 ( 158 ) 05 :13:30.562774 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 191 : ( tos 0x0, ttl 64 , id 57368 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 177 ) 10 .16.0.4.53 > 10 .16.0.6.37755: 48737 NXDomain*- 0 /1/1 ( 149 ) 05 :13:30.563523 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 187 : ( tos 0x0, ttl 64 , id 57369 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 173 ) 10 .16.0.4.53 > 10 .16.0.6.53887: 45519 NXDomain*- 0 /1/1 ( 145 ) 05 :13:30.564940 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57370 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.40846: 25745 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.565140 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57371 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.45214: 61875 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.566023 00 :00:00:a3:f5:e2 > 00 :00:00:55:e4:4e, ethertype IPv4 ( 0x0800 ) , length 80 : ( tos 0x0, ttl 64 , id 45937 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 66 ) 10 .16.0.4.44116 > 172 .18.0.1.53: 16025 + [ 1au ] AAAA? alauda.cn. ( 38 )","title":"\u914d\u7f6e\u6d41\u91cf\u63a5\u6536 Pod"},{"location":"advance/ovn-remote-port-mirroring/#_2","text":"\u5982\u679c\u4f7f\u7528 ERSPAN \u4f5c\u4e3a\u5c01\u88c5\u534f\u8bae\uff0cOVN \u8282\u70b9\u53ca\u8fdc\u7aef\u8bbe\u5907\u7684 Linux \u5185\u6838\u7248\u672c\u4e0d\u5f97\u4f4e\u4e8e 4.14\u3002\u82e5\u4f7f\u7528 ERSPAN \u4f5c\u4e3a\u5c01\u88c5\u534f\u8bae\u4e14\u4f7f\u7528 IPv6 \u4f5c\u4e3a\u4f20\u8f93\u7f51\u7edc\uff0cLinux \u5185\u6838\u7248\u672c\u4e0d\u5f97\u4f4e\u4e8e 4.16\u3002 \u955c\u50cf\u6d41\u91cf\u7684\u4f20\u8f93\u662f\u5355\u5411\u7684\uff0c\u53ea\u9700\u4fdd\u8bc1 OVN \u8282\u70b9\u80fd\u591f\u8bbf\u95ee\u8fdc\u7aef\u8bbe\u5907\u5373\u53ef\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"advance/performance-tuning/","text":"\u6027\u80fd\u8c03\u4f18 \u00b6 \u4e3a\u4e86\u4fdd\u6301\u5b89\u88c5\u7684\u7b80\u5355\u548c\u529f\u80fd\u7684\u5b8c\u5907\uff0cKube-OVN \u7684\u9ed8\u8ba4\u5b89\u88c5\u811a\u672c\u5e76\u6ca1\u6709\u5bf9\u6027\u80fd\u9488\u5bf9\u6027\u7684\u4f18\u5316\u3002\u5982\u679c\u5e94\u7528\u5bf9\u5ef6\u8fdf\u548c\u541e\u5410\u91cf\u654f\u611f\uff0c \u7ba1\u7406\u5458\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u5bf9\u6027\u80fd\u8fdb\u884c\u9488\u5bf9\u6027\u4f18\u5316\u3002 \u793e\u533a\u4f1a\u4e0d\u65ad\u8fed\u4ee3\u63a7\u5236\u9762\u677f\u548c\u4f18\u5316\u9762\u7684\u6027\u80fd\uff0c\u90e8\u5206\u901a\u7528\u6027\u80fd\u4f18\u5316\u5df2\u7ecf\u96c6\u6210\u5230\u6700\u65b0\u7248\u672c\uff0c\u5efa\u8bae\u4f7f\u7528\u6700\u65b0\u7248\u672c\u83b7\u5f97\u66f4\u597d\u7684\u9ed8\u8ba4\u6027\u80fd\u3002 \u66f4\u591a\u5173\u4e8e\u6027\u80fd\u4f18\u5316\u7684\u8fc7\u7a0b\u548c\u65b9\u6cd5\u8bba\uff0c\u53ef\u4ee5\u89c2\u770b\u89c6\u9891\u5206\u4eab\uff1a Kube-OVN \u5bb9\u5668\u6027\u80fd\u4f18\u5316\u4e4b\u65c5 \u3002 \u57fa\u51c6\u6d4b\u8bd5 \u00b6 \u7531\u4e8e\u8f6f\u786c\u4ef6\u73af\u5883\u7684\u5dee\u5f02\u6781\u5927\uff0c\u8fd9\u91cc\u63d0\u4f9b\u7684\u6027\u80fd\u6d4b\u8bd5\u6570\u636e\u53ea\u80fd\u4f5c\u4e3a\u53c2\u8003\uff0c\u5b9e\u9645\u6d4b\u8bd5\u7ed3\u679c\u4f1a\u548c\u672c\u6587\u6863\u4e2d\u7684\u7ed3\u679c\u5b58\u5728\u8f83\u5927\u5dee\u5f02\u3002 \u5efa\u8bae\u6bd4\u8f83\u4f18\u5316\u524d\u540e\u7684\u6027\u80fd\u6d4b\u8bd5\u7ed3\u679c\uff0c\u548c\u5bbf\u4e3b\u673a\u7f51\u7edc\u548c\u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u6bd4\u8f83\u3002 Overlay \u4f18\u5316\u524d\u540e\u6027\u80fd\u5bf9\u6bd4 \u00b6 \u73af\u5883\u4fe1\u606f\uff1a Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 Overlay \u6a21\u5f0f CPU: Intel(R) Xeon(R) E-2278G Network: 2*10Gbps, xmit_hash_policy=layer3+4 \u6211\u4eec\u4f7f\u7528 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw \u6d4b\u8bd5 1 \u5b57\u8282\u5c0f\u5305\u4e0b tcp/udp \u7684\u5e26\u5bbd\u548c\u5ef6\u8fdf\uff0c\u5206\u522b\u6d4b\u8bd5\u4f18\u5316\u524d\uff0c\u4f18\u5316\u540e\u4ee5\u53ca\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u6027\u80fd\uff1a Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Default 25.7 22.9 27.1 1.59 Kube-OVN Optimized 13.9 12.9 27.6 5.57 HOST Network 13.1 12.4 28.2 6.02 Overlay\uff0c Underlay \u4ee5\u53ca Calico \u4e0d\u540c\u6a21\u5f0f\u6027\u80fd\u5bf9\u6bd4 \u00b6 \u4e0b\u9762\u6211\u4eec\u4f1a\u6bd4\u8f83\u4f18\u5316\u540e Kube-OVN \u5728\u4e0d\u540c\u5305\u5927\u5c0f\u4e0b\u7684 Overlay \u548c Underlay \u6027\u80fd\uff0c\u5e76\u548c Calico \u7684 IPIP Always , IPIP never \u4ee5\u53ca\u5bbf\u4e3b\u673a\u7f51\u7edc\u505a\u6bd4\u8f83\u3002 Environment : Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 CPU: AMD EPYC 7402P 24-Core Processor Network: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Overlay 15.2 14.6 23.6 2.65 Kube-OVN Underlay 14.3 13.8 24.2 3.46 Calico IPIP 21.4 20.2 23.6 1.18 Calico NoEncap 19.3 16.9 23.6 1.76 HOST Network 16.6 15.4 24.8 2.64 qperf -t 60 <server ip> -ub -oo msg_size:1K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 16.5 15.8 10.2 2.77 Kube-OVN Underlay 15.9 14.5 9.6 3.22 Calico IPIP 22.5 21.5 1.45 1.14 Calico NoEncap 19.4 18.3 3.76 1.63 HOST Network 18.1 16.6 9.32 2.66 qperf -t 60 <server ip> -ub -oo msg_size:4K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 34.7 41.6 16.0 9.23 Kube-OVN Underlay 32.6 44 15.1 6.71 Calico IPIP 44.8 52.9 2.94 3.26 Calico NoEncap 40 49.6 6.56 4.19 HOST Network 35.9 45.9 14.6 5.59 \u5728\u90e8\u5206\u60c5\u51b5\u4e0b\u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u4f1a\u4f18\u4e8e\u5bbf\u4e3b\u673a\u7f51\u7edc\uff0c\u8fd9\u662f\u4f18\u4e8e\u7ecf\u8fc7\u4f18\u5316\u540e\u5bb9\u5668\u7f51\u7edc\u8def\u5f84\u5b8c\u5168\u7ed5\u8fc7\u4e86 netfilter\uff0c \u800c\u5bbf\u4e3b\u673a\u7f51\u7edc\u7531\u4e8e kube-proxy \u7684\u5b58\u5728\u6240\u6709\u6570\u636e\u5305\u5747\u9700\u7ecf\u8fc7 netfilter\uff0c\u4f1a\u5bfc\u81f4\u5728\u4e00\u4e9b\u73af\u5883\u4e0b\u5bb9\u5668\u7f51\u7edc \u7684\u6d88\u8017\u76f8\u5bf9\u66f4\u5c0f\uff0c\u56e0\u6b64\u4f1a\u6709\u66f4\u597d\u7684\u6027\u80fd\u8868\u73b0\u3002 \u6570\u636e\u5e73\u9762\u6027\u80fd\u4f18\u5316\u65b9\u6cd5 \u00b6 \u8fd9\u91cc\u4ecb\u7ecd\u7684\u4f18\u5316\u65b9\u6cd5\u548c\u8f6f\u786c\u4ef6\u73af\u5883\u4ee5\u53ca\u6240\u9700\u8981\u7684\u529f\u80fd\u76f8\u5173\uff0c\u8bf7\u4ed4\u7ec6\u4e86\u89e3\u4f18\u5316\u7684\u524d\u63d0\u6761\u4ef6\u518d\u8fdb\u884c\u5c1d\u8bd5\u3002 CPU \u6027\u80fd\u6a21\u5f0f\u8c03\u6574 \u00b6 \u90e8\u5206\u73af\u5883\u4e0b CPU \u8fd0\u884c\u5728\u8282\u80fd\u6a21\u5f0f\uff0c\u8be5\u6a21\u5f0f\u4e0b\u6027\u80fd\u8868\u73b0\u5c06\u4f1a\u4e0d\u7a33\u5b9a\uff0c\u5ef6\u8fdf\u4f1a\u51fa\u73b0\u660e\u663e\u589e\u52a0\uff0c\u5efa\u8bae\u4f7f\u7528 CPU \u7684\u6027\u80fd\u6a21\u5f0f\u83b7\u5f97\u66f4\u7a33\u5b9a\u7684\u6027\u80fd\u8868\u73b0\uff1a cpupower frequency-set -g performance \u7f51\u5361\u786c\u4ef6\u961f\u5217\u8c03\u6574 \u00b6 \u5728\u6d41\u91cf\u589e\u5927\u7684\u60c5\u51b5\u4e0b\uff0c\u7f13\u51b2\u961f\u5217\u8fc7\u77ed\u53ef\u80fd\u5bfc\u81f4\u8f83\u9ad8\u7684\u4e22\u5305\u7387\u5bfc\u81f4\u6027\u80fd\u663e\u8457\u4e0b\u964d\uff0c\u9700\u8981\u8fdb\u884c\u8c03\u6574 \u68c0\u67e5\u5f53\u524d\u7f51\u5361\u961f\u5217\u957f\u5ea6\uff1a # ethtool -g eno1 Ring parameters for eno1: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 255 RX Mini: 0 RX Jumbo: 0 TX: 255 \u589e\u52a0\u961f\u5217\u957f\u5ea6\u81f3\u6700\u5927\u503c\uff1a ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096 \u4f7f\u7528 tuned \u4f18\u5316\u7cfb\u7edf\u53c2\u6570 \u00b6 tuned \u53ef\u4ee5\u4f7f\u7528\u4e00\u7cfb\u5217\u9884\u7f6e\u7684 profile \u6587\u4ef6\u4fdd\u5b58\u4e86\u9488\u5bf9\u7279\u5b9a\u573a\u666f\u7684\u4e00\u7cfb\u5217\u7cfb\u7edf\u4f18\u5316\u914d\u7f6e\u3002 \u9488\u5bf9\u5ef6\u8fdf\u4f18\u5148\u573a\u666f\uff1a tuned-adm profile network-latency \u9488\u5bf9\u541e\u5410\u91cf\u4f18\u5148\u573a\u666f\uff1a tuned-adm profile network-throughput \u4e2d\u65ad\u7ed1\u5b9a \u00b6 \u6211\u4eec\u63a8\u8350\u7981\u7528 irqbalance \u5e76\u5c06\u7f51\u5361\u4e2d\u65ad\u548c\u7279\u5b9a CPU \u8fdb\u884c\u7ed1\u5b9a\uff0c\u6765\u907f\u514d\u5728\u591a\u4e2a CPU \u4e4b\u95f4\u5207\u6362\u5bfc\u81f4\u7684\u6027\u80fd\u6ce2\u52a8\u3002 \u5173\u95ed OVN LB \u00b6 OVN \u7684 L2 LB \u5b9e\u73b0\u8fc7\u7a0b\u4e2d\u9700\u8981\u8c03\u7528\u5185\u6838\u7684 conntrack \u6a21\u5757\u5e76\u8fdb\u884c recirculate \u5bfc\u81f4\u5927\u91cf\u7684 CPU \u5f00\u9500\uff0c\u7ecf\u6d4b\u8bd5\u8be5\u529f\u80fd\u4f1a\u5e26\u6765 20% \u5de6\u53f3\u7684 CPU \u5f00\u9500\uff0c \u5728 Overlay \u7f51\u7edc\u6a21\u5f0f\u4e0b\u53ef\u4ee5\u4f7f\u7528 kube-proxy \u5b8c\u6210 Service \u8f6c\u53d1\u529f\u80fd\uff0c\u83b7\u5f97\u66f4\u597d\u7684 Pod-to-Pod \u6027\u80fd\u3002\u53ef\u4ee5\u5728 kube-ovn-controller \u4e2d\u5173\u95ed\u8be5\u529f\u80fd\uff1a command : - /kube-ovn/start-controller.sh args : ... - --enable-lb=false ... Underlay \u6a21\u5f0f\u4e0b kube-proxy \u65e0\u6cd5\u4f7f\u7528 iptables \u6216 ipvs \u63a7\u5236\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\uff0c\u5982\u9700\u5173\u95ed LB \u529f\u80fd\u9700\u8981\u786e\u8ba4\u662f\u5426\u4e0d\u9700\u8981 Service \u529f\u80fd\u3002 \u5185\u6838 FastPath \u6a21\u5757 \u00b6 \u7531\u4e8e\u5bb9\u5668\u7f51\u7edc\u548c\u5bbf\u4e3b\u673a\u7f51\u7edc\u5728\u4e0d\u540c\u7684 network ns\uff0c\u6570\u636e\u5305\u5728\u8de8\u5bbf\u4e3b\u673a\u4f20\u8f93\u65f6\u4f1a\u591a\u6b21\u7ecf\u8fc7 netfilter \u6a21\u5757\uff0c\u4f1a\u5e26\u6765\u8fd1 20% \u7684 CPU \u5f00\u9500\u3002\u7531\u4e8e\u5927\u90e8\u5206\u60c5\u51b5\u4e0b \u5bb9\u5668\u7f51\u7edc\u5185\u5e94\u7528\u65e0\u987b\u4f7f\u7528 netfilter \u6a21\u5757\u7684\u529f\u80fd\uff0c FastPath \u6a21\u5757\u53ef\u4ee5\u7ed5\u8fc7 netfilter \u964d\u4f4e CPU \u5f00\u9500\u3002 \u5982\u5bb9\u5668\u7f51\u7edc\u5185\u9700\u8981\u4f7f\u7528 netfilter \u63d0\u4f9b\u7684\u529f\u80fd\u5982 iptables\uff0cipvs\uff0cnftables \u7b49\uff0c\u8be5\u6a21\u5757\u4f1a\u4f7f\u76f8\u5173\u529f\u80fd\u5931\u6548\u3002 \u7531\u4e8e\u5185\u6838\u6a21\u5757\u548c\u5185\u6838\u7248\u672c\u76f8\u5173\uff0c\u65e0\u6cd5\u63d0\u4f9b\u4e00\u4e2a\u5355\u4e00\u9002\u5e94\u6240\u6709\u5185\u6838\u7684\u5185\u6838\u6a21\u5757\u5236\u54c1\u3002\u6211\u4eec\u9884\u5148\u7f16\u8bd1\u4e86\u90e8\u5206\u5185\u6838\u7684 FastPath \u6a21\u5757\uff0c \u53ef\u4ee5\u524d\u5f80 tunning-package \u8fdb\u884c\u4e0b\u8f7d\u3002 \u4e5f\u53ef\u4ee5\u624b\u52a8\u8fdb\u884c\u7f16\u8bd1\uff0c\u65b9\u6cd5\u53c2\u8003 \u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757 \u83b7\u5f97\u5185\u6838\u6a21\u5757\u540e\u53ef\u5728\u6bcf\u4e2a\u8282\u70b9\u4f7f\u7528 insmod kube_ovn_fastpath.ko \u52a0\u8f7d FastPath \u6a21\u5757\uff0c\u5e76\u4f7f\u7528 dmesg \u9a8c\u8bc1\u6a21\u5757\u52a0\u8f7d\u6210\u529f\uff1a # dmesg ... [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in ... OVS \u5185\u6838\u6a21\u5757\u4f18\u5316 \u00b6 OVS \u7684 flow \u5904\u7406\u5305\u62ec\u54c8\u5e0c\u8ba1\u7b97\uff0c\u5339\u914d\u7b49\u64cd\u4f5c\u4f1a\u6d88\u8017\u5927\u7ea6 10% \u5de6\u53f3\u7684 CPU \u8d44\u6e90\u3002\u73b0\u4ee3 x86 CPU \u4e0a\u7684\u4e00\u4e9b\u6307\u4ee4\u96c6\u4f8b\u5982 popcnt \u548c sse4.2 \u53ef\u4ee5 \u52a0\u901f\u76f8\u5173\u8ba1\u7b97\u8fc7\u7a0b\uff0c\u4f46\u5185\u6838\u9ed8\u8ba4\u7f16\u8bd1\u672a\u5f00\u542f\u76f8\u5173\u9009\u9879\u3002\u7ecf\u6d4b\u8bd5\u5728\u5f00\u542f\u76f8\u5e94\u6307\u4ee4\u96c6\u4f18\u5316\u540e\uff0cflow \u76f8\u5173\u64cd\u4f5c CPU \u6d88\u8017\u5c06\u4f1a\u964d\u81f3 5% \u5de6\u53f3\u3002 \u548c FastPath \u6a21\u5757\u7684\u7f16\u8bd1\u7c7b\u4f3c\uff0c\u7531\u4e8e\u5185\u6838\u6a21\u5757\u548c\u5185\u6838\u7248\u672c\u76f8\u5173\uff0c\u65e0\u6cd5\u63d0\u4f9b\u4e00\u4e2a\u5355\u4e00\u9002\u5e94\u6240\u6709\u5185\u6838\u7684\u5185\u6838\u6a21\u5757\u5236\u54c1\u3002\u7528\u6237\u9700\u8981\u624b\u52a8\u7f16\u8bd1\u6216\u8005 \u524d\u5f80 tunning-package \u67e5\u770b\u662f\u5426\u6709\u5df2\u7f16\u8bd1\u597d\u7684\u5236\u54c1\u8fdb\u884c\u4e0b\u8f7d\u3002 \u4f7f\u7528\u8be5\u5185\u6838\u6a21\u5757\u524d\u8bf7\u5148\u786e\u8ba4 CPU \u662f\u5426\u652f\u6301\u76f8\u5173\u6307\u4ee4\u96c6\uff1a cat /proc/cpuinfo | grep popcnt cat /proc/cpuinfo | grep sse4_2 CentOS \u4e0b\u7f16\u8bd1\u5b89\u88c5 \u00b6 \u5b89\u88c5\u76f8\u5173\u7f16\u8bd1\u4f9d\u8d56\u548c\u5185\u6838\u5934\u6587\u4ef6\uff1a yum install -y gcc kernel-devel- $( uname -r ) python3 autoconf automake libtool rpm-build openssl-devel \u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u5e76\u751f\u6210\u5bf9\u5e94 RPM \u6587\u4ef6: git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --with-linux = /lib/modules/ $( uname -r ) /build CFLAGS = \"-g -O2 -mpopcnt -msse4.2\" make rpm-fedora-kmod cd rpm/rpmbuild/RPMS/x86_64/ \u590d\u5236 RPM \u5230\u6bcf\u4e2a\u8282\u70b9\u5e76\u8fdb\u884c\u5b89\u88c5\uff1a rpm -i openvswitch-kmod-2.15.2-1.el7.x86_64.rpm \u82e5\u4e4b\u524d\u5df2\u7ecf\u542f\u52a8\u8fc7 Kube-OVN\uff0c\u65e7\u7248\u672c OVS \u6a21\u5757\u5df2\u52a0\u8f7d\u81f3\u5185\u6838\uff0c\u5efa\u8bae\u91cd\u542f\u673a\u5668\u91cd\u65b0\u52a0\u8f7d\u65b0\u7248\u5185\u6838\u6a21\u5757\u3002 Ubuntu \u4e0b\u7f16\u8bd1\u5b89\u88c5 \u00b6 \u5b89\u88c5\u76f8\u5173\u7f16\u8bd1\u4f9d\u8d56\u548c\u5185\u6838\u5934\u6587\u4ef6\uff1a apt install -y autoconf automake libtool gcc build-essential libssl-dev \u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u5e76\u5b89\u88c5\uff1a apt install -y autoconf automake libtool gcc build-essential libssl-dev git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --prefix = /usr/ --localstatedir = /var --enable-ssl --with-linux = /lib/modules/ $( uname -r ) /build make -j ` nproc ` make install make modules_install cat > /etc/depmod.d/openvswitch.conf << EOF override openvswitch * extra override vport-* * extra EOF depmod -a cp debian/openvswitch-switch.init /etc/init.d/openvswitch-switch /etc/init.d/openvswitch-switch force-reload-kmod \u82e5\u4e4b\u524d\u5df2\u7ecf\u542f\u52a8\u8fc7 Kube-OVN\uff0c\u65e7\u7248\u672c OVS \u6a21\u5757\u5df2\u52a0\u8f7d\u81f3\u5185\u6838\uff0c\u5efa\u8bae\u91cd\u542f\u673a\u5668\u91cd\u65b0\u52a0\u8f7d\u65b0\u7248\u5185\u6838\u6a21\u5757\u3002 \u4f7f\u7528 STT \u7c7b\u578b\u96a7\u9053 \u00b6 \u5e38\u89c1\u7684\u96a7\u9053\u5c01\u88c5\u534f\u8bae\u4f8b\u5982 Geneve \u548c Vxlan \u4f7f\u7528 UDP \u534f\u8bae\u5bf9\u6570\u636e\u5305\u8fdb\u884c\u5c01\u88c5\uff0c\u5728\u5185\u6838\u4e2d\u6709\u826f\u597d\u7684\u652f\u6301\u3002\u4f46\u662f\u5f53\u4f7f\u7528 UDP \u5c01\u88c5 TCP \u6570\u636e\u5305\u65f6\uff0c \u73b0\u4ee3\u64cd\u4f5c\u7cfb\u7edf\u548c\u7f51\u5361\u9488\u5bf9 TCP \u534f\u8bae\u7684\u4f18\u5316\u548c offload \u529f\u80fd\u5c06\u65e0\u6cd5\u987a\u5229\u5de5\u4f5c\uff0c\u5bfc\u81f4 TCP \u7684\u541e\u5410\u91cf\u51fa\u73b0\u663e\u8457\u4e0b\u964d\u3002\u5728\u865a\u62df\u5316\u573a\u666f\u4e0b\u7531\u4e8e CPU \u7684\u9650\u5236\uff0c TCP \u5927\u5305\u7684\u541e\u5410\u91cf\u751a\u81f3\u53ef\u80fd\u53ea\u6709\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u5341\u5206\u4e4b\u4e00\u3002 STT \u63d0\u4f9b\u4e86\u4e00\u79cd\u521b\u65b0\u5f0f\u7684\u4f7f\u7528 TCP \u683c\u5f0f\u6570\u636e\u5305\u8fdb\u884c\u5c01\u88c5\u7684\u96a7\u9053\u534f\u8bae\uff0c\u8be5\u5c01\u88c5\u53ea\u662f\u6a21\u62df\u4e86 TCP \u534f\u8bae\u7684\u5934\u90e8\u683c\u5f0f\uff0c\u5e76\u6ca1\u6709\u771f\u6b63\u5efa\u7acb TCP \u8fde\u63a5\uff0c\u4f46\u662f\u53ef\u4ee5 \u5145\u5206\u5229\u7528\u73b0\u4ee3\u64cd\u4f5c\u7cfb\u7edf\u548c\u7f51\u5361\u7684 TCP \u4f18\u5316\u80fd\u529b\u3002\u5728\u6211\u4eec\u7684\u6d4b\u8bd5\u4e2d TCP \u5927\u5305\u7684\u541e\u5410\u91cf\u80fd\u6709\u6570\u500d\u7684\u63d0\u5347\uff0c\u8fbe\u5230\u63a5\u8fd1\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u6027\u80fd\u6c34\u5e73\u3002 STT \u96a7\u9053\u5e76\u6ca1\u6709\u9884\u5b89\u88c5\u5728\u5185\u6838\u5185\uff0c\u9700\u8981\u901a\u8fc7\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u6765\u5b89\u88c5\uff0cOVS \u5185\u6838\u6a21\u5757\u7684\u7f16\u8bd1\u65b9\u6cd5\u53ef\u4ee5\u53c2\u8003\u4e0a\u4e00\u8282\u3002 STT \u96a7\u9053\u5f00\u542f\uff1a kubectl set env daemonset/ovs-ovn -n kube-system TUNNEL_TYPE = stt kubectl delete pod -n kube-system -lapp = ovs \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6027\u80fd\u8c03\u4f18"},{"location":"advance/performance-tuning/#_1","text":"\u4e3a\u4e86\u4fdd\u6301\u5b89\u88c5\u7684\u7b80\u5355\u548c\u529f\u80fd\u7684\u5b8c\u5907\uff0cKube-OVN \u7684\u9ed8\u8ba4\u5b89\u88c5\u811a\u672c\u5e76\u6ca1\u6709\u5bf9\u6027\u80fd\u9488\u5bf9\u6027\u7684\u4f18\u5316\u3002\u5982\u679c\u5e94\u7528\u5bf9\u5ef6\u8fdf\u548c\u541e\u5410\u91cf\u654f\u611f\uff0c \u7ba1\u7406\u5458\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u5bf9\u6027\u80fd\u8fdb\u884c\u9488\u5bf9\u6027\u4f18\u5316\u3002 \u793e\u533a\u4f1a\u4e0d\u65ad\u8fed\u4ee3\u63a7\u5236\u9762\u677f\u548c\u4f18\u5316\u9762\u7684\u6027\u80fd\uff0c\u90e8\u5206\u901a\u7528\u6027\u80fd\u4f18\u5316\u5df2\u7ecf\u96c6\u6210\u5230\u6700\u65b0\u7248\u672c\uff0c\u5efa\u8bae\u4f7f\u7528\u6700\u65b0\u7248\u672c\u83b7\u5f97\u66f4\u597d\u7684\u9ed8\u8ba4\u6027\u80fd\u3002 \u66f4\u591a\u5173\u4e8e\u6027\u80fd\u4f18\u5316\u7684\u8fc7\u7a0b\u548c\u65b9\u6cd5\u8bba\uff0c\u53ef\u4ee5\u89c2\u770b\u89c6\u9891\u5206\u4eab\uff1a Kube-OVN \u5bb9\u5668\u6027\u80fd\u4f18\u5316\u4e4b\u65c5 \u3002","title":"\u6027\u80fd\u8c03\u4f18"},{"location":"advance/performance-tuning/#_2","text":"\u7531\u4e8e\u8f6f\u786c\u4ef6\u73af\u5883\u7684\u5dee\u5f02\u6781\u5927\uff0c\u8fd9\u91cc\u63d0\u4f9b\u7684\u6027\u80fd\u6d4b\u8bd5\u6570\u636e\u53ea\u80fd\u4f5c\u4e3a\u53c2\u8003\uff0c\u5b9e\u9645\u6d4b\u8bd5\u7ed3\u679c\u4f1a\u548c\u672c\u6587\u6863\u4e2d\u7684\u7ed3\u679c\u5b58\u5728\u8f83\u5927\u5dee\u5f02\u3002 \u5efa\u8bae\u6bd4\u8f83\u4f18\u5316\u524d\u540e\u7684\u6027\u80fd\u6d4b\u8bd5\u7ed3\u679c\uff0c\u548c\u5bbf\u4e3b\u673a\u7f51\u7edc\u548c\u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u6bd4\u8f83\u3002","title":"\u57fa\u51c6\u6d4b\u8bd5"},{"location":"advance/performance-tuning/#overlay","text":"\u73af\u5883\u4fe1\u606f\uff1a Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 Overlay \u6a21\u5f0f CPU: Intel(R) Xeon(R) E-2278G Network: 2*10Gbps, xmit_hash_policy=layer3+4 \u6211\u4eec\u4f7f\u7528 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw \u6d4b\u8bd5 1 \u5b57\u8282\u5c0f\u5305\u4e0b tcp/udp \u7684\u5e26\u5bbd\u548c\u5ef6\u8fdf\uff0c\u5206\u522b\u6d4b\u8bd5\u4f18\u5316\u524d\uff0c\u4f18\u5316\u540e\u4ee5\u53ca\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u6027\u80fd\uff1a Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Default 25.7 22.9 27.1 1.59 Kube-OVN Optimized 13.9 12.9 27.6 5.57 HOST Network 13.1 12.4 28.2 6.02","title":"Overlay \u4f18\u5316\u524d\u540e\u6027\u80fd\u5bf9\u6bd4"},{"location":"advance/performance-tuning/#overlay-underlay-calico","text":"\u4e0b\u9762\u6211\u4eec\u4f1a\u6bd4\u8f83\u4f18\u5316\u540e Kube-OVN \u5728\u4e0d\u540c\u5305\u5927\u5c0f\u4e0b\u7684 Overlay \u548c Underlay \u6027\u80fd\uff0c\u5e76\u548c Calico \u7684 IPIP Always , IPIP never \u4ee5\u53ca\u5bbf\u4e3b\u673a\u7f51\u7edc\u505a\u6bd4\u8f83\u3002 Environment : Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 CPU: AMD EPYC 7402P 24-Core Processor Network: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Overlay 15.2 14.6 23.6 2.65 Kube-OVN Underlay 14.3 13.8 24.2 3.46 Calico IPIP 21.4 20.2 23.6 1.18 Calico NoEncap 19.3 16.9 23.6 1.76 HOST Network 16.6 15.4 24.8 2.64 qperf -t 60 <server ip> -ub -oo msg_size:1K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 16.5 15.8 10.2 2.77 Kube-OVN Underlay 15.9 14.5 9.6 3.22 Calico IPIP 22.5 21.5 1.45 1.14 Calico NoEncap 19.4 18.3 3.76 1.63 HOST Network 18.1 16.6 9.32 2.66 qperf -t 60 <server ip> -ub -oo msg_size:4K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 34.7 41.6 16.0 9.23 Kube-OVN Underlay 32.6 44 15.1 6.71 Calico IPIP 44.8 52.9 2.94 3.26 Calico NoEncap 40 49.6 6.56 4.19 HOST Network 35.9 45.9 14.6 5.59 \u5728\u90e8\u5206\u60c5\u51b5\u4e0b\u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u4f1a\u4f18\u4e8e\u5bbf\u4e3b\u673a\u7f51\u7edc\uff0c\u8fd9\u662f\u4f18\u4e8e\u7ecf\u8fc7\u4f18\u5316\u540e\u5bb9\u5668\u7f51\u7edc\u8def\u5f84\u5b8c\u5168\u7ed5\u8fc7\u4e86 netfilter\uff0c \u800c\u5bbf\u4e3b\u673a\u7f51\u7edc\u7531\u4e8e kube-proxy \u7684\u5b58\u5728\u6240\u6709\u6570\u636e\u5305\u5747\u9700\u7ecf\u8fc7 netfilter\uff0c\u4f1a\u5bfc\u81f4\u5728\u4e00\u4e9b\u73af\u5883\u4e0b\u5bb9\u5668\u7f51\u7edc \u7684\u6d88\u8017\u76f8\u5bf9\u66f4\u5c0f\uff0c\u56e0\u6b64\u4f1a\u6709\u66f4\u597d\u7684\u6027\u80fd\u8868\u73b0\u3002","title":"Overlay\uff0c Underlay \u4ee5\u53ca Calico \u4e0d\u540c\u6a21\u5f0f\u6027\u80fd\u5bf9\u6bd4"},{"location":"advance/performance-tuning/#_3","text":"\u8fd9\u91cc\u4ecb\u7ecd\u7684\u4f18\u5316\u65b9\u6cd5\u548c\u8f6f\u786c\u4ef6\u73af\u5883\u4ee5\u53ca\u6240\u9700\u8981\u7684\u529f\u80fd\u76f8\u5173\uff0c\u8bf7\u4ed4\u7ec6\u4e86\u89e3\u4f18\u5316\u7684\u524d\u63d0\u6761\u4ef6\u518d\u8fdb\u884c\u5c1d\u8bd5\u3002","title":"\u6570\u636e\u5e73\u9762\u6027\u80fd\u4f18\u5316\u65b9\u6cd5"},{"location":"advance/performance-tuning/#cpu","text":"\u90e8\u5206\u73af\u5883\u4e0b CPU \u8fd0\u884c\u5728\u8282\u80fd\u6a21\u5f0f\uff0c\u8be5\u6a21\u5f0f\u4e0b\u6027\u80fd\u8868\u73b0\u5c06\u4f1a\u4e0d\u7a33\u5b9a\uff0c\u5ef6\u8fdf\u4f1a\u51fa\u73b0\u660e\u663e\u589e\u52a0\uff0c\u5efa\u8bae\u4f7f\u7528 CPU \u7684\u6027\u80fd\u6a21\u5f0f\u83b7\u5f97\u66f4\u7a33\u5b9a\u7684\u6027\u80fd\u8868\u73b0\uff1a cpupower frequency-set -g performance","title":"CPU \u6027\u80fd\u6a21\u5f0f\u8c03\u6574"},{"location":"advance/performance-tuning/#_4","text":"\u5728\u6d41\u91cf\u589e\u5927\u7684\u60c5\u51b5\u4e0b\uff0c\u7f13\u51b2\u961f\u5217\u8fc7\u77ed\u53ef\u80fd\u5bfc\u81f4\u8f83\u9ad8\u7684\u4e22\u5305\u7387\u5bfc\u81f4\u6027\u80fd\u663e\u8457\u4e0b\u964d\uff0c\u9700\u8981\u8fdb\u884c\u8c03\u6574 \u68c0\u67e5\u5f53\u524d\u7f51\u5361\u961f\u5217\u957f\u5ea6\uff1a # ethtool -g eno1 Ring parameters for eno1: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 255 RX Mini: 0 RX Jumbo: 0 TX: 255 \u589e\u52a0\u961f\u5217\u957f\u5ea6\u81f3\u6700\u5927\u503c\uff1a ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096","title":"\u7f51\u5361\u786c\u4ef6\u961f\u5217\u8c03\u6574"},{"location":"advance/performance-tuning/#tuned","text":"tuned \u53ef\u4ee5\u4f7f\u7528\u4e00\u7cfb\u5217\u9884\u7f6e\u7684 profile \u6587\u4ef6\u4fdd\u5b58\u4e86\u9488\u5bf9\u7279\u5b9a\u573a\u666f\u7684\u4e00\u7cfb\u5217\u7cfb\u7edf\u4f18\u5316\u914d\u7f6e\u3002 \u9488\u5bf9\u5ef6\u8fdf\u4f18\u5148\u573a\u666f\uff1a tuned-adm profile network-latency \u9488\u5bf9\u541e\u5410\u91cf\u4f18\u5148\u573a\u666f\uff1a tuned-adm profile network-throughput","title":"\u4f7f\u7528 tuned \u4f18\u5316\u7cfb\u7edf\u53c2\u6570"},{"location":"advance/performance-tuning/#_5","text":"\u6211\u4eec\u63a8\u8350\u7981\u7528 irqbalance \u5e76\u5c06\u7f51\u5361\u4e2d\u65ad\u548c\u7279\u5b9a CPU \u8fdb\u884c\u7ed1\u5b9a\uff0c\u6765\u907f\u514d\u5728\u591a\u4e2a CPU \u4e4b\u95f4\u5207\u6362\u5bfc\u81f4\u7684\u6027\u80fd\u6ce2\u52a8\u3002","title":"\u4e2d\u65ad\u7ed1\u5b9a"},{"location":"advance/performance-tuning/#ovn-lb","text":"OVN \u7684 L2 LB \u5b9e\u73b0\u8fc7\u7a0b\u4e2d\u9700\u8981\u8c03\u7528\u5185\u6838\u7684 conntrack \u6a21\u5757\u5e76\u8fdb\u884c recirculate \u5bfc\u81f4\u5927\u91cf\u7684 CPU \u5f00\u9500\uff0c\u7ecf\u6d4b\u8bd5\u8be5\u529f\u80fd\u4f1a\u5e26\u6765 20% \u5de6\u53f3\u7684 CPU \u5f00\u9500\uff0c \u5728 Overlay \u7f51\u7edc\u6a21\u5f0f\u4e0b\u53ef\u4ee5\u4f7f\u7528 kube-proxy \u5b8c\u6210 Service \u8f6c\u53d1\u529f\u80fd\uff0c\u83b7\u5f97\u66f4\u597d\u7684 Pod-to-Pod \u6027\u80fd\u3002\u53ef\u4ee5\u5728 kube-ovn-controller \u4e2d\u5173\u95ed\u8be5\u529f\u80fd\uff1a command : - /kube-ovn/start-controller.sh args : ... - --enable-lb=false ... Underlay \u6a21\u5f0f\u4e0b kube-proxy \u65e0\u6cd5\u4f7f\u7528 iptables \u6216 ipvs \u63a7\u5236\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\uff0c\u5982\u9700\u5173\u95ed LB \u529f\u80fd\u9700\u8981\u786e\u8ba4\u662f\u5426\u4e0d\u9700\u8981 Service \u529f\u80fd\u3002","title":"\u5173\u95ed OVN LB"},{"location":"advance/performance-tuning/#fastpath","text":"\u7531\u4e8e\u5bb9\u5668\u7f51\u7edc\u548c\u5bbf\u4e3b\u673a\u7f51\u7edc\u5728\u4e0d\u540c\u7684 network ns\uff0c\u6570\u636e\u5305\u5728\u8de8\u5bbf\u4e3b\u673a\u4f20\u8f93\u65f6\u4f1a\u591a\u6b21\u7ecf\u8fc7 netfilter \u6a21\u5757\uff0c\u4f1a\u5e26\u6765\u8fd1 20% \u7684 CPU \u5f00\u9500\u3002\u7531\u4e8e\u5927\u90e8\u5206\u60c5\u51b5\u4e0b \u5bb9\u5668\u7f51\u7edc\u5185\u5e94\u7528\u65e0\u987b\u4f7f\u7528 netfilter \u6a21\u5757\u7684\u529f\u80fd\uff0c FastPath \u6a21\u5757\u53ef\u4ee5\u7ed5\u8fc7 netfilter \u964d\u4f4e CPU \u5f00\u9500\u3002 \u5982\u5bb9\u5668\u7f51\u7edc\u5185\u9700\u8981\u4f7f\u7528 netfilter \u63d0\u4f9b\u7684\u529f\u80fd\u5982 iptables\uff0cipvs\uff0cnftables \u7b49\uff0c\u8be5\u6a21\u5757\u4f1a\u4f7f\u76f8\u5173\u529f\u80fd\u5931\u6548\u3002 \u7531\u4e8e\u5185\u6838\u6a21\u5757\u548c\u5185\u6838\u7248\u672c\u76f8\u5173\uff0c\u65e0\u6cd5\u63d0\u4f9b\u4e00\u4e2a\u5355\u4e00\u9002\u5e94\u6240\u6709\u5185\u6838\u7684\u5185\u6838\u6a21\u5757\u5236\u54c1\u3002\u6211\u4eec\u9884\u5148\u7f16\u8bd1\u4e86\u90e8\u5206\u5185\u6838\u7684 FastPath \u6a21\u5757\uff0c \u53ef\u4ee5\u524d\u5f80 tunning-package \u8fdb\u884c\u4e0b\u8f7d\u3002 \u4e5f\u53ef\u4ee5\u624b\u52a8\u8fdb\u884c\u7f16\u8bd1\uff0c\u65b9\u6cd5\u53c2\u8003 \u624b\u52a8\u7f16\u8bd1 FastPath \u6a21\u5757 \u83b7\u5f97\u5185\u6838\u6a21\u5757\u540e\u53ef\u5728\u6bcf\u4e2a\u8282\u70b9\u4f7f\u7528 insmod kube_ovn_fastpath.ko \u52a0\u8f7d FastPath \u6a21\u5757\uff0c\u5e76\u4f7f\u7528 dmesg \u9a8c\u8bc1\u6a21\u5757\u52a0\u8f7d\u6210\u529f\uff1a # dmesg ... [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in ...","title":"\u5185\u6838 FastPath \u6a21\u5757"},{"location":"advance/performance-tuning/#ovs","text":"OVS \u7684 flow \u5904\u7406\u5305\u62ec\u54c8\u5e0c\u8ba1\u7b97\uff0c\u5339\u914d\u7b49\u64cd\u4f5c\u4f1a\u6d88\u8017\u5927\u7ea6 10% \u5de6\u53f3\u7684 CPU \u8d44\u6e90\u3002\u73b0\u4ee3 x86 CPU \u4e0a\u7684\u4e00\u4e9b\u6307\u4ee4\u96c6\u4f8b\u5982 popcnt \u548c sse4.2 \u53ef\u4ee5 \u52a0\u901f\u76f8\u5173\u8ba1\u7b97\u8fc7\u7a0b\uff0c\u4f46\u5185\u6838\u9ed8\u8ba4\u7f16\u8bd1\u672a\u5f00\u542f\u76f8\u5173\u9009\u9879\u3002\u7ecf\u6d4b\u8bd5\u5728\u5f00\u542f\u76f8\u5e94\u6307\u4ee4\u96c6\u4f18\u5316\u540e\uff0cflow \u76f8\u5173\u64cd\u4f5c CPU \u6d88\u8017\u5c06\u4f1a\u964d\u81f3 5% \u5de6\u53f3\u3002 \u548c FastPath \u6a21\u5757\u7684\u7f16\u8bd1\u7c7b\u4f3c\uff0c\u7531\u4e8e\u5185\u6838\u6a21\u5757\u548c\u5185\u6838\u7248\u672c\u76f8\u5173\uff0c\u65e0\u6cd5\u63d0\u4f9b\u4e00\u4e2a\u5355\u4e00\u9002\u5e94\u6240\u6709\u5185\u6838\u7684\u5185\u6838\u6a21\u5757\u5236\u54c1\u3002\u7528\u6237\u9700\u8981\u624b\u52a8\u7f16\u8bd1\u6216\u8005 \u524d\u5f80 tunning-package \u67e5\u770b\u662f\u5426\u6709\u5df2\u7f16\u8bd1\u597d\u7684\u5236\u54c1\u8fdb\u884c\u4e0b\u8f7d\u3002 \u4f7f\u7528\u8be5\u5185\u6838\u6a21\u5757\u524d\u8bf7\u5148\u786e\u8ba4 CPU \u662f\u5426\u652f\u6301\u76f8\u5173\u6307\u4ee4\u96c6\uff1a cat /proc/cpuinfo | grep popcnt cat /proc/cpuinfo | grep sse4_2","title":"OVS \u5185\u6838\u6a21\u5757\u4f18\u5316"},{"location":"advance/performance-tuning/#centos","text":"\u5b89\u88c5\u76f8\u5173\u7f16\u8bd1\u4f9d\u8d56\u548c\u5185\u6838\u5934\u6587\u4ef6\uff1a yum install -y gcc kernel-devel- $( uname -r ) python3 autoconf automake libtool rpm-build openssl-devel \u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u5e76\u751f\u6210\u5bf9\u5e94 RPM \u6587\u4ef6: git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --with-linux = /lib/modules/ $( uname -r ) /build CFLAGS = \"-g -O2 -mpopcnt -msse4.2\" make rpm-fedora-kmod cd rpm/rpmbuild/RPMS/x86_64/ \u590d\u5236 RPM \u5230\u6bcf\u4e2a\u8282\u70b9\u5e76\u8fdb\u884c\u5b89\u88c5\uff1a rpm -i openvswitch-kmod-2.15.2-1.el7.x86_64.rpm \u82e5\u4e4b\u524d\u5df2\u7ecf\u542f\u52a8\u8fc7 Kube-OVN\uff0c\u65e7\u7248\u672c OVS \u6a21\u5757\u5df2\u52a0\u8f7d\u81f3\u5185\u6838\uff0c\u5efa\u8bae\u91cd\u542f\u673a\u5668\u91cd\u65b0\u52a0\u8f7d\u65b0\u7248\u5185\u6838\u6a21\u5757\u3002","title":"CentOS \u4e0b\u7f16\u8bd1\u5b89\u88c5"},{"location":"advance/performance-tuning/#ubuntu","text":"\u5b89\u88c5\u76f8\u5173\u7f16\u8bd1\u4f9d\u8d56\u548c\u5185\u6838\u5934\u6587\u4ef6\uff1a apt install -y autoconf automake libtool gcc build-essential libssl-dev \u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u5e76\u5b89\u88c5\uff1a apt install -y autoconf automake libtool gcc build-essential libssl-dev git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --prefix = /usr/ --localstatedir = /var --enable-ssl --with-linux = /lib/modules/ $( uname -r ) /build make -j ` nproc ` make install make modules_install cat > /etc/depmod.d/openvswitch.conf << EOF override openvswitch * extra override vport-* * extra EOF depmod -a cp debian/openvswitch-switch.init /etc/init.d/openvswitch-switch /etc/init.d/openvswitch-switch force-reload-kmod \u82e5\u4e4b\u524d\u5df2\u7ecf\u542f\u52a8\u8fc7 Kube-OVN\uff0c\u65e7\u7248\u672c OVS \u6a21\u5757\u5df2\u52a0\u8f7d\u81f3\u5185\u6838\uff0c\u5efa\u8bae\u91cd\u542f\u673a\u5668\u91cd\u65b0\u52a0\u8f7d\u65b0\u7248\u5185\u6838\u6a21\u5757\u3002","title":"Ubuntu \u4e0b\u7f16\u8bd1\u5b89\u88c5"},{"location":"advance/performance-tuning/#stt","text":"\u5e38\u89c1\u7684\u96a7\u9053\u5c01\u88c5\u534f\u8bae\u4f8b\u5982 Geneve \u548c Vxlan \u4f7f\u7528 UDP \u534f\u8bae\u5bf9\u6570\u636e\u5305\u8fdb\u884c\u5c01\u88c5\uff0c\u5728\u5185\u6838\u4e2d\u6709\u826f\u597d\u7684\u652f\u6301\u3002\u4f46\u662f\u5f53\u4f7f\u7528 UDP \u5c01\u88c5 TCP \u6570\u636e\u5305\u65f6\uff0c \u73b0\u4ee3\u64cd\u4f5c\u7cfb\u7edf\u548c\u7f51\u5361\u9488\u5bf9 TCP \u534f\u8bae\u7684\u4f18\u5316\u548c offload \u529f\u80fd\u5c06\u65e0\u6cd5\u987a\u5229\u5de5\u4f5c\uff0c\u5bfc\u81f4 TCP \u7684\u541e\u5410\u91cf\u51fa\u73b0\u663e\u8457\u4e0b\u964d\u3002\u5728\u865a\u62df\u5316\u573a\u666f\u4e0b\u7531\u4e8e CPU \u7684\u9650\u5236\uff0c TCP \u5927\u5305\u7684\u541e\u5410\u91cf\u751a\u81f3\u53ef\u80fd\u53ea\u6709\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u5341\u5206\u4e4b\u4e00\u3002 STT \u63d0\u4f9b\u4e86\u4e00\u79cd\u521b\u65b0\u5f0f\u7684\u4f7f\u7528 TCP \u683c\u5f0f\u6570\u636e\u5305\u8fdb\u884c\u5c01\u88c5\u7684\u96a7\u9053\u534f\u8bae\uff0c\u8be5\u5c01\u88c5\u53ea\u662f\u6a21\u62df\u4e86 TCP \u534f\u8bae\u7684\u5934\u90e8\u683c\u5f0f\uff0c\u5e76\u6ca1\u6709\u771f\u6b63\u5efa\u7acb TCP \u8fde\u63a5\uff0c\u4f46\u662f\u53ef\u4ee5 \u5145\u5206\u5229\u7528\u73b0\u4ee3\u64cd\u4f5c\u7cfb\u7edf\u548c\u7f51\u5361\u7684 TCP \u4f18\u5316\u80fd\u529b\u3002\u5728\u6211\u4eec\u7684\u6d4b\u8bd5\u4e2d TCP \u5927\u5305\u7684\u541e\u5410\u91cf\u80fd\u6709\u6570\u500d\u7684\u63d0\u5347\uff0c\u8fbe\u5230\u63a5\u8fd1\u5bbf\u4e3b\u673a\u7f51\u7edc\u7684\u6027\u80fd\u6c34\u5e73\u3002 STT \u96a7\u9053\u5e76\u6ca1\u6709\u9884\u5b89\u88c5\u5728\u5185\u6838\u5185\uff0c\u9700\u8981\u901a\u8fc7\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u6765\u5b89\u88c5\uff0cOVS \u5185\u6838\u6a21\u5757\u7684\u7f16\u8bd1\u65b9\u6cd5\u53ef\u4ee5\u53c2\u8003\u4e0a\u4e00\u8282\u3002 STT \u96a7\u9053\u5f00\u542f\uff1a kubectl set env daemonset/ovs-ovn -n kube-system TUNNEL_TYPE = stt kubectl delete pod -n kube-system -lapp = ovs \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 STT \u7c7b\u578b\u96a7\u9053"},{"location":"advance/security-group/","text":"SecurityGroup \u4f7f\u7528 \u00b6 Kube-OVN \u652f\u6301\u4e86\u5b89\u5168\u7ec4\u7684\u914d\u7f6e\uff0c\u914d\u7f6e\u5b89\u5168\u7ec4\u4f7f\u7528\u7684 CRD \u4e3a SecurityGroup\u3002 \u5b89\u5168\u7ec4\u793a\u4f8b \u00b6 apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-example spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 1 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address \u5b89\u5168\u7ec4\u5404\u5b57\u6bb5\u7684\u5177\u4f53\u542b\u4e49\uff0c\u53ef\u4ee5\u53c2\u8003 Kube-OVN \u63a5\u53e3\u89c4\u8303 \u3002 Pod \u901a\u8fc7\u6dfb\u52a0 annotation \u6765\u7ed1\u5b9a\u5b89\u5168\u7ec4\uff0c\u4f7f\u7528\u7684 annotation \u6709\u4e24\u4e2a\uff1a ovn.kubernetes.io/port_security : \"true\" ovn.kubernetes.io/security_groups : sg-example \u6ce8\u610f\u4e8b\u9879 \u00b6 \u5b89\u5168\u7ec4\u6700\u540e\u662f\u901a\u8fc7\u8bbe\u7f6e ACL \u89c4\u5219\u6765\u9650\u5236\u8bbf\u95ee\u7684\uff0cOVN \u6587\u6863\u4e2d\u63d0\u5230\uff0c\u5982\u679c\u5339\u914d\u5230\u7684\u4e24\u4e2a ACL \u89c4\u5219\u62e5\u6709\u76f8\u540c\u7684\u4f18\u5148\u7ea7\uff0c\u5b9e\u9645\u8d77\u4f5c\u7528\u7684\u662f\u54ea\u4e2a ACL \u662f\u4e0d\u786e\u5b9a\u7684\u3002\u56e0\u6b64\u8bbe\u7f6e\u5b89\u5168\u7ec4\u89c4\u5219\u7684\u65f6\u5019\uff0c\u9700\u8981\u6ce8\u610f\u533a\u5206\u4f18\u5148\u7ea7\u3002 \u5f53\u6dfb\u52a0\u5b89\u5168\u7ec4\u7684\u65f6\u5019\uff0c\u8981\u6e05\u695a\u7684\u77e5\u9053\u662f\u5728\u6dfb\u52a0\u4ec0\u4e48\u9650\u5236\u3002Kube-OVN \u4f5c\u4e3a CNI\uff0c\u521b\u5efa Pod \u540e\u4f1a\u8fdb\u884c Pod \u5230\u7f51\u5173\u7684\u8fde\u901a\u6027\u6d4b\u8bd5\uff0c\u5982\u679c\u8bbf\u95ee\u4e0d\u901a\u7f51\u5173\uff0c\u5c31\u4f1a\u5bfc\u81f4 Pod \u4e00\u76f4\u5904\u4e8e ContainerCreating \u72b6\u6001\uff0c\u65e0\u6cd5\u987a\u5229\u5207\u6362\u5230 Running \u72b6\u6001\u3002 \u5b9e\u9645\u6d4b\u8bd5 \u00b6 \u5229\u7528\u4ee5\u4e0b yaml \u521b\u5efa Pod\uff0c\u5728 annotation \u4e2d\u6307\u5b9a\u7ed1\u5b9a\u793a\u4f8b\u4e2d\u7684\u5b89\u5168\u7ec4\uff1a apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-example' name : sg-test-pod namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u5b9e\u9645\u6d4b\u8bd5\u7ed3\u679c\u663e\u793a\u5982\u4e0b\uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h32m <none> kube-ovn-worker <none> <none> test-99fff7f86-52h9r 1 /1 Running 0 5h41m 10 .16.0.14 kube-ovn-control-plane <none> <none> test-99fff7f86-qcgjw 1 /1 Running 0 5h43m 10 .16.0.13 kube-ovn-worker <none> <none> \u6267\u884c kubectl describe pod \u67e5\u770b Pod \u7684\u4fe1\u606f\uff0c\u53ef\u4ee5\u770b\u5230\u62a5\u9519\u63d0\u793a\uff1a # kubectl describe pod sg-test-pod Name: sg-test-pod Namespace: default Priority: 0 Node: kube-ovn-worker/172.18.0.2 Start Time: Tue, 28 Feb 2023 10 :29:36 +0800 Labels: app = static Annotations: ovn.kubernetes.io/allocated: true ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.15 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:FA:17:97 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/port_security: true ovn.kubernetes.io/routed: true ovn.kubernetes.io/security_groups: sg-allow-reject Status: Pending IP: IPs: <none> \u00b7 \u00b7 \u00b7 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 5m3s ( x70 over 4h59m ) kubelet ( combined from similar events ) : Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"40636e0c7f1ade5500fa958486163d74f2e2300051a71522a9afd7ba0538afb6\" : plugin type = \"kube-ovn\" failed ( add ) : RPC failed ; request ip return 500 configure nic failed 10 .16.0.15 network not ready after 200 ping 10 .16.0.1 \u4fee\u6539\u5b89\u5168\u7ec4\u7684\u89c4\u5219\uff0c\u6dfb\u52a0\u5230\u7f51\u5173\u7684\u8bbf\u95ee\u89c4\u5219\uff0c\u53c2\u8003\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-gw-both spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 2 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.1/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 2 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : icmp remoteAddress : 10.16.0.1/16 remoteType : address \u5206\u522b\u5728\u5165\u65b9\u5411\u548c\u51fa\u65b9\u5411\u89c4\u5219\u4e2d\uff0c\u6dfb\u52a0\u5141\u8bb8\u5230\u7f51\u5173\u7684\u8bbf\u95ee\u89c4\u5219\uff0c\u5e76\u4e14\u8bbe\u7f6e\u8be5\u89c4\u5219\u7684\u4f18\u5148\u7ea7\u6700\u9ad8\u3002 \u5229\u7528\u4ee5\u4e0b yaml \u7ed1\u5b9a\u5b89\u5168\u7ec4\uff0c\u90e8\u7f72 Pod \u540e\uff0c\u786e\u8ba4 Pod \u53ef\u4ee5\u6b63\u5e38\u8fd0\u884c\uff1a apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-gw-both' name : sg-gw-both namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u90e8\u7f72\u540e\u67e5\u770b Pod \u4fe1\u606f\uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h41m <none> kube-ovn-worker <none> <none> sg-gw-both 1 /1 Running 0 5h37m 10 .16.0.19 kube-ovn-worker <none> <none> \u56e0\u6b64\u5bf9\u4e8e\u5b89\u5168\u7ec4\u7684\u4f7f\u7528\uff0c\u8981\u7279\u522b\u660e\u786e\u6dfb\u52a0\u7684\u9650\u5236\u89c4\u5219\u7684\u4f5c\u7528\u3002\u5982\u679c\u5355\u7eaf\u662f\u9650\u5236\u6d41\u91cf\u8bbf\u95ee\uff0c\u53ef\u4ee5\u8003\u8651\u4f7f\u7528\u7f51\u7edc\u7b56\u7565\u5b9e\u73b0\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"SecurityGroup \u4f7f\u7528"},{"location":"advance/security-group/#securitygroup","text":"Kube-OVN \u652f\u6301\u4e86\u5b89\u5168\u7ec4\u7684\u914d\u7f6e\uff0c\u914d\u7f6e\u5b89\u5168\u7ec4\u4f7f\u7528\u7684 CRD \u4e3a SecurityGroup\u3002","title":"SecurityGroup \u4f7f\u7528"},{"location":"advance/security-group/#_1","text":"apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-example spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 1 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address \u5b89\u5168\u7ec4\u5404\u5b57\u6bb5\u7684\u5177\u4f53\u542b\u4e49\uff0c\u53ef\u4ee5\u53c2\u8003 Kube-OVN \u63a5\u53e3\u89c4\u8303 \u3002 Pod \u901a\u8fc7\u6dfb\u52a0 annotation \u6765\u7ed1\u5b9a\u5b89\u5168\u7ec4\uff0c\u4f7f\u7528\u7684 annotation \u6709\u4e24\u4e2a\uff1a ovn.kubernetes.io/port_security : \"true\" ovn.kubernetes.io/security_groups : sg-example","title":"\u5b89\u5168\u7ec4\u793a\u4f8b"},{"location":"advance/security-group/#_2","text":"\u5b89\u5168\u7ec4\u6700\u540e\u662f\u901a\u8fc7\u8bbe\u7f6e ACL \u89c4\u5219\u6765\u9650\u5236\u8bbf\u95ee\u7684\uff0cOVN \u6587\u6863\u4e2d\u63d0\u5230\uff0c\u5982\u679c\u5339\u914d\u5230\u7684\u4e24\u4e2a ACL \u89c4\u5219\u62e5\u6709\u76f8\u540c\u7684\u4f18\u5148\u7ea7\uff0c\u5b9e\u9645\u8d77\u4f5c\u7528\u7684\u662f\u54ea\u4e2a ACL \u662f\u4e0d\u786e\u5b9a\u7684\u3002\u56e0\u6b64\u8bbe\u7f6e\u5b89\u5168\u7ec4\u89c4\u5219\u7684\u65f6\u5019\uff0c\u9700\u8981\u6ce8\u610f\u533a\u5206\u4f18\u5148\u7ea7\u3002 \u5f53\u6dfb\u52a0\u5b89\u5168\u7ec4\u7684\u65f6\u5019\uff0c\u8981\u6e05\u695a\u7684\u77e5\u9053\u662f\u5728\u6dfb\u52a0\u4ec0\u4e48\u9650\u5236\u3002Kube-OVN \u4f5c\u4e3a CNI\uff0c\u521b\u5efa Pod \u540e\u4f1a\u8fdb\u884c Pod \u5230\u7f51\u5173\u7684\u8fde\u901a\u6027\u6d4b\u8bd5\uff0c\u5982\u679c\u8bbf\u95ee\u4e0d\u901a\u7f51\u5173\uff0c\u5c31\u4f1a\u5bfc\u81f4 Pod \u4e00\u76f4\u5904\u4e8e ContainerCreating \u72b6\u6001\uff0c\u65e0\u6cd5\u987a\u5229\u5207\u6362\u5230 Running \u72b6\u6001\u3002","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"advance/security-group/#_3","text":"\u5229\u7528\u4ee5\u4e0b yaml \u521b\u5efa Pod\uff0c\u5728 annotation \u4e2d\u6307\u5b9a\u7ed1\u5b9a\u793a\u4f8b\u4e2d\u7684\u5b89\u5168\u7ec4\uff1a apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-example' name : sg-test-pod namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u5b9e\u9645\u6d4b\u8bd5\u7ed3\u679c\u663e\u793a\u5982\u4e0b\uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h32m <none> kube-ovn-worker <none> <none> test-99fff7f86-52h9r 1 /1 Running 0 5h41m 10 .16.0.14 kube-ovn-control-plane <none> <none> test-99fff7f86-qcgjw 1 /1 Running 0 5h43m 10 .16.0.13 kube-ovn-worker <none> <none> \u6267\u884c kubectl describe pod \u67e5\u770b Pod \u7684\u4fe1\u606f\uff0c\u53ef\u4ee5\u770b\u5230\u62a5\u9519\u63d0\u793a\uff1a # kubectl describe pod sg-test-pod Name: sg-test-pod Namespace: default Priority: 0 Node: kube-ovn-worker/172.18.0.2 Start Time: Tue, 28 Feb 2023 10 :29:36 +0800 Labels: app = static Annotations: ovn.kubernetes.io/allocated: true ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.15 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:FA:17:97 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/port_security: true ovn.kubernetes.io/routed: true ovn.kubernetes.io/security_groups: sg-allow-reject Status: Pending IP: IPs: <none> \u00b7 \u00b7 \u00b7 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 5m3s ( x70 over 4h59m ) kubelet ( combined from similar events ) : Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"40636e0c7f1ade5500fa958486163d74f2e2300051a71522a9afd7ba0538afb6\" : plugin type = \"kube-ovn\" failed ( add ) : RPC failed ; request ip return 500 configure nic failed 10 .16.0.15 network not ready after 200 ping 10 .16.0.1 \u4fee\u6539\u5b89\u5168\u7ec4\u7684\u89c4\u5219\uff0c\u6dfb\u52a0\u5230\u7f51\u5173\u7684\u8bbf\u95ee\u89c4\u5219\uff0c\u53c2\u8003\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-gw-both spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 2 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.1/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 2 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : icmp remoteAddress : 10.16.0.1/16 remoteType : address \u5206\u522b\u5728\u5165\u65b9\u5411\u548c\u51fa\u65b9\u5411\u89c4\u5219\u4e2d\uff0c\u6dfb\u52a0\u5141\u8bb8\u5230\u7f51\u5173\u7684\u8bbf\u95ee\u89c4\u5219\uff0c\u5e76\u4e14\u8bbe\u7f6e\u8be5\u89c4\u5219\u7684\u4f18\u5148\u7ea7\u6700\u9ad8\u3002 \u5229\u7528\u4ee5\u4e0b yaml \u7ed1\u5b9a\u5b89\u5168\u7ec4\uff0c\u90e8\u7f72 Pod \u540e\uff0c\u786e\u8ba4 Pod \u53ef\u4ee5\u6b63\u5e38\u8fd0\u884c\uff1a apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-gw-both' name : sg-gw-both namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u90e8\u7f72\u540e\u67e5\u770b Pod \u4fe1\u606f\uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h41m <none> kube-ovn-worker <none> <none> sg-gw-both 1 /1 Running 0 5h37m 10 .16.0.19 kube-ovn-worker <none> <none> \u56e0\u6b64\u5bf9\u4e8e\u5b89\u5168\u7ec4\u7684\u4f7f\u7528\uff0c\u8981\u7279\u522b\u660e\u786e\u6dfb\u52a0\u7684\u9650\u5236\u89c4\u5219\u7684\u4f5c\u7528\u3002\u5982\u679c\u5355\u7eaf\u662f\u9650\u5236\u6d41\u91cf\u8bbf\u95ee\uff0c\u53ef\u4ee5\u8003\u8651\u4f7f\u7528\u7f51\u7edc\u7b56\u7565\u5b9e\u73b0\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b9e\u9645\u6d4b\u8bd5"},{"location":"advance/vip/","text":"VIP \u9884\u7559\u8bbe\u7f6e \u00b6 \u5728\u4e00\u4e9b\u573a\u666f\u4e0b\u6211\u4eec\u5e0c\u671b\u52a8\u6001\u7684\u9884\u7559\u4e00\u90e8\u5206 IP \u4f46\u662f\u5e76\u4e0d\u5206\u914d\u7ed9 Pod \u800c\u662f\u5206\u914d\u7ed9\u5176\u4ed6\u7684\u57fa\u7840\u8bbe\u65bd\u542f\u7528\uff0c\u4f8b\u5982\uff1a Kubernetes \u5d4c\u5957 Kubernetes \u7684\u573a\u666f\u4e2d\u4e0a\u5c42 Kubernetes \u4f7f\u7528 Underlay \u7f51\u7edc\u4f1a\u5360\u7528\u5e95\u5c42 Subnet \u53ef\u7528\u5730\u5740\u3002 LB \u6216\u5176\u4ed6\u7f51\u7edc\u57fa\u7840\u8bbe\u65bd\u9700\u8981\u4f7f\u7528\u4e00\u4e2a Subnet \u5185\u7684 IP\uff0c\u4f46\u4e0d\u4f1a\u5355\u72ec\u8d77 Pod\u3002 \u521b\u5efa\u968f\u673a\u5730\u5740 VIP \u00b6 \u5982\u679c\u53ea\u662f\u4e3a\u4e86\u9884\u7559\u82e5\u5e72 IP \u800c\u5bf9 IP \u5730\u5740\u672c\u8eab\u6ca1\u6709\u8981\u6c42\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684 yaml \u8fdb\u884c\u521b\u5efa\uff1a apiVersion : kubeovn.io/v1 kind : Vip metadata : name : vip-dynamic-01 spec : subnet : ovn-default type : \"\" subnet : \u5c06\u4ece\u8be5 Subnet \u4e2d\u9884\u7559 IP\u3002 type : \u76ee\u524d\u652f\u6301\u4e24\u79cd\u7c7b\u578b\uff0c\u4e3a\u7a7a\u8868\u793a\u4ec5\u7528\u4e8e ipam ip \u5360\u4f4d\uff0c switch_lb_vip \u8868\u793a\u8be5 vip \u4ec5\u7528\u4e8e switch lb \u524d\u7aef vip \u548c\u540e\u7aef ip \u9700\u5904\u4e8e\u540c\u4e00\u5b50\u7f51\u3002 \u521b\u5efa\u6210\u529f\u540e\u67e5\u8be2\u8be5 VIP\uff1a # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY vip-dynamic-01 10 .16.0.12 00 :00:00:F0:DB:25 ovn-default true \u53ef\u89c1\u8be5 VIP \u88ab\u5206\u914d\u4e86 10.16.0.12 \u7684 IP \u5730\u5740\uff0c\u8be5\u5730\u5740\u53ef\u4ee5\u4e4b\u540e\u4f9b\u5176\u4ed6\u7f51\u7edc\u57fa\u7840\u8bbe\u65bd\u4f7f\u7528\u3002 \u521b\u5efa\u56fa\u5b9a\u5730\u5740 VIP \u00b6 \u5982\u5bf9\u9884\u7559\u7684 VIP \u7684 IP \u5730\u5740\u6709\u9700\u6c42\u53ef\u4f7f\u7528\u4e0b\u9762\u7684 yaml \u8fdb\u884c\u56fa\u5b9a\u5206\u914d\uff1a apiVersion : kubeovn.io/v1 kind : Vip metadata : name : static-vip01 spec : subnet : ovn-default v4ip : \"10.16.0.121\" subnet : \u5c06\u4ece\u8be5 Subnet \u4e2d\u9884\u7559 IP\u3002 v4ip : \u56fa\u5b9a\u5206\u914d\u7684 IP \u5730\u5740\uff0c\u8be5\u5730\u5740\u9700\u5728 subnet \u7684 CIDR \u8303\u56f4\u5185\u3002 \u521b\u5efa\u6210\u529f\u540e\u67e5\u8be2\u8be5 VIP\uff1a # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY static-vip01 10 .16.0.121 00 :00:00:F0:DB:26 ovn-default true \u53ef\u89c1\u8be5 VIP \u88ab\u5206\u914d\u4e86\u6240\u9884\u671f\u7684 IP \u5730\u5740\u3002 Pod \u4f7f\u7528 VIP \u6765\u56fa\u5b9a IP \u00b6 \u8be5\u529f\u80fd\u76ee\u524d\u53ea\u5728 master \u5206\u652f\u652f\u6301\u3002 \u53ef\u4ee5\u4f7f\u7528 annotation \u5c06\u67d0\u4e2a VIP \u5206\u914d\u7ed9\u4e00\u4e2a Pod\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip annotations : ovn.kubernetes.io/vip : vip-dynamic-01 # \u6307\u5b9a vip namespace : default spec : containers : - name : static-ip image : docker.io/library/nginx:alpine StatefulSet \u548c Kubevirt VM \u4fdd\u7559 VIP \u00b6 \u9488\u5bf9 StatefulSet \u548c VM \u7684\u7279\u6b8a\u6027\uff0c\u5728\u4ed6\u4eec\u7684 Pod \u9500\u6bc1\u518d\u62c9\u8d77\u8d77\u540e\u4f1a\u91cd\u65b0\u4f7f\u7528\u4e4b\u524d\u8bbe\u7f6e\u7684 VIP\u3002 VM \u4fdd\u7559 VIP \u9700\u8981\u786e\u4fdd kube-ovn-controller \u7684 keep-vm-ip \u53c2\u6570\u4e3a true \u3002\u8bf7\u53c2\u8003 Kubevirt VM \u56fa\u5b9a\u5730\u5740\u5f00\u542f\u8bbe\u7f6e \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VIP \u9884\u7559\u8bbe\u7f6e"},{"location":"advance/vip/#vip","text":"\u5728\u4e00\u4e9b\u573a\u666f\u4e0b\u6211\u4eec\u5e0c\u671b\u52a8\u6001\u7684\u9884\u7559\u4e00\u90e8\u5206 IP \u4f46\u662f\u5e76\u4e0d\u5206\u914d\u7ed9 Pod \u800c\u662f\u5206\u914d\u7ed9\u5176\u4ed6\u7684\u57fa\u7840\u8bbe\u65bd\u542f\u7528\uff0c\u4f8b\u5982\uff1a Kubernetes \u5d4c\u5957 Kubernetes \u7684\u573a\u666f\u4e2d\u4e0a\u5c42 Kubernetes \u4f7f\u7528 Underlay \u7f51\u7edc\u4f1a\u5360\u7528\u5e95\u5c42 Subnet \u53ef\u7528\u5730\u5740\u3002 LB \u6216\u5176\u4ed6\u7f51\u7edc\u57fa\u7840\u8bbe\u65bd\u9700\u8981\u4f7f\u7528\u4e00\u4e2a Subnet \u5185\u7684 IP\uff0c\u4f46\u4e0d\u4f1a\u5355\u72ec\u8d77 Pod\u3002","title":"VIP \u9884\u7559\u8bbe\u7f6e"},{"location":"advance/vip/#vip_1","text":"\u5982\u679c\u53ea\u662f\u4e3a\u4e86\u9884\u7559\u82e5\u5e72 IP \u800c\u5bf9 IP \u5730\u5740\u672c\u8eab\u6ca1\u6709\u8981\u6c42\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684 yaml \u8fdb\u884c\u521b\u5efa\uff1a apiVersion : kubeovn.io/v1 kind : Vip metadata : name : vip-dynamic-01 spec : subnet : ovn-default type : \"\" subnet : \u5c06\u4ece\u8be5 Subnet \u4e2d\u9884\u7559 IP\u3002 type : \u76ee\u524d\u652f\u6301\u4e24\u79cd\u7c7b\u578b\uff0c\u4e3a\u7a7a\u8868\u793a\u4ec5\u7528\u4e8e ipam ip \u5360\u4f4d\uff0c switch_lb_vip \u8868\u793a\u8be5 vip \u4ec5\u7528\u4e8e switch lb \u524d\u7aef vip \u548c\u540e\u7aef ip \u9700\u5904\u4e8e\u540c\u4e00\u5b50\u7f51\u3002 \u521b\u5efa\u6210\u529f\u540e\u67e5\u8be2\u8be5 VIP\uff1a # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY vip-dynamic-01 10 .16.0.12 00 :00:00:F0:DB:25 ovn-default true \u53ef\u89c1\u8be5 VIP \u88ab\u5206\u914d\u4e86 10.16.0.12 \u7684 IP \u5730\u5740\uff0c\u8be5\u5730\u5740\u53ef\u4ee5\u4e4b\u540e\u4f9b\u5176\u4ed6\u7f51\u7edc\u57fa\u7840\u8bbe\u65bd\u4f7f\u7528\u3002","title":"\u521b\u5efa\u968f\u673a\u5730\u5740 VIP"},{"location":"advance/vip/#vip_2","text":"\u5982\u5bf9\u9884\u7559\u7684 VIP \u7684 IP \u5730\u5740\u6709\u9700\u6c42\u53ef\u4f7f\u7528\u4e0b\u9762\u7684 yaml \u8fdb\u884c\u56fa\u5b9a\u5206\u914d\uff1a apiVersion : kubeovn.io/v1 kind : Vip metadata : name : static-vip01 spec : subnet : ovn-default v4ip : \"10.16.0.121\" subnet : \u5c06\u4ece\u8be5 Subnet \u4e2d\u9884\u7559 IP\u3002 v4ip : \u56fa\u5b9a\u5206\u914d\u7684 IP \u5730\u5740\uff0c\u8be5\u5730\u5740\u9700\u5728 subnet \u7684 CIDR \u8303\u56f4\u5185\u3002 \u521b\u5efa\u6210\u529f\u540e\u67e5\u8be2\u8be5 VIP\uff1a # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY static-vip01 10 .16.0.121 00 :00:00:F0:DB:26 ovn-default true \u53ef\u89c1\u8be5 VIP \u88ab\u5206\u914d\u4e86\u6240\u9884\u671f\u7684 IP \u5730\u5740\u3002","title":"\u521b\u5efa\u56fa\u5b9a\u5730\u5740 VIP"},{"location":"advance/vip/#pod-vip-ip","text":"\u8be5\u529f\u80fd\u76ee\u524d\u53ea\u5728 master \u5206\u652f\u652f\u6301\u3002 \u53ef\u4ee5\u4f7f\u7528 annotation \u5c06\u67d0\u4e2a VIP \u5206\u914d\u7ed9\u4e00\u4e2a Pod\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip annotations : ovn.kubernetes.io/vip : vip-dynamic-01 # \u6307\u5b9a vip namespace : default spec : containers : - name : static-ip image : docker.io/library/nginx:alpine","title":"Pod \u4f7f\u7528 VIP \u6765\u56fa\u5b9a IP"},{"location":"advance/vip/#statefulset-kubevirt-vm-vip","text":"\u9488\u5bf9 StatefulSet \u548c VM \u7684\u7279\u6b8a\u6027\uff0c\u5728\u4ed6\u4eec\u7684 Pod \u9500\u6bc1\u518d\u62c9\u8d77\u8d77\u540e\u4f1a\u91cd\u65b0\u4f7f\u7528\u4e4b\u524d\u8bbe\u7f6e\u7684 VIP\u3002 VM \u4fdd\u7559 VIP \u9700\u8981\u786e\u4fdd kube-ovn-controller \u7684 keep-vm-ip \u53c2\u6570\u4e3a true \u3002\u8bf7\u53c2\u8003 Kubevirt VM \u56fa\u5b9a\u5730\u5740\u5f00\u542f\u8bbe\u7f6e \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"StatefulSet \u548c Kubevirt VM \u4fdd\u7559 VIP"},{"location":"advance/vpc-dns/","text":"\u81ea\u5b9a\u4e49 VPC DNS \u00b6 \u7531\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u548c \u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0c\u81ea\u5b9a VPC \u5185\u65e0\u6cd5\u8bbf\u95ee\u5230\u90e8\u7f72\u5728\u9ed8\u8ba4 VPC \u5185\u7684 coredns\u3002 \u5982\u679c\u7528\u6237\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 Kubernetes \u63d0\u4f9b\u7684\u96c6\u7fa4\u5185\u57df\u540d\u89e3\u6790\u80fd\u529b\uff0c\u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\uff0c\u5229\u7528 vpc-dns CRD \u6765\u5b9e\u73b0\u3002 \u8be5 CRD \u6700\u7ec8\u4f1a\u90e8\u7f72\u4e00\u4e2a coredns\uff0c\u8be5 Pod \u6709\u4e24\u4e2a\u7f51\u5361\uff0c\u4e00\u4e2a\u7f51\u5361\u5728\u7528\u6237\u81ea\u5b9a\u4e49 VPC\uff0c\u53e6\u4e00\u4e2a\u7f51\u5361\u5728\u9ed8\u8ba4 VPC \u4ece\u800c\u5b9e\u73b0\u7f51\u7edc\u4e92\u901a\uff0c\u540c\u65f6\u901a\u8fc7 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u63d0\u4f9b\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u4e00\u4e2a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u3002 \u90e8\u7f72 vpc-dns \u6240\u4f9d\u8d56\u7684\u8d44\u6e90 \u00b6 apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u9664\u4e86\u4ee5\u4e0a\u8d44\u6e90\uff0c\u8be5\u529f\u80fd\u8fd8\u4f9d\u8d56 nat-gw-pod \u955c\u50cf\u8fdb\u884c\u8def\u7531\u914d\u7f6e\u3002 \u914d\u7f6e\u9644\u52a0\u7f51\u5361 \u00b6 apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }' \u4fee\u6539 ovn-default \u5b50\u7f51\u7684 provider \u00b6 \u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn # \u53ea\u9700\u4fee\u6539\u8be5\u5b57\u6bb5 vpc : ovn-cluster \u914d\u7f6e vpc-dns \u7684 Configmap \u00b6 \u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : \"true\" nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\u662f\u5426\u542f\u7528\u529f\u80fd\uff0c\u9ed8\u8ba4 true \u3002 coredns-image \uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 coredns-template \uff1acoredns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\u83b7\u53d6\u5f53\u524d\u7248\u672c ovn \u76ee\u5f55\u4e0b coredns-template.yaml \u9ed8\u8ba4\u4e3a https://raw.githubusercontent.com/kubeovn/kube-ovn/\u5f53\u524d\u7248\u672c/yamls/coredns-template.yaml \u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u5730\u5740\u3002 k8s-service-port \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u7aef\u53e3\u3002 \u90e8\u7f72 vpc-dns \u00b6 \u914d\u7f6e vpc-dns yaml\uff1a kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u90e8\u7f72\u8d44\u6e90\u7684\u4fe1\u606f\uff1a # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72\u3002 \u9650\u5236\uff1a\u4e00\u4e2a VPC \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a VPC \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a VPC \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002 \u9a8c\u8bc1\u90e8\u7f72\u7ed3\u679c \u00b6 \u67e5\u770b vpc-dns Pod \u72b6\u6001\uff0c\u4f7f\u7528 label app=vpc-dns \uff0c\u53ef\u4ee5\u67e5\u770b\u6240\u6709 vpc-dns pod \u72b6\u6001\uff1a # kubectl -n kube-system get pods -l app=vpc-dns NAME READY STATUS RESTARTS AGE vpc-dns-test-cjh1-7b878d96b4-g5979 1 /1 Running 0 28s vpc-dns-test-cjh1-7b878d96b4-ltmf9 1 /1 Running 0 28s \u67e5\u770b slr \u72b6\u6001\u4fe1\u606f\uff1a # kubectl -n kube-system get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh1 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh1 113s \u8fdb\u5165\u8be5 VPC \u4e0b\u7684 Pod\uff0c\u6d4b\u8bd5 dns \u89e3\u6790: nslookup kubernetes.default.svc.cluster.local 10 .96.0.3 \u8be5 VPC \u4e0b\u7684 switch lb rule \u6240\u5728\u7684\u5b50\u7f51\u4ee5\u53ca\u540c\u4e00 VPC \u4e0b\u7684\u5176\u4ed6\u5b50\u7f51\u4e0b\u7684 pod \u90fd\u53ef\u4ee5\u89e3\u6790\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u81ea\u5b9a\u4e49 VPC DNS"},{"location":"advance/vpc-dns/#vpc-dns","text":"\u7531\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u548c \u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0c\u81ea\u5b9a VPC \u5185\u65e0\u6cd5\u8bbf\u95ee\u5230\u90e8\u7f72\u5728\u9ed8\u8ba4 VPC \u5185\u7684 coredns\u3002 \u5982\u679c\u7528\u6237\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 Kubernetes \u63d0\u4f9b\u7684\u96c6\u7fa4\u5185\u57df\u540d\u89e3\u6790\u80fd\u529b\uff0c\u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\uff0c\u5229\u7528 vpc-dns CRD \u6765\u5b9e\u73b0\u3002 \u8be5 CRD \u6700\u7ec8\u4f1a\u90e8\u7f72\u4e00\u4e2a coredns\uff0c\u8be5 Pod \u6709\u4e24\u4e2a\u7f51\u5361\uff0c\u4e00\u4e2a\u7f51\u5361\u5728\u7528\u6237\u81ea\u5b9a\u4e49 VPC\uff0c\u53e6\u4e00\u4e2a\u7f51\u5361\u5728\u9ed8\u8ba4 VPC \u4ece\u800c\u5b9e\u73b0\u7f51\u7edc\u4e92\u901a\uff0c\u540c\u65f6\u901a\u8fc7 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u63d0\u4f9b\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u4e00\u4e2a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u3002","title":"\u81ea\u5b9a\u4e49 VPC DNS"},{"location":"advance/vpc-dns/#vpc-dns_1","text":"apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u9664\u4e86\u4ee5\u4e0a\u8d44\u6e90\uff0c\u8be5\u529f\u80fd\u8fd8\u4f9d\u8d56 nat-gw-pod \u955c\u50cf\u8fdb\u884c\u8def\u7531\u914d\u7f6e\u3002","title":"\u90e8\u7f72 vpc-dns \u6240\u4f9d\u8d56\u7684\u8d44\u6e90"},{"location":"advance/vpc-dns/#_1","text":"apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }'","title":"\u914d\u7f6e\u9644\u52a0\u7f51\u5361"},{"location":"advance/vpc-dns/#ovn-default-provider","text":"\u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn # \u53ea\u9700\u4fee\u6539\u8be5\u5b57\u6bb5 vpc : ovn-cluster","title":"\u4fee\u6539 ovn-default \u5b50\u7f51\u7684 provider"},{"location":"advance/vpc-dns/#vpc-dns-configmap","text":"\u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : \"true\" nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\u662f\u5426\u542f\u7528\u529f\u80fd\uff0c\u9ed8\u8ba4 true \u3002 coredns-image \uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 coredns-template \uff1acoredns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\u83b7\u53d6\u5f53\u524d\u7248\u672c ovn \u76ee\u5f55\u4e0b coredns-template.yaml \u9ed8\u8ba4\u4e3a https://raw.githubusercontent.com/kubeovn/kube-ovn/\u5f53\u524d\u7248\u672c/yamls/coredns-template.yaml \u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u5730\u5740\u3002 k8s-service-port \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u7aef\u53e3\u3002","title":"\u914d\u7f6e vpc-dns \u7684 Configmap"},{"location":"advance/vpc-dns/#vpc-dns_2","text":"\u914d\u7f6e vpc-dns yaml\uff1a kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u90e8\u7f72\u8d44\u6e90\u7684\u4fe1\u606f\uff1a # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72\u3002 \u9650\u5236\uff1a\u4e00\u4e2a VPC \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a VPC \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a VPC \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002","title":"\u90e8\u7f72 vpc-dns"},{"location":"advance/vpc-dns/#_2","text":"\u67e5\u770b vpc-dns Pod \u72b6\u6001\uff0c\u4f7f\u7528 label app=vpc-dns \uff0c\u53ef\u4ee5\u67e5\u770b\u6240\u6709 vpc-dns pod \u72b6\u6001\uff1a # kubectl -n kube-system get pods -l app=vpc-dns NAME READY STATUS RESTARTS AGE vpc-dns-test-cjh1-7b878d96b4-g5979 1 /1 Running 0 28s vpc-dns-test-cjh1-7b878d96b4-ltmf9 1 /1 Running 0 28s \u67e5\u770b slr \u72b6\u6001\u4fe1\u606f\uff1a # kubectl -n kube-system get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh1 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh1 113s \u8fdb\u5165\u8be5 VPC \u4e0b\u7684 Pod\uff0c\u6d4b\u8bd5 dns \u89e3\u6790: nslookup kubernetes.default.svc.cluster.local 10 .96.0.3 \u8be5 VPC \u4e0b\u7684 switch lb rule \u6240\u5728\u7684\u5b50\u7f51\u4ee5\u53ca\u540c\u4e00 VPC \u4e0b\u7684\u5176\u4ed6\u5b50\u7f51\u4e0b\u7684 pod \u90fd\u53ef\u4ee5\u89e3\u6790\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9a8c\u8bc1\u90e8\u7f72\u7ed3\u679c"},{"location":"advance/vpc-internal-lb/","text":"\u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u00b6 Kubernetes \u63d0\u4f9b\u7684 Service \u53ef\u4ee5\u7528\u4f5c\u96c6\u7fa4\u5185\u7684\u8d1f\u8f7d\u5747\u8861\uff0c \u4f46\u662f\u5728\u81ea\u5b9a\u4e49 VPC \u6a21\u5f0f\u4e0b\uff0c \u4f7f\u7528 Service \u4f5c\u4e3a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u5b58\u5728\u5982\u4e0b\u51e0\u4e2a\u95ee\u9898\uff1a Service IP \u8303\u56f4\u4e3a\u96c6\u7fa4\u8d44\u6e90\uff0c\u6240\u6709\u81ea\u5b9a\u4e49 VPC \u5171\u4eab\uff0c\u65e0\u6cd5\u91cd\u53e0\u3002 \u7528\u6237\u65e0\u6cd5\u6309\u7167\u81ea\u5df1\u610f\u613f\u8bbe\u7f6e\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 \u4e3a\u4e86\u89e3\u51b3\u4e0a\u8ff0\u95ee\u9898\uff0cKube-OVN \u5728 1.11 \u5f15\u5165 SwitchLBRule CRD\uff0c\u7528\u6237\u53ef\u4ee5\u8bbe\u7f6e\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 SwitchLBRule \u652f\u6301\u4ee5\u4e0b\u4e24\u79cd\u65b9\u5f0f\u8bbe\u7f6e\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 Selector \u81ea\u52a8\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219 \u00b6 \u901a\u8fc7 selector \u53ef\u4ee5\u901a\u8fc7 label \u81ea\u52a8\u5173\u8054 pod \u914d\u7f6e\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 SwitchLBRule \u6837\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP selector , sessionAffinity \u548c port \u4f7f\u7528\u65b9\u5f0f\u540c Kubernetes Service\u3002 vip \uff1a\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 namespace \uff1a selector \u6240\u9009\u62e9 Pod \u6240\u5728\u547d\u540d\u7a7a\u95f4\u3002 Kube-OVN \u4f1a\u6839\u636e SwitchLBRule \u5b9a\u4e49\u9009\u62e9\u7684 Pod \u5f97\u51fa Pod \u6240\u5728 VPC \u5e76\u8bbe\u7f6e\u5bf9\u5e94\u7684 L2 LB\u3002 Endpoints \u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u89c4\u5219 \u00b6 \u901a\u8fc7 endpoints \u53ef\u4ee5\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\uff0c\u7528\u4ee5\u652f\u6301\u65e0\u6cd5\u901a\u8fc7 selector \u81ea\u52a8\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u7684\u573a\u666f\uff0c\u6bd4\u5982\u8d1f\u8f7d\u5747\u8861\u540e\u7aef\u662f kubevirt \u521b\u5efa\u7684 vm \u3002 SwitchLBRule \u6837\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default endpoints : - 192.168.0.101 - 192.168.0.102 - 192.168.0.103 ports : - name : dns port : 8888 targetPort : 80 protocol : TCP sessionAffinity \u548c port \u4f7f\u7528\u65b9\u5f0f\u540c Kubernetes Service\u3002 vip \uff1a\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 namespace \uff1a selector \u6240\u9009\u62e9 Pod \u6240\u5728\u547d\u540d\u7a7a\u95f4\u3002 endpoints \uff1a\u8d1f\u8f7d\u5747\u8861\u540e\u7aef IP \u5217\u8868\u3002 \u6ce8\uff1a \u5982\u679c\u540c\u65f6\u914d\u7f6e\u4e86 selector \u548c endpoints ,\u4f1a\u81ea\u52a8\u5ffd\u7565 selector \u914d\u7f6e\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861"},{"location":"advance/vpc-internal-lb/#vpc","text":"Kubernetes \u63d0\u4f9b\u7684 Service \u53ef\u4ee5\u7528\u4f5c\u96c6\u7fa4\u5185\u7684\u8d1f\u8f7d\u5747\u8861\uff0c \u4f46\u662f\u5728\u81ea\u5b9a\u4e49 VPC \u6a21\u5f0f\u4e0b\uff0c \u4f7f\u7528 Service \u4f5c\u4e3a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u5b58\u5728\u5982\u4e0b\u51e0\u4e2a\u95ee\u9898\uff1a Service IP \u8303\u56f4\u4e3a\u96c6\u7fa4\u8d44\u6e90\uff0c\u6240\u6709\u81ea\u5b9a\u4e49 VPC \u5171\u4eab\uff0c\u65e0\u6cd5\u91cd\u53e0\u3002 \u7528\u6237\u65e0\u6cd5\u6309\u7167\u81ea\u5df1\u610f\u613f\u8bbe\u7f6e\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 \u4e3a\u4e86\u89e3\u51b3\u4e0a\u8ff0\u95ee\u9898\uff0cKube-OVN \u5728 1.11 \u5f15\u5165 SwitchLBRule CRD\uff0c\u7528\u6237\u53ef\u4ee5\u8bbe\u7f6e\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 SwitchLBRule \u652f\u6301\u4ee5\u4e0b\u4e24\u79cd\u65b9\u5f0f\u8bbe\u7f6e\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002","title":"\u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861"},{"location":"advance/vpc-internal-lb/#selector","text":"\u901a\u8fc7 selector \u53ef\u4ee5\u901a\u8fc7 label \u81ea\u52a8\u5173\u8054 pod \u914d\u7f6e\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u3002 SwitchLBRule \u6837\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP selector , sessionAffinity \u548c port \u4f7f\u7528\u65b9\u5f0f\u540c Kubernetes Service\u3002 vip \uff1a\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 namespace \uff1a selector \u6240\u9009\u62e9 Pod \u6240\u5728\u547d\u540d\u7a7a\u95f4\u3002 Kube-OVN \u4f1a\u6839\u636e SwitchLBRule \u5b9a\u4e49\u9009\u62e9\u7684 Pod \u5f97\u51fa Pod \u6240\u5728 VPC \u5e76\u8bbe\u7f6e\u5bf9\u5e94\u7684 L2 LB\u3002","title":"Selector \u81ea\u52a8\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219"},{"location":"advance/vpc-internal-lb/#endpoints","text":"\u901a\u8fc7 endpoints \u53ef\u4ee5\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\uff0c\u7528\u4ee5\u652f\u6301\u65e0\u6cd5\u901a\u8fc7 selector \u81ea\u52a8\u751f\u6210\u8d1f\u8f7d\u5747\u8861\u89c4\u5219\u7684\u573a\u666f\uff0c\u6bd4\u5982\u8d1f\u8f7d\u5747\u8861\u540e\u7aef\u662f kubevirt \u521b\u5efa\u7684 vm \u3002 SwitchLBRule \u6837\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default endpoints : - 192.168.0.101 - 192.168.0.102 - 192.168.0.103 ports : - name : dns port : 8888 targetPort : 80 protocol : TCP sessionAffinity \u548c port \u4f7f\u7528\u65b9\u5f0f\u540c Kubernetes Service\u3002 vip \uff1a\u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u7684 IP \u5730\u5740\u3002 namespace \uff1a selector \u6240\u9009\u62e9 Pod \u6240\u5728\u547d\u540d\u7a7a\u95f4\u3002 endpoints \uff1a\u8d1f\u8f7d\u5747\u8861\u540e\u7aef IP \u5217\u8868\u3002 \u6ce8\uff1a \u5982\u679c\u540c\u65f6\u914d\u7f6e\u4e86 selector \u548c endpoints ,\u4f1a\u81ea\u52a8\u5ffd\u7565 selector \u914d\u7f6e\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Endpoints \u81ea\u5b9a\u4e49\u8d1f\u8f7d\u5747\u8861\u89c4\u5219"},{"location":"advance/vpc-peering/","text":"VPC \u4e92\u8054 \u00b6 VPC \u4e92\u8054\u63d0\u4f9b\u4e86\u4e00\u79cd\u5c06\u4e24\u4e2a VPC \u7f51\u7edc\u901a\u8fc7\u903b\u8f91\u8def\u7531\u6253\u901a\u7684\u673a\u5236\uff0c\u4ece\u800c\u4f7f\u4e24\u4e2a VPC \u5185\u7684\u5de5\u4f5c\u8d1f\u8f7d\u53ef\u4ee5\u50cf\u5728\u540c\u4e00\u4e2a\u79c1\u6709\u7f51\u7edc\u4e00\u6837\uff0c \u901a\u8fc7\u79c1\u6709\u5730\u5740\u76f8\u4e92\u8bbf\u95ee\uff0c\u65e0\u9700\u901a\u8fc7\u5916\u90e8\u7f51\u5173\u8fdb\u884c NAT \u8f6c\u53d1\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u8be5\u529f\u80fd\u53ea\u9002\u7528\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC\u3002 \u4e3a\u4e86\u907f\u514d\u8def\u7531\u91cd\u53e0\u4e24\u4e2a VPC \u5185\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 \u76ee\u524d\u53ea\u652f\u6301\u4e24\u4e2a VPC \u7684\u4e92\u8054\uff0c\u66f4\u591a\u7ec4 VPC \u4e4b\u95f4\u7684\u4e92\u8054\u6682\u4e0d\u652f\u6301\u3002 \u4f7f\u7528\u65b9\u5f0f \u00b6 \u9996\u5148\u521b\u5efa\u4e24\u4e2a\u4e0d\u4e92\u8054\u7684 VPC\uff0c\u6bcf\u4e2a VPC \u4e0b\u5404\u6709\u4e00\u4e2a Subnet\uff0cSubnet \u7684 CIDR \u4e92\u4e0d\u91cd\u53e0\u3002 kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : vpc-1 cidrBlock : 10.0.0.0/16 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : vpc-2 cidrBlock : 172.31.0.0/16 \u5728\u6bcf\u4e2a VPC \u5185\u5206\u522b\u589e\u52a0 vpcPeerings \u548c\u5bf9\u5e94\u7684\u9759\u6001\u8def\u7531\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : vpcPeerings : - remoteVpc : vpc-2 localConnectIP : 169.254.0.1/30 staticRoutes : - cidr : 172.31.0.0/16 nextHopIP : 169.254.0.2 policy : policyDst --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : vpcPeerings : - remoteVpc : vpc-1 localConnectIP : 169.254.0.2/30 staticRoutes : - cidr : 10.0.0.0/16 nextHopIP : 169.254.0.1 policy : policyDst remoteVpc : \u4e92\u8054\u7684\u53e6\u4e00\u4e2a VPC \u7684\u540d\u5b57\u3002 localConnectIP : \u4f5c\u4e3a\u4e92\u8054\u7aef\u70b9\u7684 IP \u5730\u5740\u548c CIDR\uff0c\u6ce8\u610f\u4e24\u7aef IP \u5e94\u5c5e\u4e8e\u540c\u4e00 CIDR\uff0c\u4e14\u4e0d\u80fd\u548c\u5df2\u6709\u5b50\u7f51\u51b2\u7a81\u3002 cidr \uff1a\u53e6\u4e00\u7aef Subnet \u7684 CIDR\u3002 nextHopIP \uff1a\u4e92\u8054 VPC \u53e6\u4e00\u7aef\u7684 localConnectIP \u3002 \u5206\u522b\u5728\u4e24\u4e2a Subnet \u4e0b\u521b\u5efa Pod apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 name : vpc-1-pod spec : containers : - name : vpc-1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 name : vpc-2-pod spec : containers : - name : vpc-2-pod image : docker.io/library/nginx:alpine \u6d4b\u8bd5\u7f51\u7edc\u8fde\u901a\u6027 # kubectl exec -it vpc-1-pod -- ping $(kubectl get pod vpc-2-pod -o jsonpath='{.status.podIP}') PING 172 .31.0.2 ( 172 .31.0.2 ) : 56 data bytes 64 bytes from 172 .31.0.2: seq = 0 ttl = 62 time = 0 .655 ms 64 bytes from 172 .31.0.2: seq = 1 ttl = 62 time = 0 .086 ms 64 bytes from 172 .31.0.2: seq = 2 ttl = 62 time = 0 .098 ms ^C --- 172 .31.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .086/0.279/0.655 ms # kubectl exec -it vpc-2-pod -- ping $(kubectl get pod vpc-1-pod -o jsonpath='{.status.podIP}') PING 10 .0.0.2 ( 10 .0.0.2 ) : 56 data bytes 64 bytes from 10 .0.0.2: seq = 0 ttl = 62 time = 0 .594 ms 64 bytes from 10 .0.0.2: seq = 1 ttl = 62 time = 0 .093 ms 64 bytes from 10 .0.0.2: seq = 2 ttl = 62 time = 0 .088 ms ^C --- 10 .0.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .088/0.258/0.594 ms \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC \u4e92\u8054"},{"location":"advance/vpc-peering/#vpc","text":"VPC \u4e92\u8054\u63d0\u4f9b\u4e86\u4e00\u79cd\u5c06\u4e24\u4e2a VPC \u7f51\u7edc\u901a\u8fc7\u903b\u8f91\u8def\u7531\u6253\u901a\u7684\u673a\u5236\uff0c\u4ece\u800c\u4f7f\u4e24\u4e2a VPC \u5185\u7684\u5de5\u4f5c\u8d1f\u8f7d\u53ef\u4ee5\u50cf\u5728\u540c\u4e00\u4e2a\u79c1\u6709\u7f51\u7edc\u4e00\u6837\uff0c \u901a\u8fc7\u79c1\u6709\u5730\u5740\u76f8\u4e92\u8bbf\u95ee\uff0c\u65e0\u9700\u901a\u8fc7\u5916\u90e8\u7f51\u5173\u8fdb\u884c NAT \u8f6c\u53d1\u3002","title":"VPC \u4e92\u8054"},{"location":"advance/vpc-peering/#_1","text":"\u8be5\u529f\u80fd\u53ea\u9002\u7528\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC\u3002 \u4e3a\u4e86\u907f\u514d\u8def\u7531\u91cd\u53e0\u4e24\u4e2a VPC \u5185\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 \u76ee\u524d\u53ea\u652f\u6301\u4e24\u4e2a VPC \u7684\u4e92\u8054\uff0c\u66f4\u591a\u7ec4 VPC \u4e4b\u95f4\u7684\u4e92\u8054\u6682\u4e0d\u652f\u6301\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/vpc-peering/#_2","text":"\u9996\u5148\u521b\u5efa\u4e24\u4e2a\u4e0d\u4e92\u8054\u7684 VPC\uff0c\u6bcf\u4e2a VPC \u4e0b\u5404\u6709\u4e00\u4e2a Subnet\uff0cSubnet \u7684 CIDR \u4e92\u4e0d\u91cd\u53e0\u3002 kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : vpc-1 cidrBlock : 10.0.0.0/16 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : vpc-2 cidrBlock : 172.31.0.0/16 \u5728\u6bcf\u4e2a VPC \u5185\u5206\u522b\u589e\u52a0 vpcPeerings \u548c\u5bf9\u5e94\u7684\u9759\u6001\u8def\u7531\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : vpcPeerings : - remoteVpc : vpc-2 localConnectIP : 169.254.0.1/30 staticRoutes : - cidr : 172.31.0.0/16 nextHopIP : 169.254.0.2 policy : policyDst --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : vpcPeerings : - remoteVpc : vpc-1 localConnectIP : 169.254.0.2/30 staticRoutes : - cidr : 10.0.0.0/16 nextHopIP : 169.254.0.1 policy : policyDst remoteVpc : \u4e92\u8054\u7684\u53e6\u4e00\u4e2a VPC \u7684\u540d\u5b57\u3002 localConnectIP : \u4f5c\u4e3a\u4e92\u8054\u7aef\u70b9\u7684 IP \u5730\u5740\u548c CIDR\uff0c\u6ce8\u610f\u4e24\u7aef IP \u5e94\u5c5e\u4e8e\u540c\u4e00 CIDR\uff0c\u4e14\u4e0d\u80fd\u548c\u5df2\u6709\u5b50\u7f51\u51b2\u7a81\u3002 cidr \uff1a\u53e6\u4e00\u7aef Subnet \u7684 CIDR\u3002 nextHopIP \uff1a\u4e92\u8054 VPC \u53e6\u4e00\u7aef\u7684 localConnectIP \u3002 \u5206\u522b\u5728\u4e24\u4e2a Subnet \u4e0b\u521b\u5efa Pod apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 name : vpc-1-pod spec : containers : - name : vpc-1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 name : vpc-2-pod spec : containers : - name : vpc-2-pod image : docker.io/library/nginx:alpine \u6d4b\u8bd5\u7f51\u7edc\u8fde\u901a\u6027 # kubectl exec -it vpc-1-pod -- ping $(kubectl get pod vpc-2-pod -o jsonpath='{.status.podIP}') PING 172 .31.0.2 ( 172 .31.0.2 ) : 56 data bytes 64 bytes from 172 .31.0.2: seq = 0 ttl = 62 time = 0 .655 ms 64 bytes from 172 .31.0.2: seq = 1 ttl = 62 time = 0 .086 ms 64 bytes from 172 .31.0.2: seq = 2 ttl = 62 time = 0 .098 ms ^C --- 172 .31.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .086/0.279/0.655 ms # kubectl exec -it vpc-2-pod -- ping $(kubectl get pod vpc-1-pod -o jsonpath='{.status.podIP}') PING 10 .0.0.2 ( 10 .0.0.2 ) : 56 data bytes 64 bytes from 10 .0.0.2: seq = 0 ttl = 62 time = 0 .594 ms 64 bytes from 10 .0.0.2: seq = 1 ttl = 62 time = 0 .093 ms 64 bytes from 10 .0.0.2: seq = 2 ttl = 62 time = 0 .088 ms ^C --- 10 .0.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .088/0.258/0.594 ms \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528\u65b9\u5f0f"},{"location":"advance/windows/","text":"Windows \u652f\u6301 \u00b6 Kube-OVN \u652f\u6301\u5305\u542b Windows \u7cfb\u7edf\u8282\u70b9\u7684 Kubernetes \u96c6\u7fa4\u7f51\u7edc\uff0c\u53ef\u4ee5\u5c06 Windows \u5bb9\u5668\u7684\u7f51\u7edc\u7edf\u4e00\u63a5\u5165\u8fdb\u884c\u7ba1\u7406\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u53c2\u8003 Adding Windows nodes \u589e\u52a0 Windows \u8282\u70b9\u3002 Windows \u8282\u70b9\u5fc5\u987b\u5b89\u88c5 KB4489899 \u8865\u4e01\u4ee5\u4f7f Overlay/VXLAN \u7f51\u7edc\u6b63\u5e38\u5de5\u4f5c\uff0c\u5efa\u8bae\u66f4\u65b0\u7cfb\u7edf\u81f3\u6700\u65b0\u7248\u672c\u3002 Windows \u8282\u70b9\u5fc5\u987b\u5b89\u88c5 Hyper-V \u53ca\u7ba1\u7406\u5de5\u5177\u3002 \u7531\u4e8e Windows \u9650\u5236\u96a7\u9053\u5c01\u88c5\u53ea\u80fd\u4f7f\u7528 Vxlan \u6a21\u5f0f\u3002 \u6682\u4e0d\u652f\u6301 SSL\uff0cIPv6\uff0c\u53cc\u6808\uff0cQoS \u529f\u80fd\u3002 \u6682\u4e0d\u652f\u6301\u52a8\u6001\u5b50\u7f51\uff0c\u52a8\u6001\u96a7\u9053\u63a5\u53e3\u529f\u80fd\uff0c\u9700\u5728\u5b89\u88c5 Windows \u8282\u70b9\u524d\u5b8c\u6210\u5b50\u7f51\u521b\u5efa\uff0c\u5e76\u56fa\u5b9a\u7f51\u7edc\u63a5\u53e3\u3002 \u4e0d\u652f\u6301\u591a\u4e2a ProviderNetwork \uff0c\u4e14\u65e0\u6cd5\u52a8\u6001\u8c03\u6574\u6865\u63a5\u63a5\u53e3\u914d\u7f6e\u3002 \u5b89\u88c5 OVS \u00b6 \u7531\u4e8e\u4e0a\u6e38 OVN \u548c OVS \u5bf9 Windows \u5bb9\u5668\u652f\u6301\u5b58\u5728\u4e00\u4e9b\u95ee\u9898\uff0c\u9700\u8981\u4f7f\u7528 Kube-OVN \u63d0\u4f9b\u7684\u7ecf\u8fc7\u4fee\u6539\u7684\u5b89\u88c5\u5305\u8fdb\u884c\u5b89\u88c5\u3002 \u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u6253\u5f00 Windows \u8282\u70b9\u7684 TESTSIGNING \u542f\u52a8\u9879\uff0c\u6267\u884c\u6210\u529f\u540e\u9700\u8981\u91cd\u542f\u7cfb\u7edf\u751f\u6548\uff1a bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS bcdedit /set TESTSIGNING ON bcdedit /set nointegritychecks ON \u5728 Windows \u8282\u70b9\u4e0b\u8f7d Windows \u5b89\u88c5\u5305 \u5e76\u89e3\u538b\u5b89\u88c5\u3002 \u5b89\u88c5\u5b8c\u6210\u540e\u786e\u8ba4\u670d\u52a1\u6b63\u5e38\u8fd0\u884c\uff1a PS > Get-Service | findstr ovs Running ovsdb-server Open vSwitch DB Service Running ovs-vswitchd Open vSwitch Service \u5b89\u88c5 Kube-OVN \u00b6 \u5728 Windows \u8282\u70b9\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c install.ps1 \u3002 \u8865\u5145\u76f8\u5173\u53c2\u6570\u5e76\u6267\u884c\uff1a . \\i nstall.ps1 -KubeConfig C: \\k\\a dmin.conf -ApiServer https://192.168.140.180:6443 -ServiceCIDR 10 .96.0.0/12 \u9ed8\u8ba4\u60c5\u51b5\u4e0b, Kube-OVN \u4f7f\u7528\u8282\u70b9 IP \u6240\u5728\u7684\u7f51\u5361\u4f5c\u4e3a\u96a7\u9053\u63a5\u53e3\u3002 \u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u5b83\u7f51\u5361\uff0c\u9700\u8981\u5728\u5b89\u88c5\u524d\u7ed9\u8282\u70b9\u6dfb\u52a0\u6307\u5b9a\u7684 Annotation\uff0c\u5982 ovn.kubernetes.io/tunnel_interface=Ethernet1 \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Windows \u652f\u6301"},{"location":"advance/windows/#windows","text":"Kube-OVN \u652f\u6301\u5305\u542b Windows \u7cfb\u7edf\u8282\u70b9\u7684 Kubernetes \u96c6\u7fa4\u7f51\u7edc\uff0c\u53ef\u4ee5\u5c06 Windows \u5bb9\u5668\u7684\u7f51\u7edc\u7edf\u4e00\u63a5\u5165\u8fdb\u884c\u7ba1\u7406\u3002","title":"Windows \u652f\u6301"},{"location":"advance/windows/#_1","text":"\u53c2\u8003 Adding Windows nodes \u589e\u52a0 Windows \u8282\u70b9\u3002 Windows \u8282\u70b9\u5fc5\u987b\u5b89\u88c5 KB4489899 \u8865\u4e01\u4ee5\u4f7f Overlay/VXLAN \u7f51\u7edc\u6b63\u5e38\u5de5\u4f5c\uff0c\u5efa\u8bae\u66f4\u65b0\u7cfb\u7edf\u81f3\u6700\u65b0\u7248\u672c\u3002 Windows \u8282\u70b9\u5fc5\u987b\u5b89\u88c5 Hyper-V \u53ca\u7ba1\u7406\u5de5\u5177\u3002 \u7531\u4e8e Windows \u9650\u5236\u96a7\u9053\u5c01\u88c5\u53ea\u80fd\u4f7f\u7528 Vxlan \u6a21\u5f0f\u3002 \u6682\u4e0d\u652f\u6301 SSL\uff0cIPv6\uff0c\u53cc\u6808\uff0cQoS \u529f\u80fd\u3002 \u6682\u4e0d\u652f\u6301\u52a8\u6001\u5b50\u7f51\uff0c\u52a8\u6001\u96a7\u9053\u63a5\u53e3\u529f\u80fd\uff0c\u9700\u5728\u5b89\u88c5 Windows \u8282\u70b9\u524d\u5b8c\u6210\u5b50\u7f51\u521b\u5efa\uff0c\u5e76\u56fa\u5b9a\u7f51\u7edc\u63a5\u53e3\u3002 \u4e0d\u652f\u6301\u591a\u4e2a ProviderNetwork \uff0c\u4e14\u65e0\u6cd5\u52a8\u6001\u8c03\u6574\u6865\u63a5\u63a5\u53e3\u914d\u7f6e\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/windows/#ovs","text":"\u7531\u4e8e\u4e0a\u6e38 OVN \u548c OVS \u5bf9 Windows \u5bb9\u5668\u652f\u6301\u5b58\u5728\u4e00\u4e9b\u95ee\u9898\uff0c\u9700\u8981\u4f7f\u7528 Kube-OVN \u63d0\u4f9b\u7684\u7ecf\u8fc7\u4fee\u6539\u7684\u5b89\u88c5\u5305\u8fdb\u884c\u5b89\u88c5\u3002 \u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u6253\u5f00 Windows \u8282\u70b9\u7684 TESTSIGNING \u542f\u52a8\u9879\uff0c\u6267\u884c\u6210\u529f\u540e\u9700\u8981\u91cd\u542f\u7cfb\u7edf\u751f\u6548\uff1a bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS bcdedit /set TESTSIGNING ON bcdedit /set nointegritychecks ON \u5728 Windows \u8282\u70b9\u4e0b\u8f7d Windows \u5b89\u88c5\u5305 \u5e76\u89e3\u538b\u5b89\u88c5\u3002 \u5b89\u88c5\u5b8c\u6210\u540e\u786e\u8ba4\u670d\u52a1\u6b63\u5e38\u8fd0\u884c\uff1a PS > Get-Service | findstr ovs Running ovsdb-server Open vSwitch DB Service Running ovs-vswitchd Open vSwitch Service","title":"\u5b89\u88c5 OVS"},{"location":"advance/windows/#kube-ovn","text":"\u5728 Windows \u8282\u70b9\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c install.ps1 \u3002 \u8865\u5145\u76f8\u5173\u53c2\u6570\u5e76\u6267\u884c\uff1a . \\i nstall.ps1 -KubeConfig C: \\k\\a dmin.conf -ApiServer https://192.168.140.180:6443 -ServiceCIDR 10 .96.0.0/12 \u9ed8\u8ba4\u60c5\u51b5\u4e0b, Kube-OVN \u4f7f\u7528\u8282\u70b9 IP \u6240\u5728\u7684\u7f51\u5361\u4f5c\u4e3a\u96a7\u9053\u63a5\u53e3\u3002 \u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u5b83\u7f51\u5361\uff0c\u9700\u8981\u5728\u5b89\u88c5\u524d\u7ed9\u8282\u70b9\u6dfb\u52a0\u6307\u5b9a\u7684 Annotation\uff0c\u5982 ovn.kubernetes.io/tunnel_interface=Ethernet1 \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b89\u88c5 Kube-OVN"},{"location":"advance/with-bgp/","text":"BGP \u652f\u6301 \u00b6 Kube-OVN \u652f\u6301\u5c06 Pod \u6216 Subnet \u7684 IP \u5730\u5740\u901a\u8fc7 BGP \u534f\u8bae\u5411\u5916\u90e8\u8fdb\u884c\u8def\u7531\u5e7f\u64ad\uff0c\u4ece\u800c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u5bf9\u5916\u66b4\u9732\u3002 \u5982\u679c\u9700\u8981\u4f7f\u7528\u8be5\u529f\u80fd\uff0c\u9700\u8981\u5728\u7279\u5b9a\u8282\u70b9\u5b89\u88c5 kube-ovn-speaker \u5e76\u5bf9\u9700\u8981\u5bf9\u5916\u66b4\u9732\u7684 Pod \u6216 Subnet \u589e\u52a0\u5bf9\u5e94\u7684 annotation\u3002 \u5b89\u88c5 kube-ovn-speaker \u00b6 kube-ovn-speaker \u5185\u4f7f\u7528 GoBGP \u5bf9\u5916\u53d1\u5e03\u8def\u7531\u4fe1\u606f\uff0c\u5e76\u5c06\u8bbf\u95ee\u66b4\u9732\u5730\u5740\u7684\u4e0b\u4e00\u8df3\u8def\u7531\u6307\u5411\u81ea\u8eab\u3002 \u7531\u4e8e\u90e8\u7f72 kube-ovn-speaker \u7684\u8282\u70b9\u9700\u8981\u627f\u62c5\u56de\u7a0b\u6d41\u91cf\uff0c\u56e0\u6b64\u9700\u8981\u9009\u62e9\u7279\u5b9a\u8282\u70b9\u8fdb\u884c\u90e8\u7f72\uff1a kubectl label nodes speaker-node-1 ovn.kubernetes.io/bgp = true kubectl label nodes speaker-node-2 ovn.kubernetes.io/bgp = true \u5f53\u5b58\u5728\u591a\u4e2a kube-ovn-speaker \u5b9e\u4f8b\u65f6\uff0c\u6bcf\u4e2a\u5b9e\u4f8b\u90fd\u4f1a\u5bf9\u5916\u53d1\u5e03\u8def\u7531\uff0c\u4e0a\u6e38\u8def\u7531\u5668\u9700\u8981\u652f\u6301\u591a\u8def\u5f84 ECMP\u3002 \u4e0b\u8f7d\u5bf9\u5e94 yaml: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/speaker.yaml \u4fee\u6539 yaml \u5185\u76f8\u5e94\u914d\u7f6e\uff1a --neighbor-address=10.32.32.1 --neighbor-as=65030 --cluster-as=65000 neighbor-address : BGP Peer \u7684\u5730\u5740\uff0c\u901a\u5e38\u4e3a\u8def\u7531\u5668\u7f51\u5173\u5730\u5740\u3002 neighbor-as : BGP Peer \u7684 AS \u53f7\u3002 cluster-as : \u5bb9\u5668\u7f51\u7edc\u7684 AS \u53f7\u3002 \u90e8\u7f72 yaml: kubectl apply -f speaker.yaml \u53d1\u5e03 Pod/Subnet \u8def\u7531 \u00b6 \u5982\u9700\u4f7f\u7528 BGP \u5bf9\u5916\u53d1\u5e03\u8def\u7531\uff0c\u9996\u5148\u9700\u8981\u5c06\u5bf9\u5e94 Subnet \u7684 natOutgoing \u8bbe\u7f6e\u4e3a false \uff0c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u8fdb\u5165\u5e95\u5c42\u7f51\u7edc\u3002 \u589e\u52a0 annotation \u5bf9\u5916\u53d1\u5e03\uff1a kubectl annotate pod sample ovn.kubernetes.io/bgp = true kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp = true \u5220\u9664 annotation \u53d6\u6d88\u53d1\u5e03\uff1a kubectl annotate pod perf-ovn-xzvd4 ovn.kubernetes.io/bgp- kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp- BGP \u9ad8\u7ea7\u9009\u9879 \u00b6 kube-ovn-speaker \u652f\u6301\u66f4\u591a BGP \u53c2\u6570\u8fdb\u884c\u9ad8\u7ea7\u914d\u7f6e\uff0c\u7528\u6237\u53ef\u6839\u636e\u81ea\u5df1\u7f51\u7edc\u73af\u5883\u8fdb\u884c\u8c03\u6574\uff1a announce-cluster-ip : \u662f\u5426\u5bf9\u5916\u53d1\u5e03 Service \u8def\u7531\uff0c\u9ed8\u8ba4\u4e3a false \u3002 auth-password : BGP peer \u7684\u8bbf\u95ee\u5bc6\u7801\u3002 holdtime : BGP \u90bb\u5c45\u95f4\u7684\u5fc3\u8df3\u63a2\u6d4b\u65f6\u95f4\uff0c\u8d85\u8fc7\u6539\u65f6\u95f4\u6ca1\u6709\u6d88\u606f\u7684\u90bb\u5c45\u5c06\u4f1a\u88ab\u79fb\u9664\uff0c\u9ed8\u8ba4\u4e3a 90 \u79d2\u3002 graceful-restart : \u662f\u5426\u542f\u7528 BGP Graceful Restart\u3002 graceful-restart-time : BGP Graceful restart time \u53ef\u53c2\u8003 RFC4724 3\u3002 graceful-restart-deferral-time : BGP Graceful restart deferral time \u53ef\u53c2\u8003 RFC4724 4.1\u3002 passivemode : Speaker \u8fd0\u884c\u5728 passive \u6a21\u5f0f\uff0c\u4e0d\u4e3b\u52a8\u8fde\u63a5 peer\u3002 ebgp-multihop : ebgp ttl \u9ed8\u8ba4\u503c\u4e3a 1\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"BGP \u652f\u6301"},{"location":"advance/with-bgp/#bgp","text":"Kube-OVN \u652f\u6301\u5c06 Pod \u6216 Subnet \u7684 IP \u5730\u5740\u901a\u8fc7 BGP \u534f\u8bae\u5411\u5916\u90e8\u8fdb\u884c\u8def\u7531\u5e7f\u64ad\uff0c\u4ece\u800c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u5bf9\u5916\u66b4\u9732\u3002 \u5982\u679c\u9700\u8981\u4f7f\u7528\u8be5\u529f\u80fd\uff0c\u9700\u8981\u5728\u7279\u5b9a\u8282\u70b9\u5b89\u88c5 kube-ovn-speaker \u5e76\u5bf9\u9700\u8981\u5bf9\u5916\u66b4\u9732\u7684 Pod \u6216 Subnet \u589e\u52a0\u5bf9\u5e94\u7684 annotation\u3002","title":"BGP \u652f\u6301"},{"location":"advance/with-bgp/#kube-ovn-speaker","text":"kube-ovn-speaker \u5185\u4f7f\u7528 GoBGP \u5bf9\u5916\u53d1\u5e03\u8def\u7531\u4fe1\u606f\uff0c\u5e76\u5c06\u8bbf\u95ee\u66b4\u9732\u5730\u5740\u7684\u4e0b\u4e00\u8df3\u8def\u7531\u6307\u5411\u81ea\u8eab\u3002 \u7531\u4e8e\u90e8\u7f72 kube-ovn-speaker \u7684\u8282\u70b9\u9700\u8981\u627f\u62c5\u56de\u7a0b\u6d41\u91cf\uff0c\u56e0\u6b64\u9700\u8981\u9009\u62e9\u7279\u5b9a\u8282\u70b9\u8fdb\u884c\u90e8\u7f72\uff1a kubectl label nodes speaker-node-1 ovn.kubernetes.io/bgp = true kubectl label nodes speaker-node-2 ovn.kubernetes.io/bgp = true \u5f53\u5b58\u5728\u591a\u4e2a kube-ovn-speaker \u5b9e\u4f8b\u65f6\uff0c\u6bcf\u4e2a\u5b9e\u4f8b\u90fd\u4f1a\u5bf9\u5916\u53d1\u5e03\u8def\u7531\uff0c\u4e0a\u6e38\u8def\u7531\u5668\u9700\u8981\u652f\u6301\u591a\u8def\u5f84 ECMP\u3002 \u4e0b\u8f7d\u5bf9\u5e94 yaml: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/speaker.yaml \u4fee\u6539 yaml \u5185\u76f8\u5e94\u914d\u7f6e\uff1a --neighbor-address=10.32.32.1 --neighbor-as=65030 --cluster-as=65000 neighbor-address : BGP Peer \u7684\u5730\u5740\uff0c\u901a\u5e38\u4e3a\u8def\u7531\u5668\u7f51\u5173\u5730\u5740\u3002 neighbor-as : BGP Peer \u7684 AS \u53f7\u3002 cluster-as : \u5bb9\u5668\u7f51\u7edc\u7684 AS \u53f7\u3002 \u90e8\u7f72 yaml: kubectl apply -f speaker.yaml","title":"\u5b89\u88c5 kube-ovn-speaker"},{"location":"advance/with-bgp/#podsubnet","text":"\u5982\u9700\u4f7f\u7528 BGP \u5bf9\u5916\u53d1\u5e03\u8def\u7531\uff0c\u9996\u5148\u9700\u8981\u5c06\u5bf9\u5e94 Subnet \u7684 natOutgoing \u8bbe\u7f6e\u4e3a false \uff0c\u4f7f\u5f97 Pod IP \u53ef\u4ee5\u76f4\u63a5\u8fdb\u5165\u5e95\u5c42\u7f51\u7edc\u3002 \u589e\u52a0 annotation \u5bf9\u5916\u53d1\u5e03\uff1a kubectl annotate pod sample ovn.kubernetes.io/bgp = true kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp = true \u5220\u9664 annotation \u53d6\u6d88\u53d1\u5e03\uff1a kubectl annotate pod perf-ovn-xzvd4 ovn.kubernetes.io/bgp- kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp-","title":"\u53d1\u5e03 Pod/Subnet \u8def\u7531"},{"location":"advance/with-bgp/#bgp_1","text":"kube-ovn-speaker \u652f\u6301\u66f4\u591a BGP \u53c2\u6570\u8fdb\u884c\u9ad8\u7ea7\u914d\u7f6e\uff0c\u7528\u6237\u53ef\u6839\u636e\u81ea\u5df1\u7f51\u7edc\u73af\u5883\u8fdb\u884c\u8c03\u6574\uff1a announce-cluster-ip : \u662f\u5426\u5bf9\u5916\u53d1\u5e03 Service \u8def\u7531\uff0c\u9ed8\u8ba4\u4e3a false \u3002 auth-password : BGP peer \u7684\u8bbf\u95ee\u5bc6\u7801\u3002 holdtime : BGP \u90bb\u5c45\u95f4\u7684\u5fc3\u8df3\u63a2\u6d4b\u65f6\u95f4\uff0c\u8d85\u8fc7\u6539\u65f6\u95f4\u6ca1\u6709\u6d88\u606f\u7684\u90bb\u5c45\u5c06\u4f1a\u88ab\u79fb\u9664\uff0c\u9ed8\u8ba4\u4e3a 90 \u79d2\u3002 graceful-restart : \u662f\u5426\u542f\u7528 BGP Graceful Restart\u3002 graceful-restart-time : BGP Graceful restart time \u53ef\u53c2\u8003 RFC4724 3\u3002 graceful-restart-deferral-time : BGP Graceful restart deferral time \u53ef\u53c2\u8003 RFC4724 4.1\u3002 passivemode : Speaker \u8fd0\u884c\u5728 passive \u6a21\u5f0f\uff0c\u4e0d\u4e3b\u52a8\u8fde\u63a5 peer\u3002 ebgp-multihop : ebgp ttl \u9ed8\u8ba4\u503c\u4e3a 1\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"BGP \u9ad8\u7ea7\u9009\u9879"},{"location":"advance/with-cilium/","text":"Cilium \u96c6\u6210 \u00b6 Cilium \u662f\u4e00\u6b3e\u57fa\u4e8e eBPF \u7684\u7f51\u7edc\u548c\u5b89\u5168\u7ec4\u4ef6\uff0cKube-OVN \u5229\u7528\u5176\u4e2d\u7684 CNI Chaining \u6a21\u5f0f\u6765\u5bf9\u5df2\u6709\u529f\u80fd\u8fdb\u884c\u589e\u5f3a\u3002 \u7528\u6237\u53ef\u4ee5\u540c\u65f6\u4f7f\u7528 Kube-OVN \u4e30\u5bcc\u7684\u7f51\u7edc\u62bd\u8c61\u80fd\u529b\u548c eBPF \u5e26\u6765\u7684\u76d1\u63a7\u548c\u5b89\u5168\u80fd\u529b\u3002 \u901a\u8fc7\u96c6\u6210 Cilium\uff0cKube-OVN \u7528\u6237\u53ef\u4ee5\u83b7\u5f97\u5982\u4e0b\u589e\u76ca\uff1a \u66f4\u4e30\u5bcc\u9ad8\u6548\u7684\u5b89\u5168\u7b56\u7565\u3002 \u57fa\u4e8e Hubble \u7684\u76d1\u63a7\u89c6\u56fe\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 Linux \u5185\u6838\u7248\u672c\u9ad8\u4e8e 4.19 \u6216\u5176\u4ed6\u517c\u5bb9\u5185\u6838\u4ee5\u83b7\u5f97\u5b8c\u6574 eBPF \u80fd\u529b\u652f\u6301\u3002 \u63d0\u524d\u90e8\u7f72 Helm \u4e3a\u5b89\u88c5 Cilium \u505a\u51c6\u5907\uff0c\u90e8\u7f72 Helm \u8bf7\u53c2\u8003 Installing Helm \u3002 \u914d\u7f6e Kube-OVN \u00b6 \u4e3a\u4e86\u5145\u5206\u4f7f\u7528 Cilium \u7684\u5b89\u5168\u80fd\u529b\uff0c\u9700\u8981\u5173\u95ed Kube-OVN \u5185\u7684 networkpolicy \u529f\u80fd\uff0c\u5e76\u8c03\u6574 CNI \u914d\u7f6e\u4f18\u5148\u7ea7\u3002 \u5728 install.sh \u811a\u672c\u91cc\u4fee\u6539\u4e0b\u5217\u53d8\u91cf\uff1a ENABLE_NP = false CNI_CONFIG_PRIORITY = 10 \u82e5\u5df2\u90e8\u7f72\u5b8c\u6210\uff0c\u53ef\u901a\u8fc7\u4fee\u6539 kube-ovn-controller \u7684\u542f\u52a8\u53c2\u6570\u8fdb\u884c\u8c03\u6574 networkpolicy \uff1a args : - --enable-np=false \u4fee\u6539 kube-ovn-cni \u542f\u52a8\u53c2\u6570\u8c03\u6574 CNI \u914d\u7f6e\u4f18\u5148\u7ea7\uff1a args : - --cni-conf-name=10-kube-ovn.conflist \u5728\u6bcf\u4e2a\u8282\u70b9\u8c03\u6574 Kube-OVN \u914d\u7f6e\u6587\u4ef6\u540d\u79f0\uff0c\u4ee5\u4fbf\u4f18\u5148\u4f7f\u7528 Cilium \u8fdb\u884c\u64cd\u4f5c\uff1a mv /etc/cni/net.d/01-kube-ovn.conflist /etc/cni/net.d/10-kube-ovn.conflist \u90e8\u7f72 Cilium \u00b6 \u521b\u5efa chaining.yaml \u914d\u7f6e\u6587\u4ef6\uff0c\u4f7f\u7528 Cilium \u7684 generic-veth \u6a21\u5f0f\uff1a apiVersion : v1 kind : ConfigMap metadata : name : cni-configuration namespace : kube-system data : cni-config : |- { \"name\": \"generic-veth\", \"cniVersion\": \"0.3.1\", \"plugins\": [ { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\" } }, { \"type\": \"portmap\", \"snat\": true, \"capabilities\": {\"portMappings\": true} }, { \"type\": \"cilium-cni\" } ] } \u5b89\u88c5\u914d\u7f6e\u6587\u4ef6\uff1a kubectl apply -f chaining.yaml \u4f7f\u7528 Helm \u90e8\u7f72 Cilium\uff1a helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --set cni.chainingMode = generic-veth \\ --set cni.customConf = true \\ --set cni.configMap = cni-configuration \\ --set tunnel = disabled \\ --set enableIPv4Masquerade = false \\ --set enableIdentityMark = false \u786e\u8ba4 Cilium \u5b89\u88c5\u6210\u529f\uff1a # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: disabled \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 Containers: cilium Running: 2 cilium-operator Running: 2 Cluster Pods: 8 /11 managed by Cilium Image versions cilium quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb: 2 cilium-operator quay.io/cilium/operator-generic:v1.10.5@sha256:2d2f730f219d489ff0702923bf24c0002cd93eb4b47ba344375566202f56d972: 2 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium \u96c6\u6210"},{"location":"advance/with-cilium/#cilium","text":"Cilium \u662f\u4e00\u6b3e\u57fa\u4e8e eBPF \u7684\u7f51\u7edc\u548c\u5b89\u5168\u7ec4\u4ef6\uff0cKube-OVN \u5229\u7528\u5176\u4e2d\u7684 CNI Chaining \u6a21\u5f0f\u6765\u5bf9\u5df2\u6709\u529f\u80fd\u8fdb\u884c\u589e\u5f3a\u3002 \u7528\u6237\u53ef\u4ee5\u540c\u65f6\u4f7f\u7528 Kube-OVN \u4e30\u5bcc\u7684\u7f51\u7edc\u62bd\u8c61\u80fd\u529b\u548c eBPF \u5e26\u6765\u7684\u76d1\u63a7\u548c\u5b89\u5168\u80fd\u529b\u3002 \u901a\u8fc7\u96c6\u6210 Cilium\uff0cKube-OVN \u7528\u6237\u53ef\u4ee5\u83b7\u5f97\u5982\u4e0b\u589e\u76ca\uff1a \u66f4\u4e30\u5bcc\u9ad8\u6548\u7684\u5b89\u5168\u7b56\u7565\u3002 \u57fa\u4e8e Hubble \u7684\u76d1\u63a7\u89c6\u56fe\u3002","title":"Cilium \u96c6\u6210"},{"location":"advance/with-cilium/#_1","text":"Linux \u5185\u6838\u7248\u672c\u9ad8\u4e8e 4.19 \u6216\u5176\u4ed6\u517c\u5bb9\u5185\u6838\u4ee5\u83b7\u5f97\u5b8c\u6574 eBPF \u80fd\u529b\u652f\u6301\u3002 \u63d0\u524d\u90e8\u7f72 Helm \u4e3a\u5b89\u88c5 Cilium \u505a\u51c6\u5907\uff0c\u90e8\u7f72 Helm \u8bf7\u53c2\u8003 Installing Helm \u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/with-cilium/#kube-ovn","text":"\u4e3a\u4e86\u5145\u5206\u4f7f\u7528 Cilium \u7684\u5b89\u5168\u80fd\u529b\uff0c\u9700\u8981\u5173\u95ed Kube-OVN \u5185\u7684 networkpolicy \u529f\u80fd\uff0c\u5e76\u8c03\u6574 CNI \u914d\u7f6e\u4f18\u5148\u7ea7\u3002 \u5728 install.sh \u811a\u672c\u91cc\u4fee\u6539\u4e0b\u5217\u53d8\u91cf\uff1a ENABLE_NP = false CNI_CONFIG_PRIORITY = 10 \u82e5\u5df2\u90e8\u7f72\u5b8c\u6210\uff0c\u53ef\u901a\u8fc7\u4fee\u6539 kube-ovn-controller \u7684\u542f\u52a8\u53c2\u6570\u8fdb\u884c\u8c03\u6574 networkpolicy \uff1a args : - --enable-np=false \u4fee\u6539 kube-ovn-cni \u542f\u52a8\u53c2\u6570\u8c03\u6574 CNI \u914d\u7f6e\u4f18\u5148\u7ea7\uff1a args : - --cni-conf-name=10-kube-ovn.conflist \u5728\u6bcf\u4e2a\u8282\u70b9\u8c03\u6574 Kube-OVN \u914d\u7f6e\u6587\u4ef6\u540d\u79f0\uff0c\u4ee5\u4fbf\u4f18\u5148\u4f7f\u7528 Cilium \u8fdb\u884c\u64cd\u4f5c\uff1a mv /etc/cni/net.d/01-kube-ovn.conflist /etc/cni/net.d/10-kube-ovn.conflist","title":"\u914d\u7f6e Kube-OVN"},{"location":"advance/with-cilium/#cilium_1","text":"\u521b\u5efa chaining.yaml \u914d\u7f6e\u6587\u4ef6\uff0c\u4f7f\u7528 Cilium \u7684 generic-veth \u6a21\u5f0f\uff1a apiVersion : v1 kind : ConfigMap metadata : name : cni-configuration namespace : kube-system data : cni-config : |- { \"name\": \"generic-veth\", \"cniVersion\": \"0.3.1\", \"plugins\": [ { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\" } }, { \"type\": \"portmap\", \"snat\": true, \"capabilities\": {\"portMappings\": true} }, { \"type\": \"cilium-cni\" } ] } \u5b89\u88c5\u914d\u7f6e\u6587\u4ef6\uff1a kubectl apply -f chaining.yaml \u4f7f\u7528 Helm \u90e8\u7f72 Cilium\uff1a helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --set cni.chainingMode = generic-veth \\ --set cni.customConf = true \\ --set cni.configMap = cni-configuration \\ --set tunnel = disabled \\ --set enableIPv4Masquerade = false \\ --set enableIdentityMark = false \u786e\u8ba4 Cilium \u5b89\u88c5\u6210\u529f\uff1a # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: disabled \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 Containers: cilium Running: 2 cilium-operator Running: 2 Cluster Pods: 8 /11 managed by Cilium Image versions cilium quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb: 2 cilium-operator quay.io/cilium/operator-generic:v1.10.5@sha256:2d2f730f219d489ff0702923bf24c0002cd93eb4b47ba344375566202f56d972: 2 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u90e8\u7f72 Cilium"},{"location":"advance/with-openstack/","text":"OpenStack \u96c6\u6210 \u00b6 \u5728\u4e00\u4e9b\u60c5\u51b5\u4e0b\uff0c\u7528\u6237\u9700\u8981\u4f7f\u7528 OpenStack \u8fd0\u884c\u865a\u62df\u673a\uff0c\u4f7f\u7528 Kubernetes \u8fd0\u884c\u5bb9\u5668\uff0c\u5e76\u9700\u8981\u5bb9\u5668\u548c\u865a\u673a\u4e4b\u95f4\u7f51\u7edc\u4e92\u901a\u5e76\u5904\u4e8e\u7edf\u4e00\u63a7\u5236\u5e73\u9762\u4e0b\u3002\u5982\u679c OpenStack Neutron \u4fa7\u540c\u6837\u4f7f\u7528 OVN \u4f5c\u4e3a\u5e95\u5c42\u7f51\u7edc\u63a7\u5236\uff0c\u90a3\u4e48 Kube-OVN \u53ef\u4ee5\u4f7f\u7528\u96c6\u7fa4\u4e92\u8054\u548c\u5171\u4eab\u5e95\u5c42 OVN \u4e24\u79cd\u65b9\u5f0f\u6253\u901a OpenStack \u548c Kubernetes \u7684\u7f51\u7edc\u3002 \u96c6\u7fa4\u4e92\u8054 \u00b6 \u8be5\u6a21\u5f0f\u548c \u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054 \u6253\u901a\u4e24\u4e2a Kubernetes \u96c6\u7fa4\u7f51\u7edc\u65b9\u5f0f\u7c7b\u4f3c\uff0c\u53ea\u4e0d\u8fc7\u5c06\u96c6\u7fa4\u4e24\u7aef\u6362\u6210 OpenStack \u548c Kubernetes\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u81ea\u52a8\u4e92\u8054\u6a21\u5f0f\u4e0b OpenStack \u548c Kubernetes \u5185\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u76f8\u4e92\u91cd\u53e0\u3002 \u9700\u8981\u5b58\u5728\u4e00\u7ec4\u673a\u5668\u53ef\u4ee5\u88ab\u6bcf\u4e2a\u96c6\u7fa4\u901a\u8fc7\u7f51\u7edc\u8bbf\u95ee\uff0c\u7528\u6765\u90e8\u7f72\u8de8\u96c6\u7fa4\u4e92\u8054\u7684\u63a7\u5236\u5668\u3002 \u6bcf\u4e2a\u96c6\u7fa4\u9700\u8981\u6709\u4e00\u7ec4\u53ef\u4ee5\u901a\u8fc7 IP \u8fdb\u884c\u8de8\u96c6\u7fa4\u4e92\u8bbf\u7684\u673a\u5668\u4f5c\u4e3a\u4e4b\u540e\u7684\u7f51\u5173\u8282\u70b9\u3002 \u8be5\u65b9\u6848\u53ea\u6253\u901a Kubernetes \u9ed8\u8ba4\u5b50\u7f51\u548c OpenStack \u7684\u9009\u5b9a VPC\u3002 \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93 \u00b6 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\uff1a docker run --name = ovn-ic-db -d --network = host -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh Kubernetes \u4fa7\u64cd\u4f5c \u00b6 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : \u662f\u5426\u5f00\u542f\u96c6\u7fa4\u4e92\u8054\u3002 az-name : \u533a\u5206\u4e0d\u540c\u96c6\u7fa4\u7684\u96c6\u7fa4\u540d\u79f0\uff0c\u6bcf\u4e2a\u4e92\u8054\u96c6\u7fa4\u9700\u4e0d\u540c\u3002 ic-db-host : \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u5730\u5740\u3002 ic-nb-port : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6645\u3002 ic-sb-port : OVN-IC \u5357\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6646\u3002 gw-nodes : \u96c6\u7fa4\u4e92\u8054\u4e2d\u627f\u62c5\u7f51\u5173\u5de5\u4f5c\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 auto-route : \u662f\u5426\u81ea\u52a8\u5bf9\u5916\u53d1\u5e03\u548c\u5b66\u4e60\u8def\u7531\u3002 OpenStack \u4fa7\u64cd\u4f5c \u00b6 \u521b\u5efa\u548c Kubernetes \u4e92\u8054\u7684\u903b\u8f91\u8def\u7531\u5668\uff1a # openstack router create router0 # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | d5b38655-249a-4192-8046-71aa4d2b4af1 | router0 | ACTIVE | UP | 98a29ab7388347e7b5ff8bdd181ba4f9 | +--------------------------------------+---------+--------+-------+----------------------------------+ \u5728 OpenStack \u5185\u7684 OVN \u5317\u5411\u6570\u636e\u5e93\u4e2d\u8bbe\u7f6e\u53ef\u7528\u533a\u540d\u5b57\uff0c\u8be5\u540d\u79f0\u9700\u548c\u5176\u4ed6\u4e92\u8054\u96c6\u7fa4\u4e0d\u540c\uff1a ovn-nbctl set NB_Global . name = op-az \u5728\u53ef\u8bbf\u95ee OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u542f\u52a8 OVN-IC \u63a7\u5236\u5668\uff1a /usr/share/ovn/scripts/ovn-ctl --ovn-ic-nb-db = tcp:192.168.65.3:6645 \\ --ovn-ic-sb-db = tcp:192.168.65.3:6646 \\ --ovn-northd-nb-db = unix:/run/ovn/ovnnb_db.sock \\ --ovn-northd-sb-db = unix:/run/ovn/ovnsb_db.sock \\ start_ic ovn-ic-nb-db \uff0c ovn-ic-sb-db : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u548c\u5357\u5411\u6570\u636e\u5e93\u5730\u5740\u3002 ovn-northd-nb-db \uff0c ovn-northd-sb-db : \u5f53\u524d\u96c6\u7fa4 OVN \u5317\u5411\u6570\u636e\u5e93\u548c\u5357\u5411\u6570\u636e\u5730\u5740\u3002 \u914d\u7f6e\u4e92\u8054\u7f51\u5173\u8282\u70b9\uff1a ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn = true \u63a5\u4e0b\u6765\u9700\u8981\u5728 OpenStack \u7684 OVN \u5185\u8fdb\u884c\u64cd\u4f5c\u521b\u5efa\u903b\u8f91\u62d3\u6251\u3002 \u8fde\u63a5 ts \u4e92\u8054\u4ea4\u6362\u673a\u548c router0 \u903b\u8f91\u8def\u7531\u5668\uff0c\u5e76\u8bbe\u7f6e\u76f8\u5173\u89c4\u5219\uff1a ovn-nbctl lrp-add router0 lrp-router0-ts 00 :02:ef:11:39:4f 169 .254.100.73/24 ovn-nbctl lsp-add ts lsp-ts-router0 -- lsp-set-addresses lsp-ts-router0 router \\ -- lsp-set-type lsp-ts-router0 router \\ -- lsp-set-options lsp-ts-router0 router-port = lrp-router0-ts ovn-nbctl lrp-set-gateway-chassis lrp-router0-ts { gateway chassis } 1000 ovn-nbctl set NB_Global . options:ic-route-adv = true options:ic-route-learn = true \u9a8c\u8bc1\u5df2\u5b66\u4e60\u5230 Kubernetes \u8def\u7531\u89c4\u5219\uff1a # ovn-nbctl lr-route-list router0 IPv4 Routes 10 .0.0.22 169 .254.100.34 dst-ip ( learned ) 10 .16.0.0/16 169 .254.100.34 dst-ip ( learned ) \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728 router0 \u7f51\u7edc\u4e0b\u521b\u5efa\u865a\u673a\u9a8c\u8bc1\u662f\u5426\u53ef\u4ee5\u548c Kubernetes \u4e0b Pod \u4e92\u901a\u3002 \u5171\u4eab\u5e95\u5c42 OVN \u00b6 \u5728\u8be5\u65b9\u6848\u4e0b\uff0cOpenStack \u548c Kubernetes \u5171\u4eab\u4f7f\u7528\u540c\u4e00\u4e2a OVN\uff0c\u56e0\u6b64\u53ef\u4ee5\u5c06\u4e24\u8005\u7684 VPC \u548c Subnet \u7b49\u6982\u5ff5\u62c9\u9f50\uff0c\u5b9e\u73b0\u66f4\u597d\u7684\u63a7\u5236\u548c\u4e92\u8054\u3002 \u5728\u8be5\u6a21\u5f0f\u4e0b\u6211\u4eec\u6b63\u5e38\u4f7f\u7528 Kube-OVN \u90e8\u7f72 OVN\uff0cOpenStack \u4fee\u6539 Neutron \u914d\u7f6e\u5b9e\u73b0\u8fde\u63a5\u540c\u4e00\u4e2a OVN \u6570\u636e\u5e93\u3002OpenStack \u9700\u4f7f\u7528 networking-ovn \u4f5c\u4e3a Neutron \u540e\u7aef\u5b9e\u73b0\u3002 Neutron \u914d\u7f6e\u4fee\u6539 \u00b6 \u4fee\u6539 Neutron \u914d\u7f6e\u6587\u4ef6 /etc/neutron/plugins/ml2/ml2_conf.ini \uff1a [ ovn ] ... ovn_nb_connection = tcp: [ 192 .168.137.176 ] :6641,tcp: [ 192 .168.137.177 ] :6641,tcp: [ 192 .168.137.178 ] :6641 ovn_sb_connection = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovn_l3_scheduler = OVN_L3_SCHEDULER ovn_nb_connection \uff0c ovn_sb_connection : \u5730\u5740\u9700\u4fee\u6539\u4e3a Kube-OVN \u90e8\u7f72 ovn-central \u8282\u70b9\u7684\u5730\u5740\u3002 \u4fee\u6539\u6bcf\u4e2a\u8282\u70b9\u7684 OVS \u914d\u7f6e\uff1a ovs-vsctl set open . external-ids:ovn-remote = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovs-vsctl set open . external-ids:ovn-encap-type = geneve ovs-vsctl set open . external-ids:ovn-encap-ip = 192 .168.137.200 external-ids:ovn-remote : \u5730\u5740\u9700\u4fee\u6539\u4e3a Kube-OVN \u90e8\u7f72 ovn-central \u8282\u70b9\u7684\u5730\u5740\u3002 ovn-encap-ip : \u4fee\u6539\u4e3a\u5f53\u524d\u8282\u70b9\u7684 IP \u5730\u5740\u3002 \u5728 Kubernetes \u4e2d\u4f7f\u7528 OpenStack \u5185\u8d44\u6e90 \u00b6 \u63a5\u4e0b\u6765\u4ecb\u7ecd\u5982\u4f55\u5728 Kubernetes \u4e2d\u67e5\u8be2 OpenStack \u7684\u7f51\u7edc\u8d44\u6e90\u5e76\u5728 OpenStack \u7684\u5b50\u7f51\u4e2d\u521b\u5efa Pod\u3002 \u67e5\u8be2 OpenStack \u4e2d\u5df2\u6709\u7684\u7f51\u7edc\u8d44\u6e90\uff0c\u5982\u4e0b\u8d44\u6e90\u5df2\u7ecf\u9884\u5148\u521b\u5efa\u5b8c\u6210\uff1a # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | 22040ed5-0598-4f77-bffd-e7fd4db47e93 | router0 | ACTIVE | UP | 62381a21d569404aa236a5dd8712449c | +--------------------------------------+---------+--------+-------+----------------------------------+ # openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | cd59e36a-37db-4c27-b709-d35379a7920f | provider | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | +--------------------------------------+----------+--------------------------------------+ # openstack subnet list +--------------------------------------+-------------+--------------------------------------+----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------------+--------------------------------------+----------------+ | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | provider-v4 | cd59e36a-37db-4c27-b709-d35379a7920f | 192 .168.1.0/24 | +--------------------------------------+-------------+--------------------------------------+----------------+ # openstack server list +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | 8433d622-a8d6-41a7-8b31-49abfd64f639 | provider-instance | ACTIVE | provider = 192 .168.1.61 | ubuntu | m1 | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ \u5728 Kubernetes \u4fa7\uff0c\u67e5\u8be2 VPC \u8d44\u6e90\uff1a # kubectl get vpc NAME STANDBY SUBNETS neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 true [ \"neutron-cd59e36a-37db-4c27-b709-d35379a7920f\" ] ovn-cluster true [ \"join\" , \"ovn-default\" ] neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 \u4e3a\u4ece OpenStack \u540c\u6b65\u8fc7\u6765\u7684 VPC \u8d44\u6e90\u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u6309\u7167 Kube-OVN \u539f\u751f\u7684 VPC \u548c Subnet \u64cd\u4f5c\u521b\u5efa Pod \u5e76\u8fd0\u884c\u3002 VPC, Subnet \u7ed1\u5b9a Namespace net2 \uff0c\u5e76\u521b\u5efa Pod: apiVersion : v1 kind : Namespace metadata : name : net2 --- apiVersion : kubeovn.io/v1 kind : Vpc metadata : creationTimestamp : \"2021-06-20T13:34:11Z\" generation : 2 labels : ovn.kubernetes.io/vpc_external : \"true\" name : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 resourceVersion : \"583728\" uid : 18d4c654-f511-4def-a3a0-a6434d237c1e spec : namespaces : - net2 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 namespaces : - net2 cidrBlock : 12.0.1.0/24 natOutgoing : false --- apiVersion : v1 kind : Pod metadata : name : ubuntu namespace : net2 spec : containers : - image : docker.io/kubeovn/kube-ovn:v1.8.0 command : - \"sleep\" - \"604800\" imagePullPolicy : IfNotPresent name : ubuntu restartPolicy : Always \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OpenStack \u96c6\u6210"},{"location":"advance/with-openstack/#openstack","text":"\u5728\u4e00\u4e9b\u60c5\u51b5\u4e0b\uff0c\u7528\u6237\u9700\u8981\u4f7f\u7528 OpenStack \u8fd0\u884c\u865a\u62df\u673a\uff0c\u4f7f\u7528 Kubernetes \u8fd0\u884c\u5bb9\u5668\uff0c\u5e76\u9700\u8981\u5bb9\u5668\u548c\u865a\u673a\u4e4b\u95f4\u7f51\u7edc\u4e92\u901a\u5e76\u5904\u4e8e\u7edf\u4e00\u63a7\u5236\u5e73\u9762\u4e0b\u3002\u5982\u679c OpenStack Neutron \u4fa7\u540c\u6837\u4f7f\u7528 OVN \u4f5c\u4e3a\u5e95\u5c42\u7f51\u7edc\u63a7\u5236\uff0c\u90a3\u4e48 Kube-OVN \u53ef\u4ee5\u4f7f\u7528\u96c6\u7fa4\u4e92\u8054\u548c\u5171\u4eab\u5e95\u5c42 OVN \u4e24\u79cd\u65b9\u5f0f\u6253\u901a OpenStack \u548c Kubernetes \u7684\u7f51\u7edc\u3002","title":"OpenStack \u96c6\u6210"},{"location":"advance/with-openstack/#_1","text":"\u8be5\u6a21\u5f0f\u548c \u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054 \u6253\u901a\u4e24\u4e2a Kubernetes \u96c6\u7fa4\u7f51\u7edc\u65b9\u5f0f\u7c7b\u4f3c\uff0c\u53ea\u4e0d\u8fc7\u5c06\u96c6\u7fa4\u4e24\u7aef\u6362\u6210 OpenStack \u548c Kubernetes\u3002","title":"\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-openstack/#_2","text":"\u81ea\u52a8\u4e92\u8054\u6a21\u5f0f\u4e0b OpenStack \u548c Kubernetes \u5185\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u76f8\u4e92\u91cd\u53e0\u3002 \u9700\u8981\u5b58\u5728\u4e00\u7ec4\u673a\u5668\u53ef\u4ee5\u88ab\u6bcf\u4e2a\u96c6\u7fa4\u901a\u8fc7\u7f51\u7edc\u8bbf\u95ee\uff0c\u7528\u6765\u90e8\u7f72\u8de8\u96c6\u7fa4\u4e92\u8054\u7684\u63a7\u5236\u5668\u3002 \u6bcf\u4e2a\u96c6\u7fa4\u9700\u8981\u6709\u4e00\u7ec4\u53ef\u4ee5\u901a\u8fc7 IP \u8fdb\u884c\u8de8\u96c6\u7fa4\u4e92\u8bbf\u7684\u673a\u5668\u4f5c\u4e3a\u4e4b\u540e\u7684\u7f51\u5173\u8282\u70b9\u3002 \u8be5\u65b9\u6848\u53ea\u6253\u901a Kubernetes \u9ed8\u8ba4\u5b50\u7f51\u548c OpenStack \u7684\u9009\u5b9a VPC\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/with-openstack/#ovn-ic","text":"\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\uff1a docker run --name = ovn-ic-db -d --network = host -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh","title":"\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93"},{"location":"advance/with-openstack/#kubernetes","text":"\u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : \u662f\u5426\u5f00\u542f\u96c6\u7fa4\u4e92\u8054\u3002 az-name : \u533a\u5206\u4e0d\u540c\u96c6\u7fa4\u7684\u96c6\u7fa4\u540d\u79f0\uff0c\u6bcf\u4e2a\u4e92\u8054\u96c6\u7fa4\u9700\u4e0d\u540c\u3002 ic-db-host : \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u5730\u5740\u3002 ic-nb-port : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6645\u3002 ic-sb-port : OVN-IC \u5357\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6646\u3002 gw-nodes : \u96c6\u7fa4\u4e92\u8054\u4e2d\u627f\u62c5\u7f51\u5173\u5de5\u4f5c\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 auto-route : \u662f\u5426\u81ea\u52a8\u5bf9\u5916\u53d1\u5e03\u548c\u5b66\u4e60\u8def\u7531\u3002","title":"Kubernetes \u4fa7\u64cd\u4f5c"},{"location":"advance/with-openstack/#openstack_1","text":"\u521b\u5efa\u548c Kubernetes \u4e92\u8054\u7684\u903b\u8f91\u8def\u7531\u5668\uff1a # openstack router create router0 # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | d5b38655-249a-4192-8046-71aa4d2b4af1 | router0 | ACTIVE | UP | 98a29ab7388347e7b5ff8bdd181ba4f9 | +--------------------------------------+---------+--------+-------+----------------------------------+ \u5728 OpenStack \u5185\u7684 OVN \u5317\u5411\u6570\u636e\u5e93\u4e2d\u8bbe\u7f6e\u53ef\u7528\u533a\u540d\u5b57\uff0c\u8be5\u540d\u79f0\u9700\u548c\u5176\u4ed6\u4e92\u8054\u96c6\u7fa4\u4e0d\u540c\uff1a ovn-nbctl set NB_Global . name = op-az \u5728\u53ef\u8bbf\u95ee OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u542f\u52a8 OVN-IC \u63a7\u5236\u5668\uff1a /usr/share/ovn/scripts/ovn-ctl --ovn-ic-nb-db = tcp:192.168.65.3:6645 \\ --ovn-ic-sb-db = tcp:192.168.65.3:6646 \\ --ovn-northd-nb-db = unix:/run/ovn/ovnnb_db.sock \\ --ovn-northd-sb-db = unix:/run/ovn/ovnsb_db.sock \\ start_ic ovn-ic-nb-db \uff0c ovn-ic-sb-db : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u548c\u5357\u5411\u6570\u636e\u5e93\u5730\u5740\u3002 ovn-northd-nb-db \uff0c ovn-northd-sb-db : \u5f53\u524d\u96c6\u7fa4 OVN \u5317\u5411\u6570\u636e\u5e93\u548c\u5357\u5411\u6570\u636e\u5730\u5740\u3002 \u914d\u7f6e\u4e92\u8054\u7f51\u5173\u8282\u70b9\uff1a ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn = true \u63a5\u4e0b\u6765\u9700\u8981\u5728 OpenStack \u7684 OVN \u5185\u8fdb\u884c\u64cd\u4f5c\u521b\u5efa\u903b\u8f91\u62d3\u6251\u3002 \u8fde\u63a5 ts \u4e92\u8054\u4ea4\u6362\u673a\u548c router0 \u903b\u8f91\u8def\u7531\u5668\uff0c\u5e76\u8bbe\u7f6e\u76f8\u5173\u89c4\u5219\uff1a ovn-nbctl lrp-add router0 lrp-router0-ts 00 :02:ef:11:39:4f 169 .254.100.73/24 ovn-nbctl lsp-add ts lsp-ts-router0 -- lsp-set-addresses lsp-ts-router0 router \\ -- lsp-set-type lsp-ts-router0 router \\ -- lsp-set-options lsp-ts-router0 router-port = lrp-router0-ts ovn-nbctl lrp-set-gateway-chassis lrp-router0-ts { gateway chassis } 1000 ovn-nbctl set NB_Global . options:ic-route-adv = true options:ic-route-learn = true \u9a8c\u8bc1\u5df2\u5b66\u4e60\u5230 Kubernetes \u8def\u7531\u89c4\u5219\uff1a # ovn-nbctl lr-route-list router0 IPv4 Routes 10 .0.0.22 169 .254.100.34 dst-ip ( learned ) 10 .16.0.0/16 169 .254.100.34 dst-ip ( learned ) \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728 router0 \u7f51\u7edc\u4e0b\u521b\u5efa\u865a\u673a\u9a8c\u8bc1\u662f\u5426\u53ef\u4ee5\u548c Kubernetes \u4e0b Pod \u4e92\u901a\u3002","title":"OpenStack \u4fa7\u64cd\u4f5c"},{"location":"advance/with-openstack/#ovn","text":"\u5728\u8be5\u65b9\u6848\u4e0b\uff0cOpenStack \u548c Kubernetes \u5171\u4eab\u4f7f\u7528\u540c\u4e00\u4e2a OVN\uff0c\u56e0\u6b64\u53ef\u4ee5\u5c06\u4e24\u8005\u7684 VPC \u548c Subnet \u7b49\u6982\u5ff5\u62c9\u9f50\uff0c\u5b9e\u73b0\u66f4\u597d\u7684\u63a7\u5236\u548c\u4e92\u8054\u3002 \u5728\u8be5\u6a21\u5f0f\u4e0b\u6211\u4eec\u6b63\u5e38\u4f7f\u7528 Kube-OVN \u90e8\u7f72 OVN\uff0cOpenStack \u4fee\u6539 Neutron \u914d\u7f6e\u5b9e\u73b0\u8fde\u63a5\u540c\u4e00\u4e2a OVN \u6570\u636e\u5e93\u3002OpenStack \u9700\u4f7f\u7528 networking-ovn \u4f5c\u4e3a Neutron \u540e\u7aef\u5b9e\u73b0\u3002","title":"\u5171\u4eab\u5e95\u5c42 OVN"},{"location":"advance/with-openstack/#neutron","text":"\u4fee\u6539 Neutron \u914d\u7f6e\u6587\u4ef6 /etc/neutron/plugins/ml2/ml2_conf.ini \uff1a [ ovn ] ... ovn_nb_connection = tcp: [ 192 .168.137.176 ] :6641,tcp: [ 192 .168.137.177 ] :6641,tcp: [ 192 .168.137.178 ] :6641 ovn_sb_connection = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovn_l3_scheduler = OVN_L3_SCHEDULER ovn_nb_connection \uff0c ovn_sb_connection : \u5730\u5740\u9700\u4fee\u6539\u4e3a Kube-OVN \u90e8\u7f72 ovn-central \u8282\u70b9\u7684\u5730\u5740\u3002 \u4fee\u6539\u6bcf\u4e2a\u8282\u70b9\u7684 OVS \u914d\u7f6e\uff1a ovs-vsctl set open . external-ids:ovn-remote = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovs-vsctl set open . external-ids:ovn-encap-type = geneve ovs-vsctl set open . external-ids:ovn-encap-ip = 192 .168.137.200 external-ids:ovn-remote : \u5730\u5740\u9700\u4fee\u6539\u4e3a Kube-OVN \u90e8\u7f72 ovn-central \u8282\u70b9\u7684\u5730\u5740\u3002 ovn-encap-ip : \u4fee\u6539\u4e3a\u5f53\u524d\u8282\u70b9\u7684 IP \u5730\u5740\u3002","title":"Neutron \u914d\u7f6e\u4fee\u6539"},{"location":"advance/with-openstack/#kubernetes-openstack","text":"\u63a5\u4e0b\u6765\u4ecb\u7ecd\u5982\u4f55\u5728 Kubernetes \u4e2d\u67e5\u8be2 OpenStack \u7684\u7f51\u7edc\u8d44\u6e90\u5e76\u5728 OpenStack \u7684\u5b50\u7f51\u4e2d\u521b\u5efa Pod\u3002 \u67e5\u8be2 OpenStack \u4e2d\u5df2\u6709\u7684\u7f51\u7edc\u8d44\u6e90\uff0c\u5982\u4e0b\u8d44\u6e90\u5df2\u7ecf\u9884\u5148\u521b\u5efa\u5b8c\u6210\uff1a # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | 22040ed5-0598-4f77-bffd-e7fd4db47e93 | router0 | ACTIVE | UP | 62381a21d569404aa236a5dd8712449c | +--------------------------------------+---------+--------+-------+----------------------------------+ # openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | cd59e36a-37db-4c27-b709-d35379a7920f | provider | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | +--------------------------------------+----------+--------------------------------------+ # openstack subnet list +--------------------------------------+-------------+--------------------------------------+----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------------+--------------------------------------+----------------+ | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | provider-v4 | cd59e36a-37db-4c27-b709-d35379a7920f | 192 .168.1.0/24 | +--------------------------------------+-------------+--------------------------------------+----------------+ # openstack server list +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | 8433d622-a8d6-41a7-8b31-49abfd64f639 | provider-instance | ACTIVE | provider = 192 .168.1.61 | ubuntu | m1 | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ \u5728 Kubernetes \u4fa7\uff0c\u67e5\u8be2 VPC \u8d44\u6e90\uff1a # kubectl get vpc NAME STANDBY SUBNETS neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 true [ \"neutron-cd59e36a-37db-4c27-b709-d35379a7920f\" ] ovn-cluster true [ \"join\" , \"ovn-default\" ] neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 \u4e3a\u4ece OpenStack \u540c\u6b65\u8fc7\u6765\u7684 VPC \u8d44\u6e90\u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u6309\u7167 Kube-OVN \u539f\u751f\u7684 VPC \u548c Subnet \u64cd\u4f5c\u521b\u5efa Pod \u5e76\u8fd0\u884c\u3002 VPC, Subnet \u7ed1\u5b9a Namespace net2 \uff0c\u5e76\u521b\u5efa Pod: apiVersion : v1 kind : Namespace metadata : name : net2 --- apiVersion : kubeovn.io/v1 kind : Vpc metadata : creationTimestamp : \"2021-06-20T13:34:11Z\" generation : 2 labels : ovn.kubernetes.io/vpc_external : \"true\" name : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 resourceVersion : \"583728\" uid : 18d4c654-f511-4def-a3a0-a6434d237c1e spec : namespaces : - net2 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 namespaces : - net2 cidrBlock : 12.0.1.0/24 natOutgoing : false --- apiVersion : v1 kind : Pod metadata : name : ubuntu namespace : net2 spec : containers : - image : docker.io/kubeovn/kube-ovn:v1.8.0 command : - \"sleep\" - \"604800\" imagePullPolicy : IfNotPresent name : ubuntu restartPolicy : Always \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5728 Kubernetes \u4e2d\u4f7f\u7528 OpenStack \u5185\u8d44\u6e90"},{"location":"advance/with-ovn-ic/","text":"\u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054 \u00b6 Kube-OVN \u652f\u6301\u901a\u8fc7 OVN-IC \u5c06\u4e24\u4e2a Kubernetes \u96c6\u7fa4 Pod \u7f51\u7edc\u6253\u901a\uff0c\u6253\u901a\u540e\u7684\u4e24\u4e2a\u96c6\u7fa4\u5185\u7684 Pod \u53ef\u4ee5\u901a\u8fc7 Pod IP \u8fdb\u884c\u76f4\u63a5\u901a\u4fe1\u3002 Kube-OVN \u4f7f\u7528\u96a7\u9053\u5bf9\u8de8\u96c6\u7fa4\u6d41\u91cf\u8fdb\u884c\u5c01\u88c5\uff0c\u4e24\u4e2a\u96c6\u7fa4\u4e4b\u95f4\u53ea\u8981\u5b58\u5728\u4e00\u7ec4 IP \u53ef\u8fbe\u7684\u673a\u5668\u5373\u53ef\u5b8c\u6210\u5bb9\u5668\u7f51\u7edc\u7684\u4e92\u901a\u3002 \u8be5\u6a21\u5f0f\u7684\u591a\u96c6\u7fa4\u4e92\u8054\u4e3a Overlay \u7f51\u7edc\u529f\u80fd\uff0cUnderlay \u7f51\u7edc\u5982\u679c\u60f3\u8981\u5b9e\u73b0\u96c6\u7fa4\u4e92\u8054\u9700\u8981\u5e95\u5c42\u57fa\u7840\u8bbe\u65bd\u505a\u7f51\u7edc\u6253\u901a\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u81ea\u52a8\u4e92\u8054\u6a21\u5f0f\u4e0b\u4e0d\u540c\u96c6\u7fa4\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u76f8\u4e92\u91cd\u53e0\uff0c\u9ed8\u8ba4\u5b50\u7f51\u9700\u5728\u5b89\u88c5\u65f6\u914d\u7f6e\u4e3a\u4e0d\u91cd\u53e0\u7684\u7f51\u6bb5\u3002\u82e5\u5b58\u5728\u91cd\u53e0\u9700\u53c2\u8003\u540e\u7eed\u624b\u52a8\u4e92\u8054\u8fc7\u7a0b\uff0c\u53ea\u80fd\u5c06\u4e0d\u91cd\u53e0\u7f51\u6bb5\u6253\u901a\u3002 \u9700\u8981\u5b58\u5728\u4e00\u7ec4\u673a\u5668\u53ef\u4ee5\u88ab\u6bcf\u4e2a\u96c6\u7fa4\u7684 kube-ovn-controller \u901a\u8fc7 IP \u8bbf\u95ee\uff0c\u7528\u6765\u90e8\u7f72\u8de8\u96c6\u7fa4\u4e92\u8054\u7684\u63a7\u5236\u5668\u3002 \u6bcf\u4e2a\u96c6\u7fa4\u9700\u8981\u6709\u4e00\u7ec4\u53ef\u4ee5\u901a\u8fc7 IP \u8fdb\u884c\u8de8\u96c6\u7fa4\u4e92\u8bbf\u7684\u673a\u5668\u4f5c\u4e3a\u4e4b\u540e\u7684\u7f51\u5173\u8282\u70b9\u3002 \u8be5\u529f\u80fd\u53ea\u5bf9\u9ed8\u8ba4 VPC \u751f\u6548\uff0c\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u65e0\u6cd5\u4f7f\u7528\u4e92\u8054\u529f\u80fd\u3002 \u90e8\u7f72\u5355\u8282\u70b9 OVN-IC \u6570\u636e\u5e93 \u00b6 \u5728\u6bcf\u4e2a\u96c6\u7fa4 kube-ovn-controller \u53ef\u901a\u8fc7 IP \u8bbf\u95ee\u7684\u673a\u5668\u4e0a\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\uff0c\u8be5\u8282\u70b9\u5c06\u4fdd\u5b58\u5404\u4e2a\u96c6\u7fa4\u540c\u6b65\u4e0a\u6765\u7684\u7f51\u7edc\u914d\u7f6e\u4fe1\u606f\u3002 \u90e8\u7f72 docker \u7684\u73af\u5883\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5bf9\u4e8e\u90e8\u7f72 containerd \u53d6\u4ee3 docker \u7684\u73af\u5883\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh \u81ea\u52a8\u8def\u7531\u8bbe\u7f6e \u00b6 \u5728\u81ea\u52a8\u8def\u7531\u8bbe\u7f6e\u4e0b\uff0c\u6bcf\u4e2a\u96c6\u7fa4\u4f1a\u5c06\u81ea\u5df1\u9ed8\u8ba4 VPC \u4e0b Subnet \u7684 CIDR \u4fe1\u606f\u540c\u6b65\u7ed9 OVN-IC \uff0c\u56e0\u6b64\u8981\u786e\u4fdd\u4e24\u4e2a\u96c6\u7fa4\u7684 Subnet CIDR \u4e0d\u5b58\u5728\u91cd\u53e0\u3002 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : \u662f\u5426\u5f00\u542f\u96c6\u7fa4\u4e92\u8054\u3002 az-name : \u533a\u5206\u4e0d\u540c\u96c6\u7fa4\u7684\u96c6\u7fa4\u540d\u79f0\uff0c\u6bcf\u4e2a\u4e92\u8054\u96c6\u7fa4\u9700\u4e0d\u540c\u3002 ic-db-host : \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u5730\u5740\u3002 ic-nb-port : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6645\u3002 ic-sb-port : OVN-IC \u5357\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6646\u3002 gw-nodes : \u96c6\u7fa4\u4e92\u8054\u4e2d\u627f\u62c5\u7f51\u5173\u5de5\u4f5c\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 auto-route : \u662f\u5426\u81ea\u52a8\u5bf9\u5916\u53d1\u5e03\u548c\u5b66\u4e60\u8def\u7531\u3002 \u6ce8\u610f\uff1a \u4e3a\u4e86\u4fdd\u8bc1\u64cd\u4f5c\u7684\u6b63\u786e\u6027\uff0c ovn-ic-config \u8fd9\u4e2a ConfigMap \u4e0d\u5141\u8bb8\u4fee\u6539\u3002\u5982\u6709\u53c2\u6570\u9700\u8981\u53d8\u66f4\uff0c\u8bf7\u5220\u9664\u8be5 ConfigMap\uff0c\u4fee\u6539\u540e\u518d\u5e94\u7528\u6b64 ConfigMap\u3002 \u5728 ovn-ic \u5bb9\u5668\u5185\u901a\u8fc7\u4e0b\u9762\u7684\u547d\u4ee4\u67e5\u770b\u662f\u5426\u5df2\u5efa\u7acb\u4e92\u8054\u903b\u8f91\u4ea4\u6362\u673a ts \uff1a # ovn-ic-sbctl show availability-zone az1 gateway deee03e0-af16-4f45-91e9-b50c3960f809 hostname: az1-gw type: geneve ip: 192 .168.42.145 port ts-az1 transit switch: ts address: [ \"00:00:00:50:AC:8C 169.254.100.45/24\" ] availability-zone az2 gateway e94cc831-8143-40e3-a478-90352773327b hostname: az2-gw type: geneve ip: 192 .168.42.149 port ts-az2 transit switch: ts address: [ \"00:00:00:07:4A:59 169.254.100.63/24\" ] \u5728\u6bcf\u4e2a\u96c6\u7fa4\u89c2\u5bdf\u903b\u8f91\u8def\u7531\u662f\u5426\u6709\u5b66\u4e60\u5230\u7684\u5bf9\u7aef\u8def\u7531\uff1a # kubectl ko nbctl lr-route-list ovn-cluster IPv4 Routes 10 .42.1.1 169 .254.100.45 dst-ip ( learned ) 10 .42.1.3 100 .64.0.2 dst-ip 10 .16.0.2 100 .64.0.2 src-ip 10 .16.0.3 100 .64.0.2 src-ip 10 .16.0.4 100 .64.0.2 src-ip 10 .16.0.6 100 .64.0.2 src-ip 10 .17.0.0/16 169 .254.100.45 dst-ip ( learned ) 100 .65.0.0/16 169 .254.100.45 dst-ip ( learned ) \u63a5\u4e0b\u6765\u53ef\u4ee5\u5c1d\u8bd5\u5728\u96c6\u7fa4 1 \u5185\u7684\u4e00\u4e2a Pod \u5185\u76f4\u63a5 ping \u96c6\u7fa4 2 \u5185\u7684\u4e00\u4e2a Pod IP \u89c2\u5bdf\u662f\u5426\u53ef\u4ee5\u8054\u901a\u3002 \u5bf9\u4e8e\u67d0\u4e2a\u4e0d\u60f3\u5bf9\u5916\u81ea\u52a8\u53d1\u5e03\u8def\u7531\u7684\u5b50\u7f51\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 Subnet \u91cc\u7684 disableInterConnection \u6765\u7981\u6b62\u8def\u7531\u5e7f\u64ad\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : no-advertise spec : cidrBlock : 10.199.0.0/16 disableInterConnection : true \u624b\u52a8\u8def\u7531\u8bbe\u7f6e \u00b6 \u5bf9\u4e8e\u96c6\u7fa4\u95f4\u5b58\u5728\u91cd\u53e0 CIDR \u53ea\u5e0c\u671b\u505a\u90e8\u5206\u5b50\u7f51\u6253\u901a\u7684\u60c5\u51b5\uff0c\u53ef\u4ee5\u901a\u8fc7\u4e0b\u9762\u7684\u6b65\u9aa4\u624b\u52a8\u53d1\u5e03\u5b50\u7f51\u8def\u7531\u3002 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff0c\u5e76\u5c06 auto-route \u8bbe\u7f6e\u4e3a false \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"false\" \u5728\u6bcf\u4e2a\u96c6\u7fa4\u5206\u522b\u67e5\u770b\u8fdc\u7aef\u903b\u8f91\u7aef\u53e3\u7684\u5730\u5740\uff0c\u7528\u4e8e\u4e4b\u540e\u624b\u52a8\u914d\u7f6e\u8def\u7531\uff1a [ root@az1 ~ ] # kubectl ko nbctl show switch a391d3a1-14a0-4841-9836-4bd930c447fb ( ts ) port ts-az1 type: router router-port: az1-ts port ts-az2 type: remote addresses: [ \"00:00:00:4B:E2:9F 169.254.100.31/24\" ] [ root@az2 ~ ] # kubectl ko nbctl show switch da6138b8-de81-4908-abf9-b2224ec4edf3 ( ts ) port ts-az2 type: router router-port: az2-ts port ts-az1 type: remote addresses: [ \"00:00:00:FB:2A:F7 169.254.100.79/24\" ] \u7531\u4e0a\u8f93\u51fa\u53ef\u77e5\uff0c\u96c6\u7fa4 az1 \u5230 \u96c6\u7fa4 az2 \u7684\u8fdc\u7aef\u5730\u5740\u4e3a 169.254.100.31 \uff0c az2 \u5230 az1 \u7684\u8fdc\u7aef\u5730\u5740\u4e3a 169.254.100.79 \u3002 \u4e0b\u9762\u624b\u52a8\u8bbe\u7f6e\u8def\u7531\uff0c\u5728\u8be5\u4f8b\u5b50\u4e2d\uff0c\u96c6\u7fa4 az1 \u5185\u7684\u5b50\u7f51 CIDR \u4e3a 10.16.0.0/24 \uff0c\u96c6\u7fa4 az2 \u5185\u7684\u5b50\u7f51 CIDR \u4e3a 10.17.0.0/24 \u3002 \u5728\u96c6\u7fa4 az1 \u8bbe\u7f6e\u5230\u96c6\u7fa4 az2 \u7684\u8def\u7531: kubectl ko nbctl lr-route-add ovn-cluster 10 .17.0.0/24 169 .254.100.31 \u5728\u96c6\u7fa4 az2 \u8bbe\u7f6e\u5230\u96c6\u7fa4 az1 \u7684\u8def\u7531: kubectl ko nbctl lr-route-add ovn-cluster 10 .16.0.0/24 169 .254.100.79 \u9ad8\u53ef\u7528 OVN-IC \u6570\u636e\u5e93\u90e8\u7f72 \u00b6 OVN-IC \u6570\u636e\u5e93\u4e4b\u95f4\u53ef\u4ee5\u901a\u8fc7 Raft \u534f\u8bae\u7ec4\u6210\u4e00\u4e2a\u9ad8\u53ef\u7528\u96c6\u7fa4\uff0c\u8be5\u90e8\u7f72\u6a21\u5f0f\u9700\u8981\u81f3\u5c11 3 \u4e2a\u8282\u70b9\u3002 \u9996\u5148\u5728\u7b2c\u4e00\u4e2a\u8282\u70b9\u4e0a\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\u7684 leader\u3002 \u90e8\u7f72 docker \u73af\u5883\u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.3\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5982\u679c\u662f\u90e8\u7f72 containerd \u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a \u5f53\u524d\u5bb9\u5668\u6240\u5728\u8282\u70b9 IP \u5730\u5740\u3002 NODE_IPS \uff1a \u8fd0\u884c OVN-IC \u6570\u636e\u5e93\u7684\u4e09\u4e2a\u8282\u70b9 IP \u5730\u5740\uff0c\u4f7f\u7528\u9017\u53f7\u8fdb\u884c\u5206\u9694\u3002 \u63a5\u4e0b\u6765\uff0c\u5728\u53e6\u5916\u4e24\u4e2a\u8282\u70b9\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684 follower\u3002 \u90e8\u7f72 docker \u73af\u5883\u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.2\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" -e LEADER_IP = \"192.168.65.3\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5982\u679c\u662f\u90e8\u7f72 containerd \u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.2 \"\" --env = \"LEADER_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a \u5f53\u524d\u5bb9\u5668\u6240\u5728\u8282\u70b9 IP \u5730\u5740\u3002 NODE_IPS \uff1a \u8fd0\u884c OVN-IC \u6570\u636e\u5e93\u7684\u4e09\u4e2a\u8282\u70b9 IP \u5730\u5740\uff0c\u4f7f\u7528\u9017\u53f7\u8fdb\u884c\u5206\u9694\u3002 LEADER_IP : \u8fd0\u884c OVN-IC \u6570\u636e\u5e93 leader \u8282\u70b9\u7684 IP \u5730\u5740\u3002 \u5728\u6bcf\u4e2a\u96c6\u7fa4\u521b\u5efa ovn-ic-config \u65f6\u6307\u5b9a\u591a\u4e2a OVN-IC \u6570\u636e\u5e93\u8282\u70b9\u5730\u5740\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3,192.168.65.2,192.168.65.1\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" \u624b\u52a8\u91cd\u7f6e \u00b6 \u5728\u4e00\u4e9b\u60c5\u51b5\u4e0b\uff0c\u7531\u4e8e\u914d\u7f6e\u9519\u8bef\u9700\u8981\u5bf9\u6574\u4e2a\u4e92\u8054\u914d\u7f6e\u8fdb\u884c\u6e05\u7406\uff0c\u53ef\u4ee5\u53c2\u8003\u4e0b\u9762\u7684\u6b65\u9aa4\u6e05\u7406\u73af\u5883\u3002 \u5220\u9664\u5f53\u524d\u7684 ovn-ic-config Configmap\uff1a kubectl -n kube-system delete cm ovn-ic-config \u5220\u9664 ts \u903b\u8f91\u4ea4\u6362\u673a\uff1a kubectl-ko nbctl ls-del ts \u5728\u5bf9\u7aef\u96c6\u7fa4\u91cd\u590d\u540c\u6837\u7684\u6b65\u9aa4\u3002 \u4fee\u6539 az-name \u00b6 \u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7 kubectl edit \u7684\u65b9\u5f0f\u5bf9 ovn-ic-config \u8fd9\u4e2a configmap \u4e2d\u7684 az-name \u5b57\u6bb5\u8fdb\u884c\u4fee\u6539\u3002 \u4f46\u662f\u9700\u8981\u5728\u6bcf\u4e2a ovn-cni pod \u4e0a\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u5426\u5219\u53ef\u80fd\u51fa\u73b0\u6700\u957f 10 \u5206\u949f\u7684\u8de8\u96c6\u7fa4\u7f51\u7edc\u4e2d\u65ad\u3002 ovn-appctl -t ovn-controller inc-engine/recompute \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-ovn-ic/#ovn-ic","text":"Kube-OVN \u652f\u6301\u901a\u8fc7 OVN-IC \u5c06\u4e24\u4e2a Kubernetes \u96c6\u7fa4 Pod \u7f51\u7edc\u6253\u901a\uff0c\u6253\u901a\u540e\u7684\u4e24\u4e2a\u96c6\u7fa4\u5185\u7684 Pod \u53ef\u4ee5\u901a\u8fc7 Pod IP \u8fdb\u884c\u76f4\u63a5\u901a\u4fe1\u3002 Kube-OVN \u4f7f\u7528\u96a7\u9053\u5bf9\u8de8\u96c6\u7fa4\u6d41\u91cf\u8fdb\u884c\u5c01\u88c5\uff0c\u4e24\u4e2a\u96c6\u7fa4\u4e4b\u95f4\u53ea\u8981\u5b58\u5728\u4e00\u7ec4 IP \u53ef\u8fbe\u7684\u673a\u5668\u5373\u53ef\u5b8c\u6210\u5bb9\u5668\u7f51\u7edc\u7684\u4e92\u901a\u3002 \u8be5\u6a21\u5f0f\u7684\u591a\u96c6\u7fa4\u4e92\u8054\u4e3a Overlay \u7f51\u7edc\u529f\u80fd\uff0cUnderlay \u7f51\u7edc\u5982\u679c\u60f3\u8981\u5b9e\u73b0\u96c6\u7fa4\u4e92\u8054\u9700\u8981\u5e95\u5c42\u57fa\u7840\u8bbe\u65bd\u505a\u7f51\u7edc\u6253\u901a\u3002","title":"\u4f7f\u7528 OVN-IC \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-ovn-ic/#_1","text":"\u81ea\u52a8\u4e92\u8054\u6a21\u5f0f\u4e0b\u4e0d\u540c\u96c6\u7fa4\u7684\u5b50\u7f51 CIDR \u4e0d\u80fd\u76f8\u4e92\u91cd\u53e0\uff0c\u9ed8\u8ba4\u5b50\u7f51\u9700\u5728\u5b89\u88c5\u65f6\u914d\u7f6e\u4e3a\u4e0d\u91cd\u53e0\u7684\u7f51\u6bb5\u3002\u82e5\u5b58\u5728\u91cd\u53e0\u9700\u53c2\u8003\u540e\u7eed\u624b\u52a8\u4e92\u8054\u8fc7\u7a0b\uff0c\u53ea\u80fd\u5c06\u4e0d\u91cd\u53e0\u7f51\u6bb5\u6253\u901a\u3002 \u9700\u8981\u5b58\u5728\u4e00\u7ec4\u673a\u5668\u53ef\u4ee5\u88ab\u6bcf\u4e2a\u96c6\u7fa4\u7684 kube-ovn-controller \u901a\u8fc7 IP \u8bbf\u95ee\uff0c\u7528\u6765\u90e8\u7f72\u8de8\u96c6\u7fa4\u4e92\u8054\u7684\u63a7\u5236\u5668\u3002 \u6bcf\u4e2a\u96c6\u7fa4\u9700\u8981\u6709\u4e00\u7ec4\u53ef\u4ee5\u901a\u8fc7 IP \u8fdb\u884c\u8de8\u96c6\u7fa4\u4e92\u8bbf\u7684\u673a\u5668\u4f5c\u4e3a\u4e4b\u540e\u7684\u7f51\u5173\u8282\u70b9\u3002 \u8be5\u529f\u80fd\u53ea\u5bf9\u9ed8\u8ba4 VPC \u751f\u6548\uff0c\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u65e0\u6cd5\u4f7f\u7528\u4e92\u8054\u529f\u80fd\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/with-ovn-ic/#ovn-ic_1","text":"\u5728\u6bcf\u4e2a\u96c6\u7fa4 kube-ovn-controller \u53ef\u901a\u8fc7 IP \u8bbf\u95ee\u7684\u673a\u5668\u4e0a\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\uff0c\u8be5\u8282\u70b9\u5c06\u4fdd\u5b58\u5404\u4e2a\u96c6\u7fa4\u540c\u6b65\u4e0a\u6765\u7684\u7f51\u7edc\u914d\u7f6e\u4fe1\u606f\u3002 \u90e8\u7f72 docker \u7684\u73af\u5883\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5bf9\u4e8e\u90e8\u7f72 containerd \u53d6\u4ee3 docker \u7684\u73af\u5883\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh","title":"\u90e8\u7f72\u5355\u8282\u70b9 OVN-IC \u6570\u636e\u5e93"},{"location":"advance/with-ovn-ic/#_2","text":"\u5728\u81ea\u52a8\u8def\u7531\u8bbe\u7f6e\u4e0b\uff0c\u6bcf\u4e2a\u96c6\u7fa4\u4f1a\u5c06\u81ea\u5df1\u9ed8\u8ba4 VPC \u4e0b Subnet \u7684 CIDR \u4fe1\u606f\u540c\u6b65\u7ed9 OVN-IC \uff0c\u56e0\u6b64\u8981\u786e\u4fdd\u4e24\u4e2a\u96c6\u7fa4\u7684 Subnet CIDR \u4e0d\u5b58\u5728\u91cd\u53e0\u3002 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : \u662f\u5426\u5f00\u542f\u96c6\u7fa4\u4e92\u8054\u3002 az-name : \u533a\u5206\u4e0d\u540c\u96c6\u7fa4\u7684\u96c6\u7fa4\u540d\u79f0\uff0c\u6bcf\u4e2a\u4e92\u8054\u96c6\u7fa4\u9700\u4e0d\u540c\u3002 ic-db-host : \u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684\u8282\u70b9\u5730\u5740\u3002 ic-nb-port : OVN-IC \u5317\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6645\u3002 ic-sb-port : OVN-IC \u5357\u5411\u6570\u636e\u5e93\u7aef\u53e3\uff0c\u9ed8\u8ba4\u4e3a 6646\u3002 gw-nodes : \u96c6\u7fa4\u4e92\u8054\u4e2d\u627f\u62c5\u7f51\u5173\u5de5\u4f5c\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 auto-route : \u662f\u5426\u81ea\u52a8\u5bf9\u5916\u53d1\u5e03\u548c\u5b66\u4e60\u8def\u7531\u3002 \u6ce8\u610f\uff1a \u4e3a\u4e86\u4fdd\u8bc1\u64cd\u4f5c\u7684\u6b63\u786e\u6027\uff0c ovn-ic-config \u8fd9\u4e2a ConfigMap \u4e0d\u5141\u8bb8\u4fee\u6539\u3002\u5982\u6709\u53c2\u6570\u9700\u8981\u53d8\u66f4\uff0c\u8bf7\u5220\u9664\u8be5 ConfigMap\uff0c\u4fee\u6539\u540e\u518d\u5e94\u7528\u6b64 ConfigMap\u3002 \u5728 ovn-ic \u5bb9\u5668\u5185\u901a\u8fc7\u4e0b\u9762\u7684\u547d\u4ee4\u67e5\u770b\u662f\u5426\u5df2\u5efa\u7acb\u4e92\u8054\u903b\u8f91\u4ea4\u6362\u673a ts \uff1a # ovn-ic-sbctl show availability-zone az1 gateway deee03e0-af16-4f45-91e9-b50c3960f809 hostname: az1-gw type: geneve ip: 192 .168.42.145 port ts-az1 transit switch: ts address: [ \"00:00:00:50:AC:8C 169.254.100.45/24\" ] availability-zone az2 gateway e94cc831-8143-40e3-a478-90352773327b hostname: az2-gw type: geneve ip: 192 .168.42.149 port ts-az2 transit switch: ts address: [ \"00:00:00:07:4A:59 169.254.100.63/24\" ] \u5728\u6bcf\u4e2a\u96c6\u7fa4\u89c2\u5bdf\u903b\u8f91\u8def\u7531\u662f\u5426\u6709\u5b66\u4e60\u5230\u7684\u5bf9\u7aef\u8def\u7531\uff1a # kubectl ko nbctl lr-route-list ovn-cluster IPv4 Routes 10 .42.1.1 169 .254.100.45 dst-ip ( learned ) 10 .42.1.3 100 .64.0.2 dst-ip 10 .16.0.2 100 .64.0.2 src-ip 10 .16.0.3 100 .64.0.2 src-ip 10 .16.0.4 100 .64.0.2 src-ip 10 .16.0.6 100 .64.0.2 src-ip 10 .17.0.0/16 169 .254.100.45 dst-ip ( learned ) 100 .65.0.0/16 169 .254.100.45 dst-ip ( learned ) \u63a5\u4e0b\u6765\u53ef\u4ee5\u5c1d\u8bd5\u5728\u96c6\u7fa4 1 \u5185\u7684\u4e00\u4e2a Pod \u5185\u76f4\u63a5 ping \u96c6\u7fa4 2 \u5185\u7684\u4e00\u4e2a Pod IP \u89c2\u5bdf\u662f\u5426\u53ef\u4ee5\u8054\u901a\u3002 \u5bf9\u4e8e\u67d0\u4e2a\u4e0d\u60f3\u5bf9\u5916\u81ea\u52a8\u53d1\u5e03\u8def\u7531\u7684\u5b50\u7f51\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 Subnet \u91cc\u7684 disableInterConnection \u6765\u7981\u6b62\u8def\u7531\u5e7f\u64ad\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : no-advertise spec : cidrBlock : 10.199.0.0/16 disableInterConnection : true","title":"\u81ea\u52a8\u8def\u7531\u8bbe\u7f6e"},{"location":"advance/with-ovn-ic/#_3","text":"\u5bf9\u4e8e\u96c6\u7fa4\u95f4\u5b58\u5728\u91cd\u53e0 CIDR \u53ea\u5e0c\u671b\u505a\u90e8\u5206\u5b50\u7f51\u6253\u901a\u7684\u60c5\u51b5\uff0c\u53ef\u4ee5\u901a\u8fc7\u4e0b\u9762\u7684\u6b65\u9aa4\u624b\u52a8\u53d1\u5e03\u5b50\u7f51\u8def\u7531\u3002 \u5728 kube-system Namespace \u4e0b\u521b\u5efa ovn-ic-config ConfigMap\uff0c\u5e76\u5c06 auto-route \u8bbe\u7f6e\u4e3a false \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"false\" \u5728\u6bcf\u4e2a\u96c6\u7fa4\u5206\u522b\u67e5\u770b\u8fdc\u7aef\u903b\u8f91\u7aef\u53e3\u7684\u5730\u5740\uff0c\u7528\u4e8e\u4e4b\u540e\u624b\u52a8\u914d\u7f6e\u8def\u7531\uff1a [ root@az1 ~ ] # kubectl ko nbctl show switch a391d3a1-14a0-4841-9836-4bd930c447fb ( ts ) port ts-az1 type: router router-port: az1-ts port ts-az2 type: remote addresses: [ \"00:00:00:4B:E2:9F 169.254.100.31/24\" ] [ root@az2 ~ ] # kubectl ko nbctl show switch da6138b8-de81-4908-abf9-b2224ec4edf3 ( ts ) port ts-az2 type: router router-port: az2-ts port ts-az1 type: remote addresses: [ \"00:00:00:FB:2A:F7 169.254.100.79/24\" ] \u7531\u4e0a\u8f93\u51fa\u53ef\u77e5\uff0c\u96c6\u7fa4 az1 \u5230 \u96c6\u7fa4 az2 \u7684\u8fdc\u7aef\u5730\u5740\u4e3a 169.254.100.31 \uff0c az2 \u5230 az1 \u7684\u8fdc\u7aef\u5730\u5740\u4e3a 169.254.100.79 \u3002 \u4e0b\u9762\u624b\u52a8\u8bbe\u7f6e\u8def\u7531\uff0c\u5728\u8be5\u4f8b\u5b50\u4e2d\uff0c\u96c6\u7fa4 az1 \u5185\u7684\u5b50\u7f51 CIDR \u4e3a 10.16.0.0/24 \uff0c\u96c6\u7fa4 az2 \u5185\u7684\u5b50\u7f51 CIDR \u4e3a 10.17.0.0/24 \u3002 \u5728\u96c6\u7fa4 az1 \u8bbe\u7f6e\u5230\u96c6\u7fa4 az2 \u7684\u8def\u7531: kubectl ko nbctl lr-route-add ovn-cluster 10 .17.0.0/24 169 .254.100.31 \u5728\u96c6\u7fa4 az2 \u8bbe\u7f6e\u5230\u96c6\u7fa4 az1 \u7684\u8def\u7531: kubectl ko nbctl lr-route-add ovn-cluster 10 .16.0.0/24 169 .254.100.79","title":"\u624b\u52a8\u8def\u7531\u8bbe\u7f6e"},{"location":"advance/with-ovn-ic/#ovn-ic_2","text":"OVN-IC \u6570\u636e\u5e93\u4e4b\u95f4\u53ef\u4ee5\u901a\u8fc7 Raft \u534f\u8bae\u7ec4\u6210\u4e00\u4e2a\u9ad8\u53ef\u7528\u96c6\u7fa4\uff0c\u8be5\u90e8\u7f72\u6a21\u5f0f\u9700\u8981\u81f3\u5c11 3 \u4e2a\u8282\u70b9\u3002 \u9996\u5148\u5728\u7b2c\u4e00\u4e2a\u8282\u70b9\u4e0a\u542f\u52a8 OVN-IC \u6570\u636e\u5e93\u7684 leader\u3002 \u90e8\u7f72 docker \u73af\u5883\u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.3\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5982\u679c\u662f\u90e8\u7f72 containerd \u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a \u5f53\u524d\u5bb9\u5668\u6240\u5728\u8282\u70b9 IP \u5730\u5740\u3002 NODE_IPS \uff1a \u8fd0\u884c OVN-IC \u6570\u636e\u5e93\u7684\u4e09\u4e2a\u8282\u70b9 IP \u5730\u5740\uff0c\u4f7f\u7528\u9017\u53f7\u8fdb\u884c\u5206\u9694\u3002 \u63a5\u4e0b\u6765\uff0c\u5728\u53e6\u5916\u4e24\u4e2a\u8282\u70b9\u90e8\u7f72 OVN-IC \u6570\u636e\u5e93\u7684 follower\u3002 \u90e8\u7f72 docker \u73af\u5883\u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.2\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" -e LEADER_IP = \"192.168.65.3\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh \u5982\u679c\u662f\u90e8\u7f72 containerd \u7684\u7528\u6237\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\uff1a ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.2 \"\" --env = \"LEADER_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a \u5f53\u524d\u5bb9\u5668\u6240\u5728\u8282\u70b9 IP \u5730\u5740\u3002 NODE_IPS \uff1a \u8fd0\u884c OVN-IC \u6570\u636e\u5e93\u7684\u4e09\u4e2a\u8282\u70b9 IP \u5730\u5740\uff0c\u4f7f\u7528\u9017\u53f7\u8fdb\u884c\u5206\u9694\u3002 LEADER_IP : \u8fd0\u884c OVN-IC \u6570\u636e\u5e93 leader \u8282\u70b9\u7684 IP \u5730\u5740\u3002 \u5728\u6bcf\u4e2a\u96c6\u7fa4\u521b\u5efa ovn-ic-config \u65f6\u6307\u5b9a\u591a\u4e2a OVN-IC \u6570\u636e\u5e93\u8282\u70b9\u5730\u5740\uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3,192.168.65.2,192.168.65.1\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\"","title":"\u9ad8\u53ef\u7528 OVN-IC \u6570\u636e\u5e93\u90e8\u7f72"},{"location":"advance/with-ovn-ic/#_4","text":"\u5728\u4e00\u4e9b\u60c5\u51b5\u4e0b\uff0c\u7531\u4e8e\u914d\u7f6e\u9519\u8bef\u9700\u8981\u5bf9\u6574\u4e2a\u4e92\u8054\u914d\u7f6e\u8fdb\u884c\u6e05\u7406\uff0c\u53ef\u4ee5\u53c2\u8003\u4e0b\u9762\u7684\u6b65\u9aa4\u6e05\u7406\u73af\u5883\u3002 \u5220\u9664\u5f53\u524d\u7684 ovn-ic-config Configmap\uff1a kubectl -n kube-system delete cm ovn-ic-config \u5220\u9664 ts \u903b\u8f91\u4ea4\u6362\u673a\uff1a kubectl-ko nbctl ls-del ts \u5728\u5bf9\u7aef\u96c6\u7fa4\u91cd\u590d\u540c\u6837\u7684\u6b65\u9aa4\u3002","title":"\u624b\u52a8\u91cd\u7f6e"},{"location":"advance/with-ovn-ic/#az-name","text":"\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7 kubectl edit \u7684\u65b9\u5f0f\u5bf9 ovn-ic-config \u8fd9\u4e2a configmap \u4e2d\u7684 az-name \u5b57\u6bb5\u8fdb\u884c\u4fee\u6539\u3002 \u4f46\u662f\u9700\u8981\u5728\u6bcf\u4e2a ovn-cni pod \u4e0a\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff0c\u5426\u5219\u53ef\u80fd\u51fa\u73b0\u6700\u957f 10 \u5206\u949f\u7684\u8de8\u96c6\u7fa4\u7f51\u7edc\u4e2d\u65ad\u3002 ovn-appctl -t ovn-controller inc-engine/recompute \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4fee\u6539 az-name"},{"location":"advance/with-submariner/","text":"\u4f7f\u7528 Submariner \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054 \u00b6 Submariner \u4f5c\u4e3a\u53ef\u4ee5\u6253\u901a\u591a\u4e2a Kubernetes \u96c6\u7fa4 Pod \u548c Service \u7f51\u7edc\u7684\u5f00\u6e90\u7f51\u7edc\u7ec4\u4ef6\uff0c\u80fd\u591f\u5e2e\u52a9 Kube-OVN \u5b9e\u73b0\u591a\u96c6\u7fa4\u4e92\u8054\u3002 \u76f8\u6bd4\u901a\u8fc7 OVN-IC \u6253\u901a\u591a\u96c6\u7fa4\u7f51\u7edc\u7684\u65b9\u5f0f\uff0cSubmariner \u53ef\u4ee5\u6253\u901a Kube-OVN \u548c\u975e Kube-OVN \u7684\u96c6\u7fa4\u7f51\u7edc\uff0c\u5e76 \u80fd\u63d0\u4f9b Service \u7684\u8de8\u96c6\u7fa4\u80fd\u529b\u3002\u4f46\u662f Submariner \u76ee\u524d\u53ea\u80fd\u5b9e\u73b0\u9ed8\u8ba4\u5b50\u7f51\u7684\u6253\u901a\uff0c\u65e0\u6cd5\u5b9e\u73b0\u591a\u5b50\u7f51\u9009\u62e9\u6027\u6253\u901a\u3002 \u524d\u63d0\u6761\u4ef6 \u00b6 \u4e24\u4e2a\u96c6\u7fa4\u7684 Service CIDR \u548c\u9ed8\u8ba4\u5b50\u7f51\u7684 CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 \u90e8\u7f72 Submariner \u00b6 \u4e0b\u8f7d subctl \u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u5e76\u90e8\u7f72\u5230\u76f8\u5e94\u8def\u5f84\uff1a curl -Ls https://get.submariner.io | bash export PATH = $PATH :~/.local/bin echo export PATH = \\$ PATH:~/.local/bin >> ~/.profile \u5207\u6362 kubeconfig \u81f3\u5e0c\u671b\u90e8\u7f72 submariner-broker \u7684\u96c6\u7fa4\u8fdb\u884c\u90e8\u7f72\uff1a subctl deploy-broker \u5728\u672c\u6587\u6863\u4e2d cluster0 \u7684\u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e3a 10.16.0.0/16 \uff0cjoin \u5b50\u7f51 CIDR \u4e3a 100.64.0.0/16 \uff0c cluster1 \u7684\u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e3a 11.16.0.0/16 \uff0cjoin \u5b50\u7f51 CIDR \u4e3a '100.68.0.0/16'\u3002 \u5207\u6362 kubeconfig \u81f3 cluster0 \u6ce8\u518c\u96c6\u7fa4\u81f3 broker\uff0c\u5e76\u6ce8\u518c\u7f51\u5173\u8282\u70b9: subctl join broker-info.subm --clusterid cluster0 --clustercidr 100 .64.0.0/16,10.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster0 submariner.io/gateway = true \u5207\u6362 kubeconfig \u81f3 cluster1 \u6ce8\u518c\u96c6\u7fa4\u81f3 broker\uff0c\u5e76\u6ce8\u518c\u7f51\u5173\u8282\u70b9: subctl join broker-info.subm --clusterid cluster1 --clustercidr 100 .68.0.0/16,11.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster1 submariner.io/gateway = true \u5982\u679c\u6267\u884c join \u547d\u4ee4\u4e4b\u540e\u6ca1\u6709\u65b0\u7684 gateway, routeagent pod \u51fa\u73b0\u7684\u8bdd, \u8bf7\u4e3a submariner-operator \u8fd9\u4e2a clusterrole \u589e\u52a0\u4ee5\u4e0b\u6743\u9650: - apiGroups : - \"apps\" resources : - daemonsets verbs : - create - get - list - watch - update \u5bf9\u4e8e\u591a\u8282\u70b9\u7684\u96c6\u7fa4\uff0c\u9700\u8981\u5c06\u9ed8\u8ba4\u7684 subnet ovn-default \u7684\u7f51\u5173\u914d\u7f6e\u6539\u4e3a centralized \u3002\u4e3a submariner \u914d\u7f6e\u7684 gateway \u8282\u70b9\u9700\u8981\u548c subnet \u8282\u70b9\u5b8c\u5168\u76f8\u540c\u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728\u4e24\u4e2a\u96c6\u7fa4\u5185\u5206\u522b\u542f\u52a8 Pod \u5e76\u5c1d\u8bd5\u4f7f\u7528 IP \u8fdb\u884c\u76f8\u4e92\u8bbf\u95ee\u3002 \u5982\u679c\u51fa\u73b0\u7f51\u7edc\u4e92\u901a\u95ee\u9898\u53ef\u901a\u8fc7 subctl \u547d\u4ee4\u8fdb\u884c\u8bca\u65ad\uff1a subctl show all subctl diagnose all \u66f4\u591a Submariner \u76f8\u5173\u64cd\u4f5c\u8bf7\u67e5\u770b Submariner \u7528\u6237\u624b\u518c \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 Submariner \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-submariner/#submariner","text":"Submariner \u4f5c\u4e3a\u53ef\u4ee5\u6253\u901a\u591a\u4e2a Kubernetes \u96c6\u7fa4 Pod \u548c Service \u7f51\u7edc\u7684\u5f00\u6e90\u7f51\u7edc\u7ec4\u4ef6\uff0c\u80fd\u591f\u5e2e\u52a9 Kube-OVN \u5b9e\u73b0\u591a\u96c6\u7fa4\u4e92\u8054\u3002 \u76f8\u6bd4\u901a\u8fc7 OVN-IC \u6253\u901a\u591a\u96c6\u7fa4\u7f51\u7edc\u7684\u65b9\u5f0f\uff0cSubmariner \u53ef\u4ee5\u6253\u901a Kube-OVN \u548c\u975e Kube-OVN \u7684\u96c6\u7fa4\u7f51\u7edc\uff0c\u5e76 \u80fd\u63d0\u4f9b Service \u7684\u8de8\u96c6\u7fa4\u80fd\u529b\u3002\u4f46\u662f Submariner \u76ee\u524d\u53ea\u80fd\u5b9e\u73b0\u9ed8\u8ba4\u5b50\u7f51\u7684\u6253\u901a\uff0c\u65e0\u6cd5\u5b9e\u73b0\u591a\u5b50\u7f51\u9009\u62e9\u6027\u6253\u901a\u3002","title":"\u4f7f\u7528 Submariner \u8fdb\u884c\u591a\u96c6\u7fa4\u4e92\u8054"},{"location":"advance/with-submariner/#_1","text":"\u4e24\u4e2a\u96c6\u7fa4\u7684 Service CIDR \u548c\u9ed8\u8ba4\u5b50\u7f51\u7684 CIDR \u4e0d\u80fd\u91cd\u53e0\u3002","title":"\u524d\u63d0\u6761\u4ef6"},{"location":"advance/with-submariner/#submariner_1","text":"\u4e0b\u8f7d subctl \u4e8c\u8fdb\u5236\u6587\u4ef6\uff0c\u5e76\u90e8\u7f72\u5230\u76f8\u5e94\u8def\u5f84\uff1a curl -Ls https://get.submariner.io | bash export PATH = $PATH :~/.local/bin echo export PATH = \\$ PATH:~/.local/bin >> ~/.profile \u5207\u6362 kubeconfig \u81f3\u5e0c\u671b\u90e8\u7f72 submariner-broker \u7684\u96c6\u7fa4\u8fdb\u884c\u90e8\u7f72\uff1a subctl deploy-broker \u5728\u672c\u6587\u6863\u4e2d cluster0 \u7684\u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e3a 10.16.0.0/16 \uff0cjoin \u5b50\u7f51 CIDR \u4e3a 100.64.0.0/16 \uff0c cluster1 \u7684\u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e3a 11.16.0.0/16 \uff0cjoin \u5b50\u7f51 CIDR \u4e3a '100.68.0.0/16'\u3002 \u5207\u6362 kubeconfig \u81f3 cluster0 \u6ce8\u518c\u96c6\u7fa4\u81f3 broker\uff0c\u5e76\u6ce8\u518c\u7f51\u5173\u8282\u70b9: subctl join broker-info.subm --clusterid cluster0 --clustercidr 100 .64.0.0/16,10.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster0 submariner.io/gateway = true \u5207\u6362 kubeconfig \u81f3 cluster1 \u6ce8\u518c\u96c6\u7fa4\u81f3 broker\uff0c\u5e76\u6ce8\u518c\u7f51\u5173\u8282\u70b9: subctl join broker-info.subm --clusterid cluster1 --clustercidr 100 .68.0.0/16,11.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster1 submariner.io/gateway = true \u5982\u679c\u6267\u884c join \u547d\u4ee4\u4e4b\u540e\u6ca1\u6709\u65b0\u7684 gateway, routeagent pod \u51fa\u73b0\u7684\u8bdd, \u8bf7\u4e3a submariner-operator \u8fd9\u4e2a clusterrole \u589e\u52a0\u4ee5\u4e0b\u6743\u9650: - apiGroups : - \"apps\" resources : - daemonsets verbs : - create - get - list - watch - update \u5bf9\u4e8e\u591a\u8282\u70b9\u7684\u96c6\u7fa4\uff0c\u9700\u8981\u5c06\u9ed8\u8ba4\u7684 subnet ovn-default \u7684\u7f51\u5173\u914d\u7f6e\u6539\u4e3a centralized \u3002\u4e3a submariner \u914d\u7f6e\u7684 gateway \u8282\u70b9\u9700\u8981\u548c subnet \u8282\u70b9\u5b8c\u5168\u76f8\u540c\u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728\u4e24\u4e2a\u96c6\u7fa4\u5185\u5206\u522b\u542f\u52a8 Pod \u5e76\u5c1d\u8bd5\u4f7f\u7528 IP \u8fdb\u884c\u76f8\u4e92\u8bbf\u95ee\u3002 \u5982\u679c\u51fa\u73b0\u7f51\u7edc\u4e92\u901a\u95ee\u9898\u53ef\u901a\u8fc7 subctl \u547d\u4ee4\u8fdb\u884c\u8bca\u65ad\uff1a subctl show all subctl diagnose all \u66f4\u591a Submariner \u76f8\u5173\u64cd\u4f5c\u8bf7\u67e5\u770b Submariner \u7528\u6237\u624b\u518c \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u90e8\u7f72 Submariner"},{"location":"guide/custom-routes/","text":"\u81ea\u5b9a\u4e49\u8def\u7531 \u00b6 \u53ef\u4ee5\u5728\u521b\u5efa Pod \u65f6\u901a\u8fc7 Annotations \u6765\u6307\u5b9a\u9700\u8981\u914d\u7f6e\u7684\u8def\u7531\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : v1 kind : Pod metadata : name : custom-routes annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine dst \u5b57\u6bb5\u4e3a\u7a7a\u8868\u793a\u4fee\u6539\u9ed8\u8ba4\u8def\u7531\u3002 \u5982\u679c\u5de5\u4f5c\u8d1f\u8f7d\u4e3a Deployment\u3001DaemonSet \u6216 StatefulSet\uff0c\u5bf9\u5e94\u7684 Annotation \u9700\u8981\u914d\u7f6e\u5728\u8d44\u6e90\u7684 .spec.template.metadata.annotations \u4e2d\uff0c\u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : apps/v1 kind : Deployment metadata : name : custom-routes labels : app : nginx spec : replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u81ea\u5b9a\u4e49\u8def\u7531"},{"location":"guide/custom-routes/#_1","text":"\u53ef\u4ee5\u5728\u521b\u5efa Pod \u65f6\u901a\u8fc7 Annotations \u6765\u6307\u5b9a\u9700\u8981\u914d\u7f6e\u7684\u8def\u7531\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : v1 kind : Pod metadata : name : custom-routes annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine dst \u5b57\u6bb5\u4e3a\u7a7a\u8868\u793a\u4fee\u6539\u9ed8\u8ba4\u8def\u7531\u3002 \u5982\u679c\u5de5\u4f5c\u8d1f\u8f7d\u4e3a Deployment\u3001DaemonSet \u6216 StatefulSet\uff0c\u5bf9\u5e94\u7684 Annotation \u9700\u8981\u914d\u7f6e\u5728\u8d44\u6e90\u7684 .spec.template.metadata.annotations \u4e2d\uff0c\u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : apps/v1 kind : Deployment metadata : name : custom-routes labels : app : nginx spec : replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u81ea\u5b9a\u4e49\u8def\u7531"},{"location":"guide/dual-stack/","text":"\u53cc\u6808\u4f7f\u7528 \u00b6 Kube-OVN \u4e2d\u4e0d\u540c\u7684\u5b50\u7f51\u53ef\u4ee5\u652f\u6301\u4e0d\u540c\u7684\u534f\u8bae\uff0c\u4e00\u4e2a\u96c6\u7fa4\u5185\u53ef\u4ee5\u540c\u65f6\u5b58\u5728 IPv4\uff0cIPv6 \u548c\u53cc\u6808\u7c7b\u578b\u7684\u5b50\u7f51\u3002 \u6211\u4eec\u63a8\u8350\u4e00\u4e2a\u96c6\u7fa4\u5185\u4f7f\u7528\u7edf\u4e00\u7684\u534f\u8bae\u7c7b\u578b\u4ee5\u7b80\u5316\u4f7f\u7528\u548c\u7ef4\u62a4\u3002 \u4e3a\u4e86\u652f\u6301\u53cc\u6808\uff0c\u9700\u8981\u4e3b\u673a\u7f51\u7edc\u6ee1\u8db3\u53cc\u6808\u8981\u6c42\uff0c\u540c\u65f6\u9700\u8981\u5bf9 Kubernetes \u76f8\u5173\u53c2\u6570\u505a\u8c03\u6574\uff0c \u8bf7\u53c2\u8003 Kubernetes \u7684 \u53cc\u6808\u5b98\u65b9\u6307\u5bfc \u3002 \u521b\u5efa\u53cc\u6808\u5b50\u7f51 \u00b6 \u5728\u914d\u7f6e\u53cc\u6808\u65f6\uff0c\u53ea\u9700\u8981\u8bbe\u7f6e\u5bf9\u5e94\u5b50\u7f51 CIDR \u683c\u5f0f\u4e3a cidr=<IPv4 CIDR>,<IPv6 CIDR> \u5373\u53ef\u3002 CIDR \u987a\u5e8f\u8981\u6c42 IPv4 \u5728\u524d\uff0cIPv6 \u5728\u540e\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-test spec : cidrBlock : 10.16.0.0/16,fd00:10:16::/64 excludeIps : - 10.16.0.1 - fd00:10:16::1 gateway : 10.16.0.1,fd00:10:16::1 \u5982\u679c\u9700\u8981\u5728\u5b89\u88c5\u65f6\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u53cc\u6808\uff0c\u9700\u8981\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u4fee\u6539\u5982\u4e0b\u53c2\u6570\uff1a POD_CIDR = \"10.16.0.0/16,fd00:10:16::/64\" JOIN_CIDR = \"100.64.0.0/16,fd00:100:64::/64\" \u67e5\u770b Pod \u5730\u5740 \u00b6 \u914d\u7f6e\u53cc\u6808\u7f51\u7edc\u7684 Pod \u5c06\u4f1a\u4ece\u8be5\u5b50\u7f51\u540c\u65f6\u5206\u914d IPv4 \u548c IPv6 \u7684\u5730\u5740\uff0c\u5206\u914d\u7ed3\u679c\u4f1a\u663e\u793a\u5728 Pod \u7684 annotation \u4e2d: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/allocated : \"true\" ovn.kubernetes.io/cidr : 10.16.0.0/16,fd00:10:16::/64 ovn.kubernetes.io/gateway : 10.16.0.1,fd00:10:16::1 ovn.kubernetes.io/ip_address : 10.16.0.9,fd00:10:16::9 ovn.kubernetes.io/logical_switch : ovn-default ovn.kubernetes.io/mac_address : 00:00:00:14:88:09 ovn.kubernetes.io/network_types : geneve ovn.kubernetes.io/routed : \"true\" ... podIP : 10.16.0.9 podIPs : - ip : 10.16.0.9 - ip : fd00:10:16::9 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u53cc\u6808\u4f7f\u7528"},{"location":"guide/dual-stack/#_1","text":"Kube-OVN \u4e2d\u4e0d\u540c\u7684\u5b50\u7f51\u53ef\u4ee5\u652f\u6301\u4e0d\u540c\u7684\u534f\u8bae\uff0c\u4e00\u4e2a\u96c6\u7fa4\u5185\u53ef\u4ee5\u540c\u65f6\u5b58\u5728 IPv4\uff0cIPv6 \u548c\u53cc\u6808\u7c7b\u578b\u7684\u5b50\u7f51\u3002 \u6211\u4eec\u63a8\u8350\u4e00\u4e2a\u96c6\u7fa4\u5185\u4f7f\u7528\u7edf\u4e00\u7684\u534f\u8bae\u7c7b\u578b\u4ee5\u7b80\u5316\u4f7f\u7528\u548c\u7ef4\u62a4\u3002 \u4e3a\u4e86\u652f\u6301\u53cc\u6808\uff0c\u9700\u8981\u4e3b\u673a\u7f51\u7edc\u6ee1\u8db3\u53cc\u6808\u8981\u6c42\uff0c\u540c\u65f6\u9700\u8981\u5bf9 Kubernetes \u76f8\u5173\u53c2\u6570\u505a\u8c03\u6574\uff0c \u8bf7\u53c2\u8003 Kubernetes \u7684 \u53cc\u6808\u5b98\u65b9\u6307\u5bfc \u3002","title":"\u53cc\u6808\u4f7f\u7528"},{"location":"guide/dual-stack/#_2","text":"\u5728\u914d\u7f6e\u53cc\u6808\u65f6\uff0c\u53ea\u9700\u8981\u8bbe\u7f6e\u5bf9\u5e94\u5b50\u7f51 CIDR \u683c\u5f0f\u4e3a cidr=<IPv4 CIDR>,<IPv6 CIDR> \u5373\u53ef\u3002 CIDR \u987a\u5e8f\u8981\u6c42 IPv4 \u5728\u524d\uff0cIPv6 \u5728\u540e\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-test spec : cidrBlock : 10.16.0.0/16,fd00:10:16::/64 excludeIps : - 10.16.0.1 - fd00:10:16::1 gateway : 10.16.0.1,fd00:10:16::1 \u5982\u679c\u9700\u8981\u5728\u5b89\u88c5\u65f6\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u53cc\u6808\uff0c\u9700\u8981\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u4fee\u6539\u5982\u4e0b\u53c2\u6570\uff1a POD_CIDR = \"10.16.0.0/16,fd00:10:16::/64\" JOIN_CIDR = \"100.64.0.0/16,fd00:100:64::/64\"","title":"\u521b\u5efa\u53cc\u6808\u5b50\u7f51"},{"location":"guide/dual-stack/#pod","text":"\u914d\u7f6e\u53cc\u6808\u7f51\u7edc\u7684 Pod \u5c06\u4f1a\u4ece\u8be5\u5b50\u7f51\u540c\u65f6\u5206\u914d IPv4 \u548c IPv6 \u7684\u5730\u5740\uff0c\u5206\u914d\u7ed3\u679c\u4f1a\u663e\u793a\u5728 Pod \u7684 annotation \u4e2d: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/allocated : \"true\" ovn.kubernetes.io/cidr : 10.16.0.0/16,fd00:10:16::/64 ovn.kubernetes.io/gateway : 10.16.0.1,fd00:10:16::1 ovn.kubernetes.io/ip_address : 10.16.0.9,fd00:10:16::9 ovn.kubernetes.io/logical_switch : ovn-default ovn.kubernetes.io/mac_address : 00:00:00:14:88:09 ovn.kubernetes.io/network_types : geneve ovn.kubernetes.io/routed : \"true\" ... podIP : 10.16.0.9 podIPs : - ip : 10.16.0.9 - ip : fd00:10:16::9 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u67e5\u770b Pod \u5730\u5740"},{"location":"guide/eip-snat/","text":"EIP \u548c SNAT \u914d\u7f6e \u00b6 \u8be5\u914d\u7f6e\u9488\u5bf9\u9ed8\u8ba4 VPC \u4e0b\u7684\u7f51\u7edc\uff0c\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u8bf7\u53c2\u8003 VPC \u7f51\u5173 Kube-OVN \u652f\u6301\u5229\u7528 OVN \u4e2d\u7684 L3 Gateway \u529f\u80fd\u6765\u5b9e\u73b0 Pod \u7ea7\u522b\u7684 SNAT \u548c EIP \u529f\u80fd\u3002 \u901a\u8fc7\u4f7f\u7528 SNAT\uff0c\u4e00\u7ec4 Pod \u53ef\u4ee5\u5171\u4eab\u4e00\u4e2a IP \u5730\u5740\u5bf9\u5916\u8fdb\u884c\u8bbf\u95ee\u3002 \u901a\u8fc7 EIP \u7684\u529f\u80fd\uff0c\u4e00\u4e2a Pod \u53ef\u4ee5\u76f4\u63a5\u548c\u4e00\u4e2a\u5916\u90e8 IP \u5173\u8054\uff0c \u5916\u90e8\u670d\u52a1\u53ef\u4ee5\u901a\u8fc7 EIP \u76f4\u63a5\u8bbf\u95ee Pod\uff0cPod \u4e5f\u5c06\u901a\u8fc7\u8fd9\u4e2a EIP \u8bbf\u95ee\u5916\u90e8\u670d\u52a1\u3002 \u51c6\u5907\u5de5\u4f5c \u00b6 \u4e3a\u4e86\u4f7f\u7528 OVN \u7684 L3 Gateway \u80fd\u529b\uff0c\u5fc5\u987b\u5c06\u4e00\u4e2a\u5355\u72ec\u7684\u7f51\u5361\u63a5\u5165 OVS \u7f51\u6865\u4e2d\u8fdb\u884c Overlay \u548c Underlay \u7f51\u7edc\u7684\u6253\u901a\uff0c \u4e3b\u673a\u5fc5\u987b\u6709\u5176\u4ed6\u7684\u7f51\u5361\u7528\u4e8e\u8fd0\u7ef4\u7ba1\u7406\u3002 \u7531\u4e8e\u7ecf\u8fc7 NAT \u540e\u7684\u6570\u636e\u5305\u4f1a\u76f4\u63a5\u8fdb\u5165 Underlay \u7f51\u7edc\uff0c\u5fc5\u987b\u786e\u8ba4\u5f53\u524d\u7684\u7f51\u7edc\u67b6\u6784\u4e0b\u6b64\u7c7b\u6570\u636e\u5305\u53ef\u4ee5\u5b89\u5168\u901a\u8fc7\u3002 \u76ee\u524d EIP \u548c SNAT \u5730\u5740\u6ca1\u6709\u51b2\u7a81\u68c0\u6d4b\uff0c\u9700\u8981\u7ba1\u7406\u5458\u624b\u52a8\u5206\u914d\u907f\u514d\u5730\u5740\u51b2\u7a81\u3002 \u521b\u5efa\u914d\u7f6e\u6587\u4ef6 \u00b6 \u5728 kube-system \u4e0b\u521b\u5efa ConfigMap ovn-external-gw-config \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-external-gw-config namespace : kube-system data : enable-external-gw : \"true\" external-gw-nodes : \"kube-ovn-worker\" external-gw-nic : \"eth1\" external-gw-addr : \"172.56.0.1/16\" nic-ip : \"172.56.0.254/16\" nic-mac : \"16:52:f3:13:6a:25\" enable-external-gw : \u662f\u5426\u5f00\u542f SNAT \u548c EIP \u529f\u80fd\u3002 type : centrailized \u6216 distributed \uff0c \u9ed8\u8ba4\u4e3a centralized \u5982\u679c\u4f7f\u7528 distributed \uff0c\u5219\u96c6\u7fa4\u6240\u6709\u8282\u70b9\u90fd\u9700\u8981\u6709\u540c\u540d\u7f51\u5361\u6765\u627f\u62c5\u7f51\u5173\u529f\u80fd\u3002 external-gw-nodes : centralized \u6a21\u5f0f\u4e0b\uff0c\u627f\u62c5\u7f51\u5173\u4f5c\u7528\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 external-gw-nic : \u8282\u70b9\u4e0a\u627f\u62c5\u7f51\u5173\u4f5c\u7528\u7684\u7f51\u5361\u540d\u3002 external-gw-addr : \u7269\u7406\u7f51\u7edc\u7f51\u5173\u7684 IP \u548c\u63a9\u7801\u3002 nic-ip , nic-mac : \u5206\u914d\u7ed9\u903b\u8f91\u7f51\u5173\u7aef\u53e3\u7684 IP \u548c Mac\uff0c\u9700\u4e3a\u7269\u7406\u6bb5\u672a\u88ab\u5360\u7528\u7684 IP \u548c Mac\u3002 \u89c2\u5bdf OVN \u548c OVS \u72b6\u6001\u786e\u8ba4\u914d\u7f6e\u751f\u6548 \u00b6 \u68c0\u67e5 OVN-NB \u72b6\u6001, \u786e\u8ba4 ovn-external \u903b\u8f91\u4ea4\u6362\u673a\u5b58\u5728\uff0c\u5e76\u4e14 ovn-cluster-ovn-external \u903b\u8f91\u8def\u7531\u5668\u7aef\u53e3\u4e0a \u7ed1\u5b9a\u4e86\u6b63\u786e\u7684\u5730\u5740\u548c chassis\u3002 # kubectl ko nbctl show switch 3de4cea7-1a71-43f3-8b62-435a57ef16a6 ( ovn-external ) port ln-ovn-external type: localnet addresses: [ \"unknown\" ] port ovn-external-ovn-cluster type: router router-port: ovn-cluster-ovn-external router e1eb83ad-34be-4ed5-9a02-fcc8b1d357c4 ( ovn-cluster ) port ovn-cluster-ovn-external mac: \"ac:1f:6b:2d:33:f1\" networks: [ \"172.56.0.100/16\" ] gateway chassis: [ a5682814-2e2c-46dd-9c1c-6803ef0dab66 ] \u68c0\u67e5 OVS \u72b6\u6001\uff0c\u786e\u8ba4\u76f8\u5e94\u7684\u7f51\u5361\u5df2\u7ecf\u6865\u63a5\u8fdb br-external \u7f51\u6865\uff1a # kubectl ko vsctl ${gateway node name} show e7d81150-7743-4d6e-9e6f-5c688232e130 Bridge br-external Port br-external Interface br-external type: internal Port eno2 Interface eno2 Port patch-ln-ovn-external-to-br-int Interface patch-ln-ovn-external-to-br-int type: patch options: { peer = patch-br-int-to-ln-ovn-external } Pod \u914d\u7f6e EIP \u548c SNAT \u00b6 \u53ef\u901a\u8fc7\u5728 Pod \u4e0a\u589e\u52a0 ovn.kubernetes.io/snat \u6216 ovn.kubernetes.io/eip annotation \u6765\u5206\u522b\u914d\u7f6e SNAT \u548c EIP\uff1a apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/snat : 172.56.0.200 spec : containers : - name : snat-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/eip : 172.56.0.233 spec : containers : - name : eip-pod image : docker.io/library/nginx:alpine \u53ef\u901a\u8fc7 kubectl \u6216\u5176\u4ed6\u5de5\u5177\u52a8\u6001\u8c03\u6574 Pod \u6240\u914d\u7f6e\u7684 EIP \u6216 SNAT \u89c4\u5219\uff0c\u66f4\u6539\u65f6\u8bf7\u6ce8\u610f\u8981\u540c\u65f6\u5220\u9664 ovn.kubernetes.io/routed annotation \u89e6\u53d1\u8def\u7531\u7684\u53d8\u66f4\uff1a kubectl annotate pod pod-gw ovn.kubernetes.io/eip = 172 .56.0.221 --overwrite kubectl annotate pod pod-gw ovn.kubernetes.io/routed- \u5f53 EIP \u6216 SNAT \u89c4\u5219\u751f\u6548\u540e\uff0c ovn.kubernetes.io/routed annotation \u4f1a\u88ab\u91cd\u65b0\u6dfb\u52a0\u3002 \u9ad8\u7ea7\u914d\u7f6e \u00b6 kube-ovn-controller \u7684\u90e8\u5206\u542f\u52a8\u53c2\u6570\u53ef\u5bf9 SNAT \u548c EIP \u529f\u80fd\u8fdb\u884c\u9ad8\u9636\u914d\u7f6e\uff1a --external-gateway-config-ns : Configmap ovn-external-gw-config \u6240\u5c5e Namespace\uff0c \u9ed8\u8ba4\u4e3a kube-system \u3002 --external-gateway-net : \u7269\u7406\u7f51\u5361\u6240\u6865\u63a5\u7684\u7f51\u6865\u540d\uff0c\u9ed8\u8ba4\u4e3a external \u3002 --external-gateway-vlanid : \u7269\u7406\u7f51\u7edc Vlan Tag \u53f7\uff0c\u9ed8\u8ba4\u4e3a 0\uff0c \u5373\u4e0d\u4f7f\u7528 Vlan\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"EIP \u548c SNAT \u914d\u7f6e"},{"location":"guide/eip-snat/#eip-snat","text":"\u8be5\u914d\u7f6e\u9488\u5bf9\u9ed8\u8ba4 VPC \u4e0b\u7684\u7f51\u7edc\uff0c\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u8bf7\u53c2\u8003 VPC \u7f51\u5173 Kube-OVN \u652f\u6301\u5229\u7528 OVN \u4e2d\u7684 L3 Gateway \u529f\u80fd\u6765\u5b9e\u73b0 Pod \u7ea7\u522b\u7684 SNAT \u548c EIP \u529f\u80fd\u3002 \u901a\u8fc7\u4f7f\u7528 SNAT\uff0c\u4e00\u7ec4 Pod \u53ef\u4ee5\u5171\u4eab\u4e00\u4e2a IP \u5730\u5740\u5bf9\u5916\u8fdb\u884c\u8bbf\u95ee\u3002 \u901a\u8fc7 EIP \u7684\u529f\u80fd\uff0c\u4e00\u4e2a Pod \u53ef\u4ee5\u76f4\u63a5\u548c\u4e00\u4e2a\u5916\u90e8 IP \u5173\u8054\uff0c \u5916\u90e8\u670d\u52a1\u53ef\u4ee5\u901a\u8fc7 EIP \u76f4\u63a5\u8bbf\u95ee Pod\uff0cPod \u4e5f\u5c06\u901a\u8fc7\u8fd9\u4e2a EIP \u8bbf\u95ee\u5916\u90e8\u670d\u52a1\u3002","title":"EIP \u548c SNAT \u914d\u7f6e"},{"location":"guide/eip-snat/#_1","text":"\u4e3a\u4e86\u4f7f\u7528 OVN \u7684 L3 Gateway \u80fd\u529b\uff0c\u5fc5\u987b\u5c06\u4e00\u4e2a\u5355\u72ec\u7684\u7f51\u5361\u63a5\u5165 OVS \u7f51\u6865\u4e2d\u8fdb\u884c Overlay \u548c Underlay \u7f51\u7edc\u7684\u6253\u901a\uff0c \u4e3b\u673a\u5fc5\u987b\u6709\u5176\u4ed6\u7684\u7f51\u5361\u7528\u4e8e\u8fd0\u7ef4\u7ba1\u7406\u3002 \u7531\u4e8e\u7ecf\u8fc7 NAT \u540e\u7684\u6570\u636e\u5305\u4f1a\u76f4\u63a5\u8fdb\u5165 Underlay \u7f51\u7edc\uff0c\u5fc5\u987b\u786e\u8ba4\u5f53\u524d\u7684\u7f51\u7edc\u67b6\u6784\u4e0b\u6b64\u7c7b\u6570\u636e\u5305\u53ef\u4ee5\u5b89\u5168\u901a\u8fc7\u3002 \u76ee\u524d EIP \u548c SNAT \u5730\u5740\u6ca1\u6709\u51b2\u7a81\u68c0\u6d4b\uff0c\u9700\u8981\u7ba1\u7406\u5458\u624b\u52a8\u5206\u914d\u907f\u514d\u5730\u5740\u51b2\u7a81\u3002","title":"\u51c6\u5907\u5de5\u4f5c"},{"location":"guide/eip-snat/#_2","text":"\u5728 kube-system \u4e0b\u521b\u5efa ConfigMap ovn-external-gw-config \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-external-gw-config namespace : kube-system data : enable-external-gw : \"true\" external-gw-nodes : \"kube-ovn-worker\" external-gw-nic : \"eth1\" external-gw-addr : \"172.56.0.1/16\" nic-ip : \"172.56.0.254/16\" nic-mac : \"16:52:f3:13:6a:25\" enable-external-gw : \u662f\u5426\u5f00\u542f SNAT \u548c EIP \u529f\u80fd\u3002 type : centrailized \u6216 distributed \uff0c \u9ed8\u8ba4\u4e3a centralized \u5982\u679c\u4f7f\u7528 distributed \uff0c\u5219\u96c6\u7fa4\u6240\u6709\u8282\u70b9\u90fd\u9700\u8981\u6709\u540c\u540d\u7f51\u5361\u6765\u627f\u62c5\u7f51\u5173\u529f\u80fd\u3002 external-gw-nodes : centralized \u6a21\u5f0f\u4e0b\uff0c\u627f\u62c5\u7f51\u5173\u4f5c\u7528\u7684\u8282\u70b9\u540d\uff0c\u9017\u53f7\u5206\u9694\u3002 external-gw-nic : \u8282\u70b9\u4e0a\u627f\u62c5\u7f51\u5173\u4f5c\u7528\u7684\u7f51\u5361\u540d\u3002 external-gw-addr : \u7269\u7406\u7f51\u7edc\u7f51\u5173\u7684 IP \u548c\u63a9\u7801\u3002 nic-ip , nic-mac : \u5206\u914d\u7ed9\u903b\u8f91\u7f51\u5173\u7aef\u53e3\u7684 IP \u548c Mac\uff0c\u9700\u4e3a\u7269\u7406\u6bb5\u672a\u88ab\u5360\u7528\u7684 IP \u548c Mac\u3002","title":"\u521b\u5efa\u914d\u7f6e\u6587\u4ef6"},{"location":"guide/eip-snat/#ovn-ovs","text":"\u68c0\u67e5 OVN-NB \u72b6\u6001, \u786e\u8ba4 ovn-external \u903b\u8f91\u4ea4\u6362\u673a\u5b58\u5728\uff0c\u5e76\u4e14 ovn-cluster-ovn-external \u903b\u8f91\u8def\u7531\u5668\u7aef\u53e3\u4e0a \u7ed1\u5b9a\u4e86\u6b63\u786e\u7684\u5730\u5740\u548c chassis\u3002 # kubectl ko nbctl show switch 3de4cea7-1a71-43f3-8b62-435a57ef16a6 ( ovn-external ) port ln-ovn-external type: localnet addresses: [ \"unknown\" ] port ovn-external-ovn-cluster type: router router-port: ovn-cluster-ovn-external router e1eb83ad-34be-4ed5-9a02-fcc8b1d357c4 ( ovn-cluster ) port ovn-cluster-ovn-external mac: \"ac:1f:6b:2d:33:f1\" networks: [ \"172.56.0.100/16\" ] gateway chassis: [ a5682814-2e2c-46dd-9c1c-6803ef0dab66 ] \u68c0\u67e5 OVS \u72b6\u6001\uff0c\u786e\u8ba4\u76f8\u5e94\u7684\u7f51\u5361\u5df2\u7ecf\u6865\u63a5\u8fdb br-external \u7f51\u6865\uff1a # kubectl ko vsctl ${gateway node name} show e7d81150-7743-4d6e-9e6f-5c688232e130 Bridge br-external Port br-external Interface br-external type: internal Port eno2 Interface eno2 Port patch-ln-ovn-external-to-br-int Interface patch-ln-ovn-external-to-br-int type: patch options: { peer = patch-br-int-to-ln-ovn-external }","title":"\u89c2\u5bdf OVN \u548c OVS \u72b6\u6001\u786e\u8ba4\u914d\u7f6e\u751f\u6548"},{"location":"guide/eip-snat/#pod-eip-snat","text":"\u53ef\u901a\u8fc7\u5728 Pod \u4e0a\u589e\u52a0 ovn.kubernetes.io/snat \u6216 ovn.kubernetes.io/eip annotation \u6765\u5206\u522b\u914d\u7f6e SNAT \u548c EIP\uff1a apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/snat : 172.56.0.200 spec : containers : - name : snat-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/eip : 172.56.0.233 spec : containers : - name : eip-pod image : docker.io/library/nginx:alpine \u53ef\u901a\u8fc7 kubectl \u6216\u5176\u4ed6\u5de5\u5177\u52a8\u6001\u8c03\u6574 Pod \u6240\u914d\u7f6e\u7684 EIP \u6216 SNAT \u89c4\u5219\uff0c\u66f4\u6539\u65f6\u8bf7\u6ce8\u610f\u8981\u540c\u65f6\u5220\u9664 ovn.kubernetes.io/routed annotation \u89e6\u53d1\u8def\u7531\u7684\u53d8\u66f4\uff1a kubectl annotate pod pod-gw ovn.kubernetes.io/eip = 172 .56.0.221 --overwrite kubectl annotate pod pod-gw ovn.kubernetes.io/routed- \u5f53 EIP \u6216 SNAT \u89c4\u5219\u751f\u6548\u540e\uff0c ovn.kubernetes.io/routed annotation \u4f1a\u88ab\u91cd\u65b0\u6dfb\u52a0\u3002","title":"Pod \u914d\u7f6e EIP \u548c SNAT"},{"location":"guide/eip-snat/#_3","text":"kube-ovn-controller \u7684\u90e8\u5206\u542f\u52a8\u53c2\u6570\u53ef\u5bf9 SNAT \u548c EIP \u529f\u80fd\u8fdb\u884c\u9ad8\u9636\u914d\u7f6e\uff1a --external-gateway-config-ns : Configmap ovn-external-gw-config \u6240\u5c5e Namespace\uff0c \u9ed8\u8ba4\u4e3a kube-system \u3002 --external-gateway-net : \u7269\u7406\u7f51\u5361\u6240\u6865\u63a5\u7684\u7f51\u6865\u540d\uff0c\u9ed8\u8ba4\u4e3a external \u3002 --external-gateway-vlanid : \u7269\u7406\u7f51\u7edc Vlan Tag \u53f7\uff0c\u9ed8\u8ba4\u4e3a 0\uff0c \u5373\u4e0d\u4f7f\u7528 Vlan\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9ad8\u7ea7\u914d\u7f6e"},{"location":"guide/ippool/","text":"IP \u6c60\u4f7f\u7528 \u00b6 IP \u6c60\uff08IPPool\uff09\u662f\u6bd4\u5b50\u7f51\uff08Subnet\uff09\u66f4\u7ec6\u529b\u5ea6\u7684 IPAM \u7ba1\u7406\u5355\u5143\u3002\u4f60\u53ef\u4ee5\u901a\u8fc7 IP \u6c60\u5c06\u5b50\u7f51\u7f51\u6bb5\u7ec6\u5206\u4e3a\u591a\u4e2a\u5355\u5143\uff0c\u6bcf\u4e2a\u5355\u5143\u7ed1\u5b9a\u4e00\u4e2a\u6216\u591a\u4e2a\u547d\u540d\u7a7a\u95f4\uff08Namespace\uff09\u3002 \u4f7f\u7528\u65b9\u6cd5 \u00b6 \u4f7f\u7528\u793a\u4f8b\uff1a apiVersion : kubeovn.io/v1 kind : IPPool metadata : name : pool-1 spec : subnet : ovn-default ips : - \"10.16.0.201\" - \"10.16.0.210/30\" - \"10.16.0.220..10.16.0.230\" namespaces : - ns-1 \u5b57\u6bb5\u8bf4\u660e\uff1a \u540d\u79f0 \u7528\u9014 \u5907\u6ce8 subnet \u6307\u5b9a\u6240\u5c5e\u5b50\u7f51 \u5fc5\u586b ips \u6307\u5b9a\u5305\u542b\u7684 IP \u8303\u56f4 \u652f\u6301 \u3001 \u4ee5\u53ca .. \u4e09\u79cd\u683c\u5f0f\uff0c\u652f\u6301 IPv6\u3002 namespaces \u7ed1\u5b9a\u547d\u540d\u7a7a\u95f4 \u53ef\u9009 \u6ce8\u610f\u4e8b\u9879 \u00b6 \u4e3a\u4fdd\u8bc1\u4e0e Workload \u901a\u7528 IP Pool \u56fa\u5b9a\u5730\u5740 \u7684\u517c\u5bb9\u6027\uff0cIP \u6c60\u7684\u540d\u79f0\u4e0d\u80fd\u662f\u4e00\u4e2a IP \u5730\u5740\uff1b IP \u6c60\u7684 .spec.ips \u53ef\u6307\u5b9a\u8d85\u51fa\u5b50\u7f51\u8303\u56f4\u7684 IP \u5730\u5740\uff0c\u4f46\u5b9e\u9645\u6709\u6548\u7684 IP \u5730\u5740\u662f .spec.ips \u4e0e\u5b50\u7f51 CIDR \u7684\u4ea4\u96c6\uff1b \u540c\u4e00\u4e2a\u5b50\u7f51\u7684\u4e0d\u540c IP \u6c60\uff0c\u4e0d\u80fd\u5305\u542b\u76f8\u540c\u7684\uff08\u6709\u6548\uff09IP \u5730\u5740\uff1b IP \u6c60\u7684 .spec.ips \u53ef\u52a8\u6001\u4fee\u6539\uff1b IP \u6c60\u4f1a\u7ee7\u627f\u5b50\u7f51\u7684\u4fdd\u7559 IP\uff0c\u4ece IP \u6c60\u968f\u673a\u5206\u914d IP \u5730\u5740\u65f6\uff0c\u4f1a\u8df3\u8fc7\u5305\u542b\u5728 IP \u6c60\u4e2d\u7684\u4fdd\u7559 IP\uff1b \u4ece\u5b50\u7f51\u968f\u673a\u5206\u914d IP \u5730\u5740\u65f6\uff0c\u53ea\u4f1a\u4ece\u5b50\u7f51\u6240\u6709 IP \u6c60\u4ee5\u5916\u7684\u8303\u56f4\u5206\u914d\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"IP\u6c60\u4f7f\u7528"},{"location":"guide/ippool/#ip","text":"IP \u6c60\uff08IPPool\uff09\u662f\u6bd4\u5b50\u7f51\uff08Subnet\uff09\u66f4\u7ec6\u529b\u5ea6\u7684 IPAM \u7ba1\u7406\u5355\u5143\u3002\u4f60\u53ef\u4ee5\u901a\u8fc7 IP \u6c60\u5c06\u5b50\u7f51\u7f51\u6bb5\u7ec6\u5206\u4e3a\u591a\u4e2a\u5355\u5143\uff0c\u6bcf\u4e2a\u5355\u5143\u7ed1\u5b9a\u4e00\u4e2a\u6216\u591a\u4e2a\u547d\u540d\u7a7a\u95f4\uff08Namespace\uff09\u3002","title":"IP \u6c60\u4f7f\u7528"},{"location":"guide/ippool/#_1","text":"\u4f7f\u7528\u793a\u4f8b\uff1a apiVersion : kubeovn.io/v1 kind : IPPool metadata : name : pool-1 spec : subnet : ovn-default ips : - \"10.16.0.201\" - \"10.16.0.210/30\" - \"10.16.0.220..10.16.0.230\" namespaces : - ns-1 \u5b57\u6bb5\u8bf4\u660e\uff1a \u540d\u79f0 \u7528\u9014 \u5907\u6ce8 subnet \u6307\u5b9a\u6240\u5c5e\u5b50\u7f51 \u5fc5\u586b ips \u6307\u5b9a\u5305\u542b\u7684 IP \u8303\u56f4 \u652f\u6301 \u3001 \u4ee5\u53ca .. \u4e09\u79cd\u683c\u5f0f\uff0c\u652f\u6301 IPv6\u3002 namespaces \u7ed1\u5b9a\u547d\u540d\u7a7a\u95f4 \u53ef\u9009","title":"\u4f7f\u7528\u65b9\u6cd5"},{"location":"guide/ippool/#_2","text":"\u4e3a\u4fdd\u8bc1\u4e0e Workload \u901a\u7528 IP Pool \u56fa\u5b9a\u5730\u5740 \u7684\u517c\u5bb9\u6027\uff0cIP \u6c60\u7684\u540d\u79f0\u4e0d\u80fd\u662f\u4e00\u4e2a IP \u5730\u5740\uff1b IP \u6c60\u7684 .spec.ips \u53ef\u6307\u5b9a\u8d85\u51fa\u5b50\u7f51\u8303\u56f4\u7684 IP \u5730\u5740\uff0c\u4f46\u5b9e\u9645\u6709\u6548\u7684 IP \u5730\u5740\u662f .spec.ips \u4e0e\u5b50\u7f51 CIDR \u7684\u4ea4\u96c6\uff1b \u540c\u4e00\u4e2a\u5b50\u7f51\u7684\u4e0d\u540c IP \u6c60\uff0c\u4e0d\u80fd\u5305\u542b\u76f8\u540c\u7684\uff08\u6709\u6548\uff09IP \u5730\u5740\uff1b IP \u6c60\u7684 .spec.ips \u53ef\u52a8\u6001\u4fee\u6539\uff1b IP \u6c60\u4f1a\u7ee7\u627f\u5b50\u7f51\u7684\u4fdd\u7559 IP\uff0c\u4ece IP \u6c60\u968f\u673a\u5206\u914d IP \u5730\u5740\u65f6\uff0c\u4f1a\u8df3\u8fc7\u5305\u542b\u5728 IP \u6c60\u4e2d\u7684\u4fdd\u7559 IP\uff1b \u4ece\u5b50\u7f51\u968f\u673a\u5206\u914d IP \u5730\u5740\u65f6\uff0c\u53ea\u4f1a\u4ece\u5b50\u7f51\u6240\u6709 IP \u6c60\u4ee5\u5916\u7684\u8303\u56f4\u5206\u914d\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"guide/loadbalancer-service/","text":"LoadBalancer \u7c7b\u578b Service \u00b6 Kube-OVN \u5df2\u7ecf\u652f\u6301\u4e86 VPC \u548c VPC \u7f51\u5173\u7684\u5b9e\u73b0\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 VPC \u914d\u7f6e \u3002 \u7531\u4e8e VPC \u7f51\u5173\u7684\u4f7f\u7528\u6bd4\u8f83\u590d\u6742\uff0c\u57fa\u4e8e VPC \u7f51\u5173\u7684\u5b9e\u73b0\u505a\u4e86\u7b80\u5316\uff0c\u652f\u6301\u5728\u9ed8\u8ba4 VPC \u4e0b\u521b\u5efa LoadBalancer \u7c7b\u578b\u7684 Service\uff0c\u5b9e\u73b0\u901a\u8fc7 LoadBalancerIP \u6765\u8bbf\u95ee\u9ed8\u8ba4 VPC \u4e0b\u7684 Service\u3002 \u9996\u5148\u786e\u8ba4\u73af\u5883\u4e0a\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff1a \u5b89\u88c5\u4e86 multus-cni \u548c macvlan cni \u3002 LoadBalancer Service \u7684\u652f\u6301\uff0c\u662f\u5bf9 VPC \u7f51\u5173\u4ee3\u7801\u8fdb\u884c\u7b80\u5316\u5b9e\u73b0\u7684\uff0c\u4ecd\u7136\u4f7f\u7528 vpc-nat-gw \u7684\u955c\u50cf\uff0c\u4f9d\u8d56 macvlan \u63d0\u4f9b\u591a\u7f51\u5361\u529f\u80fd\u652f\u6301\u3002 \u76ee\u524d\u53ea\u652f\u6301\u5728 \u9ed8\u8ba4 VPC \u914d\u7f6e\uff0c\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 LoadBalancer \u652f\u6301\u53ef\u4ee5\u53c2\u8003 VPC \u7684\u6587\u6863 VPC \u914d\u7f6e \u3002 \u9ed8\u8ba4 VPC LoadBalancer Service \u914d\u7f6e\u6b65\u9aa4 \u00b6 \u5f00\u542f\u7279\u6027\u5f00\u5173 \u00b6 \u4fee\u6539 kube-system namespace \u4e0b\u7684 deployment kube-ovn-controller \uff0c\u5728 args \u4e2d\u589e\u52a0\u53c2\u6570 --enable-lb-svc=true \uff0c\u5f00\u542f\u529f\u80fd\u5f00\u5173\uff0c\u8be5\u53c2\u6570\u9ed8\u8ba4\u4e3a false\u3002 containers : - args : - /kube-ovn/start-controller.sh - --default-cidr=10.16.0.0/16 - --default-gateway=10.16.0.1 - --default-gateway-check=true - --enable-lb-svc=true // \u53c2\u6570\u8bbe\u7f6e\u4e3a true \u521b\u5efa NetworkAttachmentDefinition CRD \u8d44\u6e90 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa net-attach-def \u8d44\u6e90: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : lb-svc-attachment namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", //\u7269\u7406\u7f51\u5361\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u914d\u7f6e \"mode\": \"bridge\" }' \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u901a\u8fc7\u7269\u7406\u7f51\u5361 eth0 \u6765\u5b9e\u73b0\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u4ed6\u7269\u7406\u7f51\u5361\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 master \u53d6\u503c\uff0c\u6307\u5b9a\u4f7f\u7528\u7684\u7269\u7406\u7f51\u5361\u540d\u79f0\u3002 \u521b\u5efa Subnet \u00b6 \u521b\u5efa\u7684 Subnet\uff0c\u7528\u4e8e\u7ed9 LoadBalancer Service \u5206\u914d LoadBalancerIP\uff0c\u8be5\u5730\u5740\u6b63\u5e38\u60c5\u51b5\u4e0b\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u53ef\u4ee5\u914d\u7f6e Underlay Subnet \u7528\u4e8e\u5730\u5740\u5206\u914d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa\u65b0\u5b50\u7f51\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attach-subnet spec : protocol : IPv4 provider : lb-svc-attachment.kube-system # provider \u683c\u5f0f\u56fa\u5b9a\uff0c\u7531\u4e0a\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210 cidrBlock : 172.18.0.0/16 gateway : 172.18.0.1 excludeIps : - 172.18.0.0..172.18.0.10 Subnet \u4e2d provider \u53c2\u6570\u4ee5 ovn \u6216\u8005\u4ee5 .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u8868\u793a\u8be5\u5b50\u7f51\u662f\u7531 Kube-OVN \u7ba1\u7406\u4f7f\u7528\uff0c\u9700\u8981\u5bf9\u5e94\u521b\u5efa logical switch \u8bb0\u5f55\u3002 provider \u975e ovn \u6216\u8005\u975e .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u5219 Kube-OVN \u53ea\u63d0\u4f9b IPAM \u529f\u80fd\uff0c\u8bb0\u5f55 IP \u5730\u5740\u5206\u914d\u60c5\u51b5\uff0c\u4e0d\u5bf9\u5b50\u7f51\u505a\u4e1a\u52a1\u903b\u8f91\u5904\u7406\u3002 \u521b\u5efa LoadBalancer Service \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa LoadBalancer Service\uff1a apiVersion : v1 kind : Service metadata : annotations : lb-svc-attachment.kube-system.kubernetes.io/logical_switch : attach-subnet #\u53ef\u9009 ovn.kubernetes.io/attachmentprovider : lb-svc-attachment.kube-system #\u5fc5\u987b labels : app : dynamic name : test-service namespace : default spec : loadBalancerIP : 172.18.0.18 #\u53ef\u9009 ports : - name : test protocol : TCP port : 80 targetPort : 80 selector : app : dynamic sessionAffinity : None type : LoadBalancer \u5728 yaml \u4e2d\uff0cannotation ovn.kubernetes.io/attachmentprovider \u4e3a\u5fc5\u586b\u9879\uff0c\u53d6\u503c\u7531\u7b2c\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210\u3002\u8be5 annotation \u7528\u4e8e\u5728\u521b\u5efa Pod \u65f6\uff0c\u67e5\u627e net-attach-def \u8d44\u6e90\u3002 \u53ef\u4ee5\u901a\u8fc7 annotation \u6307\u5b9a\u591a\u7f51\u5361\u5730\u5740\u5206\u914d\u4f7f\u7528\u7684\u5b50\u7f51\u3002annotation key \u683c\u5f0f\u4e3a net-attach-def \u8d44\u6e90\u7684 Name.Namespace.kubernetes.io/logical_switch \u3002\u8be5\u914d\u7f6e\u4e3a \u53ef\u9009 \u9009\u9879\uff0c\u5728\u6ca1\u6709\u6307\u5b9a LoadBalancerIP \u5730\u5740\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u4ece\u8be5\u5b50\u7f51\u52a8\u6001\u5206\u914d\u5730\u5740\uff0c\u586b\u5145\u5230 LoadBalancerIP \u5b57\u6bb5\u3002 \u5982\u679c\u9700\u8981\u9759\u6001\u914d\u7f6e LoadBalancerIP \u5730\u5740\uff0c\u53ef\u4ee5\u914d\u7f6e spec.loadBalancerIP \u5b57\u6bb5\uff0c\u8be5\u5730\u5740\u9700\u8981\u5728\u6307\u5b9a\u5b50\u7f51\u7684\u5730\u5740\u8303\u56f4\u5185\u3002 \u5728\u6267\u884c yaml \u521b\u5efa Service \u540e\uff0c\u5728 Service \u540c Namespace \u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u542f\u52a8\u4fe1\u606f\uff1a # kubectl get pod NAME READY STATUS RESTARTS AGE lb-svc-test-service-6869d98dd8-cjvll 1 /1 Running 0 107m # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE test-service LoadBalancer 10 .109.201.193 172 .18.0.18 80 :30056/TCP 107m \u6307\u5b9a service.spec.loadBalancerIP \u53c2\u6570\u65f6\uff0c\u6700\u7ec8\u5c06\u8be5\u53c2\u6570\u8d4b\u503c\u7ed9 service external-ip \u5b57\u6bb5\u3002\u4e0d\u6307\u5b9a\u7684\u60c5\u51b5\u4e0b\uff0c\u8be5\u53c2\u6570\u4e3a\u968f\u673a\u5206\u914d\u503c\u3002 \u67e5\u770b\u6d4b\u8bd5 Pod \u7684 yaml \u8f93\u51fa\uff0c\u5b58\u5728\u591a\u7f51\u5361\u5206\u914d\u7684\u5730\u5740\u4fe1\u606f\uff1a # kubectl get pod -o yaml lb-svc-test-service-6869d98dd8-cjvll apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/network-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] k8s.v1.cni.cncf.io/networks: default/test-service k8s.v1.cni.cncf.io/networks-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] ovn.kubernetes.io/allocated: \"true\" ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.2 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:45:F4:29 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/routed: \"true\" test-service.default.kubernetes.io/allocated: \"true\" test-service.default.kubernetes.io/cidr: 172 .18.0.0/16 test-service.default.kubernetes.io/gateway: 172 .18.0.1 test-service.default.kubernetes.io/ip_address: 172 .18.0.18 test-service.default.kubernetes.io/logical_switch: attach-subnet test-service.default.kubernetes.io/mac_address: 00 :00:00:AF:AA:BF test-service.default.kubernetes.io/pod_nic_type: veth-pair \u67e5\u770b Service \u7684\u4fe1\u606f\uff1a # kubectl get svc -o yaml test-service apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { \"apiVersion\" : \"v1\" , \"kind\" : \"Service\" , \"metadata\" : { \"annotations\" : { \"test-service.default.kubernetes.io/logical_switch\" : \"attach-subnet\" } , \"labels\" : { \"app\" : \"dynamic\" } , \"name\" : \"test-service\" , \"namespace\" : \"default\" } , \"spec\" : { \"ports\" : [{ \"name\" : \"test\" , \"port\" :80, \"protocol\" : \"TCP\" , \"targetPort\" :80 }] , \"selector\" : { \"app\" : \"dynamic\" } , \"sessionAffinity\" : \"None\" , \"type\" : \"LoadBalancer\" }} ovn.kubernetes.io/vpc: ovn-cluster test-service.default.kubernetes.io/logical_switch: attach-subnet creationTimestamp: \"2022-06-15T09:01:58Z\" labels: app: dynamic name: test-service namespace: default resourceVersion: \"38485\" uid: 161edee1-7f6e-40f5-9e09-5a52c44267d0 spec: allocateLoadBalancerNodePorts: true clusterIP: 10 .109.201.193 clusterIPs: - 10 .109.201.193 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: test nodePort: 30056 port: 80 protocol: TCP targetPort: 80 selector: app: dynamic sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - ip: 172 .18.0.18 \u6d4b\u8bd5 LoadBalancerIP \u8bbf\u95ee \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml, \u521b\u5efa\u6d4b\u8bd5 Pod\uff0c\u4f5c\u4e3a Service \u7684 Endpoints \u63d0\u4f9b\u670d\u52a1: apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx dnsPolicy : ClusterFirst restartPolicy : Always \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u63d0\u4f9b\u7684\u5b50\u7f51\u5730\u5740\uff0c\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u4e3a\u4e86\u7b80\u5355\u9a8c\u8bc1\uff0c\u5728\u96c6\u7fa4\u5185\u8bbf\u95ee Service \u7684 LoadBalancerIP:Port \uff0c\u67e5\u770b\u662f\u5426\u6b63\u5e38\u8bbf\u95ee\u6210\u529f\u3002 # curl 172.18.0.11:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on dynamic-7d8d7874f5-hsgc4</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u8fdb\u5165 Service \u521b\u5efa\u7684 Pod\uff0c\u67e5\u770b\u7f51\u7edc\u7684\u4fe1\u606f # ip a 4 : net1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ba:85:f7:02:9f:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172 .18.0.18/16 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::b885:f7ff:fe02:9f42/64 scope link valid_lft forever preferred_lft forever 36 : eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 00 :00:00:45:f4:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10 .16.0.2/16 brd 10 .16.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe45:f429/64 scope link valid_lft forever preferred_lft forever # ip rule 0 : from all lookup local 32764 : from all iif eth0 lookup 100 32765 : from all iif net1 lookup 100 32766 : from all lookup main 32767 : from all lookup default # ip route show table 100 default via 172 .18.0.1 dev net1 10 .109.201.193 via 10 .16.0.1 dev eth0 172 .18.0.0/16 dev net1 scope link # iptables -t nat -L -n -v Chain PREROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- * * 0 .0.0.0/0 172 .18.0.18 tcp dpt:80 to:10.109.201.193:80 Chain INPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain OUTPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain POSTROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0 .0.0.0/0 10 .109.201.193 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"LoadBalancer \u7c7b\u578b Service"},{"location":"guide/loadbalancer-service/#loadbalancer-service","text":"Kube-OVN \u5df2\u7ecf\u652f\u6301\u4e86 VPC \u548c VPC \u7f51\u5173\u7684\u5b9e\u73b0\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 VPC \u914d\u7f6e \u3002 \u7531\u4e8e VPC \u7f51\u5173\u7684\u4f7f\u7528\u6bd4\u8f83\u590d\u6742\uff0c\u57fa\u4e8e VPC \u7f51\u5173\u7684\u5b9e\u73b0\u505a\u4e86\u7b80\u5316\uff0c\u652f\u6301\u5728\u9ed8\u8ba4 VPC \u4e0b\u521b\u5efa LoadBalancer \u7c7b\u578b\u7684 Service\uff0c\u5b9e\u73b0\u901a\u8fc7 LoadBalancerIP \u6765\u8bbf\u95ee\u9ed8\u8ba4 VPC \u4e0b\u7684 Service\u3002 \u9996\u5148\u786e\u8ba4\u73af\u5883\u4e0a\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff1a \u5b89\u88c5\u4e86 multus-cni \u548c macvlan cni \u3002 LoadBalancer Service \u7684\u652f\u6301\uff0c\u662f\u5bf9 VPC \u7f51\u5173\u4ee3\u7801\u8fdb\u884c\u7b80\u5316\u5b9e\u73b0\u7684\uff0c\u4ecd\u7136\u4f7f\u7528 vpc-nat-gw \u7684\u955c\u50cf\uff0c\u4f9d\u8d56 macvlan \u63d0\u4f9b\u591a\u7f51\u5361\u529f\u80fd\u652f\u6301\u3002 \u76ee\u524d\u53ea\u652f\u6301\u5728 \u9ed8\u8ba4 VPC \u914d\u7f6e\uff0c\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 LoadBalancer \u652f\u6301\u53ef\u4ee5\u53c2\u8003 VPC \u7684\u6587\u6863 VPC \u914d\u7f6e \u3002","title":"LoadBalancer \u7c7b\u578b Service"},{"location":"guide/loadbalancer-service/#vpc-loadbalancer-service","text":"","title":"\u9ed8\u8ba4 VPC LoadBalancer Service \u914d\u7f6e\u6b65\u9aa4"},{"location":"guide/loadbalancer-service/#_1","text":"\u4fee\u6539 kube-system namespace \u4e0b\u7684 deployment kube-ovn-controller \uff0c\u5728 args \u4e2d\u589e\u52a0\u53c2\u6570 --enable-lb-svc=true \uff0c\u5f00\u542f\u529f\u80fd\u5f00\u5173\uff0c\u8be5\u53c2\u6570\u9ed8\u8ba4\u4e3a false\u3002 containers : - args : - /kube-ovn/start-controller.sh - --default-cidr=10.16.0.0/16 - --default-gateway=10.16.0.1 - --default-gateway-check=true - --enable-lb-svc=true // \u53c2\u6570\u8bbe\u7f6e\u4e3a true","title":"\u5f00\u542f\u7279\u6027\u5f00\u5173"},{"location":"guide/loadbalancer-service/#networkattachmentdefinition-crd","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa net-attach-def \u8d44\u6e90: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : lb-svc-attachment namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", //\u7269\u7406\u7f51\u5361\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u914d\u7f6e \"mode\": \"bridge\" }' \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u901a\u8fc7\u7269\u7406\u7f51\u5361 eth0 \u6765\u5b9e\u73b0\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u4ed6\u7269\u7406\u7f51\u5361\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 master \u53d6\u503c\uff0c\u6307\u5b9a\u4f7f\u7528\u7684\u7269\u7406\u7f51\u5361\u540d\u79f0\u3002","title":"\u521b\u5efa NetworkAttachmentDefinition CRD \u8d44\u6e90"},{"location":"guide/loadbalancer-service/#subnet","text":"\u521b\u5efa\u7684 Subnet\uff0c\u7528\u4e8e\u7ed9 LoadBalancer Service \u5206\u914d LoadBalancerIP\uff0c\u8be5\u5730\u5740\u6b63\u5e38\u60c5\u51b5\u4e0b\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u53ef\u4ee5\u914d\u7f6e Underlay Subnet \u7528\u4e8e\u5730\u5740\u5206\u914d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa\u65b0\u5b50\u7f51\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attach-subnet spec : protocol : IPv4 provider : lb-svc-attachment.kube-system # provider \u683c\u5f0f\u56fa\u5b9a\uff0c\u7531\u4e0a\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210 cidrBlock : 172.18.0.0/16 gateway : 172.18.0.1 excludeIps : - 172.18.0.0..172.18.0.10 Subnet \u4e2d provider \u53c2\u6570\u4ee5 ovn \u6216\u8005\u4ee5 .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u8868\u793a\u8be5\u5b50\u7f51\u662f\u7531 Kube-OVN \u7ba1\u7406\u4f7f\u7528\uff0c\u9700\u8981\u5bf9\u5e94\u521b\u5efa logical switch \u8bb0\u5f55\u3002 provider \u975e ovn \u6216\u8005\u975e .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u5219 Kube-OVN \u53ea\u63d0\u4f9b IPAM \u529f\u80fd\uff0c\u8bb0\u5f55 IP \u5730\u5740\u5206\u914d\u60c5\u51b5\uff0c\u4e0d\u5bf9\u5b50\u7f51\u505a\u4e1a\u52a1\u903b\u8f91\u5904\u7406\u3002","title":"\u521b\u5efa Subnet"},{"location":"guide/loadbalancer-service/#loadbalancer-service_1","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa LoadBalancer Service\uff1a apiVersion : v1 kind : Service metadata : annotations : lb-svc-attachment.kube-system.kubernetes.io/logical_switch : attach-subnet #\u53ef\u9009 ovn.kubernetes.io/attachmentprovider : lb-svc-attachment.kube-system #\u5fc5\u987b labels : app : dynamic name : test-service namespace : default spec : loadBalancerIP : 172.18.0.18 #\u53ef\u9009 ports : - name : test protocol : TCP port : 80 targetPort : 80 selector : app : dynamic sessionAffinity : None type : LoadBalancer \u5728 yaml \u4e2d\uff0cannotation ovn.kubernetes.io/attachmentprovider \u4e3a\u5fc5\u586b\u9879\uff0c\u53d6\u503c\u7531\u7b2c\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210\u3002\u8be5 annotation \u7528\u4e8e\u5728\u521b\u5efa Pod \u65f6\uff0c\u67e5\u627e net-attach-def \u8d44\u6e90\u3002 \u53ef\u4ee5\u901a\u8fc7 annotation \u6307\u5b9a\u591a\u7f51\u5361\u5730\u5740\u5206\u914d\u4f7f\u7528\u7684\u5b50\u7f51\u3002annotation key \u683c\u5f0f\u4e3a net-attach-def \u8d44\u6e90\u7684 Name.Namespace.kubernetes.io/logical_switch \u3002\u8be5\u914d\u7f6e\u4e3a \u53ef\u9009 \u9009\u9879\uff0c\u5728\u6ca1\u6709\u6307\u5b9a LoadBalancerIP \u5730\u5740\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u4ece\u8be5\u5b50\u7f51\u52a8\u6001\u5206\u914d\u5730\u5740\uff0c\u586b\u5145\u5230 LoadBalancerIP \u5b57\u6bb5\u3002 \u5982\u679c\u9700\u8981\u9759\u6001\u914d\u7f6e LoadBalancerIP \u5730\u5740\uff0c\u53ef\u4ee5\u914d\u7f6e spec.loadBalancerIP \u5b57\u6bb5\uff0c\u8be5\u5730\u5740\u9700\u8981\u5728\u6307\u5b9a\u5b50\u7f51\u7684\u5730\u5740\u8303\u56f4\u5185\u3002 \u5728\u6267\u884c yaml \u521b\u5efa Service \u540e\uff0c\u5728 Service \u540c Namespace \u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u542f\u52a8\u4fe1\u606f\uff1a # kubectl get pod NAME READY STATUS RESTARTS AGE lb-svc-test-service-6869d98dd8-cjvll 1 /1 Running 0 107m # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE test-service LoadBalancer 10 .109.201.193 172 .18.0.18 80 :30056/TCP 107m \u6307\u5b9a service.spec.loadBalancerIP \u53c2\u6570\u65f6\uff0c\u6700\u7ec8\u5c06\u8be5\u53c2\u6570\u8d4b\u503c\u7ed9 service external-ip \u5b57\u6bb5\u3002\u4e0d\u6307\u5b9a\u7684\u60c5\u51b5\u4e0b\uff0c\u8be5\u53c2\u6570\u4e3a\u968f\u673a\u5206\u914d\u503c\u3002 \u67e5\u770b\u6d4b\u8bd5 Pod \u7684 yaml \u8f93\u51fa\uff0c\u5b58\u5728\u591a\u7f51\u5361\u5206\u914d\u7684\u5730\u5740\u4fe1\u606f\uff1a # kubectl get pod -o yaml lb-svc-test-service-6869d98dd8-cjvll apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/network-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] k8s.v1.cni.cncf.io/networks: default/test-service k8s.v1.cni.cncf.io/networks-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] ovn.kubernetes.io/allocated: \"true\" ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.2 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:45:F4:29 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/routed: \"true\" test-service.default.kubernetes.io/allocated: \"true\" test-service.default.kubernetes.io/cidr: 172 .18.0.0/16 test-service.default.kubernetes.io/gateway: 172 .18.0.1 test-service.default.kubernetes.io/ip_address: 172 .18.0.18 test-service.default.kubernetes.io/logical_switch: attach-subnet test-service.default.kubernetes.io/mac_address: 00 :00:00:AF:AA:BF test-service.default.kubernetes.io/pod_nic_type: veth-pair \u67e5\u770b Service \u7684\u4fe1\u606f\uff1a # kubectl get svc -o yaml test-service apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { \"apiVersion\" : \"v1\" , \"kind\" : \"Service\" , \"metadata\" : { \"annotations\" : { \"test-service.default.kubernetes.io/logical_switch\" : \"attach-subnet\" } , \"labels\" : { \"app\" : \"dynamic\" } , \"name\" : \"test-service\" , \"namespace\" : \"default\" } , \"spec\" : { \"ports\" : [{ \"name\" : \"test\" , \"port\" :80, \"protocol\" : \"TCP\" , \"targetPort\" :80 }] , \"selector\" : { \"app\" : \"dynamic\" } , \"sessionAffinity\" : \"None\" , \"type\" : \"LoadBalancer\" }} ovn.kubernetes.io/vpc: ovn-cluster test-service.default.kubernetes.io/logical_switch: attach-subnet creationTimestamp: \"2022-06-15T09:01:58Z\" labels: app: dynamic name: test-service namespace: default resourceVersion: \"38485\" uid: 161edee1-7f6e-40f5-9e09-5a52c44267d0 spec: allocateLoadBalancerNodePorts: true clusterIP: 10 .109.201.193 clusterIPs: - 10 .109.201.193 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: test nodePort: 30056 port: 80 protocol: TCP targetPort: 80 selector: app: dynamic sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - ip: 172 .18.0.18","title":"\u521b\u5efa LoadBalancer Service"},{"location":"guide/loadbalancer-service/#loadbalancerip","text":"\u53c2\u8003\u4ee5\u4e0b yaml, \u521b\u5efa\u6d4b\u8bd5 Pod\uff0c\u4f5c\u4e3a Service \u7684 Endpoints \u63d0\u4f9b\u670d\u52a1: apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx dnsPolicy : ClusterFirst restartPolicy : Always \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u63d0\u4f9b\u7684\u5b50\u7f51\u5730\u5740\uff0c\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u4e3a\u4e86\u7b80\u5355\u9a8c\u8bc1\uff0c\u5728\u96c6\u7fa4\u5185\u8bbf\u95ee Service \u7684 LoadBalancerIP:Port \uff0c\u67e5\u770b\u662f\u5426\u6b63\u5e38\u8bbf\u95ee\u6210\u529f\u3002 # curl 172.18.0.11:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on dynamic-7d8d7874f5-hsgc4</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u8fdb\u5165 Service \u521b\u5efa\u7684 Pod\uff0c\u67e5\u770b\u7f51\u7edc\u7684\u4fe1\u606f # ip a 4 : net1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ba:85:f7:02:9f:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172 .18.0.18/16 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::b885:f7ff:fe02:9f42/64 scope link valid_lft forever preferred_lft forever 36 : eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 00 :00:00:45:f4:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10 .16.0.2/16 brd 10 .16.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe45:f429/64 scope link valid_lft forever preferred_lft forever # ip rule 0 : from all lookup local 32764 : from all iif eth0 lookup 100 32765 : from all iif net1 lookup 100 32766 : from all lookup main 32767 : from all lookup default # ip route show table 100 default via 172 .18.0.1 dev net1 10 .109.201.193 via 10 .16.0.1 dev eth0 172 .18.0.0/16 dev net1 scope link # iptables -t nat -L -n -v Chain PREROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- * * 0 .0.0.0/0 172 .18.0.18 tcp dpt:80 to:10.109.201.193:80 Chain INPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain OUTPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain POSTROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0 .0.0.0/0 10 .109.201.193 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6d4b\u8bd5 LoadBalancerIP \u8bbf\u95ee"},{"location":"guide/mirror/","text":"\u6d41\u91cf\u955c\u50cf \u00b6 \u6d41\u91cf\u955c\u50cf\u529f\u80fd\u53ef\u4ee5\u5c06\u8fdb\u51fa\u5bb9\u5668\u7f51\u7edc\u7684\u6570\u636e\u5305\u8fdb\u884c\u590d\u5236\u5230\u4e3b\u673a\u7684\u7279\u5b9a\u7f51\u5361\u3002\u7ba1\u7406\u5458\u6216\u5f00\u53d1\u8005 \u53ef\u4ee5\u901a\u8fc7\u76d1\u542c\u8fd9\u5757\u7f51\u5361\u83b7\u5f97\u5b8c\u6574\u7684\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u6765\u8fdb\u4e00\u6b65\u8fdb\u884c\u5206\u6790\uff0c\u76d1\u63a7\uff0c\u5b89\u5168\u5ba1\u8ba1\u7b49\u64cd\u4f5c\u3002 \u4e5f\u53ef\u548c\u4f20\u7edf\u7684 NPM \u5bf9\u63a5\u83b7\u53d6\u66f4\u7ec6\u7c92\u5ea6\u7684\u6d41\u91cf\u76d1\u63a7\u3002 \u6d41\u91cf\u955c\u50cf\u529f\u80fd\u4f1a\u5e26\u6765\u4e00\u5b9a\u7684\u6027\u80fd\u635f\u5931\uff0c\u6839\u636e CPU \u6027\u80fd\u4ee5\u53ca\u6d41\u91cf\u7684\u7279\u5f81\uff0c\u4f1a\u6709 5%~10% \u7684 \u989d\u5916 CPU \u6d88\u8017\u3002 \u5168\u5c40\u6d41\u91cf\u955c\u50cf\u914d\u7f6e \u00b6 \u6d41\u91cf\u955c\u50cf\u529f\u80fd\u9ed8\u8ba4\u4e3a\u5173\u95ed\u72b6\u6001\uff0c\u5982\u679c\u9700\u8981\u5f00\u542f\u8bf7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u542f\u52a8\u53c2\u6570\uff1a --enable-mirror=true \uff1a \u662f\u5426\u5f00\u542f\u6d41\u91cf\u955c\u50cf\u3002 --mirror-iface=mirror0 : \u6d41\u91cf\u955c\u50cf\u6240\u590d\u5236\u5230\u7684\u7f51\u5361\u540d\u3002\u8be5\u7f51\u5361\u53ef\u4e3a\u4e3b\u673a\u4e0a\u5df2\u5b58\u5728\u7684\u4e00\u5757\u7269\u7406\u7f51\u5361\uff0c \u6b64\u65f6\u8be5\u7f51\u5361\u4f1a\u88ab\u6865\u63a5\u8fdb br-int \u7f51\u6865\uff0c\u955c\u50cf\u6d41\u91cf\u4f1a\u76f4\u63a5\u63a5\u5165\u5e95\u5c42\u4ea4\u6362\u673a\u3002\u82e5\u7f51\u5361\u540d\u4e0d\u5b58\u5728\uff0cKube-OVN \u4f1a\u81ea\u52a8 \u521b\u5efa\u4e00\u5757\u540c\u540d\u7684\u865a\u62df\u7f51\u5361\uff0c\u7ba1\u7406\u5458\u6216\u5f00\u53d1\u8005\u53ef\u4ee5\u5728\u5bbf\u4e3b\u673a\u4e0a\u901a\u8fc7\u8be5\u7f51\u5361\u83b7\u53d6\u5f53\u524d\u8282\u70b9\u6240\u6709\u6d41\u91cf\u3002\u9ed8\u8ba4\u4e3a mirror0 \u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u7528 tcpdump \u6216\u5176\u4ed6\u6d41\u91cf\u5206\u6790\u5de5\u5177\u76d1\u542c mirror0 \u4e0a\u7684\u6d41\u91cf\uff1a tcpdump -ni mirror0 Pod \u7ea7\u522b\u6d41\u91cf\u955c\u50cf\u914d\u7f6e \u00b6 \u5982\u679c\u53ea\u9700\u5bf9\u90e8\u5206 Pod \u6d41\u91cf\u8fdb\u884c\u955c\u50cf\uff0c\u5219\u9700\u8981\u5173\u95ed\u5168\u5c40\u7684\u6d41\u91cf\u955c\u50cf\u529f\u80fd\uff0c\u7136\u540e\u5728\u7279\u5b9a Pod \u4e0a\u589e\u52a0 ovn.kubernetes.io/mirror annotation \u6765\u5f00\u542f Pod \u7ea7\u522b\u6d41\u91cf\u955c\u50cf\u3002 apiVersion : v1 kind : Pod metadata : name : mirror-pod namespace : ls1 annotations : ovn.kubernetes.io/mirror : \"true\" spec : containers : - name : mirror-pod image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6d41\u91cf\u955c\u50cf"},{"location":"guide/mirror/#_1","text":"\u6d41\u91cf\u955c\u50cf\u529f\u80fd\u53ef\u4ee5\u5c06\u8fdb\u51fa\u5bb9\u5668\u7f51\u7edc\u7684\u6570\u636e\u5305\u8fdb\u884c\u590d\u5236\u5230\u4e3b\u673a\u7684\u7279\u5b9a\u7f51\u5361\u3002\u7ba1\u7406\u5458\u6216\u5f00\u53d1\u8005 \u53ef\u4ee5\u901a\u8fc7\u76d1\u542c\u8fd9\u5757\u7f51\u5361\u83b7\u5f97\u5b8c\u6574\u7684\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u6765\u8fdb\u4e00\u6b65\u8fdb\u884c\u5206\u6790\uff0c\u76d1\u63a7\uff0c\u5b89\u5168\u5ba1\u8ba1\u7b49\u64cd\u4f5c\u3002 \u4e5f\u53ef\u548c\u4f20\u7edf\u7684 NPM \u5bf9\u63a5\u83b7\u53d6\u66f4\u7ec6\u7c92\u5ea6\u7684\u6d41\u91cf\u76d1\u63a7\u3002 \u6d41\u91cf\u955c\u50cf\u529f\u80fd\u4f1a\u5e26\u6765\u4e00\u5b9a\u7684\u6027\u80fd\u635f\u5931\uff0c\u6839\u636e CPU \u6027\u80fd\u4ee5\u53ca\u6d41\u91cf\u7684\u7279\u5f81\uff0c\u4f1a\u6709 5%~10% \u7684 \u989d\u5916 CPU \u6d88\u8017\u3002","title":"\u6d41\u91cf\u955c\u50cf"},{"location":"guide/mirror/#_2","text":"\u6d41\u91cf\u955c\u50cf\u529f\u80fd\u9ed8\u8ba4\u4e3a\u5173\u95ed\u72b6\u6001\uff0c\u5982\u679c\u9700\u8981\u5f00\u542f\u8bf7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u542f\u52a8\u53c2\u6570\uff1a --enable-mirror=true \uff1a \u662f\u5426\u5f00\u542f\u6d41\u91cf\u955c\u50cf\u3002 --mirror-iface=mirror0 : \u6d41\u91cf\u955c\u50cf\u6240\u590d\u5236\u5230\u7684\u7f51\u5361\u540d\u3002\u8be5\u7f51\u5361\u53ef\u4e3a\u4e3b\u673a\u4e0a\u5df2\u5b58\u5728\u7684\u4e00\u5757\u7269\u7406\u7f51\u5361\uff0c \u6b64\u65f6\u8be5\u7f51\u5361\u4f1a\u88ab\u6865\u63a5\u8fdb br-int \u7f51\u6865\uff0c\u955c\u50cf\u6d41\u91cf\u4f1a\u76f4\u63a5\u63a5\u5165\u5e95\u5c42\u4ea4\u6362\u673a\u3002\u82e5\u7f51\u5361\u540d\u4e0d\u5b58\u5728\uff0cKube-OVN \u4f1a\u81ea\u52a8 \u521b\u5efa\u4e00\u5757\u540c\u540d\u7684\u865a\u62df\u7f51\u5361\uff0c\u7ba1\u7406\u5458\u6216\u5f00\u53d1\u8005\u53ef\u4ee5\u5728\u5bbf\u4e3b\u673a\u4e0a\u901a\u8fc7\u8be5\u7f51\u5361\u83b7\u53d6\u5f53\u524d\u8282\u70b9\u6240\u6709\u6d41\u91cf\u3002\u9ed8\u8ba4\u4e3a mirror0 \u3002 \u63a5\u4e0b\u6765\u53ef\u4ee5\u7528 tcpdump \u6216\u5176\u4ed6\u6d41\u91cf\u5206\u6790\u5de5\u5177\u76d1\u542c mirror0 \u4e0a\u7684\u6d41\u91cf\uff1a tcpdump -ni mirror0","title":"\u5168\u5c40\u6d41\u91cf\u955c\u50cf\u914d\u7f6e"},{"location":"guide/mirror/#pod","text":"\u5982\u679c\u53ea\u9700\u5bf9\u90e8\u5206 Pod \u6d41\u91cf\u8fdb\u884c\u955c\u50cf\uff0c\u5219\u9700\u8981\u5173\u95ed\u5168\u5c40\u7684\u6d41\u91cf\u955c\u50cf\u529f\u80fd\uff0c\u7136\u540e\u5728\u7279\u5b9a Pod \u4e0a\u589e\u52a0 ovn.kubernetes.io/mirror annotation \u6765\u5f00\u542f Pod \u7ea7\u522b\u6d41\u91cf\u955c\u50cf\u3002 apiVersion : v1 kind : Pod metadata : name : mirror-pod namespace : ls1 annotations : ovn.kubernetes.io/mirror : \"true\" spec : containers : - name : mirror-pod image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Pod \u7ea7\u522b\u6d41\u91cf\u955c\u50cf\u914d\u7f6e"},{"location":"guide/networkpolicy-log/","text":"NetworkPolicy \u65e5\u5fd7 \u00b6 NetworkPolicy \u4e3a Kubernetes \u63d0\u4f9b\u7684\u7f51\u7edc\u7b56\u7565\u63a5\u53e3\uff0cKube-OVN \u901a\u8fc7 OVN \u7684 ACL \u8fdb\u884c\u4e86\u5b9e\u73b0\u3002 \u4f7f\u7528\u4e86 NetworkPolicy \u540e\u5982\u679c\u51fa\u73b0\u7f51\u7edc\u4e0d\u901a\u7684\u60c5\u51b5\uff0c\u96be\u4ee5\u5224\u65ad\u662f\u7f51\u7edc\u6545\u969c\u95ee\u9898\u8fd8\u662f NetworkPolicy \u89c4\u5219\u8bbe\u7f6e\u95ee\u9898\u5bfc\u81f4\u7684\u7f51\u7edc\u4e2d\u65ad\u3002 Kube-OVN \u63d0\u4f9b\u4e86 NetworkPolicy \u65e5\u5fd7\u529f\u80fd\uff0c\u5e2e\u52a9\u7ba1\u7406\u5458\u5feb\u901f\u5b9a\u4f4d NetworkPolicy Drop \u89c4\u5219\u662f\u5426\u547d\u4e2d\uff0c\u5e76\u8bb0\u5f55\u6709\u54ea\u4e9b\u975e\u6cd5\u8bbf\u95ee\u3002 NetworkPolicy \u65e5\u5fd7\u529f\u80fd\u4e00\u65e6\u5f00\u542f\uff0c\u5bf9\u6bcf\u4e2a\u547d\u4e2d Drop \u89c4\u5219\u7684\u6570\u636e\u5305\u90fd\u9700\u8981\u6253\u5370\u65e5\u5fd7\uff0c\u4f1a\u5e26\u6765\u989d\u5916\u6027\u80fd\u5f00\u9500\u3002 \u5728\u6076\u610f\u653b\u51fb\u4e0b\uff0c\u77ed\u65f6\u95f4\u5927\u91cf\u65e5\u5fd7\u53ef\u80fd\u4f1a\u8017\u5c3d CPU\u3002\u6211\u4eec\u5efa\u8bae\u5728\u751f\u4ea7\u73af\u5883\u9ed8\u8ba4\u5173\u95ed\u65e5\u5fd7\u529f\u80fd\uff0c\u5728\u9700\u8981\u6392\u67e5\u95ee\u9898\u65f6\uff0c\u52a8\u6001\u5f00\u542f\u65e5\u5fd7\u3002 \u5f00\u542f NetworkPolicy \u65e5\u5fd7 \u00b6 \u5728\u9700\u8981\u5f00\u542f\u65e5\u5fd7\u8bb0\u5f55\u7684 NetworkPolicy \u4e2d\u589e\u52a0 annotation ovn.kubernetes.io/enable_log \uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : networking.k8s.io/v1 kind : NetworkPolicy metadata : name : default-deny-ingress namespace : kube-system annotations : ovn.kubernetes.io/enable_log : \"true\" spec : podSelector : {} policyTypes : - Ingress \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728\u5bf9\u5e94 Pod \u6240\u5728\u4e3b\u673a\u7684 /var/log/ovn/ovn-controller.log \u4e2d\u89c2\u5bdf\u5230\u88ab\u4e22\u5f03\u6570\u636e\u5305\u7684\u65e5\u5fd7\uff1a # tail -f /var/log/ovn/ovn-controller.log 2022 -07-20T05:55:03.229Z | 00394 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54343 ,tp_dst = 53 2022 -07-20T05:55:06.229Z | 00395 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 44187 ,tp_dst = 53 2022 -07-20T05:55:08.230Z | 00396 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54274 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00397 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 32778 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00398 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 34188 ,tp_dst = 53 2022 -07-20T05:55:13.231Z | 00399 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 43290 ,tp_dst = 53 2022 -07-20T05:55:22.096Z | 00400 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.097Z | 00401 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.098Z | 00402 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 \u5173\u95ed NetworkPolicy \u65e5\u5fd7 \u00b6 \u5c06\u5bf9\u5e94 NetworkPolicy \u4e2d\u7684 annotation ovn.kubernetes.io/enable_log \u8bbe\u7f6e\u4e3a false \u5373\u53ef\u5173\u95ed NetworkPolicy \u65e5\u5fd7\uff1a kubectl annotate networkpolicy -n kube-system default-deny-ingress ovn.kubernetes.io/enable_log = false --overwrite \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NetworkPolicy \u65e5\u5fd7"},{"location":"guide/networkpolicy-log/#networkpolicy","text":"NetworkPolicy \u4e3a Kubernetes \u63d0\u4f9b\u7684\u7f51\u7edc\u7b56\u7565\u63a5\u53e3\uff0cKube-OVN \u901a\u8fc7 OVN \u7684 ACL \u8fdb\u884c\u4e86\u5b9e\u73b0\u3002 \u4f7f\u7528\u4e86 NetworkPolicy \u540e\u5982\u679c\u51fa\u73b0\u7f51\u7edc\u4e0d\u901a\u7684\u60c5\u51b5\uff0c\u96be\u4ee5\u5224\u65ad\u662f\u7f51\u7edc\u6545\u969c\u95ee\u9898\u8fd8\u662f NetworkPolicy \u89c4\u5219\u8bbe\u7f6e\u95ee\u9898\u5bfc\u81f4\u7684\u7f51\u7edc\u4e2d\u65ad\u3002 Kube-OVN \u63d0\u4f9b\u4e86 NetworkPolicy \u65e5\u5fd7\u529f\u80fd\uff0c\u5e2e\u52a9\u7ba1\u7406\u5458\u5feb\u901f\u5b9a\u4f4d NetworkPolicy Drop \u89c4\u5219\u662f\u5426\u547d\u4e2d\uff0c\u5e76\u8bb0\u5f55\u6709\u54ea\u4e9b\u975e\u6cd5\u8bbf\u95ee\u3002 NetworkPolicy \u65e5\u5fd7\u529f\u80fd\u4e00\u65e6\u5f00\u542f\uff0c\u5bf9\u6bcf\u4e2a\u547d\u4e2d Drop \u89c4\u5219\u7684\u6570\u636e\u5305\u90fd\u9700\u8981\u6253\u5370\u65e5\u5fd7\uff0c\u4f1a\u5e26\u6765\u989d\u5916\u6027\u80fd\u5f00\u9500\u3002 \u5728\u6076\u610f\u653b\u51fb\u4e0b\uff0c\u77ed\u65f6\u95f4\u5927\u91cf\u65e5\u5fd7\u53ef\u80fd\u4f1a\u8017\u5c3d CPU\u3002\u6211\u4eec\u5efa\u8bae\u5728\u751f\u4ea7\u73af\u5883\u9ed8\u8ba4\u5173\u95ed\u65e5\u5fd7\u529f\u80fd\uff0c\u5728\u9700\u8981\u6392\u67e5\u95ee\u9898\u65f6\uff0c\u52a8\u6001\u5f00\u542f\u65e5\u5fd7\u3002","title":"NetworkPolicy \u65e5\u5fd7"},{"location":"guide/networkpolicy-log/#networkpolicy_1","text":"\u5728\u9700\u8981\u5f00\u542f\u65e5\u5fd7\u8bb0\u5f55\u7684 NetworkPolicy \u4e2d\u589e\u52a0 annotation ovn.kubernetes.io/enable_log \uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : networking.k8s.io/v1 kind : NetworkPolicy metadata : name : default-deny-ingress namespace : kube-system annotations : ovn.kubernetes.io/enable_log : \"true\" spec : podSelector : {} policyTypes : - Ingress \u63a5\u4e0b\u6765\u53ef\u4ee5\u5728\u5bf9\u5e94 Pod \u6240\u5728\u4e3b\u673a\u7684 /var/log/ovn/ovn-controller.log \u4e2d\u89c2\u5bdf\u5230\u88ab\u4e22\u5f03\u6570\u636e\u5305\u7684\u65e5\u5fd7\uff1a # tail -f /var/log/ovn/ovn-controller.log 2022 -07-20T05:55:03.229Z | 00394 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54343 ,tp_dst = 53 2022 -07-20T05:55:06.229Z | 00395 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 44187 ,tp_dst = 53 2022 -07-20T05:55:08.230Z | 00396 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54274 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00397 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 32778 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00398 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 34188 ,tp_dst = 53 2022 -07-20T05:55:13.231Z | 00399 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 43290 ,tp_dst = 53 2022 -07-20T05:55:22.096Z | 00400 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.097Z | 00401 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.098Z | 00402 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0","title":"\u5f00\u542f NetworkPolicy \u65e5\u5fd7"},{"location":"guide/networkpolicy-log/#networkpolicy_2","text":"\u5c06\u5bf9\u5e94 NetworkPolicy \u4e2d\u7684 annotation ovn.kubernetes.io/enable_log \u8bbe\u7f6e\u4e3a false \u5373\u53ef\u5173\u95ed NetworkPolicy \u65e5\u5fd7\uff1a kubectl annotate networkpolicy -n kube-system default-deny-ingress ovn.kubernetes.io/enable_log = false --overwrite \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5173\u95ed NetworkPolicy \u65e5\u5fd7"},{"location":"guide/prometheus-grafana/","text":"\u914d\u7f6e\u76d1\u63a7\u548c\u9762\u677f \u00b6 Kube-OVN \u53ef\u4ee5\u5c06\u7f51\u7edc\u63a7\u5236\u5e73\u9762\u4fe1\u606f\u4ee5\u53ca\u7f51\u7edc\u6570\u636e\u5e73\u9762\u8d28\u91cf\u4fe1\u606f\u6307\u6807\u4ee5 Prometheus \u6240\u652f\u6301\u7684\u683c\u5f0f\u5bf9\u5916\u8f93\u51fa\u3002 \u6211\u4eec\u4f7f\u7528 kube-prometheus \u6240\u63d0\u4f9b\u7684 CRD \u6765\u5b9a\u4e49\u76f8\u5e94\u7684 Prometheus \u76d1\u63a7\u89c4\u5219\u3002 \u7528\u6237\u9700\u8981\u9884\u5148\u5b89\u88c5 kube-prometheus \u6765\u542f\u7528\u76f8\u5173\u7684 CRD\u3002Kube-OVN \u6240\u652f\u6301\u7684\u5168\u90e8\u76d1\u63a7\u6307\u6807\u8bf7\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002 \u5982\u679c\u4f7f\u7528\u539f\u751f Prometheus \u8bf7\u53c2\u8003 \u914d\u7f6e\u539f\u751f Prometheus \u83b7\u53d6\u76d1\u63a7\u6570\u636e \u8fdb\u884c\u914d\u7f6e\u3002 \u5b89\u88c5 Prometheus Monitor \u00b6 Kube-OVN \u4f7f\u7528 Prometheus Monitor CRD \u6765\u7ba1\u7406\u76d1\u63a7\u8f93\u51fa\uff1a # \u7f51\u54af\u8d28\u91cf\u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-monitor.yaml # kube-ovn-controller \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-monitor.yaml # kube-ovn-cni \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-monitor.yaml # ovn \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-monitor.yaml Prometheus \u62c9\u53d6\u76d1\u63a7\u65f6\u95f4\u95f4\u9694\u9ed8\u8ba4\u4e3a 15s\uff0c\u5982\u679c\u9700\u8981\u8c03\u6574\u9700\u8981\u4fee\u6539 yaml \u4e2d\u7684 interval \u5b57\u6bb5\u3002 \u52a0\u8f7d Grafana \u9762\u677f \u00b6 Kube-OVN \u8fd8\u63d0\u4f9b\u4e86\u9884\u5148\u5b9a\u4e49\u597d\u7684 Grafana Dashboard \u5c55\u793a\u63a7\u5236\u5e73\u9762\u548c\u6570\u636e\u5e73\u9762\u76f8\u5173\u4fe1\u606f\u3002 \u4e0b\u8f7d\u5bf9\u5e94 Dashboard \u6a21\u677f\uff1a # \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-grafana.json # kube-ovn-controller \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-grafana.json # kube-ovn-cni \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-grafana.json # ovn \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-grafana.json # ovs \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovs-grafana.json \u5728 Grafana \u4e2d\u5bfc\u5165\u6a21\u677f\uff0c\u5e76\u5c06\u6570\u636e\u6e90\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 Prometheus \u5373\u53ef\u770b\u5230\u5982\u4e0b Dashboard\uff1a kube-ovn-controller \u8fd0\u884c\u72b6\u51b5\u76f8\u5173\u9762\u677f\uff1a kube-ovn-pinger \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u9762\u677f\uff1a kube-ovn-cni \u8fd0\u884c\u72b6\u51b5\u76f8\u5173\u9762\u677f\uff1a \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u914d\u7f6e\u76d1\u63a7\u548c\u9762\u677f"},{"location":"guide/prometheus-grafana/#_1","text":"Kube-OVN \u53ef\u4ee5\u5c06\u7f51\u7edc\u63a7\u5236\u5e73\u9762\u4fe1\u606f\u4ee5\u53ca\u7f51\u7edc\u6570\u636e\u5e73\u9762\u8d28\u91cf\u4fe1\u606f\u6307\u6807\u4ee5 Prometheus \u6240\u652f\u6301\u7684\u683c\u5f0f\u5bf9\u5916\u8f93\u51fa\u3002 \u6211\u4eec\u4f7f\u7528 kube-prometheus \u6240\u63d0\u4f9b\u7684 CRD \u6765\u5b9a\u4e49\u76f8\u5e94\u7684 Prometheus \u76d1\u63a7\u89c4\u5219\u3002 \u7528\u6237\u9700\u8981\u9884\u5148\u5b89\u88c5 kube-prometheus \u6765\u542f\u7528\u76f8\u5173\u7684 CRD\u3002Kube-OVN \u6240\u652f\u6301\u7684\u5168\u90e8\u76d1\u63a7\u6307\u6807\u8bf7\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002 \u5982\u679c\u4f7f\u7528\u539f\u751f Prometheus \u8bf7\u53c2\u8003 \u914d\u7f6e\u539f\u751f Prometheus \u83b7\u53d6\u76d1\u63a7\u6570\u636e \u8fdb\u884c\u914d\u7f6e\u3002","title":"\u914d\u7f6e\u76d1\u63a7\u548c\u9762\u677f"},{"location":"guide/prometheus-grafana/#prometheus-monitor","text":"Kube-OVN \u4f7f\u7528 Prometheus Monitor CRD \u6765\u7ba1\u7406\u76d1\u63a7\u8f93\u51fa\uff1a # \u7f51\u54af\u8d28\u91cf\u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-monitor.yaml # kube-ovn-controller \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-monitor.yaml # kube-ovn-cni \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-monitor.yaml # ovn \u76f8\u5173\u76d1\u63a7\u6307\u6807 kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-monitor.yaml Prometheus \u62c9\u53d6\u76d1\u63a7\u65f6\u95f4\u95f4\u9694\u9ed8\u8ba4\u4e3a 15s\uff0c\u5982\u679c\u9700\u8981\u8c03\u6574\u9700\u8981\u4fee\u6539 yaml \u4e2d\u7684 interval \u5b57\u6bb5\u3002","title":"\u5b89\u88c5 Prometheus Monitor"},{"location":"guide/prometheus-grafana/#grafana","text":"Kube-OVN \u8fd8\u63d0\u4f9b\u4e86\u9884\u5148\u5b9a\u4e49\u597d\u7684 Grafana Dashboard \u5c55\u793a\u63a7\u5236\u5e73\u9762\u548c\u6570\u636e\u5e73\u9762\u76f8\u5173\u4fe1\u606f\u3002 \u4e0b\u8f7d\u5bf9\u5e94 Dashboard \u6a21\u677f\uff1a # \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-grafana.json # kube-ovn-controller \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-grafana.json # kube-ovn-cni \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-grafana.json # ovn \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-grafana.json # ovs \u76f8\u5173\u9762\u677f wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovs-grafana.json \u5728 Grafana \u4e2d\u5bfc\u5165\u6a21\u677f\uff0c\u5e76\u5c06\u6570\u636e\u6e90\u8bbe\u7f6e\u4e3a\u5bf9\u5e94\u7684 Prometheus \u5373\u53ef\u770b\u5230\u5982\u4e0b Dashboard\uff1a kube-ovn-controller \u8fd0\u884c\u72b6\u51b5\u76f8\u5173\u9762\u677f\uff1a kube-ovn-pinger \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u9762\u677f\uff1a kube-ovn-cni \u8fd0\u884c\u72b6\u51b5\u76f8\u5173\u9762\u677f\uff1a \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u52a0\u8f7d Grafana \u9762\u677f"},{"location":"guide/prometheus/","text":"\u914d\u7f6e\u539f\u751f Prometheus \u83b7\u53d6\u76d1\u63a7\u6570\u636e \u00b6 Kube-OVN \u63d0\u4f9b\u4e86\u4e30\u5bcc\u7684\u76d1\u63a7\u6570\u636e\uff0c\u7528\u4e8e OVN/OVS \u5065\u5eb7\u72b6\u6001\u68c0\u67e5\uff0c\u4ee5\u53ca\u5bb9\u5668\u7f51\u7edc\u548c\u4e3b\u673a\u7f51\u7edc\u7684\u8fde\u901a\u6027\u68c0\u67e5\u3002Kube-OVN \u914d\u7f6e\u4e86 ServiceMonitor\uff0c\u53ef\u4ee5\u7528\u4e8e Prometheus \u52a8\u6001\u83b7\u53d6\u76d1\u63a7\u6307\u6807\u3002 \u5728\u67d0\u4e9b\u60c5\u51b5\u4e0b\uff0c\u53ea\u5b89\u88c5\u4e86 Prometheus Server\uff0c\u6ca1\u6709\u5b89\u88c5\u5176\u4ed6\u7684\u7ec4\u4ef6\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 Prometheus \u7684\u914d\u7f6e\uff0c\u52a8\u6001\u83b7\u53d6\u96c6\u7fa4\u73af\u5883\u7684\u76d1\u63a7\u6570\u636e\u3002 Prometheus \u914d\u7f6e \u00b6 \u4ee5\u4e0b\u7684\u914d\u7f6e\u6587\u6863\uff0c\u53c2\u8003\u81ea Prometheus \u670d\u52a1\u53d1\u73b0 \u3002 \u6743\u9650\u914d\u7f6e \u00b6 Prometheus \u90e8\u7f72\u5728\u96c6\u7fa4\u5185\uff0c\u9700\u8981\u901a\u8fc7 k8s apiserver \u6765\u8bbf\u95ee\u96c6\u7fa4\u5185\u7684\u8d44\u6e90\uff0c\u4ece\u800c\u5b9e\u73b0\u67e5\u8be2\u4e1a\u52a1\u7684\u76d1\u63a7\u6570\u636e\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u914d\u7f6e Prometheus \u9700\u8981\u7684\u6743\u9650\uff1a apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : prometheus rules : - apiGroups : [ \"\" ] resources : - nodes - nodes/proxy - services - endpoints - pods verbs : [ \"get\" , \"list\" , \"watch\" ] - apiGroups : - extensions resources : - ingresses verbs : [ \"get\" , \"list\" , \"watch\" ] - nonResourceURLs : [ \"/metrics\" ] verbs : [ \"get\" ] --- apiVersion : v1 kind : ServiceAccount metadata : name : prometheus namespace : default --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : prometheus roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : prometheus subjects : - kind : ServiceAccount name : prometheus namespace : default Prometheus \u914d\u7f6e\u6587\u4ef6 \u00b6 Prometheus \u7684\u542f\u52a8\uff0c\u4f9d\u8d56\u4e8e\u914d\u7f6e\u6587\u4ef6 prometheus.yml\uff0c\u53ef\u4ee5\u5c06\u8be5\u6587\u4ef6\u5185\u5bb9\u914d\u7f6e\u5728 ConfigMap \u5185\uff0c\u52a8\u6001\u6302\u8f7d\u5230 Pod \u4e2d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa Prometheus \u4f7f\u7528\u7684 ConfigMap \u6587\u4ef6\uff1a apiVersion : v1 kind : ConfigMap metadata : name : prometheus-config data : prometheus.yml : |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'kubernetes-nodes' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node - job_name: 'kubernetes-service' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: service - job_name: 'kubernetes-endpoints' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints - job_name: 'kubernetes-ingress' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: ingress - job_name: 'kubernetes-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: pod Prometheus \u63d0\u4f9b\u4e86\u57fa\u4e8e\u89d2\u8272\u67e5\u8be2 Kubernetes \u8d44\u6e90\u76d1\u63a7\u7684\u64cd\u4f5c\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u67e5\u770b\u5b98\u65b9\u6587\u6863 kubernetes_sd_config \u3002 \u5728 Kubernetes \u96c6\u7fa4\u4e2d\uff0cPrometheus \u652f\u6301\u67e5\u8be2\u76d1\u63a7\u6307\u6807\u7684\u89d2\u8272\u5305\u542b node\u3001service\u3001pod\u3001endpoints \u548c ingress\u3002\u5728 ConfigMap \u914d\u7f6e\u6587\u4ef6\u4e2d\u7ed9\u51fa\u4e86\u4ee5\u4e0a\u5168\u90e8\u8d44\u6e90\u7684\u76d1\u63a7\u67e5\u8be2\u914d\u7f6e\u793a\u4f8b\uff0c\u53ef\u4ee5\u6839\u636e\u9700\u8981\u9009\u62e9\u914d\u7f6e\u3002 Prometheus \u90e8\u7f72 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml \u6587\u4ef6\uff0c\u90e8\u7f72 Prometheus Server\uff1a apiVersion : apps/v1 kind : Deployment metadata : labels : app : prometheus name : prometheus namespace : default spec : replicas : 1 selector : matchLabels : app : prometheus strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : prometheus spec : serviceAccountName : prometheus serviceAccount : prometheus containers : - image : docker.io/prom/prometheus:latest imagePullPolicy : IfNotPresent name : prometheus command : - \"/bin/prometheus\" args : - \"--config.file=/etc/prometheus/prometheus.yml\" ports : - containerPort : 9090 protocol : TCP volumeMounts : - mountPath : \"/etc/prometheus\" name : prometheus-config volumes : - name : prometheus-config configMap : name : prometheus-config \u5728\u90e8\u7f72\u5b8c Prometheus \u4e4b\u540e\uff0c\u53c2\u8003\u4ee5\u4e0b yaml \u6587\u4ef6\uff0c\u90e8\u7f72 Prometheus Service\uff1a kind : Service apiVersion : v1 metadata : name : prometheus namespace : default labels : name : prometheus spec : ports : - name : test protocol : TCP port : 9090 targetPort : 9090 type : NodePort selector : app : prometheus sessionAffinity : None \u5c06 Prometheus \u901a\u8fc7 NodePort \u66b4\u9732\u540e\uff0c\u5373\u53ef\u901a\u8fc7\u8282\u70b9\u6765\u8bbf\u95ee Prometheus\u3002 Prometheus \u76d1\u63a7\u6570\u636e\u9a8c\u8bc1 \u00b6 \u67e5\u770b\u73af\u5883\u4e0a Prometheus \u76f8\u5173\u7684\u4fe1\u606f\uff1a # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kubernetes ClusterIP 10 .4.0.1 <none> 443 /TCP 8d prometheus NodePort 10 .4.102.222 <none> 9090 :32611/TCP 8d # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-7544b6b84d-v9m8s 1 /1 Running 0 3d5h 10 .3.0.7 192 .168.137.219 <none> <none> # kubectl get endpoints -o wide NAME ENDPOINTS AGE kubernetes 192 .168.136.228:6443,192.168.136.232:6443,192.168.137.219:6443 8d prometheus 10 .3.0.7:9090 8d \u901a\u8fc7 NodePort \u8bbf\u95ee Prometheus\uff0c\u67e5\u770b Status/Service Discovery \u52a8\u6001\u67e5\u8be2\u5230\u7684\u6570\u636e\uff1a \u53ef\u4ee5\u770b\u5230\u5f53\u524d\u53ef\u4ee5\u67e5\u8be2\u5230\u96c6\u7fa4\u4e0a\u5168\u90e8\u7684 Service \u6570\u636e\u4fe1\u606f\u3002 \u914d\u7f6e\u67e5\u8be2\u6307\u5b9a\u7684\u8d44\u6e90 \u00b6 \u4ee5\u4e0a\u7684 ConfigMap \u914d\u7f6e\u4e2d\uff0c\u6ca1\u6709\u6dfb\u52a0\u8fc7\u6ee4\u6761\u4ef6\uff0c\u67e5\u8be2\u4e86\u6240\u6709\u7684\u8d44\u6e90\u6570\u636e\u3002\u5982\u679c\u53ea\u9700\u8981\u67d0\u4e2a\u89d2\u8272\u7684\u8d44\u6e90\u6570\u636e\uff0c\u5219\u53ef\u4ee5\u6dfb\u52a0\u8fc7\u6ee4\u6761\u4ef6\u3002 \u4ee5 Service \u4e3a\u4f8b\uff0c\u4fee\u6539 ConfigMap \u5185\u5bb9\uff0c\u53ea\u67e5\u8be2\u5173\u5fc3\u7684 Service \u76d1\u63a7\u6570\u636e\u3002 - job_name : 'kubernetes-service' tls_config : ca_file : /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file : /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs : - role : service relabel_configs : - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_scrape ] action : \"keep\" regex : \"true\" - action : labelmap regex : __meta_kubernetes_service_label_(.+) - source_labels : [ __meta_kubernetes_namespace ] target_label : kubernetes_namespace - source_labels : [ __meta_kubernetes_service_name ] target_label : kubernetes_service_name - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_path ] action : replace target_label : __metrics_path__ regex : \"(.+)\" Service \u9ed8\u8ba4\u76d1\u63a7\u8def\u5f84\u4e3a /metrics\u3002\u5982\u679c Service \u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\u662f\u5176\u4ed6\u7684\u8def\u5f84\uff0c\u53ef\u4ee5\u901a\u8fc7\u7ed9 Service \u6dfb\u52a0 annotation prometheus.io/path \u6765\u6307\u5b9a\u91c7\u96c6\u8def\u5f84\u3002 \u5e94\u7528\u4ee5\u4e0a yaml\uff0c\u66f4\u65b0 ConfigMap \u4fe1\u606f\uff0c\u91cd\u5efa Prometheus Pod\uff0c\u4f7f\u914d\u7f6e\u751f\u6548\u3002 \u67e5\u770b kube-system Namespace \u4e0b\u7684 Service \u4fe1\u606f\uff1a # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kube-dns ClusterIP 10 .4.0.10 <none> 53 /UDP,53/TCP,9153/TCP 13d kube-ovn-cni ClusterIP 10 .4.228.60 <none> 10665 /TCP 13d kube-ovn-controller ClusterIP 10 .4.172.213 <none> 10660 /TCP 13d kube-ovn-monitor ClusterIP 10 .4.242.9 <none> 10661 /TCP 13d kube-ovn-pinger ClusterIP 10 .4.122.52 <none> 8080 /TCP 13d ovn-nb ClusterIP 10 .4.80.213 <none> 6641 /TCP 13d ovn-northd ClusterIP 10 .4.126.234 <none> 6643 /TCP 13d ovn-sb ClusterIP 10 .4.216.249 <none> 6642 /TCP 13d \u7ed9 Service \u6dfb\u52a0 annotation prometheus.io/scrape=\"true\" \uff1a # kubectl annotate svc -n kube-system kube-ovn-cni prometheus.io/scrape=true service/kube-ovn-cni annotated # kubectl annotate svc -n kube-system kube-ovn-controller prometheus.io/scrape=true service/kube-ovn-controller annotated # kubectl annotate svc -n kube-system kube-ovn-monitor prometheus.io/scrape=true service/kube-ovn-monitor annotated # kubectl annotate svc -n kube-system kube-ovn-pinger prometheus.io/scrape=true service/kube-ovn-pinger annotated \u67e5\u770b\u914d\u7f6e\u540e\u7684 Service \u4fe1\u606f\uff1a # kubectl get svc -o yaml -n kube-system kube-ovn-controller apiVersion: v1 kind: Service metadata: annotations: helm.sh/chart-version: v3.10.0-alpha.55 helm.sh/original-name: kube-ovn-controller ovn.kubernetes.io/vpc: ovn-cluster prometheus.io/scrape: \"true\" // \u6dfb\u52a0\u7684 annotation labels: app: kube-ovn-controller name: kube-ovn-controller namespace: kube-system spec: clusterIP: 10 .4.172.213 clusterIPs: - 10 .4.172.213 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: metrics port: 10660 protocol: TCP targetPort: 10660 selector: app: kube-ovn-controller sessionAffinity: None type: ClusterIP status: loadBalancer: {} \u67e5\u770b Prometheus Status Targets \u4fe1\u606f\uff0c\u53ef\u4ee5\u770b\u5230\u53ea\u6709\u6dfb\u52a0\u4e86 annotation \u7684 Service \u88ab\u8fc7\u6ee4\u51fa\u6765\uff1a \u66f4\u591a\u5173\u4e8e relabel \u6dfb\u52a0\u8fc7\u6ee4\u53c2\u6570\u7684\u4fe1\u606f\uff0c\u53ef\u4ee5\u53c2\u8003 Prometheus-Relabel \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u914d\u7f6e\u539f\u751f Prometheus"},{"location":"guide/prometheus/#prometheus","text":"Kube-OVN \u63d0\u4f9b\u4e86\u4e30\u5bcc\u7684\u76d1\u63a7\u6570\u636e\uff0c\u7528\u4e8e OVN/OVS \u5065\u5eb7\u72b6\u6001\u68c0\u67e5\uff0c\u4ee5\u53ca\u5bb9\u5668\u7f51\u7edc\u548c\u4e3b\u673a\u7f51\u7edc\u7684\u8fde\u901a\u6027\u68c0\u67e5\u3002Kube-OVN \u914d\u7f6e\u4e86 ServiceMonitor\uff0c\u53ef\u4ee5\u7528\u4e8e Prometheus \u52a8\u6001\u83b7\u53d6\u76d1\u63a7\u6307\u6807\u3002 \u5728\u67d0\u4e9b\u60c5\u51b5\u4e0b\uff0c\u53ea\u5b89\u88c5\u4e86 Prometheus Server\uff0c\u6ca1\u6709\u5b89\u88c5\u5176\u4ed6\u7684\u7ec4\u4ef6\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 Prometheus \u7684\u914d\u7f6e\uff0c\u52a8\u6001\u83b7\u53d6\u96c6\u7fa4\u73af\u5883\u7684\u76d1\u63a7\u6570\u636e\u3002","title":"\u914d\u7f6e\u539f\u751f Prometheus \u83b7\u53d6\u76d1\u63a7\u6570\u636e"},{"location":"guide/prometheus/#prometheus_1","text":"\u4ee5\u4e0b\u7684\u914d\u7f6e\u6587\u6863\uff0c\u53c2\u8003\u81ea Prometheus \u670d\u52a1\u53d1\u73b0 \u3002","title":"Prometheus \u914d\u7f6e"},{"location":"guide/prometheus/#_1","text":"Prometheus \u90e8\u7f72\u5728\u96c6\u7fa4\u5185\uff0c\u9700\u8981\u901a\u8fc7 k8s apiserver \u6765\u8bbf\u95ee\u96c6\u7fa4\u5185\u7684\u8d44\u6e90\uff0c\u4ece\u800c\u5b9e\u73b0\u67e5\u8be2\u4e1a\u52a1\u7684\u76d1\u63a7\u6570\u636e\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u914d\u7f6e Prometheus \u9700\u8981\u7684\u6743\u9650\uff1a apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : prometheus rules : - apiGroups : [ \"\" ] resources : - nodes - nodes/proxy - services - endpoints - pods verbs : [ \"get\" , \"list\" , \"watch\" ] - apiGroups : - extensions resources : - ingresses verbs : [ \"get\" , \"list\" , \"watch\" ] - nonResourceURLs : [ \"/metrics\" ] verbs : [ \"get\" ] --- apiVersion : v1 kind : ServiceAccount metadata : name : prometheus namespace : default --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : prometheus roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : prometheus subjects : - kind : ServiceAccount name : prometheus namespace : default","title":"\u6743\u9650\u914d\u7f6e"},{"location":"guide/prometheus/#prometheus_2","text":"Prometheus \u7684\u542f\u52a8\uff0c\u4f9d\u8d56\u4e8e\u914d\u7f6e\u6587\u4ef6 prometheus.yml\uff0c\u53ef\u4ee5\u5c06\u8be5\u6587\u4ef6\u5185\u5bb9\u914d\u7f6e\u5728 ConfigMap \u5185\uff0c\u52a8\u6001\u6302\u8f7d\u5230 Pod \u4e2d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa Prometheus \u4f7f\u7528\u7684 ConfigMap \u6587\u4ef6\uff1a apiVersion : v1 kind : ConfigMap metadata : name : prometheus-config data : prometheus.yml : |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'kubernetes-nodes' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node - job_name: 'kubernetes-service' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: service - job_name: 'kubernetes-endpoints' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints - job_name: 'kubernetes-ingress' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: ingress - job_name: 'kubernetes-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: pod Prometheus \u63d0\u4f9b\u4e86\u57fa\u4e8e\u89d2\u8272\u67e5\u8be2 Kubernetes \u8d44\u6e90\u76d1\u63a7\u7684\u64cd\u4f5c\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u67e5\u770b\u5b98\u65b9\u6587\u6863 kubernetes_sd_config \u3002 \u5728 Kubernetes \u96c6\u7fa4\u4e2d\uff0cPrometheus \u652f\u6301\u67e5\u8be2\u76d1\u63a7\u6307\u6807\u7684\u89d2\u8272\u5305\u542b node\u3001service\u3001pod\u3001endpoints \u548c ingress\u3002\u5728 ConfigMap \u914d\u7f6e\u6587\u4ef6\u4e2d\u7ed9\u51fa\u4e86\u4ee5\u4e0a\u5168\u90e8\u8d44\u6e90\u7684\u76d1\u63a7\u67e5\u8be2\u914d\u7f6e\u793a\u4f8b\uff0c\u53ef\u4ee5\u6839\u636e\u9700\u8981\u9009\u62e9\u914d\u7f6e\u3002","title":"Prometheus \u914d\u7f6e\u6587\u4ef6"},{"location":"guide/prometheus/#prometheus_3","text":"\u53c2\u8003\u4ee5\u4e0b yaml \u6587\u4ef6\uff0c\u90e8\u7f72 Prometheus Server\uff1a apiVersion : apps/v1 kind : Deployment metadata : labels : app : prometheus name : prometheus namespace : default spec : replicas : 1 selector : matchLabels : app : prometheus strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : prometheus spec : serviceAccountName : prometheus serviceAccount : prometheus containers : - image : docker.io/prom/prometheus:latest imagePullPolicy : IfNotPresent name : prometheus command : - \"/bin/prometheus\" args : - \"--config.file=/etc/prometheus/prometheus.yml\" ports : - containerPort : 9090 protocol : TCP volumeMounts : - mountPath : \"/etc/prometheus\" name : prometheus-config volumes : - name : prometheus-config configMap : name : prometheus-config \u5728\u90e8\u7f72\u5b8c Prometheus \u4e4b\u540e\uff0c\u53c2\u8003\u4ee5\u4e0b yaml \u6587\u4ef6\uff0c\u90e8\u7f72 Prometheus Service\uff1a kind : Service apiVersion : v1 metadata : name : prometheus namespace : default labels : name : prometheus spec : ports : - name : test protocol : TCP port : 9090 targetPort : 9090 type : NodePort selector : app : prometheus sessionAffinity : None \u5c06 Prometheus \u901a\u8fc7 NodePort \u66b4\u9732\u540e\uff0c\u5373\u53ef\u901a\u8fc7\u8282\u70b9\u6765\u8bbf\u95ee Prometheus\u3002","title":"Prometheus \u90e8\u7f72"},{"location":"guide/prometheus/#prometheus_4","text":"\u67e5\u770b\u73af\u5883\u4e0a Prometheus \u76f8\u5173\u7684\u4fe1\u606f\uff1a # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kubernetes ClusterIP 10 .4.0.1 <none> 443 /TCP 8d prometheus NodePort 10 .4.102.222 <none> 9090 :32611/TCP 8d # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-7544b6b84d-v9m8s 1 /1 Running 0 3d5h 10 .3.0.7 192 .168.137.219 <none> <none> # kubectl get endpoints -o wide NAME ENDPOINTS AGE kubernetes 192 .168.136.228:6443,192.168.136.232:6443,192.168.137.219:6443 8d prometheus 10 .3.0.7:9090 8d \u901a\u8fc7 NodePort \u8bbf\u95ee Prometheus\uff0c\u67e5\u770b Status/Service Discovery \u52a8\u6001\u67e5\u8be2\u5230\u7684\u6570\u636e\uff1a \u53ef\u4ee5\u770b\u5230\u5f53\u524d\u53ef\u4ee5\u67e5\u8be2\u5230\u96c6\u7fa4\u4e0a\u5168\u90e8\u7684 Service \u6570\u636e\u4fe1\u606f\u3002","title":"Prometheus \u76d1\u63a7\u6570\u636e\u9a8c\u8bc1"},{"location":"guide/prometheus/#_2","text":"\u4ee5\u4e0a\u7684 ConfigMap \u914d\u7f6e\u4e2d\uff0c\u6ca1\u6709\u6dfb\u52a0\u8fc7\u6ee4\u6761\u4ef6\uff0c\u67e5\u8be2\u4e86\u6240\u6709\u7684\u8d44\u6e90\u6570\u636e\u3002\u5982\u679c\u53ea\u9700\u8981\u67d0\u4e2a\u89d2\u8272\u7684\u8d44\u6e90\u6570\u636e\uff0c\u5219\u53ef\u4ee5\u6dfb\u52a0\u8fc7\u6ee4\u6761\u4ef6\u3002 \u4ee5 Service \u4e3a\u4f8b\uff0c\u4fee\u6539 ConfigMap \u5185\u5bb9\uff0c\u53ea\u67e5\u8be2\u5173\u5fc3\u7684 Service \u76d1\u63a7\u6570\u636e\u3002 - job_name : 'kubernetes-service' tls_config : ca_file : /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file : /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs : - role : service relabel_configs : - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_scrape ] action : \"keep\" regex : \"true\" - action : labelmap regex : __meta_kubernetes_service_label_(.+) - source_labels : [ __meta_kubernetes_namespace ] target_label : kubernetes_namespace - source_labels : [ __meta_kubernetes_service_name ] target_label : kubernetes_service_name - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_path ] action : replace target_label : __metrics_path__ regex : \"(.+)\" Service \u9ed8\u8ba4\u76d1\u63a7\u8def\u5f84\u4e3a /metrics\u3002\u5982\u679c Service \u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\u662f\u5176\u4ed6\u7684\u8def\u5f84\uff0c\u53ef\u4ee5\u901a\u8fc7\u7ed9 Service \u6dfb\u52a0 annotation prometheus.io/path \u6765\u6307\u5b9a\u91c7\u96c6\u8def\u5f84\u3002 \u5e94\u7528\u4ee5\u4e0a yaml\uff0c\u66f4\u65b0 ConfigMap \u4fe1\u606f\uff0c\u91cd\u5efa Prometheus Pod\uff0c\u4f7f\u914d\u7f6e\u751f\u6548\u3002 \u67e5\u770b kube-system Namespace \u4e0b\u7684 Service \u4fe1\u606f\uff1a # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kube-dns ClusterIP 10 .4.0.10 <none> 53 /UDP,53/TCP,9153/TCP 13d kube-ovn-cni ClusterIP 10 .4.228.60 <none> 10665 /TCP 13d kube-ovn-controller ClusterIP 10 .4.172.213 <none> 10660 /TCP 13d kube-ovn-monitor ClusterIP 10 .4.242.9 <none> 10661 /TCP 13d kube-ovn-pinger ClusterIP 10 .4.122.52 <none> 8080 /TCP 13d ovn-nb ClusterIP 10 .4.80.213 <none> 6641 /TCP 13d ovn-northd ClusterIP 10 .4.126.234 <none> 6643 /TCP 13d ovn-sb ClusterIP 10 .4.216.249 <none> 6642 /TCP 13d \u7ed9 Service \u6dfb\u52a0 annotation prometheus.io/scrape=\"true\" \uff1a # kubectl annotate svc -n kube-system kube-ovn-cni prometheus.io/scrape=true service/kube-ovn-cni annotated # kubectl annotate svc -n kube-system kube-ovn-controller prometheus.io/scrape=true service/kube-ovn-controller annotated # kubectl annotate svc -n kube-system kube-ovn-monitor prometheus.io/scrape=true service/kube-ovn-monitor annotated # kubectl annotate svc -n kube-system kube-ovn-pinger prometheus.io/scrape=true service/kube-ovn-pinger annotated \u67e5\u770b\u914d\u7f6e\u540e\u7684 Service \u4fe1\u606f\uff1a # kubectl get svc -o yaml -n kube-system kube-ovn-controller apiVersion: v1 kind: Service metadata: annotations: helm.sh/chart-version: v3.10.0-alpha.55 helm.sh/original-name: kube-ovn-controller ovn.kubernetes.io/vpc: ovn-cluster prometheus.io/scrape: \"true\" // \u6dfb\u52a0\u7684 annotation labels: app: kube-ovn-controller name: kube-ovn-controller namespace: kube-system spec: clusterIP: 10 .4.172.213 clusterIPs: - 10 .4.172.213 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: metrics port: 10660 protocol: TCP targetPort: 10660 selector: app: kube-ovn-controller sessionAffinity: None type: ClusterIP status: loadBalancer: {} \u67e5\u770b Prometheus Status Targets \u4fe1\u606f\uff0c\u53ef\u4ee5\u770b\u5230\u53ea\u6709\u6dfb\u52a0\u4e86 annotation \u7684 Service \u88ab\u8fc7\u6ee4\u51fa\u6765\uff1a \u66f4\u591a\u5173\u4e8e relabel \u6dfb\u52a0\u8fc7\u6ee4\u53c2\u6570\u7684\u4fe1\u606f\uff0c\u53ef\u4ee5\u53c2\u8003 Prometheus-Relabel \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u914d\u7f6e\u67e5\u8be2\u6307\u5b9a\u7684\u8d44\u6e90"},{"location":"guide/qos/","text":"\u5bb9\u5668\u7f51\u7edc QoS \u914d\u7f6e \u00b6 Kube-OVN \u652f\u6301\u57fa\u4e8e\u5355\u4e2a Pod \u7684\u4e24\u79cd\u4e0d\u540c\u7c7b\u578b\u7684 QoS\uff1a \u6700\u5927\u5e26\u5bbd\u9650\u5236 QoS\u3002 linux-netem \uff0c\u6a21\u62df\u8bbe\u5907\u5e72\u6270\u4e22\u5305\u7b49\u7684 QoS\uff0c\u53ef\u7528\u4e8e\u6a21\u62df\u6d4b\u8bd5\u3002 \u76ee\u524d\u53ea\u652f\u6301 Pod \u7ea7\u522b QoS \u4e0d\u652f\u6301 Namespace \u6216 Subnet \u7ea7\u522b\u7684 QoS \u9650\u5236\u3002 \u57fa\u4e8e\u6700\u5927\u5e26\u5bbd\u9650\u5236\u7684 QoS \u00b6 \u8be5\u7c7b\u578b\u7684 QoS \u53ef\u4ee5\u901a\u8fc7 Pod annotation \u52a8\u6001\u8fdb\u884c\u914d\u7f6e\uff0c\u53ef\u4ee5\u5728\u4e0d\u4e2d\u65ad Pod \u8fd0\u884c\u7684\u60c5\u51b5\u4e0b\u8fdb\u884c\u8c03\u6574\u3002 \u5e26\u5bbd\u9650\u901f\u7684\u5355\u4f4d\u4e3a Mbit/s \u3002 apiVersion : v1 kind : Pod metadata : name : qos namespace : ls1 annotations : ovn.kubernetes.io/ingress_rate : \"3\" ovn.kubernetes.io/egress_rate : \"1\" spec : containers : - name : qos image : docker.io/library/nginx:alpine \u4f7f\u7528 annotation \u52a8\u6001\u8c03\u6574 QoS\uff1a kubectl annotate --overwrite pod nginx-74d5899f46-d7qkn ovn.kubernetes.io/ingress_rate = 3 \u6d4b\u8bd5 QoS \u8c03\u6574 \u00b6 \u90e8\u7f72\u6027\u80fd\u6d4b\u8bd5\u9700\u8981\u7684\u5bb9\u5668\uff1a kind : DaemonSet apiVersion : apps/v1 metadata : name : perf namespace : ls1 labels : app : perf spec : selector : matchLabels : app : perf template : metadata : labels : app : perf spec : containers : - name : nginx image : docker.io/kubeovn/perf \u8fdb\u5165\u5176\u4e2d\u4e00\u4e2a Pod \u5e76\u5f00\u542f iperf3 server\uff1a # kubectl exec -it perf-4n4gt -n ls1 sh # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- \u8fdb\u5165\u53e6\u4e00\u4e2a Pod \u8bf7\u6c42\u4e4b\u524d\u7684 Pod\uff1a # kubectl exec -it perf-d4mqc -n ls1 sh # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 51544 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 86 .4 MBytes 725 Mbits/sec 3 350 KBytes [ 4 ] 1 .00-2.00 sec 89 .9 MBytes 754 Mbits/sec 118 473 KBytes [ 4 ] 2 .00-3.00 sec 101 MBytes 848 Mbits/sec 184 586 KBytes [ 4 ] 3 .00-4.00 sec 104 MBytes 875 Mbits/sec 217 671 KBytes [ 4 ] 4 .00-5.00 sec 111 MBytes 935 Mbits/sec 175 772 KBytes [ 4 ] 5 .00-6.00 sec 100 MBytes 840 Mbits/sec 658 598 KBytes [ 4 ] 6 .00-7.00 sec 106 MBytes 890 Mbits/sec 742 668 KBytes [ 4 ] 7 .00-8.00 sec 102 MBytes 857 Mbits/sec 764 724 KBytes [ 4 ] 8 .00-9.00 sec 97 .4 MBytes 817 Mbits/sec 1175 764 KBytes [ 4 ] 9 .00-10.00 sec 111 MBytes 934 Mbits/sec 1083 838 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 1010 MBytes 848 Mbits/sec 5119 sender [ 4 ] 0 .00-10.00 sec 1008 MBytes 846 Mbits/sec receiver iperf Done. \u4fee\u6539\u7b2c\u4e00\u4e2a Pod \u7684\u5165\u53e3\u5e26\u5bbd QoS\uff1a kubectl annotate --overwrite pod perf-4n4gt -n ls1 ovn.kubernetes.io/ingress_rate = 30 \u518d\u6b21\u4ece\u7b2c\u4e8c\u4e2a Pod \u6d4b\u8bd5\u7b2c\u4e00\u4e2a Pod \u5e26\u5bbd\uff1a # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 52372 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 3 .66 MBytes 30 .7 Mbits/sec 2 76 .1 KBytes [ 4 ] 1 .00-2.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 104 KBytes [ 4 ] 2 .00-3.00 sec 3 .50 MBytes 29 .4 Mbits/sec 0 126 KBytes [ 4 ] 3 .00-4.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 144 KBytes [ 4 ] 4 .00-5.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 160 KBytes [ 4 ] 5 .00-6.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 175 KBytes [ 4 ] 6 .00-7.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 212 KBytes [ 4 ] 7 .00-8.00 sec 3 .68 MBytes 30 .9 Mbits/sec 0 294 KBytes [ 4 ] 8 .00-9.00 sec 3 .74 MBytes 31 .4 Mbits/sec 0 398 KBytes [ 4 ] 9 .00-10.00 sec 3 .80 MBytes 31 .9 Mbits/sec 0 526 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 35 .7 MBytes 29 .9 Mbits/sec 2 sender [ 4 ] 0 .00-10.00 sec 34 .5 MBytes 29 .0 Mbits/sec receiver iperf Done. linux-netem QoS \u00b6 Pod \u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b annotation \u914d\u7f6e linux-netem \u7c7b\u578b QoS\uff1a ovn.kubernetes.io/latency \u3001 ovn.kubernetes.io/limit \u548c ovn.kubernetes.io/loss \u3002 ovn.kubernetes.io/latency \uff1a\u8bbe\u7f6e Pod \u6d41\u91cf\u5ef6\u8fdf\uff0c\u53d6\u503c\u4e3a\u6574\u6570\uff0c\u5355\u4f4d\u4e3a ms\u3002 ovn.kubernetes.io/limit \uff1a \u4e3a qdisc \u961f\u5217\u53ef\u5bb9\u7eb3\u7684\u6700\u5927\u6570\u636e\u5305\u6570\uff0c\u53d6\u503c\u4e3a\u6574\u5f62\u6570\u503c\uff0c\u4f8b\u5982 1000\u3002 ovn.kubernetes.io/loss \uff1a \u4e3a\u8bbe\u7f6e\u7684\u62a5\u6587\u4e22\u5305\u6982\u7387\uff0c\u53d6\u503c\u4e3a float \u7c7b\u578b\uff0c\u4f8b\u5982\u53d6\u503c\u4e3a 20\uff0c\u5219\u4e3a\u8bbe\u7f6e 20% \u7684\u4e22\u5305\u6982\u7387\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5bb9\u5668\u7f51\u7edc QoS \u914d\u7f6e"},{"location":"guide/qos/#qos","text":"Kube-OVN \u652f\u6301\u57fa\u4e8e\u5355\u4e2a Pod \u7684\u4e24\u79cd\u4e0d\u540c\u7c7b\u578b\u7684 QoS\uff1a \u6700\u5927\u5e26\u5bbd\u9650\u5236 QoS\u3002 linux-netem \uff0c\u6a21\u62df\u8bbe\u5907\u5e72\u6270\u4e22\u5305\u7b49\u7684 QoS\uff0c\u53ef\u7528\u4e8e\u6a21\u62df\u6d4b\u8bd5\u3002 \u76ee\u524d\u53ea\u652f\u6301 Pod \u7ea7\u522b QoS \u4e0d\u652f\u6301 Namespace \u6216 Subnet \u7ea7\u522b\u7684 QoS \u9650\u5236\u3002","title":"\u5bb9\u5668\u7f51\u7edc QoS \u914d\u7f6e"},{"location":"guide/qos/#qos_1","text":"\u8be5\u7c7b\u578b\u7684 QoS \u53ef\u4ee5\u901a\u8fc7 Pod annotation \u52a8\u6001\u8fdb\u884c\u914d\u7f6e\uff0c\u53ef\u4ee5\u5728\u4e0d\u4e2d\u65ad Pod \u8fd0\u884c\u7684\u60c5\u51b5\u4e0b\u8fdb\u884c\u8c03\u6574\u3002 \u5e26\u5bbd\u9650\u901f\u7684\u5355\u4f4d\u4e3a Mbit/s \u3002 apiVersion : v1 kind : Pod metadata : name : qos namespace : ls1 annotations : ovn.kubernetes.io/ingress_rate : \"3\" ovn.kubernetes.io/egress_rate : \"1\" spec : containers : - name : qos image : docker.io/library/nginx:alpine \u4f7f\u7528 annotation \u52a8\u6001\u8c03\u6574 QoS\uff1a kubectl annotate --overwrite pod nginx-74d5899f46-d7qkn ovn.kubernetes.io/ingress_rate = 3","title":"\u57fa\u4e8e\u6700\u5927\u5e26\u5bbd\u9650\u5236\u7684 QoS"},{"location":"guide/qos/#qos_2","text":"\u90e8\u7f72\u6027\u80fd\u6d4b\u8bd5\u9700\u8981\u7684\u5bb9\u5668\uff1a kind : DaemonSet apiVersion : apps/v1 metadata : name : perf namespace : ls1 labels : app : perf spec : selector : matchLabels : app : perf template : metadata : labels : app : perf spec : containers : - name : nginx image : docker.io/kubeovn/perf \u8fdb\u5165\u5176\u4e2d\u4e00\u4e2a Pod \u5e76\u5f00\u542f iperf3 server\uff1a # kubectl exec -it perf-4n4gt -n ls1 sh # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- \u8fdb\u5165\u53e6\u4e00\u4e2a Pod \u8bf7\u6c42\u4e4b\u524d\u7684 Pod\uff1a # kubectl exec -it perf-d4mqc -n ls1 sh # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 51544 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 86 .4 MBytes 725 Mbits/sec 3 350 KBytes [ 4 ] 1 .00-2.00 sec 89 .9 MBytes 754 Mbits/sec 118 473 KBytes [ 4 ] 2 .00-3.00 sec 101 MBytes 848 Mbits/sec 184 586 KBytes [ 4 ] 3 .00-4.00 sec 104 MBytes 875 Mbits/sec 217 671 KBytes [ 4 ] 4 .00-5.00 sec 111 MBytes 935 Mbits/sec 175 772 KBytes [ 4 ] 5 .00-6.00 sec 100 MBytes 840 Mbits/sec 658 598 KBytes [ 4 ] 6 .00-7.00 sec 106 MBytes 890 Mbits/sec 742 668 KBytes [ 4 ] 7 .00-8.00 sec 102 MBytes 857 Mbits/sec 764 724 KBytes [ 4 ] 8 .00-9.00 sec 97 .4 MBytes 817 Mbits/sec 1175 764 KBytes [ 4 ] 9 .00-10.00 sec 111 MBytes 934 Mbits/sec 1083 838 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 1010 MBytes 848 Mbits/sec 5119 sender [ 4 ] 0 .00-10.00 sec 1008 MBytes 846 Mbits/sec receiver iperf Done. \u4fee\u6539\u7b2c\u4e00\u4e2a Pod \u7684\u5165\u53e3\u5e26\u5bbd QoS\uff1a kubectl annotate --overwrite pod perf-4n4gt -n ls1 ovn.kubernetes.io/ingress_rate = 30 \u518d\u6b21\u4ece\u7b2c\u4e8c\u4e2a Pod \u6d4b\u8bd5\u7b2c\u4e00\u4e2a Pod \u5e26\u5bbd\uff1a # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 52372 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 3 .66 MBytes 30 .7 Mbits/sec 2 76 .1 KBytes [ 4 ] 1 .00-2.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 104 KBytes [ 4 ] 2 .00-3.00 sec 3 .50 MBytes 29 .4 Mbits/sec 0 126 KBytes [ 4 ] 3 .00-4.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 144 KBytes [ 4 ] 4 .00-5.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 160 KBytes [ 4 ] 5 .00-6.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 175 KBytes [ 4 ] 6 .00-7.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 212 KBytes [ 4 ] 7 .00-8.00 sec 3 .68 MBytes 30 .9 Mbits/sec 0 294 KBytes [ 4 ] 8 .00-9.00 sec 3 .74 MBytes 31 .4 Mbits/sec 0 398 KBytes [ 4 ] 9 .00-10.00 sec 3 .80 MBytes 31 .9 Mbits/sec 0 526 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 35 .7 MBytes 29 .9 Mbits/sec 2 sender [ 4 ] 0 .00-10.00 sec 34 .5 MBytes 29 .0 Mbits/sec receiver iperf Done.","title":"\u6d4b\u8bd5 QoS \u8c03\u6574"},{"location":"guide/qos/#linux-netem-qos","text":"Pod \u53ef\u4ee5\u4f7f\u7528\u5982\u4e0b annotation \u914d\u7f6e linux-netem \u7c7b\u578b QoS\uff1a ovn.kubernetes.io/latency \u3001 ovn.kubernetes.io/limit \u548c ovn.kubernetes.io/loss \u3002 ovn.kubernetes.io/latency \uff1a\u8bbe\u7f6e Pod \u6d41\u91cf\u5ef6\u8fdf\uff0c\u53d6\u503c\u4e3a\u6574\u6570\uff0c\u5355\u4f4d\u4e3a ms\u3002 ovn.kubernetes.io/limit \uff1a \u4e3a qdisc \u961f\u5217\u53ef\u5bb9\u7eb3\u7684\u6700\u5927\u6570\u636e\u5305\u6570\uff0c\u53d6\u503c\u4e3a\u6574\u5f62\u6570\u503c\uff0c\u4f8b\u5982 1000\u3002 ovn.kubernetes.io/loss \uff1a \u4e3a\u8bbe\u7f6e\u7684\u62a5\u6587\u4e22\u5305\u6982\u7387\uff0c\u53d6\u503c\u4e3a float \u7c7b\u578b\uff0c\u4f8b\u5982\u53d6\u503c\u4e3a 20\uff0c\u5219\u4e3a\u8bbe\u7f6e 20% \u7684\u4e22\u5305\u6982\u7387\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"linux-netem QoS"},{"location":"guide/setup-options/","text":"\u5b89\u88c5\u548c\u914d\u7f6e\u9009\u9879 \u00b6 \u5728 \u4e00\u952e\u5b89\u88c5\u4e2d \u6211\u4eec\u4f7f\u7528\u9ed8\u8ba4\u914d\u7f6e\u8fdb\u884c\u5b89\u88c5\uff0cKube-OVN \u8fd8\u652f\u6301\u66f4\u591a \u81ea\u5b9a\u4e49\u914d\u7f6e\uff0c\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff0c\u6216\u8005\u4e4b\u540e\u66f4\u6539\u5404\u4e2a\u7ec4\u4ef6\u7684\u53c2\u6570\u6765\u8fdb\u884c\u914d\u7f6e\u3002\u672c\u6587\u6863\u5c06\u4f1a\u4ecb\u7ecd\u8fd9\u4e9b\u81ea\u5b9a\u4e49\u9009\u9879 \u7684\u4f5c\u7528\uff0c\u4ee5\u53ca\u5982\u4f55\u8fdb\u884c\u914d\u7f6e\u3002 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \u00b6 Kube-OVN \u5728\u5b89\u88c5\u65f6\u4f1a\u914d\u7f6e\u4e24\u4e2a\u5185\u7f6e\u5b50\u7f51\uff1a default \u5b50\u7f51\uff0c\u4f5c\u4e3a Pod \u5206\u914d IP \u4f7f\u7528\u7684\u9ed8\u8ba4\u5b50\u7f51\uff0c\u9ed8\u8ba4 CIDR \u4e3a 10.16.0.0/16 \uff0c\u7f51\u5173\u4e3a 10.16.0.1 \u3002 join \u5b50\u7f51\uff0c\u4f5c\u4e3a Node \u548c Pod \u4e4b\u95f4\u8fdb\u884c\u7f51\u7edc\u901a\u4fe1\u7684\u7279\u6b8a\u5b50\u7f51, \u9ed8\u8ba4 CIDR \u4e3a 100.64.0.0/16 \uff0c\u7f51\u5173\u4e3a 100.64.0.1 \u3002 \u5728\u5b89\u88c5\u65f6\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u5185\u7684\u914d\u7f6e\u8fdb\u884c\u66f4\u6539\uff1a POD_CIDR = \"10.16.0.0/16\" POD_GATEWAY = \"10.16.0.1\" JOIN_CIDR = \"100.64.0.0/16\" EXCLUDE_IPS = \"\" EXCLUDE_IP \u53ef\u8bbe\u7f6e POD_CIDR \u4e0d\u8fdb\u884c\u5206\u914d\u7684\u5730\u5740\u8303\u56f4\uff0c\u683c\u5f0f\u4e3a\uff1a 192.168.10.20..192.168.10.30 \u3002 \u9700\u8981\u6ce8\u610f Overlay \u60c5\u51b5\u4e0b\u8fd9\u4e24\u4e2a\u7f51\u7edc\u4e0d\u80fd\u548c\u5df2\u6709\u7684\u4e3b\u673a\u7f51\u7edc\u548c Service CIDR \u51b2\u7a81\u3002 \u5728\u5b89\u88c5\u540e\u53ef\u4ee5\u5bf9\u8fd9\u4e24\u4e2a\u7f51\u7edc\u7684\u5730\u5740\u8303\u56f4\u8fdb\u884c\u4fee\u6539\u8bf7\u53c2\u8003 \u4fee\u6539\u9ed8\u8ba4\u5b50\u7f51 \u548c \u4fee\u6539 Join \u5b50\u7f51 \u3002 Service \u7f51\u6bb5\u914d\u7f6e \u00b6 \u7531\u4e8e\u90e8\u5206 kube-proxy \u8bbe\u7f6e\u7684 iptables \u548c\u8def\u7531\u89c4\u5219\u4f1a\u548c Kube-OVN \u8bbe\u7f6e\u7684\u89c4\u5219\u4ea7\u751f\u4ea4\u96c6\uff0c\u56e0\u6b64 Kube-OVN \u9700\u8981\u77e5\u9053 Service \u7684 CIDR \u6765\u6b63\u786e\u8bbe\u7f6e\u5bf9\u5e94\u7684\u89c4\u5219\u3002 \u5728\u5b89\u88c5\u811a\u672c\u4e2d\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539\uff1a SVC_CIDR = \"10.96.0.0/12\" \u6765\u8fdb\u884c\u914d\u7f6e\u3002 \u4e5f\u53ef\u4ee5\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u4fee\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\uff1a args : - --service-cluster-ip-range=10.96.0.0/12 \u6765\u8fdb\u884c\u4fee\u6539\u3002 Overlay \u7f51\u5361\u9009\u62e9 \u00b6 \u5728\u8282\u70b9\u5b58\u5728\u591a\u5757\u7f51\u5361\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u9ed8\u8ba4\u4f1a\u9009\u62e9 Kubernetes Node IP \u5bf9\u5e94\u7684\u7f51\u5361\u4f5c\u4e3a\u5bb9\u5668\u95f4\u8de8\u8282\u70b9\u901a\u4fe1\u7684\u7f51\u5361\u5e76\u5efa\u7acb\u5bf9\u5e94\u7684\u96a7\u9053\u3002 \u5982\u679c\u9700\u8981\u9009\u62e9\u5176\u4ed6\u7684\u7f51\u5361\u5efa\u7acb\u5bb9\u5668\u96a7\u9053\uff0c\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u4fee\u6539\uff1a IFACE = eth1 \u8be5\u9009\u9879\u652f\u6301\u4ee5\u9017\u53f7\u6240\u5206\u9694\u6b63\u5219\u8868\u8fbe\u5f0f,\u4f8b\u5982 ens[a-z0-9]*,eth[a-z0-9]* \u3002 \u5b89\u88c5\u540e\u4e5f\u53ef\u901a\u8fc7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a args : - --iface=eth1 \u5982\u679c\u6bcf\u53f0\u673a\u5668\u7684\u7f51\u5361\u540d\u5747\u4e0d\u540c\uff0c\u4e14\u6ca1\u6709\u56fa\u5b9a\u89c4\u5f8b\uff0c\u53ef\u4ee5\u4f7f\u7528\u8282\u70b9 annotation ovn.kubernetes.io/tunnel_interface \u8fdb\u884c\u6bcf\u4e2a\u8282\u70b9\u7684\u9010\u4e00\u914d\u7f6e\uff0c\u62e5\u6709\u8be5 annotation \u8282\u70b9\u4f1a\u8986\u76d6 iface \u7684\u914d\u7f6e\uff0c\u4f18\u5148\u4f7f\u7528 annotation\u3002 kubectl annotate node no1 ovn.kubernetes.io/tunnel_interface = ethx MTU \u8bbe\u7f6e \u00b6 \u7531\u4e8e Overlay \u5c01\u88c5\u9700\u8981\u5360\u636e\u989d\u5916\u7684\u7a7a\u95f4\uff0cKube-OVN \u5728\u521b\u5efa\u5bb9\u5668\u7f51\u5361\u65f6\u4f1a\u6839\u636e\u9009\u62e9\u7f51\u5361\u7684 MTU \u8fdb\u884c\u5bb9\u5668\u7f51\u5361\u7684 MTU \u8c03\u6574\uff0c \u9ed8\u8ba4\u60c5\u51b5\u4e0b Overlay \u5b50\u7f51\u4e0b Pod \u7f51\u5361 MTU \u4e3a\u4e3b\u673a\u7f51\u5361 MTU - 100\uff0cUnderlay \u5b50\u7f51\u4e0b\uff0cPod \u7f51\u5361\u548c\u4e3b\u673a\u7f51\u5361\u6709\u76f8\u540c MTU\u3002 \u5982\u679c\u9700\u8981\u8c03\u6574 Overlay \u5b50\u7f51\u4e0b MTU \u7684\u5927\u5c0f\uff0c\u53ef\u4ee5\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\uff1a args : - --mtu=1333 \u5168\u5c40\u6d41\u91cf\u955c\u50cf\u5f00\u542f\u8bbe\u7f6e \u00b6 \u5728\u5f00\u542f\u5168\u5c40\u6d41\u91cf\u955c\u50cf\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u4f1a\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\u521b\u5efa\u4e00\u5757 mirror0 \u7684\u865a\u62df\u7f51\u5361\uff0c\u590d\u5236\u5f53\u524d\u673a\u5668\u6240\u6709\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u5230\u8be5\u7f51\u5361\u4e0a\uff0c \u7528\u6237\u53ef\u4ee5\u901a\u8fc7 tcpdump \u53ca\u5176\u4ed6\u5de5\u5177\u8fdb\u884c\u6d41\u91cf\u5206\u6790\uff0c\u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u901a\u8fc7\u4e0b\u9762\u7684\u914d\u7f6e\u5f00\u542f\uff1a ENABLE_MIRROR = true \u4e5f\u53ef\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\u65b9\u5f0f\u8fdb\u884c\u8c03\u6574: args : - --enable-mirror=true \u6d41\u91cf\u955c\u50cf\u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5173\u95ed\uff0c\u5982\u679c\u9700\u8981\u7ec6\u7c92\u5ea6\u7684\u6d41\u91cf\u955c\u50cf\u6216\u9700\u8981\u5c06\u6d41\u91cf\u955c\u50cf\u5230\u989d\u5916\u7684\u7f51\u5361\u8bf7\u53c2\u8003 \u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u955c\u50cf \u3002 LB \u5f00\u542f\u8bbe\u7f6e \u00b6 Kube-OVN \u4f7f\u7528 OVN \u4e2d\u7684 L2 LB \u6765\u5b9e\u73b0 Service \u8f6c\u53d1\uff0c\u5728 Overlay \u573a\u666f\u4e2d\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9\u4f7f\u7528 kube-proxy \u6765\u5b8c\u6210 Service \u6d41\u91cf\u8f6c\u53d1, \u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u5173\u95ed Kube-OVN \u7684 LB \u529f\u80fd\u4ee5\u8fbe\u5230\u63a7\u5236\u9762\u548c\u6570\u636e\u9762\u66f4\u597d\u7684\u6027\u80fd\u3002 \u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_LB = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-lb=false LB \u7684\u529f\u80fd\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableLb \uff0c\u5c06 Kube-OVN \u7684 LB \u529f\u80fd\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f LB \u529f\u80fd\u3002 kube-ovn-controller Deployment \u4e2d\u7684 enable-lb \u53c2\u6570\u4f5c\u4e3a\u5168\u5c40\u53c2\u6570\uff0c\u63a7\u5236\u662f\u5426\u521b\u5efa load-balancer \u8bb0\u5f55\uff0c\u5b50\u7f51\u4e2d\u65b0\u589e\u7684 enableLb \u53c2\u6570\u7528\u4e8e\u63a7\u5236\u5b50\u7f51\u662f\u5426\u5173\u8054 load-balancer \u8bb0\u5f55\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51 enableLb \u53c2\u6570\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002 NetworkPolicy \u5f00\u542f\u8bbe\u7f6e \u00b6 Kube-OVN \u4f7f\u7528 OVN \u4e2d\u7684 ACL \u6765\u5b9e\u73b0 NetworkPolicy\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9\u5173\u95ed NetworkPolicy \u529f\u80fd \u6216\u8005\u4f7f\u7528 Cilium Chain \u7684\u65b9\u5f0f\u5229\u7528 eBPF \u5b9e\u73b0 NetworkPolicy\uff0c \u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u5173\u95ed Kube-OVN \u7684 NetworkPolicy \u529f\u80fd\u4ee5\u8fbe\u5230\u63a7\u5236\u9762\u548c\u6570\u636e\u9762\u66f4\u597d\u7684\u6027\u80fd\u3002 \u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_NP = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-np=false NetworkPolicy \u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002 EIP \u548c SNAT \u5f00\u542f\u8bbe\u7f6e \u00b6 \u9ed8\u8ba4\u7f51\u7edc\u4e0b\u5982\u679c\u65e0\u9700\u4f7f\u7528 EIP \u548c SNAT \u7684\u80fd\u529b\uff0c\u53ef\u4ee5\u9009\u62e9\u5173\u95ed\u76f8\u5173\u529f\u80fd\uff0c\u4ee5\u51cf\u5c11 kube-ovn-controller \u5728\u521b\u5efa\u548c\u66f4\u65b0 \u7f51\u7edc\u65f6\u7684\u68c0\u67e5\u6d88\u8017\uff0c\u5728\u5927\u89c4\u6a21\u96c6\u7fa4\u73af\u5883\u4e0b\u53ef\u4ee5\u63d0\u5347\u5904\u7406\u901f\u5ea6\u3002 \u8be5\u529f\u80fd\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_EIP_SNAT = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-eip-snat=false EIP \u548c SNAT \u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002\u8be5\u529f\u80fd\u7684\u76f8\u5173\u4f7f\u7528\u548c\u5176\u4ed6\u53ef\u914d\u53c2\u6570\u8bf7\u53c2\u8003 EIP \u548c SNAT \u914d\u7f6e \u3002 Load Balancer \u7c7b\u578b Service \u652f\u6301\u5f00\u542f\u8bbe\u7f6e \u00b6 \u9ed8\u8ba4 VPC \u4e0b\u53ef\u901a\u8fc7\u5f00\u542f\u8be5\u9009\u9879\u6765\u652f\u6301 Load Balancer \u7c7b\u578b Service\u3002\u8be5\u529f\u80fd\u7684\u76f8\u5173\u4f7f\u7528\u548c\u5176\u4ed6\u53ef\u914d\u53c2\u6570\u8bf7\u53c2\u8003 LoadBalancer \u7c7b\u578b Service \u3002 \u8be5\u529f\u80fd\u9ed8\u8ba4\u5173\u95ed\uff0c\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_LB_SVC = true \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-lb-svc=true \u96c6\u4e2d\u5f0f\u7f51\u5173 ECMP \u5f00\u542f\u8bbe\u7f6e \u00b6 \u96c6\u4e2d\u5f0f\u7f51\u5173\u652f\u6301\u4e3b\u5907\u548c ECMP \u4e24\u79cd\u9ad8\u53ef\u7528\u6a21\u5f0f\uff0c\u5982\u679c\u9700\u8981\u542f\u7528 ECMP \u6a21\u5f0f\uff0c \u9700\u8981\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e: args : - --enable-ecmp=true \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableEcmp \uff0c\u5c06\u96c6\u4e2d\u5f0f\u5b50\u7f51 ECMP \u5f00\u5173\u63a7\u5236\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f ECMP \u6a21\u5f0f\u3002\u539f\u6709\u7684 kube-ovn-controller Deployment \u4e2d\u7684 enable-ecmp \u53c2\u6570\u4e0d\u518d\u4f7f\u7528\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51\u5f00\u5173\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002 \u96c6\u4e2d\u5f0f\u7f51\u5173\u9ed8\u8ba4\u5b89\u88c5\u4e0b\u4e3a\u4e3b\u5907\u6a21\u5f0f\uff0c\u66f4\u591a\u7f51\u5173\u76f8\u5173\u5185\u5bb9\u8bf7\u53c2\u8003 \u5b50\u7f51\u4f7f\u7528 \u3002 Kubevirt VM \u56fa\u5b9a\u5730\u5740\u5f00\u542f\u8bbe\u7f6e \u00b6 \u9488\u5bf9 Kubevirt \u521b\u5efa\u7684 VM \u5b9e\u4f8b\uff0c kube-ovn-controller \u53ef\u4ee5\u6309\u7167\u7c7b\u4f3c StatefulSet Pod \u7684\u65b9\u5f0f\u8fdb\u884c IP \u5730\u5740\u5206\u914d\u548c\u7ba1\u7406\u3002 \u4ee5\u8fbe\u5230 VM \u5b9e\u4f8b\u5728\u751f\u547d\u5468\u671f\u5185\u542f\u505c\uff0c\u5347\u7ea7\uff0c\u8fc1\u79fb\u7b49\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\u5730\u5740\u56fa\u5b9a\u4e0d\u53d8\uff0c\u66f4\u7b26\u865a\u62df\u5316\u5408\u7528\u6237\u7684\u5b9e\u9645\u4f7f\u7528\u4f53\u9a8c\u3002 \u8be5\u529f\u80fd\u5728 1.10.6 \u540e\u9ed8\u8ba4\u5f00\u542f\uff0c\u82e5\u8981\u5173\u95ed\u6b64\u529f\u80fd\uff0c\u9700\u8981\u5728 kube-ovn-controller Deployment \u7684\u542f\u52a8\u547d\u4ee4\u4e2d\u8bbe\u7f6e\u5982\u4e0b\u53c2\u6570\uff1a args : - --keep-vm-ip=false CNI \u914d\u7f6e\u76f8\u5173\u8bbe\u7f6e \u00b6 Kube-OVN \u9ed8\u8ba4\u4f1a\u5728 /opt/cni/bin \u76ee\u5f55\u4e0b\u5b89\u88c5 CNI \u6267\u884c\u6587\u4ef6\uff0c\u5728 /etc/cni/net.d \u76ee\u5f55\u4e0b\u5b89\u88c5 CNI \u914d\u7f6e\u6587\u4ef6 01-kube-ovn.conflist \u3002 \u5982\u679c\u9700\u8981\u66f4\u6539\u5b89\u88c5\u4f4d\u7f6e\u548c CNI \u914d\u7f6e\u6587\u4ef6\u7684\u4f18\u5148\u7ea7\uff0c\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u7684\u4e0b\u5217\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a CNI_CONF_DIR = \"/etc/cni/net.d\" CNI_BIN_DIR = \"/opt/cni/bin\" CNI_CONFIG_PRIORITY = \"01\" \u6216\u8005\u5728\u5b89\u88c5\u540e\u66f4\u6539 kube-ovn-cni DaemonSet \u7684 Volume \u6302\u8f7d\u548c\u542f\u52a8\u53c2\u6570\uff1a volumes : - name : cni-conf hostPath : path : \"/etc/cni/net.d\" - name : cni-bin hostPath : path:\"/opt/cni/bin\" ... args : - --cni-conf-name=01-kube-ovn.conflist \u96a7\u9053\u7c7b\u578b\u8bbe\u7f6e \u00b6 Kube-OVN \u9ed8\u8ba4 Overlay \u7684\u5c01\u88c5\u6a21\u5f0f\u4e3a Geneve\uff0c\u5982\u679c\u60f3\u66f4\u6362\u4e3a Vxlan \u6216 STT\uff0c\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u7684\u4e0b\u5217\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a TUNNEL_TYPE = \"vxlan\" \u6216\u8005\u5728\u5b89\u88c5\u540e\u66f4\u6539 ovs-ovn DaemonSet \u7684\u73af\u5883\u53d8\u91cf\uff1a env : - name : TUNNEL_TYPE value : \"vxlan\" \u5982\u679c\u9700\u8981\u4f7f\u7528 STT \u96a7\u9053\u9700\u8981\u989d\u5916\u7f16\u8bd1 ovs \u7684\u5185\u6838\u6a21\u5757\uff0c\u8bf7\u53c2\u8003 \u6027\u80fd\u8c03\u4f18 \u3002 \u4e0d\u540c\u534f\u8bae\u5728\u5b9e\u9645\u4f7f\u7528\u4e2d\u7684\u533a\u522b\u8bf7\u53c2\u8003 \u96a7\u9053\u534f\u8bae\u8bf4\u660e \u3002 SSL \u8bbe\u7f6e \u00b6 OVN DB \u7684 API \u63a5\u53e3\u652f\u6301 SSL \u52a0\u5bc6\u6765\u4fdd\u8bc1\u8fde\u63a5\u5b89\u5168\uff0c\u5982\u8981\u5f00\u542f\u53ef\u8c03\u6574\u5b89\u88c5\u811a\u672c\u4e2d\u7684\u5982\u4e0b\u53c2\u6570: ENABLE_SSL = true SSL \u529f\u80fd\u9ed8\u8ba4\u5b89\u88c5\u4e0b\u4e3a\u5173\u95ed\u6a21\u5f0f\u3002 \u7ed1\u5b9a\u672c\u5730 ip \u00b6 kube-ovn-controller/kube-ovn-cni/kube-ovn-monitor \u8fd9\u4e9b\u670d\u52a1\u652f\u6301\u7ed1\u5b9a\u672c\u5730 ip\uff0c\u8be5\u529f\u80fd\u8bbe\u8ba1\u539f\u56e0\u4e3b\u8981\u662f\u56e0\u4e3a\u67d0\u4e9b\u573a\u666f\u4e0b\u51fa\u4e8e\u5b89\u5168\u8003\u8651\u4e0d\u5141\u8bb8\u670d\u52a1\u7ed1\u5b9a 0.0.0.0 \uff08\u6bd4\u5982\u8be5\u670d\u52a1\u90e8\u7f72\u5728\u67d0\u4e2a\u5bf9\u5916\u7f51\u5173\u4e0a\uff0c\u5916\u90e8\u7528\u6237\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7\u516c\u7f51 ip \u5e76\u6307\u5b9a\u7aef\u53e3\u53bb\u8bbf\u95ee\u5230\u8be5\u670d\u52a1\uff09\uff0c\u8be5\u529f\u80fd\u9ed8\u8ba4\u662f\u6253\u5f00\u7684\uff0c\u7531\u5b89\u88c5\u811a\u672c\u4e2d\u5982\u4e0b\u53c2\u6570\u63a7\u5236\uff1a ENABLE_BIND_LOCAL_IP = true \u4ee5 kube-ovn-monitor \u4e3a\u4f8b\uff0c\u5f00\u542f\u529f\u80fd\u540e\u4f1a\u628a\u670d\u52a1\u7ed1\u5b9a\u672c\u5730\u7684 pod ip \u5982\u4e0b\uff1a # netstat -tunlp |grep kube-ovn tcp 0 0 172 .18.0.5:10661 0 .0.0.0:* LISTEN 2612 /./kube-ovn-mon \u5b89\u88c5\u540e\u4e5f\u53ef\u901a\u8fc7\u4fee\u6539\u670d\u52a1\u7684 deployment \u6216\u8005 daemonSet \u7684\u73af\u5883\u53d8\u91cf\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a env : - name : ENABLE_BIND_LOCAL_IP value : \"false\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b89\u88c5\u548c\u914d\u7f6e\u9009\u9879"},{"location":"guide/setup-options/#_1","text":"\u5728 \u4e00\u952e\u5b89\u88c5\u4e2d \u6211\u4eec\u4f7f\u7528\u9ed8\u8ba4\u914d\u7f6e\u8fdb\u884c\u5b89\u88c5\uff0cKube-OVN \u8fd8\u652f\u6301\u66f4\u591a \u81ea\u5b9a\u4e49\u914d\u7f6e\uff0c\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff0c\u6216\u8005\u4e4b\u540e\u66f4\u6539\u5404\u4e2a\u7ec4\u4ef6\u7684\u53c2\u6570\u6765\u8fdb\u884c\u914d\u7f6e\u3002\u672c\u6587\u6863\u5c06\u4f1a\u4ecb\u7ecd\u8fd9\u4e9b\u81ea\u5b9a\u4e49\u9009\u9879 \u7684\u4f5c\u7528\uff0c\u4ee5\u53ca\u5982\u4f55\u8fdb\u884c\u914d\u7f6e\u3002","title":"\u5b89\u88c5\u548c\u914d\u7f6e\u9009\u9879"},{"location":"guide/setup-options/#_2","text":"Kube-OVN \u5728\u5b89\u88c5\u65f6\u4f1a\u914d\u7f6e\u4e24\u4e2a\u5185\u7f6e\u5b50\u7f51\uff1a default \u5b50\u7f51\uff0c\u4f5c\u4e3a Pod \u5206\u914d IP \u4f7f\u7528\u7684\u9ed8\u8ba4\u5b50\u7f51\uff0c\u9ed8\u8ba4 CIDR \u4e3a 10.16.0.0/16 \uff0c\u7f51\u5173\u4e3a 10.16.0.1 \u3002 join \u5b50\u7f51\uff0c\u4f5c\u4e3a Node \u548c Pod \u4e4b\u95f4\u8fdb\u884c\u7f51\u7edc\u901a\u4fe1\u7684\u7279\u6b8a\u5b50\u7f51, \u9ed8\u8ba4 CIDR \u4e3a 100.64.0.0/16 \uff0c\u7f51\u5173\u4e3a 100.64.0.1 \u3002 \u5728\u5b89\u88c5\u65f6\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u5185\u7684\u914d\u7f6e\u8fdb\u884c\u66f4\u6539\uff1a POD_CIDR = \"10.16.0.0/16\" POD_GATEWAY = \"10.16.0.1\" JOIN_CIDR = \"100.64.0.0/16\" EXCLUDE_IPS = \"\" EXCLUDE_IP \u53ef\u8bbe\u7f6e POD_CIDR \u4e0d\u8fdb\u884c\u5206\u914d\u7684\u5730\u5740\u8303\u56f4\uff0c\u683c\u5f0f\u4e3a\uff1a 192.168.10.20..192.168.10.30 \u3002 \u9700\u8981\u6ce8\u610f Overlay \u60c5\u51b5\u4e0b\u8fd9\u4e24\u4e2a\u7f51\u7edc\u4e0d\u80fd\u548c\u5df2\u6709\u7684\u4e3b\u673a\u7f51\u7edc\u548c Service CIDR \u51b2\u7a81\u3002 \u5728\u5b89\u88c5\u540e\u53ef\u4ee5\u5bf9\u8fd9\u4e24\u4e2a\u7f51\u7edc\u7684\u5730\u5740\u8303\u56f4\u8fdb\u884c\u4fee\u6539\u8bf7\u53c2\u8003 \u4fee\u6539\u9ed8\u8ba4\u5b50\u7f51 \u548c \u4fee\u6539 Join \u5b50\u7f51 \u3002","title":"\u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e"},{"location":"guide/setup-options/#service","text":"\u7531\u4e8e\u90e8\u5206 kube-proxy \u8bbe\u7f6e\u7684 iptables \u548c\u8def\u7531\u89c4\u5219\u4f1a\u548c Kube-OVN \u8bbe\u7f6e\u7684\u89c4\u5219\u4ea7\u751f\u4ea4\u96c6\uff0c\u56e0\u6b64 Kube-OVN \u9700\u8981\u77e5\u9053 Service \u7684 CIDR \u6765\u6b63\u786e\u8bbe\u7f6e\u5bf9\u5e94\u7684\u89c4\u5219\u3002 \u5728\u5b89\u88c5\u811a\u672c\u4e2d\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539\uff1a SVC_CIDR = \"10.96.0.0/12\" \u6765\u8fdb\u884c\u914d\u7f6e\u3002 \u4e5f\u53ef\u4ee5\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u4fee\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\uff1a args : - --service-cluster-ip-range=10.96.0.0/12 \u6765\u8fdb\u884c\u4fee\u6539\u3002","title":"Service \u7f51\u6bb5\u914d\u7f6e"},{"location":"guide/setup-options/#overlay","text":"\u5728\u8282\u70b9\u5b58\u5728\u591a\u5757\u7f51\u5361\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u9ed8\u8ba4\u4f1a\u9009\u62e9 Kubernetes Node IP \u5bf9\u5e94\u7684\u7f51\u5361\u4f5c\u4e3a\u5bb9\u5668\u95f4\u8de8\u8282\u70b9\u901a\u4fe1\u7684\u7f51\u5361\u5e76\u5efa\u7acb\u5bf9\u5e94\u7684\u96a7\u9053\u3002 \u5982\u679c\u9700\u8981\u9009\u62e9\u5176\u4ed6\u7684\u7f51\u5361\u5efa\u7acb\u5bb9\u5668\u96a7\u9053\uff0c\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u4fee\u6539\uff1a IFACE = eth1 \u8be5\u9009\u9879\u652f\u6301\u4ee5\u9017\u53f7\u6240\u5206\u9694\u6b63\u5219\u8868\u8fbe\u5f0f,\u4f8b\u5982 ens[a-z0-9]*,eth[a-z0-9]* \u3002 \u5b89\u88c5\u540e\u4e5f\u53ef\u901a\u8fc7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a args : - --iface=eth1 \u5982\u679c\u6bcf\u53f0\u673a\u5668\u7684\u7f51\u5361\u540d\u5747\u4e0d\u540c\uff0c\u4e14\u6ca1\u6709\u56fa\u5b9a\u89c4\u5f8b\uff0c\u53ef\u4ee5\u4f7f\u7528\u8282\u70b9 annotation ovn.kubernetes.io/tunnel_interface \u8fdb\u884c\u6bcf\u4e2a\u8282\u70b9\u7684\u9010\u4e00\u914d\u7f6e\uff0c\u62e5\u6709\u8be5 annotation \u8282\u70b9\u4f1a\u8986\u76d6 iface \u7684\u914d\u7f6e\uff0c\u4f18\u5148\u4f7f\u7528 annotation\u3002 kubectl annotate node no1 ovn.kubernetes.io/tunnel_interface = ethx","title":"Overlay \u7f51\u5361\u9009\u62e9"},{"location":"guide/setup-options/#mtu","text":"\u7531\u4e8e Overlay \u5c01\u88c5\u9700\u8981\u5360\u636e\u989d\u5916\u7684\u7a7a\u95f4\uff0cKube-OVN \u5728\u521b\u5efa\u5bb9\u5668\u7f51\u5361\u65f6\u4f1a\u6839\u636e\u9009\u62e9\u7f51\u5361\u7684 MTU \u8fdb\u884c\u5bb9\u5668\u7f51\u5361\u7684 MTU \u8c03\u6574\uff0c \u9ed8\u8ba4\u60c5\u51b5\u4e0b Overlay \u5b50\u7f51\u4e0b Pod \u7f51\u5361 MTU \u4e3a\u4e3b\u673a\u7f51\u5361 MTU - 100\uff0cUnderlay \u5b50\u7f51\u4e0b\uff0cPod \u7f51\u5361\u548c\u4e3b\u673a\u7f51\u5361\u6709\u76f8\u540c MTU\u3002 \u5982\u679c\u9700\u8981\u8c03\u6574 Overlay \u5b50\u7f51\u4e0b MTU \u7684\u5927\u5c0f\uff0c\u53ef\u4ee5\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\uff1a args : - --mtu=1333","title":"MTU \u8bbe\u7f6e"},{"location":"guide/setup-options/#_3","text":"\u5728\u5f00\u542f\u5168\u5c40\u6d41\u91cf\u955c\u50cf\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u4f1a\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\u521b\u5efa\u4e00\u5757 mirror0 \u7684\u865a\u62df\u7f51\u5361\uff0c\u590d\u5236\u5f53\u524d\u673a\u5668\u6240\u6709\u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u5230\u8be5\u7f51\u5361\u4e0a\uff0c \u7528\u6237\u53ef\u4ee5\u901a\u8fc7 tcpdump \u53ca\u5176\u4ed6\u5de5\u5177\u8fdb\u884c\u6d41\u91cf\u5206\u6790\uff0c\u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u901a\u8fc7\u4e0b\u9762\u7684\u914d\u7f6e\u5f00\u542f\uff1a ENABLE_MIRROR = true \u4e5f\u53ef\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u4fee\u6539 kube-ovn-cni DaemonSet \u7684\u53c2\u6570\u65b9\u5f0f\u8fdb\u884c\u8c03\u6574: args : - --enable-mirror=true \u6d41\u91cf\u955c\u50cf\u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5173\u95ed\uff0c\u5982\u679c\u9700\u8981\u7ec6\u7c92\u5ea6\u7684\u6d41\u91cf\u955c\u50cf\u6216\u9700\u8981\u5c06\u6d41\u91cf\u955c\u50cf\u5230\u989d\u5916\u7684\u7f51\u5361\u8bf7\u53c2\u8003 \u5bb9\u5668\u7f51\u7edc\u6d41\u91cf\u955c\u50cf \u3002","title":"\u5168\u5c40\u6d41\u91cf\u955c\u50cf\u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#lb","text":"Kube-OVN \u4f7f\u7528 OVN \u4e2d\u7684 L2 LB \u6765\u5b9e\u73b0 Service \u8f6c\u53d1\uff0c\u5728 Overlay \u573a\u666f\u4e2d\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9\u4f7f\u7528 kube-proxy \u6765\u5b8c\u6210 Service \u6d41\u91cf\u8f6c\u53d1, \u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u5173\u95ed Kube-OVN \u7684 LB \u529f\u80fd\u4ee5\u8fbe\u5230\u63a7\u5236\u9762\u548c\u6570\u636e\u9762\u66f4\u597d\u7684\u6027\u80fd\u3002 \u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_LB = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-lb=false LB \u7684\u529f\u80fd\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableLb \uff0c\u5c06 Kube-OVN \u7684 LB \u529f\u80fd\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f LB \u529f\u80fd\u3002 kube-ovn-controller Deployment \u4e2d\u7684 enable-lb \u53c2\u6570\u4f5c\u4e3a\u5168\u5c40\u53c2\u6570\uff0c\u63a7\u5236\u662f\u5426\u521b\u5efa load-balancer \u8bb0\u5f55\uff0c\u5b50\u7f51\u4e2d\u65b0\u589e\u7684 enableLb \u53c2\u6570\u7528\u4e8e\u63a7\u5236\u5b50\u7f51\u662f\u5426\u5173\u8054 load-balancer \u8bb0\u5f55\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51 enableLb \u53c2\u6570\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002","title":"LB \u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#networkpolicy","text":"Kube-OVN \u4f7f\u7528 OVN \u4e2d\u7684 ACL \u6765\u5b9e\u73b0 NetworkPolicy\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9\u5173\u95ed NetworkPolicy \u529f\u80fd \u6216\u8005\u4f7f\u7528 Cilium Chain \u7684\u65b9\u5f0f\u5229\u7528 eBPF \u5b9e\u73b0 NetworkPolicy\uff0c \u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u5173\u95ed Kube-OVN \u7684 NetworkPolicy \u529f\u80fd\u4ee5\u8fbe\u5230\u63a7\u5236\u9762\u548c\u6570\u636e\u9762\u66f4\u597d\u7684\u6027\u80fd\u3002 \u8be5\u529f\u80fd\u53ef\u4ee5\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_NP = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-np=false NetworkPolicy \u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002","title":"NetworkPolicy \u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#eip-snat","text":"\u9ed8\u8ba4\u7f51\u7edc\u4e0b\u5982\u679c\u65e0\u9700\u4f7f\u7528 EIP \u548c SNAT \u7684\u80fd\u529b\uff0c\u53ef\u4ee5\u9009\u62e9\u5173\u95ed\u76f8\u5173\u529f\u80fd\uff0c\u4ee5\u51cf\u5c11 kube-ovn-controller \u5728\u521b\u5efa\u548c\u66f4\u65b0 \u7f51\u7edc\u65f6\u7684\u68c0\u67e5\u6d88\u8017\uff0c\u5728\u5927\u89c4\u6a21\u96c6\u7fa4\u73af\u5883\u4e0b\u53ef\u4ee5\u63d0\u5347\u5904\u7406\u901f\u5ea6\u3002 \u8be5\u529f\u80fd\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_EIP_SNAT = false \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-eip-snat=false EIP \u548c SNAT \u7684\u80fd\u529b\u5728\u9ed8\u8ba4\u5b89\u88c5\u4e2d\u4e3a\u5f00\u542f\u3002\u8be5\u529f\u80fd\u7684\u76f8\u5173\u4f7f\u7528\u548c\u5176\u4ed6\u53ef\u914d\u53c2\u6570\u8bf7\u53c2\u8003 EIP \u548c SNAT \u914d\u7f6e \u3002","title":"EIP \u548c SNAT \u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#load-balancer-service","text":"\u9ed8\u8ba4 VPC \u4e0b\u53ef\u901a\u8fc7\u5f00\u542f\u8be5\u9009\u9879\u6765\u652f\u6301 Load Balancer \u7c7b\u578b Service\u3002\u8be5\u529f\u80fd\u7684\u76f8\u5173\u4f7f\u7528\u548c\u5176\u4ed6\u53ef\u914d\u53c2\u6570\u8bf7\u53c2\u8003 LoadBalancer \u7c7b\u578b Service \u3002 \u8be5\u529f\u80fd\u9ed8\u8ba4\u5173\u95ed\uff0c\u53ef\u5728\u5b89\u88c5\u811a\u672c\u4e2d\u8fdb\u884c\u914d\u7f6e\uff1a ENABLE_LB_SVC = true \u6216\u8005\u5728\u5b89\u88c5\u540e\u901a\u8fc7\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e\uff1a args : - --enable-lb-svc=true","title":"Load Balancer \u7c7b\u578b Service \u652f\u6301\u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#ecmp","text":"\u96c6\u4e2d\u5f0f\u7f51\u5173\u652f\u6301\u4e3b\u5907\u548c ECMP \u4e24\u79cd\u9ad8\u53ef\u7528\u6a21\u5f0f\uff0c\u5982\u679c\u9700\u8981\u542f\u7528 ECMP \u6a21\u5f0f\uff0c \u9700\u8981\u66f4\u6539 kube-ovn-controller Deployment \u7684\u53c2\u6570\u8fdb\u884c\u914d\u7f6e: args : - --enable-ecmp=true \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableEcmp \uff0c\u5c06\u96c6\u4e2d\u5f0f\u5b50\u7f51 ECMP \u5f00\u5173\u63a7\u5236\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f ECMP \u6a21\u5f0f\u3002\u539f\u6709\u7684 kube-ovn-controller Deployment \u4e2d\u7684 enable-ecmp \u53c2\u6570\u4e0d\u518d\u4f7f\u7528\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51\u5f00\u5173\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002 \u96c6\u4e2d\u5f0f\u7f51\u5173\u9ed8\u8ba4\u5b89\u88c5\u4e0b\u4e3a\u4e3b\u5907\u6a21\u5f0f\uff0c\u66f4\u591a\u7f51\u5173\u76f8\u5173\u5185\u5bb9\u8bf7\u53c2\u8003 \u5b50\u7f51\u4f7f\u7528 \u3002","title":"\u96c6\u4e2d\u5f0f\u7f51\u5173 ECMP \u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#kubevirt-vm","text":"\u9488\u5bf9 Kubevirt \u521b\u5efa\u7684 VM \u5b9e\u4f8b\uff0c kube-ovn-controller \u53ef\u4ee5\u6309\u7167\u7c7b\u4f3c StatefulSet Pod \u7684\u65b9\u5f0f\u8fdb\u884c IP \u5730\u5740\u5206\u914d\u548c\u7ba1\u7406\u3002 \u4ee5\u8fbe\u5230 VM \u5b9e\u4f8b\u5728\u751f\u547d\u5468\u671f\u5185\u542f\u505c\uff0c\u5347\u7ea7\uff0c\u8fc1\u79fb\u7b49\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\u5730\u5740\u56fa\u5b9a\u4e0d\u53d8\uff0c\u66f4\u7b26\u865a\u62df\u5316\u5408\u7528\u6237\u7684\u5b9e\u9645\u4f7f\u7528\u4f53\u9a8c\u3002 \u8be5\u529f\u80fd\u5728 1.10.6 \u540e\u9ed8\u8ba4\u5f00\u542f\uff0c\u82e5\u8981\u5173\u95ed\u6b64\u529f\u80fd\uff0c\u9700\u8981\u5728 kube-ovn-controller Deployment \u7684\u542f\u52a8\u547d\u4ee4\u4e2d\u8bbe\u7f6e\u5982\u4e0b\u53c2\u6570\uff1a args : - --keep-vm-ip=false","title":"Kubevirt VM \u56fa\u5b9a\u5730\u5740\u5f00\u542f\u8bbe\u7f6e"},{"location":"guide/setup-options/#cni","text":"Kube-OVN \u9ed8\u8ba4\u4f1a\u5728 /opt/cni/bin \u76ee\u5f55\u4e0b\u5b89\u88c5 CNI \u6267\u884c\u6587\u4ef6\uff0c\u5728 /etc/cni/net.d \u76ee\u5f55\u4e0b\u5b89\u88c5 CNI \u914d\u7f6e\u6587\u4ef6 01-kube-ovn.conflist \u3002 \u5982\u679c\u9700\u8981\u66f4\u6539\u5b89\u88c5\u4f4d\u7f6e\u548c CNI \u914d\u7f6e\u6587\u4ef6\u7684\u4f18\u5148\u7ea7\uff0c\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u7684\u4e0b\u5217\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a CNI_CONF_DIR = \"/etc/cni/net.d\" CNI_BIN_DIR = \"/opt/cni/bin\" CNI_CONFIG_PRIORITY = \"01\" \u6216\u8005\u5728\u5b89\u88c5\u540e\u66f4\u6539 kube-ovn-cni DaemonSet \u7684 Volume \u6302\u8f7d\u548c\u542f\u52a8\u53c2\u6570\uff1a volumes : - name : cni-conf hostPath : path : \"/etc/cni/net.d\" - name : cni-bin hostPath : path:\"/opt/cni/bin\" ... args : - --cni-conf-name=01-kube-ovn.conflist","title":"CNI \u914d\u7f6e\u76f8\u5173\u8bbe\u7f6e"},{"location":"guide/setup-options/#_4","text":"Kube-OVN \u9ed8\u8ba4 Overlay \u7684\u5c01\u88c5\u6a21\u5f0f\u4e3a Geneve\uff0c\u5982\u679c\u60f3\u66f4\u6362\u4e3a Vxlan \u6216 STT\uff0c\u53ef\u4ee5\u901a\u8fc7\u5b89\u88c5\u811a\u672c\u7684\u4e0b\u5217\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a TUNNEL_TYPE = \"vxlan\" \u6216\u8005\u5728\u5b89\u88c5\u540e\u66f4\u6539 ovs-ovn DaemonSet \u7684\u73af\u5883\u53d8\u91cf\uff1a env : - name : TUNNEL_TYPE value : \"vxlan\" \u5982\u679c\u9700\u8981\u4f7f\u7528 STT \u96a7\u9053\u9700\u8981\u989d\u5916\u7f16\u8bd1 ovs \u7684\u5185\u6838\u6a21\u5757\uff0c\u8bf7\u53c2\u8003 \u6027\u80fd\u8c03\u4f18 \u3002 \u4e0d\u540c\u534f\u8bae\u5728\u5b9e\u9645\u4f7f\u7528\u4e2d\u7684\u533a\u522b\u8bf7\u53c2\u8003 \u96a7\u9053\u534f\u8bae\u8bf4\u660e \u3002","title":"\u96a7\u9053\u7c7b\u578b\u8bbe\u7f6e"},{"location":"guide/setup-options/#ssl","text":"OVN DB \u7684 API \u63a5\u53e3\u652f\u6301 SSL \u52a0\u5bc6\u6765\u4fdd\u8bc1\u8fde\u63a5\u5b89\u5168\uff0c\u5982\u8981\u5f00\u542f\u53ef\u8c03\u6574\u5b89\u88c5\u811a\u672c\u4e2d\u7684\u5982\u4e0b\u53c2\u6570: ENABLE_SSL = true SSL \u529f\u80fd\u9ed8\u8ba4\u5b89\u88c5\u4e0b\u4e3a\u5173\u95ed\u6a21\u5f0f\u3002","title":"SSL \u8bbe\u7f6e"},{"location":"guide/setup-options/#ip","text":"kube-ovn-controller/kube-ovn-cni/kube-ovn-monitor \u8fd9\u4e9b\u670d\u52a1\u652f\u6301\u7ed1\u5b9a\u672c\u5730 ip\uff0c\u8be5\u529f\u80fd\u8bbe\u8ba1\u539f\u56e0\u4e3b\u8981\u662f\u56e0\u4e3a\u67d0\u4e9b\u573a\u666f\u4e0b\u51fa\u4e8e\u5b89\u5168\u8003\u8651\u4e0d\u5141\u8bb8\u670d\u52a1\u7ed1\u5b9a 0.0.0.0 \uff08\u6bd4\u5982\u8be5\u670d\u52a1\u90e8\u7f72\u5728\u67d0\u4e2a\u5bf9\u5916\u7f51\u5173\u4e0a\uff0c\u5916\u90e8\u7528\u6237\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7\u516c\u7f51 ip \u5e76\u6307\u5b9a\u7aef\u53e3\u53bb\u8bbf\u95ee\u5230\u8be5\u670d\u52a1\uff09\uff0c\u8be5\u529f\u80fd\u9ed8\u8ba4\u662f\u6253\u5f00\u7684\uff0c\u7531\u5b89\u88c5\u811a\u672c\u4e2d\u5982\u4e0b\u53c2\u6570\u63a7\u5236\uff1a ENABLE_BIND_LOCAL_IP = true \u4ee5 kube-ovn-monitor \u4e3a\u4f8b\uff0c\u5f00\u542f\u529f\u80fd\u540e\u4f1a\u628a\u670d\u52a1\u7ed1\u5b9a\u672c\u5730\u7684 pod ip \u5982\u4e0b\uff1a # netstat -tunlp |grep kube-ovn tcp 0 0 172 .18.0.5:10661 0 .0.0.0:* LISTEN 2612 /./kube-ovn-mon \u5b89\u88c5\u540e\u4e5f\u53ef\u901a\u8fc7\u4fee\u6539\u670d\u52a1\u7684 deployment \u6216\u8005 daemonSet \u7684\u73af\u5883\u53d8\u91cf\u53c2\u6570\u8fdb\u884c\u8c03\u6574\uff1a env : - name : ENABLE_BIND_LOCAL_IP value : \"false\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7ed1\u5b9a\u672c\u5730 ip"},{"location":"guide/static-ip-mac/","text":"\u56fa\u5b9a\u5730\u5740 \u00b6 Kube-OVN \u9ed8\u8ba4\u4f1a\u6839\u636e Pod \u6240\u5728 Namespace \u6240\u5c5e\u7684\u5b50\u7f51\u4e2d\u968f\u673a\u5206\u914d IP \u548c Mac\u3002 \u9488\u5bf9\u5de5\u4f5c\u8d1f\u8f7d\u9700\u8981\u56fa\u5b9a\u5730\u5740\u7684\u60c5\u51b5\uff0cKube-OVN \u6839\u636e\u4e0d\u540c\u7684\u573a\u666f\uff0c\u63d0\u4f9b\u4e86\u591a\u79cd\u56fa\u5b9a\u5730\u5740\u7684\u65b9\u6cd5\uff1a \u5355\u4e2a Pod \u56fa\u5b9a IP/Mac\u3002 Workload \u901a\u7528 IP Pool \u65b9\u5f0f\u6307\u5b9a\u56fa\u5b9a\u5730\u5740\u8303\u56f4\u3002 StatefulSet \u56fa\u5b9a\u5730\u5740\u3002 KubeVirt VM \u56fa\u5b9a\u5730\u5740\u3002 \u5355\u4e2a Pod \u56fa\u5b9a IP \u548c Mac \u00b6 \u53ef\u4ee5\u5728\u521b\u5efa Pod \u65f6\u901a\u8fc7 annotation \u6765\u6307\u5b9a Pod \u8fd0\u884c\u65f6\u6240\u9700\u7684 IP/Mac, kube-ovn-controller \u8fd0\u884c\u65f6\u5c06\u4f1a\u8df3\u8fc7\u5730\u5740\u968f\u673a\u5206\u914d\u9636\u6bb5\uff0c\u7ecf\u8fc7\u51b2\u7a81\u68c0\u6d4b\u540e\u76f4\u63a5\u4f7f\u7528\u6307\u5b9a\u5730\u5740\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u9017\u53f7\u5206\u9694 10.16.0.15,fd00:10:16::15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 spec : containers : - name : static-ip image : docker.io/library/nginx:alpine \u5728\u4f7f\u7528 annotation \u5b9a\u4e49\u5355\u4e2a Pod IP/Mac \u65f6\u9700\u8981\u6ce8\u610f\u4ee5\u4e0b\u51e0\u70b9\uff1a \u6240\u4f7f\u7528\u7684 IP/Mac \u4e0d\u80fd\u548c\u5df2\u6709\u7684 IP/Mac \u51b2\u7a81\u3002 IP \u5fc5\u987b\u5728\u6240\u5c5e\u5b50\u7f51\u7684 CIDR \u5185\u3002 \u53ef\u4ee5\u53ea\u6307\u5b9a IP \u6216 Mac\uff0c\u53ea\u6307\u5b9a\u4e00\u4e2a\u65f6\uff0c\u53e6\u4e00\u4e2a\u4f1a\u968f\u673a\u5206\u914d\u3002 Workload \u901a\u7528 IP Pool \u56fa\u5b9a\u5730\u5740 \u00b6 Kube-OVN \u652f\u6301\u901a\u8fc7 annotation ovn.kubernetes.io/ip_pool \u7ed9 Workload\uff08Deployment/StatefulSet/DaemonSet/Job/CronJob\uff09\u8bbe\u7f6e\u56fa\u5b9a IP\u3002 kube-ovn-controllerr \u4f1a\u81ea\u52a8\u9009\u62e9 ovn.kubernetes.io/ip_pool \u4e2d\u6307\u5b9a\u7684 IP \u5e76\u8fdb\u884c\u51b2\u7a81\u68c0\u6d4b\u3002 IP Pool \u7684 Annotation \u9700\u8981\u52a0\u5728 template \u5185\u7684 annotation \u5b57\u6bb5\uff0c\u9664\u4e86 Kubernetes \u5185\u7f6e\u7684 Workload \u7c7b\u578b\uff0c \u5176\u4ed6\u7528\u6237\u81ea\u5b9a\u4e49\u7684 Workload \u4e5f\u53ef\u4ee5\u4f7f\u7528\u540c\u6837\u7684\u65b9\u5f0f\u8fdb\u884c\u56fa\u5b9a\u5730\u5740\u5206\u914d\u3002 Deployment \u56fa\u5b9a IP \u793a\u4f8b \u00b6 apiVersion : apps/v1 kind : Deployment metadata : name : ippool labels : app : ippool spec : replicas : 2 selector : matchLabels : app : ippool template : metadata : labels : app : ippool annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u5206\u53f7\u8fdb\u884c\u5206\u9694 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::000F;10.16.0.17,fd00:10:16::0010 spec : containers : - name : ippool image : docker.io/library/nginx:alpine \u5bf9 Workload \u4f7f\u7528\u56fa\u5b9a IP \u9700\u8981\u6ce8\u610f\u4ee5\u4e0b\u51e0\u70b9\uff1a ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u5e94\u8be5\u5c5e\u4e8e\u6240\u5728\u5b50\u7f51\u7684 CIDR \u5185\u3002 ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u4e0d\u80fd\u548c\u5df2\u4f7f\u7528\u7684 IP \u51b2\u7a81\u3002 \u5f53 ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u6570\u91cf\u5c0f\u4e8e replicas \u6570\u91cf\u65f6\uff0c\u591a\u51fa\u7684 Pod \u5c06\u65e0\u6cd5\u521b\u5efa\u3002\u4f60\u9700\u8981\u6839\u636e Workload \u7684\u66f4\u65b0\u7b56\u7565\u4ee5\u53ca\u6269\u5bb9\u89c4\u5212\u8c03\u6574 ovn.kubernetes.io/ip_pool \u4e2d IP \u7684\u6570\u91cf\u3002 StatefulSet \u56fa\u5b9a\u5730\u5740 \u00b6 StatefulSet \u548c\u5176\u4ed6 Workload \u76f8\u540c\u53ef\u4ee5\u4f7f\u7528 ovn.kubernetes.io/ip_pool \u6765\u6307\u5b9a Pod \u4f7f\u7528\u7684 IP\u3002 \u7531\u4e8e StatefulSet \u591a\u7528\u4e8e\u6709\u72b6\u6001\u670d\u52a1\uff0c\u5bf9\u7f51\u7edc\u6807\u793a\u7684\u56fa\u5b9a\u6709\u66f4\u9ad8\u7684\u8981\u6c42\uff0cKube-OVN \u505a\u4e86\u7279\u6b8a\u7684\u5f3a\u5316\uff1a Pod \u4f1a\u6309\u987a\u5e8f\u5206\u914d ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP\u3002\u4f8b\u5982 StatefulSet \u7684\u540d\u5b57\u4e3a web\uff0c\u5219 web-0 \u4f1a\u4f7f\u7528 ovn.kubernetes.io/ip_pool \u4e2d\u7684\u7b2c\u4e00\u4e2a IP\uff0c web-1 \u4f1a\u4f7f\u7528\u7b2c\u4e8c\u4e2a IP\uff0c\u4ee5\u6b64\u7c7b\u63a8\u3002 StatefulSet Pod \u5728\u66f4\u65b0\u6216\u5220\u9664\u7684\u8fc7\u7a0b\u4e2d OVN \u4e2d\u7684 logical_switch_port \u4e0d\u4f1a\u5220\u9664\uff0c\u65b0\u751f\u6210\u7684 Pod \u76f4\u63a5\u590d\u7528\u65e7\u7684 interface \u4fe1\u606f\u3002\u56e0\u6b64 Pod \u53ef\u4ee5\u590d\u7528 IP/Mac \u53ca\u5176\u4ed6\u7f51\u7edc\u4fe1\u606f\uff0c\u8fbe\u5230\u548c StatefulSet Volume \u7c7b\u4f3c\u7684\u72b6\u6001\u4fdd\u7559\u529f\u80fd\u3002 \u57fa\u4e8e 2 \u7684\u80fd\u529b\uff0c\u5bf9\u4e8e\u6ca1\u6709 ovn.kubernetes.io/ip_pool \u6ce8\u89e3\u7684 StatefulSet\uff0cPod \u7b2c\u4e00\u6b21\u751f\u6210\u65f6\u4f1a\u968f\u673a\u5206\u914d IP/Mac\uff0c\u4e4b\u540e\u5728\u6574\u4e2a StatefulSet \u7684\u751f\u547d\u5468\u671f\u5185\uff0c\u7f51\u7edc\u4fe1\u606f\u90fd\u4f1a\u4fdd\u6301\u56fa\u5b9a\u3002 StatefulSet \u793a\u4f8b \u00b6 apiVersion : apps/v1 kind : StatefulSet metadata : name : web spec : serviceName : \"nginx\" replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx spec : containers : - name : nginx image : docker.io/library/nginx:alpine ports : - containerPort : 80 name : web \u53ef\u4ee5\u5c1d\u8bd5\u5220\u9664 StatefulSet \u4e0b Pod \u89c2\u5bdf Pod IP \u53d8\u5316\u4fe1\u606f\u3002 KubeVirt VM \u56fa\u5b9a\u5730\u5740 \u00b6 \u9488\u5bf9 KubeVirt \u521b\u5efa\u7684 VM \u5b9e\u4f8b\uff0c kube-ovn-controller \u53ef\u4ee5\u6309\u7167\u7c7b\u4f3c StatefulSet Pod \u7684\u65b9\u5f0f\u8fdb\u884c IP \u5730\u5740\u5206\u914d\u548c\u7ba1\u7406\u3002 \u4ee5\u8fbe\u5230 VM \u5b9e\u4f8b\u5728\u751f\u547d\u5468\u671f\u5185\u542f\u505c\uff0c\u5347\u7ea7\uff0c\u8fc1\u79fb\u7b49\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\u5730\u5740\u56fa\u5b9a\u4e0d\u53d8\uff0c\u66f4\u7b26\u865a\u62df\u5316\u5408\u7528\u6237\u7684\u5b9e\u9645\u4f7f\u7528\u4f53\u9a8c\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u56fa\u5b9a\u5730\u5740"},{"location":"guide/static-ip-mac/#_1","text":"Kube-OVN \u9ed8\u8ba4\u4f1a\u6839\u636e Pod \u6240\u5728 Namespace \u6240\u5c5e\u7684\u5b50\u7f51\u4e2d\u968f\u673a\u5206\u914d IP \u548c Mac\u3002 \u9488\u5bf9\u5de5\u4f5c\u8d1f\u8f7d\u9700\u8981\u56fa\u5b9a\u5730\u5740\u7684\u60c5\u51b5\uff0cKube-OVN \u6839\u636e\u4e0d\u540c\u7684\u573a\u666f\uff0c\u63d0\u4f9b\u4e86\u591a\u79cd\u56fa\u5b9a\u5730\u5740\u7684\u65b9\u6cd5\uff1a \u5355\u4e2a Pod \u56fa\u5b9a IP/Mac\u3002 Workload \u901a\u7528 IP Pool \u65b9\u5f0f\u6307\u5b9a\u56fa\u5b9a\u5730\u5740\u8303\u56f4\u3002 StatefulSet \u56fa\u5b9a\u5730\u5740\u3002 KubeVirt VM \u56fa\u5b9a\u5730\u5740\u3002","title":"\u56fa\u5b9a\u5730\u5740"},{"location":"guide/static-ip-mac/#pod-ip-mac","text":"\u53ef\u4ee5\u5728\u521b\u5efa Pod \u65f6\u901a\u8fc7 annotation \u6765\u6307\u5b9a Pod \u8fd0\u884c\u65f6\u6240\u9700\u7684 IP/Mac, kube-ovn-controller \u8fd0\u884c\u65f6\u5c06\u4f1a\u8df3\u8fc7\u5730\u5740\u968f\u673a\u5206\u914d\u9636\u6bb5\uff0c\u7ecf\u8fc7\u51b2\u7a81\u68c0\u6d4b\u540e\u76f4\u63a5\u4f7f\u7528\u6307\u5b9a\u5730\u5740\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u9017\u53f7\u5206\u9694 10.16.0.15,fd00:10:16::15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 spec : containers : - name : static-ip image : docker.io/library/nginx:alpine \u5728\u4f7f\u7528 annotation \u5b9a\u4e49\u5355\u4e2a Pod IP/Mac \u65f6\u9700\u8981\u6ce8\u610f\u4ee5\u4e0b\u51e0\u70b9\uff1a \u6240\u4f7f\u7528\u7684 IP/Mac \u4e0d\u80fd\u548c\u5df2\u6709\u7684 IP/Mac \u51b2\u7a81\u3002 IP \u5fc5\u987b\u5728\u6240\u5c5e\u5b50\u7f51\u7684 CIDR \u5185\u3002 \u53ef\u4ee5\u53ea\u6307\u5b9a IP \u6216 Mac\uff0c\u53ea\u6307\u5b9a\u4e00\u4e2a\u65f6\uff0c\u53e6\u4e00\u4e2a\u4f1a\u968f\u673a\u5206\u914d\u3002","title":"\u5355\u4e2a Pod \u56fa\u5b9a IP \u548c Mac"},{"location":"guide/static-ip-mac/#workload-ip-pool","text":"Kube-OVN \u652f\u6301\u901a\u8fc7 annotation ovn.kubernetes.io/ip_pool \u7ed9 Workload\uff08Deployment/StatefulSet/DaemonSet/Job/CronJob\uff09\u8bbe\u7f6e\u56fa\u5b9a IP\u3002 kube-ovn-controllerr \u4f1a\u81ea\u52a8\u9009\u62e9 ovn.kubernetes.io/ip_pool \u4e2d\u6307\u5b9a\u7684 IP \u5e76\u8fdb\u884c\u51b2\u7a81\u68c0\u6d4b\u3002 IP Pool \u7684 Annotation \u9700\u8981\u52a0\u5728 template \u5185\u7684 annotation \u5b57\u6bb5\uff0c\u9664\u4e86 Kubernetes \u5185\u7f6e\u7684 Workload \u7c7b\u578b\uff0c \u5176\u4ed6\u7528\u6237\u81ea\u5b9a\u4e49\u7684 Workload \u4e5f\u53ef\u4ee5\u4f7f\u7528\u540c\u6837\u7684\u65b9\u5f0f\u8fdb\u884c\u56fa\u5b9a\u5730\u5740\u5206\u914d\u3002","title":"Workload \u901a\u7528 IP Pool \u56fa\u5b9a\u5730\u5740"},{"location":"guide/static-ip-mac/#deployment-ip","text":"apiVersion : apps/v1 kind : Deployment metadata : name : ippool labels : app : ippool spec : replicas : 2 selector : matchLabels : app : ippool template : metadata : labels : app : ippool annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u5206\u53f7\u8fdb\u884c\u5206\u9694 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::000F;10.16.0.17,fd00:10:16::0010 spec : containers : - name : ippool image : docker.io/library/nginx:alpine \u5bf9 Workload \u4f7f\u7528\u56fa\u5b9a IP \u9700\u8981\u6ce8\u610f\u4ee5\u4e0b\u51e0\u70b9\uff1a ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u5e94\u8be5\u5c5e\u4e8e\u6240\u5728\u5b50\u7f51\u7684 CIDR \u5185\u3002 ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u4e0d\u80fd\u548c\u5df2\u4f7f\u7528\u7684 IP \u51b2\u7a81\u3002 \u5f53 ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP \u6570\u91cf\u5c0f\u4e8e replicas \u6570\u91cf\u65f6\uff0c\u591a\u51fa\u7684 Pod \u5c06\u65e0\u6cd5\u521b\u5efa\u3002\u4f60\u9700\u8981\u6839\u636e Workload \u7684\u66f4\u65b0\u7b56\u7565\u4ee5\u53ca\u6269\u5bb9\u89c4\u5212\u8c03\u6574 ovn.kubernetes.io/ip_pool \u4e2d IP \u7684\u6570\u91cf\u3002","title":"Deployment \u56fa\u5b9a IP \u793a\u4f8b"},{"location":"guide/static-ip-mac/#statefulset","text":"StatefulSet \u548c\u5176\u4ed6 Workload \u76f8\u540c\u53ef\u4ee5\u4f7f\u7528 ovn.kubernetes.io/ip_pool \u6765\u6307\u5b9a Pod \u4f7f\u7528\u7684 IP\u3002 \u7531\u4e8e StatefulSet \u591a\u7528\u4e8e\u6709\u72b6\u6001\u670d\u52a1\uff0c\u5bf9\u7f51\u7edc\u6807\u793a\u7684\u56fa\u5b9a\u6709\u66f4\u9ad8\u7684\u8981\u6c42\uff0cKube-OVN \u505a\u4e86\u7279\u6b8a\u7684\u5f3a\u5316\uff1a Pod \u4f1a\u6309\u987a\u5e8f\u5206\u914d ovn.kubernetes.io/ip_pool \u4e2d\u7684 IP\u3002\u4f8b\u5982 StatefulSet \u7684\u540d\u5b57\u4e3a web\uff0c\u5219 web-0 \u4f1a\u4f7f\u7528 ovn.kubernetes.io/ip_pool \u4e2d\u7684\u7b2c\u4e00\u4e2a IP\uff0c web-1 \u4f1a\u4f7f\u7528\u7b2c\u4e8c\u4e2a IP\uff0c\u4ee5\u6b64\u7c7b\u63a8\u3002 StatefulSet Pod \u5728\u66f4\u65b0\u6216\u5220\u9664\u7684\u8fc7\u7a0b\u4e2d OVN \u4e2d\u7684 logical_switch_port \u4e0d\u4f1a\u5220\u9664\uff0c\u65b0\u751f\u6210\u7684 Pod \u76f4\u63a5\u590d\u7528\u65e7\u7684 interface \u4fe1\u606f\u3002\u56e0\u6b64 Pod \u53ef\u4ee5\u590d\u7528 IP/Mac \u53ca\u5176\u4ed6\u7f51\u7edc\u4fe1\u606f\uff0c\u8fbe\u5230\u548c StatefulSet Volume \u7c7b\u4f3c\u7684\u72b6\u6001\u4fdd\u7559\u529f\u80fd\u3002 \u57fa\u4e8e 2 \u7684\u80fd\u529b\uff0c\u5bf9\u4e8e\u6ca1\u6709 ovn.kubernetes.io/ip_pool \u6ce8\u89e3\u7684 StatefulSet\uff0cPod \u7b2c\u4e00\u6b21\u751f\u6210\u65f6\u4f1a\u968f\u673a\u5206\u914d IP/Mac\uff0c\u4e4b\u540e\u5728\u6574\u4e2a StatefulSet \u7684\u751f\u547d\u5468\u671f\u5185\uff0c\u7f51\u7edc\u4fe1\u606f\u90fd\u4f1a\u4fdd\u6301\u56fa\u5b9a\u3002","title":"StatefulSet \u56fa\u5b9a\u5730\u5740"},{"location":"guide/static-ip-mac/#statefulset_1","text":"apiVersion : apps/v1 kind : StatefulSet metadata : name : web spec : serviceName : \"nginx\" replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx spec : containers : - name : nginx image : docker.io/library/nginx:alpine ports : - containerPort : 80 name : web \u53ef\u4ee5\u5c1d\u8bd5\u5220\u9664 StatefulSet \u4e0b Pod \u89c2\u5bdf Pod IP \u53d8\u5316\u4fe1\u606f\u3002","title":"StatefulSet \u793a\u4f8b"},{"location":"guide/static-ip-mac/#kubevirt-vm","text":"\u9488\u5bf9 KubeVirt \u521b\u5efa\u7684 VM \u5b9e\u4f8b\uff0c kube-ovn-controller \u53ef\u4ee5\u6309\u7167\u7c7b\u4f3c StatefulSet Pod \u7684\u65b9\u5f0f\u8fdb\u884c IP \u5730\u5740\u5206\u914d\u548c\u7ba1\u7406\u3002 \u4ee5\u8fbe\u5230 VM \u5b9e\u4f8b\u5728\u751f\u547d\u5468\u671f\u5185\u542f\u505c\uff0c\u5347\u7ea7\uff0c\u8fc1\u79fb\u7b49\u64cd\u4f5c\u8fc7\u7a0b\u4e2d\u5730\u5740\u56fa\u5b9a\u4e0d\u53d8\uff0c\u66f4\u7b26\u865a\u62df\u5316\u5408\u7528\u6237\u7684\u5b9e\u9645\u4f7f\u7528\u4f53\u9a8c\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"KubeVirt VM \u56fa\u5b9a\u5730\u5740"},{"location":"guide/subnet/","text":"\u5b50\u7f51\u4f7f\u7528 \u00b6 \u5b50\u7f51\u662f Kube-OVN \u4e2d\u7684\u4e00\u4e2a\u6838\u5fc3\u6982\u5ff5\u548c\u57fa\u672c\u4f7f\u7528\u5355\u5143\uff0cKube-OVN \u4f1a\u4ee5\u5b50\u7f51\u6765\u7ec4\u7ec7 IP \u548c\u7f51\u7edc\u914d\u7f6e\uff0c\u6bcf\u4e2a Namespace \u53ef\u4ee5\u5f52\u5c5e\u4e8e\u7279\u5b9a\u7684\u5b50\u7f51\uff0c Namespace \u4e0b\u7684 Pod \u4f1a\u81ea\u52a8\u4ece\u6240\u5c5e\u7684\u5b50\u7f51\u4e2d\u83b7\u53d6 IP \u5e76\u5171\u4eab\u5b50\u7f51\u7684\u7f51\u7edc\u914d\u7f6e\uff08CIDR\uff0c\u7f51\u5173\u7c7b\u578b\uff0c\u8bbf\u95ee\u63a7\u5236\uff0cNAT \u63a7\u5236\u7b49\uff09\u3002 \u548c\u5176\u4ed6 CNI \u7684\u6bcf\u4e2a\u8282\u70b9\u7ed1\u5b9a\u4e00\u4e2a\u5b50\u7f51\u7684\u5b9e\u73b0\u4e0d\u540c\uff0c\u5728 Kube-OVN \u4e2d\u5b50\u7f51\u4e3a\u4e00\u4e2a\u5168\u5c40\u7684\u865a\u62df\u7f51\u7edc\u914d\u7f6e\uff0c\u540c\u4e00\u4e2a\u5b50\u7f51\u7684\u5730\u5740\u53ef\u4ee5\u5206\u5e03\u5728\u4efb\u610f\u4e00\u4e2a\u8282\u70b9\u4e0a\u3002 Overlay \u548c Underlay \u7684\u5b50\u7f51\u5728\u4f7f\u7528\u548c\u914d\u7f6e\u4e0a\u5b58\u5728\u4e00\u4e9b\u5dee\u5f02\uff0c\u672c\u6587\u6863\u5c06\u4f1a\u4ecb\u7ecd\u4e0d\u540c\u7c7b\u578b\u5b50\u7f51\u7684\u4e00\u4e9b\u5171\u540c\u914d\u7f6e\u548c\u5dee\u5f02\u5316\u529f\u80fd\u3002 \u9ed8\u8ba4\u5b50\u7f51 \u00b6 \u4e3a\u4e86\u65b9\u4fbf\u7528\u6237\u7684\u5feb\u901f\u4e0a\u624b\u4f7f\u7528\uff0cKube-OVN \u5185\u7f6e\u4e86\u4e00\u4e2a\u9ed8\u8ba4\u5b50\u7f51\uff0c\u6240\u6709\u672a\u663e\u5f0f\u58f0\u660e\u5b50\u7f51\u5f52\u5c5e\u7684 Namespace \u4f1a\u81ea\u52a8\u4ece\u9ed8\u8ba4\u5b50\u7f51\u4e2d\u5206\u914d IP\uff0c \u5e76\u4f7f\u7528\u9ed8\u8ba4\u5b50\u7f51\u7684\u7f51\u7edc\u4fe1\u606f\u3002\u8be5\u5b50\u7f51\u7684\u914d\u7f6e\u4e3a\u5b89\u88c5\u65f6\u6307\u5b9a\uff0c\u53ef\u4ee5\u53c2\u8003 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \uff0c \u5982\u679c\u8981\u5728\u5b89\u88c5\u540e\u4fee\u6539\u9ed8\u8ba4\u7f51\u7edc\u7684 CIDR \u8bf7\u53c2\u8003 \u4fee\u6539\u9ed8\u8ba4\u7f51\u7edc \u3002 \u5728 Overlay \u6a21\u5f0f\u4e0b\uff0c\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u4e86\u5206\u5e03\u5f0f\u7f51\u5173\u5e76\u5bf9\u51fa\u7f51\u6d41\u91cf\u8fdb\u884c NAT \u8f6c\u6362\uff0c\u5176\u884c\u4e3a\u548c Flannel \u7684\u9ed8\u8ba4\u884c\u4e3a\u57fa\u672c\u4e00\u81f4\uff0c \u7528\u6237\u65e0\u9700\u989d\u5916\u7684\u914d\u7f6e\u5373\u53ef\u4f7f\u7528\u5230\u5927\u90e8\u5206\u7684\u7f51\u7edc\u529f\u80fd\u3002 \u5728 Underlay \u6a21\u5f0f\u4e0b\uff0c\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u7269\u7406\u7f51\u5173\u4f5c\u4e3a\u51fa\u7f51\u7f51\u5173\uff0c\u5e76\u5f00\u542f arping \u68c0\u67e5\u7f51\u7edc\u8fde\u901a\u6027\u3002 \u67e5\u770b\u9ed8\u8ba4\u5b50\u7f51 \u00b6 \u9ed8\u8ba4\u5b50\u7f51 spec \u4e2d\u7684 default \u5b57\u6bb5\u4e3a true\uff0c\u4e00\u4e2a\u96c6\u7fa4\u4e0b\u53ea\u6709\u4e00\u4e2a\u9ed8\u8ba4\u5b50\u7f51\uff0c\u9ed8\u8ba4\u540d\u4e3a ovn-default \u3002 \u67e5\u770b\u9ed8\u8ba4\u5b50\u7f51\uff1a # kubectl get subnet ovn-default -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: ovn-default resourceVersion: \"1571334\" selfLink: /apis/kubeovn.io/v1/subnets/ovn-default uid: 7e2451f8-fb44-4f7f-b3e0-cfd27f6fd5d6 spec: cidrBlock: 10 .16.0.0/16 default: true excludeIps: - 10 .16.0.1 gateway: 10 .16.0.1 gatewayType: distributed natOutgoing: true private: false protocol: IPv4 Join \u5b50\u7f51 \u00b6 \u5728 Kubernetes \u7684\u7f51\u7edc\u89c4\u8303\u4e2d\uff0c\u8981\u6c42 Node \u53ef\u4ee5\u548c\u6240\u6709\u7684 Pod \u76f4\u63a5\u901a\u4fe1\u3002 \u4e3a\u4e86\u5728 Overlay \u7f51\u7edc\u6a21\u5f0f\u4e0b\u8fbe\u5230\u8fd9\u4e2a\u76ee\u7684\uff0c Kube-OVN \u521b\u5efa\u4e86\u4e00\u4e2a join \u5b50\u7f51\uff0c \u5e76\u5728\u6bcf\u4e2a Node \u8282\u70b9\u521b\u5efa\u4e86\u4e00\u5757\u865a\u62df\u7f51\u5361 ovn0 \u63a5\u5165 join \u5b50\u7f51\uff0c\u901a\u8fc7\u8be5\u7f51\u7edc\u5b8c\u6210\u8282\u70b9\u548c Pod \u4e4b\u95f4\u7684\u7f51\u7edc\u4e92\u901a\u3002 \u8be5\u5b50\u7f51\u7684\u914d\u7f6e\u4e3a\u5b89\u88c5\u65f6\u6307\u5b9a\uff0c\u53ef\u4ee5\u53c2\u8003 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \uff0c\u5982\u679c\u8981\u5728\u5b89\u88c5\u540e\u4fee\u6539\u3002 join \u5b50\u7f51\u7684 CIDR \u8bf7\u53c2\u8003 \u4fee\u6539 Join \u5b50\u7f51 \u67e5\u770b Join \u5b50\u7f51 \u00b6 \u8be5\u5b50\u7f51\u9ed8\u8ba4\u540d\u4e3a join \u4e00\u822c\u65e0\u9700\u5bf9\u8be5\u5b50\u7f51 CIDR \u5916\u7684\u5176\u4ed6\u7f51\u7edc\u914d\u7f6e\u8fdb\u884c\u4fee\u6539\u3002 # kubectl get subnet join -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: join resourceVersion: \"1571333\" selfLink: /apis/kubeovn.io/v1/subnets/join uid: 9c744810-c678-4d50-8a7d-b8ec12ef91b8 spec: cidrBlock: 100 .64.0.0/16 default: false excludeIps: - 100 .64.0.1 gateway: 100 .64.0.1 gatewayNode: \"\" gatewayType: \"\" natOutgoing: false private: false protocol: IPv4 \u5728 node \u8282\u70b9\u67e5\u770b ovn0 \u7f51\u5361\uff1a # ifconfig ovn0 ovn0: flags = 4163 <UP,BROADCAST,RUNNING,MULTICAST> mtu 1420 inet 100 .64.0.4 netmask 255 .255.0.0 broadcast 100 .64.255.255 inet6 fe80::800:ff:fe40:5 prefixlen 64 scopeid 0x20<link> ether 0a:00:00:40:00:05 txqueuelen 1000 ( Ethernet ) RX packets 18 bytes 1428 ( 1 .3 KiB ) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 19 bytes 1810 ( 1 .7 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 \u521b\u5efa\u81ea\u5b9a\u4e49\u5b50\u7f51 \u00b6 \u8fd9\u91cc\u6211\u4eec\u4ecb\u7ecd\u521b\u5efa\u4e00\u4e2a\u5b50\u7f51\uff0c\u5e76\u5c06\u5176\u548c\u67d0\u4e2a Namespace \u505a\u5173\u8054\u7684\u57fa\u672c\u64cd\u4f5c\uff0c\u66f4\u591a\u9ad8\u7ea7\u914d\u7f6e\u8bf7\u53c2\u8003\u540e\u7eed\u5185\u5bb9\u3002 \u521b\u5efa\u5b50\u7f51 \u00b6 cat <<EOF | kubectl create -f - apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 10.66.0.0/16 excludeIps: - 10.66.0.1..10.66.0.10 - 10.66.0.101..10.66.0.151 gateway: 10.66.0.1 gatewayType: distributed natOutgoing: true routeTable: \"\" namespaces: - ns1 - ns2 EOF cidrBlock : \u5b50\u7f51 CIDR \u8303\u56f4\uff0c\u540c\u4e00\u4e2a VPC \u4e0b\u7684\u4e0d\u540c Subnet CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 excludeIps : \u4fdd\u7559\u5730\u5740\u5217\u8868\uff0c\u5bb9\u5668\u7f51\u7edc\u5c06\u4e0d\u4f1a\u81ea\u52a8\u5206\u914d\u5217\u8868\u5185\u7684\u5730\u5740\uff0c\u53ef\u7528\u505a\u56fa\u5b9a IP \u5730\u5740\u5206\u914d\u6bb5\uff0c\u4e5f\u53ef\u5728 Underlay \u6a21\u5f0f\u4e0b\u907f\u514d\u548c\u7269\u7406\u7f51\u7edc\u4e2d\u5df2\u6709\u8bbe\u5907\u51b2\u7a81\u3002 gateway \uff1a\u8be5\u5b50\u7f51\u7f51\u5173\u5730\u5740\uff0cOverlay \u6a21\u5f0f\u4e0b Kube-OVN \u4f1a\u81ea\u52a8\u5206\u914d\u5bf9\u5e94\u7684\u903b\u8f91\u7f51\u5173\uff0cUnderlay \u6a21\u5f0f\u4e0b\u8be5\u5730\u5740\u9700\u4e3a\u5e95\u5c42\u7269\u7406\u7f51\u5173\u5730\u5740\u3002 namespaces : \u7ed1\u5b9a\u8be5\u5b50\u7f51\u7684 Namespace \u5217\u8868\uff0c\u7ed1\u5b9a\u540e Namespace \u4e0b\u7684 Pod \u5c06\u4f1a\u4ece\u5f53\u524d\u5b50\u7f51\u5206\u914d\u5730\u5740\u3002 routeTable : \u5173\u8054\u7684\u8def\u7531\u8868\uff0c\u9ed8\u8ba4\u5173\u8054\u4e3b\u8def\u7531\u8868\uff0c\u8def\u7531\u8868\u5b9a\u4e49\u8bf7\u53c2\u8003 \u9759\u6001\u8def\u7531 \u9a8c\u8bc1\u5b50\u7f51\u7ed1\u5b9a\u751f\u6548 \u00b6 # kubectl create ns ns1 namespace/ns1 created # kubectl run nginx --image=docker.io/library/nginx:alpine -n ns1 deployment.apps/nginx created # kubectl get pod -n ns1 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-74d5899f46-n8wtg 1 /1 Running 0 10s 10 .66.0.11 node1 <none> <none> Overlay \u5b50\u7f51\u7f51\u5173\u914d\u7f6e \u00b6 \u8be5\u529f\u80fd\u53ea\u5bf9 Overlay \u6a21\u5f0f\u5b50\u7f51\u751f\u6548\uff0cUnderlay \u7c7b\u578b\u5b50\u7f51\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u9700\u8981\u501f\u52a9\u5e95\u5c42\u7269\u7406\u7f51\u5173\u3002 Overlay \u5b50\u7f51\u4e0b\u7684 Pod \u9700\u8981\u901a\u8fc7\u7f51\u5173\u6765\u8bbf\u95ee\u96c6\u7fa4\u5916\u90e8\u7f51\u7edc\uff0cKube-OVN \u76ee\u524d\u652f\u6301\u4e24\u79cd\u7c7b\u578b\u7684\u7f51\u5173\uff1a \u5206\u5e03\u5f0f\u7f51\u5173\u548c\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u7528\u6237\u53ef\u4ee5\u5728\u5b50\u7f51\u4e2d\u5bf9\u7f51\u5173\u7684\u7c7b\u578b\u8fdb\u884c\u8c03\u6574\u3002 \u4e24\u79cd\u7c7b\u578b\u7f51\u5173\u5747\u652f\u6301 natOutgoing \u8bbe\u7f6e\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9 Pod \u8bbf\u95ee\u5916\u7f51\u65f6\u662f\u5426\u9700\u8981\u8fdb\u884c snat\u3002 \u5206\u5e03\u5f0f\u7f51\u5173 \u00b6 \u5b50\u7f51\u7684\u9ed8\u8ba4\u7c7b\u578b\u7f51\u5173\uff0c\u6bcf\u4e2a node \u4f1a\u4f5c\u4e3a\u5f53\u524d node \u4e0a pod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u7684\u7f51\u5173\u3002 \u6570\u636e\u5305\u4f1a\u901a\u8fc7\u672c\u673a\u7684 ovn0 \u7f51\u5361\u6d41\u5165\u4e3b\u673a\u7f51\u7edc\u6808\uff0c\u518d\u6839\u636e\u4e3b\u673a\u7684\u8def\u7531\u89c4\u5219\u8fdb\u884c\u51fa\u7f51\u3002 \u5f53 natOutgoing \u4e3a true \u65f6\uff0cPod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u5c06\u4f1a\u4f7f\u7528\u5f53\u524d\u6240\u5728\u5bbf\u4e3b\u673a\u7684 IP\u3002 \u5b50\u7f51\u793a\u4f8b\uff0c\u5176\u4e2d gatewayType \u5b57\u6bb5\u4e3a distributed \uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : distributed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : true \u96c6\u4e2d\u5f0f\u7f51\u5173 \u00b6 \u5982\u679c\u5e0c\u671b\u5b50\u7f51\u5185\u6d41\u91cf\u8bbf\u95ee\u5916\u7f51\u4f7f\u7528\u56fa\u5b9a\u7684 IP\uff0c\u4ee5\u4fbf\u5ba1\u8ba1\u548c\u767d\u540d\u5355\u7b49\u5b89\u5168\u64cd\u4f5c\uff0c\u53ef\u4ee5\u5728\u5b50\u7f51\u4e2d\u8bbe\u7f6e\u7f51\u5173\u7c7b\u578b\u4e3a\u96c6\u4e2d\u5f0f\u7f51\u5173\u3002 \u5728\u96c6\u4e2d\u5f0f\u7f51\u5173\u6a21\u5f0f\u4e0b\uff0cPod \u8bbf\u95ee\u5916\u7f51\u7684\u6570\u636e\u5305\u4f1a\u9996\u5148\u88ab\u8def\u7531\u5230\u7279\u5b9a\u8282\u70b9\u7684 ovn0 \u7f51\u5361\uff0c\u518d\u901a\u8fc7\u4e3b\u673a\u7684\u8def\u7531\u89c4\u5219\u8fdb\u884c\u51fa\u7f51\u3002 \u5f53 natOutgoing \u4e3a true \u65f6\uff0cPod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u5c06\u4f1a\u4f7f\u7528\u7279\u5b9a\u5bbf\u4e3b\u673a\u7684 IP\u3002 \u5b50\u7f51\u793a\u4f8b\uff0c\u5176\u4e2d gatewayType \u5b57\u6bb5\u4e3a centralized \uff0c gatewayNode \u4e3a\u7279\u5b9a\u673a\u5668\u5728 Kubernetes \u4e2d\u7684 NodeName\u3002 \u5176\u4e2d gatewayNode \u5b57\u6bb5\u53ef\u4ee5\u4e3a\u9017\u53f7\u5206\u9694\u7684\u591a\u53f0\u4e3b\u673a\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : centralized spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1,node2\" natOutgoing : true \u96c6\u4e2d\u5f0f\u7f51\u5173\u5982\u679c\u5e0c\u671b\u6307\u5b9a\u673a\u5668\u7684\u7279\u5b9a\u7f51\u5361\u8fdb\u884c\u51fa\u7f51\uff0c gatewayNode \u53ef\u66f4\u6539\u4e3a kube-ovn-worker:172.18.0.2, kube-ovn-control-plane:172.18.0.3 \u683c\u5f0f\u3002 \u96c6\u4e2d\u5f0f\u7f51\u5173\u9ed8\u8ba4\u4e3a\u4e3b\u5907\u6a21\u5f0f\uff0c\u53ea\u6709\u4e3b\u8282\u70b9\u8fdb\u884c\u6d41\u91cf\u8f6c\u53d1\uff0c \u5982\u679c\u9700\u8981\u5207\u6362\u4e3a ECMP \u6a21\u5f0f\uff0c\u8bf7\u53c2\u8003 \u96c6\u4e2d\u5f0f\u7f51\u5173 ECMP \u5f00\u542f\u8bbe\u7f6e \u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableEcmp \uff0c\u5c06\u96c6\u4e2d\u5f0f\u5b50\u7f51 ECMP \u5f00\u5173\u63a7\u5236\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f ECMP \u6a21\u5f0f\u3002\u539f\u6709\u7684 kube-ovn-controller Deployment \u4e2d\u7684 enable-ecmp \u53c2\u6570\u4e0d\u518d\u4f7f\u7528\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51\u5f00\u5173\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002 \u5b50\u7f51 ACL \u8bbe\u7f6e \u00b6 \u5bf9\u4e8e\u6709\u7ec6\u7c92\u5ea6 ACL \u63a7\u5236\u7684\u573a\u666f\uff0cKube-OVN \u7684 Subnet \u63d0\u4f9b\u4e86 ACL \u89c4\u5219\u7684\u8bbe\u7f6e\uff0c\u53ef\u4ee5\u5b9e\u73b0\u7f51\u7edc\u89c4\u5219\u7684\u7cbe\u7ec6\u63a7\u5236\u3002 Subnet \u4e2d\u7684 ACL \u89c4\u5219\u548c OVN \u7684 ACL \u89c4\u5219\u4e00\u81f4\uff0c\u76f8\u5173\u5b57\u6bb5\u5185\u5bb9\u53ef\u4ee5\u53c2\u8003 ovn-nb ACL Table \uff0c match \u5b57\u6bb5\u652f\u6301\u7684\u5b57\u6bb5\u53ef\u53c2\u8003 ovn-sb Logical Flow Table \u3002 \u5141\u8bb8 IP \u5730\u5740\u4e3a 10.10.0.2 \u7684 Pod \u8bbf\u95ee\u6240\u6709\u5730\u5740\uff0c\u4f46\u4e0d\u5141\u8bb8\u5176\u4ed6\u5730\u5740\u4e3b\u52a8\u8bbf\u95ee\u81ea\u5df1\u7684 ACL \u89c4\u5219\u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : acl spec : acls : - action : drop direction : to-lport match : ip4.dst == 10.10.0.2 && ip priority : 1002 - action : allow-related direction : from-lport match : ip4.src == 10.10.0.2 && ip priority : 1002 cidrBlock : 10.10.0.0/24 \u5b50\u7f51\u9694\u79bb\u8bbe\u7f6e \u00b6 \u5b50\u7f51 ACL \u7684\u529f\u80fd\u53ef\u4ee5\u8986\u76d6\u5b50\u7f51\u9694\u79bb\u7684\u529f\u80fd\uff0c\u5e76\u6709\u66f4\u597d\u7684\u7075\u6d3b\u6027\uff0c\u6211\u4eec\u63a8\u8350\u4f7f\u7528\u5b50\u7f51 ACL \u6765\u505a\u76f8\u5e94\u7684\u914d\u7f6e\u3002 \u9ed8\u8ba4\u60c5\u51b5\u4e0b Kube-OVN \u521b\u5efa\u7684\u5b50\u7f51\u4e4b\u95f4\u53ef\u4ee5\u76f8\u4e92\u901a\u4fe1\uff0cPod \u4e5f\u53ef\u4ee5\u901a\u8fc7\u7f51\u5173\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u3002 \u5982\u9700\u5bf9\u5b50\u7f51\u95f4\u7684\u8bbf\u95ee\u8fdb\u884c\u63a7\u5236\uff0c\u53ef\u4ee5\u5728\u5b50\u7f51 CRD \u4e2d\u5c06 private \u8bbe\u7f6e\u4e3a true\uff0c\u5219\u8be5\u5b50\u7f51\u5c06\u548c\u5176\u4ed6\u5b50\u7f51\u4ee5\u53ca\u5916\u90e8\u7f51\u7edc\u9694\u79bb\uff0c \u53ea\u80fd\u8fdb\u884c\u5b50\u7f51\u5185\u90e8\u7684\u901a\u4fe1\u3002\u5982\u9700\u5f00\u767d\u540d\u5355\uff0c\u53ef\u4ee5\u901a\u8fc7 allowSubnets \u8fdb\u884c\u8bbe\u7f6e\u3002 allowSubnets \u5185\u7684\u7f51\u6bb5\u548c\u8be5\u5b50\u7f51\u53ef\u4ee5\u53cc\u5411\u4e92\u8bbf\u3002 \u5f00\u542f\u8bbf\u95ee\u63a7\u5236\u7684\u5b50\u7f51\u793a\u4f8b \u00b6 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : private spec : protocol : IPv4 default : false namespaces : - ns1 - ns2 cidrBlock : 10.69.0.0/16 private : true allowSubnets : - 10.16.0.0/16 - 10.18.0.0/16 Underlay \u76f8\u5173\u9009\u9879 \u00b6 \u8be5\u90e8\u5206\u529f\u80fd\u53ea\u5bf9 Underlay \u7c7b\u578b\u5b50\u7f51\u751f\u6548\u3002 vlan : \u5982\u679c\u4f7f\u7528 Underlay \u7f51\u7edc\uff0c\u8be5\u5b57\u6bb5\u7528\u6765\u63a7\u5236\u8be5 Subnet \u548c\u54ea\u4e2a Vlan CR \u8fdb\u884c\u7ed1\u5b9a\u3002\u8be5\u9009\u9879\u9ed8\u8ba4\u4e3a\u7a7a\u5b57\u7b26\u4e32\uff0c\u5373\u4e0d\u4f7f\u7528 Underlay \u7f51\u7edc\u3002 logicalGateway : \u4e00\u4e9b Underlay \u73af\u5883\u4e3a\u7eaf\u4e8c\u5c42\u7f51\u7edc\uff0c\u4e0d\u5b58\u5728\u7269\u7406\u7684\u4e09\u5c42\u7f51\u5173\u3002\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u501f\u52a9 OVN \u672c\u8eab\u7684\u80fd\u529b\u8bbe\u7f6e\u4e00\u4e2a\u865a\u62df\u7f51\u5173\uff0c\u5c06 Underlay \u548c Overlay \u7f51\u7edc\u6253\u901a\u3002\u9ed8\u8ba4\u503c\u4e3a\uff1a false \u3002 \u7f51\u5173\u68c0\u67e5\u8bbe\u7f6e \u00b6 \u9ed8\u8ba4\u60c5\u51b5\u4e0b kube-ovn-cni \u5728\u542f\u52a8 Pod \u540e\u4f1a\u4f7f\u7528 ICMP \u6216 ARP \u534f\u8bae\u8bf7\u6c42\u7f51\u5173\u5e76\u7b49\u5f85\u8fd4\u56de\uff0c \u4ee5\u9a8c\u8bc1\u7f51\u7edc\u5de5\u4f5c\u6b63\u5e38\uff0c\u5728\u90e8\u5206 Underlay \u73af\u5883\u7f51\u5173\u65e0\u6cd5\u54cd\u5e94 ARP \u8bf7\u6c42\uff0c\u6216\u65e0\u9700\u7f51\u7edc\u5916\u90e8\u8054\u901a\u7684\u573a\u666f \u53ef\u4ee5\u5173\u95ed\u7f51\u5173\u68c0\u67e5\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : disable-gw-check spec : disableGatewayCheck : true \u5176\u4ed6\u9ad8\u7ea7\u8bbe\u7f6e \u00b6 IP \u6c60\u4f7f\u7528 QoS \u8bbe\u7f6e \u591a\u7f51\u5361\u7ba1\u7406 DHCP \u9009\u9879 \u5916\u90e8\u7f51\u5173\u8bbe\u7f6e \u96c6\u7fa4\u4e92\u8054\u8bbe\u7f6e \u865a\u62df IP \u8bbe\u7f6e \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5b50\u7f51\u4f7f\u7528"},{"location":"guide/subnet/#_1","text":"\u5b50\u7f51\u662f Kube-OVN \u4e2d\u7684\u4e00\u4e2a\u6838\u5fc3\u6982\u5ff5\u548c\u57fa\u672c\u4f7f\u7528\u5355\u5143\uff0cKube-OVN \u4f1a\u4ee5\u5b50\u7f51\u6765\u7ec4\u7ec7 IP \u548c\u7f51\u7edc\u914d\u7f6e\uff0c\u6bcf\u4e2a Namespace \u53ef\u4ee5\u5f52\u5c5e\u4e8e\u7279\u5b9a\u7684\u5b50\u7f51\uff0c Namespace \u4e0b\u7684 Pod \u4f1a\u81ea\u52a8\u4ece\u6240\u5c5e\u7684\u5b50\u7f51\u4e2d\u83b7\u53d6 IP \u5e76\u5171\u4eab\u5b50\u7f51\u7684\u7f51\u7edc\u914d\u7f6e\uff08CIDR\uff0c\u7f51\u5173\u7c7b\u578b\uff0c\u8bbf\u95ee\u63a7\u5236\uff0cNAT \u63a7\u5236\u7b49\uff09\u3002 \u548c\u5176\u4ed6 CNI \u7684\u6bcf\u4e2a\u8282\u70b9\u7ed1\u5b9a\u4e00\u4e2a\u5b50\u7f51\u7684\u5b9e\u73b0\u4e0d\u540c\uff0c\u5728 Kube-OVN \u4e2d\u5b50\u7f51\u4e3a\u4e00\u4e2a\u5168\u5c40\u7684\u865a\u62df\u7f51\u7edc\u914d\u7f6e\uff0c\u540c\u4e00\u4e2a\u5b50\u7f51\u7684\u5730\u5740\u53ef\u4ee5\u5206\u5e03\u5728\u4efb\u610f\u4e00\u4e2a\u8282\u70b9\u4e0a\u3002 Overlay \u548c Underlay \u7684\u5b50\u7f51\u5728\u4f7f\u7528\u548c\u914d\u7f6e\u4e0a\u5b58\u5728\u4e00\u4e9b\u5dee\u5f02\uff0c\u672c\u6587\u6863\u5c06\u4f1a\u4ecb\u7ecd\u4e0d\u540c\u7c7b\u578b\u5b50\u7f51\u7684\u4e00\u4e9b\u5171\u540c\u914d\u7f6e\u548c\u5dee\u5f02\u5316\u529f\u80fd\u3002","title":"\u5b50\u7f51\u4f7f\u7528"},{"location":"guide/subnet/#_2","text":"\u4e3a\u4e86\u65b9\u4fbf\u7528\u6237\u7684\u5feb\u901f\u4e0a\u624b\u4f7f\u7528\uff0cKube-OVN \u5185\u7f6e\u4e86\u4e00\u4e2a\u9ed8\u8ba4\u5b50\u7f51\uff0c\u6240\u6709\u672a\u663e\u5f0f\u58f0\u660e\u5b50\u7f51\u5f52\u5c5e\u7684 Namespace \u4f1a\u81ea\u52a8\u4ece\u9ed8\u8ba4\u5b50\u7f51\u4e2d\u5206\u914d IP\uff0c \u5e76\u4f7f\u7528\u9ed8\u8ba4\u5b50\u7f51\u7684\u7f51\u7edc\u4fe1\u606f\u3002\u8be5\u5b50\u7f51\u7684\u914d\u7f6e\u4e3a\u5b89\u88c5\u65f6\u6307\u5b9a\uff0c\u53ef\u4ee5\u53c2\u8003 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \uff0c \u5982\u679c\u8981\u5728\u5b89\u88c5\u540e\u4fee\u6539\u9ed8\u8ba4\u7f51\u7edc\u7684 CIDR \u8bf7\u53c2\u8003 \u4fee\u6539\u9ed8\u8ba4\u7f51\u7edc \u3002 \u5728 Overlay \u6a21\u5f0f\u4e0b\uff0c\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u4e86\u5206\u5e03\u5f0f\u7f51\u5173\u5e76\u5bf9\u51fa\u7f51\u6d41\u91cf\u8fdb\u884c NAT \u8f6c\u6362\uff0c\u5176\u884c\u4e3a\u548c Flannel \u7684\u9ed8\u8ba4\u884c\u4e3a\u57fa\u672c\u4e00\u81f4\uff0c \u7528\u6237\u65e0\u9700\u989d\u5916\u7684\u914d\u7f6e\u5373\u53ef\u4f7f\u7528\u5230\u5927\u90e8\u5206\u7684\u7f51\u7edc\u529f\u80fd\u3002 \u5728 Underlay \u6a21\u5f0f\u4e0b\uff0c\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528\u7269\u7406\u7f51\u5173\u4f5c\u4e3a\u51fa\u7f51\u7f51\u5173\uff0c\u5e76\u5f00\u542f arping \u68c0\u67e5\u7f51\u7edc\u8fde\u901a\u6027\u3002","title":"\u9ed8\u8ba4\u5b50\u7f51"},{"location":"guide/subnet/#_3","text":"\u9ed8\u8ba4\u5b50\u7f51 spec \u4e2d\u7684 default \u5b57\u6bb5\u4e3a true\uff0c\u4e00\u4e2a\u96c6\u7fa4\u4e0b\u53ea\u6709\u4e00\u4e2a\u9ed8\u8ba4\u5b50\u7f51\uff0c\u9ed8\u8ba4\u540d\u4e3a ovn-default \u3002 \u67e5\u770b\u9ed8\u8ba4\u5b50\u7f51\uff1a # kubectl get subnet ovn-default -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: ovn-default resourceVersion: \"1571334\" selfLink: /apis/kubeovn.io/v1/subnets/ovn-default uid: 7e2451f8-fb44-4f7f-b3e0-cfd27f6fd5d6 spec: cidrBlock: 10 .16.0.0/16 default: true excludeIps: - 10 .16.0.1 gateway: 10 .16.0.1 gatewayType: distributed natOutgoing: true private: false protocol: IPv4","title":"\u67e5\u770b\u9ed8\u8ba4\u5b50\u7f51"},{"location":"guide/subnet/#join","text":"\u5728 Kubernetes \u7684\u7f51\u7edc\u89c4\u8303\u4e2d\uff0c\u8981\u6c42 Node \u53ef\u4ee5\u548c\u6240\u6709\u7684 Pod \u76f4\u63a5\u901a\u4fe1\u3002 \u4e3a\u4e86\u5728 Overlay \u7f51\u7edc\u6a21\u5f0f\u4e0b\u8fbe\u5230\u8fd9\u4e2a\u76ee\u7684\uff0c Kube-OVN \u521b\u5efa\u4e86\u4e00\u4e2a join \u5b50\u7f51\uff0c \u5e76\u5728\u6bcf\u4e2a Node \u8282\u70b9\u521b\u5efa\u4e86\u4e00\u5757\u865a\u62df\u7f51\u5361 ovn0 \u63a5\u5165 join \u5b50\u7f51\uff0c\u901a\u8fc7\u8be5\u7f51\u7edc\u5b8c\u6210\u8282\u70b9\u548c Pod \u4e4b\u95f4\u7684\u7f51\u7edc\u4e92\u901a\u3002 \u8be5\u5b50\u7f51\u7684\u914d\u7f6e\u4e3a\u5b89\u88c5\u65f6\u6307\u5b9a\uff0c\u53ef\u4ee5\u53c2\u8003 \u5185\u7f6e\u7f51\u7edc\u8bbe\u7f6e \uff0c\u5982\u679c\u8981\u5728\u5b89\u88c5\u540e\u4fee\u6539\u3002 join \u5b50\u7f51\u7684 CIDR \u8bf7\u53c2\u8003 \u4fee\u6539 Join \u5b50\u7f51","title":"Join \u5b50\u7f51"},{"location":"guide/subnet/#join_1","text":"\u8be5\u5b50\u7f51\u9ed8\u8ba4\u540d\u4e3a join \u4e00\u822c\u65e0\u9700\u5bf9\u8be5\u5b50\u7f51 CIDR \u5916\u7684\u5176\u4ed6\u7f51\u7edc\u914d\u7f6e\u8fdb\u884c\u4fee\u6539\u3002 # kubectl get subnet join -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: join resourceVersion: \"1571333\" selfLink: /apis/kubeovn.io/v1/subnets/join uid: 9c744810-c678-4d50-8a7d-b8ec12ef91b8 spec: cidrBlock: 100 .64.0.0/16 default: false excludeIps: - 100 .64.0.1 gateway: 100 .64.0.1 gatewayNode: \"\" gatewayType: \"\" natOutgoing: false private: false protocol: IPv4 \u5728 node \u8282\u70b9\u67e5\u770b ovn0 \u7f51\u5361\uff1a # ifconfig ovn0 ovn0: flags = 4163 <UP,BROADCAST,RUNNING,MULTICAST> mtu 1420 inet 100 .64.0.4 netmask 255 .255.0.0 broadcast 100 .64.255.255 inet6 fe80::800:ff:fe40:5 prefixlen 64 scopeid 0x20<link> ether 0a:00:00:40:00:05 txqueuelen 1000 ( Ethernet ) RX packets 18 bytes 1428 ( 1 .3 KiB ) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 19 bytes 1810 ( 1 .7 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0","title":"\u67e5\u770b Join \u5b50\u7f51"},{"location":"guide/subnet/#_4","text":"\u8fd9\u91cc\u6211\u4eec\u4ecb\u7ecd\u521b\u5efa\u4e00\u4e2a\u5b50\u7f51\uff0c\u5e76\u5c06\u5176\u548c\u67d0\u4e2a Namespace \u505a\u5173\u8054\u7684\u57fa\u672c\u64cd\u4f5c\uff0c\u66f4\u591a\u9ad8\u7ea7\u914d\u7f6e\u8bf7\u53c2\u8003\u540e\u7eed\u5185\u5bb9\u3002","title":"\u521b\u5efa\u81ea\u5b9a\u4e49\u5b50\u7f51"},{"location":"guide/subnet/#_5","text":"cat <<EOF | kubectl create -f - apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 10.66.0.0/16 excludeIps: - 10.66.0.1..10.66.0.10 - 10.66.0.101..10.66.0.151 gateway: 10.66.0.1 gatewayType: distributed natOutgoing: true routeTable: \"\" namespaces: - ns1 - ns2 EOF cidrBlock : \u5b50\u7f51 CIDR \u8303\u56f4\uff0c\u540c\u4e00\u4e2a VPC \u4e0b\u7684\u4e0d\u540c Subnet CIDR \u4e0d\u80fd\u91cd\u53e0\u3002 excludeIps : \u4fdd\u7559\u5730\u5740\u5217\u8868\uff0c\u5bb9\u5668\u7f51\u7edc\u5c06\u4e0d\u4f1a\u81ea\u52a8\u5206\u914d\u5217\u8868\u5185\u7684\u5730\u5740\uff0c\u53ef\u7528\u505a\u56fa\u5b9a IP \u5730\u5740\u5206\u914d\u6bb5\uff0c\u4e5f\u53ef\u5728 Underlay \u6a21\u5f0f\u4e0b\u907f\u514d\u548c\u7269\u7406\u7f51\u7edc\u4e2d\u5df2\u6709\u8bbe\u5907\u51b2\u7a81\u3002 gateway \uff1a\u8be5\u5b50\u7f51\u7f51\u5173\u5730\u5740\uff0cOverlay \u6a21\u5f0f\u4e0b Kube-OVN \u4f1a\u81ea\u52a8\u5206\u914d\u5bf9\u5e94\u7684\u903b\u8f91\u7f51\u5173\uff0cUnderlay \u6a21\u5f0f\u4e0b\u8be5\u5730\u5740\u9700\u4e3a\u5e95\u5c42\u7269\u7406\u7f51\u5173\u5730\u5740\u3002 namespaces : \u7ed1\u5b9a\u8be5\u5b50\u7f51\u7684 Namespace \u5217\u8868\uff0c\u7ed1\u5b9a\u540e Namespace \u4e0b\u7684 Pod \u5c06\u4f1a\u4ece\u5f53\u524d\u5b50\u7f51\u5206\u914d\u5730\u5740\u3002 routeTable : \u5173\u8054\u7684\u8def\u7531\u8868\uff0c\u9ed8\u8ba4\u5173\u8054\u4e3b\u8def\u7531\u8868\uff0c\u8def\u7531\u8868\u5b9a\u4e49\u8bf7\u53c2\u8003 \u9759\u6001\u8def\u7531","title":"\u521b\u5efa\u5b50\u7f51"},{"location":"guide/subnet/#_6","text":"# kubectl create ns ns1 namespace/ns1 created # kubectl run nginx --image=docker.io/library/nginx:alpine -n ns1 deployment.apps/nginx created # kubectl get pod -n ns1 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-74d5899f46-n8wtg 1 /1 Running 0 10s 10 .66.0.11 node1 <none> <none>","title":"\u9a8c\u8bc1\u5b50\u7f51\u7ed1\u5b9a\u751f\u6548"},{"location":"guide/subnet/#overlay","text":"\u8be5\u529f\u80fd\u53ea\u5bf9 Overlay \u6a21\u5f0f\u5b50\u7f51\u751f\u6548\uff0cUnderlay \u7c7b\u578b\u5b50\u7f51\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u9700\u8981\u501f\u52a9\u5e95\u5c42\u7269\u7406\u7f51\u5173\u3002 Overlay \u5b50\u7f51\u4e0b\u7684 Pod \u9700\u8981\u901a\u8fc7\u7f51\u5173\u6765\u8bbf\u95ee\u96c6\u7fa4\u5916\u90e8\u7f51\u7edc\uff0cKube-OVN \u76ee\u524d\u652f\u6301\u4e24\u79cd\u7c7b\u578b\u7684\u7f51\u5173\uff1a \u5206\u5e03\u5f0f\u7f51\u5173\u548c\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u7528\u6237\u53ef\u4ee5\u5728\u5b50\u7f51\u4e2d\u5bf9\u7f51\u5173\u7684\u7c7b\u578b\u8fdb\u884c\u8c03\u6574\u3002 \u4e24\u79cd\u7c7b\u578b\u7f51\u5173\u5747\u652f\u6301 natOutgoing \u8bbe\u7f6e\uff0c\u7528\u6237\u53ef\u4ee5\u9009\u62e9 Pod \u8bbf\u95ee\u5916\u7f51\u65f6\u662f\u5426\u9700\u8981\u8fdb\u884c snat\u3002","title":"Overlay \u5b50\u7f51\u7f51\u5173\u914d\u7f6e"},{"location":"guide/subnet/#_7","text":"\u5b50\u7f51\u7684\u9ed8\u8ba4\u7c7b\u578b\u7f51\u5173\uff0c\u6bcf\u4e2a node \u4f1a\u4f5c\u4e3a\u5f53\u524d node \u4e0a pod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u7684\u7f51\u5173\u3002 \u6570\u636e\u5305\u4f1a\u901a\u8fc7\u672c\u673a\u7684 ovn0 \u7f51\u5361\u6d41\u5165\u4e3b\u673a\u7f51\u7edc\u6808\uff0c\u518d\u6839\u636e\u4e3b\u673a\u7684\u8def\u7531\u89c4\u5219\u8fdb\u884c\u51fa\u7f51\u3002 \u5f53 natOutgoing \u4e3a true \u65f6\uff0cPod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u5c06\u4f1a\u4f7f\u7528\u5f53\u524d\u6240\u5728\u5bbf\u4e3b\u673a\u7684 IP\u3002 \u5b50\u7f51\u793a\u4f8b\uff0c\u5176\u4e2d gatewayType \u5b57\u6bb5\u4e3a distributed \uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : distributed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : true","title":"\u5206\u5e03\u5f0f\u7f51\u5173"},{"location":"guide/subnet/#_8","text":"\u5982\u679c\u5e0c\u671b\u5b50\u7f51\u5185\u6d41\u91cf\u8bbf\u95ee\u5916\u7f51\u4f7f\u7528\u56fa\u5b9a\u7684 IP\uff0c\u4ee5\u4fbf\u5ba1\u8ba1\u548c\u767d\u540d\u5355\u7b49\u5b89\u5168\u64cd\u4f5c\uff0c\u53ef\u4ee5\u5728\u5b50\u7f51\u4e2d\u8bbe\u7f6e\u7f51\u5173\u7c7b\u578b\u4e3a\u96c6\u4e2d\u5f0f\u7f51\u5173\u3002 \u5728\u96c6\u4e2d\u5f0f\u7f51\u5173\u6a21\u5f0f\u4e0b\uff0cPod \u8bbf\u95ee\u5916\u7f51\u7684\u6570\u636e\u5305\u4f1a\u9996\u5148\u88ab\u8def\u7531\u5230\u7279\u5b9a\u8282\u70b9\u7684 ovn0 \u7f51\u5361\uff0c\u518d\u901a\u8fc7\u4e3b\u673a\u7684\u8def\u7531\u89c4\u5219\u8fdb\u884c\u51fa\u7f51\u3002 \u5f53 natOutgoing \u4e3a true \u65f6\uff0cPod \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u5c06\u4f1a\u4f7f\u7528\u7279\u5b9a\u5bbf\u4e3b\u673a\u7684 IP\u3002 \u5b50\u7f51\u793a\u4f8b\uff0c\u5176\u4e2d gatewayType \u5b57\u6bb5\u4e3a centralized \uff0c gatewayNode \u4e3a\u7279\u5b9a\u673a\u5668\u5728 Kubernetes \u4e2d\u7684 NodeName\u3002 \u5176\u4e2d gatewayNode \u5b57\u6bb5\u53ef\u4ee5\u4e3a\u9017\u53f7\u5206\u9694\u7684\u591a\u53f0\u4e3b\u673a\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : centralized spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1,node2\" natOutgoing : true \u96c6\u4e2d\u5f0f\u7f51\u5173\u5982\u679c\u5e0c\u671b\u6307\u5b9a\u673a\u5668\u7684\u7279\u5b9a\u7f51\u5361\u8fdb\u884c\u51fa\u7f51\uff0c gatewayNode \u53ef\u66f4\u6539\u4e3a kube-ovn-worker:172.18.0.2, kube-ovn-control-plane:172.18.0.3 \u683c\u5f0f\u3002 \u96c6\u4e2d\u5f0f\u7f51\u5173\u9ed8\u8ba4\u4e3a\u4e3b\u5907\u6a21\u5f0f\uff0c\u53ea\u6709\u4e3b\u8282\u70b9\u8fdb\u884c\u6d41\u91cf\u8f6c\u53d1\uff0c \u5982\u679c\u9700\u8981\u5207\u6362\u4e3a ECMP \u6a21\u5f0f\uff0c\u8bf7\u53c2\u8003 \u96c6\u4e2d\u5f0f\u7f51\u5173 ECMP \u5f00\u542f\u8bbe\u7f6e \u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u5728 subnet crd \u5b9a\u4e49\u4e2d\u589e\u52a0\u4e86 spec \u5b57\u6bb5 enableEcmp \uff0c\u5c06\u96c6\u4e2d\u5f0f\u5b50\u7f51 ECMP \u5f00\u5173\u63a7\u5236\u8fc1\u79fb\u5230\u5b50\u7f51\u5c42\u7ea7\uff0c\u53ef\u4ee5\u57fa\u4e8e\u4e0d\u540c\u7684\u5b50\u7f51\u5206\u522b\u8bbe\u7f6e\u662f\u5426\u5f00\u542f ECMP \u6a21\u5f0f\u3002\u539f\u6709\u7684 kube-ovn-controller Deployment \u4e2d\u7684 enable-ecmp \u53c2\u6570\u4e0d\u518d\u4f7f\u7528\u3002\u4e4b\u524d\u7248\u672c\u5347\u7ea7\u5230 v1.12.0 \u4e4b\u540e\uff0c\u5b50\u7f51\u5f00\u5173\u4f1a\u81ea\u52a8\u7ee7\u627f\u539f\u6709\u7684\u5168\u5c40\u5f00\u5173\u53c2\u6570\u53d6\u503c\u3002","title":"\u96c6\u4e2d\u5f0f\u7f51\u5173"},{"location":"guide/subnet/#acl","text":"\u5bf9\u4e8e\u6709\u7ec6\u7c92\u5ea6 ACL \u63a7\u5236\u7684\u573a\u666f\uff0cKube-OVN \u7684 Subnet \u63d0\u4f9b\u4e86 ACL \u89c4\u5219\u7684\u8bbe\u7f6e\uff0c\u53ef\u4ee5\u5b9e\u73b0\u7f51\u7edc\u89c4\u5219\u7684\u7cbe\u7ec6\u63a7\u5236\u3002 Subnet \u4e2d\u7684 ACL \u89c4\u5219\u548c OVN \u7684 ACL \u89c4\u5219\u4e00\u81f4\uff0c\u76f8\u5173\u5b57\u6bb5\u5185\u5bb9\u53ef\u4ee5\u53c2\u8003 ovn-nb ACL Table \uff0c match \u5b57\u6bb5\u652f\u6301\u7684\u5b57\u6bb5\u53ef\u53c2\u8003 ovn-sb Logical Flow Table \u3002 \u5141\u8bb8 IP \u5730\u5740\u4e3a 10.10.0.2 \u7684 Pod \u8bbf\u95ee\u6240\u6709\u5730\u5740\uff0c\u4f46\u4e0d\u5141\u8bb8\u5176\u4ed6\u5730\u5740\u4e3b\u52a8\u8bbf\u95ee\u81ea\u5df1\u7684 ACL \u89c4\u5219\u793a\u4f8b\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : acl spec : acls : - action : drop direction : to-lport match : ip4.dst == 10.10.0.2 && ip priority : 1002 - action : allow-related direction : from-lport match : ip4.src == 10.10.0.2 && ip priority : 1002 cidrBlock : 10.10.0.0/24","title":"\u5b50\u7f51 ACL \u8bbe\u7f6e"},{"location":"guide/subnet/#_9","text":"\u5b50\u7f51 ACL \u7684\u529f\u80fd\u53ef\u4ee5\u8986\u76d6\u5b50\u7f51\u9694\u79bb\u7684\u529f\u80fd\uff0c\u5e76\u6709\u66f4\u597d\u7684\u7075\u6d3b\u6027\uff0c\u6211\u4eec\u63a8\u8350\u4f7f\u7528\u5b50\u7f51 ACL \u6765\u505a\u76f8\u5e94\u7684\u914d\u7f6e\u3002 \u9ed8\u8ba4\u60c5\u51b5\u4e0b Kube-OVN \u521b\u5efa\u7684\u5b50\u7f51\u4e4b\u95f4\u53ef\u4ee5\u76f8\u4e92\u901a\u4fe1\uff0cPod \u4e5f\u53ef\u4ee5\u901a\u8fc7\u7f51\u5173\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u3002 \u5982\u9700\u5bf9\u5b50\u7f51\u95f4\u7684\u8bbf\u95ee\u8fdb\u884c\u63a7\u5236\uff0c\u53ef\u4ee5\u5728\u5b50\u7f51 CRD \u4e2d\u5c06 private \u8bbe\u7f6e\u4e3a true\uff0c\u5219\u8be5\u5b50\u7f51\u5c06\u548c\u5176\u4ed6\u5b50\u7f51\u4ee5\u53ca\u5916\u90e8\u7f51\u7edc\u9694\u79bb\uff0c \u53ea\u80fd\u8fdb\u884c\u5b50\u7f51\u5185\u90e8\u7684\u901a\u4fe1\u3002\u5982\u9700\u5f00\u767d\u540d\u5355\uff0c\u53ef\u4ee5\u901a\u8fc7 allowSubnets \u8fdb\u884c\u8bbe\u7f6e\u3002 allowSubnets \u5185\u7684\u7f51\u6bb5\u548c\u8be5\u5b50\u7f51\u53ef\u4ee5\u53cc\u5411\u4e92\u8bbf\u3002","title":"\u5b50\u7f51\u9694\u79bb\u8bbe\u7f6e"},{"location":"guide/subnet/#_10","text":"apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : private spec : protocol : IPv4 default : false namespaces : - ns1 - ns2 cidrBlock : 10.69.0.0/16 private : true allowSubnets : - 10.16.0.0/16 - 10.18.0.0/16","title":"\u5f00\u542f\u8bbf\u95ee\u63a7\u5236\u7684\u5b50\u7f51\u793a\u4f8b"},{"location":"guide/subnet/#underlay","text":"\u8be5\u90e8\u5206\u529f\u80fd\u53ea\u5bf9 Underlay \u7c7b\u578b\u5b50\u7f51\u751f\u6548\u3002 vlan : \u5982\u679c\u4f7f\u7528 Underlay \u7f51\u7edc\uff0c\u8be5\u5b57\u6bb5\u7528\u6765\u63a7\u5236\u8be5 Subnet \u548c\u54ea\u4e2a Vlan CR \u8fdb\u884c\u7ed1\u5b9a\u3002\u8be5\u9009\u9879\u9ed8\u8ba4\u4e3a\u7a7a\u5b57\u7b26\u4e32\uff0c\u5373\u4e0d\u4f7f\u7528 Underlay \u7f51\u7edc\u3002 logicalGateway : \u4e00\u4e9b Underlay \u73af\u5883\u4e3a\u7eaf\u4e8c\u5c42\u7f51\u7edc\uff0c\u4e0d\u5b58\u5728\u7269\u7406\u7684\u4e09\u5c42\u7f51\u5173\u3002\u5728\u8fd9\u79cd\u60c5\u51b5\u4e0b\u53ef\u4ee5\u501f\u52a9 OVN \u672c\u8eab\u7684\u80fd\u529b\u8bbe\u7f6e\u4e00\u4e2a\u865a\u62df\u7f51\u5173\uff0c\u5c06 Underlay \u548c Overlay \u7f51\u7edc\u6253\u901a\u3002\u9ed8\u8ba4\u503c\u4e3a\uff1a false \u3002","title":"Underlay \u76f8\u5173\u9009\u9879"},{"location":"guide/subnet/#_11","text":"\u9ed8\u8ba4\u60c5\u51b5\u4e0b kube-ovn-cni \u5728\u542f\u52a8 Pod \u540e\u4f1a\u4f7f\u7528 ICMP \u6216 ARP \u534f\u8bae\u8bf7\u6c42\u7f51\u5173\u5e76\u7b49\u5f85\u8fd4\u56de\uff0c \u4ee5\u9a8c\u8bc1\u7f51\u7edc\u5de5\u4f5c\u6b63\u5e38\uff0c\u5728\u90e8\u5206 Underlay \u73af\u5883\u7f51\u5173\u65e0\u6cd5\u54cd\u5e94 ARP \u8bf7\u6c42\uff0c\u6216\u65e0\u9700\u7f51\u7edc\u5916\u90e8\u8054\u901a\u7684\u573a\u666f \u53ef\u4ee5\u5173\u95ed\u7f51\u5173\u68c0\u67e5\u3002 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : disable-gw-check spec : disableGatewayCheck : true","title":"\u7f51\u5173\u68c0\u67e5\u8bbe\u7f6e"},{"location":"guide/subnet/#_12","text":"IP \u6c60\u4f7f\u7528 QoS \u8bbe\u7f6e \u591a\u7f51\u5361\u7ba1\u7406 DHCP \u9009\u9879 \u5916\u90e8\u7f51\u5173\u8bbe\u7f6e \u96c6\u7fa4\u4e92\u8054\u8bbe\u7f6e \u865a\u62df IP \u8bbe\u7f6e \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5176\u4ed6\u9ad8\u7ea7\u8bbe\u7f6e"},{"location":"guide/vpc-qos/","text":"VPC QoS \u00b6 Kube-OVN \u652f\u6301\u4f7f\u7528 QoSPolicy CRD \u5bf9\u81ea\u5b9a\u4e49 VPC \u7684\u6d41\u91cf\u901f\u7387\u8fdb\u884c\u9650\u5236\u3002 EIP QoS \u00b6 \u5bf9 EIP \u8fdb\u884c\u9650\u901f\uff0c\u9650\u901f\u503c\u4e3a 1Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 1\uff0c\u8fd9\u91cc shared=false \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ea\u80fd\u7ed9\u8fd9\u4e2a EIP \u4f7f\u7528\u4e14\u652f\u6301\u52a8\u6001\u4fee\u6539 QoSPolicy \u53bb\u53d8\u66f4 QoS \u89c4\u5219\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-eip-example spec : shared : false bindingType : EIP bandwidthLimitRules : - name : eip-ingress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : ingress - name : eip-egress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : egress IptablesEIP \u914d\u7f6e\u5982\u4e0b\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-1 spec : natGwDp : gw1 qosPolicy : qos-eip-example .spec.qosPolicy \u7684\u503c\u652f\u6301\u521b\u5efa\u65f6\u4f20\u5165\uff0c\u4e5f\u652f\u6301\u521b\u5efa\u540e\u4fee\u6539\u3002 \u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 EIP \u00b6 \u901a\u8fc7 label \u67e5\u770b\u5df2\u7ecf\u8bbe\u7f6e\u5bf9\u5e94 qos \u7684 eip\uff1a # kubectl get eip -l ovn.kubernetes.io/qos=qos-eip-example NAME IP MAC NAT NATGWDP READY eip-1 172 .18.11.24 00 :00:00:34:41:0B fip gw1 true VPC NATGW net1 \u7f51\u5361 QoS \u00b6 \u5bf9 VPC NATGW \u7684 net1 \u7f51\u5361\u901f\u7387\u8fdb\u884c\u9650\u5236\uff0c\u9650\u901f\u503c\u4e3a 10Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 3\uff0c\u8fd9\u91cc shared=true \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ef\u4ee5\u540c\u65f6\u7ed9\u591a\u4e2a\u8d44\u6e90\u4f7f\u7528\uff0c\u8fd9\u79cd\u573a\u666f\u4e0b\u4e0d\u5141\u8bb8\u4fee\u6539 QoSPolicy \u7684\u5185\u5bb9\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-ingress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : ingress - name : net1-egress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : egress VpcNatGateway \u914d\u7f6e\u5982\u4e0b\uff1a kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" .spec.qosPolicy \u7684\u503c\u652f\u6301\u521b\u5efa\u4f20\u5165\uff0c\u4e5f\u652f\u6301\u540e\u7eed\u4fee\u6539\u3002 net1 \u7f51\u5361\u7279\u5b9a\u6d41\u91cf QoS \u00b6 \u5bf9 net1 \u7f51\u5361\u4e0a\u7279\u5b9a\u6d41\u91cf\u8fdb\u884c\u9650\u901f\uff0c\u9650\u901f\u503c\u4e3a 5Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 2\uff0c\u8fd9\u91cc shared=true \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ef\u4ee5\u540c\u65f6\u7ed9\u591a\u4e2a\u8d44\u6e90\u4f7f\u7528\uff0c\u6b64\u65f6\u4e0d\u5141\u8bb8\u4fee\u6539 QoSPolicy \u7684\u5185\u5bb9\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-extip-ingress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : ingress matchType : ip matchValue : src 172.18.11.22/32 - name : net1-extip-egress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : egress matchType : ip matchValue : dst 172.18.11.23/32 VpcNatGateway \u914d\u7f6e\u5982\u4e0b\uff1a kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" \u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 NATGW \u00b6 \u901a\u8fc7 label \u67e5\u770b\u5df2\u7ecf\u8bbe\u7f6e\u5bf9\u5e94 qos \u7684 eip\uff1a # kubectl get vpc-nat-gw -l ovn.kubernetes.io/qos=qos-natgw-example NAME VPC SUBNET LANIP gw1 test-vpc-1 net1 10 .0.1.254 \u67e5\u770b qos \u89c4\u5219 \u00b6 # kubectl get qos -A NAME SHARED BINDINGTYPE qos-eip-example false EIP qos-natgw-example true NATGW \u9650\u5236 \u00b6 \u53ea\u6709\u5728\u672a\u4f7f\u7528\u65f6\u624d\u80fd\u5220\u9664 QoS \u7b56\u7565\u3002\u56e0\u6b64\uff0c\u5728\u5220\u9664 QoS \u7b56\u7565\u4e4b\u524d\uff0c\u8bf7\u5148\u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 EIP \u548c NATGW\uff0c\u53bb\u6389\u5b83\u4eec\u7684 spec.qosPolicy \u914d\u7f6e\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC QoS \u914d\u7f6e"},{"location":"guide/vpc-qos/#vpc-qos","text":"Kube-OVN \u652f\u6301\u4f7f\u7528 QoSPolicy CRD \u5bf9\u81ea\u5b9a\u4e49 VPC \u7684\u6d41\u91cf\u901f\u7387\u8fdb\u884c\u9650\u5236\u3002","title":"VPC QoS"},{"location":"guide/vpc-qos/#eip-qos","text":"\u5bf9 EIP \u8fdb\u884c\u9650\u901f\uff0c\u9650\u901f\u503c\u4e3a 1Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 1\uff0c\u8fd9\u91cc shared=false \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ea\u80fd\u7ed9\u8fd9\u4e2a EIP \u4f7f\u7528\u4e14\u652f\u6301\u52a8\u6001\u4fee\u6539 QoSPolicy \u53bb\u53d8\u66f4 QoS \u89c4\u5219\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-eip-example spec : shared : false bindingType : EIP bandwidthLimitRules : - name : eip-ingress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : ingress - name : eip-egress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : egress IptablesEIP \u914d\u7f6e\u5982\u4e0b\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-1 spec : natGwDp : gw1 qosPolicy : qos-eip-example .spec.qosPolicy \u7684\u503c\u652f\u6301\u521b\u5efa\u65f6\u4f20\u5165\uff0c\u4e5f\u652f\u6301\u521b\u5efa\u540e\u4fee\u6539\u3002","title":"EIP QoS"},{"location":"guide/vpc-qos/#qos-eip","text":"\u901a\u8fc7 label \u67e5\u770b\u5df2\u7ecf\u8bbe\u7f6e\u5bf9\u5e94 qos \u7684 eip\uff1a # kubectl get eip -l ovn.kubernetes.io/qos=qos-eip-example NAME IP MAC NAT NATGWDP READY eip-1 172 .18.11.24 00 :00:00:34:41:0B fip gw1 true","title":"\u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 EIP"},{"location":"guide/vpc-qos/#vpc-natgw-net1-qos","text":"\u5bf9 VPC NATGW \u7684 net1 \u7f51\u5361\u901f\u7387\u8fdb\u884c\u9650\u5236\uff0c\u9650\u901f\u503c\u4e3a 10Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 3\uff0c\u8fd9\u91cc shared=true \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ef\u4ee5\u540c\u65f6\u7ed9\u591a\u4e2a\u8d44\u6e90\u4f7f\u7528\uff0c\u8fd9\u79cd\u573a\u666f\u4e0b\u4e0d\u5141\u8bb8\u4fee\u6539 QoSPolicy \u7684\u5185\u5bb9\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-ingress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : ingress - name : net1-egress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : egress VpcNatGateway \u914d\u7f6e\u5982\u4e0b\uff1a kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" .spec.qosPolicy \u7684\u503c\u652f\u6301\u521b\u5efa\u4f20\u5165\uff0c\u4e5f\u652f\u6301\u540e\u7eed\u4fee\u6539\u3002","title":"VPC NATGW net1 \u7f51\u5361 QoS"},{"location":"guide/vpc-qos/#net1-qos","text":"\u5bf9 net1 \u7f51\u5361\u4e0a\u7279\u5b9a\u6d41\u91cf\u8fdb\u884c\u9650\u901f\uff0c\u9650\u901f\u503c\u4e3a 5Mbps\uff0c\u4f18\u5148\u7ea7\u4e3a 2\uff0c\u8fd9\u91cc shared=true \uff0c\u8868\u793a\u8fd9\u4e2a QoSPolicy \u53ef\u4ee5\u540c\u65f6\u7ed9\u591a\u4e2a\u8d44\u6e90\u4f7f\u7528\uff0c\u6b64\u65f6\u4e0d\u5141\u8bb8\u4fee\u6539 QoSPolicy \u7684\u5185\u5bb9\u3002 QoSPolicy \u914d\u7f6e\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-extip-ingress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : ingress matchType : ip matchValue : src 172.18.11.22/32 - name : net1-extip-egress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : egress matchType : ip matchValue : dst 172.18.11.23/32 VpcNatGateway \u914d\u7f6e\u5982\u4e0b\uff1a kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\"","title":"net1 \u7f51\u5361\u7279\u5b9a\u6d41\u91cf QoS"},{"location":"guide/vpc-qos/#qos-natgw","text":"\u901a\u8fc7 label \u67e5\u770b\u5df2\u7ecf\u8bbe\u7f6e\u5bf9\u5e94 qos \u7684 eip\uff1a # kubectl get vpc-nat-gw -l ovn.kubernetes.io/qos=qos-natgw-example NAME VPC SUBNET LANIP gw1 test-vpc-1 net1 10 .0.1.254","title":"\u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 NATGW"},{"location":"guide/vpc-qos/#qos","text":"# kubectl get qos -A NAME SHARED BINDINGTYPE qos-eip-example false EIP qos-natgw-example true NATGW","title":"\u67e5\u770b qos \u89c4\u5219"},{"location":"guide/vpc-qos/#_1","text":"\u53ea\u6709\u5728\u672a\u4f7f\u7528\u65f6\u624d\u80fd\u5220\u9664 QoS \u7b56\u7565\u3002\u56e0\u6b64\uff0c\u5728\u5220\u9664 QoS \u7b56\u7565\u4e4b\u524d\uff0c\u8bf7\u5148\u67e5\u770b\u5df2\u542f\u7528 QoS \u7684 EIP \u548c NATGW\uff0c\u53bb\u6389\u5b83\u4eec\u7684 spec.qosPolicy \u914d\u7f6e\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9650\u5236"},{"location":"guide/vpc/","text":"VPC \u4f7f\u7528 \u00b6 Kube-OVN \u652f\u6301\u591a\u79df\u6237\u9694\u79bb\u7ea7\u522b\u7684 VPC \u7f51\u7edc\u3002\u4e0d\u540c VPC \u7f51\u7edc\u76f8\u4e92\u72ec\u7acb\uff0c\u53ef\u4ee5\u5206\u522b\u914d\u7f6e Subnet \u7f51\u6bb5\uff0c \u8def\u7531\u7b56\u7565\uff0c\u5b89\u5168\u7b56\u7565\uff0c\u51fa\u7f51\u7f51\u5173\uff0cEIP \u7b49\u914d\u7f6e\u3002 VPC \u4e3b\u8981\u7528\u4e8e\u6709\u591a\u79df\u6237\u7f51\u7edc\u5f3a\u9694\u79bb\u7684\u573a\u666f\uff0c\u90e8\u5206 Kubernetes \u7f51\u7edc\u529f\u80fd\u5728\u591a\u79df\u6237\u7f51\u7edc\u4e0b\u5b58\u5728\u51b2\u7a81\u3002 \u4f8b\u5982\u8282\u70b9\u548c Pod \u4e92\u8bbf\uff0cNodePort \u529f\u80fd\uff0c\u57fa\u4e8e\u7f51\u7edc\u8bbf\u95ee\u7684\u5065\u5eb7\u68c0\u67e5\u548c DNS \u80fd\u529b\u5728\u591a\u79df\u6237\u7f51\u7edc\u573a\u666f\u6682\u4e0d\u652f\u6301\u3002 \u4e3a\u4e86\u65b9\u4fbf\u5e38\u89c1 Kubernetes \u7684\u4f7f\u7528\u573a\u666f\uff0cKube-OVN \u9ed8\u8ba4 VPC \u505a\u4e86\u7279\u6b8a\u8bbe\u8ba1\uff0c\u8be5 VPC \u4e0b\u7684 Subnet \u53ef\u4ee5\u6ee1\u8db3 Kubernetes \u89c4\u8303\u3002\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u652f\u6301\u672c\u6587\u6863\u4ecb\u7ecd\u7684\u9759\u6001\u8def\u7531\uff0cEIP \u548c NAT \u7f51\u5173\u7b49\u529f\u80fd\u3002 \u5e38\u89c1\u9694\u79bb\u9700\u6c42\u53ef\u901a\u8fc7\u9ed8\u8ba4 VPC \u4e0b\u7684\u7f51\u7edc\u7b56\u7565\u548c\u5b50\u7f51 ACL \u5b9e\u73b0\uff0c\u5728\u4f7f\u7528\u81ea\u5b9a\u4e49 VPC \u524d\u8bf7\u660e\u786e\u662f\u5426\u9700\u8981 VPC \u7ea7\u522b\u7684\u9694\u79bb\uff0c\u5e76\u4e86\u89e3\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684\u9650\u5236\u3002 \u5728 Underlay \u7f51\u7edc\u4e0b\uff0c\u7269\u7406\u4ea4\u6362\u673a\u8d1f\u8d23\u6570\u636e\u9762\u8f6c\u53d1\uff0cVPC \u65e0\u6cd5\u5bf9 Underlay \u5b50\u7f51\u8fdb\u884c\u9694\u79bb\u3002 \u521b\u5efa\u81ea\u5b9a\u4e49 VPC \u00b6 \u521b\u5efa\u4e24\u4e2a VPC\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : namespaces : - ns1 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-2 spec : namespaces : - ns2 namespaces \u53ef\u4ee5\u9650\u5b9a\u53ea\u6709\u54ea\u4e9b Namespace \u53ef\u4ee5\u4f7f\u7528\u5f53\u524d VPC\uff0c\u82e5\u4e3a\u7a7a\u5219\u4e0d\u9650\u5b9a\u3002 \u521b\u5efa\u4e24\u4e2a\u5b50\u7f51\uff0c\u5206\u5c5e\u4e24\u4e2a\u4e0d\u540c\u7684 VPC \u5e76\u6709\u76f8\u540c\u7684 CIDR: kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : test-vpc-1 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns1 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : test-vpc-2 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns2 \u5206\u522b\u5728\u4e24\u4e2a Namespace \u4e0b\u521b\u5efa Pod: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 namespace : ns1 name : vpc1-pod spec : containers : - name : vpc1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 namespace : ns2 name : vpc2-pod spec : containers : - name : vpc2-pod image : docker.io/library/nginx:alpine \u8fd0\u884c\u6210\u529f\u540e\u53ef\u89c2\u5bdf\u4e24\u4e2a Pod \u5730\u5740\u5c5e\u4e8e\u540c\u4e00\u4e2a CIDR\uff0c\u4f46\u7531\u4e8e\u8fd0\u884c\u5728\u4e0d\u540c\u7684\u79df\u6237 VPC\uff0c\u4e24\u4e2a Pod \u65e0\u6cd5\u76f8\u4e92\u8bbf\u95ee\u3002 \u81ea\u5b9a\u4e49 VPC Pod \u652f\u6301 livenessProbe \u548c readinessProbe \u00b6 \u7531\u4e8e\u5e38\u89c4\u914d\u7f6e\u4e0b\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 Pod \u548c\u8282\u70b9\u7684\u7f51\u7edc\u4e4b\u95f4\u5e76\u4e0d\u4e92\u901a\uff0c\u6240\u4ee5 kubelet \u53d1\u9001\u7684\u63a2\u6d4b\u62a5\u6587\u65e0\u6cd5\u5230\u8fbe\u81ea\u5b9a VPC \u5185\u7684 Pod\u3002Kube-OVN \u901a\u8fc7 TProxy \u5c06 kubelet \u53d1\u9001\u7684\u63a2\u6d4b\u62a5\u6587\u91cd\u5b9a\u5411\u5230\u81ea\u5b9a\u4e49 VPC \u5185\u7684 Pod\uff0c\u4ece\u800c\u5b9e\u73b0\u8fd9\u4e00\u529f\u80fd\u3002 \u914d\u7f6e\u65b9\u6cd5\u5982\u4e0b\uff0c\u5728 Daemonset kube-ovn-cni \u4e2d\u589e\u52a0\u53c2\u6570 --enable-tproxy=true \uff1a spec : template : spec : containers : - args : - --enable-tproxy=true \u8be5\u529f\u80fd\u9650\u5236\u6761\u4ef6\uff1a \u5f53\u540c\u4e00\u4e2a\u8282\u70b9\u4e0b\u51fa\u73b0\u4e0d\u540c VPC \u4e0b\u7684 Pod \u5177\u6709\u76f8\u540c\u7684 IP\uff0c\u63a2\u6d4b\u529f\u80fd\u5931\u6548\u3002 \u76ee\u524d\u6682\u65f6\u53ea\u652f\u6301 tcpSocket \u548c httpGet \u4e24\u79cd\u63a2\u6d4b\u65b9\u5f0f\u3002 \u521b\u5efa VPC \u7f51\u5173 \u00b6 \u81ea\u5b9a\u4e49 VPC \u4e0b\u7684\u5b50\u7f51\u4e0d\u652f\u6301\u9ed8\u8ba4 VPC \u4e0b\u7684\u5206\u5e03\u5f0f\u7f51\u5173\u548c\u96c6\u4e2d\u5f0f\u7f51\u5173\u3002 VPC \u5185\u5bb9\u5668\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u9700\u8981\u901a\u8fc7 VPC \u7f51\u5173\uff0cVPC \u7f51\u5173\u53ef\u4ee5\u6253\u901a\u7269\u7406\u7f51\u7edc\u548c\u79df\u6237\u7f51\u7edc\uff0c\u5e76\u63d0\u4f9b \u6d6e\u52a8 IP\uff0cSNAT \u548c DNAT \u529f\u80fd\u3002 VPC \u7f51\u5173\u529f\u80fd\u4f9d\u8d56 Multus-CNI \u7684\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5b89\u88c5\u8bf7\u53c2\u8003 multus-cni \u3002 \u914d\u7f6e\u5916\u90e8\u7f51\u7edc \u00b6 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-vpc-external-network spec : protocol : IPv4 provider : ovn-vpc-external-network.kube-system cidrBlock : 192.168.0.0/24 gateway : 192.168.0.1 # IP address of the physical gateway excludeIps : - 192.168.0.1..192.168.0.10 --- apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-vpc-external-network namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth1\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-vpc-external-network.kube-system\" } }' \u8be5 Subnet \u7528\u6765\u7ba1\u7406\u53ef\u7528\u7684\u5916\u90e8\u5730\u5740\uff0c\u7f51\u6bb5\u5185\u7684\u5730\u5740\u5c06\u4f1a\u901a\u8fc7 Macvlan \u5206\u914d\u7ed9 VPC \u7f51\u5173\uff0c\u8bf7\u548c\u7f51\u7edc\u7ba1\u7406\u6c9f\u901a\u7ed9\u51fa\u53ef\u7528\u7684\u7269\u7406\u6bb5 IP\u3002 VPC \u7f51\u5173\u4f7f\u7528 Macvlan \u505a\u7269\u7406\u7f51\u7edc\u914d\u7f6e\uff0c NetworkAttachmentDefinition \u7684 master \u9700\u4e3a\u5bf9\u5e94\u7269\u7406\u7f51\u8def\u7f51\u5361\u7684\u7f51\u5361\u540d\u3002 name \u5916\u90e8\u7f51\u7edc\u540d\u79f0\u3002 \u5728 Macvlan \u6a21\u5f0f\u4e0b\uff0c\u9644\u5c5e\u7f51\u5361\u4f1a\u5c06\u6570\u636e\u5305\u76f4\u63a5\u901a\u8fc7\u8be5\u8282\u70b9\u7f51\u5361\u5bf9\u5916\u53d1\u9001\uff0cL2/L3 \u5c42\u9762\u7684\u8f6c\u53d1\u80fd\u529b\u9700\u8981\u4f9d\u8d56\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u3002 \u9700\u8981\u9884\u5148\u5728\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u914d\u7f6e\u5bf9\u5e94\u7684\u7f51\u5173\u3001Vlan \u548c\u5b89\u5168\u7b56\u7565\u7b49\u914d\u7f6e\u3002 \u5bf9\u4e8e OpenStack \u7684 VM \u73af\u5883\uff0c\u9700\u8981\u5c06\u5bf9\u5e94\u7f51\u7edc\u7aef\u53e3\u7684 PortSecurity \u5173\u95ed\u3002 \u5bf9\u4e8e VMware \u7684 vSwitch \u7f51\u7edc\uff0c\u9700\u8981\u5c06 MAC Address Changes , Forged Transmits \u548c Promiscuous Mode Operation \u8bbe\u7f6e\u4e3a allow \u3002 \u5bf9\u4e8e Hyper-V \u865a\u62df\u5316\uff0c\u9700\u8981\u5f00\u542f\u865a\u62df\u673a\u7f51\u5361\u9ad8\u7ea7\u529f\u80fd\u4e2d\u7684 MAC Address Spoofing \u3002 \u516c\u6709\u4e91\uff0c\u4f8b\u5982 AWS\u3001GCE\u3001\u963f\u91cc\u4e91\u7b49\u7531\u4e8e\u4e0d\u652f\u6301\u7528\u6237\u81ea\u5b9a\u4e49 Mac \u65e0\u6cd5\u652f\u6301 Macvlan \u6a21\u5f0f\u7f51\u7edc\u3002 \u7531\u4e8e Macvlan \u672c\u8eab\u7684\u9650\u5236\uff0cMacvlan \u5b50\u63a5\u53e3\u65e0\u6cd5\u8bbf\u95ee\u7236\u63a5\u53e3\u5730\u5740\u3002 \u5982\u679c\u7269\u7406\u7f51\u5361\u5bf9\u5e94\u4ea4\u6362\u673a\u63a5\u53e3\u4e3a Trunk \u6a21\u5f0f\uff0c\u9700\u8981\u5728\u8be5\u7f51\u5361\u4e0a\u521b\u5efa\u5b50\u63a5\u53e3\u518d\u63d0\u4f9b\u7ed9 Macvlan \u4f7f\u7528\u3002 \u5f00\u542f VPC \u7f51\u5173\u529f\u80fd \u00b6 VPC \u7f51\u5173\u529f\u80fd\u9700\u8981\u901a\u8fc7 kube-system \u4e0b\u7684 ovn-vpc-nat-gw-config \u5f00\u542f\uff1a --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-config namespace : kube-system data : image : 'docker.io/kubeovn/vpc-nat-gateway:v1.12.0' --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-gw-config namespace : kube-system data : enable-vpc-nat-gw : 'true' image : \u7f51\u5173 Pod \u6240\u4f7f\u7528\u7684\u955c\u50cf\u3002 enable-vpc-nat-gw \uff1a \u63a7\u5236\u662f\u5426\u542f\u7528 VPC \u7f51\u5173\u529f\u80fd\u3002 \u521b\u5efa VPC \u7f51\u5173\u5e76\u914d\u7f6e\u9ed8\u8ba4\u8def\u7531 \u00b6 kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" externalSubnets : - ovn-vpc-external-network vpc \uff1a\u8be5 VpcNatGateway \u6240\u5c5e\u7684 VPC\u3002 subnet \uff1a\u4e3a VPC \u5185\u67d0\u4e2a Subnet \u540d\uff0cVPC \u7f51\u5173 Pod \u4f1a\u5728\u8be5\u5b50\u7f51\u4e0b\u7528 lanIp \u6765\u8fde\u63a5\u79df\u6237\u7f51\u7edc\u3002 lanIp \uff1a subnet \u5185\u67d0\u4e2a\u672a\u88ab\u4f7f\u7528\u7684 IP\uff0cVPC \u7f51\u5173 Pod \u6700\u7ec8\u4f1a\u4f7f\u7528\u8be5 Pod\u3002\u5f53 VPC \u914d\u7f6e\u8def\u7531\u9700\u8981\u6307\u5411\u5f53\u524d VpcNatGateway \u65f6 nextHopIP \u9700\u8981\u8bbe\u7f6e\u4e3a\u8fd9\u4e2a lanIp \u3002 selector \uff1aVpcNatGateway Pod \u7684\u8282\u70b9\u9009\u62e9\u5668\uff0c\u683c\u5f0f\u548c Kubernetes \u4e2d\u7684 NodeSelector \u683c\u5f0f\u76f8\u540c\u3002 externalSubnets \uff1a VPC \u7f51\u5173\u4f7f\u7528\u7684\u5916\u90e8\u7f51\u7edc\uff0c\u5982\u679c\u4e0d\u914d\u7f6e\u5219\u9ed8\u8ba4\u4f7f\u7528 ovn-vpc-external-network \uff0c\u5f53\u524d\u7248\u672c\u53ea\u652f\u6301\u914d\u7f6e\u4e00\u4e2a\u5916\u90e8\u7f51\u7edc\u3002 \u5176\u4ed6\u53ef\u914d\u53c2\u6570\uff1a tolerations : \u4e3a VPC \u7f51\u5173\u914d\u7f6e\u5bb9\u5fcd\u5ea6\uff0c\u5177\u4f53\u914d\u7f6e\u53c2\u8003 \u6c61\u70b9\u548c\u5bb9\u5fcd\u5ea6 \u3002 affinity : \u4e3a VPC \u7f51\u5173 Pod \u6216\u8282\u70b9\u914d\u7f6e\u4eb2\u548c\u6027\uff0c\u5177\u4f53\u8bbe\u7f6e\u53c2\u8003 \u4eb2\u548c\u6027\u4e0e\u53cd\u4eb2\u548c\u6027 \u3002 \u521b\u5efa EIP \u00b6 EIP \u4e3a\u5916\u90e8\u7f51\u7edc\u6bb5\u7684\u67d0\u4e2a IP \u5206\u914d\u7ed9 VPC \u7f51\u5173\u540e\u53ef\u8fdb\u884c\u6d6e\u52a8 IP\uff0cSNAT \u548c DNAT \u64cd\u4f5c\u3002 \u968f\u673a\u5206\u914d\u4e00\u4e2a\u5730\u5740\u7ed9 EIP\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 \u56fa\u5b9a EIP \u5730\u5740\u5206\u914d\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : natGwDp : gw1 v4ip : 10.0.1.111 \u6307\u5b9a EIP \u6240\u5728\u7684\u5916\u90e8\u7f51\u7edc\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 externalSubnet : ovn-vpc-external-network externalSubnet \uff1a EIP \u6240\u5728\u5916\u90e8\u7f51\u7edc\u540d\u79f0\uff0c\u5982\u679c\u4e0d\u6307\u5b9a\u5219\u9ed8\u8ba4\u4e3a ovn-vpc-external-network \uff0c\u5982\u679c\u6307\u5b9a\u5219\u5fc5\u987b\u4e3a\u6240\u5728 VPC \u7f51\u5173\u7684 externalSubnets \u4e2d\u7684\u4e00\u4e2a\u3002 \u521b\u5efa DNAT \u89c4\u5219 \u00b6 kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipd01 spec : natGwDp : gw1 --- kind : IptablesDnatRule apiVersion : kubeovn.io/v1 metadata : name : dnat01 spec : eip : eipd01 externalPort : '8888' internalIp : 10.0.1.10 internalPort : '80' protocol : tcp \u521b\u5efa SNAT \u89c4\u5219 \u00b6 --- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eips01 spec : natGwDp : gw1 --- kind : IptablesSnatRule apiVersion : kubeovn.io/v1 metadata : name : snat01 spec : eip : eips01 internalCIDR : 10.0.1.0/24 \u521b\u5efa\u6d6e\u52a8 IP \u00b6 --- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipf01 spec : natGwDp : gw1 --- kind : IptablesFIPRule apiVersion : kubeovn.io/v1 metadata : name : fip01 spec : eip : eipf01 internalIp : 10.0.1.5 \u81ea\u5b9a\u4e49\u8def\u7531 \u00b6 \u5728\u81ea\u5b9a\u4e49 VPC \u5185\uff0c\u7528\u6237\u53ef\u4ee5\u81ea\u5b9a\u4e49\u7f51\u7edc\u5185\u90e8\u7684\u8def\u7531\u89c4\u5219\uff0c\u7ed3\u5408\u7f51\u5173\u5b9e\u73b0\u66f4\u7075\u6d3b\u7684\u8f6c\u53d1\u3002 Kube-OVN \u652f\u6301\u9759\u6001\u8def\u7531\u548c\u66f4\u4e3a\u7075\u6d3b\u7684\u7b56\u7565\u8def\u7531\u3002 \u9759\u6001\u8def\u7531 \u00b6 kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : staticRoutes : - cidr : 0.0.0.0/0 nextHopIP : 10.0.1.254 policy : policyDst - cidr : 172.31.0.0/24 nextHopIP : 10.0.1.253 policy : policySrc routeTable : \"rtb1\" policy : \u652f\u6301\u76ee\u7684\u5730\u5740\u8def\u7531 policyDst \u548c\u6e90\u5730\u5740\u8def\u7531 policySrc \u3002 \u5f53\u8def\u7531\u89c4\u5219\u5b58\u5728\u91cd\u53e0\u65f6\uff0cCIDR \u63a9\u7801\u8f83\u957f\u7684\u89c4\u5219\u4f18\u5148\u7ea7\u66f4\u9ad8\uff0c\u82e5\u63a9\u7801\u957f\u5ea6\u76f8\u540c\u5219\u76ee\u7684\u5730\u5740\u8def\u7531\u4f18\u5148\u4e8e\u6e90\u5730\u5740\u8def\u7531\u3002 routeTable : \u53ef\u6307\u5b9a\u9759\u6001\u8def\u7531\u6240\u5728\u7684\u8def\u7531\u8868\uff0c\u9ed8\u8ba4\u5728\u4e3b\u8def\u7531\u8868\u3002\u5b50\u7f51\u5173\u8054\u8def\u7531\u8868\u8bf7\u53c2\u8003 \u521b\u5efa\u5b50\u7f51 \u7b56\u7565\u8def\u7531 \u00b6 \u9488\u5bf9\u9759\u6001\u8def\u7531\u5339\u914d\u7684\u6d41\u91cf\uff0c\u53ef\u901a\u8fc7\u7b56\u7565\u8def\u7531\u8fdb\u884c\u66f4\u7ec6\u7c92\u5ea6\u7684\u63a7\u5236\u3002\u7b56\u7565\u8def\u7531\u63d0\u4f9b\u4e86\u66f4\u7cbe\u786e\u7684\u5339\u914d\u89c4\u5219\uff0c\u4f18\u5148\u7ea7\u63a7\u5236 \u548c\u66f4\u591a\u7684\u8f6c\u53d1\u52a8\u4f5c\u3002\u8be5\u529f\u80fd\u4e3a OVN \u5185\u90e8\u903b\u8f91\u8def\u7531\u5668\u7b56\u7565\u529f\u80fd\u7684\u4e00\u4e2a\u5bf9\u5916\u66b4\u9732\uff0c\u66f4\u591a\u4f7f\u7528\u4fe1\u606f\u8bf7\u53c2\u8003 Logical Router Policy \u3002 \u7b80\u5355\u793a\u4f8b\u5982\u4e0b\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : policyRoutes : - action : drop match : ip4.src==10.0.1.0/24 && ip4.dst==10.0.1.250 priority : 11 - action : reroute match : ip4.src==10.0.1.0/24 nextHopIP : 10.0.1.252 priority : 10 \u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219 \u00b6 Kubernetes \u672c\u8eab\u63d0\u4f9b\u7684 Service \u80fd\u529b\u53ef\u4ee5\u5b8c\u6210\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u529f\u80fd\uff0c\u4f46\u662f\u53d7\u9650\u4e8e Kubernetes \u5b9e\u73b0\uff0c Service \u7684 IP \u5730\u5740\u662f\u5168\u5c40\u5206\u914d\u4e14\u4e0d\u80fd\u91cd\u590d\u3002\u5bf9\u4e8e VPC \u7684\u4f7f\u7528\u573a\u666f\uff0c\u7528\u6237\u5e0c\u671b\u80fd\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740 \u8303\u56f4\uff0c\u4e0d\u540c VPC \u4e0b\u7684\u8d1f\u8f7d\u5747\u8861\u5730\u5740\u53ef\u80fd\u91cd\u53e0\uff0cKubernetes \u5185\u7f6e\u7684 Service \u529f\u80fd\u65e0\u6cd5\u5b8c\u5168\u6ee1\u8db3\u3002 \u9488\u5bf9\u8fd9\u7c7b\u573a\u666f\uff0cKube-OVN \u63d0\u4f9b\u4e86 SwitchLBRule \u8d44\u6e90\uff0c\u7528\u6237\u53ef\u4ee5\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740\u8303\u56f4\u3002 \u4e00\u4e2a `SwitchLBRule`` \u4f8b\u5b50\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP vip \uff1a\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740\u3002 namespace \uff1a\u8d1f\u8f7d\u5747\u8861\u5668\u540e\u7aef Pod \u6240\u5728\u7684 Namespace\u3002 sessionAffinity \uff1a\u548c Service \u7684 sessionAffinity \u529f\u80fd\u76f8\u540c\u3002 selector \uff1a\u548c Service \u7684 selector \u529f\u80fd\u76f8\u540c\u3002 ports \uff1a\u548c Service \u7684 port \u529f\u80fd\u76f8\u540c\u3002 \u67e5\u770b\u90e8\u7f72\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u5668\u89c4\u5219\uff1a # kubectl get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh2 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh2 88m \u81ea\u5b9a\u4e49 vpc-dns \u00b6 \u7531\u4e8e\u81ea\u5b9a\u4e49 VPC \u548c\u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0cVPC \u5185 Pod \u65e0\u6cd5\u4f7f\u7528\u9ed8\u8ba4\u7684 coredns \u670d\u52a1\u8fdb\u884c\u57df\u540d\u89e3\u6790\u3002 \u5982\u679c\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 coredns \u89e3\u6790\u96c6\u7fa4\u5185 Service \u57df\u540d\uff0c\u53ef\u4ee5\u901a\u8fc7 Kube-OVN \u63d0\u4f9b\u7684 vpc-dns \u8d44\u6e90\u6765\u5b9e\u73b0\u3002 \u521b\u5efa\u9644\u52a0\u7f51\u5361 \u00b6 apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }' \u4fee\u6539 ovn-default \u903b\u8f91\u4ea4\u6362\u673a\u7684 provider \u00b6 \u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn \uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn vpc : ovn-cluster \u914d\u7f6e vpc-dns \u7684 ConfigMap \u00b6 \u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : true nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\uff08\u53ef\u7f3a\u7701\uff09 true \u542f\u7528\u529f\u80fd\uff0c false \u5173\u95ed\u529f\u80fd\u3002\u9ed8\u8ba4 true \u3002 coredns-image \uff1a\uff08\u53ef\u7701\u7565\uff09\uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-template \uff1a\uff08\u53ef\u7701\u7565\uff09\uff1adns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\uff1a\u5f53\u524d\u7248\u672c\u4ed3\u5e93\u91cc\u7684 yamls/coredns-template.yaml \u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\uff08\u53ef\u7f3a\u7701\uff09 \u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\u3002 k8s-service-port \uff1a\uff08\u53ef\u7f3a\u7701\uff09\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\u3002 \u90e8\u7f72 vpc-dns \u4f9d\u8d56\u8d44\u6e90 \u00b6 apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u90e8\u7f72 vpc-dns \u00b6 kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u8d44\u6e90\u4fe1\u606f\uff1a [ root@hci-dev-mst-1 kubeovn ] # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u6210\u529f\u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72 \u9650\u5236 \u00b6 \u4e00\u4e2a vpc \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a vpc \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a vpc \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC \u4f7f\u7528"},{"location":"guide/vpc/#vpc","text":"Kube-OVN \u652f\u6301\u591a\u79df\u6237\u9694\u79bb\u7ea7\u522b\u7684 VPC \u7f51\u7edc\u3002\u4e0d\u540c VPC \u7f51\u7edc\u76f8\u4e92\u72ec\u7acb\uff0c\u53ef\u4ee5\u5206\u522b\u914d\u7f6e Subnet \u7f51\u6bb5\uff0c \u8def\u7531\u7b56\u7565\uff0c\u5b89\u5168\u7b56\u7565\uff0c\u51fa\u7f51\u7f51\u5173\uff0cEIP \u7b49\u914d\u7f6e\u3002 VPC \u4e3b\u8981\u7528\u4e8e\u6709\u591a\u79df\u6237\u7f51\u7edc\u5f3a\u9694\u79bb\u7684\u573a\u666f\uff0c\u90e8\u5206 Kubernetes \u7f51\u7edc\u529f\u80fd\u5728\u591a\u79df\u6237\u7f51\u7edc\u4e0b\u5b58\u5728\u51b2\u7a81\u3002 \u4f8b\u5982\u8282\u70b9\u548c Pod \u4e92\u8bbf\uff0cNodePort \u529f\u80fd\uff0c\u57fa\u4e8e\u7f51\u7edc\u8bbf\u95ee\u7684\u5065\u5eb7\u68c0\u67e5\u548c DNS \u80fd\u529b\u5728\u591a\u79df\u6237\u7f51\u7edc\u573a\u666f\u6682\u4e0d\u652f\u6301\u3002 \u4e3a\u4e86\u65b9\u4fbf\u5e38\u89c1 Kubernetes \u7684\u4f7f\u7528\u573a\u666f\uff0cKube-OVN \u9ed8\u8ba4 VPC \u505a\u4e86\u7279\u6b8a\u8bbe\u8ba1\uff0c\u8be5 VPC \u4e0b\u7684 Subnet \u53ef\u4ee5\u6ee1\u8db3 Kubernetes \u89c4\u8303\u3002\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u652f\u6301\u672c\u6587\u6863\u4ecb\u7ecd\u7684\u9759\u6001\u8def\u7531\uff0cEIP \u548c NAT \u7f51\u5173\u7b49\u529f\u80fd\u3002 \u5e38\u89c1\u9694\u79bb\u9700\u6c42\u53ef\u901a\u8fc7\u9ed8\u8ba4 VPC \u4e0b\u7684\u7f51\u7edc\u7b56\u7565\u548c\u5b50\u7f51 ACL \u5b9e\u73b0\uff0c\u5728\u4f7f\u7528\u81ea\u5b9a\u4e49 VPC \u524d\u8bf7\u660e\u786e\u662f\u5426\u9700\u8981 VPC \u7ea7\u522b\u7684\u9694\u79bb\uff0c\u5e76\u4e86\u89e3\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684\u9650\u5236\u3002 \u5728 Underlay \u7f51\u7edc\u4e0b\uff0c\u7269\u7406\u4ea4\u6362\u673a\u8d1f\u8d23\u6570\u636e\u9762\u8f6c\u53d1\uff0cVPC \u65e0\u6cd5\u5bf9 Underlay \u5b50\u7f51\u8fdb\u884c\u9694\u79bb\u3002","title":"VPC \u4f7f\u7528"},{"location":"guide/vpc/#vpc_1","text":"\u521b\u5efa\u4e24\u4e2a VPC\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : namespaces : - ns1 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-2 spec : namespaces : - ns2 namespaces \u53ef\u4ee5\u9650\u5b9a\u53ea\u6709\u54ea\u4e9b Namespace \u53ef\u4ee5\u4f7f\u7528\u5f53\u524d VPC\uff0c\u82e5\u4e3a\u7a7a\u5219\u4e0d\u9650\u5b9a\u3002 \u521b\u5efa\u4e24\u4e2a\u5b50\u7f51\uff0c\u5206\u5c5e\u4e24\u4e2a\u4e0d\u540c\u7684 VPC \u5e76\u6709\u76f8\u540c\u7684 CIDR: kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : test-vpc-1 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns1 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : test-vpc-2 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns2 \u5206\u522b\u5728\u4e24\u4e2a Namespace \u4e0b\u521b\u5efa Pod: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 namespace : ns1 name : vpc1-pod spec : containers : - name : vpc1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 namespace : ns2 name : vpc2-pod spec : containers : - name : vpc2-pod image : docker.io/library/nginx:alpine \u8fd0\u884c\u6210\u529f\u540e\u53ef\u89c2\u5bdf\u4e24\u4e2a Pod \u5730\u5740\u5c5e\u4e8e\u540c\u4e00\u4e2a CIDR\uff0c\u4f46\u7531\u4e8e\u8fd0\u884c\u5728\u4e0d\u540c\u7684\u79df\u6237 VPC\uff0c\u4e24\u4e2a Pod \u65e0\u6cd5\u76f8\u4e92\u8bbf\u95ee\u3002","title":"\u521b\u5efa\u81ea\u5b9a\u4e49 VPC"},{"location":"guide/vpc/#vpc-pod-livenessprobe-readinessprobe","text":"\u7531\u4e8e\u5e38\u89c4\u914d\u7f6e\u4e0b\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 Pod \u548c\u8282\u70b9\u7684\u7f51\u7edc\u4e4b\u95f4\u5e76\u4e0d\u4e92\u901a\uff0c\u6240\u4ee5 kubelet \u53d1\u9001\u7684\u63a2\u6d4b\u62a5\u6587\u65e0\u6cd5\u5230\u8fbe\u81ea\u5b9a VPC \u5185\u7684 Pod\u3002Kube-OVN \u901a\u8fc7 TProxy \u5c06 kubelet \u53d1\u9001\u7684\u63a2\u6d4b\u62a5\u6587\u91cd\u5b9a\u5411\u5230\u81ea\u5b9a\u4e49 VPC \u5185\u7684 Pod\uff0c\u4ece\u800c\u5b9e\u73b0\u8fd9\u4e00\u529f\u80fd\u3002 \u914d\u7f6e\u65b9\u6cd5\u5982\u4e0b\uff0c\u5728 Daemonset kube-ovn-cni \u4e2d\u589e\u52a0\u53c2\u6570 --enable-tproxy=true \uff1a spec : template : spec : containers : - args : - --enable-tproxy=true \u8be5\u529f\u80fd\u9650\u5236\u6761\u4ef6\uff1a \u5f53\u540c\u4e00\u4e2a\u8282\u70b9\u4e0b\u51fa\u73b0\u4e0d\u540c VPC \u4e0b\u7684 Pod \u5177\u6709\u76f8\u540c\u7684 IP\uff0c\u63a2\u6d4b\u529f\u80fd\u5931\u6548\u3002 \u76ee\u524d\u6682\u65f6\u53ea\u652f\u6301 tcpSocket \u548c httpGet \u4e24\u79cd\u63a2\u6d4b\u65b9\u5f0f\u3002","title":"\u81ea\u5b9a\u4e49 VPC Pod \u652f\u6301 livenessProbe \u548c readinessProbe"},{"location":"guide/vpc/#vpc_2","text":"\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684\u5b50\u7f51\u4e0d\u652f\u6301\u9ed8\u8ba4 VPC \u4e0b\u7684\u5206\u5e03\u5f0f\u7f51\u5173\u548c\u96c6\u4e2d\u5f0f\u7f51\u5173\u3002 VPC \u5185\u5bb9\u5668\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u9700\u8981\u901a\u8fc7 VPC \u7f51\u5173\uff0cVPC \u7f51\u5173\u53ef\u4ee5\u6253\u901a\u7269\u7406\u7f51\u7edc\u548c\u79df\u6237\u7f51\u7edc\uff0c\u5e76\u63d0\u4f9b \u6d6e\u52a8 IP\uff0cSNAT \u548c DNAT \u529f\u80fd\u3002 VPC \u7f51\u5173\u529f\u80fd\u4f9d\u8d56 Multus-CNI \u7684\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5b89\u88c5\u8bf7\u53c2\u8003 multus-cni \u3002","title":"\u521b\u5efa VPC \u7f51\u5173"},{"location":"guide/vpc/#_1","text":"apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-vpc-external-network spec : protocol : IPv4 provider : ovn-vpc-external-network.kube-system cidrBlock : 192.168.0.0/24 gateway : 192.168.0.1 # IP address of the physical gateway excludeIps : - 192.168.0.1..192.168.0.10 --- apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-vpc-external-network namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth1\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-vpc-external-network.kube-system\" } }' \u8be5 Subnet \u7528\u6765\u7ba1\u7406\u53ef\u7528\u7684\u5916\u90e8\u5730\u5740\uff0c\u7f51\u6bb5\u5185\u7684\u5730\u5740\u5c06\u4f1a\u901a\u8fc7 Macvlan \u5206\u914d\u7ed9 VPC \u7f51\u5173\uff0c\u8bf7\u548c\u7f51\u7edc\u7ba1\u7406\u6c9f\u901a\u7ed9\u51fa\u53ef\u7528\u7684\u7269\u7406\u6bb5 IP\u3002 VPC \u7f51\u5173\u4f7f\u7528 Macvlan \u505a\u7269\u7406\u7f51\u7edc\u914d\u7f6e\uff0c NetworkAttachmentDefinition \u7684 master \u9700\u4e3a\u5bf9\u5e94\u7269\u7406\u7f51\u8def\u7f51\u5361\u7684\u7f51\u5361\u540d\u3002 name \u5916\u90e8\u7f51\u7edc\u540d\u79f0\u3002 \u5728 Macvlan \u6a21\u5f0f\u4e0b\uff0c\u9644\u5c5e\u7f51\u5361\u4f1a\u5c06\u6570\u636e\u5305\u76f4\u63a5\u901a\u8fc7\u8be5\u8282\u70b9\u7f51\u5361\u5bf9\u5916\u53d1\u9001\uff0cL2/L3 \u5c42\u9762\u7684\u8f6c\u53d1\u80fd\u529b\u9700\u8981\u4f9d\u8d56\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u3002 \u9700\u8981\u9884\u5148\u5728\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u914d\u7f6e\u5bf9\u5e94\u7684\u7f51\u5173\u3001Vlan \u548c\u5b89\u5168\u7b56\u7565\u7b49\u914d\u7f6e\u3002 \u5bf9\u4e8e OpenStack \u7684 VM \u73af\u5883\uff0c\u9700\u8981\u5c06\u5bf9\u5e94\u7f51\u7edc\u7aef\u53e3\u7684 PortSecurity \u5173\u95ed\u3002 \u5bf9\u4e8e VMware \u7684 vSwitch \u7f51\u7edc\uff0c\u9700\u8981\u5c06 MAC Address Changes , Forged Transmits \u548c Promiscuous Mode Operation \u8bbe\u7f6e\u4e3a allow \u3002 \u5bf9\u4e8e Hyper-V \u865a\u62df\u5316\uff0c\u9700\u8981\u5f00\u542f\u865a\u62df\u673a\u7f51\u5361\u9ad8\u7ea7\u529f\u80fd\u4e2d\u7684 MAC Address Spoofing \u3002 \u516c\u6709\u4e91\uff0c\u4f8b\u5982 AWS\u3001GCE\u3001\u963f\u91cc\u4e91\u7b49\u7531\u4e8e\u4e0d\u652f\u6301\u7528\u6237\u81ea\u5b9a\u4e49 Mac \u65e0\u6cd5\u652f\u6301 Macvlan \u6a21\u5f0f\u7f51\u7edc\u3002 \u7531\u4e8e Macvlan \u672c\u8eab\u7684\u9650\u5236\uff0cMacvlan \u5b50\u63a5\u53e3\u65e0\u6cd5\u8bbf\u95ee\u7236\u63a5\u53e3\u5730\u5740\u3002 \u5982\u679c\u7269\u7406\u7f51\u5361\u5bf9\u5e94\u4ea4\u6362\u673a\u63a5\u53e3\u4e3a Trunk \u6a21\u5f0f\uff0c\u9700\u8981\u5728\u8be5\u7f51\u5361\u4e0a\u521b\u5efa\u5b50\u63a5\u53e3\u518d\u63d0\u4f9b\u7ed9 Macvlan \u4f7f\u7528\u3002","title":"\u914d\u7f6e\u5916\u90e8\u7f51\u7edc"},{"location":"guide/vpc/#vpc_3","text":"VPC \u7f51\u5173\u529f\u80fd\u9700\u8981\u901a\u8fc7 kube-system \u4e0b\u7684 ovn-vpc-nat-gw-config \u5f00\u542f\uff1a --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-config namespace : kube-system data : image : 'docker.io/kubeovn/vpc-nat-gateway:v1.12.0' --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-gw-config namespace : kube-system data : enable-vpc-nat-gw : 'true' image : \u7f51\u5173 Pod \u6240\u4f7f\u7528\u7684\u955c\u50cf\u3002 enable-vpc-nat-gw \uff1a \u63a7\u5236\u662f\u5426\u542f\u7528 VPC \u7f51\u5173\u529f\u80fd\u3002","title":"\u5f00\u542f VPC \u7f51\u5173\u529f\u80fd"},{"location":"guide/vpc/#vpc_4","text":"kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" externalSubnets : - ovn-vpc-external-network vpc \uff1a\u8be5 VpcNatGateway \u6240\u5c5e\u7684 VPC\u3002 subnet \uff1a\u4e3a VPC \u5185\u67d0\u4e2a Subnet \u540d\uff0cVPC \u7f51\u5173 Pod \u4f1a\u5728\u8be5\u5b50\u7f51\u4e0b\u7528 lanIp \u6765\u8fde\u63a5\u79df\u6237\u7f51\u7edc\u3002 lanIp \uff1a subnet \u5185\u67d0\u4e2a\u672a\u88ab\u4f7f\u7528\u7684 IP\uff0cVPC \u7f51\u5173 Pod \u6700\u7ec8\u4f1a\u4f7f\u7528\u8be5 Pod\u3002\u5f53 VPC \u914d\u7f6e\u8def\u7531\u9700\u8981\u6307\u5411\u5f53\u524d VpcNatGateway \u65f6 nextHopIP \u9700\u8981\u8bbe\u7f6e\u4e3a\u8fd9\u4e2a lanIp \u3002 selector \uff1aVpcNatGateway Pod \u7684\u8282\u70b9\u9009\u62e9\u5668\uff0c\u683c\u5f0f\u548c Kubernetes \u4e2d\u7684 NodeSelector \u683c\u5f0f\u76f8\u540c\u3002 externalSubnets \uff1a VPC \u7f51\u5173\u4f7f\u7528\u7684\u5916\u90e8\u7f51\u7edc\uff0c\u5982\u679c\u4e0d\u914d\u7f6e\u5219\u9ed8\u8ba4\u4f7f\u7528 ovn-vpc-external-network \uff0c\u5f53\u524d\u7248\u672c\u53ea\u652f\u6301\u914d\u7f6e\u4e00\u4e2a\u5916\u90e8\u7f51\u7edc\u3002 \u5176\u4ed6\u53ef\u914d\u53c2\u6570\uff1a tolerations : \u4e3a VPC \u7f51\u5173\u914d\u7f6e\u5bb9\u5fcd\u5ea6\uff0c\u5177\u4f53\u914d\u7f6e\u53c2\u8003 \u6c61\u70b9\u548c\u5bb9\u5fcd\u5ea6 \u3002 affinity : \u4e3a VPC \u7f51\u5173 Pod \u6216\u8282\u70b9\u914d\u7f6e\u4eb2\u548c\u6027\uff0c\u5177\u4f53\u8bbe\u7f6e\u53c2\u8003 \u4eb2\u548c\u6027\u4e0e\u53cd\u4eb2\u548c\u6027 \u3002","title":"\u521b\u5efa VPC \u7f51\u5173\u5e76\u914d\u7f6e\u9ed8\u8ba4\u8def\u7531"},{"location":"guide/vpc/#eip","text":"EIP \u4e3a\u5916\u90e8\u7f51\u7edc\u6bb5\u7684\u67d0\u4e2a IP \u5206\u914d\u7ed9 VPC \u7f51\u5173\u540e\u53ef\u8fdb\u884c\u6d6e\u52a8 IP\uff0cSNAT \u548c DNAT \u64cd\u4f5c\u3002 \u968f\u673a\u5206\u914d\u4e00\u4e2a\u5730\u5740\u7ed9 EIP\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 \u56fa\u5b9a EIP \u5730\u5740\u5206\u914d\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : natGwDp : gw1 v4ip : 10.0.1.111 \u6307\u5b9a EIP \u6240\u5728\u7684\u5916\u90e8\u7f51\u7edc\uff1a kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 externalSubnet : ovn-vpc-external-network externalSubnet \uff1a EIP \u6240\u5728\u5916\u90e8\u7f51\u7edc\u540d\u79f0\uff0c\u5982\u679c\u4e0d\u6307\u5b9a\u5219\u9ed8\u8ba4\u4e3a ovn-vpc-external-network \uff0c\u5982\u679c\u6307\u5b9a\u5219\u5fc5\u987b\u4e3a\u6240\u5728 VPC \u7f51\u5173\u7684 externalSubnets \u4e2d\u7684\u4e00\u4e2a\u3002","title":"\u521b\u5efa EIP"},{"location":"guide/vpc/#dnat","text":"kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipd01 spec : natGwDp : gw1 --- kind : IptablesDnatRule apiVersion : kubeovn.io/v1 metadata : name : dnat01 spec : eip : eipd01 externalPort : '8888' internalIp : 10.0.1.10 internalPort : '80' protocol : tcp","title":"\u521b\u5efa DNAT \u89c4\u5219"},{"location":"guide/vpc/#snat","text":"--- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eips01 spec : natGwDp : gw1 --- kind : IptablesSnatRule apiVersion : kubeovn.io/v1 metadata : name : snat01 spec : eip : eips01 internalCIDR : 10.0.1.0/24","title":"\u521b\u5efa SNAT \u89c4\u5219"},{"location":"guide/vpc/#ip","text":"--- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipf01 spec : natGwDp : gw1 --- kind : IptablesFIPRule apiVersion : kubeovn.io/v1 metadata : name : fip01 spec : eip : eipf01 internalIp : 10.0.1.5","title":"\u521b\u5efa\u6d6e\u52a8 IP"},{"location":"guide/vpc/#_2","text":"\u5728\u81ea\u5b9a\u4e49 VPC \u5185\uff0c\u7528\u6237\u53ef\u4ee5\u81ea\u5b9a\u4e49\u7f51\u7edc\u5185\u90e8\u7684\u8def\u7531\u89c4\u5219\uff0c\u7ed3\u5408\u7f51\u5173\u5b9e\u73b0\u66f4\u7075\u6d3b\u7684\u8f6c\u53d1\u3002 Kube-OVN \u652f\u6301\u9759\u6001\u8def\u7531\u548c\u66f4\u4e3a\u7075\u6d3b\u7684\u7b56\u7565\u8def\u7531\u3002","title":"\u81ea\u5b9a\u4e49\u8def\u7531"},{"location":"guide/vpc/#_3","text":"kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : staticRoutes : - cidr : 0.0.0.0/0 nextHopIP : 10.0.1.254 policy : policyDst - cidr : 172.31.0.0/24 nextHopIP : 10.0.1.253 policy : policySrc routeTable : \"rtb1\" policy : \u652f\u6301\u76ee\u7684\u5730\u5740\u8def\u7531 policyDst \u548c\u6e90\u5730\u5740\u8def\u7531 policySrc \u3002 \u5f53\u8def\u7531\u89c4\u5219\u5b58\u5728\u91cd\u53e0\u65f6\uff0cCIDR \u63a9\u7801\u8f83\u957f\u7684\u89c4\u5219\u4f18\u5148\u7ea7\u66f4\u9ad8\uff0c\u82e5\u63a9\u7801\u957f\u5ea6\u76f8\u540c\u5219\u76ee\u7684\u5730\u5740\u8def\u7531\u4f18\u5148\u4e8e\u6e90\u5730\u5740\u8def\u7531\u3002 routeTable : \u53ef\u6307\u5b9a\u9759\u6001\u8def\u7531\u6240\u5728\u7684\u8def\u7531\u8868\uff0c\u9ed8\u8ba4\u5728\u4e3b\u8def\u7531\u8868\u3002\u5b50\u7f51\u5173\u8054\u8def\u7531\u8868\u8bf7\u53c2\u8003 \u521b\u5efa\u5b50\u7f51","title":"\u9759\u6001\u8def\u7531"},{"location":"guide/vpc/#_4","text":"\u9488\u5bf9\u9759\u6001\u8def\u7531\u5339\u914d\u7684\u6d41\u91cf\uff0c\u53ef\u901a\u8fc7\u7b56\u7565\u8def\u7531\u8fdb\u884c\u66f4\u7ec6\u7c92\u5ea6\u7684\u63a7\u5236\u3002\u7b56\u7565\u8def\u7531\u63d0\u4f9b\u4e86\u66f4\u7cbe\u786e\u7684\u5339\u914d\u89c4\u5219\uff0c\u4f18\u5148\u7ea7\u63a7\u5236 \u548c\u66f4\u591a\u7684\u8f6c\u53d1\u52a8\u4f5c\u3002\u8be5\u529f\u80fd\u4e3a OVN \u5185\u90e8\u903b\u8f91\u8def\u7531\u5668\u7b56\u7565\u529f\u80fd\u7684\u4e00\u4e2a\u5bf9\u5916\u66b4\u9732\uff0c\u66f4\u591a\u4f7f\u7528\u4fe1\u606f\u8bf7\u53c2\u8003 Logical Router Policy \u3002 \u7b80\u5355\u793a\u4f8b\u5982\u4e0b\uff1a kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : policyRoutes : - action : drop match : ip4.src==10.0.1.0/24 && ip4.dst==10.0.1.250 priority : 11 - action : reroute match : ip4.src==10.0.1.0/24 nextHopIP : 10.0.1.252 priority : 10","title":"\u7b56\u7565\u8def\u7531"},{"location":"guide/vpc/#_5","text":"Kubernetes \u672c\u8eab\u63d0\u4f9b\u7684 Service \u80fd\u529b\u53ef\u4ee5\u5b8c\u6210\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u529f\u80fd\uff0c\u4f46\u662f\u53d7\u9650\u4e8e Kubernetes \u5b9e\u73b0\uff0c Service \u7684 IP \u5730\u5740\u662f\u5168\u5c40\u5206\u914d\u4e14\u4e0d\u80fd\u91cd\u590d\u3002\u5bf9\u4e8e VPC \u7684\u4f7f\u7528\u573a\u666f\uff0c\u7528\u6237\u5e0c\u671b\u80fd\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740 \u8303\u56f4\uff0c\u4e0d\u540c VPC \u4e0b\u7684\u8d1f\u8f7d\u5747\u8861\u5730\u5740\u53ef\u80fd\u91cd\u53e0\uff0cKubernetes \u5185\u7f6e\u7684 Service \u529f\u80fd\u65e0\u6cd5\u5b8c\u5168\u6ee1\u8db3\u3002 \u9488\u5bf9\u8fd9\u7c7b\u573a\u666f\uff0cKube-OVN \u63d0\u4f9b\u4e86 SwitchLBRule \u8d44\u6e90\uff0c\u7528\u6237\u53ef\u4ee5\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740\u8303\u56f4\u3002 \u4e00\u4e2a `SwitchLBRule`` \u4f8b\u5b50\u5982\u4e0b\uff1a apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP vip \uff1a\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u7684\u5730\u5740\u3002 namespace \uff1a\u8d1f\u8f7d\u5747\u8861\u5668\u540e\u7aef Pod \u6240\u5728\u7684 Namespace\u3002 sessionAffinity \uff1a\u548c Service \u7684 sessionAffinity \u529f\u80fd\u76f8\u540c\u3002 selector \uff1a\u548c Service \u7684 selector \u529f\u80fd\u76f8\u540c\u3002 ports \uff1a\u548c Service \u7684 port \u529f\u80fd\u76f8\u540c\u3002 \u67e5\u770b\u90e8\u7f72\u7684\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u5668\u89c4\u5219\uff1a # kubectl get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh2 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh2 88m","title":"\u81ea\u5b9a\u4e49\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u89c4\u5219"},{"location":"guide/vpc/#vpc-dns","text":"\u7531\u4e8e\u81ea\u5b9a\u4e49 VPC \u548c\u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0cVPC \u5185 Pod \u65e0\u6cd5\u4f7f\u7528\u9ed8\u8ba4\u7684 coredns \u670d\u52a1\u8fdb\u884c\u57df\u540d\u89e3\u6790\u3002 \u5982\u679c\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 coredns \u89e3\u6790\u96c6\u7fa4\u5185 Service \u57df\u540d\uff0c\u53ef\u4ee5\u901a\u8fc7 Kube-OVN \u63d0\u4f9b\u7684 vpc-dns \u8d44\u6e90\u6765\u5b9e\u73b0\u3002","title":"\u81ea\u5b9a\u4e49 vpc-dns"},{"location":"guide/vpc/#_6","text":"apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }'","title":"\u521b\u5efa\u9644\u52a0\u7f51\u5361"},{"location":"guide/vpc/#ovn-default-provider","text":"\u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn \uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn vpc : ovn-cluster","title":"\u4fee\u6539 ovn-default \u903b\u8f91\u4ea4\u6362\u673a\u7684 provider"},{"location":"guide/vpc/#vpc-dns-configmap","text":"\u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : true nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\uff08\u53ef\u7f3a\u7701\uff09 true \u542f\u7528\u529f\u80fd\uff0c false \u5173\u95ed\u529f\u80fd\u3002\u9ed8\u8ba4 true \u3002 coredns-image \uff1a\uff08\u53ef\u7701\u7565\uff09\uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-template \uff1a\uff08\u53ef\u7701\u7565\uff09\uff1adns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\uff1a\u5f53\u524d\u7248\u672c\u4ed3\u5e93\u91cc\u7684 yamls/coredns-template.yaml \u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\uff08\u53ef\u7f3a\u7701\uff09 \u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\u3002 k8s-service-port \uff1a\uff08\u53ef\u7f3a\u7701\uff09\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\u3002","title":"\u914d\u7f6e vpc-dns \u7684 ConfigMap"},{"location":"guide/vpc/#vpc-dns_1","text":"apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance }","title":"\u90e8\u7f72 vpc-dns \u4f9d\u8d56\u8d44\u6e90"},{"location":"guide/vpc/#vpc-dns_2","text":"kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u8d44\u6e90\u4fe1\u606f\uff1a [ root@hci-dev-mst-1 kubeovn ] # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u6210\u529f\u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72","title":"\u90e8\u7f72 vpc-dns"},{"location":"guide/vpc/#_7","text":"\u4e00\u4e2a vpc \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a vpc \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a vpc \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9650\u5236"},{"location":"guide/webhook/","text":"Webhook \u4f7f\u7528 \u00b6 \u4f7f\u7528 Webhook \u53ef\u4ee5\u5bf9 Kube-OVN \u5185\u7684 CRD \u8d44\u6e90\u8fdb\u884c\u6821\u9a8c\uff0c\u76ee\u524d Webhook \u4e3b\u8981\u5b8c\u6210 \u56fa\u5b9a IP \u5730\u5740\u51b2\u7a81\u68c0\u6d4b\u548c Subnet CIDR \u7684\u51b2\u7a81\u68c0\u6d4b\uff0c\u5e76\u5728\u8fd9\u7c7b\u8d44\u6e90\u521b\u5efa\u51b2\u7a81\u65f6\u63d0\u793a\u9519\u8bef\u3002 \u7531\u4e8e Webhook \u4f1a\u62e6\u622a\u6240\u6709\u7684 Subnet \u548c Pod \u521b\u5efa\u7684\u8bf7\u6c42\uff0c\u56e0\u6b64\u9700\u8981\u5148\u90e8\u7f72 Kube-OVN \u540e\u90e8\u7f72 Webhook \u907f\u514d\u65e0\u6cd5\u521b\u5efa Pod\u3002 Cert-Manager \u5b89\u88c5 \u00b6 Webhook \u90e8\u7f72\u9700\u8981\u76f8\u5173\u8bc1\u4e66\u52a0\u5bc6\uff0c\u6211\u4eec\u4f7f\u7528 cert-manager \u751f\u6210\u76f8\u5173\u8bc1\u4e66\uff0c\u6211\u4eec\u9700\u8981\u5728\u90e8\u7f72 Webhook \u524d\u5148\u90e8\u7f72 cert-manager\u3002 \u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u6765\u90e8\u7f72 cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml \u66f4\u591a cert-manager \u4f7f\u7528\u8bf7\u53c2\u8003 cert-manager \u6587\u6863 \u3002 \u5b89\u88c5 Webhook \u00b6 \u4e0b\u8f7d Webhook \u5bf9\u5e94\u7684 yaml \u8fdb\u884c\u5b89\u88c5: # kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/webhook.yaml deployment.apps/kube-ovn-webhook created service/kube-ovn-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/kube-ovn-webhook created certificate.cert-manager.io/kube-ovn-webhook-serving-cert created issuer.cert-manager.io/kube-ovn-webhook-selfsigned-issuer created \u9a8c\u8bc1 Webhook \u751f\u6548 \u00b6 \u67e5\u770b\u5df2\u8fd0\u884c Pod\uff0c\u5f97\u5230 Pod IP 10.16.0.15 \uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES static-7584848b74-fw9dm 1 /1 Running 0 2d13h 10 .16.0.15 kube-ovn-worker <none> \u7f16\u5199 yaml \u521b\u5efa\u76f8\u540c IP \u7684 Pod\uff1a apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 labels : app : static managedFields : name : staticip-pod namespace : default spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u4f7f\u7528\u4ee5\u4e0a yaml \u521b\u5efa\u9759\u6001\u5730\u5740 Pod \u7684\u65f6\u5019\uff0c\u63d0\u793a IP \u5730\u5740\u51b2\u7a81\uff1a # kubectl apply -f pod-static.yaml Error from server ( annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 ) : error when creating \"pod-static.yaml\" : admission webhook \"pod-ip-validaing.kube-ovn.io\" denied the request: annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Webhook \u4f7f\u7528"},{"location":"guide/webhook/#webhook","text":"\u4f7f\u7528 Webhook \u53ef\u4ee5\u5bf9 Kube-OVN \u5185\u7684 CRD \u8d44\u6e90\u8fdb\u884c\u6821\u9a8c\uff0c\u76ee\u524d Webhook \u4e3b\u8981\u5b8c\u6210 \u56fa\u5b9a IP \u5730\u5740\u51b2\u7a81\u68c0\u6d4b\u548c Subnet CIDR \u7684\u51b2\u7a81\u68c0\u6d4b\uff0c\u5e76\u5728\u8fd9\u7c7b\u8d44\u6e90\u521b\u5efa\u51b2\u7a81\u65f6\u63d0\u793a\u9519\u8bef\u3002 \u7531\u4e8e Webhook \u4f1a\u62e6\u622a\u6240\u6709\u7684 Subnet \u548c Pod \u521b\u5efa\u7684\u8bf7\u6c42\uff0c\u56e0\u6b64\u9700\u8981\u5148\u90e8\u7f72 Kube-OVN \u540e\u90e8\u7f72 Webhook \u907f\u514d\u65e0\u6cd5\u521b\u5efa Pod\u3002","title":"Webhook \u4f7f\u7528"},{"location":"guide/webhook/#cert-manager","text":"Webhook \u90e8\u7f72\u9700\u8981\u76f8\u5173\u8bc1\u4e66\u52a0\u5bc6\uff0c\u6211\u4eec\u4f7f\u7528 cert-manager \u751f\u6210\u76f8\u5173\u8bc1\u4e66\uff0c\u6211\u4eec\u9700\u8981\u5728\u90e8\u7f72 Webhook \u524d\u5148\u90e8\u7f72 cert-manager\u3002 \u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u6765\u90e8\u7f72 cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml \u66f4\u591a cert-manager \u4f7f\u7528\u8bf7\u53c2\u8003 cert-manager \u6587\u6863 \u3002","title":"Cert-Manager \u5b89\u88c5"},{"location":"guide/webhook/#webhook_1","text":"\u4e0b\u8f7d Webhook \u5bf9\u5e94\u7684 yaml \u8fdb\u884c\u5b89\u88c5: # kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/webhook.yaml deployment.apps/kube-ovn-webhook created service/kube-ovn-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/kube-ovn-webhook created certificate.cert-manager.io/kube-ovn-webhook-serving-cert created issuer.cert-manager.io/kube-ovn-webhook-selfsigned-issuer created","title":"\u5b89\u88c5 Webhook"},{"location":"guide/webhook/#webhook_2","text":"\u67e5\u770b\u5df2\u8fd0\u884c Pod\uff0c\u5f97\u5230 Pod IP 10.16.0.15 \uff1a # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES static-7584848b74-fw9dm 1 /1 Running 0 2d13h 10 .16.0.15 kube-ovn-worker <none> \u7f16\u5199 yaml \u521b\u5efa\u76f8\u540c IP \u7684 Pod\uff1a apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 labels : app : static managedFields : name : staticip-pod namespace : default spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest \u4f7f\u7528\u4ee5\u4e0a yaml \u521b\u5efa\u9759\u6001\u5730\u5740 Pod \u7684\u65f6\u5019\uff0c\u63d0\u793a IP \u5730\u5740\u51b2\u7a81\uff1a # kubectl apply -f pod-static.yaml Error from server ( annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 ) : error when creating \"pod-static.yaml\" : admission webhook \"pod-ip-validaing.kube-ovn.io\" denied the request: annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9a8c\u8bc1 Webhook \u751f\u6548"},{"location":"ops/change-default-subnet/","text":"\u4fee\u6539\u5b50\u7f51 CIDR \u00b6 \u5982\u679c\u521b\u5efa\u7684\u5b50\u7f51 CIDR \u51b2\u7a81\u6216\u4e0d\u7b26\u5408\u9884\u671f\uff0c\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u7684\u6b65\u9aa4\u8fdb\u884c\u4fee\u6539\u3002 \u4fee\u6539\u5b50\u7f51 CIDR \u540e\u4e4b\u524d\u521b\u5efa\u7684 Pod \u5c06\u65e0\u6cd5\u6b63\u5e38\u8bbf\u95ee\u7f51\u7edc\u9700\u8981\u8fdb\u884c\u91cd\u5efa\u3002 \u5efa\u8bae\u64cd\u4f5c\u524d\u614e\u91cd\u8003\u8651\u3002\u672c\u6587\u53ea\u9488\u5bf9\u4e1a\u52a1\u5b50\u7f51 CIDR \u66f4\u6539\u8fdb\u884c\u64cd\u4f5c\uff0c\u5982\u9700 \u66f4\u6539 Join \u5b50\u7f51 CIDR \u8bf7\u53c2\u8003 \u66f4\u6539 Join \u5b50\u7f51 CIDR \u3002 \u7f16\u8f91\u5b50\u7f51 \u00b6 \u4f7f\u7528 kubectl edit \u4fee\u6539\u5b50\u7f51 cidrBlock \uff0c gateway \u548c excludeIps \u3002 kubectl edit subnet test-subnet \u91cd\u5efa\u8be5\u5b50\u7f51\u7ed1\u5b9a\u7684 Namespace \u4e0b\u6240\u6709 Pod \u00b6 \u4ee5\u5b50\u7f51\u7ed1\u5b9a test Namespace \u4e3a\u4f8b\uff1a for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n test --ignore-not-found done \u82e5\u53ea\u4f7f\u7528\u4e86\u9ed8\u8ba4\u5b50\u7f51\uff0c\u53ef\u4ee5\u4f7f\u7528\u4e0b\u5217\u547d\u4ee4\u5220\u9664\u6240\u6709\u975e host \u7f51\u7edc\u6a21\u5f0f\u7684 Pod\uff1a for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done \u66f4\u6539\u9ed8\u8ba4\u5b50\u7f51\u914d\u7f6e \u00b6 \u82e5\u4fee\u6539\u7684\u4e3a\u9ed8\u8ba4\u5b50\u7f51\u7684 CIDR \u8fd8\u9700\u8981\u66f4\u6539 kube-ovn-controller Deployment \u7684\u542f\u52a8\u53c2\u6570\uff1a args : - --default-cidr=10.17.0.0/16 - --default-gateway=10.17.0.1 - --default-exclude-ips=10.17.0.1 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4fee\u6539\u5b50\u7f51 CIDR"},{"location":"ops/change-default-subnet/#cidr","text":"\u5982\u679c\u521b\u5efa\u7684\u5b50\u7f51 CIDR \u51b2\u7a81\u6216\u4e0d\u7b26\u5408\u9884\u671f\uff0c\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u7684\u6b65\u9aa4\u8fdb\u884c\u4fee\u6539\u3002 \u4fee\u6539\u5b50\u7f51 CIDR \u540e\u4e4b\u524d\u521b\u5efa\u7684 Pod \u5c06\u65e0\u6cd5\u6b63\u5e38\u8bbf\u95ee\u7f51\u7edc\u9700\u8981\u8fdb\u884c\u91cd\u5efa\u3002 \u5efa\u8bae\u64cd\u4f5c\u524d\u614e\u91cd\u8003\u8651\u3002\u672c\u6587\u53ea\u9488\u5bf9\u4e1a\u52a1\u5b50\u7f51 CIDR \u66f4\u6539\u8fdb\u884c\u64cd\u4f5c\uff0c\u5982\u9700 \u66f4\u6539 Join \u5b50\u7f51 CIDR \u8bf7\u53c2\u8003 \u66f4\u6539 Join \u5b50\u7f51 CIDR \u3002","title":"\u4fee\u6539\u5b50\u7f51 CIDR"},{"location":"ops/change-default-subnet/#_1","text":"\u4f7f\u7528 kubectl edit \u4fee\u6539\u5b50\u7f51 cidrBlock \uff0c gateway \u548c excludeIps \u3002 kubectl edit subnet test-subnet","title":"\u7f16\u8f91\u5b50\u7f51"},{"location":"ops/change-default-subnet/#namespace-pod","text":"\u4ee5\u5b50\u7f51\u7ed1\u5b9a test Namespace \u4e3a\u4f8b\uff1a for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n test --ignore-not-found done \u82e5\u53ea\u4f7f\u7528\u4e86\u9ed8\u8ba4\u5b50\u7f51\uff0c\u53ef\u4ee5\u4f7f\u7528\u4e0b\u5217\u547d\u4ee4\u5220\u9664\u6240\u6709\u975e host \u7f51\u7edc\u6a21\u5f0f\u7684 Pod\uff1a for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done","title":"\u91cd\u5efa\u8be5\u5b50\u7f51\u7ed1\u5b9a\u7684 Namespace \u4e0b\u6240\u6709 Pod"},{"location":"ops/change-default-subnet/#_2","text":"\u82e5\u4fee\u6539\u7684\u4e3a\u9ed8\u8ba4\u5b50\u7f51\u7684 CIDR \u8fd8\u9700\u8981\u66f4\u6539 kube-ovn-controller Deployment \u7684\u542f\u52a8\u53c2\u6570\uff1a args : - --default-cidr=10.17.0.0/16 - --default-gateway=10.17.0.1 - --default-exclude-ips=10.17.0.1 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u66f4\u6539\u9ed8\u8ba4\u5b50\u7f51\u914d\u7f6e"},{"location":"ops/change-join-subnet/","text":"\u4fee\u6539 Join \u5b50\u7f51 CIDR \u00b6 \u82e5\u53d1\u73b0\u521b\u5efa\u7684 Join \u5b50\u7f51 CIDR \u51b2\u7a81\u6216\u4e0d\u7b26\u5408\u9884\u671f\uff0c\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u8fdb\u884c\u4fee\u6539\u3002 \u4fee\u6539 Join \u5b50\u7f51 CIDR \u540e\u4e4b\u524d\u521b\u5efa\u7684 Pod \u5c06\u65e0\u6cd5\u6b63\u5e38\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\uff0c\u9700\u8981\u7b49\u91cd\u5efa\u5b8c\u6210, \u5efa\u8bae\u524d\u64cd\u4f5c\u65f6\u614e\u91cd\u8003\u8651\u3002 \u5220\u9664 Join \u5b50\u7f51 \u00b6 kubectl patch subnet join --type = 'json' -p '[{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\": []}]' kubectl delete subnet join \u6e05\u7406\u76f8\u5173\u5206\u914d\u4fe1\u606f \u00b6 kubectl annotate node ovn.kubernetes.io/allocated = false --all --overwrite \u4fee\u6539 Join \u5b50\u7f51\u76f8\u5173\u4fe1\u606f \u00b6 \u4fee\u6539 kube-ovn-controller \u5185 Join \u5b50\u7f51\u76f8\u5173\u4fe1\u606f\uff1a kubectl edit deployment -n kube-system kube-ovn-controller \u4fee\u6539\u4e0b\u5217\u53c2\u6570\uff1a args : - --node-switch-cidr=100.51.0.0/16 \u91cd\u542f kube-ovn-controller \u91cd\u5efa join \u5b50\u7f51\uff1a kubectl delete pod -n kube-system -lapp = kube-ovn-controller \u67e5\u770b\u65b0\u7684 Join \u5b50\u7f51\u4fe1\u606f\uff1a # kubectl get subnet NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS join ovn ovn-cluster IPv4 100 .51.0.0/16 false false false distributed 2 65531 0 0 [ \"100.51.0.1\" ] ovn-default ovn ovn-cluster IPv4 10 .17.0.0/16 false true true distributed 5 65528 0 0 [ \"10.17.0.1\" ] \u91cd\u65b0\u914d\u7f6e ovn0 \u7f51\u5361\u5730\u5740 \u00b6 \u6bcf\u4e2a\u8282\u70b9\u7684 ovn0 \u7f51\u5361\u4fe1\u606f\u9700\u8981\u91cd\u65b0\u66f4\u65b0\uff0c\u53ef\u901a\u8fc7\u91cd\u542f kube-ovn-cni \u6765\u5b8c\u6210\uff1a kubectl delete pod -n kube-system -l app = kube-ovn-cni \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4fee\u6539 Join \u5b50\u7f51 CIDR"},{"location":"ops/change-join-subnet/#join-cidr","text":"\u82e5\u53d1\u73b0\u521b\u5efa\u7684 Join \u5b50\u7f51 CIDR \u51b2\u7a81\u6216\u4e0d\u7b26\u5408\u9884\u671f\uff0c\u53ef\u4ee5\u901a\u8fc7\u672c\u6587\u6863\u8fdb\u884c\u4fee\u6539\u3002 \u4fee\u6539 Join \u5b50\u7f51 CIDR \u540e\u4e4b\u524d\u521b\u5efa\u7684 Pod \u5c06\u65e0\u6cd5\u6b63\u5e38\u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\uff0c\u9700\u8981\u7b49\u91cd\u5efa\u5b8c\u6210, \u5efa\u8bae\u524d\u64cd\u4f5c\u65f6\u614e\u91cd\u8003\u8651\u3002","title":"\u4fee\u6539 Join \u5b50\u7f51 CIDR"},{"location":"ops/change-join-subnet/#join","text":"kubectl patch subnet join --type = 'json' -p '[{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\": []}]' kubectl delete subnet join","title":"\u5220\u9664 Join \u5b50\u7f51"},{"location":"ops/change-join-subnet/#_1","text":"kubectl annotate node ovn.kubernetes.io/allocated = false --all --overwrite","title":"\u6e05\u7406\u76f8\u5173\u5206\u914d\u4fe1\u606f"},{"location":"ops/change-join-subnet/#join_1","text":"\u4fee\u6539 kube-ovn-controller \u5185 Join \u5b50\u7f51\u76f8\u5173\u4fe1\u606f\uff1a kubectl edit deployment -n kube-system kube-ovn-controller \u4fee\u6539\u4e0b\u5217\u53c2\u6570\uff1a args : - --node-switch-cidr=100.51.0.0/16 \u91cd\u542f kube-ovn-controller \u91cd\u5efa join \u5b50\u7f51\uff1a kubectl delete pod -n kube-system -lapp = kube-ovn-controller \u67e5\u770b\u65b0\u7684 Join \u5b50\u7f51\u4fe1\u606f\uff1a # kubectl get subnet NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS join ovn ovn-cluster IPv4 100 .51.0.0/16 false false false distributed 2 65531 0 0 [ \"100.51.0.1\" ] ovn-default ovn ovn-cluster IPv4 10 .17.0.0/16 false true true distributed 5 65528 0 0 [ \"10.17.0.1\" ]","title":"\u4fee\u6539 Join \u5b50\u7f51\u76f8\u5173\u4fe1\u606f"},{"location":"ops/change-join-subnet/#ovn0","text":"\u6bcf\u4e2a\u8282\u70b9\u7684 ovn0 \u7f51\u5361\u4fe1\u606f\u9700\u8981\u91cd\u65b0\u66f4\u65b0\uff0c\u53ef\u901a\u8fc7\u91cd\u542f kube-ovn-cni \u6765\u5b8c\u6210\uff1a kubectl delete pod -n kube-system -l app = kube-ovn-cni \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u91cd\u65b0\u914d\u7f6e ovn0 \u7f51\u5361\u5730\u5740"},{"location":"ops/change-log-level/","text":"\u8c03\u6574\u65e5\u5fd7\u7b49\u7ea7 \u00b6 \u6253\u5f00 kube-ovn.yaml \uff0c\u5728\u670d\u52a1\u542f\u52a8\u811a\u672c\u7684\u53c2\u6570\u5217\u8868\u4e2d\u8bbe\u7f6e log \u7b49\u7ea7\uff0c\u6bd4\u5982\uff1a vi kube-ovn.yaml # ... - name: kube-ovn-controller image: \"docker.io/kubeovn/kube-ovn:v1.12.0\" imagePullPolicy: IfNotPresent args: - /kube-ovn/start-controller.sh - --v = 3 # ... # log \u7b49\u7ea7\u8d8a\u9ad8\uff0clog \u5c31\u8d8a\u8be6\u7ec6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8c03\u6574\u65e5\u5fd7\u7b49\u7ea7"},{"location":"ops/change-log-level/#_1","text":"\u6253\u5f00 kube-ovn.yaml \uff0c\u5728\u670d\u52a1\u542f\u52a8\u811a\u672c\u7684\u53c2\u6570\u5217\u8868\u4e2d\u8bbe\u7f6e log \u7b49\u7ea7\uff0c\u6bd4\u5982\uff1a vi kube-ovn.yaml # ... - name: kube-ovn-controller image: \"docker.io/kubeovn/kube-ovn:v1.12.0\" imagePullPolicy: IfNotPresent args: - /kube-ovn/start-controller.sh - --v = 3 # ... # log \u7b49\u7ea7\u8d8a\u9ad8\uff0clog \u5c31\u8d8a\u8be6\u7ec6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8c03\u6574\u65e5\u5fd7\u7b49\u7ea7"},{"location":"ops/change-ovn-central-node/","text":"\u66f4\u6362 ovn-central \u8282\u70b9 \u00b6 \u7531\u4e8e ovn-central \u5185\u7684 ovn-nb \u548c ovn-sb \u5206\u522b\u5efa\u7acb\u4e86\u7c7b\u4f3c etcd \u7684 raft \u96c6\u7fa4\uff0c\u56e0\u6b64\u66f4\u6362 ovn-central \u8282\u70b9\u9700\u8981\u989d\u5916\u7684\u64cd\u4f5c\uff0c\u4fdd\u8bc1\u96c6\u7fa4\u72b6\u6001\u7684\u6b63\u786e\u548c\u6570\u636e\u7684\u4e00\u81f4\u3002\u5efa\u8bae\u6bcf\u6b21\u53ea\u5bf9\u4e00\u4e2a\u8282\u70b9\u8fdb\u884c\u4e0a\u4e0b\u7ebf\u5904\u7406\uff0c\u4ee5\u907f\u514d\u96c6\u7fa4\u8fdb\u5165\u4e0d\u53ef\u7528 \u72b6\u6001\uff0c\u5f71\u54cd\u96c6\u7fa4\u6574\u4f53\u7f51\u7edc\u3002 ovn-central \u8282\u70b9\u4e0b\u7ebf \u00b6 \u672c\u6587\u6863\u9488\u5bf9\u5982\u4e0b\u7684\u96c6\u7fa4\u60c5\u51b5\uff0c\u4ee5\u4e0b\u7ebf kube-ovn-control-plane2 \u8282\u70b9\u4e3a\u4f8b\uff0c\u4ecb\u7ecd\u5982\u4f55\u5c06\u5176\u4ece ovn-central \u96c6\u7fa4\u4e2d\u79fb\u9664\u3002 # kubectl -n kube-system get pod -o wide | grep central ovn-central-6bf58cbc97-2cdhg 1 /1 Running 0 21m 172 .18.0.3 kube-ovn-control-plane <none> <none> ovn-central-6bf58cbc97-crmfp 1 /1 Running 0 21m 172 .18.0.5 kube-ovn-control-plane2 <none> <none> ovn-central-6bf58cbc97-lxmpl 1 /1 Running 0 21m 172 .18.0.4 kube-ovn-control-plane3 <none> <none> \u4e0b\u7ebf ovn-nb \u96c6\u7fa4\u5185\u5bf9\u5e94\u8282\u70b9 \u00b6 \u9996\u5148\u67e5\u770b\u8282\u70b9\u5728\u96c6\u7fa4\u5185\u7684 ID\uff0c\u4ee5\u4fbf\u540e\u7eed\u64cd\u4f5c\u3002 # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2135194 ms ago, reason: timeout Last Election won: 2135188 ms ago Election timer: 5000 Log: [ 135 , 135 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-d64b ->d64b <-4984 ->4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 134 d64b ( d64b at tcp: [ 172 .18.0.5 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago status: ok kube-ovn-control-plane2 \u5bf9\u5e94\u8282\u70b9 IP \u4e3a 172.18.0.5 \uff0c\u96c6\u7fa4\u5185\u5bf9\u5e94\u7684 ID \u4e3a d64b \u3002\u63a5\u4e0b\u6765\u4ece ovn-nb \u96c6\u7fa4\u4e2d\u8e22\u51fa\u8be5\u8282\u70b9\uff1a # kubectl ko nb kick d64b started removal \u786e\u8ba4\u8282\u70b9\u8e22\u51fa\u6210\u529f\uff1a # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2297649 ms ago, reason: timeout Last Election won: 2297643 ms ago Election timer: 5000 Log: [ 136 , 136 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-4984 ->4984 Disconnections: 2 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 136 match_index = 135 last msg 1270 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 135 status: ok \u4e0b\u7ebf ovn-sb \u96c6\u7fa4\u5185\u5bf9\u5e94\u8282\u70b9 \u00b6 \u63a5\u4e0b\u6765\u9700\u8981\u64cd\u4f5c ovn-sb \u96c6\u7fa4\uff0c\u9996\u5148\u67e5\u770b\u8282\u70b9\u5728\u96c6\u7fa4\u5185\u7684 ID\uff0c\u4ee5\u4fbf\u540e\u7eed\u64cd\u4f5c\uff1a kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2395317 ms ago, reason: timeout Last Election won: 2395316 ms ago Election timer: 5000 Log: [ 130 , 130 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-e9f7 ->e9f7 <-6e84 ->6e84 Disconnections: 0 Servers: e9f7 ( e9f7 at tcp: [ 172 .18.0.5 ] :6644 ) next_index = 130 match_index = 129 last msg 1006 ms ago 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 130 match_index = 129 last msg 1004 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 129 status: ok kube-ovn-control-plane2 \u5bf9\u5e94\u8282\u70b9 IP \u4e3a 172.18.0.5 \uff0c\u96c6\u7fa4\u5185\u5bf9\u5e94\u7684 ID \u4e3a e9f7 \u3002\u63a5\u4e0b\u6765\u4ece ovn-sb \u96c6\u7fa4\u4e2d\u8e22\u51fa\u8be5\u8282\u70b9\uff1a # kubectl ko sb kick e9f7 started removal \u786e\u8ba4\u8282\u70b9\u8e22\u51fa\u6210\u529f\uff1a # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2481636 ms ago, reason: timeout Last Election won: 2481635 ms ago Election timer: 5000 Log: [ 131 , 131 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-6e84 ->6e84 Disconnections: 2 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 131 match_index = 130 last msg 642 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 130 status: ok \u5220\u9664\u8282\u70b9\u6807\u7b7e\uff0c\u5e76\u7f29\u5bb9 ovn-central \u00b6 \u6ce8\u610f\u9700\u5728 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u7684\u8282\u70b9\u5730\u5740\u4e2d\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u3002 kubectl label node kube-ovn-control-plane2 kube-ovn/role- kubectl scale deployment -n kube-system ovn-central --replicas = 2 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4\" kubectl rollout status deployment/ovn-central -n kube-system \u4fee\u6539\u5176\u4ed6\u7ec4\u4ef6\u8fde\u63a5 ovn-central \u5730\u5740 \u00b6 \u4fee\u6539 ovs-ovn \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u5730\u5740\u3002 # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted \u4fee\u6539 kube-ovn-controller \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u5730\u5740\u3002 # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u6e05\u7406\u8282\u70b9 \u00b6 \u5220\u9664 kube-ovn-control-plane2 \u8282\u70b9\u5185\u7684\u6570\u636e\u5e93\u6587\u4ef6\uff0c\u907f\u514d\u91cd\u590d\u6dfb\u52a0\u8282\u70b9\u65f6\u53d1\u751f\u5f02\u5e38\uff1a rm -rf /etc/origin/ovn \u5982\u9700\u5c06\u8282\u70b9\u4ece\u6574\u4e2a Kubernetes \u96c6\u7fa4\u4e0b\u7ebf\uff0c\u8fd8\u9700\u7ee7\u7eed\u53c2\u8003 \u5220\u9664\u5de5\u4f5c\u8282\u70b9 \u8fdb\u884c\u64cd\u4f5c\u3002 ovn-central \u8282\u70b9\u4e0a\u7ebf \u00b6 \u4e0b\u5217\u6b65\u9aa4\u4f1a\u5c06\u4e00\u4e2a\u65b0\u7684 Kubernetes \u8282\u70b9\u52a0\u5165 ovn-central \u96c6\u7fa4\u3002 \u76ee\u5f55\u68c0\u67e5 \u00b6 \u68c0\u67e5\u65b0\u589e\u8282\u70b9\u7684 /etc/origin/ovn \u76ee\u5f55\u4e2d\u662f\u5426\u5b58\u5728 ovnnb_db.db \u6216 ovnsb_db.db \u6587\u4ef6\uff0c\u82e5\u5b58\u5728\u9700\u63d0\u524d\u5220\u9664\uff1a rm -rf /etc/origin/ovn \u786e\u8ba4\u5f53\u524d ovn-central \u96c6\u7fa4\u72b6\u6001\u6b63\u5e38 \u00b6 \u82e5\u5f53\u524d ovn-central \u96c6\u7fa4\u72b6\u6001\u5df2\u7ecf\u5f02\u5e38\uff0c\u65b0\u589e\u8282\u70b9\u53ef\u80fd\u5bfc\u81f4\u6295\u7968\u9009\u4e3e\u65e0\u6cd5\u8fc7\u534a\u6570\uff0c\u5f71\u54cd\u540e\u7eed\u64cd\u4f5c\u3002 # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 44 Leader: self Vote: self Last Election started 1855739 ms ago, reason: timeout Last Election won: 1855729 ms ago Election timer: 5000 Log: [ 147 , 147 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->4984 <-4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 147 match_index = 146 last msg 367 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 140 match_index = 146 status: ok # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 33 Leader: self Vote: self Last Election started 1868589 ms ago, reason: timeout Last Election won: 1868579 ms ago Election timer: 5000 Log: [ 142 , 142 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->6e84 <-6e84 Disconnections: 0 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 142 match_index = 141 last msg 728 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 134 match_index = 141 status: ok \u7ed9\u8282\u70b9\u589e\u52a0\u6807\u7b7e\u5e76\u6269\u5bb9 \u00b6 \u6ce8\u610f\u9700\u5728 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u7684\u8282\u70b9\u5730\u5740\u4e2d\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\u3002 kubectl label node kube-ovn-control-plane2 kube-ovn/role = master kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4,172.18.0.5\" kubectl rollout status deployment/ovn-central -n kube-system \u4fee\u6539\u5176\u4ed6\u7ec4\u4ef6\u8fde\u63a5 ovn-central \u5730\u5740 \u00b6 \u4fee\u6539 ovs-ovn \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\uff1a # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted \u4fee\u6539 kube-ovn-controller \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\uff1a # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u66f4\u6362 ovn-central \u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-central","text":"\u7531\u4e8e ovn-central \u5185\u7684 ovn-nb \u548c ovn-sb \u5206\u522b\u5efa\u7acb\u4e86\u7c7b\u4f3c etcd \u7684 raft \u96c6\u7fa4\uff0c\u56e0\u6b64\u66f4\u6362 ovn-central \u8282\u70b9\u9700\u8981\u989d\u5916\u7684\u64cd\u4f5c\uff0c\u4fdd\u8bc1\u96c6\u7fa4\u72b6\u6001\u7684\u6b63\u786e\u548c\u6570\u636e\u7684\u4e00\u81f4\u3002\u5efa\u8bae\u6bcf\u6b21\u53ea\u5bf9\u4e00\u4e2a\u8282\u70b9\u8fdb\u884c\u4e0a\u4e0b\u7ebf\u5904\u7406\uff0c\u4ee5\u907f\u514d\u96c6\u7fa4\u8fdb\u5165\u4e0d\u53ef\u7528 \u72b6\u6001\uff0c\u5f71\u54cd\u96c6\u7fa4\u6574\u4f53\u7f51\u7edc\u3002","title":"\u66f4\u6362 ovn-central \u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-central_1","text":"\u672c\u6587\u6863\u9488\u5bf9\u5982\u4e0b\u7684\u96c6\u7fa4\u60c5\u51b5\uff0c\u4ee5\u4e0b\u7ebf kube-ovn-control-plane2 \u8282\u70b9\u4e3a\u4f8b\uff0c\u4ecb\u7ecd\u5982\u4f55\u5c06\u5176\u4ece ovn-central \u96c6\u7fa4\u4e2d\u79fb\u9664\u3002 # kubectl -n kube-system get pod -o wide | grep central ovn-central-6bf58cbc97-2cdhg 1 /1 Running 0 21m 172 .18.0.3 kube-ovn-control-plane <none> <none> ovn-central-6bf58cbc97-crmfp 1 /1 Running 0 21m 172 .18.0.5 kube-ovn-control-plane2 <none> <none> ovn-central-6bf58cbc97-lxmpl 1 /1 Running 0 21m 172 .18.0.4 kube-ovn-control-plane3 <none> <none>","title":"ovn-central \u8282\u70b9\u4e0b\u7ebf"},{"location":"ops/change-ovn-central-node/#ovn-nb","text":"\u9996\u5148\u67e5\u770b\u8282\u70b9\u5728\u96c6\u7fa4\u5185\u7684 ID\uff0c\u4ee5\u4fbf\u540e\u7eed\u64cd\u4f5c\u3002 # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2135194 ms ago, reason: timeout Last Election won: 2135188 ms ago Election timer: 5000 Log: [ 135 , 135 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-d64b ->d64b <-4984 ->4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 134 d64b ( d64b at tcp: [ 172 .18.0.5 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago status: ok kube-ovn-control-plane2 \u5bf9\u5e94\u8282\u70b9 IP \u4e3a 172.18.0.5 \uff0c\u96c6\u7fa4\u5185\u5bf9\u5e94\u7684 ID \u4e3a d64b \u3002\u63a5\u4e0b\u6765\u4ece ovn-nb \u96c6\u7fa4\u4e2d\u8e22\u51fa\u8be5\u8282\u70b9\uff1a # kubectl ko nb kick d64b started removal \u786e\u8ba4\u8282\u70b9\u8e22\u51fa\u6210\u529f\uff1a # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2297649 ms ago, reason: timeout Last Election won: 2297643 ms ago Election timer: 5000 Log: [ 136 , 136 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-4984 ->4984 Disconnections: 2 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 136 match_index = 135 last msg 1270 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 135 status: ok","title":"\u4e0b\u7ebf ovn-nb \u96c6\u7fa4\u5185\u5bf9\u5e94\u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-sb","text":"\u63a5\u4e0b\u6765\u9700\u8981\u64cd\u4f5c ovn-sb \u96c6\u7fa4\uff0c\u9996\u5148\u67e5\u770b\u8282\u70b9\u5728\u96c6\u7fa4\u5185\u7684 ID\uff0c\u4ee5\u4fbf\u540e\u7eed\u64cd\u4f5c\uff1a kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2395317 ms ago, reason: timeout Last Election won: 2395316 ms ago Election timer: 5000 Log: [ 130 , 130 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-e9f7 ->e9f7 <-6e84 ->6e84 Disconnections: 0 Servers: e9f7 ( e9f7 at tcp: [ 172 .18.0.5 ] :6644 ) next_index = 130 match_index = 129 last msg 1006 ms ago 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 130 match_index = 129 last msg 1004 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 129 status: ok kube-ovn-control-plane2 \u5bf9\u5e94\u8282\u70b9 IP \u4e3a 172.18.0.5 \uff0c\u96c6\u7fa4\u5185\u5bf9\u5e94\u7684 ID \u4e3a e9f7 \u3002\u63a5\u4e0b\u6765\u4ece ovn-sb \u96c6\u7fa4\u4e2d\u8e22\u51fa\u8be5\u8282\u70b9\uff1a # kubectl ko sb kick e9f7 started removal \u786e\u8ba4\u8282\u70b9\u8e22\u51fa\u6210\u529f\uff1a # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2481636 ms ago, reason: timeout Last Election won: 2481635 ms ago Election timer: 5000 Log: [ 131 , 131 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-6e84 ->6e84 Disconnections: 2 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 131 match_index = 130 last msg 642 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 130 status: ok","title":"\u4e0b\u7ebf ovn-sb \u96c6\u7fa4\u5185\u5bf9\u5e94\u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-central_2","text":"\u6ce8\u610f\u9700\u5728 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u7684\u8282\u70b9\u5730\u5740\u4e2d\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u3002 kubectl label node kube-ovn-control-plane2 kube-ovn/role- kubectl scale deployment -n kube-system ovn-central --replicas = 2 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4\" kubectl rollout status deployment/ovn-central -n kube-system","title":"\u5220\u9664\u8282\u70b9\u6807\u7b7e\uff0c\u5e76\u7f29\u5bb9 ovn-central"},{"location":"ops/change-ovn-central-node/#ovn-central_3","text":"\u4fee\u6539 ovs-ovn \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u5730\u5740\u3002 # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted \u4fee\u6539 kube-ovn-controller \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u5220\u9664\u4e0b\u7ebf\u8282\u70b9\u5730\u5740\u3002 # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out","title":"\u4fee\u6539\u5176\u4ed6\u7ec4\u4ef6\u8fde\u63a5 ovn-central \u5730\u5740"},{"location":"ops/change-ovn-central-node/#_1","text":"\u5220\u9664 kube-ovn-control-plane2 \u8282\u70b9\u5185\u7684\u6570\u636e\u5e93\u6587\u4ef6\uff0c\u907f\u514d\u91cd\u590d\u6dfb\u52a0\u8282\u70b9\u65f6\u53d1\u751f\u5f02\u5e38\uff1a rm -rf /etc/origin/ovn \u5982\u9700\u5c06\u8282\u70b9\u4ece\u6574\u4e2a Kubernetes \u96c6\u7fa4\u4e0b\u7ebf\uff0c\u8fd8\u9700\u7ee7\u7eed\u53c2\u8003 \u5220\u9664\u5de5\u4f5c\u8282\u70b9 \u8fdb\u884c\u64cd\u4f5c\u3002","title":"\u6e05\u7406\u8282\u70b9"},{"location":"ops/change-ovn-central-node/#ovn-central_4","text":"\u4e0b\u5217\u6b65\u9aa4\u4f1a\u5c06\u4e00\u4e2a\u65b0\u7684 Kubernetes \u8282\u70b9\u52a0\u5165 ovn-central \u96c6\u7fa4\u3002","title":"ovn-central \u8282\u70b9\u4e0a\u7ebf"},{"location":"ops/change-ovn-central-node/#_2","text":"\u68c0\u67e5\u65b0\u589e\u8282\u70b9\u7684 /etc/origin/ovn \u76ee\u5f55\u4e2d\u662f\u5426\u5b58\u5728 ovnnb_db.db \u6216 ovnsb_db.db \u6587\u4ef6\uff0c\u82e5\u5b58\u5728\u9700\u63d0\u524d\u5220\u9664\uff1a rm -rf /etc/origin/ovn","title":"\u76ee\u5f55\u68c0\u67e5"},{"location":"ops/change-ovn-central-node/#ovn-central_5","text":"\u82e5\u5f53\u524d ovn-central \u96c6\u7fa4\u72b6\u6001\u5df2\u7ecf\u5f02\u5e38\uff0c\u65b0\u589e\u8282\u70b9\u53ef\u80fd\u5bfc\u81f4\u6295\u7968\u9009\u4e3e\u65e0\u6cd5\u8fc7\u534a\u6570\uff0c\u5f71\u54cd\u540e\u7eed\u64cd\u4f5c\u3002 # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 44 Leader: self Vote: self Last Election started 1855739 ms ago, reason: timeout Last Election won: 1855729 ms ago Election timer: 5000 Log: [ 147 , 147 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->4984 <-4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 147 match_index = 146 last msg 367 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 140 match_index = 146 status: ok # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 33 Leader: self Vote: self Last Election started 1868589 ms ago, reason: timeout Last Election won: 1868579 ms ago Election timer: 5000 Log: [ 142 , 142 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->6e84 <-6e84 Disconnections: 0 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 142 match_index = 141 last msg 728 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 134 match_index = 141 status: ok","title":"\u786e\u8ba4\u5f53\u524d ovn-central \u96c6\u7fa4\u72b6\u6001\u6b63\u5e38"},{"location":"ops/change-ovn-central-node/#_3","text":"\u6ce8\u610f\u9700\u5728 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u7684\u8282\u70b9\u5730\u5740\u4e2d\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\u3002 kubectl label node kube-ovn-control-plane2 kube-ovn/role = master kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4,172.18.0.5\" kubectl rollout status deployment/ovn-central -n kube-system","title":"\u7ed9\u8282\u70b9\u589e\u52a0\u6807\u7b7e\u5e76\u6269\u5bb9"},{"location":"ops/change-ovn-central-node/#ovn-central_6","text":"\u4fee\u6539 ovs-ovn \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\uff1a # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted \u4fee\u6539 kube-ovn-controller \u5185\u8fde\u63a5\u4fe1\u606f\uff0c\u589e\u52a0\u4e0a\u7ebf\u8282\u70b9\u5730\u5740\uff1a # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4fee\u6539\u5176\u4ed6\u7ec4\u4ef6\u8fde\u63a5 ovn-central \u5730\u5740"},{"location":"ops/delete-worker-node/","text":"\u5220\u9664\u5de5\u4f5c\u8282\u70b9 \u00b6 \u5982\u679c\u53ea\u662f\u7b80\u5355\u4ece Kubernetes \u4e2d\u5220\u9664\u8282\u70b9\uff0c\u7531\u4e8e\u8282\u70b9\u4e0a ovs-ovn \u4e2d\u8fd0\u884c\u7684 ovn-controller \u8fdb\u7a0b\u4ecd\u5728\u8fd0\u884c\u4f1a\u5b9a\u671f\u8fde\u63a5 ovn-central \u6ce8\u518c\u76f8\u5173\u7f51\u7edc\u4fe1\u606f\uff0c \u4f1a\u5bfc\u81f4\u989d\u5916\u8d44\u6e90\u6d6a\u8d39\u5e76\u6709\u6f5c\u5728\u7684\u89c4\u5219\u51b2\u7a81\u98ce\u9669\u3002 \u56e0\u6b64\u5728\u4ece Kubernetes \u5185\u5220\u9664\u8282\u70b9\u65f6\uff0c\u8bf7\u6309\u7167\u4e0b\u9762\u7684\u6b65\u9aa4\u6765\u4fdd\u8bc1\u7f51\u7edc\u4fe1\u606f\u53ef\u4ee5\u6b63\u5e38\u88ab\u6e05\u7406\u3002 \u8be5\u6587\u6863\u4ecb\u7ecd\u5220\u9664\u5de5\u4f5c\u8282\u70b9\u7684\u6b65\u9aa4\uff0c\u5982\u9700\u66f4\u6362 ovn-central \u6240\u5728\u8282\u70b9\uff0c\u8bf7\u53c2\u8003 \u66f4\u6362 ovn-central \u8282\u70b9 \u3002 \u9a71\u9010\u8282\u70b9\u4e0a\u6240\u6709\u5bb9\u5668 \u00b6 # kubectl drain kube-ovn-worker --ignore-daemonsets --force node/kube-ovn-worker cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-ovn-cni-zt74b, kube-system/kube-ovn-pinger-5rxfs, kube-system/kube-proxy-jpmnm, kube-system/ovs-ovn-v2kll evicting pod kube-system/coredns-64897985d-qsgpt evicting pod local-path-storage/local-path-provisioner-5ddd94ff66-llss6 evicting pod kube-system/kube-ovn-controller-8459db5ff4-94lxb pod/kube-ovn-controller-8459db5ff4-94lxb evicted pod/coredns-64897985d-qsgpt evicted pod/local-path-provisioner-5ddd94ff66-llss6 evicted node/kube-ovn-worker drained \u505c\u6b62 kubelet \u548c docker \u00b6 \u8be5\u6b65\u9aa4\u4f1a\u505c\u6b62 ovs-ovn \u5bb9\u5668\uff0c\u4ee5\u907f\u514d\u5411 ovn-central \u8fdb\u884c\u4fe1\u606f\u6ce8\u518c\uff0c\u767b\u5f55\u5230\u5bf9\u5e94\u8282\u70b9\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a systemctl stop kubelet systemctl stop docker \u5982\u679c\u4f7f\u7528\u7684 CRI \u4e3a containerd\uff0c\u9700\u8981\u6267\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u6765\u505c\u6b62 ovs-ovn \u5bb9\u5668\uff1a crictl rm -f $( crictl ps | grep openvswitch | awk '{print $1}' ) \u6e05\u7406 Node \u4e0a\u7684\u6b8b\u7559\u6570\u636e \u00b6 rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn \u4f7f\u7528 kubectl \u5220\u9664\u8282\u70b9 \u00b6 kubectl delete no kube-ovn-01 \u68c0\u67e5\u5bf9\u5e94\u8282\u70b9\u662f\u5426\u4ece ovn-sb \u4e2d\u5220\u9664 \u00b6 \u4e0b\u9762\u7684\u793a\u4f8b\u4e3a kube-ovn-worker \u4f9d\u7136\u672a\u88ab\u5220\u9664\uff1a # kubectl ko sbctl show Chassis \"b0564934-5a0d-4804-a4c0-476c93596a17\" hostname: kube-ovn-worker Encap geneve ip: \"172.18.0.2\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-5rxfs.kube-system Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u82e5\u8282\u70b9\u5bf9\u5e94\u7684 chassis \u4f9d\u7136\u5b58\u5728\uff0c\u624b\u52a8\u8fdb\u884c\u5220\u9664 \u00b6 uuid \u4e3a\u4e4b\u524d\u547d\u4ee4\u6240\u67e5\u51fa\u7684 Chassis \u5bf9\u5e94 id\uff1a # kubectl ko sbctl chassis-del b0564934-5a0d-4804-a4c0-476c93596a17 # kubectl ko sbctl show Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5220\u9664\u5de5\u4f5c\u8282\u70b9"},{"location":"ops/delete-worker-node/#_1","text":"\u5982\u679c\u53ea\u662f\u7b80\u5355\u4ece Kubernetes \u4e2d\u5220\u9664\u8282\u70b9\uff0c\u7531\u4e8e\u8282\u70b9\u4e0a ovs-ovn \u4e2d\u8fd0\u884c\u7684 ovn-controller \u8fdb\u7a0b\u4ecd\u5728\u8fd0\u884c\u4f1a\u5b9a\u671f\u8fde\u63a5 ovn-central \u6ce8\u518c\u76f8\u5173\u7f51\u7edc\u4fe1\u606f\uff0c \u4f1a\u5bfc\u81f4\u989d\u5916\u8d44\u6e90\u6d6a\u8d39\u5e76\u6709\u6f5c\u5728\u7684\u89c4\u5219\u51b2\u7a81\u98ce\u9669\u3002 \u56e0\u6b64\u5728\u4ece Kubernetes \u5185\u5220\u9664\u8282\u70b9\u65f6\uff0c\u8bf7\u6309\u7167\u4e0b\u9762\u7684\u6b65\u9aa4\u6765\u4fdd\u8bc1\u7f51\u7edc\u4fe1\u606f\u53ef\u4ee5\u6b63\u5e38\u88ab\u6e05\u7406\u3002 \u8be5\u6587\u6863\u4ecb\u7ecd\u5220\u9664\u5de5\u4f5c\u8282\u70b9\u7684\u6b65\u9aa4\uff0c\u5982\u9700\u66f4\u6362 ovn-central \u6240\u5728\u8282\u70b9\uff0c\u8bf7\u53c2\u8003 \u66f4\u6362 ovn-central \u8282\u70b9 \u3002","title":"\u5220\u9664\u5de5\u4f5c\u8282\u70b9"},{"location":"ops/delete-worker-node/#_2","text":"# kubectl drain kube-ovn-worker --ignore-daemonsets --force node/kube-ovn-worker cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-ovn-cni-zt74b, kube-system/kube-ovn-pinger-5rxfs, kube-system/kube-proxy-jpmnm, kube-system/ovs-ovn-v2kll evicting pod kube-system/coredns-64897985d-qsgpt evicting pod local-path-storage/local-path-provisioner-5ddd94ff66-llss6 evicting pod kube-system/kube-ovn-controller-8459db5ff4-94lxb pod/kube-ovn-controller-8459db5ff4-94lxb evicted pod/coredns-64897985d-qsgpt evicted pod/local-path-provisioner-5ddd94ff66-llss6 evicted node/kube-ovn-worker drained","title":"\u9a71\u9010\u8282\u70b9\u4e0a\u6240\u6709\u5bb9\u5668"},{"location":"ops/delete-worker-node/#kubelet-docker","text":"\u8be5\u6b65\u9aa4\u4f1a\u505c\u6b62 ovs-ovn \u5bb9\u5668\uff0c\u4ee5\u907f\u514d\u5411 ovn-central \u8fdb\u884c\u4fe1\u606f\u6ce8\u518c\uff0c\u767b\u5f55\u5230\u5bf9\u5e94\u8282\u70b9\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a systemctl stop kubelet systemctl stop docker \u5982\u679c\u4f7f\u7528\u7684 CRI \u4e3a containerd\uff0c\u9700\u8981\u6267\u884c\u4e0b\u9762\u7684\u547d\u4ee4\u6765\u505c\u6b62 ovs-ovn \u5bb9\u5668\uff1a crictl rm -f $( crictl ps | grep openvswitch | awk '{print $1}' )","title":"\u505c\u6b62 kubelet \u548c docker"},{"location":"ops/delete-worker-node/#node","text":"rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn","title":"\u6e05\u7406 Node \u4e0a\u7684\u6b8b\u7559\u6570\u636e"},{"location":"ops/delete-worker-node/#kubectl","text":"kubectl delete no kube-ovn-01","title":"\u4f7f\u7528 kubectl \u5220\u9664\u8282\u70b9"},{"location":"ops/delete-worker-node/#ovn-sb","text":"\u4e0b\u9762\u7684\u793a\u4f8b\u4e3a kube-ovn-worker \u4f9d\u7136\u672a\u88ab\u5220\u9664\uff1a # kubectl ko sbctl show Chassis \"b0564934-5a0d-4804-a4c0-476c93596a17\" hostname: kube-ovn-worker Encap geneve ip: \"172.18.0.2\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-5rxfs.kube-system Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system","title":"\u68c0\u67e5\u5bf9\u5e94\u8282\u70b9\u662f\u5426\u4ece ovn-sb \u4e2d\u5220\u9664"},{"location":"ops/delete-worker-node/#chassis","text":"uuid \u4e3a\u4e4b\u524d\u547d\u4ee4\u6240\u67e5\u51fa\u7684 Chassis \u5bf9\u5e94 id\uff1a # kubectl ko sbctl chassis-del b0564934-5a0d-4804-a4c0-476c93596a17 # kubectl ko sbctl show Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u82e5\u8282\u70b9\u5bf9\u5e94\u7684 chassis \u4f9d\u7136\u5b58\u5728\uff0c\u624b\u52a8\u8fdb\u884c\u5220\u9664"},{"location":"ops/faq/","text":"\u5176\u4ed6\u5e38\u89c1\u95ee\u9898 \u00b6 \u9e92\u9e9f ARM \u7cfb\u7edf\u8de8\u4e3b\u673a\u5bb9\u5668\u8bbf\u95ee\u95f4\u6b47\u5931\u8d25 \u00b6 \u73b0\u8c61 \u00b6 \u9e92\u9e9f ARM \u7cfb\u7edf\u548c\u90e8\u5206\u56fd\u4ea7\u5316\u7f51\u5361 offload \u914d\u5408\u5b58\u5728\u95ee\u9898\uff0c\u4f1a\u5bfc\u81f4\u5bb9\u5668\u7f51\u7edc\u95f4\u6b47\u6545\u969c\u3002 \u4f7f\u7528 netstat \u786e\u8ba4\u95ee\u9898\uff1a # netstat -us IcmpMsg: InType0: 22 InType3: 24 InType8: 117852 OutType0: 117852 OutType3: 29 OutType8: 22 Udp: 3040636 packets received 0 packets to unknown port received. 4 packet receive errors 602 packets sent 0 receive buffer errors 0 send buffer errors InCsumErrors: 4 UdpLite: IpExt: InBcastPkts: 10244 InOctets: 4446320361 OutOctets: 1496815600 InBcastOctets: 3095950 InNoECTPkts: 7683903 \u82e5\u5b58\u5728 InCsumErrors \uff0c\u4e14\u968f\u7740\u8bbf\u95ee\u5931\u8d25\u589e\u52a0\uff0c\u53ef\u786e\u8ba4\u662f\u8be5\u95ee\u9898\u3002 \u89e3\u51b3\u65b9\u6cd5 \u00b6 \u6839\u672c\u89e3\u51b3\u9700\u8981\u548c\u9e92\u9e9f\u4ee5\u53ca\u5bf9\u5e94\u7f51\u5361\u5382\u5546\u6c9f\u901a\uff0c\u66f4\u65b0\u7cfb\u7edf\u548c\u9a71\u52a8\u3002\u4e34\u65f6\u89e3\u51b3\u53ef\u5148\u5173\u95ed\u7269\u7406 \u7f51\u5361\u7684 tx offload \u4f46\u662f\u4f1a\u5bfc\u81f4 tcp \u6027\u80fd\u6709\u8f83\u660e\u663e\u4e0b\u964d\u3002 ethtool -K eth0 tx off \u7ecf\u793e\u533a\u53cd\u9988\u4f7f\u7528 4.19.90-25.16.v2101 \u5185\u6838\u540e\u53ef\u4ee5\u89e3\u51b3\u8be5\u95ee\u9898\u3002 Pod \u8bbf\u95ee Service \u4e0d\u901a \u00b6 \u73b0\u8c61 \u00b6 Pod \u5185\u65e0\u6cd5\u8bbf\u95ee Service \u5bf9\u5e94\u7684\u670d\u52a1\uff0c dmesg \u663e\u793a\u5f02\u5e38\uff1a netlink: Unknown conntrack attr ( type = 6 , max = 5 ) openvswitch: netlink: Flow actions may not be safe on all matching packets. \u8be5\u65e5\u5fd7\u8bf4\u660e\u5185\u6838\u5185 OVS \u7248\u672c\u8fc7\u4f4e\u4e0d\u652f\u6301\u5bf9\u5e94 NAT \u64cd\u4f5c\u3002 \u89e3\u51b3\u65b9\u6cd5 \u00b6 \u5347\u7ea7\u5185\u6838\u6a21\u5757\u6216\u624b\u52a8\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u3002 \u82e5\u53ea\u4f7f\u7528 Overlay \u7f51\u7edc\u53ef\u4ee5\u66f4\u6539 kube-ovn-controller \u542f\u52a8\u53c2\u6570\u8bbe\u7f6e --enable-lb=false \u5173\u95ed OVN LB \u4f7f\u7528 kube-proxy \u8fdb\u884c Service \u8f6c\u53d1\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5176\u4ed6\u5e38\u89c1\u95ee\u9898"},{"location":"ops/faq/#_1","text":"","title":"\u5176\u4ed6\u5e38\u89c1\u95ee\u9898"},{"location":"ops/faq/#arm","text":"","title":"\u9e92\u9e9f ARM \u7cfb\u7edf\u8de8\u4e3b\u673a\u5bb9\u5668\u8bbf\u95ee\u95f4\u6b47\u5931\u8d25"},{"location":"ops/faq/#_2","text":"\u9e92\u9e9f ARM \u7cfb\u7edf\u548c\u90e8\u5206\u56fd\u4ea7\u5316\u7f51\u5361 offload \u914d\u5408\u5b58\u5728\u95ee\u9898\uff0c\u4f1a\u5bfc\u81f4\u5bb9\u5668\u7f51\u7edc\u95f4\u6b47\u6545\u969c\u3002 \u4f7f\u7528 netstat \u786e\u8ba4\u95ee\u9898\uff1a # netstat -us IcmpMsg: InType0: 22 InType3: 24 InType8: 117852 OutType0: 117852 OutType3: 29 OutType8: 22 Udp: 3040636 packets received 0 packets to unknown port received. 4 packet receive errors 602 packets sent 0 receive buffer errors 0 send buffer errors InCsumErrors: 4 UdpLite: IpExt: InBcastPkts: 10244 InOctets: 4446320361 OutOctets: 1496815600 InBcastOctets: 3095950 InNoECTPkts: 7683903 \u82e5\u5b58\u5728 InCsumErrors \uff0c\u4e14\u968f\u7740\u8bbf\u95ee\u5931\u8d25\u589e\u52a0\uff0c\u53ef\u786e\u8ba4\u662f\u8be5\u95ee\u9898\u3002","title":"\u73b0\u8c61"},{"location":"ops/faq/#_3","text":"\u6839\u672c\u89e3\u51b3\u9700\u8981\u548c\u9e92\u9e9f\u4ee5\u53ca\u5bf9\u5e94\u7f51\u5361\u5382\u5546\u6c9f\u901a\uff0c\u66f4\u65b0\u7cfb\u7edf\u548c\u9a71\u52a8\u3002\u4e34\u65f6\u89e3\u51b3\u53ef\u5148\u5173\u95ed\u7269\u7406 \u7f51\u5361\u7684 tx offload \u4f46\u662f\u4f1a\u5bfc\u81f4 tcp \u6027\u80fd\u6709\u8f83\u660e\u663e\u4e0b\u964d\u3002 ethtool -K eth0 tx off \u7ecf\u793e\u533a\u53cd\u9988\u4f7f\u7528 4.19.90-25.16.v2101 \u5185\u6838\u540e\u53ef\u4ee5\u89e3\u51b3\u8be5\u95ee\u9898\u3002","title":"\u89e3\u51b3\u65b9\u6cd5"},{"location":"ops/faq/#pod-service","text":"","title":"Pod \u8bbf\u95ee Service \u4e0d\u901a"},{"location":"ops/faq/#_4","text":"Pod \u5185\u65e0\u6cd5\u8bbf\u95ee Service \u5bf9\u5e94\u7684\u670d\u52a1\uff0c dmesg \u663e\u793a\u5f02\u5e38\uff1a netlink: Unknown conntrack attr ( type = 6 , max = 5 ) openvswitch: netlink: Flow actions may not be safe on all matching packets. \u8be5\u65e5\u5fd7\u8bf4\u660e\u5185\u6838\u5185 OVS \u7248\u672c\u8fc7\u4f4e\u4e0d\u652f\u6301\u5bf9\u5e94 NAT \u64cd\u4f5c\u3002","title":"\u73b0\u8c61"},{"location":"ops/faq/#_5","text":"\u5347\u7ea7\u5185\u6838\u6a21\u5757\u6216\u624b\u52a8\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\u3002 \u82e5\u53ea\u4f7f\u7528 Overlay \u7f51\u7edc\u53ef\u4ee5\u66f4\u6539 kube-ovn-controller \u542f\u52a8\u53c2\u6570\u8bbe\u7f6e --enable-lb=false \u5173\u95ed OVN LB \u4f7f\u7528 kube-proxy \u8fdb\u884c Service \u8f6c\u53d1\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u89e3\u51b3\u65b9\u6cd5"},{"location":"ops/from-calico/","text":"\u5378\u8f7d Calico \u5b89\u88c5 Kube-OVN \u00b6 \u82e5 Kubernetes \u96c6\u7fa4\u5df2\u5b89\u88c5 Calico \u9700\u8981\u53d8\u66f4\u4e3a Kube-OVN \u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\u3002 \u672c\u6587\u4ee5 Calico v3.24.1 \u4e3a\u4f8b\uff0c\u5176\u5b83 Calico \u7248\u672c\u9700\u8981\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u8fdb\u884c\u8c03\u6574\u3002 \u51c6\u5907\u5de5\u4f5c \u00b6 \u4e3a\u4e86\u4fdd\u8bc1\u5207\u6362 CNI \u8fc7\u7a0b\u4e2d\u96c6\u7fa4\u7f51\u7edc\u4fdd\u6301\u7545\u901a\uff0cCalico ippool \u9700\u8981\u5f00\u542f nat outgoing\uff0c \u6216 \u5728\u6240\u6709\u8282\u70b9\u4e0a\u5173\u95ed rp_filter\uff1a sysctl net.ipv4.conf.all.rp_filter = 0 sysctl net.ipv4.conf.default.rp_filter = 0 # IPIP \u6a21\u5f0f sysctl net.ipv4.conf.tunl0.rp_filter = 0 # VXLAN \u6a21\u5f0f sysctl net.ipv4.conf.vxlan/calico.rp_filter = 0 # \u8def\u7531\u6a21\u5f0f\uff0ceth0 \u9700\u8981\u4fee\u6539\u4e3a\u5b9e\u9645\u4f7f\u7528\u7684\u7f51\u5361 sysctl net.ipv4.conf.eth0.rp_filter = 0 \u90e8\u7f72 Kube-OVN \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c \u00b6 wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u5b89\u88c5\u811a\u672c \u00b6 \u5c06\u5b89\u88c5\u811a\u672c\u4e2d\u91cd\u5efa Pod \u7684\u90e8\u5206\u5220\u9664\uff1a echo \"[Step 4/6] Delete pod that not in host network mode\" for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done \u6309\u9700\u4fee\u6539\u4ee5\u4e0b\u914d\u7f6e\uff1a REGISTRY = \"kubeovn\" # \u955c\u50cf\u4ed3\u5e93\u5730\u5740 VERSION = \"v1.12.0\" # \u955c\u50cf\u7248\u672c/Tag POD_CIDR = \"10.16.0.0/16\" # \u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e0d\u8981\u548c SVC/NODE/JOIN CIDR \u91cd\u53e0 SVC_CIDR = \"10.96.0.0/12\" # \u9700\u8981\u548c apiserver \u7684 service-cluster-ip-range \u4fdd\u6301\u4e00\u81f4 JOIN_CIDR = \"100.64.0.0/16\" # Pod \u548c\u4e3b\u673a\u901a\u4fe1\u7f51\u7edc CIDR\uff0c\u4e0d\u8981\u548c SVC/NODE/POD CIDR \u91cd\u53e0 LABEL = \"node-role.kubernetes.io/master\" # \u90e8\u7f72 OVN DB \u8282\u70b9\u7684\u6807\u7b7e IFACE = \"\" # \u5bb9\u5668\u7f51\u7edc\u6240\u4f7f\u7528\u7684\u7684\u5bbf\u4e3b\u673a\u7f51\u5361\u540d\uff0c\u5982\u679c\u4e3a\u7a7a\u5219\u4f7f\u7528 Kubernetes \u4e2d\u7684 Node IP \u6240\u5728\u7f51\u5361 TUNNEL_TYPE = \"geneve\" # \u96a7\u9053\u5c01\u88c5\u534f\u8bae\uff0c\u53ef\u9009 geneve, vxlan \u6216 stt\uff0cstt \u9700\u8981\u5355\u72ec\u7f16\u8bd1 ovs \u5185\u6838\u6a21\u5757 \u6ce8\u610f \uff1aPOD_CIDR \u53ca JOIN_CIDR \u4e0d\u53ef\u4e0e Calico ippool \u7684 CIDR \u51b2\u7a81\uff0c\u4e14 POD_CIDR \u9700\u8981\u5305\u542b\u8db3\u591f\u591a\u7684 IP \u6765\u5bb9\u7eb3\u96c6\u7fa4\u4e2d\u5df2\u6709\u7684 Pod\u3002 \u6267\u884c\u5b89\u88c5\u811a\u672c \u00b6 bash install.sh \u9010\u4e2a\u8282\u70b9\u8fc1\u79fb \u00b6 \u6309\u7167\u4ee5\u4e0b\u65b9\u6cd5\u4e3a\u6bcf\u4e2a\u8282\u70b9\u9010\u4e2a\u8fdb\u884c\u8fc1\u79fb\u3002 \u6ce8\u610f \uff1a\u547d\u4ee4\u4e2d\u7684 \\<NODE> \u9700\u8981\u66ff\u6362\u4e3a\u8282\u70b9\u540d\u79f0\u3002 \u9a71\u9010\u8282\u70b9 \u00b6 kubectl drain --ignore-daemonsets <NODE> \u82e5\u6b64\u547d\u4ee4\u4e00\u76f4\u7b49\u5f85 Pod \u88ab\u9a71\u9010\uff0c\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\u5f3a\u5236\u5220\u9664\u88ab\u9a71\u9010\u7684 Pod\uff1a kubectl get pod -A --field-selector = spec.nodeName = <NODE> --no-headers | \\ awk '$4==\"Terminating\" {print $1\" \"$2}' | \\ while read s ; do kubectl delete pod --force -n $s ; done \u91cd\u542f\u8282\u70b9 \u00b6 \u5728\u8282\u70b9\u4e2d\u6267\u884c\uff1a shutdown -r 0 \u6062\u590d\u8282\u70b9 \u00b6 kubectl uncordon <NODE> \u5378\u8f7d Calico \u00b6 \u5220\u9664 k8s \u8d44\u6e90 \u00b6 kubectl -n kube-system delete deploy calico-kube-controllers kubectl -n kube-system delete ds calico-node kubectl -n kube-system delete cm calico-config # \u5220\u9664 CRD \u53ca\u76f8\u5173\u8d44\u6e90 kubectl get crd -o jsonpath = '{range .items[*]}{.metadata.name}{\"\\n\"}{end}' | while read crd ; do if ! echo $crd | grep '.crd.projectcalico.org$' >/dev/null ; then continue fi for name in $( kubectl get $crd -o jsonpath = '{.items[*].metadata.name}' ) ; do kubectl delete $crd $name done kubectl delete crd $crd done # \u5176\u5b83\u8d44\u6e90 kubectl delete --ignore-not-found clusterrolebinding calico-node calico-kube-controllers kubectl delete --ignore-not-found clusterrole calico-node calico-kube-controllers kubectl delete --ignore-not-found sa -n kube-system calico-kube-controllers calico-node kubectl delete --ignore-not-found pdb -n kube-system calico-kube-controllers \u6e05\u7406\u8282\u70b9\u6587\u4ef6 \u00b6 \u5728\u6bcf\u4e2a\u8282\u70b9\u4e2d\u6267\u884c\uff1a rm -f /etc/cni/net.d/10-calico.conflist /etc/cni/net.d/calico-kubeconfig rm -f /opt/cni/bin/calico /opt/cni/bin/calico-ipam \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5378\u8f7d Calico \u5b89\u88c5 Kube-OVN"},{"location":"ops/from-calico/#calico-kube-ovn","text":"\u82e5 Kubernetes \u96c6\u7fa4\u5df2\u5b89\u88c5 Calico \u9700\u8981\u53d8\u66f4\u4e3a Kube-OVN \u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\u3002 \u672c\u6587\u4ee5 Calico v3.24.1 \u4e3a\u4f8b\uff0c\u5176\u5b83 Calico \u7248\u672c\u9700\u8981\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u8fdb\u884c\u8c03\u6574\u3002","title":"\u5378\u8f7d Calico \u5b89\u88c5 Kube-OVN"},{"location":"ops/from-calico/#_1","text":"\u4e3a\u4e86\u4fdd\u8bc1\u5207\u6362 CNI \u8fc7\u7a0b\u4e2d\u96c6\u7fa4\u7f51\u7edc\u4fdd\u6301\u7545\u901a\uff0cCalico ippool \u9700\u8981\u5f00\u542f nat outgoing\uff0c \u6216 \u5728\u6240\u6709\u8282\u70b9\u4e0a\u5173\u95ed rp_filter\uff1a sysctl net.ipv4.conf.all.rp_filter = 0 sysctl net.ipv4.conf.default.rp_filter = 0 # IPIP \u6a21\u5f0f sysctl net.ipv4.conf.tunl0.rp_filter = 0 # VXLAN \u6a21\u5f0f sysctl net.ipv4.conf.vxlan/calico.rp_filter = 0 # \u8def\u7531\u6a21\u5f0f\uff0ceth0 \u9700\u8981\u4fee\u6539\u4e3a\u5b9e\u9645\u4f7f\u7528\u7684\u7f51\u5361 sysctl net.ipv4.conf.eth0.rp_filter = 0","title":"\u51c6\u5907\u5de5\u4f5c"},{"location":"ops/from-calico/#kube-ovn","text":"","title":"\u90e8\u7f72 Kube-OVN"},{"location":"ops/from-calico/#_2","text":"wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c"},{"location":"ops/from-calico/#_3","text":"\u5c06\u5b89\u88c5\u811a\u672c\u4e2d\u91cd\u5efa Pod \u7684\u90e8\u5206\u5220\u9664\uff1a echo \"[Step 4/6] Delete pod that not in host network mode\" for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done \u6309\u9700\u4fee\u6539\u4ee5\u4e0b\u914d\u7f6e\uff1a REGISTRY = \"kubeovn\" # \u955c\u50cf\u4ed3\u5e93\u5730\u5740 VERSION = \"v1.12.0\" # \u955c\u50cf\u7248\u672c/Tag POD_CIDR = \"10.16.0.0/16\" # \u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e0d\u8981\u548c SVC/NODE/JOIN CIDR \u91cd\u53e0 SVC_CIDR = \"10.96.0.0/12\" # \u9700\u8981\u548c apiserver \u7684 service-cluster-ip-range \u4fdd\u6301\u4e00\u81f4 JOIN_CIDR = \"100.64.0.0/16\" # Pod \u548c\u4e3b\u673a\u901a\u4fe1\u7f51\u7edc CIDR\uff0c\u4e0d\u8981\u548c SVC/NODE/POD CIDR \u91cd\u53e0 LABEL = \"node-role.kubernetes.io/master\" # \u90e8\u7f72 OVN DB \u8282\u70b9\u7684\u6807\u7b7e IFACE = \"\" # \u5bb9\u5668\u7f51\u7edc\u6240\u4f7f\u7528\u7684\u7684\u5bbf\u4e3b\u673a\u7f51\u5361\u540d\uff0c\u5982\u679c\u4e3a\u7a7a\u5219\u4f7f\u7528 Kubernetes \u4e2d\u7684 Node IP \u6240\u5728\u7f51\u5361 TUNNEL_TYPE = \"geneve\" # \u96a7\u9053\u5c01\u88c5\u534f\u8bae\uff0c\u53ef\u9009 geneve, vxlan \u6216 stt\uff0cstt \u9700\u8981\u5355\u72ec\u7f16\u8bd1 ovs \u5185\u6838\u6a21\u5757 \u6ce8\u610f \uff1aPOD_CIDR \u53ca JOIN_CIDR \u4e0d\u53ef\u4e0e Calico ippool \u7684 CIDR \u51b2\u7a81\uff0c\u4e14 POD_CIDR \u9700\u8981\u5305\u542b\u8db3\u591f\u591a\u7684 IP \u6765\u5bb9\u7eb3\u96c6\u7fa4\u4e2d\u5df2\u6709\u7684 Pod\u3002","title":"\u4fee\u6539\u5b89\u88c5\u811a\u672c"},{"location":"ops/from-calico/#_4","text":"bash install.sh","title":"\u6267\u884c\u5b89\u88c5\u811a\u672c"},{"location":"ops/from-calico/#_5","text":"\u6309\u7167\u4ee5\u4e0b\u65b9\u6cd5\u4e3a\u6bcf\u4e2a\u8282\u70b9\u9010\u4e2a\u8fdb\u884c\u8fc1\u79fb\u3002 \u6ce8\u610f \uff1a\u547d\u4ee4\u4e2d\u7684 \\<NODE> \u9700\u8981\u66ff\u6362\u4e3a\u8282\u70b9\u540d\u79f0\u3002","title":"\u9010\u4e2a\u8282\u70b9\u8fc1\u79fb"},{"location":"ops/from-calico/#_6","text":"kubectl drain --ignore-daemonsets <NODE> \u82e5\u6b64\u547d\u4ee4\u4e00\u76f4\u7b49\u5f85 Pod \u88ab\u9a71\u9010\uff0c\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\u5f3a\u5236\u5220\u9664\u88ab\u9a71\u9010\u7684 Pod\uff1a kubectl get pod -A --field-selector = spec.nodeName = <NODE> --no-headers | \\ awk '$4==\"Terminating\" {print $1\" \"$2}' | \\ while read s ; do kubectl delete pod --force -n $s ; done","title":"\u9a71\u9010\u8282\u70b9"},{"location":"ops/from-calico/#_7","text":"\u5728\u8282\u70b9\u4e2d\u6267\u884c\uff1a shutdown -r 0","title":"\u91cd\u542f\u8282\u70b9"},{"location":"ops/from-calico/#_8","text":"kubectl uncordon <NODE>","title":"\u6062\u590d\u8282\u70b9"},{"location":"ops/from-calico/#calico","text":"","title":"\u5378\u8f7d Calico"},{"location":"ops/from-calico/#k8s","text":"kubectl -n kube-system delete deploy calico-kube-controllers kubectl -n kube-system delete ds calico-node kubectl -n kube-system delete cm calico-config # \u5220\u9664 CRD \u53ca\u76f8\u5173\u8d44\u6e90 kubectl get crd -o jsonpath = '{range .items[*]}{.metadata.name}{\"\\n\"}{end}' | while read crd ; do if ! echo $crd | grep '.crd.projectcalico.org$' >/dev/null ; then continue fi for name in $( kubectl get $crd -o jsonpath = '{.items[*].metadata.name}' ) ; do kubectl delete $crd $name done kubectl delete crd $crd done # \u5176\u5b83\u8d44\u6e90 kubectl delete --ignore-not-found clusterrolebinding calico-node calico-kube-controllers kubectl delete --ignore-not-found clusterrole calico-node calico-kube-controllers kubectl delete --ignore-not-found sa -n kube-system calico-kube-controllers calico-node kubectl delete --ignore-not-found pdb -n kube-system calico-kube-controllers","title":"\u5220\u9664 k8s \u8d44\u6e90"},{"location":"ops/from-calico/#_9","text":"\u5728\u6bcf\u4e2a\u8282\u70b9\u4e2d\u6267\u884c\uff1a rm -f /etc/cni/net.d/10-calico.conflist /etc/cni/net.d/calico-kubeconfig rm -f /opt/cni/bin/calico /opt/cni/bin/calico-ipam \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6e05\u7406\u8282\u70b9\u6587\u4ef6"},{"location":"ops/kubectl-ko/","text":"kubectl \u63d2\u4ef6\u4f7f\u7528 \u00b6 \u4e3a\u4e86\u65b9\u4fbf\u65e5\u5e38\u7684\u8fd0\u7ef4\u64cd\u4f5c\uff0cKube-OVN \u63d0\u4f9b\u4e86 kubectl \u63d2\u4ef6\u5de5\u5177\uff0c\u7f51\u7edc\u7ba1\u7406\u5458 \u53ef\u4ee5\u901a\u8fc7\u8be5\u547d\u4ee4\u8fdb\u884c\u65e5\u5e38\u64cd\u4f5c\uff0c\u4f8b\u5982\uff1a\u67e5\u770b OVN \u6570\u636e\u5e93\u4fe1\u606f\u548c\u72b6\u6001\uff0cOVN \u6570\u636e\u5e93 \u5907\u4efd\u548c\u6062\u590d\uff0cOVS \u76f8\u5173\u4fe1\u606f\u67e5\u770b\uff0ctcpdump \u7279\u5b9a\u5bb9\u5668\uff0c\u7279\u5b9a\u94fe\u8def\u903b\u8f91\u62d3\u6251\u5c55\u793a\uff0c \u7f51\u7edc\u95ee\u9898\u8bca\u65ad\u548c\u6027\u80fd\u4f18\u5316\u3002 \u63d2\u4ef6\u5b89\u88c5 \u00b6 Kube-OVN \u5b89\u88c5\u65f6\u9ed8\u8ba4\u4f1a\u90e8\u7f72\u63d2\u4ef6\u5230\u6bcf\u4e2a\u8282\u70b9\uff0c\u82e5\u6267\u884c kubectl \u7684\u673a\u5668\u4e0d\u5728\u96c6\u7fa4\u5185\uff0c \u6216\u9700\u8981\u91cd\u88c5\u63d2\u4ef6\uff0c\u53ef\u53c2\u8003\u4e0b\u9762\u7684\u6b65\u9aa4\uff1a \u4e0b\u8f7d kubectl-ko \u6587\u4ef6\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/kubectl-ko \u5c06\u8be5\u6587\u4ef6\u79fb\u52a8\u81f3 $PATH \u76ee\u5f55\u4e0b\uff1a mv kubectl-ko /usr/local/bin/kubectl-ko \u589e\u52a0\u53ef\u6267\u884c\u6743\u9650\uff1a chmod +x /usr/local/bin/kubectl-ko \u68c0\u67e5\u63d2\u4ef6\u662f\u5426\u53ef\u4ee5\u6b63\u5e38\u4f7f\u7528\uff1a # kubectl plugin list The following compatible plugins are available: /usr/local/bin/kubectl-ko \u63d2\u4ef6\u4f7f\u7528 \u00b6 \u8fd0\u884c kubectl ko \u4f1a\u5c55\u793a\u8be5\u63d2\u4ef6\u6240\u6709\u53ef\u7528\u7684\u547d\u4ee4\u548c\u7528\u6cd5\u63cf\u8ff0\uff0c\u5982\u4e0b\u6240\u793a\uff1a # kubectl ko kubectl ko { subcommand } [ option... ] Available Subcommands: [ nb | sb ] [ status | kick | backup | dbstatus | restore ] ovn-db operations show cluster status, kick stale server, backup database, get db consistency status or restore ovn nb db when met 'inconsistent data' error nbctl [ ovn-nbctl options ... ] invoke ovn-nbctl sbctl [ ovn-sbctl options ... ] invoke ovn-sbctl vsctl { nodeName } [ ovs-vsctl options ... ] invoke ovs-vsctl on the specified node ofctl { nodeName } [ ovs-ofctl options ... ] invoke ovs-ofctl on the specified node dpctl { nodeName } [ ovs-dpctl options ... ] invoke ovs-dpctl on the specified node appctl { nodeName } [ ovs-appctl options ... ] invoke ovs-appctl on the specified node tcpdump { namespace/podname } [ tcpdump options ... ] capture pod traffic trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply echo \" diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] diagnose connectivity of all nodes or a specific node or specify subnet's ds pod or IPPorts like 'tcp-172.18.0.2-53,udp-172.18.0.3-53'\" tuning { install-fastpath | local-install-fastpath | remove-fastpath | install-stt | local-install-stt | remove-stt } { centos7 | centos8 }} [ kernel-devel-version ] deploy kernel optimisation components to the system reload restart all kube-ovn components log { kube-ovn | ovn | ovs | linux | all } save log to ./kubectl-ko-log/ perf [ image ] performance test default image is kubeovn/test:v1.12.0 \u4e0b\u9762\u5c06\u4ecb\u7ecd\u6bcf\u4e2a\u547d\u4ee4\u7684\u5177\u4f53\u529f\u80fd\u548c\u4f7f\u7528\u3002 [nb | sb] [status | kick | backup | dbstatus | restore] \u00b6 \u8be5\u5b50\u547d\u4ee4\u4e3b\u8981\u5bf9 OVN \u5317\u5411\u6216\u5357\u5411\u6570\u636e\u5e93\u8fdb\u884c\u64cd\u4f5c\uff0c\u5305\u62ec\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u67e5\u770b\uff0c\u6570\u636e\u5e93\u8282\u70b9\u4e0b\u7ebf\uff0c \u6570\u636e\u5e93\u5907\u4efd\uff0c\u6570\u636e\u5e93\u5b58\u50a8\u72b6\u6001\u67e5\u770b\u548c\u6570\u636e\u5e93\u4fee\u590d\u3002 \u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u67e5\u770b \u00b6 \u8be5\u547d\u4ee4\u4f1a\u5728\u5bf9\u5e94 OVN \u6570\u636e\u5e93\u7684 leader \u8282\u70b9\u6267\u884c ovs-appctl cluster/status \u5c55\u793a\u96c6\u7fa4\u72b6\u6001: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 280309 ms ago, reason: timeout Last Election won: 280309 ms ago Election timer: 5000 Log: [ 139 , 139 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-8723 ->8723 <-85d6 ->85d6 Disconnections: 0 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 8723 ( 8723 at tcp: [ 172 .18.0.3 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 138 status: ok \u82e5 Server \u4e0b\u7684 match_index \u51fa\u73b0\u8f83\u5927\u5dee\u522b\uff0c\u4e14 last msg \u65f6\u95f4\u8f83\u957f\u5219\u5bf9\u5e94 Server \u53ef\u80fd\u957f\u65f6\u95f4\u6ca1\u6709\u54cd\u5e94\uff0c \u9700\u8981\u8fdb\u4e00\u6b65\u67e5\u770b\u3002 \u6570\u636e\u5e93\u8282\u70b9\u4e0b\u7ebf \u00b6 \u8be5\u547d\u4ee4\u4f1a\u5c06\u67d0\u4e2a\u8282\u70b9\u4ece OVN \u6570\u636e\u5e93\u4e2d\u79fb\u9664\uff0c\u5728\u8282\u70b9\u4e0b\u7ebf\u6216\u66f4\u6362\u8282\u70b9\u65f6\u9700\u8981\u7528\u5230\u3002 \u4e0b\u9762\u5c06\u4ee5\u4e0a\u4e00\u6761\u547d\u4ee4\u6240\u67e5\u770b\u5230\u7684\u96c6\u7fa4\u72b6\u6001\u4e3a\u4f8b\uff0c\u4e0b\u7ebf 172.18.0.3 \u8282\u70b9: # kubectl ko nb kick 8723 started removal \u518d\u6b21\u67e5\u770b\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u786e\u8ba4\u8282\u70b9\u5df2\u79fb\u9664\uff1a # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 324356 ms ago, reason: timeout Last Election won: 324356 ms ago Election timer: 5000 Log: [ 140 , 140 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-85d6 ->85d6 Disconnections: 2 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 140 match_index = 139 last msg 848 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 139 status: ok \u6570\u636e\u5e93\u5907\u4efd \u00b6 \u8be5\u5b50\u547d\u4ee4\u4f1a\u5907\u4efd\u5f53\u524d OVN \u6570\u636e\u5e93\u81f3\u672c\u5730\uff0c\u53ef\u7528\u4e8e\u707e\u5907\u548c\u6062\u590d\uff1a # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup \u6570\u636e\u5e93\u5b58\u50a8\u72b6\u6001\u67e5\u770b \u00b6 \u8be5\u547d\u4ee4\u7528\u6765\u67e5\u770b\u6570\u636e\u5e93\u6587\u4ef6\u662f\u5426\u5b58\u5728\u635f\u574f\uff1a # kubectl ko nb dbstatus status: ok \u82e5\u5f02\u5e38\u5219\u663e\u793a inconsistent data \u9700\u8981\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u8fdb\u884c\u4fee\u590d\u3002 \u6570\u636e\u5e93\u4fee\u590d \u00b6 \u82e5\u6570\u636e\u5e93\u72b6\u6001\u8fdb\u5165 inconsistent data \u53ef\u4f7f\u7528\u8be5\u547d\u4ee4\u8fdb\u884c\u4fee\u590d\uff1a # kubectl ko nb restore deployment.apps/ovn-central scaled ovn-central original replicas is 3 first nodeIP is 172 .18.0.5 ovs-ovn pod on node 172 .18.0.5 is ovs-ovn-8jxv9 ovs-ovn pod on node 172 .18.0.3 is ovs-ovn-sjzb6 ovs-ovn pod on node 172 .18.0.4 is ovs-ovn-t87zk backup nb db file restore nb db file, operate in pod ovs-ovn-8jxv9 deployment.apps/ovn-central scaled finish restore nb db file and ovn-central replicas recreate ovs-ovn pods pod \"ovs-ovn-8jxv9\" deleted pod \"ovs-ovn-sjzb6\" deleted pod \"ovs-ovn-t87zk\" deleted [nbctl | sbctl] [options ...] \u00b6 \u8be5\u5b50\u547d\u4ee4\u4f1a\u76f4\u63a5\u8fdb\u5165 OVN \u5317\u5411\u6570\u636e\u5e93\u6216\u5357\u5411\u6570\u636e\u5e93 \u7684 leader \u8282\u70b9\u5206\u522b\u6267\u884c ovn-nbctl \u548c ovn-sbctl \u547d\u4ee4\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVN \u7684\u5b98\u65b9\u6587\u6863 ovn-nbctl(8) \u548c ovn-sbctl(8) \u3002 # kubectl ko nbctl show switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port node-liumengxin-ovn2-192.168.137.177 addresses: [ \"00:00:00:D9:58:B8 100.64.0.3\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port alertmanager-main-0.monitoring addresses: [ \"00:00:00:6C:DF:A3 10.16.0.19\" ] port kube-state-metrics-5d6885d89-4nf8h.monitoring addresses: [ \"00:00:00:6F:02:1C 10.16.0.15\" ] port fake-kubelet-67c55dfd89-pv86k.kube-system addresses: [ \"00:00:00:5C:12:E8 10.16.19.177\" ] port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] vsctl {nodeName} [options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-vsctl \u547d\u4ee4\uff0c\u67e5\u8be2\u5e76\u914d\u7f6e vswitchd \u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-vsctl(8) \u3002 # kubectl ko vsctl kube-ovn-01 show 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port mirror0 Interface mirror0 type: internal Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" ofctl {nodeName} [options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-ofctl \u547d\u4ee4\uff0c\u67e5\u8be2\u6216\u7ba1\u7406 OpenFlow\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-ofctl(8) \u3002 # kubectl ko ofctl kube-ovn-01 dump-flows br-int NXST_FLOW reply ( xid = 0x4 ) : flags =[ more ] cookie = 0xcf3429e6, duration = 671791 .432s, table = 0 , n_packets = 0 , n_bytes = 0 , idle_age = 65534 , hard_age = 65534 , priority = 100 ,in_port = 2 actions = load:0x4->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x1->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xc91413c6, duration = 671791 .431s, table = 0 , n_packets = 907489 , n_bytes = 99978275 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 7 actions = load:0x1->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x4->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xf180459, duration = 671791 .431s, table = 0 , n_packets = 17348582 , n_bytes = 2667811214 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 6317 actions = load:0xa->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x9->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0x7806dd90, duration = 671791 .431s, table = 0 , n_packets = 3235428 , n_bytes = 833821312 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 1 actions = load:0xd->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x3->NXM_NX_REG14 [] ,resubmit ( ,8 ) ... dpctl {nodeName} [options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-dpctl \u547d\u4ee4\uff0c\u67e5\u8be2\u6216\u7ba1\u7406 OVS datapath\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-dpctl(8) \u3002 # kubectl ko dpctl kube-ovn-01 show system@ovs-system: lookups: hit:350805055 missed:21983648 lost:73 flows: 105 masks: hit:1970748791 total:22 hit/pkt:5.29 port 0 : ovs-system ( internal ) port 1 : ovn0 ( internal ) port 2 : mirror0 ( internal ) port 3 : br-int ( internal ) port 4 : stt_sys_7471 ( stt: packet_type = ptap ) port 5 : eeb4d9e51b5d_h port 6 : a2c1a8a8b83a_h port 7 : 4fa5c4cbb1a5_h appctl {nodeName} [options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-appctl \u547d\u4ee4\uff0c\u6765\u64cd\u4f5c\u76f8\u5173 daemon \u8fdb\u7a0b\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-appctl(8) \u3002 # kubectl ko appctl kube-ovn-01 vlog/list console syslog file ------- ------ ------ backtrace OFF ERR INFO bfd OFF ERR INFO bond OFF ERR INFO bridge OFF ERR INFO bundle OFF ERR INFO bundles OFF ERR INFO ... tcpdump {namespace/podname} [tcpdump options ...] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165 namespace/podname \u6240\u5728\u673a\u5668\u7684 kube-ovn-cni \u5bb9\u5668\uff0c\u5e76\u6267\u884c tcpdump \u6293\u53d6\u5bf9\u5e94\u5bb9\u5668 veth \u7f51\u5361 \u7aef\u7684\u6d41\u91cf\uff0c\u53ef\u4ee5\u65b9\u4fbf\u6392\u67e5\u7f51\u7edc\u76f8\u5173\u95ee\u9898\uff0c\u5982\u4e0b\u6240\u793a\uff1a # kubectl ko tcpdump default/ds1-l6n7p icmp + kubectl exec -it kube-ovn-cni-wlg4s -n kube-ovn -- tcpdump -nn -i d7176fe7b4e0_h icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on d7176fe7b4e0_h, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 06 :52:36.619688 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 1 , length 64 06 :52:36.619746 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 1 , length 64 06 :52:37.619588 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 2 , length 64 06 :52:37.619630 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 2 , length 64 06 :52:38.619933 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 3 , length 64 06 :52:38.619973 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 3 , length 64 trace [arguments ...] \u00b6 \u8be5\u547d\u4ee4\u5c06\u4f1a\u6253\u5370 Pod \u6216\u8282\u70b9\u901a\u8fc7\u7279\u5b9a\u534f\u8bae\u8bbf\u95ee\u67d0\u5730\u5740\u65f6\u5bf9\u5e94\u7684 OVN \u903b\u8f91\u6d41\u8868\u548c\u6700\u7ec8\u7684 Openflow \u6d41\u8868\uff0c \u65b9\u4fbf\u5f00\u53d1\u6216\u8fd0\u7ef4\u65f6\u5b9a\u4f4d\u6d41\u8868\u76f8\u5173\u95ee\u9898\u3002 \u652f\u6301\u7684\u547d\u4ee4\uff1a kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } kubectl ko trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } \u793a\u4f8b\uff1a # kubectl ko trace default/ds1-l6n7p 8.8.8.8 icmp + kubectl exec ovn-central-5bc494cb5-np9hm -n kube-ovn -- ovn-trace --ct = new ovn-default 'inport == \"ds1-l6n7p.default\" && ip.ttl == 64 && icmp && eth.src == 0a:00:00:10:00:05 && ip4.src == 10.16.0.4 && eth.dst == 00:00:00:B8:CA:43 && ip4.dst == 8.8.8.8' # icmp,reg14=0xf,vlan_tci=0x0000,dl_src=0a:00:00:10:00:05,dl_dst=00:00:00:b8:ca:43,nw_src=10.16.0.4,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 ingress ( dp = \"ovn-default\" , inport = \"ds1-l6n7p.default\" ) ----------------------------------------------------- 0 . ls_in_port_sec_l2 ( ovn-northd.c:4143 ) : inport == \"ds1-l6n7p.default\" && eth.src == { 0a:00:00:10:00:05 } , priority 50 , uuid 39453393 next ; 1 . ls_in_port_sec_ip ( ovn-northd.c:2898 ) : inport == \"ds1-l6n7p.default\" && eth.src == 0a:00:00:10:00:05 && ip4.src == { 10 .16.0.4 } , priority 90 , uuid 81bcd485 next ; 3 . ls_in_pre_acl ( ovn-northd.c:3269 ) : ip, priority 100 , uuid 7b4f4971 reg0 [ 0 ] = 1 ; next ; 5 . ls_in_pre_stateful ( ovn-northd.c:3396 ) : reg0 [ 0 ] == 1 , priority 100 , uuid 36cdd577 ct_next ; ct_next ( ct_state = new | trk ) ------------------------- 6 . ls_in_acl ( ovn-northd.c:3759 ) : ip && ( !ct.est || ( ct.est && ct_label.blocked == 1 )) , priority 1 , uuid 7608af5b reg0 [ 1 ] = 1 ; next ; 10 . ls_in_stateful ( ovn-northd.c:3995 ) : reg0 [ 1 ] == 1 , priority 100 , uuid 2aba1b90 ct_commit ( ct_label = 0 /0x1 ) ; next ; 16 . ls_in_l2_lkup ( ovn-northd.c:4470 ) : eth.dst == 00 :00:00:b8:ca:43, priority 50 , uuid 5c9c3c9f outport = \"ovn-default-ovn-cluster\" ; output ; ... \u82e5 trace \u5bf9\u8c61\u4e3a\u8fd0\u884c\u4e8e Underlay \u7f51\u7edc\u4e0b\u7684\u865a\u62df\u673a\uff0c\u9700\u8981\u6dfb\u52a0\u989d\u5916\u53c2\u6570\u6765\u6307\u5b9a\u76ee\u7684 Mac \u5730\u5740\uff1a kubectl ko trace default/virt-handler-7lvml 8 .8.8.8 82 :7c:9f:83:8c:01 icmp diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] \u00b6 \u8bca\u65ad\u96c6\u7fa4\u7f51\u7edc\u7ec4\u4ef6\u72b6\u6001\uff0c\u5e76\u53bb\u5bf9\u5e94\u8282\u70b9\u7684 kube-ovn-pinger \u68c0\u6d4b\u5f53\u524d\u8282\u70b9\u5230\u5176\u4ed6\u8282\u70b9\u548c\u5173\u952e\u670d\u52a1\u7684\u8fde\u901a\u6027\u548c\u7f51\u7edc\u5ef6\u8fdf\uff1a # kubectl ko diagnose all switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default port prometheus-k8s-1.monitoring addresses: [ \"00:00:00:AA:37:DF 10.16.0.23\" ] router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] Routing Policies 31000 ip4.dst == 10 .16.0.0/16 allow 31000 ip4.dst == 100 .64.0.0/16 allow 30000 ip4.dst == 192 .168.137.177 reroute 100 .64.0.3 30000 ip4.dst == 192 .168.137.178 reroute 100 .64.0.4 29000 ip4.src == $ovn .default.fake.6_ip4 reroute 100 .64.0.22 29000 ip4.src == $ovn .default.fake.7_ip4 reroute 100 .64.0.21 29000 ip4.src == $ovn .default.fake.8_ip4 reroute 100 .64.0.23 29000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 reroute 100 .64.0.4 20000 ip4.src == $ovn .default.liumengxin.ovn1.192.168.137.176_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.2 20000 ip4.src == $ovn .default.liumengxin.ovn2.192.168.137.177_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.3 20000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.4 IPv4 Routes Route Table <main>: 0 .0.0.0/0 100 .64.0.1 dst-ip UUID LB PROTO VIP IPs e9bcfd9d-793e-4431-9073-6dec96b75d71 cluster-tcp-load tcp 10 .100.209.132:10660 192 .168.137.176:10660 tcp 10 .101.239.192:6641 192 .168.137.177:6641 tcp 10 .101.240.101:3000 10 .16.0.7:3000 tcp 10 .103.184.186:6642 192 .168.137.177:6642 35d2b7a5-e3a7-485a-a4b7-b4970eb0e63b cluster-tcp-sess tcp 10 .100.158.128:8080 10 .16.0.10:8080,10.16.0.5:8080,10.16.63.30:8080 tcp 10 .107.26.215:8080 10 .16.0.19:8080,10.16.0.20:8080,10.16.0.21:8080 tcp 10 .107.26.215:9093 10 .16.0.19:9093,10.16.0.20:9093,10.16.0.21:9093 tcp 10 .98.187.99:8080 10 .16.0.22:8080,10.16.0.23:8080 tcp 10 .98.187.99:9090 10 .16.0.22:9090,10.16.0.23:9090 f43303e4-89aa-4d3e-a3dc-278a552fe27b cluster-udp-load udp 10 .96.0.10:53 10 .16.0.4:53,10.16.0.9:53 _uuid : 06776304 -5a96-43ed-90c4-c4854c251699 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn2_192.168.137.177_underlay_v6 _uuid : 62690625 -87d5-491c-8675-9fd83b1f433c addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn1_192.168.137.176_underlay_v6 _uuid : b03a9bae-94d5-4562-b34c-b5f6198e180b addresses : [ \"10.16.0.0/16\" , \"100.64.0.0/16\" , \"172.22.0.0/16\" , \"192.101.0.0/16\" ] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv4 _uuid : e1056f3a-24cc-4666-8a91-75ee6c3c2426 addresses : [] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv6 _uuid : 3e5d5fff-e670-47b2-a2f5-a39f4698a8c5 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn3_192.168.137.178_underlay_v6 _uuid : 2d85dbdc-d0db-4abe-b19e-cc806d32b492 action : drop direction : from-lport external_ids : {} label : 0 log : false match : \"inport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] _uuid : de790cc8-f155-405f-bb32-5a51f30c545f action : drop direction : to-lport external_ids : {} label : 0 log : false match : \"outport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] Chassis \"e15ed4d4-1780-4d50-b09e-ea8372ed48b8\" hostname: liumengxin-ovn1-192.168.137.176 Encap stt ip: \"192.168.137.176\" options: { csum = \"true\" } Port_Binding node-liumengxin-ovn1-192.168.137.176 Port_Binding perf-6vxkn.default Port_Binding kube-state-metrics-5d6885d89-4nf8h.monitoring Port_Binding alertmanager-main-0.monitoring Port_Binding kube-ovn-pinger-6ftdf.kube-system Port_Binding fake-kubelet-67c55dfd89-pv86k.kube-system Port_Binding prometheus-k8s-0.monitoring Chassis \"eef07da1-f8ad-4775-b14d-bd6a3b4eb0d5\" hostname: liumengxin-ovn3-192.168.137.178 Encap stt ip: \"192.168.137.178\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-7twb4.kube-system Port_Binding prometheus-adapter-86df476d87-rl88g.monitoring Port_Binding prometheus-k8s-1.monitoring Port_Binding node-liumengxin-ovn3-192.168.137.178 Port_Binding perf-ff475.default Port_Binding alertmanager-main-1.monitoring Port_Binding blackbox-exporter-676d976865-tvsjd.monitoring Chassis \"efa253c9-494d-4719-83ae-b48ab0f11c03\" hostname: liumengxin-ovn2-192.168.137.177 Encap stt ip: \"192.168.137.177\" options: { csum = \"true\" } Port_Binding grafana-6c4c6b8fb7-pzd2c.monitoring Port_Binding node-liumengxin-ovn2-192.168.137.177 Port_Binding alertmanager-main-2.monitoring Port_Binding coredns-6789c94dd8-9jqsz.kube-system Port_Binding coredns-6789c94dd8-25d4r.kube-system Port_Binding prometheus-operator-7bbc99fc8b-wgjm4.monitoring Port_Binding prometheus-adapter-86df476d87-gdxmc.monitoring Port_Binding perf-fjnws.default Port_Binding kube-ovn-pinger-vh2xg.kube-system ds kube-proxy ready kube-proxy ready deployment ovn-central ready deployment kube-ovn-controller ready ds kube-ovn-cni ready ds ovs-ovn ready deployment coredns ready ovn-nb leader check ok ovn-sb leader check ok ovn-northd leader check ok ### kube-ovn-controller recent log ### start to diagnose node liumengxin-ovn1-192.168.137.176 #### ovn-controller log: 2022 -06-03T00:56:44.897Z | 16722 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:06:44.912Z | 16723 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:16:44.925Z | 16724 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:26:44.936Z | 16725 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:36:44.959Z | 16726 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:46:44.974Z | 16727 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:56:44.988Z | 16728 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:06:45.001Z | 16729 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:16:45.025Z | 16730 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:26:45.040Z | 16731 | inc_proc_eng | INFO | User triggered force recompute. #### ovs-vswitchd log: 2022 -06-02T23:03:00.137Z | 00079 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:f9d1 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-02T23:23:31.840Z | 00080 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:15b2 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:09:15.659Z | 00081 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:dc:e3:63,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.63.30,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:e5a5 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x150017000004,src = 192 .168.137.178,dst = 192 .168.137.176,ttl = 64 ,tp_src = 9239 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.63.30,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:30:13.409Z | 00064 | dpif ( handler2 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:6b4a with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T02:02:33.832Z | 00082 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:a819 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 #### ovs-vsctl show results: 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port \"04d03360e9a0_h\" Interface \"04d03360e9a0_h\" Port eeb4d9e51b5d_h Interface eeb4d9e51b5d_h Port mirror0 Interface mirror0 type: internal Port \"8e5d887ccd80_h\" Interface \"8e5d887ccd80_h\" Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port \"17512d5be1f1_h\" Interface \"17512d5be1f1_h\" Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" #### pinger diagnose results: I0603 10 :35:04.349404 17619 pinger.go:19 ] ------------------------------------------------------------------------------- Kube-OVN: Version: v1.12.0 Build: 2022 -04-24_08:02:50 Commit: git-73f9d15 Go Version: go1.17.8 Arch: amd64 ------------------------------------------------------------------------------- I0603 10 :35:04.376797 17619 config.go:166 ] pinger config is & { KubeConfigFile: KubeClient:0xc000493380 Port:8080 DaemonSetNamespace:kube-system DaemonSetName:kube-ovn-pinger Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:liumengxin-ovn1-192.168.137.176 HostIP:192.168.137.176 PodName:kube-ovn-pinger-6ftdf PodIP:10.16.0.10 PodProtocols: [ IPv4 ] ExternalAddress: NetworkMode:kube-ovn PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid } I0603 10 :35:04.449166 17619 exporter.go:75 ] liumengxin-ovn1-192.168.137.176: exporter connect successfully I0603 10 :35:04.554011 17619 ovn.go:21 ] ovs-vswitchd and ovsdb are up I0603 10 :35:04.651293 17619 ovn.go:33 ] ovn_controller is up I0603 10 :35:04.651342 17619 ovn.go:39 ] start to check port binding I0603 10 :35:04.749613 17619 ovn.go:135 ] chassis id is 1d7f3d6c-eec5-4b3c-adca-2969d9cdfd80 I0603 10 :35:04.763487 17619 ovn.go:49 ] port in sb is [ node-liumengxin-ovn1-192.168.137.176 perf-6vxkn.default kube-state-metrics-5d6885d89-4nf8h.monitoring alertmanager-main-0.monitoring kube-ovn-pinger-6ftdf.kube-system fake-kubelet-67c55dfd89-pv86k.kube-system prometheus-k8s-0.monitoring ] I0603 10 :35:04.763583 17619 ovn.go:61 ] ovs and ovn-sb binding check passed I0603 10 :35:05.049309 17619 ping.go:259 ] start to check apiserver connectivity I0603 10 :35:05.053666 17619 ping.go:268 ] connect to apiserver success in 4 .27ms I0603 10 :35:05.053786 17619 ping.go:129 ] start to check pod connectivity I0603 10 :35:05.249590 17619 ping.go:159 ] ping pod: kube-ovn-pinger-6ftdf 10 .16.0.10, count: 3 , loss count 0 , average rtt 16 .30ms I0603 10 :35:05.354135 17619 ping.go:159 ] ping pod: kube-ovn-pinger-7twb4 10 .16.63.30, count: 3 , loss count 0 , average rtt 1 .81ms I0603 10 :35:05.458460 17619 ping.go:159 ] ping pod: kube-ovn-pinger-vh2xg 10 .16.0.5, count: 3 , loss count 0 , average rtt 1 .92ms I0603 10 :35:05.458523 17619 ping.go:83 ] start to check node connectivity \u5982\u679c diagnose \u7684\u76ee\u6807\u6307\u5b9a\u4e3a subnet \u8be5\u811a\u672c\u4f1a\u5728 subnet \u4e0a\u5efa\u7acb daemonset\uff0c\u7531 kube-ovn-pinger \u53bb\u63a2\u6d4b\u8fd9\u4e2a daemonset \u7684\u6240\u6709 pod \u7684\u8fde\u901a\u6027\u548c\u7f51\u7edc\u5ef6\u65f6\uff0c\u6d4b\u8bd5\u5b8c\u540e\u81ea\u52a8\u9500\u6bc1\u8be5 daemonset\u3002 \u5982\u679c diagnose \u7684\u76ee\u6807\u6307\u5b9a\u4e3a IPPorts \u8be5\u811a\u672c\u4f1a\u8ba9\u6bcf\u4e2a kube-ovn-pinger pod \u53bb\u63a2\u6d4b\u76ee\u6807\u534f\u8bae\uff0cIP\uff0cPort \u662f\u5426\u53ef\u8fbe\u3002 tuning {install-fastpath|local-install-fastpath|remove-fastpath|install-stt|local-install-stt|remove-stt} {centos7|centos8}} [kernel-devel-version] \u00b6 \u8be5\u547d\u4ee4\u6267\u884c\u6027\u80fd\u8c03\u4f18\u76f8\u5173\u64cd\u4f5c\uff0c\u5177\u4f53\u4f7f\u7528\u8bf7\u53c2\u8003 \u6027\u80fd\u8c03\u4f18 \u3002 reload \u00b6 \u8be5\u547d\u4ee4\u91cd\u542f\u6240\u6709 Kube-OVN \u76f8\u5173\u7ec4\u4ef6\uff1a # kubectl ko reload pod \"ovn-central-8684dd94bd-vzgcr\" deleted Waiting for deployment \"ovn-central\" rollout to finish: 0 of 1 updated replicas are available... deployment \"ovn-central\" successfully rolled out pod \"ovs-ovn-bsnvz\" deleted pod \"ovs-ovn-m9b98\" deleted pod \"kube-ovn-controller-8459db5ff4-64c62\" deleted Waiting for deployment \"kube-ovn-controller\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out pod \"kube-ovn-cni-2klnh\" deleted pod \"kube-ovn-cni-t2jz4\" deleted Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-cni\" successfully rolled out pod \"kube-ovn-pinger-ln72z\" deleted pod \"kube-ovn-pinger-w8lrk\" deleted Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-pinger\" successfully rolled out pod \"kube-ovn-monitor-7fb67d5488-7q6zb\" deleted Waiting for deployment \"kube-ovn-monitor\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-monitor\" successfully rolled out log \u00b6 \u4f7f\u7528\u8be5\u547d\u4ee4\u4f1a\u6293\u53d6 kube-ovn \u6240\u6709\u8282\u70b9\u4e0a\u7684 Kube-OVN\uff0cOVN\uff0cOpenvswitch \u7684 log \u4ee5\u53ca linux \u5e38\u7528\u7684\u4e00\u4e9b debug \u4fe1\u606f\u3002 # kubectl ko log all Collecting kube-ovn logging files Collecting ovn logging files Collecting openvswitch logging files Collecting linux dmesg files Collecting linux iptables-legacy files Collecting linux iptables-nft files Collecting linux route files Collecting linux link files Collecting linux neigh files Collecting linux memory files Collecting linux top files Collecting linux sysctl files Collecting linux netstat files Collecting linux addr files Collecting linux ipset files Collecting linux tcp files Collected files have been saved in the directory /root/kubectl-ko-log \u76ee\u5f55\u5982\u4e0b\uff1a # tree kubectl-ko-log/ kubectl-ko-log/ | -- kube-ovn-control-plane | | -- kube-ovn | | | -- kube-ovn-cni.log | | | -- kube-ovn-monitor.log | | ` -- kube-ovn-pinger.log | | -- linux | | | -- addr.log | | | -- dmesg.log | | | -- ipset.log | | | -- iptables-legacy.log | | | -- iptables-nft.log | | | -- link.log | | | -- memory.log | | | -- neigh.log | | | -- netstat.log | | | -- route.log | | | -- sysctl.log | | | -- tcp.log | | ` -- top.log | | -- openvswitch | | | -- ovs-vswitchd.log | | ` -- ovsdb-server.log | ` -- ovn | | -- ovn-controller.log | | -- ovn-northd.log | | -- ovsdb-server-nb.log | ` -- ovsdb-server-sb.log perf [image] \u00b6 \u8be5\u547d\u4ee4\u4f1a\u53bb\u6d4b\u8bd5 Kube-OVN \u7684\u4e00\u4e9b\u6027\u80fd\u6307\u6807\u5982\u4e0b\uff1a \u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u6307\u6807\uff1b Hostnetwork \u7f51\u7edc\u6027\u80fd\u6307\u6807\uff1b \u5bb9\u5668\u7f51\u7edc\u7ec4\u64ad\u62a5\u6587\u6027\u80fd\u6307\u6807\uff1b OVN-NB, OVN-SB, OVN-Northd leader \u5220\u9664\u6062\u590d\u6240\u9700\u65f6\u95f4\u3002 \u53c2\u6570 image \u7528\u4e8e\u6307\u5b9a\u6027\u80fd\u6d4b\u8bd5 pod \u6240\u7528\u7684\u955c\u50cf\uff0c\u9ed8\u8ba4\u60c5\u51b5\u4e0b\u662f kubeovn/test:v1.12.0 , \u8bbe\u7f6e\u8be5\u53c2\u6570\u4e3b\u8981\u662f\u4e3a\u4e86\u79bb\u7ebf\u573a\u666f\uff0c\u5c06\u955c\u50cf\u62c9\u5230\u5185\u7f51\u73af\u5883\u53ef\u80fd\u4f1a\u6709\u955c\u50cf\u540d\u53d8\u5316\u3002 # kubectl ko perf ============================== Prepareing Performance Test Resources =============================== pod/test-client created pod/test-host-client created pod/test-server created pod/test-host-server created service/test-server created pod/test-client condition met pod/test-host-client condition met pod/test-host-server condition met pod/test-server condition met ==================================================================================================== ============================ Start Pod Network Unicast Performance Test ============================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 82 .8 us 97 .7 Mbits/sec 67 .6 us ( 0 % ) 8 .42 Mbits/sec 128 85 .4 us 167 Mbits/sec 67 .2 us ( 0 % ) 17 .2 Mbits/sec 512 85 .8 us 440 Mbits/sec 68 .7 us ( 0 % ) 68 .4 Mbits/sec 1k 85 .1 us 567 Mbits/sec 68 .7 us ( 0 % ) 134 Mbits/sec 4k 138 us 826 Mbits/sec 78 .1 us ( 1 .4% ) 503 Mbits/sec ==================================================================================================== =============================== Start Host Network Performance Test ================================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 49 .7 us 120 Mbits/sec 37 .9 us ( 0 % ) 18 .6 Mbits/sec 128 49 .7 us 200 Mbits/sec 38 .1 us ( 0 % ) 35 .5 Mbits/sec 512 51 .9 us 588 Mbits/sec 38 .9 us ( 0 % ) 142 Mbits/sec 1k 51 .7 us 944 Mbits/sec 37 .2 us ( 0 % ) 279 Mbits/sec 4k 74 .9 us 1 .66 Gbits/sec 39 .9 us ( 0 % ) 1 .20 Gbits/sec ==================================================================================================== ============================== Start Service Network Performance Test ============================== Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 111 us 96 .3 Mbits/sec 88 .4 us ( 0 % ) 7 .59 Mbits/sec 128 83 .7 us 150 Mbits/sec 69 .2 us ( 0 % ) 16 .9 Mbits/sec 512 87 .4 us 374 Mbits/sec 75 .8 us ( 0 % ) 60 .9 Mbits/sec 1k 88 .2 us 521 Mbits/sec 73 .1 us ( 0 % ) 123 Mbits/sec 4k 148 us 813 Mbits/sec 77 .6 us ( 0 .0044% ) 451 Mbits/sec ==================================================================================================== =========================== Start Pod Multicast Network Performance Test =========================== Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .014 ms ( 0 .17% ) 5 .80 Mbits/sec 128 0 .012 ms ( 0 % ) 11 .4 Mbits/sec 512 0 .016 ms ( 0 % ) 46 .1 Mbits/sec 1k 0 .023 ms ( 0 .073% ) 89 .8 Mbits/sec 4k 0 .035 ms ( 1 .3% ) 126 Mbits/sec ==================================================================================================== ============================= Start Host Multicast Network Performance ============================= Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .007 ms ( 0 % ) 9 .95 Mbits/sec 128 0 .005 ms ( 0 % ) 21 .8 Mbits/sec 512 0 .008 ms ( 0 % ) 86 .8 Mbits/sec 1k 0 .013 ms ( 0 .045% ) 168 Mbits/sec 4k 0 .010 ms ( 0 .31% ) 242 Mbits/sec ==================================================================================================== ================================== Start Leader Recover Time Test ================================== Delete ovn central nb pod pod \"ovn-central-5cb9c67d75-tlz9w\" deleted Waiting for ovn central nb pod running =============================== OVN nb Recovery takes 3 .305236803 s ================================ Delete ovn central sb pod pod \"ovn-central-5cb9c67d75-szx4c\" deleted Waiting for ovn central sb pod running =============================== OVN sb Recovery takes 3 .462698535 s ================================ Delete ovn central northd pod pod \"ovn-central-5cb9c67d75-zqmqv\" deleted Waiting for ovn central northd pod running ============================= OVN northd Recovery takes 2 .691291403 s ============================== ==================================================================================================== ================================= Remove Performance Test Resource ================================= rm -f unicast-test-client.log rm -f unicast-test-host-client.log rm -f unicast-test-client.log kubectl ko nbctl lb-del test-server rm -f multicast-test-server.log kubectl exec ovs-ovn-gxdrf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 kubectl exec ovs-ovn-h57bf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 rm -f multicast-test-host-server.log pod \"test-client\" deleted pod \"test-host-client\" deleted pod \"test-host-server\" deleted pod \"test-server\" deleted service \"test-server\" deleted ==================================================================================================== \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kubectl \u63d2\u4ef6\u4f7f\u7528"},{"location":"ops/kubectl-ko/#kubectl","text":"\u4e3a\u4e86\u65b9\u4fbf\u65e5\u5e38\u7684\u8fd0\u7ef4\u64cd\u4f5c\uff0cKube-OVN \u63d0\u4f9b\u4e86 kubectl \u63d2\u4ef6\u5de5\u5177\uff0c\u7f51\u7edc\u7ba1\u7406\u5458 \u53ef\u4ee5\u901a\u8fc7\u8be5\u547d\u4ee4\u8fdb\u884c\u65e5\u5e38\u64cd\u4f5c\uff0c\u4f8b\u5982\uff1a\u67e5\u770b OVN \u6570\u636e\u5e93\u4fe1\u606f\u548c\u72b6\u6001\uff0cOVN \u6570\u636e\u5e93 \u5907\u4efd\u548c\u6062\u590d\uff0cOVS \u76f8\u5173\u4fe1\u606f\u67e5\u770b\uff0ctcpdump \u7279\u5b9a\u5bb9\u5668\uff0c\u7279\u5b9a\u94fe\u8def\u903b\u8f91\u62d3\u6251\u5c55\u793a\uff0c \u7f51\u7edc\u95ee\u9898\u8bca\u65ad\u548c\u6027\u80fd\u4f18\u5316\u3002","title":"kubectl \u63d2\u4ef6\u4f7f\u7528"},{"location":"ops/kubectl-ko/#_1","text":"Kube-OVN \u5b89\u88c5\u65f6\u9ed8\u8ba4\u4f1a\u90e8\u7f72\u63d2\u4ef6\u5230\u6bcf\u4e2a\u8282\u70b9\uff0c\u82e5\u6267\u884c kubectl \u7684\u673a\u5668\u4e0d\u5728\u96c6\u7fa4\u5185\uff0c \u6216\u9700\u8981\u91cd\u88c5\u63d2\u4ef6\uff0c\u53ef\u53c2\u8003\u4e0b\u9762\u7684\u6b65\u9aa4\uff1a \u4e0b\u8f7d kubectl-ko \u6587\u4ef6\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/kubectl-ko \u5c06\u8be5\u6587\u4ef6\u79fb\u52a8\u81f3 $PATH \u76ee\u5f55\u4e0b\uff1a mv kubectl-ko /usr/local/bin/kubectl-ko \u589e\u52a0\u53ef\u6267\u884c\u6743\u9650\uff1a chmod +x /usr/local/bin/kubectl-ko \u68c0\u67e5\u63d2\u4ef6\u662f\u5426\u53ef\u4ee5\u6b63\u5e38\u4f7f\u7528\uff1a # kubectl plugin list The following compatible plugins are available: /usr/local/bin/kubectl-ko","title":"\u63d2\u4ef6\u5b89\u88c5"},{"location":"ops/kubectl-ko/#_2","text":"\u8fd0\u884c kubectl ko \u4f1a\u5c55\u793a\u8be5\u63d2\u4ef6\u6240\u6709\u53ef\u7528\u7684\u547d\u4ee4\u548c\u7528\u6cd5\u63cf\u8ff0\uff0c\u5982\u4e0b\u6240\u793a\uff1a # kubectl ko kubectl ko { subcommand } [ option... ] Available Subcommands: [ nb | sb ] [ status | kick | backup | dbstatus | restore ] ovn-db operations show cluster status, kick stale server, backup database, get db consistency status or restore ovn nb db when met 'inconsistent data' error nbctl [ ovn-nbctl options ... ] invoke ovn-nbctl sbctl [ ovn-sbctl options ... ] invoke ovn-sbctl vsctl { nodeName } [ ovs-vsctl options ... ] invoke ovs-vsctl on the specified node ofctl { nodeName } [ ovs-ofctl options ... ] invoke ovs-ofctl on the specified node dpctl { nodeName } [ ovs-dpctl options ... ] invoke ovs-dpctl on the specified node appctl { nodeName } [ ovs-appctl options ... ] invoke ovs-appctl on the specified node tcpdump { namespace/podname } [ tcpdump options ... ] capture pod traffic trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply echo \" diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] diagnose connectivity of all nodes or a specific node or specify subnet's ds pod or IPPorts like 'tcp-172.18.0.2-53,udp-172.18.0.3-53'\" tuning { install-fastpath | local-install-fastpath | remove-fastpath | install-stt | local-install-stt | remove-stt } { centos7 | centos8 }} [ kernel-devel-version ] deploy kernel optimisation components to the system reload restart all kube-ovn components log { kube-ovn | ovn | ovs | linux | all } save log to ./kubectl-ko-log/ perf [ image ] performance test default image is kubeovn/test:v1.12.0 \u4e0b\u9762\u5c06\u4ecb\u7ecd\u6bcf\u4e2a\u547d\u4ee4\u7684\u5177\u4f53\u529f\u80fd\u548c\u4f7f\u7528\u3002","title":"\u63d2\u4ef6\u4f7f\u7528"},{"location":"ops/kubectl-ko/#nb-sb-status-kick-backup-dbstatus-restore","text":"\u8be5\u5b50\u547d\u4ee4\u4e3b\u8981\u5bf9 OVN \u5317\u5411\u6216\u5357\u5411\u6570\u636e\u5e93\u8fdb\u884c\u64cd\u4f5c\uff0c\u5305\u62ec\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u67e5\u770b\uff0c\u6570\u636e\u5e93\u8282\u70b9\u4e0b\u7ebf\uff0c \u6570\u636e\u5e93\u5907\u4efd\uff0c\u6570\u636e\u5e93\u5b58\u50a8\u72b6\u6001\u67e5\u770b\u548c\u6570\u636e\u5e93\u4fee\u590d\u3002","title":"[nb | sb] [status | kick | backup | dbstatus | restore]"},{"location":"ops/kubectl-ko/#_3","text":"\u8be5\u547d\u4ee4\u4f1a\u5728\u5bf9\u5e94 OVN \u6570\u636e\u5e93\u7684 leader \u8282\u70b9\u6267\u884c ovs-appctl cluster/status \u5c55\u793a\u96c6\u7fa4\u72b6\u6001: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 280309 ms ago, reason: timeout Last Election won: 280309 ms ago Election timer: 5000 Log: [ 139 , 139 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-8723 ->8723 <-85d6 ->85d6 Disconnections: 0 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 8723 ( 8723 at tcp: [ 172 .18.0.3 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 138 status: ok \u82e5 Server \u4e0b\u7684 match_index \u51fa\u73b0\u8f83\u5927\u5dee\u522b\uff0c\u4e14 last msg \u65f6\u95f4\u8f83\u957f\u5219\u5bf9\u5e94 Server \u53ef\u80fd\u957f\u65f6\u95f4\u6ca1\u6709\u54cd\u5e94\uff0c \u9700\u8981\u8fdb\u4e00\u6b65\u67e5\u770b\u3002","title":"\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u67e5\u770b"},{"location":"ops/kubectl-ko/#_4","text":"\u8be5\u547d\u4ee4\u4f1a\u5c06\u67d0\u4e2a\u8282\u70b9\u4ece OVN \u6570\u636e\u5e93\u4e2d\u79fb\u9664\uff0c\u5728\u8282\u70b9\u4e0b\u7ebf\u6216\u66f4\u6362\u8282\u70b9\u65f6\u9700\u8981\u7528\u5230\u3002 \u4e0b\u9762\u5c06\u4ee5\u4e0a\u4e00\u6761\u547d\u4ee4\u6240\u67e5\u770b\u5230\u7684\u96c6\u7fa4\u72b6\u6001\u4e3a\u4f8b\uff0c\u4e0b\u7ebf 172.18.0.3 \u8282\u70b9: # kubectl ko nb kick 8723 started removal \u518d\u6b21\u67e5\u770b\u6570\u636e\u5e93\u96c6\u7fa4\u72b6\u6001\u786e\u8ba4\u8282\u70b9\u5df2\u79fb\u9664\uff1a # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 324356 ms ago, reason: timeout Last Election won: 324356 ms ago Election timer: 5000 Log: [ 140 , 140 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-85d6 ->85d6 Disconnections: 2 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 140 match_index = 139 last msg 848 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 139 status: ok","title":"\u6570\u636e\u5e93\u8282\u70b9\u4e0b\u7ebf"},{"location":"ops/kubectl-ko/#_5","text":"\u8be5\u5b50\u547d\u4ee4\u4f1a\u5907\u4efd\u5f53\u524d OVN \u6570\u636e\u5e93\u81f3\u672c\u5730\uff0c\u53ef\u7528\u4e8e\u707e\u5907\u548c\u6062\u590d\uff1a # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup","title":"\u6570\u636e\u5e93\u5907\u4efd"},{"location":"ops/kubectl-ko/#_6","text":"\u8be5\u547d\u4ee4\u7528\u6765\u67e5\u770b\u6570\u636e\u5e93\u6587\u4ef6\u662f\u5426\u5b58\u5728\u635f\u574f\uff1a # kubectl ko nb dbstatus status: ok \u82e5\u5f02\u5e38\u5219\u663e\u793a inconsistent data \u9700\u8981\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u8fdb\u884c\u4fee\u590d\u3002","title":"\u6570\u636e\u5e93\u5b58\u50a8\u72b6\u6001\u67e5\u770b"},{"location":"ops/kubectl-ko/#_7","text":"\u82e5\u6570\u636e\u5e93\u72b6\u6001\u8fdb\u5165 inconsistent data \u53ef\u4f7f\u7528\u8be5\u547d\u4ee4\u8fdb\u884c\u4fee\u590d\uff1a # kubectl ko nb restore deployment.apps/ovn-central scaled ovn-central original replicas is 3 first nodeIP is 172 .18.0.5 ovs-ovn pod on node 172 .18.0.5 is ovs-ovn-8jxv9 ovs-ovn pod on node 172 .18.0.3 is ovs-ovn-sjzb6 ovs-ovn pod on node 172 .18.0.4 is ovs-ovn-t87zk backup nb db file restore nb db file, operate in pod ovs-ovn-8jxv9 deployment.apps/ovn-central scaled finish restore nb db file and ovn-central replicas recreate ovs-ovn pods pod \"ovs-ovn-8jxv9\" deleted pod \"ovs-ovn-sjzb6\" deleted pod \"ovs-ovn-t87zk\" deleted","title":"\u6570\u636e\u5e93\u4fee\u590d"},{"location":"ops/kubectl-ko/#nbctl-sbctl-options","text":"\u8be5\u5b50\u547d\u4ee4\u4f1a\u76f4\u63a5\u8fdb\u5165 OVN \u5317\u5411\u6570\u636e\u5e93\u6216\u5357\u5411\u6570\u636e\u5e93 \u7684 leader \u8282\u70b9\u5206\u522b\u6267\u884c ovn-nbctl \u548c ovn-sbctl \u547d\u4ee4\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVN \u7684\u5b98\u65b9\u6587\u6863 ovn-nbctl(8) \u548c ovn-sbctl(8) \u3002 # kubectl ko nbctl show switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port node-liumengxin-ovn2-192.168.137.177 addresses: [ \"00:00:00:D9:58:B8 100.64.0.3\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port alertmanager-main-0.monitoring addresses: [ \"00:00:00:6C:DF:A3 10.16.0.19\" ] port kube-state-metrics-5d6885d89-4nf8h.monitoring addresses: [ \"00:00:00:6F:02:1C 10.16.0.15\" ] port fake-kubelet-67c55dfd89-pv86k.kube-system addresses: [ \"00:00:00:5C:12:E8 10.16.19.177\" ] port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ]","title":"[nbctl | sbctl] [options ...]"},{"location":"ops/kubectl-ko/#vsctl-nodename-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-vsctl \u547d\u4ee4\uff0c\u67e5\u8be2\u5e76\u914d\u7f6e vswitchd \u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-vsctl(8) \u3002 # kubectl ko vsctl kube-ovn-01 show 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port mirror0 Interface mirror0 type: internal Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port br-int Interface br-int type: internal ovs_version: \"2.17.2\"","title":"vsctl {nodeName} [options ...]"},{"location":"ops/kubectl-ko/#ofctl-nodename-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-ofctl \u547d\u4ee4\uff0c\u67e5\u8be2\u6216\u7ba1\u7406 OpenFlow\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-ofctl(8) \u3002 # kubectl ko ofctl kube-ovn-01 dump-flows br-int NXST_FLOW reply ( xid = 0x4 ) : flags =[ more ] cookie = 0xcf3429e6, duration = 671791 .432s, table = 0 , n_packets = 0 , n_bytes = 0 , idle_age = 65534 , hard_age = 65534 , priority = 100 ,in_port = 2 actions = load:0x4->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x1->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xc91413c6, duration = 671791 .431s, table = 0 , n_packets = 907489 , n_bytes = 99978275 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 7 actions = load:0x1->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x4->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xf180459, duration = 671791 .431s, table = 0 , n_packets = 17348582 , n_bytes = 2667811214 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 6317 actions = load:0xa->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x9->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0x7806dd90, duration = 671791 .431s, table = 0 , n_packets = 3235428 , n_bytes = 833821312 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 1 actions = load:0xd->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x3->NXM_NX_REG14 [] ,resubmit ( ,8 ) ...","title":"ofctl {nodeName} [options ...]"},{"location":"ops/kubectl-ko/#dpctl-nodename-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-dpctl \u547d\u4ee4\uff0c\u67e5\u8be2\u6216\u7ba1\u7406 OVS datapath\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-dpctl(8) \u3002 # kubectl ko dpctl kube-ovn-01 show system@ovs-system: lookups: hit:350805055 missed:21983648 lost:73 flows: 105 masks: hit:1970748791 total:22 hit/pkt:5.29 port 0 : ovs-system ( internal ) port 1 : ovn0 ( internal ) port 2 : mirror0 ( internal ) port 3 : br-int ( internal ) port 4 : stt_sys_7471 ( stt: packet_type = ptap ) port 5 : eeb4d9e51b5d_h port 6 : a2c1a8a8b83a_h port 7 : 4fa5c4cbb1a5_h","title":"dpctl {nodeName} [options ...]"},{"location":"ops/kubectl-ko/#appctl-nodename-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165\u5bf9\u5e94 nodeName \u4e0a\u7684 ovs-ovn \u5bb9\u5668\uff0c\u5e76\u6267\u884c\u76f8\u5e94\u7684 ovs-appctl \u547d\u4ee4\uff0c\u6765\u64cd\u4f5c\u76f8\u5173 daemon \u8fdb\u7a0b\u3002 \u66f4\u591a\u8be5\u547d\u4ee4\u7684\u8be6\u7ec6\u7528\u6cd5\u8bf7\u67e5\u8be2\u4e0a\u6e38 OVS \u7684\u5b98\u65b9\u6587\u6863 ovs-appctl(8) \u3002 # kubectl ko appctl kube-ovn-01 vlog/list console syslog file ------- ------ ------ backtrace OFF ERR INFO bfd OFF ERR INFO bond OFF ERR INFO bridge OFF ERR INFO bundle OFF ERR INFO bundles OFF ERR INFO ...","title":"appctl {nodeName} [options ...]"},{"location":"ops/kubectl-ko/#tcpdump-namespacepodname-tcpdump-options","text":"\u8be5\u547d\u4ee4\u4f1a\u8fdb\u5165 namespace/podname \u6240\u5728\u673a\u5668\u7684 kube-ovn-cni \u5bb9\u5668\uff0c\u5e76\u6267\u884c tcpdump \u6293\u53d6\u5bf9\u5e94\u5bb9\u5668 veth \u7f51\u5361 \u7aef\u7684\u6d41\u91cf\uff0c\u53ef\u4ee5\u65b9\u4fbf\u6392\u67e5\u7f51\u7edc\u76f8\u5173\u95ee\u9898\uff0c\u5982\u4e0b\u6240\u793a\uff1a # kubectl ko tcpdump default/ds1-l6n7p icmp + kubectl exec -it kube-ovn-cni-wlg4s -n kube-ovn -- tcpdump -nn -i d7176fe7b4e0_h icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on d7176fe7b4e0_h, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 06 :52:36.619688 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 1 , length 64 06 :52:36.619746 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 1 , length 64 06 :52:37.619588 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 2 , length 64 06 :52:37.619630 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 2 , length 64 06 :52:38.619933 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 3 , length 64 06 :52:38.619973 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 3 , length 64","title":"tcpdump {namespace/podname} [tcpdump options ...]"},{"location":"ops/kubectl-ko/#trace-arguments","text":"\u8be5\u547d\u4ee4\u5c06\u4f1a\u6253\u5370 Pod \u6216\u8282\u70b9\u901a\u8fc7\u7279\u5b9a\u534f\u8bae\u8bbf\u95ee\u67d0\u5730\u5740\u65f6\u5bf9\u5e94\u7684 OVN \u903b\u8f91\u6d41\u8868\u548c\u6700\u7ec8\u7684 Openflow \u6d41\u8868\uff0c \u65b9\u4fbf\u5f00\u53d1\u6216\u8fd0\u7ef4\u65f6\u5b9a\u4f4d\u6d41\u8868\u76f8\u5173\u95ee\u9898\u3002 \u652f\u6301\u7684\u547d\u4ee4\uff1a kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } kubectl ko trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } \u793a\u4f8b\uff1a # kubectl ko trace default/ds1-l6n7p 8.8.8.8 icmp + kubectl exec ovn-central-5bc494cb5-np9hm -n kube-ovn -- ovn-trace --ct = new ovn-default 'inport == \"ds1-l6n7p.default\" && ip.ttl == 64 && icmp && eth.src == 0a:00:00:10:00:05 && ip4.src == 10.16.0.4 && eth.dst == 00:00:00:B8:CA:43 && ip4.dst == 8.8.8.8' # icmp,reg14=0xf,vlan_tci=0x0000,dl_src=0a:00:00:10:00:05,dl_dst=00:00:00:b8:ca:43,nw_src=10.16.0.4,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 ingress ( dp = \"ovn-default\" , inport = \"ds1-l6n7p.default\" ) ----------------------------------------------------- 0 . ls_in_port_sec_l2 ( ovn-northd.c:4143 ) : inport == \"ds1-l6n7p.default\" && eth.src == { 0a:00:00:10:00:05 } , priority 50 , uuid 39453393 next ; 1 . ls_in_port_sec_ip ( ovn-northd.c:2898 ) : inport == \"ds1-l6n7p.default\" && eth.src == 0a:00:00:10:00:05 && ip4.src == { 10 .16.0.4 } , priority 90 , uuid 81bcd485 next ; 3 . ls_in_pre_acl ( ovn-northd.c:3269 ) : ip, priority 100 , uuid 7b4f4971 reg0 [ 0 ] = 1 ; next ; 5 . ls_in_pre_stateful ( ovn-northd.c:3396 ) : reg0 [ 0 ] == 1 , priority 100 , uuid 36cdd577 ct_next ; ct_next ( ct_state = new | trk ) ------------------------- 6 . ls_in_acl ( ovn-northd.c:3759 ) : ip && ( !ct.est || ( ct.est && ct_label.blocked == 1 )) , priority 1 , uuid 7608af5b reg0 [ 1 ] = 1 ; next ; 10 . ls_in_stateful ( ovn-northd.c:3995 ) : reg0 [ 1 ] == 1 , priority 100 , uuid 2aba1b90 ct_commit ( ct_label = 0 /0x1 ) ; next ; 16 . ls_in_l2_lkup ( ovn-northd.c:4470 ) : eth.dst == 00 :00:00:b8:ca:43, priority 50 , uuid 5c9c3c9f outport = \"ovn-default-ovn-cluster\" ; output ; ... \u82e5 trace \u5bf9\u8c61\u4e3a\u8fd0\u884c\u4e8e Underlay \u7f51\u7edc\u4e0b\u7684\u865a\u62df\u673a\uff0c\u9700\u8981\u6dfb\u52a0\u989d\u5916\u53c2\u6570\u6765\u6307\u5b9a\u76ee\u7684 Mac \u5730\u5740\uff1a kubectl ko trace default/virt-handler-7lvml 8 .8.8.8 82 :7c:9f:83:8c:01 icmp","title":"trace [arguments ...]"},{"location":"ops/kubectl-ko/#diagnose-allnodesubnetipports-nodenamesubnetnameproto1-ip1-port1proto2-ip2-port2","text":"\u8bca\u65ad\u96c6\u7fa4\u7f51\u7edc\u7ec4\u4ef6\u72b6\u6001\uff0c\u5e76\u53bb\u5bf9\u5e94\u8282\u70b9\u7684 kube-ovn-pinger \u68c0\u6d4b\u5f53\u524d\u8282\u70b9\u5230\u5176\u4ed6\u8282\u70b9\u548c\u5173\u952e\u670d\u52a1\u7684\u8fde\u901a\u6027\u548c\u7f51\u7edc\u5ef6\u8fdf\uff1a # kubectl ko diagnose all switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default port prometheus-k8s-1.monitoring addresses: [ \"00:00:00:AA:37:DF 10.16.0.23\" ] router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] Routing Policies 31000 ip4.dst == 10 .16.0.0/16 allow 31000 ip4.dst == 100 .64.0.0/16 allow 30000 ip4.dst == 192 .168.137.177 reroute 100 .64.0.3 30000 ip4.dst == 192 .168.137.178 reroute 100 .64.0.4 29000 ip4.src == $ovn .default.fake.6_ip4 reroute 100 .64.0.22 29000 ip4.src == $ovn .default.fake.7_ip4 reroute 100 .64.0.21 29000 ip4.src == $ovn .default.fake.8_ip4 reroute 100 .64.0.23 29000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 reroute 100 .64.0.4 20000 ip4.src == $ovn .default.liumengxin.ovn1.192.168.137.176_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.2 20000 ip4.src == $ovn .default.liumengxin.ovn2.192.168.137.177_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.3 20000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.4 IPv4 Routes Route Table <main>: 0 .0.0.0/0 100 .64.0.1 dst-ip UUID LB PROTO VIP IPs e9bcfd9d-793e-4431-9073-6dec96b75d71 cluster-tcp-load tcp 10 .100.209.132:10660 192 .168.137.176:10660 tcp 10 .101.239.192:6641 192 .168.137.177:6641 tcp 10 .101.240.101:3000 10 .16.0.7:3000 tcp 10 .103.184.186:6642 192 .168.137.177:6642 35d2b7a5-e3a7-485a-a4b7-b4970eb0e63b cluster-tcp-sess tcp 10 .100.158.128:8080 10 .16.0.10:8080,10.16.0.5:8080,10.16.63.30:8080 tcp 10 .107.26.215:8080 10 .16.0.19:8080,10.16.0.20:8080,10.16.0.21:8080 tcp 10 .107.26.215:9093 10 .16.0.19:9093,10.16.0.20:9093,10.16.0.21:9093 tcp 10 .98.187.99:8080 10 .16.0.22:8080,10.16.0.23:8080 tcp 10 .98.187.99:9090 10 .16.0.22:9090,10.16.0.23:9090 f43303e4-89aa-4d3e-a3dc-278a552fe27b cluster-udp-load udp 10 .96.0.10:53 10 .16.0.4:53,10.16.0.9:53 _uuid : 06776304 -5a96-43ed-90c4-c4854c251699 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn2_192.168.137.177_underlay_v6 _uuid : 62690625 -87d5-491c-8675-9fd83b1f433c addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn1_192.168.137.176_underlay_v6 _uuid : b03a9bae-94d5-4562-b34c-b5f6198e180b addresses : [ \"10.16.0.0/16\" , \"100.64.0.0/16\" , \"172.22.0.0/16\" , \"192.101.0.0/16\" ] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv4 _uuid : e1056f3a-24cc-4666-8a91-75ee6c3c2426 addresses : [] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv6 _uuid : 3e5d5fff-e670-47b2-a2f5-a39f4698a8c5 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn3_192.168.137.178_underlay_v6 _uuid : 2d85dbdc-d0db-4abe-b19e-cc806d32b492 action : drop direction : from-lport external_ids : {} label : 0 log : false match : \"inport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] _uuid : de790cc8-f155-405f-bb32-5a51f30c545f action : drop direction : to-lport external_ids : {} label : 0 log : false match : \"outport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] Chassis \"e15ed4d4-1780-4d50-b09e-ea8372ed48b8\" hostname: liumengxin-ovn1-192.168.137.176 Encap stt ip: \"192.168.137.176\" options: { csum = \"true\" } Port_Binding node-liumengxin-ovn1-192.168.137.176 Port_Binding perf-6vxkn.default Port_Binding kube-state-metrics-5d6885d89-4nf8h.monitoring Port_Binding alertmanager-main-0.monitoring Port_Binding kube-ovn-pinger-6ftdf.kube-system Port_Binding fake-kubelet-67c55dfd89-pv86k.kube-system Port_Binding prometheus-k8s-0.monitoring Chassis \"eef07da1-f8ad-4775-b14d-bd6a3b4eb0d5\" hostname: liumengxin-ovn3-192.168.137.178 Encap stt ip: \"192.168.137.178\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-7twb4.kube-system Port_Binding prometheus-adapter-86df476d87-rl88g.monitoring Port_Binding prometheus-k8s-1.monitoring Port_Binding node-liumengxin-ovn3-192.168.137.178 Port_Binding perf-ff475.default Port_Binding alertmanager-main-1.monitoring Port_Binding blackbox-exporter-676d976865-tvsjd.monitoring Chassis \"efa253c9-494d-4719-83ae-b48ab0f11c03\" hostname: liumengxin-ovn2-192.168.137.177 Encap stt ip: \"192.168.137.177\" options: { csum = \"true\" } Port_Binding grafana-6c4c6b8fb7-pzd2c.monitoring Port_Binding node-liumengxin-ovn2-192.168.137.177 Port_Binding alertmanager-main-2.monitoring Port_Binding coredns-6789c94dd8-9jqsz.kube-system Port_Binding coredns-6789c94dd8-25d4r.kube-system Port_Binding prometheus-operator-7bbc99fc8b-wgjm4.monitoring Port_Binding prometheus-adapter-86df476d87-gdxmc.monitoring Port_Binding perf-fjnws.default Port_Binding kube-ovn-pinger-vh2xg.kube-system ds kube-proxy ready kube-proxy ready deployment ovn-central ready deployment kube-ovn-controller ready ds kube-ovn-cni ready ds ovs-ovn ready deployment coredns ready ovn-nb leader check ok ovn-sb leader check ok ovn-northd leader check ok ### kube-ovn-controller recent log ### start to diagnose node liumengxin-ovn1-192.168.137.176 #### ovn-controller log: 2022 -06-03T00:56:44.897Z | 16722 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:06:44.912Z | 16723 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:16:44.925Z | 16724 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:26:44.936Z | 16725 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:36:44.959Z | 16726 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:46:44.974Z | 16727 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:56:44.988Z | 16728 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:06:45.001Z | 16729 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:16:45.025Z | 16730 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:26:45.040Z | 16731 | inc_proc_eng | INFO | User triggered force recompute. #### ovs-vswitchd log: 2022 -06-02T23:03:00.137Z | 00079 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:f9d1 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-02T23:23:31.840Z | 00080 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:15b2 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:09:15.659Z | 00081 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:dc:e3:63,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.63.30,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:e5a5 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x150017000004,src = 192 .168.137.178,dst = 192 .168.137.176,ttl = 64 ,tp_src = 9239 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.63.30,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:30:13.409Z | 00064 | dpif ( handler2 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:6b4a with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T02:02:33.832Z | 00082 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:a819 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 #### ovs-vsctl show results: 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port \"04d03360e9a0_h\" Interface \"04d03360e9a0_h\" Port eeb4d9e51b5d_h Interface eeb4d9e51b5d_h Port mirror0 Interface mirror0 type: internal Port \"8e5d887ccd80_h\" Interface \"8e5d887ccd80_h\" Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port \"17512d5be1f1_h\" Interface \"17512d5be1f1_h\" Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" #### pinger diagnose results: I0603 10 :35:04.349404 17619 pinger.go:19 ] ------------------------------------------------------------------------------- Kube-OVN: Version: v1.12.0 Build: 2022 -04-24_08:02:50 Commit: git-73f9d15 Go Version: go1.17.8 Arch: amd64 ------------------------------------------------------------------------------- I0603 10 :35:04.376797 17619 config.go:166 ] pinger config is & { KubeConfigFile: KubeClient:0xc000493380 Port:8080 DaemonSetNamespace:kube-system DaemonSetName:kube-ovn-pinger Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:liumengxin-ovn1-192.168.137.176 HostIP:192.168.137.176 PodName:kube-ovn-pinger-6ftdf PodIP:10.16.0.10 PodProtocols: [ IPv4 ] ExternalAddress: NetworkMode:kube-ovn PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid } I0603 10 :35:04.449166 17619 exporter.go:75 ] liumengxin-ovn1-192.168.137.176: exporter connect successfully I0603 10 :35:04.554011 17619 ovn.go:21 ] ovs-vswitchd and ovsdb are up I0603 10 :35:04.651293 17619 ovn.go:33 ] ovn_controller is up I0603 10 :35:04.651342 17619 ovn.go:39 ] start to check port binding I0603 10 :35:04.749613 17619 ovn.go:135 ] chassis id is 1d7f3d6c-eec5-4b3c-adca-2969d9cdfd80 I0603 10 :35:04.763487 17619 ovn.go:49 ] port in sb is [ node-liumengxin-ovn1-192.168.137.176 perf-6vxkn.default kube-state-metrics-5d6885d89-4nf8h.monitoring alertmanager-main-0.monitoring kube-ovn-pinger-6ftdf.kube-system fake-kubelet-67c55dfd89-pv86k.kube-system prometheus-k8s-0.monitoring ] I0603 10 :35:04.763583 17619 ovn.go:61 ] ovs and ovn-sb binding check passed I0603 10 :35:05.049309 17619 ping.go:259 ] start to check apiserver connectivity I0603 10 :35:05.053666 17619 ping.go:268 ] connect to apiserver success in 4 .27ms I0603 10 :35:05.053786 17619 ping.go:129 ] start to check pod connectivity I0603 10 :35:05.249590 17619 ping.go:159 ] ping pod: kube-ovn-pinger-6ftdf 10 .16.0.10, count: 3 , loss count 0 , average rtt 16 .30ms I0603 10 :35:05.354135 17619 ping.go:159 ] ping pod: kube-ovn-pinger-7twb4 10 .16.63.30, count: 3 , loss count 0 , average rtt 1 .81ms I0603 10 :35:05.458460 17619 ping.go:159 ] ping pod: kube-ovn-pinger-vh2xg 10 .16.0.5, count: 3 , loss count 0 , average rtt 1 .92ms I0603 10 :35:05.458523 17619 ping.go:83 ] start to check node connectivity \u5982\u679c diagnose \u7684\u76ee\u6807\u6307\u5b9a\u4e3a subnet \u8be5\u811a\u672c\u4f1a\u5728 subnet \u4e0a\u5efa\u7acb daemonset\uff0c\u7531 kube-ovn-pinger \u53bb\u63a2\u6d4b\u8fd9\u4e2a daemonset \u7684\u6240\u6709 pod \u7684\u8fde\u901a\u6027\u548c\u7f51\u7edc\u5ef6\u65f6\uff0c\u6d4b\u8bd5\u5b8c\u540e\u81ea\u52a8\u9500\u6bc1\u8be5 daemonset\u3002 \u5982\u679c diagnose \u7684\u76ee\u6807\u6307\u5b9a\u4e3a IPPorts \u8be5\u811a\u672c\u4f1a\u8ba9\u6bcf\u4e2a kube-ovn-pinger pod \u53bb\u63a2\u6d4b\u76ee\u6807\u534f\u8bae\uff0cIP\uff0cPort \u662f\u5426\u53ef\u8fbe\u3002","title":"diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}]"},{"location":"ops/kubectl-ko/#tuning-install-fastpathlocal-install-fastpathremove-fastpathinstall-sttlocal-install-sttremove-stt-centos7centos8-kernel-devel-version","text":"\u8be5\u547d\u4ee4\u6267\u884c\u6027\u80fd\u8c03\u4f18\u76f8\u5173\u64cd\u4f5c\uff0c\u5177\u4f53\u4f7f\u7528\u8bf7\u53c2\u8003 \u6027\u80fd\u8c03\u4f18 \u3002","title":"tuning {install-fastpath|local-install-fastpath|remove-fastpath|install-stt|local-install-stt|remove-stt} {centos7|centos8}} [kernel-devel-version]"},{"location":"ops/kubectl-ko/#reload","text":"\u8be5\u547d\u4ee4\u91cd\u542f\u6240\u6709 Kube-OVN \u76f8\u5173\u7ec4\u4ef6\uff1a # kubectl ko reload pod \"ovn-central-8684dd94bd-vzgcr\" deleted Waiting for deployment \"ovn-central\" rollout to finish: 0 of 1 updated replicas are available... deployment \"ovn-central\" successfully rolled out pod \"ovs-ovn-bsnvz\" deleted pod \"ovs-ovn-m9b98\" deleted pod \"kube-ovn-controller-8459db5ff4-64c62\" deleted Waiting for deployment \"kube-ovn-controller\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out pod \"kube-ovn-cni-2klnh\" deleted pod \"kube-ovn-cni-t2jz4\" deleted Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-cni\" successfully rolled out pod \"kube-ovn-pinger-ln72z\" deleted pod \"kube-ovn-pinger-w8lrk\" deleted Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-pinger\" successfully rolled out pod \"kube-ovn-monitor-7fb67d5488-7q6zb\" deleted Waiting for deployment \"kube-ovn-monitor\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-monitor\" successfully rolled out","title":"reload"},{"location":"ops/kubectl-ko/#log","text":"\u4f7f\u7528\u8be5\u547d\u4ee4\u4f1a\u6293\u53d6 kube-ovn \u6240\u6709\u8282\u70b9\u4e0a\u7684 Kube-OVN\uff0cOVN\uff0cOpenvswitch \u7684 log \u4ee5\u53ca linux \u5e38\u7528\u7684\u4e00\u4e9b debug \u4fe1\u606f\u3002 # kubectl ko log all Collecting kube-ovn logging files Collecting ovn logging files Collecting openvswitch logging files Collecting linux dmesg files Collecting linux iptables-legacy files Collecting linux iptables-nft files Collecting linux route files Collecting linux link files Collecting linux neigh files Collecting linux memory files Collecting linux top files Collecting linux sysctl files Collecting linux netstat files Collecting linux addr files Collecting linux ipset files Collecting linux tcp files Collected files have been saved in the directory /root/kubectl-ko-log \u76ee\u5f55\u5982\u4e0b\uff1a # tree kubectl-ko-log/ kubectl-ko-log/ | -- kube-ovn-control-plane | | -- kube-ovn | | | -- kube-ovn-cni.log | | | -- kube-ovn-monitor.log | | ` -- kube-ovn-pinger.log | | -- linux | | | -- addr.log | | | -- dmesg.log | | | -- ipset.log | | | -- iptables-legacy.log | | | -- iptables-nft.log | | | -- link.log | | | -- memory.log | | | -- neigh.log | | | -- netstat.log | | | -- route.log | | | -- sysctl.log | | | -- tcp.log | | ` -- top.log | | -- openvswitch | | | -- ovs-vswitchd.log | | ` -- ovsdb-server.log | ` -- ovn | | -- ovn-controller.log | | -- ovn-northd.log | | -- ovsdb-server-nb.log | ` -- ovsdb-server-sb.log","title":"log"},{"location":"ops/kubectl-ko/#perf-image","text":"\u8be5\u547d\u4ee4\u4f1a\u53bb\u6d4b\u8bd5 Kube-OVN \u7684\u4e00\u4e9b\u6027\u80fd\u6307\u6807\u5982\u4e0b\uff1a \u5bb9\u5668\u7f51\u7edc\u7684\u6027\u80fd\u6307\u6807\uff1b Hostnetwork \u7f51\u7edc\u6027\u80fd\u6307\u6807\uff1b \u5bb9\u5668\u7f51\u7edc\u7ec4\u64ad\u62a5\u6587\u6027\u80fd\u6307\u6807\uff1b OVN-NB, OVN-SB, OVN-Northd leader \u5220\u9664\u6062\u590d\u6240\u9700\u65f6\u95f4\u3002 \u53c2\u6570 image \u7528\u4e8e\u6307\u5b9a\u6027\u80fd\u6d4b\u8bd5 pod \u6240\u7528\u7684\u955c\u50cf\uff0c\u9ed8\u8ba4\u60c5\u51b5\u4e0b\u662f kubeovn/test:v1.12.0 , \u8bbe\u7f6e\u8be5\u53c2\u6570\u4e3b\u8981\u662f\u4e3a\u4e86\u79bb\u7ebf\u573a\u666f\uff0c\u5c06\u955c\u50cf\u62c9\u5230\u5185\u7f51\u73af\u5883\u53ef\u80fd\u4f1a\u6709\u955c\u50cf\u540d\u53d8\u5316\u3002 # kubectl ko perf ============================== Prepareing Performance Test Resources =============================== pod/test-client created pod/test-host-client created pod/test-server created pod/test-host-server created service/test-server created pod/test-client condition met pod/test-host-client condition met pod/test-host-server condition met pod/test-server condition met ==================================================================================================== ============================ Start Pod Network Unicast Performance Test ============================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 82 .8 us 97 .7 Mbits/sec 67 .6 us ( 0 % ) 8 .42 Mbits/sec 128 85 .4 us 167 Mbits/sec 67 .2 us ( 0 % ) 17 .2 Mbits/sec 512 85 .8 us 440 Mbits/sec 68 .7 us ( 0 % ) 68 .4 Mbits/sec 1k 85 .1 us 567 Mbits/sec 68 .7 us ( 0 % ) 134 Mbits/sec 4k 138 us 826 Mbits/sec 78 .1 us ( 1 .4% ) 503 Mbits/sec ==================================================================================================== =============================== Start Host Network Performance Test ================================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 49 .7 us 120 Mbits/sec 37 .9 us ( 0 % ) 18 .6 Mbits/sec 128 49 .7 us 200 Mbits/sec 38 .1 us ( 0 % ) 35 .5 Mbits/sec 512 51 .9 us 588 Mbits/sec 38 .9 us ( 0 % ) 142 Mbits/sec 1k 51 .7 us 944 Mbits/sec 37 .2 us ( 0 % ) 279 Mbits/sec 4k 74 .9 us 1 .66 Gbits/sec 39 .9 us ( 0 % ) 1 .20 Gbits/sec ==================================================================================================== ============================== Start Service Network Performance Test ============================== Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 111 us 96 .3 Mbits/sec 88 .4 us ( 0 % ) 7 .59 Mbits/sec 128 83 .7 us 150 Mbits/sec 69 .2 us ( 0 % ) 16 .9 Mbits/sec 512 87 .4 us 374 Mbits/sec 75 .8 us ( 0 % ) 60 .9 Mbits/sec 1k 88 .2 us 521 Mbits/sec 73 .1 us ( 0 % ) 123 Mbits/sec 4k 148 us 813 Mbits/sec 77 .6 us ( 0 .0044% ) 451 Mbits/sec ==================================================================================================== =========================== Start Pod Multicast Network Performance Test =========================== Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .014 ms ( 0 .17% ) 5 .80 Mbits/sec 128 0 .012 ms ( 0 % ) 11 .4 Mbits/sec 512 0 .016 ms ( 0 % ) 46 .1 Mbits/sec 1k 0 .023 ms ( 0 .073% ) 89 .8 Mbits/sec 4k 0 .035 ms ( 1 .3% ) 126 Mbits/sec ==================================================================================================== ============================= Start Host Multicast Network Performance ============================= Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .007 ms ( 0 % ) 9 .95 Mbits/sec 128 0 .005 ms ( 0 % ) 21 .8 Mbits/sec 512 0 .008 ms ( 0 % ) 86 .8 Mbits/sec 1k 0 .013 ms ( 0 .045% ) 168 Mbits/sec 4k 0 .010 ms ( 0 .31% ) 242 Mbits/sec ==================================================================================================== ================================== Start Leader Recover Time Test ================================== Delete ovn central nb pod pod \"ovn-central-5cb9c67d75-tlz9w\" deleted Waiting for ovn central nb pod running =============================== OVN nb Recovery takes 3 .305236803 s ================================ Delete ovn central sb pod pod \"ovn-central-5cb9c67d75-szx4c\" deleted Waiting for ovn central sb pod running =============================== OVN sb Recovery takes 3 .462698535 s ================================ Delete ovn central northd pod pod \"ovn-central-5cb9c67d75-zqmqv\" deleted Waiting for ovn central northd pod running ============================= OVN northd Recovery takes 2 .691291403 s ============================== ==================================================================================================== ================================= Remove Performance Test Resource ================================= rm -f unicast-test-client.log rm -f unicast-test-host-client.log rm -f unicast-test-client.log kubectl ko nbctl lb-del test-server rm -f multicast-test-server.log kubectl exec ovs-ovn-gxdrf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 kubectl exec ovs-ovn-h57bf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 rm -f multicast-test-host-server.log pod \"test-client\" deleted pod \"test-host-client\" deleted pod \"test-host-server\" deleted pod \"test-server\" deleted service \"test-server\" deleted ==================================================================================================== \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"perf [image]"},{"location":"ops/recover-db/","text":"OVN \u6570\u636e\u5e93\u5907\u4efd\u548c\u6062\u590d \u00b6 \u672c\u6587\u6863\u4ecb\u7ecd\u5982\u4f55\u8fdb\u884c\u6570\u636e\u5e93\u5907\u4efd\uff0c\u4ee5\u53ca\u5728\u4e0d\u540c\u60c5\u51b5\u4e0b\u5982\u4f55\u901a\u8fc7\u5df2\u6709\u7684\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u96c6\u7fa4\u6062\u590d\u3002 \u6570\u636e\u5e93\u5907\u4efd \u00b6 \u5229\u7528 kubectl \u63d2\u4ef6\u7684 backup \u547d\u4ee4\u53ef\u4ee5\u5bf9\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u5907\u4efd\uff0c\u4ee5\u7528\u4e8e\u6545\u969c\u65f6\u6062\u590d\uff1a # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup # kubectl ko sb backup tar: Removing leading `/' from member names backup ovn-nb db to /root/ovnsb_db.060223191654183154.backup \u96c6\u7fa4\u90e8\u5206\u6545\u969c\u6062\u590d \u00b6 \u82e5\u96c6\u7fa4\u4e2d\u5b58\u5728\u90e8\u5206\u8282\u70b9\u56e0\u4e3a\u65ad\u7535\uff0c\u6587\u4ef6\u7cfb\u7edf\u6545\u969c\u6216\u78c1\u76d8\u7a7a\u95f4\u4e0d\u8db3\u5bfc\u81f4\u5de5\u4f5c\u5f02\u5e38\uff0c \u4f46\u662f\u96c6\u7fa4\u4ecd\u53ef\u6b63\u5e38\u5de5\u4f5c\u53ef\u4ee5\u901a\u8fc7\u5982\u4e0b\u6b65\u9aa4\u8fdb\u884c\u6062\u590d\u3002 \u67e5\u770b\u65e5\u5fd7\u786e\u8ba4\u72b6\u6001\u5f02\u5e38 \u00b6 \u67e5\u770b\u5bf9\u5e94\u8282\u70b9 /var/log/ovn/ovn-northd.log \uff0c\u82e5\u63d0\u793a\u7c7b\u4f3c\u9519\u8bef\u5219\u53ef\u5224\u65ad\u6570\u636e\u5e93\u5b58\u5728\u5f02\u5e38 * ovn-northd is not running ovsdb-server: ovsdb error: error reading record 2739 from OVN_Northbound log: record 2739 advances commit index to 6308 but last log index is 6307 * Starting ovsdb-nb \u4ece\u96c6\u7fa4\u4e2d\u8e22\u51fa\u5bf9\u5e94\u8282\u70b9 \u00b6 \u6839\u636e\u65e5\u5fd7\u63d0\u793a\u662f OVN_Northbound \u8fd8\u662f OVN_Southbound \u9009\u62e9\u5bf9\u5e94\u7684\u6570\u636e\u5e93\u8fdb\u884c\u64cd\u4f5c\u3002 \u4e0a\u8ff0\u65e5\u5fd7\u63d0\u793a\u4e3a OVN_Northbound \u5219\u5bf9 ovn-nb \u8fdb\u884c\u64cd\u4f5c\uff1a # kubectl ko nb status 9182 Name: OVN_Northbound Cluster ID: e75f ( e75fa340-49ed-45ab-990e-26cb865ebc85 ) Server ID: 9182 ( 9182e8dd-b5b0-4dd8-8518-598cc1e374f3 ) Address: tcp: [ 10 .0.128.61 ] :6643 Status: cluster member Role: leader Term: 1454 Leader: self Vote: self Last Election started 1732603 ms ago, reason: timeout Last Election won: 1732587 ms ago Election timer: 1000 Log: [ 7332 , 12512 ] Entries not yet committed: 1 Entries not yet applied: 1 Connections: ->f080 <-f080 <-e631 ->e631 Disconnections: 1 Servers: f080 ( f080 at tcp: [ 10 .0.129.139 ] :6643 ) next_index = 12512 match_index = 12510 last msg 63 ms ago 9182 ( 9182 at tcp: [ 10 .0.128.61 ] :6643 ) ( self ) next_index = 10394 match_index = 12510 e631 ( e631 at tcp: [ 10 .0.131.173 ] :6643 ) next_index = 12512 match_index = 0 \u4ece\u96c6\u7fa4\u4e2d\u8e22\u51fa\u72b6\u6001\u5f02\u5e38\u8282\u70b9\uff1a kubectl ko nb kick e631 \u767b\u5f55\u5f02\u5e38\u8282\u70b9\uff0c\u5220\u9664\u5bf9\u5e94\u7684\u6570\u636e\u5e93\u6587\u4ef6\uff1a mv /etc/origin/ovn/ovnnb_db.db /tmp \u5220\u9664\u5bf9\u5e94\u8282\u70b9\u7684 ovn-central Pod\uff0c\u7b49\u5f85\u96c6\u7fa4\u81ea\u52a8\u6062\u590d\uff1a kubectl delete pod -n kube-system ovn-central-xxxx \u96c6\u7fa4\u4e0d\u80fd\u6b63\u5e38\u5de5\u4f5c\u4e0b\u7684\u6062\u590d \u00b6 \u82e5\u96c6\u7fa4\u591a\u6570\u8282\u70b9\u53d7\u635f\u65e0\u6cd5\u9009\u4e3e\u51fa leader\uff0c\u8bf7\u53c2\u7167\u4e0b\u9762\u7684\u6b65\u9aa4\u8fdb\u884c\u6062\u590d\u3002 \u505c\u6b62 ovn-central \u00b6 \u8bb0\u5f55\u5f53\u524d ovn-central \u526f\u672c\u6570\u91cf\uff0c\u5e76\u505c\u6b62 ovn-central \u907f\u514d\u65b0\u7684\u6570\u636e\u5e93\u53d8\u66f4\u5f71\u54cd\u6062\u590d\uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 0 \u9009\u62e9\u5907\u4efd \u00b6 \u7531\u4e8e\u591a\u6570\u8282\u70b9\u53d7\u635f\uff0c\u9700\u8981\u4ece\u67d0\u4e2a\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u6062\u590d\u91cd\u5efa\u96c6\u7fa4\u3002\u5982\u679c\u4e4b\u524d\u5907\u4efd\u8fc7\u6570\u636e\u5e93 \u53ef\u4f7f\u7528\u4e4b\u524d\u7684\u5907\u4efd\u6587\u4ef6\u8fdb\u884c\u6062\u590d\u3002\u5982\u679c\u6ca1\u6709\u8fdb\u884c\u8fc7\u5907\u4efd\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u6b65\u9aa4\u4ece\u5df2\u6709\u7684\u6570\u636e\u5e93\u6587\u4ef6 \u4e2d\u751f\u6210\u4e00\u4e2a\u5907\u4efd\u3002 \u7531\u4e8e\u9ed8\u8ba4\u6587\u4ef6\u5939\u4e0b\u7684\u6570\u636e\u5e93\u6587\u4ef6\u4e3a\u96c6\u7fa4\u683c\u5f0f\u6570\u636e\u5e93\u6587\u4ef6\uff0c\u5305\u542b\u5f53\u524d\u96c6\u7fa4\u7684\u4fe1\u606f\uff0c\u65e0\u6cd5\u76f4\u63a5 \u7528\u8be5\u6587\u4ef6\u91cd\u5efa\u6570\u636e\u5e93\uff0c\u9700\u8981\u4f7f\u7528 ovsdb-tool cluster-to-standalone \u8fdb\u884c\u683c\u5f0f\u8f6c\u6362\u3002 \u9009\u62e9 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u4e2d\u6392\u7b2c\u4e00\u7684\u8282\u70b9\u6062\u590d\u6570\u636e\u5e93\u6587\u4ef6\uff0c \u5982\u679c\u7b2c\u4e00\u4e2a\u8282\u70b9\u6570\u636e\u5e93\u6587\u4ef6\u5df2\u635f\u574f\uff0c\u4ece\u5176\u4ed6\u673a\u5668 /etc/origin/ovn \u4e0b\u590d\u5236\u6587\u4ef6\u5230\u7b2c\u4e00\u53f0\u673a\u5668 \uff0c \u6267\u884c\u4e0b\u5217\u547d\u4ee4\u751f\u6210\u6570\u636e\u5e93\u6587\u4ef6\u5907\u4efd\u3002 docker run -it -v /etc/origin/ovn:/etc/ovn kubeovn/kube-ovn:v1.12.0 bash cd /etc/ovn/ ovsdb-tool cluster-to-standalone ovnnb_db_standalone.db ovnnb_db.db ovsdb-tool cluster-to-standalone ovnsb_db_standalone.db ovnsb_db.db \u5220\u9664\u6bcf\u4e2a ovn-central \u8282\u70b9\u4e0a\u7684\u6570\u636e\u5e93\u6587\u4ef6 \u00b6 \u4e3a\u4e86\u907f\u514d\u91cd\u5efa\u96c6\u7fa4\u65f6\u4f7f\u7528\u5230\u9519\u8bef\u7684\u6570\u636e\uff0c\u9700\u8981\u5bf9\u5df2\u6709\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u6e05\u7406\uff1a mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp \u6062\u590d\u6570\u636e\u5e93\u96c6\u7fa4 \u00b6 \u5c06\u5907\u4efd\u6570\u636e\u5e93\u5206\u522b\u91cd\u547d\u540d\u4e3a ovnnb_db.db \u548c ovnsb_db.db \uff0c\u5e76\u590d\u5236\u5230 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u4e2d\u6392\u7b2c\u4e00\u673a\u5668\u7684 /etc/origin/ovn/ \u76ee\u5f55\u4e0b\uff1a mv /etc/origin/ovn/ovnnb_db_standalone.db /etc/origin/ovn/ovnnb_db.db mv /etc/origin/ovn/ovnsb_db_standalone.db /etc/origin/ovn/ovnsb_db.db \u6062\u590d ovn-central \u7684\u526f\u672c\u6570\uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl rollout status deployment/ovn-central -n kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN \u6570\u636e\u5e93\u5907\u4efd\u548c\u6062\u590d"},{"location":"ops/recover-db/#ovn","text":"\u672c\u6587\u6863\u4ecb\u7ecd\u5982\u4f55\u8fdb\u884c\u6570\u636e\u5e93\u5907\u4efd\uff0c\u4ee5\u53ca\u5728\u4e0d\u540c\u60c5\u51b5\u4e0b\u5982\u4f55\u901a\u8fc7\u5df2\u6709\u7684\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u96c6\u7fa4\u6062\u590d\u3002","title":"OVN \u6570\u636e\u5e93\u5907\u4efd\u548c\u6062\u590d"},{"location":"ops/recover-db/#_1","text":"\u5229\u7528 kubectl \u63d2\u4ef6\u7684 backup \u547d\u4ee4\u53ef\u4ee5\u5bf9\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u5907\u4efd\uff0c\u4ee5\u7528\u4e8e\u6545\u969c\u65f6\u6062\u590d\uff1a # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup # kubectl ko sb backup tar: Removing leading `/' from member names backup ovn-nb db to /root/ovnsb_db.060223191654183154.backup","title":"\u6570\u636e\u5e93\u5907\u4efd"},{"location":"ops/recover-db/#_2","text":"\u82e5\u96c6\u7fa4\u4e2d\u5b58\u5728\u90e8\u5206\u8282\u70b9\u56e0\u4e3a\u65ad\u7535\uff0c\u6587\u4ef6\u7cfb\u7edf\u6545\u969c\u6216\u78c1\u76d8\u7a7a\u95f4\u4e0d\u8db3\u5bfc\u81f4\u5de5\u4f5c\u5f02\u5e38\uff0c \u4f46\u662f\u96c6\u7fa4\u4ecd\u53ef\u6b63\u5e38\u5de5\u4f5c\u53ef\u4ee5\u901a\u8fc7\u5982\u4e0b\u6b65\u9aa4\u8fdb\u884c\u6062\u590d\u3002","title":"\u96c6\u7fa4\u90e8\u5206\u6545\u969c\u6062\u590d"},{"location":"ops/recover-db/#_3","text":"\u67e5\u770b\u5bf9\u5e94\u8282\u70b9 /var/log/ovn/ovn-northd.log \uff0c\u82e5\u63d0\u793a\u7c7b\u4f3c\u9519\u8bef\u5219\u53ef\u5224\u65ad\u6570\u636e\u5e93\u5b58\u5728\u5f02\u5e38 * ovn-northd is not running ovsdb-server: ovsdb error: error reading record 2739 from OVN_Northbound log: record 2739 advances commit index to 6308 but last log index is 6307 * Starting ovsdb-nb","title":"\u67e5\u770b\u65e5\u5fd7\u786e\u8ba4\u72b6\u6001\u5f02\u5e38"},{"location":"ops/recover-db/#_4","text":"\u6839\u636e\u65e5\u5fd7\u63d0\u793a\u662f OVN_Northbound \u8fd8\u662f OVN_Southbound \u9009\u62e9\u5bf9\u5e94\u7684\u6570\u636e\u5e93\u8fdb\u884c\u64cd\u4f5c\u3002 \u4e0a\u8ff0\u65e5\u5fd7\u63d0\u793a\u4e3a OVN_Northbound \u5219\u5bf9 ovn-nb \u8fdb\u884c\u64cd\u4f5c\uff1a # kubectl ko nb status 9182 Name: OVN_Northbound Cluster ID: e75f ( e75fa340-49ed-45ab-990e-26cb865ebc85 ) Server ID: 9182 ( 9182e8dd-b5b0-4dd8-8518-598cc1e374f3 ) Address: tcp: [ 10 .0.128.61 ] :6643 Status: cluster member Role: leader Term: 1454 Leader: self Vote: self Last Election started 1732603 ms ago, reason: timeout Last Election won: 1732587 ms ago Election timer: 1000 Log: [ 7332 , 12512 ] Entries not yet committed: 1 Entries not yet applied: 1 Connections: ->f080 <-f080 <-e631 ->e631 Disconnections: 1 Servers: f080 ( f080 at tcp: [ 10 .0.129.139 ] :6643 ) next_index = 12512 match_index = 12510 last msg 63 ms ago 9182 ( 9182 at tcp: [ 10 .0.128.61 ] :6643 ) ( self ) next_index = 10394 match_index = 12510 e631 ( e631 at tcp: [ 10 .0.131.173 ] :6643 ) next_index = 12512 match_index = 0 \u4ece\u96c6\u7fa4\u4e2d\u8e22\u51fa\u72b6\u6001\u5f02\u5e38\u8282\u70b9\uff1a kubectl ko nb kick e631 \u767b\u5f55\u5f02\u5e38\u8282\u70b9\uff0c\u5220\u9664\u5bf9\u5e94\u7684\u6570\u636e\u5e93\u6587\u4ef6\uff1a mv /etc/origin/ovn/ovnnb_db.db /tmp \u5220\u9664\u5bf9\u5e94\u8282\u70b9\u7684 ovn-central Pod\uff0c\u7b49\u5f85\u96c6\u7fa4\u81ea\u52a8\u6062\u590d\uff1a kubectl delete pod -n kube-system ovn-central-xxxx","title":"\u4ece\u96c6\u7fa4\u4e2d\u8e22\u51fa\u5bf9\u5e94\u8282\u70b9"},{"location":"ops/recover-db/#_5","text":"\u82e5\u96c6\u7fa4\u591a\u6570\u8282\u70b9\u53d7\u635f\u65e0\u6cd5\u9009\u4e3e\u51fa leader\uff0c\u8bf7\u53c2\u7167\u4e0b\u9762\u7684\u6b65\u9aa4\u8fdb\u884c\u6062\u590d\u3002","title":"\u96c6\u7fa4\u4e0d\u80fd\u6b63\u5e38\u5de5\u4f5c\u4e0b\u7684\u6062\u590d"},{"location":"ops/recover-db/#ovn-central","text":"\u8bb0\u5f55\u5f53\u524d ovn-central \u526f\u672c\u6570\u91cf\uff0c\u5e76\u505c\u6b62 ovn-central \u907f\u514d\u65b0\u7684\u6570\u636e\u5e93\u53d8\u66f4\u5f71\u54cd\u6062\u590d\uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 0","title":"\u505c\u6b62 ovn-central"},{"location":"ops/recover-db/#_6","text":"\u7531\u4e8e\u591a\u6570\u8282\u70b9\u53d7\u635f\uff0c\u9700\u8981\u4ece\u67d0\u4e2a\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u6062\u590d\u91cd\u5efa\u96c6\u7fa4\u3002\u5982\u679c\u4e4b\u524d\u5907\u4efd\u8fc7\u6570\u636e\u5e93 \u53ef\u4f7f\u7528\u4e4b\u524d\u7684\u5907\u4efd\u6587\u4ef6\u8fdb\u884c\u6062\u590d\u3002\u5982\u679c\u6ca1\u6709\u8fdb\u884c\u8fc7\u5907\u4efd\u53ef\u4ee5\u4f7f\u7528\u4e0b\u9762\u7684\u6b65\u9aa4\u4ece\u5df2\u6709\u7684\u6570\u636e\u5e93\u6587\u4ef6 \u4e2d\u751f\u6210\u4e00\u4e2a\u5907\u4efd\u3002 \u7531\u4e8e\u9ed8\u8ba4\u6587\u4ef6\u5939\u4e0b\u7684\u6570\u636e\u5e93\u6587\u4ef6\u4e3a\u96c6\u7fa4\u683c\u5f0f\u6570\u636e\u5e93\u6587\u4ef6\uff0c\u5305\u542b\u5f53\u524d\u96c6\u7fa4\u7684\u4fe1\u606f\uff0c\u65e0\u6cd5\u76f4\u63a5 \u7528\u8be5\u6587\u4ef6\u91cd\u5efa\u6570\u636e\u5e93\uff0c\u9700\u8981\u4f7f\u7528 ovsdb-tool cluster-to-standalone \u8fdb\u884c\u683c\u5f0f\u8f6c\u6362\u3002 \u9009\u62e9 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u4e2d\u6392\u7b2c\u4e00\u7684\u8282\u70b9\u6062\u590d\u6570\u636e\u5e93\u6587\u4ef6\uff0c \u5982\u679c\u7b2c\u4e00\u4e2a\u8282\u70b9\u6570\u636e\u5e93\u6587\u4ef6\u5df2\u635f\u574f\uff0c\u4ece\u5176\u4ed6\u673a\u5668 /etc/origin/ovn \u4e0b\u590d\u5236\u6587\u4ef6\u5230\u7b2c\u4e00\u53f0\u673a\u5668 \uff0c \u6267\u884c\u4e0b\u5217\u547d\u4ee4\u751f\u6210\u6570\u636e\u5e93\u6587\u4ef6\u5907\u4efd\u3002 docker run -it -v /etc/origin/ovn:/etc/ovn kubeovn/kube-ovn:v1.12.0 bash cd /etc/ovn/ ovsdb-tool cluster-to-standalone ovnnb_db_standalone.db ovnnb_db.db ovsdb-tool cluster-to-standalone ovnsb_db_standalone.db ovnsb_db.db","title":"\u9009\u62e9\u5907\u4efd"},{"location":"ops/recover-db/#ovn-central_1","text":"\u4e3a\u4e86\u907f\u514d\u91cd\u5efa\u96c6\u7fa4\u65f6\u4f7f\u7528\u5230\u9519\u8bef\u7684\u6570\u636e\uff0c\u9700\u8981\u5bf9\u5df2\u6709\u6570\u636e\u5e93\u6587\u4ef6\u8fdb\u884c\u6e05\u7406\uff1a mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp","title":"\u5220\u9664\u6bcf\u4e2a ovn-central \u8282\u70b9\u4e0a\u7684\u6570\u636e\u5e93\u6587\u4ef6"},{"location":"ops/recover-db/#_7","text":"\u5c06\u5907\u4efd\u6570\u636e\u5e93\u5206\u522b\u91cd\u547d\u540d\u4e3a ovnnb_db.db \u548c ovnsb_db.db \uff0c\u5e76\u590d\u5236\u5230 ovn-central \u73af\u5883\u53d8\u91cf NODE_IPS \u4e2d\u6392\u7b2c\u4e00\u673a\u5668\u7684 /etc/origin/ovn/ \u76ee\u5f55\u4e0b\uff1a mv /etc/origin/ovn/ovnnb_db_standalone.db /etc/origin/ovn/ovnnb_db.db mv /etc/origin/ovn/ovnsb_db_standalone.db /etc/origin/ovn/ovnsb_db.db \u6062\u590d ovn-central \u7684\u526f\u672c\u6570\uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl rollout status deployment/ovn-central -n kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6062\u590d\u6570\u636e\u5e93\u96c6\u7fa4"},{"location":"reference/architecture/","text":"\u603b\u4f53\u67b6\u6784 \u00b6 \u672c\u6587\u6863\u5c06\u4ecb\u7ecd Kube-OVN \u7684\u603b\u4f53\u67b6\u6784\uff0c\u548c\u5404\u4e2a\u7ec4\u4ef6\u7684\u529f\u80fd\u4ee5\u53ca\u5176\u4e4b\u95f4\u7684\u4ea4\u4e92\u3002 \u603b\u4f53\u6765\u770b\uff0cKube-OVN \u4f5c\u4e3a Kubernetes \u548c OVN \u4e4b\u95f4\u7684\u4e00\u4e2a\u6865\u6881\uff0c\u5c06\u6210\u719f\u7684 SDN \u548c\u4e91\u539f\u751f\u76f8\u7ed3\u5408\u3002 \u8fd9\u610f\u5473\u7740 Kube-OVN \u4e0d\u4ec5\u901a\u8fc7 OVN \u5b9e\u73b0\u4e86 Kubernetes \u4e0b\u7684\u7f51\u7edc\u89c4\u8303\uff0c\u4f8b\u5982 CNI\uff0cService \u548c Networkpolicy\uff0c\u8fd8\u5c06\u5927\u91cf\u7684 SDN \u9886\u57df\u80fd\u529b\u5e26\u5165\u4e91\u539f\u751f\uff0c\u4f8b\u5982\u903b\u8f91\u4ea4\u6362\u673a\uff0c\u903b\u8f91\u8def\u7531\u5668\uff0cVPC\uff0c\u7f51\u5173\uff0cQoS\uff0cACL \u548c\u6d41\u91cf\u955c\u50cf\u3002 \u540c\u65f6 Kube-OVN \u8fd8\u4fdd\u6301\u4e86\u826f\u597d\u7684\u5f00\u653e\u6027\u53ef\u4ee5\u548c\u8bf8\u591a\u6280\u672f\u65b9\u6848\u96c6\u6210\uff0c\u4f8b\u5982 Cilium\uff0cSubmariner\uff0cPrometheus\uff0cKubeVirt \u7b49\u7b49\u3002 \u7ec4\u4ef6\u4ecb\u7ecd \u00b6 Kube-OVN \u7684\u7ec4\u4ef6\u53ef\u4ee5\u5927\u81f4\u5206\u4e3a\u4e09\u7c7b\uff1a \u4e0a\u6e38 OVN/OVS \u7ec4\u4ef6\u3002 \u6838\u5fc3\u63a7\u5236\u5668\u548c Agent\u3002 \u76d1\u63a7\uff0c\u8fd0\u7ef4\u5de5\u5177\u548c\u6269\u5c55\u7ec4\u4ef6\u3002 \u4e0a\u6e38 OVN/OVS \u7ec4\u4ef6 \u00b6 \u8be5\u7c7b\u578b\u7ec4\u4ef6\u6765\u81ea OVN/OVS \u793e\u533a\uff0c\u5e76\u9488\u5bf9 Kube-OVN \u7684\u4f7f\u7528\u573a\u666f\u505a\u4e86\u7279\u5b9a\u4fee\u6539\u3002 OVN/OVS \u672c\u8eab\u662f\u4e00\u5957\u6210\u719f\u7684\u7ba1\u7406\u865a\u673a\u548c\u5bb9\u5668\u7684 SDN \u7cfb\u7edf\uff0c\u6211\u4eec\u5f3a\u70c8\u5efa\u8bae \u5bf9 Kube-OVN \u5b9e\u73b0\u611f\u5174\u8da3\u7684\u7528\u6237\u5148\u53bb\u8bfb\u4e00\u4e0b ovn-architecture(7) \u6765\u4e86\u89e3\u4ec0\u4e48\u662f OVN \u4ee5\u53ca \u5982\u4f55\u548c\u5b83\u8fdb\u884c\u96c6\u6210\u3002Kube-OVN \u4f7f\u7528 OVN \u7684\u5317\u5411\u63a5\u53e3\u521b\u5efa\u548c\u8c03\u6574\u865a\u62df\u7f51\u7edc\uff0c\u5e76\u5c06\u5176\u4e2d\u7684\u7f51\u7edc\u6982\u5ff5\u6620\u5c04\u5230 Kubernetes \u4e4b\u5185\u3002 \u6240\u6709 OVN/OVS \u76f8\u5173\u7ec4\u4ef6\u90fd\u5df2\u6253\u5305\u6210\u5bf9\u5e94\u955c\u50cf\uff0c\u5e76\u53ef\u5728 Kubernetes \u4e2d\u8fd0\u884c\u3002 ovn-central \u00b6 ovn-central Deployment \u8fd0\u884c OVN \u7684\u7ba1\u7406\u5e73\u9762\u7ec4\u4ef6\uff0c\u5305\u62ec ovn-nb , ovn-sb , \u548c ovn-northd \u3002 ovn-nb \uff1a \u4fdd\u5b58\u865a\u62df\u7f51\u7edc\u914d\u7f6e\uff0c\u5e76\u63d0\u4f9b API \u8fdb\u884c\u865a\u62df\u7f51\u7edc\u7ba1\u7406\u3002 kube-ovn-controller \u5c06\u4f1a\u4e3b\u8981\u548c ovn-nb \u8fdb\u884c\u4ea4\u4e92\u914d\u7f6e\u865a\u62df\u7f51\u7edc\u3002 ovn-sb \uff1a \u4fdd\u5b58\u4ece ovn-nb \u7684\u903b\u8f91\u7f51\u7edc\u751f\u6210\u7684\u903b\u8f91\u6d41\u8868\uff0c\u4ee5\u53ca\u5404\u4e2a\u8282\u70b9\u7684\u5b9e\u9645\u7269\u7406\u7f51\u7edc\u72b6\u6001\u3002 ovn-northd \uff1a\u5c06 ovn-nb \u7684\u865a\u62df\u7f51\u7edc\u7ffb\u8bd1\u6210 ovn-sb \u4e2d\u7684\u903b\u8f91\u6d41\u8868\u3002 \u591a\u4e2a ovn-central \u5b9e\u4f8b\u4f1a\u901a\u8fc7 Raft \u534f\u8bae\u540c\u6b65\u6570\u636e\u4fdd\u8bc1\u9ad8\u53ef\u7528\u3002 ovs-ovn \u00b6 ovs-ovn \u4ee5 DaemonSet \u5f62\u5f0f\u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\uff0c\u5728 Pod \u5185\u8fd0\u884c\u4e86 openvswitch , ovsdb , \u548c ovn-controller \u3002\u8fd9\u4e9b\u7ec4\u4ef6\u4f5c\u4e3a ovn-central \u7684 Agent \u5c06\u903b\u8f91\u6d41\u8868\u7ffb\u8bd1\u6210\u771f\u5b9e\u7684\u7f51\u7edc\u914d\u7f6e\u3002 \u6838\u5fc3\u63a7\u5236\u5668\u548c Agent \u00b6 \u8be5\u90e8\u5206\u4e3a Kube-OVN \u7684\u6838\u5fc3\u7ec4\u4ef6\uff0c\u4f5c\u4e3a OVN \u548c Kubernetes \u4e4b\u95f4\u7684\u4e00\u4e2a\u6865\u6881\uff0c\u5c06\u4e24\u4e2a\u7cfb\u7edf\u6253\u901a\u5e76\u5c06\u7f51\u7edc\u6982\u5ff5\u8fdb\u884c\u76f8\u4e92\u8f6c\u6362\u3002 \u5927\u90e8\u5206\u7684\u6838\u5fc3\u529f\u80fd\u90fd\u5728\u8be5\u90e8\u5206\u7ec4\u4ef6\u4e2d\u5b9e\u73b0\u3002 kube-ovn-controller \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a Deployment \u6267\u884c\u6240\u6709 Kubernetes \u5185\u8d44\u6e90\u5230 OVN \u8d44\u6e90\u7684\u7ffb\u8bd1\u5de5\u4f5c\uff0c\u5176\u4f5c\u7528\u76f8\u5f53\u4e8e\u6574\u4e2a Kube-OVN \u7cfb\u7edf\u7684\u63a7\u5236\u5e73\u9762\u3002 kube-ovn-controller \u76d1\u542c\u4e86\u6240\u6709\u548c\u7f51\u7edc\u529f\u80fd\u76f8\u5173\u8d44\u6e90\u7684\u4e8b\u4ef6\uff0c\u5e76\u6839\u636e\u8d44\u6e90\u53d8\u5316\u60c5\u51b5\u66f4\u65b0 OVN \u5185\u7684\u903b\u8f91\u7f51\u7edc\u3002\u4e3b\u8981\u76d1\u542c\u7684\u8d44\u6e90\u5305\u62ec\uff1a Pod\uff0cService\uff0cEndpoint\uff0cNode\uff0cNetworkPolicy\uff0cVPC\uff0cSubnet\uff0cVlan\uff0cProviderNetwork\u3002 \u4ee5 Pod \u4e8b\u4ef6\u4e3a\u4f8b\uff0c kube-ovn-controller \u76d1\u542c\u5230 Pod \u521b\u5efa\u4e8b\u4ef6\u540e\uff0c\u901a\u8fc7\u5185\u7f6e\u7684\u5185\u5b58 IPAM \u529f\u80fd\u5206\u914d\u5730\u5740\uff0c\u5e76\u8c03\u7528 ovn-central \u521b\u5efa \u903b\u8f91\u7aef\u53e3\uff0c\u9759\u6001\u8def\u7531\u548c\u53ef\u80fd\u7684 ACL \u89c4\u5219\u3002\u63a5\u4e0b\u6765 kube-ovn-controller \u5c06\u5206\u914d\u5230\u7684\u5730\u5740\uff0c\u548c\u5b50\u7f51\u4fe1\u606f\u4f8b\u5982 CIDR\uff0c\u7f51\u5173\uff0c\u8def\u7531\u7b49\u4fe1\u606f\u5199\u4f1a\u5230 Pod \u7684 annotation \u4e2d\u3002\u8be5 annotation \u540e\u7eed\u4f1a\u88ab kube-ovn-cni \u8bfb\u53d6\u7528\u6765\u914d\u7f6e\u672c\u5730\u7f51\u7edc\u3002 kube-ovn-cni \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\uff0c\u5b9e\u73b0 CNI \u63a5\u53e3\uff0c\u5e76\u64cd\u4f5c\u672c\u5730\u7684 OVS \u914d\u7f6e\u5355\u673a\u7f51\u7edc\u3002 \u8be5 DaemonSet \u4f1a\u590d\u5236 kube-ovn \u4e8c\u8fdb\u5236\u6587\u4ef6\u5230\u6bcf\u53f0\u673a\u5668\uff0c\u4f5c\u4e3a kubelet \u548c kube-ovn-cni \u4e4b\u95f4\u7684\u4ea4\u4e92\u5de5\u5177\uff0c\u5c06\u76f8\u5e94 CNI \u8bf7\u6c42 \u53d1\u9001\u7ed9 kube-ovn-cni \u6267\u884c\u3002\u8be5\u4e8c\u8fdb\u5236\u6587\u4ef6\u9ed8\u8ba4\u4f1a\u88ab\u590d\u5236\u5230 /opt/cni/bin \u76ee\u5f55\u4e0b\u3002 kube-ovn-cni \u4f1a\u914d\u7f6e\u5177\u4f53\u7684\u7f51\u7edc\u6765\u6267\u884c\u76f8\u5e94\u6d41\u91cf\u64cd\u4f5c\uff0c\u4e3b\u8981\u5de5\u4f5c\u5305\u62ec\uff1a \u914d\u7f6e ovn-controller \u548c vswitchd \u3002 \u5904\u7406 CNI add/del \u8bf7\u6c42\uff1a \u521b\u5efa\u5220\u9664 veth \u5e76\u548c OVS \u7aef\u53e3\u7ed1\u5b9a\u3002 \u914d\u7f6e OVS \u7aef\u53e3\u4fe1\u606f\u3002 \u66f4\u65b0\u5bbf\u4e3b\u673a\u7684 iptables/ipset/route \u7b49\u89c4\u5219\u3002 \u52a8\u6001\u66f4\u65b0\u5bb9\u5668 QoS. \u521b\u5efa\u5e76\u914d\u7f6e ovn0 \u7f51\u5361\u8054\u901a\u5bb9\u5668\u7f51\u7edc\u548c\u4e3b\u673a\u7f51\u7edc\u3002 \u914d\u7f6e\u4e3b\u673a\u7f51\u5361\u6765\u5b9e\u73b0 Vlan/Underlay/EIP \u7b49\u529f\u80fd\u3002 \u52a8\u6001\u914d\u7f6e\u96c6\u7fa4\u4e92\u8054\u7f51\u5173\u3002 \u76d1\u63a7\uff0c\u8fd0\u7ef4\u5de5\u5177\u548c\u6269\u5c55\u7ec4\u4ef6 \u00b6 \u8be5\u90e8\u5206\u7ec4\u4ef6\u4e3b\u8981\u63d0\u4f9b\u76d1\u63a7\uff0c\u8bca\u65ad\uff0c\u8fd0\u7ef4\u64cd\u4f5c\u4ee5\u53ca\u548c\u5916\u90e8\u8fdb\u884c\u5bf9\u63a5\uff0c\u5bf9 Kube-OVN \u7684\u6838\u5fc3\u7f51\u7edc\u80fd\u529b\u8fdb\u884c\u6269\u5c55\uff0c\u5e76\u7b80\u5316\u65e5\u5e38\u8fd0\u7ef4\u64cd\u4f5c\u3002 kube-ovn-speaker \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u7279\u5b9a\u6807\u7b7e\u7684\u8282\u70b9\u4e0a\uff0c\u5bf9\u5916\u53d1\u5e03\u5bb9\u5668\u7f51\u7edc\u7684\u8def\u7531\uff0c\u4f7f\u5f97\u5916\u90e8\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7 Pod IP \u8bbf\u95ee\u5bb9\u5668\u3002 \u66f4\u591a\u76f8\u5173\u4f7f\u7528\u65b9\u5f0f\u8bf7\u53c2\u8003 BGP \u652f\u6301 \u3002 kube-ovn-pinger \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\u6536\u96c6 OVS \u8fd0\u884c\u4fe1\u606f\uff0c\u8282\u70b9\u7f51\u7edc\u8d28\u91cf\uff0c\u7f51\u7edc\u5ef6\u8fdf\u7b49\u4fe1\u606f\uff0c\u6536\u96c6\u7684\u76d1\u63a7\u6307\u6807\u53ef\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002 kube-ovn-monitor \u00b6 \u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a Deployment \u6536\u96c6 OVN \u7684\u8fd0\u884c\u4fe1\u606f\uff0c\u6536\u96c6\u7684\u76d1\u63a7\u6307\u6807\u53ef\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002 kubectl-ko \u00b6 \u8be5\u7ec4\u4ef6\u4e3a kubectl \u63d2\u4ef6\uff0c\u53ef\u4ee5\u5feb\u901f\u8fd0\u884c\u5e38\u89c1\u8fd0\u7ef4\u64cd\u4f5c\uff0c\u66f4\u591a\u4f7f\u7528\u8bf7\u53c2\u8003 kubectl \u63d2\u4ef6\u4f7f\u7528 \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u603b\u4f53\u67b6\u6784"},{"location":"reference/architecture/#_1","text":"\u672c\u6587\u6863\u5c06\u4ecb\u7ecd Kube-OVN \u7684\u603b\u4f53\u67b6\u6784\uff0c\u548c\u5404\u4e2a\u7ec4\u4ef6\u7684\u529f\u80fd\u4ee5\u53ca\u5176\u4e4b\u95f4\u7684\u4ea4\u4e92\u3002 \u603b\u4f53\u6765\u770b\uff0cKube-OVN \u4f5c\u4e3a Kubernetes \u548c OVN \u4e4b\u95f4\u7684\u4e00\u4e2a\u6865\u6881\uff0c\u5c06\u6210\u719f\u7684 SDN \u548c\u4e91\u539f\u751f\u76f8\u7ed3\u5408\u3002 \u8fd9\u610f\u5473\u7740 Kube-OVN \u4e0d\u4ec5\u901a\u8fc7 OVN \u5b9e\u73b0\u4e86 Kubernetes \u4e0b\u7684\u7f51\u7edc\u89c4\u8303\uff0c\u4f8b\u5982 CNI\uff0cService \u548c Networkpolicy\uff0c\u8fd8\u5c06\u5927\u91cf\u7684 SDN \u9886\u57df\u80fd\u529b\u5e26\u5165\u4e91\u539f\u751f\uff0c\u4f8b\u5982\u903b\u8f91\u4ea4\u6362\u673a\uff0c\u903b\u8f91\u8def\u7531\u5668\uff0cVPC\uff0c\u7f51\u5173\uff0cQoS\uff0cACL \u548c\u6d41\u91cf\u955c\u50cf\u3002 \u540c\u65f6 Kube-OVN \u8fd8\u4fdd\u6301\u4e86\u826f\u597d\u7684\u5f00\u653e\u6027\u53ef\u4ee5\u548c\u8bf8\u591a\u6280\u672f\u65b9\u6848\u96c6\u6210\uff0c\u4f8b\u5982 Cilium\uff0cSubmariner\uff0cPrometheus\uff0cKubeVirt \u7b49\u7b49\u3002","title":"\u603b\u4f53\u67b6\u6784"},{"location":"reference/architecture/#_2","text":"Kube-OVN \u7684\u7ec4\u4ef6\u53ef\u4ee5\u5927\u81f4\u5206\u4e3a\u4e09\u7c7b\uff1a \u4e0a\u6e38 OVN/OVS \u7ec4\u4ef6\u3002 \u6838\u5fc3\u63a7\u5236\u5668\u548c Agent\u3002 \u76d1\u63a7\uff0c\u8fd0\u7ef4\u5de5\u5177\u548c\u6269\u5c55\u7ec4\u4ef6\u3002","title":"\u7ec4\u4ef6\u4ecb\u7ecd"},{"location":"reference/architecture/#ovnovs","text":"\u8be5\u7c7b\u578b\u7ec4\u4ef6\u6765\u81ea OVN/OVS \u793e\u533a\uff0c\u5e76\u9488\u5bf9 Kube-OVN \u7684\u4f7f\u7528\u573a\u666f\u505a\u4e86\u7279\u5b9a\u4fee\u6539\u3002 OVN/OVS \u672c\u8eab\u662f\u4e00\u5957\u6210\u719f\u7684\u7ba1\u7406\u865a\u673a\u548c\u5bb9\u5668\u7684 SDN \u7cfb\u7edf\uff0c\u6211\u4eec\u5f3a\u70c8\u5efa\u8bae \u5bf9 Kube-OVN \u5b9e\u73b0\u611f\u5174\u8da3\u7684\u7528\u6237\u5148\u53bb\u8bfb\u4e00\u4e0b ovn-architecture(7) \u6765\u4e86\u89e3\u4ec0\u4e48\u662f OVN \u4ee5\u53ca \u5982\u4f55\u548c\u5b83\u8fdb\u884c\u96c6\u6210\u3002Kube-OVN \u4f7f\u7528 OVN \u7684\u5317\u5411\u63a5\u53e3\u521b\u5efa\u548c\u8c03\u6574\u865a\u62df\u7f51\u7edc\uff0c\u5e76\u5c06\u5176\u4e2d\u7684\u7f51\u7edc\u6982\u5ff5\u6620\u5c04\u5230 Kubernetes \u4e4b\u5185\u3002 \u6240\u6709 OVN/OVS \u76f8\u5173\u7ec4\u4ef6\u90fd\u5df2\u6253\u5305\u6210\u5bf9\u5e94\u955c\u50cf\uff0c\u5e76\u53ef\u5728 Kubernetes \u4e2d\u8fd0\u884c\u3002","title":"\u4e0a\u6e38 OVN/OVS \u7ec4\u4ef6"},{"location":"reference/architecture/#ovn-central","text":"ovn-central Deployment \u8fd0\u884c OVN \u7684\u7ba1\u7406\u5e73\u9762\u7ec4\u4ef6\uff0c\u5305\u62ec ovn-nb , ovn-sb , \u548c ovn-northd \u3002 ovn-nb \uff1a \u4fdd\u5b58\u865a\u62df\u7f51\u7edc\u914d\u7f6e\uff0c\u5e76\u63d0\u4f9b API \u8fdb\u884c\u865a\u62df\u7f51\u7edc\u7ba1\u7406\u3002 kube-ovn-controller \u5c06\u4f1a\u4e3b\u8981\u548c ovn-nb \u8fdb\u884c\u4ea4\u4e92\u914d\u7f6e\u865a\u62df\u7f51\u7edc\u3002 ovn-sb \uff1a \u4fdd\u5b58\u4ece ovn-nb \u7684\u903b\u8f91\u7f51\u7edc\u751f\u6210\u7684\u903b\u8f91\u6d41\u8868\uff0c\u4ee5\u53ca\u5404\u4e2a\u8282\u70b9\u7684\u5b9e\u9645\u7269\u7406\u7f51\u7edc\u72b6\u6001\u3002 ovn-northd \uff1a\u5c06 ovn-nb \u7684\u865a\u62df\u7f51\u7edc\u7ffb\u8bd1\u6210 ovn-sb \u4e2d\u7684\u903b\u8f91\u6d41\u8868\u3002 \u591a\u4e2a ovn-central \u5b9e\u4f8b\u4f1a\u901a\u8fc7 Raft \u534f\u8bae\u540c\u6b65\u6570\u636e\u4fdd\u8bc1\u9ad8\u53ef\u7528\u3002","title":"ovn-central"},{"location":"reference/architecture/#ovs-ovn","text":"ovs-ovn \u4ee5 DaemonSet \u5f62\u5f0f\u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\uff0c\u5728 Pod \u5185\u8fd0\u884c\u4e86 openvswitch , ovsdb , \u548c ovn-controller \u3002\u8fd9\u4e9b\u7ec4\u4ef6\u4f5c\u4e3a ovn-central \u7684 Agent \u5c06\u903b\u8f91\u6d41\u8868\u7ffb\u8bd1\u6210\u771f\u5b9e\u7684\u7f51\u7edc\u914d\u7f6e\u3002","title":"ovs-ovn"},{"location":"reference/architecture/#agent","text":"\u8be5\u90e8\u5206\u4e3a Kube-OVN \u7684\u6838\u5fc3\u7ec4\u4ef6\uff0c\u4f5c\u4e3a OVN \u548c Kubernetes \u4e4b\u95f4\u7684\u4e00\u4e2a\u6865\u6881\uff0c\u5c06\u4e24\u4e2a\u7cfb\u7edf\u6253\u901a\u5e76\u5c06\u7f51\u7edc\u6982\u5ff5\u8fdb\u884c\u76f8\u4e92\u8f6c\u6362\u3002 \u5927\u90e8\u5206\u7684\u6838\u5fc3\u529f\u80fd\u90fd\u5728\u8be5\u90e8\u5206\u7ec4\u4ef6\u4e2d\u5b9e\u73b0\u3002","title":"\u6838\u5fc3\u63a7\u5236\u5668\u548c Agent"},{"location":"reference/architecture/#kube-ovn-controller","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a Deployment \u6267\u884c\u6240\u6709 Kubernetes \u5185\u8d44\u6e90\u5230 OVN \u8d44\u6e90\u7684\u7ffb\u8bd1\u5de5\u4f5c\uff0c\u5176\u4f5c\u7528\u76f8\u5f53\u4e8e\u6574\u4e2a Kube-OVN \u7cfb\u7edf\u7684\u63a7\u5236\u5e73\u9762\u3002 kube-ovn-controller \u76d1\u542c\u4e86\u6240\u6709\u548c\u7f51\u7edc\u529f\u80fd\u76f8\u5173\u8d44\u6e90\u7684\u4e8b\u4ef6\uff0c\u5e76\u6839\u636e\u8d44\u6e90\u53d8\u5316\u60c5\u51b5\u66f4\u65b0 OVN \u5185\u7684\u903b\u8f91\u7f51\u7edc\u3002\u4e3b\u8981\u76d1\u542c\u7684\u8d44\u6e90\u5305\u62ec\uff1a Pod\uff0cService\uff0cEndpoint\uff0cNode\uff0cNetworkPolicy\uff0cVPC\uff0cSubnet\uff0cVlan\uff0cProviderNetwork\u3002 \u4ee5 Pod \u4e8b\u4ef6\u4e3a\u4f8b\uff0c kube-ovn-controller \u76d1\u542c\u5230 Pod \u521b\u5efa\u4e8b\u4ef6\u540e\uff0c\u901a\u8fc7\u5185\u7f6e\u7684\u5185\u5b58 IPAM \u529f\u80fd\u5206\u914d\u5730\u5740\uff0c\u5e76\u8c03\u7528 ovn-central \u521b\u5efa \u903b\u8f91\u7aef\u53e3\uff0c\u9759\u6001\u8def\u7531\u548c\u53ef\u80fd\u7684 ACL \u89c4\u5219\u3002\u63a5\u4e0b\u6765 kube-ovn-controller \u5c06\u5206\u914d\u5230\u7684\u5730\u5740\uff0c\u548c\u5b50\u7f51\u4fe1\u606f\u4f8b\u5982 CIDR\uff0c\u7f51\u5173\uff0c\u8def\u7531\u7b49\u4fe1\u606f\u5199\u4f1a\u5230 Pod \u7684 annotation \u4e2d\u3002\u8be5 annotation \u540e\u7eed\u4f1a\u88ab kube-ovn-cni \u8bfb\u53d6\u7528\u6765\u914d\u7f6e\u672c\u5730\u7f51\u7edc\u3002","title":"kube-ovn-controller"},{"location":"reference/architecture/#kube-ovn-cni","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\uff0c\u5b9e\u73b0 CNI \u63a5\u53e3\uff0c\u5e76\u64cd\u4f5c\u672c\u5730\u7684 OVS \u914d\u7f6e\u5355\u673a\u7f51\u7edc\u3002 \u8be5 DaemonSet \u4f1a\u590d\u5236 kube-ovn \u4e8c\u8fdb\u5236\u6587\u4ef6\u5230\u6bcf\u53f0\u673a\u5668\uff0c\u4f5c\u4e3a kubelet \u548c kube-ovn-cni \u4e4b\u95f4\u7684\u4ea4\u4e92\u5de5\u5177\uff0c\u5c06\u76f8\u5e94 CNI \u8bf7\u6c42 \u53d1\u9001\u7ed9 kube-ovn-cni \u6267\u884c\u3002\u8be5\u4e8c\u8fdb\u5236\u6587\u4ef6\u9ed8\u8ba4\u4f1a\u88ab\u590d\u5236\u5230 /opt/cni/bin \u76ee\u5f55\u4e0b\u3002 kube-ovn-cni \u4f1a\u914d\u7f6e\u5177\u4f53\u7684\u7f51\u7edc\u6765\u6267\u884c\u76f8\u5e94\u6d41\u91cf\u64cd\u4f5c\uff0c\u4e3b\u8981\u5de5\u4f5c\u5305\u62ec\uff1a \u914d\u7f6e ovn-controller \u548c vswitchd \u3002 \u5904\u7406 CNI add/del \u8bf7\u6c42\uff1a \u521b\u5efa\u5220\u9664 veth \u5e76\u548c OVS \u7aef\u53e3\u7ed1\u5b9a\u3002 \u914d\u7f6e OVS \u7aef\u53e3\u4fe1\u606f\u3002 \u66f4\u65b0\u5bbf\u4e3b\u673a\u7684 iptables/ipset/route \u7b49\u89c4\u5219\u3002 \u52a8\u6001\u66f4\u65b0\u5bb9\u5668 QoS. \u521b\u5efa\u5e76\u914d\u7f6e ovn0 \u7f51\u5361\u8054\u901a\u5bb9\u5668\u7f51\u7edc\u548c\u4e3b\u673a\u7f51\u7edc\u3002 \u914d\u7f6e\u4e3b\u673a\u7f51\u5361\u6765\u5b9e\u73b0 Vlan/Underlay/EIP \u7b49\u529f\u80fd\u3002 \u52a8\u6001\u914d\u7f6e\u96c6\u7fa4\u4e92\u8054\u7f51\u5173\u3002","title":"kube-ovn-cni"},{"location":"reference/architecture/#_3","text":"\u8be5\u90e8\u5206\u7ec4\u4ef6\u4e3b\u8981\u63d0\u4f9b\u76d1\u63a7\uff0c\u8bca\u65ad\uff0c\u8fd0\u7ef4\u64cd\u4f5c\u4ee5\u53ca\u548c\u5916\u90e8\u8fdb\u884c\u5bf9\u63a5\uff0c\u5bf9 Kube-OVN \u7684\u6838\u5fc3\u7f51\u7edc\u80fd\u529b\u8fdb\u884c\u6269\u5c55\uff0c\u5e76\u7b80\u5316\u65e5\u5e38\u8fd0\u7ef4\u64cd\u4f5c\u3002","title":"\u76d1\u63a7\uff0c\u8fd0\u7ef4\u5de5\u5177\u548c\u6269\u5c55\u7ec4\u4ef6"},{"location":"reference/architecture/#kube-ovn-speaker","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u7279\u5b9a\u6807\u7b7e\u7684\u8282\u70b9\u4e0a\uff0c\u5bf9\u5916\u53d1\u5e03\u5bb9\u5668\u7f51\u7edc\u7684\u8def\u7531\uff0c\u4f7f\u5f97\u5916\u90e8\u53ef\u4ee5\u76f4\u63a5\u901a\u8fc7 Pod IP \u8bbf\u95ee\u5bb9\u5668\u3002 \u66f4\u591a\u76f8\u5173\u4f7f\u7528\u65b9\u5f0f\u8bf7\u53c2\u8003 BGP \u652f\u6301 \u3002","title":"kube-ovn-speaker"},{"location":"reference/architecture/#kube-ovn-pinger","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a DaemonSet \u8fd0\u884c\u5728\u6bcf\u4e2a\u8282\u70b9\u4e0a\u6536\u96c6 OVS \u8fd0\u884c\u4fe1\u606f\uff0c\u8282\u70b9\u7f51\u7edc\u8d28\u91cf\uff0c\u7f51\u7edc\u5ef6\u8fdf\u7b49\u4fe1\u606f\uff0c\u6536\u96c6\u7684\u76d1\u63a7\u6307\u6807\u53ef\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002","title":"kube-ovn-pinger"},{"location":"reference/architecture/#kube-ovn-monitor","text":"\u8be5\u7ec4\u4ef6\u4e3a\u4e00\u4e2a Deployment \u6536\u96c6 OVN \u7684\u8fd0\u884c\u4fe1\u606f\uff0c\u6536\u96c6\u7684\u76d1\u63a7\u6307\u6807\u53ef\u53c2\u8003 Kube-OVN \u76d1\u63a7\u6307\u6807 \u3002","title":"kube-ovn-monitor"},{"location":"reference/architecture/#kubectl-ko","text":"\u8be5\u7ec4\u4ef6\u4e3a kubectl \u63d2\u4ef6\uff0c\u53ef\u4ee5\u5feb\u901f\u8fd0\u884c\u5e38\u89c1\u8fd0\u7ef4\u64cd\u4f5c\uff0c\u66f4\u591a\u4f7f\u7528\u8bf7\u53c2\u8003 kubectl \u63d2\u4ef6\u4f7f\u7528 \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kubectl-ko"},{"location":"reference/dev-env/","text":"\u5f00\u53d1\u73af\u5883\u6784\u5efa \u00b6 \u73af\u5883\u51c6\u5907 \u00b6 Kube-OVN \u4f7f\u7528 Go 1.20 \u5f00\u53d1\u5e76\u4f7f\u7528 Go Modules \u7ba1\u7406\u4f9d\u8d56\uff0c \u8bf7\u786e\u8ba4\u73af\u5883\u53d8\u91cf GO111MODULE=\"on\" \u3002 gosec \u88ab\u7528\u6765\u626b\u63cf\u4ee3\u7801\u5b89\u5168\u76f8\u5173\u95ee\u9898\uff0c\u9700\u8981\u5728\u5f00\u53d1\u73af\u5883\u5b89\u88c5\uff1a go get github.com/securego/gosec/v2/cmd/gosec \u4e3a\u4e86\u964d\u4f4e\u6700\u7ec8\u751f\u6210\u955c\u50cf\u5927\u5c0f\uff0cKube-OVN \u4f7f\u7528\u4e86\u90e8\u5206 Docker buildx \u8bd5\u9a8c\u7279\u6027\uff0c\u8bf7\u66f4\u65b0 Docker \u81f3\u6700\u65b0\u7248\u672c \u5e76\u5f00\u542f buildx: docker buildx create --use \u6784\u5efa\u955c\u50cf \u00b6 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u4ee3\u7801\uff0c\u5e76\u751f\u6210\u8fd0\u884c Kube-OVN \u6240\u9700\u955c\u50cf\uff1a git clone https://github.com/kubeovn/kube-ovn.git cd kube-ovn make release \u5982\u9700\u6784\u5efa\u5728 ARM \u73af\u5883\u4e0b\u8fd0\u884c\u7684\u955c\u50cf\uff0c\u8bf7\u6267\u884c\u4e0b\u9762\u7684\u547d\u4ee4\uff1a make release-arm \u6784\u5efa base \u955c\u50cf \u00b6 \u5982\u9700\u8981\u66f4\u6539\u64cd\u4f5c\u7cfb\u7edf\u7248\u672c\uff0c\u4f9d\u8d56\u5e93\uff0cOVS/OVN \u4ee3\u7801\u7b49\uff0c\u9700\u8981\u5bf9 base \u955c\u50cf\u8fdb\u884c\u91cd\u65b0\u6784\u5efa\u3002 base \u955c\u50cf\u4f7f\u7528\u7684 Dockerfile \u4e3a dist/images/Dockerfile.base \u3002 \u6784\u5efa\u65b9\u6cd5\uff1a # build x86 base image make base-amd64 # build arm base image make base-arm64 \u8fd0\u884c E2E \u00b6 Kube-OVN \u4f7f\u7528 KIND \u6784\u5efa\u672c\u5730 Kubernetes \u96c6\u7fa4\uff0c j2cli \u6e32\u67d3\u6a21\u677f\uff0c Ginkgo \u6765\u8fd0\u884c\u6d4b\u8bd5\u4ee3\u7801\u3002\u8bf7\u53c2\u8003\u76f8\u5173\u6587\u6863\u8fdb\u884c\u4f9d\u8d56\u5b89\u88c5\u3002 \u672c\u5730\u6267\u884c E2E \u6d4b\u8bd5\uff1a make kind-init make kind-install make e2e \u5982\u9700\u8fd0\u884c Underlay E2E \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install-underlay make e2e-underlay-single-nic \u5982\u9700\u8fd0\u884c ovn vpc nat gw eip, fip, snat, dnat \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make ovn-vpc-nat-gw-conformance-e2e \u5982\u9700\u8fd0\u884c iptables vpc nat gw eip, fip, snat, dnat \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make kind-install-vpc-nat-gw make iptables-vpc-nat-gw-conformance-e2e \u5982\u9700\u8fd0\u884c loadbalancer service \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make kind-install-lb-svc make kube-ovn-lb-svc-conformance-e2e \u5982\u9700\u6e05\u7406\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-clean \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5f00\u53d1\u73af\u5883\u6784\u5efa"},{"location":"reference/dev-env/#_1","text":"","title":"\u5f00\u53d1\u73af\u5883\u6784\u5efa"},{"location":"reference/dev-env/#_2","text":"Kube-OVN \u4f7f\u7528 Go 1.20 \u5f00\u53d1\u5e76\u4f7f\u7528 Go Modules \u7ba1\u7406\u4f9d\u8d56\uff0c \u8bf7\u786e\u8ba4\u73af\u5883\u53d8\u91cf GO111MODULE=\"on\" \u3002 gosec \u88ab\u7528\u6765\u626b\u63cf\u4ee3\u7801\u5b89\u5168\u76f8\u5173\u95ee\u9898\uff0c\u9700\u8981\u5728\u5f00\u53d1\u73af\u5883\u5b89\u88c5\uff1a go get github.com/securego/gosec/v2/cmd/gosec \u4e3a\u4e86\u964d\u4f4e\u6700\u7ec8\u751f\u6210\u955c\u50cf\u5927\u5c0f\uff0cKube-OVN \u4f7f\u7528\u4e86\u90e8\u5206 Docker buildx \u8bd5\u9a8c\u7279\u6027\uff0c\u8bf7\u66f4\u65b0 Docker \u81f3\u6700\u65b0\u7248\u672c \u5e76\u5f00\u542f buildx: docker buildx create --use","title":"\u73af\u5883\u51c6\u5907"},{"location":"reference/dev-env/#_3","text":"\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u4ee3\u7801\uff0c\u5e76\u751f\u6210\u8fd0\u884c Kube-OVN \u6240\u9700\u955c\u50cf\uff1a git clone https://github.com/kubeovn/kube-ovn.git cd kube-ovn make release \u5982\u9700\u6784\u5efa\u5728 ARM \u73af\u5883\u4e0b\u8fd0\u884c\u7684\u955c\u50cf\uff0c\u8bf7\u6267\u884c\u4e0b\u9762\u7684\u547d\u4ee4\uff1a make release-arm","title":"\u6784\u5efa\u955c\u50cf"},{"location":"reference/dev-env/#base","text":"\u5982\u9700\u8981\u66f4\u6539\u64cd\u4f5c\u7cfb\u7edf\u7248\u672c\uff0c\u4f9d\u8d56\u5e93\uff0cOVS/OVN \u4ee3\u7801\u7b49\uff0c\u9700\u8981\u5bf9 base \u955c\u50cf\u8fdb\u884c\u91cd\u65b0\u6784\u5efa\u3002 base \u955c\u50cf\u4f7f\u7528\u7684 Dockerfile \u4e3a dist/images/Dockerfile.base \u3002 \u6784\u5efa\u65b9\u6cd5\uff1a # build x86 base image make base-amd64 # build arm base image make base-arm64","title":"\u6784\u5efa base \u955c\u50cf"},{"location":"reference/dev-env/#e2e","text":"Kube-OVN \u4f7f\u7528 KIND \u6784\u5efa\u672c\u5730 Kubernetes \u96c6\u7fa4\uff0c j2cli \u6e32\u67d3\u6a21\u677f\uff0c Ginkgo \u6765\u8fd0\u884c\u6d4b\u8bd5\u4ee3\u7801\u3002\u8bf7\u53c2\u8003\u76f8\u5173\u6587\u6863\u8fdb\u884c\u4f9d\u8d56\u5b89\u88c5\u3002 \u672c\u5730\u6267\u884c E2E \u6d4b\u8bd5\uff1a make kind-init make kind-install make e2e \u5982\u9700\u8fd0\u884c Underlay E2E \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install-underlay make e2e-underlay-single-nic \u5982\u9700\u8fd0\u884c ovn vpc nat gw eip, fip, snat, dnat \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make ovn-vpc-nat-gw-conformance-e2e \u5982\u9700\u8fd0\u884c iptables vpc nat gw eip, fip, snat, dnat \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make kind-install-vpc-nat-gw make iptables-vpc-nat-gw-conformance-e2e \u5982\u9700\u8fd0\u884c loadbalancer service \u6d4b\u8bd5\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-init make kind-install make kind-install-lb-svc make kube-ovn-lb-svc-conformance-e2e \u5982\u9700\u6e05\u7406\uff0c\u6267\u884c\u4e0b\u5217\u547d\u4ee4\uff1a make kind-clean \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u8fd0\u884c E2E"},{"location":"reference/document-convention/","text":"\u6587\u6863\u89c4\u8303 \u00b6 \u4e3a\u4e86\u4fdd\u8bc1\u6587\u6863\u98ce\u683c\u4e00\u81f4\uff0c\u8bf7\u5728\u63d0\u4ea4\u6587\u6863\u65f6\u9075\u5faa\u4e0b\u5217\u7684\u98ce\u683c\u89c4\u8303\u3002 \u6807\u70b9 \u00b6 \u4e2d\u6587\u6587\u6863\u4e2d\u6587\u672c\u5185\u5bb9\u6240\u6709\u6807\u70b9\u5e94\u4f7f\u7528\u4e2d\u6587\u683c\u5f0f\u6807\u70b9\uff0c\u82f1\u6587\u6587\u6863\u4e2d\u6240\u6709\u6587\u672c\u5185\u5bb9\u4e2d\u5e94\u4f7f\u7528\u82f1\u6587\u6807\u70b9\u3002 Bad Good \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c,\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528,\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc. \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002 \u82f1\u6587\u6570\u5b57\u548c\u4e2d\u6587\u5e94\u8be5\u7528\u7a7a\u683c\u8fdb\u884c\u5206\u9694\u3002 Bad Good Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 \u793a\u4f8b\u5185\u5bb9\u5e94\u8be5\u4ee5 \uff1a \u5f00\u542f\uff0c\u5176\u4ed6\u53e5\u5c3e\u9700\u8981\u7528 \u3002 \u7ed3\u675f\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\u3002 wget 127 .0.0.1 \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget 127 .0.0.1 \u4ee3\u7801\u5757 \u00b6 yaml \u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a yaml\u3002 Bad Good ```` apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` ````yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` \u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a bash\u3002 Bad Good ```` wget 127.0.0.1 ```` ````bash wget 127.0.0.1 ```` \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u5305\u542b\u8f93\u51fa\u5185\u5bb9\uff0c\u5219\u6240\u6267\u884c\u547d\u4ee4\u9700\u8981\u4ee5 # \u5f00\u59cb\uff0c\u4ee5\u533a\u5206\u8f93\u5165\u4e0e\u8f93\u51fa\u3002 Bad Good oilbeater@macdeMac-3 ~ ping 114 .114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms # ping 114.114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u53ea\u5305\u542b\u6267\u884c\u547d\u4ee4\uff0c\u6ca1\u6709\u8f93\u51fa\u7ed3\u679c\uff0c\u5219\u591a\u6761\u547d\u4ee4\u65e0\u9700 # \u5f00\u59cb\u3002 Bad Good # mv /etc/origin/ovn/ovnnb_db.db /tmp # mv /etc/origin/ovn/ovnsb_db.db /tmp mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp \u94fe\u63a5 \u00b6 \u7ad9\u5185\u94fe\u63a5\u4f7f\u7528\u5bf9\u5e94 md \u6587\u4ef6\u8def\u5f84\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](http://kubeovn.github.io/prepare)\u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](./prepare.md)\u3002 Bad Good \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io )\u3002 \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io ){: target=\"_blank\" }\u3002 \u7a7a\u884c \u00b6 \u4e0d\u540c\u903b\u8f91\u5757\uff0c\u4f8b\u5982\u6807\u9898\u548c\u6587\u672c\uff0c\u6587\u672c\u548c\u4ee3\u7801\uff0c\u6587\u672c\u548c\u7f16\u53f7\u4e4b\u95f4\u9700\u8981\u7528\u7a7a\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0d\u540c\u903b\u8f91\u5757\u4e4b\u95f4\u53ea\u4f7f\u7528 \u4e00\u4e2a \u7a7a\u884c\u8fdb\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6587\u6863\u89c4\u8303"},{"location":"reference/document-convention/#_1","text":"\u4e3a\u4e86\u4fdd\u8bc1\u6587\u6863\u98ce\u683c\u4e00\u81f4\uff0c\u8bf7\u5728\u63d0\u4ea4\u6587\u6863\u65f6\u9075\u5faa\u4e0b\u5217\u7684\u98ce\u683c\u89c4\u8303\u3002","title":"\u6587\u6863\u89c4\u8303"},{"location":"reference/document-convention/#_2","text":"\u4e2d\u6587\u6587\u6863\u4e2d\u6587\u672c\u5185\u5bb9\u6240\u6709\u6807\u70b9\u5e94\u4f7f\u7528\u4e2d\u6587\u683c\u5f0f\u6807\u70b9\uff0c\u82f1\u6587\u6587\u6863\u4e2d\u6240\u6709\u6587\u672c\u5185\u5bb9\u4e2d\u5e94\u4f7f\u7528\u82f1\u6587\u6807\u70b9\u3002 Bad Good \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c,\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528,\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc. \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002 \u82f1\u6587\u6570\u5b57\u548c\u4e2d\u6587\u5e94\u8be5\u7528\u7a7a\u683c\u8fdb\u884c\u5206\u9694\u3002 Bad Good Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 \u793a\u4f8b\u5185\u5bb9\u5e94\u8be5\u4ee5 \uff1a \u5f00\u542f\uff0c\u5176\u4ed6\u53e5\u5c3e\u9700\u8981\u7528 \u3002 \u7ed3\u675f\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\u3002 wget 127 .0.0.1 \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget 127 .0.0.1","title":"\u6807\u70b9"},{"location":"reference/document-convention/#_3","text":"yaml \u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a yaml\u3002 Bad Good ```` apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` ````yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` \u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a bash\u3002 Bad Good ```` wget 127.0.0.1 ```` ````bash wget 127.0.0.1 ```` \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u5305\u542b\u8f93\u51fa\u5185\u5bb9\uff0c\u5219\u6240\u6267\u884c\u547d\u4ee4\u9700\u8981\u4ee5 # \u5f00\u59cb\uff0c\u4ee5\u533a\u5206\u8f93\u5165\u4e0e\u8f93\u51fa\u3002 Bad Good oilbeater@macdeMac-3 ~ ping 114 .114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms # ping 114.114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u53ea\u5305\u542b\u6267\u884c\u547d\u4ee4\uff0c\u6ca1\u6709\u8f93\u51fa\u7ed3\u679c\uff0c\u5219\u591a\u6761\u547d\u4ee4\u65e0\u9700 # \u5f00\u59cb\u3002 Bad Good # mv /etc/origin/ovn/ovnnb_db.db /tmp # mv /etc/origin/ovn/ovnsb_db.db /tmp mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp","title":"\u4ee3\u7801\u5757"},{"location":"reference/document-convention/#_4","text":"\u7ad9\u5185\u94fe\u63a5\u4f7f\u7528\u5bf9\u5e94 md \u6587\u4ef6\u8def\u5f84\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](http://kubeovn.github.io/prepare)\u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](./prepare.md)\u3002 Bad Good \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io )\u3002 \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io ){: target=\"_blank\" }\u3002","title":"\u94fe\u63a5"},{"location":"reference/document-convention/#_5","text":"\u4e0d\u540c\u903b\u8f91\u5757\uff0c\u4f8b\u5982\u6807\u9898\u548c\u6587\u672c\uff0c\u6587\u672c\u548c\u4ee3\u7801\uff0c\u6587\u672c\u548c\u7f16\u53f7\u4e4b\u95f4\u9700\u8981\u7528\u7a7a\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0d\u540c\u903b\u8f91\u5757\u4e4b\u95f4\u53ea\u4f7f\u7528 \u4e00\u4e2a \u7a7a\u884c\u8fdb\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7a7a\u884c"},{"location":"reference/feature-stage/","text":"\u529f\u80fd\u6210\u719f\u5ea6 \u00b6 \u5728 Kube-OVN \u4e2d\u6839\u636e\u529f\u80fd\u4f7f\u7528\u5ea6\uff0c\u6587\u6863\u5b8c\u5584\u7a0b\u5ea6\u548c\u6d4b\u8bd5\u8986\u76d6\u7a0b\u5ea6\u5c06\u529f\u80fd\u6210\u719f\u5ea6\u5206\u4e3a Alpha \uff0c Beta \u548c GA \u4e09\u4e2a\u9636\u6bb5\u3002 \u6210\u719f\u5ea6\u5b9a\u4e49 \u00b6 \u5bf9\u4e8e Alpha \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6ca1\u6709\u5b8c\u6574\u7684\u6587\u6863\u548c\u5b8c\u5584\u7684\u6d4b\u8bd5\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u672a\u6765\u53ef\u80fd\u4f1a\u53d1\u751f\u53d8\u5316\u751a\u81f3\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd API \u4e0d\u4fdd\u8bc1\u7a33\u5b9a\uff0c\u53ef\u80fd\u4f1a\u88ab\u79fb\u9664\u3002 \u8be5\u529f\u80fd\u7684\u793e\u533a\u652f\u6301\u4f18\u5148\u7ea7\u8f83\u4f4e\uff0c\u4e14\u65e0\u6cd5\u4fdd\u8bc1\u957f\u671f\u652f\u6301\u3002 \u7531\u4e8e\u529f\u80fd\u7a33\u5b9a\u6027\u548c\u957f\u671f\u652f\u6301\u65e0\u6cd5\u4fdd\u8bc1\uff0c\u53ef\u4ee5\u8fdb\u884c\u6d4b\u8bd5\u9a8c\u8bc1\uff0c\u4f46\u4e0d\u63a8\u8350\u751f\u4ea7\u4f7f\u7528\u3002 \u5bf9\u4e8e Beta \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6709\u90e8\u5206\u6587\u6863\u548c\u6d4b\u8bd5\uff0c\u4f46\u662f\u4e0d\u4fdd\u8bc1\u5b8c\u6574\u7684\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u672a\u6765\u53ef\u80fd\u53d1\u751f\u53d8\u5316\uff0c\u5347\u7ea7\u53ef\u80fd\u4f1a\u5f71\u54cd\u7f51\u7edc\uff0c\u4f46\u4e0d\u4f1a\u88ab\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd API \u672a\u6765\u53ef\u80fd\u4f1a\u53d1\u751f\u53d8\u5316\uff0c\u5b57\u6bb5\u53ef\u80fd\u4f1a\u8fdb\u884c\u8c03\u6574\uff0c\u4f46\u4e0d\u4f1a\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd\u4f1a\u5f97\u5230\u793e\u533a\u7684\u957f\u671f\u652f\u6301\u3002 \u7531\u4e8e\u529f\u80fd\u4f1a\u5f97\u5230\u957f\u671f\u652f\u6301\uff0c\u53ef\u4ee5\u5728\u975e\u5173\u952e\u4e1a\u52a1\u4e0a\u8fdb\u884c\u4f7f\u7528\uff0c\u4f46\u662f\u7531\u4e8e\u529f\u80fd\u548c API \u5b58\u5728\u53d8\u5316\u7684\u53ef\u80fd\uff0c\u53ef\u80fd\u4f1a\u5728\u5347\u7ea7\u4e2d\u51fa\u73b0\u4e2d\u65ad\uff0c\u4e0d\u63a8\u8350\u5728\u5173\u952e\u751f\u4ea7\u4e1a\u52a1\u4e0a\u4f7f\u7528\u3002 \u5bf9\u4e8e GA \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6709\u5b8c\u6574\u7684\u6587\u6863\u548c\u6d4b\u8bd5\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u4f1a\u4fdd\u6301\u7a33\u5b9a\uff0c\u5347\u7ea7\u4f1a\u4fdd\u8bc1\u5e73\u6ed1\u3002 \u8be5\u529f\u80fd API \u4e0d\u4f1a\u53d1\u751f\u7834\u574f\u6027\u53d8\u5316\u3002 \u8be5\u529f\u80fd\u4f1a\u5f97\u5230\u793e\u533a\u9ad8\u4f18\u5148\u7ea7\u652f\u6301\uff0c\u5e76\u4f1a\u4fdd\u8bc1\u957f\u671f\u652f\u6301\u3002 \u6210\u719f\u5ea6\u5217\u8868 \u00b6 \u672c\u5217\u8868\u7edf\u8ba1\u4ece v1.8 \u7248\u672c\u4e2d\u5305\u542b\u7684\u529f\u80fd\u5bf9\u5e94\u6210\u719f\u5ea6\u3002 \u529f\u80fd \u9ed8\u8ba4\u5f00\u542f \u72b6\u6001 \u5f00\u59cb\uff08Since\uff09 \u7ed3\u675f\uff08Until\uff09 Namespaced Subnet true GA 1.8 \u5206\u5e03\u5f0f\u7f51\u5173 true GA 1.8 \u4e3b\u4ece\u6a21\u5f0f\u96c6\u4e2d\u5f0f\u7f51\u5173 true GA 1.8 ECMP \u6a21\u5f0f\u96c6\u4e2d\u5f0f\u7f51\u5173 false Beta 1.8 \u5b50\u7f51 ACL true Alpha 1.9 \u5b50\u7f51\u9694\u79bb (\u672a\u6765\u4f1a\u548c\u5b50\u7f51 ACL \u5408\u5e76) true Beta 1.8 Underlay \u5b50\u7f51 true GA 1.8 \u591a\u7f51\u5361\u7ba1\u7406 true Beta 1.8 \u5b50\u7f51 DHCP false Alpha 1.10 \u5b50\u7f51\u8bbe\u7f6e\u5916\u90e8\u7f51\u5173 false Alpha 1.8 \u4f7f\u7528 OVN-IC \u8fdb\u884c\u96c6\u7fa4\u4e92\u8054 false Beta 1.8 \u4f7f\u7528 Submariner \u8fdb\u884c\u96c6\u7fa4\u4e92\u8054 false Alpha 1.9 \u5b50\u7f51 VIP \u9884\u7559 true Alpha 1.10 \u521b\u5efa\u81ea\u5b9a\u4e49 VPC true Beta 1.8 \u81ea\u5b9a\u4e49 VPC \u6d6e\u52a8 IP/SNAT/DNAT true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u9759\u6001\u8def\u7531 true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u7b56\u7565\u8def\u7531 true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u5b89\u5168\u7ec4 true Alpha 1.10 \u5bb9\u5668\u6700\u5927\u5e26\u5bbd QoS true GA 1.8 linux-netem QoS true Alpha 1.9 Prometheus \u96c6\u6210 false GA 1.8 Grafana \u96c6\u6210 false GA 1.8 \u53cc\u6808\u7f51\u7edc false GA 1.8 \u9ed8\u8ba4 VPC EIP/SNAT false Beta 1.8 \u6d41\u91cf\u955c\u50cf false GA 1.8 NetworkPolicy true Beta 1.8 Webhook false Alpha 1.10 \u6027\u80fd\u8c03\u4f18 false Beta 1.8 Overlay \u5b50\u7f51\u9759\u6001\u8def\u7531\u5bf9\u5916\u66b4\u9732 false Alpha 1.8 Overlay \u5b50\u7f51 BGP \u5bf9\u5916\u66b4\u9732 false Alpha 1.9 Cilium \u96c6\u6210 false Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u4e92\u8054 false Alpha 1.10 Mellanox Offload false Alpha 1.8 \u82af\u542f\u6e90 Offload false Alpha 1.10 Windows \u652f\u6301 false Alpha 1.10 DPDK \u652f\u6301 false Alpha 1.10 OpenStack \u96c6\u6210 false Alpha 1.9 \u5355\u4e2a Pod \u56fa\u5b9a IP/Mac true GA 1.8 Workload \u56fa\u5b9a IP true GA 1.8 StatefulSet \u56fa\u5b9a IP true GA 1.8 VM \u56fa\u5b9a IP false Beta 1.9 \u9ed8\u8ba4 VPC Load Balancer \u7c7b\u578b Service false Alpha 1.11 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 false Alpha 1.11 \u81ea\u5b9a\u4e49 VPC DNS false Alpha 1.11 Underlay \u548c Overlay \u4e92\u901a false Alpha 1.11 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u529f\u80fd\u6210\u719f\u5ea6"},{"location":"reference/feature-stage/#_1","text":"\u5728 Kube-OVN \u4e2d\u6839\u636e\u529f\u80fd\u4f7f\u7528\u5ea6\uff0c\u6587\u6863\u5b8c\u5584\u7a0b\u5ea6\u548c\u6d4b\u8bd5\u8986\u76d6\u7a0b\u5ea6\u5c06\u529f\u80fd\u6210\u719f\u5ea6\u5206\u4e3a Alpha \uff0c Beta \u548c GA \u4e09\u4e2a\u9636\u6bb5\u3002","title":"\u529f\u80fd\u6210\u719f\u5ea6"},{"location":"reference/feature-stage/#_2","text":"\u5bf9\u4e8e Alpha \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6ca1\u6709\u5b8c\u6574\u7684\u6587\u6863\u548c\u5b8c\u5584\u7684\u6d4b\u8bd5\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u672a\u6765\u53ef\u80fd\u4f1a\u53d1\u751f\u53d8\u5316\u751a\u81f3\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd API \u4e0d\u4fdd\u8bc1\u7a33\u5b9a\uff0c\u53ef\u80fd\u4f1a\u88ab\u79fb\u9664\u3002 \u8be5\u529f\u80fd\u7684\u793e\u533a\u652f\u6301\u4f18\u5148\u7ea7\u8f83\u4f4e\uff0c\u4e14\u65e0\u6cd5\u4fdd\u8bc1\u957f\u671f\u652f\u6301\u3002 \u7531\u4e8e\u529f\u80fd\u7a33\u5b9a\u6027\u548c\u957f\u671f\u652f\u6301\u65e0\u6cd5\u4fdd\u8bc1\uff0c\u53ef\u4ee5\u8fdb\u884c\u6d4b\u8bd5\u9a8c\u8bc1\uff0c\u4f46\u4e0d\u63a8\u8350\u751f\u4ea7\u4f7f\u7528\u3002 \u5bf9\u4e8e Beta \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6709\u90e8\u5206\u6587\u6863\u548c\u6d4b\u8bd5\uff0c\u4f46\u662f\u4e0d\u4fdd\u8bc1\u5b8c\u6574\u7684\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u672a\u6765\u53ef\u80fd\u53d1\u751f\u53d8\u5316\uff0c\u5347\u7ea7\u53ef\u80fd\u4f1a\u5f71\u54cd\u7f51\u7edc\uff0c\u4f46\u4e0d\u4f1a\u88ab\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd API \u672a\u6765\u53ef\u80fd\u4f1a\u53d1\u751f\u53d8\u5316\uff0c\u5b57\u6bb5\u53ef\u80fd\u4f1a\u8fdb\u884c\u8c03\u6574\uff0c\u4f46\u4e0d\u4f1a\u6574\u4f53\u79fb\u9664\u3002 \u8be5\u529f\u80fd\u4f1a\u5f97\u5230\u793e\u533a\u7684\u957f\u671f\u652f\u6301\u3002 \u7531\u4e8e\u529f\u80fd\u4f1a\u5f97\u5230\u957f\u671f\u652f\u6301\uff0c\u53ef\u4ee5\u5728\u975e\u5173\u952e\u4e1a\u52a1\u4e0a\u8fdb\u884c\u4f7f\u7528\uff0c\u4f46\u662f\u7531\u4e8e\u529f\u80fd\u548c API \u5b58\u5728\u53d8\u5316\u7684\u53ef\u80fd\uff0c\u53ef\u80fd\u4f1a\u5728\u5347\u7ea7\u4e2d\u51fa\u73b0\u4e2d\u65ad\uff0c\u4e0d\u63a8\u8350\u5728\u5173\u952e\u751f\u4ea7\u4e1a\u52a1\u4e0a\u4f7f\u7528\u3002 \u5bf9\u4e8e GA \u529f\u80fd\uff1a \u8be5\u529f\u80fd\u6709\u5b8c\u6574\u7684\u6587\u6863\u548c\u6d4b\u8bd5\u8986\u76d6\u3002 \u8be5\u529f\u80fd\u4f1a\u4fdd\u6301\u7a33\u5b9a\uff0c\u5347\u7ea7\u4f1a\u4fdd\u8bc1\u5e73\u6ed1\u3002 \u8be5\u529f\u80fd API \u4e0d\u4f1a\u53d1\u751f\u7834\u574f\u6027\u53d8\u5316\u3002 \u8be5\u529f\u80fd\u4f1a\u5f97\u5230\u793e\u533a\u9ad8\u4f18\u5148\u7ea7\u652f\u6301\uff0c\u5e76\u4f1a\u4fdd\u8bc1\u957f\u671f\u652f\u6301\u3002","title":"\u6210\u719f\u5ea6\u5b9a\u4e49"},{"location":"reference/feature-stage/#_3","text":"\u672c\u5217\u8868\u7edf\u8ba1\u4ece v1.8 \u7248\u672c\u4e2d\u5305\u542b\u7684\u529f\u80fd\u5bf9\u5e94\u6210\u719f\u5ea6\u3002 \u529f\u80fd \u9ed8\u8ba4\u5f00\u542f \u72b6\u6001 \u5f00\u59cb\uff08Since\uff09 \u7ed3\u675f\uff08Until\uff09 Namespaced Subnet true GA 1.8 \u5206\u5e03\u5f0f\u7f51\u5173 true GA 1.8 \u4e3b\u4ece\u6a21\u5f0f\u96c6\u4e2d\u5f0f\u7f51\u5173 true GA 1.8 ECMP \u6a21\u5f0f\u96c6\u4e2d\u5f0f\u7f51\u5173 false Beta 1.8 \u5b50\u7f51 ACL true Alpha 1.9 \u5b50\u7f51\u9694\u79bb (\u672a\u6765\u4f1a\u548c\u5b50\u7f51 ACL \u5408\u5e76) true Beta 1.8 Underlay \u5b50\u7f51 true GA 1.8 \u591a\u7f51\u5361\u7ba1\u7406 true Beta 1.8 \u5b50\u7f51 DHCP false Alpha 1.10 \u5b50\u7f51\u8bbe\u7f6e\u5916\u90e8\u7f51\u5173 false Alpha 1.8 \u4f7f\u7528 OVN-IC \u8fdb\u884c\u96c6\u7fa4\u4e92\u8054 false Beta 1.8 \u4f7f\u7528 Submariner \u8fdb\u884c\u96c6\u7fa4\u4e92\u8054 false Alpha 1.9 \u5b50\u7f51 VIP \u9884\u7559 true Alpha 1.10 \u521b\u5efa\u81ea\u5b9a\u4e49 VPC true Beta 1.8 \u81ea\u5b9a\u4e49 VPC \u6d6e\u52a8 IP/SNAT/DNAT true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u9759\u6001\u8def\u7531 true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u7b56\u7565\u8def\u7531 true Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u5b89\u5168\u7ec4 true Alpha 1.10 \u5bb9\u5668\u6700\u5927\u5e26\u5bbd QoS true GA 1.8 linux-netem QoS true Alpha 1.9 Prometheus \u96c6\u6210 false GA 1.8 Grafana \u96c6\u6210 false GA 1.8 \u53cc\u6808\u7f51\u7edc false GA 1.8 \u9ed8\u8ba4 VPC EIP/SNAT false Beta 1.8 \u6d41\u91cf\u955c\u50cf false GA 1.8 NetworkPolicy true Beta 1.8 Webhook false Alpha 1.10 \u6027\u80fd\u8c03\u4f18 false Beta 1.8 Overlay \u5b50\u7f51\u9759\u6001\u8def\u7531\u5bf9\u5916\u66b4\u9732 false Alpha 1.8 Overlay \u5b50\u7f51 BGP \u5bf9\u5916\u66b4\u9732 false Alpha 1.9 Cilium \u96c6\u6210 false Alpha 1.10 \u81ea\u5b9a\u4e49 VPC \u4e92\u8054 false Alpha 1.10 Mellanox Offload false Alpha 1.8 \u82af\u542f\u6e90 Offload false Alpha 1.10 Windows \u652f\u6301 false Alpha 1.10 DPDK \u652f\u6301 false Alpha 1.10 OpenStack \u96c6\u6210 false Alpha 1.9 \u5355\u4e2a Pod \u56fa\u5b9a IP/Mac true GA 1.8 Workload \u56fa\u5b9a IP true GA 1.8 StatefulSet \u56fa\u5b9a IP true GA 1.8 VM \u56fa\u5b9a IP false Beta 1.9 \u9ed8\u8ba4 VPC Load Balancer \u7c7b\u578b Service false Alpha 1.11 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 false Alpha 1.11 \u81ea\u5b9a\u4e49 VPC DNS false Alpha 1.11 Underlay \u548c Overlay \u4e92\u901a false Alpha 1.11 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6210\u719f\u5ea6\u5217\u8868"},{"location":"reference/iptables-rules/","text":"Iptables \u89c4\u5219 \u00b6 Kube-OVN \u4f7f\u7528 ipset \u53ca iptables \u8f85\u52a9\u5b9e\u73b0\u9ed8\u8ba4 VPC \u4e0b\u5bb9\u5668\u7f51\u7edc\uff08Overlay\uff09\u7f51\u5173 NAT \u7684\u529f\u80fd\u3002 \u4f7f\u7528\u7684 ipset \u5982\u4e0b\u8868\u6240\u793a\uff1a \u540d\u79f0\uff08IPv4/IPv6\uff09 \u7c7b\u578b \u5b58\u50a8\u5bf9\u8c61 ovn40services/ovn60services hash:net Service \u7f51\u6bb5 ovn40subnets/ovn60subnets hash:net Overlay \u5b50\u7f51\u7f51\u6bb5\u4ee5\u53ca NodeLocal DNS IP \u5730\u5740 ovn40subnets-nat/ovn60subnets-nat hash:net \u5f00\u542f NatOutgoing \u7684 Overlay \u5b50\u7f51\u7f51\u6bb5 ovn40subnets-distributed-gw/ovn60subnets-distributed-gw hash:net \u5f00\u542f\u5206\u5e03\u5f0f\u7f51\u5173\u7684 Overlay \u5b50\u7f51\u7f51\u6bb5 ovn40other-node/ovn60other-node hash:net \u5176\u5b83\u8282\u70b9\u7684\u5185\u90e8 IP \u5730\u5740 ovn40local-pod-ip-nat/ovn60local-pod-ip-nat hash:ip \u5df2\u5f03\u7528 ovn40subnets-nat-policy hash:net \u914d\u7f6e\u4e86 natOutgoingPolicyRules \u7684\u6240\u6709\u5b50\u7f51\u7f51\u6bb5 ovn40natpr-418e79269dc5-dst hash:net natOutgoingPolicyRules \u4e2d rule \u5bf9\u5e94\u7684 dstIPs ovn40natpr-418e79269dc5-src hash:net natOutgoingPolicyRules \u4e2d rule \u5bf9\u5e94\u7684 srcIPs \u4f7f\u7528\u7684 iptables \u89c4\u5219\uff08IPv4\uff09\u5982\u4e0b\u8868\u6240\u793a\uff1a \u8868 \u94fe \u89c4\u5219 \u7528\u9014 \u5907\u6ce8 filter INPUT -m set --match-set ovn40services src -j ACCEPT \u5141\u8bb8 k8s Service \u548c Pod \u76f8\u5173\u6d41\u91cf\u901a\u8fc7 -- filter INPUT -m set --match-set ovn40services dst -j ACCEPT \u540c\u4e0a -- filter INPUT -m set --match-set ovn40subnets src -j ACCEPT \u540c\u4e0a -- filter INPUT -m set --match-set ovn40subnets dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40services src -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40services dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40subnets src -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40subnets dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -s 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" \u7528\u4e8e\u8ba1\u6570\u4ece subnet \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u7684\u62a5\u6587 10.16.0.0/16 \u4e3a subnet \u7684 cidr \uff0ccomment \u4e2d\u9017\u53f7\u524d\u9762\u7684 ovn-subnet-gateway \u7528\u4e8e\u6807\u8bc6\u8be5 iptables \u89c4\u5219\u7528\u4e8e subnet \u51fa\u5165\u7f51\u5173\u62a5\u6587\u8ba1\u6570\uff0c\u9017\u53f7\u540e\u9762 ovn-default \u662f\u8be5 subnet \u7684\u540d\u5b57 filter FORWARD -d 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" \u7528\u4e8e\u8ba1\u6570\u4ece\u5916\u90e8\u7f51\u7edc\u8bbf\u95ee subnet \u7684\u62a5\u6587 \u540c\u4e0a filter OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0 \u6e05\u9664\u6d41\u91cf\u6807\u8bb0\uff0c\u907f\u514d\u6267\u884c SNAT UDP: bad checksum on VXLAN interface nat PREROUTING -m comment --comment \"kube-ovn prerouting rules\" -j OVN-PREROUTING \u8fdb\u5165 OVN-PREROUTING \u94fe\u5904\u7406 -- nat POSTROUTING -m comment --comment \"kube-ovn postrouting rules\" -j OVN-POSTROUTING \u8fdb\u5165 OVN-POSTROUTING \u94fe\u5904\u7406 -- nat OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set ovn40services dst -j MARK --set-xmark 0x4000/0x4000 \u4e3a Pod \u8bbf\u95ee Service \u6d41\u91cf\u6dfb\u52a0 masquerade \u6807\u8bb0 \u4f5c\u7528\u4e8e\u5173\u95ed\u5185\u7f6e LB \u7684\u573a\u666f nat OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000 \u4e3a ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff08TCP\uff09\u6dfb\u52a0\u7279\u5b9a\u6807\u8bb0 \u4ec5 kube-proxy \u4f7f\u7528 ipvs \u6a21\u5f0f\u65f6\u5b58\u5728 nat OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000 \u4e3a ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff08UDP\uff09\u6dfb\u52a0\u7279\u5b9a\u6807\u8bb0 \u540c\u4e0a nat OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source \u5f53\u8282\u70b9\u901a\u8fc7 Service IP \u8bbf\u95ee Overlay Pod \u65f6\uff0c\u4fdd\u6301\u6e90 IP \u4e3a\u8282\u70b9 IP\u3002 \u4ec5 kube-proxy \u4f7f\u7528 ipvs \u6a21\u5f0f\u65f6\u751f\u6548 nat OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE \u4e3a\u7279\u5b9a\u6807\u8bb0\u7684\u6d41\u91cf\u6267\u884c SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j MASQUERADE \u4e3a\u901a\u8fc7\u8282\u70b9\u7684 Pod \u4e4b\u95f4\u7684 Service \u6d41\u91cf\u6267\u884c SNAT -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN \u5bf9\u4e8e ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff0c\u82e5 Endpoint \u4f7f\u7528\u5206\u5e03\u5f0f\u7f51\u5173\uff0c\u65e0\u9700\u6267\u884c SNAT -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j MASQUERADE \u5bf9\u4e8e ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff0c\u82e5 Endpoint \u4f7f\u7528\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u6267\u884c SNAT -- nat OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN Pod IP \u5bf9\u5916\u66b4\u9732\u65f6\uff0c\u4e0d\u6267\u884c SNAT -- nat OVN-POSTROUTING -s 10.16.0.0/16 -m set ! --match-set ovn40subnets dst -j SNAT --to-source 192.168.0.101 Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f NatOutgoing \u4e14\u4f7f\u7528\u6307\u5b9a IP \u7684\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u6267\u884c SNAT 10.16.0.0/16 \u4e3a\u5b50\u7f51\u7f51\u6bb5\uff0c192.168.0.101 \u4e3a\u6307\u5b9a\u7684\u7f51\u5173\u8282\u70b9 IP nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j MASQUERADE Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f NatOutgoing\uff0c\u6267\u884c SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f natOutgoingPolicyRules\uff0c\u6307\u5b9a\u7b56\u7565\u7684\u62a5\u6587\u6267\u884c SNAT \u914d\u7f6e\u4e86 natOutgoingPolicyRules \u5b50\u7f51\u7684\u51fa\u5916\u7f51\u62a5\u6587\u7684\u8fdb\u5165\u94fe OVN-NAT-POLICY nat OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j MASQUERADE --random-fully \u540c\u4e0a \u4ece OVN-NAT-POLICY \u51fa\u6765\u540e\uff0c\u5982\u679c\u88ab\u6253\u4e0a tag 0x90001/0x90001 \u5c31\u4f1a\u505a SNAT nat OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN \u540c\u4e0a \u4ece OVN-NAT-POLICY \u51fa\u6765\u540e, \u5982\u679c\u88ab\u6253\u4e0a tag 0x90002/0x90002 \u4e0d\u4f1a\u505a SNAT nat OVN-NAT-POLICY -s 10.0.11.0/24 -m comment --comment natPolicySubnet-net1 -j OVN-NAT-PSUBNET-aa98851157c5 \u540c\u4e0a 10.0.11.0/24 \u8868\u793a\u5b50\u7f51 net1 \u7684 CIDR\uff0c OVN-NAT-PSUBNET-aa98851157c5 \u8fd9\u6761\u94fe\u4e0b\u7684\u89c4\u5219\u5c31\u5bf9\u5e94\u8fd9\u4e2a\u5b50\u7f51\u7684 natOutgoingPolicyRules \u914d\u7f6e nat OVN-NAT-PSUBNET-xxxxxxxxxxxx -m set --match-set ovn40natpr-418e79269dc5-src src -m set --match-set ovn40natpr-418e79269dc5-dst dst -j MARK --set-xmark 0x90002/0x90002 \u540c\u4e0a 418e79269dc5 \u8868\u793a natOutgoingPolicyRules \u4e2d\u7684\u4e00\u6761\u89c4\u5219\u7684 ID\uff0c\u53ef\u4ee5\u901a\u8fc7 status.natOutgoingPolicyRules[index].RuleID \u67e5\u770b\u5230\uff0c \u8868\u793a srcIPs \u6ee1\u8db3 ovn40natpr-418e79269dc5-src\uff0c dstIPS \u6ee1\u8db3 ovn40natpr-418e79269dc5-dst \u4f1a\u6253\u4e0a tag 0x90002 mangle OVN-OUTPUT -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003 \u5c06 kubelet \u7684\u63a2\u6d4b\u6d41\u91cf\u52a0\u4e0a\u7279\u5b9a\u6807\u8bb0\u4ece\u800c\u5f15\u5165\u5230 tproxy mangle OVN-PREROUTING -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.3 --tproxy-mark 0x90004/0x90004 \u5c06 kubelet \u7684\u63a2\u6d4b\u6d41\u91cf\u52a0\u4e0a\u7279\u5b9a\u6807\u8bb0\u4ece\u800c\u5f15\u5165\u5230 tproxy \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Iptables \u89c4\u5219"},{"location":"reference/iptables-rules/#iptables","text":"Kube-OVN \u4f7f\u7528 ipset \u53ca iptables \u8f85\u52a9\u5b9e\u73b0\u9ed8\u8ba4 VPC \u4e0b\u5bb9\u5668\u7f51\u7edc\uff08Overlay\uff09\u7f51\u5173 NAT \u7684\u529f\u80fd\u3002 \u4f7f\u7528\u7684 ipset \u5982\u4e0b\u8868\u6240\u793a\uff1a \u540d\u79f0\uff08IPv4/IPv6\uff09 \u7c7b\u578b \u5b58\u50a8\u5bf9\u8c61 ovn40services/ovn60services hash:net Service \u7f51\u6bb5 ovn40subnets/ovn60subnets hash:net Overlay \u5b50\u7f51\u7f51\u6bb5\u4ee5\u53ca NodeLocal DNS IP \u5730\u5740 ovn40subnets-nat/ovn60subnets-nat hash:net \u5f00\u542f NatOutgoing \u7684 Overlay \u5b50\u7f51\u7f51\u6bb5 ovn40subnets-distributed-gw/ovn60subnets-distributed-gw hash:net \u5f00\u542f\u5206\u5e03\u5f0f\u7f51\u5173\u7684 Overlay \u5b50\u7f51\u7f51\u6bb5 ovn40other-node/ovn60other-node hash:net \u5176\u5b83\u8282\u70b9\u7684\u5185\u90e8 IP \u5730\u5740 ovn40local-pod-ip-nat/ovn60local-pod-ip-nat hash:ip \u5df2\u5f03\u7528 ovn40subnets-nat-policy hash:net \u914d\u7f6e\u4e86 natOutgoingPolicyRules \u7684\u6240\u6709\u5b50\u7f51\u7f51\u6bb5 ovn40natpr-418e79269dc5-dst hash:net natOutgoingPolicyRules \u4e2d rule \u5bf9\u5e94\u7684 dstIPs ovn40natpr-418e79269dc5-src hash:net natOutgoingPolicyRules \u4e2d rule \u5bf9\u5e94\u7684 srcIPs \u4f7f\u7528\u7684 iptables \u89c4\u5219\uff08IPv4\uff09\u5982\u4e0b\u8868\u6240\u793a\uff1a \u8868 \u94fe \u89c4\u5219 \u7528\u9014 \u5907\u6ce8 filter INPUT -m set --match-set ovn40services src -j ACCEPT \u5141\u8bb8 k8s Service \u548c Pod \u76f8\u5173\u6d41\u91cf\u901a\u8fc7 -- filter INPUT -m set --match-set ovn40services dst -j ACCEPT \u540c\u4e0a -- filter INPUT -m set --match-set ovn40subnets src -j ACCEPT \u540c\u4e0a -- filter INPUT -m set --match-set ovn40subnets dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40services src -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40services dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40subnets src -j ACCEPT \u540c\u4e0a -- filter FORWARD -m set --match-set ovn40subnets dst -j ACCEPT \u540c\u4e0a -- filter FORWARD -s 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" \u7528\u4e8e\u8ba1\u6570\u4ece subnet \u8bbf\u95ee\u5916\u90e8\u7f51\u7edc\u7684\u62a5\u6587 10.16.0.0/16 \u4e3a subnet \u7684 cidr \uff0ccomment \u4e2d\u9017\u53f7\u524d\u9762\u7684 ovn-subnet-gateway \u7528\u4e8e\u6807\u8bc6\u8be5 iptables \u89c4\u5219\u7528\u4e8e subnet \u51fa\u5165\u7f51\u5173\u62a5\u6587\u8ba1\u6570\uff0c\u9017\u53f7\u540e\u9762 ovn-default \u662f\u8be5 subnet \u7684\u540d\u5b57 filter FORWARD -d 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" \u7528\u4e8e\u8ba1\u6570\u4ece\u5916\u90e8\u7f51\u7edc\u8bbf\u95ee subnet \u7684\u62a5\u6587 \u540c\u4e0a filter OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0 \u6e05\u9664\u6d41\u91cf\u6807\u8bb0\uff0c\u907f\u514d\u6267\u884c SNAT UDP: bad checksum on VXLAN interface nat PREROUTING -m comment --comment \"kube-ovn prerouting rules\" -j OVN-PREROUTING \u8fdb\u5165 OVN-PREROUTING \u94fe\u5904\u7406 -- nat POSTROUTING -m comment --comment \"kube-ovn postrouting rules\" -j OVN-POSTROUTING \u8fdb\u5165 OVN-POSTROUTING \u94fe\u5904\u7406 -- nat OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set ovn40services dst -j MARK --set-xmark 0x4000/0x4000 \u4e3a Pod \u8bbf\u95ee Service \u6d41\u91cf\u6dfb\u52a0 masquerade \u6807\u8bb0 \u4f5c\u7528\u4e8e\u5173\u95ed\u5185\u7f6e LB \u7684\u573a\u666f nat OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000 \u4e3a ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff08TCP\uff09\u6dfb\u52a0\u7279\u5b9a\u6807\u8bb0 \u4ec5 kube-proxy \u4f7f\u7528 ipvs \u6a21\u5f0f\u65f6\u5b58\u5728 nat OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000 \u4e3a ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff08UDP\uff09\u6dfb\u52a0\u7279\u5b9a\u6807\u8bb0 \u540c\u4e0a nat OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source \u5f53\u8282\u70b9\u901a\u8fc7 Service IP \u8bbf\u95ee Overlay Pod \u65f6\uff0c\u4fdd\u6301\u6e90 IP \u4e3a\u8282\u70b9 IP\u3002 \u4ec5 kube-proxy \u4f7f\u7528 ipvs \u6a21\u5f0f\u65f6\u751f\u6548 nat OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE \u4e3a\u7279\u5b9a\u6807\u8bb0\u7684\u6d41\u91cf\u6267\u884c SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j MASQUERADE \u4e3a\u901a\u8fc7\u8282\u70b9\u7684 Pod \u4e4b\u95f4\u7684 Service \u6d41\u91cf\u6267\u884c SNAT -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN \u5bf9\u4e8e ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff0c\u82e5 Endpoint \u4f7f\u7528\u5206\u5e03\u5f0f\u7f51\u5173\uff0c\u65e0\u9700\u6267\u884c SNAT -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j MASQUERADE \u5bf9\u4e8e ExternalTrafficPolicy \u4e3a Local \u7684 Service \u6d41\u91cf\uff0c\u82e5 Endpoint \u4f7f\u7528\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u6267\u884c SNAT -- nat OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN Pod IP \u5bf9\u5916\u66b4\u9732\u65f6\uff0c\u4e0d\u6267\u884c SNAT -- nat OVN-POSTROUTING -s 10.16.0.0/16 -m set ! --match-set ovn40subnets dst -j SNAT --to-source 192.168.0.101 Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f NatOutgoing \u4e14\u4f7f\u7528\u6307\u5b9a IP \u7684\u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u6267\u884c SNAT 10.16.0.0/16 \u4e3a\u5b50\u7f51\u7f51\u6bb5\uff0c192.168.0.101 \u4e3a\u6307\u5b9a\u7684\u7f51\u5173\u8282\u70b9 IP nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j MASQUERADE Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f NatOutgoing\uff0c\u6267\u884c SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY Pod \u8bbf\u95ee\u96c6\u7fa4\u5916\u7f51\u7edc\u65f6\uff0c\u82e5\u5b50\u7f51\u5f00\u542f natOutgoingPolicyRules\uff0c\u6307\u5b9a\u7b56\u7565\u7684\u62a5\u6587\u6267\u884c SNAT \u914d\u7f6e\u4e86 natOutgoingPolicyRules \u5b50\u7f51\u7684\u51fa\u5916\u7f51\u62a5\u6587\u7684\u8fdb\u5165\u94fe OVN-NAT-POLICY nat OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j MASQUERADE --random-fully \u540c\u4e0a \u4ece OVN-NAT-POLICY \u51fa\u6765\u540e\uff0c\u5982\u679c\u88ab\u6253\u4e0a tag 0x90001/0x90001 \u5c31\u4f1a\u505a SNAT nat OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN \u540c\u4e0a \u4ece OVN-NAT-POLICY \u51fa\u6765\u540e, \u5982\u679c\u88ab\u6253\u4e0a tag 0x90002/0x90002 \u4e0d\u4f1a\u505a SNAT nat OVN-NAT-POLICY -s 10.0.11.0/24 -m comment --comment natPolicySubnet-net1 -j OVN-NAT-PSUBNET-aa98851157c5 \u540c\u4e0a 10.0.11.0/24 \u8868\u793a\u5b50\u7f51 net1 \u7684 CIDR\uff0c OVN-NAT-PSUBNET-aa98851157c5 \u8fd9\u6761\u94fe\u4e0b\u7684\u89c4\u5219\u5c31\u5bf9\u5e94\u8fd9\u4e2a\u5b50\u7f51\u7684 natOutgoingPolicyRules \u914d\u7f6e nat OVN-NAT-PSUBNET-xxxxxxxxxxxx -m set --match-set ovn40natpr-418e79269dc5-src src -m set --match-set ovn40natpr-418e79269dc5-dst dst -j MARK --set-xmark 0x90002/0x90002 \u540c\u4e0a 418e79269dc5 \u8868\u793a natOutgoingPolicyRules \u4e2d\u7684\u4e00\u6761\u89c4\u5219\u7684 ID\uff0c\u53ef\u4ee5\u901a\u8fc7 status.natOutgoingPolicyRules[index].RuleID \u67e5\u770b\u5230\uff0c \u8868\u793a srcIPs \u6ee1\u8db3 ovn40natpr-418e79269dc5-src\uff0c dstIPS \u6ee1\u8db3 ovn40natpr-418e79269dc5-dst \u4f1a\u6253\u4e0a tag 0x90002 mangle OVN-OUTPUT -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003 \u5c06 kubelet \u7684\u63a2\u6d4b\u6d41\u91cf\u52a0\u4e0a\u7279\u5b9a\u6807\u8bb0\u4ece\u800c\u5f15\u5165\u5230 tproxy mangle OVN-PREROUTING -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.3 --tproxy-mark 0x90004/0x90004 \u5c06 kubelet \u7684\u63a2\u6d4b\u6d41\u91cf\u52a0\u4e0a\u7279\u5b9a\u6807\u8bb0\u4ece\u800c\u5f15\u5165\u5230 tproxy \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Iptables \u89c4\u5219"},{"location":"reference/kube-ovn-api/","text":"Kube-OVN \u63a5\u53e3\u89c4\u8303 \u00b6 \u57fa\u4e8e Kube-OVN v1.12.0 \u7248\u672c\uff0c\u6574\u7406\u4e86 Kube-OVN \u652f\u6301\u7684 CRD \u8d44\u6e90\u5217\u8868\uff0c\u5217\u51fa CRD \u5b9a\u4e49\u5404\u5b57\u6bb5\u7684\u53d6\u503c\u7c7b\u578b\u548c\u542b\u4e49\uff0c\u4ee5\u4f9b\u53c2\u8003\u3002 \u901a\u7528\u7684 Condition \u5b9a\u4e49 \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 type String \u72b6\u6001\u7c7b\u578b status String \u72b6\u6001\u503c\uff0c\u53d6\u503c\u4e3a True \uff0c False \u6216 Unknown reason String \u72b6\u6001\u53d8\u5316\u7684\u539f\u56e0 message String \u72b6\u6001\u53d8\u5316\u7684\u5177\u4f53\u4fe1\u606f lastUpdateTime Time \u4e0a\u6b21\u72b6\u6001\u66f4\u65b0\u65f6\u95f4 lastTransitionTime Time \u4e0a\u6b21\u72b6\u6001\u7c7b\u578b\u53d1\u751f\u53d8\u5316\u7684\u65f6\u95f4 \u5728\u5404 CRD \u7684\u5b9a\u4e49\u4e2d\uff0cStatus \u4e2d\u7684 Condition \u5b57\u6bb5\uff0c\u90fd\u9075\u5faa\u4e0a\u8ff0\u683c\u5f0f\uff0c\u56e0\u6b64\u63d0\u524d\u8fdb\u884c\u8bf4\u660e\u3002 Subnet \u5b9a\u4e49 \u00b6 Subnet \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Subnet metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SubnetSpec Subnet \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SubnetStatus Subnet \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5 SubnetSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 default Bool \u8be5\u5b50\u7f51\u662f\u5426\u4e3a\u9ed8\u8ba4\u5b50\u7f51 vpc String \u5b50\u7f51\u6240\u5c5e Vpc\uff0c\u9ed8\u8ba4\u4e3a ovn-cluster protocol String IP \u534f\u8bae\uff0c\u53d6\u503c\u53ef\u4ee5\u4e3a IPv4 \uff0c IPv6 \u6216 Dual namespaces []String \u8be5\u5b50\u7f51\u6240\u7ed1\u5b9a\u7684 namespace \u5217\u8868 cidrBlock String \u5b50\u7f51\u7684\u7f51\u6bb5\u8303\u56f4\uff0c\u5982 10.16.0.0/16 gateway String \u5b50\u7f51\u7f51\u5173\u5730\u5740\uff0c\u9ed8\u8ba4\u4e3a\u8be5\u5b50\u7f51 CIDRBlock \u4e0b\u7684\u7b2c\u4e00\u4e2a\u53ef\u7528\u5730\u5740 excludeIps []String \u8be5\u5b50\u7f51\u4e0b\u4e0d\u4f1a\u88ab\u81ea\u52a8\u5206\u914d\u7684\u5730\u5740\u8303\u56f4 provider String \u9ed8\u8ba4\u4e3a ovn\u3002\u591a\u7f51\u5361\u60c5\u51b5\u4e0b\u53ef\u4ee5\u914d\u7f6e\u53d6\u503c\u4e3a NetworkAttachmentDefinition \u7684 . \uff0cKube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90 gatewayType String Overlay \u6a21\u5f0f\u4e0b\u7684\u7f51\u5173\u7c7b\u578b\uff0c\u53d6\u503c\u53ef\u4ee5\u4e3a distributed \u6216 centralized gatewayNode String \u5f53\u7f51\u5173\u6a21\u5f0f\u4e3a centralized \u65f6\u7684\u7f51\u5173\u8282\u70b9\uff0c\u53ef\u4ee5\u4e3a\u9017\u53f7\u5206\u9694\u7684\u591a\u4e2a\u8282\u70b9 natOutgoing Bool \u51fa\u7f51\u6d41\u91cf\u662f\u5426\u8fdb\u884c NAT\u3002\u8be5\u53c2\u6570\u548c externalEgressGateway \u53c2\u6570\u4e0d\u80fd\u540c\u65f6\u8bbe\u7f6e\u3002 externalEgressGateway String \u5916\u90e8\u7f51\u5173\u5730\u5740\u3002\u9700\u8981\u548c\u5b50\u7f51\u7f51\u5173\u8282\u70b9\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u53ef\u8fbe\u57df\uff0c\u8be5\u53c2\u6570\u548c natOutgoing \u53c2\u6570\u4e0d\u80fd\u540c\u65f6\u8bbe\u7f6e policyRoutingPriority Uint32 \u7b56\u7565\u8def\u7531\u4f18\u5148\u7ea7\u3002\u6dfb\u52a0\u7b56\u7565\u8def\u7531\u4f7f\u7528\u53c2\u6570\uff0c\u63a7\u5236\u6d41\u91cf\u7ecf\u5b50\u7f51\u7f51\u5173\u4e4b\u540e\uff0c\u8f6c\u53d1\u5230\u5916\u90e8\u7f51\u5173\u5730\u5740 policyRoutingTableID Uint32 \u4f7f\u7528\u7684\u672c\u5730\u7b56\u7565\u8def\u7531\u8868\u7684 TableID\uff0c\u6bcf\u4e2a\u5b50\u7f51\u5747\u9700\u4e0d\u540c\u4ee5\u907f\u514d\u51b2\u7a81 private Bool \u6807\u8bc6\u8be5\u5b50\u7f51\u662f\u5426\u4e3a\u79c1\u6709\u5b50\u7f51\uff0c\u79c1\u6709\u5b50\u7f51\u9ed8\u8ba4\u62d2\u7edd\u5b50\u7f51\u5916\u7684\u5730\u5740\u8bbf\u95ee allowSubnets []String \u5b50\u7f51\u4e3a\u79c1\u6709\u5b50\u7f51\u7684\u60c5\u51b5\u4e0b\uff0c\u5141\u8bb8\u8bbf\u95ee\u8be5\u5b50\u7f51\u5730\u5740\u7684\u96c6\u5408 vlan String \u5b50\u7f51\u7ed1\u5b9a\u7684 Vlan \u540d\u79f0 vips []String \u5b50\u7f51\u4e0b virtual \u7c7b\u578b lsp \u7684 virtual-ip \u53c2\u6570\u4fe1\u606f logicalGateway Bool \u662f\u5426\u542f\u7528\u903b\u8f91\u7f51\u5173 disableGatewayCheck Bool \u521b\u5efa Pod \u65f6\u662f\u5426\u8df3\u8fc7\u7f51\u5173\u8054\u901a\u6027\u68c0\u67e5 disableInterConnection Bool \u63a7\u5236\u662f\u5426\u5f00\u542f\u5b50\u7f51\u8de8\u96c6\u7fa4\u4e92\u8054 enableDHCP Bool \u63a7\u5236\u662f\u5426\u914d\u7f6e\u5b50\u7f51\u4e0b lsp \u7684 dhcp \u914d\u7f6e\u9009\u9879 dhcpV4Options String \u5b50\u7f51\u4e0b lsp dhcpv4_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55 dhcpV6Options String \u5b50\u7f51\u4e0b lsp dhcpv6_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55 enableIPv6RA Bool \u63a7\u5236\u5b50\u7f51\u8fde\u63a5\u8def\u7531\u5668\u7684 lrp \u7aef\u53e3\uff0c\u662f\u5426\u914d\u7f6e ipv6_ra_configs \u53c2\u6570 ipv6RAConfigs String \u5b50\u7f51\u8fde\u63a5\u8def\u7531\u5668\u7684 lrp \u7aef\u53e3\uff0cipv6_ra_configs \u53c2\u6570\u914d\u7f6e\u4fe1\u606f acls []Acl \u5b50\u7f51\u5bf9\u5e94 logical-switch \u5173\u8054\u7684 acls \u8bb0\u5f55 u2oInterconnection Bool \u662f\u5426\u5f00\u542f Overlay/Underlay \u7684\u4e92\u8054\u6a21\u5f0f enableLb *Bool \u63a7\u5236\u5b50\u7f51\u5bf9\u5e94\u7684 logical-switch \u662f\u5426\u5173\u8054 load-balancer \u8bb0\u5f55 enableEcmp Bool \u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u662f\u5426\u5f00\u542f ECMP \u8def\u7531 Acl \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 direction String Acl \u9650\u5236\u65b9\u5411\uff0c\u53d6\u503c\u4e3a from-lport \u6216\u8005 to-lport priority Int Acl \u4f18\u5148\u7ea7\uff0c\u53d6\u503c\u8303\u56f4 0 \u5230 32767 match String Acl \u89c4\u5219\u5339\u914d\u8868\u8fbe\u5f0f action String Acl \u89c4\u5219\u52a8\u4f5c\uff0c\u53d6\u503c\u4e3a allow-related , allow-stateless , allow , drop , reject \u5176\u4e2d\u4e00\u4e2a SubnetStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []SubnetCondition \u5b50\u7f51\u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 v4AvailableIPs Float64 \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv4 IP \u5730\u5740\u6570\u91cf v4availableIPrange String \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv4 \u5730\u5740\u8303\u56f4 v4UsingIPs Float64 \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv4 IP \u5730\u5740\u6570\u91cf v4usingIPrange String \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv4 \u5730\u5740\u8303\u56f4 v6AvailableIPs Float64 \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv6 IP \u5730\u5740\u6570\u91cf v6availableIPrange String \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv6 \u5730\u5740\u8303\u56f4 v6UsingIPs Float64 \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv6 IP \u5730\u5740\u6570\u91cf v6usingIPrange String \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv6 \u5730\u5740\u8303\u56f4 sctivateGateway String \u96c6\u4e2d\u5f0f\u5b50\u7f51\uff0c\u4e3b\u5907\u6a21\u5f0f\u4e0b\u5f53\u524d\u6b63\u5728\u5de5\u4f5c\u7684\u7f51\u5173\u8282\u70b9 dhcpV4OptionsUUID String \u5b50\u7f51\u4e0b lsp dhcpv4_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55\u6807\u8bc6 dhcpV6OptionsUUID String \u5b50\u7f51\u4e0b lsp dhcpv6_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55\u6807\u8bc6 u2oInterconnectionIP String \u5f00\u542f Overlay/Underlay \u4e92\u8054\u6a21\u5f0f\u540e\uff0c\u6240\u5360\u7528\u7684\u7528\u4e8e\u4e92\u8054\u7684 IP \u5730\u5740 IP \u5b9a\u4e49 \u00b6 IP \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IP metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IPSpec IP \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 IPSepc \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 podName String \u7ed1\u5b9a Pod \u540d\u79f0 namespace String \u7ed1\u5b9a Pod \u6240\u5728 Namespace \u540d\u79f0 subnet String IP \u6240\u5c5e Subnet attachSubnets []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e\u5b50\u7f51\u540d\u79f0\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 nodeName String \u7ed1\u5b9a Pod \u6240\u5728\u7684\u8282\u70b9\u540d\u79f0 ipAddress String IP \u5730\u5740\uff0c\u53cc\u6808\u60c5\u51b5\u4e0b\u4e3a v4IP,v6IP \u683c\u5f0f v4IPAddress String IPv4 IP \u5730\u5740 v6IPAddress String IPv6 IP \u5730\u5740 attachIPs []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e IP \u5730\u5740\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 macAddress String \u7ed1\u5b9a Pod \u7684 Mac \u5730\u5740 attachMacs []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e Mac \u5730\u5740\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 containerID String \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 Container ID podType String \u7279\u6b8a\u5de5\u4f5c\u8d1f\u8f7d Pod\uff0c\u53ef\u4e3a StatefulSet \uff0c VirtualMachine \u6216\u7a7a Underlay \u914d\u7f6e \u00b6 Vlan \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vlan metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VlanSpec Vlan \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VlanStatus Vlan \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5 VlanSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 id Int Vlan tag \u53f7\uff0c\u53d6\u503c\u8303\u56f4\u4e3a 0~4096 provider String Vlan \u7ed1\u5b9a\u7684 ProviderNetwork \u540d\u79f0 VlanStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 subnets []String Vlan \u7ed1\u5b9a\u7684\u5b50\u7f51\u5217\u8868 conditions []VlanCondition Vlan \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ProviderNetwork \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a ProviderNetwork metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec ProviderNetworkSpec ProviderNetwork \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status ProviderNetworkStatus ProviderNetwork \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5 ProviderNetworkSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 defaultInterface String \u8be5\u6865\u63a5\u7f51\u7edc\u9ed8\u8ba4\u4f7f\u7528\u7684\u7f51\u5361\u63a5\u53e3\u540d\u79f0 customInterfaces []CustomInterface \u8be5\u6865\u63a5\u7f51\u7edc\u7279\u6b8a\u4f7f\u7528\u7684\u7f51\u5361\u914d\u7f6e excludeNodes []String \u8be5\u6865\u63a5\u7f51\u7edc\u4e0d\u4f1a\u7ed1\u5b9a\u7684\u8282\u70b9\u540d\u79f0 exchangeLinkName Bool \u662f\u5426\u4ea4\u6362\u6865\u63a5\u7f51\u5361\u548c\u5bf9\u5e94 OVS \u7f51\u6865\u540d\u79f0 CustomInterface \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 interface String Underlay \u4f7f\u7528\u7f51\u5361\u63a5\u53e3\u540d\u79f0 nodes []String \u4f7f\u7528\u81ea\u5b9a\u4e49\u7f51\u5361\u63a5\u53e3\u7684\u8282\u70b9\u5217\u8868 ProviderNetworkStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool \u5f53\u524d\u6865\u63a5\u7f51\u7edc\u662f\u5426\u8fdb\u5165\u5c31\u7eea\u72b6\u6001 readyNodes []String \u6865\u63a5\u7f51\u7edc\u8fdb\u5165\u5c31\u7eea\u72b6\u6001\u7684\u8282\u70b9\u540d\u79f0 notReadyNodes []String \u6865\u63a5\u7f51\u7edc\u672a\u8fdb\u5165\u5c31\u7eea\u72b6\u6001\u7684\u8282\u70b9\u540d\u79f0 vlans []String \u6865\u63a5\u7f51\u7edc\u7ed1\u5b9a\u7684 Vlan \u540d\u79f0 conditions []ProviderNetworkCondition ProviderNetwork \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 Vpc \u5b9a\u4e49 \u00b6 Vpc \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vpc metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcSpec Vpc \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VpcStatus Vpc \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5 VpcSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 namespaces []String Vpc \u7ed1\u5b9a\u7684\u547d\u540d\u7a7a\u95f4\u5217\u8868 staticRoutes []*StaticRoute Vpc \u4e0b\u914d\u7f6e\u7684\u9759\u6001\u8def\u7531\u4fe1\u606f policyRoutes []*PolicyRoute Vpc \u4e0b\u914d\u7f6e\u7684\u7b56\u7565\u8def\u7531\u4fe1\u606f vpcPeerings []*VpcPeering Vpc \u4e92\u8054\u4fe1\u606f enableExternal Bool Vpc \u662f\u5426\u8fde\u63a5\u5230\u5916\u90e8\u4ea4\u6362\u673a StaticRoute \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 policy String \u8def\u7531\u7b56\u7565\uff0c\u53d6\u503c\u4e3a policySrc \u6216\u8005 policyDst cidr String \u8def\u7531 Cidr \u7f51\u6bb5 nextHopIP String \u8def\u7531\u4e0b\u4e00\u8df3\u4fe1\u606f PolicyRoute \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 priority Int32 \u7b56\u7565\u8def\u7531\u4f18\u5148\u7ea7 match String \u7b56\u7565\u8def\u7531\u5339\u914d\u6761\u4ef6 action String \u7b56\u7565\u8def\u7531\u52a8\u4f5c\uff0c\u53d6\u503c\u4e3a allow \u3001 drop \u6216\u8005 reroute nextHopIP String \u7b56\u7565\u8def\u7531\u4e0b\u4e00\u8df3\u4fe1\u606f\uff0cECMP \u8def\u7531\u60c5\u51b5\u4e0b\u4e0b\u4e00\u8df3\u5730\u5740\u4f7f\u7528\u9017\u53f7\u9694\u5f00 VpcPeering \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 remoteVpc String Vpc \u4e92\u8054\u5bf9\u7aef Vpc \u540d\u79f0 localConnectIP String Vpc \u4e92\u8054\u672c\u7aef IP \u5730\u5740 VpcStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VpcCondition Vpc \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 standby Bool \u6807\u8bc6 Vpc \u662f\u5426\u521b\u5efa\u5b8c\u6210\uff0cVpc \u4e0b\u7684 Subnet \u9700\u8981\u7b49 Vpc \u521b\u5efa\u5b8c\u6210\u8f6c\u6362\u518d\u7ee7\u7eed\u5904\u7406 default Bool \u662f\u5426\u662f\u9ed8\u8ba4 Vpc defaultLogicalSwitch String Vpc \u4e0b\u7684\u9ed8\u8ba4\u5b50\u7f51 router String Vpc \u5bf9\u5e94\u7684 logical-router \u540d\u79f0 tcpLoadBalancer String Vpc \u4e0b\u7684 TCP LB \u4fe1\u606f udpLoadBalancer String Vpc \u4e0b\u7684 UDP LB \u4fe1\u606f tcpSessionLoadBalancer String Vpc \u4e0b\u7684 TCP \u4f1a\u8bdd\u4fdd\u6301 LB \u4fe1\u606f udpSessionLoadBalancer String Vpc \u4e0b\u7684 UDP \u4f1a\u8bdd\u4fdd\u6301 LB \u4fe1\u606f subnets []String Vpc \u4e0b\u7684\u5b50\u7f51\u5217\u8868 vpcPeerings []String Vpc \u4e92\u8054\u7684\u5bf9\u7aef Vpc \u5217\u8868 enableExternal Bool Vpc \u662f\u5426\u8fde\u63a5\u5230\u5916\u90e8\u4ea4\u6362\u673a VpcNatGateway \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a VpcNatGateway metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcNatSpec Vpc \u7f51\u5173\u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 VpcNatSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vpc String Vpc \u7f51\u5173 Pod \u6240\u5728\u7684 Vpc \u540d\u79f0 subnet String Vpc \u7f51\u5173 Pod \u6240\u5c5e\u7684\u5b50\u7f51\u540d\u79f0 lanIp String Vpc \u7f51\u5173 Pod \u6307\u5b9a\u5206\u914d\u7684 IP \u5730\u5740 selector []String \u6807\u51c6 Kubernetes Selector \u5339\u914d\u4fe1\u606f tolerations []VpcNatToleration \u6807\u51c6 Kubernetes \u5bb9\u5fcd\u4fe1\u606f VpcNatToleration \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 key String \u5bb9\u5fcd\u6c61\u70b9\u7684 key \u4fe1\u606f operator String \u53d6\u503c\u4e3a Exists \u6216\u8005 Equal value String \u5bb9\u5fcd\u6c61\u70b9\u7684 value \u4fe1\u606f effect String \u5bb9\u5fcd\u6c61\u70b9\u7684\u4f5c\u7528\u6548\u679c\uff0c\u53d6\u503c\u4e3a NoExecute \u3001 NoSchedule \u6216\u8005 PreferNoSchedule tolerationSeconds Int64 \u6dfb\u52a0\u6c61\u70b9\u540e\uff0cPod \u8fd8\u80fd\u7ee7\u7eed\u5728\u8282\u70b9\u4e0a\u8fd0\u884c\u7684\u65f6\u95f4 \u4ee5\u4e0a\u5bb9\u5fcd\u5b57\u6bb5\u7684\u542b\u4e49\uff0c\u53ef\u4ee5\u53c2\u8003 Kubernetes \u5b98\u65b9\u6587\u6863 \u6c61\u70b9\u548c\u5bb9\u5fcd\u5ea6 \u3002 IptablesEIP \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesEIP metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesEipSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesEIP \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesEipStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesEIP \u72b6\u6001\u4fe1\u606f IptablesEipSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 v4ip String IptablesEIP v4 \u5730\u5740 v6ip String IptablesEIP v6 \u5730\u5740 macAddress String IptablesEIP crd \u8bb0\u5f55\u5206\u914d\u7684 mac \u5730\u5740\uff0c\u6ca1\u6709\u5b9e\u9645\u4f7f\u7528 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 IptablesEipStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesEIP \u662f\u5426\u914d\u7f6e\u5b8c\u6210 ip String IptablesEIP \u4f7f\u7528\u7684 IP \u5730\u5740\uff0c\u76ee\u524d\u53ea\u652f\u6301\u4e86 IPv4 \u5730\u5740 redo String IptablesEIP crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 nat String IptablesEIP \u7684\u4f7f\u7528\u7c7b\u578b\uff0c\u53d6\u503c\u4e3a fip \u3001 snat \u6216\u8005 dnat conditions []IptablesEIPCondition IptablesEIP \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 IptablesFIPRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesFIPRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesFIPRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesFIPRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesFIPRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesFIPRule \u72b6\u6001\u4fe1\u606f IptablesFIPRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip String IptablesFIPRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 internalIp String IptablesFIPRule \u5bf9\u5e94\u7684\u5185\u90e8\u7684 IP \u5730\u5740 IptablesFIPRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesFIPRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesEIP \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesEIP \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesFIPRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesFIPRuleCondition IptablesFIPRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 IptablesSnatRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesSnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesSnatRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesSnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesSnatRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesSnatRule \u72b6\u6001\u4fe1\u606f IptablesSnatRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip String IptablesSnatRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 internalIp String IptablesSnatRule \u5bf9\u5e94\u7684\u5185\u90e8\u7684 IP \u5730\u5740 IptablesSnatRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesSnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesSnatRule \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesSnatRule \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesSnatRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesSnatRuleCondition IptablesSnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 IptablesDnatRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesDnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesDnatRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesDnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesDnatRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesDnatRule \u72b6\u6001\u4fe1\u606f IptablesDnatRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 externalPort Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5916\u90e8\u7aef\u53e3 protocol Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u7684\u534f\u8bae\u7c7b\u578b internalIp Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5185\u90e8 IP \u5730\u5740 internalPort Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5185\u90e8\u7aef\u53e3 IptablesDnatRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesDnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesDnatRule \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesDnatRule \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesDnatRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesDnatRuleCondition IptablesDnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 VpcDns \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a VpcDns metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcDnsSpec VpcDns \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VpcDnsStatus VpcDns \u72b6\u6001\u4fe1\u606f VpcDnsSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vpc String VpcDns \u6240\u5728\u7684 Vpc \u540d\u79f0 subnet String VpcDns Pod \u5206\u914d\u5730\u5740\u7684 Subnet \u540d\u79f0 VpcDnsStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VpcDnsCondition VpcDns \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 active Bool VpcDns \u662f\u5426\u6b63\u5728\u4f7f\u7528 VpcDns \u7684\u8be6\u7ec6\u4f7f\u7528\u6587\u6863\uff0c\u53ef\u4ee5\u53c2\u8003 \u81ea\u5b9a\u4e49 VPC DNS \u3002 SwitchLBRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a SwitchLBRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SwitchLBRuleSpec SwitchLBRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SwitchLBRuleStatus SwitchLBRule \u72b6\u6001\u4fe1\u606f SwitchLBRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vip String SwitchLBRule \u914d\u7f6e\u7684 vip \u5730\u5740 namespace String SwitchLBRule \u7684\u547d\u540d\u7a7a\u95f4 selector []String \u6807\u51c6 Kubernetes Selector \u5339\u914d\u4fe1\u606f sessionAffinity String \u6807\u51c6 Kubernetes Service \u4e2d sessionAffinity \u53d6\u503c ports []SlrPort SwitchLBRule \u7aef\u53e3\u5217\u8868 SwitchLBRule \u7684\u8be6\u7ec6\u914d\u7f6e\u4fe1\u606f\uff0c\u53ef\u4ee5\u53c2\u8003 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u3002 SlrPort \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 name String \u7aef\u53e3\u540d\u79f0 port Int32 \u7aef\u53e3\u53f7 targetPort Int32 \u76ee\u6807\u7aef\u53e3\u53f7 protocol String \u534f\u8bae\u7c7b\u578b SwitchLBRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []SwitchLBRuleCondition SwitchLBRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ports String SwitchLBRule \u7aef\u53e3\u4fe1\u606f service String SwitchLBRule \u63d0\u4f9b\u670d\u52a1\u7684 service \u540d\u79f0 \u5b89\u5168\u7ec4\u4e0e Vip \u00b6 SecurityGroup \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a SecurityGroup metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SecurityGroupSpec \u5b89\u5168\u7ec4\u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SecurityGroupStatus \u5b89\u5168\u7ec4\u72b6\u6001\u4fe1\u606f SecurityGroupSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ingressRules []*SgRule \u5165\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 egressRules []*SgRule \u51fa\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 allowSameGroupTraffic Bool \u540c\u4e00\u5b89\u5168\u7ec4\u5185\u7684 lsp \u662f\u5426\u53ef\u4ee5\u4e92\u901a\uff0c\u4ee5\u53ca\u6d41\u91cf\u89c4\u5219\u662f\u5426\u9700\u8981\u66f4\u65b0 SgRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ipVersion String IP \u7248\u672c\u53f7\uff0c\u53d6\u503c\u4e3a ipv4 \u6216\u8005 ipv6 protocol String \u53d6\u503c\u4e3a all \u3001 icmp \u3001 tcp \u6216\u8005 udp priority Int Acl \u4f18\u5148\u7ea7\uff0c\u53d6\u503c\u8303\u56f4\u4e3a 1-200\uff0c\u6570\u503c\u8d8a\u5c0f\uff0c\u4f18\u5148\u7ea7\u8d8a\u9ad8 remoteType String \u53d6\u503c\u4e3a address \u6216\u8005 securityGroup remoteAddress String \u5bf9\u7aef\u5730\u5740 remoteSecurityGroup String \u5bf9\u7aef\u5b89\u5168\u7ec4 portRangeMin Int \u7aef\u53e3\u8303\u56f4\u8d77\u59cb\u503c\uff0c\u6700\u5c0f\u53d6\u503c\u4e3a 1 portRangeMax Int \u7aef\u53e3\u8303\u56f4\u6700\u5927\u503c\uff0c\u6700\u5927\u53d6\u503c\u4e3a 65535 policy String \u53d6\u503c\u4e3a allow \u6216\u8005 drop SecurityGroupStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 portGroup String \u5b89\u5168\u7ec4\u5bf9\u5e94\u7684 port-group \u540d\u79f0 allowSameGroupTraffic Bool \u540c\u4e00\u5b89\u5168\u7ec4\u5185\u7684 lsp \u662f\u5426\u53ef\u4ee5\u4e92\u901a\uff0c\u4ee5\u53ca\u5b89\u5168\u7ec4\u7684\u6d41\u91cf\u89c4\u5219\u662f\u5426\u9700\u8981\u66f4\u65b0 ingressMd5 String \u5165\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 MD5 \u53d6\u503c egressMd5 String \u51fa\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 MD5 \u53d6\u503c ingressLastSyncSuccess Bool \u5165\u65b9\u5411\u89c4\u5219\u4e0a\u4e00\u6b21\u540c\u6b65\u662f\u5426\u6210\u529f egressLastSyncSuccess Bool \u51fa\u65b9\u5411\u89c4\u5219\u4e0a\u4e00\u6b21\u540c\u6b65\u662f\u5426\u6210\u529f Vip \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VipSpec Vip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VipStatus Vip \u72b6\u6001\u4fe1\u606f VipSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 namespace String Vip \u6240\u5728\u547d\u540d\u7a7a\u95f4 subnet String Vip \u6240\u5c5e\u5b50\u7f51 v4ip String Vip v4 IP \u5730\u5740 v6ip String Vip v6 IP \u5730\u5740 macAddress String Vip mac \u5730\u5740 parentV4ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 parentV6ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 parentMac String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 attachSubnets []String \u8be5\u5b57\u6bb5\u5e9f\u5f03\uff0c\u4e0d\u518d\u4f7f\u7528 VipStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VipCondition Vip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ready Bool Vip \u662f\u5426\u51c6\u5907\u597d v4ip String Vip v4 IP \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 v6ip String Vip v6 IP \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 mac String Vip mac \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 pv4ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 pv6ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 pmac String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 OvnEip \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnEip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnEipSpec \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnEip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnEipStatus \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnEip \u72b6\u6001\u4fe1\u606f OvnEipSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 externalSubnet String OvnEip \u6240\u5728\u7684\u5b50\u7f51\u540d\u79f0 v4ip String OvnEip IP \u5730\u5740 macAddress String OvnEip Mac \u5730\u5740 type String OvnEip \u4f7f\u7528\u7c7b\u578b\uff0c\u53d6\u503c\u6709 fip \u3001 snat \u6216\u8005 lrp OvnEipStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []OvnEipCondition \u9ed8\u8ba4 Vpc OvnEip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 v4ip String OvnEip \u4f7f\u7528\u7684 v4 IP \u5730\u5740 macAddress String OvnEip \u4f7f\u7528\u7684 Mac \u5730\u5740 OvnFip \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnFip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnFipSpec \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnFip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnFipStatus \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnFip \u72b6\u6001\u4fe1\u606f OvnFipSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ovnEip String OvnFip \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 ipName String OvnFip \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 IP crd \u540d\u79f0 OvnFipStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool OvnFip \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4Eip String OvnFip \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 v4Ip String OvnFip \u5f53\u524d\u4f7f\u7528\u7684 OvnEip \u5730\u5740 macAddress String OvnFip \u914d\u7f6e\u7684 Mac \u5730\u5740 vpc String OvnFip \u6240\u5728\u7684 Vpc \u540d\u79f0 conditions []OvnFipCondition OvnFip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 OvnSnatRule \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnSnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnSnatRuleSpec \u9ed8\u8ba4 Vpc OvnSnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnSnatRuleStatus \u9ed8\u8ba4 Vpc OvnSnatRule \u72b6\u6001\u4fe1\u606f OvnSnatRuleSpec \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ovnEip String OvnSnatRule \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 vpcSubnet String OvnSnatRule \u914d\u7f6e\u7684\u5b50\u7f51\u540d\u79f0 ipName String OvnSnatRule \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 IP crd \u540d\u79f0 OvnSnatRuleStatus \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool OvnSnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4Eip String OvnSnatRule \u7ed1\u5b9a\u7684 OvnEip \u5730\u5740 v4IpCidr String \u5728 logical-router \u4e2d\u914d\u7f6e snat \u8f6c\u6362\u4f7f\u7528\u7684 cidr \u5730\u5740 vpc String OvnSnatRule \u6240\u5728\u7684 Vpc \u540d\u79f0 conditions []OvnSnatRuleCondition OvnSnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN \u63a5\u53e3\u89c4\u8303"},{"location":"reference/kube-ovn-api/#kube-ovn","text":"\u57fa\u4e8e Kube-OVN v1.12.0 \u7248\u672c\uff0c\u6574\u7406\u4e86 Kube-OVN \u652f\u6301\u7684 CRD \u8d44\u6e90\u5217\u8868\uff0c\u5217\u51fa CRD \u5b9a\u4e49\u5404\u5b57\u6bb5\u7684\u53d6\u503c\u7c7b\u578b\u548c\u542b\u4e49\uff0c\u4ee5\u4f9b\u53c2\u8003\u3002","title":"Kube-OVN \u63a5\u53e3\u89c4\u8303"},{"location":"reference/kube-ovn-api/#condition","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 type String \u72b6\u6001\u7c7b\u578b status String \u72b6\u6001\u503c\uff0c\u53d6\u503c\u4e3a True \uff0c False \u6216 Unknown reason String \u72b6\u6001\u53d8\u5316\u7684\u539f\u56e0 message String \u72b6\u6001\u53d8\u5316\u7684\u5177\u4f53\u4fe1\u606f lastUpdateTime Time \u4e0a\u6b21\u72b6\u6001\u66f4\u65b0\u65f6\u95f4 lastTransitionTime Time \u4e0a\u6b21\u72b6\u6001\u7c7b\u578b\u53d1\u751f\u53d8\u5316\u7684\u65f6\u95f4 \u5728\u5404 CRD \u7684\u5b9a\u4e49\u4e2d\uff0cStatus \u4e2d\u7684 Condition \u5b57\u6bb5\uff0c\u90fd\u9075\u5faa\u4e0a\u8ff0\u683c\u5f0f\uff0c\u56e0\u6b64\u63d0\u524d\u8fdb\u884c\u8bf4\u660e\u3002","title":"\u901a\u7528\u7684 Condition \u5b9a\u4e49"},{"location":"reference/kube-ovn-api/#subnet","text":"","title":"Subnet \u5b9a\u4e49"},{"location":"reference/kube-ovn-api/#subnet_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Subnet metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SubnetSpec Subnet \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SubnetStatus Subnet \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5","title":"Subnet"},{"location":"reference/kube-ovn-api/#subnetspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 default Bool \u8be5\u5b50\u7f51\u662f\u5426\u4e3a\u9ed8\u8ba4\u5b50\u7f51 vpc String \u5b50\u7f51\u6240\u5c5e Vpc\uff0c\u9ed8\u8ba4\u4e3a ovn-cluster protocol String IP \u534f\u8bae\uff0c\u53d6\u503c\u53ef\u4ee5\u4e3a IPv4 \uff0c IPv6 \u6216 Dual namespaces []String \u8be5\u5b50\u7f51\u6240\u7ed1\u5b9a\u7684 namespace \u5217\u8868 cidrBlock String \u5b50\u7f51\u7684\u7f51\u6bb5\u8303\u56f4\uff0c\u5982 10.16.0.0/16 gateway String \u5b50\u7f51\u7f51\u5173\u5730\u5740\uff0c\u9ed8\u8ba4\u4e3a\u8be5\u5b50\u7f51 CIDRBlock \u4e0b\u7684\u7b2c\u4e00\u4e2a\u53ef\u7528\u5730\u5740 excludeIps []String \u8be5\u5b50\u7f51\u4e0b\u4e0d\u4f1a\u88ab\u81ea\u52a8\u5206\u914d\u7684\u5730\u5740\u8303\u56f4 provider String \u9ed8\u8ba4\u4e3a ovn\u3002\u591a\u7f51\u5361\u60c5\u51b5\u4e0b\u53ef\u4ee5\u914d\u7f6e\u53d6\u503c\u4e3a NetworkAttachmentDefinition \u7684 . \uff0cKube-OVN \u5c06\u4f1a\u4f7f\u7528\u8fd9\u4e9b\u4fe1\u606f\u627e\u5230\u5bf9\u5e94\u7684 Subnet \u8d44\u6e90 gatewayType String Overlay \u6a21\u5f0f\u4e0b\u7684\u7f51\u5173\u7c7b\u578b\uff0c\u53d6\u503c\u53ef\u4ee5\u4e3a distributed \u6216 centralized gatewayNode String \u5f53\u7f51\u5173\u6a21\u5f0f\u4e3a centralized \u65f6\u7684\u7f51\u5173\u8282\u70b9\uff0c\u53ef\u4ee5\u4e3a\u9017\u53f7\u5206\u9694\u7684\u591a\u4e2a\u8282\u70b9 natOutgoing Bool \u51fa\u7f51\u6d41\u91cf\u662f\u5426\u8fdb\u884c NAT\u3002\u8be5\u53c2\u6570\u548c externalEgressGateway \u53c2\u6570\u4e0d\u80fd\u540c\u65f6\u8bbe\u7f6e\u3002 externalEgressGateway String \u5916\u90e8\u7f51\u5173\u5730\u5740\u3002\u9700\u8981\u548c\u5b50\u7f51\u7f51\u5173\u8282\u70b9\u5728\u540c\u4e00\u4e2a\u4e8c\u5c42\u53ef\u8fbe\u57df\uff0c\u8be5\u53c2\u6570\u548c natOutgoing \u53c2\u6570\u4e0d\u80fd\u540c\u65f6\u8bbe\u7f6e policyRoutingPriority Uint32 \u7b56\u7565\u8def\u7531\u4f18\u5148\u7ea7\u3002\u6dfb\u52a0\u7b56\u7565\u8def\u7531\u4f7f\u7528\u53c2\u6570\uff0c\u63a7\u5236\u6d41\u91cf\u7ecf\u5b50\u7f51\u7f51\u5173\u4e4b\u540e\uff0c\u8f6c\u53d1\u5230\u5916\u90e8\u7f51\u5173\u5730\u5740 policyRoutingTableID Uint32 \u4f7f\u7528\u7684\u672c\u5730\u7b56\u7565\u8def\u7531\u8868\u7684 TableID\uff0c\u6bcf\u4e2a\u5b50\u7f51\u5747\u9700\u4e0d\u540c\u4ee5\u907f\u514d\u51b2\u7a81 private Bool \u6807\u8bc6\u8be5\u5b50\u7f51\u662f\u5426\u4e3a\u79c1\u6709\u5b50\u7f51\uff0c\u79c1\u6709\u5b50\u7f51\u9ed8\u8ba4\u62d2\u7edd\u5b50\u7f51\u5916\u7684\u5730\u5740\u8bbf\u95ee allowSubnets []String \u5b50\u7f51\u4e3a\u79c1\u6709\u5b50\u7f51\u7684\u60c5\u51b5\u4e0b\uff0c\u5141\u8bb8\u8bbf\u95ee\u8be5\u5b50\u7f51\u5730\u5740\u7684\u96c6\u5408 vlan String \u5b50\u7f51\u7ed1\u5b9a\u7684 Vlan \u540d\u79f0 vips []String \u5b50\u7f51\u4e0b virtual \u7c7b\u578b lsp \u7684 virtual-ip \u53c2\u6570\u4fe1\u606f logicalGateway Bool \u662f\u5426\u542f\u7528\u903b\u8f91\u7f51\u5173 disableGatewayCheck Bool \u521b\u5efa Pod \u65f6\u662f\u5426\u8df3\u8fc7\u7f51\u5173\u8054\u901a\u6027\u68c0\u67e5 disableInterConnection Bool \u63a7\u5236\u662f\u5426\u5f00\u542f\u5b50\u7f51\u8de8\u96c6\u7fa4\u4e92\u8054 enableDHCP Bool \u63a7\u5236\u662f\u5426\u914d\u7f6e\u5b50\u7f51\u4e0b lsp \u7684 dhcp \u914d\u7f6e\u9009\u9879 dhcpV4Options String \u5b50\u7f51\u4e0b lsp dhcpv4_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55 dhcpV6Options String \u5b50\u7f51\u4e0b lsp dhcpv6_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55 enableIPv6RA Bool \u63a7\u5236\u5b50\u7f51\u8fde\u63a5\u8def\u7531\u5668\u7684 lrp \u7aef\u53e3\uff0c\u662f\u5426\u914d\u7f6e ipv6_ra_configs \u53c2\u6570 ipv6RAConfigs String \u5b50\u7f51\u8fde\u63a5\u8def\u7531\u5668\u7684 lrp \u7aef\u53e3\uff0cipv6_ra_configs \u53c2\u6570\u914d\u7f6e\u4fe1\u606f acls []Acl \u5b50\u7f51\u5bf9\u5e94 logical-switch \u5173\u8054\u7684 acls \u8bb0\u5f55 u2oInterconnection Bool \u662f\u5426\u5f00\u542f Overlay/Underlay \u7684\u4e92\u8054\u6a21\u5f0f enableLb *Bool \u63a7\u5236\u5b50\u7f51\u5bf9\u5e94\u7684 logical-switch \u662f\u5426\u5173\u8054 load-balancer \u8bb0\u5f55 enableEcmp Bool \u96c6\u4e2d\u5f0f\u7f51\u5173\uff0c\u662f\u5426\u5f00\u542f ECMP \u8def\u7531","title":"SubnetSpec"},{"location":"reference/kube-ovn-api/#acl","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 direction String Acl \u9650\u5236\u65b9\u5411\uff0c\u53d6\u503c\u4e3a from-lport \u6216\u8005 to-lport priority Int Acl \u4f18\u5148\u7ea7\uff0c\u53d6\u503c\u8303\u56f4 0 \u5230 32767 match String Acl \u89c4\u5219\u5339\u914d\u8868\u8fbe\u5f0f action String Acl \u89c4\u5219\u52a8\u4f5c\uff0c\u53d6\u503c\u4e3a allow-related , allow-stateless , allow , drop , reject \u5176\u4e2d\u4e00\u4e2a","title":"Acl"},{"location":"reference/kube-ovn-api/#subnetstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []SubnetCondition \u5b50\u7f51\u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 v4AvailableIPs Float64 \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv4 IP \u5730\u5740\u6570\u91cf v4availableIPrange String \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv4 \u5730\u5740\u8303\u56f4 v4UsingIPs Float64 \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv4 IP \u5730\u5740\u6570\u91cf v4usingIPrange String \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv4 \u5730\u5740\u8303\u56f4 v6AvailableIPs Float64 \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv6 IP \u5730\u5740\u6570\u91cf v6availableIPrange String \u5b50\u7f51\u73b0\u5728\u53ef\u7528\u7684 IPv6 \u5730\u5740\u8303\u56f4 v6UsingIPs Float64 \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv6 IP \u5730\u5740\u6570\u91cf v6usingIPrange String \u5b50\u7f51\u73b0\u5728\u5df2\u7528\u7684 IPv6 \u5730\u5740\u8303\u56f4 sctivateGateway String \u96c6\u4e2d\u5f0f\u5b50\u7f51\uff0c\u4e3b\u5907\u6a21\u5f0f\u4e0b\u5f53\u524d\u6b63\u5728\u5de5\u4f5c\u7684\u7f51\u5173\u8282\u70b9 dhcpV4OptionsUUID String \u5b50\u7f51\u4e0b lsp dhcpv4_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55\u6807\u8bc6 dhcpV6OptionsUUID String \u5b50\u7f51\u4e0b lsp dhcpv6_options \u5173\u8054\u7684 DHCP_Options \u8bb0\u5f55\u6807\u8bc6 u2oInterconnectionIP String \u5f00\u542f Overlay/Underlay \u4e92\u8054\u6a21\u5f0f\u540e\uff0c\u6240\u5360\u7528\u7684\u7528\u4e8e\u4e92\u8054\u7684 IP \u5730\u5740","title":"SubnetStatus"},{"location":"reference/kube-ovn-api/#ip","text":"","title":"IP \u5b9a\u4e49"},{"location":"reference/kube-ovn-api/#ip_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IP metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IPSpec IP \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5","title":"IP"},{"location":"reference/kube-ovn-api/#ipsepc","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 podName String \u7ed1\u5b9a Pod \u540d\u79f0 namespace String \u7ed1\u5b9a Pod \u6240\u5728 Namespace \u540d\u79f0 subnet String IP \u6240\u5c5e Subnet attachSubnets []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e\u5b50\u7f51\u540d\u79f0\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 nodeName String \u7ed1\u5b9a Pod \u6240\u5728\u7684\u8282\u70b9\u540d\u79f0 ipAddress String IP \u5730\u5740\uff0c\u53cc\u6808\u60c5\u51b5\u4e0b\u4e3a v4IP,v6IP \u683c\u5f0f v4IPAddress String IPv4 IP \u5730\u5740 v6IPAddress String IPv6 IP \u5730\u5740 attachIPs []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e IP \u5730\u5740\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 macAddress String \u7ed1\u5b9a Pod \u7684 Mac \u5730\u5740 attachMacs []String \u8be5\u4e3b IP \u4e0b\u5176\u4ed6\u9644\u5c5e Mac \u5730\u5740\uff08\u5b57\u6bb5\u5e9f\u5f03\u4e0d\u518d\u4f7f\u7528\uff09 containerID String \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 Container ID podType String \u7279\u6b8a\u5de5\u4f5c\u8d1f\u8f7d Pod\uff0c\u53ef\u4e3a StatefulSet \uff0c VirtualMachine \u6216\u7a7a","title":"IPSepc"},{"location":"reference/kube-ovn-api/#underlay","text":"","title":"Underlay \u914d\u7f6e"},{"location":"reference/kube-ovn-api/#vlan","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vlan metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VlanSpec Vlan \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VlanStatus Vlan \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5","title":"Vlan"},{"location":"reference/kube-ovn-api/#vlanspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 id Int Vlan tag \u53f7\uff0c\u53d6\u503c\u8303\u56f4\u4e3a 0~4096 provider String Vlan \u7ed1\u5b9a\u7684 ProviderNetwork \u540d\u79f0","title":"VlanSpec"},{"location":"reference/kube-ovn-api/#vlanstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 subnets []String Vlan \u7ed1\u5b9a\u7684\u5b50\u7f51\u5217\u8868 conditions []VlanCondition Vlan \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"VlanStatus"},{"location":"reference/kube-ovn-api/#providernetwork","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a ProviderNetwork metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec ProviderNetworkSpec ProviderNetwork \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status ProviderNetworkStatus ProviderNetwork \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5","title":"ProviderNetwork"},{"location":"reference/kube-ovn-api/#providernetworkspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 defaultInterface String \u8be5\u6865\u63a5\u7f51\u7edc\u9ed8\u8ba4\u4f7f\u7528\u7684\u7f51\u5361\u63a5\u53e3\u540d\u79f0 customInterfaces []CustomInterface \u8be5\u6865\u63a5\u7f51\u7edc\u7279\u6b8a\u4f7f\u7528\u7684\u7f51\u5361\u914d\u7f6e excludeNodes []String \u8be5\u6865\u63a5\u7f51\u7edc\u4e0d\u4f1a\u7ed1\u5b9a\u7684\u8282\u70b9\u540d\u79f0 exchangeLinkName Bool \u662f\u5426\u4ea4\u6362\u6865\u63a5\u7f51\u5361\u548c\u5bf9\u5e94 OVS \u7f51\u6865\u540d\u79f0","title":"ProviderNetworkSpec"},{"location":"reference/kube-ovn-api/#custominterface","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 interface String Underlay \u4f7f\u7528\u7f51\u5361\u63a5\u53e3\u540d\u79f0 nodes []String \u4f7f\u7528\u81ea\u5b9a\u4e49\u7f51\u5361\u63a5\u53e3\u7684\u8282\u70b9\u5217\u8868","title":"CustomInterface"},{"location":"reference/kube-ovn-api/#providernetworkstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool \u5f53\u524d\u6865\u63a5\u7f51\u7edc\u662f\u5426\u8fdb\u5165\u5c31\u7eea\u72b6\u6001 readyNodes []String \u6865\u63a5\u7f51\u7edc\u8fdb\u5165\u5c31\u7eea\u72b6\u6001\u7684\u8282\u70b9\u540d\u79f0 notReadyNodes []String \u6865\u63a5\u7f51\u7edc\u672a\u8fdb\u5165\u5c31\u7eea\u72b6\u6001\u7684\u8282\u70b9\u540d\u79f0 vlans []String \u6865\u63a5\u7f51\u7edc\u7ed1\u5b9a\u7684 Vlan \u540d\u79f0 conditions []ProviderNetworkCondition ProviderNetwork \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"ProviderNetworkStatus"},{"location":"reference/kube-ovn-api/#vpc","text":"","title":"Vpc \u5b9a\u4e49"},{"location":"reference/kube-ovn-api/#vpc_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vpc metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcSpec Vpc \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VpcStatus Vpc \u72b6\u6001\u4fe1\u606f\u5b57\u6bb5","title":"Vpc"},{"location":"reference/kube-ovn-api/#vpcspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 namespaces []String Vpc \u7ed1\u5b9a\u7684\u547d\u540d\u7a7a\u95f4\u5217\u8868 staticRoutes []*StaticRoute Vpc \u4e0b\u914d\u7f6e\u7684\u9759\u6001\u8def\u7531\u4fe1\u606f policyRoutes []*PolicyRoute Vpc \u4e0b\u914d\u7f6e\u7684\u7b56\u7565\u8def\u7531\u4fe1\u606f vpcPeerings []*VpcPeering Vpc \u4e92\u8054\u4fe1\u606f enableExternal Bool Vpc \u662f\u5426\u8fde\u63a5\u5230\u5916\u90e8\u4ea4\u6362\u673a","title":"VpcSpec"},{"location":"reference/kube-ovn-api/#staticroute","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 policy String \u8def\u7531\u7b56\u7565\uff0c\u53d6\u503c\u4e3a policySrc \u6216\u8005 policyDst cidr String \u8def\u7531 Cidr \u7f51\u6bb5 nextHopIP String \u8def\u7531\u4e0b\u4e00\u8df3\u4fe1\u606f","title":"StaticRoute"},{"location":"reference/kube-ovn-api/#policyroute","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 priority Int32 \u7b56\u7565\u8def\u7531\u4f18\u5148\u7ea7 match String \u7b56\u7565\u8def\u7531\u5339\u914d\u6761\u4ef6 action String \u7b56\u7565\u8def\u7531\u52a8\u4f5c\uff0c\u53d6\u503c\u4e3a allow \u3001 drop \u6216\u8005 reroute nextHopIP String \u7b56\u7565\u8def\u7531\u4e0b\u4e00\u8df3\u4fe1\u606f\uff0cECMP \u8def\u7531\u60c5\u51b5\u4e0b\u4e0b\u4e00\u8df3\u5730\u5740\u4f7f\u7528\u9017\u53f7\u9694\u5f00","title":"PolicyRoute"},{"location":"reference/kube-ovn-api/#vpcpeering","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 remoteVpc String Vpc \u4e92\u8054\u5bf9\u7aef Vpc \u540d\u79f0 localConnectIP String Vpc \u4e92\u8054\u672c\u7aef IP \u5730\u5740","title":"VpcPeering"},{"location":"reference/kube-ovn-api/#vpcstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VpcCondition Vpc \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 standby Bool \u6807\u8bc6 Vpc \u662f\u5426\u521b\u5efa\u5b8c\u6210\uff0cVpc \u4e0b\u7684 Subnet \u9700\u8981\u7b49 Vpc \u521b\u5efa\u5b8c\u6210\u8f6c\u6362\u518d\u7ee7\u7eed\u5904\u7406 default Bool \u662f\u5426\u662f\u9ed8\u8ba4 Vpc defaultLogicalSwitch String Vpc \u4e0b\u7684\u9ed8\u8ba4\u5b50\u7f51 router String Vpc \u5bf9\u5e94\u7684 logical-router \u540d\u79f0 tcpLoadBalancer String Vpc \u4e0b\u7684 TCP LB \u4fe1\u606f udpLoadBalancer String Vpc \u4e0b\u7684 UDP LB \u4fe1\u606f tcpSessionLoadBalancer String Vpc \u4e0b\u7684 TCP \u4f1a\u8bdd\u4fdd\u6301 LB \u4fe1\u606f udpSessionLoadBalancer String Vpc \u4e0b\u7684 UDP \u4f1a\u8bdd\u4fdd\u6301 LB \u4fe1\u606f subnets []String Vpc \u4e0b\u7684\u5b50\u7f51\u5217\u8868 vpcPeerings []String Vpc \u4e92\u8054\u7684\u5bf9\u7aef Vpc \u5217\u8868 enableExternal Bool Vpc \u662f\u5426\u8fde\u63a5\u5230\u5916\u90e8\u4ea4\u6362\u673a","title":"VpcStatus"},{"location":"reference/kube-ovn-api/#vpcnatgateway","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a VpcNatGateway metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcNatSpec Vpc \u7f51\u5173\u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5","title":"VpcNatGateway"},{"location":"reference/kube-ovn-api/#vpcnatspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vpc String Vpc \u7f51\u5173 Pod \u6240\u5728\u7684 Vpc \u540d\u79f0 subnet String Vpc \u7f51\u5173 Pod \u6240\u5c5e\u7684\u5b50\u7f51\u540d\u79f0 lanIp String Vpc \u7f51\u5173 Pod \u6307\u5b9a\u5206\u914d\u7684 IP \u5730\u5740 selector []String \u6807\u51c6 Kubernetes Selector \u5339\u914d\u4fe1\u606f tolerations []VpcNatToleration \u6807\u51c6 Kubernetes \u5bb9\u5fcd\u4fe1\u606f","title":"VpcNatSpec"},{"location":"reference/kube-ovn-api/#vpcnattoleration","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 key String \u5bb9\u5fcd\u6c61\u70b9\u7684 key \u4fe1\u606f operator String \u53d6\u503c\u4e3a Exists \u6216\u8005 Equal value String \u5bb9\u5fcd\u6c61\u70b9\u7684 value \u4fe1\u606f effect String \u5bb9\u5fcd\u6c61\u70b9\u7684\u4f5c\u7528\u6548\u679c\uff0c\u53d6\u503c\u4e3a NoExecute \u3001 NoSchedule \u6216\u8005 PreferNoSchedule tolerationSeconds Int64 \u6dfb\u52a0\u6c61\u70b9\u540e\uff0cPod \u8fd8\u80fd\u7ee7\u7eed\u5728\u8282\u70b9\u4e0a\u8fd0\u884c\u7684\u65f6\u95f4 \u4ee5\u4e0a\u5bb9\u5fcd\u5b57\u6bb5\u7684\u542b\u4e49\uff0c\u53ef\u4ee5\u53c2\u8003 Kubernetes \u5b98\u65b9\u6587\u6863 \u6c61\u70b9\u548c\u5bb9\u5fcd\u5ea6 \u3002","title":"VpcNatToleration"},{"location":"reference/kube-ovn-api/#iptableseip","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesEIP metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesEipSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesEIP \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesEipStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesEIP \u72b6\u6001\u4fe1\u606f","title":"IptablesEIP"},{"location":"reference/kube-ovn-api/#iptableseipspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 v4ip String IptablesEIP v4 \u5730\u5740 v6ip String IptablesEIP v6 \u5730\u5740 macAddress String IptablesEIP crd \u8bb0\u5f55\u5206\u914d\u7684 mac \u5730\u5740\uff0c\u6ca1\u6709\u5b9e\u9645\u4f7f\u7528 natGwDp String Vpc \u7f51\u5173\u540d\u79f0","title":"IptablesEipSpec"},{"location":"reference/kube-ovn-api/#iptableseipstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesEIP \u662f\u5426\u914d\u7f6e\u5b8c\u6210 ip String IptablesEIP \u4f7f\u7528\u7684 IP \u5730\u5740\uff0c\u76ee\u524d\u53ea\u652f\u6301\u4e86 IPv4 \u5730\u5740 redo String IptablesEIP crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 nat String IptablesEIP \u7684\u4f7f\u7528\u7c7b\u578b\uff0c\u53d6\u503c\u4e3a fip \u3001 snat \u6216\u8005 dnat conditions []IptablesEIPCondition IptablesEIP \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"IptablesEipStatus"},{"location":"reference/kube-ovn-api/#iptablesfiprule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesFIPRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesFIPRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesFIPRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesFIPRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesFIPRule \u72b6\u6001\u4fe1\u606f","title":"IptablesFIPRule"},{"location":"reference/kube-ovn-api/#iptablesfiprulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip String IptablesFIPRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 internalIp String IptablesFIPRule \u5bf9\u5e94\u7684\u5185\u90e8\u7684 IP \u5730\u5740","title":"IptablesFIPRuleSpec"},{"location":"reference/kube-ovn-api/#iptablesfiprulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesFIPRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesEIP \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesEIP \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesFIPRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesFIPRuleCondition IptablesFIPRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"IptablesFIPRuleStatus"},{"location":"reference/kube-ovn-api/#iptablessnatrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesSnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesSnatRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesSnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesSnatRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesSnatRule \u72b6\u6001\u4fe1\u606f","title":"IptablesSnatRule"},{"location":"reference/kube-ovn-api/#iptablessnatrulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip String IptablesSnatRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 internalIp String IptablesSnatRule \u5bf9\u5e94\u7684\u5185\u90e8\u7684 IP \u5730\u5740","title":"IptablesSnatRuleSpec"},{"location":"reference/kube-ovn-api/#iptablessnatrulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesSnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesSnatRule \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesSnatRule \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesSnatRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesSnatRuleCondition IptablesSnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"IptablesSnatRuleStatus"},{"location":"reference/kube-ovn-api/#iptablesdnatrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a IptablesDnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec IptablesDnatRuleSpec Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesDnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status IptablesDnatRuleStatus Vpc \u7f51\u5173\u4f7f\u7528\u7684 IptablesDnatRule \u72b6\u6001\u4fe1\u606f","title":"IptablesDnatRule"},{"location":"reference/kube-ovn-api/#iptablesdnatrulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 eip Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684 IptablesEIP \u540d\u79f0 externalPort Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5916\u90e8\u7aef\u53e3 protocol Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u7684\u534f\u8bae\u7c7b\u578b internalIp Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5185\u90e8 IP \u5730\u5740 internalPort Sting Vpc \u7f51\u5173\u914d\u7f6e IptablesDnatRule \u4f7f\u7528\u7684\u5185\u90e8\u7aef\u53e3","title":"IptablesDnatRuleSpec"},{"location":"reference/kube-ovn-api/#iptablesdnatrulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool IptablesDnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4ip String IptablesDnatRule \u4f7f\u7528\u7684 v4 IP \u5730\u5740 v6ip String IptablesDnatRule \u4f7f\u7528\u7684 v6 IP \u5730\u5740 natGwDp String Vpc \u7f51\u5173\u540d\u79f0 redo String IptablesDnatRule crd \u521b\u5efa\u6216\u8005\u66f4\u65b0\u65f6\u95f4 conditions []IptablesDnatRuleCondition IptablesDnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"IptablesDnatRuleStatus"},{"location":"reference/kube-ovn-api/#vpcdns","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a VpcDns metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VpcDnsSpec VpcDns \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VpcDnsStatus VpcDns \u72b6\u6001\u4fe1\u606f","title":"VpcDns"},{"location":"reference/kube-ovn-api/#vpcdnsspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vpc String VpcDns \u6240\u5728\u7684 Vpc \u540d\u79f0 subnet String VpcDns Pod \u5206\u914d\u5730\u5740\u7684 Subnet \u540d\u79f0","title":"VpcDnsSpec"},{"location":"reference/kube-ovn-api/#vpcdnsstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VpcDnsCondition VpcDns \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 active Bool VpcDns \u662f\u5426\u6b63\u5728\u4f7f\u7528 VpcDns \u7684\u8be6\u7ec6\u4f7f\u7528\u6587\u6863\uff0c\u53ef\u4ee5\u53c2\u8003 \u81ea\u5b9a\u4e49 VPC DNS \u3002","title":"VpcDnsStatus"},{"location":"reference/kube-ovn-api/#switchlbrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a SwitchLBRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SwitchLBRuleSpec SwitchLBRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SwitchLBRuleStatus SwitchLBRule \u72b6\u6001\u4fe1\u606f","title":"SwitchLBRule"},{"location":"reference/kube-ovn-api/#switchlbrulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 vip String SwitchLBRule \u914d\u7f6e\u7684 vip \u5730\u5740 namespace String SwitchLBRule \u7684\u547d\u540d\u7a7a\u95f4 selector []String \u6807\u51c6 Kubernetes Selector \u5339\u914d\u4fe1\u606f sessionAffinity String \u6807\u51c6 Kubernetes Service \u4e2d sessionAffinity \u53d6\u503c ports []SlrPort SwitchLBRule \u7aef\u53e3\u5217\u8868 SwitchLBRule \u7684\u8be6\u7ec6\u914d\u7f6e\u4fe1\u606f\uff0c\u53ef\u4ee5\u53c2\u8003 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u3002","title":"SwitchLBRuleSpec"},{"location":"reference/kube-ovn-api/#slrport","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 name String \u7aef\u53e3\u540d\u79f0 port Int32 \u7aef\u53e3\u53f7 targetPort Int32 \u76ee\u6807\u7aef\u53e3\u53f7 protocol String \u534f\u8bae\u7c7b\u578b","title":"SlrPort"},{"location":"reference/kube-ovn-api/#switchlbrulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []SwitchLBRuleCondition SwitchLBRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ports String SwitchLBRule \u7aef\u53e3\u4fe1\u606f service String SwitchLBRule \u63d0\u4f9b\u670d\u52a1\u7684 service \u540d\u79f0","title":"SwitchLBRuleStatus"},{"location":"reference/kube-ovn-api/#vip","text":"","title":"\u5b89\u5168\u7ec4\u4e0e Vip"},{"location":"reference/kube-ovn-api/#securitygroup","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a SecurityGroup metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec SecurityGroupSpec \u5b89\u5168\u7ec4\u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status SecurityGroupStatus \u5b89\u5168\u7ec4\u72b6\u6001\u4fe1\u606f","title":"SecurityGroup"},{"location":"reference/kube-ovn-api/#securitygroupspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ingressRules []*SgRule \u5165\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 egressRules []*SgRule \u51fa\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 allowSameGroupTraffic Bool \u540c\u4e00\u5b89\u5168\u7ec4\u5185\u7684 lsp \u662f\u5426\u53ef\u4ee5\u4e92\u901a\uff0c\u4ee5\u53ca\u6d41\u91cf\u89c4\u5219\u662f\u5426\u9700\u8981\u66f4\u65b0","title":"SecurityGroupSpec"},{"location":"reference/kube-ovn-api/#sgrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ipVersion String IP \u7248\u672c\u53f7\uff0c\u53d6\u503c\u4e3a ipv4 \u6216\u8005 ipv6 protocol String \u53d6\u503c\u4e3a all \u3001 icmp \u3001 tcp \u6216\u8005 udp priority Int Acl \u4f18\u5148\u7ea7\uff0c\u53d6\u503c\u8303\u56f4\u4e3a 1-200\uff0c\u6570\u503c\u8d8a\u5c0f\uff0c\u4f18\u5148\u7ea7\u8d8a\u9ad8 remoteType String \u53d6\u503c\u4e3a address \u6216\u8005 securityGroup remoteAddress String \u5bf9\u7aef\u5730\u5740 remoteSecurityGroup String \u5bf9\u7aef\u5b89\u5168\u7ec4 portRangeMin Int \u7aef\u53e3\u8303\u56f4\u8d77\u59cb\u503c\uff0c\u6700\u5c0f\u53d6\u503c\u4e3a 1 portRangeMax Int \u7aef\u53e3\u8303\u56f4\u6700\u5927\u503c\uff0c\u6700\u5927\u53d6\u503c\u4e3a 65535 policy String \u53d6\u503c\u4e3a allow \u6216\u8005 drop","title":"SgRule"},{"location":"reference/kube-ovn-api/#securitygroupstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 portGroup String \u5b89\u5168\u7ec4\u5bf9\u5e94\u7684 port-group \u540d\u79f0 allowSameGroupTraffic Bool \u540c\u4e00\u5b89\u5168\u7ec4\u5185\u7684 lsp \u662f\u5426\u53ef\u4ee5\u4e92\u901a\uff0c\u4ee5\u53ca\u5b89\u5168\u7ec4\u7684\u6d41\u91cf\u89c4\u5219\u662f\u5426\u9700\u8981\u66f4\u65b0 ingressMd5 String \u5165\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 MD5 \u53d6\u503c egressMd5 String \u51fa\u65b9\u5411\u5b89\u5168\u7ec4\u89c4\u5219 MD5 \u53d6\u503c ingressLastSyncSuccess Bool \u5165\u65b9\u5411\u89c4\u5219\u4e0a\u4e00\u6b21\u540c\u6b65\u662f\u5426\u6210\u529f egressLastSyncSuccess Bool \u51fa\u65b9\u5411\u89c4\u5219\u4e0a\u4e00\u6b21\u540c\u6b65\u662f\u5426\u6210\u529f","title":"SecurityGroupStatus"},{"location":"reference/kube-ovn-api/#vip_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a Vip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec VipSpec Vip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status VipStatus Vip \u72b6\u6001\u4fe1\u606f","title":"Vip"},{"location":"reference/kube-ovn-api/#vipspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 namespace String Vip \u6240\u5728\u547d\u540d\u7a7a\u95f4 subnet String Vip \u6240\u5c5e\u5b50\u7f51 v4ip String Vip v4 IP \u5730\u5740 v6ip String Vip v6 IP \u5730\u5740 macAddress String Vip mac \u5730\u5740 parentV4ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 parentV6ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 parentMac String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 attachSubnets []String \u8be5\u5b57\u6bb5\u5e9f\u5f03\uff0c\u4e0d\u518d\u4f7f\u7528","title":"VipSpec"},{"location":"reference/kube-ovn-api/#vipstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []VipCondition Vip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 ready Bool Vip \u662f\u5426\u51c6\u5907\u597d v4ip String Vip v4 IP \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 v6ip String Vip v6 IP \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 mac String Vip mac \u5730\u5740\uff0c\u5e94\u8be5\u548c spec \u5b57\u6bb5\u53d6\u503c\u4e00\u81f4 pv4ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 pv6ip String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528 pmac String \u76ee\u524d\u6ca1\u6709\u4f7f\u7528","title":"VipStatus"},{"location":"reference/kube-ovn-api/#ovneip","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnEip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnEipSpec \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnEip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnEipStatus \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnEip \u72b6\u6001\u4fe1\u606f","title":"OvnEip"},{"location":"reference/kube-ovn-api/#ovneipspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 externalSubnet String OvnEip \u6240\u5728\u7684\u5b50\u7f51\u540d\u79f0 v4ip String OvnEip IP \u5730\u5740 macAddress String OvnEip Mac \u5730\u5740 type String OvnEip \u4f7f\u7528\u7c7b\u578b\uff0c\u53d6\u503c\u6709 fip \u3001 snat \u6216\u8005 lrp","title":"OvnEipSpec"},{"location":"reference/kube-ovn-api/#ovneipstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 conditions []OvnEipCondition \u9ed8\u8ba4 Vpc OvnEip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 v4ip String OvnEip \u4f7f\u7528\u7684 v4 IP \u5730\u5740 macAddress String OvnEip \u4f7f\u7528\u7684 Mac \u5730\u5740","title":"OvnEipStatus"},{"location":"reference/kube-ovn-api/#ovnfip","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnFip metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnFipSpec \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnFip \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnFipStatus \u9ed8\u8ba4 Vpc \u4f7f\u7528 OvnFip \u72b6\u6001\u4fe1\u606f","title":"OvnFip"},{"location":"reference/kube-ovn-api/#ovnfipspec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ovnEip String OvnFip \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 ipName String OvnFip \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 IP crd \u540d\u79f0","title":"OvnFipSpec"},{"location":"reference/kube-ovn-api/#ovnfipstatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool OvnFip \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4Eip String OvnFip \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 v4Ip String OvnFip \u5f53\u524d\u4f7f\u7528\u7684 OvnEip \u5730\u5740 macAddress String OvnFip \u914d\u7f6e\u7684 Mac \u5730\u5740 vpc String OvnFip \u6240\u5728\u7684 Vpc \u540d\u79f0 conditions []OvnFipCondition OvnFip \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49","title":"OvnFipStatus"},{"location":"reference/kube-ovn-api/#ovnsnatrule","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 apiVersion String \u6807\u51c6 Kubernetes \u7248\u672c\u4fe1\u606f\u5b57\u6bb5\uff0c\u6240\u6709\u81ea\u5b9a\u4e49\u8d44\u6e90\u8be5\u503c\u5747\u4e3a kubeovn.io/v1 kind String \u6807\u51c6 Kubernetes \u8d44\u6e90\u7c7b\u578b\u5b57\u6bb5\uff0c\u672c\u8d44\u6e90\u6240\u6709\u5b9e\u4f8b\u8be5\u503c\u5747\u4e3a OvnSnatRule metadata ObjectMeta \u6807\u51c6 Kubernetes \u8d44\u6e90\u5143\u6570\u636e\u4fe1\u606f spec OvnSnatRuleSpec \u9ed8\u8ba4 Vpc OvnSnatRule \u5177\u4f53\u914d\u7f6e\u4fe1\u606f\u5b57\u6bb5 status OvnSnatRuleStatus \u9ed8\u8ba4 Vpc OvnSnatRule \u72b6\u6001\u4fe1\u606f","title":"OvnSnatRule"},{"location":"reference/kube-ovn-api/#ovnsnatrulespec","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ovnEip String OvnSnatRule \u7ed1\u5b9a\u7684 OvnEip \u540d\u79f0 vpcSubnet String OvnSnatRule \u914d\u7f6e\u7684\u5b50\u7f51\u540d\u79f0 ipName String OvnSnatRule \u7ed1\u5b9a Pod \u5bf9\u5e94\u7684 IP crd \u540d\u79f0","title":"OvnSnatRuleSpec"},{"location":"reference/kube-ovn-api/#ovnsnatrulestatus","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 ready Bool OvnSnatRule \u662f\u5426\u914d\u7f6e\u5b8c\u6210 v4Eip String OvnSnatRule \u7ed1\u5b9a\u7684 OvnEip \u5730\u5740 v4IpCidr String \u5728 logical-router \u4e2d\u914d\u7f6e snat \u8f6c\u6362\u4f7f\u7528\u7684 cidr \u5730\u5740 vpc String OvnSnatRule \u6240\u5728\u7684 Vpc \u540d\u79f0 conditions []OvnSnatRuleCondition OvnSnatRule \u72b6\u6001\u53d8\u5316\u4fe1\u606f\uff0c\u5177\u4f53\u5b57\u6bb5\u53c2\u8003\u6587\u6863\u5f00\u5934 Condition \u5b9a\u4e49 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OvnSnatRuleStatus"},{"location":"reference/kube-ovn-pinger-args/","text":"Kube-OVN-Pinger \u53c2\u6570\u53c2\u8003 \u00b6 \u57fa\u4e8e Kube-OVN v1.12.0 \u7248\u672c\uff0c\u6574\u7406\u4e86 Kube-ovn-pinger \u652f\u6301\u7684\u53c2\u6570\uff0c\u5217\u51fa\u53c2\u6570\u5b9a\u4e49\u5404\u5b57\u6bb5\u7684\u53d6\u503c\u7c7b\u578b\uff0c\u542b\u4e49\u548c\u9ed8\u8ba4\u503c\uff0c\u4ee5\u4f9b\u53c2\u8003 \u53c2\u6570\u63cf\u8ff0 \u00b6 \u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 \u9ed8\u8ba4\u503c port Int metrics \u7aef\u53e3 8080 kubeconfig String \u5177\u6709\u8ba4\u8bc1\u4fe1\u606f\u7684 kubeconfig \u6587\u4ef6\u8def\u5f84\uff0c \u5982\u679c\u672a\u8bbe\u7f6e\uff0c\u4f7f\u7528 inCluster \u4ee4\u724c\u3002 \"\" ds-namespace String kube-ovn-pinger \u5b88\u62a4\u8fdb\u7a0b\u547d\u540d\u7a7a\u95f4 \"kube-system\" ds-name String kube-ovn-pinger \u5b88\u62a4\u8fdb\u7a0b\u540d\u5b57 \"kube-ovn-pinger\" interval Int \u8fde\u7eed ping \u4e4b\u95f4\u7684\u95f4\u9694\u79d2\u6570 5 mode String \u670d\u52a1\u5668\u6216\u5de5\u4f5c\u6a21\u5f0f \"server\" exit-code Int \u5931\u8d25\u65f6\u9000\u51fa\u4ee3\u7801 0 internal-dns String \u4ece pod \u5185\u89e3\u6790\u5185\u90e8 dns \"kubernetes.default\" external-dns String \u4ece pod \u5185\u89e3\u6790\u5916\u90e8 dns \"\" external-address String \u68c0\u67e5\u4e0e\u5916\u90e8\u5730\u5740\u7684 ping \u8fde\u901a \"114.114.114.114\" network-mode String \u5f53\u524d\u96c6\u7fa4\u4f7f\u7528\u7684 cni \u63d2\u4ef6 \"kube-ovn\" enable-metrics Bool \u662f\u5426\u652f\u6301 metrics \u67e5\u8be2 true ovs.timeout Int \u5bf9 OVS \u7684 JSON-RPC \u8bf7\u6c42\u8d85\u65f6\u3002 2 system.run.dir String OVS \u9ed8\u8ba4\u8fd0\u884c\u76ee\u5f55\u3002 \"/var/run/openvswitch\" database.vswitch.name String OVS \u6570\u636e\u5e93\u7684\u540d\u79f0\u3002 \"Open_vSwitch\" database.vswitch.socket.remote String JSON-RPC unix \u5957\u63a5\u5b57\u5230 OVS \u6570\u636e\u5e93\u3002 \"unix:/var/run/openvswitch/db.sock\" database.vswitch.file.data.path String OVS \u6570\u636e\u5e93\u6587\u4ef6\u3002 \"/etc/openvswitch/conf.db\" database.vswitch.file.log.path String OVS \u6570\u636e\u5e93\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/openvswitch/ovsdb-server.log\" database.vswitch.file.pid.path String OVS \u6570\u636e\u5e93\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/openvswitch/ovsdb-server.pid\" database.vswitch.file.system.id.path String OVS \u7cfb\u7edf\u6807\u8bc6\u6587\u4ef6\u3002 \"/etc/openvswitch/system-id.conf\" service.vswitchd.file.log.path String OVS vswitchd \u5b88\u62a4\u8fdb\u7a0b\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/openvswitch/ovs-vswitchd.log\" service.vswitchd.file.pid.path String OVS vswitchd \u5b88\u62a4\u8fdb\u7a0b\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/openvswitch/ovs-vswitchd.pid\" service.ovncontroller.file.log.path String OVN \u63a7\u5236\u5668\u5b88\u62a4\u8fdb\u7a0b\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/ovn/ovn-controller.log\" service.ovncontroller.file.pid.path String OVN \u63a7\u5236\u5668\u5b88\u62a4\u8fdb\u7a0b\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/ovn/ovn-controller.pid\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kube-ovn-pinger \u53c2\u6570\u63cf\u8ff0"},{"location":"reference/kube-ovn-pinger-args/#kube-ovn-pinger","text":"\u57fa\u4e8e Kube-OVN v1.12.0 \u7248\u672c\uff0c\u6574\u7406\u4e86 Kube-ovn-pinger \u652f\u6301\u7684\u53c2\u6570\uff0c\u5217\u51fa\u53c2\u6570\u5b9a\u4e49\u5404\u5b57\u6bb5\u7684\u53d6\u503c\u7c7b\u578b\uff0c\u542b\u4e49\u548c\u9ed8\u8ba4\u503c\uff0c\u4ee5\u4f9b\u53c2\u8003","title":"Kube-OVN-Pinger \u53c2\u6570\u53c2\u8003"},{"location":"reference/kube-ovn-pinger-args/#_1","text":"\u5c5e\u6027\u540d\u79f0 \u7c7b\u578b \u63cf\u8ff0 \u9ed8\u8ba4\u503c port Int metrics \u7aef\u53e3 8080 kubeconfig String \u5177\u6709\u8ba4\u8bc1\u4fe1\u606f\u7684 kubeconfig \u6587\u4ef6\u8def\u5f84\uff0c \u5982\u679c\u672a\u8bbe\u7f6e\uff0c\u4f7f\u7528 inCluster \u4ee4\u724c\u3002 \"\" ds-namespace String kube-ovn-pinger \u5b88\u62a4\u8fdb\u7a0b\u547d\u540d\u7a7a\u95f4 \"kube-system\" ds-name String kube-ovn-pinger \u5b88\u62a4\u8fdb\u7a0b\u540d\u5b57 \"kube-ovn-pinger\" interval Int \u8fde\u7eed ping \u4e4b\u95f4\u7684\u95f4\u9694\u79d2\u6570 5 mode String \u670d\u52a1\u5668\u6216\u5de5\u4f5c\u6a21\u5f0f \"server\" exit-code Int \u5931\u8d25\u65f6\u9000\u51fa\u4ee3\u7801 0 internal-dns String \u4ece pod \u5185\u89e3\u6790\u5185\u90e8 dns \"kubernetes.default\" external-dns String \u4ece pod \u5185\u89e3\u6790\u5916\u90e8 dns \"\" external-address String \u68c0\u67e5\u4e0e\u5916\u90e8\u5730\u5740\u7684 ping \u8fde\u901a \"114.114.114.114\" network-mode String \u5f53\u524d\u96c6\u7fa4\u4f7f\u7528\u7684 cni \u63d2\u4ef6 \"kube-ovn\" enable-metrics Bool \u662f\u5426\u652f\u6301 metrics \u67e5\u8be2 true ovs.timeout Int \u5bf9 OVS \u7684 JSON-RPC \u8bf7\u6c42\u8d85\u65f6\u3002 2 system.run.dir String OVS \u9ed8\u8ba4\u8fd0\u884c\u76ee\u5f55\u3002 \"/var/run/openvswitch\" database.vswitch.name String OVS \u6570\u636e\u5e93\u7684\u540d\u79f0\u3002 \"Open_vSwitch\" database.vswitch.socket.remote String JSON-RPC unix \u5957\u63a5\u5b57\u5230 OVS \u6570\u636e\u5e93\u3002 \"unix:/var/run/openvswitch/db.sock\" database.vswitch.file.data.path String OVS \u6570\u636e\u5e93\u6587\u4ef6\u3002 \"/etc/openvswitch/conf.db\" database.vswitch.file.log.path String OVS \u6570\u636e\u5e93\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/openvswitch/ovsdb-server.log\" database.vswitch.file.pid.path String OVS \u6570\u636e\u5e93\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/openvswitch/ovsdb-server.pid\" database.vswitch.file.system.id.path String OVS \u7cfb\u7edf\u6807\u8bc6\u6587\u4ef6\u3002 \"/etc/openvswitch/system-id.conf\" service.vswitchd.file.log.path String OVS vswitchd \u5b88\u62a4\u8fdb\u7a0b\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/openvswitch/ovs-vswitchd.log\" service.vswitchd.file.pid.path String OVS vswitchd \u5b88\u62a4\u8fdb\u7a0b\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/openvswitch/ovs-vswitchd.pid\" service.ovncontroller.file.log.path String OVN \u63a7\u5236\u5668\u5b88\u62a4\u8fdb\u7a0b\u65e5\u5fd7\u6587\u4ef6\u3002 \"/var/log/ovn/ovn-controller.log\" service.ovncontroller.file.pid.path String OVN \u63a7\u5236\u5668\u5b88\u62a4\u8fdb\u7a0b\u8fdb\u7a0b ID \u6587\u4ef6\u3002 \"/var/run/ovn/ovn-controller.pid\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u53c2\u6570\u63cf\u8ff0"},{"location":"reference/metrics/","text":"Kube-OVN \u76d1\u63a7\u6307\u6807 \u00b6 \u672c\u6587\u6863\u5217\u4e3e Kube-OVN \u6240\u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\u3002 ovn-monitor \u00b6 OVN \u81ea\u8eab\u72b6\u6001\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge kube_ovn_ovn_status OVN \u89d2\u8272\u72b6\u6001\uff0c (2) \u4e3a follower\uff1b (1) \u4e3a leader, (0) \u4e3a\u5f02\u5e38\u72b6\u6001\u3002 Gauge kube_ovn_failed_req_count OVN \u5931\u8d25\u8bf7\u6c42\u6570\u91cf\u3002 Gauge kube_ovn_log_file_size_bytes OVN \u7ec4\u4ef6\u65e5\u5fd7\u6587\u4ef6\u5927\u5c0f\u3002 Gauge kube_ovn_db_file_size_bytes OVN \u7ec4\u4ef6\u6570\u636e\u5e93\u6587\u4ef6\u5927\u5c0f\u3002 Gauge kube_ovn_chassis_info OVN chassis \u72b6\u6001 (1) \u8fd0\u884c\u4e2d\uff0c(0) \u505c\u6b62\u3002 Gauge kube_ovn_db_status OVN \u6570\u636e\u5e93\u72b6\u6001, (1) \u4e3a\u6b63\u5e38\uff1b (0) \u4e3a\u5f02\u5e38\u3002 Gauge kube_ovn_logical_switch_info OVN logical switch \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b logical switch \u540d\u5b57\u3002 Gauge kube_ovn_logical_switch_external_id OVN logical switch external_id \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b external-id \u5185\u5bb9\u3002 Gauge kube_ovn_logical_switch_port_binding OVN logical switch \u548c logical switch port \u5173\u8054\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u901a\u8fc7\u6807\u7b7e\u8fdb\u884c\u5173\u8054\u3002 Gauge kube_ovn_logical_switch_tunnel_key \u548c OVN logical switch \u5173\u8054\u7684 tunnel key \u4fe1\u606f\u3002 Gauge kube_ovn_logical_switch_ports_num OVN logical switch \u4e0a logical port \u7684\u6570\u91cf\u3002 Gauge kube_ovn_logical_switch_port_info OVN logical switch port \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5177\u4f53\u4fe1\u606f\u3002 Gauge kube_ovn_logical_switch_port_tunnel_key \u548c OVN logical switch port \u5173\u8054\u7684 tunnel key \u4fe1\u606f\u3002 Gauge kube_ovn_cluster_enabled (1) OVN \u6570\u636e\u5e93\u4e3a\u96c6\u7fa4\u6a21\u5f0f\uff1b (0) OVN \u6570\u636e\u5e93\u4e3a\u975e\u96c6\u7fa4\u6a21\u5f0f\u3002 Gauge kube_ovn_cluster_role \u6bcf\u4e2a\u6570\u636e\u5e93\u5b9e\u4f8b\u7684\u89d2\u8272\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u89d2\u8272\u4fe1\u606f\u3002 Gauge kube_ovn_cluster_status \u6bcf\u4e2a\u6570\u636e\u5e93\u5b9e\u4f8b\u7684\u72b6\u6001\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u72b6\u6001\u4fe1\u606f\u3002 Gauge kube_ovn_cluster_term RAFT term \u4fe1\u606f\u3002 Gauge kube_ovn_cluster_leader_self \u5f53\u524d\u6570\u636e\u5e93\u5b9e\u4f8b\u662f\u5426\u4e3a leader (1) \u662f\uff0c (0) \u4e0d\u662f\u3002 Gauge kube_ovn_cluster_vote_self \u5f53\u524d\u6570\u636e\u5e93\u5b9e\u4f8b\u662f\u5426\u9009\u4e3e\u81ea\u5df1\u4e3a leader (1) \u662f\uff0c (0) \u4e0d\u662f\u3002 Gauge kube_ovn_cluster_election_timer \u5f53\u524d election timer \u503c\u3002 Gauge kube_ovn_cluster_log_not_committed \u672a commit \u7684 RAFT \u65e5\u5fd7\u6570\u91cf\u3002 Gauge kube_ovn_cluster_log_not_applied \u672a apply \u7684 RAFT \u65e5\u5fd7\u6570\u91cf\u3002 Gauge kube_ovn_cluster_log_index_start \u5f53\u524d RAFT \u65e5\u5fd7\u6761\u76ee\u7684\u8d77\u59cb\u503c\u3002 Gauge kube_ovn_cluster_log_index_next RAFT \u65e5\u5fd7\u6761\u76ee\u7684\u4e0b\u4e00\u4e2a\u503c\u3002 Gauge kube_ovn_cluster_inbound_connections_total \u5f53\u524d\u5b9e\u4f8b\u7684\u5165\u5411\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_outbound_connections_total \u5f53\u524d\u5b9e\u4f8b\u7684\u51fa\u5411\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_inbound_connections_error_total \u5f53\u524d\u5b9e\u4f8b\u7684\u5165\u5411\u9519\u8bef\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_outbound_connections_error_total \u5f53\u524d\u5b9e\u4f8b\u7684\u51fa\u5411\u9519\u8bef\u8fde\u63a5\u6570\u91cf\u3002 ovs-monitor \u00b6 ovsdb \u548c vswitchd \u81ea\u8eab\u72b6\u6001\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge ovs_status OVS \u5065\u5eb7\u72b6\u6001\uff0c (1) \u4e3a\u6b63\u5e38\uff0c(0) \u4e3a\u5f02\u5e38\u3002 Gauge ovs_info OVS \u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge failed_req_count OVS \u5931\u8d25\u8bf7\u6c42\u6570\u91cf\u3002 Gauge log_file_size OVS \u7ec4\u4ef6\u65e5\u5fd7\u6587\u4ef6\u5927\u5c0f\u3002 Gauge db_file_size OVS \u7ec4\u4ef6\u6570\u636e\u5e93\u6587\u4ef6\u5927\u5c0f\u3002 Gauge datapath Datapath \u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge dp_total \u5f53\u524d OVS \u4e2d datapath \u6570\u91cf\u3002 Gauge dp_if Datapath \u63a5\u53e3\u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge dp_if_total \u5f53\u524d datapath \u4e2d port \u6570\u91cf\u3002 Gauge dp_flows_total Datapath \u4e2d flow \u6570\u91cf\u3002 Gauge dp_flows_lookup_hit Datapath \u4e2d\u547d\u4e2d\u5f53\u524d flow \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_flows_lookup_missed Datapath \u4e2d\u672a\u547d\u4e2d\u5f53\u524d flow \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_flows_lookup_lost Datapath \u4e2d\u9700\u8981\u53d1\u9001\u7ed9 userspace \u5904\u7406\u7684\u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_masks_hit Datapath \u4e2d\u547d\u4e2d\u5f53\u524d mask \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_masks_total Datapath \u4e2d mask \u7684\u6570\u91cf\u3002 Gauge dp_masks_hit_ratio Datapath \u4e2d \u6570\u636e\u5305\u547d\u4e2d mask \u7684\u6bd4\u7387\u3002 Gauge interface OVS \u63a5\u53e3\u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge interface_admin_state \u63a5\u53e3\u7ba1\u7406\u72b6\u6001\u4fe1\u606f (0) \u4e3a down, (1) \u4e3a up, (2) \u4e3a\u5176\u4ed6\u72b6\u6001\u3002 Gauge interface_link_state \u63a5\u53e3\u94fe\u8def\u72b6\u6001\u4fe1\u606f (0) \u4e3a down, (1) \u4e3a up, (2) \u4e3a\u5176\u4ed6\u72b6\u6001\u3002 Gauge interface_mac_in_use OVS Interface \u4f7f\u7528\u7684 MAC \u5730\u5740 Gauge interface_mtu OVS Interface \u4f7f\u7528\u7684 MTU\u3002 Gauge interface_of_port OVS Interface \u5173\u8054\u7684 OpenFlow Port ID\u3002 Gauge interface_if_index OVS Interface \u5173\u8054\u7684 Index\u3002 Gauge interface_tx_packets OVS Interface \u53d1\u9001\u5305\u6570\u91cf\u3002 Gauge interface_tx_bytes OVS Interface \u53d1\u9001\u5305\u5927\u5c0f\u3002 Gauge interface_rx_packets OVS Interface \u63a5\u6536\u5305\u6570\u91cf\u3002 Gauge interface_rx_bytes OVS Interface \u63a5\u6536\u5305\u5927\u5c0f\u3002 Gauge interface_rx_crc_err OVS Interface \u63a5\u6536\u5305\u6821\u9a8c\u548c\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_dropped OVS Interface \u63a5\u6536\u5305\u4e22\u5f03\u6570\u91cf\u3002 Gauge interface_rx_errors OVS Interface \u63a5\u6536\u5305\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_frame_err OVS Interface \u63a5\u6536\u5e27\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_missed_err OVS Interface \u63a5\u6536\u5305 miss \u6570\u91cf\u3002 Gauge interface_rx_over_err OVS Interface \u63a5\u6536\u5305 overrun \u6570\u91cf\u3002 Gauge interface_tx_dropped OVS Interface \u53d1\u9001\u5305\u4e22\u5f03\u6570\u91cf\u3002 Gauge interface_tx_errors OVS Interface \u53d1\u9001\u5305\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_collisions OVS interface \u51b2\u7a81\u6570\u91cf\u3002 kube-ovn-pinger \u00b6 \u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge pinger_ovs_up \u8282\u70b9 OVS \u8fd0\u884c\u3002 Gauge pinger_ovs_down \u8282\u70b9 OVS \u505c\u6b62\u3002 Gauge pinger_ovn_controller_up \u8282\u70b9 ovn-controller \u8fd0\u884c\u3002 Gauge pinger_ovn_controller_down \u8282\u70b9 ovn-controller \u505c\u6b62\u3002 Gauge pinger_inconsistent_port_binding OVN-SB \u91cc portbinding \u6570\u91cf\u548c\u4e3b\u673a OVS interface \u4e0d\u4e00\u81f4\u7684\u6570\u91cf\u3002 Gauge pinger_apiserver_healthy kube-ovn-pinger \u53ef\u4ee5\u8054\u901a apiserver\u3002 Gauge pinger_apiserver_unhealthy kube-ovn-pinger \u65e0\u6cd5\u8054\u901a apiserver\u3002 Histogram pinger_apiserver_latency_ms kube-ovn-pinger \u8bbf\u95ee apiserver \u5ef6\u8fdf\u3002 Gauge pinger_internal_dns_healthy kube-ovn-pinger \u53ef\u4ee5\u89e3\u6790\u5185\u90e8\u57df\u540d\u3002 Gauge pinger_internal_dns_unhealthy kube-ovn-pinger \u65e0\u6cd5\u89e3\u6790\u5185\u90e8\u57df\u540d\u3002 Histogram pinger_internal_dns_latency_ms kube-ovn-pinger \u89e3\u6790\u5185\u90e8\u57df\u540d\u5ef6\u8fdf\u3002 Gauge pinger_external_dns_health kube-ovn-pinger \u53ef\u4ee5\u89e3\u6790\u5916\u90e8\u57df\u540d\u3002 Gauge pinger_external_dns_unhealthy kube-ovn-pinger \u65e0\u6cd5\u89e3\u6790\u5916\u90e8\u57df\u540d\u3002 Histogram pinger_external_dns_latency_ms kube-ovn-pinger \u89e3\u6790\u5916\u90e8\u57df\u540d\u5ef6\u8fdf\u3002 Histogram pinger_pod_ping_latency_ms kube-ovn-pinger ping Pod \u5ef6\u8fdf\u3002 Gauge pinger_pod_ping_lost_total kube-ovn-pinger ping Pod \u4e22\u5305\u6570\u91cf\u3002 Gauge pinger_pod_ping_count_total kube-ovn-pinger ping Pod \u6570\u91cf\u3002 Histogram pinger_node_ping_latency_ms kube-ovn-pinger ping Node \u5ef6\u8fdf\u3002 Gauge pinger_node_ping_lost_total kube-ovn-pinger ping Node \u4e22\u5305\u3002 Gauge pinger_node_ping_count_total kube-ovn-pinger ping Node \u6570\u91cf\u3002 Histogram pinger_external_ping_latency_ms kube-ovn-pinger ping \u5916\u90e8\u5730\u5740 \u5ef6\u8fdf\u3002 Gauge pinger_external_lost_total kube-ovn-pinger ping \u5916\u90e8\u4e22\u5305\u6570\u91cf\u3002 kube-ovn-controller \u00b6 kube-ovn-controller \u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Histogram rest_client_request_latency_seconds \u8bf7\u6c42 apiserver \u5ef6\u8fdf\u3002 Counter rest_client_requests_total \u8bf7\u6c42 apiserver \u6570\u91cf\u3002 Counter lists_total API list \u8bf7\u6c42\u6570\u91cf\u3002 Summary list_duration_seconds API list \u8bf7\u6c42\u5ef6\u8fdf\u3002 Summary items_per_list API list \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Counter watches_total API watch \u8bf7\u6c42\u6570\u91cf\u3002 Counter short_watches_total \u77ed\u65f6\u95f4 API watch \u8bf7\u6c42\u6570\u91cf\u3002 Summary watch_duration_seconds API watch \u6301\u7eed\u65f6\u95f4\u3002 Summary items_per_watch API watch \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Gauge last_resource_version \u6700\u65b0\u7684 resource version\u3002 Histogram ovs_client_request_latency_milliseconds \u8bf7\u6c42 OVN \u7ec4\u4ef6\u5ef6\u8fdf\u3002 Gauge subnet_available_ip_count \u5b50\u7f51\u53ef\u7528 IP \u6570\u91cf\u3002 Gauge subnet_used_ip_count \u5b50\u7f51\u5df2\u7528 IP \u6570\u91cf\u3002 kube-ovn-cni \u00b6 kube-ovn-cni \u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Histogram cni_op_latency_seconds CNI \u64cd\u4f5c\u5ef6\u8fdf\u3002 Counter cni_wait_address_seconds_total CNI \u7b49\u5f85\u5730\u5740\u5c31\u7eea\u65f6\u95f4\u3002 Counter cni_wait_connectivity_seconds_total CNI \u7b49\u5f85\u8fde\u63a5\u5c31\u7eea\u65f6\u95f4\u3002 Counter cni_wait_route_seconds_total CNI \u7b49\u5f85\u8def\u7531\u5c31\u7eea\u65f6\u95f4\u3002 Histogram rest_client_request_latency_seconds \u8bf7\u6c42 apiserver \u5ef6\u8fdf\u3002 Counter rest_client_requests_total \u8bf7\u6c42 apiserver \u6570\u91cf\u3002 Counter lists_total API list \u8bf7\u6c42\u6570\u91cf\u3002 Summary list_duration_seconds API list \u8bf7\u6c42\u5ef6\u8fdf\u3002 Summary items_per_list API list \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Counter watches_total API watch \u8bf7\u6c42\u6570\u91cf\u3002 Counter short_watches_total \u77ed\u65f6\u95f4 API watch \u8bf7\u6c42\u6570\u91cf\u3002 Summary watch_duration_seconds API watch \u6301\u7eed\u65f6\u95f4\u3002 Summary items_per_watch API watch \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Gauge last_resource_version \u6700\u65b0\u7684 resource version\u3002 Histogram ovs_client_request_latency_milliseconds \u8bf7\u6c42 OVN \u7ec4\u4ef6\u5ef6\u8fdf\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN \u76d1\u63a7\u6307\u6807"},{"location":"reference/metrics/#kube-ovn","text":"\u672c\u6587\u6863\u5217\u4e3e Kube-OVN \u6240\u63d0\u4f9b\u7684\u76d1\u63a7\u6307\u6807\u3002","title":"Kube-OVN \u76d1\u63a7\u6307\u6807"},{"location":"reference/metrics/#ovn-monitor","text":"OVN \u81ea\u8eab\u72b6\u6001\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge kube_ovn_ovn_status OVN \u89d2\u8272\u72b6\u6001\uff0c (2) \u4e3a follower\uff1b (1) \u4e3a leader, (0) \u4e3a\u5f02\u5e38\u72b6\u6001\u3002 Gauge kube_ovn_failed_req_count OVN \u5931\u8d25\u8bf7\u6c42\u6570\u91cf\u3002 Gauge kube_ovn_log_file_size_bytes OVN \u7ec4\u4ef6\u65e5\u5fd7\u6587\u4ef6\u5927\u5c0f\u3002 Gauge kube_ovn_db_file_size_bytes OVN \u7ec4\u4ef6\u6570\u636e\u5e93\u6587\u4ef6\u5927\u5c0f\u3002 Gauge kube_ovn_chassis_info OVN chassis \u72b6\u6001 (1) \u8fd0\u884c\u4e2d\uff0c(0) \u505c\u6b62\u3002 Gauge kube_ovn_db_status OVN \u6570\u636e\u5e93\u72b6\u6001, (1) \u4e3a\u6b63\u5e38\uff1b (0) \u4e3a\u5f02\u5e38\u3002 Gauge kube_ovn_logical_switch_info OVN logical switch \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b logical switch \u540d\u5b57\u3002 Gauge kube_ovn_logical_switch_external_id OVN logical switch external_id \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b external-id \u5185\u5bb9\u3002 Gauge kube_ovn_logical_switch_port_binding OVN logical switch \u548c logical switch port \u5173\u8054\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u901a\u8fc7\u6807\u7b7e\u8fdb\u884c\u5173\u8054\u3002 Gauge kube_ovn_logical_switch_tunnel_key \u548c OVN logical switch \u5173\u8054\u7684 tunnel key \u4fe1\u606f\u3002 Gauge kube_ovn_logical_switch_ports_num OVN logical switch \u4e0a logical port \u7684\u6570\u91cf\u3002 Gauge kube_ovn_logical_switch_port_info OVN logical switch port \u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5177\u4f53\u4fe1\u606f\u3002 Gauge kube_ovn_logical_switch_port_tunnel_key \u548c OVN logical switch port \u5173\u8054\u7684 tunnel key \u4fe1\u606f\u3002 Gauge kube_ovn_cluster_enabled (1) OVN \u6570\u636e\u5e93\u4e3a\u96c6\u7fa4\u6a21\u5f0f\uff1b (0) OVN \u6570\u636e\u5e93\u4e3a\u975e\u96c6\u7fa4\u6a21\u5f0f\u3002 Gauge kube_ovn_cluster_role \u6bcf\u4e2a\u6570\u636e\u5e93\u5b9e\u4f8b\u7684\u89d2\u8272\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u89d2\u8272\u4fe1\u606f\u3002 Gauge kube_ovn_cluster_status \u6bcf\u4e2a\u6570\u636e\u5e93\u5b9e\u4f8b\u7684\u72b6\u6001\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u72b6\u6001\u4fe1\u606f\u3002 Gauge kube_ovn_cluster_term RAFT term \u4fe1\u606f\u3002 Gauge kube_ovn_cluster_leader_self \u5f53\u524d\u6570\u636e\u5e93\u5b9e\u4f8b\u662f\u5426\u4e3a leader (1) \u662f\uff0c (0) \u4e0d\u662f\u3002 Gauge kube_ovn_cluster_vote_self \u5f53\u524d\u6570\u636e\u5e93\u5b9e\u4f8b\u662f\u5426\u9009\u4e3e\u81ea\u5df1\u4e3a leader (1) \u662f\uff0c (0) \u4e0d\u662f\u3002 Gauge kube_ovn_cluster_election_timer \u5f53\u524d election timer \u503c\u3002 Gauge kube_ovn_cluster_log_not_committed \u672a commit \u7684 RAFT \u65e5\u5fd7\u6570\u91cf\u3002 Gauge kube_ovn_cluster_log_not_applied \u672a apply \u7684 RAFT \u65e5\u5fd7\u6570\u91cf\u3002 Gauge kube_ovn_cluster_log_index_start \u5f53\u524d RAFT \u65e5\u5fd7\u6761\u76ee\u7684\u8d77\u59cb\u503c\u3002 Gauge kube_ovn_cluster_log_index_next RAFT \u65e5\u5fd7\u6761\u76ee\u7684\u4e0b\u4e00\u4e2a\u503c\u3002 Gauge kube_ovn_cluster_inbound_connections_total \u5f53\u524d\u5b9e\u4f8b\u7684\u5165\u5411\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_outbound_connections_total \u5f53\u524d\u5b9e\u4f8b\u7684\u51fa\u5411\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_inbound_connections_error_total \u5f53\u524d\u5b9e\u4f8b\u7684\u5165\u5411\u9519\u8bef\u8fde\u63a5\u6570\u91cf\u3002 Gauge kube_ovn_cluster_outbound_connections_error_total \u5f53\u524d\u5b9e\u4f8b\u7684\u51fa\u5411\u9519\u8bef\u8fde\u63a5\u6570\u91cf\u3002","title":"ovn-monitor"},{"location":"reference/metrics/#ovs-monitor","text":"ovsdb \u548c vswitchd \u81ea\u8eab\u72b6\u6001\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge ovs_status OVS \u5065\u5eb7\u72b6\u6001\uff0c (1) \u4e3a\u6b63\u5e38\uff0c(0) \u4e3a\u5f02\u5e38\u3002 Gauge ovs_info OVS \u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge failed_req_count OVS \u5931\u8d25\u8bf7\u6c42\u6570\u91cf\u3002 Gauge log_file_size OVS \u7ec4\u4ef6\u65e5\u5fd7\u6587\u4ef6\u5927\u5c0f\u3002 Gauge db_file_size OVS \u7ec4\u4ef6\u6570\u636e\u5e93\u6587\u4ef6\u5927\u5c0f\u3002 Gauge datapath Datapath \u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge dp_total \u5f53\u524d OVS \u4e2d datapath \u6570\u91cf\u3002 Gauge dp_if Datapath \u63a5\u53e3\u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge dp_if_total \u5f53\u524d datapath \u4e2d port \u6570\u91cf\u3002 Gauge dp_flows_total Datapath \u4e2d flow \u6570\u91cf\u3002 Gauge dp_flows_lookup_hit Datapath \u4e2d\u547d\u4e2d\u5f53\u524d flow \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_flows_lookup_missed Datapath \u4e2d\u672a\u547d\u4e2d\u5f53\u524d flow \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_flows_lookup_lost Datapath \u4e2d\u9700\u8981\u53d1\u9001\u7ed9 userspace \u5904\u7406\u7684\u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_masks_hit Datapath \u4e2d\u547d\u4e2d\u5f53\u524d mask \u6570\u636e\u5305\u6570\u91cf\u3002 Gauge dp_masks_total Datapath \u4e2d mask \u7684\u6570\u91cf\u3002 Gauge dp_masks_hit_ratio Datapath \u4e2d \u6570\u636e\u5305\u547d\u4e2d mask \u7684\u6bd4\u7387\u3002 Gauge interface OVS \u63a5\u53e3\u57fa\u7840\u4fe1\u606f\uff0c\u503c\u4e3a (1)\uff0c\u6807\u7b7e\u4e2d\u5305\u542b\u5bf9\u5e94\u4fe1\u606f\u3002 Gauge interface_admin_state \u63a5\u53e3\u7ba1\u7406\u72b6\u6001\u4fe1\u606f (0) \u4e3a down, (1) \u4e3a up, (2) \u4e3a\u5176\u4ed6\u72b6\u6001\u3002 Gauge interface_link_state \u63a5\u53e3\u94fe\u8def\u72b6\u6001\u4fe1\u606f (0) \u4e3a down, (1) \u4e3a up, (2) \u4e3a\u5176\u4ed6\u72b6\u6001\u3002 Gauge interface_mac_in_use OVS Interface \u4f7f\u7528\u7684 MAC \u5730\u5740 Gauge interface_mtu OVS Interface \u4f7f\u7528\u7684 MTU\u3002 Gauge interface_of_port OVS Interface \u5173\u8054\u7684 OpenFlow Port ID\u3002 Gauge interface_if_index OVS Interface \u5173\u8054\u7684 Index\u3002 Gauge interface_tx_packets OVS Interface \u53d1\u9001\u5305\u6570\u91cf\u3002 Gauge interface_tx_bytes OVS Interface \u53d1\u9001\u5305\u5927\u5c0f\u3002 Gauge interface_rx_packets OVS Interface \u63a5\u6536\u5305\u6570\u91cf\u3002 Gauge interface_rx_bytes OVS Interface \u63a5\u6536\u5305\u5927\u5c0f\u3002 Gauge interface_rx_crc_err OVS Interface \u63a5\u6536\u5305\u6821\u9a8c\u548c\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_dropped OVS Interface \u63a5\u6536\u5305\u4e22\u5f03\u6570\u91cf\u3002 Gauge interface_rx_errors OVS Interface \u63a5\u6536\u5305\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_frame_err OVS Interface \u63a5\u6536\u5e27\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_rx_missed_err OVS Interface \u63a5\u6536\u5305 miss \u6570\u91cf\u3002 Gauge interface_rx_over_err OVS Interface \u63a5\u6536\u5305 overrun \u6570\u91cf\u3002 Gauge interface_tx_dropped OVS Interface \u53d1\u9001\u5305\u4e22\u5f03\u6570\u91cf\u3002 Gauge interface_tx_errors OVS Interface \u53d1\u9001\u5305\u9519\u8bef\u6570\u91cf\u3002 Gauge interface_collisions OVS interface \u51b2\u7a81\u6570\u91cf\u3002","title":"ovs-monitor"},{"location":"reference/metrics/#kube-ovn-pinger","text":"\u7f51\u7edc\u8d28\u91cf\u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Gauge pinger_ovs_up \u8282\u70b9 OVS \u8fd0\u884c\u3002 Gauge pinger_ovs_down \u8282\u70b9 OVS \u505c\u6b62\u3002 Gauge pinger_ovn_controller_up \u8282\u70b9 ovn-controller \u8fd0\u884c\u3002 Gauge pinger_ovn_controller_down \u8282\u70b9 ovn-controller \u505c\u6b62\u3002 Gauge pinger_inconsistent_port_binding OVN-SB \u91cc portbinding \u6570\u91cf\u548c\u4e3b\u673a OVS interface \u4e0d\u4e00\u81f4\u7684\u6570\u91cf\u3002 Gauge pinger_apiserver_healthy kube-ovn-pinger \u53ef\u4ee5\u8054\u901a apiserver\u3002 Gauge pinger_apiserver_unhealthy kube-ovn-pinger \u65e0\u6cd5\u8054\u901a apiserver\u3002 Histogram pinger_apiserver_latency_ms kube-ovn-pinger \u8bbf\u95ee apiserver \u5ef6\u8fdf\u3002 Gauge pinger_internal_dns_healthy kube-ovn-pinger \u53ef\u4ee5\u89e3\u6790\u5185\u90e8\u57df\u540d\u3002 Gauge pinger_internal_dns_unhealthy kube-ovn-pinger \u65e0\u6cd5\u89e3\u6790\u5185\u90e8\u57df\u540d\u3002 Histogram pinger_internal_dns_latency_ms kube-ovn-pinger \u89e3\u6790\u5185\u90e8\u57df\u540d\u5ef6\u8fdf\u3002 Gauge pinger_external_dns_health kube-ovn-pinger \u53ef\u4ee5\u89e3\u6790\u5916\u90e8\u57df\u540d\u3002 Gauge pinger_external_dns_unhealthy kube-ovn-pinger \u65e0\u6cd5\u89e3\u6790\u5916\u90e8\u57df\u540d\u3002 Histogram pinger_external_dns_latency_ms kube-ovn-pinger \u89e3\u6790\u5916\u90e8\u57df\u540d\u5ef6\u8fdf\u3002 Histogram pinger_pod_ping_latency_ms kube-ovn-pinger ping Pod \u5ef6\u8fdf\u3002 Gauge pinger_pod_ping_lost_total kube-ovn-pinger ping Pod \u4e22\u5305\u6570\u91cf\u3002 Gauge pinger_pod_ping_count_total kube-ovn-pinger ping Pod \u6570\u91cf\u3002 Histogram pinger_node_ping_latency_ms kube-ovn-pinger ping Node \u5ef6\u8fdf\u3002 Gauge pinger_node_ping_lost_total kube-ovn-pinger ping Node \u4e22\u5305\u3002 Gauge pinger_node_ping_count_total kube-ovn-pinger ping Node \u6570\u91cf\u3002 Histogram pinger_external_ping_latency_ms kube-ovn-pinger ping \u5916\u90e8\u5730\u5740 \u5ef6\u8fdf\u3002 Gauge pinger_external_lost_total kube-ovn-pinger ping \u5916\u90e8\u4e22\u5305\u6570\u91cf\u3002","title":"kube-ovn-pinger"},{"location":"reference/metrics/#kube-ovn-controller","text":"kube-ovn-controller \u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Histogram rest_client_request_latency_seconds \u8bf7\u6c42 apiserver \u5ef6\u8fdf\u3002 Counter rest_client_requests_total \u8bf7\u6c42 apiserver \u6570\u91cf\u3002 Counter lists_total API list \u8bf7\u6c42\u6570\u91cf\u3002 Summary list_duration_seconds API list \u8bf7\u6c42\u5ef6\u8fdf\u3002 Summary items_per_list API list \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Counter watches_total API watch \u8bf7\u6c42\u6570\u91cf\u3002 Counter short_watches_total \u77ed\u65f6\u95f4 API watch \u8bf7\u6c42\u6570\u91cf\u3002 Summary watch_duration_seconds API watch \u6301\u7eed\u65f6\u95f4\u3002 Summary items_per_watch API watch \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Gauge last_resource_version \u6700\u65b0\u7684 resource version\u3002 Histogram ovs_client_request_latency_milliseconds \u8bf7\u6c42 OVN \u7ec4\u4ef6\u5ef6\u8fdf\u3002 Gauge subnet_available_ip_count \u5b50\u7f51\u53ef\u7528 IP \u6570\u91cf\u3002 Gauge subnet_used_ip_count \u5b50\u7f51\u5df2\u7528 IP \u6570\u91cf\u3002","title":"kube-ovn-controller"},{"location":"reference/metrics/#kube-ovn-cni","text":"kube-ovn-cni \u76f8\u5173\u76d1\u63a7\u6307\u6807\uff1a \u7c7b\u578b \u6307\u6807\u9879 \u63cf\u8ff0 Histogram cni_op_latency_seconds CNI \u64cd\u4f5c\u5ef6\u8fdf\u3002 Counter cni_wait_address_seconds_total CNI \u7b49\u5f85\u5730\u5740\u5c31\u7eea\u65f6\u95f4\u3002 Counter cni_wait_connectivity_seconds_total CNI \u7b49\u5f85\u8fde\u63a5\u5c31\u7eea\u65f6\u95f4\u3002 Counter cni_wait_route_seconds_total CNI \u7b49\u5f85\u8def\u7531\u5c31\u7eea\u65f6\u95f4\u3002 Histogram rest_client_request_latency_seconds \u8bf7\u6c42 apiserver \u5ef6\u8fdf\u3002 Counter rest_client_requests_total \u8bf7\u6c42 apiserver \u6570\u91cf\u3002 Counter lists_total API list \u8bf7\u6c42\u6570\u91cf\u3002 Summary list_duration_seconds API list \u8bf7\u6c42\u5ef6\u8fdf\u3002 Summary items_per_list API list \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Counter watches_total API watch \u8bf7\u6c42\u6570\u91cf\u3002 Counter short_watches_total \u77ed\u65f6\u95f4 API watch \u8bf7\u6c42\u6570\u91cf\u3002 Summary watch_duration_seconds API watch \u6301\u7eed\u65f6\u95f4\u3002 Summary items_per_watch API watch \u8fd4\u56de\u7ed3\u679c\u6570\u91cf\u3002 Gauge last_resource_version \u6700\u65b0\u7684 resource version\u3002 Histogram ovs_client_request_latency_milliseconds \u8bf7\u6c42 OVN \u7ec4\u4ef6\u5ef6\u8fdf\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kube-ovn-cni"},{"location":"reference/ovs-ovn-customized/","text":"\u5bf9\u4e0a\u6e38 OVS/OVN \u4fee\u6539 \u00b6 \u4e0a\u6e38 OVN/OVS \u6700\u521d\u8bbe\u8ba1\u76ee\u6807\u4e3a\u901a\u7528 SDN \u63a7\u5236\u5668\u548c\u6570\u636e\u5e73\u9762\u3002\u7531\u4e8e Kubernetes \u7f51\u7edc\u5b58\u5728\u4e00\u4e9b\u7279\u6b8a\u7684\u7528\u6cd5\uff0c \u5e76\u4e14 Kube-OVN \u53ea\u91cd\u70b9\u4f7f\u7528\u4e86\u90e8\u5206\u529f\u80fd\uff0c\u4e3a\u4e86 \u8fbe\u5230\u66f4\u597d\u7684\u6027\u80fd\u3001\u7a33\u5b9a\u6027\u548c\u7279\u5b9a\u7684\u529f\u80fd\uff0cKube-OVN \u5bf9\u4e0a\u6e38 OVN/OVS \u505a\u4e86\u90e8\u5206\u4fee\u6539\u3002\u7528\u6237\u5982\u679c\u4f7f\u7528\u81ea\u5df1\u7684 OVN/OVS \u914d\u5408 Kube-OVN \u7684\u63a7\u5236\u5668\u8fdb\u884c\u5de5\u4f5c\u65f6\u9700\u8981\u6ce8\u610f \u4e0b\u8ff0\u7684\u6539\u52a8\u53ef\u80fd\u9020\u6210\u7684\u5f71\u54cd\u3002 \u672a\u5408\u5165\u4e0a\u6e38\u4fee\u6539\uff1a 38df6fa3f7 \u8c03\u6574\u9009\u4e3e timer\uff0c\u907f\u514d\u5927\u89c4\u6a21\u96c6\u7fa4\u9009\u4e3e\u6296\u52a8\u3002 d4888c4e75 \u6dfb\u52a0 fdb \u66f4\u65b0\u65e5\u5fd7\u3002 d4888c4e75 \u4fee\u590d hairpin \u73af\u5883\u4e0b fdb \u5b66\u4e60\u9519\u8bef\u7684\u95ee\u9898\u3002 9a81b91368 \u4e3a ovsdb-tool \u7684 join-cluster \u5b50\u547d\u4ee4\u6dfb\u52a0 Server ID \u53c2\u6570\u3002 62d4969877 \u4fee\u590d\u5f00\u542f SSL \u540e OVSDB \u76d1\u542c\u5730\u5740\u9519\u8bef\u7684\u95ee\u9898\u3002 0700cb90f9 \u76ee\u7684\u5730\u5740\u975e Service \u6d41\u91cf\u7ed5\u8fc7 conntrack \u4ee5\u63d0\u9ad8\u7279\u5b9a\u6570\u636e\u94fe\u8def\u6027\u80fd\u3002 c48049a64f ECMP \u7b97\u6cd5\u7531 dp_hash \u8c03\u6574\u4e3a hash\uff0c\u907f\u514d\u90e8\u5206\u5185\u6838\u51fa\u73b0\u7684\u54c8\u5e0c\u9519\u8bef\u95ee\u9898\u3002 64383c14a9 \u4fee\u590d Windows \u4e0b\u5185\u6838 Crash \u95ee\u9898\u3002 08a95db2ca \u652f\u6301 Windows \u4e0b\u7684 github action \u6784\u5efa\u3002 680e77a190 Windows \u4e0b\u9ed8\u8ba4\u4f7f\u7528 tcp \u76d1\u542c\u3002 05e57b3227 \u652f\u6301 Windows \u7f16\u8bd1\u3002 b3801ecb73 \u4fee\u6539\u6e90\u8def\u7531\u7684\u4f18\u5148\u7ea7\u3002 977e569539 \u4fee\u590d Underlay \u6a21\u5f0f\u4e0b Pod \u6570\u91cf\u8fc7\u591a\u5bfc\u81f4 OVS \u6d41\u8868 resubmit \u6b21\u6570\u8d85\u8fc7\u4e0a\u9650\u7684\u95ee\u9898\u3002 45a4a22161 ovn-nbctl\uff1avips \u4e3a\u7a7a\u65f6\u4e0d\u5220\u9664 Load Balancer\u3002 540592b9ff DNAT \u540e\u66ff\u6362 Mac \u5730\u5740\u4e3a\u76ee\u6807\u5730\u5740\uff0c\u51cf\u5c11\u989d\u5916\u6027\u80fd\u5f00\u9500\u3002 10972d9632 \u4fee\u590d vswitchd ofport_usage \u5185\u5b58\u6cc4\u9732\u3002 \u5df2\u5408\u5165\u4e0a\u6e38\u4fee\u6539\uff1a 20626ea909 \u7ec4\u64ad\u6d41\u91cf\u7ed5\u8fc7 LB \u548c ACL \u5904\u7406\u9636\u6bb5\uff0c\u4ee5\u63d0\u9ad8\u7279\u5b9a\u6570\u636e\u94fe\u8def\u6027\u80fd\u3002 a2d9ff3ccd Deb \u6784\u5efa\u589e\u52a0\u7f16\u8bd1\u4f18\u5316\u9009\u9879\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5bf9\u4e0a\u6e38 OVS/OVN \u4fee\u6539"},{"location":"reference/ovs-ovn-customized/#ovsovn","text":"\u4e0a\u6e38 OVN/OVS \u6700\u521d\u8bbe\u8ba1\u76ee\u6807\u4e3a\u901a\u7528 SDN \u63a7\u5236\u5668\u548c\u6570\u636e\u5e73\u9762\u3002\u7531\u4e8e Kubernetes \u7f51\u7edc\u5b58\u5728\u4e00\u4e9b\u7279\u6b8a\u7684\u7528\u6cd5\uff0c \u5e76\u4e14 Kube-OVN \u53ea\u91cd\u70b9\u4f7f\u7528\u4e86\u90e8\u5206\u529f\u80fd\uff0c\u4e3a\u4e86 \u8fbe\u5230\u66f4\u597d\u7684\u6027\u80fd\u3001\u7a33\u5b9a\u6027\u548c\u7279\u5b9a\u7684\u529f\u80fd\uff0cKube-OVN \u5bf9\u4e0a\u6e38 OVN/OVS \u505a\u4e86\u90e8\u5206\u4fee\u6539\u3002\u7528\u6237\u5982\u679c\u4f7f\u7528\u81ea\u5df1\u7684 OVN/OVS \u914d\u5408 Kube-OVN \u7684\u63a7\u5236\u5668\u8fdb\u884c\u5de5\u4f5c\u65f6\u9700\u8981\u6ce8\u610f \u4e0b\u8ff0\u7684\u6539\u52a8\u53ef\u80fd\u9020\u6210\u7684\u5f71\u54cd\u3002 \u672a\u5408\u5165\u4e0a\u6e38\u4fee\u6539\uff1a 38df6fa3f7 \u8c03\u6574\u9009\u4e3e timer\uff0c\u907f\u514d\u5927\u89c4\u6a21\u96c6\u7fa4\u9009\u4e3e\u6296\u52a8\u3002 d4888c4e75 \u6dfb\u52a0 fdb \u66f4\u65b0\u65e5\u5fd7\u3002 d4888c4e75 \u4fee\u590d hairpin \u73af\u5883\u4e0b fdb \u5b66\u4e60\u9519\u8bef\u7684\u95ee\u9898\u3002 9a81b91368 \u4e3a ovsdb-tool \u7684 join-cluster \u5b50\u547d\u4ee4\u6dfb\u52a0 Server ID \u53c2\u6570\u3002 62d4969877 \u4fee\u590d\u5f00\u542f SSL \u540e OVSDB \u76d1\u542c\u5730\u5740\u9519\u8bef\u7684\u95ee\u9898\u3002 0700cb90f9 \u76ee\u7684\u5730\u5740\u975e Service \u6d41\u91cf\u7ed5\u8fc7 conntrack \u4ee5\u63d0\u9ad8\u7279\u5b9a\u6570\u636e\u94fe\u8def\u6027\u80fd\u3002 c48049a64f ECMP \u7b97\u6cd5\u7531 dp_hash \u8c03\u6574\u4e3a hash\uff0c\u907f\u514d\u90e8\u5206\u5185\u6838\u51fa\u73b0\u7684\u54c8\u5e0c\u9519\u8bef\u95ee\u9898\u3002 64383c14a9 \u4fee\u590d Windows \u4e0b\u5185\u6838 Crash \u95ee\u9898\u3002 08a95db2ca \u652f\u6301 Windows \u4e0b\u7684 github action \u6784\u5efa\u3002 680e77a190 Windows \u4e0b\u9ed8\u8ba4\u4f7f\u7528 tcp \u76d1\u542c\u3002 05e57b3227 \u652f\u6301 Windows \u7f16\u8bd1\u3002 b3801ecb73 \u4fee\u6539\u6e90\u8def\u7531\u7684\u4f18\u5148\u7ea7\u3002 977e569539 \u4fee\u590d Underlay \u6a21\u5f0f\u4e0b Pod \u6570\u91cf\u8fc7\u591a\u5bfc\u81f4 OVS \u6d41\u8868 resubmit \u6b21\u6570\u8d85\u8fc7\u4e0a\u9650\u7684\u95ee\u9898\u3002 45a4a22161 ovn-nbctl\uff1avips \u4e3a\u7a7a\u65f6\u4e0d\u5220\u9664 Load Balancer\u3002 540592b9ff DNAT \u540e\u66ff\u6362 Mac \u5730\u5740\u4e3a\u76ee\u6807\u5730\u5740\uff0c\u51cf\u5c11\u989d\u5916\u6027\u80fd\u5f00\u9500\u3002 10972d9632 \u4fee\u590d vswitchd ofport_usage \u5185\u5b58\u6cc4\u9732\u3002 \u5df2\u5408\u5165\u4e0a\u6e38\u4fee\u6539\uff1a 20626ea909 \u7ec4\u64ad\u6d41\u91cf\u7ed5\u8fc7 LB \u548c ACL \u5904\u7406\u9636\u6bb5\uff0c\u4ee5\u63d0\u9ad8\u7279\u5b9a\u6570\u636e\u94fe\u8def\u6027\u80fd\u3002 a2d9ff3ccd Deb \u6784\u5efa\u589e\u52a0\u7f16\u8bd1\u4f18\u5316\u9009\u9879\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5bf9\u4e0a\u6e38 OVS/OVN \u4fee\u6539"},{"location":"reference/tunnel-protocol/","text":"\u96a7\u9053\u534f\u8bae\u8bf4\u660e \u00b6 Kube-OVN \u4f7f\u7528 OVN/OVS \u4f5c\u4e3a\u6570\u636e\u5e73\u9762\u5b9e\u73b0\uff0c\u76ee\u524d\u652f\u6301 Geneve \uff0c Vxlan \u548c STT \u4e09\u79cd\u96a7\u9053\u5c01\u88c5\u534f\u8bae\u3002 \u8fd9\u4e09\u79cd\u534f\u8bae\u5728\u529f\u80fd\uff0c\u6027\u80fd\u548c\u6613\u7528\u6027\u4e0a\u5b58\u5728\u7740\u533a\u522b\uff0c\u672c\u6587\u6863\u5c06\u4ecb\u7ecd\u4e09\u79cd\u534f\u8bae\u5728\u4f7f\u7528\u4e2d\u7684\u5dee\u5f02\uff0c\u7528\u6237\u53ef\u6839\u636e\u81ea\u5df1\u7684\u60c5\u51b5\u8fdb\u884c\u9009\u62e9\u3002 Geneve \u00b6 Geneve \u534f\u8bae\u4e3a Kube-OVN \u90e8\u7f72\u65f6\u9009\u62e9\u7684\u9ed8\u8ba4\u96a7\u9053\u534f\u8bae\uff0c\u4e5f\u662f OVN \u9ed8\u8ba4\u63a8\u8350\u7684\u96a7\u9053\u534f\u8bae\u3002\u8be5\u534f\u8bae\u5728\u5185\u6838\u4e2d\u5f97\u5230\u4e86\u5e7f\u6cdb\u7684\u652f\u6301\uff0c \u5e76\u53ef\u4ee5\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684\u901a\u7528 Offload \u80fd\u529b\u8fdb\u884c\u52a0\u901f\u3002\u7531\u4e8e Geneve \u6709\u7740\u53ef\u53d8\u957f\u7684\u5934\u90e8\uff0c\u53ef\u4ee5\u4f7f\u7528 24bit \u7a7a\u95f4\u6765\u6807\u5fd7\u4e0d\u540c\u7684 datapath \u7528\u6237\u53ef\u4ee5\u521b\u5efa\u66f4\u591a\u6570\u91cf\u7684\u865a\u62df\u7f51\u7edc\u3002 \u5982\u679c\u4f7f\u7528 Mellanox \u6216\u82af\u542f\u6e90\u7684\u667a\u80fd\u7f51\u5361 OVS \u5378\u8f7d\uff0c Geneve \u9700\u8981\u8f83\u9ad8\u7248\u672c\u7684\u5185\u6838\u652f\u6301\uff0c\u9700\u8981\u9009\u62e9 5.4 \u4ee5\u4e0a\u7684\u4e0a\u6e38\u5185\u6838\uff0c \u6216 backport \u4e86\u8be5\u529f\u80fd\u7684\u5176\u4ed6\u517c\u5bb9\u5185\u6838\u3002 \u7531\u4e8e\u4f7f\u7528 UDP \u8fdb\u884c\u5c01\u88c5\uff0c\u8be5\u534f\u8bae\u5728\u5904\u7406 TCP over UDP \u65f6\u4e0d\u80fd\u5f88\u597d\u7684\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684 TCP \u76f8\u5173\u5378\u8f7d\uff0c\u5728\u5904\u7406\u5927\u5305\u65f6\u4f1a\u6d88\u8017\u8f83\u591a CPU \u8d44\u6e90\u3002 Vxlan \u00b6 Vxlan \u4e3a\u4e0a\u6e38 OVN \u8fd1\u671f\u652f\u6301\u7684\u534f\u8bae\uff0c\u8be5\u534f\u8bae\u5728\u5185\u6838\u4e2d\u5f97\u5230\u4e86\u5e7f\u6cdb\u7684\u652f\u6301\uff0c \u5e76\u53ef\u4ee5\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684\u901a\u7528 Offload \u80fd\u529b\u8fdb\u884c\u52a0\u901f\u3002 \u7531\u4e8e\u8be5\u534f\u8bae\u5934\u90e8\u957f\u5ea6\u6709\u9650\uff0c\u5e76\u4e14 OVN \u9700\u8981\u4f7f\u7528\u989d\u5916\u7684\u7a7a\u95f4\u8fdb\u884c\u7f16\u6392\uff0cdatapath \u7684\u6570\u91cf\u5b58\u5728\u9650\u5236\uff0c\u6700\u591a\u53ea\u80fd\u521b\u5efa 4096 \u4e2a datapath\uff0c \u6bcf\u4e2a datapath \u4e0b\u6700\u591a 4096 \u4e2a\u7aef\u53e3\u3002\u540c\u65f6\u7531\u4e8e\u7a7a\u95f4\u6709\u9650\uff0c\u57fa\u4e8e inport \u7684 ACL \u6ca1\u6709\u8fdb\u884c\u652f\u6301\u3002 \u5982\u679c\u4f7f\u7528 Mellanox \u6216\u82af\u542f\u6e90\u7684\u667a\u80fd\u7f51\u5361 OVS \u5378\u8f7d\uff0c Vxlan \u7684\u5378\u8f7d\u5728\u5e38\u89c1\u5185\u6838\u4e2d\u5df2\u83b7\u5f97\u652f\u6301\u3002 \u7531\u4e8e\u4f7f\u7528 UDP \u8fdb\u884c\u5c01\u88c5\uff0c\u8be5\u534f\u8bae\u5728\u5904\u7406 TCP over UDP \u65f6\u4e0d\u80fd\u5f88\u597d\u7684\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684 TCP \u76f8\u5173\u5378\u8f7d\uff0c\u5728\u5904\u7406\u5927\u5305\u65f6\u4f1a\u6d88\u8017\u8f83\u591a CPU \u8d44\u6e90\u3002 STT \u00b6 STT \u534f\u8bae\u4e3a OVN \u8f83\u65e9\u652f\u6301\u7684\u96a7\u9053\u534f\u8bae\uff0c\u8be5\u534f\u8bae\u4f7f\u7528\u7c7b TCP \u7684\u5934\u90e8\uff0c\u53ef\u4ee5\u5145\u5206\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u901a\u7528\u7684 TCP \u5378\u8f7d\u80fd\u529b\uff0c\u5927\u5e45\u63d0\u5347 TCP \u7684\u541e\u5410\u91cf\u3002\u540c\u65f6\u8be5\u534f\u8bae\u5934\u90e8\u8f83\u957f\u53ef\u652f\u6301\u5b8c\u6574\u7684 OVN \u80fd\u529b\u548c\u5927\u89c4\u6a21\u7684 datapath\u3002 \u8be5\u534f\u8bae\u672a\u5728\u5185\u6838\u4e2d\u652f\u6301\uff0c\u82e5\u8981\u4f7f\u7528\u9700\u8981\u989d\u5916\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\uff0c\u5e76\u5728\u5347\u7ea7\u5185\u6838\u65f6\u5bf9\u5e94\u518d\u6b21\u7f16\u8bd1\u65b0\u7248\u672c\u5185\u6838\u6a21\u5757\u3002 \u8be5\u534f\u8bae\u76ee\u524d\u672a\u88ab\u667a\u80fd\u7f51\u5361\u652f\u6301\uff0c\u65e0\u6cd5\u4f7f\u7528 OVS \u7684\u5378\u8f7d\u80fd\u529b\u3002 \u53c2\u8003\u8d44\u6599 \u00b6 https://ipwithease.com/vxlan-vs-geneve-understand-the-difference/ OVN FAQ What is Geneve \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u96a7\u9053\u534f\u8bae\u8bf4\u660e"},{"location":"reference/tunnel-protocol/#_1","text":"Kube-OVN \u4f7f\u7528 OVN/OVS \u4f5c\u4e3a\u6570\u636e\u5e73\u9762\u5b9e\u73b0\uff0c\u76ee\u524d\u652f\u6301 Geneve \uff0c Vxlan \u548c STT \u4e09\u79cd\u96a7\u9053\u5c01\u88c5\u534f\u8bae\u3002 \u8fd9\u4e09\u79cd\u534f\u8bae\u5728\u529f\u80fd\uff0c\u6027\u80fd\u548c\u6613\u7528\u6027\u4e0a\u5b58\u5728\u7740\u533a\u522b\uff0c\u672c\u6587\u6863\u5c06\u4ecb\u7ecd\u4e09\u79cd\u534f\u8bae\u5728\u4f7f\u7528\u4e2d\u7684\u5dee\u5f02\uff0c\u7528\u6237\u53ef\u6839\u636e\u81ea\u5df1\u7684\u60c5\u51b5\u8fdb\u884c\u9009\u62e9\u3002","title":"\u96a7\u9053\u534f\u8bae\u8bf4\u660e"},{"location":"reference/tunnel-protocol/#geneve","text":"Geneve \u534f\u8bae\u4e3a Kube-OVN \u90e8\u7f72\u65f6\u9009\u62e9\u7684\u9ed8\u8ba4\u96a7\u9053\u534f\u8bae\uff0c\u4e5f\u662f OVN \u9ed8\u8ba4\u63a8\u8350\u7684\u96a7\u9053\u534f\u8bae\u3002\u8be5\u534f\u8bae\u5728\u5185\u6838\u4e2d\u5f97\u5230\u4e86\u5e7f\u6cdb\u7684\u652f\u6301\uff0c \u5e76\u53ef\u4ee5\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684\u901a\u7528 Offload \u80fd\u529b\u8fdb\u884c\u52a0\u901f\u3002\u7531\u4e8e Geneve \u6709\u7740\u53ef\u53d8\u957f\u7684\u5934\u90e8\uff0c\u53ef\u4ee5\u4f7f\u7528 24bit \u7a7a\u95f4\u6765\u6807\u5fd7\u4e0d\u540c\u7684 datapath \u7528\u6237\u53ef\u4ee5\u521b\u5efa\u66f4\u591a\u6570\u91cf\u7684\u865a\u62df\u7f51\u7edc\u3002 \u5982\u679c\u4f7f\u7528 Mellanox \u6216\u82af\u542f\u6e90\u7684\u667a\u80fd\u7f51\u5361 OVS \u5378\u8f7d\uff0c Geneve \u9700\u8981\u8f83\u9ad8\u7248\u672c\u7684\u5185\u6838\u652f\u6301\uff0c\u9700\u8981\u9009\u62e9 5.4 \u4ee5\u4e0a\u7684\u4e0a\u6e38\u5185\u6838\uff0c \u6216 backport \u4e86\u8be5\u529f\u80fd\u7684\u5176\u4ed6\u517c\u5bb9\u5185\u6838\u3002 \u7531\u4e8e\u4f7f\u7528 UDP \u8fdb\u884c\u5c01\u88c5\uff0c\u8be5\u534f\u8bae\u5728\u5904\u7406 TCP over UDP \u65f6\u4e0d\u80fd\u5f88\u597d\u7684\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684 TCP \u76f8\u5173\u5378\u8f7d\uff0c\u5728\u5904\u7406\u5927\u5305\u65f6\u4f1a\u6d88\u8017\u8f83\u591a CPU \u8d44\u6e90\u3002","title":"Geneve"},{"location":"reference/tunnel-protocol/#vxlan","text":"Vxlan \u4e3a\u4e0a\u6e38 OVN \u8fd1\u671f\u652f\u6301\u7684\u534f\u8bae\uff0c\u8be5\u534f\u8bae\u5728\u5185\u6838\u4e2d\u5f97\u5230\u4e86\u5e7f\u6cdb\u7684\u652f\u6301\uff0c \u5e76\u53ef\u4ee5\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684\u901a\u7528 Offload \u80fd\u529b\u8fdb\u884c\u52a0\u901f\u3002 \u7531\u4e8e\u8be5\u534f\u8bae\u5934\u90e8\u957f\u5ea6\u6709\u9650\uff0c\u5e76\u4e14 OVN \u9700\u8981\u4f7f\u7528\u989d\u5916\u7684\u7a7a\u95f4\u8fdb\u884c\u7f16\u6392\uff0cdatapath \u7684\u6570\u91cf\u5b58\u5728\u9650\u5236\uff0c\u6700\u591a\u53ea\u80fd\u521b\u5efa 4096 \u4e2a datapath\uff0c \u6bcf\u4e2a datapath \u4e0b\u6700\u591a 4096 \u4e2a\u7aef\u53e3\u3002\u540c\u65f6\u7531\u4e8e\u7a7a\u95f4\u6709\u9650\uff0c\u57fa\u4e8e inport \u7684 ACL \u6ca1\u6709\u8fdb\u884c\u652f\u6301\u3002 \u5982\u679c\u4f7f\u7528 Mellanox \u6216\u82af\u542f\u6e90\u7684\u667a\u80fd\u7f51\u5361 OVS \u5378\u8f7d\uff0c Vxlan \u7684\u5378\u8f7d\u5728\u5e38\u89c1\u5185\u6838\u4e2d\u5df2\u83b7\u5f97\u652f\u6301\u3002 \u7531\u4e8e\u4f7f\u7528 UDP \u8fdb\u884c\u5c01\u88c5\uff0c\u8be5\u534f\u8bae\u5728\u5904\u7406 TCP over UDP \u65f6\u4e0d\u80fd\u5f88\u597d\u7684\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u7684 TCP \u76f8\u5173\u5378\u8f7d\uff0c\u5728\u5904\u7406\u5927\u5305\u65f6\u4f1a\u6d88\u8017\u8f83\u591a CPU \u8d44\u6e90\u3002","title":"Vxlan"},{"location":"reference/tunnel-protocol/#stt","text":"STT \u534f\u8bae\u4e3a OVN \u8f83\u65e9\u652f\u6301\u7684\u96a7\u9053\u534f\u8bae\uff0c\u8be5\u534f\u8bae\u4f7f\u7528\u7c7b TCP \u7684\u5934\u90e8\uff0c\u53ef\u4ee5\u5145\u5206\u5229\u7528\u73b0\u4ee3\u7f51\u5361\u901a\u7528\u7684 TCP \u5378\u8f7d\u80fd\u529b\uff0c\u5927\u5e45\u63d0\u5347 TCP \u7684\u541e\u5410\u91cf\u3002\u540c\u65f6\u8be5\u534f\u8bae\u5934\u90e8\u8f83\u957f\u53ef\u652f\u6301\u5b8c\u6574\u7684 OVN \u80fd\u529b\u548c\u5927\u89c4\u6a21\u7684 datapath\u3002 \u8be5\u534f\u8bae\u672a\u5728\u5185\u6838\u4e2d\u652f\u6301\uff0c\u82e5\u8981\u4f7f\u7528\u9700\u8981\u989d\u5916\u7f16\u8bd1 OVS \u5185\u6838\u6a21\u5757\uff0c\u5e76\u5728\u5347\u7ea7\u5185\u6838\u65f6\u5bf9\u5e94\u518d\u6b21\u7f16\u8bd1\u65b0\u7248\u672c\u5185\u6838\u6a21\u5757\u3002 \u8be5\u534f\u8bae\u76ee\u524d\u672a\u88ab\u667a\u80fd\u7f51\u5361\u652f\u6301\uff0c\u65e0\u6cd5\u4f7f\u7528 OVS \u7684\u5378\u8f7d\u80fd\u529b\u3002","title":"STT"},{"location":"reference/tunnel-protocol/#_2","text":"https://ipwithease.com/vxlan-vs-geneve-understand-the-difference/ OVN FAQ What is Geneve \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u53c2\u8003\u8d44\u6599"},{"location":"reference/underlay-topology/","text":"Underlay \u6d41\u91cf\u62d3\u6251 \u00b6 \u672c\u6587\u6863\u4ecb\u7ecd Underlay \u6a21\u5f0f\u4e0b\u6d41\u91cf\u5728\u4e0d\u540c\u60c5\u51b5\u4e0b\u7684\u8f6c\u53d1\u8def\u5f84\u3002 \u540c\u8282\u70b9\u540c\u5b50\u7f51 \u00b6 \u5185\u90e8\u903b\u8f91\u4ea4\u6362\u673a\u76f4\u63a5\u4ea4\u6362\u6570\u636e\u5305\uff0c\u4e0d\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\u3002 \u8de8\u8282\u70b9\u540c\u5b50\u7f51 \u00b6 \u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u4ea4\u6362\u673a\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u8fdb\u884c\u4ea4\u6362\u3002 \u540c\u8282\u70b9\u4e0d\u540c\u5b50\u7f51 \u00b6 \u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u6b64\u5904 br-provider-1 \u548c br-provider-2 \u53ef\u4ee5\u662f\u540c\u4e00\u4e2a OVS \u7f51\u6865\uff0c\u5373\u591a\u4e2a\u4e0d\u540c\u5b50\u7f51\u53ef\u4ee5\u4f7f\u7528\u540c\u4e00\u4e2a Provider Network\u3002 \u8de8\u8282\u70b9\u4e0d\u540c\u5b50\u7f51 \u00b6 \u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u8bbf\u95ee\u5916\u90e8 \u00b6 \u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u8282\u70b9\u4e0e Pod \u4e4b\u95f4\u7684\u901a\u4fe1\u5927\u4f53\u4e0a\u4e5f\u9075\u5faa\u6b64\u903b\u8f91\u3002 \u65e0 Vlan Tag \u4e0b\u603b\u89c8 \u00b6 \u591a VLAN \u603b\u89c8 \u00b6 Pod \u8bbf\u95ee Service IP \u00b6 Kube-OVN \u4e3a\u6bcf\u4e2a Kubernetes Service \u5728\u6bcf\u4e2a\u5b50\u7f51\u7684\u903b\u8f91\u4ea4\u6362\u673a\u4e0a\u914d\u7f6e\u4e86\u8d1f\u8f7d\u5747\u8861\u3002 \u5f53 Pod \u901a\u8fc7\u8bbf\u95ee Service IP \u8bbf\u95ee\u5176\u5b83 Pod \u65f6\uff0c\u4f1a\u6784\u9020\u4e00\u4e2a\u76ee\u7684\u5730\u5740\u4e3a Service IP\u3001\u76ee\u7684 MAC \u5730\u5740\u4e3a\u7f51\u5173 MAC \u5730\u5740\u7684\u7f51\u7edc\u5305\u3002 \u7f51\u7edc\u5305\u8fdb\u5165\u903b\u8f91\u4ea4\u6362\u673a\u540e\uff0c\u8d1f\u8f7d\u5747\u8861\u4f1a\u5bf9\u7f51\u7edc\u5305\u8fdb\u884c\u62e6\u622a\u548c DNAT \u5904\u7406\uff0c\u5c06\u76ee\u7684 IP \u548c\u7aef\u53e3\u4fee\u6539\u4e3a Service \u5bf9\u5e94\u7684\u67d0\u4e2a Endpoint \u7684 IP \u548c\u7aef\u53e3\u3002 \u7531\u4e8e\u903b\u8f91\u4ea4\u6362\u673a\u5e76\u672a\u4fee\u6539\u7f51\u7edc\u5305\u7684\u4e8c\u5c42\u76ee\u7684 MAC \u5730\u5740\uff0c\u7f51\u7edc\u5305\u5728\u8fdb\u5165\u5916\u90e8\u4ea4\u6362\u673a\u540e\u4ecd\u7136\u4f1a\u9001\u5230\u5916\u90e8\u7f51\u5173\uff0c\u6b64\u65f6\u9700\u8981\u5916\u90e8\u7f51\u5173\u5bf9\u7f51\u7edc\u5305\u8fdb\u884c\u8f6c\u53d1\u3002 Service \u540e\u7aef\u4e3a\u540c\u8282\u70b9\u540c\u5b50\u7f51 Pod \u00b6 Service \u540e\u7aef\u4e3a\u540c\u8282\u70b9\u4e0d\u540c\u5b50\u7f51 Pod \u00b6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Underlay \u6d41\u91cf\u62d3\u6251"},{"location":"reference/underlay-topology/#underlay","text":"\u672c\u6587\u6863\u4ecb\u7ecd Underlay \u6a21\u5f0f\u4e0b\u6d41\u91cf\u5728\u4e0d\u540c\u60c5\u51b5\u4e0b\u7684\u8f6c\u53d1\u8def\u5f84\u3002","title":"Underlay \u6d41\u91cf\u62d3\u6251"},{"location":"reference/underlay-topology/#_1","text":"\u5185\u90e8\u903b\u8f91\u4ea4\u6362\u673a\u76f4\u63a5\u4ea4\u6362\u6570\u636e\u5305\uff0c\u4e0d\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\u3002","title":"\u540c\u8282\u70b9\u540c\u5b50\u7f51"},{"location":"reference/underlay-topology/#_2","text":"\u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u4ea4\u6362\u673a\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u8fdb\u884c\u4ea4\u6362\u3002","title":"\u8de8\u8282\u70b9\u540c\u5b50\u7f51"},{"location":"reference/underlay-topology/#_3","text":"\u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u6b64\u5904 br-provider-1 \u548c br-provider-2 \u53ef\u4ee5\u662f\u540c\u4e00\u4e2a OVS \u7f51\u6865\uff0c\u5373\u591a\u4e2a\u4e0d\u540c\u5b50\u7f51\u53ef\u4ee5\u4f7f\u7528\u540c\u4e00\u4e2a Provider Network\u3002","title":"\u540c\u8282\u70b9\u4e0d\u540c\u5b50\u7f51"},{"location":"reference/underlay-topology/#_4","text":"\u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002","title":"\u8de8\u8282\u70b9\u4e0d\u540c\u5b50\u7f51"},{"location":"reference/underlay-topology/#_5","text":"\u6570\u636e\u5305\u7ecf\u7531\u8282\u70b9\u7f51\u5361\u8fdb\u5165\u5916\u90e8\u7f51\u7edc\uff0c\u7531\u5916\u90e8\u4ea4\u6362\u673a\u53ca\u8def\u7531\u5668\u8fdb\u884c\u4ea4\u6362\u548c\u8def\u7531\u8f6c\u53d1\u3002 \u8282\u70b9\u4e0e Pod \u4e4b\u95f4\u7684\u901a\u4fe1\u5927\u4f53\u4e0a\u4e5f\u9075\u5faa\u6b64\u903b\u8f91\u3002","title":"\u8bbf\u95ee\u5916\u90e8"},{"location":"reference/underlay-topology/#vlan-tag","text":"","title":"\u65e0 Vlan Tag \u4e0b\u603b\u89c8"},{"location":"reference/underlay-topology/#vlan","text":"","title":"\u591a VLAN \u603b\u89c8"},{"location":"reference/underlay-topology/#pod-service-ip","text":"Kube-OVN \u4e3a\u6bcf\u4e2a Kubernetes Service \u5728\u6bcf\u4e2a\u5b50\u7f51\u7684\u903b\u8f91\u4ea4\u6362\u673a\u4e0a\u914d\u7f6e\u4e86\u8d1f\u8f7d\u5747\u8861\u3002 \u5f53 Pod \u901a\u8fc7\u8bbf\u95ee Service IP \u8bbf\u95ee\u5176\u5b83 Pod \u65f6\uff0c\u4f1a\u6784\u9020\u4e00\u4e2a\u76ee\u7684\u5730\u5740\u4e3a Service IP\u3001\u76ee\u7684 MAC \u5730\u5740\u4e3a\u7f51\u5173 MAC \u5730\u5740\u7684\u7f51\u7edc\u5305\u3002 \u7f51\u7edc\u5305\u8fdb\u5165\u903b\u8f91\u4ea4\u6362\u673a\u540e\uff0c\u8d1f\u8f7d\u5747\u8861\u4f1a\u5bf9\u7f51\u7edc\u5305\u8fdb\u884c\u62e6\u622a\u548c DNAT \u5904\u7406\uff0c\u5c06\u76ee\u7684 IP \u548c\u7aef\u53e3\u4fee\u6539\u4e3a Service \u5bf9\u5e94\u7684\u67d0\u4e2a Endpoint \u7684 IP \u548c\u7aef\u53e3\u3002 \u7531\u4e8e\u903b\u8f91\u4ea4\u6362\u673a\u5e76\u672a\u4fee\u6539\u7f51\u7edc\u5305\u7684\u4e8c\u5c42\u76ee\u7684 MAC \u5730\u5740\uff0c\u7f51\u7edc\u5305\u5728\u8fdb\u5165\u5916\u90e8\u4ea4\u6362\u673a\u540e\u4ecd\u7136\u4f1a\u9001\u5230\u5916\u90e8\u7f51\u5173\uff0c\u6b64\u65f6\u9700\u8981\u5916\u90e8\u7f51\u5173\u5bf9\u7f51\u7edc\u5305\u8fdb\u884c\u8f6c\u53d1\u3002","title":"Pod \u8bbf\u95ee Service IP"},{"location":"reference/underlay-topology/#service-pod","text":"","title":"Service \u540e\u7aef\u4e3a\u540c\u8282\u70b9\u540c\u5b50\u7f51 Pod"},{"location":"reference/underlay-topology/#service-pod_1","text":"\u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Service \u540e\u7aef\u4e3a\u540c\u8282\u70b9\u4e0d\u540c\u5b50\u7f51 Pod"},{"location":"start/one-step-install/","text":"\u4e00\u952e\u5b89\u88c5 \u00b6 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684 Kube-OVN \u5bb9\u5668\u7f51\u7edc\uff0c\u9ed8\u8ba4\u90e8\u7f72\u4e3a Overlay \u7c7b\u578b\u7f51\u7edc\u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u652f\u6301 Helm Chart \u5b89\u88c5\uff0c\u9ed8\u8ba4\u90e8\u7f72\u4e3a Overlay \u7c7b\u578b\u7f51\u7edc\u3002 \u5982\u679c\u9ed8\u8ba4\u7f51\u7edc\u9700\u8981\u642d\u5efa Underlay/Vlan \u7f51\u7edc\uff0c\u8bf7\u53c2\u8003 Underlay \u7f51\u7edc\u652f\u6301 \u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003 \u51c6\u5907\u5de5\u4f5c \u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u811a\u672c\u5b89\u88c5 \u00b6 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c \u00b6 \u6211\u4eec\u63a8\u8350\u5728\u751f\u4ea7\u73af\u5883\u4f7f\u7528\u7a33\u5b9a\u7684 release \u7248\u672c\uff0c\u8bf7\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u7a33\u5b9a\u7248\u672c\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u5982\u679c\u5bf9 master \u5206\u652f\u7684\u6700\u65b0\u529f\u80fd\u611f\u5174\u8da3\uff0c\u60f3\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5f00\u53d1\u7248\u672c\u90e8\u7f72\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u914d\u7f6e\u53c2\u6570 \u00b6 \u4f7f\u7528\u7f16\u8f91\u5668\u6253\u5f00\u811a\u672c\uff0c\u5e76\u4fee\u6539\u4e0b\u5217\u53d8\u91cf\u4e3a\u9884\u671f\u503c\uff1a REGISTRY = \"kubeovn\" # \u955c\u50cf\u4ed3\u5e93\u5730\u5740 VERSION = \"v1.12.0\" # \u955c\u50cf\u7248\u672c/Tag POD_CIDR = \"10.16.0.0/16\" # \u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e0d\u8981\u548c SVC/NODE/JOIN CIDR \u91cd\u53e0 SVC_CIDR = \"10.96.0.0/12\" # \u9700\u8981\u548c apiserver \u7684 service-cluster-ip-range \u4fdd\u6301\u4e00\u81f4 JOIN_CIDR = \"100.64.0.0/16\" # Pod \u548c\u4e3b\u673a\u901a\u4fe1\u7f51\u7edc CIDR\uff0c\u4e0d\u8981\u548c SVC/NODE/POD CIDR \u91cd\u53e0 LABEL = \"node-role.kubernetes.io/master\" # \u90e8\u7f72 OVN DB \u8282\u70b9\u7684\u6807\u7b7e IFACE = \"\" # \u5bb9\u5668\u7f51\u7edc\u6240\u4f7f\u7528\u7684\u7684\u5bbf\u4e3b\u673a\u7f51\u5361\u540d\uff0c\u5982\u679c\u4e3a\u7a7a\u5219\u4f7f\u7528 Kubernetes \u4e2d\u7684 Node IP \u6240\u5728\u7f51\u5361 TUNNEL_TYPE = \"geneve\" # \u96a7\u9053\u5c01\u88c5\u534f\u8bae\uff0c\u53ef\u9009 geneve, vxlan \u6216 stt\uff0cstt \u9700\u8981\u5355\u72ec\u7f16\u8bd1 ovs \u5185\u6838\u6a21\u5757 \u53ef\u4f7f\u7528\u6b63\u5219\u8868\u8fbe\u5f0f\u6765\u5339\u914d\u7f51\u5361\u540d\uff0c\u4f8b\u5982 IFACE=enp6s0f0,eth.* \u3002 \u6267\u884c\u5b89\u88c5\u811a\u672c \u00b6 bash install.sh \u7b49\u5f85\u5b89\u88c5\u5b8c\u6210\u3002 Helm Chart \u5b89\u88c5 \u00b6 \u7531\u4e8e Kube-OVN \u7684\u5b89\u88c5\uff0c\u9700\u8981\u8bbe\u7f6e\u4e00\u4e9b\u53c2\u6570\uff0c\u56e0\u6b64\u4f7f\u7528 Helm \u5b89\u88c5 Kube-OVN\uff0c\u9700\u8981\u6309\u7167\u4ee5\u4e0b\u6b65\u9aa4\u6267\u884c\u3002 \u67e5\u770b\u8282\u70b9 IP \u5730\u5740 \u00b6 $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-ovn-control-plane NotReady control-plane 20h v1.26.0 172 .18.0.3 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 kube-ovn-worker NotReady <none> 20h v1.26.0 172 .18.0.2 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 \u53bb\u6389\u96c6\u7fa4 master \u8282\u70b9\u6c61\u70b9 \u00b6 $ kubectl taint node kube-ovn-control-plane node-role.kubernetes.io/control-plane:NoSchedule- node/kube-ovn-control-plane untainted \u5982\u679c\u786e\u5b9a\u4e0d\u9700\u8981\u5728 master \u8282\u70b9\u8c03\u5ea6\u4e1a\u52a1 Pod\uff0c\u8fd9\u4e00\u6b65\u53ef\u4ee5\u8df3\u8fc7\u3002 \u7ed9\u8282\u70b9\u6dfb\u52a0 label \u00b6 $ kubectl label node -lbeta.kubernetes.io/os = linux kubernetes.io/os = linux --overwrite node/kube-ovn-control-plane not labeled node/kube-ovn-worker not labeled $ kubectl label node -lnode-role.kubernetes.io/control-plane kube-ovn/role = master --overwrite node/kube-ovn-control-plane labeled # \u4ee5\u4e0b label \u7528\u4e8e dpdk \u955c\u50cf\u7684\u5b89\u88c5\uff0c\u975e dpdk \u60c5\u51b5\uff0c\u53ef\u4ee5\u5ffd\u7565 $ kubectl label node -lovn.kubernetes.io/ovs_dp_type! = userspace ovn.kubernetes.io/ovs_dp_type = kernel --overwrite node/kube-ovn-control-plane labeled node/kube-ovn-worker labeled \u6dfb\u52a0 Helm Repo \u4fe1\u606f \u00b6 $ helm repo add kubeovn https://kubeovn.github.io/kube-ovn/ \"kubeovn\" has been added to your repositories $ helm repo list NAME URL kubeovn https://kubeovn.github.io/kube-ovn/ $ helm search repo kubeovn NAME CHART VERSION APP VERSION DESCRIPTION kubeovn/kube-ovn 0 .1.0 1 .12.0 Helm chart for Kube-OVN \u6267\u884c helm install \u5b89\u88c5 Kube-OVN \u00b6 Node0IP\u3001Node1IP\u3001Node2IP \u53c2\u6570\u5206\u522b\u4e3a\u96c6\u7fa4 master \u8282\u70b9\u7684 IP \u5730\u5740\u3002\u5176\u4ed6\u53c2\u6570\u7684\u8bbe\u7f6e\uff0c\u53ef\u4ee5\u53c2\u8003 values.yaml \u6587\u4ef6\u4e2d\u53d8\u91cf\u5b9a\u4e49\u3002 # \u5355 master \u8282\u70b9\u73af\u5883\u5b89\u88c5 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } # \u4ee5\u4e0a\u8fb9\u7684 node \u4fe1\u606f\u4e3a\u4f8b\uff0c\u6267\u884c\u5b89\u88c5\u547d\u4ee4 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = 172 .18.0.3 NAME: kube-ovn LAST DEPLOYED: Fri Mar 31 12 :43:43 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None # \u9ad8\u53ef\u7528\u96c6\u7fa4\u5b89\u88c5 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } , ${ Node1IP } , ${ Node2IP } , --set replicaCount = 3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4e00\u952e\u5b89\u88c5"},{"location":"start/one-step-install/#_1","text":"Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684 Kube-OVN \u5bb9\u5668\u7f51\u7edc\uff0c\u9ed8\u8ba4\u90e8\u7f72\u4e3a Overlay \u7c7b\u578b\u7f51\u7edc\u3002 \u4ece Kube-OVN v1.12.0 \u7248\u672c\u5f00\u59cb\uff0c\u652f\u6301 Helm Chart \u5b89\u88c5\uff0c\u9ed8\u8ba4\u90e8\u7f72\u4e3a Overlay \u7c7b\u578b\u7f51\u7edc\u3002 \u5982\u679c\u9ed8\u8ba4\u7f51\u7edc\u9700\u8981\u642d\u5efa Underlay/Vlan \u7f51\u7edc\uff0c\u8bf7\u53c2\u8003 Underlay \u7f51\u7edc\u652f\u6301 \u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003 \u51c6\u5907\u5de5\u4f5c \u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002","title":"\u4e00\u952e\u5b89\u88c5"},{"location":"start/one-step-install/#_2","text":"","title":"\u811a\u672c\u5b89\u88c5"},{"location":"start/one-step-install/#_3","text":"\u6211\u4eec\u63a8\u8350\u5728\u751f\u4ea7\u73af\u5883\u4f7f\u7528\u7a33\u5b9a\u7684 release \u7248\u672c\uff0c\u8bf7\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u7a33\u5b9a\u7248\u672c\u5b89\u88c5\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u5982\u679c\u5bf9 master \u5206\u652f\u7684\u6700\u65b0\u529f\u80fd\u611f\u5174\u8da3\uff0c\u60f3\u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5f00\u53d1\u7248\u672c\u90e8\u7f72\u811a\u672c\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c"},{"location":"start/one-step-install/#_4","text":"\u4f7f\u7528\u7f16\u8f91\u5668\u6253\u5f00\u811a\u672c\uff0c\u5e76\u4fee\u6539\u4e0b\u5217\u53d8\u91cf\u4e3a\u9884\u671f\u503c\uff1a REGISTRY = \"kubeovn\" # \u955c\u50cf\u4ed3\u5e93\u5730\u5740 VERSION = \"v1.12.0\" # \u955c\u50cf\u7248\u672c/Tag POD_CIDR = \"10.16.0.0/16\" # \u9ed8\u8ba4\u5b50\u7f51 CIDR \u4e0d\u8981\u548c SVC/NODE/JOIN CIDR \u91cd\u53e0 SVC_CIDR = \"10.96.0.0/12\" # \u9700\u8981\u548c apiserver \u7684 service-cluster-ip-range \u4fdd\u6301\u4e00\u81f4 JOIN_CIDR = \"100.64.0.0/16\" # Pod \u548c\u4e3b\u673a\u901a\u4fe1\u7f51\u7edc CIDR\uff0c\u4e0d\u8981\u548c SVC/NODE/POD CIDR \u91cd\u53e0 LABEL = \"node-role.kubernetes.io/master\" # \u90e8\u7f72 OVN DB \u8282\u70b9\u7684\u6807\u7b7e IFACE = \"\" # \u5bb9\u5668\u7f51\u7edc\u6240\u4f7f\u7528\u7684\u7684\u5bbf\u4e3b\u673a\u7f51\u5361\u540d\uff0c\u5982\u679c\u4e3a\u7a7a\u5219\u4f7f\u7528 Kubernetes \u4e2d\u7684 Node IP \u6240\u5728\u7f51\u5361 TUNNEL_TYPE = \"geneve\" # \u96a7\u9053\u5c01\u88c5\u534f\u8bae\uff0c\u53ef\u9009 geneve, vxlan \u6216 stt\uff0cstt \u9700\u8981\u5355\u72ec\u7f16\u8bd1 ovs \u5185\u6838\u6a21\u5757 \u53ef\u4f7f\u7528\u6b63\u5219\u8868\u8fbe\u5f0f\u6765\u5339\u914d\u7f51\u5361\u540d\uff0c\u4f8b\u5982 IFACE=enp6s0f0,eth.* \u3002","title":"\u4fee\u6539\u914d\u7f6e\u53c2\u6570"},{"location":"start/one-step-install/#_5","text":"bash install.sh \u7b49\u5f85\u5b89\u88c5\u5b8c\u6210\u3002","title":"\u6267\u884c\u5b89\u88c5\u811a\u672c"},{"location":"start/one-step-install/#helm-chart","text":"\u7531\u4e8e Kube-OVN \u7684\u5b89\u88c5\uff0c\u9700\u8981\u8bbe\u7f6e\u4e00\u4e9b\u53c2\u6570\uff0c\u56e0\u6b64\u4f7f\u7528 Helm \u5b89\u88c5 Kube-OVN\uff0c\u9700\u8981\u6309\u7167\u4ee5\u4e0b\u6b65\u9aa4\u6267\u884c\u3002","title":"Helm Chart \u5b89\u88c5"},{"location":"start/one-step-install/#ip","text":"$ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-ovn-control-plane NotReady control-plane 20h v1.26.0 172 .18.0.3 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 kube-ovn-worker NotReady <none> 20h v1.26.0 172 .18.0.2 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9","title":"\u67e5\u770b\u8282\u70b9 IP \u5730\u5740"},{"location":"start/one-step-install/#master","text":"$ kubectl taint node kube-ovn-control-plane node-role.kubernetes.io/control-plane:NoSchedule- node/kube-ovn-control-plane untainted \u5982\u679c\u786e\u5b9a\u4e0d\u9700\u8981\u5728 master \u8282\u70b9\u8c03\u5ea6\u4e1a\u52a1 Pod\uff0c\u8fd9\u4e00\u6b65\u53ef\u4ee5\u8df3\u8fc7\u3002","title":"\u53bb\u6389\u96c6\u7fa4 master \u8282\u70b9\u6c61\u70b9"},{"location":"start/one-step-install/#label","text":"$ kubectl label node -lbeta.kubernetes.io/os = linux kubernetes.io/os = linux --overwrite node/kube-ovn-control-plane not labeled node/kube-ovn-worker not labeled $ kubectl label node -lnode-role.kubernetes.io/control-plane kube-ovn/role = master --overwrite node/kube-ovn-control-plane labeled # \u4ee5\u4e0b label \u7528\u4e8e dpdk \u955c\u50cf\u7684\u5b89\u88c5\uff0c\u975e dpdk \u60c5\u51b5\uff0c\u53ef\u4ee5\u5ffd\u7565 $ kubectl label node -lovn.kubernetes.io/ovs_dp_type! = userspace ovn.kubernetes.io/ovs_dp_type = kernel --overwrite node/kube-ovn-control-plane labeled node/kube-ovn-worker labeled","title":"\u7ed9\u8282\u70b9\u6dfb\u52a0 label"},{"location":"start/one-step-install/#helm-repo","text":"$ helm repo add kubeovn https://kubeovn.github.io/kube-ovn/ \"kubeovn\" has been added to your repositories $ helm repo list NAME URL kubeovn https://kubeovn.github.io/kube-ovn/ $ helm search repo kubeovn NAME CHART VERSION APP VERSION DESCRIPTION kubeovn/kube-ovn 0 .1.0 1 .12.0 Helm chart for Kube-OVN","title":"\u6dfb\u52a0 Helm Repo \u4fe1\u606f"},{"location":"start/one-step-install/#helm-install-kube-ovn","text":"Node0IP\u3001Node1IP\u3001Node2IP \u53c2\u6570\u5206\u522b\u4e3a\u96c6\u7fa4 master \u8282\u70b9\u7684 IP \u5730\u5740\u3002\u5176\u4ed6\u53c2\u6570\u7684\u8bbe\u7f6e\uff0c\u53ef\u4ee5\u53c2\u8003 values.yaml \u6587\u4ef6\u4e2d\u53d8\u91cf\u5b9a\u4e49\u3002 # \u5355 master \u8282\u70b9\u73af\u5883\u5b89\u88c5 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } # \u4ee5\u4e0a\u8fb9\u7684 node \u4fe1\u606f\u4e3a\u4f8b\uff0c\u6267\u884c\u5b89\u88c5\u547d\u4ee4 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = 172 .18.0.3 NAME: kube-ovn LAST DEPLOYED: Fri Mar 31 12 :43:43 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None # \u9ad8\u53ef\u7528\u96c6\u7fa4\u5b89\u88c5 $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } , ${ Node1IP } , ${ Node2IP } , --set replicaCount = 3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6267\u884c helm install \u5b89\u88c5 Kube-OVN"},{"location":"start/prepare/","text":"\u51c6\u5907\u5de5\u4f5c \u00b6 Kube-OVN \u662f\u4e00\u4e2a\u7b26\u5408 CNI \u89c4\u8303\u7684\u7f51\u7edc\u7ec4\u4ef6\uff0c\u5176\u8fd0\u884c\u9700\u8981\u4f9d\u8d56 Kubernetes \u73af\u5883\u53ca\u5bf9\u5e94\u7684\u5185\u6838\u7f51\u7edc\u6a21\u5757\u3002 \u4ee5\u4e0b\u662f\u901a\u8fc7\u6d4b\u8bd5\u7684\u64cd\u4f5c\u7cfb\u7edf\u548c\u8f6f\u4ef6\u7248\u672c\uff0c\u73af\u5883\u914d\u7f6e\u548c\u6240\u9700\u8981\u5f00\u653e\u7684\u7aef\u53e3\u4fe1\u606f\u3002 \u8f6f\u4ef6\u7248\u672c \u00b6 Kubernetes >= 1.23\u3002 Docker >= 1.12.6, Containerd >= 1.3.4\u3002 \u64cd\u4f5c\u7cfb\u7edf: CentOS 7/8, Ubuntu 16.04/18.04/20.04\u3002 \u5176\u4ed6 Linux \u53d1\u884c\u7248\uff0c\u9700\u8981\u68c0\u67e5\u4e00\u4e0b\u5185\u6838\u6a21\u5757\u662f\u5426\u5b58\u5728 geneve , openvswitch , ip_tables \u548c iptable_nat \uff0cKube-OVN \u6b63\u5e38\u5de5\u4f5c\u4f9d\u8d56\u4e0a\u8ff0\u6a21\u5757\u3002 \u6ce8\u610f\u4e8b\u9879 \uff1a \u5982\u679c\u5185\u6838\u7248\u672c\u4e3a 3.10.0-862 \u5185\u6838 netfilter \u6a21\u5757\u5b58\u5728 bug \u4f1a\u5bfc\u81f4 Kube-OVN \u5185\u7f6e\u8d1f\u8f7d\u5747\u8861\u5668\u65e0\u6cd5\u5de5\u4f5c\uff0c\u9700\u8981\u5bf9\u5185\u6838\u5347\u7ea7\uff0c\u5efa\u8bae\u4f7f\u7528 CentOS \u5b98\u65b9\u5bf9\u5e94\u7248\u672c\u6700\u65b0\u5185\u6838\u4fdd\u8bc1\u7cfb\u7edf\u7684\u5b89\u5168\u3002\u76f8\u5173\u5185\u6838 bug \u53c2\u8003 Floating IPs broken after kernel upgrade to Centos/RHEL 7.5 - DNAT not working \u3002 Rocky Linux 8.6 \u7684\u5185\u6838 4.18.0-372.9.1.el8.x86_64 \u5b58\u5728 TCP \u901a\u4fe1\u95ee\u9898 TCP connection failed in Rocky Linux 8.6 \uff0c\u8bf7\u5347\u7ea7\u5185\u6838\u81f3 4.18.0-372.13.1.el8_6.x86_64 \u6216\u66f4\u9ad8\u7248\u672c\u3002 \u5982\u679c\u5185\u6838\u7248\u672c\u4e3a 4.4 \u5219\u5bf9\u5e94\u7684\u5185\u6838 openvswitch \u6a21\u5757\u5b58\u5728\u95ee\u9898\uff0c\u5efa\u8bae\u5347\u7ea7\u6216\u624b\u52a8\u7f16\u8bd1 openvswitch \u65b0\u7248\u672c\u6a21\u5757\u8fdb\u884c\u66f4\u65b0 Geneve \u96a7\u9053\u5efa\u7acb\u9700\u8981\u68c0\u67e5 IPv6\uff0c\u53ef\u901a\u8fc7 cat /proc/cmdline \u68c0\u67e5\u5185\u6838\u542f\u52a8\u53c2\u6570\uff0c \u76f8\u5173\u5185\u6838 bug \u8bf7\u53c2\u8003 Geneve tunnels don't work when ipv6 is disabled \u3002 \u73af\u5883\u914d\u7f6e \u00b6 Kernel \u542f\u52a8\u9700\u8981\u5f00\u542f ipv6, \u5982\u679c kernel \u542f\u52a8\u53c2\u6570\u5305\u542b ipv6.disable=1 \u9700\u8981\u5c06\u5176\u8bbe\u7f6e\u4e3a 0\u3002 kube-proxy \u6b63\u5e38\u5de5\u4f5c\uff0cKube-OVN \u53ef\u4ee5\u901a\u8fc7 SVC IP \u8bbf\u95ee\u5230 kube-apiserver \u3002 \u786e\u8ba4 kubelet \u914d\u7f6e\u53c2\u6570\u5f00\u542f\u4e86 CNI\uff0c\u5e76\u4e14\u914d\u7f6e\u5728\u6807\u51c6\u8def\u5f84\u4e0b, kubelet \u542f\u52a8\u65f6\u5e94\u5305\u542b\u5982\u4e0b\u53c2\u6570 --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d \u3002 \u786e\u8ba4\u672a\u5b89\u88c5\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u6216\u8005\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u5df2\u7ecf\u88ab\u6e05\u9664\uff0c\u68c0\u67e5 /etc/cni/net.d/ \u8def\u5f84\u4e0b\u65e0\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u914d\u7f6e\u6587\u4ef6\u3002\u5982\u679c\u4e4b\u524d\u5b89\u88c5\u8fc7\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u5efa\u8bae\u5220\u9664\u540e\u91cd\u542f\u673a\u5668\u6e05\u7406\u6b8b\u7559\u7f51\u7edc\u8d44\u6e90\u3002 \u7aef\u53e3\u4fe1\u606f \u00b6 \u7ec4\u4ef6 \u7aef\u53e3 \u7528\u9014 ovn-central 6641/tcp, 6642/tcp, 6643/tcp, 6644/tcp ovn-db \u548c raft server \u76d1\u542c\u7aef\u53e3 ovs-ovn Geneve 6081/udp, STT 7471/tcp, Vxlan 4789/udp \u96a7\u9053\u7aef\u53e3 kube-ovn-controller 10660/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 kube-ovn-daemon 10665/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 kube-ovn-monitor 10661/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u51c6\u5907\u5de5\u4f5c"},{"location":"start/prepare/#_1","text":"Kube-OVN \u662f\u4e00\u4e2a\u7b26\u5408 CNI \u89c4\u8303\u7684\u7f51\u7edc\u7ec4\u4ef6\uff0c\u5176\u8fd0\u884c\u9700\u8981\u4f9d\u8d56 Kubernetes \u73af\u5883\u53ca\u5bf9\u5e94\u7684\u5185\u6838\u7f51\u7edc\u6a21\u5757\u3002 \u4ee5\u4e0b\u662f\u901a\u8fc7\u6d4b\u8bd5\u7684\u64cd\u4f5c\u7cfb\u7edf\u548c\u8f6f\u4ef6\u7248\u672c\uff0c\u73af\u5883\u914d\u7f6e\u548c\u6240\u9700\u8981\u5f00\u653e\u7684\u7aef\u53e3\u4fe1\u606f\u3002","title":"\u51c6\u5907\u5de5\u4f5c"},{"location":"start/prepare/#_2","text":"Kubernetes >= 1.23\u3002 Docker >= 1.12.6, Containerd >= 1.3.4\u3002 \u64cd\u4f5c\u7cfb\u7edf: CentOS 7/8, Ubuntu 16.04/18.04/20.04\u3002 \u5176\u4ed6 Linux \u53d1\u884c\u7248\uff0c\u9700\u8981\u68c0\u67e5\u4e00\u4e0b\u5185\u6838\u6a21\u5757\u662f\u5426\u5b58\u5728 geneve , openvswitch , ip_tables \u548c iptable_nat \uff0cKube-OVN \u6b63\u5e38\u5de5\u4f5c\u4f9d\u8d56\u4e0a\u8ff0\u6a21\u5757\u3002 \u6ce8\u610f\u4e8b\u9879 \uff1a \u5982\u679c\u5185\u6838\u7248\u672c\u4e3a 3.10.0-862 \u5185\u6838 netfilter \u6a21\u5757\u5b58\u5728 bug \u4f1a\u5bfc\u81f4 Kube-OVN \u5185\u7f6e\u8d1f\u8f7d\u5747\u8861\u5668\u65e0\u6cd5\u5de5\u4f5c\uff0c\u9700\u8981\u5bf9\u5185\u6838\u5347\u7ea7\uff0c\u5efa\u8bae\u4f7f\u7528 CentOS \u5b98\u65b9\u5bf9\u5e94\u7248\u672c\u6700\u65b0\u5185\u6838\u4fdd\u8bc1\u7cfb\u7edf\u7684\u5b89\u5168\u3002\u76f8\u5173\u5185\u6838 bug \u53c2\u8003 Floating IPs broken after kernel upgrade to Centos/RHEL 7.5 - DNAT not working \u3002 Rocky Linux 8.6 \u7684\u5185\u6838 4.18.0-372.9.1.el8.x86_64 \u5b58\u5728 TCP \u901a\u4fe1\u95ee\u9898 TCP connection failed in Rocky Linux 8.6 \uff0c\u8bf7\u5347\u7ea7\u5185\u6838\u81f3 4.18.0-372.13.1.el8_6.x86_64 \u6216\u66f4\u9ad8\u7248\u672c\u3002 \u5982\u679c\u5185\u6838\u7248\u672c\u4e3a 4.4 \u5219\u5bf9\u5e94\u7684\u5185\u6838 openvswitch \u6a21\u5757\u5b58\u5728\u95ee\u9898\uff0c\u5efa\u8bae\u5347\u7ea7\u6216\u624b\u52a8\u7f16\u8bd1 openvswitch \u65b0\u7248\u672c\u6a21\u5757\u8fdb\u884c\u66f4\u65b0 Geneve \u96a7\u9053\u5efa\u7acb\u9700\u8981\u68c0\u67e5 IPv6\uff0c\u53ef\u901a\u8fc7 cat /proc/cmdline \u68c0\u67e5\u5185\u6838\u542f\u52a8\u53c2\u6570\uff0c \u76f8\u5173\u5185\u6838 bug \u8bf7\u53c2\u8003 Geneve tunnels don't work when ipv6 is disabled \u3002","title":"\u8f6f\u4ef6\u7248\u672c"},{"location":"start/prepare/#_3","text":"Kernel \u542f\u52a8\u9700\u8981\u5f00\u542f ipv6, \u5982\u679c kernel \u542f\u52a8\u53c2\u6570\u5305\u542b ipv6.disable=1 \u9700\u8981\u5c06\u5176\u8bbe\u7f6e\u4e3a 0\u3002 kube-proxy \u6b63\u5e38\u5de5\u4f5c\uff0cKube-OVN \u53ef\u4ee5\u901a\u8fc7 SVC IP \u8bbf\u95ee\u5230 kube-apiserver \u3002 \u786e\u8ba4 kubelet \u914d\u7f6e\u53c2\u6570\u5f00\u542f\u4e86 CNI\uff0c\u5e76\u4e14\u914d\u7f6e\u5728\u6807\u51c6\u8def\u5f84\u4e0b, kubelet \u542f\u52a8\u65f6\u5e94\u5305\u542b\u5982\u4e0b\u53c2\u6570 --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d \u3002 \u786e\u8ba4\u672a\u5b89\u88c5\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u6216\u8005\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u5df2\u7ecf\u88ab\u6e05\u9664\uff0c\u68c0\u67e5 /etc/cni/net.d/ \u8def\u5f84\u4e0b\u65e0\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u914d\u7f6e\u6587\u4ef6\u3002\u5982\u679c\u4e4b\u524d\u5b89\u88c5\u8fc7\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u5efa\u8bae\u5220\u9664\u540e\u91cd\u542f\u673a\u5668\u6e05\u7406\u6b8b\u7559\u7f51\u7edc\u8d44\u6e90\u3002","title":"\u73af\u5883\u914d\u7f6e"},{"location":"start/prepare/#_4","text":"\u7ec4\u4ef6 \u7aef\u53e3 \u7528\u9014 ovn-central 6641/tcp, 6642/tcp, 6643/tcp, 6644/tcp ovn-db \u548c raft server \u76d1\u542c\u7aef\u53e3 ovs-ovn Geneve 6081/udp, STT 7471/tcp, Vxlan 4789/udp \u96a7\u9053\u7aef\u53e3 kube-ovn-controller 10660/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 kube-ovn-daemon 10665/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 kube-ovn-monitor 10661/tcp \u76d1\u63a7\u76d1\u542c\u7aef\u53e3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7aef\u53e3\u4fe1\u606f"},{"location":"start/sealos-install/","text":"\u4f7f\u7528 sealos \u4e00\u952e\u90e8\u7f72 Kubernetes \u548c Kube-OVN \u00b6 sealos \u4f5c\u4e3a Kubernetes \u7684\u4e00\u4e2a\u53d1\u884c\u7248\uff0c\u901a\u8fc7\u6781\u7b80\u7684\u4f7f\u7528\u65b9\u5f0f\u548c\u56fd\u5185\u7684\u955c\u50cf\u4ed3\u5e93\uff0c\u53ef\u4ee5\u5e2e\u52a9\u7528\u6237\u5feb\u901f\u4ece\u96f6\u521d\u59cb\u5316\u4e00\u4e2a\u5bb9\u5668\u96c6\u7fa4\u3002 \u901a\u8fc7\u4f7f\u7528 sealos \u7528\u6237\u53ef\u4ee5\u901a\u8fc7\u4e00\u6761\u547d\u4ee4\u5728\u51e0\u5206\u949f\u5185\u90e8\u7f72\u51fa\u4e00\u4e2a\u5b89\u88c5\u597d Kube-OVN \u7684 Kubernetes \u96c6\u7fa4\u3002 \u4e0b\u8f7d\u5b89\u88c5 sealos \u00b6 AMD64 ARM64 wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_amd64.tar.gz && \\ tar -zxvf sealos_4.1.4_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_arm64.tar.gz && \\ tar -zxvf sealos_4.1.4_linux_arm64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin \u90e8\u7f72 Kubernetes \u548c Kube-OVN \u00b6 ```bash sealos run labring/kubernetes:v1.24.3 labring/kube-ovn:v1.10.5 \\ --masters [masters ips seperated by comma] \\ --nodes [nodes ips seperated by comma] -p [your-ssh-passwd] ``` \u7b49\u5f85\u90e8\u7f72\u5b8c\u6210 \u00b6 ```bash [Step 6/6] Finish ,,,, ,::, ,,::,,,, ,,,,,::::::::::::,,,,, ,,,::::::::::::::::::::::,,, ,,::::::::::::::::::::::::::::,, ,,::::::::::::::::::::::::::::::::,, ,::::::::::::::::::::::::::::::::::::, ,:::::::::::::,, ,,:::::,,,::::::::::, ,,:::::::::::::, ,::, ,:::::::::, ,:::::::::::::, :x, ,:: :, ,:::::::::, ,:::::::::::::::, ,,, ,::, ,, ,::::::::::, ,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x ,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x ,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx ,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx ,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,:: ,::::, ,::::, ,::::, ,::::, ,::::, ,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,, ,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::, ,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::, ,::::, ,::::, ,::::, ,::::::::::::,,,,, ,,,, ,::::, ,,,, ,,,::::,,,, ,::::, ,,::, Thanks for choosing Kube-OVN! For more advanced features, please read https://github.com/kubeovn/kube-ovn#documents If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose 2022-08-10T16:31:34 info succeeded in creating a new cluster, enjoy it! 2022-08-10T16:31:34 info ___ ___ ___ ___ ___ ___ /\\ \\ /\\ \\ /\\ \\ /\\__\\ /\\ \\ /\\ \\ /::\\ \\ /::\\ \\ /::\\ \\ /:/ / /::\\ \\ /::\\ \\ /:/\\ \\ \\ /:/\\:\\ \\ /:/\\:\\ \\ /:/ / /:/\\:\\ \\ /:/\\ \\ \\ _\\:\\~\\ \\ \\ /::\\~\\:\\ \\ /::\\~\\:\\ \\ /:/ / /:/ \\:\\ \\ _\\:\\~\\ \\ \\ /\\ \\:\\ \\ \\__\\ /:/\\:\\ \\:\\__\\ /:/\\:\\ \\:\\__\\ /:/__/ /:/__/ \\:\\__\\ /\\ \\:\\ \\ \\__\\ \\:\\ \\:\\ \\/__/ \\:\\~\\:\\ \\/__/ \\/__\\:\\/:/ / \\:\\ \\ \\:\\ \\ /:/ / \\:\\ \\:\\ \\/__/ \\:\\ \\:\\__\\ \\:\\ \\:\\__\\ \\::/ / \\:\\ \\ \\:\\ /:/ / \\:\\ \\:\\__\\ \\:\\/:/ / \\:\\ \\/__/ /:/ / \\:\\ \\ \\:\\/:/ / \\:\\/:/ / \\::/ / \\:\\__\\ /:/ / \\:\\__\\ \\::/ / \\::/ / \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ Website :https://www.sealos.io/ Address :github.com/labring/sealos ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u4f7f\u7528 Sealos \u4e00\u952e\u90e8\u7f72 Kubernetes \u548c Kube-OVN"},{"location":"start/sealos-install/#sealos-kubernetes-kube-ovn","text":"sealos \u4f5c\u4e3a Kubernetes \u7684\u4e00\u4e2a\u53d1\u884c\u7248\uff0c\u901a\u8fc7\u6781\u7b80\u7684\u4f7f\u7528\u65b9\u5f0f\u548c\u56fd\u5185\u7684\u955c\u50cf\u4ed3\u5e93\uff0c\u53ef\u4ee5\u5e2e\u52a9\u7528\u6237\u5feb\u901f\u4ece\u96f6\u521d\u59cb\u5316\u4e00\u4e2a\u5bb9\u5668\u96c6\u7fa4\u3002 \u901a\u8fc7\u4f7f\u7528 sealos \u7528\u6237\u53ef\u4ee5\u901a\u8fc7\u4e00\u6761\u547d\u4ee4\u5728\u51e0\u5206\u949f\u5185\u90e8\u7f72\u51fa\u4e00\u4e2a\u5b89\u88c5\u597d Kube-OVN \u7684 Kubernetes \u96c6\u7fa4\u3002","title":"\u4f7f\u7528 sealos \u4e00\u952e\u90e8\u7f72 Kubernetes \u548c Kube-OVN"},{"location":"start/sealos-install/#sealos","text":"AMD64 ARM64 wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_amd64.tar.gz && \\ tar -zxvf sealos_4.1.4_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_arm64.tar.gz && \\ tar -zxvf sealos_4.1.4_linux_arm64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin","title":"\u4e0b\u8f7d\u5b89\u88c5 sealos"},{"location":"start/sealos-install/#kubernetes-kube-ovn","text":"```bash sealos run labring/kubernetes:v1.24.3 labring/kube-ovn:v1.10.5 \\ --masters [masters ips seperated by comma] \\ --nodes [nodes ips seperated by comma] -p [your-ssh-passwd] ```","title":"\u90e8\u7f72 Kubernetes \u548c Kube-OVN"},{"location":"start/sealos-install/#_1","text":"```bash [Step 6/6] Finish ,,,, ,::, ,,::,,,, ,,,,,::::::::::::,,,,, ,,,::::::::::::::::::::::,,, ,,::::::::::::::::::::::::::::,, ,,::::::::::::::::::::::::::::::::,, ,::::::::::::::::::::::::::::::::::::, ,:::::::::::::,, ,,:::::,,,::::::::::, ,,:::::::::::::, ,::, ,:::::::::, ,:::::::::::::, :x, ,:: :, ,:::::::::, ,:::::::::::::::, ,,, ,::, ,, ,::::::::::, ,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x ,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x ,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx ,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx ,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,:: ,::::, ,::::, ,::::, ,::::, ,::::, ,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,, ,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::, ,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::, ,::::, ,::::, ,::::, ,::::::::::::,,,,, ,,,, ,::::, ,,,, ,,,::::,,,, ,::::, ,,::, Thanks for choosing Kube-OVN! For more advanced features, please read https://github.com/kubeovn/kube-ovn#documents If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose 2022-08-10T16:31:34 info succeeded in creating a new cluster, enjoy it! 2022-08-10T16:31:34 info ___ ___ ___ ___ ___ ___ /\\ \\ /\\ \\ /\\ \\ /\\__\\ /\\ \\ /\\ \\ /::\\ \\ /::\\ \\ /::\\ \\ /:/ / /::\\ \\ /::\\ \\ /:/\\ \\ \\ /:/\\:\\ \\ /:/\\:\\ \\ /:/ / /:/\\:\\ \\ /:/\\ \\ \\ _\\:\\~\\ \\ \\ /::\\~\\:\\ \\ /::\\~\\:\\ \\ /:/ / /:/ \\:\\ \\ _\\:\\~\\ \\ \\ /\\ \\:\\ \\ \\__\\ /:/\\:\\ \\:\\__\\ /:/\\:\\ \\:\\__\\ /:/__/ /:/__/ \\:\\__\\ /\\ \\:\\ \\ \\__\\ \\:\\ \\:\\ \\/__/ \\:\\~\\:\\ \\/__/ \\/__\\:\\/:/ / \\:\\ \\ \\:\\ \\ /:/ / \\:\\ \\:\\ \\/__/ \\:\\ \\:\\__\\ \\:\\ \\:\\__\\ \\::/ / \\:\\ \\ \\:\\ /:/ / \\:\\ \\:\\__\\ \\:\\/:/ / \\:\\ \\/__/ /:/ / \\:\\ \\ \\:\\/:/ / \\:\\/:/ / \\::/ / \\:\\__\\ /:/ / \\:\\__\\ \\::/ / \\::/ / \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ Website :https://www.sealos.io/ Address :github.com/labring/sealos ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7b49\u5f85\u90e8\u7f72\u5b8c\u6210"},{"location":"start/underlay/","text":"Underlay \u7f51\u7edc\u5b89\u88c5 \u00b6 \u9ed8\u8ba4\u60c5\u51b5\u4e0b Kube-OVN \u7684\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528 Geneve \u5bf9\u8de8\u4e3b\u673a\u6d41\u91cf\u8fdb\u884c\u5c01\u88c5\uff0c\u5728\u57fa\u7840\u8bbe\u65bd\u4e4b\u4e0a\u62bd\u8c61\u51fa\u4e00\u5c42\u865a\u62df\u7684 Overlay \u7f51\u7edc\u3002 \u5bf9\u4e8e\u5e0c\u671b\u5bb9\u5668\u7f51\u7edc\u76f4\u63a5\u4f7f\u7528\u7269\u7406\u7f51\u7edc\u5730\u5740\u6bb5\u60c5\u51b5\uff0c\u53ef\u4ee5\u5c06 Kube-OVN \u7684\u9ed8\u8ba4\u5b50\u7f51\u5de5\u4f5c\u5728 Underlay \u6a21\u5f0f\uff0c\u53ef\u4ee5\u76f4\u63a5\u7ed9\u5bb9\u5668\u5206\u914d\u7269\u7406\u7f51\u7edc\u4e2d\u7684\u5730\u5740\u8d44\u6e90\uff0c\u8fbe\u5230\u66f4\u597d\u7684\u6027\u80fd\u4ee5\u53ca\u548c\u7269\u7406\u7f51\u7edc\u7684\u8fde\u901a\u6027\u3002 \u529f\u80fd\u9650\u5236 \u00b6 \u7531\u4e8e\u8be5\u6a21\u5f0f\u4e0b\u5bb9\u5668\u7f51\u7edc\u76f4\u63a5\u4f7f\u7528\u7269\u7406\u7f51\u7edc\u8fdb\u884c\u4e8c\u5c42\u5305\u8f6c\u53d1\uff0cOverlay \u6a21\u5f0f\u4e0b\u7684 SNAT/EIP\uff0c \u5206\u5e03\u5f0f\u7f51\u5173/\u96c6\u4e2d\u5f0f\u7f51\u5173\u7b49 L3 \u529f\u80fd\u65e0\u6cd5\u4f7f\u7528\uff0cVPC \u7ea7\u522b\u7684\u9694\u79bb\u4e5f\u65e0\u6cd5\u5bf9 Underlay \u5b50\u7f51\u751f\u6548\u3002 \u548c Macvlan \u6bd4\u8f83 \u00b6 Kube-OVN \u7684 Underlay \u6a21\u5f0f\u548c Macvlan \u5de5\u4f5c\u6a21\u5f0f\u5341\u5206\u7c7b\u4f3c\uff0c\u5728\u529f\u80fd\u548c\u6027\u80fd\u4e0a\u4e3b\u8981\u6709\u4ee5\u4e0b\u51e0\u4e2a\u533a\u522b\uff1a \u7531\u4e8e Macvlan \u7684\u5185\u6838\u8def\u5f84\u66f4\u77ed\uff0c\u5e76\u4e14\u4e0d\u9700\u8981 OVS \u5bf9\u6570\u636e\u5305\u8fdb\u884c\u5904\u7406\uff0cMacvlan \u5728\u541e\u5410\u91cf\u548c\u5ef6\u8fdf\u6027\u80fd\u6307\u6807\u4e0a\u8868\u73b0\u4f1a\u66f4\u597d\u3002 Kube-OVN \u901a\u8fc7\u6d41\u8868\u63d0\u4f9b\u4e86 arp-proxy \u529f\u80fd\uff0c\u53ef\u4ee5\u7f13\u89e3\u5927\u89c4\u6a21\u7f51\u7edc\u4e0b\u7684 arp \u5e7f\u64ad\u98ce\u66b4\u98ce\u9669\u3002 \u7531\u4e8e Macvlan \u5de5\u4f5c\u5728\u5185\u6838\u5e95\u5c42\uff0c\u4f1a\u7ed5\u8fc7\u5bbf\u4e3b\u673a\u7684 netfilter\uff0cService \u548c NetworkPolicy \u529f\u80fd\u9700\u8981\u989d\u5916\u5f00\u53d1\u3002Kube-OVN \u901a\u8fc7 OVS \u6d41\u8868\u63d0\u4f9b\u4e86 Service \u548c NetworkPolicy \u7684\u80fd\u529b\u3002 Kube-OVN \u7684 Underlay \u6a21\u5f0f\u76f8\u6bd4 Macvlan \u989d\u5916\u63d0\u4f9b\u4e86\u5730\u5740\u7ba1\u7406\uff0c\u56fa\u5b9a IP \u548c QoS \u7b49\u529f\u80fd\u3002 \u73af\u5883\u8981\u6c42 \u00b6 \u5728 Underlay \u6a21\u5f0f\u4e0b\uff0cOVS \u5c06\u4f1a\u6865\u63a5\u4e00\u4e2a\u8282\u70b9\u7f51\u5361\u5230 OVS \u7f51\u6865\uff0c\u5e76\u5c06\u6570\u636e\u5305\u76f4\u63a5\u901a\u8fc7\u8be5\u8282\u70b9\u7f51\u5361\u5bf9\u5916\u53d1\u9001\uff0cL2/L3 \u5c42\u9762\u7684\u8f6c\u53d1\u80fd\u529b\u9700\u8981\u4f9d\u8d56\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u3002 \u9700\u8981\u9884\u5148\u5728\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u914d\u7f6e\u5bf9\u5e94\u7684\u7f51\u5173\u3001Vlan \u548c\u5b89\u5168\u7b56\u7565\u7b49\u914d\u7f6e\u3002 \u5bf9\u4e8e OpenStack \u7684 VM \u73af\u5883\uff0c\u9700\u8981\u5c06\u5bf9\u5e94\u7f51\u7edc\u7aef\u53e3\u7684 PortSecurity \u5173\u95ed\u3002 \u5bf9\u4e8e VMware \u7684 vSwitch \u7f51\u7edc\uff0c\u9700\u8981\u5c06 MAC Address Changes , Forged Transmits \u548c Promiscuous Mode Operation \u8bbe\u7f6e\u4e3a allow \u3002 \u5bf9\u4e8e Hyper-V \u865a\u62df\u5316\uff0c\u9700\u8981\u5f00\u542f\u865a\u62df\u673a\u7f51\u5361\u9ad8\u7ea7\u529f\u80fd\u4e2d\u7684 MAC Address Spoofing \u3002 \u516c\u6709\u4e91\uff0c\u4f8b\u5982 AWS\u3001GCE\u3001\u963f\u91cc\u4e91\u7b49\u7531\u4e8e\u4e0d\u652f\u6301\u7528\u6237\u81ea\u5b9a\u4e49 Mac \u65e0\u6cd5\u652f\u6301 Underlay \u6a21\u5f0f\u7f51\u7edc\uff0c\u5728\u8fd9\u79cd\u573a\u666f\u4e0b\u5982\u679c\u60f3\u4f7f\u7528 Underlay \u63a8\u8350\u4f7f\u7528\u5bf9\u5e94\u516c\u6709\u4e91\u5382\u5546\u63d0\u4f9b\u7684 VPC-CNI\u3002 \u6865\u63a5\u7f51\u5361\u4e0d\u80fd\u4e3a Linux Bridge\u3002 \u5bf9\u4e8e\u7ba1\u7406\u7f51\u548c\u5bb9\u5668\u7f51\u4f7f\u7528\u540c\u4e00\u4e2a\u7f51\u5361\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u7684 Mac \u5730\u5740\u3001IP \u5730\u5740\u3001\u8def\u7531\u4ee5\u53ca MTU \u5c06\u8f6c\u79fb\u6216\u590d\u5236\u81f3\u5bf9\u5e94\u7684 OVS Bridge\uff0c \u4ee5\u652f\u6301\u5355\u7f51\u5361\u90e8\u7f72 Underlay \u7f51\u7edc\u3002OVS Bridge \u540d\u79f0\u683c\u5f0f\u4e3a br-PROVIDER_NAME \uff0c PROVIDER_NAME \u4e3a Provider \u7f51\u7edc\u540d\u79f0\uff08\u9ed8\u8ba4\u4e3a provider\uff09\u3002 \u90e8\u7f72\u65f6\u6307\u5b9a\u7f51\u7edc\u6a21\u5f0f \u00b6 \u8be5\u90e8\u7f72\u6a21\u5f0f\u5c06\u9ed8\u8ba4\u5b50\u7f51\u8bbe\u7f6e\u4e3a Underlay \u6a21\u5f0f\uff0c\u6240\u6709\u672a\u6307\u5b9a\u5b50\u7f51\u7684 Pod \u5747\u4f1a\u9ed8\u8ba4\u8fd0\u884c\u5728 Underlay \u7f51\u7edc\u4e2d\u3002 \u4e0b\u8f7d\u5b89\u88c5\u811a\u672c \u00b6 wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh \u4fee\u6539\u811a\u672c\u4e2d\u76f8\u5e94\u914d\u7f6e \u00b6 ENABLE_ARP_DETECT_IP_CONFLICT # \u5982\u6709\u9700\u8981\uff0c\u53ef\u4ee5\u9009\u62e9\u5173\u95ed vlan \u7f51\u7edc arp \u51b2\u7a81\u68c0\u6d4b NETWORK_TYPE # \u8bbe\u7f6e\u4e3a vlan VLAN_INTERFACE_NAME # \u8bbe\u7f6e\u4e3a\u5bbf\u4e3b\u673a\u4e0a\u627f\u62c5\u5bb9\u5668\u6d41\u91cf\u7684\u7f51\u5361\uff0c\u4f8b\u5982 eth1 VLAN_ID # \u4ea4\u6362\u673a\u6240\u63a5\u53d7\u7684 VLAN Tag\uff0c\u82e5\u8bbe\u7f6e\u4e3a 0 \u5219\u4e0d\u505a VLAN \u5c01\u88c5 POD_CIDR # \u8bbe\u7f6e\u4e3a\u7269\u7406\u7f51\u7edc CIDR\uff0c \u4f8b\u5982 192.168.1.0/24 POD_GATEWAY # \u8bbe\u7f6e\u4e3a\u7269\u7406\u7f51\u7edc\u7f51\u5173\uff0c\u4f8b\u5982 192.168.1.1 EXCLUDE_IPS # \u6392\u9664\u8303\u56f4\uff0c\u907f\u514d\u5bb9\u5668\u7f51\u6bb5\u548c\u7269\u7406\u7f51\u7edc\u5df2\u7528 IP \u51b2\u7a81\uff0c\u4f8b\u5982 192.168.1.1..192.168.1.100 ENABLE_LB # \u5982\u679c Underlay \u5b50\u7f51\u9700\u8981\u4f7f\u7528 Service \u9700\u8981\u8bbe\u7f6e\u4e3a true EXCHANGE_LINK_NAME # \u662f\u5426\u4ea4\u6362\u9ed8\u8ba4 provider-network \u4e0b OVS \u7f51\u6865\u548c\u6865\u63a5\u7f51\u5361\u7684\u540d\u5b57\uff0c\u9ed8\u8ba4\u4e3a false LS_DNAT_MOD_DL_DST # DNAT \u65f6\u662f\u5426\u5bf9 MAC \u5730\u5740\u8fdb\u884c\u8f6c\u6362\uff0c\u53ef\u52a0\u901f Service \u7684\u8bbf\u95ee\uff0c\u9ed8\u8ba4\u4e3a true \u8fd0\u884c\u5b89\u88c5\u811a\u672c \u00b6 bash install.sh \u901a\u8fc7 CRD \u52a8\u6001\u521b\u5efa Underlay \u7f51\u7edc \u00b6 \u8be5\u65b9\u5f0f\u53ef\u5728\u5b89\u88c5\u540e\u52a8\u6001\u7684\u521b\u5efa\u67d0\u4e2a Underlay \u5b50\u7f51\u4f9b Pod \u4f7f\u7528\u3002\u9700\u8981\u914d\u7f6e ProviderNetwork \uff0c Vlan \u548c Subnet \u4e09\u79cd\u81ea\u5b9a\u4e49\u8d44\u6e90\u3002 \u521b\u5efa ProviderNetwork \u00b6 ProviderNetwork \u63d0\u4f9b\u4e86\u4e3b\u673a\u7f51\u5361\u5230\u7269\u7406\u7f51\u7edc\u6620\u5c04\u7684\u62bd\u8c61\uff0c\u5c06\u540c\u5c5e\u4e00\u4e2a\u7f51\u7edc\u7684\u7f51\u5361\u8fdb\u884c\u7edf\u4e00\u7ba1\u7406\uff0c \u5e76\u89e3\u51b3\u5728\u590d\u6742\u73af\u5883\u4e0b\u540c\u673a\u5668\u591a\u7f51\u5361\u3001\u7f51\u5361\u540d\u4e0d\u4e00\u81f4\u3001\u5bf9\u5e94 Underlay \u7f51\u7edc\u4e0d\u4e00\u81f4\u7b49\u60c5\u51b5\u4e0b\u7684\u914d\u7f6e\u95ee\u9898\u3002 \u521b\u5efa\u5982\u4e0b ProviderNetwork \u5e76\u5e94\u7528: apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: net1 spec: defaultInterface: eth1 customInterfaces: - interface: eth2 nodes: - node1 excludeNodes: - node2 \u6ce8\u610f\uff1aProviderNetwork \u8d44\u6e90\u540d\u79f0\u7684\u957f\u5ea6\u4e0d\u5f97\u8d85\u8fc7 12\u3002 defaultInterface : \u4e3a\u9ed8\u8ba4\u4f7f\u7528\u7684\u8282\u70b9\u7f51\u5361\u540d\u79f0\u3002 ProviderNetwork \u521b\u5efa\u6210\u529f\u540e\uff0c\u5404\u8282\u70b9\uff08\u9664 excludeNodes \u5916\uff09\u4e2d\u4f1a\u521b\u5efa\u540d\u4e3a br-net1\uff08\u683c\u5f0f\u4e3a br-NAME \uff09\u7684 OVS \u7f51\u6865\uff0c\u5e76\u5c06\u6307\u5b9a\u7684\u8282\u70b9\u7f51\u5361\u6865\u63a5\u81f3\u6b64\u7f51\u6865\u3002 customInterfaces : \u4e3a\u53ef\u9009\u9879\uff0c\u53ef\u9488\u5bf9\u7279\u5b9a\u8282\u70b9\u6307\u5b9a\u9700\u8981\u4f7f\u7528\u7684\u7f51\u5361\u3002 excludeNodes : \u53ef\u9009\u9879\uff0c\u7528\u4e8e\u6307\u5b9a\u4e0d\u6865\u63a5\u7f51\u5361\u7684\u8282\u70b9\u3002\u8be5\u5217\u8868\u4e2d\u7684\u8282\u70b9\u4f1a\u88ab\u6dfb\u52a0 net1.provider-network.ovn.kubernetes.io/exclude=true \u6807\u7b7e\u3002 \u5176\u5b83\u8282\u70b9\u4f1a\u88ab\u6dfb\u52a0\u5982\u4e0b\u6807\u7b7e\uff1a Key Value \u63cf\u8ff0 net1.provider-network.ovn.kubernetes.io/ready true \u8282\u70b9\u4e2d\u7684\u6865\u63a5\u5de5\u4f5c\u5df2\u5b8c\u6210\uff0cProviderNetwork \u5728\u8282\u70b9\u4e2d\u53ef\u7528 net1.provider-network.ovn.kubernetes.io/interface eth1 \u8282\u70b9\u4e2d\u88ab\u6865\u63a5\u7684\u7f51\u5361\u7684\u540d\u79f0 net1.provider-network.ovn.kubernetes.io/mtu 1500 \u8282\u70b9\u4e2d\u88ab\u6865\u63a5\u7684\u7f51\u5361\u7684 MTU \u5982\u679c\u8282\u70b9\u7f51\u5361\u4e0a\u5df2\u7ecf\u914d\u7f6e\u4e86 IP\uff0c\u5219 IP \u5730\u5740\u548c\u7f51\u5361\u4e0a\u7684\u8def\u7531\u4f1a\u88ab\u8f6c\u79fb\u81f3\u5bf9\u5e94\u7684 OVS \u7f51\u6865\u3002 \u521b\u5efa VLAN \u00b6 Vlan \u63d0\u4f9b\u4e86\u5c06 Vlan Tag \u548c ProviderNetwork \u8fdb\u884c\u7ed1\u5b9a\u7684\u80fd\u529b\u3002 \u521b\u5efa\u5982\u4e0b VLAN \u5e76\u5e94\u7528\uff1a apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan1 spec: id: 0 provider: net1 id : \u4e3a VLAN ID/Tag\uff0cKube-OVN \u4f1a\u5bf9\u5bf9\u8be5 Vlan \u4e0b\u7684\u6d41\u91cf\u589e\u52a0 Vlan \u6807\u7b7e\uff0c\u4e3a 0 \u65f6\u4e0d\u589e\u52a0\u4efb\u4f55\u6807\u7b7e\u3002 provider : \u4e3a\u9700\u8981\u4f7f\u7528\u7684 ProviderNetwork \u8d44\u6e90\u7684\u540d\u79f0\u3002\u591a\u4e2a VLAN \u53ef\u4ee5\u5f15\u7528\u540c\u4e00\u4e2a ProviderNetwork\u3002 \u521b\u5efa Subnet \u00b6 \u5c06 Vlan \u548c\u4e00\u4e2a\u5b50\u7f51\u7ed1\u5b9a\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 \u5c06 vlan \u7684\u503c\u6307\u5b9a\u4e3a\u9700\u8981\u4f7f\u7528\u7684 VLAN \u540d\u79f0\u5373\u53ef\u3002\u591a\u4e2a Subnet \u53ef\u4ee5\u5f15\u7528\u540c\u4e00\u4e2a VLAN\u3002 \u5bb9\u5668\u521b\u5efa \u00b6 \u53ef\u6309\u6b63\u5e38\u5bb9\u5668\u521b\u5efa\u65b9\u5f0f\u8fdb\u884c\u521b\u5efa\uff0c\u67e5\u770b\u5bb9\u5668 IP \u662f\u5426\u5728\u89c4\u5b9a\u8303\u56f4\u5185\uff0c\u4ee5\u53ca\u5bb9\u5668\u662f\u5426\u53ef\u4ee5\u548c\u7269\u7406\u7f51\u7edc\u4e92\u901a\u3002 \u5982\u6709\u56fa\u5b9a IP \u9700\u6c42\uff0c\u53ef\u53c2\u8003 Pod \u56fa\u5b9a IP \u548c Mac \u4f7f\u7528\u903b\u8f91\u7f51\u5173 \u00b6 \u5bf9\u4e8e\u7269\u7406\u7f51\u7edc\u4e0d\u5b58\u5728\u7f51\u5173\u7684\u60c5\u51b5\uff0cKube-OVN \u652f\u6301\u5728 Underlay \u6a21\u5f0f\u7684\u5b50\u7f51\u4e2d\u914d\u7f6e\u4f7f\u7528\u903b\u8f91\u7f51\u5173\u3002 \u82e5\u8981\u4f7f\u7528\u6b64\u529f\u80fd\uff0c\u8bbe\u7f6e\u5b50\u7f51\u7684 spec.logicalGateway \u4e3a true \u5373\u53ef\uff1a apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 172.17.0.0/16 gateway: 172.17.0.1 vlan: vlan1 logicalGateway: true \u5f00\u542f\u6b64\u529f\u80fd\u540e\uff0cPod \u4e0d\u4f7f\u7528\u5916\u90e8\u7f51\u5173\uff0c\u800c\u662f\u4f7f\u7528 Kube-OVN \u521b\u5efa\u7684\u903b\u8f91\u8def\u7531\u5668\uff08Logical Router\uff09\u5bf9\u4e8e\u8de8\u7f51\u6bb5\u901a\u4fe1\u8fdb\u884c\u8f6c\u53d1\u3002 Underlay \u548c Overlay \u7f51\u7edc\u4e92\u901a \u00b6 \u5982\u679c\u4e00\u4e2a\u96c6\u7fa4\u540c\u65f6\u5b58\u5728 Underlay \u548c Overlay \u5b50\u7f51\uff0c\u9ed8\u8ba4\u60c5\u51b5\u4e0b Overlay \u5b50\u7f51\u4e0b\u7684 Pod \u53ef\u4ee5\u901a\u8fc7\u7f51\u5173\u4ee5 NAT \u7684\u65b9\u5f0f\u8bbf\u95ee Underlay \u5b50\u7f51\u4e0b\u7684 Pod IP\u3002 \u5728 Underlay \u5b50\u7f51\u7684 Pod \u770b\u6765 Overlay \u5b50\u7f51\u7684\u5730\u5740\u662f\u4e00\u4e2a\u5916\u90e8\u7684\u5730\u5740\uff0c\u9700\u8981\u901a\u8fc7\u5e95\u5c42\u7269\u7406\u8bbe\u5907\u53bb\u8f6c\u53d1\uff0c\u4f46\u5e95\u5c42\u7269\u7406\u8bbe\u5907\u5e76\u4e0d\u6e05\u695a Overlay \u5b50\u7f51\u7684\u5730\u5740\u65e0\u6cd5\u8fdb\u884c\u8f6c\u53d1\u3002 \u56e0\u6b64 Underlay \u5b50\u7f51\u4e0b\u7684 Pod \u65e0\u6cd5\u901a\u8fc7 Pod IP \u76f4\u63a5\u8bbf\u95ee Overlay \u5b50\u7f51\u7684 Pod\u3002 \u5982\u679c\u9700\u8981 Underlay \u548c Overlay \u4e92\u901a\u9700\u8981\u5c06\u5b50\u7f51\u7684 u2oInterconnection \u8bbe\u7f6e\u4e3a true \uff0c\u5728\u8fd9\u4e2a\u60c5\u51b5\u4e0b Kube-OVN \u4f1a\u989d\u5916\u4f7f\u7528\u4e00\u4e2a Underlay IP \u5c06 Underlay \u5b50\u7f51 \u548c ovn-cluster \u903b\u8f91\u8def\u7531\u5668\u8fde\u63a5\uff0c\u5e76\u8bbe\u7f6e\u5bf9\u5e94\u7684\u8def\u7531\u89c4\u5219\u5b9e\u73b0\u4e92\u901a\u3002 \u548c\u903b\u8f91\u7f51\u5173\u4e0d\u540c\uff0c\u8be5\u65b9\u6848\u53ea\u4f1a\u8fde\u63a5 Kube-OVN \u5185\u90e8\u7684 Underlay \u548c Overlay \u5b50\u7f51\uff0c\u5176\u4ed6\u8bbf\u95ee\u5916\u7f51\u7684\u6d41\u91cf\u8fd8\u662f\u4f1a\u901a\u8fc7\u7269\u7406\u7f51\u5173\u8fdb\u884c\u8f6c\u53d1\u3002 \u6307\u5b9a\u903b\u8f91\u7f51\u5173 IP \u00b6 \u5f00\u542f\u4e92\u901a\u529f\u80fd\u540e\uff0c\u4f1a\u968f\u673a\u4ece subnet \u5185\u7684\u53d6\u4e00\u4e2a IP \u4f5c\u4e3a\u903b\u8f91\u7f51\u5173\uff0c\u5982\u679c\u9700\u8981\u6307\u5b9a Underlay Subnet \u7684\u903b\u8f91\u7f51\u5173\u53ef\u4ee5\u6307\u5b9a\u5b57\u6bb5 u2oInterconnectionIP \u3002 \u6307\u5b9a Underlay Subnet \u8fde\u63a5\u7684\u81ea\u5b9a\u4e49 VPC \u00b6 \u9ed8\u8ba4\u60c5\u51b5\u4e0b Underlay Subnet \u4f1a\u548c\u9ed8\u8ba4 VPC \u4e0a\u7684 Overlay Subnet \u4e92\u901a\uff0c\u5982\u679c\u8981\u6307\u5b9a\u548c\u67d0\u4e2a VPC \u4e92\u901a\uff0c\u5728 u2oInterconnection \u8bbe\u7f6e\u4e3a true \u540e\uff0c\u6307\u5b9a subnet.spec.vpc \u5b57\u6bb5\u4e3a\u8be5 VPC \u540d\u5b57\u5373\u53ef\u3002 \u6ce8\u610f\u4e8b\u9879 \u00b6 \u5982\u679c\u60a8\u4f7f\u7528\u7684\u8282\u70b9\u7f51\u5361\u4e0a\u914d\u7f6e\u6709 IP \u5730\u5740\uff0c\u4e14\u64cd\u4f5c\u7cfb\u7edf\u662f Ubuntu \u5e76\u901a\u8fc7 Netplan \u914d\u7f6e\u7f51\u7edc\uff0c\u5efa\u8bae\u60a8\u5c06 Netplan \u7684 renderer \u8bbe\u7f6e\u4e3a NetworkManager\uff0c\u5e76\u4e3a\u8282\u70b9\u7f51\u5361\u914d\u7f6e\u9759\u6001 IP \u5730\u5740\uff08\u5173\u95ed DHCP\uff09\uff1a network : renderer : NetworkManager ethernets : eth0 : dhcp4 : no addresses : - 172.16.143.129/24 version : 2 \u82e5\u8282\u70b9\u7f51\u7edc\u7ba1\u7406\u670d\u52a1\u4e3a NetworkManager\uff0c\u5728\u4f7f\u7528\u8282\u70b9\u7f51\u5361\u521b\u5efa ProviderNetwork \u540e\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u4ece NetworkManager \u7ba1\u7406\u5217\u8868\u4e2d\u79fb\u9664\uff08managed \u5c5e\u6027\u4e3a no\uff09\uff1a root@ubuntu:~# nmcli device status DEVICE TYPE STATE CONNECTION eth0 ethernet unmanaged netplan-eth0 \u5982\u679c\u60a8\u8981\u4fee\u6539\u7f51\u5361\u7684 IP \u6216\u8def\u7531\u914d\u7f6e\uff0c\u9700\u8981\u624b\u52a8\u5c06\u7f51\u5361\u91cd\u65b0\u52a0\u5165 NetworkManager \u7ba1\u7406\u5217\u8868\uff1a nmcli device set eth0 managed yes \u6267\u884c\u4ee5\u4e0a\u547d\u4ee4\u540e\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u4e0a\u7684 IP \u53ca\u8def\u7531\u91cd\u65b0\u8f6c\u79fb\u81f3 OVS \u7f51\u6865\uff0c\u5e76\u518d\u6b21\u5c06\u7f51\u5361\u4ece NetworkManager \u7ba1\u7406\u5217\u8868\u4e2d\u79fb\u9664\u3002 \u6ce8\u610f \uff1a\u8282\u70b9\u7f51\u5361\u914d\u7f6e\u7684\u52a8\u6001\u4fee\u6539\u4ec5\u652f\u6301 IP \u548c\u8def\u7531\uff0c\u4e0d\u652f\u6301 MAC \u5730\u5740\u7684\u4fee\u6539\u3002 \u5df2\u77e5\u95ee\u9898 \u00b6 \u7269\u7406\u7f51\u7edc\u5f00\u542f hairpin \u65f6 Pod \u7f51\u7edc\u5f02\u5e38 \u00b6 \u5f53\u7269\u7406\u7f51\u7edc\u5f00\u542f hairpin \u6216\u7c7b\u4f3c\u884c\u4e3a\u65f6\uff0c\u53ef\u80fd\u51fa\u73b0\u521b\u5efa Pod \u65f6\u7f51\u5173\u68c0\u67e5\u5931\u8d25\u3001Pod \u7f51\u7edc\u901a\u4fe1\u5f02\u5e38\u7b49\u95ee\u9898\u3002\u8fd9\u662f\u56e0\u4e3a OVS \u7f51\u6865\u9ed8\u8ba4\u7684 MAC \u5b66\u4e60\u529f\u80fd\u4e0d\u652f\u6301\u8fd9\u79cd\u7f51\u7edc\u73af\u5883\u3002 \u8981\u89e3\u51b3\u6b64\u95ee\u9898\uff0c\u9700\u8981\u5173\u95ed hairpin\uff08\u6216\u4fee\u6539\u7269\u7406\u7f51\u7edc\u7684\u76f8\u5173\u914d\u7f6e\uff09\uff0c\u6216\u66f4\u65b0 Kube-OVN \u7248\u672c\u3002 Pod \u6570\u91cf\u8f83\u591a\u65f6\u65b0\u5efa Pod \u7f51\u5173\u68c0\u67e5\u5931\u8d25 \u00b6 \u82e5\u540c\u4e00\u4e2a\u8282\u70b9\u4e0a\u8fd0\u884c\u7684 Pod \u6570\u91cf\u8f83\u591a\uff08\u5927\u4e8e 300\uff09\uff0c\u53ef\u80fd\u4f1a\u51fa\u73b0 ARP \u5e7f\u64ad\u5305\u7684 OVS \u6d41\u8868 resubmit \u6b21\u6570\u8d85\u8fc7\u4e0a\u9650\u5bfc\u81f4\u4e22\u5305\u7684\u73b0\u8c61\uff1a 2022-11-13T08:43:46.782Z|00222|ofproto_dpif_upcall(handler5)|WARN|Flow: arp,in_port=331,vlan_tci=0x0000,dl_src=00:00:00:25:eb:39,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.131.240,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:25:eb:39,arp_tha=ff:ff:ff:ff:ff:ff bridge(\"br-int\") ---------------- 0. No match. >>>> received packet on unknown port 331 <<<< drop Final flow: unchanged Megaflow: recirc_id=0,eth,arp,in_port=331,dl_src=00:00:00:25:eb:39 Datapath actions: drop 2022-11-13T08:44:34.077Z|00224|ofproto_dpif_xlate(handler5)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,in_port=13483,vlan_tci=0x0000,dl_src=00:00:00:59:ef:13,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.152.3,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:59:ef:13,arp_tha=ff:ff:ff:ff:ff:ff \u8981\u89e3\u51b3\u6b64\u95ee\u9898\uff0c\u53ef\u4fee\u6539 OVN NB \u9009\u9879 bcast_arp_req_flood \u4e3a false \uff1a kubectl ko nbctl set NB_Global . options:bcast_arp_req_flood = false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Underlay \u7f51\u7edc\u5b89\u88c5"},{"location":"start/underlay/#underlay","text":"\u9ed8\u8ba4\u60c5\u51b5\u4e0b Kube-OVN \u7684\u9ed8\u8ba4\u5b50\u7f51\u4f7f\u7528 Geneve \u5bf9\u8de8\u4e3b\u673a\u6d41\u91cf\u8fdb\u884c\u5c01\u88c5\uff0c\u5728\u57fa\u7840\u8bbe\u65bd\u4e4b\u4e0a\u62bd\u8c61\u51fa\u4e00\u5c42\u865a\u62df\u7684 Overlay \u7f51\u7edc\u3002 \u5bf9\u4e8e\u5e0c\u671b\u5bb9\u5668\u7f51\u7edc\u76f4\u63a5\u4f7f\u7528\u7269\u7406\u7f51\u7edc\u5730\u5740\u6bb5\u60c5\u51b5\uff0c\u53ef\u4ee5\u5c06 Kube-OVN \u7684\u9ed8\u8ba4\u5b50\u7f51\u5de5\u4f5c\u5728 Underlay \u6a21\u5f0f\uff0c\u53ef\u4ee5\u76f4\u63a5\u7ed9\u5bb9\u5668\u5206\u914d\u7269\u7406\u7f51\u7edc\u4e2d\u7684\u5730\u5740\u8d44\u6e90\uff0c\u8fbe\u5230\u66f4\u597d\u7684\u6027\u80fd\u4ee5\u53ca\u548c\u7269\u7406\u7f51\u7edc\u7684\u8fde\u901a\u6027\u3002","title":"Underlay \u7f51\u7edc\u5b89\u88c5"},{"location":"start/underlay/#_1","text":"\u7531\u4e8e\u8be5\u6a21\u5f0f\u4e0b\u5bb9\u5668\u7f51\u7edc\u76f4\u63a5\u4f7f\u7528\u7269\u7406\u7f51\u7edc\u8fdb\u884c\u4e8c\u5c42\u5305\u8f6c\u53d1\uff0cOverlay \u6a21\u5f0f\u4e0b\u7684 SNAT/EIP\uff0c \u5206\u5e03\u5f0f\u7f51\u5173/\u96c6\u4e2d\u5f0f\u7f51\u5173\u7b49 L3 \u529f\u80fd\u65e0\u6cd5\u4f7f\u7528\uff0cVPC \u7ea7\u522b\u7684\u9694\u79bb\u4e5f\u65e0\u6cd5\u5bf9 Underlay \u5b50\u7f51\u751f\u6548\u3002","title":"\u529f\u80fd\u9650\u5236"},{"location":"start/underlay/#macvlan","text":"Kube-OVN \u7684 Underlay \u6a21\u5f0f\u548c Macvlan \u5de5\u4f5c\u6a21\u5f0f\u5341\u5206\u7c7b\u4f3c\uff0c\u5728\u529f\u80fd\u548c\u6027\u80fd\u4e0a\u4e3b\u8981\u6709\u4ee5\u4e0b\u51e0\u4e2a\u533a\u522b\uff1a \u7531\u4e8e Macvlan \u7684\u5185\u6838\u8def\u5f84\u66f4\u77ed\uff0c\u5e76\u4e14\u4e0d\u9700\u8981 OVS \u5bf9\u6570\u636e\u5305\u8fdb\u884c\u5904\u7406\uff0cMacvlan \u5728\u541e\u5410\u91cf\u548c\u5ef6\u8fdf\u6027\u80fd\u6307\u6807\u4e0a\u8868\u73b0\u4f1a\u66f4\u597d\u3002 Kube-OVN \u901a\u8fc7\u6d41\u8868\u63d0\u4f9b\u4e86 arp-proxy \u529f\u80fd\uff0c\u53ef\u4ee5\u7f13\u89e3\u5927\u89c4\u6a21\u7f51\u7edc\u4e0b\u7684 arp \u5e7f\u64ad\u98ce\u66b4\u98ce\u9669\u3002 \u7531\u4e8e Macvlan \u5de5\u4f5c\u5728\u5185\u6838\u5e95\u5c42\uff0c\u4f1a\u7ed5\u8fc7\u5bbf\u4e3b\u673a\u7684 netfilter\uff0cService \u548c NetworkPolicy \u529f\u80fd\u9700\u8981\u989d\u5916\u5f00\u53d1\u3002Kube-OVN \u901a\u8fc7 OVS \u6d41\u8868\u63d0\u4f9b\u4e86 Service \u548c NetworkPolicy \u7684\u80fd\u529b\u3002 Kube-OVN \u7684 Underlay \u6a21\u5f0f\u76f8\u6bd4 Macvlan \u989d\u5916\u63d0\u4f9b\u4e86\u5730\u5740\u7ba1\u7406\uff0c\u56fa\u5b9a IP \u548c QoS \u7b49\u529f\u80fd\u3002","title":"\u548c Macvlan \u6bd4\u8f83"},{"location":"start/underlay/#_2","text":"\u5728 Underlay \u6a21\u5f0f\u4e0b\uff0cOVS \u5c06\u4f1a\u6865\u63a5\u4e00\u4e2a\u8282\u70b9\u7f51\u5361\u5230 OVS \u7f51\u6865\uff0c\u5e76\u5c06\u6570\u636e\u5305\u76f4\u63a5\u901a\u8fc7\u8be5\u8282\u70b9\u7f51\u5361\u5bf9\u5916\u53d1\u9001\uff0cL2/L3 \u5c42\u9762\u7684\u8f6c\u53d1\u80fd\u529b\u9700\u8981\u4f9d\u8d56\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u3002 \u9700\u8981\u9884\u5148\u5728\u5e95\u5c42\u7f51\u7edc\u8bbe\u5907\u914d\u7f6e\u5bf9\u5e94\u7684\u7f51\u5173\u3001Vlan \u548c\u5b89\u5168\u7b56\u7565\u7b49\u914d\u7f6e\u3002 \u5bf9\u4e8e OpenStack \u7684 VM \u73af\u5883\uff0c\u9700\u8981\u5c06\u5bf9\u5e94\u7f51\u7edc\u7aef\u53e3\u7684 PortSecurity \u5173\u95ed\u3002 \u5bf9\u4e8e VMware \u7684 vSwitch \u7f51\u7edc\uff0c\u9700\u8981\u5c06 MAC Address Changes , Forged Transmits \u548c Promiscuous Mode Operation \u8bbe\u7f6e\u4e3a allow \u3002 \u5bf9\u4e8e Hyper-V \u865a\u62df\u5316\uff0c\u9700\u8981\u5f00\u542f\u865a\u62df\u673a\u7f51\u5361\u9ad8\u7ea7\u529f\u80fd\u4e2d\u7684 MAC Address Spoofing \u3002 \u516c\u6709\u4e91\uff0c\u4f8b\u5982 AWS\u3001GCE\u3001\u963f\u91cc\u4e91\u7b49\u7531\u4e8e\u4e0d\u652f\u6301\u7528\u6237\u81ea\u5b9a\u4e49 Mac \u65e0\u6cd5\u652f\u6301 Underlay \u6a21\u5f0f\u7f51\u7edc\uff0c\u5728\u8fd9\u79cd\u573a\u666f\u4e0b\u5982\u679c\u60f3\u4f7f\u7528 Underlay \u63a8\u8350\u4f7f\u7528\u5bf9\u5e94\u516c\u6709\u4e91\u5382\u5546\u63d0\u4f9b\u7684 VPC-CNI\u3002 \u6865\u63a5\u7f51\u5361\u4e0d\u80fd\u4e3a Linux Bridge\u3002 \u5bf9\u4e8e\u7ba1\u7406\u7f51\u548c\u5bb9\u5668\u7f51\u4f7f\u7528\u540c\u4e00\u4e2a\u7f51\u5361\u7684\u60c5\u51b5\u4e0b\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u7684 Mac \u5730\u5740\u3001IP \u5730\u5740\u3001\u8def\u7531\u4ee5\u53ca MTU \u5c06\u8f6c\u79fb\u6216\u590d\u5236\u81f3\u5bf9\u5e94\u7684 OVS Bridge\uff0c \u4ee5\u652f\u6301\u5355\u7f51\u5361\u90e8\u7f72 Underlay \u7f51\u7edc\u3002OVS Bridge \u540d\u79f0\u683c\u5f0f\u4e3a br-PROVIDER_NAME \uff0c PROVIDER_NAME \u4e3a Provider \u7f51\u7edc\u540d\u79f0\uff08\u9ed8\u8ba4\u4e3a provider\uff09\u3002","title":"\u73af\u5883\u8981\u6c42"},{"location":"start/underlay/#_3","text":"\u8be5\u90e8\u7f72\u6a21\u5f0f\u5c06\u9ed8\u8ba4\u5b50\u7f51\u8bbe\u7f6e\u4e3a Underlay \u6a21\u5f0f\uff0c\u6240\u6709\u672a\u6307\u5b9a\u5b50\u7f51\u7684 Pod \u5747\u4f1a\u9ed8\u8ba4\u8fd0\u884c\u5728 Underlay \u7f51\u7edc\u4e2d\u3002","title":"\u90e8\u7f72\u65f6\u6307\u5b9a\u7f51\u7edc\u6a21\u5f0f"},{"location":"start/underlay/#_4","text":"wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c"},{"location":"start/underlay/#_5","text":"ENABLE_ARP_DETECT_IP_CONFLICT # \u5982\u6709\u9700\u8981\uff0c\u53ef\u4ee5\u9009\u62e9\u5173\u95ed vlan \u7f51\u7edc arp \u51b2\u7a81\u68c0\u6d4b NETWORK_TYPE # \u8bbe\u7f6e\u4e3a vlan VLAN_INTERFACE_NAME # \u8bbe\u7f6e\u4e3a\u5bbf\u4e3b\u673a\u4e0a\u627f\u62c5\u5bb9\u5668\u6d41\u91cf\u7684\u7f51\u5361\uff0c\u4f8b\u5982 eth1 VLAN_ID # \u4ea4\u6362\u673a\u6240\u63a5\u53d7\u7684 VLAN Tag\uff0c\u82e5\u8bbe\u7f6e\u4e3a 0 \u5219\u4e0d\u505a VLAN \u5c01\u88c5 POD_CIDR # \u8bbe\u7f6e\u4e3a\u7269\u7406\u7f51\u7edc CIDR\uff0c \u4f8b\u5982 192.168.1.0/24 POD_GATEWAY # \u8bbe\u7f6e\u4e3a\u7269\u7406\u7f51\u7edc\u7f51\u5173\uff0c\u4f8b\u5982 192.168.1.1 EXCLUDE_IPS # \u6392\u9664\u8303\u56f4\uff0c\u907f\u514d\u5bb9\u5668\u7f51\u6bb5\u548c\u7269\u7406\u7f51\u7edc\u5df2\u7528 IP \u51b2\u7a81\uff0c\u4f8b\u5982 192.168.1.1..192.168.1.100 ENABLE_LB # \u5982\u679c Underlay \u5b50\u7f51\u9700\u8981\u4f7f\u7528 Service \u9700\u8981\u8bbe\u7f6e\u4e3a true EXCHANGE_LINK_NAME # \u662f\u5426\u4ea4\u6362\u9ed8\u8ba4 provider-network \u4e0b OVS \u7f51\u6865\u548c\u6865\u63a5\u7f51\u5361\u7684\u540d\u5b57\uff0c\u9ed8\u8ba4\u4e3a false LS_DNAT_MOD_DL_DST # DNAT \u65f6\u662f\u5426\u5bf9 MAC \u5730\u5740\u8fdb\u884c\u8f6c\u6362\uff0c\u53ef\u52a0\u901f Service \u7684\u8bbf\u95ee\uff0c\u9ed8\u8ba4\u4e3a true","title":"\u4fee\u6539\u811a\u672c\u4e2d\u76f8\u5e94\u914d\u7f6e"},{"location":"start/underlay/#_6","text":"bash install.sh","title":"\u8fd0\u884c\u5b89\u88c5\u811a\u672c"},{"location":"start/underlay/#crd-underlay","text":"\u8be5\u65b9\u5f0f\u53ef\u5728\u5b89\u88c5\u540e\u52a8\u6001\u7684\u521b\u5efa\u67d0\u4e2a Underlay \u5b50\u7f51\u4f9b Pod \u4f7f\u7528\u3002\u9700\u8981\u914d\u7f6e ProviderNetwork \uff0c Vlan \u548c Subnet \u4e09\u79cd\u81ea\u5b9a\u4e49\u8d44\u6e90\u3002","title":"\u901a\u8fc7 CRD \u52a8\u6001\u521b\u5efa Underlay \u7f51\u7edc"},{"location":"start/underlay/#providernetwork","text":"ProviderNetwork \u63d0\u4f9b\u4e86\u4e3b\u673a\u7f51\u5361\u5230\u7269\u7406\u7f51\u7edc\u6620\u5c04\u7684\u62bd\u8c61\uff0c\u5c06\u540c\u5c5e\u4e00\u4e2a\u7f51\u7edc\u7684\u7f51\u5361\u8fdb\u884c\u7edf\u4e00\u7ba1\u7406\uff0c \u5e76\u89e3\u51b3\u5728\u590d\u6742\u73af\u5883\u4e0b\u540c\u673a\u5668\u591a\u7f51\u5361\u3001\u7f51\u5361\u540d\u4e0d\u4e00\u81f4\u3001\u5bf9\u5e94 Underlay \u7f51\u7edc\u4e0d\u4e00\u81f4\u7b49\u60c5\u51b5\u4e0b\u7684\u914d\u7f6e\u95ee\u9898\u3002 \u521b\u5efa\u5982\u4e0b ProviderNetwork \u5e76\u5e94\u7528: apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: net1 spec: defaultInterface: eth1 customInterfaces: - interface: eth2 nodes: - node1 excludeNodes: - node2 \u6ce8\u610f\uff1aProviderNetwork \u8d44\u6e90\u540d\u79f0\u7684\u957f\u5ea6\u4e0d\u5f97\u8d85\u8fc7 12\u3002 defaultInterface : \u4e3a\u9ed8\u8ba4\u4f7f\u7528\u7684\u8282\u70b9\u7f51\u5361\u540d\u79f0\u3002 ProviderNetwork \u521b\u5efa\u6210\u529f\u540e\uff0c\u5404\u8282\u70b9\uff08\u9664 excludeNodes \u5916\uff09\u4e2d\u4f1a\u521b\u5efa\u540d\u4e3a br-net1\uff08\u683c\u5f0f\u4e3a br-NAME \uff09\u7684 OVS \u7f51\u6865\uff0c\u5e76\u5c06\u6307\u5b9a\u7684\u8282\u70b9\u7f51\u5361\u6865\u63a5\u81f3\u6b64\u7f51\u6865\u3002 customInterfaces : \u4e3a\u53ef\u9009\u9879\uff0c\u53ef\u9488\u5bf9\u7279\u5b9a\u8282\u70b9\u6307\u5b9a\u9700\u8981\u4f7f\u7528\u7684\u7f51\u5361\u3002 excludeNodes : \u53ef\u9009\u9879\uff0c\u7528\u4e8e\u6307\u5b9a\u4e0d\u6865\u63a5\u7f51\u5361\u7684\u8282\u70b9\u3002\u8be5\u5217\u8868\u4e2d\u7684\u8282\u70b9\u4f1a\u88ab\u6dfb\u52a0 net1.provider-network.ovn.kubernetes.io/exclude=true \u6807\u7b7e\u3002 \u5176\u5b83\u8282\u70b9\u4f1a\u88ab\u6dfb\u52a0\u5982\u4e0b\u6807\u7b7e\uff1a Key Value \u63cf\u8ff0 net1.provider-network.ovn.kubernetes.io/ready true \u8282\u70b9\u4e2d\u7684\u6865\u63a5\u5de5\u4f5c\u5df2\u5b8c\u6210\uff0cProviderNetwork \u5728\u8282\u70b9\u4e2d\u53ef\u7528 net1.provider-network.ovn.kubernetes.io/interface eth1 \u8282\u70b9\u4e2d\u88ab\u6865\u63a5\u7684\u7f51\u5361\u7684\u540d\u79f0 net1.provider-network.ovn.kubernetes.io/mtu 1500 \u8282\u70b9\u4e2d\u88ab\u6865\u63a5\u7684\u7f51\u5361\u7684 MTU \u5982\u679c\u8282\u70b9\u7f51\u5361\u4e0a\u5df2\u7ecf\u914d\u7f6e\u4e86 IP\uff0c\u5219 IP \u5730\u5740\u548c\u7f51\u5361\u4e0a\u7684\u8def\u7531\u4f1a\u88ab\u8f6c\u79fb\u81f3\u5bf9\u5e94\u7684 OVS \u7f51\u6865\u3002","title":"\u521b\u5efa ProviderNetwork"},{"location":"start/underlay/#vlan","text":"Vlan \u63d0\u4f9b\u4e86\u5c06 Vlan Tag \u548c ProviderNetwork \u8fdb\u884c\u7ed1\u5b9a\u7684\u80fd\u529b\u3002 \u521b\u5efa\u5982\u4e0b VLAN \u5e76\u5e94\u7528\uff1a apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan1 spec: id: 0 provider: net1 id : \u4e3a VLAN ID/Tag\uff0cKube-OVN \u4f1a\u5bf9\u5bf9\u8be5 Vlan \u4e0b\u7684\u6d41\u91cf\u589e\u52a0 Vlan \u6807\u7b7e\uff0c\u4e3a 0 \u65f6\u4e0d\u589e\u52a0\u4efb\u4f55\u6807\u7b7e\u3002 provider : \u4e3a\u9700\u8981\u4f7f\u7528\u7684 ProviderNetwork \u8d44\u6e90\u7684\u540d\u79f0\u3002\u591a\u4e2a VLAN \u53ef\u4ee5\u5f15\u7528\u540c\u4e00\u4e2a ProviderNetwork\u3002","title":"\u521b\u5efa VLAN"},{"location":"start/underlay/#subnet","text":"\u5c06 Vlan \u548c\u4e00\u4e2a\u5b50\u7f51\u7ed1\u5b9a\uff0c\u5982\u4e0b\u6240\u793a\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 \u5c06 vlan \u7684\u503c\u6307\u5b9a\u4e3a\u9700\u8981\u4f7f\u7528\u7684 VLAN \u540d\u79f0\u5373\u53ef\u3002\u591a\u4e2a Subnet \u53ef\u4ee5\u5f15\u7528\u540c\u4e00\u4e2a VLAN\u3002","title":"\u521b\u5efa Subnet"},{"location":"start/underlay/#_7","text":"\u53ef\u6309\u6b63\u5e38\u5bb9\u5668\u521b\u5efa\u65b9\u5f0f\u8fdb\u884c\u521b\u5efa\uff0c\u67e5\u770b\u5bb9\u5668 IP \u662f\u5426\u5728\u89c4\u5b9a\u8303\u56f4\u5185\uff0c\u4ee5\u53ca\u5bb9\u5668\u662f\u5426\u53ef\u4ee5\u548c\u7269\u7406\u7f51\u7edc\u4e92\u901a\u3002 \u5982\u6709\u56fa\u5b9a IP \u9700\u6c42\uff0c\u53ef\u53c2\u8003 Pod \u56fa\u5b9a IP \u548c Mac","title":"\u5bb9\u5668\u521b\u5efa"},{"location":"start/underlay/#_8","text":"\u5bf9\u4e8e\u7269\u7406\u7f51\u7edc\u4e0d\u5b58\u5728\u7f51\u5173\u7684\u60c5\u51b5\uff0cKube-OVN \u652f\u6301\u5728 Underlay \u6a21\u5f0f\u7684\u5b50\u7f51\u4e2d\u914d\u7f6e\u4f7f\u7528\u903b\u8f91\u7f51\u5173\u3002 \u82e5\u8981\u4f7f\u7528\u6b64\u529f\u80fd\uff0c\u8bbe\u7f6e\u5b50\u7f51\u7684 spec.logicalGateway \u4e3a true \u5373\u53ef\uff1a apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 172.17.0.0/16 gateway: 172.17.0.1 vlan: vlan1 logicalGateway: true \u5f00\u542f\u6b64\u529f\u80fd\u540e\uff0cPod \u4e0d\u4f7f\u7528\u5916\u90e8\u7f51\u5173\uff0c\u800c\u662f\u4f7f\u7528 Kube-OVN \u521b\u5efa\u7684\u903b\u8f91\u8def\u7531\u5668\uff08Logical Router\uff09\u5bf9\u4e8e\u8de8\u7f51\u6bb5\u901a\u4fe1\u8fdb\u884c\u8f6c\u53d1\u3002","title":"\u4f7f\u7528\u903b\u8f91\u7f51\u5173"},{"location":"start/underlay/#underlay-overlay","text":"\u5982\u679c\u4e00\u4e2a\u96c6\u7fa4\u540c\u65f6\u5b58\u5728 Underlay \u548c Overlay \u5b50\u7f51\uff0c\u9ed8\u8ba4\u60c5\u51b5\u4e0b Overlay \u5b50\u7f51\u4e0b\u7684 Pod \u53ef\u4ee5\u901a\u8fc7\u7f51\u5173\u4ee5 NAT \u7684\u65b9\u5f0f\u8bbf\u95ee Underlay \u5b50\u7f51\u4e0b\u7684 Pod IP\u3002 \u5728 Underlay \u5b50\u7f51\u7684 Pod \u770b\u6765 Overlay \u5b50\u7f51\u7684\u5730\u5740\u662f\u4e00\u4e2a\u5916\u90e8\u7684\u5730\u5740\uff0c\u9700\u8981\u901a\u8fc7\u5e95\u5c42\u7269\u7406\u8bbe\u5907\u53bb\u8f6c\u53d1\uff0c\u4f46\u5e95\u5c42\u7269\u7406\u8bbe\u5907\u5e76\u4e0d\u6e05\u695a Overlay \u5b50\u7f51\u7684\u5730\u5740\u65e0\u6cd5\u8fdb\u884c\u8f6c\u53d1\u3002 \u56e0\u6b64 Underlay \u5b50\u7f51\u4e0b\u7684 Pod \u65e0\u6cd5\u901a\u8fc7 Pod IP \u76f4\u63a5\u8bbf\u95ee Overlay \u5b50\u7f51\u7684 Pod\u3002 \u5982\u679c\u9700\u8981 Underlay \u548c Overlay \u4e92\u901a\u9700\u8981\u5c06\u5b50\u7f51\u7684 u2oInterconnection \u8bbe\u7f6e\u4e3a true \uff0c\u5728\u8fd9\u4e2a\u60c5\u51b5\u4e0b Kube-OVN \u4f1a\u989d\u5916\u4f7f\u7528\u4e00\u4e2a Underlay IP \u5c06 Underlay \u5b50\u7f51 \u548c ovn-cluster \u903b\u8f91\u8def\u7531\u5668\u8fde\u63a5\uff0c\u5e76\u8bbe\u7f6e\u5bf9\u5e94\u7684\u8def\u7531\u89c4\u5219\u5b9e\u73b0\u4e92\u901a\u3002 \u548c\u903b\u8f91\u7f51\u5173\u4e0d\u540c\uff0c\u8be5\u65b9\u6848\u53ea\u4f1a\u8fde\u63a5 Kube-OVN \u5185\u90e8\u7684 Underlay \u548c Overlay \u5b50\u7f51\uff0c\u5176\u4ed6\u8bbf\u95ee\u5916\u7f51\u7684\u6d41\u91cf\u8fd8\u662f\u4f1a\u901a\u8fc7\u7269\u7406\u7f51\u5173\u8fdb\u884c\u8f6c\u53d1\u3002","title":"Underlay \u548c Overlay \u7f51\u7edc\u4e92\u901a"},{"location":"start/underlay/#ip","text":"\u5f00\u542f\u4e92\u901a\u529f\u80fd\u540e\uff0c\u4f1a\u968f\u673a\u4ece subnet \u5185\u7684\u53d6\u4e00\u4e2a IP \u4f5c\u4e3a\u903b\u8f91\u7f51\u5173\uff0c\u5982\u679c\u9700\u8981\u6307\u5b9a Underlay Subnet \u7684\u903b\u8f91\u7f51\u5173\u53ef\u4ee5\u6307\u5b9a\u5b57\u6bb5 u2oInterconnectionIP \u3002","title":"\u6307\u5b9a\u903b\u8f91\u7f51\u5173 IP"},{"location":"start/underlay/#underlay-subnet-vpc","text":"\u9ed8\u8ba4\u60c5\u51b5\u4e0b Underlay Subnet \u4f1a\u548c\u9ed8\u8ba4 VPC \u4e0a\u7684 Overlay Subnet \u4e92\u901a\uff0c\u5982\u679c\u8981\u6307\u5b9a\u548c\u67d0\u4e2a VPC \u4e92\u901a\uff0c\u5728 u2oInterconnection \u8bbe\u7f6e\u4e3a true \u540e\uff0c\u6307\u5b9a subnet.spec.vpc \u5b57\u6bb5\u4e3a\u8be5 VPC \u540d\u5b57\u5373\u53ef\u3002","title":"\u6307\u5b9a Underlay Subnet \u8fde\u63a5\u7684\u81ea\u5b9a\u4e49 VPC"},{"location":"start/underlay/#_9","text":"\u5982\u679c\u60a8\u4f7f\u7528\u7684\u8282\u70b9\u7f51\u5361\u4e0a\u914d\u7f6e\u6709 IP \u5730\u5740\uff0c\u4e14\u64cd\u4f5c\u7cfb\u7edf\u662f Ubuntu \u5e76\u901a\u8fc7 Netplan \u914d\u7f6e\u7f51\u7edc\uff0c\u5efa\u8bae\u60a8\u5c06 Netplan \u7684 renderer \u8bbe\u7f6e\u4e3a NetworkManager\uff0c\u5e76\u4e3a\u8282\u70b9\u7f51\u5361\u914d\u7f6e\u9759\u6001 IP \u5730\u5740\uff08\u5173\u95ed DHCP\uff09\uff1a network : renderer : NetworkManager ethernets : eth0 : dhcp4 : no addresses : - 172.16.143.129/24 version : 2 \u82e5\u8282\u70b9\u7f51\u7edc\u7ba1\u7406\u670d\u52a1\u4e3a NetworkManager\uff0c\u5728\u4f7f\u7528\u8282\u70b9\u7f51\u5361\u521b\u5efa ProviderNetwork \u540e\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u4ece NetworkManager \u7ba1\u7406\u5217\u8868\u4e2d\u79fb\u9664\uff08managed \u5c5e\u6027\u4e3a no\uff09\uff1a root@ubuntu:~# nmcli device status DEVICE TYPE STATE CONNECTION eth0 ethernet unmanaged netplan-eth0 \u5982\u679c\u60a8\u8981\u4fee\u6539\u7f51\u5361\u7684 IP \u6216\u8def\u7531\u914d\u7f6e\uff0c\u9700\u8981\u624b\u52a8\u5c06\u7f51\u5361\u91cd\u65b0\u52a0\u5165 NetworkManager \u7ba1\u7406\u5217\u8868\uff1a nmcli device set eth0 managed yes \u6267\u884c\u4ee5\u4e0a\u547d\u4ee4\u540e\uff0cKube-OVN \u4f1a\u5c06\u7f51\u5361\u4e0a\u7684 IP \u53ca\u8def\u7531\u91cd\u65b0\u8f6c\u79fb\u81f3 OVS \u7f51\u6865\uff0c\u5e76\u518d\u6b21\u5c06\u7f51\u5361\u4ece NetworkManager \u7ba1\u7406\u5217\u8868\u4e2d\u79fb\u9664\u3002 \u6ce8\u610f \uff1a\u8282\u70b9\u7f51\u5361\u914d\u7f6e\u7684\u52a8\u6001\u4fee\u6539\u4ec5\u652f\u6301 IP \u548c\u8def\u7531\uff0c\u4e0d\u652f\u6301 MAC \u5730\u5740\u7684\u4fee\u6539\u3002","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"start/underlay/#_10","text":"","title":"\u5df2\u77e5\u95ee\u9898"},{"location":"start/underlay/#hairpin-pod","text":"\u5f53\u7269\u7406\u7f51\u7edc\u5f00\u542f hairpin \u6216\u7c7b\u4f3c\u884c\u4e3a\u65f6\uff0c\u53ef\u80fd\u51fa\u73b0\u521b\u5efa Pod \u65f6\u7f51\u5173\u68c0\u67e5\u5931\u8d25\u3001Pod \u7f51\u7edc\u901a\u4fe1\u5f02\u5e38\u7b49\u95ee\u9898\u3002\u8fd9\u662f\u56e0\u4e3a OVS \u7f51\u6865\u9ed8\u8ba4\u7684 MAC \u5b66\u4e60\u529f\u80fd\u4e0d\u652f\u6301\u8fd9\u79cd\u7f51\u7edc\u73af\u5883\u3002 \u8981\u89e3\u51b3\u6b64\u95ee\u9898\uff0c\u9700\u8981\u5173\u95ed hairpin\uff08\u6216\u4fee\u6539\u7269\u7406\u7f51\u7edc\u7684\u76f8\u5173\u914d\u7f6e\uff09\uff0c\u6216\u66f4\u65b0 Kube-OVN \u7248\u672c\u3002","title":"\u7269\u7406\u7f51\u7edc\u5f00\u542f hairpin \u65f6 Pod \u7f51\u7edc\u5f02\u5e38"},{"location":"start/underlay/#pod-pod","text":"\u82e5\u540c\u4e00\u4e2a\u8282\u70b9\u4e0a\u8fd0\u884c\u7684 Pod \u6570\u91cf\u8f83\u591a\uff08\u5927\u4e8e 300\uff09\uff0c\u53ef\u80fd\u4f1a\u51fa\u73b0 ARP \u5e7f\u64ad\u5305\u7684 OVS \u6d41\u8868 resubmit \u6b21\u6570\u8d85\u8fc7\u4e0a\u9650\u5bfc\u81f4\u4e22\u5305\u7684\u73b0\u8c61\uff1a 2022-11-13T08:43:46.782Z|00222|ofproto_dpif_upcall(handler5)|WARN|Flow: arp,in_port=331,vlan_tci=0x0000,dl_src=00:00:00:25:eb:39,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.131.240,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:25:eb:39,arp_tha=ff:ff:ff:ff:ff:ff bridge(\"br-int\") ---------------- 0. No match. >>>> received packet on unknown port 331 <<<< drop Final flow: unchanged Megaflow: recirc_id=0,eth,arp,in_port=331,dl_src=00:00:00:25:eb:39 Datapath actions: drop 2022-11-13T08:44:34.077Z|00224|ofproto_dpif_xlate(handler5)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,in_port=13483,vlan_tci=0x0000,dl_src=00:00:00:59:ef:13,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.152.3,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:59:ef:13,arp_tha=ff:ff:ff:ff:ff:ff \u8981\u89e3\u51b3\u6b64\u95ee\u9898\uff0c\u53ef\u4fee\u6539 OVN NB \u9009\u9879 bcast_arp_req_flood \u4e3a false \uff1a kubectl ko nbctl set NB_Global . options:bcast_arp_req_flood = false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Pod \u6570\u91cf\u8f83\u591a\u65f6\u65b0\u5efa Pod \u7f51\u5173\u68c0\u67e5\u5931\u8d25"},{"location":"start/uninstall/","text":"\u5378\u8f7d \u00b6 \u5982\u679c\u9700\u8981\u5220\u9664 Kube-OVN \u5e76\u66f4\u6362\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u8bf7\u6309\u7167\u4e0b\u5217\u7684\u6b65\u9aa4\u5220\u9664\u5bf9\u5e94\u7684 Kube-OVN \u7ec4\u4ef6\u4ee5\u53ca OVS \u914d\u7f6e\uff0c\u4ee5\u907f\u514d\u5bf9\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4ea7\u751f\u5e72\u6270\u3002 \u4e5f\u6b22\u8fce\u63d0 issue \u8054\u7cfb\u6211\u4eec\u53cd\u9988\u4e0d\u4f7f\u7528 Kube-OVN \u7684\u539f\u56e0\u5e2e\u52a9\u6211\u4eec\u6539\u8fdb\u3002 \u5220\u9664\u5728 Kubernetes \u4e2d\u521b\u5efa\u7684\u8d44\u6e90 \u00b6 \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u6267\u884c\u811a\u672c\u5220\u9664\u5728 Kubernetes \u4e2d\u521b\u5efa\u7684\u8d44\u6e90\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/cleanup.sh bash cleanup.sh \u6e05\u7406\u4e3b\u673a\u4e0a\u7684\u65e5\u5fd7\u548c\u914d\u7f6e\u6587\u4ef6 \u00b6 \u5728\u6bcf\u53f0\u673a\u5668\u4e0a\u6267\u884c\u4e0b\u5217\u64cd\u4f5c\uff0c\u6e05\u7406 ovsdb \u4ee5\u53ca openvswitch \u4fdd\u5b58\u7684\u914d\u7f6e\uff1a rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn rm -fr /var/log/kube-ovn \u91cd\u542f\u8282\u70b9 \u00b6 \u91cd\u542f\u673a\u5668\u786e\u4fdd\u5bf9\u5e94\u7684\u7f51\u5361\u4fe1\u606f\uff0ciptable/ipset \u89c4\u5219\u5f97\u4ee5\u6e05\u9664\uff0c\u907f\u514d\u5bf9\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u7684\u5f71\u54cd\uff1a reboot \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u5378\u8f7d"},{"location":"start/uninstall/#_1","text":"\u5982\u679c\u9700\u8981\u5220\u9664 Kube-OVN \u5e76\u66f4\u6362\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\uff0c\u8bf7\u6309\u7167\u4e0b\u5217\u7684\u6b65\u9aa4\u5220\u9664\u5bf9\u5e94\u7684 Kube-OVN \u7ec4\u4ef6\u4ee5\u53ca OVS \u914d\u7f6e\uff0c\u4ee5\u907f\u514d\u5bf9\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u4ea7\u751f\u5e72\u6270\u3002 \u4e5f\u6b22\u8fce\u63d0 issue \u8054\u7cfb\u6211\u4eec\u53cd\u9988\u4e0d\u4f7f\u7528 Kube-OVN \u7684\u539f\u56e0\u5e2e\u52a9\u6211\u4eec\u6539\u8fdb\u3002","title":"\u5378\u8f7d"},{"location":"start/uninstall/#kubernetes","text":"\u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u6267\u884c\u811a\u672c\u5220\u9664\u5728 Kubernetes \u4e2d\u521b\u5efa\u7684\u8d44\u6e90\uff1a wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/cleanup.sh bash cleanup.sh","title":"\u5220\u9664\u5728 Kubernetes \u4e2d\u521b\u5efa\u7684\u8d44\u6e90"},{"location":"start/uninstall/#_2","text":"\u5728\u6bcf\u53f0\u673a\u5668\u4e0a\u6267\u884c\u4e0b\u5217\u64cd\u4f5c\uff0c\u6e05\u7406 ovsdb \u4ee5\u53ca openvswitch \u4fdd\u5b58\u7684\u914d\u7f6e\uff1a rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn rm -fr /var/log/kube-ovn","title":"\u6e05\u7406\u4e3b\u673a\u4e0a\u7684\u65e5\u5fd7\u548c\u914d\u7f6e\u6587\u4ef6"},{"location":"start/uninstall/#_3","text":"\u91cd\u542f\u673a\u5668\u786e\u4fdd\u5bf9\u5e94\u7684\u7f51\u5361\u4fe1\u606f\uff0ciptable/ipset \u89c4\u5219\u5f97\u4ee5\u6e05\u9664\uff0c\u907f\u514d\u5bf9\u5176\u4ed6\u7f51\u7edc\u63d2\u4ef6\u7684\u5f71\u54cd\uff1a reboot \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u91cd\u542f\u8282\u70b9"},{"location":"en/","text":"Kube-OVN \u00b6 Kube-OVN, a CNCF Sandbox Project, bridges the SDN into Cloud Native. It offers an advanced Container Network Fabric for Enterprises with the most functions, extreme performance and the easiest operation. Most Functions: If you miss the rich networking capabilities of the SDN age but are struggling to find them in the cloud-native age, Kube-OVN should be your best choice. Leveraging the proven capabilities of OVS/OVN in the SDN, Kube-OVN brings the rich capabilities of network virtualization to the cloud-native space. It currently supports Subnet Management , Static IP Allocation , Distributed/Centralized Gateways , Underlay/Overlay Hybrid Networks , VPC Multi-Tenant Networks , Cross-Cluster Interconnect , QoS Management , Multi-NIC Management , ACL , Traffic Mirroring , ARM Support, Windows Support , and many more. Extreme Performance: If you're concerned about the additional performance loss associated with container networks, then take a look at How Kube-OVN is doing everything it can to optimize performance . In the data plane, through a series of carefully optimized flow and kernel optimizations, and with emerging technologies such as eBPF , DPDK and SmartNIC Offload , Kube-OVN can approximate or exceed host network performance in terms of latency and throughput. In the control plane, Kube-OVN can support large-scale clusters of thousands of nodes and tens of thousands of Pods through the tailoring of OVN upstream flow tables and the use and tuning of various caching techniques. In addition, Kube-OVN is continuously optimizing the usage of resources such as CPU and memory to accommodate resource-limited scenarios such as the edge. Easiest Operation: If you're worried about container network operations, Kube-OVN has a number of built-in tools to help you simplify your operations. Kube-OVN provides one-click installation scripts to help users quickly build production-ready container networks. Also built-in rich monitoring metrics and Grafana dashboard help users to quickly set up monitoring system. Powerful command line tools simplify daily operations and maintenance for users. By combining with Cilium , users can enhance the observability of their networks with eBPF capabilities. In addition, the ability to mirror traffic makes it easy to customize traffic monitoring and interface with traditional NPM systems. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVERVIEW"},{"location":"en/#kube-ovn","text":"Kube-OVN, a CNCF Sandbox Project, bridges the SDN into Cloud Native. It offers an advanced Container Network Fabric for Enterprises with the most functions, extreme performance and the easiest operation. Most Functions: If you miss the rich networking capabilities of the SDN age but are struggling to find them in the cloud-native age, Kube-OVN should be your best choice. Leveraging the proven capabilities of OVS/OVN in the SDN, Kube-OVN brings the rich capabilities of network virtualization to the cloud-native space. It currently supports Subnet Management , Static IP Allocation , Distributed/Centralized Gateways , Underlay/Overlay Hybrid Networks , VPC Multi-Tenant Networks , Cross-Cluster Interconnect , QoS Management , Multi-NIC Management , ACL , Traffic Mirroring , ARM Support, Windows Support , and many more. Extreme Performance: If you're concerned about the additional performance loss associated with container networks, then take a look at How Kube-OVN is doing everything it can to optimize performance . In the data plane, through a series of carefully optimized flow and kernel optimizations, and with emerging technologies such as eBPF , DPDK and SmartNIC Offload , Kube-OVN can approximate or exceed host network performance in terms of latency and throughput. In the control plane, Kube-OVN can support large-scale clusters of thousands of nodes and tens of thousands of Pods through the tailoring of OVN upstream flow tables and the use and tuning of various caching techniques. In addition, Kube-OVN is continuously optimizing the usage of resources such as CPU and memory to accommodate resource-limited scenarios such as the edge. Easiest Operation: If you're worried about container network operations, Kube-OVN has a number of built-in tools to help you simplify your operations. Kube-OVN provides one-click installation scripts to help users quickly build production-ready container networks. Also built-in rich monitoring metrics and Grafana dashboard help users to quickly set up monitoring system. Powerful command line tools simplify daily operations and maintenance for users. By combining with Cilium , users can enhance the observability of their networks with eBPF capabilities. In addition, the ability to mirror traffic makes it easy to customize traffic monitoring and interface with traditional NPM systems. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN"},{"location":"en/contact/","text":"Contact US \u00b6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"CONTACT US"},{"location":"en/contact/#contact-us","text":"\u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Contact US"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/","text":"Accelerate TCP Communication in Node with eBPF \u00b6 At some edge and 5G scenarios, there will be a lot of TCP communication between Pods on the same node. By using the open source istio-tcpip-bypass project from Intel, Pods can use the ability of eBPF to bypass the host's TCP/IP protocol stack and communicate directly through sockets, thereby greatly reducing latency and improving throughput. Basic Principle \u00b6 At present, two Pods on the same host need to go through a lot of network stacks, including TCP/IP, netfilter, OVS, etc., as shown in the following figure: istio-tcpip-bypass plugin can automatically analyze and identify TCP communication within the same host, and bypass the complex kernel stack so that socket data transmission can be performed directly to reduce network stack processing overhead, as shown in the following figure: Due to the fact that this component can automatically identify TCP communication within the same host and optimize it. In the Service Mesh environment based on the proxy mode, this component can also enhance the performance of Service Mesh. For more technical implementation details, please refer to Tanzu Service Mesh Acceleration using eBPF . Prerequisites \u00b6 eBPF requires a kernel version of at least 5.4.0-74-generic. It is recommended to use Ubuntu 20.04 and Linux 5.4.0-74-generic kernel version for testing. Experimental Steps \u00b6 Deploy two performance test Pods on the same node. If there are multiple machines in the cluster, you need to specify nodeSelector : # kubectl create deployment perf --image=kubeovn/perf:dev --replicas=2 deployment.apps/perf created # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES perf-7697bc6ddf-b2cpv 1 /1 Running 0 28s 100 .64.0.3 sealos <none> <none> perf-7697bc6ddf-p2xpt 1 /1 Running 0 28s 100 .64.0.2 sealos <none> <none> Enter one of the Pods to start the qperf server, and start the qperf client in another Pod for performance testing: # kubectl exec -it perf-7697bc6ddf-b2cpv sh / # qperf # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw Deploy the istio-tcpip-bypass plugin: kubectl apply -f https://raw.githubusercontent.com/intel/istio-tcpip-bypass/main/bypass-tcpip-daemonset.yaml Enter the perf client container again for performance testing: # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw Test Results \u00b6 According to the test results, the TCP latency will decrease by 40% ~ 60% under different packet sizes, and the throughput will increase by 40% ~ 80% when the packet size is greater than 1024 bytes. Packet Size (byte) eBPF tcp_lat (us) Default tcp_lat (us) eBPF tcp_bw (Mb/s) Default tcp_bw(Mb/s) 1 20.2 44.5 1.36 4.27 4 20.2 48.7 5.48 16.7 16 19.6 41.6 21.7 63.5 64 18.8 41.3 96.8 201 256 19.2 36 395 539 1024 18.3 42.4 1360 846 4096 16.5 62.6 4460 2430 16384 20.2 58.8 9600 6900 In the hardware environment under test, when the packet size is less than 512 bytes, the throughput indicator optimized by eBPF is lower than the throughput under the default configuration. This situation may be related to the TCP aggregation optimization of the network card under the default configuration. If the application scenario is sensitive to small packet throughput, you need to test in the corresponding environment Determine whether to enable eBPF optimization. We will also optimize the throughput of eBPF TCP small packet scenarios in the future. References \u00b6 istio-tcpip-bypass Deep Dive TCP/IP Bypass with eBPF in Service Mesh Tanzu Service Mesh Acceleration using eBPF \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Accelerate TCP Communication in Node with eBPF"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#accelerate-tcp-communication-in-node-with-ebpf","text":"At some edge and 5G scenarios, there will be a lot of TCP communication between Pods on the same node. By using the open source istio-tcpip-bypass project from Intel, Pods can use the ability of eBPF to bypass the host's TCP/IP protocol stack and communicate directly through sockets, thereby greatly reducing latency and improving throughput.","title":"Accelerate TCP Communication in Node with eBPF"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#basic-principle","text":"At present, two Pods on the same host need to go through a lot of network stacks, including TCP/IP, netfilter, OVS, etc., as shown in the following figure: istio-tcpip-bypass plugin can automatically analyze and identify TCP communication within the same host, and bypass the complex kernel stack so that socket data transmission can be performed directly to reduce network stack processing overhead, as shown in the following figure: Due to the fact that this component can automatically identify TCP communication within the same host and optimize it. In the Service Mesh environment based on the proxy mode, this component can also enhance the performance of Service Mesh. For more technical implementation details, please refer to Tanzu Service Mesh Acceleration using eBPF .","title":"Basic Principle"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#prerequisites","text":"eBPF requires a kernel version of at least 5.4.0-74-generic. It is recommended to use Ubuntu 20.04 and Linux 5.4.0-74-generic kernel version for testing.","title":"Prerequisites"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#experimental-steps","text":"Deploy two performance test Pods on the same node. If there are multiple machines in the cluster, you need to specify nodeSelector : # kubectl create deployment perf --image=kubeovn/perf:dev --replicas=2 deployment.apps/perf created # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES perf-7697bc6ddf-b2cpv 1 /1 Running 0 28s 100 .64.0.3 sealos <none> <none> perf-7697bc6ddf-p2xpt 1 /1 Running 0 28s 100 .64.0.2 sealos <none> <none> Enter one of the Pods to start the qperf server, and start the qperf client in another Pod for performance testing: # kubectl exec -it perf-7697bc6ddf-b2cpv sh / # qperf # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw Deploy the istio-tcpip-bypass plugin: kubectl apply -f https://raw.githubusercontent.com/intel/istio-tcpip-bypass/main/bypass-tcpip-daemonset.yaml Enter the perf client container again for performance testing: # kubectl exec -it perf-7697bc6ddf-p2xpt sh / # qperf -t 60 100.64.0.3 -ub -oo msg_size:1:16K:*4 -vu tcp_lat tcp_bw","title":"Experimental Steps"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#test-results","text":"According to the test results, the TCP latency will decrease by 40% ~ 60% under different packet sizes, and the throughput will increase by 40% ~ 80% when the packet size is greater than 1024 bytes. Packet Size (byte) eBPF tcp_lat (us) Default tcp_lat (us) eBPF tcp_bw (Mb/s) Default tcp_bw(Mb/s) 1 20.2 44.5 1.36 4.27 4 20.2 48.7 5.48 16.7 16 19.6 41.6 21.7 63.5 64 18.8 41.3 96.8 201 256 19.2 36 395 539 1024 18.3 42.4 1360 846 4096 16.5 62.6 4460 2430 16384 20.2 58.8 9600 6900 In the hardware environment under test, when the packet size is less than 512 bytes, the throughput indicator optimized by eBPF is lower than the throughput under the default configuration. This situation may be related to the TCP aggregation optimization of the network card under the default configuration. If the application scenario is sensitive to small packet throughput, you need to test in the corresponding environment Determine whether to enable eBPF optimization. We will also optimize the throughput of eBPF TCP small packet scenarios in the future.","title":"Test Results"},{"location":"en/advance/accelerate-intra-node-tcp-with-ebpf/#references","text":"istio-tcpip-bypass Deep Dive TCP/IP Bypass with eBPF in Service Mesh Tanzu Service Mesh Acceleration using eBPF \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"References"},{"location":"en/advance/cilium-hubble-observe/","text":"Cilium Network Traffic Observation \u00b6 Kube-OVN supports Cilium integration, please refer to Cilium integration for details. Cilium provides rich network traffic observation capabilities, and the flow observability is provided by Hubble. Hubble can observe the traffic across nodes, clusters, and even multi-cluster scenarios. Install Hubble \u00b6 In the default Cilium integration installation, the Hubble related components are not installed, so to support traffic observation, you need to supplement the installation of Hubble on the environment. Execute the following command to install Hubble using helm: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true After installing Hubble, execute cilium status to check the status of the component and confirm that the installation is successful. # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: OK \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ Deployment hubble-relay Desired: 1 , Ready: 1 /1, Available: 1 /1 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment hubble-ui Desired: 1 , Ready: 1 /1, Available: 1 /1 Containers: cilium Running: 2 hubble-ui Running: 1 hubble-relay Running: 1 cilium-operator Running: 2 Cluster Pods: 16 /17 managed by Cilium Image versions hubble-relay quay.io/cilium/hubble-relay:v1.11.6@sha256:fd9034a2d04d5b973f1e8ed44f230ea195b89c37955ff32e34e5aa68f3ed675a: 1 cilium-operator quay.io/cilium/operator-generic:v1.11.6@sha256:9f6063c7bcaede801a39315ec7c166309f6a6783e98665f6693939cf1701bc17: 2 cilium quay.io/cilium/cilium:v1.11.6@sha256:f7f93c26739b6641a3fa3d76b1e1605b15989f25d06625260099e01c8243f54c: 2 hubble-ui quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1 apple@bogon cilium % After installing the Hubble component, you need to install the command line to view the traffic information in the environment. Execute the following command to install Hubble CLI: curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/v0.10.0/hubble-linux-amd64.tar.gz sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin Deploy and test \u00b6 Cilium offers a traffic test deployment solution, you can directly use the official deployment solution to deploy the test. Execute the command cilium connectivity test , Cilium will automatically create the cilium-test namespace, and deploy the test under cilium-test. After the normal deployment, you can view the resource information under the cilium-test namespace, as follows: # kubectl get all -n cilium-test NAME READY STATUS RESTARTS AGE pod/client-7df6cfbf7b-z5t2j 1 /1 Running 0 21s pod/client2-547996d7d8-nvgxg 1 /1 Running 0 21s pod/echo-other-node-d79544ccf-hl4gg 2 /2 Running 0 21s pod/echo-same-node-5d466d5444-ml7tc 2 /2 Running 0 21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE service/echo-other-node NodePort 10 .109.58.126 <none> 8080 :32269/TCP 21s service/echo-same-node NodePort 10 .108.70.32 <none> 8080 :32490/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/client 1 /1 1 1 21s deployment.apps/client2 1 /1 1 1 21s deployment.apps/echo-other-node 1 /1 1 1 21s deployment.apps/echo-same-node 1 /1 1 1 21s NAME DESIRED CURRENT READY AGE replicaset.apps/client-7df6cfbf7b 1 1 1 21s replicaset.apps/client2-547996d7d8 1 1 1 21s replicaset.apps/echo-other-node-d79544ccf 1 1 1 21s replicaset.apps/echo-same-node-5d466d5444 1 1 1 21s Use the command line to observe traffic \u00b6 By default, the network traffic observation only provides the traffic observed by the Cilium agent on each node. Execute the hubble observe command in the Cilium agent pod under the kube-system namespace to view the traffic information on the node. # kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cilium-d6h56 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-c7sb2 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-wj8gt 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> cilium-tq5xb 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> kube-ovn-pinger-7lgk8 1 /1 Running 0 21h 10 .16.0.19 kube-ovn-control-plane <none> <none> kube-ovn-pinger-msvcn 1 /1 Running 0 21h 10 .16.0.18 kube-ovn-worker <none> <none> # kubectl exec -it -n kube-system cilium-d6h56 -- bash root@kube-ovn-worker:/home/cilium# hubble observe --from-namespace kube-system Jul 29 03 :24:25.551: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: SYN ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.651: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.761: kube-system/kube-ovn-pinger-msvcn:52004 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn -> kube-system/kube-ovn-pinger-7lgk8 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn <- kube-system/kube-ovn-pinger-7lgk8 to-endpoint FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:80 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 -> kube-system/kube-ovn-pinger-msvcn to-endpoint FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 <- kube-system/kube-ovn-pinger-msvcn to-stack FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.979: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.3 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:26.037: kube-system/coredns-6d4b75cb6d-lbgjg:36430 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:26.282: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.2 to-stack FORWARDED ( ICMPv4 EchoRequest ) After deploying Hubble Relay, Hubble can provide complete cluster-wide network traffic observation. Configure port forwarding \u00b6 In order to access the Hubble API normally, you need to create a port forwarding to forward the local request to the Hubble Service. You can execute the kubectl port-forward deployment/hubble-relay -n kube-system 4245:4245 command to open the port forwarding in the current terminal. The port forwarding configuration can refer to Port Forwarding . kubectl port-forward is a blocking command, you can open a new terminal to execute the following command to observe the traffic information. After configuring the port forwarding, execute the hubble status command in the terminal. If there is an output similar to the following, the port forwarding configuration is correct, and you can use the command line to observe the traffic. # hubble status Healthcheck ( via localhost:4245 ) : Ok Current/Max Flows: 8 ,190/8,190 ( 100 .00% ) Flows/s: 22 .86 Connected Nodes: 2 /2 Use the command line to observe traffic \u00b6 Execute the hubble observe command in the terminal to view the traffic information of the cluster. The traffic observed by the cilium-test namespace is as follows: Pay attention to the hubble observe command display result, which is the traffic information queried when the current command line is executed. Executing the command line multiple times can view different traffic information. For more detailed observation information, you can execute the hubble help observe command to view the detailed usage of Hubble CLI. Use UI to observe traffic \u00b6 Execute the cilium status command to confirm that the Hubble UI has been successfully installed. In the second step of the Hubble installation, the installation of the UI has been supplemented. Execute the command cilium hubble ui to automatically create port forwarding and map the hubble-ui service to the local port. When the command is executed normally, the local browser will be automatically opened and jump to the Hubble UI interface. If it does not jump automatically, enter http://localhost:12000 in the browser to open the UI observation interface. On the top left of the UI, select the cilium-test namespace to view the test traffic information provided by Cilium. Hubble Traffic Monitoring \u00b6 Hubble component provides monitoring of Pod network behavior in the cluster. In order to support viewing the monitoring data provided by Hubble, you need to enable monitoring statistics. Refer to the following command to supplement the hubble.metrics.enabled configuration item: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \\ --set hubble.metrics.enabled = \"{dns,drop,tcp,flow,icmp,http}\" After the deployment is completed, you can view the monitoring data provided by Hubble through the hubble-metrics service. Execute the following command to view the monitoring data: # curl 172.18.0.2:9091/metrics # HELP hubble_drop_total Number of drops # TYPE hubble_drop_total counter hubble_drop_total { protocol = \"ICMPv6\" ,reason = \"Unsupported L3 protocol\" } 2 # HELP hubble_flows_processed_total Total number of flows processed # TYPE hubble_flows_processed_total counter hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv6\" ,subtype = \"\" ,type = \"Drop\" ,verdict = \"DROPPED\" } 2 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 8282 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 6767 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 # HELP hubble_icmp_total Number of ICMP messages # TYPE hubble_icmp_total counter hubble_icmp_total { family = \"IPv4\" ,type = \"EchoReply\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"EchoRequest\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"RouterSolicitation\" } 2 # HELP hubble_tcp_flags_total TCP flag occurrences # TYPE hubble_tcp_flags_total counter hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"FIN\" } 2043 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"RST\" } 301 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN\" } 1169 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN-ACK\" } 1169 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium Network Traffic Observation"},{"location":"en/advance/cilium-hubble-observe/#cilium-network-traffic-observation","text":"Kube-OVN supports Cilium integration, please refer to Cilium integration for details. Cilium provides rich network traffic observation capabilities, and the flow observability is provided by Hubble. Hubble can observe the traffic across nodes, clusters, and even multi-cluster scenarios.","title":"Cilium Network Traffic Observation"},{"location":"en/advance/cilium-hubble-observe/#install-hubble","text":"In the default Cilium integration installation, the Hubble related components are not installed, so to support traffic observation, you need to supplement the installation of Hubble on the environment. Execute the following command to install Hubble using helm: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true After installing Hubble, execute cilium status to check the status of the component and confirm that the installation is successful. # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: OK \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ Deployment hubble-relay Desired: 1 , Ready: 1 /1, Available: 1 /1 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment hubble-ui Desired: 1 , Ready: 1 /1, Available: 1 /1 Containers: cilium Running: 2 hubble-ui Running: 1 hubble-relay Running: 1 cilium-operator Running: 2 Cluster Pods: 16 /17 managed by Cilium Image versions hubble-relay quay.io/cilium/hubble-relay:v1.11.6@sha256:fd9034a2d04d5b973f1e8ed44f230ea195b89c37955ff32e34e5aa68f3ed675a: 1 cilium-operator quay.io/cilium/operator-generic:v1.11.6@sha256:9f6063c7bcaede801a39315ec7c166309f6a6783e98665f6693939cf1701bc17: 2 cilium quay.io/cilium/cilium:v1.11.6@sha256:f7f93c26739b6641a3fa3d76b1e1605b15989f25d06625260099e01c8243f54c: 2 hubble-ui quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1 hubble-ui quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1 apple@bogon cilium % After installing the Hubble component, you need to install the command line to view the traffic information in the environment. Execute the following command to install Hubble CLI: curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/v0.10.0/hubble-linux-amd64.tar.gz sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin","title":"Install Hubble"},{"location":"en/advance/cilium-hubble-observe/#deploy-and-test","text":"Cilium offers a traffic test deployment solution, you can directly use the official deployment solution to deploy the test. Execute the command cilium connectivity test , Cilium will automatically create the cilium-test namespace, and deploy the test under cilium-test. After the normal deployment, you can view the resource information under the cilium-test namespace, as follows: # kubectl get all -n cilium-test NAME READY STATUS RESTARTS AGE pod/client-7df6cfbf7b-z5t2j 1 /1 Running 0 21s pod/client2-547996d7d8-nvgxg 1 /1 Running 0 21s pod/echo-other-node-d79544ccf-hl4gg 2 /2 Running 0 21s pod/echo-same-node-5d466d5444-ml7tc 2 /2 Running 0 21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE service/echo-other-node NodePort 10 .109.58.126 <none> 8080 :32269/TCP 21s service/echo-same-node NodePort 10 .108.70.32 <none> 8080 :32490/TCP 21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/client 1 /1 1 1 21s deployment.apps/client2 1 /1 1 1 21s deployment.apps/echo-other-node 1 /1 1 1 21s deployment.apps/echo-same-node 1 /1 1 1 21s NAME DESIRED CURRENT READY AGE replicaset.apps/client-7df6cfbf7b 1 1 1 21s replicaset.apps/client2-547996d7d8 1 1 1 21s replicaset.apps/echo-other-node-d79544ccf 1 1 1 21s replicaset.apps/echo-same-node-5d466d5444 1 1 1 21s","title":"Deploy and test"},{"location":"en/advance/cilium-hubble-observe/#use-the-command-line-to-observe-traffic","text":"By default, the network traffic observation only provides the traffic observed by the Cilium agent on each node. Execute the hubble observe command in the Cilium agent pod under the kube-system namespace to view the traffic information on the node. # kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cilium-d6h56 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-c7sb2 1 /1 Running 0 2d20h 172 .18.0.2 kube-ovn-worker <none> <none> cilium-operator-5887f78bbb-wj8gt 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> cilium-tq5xb 1 /1 Running 0 2d20h 172 .18.0.3 kube-ovn-control-plane <none> <none> kube-ovn-pinger-7lgk8 1 /1 Running 0 21h 10 .16.0.19 kube-ovn-control-plane <none> <none> kube-ovn-pinger-msvcn 1 /1 Running 0 21h 10 .16.0.18 kube-ovn-worker <none> <none> # kubectl exec -it -n kube-system cilium-d6h56 -- bash root@kube-ovn-worker:/home/cilium# hubble observe --from-namespace kube-system Jul 29 03 :24:25.551: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.561: kube-system/kube-ovn-pinger-msvcn:35576 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: SYN ) Jul 29 03 :24:25.572: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.651: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: RST ) Jul 29 03 :24:25.661: kube-system/kube-ovn-pinger-msvcn:35578 -> 172 .18.0.3:6642 to-stack FORWARDED ( TCP Flags: ACK, RST ) Jul 29 03 :24:25.761: kube-system/kube-ovn-pinger-msvcn:52004 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK, PSH ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn -> kube-system/kube-ovn-pinger-7lgk8 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.779: kube-system/kube-ovn-pinger-msvcn <- kube-system/kube-ovn-pinger-7lgk8 to-endpoint FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 <- kube-system/hubble-relay-959988db5-zc5vv:80 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.866: kube-system/hubble-ui-7596f7ff6f-7j6f2:55836 -> kube-system/hubble-relay-959988db5-zc5vv:4245 to-endpoint FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 -> kube-system/kube-ovn-pinger-msvcn to-endpoint FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:25.975: kube-system/kube-ovn-pinger-7lgk8 <- kube-system/kube-ovn-pinger-msvcn to-stack FORWARDED ( ICMPv4 EchoReply ) Jul 29 03 :24:25.979: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.3 to-stack FORWARDED ( ICMPv4 EchoRequest ) Jul 29 03 :24:26.037: kube-system/coredns-6d4b75cb6d-lbgjg:36430 -> 172 .18.0.3:6443 to-stack FORWARDED ( TCP Flags: ACK ) Jul 29 03 :24:26.282: kube-system/kube-ovn-pinger-msvcn -> 172 .18.0.2 to-stack FORWARDED ( ICMPv4 EchoRequest ) After deploying Hubble Relay, Hubble can provide complete cluster-wide network traffic observation.","title":"Use the command line to observe traffic"},{"location":"en/advance/cilium-hubble-observe/#configure-port-forwarding","text":"In order to access the Hubble API normally, you need to create a port forwarding to forward the local request to the Hubble Service. You can execute the kubectl port-forward deployment/hubble-relay -n kube-system 4245:4245 command to open the port forwarding in the current terminal. The port forwarding configuration can refer to Port Forwarding . kubectl port-forward is a blocking command, you can open a new terminal to execute the following command to observe the traffic information. After configuring the port forwarding, execute the hubble status command in the terminal. If there is an output similar to the following, the port forwarding configuration is correct, and you can use the command line to observe the traffic. # hubble status Healthcheck ( via localhost:4245 ) : Ok Current/Max Flows: 8 ,190/8,190 ( 100 .00% ) Flows/s: 22 .86 Connected Nodes: 2 /2","title":"Configure port forwarding"},{"location":"en/advance/cilium-hubble-observe/#use-the-command-line-to-observe-traffic_1","text":"Execute the hubble observe command in the terminal to view the traffic information of the cluster. The traffic observed by the cilium-test namespace is as follows: Pay attention to the hubble observe command display result, which is the traffic information queried when the current command line is executed. Executing the command line multiple times can view different traffic information. For more detailed observation information, you can execute the hubble help observe command to view the detailed usage of Hubble CLI.","title":"Use the command line to observe traffic"},{"location":"en/advance/cilium-hubble-observe/#use-ui-to-observe-traffic","text":"Execute the cilium status command to confirm that the Hubble UI has been successfully installed. In the second step of the Hubble installation, the installation of the UI has been supplemented. Execute the command cilium hubble ui to automatically create port forwarding and map the hubble-ui service to the local port. When the command is executed normally, the local browser will be automatically opened and jump to the Hubble UI interface. If it does not jump automatically, enter http://localhost:12000 in the browser to open the UI observation interface. On the top left of the UI, select the cilium-test namespace to view the test traffic information provided by Cilium.","title":"Use UI to observe traffic"},{"location":"en/advance/cilium-hubble-observe/#hubble-traffic-monitoring","text":"Hubble component provides monitoring of Pod network behavior in the cluster. In order to support viewing the monitoring data provided by Hubble, you need to enable monitoring statistics. Refer to the following command to supplement the hubble.metrics.enabled configuration item: helm upgrade cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --reuse-values \\ --set hubble.relay.enabled = true \\ --set hubble.ui.enabled = true \\ --set hubble.metrics.enabled = \"{dns,drop,tcp,flow,icmp,http}\" After the deployment is completed, you can view the monitoring data provided by Hubble through the hubble-metrics service. Execute the following command to view the monitoring data: # curl 172.18.0.2:9091/metrics # HELP hubble_drop_total Number of drops # TYPE hubble_drop_total counter hubble_drop_total { protocol = \"ICMPv6\" ,reason = \"Unsupported L3 protocol\" } 2 # HELP hubble_flows_processed_total Total number of flows processed # TYPE hubble_flows_processed_total counter hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv4\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 335 hubble_flows_processed_total { protocol = \"ICMPv6\" ,subtype = \"\" ,type = \"Drop\" ,verdict = \"DROPPED\" } 2 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 8282 hubble_flows_processed_total { protocol = \"TCP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 6767 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-endpoint\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 hubble_flows_processed_total { protocol = \"UDP\" ,subtype = \"to-stack\" ,type = \"Trace\" ,verdict = \"FORWARDED\" } 1642 # HELP hubble_icmp_total Number of ICMP messages # TYPE hubble_icmp_total counter hubble_icmp_total { family = \"IPv4\" ,type = \"EchoReply\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"EchoRequest\" } 335 hubble_icmp_total { family = \"IPv4\" ,type = \"RouterSolicitation\" } 2 # HELP hubble_tcp_flags_total TCP flag occurrences # TYPE hubble_tcp_flags_total counter hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"FIN\" } 2043 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"RST\" } 301 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN\" } 1169 hubble_tcp_flags_total { family = \"IPv4\" ,flag = \"SYN-ACK\" } 1169 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Hubble Traffic Monitoring"},{"location":"en/advance/cilium-networkpolicy/","text":"Cilium NetworkPolicy Support \u00b6 Kube-OVN currently supports integration with Cilium, and the specific operation can refer to Cilium integration . After integrating Cilium, you can use Cilium's excellent network policy capabilities to control the access of Pods in the cluster.The following documents provide integration verification of Cilium L3 and L4 network policy capabilities. Verification Steps \u00b6 Create test Pod \u00b6 Create namespace test . Refer to the following yaml, create Pod with label app=test in namespace test as the destination Pod for testing access. apiVersion : apps/v1 kind : Deployment metadata : labels : app : test name : test namespace : test spec : replicas : 1 selector : matchLabels : app : test strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : test spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx Similarly, refer to the following yaml, create Pod with label app=dynamic in namespace default as the Pod for testing access. apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx View the test Pod and Label information: # kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-9v5c4 1 /1 Running 0 28h 10 .16.0.35 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-s8z2n 1 /1 Running 0 28h 10 .16.0.36 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 # kubectl get pod -o wide -n test --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-6dsg6 1 /1 Running 0 7h20m 10 .16.0.2 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-tjgtp 1 /1 Running 0 7h46m 10 .16.0.42 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 label-test1-77b6764857-swq4k 1 /1 Running 0 3h43m 10 .16.0.12 kube-ovn-worker <none> <none> app = test1,pod-template-hash = 77b6764857 // As the destination Pod for testing access. test-54c98bc466-mft5s 1 /1 Running 0 8h 10 .16.0.41 kube-ovn-worker <none> <none> app = test,pod-template-hash = 54c98bc466 L3 Network Policy Test \u00b6 Refer to the following yaml, create CiliumNetworkPolicy resource: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l3-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic At this point, the test Pod in the default namespace cannot access the destination Pod, but the test Pod to the destination Pod in the test namespace is accessible. Test results in the default namespace: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss Test results in the test namespace: # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .558 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .223 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .304 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .223/1.028/2.558 ms Look at the Cilium official document explanation, the CiliumNetworkPolicy resource limits the control at the namespace level. For more information, please refer to Cilium Limitations . If there is a network policy rule match, only the Pod in the same namespace can access according to the rule, and the Pod in the other namespace is denied access by default. If you want to implement cross-namespace access, you need to specify the namespace information in the rule. Refer to the document, modify the CiliumNetworkPolicy resource, and add namespace information: ingress : - fromEndpoints : - matchLabels : app : dynamic k8s:io.kubernetes.pod.namespace : default // control the Pod access in other namespace Look at the modified CiliumNetworkPolicy resource information: # kubectl get cnp -n test -o yaml l3-rule apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: l3-rule namespace: test spec: endpointSelector: matchLabels: app: test ingress: - fromEndpoints: - matchLabels: app: dynamic - matchLabels: app: dynamic k8s:io.kubernetes.pod.namespace: default Test the Pod access in the default namespace again, and the destination Pod access is normal: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -n test -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .383 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .115 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .142 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .115/0.880/2.383 ms Using the standard Kubernetes network policy networkpolicy , the test results show that Cilium also restricts access within the same namespace, and cross-namespace access is prohibited. It is different from Kube-OVN implementation. Kube-OVN supports standard k8s network policy, which restricts the destination Pod in a specific namespace, but there is no namespace restriction on the source Pod. Any Pod that meets the restriction rules in any namespace can access the destination Pod. L4 Network Policy Test \u00b6 Refer to the following yaml, create CiliumNetworkPolicy resource: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l4-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic toPorts : - ports : - port : \"80\" protocol : TCP Test the access of the Pod that meets the network policy rules in the same namespace # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl 10 .16.0.41:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on test-54c98bc466-mft5s</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> The Pod that does not meet the network policy rules in the same namespace cannot access # kubectl exec -it -n test label-test1-77b6764857-swq4k -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl -v 10 .16.0.41:80 --connect-timeout 10 * Trying 10 .16.0.41:80... * After 10000ms connect time, move on! * connect to 10 .16.0.41 port 80 failed: Operation timed out * Connection timeout after 10001 ms * Closing connection 0 curl: ( 28 ) Connection timeout after 10001 ms After the network policy takes effect, cross-namespace access is still prohibited, which is consistent with the L3 network policy test results. After the L4 network policy takes effect, ping cannot be used, but TCP access that meets the policy rules can be executed normally. About the restriction of ICMP, please refer to the official description L4 Limitation Description . L7 Network Policy Test \u00b6 chaining mode, L7 network policy currently has problems. In the Cilium official document, there is an explanation for this situation, please refer to Generic Veth Chaining . This problem is tracked using issue 12454 , and it has not been resolved yet. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cilium NetworkPolicy Support"},{"location":"en/advance/cilium-networkpolicy/#cilium-networkpolicy-support","text":"Kube-OVN currently supports integration with Cilium, and the specific operation can refer to Cilium integration . After integrating Cilium, you can use Cilium's excellent network policy capabilities to control the access of Pods in the cluster.The following documents provide integration verification of Cilium L3 and L4 network policy capabilities.","title":"Cilium NetworkPolicy Support"},{"location":"en/advance/cilium-networkpolicy/#verification-steps","text":"","title":"Verification Steps"},{"location":"en/advance/cilium-networkpolicy/#create-test-pod","text":"Create namespace test . Refer to the following yaml, create Pod with label app=test in namespace test as the destination Pod for testing access. apiVersion : apps/v1 kind : Deployment metadata : labels : app : test name : test namespace : test spec : replicas : 1 selector : matchLabels : app : test strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : test spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx Similarly, refer to the following yaml, create Pod with label app=dynamic in namespace default as the Pod for testing access. apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx View the test Pod and Label information: # kubectl get pod -o wide --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-9v5c4 1 /1 Running 0 28h 10 .16.0.35 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-s8z2n 1 /1 Running 0 28h 10 .16.0.36 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 # kubectl get pod -o wide -n test --show-labels NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS dynamic-7d8d7874f5-6dsg6 1 /1 Running 0 7h20m 10 .16.0.2 kube-ovn-control-plane <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 dynamic-7d8d7874f5-tjgtp 1 /1 Running 0 7h46m 10 .16.0.42 kube-ovn-worker <none> <none> app = dynamic,pod-template-hash = 7d8d7874f5 label-test1-77b6764857-swq4k 1 /1 Running 0 3h43m 10 .16.0.12 kube-ovn-worker <none> <none> app = test1,pod-template-hash = 77b6764857 // As the destination Pod for testing access. test-54c98bc466-mft5s 1 /1 Running 0 8h 10 .16.0.41 kube-ovn-worker <none> <none> app = test,pod-template-hash = 54c98bc466","title":"Create test Pod"},{"location":"en/advance/cilium-networkpolicy/#l3-network-policy-test","text":"Refer to the following yaml, create CiliumNetworkPolicy resource: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l3-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic At this point, the test Pod in the default namespace cannot access the destination Pod, but the test Pod to the destination Pod in the test namespace is accessible. Test results in the default namespace: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss Test results in the test namespace: # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .558 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .223 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .304 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .223/1.028/2.558 ms Look at the Cilium official document explanation, the CiliumNetworkPolicy resource limits the control at the namespace level. For more information, please refer to Cilium Limitations . If there is a network policy rule match, only the Pod in the same namespace can access according to the rule, and the Pod in the other namespace is denied access by default. If you want to implement cross-namespace access, you need to specify the namespace information in the rule. Refer to the document, modify the CiliumNetworkPolicy resource, and add namespace information: ingress : - fromEndpoints : - matchLabels : app : dynamic k8s:io.kubernetes.pod.namespace : default // control the Pod access in other namespace Look at the modified CiliumNetworkPolicy resource information: # kubectl get cnp -n test -o yaml l3-rule apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: l3-rule namespace: test spec: endpointSelector: matchLabels: app: test ingress: - fromEndpoints: - matchLabels: app: dynamic - matchLabels: app: dynamic k8s:io.kubernetes.pod.namespace: default Test the Pod access in the default namespace again, and the destination Pod access is normal: # kubectl exec -it dynamic-7d8d7874f5-9v5c4 -n test -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes 64 bytes from 10 .16.0.41: seq = 0 ttl = 64 time = 2 .383 ms 64 bytes from 10 .16.0.41: seq = 1 ttl = 64 time = 0 .115 ms 64 bytes from 10 .16.0.41: seq = 2 ttl = 64 time = 0 .142 ms --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .115/0.880/2.383 ms Using the standard Kubernetes network policy networkpolicy , the test results show that Cilium also restricts access within the same namespace, and cross-namespace access is prohibited. It is different from Kube-OVN implementation. Kube-OVN supports standard k8s network policy, which restricts the destination Pod in a specific namespace, but there is no namespace restriction on the source Pod. Any Pod that meets the restriction rules in any namespace can access the destination Pod.","title":"L3 Network Policy Test"},{"location":"en/advance/cilium-networkpolicy/#l4-network-policy-test","text":"Refer to the following yaml, create CiliumNetworkPolicy resource: apiVersion : \"cilium.io/v2\" kind : CiliumNetworkPolicy metadata : name : \"l4-rule\" namespace : test spec : endpointSelector : matchLabels : app : test ingress : - fromEndpoints : - matchLabels : app : dynamic toPorts : - ports : - port : \"80\" protocol : TCP Test the access of the Pod that meets the network policy rules in the same namespace # kubectl exec -it -n test dynamic-7d8d7874f5-6dsg6 -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl 10 .16.0.41:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on test-54c98bc466-mft5s</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> The Pod that does not meet the network policy rules in the same namespace cannot access # kubectl exec -it -n test label-test1-77b6764857-swq4k -- bash bash-5.0# ping -c 3 10 .16.0.41 PING 10 .16.0.41 ( 10 .16.0.41 ) : 56 data bytes --- 10 .16.0.41 ping statistics --- 3 packets transmitted, 0 packets received, 100 % packet loss bash-5.0# bash-5.0# curl -v 10 .16.0.41:80 --connect-timeout 10 * Trying 10 .16.0.41:80... * After 10000ms connect time, move on! * connect to 10 .16.0.41 port 80 failed: Operation timed out * Connection timeout after 10001 ms * Closing connection 0 curl: ( 28 ) Connection timeout after 10001 ms After the network policy takes effect, cross-namespace access is still prohibited, which is consistent with the L3 network policy test results. After the L4 network policy takes effect, ping cannot be used, but TCP access that meets the policy rules can be executed normally. About the restriction of ICMP, please refer to the official description L4 Limitation Description .","title":"L4 Network Policy Test"},{"location":"en/advance/cilium-networkpolicy/#l7-network-policy-test","text":"chaining mode, L7 network policy currently has problems. In the Cilium official document, there is an explanation for this situation, please refer to Generic Veth Chaining . This problem is tracked using issue 12454 , and it has not been resolved yet. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"L7 Network Policy Test"},{"location":"en/advance/dhcp/","text":"DHCP \u00b6 When using SR-IOV or DPDK type networks, KubeVirt's built-in DHCP does not work in this network mode. Kube-OVN can use the DHCP capabilities of OVN to set DHCP options at the subnet level to help KubeVirt VMs of these network types to properly use DHCP to obtain assigned IP addresses. Kube-OVN supports both DHCPv4 and DHCPv6. The subnet DHCP is configured as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : sn-dual spec : cidrBlock : \"10.0.0.0/24,240e::a00/120\" default : false disableGatewayCheck : true disableInterConnection : false excludeIps : - 10.0.0.1 - 240e::a01 gateway : 10.0.0.1,240e::a01 gatewayNode : '' gatewayType : distributed natOutgoing : false private : false protocol : Dual provider : ovn vpc : vpc-test enableDHCP : true dhcpV4Options : \"lease_time=3600,router=10.0.0.1,server_id=169.254.0.254,server_mac=00:00:00:2E:2F:B8\" dhcpV6Options : \"server_id=00:00:00:2E:2F:C5\" enableIPv6RA : true ipv6RAConfigs : \"address_mode=dhcpv6_stateful,max_interval=30,min_interval=5,send_periodic=true\" enableDHCP : Whether to enable the DHCP function for the subnet. dhcpV4Options , dhcpV6Options : This field directly exposes DHCP-related options within ovn-nb, please reade DHCP Options for more detail. The default value is \"lease_time=3600, router=$ipv4_gateway, server_id=169.254.0.254, server_mac=$random_mac\" and server_id=$random_mac \u3002 enableIPv6RA : Whether to enable the route broadcast function of DHCPv6. ipv6RAConfigs \uff1aThis field directly exposes DHCP-related options within ovn-nb Logical_Router_Port, please read Logical Router Port for more detail. The default value is address_mode=dhcpv6_stateful, max_interval=30, min_interval=5, send_periodic=true \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DHCP"},{"location":"en/advance/dhcp/#dhcp","text":"When using SR-IOV or DPDK type networks, KubeVirt's built-in DHCP does not work in this network mode. Kube-OVN can use the DHCP capabilities of OVN to set DHCP options at the subnet level to help KubeVirt VMs of these network types to properly use DHCP to obtain assigned IP addresses. Kube-OVN supports both DHCPv4 and DHCPv6. The subnet DHCP is configured as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : sn-dual spec : cidrBlock : \"10.0.0.0/24,240e::a00/120\" default : false disableGatewayCheck : true disableInterConnection : false excludeIps : - 10.0.0.1 - 240e::a01 gateway : 10.0.0.1,240e::a01 gatewayNode : '' gatewayType : distributed natOutgoing : false private : false protocol : Dual provider : ovn vpc : vpc-test enableDHCP : true dhcpV4Options : \"lease_time=3600,router=10.0.0.1,server_id=169.254.0.254,server_mac=00:00:00:2E:2F:B8\" dhcpV6Options : \"server_id=00:00:00:2E:2F:C5\" enableIPv6RA : true ipv6RAConfigs : \"address_mode=dhcpv6_stateful,max_interval=30,min_interval=5,send_periodic=true\" enableDHCP : Whether to enable the DHCP function for the subnet. dhcpV4Options , dhcpV6Options : This field directly exposes DHCP-related options within ovn-nb, please reade DHCP Options for more detail. The default value is \"lease_time=3600, router=$ipv4_gateway, server_id=169.254.0.254, server_mac=$random_mac\" and server_id=$random_mac \u3002 enableIPv6RA : Whether to enable the route broadcast function of DHCPv6. ipv6RAConfigs \uff1aThis field directly exposes DHCP-related options within ovn-nb Logical_Router_Port, please read Logical Router Port for more detail. The default value is address_mode=dhcpv6_stateful, max_interval=30, min_interval=5, send_periodic=true \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DHCP"},{"location":"en/advance/dpdk/","text":"DPDK Support \u00b6 This document describes how Kube-OVN combines with OVS-DPDK to provide a DPDK-type network interface to KubeVirt's virtual machines. Upstream KubeVirt does not currently support OVS-DPDK, users need to use the downstream patch Vhostuser implementation to build KubeVirt by themselves or KVM Device Plugin to use OVS-DPDK. Prerequisites \u00b6 The node needs to provide a dedicated NIC for the DPDK driver to run. The node needs to have Hugepages enabled. Set DPDK driver \u00b6 Here we use driverctl for example, please refer to the DPDK documentation for specific parameters and other driver usage: driverctl set-override 0000 :00:0b.0 uio_pci_generic Configure Nodes \u00b6 Labeling OVS-DPDK-enabled nodes for Kube-OVN to recognize: kubectl label nodes <node> ovn.kubernetes.io/ovs_dp_type = \"userspace\" Create the configuration file ovs-dpdk-config in the /opt/ovs-config directory on nodes that support DPDK. ENCAP_IP = 192 .168.122.193/24 DPDK_DEV = 0000 :00:0b.0 ENCAP_IP : The tunnel endpoint address. DPDK_DEV : The PCI ID of the device. Install Kube-OVN \u00b6 Download scripts: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh Enable the DPDK installation option: bash install.sh --with-hybrid-dpdk Usage \u00b6 Here we verify the OVS-DPDK functionality by creating a virtual machine with a vhostuser type NIC. Here we use the KVM Device Plugin to create virtual machines. For more information on how to use it, please refer to [KVM Device Plugin].( https://github.com/kubevirt/kubernetes-device-plugins/blob/master/docs/README.kvm.md ). kubectl apply -f https://raw.githubusercontent.com/kubevirt/kubernetes-device-plugins/master/manifests/kvm-ds.yml Create NetworkAttachmentDefinition: apiVersion : k8s.cni.cncf.io/v1 kind : NetworkAttachmentDefinition metadata : name : ovn-dpdk namespace : default spec : config : >- { \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-dpdk.default.ovn\", \"vhost_user_socket_volume_name\": \"vhostuser-sockets\", \"vhost_user_socket_name\": \"sock\" } Create a VM image using the following Dockerfile: FROM quay.io/kubevirt/virt-launcher:v0.46.1 # wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 COPY CentOS-7-x86_64-GenericCloud.qcow2 /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2 Create a virtual machine: apiVersion : v1 kind : ConfigMap metadata : name : vm-config data : start.sh : | chmod u+w /etc/libvirt/qemu.conf echo \"hugetlbfs_mount = \\\"/dev/hugepages\\\"\" >> /etc/libvirt/qemu.conf virtlogd & libvirtd & mkdir /var/lock sleep 5 virsh define /root/vm/vm.xml virsh start vm tail -f /dev/null vm.xml : | <domain type='kvm'> <name>vm</name> <uuid>4a9b3f53-fa2a-47f3-a757-dd87720d9d1d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <hugepages> <page size='2' unit='M' nodeset='0'/> </hugepages> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <emulatorpin cpuset='1,3'/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='2' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='2097152' unit='KiB' memAccess='shared'/> </numa> </cpu> <on_reboot>restart</on_reboot> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <interface type='vhostuser'> <mac address='00:00:00:0A:30:89'/> <source type='unix' path='/var/run/vm/sock' mode='server'/> <model type='virtio'/> <driver queues='2'> <host mrg_rxbuf='off'/> </driver> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-vm/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> </devices> </domain> --- apiVersion : apps/v1 kind : Deployment metadata : name : vm-deployment labels : app : vm spec : replicas : 1 selector : matchLabels : app : vm template : metadata : labels : app : vm annotations : k8s.v1.cni.cncf.io/networks : default/ovn-dpdk ovn-dpdk.default.ovn.kubernetes.io/ip_address : 10.16.0.96 ovn-dpdk.default.ovn.kubernetes.io/mac_address : 00:00:00:0A:30:89 spec : nodeSelector : ovn.kubernetes.io/ovs_dp_type : userspace securityContext : runAsUser : 0 volumes : - name : vhostuser-sockets emptyDir : {} - name : xml configMap : name : vm-config - name : hugepage emptyDir : medium : HugePages-2Mi - name : libvirt-runtime emptyDir : {} containers : - name : vm image : vm-vhostuser:latest command : [ \"bash\" , \"/root/vm/start.sh\" ] securityContext : capabilities : add : - NET_BIND_SERVICE - SYS_NICE - NET_RAW - NET_ADMIN privileged : false runAsUser : 0 resources : limits : cpu : '2' devices.kubevirt.io/kvm : '1' memory : '8784969729' hugepages-2Mi : 2Gi requests : cpu : 666m devices.kubevirt.io/kvm : '1' ephemeral-storage : 50M memory : '4490002433' volumeMounts : - name : vhostuser-sockets mountPath : /var/run/vm - name : xml mountPath : /root/vm/ - mountPath : /dev/hugepages name : hugepage - name : libvirt-runtime mountPath : /var/run/libvirt Wait for the virtual machine to be created successfully and then go to the Pod to configure the virtual machine: # virsh set-user-password vm root 12345 Password set successfully for root in vm # virsh console vm Connected to domain 'vm' Escape character is ^ ] ( Ctrl + ]) CentOS Linux 7 ( Core ) Kernel 3 .10.0-1127.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Fri Feb 25 09 :52:54 on ttyS0 Next, you can log into the virtual machine for network configuration and test: ip link set eth0 mtu 1400 ip addr add 10 .16.0.96/16 dev eth0 ip ro add default via 10 .16.0.1 ping 114 .114.114.114 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DPDK Support"},{"location":"en/advance/dpdk/#dpdk-support","text":"This document describes how Kube-OVN combines with OVS-DPDK to provide a DPDK-type network interface to KubeVirt's virtual machines. Upstream KubeVirt does not currently support OVS-DPDK, users need to use the downstream patch Vhostuser implementation to build KubeVirt by themselves or KVM Device Plugin to use OVS-DPDK.","title":"DPDK Support"},{"location":"en/advance/dpdk/#prerequisites","text":"The node needs to provide a dedicated NIC for the DPDK driver to run. The node needs to have Hugepages enabled.","title":"Prerequisites"},{"location":"en/advance/dpdk/#set-dpdk-driver","text":"Here we use driverctl for example, please refer to the DPDK documentation for specific parameters and other driver usage: driverctl set-override 0000 :00:0b.0 uio_pci_generic","title":"Set DPDK driver"},{"location":"en/advance/dpdk/#configure-nodes","text":"Labeling OVS-DPDK-enabled nodes for Kube-OVN to recognize: kubectl label nodes <node> ovn.kubernetes.io/ovs_dp_type = \"userspace\" Create the configuration file ovs-dpdk-config in the /opt/ovs-config directory on nodes that support DPDK. ENCAP_IP = 192 .168.122.193/24 DPDK_DEV = 0000 :00:0b.0 ENCAP_IP : The tunnel endpoint address. DPDK_DEV : The PCI ID of the device.","title":"Configure Nodes"},{"location":"en/advance/dpdk/#install-kube-ovn","text":"Download scripts: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh Enable the DPDK installation option: bash install.sh --with-hybrid-dpdk","title":"Install Kube-OVN"},{"location":"en/advance/dpdk/#usage","text":"Here we verify the OVS-DPDK functionality by creating a virtual machine with a vhostuser type NIC. Here we use the KVM Device Plugin to create virtual machines. For more information on how to use it, please refer to [KVM Device Plugin].( https://github.com/kubevirt/kubernetes-device-plugins/blob/master/docs/README.kvm.md ). kubectl apply -f https://raw.githubusercontent.com/kubevirt/kubernetes-device-plugins/master/manifests/kvm-ds.yml Create NetworkAttachmentDefinition: apiVersion : k8s.cni.cncf.io/v1 kind : NetworkAttachmentDefinition metadata : name : ovn-dpdk namespace : default spec : config : >- { \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-dpdk.default.ovn\", \"vhost_user_socket_volume_name\": \"vhostuser-sockets\", \"vhost_user_socket_name\": \"sock\" } Create a VM image using the following Dockerfile: FROM quay.io/kubevirt/virt-launcher:v0.46.1 # wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 COPY CentOS-7-x86_64-GenericCloud.qcow2 /var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2 Create a virtual machine: apiVersion : v1 kind : ConfigMap metadata : name : vm-config data : start.sh : | chmod u+w /etc/libvirt/qemu.conf echo \"hugetlbfs_mount = \\\"/dev/hugepages\\\"\" >> /etc/libvirt/qemu.conf virtlogd & libvirtd & mkdir /var/lock sleep 5 virsh define /root/vm/vm.xml virsh start vm tail -f /dev/null vm.xml : | <domain type='kvm'> <name>vm</name> <uuid>4a9b3f53-fa2a-47f3-a757-dd87720d9d1d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <hugepages> <page size='2' unit='M' nodeset='0'/> </hugepages> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <shares>4096</shares> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <emulatorpin cpuset='1,3'/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='2' threads='1'/> <numa> <cell id='0' cpus='0-1' memory='2097152' unit='KiB' memAccess='shared'/> </numa> </cpu> <on_reboot>restart</on_reboot> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/CentOS-7-x86_64-GenericCloud.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <interface type='vhostuser'> <mac address='00:00:00:0A:30:89'/> <source type='unix' path='/var/run/vm/sock' mode='server'/> <model type='virtio'/> <driver queues='2'> <host mrg_rxbuf='off'/> </driver> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-vm/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> </devices> </domain> --- apiVersion : apps/v1 kind : Deployment metadata : name : vm-deployment labels : app : vm spec : replicas : 1 selector : matchLabels : app : vm template : metadata : labels : app : vm annotations : k8s.v1.cni.cncf.io/networks : default/ovn-dpdk ovn-dpdk.default.ovn.kubernetes.io/ip_address : 10.16.0.96 ovn-dpdk.default.ovn.kubernetes.io/mac_address : 00:00:00:0A:30:89 spec : nodeSelector : ovn.kubernetes.io/ovs_dp_type : userspace securityContext : runAsUser : 0 volumes : - name : vhostuser-sockets emptyDir : {} - name : xml configMap : name : vm-config - name : hugepage emptyDir : medium : HugePages-2Mi - name : libvirt-runtime emptyDir : {} containers : - name : vm image : vm-vhostuser:latest command : [ \"bash\" , \"/root/vm/start.sh\" ] securityContext : capabilities : add : - NET_BIND_SERVICE - SYS_NICE - NET_RAW - NET_ADMIN privileged : false runAsUser : 0 resources : limits : cpu : '2' devices.kubevirt.io/kvm : '1' memory : '8784969729' hugepages-2Mi : 2Gi requests : cpu : 666m devices.kubevirt.io/kvm : '1' ephemeral-storage : 50M memory : '4490002433' volumeMounts : - name : vhostuser-sockets mountPath : /var/run/vm - name : xml mountPath : /root/vm/ - mountPath : /dev/hugepages name : hugepage - name : libvirt-runtime mountPath : /var/run/libvirt Wait for the virtual machine to be created successfully and then go to the Pod to configure the virtual machine: # virsh set-user-password vm root 12345 Password set successfully for root in vm # virsh console vm Connected to domain 'vm' Escape character is ^ ] ( Ctrl + ]) CentOS Linux 7 ( Core ) Kernel 3 .10.0-1127.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Fri Feb 25 09 :52:54 on ttyS0 Next, you can log into the virtual machine for network configuration and test: ip link set eth0 mtu 1400 ip addr add 10 .16.0.96/16 dev eth0 ip ro add default via 10 .16.0.1 ping 114 .114.114.114 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Usage"},{"location":"en/advance/external-gateway/","text":"External Gateway \u00b6 In some scenarios, all container traffic access to the outside needs to be managed and audited through an external gateway. Kube-OVN can forward outbound traffic to the corresponding external gateway by configuring the appropriate routes in the subnet. Usage \u00b6 kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : external spec : cidrBlock : 172.31.0.0/16 gatewayType : centralized natOutgoing : false externalEgressGateway : 192.168.0.1 policyRoutingTableID : 1000 policyRoutingPriority : 1500 natOutgoing : needs to be set to false . externalEgressGateway : Set to the address of the external gateway, which needs to be in the same Layer 2 reachable domain as the gateway node. policyRoutingTableID : The TableID of the local policy routing table used needs to be different for each subnet to avoid conflicts. policyRoutingPriority : Route priority, in order to avoid subsequent user customization of other routing operations conflict, here you can specify the route priority. If no special needs, you can fill in any value. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"External Gateway"},{"location":"en/advance/external-gateway/#external-gateway","text":"In some scenarios, all container traffic access to the outside needs to be managed and audited through an external gateway. Kube-OVN can forward outbound traffic to the corresponding external gateway by configuring the appropriate routes in the subnet.","title":"External Gateway"},{"location":"en/advance/external-gateway/#usage","text":"kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : external spec : cidrBlock : 172.31.0.0/16 gatewayType : centralized natOutgoing : false externalEgressGateway : 192.168.0.1 policyRoutingTableID : 1000 policyRoutingPriority : 1500 natOutgoing : needs to be set to false . externalEgressGateway : Set to the address of the external gateway, which needs to be in the same Layer 2 reachable domain as the gateway node. policyRoutingTableID : The TableID of the local policy routing table used needs to be different for each subnet to avoid conflicts. policyRoutingPriority : Route priority, in order to avoid subsequent user customization of other routing operations conflict, here you can specify the route priority. If no special needs, you can fill in any value. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Usage"},{"location":"en/advance/fastpath/","text":"Compile FastPath Module \u00b6 After a data plane performance profile, netfilter consumes about 20% of CPU resources for related processing within the container and on the host. The FastPath module can bypass netfilter to reduce CPU consumption and latency, and increase throughput. This document will describe how to compile the FastPath module manually. Download Related Code \u00b6 git clone --depth = 1 https://github.com/kubeovn/kube-ovn.git Install Dependencies \u00b6 Here is an example of CentOS dependencies to download: yum install -y kernel-devel- $( uname -r ) gcc elfutils-libelf-devel Compile the Module \u00b6 For the 3.x kernel: cd kube-ovn/fastpath make all For the 4.x kernel: cd kube-ovn/fastpath/4.18 cp ../Makefile . make all Instal the Kernel Module \u00b6 Copy kube_ovn_fastpath.ko to each node that needs performance optimization, and run the following command: insmod kube_ovn_fastpath.ko Use dmesg to confirm successful installation: # dmesg [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in To uninstall a module, use the following command. rmmod kube_ovn_fastpath.ko This module will not be loaded automatically after machine reboot. If you want to load it automatically, please write the corresponding autostart script according to the system configuration. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Compile FastPath Module"},{"location":"en/advance/fastpath/#compile-fastpath-module","text":"After a data plane performance profile, netfilter consumes about 20% of CPU resources for related processing within the container and on the host. The FastPath module can bypass netfilter to reduce CPU consumption and latency, and increase throughput. This document will describe how to compile the FastPath module manually.","title":"Compile FastPath Module"},{"location":"en/advance/fastpath/#download-related-code","text":"git clone --depth = 1 https://github.com/kubeovn/kube-ovn.git","title":"Download Related Code"},{"location":"en/advance/fastpath/#install-dependencies","text":"Here is an example of CentOS dependencies to download: yum install -y kernel-devel- $( uname -r ) gcc elfutils-libelf-devel","title":"Install Dependencies"},{"location":"en/advance/fastpath/#compile-the-module","text":"For the 3.x kernel: cd kube-ovn/fastpath make all For the 4.x kernel: cd kube-ovn/fastpath/4.18 cp ../Makefile . make all","title":"Compile the Module"},{"location":"en/advance/fastpath/#instal-the-kernel-module","text":"Copy kube_ovn_fastpath.ko to each node that needs performance optimization, and run the following command: insmod kube_ovn_fastpath.ko Use dmesg to confirm successful installation: # dmesg [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in To uninstall a module, use the following command. rmmod kube_ovn_fastpath.ko This module will not be loaded automatically after machine reboot. If you want to load it automatically, please write the corresponding autostart script according to the system configuration. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Instal the Kernel Module"},{"location":"en/advance/multi-nic/","text":"Manage Multiple Interface \u00b6 Kube-OVN can provide cluster-level IPAM capabilities for other CNI network plugins such as macvlan, vlan, host-device, etc. Other network plugins can then use the subnet and fixed IP capabilities in Kube-OVN. Kube-OVN also supports address management when multiple NICs are all of Kube-OVN type. Working Principle \u00b6 By using Multus CNI , we can add multiple NICs of different networks to a Pod. However, we still lack the ability to manage the IP addresses of different networks within a cluster. In Kube-OVN, we have been able to perform advanced IP management such as subnet management, IP reservation, random assignment, fixed assignment, etc. through CRD of Subnet and IP. Now Kube-OVN extend the subnet to integrate with other different network plugins, so that other network plugins can also use the IPAM functionality of Kube-OVN. Workflow \u00b6 The above diagram shows how to manage the IP addresses of other network plugins via Kube-OVN. The eth0 NIC of the container is connected to the OVN network and the net1 NIC is connected to other CNI networks. The network definition for the net1 network is taken from the NetworkAttachmentDefinition resource definition in multus-cni. When a Pod is created, kube-ovn-controller will get the Pod add event, find the corresponding Subnet according to the annotation in the Pod, then manage the address from it, and write the address information assigned to the Pod back to the Pod annotation. The CNI on the container machine can configure kube-ovn-cni as the ipam plugin. kube-ovn-cni will read the Pod annotation and return the address information to the corresponding CNI plugin using the standard format of the CNI protocol. Usage \u00b6 Install Kube-OVN and Multus \u00b6 Please refer One-Click Installation and Multus how to use to install Kube-OVN and Multus-CNI. Provide IPAM for other types of CNI \u00b6 Create NetworkAttachmentDefinition \u00b6 Here we use macvlan as the second network of the container network and set its ipam to kube-ovn : apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : macvlan namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"macvlan.default\" } }' spec.config.ipam.type : Need to be set to kube-ovn to call the kube-ovn plugin to get the address information. server_socket : The socket file used for communication to Kube-OVN. The default location is /run/openvswitch/kube-ovn-daemon.sock . provider : The current NetworkAttachmentDefinition's <name>. <namespace> , Kube-OVN will use this information to find the corresponding Subnet resource. The attached NIC is a Kube-OVN type NIC \u00b6 At this point, the multiple NICs are all Kube-OVN type NICs. Create NetworkAttachmentDefinition \u00b6 Set the provider suffix to ovn : apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" }' spec.config.ipam.type : Need to be set to kube-ovn to call the kube-ovn plugin to get the address information. server_socket : The socket file used for communication to Kube-OVN. The default location is /run/openvswitch/kube-ovn-daemon.sock . provider : The current NetworkAttachmentDefinition's <name>. <namespace> , Kube-OVN will use this information to find the corresponding Subnet resource. It should have the suffix ovn here. Create a Kube-OVN Subnet \u00b6 Create a Kube-OVN Subnet, set the corresponding cidrBlock and exclude_ips , the provider should be set to the <name>. <namespace> of corresponding NetworkAttachmentDefinition. For example, to provide additional NICs with macvlan, create a Subnet as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : macvlan spec : protocol : IPv4 provider : macvlan.default cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 gateway , private , nat are only valid for networks with provider type ovn, not for attachment networks. If you are using Kube-OVN as an attached NIC, provider should be set to the <name>. <namespace>.ovn of the corresponding NetworkAttachmentDefinition, and should end with ovn as a suffix. An example of creating a Subnet with an additional NIC provided by Kube-OVN is as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attachnet spec : protocol : IPv4 provider : attachnet.default.ovn cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 Create a Pod with Multiple NIC \u00b6 For Pods with randomly assigned addresses, simply add the following annotation k8s.v1.cni.cncf.io/networks , taking the value <namespace>/<name> of the corresponding NetworkAttachmentDefinition.\uff1a apiVersion : v1 kind : Pod metadata : name : samplepod namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan spec : containers : - name : samplepod command : [ \"/bin/ash\" , \"-c\" , \"trap : TERM INT; sleep infinity & wait\" ] image : docker.io/library/alpine:edge Create Pod with a Fixed IP \u00b6 For Pods with fixed IPs, add <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_address annotation\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 macvlan.default.kubernetes.io/ip_address : 172.17.0.100 macvlan.default.kubernetes.io/mac_address : 00:00:00:53:6B:BB spec : containers : - name : static-ip image : docker.io/library/nginx:alpine Create Workloads with Fixed IPs \u00b6 For workloads that use ippool, add <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_pool annotations: apiVersion : apps/v1 kind : Deployment metadata : namespace : default name : static-workload labels : app : static-workload spec : replicas : 2 selector : matchLabels : app : static-workload template : metadata : labels : app : static-workload annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 macvlan.default.kubernetes.io/ip_pool : 172.17.0.200,172.17.0.201,172.17.0.202 spec : containers : - name : static-workload image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Manage Multiple Interface"},{"location":"en/advance/multi-nic/#manage-multiple-interface","text":"Kube-OVN can provide cluster-level IPAM capabilities for other CNI network plugins such as macvlan, vlan, host-device, etc. Other network plugins can then use the subnet and fixed IP capabilities in Kube-OVN. Kube-OVN also supports address management when multiple NICs are all of Kube-OVN type.","title":"Manage Multiple Interface"},{"location":"en/advance/multi-nic/#working-principle","text":"By using Multus CNI , we can add multiple NICs of different networks to a Pod. However, we still lack the ability to manage the IP addresses of different networks within a cluster. In Kube-OVN, we have been able to perform advanced IP management such as subnet management, IP reservation, random assignment, fixed assignment, etc. through CRD of Subnet and IP. Now Kube-OVN extend the subnet to integrate with other different network plugins, so that other network plugins can also use the IPAM functionality of Kube-OVN.","title":"Working Principle"},{"location":"en/advance/multi-nic/#workflow","text":"The above diagram shows how to manage the IP addresses of other network plugins via Kube-OVN. The eth0 NIC of the container is connected to the OVN network and the net1 NIC is connected to other CNI networks. The network definition for the net1 network is taken from the NetworkAttachmentDefinition resource definition in multus-cni. When a Pod is created, kube-ovn-controller will get the Pod add event, find the corresponding Subnet according to the annotation in the Pod, then manage the address from it, and write the address information assigned to the Pod back to the Pod annotation. The CNI on the container machine can configure kube-ovn-cni as the ipam plugin. kube-ovn-cni will read the Pod annotation and return the address information to the corresponding CNI plugin using the standard format of the CNI protocol.","title":"Workflow"},{"location":"en/advance/multi-nic/#usage","text":"","title":"Usage"},{"location":"en/advance/multi-nic/#install-kube-ovn-and-multus","text":"Please refer One-Click Installation and Multus how to use to install Kube-OVN and Multus-CNI.","title":"Install Kube-OVN and Multus"},{"location":"en/advance/multi-nic/#provide-ipam-for-other-types-of-cni","text":"","title":"Provide IPAM for other types of CNI"},{"location":"en/advance/multi-nic/#create-networkattachmentdefinition","text":"Here we use macvlan as the second network of the container network and set its ipam to kube-ovn : apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : macvlan namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"macvlan.default\" } }' spec.config.ipam.type : Need to be set to kube-ovn to call the kube-ovn plugin to get the address information. server_socket : The socket file used for communication to Kube-OVN. The default location is /run/openvswitch/kube-ovn-daemon.sock . provider : The current NetworkAttachmentDefinition's <name>. <namespace> , Kube-OVN will use this information to find the corresponding Subnet resource.","title":"Create NetworkAttachmentDefinition"},{"location":"en/advance/multi-nic/#the-attached-nic-is-a-kube-ovn-type-nic","text":"At this point, the multiple NICs are all Kube-OVN type NICs.","title":"The attached NIC is a Kube-OVN type NIC"},{"location":"en/advance/multi-nic/#create-networkattachmentdefinition_1","text":"Set the provider suffix to ovn : apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" }' spec.config.ipam.type : Need to be set to kube-ovn to call the kube-ovn plugin to get the address information. server_socket : The socket file used for communication to Kube-OVN. The default location is /run/openvswitch/kube-ovn-daemon.sock . provider : The current NetworkAttachmentDefinition's <name>. <namespace> , Kube-OVN will use this information to find the corresponding Subnet resource. It should have the suffix ovn here.","title":"Create NetworkAttachmentDefinition"},{"location":"en/advance/multi-nic/#create-a-kube-ovn-subnet","text":"Create a Kube-OVN Subnet, set the corresponding cidrBlock and exclude_ips , the provider should be set to the <name>. <namespace> of corresponding NetworkAttachmentDefinition. For example, to provide additional NICs with macvlan, create a Subnet as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : macvlan spec : protocol : IPv4 provider : macvlan.default cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10 gateway , private , nat are only valid for networks with provider type ovn, not for attachment networks. If you are using Kube-OVN as an attached NIC, provider should be set to the <name>. <namespace>.ovn of the corresponding NetworkAttachmentDefinition, and should end with ovn as a suffix. An example of creating a Subnet with an additional NIC provided by Kube-OVN is as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attachnet spec : protocol : IPv4 provider : attachnet.default.ovn cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 excludeIps : - 172.17.0.0..172.17.0.10","title":"Create a Kube-OVN Subnet"},{"location":"en/advance/multi-nic/#create-a-pod-with-multiple-nic","text":"For Pods with randomly assigned addresses, simply add the following annotation k8s.v1.cni.cncf.io/networks , taking the value <namespace>/<name> of the corresponding NetworkAttachmentDefinition.\uff1a apiVersion : v1 kind : Pod metadata : name : samplepod namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan spec : containers : - name : samplepod command : [ \"/bin/ash\" , \"-c\" , \"trap : TERM INT; sleep infinity & wait\" ] image : docker.io/library/alpine:edge","title":"Create a Pod with Multiple NIC"},{"location":"en/advance/multi-nic/#create-pod-with-a-fixed-ip","text":"For Pods with fixed IPs, add <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_address annotation\uff1a apiVersion : v1 kind : Pod metadata : name : static-ip namespace : default annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 macvlan.default.kubernetes.io/ip_address : 172.17.0.100 macvlan.default.kubernetes.io/mac_address : 00:00:00:53:6B:BB spec : containers : - name : static-ip image : docker.io/library/nginx:alpine","title":"Create Pod with a Fixed IP"},{"location":"en/advance/multi-nic/#create-workloads-with-fixed-ips","text":"For workloads that use ippool, add <networkAttachmentName>.<networkAttachmentNamespace>.kubernetes.io/ip_pool annotations: apiVersion : apps/v1 kind : Deployment metadata : namespace : default name : static-workload labels : app : static-workload spec : replicas : 2 selector : matchLabels : app : static-workload template : metadata : labels : app : static-workload annotations : k8s.v1.cni.cncf.io/networks : default/macvlan ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 macvlan.default.kubernetes.io/ip_pool : 172.17.0.200,172.17.0.201,172.17.0.202 spec : containers : - name : static-workload image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Create Workloads with Fixed IPs"},{"location":"en/advance/nat-policy-rule/","text":"NAT Policy Rule Function \u00b6 NAT Policy Rule Function Purpose \u00b6 In the Overlay subnet under the default VPC, when the natOutgoing switch is turned on, all Pods in the subnet need to do SNAT to access the external network to the IP of the current node, but in some scenarios we do not want all Pods in the subnet to access the external network by SNAT. So the NAT Policy Rule is to provide an interface for users to decide which CIDRs or IPs in the subnet to access the external network for SNAT. How to use NAT Policy Rules \u00b6 Enable the natOutgoing switch in subnet.Spec , and add the field natOutgoingPolicyRules as follows: spec : natOutgoing : true natOutgoingPolicyRules : - action : forward match : srcIPs : 10.0.11.0/30,10.0.11.254 - action : nat match : srcIPs : 10.0.11.128/26 dstIPs : 114.114.114.114,8.8.8.8 The above case shows that there are two NAT policy rules: Packets with source IP 10.0.11.0/30 or 10.0.11.254 will not perform SNAT when accessing the external network. When a packet with source IP 10.0.11.128/26 and destination IP 114.114.114.114 or 8.8.8.8 accesses the external network, SNAT will be performed. Field description: Action: Indicates the action that will be executed for the message that meets the corresponding conditions of the \"match\". The action is divided into two types: forward and nat . SNAT. When natOutgoingPolicyRules is not configured, packets are still SNAT by default. match: Indicates the matching segment of the message, the matching segment includes srcIPs and dstIPs, here indicates the source IP and destination IP of the message from the subnet to the external network. match.srcIPs and match.dstIPs support multiple cidr and ip, separated by commas. If several matches are repeated but the actions are different, the array position of natOutgoingPolicyRules shall prevail, and the lower the array index, the higher the priority. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NAT Policy Rule Function"},{"location":"en/advance/nat-policy-rule/#nat-policy-rule-function","text":"","title":"NAT Policy Rule Function"},{"location":"en/advance/nat-policy-rule/#nat-policy-rule-function-purpose","text":"In the Overlay subnet under the default VPC, when the natOutgoing switch is turned on, all Pods in the subnet need to do SNAT to access the external network to the IP of the current node, but in some scenarios we do not want all Pods in the subnet to access the external network by SNAT. So the NAT Policy Rule is to provide an interface for users to decide which CIDRs or IPs in the subnet to access the external network for SNAT.","title":"NAT Policy Rule Function Purpose"},{"location":"en/advance/nat-policy-rule/#how-to-use-nat-policy-rules","text":"Enable the natOutgoing switch in subnet.Spec , and add the field natOutgoingPolicyRules as follows: spec : natOutgoing : true natOutgoingPolicyRules : - action : forward match : srcIPs : 10.0.11.0/30,10.0.11.254 - action : nat match : srcIPs : 10.0.11.128/26 dstIPs : 114.114.114.114,8.8.8.8 The above case shows that there are two NAT policy rules: Packets with source IP 10.0.11.0/30 or 10.0.11.254 will not perform SNAT when accessing the external network. When a packet with source IP 10.0.11.128/26 and destination IP 114.114.114.114 or 8.8.8.8 accesses the external network, SNAT will be performed. Field description: Action: Indicates the action that will be executed for the message that meets the corresponding conditions of the \"match\". The action is divided into two types: forward and nat . SNAT. When natOutgoingPolicyRules is not configured, packets are still SNAT by default. match: Indicates the matching segment of the message, the matching segment includes srcIPs and dstIPs, here indicates the source IP and destination IP of the message from the subnet to the external network. match.srcIPs and match.dstIPs support multiple cidr and ip, separated by commas. If several matches are repeated but the actions are different, the array position of natOutgoingPolicyRules shall prevail, and the lower the array index, the higher the priority. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"How to use NAT Policy Rules"},{"location":"en/advance/node-local-dns/","text":"NodeLocal DNSCache and Kube-OVN adaptation \u00b6 NodeLocal DNSCache improves cluster DNS performance by running DNS cache as a DaemonSet on cluster nodes. This function can also be adapted to Kube-OVN. Nodelocal DNSCache deployment \u00b6 Deploy Kubernetes NodeLocal DNScache \u00b6 This step refers to Kubernetes official website configuration nodelocaldnscache . Deploy with the following script: #!bin/bash localdns = 169 .254.20.10 domain = cluster.local kubedns = 10 .96.0.10 wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml sed -i \"s/__PILLAR__LOCAL__DNS__/ $localdns /g; s/__PILLAR__DNS__DOMAIN__/ $domain /g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/ $kubedns /g\" nodelocaldns.yaml kubectl apply -f nodelocaldns.yaml Modify the kubelet configuration file on each node, modify the clusterDNS field in /var/lib/kubelet/config.yaml to the local DNS IP 169.254.20.10, and then restart the kubelet service. Kube-OVN corresponding DNS configuration \u00b6 After deploying the Nodelocal DNScache component of Kubernetes, Kube-OVN needs to make the following modifications: Underlay subnet enable U2O switch \u00b6 If the underlay subnet needs to use the local DNS function, you need to enable the U2O function, that is, configure spec.u2oInterconnection = true in kubectl edit subnet {your subnet} . If it is an overlay subnet, this step is not required. Specify the corresponding local DNS IP for kube-ovn-controller \u00b6 kubectl edit deployment kube-ovn-controller -n kube-system Add field to spec.template.spec.containers.args --node-local-dns-ip=169.254.20.10 Rebuild the created Pods \u00b6 The reason for this step is to let the Pod regenerate /etc/resolv.conf so that the nameserver points to the local DNS IP. If the nameserver of the Pod is not rebuilt, it will still use the DNS ClusterIP of the cluster. At the same time, if the u2o switch is turned on, the Pod needs to be rebuilt to regenerate the Pod gateway. Validator local DNS cache function \u00b6 After the above configuration is completed, you can find the Pod verification as follows. You can see that the Pod's DNS server points to the local 169.254.20.10 and successfully resolves the domain name: # kubectl exec -it pod1 -- nslookup github.com Server: 169 .254.20.10 Address: 169 .254.20.10:53 Name: github.com Address: 20 .205.243.166 You can also capture packets at the node and verify as follows. You can see that the DNS query message reaches the local DNS service through the ovn0 network card, and the DNS response message returns in the same way: # tcpdump -i any port 53 06 :20:00.441889 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441889 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441950 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.441950 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.442203 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442219 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442273 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) 06 :20:00.442278 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Node Local DNS Cache and Kube-OVN Adaptation"},{"location":"en/advance/node-local-dns/#nodelocal-dnscache-and-kube-ovn-adaptation","text":"NodeLocal DNSCache improves cluster DNS performance by running DNS cache as a DaemonSet on cluster nodes. This function can also be adapted to Kube-OVN.","title":"NodeLocal DNSCache and Kube-OVN adaptation"},{"location":"en/advance/node-local-dns/#nodelocal-dnscache-deployment","text":"","title":"Nodelocal DNSCache deployment"},{"location":"en/advance/node-local-dns/#deploy-kubernetes-nodelocal-dnscache","text":"This step refers to Kubernetes official website configuration nodelocaldnscache . Deploy with the following script: #!bin/bash localdns = 169 .254.20.10 domain = cluster.local kubedns = 10 .96.0.10 wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml sed -i \"s/__PILLAR__LOCAL__DNS__/ $localdns /g; s/__PILLAR__DNS__DOMAIN__/ $domain /g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/ $kubedns /g\" nodelocaldns.yaml kubectl apply -f nodelocaldns.yaml Modify the kubelet configuration file on each node, modify the clusterDNS field in /var/lib/kubelet/config.yaml to the local DNS IP 169.254.20.10, and then restart the kubelet service.","title":"Deploy Kubernetes NodeLocal DNScache"},{"location":"en/advance/node-local-dns/#kube-ovn-corresponding-dns-configuration","text":"After deploying the Nodelocal DNScache component of Kubernetes, Kube-OVN needs to make the following modifications:","title":"Kube-OVN corresponding DNS configuration"},{"location":"en/advance/node-local-dns/#underlay-subnet-enable-u2o-switch","text":"If the underlay subnet needs to use the local DNS function, you need to enable the U2O function, that is, configure spec.u2oInterconnection = true in kubectl edit subnet {your subnet} . If it is an overlay subnet, this step is not required.","title":"Underlay subnet enable U2O switch"},{"location":"en/advance/node-local-dns/#specify-the-corresponding-local-dns-ip-for-kube-ovn-controller","text":"kubectl edit deployment kube-ovn-controller -n kube-system Add field to spec.template.spec.containers.args --node-local-dns-ip=169.254.20.10","title":"Specify the corresponding local DNS IP for kube-ovn-controller"},{"location":"en/advance/node-local-dns/#rebuild-the-created-pods","text":"The reason for this step is to let the Pod regenerate /etc/resolv.conf so that the nameserver points to the local DNS IP. If the nameserver of the Pod is not rebuilt, it will still use the DNS ClusterIP of the cluster. At the same time, if the u2o switch is turned on, the Pod needs to be rebuilt to regenerate the Pod gateway.","title":"Rebuild the created Pods"},{"location":"en/advance/node-local-dns/#validator-local-dns-cache-function","text":"After the above configuration is completed, you can find the Pod verification as follows. You can see that the Pod's DNS server points to the local 169.254.20.10 and successfully resolves the domain name: # kubectl exec -it pod1 -- nslookup github.com Server: 169 .254.20.10 Address: 169 .254.20.10:53 Name: github.com Address: 20 .205.243.166 You can also capture packets at the node and verify as follows. You can see that the DNS query message reaches the local DNS service through the ovn0 network card, and the DNS response message returns in the same way: # tcpdump -i any port 53 06 :20:00.441889 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441889 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1291 + A? baidu.com. ( 27 ) 06 :20:00.441950 659246098c56_h P ifindex 17 00 :00:00:73:f1:06 ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.441950 ovn0 In ifindex 7 00 :00:00:50:32:cd ethertype IPv4 ( 0x0800 ) , length 75 : 10 .16.0.2.40230 > 169 .254.20.10.53: 1611 + AAAA? baidu.com. ( 27 ) 06 :20:00.442203 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442219 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 145 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1611 * 0 /1/0 ( 97 ) 06 :20:00.442273 ovn0 Out ifindex 7 00 :00:00:52:99:d8 ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) 06 :20:00.442278 659246098c56_h Out ifindex 17 00 :00:00:ea:b3:5e ethertype IPv4 ( 0x0800 ) , length 125 : 169 .254.20.10.53 > 10 .16.0.2.40230: 1291 * 2 /0/0 A 39 .156.66.10, A 110 .242.68.66 ( 77 ) \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Validator local DNS cache function"},{"location":"en/advance/offload-corigine/","text":"Offload with Corigine \u00b6 Kube-OVN uses OVS for traffic forwarding in the final data plane, and the associated flow table matching, tunnel encapsulation and other functions are CPU-intensive, which consumes a lot of CPU resources and leads to higher latency and lower throughput under heavy traffic. Corigine Agilio CX series SmartNIC can offload OVS-related operations to the hardware. This technology can shorten the data path without modifying the OVS control plane, avoiding the use of host CPU resources, which dramatically reduce latency and significantly increase the throughput. Prerequisites \u00b6 Corigine Agilio CX series SmartNIC. CentOS 8 Stream or Linux 5.7 above. Since the current NIC does not support dp_hash and hash operation offload, OVN LB function should be disabled. Setup SR-IOV \u00b6 Please read Agilio Open vSwitch TC User Guide for the detail usage of this SmartNIC. The following scripts are saved for subsequent execution of firmware-related operations: #!/bin/bash DEVICE = ${ 1 } DEFAULT_ASSY = scan ASSY = ${ 2 :- ${ DEFAULT_ASSY }} APP = ${ 3 :- flower } if [ \"x ${ DEVICE } \" = \"x\" -o ! -e /sys/class/net/ ${ DEVICE } ] ; then echo Syntax: ${ 0 } device [ ASSY ] [ APP ] echo echo This script associates the TC Offload firmware echo with a Netronome SmartNIC. echo echo device: is the network device associated with the SmartNIC echo ASSY: defaults to ${ DEFAULT_ASSY } echo APP: defaults to flower. flower-next is supported if updated echo firmware has been installed. exit 1 fi # It is recommended that the assembly be determined by inspection # The following code determines the value via the debug interface if [ \" ${ ASSY } x\" = \"scanx\" ] ; then ethtool -W ${ DEVICE } 0 DEBUG = $( ethtool -w ${ DEVICE } data /dev/stdout | strings ) SERIAL = $( echo \" ${ DEBUG } \" | grep \"^SN:\" ) ASSY = $( echo ${ SERIAL } | grep -oE AMDA [ 0 -9 ]{ 4 } ) fi PCIADDR = $( basename $( readlink -e /sys/class/net/ ${ DEVICE } /device )) FWDIR = \"/lib/firmware/netronome\" # AMDA0081 and AMDA0097 uses the same firmware if [ \" ${ ASSY } \" = \"AMDA0081\" ] ; then if [ ! -e ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw ] ; then ln -sf nic_AMDA0097.nffw ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw fi fi FW = \" ${ FWDIR } /pci- ${ PCIADDR } .nffw\" ln -sf \" ${ APP } /nic_ ${ ASSY } .nffw\" \" ${ FW } \" # insert distro-specific initramfs section here... Switching firmware options and reloading the driver: ./agilio-tc-fw-select.sh ens47np0 scan rmmod nfp modprobe nfp Check the number of available VFs and create VFs. # cat /sys/class/net/ens3/device/sriov_totalvfs 65 # echo 4 > /sys/class/net/ens47/device/sriov_numvfs Install SR-IOV Device Plugin \u00b6 Since each machine has a limited number of VFs and each Pod that uses acceleration will take up VF resources, we need to use the SR-IOV Device Plugin to manage the corresponding resources so that the scheduler knows how to schedule. Create SR-IOV Configmap: apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"corigine.com\", \"resourceName\": \"agilio_sriov\", \"selectors\": { \"vendors\": [\"19ee\"], \"devices\": [\"6003\"], \"drivers\": [\"nfp_netvf\"] } } ] } Please read the SR-IOV device plugin to deploy: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml Check if SR-IOV resources have been registered to Kubernetes Node: kubectl describe no containerserver | grep corigine corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov 0 0 Install Multus-CNI \u00b6 The device IDs obtained during SR-IOV Device Plugin scheduling need to be passed to Kube-OVN via Multus-CNI, so Multus-CNI needs to be configured to perform the related tasks. Please read Multus-CNI Document to deploy\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml Create NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : corigine.com/agilio_sriov spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : the format should be {name}.{namespace}.ovn of related NetworkAttachmentDefinition . Enable Offload in Kube-OVN \u00b6 Download the scripts: wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh Change the related options\uff0c IFACE should be the physic NIC and has an IP: ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" Install Kube-OVN\uff1a bash install.sh Create Pods with VF NICs \u00b6 Pods that use VF for network offload acceleration can be created using the following yaml: apiVersion : v1 kind : Pod metadata : name : nginx namespace : default annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : corigine.com/agilio_sriov : '1' limits : corigine.com/agilio_sriov : '1' v1.multus-cni.io/default-network : should be the {namespace}/{name} of related NetworkAttachmentDefinition . Running the following command in the ovs-ovn container of the Pod run node to observe if offload success. # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h If there is offloaded:yes, dp:tc content, the offloading is successful. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Offload with Corigine"},{"location":"en/advance/offload-corigine/#offload-with-corigine","text":"Kube-OVN uses OVS for traffic forwarding in the final data plane, and the associated flow table matching, tunnel encapsulation and other functions are CPU-intensive, which consumes a lot of CPU resources and leads to higher latency and lower throughput under heavy traffic. Corigine Agilio CX series SmartNIC can offload OVS-related operations to the hardware. This technology can shorten the data path without modifying the OVS control plane, avoiding the use of host CPU resources, which dramatically reduce latency and significantly increase the throughput.","title":"Offload with Corigine"},{"location":"en/advance/offload-corigine/#prerequisites","text":"Corigine Agilio CX series SmartNIC. CentOS 8 Stream or Linux 5.7 above. Since the current NIC does not support dp_hash and hash operation offload, OVN LB function should be disabled.","title":"Prerequisites"},{"location":"en/advance/offload-corigine/#setup-sr-iov","text":"Please read Agilio Open vSwitch TC User Guide for the detail usage of this SmartNIC. The following scripts are saved for subsequent execution of firmware-related operations: #!/bin/bash DEVICE = ${ 1 } DEFAULT_ASSY = scan ASSY = ${ 2 :- ${ DEFAULT_ASSY }} APP = ${ 3 :- flower } if [ \"x ${ DEVICE } \" = \"x\" -o ! -e /sys/class/net/ ${ DEVICE } ] ; then echo Syntax: ${ 0 } device [ ASSY ] [ APP ] echo echo This script associates the TC Offload firmware echo with a Netronome SmartNIC. echo echo device: is the network device associated with the SmartNIC echo ASSY: defaults to ${ DEFAULT_ASSY } echo APP: defaults to flower. flower-next is supported if updated echo firmware has been installed. exit 1 fi # It is recommended that the assembly be determined by inspection # The following code determines the value via the debug interface if [ \" ${ ASSY } x\" = \"scanx\" ] ; then ethtool -W ${ DEVICE } 0 DEBUG = $( ethtool -w ${ DEVICE } data /dev/stdout | strings ) SERIAL = $( echo \" ${ DEBUG } \" | grep \"^SN:\" ) ASSY = $( echo ${ SERIAL } | grep -oE AMDA [ 0 -9 ]{ 4 } ) fi PCIADDR = $( basename $( readlink -e /sys/class/net/ ${ DEVICE } /device )) FWDIR = \"/lib/firmware/netronome\" # AMDA0081 and AMDA0097 uses the same firmware if [ \" ${ ASSY } \" = \"AMDA0081\" ] ; then if [ ! -e ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw ] ; then ln -sf nic_AMDA0097.nffw ${ FWDIR } / ${ APP } /nic_AMDA0081.nffw fi fi FW = \" ${ FWDIR } /pci- ${ PCIADDR } .nffw\" ln -sf \" ${ APP } /nic_ ${ ASSY } .nffw\" \" ${ FW } \" # insert distro-specific initramfs section here... Switching firmware options and reloading the driver: ./agilio-tc-fw-select.sh ens47np0 scan rmmod nfp modprobe nfp Check the number of available VFs and create VFs. # cat /sys/class/net/ens3/device/sriov_totalvfs 65 # echo 4 > /sys/class/net/ens47/device/sriov_numvfs","title":"Setup SR-IOV"},{"location":"en/advance/offload-corigine/#install-sr-iov-device-plugin","text":"Since each machine has a limited number of VFs and each Pod that uses acceleration will take up VF resources, we need to use the SR-IOV Device Plugin to manage the corresponding resources so that the scheduler knows how to schedule. Create SR-IOV Configmap: apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"corigine.com\", \"resourceName\": \"agilio_sriov\", \"selectors\": { \"vendors\": [\"19ee\"], \"devices\": [\"6003\"], \"drivers\": [\"nfp_netvf\"] } } ] } Please read the SR-IOV device plugin to deploy: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml Check if SR-IOV resources have been registered to Kubernetes Node: kubectl describe no containerserver | grep corigine corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov: 4 corigine.com/agilio_sriov 0 0","title":"Install SR-IOV Device Plugin"},{"location":"en/advance/offload-corigine/#install-multus-cni","text":"The device IDs obtained during SR-IOV Device Plugin scheduling need to be passed to Kube-OVN via Multus-CNI, so Multus-CNI needs to be configured to perform the related tasks. Please read Multus-CNI Document to deploy\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml Create NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : corigine.com/agilio_sriov spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : the format should be {name}.{namespace}.ovn of related NetworkAttachmentDefinition .","title":"Install Multus-CNI"},{"location":"en/advance/offload-corigine/#enable-offload-in-kube-ovn","text":"Download the scripts: wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh Change the related options\uff0c IFACE should be the physic NIC and has an IP: ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" Install Kube-OVN\uff1a bash install.sh","title":"Enable Offload in Kube-OVN"},{"location":"en/advance/offload-corigine/#create-pods-with-vf-nics","text":"Pods that use VF for network offload acceleration can be created using the following yaml: apiVersion : v1 kind : Pod metadata : name : nginx namespace : default annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : corigine.com/agilio_sriov : '1' limits : corigine.com/agilio_sriov : '1' v1.multus-cni.io/default-network : should be the {namespace}/{name} of related NetworkAttachmentDefinition . Running the following command in the ovs-ovn container of the Pod run node to observe if offload success. # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h If there is offloaded:yes, dp:tc content, the offloading is successful. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Create Pods with VF NICs"},{"location":"en/advance/offload-mellanox/","text":"Offload with Mellanox \u00b6 Kube-OVN uses OVS for traffic forwarding in the final data plane, and the associated flow table matching, tunnel encapsulation and other functions are CPU-intensive, which consumes a lot of CPU resources and leads to higher latency and lower throughput under heavy traffic. Mellanox Accelerated Switching And Packet Processing (ASAP\u00b2) technology offloads OVS-related operations to an eSwitch within the eSwitch in the hardware. This technology can shorten the data path without modifying the OVS control plane, avoiding the use of host CPU resources, which dramatically reduce latency and significantly increase the throughput. Prerequisites \u00b6 Mellanox CX5/CX6/BlueField that support ASAP\u00b2. CentOS 8 Stream or Linux 5.7 above. Since the current NIC does not support dp_hash and hash operation offload, OVN LB function should be disabled. In order to support offload mode, the NIC cannot do bond. Setup SR-IOV \u00b6 Check the device ID of the NIC, in the following example it is 42:00.0 : # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] Find the corresponding NIC by its device ID: # ls -l /sys/class/net/ | grep 42:00.0 lrwxrwxrwx. 1 root root 0 Jul 22 23 :16 p4p1 -> ../../devices/pci0000:40/0000:40:02.0/0000:42:00.0/net/p4p1 Check the number of available VFs: # cat /sys/class/net/p4p1/device/sriov_totalvfs 8 Create VFs and do not exceeding the number found above: # echo '4' > /sys/class/net/p4p1/device/sriov_numvfs # ip link show p4p1 10 : p4p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:59:9f:c1:ec:12 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 1 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 2 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 3 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off # ip link set p4p1 up Find the device IDs corresponding to the above VFs: # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] Unbound the VFs from the driver: echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind Enable eSwitch mode and set up hardware offload: devlink dev eswitch set pci/0000:42:00.0 mode switchdev ethtool -K enp66s0f0 hw-tc-offload on Rebind the driver and complete the VF setup: echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/bind Some behaviors of NetworkManager may cause driver exceptions, if offloading problems occur we recommended to close NetworkManager and try again. systemctl stop NetworkManager systemctl disable NetworkManager Install SR-IOV Device Plugin \u00b6 Since each machine has a limited number of VFs and each Pod that uses acceleration will take up VF resources, we need to use the SR-IOV Device Plugin to manage the corresponding resources so that the scheduler knows how to schedule. Create SR-IOV Configmap: apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"mellanox.com\", \"resourceName\": \"cx5_sriov_switchdev\", \"selectors\": { \"vendors\": [\"15b3\"], \"devices\": [\"1018\"], \"drivers\": [\"mlx5_core\"] } } ] } Please read the SR-IOV device plugin to deploy: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml Check if SR-IOV resources have been registered to Kubernetes Node: kubectl describe node kube-ovn-01 | grep mellanox mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev 0 0 Install Multus-CNI \u00b6 The device IDs obtained during SR-IOV Device Plugin scheduling need to be passed to Kube-OVN via Multus-CNI, so Multus-CNI needs to be configured to perform the related tasks. Please read Multus-CNI Document to deploy\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml Create NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : mellanox.com/cx5_sriov_switchdev spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : the format should be {name}.{namespace}.ovn of related NetworkAttachmentDefinition . Enable Offload in Kube-OVN \u00b6 Download the scripts: wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh Change the related options\uff0c IFACE should be the physic NIC and has an IP: ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" Install Kube-OVN\uff1a bash install.sh Create Pods with VF NICs \u00b6 Pods that use VF for network offload acceleration can be created using the following yaml: apiVersion : v1 kind : Pod metadata : name : nginx annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : mellanox.com/cx5_sriov_switchdev : '1' limits : mellanox.com/cx5_sriov_switchdev : '1' v1.multus-cni.io/default-network : should be the {namespace}/{name} of related NetworkAttachmentDefinition . Running the following command in the ovs-ovn container of the Pod run node to observe if offload success. # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h If there is offloaded:yes, dp:tc content, the offloading is successful. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Offload with Mellanox"},{"location":"en/advance/offload-mellanox/#offload-with-mellanox","text":"Kube-OVN uses OVS for traffic forwarding in the final data plane, and the associated flow table matching, tunnel encapsulation and other functions are CPU-intensive, which consumes a lot of CPU resources and leads to higher latency and lower throughput under heavy traffic. Mellanox Accelerated Switching And Packet Processing (ASAP\u00b2) technology offloads OVS-related operations to an eSwitch within the eSwitch in the hardware. This technology can shorten the data path without modifying the OVS control plane, avoiding the use of host CPU resources, which dramatically reduce latency and significantly increase the throughput.","title":"Offload with Mellanox"},{"location":"en/advance/offload-mellanox/#prerequisites","text":"Mellanox CX5/CX6/BlueField that support ASAP\u00b2. CentOS 8 Stream or Linux 5.7 above. Since the current NIC does not support dp_hash and hash operation offload, OVN LB function should be disabled. In order to support offload mode, the NIC cannot do bond.","title":"Prerequisites"},{"location":"en/advance/offload-mellanox/#setup-sr-iov","text":"Check the device ID of the NIC, in the following example it is 42:00.0 : # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] Find the corresponding NIC by its device ID: # ls -l /sys/class/net/ | grep 42:00.0 lrwxrwxrwx. 1 root root 0 Jul 22 23 :16 p4p1 -> ../../devices/pci0000:40/0000:40:02.0/0000:42:00.0/net/p4p1 Check the number of available VFs: # cat /sys/class/net/p4p1/device/sriov_totalvfs 8 Create VFs and do not exceeding the number found above: # echo '4' > /sys/class/net/p4p1/device/sriov_numvfs # ip link show p4p1 10 : p4p1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 link/ether b8:59:9f:c1:ec:12 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 1 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 2 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off vf 3 MAC 00 :00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off # ip link set p4p1 up Find the device IDs corresponding to the above VFs: # lspci -nn | grep ConnectX-5 42 :00.0 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.1 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 ] [ 15b3:1017 ] 42 :00.2 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.3 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.4 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] 42 :00.5 Ethernet controller [ 0200 ] : Mellanox Technologies MT27800 Family [ ConnectX-5 Virtual Function ] [ 15b3:1018 ] Unbound the VFs from the driver: echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind Enable eSwitch mode and set up hardware offload: devlink dev eswitch set pci/0000:42:00.0 mode switchdev ethtool -K enp66s0f0 hw-tc-offload on Rebind the driver and complete the VF setup: echo 0000 :42:00.2 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.3 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.4 > /sys/bus/pci/drivers/mlx5_core/bind echo 0000 :42:00.5 > /sys/bus/pci/drivers/mlx5_core/bind Some behaviors of NetworkManager may cause driver exceptions, if offloading problems occur we recommended to close NetworkManager and try again. systemctl stop NetworkManager systemctl disable NetworkManager","title":"Setup SR-IOV"},{"location":"en/advance/offload-mellanox/#install-sr-iov-device-plugin","text":"Since each machine has a limited number of VFs and each Pod that uses acceleration will take up VF resources, we need to use the SR-IOV Device Plugin to manage the corresponding resources so that the scheduler knows how to schedule. Create SR-IOV Configmap: apiVersion : v1 kind : ConfigMap metadata : name : sriovdp-config namespace : kube-system data : config.json : | { \"resourceList\": [{ \"resourcePrefix\": \"mellanox.com\", \"resourceName\": \"cx5_sriov_switchdev\", \"selectors\": { \"vendors\": [\"15b3\"], \"devices\": [\"1018\"], \"drivers\": [\"mlx5_core\"] } } ] } Please read the SR-IOV device plugin to deploy: kubectl apply -f https://raw.githubusercontent.com/intel/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml Check if SR-IOV resources have been registered to Kubernetes Node: kubectl describe node kube-ovn-01 | grep mellanox mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev: 4 mellanox.com/cx5_sriov_switchdev 0 0","title":"Install SR-IOV Device Plugin"},{"location":"en/advance/offload-mellanox/#install-multus-cni","text":"The device IDs obtained during SR-IOV Device Plugin scheduling need to be passed to Kube-OVN via Multus-CNI, so Multus-CNI needs to be configured to perform the related tasks. Please read Multus-CNI Document to deploy\uff1a kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml Create NetworkAttachmentDefinition \uff1a apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : default namespace : default annotations : k8s.v1.cni.cncf.io/resourceName : mellanox.com/cx5_sriov_switchdev spec : config : '{ \"cniVersion\": \"0.3.1\", \"name\": \"kube-ovn\", \"plugins\":[ { \"type\":\"kube-ovn\", \"server_socket\":\"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"default.default.ovn\" }, { \"type\":\"portmap\", \"capabilities\":{ \"portMappings\":true } } ] }' provider : the format should be {name}.{namespace}.ovn of related NetworkAttachmentDefinition .","title":"Install Multus-CNI"},{"location":"en/advance/offload-mellanox/#enable-offload-in-kube-ovn","text":"Download the scripts: wget https://raw.githubusercontent.com/alauda/kube-ovn/master/dist/images/install.sh Change the related options\uff0c IFACE should be the physic NIC and has an IP: ENABLE_MIRROR = ${ ENABLE_MIRROR :- false } HW_OFFLOAD = ${ HW_OFFLOAD :- true } ENABLE_LB = ${ ENABLE_LB :- false } IFACE = \"ensp01\" Install Kube-OVN\uff1a bash install.sh","title":"Enable Offload in Kube-OVN"},{"location":"en/advance/offload-mellanox/#create-pods-with-vf-nics","text":"Pods that use VF for network offload acceleration can be created using the following yaml: apiVersion : v1 kind : Pod metadata : name : nginx annotations : v1.multus-cni.io/default-network : default/default spec : containers : - name : nginx image : docker.io/library/nginx:alpine resources : requests : mellanox.com/cx5_sriov_switchdev : '1' limits : mellanox.com/cx5_sriov_switchdev : '1' v1.multus-cni.io/default-network : should be the {namespace}/{name} of related NetworkAttachmentDefinition . Running the following command in the ovs-ovn container of the Pod run node to observe if offload success. # ovs-appctl dpctl/dump-flows -m type=offloaded ufid:91cc45de-e7e9-4935-8f82-1890430b0f66, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 5b45c61b307e_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:c5:6d:4e,dst = 00 :00:00:e7:16:ce ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:941539, bytes:62142230, used:0.260s, offloaded:yes, dp:tc, actions:54235e5753b8_h ufid:e00768d7-e652-4d79-8182-3291d852b791, skb_priority ( 0 /0 ) ,skb_mark ( 0 /0 ) ,ct_state ( 0 /0x23 ) ,ct_zone ( 0 /0 ) ,ct_mark ( 0 /0 ) ,ct_label ( 0 /0x1 ) ,recirc_id ( 0 ) ,dp_hash ( 0 /0 ) ,in_port ( 54235e5753b8_h ) ,packet_type ( ns = 0 /0,id = 0 /0 ) ,eth ( src = 00 :00:00:e7:16:ce,dst = 00 :00:00:c5:6d:4e ) ,eth_type ( 0x0800 ) ,ipv4 ( src = 0 .0.0.0/0.0.0.0,dst = 0 .0.0.0/0.0.0.0,proto = 0 /0,tos = 0 /0,ttl = 0 /0,frag = no ) , packets:82386659, bytes:115944854173, used:0.260s, offloaded:yes, dp:tc, actions:5b45c61b307e_h If there is offloaded:yes, dp:tc content, the offloading is successful. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Create Pods with VF NICs"},{"location":"en/advance/overlay-with-route/","text":"Interconnection with Routes in Overlay Mode \u00b6 In some scenarios, the network environment does not support Underlay mode, but still need Pods and external devices directly access through IP, then you can use the routing method to connect the container network and the external. Only Overlay Subnets in default VPC support this method. In this case, the Pod IP goes directly to the underlying network, which needs to disable IP checks for source and destination addresses. Prerequisites \u00b6 In this mode, the host needs to open the ip_forward . Check if there is a Drop rule in the forward chain in the host iptables that should be modified for container-related traffic. Due to the possibility of asymmetric routing, the host needs to allow packets with a ct status of INVALID . Steps \u00b6 For subnets that require direct external routing, you need to set natOutgoing of the subnet to false to turn off nat mapping and make the Pod IP directly accessible to the external network. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : false At this point, the Pod's packets can reach the peer node via the host route, but the peer node does not yet know where the return packets should be sent to and needs to add a return route. If the peer host and the container host are on the same Layer 2 network, we can add a static route directly to the peer host to point the next hop of the container network to any machine in the Kubernetes cluster. ip route add 10 .166.0.0/16 via 192 .168.2.10 dev eth0 10.166.0.0/16 is the container subnet CIDR, and 192.168.2.10 is one node in the Kubernetes cluster. If the peer host and the container host are not in the same layer 2 network, you need to configure the corresponding rules on the router. In some virtualized environments, the virtual network identifies asymmetric traffic as illegal traffic and drops it. In this case, you need to adjust the gatewayType of the Subnet to centralized and set the next hop to the IP of the gatewayNode node during route setup. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1\" natOutgoing : false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Interconnection with Routes in Overlay Mode"},{"location":"en/advance/overlay-with-route/#interconnection-with-routes-in-overlay-mode","text":"In some scenarios, the network environment does not support Underlay mode, but still need Pods and external devices directly access through IP, then you can use the routing method to connect the container network and the external. Only Overlay Subnets in default VPC support this method. In this case, the Pod IP goes directly to the underlying network, which needs to disable IP checks for source and destination addresses.","title":"Interconnection with Routes in Overlay Mode"},{"location":"en/advance/overlay-with-route/#prerequisites","text":"In this mode, the host needs to open the ip_forward . Check if there is a Drop rule in the forward chain in the host iptables that should be modified for container-related traffic. Due to the possibility of asymmetric routing, the host needs to allow packets with a ct status of INVALID .","title":"Prerequisites"},{"location":"en/advance/overlay-with-route/#steps","text":"For subnets that require direct external routing, you need to set natOutgoing of the subnet to false to turn off nat mapping and make the Pod IP directly accessible to the external network. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : false At this point, the Pod's packets can reach the peer node via the host route, but the peer node does not yet know where the return packets should be sent to and needs to add a return route. If the peer host and the container host are on the same Layer 2 network, we can add a static route directly to the peer host to point the next hop of the container network to any machine in the Kubernetes cluster. ip route add 10 .166.0.0/16 via 192 .168.2.10 dev eth0 10.166.0.0/16 is the container subnet CIDR, and 192.168.2.10 is one node in the Kubernetes cluster. If the peer host and the container host are not in the same layer 2 network, you need to configure the corresponding rules on the router. In some virtualized environments, the virtual network identifies asymmetric traffic as illegal traffic and drops it. In this case, you need to adjust the gatewayType of the Subnet to centralized and set the next hop to the IP of the gatewayNode node during route setup. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : routed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1\" natOutgoing : false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Steps"},{"location":"en/advance/ovn-eip-fip-snat/","text":"Support OVN EIP,FIP and SNAT \u00b6 graph LR pod-->vpc1-subnet-->vpc1-->snat-->lrp-->external-subnet-->gw-node-external-nic The pod access the public network based on the snat graph LR pod-->vpc1-subnet-->vpc1-->fip-->lrp-->external-subnet-->local-node-external-nic The pod access the public network based on the fip The CRD supported by this function is basically the same as the iptable nat gw public network solution. ovn eip: occupies a public ip address and is allocated from the underlay provider network vlan subnet ovn fip: one-to-one dnat snat, which provides direct public network access for ip addresses and vip in a vpc ovn snat: a subnet cidr or a single vpc ip or vip can access public networks based on snat ovn dnat: based router lb, which enables direct access to a group of endpoints in a vpc based on a public endpoint 1. Deployment \u00b6 Currently allows all vpcs to share the same provider vlan subnet resources, similar to neutron ovn mode. Compatible with previous scenarios default VPC EIP/SNAT . During the deployment phase, you may need to specify a default public network logical switch based on actual conditions. If no vlan is in use (vlan 0 is used), the following startup parameters do not need to be configured. # When deploying you need to refer to the above scenario and specify the following parameters as needed according to the actual situation # 1. kube-ovn-controller Startup parameters to be configured\uff1a - --external-gateway-vlanid = 204 - --external-gateway-switch = external204 # 2. kube-ovn-cni Startup parameters to be configured: - --external-gateway-switch = external204 # The above configuration is consistent with the following public network configuration vlan id and resource name, # currently only support to specify one underlay public network as the default external public network. The design and use of this configuration item takes into account the following factors\uff1a Based on this configuration item can be docked to the provider network, vlan, subnet resources. Based on this configuration item, the default vpc enable_eip_snat function can be docked to the existing vlan, subnet resources, while supporting the ipam If only the default vpc's enable_eip_snat mode is used with the old pod annotaion based eip fip snat, then the following configuration is not required. Based on this configuration you can not use the default vpc enable_eip_snat process, only by corresponding to vlan, subnet process, can be compatible with only custom vpc use eip snat usage scenarios. The neutron ovn mode also has a certain static file configuration designation that is, for now, generally consistent. 1.1 Create the underlay public network \u00b6 # provider-network\uff0c vlan\uff0c subnet # cat 01-provider-network.yaml apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: external204 spec: defaultInterface: vlan # cat 02-vlan.yaml apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan204 spec: id: 204 provider: external204 # cat 03-vlan-subnet.yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: external204 spec: protocol: IPv4 cidrBlock: 10 .5.204.0/24 gateway: 10 .5.204.254 vlan: vlan204 excludeIps: - 10 .5.204.1..10.5.204.100 1.2 Default vpc enable eip_snat \u00b6 # Enable the default vpc and the above underlay public provider subnet interconnection cat 00 -centralized-external-gw-no-ip.yaml apiVersion: v1 kind: ConfigMap metadata: name: ovn-external-gw-config namespace: kube-system data: enable-external-gw: \"true\" external-gw-nodes: \"pc-node-1,pc-node-2,pc-node-3\" type: \"centralized\" external-gw-nic: \"vlan\" external-gw-addr: \"10.5.204.254/24\" This feature currently supports the ability to create lrp type ovn eip resources without specifying the lrp ip and mac, which is already supported for automatic acquisition. If specified, it is equivalent to specifying the ip to create an ovn-eip of type lrp. Of course, you can also manually create the lrp type ovn eip in advance. 1.3 Custom vpc enable eip snat fip function \u00b6 # cat 00-ns.yml apiVersion: v1 kind: Namespace metadata: name: vpc1 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true # vpc enableExternal will automatically create an lrp association to the public network specified above # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 After the above template is applied, you should see the following resources exist # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # The route currently supports automatic maintenance 2. ovn-eip \u00b6 This function is designed and used in the same way as iptables-eip, ovn-eip currently has three types nat: indicates ovn dnat, fip, and snat. These nat types are recorded in status lrp: indicates the resource used to connect a vpc to the public network lsp: In the ovn BFD-based ecmp static route scenario, an ovs internal port is provided on the gateway node as the next hop of the ecmp route --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: nat # Dynamically allocate an eip resource that is reserved for fip dnat_and_snat scenarios 2.1 Create an fip for pod \u00b6 # k get po -o wide -n vpc1 vpc-1-busybox01 NAME READY STATUS RESTARTS AGE IP NODE vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 # k get ip vpc-1-busybox01.vpc1 NAME V4IP V6IP MAC NODE SUBNET vpc-1-busybox01.vpc1 192 .168.0.2 00 :00:00:0A:DD:27 pc-node-2 vpc1-subnet1 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: nat --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: ovnEip: eip-static ipName: vpc-1-busybox01.vpc1 # the name of the ip crd, which is unique # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 # k get ofip eip-static NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 [ root@pc-node-1 03 -cust-vpc ] # ping 10.5.204.101 PING 10 .5.204.101 ( 10 .5.204.101 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.101: icmp_seq = 2 ttl = 62 time = 1 .21 ms 64 bytes from 10 .5.204.101: icmp_seq = 3 ttl = 62 time = 0 .624 ms 64 bytes from 10 .5.204.101: icmp_seq = 4 ttl = 62 time = 0 .368 ms ^C --- 10 .5.204.101 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3049ms rtt min/avg/max/mdev = 0 .368/0.734/1.210/0.352 ms [ root@pc-node-1 03 -cust-vpc ] # # pod <--> node ping is working # The key resources that this public ip can pass include the following ovn nb resources # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 813523e7-c68c-408f-bd8c-cba30cb2e4f4 external ip: \"10.5.204.101\" logical ip: \"192.168.0.2\" type: \"dnat_and_snat\" 2.2 Create an fip for vip \u00b6 In order to facilitate the use of some vip scenarios, such as inside kubevirt VM, keepalived use vip, kube-vip use vip, etc. the vip need public network access. # First create vip, eip, then bind eip to vip # cat vip.yaml apiVersion: kubeovn.io/v1 kind: Vip metadata: name: test-fip-vip spec: subnet: vpc1-subnet1 # cat 04-fip.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: externalSubnet: external204 type: nat --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: ovnEip: eip-for-vip ipType: vip # By default fip is for pod ip, here you need to specify the docking to vip resources ipName: test-fip-vip # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip [ root@pc-node-1 fip-vip ] # ping 10.5.204.106 PING 10 .5.204.106 ( 10 .5.204.106 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.106: icmp_seq = 1 ttl = 62 time = 0 .694 ms 64 bytes from 10 .5.204.106: icmp_seq = 2 ttl = 62 time = 0 .436 ms # node <--> pod fip is working # The way ip is used inside the pod is roughly as follows [ root@pc-node-1 fip-vip ] # k -n vpc1 exec -it vpc-1-busybox03 -- bash [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1568 : eth0@if1569: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:56:40:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.5/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 192 .168.0.3/24 scope global secondary eth0 # vip here valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe56:40e5/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox03 / ] # tcpdump -i eth0 host 192.168.0.3 -netvv tcpdump: listening on eth0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:ed:8e:c7 > 00 :00:00:56:40:e5, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 44830 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.32.51 > 192 .168.0.3: ICMP echo request, id 177 , seq 1 , length 64 00 :00:00:56:40:e5 > 00 :00:00:ed:8e:c7, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 64 , id 43962 , offset 0 , flags [ none ] , proto ICMP ( 1 ) , length 84 ) 192 .168.0.3 > 10 .5.32.51: ICMP echo reply, id 177 , seq 1 , length 64 # pod internal can catch fip related icmp packets 3. ovn-snat \u00b6 3.1 ovn-snat corresponds to the CIDR of a subnet \u00b6 This feature is designed and used in much the same way as iptables-snat # cat 03-subnet-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: externalSubnet: external204 type: nat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: ovnEip: snat-for-subnet-in-vpc vpcSubnet: vpc1-subnet1 # eip corresponds to the entire network segment 3.2 ovn-fip corresponds to a pod IP \u00b6 This feature is designed and used in much the same way as iptables-fip # cat 03-pod-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-pod-vpc-ip spec: externalSubnet: external204 type: nat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat01 spec: ovnEip: snat-for-pod-vpc-ip ipName: vpc-1-busybox02.vpc1 # eip corresponds to a single pod ip After the above resources are created, you can see the following resources that the snat public network feature depends on. # kubectl ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" nat da77a11f-c523-439c-b1d1-72c664196a0f external ip: \"10.5.204.116\" logical ip: \"192.168.0.4\" type: \"snat\" [ root@pc-node-1 03 -cust-vpc ] # k get po -A -o wide | grep busy vpc1 vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 <none> <none> vpc1 vpc-1-busybox02 1 /1 Running 0 17h 192 .168.0.4 pc-node-1 <none> <none> vpc1 vpc-1-busybox03 1 /1 Running 0 17h 192 .168.0.5 pc-node-1 <none> <none> vpc1 vpc-1-busybox04 1 /1 Running 0 17h 192 .168.0.6 pc-node-3 <none> <none> vpc1 vpc-1-busybox05 1 /1 Running 0 17h 192 .168.0.7 pc-node-1 <none> <none> # k exec -it -n vpc1 vpc-1-busybox04 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17095 : eth0@if17096: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:76:94:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.6/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe76:9455/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox04 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 114 time = 22 .2 ms 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 21 .8 ms [ root@pc-node-1 03 -cust-vpc ] # k exec -it -n vpc1 vpc-1-busybox02 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1566 : eth0@if1567: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:0b:e9:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.4/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe0b:e9d0/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox02 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 22 .7 ms 64 bytes from 223 .5.5.5: icmp_seq = 3 ttl = 114 time = 22 .6 ms 64 bytes from 223 .5.5.5: icmp_seq = 4 ttl = 114 time = 22 .1 ms ^C --- 223 .5.5.5 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3064ms rtt min/avg/max/mdev = 22 .126/22.518/22.741/0.278 ms # the two pods can access the external network based on these two type snat resources respectively 4. ovn-dnat \u00b6 4.1 ovn-dnat binds a DNAT to a pod \u00b6 kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : externalSubnet : underlay type : nat --- kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ovnEip : eip-dnat ipName : vpc-1-busybox01.vpc1 # Note that this is the name of the pod IP CRD and it is unique protocol : tcp internalPort : \"22\" externalPort : \"22\" The configuration of OvnDnatRule is similar to that of IptablesDnatRule. # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.3 22 22 vpc-1-busybox01.vpc1 true 4.2 ovn-dnat binds a DNAT to a VIP \u00b6 kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ipType : vip # By default, Dnat is oriented towards pod IPs. Here, it is necessary to specify that it is connected to VIP resources ovnEip : eip-dnat ipName : test-dnat-vip protocol : tcp internalPort : \"22\" externalPort : \"22\" The configuration of OvnDnatRule is similar to that of IptablesDnatRule. # kubectl get vip test-dnat-vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY test-dnat-vip 192 .168.0.4 00 :00:00:D0:C0:B5 vpc1-subnet1 true # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat eip-dnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.4 22 22 test-dnat-vip true \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Support OVN EIP,FIP and SNAT"},{"location":"en/advance/ovn-eip-fip-snat/#support-ovn-eipfip-and-snat","text":"graph LR pod-->vpc1-subnet-->vpc1-->snat-->lrp-->external-subnet-->gw-node-external-nic The pod access the public network based on the snat graph LR pod-->vpc1-subnet-->vpc1-->fip-->lrp-->external-subnet-->local-node-external-nic The pod access the public network based on the fip The CRD supported by this function is basically the same as the iptable nat gw public network solution. ovn eip: occupies a public ip address and is allocated from the underlay provider network vlan subnet ovn fip: one-to-one dnat snat, which provides direct public network access for ip addresses and vip in a vpc ovn snat: a subnet cidr or a single vpc ip or vip can access public networks based on snat ovn dnat: based router lb, which enables direct access to a group of endpoints in a vpc based on a public endpoint","title":"Support OVN EIP,FIP and SNAT"},{"location":"en/advance/ovn-eip-fip-snat/#1-deployment","text":"Currently allows all vpcs to share the same provider vlan subnet resources, similar to neutron ovn mode. Compatible with previous scenarios default VPC EIP/SNAT . During the deployment phase, you may need to specify a default public network logical switch based on actual conditions. If no vlan is in use (vlan 0 is used), the following startup parameters do not need to be configured. # When deploying you need to refer to the above scenario and specify the following parameters as needed according to the actual situation # 1. kube-ovn-controller Startup parameters to be configured\uff1a - --external-gateway-vlanid = 204 - --external-gateway-switch = external204 # 2. kube-ovn-cni Startup parameters to be configured: - --external-gateway-switch = external204 # The above configuration is consistent with the following public network configuration vlan id and resource name, # currently only support to specify one underlay public network as the default external public network. The design and use of this configuration item takes into account the following factors\uff1a Based on this configuration item can be docked to the provider network, vlan, subnet resources. Based on this configuration item, the default vpc enable_eip_snat function can be docked to the existing vlan, subnet resources, while supporting the ipam If only the default vpc's enable_eip_snat mode is used with the old pod annotaion based eip fip snat, then the following configuration is not required. Based on this configuration you can not use the default vpc enable_eip_snat process, only by corresponding to vlan, subnet process, can be compatible with only custom vpc use eip snat usage scenarios. The neutron ovn mode also has a certain static file configuration designation that is, for now, generally consistent.","title":"1. Deployment"},{"location":"en/advance/ovn-eip-fip-snat/#11-create-the-underlay-public-network","text":"# provider-network\uff0c vlan\uff0c subnet # cat 01-provider-network.yaml apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: external204 spec: defaultInterface: vlan # cat 02-vlan.yaml apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan204 spec: id: 204 provider: external204 # cat 03-vlan-subnet.yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: external204 spec: protocol: IPv4 cidrBlock: 10 .5.204.0/24 gateway: 10 .5.204.254 vlan: vlan204 excludeIps: - 10 .5.204.1..10.5.204.100","title":"1.1 Create the underlay public network"},{"location":"en/advance/ovn-eip-fip-snat/#12-default-vpc-enable-eip_snat","text":"# Enable the default vpc and the above underlay public provider subnet interconnection cat 00 -centralized-external-gw-no-ip.yaml apiVersion: v1 kind: ConfigMap metadata: name: ovn-external-gw-config namespace: kube-system data: enable-external-gw: \"true\" external-gw-nodes: \"pc-node-1,pc-node-2,pc-node-3\" type: \"centralized\" external-gw-nic: \"vlan\" external-gw-addr: \"10.5.204.254/24\" This feature currently supports the ability to create lrp type ovn eip resources without specifying the lrp ip and mac, which is already supported for automatic acquisition. If specified, it is equivalent to specifying the ip to create an ovn-eip of type lrp. Of course, you can also manually create the lrp type ovn eip in advance.","title":"1.2 Default vpc enable eip_snat"},{"location":"en/advance/ovn-eip-fip-snat/#13-custom-vpc-enable-eip-snat-fip-function","text":"# cat 00-ns.yml apiVersion: v1 kind: Namespace metadata: name: vpc1 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true # vpc enableExternal will automatically create an lrp association to the public network specified above # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 After the above template is applied, you should see the following resources exist # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # The route currently supports automatic maintenance","title":"1.3 Custom vpc enable eip snat fip function"},{"location":"en/advance/ovn-eip-fip-snat/#2-ovn-eip","text":"This function is designed and used in the same way as iptables-eip, ovn-eip currently has three types nat: indicates ovn dnat, fip, and snat. These nat types are recorded in status lrp: indicates the resource used to connect a vpc to the public network lsp: In the ovn BFD-based ecmp static route scenario, an ovs internal port is provided on the gateway node as the next hop of the ecmp route --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: nat # Dynamically allocate an eip resource that is reserved for fip dnat_and_snat scenarios","title":"2. ovn-eip"},{"location":"en/advance/ovn-eip-fip-snat/#21-create-an-fip-for-pod","text":"# k get po -o wide -n vpc1 vpc-1-busybox01 NAME READY STATUS RESTARTS AGE IP NODE vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 # k get ip vpc-1-busybox01.vpc1 NAME V4IP V6IP MAC NODE SUBNET vpc-1-busybox01.vpc1 192 .168.0.2 00 :00:00:0A:DD:27 pc-node-2 vpc1-subnet1 --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: externalSubnet: external204 type: nat --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-static spec: ovnEip: eip-static ipName: vpc-1-busybox01.vpc1 # the name of the ip crd, which is unique # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 # k get ofip eip-static NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-static vpc1 10 .5.204.101 192 .168.0.2 true vpc-1-busybox01.vpc1 [ root@pc-node-1 03 -cust-vpc ] # ping 10.5.204.101 PING 10 .5.204.101 ( 10 .5.204.101 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.101: icmp_seq = 2 ttl = 62 time = 1 .21 ms 64 bytes from 10 .5.204.101: icmp_seq = 3 ttl = 62 time = 0 .624 ms 64 bytes from 10 .5.204.101: icmp_seq = 4 ttl = 62 time = 0 .368 ms ^C --- 10 .5.204.101 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3049ms rtt min/avg/max/mdev = 0 .368/0.734/1.210/0.352 ms [ root@pc-node-1 03 -cust-vpc ] # # pod <--> node ping is working # The key resources that this public ip can pass include the following ovn nb resources # k ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 813523e7-c68c-408f-bd8c-cba30cb2e4f4 external ip: \"10.5.204.101\" logical ip: \"192.168.0.2\" type: \"dnat_and_snat\"","title":"2.1 Create an fip for pod"},{"location":"en/advance/ovn-eip-fip-snat/#22-create-an-fip-for-vip","text":"In order to facilitate the use of some vip scenarios, such as inside kubevirt VM, keepalived use vip, kube-vip use vip, etc. the vip need public network access. # First create vip, eip, then bind eip to vip # cat vip.yaml apiVersion: kubeovn.io/v1 kind: Vip metadata: name: test-fip-vip spec: subnet: vpc1-subnet1 # cat 04-fip.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: externalSubnet: external204 type: nat --- kind: OvnFip apiVersion: kubeovn.io/v1 metadata: name: eip-for-vip spec: ovnEip: eip-for-vip ipType: vip # By default fip is for pod ip, here you need to specify the docking to vip resources ipName: test-fip-vip # k get ofip NAME VPC V4EIP V4IP READY IPTYPE IPNAME eip-for-vip vpc1 10 .5.204.106 192 .168.0.3 true vip test-fip-vip [ root@pc-node-1 fip-vip ] # ping 10.5.204.106 PING 10 .5.204.106 ( 10 .5.204.106 ) 56 ( 84 ) bytes of data. 64 bytes from 10 .5.204.106: icmp_seq = 1 ttl = 62 time = 0 .694 ms 64 bytes from 10 .5.204.106: icmp_seq = 2 ttl = 62 time = 0 .436 ms # node <--> pod fip is working # The way ip is used inside the pod is roughly as follows [ root@pc-node-1 fip-vip ] # k -n vpc1 exec -it vpc-1-busybox03 -- bash [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # [ root@vpc-1-busybox03 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1568 : eth0@if1569: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:56:40:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.5/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet 192 .168.0.3/24 scope global secondary eth0 # vip here valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe56:40e5/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox03 / ] # tcpdump -i eth0 host 192.168.0.3 -netvv tcpdump: listening on eth0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:ed:8e:c7 > 00 :00:00:56:40:e5, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 44830 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.32.51 > 192 .168.0.3: ICMP echo request, id 177 , seq 1 , length 64 00 :00:00:56:40:e5 > 00 :00:00:ed:8e:c7, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 64 , id 43962 , offset 0 , flags [ none ] , proto ICMP ( 1 ) , length 84 ) 192 .168.0.3 > 10 .5.32.51: ICMP echo reply, id 177 , seq 1 , length 64 # pod internal can catch fip related icmp packets","title":"2.2 Create an fip for vip"},{"location":"en/advance/ovn-eip-fip-snat/#3-ovn-snat","text":"","title":"3. ovn-snat"},{"location":"en/advance/ovn-eip-fip-snat/#31-ovn-snat-corresponds-to-the-cidr-of-a-subnet","text":"This feature is designed and used in much the same way as iptables-snat # cat 03-subnet-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: externalSubnet: external204 type: nat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat-for-subnet-in-vpc spec: ovnEip: snat-for-subnet-in-vpc vpcSubnet: vpc1-subnet1 # eip corresponds to the entire network segment","title":"3.1 ovn-snat corresponds to the CIDR of a subnet"},{"location":"en/advance/ovn-eip-fip-snat/#32-ovn-fip-corresponds-to-a-pod-ip","text":"This feature is designed and used in much the same way as iptables-fip # cat 03-pod-snat.yaml --- kind: OvnEip apiVersion: kubeovn.io/v1 metadata: name: snat-for-pod-vpc-ip spec: externalSubnet: external204 type: nat --- kind: OvnSnatRule apiVersion: kubeovn.io/v1 metadata: name: snat01 spec: ovnEip: snat-for-pod-vpc-ip ipName: vpc-1-busybox02.vpc1 # eip corresponds to a single pod ip After the above resources are created, you can see the following resources that the snat public network feature depends on. # kubectl ko nbctl show vpc1 router 87ad06fd-71d5-4ff8-a1f0-54fa3bba1a7f ( vpc1 ) port vpc1-vpc1-subnet1 mac: \"00:00:00:ED:8E:C7\" networks: [ \"192.168.0.1/24\" ] port vpc1-external204 mac: \"00:00:00:EF:05:C7\" networks: [ \"10.5.204.105/24\" ] gateway chassis: [ 7cedd14f-265b-42e5-ac17-e03e7a1f2342 276baccb-fe9c-4476-b41d-05872a94976d fd9f140c-c45d-43db-a6c0-0d4f8ea298dd ] nat 21d853b0-f7b4-40bd-9a53-31d2e2745739 external ip: \"10.5.204.115\" logical ip: \"192.168.0.0/24\" type: \"snat\" nat da77a11f-c523-439c-b1d1-72c664196a0f external ip: \"10.5.204.116\" logical ip: \"192.168.0.4\" type: \"snat\" [ root@pc-node-1 03 -cust-vpc ] # k get po -A -o wide | grep busy vpc1 vpc-1-busybox01 1 /1 Running 0 3d15h 192 .168.0.2 pc-node-2 <none> <none> vpc1 vpc-1-busybox02 1 /1 Running 0 17h 192 .168.0.4 pc-node-1 <none> <none> vpc1 vpc-1-busybox03 1 /1 Running 0 17h 192 .168.0.5 pc-node-1 <none> <none> vpc1 vpc-1-busybox04 1 /1 Running 0 17h 192 .168.0.6 pc-node-3 <none> <none> vpc1 vpc-1-busybox05 1 /1 Running 0 17h 192 .168.0.7 pc-node-1 <none> <none> # k exec -it -n vpc1 vpc-1-busybox04 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # [ root@vpc-1-busybox04 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 17095 : eth0@if17096: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:76:94:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.6/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe76:9455/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox04 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 114 time = 22 .2 ms 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 21 .8 ms [ root@pc-node-1 03 -cust-vpc ] # k exec -it -n vpc1 vpc-1-busybox02 bash kubectl exec [ POD ] [ COMMAND ] is DEPRECATED and will be removed in a future version. Use kubectl exec [ POD ] -- [ COMMAND ] instead. [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # [ root@vpc-1-busybox02 / ] # ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1566 : eth0@if1567: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 00 :00:00:0b:e9:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192 .168.0.4/24 brd 192 .168.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe0b:e9d0/64 scope link valid_lft forever preferred_lft forever [ root@vpc-1-busybox02 / ] # ping 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 2 ttl = 114 time = 22 .7 ms 64 bytes from 223 .5.5.5: icmp_seq = 3 ttl = 114 time = 22 .6 ms 64 bytes from 223 .5.5.5: icmp_seq = 4 ttl = 114 time = 22 .1 ms ^C --- 223 .5.5.5 ping statistics --- 4 packets transmitted, 3 received, 25 % packet loss, time 3064ms rtt min/avg/max/mdev = 22 .126/22.518/22.741/0.278 ms # the two pods can access the external network based on these two type snat resources respectively","title":"3.2 ovn-fip corresponds to a pod IP"},{"location":"en/advance/ovn-eip-fip-snat/#4-ovn-dnat","text":"","title":"4. ovn-dnat"},{"location":"en/advance/ovn-eip-fip-snat/#41-ovn-dnat-binds-a-dnat-to-a-pod","text":"kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : externalSubnet : underlay type : nat --- kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ovnEip : eip-dnat ipName : vpc-1-busybox01.vpc1 # Note that this is the name of the pod IP CRD and it is unique protocol : tcp internalPort : \"22\" externalPort : \"22\" The configuration of OvnDnatRule is similar to that of IptablesDnatRule. # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.3 22 22 vpc-1-busybox01.vpc1 true","title":"4.1 ovn-dnat binds a DNAT to a pod"},{"location":"en/advance/ovn-eip-fip-snat/#42-ovn-dnat-binds-a-dnat-to-a-vip","text":"kind : OvnDnatRule apiVersion : kubeovn.io/v1 metadata : name : eip-dnat spec : ipType : vip # By default, Dnat is oriented towards pod IPs. Here, it is necessary to specify that it is connected to VIP resources ovnEip : eip-dnat ipName : test-dnat-vip protocol : tcp internalPort : \"22\" externalPort : \"22\" The configuration of OvnDnatRule is similar to that of IptablesDnatRule. # kubectl get vip test-dnat-vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY test-dnat-vip 192 .168.0.4 00 :00:00:D0:C0:B5 vpc1-subnet1 true # kubectl get oeip eip-dnat NAME V4IP V6IP MAC TYPE READY eip-dnat 10 .5.49.4 00 :00:00:4D:CE:49 dnat true # kubectl get odnat eip-dnat NAME EIP PROTOCOL V4EIP V4IP INTERNALPORT EXTERNALPORT IPNAME READY eip-dnat eip-dnat tcp 10 .5.49.4 192 .168.0.4 22 22 test-dnat-vip true \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"4.2 ovn-dnat binds a DNAT to a VIP"},{"location":"en/advance/ovn-ipsec/","text":"Encrypt inter-node communication using IPsec \u00b6 This function is supported after v1.10.11 and v1.11.4, and the kernel version is at least 3.10.0 or above. start IPsec \u00b6 Copy the script from the Kube-OVN source code ipsec.sh , execute the command as follows, the script will call ovs-pki to generate and distribute the certificate required for encryption: bash ipsec.sh init After the execution is completed, the nodes will negotiate for a period of time to establish an IPsec tunnel. The experience value is between ten seconds and one minute.You can check the IPsec status with the following command: # bash ipsec.sh status Pod { ovs-ovn-d7hdt } ipsec status... Interface name: ovn-a4718e-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.2 Remote IP: 172 .18.0.4 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem Remote cert: None Remote name: a4718e55-5b85-4f46-90e6-63527d080590 CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 2 CFM state: Disabled Kernel policies installed: src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 Kernel security associations installed: sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 IPsec connections that are active: Pod { ovs-ovn-fvbbj } ipsec status... Interface name: ovn-8aebd9-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.4 Remote IP: 172 .18.0.2 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem Local name: a4718e55-5b85-4f46-90e6-63527d080590 Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem Remote cert: None Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 1 CFM state: Disabled Kernel policies installed: src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 Kernel security associations installed: sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 IPsec connections that are active: After the establishment is complete, you can capture packets and observe that the packets have been encrypted: # tcpdump -i eth0 -nel esp 10 :01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP ( spi = 0xcc91322a,seq = 0x13d0 ) , length 156 10 :01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP ( spi = 0xc8df4221,seq = 0x1d37 ) , length 156 After executing the script, you can turn off IPsec by executing the command: # bash ipsec.sh stop Or execute the command to open it again: # bash ipsec.sh start \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN IPsec Support"},{"location":"en/advance/ovn-ipsec/#encrypt-inter-node-communication-using-ipsec","text":"This function is supported after v1.10.11 and v1.11.4, and the kernel version is at least 3.10.0 or above.","title":"Encrypt inter-node communication using IPsec"},{"location":"en/advance/ovn-ipsec/#start-ipsec","text":"Copy the script from the Kube-OVN source code ipsec.sh , execute the command as follows, the script will call ovs-pki to generate and distribute the certificate required for encryption: bash ipsec.sh init After the execution is completed, the nodes will negotiate for a period of time to establish an IPsec tunnel. The experience value is between ten seconds and one minute.You can check the IPsec status with the following command: # bash ipsec.sh status Pod { ovs-ovn-d7hdt } ipsec status... Interface name: ovn-a4718e-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.2 Remote IP: 172 .18.0.4 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/8aebd9df-46ef-47b9-85e3-73e9a765296d-cert.pem Local name: 8aebd9df-46ef-47b9-85e3-73e9a765296d Local key: /etc/ipsec.d/private/8aebd9df-46ef-47b9-85e3-73e9a765296d-privkey.pem Remote cert: None Remote name: a4718e55-5b85-4f46-90e6-63527d080590 CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 2 CFM state: Disabled Kernel policies installed: src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 Kernel security associations installed: sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 IPsec connections that are active: Pod { ovs-ovn-fvbbj } ipsec status... Interface name: ovn-8aebd9-0 v1 ( CONFIGURED ) Tunnel Type: geneve Local IP: 172 .18.0.4 Remote IP: 172 .18.0.2 Address Family: IPv4 SKB mark: None Local cert: /etc/ipsec.d/certs/a4718e55-5b85-4f46-90e6-63527d080590-cert.pem Local name: a4718e55-5b85-4f46-90e6-63527d080590 Local key: /etc/ipsec.d/private/a4718e55-5b85-4f46-90e6-63527d080590-privkey.pem Remote cert: None Remote name: 8aebd9df-46ef-47b9-85e3-73e9a765296d CA cert: /etc/ipsec.d/cacerts/cacert.pem PSK: None Custom Options: {} Ofport: 1 CFM state: Disabled Kernel policies installed: src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 Kernel security associations installed: sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp dport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp sport 6081 sel src 172 .18.0.4/32 dst 172 .18.0.2/32 proto udp sport 6081 sel src 172 .18.0.2/32 dst 172 .18.0.4/32 proto udp dport 6081 IPsec connections that are active: After the establishment is complete, you can capture packets and observe that the packets have been encrypted: # tcpdump -i eth0 -nel esp 10 :01:40.349896 IP kube-ovn-worker > kube-ovn-control-plane.kind: ESP ( spi = 0xcc91322a,seq = 0x13d0 ) , length 156 10 :01:40.350015 IP kube-ovn-control-plane.kind > kube-ovn-worker: ESP ( spi = 0xc8df4221,seq = 0x1d37 ) , length 156 After executing the script, you can turn off IPsec by executing the command: # bash ipsec.sh stop Or execute the command to open it again: # bash ipsec.sh start \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"start IPsec"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/","text":"Support OVN SNAT L3 HA Based ECMP and BFD Static Route \u00b6 Custom vpc based on ovn snat after ecmp based static route hash to multiple gw node ovnext0 NICs out of the public network Supports bfd-based high availability Only supports hash load balancing graph LR pod-->vpc-subnet-->vpc-->snat-->ecmp-->external-subnet-->gw-node1-ovnext0--> node1-external-switch external-subnet-->gw-node2-ovnext0--> node2-external-switch external-subnet-->gw-node3-ovnext0--> node3-external-switch This functions basically the same as ovn-eip-fip-snat.md . As for the different parts, which will be specified in the following sections, mainly including the creation of ovn-eip of lsp type and the automatic maintenance of bfd as well as ecmp static routes based on vpc enable_bfd. 1. Deployment \u00b6 1.1 Create the underlay public network \u00b6 1.2 Default vpc enable eip_snat \u00b6 1.3 Custom vpc enable eip snat fip function \u00b6 The above section is exactly the same with ovn-eip-fip-snat.md . After these functions are verified, the vpc can be switched directly to the ecmp-based bfd static route based on the following way, or of course, switched directly back. Before customizing vpc to use this feature, you need to provide some gateway nodes, at least 2. Note that the name of the current implementation of ovn-eip must be consistent with the gateway node name, no automated maintenance is currently done for this resource. # cat gw-node-eip.yaml --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-1 spec : externalSubnet : external204 type : lsp --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-2 spec : externalSubnet : external204 type : lsp --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-3 spec : externalSubnet : external204 type : lsp Since this scenario is currently designed for vpc ecmp out of the public network, the gateway node above will not trigger the creation of a gateway NIC when there is no vpc enabled bfd, i.e. when there is no ovn eip (lrp) with enable bfd labeled, and will not be able to successfully start listening to the bfd session on the other side. 2. Custom vpc enable ecmp bfd L3 HA public network function \u00b6 # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true enableBfd: true # bfd switch can be switched at will #enableBfd: false # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true # enable ecmp gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 note: Customize ecmp under vpc to use only static ecmp bfd routes. vpc enableBfd and subnet enableEcmp will only take effect if they are enabled at the same time, before static ecmp bfd routes are automatically managed. If the above configuration is turned off, it will automatically switch back to the regular default static route. This feature is not available for the default vpc, only custom vpc is supported, the default vpc has more complex policy routing. The enableEcmp of the subnet of the custom vpc uses only static routes, the gateway type gatewayType has no effect. When EnableExternal is turned off in vpc, the external network cannot be passed inside vpc. When EnableExternal is enabled on vpc, when EnableBfd is turned off, it will be based on the normal default route to the external network and will not have high availability. # After the above template is applied the ovn logic layer should see the following resources # k get vpc NAME ENABLEEXTERNAL ENABLEBFD STANDBY SUBNETS NAMESPACES ovn-cluster true true [ \"external204\" , \"join\" , \"ovn-default\" ] vpc1 true true true [ \"vpc1-subnet1\" ] [ \"vpc1\" ] # Default vpc does not support ENABLEBFD # Custom vpc is supported and enabled # 1. bfd table created # k ko nbctl list bfd _uuid : be7df545-2c4c-4751-878f-b3507987f050 detect_mult : 3 dst_ip : \"10.5.204.121\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : 684c4489-5b59-4693-8d8c-3beab93f8093 detect_mult : 3 dst_ip : \"10.5.204.109\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : f0f62077-2ae9-4e79-b4f8-a446ec6e784c detect_mult : 3 dst_ip : \"10.5.204.108\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up ### Note that all statuses should normally be up # 2. bfd ecmp static routes table created # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 192 .168.0.0/24 10 .5.204.108 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.109 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.121 src-ip ecmp ecmp-symmetric-reply bfd # 3. Static Route Details # k ko nbctl find Logical_Router_Static_Route policy=src-ip options=ecmp_symmetric_reply=\"true\" _uuid : 3aacb384-d5ee-4b14-aebf-59e8c11717ba bfd : 684c4489-5b59-4693-8d8c-3beab93f8093 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.109\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 18bcc585-bc05-430b-925b-ef673c8e1aef bfd : f0f62077-2ae9-4e79-b4f8-a446ec6e784c external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.108\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 7d0a4e6b-cde0-4110-8176-fbaf19738498 bfd : be7df545-2c4c-4751-878f-b3507987f050 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.121\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" # Also, the following resources should be available at all gateway nodes [ root@pc-node-1 ~ ] # ip netns exec ovnext bash ip a /usr/sbin/ip: /usr/sbin/ip: cannot execute binary file [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1541 : ovnext0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 00 :00:00:ab:bd:87 brd ff:ff:ff:ff:ff:ff inet 10 .5.204.108/24 brd 10 .5.204.255 scope global ovnext0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:feab:bd87/64 scope link valid_lft forever preferred_lft forever [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0 .0.0.0 10 .5.204.254 0 .0.0.0 UG 0 0 0 ovnext0 10 .5.204.0 0 .0.0.0 255 .255.255.0 U 0 0 0 ovnext0 [ root@pc-node-1 ~ ] # ip netns exec ovnext bfdd-control status There are 1 sessions: Session 1 id = 1 local = 10 .5.204.108 ( p ) remote = 10 .5.204.122 state = Up ## This is the other end of the lrp bfd session and one of the next hops of the lrp ecmp [ root@pc-node-1 ~ ] # ip netns exec ovnext ping -c1 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 115 time = 21 .6 ms # No problem to the public network catch outgoing packets within the ovnext ns of a gateway node # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-1 ~ ] # exit [ root@pc-node-1 ~ ] # ssh pc-node-2 Last login: Thu Feb 23 09 :21:08 2023 from 10 .5.32.51 [ root@pc-node-2 ~ ] # ip netns exec ovnext bash [ root@pc-node-2 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-2 ~ ] # exit [ root@pc-node-2 ~ ] # logout Connection to pc-node-2 closed. [ root@pc-node-1 ~ ] # ssh pc-node-3 Last login: Thu Feb 23 08 :32:41 2023 from 10 .5.32.51 [ root@pc-node-3 ~ ] # ip netns exec ovnext bash [ root@pc-node-3 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:2d:f8:ce > 00 :00:00:fd:b2:a4, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 63 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 00 :00:00:fd:b2:a4 > dc:ef:80:5a:44:1a, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [ root@pc-node-3 ~ ] # 3. Turn off bfd mode \u00b6 In some scenarios, you may want to use a (centralized) single gateway directly out of the public network, which is the same as the default vpc enable_eip_snat usage pattern # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc2 spec: namespaces: - vpc2 enableExternal: true #enableBfd: true enableBfd: false ## set it false add apply # k ko nbctl lr-route-list vpc2 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # After application the route will switch back to the normal default static route # nbctl list bfd, the bfd session associated with lrp has been removed # And the opposite side of the bfd session in ovnext ns is automatically removed \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Support OVN SNAT L3 HA Based ECMP and BFD Static Route"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#support-ovn-snat-l3-ha-based-ecmp-and-bfd-static-route","text":"Custom vpc based on ovn snat after ecmp based static route hash to multiple gw node ovnext0 NICs out of the public network Supports bfd-based high availability Only supports hash load balancing graph LR pod-->vpc-subnet-->vpc-->snat-->ecmp-->external-subnet-->gw-node1-ovnext0--> node1-external-switch external-subnet-->gw-node2-ovnext0--> node2-external-switch external-subnet-->gw-node3-ovnext0--> node3-external-switch This functions basically the same as ovn-eip-fip-snat.md . As for the different parts, which will be specified in the following sections, mainly including the creation of ovn-eip of lsp type and the automatic maintenance of bfd as well as ecmp static routes based on vpc enable_bfd.","title":"Support OVN SNAT L3 HA Based ECMP and BFD Static Route"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#1-deployment","text":"","title":"1. Deployment"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#11-create-the-underlay-public-network","text":"","title":"1.1 Create the underlay public network"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#12-default-vpc-enable-eip_snat","text":"","title":"1.2 Default vpc enable eip_snat"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#13-custom-vpc-enable-eip-snat-fip-function","text":"The above section is exactly the same with ovn-eip-fip-snat.md . After these functions are verified, the vpc can be switched directly to the ecmp-based bfd static route based on the following way, or of course, switched directly back. Before customizing vpc to use this feature, you need to provide some gateway nodes, at least 2. Note that the name of the current implementation of ovn-eip must be consistent with the gateway node name, no automated maintenance is currently done for this resource. # cat gw-node-eip.yaml --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-1 spec : externalSubnet : external204 type : lsp --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-2 spec : externalSubnet : external204 type : lsp --- kind : OvnEip apiVersion : kubeovn.io/v1 metadata : name : pc-node-3 spec : externalSubnet : external204 type : lsp Since this scenario is currently designed for vpc ecmp out of the public network, the gateway node above will not trigger the creation of a gateway NIC when there is no vpc enabled bfd, i.e. when there is no ovn eip (lrp) with enable bfd labeled, and will not be able to successfully start listening to the bfd session on the other side.","title":"1.3 Custom vpc enable eip snat fip function"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#2-custom-vpc-enable-ecmp-bfd-l3-ha-public-network-function","text":"# cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc1 spec: namespaces: - vpc1 enableExternal: true enableBfd: true # bfd switch can be switched at will #enableBfd: false # cat 02-subnet.yml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: vpc1-subnet1 spec: cidrBlock: 192 .168.0.0/24 default: false disableGatewayCheck: false disableInterConnection: true enableEcmp: true # enable ecmp gatewayNode: \"\" gatewayType: distributed #gatewayType: centralized natOutgoing: false private: false protocol: IPv4 provider: ovn vpc: vpc1 namespaces: - vpc1 note: Customize ecmp under vpc to use only static ecmp bfd routes. vpc enableBfd and subnet enableEcmp will only take effect if they are enabled at the same time, before static ecmp bfd routes are automatically managed. If the above configuration is turned off, it will automatically switch back to the regular default static route. This feature is not available for the default vpc, only custom vpc is supported, the default vpc has more complex policy routing. The enableEcmp of the subnet of the custom vpc uses only static routes, the gateway type gatewayType has no effect. When EnableExternal is turned off in vpc, the external network cannot be passed inside vpc. When EnableExternal is enabled on vpc, when EnableBfd is turned off, it will be based on the normal default route to the external network and will not have high availability. # After the above template is applied the ovn logic layer should see the following resources # k get vpc NAME ENABLEEXTERNAL ENABLEBFD STANDBY SUBNETS NAMESPACES ovn-cluster true true [ \"external204\" , \"join\" , \"ovn-default\" ] vpc1 true true true [ \"vpc1-subnet1\" ] [ \"vpc1\" ] # Default vpc does not support ENABLEBFD # Custom vpc is supported and enabled # 1. bfd table created # k ko nbctl list bfd _uuid : be7df545-2c4c-4751-878f-b3507987f050 detect_mult : 3 dst_ip : \"10.5.204.121\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : 684c4489-5b59-4693-8d8c-3beab93f8093 detect_mult : 3 dst_ip : \"10.5.204.109\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up _uuid : f0f62077-2ae9-4e79-b4f8-a446ec6e784c detect_mult : 3 dst_ip : \"10.5.204.108\" external_ids : {} logical_port : vpc1-external204 min_rx : 100 min_tx : 100 options : {} status : up ### Note that all statuses should normally be up # 2. bfd ecmp static routes table created # k ko nbctl lr-route-list vpc1 IPv4 Routes Route Table <main>: 192 .168.0.0/24 10 .5.204.108 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.109 src-ip ecmp ecmp-symmetric-reply bfd 192 .168.0.0/24 10 .5.204.121 src-ip ecmp ecmp-symmetric-reply bfd # 3. Static Route Details # k ko nbctl find Logical_Router_Static_Route policy=src-ip options=ecmp_symmetric_reply=\"true\" _uuid : 3aacb384-d5ee-4b14-aebf-59e8c11717ba bfd : 684c4489-5b59-4693-8d8c-3beab93f8093 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.109\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 18bcc585-bc05-430b-925b-ef673c8e1aef bfd : f0f62077-2ae9-4e79-b4f8-a446ec6e784c external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.108\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" _uuid : 7d0a4e6b-cde0-4110-8176-fbaf19738498 bfd : be7df545-2c4c-4751-878f-b3507987f050 external_ids : {} ip_prefix : \"192.168.0.0/24\" nexthop : \"10.5.204.121\" options : { ecmp_symmetric_reply = \"true\" } output_port : [] policy : src-ip route_table : \"\" # Also, the following resources should be available at all gateway nodes [ root@pc-node-1 ~ ] # ip netns exec ovnext bash ip a /usr/sbin/ip: /usr/sbin/ip: cannot execute binary file [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext ip a 1 : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00 :00:00:00:00:00 brd 00 :00:00:00:00:00 inet 127 .0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1541 : ovnext0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 00 :00:00:ab:bd:87 brd ff:ff:ff:ff:ff:ff inet 10 .5.204.108/24 brd 10 .5.204.255 scope global ovnext0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:feab:bd87/64 scope link valid_lft forever preferred_lft forever [ root@pc-node-1 ~ ] # [ root@pc-node-1 ~ ] # ip netns exec ovnext route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0 .0.0.0 10 .5.204.254 0 .0.0.0 UG 0 0 0 ovnext0 10 .5.204.0 0 .0.0.0 255 .255.255.0 U 0 0 0 ovnext0 [ root@pc-node-1 ~ ] # ip netns exec ovnext bfdd-control status There are 1 sessions: Session 1 id = 1 local = 10 .5.204.108 ( p ) remote = 10 .5.204.122 state = Up ## This is the other end of the lrp bfd session and one of the next hops of the lrp ecmp [ root@pc-node-1 ~ ] # ip netns exec ovnext ping -c1 223.5.5.5 PING 223 .5.5.5 ( 223 .5.5.5 ) 56 ( 84 ) bytes of data. 64 bytes from 223 .5.5.5: icmp_seq = 1 ttl = 115 time = 21 .6 ms # No problem to the public network catch outgoing packets within the ovnext ns of a gateway node # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-1 ~ ] # exit [ root@pc-node-1 ~ ] # ssh pc-node-2 Last login: Thu Feb 23 09 :21:08 2023 from 10 .5.32.51 [ root@pc-node-2 ~ ] # ip netns exec ovnext bash [ root@pc-node-2 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel [ root@pc-node-2 ~ ] # exit [ root@pc-node-2 ~ ] # logout Connection to pc-node-2 closed. [ root@pc-node-1 ~ ] # ssh pc-node-3 Last login: Thu Feb 23 08 :32:41 2023 from 10 .5.32.51 [ root@pc-node-3 ~ ] # ip netns exec ovnext bash [ root@pc-node-3 ~ ] # tcpdump -i ovnext0 host 223.5.5.5 -netvv dropped privs to tcpdump tcpdump: listening on ovnext0, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 00 :00:00:2d:f8:ce > 00 :00:00:fd:b2:a4, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 63 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 00 :00:00:fd:b2:a4 > dc:ef:80:5a:44:1a, ethertype IPv4 ( 0x0800 ) , length 98 : ( tos 0x0, ttl 62 , id 57978 , offset 0 , flags [ DF ] , proto ICMP ( 1 ) , length 84 ) 10 .5.204.102 > 223 .5.5.5: ICMP echo request, id 22 , seq 71 , length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [ root@pc-node-3 ~ ] #","title":"2. Custom vpc enable ecmp bfd L3 HA public network function"},{"location":"en/advance/ovn-l3-ha-based-ecmp-with-bfd/#3-turn-off-bfd-mode","text":"In some scenarios, you may want to use a (centralized) single gateway directly out of the public network, which is the same as the default vpc enable_eip_snat usage pattern # cat 01-vpc-ecmp-enable-external-bfd.yml kind: Vpc apiVersion: kubeovn.io/v1 metadata: name: vpc2 spec: namespaces: - vpc2 enableExternal: true #enableBfd: true enableBfd: false ## set it false add apply # k ko nbctl lr-route-list vpc2 IPv4 Routes Route Table <main>: 0 .0.0.0/0 10 .5.204.254 dst-ip # After application the route will switch back to the normal default static route # nbctl list bfd, the bfd session associated with lrp has been removed # And the opposite side of the bfd session in ovnext ns is automatically removed \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"3. Turn off bfd mode"},{"location":"en/advance/ovn-remote-port-mirroring/","text":"OVN Remote Port Mirroring \u00b6 This feature provides ability to mirror the traffic of the specified Pod and direction, and to send the mirrored traffic to a remote destination. This feature requires Kube-OVN version not lower than v1.12. Install Multus-CNI \u00b6 Install Multus-CNI by referring the Multus-CNI Document . Create NetworkAttachmentDefinition \u00b6 Create the following NetworkAttachmentDefinition: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : | { \"cniVersion\": \"0.3.1\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" } Format of the provider field is <NAME>.<NAMESPACE>.ovn . Create Underlay Network \u00b6 The mirrored traffic is encapsulated before transmition, so MTU of the network used to transmit the traffic should be greater than the mirrored LSP/Pod. Here we are using an underlay network. Create the following underlay network: apiVersion : kubeovn.io/v1 kind : ProviderNetwork metadata : name : net1 spec : defaultInterface : eth1 --- apiVersion : kubeovn.io/v1 kind : Vlan metadata : name : vlan1 spec : id : 0 provider : net1 --- apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.19.0.0/16 excludeIps : - 172.19.0.2..172.19.0.20 gateway : 172.19.0.1 vlan : vlan1 provider : attachnet.default.ovn The subnet's provider MUST be the same as the provider of the NetworkAttachmentDefinition created above. Create Receiving Pod \u00b6 Create the following Pod\uff1a apiVersion : v1 kind : Pod metadata : name : pod1 annotations : k8s.v1.cni.cncf.io/networks : default/attachnet spec : containers : - name : bash image : docker.io/kubeovn/kube-ovn:v1.12.0 args : - bash - -c - sleep infinity securityContext : privileged : true After the Pod has been created, checkout the IP addresses: $ kubectl get ips | grep pod1 pod1.default 10 .16.0.12 00 :00:00:FF:34:24 kube-ovn-worker ovn-default pod1.default.attachnet.default.ovn 172 .19.0.21 00 :00:00:A0:30:68 kube-ovn-worker subnet1 The IP address 172.19.0.21 will be used later. Create OVN Remote Port Mirroring \u00b6 Create the following OVN remote port mirroring\uff1a kubectl ko nbctl mirror-add mirror1 gre 99 from-lport 172 .19.0.21 kubectl ko nbctl lsp-attach-mirror coredns-787d4945fb-gpnkb.kube-system mirror1 coredns-787d4945fb-gpnkb.kube-system is the OVN LSP name with a format <POD_NAME>.<POD_NAMESPACE> . Here is the OVN command usage: ovn-nbctl mirror-add <NAME> <TYPE> <INDEX> <FILTER> <IP> NAME - add a mirror with given name TYPE - specify TYPE 'gre' or 'erspan' INDEX - specify the tunnel INDEX value (indicates key if GRE, erpsan_idx if ERSPAN) FILTER - specify FILTER for mirroring selection ('to-lport' / 'from-lport') IP - specify Sink / Destination i.e. Remote IP ovn-nbctl mirror-del [NAME] remove mirrors ovn-nbctl mirror-list print mirrors ovn-nbctl lsp-attach-mirror PORT MIRROR attach source PORT to MIRROR ovn-nbctl lsp-detach-mirror PORT MIRROR detach source PORT from MIRROR Configure Receiving Pod \u00b6 Execute the following commands in the Pod: root@pod1:/kube-ovn# ip link add mirror1 type gretap local 172 .19.0.21 key 99 dev net1 root@pod1:/kube-ovn# ip link set mirror1 up Now you can capture the mirrored packets: root@pod1:/kube-ovn# tcpdump -i mirror1 -nnve tcpdump: listening on mirror1, link-type EN10MB ( Ethernet ) , snapshot length 262144 bytes 05 :13:30.328808 00 :00:00:a3:f5:e2 > 00 :00:00:97:0f:6e, ethertype ARP ( 0x0806 ) , length 42 : Ethernet ( len 6 ) , IPv4 ( len 4 ) , Request who-has 10 .16.0.7 tell 10 .16.0.4, length 28 05 :13:30.559167 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57364 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.50472: 34511 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.559343 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57365 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.45177: 1659 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.560625 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 200 : ( tos 0x0, ttl 64 , id 57367 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 186 ) 10 .16.0.4.53 > 10 .16.0.6.43848: 2636 *- 0 /1/1 ( 158 ) 05 :13:30.562774 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 191 : ( tos 0x0, ttl 64 , id 57368 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 177 ) 10 .16.0.4.53 > 10 .16.0.6.37755: 48737 NXDomain*- 0 /1/1 ( 149 ) 05 :13:30.563523 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 187 : ( tos 0x0, ttl 64 , id 57369 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 173 ) 10 .16.0.4.53 > 10 .16.0.6.53887: 45519 NXDomain*- 0 /1/1 ( 145 ) 05 :13:30.564940 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57370 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.40846: 25745 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.565140 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57371 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.45214: 61875 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.566023 00 :00:00:a3:f5:e2 > 00 :00:00:55:e4:4e, ethertype IPv4 ( 0x0800 ) , length 80 : ( tos 0x0, ttl 64 , id 45937 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 66 ) 10 .16.0.4.44116 > 172 .18.0.1.53: 16025 + [ 1au ] AAAA? alauda.cn. ( 38 ) Notice \u00b6 If you are using ERSPAN as the encapsulation protocol, the Linux kernel version of the OVN nodes and remote devices must not be lower than 4.14. If you are using ERSPAN as the encapsulation protocol and using IPv6 as the transport network, the Linux kernel version must not be lower than 4.16. The transmission of mirrored traffic is unidirectional, so you only need to ensure that the OVN node can access the remote device. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN Remote Port Mirroring"},{"location":"en/advance/ovn-remote-port-mirroring/#ovn-remote-port-mirroring","text":"This feature provides ability to mirror the traffic of the specified Pod and direction, and to send the mirrored traffic to a remote destination. This feature requires Kube-OVN version not lower than v1.12.","title":"OVN Remote Port Mirroring"},{"location":"en/advance/ovn-remote-port-mirroring/#install-multus-cni","text":"Install Multus-CNI by referring the Multus-CNI Document .","title":"Install Multus-CNI"},{"location":"en/advance/ovn-remote-port-mirroring/#create-networkattachmentdefinition","text":"Create the following NetworkAttachmentDefinition: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : attachnet namespace : default spec : config : | { \"cniVersion\": \"0.3.1\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"attachnet.default.ovn\" } Format of the provider field is <NAME>.<NAMESPACE>.ovn .","title":"Create NetworkAttachmentDefinition"},{"location":"en/advance/ovn-remote-port-mirroring/#create-underlay-network","text":"The mirrored traffic is encapsulated before transmition, so MTU of the network used to transmit the traffic should be greater than the mirrored LSP/Pod. Here we are using an underlay network. Create the following underlay network: apiVersion : kubeovn.io/v1 kind : ProviderNetwork metadata : name : net1 spec : defaultInterface : eth1 --- apiVersion : kubeovn.io/v1 kind : Vlan metadata : name : vlan1 spec : id : 0 provider : net1 --- apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.19.0.0/16 excludeIps : - 172.19.0.2..172.19.0.20 gateway : 172.19.0.1 vlan : vlan1 provider : attachnet.default.ovn The subnet's provider MUST be the same as the provider of the NetworkAttachmentDefinition created above.","title":"Create Underlay Network"},{"location":"en/advance/ovn-remote-port-mirroring/#create-receiving-pod","text":"Create the following Pod\uff1a apiVersion : v1 kind : Pod metadata : name : pod1 annotations : k8s.v1.cni.cncf.io/networks : default/attachnet spec : containers : - name : bash image : docker.io/kubeovn/kube-ovn:v1.12.0 args : - bash - -c - sleep infinity securityContext : privileged : true After the Pod has been created, checkout the IP addresses: $ kubectl get ips | grep pod1 pod1.default 10 .16.0.12 00 :00:00:FF:34:24 kube-ovn-worker ovn-default pod1.default.attachnet.default.ovn 172 .19.0.21 00 :00:00:A0:30:68 kube-ovn-worker subnet1 The IP address 172.19.0.21 will be used later.","title":"Create Receiving Pod"},{"location":"en/advance/ovn-remote-port-mirroring/#create-ovn-remote-port-mirroring","text":"Create the following OVN remote port mirroring\uff1a kubectl ko nbctl mirror-add mirror1 gre 99 from-lport 172 .19.0.21 kubectl ko nbctl lsp-attach-mirror coredns-787d4945fb-gpnkb.kube-system mirror1 coredns-787d4945fb-gpnkb.kube-system is the OVN LSP name with a format <POD_NAME>.<POD_NAMESPACE> . Here is the OVN command usage: ovn-nbctl mirror-add <NAME> <TYPE> <INDEX> <FILTER> <IP> NAME - add a mirror with given name TYPE - specify TYPE 'gre' or 'erspan' INDEX - specify the tunnel INDEX value (indicates key if GRE, erpsan_idx if ERSPAN) FILTER - specify FILTER for mirroring selection ('to-lport' / 'from-lport') IP - specify Sink / Destination i.e. Remote IP ovn-nbctl mirror-del [NAME] remove mirrors ovn-nbctl mirror-list print mirrors ovn-nbctl lsp-attach-mirror PORT MIRROR attach source PORT to MIRROR ovn-nbctl lsp-detach-mirror PORT MIRROR detach source PORT from MIRROR","title":"Create OVN Remote Port Mirroring"},{"location":"en/advance/ovn-remote-port-mirroring/#configure-receiving-pod","text":"Execute the following commands in the Pod: root@pod1:/kube-ovn# ip link add mirror1 type gretap local 172 .19.0.21 key 99 dev net1 root@pod1:/kube-ovn# ip link set mirror1 up Now you can capture the mirrored packets: root@pod1:/kube-ovn# tcpdump -i mirror1 -nnve tcpdump: listening on mirror1, link-type EN10MB ( Ethernet ) , snapshot length 262144 bytes 05 :13:30.328808 00 :00:00:a3:f5:e2 > 00 :00:00:97:0f:6e, ethertype ARP ( 0x0806 ) , length 42 : Ethernet ( len 6 ) , IPv4 ( len 4 ) , Request who-has 10 .16.0.7 tell 10 .16.0.4, length 28 05 :13:30.559167 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57364 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.50472: 34511 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.559343 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 212 : ( tos 0x0, ttl 64 , id 57365 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 198 ) 10 .16.0.4.53 > 10 .16.0.6.45177: 1659 NXDomain*- 0 /1/1 ( 170 ) 05 :13:30.560625 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 200 : ( tos 0x0, ttl 64 , id 57367 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 186 ) 10 .16.0.4.53 > 10 .16.0.6.43848: 2636 *- 0 /1/1 ( 158 ) 05 :13:30.562774 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 191 : ( tos 0x0, ttl 64 , id 57368 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 177 ) 10 .16.0.4.53 > 10 .16.0.6.37755: 48737 NXDomain*- 0 /1/1 ( 149 ) 05 :13:30.563523 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 187 : ( tos 0x0, ttl 64 , id 57369 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 173 ) 10 .16.0.4.53 > 10 .16.0.6.53887: 45519 NXDomain*- 0 /1/1 ( 145 ) 05 :13:30.564940 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57370 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.40846: 25745 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.565140 00 :00:00:a3:f5:e2 > 00 :00:00:89:d5:cc, ethertype IPv4 ( 0x0800 ) , length 201 : ( tos 0x0, ttl 64 , id 57371 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 187 ) 10 .16.0.4.53 > 10 .16.0.6.45214: 61875 NXDomain*- 0 /1/1 ( 159 ) 05 :13:30.566023 00 :00:00:a3:f5:e2 > 00 :00:00:55:e4:4e, ethertype IPv4 ( 0x0800 ) , length 80 : ( tos 0x0, ttl 64 , id 45937 , offset 0 , flags [ DF ] , proto UDP ( 17 ) , length 66 ) 10 .16.0.4.44116 > 172 .18.0.1.53: 16025 + [ 1au ] AAAA? alauda.cn. ( 38 )","title":"Configure Receiving Pod"},{"location":"en/advance/ovn-remote-port-mirroring/#notice","text":"If you are using ERSPAN as the encapsulation protocol, the Linux kernel version of the OVN nodes and remote devices must not be lower than 4.14. If you are using ERSPAN as the encapsulation protocol and using IPv6 as the transport network, the Linux kernel version must not be lower than 4.16. The transmission of mirrored traffic is unidirectional, so you only need to ensure that the OVN node can access the remote device. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Notice"},{"location":"en/advance/performance-tuning/","text":"Performance Tuning \u00b6 To keep the installation simple and feature-complete, the default installation script for Kube-OVN does not have performance-specific optimizations. If the applications are sensitive to latency and throughput, administrators can use this document to make specific performance optimizations. The community will continue to iterate on the performance. Some general performance optimizations have been integrated into the latest version, so it is recommended to use the latest version to get better default performance. For more on the process and methodology of performance optimization, please watch the video Kube-OVN \u5bb9\u5668\u6027\u80fd\u4f18\u5316\u4e4b\u65c5 \u3002 Benchmarking \u00b6 Because the hardware and software environments vary greatly, the performance test data provided here can only be used as a reference, and the actual test results may differ significantly from the results in this document. It is recommended to compare the performance test results before and after optimization, and the performance comparison between the host network and the container network. Overlay Performance Comparison before and after Optimization \u00b6 Environment: Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 Overlay Mode CPU: Intel(R) Xeon(R) E-2278G Network: 2*10Gbps, xmit_hash_policy=layer3+4 We use qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw to test bandwidth and latency of tcp/udp in 1-byte packets and the host network, respectively. Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Default 25.7 22.9 27.1 1.59 Kube-OVN Optimized 13.9 12.9 27.6 5.57 HOST Network 13.1 12.4 28.2 6.02 Overlay\uff0c Underlay and Calico Comparison \u00b6 Next, we compare the overlay and underlay performance of the optimized Kube-OVN at different packet sizes with Calico's IPIP Always , IPIP never and the host network. Environment : Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 CPU: AMD EPYC 7402P 24-Core Processor Network: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Overlay 15.2 14.6 23.6 2.65 Kube-OVN Underlay 14.3 13.8 24.2 3.46 Calico IPIP 21.4 20.2 23.6 1.18 Calico NoEncap 19.3 16.9 23.6 1.76 HOST Network 16.6 15.4 24.8 2.64 qperf -t 60 <server ip> -ub -oo msg_size:1K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 16.5 15.8 10.2 2.77 Kube-OVN Underlay 15.9 14.5 9.6 3.22 Calico IPIP 22.5 21.5 1.45 1.14 Calico NoEncap 19.4 18.3 3.76 1.63 HOST Network 18.1 16.6 9.32 2.66 qperf -t 60 <server ip> -ub -oo msg_size:4K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 34.7 41.6 16.0 9.23 Kube-OVN Underlay 32.6 44 15.1 6.71 Calico IPIP 44.8 52.9 2.94 3.26 Calico NoEncap 40 49.6 6.56 4.19 HOST Network 35.9 45.9 14.6 5.59 In some cases the container network outperforms the host network, this is because the container network path is optimized to completely bypass netfilter. Due to the existence of kube-proxy , all packets in host network have to go through netfilter, which will lead to more CPU consumption, so that container network in some environments has better performance. Dataplane performance optimization methods \u00b6 The optimization methods described here are related to the hardware and software environment and the desired functionality, so please carefully understand the prerequisites for optimization before attempting it. CPU Performance Mode Tuning \u00b6 In some environments the CPU is running in power saving mode, performance in this mode will be unstable and latency will increase significantly, it is recommended to use the CPU's performance mode for more stable performance. cpupower frequency-set -g performance NIC Hardware Queue Adjustment \u00b6 In the case of increased traffic, a small buffer queue may lead to significant performance degradation due to a high packet loss rate and needs to be tuned. Check the current NIC queue length: # ethtool -g eno1 Ring parameters for eno1: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 255 RX Mini: 0 RX Jumbo: 0 TX: 255 Increase the queue length to the maximum: ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096 Optimize with tuned \u00b6 tuned can use a series of preconfigured profile files to perform system optimizations for a specific scenario. For latency-first scenarios: tuned-adm profile network-latency For throughput-first scenarios: tuned-adm profile network-throughput Interrupt Binding \u00b6 We recommend disabling irqbalance and binding NIC interrupts to specific CPUs to avoid performance fluctuations caused by switching between multiple CPUs. Disable OVN LB \u00b6 The L2 LB implementation of OVN requires calling the kernel's conntrack module and recirculate, resulting in a significant CPU overhead, which is tested to be around 20%. For Overlay networks you can use kube-proxy to complete the service forwarding function for better Pod-to-Pod performance. This can be turned off in kube-ovn-controller args: command : - /kube-ovn/start-controller.sh args : ... - --enable-lb=false ... In Underlay mode kube-proxy cannot use iptables or ipvs to control container network traffic, if you want to disable the LB function, you need to confirm whether you do not need the Service function. FastPath Kernel Module \u00b6 Since the container network and the host network are on different network ns, the packets will pass through the netfilter module several times when they are transmitted across the host, which results in a CPU overhead of nearly 20%. The FastPath module can reduce CPU overhead by bypassing netfilter, since in most cases applications within a container network do not need to use the functionality of the netfilter module. If you need to use the functions provided by netfilter such as iptables, ipvs, nftables, etc. in the container network, this module will disable the related functions. Since kernel modules are kernel version dependent, it is not possible to provide a single kernel module artifact that adapts to all kernels. We pre-compiled the FastPath module for part of the kernels, which can be accessed by tunning-package . You can also compile it manually, see Compiling FastPath Module After obtaining the kernel module, you can load the FastPath module on each node using insmod kube_ovn_fastpath.ko and verify that the module was loaded successfully using dmesg : # dmesg ... [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in ... OVS Kernel Module Optimization \u00b6 OVS flow processing including hashing, matching, etc. consumes about 10% of the CPU resources. Some instruction sets on modern x86 CPUs such as popcnt and sse4.2 can speed up the computation process, but the kernel is not compiled with these options enabled. It has been tested that the CPU consumption of flow-related operations is reduced to about 5% when the corresponding instruction set optimizations are enabled. Similar to the compilation of the FastPath module, it is not possible to provide a single kernel module artifact for all kernels. Users need to compile manually or go to tunning-package to see if a compiled package is available for download. Before using this kernel module, please check if the CPU supports the following instruction set: cat /proc/cpuinfo | grep popcnt cat /proc/cpuinfo | grep sse4_2 Compile and Install in CentOS \u00b6 Install the relevant compilation dependencies and kernel headers: yum install -y gcc kernel-devel- $( uname -r ) python3 autoconf automake libtool rpm-build openssl-devel Compile the OVS kernel module and generate the corresponding RPM: git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --with-linux = /lib/modules/ $( uname -r ) /build CFLAGS = \"-g -O2 -mpopcnt -msse4.2\" make rpm-fedora-kmod cd rpm/rpmbuild/RPMS/x86_64/ Copy the RPM to each node and install: rpm -i openvswitch-kmod-2.15.2-1.el7.x86_64.rpm If you have previously started Kube-OVN and the older version of the OVS module has been loaded into the kernel. It is recommended to reboot the machine to reload the new version of the kernel module. Compile and Install in Ubuntu \u00b6 Install the relevant compilation dependencies and kernel headers: apt install -y autoconf automake libtool gcc build-essential libssl-dev Compile the OVS kernel module and install: apt install -y autoconf automake libtool gcc build-essential libssl-dev git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --prefix = /usr/ --localstatedir = /var --enable-ssl --with-linux = /lib/modules/ $( uname -r ) /build make -j ` nproc ` make install make modules_install cat > /etc/depmod.d/openvswitch.conf << EOF override openvswitch * extra override vport-* * extra EOF depmod -a cp debian/openvswitch-switch.init /etc/init.d/openvswitch-switch /etc/init.d/openvswitch-switch force-reload-kmod If you have previously started Kube-OVN and the older version of the OVS module has been loaded into the kernel. It is recommended to reboot the machine to reload the new version of the kernel module. Using STT Type Tunnel \u00b6 Common tunnel encapsulation protocols such as Geneve and Vxlan use the UDP protocol to encapsulate packets and are well supported in the kernel. However, when TCP packets are encapsulated using UDP, the optimization and offload features of modern operating systems and network cards for the TCP protocol do not work well, resulting in a significant drop in TCP throughput. In some virtualization scenarios, due to CPU limitations, TCP packet throughput may even be a tenth of that of the host network. STT provides an innovative tunneling protocol that uses TCP formatted header for encapsulation. This encapsulation only emulates the TCP protocol header format without actually establishing a TCP connection, but can take full advantage of the TCP optimization capabilities of modern operating systems and network cards. In our tests TCP packet throughput can be improved several times, reaching performance levels close to those of the host network. The STT tunnel is not pre-installed in the kernel and needs to be installed by compiling the OVS kernel module, which can be found in the previous section. Enable STT tunnel: kubectl set env daemonset/ovs-ovn -n kube-system TUNNEL_TYPE = stt kubectl delete pod -n kube-system -lapp = ovs \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Performance Tuning"},{"location":"en/advance/performance-tuning/#performance-tuning","text":"To keep the installation simple and feature-complete, the default installation script for Kube-OVN does not have performance-specific optimizations. If the applications are sensitive to latency and throughput, administrators can use this document to make specific performance optimizations. The community will continue to iterate on the performance. Some general performance optimizations have been integrated into the latest version, so it is recommended to use the latest version to get better default performance. For more on the process and methodology of performance optimization, please watch the video Kube-OVN \u5bb9\u5668\u6027\u80fd\u4f18\u5316\u4e4b\u65c5 \u3002","title":"Performance Tuning"},{"location":"en/advance/performance-tuning/#benchmarking","text":"Because the hardware and software environments vary greatly, the performance test data provided here can only be used as a reference, and the actual test results may differ significantly from the results in this document. It is recommended to compare the performance test results before and after optimization, and the performance comparison between the host network and the container network.","title":"Benchmarking"},{"location":"en/advance/performance-tuning/#overlay-performance-comparison-before-and-after-optimization","text":"Environment: Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 Overlay Mode CPU: Intel(R) Xeon(R) E-2278G Network: 2*10Gbps, xmit_hash_policy=layer3+4 We use qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw to test bandwidth and latency of tcp/udp in 1-byte packets and the host network, respectively. Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Default 25.7 22.9 27.1 1.59 Kube-OVN Optimized 13.9 12.9 27.6 5.57 HOST Network 13.1 12.4 28.2 6.02","title":"Overlay Performance Comparison before and after Optimization"},{"location":"en/advance/performance-tuning/#overlay-underlay-and-calico-comparison","text":"Next, we compare the overlay and underlay performance of the optimized Kube-OVN at different packet sizes with Calico's IPIP Always , IPIP never and the host network. Environment : Kubernetes: 1.22.0 OS: CentOS 7 Kube-OVN: 1.8.0 CPU: AMD EPYC 7402P 24-Core Processor Network: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 qperf -t 60 <server ip> -ub -oo msg_size:1 -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Mb/s) udp_bw(Mb/s) Kube-OVN Overlay 15.2 14.6 23.6 2.65 Kube-OVN Underlay 14.3 13.8 24.2 3.46 Calico IPIP 21.4 20.2 23.6 1.18 Calico NoEncap 19.3 16.9 23.6 1.76 HOST Network 16.6 15.4 24.8 2.64 qperf -t 60 <server ip> -ub -oo msg_size:1K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 16.5 15.8 10.2 2.77 Kube-OVN Underlay 15.9 14.5 9.6 3.22 Calico IPIP 22.5 21.5 1.45 1.14 Calico NoEncap 19.4 18.3 3.76 1.63 HOST Network 18.1 16.6 9.32 2.66 qperf -t 60 <server ip> -ub -oo msg_size:4K -vu tcp_lat tcp_bw udp_lat udp_bw Type tcp_lat (us) udp_lat (us) tcp_bw (Gb/s) udp_bw(Gb/s) Kube-OVN Overlay 34.7 41.6 16.0 9.23 Kube-OVN Underlay 32.6 44 15.1 6.71 Calico IPIP 44.8 52.9 2.94 3.26 Calico NoEncap 40 49.6 6.56 4.19 HOST Network 35.9 45.9 14.6 5.59 In some cases the container network outperforms the host network, this is because the container network path is optimized to completely bypass netfilter. Due to the existence of kube-proxy , all packets in host network have to go through netfilter, which will lead to more CPU consumption, so that container network in some environments has better performance.","title":"Overlay\uff0c Underlay and Calico Comparison"},{"location":"en/advance/performance-tuning/#dataplane-performance-optimization-methods","text":"The optimization methods described here are related to the hardware and software environment and the desired functionality, so please carefully understand the prerequisites for optimization before attempting it.","title":"Dataplane performance optimization methods"},{"location":"en/advance/performance-tuning/#cpu-performance-mode-tuning","text":"In some environments the CPU is running in power saving mode, performance in this mode will be unstable and latency will increase significantly, it is recommended to use the CPU's performance mode for more stable performance. cpupower frequency-set -g performance","title":"CPU Performance Mode Tuning"},{"location":"en/advance/performance-tuning/#nic-hardware-queue-adjustment","text":"In the case of increased traffic, a small buffer queue may lead to significant performance degradation due to a high packet loss rate and needs to be tuned. Check the current NIC queue length: # ethtool -g eno1 Ring parameters for eno1: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 255 RX Mini: 0 RX Jumbo: 0 TX: 255 Increase the queue length to the maximum: ethtool -G eno1 rx 4096 ethtool -G eno1 tx 4096","title":"NIC Hardware Queue Adjustment"},{"location":"en/advance/performance-tuning/#optimize-with-tuned","text":"tuned can use a series of preconfigured profile files to perform system optimizations for a specific scenario. For latency-first scenarios: tuned-adm profile network-latency For throughput-first scenarios: tuned-adm profile network-throughput","title":"Optimize with tuned"},{"location":"en/advance/performance-tuning/#interrupt-binding","text":"We recommend disabling irqbalance and binding NIC interrupts to specific CPUs to avoid performance fluctuations caused by switching between multiple CPUs.","title":"Interrupt Binding"},{"location":"en/advance/performance-tuning/#disable-ovn-lb","text":"The L2 LB implementation of OVN requires calling the kernel's conntrack module and recirculate, resulting in a significant CPU overhead, which is tested to be around 20%. For Overlay networks you can use kube-proxy to complete the service forwarding function for better Pod-to-Pod performance. This can be turned off in kube-ovn-controller args: command : - /kube-ovn/start-controller.sh args : ... - --enable-lb=false ... In Underlay mode kube-proxy cannot use iptables or ipvs to control container network traffic, if you want to disable the LB function, you need to confirm whether you do not need the Service function.","title":"Disable OVN LB"},{"location":"en/advance/performance-tuning/#fastpath-kernel-module","text":"Since the container network and the host network are on different network ns, the packets will pass through the netfilter module several times when they are transmitted across the host, which results in a CPU overhead of nearly 20%. The FastPath module can reduce CPU overhead by bypassing netfilter, since in most cases applications within a container network do not need to use the functionality of the netfilter module. If you need to use the functions provided by netfilter such as iptables, ipvs, nftables, etc. in the container network, this module will disable the related functions. Since kernel modules are kernel version dependent, it is not possible to provide a single kernel module artifact that adapts to all kernels. We pre-compiled the FastPath module for part of the kernels, which can be accessed by tunning-package . You can also compile it manually, see Compiling FastPath Module After obtaining the kernel module, you can load the FastPath module on each node using insmod kube_ovn_fastpath.ko and verify that the module was loaded successfully using dmesg : # dmesg ... [ 619631 .323788 ] init_module,kube_ovn_fastpath_local_out [ 619631 .323798 ] init_module,kube_ovn_fastpath_post_routing [ 619631 .323800 ] init_module,kube_ovn_fastpath_pre_routing [ 619631 .323801 ] init_module,kube_ovn_fastpath_local_in ...","title":"FastPath Kernel Module"},{"location":"en/advance/performance-tuning/#ovs-kernel-module-optimization","text":"OVS flow processing including hashing, matching, etc. consumes about 10% of the CPU resources. Some instruction sets on modern x86 CPUs such as popcnt and sse4.2 can speed up the computation process, but the kernel is not compiled with these options enabled. It has been tested that the CPU consumption of flow-related operations is reduced to about 5% when the corresponding instruction set optimizations are enabled. Similar to the compilation of the FastPath module, it is not possible to provide a single kernel module artifact for all kernels. Users need to compile manually or go to tunning-package to see if a compiled package is available for download. Before using this kernel module, please check if the CPU supports the following instruction set: cat /proc/cpuinfo | grep popcnt cat /proc/cpuinfo | grep sse4_2","title":"OVS Kernel Module Optimization"},{"location":"en/advance/performance-tuning/#compile-and-install-in-centos","text":"Install the relevant compilation dependencies and kernel headers: yum install -y gcc kernel-devel- $( uname -r ) python3 autoconf automake libtool rpm-build openssl-devel Compile the OVS kernel module and generate the corresponding RPM: git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --with-linux = /lib/modules/ $( uname -r ) /build CFLAGS = \"-g -O2 -mpopcnt -msse4.2\" make rpm-fedora-kmod cd rpm/rpmbuild/RPMS/x86_64/ Copy the RPM to each node and install: rpm -i openvswitch-kmod-2.15.2-1.el7.x86_64.rpm If you have previously started Kube-OVN and the older version of the OVS module has been loaded into the kernel. It is recommended to reboot the machine to reload the new version of the kernel module.","title":"Compile and Install in CentOS"},{"location":"en/advance/performance-tuning/#compile-and-install-in-ubuntu","text":"Install the relevant compilation dependencies and kernel headers: apt install -y autoconf automake libtool gcc build-essential libssl-dev Compile the OVS kernel module and install: apt install -y autoconf automake libtool gcc build-essential libssl-dev git clone -b branch-2.17 --depth = 1 https://github.com/openvswitch/ovs.git cd ovs curl -s https://github.com/kubeovn/ovs/commit/2d2c83c26d4217446918f39d5cd5838e9ac27b32.patch | git apply ./boot.sh ./configure --prefix = /usr/ --localstatedir = /var --enable-ssl --with-linux = /lib/modules/ $( uname -r ) /build make -j ` nproc ` make install make modules_install cat > /etc/depmod.d/openvswitch.conf << EOF override openvswitch * extra override vport-* * extra EOF depmod -a cp debian/openvswitch-switch.init /etc/init.d/openvswitch-switch /etc/init.d/openvswitch-switch force-reload-kmod If you have previously started Kube-OVN and the older version of the OVS module has been loaded into the kernel. It is recommended to reboot the machine to reload the new version of the kernel module.","title":"Compile and Install in Ubuntu"},{"location":"en/advance/performance-tuning/#using-stt-type-tunnel","text":"Common tunnel encapsulation protocols such as Geneve and Vxlan use the UDP protocol to encapsulate packets and are well supported in the kernel. However, when TCP packets are encapsulated using UDP, the optimization and offload features of modern operating systems and network cards for the TCP protocol do not work well, resulting in a significant drop in TCP throughput. In some virtualization scenarios, due to CPU limitations, TCP packet throughput may even be a tenth of that of the host network. STT provides an innovative tunneling protocol that uses TCP formatted header for encapsulation. This encapsulation only emulates the TCP protocol header format without actually establishing a TCP connection, but can take full advantage of the TCP optimization capabilities of modern operating systems and network cards. In our tests TCP packet throughput can be improved several times, reaching performance levels close to those of the host network. The STT tunnel is not pre-installed in the kernel and needs to be installed by compiling the OVS kernel module, which can be found in the previous section. Enable STT tunnel: kubectl set env daemonset/ovs-ovn -n kube-system TUNNEL_TYPE = stt kubectl delete pod -n kube-system -lapp = ovs \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Using STT Type Tunnel"},{"location":"en/advance/security-group/","text":"SecurityGroup Usage \u00b6 Kube-OVN has supported the configuration of security-groups, and the CRD used to configure security-groups is SecurityGroup. SecurityGroup Example \u00b6 apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-example spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 1 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address The specific meaning of each field of the SecurityGroup can be found in the Kube-OVN API Reference . Pods bind security-groups by adding annotations, two annotations are used. ovn.kubernetes.io/port_security : \"true\" ovn.kubernetes.io/security_groups : sg-example Caution \u00b6 Security-groups are finally restricted by setting ACL rules, and as mentioned in the OVN documentation, if two ACL rules match with the same priority, it is uncertain which ACL will actually work. Therefore, when setting up security-group rules, you need to be careful to differentiate the priority. When adding a security-group, it is important to know what restrictions are being added. As a CNI, Kube-OVN will perform a Pod-to-Gateway connectivity test after creating a Pod. Actual test \u00b6 Create a Pod using the following yaml, and specify the security-group in the annotation for the pod. apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-example' name : sg-test-pod namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest The actual test results show as follows: # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h32m <none> kube-ovn-worker <none> <none> test-99fff7f86-52h9r 1 /1 Running 0 5h41m 10 .16.0.14 kube-ovn-control-plane <none> <none> test-99fff7f86-qcgjw 1 /1 Running 0 5h43m 10 .16.0.13 kube-ovn-worker <none> <none> Execute kubectl describe pod to see information about the pod, and you can see the error message: # kubectl describe pod sg-test-pod Name: sg-test-pod Namespace: default Priority: 0 Node: kube-ovn-worker/172.18.0.2 Start Time: Tue, 28 Feb 2023 10 :29:36 +0800 Labels: app = static Annotations: ovn.kubernetes.io/allocated: true ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.15 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:FA:17:97 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/port_security: true ovn.kubernetes.io/routed: true ovn.kubernetes.io/security_groups: sg-allow-reject Status: Pending IP: IPs: <none> - - - - - Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 5m3s ( x70 over 4h59m ) kubelet ( combined from similar events ) : Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"40636e0c7f1ade5500fa958486163d74f2e2300051a71522a9afd7ba0538afb6\" : plugin type = \"kube-ovn\" failed ( add ) : RPC failed ; request ip return 500 configure nic failed 10 .16.0.15 network not ready after 200 ping 10 .16.0.1 Modify the rules for the security group to add access rules to the gateway, refer to the following: apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-gw-both spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 2 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.1/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 2 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : icmp remoteAddress : 10.16.0.1/16 remoteType : address In the inbound and outbound rules respectively, add a rule to allow access to the gateway, and set the rule to have the highest priority. Deploying with the following yaml to bind security group, confirm that the Pod is operational: apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-gw-both' name : sg-gw-both namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest To view Pod information after deployment: # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h41m <none> kube-ovn-worker <none> <none> sg-gw-both 1 /1 Running 0 5h37m 10 .16.0.19 kube-ovn-worker <none> <none> So for the use of security groups, be particularly clear about the effect of the added restriction rules. If it is simply to restrict traffic access, consider using a network policy instead. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"SecurityGroup Usage"},{"location":"en/advance/security-group/#securitygroup-usage","text":"Kube-OVN has supported the configuration of security-groups, and the CRD used to configure security-groups is SecurityGroup.","title":"SecurityGroup Usage"},{"location":"en/advance/security-group/#securitygroup-example","text":"apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-example spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 1 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address The specific meaning of each field of the SecurityGroup can be found in the Kube-OVN API Reference . Pods bind security-groups by adding annotations, two annotations are used. ovn.kubernetes.io/port_security : \"true\" ovn.kubernetes.io/security_groups : sg-example","title":"SecurityGroup Example"},{"location":"en/advance/security-group/#caution","text":"Security-groups are finally restricted by setting ACL rules, and as mentioned in the OVN documentation, if two ACL rules match with the same priority, it is uncertain which ACL will actually work. Therefore, when setting up security-group rules, you need to be careful to differentiate the priority. When adding a security-group, it is important to know what restrictions are being added. As a CNI, Kube-OVN will perform a Pod-to-Gateway connectivity test after creating a Pod.","title":"Caution"},{"location":"en/advance/security-group/#actual-test","text":"Create a Pod using the following yaml, and specify the security-group in the annotation for the pod. apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-example' name : sg-test-pod namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest The actual test results show as follows: # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h32m <none> kube-ovn-worker <none> <none> test-99fff7f86-52h9r 1 /1 Running 0 5h41m 10 .16.0.14 kube-ovn-control-plane <none> <none> test-99fff7f86-qcgjw 1 /1 Running 0 5h43m 10 .16.0.13 kube-ovn-worker <none> <none> Execute kubectl describe pod to see information about the pod, and you can see the error message: # kubectl describe pod sg-test-pod Name: sg-test-pod Namespace: default Priority: 0 Node: kube-ovn-worker/172.18.0.2 Start Time: Tue, 28 Feb 2023 10 :29:36 +0800 Labels: app = static Annotations: ovn.kubernetes.io/allocated: true ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.15 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:FA:17:97 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/port_security: true ovn.kubernetes.io/routed: true ovn.kubernetes.io/security_groups: sg-allow-reject Status: Pending IP: IPs: <none> - - - - - Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 5m3s ( x70 over 4h59m ) kubelet ( combined from similar events ) : Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"40636e0c7f1ade5500fa958486163d74f2e2300051a71522a9afd7ba0538afb6\" : plugin type = \"kube-ovn\" failed ( add ) : RPC failed ; request ip return 500 configure nic failed 10 .16.0.15 network not ready after 200 ping 10 .16.0.1 Modify the rules for the security group to add access rules to the gateway, refer to the following: apiVersion : kubeovn.io/v1 kind : SecurityGroup metadata : name : sg-gw-both spec : allowSameGroupTraffic : true egressRules : - ipVersion : ipv4 policy : allow priority : 2 protocol : all remoteAddress : 10.16.0.13/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : all remoteAddress : 10.16.0.1/16 remoteType : address ingressRules : - ipVersion : ipv4 policy : deny priority : 2 protocol : icmp remoteAddress : 10.16.0.14/16 remoteType : address - ipVersion : ipv4 policy : allow priority : 1 protocol : icmp remoteAddress : 10.16.0.1/16 remoteType : address In the inbound and outbound rules respectively, add a rule to allow access to the gateway, and set the rule to have the highest priority. Deploying with the following yaml to bind security group, confirm that the Pod is operational: apiVersion : v1 kind : Pod metadata : labels : app : static annotations : ovn.kubernetes.io/port_security : 'true' ovn.kubernetes.io/security_groups : 'sg-gw-both' name : sg-gw-both namespace : default spec : nodeName : kube-ovn-worker containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest To view Pod information after deployment: # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sg-test-pod 0 /1 ContainerCreating 0 5h41m <none> kube-ovn-worker <none> <none> sg-gw-both 1 /1 Running 0 5h37m 10 .16.0.19 kube-ovn-worker <none> <none> So for the use of security groups, be particularly clear about the effect of the added restriction rules. If it is simply to restrict traffic access, consider using a network policy instead. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Actual test"},{"location":"en/advance/vip/","text":"VIP Reservation \u00b6 In some scenarios we want to dynamically reserve part of the IP but not assign it to Pods but to other infrastructure e.g: Kubernetes nested Kubernetes scenarios where the upper Kubernetes uses the Underlay network take up the available addresses of the underlying Subnet. LB or other network infrastructure requires the use of an IP within a Subnet. Create Random Address VIP \u00b6 If you just want to set aside a number of IPs and have no requirement for the IP addresses themselves, you can use the following yaml to create them: apiVersion : kubeovn.io/v1 kind : Vip metadata : name : vip-dynamic-01 spec : subnet : ovn-default type : \"\" subnet : reserve the IP from this Subnet. type : Currently, two types are supported. If the value is empty, it indicates that it is only used for occupying ip addresses of ipam. switch_lb_vip The front-end vip address and back-end ip address of the switch lb must be on the same subnet. Query the VIP after creation. # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY vip-dynamic-01 10 .16.0.12 00 :00:00:F0:DB:25 ovn-default true It can be seen that the VIP is assigned the IP address 10.16.0.12 , which can later be used by other network infrastructures. Create a fixed address VIP \u00b6 The IP address of the reserved VIP can be fixed using the following yaml: apiVersion : kubeovn.io/v1 kind : Vip metadata : name : static-vip01 spec : subnet : ovn-default V4ip : \"10.16.0.121\" subnet : reserve the IP from this Subnet. V4ip : A fixed-assigned IP address that should within the CIDR range of subnet . Query the VIP after creation: # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY static-vip01 10 .16.0.121 00 :00:00:F0:DB:26 ovn-default true It can be seen that the VIP has been assigned the expected IP address. StatefulSet & Kubevirt VM keep VIP \u00b6 Specify for StatefulSet and VM resources, these Pods their owned will reuse the VIP when these Pods recreating. VM keep VIP must be enable the keep-vm-ip param in kube-ovn-controller . Refer Kubevirt VM Fixed Address Settings \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VIP Reservation"},{"location":"en/advance/vip/#vip-reservation","text":"In some scenarios we want to dynamically reserve part of the IP but not assign it to Pods but to other infrastructure e.g: Kubernetes nested Kubernetes scenarios where the upper Kubernetes uses the Underlay network take up the available addresses of the underlying Subnet. LB or other network infrastructure requires the use of an IP within a Subnet.","title":"VIP Reservation"},{"location":"en/advance/vip/#create-random-address-vip","text":"If you just want to set aside a number of IPs and have no requirement for the IP addresses themselves, you can use the following yaml to create them: apiVersion : kubeovn.io/v1 kind : Vip metadata : name : vip-dynamic-01 spec : subnet : ovn-default type : \"\" subnet : reserve the IP from this Subnet. type : Currently, two types are supported. If the value is empty, it indicates that it is only used for occupying ip addresses of ipam. switch_lb_vip The front-end vip address and back-end ip address of the switch lb must be on the same subnet. Query the VIP after creation. # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY vip-dynamic-01 10 .16.0.12 00 :00:00:F0:DB:25 ovn-default true It can be seen that the VIP is assigned the IP address 10.16.0.12 , which can later be used by other network infrastructures.","title":"Create Random Address VIP"},{"location":"en/advance/vip/#create-a-fixed-address-vip","text":"The IP address of the reserved VIP can be fixed using the following yaml: apiVersion : kubeovn.io/v1 kind : Vip metadata : name : static-vip01 spec : subnet : ovn-default V4ip : \"10.16.0.121\" subnet : reserve the IP from this Subnet. V4ip : A fixed-assigned IP address that should within the CIDR range of subnet . Query the VIP after creation: # kubectl get vip NAME V4IP PV4IP MAC PMAC V6IP PV6IP SUBNET READY static-vip01 10 .16.0.121 00 :00:00:F0:DB:26 ovn-default true It can be seen that the VIP has been assigned the expected IP address.","title":"Create a fixed address VIP"},{"location":"en/advance/vip/#statefulset-kubevirt-vm-keep-vip","text":"Specify for StatefulSet and VM resources, these Pods their owned will reuse the VIP when these Pods recreating. VM keep VIP must be enable the keep-vm-ip param in kube-ovn-controller . Refer Kubevirt VM Fixed Address Settings \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"StatefulSet &amp; Kubevirt VM keep VIP"},{"location":"en/advance/vpc-dns/","text":"\u81ea\u5b9a\u4e49 VPC DNS \u00b6 \u7531\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u548c \u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0c\u81ea\u5b9a VPC \u5185\u65e0\u6cd5\u8bbf\u95ee\u5230\u90e8\u7f72\u5728\u9ed8\u8ba4 VPC \u5185\u7684 coredns\u3002 \u5982\u679c\u7528\u6237\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 Kubernetes \u63d0\u4f9b\u7684\u96c6\u7fa4\u5185\u57df\u540d\u89e3\u6790\u80fd\u529b\uff0c\u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\uff0c\u5229\u7528 vpc-dns CRD \u6765\u5b9e\u73b0\u3002 \u8be5 CRD \u6700\u7ec8\u4f1a\u90e8\u7f72\u4e00\u4e2a coredns\uff0c\u8be5 Pod \u6709\u4e24\u4e2a\u7f51\u5361\uff0c\u4e00\u4e2a\u7f51\u5361\u5728\u7528\u6237\u81ea\u5b9a\u4e49 VPC\uff0c\u53e6\u4e00\u4e2a\u7f51\u5361\u5728\u9ed8\u8ba4 VPC \u4ece\u800c\u5b9e\u73b0\u7f51\u7edc\u4e92\u901a\uff0c\u540c\u65f6\u901a\u8fc7 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u63d0\u4f9b\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u4e00\u4e2a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u3002 \u90e8\u7f72 vpc-dns \u6240\u4f9d\u8d56\u7684\u8d44\u6e90 \u00b6 apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u9664\u4e86\u4ee5\u4e0a\u8d44\u6e90\uff0c\u8be5\u529f\u80fd\u8fd8\u4f9d\u8d56 nat-gw-pod \u955c\u50cf\u8fdb\u884c\u8def\u7531\u914d\u7f6e\u3002 \u914d\u7f6e\u9644\u52a0\u7f51\u5361 \u00b6 apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }' \u4fee\u6539 ovn-default \u5b50\u7f51\u7684 provider \u00b6 \u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn # \u53ea\u9700\u4fee\u6539\u8be5\u5b57\u6bb5 vpc : ovn-cluster \u914d\u7f6e vpc-dns \u7684 Configmap \u00b6 \u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : \"true\" nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\u662f\u5426\u542f\u7528\u529f\u80fd\uff0c\u9ed8\u8ba4 true \u3002 coredns-image \uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 coredns-template \uff1acoredns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\u83b7\u53d6\u5f53\u524d\u7248\u672c ovn \u76ee\u5f55\u4e0b coredns-template.yaml \u9ed8\u8ba4\u4e3a https://raw.githubusercontent.com/kubeovn/kube-ovn/\u5f53\u524d\u7248\u672c/yamls/coredns-template.yaml \u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u5730\u5740\u3002 k8s-service-port \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u7aef\u53e3\u3002 \u90e8\u7f72 vpc-dns \u00b6 \u914d\u7f6e vpc-dns yaml\uff1a kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u90e8\u7f72\u8d44\u6e90\u7684\u4fe1\u606f\uff1a # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72\u3002 \u9650\u5236\uff1a\u4e00\u4e2a VPC \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a VPC \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a VPC \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002 \u9a8c\u8bc1\u90e8\u7f72\u7ed3\u679c \u00b6 \u67e5\u770b vpc-dns Pod \u72b6\u6001\uff0c\u4f7f\u7528 label app=vpc-dns \uff0c\u53ef\u4ee5\u67e5\u770b\u6240\u6709 vpc-dns pod \u72b6\u6001\uff1a # kubectl -n kube-system get pods -l app=vpc-dns NAME READY STATUS RESTARTS AGE vpc-dns-test-cjh1-7b878d96b4-g5979 1 /1 Running 0 28s vpc-dns-test-cjh1-7b878d96b4-ltmf9 1 /1 Running 0 28s \u67e5\u770b slr \u72b6\u6001\u4fe1\u606f\uff1a # kubectl -n kube-system get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh1 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh1 113s \u8fdb\u5165\u8be5 VPC \u4e0b\u7684 Pod\uff0c\u6d4b\u8bd5 dns \u89e3\u6790: nslookup kubernetes.default.svc.cluster.local 10 .96.0.3 \u8be5 VPC \u4e0b\u7684 switch lb rule \u6240\u5728\u7684\u5b50\u7f51\u4ee5\u53ca\u540c\u4e00 VPC \u4e0b\u7684\u5176\u4ed6\u5b50\u7f51\u4e0b\u7684 pod \u90fd\u53ef\u4ee5\u89e3\u6790\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC DNS"},{"location":"en/advance/vpc-dns/#vpc-dns","text":"\u7531\u4e8e\u7528\u6237\u81ea\u5b9a\u4e49 VPC \u548c \u9ed8\u8ba4 VPC \u7f51\u7edc\u76f8\u4e92\u9694\u79bb\uff0c\u81ea\u5b9a VPC \u5185\u65e0\u6cd5\u8bbf\u95ee\u5230\u90e8\u7f72\u5728\u9ed8\u8ba4 VPC \u5185\u7684 coredns\u3002 \u5982\u679c\u7528\u6237\u5e0c\u671b\u5728\u81ea\u5b9a\u4e49 VPC \u5185\u4f7f\u7528 Kubernetes \u63d0\u4f9b\u7684\u96c6\u7fa4\u5185\u57df\u540d\u89e3\u6790\u80fd\u529b\uff0c\u53ef\u4ee5\u53c2\u8003\u672c\u6587\u6863\uff0c\u5229\u7528 vpc-dns CRD \u6765\u5b9e\u73b0\u3002 \u8be5 CRD \u6700\u7ec8\u4f1a\u90e8\u7f72\u4e00\u4e2a coredns\uff0c\u8be5 Pod \u6709\u4e24\u4e2a\u7f51\u5361\uff0c\u4e00\u4e2a\u7f51\u5361\u5728\u7528\u6237\u81ea\u5b9a\u4e49 VPC\uff0c\u53e6\u4e00\u4e2a\u7f51\u5361\u5728\u9ed8\u8ba4 VPC \u4ece\u800c\u5b9e\u73b0\u7f51\u7edc\u4e92\u901a\uff0c\u540c\u65f6\u901a\u8fc7 \u81ea\u5b9a\u4e49 VPC \u5185\u90e8\u8d1f\u8f7d\u5747\u8861 \u63d0\u4f9b\u81ea\u5b9a\u4e49 VPC \u5185\u7684\u4e00\u4e2a\u5185\u90e8\u8d1f\u8f7d\u5747\u8861\u3002","title":"\u81ea\u5b9a\u4e49 VPC DNS"},{"location":"en/advance/vpc-dns/#vpc-dns_1","text":"apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } \u9664\u4e86\u4ee5\u4e0a\u8d44\u6e90\uff0c\u8be5\u529f\u80fd\u8fd8\u4f9d\u8d56 nat-gw-pod \u955c\u50cf\u8fdb\u884c\u8def\u7531\u914d\u7f6e\u3002","title":"\u90e8\u7f72 vpc-dns \u6240\u4f9d\u8d56\u7684\u8d44\u6e90"},{"location":"en/advance/vpc-dns/#_1","text":"apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }'","title":"\u914d\u7f6e\u9644\u52a0\u7f51\u5361"},{"location":"en/advance/vpc-dns/#ovn-default-provider","text":"\u4fee\u6539 ovn-default \u7684 provider\uff0c\u4e3a\u4e0a\u9762 nad \u914d\u7f6e\u7684 provider ovn-nad.default.ovn apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn # \u53ea\u9700\u4fee\u6539\u8be5\u5b57\u6bb5 vpc : ovn-cluster","title":"\u4fee\u6539 ovn-default \u5b50\u7f51\u7684 provider"},{"location":"en/advance/vpc-dns/#vpc-dns-configmap","text":"\u5728 kube-system \u547d\u540d\u7a7a\u95f4\u4e0b\u521b\u5efa configmap\uff0c\u914d\u7f6e vpc-dns \u4f7f\u7528\u53c2\u6570\uff0c\u7528\u4e8e\u540e\u9762\u542f\u52a8 vpc-dns \u529f\u80fd\uff1a apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : \"true\" nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns \uff1a\u662f\u5426\u542f\u7528\u529f\u80fd\uff0c\u9ed8\u8ba4 true \u3002 coredns-image \uff1adns \u90e8\u7f72\u955c\u50cf\u3002\u9ed8\u8ba4\u4e3a\u96c6\u7fa4 coredns \u90e8\u7f72\u7248\u672c\u3002 coredns-vip \uff1a\u4e3a coredns \u63d0\u4f9b lb \u670d\u52a1\u7684 vip\u3002 coredns-template \uff1acoredns \u90e8\u7f72\u6a21\u677f\u6240\u5728\u7684 URL\u3002\u9ed8\u8ba4\u83b7\u53d6\u5f53\u524d\u7248\u672c ovn \u76ee\u5f55\u4e0b coredns-template.yaml \u9ed8\u8ba4\u4e3a https://raw.githubusercontent.com/kubeovn/kube-ovn/\u5f53\u524d\u7248\u672c/yamls/coredns-template.yaml \u3002 nad-name \uff1a\u914d\u7f6e\u7684 network-attachment-definitions \u8d44\u6e90\u540d\u79f0\u3002 nad-provider \uff1a\u4f7f\u7528\u7684 provider \u540d\u79f0\u3002 k8s-service-host \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 ip\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u5730\u5740\u3002 k8s-service-port \uff1a\u7528\u4e8e coredns \u8bbf\u95ee k8s apiserver \u670d\u52a1\u7684 port\uff0c\u9ed8\u8ba4\u4e3a\u96c6\u7fa4\u5185 apiserver \u7aef\u53e3\u3002","title":"\u914d\u7f6e vpc-dns \u7684 Configmap"},{"location":"en/advance/vpc-dns/#vpc-dns_2","text":"\u914d\u7f6e vpc-dns yaml\uff1a kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc \uff1a \u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684 vpc \u540d\u79f0\u3002 subnet \uff1a\u7528\u4e8e\u90e8\u7f72 dns \u7ec4\u4ef6\u7684\u5b50\u540d\u79f0\u3002 \u67e5\u770b\u90e8\u7f72\u8d44\u6e90\u7684\u4fe1\u606f\uff1a # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : true \u90e8\u7f72\u4e86\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6\uff0c false \u65e0\u90e8\u7f72\u3002 \u9650\u5236\uff1a\u4e00\u4e2a VPC \u4e0b\u53ea\u4f1a\u90e8\u7f72\u4e00\u4e2a\u81ea\u5b9a\u4e49 dns \u7ec4\u4ef6; \u5f53\u4e00\u4e2a VPC \u4e0b\u914d\u7f6e\u591a\u4e2a vpc-dns \u8d44\u6e90\uff08\u5373\u540c\u4e00\u4e2a VPC \u4e0d\u540c\u7684 subnet\uff09\uff0c\u53ea\u6709\u4e00\u4e2a vpc-dns \u8d44\u6e90\u72b6\u6001 true \uff0c\u5176\u4ed6\u4e3a fasle ; \u5f53 true \u7684 vpc-dns \u88ab\u5220\u9664\u6389\uff0c\u4f1a\u83b7\u53d6\u5176\u4ed6 false \u7684 vpc-dns \u8fdb\u884c\u90e8\u7f72\u3002","title":"\u90e8\u7f72 vpc-dns"},{"location":"en/advance/vpc-dns/#_2","text":"\u67e5\u770b vpc-dns Pod \u72b6\u6001\uff0c\u4f7f\u7528 label app=vpc-dns \uff0c\u53ef\u4ee5\u67e5\u770b\u6240\u6709 vpc-dns pod \u72b6\u6001\uff1a # kubectl -n kube-system get pods -l app=vpc-dns NAME READY STATUS RESTARTS AGE vpc-dns-test-cjh1-7b878d96b4-g5979 1 /1 Running 0 28s vpc-dns-test-cjh1-7b878d96b4-ltmf9 1 /1 Running 0 28s \u67e5\u770b slr \u72b6\u6001\u4fe1\u606f\uff1a # kubectl -n kube-system get slr NAME VIP PORT ( S ) SERVICE AGE vpc-dns-test-cjh1 10 .96.0.3 53 /UDP,53/TCP,9153/TCP kube-system/slr-vpc-dns-test-cjh1 113s \u8fdb\u5165\u8be5 VPC \u4e0b\u7684 Pod\uff0c\u6d4b\u8bd5 dns \u89e3\u6790: nslookup kubernetes.default.svc.cluster.local 10 .96.0.3 \u8be5 VPC \u4e0b\u7684 switch lb rule \u6240\u5728\u7684\u5b50\u7f51\u4ee5\u53ca\u540c\u4e00 VPC \u4e0b\u7684\u5176\u4ed6\u5b50\u7f51\u4e0b\u7684 pod \u90fd\u53ef\u4ee5\u89e3\u6790\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u9a8c\u8bc1\u90e8\u7f72\u7ed3\u679c"},{"location":"en/advance/vpc-internal-lb/","text":"Customize VPC Internal Load Balancing \u00b6 The Service provided by Kubernetes can be used for load balancing within the cluster. However, there are several issues with using Service as internal load balancing in customize VPC mode: The Service IP range is a cluster resource, shared by all customize VPCs, and cannot overlap. Users cannot set internal load balancing IP addresses according to their own preferences. To address the above issues, Kube OVN introduced the SwitchLBRule CRD in 1.11, allowing users to set internal load balancing rules within customize VPCs. SwitchLBRule support the following two ways to set internal load balancing rules within a customize VPC. Automatically Generate Load Balancing Rules by Selector \u00b6 Load balancing rules can be generated by selector automatic association with pod configuration through label . example of SwitchLBRule is as follows: apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP usage of selector , sessionAffinity , and port is the same as Kubernetes Service. vip \uff1acustomize load balancing IP address. namespace \uff1anamespace of the pod selected by selector . Kube OVN will determine the VPC of the selected pod based on the SwitchLBRule definition and set the corresponding L2 LB. Manually Defined Load Balancing Rules by Endpoints \u00b6 Load balancing rules can be customized configured by endpoints , to support scenarios where load balancing rules cannot be automatically generated through selector . For example, the load balancing backend is vm created by kubevirt . example of SwitchLBRule is as follows: apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default endpoints : - 192.168.0.101 - 192.168.0.102 - 192.168.0.103 ports : - name : dns port : 8888 targetPort : 80 protocol : TCP usage of sessionAffinity , and port is the same as Kubernetes Service. vip \uff1acustomize load balancing IP address. namespace \uff1anamespace of the pod selected by selector . endpoints \uff1aload balancing backend IP list. attention\uff1a If both selector and endpoints are configured, the selector configuration will be automatically ignored. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC Internal Load Balancer"},{"location":"en/advance/vpc-internal-lb/#customize-vpc-internal-load-balancing","text":"The Service provided by Kubernetes can be used for load balancing within the cluster. However, there are several issues with using Service as internal load balancing in customize VPC mode: The Service IP range is a cluster resource, shared by all customize VPCs, and cannot overlap. Users cannot set internal load balancing IP addresses according to their own preferences. To address the above issues, Kube OVN introduced the SwitchLBRule CRD in 1.11, allowing users to set internal load balancing rules within customize VPCs. SwitchLBRule support the following two ways to set internal load balancing rules within a customize VPC.","title":"Customize VPC Internal Load Balancing"},{"location":"en/advance/vpc-internal-lb/#automatically-generate-load-balancing-rules-by-selector","text":"Load balancing rules can be generated by selector automatic association with pod configuration through label . example of SwitchLBRule is as follows: apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default selector : - app:nginx ports : - name : dns port : 8888 targetPort : 80 protocol : TCP usage of selector , sessionAffinity , and port is the same as Kubernetes Service. vip \uff1acustomize load balancing IP address. namespace \uff1anamespace of the pod selected by selector . Kube OVN will determine the VPC of the selected pod based on the SwitchLBRule definition and set the corresponding L2 LB.","title":"Automatically Generate Load Balancing Rules by Selector"},{"location":"en/advance/vpc-internal-lb/#manually-defined-load-balancing-rules-by-endpoints","text":"Load balancing rules can be customized configured by endpoints , to support scenarios where load balancing rules cannot be automatically generated through selector . For example, the load balancing backend is vm created by kubevirt . example of SwitchLBRule is as follows: apiVersion : kubeovn.io/v1 kind : SwitchLBRule metadata : name : cjh-slr-nginx spec : vip : 1.1.1.1 sessionAffinity : ClientIP namespace : default endpoints : - 192.168.0.101 - 192.168.0.102 - 192.168.0.103 ports : - name : dns port : 8888 targetPort : 80 protocol : TCP usage of sessionAffinity , and port is the same as Kubernetes Service. vip \uff1acustomize load balancing IP address. namespace \uff1anamespace of the pod selected by selector . endpoints \uff1aload balancing backend IP list. attention\uff1a If both selector and endpoints are configured, the selector configuration will be automatically ignored. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Manually Defined Load Balancing Rules by Endpoints"},{"location":"en/advance/vpc-peering/","text":"VPC Peering \u00b6 VPC peering provides a mechanism for bridging two VPC networks through logical routes so that workloads within two VPCs can access each other through private addresses as if they were on the same private network, without the need for NAT forwarding through a gateway. Prerequisites \u00b6 This feature is only available for customized VPCs. To avoid route overlap the subnet CIDRs within the two VPCs cannot overlap. Currently, only interconnection of two VPCs is supported. Usage \u00b6 First create two non-interconnected VPCs with one Subnet under each VPC, and the CIDRs of the Subnets do not overlap with each other. kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : vpc-1 cidrBlock : 10.0.0.0/16 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : vpc-2 cidrBlock : 172.31.0.0/16 Add vpcPeerings and the corresponding static routes within each VPC: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : vpcPeerings : - remoteVpc : vpc-2 localConnectIP : 169.254.0.1/30 staticRoutes : - cidr : 172.31.0.0/16 nextHopIP : 169.254.0.2 policy : policyDst --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : vpcPeerings : - remoteVpc : vpc-1 localConnectIP : 169.254.0.2/30 staticRoutes : - cidr : 10.0.0.0/16 nextHopIP : 169.254.0.1 policy : policyDst remoteVpc : The name of another peering VPC. localConnectIP : As the IP address and CIDR of the interconnection endpoint. Note that both IPs should belong to the same CIDR and should not conflict with existing subnets. cidr \uff1aCIDR of the peering Subnet. nextHopIP \uff1aThe localConnectIP on the other end of the peering VPC. Create Pods under the two Subnets apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 name : vpc-1-pod spec : containers : - name : vpc-1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 name : vpc-2-pod spec : containers : - name : vpc-2-pod image : docker.io/library/nginx:alpine Test the network connectivity # kubectl exec -it vpc-1-pod -- ping $(kubectl get pod vpc-2-pod -o jsonpath='{.status.podIP}') PING 172 .31.0.2 ( 172 .31.0.2 ) : 56 data bytes 64 bytes from 172 .31.0.2: seq = 0 ttl = 62 time = 0 .655 ms 64 bytes from 172 .31.0.2: seq = 1 ttl = 62 time = 0 .086 ms 64 bytes from 172 .31.0.2: seq = 2 ttl = 62 time = 0 .098 ms ^C --- 172 .31.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .086/0.279/0.655 ms # kubectl exec -it vpc-2-pod -- ping $(kubectl get pod vpc-1-pod -o jsonpath='{.status.podIP}') PING 10 .0.0.2 ( 10 .0.0.2 ) : 56 data bytes 64 bytes from 10 .0.0.2: seq = 0 ttl = 62 time = 0 .594 ms 64 bytes from 10 .0.0.2: seq = 1 ttl = 62 time = 0 .093 ms 64 bytes from 10 .0.0.2: seq = 2 ttl = 62 time = 0 .088 ms ^C --- 10 .0.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .088/0.258/0.594 ms \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC Peering"},{"location":"en/advance/vpc-peering/#vpc-peering","text":"VPC peering provides a mechanism for bridging two VPC networks through logical routes so that workloads within two VPCs can access each other through private addresses as if they were on the same private network, without the need for NAT forwarding through a gateway.","title":"VPC Peering"},{"location":"en/advance/vpc-peering/#prerequisites","text":"This feature is only available for customized VPCs. To avoid route overlap the subnet CIDRs within the two VPCs cannot overlap. Currently, only interconnection of two VPCs is supported.","title":"Prerequisites"},{"location":"en/advance/vpc-peering/#usage","text":"First create two non-interconnected VPCs with one Subnet under each VPC, and the CIDRs of the Subnets do not overlap with each other. kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : vpc-1 cidrBlock : 10.0.0.0/16 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : {} --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : vpc-2 cidrBlock : 172.31.0.0/16 Add vpcPeerings and the corresponding static routes within each VPC: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-1 spec : vpcPeerings : - remoteVpc : vpc-2 localConnectIP : 169.254.0.1/30 staticRoutes : - cidr : 172.31.0.0/16 nextHopIP : 169.254.0.2 policy : policyDst --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : vpc-2 spec : vpcPeerings : - remoteVpc : vpc-1 localConnectIP : 169.254.0.2/30 staticRoutes : - cidr : 10.0.0.0/16 nextHopIP : 169.254.0.1 policy : policyDst remoteVpc : The name of another peering VPC. localConnectIP : As the IP address and CIDR of the interconnection endpoint. Note that both IPs should belong to the same CIDR and should not conflict with existing subnets. cidr \uff1aCIDR of the peering Subnet. nextHopIP \uff1aThe localConnectIP on the other end of the peering VPC. Create Pods under the two Subnets apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 name : vpc-1-pod spec : containers : - name : vpc-1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 name : vpc-2-pod spec : containers : - name : vpc-2-pod image : docker.io/library/nginx:alpine Test the network connectivity # kubectl exec -it vpc-1-pod -- ping $(kubectl get pod vpc-2-pod -o jsonpath='{.status.podIP}') PING 172 .31.0.2 ( 172 .31.0.2 ) : 56 data bytes 64 bytes from 172 .31.0.2: seq = 0 ttl = 62 time = 0 .655 ms 64 bytes from 172 .31.0.2: seq = 1 ttl = 62 time = 0 .086 ms 64 bytes from 172 .31.0.2: seq = 2 ttl = 62 time = 0 .098 ms ^C --- 172 .31.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .086/0.279/0.655 ms # kubectl exec -it vpc-2-pod -- ping $(kubectl get pod vpc-1-pod -o jsonpath='{.status.podIP}') PING 10 .0.0.2 ( 10 .0.0.2 ) : 56 data bytes 64 bytes from 10 .0.0.2: seq = 0 ttl = 62 time = 0 .594 ms 64 bytes from 10 .0.0.2: seq = 1 ttl = 62 time = 0 .093 ms 64 bytes from 10 .0.0.2: seq = 2 ttl = 62 time = 0 .088 ms ^C --- 10 .0.0.2 ping statistics --- 3 packets transmitted, 3 packets received, 0 % packet loss round-trip min/avg/max = 0 .088/0.258/0.594 ms \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Usage"},{"location":"en/advance/windows/","text":"Windows Support \u00b6 Kube-OVN supports Kubernetes cluster networks that include Windows system nodes, allowing unified containers network management. Prerequisites \u00b6 Read Adding Windows nodes to add Windows nodes. Windows nodes must have the KB4489899 patch installed for Overlay/VXLAN networks to work properly, and it is recommended to update your system to the latest version. Hyper-V and management tools must be installed on the Windows node. Due to Windows restrictions tunnel encapsulation can only be used in Vxlan mode. SSL, IPv6, dual-stack, QoS features are not supported at this time. Dynamic subnet and dynamic tunnel interface are not supported at this time. You need to create the subnet and select the network interface before installing the Windows node. Multiple ProviderNetwork s are not supported, and the bridge interface configuration cannot be dynamically adjusted. Install OVS on Windows \u00b6 Due to some issues with upstream OVN and OVS support for Windows containers, a modified installation package provided by Kube-OVN is required. Use the following command to enable the TESTSIGNING startup item on the Windows node, which requires a system reboot to take effect. bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS bcdedit /set TESTSIGNING ON bcdedit /set nointegritychecks ON Download Windows package on Windows node and install. Confirm that the service is running properly after installation: PS > Get-Service | findstr ovs Running ovsdb-server Open vSwitch DB Service Running ovs-vswitchd Open vSwitch Service Install Kube-OVN \u00b6 Download the installation script in the Windows node install.ps1 . Add relevant parameters and run: . \\i nstall.ps1 -KubeConfig C: \\k\\a dmin.conf -ApiServer https://192.168.140.180:6443 -ServiceCIDR 10 .96.0.0/12 By default, Kube-OVN uses the NIC where the node IP is located as the tunnel interface. If you need to use another NIC, you need to add the specified annotation to the Node before installation, e.g. ovn.kubernetes.io/tunnel_interface=Ethernet1 . \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Windows Support"},{"location":"en/advance/windows/#windows-support","text":"Kube-OVN supports Kubernetes cluster networks that include Windows system nodes, allowing unified containers network management.","title":"Windows Support"},{"location":"en/advance/windows/#prerequisites","text":"Read Adding Windows nodes to add Windows nodes. Windows nodes must have the KB4489899 patch installed for Overlay/VXLAN networks to work properly, and it is recommended to update your system to the latest version. Hyper-V and management tools must be installed on the Windows node. Due to Windows restrictions tunnel encapsulation can only be used in Vxlan mode. SSL, IPv6, dual-stack, QoS features are not supported at this time. Dynamic subnet and dynamic tunnel interface are not supported at this time. You need to create the subnet and select the network interface before installing the Windows node. Multiple ProviderNetwork s are not supported, and the bridge interface configuration cannot be dynamically adjusted.","title":"Prerequisites"},{"location":"en/advance/windows/#install-ovs-on-windows","text":"Due to some issues with upstream OVN and OVS support for Windows containers, a modified installation package provided by Kube-OVN is required. Use the following command to enable the TESTSIGNING startup item on the Windows node, which requires a system reboot to take effect. bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS bcdedit /set TESTSIGNING ON bcdedit /set nointegritychecks ON Download Windows package on Windows node and install. Confirm that the service is running properly after installation: PS > Get-Service | findstr ovs Running ovsdb-server Open vSwitch DB Service Running ovs-vswitchd Open vSwitch Service","title":"Install OVS on Windows"},{"location":"en/advance/windows/#install-kube-ovn","text":"Download the installation script in the Windows node install.ps1 . Add relevant parameters and run: . \\i nstall.ps1 -KubeConfig C: \\k\\a dmin.conf -ApiServer https://192.168.140.180:6443 -ServiceCIDR 10 .96.0.0/12 By default, Kube-OVN uses the NIC where the node IP is located as the tunnel interface. If you need to use another NIC, you need to add the specified annotation to the Node before installation, e.g. ovn.kubernetes.io/tunnel_interface=Ethernet1 . \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Install Kube-OVN"},{"location":"en/advance/with-bgp/","text":"BGP Support \u00b6 Kube-OVN supports broadcasting the IP address of the Pod or Subnet to the outside world via BGP protocol, so that the outside world can access the Pod directly through the Pod IP. To use this feature, you need to install kube-ovn-speaker on specific nodes and add the corresponding annotation to the Pod or Subnet that needs to be exposed to the outside world. Install kube-ovn-speaker \u00b6 kube-ovn-speaker use GoBGP to publish routing information to the outside world and set the next-hop route to itself. Since the node where kube-ovn-speaker is deployed needs to carry return traffic, specific labeled nodes need to be selected for deployment: kubectl label nodes speaker-node-1 ovn.kubernetes.io/bgp = true kubectl label nodes speaker-node-2 ovn.kubernetes.io/bgp = true When there are multiple instances of kube-ovn-speaker, each of them will publish routes to the outside world, the upstream router needs to support multi-path ECMP. Download the corresponding yaml: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/speaker.yaml Modify the corresponding configuration in yaml: --neighbor-address=10.32.32.1 --neighbor-as=65030 --cluster-as=65000 neighbor-address : The address of the BGP Peer, usually the router gateway address. neighbor-as : The AS number of the BGP Peer. cluster-as : The AS number of the container network. Deploy yaml: kubectl apply -f speaker.yaml Publish Pod/Subnet Routes \u00b6 To use BGP for external routing, first set natOutgoing to false for the corresponding Subnet to allow the Pod IP to enter the underlying network directly. Add annotation to publish routes: kubectl annotate pod sample ovn.kubernetes.io/bgp = true kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp = true Delete annotation to disable the publishing: kubectl annotate pod perf-ovn-xzvd4 ovn.kubernetes.io/bgp- kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp- BGP Advance Options \u00b6 kube-ovn-speaker supports more BGP parameters for advanced configuration, which can be adjusted by users according to their network environment: announce-cluster-ip : Whether to publish Service routes to the public, default is false . auth-password : The access password for the BGP peer. holdtime : The heartbeat detection time between BGP neighbors. Neighbors with no messages after the change time will be removed, the default is 90 seconds. graceful-restart : Whether to enable BGP Graceful Restart. graceful-restart-time : BGP Graceful restart time refer to RFC4724 3. graceful-restart-deferral-time : BGP Graceful restart deferral time refer to RFC4724 4.1. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"BGP Support"},{"location":"en/advance/with-bgp/#bgp-support","text":"Kube-OVN supports broadcasting the IP address of the Pod or Subnet to the outside world via BGP protocol, so that the outside world can access the Pod directly through the Pod IP. To use this feature, you need to install kube-ovn-speaker on specific nodes and add the corresponding annotation to the Pod or Subnet that needs to be exposed to the outside world.","title":"BGP Support"},{"location":"en/advance/with-bgp/#install-kube-ovn-speaker","text":"kube-ovn-speaker use GoBGP to publish routing information to the outside world and set the next-hop route to itself. Since the node where kube-ovn-speaker is deployed needs to carry return traffic, specific labeled nodes need to be selected for deployment: kubectl label nodes speaker-node-1 ovn.kubernetes.io/bgp = true kubectl label nodes speaker-node-2 ovn.kubernetes.io/bgp = true When there are multiple instances of kube-ovn-speaker, each of them will publish routes to the outside world, the upstream router needs to support multi-path ECMP. Download the corresponding yaml: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/speaker.yaml Modify the corresponding configuration in yaml: --neighbor-address=10.32.32.1 --neighbor-as=65030 --cluster-as=65000 neighbor-address : The address of the BGP Peer, usually the router gateway address. neighbor-as : The AS number of the BGP Peer. cluster-as : The AS number of the container network. Deploy yaml: kubectl apply -f speaker.yaml","title":"Install kube-ovn-speaker"},{"location":"en/advance/with-bgp/#publish-podsubnet-routes","text":"To use BGP for external routing, first set natOutgoing to false for the corresponding Subnet to allow the Pod IP to enter the underlying network directly. Add annotation to publish routes: kubectl annotate pod sample ovn.kubernetes.io/bgp = true kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp = true Delete annotation to disable the publishing: kubectl annotate pod perf-ovn-xzvd4 ovn.kubernetes.io/bgp- kubectl annotate subnet ovn-default ovn.kubernetes.io/bgp-","title":"Publish Pod/Subnet Routes"},{"location":"en/advance/with-bgp/#bgp-advance-options","text":"kube-ovn-speaker supports more BGP parameters for advanced configuration, which can be adjusted by users according to their network environment: announce-cluster-ip : Whether to publish Service routes to the public, default is false . auth-password : The access password for the BGP peer. holdtime : The heartbeat detection time between BGP neighbors. Neighbors with no messages after the change time will be removed, the default is 90 seconds. graceful-restart : Whether to enable BGP Graceful Restart. graceful-restart-time : BGP Graceful restart time refer to RFC4724 3. graceful-restart-deferral-time : BGP Graceful restart deferral time refer to RFC4724 4.1. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"BGP Advance Options"},{"location":"en/advance/with-cilium/","text":"Integration with Cilium \u00b6 Cilium is an eBPF-based networking and security component. Kube-OVN uses the CNI Chaining mode to enhance existing features. Users can use both the rich network abstraction capabilities of Kube-OVN and the monitoring and security capabilities that come with eBPF. By integrating Cilium, Kube-OVN users can have the following gains: Richer and more efficient security policies. Hubble-based monitoring and UI. Prerequisites \u00b6 Linux kernel version above 4.19 or other compatible kernel for full eBPF capability support. Install Helm in advance to prepare for the installation of Cilium, please refer to Installing Helm to deploy Helm. Configure Kube-OVN \u00b6 In order to fully utilize the security capabilities of Cilium, you need to disable the networkpolicy feature within Kube-OVN and adjust the CNI configuration priority. Change the following variables in the install.sh script: ENABLE_NP = false CNI_CONFIG_PRIORITY = 10 If the deployment is complete, you can adjust the args of kube-ovn-controller : args : - --enable-np=false Modify the kube-ovn-cni args to adjust the CNI configuration priority: args : - --cni-conf-name=10-kube-ovn.conflist Adjust the Kube-OVN cni configuration name on each node: mv /etc/cni/net.d/01-kube-ovn.conflist /etc/cni/net.d/10-kube-ovn.conflist Deploy Cilium \u00b6 Create the chaining.yaml configuration file to use Cilium's generic-veth mode: apiVersion : v1 kind : ConfigMap metadata : name : cni-configuration namespace : kube-system data : cni-config : |- { \"name\": \"generic-veth\", \"cniVersion\": \"0.3.1\", \"plugins\": [ { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\" } }, { \"type\": \"portmap\", \"snat\": true, \"capabilities\": {\"portMappings\": true} }, { \"type\": \"cilium-cni\" } ] } Installation the chaining config: kubectl apply -f chaining.yaml Deploying Cilium with Helm: helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --set cni.chainingMode = generic-veth \\ --set cni.customConf = true \\ --set cni.configMap = cni-configuration \\ --set tunnel = disabled \\ --set enableIPv4Masquerade = false \\ --set enableIdentityMark = false Confirm that the Cilium installation was successful: # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: disabled \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 Containers: cilium Running: 2 cilium-operator Running: 2 Cluster Pods: 8 /11 managed by Cilium Image versions cilium quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb: 2 cilium-operator quay.io/cilium/operator-generic:v1.10.5@sha256:2d2f730f219d489ff0702923bf24c0002cd93eb4b47ba344375566202f56d972: 2 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Integration with Cilium"},{"location":"en/advance/with-cilium/#integration-with-cilium","text":"Cilium is an eBPF-based networking and security component. Kube-OVN uses the CNI Chaining mode to enhance existing features. Users can use both the rich network abstraction capabilities of Kube-OVN and the monitoring and security capabilities that come with eBPF. By integrating Cilium, Kube-OVN users can have the following gains: Richer and more efficient security policies. Hubble-based monitoring and UI.","title":"Integration with Cilium"},{"location":"en/advance/with-cilium/#prerequisites","text":"Linux kernel version above 4.19 or other compatible kernel for full eBPF capability support. Install Helm in advance to prepare for the installation of Cilium, please refer to Installing Helm to deploy Helm.","title":"Prerequisites"},{"location":"en/advance/with-cilium/#configure-kube-ovn","text":"In order to fully utilize the security capabilities of Cilium, you need to disable the networkpolicy feature within Kube-OVN and adjust the CNI configuration priority. Change the following variables in the install.sh script: ENABLE_NP = false CNI_CONFIG_PRIORITY = 10 If the deployment is complete, you can adjust the args of kube-ovn-controller : args : - --enable-np=false Modify the kube-ovn-cni args to adjust the CNI configuration priority: args : - --cni-conf-name=10-kube-ovn.conflist Adjust the Kube-OVN cni configuration name on each node: mv /etc/cni/net.d/01-kube-ovn.conflist /etc/cni/net.d/10-kube-ovn.conflist","title":"Configure Kube-OVN"},{"location":"en/advance/with-cilium/#deploy-cilium","text":"Create the chaining.yaml configuration file to use Cilium's generic-veth mode: apiVersion : v1 kind : ConfigMap metadata : name : cni-configuration namespace : kube-system data : cni-config : |- { \"name\": \"generic-veth\", \"cniVersion\": \"0.3.1\", \"plugins\": [ { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\" } }, { \"type\": \"portmap\", \"snat\": true, \"capabilities\": {\"portMappings\": true} }, { \"type\": \"cilium-cni\" } ] } Installation the chaining config: kubectl apply -f chaining.yaml Deploying Cilium with Helm: helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium --version 1 .11.6 \\ --namespace kube-system \\ --set cni.chainingMode = generic-veth \\ --set cni.customConf = true \\ --set cni.configMap = cni-configuration \\ --set tunnel = disabled \\ --set enableIPv4Masquerade = false \\ --set enableIdentityMark = false Confirm that the Cilium installation was successful: # cilium status /\u00af\u00af \\ /\u00af\u00af \\_ _/\u00af\u00af \\ Cilium: OK \\_ _/\u00af\u00af \\_ _/ Operator: OK /\u00af\u00af \\_ _/\u00af\u00af \\ Hubble: disabled \\_ _/\u00af\u00af \\_ _/ ClusterMesh: disabled \\_ _/ DaemonSet cilium Desired: 2 , Ready: 2 /2, Available: 2 /2 Deployment cilium-operator Desired: 2 , Ready: 2 /2, Available: 2 /2 Containers: cilium Running: 2 cilium-operator Running: 2 Cluster Pods: 8 /11 managed by Cilium Image versions cilium quay.io/cilium/cilium:v1.10.5@sha256:0612218e28288db360c63677c09fafa2d17edda4f13867bcabf87056046b33bb: 2 cilium-operator quay.io/cilium/operator-generic:v1.10.5@sha256:2d2f730f219d489ff0702923bf24c0002cd93eb4b47ba344375566202f56d972: 2 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Deploy Cilium"},{"location":"en/advance/with-openstack/","text":"Integration with OpenStack \u00b6 In some cases, users need to run virtual machines with OpenStack and containers with Kubernetes, and need the network to interoperate between containers and virtual machines and be under a unified control plane. If the OpenStack Neutron side also uses OVN as the underlying network, then Kube-OVN can use either cluster interconnection or shared underlying OVN to connect the OpenStack and Kubernetes networks. Cluster Interconnection \u00b6 This pattern is similar to Cluster Inter-Connection with OVN-IC to connect two Kubernetes cluster networks, except that the two ends of the cluster are replaced with OpenStack and Kubernetes\u3002 Prerequisites \u00b6 The subnet CIDRs within OpenStack and Kubernetes cannot overlap with each other in auto-route mode. A set of machines needs to exist that can be accessed by each cluster over the network and used to deploy controllers that interconnect across clusters. Each cluster needs to have a set of machines that can access each other across clusters via IP as the gateway nodes. This solution only connects to the Kubernetes default subnet with selected VPC in OpenStack. Deploy OVN-IC DB \u00b6 Start the OVN-IC DB with the following command: docker run --name = ovn-ic-db -d --network = host -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh Kubernetes Side Operations \u00b6 Create ovn-ic-config ConfigMap in kube-system Namespace \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : Whether to enable cluster interconnection. az-name : Distinguish the cluster names of different clusters, each interconnected cluster needs to be different. ic-db-host : Address of the node where the OVN-IC DB is deployed. ic-nb-port : OVN-IC Northbound Database port, default 6645. ic-sb-port : OVN-IC Southbound Database port, default 6645. gw-nodes : The name of the nodes in the cluster interconnection that takes on the work of the gateways, separated by commas. auto-route : Whether to automatically publish and learn routes. OpenStack Side Operations \u00b6 Create logical routers that interconnect with Kubernetes: # openstack router create router0 # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | d5b38655-249a-4192-8046-71aa4d2b4af1 | router0 | ACTIVE | UP | 98a29ab7388347e7b5ff8bdd181ba4f9 | +--------------------------------------+---------+--------+-------+----------------------------------+ Set the availability zone name in the OVN northbound database within OpenStack, which needs to be different from the other interconnected clusters: ovn-nbctl set NB_Global . name = op-az Start the OVN-IC controller at a node that has access to the OVN-IC DB: /usr/share/ovn/scripts/ovn-ctl --ovn-ic-nb-db = tcp:192.168.65.3:6645 \\ --ovn-ic-sb-db = tcp:192.168.65.3:6646 \\ --ovn-northd-nb-db = unix:/run/ovn/ovnnb_db.sock \\ --ovn-northd-sb-db = unix:/run/ovn/ovnsb_db.sock \\ start_ic ovn-ic-nb-db \uff0c ovn-ic-sb-db : OVN-IC Northbound database and southbound database addresses. ovn-northd-nb-db \uff0c ovn-northd-sb-db : Current cluster OVN northbound database and southbound data address. Configuration gateway nodes: ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn = true The next step is to create a logical topology by operating the OVN in OpenStack. Connect the ts interconnect switch and the router0 logical router, and set the relevant rules: ovn-nbctl lrp-add router0 lrp-router0-ts 00 :02:ef:11:39:4f 169 .254.100.73/24 ovn-nbctl lsp-add ts lsp-ts-router0 -- lsp-set-addresses lsp-ts-router0 router \\ -- lsp-set-type lsp-ts-router0 router \\ -- lsp-set-options lsp-ts-router0 router-port = lrp-router0-ts ovn-nbctl lrp-set-gateway-chassis lrp-router0-ts { gateway chassis } 1000 ovn-nbctl set NB_Global . options:ic-route-adv = true options:ic-route-learn = true Verify that OpenStack has learned the Kubernetes routing rules: # ovn-nbctl lr-route-list router0 IPv4 Routes 10 .0.0.22 169 .254.100.34 dst-ip ( learned ) 10 .16.0.0/16 169 .254.100.34 dst-ip ( learned ) Next, you can create a virtual machine under the router0 network to verify that it can interconnect with Pods under Kubernetes. Shared Underlay OVN \u00b6 In this scenario, OpenStack and Kubernetes share the same OVN, so concepts such as VPC and Subnet can be pulled together for better control and interconnection. In this mode we deploy the OVN normally using Kube-OVN, and OpenStack modifies the Neutron configuration to connect to the same OVN DB. OpenStack requires networking-ovn as a Neutron backend implementation. Neutron Modification \u00b6 Modify the Neutron configuration file /etc/neutron/plugins/ml2/ml2_conf.ini \uff1a [ ovn ] ... ovn_nb_connection = tcp: [ 192 .168.137.176 ] :6641,tcp: [ 192 .168.137.177 ] :6641,tcp: [ 192 .168.137.178 ] :6641 ovn_sb_connection = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovn_l3_scheduler = OVN_L3_SCHEDULER ovn_nb_connection \uff0c ovn_sb_connection : The address needs to be changed to the address of the ovn-central nodes deployed by Kube-OVN. Modify the OVS configuration for each node: ovs-vsctl set open . external-ids:ovn-remote = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovs-vsctl set open . external-ids:ovn-encap-type = geneve ovs-vsctl set open . external-ids:ovn-encap-ip = 192 .168.137.200 external-ids:ovn-remote : The address needs to be changed to the address of the ovn-central nodes deployed by Kube-OVN. ovn-encap-ip : Change to the IP address of the current node. Using OpenStack Internal Resources in Kubernetes \u00b6 The next section describes how to query OpenStack's network resources in Kubernetes and create Pods in the subnet from OpenStack. Query the existing network resources in OpenStack for the following resources that have been pre-created. # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | 22040ed5-0598-4f77-bffd-e7fd4db47e93 | router0 | ACTIVE | UP | 62381a21d569404aa236a5dd8712449c | +--------------------------------------+---------+--------+-------+----------------------------------+ # openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | cd59e36a-37db-4c27-b709-d35379a7920f | provider | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | +--------------------------------------+----------+--------------------------------------+ # openstack subnet list +--------------------------------------+-------------+--------------------------------------+----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------------+--------------------------------------+----------------+ | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | provider-v4 | cd59e36a-37db-4c27-b709-d35379a7920f | 192 .168.1.0/24 | +--------------------------------------+-------------+--------------------------------------+----------------+ # openstack server list +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | 8433d622-a8d6-41a7-8b31-49abfd64f639 | provider-instance | ACTIVE | provider = 192 .168.1.61 | ubuntu | m1 | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ On the Kubernetes side, query the VPC resources from OpenStack: # kubectl get vpc NAME STANDBY SUBNETS neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 true [ \"neutron-cd59e36a-37db-4c27-b709-d35379a7920f\" ] ovn-cluster true [ \"join\" , \"ovn-default\" ] neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 is the VPC resources synchronized from OpenStack. Next, you can create Pods and run them according to Kube-OVN's native VPC and Subnet operations. Bind VPC, Subnet to Namespace net2 and create Pod: apiVersion : v1 kind : Namespace metadata : name : net2 --- apiVersion : kubeovn.io/v1 kind : Vpc metadata : creationTimestamp : \"2021-06-20T13:34:11Z\" generation : 2 labels : ovn.kubernetes.io/vpc_external : \"true\" name : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 resourceVersion : \"583728\" uid : 18d4c654-f511-4def-a3a0-a6434d237c1e spec : namespaces : - net2 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 namespaces : - net2 cidrBlock : 12.0.1.0/24 natOutgoing : false --- apiVersion : v1 kind : Pod metadata : name : ubuntu namespace : net2 spec : containers : - image : docker.io/kubeovn/kube-ovn:v1.8.0 command : - \"sleep\" - \"604800\" imagePullPolicy : IfNotPresent name : ubuntu restartPolicy : Always \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Integration with OpenStack"},{"location":"en/advance/with-openstack/#integration-with-openstack","text":"In some cases, users need to run virtual machines with OpenStack and containers with Kubernetes, and need the network to interoperate between containers and virtual machines and be under a unified control plane. If the OpenStack Neutron side also uses OVN as the underlying network, then Kube-OVN can use either cluster interconnection or shared underlying OVN to connect the OpenStack and Kubernetes networks.","title":"Integration with OpenStack"},{"location":"en/advance/with-openstack/#cluster-interconnection","text":"This pattern is similar to Cluster Inter-Connection with OVN-IC to connect two Kubernetes cluster networks, except that the two ends of the cluster are replaced with OpenStack and Kubernetes\u3002","title":"Cluster Interconnection"},{"location":"en/advance/with-openstack/#prerequisites","text":"The subnet CIDRs within OpenStack and Kubernetes cannot overlap with each other in auto-route mode. A set of machines needs to exist that can be accessed by each cluster over the network and used to deploy controllers that interconnect across clusters. Each cluster needs to have a set of machines that can access each other across clusters via IP as the gateway nodes. This solution only connects to the Kubernetes default subnet with selected VPC in OpenStack.","title":"Prerequisites"},{"location":"en/advance/with-openstack/#deploy-ovn-ic-db","text":"Start the OVN-IC DB with the following command: docker run --name = ovn-ic-db -d --network = host -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh","title":"Deploy OVN-IC DB"},{"location":"en/advance/with-openstack/#kubernetes-side-operations","text":"Create ovn-ic-config ConfigMap in kube-system Namespace \uff1a apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : Whether to enable cluster interconnection. az-name : Distinguish the cluster names of different clusters, each interconnected cluster needs to be different. ic-db-host : Address of the node where the OVN-IC DB is deployed. ic-nb-port : OVN-IC Northbound Database port, default 6645. ic-sb-port : OVN-IC Southbound Database port, default 6645. gw-nodes : The name of the nodes in the cluster interconnection that takes on the work of the gateways, separated by commas. auto-route : Whether to automatically publish and learn routes.","title":"Kubernetes Side Operations"},{"location":"en/advance/with-openstack/#openstack-side-operations","text":"Create logical routers that interconnect with Kubernetes: # openstack router create router0 # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | d5b38655-249a-4192-8046-71aa4d2b4af1 | router0 | ACTIVE | UP | 98a29ab7388347e7b5ff8bdd181ba4f9 | +--------------------------------------+---------+--------+-------+----------------------------------+ Set the availability zone name in the OVN northbound database within OpenStack, which needs to be different from the other interconnected clusters: ovn-nbctl set NB_Global . name = op-az Start the OVN-IC controller at a node that has access to the OVN-IC DB: /usr/share/ovn/scripts/ovn-ctl --ovn-ic-nb-db = tcp:192.168.65.3:6645 \\ --ovn-ic-sb-db = tcp:192.168.65.3:6646 \\ --ovn-northd-nb-db = unix:/run/ovn/ovnnb_db.sock \\ --ovn-northd-sb-db = unix:/run/ovn/ovnsb_db.sock \\ start_ic ovn-ic-nb-db \uff0c ovn-ic-sb-db : OVN-IC Northbound database and southbound database addresses. ovn-northd-nb-db \uff0c ovn-northd-sb-db : Current cluster OVN northbound database and southbound data address. Configuration gateway nodes: ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn = true The next step is to create a logical topology by operating the OVN in OpenStack. Connect the ts interconnect switch and the router0 logical router, and set the relevant rules: ovn-nbctl lrp-add router0 lrp-router0-ts 00 :02:ef:11:39:4f 169 .254.100.73/24 ovn-nbctl lsp-add ts lsp-ts-router0 -- lsp-set-addresses lsp-ts-router0 router \\ -- lsp-set-type lsp-ts-router0 router \\ -- lsp-set-options lsp-ts-router0 router-port = lrp-router0-ts ovn-nbctl lrp-set-gateway-chassis lrp-router0-ts { gateway chassis } 1000 ovn-nbctl set NB_Global . options:ic-route-adv = true options:ic-route-learn = true Verify that OpenStack has learned the Kubernetes routing rules: # ovn-nbctl lr-route-list router0 IPv4 Routes 10 .0.0.22 169 .254.100.34 dst-ip ( learned ) 10 .16.0.0/16 169 .254.100.34 dst-ip ( learned ) Next, you can create a virtual machine under the router0 network to verify that it can interconnect with Pods under Kubernetes.","title":"OpenStack Side Operations"},{"location":"en/advance/with-openstack/#shared-underlay-ovn","text":"In this scenario, OpenStack and Kubernetes share the same OVN, so concepts such as VPC and Subnet can be pulled together for better control and interconnection. In this mode we deploy the OVN normally using Kube-OVN, and OpenStack modifies the Neutron configuration to connect to the same OVN DB. OpenStack requires networking-ovn as a Neutron backend implementation.","title":"Shared Underlay OVN"},{"location":"en/advance/with-openstack/#neutron-modification","text":"Modify the Neutron configuration file /etc/neutron/plugins/ml2/ml2_conf.ini \uff1a [ ovn ] ... ovn_nb_connection = tcp: [ 192 .168.137.176 ] :6641,tcp: [ 192 .168.137.177 ] :6641,tcp: [ 192 .168.137.178 ] :6641 ovn_sb_connection = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovn_l3_scheduler = OVN_L3_SCHEDULER ovn_nb_connection \uff0c ovn_sb_connection : The address needs to be changed to the address of the ovn-central nodes deployed by Kube-OVN. Modify the OVS configuration for each node: ovs-vsctl set open . external-ids:ovn-remote = tcp: [ 192 .168.137.176 ] :6642,tcp: [ 192 .168.137.177 ] :6642,tcp: [ 192 .168.137.178 ] :6642 ovs-vsctl set open . external-ids:ovn-encap-type = geneve ovs-vsctl set open . external-ids:ovn-encap-ip = 192 .168.137.200 external-ids:ovn-remote : The address needs to be changed to the address of the ovn-central nodes deployed by Kube-OVN. ovn-encap-ip : Change to the IP address of the current node.","title":"Neutron Modification"},{"location":"en/advance/with-openstack/#using-openstack-internal-resources-in-kubernetes","text":"The next section describes how to query OpenStack's network resources in Kubernetes and create Pods in the subnet from OpenStack. Query the existing network resources in OpenStack for the following resources that have been pre-created. # openstack router list +--------------------------------------+---------+--------+-------+----------------------------------+ | ID | Name | Status | State | Project | +--------------------------------------+---------+--------+-------+----------------------------------+ | 22040ed5-0598-4f77-bffd-e7fd4db47e93 | router0 | ACTIVE | UP | 62381a21d569404aa236a5dd8712449c | +--------------------------------------+---------+--------+-------+----------------------------------+ # openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | cd59e36a-37db-4c27-b709-d35379a7920f | provider | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | +--------------------------------------+----------+--------------------------------------+ # openstack subnet list +--------------------------------------+-------------+--------------------------------------+----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------------+--------------------------------------+----------------+ | 01d73d9f-fdaa-426c-9b60-aa34abbfacae | provider-v4 | cd59e36a-37db-4c27-b709-d35379a7920f | 192 .168.1.0/24 | +--------------------------------------+-------------+--------------------------------------+----------------+ # openstack server list +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ | 8433d622-a8d6-41a7-8b31-49abfd64f639 | provider-instance | ACTIVE | provider = 192 .168.1.61 | ubuntu | m1 | +--------------------------------------+-------------------+--------+-----------------------+--------+--------+ On the Kubernetes side, query the VPC resources from OpenStack: # kubectl get vpc NAME STANDBY SUBNETS neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 true [ \"neutron-cd59e36a-37db-4c27-b709-d35379a7920f\" ] ovn-cluster true [ \"join\" , \"ovn-default\" ] neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 is the VPC resources synchronized from OpenStack. Next, you can create Pods and run them according to Kube-OVN's native VPC and Subnet operations. Bind VPC, Subnet to Namespace net2 and create Pod: apiVersion : v1 kind : Namespace metadata : name : net2 --- apiVersion : kubeovn.io/v1 kind : Vpc metadata : creationTimestamp : \"2021-06-20T13:34:11Z\" generation : 2 labels : ovn.kubernetes.io/vpc_external : \"true\" name : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 resourceVersion : \"583728\" uid : 18d4c654-f511-4def-a3a0-a6434d237c1e spec : namespaces : - net2 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : neutron-22040ed5-0598-4f77-bffd-e7fd4db47e93 namespaces : - net2 cidrBlock : 12.0.1.0/24 natOutgoing : false --- apiVersion : v1 kind : Pod metadata : name : ubuntu namespace : net2 spec : containers : - image : docker.io/kubeovn/kube-ovn:v1.8.0 command : - \"sleep\" - \"604800\" imagePullPolicy : IfNotPresent name : ubuntu restartPolicy : Always \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Using OpenStack Internal Resources in Kubernetes"},{"location":"en/advance/with-ovn-ic/","text":"Cluster Inter-Connection with OVN-IC \u00b6 Kube-OVN supports interconnecting two Kubernetes cluster Pod networks via OVN-IC , and the Pods in the two clusters can communicate directly via Pod IPs . Kube-OVN uses tunnels to encapsulate cross-cluster traffic, allowing container networks to interconnect between two clusters as long as there is a set of IP reachable machines. This mode of multi-cluster interconnection is for Overlay network. For Underlay network, it needs the underlying infrastructure to do the inter-connection work. Prerequisites \u00b6 The subnet CIDRs within OpenStack and Kubernetes cannot overlap with each other in auto-interconnect mode. If there is overlap, you need to refer to the subsequent manual interconnection process, which can only connect non-overlapping Subnets. A set of machines needs to exist that can be accessed by each cluster over the network and used to deploy controllers that interconnect across clusters. Each cluster needs to have a set of machines that can access each other across clusters via IP as the gateway nodes. This solution only connects to the Kubernetes default VPCs. Deploy a single-node OVN-IC DB \u00b6 Deploy the OVN-IC DB on a machine accessible by kube-ovn-controller , This DB will hold the network configuration information synchronized up from each cluster. An environment deploying docker can start the OVN-IC DB with the following command. docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh For deploying a containerd environment instead of docker you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh Automatic Routing Mode \u00b6 In auto-routing mode, each cluster synchronizes the CIDR information of the Subnet under its own default VPC to OVN-IC , so make sure there is no overlap between the Subnet CIDRs of the two clusters. Create ovn-ic-config ConfigMap in kube-system Namespace: apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : Whether to enable cluster interconnection. az-name : Distinguish the cluster names of different clusters, each interconnected cluster needs to be different. ic-db-host : Address of the node where the OVN-IC DB is deployed. ic-nb-port : OVN-IC Northbound Database port, default 6645. ic-sb-port : OVN-IC Southbound Database port, default 6645. gw-nodes : The name of the nodes in the cluster interconnection that takes on the work of the gateways, separated by commas. auto-route : Whether to automatically publish and learn routes. Note: To ensure the correct operation, the ConfigMap ovn-ic-config is not allowed to be modified. If any parameter needs to be changed, please delete this ConfigMap, modify it and then apply it again. Check if the interconnected logical switch ts has been established in the ovn-ic container with the following command\uff1a # ovn-ic-sbctl show availability-zone az1 gateway deee03e0-af16-4f45-91e9-b50c3960f809 hostname: az1-gw type: geneve ip: 192 .168.42.145 port ts-az1 transit switch: ts address: [ \"00:00:00:50:AC:8C 169.254.100.45/24\" ] availability-zone az2 gateway e94cc831-8143-40e3-a478-90352773327b hostname: az2-gw type: geneve ip: 192 .168.42.149 port ts-az2 transit switch: ts address: [ \"00:00:00:07:4A:59 169.254.100.63/24\" ] At each cluster observe if logical routes have learned peer routes: # kubectl ko nbctl lr-route-list ovn-cluster IPv4 Routes 10 .42.1.1 169 .254.100.45 dst-ip ( learned ) 10 .42.1.3 100 .64.0.2 dst-ip 10 .16.0.2 100 .64.0.2 src-ip 10 .16.0.3 100 .64.0.2 src-ip 10 .16.0.4 100 .64.0.2 src-ip 10 .16.0.6 100 .64.0.2 src-ip 10 .17.0.0/16 169 .254.100.45 dst-ip ( learned ) 100 .65.0.0/16 169 .254.100.45 dst-ip ( learned ) Next, you can try ping a Pod IP in Cluster 1 directly from a Pod in Cluster 2 to see if you can work. For a subnet that does not want to automatically publish routes to the other end, you can disable route broadcasting by modifying disableInterConnection in the Subnet spec. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : no-advertise spec : cidrBlock : 10.199.0.0/16 disableInterConnection : true Manual Routing Mode \u00b6 For cases where there are overlapping CIDRs between clusters, and you only want to do partial subnet interconnection, you can manually publish subnet routing by following the steps below. Create ovn-ic-config ConfigMap in kube-system Namespace, and set auto-route to false : apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"false\" Find the address of the remote logical ports in each cluster separately, for later manual configuration of the route: [ root@az1 ~ ] # kubectl ko nbctl show switch a391d3a1-14a0-4841-9836-4bd930c447fb ( ts ) port ts-az1 type: router router-port: az1-ts port ts-az2 type: remote addresses: [ \"00:00:00:4B:E2:9F 169.254.100.31/24\" ] [ root@az2 ~ ] # kubectl ko nbctl show switch da6138b8-de81-4908-abf9-b2224ec4edf3 ( ts ) port ts-az2 type: router router-port: az2-ts port ts-az1 type: remote addresses: [ \"00:00:00:FB:2A:F7 169.254.100.79/24\" ] The output above shows that the remote address from cluster az1 to cluster az2 is 169.254.100.31 and the remote address from az2 to az1 is 169.254.100.79 . In this example, the subnet CIDR within cluster az1 is 10.16.0.0/24 and the subnet CIDR within cluster az2 is 10.17.0.0/24 . Set up a route from cluster az1 to cluster az2 in cluster az1 : kubectl ko nbctl lr-route-add ovn-cluster 10 .17.0.0/24 169 .254.100.31 Set up a route to cluster az1 in cluster az2 : kubectl ko nbctl lr-route-add ovn-cluster 10 .16.0.0/24 169 .254.100.79 Highly Available OVN-IC DB Installation \u00b6 A highly available cluster can be formed between OVN-IC DB via the Raft protocol, which requires a minimum of 3 nodes for this deployment model. First start the leader of the OVN-IC DB on the first node. Users deploying a docker environment can use the following command: docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.3\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh If you are using containerd you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a The IP address of the node where the current container is located. NODE_IPS \uff1a The IP addresses of the three nodes running the OVN-IC database, separated by commas. Next, deploy the follower of the OVN-IC DB on the other two nodes. docker environment can use the following command. docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.2\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" -e LEADER_IP = \"192.168.65.3\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh If using containerd you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.2 \"\" --env = \"LEADER_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a The IP address of the node where the current container is located. NODE_IPS \uff1a The IP addresses of the three nodes running the OVN-IC database, separated by commas. LEADER_IP : The IP address of the OVN-IC DB leader node. Specify multiple OVN-IC database node addresses when creating ovn-ic-config for each cluster: apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3,192.168.65.2,192.168.65.1\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" Manual Reset \u00b6 In some cases, the entire interconnection configuration needs to be cleaned up due to configuration errors, you can refer to the following steps to clean up your environment. Delete the current ovn-ic-config Configmap: kubectl -n kube-system delete cm ovn-ic-config Delete ts logical switch: kubectl-ko nbctl ls-del ts Repeat the same steps at the peer cluster. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cluster Inter-Connection with OVN-IC"},{"location":"en/advance/with-ovn-ic/#cluster-inter-connection-with-ovn-ic","text":"Kube-OVN supports interconnecting two Kubernetes cluster Pod networks via OVN-IC , and the Pods in the two clusters can communicate directly via Pod IPs . Kube-OVN uses tunnels to encapsulate cross-cluster traffic, allowing container networks to interconnect between two clusters as long as there is a set of IP reachable machines. This mode of multi-cluster interconnection is for Overlay network. For Underlay network, it needs the underlying infrastructure to do the inter-connection work.","title":"Cluster Inter-Connection with OVN-IC"},{"location":"en/advance/with-ovn-ic/#prerequisites","text":"The subnet CIDRs within OpenStack and Kubernetes cannot overlap with each other in auto-interconnect mode. If there is overlap, you need to refer to the subsequent manual interconnection process, which can only connect non-overlapping Subnets. A set of machines needs to exist that can be accessed by each cluster over the network and used to deploy controllers that interconnect across clusters. Each cluster needs to have a set of machines that can access each other across clusters via IP as the gateway nodes. This solution only connects to the Kubernetes default VPCs.","title":"Prerequisites"},{"location":"en/advance/with-ovn-ic/#deploy-a-single-node-ovn-ic-db","text":"Deploy the OVN-IC DB on a machine accessible by kube-ovn-controller , This DB will hold the network configuration information synchronized up from each cluster. An environment deploying docker can start the OVN-IC DB with the following command. docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh For deploying a containerd environment instead of docker you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh","title":"Deploy a single-node OVN-IC DB"},{"location":"en/advance/with-ovn-ic/#automatic-routing-mode","text":"In auto-routing mode, each cluster synchronizes the CIDR information of the Subnet under its own default VPC to OVN-IC , so make sure there is no overlap between the Subnet CIDRs of the two clusters. Create ovn-ic-config ConfigMap in kube-system Namespace: apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\" enable-ic : Whether to enable cluster interconnection. az-name : Distinguish the cluster names of different clusters, each interconnected cluster needs to be different. ic-db-host : Address of the node where the OVN-IC DB is deployed. ic-nb-port : OVN-IC Northbound Database port, default 6645. ic-sb-port : OVN-IC Southbound Database port, default 6645. gw-nodes : The name of the nodes in the cluster interconnection that takes on the work of the gateways, separated by commas. auto-route : Whether to automatically publish and learn routes. Note: To ensure the correct operation, the ConfigMap ovn-ic-config is not allowed to be modified. If any parameter needs to be changed, please delete this ConfigMap, modify it and then apply it again. Check if the interconnected logical switch ts has been established in the ovn-ic container with the following command\uff1a # ovn-ic-sbctl show availability-zone az1 gateway deee03e0-af16-4f45-91e9-b50c3960f809 hostname: az1-gw type: geneve ip: 192 .168.42.145 port ts-az1 transit switch: ts address: [ \"00:00:00:50:AC:8C 169.254.100.45/24\" ] availability-zone az2 gateway e94cc831-8143-40e3-a478-90352773327b hostname: az2-gw type: geneve ip: 192 .168.42.149 port ts-az2 transit switch: ts address: [ \"00:00:00:07:4A:59 169.254.100.63/24\" ] At each cluster observe if logical routes have learned peer routes: # kubectl ko nbctl lr-route-list ovn-cluster IPv4 Routes 10 .42.1.1 169 .254.100.45 dst-ip ( learned ) 10 .42.1.3 100 .64.0.2 dst-ip 10 .16.0.2 100 .64.0.2 src-ip 10 .16.0.3 100 .64.0.2 src-ip 10 .16.0.4 100 .64.0.2 src-ip 10 .16.0.6 100 .64.0.2 src-ip 10 .17.0.0/16 169 .254.100.45 dst-ip ( learned ) 100 .65.0.0/16 169 .254.100.45 dst-ip ( learned ) Next, you can try ping a Pod IP in Cluster 1 directly from a Pod in Cluster 2 to see if you can work. For a subnet that does not want to automatically publish routes to the other end, you can disable route broadcasting by modifying disableInterConnection in the Subnet spec. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : no-advertise spec : cidrBlock : 10.199.0.0/16 disableInterConnection : true","title":"Automatic Routing Mode"},{"location":"en/advance/with-ovn-ic/#manual-routing-mode","text":"For cases where there are overlapping CIDRs between clusters, and you only want to do partial subnet interconnection, you can manually publish subnet routing by following the steps below. Create ovn-ic-config ConfigMap in kube-system Namespace, and set auto-route to false : apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"false\" Find the address of the remote logical ports in each cluster separately, for later manual configuration of the route: [ root@az1 ~ ] # kubectl ko nbctl show switch a391d3a1-14a0-4841-9836-4bd930c447fb ( ts ) port ts-az1 type: router router-port: az1-ts port ts-az2 type: remote addresses: [ \"00:00:00:4B:E2:9F 169.254.100.31/24\" ] [ root@az2 ~ ] # kubectl ko nbctl show switch da6138b8-de81-4908-abf9-b2224ec4edf3 ( ts ) port ts-az2 type: router router-port: az2-ts port ts-az1 type: remote addresses: [ \"00:00:00:FB:2A:F7 169.254.100.79/24\" ] The output above shows that the remote address from cluster az1 to cluster az2 is 169.254.100.31 and the remote address from az2 to az1 is 169.254.100.79 . In this example, the subnet CIDR within cluster az1 is 10.16.0.0/24 and the subnet CIDR within cluster az2 is 10.17.0.0/24 . Set up a route from cluster az1 to cluster az2 in cluster az1 : kubectl ko nbctl lr-route-add ovn-cluster 10 .17.0.0/24 169 .254.100.31 Set up a route to cluster az1 in cluster az2 : kubectl ko nbctl lr-route-add ovn-cluster 10 .16.0.0/24 169 .254.100.79","title":"Manual Routing Mode"},{"location":"en/advance/with-ovn-ic/#highly-available-ovn-ic-db-installation","text":"A highly available cluster can be formed between OVN-IC DB via the Raft protocol, which requires a minimum of 3 nodes for this deployment model. First start the leader of the OVN-IC DB on the first node. Users deploying a docker environment can use the following command: docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.3\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh If you are using containerd you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a The IP address of the node where the current container is located. NODE_IPS \uff1a The IP addresses of the three nodes running the OVN-IC database, separated by commas. Next, deploy the follower of the OVN-IC DB on the other two nodes. docker environment can use the following command. docker run --name = ovn-ic-db -d --network = host --privileged -v /etc/ovn/:/etc/ovn -v /var/run/ovn:/var/run/ovn -v /var/log/ovn:/var/log/ovn -e LOCAL_IP = \"192.168.65.2\" -e NODE_IPS = \"192.168.65.3,192.168.65.2,192.168.65.1\" -e LEADER_IP = \"192.168.65.3\" kubeovn/kube-ovn:v1.12.0 bash start-ic-db.sh If using containerd you can use the following command: ctr -n k8s.io run -d --net-host --privileged --mount = \"type=bind,src=/etc/ovn/,dst=/etc/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/run/ovn,dst=/var/run/ovn,options=rbind:rw\" --mount = \"type=bind,src=/var/log/ovn,dst=/var/log/ovn,options=rbind:rw\" --env = \"NODE_IPS=\" 192 .168.65.3,192.168.65.2,192.168.65.1 \"\" --env = \"LOCAL_IP=\" 192 .168.65.2 \"\" --env = \"LEADER_IP=\" 192 .168.65.3 \"\" docker.io/kubeovn/kube-ovn:v1.12.0 ovn-ic-db bash start-ic-db.sh LOCAL_IP \uff1a The IP address of the node where the current container is located. NODE_IPS \uff1a The IP addresses of the three nodes running the OVN-IC database, separated by commas. LEADER_IP : The IP address of the OVN-IC DB leader node. Specify multiple OVN-IC database node addresses when creating ovn-ic-config for each cluster: apiVersion : v1 kind : ConfigMap metadata : name : ovn-ic-config namespace : kube-system data : enable-ic : \"true\" az-name : \"az1\" ic-db-host : \"192.168.65.3,192.168.65.2,192.168.65.1\" ic-nb-port : \"6645\" ic-sb-port : \"6646\" gw-nodes : \"az1-gw\" auto-route : \"true\"","title":"Highly Available OVN-IC DB Installation"},{"location":"en/advance/with-ovn-ic/#manual-reset","text":"In some cases, the entire interconnection configuration needs to be cleaned up due to configuration errors, you can refer to the following steps to clean up your environment. Delete the current ovn-ic-config Configmap: kubectl -n kube-system delete cm ovn-ic-config Delete ts logical switch: kubectl-ko nbctl ls-del ts Repeat the same steps at the peer cluster. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Manual Reset"},{"location":"en/advance/with-submariner/","text":"Cluster Inter-Connection with Submariner \u00b6 Submariner is an open source networking component that connects multiple Kubernetes cluster Pod and Service networks which can help Kube-OVN interconnect multiple clusters. Compared to OVN-IC , Submariner can connect Kube-OVN and non-Kube-OVN cluster networks, and Submariner can provide cross-cluster capability for services. However, Submariner currently only enables the default subnets to be connected, and cannot selectively connect multiple subnets. Prerequisites \u00b6 The Service CIDRs of the two clusters and the CIDR of the default Subnet cannot overlap. Install Submariner \u00b6 Download the subctl binary and deploy it to the appropriate path: curl -Ls https://get.submariner.io | bash export PATH = $PATH :~/.local/bin echo export PATH = \\$ PATH:~/.local/bin >> ~/.profile Change kubeconfig context to the cluster that need to deploy submariner-broker : subctl deploy-broker In this document the default subnet CIDR for cluster0 is 10.16.0.0/16 and the join subnet CIDR for cluster0 is 100.64.0.0/16 , the default subnet CIDR for cluster1 is 11.16.0.0/16 and the join subnet CIDR for cluster1 is 100.68.0.0/16 . Switch kubeconfig to cluster0 to register the cluster to the broker, and register the gateway node: subctl join broker-info.subm --clusterid cluster0 --clustercidr 100 .64.0.0/16,10.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster0 submariner.io/gateway = true Switch kubeconfig to cluster1 to register the cluster to the broker, and register the gateway node: subctl join broker-info.subm --clusterid cluster1 --clustercidr 100 .68.0.0/16,11.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster1 submariner.io/gateway = true Next, you can start Pods in each of the two clusters and try to access each other using IPs. Network communication problems can be diagnosed by using the subctl command: subctl show all subctl diagnose all For more Submariner operations please read Submariner Usage . \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Cluster Inter-Connection with Submariner"},{"location":"en/advance/with-submariner/#cluster-inter-connection-with-submariner","text":"Submariner is an open source networking component that connects multiple Kubernetes cluster Pod and Service networks which can help Kube-OVN interconnect multiple clusters. Compared to OVN-IC , Submariner can connect Kube-OVN and non-Kube-OVN cluster networks, and Submariner can provide cross-cluster capability for services. However, Submariner currently only enables the default subnets to be connected, and cannot selectively connect multiple subnets.","title":"Cluster Inter-Connection with Submariner"},{"location":"en/advance/with-submariner/#prerequisites","text":"The Service CIDRs of the two clusters and the CIDR of the default Subnet cannot overlap.","title":"Prerequisites"},{"location":"en/advance/with-submariner/#install-submariner","text":"Download the subctl binary and deploy it to the appropriate path: curl -Ls https://get.submariner.io | bash export PATH = $PATH :~/.local/bin echo export PATH = \\$ PATH:~/.local/bin >> ~/.profile Change kubeconfig context to the cluster that need to deploy submariner-broker : subctl deploy-broker In this document the default subnet CIDR for cluster0 is 10.16.0.0/16 and the join subnet CIDR for cluster0 is 100.64.0.0/16 , the default subnet CIDR for cluster1 is 11.16.0.0/16 and the join subnet CIDR for cluster1 is 100.68.0.0/16 . Switch kubeconfig to cluster0 to register the cluster to the broker, and register the gateway node: subctl join broker-info.subm --clusterid cluster0 --clustercidr 100 .64.0.0/16,10.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster0 submariner.io/gateway = true Switch kubeconfig to cluster1 to register the cluster to the broker, and register the gateway node: subctl join broker-info.subm --clusterid cluster1 --clustercidr 100 .68.0.0/16,11.16.0.0/16 --natt = false --cable-driver vxlan --health-check = false kubectl label nodes cluster1 submariner.io/gateway = true Next, you can start Pods in each of the two clusters and try to access each other using IPs. Network communication problems can be diagnosed by using the subctl command: subctl show all subctl diagnose all For more Submariner operations please read Submariner Usage . \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Install Submariner"},{"location":"en/guide/custom-routes/","text":"Custom Routes \u00b6 Custom routes can be configured via Pod's annotations. Here is an example: apiVersion : v1 kind : Pod metadata : name : custom-routes annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine Do not set the dst field if you want to configure the default route. For workloads such as Deployment, DaemonSet and StatefulSet, custom routes must be configured via .spec.template.metadata.annotations : apiVersion : apps/v1 kind : Deployment metadata : name : custom-routes labels : app : nginx spec : replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Custom Routes"},{"location":"en/guide/custom-routes/#custom-routes","text":"Custom routes can be configured via Pod's annotations. Here is an example: apiVersion : v1 kind : Pod metadata : name : custom-routes annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine Do not set the dst field if you want to configure the default route. For workloads such as Deployment, DaemonSet and StatefulSet, custom routes must be configured via .spec.template.metadata.annotations : apiVersion : apps/v1 kind : Deployment metadata : name : custom-routes labels : app : nginx spec : replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx annotations : ovn.kubernetes.io/routes : | [{ \"dst\": \"192.168.0.101/24\", \"gw\": \"10.16.0.254\" }, { \"gw\": \"10.16.0.254\" }]' spec : containers : - name : nginx image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Custom Routes"},{"location":"en/guide/dual-stack/","text":"DualStack \u00b6 Different subnets in Kube-OVN can support different IP protocols. IPv4, IPv6 and dual-stack types of subnets can exist within one cluster. However, it is recommended to use a uniform protocol type within a cluster to simplify usage and maintenance. In order to support dual-stack, the host network needs to meet the dual-stack requirements, and the Kubernetes-related parameters need to be adjusted, please refer to official guide to dual-stack . Create dual-stack Subnet \u00b6 When configuring a dual stack Subnet, you only need to set the corresponding subnet CIDR format as cidr=<IPv4 CIDR>,<IPv6 CIDR> . The CIDR order requires IPv4 to come first and IPv6 to come second, as follows. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-test spec : cidrBlock : 10.16.0.0/16,fd00:10:16::/64 excludeIps : - 10.16.0.1 - fd00:10:16::1 gateway : 10.16.0.1,fd00:10:16::1 If you need to use a dual stack for the default subnet during installation, you need to change the following parameters in the installation script: POD_CIDR = \"10.16.0.0/16,fd00:10:16::/64\" JOIN_CIDR = \"100.64.0.0/16,fd00:100:64::/64\" Check Pod Address \u00b6 Pods configured for dual-stack networks will be assigned both IPv4 and IPv6 addresses from that subnet, and the results will be displayed in the annotation of the Pod: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/allocated : \"true\" ovn.kubernetes.io/cidr : 10.16.0.0/16,fd00:10:16::/64 ovn.kubernetes.io/gateway : 10.16.0.1,fd00:10:16::1 ovn.kubernetes.io/ip_address : 10.16.0.9,fd00:10:16::9 ovn.kubernetes.io/logical_switch : ovn-default ovn.kubernetes.io/mac_address : 00:00:00:14:88:09 ovn.kubernetes.io/network_types : geneve ovn.kubernetes.io/routed : \"true\" ... podIP : 10.16.0.9 podIPs : - ip : 10.16.0.9 - ip : fd00:10:16::9 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"DualStack"},{"location":"en/guide/dual-stack/#dualstack","text":"Different subnets in Kube-OVN can support different IP protocols. IPv4, IPv6 and dual-stack types of subnets can exist within one cluster. However, it is recommended to use a uniform protocol type within a cluster to simplify usage and maintenance. In order to support dual-stack, the host network needs to meet the dual-stack requirements, and the Kubernetes-related parameters need to be adjusted, please refer to official guide to dual-stack .","title":"DualStack"},{"location":"en/guide/dual-stack/#create-dual-stack-subnet","text":"When configuring a dual stack Subnet, you only need to set the corresponding subnet CIDR format as cidr=<IPv4 CIDR>,<IPv6 CIDR> . The CIDR order requires IPv4 to come first and IPv6 to come second, as follows. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-test spec : cidrBlock : 10.16.0.0/16,fd00:10:16::/64 excludeIps : - 10.16.0.1 - fd00:10:16::1 gateway : 10.16.0.1,fd00:10:16::1 If you need to use a dual stack for the default subnet during installation, you need to change the following parameters in the installation script: POD_CIDR = \"10.16.0.0/16,fd00:10:16::/64\" JOIN_CIDR = \"100.64.0.0/16,fd00:100:64::/64\"","title":"Create dual-stack Subnet"},{"location":"en/guide/dual-stack/#check-pod-address","text":"Pods configured for dual-stack networks will be assigned both IPv4 and IPv6 addresses from that subnet, and the results will be displayed in the annotation of the Pod: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/allocated : \"true\" ovn.kubernetes.io/cidr : 10.16.0.0/16,fd00:10:16::/64 ovn.kubernetes.io/gateway : 10.16.0.1,fd00:10:16::1 ovn.kubernetes.io/ip_address : 10.16.0.9,fd00:10:16::9 ovn.kubernetes.io/logical_switch : ovn-default ovn.kubernetes.io/mac_address : 00:00:00:14:88:09 ovn.kubernetes.io/network_types : geneve ovn.kubernetes.io/routed : \"true\" ... podIP : 10.16.0.9 podIPs : - ip : 10.16.0.9 - ip : fd00:10:16::9 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Check Pod Address"},{"location":"en/guide/eip-snat/","text":"EIP and SNAT \u00b6 This configuration is for the network under default VPC, for custom VPC please refer to VPC Gateway Kube-OVN supports SNAT and EIP functionality at the Pod level using the L3 Gateway feature in OVN. By using SNAT, a group of Pods can share an IP address for external access. With the EIP feature, a Pod can be directly associated with an external IP. External services can access the Pod directly through the EIP, and the Pod will also access external services through this EIP. Preparation \u00b6 In order to use the OVN's L3 Gateway capability, a separate NIC must be bridged into the OVS bridge for overlay and underlay network communication. The host must have other NICs for management. Since packets passing through NAT will go directly to the Underlay network, it is important to confirm that such packets can pass safely on the current network architecture. Currently, there is no conflict detection for EIP and SNAT addresses, and an administrator needs to manually assign them to avoid address conflicts. Create Config \u00b6 Create ConfigMap ovn-external-gw-config in kube-system Namespace: apiVersion : v1 kind : ConfigMap metadata : name : ovn-external-gw-config namespace : kube-system data : enable-external-gw : \"true\" external-gw-nodes : \"kube-ovn-worker\" external-gw-nic : \"eth1\" external-gw-addr : \"172.56.0.1/16\" nic-ip : \"172.56.0.254/16\" nic-mac : \"16:52:f3:13:6a:25\" enable-external-gw : Whether to enable SNAT and EIP functions. type : centrailized or distributed \uff0c Default is centralized If distributed is used, all nodes of the cluster need to have the same name NIC to perform the gateway function. external-gw-nodes : In centralized mode\uff0cThe names of the node performing the gateway role, comma separated. external-gw-nic : The name of the NIC that performs the role of a gateway on the node. external-gw-addr : The IP and mask of the physical network gateway. nic-ip , nic-mac : The IP and Mac assigned to the logical gateway port needs to be an unoccupied IP and Mac for the physical subnet. Confirm the Configuration Take Effect \u00b6 Check the OVN-NB status to confirm that the ovn-external logical switch exists and that the correct address and chassis are bound to the ovn-cluster-ovn-external logical router port. # kubectl ko nbctl show switch 3de4cea7-1a71-43f3-8b62-435a57ef16a6 ( ovn-external ) port ln-ovn-external type: localnet addresses: [ \"unknown\" ] port ovn-external-ovn-cluster type: router router-port: ovn-cluster-ovn-external router e1eb83ad-34be-4ed5-9a02-fcc8b1d357c4 ( ovn-cluster ) port ovn-cluster-ovn-external mac: \"ac:1f:6b:2d:33:f1\" networks: [ \"172.56.0.100/16\" ] gateway chassis: [ a5682814-2e2c-46dd-9c1c-6803ef0dab66 ] Check the OVS status to confirm that the corresponding NIC is bridged into the br-external bridge: # kubectl ko vsctl ${gateway node name} show e7d81150-7743-4d6e-9e6f-5c688232e130 Bridge br-external Port br-external Interface br-external type: internal Port eno2 Interface eno2 Port patch-ln-ovn-external-to-br-int Interface patch-ln-ovn-external-to-br-int type: patch options: { peer = patch-br-int-to-ln-ovn-external } Config EIP amd SNAT on Pod \u00b6 SNAT and EIP can be configured by adding the ovn.kubernetes.io/snat or ovn.kubernetes.io/eip annotation to the Pod, respectively: apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/snat : 172.56.0.200 spec : containers : - name : snat-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/eip : 172.56.0.233 spec : containers : - name : eip-pod image : docker.io/library/nginx:alpine The EIP or SNAT rules configured by the Pod can be dynamically adjusted via kubectl or other tools, remember to remove the ovn.kubernetes.io/routed annotation to trigger the routing change. kubectl annotate pod pod-gw ovn.kubernetes.io/eip = 172 .56.0.221 --overwrite kubectl annotate pod pod-gw ovn.kubernetes.io/routed- When the EIP or SNAT takes into effect, the ovn.kubernetes.io/routed annotation will be added back. Advanced Configuration \u00b6 Some args of kube-ovn-controller allow for advanced configuration of SNAT and EIP: --external-gateway-config-ns : The Namespace of Configmap ovn-external-gw-config , default is kube-system \u3002 --external-gateway-net : The name of the bridge to which the physical NIC is bridged, default is external . --external-gateway-vlanid : Physical network Vlan Tag number, default is 0, i.e. no Vlan is used. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"EIP and SNAT"},{"location":"en/guide/eip-snat/#eip-and-snat","text":"This configuration is for the network under default VPC, for custom VPC please refer to VPC Gateway Kube-OVN supports SNAT and EIP functionality at the Pod level using the L3 Gateway feature in OVN. By using SNAT, a group of Pods can share an IP address for external access. With the EIP feature, a Pod can be directly associated with an external IP. External services can access the Pod directly through the EIP, and the Pod will also access external services through this EIP.","title":"EIP and SNAT"},{"location":"en/guide/eip-snat/#preparation","text":"In order to use the OVN's L3 Gateway capability, a separate NIC must be bridged into the OVS bridge for overlay and underlay network communication. The host must have other NICs for management. Since packets passing through NAT will go directly to the Underlay network, it is important to confirm that such packets can pass safely on the current network architecture. Currently, there is no conflict detection for EIP and SNAT addresses, and an administrator needs to manually assign them to avoid address conflicts.","title":"Preparation"},{"location":"en/guide/eip-snat/#create-config","text":"Create ConfigMap ovn-external-gw-config in kube-system Namespace: apiVersion : v1 kind : ConfigMap metadata : name : ovn-external-gw-config namespace : kube-system data : enable-external-gw : \"true\" external-gw-nodes : \"kube-ovn-worker\" external-gw-nic : \"eth1\" external-gw-addr : \"172.56.0.1/16\" nic-ip : \"172.56.0.254/16\" nic-mac : \"16:52:f3:13:6a:25\" enable-external-gw : Whether to enable SNAT and EIP functions. type : centrailized or distributed \uff0c Default is centralized If distributed is used, all nodes of the cluster need to have the same name NIC to perform the gateway function. external-gw-nodes : In centralized mode\uff0cThe names of the node performing the gateway role, comma separated. external-gw-nic : The name of the NIC that performs the role of a gateway on the node. external-gw-addr : The IP and mask of the physical network gateway. nic-ip , nic-mac : The IP and Mac assigned to the logical gateway port needs to be an unoccupied IP and Mac for the physical subnet.","title":"Create Config"},{"location":"en/guide/eip-snat/#confirm-the-configuration-take-effect","text":"Check the OVN-NB status to confirm that the ovn-external logical switch exists and that the correct address and chassis are bound to the ovn-cluster-ovn-external logical router port. # kubectl ko nbctl show switch 3de4cea7-1a71-43f3-8b62-435a57ef16a6 ( ovn-external ) port ln-ovn-external type: localnet addresses: [ \"unknown\" ] port ovn-external-ovn-cluster type: router router-port: ovn-cluster-ovn-external router e1eb83ad-34be-4ed5-9a02-fcc8b1d357c4 ( ovn-cluster ) port ovn-cluster-ovn-external mac: \"ac:1f:6b:2d:33:f1\" networks: [ \"172.56.0.100/16\" ] gateway chassis: [ a5682814-2e2c-46dd-9c1c-6803ef0dab66 ] Check the OVS status to confirm that the corresponding NIC is bridged into the br-external bridge: # kubectl ko vsctl ${gateway node name} show e7d81150-7743-4d6e-9e6f-5c688232e130 Bridge br-external Port br-external Interface br-external type: internal Port eno2 Interface eno2 Port patch-ln-ovn-external-to-br-int Interface patch-ln-ovn-external-to-br-int type: patch options: { peer = patch-br-int-to-ln-ovn-external }","title":"Confirm the Configuration Take Effect"},{"location":"en/guide/eip-snat/#config-eip-amd-snat-on-pod","text":"SNAT and EIP can be configured by adding the ovn.kubernetes.io/snat or ovn.kubernetes.io/eip annotation to the Pod, respectively: apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/snat : 172.56.0.200 spec : containers : - name : snat-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : name : pod-gw annotations : ovn.kubernetes.io/eip : 172.56.0.233 spec : containers : - name : eip-pod image : docker.io/library/nginx:alpine The EIP or SNAT rules configured by the Pod can be dynamically adjusted via kubectl or other tools, remember to remove the ovn.kubernetes.io/routed annotation to trigger the routing change. kubectl annotate pod pod-gw ovn.kubernetes.io/eip = 172 .56.0.221 --overwrite kubectl annotate pod pod-gw ovn.kubernetes.io/routed- When the EIP or SNAT takes into effect, the ovn.kubernetes.io/routed annotation will be added back.","title":"Config EIP amd SNAT on Pod"},{"location":"en/guide/eip-snat/#advanced-configuration","text":"Some args of kube-ovn-controller allow for advanced configuration of SNAT and EIP: --external-gateway-config-ns : The Namespace of Configmap ovn-external-gw-config , default is kube-system \u3002 --external-gateway-net : The name of the bridge to which the physical NIC is bridged, default is external . --external-gateway-vlanid : Physical network Vlan Tag number, default is 0, i.e. no Vlan is used. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Advanced Configuration"},{"location":"en/guide/ippool/","text":"Configure IPPool \u00b6 IPPool is a more granular IPAM management unit than Subnet. You can subdivide the subnet segment into multiple units through IPPool, and each unit is bound to one or more namespaces. Instructions \u00b6 Below is an example\uff1a apiVersion : kubeovn.io/v1 kind : IPPool metadata : name : pool-1 spec : subnet : ovn-default ips : - \"10.16.0.201\" - \"10.16.0.210/30\" - \"10.16.0.220..10.16.0.230\" namespaces : - ns-1 Field description: Field Usage Comment subnet Specify the subnet to which it belongs Required ips Specify IP ranges Support three formats: , and .. . Support IPv6. namespaces Specifies the bound namespaces Optional \u6ce8\u610f\u4e8b\u9879 \u00b6 To ensure compatibility with Workload Universal IP Pool Fixed Address , the name of the IP pool cannot be an IP address; The .spec.ips of the IP pool can specify an IP address beyond the scope of the subnet, but the actual effective IP address is the intersection of .spec.ips and the CIDR of the subnet; Different IP pools of the same subnet cannot contain the same (effective) IP address; The .spec.ips of the IP pool can be modified dynamically; The IP pool will inherit the reserved IP of the subnet. When randomly assigning an IP address from the IP pool, the reserved IP included in the IP pool will be skipped; When randomly assigning an IP address from a subnet, it will only be assigned from a range other than all IP pools in the subnet. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"IP\u6c60\u4f7f\u7528"},{"location":"en/guide/ippool/#configure-ippool","text":"IPPool is a more granular IPAM management unit than Subnet. You can subdivide the subnet segment into multiple units through IPPool, and each unit is bound to one or more namespaces.","title":"Configure IPPool"},{"location":"en/guide/ippool/#instructions","text":"Below is an example\uff1a apiVersion : kubeovn.io/v1 kind : IPPool metadata : name : pool-1 spec : subnet : ovn-default ips : - \"10.16.0.201\" - \"10.16.0.210/30\" - \"10.16.0.220..10.16.0.230\" namespaces : - ns-1 Field description: Field Usage Comment subnet Specify the subnet to which it belongs Required ips Specify IP ranges Support three formats: , and .. . Support IPv6. namespaces Specifies the bound namespaces Optional","title":"Instructions"},{"location":"en/guide/ippool/#_1","text":"To ensure compatibility with Workload Universal IP Pool Fixed Address , the name of the IP pool cannot be an IP address; The .spec.ips of the IP pool can specify an IP address beyond the scope of the subnet, but the actual effective IP address is the intersection of .spec.ips and the CIDR of the subnet; Different IP pools of the same subnet cannot contain the same (effective) IP address; The .spec.ips of the IP pool can be modified dynamically; The IP pool will inherit the reserved IP of the subnet. When randomly assigning an IP address from the IP pool, the reserved IP included in the IP pool will be skipped; When randomly assigning an IP address from a subnet, it will only be assigned from a range other than all IP pools in the subnet. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6ce8\u610f\u4e8b\u9879"},{"location":"en/guide/loadbalancer-service/","text":"LoadBalancer \u7c7b\u578b Service \u00b6 Kube-OVN \u5df2\u7ecf\u652f\u6301\u4e86 VPC \u548c VPC \u7f51\u5173\u7684\u5b9e\u73b0\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 VPC \u914d\u7f6e \u3002 \u7531\u4e8e VPC \u7f51\u5173\u7684\u4f7f\u7528\u6bd4\u8f83\u590d\u6742\uff0c\u57fa\u4e8e VPC \u7f51\u5173\u7684\u5b9e\u73b0\u505a\u4e86\u7b80\u5316\uff0c\u652f\u6301\u5728\u9ed8\u8ba4 VPC \u4e0b\u521b\u5efa LoadBalancer \u7c7b\u578b\u7684 Service\uff0c\u5b9e\u73b0\u901a\u8fc7 LoadBalancerIP \u6765\u8bbf\u95ee\u9ed8\u8ba4 VPC \u4e0b\u7684 Service\u3002 \u9996\u5148\u786e\u8ba4\u73af\u5883\u4e0a\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff1a \u5b89\u88c5\u4e86 multus-cni \u548c macvlan cni \u3002 LoadBalancer Service \u7684\u652f\u6301\uff0c\u662f\u5bf9 VPC \u7f51\u5173\u4ee3\u7801\u8fdb\u884c\u7b80\u5316\u5b9e\u73b0\u7684\uff0c\u4ecd\u7136\u4f7f\u7528 vpc-nat-gw \u7684\u955c\u50cf\uff0c\u4f9d\u8d56 macvlan \u63d0\u4f9b\u591a\u7f51\u5361\u529f\u80fd\u652f\u6301\u3002 \u76ee\u524d\u53ea\u652f\u6301\u5728 \u9ed8\u8ba4 VPC \u914d\u7f6e\uff0c\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 LoadBalancer \u652f\u6301\u53ef\u4ee5\u53c2\u8003 VPC \u7684\u6587\u6863 VPC \u914d\u7f6e \u3002 \u9ed8\u8ba4 VPC LoadBalancer Service \u914d\u7f6e\u6b65\u9aa4 \u00b6 \u5f00\u542f\u7279\u6027\u5f00\u5173 \u00b6 \u4fee\u6539 kube-system namespace \u4e0b\u7684 deployment kube-ovn-controller \uff0c\u5728 args \u4e2d\u589e\u52a0\u53c2\u6570 --enable-lb-svc=true \uff0c\u5f00\u542f\u529f\u80fd\u5f00\u5173\uff0c\u8be5\u53c2\u6570\u9ed8\u8ba4\u4e3a false\u3002 containers : - args : - /kube-ovn/start-controller.sh - --default-cidr=10.16.0.0/16 - --default-gateway=10.16.0.1 - --default-gateway-check=true - --enable-lb-svc=true // \u53c2\u6570\u8bbe\u7f6e\u4e3a true \u521b\u5efa NetworkAttachmentDefinition CRD \u8d44\u6e90 \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa net-attach-def \u8d44\u6e90: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : lb-svc-attachment namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", //\u7269\u7406\u7f51\u5361\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u914d\u7f6e \"mode\": \"bridge\" }' \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u901a\u8fc7\u7269\u7406\u7f51\u5361 eth0 \u6765\u5b9e\u73b0\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u4ed6\u7269\u7406\u7f51\u5361\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 master \u53d6\u503c\uff0c\u6307\u5b9a\u4f7f\u7528\u7684\u7269\u7406\u7f51\u5361\u540d\u79f0\u3002 \u521b\u5efa Subnet \u00b6 \u521b\u5efa\u7684 Subnet\uff0c\u7528\u4e8e\u7ed9 LoadBalancer Service \u5206\u914d LoadBalancerIP\uff0c\u8be5\u5730\u5740\u6b63\u5e38\u60c5\u51b5\u4e0b\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u53ef\u4ee5\u914d\u7f6e Underlay Subnet \u7528\u4e8e\u5730\u5740\u5206\u914d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa\u65b0\u5b50\u7f51\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attach-subnet spec : protocol : IPv4 provider : lb-svc-attachment.kube-system # provider \u683c\u5f0f\u56fa\u5b9a\uff0c\u7531\u4e0a\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210 cidrBlock : 172.18.0.0/16 gateway : 172.18.0.1 excludeIps : - 172.18.0.0..172.18.0.10 Subnet \u4e2d provider \u53c2\u6570\u4ee5 ovn \u6216\u8005\u4ee5 .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u8868\u793a\u8be5\u5b50\u7f51\u662f\u7531 Kube-OVN \u7ba1\u7406\u4f7f\u7528\uff0c\u9700\u8981\u5bf9\u5e94\u521b\u5efa logical switch \u8bb0\u5f55\u3002 provider \u975e ovn \u6216\u8005\u975e .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u5219 Kube-OVN \u53ea\u63d0\u4f9b IPAM \u529f\u80fd\uff0c\u8bb0\u5f55 IP \u5730\u5740\u5206\u914d\u60c5\u51b5\uff0c\u4e0d\u5bf9\u5b50\u7f51\u505a\u4e1a\u52a1\u903b\u8f91\u5904\u7406\u3002 \u521b\u5efa LoadBalancer Service \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa LoadBalancer Service\uff1a apiVersion : v1 kind : Service metadata : annotations : lb-svc-attachment.kube-system.kubernetes.io/logical_switch : attach-subnet #\u53ef\u9009 ovn.kubernetes.io/attachmentprovider : lb-svc-attachment.kube-system #\u5fc5\u987b labels : app : dynamic name : test-service namespace : default spec : loadBalancerIP : 172.18.0.18 #\u53ef\u9009 ports : - name : test protocol : TCP port : 80 targetPort : 80 selector : app : dynamic sessionAffinity : None type : LoadBalancer \u5728 yaml \u4e2d\uff0cannotation ovn.kubernetes.io/attachmentprovider \u4e3a\u5fc5\u586b\u9879\uff0c\u53d6\u503c\u7531\u7b2c\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210\u3002\u8be5 annotation \u7528\u4e8e\u5728\u521b\u5efa Pod \u65f6\uff0c\u67e5\u627e net-attach-def \u8d44\u6e90\u3002 \u53ef\u4ee5\u901a\u8fc7 annotation \u6307\u5b9a\u591a\u7f51\u5361\u5730\u5740\u5206\u914d\u4f7f\u7528\u7684\u5b50\u7f51\u3002annotation key \u683c\u5f0f\u4e3a net-attach-def \u8d44\u6e90\u7684 Name.Namespace.kubernetes.io/logical_switch \u3002\u8be5\u914d\u7f6e\u4e3a \u53ef\u9009 \u9009\u9879\uff0c\u5728\u6ca1\u6709\u6307\u5b9a LoadBalancerIP \u5730\u5740\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u4ece\u8be5\u5b50\u7f51\u52a8\u6001\u5206\u914d\u5730\u5740\uff0c\u586b\u5145\u5230 LoadBalancerIP \u5b57\u6bb5\u3002 \u5982\u679c\u9700\u8981\u9759\u6001\u914d\u7f6e LoadBalancerIP \u5730\u5740\uff0c\u53ef\u4ee5\u914d\u7f6e spec.loadBalancerIP \u5b57\u6bb5\uff0c\u8be5\u5730\u5740\u9700\u8981\u5728\u6307\u5b9a\u5b50\u7f51\u7684\u5730\u5740\u8303\u56f4\u5185\u3002 \u5728\u6267\u884c yaml \u521b\u5efa Service \u540e\uff0c\u5728 Service \u540c Namespace \u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u542f\u52a8\u4fe1\u606f\uff1a # kubectl get pod NAME READY STATUS RESTARTS AGE lb-svc-test-service-6869d98dd8-cjvll 1 /1 Running 0 107m # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE test-service LoadBalancer 10 .109.201.193 172 .18.0.18 80 :30056/TCP 107m \u6307\u5b9a service.spec.loadBalancerIP \u53c2\u6570\u65f6\uff0c\u6700\u7ec8\u5c06\u8be5\u53c2\u6570\u8d4b\u503c\u7ed9 service external-ip \u5b57\u6bb5\u3002\u4e0d\u6307\u5b9a\u7684\u60c5\u51b5\u4e0b\uff0c\u8be5\u53c2\u6570\u4e3a\u968f\u673a\u5206\u914d\u503c\u3002 \u67e5\u770b\u6d4b\u8bd5 Pod \u7684 yaml \u8f93\u51fa\uff0c\u5b58\u5728\u591a\u7f51\u5361\u5206\u914d\u7684\u5730\u5740\u4fe1\u606f\uff1a # kubectl get pod -o yaml lb-svc-test-service-6869d98dd8-cjvll apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/network-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] k8s.v1.cni.cncf.io/networks: default/test-service k8s.v1.cni.cncf.io/networks-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] ovn.kubernetes.io/allocated: \"true\" ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.2 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:45:F4:29 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/routed: \"true\" test-service.default.kubernetes.io/allocated: \"true\" test-service.default.kubernetes.io/cidr: 172 .18.0.0/16 test-service.default.kubernetes.io/gateway: 172 .18.0.1 test-service.default.kubernetes.io/ip_address: 172 .18.0.18 test-service.default.kubernetes.io/logical_switch: attach-subnet test-service.default.kubernetes.io/mac_address: 00 :00:00:AF:AA:BF test-service.default.kubernetes.io/pod_nic_type: veth-pair \u67e5\u770b Service \u7684\u4fe1\u606f\uff1a # kubectl get svc -o yaml test-service apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { \"apiVersion\" : \"v1\" , \"kind\" : \"Service\" , \"metadata\" : { \"annotations\" : { \"test-service.default.kubernetes.io/logical_switch\" : \"attach-subnet\" } , \"labels\" : { \"app\" : \"dynamic\" } , \"name\" : \"test-service\" , \"namespace\" : \"default\" } , \"spec\" : { \"ports\" : [{ \"name\" : \"test\" , \"port\" :80, \"protocol\" : \"TCP\" , \"targetPort\" :80 }] , \"selector\" : { \"app\" : \"dynamic\" } , \"sessionAffinity\" : \"None\" , \"type\" : \"LoadBalancer\" }} ovn.kubernetes.io/vpc: ovn-cluster test-service.default.kubernetes.io/logical_switch: attach-subnet creationTimestamp: \"2022-06-15T09:01:58Z\" labels: app: dynamic name: test-service namespace: default resourceVersion: \"38485\" uid: 161edee1-7f6e-40f5-9e09-5a52c44267d0 spec: allocateLoadBalancerNodePorts: true clusterIP: 10 .109.201.193 clusterIPs: - 10 .109.201.193 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: test nodePort: 30056 port: 80 protocol: TCP targetPort: 80 selector: app: dynamic sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - ip: 172 .18.0.18 \u6d4b\u8bd5 LoadBalancerIP \u8bbf\u95ee \u00b6 \u53c2\u8003\u4ee5\u4e0b yaml, \u521b\u5efa\u6d4b\u8bd5 Pod\uff0c\u4f5c\u4e3a Service \u7684 Endpoints \u63d0\u4f9b\u670d\u52a1: apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx dnsPolicy : ClusterFirst restartPolicy : Always \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u63d0\u4f9b\u7684\u5b50\u7f51\u5730\u5740\uff0c\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u4e3a\u4e86\u7b80\u5355\u9a8c\u8bc1\uff0c\u5728\u96c6\u7fa4\u5185\u8bbf\u95ee Service \u7684 LoadBalancerIP:Port \uff0c\u67e5\u770b\u662f\u5426\u6b63\u5e38\u8bbf\u95ee\u6210\u529f\u3002 # curl 172.18.0.11:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on dynamic-7d8d7874f5-hsgc4</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u8fdb\u5165 Service \u521b\u5efa\u7684 Pod\uff0c\u67e5\u770b\u7f51\u7edc\u7684\u4fe1\u606f # ip a 4 : net1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ba:85:f7:02:9f:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172 .18.0.18/16 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::b885:f7ff:fe02:9f42/64 scope link valid_lft forever preferred_lft forever 36 : eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 00 :00:00:45:f4:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10 .16.0.2/16 brd 10 .16.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe45:f429/64 scope link valid_lft forever preferred_lft forever # ip rule 0 : from all lookup local 32764 : from all iif eth0 lookup 100 32765 : from all iif net1 lookup 100 32766 : from all lookup main 32767 : from all lookup default # ip route show table 100 default via 172 .18.0.1 dev net1 10 .109.201.193 via 10 .16.0.1 dev eth0 172 .18.0.0/16 dev net1 scope link # iptables -t nat -L -n -v Chain PREROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- * * 0 .0.0.0/0 172 .18.0.18 tcp dpt:80 to:10.109.201.193:80 Chain INPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain OUTPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain POSTROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0 .0.0.0/0 10 .109.201.193 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"LoadBalancer Type Service"},{"location":"en/guide/loadbalancer-service/#loadbalancer-service","text":"Kube-OVN \u5df2\u7ecf\u652f\u6301\u4e86 VPC \u548c VPC \u7f51\u5173\u7684\u5b9e\u73b0\uff0c\u5177\u4f53\u914d\u7f6e\u53ef\u4ee5\u53c2\u8003 VPC \u914d\u7f6e \u3002 \u7531\u4e8e VPC \u7f51\u5173\u7684\u4f7f\u7528\u6bd4\u8f83\u590d\u6742\uff0c\u57fa\u4e8e VPC \u7f51\u5173\u7684\u5b9e\u73b0\u505a\u4e86\u7b80\u5316\uff0c\u652f\u6301\u5728\u9ed8\u8ba4 VPC \u4e0b\u521b\u5efa LoadBalancer \u7c7b\u578b\u7684 Service\uff0c\u5b9e\u73b0\u901a\u8fc7 LoadBalancerIP \u6765\u8bbf\u95ee\u9ed8\u8ba4 VPC \u4e0b\u7684 Service\u3002 \u9996\u5148\u786e\u8ba4\u73af\u5883\u4e0a\u6ee1\u8db3\u4ee5\u4e0b\u6761\u4ef6\uff1a \u5b89\u88c5\u4e86 multus-cni \u548c macvlan cni \u3002 LoadBalancer Service \u7684\u652f\u6301\uff0c\u662f\u5bf9 VPC \u7f51\u5173\u4ee3\u7801\u8fdb\u884c\u7b80\u5316\u5b9e\u73b0\u7684\uff0c\u4ecd\u7136\u4f7f\u7528 vpc-nat-gw \u7684\u955c\u50cf\uff0c\u4f9d\u8d56 macvlan \u63d0\u4f9b\u591a\u7f51\u5361\u529f\u80fd\u652f\u6301\u3002 \u76ee\u524d\u53ea\u652f\u6301\u5728 \u9ed8\u8ba4 VPC \u914d\u7f6e\uff0c\u81ea\u5b9a\u4e49 VPC \u4e0b\u7684 LoadBalancer \u652f\u6301\u53ef\u4ee5\u53c2\u8003 VPC \u7684\u6587\u6863 VPC \u914d\u7f6e \u3002","title":"LoadBalancer \u7c7b\u578b Service"},{"location":"en/guide/loadbalancer-service/#vpc-loadbalancer-service","text":"","title":"\u9ed8\u8ba4 VPC LoadBalancer Service \u914d\u7f6e\u6b65\u9aa4"},{"location":"en/guide/loadbalancer-service/#_1","text":"\u4fee\u6539 kube-system namespace \u4e0b\u7684 deployment kube-ovn-controller \uff0c\u5728 args \u4e2d\u589e\u52a0\u53c2\u6570 --enable-lb-svc=true \uff0c\u5f00\u542f\u529f\u80fd\u5f00\u5173\uff0c\u8be5\u53c2\u6570\u9ed8\u8ba4\u4e3a false\u3002 containers : - args : - /kube-ovn/start-controller.sh - --default-cidr=10.16.0.0/16 - --default-gateway=10.16.0.1 - --default-gateway-check=true - --enable-lb-svc=true // \u53c2\u6570\u8bbe\u7f6e\u4e3a true","title":"\u5f00\u542f\u7279\u6027\u5f00\u5173"},{"location":"en/guide/loadbalancer-service/#networkattachmentdefinition-crd","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa net-attach-def \u8d44\u6e90: apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : lb-svc-attachment namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth0\", //\u7269\u7406\u7f51\u5361\uff0c\u6839\u636e\u5b9e\u9645\u60c5\u51b5\u914d\u7f6e \"mode\": \"bridge\" }' \u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u901a\u8fc7\u7269\u7406\u7f51\u5361 eth0 \u6765\u5b9e\u73b0\u591a\u7f51\u5361\u529f\u80fd\uff0c\u5982\u679c\u9700\u8981\u4f7f\u7528\u5176\u4ed6\u7269\u7406\u7f51\u5361\uff0c\u53ef\u4ee5\u901a\u8fc7\u4fee\u6539 master \u53d6\u503c\uff0c\u6307\u5b9a\u4f7f\u7528\u7684\u7269\u7406\u7f51\u5361\u540d\u79f0\u3002","title":"\u521b\u5efa NetworkAttachmentDefinition CRD \u8d44\u6e90"},{"location":"en/guide/loadbalancer-service/#subnet","text":"\u521b\u5efa\u7684 Subnet\uff0c\u7528\u4e8e\u7ed9 LoadBalancer Service \u5206\u914d LoadBalancerIP\uff0c\u8be5\u5730\u5740\u6b63\u5e38\u60c5\u51b5\u4e0b\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u53ef\u4ee5\u914d\u7f6e Underlay Subnet \u7528\u4e8e\u5730\u5740\u5206\u914d\u3002 \u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa\u65b0\u5b50\u7f51\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : attach-subnet spec : protocol : IPv4 provider : lb-svc-attachment.kube-system # provider \u683c\u5f0f\u56fa\u5b9a\uff0c\u7531\u4e0a\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210 cidrBlock : 172.18.0.0/16 gateway : 172.18.0.1 excludeIps : - 172.18.0.0..172.18.0.10 Subnet \u4e2d provider \u53c2\u6570\u4ee5 ovn \u6216\u8005\u4ee5 .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u8868\u793a\u8be5\u5b50\u7f51\u662f\u7531 Kube-OVN \u7ba1\u7406\u4f7f\u7528\uff0c\u9700\u8981\u5bf9\u5e94\u521b\u5efa logical switch \u8bb0\u5f55\u3002 provider \u975e ovn \u6216\u8005\u975e .ovn \u4e3a\u540e\u7f00\u7ed3\u675f\uff0c\u5219 Kube-OVN \u53ea\u63d0\u4f9b IPAM \u529f\u80fd\uff0c\u8bb0\u5f55 IP \u5730\u5740\u5206\u914d\u60c5\u51b5\uff0c\u4e0d\u5bf9\u5b50\u7f51\u505a\u4e1a\u52a1\u903b\u8f91\u5904\u7406\u3002","title":"\u521b\u5efa Subnet"},{"location":"en/guide/loadbalancer-service/#loadbalancer-service_1","text":"\u53c2\u8003\u4ee5\u4e0b yaml\uff0c\u521b\u5efa LoadBalancer Service\uff1a apiVersion : v1 kind : Service metadata : annotations : lb-svc-attachment.kube-system.kubernetes.io/logical_switch : attach-subnet #\u53ef\u9009 ovn.kubernetes.io/attachmentprovider : lb-svc-attachment.kube-system #\u5fc5\u987b labels : app : dynamic name : test-service namespace : default spec : loadBalancerIP : 172.18.0.18 #\u53ef\u9009 ports : - name : test protocol : TCP port : 80 targetPort : 80 selector : app : dynamic sessionAffinity : None type : LoadBalancer \u5728 yaml \u4e2d\uff0cannotation ovn.kubernetes.io/attachmentprovider \u4e3a\u5fc5\u586b\u9879\uff0c\u53d6\u503c\u7531\u7b2c\u4e00\u6b65\u521b\u5efa\u7684 net-attach-def \u8d44\u6e90\u7684 Name.Namespace \u7ec4\u6210\u3002\u8be5 annotation \u7528\u4e8e\u5728\u521b\u5efa Pod \u65f6\uff0c\u67e5\u627e net-attach-def \u8d44\u6e90\u3002 \u53ef\u4ee5\u901a\u8fc7 annotation \u6307\u5b9a\u591a\u7f51\u5361\u5730\u5740\u5206\u914d\u4f7f\u7528\u7684\u5b50\u7f51\u3002annotation key \u683c\u5f0f\u4e3a net-attach-def \u8d44\u6e90\u7684 Name.Namespace.kubernetes.io/logical_switch \u3002\u8be5\u914d\u7f6e\u4e3a \u53ef\u9009 \u9009\u9879\uff0c\u5728\u6ca1\u6709\u6307\u5b9a LoadBalancerIP \u5730\u5740\u7684\u60c5\u51b5\u4e0b\uff0c\u5c06\u4ece\u8be5\u5b50\u7f51\u52a8\u6001\u5206\u914d\u5730\u5740\uff0c\u586b\u5145\u5230 LoadBalancerIP \u5b57\u6bb5\u3002 \u5982\u679c\u9700\u8981\u9759\u6001\u914d\u7f6e LoadBalancerIP \u5730\u5740\uff0c\u53ef\u4ee5\u914d\u7f6e spec.loadBalancerIP \u5b57\u6bb5\uff0c\u8be5\u5730\u5740\u9700\u8981\u5728\u6307\u5b9a\u5b50\u7f51\u7684\u5730\u5740\u8303\u56f4\u5185\u3002 \u5728\u6267\u884c yaml \u521b\u5efa Service \u540e\uff0c\u5728 Service \u540c Namespace \u4e0b\uff0c\u53ef\u4ee5\u770b\u5230 Pod \u542f\u52a8\u4fe1\u606f\uff1a # kubectl get pod NAME READY STATUS RESTARTS AGE lb-svc-test-service-6869d98dd8-cjvll 1 /1 Running 0 107m # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE test-service LoadBalancer 10 .109.201.193 172 .18.0.18 80 :30056/TCP 107m \u6307\u5b9a service.spec.loadBalancerIP \u53c2\u6570\u65f6\uff0c\u6700\u7ec8\u5c06\u8be5\u53c2\u6570\u8d4b\u503c\u7ed9 service external-ip \u5b57\u6bb5\u3002\u4e0d\u6307\u5b9a\u7684\u60c5\u51b5\u4e0b\uff0c\u8be5\u53c2\u6570\u4e3a\u968f\u673a\u5206\u914d\u503c\u3002 \u67e5\u770b\u6d4b\u8bd5 Pod \u7684 yaml \u8f93\u51fa\uff0c\u5b58\u5728\u591a\u7f51\u5361\u5206\u914d\u7684\u5730\u5740\u4fe1\u606f\uff1a # kubectl get pod -o yaml lb-svc-test-service-6869d98dd8-cjvll apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/network-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] k8s.v1.cni.cncf.io/networks: default/test-service k8s.v1.cni.cncf.io/networks-status: | - [{ \"name\" : \"kube-ovn\" , \"ips\" : [ \"10.16.0.2\" ] , \"default\" : true, \"dns\" : {} } , { \"name\" : \"default/test-service\" , \"interface\" : \"net1\" , \"mac\" : \"ba:85:f7:02:9f:42\" , \"dns\" : {} }] ovn.kubernetes.io/allocated: \"true\" ovn.kubernetes.io/cidr: 10 .16.0.0/16 ovn.kubernetes.io/gateway: 10 .16.0.1 ovn.kubernetes.io/ip_address: 10 .16.0.2 ovn.kubernetes.io/logical_router: ovn-cluster ovn.kubernetes.io/logical_switch: ovn-default ovn.kubernetes.io/mac_address: 00 :00:00:45:F4:29 ovn.kubernetes.io/pod_nic_type: veth-pair ovn.kubernetes.io/routed: \"true\" test-service.default.kubernetes.io/allocated: \"true\" test-service.default.kubernetes.io/cidr: 172 .18.0.0/16 test-service.default.kubernetes.io/gateway: 172 .18.0.1 test-service.default.kubernetes.io/ip_address: 172 .18.0.18 test-service.default.kubernetes.io/logical_switch: attach-subnet test-service.default.kubernetes.io/mac_address: 00 :00:00:AF:AA:BF test-service.default.kubernetes.io/pod_nic_type: veth-pair \u67e5\u770b Service \u7684\u4fe1\u606f\uff1a # kubectl get svc -o yaml test-service apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | { \"apiVersion\" : \"v1\" , \"kind\" : \"Service\" , \"metadata\" : { \"annotations\" : { \"test-service.default.kubernetes.io/logical_switch\" : \"attach-subnet\" } , \"labels\" : { \"app\" : \"dynamic\" } , \"name\" : \"test-service\" , \"namespace\" : \"default\" } , \"spec\" : { \"ports\" : [{ \"name\" : \"test\" , \"port\" :80, \"protocol\" : \"TCP\" , \"targetPort\" :80 }] , \"selector\" : { \"app\" : \"dynamic\" } , \"sessionAffinity\" : \"None\" , \"type\" : \"LoadBalancer\" }} ovn.kubernetes.io/vpc: ovn-cluster test-service.default.kubernetes.io/logical_switch: attach-subnet creationTimestamp: \"2022-06-15T09:01:58Z\" labels: app: dynamic name: test-service namespace: default resourceVersion: \"38485\" uid: 161edee1-7f6e-40f5-9e09-5a52c44267d0 spec: allocateLoadBalancerNodePorts: true clusterIP: 10 .109.201.193 clusterIPs: - 10 .109.201.193 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: test nodePort: 30056 port: 80 protocol: TCP targetPort: 80 selector: app: dynamic sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - ip: 172 .18.0.18","title":"\u521b\u5efa LoadBalancer Service"},{"location":"en/guide/loadbalancer-service/#loadbalancerip","text":"\u53c2\u8003\u4ee5\u4e0b yaml, \u521b\u5efa\u6d4b\u8bd5 Pod\uff0c\u4f5c\u4e3a Service \u7684 Endpoints \u63d0\u4f9b\u670d\u52a1: apiVersion : apps/v1 kind : Deployment metadata : labels : app : dynamic name : dynamic namespace : default spec : replicas : 2 selector : matchLabels : app : dynamic strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : creationTimestamp : null labels : app : dynamic spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : nginx dnsPolicy : ClusterFirst restartPolicy : Always \u6b63\u5e38\u60c5\u51b5\u4e0b\uff0c\u63d0\u4f9b\u7684\u5b50\u7f51\u5730\u5740\uff0c\u5728\u96c6\u7fa4\u5916\u5e94\u8be5\u53ef\u4ee5\u8bbf\u95ee\u5230\u3002\u4e3a\u4e86\u7b80\u5355\u9a8c\u8bc1\uff0c\u5728\u96c6\u7fa4\u5185\u8bbf\u95ee Service \u7684 LoadBalancerIP:Port \uff0c\u67e5\u770b\u662f\u5426\u6b63\u5e38\u8bbf\u95ee\u6210\u529f\u3002 # curl 172.18.0.11:80 <html> <head> <title>Hello World!</title> <link href = '//fonts.googleapis.com/css?family=Open+Sans:400,700' rel = 'stylesheet' type = 'text/css' > <style> body { background-color: white ; text-align: center ; padding: 50px ; font-family: \"Open Sans\" , \"Helvetica Neue\" ,Helvetica,Arial,sans-serif ; } #logo { margin-bottom: 40px ; } </style> </head> <body> <h1>Hello World!</h1> <h3>Links found</h3> <h3>I am on dynamic-7d8d7874f5-hsgc4</h3> <h3>Cookie = </h3> <b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br /> <h3>my name is hanhouchao!</h3> <h3> RequestURI = '/' </h3> </body> </html> \u8fdb\u5165 Service \u521b\u5efa\u7684 Pod\uff0c\u67e5\u770b\u7f51\u7edc\u7684\u4fe1\u606f # ip a 4 : net1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ba:85:f7:02:9f:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172 .18.0.18/16 scope global net1 valid_lft forever preferred_lft forever inet6 fe80::b885:f7ff:fe02:9f42/64 scope link valid_lft forever preferred_lft forever 36 : eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 00 :00:00:45:f4:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10 .16.0.2/16 brd 10 .16.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::200:ff:fe45:f429/64 scope link valid_lft forever preferred_lft forever # ip rule 0 : from all lookup local 32764 : from all iif eth0 lookup 100 32765 : from all iif net1 lookup 100 32766 : from all lookup main 32767 : from all lookup default # ip route show table 100 default via 172 .18.0.1 dev net1 10 .109.201.193 via 10 .16.0.1 dev eth0 172 .18.0.0/16 dev net1 scope link # iptables -t nat -L -n -v Chain PREROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 DNAT tcp -- * * 0 .0.0.0/0 172 .18.0.18 tcp dpt:80 to:10.109.201.193:80 Chain INPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain OUTPUT ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination Chain POSTROUTING ( policy ACCEPT 0 packets, 0 bytes ) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0 .0.0.0/0 10 .109.201.193 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u6d4b\u8bd5 LoadBalancerIP \u8bbf\u95ee"},{"location":"en/guide/mirror/","text":"Traffic Mirror \u00b6 The traffic mirroring feature allows packets to and from the container network to be copied to a specific NIC of the host. Administrators or developers can listen to this NIC to get the complete container network traffic for further analysis, monitoring, security auditing and other operations. It can also be integrated with traditional NPM for more fine-grained traffic visibility. The traffic mirroring feature introduces some performance loss, with an additional CPU consumption of 5% to 10% depending on CPU performance and traffic characteristics. Global Traffic Mirroring Settings \u00b6 The traffic mirroring is disabled by default, please modify the args of kube-ovn-cni DaemonSet to enable it: --enable-mirror=true : Whether to enable traffic mirroring. --mirror-iface=mirror0 : The name of the NIC that the traffic mirror is copied to. This NIC can be a physical NIC that already exists on the host machine. At this point the NIC will be bridged into the br-int bridge and the mirrored traffic will go directly to the underlying switch. If the NIC name does not exist, Kube-OVN will automatically create a virtual NIC with the same name, through which the administrator or developer can access all traffic on the current node on the host. The default is mirror0 . Next, you can listen to the traffic on mirror0 with tcpdump or other traffic analysis tools. tcpdump -ni mirror0 Pod Level Mirroring Settings \u00b6 If you only need to mirror some Pod traffic, you need to disable the global traffic mirroring and then add the ovn.kubernetes.io/mirror annotation on a specific Pod to enable Pod-level traffic mirroring. apiVersion : v1 kind : Pod metadata : name : mirror-pod namespace : ls1 annotations : ovn.kubernetes.io/mirror : \"true\" spec : containers : - name : mirror-pod image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Traffic Mirror"},{"location":"en/guide/mirror/#traffic-mirror","text":"The traffic mirroring feature allows packets to and from the container network to be copied to a specific NIC of the host. Administrators or developers can listen to this NIC to get the complete container network traffic for further analysis, monitoring, security auditing and other operations. It can also be integrated with traditional NPM for more fine-grained traffic visibility. The traffic mirroring feature introduces some performance loss, with an additional CPU consumption of 5% to 10% depending on CPU performance and traffic characteristics.","title":"Traffic Mirror"},{"location":"en/guide/mirror/#global-traffic-mirroring-settings","text":"The traffic mirroring is disabled by default, please modify the args of kube-ovn-cni DaemonSet to enable it: --enable-mirror=true : Whether to enable traffic mirroring. --mirror-iface=mirror0 : The name of the NIC that the traffic mirror is copied to. This NIC can be a physical NIC that already exists on the host machine. At this point the NIC will be bridged into the br-int bridge and the mirrored traffic will go directly to the underlying switch. If the NIC name does not exist, Kube-OVN will automatically create a virtual NIC with the same name, through which the administrator or developer can access all traffic on the current node on the host. The default is mirror0 . Next, you can listen to the traffic on mirror0 with tcpdump or other traffic analysis tools. tcpdump -ni mirror0","title":"Global Traffic Mirroring Settings"},{"location":"en/guide/mirror/#pod-level-mirroring-settings","text":"If you only need to mirror some Pod traffic, you need to disable the global traffic mirroring and then add the ovn.kubernetes.io/mirror annotation on a specific Pod to enable Pod-level traffic mirroring. apiVersion : v1 kind : Pod metadata : name : mirror-pod namespace : ls1 annotations : ovn.kubernetes.io/mirror : \"true\" spec : containers : - name : mirror-pod image : docker.io/library/nginx:alpine \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Pod Level Mirroring Settings"},{"location":"en/guide/networkpolicy-log/","text":"NetworkPolicy Logging \u00b6 NetworkPolicy is a interface provided by Kubernetes and implemented by Kube-OVN through OVN's ACLs. With NetworkPolicy, if the networks are down, it is difficult to determine whether it is caused by a network failure or a NetworkPolicy rule problem. Kube-OVN provides NetworkPolicy logging to help administrators quickly locate whether a NetworkPolicy drop rule has been hit, and to record the illegal accesses. Once NetworkPolicy logging is turned on, logs need to be printed for every packet that hits a Drop rule, which introduces additional performance overhead. Under a malicious attack, a large number of logs in a short period of time may exhaust the CPU. We recommend turning off logging by default in production environments and dynamically turning it on when you need to troubleshoot problems. Enable NetworkPolicy Logging \u00b6 Add the annotation ovn.kubernetes.io/enable_log to the NetworkPolicy where logging needs to be enabled, as follows: apiVersion : networking.k8s.io/v1 kind : NetworkPolicy metadata : name : default-deny-ingress namespace : kube-system annotations : ovn.kubernetes.io/enable_log : \"true\" spec : podSelector : {} policyTypes : - Ingress Next, you can observe the log of dropped packets in /var/log/ovn/ovn-controller.log on the host of the corresponding Pod: # tail -f /var/log/ovn/ovn-controller.log 2022 -07-20T05:55:03.229Z | 00394 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54343 ,tp_dst = 53 2022 -07-20T05:55:06.229Z | 00395 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 44187 ,tp_dst = 53 2022 -07-20T05:55:08.230Z | 00396 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54274 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00397 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 32778 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00398 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 34188 ,tp_dst = 53 2022 -07-20T05:55:13.231Z | 00399 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 43290 ,tp_dst = 53 2022 -07-20T05:55:22.096Z | 00400 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.097Z | 00401 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.098Z | 00402 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 Disable NetworkPolicy Logging \u00b6 Set annotation ovn.kubernetes.io/enable_log in the corresponding NetworkPolicy to false to disable NetworkPolicy logging: kubectl annotate networkpolicy -n kube-system default-deny-ingress ovn.kubernetes.io/enable_log = false --overwrite \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"NetworkPolicy Logging"},{"location":"en/guide/networkpolicy-log/#networkpolicy-logging","text":"NetworkPolicy is a interface provided by Kubernetes and implemented by Kube-OVN through OVN's ACLs. With NetworkPolicy, if the networks are down, it is difficult to determine whether it is caused by a network failure or a NetworkPolicy rule problem. Kube-OVN provides NetworkPolicy logging to help administrators quickly locate whether a NetworkPolicy drop rule has been hit, and to record the illegal accesses. Once NetworkPolicy logging is turned on, logs need to be printed for every packet that hits a Drop rule, which introduces additional performance overhead. Under a malicious attack, a large number of logs in a short period of time may exhaust the CPU. We recommend turning off logging by default in production environments and dynamically turning it on when you need to troubleshoot problems.","title":"NetworkPolicy Logging"},{"location":"en/guide/networkpolicy-log/#enable-networkpolicy-logging","text":"Add the annotation ovn.kubernetes.io/enable_log to the NetworkPolicy where logging needs to be enabled, as follows: apiVersion : networking.k8s.io/v1 kind : NetworkPolicy metadata : name : default-deny-ingress namespace : kube-system annotations : ovn.kubernetes.io/enable_log : \"true\" spec : podSelector : {} policyTypes : - Ingress Next, you can observe the log of dropped packets in /var/log/ovn/ovn-controller.log on the host of the corresponding Pod: # tail -f /var/log/ovn/ovn-controller.log 2022 -07-20T05:55:03.229Z | 00394 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54343 ,tp_dst = 53 2022 -07-20T05:55:06.229Z | 00395 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 44187 ,tp_dst = 53 2022 -07-20T05:55:08.230Z | 00396 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 54274 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00397 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 32778 ,tp_dst = 53 2022 -07-20T05:55:11.231Z | 00398 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 34188 ,tp_dst = 53 2022 -07-20T05:55:13.231Z | 00399 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: udp,vlan_tci = 0x0000,dl_src = 00 :00:00:21:b7:d1,dl_dst = 00 :00:00:8d:0b:86,nw_src = 10 .16.0.10,nw_dst = 10 .16.0.7,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 63 ,tp_src = 43290 ,tp_dst = 53 2022 -07-20T05:55:22.096Z | 00400 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.097Z | 00401 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 2022 -07-20T05:55:22.098Z | 00402 | acl_log ( ovn_pinctrl0 ) | INFO | name = \"<unnamed>\" , verdict = drop, severity = warning, direction = to-lport: icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:6c:42:91,dl_dst = 00 :00:00:a5:d7:63,nw_src = 10 .16.0.9,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0","title":"Enable NetworkPolicy Logging"},{"location":"en/guide/networkpolicy-log/#disable-networkpolicy-logging","text":"Set annotation ovn.kubernetes.io/enable_log in the corresponding NetworkPolicy to false to disable NetworkPolicy logging: kubectl annotate networkpolicy -n kube-system default-deny-ingress ovn.kubernetes.io/enable_log = false --overwrite \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Disable NetworkPolicy Logging"},{"location":"en/guide/prometheus-grafana/","text":"Monitor and Dashboard \u00b6 Kube-OVN can export network control plane information and network data plane quality information metrics to the external in formats supported by Prometheus. We use the CRD provided by kube-prometheus to define the corresponding Prometheus monitoring rules. For all monitoring metrics supported by Kube-OVN, please refer to Kube-OVN Monitoring Metrics . If you are using native Prometheus, please refer to Configuring Native Prometheus for configuration. Install Prometheus Monitor \u00b6 Kube-OVN uses Prometheus Monitor CRD to manage the monitoring output. # network quality related monitoring metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-monitor.yaml # kube-ovn-controller metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-monitor.yaml # kube-ovn-cni metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-monitor.yaml # ovn metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-monitor.yaml The default interval for Prometheus pull is 15s, if you need to adjust it, modify the interval value in yaml. Import Grafana Dashboard \u00b6 Kube-OVN provides a predefined Grafana Dashboard to display control plane and data plane related metrics. Download the corresponding Dashboard template: # network quality related monitoring dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-grafana.json # kube-ovn-controller dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-grafana.json # kube-ovn-cni dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-grafana.json # ovn dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-grafana.json # ovs dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovs-grafana.json Import these templates into Grafana and set the data source to the corresponding Prometheus to see the following Dashboards. kube-ovn-controller dashboard: kube-ovn-pinger dashboard: kube-ovn-cni dashboard: \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Monitor and Dashboard"},{"location":"en/guide/prometheus-grafana/#monitor-and-dashboard","text":"Kube-OVN can export network control plane information and network data plane quality information metrics to the external in formats supported by Prometheus. We use the CRD provided by kube-prometheus to define the corresponding Prometheus monitoring rules. For all monitoring metrics supported by Kube-OVN, please refer to Kube-OVN Monitoring Metrics . If you are using native Prometheus, please refer to Configuring Native Prometheus for configuration.","title":"Monitor and Dashboard"},{"location":"en/guide/prometheus-grafana/#install-prometheus-monitor","text":"Kube-OVN uses Prometheus Monitor CRD to manage the monitoring output. # network quality related monitoring metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-monitor.yaml # kube-ovn-controller metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-monitor.yaml # kube-ovn-cni metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-monitor.yaml # ovn metrics kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-monitor.yaml The default interval for Prometheus pull is 15s, if you need to adjust it, modify the interval value in yaml.","title":"Install Prometheus Monitor"},{"location":"en/guide/prometheus-grafana/#import-grafana-dashboard","text":"Kube-OVN provides a predefined Grafana Dashboard to display control plane and data plane related metrics. Download the corresponding Dashboard template: # network quality related monitoring dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/pinger-grafana.json # kube-ovn-controller dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/controller-grafana.json # kube-ovn-cni dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/cni-grafana.json # ovn dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovn-grafana.json # ovs dashboard wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/monitoring/ovs-grafana.json Import these templates into Grafana and set the data source to the corresponding Prometheus to see the following Dashboards. kube-ovn-controller dashboard: kube-ovn-pinger dashboard: kube-ovn-cni dashboard: \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Import Grafana Dashboard"},{"location":"en/guide/prometheus/","text":"Config Native Prometheus \u00b6 Kube-OVN provides rich monitoring data for OVN/OVS health status checks and connectivity checks of container and host networks, and Kube-OVN is configured with ServiceMonitor for Prometheus to dynamically obtain monitoring metrics. In some cases, where only Prometheus Server is installed and no other components are installed, you can dynamically obtain monitoring data for the cluster environment by modifying the configuration of Prometheus. Config Prometheus \u00b6 The following configuration documentation, referenced from Prometheus Service Discovery . Permission Configuration \u00b6 Prometheus is deployed in the cluster and needs to access the k8s apiserver to query the monitoring data of the containers. Refer to the following yaml to configure the permissions required by Prometheus: apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : prometheus rules : - apiGroups : [ \"\" ] resources : - nodes - nodes/proxy - services - endpoints - pods verbs : [ \"get\" , \"list\" , \"watch\" ] - apiGroups : - extensions resources : - ingresses verbs : [ \"get\" , \"list\" , \"watch\" ] - nonResourceURLs : [ \"/metrics\" ] verbs : [ \"get\" ] --- apiVersion : v1 kind : ServiceAccount metadata : name : prometheus namespace : default --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : prometheus roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : prometheus subjects : - kind : ServiceAccount name : prometheus namespace : default Prometheus ConfigMap \u00b6 The startup of Prometheus relies on the configuration file prometheus.yml, the contents of which can be configured in ConfigMap and dynamically mounted to the Pod. Create the ConfigMap file used by Prometheus by referring to the following yaml: apiVersion : v1 kind : ConfigMap metadata : name : prometheus-config data : prometheus.yml : |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'kubernetes-nodes' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node - job_name: 'kubernetes-service' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: service - job_name: 'kubernetes-endpoints' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints - job_name: 'kubernetes-ingress' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: ingress - job_name: 'kubernetes-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: pod Prometheus provides role-based querying of Kubernetes resource monitoring operations, which can be configured in the official documentation kubernetes_sd_config \u3002 Deploy Prometheus \u00b6 Deploy Prometheus Server by referring to the following yaml: apiVersion : apps/v1 kind : Deployment metadata : labels : app : prometheus name : prometheus namespace : default spec : replicas : 1 selector : matchLabels : app : prometheus strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : prometheus spec : serviceAccountName : prometheus serviceAccount : prometheus containers : - image : docker.io/prom/prometheus:latest imagePullPolicy : IfNotPresent name : prometheus command : - \"/bin/prometheus\" args : - \"--config.file=/etc/prometheus/prometheus.yml\" ports : - containerPort : 9090 protocol : TCP volumeMounts : - mountPath : \"/etc/prometheus\" name : prometheus-config volumes : - name : prometheus-config configMap : name : prometheus-config Deploy Prometheus Service by referring to the following yaml: kind : Service apiVersion : v1 metadata : name : prometheus namespace : default labels : name : prometheus spec : ports : - name : test protocol : TCP port : 9090 targetPort : 9090 type : NodePort selector : app : prometheus sessionAffinity : None After exposing Prometheus through NodePort, Prometheus can be accessed through the node address. Prometheus Metrics Config \u00b6 View information about Prometheus on the environment: # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kubernetes ClusterIP 10 .4.0.1 <none> 443 /TCP 8d prometheus NodePort 10 .4.102.222 <none> 9090 :32611/TCP 8d # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-7544b6b84d-v9m8s 1 /1 Running 0 3d5h 10 .3.0.7 192 .168.137.219 <none> <none> # kubectl get endpoints -o wide NAME ENDPOINTS AGE kubernetes 192 .168.136.228:6443,192.168.136.232:6443,192.168.137.219:6443 8d prometheus 10 .3.0.7:9090 8d Access Prometheus via NodePort to see the data dynamically queried by Status/Service Discovery: You can see that you can currently query all the service data information on the cluster. Configure to Query Specified Resource \u00b6 The ConfigMap configuration above queries all resource data. If you only need resource data for a certain role, you can add filter conditions. Take Service as an example, modify the ConfigMap content to query only the service monitoring data: - job_name : 'kubernetes-service' tls_config : ca_file : /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file : /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs : - role : service relabel_configs : - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_scrape ] action : \"keep\" regex : \"true\" - action : labelmap regex : __meta_kubernetes_service_label_(.+) - source_labels : [ __meta_kubernetes_namespace ] target_label : kubernetes_namespace - source_labels : [ __meta_kubernetes_service_name ] target_label : kubernetes_service_name - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_path ] action : replace target_label : __metrics_path__ regex : \"(.+)\" Check the Kube-OVN Service in kube-system Namespace: # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kube-dns ClusterIP 10 .4.0.10 <none> 53 /UDP,53/TCP,9153/TCP 13d kube-ovn-cni ClusterIP 10 .4.228.60 <none> 10665 /TCP 13d kube-ovn-controller ClusterIP 10 .4.172.213 <none> 10660 /TCP 13d kube-ovn-monitor ClusterIP 10 .4.242.9 <none> 10661 /TCP 13d kube-ovn-pinger ClusterIP 10 .4.122.52 <none> 8080 /TCP 13d ovn-nb ClusterIP 10 .4.80.213 <none> 6641 /TCP 13d ovn-northd ClusterIP 10 .4.126.234 <none> 6643 /TCP 13d ovn-sb ClusterIP 10 .4.216.249 <none> 6642 /TCP 13d Add annotation prometheus.io/scrape=\"true\" to Service\uff1a # kubectl annotate svc -n kube-system kube-ovn-cni prometheus.io/scrape=true service/kube-ovn-cni annotated # kubectl annotate svc -n kube-system kube-ovn-controller prometheus.io/scrape=true service/kube-ovn-controller annotated # kubectl annotate svc -n kube-system kube-ovn-monitor prometheus.io/scrape=true service/kube-ovn-monitor annotated # kubectl annotate svc -n kube-system kube-ovn-pinger prometheus.io/scrape=true service/kube-ovn-pinger annotated Check the configured Service information: # kubectl get svc -o yaml -n kube-system kube-ovn-controller apiVersion: v1 kind: Service metadata: annotations: helm.sh/chart-version: v3.10.0-alpha.55 helm.sh/original-name: kube-ovn-controller ovn.kubernetes.io/vpc: ovn-cluster prometheus.io/scrape: \"true\" // added annotation labels: app: kube-ovn-controller name: kube-ovn-controller namespace: kube-system spec: clusterIP: 10 .4.172.213 clusterIPs: - 10 .4.172.213 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: metrics port: 10660 protocol: TCP targetPort: 10660 selector: app: kube-ovn-controller sessionAffinity: None type: ClusterIP status: loadBalancer: {} Looking at the Prometheus Status Targets information, you can only see the Services with annotation: For more information about adding filter parameters to relabel, please check Prometheus-Relabel \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Config Native Prometheus"},{"location":"en/guide/prometheus/#config-native-prometheus","text":"Kube-OVN provides rich monitoring data for OVN/OVS health status checks and connectivity checks of container and host networks, and Kube-OVN is configured with ServiceMonitor for Prometheus to dynamically obtain monitoring metrics. In some cases, where only Prometheus Server is installed and no other components are installed, you can dynamically obtain monitoring data for the cluster environment by modifying the configuration of Prometheus.","title":"Config Native Prometheus"},{"location":"en/guide/prometheus/#config-prometheus","text":"The following configuration documentation, referenced from Prometheus Service Discovery .","title":"Config Prometheus"},{"location":"en/guide/prometheus/#permission-configuration","text":"Prometheus is deployed in the cluster and needs to access the k8s apiserver to query the monitoring data of the containers. Refer to the following yaml to configure the permissions required by Prometheus: apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : name : prometheus rules : - apiGroups : [ \"\" ] resources : - nodes - nodes/proxy - services - endpoints - pods verbs : [ \"get\" , \"list\" , \"watch\" ] - apiGroups : - extensions resources : - ingresses verbs : [ \"get\" , \"list\" , \"watch\" ] - nonResourceURLs : [ \"/metrics\" ] verbs : [ \"get\" ] --- apiVersion : v1 kind : ServiceAccount metadata : name : prometheus namespace : default --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : name : prometheus roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : prometheus subjects : - kind : ServiceAccount name : prometheus namespace : default","title":"Permission Configuration"},{"location":"en/guide/prometheus/#prometheus-configmap","text":"The startup of Prometheus relies on the configuration file prometheus.yml, the contents of which can be configured in ConfigMap and dynamically mounted to the Pod. Create the ConfigMap file used by Prometheus by referring to the following yaml: apiVersion : v1 kind : ConfigMap metadata : name : prometheus-config data : prometheus.yml : |- global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'kubernetes-nodes' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node - job_name: 'kubernetes-service' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: service - job_name: 'kubernetes-endpoints' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints - job_name: 'kubernetes-ingress' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: ingress - job_name: 'kubernetes-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: pod Prometheus provides role-based querying of Kubernetes resource monitoring operations, which can be configured in the official documentation kubernetes_sd_config \u3002","title":"Prometheus ConfigMap"},{"location":"en/guide/prometheus/#deploy-prometheus","text":"Deploy Prometheus Server by referring to the following yaml: apiVersion : apps/v1 kind : Deployment metadata : labels : app : prometheus name : prometheus namespace : default spec : replicas : 1 selector : matchLabels : app : prometheus strategy : rollingUpdate : maxSurge : 25% maxUnavailable : 25% type : RollingUpdate template : metadata : labels : app : prometheus spec : serviceAccountName : prometheus serviceAccount : prometheus containers : - image : docker.io/prom/prometheus:latest imagePullPolicy : IfNotPresent name : prometheus command : - \"/bin/prometheus\" args : - \"--config.file=/etc/prometheus/prometheus.yml\" ports : - containerPort : 9090 protocol : TCP volumeMounts : - mountPath : \"/etc/prometheus\" name : prometheus-config volumes : - name : prometheus-config configMap : name : prometheus-config Deploy Prometheus Service by referring to the following yaml: kind : Service apiVersion : v1 metadata : name : prometheus namespace : default labels : name : prometheus spec : ports : - name : test protocol : TCP port : 9090 targetPort : 9090 type : NodePort selector : app : prometheus sessionAffinity : None After exposing Prometheus through NodePort, Prometheus can be accessed through the node address.","title":"Deploy Prometheus"},{"location":"en/guide/prometheus/#prometheus-metrics-config","text":"View information about Prometheus on the environment: # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kubernetes ClusterIP 10 .4.0.1 <none> 443 /TCP 8d prometheus NodePort 10 .4.102.222 <none> 9090 :32611/TCP 8d # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES prometheus-7544b6b84d-v9m8s 1 /1 Running 0 3d5h 10 .3.0.7 192 .168.137.219 <none> <none> # kubectl get endpoints -o wide NAME ENDPOINTS AGE kubernetes 192 .168.136.228:6443,192.168.136.232:6443,192.168.137.219:6443 8d prometheus 10 .3.0.7:9090 8d Access Prometheus via NodePort to see the data dynamically queried by Status/Service Discovery: You can see that you can currently query all the service data information on the cluster.","title":"Prometheus Metrics Config"},{"location":"en/guide/prometheus/#configure-to-query-specified-resource","text":"The ConfigMap configuration above queries all resource data. If you only need resource data for a certain role, you can add filter conditions. Take Service as an example, modify the ConfigMap content to query only the service monitoring data: - job_name : 'kubernetes-service' tls_config : ca_file : /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file : /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs : - role : service relabel_configs : - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_scrape ] action : \"keep\" regex : \"true\" - action : labelmap regex : __meta_kubernetes_service_label_(.+) - source_labels : [ __meta_kubernetes_namespace ] target_label : kubernetes_namespace - source_labels : [ __meta_kubernetes_service_name ] target_label : kubernetes_service_name - source_labels : [ __meta_kubernetes_service_annotation_prometheus_io_path ] action : replace target_label : __metrics_path__ regex : \"(.+)\" Check the Kube-OVN Service in kube-system Namespace: # kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT ( S ) AGE kube-dns ClusterIP 10 .4.0.10 <none> 53 /UDP,53/TCP,9153/TCP 13d kube-ovn-cni ClusterIP 10 .4.228.60 <none> 10665 /TCP 13d kube-ovn-controller ClusterIP 10 .4.172.213 <none> 10660 /TCP 13d kube-ovn-monitor ClusterIP 10 .4.242.9 <none> 10661 /TCP 13d kube-ovn-pinger ClusterIP 10 .4.122.52 <none> 8080 /TCP 13d ovn-nb ClusterIP 10 .4.80.213 <none> 6641 /TCP 13d ovn-northd ClusterIP 10 .4.126.234 <none> 6643 /TCP 13d ovn-sb ClusterIP 10 .4.216.249 <none> 6642 /TCP 13d Add annotation prometheus.io/scrape=\"true\" to Service\uff1a # kubectl annotate svc -n kube-system kube-ovn-cni prometheus.io/scrape=true service/kube-ovn-cni annotated # kubectl annotate svc -n kube-system kube-ovn-controller prometheus.io/scrape=true service/kube-ovn-controller annotated # kubectl annotate svc -n kube-system kube-ovn-monitor prometheus.io/scrape=true service/kube-ovn-monitor annotated # kubectl annotate svc -n kube-system kube-ovn-pinger prometheus.io/scrape=true service/kube-ovn-pinger annotated Check the configured Service information: # kubectl get svc -o yaml -n kube-system kube-ovn-controller apiVersion: v1 kind: Service metadata: annotations: helm.sh/chart-version: v3.10.0-alpha.55 helm.sh/original-name: kube-ovn-controller ovn.kubernetes.io/vpc: ovn-cluster prometheus.io/scrape: \"true\" // added annotation labels: app: kube-ovn-controller name: kube-ovn-controller namespace: kube-system spec: clusterIP: 10 .4.172.213 clusterIPs: - 10 .4.172.213 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: metrics port: 10660 protocol: TCP targetPort: 10660 selector: app: kube-ovn-controller sessionAffinity: None type: ClusterIP status: loadBalancer: {} Looking at the Prometheus Status Targets information, you can only see the Services with annotation: For more information about adding filter parameters to relabel, please check Prometheus-Relabel \u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Configure to Query Specified Resource"},{"location":"en/guide/qos/","text":"Manage QoS \u00b6 Kube-OVN supports two types of Pod level QoS: Maximum bandwidth limit QoS. linux-netem , QoS for simulating latency and packet loss that can be used for simulation testing. Currently, only Pod level QoS is supported, and QoS restrictions at the Namespace or Subnet level are not supported. Maximum Bandwidth Limit QoS \u00b6 This type of QoS can be dynamically configured via Pod annotation and can be adjusted without restarting running Pod. Bandwidth speed limit unit is Mbit/s . apiVersion : v1 kind : Pod metadata : name : qos namespace : ls1 annotations : ovn.kubernetes.io/ingress_rate : \"3\" ovn.kubernetes.io/egress_rate : \"1\" spec : containers : - name : qos image : docker.io/library/nginx:alpine Use annotation to dynamically adjust QoS: kubectl annotate --overwrite pod nginx-74d5899f46-d7qkn ovn.kubernetes.io/ingress_rate = 3 Test QoS \u00b6 Deploy the containers needed for performance testing: kind : DaemonSet apiVersion : apps/v1 metadata : name : perf namespace : ls1 labels : app : perf spec : selector : matchLabels : app : perf template : metadata : labels : app : perf spec : containers : - name : nginx image : docker.io/kubeovn/perf Exec into one Pod and run iperf3 server: # kubectl exec -it perf-4n4gt -n ls1 sh # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Exec into the other Pod and run iperf3 client to connect above server address: # kubectl exec -it perf-d4mqc -n ls1 sh # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 51544 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 86 .4 MBytes 725 Mbits/sec 3 350 KBytes [ 4 ] 1 .00-2.00 sec 89 .9 MBytes 754 Mbits/sec 118 473 KBytes [ 4 ] 2 .00-3.00 sec 101 MBytes 848 Mbits/sec 184 586 KBytes [ 4 ] 3 .00-4.00 sec 104 MBytes 875 Mbits/sec 217 671 KBytes [ 4 ] 4 .00-5.00 sec 111 MBytes 935 Mbits/sec 175 772 KBytes [ 4 ] 5 .00-6.00 sec 100 MBytes 840 Mbits/sec 658 598 KBytes [ 4 ] 6 .00-7.00 sec 106 MBytes 890 Mbits/sec 742 668 KBytes [ 4 ] 7 .00-8.00 sec 102 MBytes 857 Mbits/sec 764 724 KBytes [ 4 ] 8 .00-9.00 sec 97 .4 MBytes 817 Mbits/sec 1175 764 KBytes [ 4 ] 9 .00-10.00 sec 111 MBytes 934 Mbits/sec 1083 838 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 1010 MBytes 848 Mbits/sec 5119 sender [ 4 ] 0 .00-10.00 sec 1008 MBytes 846 Mbits/sec receiver iperf Done. Modify the ingress bandwidth QoS for the first Pod: kubectl annotate --overwrite pod perf-4n4gt -n ls1 ovn.kubernetes.io/ingress_rate = 30 Test the Pod bandwidth again from the second Pod: # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 52372 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 3 .66 MBytes 30 .7 Mbits/sec 2 76 .1 KBytes [ 4 ] 1 .00-2.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 104 KBytes [ 4 ] 2 .00-3.00 sec 3 .50 MBytes 29 .4 Mbits/sec 0 126 KBytes [ 4 ] 3 .00-4.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 144 KBytes [ 4 ] 4 .00-5.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 160 KBytes [ 4 ] 5 .00-6.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 175 KBytes [ 4 ] 6 .00-7.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 212 KBytes [ 4 ] 7 .00-8.00 sec 3 .68 MBytes 30 .9 Mbits/sec 0 294 KBytes [ 4 ] 8 .00-9.00 sec 3 .74 MBytes 31 .4 Mbits/sec 0 398 KBytes [ 4 ] 9 .00-10.00 sec 3 .80 MBytes 31 .9 Mbits/sec 0 526 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 35 .7 MBytes 29 .9 Mbits/sec 2 sender [ 4 ] 0 .00-10.00 sec 34 .5 MBytes 29 .0 Mbits/sec receiver iperf Done. linux-netem QoS \u00b6 Pod can use annotation below to config linux-netem type QoS\uff1a ovn.kubernetes.io/latency \u3001 ovn.kubernetes.io/limit and ovn.kubernetes.io/loss \u3002 ovn.kubernetes.io/latency : Set the Pod traffic delay to an integer value in ms. ovn.kubernetes.io/limit \uff1a Set the maximum number of packets that the qdisc queue can hold, and takes an integer value, such as 1000. ovn.kubernetes.io/loss \uff1a Set packet loss probability, the value is float type, for example, the value is 20, then it is set 20% packet loss probability. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Manage QoS"},{"location":"en/guide/qos/#manage-qos","text":"Kube-OVN supports two types of Pod level QoS: Maximum bandwidth limit QoS. linux-netem , QoS for simulating latency and packet loss that can be used for simulation testing. Currently, only Pod level QoS is supported, and QoS restrictions at the Namespace or Subnet level are not supported.","title":"Manage QoS"},{"location":"en/guide/qos/#maximum-bandwidth-limit-qos","text":"This type of QoS can be dynamically configured via Pod annotation and can be adjusted without restarting running Pod. Bandwidth speed limit unit is Mbit/s . apiVersion : v1 kind : Pod metadata : name : qos namespace : ls1 annotations : ovn.kubernetes.io/ingress_rate : \"3\" ovn.kubernetes.io/egress_rate : \"1\" spec : containers : - name : qos image : docker.io/library/nginx:alpine Use annotation to dynamically adjust QoS: kubectl annotate --overwrite pod nginx-74d5899f46-d7qkn ovn.kubernetes.io/ingress_rate = 3","title":"Maximum Bandwidth Limit QoS"},{"location":"en/guide/qos/#test-qos","text":"Deploy the containers needed for performance testing: kind : DaemonSet apiVersion : apps/v1 metadata : name : perf namespace : ls1 labels : app : perf spec : selector : matchLabels : app : perf template : metadata : labels : app : perf spec : containers : - name : nginx image : docker.io/kubeovn/perf Exec into one Pod and run iperf3 server: # kubectl exec -it perf-4n4gt -n ls1 sh # iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Exec into the other Pod and run iperf3 client to connect above server address: # kubectl exec -it perf-d4mqc -n ls1 sh # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 51544 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 86 .4 MBytes 725 Mbits/sec 3 350 KBytes [ 4 ] 1 .00-2.00 sec 89 .9 MBytes 754 Mbits/sec 118 473 KBytes [ 4 ] 2 .00-3.00 sec 101 MBytes 848 Mbits/sec 184 586 KBytes [ 4 ] 3 .00-4.00 sec 104 MBytes 875 Mbits/sec 217 671 KBytes [ 4 ] 4 .00-5.00 sec 111 MBytes 935 Mbits/sec 175 772 KBytes [ 4 ] 5 .00-6.00 sec 100 MBytes 840 Mbits/sec 658 598 KBytes [ 4 ] 6 .00-7.00 sec 106 MBytes 890 Mbits/sec 742 668 KBytes [ 4 ] 7 .00-8.00 sec 102 MBytes 857 Mbits/sec 764 724 KBytes [ 4 ] 8 .00-9.00 sec 97 .4 MBytes 817 Mbits/sec 1175 764 KBytes [ 4 ] 9 .00-10.00 sec 111 MBytes 934 Mbits/sec 1083 838 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 1010 MBytes 848 Mbits/sec 5119 sender [ 4 ] 0 .00-10.00 sec 1008 MBytes 846 Mbits/sec receiver iperf Done. Modify the ingress bandwidth QoS for the first Pod: kubectl annotate --overwrite pod perf-4n4gt -n ls1 ovn.kubernetes.io/ingress_rate = 30 Test the Pod bandwidth again from the second Pod: # iperf3 -c 10.66.0.12 Connecting to host 10 .66.0.12, port 5201 [ 4 ] local 10 .66.0.14 port 52372 connected to 10 .66.0.12 port 5201 [ ID ] Interval Transfer Bandwidth Retr Cwnd [ 4 ] 0 .00-1.00 sec 3 .66 MBytes 30 .7 Mbits/sec 2 76 .1 KBytes [ 4 ] 1 .00-2.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 104 KBytes [ 4 ] 2 .00-3.00 sec 3 .50 MBytes 29 .4 Mbits/sec 0 126 KBytes [ 4 ] 3 .00-4.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 144 KBytes [ 4 ] 4 .00-5.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 160 KBytes [ 4 ] 5 .00-6.00 sec 3 .43 MBytes 28 .8 Mbits/sec 0 175 KBytes [ 4 ] 6 .00-7.00 sec 3 .50 MBytes 29 .3 Mbits/sec 0 212 KBytes [ 4 ] 7 .00-8.00 sec 3 .68 MBytes 30 .9 Mbits/sec 0 294 KBytes [ 4 ] 8 .00-9.00 sec 3 .74 MBytes 31 .4 Mbits/sec 0 398 KBytes [ 4 ] 9 .00-10.00 sec 3 .80 MBytes 31 .9 Mbits/sec 0 526 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID ] Interval Transfer Bandwidth Retr [ 4 ] 0 .00-10.00 sec 35 .7 MBytes 29 .9 Mbits/sec 2 sender [ 4 ] 0 .00-10.00 sec 34 .5 MBytes 29 .0 Mbits/sec receiver iperf Done.","title":"Test QoS"},{"location":"en/guide/qos/#linux-netem-qos","text":"Pod can use annotation below to config linux-netem type QoS\uff1a ovn.kubernetes.io/latency \u3001 ovn.kubernetes.io/limit and ovn.kubernetes.io/loss \u3002 ovn.kubernetes.io/latency : Set the Pod traffic delay to an integer value in ms. ovn.kubernetes.io/limit \uff1a Set the maximum number of packets that the qdisc queue can hold, and takes an integer value, such as 1000. ovn.kubernetes.io/loss \uff1a Set packet loss probability, the value is float type, for example, the value is 20, then it is set 20% packet loss probability. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"linux-netem QoS"},{"location":"en/guide/setup-options/","text":"Installation and Configuration Options \u00b6 In One-Click Installation we use the default configuration for installation. Kube-OVN also supports more custom configurations, which can be configured in the installation script, or later by changing the parameters of individual components. This document will describe what these customization options do, and how to configure them. Built-in Network Settings \u00b6 Kube-OVN will configure two built-in Subnets during installation: default Subnet, as the default subnet used by the Pod to assign IPs, with a default CIDR of 10.16.0.0/16 and a gateway of 10.16.0.1 . The join subnet, as a special subnet for network communication between the Node and Pod, has a default CIDR of 100.64.0.0/16 and a gateway of 100.64.0.1 . The configuration of these two subnets can be changed during installation via the installation scripts variables: POD_CIDR = \"10.16.0.0/16\" POD_GATEWAY = \"10.16.0.1\" JOIN_CIDR = \"100.64.0.0/16\" EXCLUDE_IPS = \"\" EXCLUDE_IP sets the address range for which kube-ovn-controller will not automatically assign from it, the format is: 192.168.10.20..192.168.10.30 . Note that in the Overlay case these two Subnets CIDRs cannot conflict with existing host networks and Service CIDRs. You can change the address range of both Subnets after installation by referring to Change Subnet CIDR and Change Join Subnet CIDR . Config Service CIDR \u00b6 Since some of the iptables and routing rules set by kube-proxy will conflict with the rules set by Kube-OVN, Kube-OVN needs to know the CIDR of the service to set the corresponding rules correctly. This can be done by modifying the installation script: SVC_CIDR = \"10.96.0.0/12\" You can also modify the args of the kube-ovn-controller Deployment after installation: args : - --service-cluster-ip-range=10.96.0.0/12 Overlay NIC Selection \u00b6 In the case of multiple NICs on a node, Kube-OVN will select the NIC corresponding to the Kubernetes Node IP as the NIC for cross-node communication between containers and establish the corresponding tunnel. If you need to select another NIC to create a container tunnel, you can change it in the installation script: IFACE = eth1 This option supports regular expressions separated by commas, e.g. 'ens[a-z0-9] ,eth[a-z0-9] '. It can also be adjusted after installation by modifying the args of the kube-ovn-cni DaemonSet: args : - --iface=eth1 If each machine has a different NIC name and there is no fixed pattern, you can use the node annotation ovn.kubernetes.io/tunnel_interface to configure each node one by one. This annotation will override the configuration of iface . kubectl annotate node no1 ovn.kubernetes.io/tunnel_interface = ethx Config MTU \u00b6 Since Overlay encapsulation requires additional space, Kube-OVN will adjust the MTU of the container NIC based on the MTU of the selected NIC when creating the container NIC. By default, the Pod NIC MTU is the host NIC MTU - 100 on the Overlay Subnet, and the Pod NIC and host NIC have the same MTU on the Underlay Subnet. If you need to adjust the size of the MTU under the Overlay subnet, you can modify the parameters of the kube-ovn-cni DaemonSet: args : - --mtu=1333 Global Traffic Mirroring Setting \u00b6 When global traffic mirroring is enabled, Kube-OVN will create a mirror0 virtual NIC on each node and copy all container network traffic from the current machine to that NIC\uff0c Users can perform traffic analysis with tcpdump and other tools. This function can be enabled in the installation script: ENABLE_MIRROR = true It can also be adjusted after installation by modifying the args of the kube-ovn-cni DaemonSet: args : - --enable-mirror=true The ability to mirror traffic is disabled in the default installation, if you need fine-grained traffic mirroring or need to mirror traffic to additional NICs please refer to Traffic Mirror . LB Settings \u00b6 Kube-OVN uses L2 LB in OVN to implement service forwarding. In Overlay scenarios, users can choose to use kube-proxy for service traffic forwarding, in which case the LB function of Kube-OVN can be disabled to achieve better performance on the control plane and data plane. This feature can be configured in the installation script: ENABLE_LB = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-lb=false The LB feature is enabled in the default installation. The spec field enableLb has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the LB function of Kube-OVN to the subnet level. You can set whether to enable the LB function based on different subnets. The enable-lb parameter in the kube-ovn-controller deployment is used as a global switch to control whether to create a load-balancer record. The enableLb parameter added in the subnet is used to control whether the subnet is associated with a load-balancer record. After the previous version is upgraded to v1.12.0, the enableLb parameter of the subnet will automatically inherit the value of the original global switch parameter. NetworkPolicy Settings \u00b6 Kube-OVN uses ACLs in OVN to implement NetworkPolicy. Users can choose to disable the NetworkPolicy feature or use the Cilium Chain approach to implement NetworkPolicy using eBPF. In this case, the NetworkPolicy feature of Kube-OVN can be disabled to achieve better performance on the control plane and data plane. This feature can be configured in the installation script: ENABLE_NP = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-np=false NetworkPolicy is enabled by default. EIP and SNAT Settings \u00b6 If the EIP and SNAT capabilities are not required on the default VPC, users can choose to disable them to reduce the performance overhead of kube-ovn-controller in large scale cluster environments and improve processing speed. This feature can be configured in the installation script: ENABLE_EIP_SNAT = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-eip-snat=false EIP and SNAT is enabled by default. More information can refer to EIP and SNAT \u3002 Centralized Gateway ECMP Settings \u00b6 The centralized gateway supports two mode of high availability, primary-backup and ECMP. If you want to enable ECMP mode, you need to change the args of kube-ovn-controller Deployment: args : - --enable-ecmp=true Centralized gateway default installation under the primary-backup mode, more gateway-related content please refer to Config Subnet . The spec field enableEcmp has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the ECMP switch to the subnet level. You can set whether to enable ECMP mode based on different subnets. The enable-ecmp parameter in the kube-ovn-controller deployment is no longer used. After the previous version is upgraded to v1.12.0, the subnet switch will automatically inherit the value of the original global switch parameter. Kubevirt VM Fixed Address Settings \u00b6 For VM instances created by Kubevirt, kube-ovn-controller can assign and manage IP addresses in a similar way to the StatefulSet Pod. This allows VM instances address fixed during start-up, shutdown, upgrade, migration, and other operations throughout their lifecycle, making them more compatible with the actual virtualization user experience. This feature is enable by default after v1.10.6. To disable this feature, you need to change the following args in the kube-ovn-controller Deployment: args : - --keep-vm-ip=false CNI Settings \u00b6 KBy default, Kube-OVN installs the CNI binary in the /opt/cni/bin directory and the CNI configuration file 01-kube-ovn.conflist in the /etc/cni/net.d directory. If you need to change the installation location and the priority of the CNI configuration file, you can modify the following parameters of the installation script. CNI_CONF_DIR = \"/etc/cni/net.d\" CNI_BIN_DIR = \"/opt/cni/bin\" CNI_CONFIG_PRIORITY = \"01\" Or change the Volume mount and args of the kube-ovn-cni DaemonSet after installation: volumes : - name : cni-conf hostPath : path : \"/etc/cni/net.d\" - name : cni-bin hostPath : path:\"/opt/cni/bin\" ... args : - --cni-conf-name=01-kube-ovn.conflist Tunnel Type Settings \u00b6 The default encapsulation mode of Kube-OVN Overlay is Geneve, if you want to change it to Vxlan or STT, please adjust the following parameters in the installation script: TUNNEL_TYPE = \"vxlan\" Or change the environment variables of ovs-ovn DaemonSet after installation: env : - name : TUNNEL_TYPE value : \"vxlan\" If you need to use the STT tunnel and need to compile additional kernel modules for ovs, please refer to Performance Tunning \u3002 Please refer to Tunneling Protocol Selection for the differences between the different protocols in practice. SSL Settings \u00b6 The OVN DB API interface supports SSL encryption to secure the connection. To enable it, adjust the following parameters in the installation script: ENABLE_SSL = true The SSL is disabled by default. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Installation and Configuration Options"},{"location":"en/guide/setup-options/#installation-and-configuration-options","text":"In One-Click Installation we use the default configuration for installation. Kube-OVN also supports more custom configurations, which can be configured in the installation script, or later by changing the parameters of individual components. This document will describe what these customization options do, and how to configure them.","title":"Installation and Configuration Options"},{"location":"en/guide/setup-options/#built-in-network-settings","text":"Kube-OVN will configure two built-in Subnets during installation: default Subnet, as the default subnet used by the Pod to assign IPs, with a default CIDR of 10.16.0.0/16 and a gateway of 10.16.0.1 . The join subnet, as a special subnet for network communication between the Node and Pod, has a default CIDR of 100.64.0.0/16 and a gateway of 100.64.0.1 . The configuration of these two subnets can be changed during installation via the installation scripts variables: POD_CIDR = \"10.16.0.0/16\" POD_GATEWAY = \"10.16.0.1\" JOIN_CIDR = \"100.64.0.0/16\" EXCLUDE_IPS = \"\" EXCLUDE_IP sets the address range for which kube-ovn-controller will not automatically assign from it, the format is: 192.168.10.20..192.168.10.30 . Note that in the Overlay case these two Subnets CIDRs cannot conflict with existing host networks and Service CIDRs. You can change the address range of both Subnets after installation by referring to Change Subnet CIDR and Change Join Subnet CIDR .","title":"Built-in Network Settings"},{"location":"en/guide/setup-options/#config-service-cidr","text":"Since some of the iptables and routing rules set by kube-proxy will conflict with the rules set by Kube-OVN, Kube-OVN needs to know the CIDR of the service to set the corresponding rules correctly. This can be done by modifying the installation script: SVC_CIDR = \"10.96.0.0/12\" You can also modify the args of the kube-ovn-controller Deployment after installation: args : - --service-cluster-ip-range=10.96.0.0/12","title":"Config Service CIDR"},{"location":"en/guide/setup-options/#overlay-nic-selection","text":"In the case of multiple NICs on a node, Kube-OVN will select the NIC corresponding to the Kubernetes Node IP as the NIC for cross-node communication between containers and establish the corresponding tunnel. If you need to select another NIC to create a container tunnel, you can change it in the installation script: IFACE = eth1 This option supports regular expressions separated by commas, e.g. 'ens[a-z0-9] ,eth[a-z0-9] '. It can also be adjusted after installation by modifying the args of the kube-ovn-cni DaemonSet: args : - --iface=eth1 If each machine has a different NIC name and there is no fixed pattern, you can use the node annotation ovn.kubernetes.io/tunnel_interface to configure each node one by one. This annotation will override the configuration of iface . kubectl annotate node no1 ovn.kubernetes.io/tunnel_interface = ethx","title":"Overlay NIC Selection"},{"location":"en/guide/setup-options/#config-mtu","text":"Since Overlay encapsulation requires additional space, Kube-OVN will adjust the MTU of the container NIC based on the MTU of the selected NIC when creating the container NIC. By default, the Pod NIC MTU is the host NIC MTU - 100 on the Overlay Subnet, and the Pod NIC and host NIC have the same MTU on the Underlay Subnet. If you need to adjust the size of the MTU under the Overlay subnet, you can modify the parameters of the kube-ovn-cni DaemonSet: args : - --mtu=1333","title":"Config MTU"},{"location":"en/guide/setup-options/#global-traffic-mirroring-setting","text":"When global traffic mirroring is enabled, Kube-OVN will create a mirror0 virtual NIC on each node and copy all container network traffic from the current machine to that NIC\uff0c Users can perform traffic analysis with tcpdump and other tools. This function can be enabled in the installation script: ENABLE_MIRROR = true It can also be adjusted after installation by modifying the args of the kube-ovn-cni DaemonSet: args : - --enable-mirror=true The ability to mirror traffic is disabled in the default installation, if you need fine-grained traffic mirroring or need to mirror traffic to additional NICs please refer to Traffic Mirror .","title":"Global Traffic Mirroring Setting"},{"location":"en/guide/setup-options/#lb-settings","text":"Kube-OVN uses L2 LB in OVN to implement service forwarding. In Overlay scenarios, users can choose to use kube-proxy for service traffic forwarding, in which case the LB function of Kube-OVN can be disabled to achieve better performance on the control plane and data plane. This feature can be configured in the installation script: ENABLE_LB = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-lb=false The LB feature is enabled in the default installation. The spec field enableLb has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the LB function of Kube-OVN to the subnet level. You can set whether to enable the LB function based on different subnets. The enable-lb parameter in the kube-ovn-controller deployment is used as a global switch to control whether to create a load-balancer record. The enableLb parameter added in the subnet is used to control whether the subnet is associated with a load-balancer record. After the previous version is upgraded to v1.12.0, the enableLb parameter of the subnet will automatically inherit the value of the original global switch parameter.","title":"LB Settings"},{"location":"en/guide/setup-options/#networkpolicy-settings","text":"Kube-OVN uses ACLs in OVN to implement NetworkPolicy. Users can choose to disable the NetworkPolicy feature or use the Cilium Chain approach to implement NetworkPolicy using eBPF. In this case, the NetworkPolicy feature of Kube-OVN can be disabled to achieve better performance on the control plane and data plane. This feature can be configured in the installation script: ENABLE_NP = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-np=false NetworkPolicy is enabled by default.","title":"NetworkPolicy Settings"},{"location":"en/guide/setup-options/#eip-and-snat-settings","text":"If the EIP and SNAT capabilities are not required on the default VPC, users can choose to disable them to reduce the performance overhead of kube-ovn-controller in large scale cluster environments and improve processing speed. This feature can be configured in the installation script: ENABLE_EIP_SNAT = false It can also be configured after installation by changing the args of the kube-ovn-controller Deployment: args : - --enable-eip-snat=false EIP and SNAT is enabled by default. More information can refer to EIP and SNAT \u3002","title":"EIP and SNAT Settings"},{"location":"en/guide/setup-options/#centralized-gateway-ecmp-settings","text":"The centralized gateway supports two mode of high availability, primary-backup and ECMP. If you want to enable ECMP mode, you need to change the args of kube-ovn-controller Deployment: args : - --enable-ecmp=true Centralized gateway default installation under the primary-backup mode, more gateway-related content please refer to Config Subnet . The spec field enableEcmp has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the ECMP switch to the subnet level. You can set whether to enable ECMP mode based on different subnets. The enable-ecmp parameter in the kube-ovn-controller deployment is no longer used. After the previous version is upgraded to v1.12.0, the subnet switch will automatically inherit the value of the original global switch parameter.","title":"Centralized Gateway ECMP Settings"},{"location":"en/guide/setup-options/#kubevirt-vm-fixed-address-settings","text":"For VM instances created by Kubevirt, kube-ovn-controller can assign and manage IP addresses in a similar way to the StatefulSet Pod. This allows VM instances address fixed during start-up, shutdown, upgrade, migration, and other operations throughout their lifecycle, making them more compatible with the actual virtualization user experience. This feature is enable by default after v1.10.6. To disable this feature, you need to change the following args in the kube-ovn-controller Deployment: args : - --keep-vm-ip=false","title":"Kubevirt VM Fixed Address Settings"},{"location":"en/guide/setup-options/#cni-settings","text":"KBy default, Kube-OVN installs the CNI binary in the /opt/cni/bin directory and the CNI configuration file 01-kube-ovn.conflist in the /etc/cni/net.d directory. If you need to change the installation location and the priority of the CNI configuration file, you can modify the following parameters of the installation script. CNI_CONF_DIR = \"/etc/cni/net.d\" CNI_BIN_DIR = \"/opt/cni/bin\" CNI_CONFIG_PRIORITY = \"01\" Or change the Volume mount and args of the kube-ovn-cni DaemonSet after installation: volumes : - name : cni-conf hostPath : path : \"/etc/cni/net.d\" - name : cni-bin hostPath : path:\"/opt/cni/bin\" ... args : - --cni-conf-name=01-kube-ovn.conflist","title":"CNI Settings"},{"location":"en/guide/setup-options/#tunnel-type-settings","text":"The default encapsulation mode of Kube-OVN Overlay is Geneve, if you want to change it to Vxlan or STT, please adjust the following parameters in the installation script: TUNNEL_TYPE = \"vxlan\" Or change the environment variables of ovs-ovn DaemonSet after installation: env : - name : TUNNEL_TYPE value : \"vxlan\" If you need to use the STT tunnel and need to compile additional kernel modules for ovs, please refer to Performance Tunning \u3002 Please refer to Tunneling Protocol Selection for the differences between the different protocols in practice.","title":"Tunnel Type Settings"},{"location":"en/guide/setup-options/#ssl-settings","text":"The OVN DB API interface supports SSL encryption to secure the connection. To enable it, adjust the following parameters in the installation script: ENABLE_SSL = true The SSL is disabled by default. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"SSL Settings"},{"location":"en/guide/static-ip-mac/","text":"Fixed Addresses \u00b6 By default, Kube-OVN randomly assigns IPs and Macs based on the Subnet to which the Pod's Namespace belongs. For workloads that require fixed addresses, Kube-OVN provides multiple methods of fixing addresses depending on the scenario. Single Pod fixed IP/Mac. Workload IP Pool to specify fixed addresses. StatefulSet fixed address. KubeVirt VM fixed address. Single Pod Fixed IP/Mac \u00b6 You can specify the IP/Mac required for the Pod by annotation when creating the Pod. The kube-ovn-controller will skip the address random assignment phase and use the specified address directly after conflict detection, as follows: apiVersion : apps/v1 kind : Deployment metadata : name : ippool labels : app : ippool spec : replicas : 2 selector : matchLabels : app : ippool template : metadata : labels : app : ippool annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u5206\u53f7\u8fdb\u884c\u5206\u9694 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::0 spec : containers : - name : ippool image : docker.io/library/nginx:alpine The following points need to be noted when using annotation. The IP/Mac used cannot conflict with an existing IP/Mac. The IP must be in the CIDR range of the Subnet it belongs to. You can specify only IP or Mac. When you specify only one, the other one will be assigned randomly. Workload IP Pool \u00b6 Kube-OVN supports setting fixed IPs for Workloads (Deployment/StatefulSet/DaemonSet/Job/CronJob) via annotation ovn.kubernetes.io/ip_pool . kube-ovn-controllerr will automatically select the IP specified in ovn.kubernetes.io/ip_pool and perform conflict detection. The Annotation of the IP Pool needs to be added to the annotation field in the template . In addition to Kubernetes built-in workload types, other user-defined workloads can also be assigned fixed addresses using the same approach. Deployment With Fixed IPs \u00b6 apiVersion : apps/v1 kind : Deployment metadata : namespace : ls1 name : starter-backend labels : app : starter-backend spec : replicas : 2 selector : matchLabels : app : starter-backend template : metadata : labels : app : starter-backend annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // for dualstack ippool use semicolon to separate addresses 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::000F;10.16.0.17,fd00:10:16::0010 spec : containers : - name : backend image : docker.io/library/nginx:alpine Using a fixed IP for Workload requires the following: The IP in ovn.kubernetes.io/ip_pool should belong to the CIDR of the Subnet. The IP in ovn.kubernetes.io/ip_pool cannot conflict with an IP already in use. When the number of IPs in ovn.kubernetes.io/ip_pool is less than the number of replicas, the extra Pods will not be created. You need to adjust the number of IPs in ovn.kubernetes.io/ip_pool according to the update policy of the workload and the scaling plan. StatefulSet Fixed Address \u00b6 StatefulSet, like other workloads, can use ovn.kubernetes.io/ip_pool to specify the IP used by the Pod. Since StatefulSet is mostly used for stateful services, which have higher requirements for fixed addresses, Kube-OVN has made special enhancements: Pods are assigned IPs in ovn.kubernetes.io/ip_pool in order. For example, if the name of the StatefulSet is web, web-0 will use the first IP in ovn.kubernetes.io/ip_pool , web-1 will use the second IP, and so on. The logical_switch_port in the OVN is not deleted during update or deletion of the StatefulSet Pod, and the newly generated Pod directly reuses the old logical port information. Pods can therefore reuse IP/Mac and other network information to achieve similar state retention as StatefulSet Volumes. Based on the capabilities of 2, for StatefulSet without the ovn.kubernetes.io/ip_pool annotation, a Pod is randomly assigned an IP/Mac when it is first generated, and then the network information remains fixed for the lifetime of the StatefulSet. StatefulSet Example \u00b6 apiVersion : apps/v1 kind : StatefulSet metadata : name : web spec : serviceName : \"nginx\" replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx spec : containers : - name : nginx image : docker.io/library/nginx:alpine ports : - containerPort : 80 name : web You can try to delete the Pod under StatefulSet to observe if the Pod IP changes. KubeVirt VM Fixed Address \u00b6 For VM instances created by KubeVirt, kube-ovn-controller can assign and manage IP addresses in a similar way to the StatefulSet Pod. This allows VM instances address fixed during start-up, shutdown, upgrade, migration, and other operations throughout their lifecycle, making them more compatible with the actual virtualization user experience. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Fixed Addresses"},{"location":"en/guide/static-ip-mac/#fixed-addresses","text":"By default, Kube-OVN randomly assigns IPs and Macs based on the Subnet to which the Pod's Namespace belongs. For workloads that require fixed addresses, Kube-OVN provides multiple methods of fixing addresses depending on the scenario. Single Pod fixed IP/Mac. Workload IP Pool to specify fixed addresses. StatefulSet fixed address. KubeVirt VM fixed address.","title":"Fixed Addresses"},{"location":"en/guide/static-ip-mac/#single-pod-fixed-ipmac","text":"You can specify the IP/Mac required for the Pod by annotation when creating the Pod. The kube-ovn-controller will skip the address random assignment phase and use the specified address directly after conflict detection, as follows: apiVersion : apps/v1 kind : Deployment metadata : name : ippool labels : app : ippool spec : replicas : 2 selector : matchLabels : app : ippool template : metadata : labels : app : ippool annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // \u53cc\u6808\u5730\u5740\u4f7f\u7528\u5206\u53f7\u8fdb\u884c\u5206\u9694 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::0 spec : containers : - name : ippool image : docker.io/library/nginx:alpine The following points need to be noted when using annotation. The IP/Mac used cannot conflict with an existing IP/Mac. The IP must be in the CIDR range of the Subnet it belongs to. You can specify only IP or Mac. When you specify only one, the other one will be assigned randomly.","title":"Single Pod Fixed IP/Mac"},{"location":"en/guide/static-ip-mac/#workload-ip-pool","text":"Kube-OVN supports setting fixed IPs for Workloads (Deployment/StatefulSet/DaemonSet/Job/CronJob) via annotation ovn.kubernetes.io/ip_pool . kube-ovn-controllerr will automatically select the IP specified in ovn.kubernetes.io/ip_pool and perform conflict detection. The Annotation of the IP Pool needs to be added to the annotation field in the template . In addition to Kubernetes built-in workload types, other user-defined workloads can also be assigned fixed addresses using the same approach.","title":"Workload IP Pool"},{"location":"en/guide/static-ip-mac/#deployment-with-fixed-ips","text":"apiVersion : apps/v1 kind : Deployment metadata : namespace : ls1 name : starter-backend labels : app : starter-backend spec : replicas : 2 selector : matchLabels : app : starter-backend template : metadata : labels : app : starter-backend annotations : ovn.kubernetes.io/ip_pool : 10.16.0.15,10.16.0.16,10.16.0.17 // for dualstack ippool use semicolon to separate addresses 10.16.0.15,fd00:10:16::000E;10.16.0.16,fd00:10:16::000F;10.16.0.17,fd00:10:16::0010 spec : containers : - name : backend image : docker.io/library/nginx:alpine Using a fixed IP for Workload requires the following: The IP in ovn.kubernetes.io/ip_pool should belong to the CIDR of the Subnet. The IP in ovn.kubernetes.io/ip_pool cannot conflict with an IP already in use. When the number of IPs in ovn.kubernetes.io/ip_pool is less than the number of replicas, the extra Pods will not be created. You need to adjust the number of IPs in ovn.kubernetes.io/ip_pool according to the update policy of the workload and the scaling plan.","title":"Deployment With Fixed IPs"},{"location":"en/guide/static-ip-mac/#statefulset-fixed-address","text":"StatefulSet, like other workloads, can use ovn.kubernetes.io/ip_pool to specify the IP used by the Pod. Since StatefulSet is mostly used for stateful services, which have higher requirements for fixed addresses, Kube-OVN has made special enhancements: Pods are assigned IPs in ovn.kubernetes.io/ip_pool in order. For example, if the name of the StatefulSet is web, web-0 will use the first IP in ovn.kubernetes.io/ip_pool , web-1 will use the second IP, and so on. The logical_switch_port in the OVN is not deleted during update or deletion of the StatefulSet Pod, and the newly generated Pod directly reuses the old logical port information. Pods can therefore reuse IP/Mac and other network information to achieve similar state retention as StatefulSet Volumes. Based on the capabilities of 2, for StatefulSet without the ovn.kubernetes.io/ip_pool annotation, a Pod is randomly assigned an IP/Mac when it is first generated, and then the network information remains fixed for the lifetime of the StatefulSet.","title":"StatefulSet Fixed Address"},{"location":"en/guide/static-ip-mac/#statefulset-example","text":"apiVersion : apps/v1 kind : StatefulSet metadata : name : web spec : serviceName : \"nginx\" replicas : 2 selector : matchLabels : app : nginx template : metadata : labels : app : nginx spec : containers : - name : nginx image : docker.io/library/nginx:alpine ports : - containerPort : 80 name : web You can try to delete the Pod under StatefulSet to observe if the Pod IP changes.","title":"StatefulSet Example"},{"location":"en/guide/static-ip-mac/#kubevirt-vm-fixed-address","text":"For VM instances created by KubeVirt, kube-ovn-controller can assign and manage IP addresses in a similar way to the StatefulSet Pod. This allows VM instances address fixed during start-up, shutdown, upgrade, migration, and other operations throughout their lifecycle, making them more compatible with the actual virtualization user experience. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"KubeVirt VM Fixed Address"},{"location":"en/guide/subnet/","text":"Config Subnet \u00b6 Subnet is a core concept and basic unit of use in Kube-OVN, and Kube-OVN organizes IP and network configuration in terms of Subnet. Each Namespace can belong to a specific Subnet, and Pods under the Namespace automatically obtain IPs from the Subnet they belong to and share the network configuration (CIDR, gateway type, access control, NAT control, etc.). Unlike other CNI implementations where each node is bound to a subnet, in Kube-OVN the Subnet is a global level virtual network configuration, and the addresses of one Subnet can be distributed on any node. There are some differences in the usage and configuration of Overlay and Underlay Subnets, and this document will describe the common configurations and differentiated features of the different types of Subnets. Default Subnet \u00b6 To make it easier for users to get started quickly, Kube-OVN has a built-in default Subnet, all Namespaces that do not explicitly declare subnet affiliation are automatically assigned IPs from the default subnet and the network information. The configuration of this Subnet is specified at installation time, you can refer to Built-in Network Settings for more details. To change the CIDR of the default Subnet after installation please refer to Change Subnet CIDR . In Overlay mode, the default Subnet uses a distributed gateway and NAT translation for outbound traffic, which behaves much the same as the Flannel's default behavior, allowing users to use most of the network features without additional configuration. In Underlay mode, the default Subnet uses the physical gateway as the outgoing gateway and enables arping to check network connectivity. Check the Default Subnet \u00b6 The default field in the default Subnet spec is set to true , and there is only one default Subnet in a cluster, named ovn-default . # kubectl get subnet ovn-default -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: ovn-default resourceVersion: \"1571334\" selfLink: /apis/kubeovn.io/v1/subnets/ovn-default uid: 7e2451f8-fb44-4f7f-b3e0-cfd27f6fd5d6 spec: cidrBlock: 10 .16.0.0/16 default: true excludeIps: - 10 .16.0.1 gateway: 10 .16.0.1 gatewayType: distributed natOutgoing: true private: false protocol: IPv4 Join Subnet \u00b6 In the Kubernetes network specification, it is required that Nodes can communicate directly with all Pods. To achieve this in Overlay network mode, Kube-OVN creates a join Subnet and creates a virtual NIC ovn0 at each node that connect to the join subnet, through which the nodes and Pods can communicate with each other. The configuration of this Subnet is specified at installation time, you can refer to Built-in Network Settings for more details. To change the CIDR of the Join Subnet after installation please refer to Change Join CIDR . Check the Join Subnet \u00b6 The default name of this subnet is join . There is generally no need to make changes to the network configuration except the CIDR. # kubectl get subnet join -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: join resourceVersion: \"1571333\" selfLink: /apis/kubeovn.io/v1/subnets/join uid: 9c744810-c678-4d50-8a7d-b8ec12ef91b8 spec: cidrBlock: 100 .64.0.0/16 default: false excludeIps: - 100 .64.0.1 gateway: 100 .64.0.1 gatewayNode: \"\" gatewayType: \"\" natOutgoing: false private: false protocol: IPv4 Check the ovn0 NIC at the node: # ifconfig ovn0 ovn0: flags = 4163 <UP,BROADCAST,RUNNING,MULTICAST> mtu 1420 inet 100 .64.0.4 netmask 255 .255.0.0 broadcast 100 .64.255.255 inet6 fe80::800:ff:fe40:5 prefixlen 64 scopeid 0x20<link> ether 0a:00:00:40:00:05 txqueuelen 1000 ( Ethernet ) RX packets 18 bytes 1428 ( 1 .3 KiB ) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 19 bytes 1810 ( 1 .7 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Create Custom Subnets \u00b6 Here we describe the basic operation of how to create a Subnet and associate it with a Namespace, for more advanced configuration, please refer to the subsequent content. Create Subnet \u00b6 cat <<EOF | kubectl create -f - apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 10.66.0.0/16 excludeIps: - 10.66.0.1..10.66.0.10 - 10.66.0.101..10.66.0.151 gateway: 10.66.0.1 gatewayType: distributed natOutgoing: true routeTable: \"\" namespaces: - ns1 - ns2 EOF cidrBlock : Subnet CIDR range, different Subnet CIDRs under the same VPC cannot overlap. excludeIps : The address list is reserved so that the container network will not automatically assign addresses in the list, which can be used as a fixed IP address assignment segment or to avoid conflicts with existing devices in the physical network in Underlay mode. gateway \uff1aFor this subnet gateway address, Kube-OVN will automatically assign the corresponding logical gateway in Overlay mode, and the address should be the underlying physical gateway address in Underlay mode. namespaces : Bind the list of Namespace for this Subnet. Pods under the Namespace will be assigned addresses from the current Subnet after binding. routeTable : Associate the route table, default is main table, route table definition please defer to Static Routes Create Pod in the Subnet \u00b6 # kubectl create ns ns1 namespace/ns1 created # kubectl run nginx --image=docker.io/library/nginx:alpine -n ns1 deployment.apps/nginx created # kubectl get pod -n ns1 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-74d5899f46-n8wtg 1 /1 Running 0 10s 10 .66.0.11 node1 <none> <none> Overlay Subnet Gateway Settings \u00b6 This feature only works for Overlay mode Subnets, Underlay type Subnets need to use the underlying physical gateway to access the external network. Pods under the Overlay Subnet need to access the external network through a gateway, and Kube-OVN currently supports two types of gateways: distributed gateway and centralized gateway which can be changed in the Subnet spec. Both types of gateways support the natOutgoing setting, which allows the user to choose whether snat is required when the Pod accesses the external network. Distributed Gateway \u00b6 The default type of gateway for the Subnet, each node will act as a gateway for the pod on the current node to access the external network. The packets from container will flow into the host network stack from the local ovn0 NIC, and then forwarding the network according to the host's routing rules. When natOutgoing is true , the Pod will use the IP of the current host when accessing the external network. Example of a Subnet, where the gatewayType field is distributed : apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : distributed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : true Centralized Gateway \u00b6 If you want traffic within the Subnet to access the external network using a fixed IP for security operations such as auditing and whitelisting, you can set the gateway type in the Subnet to centralized. In centralized gateway mode, packets from Pods accessing the external network are first routed to the ovn0 NIC of a specific nodes, and then outbound through the host's routing rules. When natOutgoing is true , the Pod will use the IP of a specific nodes when accessing the external network. The centralized gateway example is as follows, where the gatewayType field is centralized and gatewayNode is the NodeName of the particular machine in Kubernetes. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : centralized spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1,node2\" natOutgoing : true If a centralized gateway wants to specify a specific NIC of a machine for outbound networking, gatewayNode format can be changed to kube-ovn-worker:172.18.0.2, kube-ovn-control-plane:172.18.0.3 . The centralized gateway defaults to primary-backup mode, with only the primary node performing traffic forwarding. If you need to switch to ECMP mode, please refer to ECMP Settings . The spec field enableEcmp has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the ECMP switch to the subnet level. You can set whether to enable ECMP mode based on different subnets. The enable-ecmp parameter in the kube-ovn-controller deployment is no longer used. After the previous version is upgraded to v1.12.0, the subnet switch will automatically inherit the value of the original global switch parameter. Subnet ACL \u00b6 For scenarios with fine-grained ACL control, Subnet of Kube-OVN provides ACL to enable fine-grained rules. The ACL rules in Subnet are the same as the ACL rules in OVN, and you can refer to ovn-nb ACL Table for more details. The supported filed in match can refer to ovn-sb Logical Flow Table . Example of an ACL rule that allows Pods with IP address 10.10.0.2 to access all addresses, but does not allow other addresses to access itself, is as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : acl spec : acls : - action : drop direction : to-lport match : ip4.dst == 10.10.0.2 && ip priority : 1002 - action : allow-related direction : from-lport match : ip4.src == 10.10.0.2 && ip priority : 1002 cidrBlock : 10.10.0.0/24 Subnet Isolation \u00b6 The function of Subnet ACL can cover the function of Subnet isolation with better flexibility, we recommend using Subnet ACL to do the corresponding configuration. By default the Subnets created by Kube-OVN can communicate with each other, and Pods can also access external networks through the gateway. To control access between Subnets, set private to true in the subnet spec, and the Subnet will be isolated from other Subnets and external networks and can only communicate within the Subnet. If you want to open a whitelist, you can set it by allowSubnets . The CIDRs in allowSubnets can access the Subnet bidirectionally. Enable Subnet Isolation Examples \u00b6 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : private spec : protocol : IPv4 default : false namespaces : - ns1 - ns2 cidrBlock : 10.69.0.0/16 private : true allowSubnets : - 10.16.0.0/16 - 10.18.0.0/16 Underlay Settings \u00b6 This part of the feature is only available for Underlay type Subnets. vlan : If an Underlay network is used, this field is used to control which Vlan CR the Subnet is bound to. This option defaults to the empty string, meaning that the Underlay network is not used. logicalGateway : Some Underlay environments are pure Layer 2 networks, with no physical Layer 3 gateway. In this case a virtual gateway can be set up with the OVN to connect the Underlay and Overlay networks. The default value is: false . Gateway Check Settings \u00b6 By default kube-ovn-cni will request the gateway using ICMP or ARP protocol after starting the Pod and wait for the return to verify that the network is working properly. Some Underlay environment gateways cannot respond to ARP requests, or scenarios that do not require external connectivity, the checking can be disabled . apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : disable-gw-check spec : disableGatewayCheck : true Other Advanced Settings \u00b6 Configure IPPool Manage QoS Manage Multiple Interface DHCP External Gateway Cluster Inter-Connection with OVN-IC VIP Reservation \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Config Subnet"},{"location":"en/guide/subnet/#config-subnet","text":"Subnet is a core concept and basic unit of use in Kube-OVN, and Kube-OVN organizes IP and network configuration in terms of Subnet. Each Namespace can belong to a specific Subnet, and Pods under the Namespace automatically obtain IPs from the Subnet they belong to and share the network configuration (CIDR, gateway type, access control, NAT control, etc.). Unlike other CNI implementations where each node is bound to a subnet, in Kube-OVN the Subnet is a global level virtual network configuration, and the addresses of one Subnet can be distributed on any node. There are some differences in the usage and configuration of Overlay and Underlay Subnets, and this document will describe the common configurations and differentiated features of the different types of Subnets.","title":"Config Subnet"},{"location":"en/guide/subnet/#default-subnet","text":"To make it easier for users to get started quickly, Kube-OVN has a built-in default Subnet, all Namespaces that do not explicitly declare subnet affiliation are automatically assigned IPs from the default subnet and the network information. The configuration of this Subnet is specified at installation time, you can refer to Built-in Network Settings for more details. To change the CIDR of the default Subnet after installation please refer to Change Subnet CIDR . In Overlay mode, the default Subnet uses a distributed gateway and NAT translation for outbound traffic, which behaves much the same as the Flannel's default behavior, allowing users to use most of the network features without additional configuration. In Underlay mode, the default Subnet uses the physical gateway as the outgoing gateway and enables arping to check network connectivity.","title":"Default Subnet"},{"location":"en/guide/subnet/#check-the-default-subnet","text":"The default field in the default Subnet spec is set to true , and there is only one default Subnet in a cluster, named ovn-default . # kubectl get subnet ovn-default -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: ovn-default resourceVersion: \"1571334\" selfLink: /apis/kubeovn.io/v1/subnets/ovn-default uid: 7e2451f8-fb44-4f7f-b3e0-cfd27f6fd5d6 spec: cidrBlock: 10 .16.0.0/16 default: true excludeIps: - 10 .16.0.1 gateway: 10 .16.0.1 gatewayType: distributed natOutgoing: true private: false protocol: IPv4","title":"Check the Default Subnet"},{"location":"en/guide/subnet/#join-subnet","text":"In the Kubernetes network specification, it is required that Nodes can communicate directly with all Pods. To achieve this in Overlay network mode, Kube-OVN creates a join Subnet and creates a virtual NIC ovn0 at each node that connect to the join subnet, through which the nodes and Pods can communicate with each other. The configuration of this Subnet is specified at installation time, you can refer to Built-in Network Settings for more details. To change the CIDR of the Join Subnet after installation please refer to Change Join CIDR .","title":"Join Subnet"},{"location":"en/guide/subnet/#check-the-join-subnet","text":"The default name of this subnet is join . There is generally no need to make changes to the network configuration except the CIDR. # kubectl get subnet join -o yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: creationTimestamp: \"2019-08-06T09:33:43Z\" generation: 1 name: join resourceVersion: \"1571333\" selfLink: /apis/kubeovn.io/v1/subnets/join uid: 9c744810-c678-4d50-8a7d-b8ec12ef91b8 spec: cidrBlock: 100 .64.0.0/16 default: false excludeIps: - 100 .64.0.1 gateway: 100 .64.0.1 gatewayNode: \"\" gatewayType: \"\" natOutgoing: false private: false protocol: IPv4 Check the ovn0 NIC at the node: # ifconfig ovn0 ovn0: flags = 4163 <UP,BROADCAST,RUNNING,MULTICAST> mtu 1420 inet 100 .64.0.4 netmask 255 .255.0.0 broadcast 100 .64.255.255 inet6 fe80::800:ff:fe40:5 prefixlen 64 scopeid 0x20<link> ether 0a:00:00:40:00:05 txqueuelen 1000 ( Ethernet ) RX packets 18 bytes 1428 ( 1 .3 KiB ) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 19 bytes 1810 ( 1 .7 KiB ) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0","title":"Check the Join Subnet"},{"location":"en/guide/subnet/#create-custom-subnets","text":"Here we describe the basic operation of how to create a Subnet and associate it with a Namespace, for more advanced configuration, please refer to the subsequent content.","title":"Create Custom Subnets"},{"location":"en/guide/subnet/#create-subnet","text":"cat <<EOF | kubectl create -f - apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: subnet1 spec: protocol: IPv4 cidrBlock: 10.66.0.0/16 excludeIps: - 10.66.0.1..10.66.0.10 - 10.66.0.101..10.66.0.151 gateway: 10.66.0.1 gatewayType: distributed natOutgoing: true routeTable: \"\" namespaces: - ns1 - ns2 EOF cidrBlock : Subnet CIDR range, different Subnet CIDRs under the same VPC cannot overlap. excludeIps : The address list is reserved so that the container network will not automatically assign addresses in the list, which can be used as a fixed IP address assignment segment or to avoid conflicts with existing devices in the physical network in Underlay mode. gateway \uff1aFor this subnet gateway address, Kube-OVN will automatically assign the corresponding logical gateway in Overlay mode, and the address should be the underlying physical gateway address in Underlay mode. namespaces : Bind the list of Namespace for this Subnet. Pods under the Namespace will be assigned addresses from the current Subnet after binding. routeTable : Associate the route table, default is main table, route table definition please defer to Static Routes","title":"Create Subnet"},{"location":"en/guide/subnet/#create-pod-in-the-subnet","text":"# kubectl create ns ns1 namespace/ns1 created # kubectl run nginx --image=docker.io/library/nginx:alpine -n ns1 deployment.apps/nginx created # kubectl get pod -n ns1 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-74d5899f46-n8wtg 1 /1 Running 0 10s 10 .66.0.11 node1 <none> <none>","title":"Create Pod in the Subnet"},{"location":"en/guide/subnet/#overlay-subnet-gateway-settings","text":"This feature only works for Overlay mode Subnets, Underlay type Subnets need to use the underlying physical gateway to access the external network. Pods under the Overlay Subnet need to access the external network through a gateway, and Kube-OVN currently supports two types of gateways: distributed gateway and centralized gateway which can be changed in the Subnet spec. Both types of gateways support the natOutgoing setting, which allows the user to choose whether snat is required when the Pod accesses the external network.","title":"Overlay Subnet Gateway Settings"},{"location":"en/guide/subnet/#distributed-gateway","text":"The default type of gateway for the Subnet, each node will act as a gateway for the pod on the current node to access the external network. The packets from container will flow into the host network stack from the local ovn0 NIC, and then forwarding the network according to the host's routing rules. When natOutgoing is true , the Pod will use the IP of the current host when accessing the external network. Example of a Subnet, where the gatewayType field is distributed : apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : distributed spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : distributed natOutgoing : true","title":"Distributed Gateway"},{"location":"en/guide/subnet/#centralized-gateway","text":"If you want traffic within the Subnet to access the external network using a fixed IP for security operations such as auditing and whitelisting, you can set the gateway type in the Subnet to centralized. In centralized gateway mode, packets from Pods accessing the external network are first routed to the ovn0 NIC of a specific nodes, and then outbound through the host's routing rules. When natOutgoing is true , the Pod will use the IP of a specific nodes when accessing the external network. The centralized gateway example is as follows, where the gatewayType field is centralized and gatewayNode is the NodeName of the particular machine in Kubernetes. apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : centralized spec : protocol : IPv4 cidrBlock : 10.166.0.0/16 default : false excludeIps : - 10.166.0.1 gateway : 10.166.0.1 gatewayType : centralized gatewayNode : \"node1,node2\" natOutgoing : true If a centralized gateway wants to specify a specific NIC of a machine for outbound networking, gatewayNode format can be changed to kube-ovn-worker:172.18.0.2, kube-ovn-control-plane:172.18.0.3 . The centralized gateway defaults to primary-backup mode, with only the primary node performing traffic forwarding. If you need to switch to ECMP mode, please refer to ECMP Settings . The spec field enableEcmp has been added to the subnet crd definition since Kube-OVN v1.12.0 to migrate the ECMP switch to the subnet level. You can set whether to enable ECMP mode based on different subnets. The enable-ecmp parameter in the kube-ovn-controller deployment is no longer used. After the previous version is upgraded to v1.12.0, the subnet switch will automatically inherit the value of the original global switch parameter.","title":"Centralized Gateway"},{"location":"en/guide/subnet/#subnet-acl","text":"For scenarios with fine-grained ACL control, Subnet of Kube-OVN provides ACL to enable fine-grained rules. The ACL rules in Subnet are the same as the ACL rules in OVN, and you can refer to ovn-nb ACL Table for more details. The supported filed in match can refer to ovn-sb Logical Flow Table . Example of an ACL rule that allows Pods with IP address 10.10.0.2 to access all addresses, but does not allow other addresses to access itself, is as follows: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : acl spec : acls : - action : drop direction : to-lport match : ip4.dst == 10.10.0.2 && ip priority : 1002 - action : allow-related direction : from-lport match : ip4.src == 10.10.0.2 && ip priority : 1002 cidrBlock : 10.10.0.0/24","title":"Subnet ACL"},{"location":"en/guide/subnet/#subnet-isolation","text":"The function of Subnet ACL can cover the function of Subnet isolation with better flexibility, we recommend using Subnet ACL to do the corresponding configuration. By default the Subnets created by Kube-OVN can communicate with each other, and Pods can also access external networks through the gateway. To control access between Subnets, set private to true in the subnet spec, and the Subnet will be isolated from other Subnets and external networks and can only communicate within the Subnet. If you want to open a whitelist, you can set it by allowSubnets . The CIDRs in allowSubnets can access the Subnet bidirectionally.","title":"Subnet Isolation"},{"location":"en/guide/subnet/#enable-subnet-isolation-examples","text":"apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : private spec : protocol : IPv4 default : false namespaces : - ns1 - ns2 cidrBlock : 10.69.0.0/16 private : true allowSubnets : - 10.16.0.0/16 - 10.18.0.0/16","title":"Enable Subnet Isolation Examples"},{"location":"en/guide/subnet/#underlay-settings","text":"This part of the feature is only available for Underlay type Subnets. vlan : If an Underlay network is used, this field is used to control which Vlan CR the Subnet is bound to. This option defaults to the empty string, meaning that the Underlay network is not used. logicalGateway : Some Underlay environments are pure Layer 2 networks, with no physical Layer 3 gateway. In this case a virtual gateway can be set up with the OVN to connect the Underlay and Overlay networks. The default value is: false .","title":"Underlay Settings"},{"location":"en/guide/subnet/#gateway-check-settings","text":"By default kube-ovn-cni will request the gateway using ICMP or ARP protocol after starting the Pod and wait for the return to verify that the network is working properly. Some Underlay environment gateways cannot respond to ARP requests, or scenarios that do not require external connectivity, the checking can be disabled . apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : disable-gw-check spec : disableGatewayCheck : true","title":"Gateway Check Settings"},{"location":"en/guide/subnet/#other-advanced-settings","text":"Configure IPPool Manage QoS Manage Multiple Interface DHCP External Gateway Cluster Inter-Connection with OVN-IC VIP Reservation \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Other Advanced Settings"},{"location":"en/guide/vpc-qos/","text":"VPC QoS \u00b6 Kube-OVN supports using QoSPolicy CRD to limit the traffic rate of custom VPC. EIP QoS \u00b6 Limit the speed of EIP to 1Mbps and the priority to 1, and shared=false here means that this QoSPolicy can only be used for this EIP and support dynamically modifying QoSPolicy to change QoS rules. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-eip-example spec : shared : false bindingType : EIP bandwidthLimitRules : - name : eip-ingress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : ingress - name : eip-egress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : egress The IptablesEIP configuration is as follows: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-1 spec : natGwDp : gw1 qosPolicy : qos-eip-example The value of .spec.qosPolicy supports being specified during creation and also supports modification after creation. View EIPs with QoS enabled \u00b6 View the corresponding EIPs that have been set up using label : # kubectl get eip -l ovn.kubernetes.io/qos=qos-eip-example NAME IP MAC NAT NATGWDP READY eip-1 172 .18.11.24 00 :00:00:34:41:0B fip gw1 true QoS for VPC NATGW net1 NIC \u00b6 Limit the speed of the net1 NIC on VPC NATGW to 10Mbps and set the priority to 3. Here shared=true , which means that this QoSPolicy can be used by multiple resources at the same time, and does not allow the modification of the contents of the QoSPolicy in this scenario. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-ingress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : ingress - name : net1-egress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : egress The VpcNatGateway configuration is as follows: kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" The value of .spec.qosPolicy supports both creation and subsequent modification. QoS for specific traffic on net1 NIC \u00b6 Limit the specific traffic on net1 NIC to 5Mbps and set the priority to 2. Here shared=true , which means that this QoSPolicy can be used by multiple resources at the same time, and does not allow the modification of the contents of the QoSPolicy in this scenario. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-extip-ingress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : ingress matchType : ip matchValue : src 172.18.11.22/32 - name : net1-extip-egress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : egress matchType : ip matchValue : dst 172.18.11.23/32 The VpcNatGateway configuration is as follows: kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" View NATGWs with QoS enabled \u00b6 View the corresponding NATGWs that have been set up using label : # kubectl get vpc-nat-gw -l ovn.kubernetes.io/qos=qos-natgw-example NAME VPC SUBNET LANIP gw1 test-vpc-1 net1 10 .0.1.254 View QoS rules \u00b6 # kubectl get qos -A NAME SHARED BINDINGTYPE qos-eip-example false EIP qos-natgw-example true NATGW Limitations \u00b6 QoSPolicy can only be deleted when it is not in use. Therefore, before deleting the QoSPolicy, please check the EIP and NATGW that have enabled QoS, and remove their spec.qosPolicy configuration. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"VPC QoS"},{"location":"en/guide/vpc-qos/#vpc-qos","text":"Kube-OVN supports using QoSPolicy CRD to limit the traffic rate of custom VPC.","title":"VPC QoS"},{"location":"en/guide/vpc-qos/#eip-qos","text":"Limit the speed of EIP to 1Mbps and the priority to 1, and shared=false here means that this QoSPolicy can only be used for this EIP and support dynamically modifying QoSPolicy to change QoS rules. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-eip-example spec : shared : false bindingType : EIP bandwidthLimitRules : - name : eip-ingress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : ingress - name : eip-egress rateMax : \"1\" # Mbps burstMax : \"1\" # Mbps priority : 1 direction : egress The IptablesEIP configuration is as follows: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-1 spec : natGwDp : gw1 qosPolicy : qos-eip-example The value of .spec.qosPolicy supports being specified during creation and also supports modification after creation.","title":"EIP QoS"},{"location":"en/guide/vpc-qos/#view-eips-with-qos-enabled","text":"View the corresponding EIPs that have been set up using label : # kubectl get eip -l ovn.kubernetes.io/qos=qos-eip-example NAME IP MAC NAT NATGWDP READY eip-1 172 .18.11.24 00 :00:00:34:41:0B fip gw1 true","title":"View EIPs with QoS enabled"},{"location":"en/guide/vpc-qos/#qos-for-vpc-natgw-net1-nic","text":"Limit the speed of the net1 NIC on VPC NATGW to 10Mbps and set the priority to 3. Here shared=true , which means that this QoSPolicy can be used by multiple resources at the same time, and does not allow the modification of the contents of the QoSPolicy in this scenario. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-ingress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : ingress - name : net1-egress interface : net1 rateMax : \"10\" # Mbps burstMax : \"10\" # Mbps priority : 3 direction : egress The VpcNatGateway configuration is as follows: kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" The value of .spec.qosPolicy supports both creation and subsequent modification.","title":"QoS for VPC NATGW net1 NIC"},{"location":"en/guide/vpc-qos/#qos-for-specific-traffic-on-net1-nic","text":"Limit the specific traffic on net1 NIC to 5Mbps and set the priority to 2. Here shared=true , which means that this QoSPolicy can be used by multiple resources at the same time, and does not allow the modification of the contents of the QoSPolicy in this scenario. The QoSPolicy configuration is as follows: apiVersion : kubeovn.io/v1 kind : QoSPolicy metadata : name : qos-natgw-example spec : shared : true bindingType : NATGW bandwidthLimitRules : - name : net1-extip-ingress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : ingress matchType : ip matchValue : src 172.18.11.22/32 - name : net1-extip-egress interface : net1 rateMax : \"5\" # Mbps burstMax : \"5\" # Mbps priority : 2 direction : egress matchType : ip matchValue : dst 172.18.11.23/32 The VpcNatGateway configuration is as follows: kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 qosPolicy : qos-natgw-example selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\"","title":"QoS for specific traffic on net1 NIC"},{"location":"en/guide/vpc-qos/#view-natgws-with-qos-enabled","text":"View the corresponding NATGWs that have been set up using label : # kubectl get vpc-nat-gw -l ovn.kubernetes.io/qos=qos-natgw-example NAME VPC SUBNET LANIP gw1 test-vpc-1 net1 10 .0.1.254","title":"View NATGWs with QoS enabled"},{"location":"en/guide/vpc-qos/#view-qos-rules","text":"# kubectl get qos -A NAME SHARED BINDINGTYPE qos-eip-example false EIP qos-natgw-example true NATGW","title":"View QoS rules"},{"location":"en/guide/vpc-qos/#limitations","text":"QoSPolicy can only be deleted when it is not in use. Therefore, before deleting the QoSPolicy, please check the EIP and NATGW that have enabled QoS, and remove their spec.qosPolicy configuration. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Limitations"},{"location":"en/guide/vpc/","text":"Config VPC \u00b6 Kube-OVN supports multi-tenant isolation level VPC networks. Different VPC networks are independent of each other and can be configured separately with Subnet CIDRs, routing policies, security policies, outbound gateways, EIP, etc. VPC is mainly used in scenarios where there requires strong isolation of multi-tenant networks and some Kubernetes networking features conflict under multi-tenant networks. For example, node and pod access, NodePort functionality, network access-based health checks, and DNS capabilities are not supported in multi-tenant network scenarios at this time. In order to facilitate common Kubernetes usage scenarios, Kube-OVN has a special design for the default VPC where the Subnet under the VPC can meet the Kubernetes specification. The custom VPC supports static routing, EIP and NAT gateways as described in this document. Common isolation requirements can be achieved through network policies and Subnet ACLs under the default VPC, so before using a custom VPC, please make sure whether you need VPC-level isolation and understand the limitations under the custom VPC. For Underlay subnets, physical switches are responsible for data-plane forwarding, so VPCs cannot isolate Underlay subnets. Creating Custom VPCs \u00b6 Create two VPCs: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : namespaces : - ns1 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-2 spec : namespaces : - ns2 namespaces : Limit which namespaces can use this VPC. If empty, all namespaces can use this VPC. Create two Subnets, belonging to two different VPCs and having the same CIDR: kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : test-vpc-1 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns1 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : test-vpc-2 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns2 Create Pods under two separate Namespaces: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 namespace : ns1 name : vpc1-pod spec : containers : - name : vpc1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 namespace : ns2 name : vpc2-pod spec : containers : - name : vpc2-pod image : docker.io/library/nginx:alpine After running successfully, you can observe that the two Pod addresses belong to the same CIDR, but the two Pods cannot access each other because they are running on different tenant VPCs. Custom VPC Pod supports livenessProbe and readinessProbe \u00b6 Since the Pods under the custom VPC do not communicate with the network of the node, the probe packets sent by the kubelet cannot reach the Pods in the custom VPC. Kube-OVN uses TProxy to redirect the detection packets sent by kubelet to Pods in the custom VPC to achieve this function. The configuration method is as follows, add the parameter --enable-tproxy=true in Daemonset kube-ovn-cni : spec : template : spec : containers : - args : - --enable-tproxy=true Restrictions for this feature: When Pods under different VPCs have the same IP under the same node, the detection function fails. Currently, only tcpSocket and httpGet are supported. Create VPC NAT Gateway \u00b6 Subnets under custom VPCs do not support distributed gateways and centralized gateways under default VPCs. Pod access to the external network within the VPC requires a VPC gateway, which bridges the physical and tenant networks and provides floating IP, SNAT and DNAT capabilities. The VPC gateway function relies on Multus-CNI function, please refer to multus-cni . Configuring the External Network \u00b6 apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-vpc-external-network spec : protocol : IPv4 provider : ovn-vpc-external-network.kube-system cidrBlock : 192.168.0.0/24 gateway : 192.168.0.1 # IP address of the physical gateway excludeIps : - 192.168.0.1..192.168.0.10 --- apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-vpc-external-network namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth1\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-vpc-external-network.kube-system\" } }' This Subnet is used to manage the available external addresses and the address will be allocated to VPC NAT Gateway through Macvlan, so please communicate with your network management to give you the available physical segment IPs. The VPC gateway uses Macvlan for physical network configuration, and master of NetworkAttachmentDefinition should be the NIC name of the corresponding physical network NIC. name : External network name. For macvlan mode, the nic will send packets directly through that node NIC, relying on the underlying network devices for L2/L3 level forwarding capabilities. You need to configure the corresponding gateway, Vlan and security policy in the underlying network device in advance. For OpenStack VM environments, you need to turn off PortSecurity on the corresponding network port. For VMware vSwitch networks, MAC Address Changes , Forged Transmits and Promiscuous Mode Operation should be set to allow . For Hyper-V virtualization, MAC Address Spoofing should be enabled in VM nic advanced features. Public clouds, such as AWS, GCE, AliCloud, etc., do not support user-defined Mac, so they cannot support Macvlan mode network. Due to the limitations of Macvlan, the Macvlan sub-interface cannot access the parent interface address. If the physical network card corresponds to a switch interface in Trunk mode, a sub-interface needs to be created on the network card and provided to Macvlan for use. Enabling the VPC Gateway \u00b6 VPC gateway functionality needs to be enabled via ovn-vpc-nat-gw-config under kube-system : --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-config namespace : kube-system data : image : docker.io/kubeovn/vpc-nat-gateway:v1.12.0 --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-gw-config namespace : kube-system data : enable-vpc-nat-gw : 'true' image : The image used by the Gateway Pod. enable-vpc-nat-gw : Controls whether the VPC Gateway feature is enabled. Create VPC Gateway and Set the Default Route \u00b6 kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" externalSubnets : - ovn-vpc-external-network vpc : The VPC to which this VpcNatGateway belongs. subnet : A Subnet within the VPC, the VPC Gateway Pod will use lanIp to connect to the tenant network under that subnet. lanIp : An unused IP within the subnet that the VPC Gateway Pod will eventually use for the Pod. When configuring routing for a VPC, the nextHopIP needs to be set to the lanIp of the current VpcNatGateway. selector : The node selector for VpcNatGateway Pod has the same format as NodeSelector in Kubernetes. externalSubnets : External network used by the VPC gateway, if not configured, ovn-vpc-external-network is used by default, and only one external network is supported in the current version. Other configurable parameters: tolerations : Configure tolerance for the VPC gateway. For details, see Taints and Tolerations affinity : Configure affinity for the Pod or node of the VPC gateway. For details, see Assigning Pods to Nodes Create EIP \u00b6 EIP allows for floating IP, SNAT, and DNAT operations after assigning an IP from an external network segment to a VPC gateway. Randomly assign an address to the EIP: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 Fixed EIP address assignment: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : natGwDp : gw1 v4ip : 10.0.1.111 Specify the external network on which the EIP is located: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 externalSubnet : ovn-vpc-external-network externalSubnet : The name of the external network on which the EIP is located. If not specified, it defaults to ovn-vpc-external-network . If specified, it must be one of the externalSubnets of the VPC gateway. Create DNAT Rules \u00b6 kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipd01 spec : natGwDp : gw1 --- kind : IptablesDnatRule apiVersion : kubeovn.io/v1 metadata : name : dnat01 spec : eip : eipd01 externalPort : '8888' internalIp : 10.0.1.10 internalPort : '80' protocol : tcp Create SNAT Rules \u00b6 --- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eips01 spec : natGwDp : gw1 --- kind : IptablesSnatRule apiVersion : kubeovn.io/v1 metadata : name : snat01 spec : eip : eips01 internalCIDR : 10.0.1.0/24 Create Floating IP \u00b6 --- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipf01 spec : natGwDp : gw1 --- kind : IptablesFIPRule apiVersion : kubeovn.io/v1 metadata : name : fip01 spec : eip : eipf01 internalIp : 10.0.1.5 Custom Routing \u00b6 Within the custom VPC, users can customize the routing rules within the VPC and combine it with the gateway for more flexible forwarding. Kube-OVN supports static routes and more flexible policy routes. Static Routes \u00b6 kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : staticRoutes : - cidr : 0.0.0.0/0 nextHopIP : 10.0.1.254 policy : policyDst - cidr : 172.31.0.0/24 nextHopIP : 10.0.1.253 policy : policySrc routeTable : \"rtb1\" policy : Supports destination routing policyDst and source routing policySrc . When there are overlapping routing rules, the rule with the longer CIDR mask has higher priority, and if the mask length is the same, the destination route has a higher priority over the source route. routeTable : You can store the route in specific table, default is main table. Associate with subnet please defer to Create Custom Subnets Policy Routes \u00b6 Traffic matched by static routes can be controlled at a finer granularity by policy routing. Policy routing provides more precise matching rules, priority control and more forwarding actions. This feature brings the OVN internal logical router policy function directly to the outside world, for more information on its use, please refer to Logical Router Policy . An example of policy routes: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : policyRoutes : - action : drop match : ip4.src==10.0.1.0/24 && ip4.dst==10.0.1.250 priority : 11 - action : reroute match : ip4.src==10.0.1.0/24 nextHopIP : 10.0.1.252 priority : 10 Custom vpc-dns \u00b6 Due to the isolation between custom VPCs and default VPC networks, Pods in VPCs cannot use the default coredns service for domain name resolution. If you want to use coredns to resolve Service domain names within the custom VPC, you can use the vpc-dns resource provided by Kube-OVN. Create an Additional Network \u00b6 apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }' Modify the Provider of the ovn-default Logical Switch \u00b6 Modify the provider of ovn-default to the provider ovn-nad.default.ovn configured above in nad\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn vpc : ovn-cluster Modify the vpc-dns ConfigMap \u00b6 Create a ConfigMap in the kube-system namespace, configure the vpc-dns parameters to be used for the subsequent vpc-dns feature activation: apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : true nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns : (optional) true to enable the feature, false to disable the feature. Default true . coredns-image : (optional): DNS deployment image. Default is the cluster coredns deployment version. coredns-template : (optional): URL of the DNS deployment template. Default: yamls/coredns-template.yaml in the current version repository. coredns-vip : VIP providing LB service for coredns. nad-name : Name of the configured network-attachment-definitions resource. nad-provider : Name of the used provider. k8s-service-host : (optional) IP used by coredns to access the k8s apiserver service. k8s-service-port : (optional) Port used by coredns to access the k8s apiserver service. Deploying VPC-DNS Dependent Resources \u00b6 apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance } Deploy vpc-dns \u00b6 kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc : The VPC name used to deploy the DNS component. subnet : The subnet name used to deploy the DNS component. View resource information: [ root@hci-dev-mst-1 kubeovn ] # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : if the custom vpc-dns is ready. Restrictions \u00b6 Only one custom DNS component will be deployed in one VPC; When multiple VPC-DNS resources (i.e. different subnets in the same VPC) are configured in one VPC, only one VPC-DNS resource with status true will be active, while the others will be false ; When the true VPC-DNS is deleted, another false VPC-DNS will be deployed. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Config VPC"},{"location":"en/guide/vpc/#config-vpc","text":"Kube-OVN supports multi-tenant isolation level VPC networks. Different VPC networks are independent of each other and can be configured separately with Subnet CIDRs, routing policies, security policies, outbound gateways, EIP, etc. VPC is mainly used in scenarios where there requires strong isolation of multi-tenant networks and some Kubernetes networking features conflict under multi-tenant networks. For example, node and pod access, NodePort functionality, network access-based health checks, and DNS capabilities are not supported in multi-tenant network scenarios at this time. In order to facilitate common Kubernetes usage scenarios, Kube-OVN has a special design for the default VPC where the Subnet under the VPC can meet the Kubernetes specification. The custom VPC supports static routing, EIP and NAT gateways as described in this document. Common isolation requirements can be achieved through network policies and Subnet ACLs under the default VPC, so before using a custom VPC, please make sure whether you need VPC-level isolation and understand the limitations under the custom VPC. For Underlay subnets, physical switches are responsible for data-plane forwarding, so VPCs cannot isolate Underlay subnets.","title":"Config VPC"},{"location":"en/guide/vpc/#creating-custom-vpcs","text":"Create two VPCs: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : namespaces : - ns1 --- kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-2 spec : namespaces : - ns2 namespaces : Limit which namespaces can use this VPC. If empty, all namespaces can use this VPC. Create two Subnets, belonging to two different VPCs and having the same CIDR: kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net1 spec : vpc : test-vpc-1 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns1 --- kind : Subnet apiVersion : kubeovn.io/v1 metadata : name : net2 spec : vpc : test-vpc-2 cidrBlock : 10.0.1.0/24 protocol : IPv4 namespaces : - ns2 Create Pods under two separate Namespaces: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net1 namespace : ns1 name : vpc1-pod spec : containers : - name : vpc1-pod image : docker.io/library/nginx:alpine --- apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/logical_switch : net2 namespace : ns2 name : vpc2-pod spec : containers : - name : vpc2-pod image : docker.io/library/nginx:alpine After running successfully, you can observe that the two Pod addresses belong to the same CIDR, but the two Pods cannot access each other because they are running on different tenant VPCs.","title":"Creating Custom VPCs"},{"location":"en/guide/vpc/#custom-vpc-pod-supports-livenessprobe-and-readinessprobe","text":"Since the Pods under the custom VPC do not communicate with the network of the node, the probe packets sent by the kubelet cannot reach the Pods in the custom VPC. Kube-OVN uses TProxy to redirect the detection packets sent by kubelet to Pods in the custom VPC to achieve this function. The configuration method is as follows, add the parameter --enable-tproxy=true in Daemonset kube-ovn-cni : spec : template : spec : containers : - args : - --enable-tproxy=true Restrictions for this feature: When Pods under different VPCs have the same IP under the same node, the detection function fails. Currently, only tcpSocket and httpGet are supported.","title":"Custom VPC Pod supports livenessProbe and readinessProbe"},{"location":"en/guide/vpc/#create-vpc-nat-gateway","text":"Subnets under custom VPCs do not support distributed gateways and centralized gateways under default VPCs. Pod access to the external network within the VPC requires a VPC gateway, which bridges the physical and tenant networks and provides floating IP, SNAT and DNAT capabilities. The VPC gateway function relies on Multus-CNI function, please refer to multus-cni .","title":"Create VPC NAT Gateway"},{"location":"en/guide/vpc/#configuring-the-external-network","text":"apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-vpc-external-network spec : protocol : IPv4 provider : ovn-vpc-external-network.kube-system cidrBlock : 192.168.0.0/24 gateway : 192.168.0.1 # IP address of the physical gateway excludeIps : - 192.168.0.1..192.168.0.10 --- apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-vpc-external-network namespace : kube-system spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"macvlan\", \"master\": \"eth1\", \"mode\": \"bridge\", \"ipam\": { \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-vpc-external-network.kube-system\" } }' This Subnet is used to manage the available external addresses and the address will be allocated to VPC NAT Gateway through Macvlan, so please communicate with your network management to give you the available physical segment IPs. The VPC gateway uses Macvlan for physical network configuration, and master of NetworkAttachmentDefinition should be the NIC name of the corresponding physical network NIC. name : External network name. For macvlan mode, the nic will send packets directly through that node NIC, relying on the underlying network devices for L2/L3 level forwarding capabilities. You need to configure the corresponding gateway, Vlan and security policy in the underlying network device in advance. For OpenStack VM environments, you need to turn off PortSecurity on the corresponding network port. For VMware vSwitch networks, MAC Address Changes , Forged Transmits and Promiscuous Mode Operation should be set to allow . For Hyper-V virtualization, MAC Address Spoofing should be enabled in VM nic advanced features. Public clouds, such as AWS, GCE, AliCloud, etc., do not support user-defined Mac, so they cannot support Macvlan mode network. Due to the limitations of Macvlan, the Macvlan sub-interface cannot access the parent interface address. If the physical network card corresponds to a switch interface in Trunk mode, a sub-interface needs to be created on the network card and provided to Macvlan for use.","title":"Configuring the External Network"},{"location":"en/guide/vpc/#enabling-the-vpc-gateway","text":"VPC gateway functionality needs to be enabled via ovn-vpc-nat-gw-config under kube-system : --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-config namespace : kube-system data : image : docker.io/kubeovn/vpc-nat-gateway:v1.12.0 --- kind : ConfigMap apiVersion : v1 metadata : name : ovn-vpc-nat-gw-config namespace : kube-system data : enable-vpc-nat-gw : 'true' image : The image used by the Gateway Pod. enable-vpc-nat-gw : Controls whether the VPC Gateway feature is enabled.","title":"Enabling the VPC Gateway"},{"location":"en/guide/vpc/#create-vpc-gateway-and-set-the-default-route","text":"kind : VpcNatGateway apiVersion : kubeovn.io/v1 metadata : name : gw1 spec : vpc : test-vpc-1 subnet : net1 lanIp : 10.0.1.254 selector : - \"kubernetes.io/hostname: kube-ovn-worker\" - \"kubernetes.io/os: linux\" externalSubnets : - ovn-vpc-external-network vpc : The VPC to which this VpcNatGateway belongs. subnet : A Subnet within the VPC, the VPC Gateway Pod will use lanIp to connect to the tenant network under that subnet. lanIp : An unused IP within the subnet that the VPC Gateway Pod will eventually use for the Pod. When configuring routing for a VPC, the nextHopIP needs to be set to the lanIp of the current VpcNatGateway. selector : The node selector for VpcNatGateway Pod has the same format as NodeSelector in Kubernetes. externalSubnets : External network used by the VPC gateway, if not configured, ovn-vpc-external-network is used by default, and only one external network is supported in the current version. Other configurable parameters: tolerations : Configure tolerance for the VPC gateway. For details, see Taints and Tolerations affinity : Configure affinity for the Pod or node of the VPC gateway. For details, see Assigning Pods to Nodes","title":"Create VPC Gateway and Set the Default Route"},{"location":"en/guide/vpc/#create-eip","text":"EIP allows for floating IP, SNAT, and DNAT operations after assigning an IP from an external network segment to a VPC gateway. Randomly assign an address to the EIP: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 Fixed EIP address assignment: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-static spec : natGwDp : gw1 v4ip : 10.0.1.111 Specify the external network on which the EIP is located: kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eip-random spec : natGwDp : gw1 externalSubnet : ovn-vpc-external-network externalSubnet : The name of the external network on which the EIP is located. If not specified, it defaults to ovn-vpc-external-network . If specified, it must be one of the externalSubnets of the VPC gateway.","title":"Create EIP"},{"location":"en/guide/vpc/#create-dnat-rules","text":"kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipd01 spec : natGwDp : gw1 --- kind : IptablesDnatRule apiVersion : kubeovn.io/v1 metadata : name : dnat01 spec : eip : eipd01 externalPort : '8888' internalIp : 10.0.1.10 internalPort : '80' protocol : tcp","title":"Create DNAT Rules"},{"location":"en/guide/vpc/#create-snat-rules","text":"--- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eips01 spec : natGwDp : gw1 --- kind : IptablesSnatRule apiVersion : kubeovn.io/v1 metadata : name : snat01 spec : eip : eips01 internalCIDR : 10.0.1.0/24","title":"Create SNAT Rules"},{"location":"en/guide/vpc/#create-floating-ip","text":"--- kind : IptablesEIP apiVersion : kubeovn.io/v1 metadata : name : eipf01 spec : natGwDp : gw1 --- kind : IptablesFIPRule apiVersion : kubeovn.io/v1 metadata : name : fip01 spec : eip : eipf01 internalIp : 10.0.1.5","title":"Create Floating IP"},{"location":"en/guide/vpc/#custom-routing","text":"Within the custom VPC, users can customize the routing rules within the VPC and combine it with the gateway for more flexible forwarding. Kube-OVN supports static routes and more flexible policy routes.","title":"Custom Routing"},{"location":"en/guide/vpc/#static-routes","text":"kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : staticRoutes : - cidr : 0.0.0.0/0 nextHopIP : 10.0.1.254 policy : policyDst - cidr : 172.31.0.0/24 nextHopIP : 10.0.1.253 policy : policySrc routeTable : \"rtb1\" policy : Supports destination routing policyDst and source routing policySrc . When there are overlapping routing rules, the rule with the longer CIDR mask has higher priority, and if the mask length is the same, the destination route has a higher priority over the source route. routeTable : You can store the route in specific table, default is main table. Associate with subnet please defer to Create Custom Subnets","title":"Static Routes"},{"location":"en/guide/vpc/#policy-routes","text":"Traffic matched by static routes can be controlled at a finer granularity by policy routing. Policy routing provides more precise matching rules, priority control and more forwarding actions. This feature brings the OVN internal logical router policy function directly to the outside world, for more information on its use, please refer to Logical Router Policy . An example of policy routes: kind : Vpc apiVersion : kubeovn.io/v1 metadata : name : test-vpc-1 spec : policyRoutes : - action : drop match : ip4.src==10.0.1.0/24 && ip4.dst==10.0.1.250 priority : 11 - action : reroute match : ip4.src==10.0.1.0/24 nextHopIP : 10.0.1.252 priority : 10","title":"Policy Routes"},{"location":"en/guide/vpc/#custom-vpc-dns","text":"Due to the isolation between custom VPCs and default VPC networks, Pods in VPCs cannot use the default coredns service for domain name resolution. If you want to use coredns to resolve Service domain names within the custom VPC, you can use the vpc-dns resource provided by Kube-OVN.","title":"Custom vpc-dns"},{"location":"en/guide/vpc/#create-an-additional-network","text":"apiVersion : \"k8s.cni.cncf.io/v1\" kind : NetworkAttachmentDefinition metadata : name : ovn-nad namespace : default spec : config : '{ \"cniVersion\": \"0.3.0\", \"type\": \"kube-ovn\", \"server_socket\": \"/run/openvswitch/kube-ovn-daemon.sock\", \"provider\": \"ovn-nad.default.ovn\" }'","title":"Create an Additional Network"},{"location":"en/guide/vpc/#modify-the-provider-of-the-ovn-default-logical-switch","text":"Modify the provider of ovn-default to the provider ovn-nad.default.ovn configured above in nad\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : ovn-default spec : cidrBlock : 10.16.0.0/16 default : true disableGatewayCheck : false disableInterConnection : false enableDHCP : false enableIPv6RA : false excludeIps : - 10.16.0.1 gateway : 10.16.0.1 gatewayType : distributed logicalGateway : false natOutgoing : true private : false protocol : IPv4 provider : ovn-nad.default.ovn vpc : ovn-cluster","title":"Modify the Provider of the ovn-default Logical Switch"},{"location":"en/guide/vpc/#modify-the-vpc-dns-configmap","text":"Create a ConfigMap in the kube-system namespace, configure the vpc-dns parameters to be used for the subsequent vpc-dns feature activation: apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-config namespace : kube-system data : coredns-vip : 10.96.0.3 enable-vpc-dns : true nad-name : ovn-nad nad-provider : ovn-nad.default.ovn enable-vpc-dns : (optional) true to enable the feature, false to disable the feature. Default true . coredns-image : (optional): DNS deployment image. Default is the cluster coredns deployment version. coredns-template : (optional): URL of the DNS deployment template. Default: yamls/coredns-template.yaml in the current version repository. coredns-vip : VIP providing LB service for coredns. nad-name : Name of the configured network-attachment-definitions resource. nad-provider : Name of the used provider. k8s-service-host : (optional) IP used by coredns to access the k8s apiserver service. k8s-service-port : (optional) Port used by coredns to access the k8s apiserver service.","title":"Modify the vpc-dns ConfigMap"},{"location":"en/guide/vpc/#deploying-vpc-dns-dependent-resources","text":"apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRole metadata : labels : kubernetes.io/bootstrapping : rbac-defaults name : system:vpc-dns rules : - apiGroups : - \"\" resources : - endpoints - services - pods - namespaces verbs : - list - watch - apiGroups : - discovery.k8s.io resources : - endpointslices verbs : - list - watch --- apiVersion : rbac.authorization.k8s.io/v1 kind : ClusterRoleBinding metadata : annotations : rbac.authorization.kubernetes.io/autoupdate : \"true\" labels : kubernetes.io/bootstrapping : rbac-defaults name : vpc-dns roleRef : apiGroup : rbac.authorization.k8s.io kind : ClusterRole name : system:vpc-dns subjects : - kind : ServiceAccount name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ServiceAccount metadata : name : vpc-dns namespace : kube-system --- apiVersion : v1 kind : ConfigMap metadata : name : vpc-dns-corefile namespace : kube-system data : Corefile : | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp } cache 30 loop reload loadbalance }","title":"Deploying VPC-DNS Dependent Resources"},{"location":"en/guide/vpc/#deploy-vpc-dns","text":"kind : VpcDns apiVersion : kubeovn.io/v1 metadata : name : test-cjh1 spec : vpc : cjh-vpc-1 subnet : cjh-subnet-1 vpc : The VPC name used to deploy the DNS component. subnet : The subnet name used to deploy the DNS component. View resource information: [ root@hci-dev-mst-1 kubeovn ] # kubectl get vpc-dns NAME ACTIVE VPC SUBNET test-cjh1 false cjh-vpc-1 cjh-subnet-1 test-cjh2 true cjh-vpc-1 cjh-subnet-2 ACTIVE : if the custom vpc-dns is ready.","title":"Deploy vpc-dns"},{"location":"en/guide/vpc/#restrictions","text":"Only one custom DNS component will be deployed in one VPC; When multiple VPC-DNS resources (i.e. different subnets in the same VPC) are configured in one VPC, only one VPC-DNS resource with status true will be active, while the others will be false ; When the true VPC-DNS is deleted, another false VPC-DNS will be deployed. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Restrictions"},{"location":"en/guide/webhook/","text":"Webhook \u00b6 Using Webhook, you can verify CRD resources within Kube-OVN. Currently, Webhook mainly performs fixed IP address conflict detection and Subnet CIDR conflict detection, and prompts errors when such conflicts happen. Since Webhook intercepts all Subnet and Pod creation requests, you need to deploy Kube-OVN first and Webhook later. Install Cert-Manager \u00b6 Webhook deployment requires certificate, we use cert-manager to generate the associated certificate, we need to deploy cert-manager before deploying Webhook. You can use the following command to deploy cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml More cert-manager usage please refer to cert-manager document \u3002 Install Webhook \u00b6 Download Webhook yaml and install: # kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/webhook.yaml deployment.apps/kube-ovn-webhook created service/kube-ovn-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/kube-ovn-webhook created certificate.cert-manager.io/kube-ovn-webhook-serving-cert created issuer.cert-manager.io/kube-ovn-webhook-selfsigned-issuer created Verify Webhook Take Effect \u00b6 Check the running Pod and get the Pod IP 10.16.0.15 : # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES static-7584848b74-fw9dm 1 /1 Running 0 2d13h 10 .16.0.15 kube-ovn-worker <none> Write yaml to create a Pod with the same IP: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 labels : app : static managedFields : name : staticip-pod namespace : default spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest When using the above yaml to create a fixed address Pod, it prompts an IP address conflict: # kubectl apply -f pod-static.yaml Error from server ( annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 ) : error when creating \"pod-static.yaml\" : admission webhook \"pod-ip-validaing.kube-ovn.io\" denied the request: annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Webhook"},{"location":"en/guide/webhook/#webhook","text":"Using Webhook, you can verify CRD resources within Kube-OVN. Currently, Webhook mainly performs fixed IP address conflict detection and Subnet CIDR conflict detection, and prompts errors when such conflicts happen. Since Webhook intercepts all Subnet and Pod creation requests, you need to deploy Kube-OVN first and Webhook later.","title":"Webhook"},{"location":"en/guide/webhook/#install-cert-manager","text":"Webhook deployment requires certificate, we use cert-manager to generate the associated certificate, we need to deploy cert-manager before deploying Webhook. You can use the following command to deploy cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml More cert-manager usage please refer to cert-manager document \u3002","title":"Install Cert-Manager"},{"location":"en/guide/webhook/#install-webhook","text":"Download Webhook yaml and install: # kubectl apply -f https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/webhook.yaml deployment.apps/kube-ovn-webhook created service/kube-ovn-webhook created validatingwebhookconfiguration.admissionregistration.k8s.io/kube-ovn-webhook created certificate.cert-manager.io/kube-ovn-webhook-serving-cert created issuer.cert-manager.io/kube-ovn-webhook-selfsigned-issuer created","title":"Install Webhook"},{"location":"en/guide/webhook/#verify-webhook-take-effect","text":"Check the running Pod and get the Pod IP 10.16.0.15 : # kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES static-7584848b74-fw9dm 1 /1 Running 0 2d13h 10 .16.0.15 kube-ovn-worker <none> Write yaml to create a Pod with the same IP: apiVersion : v1 kind : Pod metadata : annotations : ovn.kubernetes.io/ip_address : 10.16.0.15 ovn.kubernetes.io/mac_address : 00:00:00:53:6B:B6 labels : app : static managedFields : name : staticip-pod namespace : default spec : containers : - image : docker.io/library/nginx:alpine imagePullPolicy : IfNotPresent name : qatest When using the above yaml to create a fixed address Pod, it prompts an IP address conflict: # kubectl apply -f pod-static.yaml Error from server ( annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 ) : error when creating \"pod-static.yaml\" : admission webhook \"pod-ip-validaing.kube-ovn.io\" denied the request: annotation ip address 10 .16.0.15 is conflict with ip crd static-7584848b74-fw9dm.default 10 .16.0.15 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Verify Webhook Take Effect"},{"location":"en/ops/change-default-subnet/","text":"Change Subnet CIDR \u00b6 If a subnet CIDR is created that conflicts or does not meet expectations, it can be modified by following the steps in this document. After modifying the subnet CIDR, the previously created Pods will not be able to access the network properly and need to be rebuilt. Careful consideration is recommended before operating\u3002This document is only for business subnet CIDR changes, if you need to Change the Join subnet CIDR, please refer to Change Join CIDR . Edit Subnet \u00b6 Use kubectl edit to modify cidrBlock \uff0c gateway and excludeIps . kubectl edit subnet test-subnet Rebuild all Pods under this Subnet \u00b6 Take the subnet binding test Namespace as example: for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n test --ignore-not-found done If only the default subnet is used, you can delete all Pods that are not in host network mode using the following command: for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done Change Default Subnet Settings \u00b6 If you are modifying the CIDR for the default Subnet, you also need to change the args of the kube-ovn-controller Deployment: args : - --default-cidr=10.17.0.0/16 - --default-gateway=10.17.0.1 - --default-exclude-ips=10.17.0.1 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Subnet CIDR"},{"location":"en/ops/change-default-subnet/#change-subnet-cidr","text":"If a subnet CIDR is created that conflicts or does not meet expectations, it can be modified by following the steps in this document. After modifying the subnet CIDR, the previously created Pods will not be able to access the network properly and need to be rebuilt. Careful consideration is recommended before operating\u3002This document is only for business subnet CIDR changes, if you need to Change the Join subnet CIDR, please refer to Change Join CIDR .","title":"Change Subnet CIDR"},{"location":"en/ops/change-default-subnet/#edit-subnet","text":"Use kubectl edit to modify cidrBlock \uff0c gateway and excludeIps . kubectl edit subnet test-subnet","title":"Edit Subnet"},{"location":"en/ops/change-default-subnet/#rebuild-all-pods-under-this-subnet","text":"Take the subnet binding test Namespace as example: for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n test --ignore-not-found done If only the default subnet is used, you can delete all Pods that are not in host network mode using the following command: for ns in $( kubectl get ns --no-headers -o custom-columns = NAME:.metadata.name ) ; do for pod in $( kubectl get pod --no-headers -n \" $ns \" --field-selector spec.restartPolicy = Always -o custom-columns = NAME:.metadata.name,HOST:spec.hostNetwork | awk '{if ($2!=\"true\") print $1}' ) ; do kubectl delete pod \" $pod \" -n \" $ns \" --ignore-not-found done done","title":"Rebuild all Pods under this Subnet"},{"location":"en/ops/change-default-subnet/#change-default-subnet-settings","text":"If you are modifying the CIDR for the default Subnet, you also need to change the args of the kube-ovn-controller Deployment: args : - --default-cidr=10.17.0.0/16 - --default-gateway=10.17.0.1 - --default-exclude-ips=10.17.0.1 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Default Subnet Settings"},{"location":"en/ops/change-join-subnet/","text":"Change Join Subnet CIDR \u00b6 If the Join subnet CIDR created conflicts or does not meet expectations, you can use this document to modify. After modifying the Join Subnet CIDR, the previously created Pods will not be able to access the external network normally and need to wait for the rebuild completed. Delete Join Subnet \u00b6 kubectl patch subnet join --type = 'json' -p '[{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\": []}]' kubectl delete subnet join Cleanup Allocated Config \u00b6 kubectl annotate node ovn.kubernetes.io/allocated = false --all --overwrite Modify Join Subnet \u00b6 Change Join Subnet args in kube-ovn-controller : kubectl edit deployment -n kube-system kube-ovn-controller Change the CIDR below: args : - --node-switch-cidr=100.51.0.0/16 Reboot the kube-ovn-controller and rebuild join Subnet: kubectl delete pod -n kube-system -lapp = kube-ovn-controller Check the new Join Subnet information: # kubectl get subnet NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS join ovn ovn-cluster IPv4 100 .51.0.0/16 false false false distributed 2 65531 0 0 [ \"100.51.0.1\" ] ovn-default ovn ovn-cluster IPv4 10 .17.0.0/16 false true true distributed 5 65528 0 0 [ \"10.17.0.1\" ] Reconfigure ovn0 NIC Address \u00b6 The ovn0 NIC information for each node needs to be re-updated, which can be done by restarting kube-ovn-cni : kubectl delete pod -n kube-system -l app = kube-ovn-cni \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Join Subnet CIDR"},{"location":"en/ops/change-join-subnet/#change-join-subnet-cidr","text":"If the Join subnet CIDR created conflicts or does not meet expectations, you can use this document to modify. After modifying the Join Subnet CIDR, the previously created Pods will not be able to access the external network normally and need to wait for the rebuild completed.","title":"Change Join Subnet CIDR"},{"location":"en/ops/change-join-subnet/#delete-join-subnet","text":"kubectl patch subnet join --type = 'json' -p '[{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\": []}]' kubectl delete subnet join","title":"Delete Join Subnet"},{"location":"en/ops/change-join-subnet/#cleanup-allocated-config","text":"kubectl annotate node ovn.kubernetes.io/allocated = false --all --overwrite","title":"Cleanup Allocated Config"},{"location":"en/ops/change-join-subnet/#modify-join-subnet","text":"Change Join Subnet args in kube-ovn-controller : kubectl edit deployment -n kube-system kube-ovn-controller Change the CIDR below: args : - --node-switch-cidr=100.51.0.0/16 Reboot the kube-ovn-controller and rebuild join Subnet: kubectl delete pod -n kube-system -lapp = kube-ovn-controller Check the new Join Subnet information: # kubectl get subnet NAME PROVIDER VPC PROTOCOL CIDR PRIVATE NAT DEFAULT GATEWAYTYPE V4USED V4AVAILABLE V6USED V6AVAILABLE EXCLUDEIPS join ovn ovn-cluster IPv4 100 .51.0.0/16 false false false distributed 2 65531 0 0 [ \"100.51.0.1\" ] ovn-default ovn ovn-cluster IPv4 10 .17.0.0/16 false true true distributed 5 65528 0 0 [ \"10.17.0.1\" ]","title":"Modify Join Subnet"},{"location":"en/ops/change-join-subnet/#reconfigure-ovn0-nic-address","text":"The ovn0 NIC information for each node needs to be re-updated, which can be done by restarting kube-ovn-cni : kubectl delete pod -n kube-system -l app = kube-ovn-cni \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Reconfigure ovn0 NIC Address"},{"location":"en/ops/change-log-level/","text":"Change Log Level \u00b6 Open kube-ovn.yaml and set the log level in the parameter list of the service startup script, such as: vi kube-ovn.yaml # ... - name: kube-ovn-controller image: \"docker.io/kubeovn/kube-ovn:v1.12.0\" imagePullPolicy: IfNotPresent args: - /kube-ovn/start-controller.sh - --v = 3 # ... # The higher the log level, the more detailed the log \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Log Level"},{"location":"en/ops/change-log-level/#change-log-level","text":"Open kube-ovn.yaml and set the log level in the parameter list of the service startup script, such as: vi kube-ovn.yaml # ... - name: kube-ovn-controller image: \"docker.io/kubeovn/kube-ovn:v1.12.0\" imagePullPolicy: IfNotPresent args: - /kube-ovn/start-controller.sh - --v = 3 # ... # The higher the log level, the more detailed the log \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Change Log Level"},{"location":"en/ops/change-ovn-central-node/","text":"Replace ovn-central Node \u00b6 Since ovn-nb and ovn-sb within ovn-central create separate etcd-like raft clusters, replacing the ovn-central node requires additional operations to ensure correct cluster state and consistent data. It is recommended that only one node be up and down at a time to avoid the cluster going into an unavailable state and affecting the overall cluster network. ovn-central Nodes Offline \u00b6 This document use the cluster below to describes how to remove the kube-ovn-control-plane2 node from the ovn-central as an example. # kubectl -n kube-system get pod -o wide | grep central ovn-central-6bf58cbc97-2cdhg 1 /1 Running 0 21m 172 .18.0.3 kube-ovn-control-plane <none> <none> ovn-central-6bf58cbc97-crmfp 1 /1 Running 0 21m 172 .18.0.5 kube-ovn-control-plane2 <none> <none> ovn-central-6bf58cbc97-lxmpl 1 /1 Running 0 21m 172 .18.0.4 kube-ovn-control-plane3 <none> <none> Kick Node in ovn-nb \u00b6 First check the ID of the node within the cluster for subsequent operations. # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2135194 ms ago, reason: timeout Last Election won: 2135188 ms ago Election timer: 5000 Log: [ 135 , 135 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-d64b ->d64b <-4984 ->4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 134 d64b ( d64b at tcp: [ 172 .18.0.5 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago status: ok kube-ovn-control-plane2 corresponds to a node IP of 172.18.0.5 and the corresponding ID within the cluster is d64b . Next, kick the node out of the ovn-nb cluster. # kubectl ko nb kick d64b started removal Check if the node has been kicked: # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2297649 ms ago, reason: timeout Last Election won: 2297643 ms ago Election timer: 5000 Log: [ 136 , 136 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-4984 ->4984 Disconnections: 2 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 136 match_index = 135 last msg 1270 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 135 status: ok Kick Node in ovn-sb \u00b6 Next, for the ovn-sb cluster, you need to first check the ID of the node within the cluster for subsequent operations. kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2395317 ms ago, reason: timeout Last Election won: 2395316 ms ago Election timer: 5000 Log: [ 130 , 130 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-e9f7 ->e9f7 <-6e84 ->6e84 Disconnections: 0 Servers: e9f7 ( e9f7 at tcp: [ 172 .18.0.5 ] :6644 ) next_index = 130 match_index = 129 last msg 1006 ms ago 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 130 match_index = 129 last msg 1004 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 129 status: ok kube-ovn-control-plane2 corresponds to node IP 172.18.0.5 and the corresponding ID within the cluster is e9f7 . Next, kick the node out of the ovn-sb cluster. # kubectl ko sb kick e9f7 started removal Check if the node has been kicked: # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2481636 ms ago, reason: timeout Last Election won: 2481635 ms ago Election timer: 5000 Log: [ 131 , 131 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-6e84 ->6e84 Disconnections: 2 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 131 match_index = 130 last msg 642 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 130 status: ok Delete Node Label and Downscale ovn-central \u00b6 Note that you need to remove the offline node from the node address of the ovn-central environment variable NODE_IPS . kubectl label node kube-ovn-control-plane2 kube-ovn/role- kubectl scale deployment -n kube-system ovn-central --replicas = 2 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4\" kubectl rollout status deployment/ovn-central -n kube-system Modify Components Address to ovn-central \u00b6 Modify ovs-ovn to remove the offline Node address: # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted Modify kube-ovn-controller to remove the offline Node address: # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out Clean Node \u00b6 Delete the database files in the kube-ovn-control-plane2 node to avoid errors when adding the node again: rm -rf /etc/origin/ovn To take a node offline from a Kubernetes cluster entirely, please continue with Delete Work Node . ovn-central Online \u00b6 The following steps will add a new Kubernetes node to the ovn-central cluster. Directory Check \u00b6 Check if the ovnnb_db.db or ovnsb_db.db file exists in the /etc/origin/ovn directory of the new node, and if so, delete it: rm -rf /etc/origin/ovn Check Current ovn-central Status \u00b6 If the current ovn-central cluster state is already abnormal, adding new nodes may cause the voting election to fail to pass the majority, affecting subsequent operations. # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 44 Leader: self Vote: self Last Election started 1855739 ms ago, reason: timeout Last Election won: 1855729 ms ago Election timer: 5000 Log: [ 147 , 147 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->4984 <-4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 147 match_index = 146 last msg 367 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 140 match_index = 146 status: ok # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 33 Leader: self Vote: self Last Election started 1868589 ms ago, reason: timeout Last Election won: 1868579 ms ago Election timer: 5000 Log: [ 142 , 142 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->6e84 <-6e84 Disconnections: 0 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 142 match_index = 141 last msg 728 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 134 match_index = 141 status: ok Label Node and Scale ovn-central \u00b6 Note that you need to add the online node address to the node address of the ovn-central environment variable NODE_IPS . kubectl label node kube-ovn-control-plane2 kube-ovn/role = master kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4,172.18.0.5\" kubectl rollout status deployment/ovn-central -n kube-system Modify Components Address to ovn-central \u00b6 Modify ovs-ovn to add the online Node address: # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted Modify kube-ovn-controller to add the online Node address: # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Replace ovn-central Node"},{"location":"en/ops/change-ovn-central-node/#replace-ovn-central-node","text":"Since ovn-nb and ovn-sb within ovn-central create separate etcd-like raft clusters, replacing the ovn-central node requires additional operations to ensure correct cluster state and consistent data. It is recommended that only one node be up and down at a time to avoid the cluster going into an unavailable state and affecting the overall cluster network.","title":"Replace ovn-central Node"},{"location":"en/ops/change-ovn-central-node/#ovn-central-nodes-offline","text":"This document use the cluster below to describes how to remove the kube-ovn-control-plane2 node from the ovn-central as an example. # kubectl -n kube-system get pod -o wide | grep central ovn-central-6bf58cbc97-2cdhg 1 /1 Running 0 21m 172 .18.0.3 kube-ovn-control-plane <none> <none> ovn-central-6bf58cbc97-crmfp 1 /1 Running 0 21m 172 .18.0.5 kube-ovn-control-plane2 <none> <none> ovn-central-6bf58cbc97-lxmpl 1 /1 Running 0 21m 172 .18.0.4 kube-ovn-control-plane3 <none> <none>","title":"ovn-central Nodes Offline"},{"location":"en/ops/change-ovn-central-node/#kick-node-in-ovn-nb","text":"First check the ID of the node within the cluster for subsequent operations. # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2135194 ms ago, reason: timeout Last Election won: 2135188 ms ago Election timer: 5000 Log: [ 135 , 135 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-d64b ->d64b <-4984 ->4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 134 d64b ( d64b at tcp: [ 172 .18.0.5 ] :6643 ) next_index = 135 match_index = 134 last msg 1084 ms ago status: ok kube-ovn-control-plane2 corresponds to a node IP of 172.18.0.5 and the corresponding ID within the cluster is d64b . Next, kick the node out of the ovn-nb cluster. # kubectl ko nb kick d64b started removal Check if the node has been kicked: # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2297649 ms ago, reason: timeout Last Election won: 2297643 ms ago Election timer: 5000 Log: [ 136 , 136 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-4984 ->4984 Disconnections: 2 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 136 match_index = 135 last msg 1270 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 2 match_index = 135 status: ok","title":"Kick Node in ovn-nb"},{"location":"en/ops/change-ovn-central-node/#kick-node-in-ovn-sb","text":"Next, for the ovn-sb cluster, you need to first check the ID of the node within the cluster for subsequent operations. kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2395317 ms ago, reason: timeout Last Election won: 2395316 ms ago Election timer: 5000 Log: [ 130 , 130 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-e9f7 ->e9f7 <-6e84 ->6e84 Disconnections: 0 Servers: e9f7 ( e9f7 at tcp: [ 172 .18.0.5 ] :6644 ) next_index = 130 match_index = 129 last msg 1006 ms ago 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 130 match_index = 129 last msg 1004 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 129 status: ok kube-ovn-control-plane2 corresponds to node IP 172.18.0.5 and the corresponding ID within the cluster is e9f7 . Next, kick the node out of the ovn-sb cluster. # kubectl ko sb kick e9f7 started removal Check if the node has been kicked: # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 2481636 ms ago, reason: timeout Last Election won: 2481635 ms ago Election timer: 5000 Log: [ 131 , 131 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-6e84 ->6e84 Disconnections: 2 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 131 match_index = 130 last msg 642 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 2 match_index = 130 status: ok","title":"Kick Node in ovn-sb"},{"location":"en/ops/change-ovn-central-node/#delete-node-label-and-downscale-ovn-central","text":"Note that you need to remove the offline node from the node address of the ovn-central environment variable NODE_IPS . kubectl label node kube-ovn-control-plane2 kube-ovn/role- kubectl scale deployment -n kube-system ovn-central --replicas = 2 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4\" kubectl rollout status deployment/ovn-central -n kube-system","title":"Delete Node Label and Downscale ovn-central"},{"location":"en/ops/change-ovn-central-node/#modify-components-address-to-ovn-central","text":"Modify ovs-ovn to remove the offline Node address: # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted Modify kube-ovn-controller to remove the offline Node address: # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out","title":"Modify Components Address to ovn-central"},{"location":"en/ops/change-ovn-central-node/#clean-node","text":"Delete the database files in the kube-ovn-control-plane2 node to avoid errors when adding the node again: rm -rf /etc/origin/ovn To take a node offline from a Kubernetes cluster entirely, please continue with Delete Work Node .","title":"Clean Node"},{"location":"en/ops/change-ovn-central-node/#ovn-central-online","text":"The following steps will add a new Kubernetes node to the ovn-central cluster.","title":"ovn-central Online"},{"location":"en/ops/change-ovn-central-node/#directory-check","text":"Check if the ovnnb_db.db or ovnsb_db.db file exists in the /etc/origin/ovn directory of the new node, and if so, delete it: rm -rf /etc/origin/ovn","title":"Directory Check"},{"location":"en/ops/change-ovn-central-node/#check-current-ovn-central-status","text":"If the current ovn-central cluster state is already abnormal, adding new nodes may cause the voting election to fail to pass the majority, affecting subsequent operations. # kubectl ko nb status 1b9a Name: OVN_Northbound Cluster ID: 32ca ( 32ca07fb-739b-4257-b510-12fa18e7cce8 ) Server ID: 1b9a ( 1b9a5d76-e69b-410c-8085-39943d0cd38c ) Address: tcp: [ 172 .18.0.3 ] :6643 Status: cluster member Role: leader Term: 44 Leader: self Vote: self Last Election started 1855739 ms ago, reason: timeout Last Election won: 1855729 ms ago Election timer: 5000 Log: [ 147 , 147 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->4984 <-4984 Disconnections: 0 Servers: 4984 ( 4984 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 147 match_index = 146 last msg 367 ms ago 1b9a ( 1b9a at tcp: [ 172 .18.0.3 ] :6643 ) ( self ) next_index = 140 match_index = 146 status: ok # kubectl ko sb status 3722 Name: OVN_Southbound Cluster ID: d4bd ( d4bd37a4-0400-499f-b4df-b4fd389780f0 ) Server ID: 3722 ( 3722d5ae-2ced-4820-a6b2-8b744d11fb3e ) Address: tcp: [ 172 .18.0.3 ] :6644 Status: cluster member Role: leader Term: 33 Leader: self Vote: self Last Election started 1868589 ms ago, reason: timeout Last Election won: 1868579 ms ago Election timer: 5000 Log: [ 142 , 142 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: ->6e84 <-6e84 Disconnections: 0 Servers: 6e84 ( 6e84 at tcp: [ 172 .18.0.4 ] :6644 ) next_index = 142 match_index = 141 last msg 728 ms ago 3722 ( 3722 at tcp: [ 172 .18.0.3 ] :6644 ) ( self ) next_index = 134 match_index = 141 status: ok","title":"Check Current ovn-central Status"},{"location":"en/ops/change-ovn-central-node/#label-node-and-scale-ovn-central","text":"Note that you need to add the online node address to the node address of the ovn-central environment variable NODE_IPS . kubectl label node kube-ovn-control-plane2 kube-ovn/role = master kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl set env deployment/ovn-central -n kube-system NODE_IPS = \"172.18.0.3,172.18.0.4,172.18.0.5\" kubectl rollout status deployment/ovn-central -n kube-system","title":"Label Node and Scale ovn-central"},{"location":"en/ops/change-ovn-central-node/#modify-components-address-to-ovn-central_1","text":"Modify ovs-ovn to add the online Node address: # kubectl set env daemonset/ovs-ovn -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" daemonset.apps/ovs-ovn env updated # kubectl delete pod -n kube-system -lapp=ovs pod \"ovs-ovn-4f6jc\" deleted pod \"ovs-ovn-csn2w\" deleted pod \"ovs-ovn-mpbmb\" deleted Modify kube-ovn-controller to add the online Node address: # kubectl set env deployment/kube-ovn-controller -n kube-system OVN_DB_IPS=\"172.18.0.3,172.18.0.4,172.18.0.5\" deployment.apps/kube-ovn-controller env updated # kubectl rollout status deployment/kube-ovn-controller -n kube-system Waiting for deployment \"kube-ovn-controller\" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment \"kube-ovn-controller\" rollout to finish: 2 of 3 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Modify Components Address to ovn-central"},{"location":"en/ops/delete-worker-node/","text":"Delete Work Node \u00b6 If the node is simply removed from Kubernetes, the ovn-controller process running in ovs-ovn on the node will periodically connect to ovn-central to register relevant network information. This leads to additional resource waste and potential rule conflict risk\u3002 Therefore, when removing nodes from within Kubernetes, follow the steps below to ensure that related resources are cleaned up properly. This document describes the steps to delete a worker node, if you want to change the node where ovn-central is located, please refer to Replace ovn-central Node . Evict Pods on the Node \u00b6 # kubectl drain kube-ovn-worker --ignore-daemonsets --force node/kube-ovn-worker cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-ovn-cni-zt74b, kube-system/kube-ovn-pinger-5rxfs, kube-system/kube-proxy-jpmnm, kube-system/ovs-ovn-v2kll evicting pod kube-system/coredns-64897985d-qsgpt evicting pod local-path-storage/local-path-provisioner-5ddd94ff66-llss6 evicting pod kube-system/kube-ovn-controller-8459db5ff4-94lxb pod/kube-ovn-controller-8459db5ff4-94lxb evicted pod/coredns-64897985d-qsgpt evicted pod/local-path-provisioner-5ddd94ff66-llss6 evicted node/kube-ovn-worker drained Stop kubelet and docker \u00b6 This step stops the ovs-ovn container to avoid registering information to ovn-central . Log into to the corresponding node and ruu the following commands: systemctl stop kubelet systemctl stop docker If using containerd as the CRI, the following command needs to be executed to stop the ovs-ovn container: crictl rm -f $( crictl ps | grep openvswitch | awk '{print $1}' ) Cleanup Files on Node \u00b6 rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn Delete the Node \u00b6 kubectl delete no kube-ovn-01 Check If Node Removed from OVN-SB \u00b6 In the example below, the node kube-ovn-worker is not removed: # kubectl ko sbctl show Chassis \"b0564934-5a0d-4804-a4c0-476c93596a17\" hostname: kube-ovn-worker Encap geneve ip: \"172.18.0.2\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-5rxfs.kube-system Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system Delete the Chassis Manually \u00b6 Use the uuid find above to delete the chassis: # kubectl ko sbctl chassis-del b0564934-5a0d-4804-a4c0-476c93596a17 # kubectl ko sbctl show Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Deleting Work Node"},{"location":"en/ops/delete-worker-node/#delete-work-node","text":"If the node is simply removed from Kubernetes, the ovn-controller process running in ovs-ovn on the node will periodically connect to ovn-central to register relevant network information. This leads to additional resource waste and potential rule conflict risk\u3002 Therefore, when removing nodes from within Kubernetes, follow the steps below to ensure that related resources are cleaned up properly. This document describes the steps to delete a worker node, if you want to change the node where ovn-central is located, please refer to Replace ovn-central Node .","title":"Delete Work Node"},{"location":"en/ops/delete-worker-node/#evict-pods-on-the-node","text":"# kubectl drain kube-ovn-worker --ignore-daemonsets --force node/kube-ovn-worker cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-ovn-cni-zt74b, kube-system/kube-ovn-pinger-5rxfs, kube-system/kube-proxy-jpmnm, kube-system/ovs-ovn-v2kll evicting pod kube-system/coredns-64897985d-qsgpt evicting pod local-path-storage/local-path-provisioner-5ddd94ff66-llss6 evicting pod kube-system/kube-ovn-controller-8459db5ff4-94lxb pod/kube-ovn-controller-8459db5ff4-94lxb evicted pod/coredns-64897985d-qsgpt evicted pod/local-path-provisioner-5ddd94ff66-llss6 evicted node/kube-ovn-worker drained","title":"Evict Pods on the Node"},{"location":"en/ops/delete-worker-node/#stop-kubelet-and-docker","text":"This step stops the ovs-ovn container to avoid registering information to ovn-central . Log into to the corresponding node and ruu the following commands: systemctl stop kubelet systemctl stop docker If using containerd as the CRI, the following command needs to be executed to stop the ovs-ovn container: crictl rm -f $( crictl ps | grep openvswitch | awk '{print $1}' )","title":"Stop kubelet and docker"},{"location":"en/ops/delete-worker-node/#cleanup-files-on-node","text":"rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn","title":"Cleanup Files on Node"},{"location":"en/ops/delete-worker-node/#delete-the-node","text":"kubectl delete no kube-ovn-01","title":"Delete the Node"},{"location":"en/ops/delete-worker-node/#check-if-node-removed-from-ovn-sb","text":"In the example below, the node kube-ovn-worker is not removed: # kubectl ko sbctl show Chassis \"b0564934-5a0d-4804-a4c0-476c93596a17\" hostname: kube-ovn-worker Encap geneve ip: \"172.18.0.2\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-5rxfs.kube-system Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system","title":"Check If Node Removed from OVN-SB"},{"location":"en/ops/delete-worker-node/#delete-the-chassis-manually","text":"Use the uuid find above to delete the chassis: # kubectl ko sbctl chassis-del b0564934-5a0d-4804-a4c0-476c93596a17 # kubectl ko sbctl show Chassis \"6a29de7e-d731-4eaf-bacd-2f239ee52b28\" hostname: kube-ovn-control-plane Encap geneve ip: \"172.18.0.3\" options: { csum = \"true\" } Port_Binding coredns-64897985d-nbfln.kube-system Port_Binding node-kube-ovn-control-plane Port_Binding local-path-provisioner-5ddd94ff66-h4tn9.local-path-storage Port_Binding kube-ovn-pinger-hf2p6.kube-system Port_Binding coredns-64897985d-fhwlw.kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Delete the Chassis Manually"},{"location":"en/ops/faq/","text":"FAQ \u00b6 Kylin ARM system cross-host container access intermittently fails \u00b6 Behavior \u00b6 There is a problem with Kylin ARM system and some NIC offload, which can cause intermittent container network failure. Use netstat to identify the problem: # netstat -us IcmpMsg: InType0: 22 InType3: 24 InType8: 117852 OutType0: 117852 OutType3: 29 OutType8: 22 Udp: 3040636 packets received 0 packets to unknown port received. 4 packet receive errors 602 packets sent 0 receive buffer errors 0 send buffer errors InCsumErrors: 4 UdpLite: IpExt: InBcastPkts: 10244 InOctets: 4446320361 OutOctets: 1496815600 InBcastOctets: 3095950 InNoECTPkts: 7683903 If InCsumErrors is present and increases with netwoork failures, you can confirm that this is the problem. Solution \u00b6 The fundamental solution requires communication with Kylin and the corresponding network card manufacturer to update the system and drivers. A temporary solution would be to turn off tx offload on the physical NIC, but this would cause a significant degradation in tcp performance. ethtool -K eth0 tx off From the community feedback, the problem can be solved by the 4.19.90-25.16.v2101 kernel. Pod can not Access Service \u00b6 Behavior \u00b6 Pod can not access Service, and dmesg show errors: netlink\uff1aUnknown conntrack attr ( type = 6 , max = 5 ) openvswitch: netlink: Flow actions may not be safe on all matching packets. This log indicates that the in-kernel OVS version is too low to support the corresponding NAT operation. Solution \u00b6 Upgrade the kernel module or compile the OVS kernel module manually. If you are using an Overlay network you can change the kube-ovn-controller args, setting --enable-lb=false to disable the OVN LB to use kube-proxy for service forwarding. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"FAQ"},{"location":"en/ops/faq/#faq","text":"","title":"FAQ"},{"location":"en/ops/faq/#kylin-arm-system-cross-host-container-access-intermittently-fails","text":"","title":"Kylin ARM system cross-host container access intermittently fails"},{"location":"en/ops/faq/#behavior","text":"There is a problem with Kylin ARM system and some NIC offload, which can cause intermittent container network failure. Use netstat to identify the problem: # netstat -us IcmpMsg: InType0: 22 InType3: 24 InType8: 117852 OutType0: 117852 OutType3: 29 OutType8: 22 Udp: 3040636 packets received 0 packets to unknown port received. 4 packet receive errors 602 packets sent 0 receive buffer errors 0 send buffer errors InCsumErrors: 4 UdpLite: IpExt: InBcastPkts: 10244 InOctets: 4446320361 OutOctets: 1496815600 InBcastOctets: 3095950 InNoECTPkts: 7683903 If InCsumErrors is present and increases with netwoork failures, you can confirm that this is the problem.","title":"Behavior"},{"location":"en/ops/faq/#solution","text":"The fundamental solution requires communication with Kylin and the corresponding network card manufacturer to update the system and drivers. A temporary solution would be to turn off tx offload on the physical NIC, but this would cause a significant degradation in tcp performance. ethtool -K eth0 tx off From the community feedback, the problem can be solved by the 4.19.90-25.16.v2101 kernel.","title":"Solution"},{"location":"en/ops/faq/#pod-can-not-access-service","text":"","title":"Pod can not Access Service"},{"location":"en/ops/faq/#behavior_1","text":"Pod can not access Service, and dmesg show errors: netlink\uff1aUnknown conntrack attr ( type = 6 , max = 5 ) openvswitch: netlink: Flow actions may not be safe on all matching packets. This log indicates that the in-kernel OVS version is too low to support the corresponding NAT operation.","title":"Behavior"},{"location":"en/ops/faq/#solution_1","text":"Upgrade the kernel module or compile the OVS kernel module manually. If you are using an Overlay network you can change the kube-ovn-controller args, setting --enable-lb=false to disable the OVN LB to use kube-proxy for service forwarding. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Solution"},{"location":"en/ops/from-calico/","text":"Install Kube-OVN From Calico \u00b6 If a Kubernetes cluster already has Calico installed and needs to change to Kube-OVN you can refer to this document. Since the installation of Calico may vary from version to version and the existing Pod network may be disrupted during the replacement process, it is recommended that you plan ahead and compare the differences in Calico installation from version to version. Uninstall Calico \u00b6 For Calico installed from an Operator: kubectl delete -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml kubectl delete -f https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml For Calico installed from manifests: kubectl delete -f https://projectcalico.docs.tigera.io/manifests/calico.yaml Cleanup Config Files \u00b6 Delete the CNI-related configuration files on each machine, depending on the environment: rm -f /etc/cni/net.d/10-calico.conflist rm -f /etc/cni/net.d/calico-kubeconfig Calico still leaves routing rules, iptables rules, veth network interfaces and other configuration information on the node, so it is recommended to reboot the node to clean up the relevant configuration to avoid problems that are difficult to troubleshoot. Install Kube-OVN \u00b6 You can refer to One Click Installation to install Kube-OVN as usual. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Install Kube-OVN From Calico"},{"location":"en/ops/from-calico/#install-kube-ovn-from-calico","text":"If a Kubernetes cluster already has Calico installed and needs to change to Kube-OVN you can refer to this document. Since the installation of Calico may vary from version to version and the existing Pod network may be disrupted during the replacement process, it is recommended that you plan ahead and compare the differences in Calico installation from version to version.","title":"Install Kube-OVN From Calico"},{"location":"en/ops/from-calico/#uninstall-calico","text":"For Calico installed from an Operator: kubectl delete -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml kubectl delete -f https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml For Calico installed from manifests: kubectl delete -f https://projectcalico.docs.tigera.io/manifests/calico.yaml","title":"Uninstall Calico"},{"location":"en/ops/from-calico/#cleanup-config-files","text":"Delete the CNI-related configuration files on each machine, depending on the environment: rm -f /etc/cni/net.d/10-calico.conflist rm -f /etc/cni/net.d/calico-kubeconfig Calico still leaves routing rules, iptables rules, veth network interfaces and other configuration information on the node, so it is recommended to reboot the node to clean up the relevant configuration to avoid problems that are difficult to troubleshoot.","title":"Cleanup Config Files"},{"location":"en/ops/from-calico/#install-kube-ovn","text":"You can refer to One Click Installation to install Kube-OVN as usual. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Install Kube-OVN"},{"location":"en/ops/kubectl-ko/","text":"Kubectl Plugin \u00b6 To facilitate daily operations and maintenance, Kube-OVN provides the kubectl plug-in tool, which allows administrators to perform daily operations through this command. For examples: Check OVN database information and status, OVN database backup and restore, OVS related information, tcpdump specific containers, specific link logical topology, network problem diagnosis and performance optimization. Plugin Installation \u00b6 Kube-OVN installation will deploy the plugin to each node by default. If the machine that runs kubectl is not in the cluster, or if you need to reinstall the plugin, please refer to the following steps: Download kubectl-ko file: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/kubectl-ko Move file to $PATH : mv kubectl-ko /usr/local/bin/kubectl-ko Add executable permissions: chmod +x /usr/local/bin/kubectl-ko Check if the plugin works properly: # kubectl plugin list The following compatible plugins are available: /usr/local/bin/kubectl-ko Plugin Usage \u00b6 Running kubectl ko will show all the available commands and usage descriptions, as follows: # kubectl ko kubectl ko { subcommand } [ option... ] Available Subcommands: [ nb | sb ] [ status | kick | backup | dbstatus | restore ] ovn-db operations show cluster status, kick stale server, backup database, get db consistency status or restore ovn nb db when met 'inconsistent data' error nbctl [ ovn-nbctl options ... ] invoke ovn-nbctl sbctl [ ovn-sbctl options ... ] invoke ovn-sbctl vsctl { nodeName } [ ovs-vsctl options ... ] invoke ovs-vsctl on the specified node ofctl { nodeName } [ ovs-ofctl options ... ] invoke ovs-ofctl on the specified node dpctl { nodeName } [ ovs-dpctl options ... ] invoke ovs-dpctl on the specified node appctl { nodeName } [ ovs-appctl options ... ] invoke ovs-appctl on the specified node tcpdump { namespace/podname } [ tcpdump options ... ] capture pod traffic trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply echo \" diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] diagnose connectivity of all nodes or a specific node or specify subnet's ds pod or IPPorts like 'tcp-172.18.0.2-53,udp-172.18.0.3-53'\" tuning { install-fastpath | local-install-fastpath | remove-fastpath | install-stt | local-install-stt | remove-stt } { centos7 | centos8 }} [ kernel-devel-version ] deploy kernel optimisation components to the system reload restart all kube-ovn components log { kube-ovn | ovn | ovs | linux | all } save log to ./kubectl-ko-log/ perf [ image ] performance test default image is kubeovn/test:v1.12.0 The specific functions and usage of each command are described below. [nb | sb] [status | kick | backup | dbstatus | restore] \u00b6 This subcommand mainly operates on OVN northbound or southbound databases, including database cluster status check, database node offline, database backup, database storage status check and database repair. DB Cluster Status Check \u00b6 This command executes ovs-appctl cluster/status on the leader node of the corresponding OVN database to show the cluster status: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 280309 ms ago, reason: timeout Last Election won: 280309 ms ago Election timer: 5000 Log: [ 139 , 139 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-8723 ->8723 <-85d6 ->85d6 Disconnections: 0 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 8723 ( 8723 at tcp: [ 172 .18.0.3 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 138 status: ok If the match_index under Server has a large difference and the last msg time is long, the corresponding Server may not respond for a long time and needs to be checked further. DB Nodes Offline \u00b6 This command removes a node from the OVN database and is required when a node is taken offline or replaced. The following is an example of the cluster status from the previous command, to offline the 172.18.0.3 node: # kubectl ko nb kick 8723 started removal Check the database cluster status again to confirm that the node has been removed: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 324356 ms ago, reason: timeout Last Election won: 324356 ms ago Election timer: 5000 Log: [ 140 , 140 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-85d6 ->85d6 Disconnections: 2 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 140 match_index = 139 last msg 848 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 139 status: ok DB Backup \u00b6 This subcommand backs up the current OVN database locally and can be used for disaster recovery: # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup Database Storage Status Check \u00b6 This command is used to check if the database file is corrupt: # kubectl ko nb dbstatus status: ok If error happens, inconsistent data is displayed and needs to be fixed with the following command. Database Repair \u00b6 If the database status goes to inconsistent data , this command can be used to repair: # kubectl ko nb restore deployment.apps/ovn-central scaled ovn-central original replicas is 3 first nodeIP is 172 .18.0.5 ovs-ovn pod on node 172 .18.0.5 is ovs-ovn-8jxv9 ovs-ovn pod on node 172 .18.0.3 is ovs-ovn-sjzb6 ovs-ovn pod on node 172 .18.0.4 is ovs-ovn-t87zk backup nb db file restore nb db file, operate in pod ovs-ovn-8jxv9 deployment.apps/ovn-central scaled finish restore nb db file and ovn-central replicas recreate ovs-ovn pods pod \"ovs-ovn-8jxv9\" deleted pod \"ovs-ovn-sjzb6\" deleted pod \"ovs-ovn-t87zk\" deleted [nbctl | sbctl] [options ...] \u00b6 This subcommand executes the ovn-nbctl and ovn-sbctl commands directly into the leader node of the OVN northbound or southbound database. For more detailed usage of this command, please refer to the official documentation of the upstream OVN ovn-nbctl(8) \u548c ovn-sbctl(8) \u3002 # kubectl ko nbctl show switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port node-liumengxin-ovn2-192.168.137.177 addresses: [ \"00:00:00:D9:58:B8 100.64.0.3\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port alertmanager-main-0.monitoring addresses: [ \"00:00:00:6C:DF:A3 10.16.0.19\" ] port kube-state-metrics-5d6885d89-4nf8h.monitoring addresses: [ \"00:00:00:6F:02:1C 10.16.0.15\" ] port fake-kubelet-67c55dfd89-pv86k.kube-system addresses: [ \"00:00:00:5C:12:E8 10.16.19.177\" ] port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] vsctl {nodeName} [options ...] \u00b6 This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-vsctl command to query and configure vswitchd . For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-vsctl(8) \u3002 # kubectl ko vsctl kube-ovn-01 show 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port mirror0 Interface mirror0 type: internal Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" ofctl {nodeName} [options ...] \u00b6 This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-ofctl command to query or manage OpenFlow. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-ofctl(8) \u3002 # kubectl ko ofctl kube-ovn-01 dump-flows br-int NXST_FLOW reply ( xid = 0x4 ) : flags =[ more ] cookie = 0xcf3429e6, duration = 671791 .432s, table = 0 , n_packets = 0 , n_bytes = 0 , idle_age = 65534 , hard_age = 65534 , priority = 100 ,in_port = 2 actions = load:0x4->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x1->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xc91413c6, duration = 671791 .431s, table = 0 , n_packets = 907489 , n_bytes = 99978275 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 7 actions = load:0x1->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x4->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xf180459, duration = 671791 .431s, table = 0 , n_packets = 17348582 , n_bytes = 2667811214 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 6317 actions = load:0xa->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x9->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0x7806dd90, duration = 671791 .431s, table = 0 , n_packets = 3235428 , n_bytes = 833821312 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 1 actions = load:0xd->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x3->NXM_NX_REG14 [] ,resubmit ( ,8 ) ... dpctl {nodeName} [options ...] \u00b6 This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-dpctl command to query or manage the OVS datapath. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-dpctl(8) \u3002 # kubectl ko dpctl kube-ovn-01 show system@ovs-system: lookups: hit:350805055 missed:21983648 lost:73 flows: 105 masks: hit:1970748791 total:22 hit/pkt:5.29 port 0 : ovs-system ( internal ) port 1 : ovn0 ( internal ) port 2 : mirror0 ( internal ) port 3 : br-int ( internal ) port 4 : stt_sys_7471 ( stt: packet_type = ptap ) port 5 : eeb4d9e51b5d_h port 6 : a2c1a8a8b83a_h port 7 : 4fa5c4cbb1a5_h appctl {nodeName} [options ...] \u00b6 This command will enter the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-appctl command to operate the associated daemon process. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-appctl(8) \u3002 # kubectl ko appctl kube-ovn-01 vlog/list console syslog file ------- ------ ------ backtrace OFF ERR INFO bfd OFF ERR INFO bond OFF ERR INFO bridge OFF ERR INFO bundle OFF ERR INFO bundles OFF ERR INFO ... tcpdump {namespace/podname} [tcpdump options ...] \u00b6 This command will enter the kube-ovn-cni container on the machine where namespace/podname is located, and run tcpdump to capture the traffic on the veth NIC of the corresponding container, which can be used to troubleshoot network-related problems. # kubectl ko tcpdump default/ds1-l6n7p icmp + kubectl exec -it kube-ovn-cni-wlg4s -n kube-ovn -- tcpdump -nn -i d7176fe7b4e0_h icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on d7176fe7b4e0_h, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 06 :52:36.619688 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 1 , length 64 06 :52:36.619746 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 1 , length 64 06 :52:37.619588 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 2 , length 64 06 :52:37.619630 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 2 , length 64 06 :52:38.619933 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 3 , length 64 06 :52:38.619973 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 3 , length 64 trace [arguments ...] \u00b6 This command will print the OVN logical flow table and the final Openflow flow table when the Pod/node accesses an address through a specific protocol, so that it make locate flow table related problems during development or troubleshooting much easy. Supported commands: kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } kubectl ko trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } Example: # kubectl ko trace default/ds1-l6n7p 8.8.8.8 icmp + kubectl exec ovn-central-5bc494cb5-np9hm -n kube-ovn -- ovn-trace --ct = new ovn-default 'inport == \"ds1-l6n7p.default\" && ip.ttl == 64 && icmp && eth.src == 0a:00:00:10:00:05 && ip4.src == 10.16.0.4 && eth.dst == 00:00:00:B8:CA:43 && ip4.dst == 8.8.8.8' # icmp,reg14=0xf,vlan_tci=0x0000,dl_src=0a:00:00:10:00:05,dl_dst=00:00:00:b8:ca:43,nw_src=10.16.0.4,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 ingress ( dp = \"ovn-default\" , inport = \"ds1-l6n7p.default\" ) ----------------------------------------------------- 0 . ls_in_port_sec_l2 ( ovn-northd.c:4143 ) : inport == \"ds1-l6n7p.default\" && eth.src == { 0a:00:00:10:00:05 } , priority 50 , uuid 39453393 next ; 1 . ls_in_port_sec_ip ( ovn-northd.c:2898 ) : inport == \"ds1-l6n7p.default\" && eth.src == 0a:00:00:10:00:05 && ip4.src == { 10 .16.0.4 } , priority 90 , uuid 81bcd485 next ; 3 . ls_in_pre_acl ( ovn-northd.c:3269 ) : ip, priority 100 , uuid 7b4f4971 reg0 [ 0 ] = 1 ; next ; 5 . ls_in_pre_stateful ( ovn-northd.c:3396 ) : reg0 [ 0 ] == 1 , priority 100 , uuid 36cdd577 ct_next ; ct_next ( ct_state = new | trk ) ------------------------- 6 . ls_in_acl ( ovn-northd.c:3759 ) : ip && ( !ct.est || ( ct.est && ct_label.blocked == 1 )) , priority 1 , uuid 7608af5b reg0 [ 1 ] = 1 ; next ; 10 . ls_in_stateful ( ovn-northd.c:3995 ) : reg0 [ 1 ] == 1 , priority 100 , uuid 2aba1b90 ct_commit ( ct_label = 0 /0x1 ) ; next ; 16 . ls_in_l2_lkup ( ovn-northd.c:4470 ) : eth.dst == 00 :00:00:b8:ca:43, priority 50 , uuid 5c9c3c9f outport = \"ovn-default-ovn-cluster\" ; output ; ... If the trace object is a virtual machine running in Underlay network, additional parameters is needed to specify the destination Mac address. kubectl ko trace default/virt-handler-7lvml 8 .8.8.8 82 :7c:9f:83:8c:01 icmp diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] \u00b6 Diagnose the status of cluster network components and go to the corresponding node's kube-ovn-pinger to detect connectivity and network latency from the current node to other nodes and critical services. # kubectl ko diagnose all switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default port prometheus-k8s-1.monitoring addresses: [ \"00:00:00:AA:37:DF 10.16.0.23\" ] router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] Routing Policies 31000 ip4.dst == 10 .16.0.0/16 allow 31000 ip4.dst == 100 .64.0.0/16 allow 30000 ip4.dst == 192 .168.137.177 reroute 100 .64.0.3 30000 ip4.dst == 192 .168.137.178 reroute 100 .64.0.4 29000 ip4.src == $ovn .default.fake.6_ip4 reroute 100 .64.0.22 29000 ip4.src == $ovn .default.fake.7_ip4 reroute 100 .64.0.21 29000 ip4.src == $ovn .default.fake.8_ip4 reroute 100 .64.0.23 29000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 reroute 100 .64.0.4 20000 ip4.src == $ovn .default.liumengxin.ovn1.192.168.137.176_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.2 20000 ip4.src == $ovn .default.liumengxin.ovn2.192.168.137.177_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.3 20000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.4 IPv4 Routes Route Table <main>: 0 .0.0.0/0 100 .64.0.1 dst-ip UUID LB PROTO VIP IPs e9bcfd9d-793e-4431-9073-6dec96b75d71 cluster-tcp-load tcp 10 .100.209.132:10660 192 .168.137.176:10660 tcp 10 .101.239.192:6641 192 .168.137.177:6641 tcp 10 .101.240.101:3000 10 .16.0.7:3000 tcp 10 .103.184.186:6642 192 .168.137.177:6642 35d2b7a5-e3a7-485a-a4b7-b4970eb0e63b cluster-tcp-sess tcp 10 .100.158.128:8080 10 .16.0.10:8080,10.16.0.5:8080,10.16.63.30:8080 tcp 10 .107.26.215:8080 10 .16.0.19:8080,10.16.0.20:8080,10.16.0.21:8080 tcp 10 .107.26.215:9093 10 .16.0.19:9093,10.16.0.20:9093,10.16.0.21:9093 tcp 10 .98.187.99:8080 10 .16.0.22:8080,10.16.0.23:8080 tcp 10 .98.187.99:9090 10 .16.0.22:9090,10.16.0.23:9090 f43303e4-89aa-4d3e-a3dc-278a552fe27b cluster-udp-load udp 10 .96.0.10:53 10 .16.0.4:53,10.16.0.9:53 _uuid : 06776304 -5a96-43ed-90c4-c4854c251699 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn2_192.168.137.177_underlay_v6 _uuid : 62690625 -87d5-491c-8675-9fd83b1f433c addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn1_192.168.137.176_underlay_v6 _uuid : b03a9bae-94d5-4562-b34c-b5f6198e180b addresses : [ \"10.16.0.0/16\" , \"100.64.0.0/16\" , \"172.22.0.0/16\" , \"192.101.0.0/16\" ] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv4 _uuid : e1056f3a-24cc-4666-8a91-75ee6c3c2426 addresses : [] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv6 _uuid : 3e5d5fff-e670-47b2-a2f5-a39f4698a8c5 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn3_192.168.137.178_underlay_v6 _uuid : 2d85dbdc-d0db-4abe-b19e-cc806d32b492 action : drop direction : from-lport external_ids : {} label : 0 log : false match : \"inport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] _uuid : de790cc8-f155-405f-bb32-5a51f30c545f action : drop direction : to-lport external_ids : {} label : 0 log : false match : \"outport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] Chassis \"e15ed4d4-1780-4d50-b09e-ea8372ed48b8\" hostname: liumengxin-ovn1-192.168.137.176 Encap stt ip: \"192.168.137.176\" options: { csum = \"true\" } Port_Binding node-liumengxin-ovn1-192.168.137.176 Port_Binding perf-6vxkn.default Port_Binding kube-state-metrics-5d6885d89-4nf8h.monitoring Port_Binding alertmanager-main-0.monitoring Port_Binding kube-ovn-pinger-6ftdf.kube-system Port_Binding fake-kubelet-67c55dfd89-pv86k.kube-system Port_Binding prometheus-k8s-0.monitoring Chassis \"eef07da1-f8ad-4775-b14d-bd6a3b4eb0d5\" hostname: liumengxin-ovn3-192.168.137.178 Encap stt ip: \"192.168.137.178\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-7twb4.kube-system Port_Binding prometheus-adapter-86df476d87-rl88g.monitoring Port_Binding prometheus-k8s-1.monitoring Port_Binding node-liumengxin-ovn3-192.168.137.178 Port_Binding perf-ff475.default Port_Binding alertmanager-main-1.monitoring Port_Binding blackbox-exporter-676d976865-tvsjd.monitoring Chassis \"efa253c9-494d-4719-83ae-b48ab0f11c03\" hostname: liumengxin-ovn2-192.168.137.177 Encap stt ip: \"192.168.137.177\" options: { csum = \"true\" } Port_Binding grafana-6c4c6b8fb7-pzd2c.monitoring Port_Binding node-liumengxin-ovn2-192.168.137.177 Port_Binding alertmanager-main-2.monitoring Port_Binding coredns-6789c94dd8-9jqsz.kube-system Port_Binding coredns-6789c94dd8-25d4r.kube-system Port_Binding prometheus-operator-7bbc99fc8b-wgjm4.monitoring Port_Binding prometheus-adapter-86df476d87-gdxmc.monitoring Port_Binding perf-fjnws.default Port_Binding kube-ovn-pinger-vh2xg.kube-system ds kube-proxy ready kube-proxy ready deployment ovn-central ready deployment kube-ovn-controller ready ds kube-ovn-cni ready ds ovs-ovn ready deployment coredns ready ovn-nb leader check ok ovn-sb leader check ok ovn-northd leader check ok ### kube-ovn-controller recent log ### start to diagnose node liumengxin-ovn1-192.168.137.176 #### ovn-controller log: 2022 -06-03T00:56:44.897Z | 16722 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:06:44.912Z | 16723 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:16:44.925Z | 16724 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:26:44.936Z | 16725 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:36:44.959Z | 16726 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:46:44.974Z | 16727 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:56:44.988Z | 16728 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:06:45.001Z | 16729 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:16:45.025Z | 16730 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:26:45.040Z | 16731 | inc_proc_eng | INFO | User triggered force recompute. #### ovs-vswitchd log: 2022 -06-02T23:03:00.137Z | 00079 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:f9d1 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-02T23:23:31.840Z | 00080 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:15b2 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:09:15.659Z | 00081 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:dc:e3:63,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.63.30,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:e5a5 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x150017000004,src = 192 .168.137.178,dst = 192 .168.137.176,ttl = 64 ,tp_src = 9239 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.63.30,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:30:13.409Z | 00064 | dpif ( handler2 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:6b4a with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T02:02:33.832Z | 00082 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:a819 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 #### ovs-vsctl show results: 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port \"04d03360e9a0_h\" Interface \"04d03360e9a0_h\" Port eeb4d9e51b5d_h Interface eeb4d9e51b5d_h Port mirror0 Interface mirror0 type: internal Port \"8e5d887ccd80_h\" Interface \"8e5d887ccd80_h\" Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port \"17512d5be1f1_h\" Interface \"17512d5be1f1_h\" Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" #### pinger diagnose results: I0603 10 :35:04.349404 17619 pinger.go:19 ] ------------------------------------------------------------------------------- Kube-OVN: Version: v1.12.0 Build: 2022 -04-24_08:02:50 Commit: git-73f9d15 Go Version: go1.17.8 Arch: amd64 ------------------------------------------------------------------------------- I0603 10 :35:04.376797 17619 config.go:166 ] pinger config is & { KubeConfigFile: KubeClient:0xc000493380 Port:8080 DaemonSetNamespace:kube-system DaemonSetName:kube-ovn-pinger Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:liumengxin-ovn1-192.168.137.176 HostIP:192.168.137.176 PodName:kube-ovn-pinger-6ftdf PodIP:10.16.0.10 PodProtocols: [ IPv4 ] ExternalAddress: NetworkMode:kube-ovn PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid } I0603 10 :35:04.449166 17619 exporter.go:75 ] liumengxin-ovn1-192.168.137.176: exporter connect successfully I0603 10 :35:04.554011 17619 ovn.go:21 ] ovs-vswitchd and ovsdb are up I0603 10 :35:04.651293 17619 ovn.go:33 ] ovn_controller is up I0603 10 :35:04.651342 17619 ovn.go:39 ] start to check port binding I0603 10 :35:04.749613 17619 ovn.go:135 ] chassis id is 1d7f3d6c-eec5-4b3c-adca-2969d9cdfd80 I0603 10 :35:04.763487 17619 ovn.go:49 ] port in sb is [ node-liumengxin-ovn1-192.168.137.176 perf-6vxkn.default kube-state-metrics-5d6885d89-4nf8h.monitoring alertmanager-main-0.monitoring kube-ovn-pinger-6ftdf.kube-system fake-kubelet-67c55dfd89-pv86k.kube-system prometheus-k8s-0.monitoring ] I0603 10 :35:04.763583 17619 ovn.go:61 ] ovs and ovn-sb binding check passed I0603 10 :35:05.049309 17619 ping.go:259 ] start to check apiserver connectivity I0603 10 :35:05.053666 17619 ping.go:268 ] connect to apiserver success in 4 .27ms I0603 10 :35:05.053786 17619 ping.go:129 ] start to check pod connectivity I0603 10 :35:05.249590 17619 ping.go:159 ] ping pod: kube-ovn-pinger-6ftdf 10 .16.0.10, count: 3 , loss count 0 , average rtt 16 .30ms I0603 10 :35:05.354135 17619 ping.go:159 ] ping pod: kube-ovn-pinger-7twb4 10 .16.63.30, count: 3 , loss count 0 , average rtt 1 .81ms I0603 10 :35:05.458460 17619 ping.go:159 ] ping pod: kube-ovn-pinger-vh2xg 10 .16.0.5, count: 3 , loss count 0 , average rtt 1 .92ms I0603 10 :35:05.458523 17619 ping.go:83 ] start to check node connectivity If the target of diagnose is specified as subnet, the script will create a daemonset on the subnet, and kube-ovn-pinger will detect the connectivity and network delay of all pods in this daemonset, and automatically destroy the daemonset after the test. If the target of diagnose is specified as IPPorts, the script will let each kube-ovn-pinger pod detect whether the target protocol, IP, and Port are reachable. tuning {install-fastpath|local-install-fastpath|remove-fastpath|install-stt|local-install-stt|remove-stt} {centos7|centos8}} [kernel-devel-version] \u00b6 This command performs performance tuning related operations, please refer to Performance Tunning . reload \u00b6 This command restarts all Kube-OVN related components: # kubectl ko reload pod \"ovn-central-8684dd94bd-vzgcr\" deleted Waiting for deployment \"ovn-central\" rollout to finish: 0 of 1 updated replicas are available... deployment \"ovn-central\" successfully rolled out pod \"ovs-ovn-bsnvz\" deleted pod \"ovs-ovn-m9b98\" deleted pod \"kube-ovn-controller-8459db5ff4-64c62\" deleted Waiting for deployment \"kube-ovn-controller\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out pod \"kube-ovn-cni-2klnh\" deleted pod \"kube-ovn-cni-t2jz4\" deleted Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-cni\" successfully rolled out pod \"kube-ovn-pinger-ln72z\" deleted pod \"kube-ovn-pinger-w8lrk\" deleted Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-pinger\" successfully rolled out pod \"kube-ovn-monitor-7fb67d5488-7q6zb\" deleted Waiting for deployment \"kube-ovn-monitor\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-monitor\" successfully rolled out log \u00b6 Using this command will capture the logs of Kube-OVN, OVN, Openvswitch on all nodes of kube-ovn and some debug information commonly used in linux. # kubectl ko log all Collecting kube-ovn logging files Collecting ovn logging files Collecting openvswitch logging files Collecting linux dmesg files Collecting linux iptables-legacy files Collecting linux iptables-nft files Collecting linux route files Collecting linux link files Collecting linux neigh files Collecting linux memory files Collecting linux top files Collecting linux sysctl files Collecting linux netstat files Collecting linux addr files Collecting linux ipset files Collecting linux tcp files Collected files have been saved in the directory /root/kubectl-ko-log The directory is as follows: # tree kubectl-ko-log/ kubectl-ko-log/ | -- kube-ovn-control-plane | | -- kube-ovn | | | -- kube-ovn-cni.log | | | -- kube-ovn-monitor.log | | ` -- kube-ovn-pinger.log | | -- linux | | | -- addr.log | | | -- dmesg.log | | | -- ipset.log | | | -- iptables-legacy.log | | | -- iptables-nft.log | | | -- link.log | | | -- memory.log | | | -- neigh.log | | | -- netstat.log | | | -- route.log | | | -- sysctl.log | | | -- tcp.log | | ` -- top.log | | -- openvswitch | | | -- ovs-vswitchd.log | | ` -- ovsdb-server.log | ` -- ovn | | -- ovn-controller.log | | -- ovn-northd.log | | -- ovsdb-server-nb.log | ` -- ovsdb-server-sb.log perf [image] \u00b6 This command will test some performance indicators of Kube-OVN as follows: The performance indicators of the container network; Hostnetwork network performance indicators; Container network multicast packet performance indicators; Time required for OVN-NB, OVN-SB, and OVN-Northd leader deletion recovery. The parameter image is used to specify the image used by the performance test pod. By default, it is kubeovn/test:v1.12.0 . This parameter is mainly set for offline scenarios, and the image name may change when the image is pulled to the intranet environment. # kubectl ko perf ============================== Prepareing Performance Test Resources =============================== pod/test-client created pod/test-host-client created pod/test-server created pod/test-host-server created service/test-server created pod/test-client condition met pod/test-host-client condition met pod/test-host-server condition met pod/test-server condition met ==================================================================================================== ============================ Start Pod Network Unicast Performance Test ============================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 82 .8 us 97 .7 Mbits/sec 67 .6 us ( 0 % ) 8 .42 Mbits/sec 128 85 .4 us 167 Mbits/sec 67 .2 us ( 0 % ) 17 .2 Mbits/sec 512 85 .8 us 440 Mbits/sec 68 .7 us ( 0 % ) 68 .4 Mbits/sec 1k 85 .1 us 567 Mbits/sec 68 .7 us ( 0 % ) 134 Mbits/sec 4k 138 us 826 Mbits/sec 78 .1 us ( 1 .4% ) 503 Mbits/sec ==================================================================================================== =============================== Start Host Network Performance Test ================================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 49 .7 us 120 Mbits/sec 37 .9 us ( 0 % ) 18 .6 Mbits/sec 128 49 .7 us 200 Mbits/sec 38 .1 us ( 0 % ) 35 .5 Mbits/sec 512 51 .9 us 588 Mbits/sec 38 .9 us ( 0 % ) 142 Mbits/sec 1k 51 .7 us 944 Mbits/sec 37 .2 us ( 0 % ) 279 Mbits/sec 4k 74 .9 us 1 .66 Gbits/sec 39 .9 us ( 0 % ) 1 .20 Gbits/sec ==================================================================================================== ============================== Start Service Network Performance Test ============================== Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 111 us 96 .3 Mbits/sec 88 .4 us ( 0 % ) 7 .59 Mbits/sec 128 83 .7 us 150 Mbits/sec 69 .2 us ( 0 % ) 16 .9 Mbits/sec 512 87 .4 us 374 Mbits/sec 75 .8 us ( 0 % ) 60 .9 Mbits/sec 1k 88 .2 us 521 Mbits/sec 73 .1 us ( 0 % ) 123 Mbits/sec 4k 148 us 813 Mbits/sec 77 .6 us ( 0 .0044% ) 451 Mbits/sec ==================================================================================================== =========================== Start Pod Multicast Network Performance Test =========================== Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .014 ms ( 0 .17% ) 5 .80 Mbits/sec 128 0 .012 ms ( 0 % ) 11 .4 Mbits/sec 512 0 .016 ms ( 0 % ) 46 .1 Mbits/sec 1k 0 .023 ms ( 0 .073% ) 89 .8 Mbits/sec 4k 0 .035 ms ( 1 .3% ) 126 Mbits/sec ==================================================================================================== ============================= Start Host Multicast Network Performance ============================= Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .007 ms ( 0 % ) 9 .95 Mbits/sec 128 0 .005 ms ( 0 % ) 21 .8 Mbits/sec 512 0 .008 ms ( 0 % ) 86 .8 Mbits/sec 1k 0 .013 ms ( 0 .045% ) 168 Mbits/sec 4k 0 .010 ms ( 0 .31% ) 242 Mbits/sec ==================================================================================================== ================================== Start Leader Recover Time Test ================================== Delete ovn central nb pod pod \"ovn-central-5cb9c67d75-tlz9w\" deleted Waiting for ovn central nb pod running =============================== OVN nb Recovery takes 3 .305236803 s ================================ Delete ovn central sb pod pod \"ovn-central-5cb9c67d75-szx4c\" deleted Waiting for ovn central sb pod running =============================== OVN sb Recovery takes 3 .462698535 s ================================ Delete ovn central northd pod pod \"ovn-central-5cb9c67d75-zqmqv\" deleted Waiting for ovn central northd pod running ============================= OVN northd Recovery takes 2 .691291403 s ============================== ==================================================================================================== ================================= Remove Performance Test Resource ================================= rm -f unicast-test-client.log rm -f unicast-test-host-client.log rm -f unicast-test-client.log kubectl ko nbctl lb-del test-server rm -f multicast-test-server.log kubectl exec ovs-ovn-gxdrf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 kubectl exec ovs-ovn-h57bf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 rm -f multicast-test-host-server.log pod \"test-client\" deleted pod \"test-host-client\" deleted pod \"test-host-server\" deleted pod \"test-server\" deleted service \"test-server\" deleted ==================================================================================================== \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kubectl Plugin"},{"location":"en/ops/kubectl-ko/#kubectl-plugin","text":"To facilitate daily operations and maintenance, Kube-OVN provides the kubectl plug-in tool, which allows administrators to perform daily operations through this command. For examples: Check OVN database information and status, OVN database backup and restore, OVS related information, tcpdump specific containers, specific link logical topology, network problem diagnosis and performance optimization.","title":"Kubectl Plugin"},{"location":"en/ops/kubectl-ko/#plugin-installation","text":"Kube-OVN installation will deploy the plugin to each node by default. If the machine that runs kubectl is not in the cluster, or if you need to reinstall the plugin, please refer to the following steps: Download kubectl-ko file: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/kubectl-ko Move file to $PATH : mv kubectl-ko /usr/local/bin/kubectl-ko Add executable permissions: chmod +x /usr/local/bin/kubectl-ko Check if the plugin works properly: # kubectl plugin list The following compatible plugins are available: /usr/local/bin/kubectl-ko","title":"Plugin Installation"},{"location":"en/ops/kubectl-ko/#plugin-usage","text":"Running kubectl ko will show all the available commands and usage descriptions, as follows: # kubectl ko kubectl ko { subcommand } [ option... ] Available Subcommands: [ nb | sb ] [ status | kick | backup | dbstatus | restore ] ovn-db operations show cluster status, kick stale server, backup database, get db consistency status or restore ovn nb db when met 'inconsistent data' error nbctl [ ovn-nbctl options ... ] invoke ovn-nbctl sbctl [ ovn-sbctl options ... ] invoke ovn-sbctl vsctl { nodeName } [ ovs-vsctl options ... ] invoke ovs-vsctl on the specified node ofctl { nodeName } [ ovs-ofctl options ... ] invoke ovs-ofctl on the specified node dpctl { nodeName } [ ovs-dpctl options ... ] invoke ovs-dpctl on the specified node appctl { nodeName } [ ovs-appctl options ... ] invoke ovs-appctl on the specified node tcpdump { namespace/podname } [ tcpdump options ... ] capture pod traffic trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] trace ICMP/TCP/UDP trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } trace ARP request/reply echo \" diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}] diagnose connectivity of all nodes or a specific node or specify subnet's ds pod or IPPorts like 'tcp-172.18.0.2-53,udp-172.18.0.3-53'\" tuning { install-fastpath | local-install-fastpath | remove-fastpath | install-stt | local-install-stt | remove-stt } { centos7 | centos8 }} [ kernel-devel-version ] deploy kernel optimisation components to the system reload restart all kube-ovn components log { kube-ovn | ovn | ovs | linux | all } save log to ./kubectl-ko-log/ perf [ image ] performance test default image is kubeovn/test:v1.12.0 The specific functions and usage of each command are described below.","title":"Plugin Usage"},{"location":"en/ops/kubectl-ko/#nb-sb-status-kick-backup-dbstatus-restore","text":"This subcommand mainly operates on OVN northbound or southbound databases, including database cluster status check, database node offline, database backup, database storage status check and database repair.","title":"[nb | sb] [status | kick | backup | dbstatus | restore]"},{"location":"en/ops/kubectl-ko/#db-cluster-status-check","text":"This command executes ovs-appctl cluster/status on the leader node of the corresponding OVN database to show the cluster status: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 280309 ms ago, reason: timeout Last Election won: 280309 ms ago Election timer: 5000 Log: [ 139 , 139 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-8723 ->8723 <-85d6 ->85d6 Disconnections: 0 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 8723 ( 8723 at tcp: [ 172 .18.0.3 ] :6643 ) next_index = 139 match_index = 138 last msg 763 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 138 status: ok If the match_index under Server has a large difference and the last msg time is long, the corresponding Server may not respond for a long time and needs to be checked further.","title":"DB Cluster Status Check"},{"location":"en/ops/kubectl-ko/#db-nodes-offline","text":"This command removes a node from the OVN database and is required when a node is taken offline or replaced. The following is an example of the cluster status from the previous command, to offline the 172.18.0.3 node: # kubectl ko nb kick 8723 started removal Check the database cluster status again to confirm that the node has been removed: # kubectl ko nb status 306b Name: OVN_Northbound Cluster ID: 9a87 ( 9a872522-3e7d-47ca-83a3-d74333e1a7ca ) Server ID: 306b ( 306b256b-b5e1-4eb0-be91-4ca96adf6bad ) Address: tcp: [ 172 .18.0.2 ] :6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election started 324356 ms ago, reason: timeout Last Election won: 324356 ms ago Election timer: 5000 Log: [ 140 , 140 ] Entries not yet committed: 0 Entries not yet applied: 0 Connections: <-85d6 ->85d6 Disconnections: 2 Servers: 85d6 ( 85d6 at tcp: [ 172 .18.0.4 ] :6643 ) next_index = 140 match_index = 139 last msg 848 ms ago 306b ( 306b at tcp: [ 172 .18.0.2 ] :6643 ) ( self ) next_index = 2 match_index = 139 status: ok","title":"DB Nodes Offline"},{"location":"en/ops/kubectl-ko/#db-backup","text":"This subcommand backs up the current OVN database locally and can be used for disaster recovery: # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup","title":"DB Backup"},{"location":"en/ops/kubectl-ko/#database-storage-status-check","text":"This command is used to check if the database file is corrupt: # kubectl ko nb dbstatus status: ok If error happens, inconsistent data is displayed and needs to be fixed with the following command.","title":"Database Storage Status Check"},{"location":"en/ops/kubectl-ko/#database-repair","text":"If the database status goes to inconsistent data , this command can be used to repair: # kubectl ko nb restore deployment.apps/ovn-central scaled ovn-central original replicas is 3 first nodeIP is 172 .18.0.5 ovs-ovn pod on node 172 .18.0.5 is ovs-ovn-8jxv9 ovs-ovn pod on node 172 .18.0.3 is ovs-ovn-sjzb6 ovs-ovn pod on node 172 .18.0.4 is ovs-ovn-t87zk backup nb db file restore nb db file, operate in pod ovs-ovn-8jxv9 deployment.apps/ovn-central scaled finish restore nb db file and ovn-central replicas recreate ovs-ovn pods pod \"ovs-ovn-8jxv9\" deleted pod \"ovs-ovn-sjzb6\" deleted pod \"ovs-ovn-t87zk\" deleted","title":"Database Repair"},{"location":"en/ops/kubectl-ko/#nbctl-sbctl-options","text":"This subcommand executes the ovn-nbctl and ovn-sbctl commands directly into the leader node of the OVN northbound or southbound database. For more detailed usage of this command, please refer to the official documentation of the upstream OVN ovn-nbctl(8) \u548c ovn-sbctl(8) \u3002 # kubectl ko nbctl show switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port node-liumengxin-ovn2-192.168.137.177 addresses: [ \"00:00:00:D9:58:B8 100.64.0.3\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port alertmanager-main-0.monitoring addresses: [ \"00:00:00:6C:DF:A3 10.16.0.19\" ] port kube-state-metrics-5d6885d89-4nf8h.monitoring addresses: [ \"00:00:00:6F:02:1C 10.16.0.15\" ] port fake-kubelet-67c55dfd89-pv86k.kube-system addresses: [ \"00:00:00:5C:12:E8 10.16.19.177\" ] port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ]","title":"[nbctl | sbctl] [options ...]"},{"location":"en/ops/kubectl-ko/#vsctl-nodename-options","text":"This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-vsctl command to query and configure vswitchd . For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-vsctl(8) \u3002 # kubectl ko vsctl kube-ovn-01 show 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port mirror0 Interface mirror0 type: internal Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port br-int Interface br-int type: internal ovs_version: \"2.17.2\"","title":"vsctl {nodeName} [options ...]"},{"location":"en/ops/kubectl-ko/#ofctl-nodename-options","text":"This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-ofctl command to query or manage OpenFlow. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-ofctl(8) \u3002 # kubectl ko ofctl kube-ovn-01 dump-flows br-int NXST_FLOW reply ( xid = 0x4 ) : flags =[ more ] cookie = 0xcf3429e6, duration = 671791 .432s, table = 0 , n_packets = 0 , n_bytes = 0 , idle_age = 65534 , hard_age = 65534 , priority = 100 ,in_port = 2 actions = load:0x4->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x1->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xc91413c6, duration = 671791 .431s, table = 0 , n_packets = 907489 , n_bytes = 99978275 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 7 actions = load:0x1->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x4->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0xf180459, duration = 671791 .431s, table = 0 , n_packets = 17348582 , n_bytes = 2667811214 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 6317 actions = load:0xa->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x9->NXM_NX_REG14 [] ,resubmit ( ,8 ) cookie = 0x7806dd90, duration = 671791 .431s, table = 0 , n_packets = 3235428 , n_bytes = 833821312 , idle_age = 0 , hard_age = 65534 , priority = 100 ,in_port = 1 actions = load:0xd->NXM_NX_REG13 [] ,load:0x9->NXM_NX_REG11 [] ,load:0xb->NXM_NX_REG12 [] ,load:0x4->OXM_OF_METADATA [] ,load:0x3->NXM_NX_REG14 [] ,resubmit ( ,8 ) ...","title":"ofctl {nodeName} [options ...]"},{"location":"en/ops/kubectl-ko/#dpctl-nodename-options","text":"This command will go to the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-dpctl command to query or manage the OVS datapath. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-dpctl(8) \u3002 # kubectl ko dpctl kube-ovn-01 show system@ovs-system: lookups: hit:350805055 missed:21983648 lost:73 flows: 105 masks: hit:1970748791 total:22 hit/pkt:5.29 port 0 : ovs-system ( internal ) port 1 : ovn0 ( internal ) port 2 : mirror0 ( internal ) port 3 : br-int ( internal ) port 4 : stt_sys_7471 ( stt: packet_type = ptap ) port 5 : eeb4d9e51b5d_h port 6 : a2c1a8a8b83a_h port 7 : 4fa5c4cbb1a5_h","title":"dpctl {nodeName} [options ...]"},{"location":"en/ops/kubectl-ko/#appctl-nodename-options","text":"This command will enter the ovs-ovn container on the corresponding nodeName and execute the corresponding ovs-appctl command to operate the associated daemon process. For more detailed usage of this command, please refer to the official documentation of the upstream OVS ovs-appctl(8) \u3002 # kubectl ko appctl kube-ovn-01 vlog/list console syslog file ------- ------ ------ backtrace OFF ERR INFO bfd OFF ERR INFO bond OFF ERR INFO bridge OFF ERR INFO bundle OFF ERR INFO bundles OFF ERR INFO ...","title":"appctl {nodeName} [options ...]"},{"location":"en/ops/kubectl-ko/#tcpdump-namespacepodname-tcpdump-options","text":"This command will enter the kube-ovn-cni container on the machine where namespace/podname is located, and run tcpdump to capture the traffic on the veth NIC of the corresponding container, which can be used to troubleshoot network-related problems. # kubectl ko tcpdump default/ds1-l6n7p icmp + kubectl exec -it kube-ovn-cni-wlg4s -n kube-ovn -- tcpdump -nn -i d7176fe7b4e0_h icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on d7176fe7b4e0_h, link-type EN10MB ( Ethernet ) , capture size 262144 bytes 06 :52:36.619688 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 1 , length 64 06 :52:36.619746 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 1 , length 64 06 :52:37.619588 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 2 , length 64 06 :52:37.619630 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 2 , length 64 06 :52:38.619933 IP 100 .64.0.3 > 10 .16.0.4: ICMP echo request, id 2 , seq 3 , length 64 06 :52:38.619973 IP 10 .16.0.4 > 100 .64.0.3: ICMP echo reply, id 2 , seq 3 , length 64","title":"tcpdump {namespace/podname} [tcpdump options ...]"},{"location":"en/ops/kubectl-ko/#trace-arguments","text":"This command will print the OVN logical flow table and the final Openflow flow table when the Pod/node accesses an address through a specific protocol, so that it make locate flow table related problems during development or troubleshooting much easy. Supported commands: kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { namespace/podname } { target ip address } [ target mac address ] arp { request | reply } kubectl ko trace { node//nodename } { target ip address } [ target mac address ] { icmp | tcp | udp } [ target tcp/udp port ] kubectl ko trace { node//nodename } { target ip address } [ target mac address ] arp { request | reply } Example: # kubectl ko trace default/ds1-l6n7p 8.8.8.8 icmp + kubectl exec ovn-central-5bc494cb5-np9hm -n kube-ovn -- ovn-trace --ct = new ovn-default 'inport == \"ds1-l6n7p.default\" && ip.ttl == 64 && icmp && eth.src == 0a:00:00:10:00:05 && ip4.src == 10.16.0.4 && eth.dst == 00:00:00:B8:CA:43 && ip4.dst == 8.8.8.8' # icmp,reg14=0xf,vlan_tci=0x0000,dl_src=0a:00:00:10:00:05,dl_dst=00:00:00:b8:ca:43,nw_src=10.16.0.4,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 ingress ( dp = \"ovn-default\" , inport = \"ds1-l6n7p.default\" ) ----------------------------------------------------- 0 . ls_in_port_sec_l2 ( ovn-northd.c:4143 ) : inport == \"ds1-l6n7p.default\" && eth.src == { 0a:00:00:10:00:05 } , priority 50 , uuid 39453393 next ; 1 . ls_in_port_sec_ip ( ovn-northd.c:2898 ) : inport == \"ds1-l6n7p.default\" && eth.src == 0a:00:00:10:00:05 && ip4.src == { 10 .16.0.4 } , priority 90 , uuid 81bcd485 next ; 3 . ls_in_pre_acl ( ovn-northd.c:3269 ) : ip, priority 100 , uuid 7b4f4971 reg0 [ 0 ] = 1 ; next ; 5 . ls_in_pre_stateful ( ovn-northd.c:3396 ) : reg0 [ 0 ] == 1 , priority 100 , uuid 36cdd577 ct_next ; ct_next ( ct_state = new | trk ) ------------------------- 6 . ls_in_acl ( ovn-northd.c:3759 ) : ip && ( !ct.est || ( ct.est && ct_label.blocked == 1 )) , priority 1 , uuid 7608af5b reg0 [ 1 ] = 1 ; next ; 10 . ls_in_stateful ( ovn-northd.c:3995 ) : reg0 [ 1 ] == 1 , priority 100 , uuid 2aba1b90 ct_commit ( ct_label = 0 /0x1 ) ; next ; 16 . ls_in_l2_lkup ( ovn-northd.c:4470 ) : eth.dst == 00 :00:00:b8:ca:43, priority 50 , uuid 5c9c3c9f outport = \"ovn-default-ovn-cluster\" ; output ; ... If the trace object is a virtual machine running in Underlay network, additional parameters is needed to specify the destination Mac address. kubectl ko trace default/virt-handler-7lvml 8 .8.8.8 82 :7c:9f:83:8c:01 icmp","title":"trace [arguments ...]"},{"location":"en/ops/kubectl-ko/#diagnose-allnodesubnetipports-nodenamesubnetnameproto1-ip1-port1proto2-ip2-port2","text":"Diagnose the status of cluster network components and go to the corresponding node's kube-ovn-pinger to detect connectivity and network latency from the current node to other nodes and critical services. # kubectl ko diagnose all switch c7cd17e8-ceee-4a91-9bb3-e5a313fe1ece ( snat ) port snat-ovn-cluster type: router router-port: ovn-cluster-snat switch 20e0c6d0-023a-4756-aec5-200e0c60f95d ( join ) port node-liumengxin-ovn3-192.168.137.178 addresses: [ \"00:00:00:64:FF:A8 100.64.0.4\" ] port node-liumengxin-ovn1-192.168.137.176 addresses: [ \"00:00:00:AF:98:62 100.64.0.2\" ] port join-ovn-cluster type: router router-port: ovn-cluster-join switch 0191705c-f827-427b-9de3-3c3b7d971ba5 ( central ) port central-ovn-cluster type: router router-port: ovn-cluster-central switch 2a45ff05-388d-4f85-9daf-e6fccd5833dc ( ovn-default ) port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default port prometheus-k8s-1.monitoring addresses: [ \"00:00:00:AA:37:DF 10.16.0.23\" ] router 212f73dd-d63d-4d72-864b-a537e9afbee1 ( ovn-cluster ) port ovn-cluster-snat mac: \"00:00:00:7A:82:8F\" networks: [ \"172.22.0.1/16\" ] port ovn-cluster-join mac: \"00:00:00:F8:18:5A\" networks: [ \"100.64.0.1/16\" ] port ovn-cluster-central mac: \"00:00:00:4D:8C:F5\" networks: [ \"192.101.0.1/16\" ] port ovn-cluster-ovn-default mac: \"00:00:00:A3:F8:18\" networks: [ \"10.16.0.1/16\" ] Routing Policies 31000 ip4.dst == 10 .16.0.0/16 allow 31000 ip4.dst == 100 .64.0.0/16 allow 30000 ip4.dst == 192 .168.137.177 reroute 100 .64.0.3 30000 ip4.dst == 192 .168.137.178 reroute 100 .64.0.4 29000 ip4.src == $ovn .default.fake.6_ip4 reroute 100 .64.0.22 29000 ip4.src == $ovn .default.fake.7_ip4 reroute 100 .64.0.21 29000 ip4.src == $ovn .default.fake.8_ip4 reroute 100 .64.0.23 29000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 reroute 100 .64.0.4 20000 ip4.src == $ovn .default.liumengxin.ovn1.192.168.137.176_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.2 20000 ip4.src == $ovn .default.liumengxin.ovn2.192.168.137.177_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.3 20000 ip4.src == $ovn .default.liumengxin.ovn3.192.168.137.178_ip4 && ip4.dst ! = $ovn .cluster.overlay.subnets.IPv4 reroute 100 .64.0.4 IPv4 Routes Route Table <main>: 0 .0.0.0/0 100 .64.0.1 dst-ip UUID LB PROTO VIP IPs e9bcfd9d-793e-4431-9073-6dec96b75d71 cluster-tcp-load tcp 10 .100.209.132:10660 192 .168.137.176:10660 tcp 10 .101.239.192:6641 192 .168.137.177:6641 tcp 10 .101.240.101:3000 10 .16.0.7:3000 tcp 10 .103.184.186:6642 192 .168.137.177:6642 35d2b7a5-e3a7-485a-a4b7-b4970eb0e63b cluster-tcp-sess tcp 10 .100.158.128:8080 10 .16.0.10:8080,10.16.0.5:8080,10.16.63.30:8080 tcp 10 .107.26.215:8080 10 .16.0.19:8080,10.16.0.20:8080,10.16.0.21:8080 tcp 10 .107.26.215:9093 10 .16.0.19:9093,10.16.0.20:9093,10.16.0.21:9093 tcp 10 .98.187.99:8080 10 .16.0.22:8080,10.16.0.23:8080 tcp 10 .98.187.99:9090 10 .16.0.22:9090,10.16.0.23:9090 f43303e4-89aa-4d3e-a3dc-278a552fe27b cluster-udp-load udp 10 .96.0.10:53 10 .16.0.4:53,10.16.0.9:53 _uuid : 06776304 -5a96-43ed-90c4-c4854c251699 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn2_192.168.137.177_underlay_v6 _uuid : 62690625 -87d5-491c-8675-9fd83b1f433c addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn1_192.168.137.176_underlay_v6 _uuid : b03a9bae-94d5-4562-b34c-b5f6198e180b addresses : [ \"10.16.0.0/16\" , \"100.64.0.0/16\" , \"172.22.0.0/16\" , \"192.101.0.0/16\" ] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv4 _uuid : e1056f3a-24cc-4666-8a91-75ee6c3c2426 addresses : [] external_ids : { vendor = kube-ovn } name : ovn.cluster.overlay.subnets.IPv6 _uuid : 3e5d5fff-e670-47b2-a2f5-a39f4698a8c5 addresses : [] external_ids : { vendor = kube-ovn } name : node_liumengxin_ovn3_192.168.137.178_underlay_v6 _uuid : 2d85dbdc-d0db-4abe-b19e-cc806d32b492 action : drop direction : from-lport external_ids : {} label : 0 log : false match : \"inport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] _uuid : de790cc8-f155-405f-bb32-5a51f30c545f action : drop direction : to-lport external_ids : {} label : 0 log : false match : \"outport==@ovn.sg.kubeovn_deny_all && ip\" meter : [] name : [] options : {} priority : 2003 severity : [] Chassis \"e15ed4d4-1780-4d50-b09e-ea8372ed48b8\" hostname: liumengxin-ovn1-192.168.137.176 Encap stt ip: \"192.168.137.176\" options: { csum = \"true\" } Port_Binding node-liumengxin-ovn1-192.168.137.176 Port_Binding perf-6vxkn.default Port_Binding kube-state-metrics-5d6885d89-4nf8h.monitoring Port_Binding alertmanager-main-0.monitoring Port_Binding kube-ovn-pinger-6ftdf.kube-system Port_Binding fake-kubelet-67c55dfd89-pv86k.kube-system Port_Binding prometheus-k8s-0.monitoring Chassis \"eef07da1-f8ad-4775-b14d-bd6a3b4eb0d5\" hostname: liumengxin-ovn3-192.168.137.178 Encap stt ip: \"192.168.137.178\" options: { csum = \"true\" } Port_Binding kube-ovn-pinger-7twb4.kube-system Port_Binding prometheus-adapter-86df476d87-rl88g.monitoring Port_Binding prometheus-k8s-1.monitoring Port_Binding node-liumengxin-ovn3-192.168.137.178 Port_Binding perf-ff475.default Port_Binding alertmanager-main-1.monitoring Port_Binding blackbox-exporter-676d976865-tvsjd.monitoring Chassis \"efa253c9-494d-4719-83ae-b48ab0f11c03\" hostname: liumengxin-ovn2-192.168.137.177 Encap stt ip: \"192.168.137.177\" options: { csum = \"true\" } Port_Binding grafana-6c4c6b8fb7-pzd2c.monitoring Port_Binding node-liumengxin-ovn2-192.168.137.177 Port_Binding alertmanager-main-2.monitoring Port_Binding coredns-6789c94dd8-9jqsz.kube-system Port_Binding coredns-6789c94dd8-25d4r.kube-system Port_Binding prometheus-operator-7bbc99fc8b-wgjm4.monitoring Port_Binding prometheus-adapter-86df476d87-gdxmc.monitoring Port_Binding perf-fjnws.default Port_Binding kube-ovn-pinger-vh2xg.kube-system ds kube-proxy ready kube-proxy ready deployment ovn-central ready deployment kube-ovn-controller ready ds kube-ovn-cni ready ds ovs-ovn ready deployment coredns ready ovn-nb leader check ok ovn-sb leader check ok ovn-northd leader check ok ### kube-ovn-controller recent log ### start to diagnose node liumengxin-ovn1-192.168.137.176 #### ovn-controller log: 2022 -06-03T00:56:44.897Z | 16722 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:06:44.912Z | 16723 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:16:44.925Z | 16724 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:26:44.936Z | 16725 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:36:44.959Z | 16726 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:46:44.974Z | 16727 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T01:56:44.988Z | 16728 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:06:45.001Z | 16729 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:16:45.025Z | 16730 | inc_proc_eng | INFO | User triggered force recompute. 2022 -06-03T02:26:45.040Z | 16731 | inc_proc_eng | INFO | User triggered force recompute. #### ovs-vswitchd log: 2022 -06-02T23:03:00.137Z | 00079 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:f9d1 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-02T23:23:31.840Z | 00080 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:15b2 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:09:15.659Z | 00081 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:dc:e3:63,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.63.30,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:e5a5 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x150017000004,src = 192 .168.137.178,dst = 192 .168.137.176,ttl = 64 ,tp_src = 9239 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.63.30,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T00:30:13.409Z | 00064 | dpif ( handler2 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:6b4a with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 2022 -06-03T02:02:33.832Z | 00082 | dpif ( handler1 ) | WARN | system@ovs-system: execute ct ( commit,zone = 14 ,label = 0 /0x1,nat ( src )) ,8 failed ( Invalid argument ) on packet icmp,vlan_tci = 0x0000,dl_src = 00 :00:00:f8:07:c8,dl_dst = 00 :00:00:fa:1e:50,nw_src = 10 .16.0.5,nw_dst = 10 .16.0.10,nw_tos = 0 ,nw_ecn = 0 ,nw_ttl = 64 ,icmp_type = 8 ,icmp_code = 0 icmp_csum:a819 with metadata skb_priority ( 0 ) ,tunnel ( tun_id = 0x160017000004,src = 192 .168.137.177,dst = 192 .168.137.176,ttl = 64 ,tp_src = 38881 ,tp_dst = 7471 ,flags ( csum | key )) ,skb_mark ( 0 ) ,ct_state ( 0x21 ) ,ct_zone ( 0xe ) ,ct_tuple4 ( src = 10 .16.0.5,dst = 10 .16.0.10,proto = 1 ,tp_src = 8 ,tp_dst = 0 ) ,in_port ( 4 ) mtu 0 #### ovs-vsctl show results: 0d4c4675-c9cc-440a-8c1a-878e17f81b88 Bridge br-int fail_mode: secure datapath_type: system Port a2c1a8a8b83a_h Interface a2c1a8a8b83a_h Port \"4fa5c4cbb1a5_h\" Interface \"4fa5c4cbb1a5_h\" Port ovn-eef07d-0 Interface ovn-eef07d-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.178\" } Port ovn0 Interface ovn0 type: internal Port \"04d03360e9a0_h\" Interface \"04d03360e9a0_h\" Port eeb4d9e51b5d_h Interface eeb4d9e51b5d_h Port mirror0 Interface mirror0 type: internal Port \"8e5d887ccd80_h\" Interface \"8e5d887ccd80_h\" Port ovn-efa253-0 Interface ovn-efa253-0 type: stt options: { csum = \"true\" , key = flow, remote_ip = \"192.168.137.177\" } Port \"17512d5be1f1_h\" Interface \"17512d5be1f1_h\" Port br-int Interface br-int type: internal ovs_version: \"2.17.2\" #### pinger diagnose results: I0603 10 :35:04.349404 17619 pinger.go:19 ] ------------------------------------------------------------------------------- Kube-OVN: Version: v1.12.0 Build: 2022 -04-24_08:02:50 Commit: git-73f9d15 Go Version: go1.17.8 Arch: amd64 ------------------------------------------------------------------------------- I0603 10 :35:04.376797 17619 config.go:166 ] pinger config is & { KubeConfigFile: KubeClient:0xc000493380 Port:8080 DaemonSetNamespace:kube-system DaemonSetName:kube-ovn-pinger Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:liumengxin-ovn1-192.168.137.176 HostIP:192.168.137.176 PodName:kube-ovn-pinger-6ftdf PodIP:10.16.0.10 PodProtocols: [ IPv4 ] ExternalAddress: NetworkMode:kube-ovn PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid } I0603 10 :35:04.449166 17619 exporter.go:75 ] liumengxin-ovn1-192.168.137.176: exporter connect successfully I0603 10 :35:04.554011 17619 ovn.go:21 ] ovs-vswitchd and ovsdb are up I0603 10 :35:04.651293 17619 ovn.go:33 ] ovn_controller is up I0603 10 :35:04.651342 17619 ovn.go:39 ] start to check port binding I0603 10 :35:04.749613 17619 ovn.go:135 ] chassis id is 1d7f3d6c-eec5-4b3c-adca-2969d9cdfd80 I0603 10 :35:04.763487 17619 ovn.go:49 ] port in sb is [ node-liumengxin-ovn1-192.168.137.176 perf-6vxkn.default kube-state-metrics-5d6885d89-4nf8h.monitoring alertmanager-main-0.monitoring kube-ovn-pinger-6ftdf.kube-system fake-kubelet-67c55dfd89-pv86k.kube-system prometheus-k8s-0.monitoring ] I0603 10 :35:04.763583 17619 ovn.go:61 ] ovs and ovn-sb binding check passed I0603 10 :35:05.049309 17619 ping.go:259 ] start to check apiserver connectivity I0603 10 :35:05.053666 17619 ping.go:268 ] connect to apiserver success in 4 .27ms I0603 10 :35:05.053786 17619 ping.go:129 ] start to check pod connectivity I0603 10 :35:05.249590 17619 ping.go:159 ] ping pod: kube-ovn-pinger-6ftdf 10 .16.0.10, count: 3 , loss count 0 , average rtt 16 .30ms I0603 10 :35:05.354135 17619 ping.go:159 ] ping pod: kube-ovn-pinger-7twb4 10 .16.63.30, count: 3 , loss count 0 , average rtt 1 .81ms I0603 10 :35:05.458460 17619 ping.go:159 ] ping pod: kube-ovn-pinger-vh2xg 10 .16.0.5, count: 3 , loss count 0 , average rtt 1 .92ms I0603 10 :35:05.458523 17619 ping.go:83 ] start to check node connectivity If the target of diagnose is specified as subnet, the script will create a daemonset on the subnet, and kube-ovn-pinger will detect the connectivity and network delay of all pods in this daemonset, and automatically destroy the daemonset after the test. If the target of diagnose is specified as IPPorts, the script will let each kube-ovn-pinger pod detect whether the target protocol, IP, and Port are reachable.","title":"diagnose {all|node|subnet|IPPorts} [nodename|subnetName|{proto1}-{IP1}-{Port1},{proto2}-{IP2}-{Port2}]"},{"location":"en/ops/kubectl-ko/#tuning-install-fastpathlocal-install-fastpathremove-fastpathinstall-sttlocal-install-sttremove-stt-centos7centos8-kernel-devel-version","text":"This command performs performance tuning related operations, please refer to Performance Tunning .","title":"tuning {install-fastpath|local-install-fastpath|remove-fastpath|install-stt|local-install-stt|remove-stt} {centos7|centos8}} [kernel-devel-version]"},{"location":"en/ops/kubectl-ko/#reload","text":"This command restarts all Kube-OVN related components: # kubectl ko reload pod \"ovn-central-8684dd94bd-vzgcr\" deleted Waiting for deployment \"ovn-central\" rollout to finish: 0 of 1 updated replicas are available... deployment \"ovn-central\" successfully rolled out pod \"ovs-ovn-bsnvz\" deleted pod \"ovs-ovn-m9b98\" deleted pod \"kube-ovn-controller-8459db5ff4-64c62\" deleted Waiting for deployment \"kube-ovn-controller\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-controller\" successfully rolled out pod \"kube-ovn-cni-2klnh\" deleted pod \"kube-ovn-cni-t2jz4\" deleted Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-cni\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-cni\" successfully rolled out pod \"kube-ovn-pinger-ln72z\" deleted pod \"kube-ovn-pinger-w8lrk\" deleted Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 0 of 2 updated pods are available... Waiting for daemon set \"kube-ovn-pinger\" rollout to finish: 1 of 2 updated pods are available... daemon set \"kube-ovn-pinger\" successfully rolled out pod \"kube-ovn-monitor-7fb67d5488-7q6zb\" deleted Waiting for deployment \"kube-ovn-monitor\" rollout to finish: 0 of 1 updated replicas are available... deployment \"kube-ovn-monitor\" successfully rolled out","title":"reload"},{"location":"en/ops/kubectl-ko/#log","text":"Using this command will capture the logs of Kube-OVN, OVN, Openvswitch on all nodes of kube-ovn and some debug information commonly used in linux. # kubectl ko log all Collecting kube-ovn logging files Collecting ovn logging files Collecting openvswitch logging files Collecting linux dmesg files Collecting linux iptables-legacy files Collecting linux iptables-nft files Collecting linux route files Collecting linux link files Collecting linux neigh files Collecting linux memory files Collecting linux top files Collecting linux sysctl files Collecting linux netstat files Collecting linux addr files Collecting linux ipset files Collecting linux tcp files Collected files have been saved in the directory /root/kubectl-ko-log The directory is as follows: # tree kubectl-ko-log/ kubectl-ko-log/ | -- kube-ovn-control-plane | | -- kube-ovn | | | -- kube-ovn-cni.log | | | -- kube-ovn-monitor.log | | ` -- kube-ovn-pinger.log | | -- linux | | | -- addr.log | | | -- dmesg.log | | | -- ipset.log | | | -- iptables-legacy.log | | | -- iptables-nft.log | | | -- link.log | | | -- memory.log | | | -- neigh.log | | | -- netstat.log | | | -- route.log | | | -- sysctl.log | | | -- tcp.log | | ` -- top.log | | -- openvswitch | | | -- ovs-vswitchd.log | | ` -- ovsdb-server.log | ` -- ovn | | -- ovn-controller.log | | -- ovn-northd.log | | -- ovsdb-server-nb.log | ` -- ovsdb-server-sb.log","title":"log"},{"location":"en/ops/kubectl-ko/#perf-image","text":"This command will test some performance indicators of Kube-OVN as follows: The performance indicators of the container network; Hostnetwork network performance indicators; Container network multicast packet performance indicators; Time required for OVN-NB, OVN-SB, and OVN-Northd leader deletion recovery. The parameter image is used to specify the image used by the performance test pod. By default, it is kubeovn/test:v1.12.0 . This parameter is mainly set for offline scenarios, and the image name may change when the image is pulled to the intranet environment. # kubectl ko perf ============================== Prepareing Performance Test Resources =============================== pod/test-client created pod/test-host-client created pod/test-server created pod/test-host-server created service/test-server created pod/test-client condition met pod/test-host-client condition met pod/test-host-server condition met pod/test-server condition met ==================================================================================================== ============================ Start Pod Network Unicast Performance Test ============================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 82 .8 us 97 .7 Mbits/sec 67 .6 us ( 0 % ) 8 .42 Mbits/sec 128 85 .4 us 167 Mbits/sec 67 .2 us ( 0 % ) 17 .2 Mbits/sec 512 85 .8 us 440 Mbits/sec 68 .7 us ( 0 % ) 68 .4 Mbits/sec 1k 85 .1 us 567 Mbits/sec 68 .7 us ( 0 % ) 134 Mbits/sec 4k 138 us 826 Mbits/sec 78 .1 us ( 1 .4% ) 503 Mbits/sec ==================================================================================================== =============================== Start Host Network Performance Test ================================ Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 49 .7 us 120 Mbits/sec 37 .9 us ( 0 % ) 18 .6 Mbits/sec 128 49 .7 us 200 Mbits/sec 38 .1 us ( 0 % ) 35 .5 Mbits/sec 512 51 .9 us 588 Mbits/sec 38 .9 us ( 0 % ) 142 Mbits/sec 1k 51 .7 us 944 Mbits/sec 37 .2 us ( 0 % ) 279 Mbits/sec 4k 74 .9 us 1 .66 Gbits/sec 39 .9 us ( 0 % ) 1 .20 Gbits/sec ==================================================================================================== ============================== Start Service Network Performance Test ============================== Size TCP Latency TCP Bandwidth UDP Latency UDP Lost Rate UDP Bandwidth 64 111 us 96 .3 Mbits/sec 88 .4 us ( 0 % ) 7 .59 Mbits/sec 128 83 .7 us 150 Mbits/sec 69 .2 us ( 0 % ) 16 .9 Mbits/sec 512 87 .4 us 374 Mbits/sec 75 .8 us ( 0 % ) 60 .9 Mbits/sec 1k 88 .2 us 521 Mbits/sec 73 .1 us ( 0 % ) 123 Mbits/sec 4k 148 us 813 Mbits/sec 77 .6 us ( 0 .0044% ) 451 Mbits/sec ==================================================================================================== =========================== Start Pod Multicast Network Performance Test =========================== Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .014 ms ( 0 .17% ) 5 .80 Mbits/sec 128 0 .012 ms ( 0 % ) 11 .4 Mbits/sec 512 0 .016 ms ( 0 % ) 46 .1 Mbits/sec 1k 0 .023 ms ( 0 .073% ) 89 .8 Mbits/sec 4k 0 .035 ms ( 1 .3% ) 126 Mbits/sec ==================================================================================================== ============================= Start Host Multicast Network Performance ============================= Size UDP Latency UDP Lost Rate UDP Bandwidth 64 0 .007 ms ( 0 % ) 9 .95 Mbits/sec 128 0 .005 ms ( 0 % ) 21 .8 Mbits/sec 512 0 .008 ms ( 0 % ) 86 .8 Mbits/sec 1k 0 .013 ms ( 0 .045% ) 168 Mbits/sec 4k 0 .010 ms ( 0 .31% ) 242 Mbits/sec ==================================================================================================== ================================== Start Leader Recover Time Test ================================== Delete ovn central nb pod pod \"ovn-central-5cb9c67d75-tlz9w\" deleted Waiting for ovn central nb pod running =============================== OVN nb Recovery takes 3 .305236803 s ================================ Delete ovn central sb pod pod \"ovn-central-5cb9c67d75-szx4c\" deleted Waiting for ovn central sb pod running =============================== OVN sb Recovery takes 3 .462698535 s ================================ Delete ovn central northd pod pod \"ovn-central-5cb9c67d75-zqmqv\" deleted Waiting for ovn central northd pod running ============================= OVN northd Recovery takes 2 .691291403 s ============================== ==================================================================================================== ================================= Remove Performance Test Resource ================================= rm -f unicast-test-client.log rm -f unicast-test-host-client.log rm -f unicast-test-client.log kubectl ko nbctl lb-del test-server rm -f multicast-test-server.log kubectl exec ovs-ovn-gxdrf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 kubectl exec ovs-ovn-h57bf -n kube-system -- ip maddr del 01 :00:5e:00:00:64 dev eth0 rm -f multicast-test-host-server.log pod \"test-client\" deleted pod \"test-host-client\" deleted pod \"test-host-server\" deleted pod \"test-server\" deleted service \"test-server\" deleted ==================================================================================================== \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"perf [image]"},{"location":"en/ops/recover-db/","text":"OVN DB Backup and Recovery \u00b6 This document describes how to perform database backups and how to perform cluster recovery from existing database files in different situations. Database Backup \u00b6 The database files can be backed up for recovery in case of failure. Use the backup command of the kubectl plugin: # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup # kubectl ko sb backup tar: Removing leading `/' from member names backup ovn-nb db to /root/ovnsb_db.060223191654183154.backup Cluster Partial Nodes Failure Recovery \u00b6 If some nodes in the cluster are working abnormally due to power failure, file system failure or lack of disk space, but the cluster is still working normally, you can recover it by following the steps below. Check the Logs to Confirm Status \u00b6 Check the log in /var/log/ovn/ovn-northd.log , if it shows similar error as follows, you can make sue that there is an exception in the database: * ovn-northd is not running ovsdb-server: ovsdb error: error reading record 2739 from OVN_Northbound log: record 2739 advances commit index to 6308 but last log index is 6307 * Starting ovsdb-nb Kick Node from Cluster \u00b6 Select the corresponding database for the operation based on whether the log prompt is OVN_Northbound or OVN_Southbound . The above log prompt is OVN_Northbound then for ovn-nb do the following: # kubectl ko nb status 9182 Name: OVN_Northbound Cluster ID: e75f ( e75fa340-49ed-45ab-990e-26cb865ebc85 ) Server ID: 9182 ( 9182e8dd-b5b0-4dd8-8518-598cc1e374f3 ) Address: tcp: [ 10 .0.128.61 ] :6643 Status: cluster member Role: leader Term: 1454 Leader: self Vote: self Last Election started 1732603 ms ago, reason: timeout Last Election won: 1732587 ms ago Election timer: 1000 Log: [ 7332 , 12512 ] Entries not yet committed: 1 Entries not yet applied: 1 Connections: ->f080 <-f080 <-e631 ->e631 Disconnections: 1 Servers: f080 ( f080 at tcp: [ 10 .0.129.139 ] :6643 ) next_index = 12512 match_index = 12510 last msg 63 ms ago 9182 ( 9182 at tcp: [ 10 .0.128.61 ] :6643 ) ( self ) next_index = 10394 match_index = 12510 e631 ( e631 at tcp: [ 10 .0.131.173 ] :6643 ) next_index = 12512 match_index = 0 Kick abnormal nodes from the cluster: kubectl ko nb kick e631 Log in to the abnormal node and delete the database file: mv /etc/origin/ovn/ovnnb_db.db /tmp Delete the ovn-central pod of the corresponding node and wait for the cluster to recover\uff1a kubectl delete pod -n kube-system ovn-central-xxxx Recover when Total Cluster Failed \u00b6 If the majority of the cluster nodes are broken and the leader cannot be elected, please refer to the following steps to recover. Stop ovn-central \u00b6 Record the current replicas of ovn-central and stop ovn-central to avoid new database changes that affect recovery: kubectl scale deployment -n kube-system ovn-central --replicas = 0 Select a Backup \u00b6 As most of the nodes are damaged, the cluster needs to be rebuilt by recovering from one of the database files. If you have previously backed up the database you can use the previous backup file to restore it. If not you can use the following steps to generate a backup from an existing file. Since the database file in the default folder is a cluster format database file containing information about the current cluster, you can't rebuild the database directly with this file, you need to use ovsdb-tool cluster-to-standalone to convert the format. Select the first node in the ovn-central environment variable NODE_IPS to restore the database files. If the database file of the first node is corrupted, copy the file from the other machine /etc/origin/ovn to the first machine. Run the following command to generate a database file backup. docker run -it -v /etc/origin/ovn:/etc/ovn kubeovn/kube-ovn:v1.12.0 bash cd /etc/ovn/ ovsdb-tool cluster-to-standalone ovnnb_db_standalone.db ovnnb_db.db ovsdb-tool cluster-to-standalone ovnsb_db_standalone.db ovnsb_db.db Delete the Database Files on All ovn-central Nodes \u00b6 In order to avoid rebuilding the cluster with the wrong data, the existing database files need to be cleaned up: mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp Recovering Database Cluster \u00b6 Rename the backup databases to ovnnb_db.db and ovnsb_db.db respectively, and copy them to the /etc/origin/ovn/ directory of the first machine in the ovn-central environment variable NODE_IPS \uff1a mv /etc/origin/ovn/ovnnb_db_standalone.db /etc/origin/ovn/ovnnb_db.db mv /etc/origin/ovn/ovnsb_db_standalone.db /etc/origin/ovn/ovnsb_db.db Restore the number of replicas of ovn-central \uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl rollout status deployment/ovn-central -n kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVN DB Backup and Recovery"},{"location":"en/ops/recover-db/#ovn-db-backup-and-recovery","text":"This document describes how to perform database backups and how to perform cluster recovery from existing database files in different situations.","title":"OVN DB Backup and Recovery"},{"location":"en/ops/recover-db/#database-backup","text":"The database files can be backed up for recovery in case of failure. Use the backup command of the kubectl plugin: # kubectl ko nb backup tar: Removing leading ` / ' from member names backup ovn-nb db to /root/ovnnb_db.060223191654183154.backup # kubectl ko sb backup tar: Removing leading `/' from member names backup ovn-nb db to /root/ovnsb_db.060223191654183154.backup","title":"Database Backup"},{"location":"en/ops/recover-db/#cluster-partial-nodes-failure-recovery","text":"If some nodes in the cluster are working abnormally due to power failure, file system failure or lack of disk space, but the cluster is still working normally, you can recover it by following the steps below.","title":"Cluster Partial Nodes Failure Recovery"},{"location":"en/ops/recover-db/#check-the-logs-to-confirm-status","text":"Check the log in /var/log/ovn/ovn-northd.log , if it shows similar error as follows, you can make sue that there is an exception in the database: * ovn-northd is not running ovsdb-server: ovsdb error: error reading record 2739 from OVN_Northbound log: record 2739 advances commit index to 6308 but last log index is 6307 * Starting ovsdb-nb","title":"Check the Logs to Confirm Status"},{"location":"en/ops/recover-db/#kick-node-from-cluster","text":"Select the corresponding database for the operation based on whether the log prompt is OVN_Northbound or OVN_Southbound . The above log prompt is OVN_Northbound then for ovn-nb do the following: # kubectl ko nb status 9182 Name: OVN_Northbound Cluster ID: e75f ( e75fa340-49ed-45ab-990e-26cb865ebc85 ) Server ID: 9182 ( 9182e8dd-b5b0-4dd8-8518-598cc1e374f3 ) Address: tcp: [ 10 .0.128.61 ] :6643 Status: cluster member Role: leader Term: 1454 Leader: self Vote: self Last Election started 1732603 ms ago, reason: timeout Last Election won: 1732587 ms ago Election timer: 1000 Log: [ 7332 , 12512 ] Entries not yet committed: 1 Entries not yet applied: 1 Connections: ->f080 <-f080 <-e631 ->e631 Disconnections: 1 Servers: f080 ( f080 at tcp: [ 10 .0.129.139 ] :6643 ) next_index = 12512 match_index = 12510 last msg 63 ms ago 9182 ( 9182 at tcp: [ 10 .0.128.61 ] :6643 ) ( self ) next_index = 10394 match_index = 12510 e631 ( e631 at tcp: [ 10 .0.131.173 ] :6643 ) next_index = 12512 match_index = 0 Kick abnormal nodes from the cluster: kubectl ko nb kick e631 Log in to the abnormal node and delete the database file: mv /etc/origin/ovn/ovnnb_db.db /tmp Delete the ovn-central pod of the corresponding node and wait for the cluster to recover\uff1a kubectl delete pod -n kube-system ovn-central-xxxx","title":"Kick Node from Cluster"},{"location":"en/ops/recover-db/#recover-when-total-cluster-failed","text":"If the majority of the cluster nodes are broken and the leader cannot be elected, please refer to the following steps to recover.","title":"Recover when Total Cluster Failed"},{"location":"en/ops/recover-db/#stop-ovn-central","text":"Record the current replicas of ovn-central and stop ovn-central to avoid new database changes that affect recovery: kubectl scale deployment -n kube-system ovn-central --replicas = 0","title":"Stop ovn-central"},{"location":"en/ops/recover-db/#select-a-backup","text":"As most of the nodes are damaged, the cluster needs to be rebuilt by recovering from one of the database files. If you have previously backed up the database you can use the previous backup file to restore it. If not you can use the following steps to generate a backup from an existing file. Since the database file in the default folder is a cluster format database file containing information about the current cluster, you can't rebuild the database directly with this file, you need to use ovsdb-tool cluster-to-standalone to convert the format. Select the first node in the ovn-central environment variable NODE_IPS to restore the database files. If the database file of the first node is corrupted, copy the file from the other machine /etc/origin/ovn to the first machine. Run the following command to generate a database file backup. docker run -it -v /etc/origin/ovn:/etc/ovn kubeovn/kube-ovn:v1.12.0 bash cd /etc/ovn/ ovsdb-tool cluster-to-standalone ovnnb_db_standalone.db ovnnb_db.db ovsdb-tool cluster-to-standalone ovnsb_db_standalone.db ovnsb_db.db","title":"Select a Backup"},{"location":"en/ops/recover-db/#delete-the-database-files-on-all-ovn-central-nodes","text":"In order to avoid rebuilding the cluster with the wrong data, the existing database files need to be cleaned up: mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp","title":"Delete the Database Files on All ovn-central Nodes"},{"location":"en/ops/recover-db/#recovering-database-cluster","text":"Rename the backup databases to ovnnb_db.db and ovnsb_db.db respectively, and copy them to the /etc/origin/ovn/ directory of the first machine in the ovn-central environment variable NODE_IPS \uff1a mv /etc/origin/ovn/ovnnb_db_standalone.db /etc/origin/ovn/ovnnb_db.db mv /etc/origin/ovn/ovnsb_db_standalone.db /etc/origin/ovn/ovnsb_db.db Restore the number of replicas of ovn-central \uff1a kubectl scale deployment -n kube-system ovn-central --replicas = 3 kubectl rollout status deployment/ovn-central -n kube-system \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Recovering Database Cluster"},{"location":"en/reference/architecture/","text":"Architecture \u00b6 This document describes the general architecture of Kube-OVN, the functionality of each component and how they interact with each other. Overall, Kube-OVN serves as a bridge between Kubernetes and OVN, combining proven SDN with Cloud Native. This means that Kube-OVN not only implements network specifications under Kubernetes, such as CNI, Service and Networkpolicy, but also brings a large number of SDN domain capabilities to cloud-native, such as logical switches, logical routers, VPCs, gateways, QoS, ACLs and traffic mirroring. Kube-OVN also maintains a good openness to integrate with many technology solutions, such as Cilium, Submariner, Prometheus, KubeVirt, etc. Component Introduction \u00b6 The components of Kube-OVN can be broadly divided into three categories. Upstream OVN/OVS components. Core Controller and Agent. Monitoring, operation and maintenance tools and extension components. Upstream OVN/OVS Components \u00b6 This type of component comes from the OVN/OVS community with specific modifications for Kube-OVN usage scenarios. OVN/OVS itself is a mature SDN system for managing virtual machines and containers, and we strongly recommend that users interested in the Kube-OVN implementation read ovn-architecture(7) first to understand what OVN is and how to integrate with it. Kube-OVN uses the northbound interface of OVN to create and coordinate virtual networks and map the network concepts into Kubernetes. All OVN/OVS-related components have been packaged into images and are ready to run in Kubernetes. ovn-central \u00b6 The ovn-central Deployment runs the control plane components of OVN, including ovn-nb , ovn-sb , and ovn-northd . ovn-nb : Saves the virtual network configuration and provides an API for virtual network management. kube-ovn-controller will mainly interact with ovn-nb to configure the virtual network. ovn-sb : Holds the logical flow table generated from the logical network of ovn-nb , as well as the actual physical network state of each node. ovn-northd : translates the virtual network of ovn-nb into a logical flow table in ovn-sb . Multiple instances of ovn-central will synchronize data via the Raft protocol to ensure high availability. ovs-ovn \u00b6 ovs-ovn runs as a DaemonSet on each node, with openvswitch , ovsdb , and ovn-controller running inside the Pod. These components act as agents for ovn-central to translate logical flow tables into real network configurations. Core Controller and Agent \u00b6 This part is the core component of Kube-OVN, serving as a bridge between OVN and Kubernetes, bridging the two systems and translating network concepts between them. Most of the core functions are implemented in these components. kube-ovn-controller \u00b6 This component performs the translation of all resources within Kubernetes to OVN resources and acts as the control plane for the entire Kube-OVN system. The kube-ovn-controller listens for events on all resources related to network functionality and updates the logical network within the OVN based on resource changes. The main resources listened including: Pod\uff0cService\uff0cEndpoint\uff0cNode\uff0cNetworkPolicy\uff0cVPC\uff0cSubnet\uff0cVlan\uff0cProviderNetwork\u3002 Taking the Pod event as an example, kube-ovn-controller listens to the Pod creation event, allocates the address via the built-in in-memory IPAM function, and calls ovn-central to create logical ports, static routes and possible ACL rules. Next, kube-ovn-controller writes the assigned address and subnet information such as CIDR, gateway, route, etc. to the annotation of the Pod. This annotation is then read by kube-ovn-cni and used to configure the local network. kube-ovn-cni \u00b6 This component runs on each node as a DaemonSet, implements the CNI interface, and operates the local OVS to configure the local network. This DaemonSet copies the kube-ovn binary to each machine as a tool for interaction between kubelet and kube-ovn-cni . This binary sends the corresponding CNI request to kube-ovn-cni for further operation. The binary will be copied to the /opt/cni/bin directory by default. kube-ovn-cni will configure the specific network to perform the appropriate traffic operations, and the main tasks including: Config ovn-controller and vswitchd . Handle CNI Add/Del requests: Create or delete veth pair and bind or unbind to OVS ports. Configure OVS ports Update host iptables/ipset/route rules. Dynamically update the network QoS. Create and configure the ovn0 NIC to connect the container network and the host network. Configure the host NIC to implement Vlan/Underlay/EIP. Dynamically config inter-cluster gateways. Monitoring, Operation and Maintenance Tools and Extension Components \u00b6 These components provide monitoring, diagnostics, operations tools, and external interface to extend the core network capabilities of Kube-OVN and simplify daily operations and maintenance. kube-ovn-speaker \u00b6 This component is a DaemonSet running on a specific labeled nodes that publish routes to the external, allowing external access to the container directly through the Pod IP. For more information on how to use it, please refer to BGP Support . kube-ovn-pinger \u00b6 This component is a DaemonSet running on each node to collect OVS status information, node network quality, network latency, etc. The monitoring metrics collected can be found in Metrics . kube-ovn-monitor \u00b6 This component collects OVN status information and the monitoring metrics, all metrics can be found in Metrics . kubectl-ko \u00b6 This component is a kubectl plugin, which can quickly run common operations, for more usage, please refer to [kubectl plugin].(../ops/kubectl-ko.en.md)\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Architecture"},{"location":"en/reference/architecture/#architecture","text":"This document describes the general architecture of Kube-OVN, the functionality of each component and how they interact with each other. Overall, Kube-OVN serves as a bridge between Kubernetes and OVN, combining proven SDN with Cloud Native. This means that Kube-OVN not only implements network specifications under Kubernetes, such as CNI, Service and Networkpolicy, but also brings a large number of SDN domain capabilities to cloud-native, such as logical switches, logical routers, VPCs, gateways, QoS, ACLs and traffic mirroring. Kube-OVN also maintains a good openness to integrate with many technology solutions, such as Cilium, Submariner, Prometheus, KubeVirt, etc.","title":"Architecture"},{"location":"en/reference/architecture/#component-introduction","text":"The components of Kube-OVN can be broadly divided into three categories. Upstream OVN/OVS components. Core Controller and Agent. Monitoring, operation and maintenance tools and extension components.","title":"Component Introduction"},{"location":"en/reference/architecture/#upstream-ovnovs-components","text":"This type of component comes from the OVN/OVS community with specific modifications for Kube-OVN usage scenarios. OVN/OVS itself is a mature SDN system for managing virtual machines and containers, and we strongly recommend that users interested in the Kube-OVN implementation read ovn-architecture(7) first to understand what OVN is and how to integrate with it. Kube-OVN uses the northbound interface of OVN to create and coordinate virtual networks and map the network concepts into Kubernetes. All OVN/OVS-related components have been packaged into images and are ready to run in Kubernetes.","title":"Upstream OVN/OVS Components"},{"location":"en/reference/architecture/#ovn-central","text":"The ovn-central Deployment runs the control plane components of OVN, including ovn-nb , ovn-sb , and ovn-northd . ovn-nb : Saves the virtual network configuration and provides an API for virtual network management. kube-ovn-controller will mainly interact with ovn-nb to configure the virtual network. ovn-sb : Holds the logical flow table generated from the logical network of ovn-nb , as well as the actual physical network state of each node. ovn-northd : translates the virtual network of ovn-nb into a logical flow table in ovn-sb . Multiple instances of ovn-central will synchronize data via the Raft protocol to ensure high availability.","title":"ovn-central"},{"location":"en/reference/architecture/#ovs-ovn","text":"ovs-ovn runs as a DaemonSet on each node, with openvswitch , ovsdb , and ovn-controller running inside the Pod. These components act as agents for ovn-central to translate logical flow tables into real network configurations.","title":"ovs-ovn"},{"location":"en/reference/architecture/#core-controller-and-agent","text":"This part is the core component of Kube-OVN, serving as a bridge between OVN and Kubernetes, bridging the two systems and translating network concepts between them. Most of the core functions are implemented in these components.","title":"Core Controller and Agent"},{"location":"en/reference/architecture/#kube-ovn-controller","text":"This component performs the translation of all resources within Kubernetes to OVN resources and acts as the control plane for the entire Kube-OVN system. The kube-ovn-controller listens for events on all resources related to network functionality and updates the logical network within the OVN based on resource changes. The main resources listened including: Pod\uff0cService\uff0cEndpoint\uff0cNode\uff0cNetworkPolicy\uff0cVPC\uff0cSubnet\uff0cVlan\uff0cProviderNetwork\u3002 Taking the Pod event as an example, kube-ovn-controller listens to the Pod creation event, allocates the address via the built-in in-memory IPAM function, and calls ovn-central to create logical ports, static routes and possible ACL rules. Next, kube-ovn-controller writes the assigned address and subnet information such as CIDR, gateway, route, etc. to the annotation of the Pod. This annotation is then read by kube-ovn-cni and used to configure the local network.","title":"kube-ovn-controller"},{"location":"en/reference/architecture/#kube-ovn-cni","text":"This component runs on each node as a DaemonSet, implements the CNI interface, and operates the local OVS to configure the local network. This DaemonSet copies the kube-ovn binary to each machine as a tool for interaction between kubelet and kube-ovn-cni . This binary sends the corresponding CNI request to kube-ovn-cni for further operation. The binary will be copied to the /opt/cni/bin directory by default. kube-ovn-cni will configure the specific network to perform the appropriate traffic operations, and the main tasks including: Config ovn-controller and vswitchd . Handle CNI Add/Del requests: Create or delete veth pair and bind or unbind to OVS ports. Configure OVS ports Update host iptables/ipset/route rules. Dynamically update the network QoS. Create and configure the ovn0 NIC to connect the container network and the host network. Configure the host NIC to implement Vlan/Underlay/EIP. Dynamically config inter-cluster gateways.","title":"kube-ovn-cni"},{"location":"en/reference/architecture/#monitoring-operation-and-maintenance-tools-and-extension-components","text":"These components provide monitoring, diagnostics, operations tools, and external interface to extend the core network capabilities of Kube-OVN and simplify daily operations and maintenance.","title":"Monitoring, Operation and Maintenance Tools and Extension Components"},{"location":"en/reference/architecture/#kube-ovn-speaker","text":"This component is a DaemonSet running on a specific labeled nodes that publish routes to the external, allowing external access to the container directly through the Pod IP. For more information on how to use it, please refer to BGP Support .","title":"kube-ovn-speaker"},{"location":"en/reference/architecture/#kube-ovn-pinger","text":"This component is a DaemonSet running on each node to collect OVS status information, node network quality, network latency, etc. The monitoring metrics collected can be found in Metrics .","title":"kube-ovn-pinger"},{"location":"en/reference/architecture/#kube-ovn-monitor","text":"This component collects OVN status information and the monitoring metrics, all metrics can be found in Metrics .","title":"kube-ovn-monitor"},{"location":"en/reference/architecture/#kubectl-ko","text":"This component is a kubectl plugin, which can quickly run common operations, for more usage, please refer to [kubectl plugin].(../ops/kubectl-ko.en.md)\u3002 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kubectl-ko"},{"location":"en/reference/dev-env/","text":"Development Setup \u00b6 Environmental Preparation \u00b6 Kube-OVN uses Golang 1.20 to develop and Go Modules to manage dependency, please check env GO111MODULE=\"on\" \u3002 gosec is used to scan for code security related issues and requires to be installed in the development environment: go get github.com/securego/gosec/v2/cmd/gosec To reduce the size of the final generated image, Kube-OVN uses some of the Docker buildx experimental features, please update Docker to the latest version and enable buildx: docker buildx create --use Build Image \u00b6 Use the following command to download the code and generate the image required to run Kube-OVN: git clone https://github.com/kubeovn/kube-ovn.git cd kube-ovn make release To build an image to run in an ARM environment, run the following command: make release-arm Building the Base Image \u00b6 If you need to change the operating system version, dependencies, OVS/OVN code, etc., you need to rebuild the base image. The Dockerfile used for the base image is dist/images/Dockerfile.base . Build instructions: # build x86 base image make base-amd64 # build arm base image make base-arm64 Run E2E \u00b6 Kube-OVN uses KIND to build local Kubernetes cluster, j2cli to render templates, and Ginkgo to run test cases. Please refer to the relevant documentation for dependency installation. Run E2E locally: make kind-init make kind-install make e2e To run the Underlay E2E test, run the following commands: make kind-init make kind-install-underlay make e2e-underlay-single-nic To run the ovn vpc nat gw eip, fip, snat, dnat E2E test, run the following commands: make kind-init make kind-install make ovn-vpc-nat-gw-conformance-e2e To run the iptables vpc nat gw eip, fip, snat, dnat E2E test, run the following commands: make kind-init make kind-install make kind-install-vpc-nat-gw make iptables-vpc-nat-gw-conformance-e2e To run the loadbalancer service E2E test, run the following commands: make kind-init make kind-install make kind-install-lb-svc make kube-ovn-lb-svc-conformance-e2e To clean, run the following commands: make kind-clean \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Development Setup"},{"location":"en/reference/dev-env/#development-setup","text":"","title":"Development Setup"},{"location":"en/reference/dev-env/#environmental-preparation","text":"Kube-OVN uses Golang 1.20 to develop and Go Modules to manage dependency, please check env GO111MODULE=\"on\" \u3002 gosec is used to scan for code security related issues and requires to be installed in the development environment: go get github.com/securego/gosec/v2/cmd/gosec To reduce the size of the final generated image, Kube-OVN uses some of the Docker buildx experimental features, please update Docker to the latest version and enable buildx: docker buildx create --use","title":"Environmental Preparation"},{"location":"en/reference/dev-env/#build-image","text":"Use the following command to download the code and generate the image required to run Kube-OVN: git clone https://github.com/kubeovn/kube-ovn.git cd kube-ovn make release To build an image to run in an ARM environment, run the following command: make release-arm","title":"Build Image"},{"location":"en/reference/dev-env/#building-the-base-image","text":"If you need to change the operating system version, dependencies, OVS/OVN code, etc., you need to rebuild the base image. The Dockerfile used for the base image is dist/images/Dockerfile.base . Build instructions: # build x86 base image make base-amd64 # build arm base image make base-arm64","title":"Building the Base Image"},{"location":"en/reference/dev-env/#run-e2e","text":"Kube-OVN uses KIND to build local Kubernetes cluster, j2cli to render templates, and Ginkgo to run test cases. Please refer to the relevant documentation for dependency installation. Run E2E locally: make kind-init make kind-install make e2e To run the Underlay E2E test, run the following commands: make kind-init make kind-install-underlay make e2e-underlay-single-nic To run the ovn vpc nat gw eip, fip, snat, dnat E2E test, run the following commands: make kind-init make kind-install make ovn-vpc-nat-gw-conformance-e2e To run the iptables vpc nat gw eip, fip, snat, dnat E2E test, run the following commands: make kind-init make kind-install make kind-install-vpc-nat-gw make iptables-vpc-nat-gw-conformance-e2e To run the loadbalancer service E2E test, run the following commands: make kind-init make kind-install make kind-install-lb-svc make kube-ovn-lb-svc-conformance-e2e To clean, run the following commands: make kind-clean \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Run E2E"},{"location":"en/reference/document-convention/","text":"\u6587\u6863\u89c4\u8303 \u00b6 \u4e3a\u4e86\u4fdd\u8bc1\u6587\u6863\u98ce\u683c\u4e00\u81f4\uff0c\u8bf7\u5728\u63d0\u4ea4\u6587\u6863\u65f6\u9075\u5faa\u4e0b\u5217\u7684\u98ce\u683c\u89c4\u8303\u3002 \u6807\u70b9 \u00b6 \u4e2d\u6587\u6587\u6863\u4e2d\u6587\u672c\u5185\u5bb9\u6240\u6709\u6807\u70b9\u5e94\u4f7f\u7528\u4e2d\u6587\u683c\u5f0f\u6807\u70b9\uff0c\u82f1\u6587\u6587\u6863\u4e2d\u6240\u6709\u6587\u672c\u5185\u5bb9\u4e2d\u5e94\u4f7f\u7528\u82f1\u6587\u6807\u70b9\u3002 Bad Good \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c,\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528,\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc. \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002 \u82f1\u6587\u6570\u5b57\u548c\u4e2d\u6587\u5e94\u8be5\u7528\u7a7a\u683c\u8fdb\u884c\u5206\u9694\u3002 Bad Good Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 \u793a\u4f8b\u5185\u5bb9\u5e94\u8be5\u4ee5 \uff1a \u5f00\u542f\uff0c\u5176\u4ed6\u53e5\u5c3e\u9700\u8981\u7528 \u3002 \u7ed3\u675f\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\u3002 wget 127 .0.0.1 \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget 127 .0.0.1 \u4ee3\u7801\u5757 \u00b6 yaml \u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a yaml\u3002 Bad Good ```` apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` ````yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` \u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a bash\u3002 Bad Good ```` wget 127.0.0.1 ```` ````bash wget 127.0.0.1 ```` \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u5305\u542b\u8f93\u51fa\u5185\u5bb9\uff0c\u5219\u6240\u6267\u884c\u547d\u4ee4\u9700\u8981\u4ee5 # \u5f00\u59cb\uff0c\u4ee5\u533a\u5206\u8f93\u5165\u4e0e\u8f93\u51fa\u3002 Bad Good oilbeater@macdeMac-3 ~ ping 114 .114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms # ping 114.114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u53ea\u5305\u542b\u6267\u884c\u547d\u4ee4\uff0c\u6ca1\u6709\u8f93\u51fa\u7ed3\u679c\uff0c\u5219\u591a\u6761\u547d\u4ee4\u65e0\u9700 # \u5f00\u59cb\u3002 Bad Good # mv /etc/origin/ovn/ovnnb_db.db /tmp # mv /etc/origin/ovn/ovnsb_db.db /tmp mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp \u94fe\u63a5 \u00b6 \u7ad9\u5185\u94fe\u63a5\u4f7f\u7528\u5bf9\u5e94 md \u6587\u4ef6\u8def\u5f84\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](http://kubeovn.github.io/prepare)\u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](./prepare.md)\u3002 Bad Good \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io )\u3002 \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io ){: target=\"_blank\" }\u3002 \u7a7a\u884c \u00b6 \u4e0d\u540c\u903b\u8f91\u5757\uff0c\u4f8b\u5982\u6807\u9898\u548c\u6587\u672c\uff0c\u6587\u672c\u548c\u4ee3\u7801\uff0c\u6587\u672c\u548c\u7f16\u53f7\u4e4b\u95f4\u9700\u8981\u7528\u7a7a\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0d\u540c\u903b\u8f91\u5757\u4e4b\u95f4\u53ea\u4f7f\u7528 \u4e00\u4e2a \u7a7a\u884c\u8fdb\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Document Convention"},{"location":"en/reference/document-convention/#_1","text":"\u4e3a\u4e86\u4fdd\u8bc1\u6587\u6863\u98ce\u683c\u4e00\u81f4\uff0c\u8bf7\u5728\u63d0\u4ea4\u6587\u6863\u65f6\u9075\u5faa\u4e0b\u5217\u7684\u98ce\u683c\u89c4\u8303\u3002","title":"\u6587\u6863\u89c4\u8303"},{"location":"en/reference/document-convention/#_2","text":"\u4e2d\u6587\u6587\u6863\u4e2d\u6587\u672c\u5185\u5bb9\u6240\u6709\u6807\u70b9\u5e94\u4f7f\u7528\u4e2d\u6587\u683c\u5f0f\u6807\u70b9\uff0c\u82f1\u6587\u6587\u6863\u4e2d\u6240\u6709\u6587\u672c\u5185\u5bb9\u4e2d\u5e94\u4f7f\u7528\u82f1\u6587\u6807\u70b9\u3002 Bad Good \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c,\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528,\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc. \u8fd9\u91cc\u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\uff0c\u53ef\u4ee5\u5e2e\u52a9\u4f60\u5feb\u901f\u5b89\u88c5\u4e00\u4e2a\u9ad8\u53ef\u7528\uff0c\u751f\u4ea7\u5c31\u7eea\u7684\u5bb9\u5668\u7f51\u7edc\u3002 \u82f1\u6587\u6570\u5b57\u548c\u4e2d\u6587\u5e94\u8be5\u7528\u7a7a\u683c\u8fdb\u884c\u5206\u9694\u3002 Bad Good Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 Kube-OVN \u63d0\u4f9b\u4e86\u4e00\u952e\u5b89\u88c5\u811a\u672c\u6765\u5b89\u88c5 1.10 \u7248\u672c Kube-OVN\u3002 \u793a\u4f8b\u5185\u5bb9\u5e94\u8be5\u4ee5 \uff1a \u5f00\u542f\uff0c\u5176\u4ed6\u53e5\u5c3e\u9700\u8981\u7528 \u3002 \u7ed3\u675f\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\u3002 wget 127 .0.0.1 \u5b89\u88c5\u524d\u8bf7\u786e\u8ba4\u73af\u5883\u914d\u7f6e\u6b63\u786e\u3002 \u4f7f\u7528\u4e0b\u9762\u7684\u547d\u4ee4\u4e0b\u8f7d\u5b89\u88c5\u811a\u672c\uff1a wget 127 .0.0.1","title":"\u6807\u70b9"},{"location":"en/reference/document-convention/#_3","text":"yaml \u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a yaml\u3002 Bad Good ```` apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` ````yaml apiVersion: kubeovn.io/v1 kind: Subnet metadata: name: attach-subnet ```` \u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4ee3\u7801\u5757\u9700\u8981\u6807\u8bc6\u4e3a bash\u3002 Bad Good ```` wget 127.0.0.1 ```` ````bash wget 127.0.0.1 ```` \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u5305\u542b\u8f93\u51fa\u5185\u5bb9\uff0c\u5219\u6240\u6267\u884c\u547d\u4ee4\u9700\u8981\u4ee5 # \u5f00\u59cb\uff0c\u4ee5\u533a\u5206\u8f93\u5165\u4e0e\u8f93\u51fa\u3002 Bad Good oilbeater@macdeMac-3 ~ ping 114 .114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms # ping 114.114.114.114 -c 3 PING 114 .114.114.114 ( 114 .114.114.114 ) : 56 data bytes 64 bytes from 114 .114.114.114: icmp_seq = 0 ttl = 83 time = 10 .429 ms 64 bytes from 114 .114.114.114: icmp_seq = 1 ttl = 79 time = 11 .360 ms 64 bytes from 114 .114.114.114: icmp_seq = 2 ttl = 76 time = 10 .794 ms --- 114 .114.114.114 ping statistics --- 3 packets transmitted, 3 packets received, 0 .0% packet loss round-trip min/avg/max/stddev = 10 .429/10.861/11.360/0.383 ms \u5982\u679c\u547d\u4ee4\u884c\u64cd\u4f5c\u793a\u4f8b\u4e2d\u53ea\u5305\u542b\u6267\u884c\u547d\u4ee4\uff0c\u6ca1\u6709\u8f93\u51fa\u7ed3\u679c\uff0c\u5219\u591a\u6761\u547d\u4ee4\u65e0\u9700 # \u5f00\u59cb\u3002 Bad Good # mv /etc/origin/ovn/ovnnb_db.db /tmp # mv /etc/origin/ovn/ovnsb_db.db /tmp mv /etc/origin/ovn/ovnnb_db.db /tmp mv /etc/origin/ovn/ovnsb_db.db /tmp","title":"\u4ee3\u7801\u5757"},{"location":"en/reference/document-convention/#_4","text":"\u7ad9\u5185\u94fe\u63a5\u4f7f\u7528\u5bf9\u5e94 md \u6587\u4ef6\u8def\u5f84\u3002 Bad Good \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](http://kubeovn.github.io/prepare)\u3002 \u5b89\u88c5\u524d\u8bf7\u53c2\u8003[\u51c6\u5907\u5de5\u4f5c](./prepare.md)\u3002 Bad Good \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io )\u3002 \u5982\u6709\u95ee\u9898\u8bf7\u53c2\u8003 [ Kubernetes \u6587\u6863 ]( http://kubernetes.io ){: target=\"_blank\" }\u3002","title":"\u94fe\u63a5"},{"location":"en/reference/document-convention/#_5","text":"\u4e0d\u540c\u903b\u8f91\u5757\uff0c\u4f8b\u5982\u6807\u9898\u548c\u6587\u672c\uff0c\u6587\u672c\u548c\u4ee3\u7801\uff0c\u6587\u672c\u548c\u7f16\u53f7\u4e4b\u95f4\u9700\u8981\u7528\u7a7a\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0d\u540c\u903b\u8f91\u5757\u4e4b\u95f4\u53ea\u4f7f\u7528 \u4e00\u4e2a \u7a7a\u884c\u8fdb\u884c\u5206\u9694\u3002 Bad Good \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u4e0b\u8f7d\u4e0b\u9762\u7684\u811a\u672c\uff0c\u8fdb\u884c\u5b89\u88c5\uff1a ```bash wget 127 .0.0.1 ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"\u7a7a\u884c"},{"location":"en/reference/feature-stage/","text":"Feature Stage \u00b6 In Kube-OVN, feature stage is classified into Alpha , Beta and GA , based on the degree of feature usage, documentation and test coverage. Definition of Stage \u00b6 For Alpha stage functions: The feature is not fully documented and well tested. This feature may change or even be removed in the future. This feature API is not guaranteed to be stable and may be removed. Community provides low priority support for this feature and long-term support cannot be guaranteed. Since feature stability and long-term support cannot be guaranteed, it can be tested and verified, but is not recommended for production use. For Beta stage functions: This feature is partially documented and tested, but complete coverage is not guaranteed. This feature may change in the future and the upgrade may affect the network, but it will not be removed as a whole. This feature API may change in the future and the fields may be adjusted, but not removed as a whole. This feature will be supported by the community in the long term. It can be used on non-critical services as the functionality will be supported for a long time, but it is not recommended for critical production service as there is a possibility of changes in functionality and APIs that may break the network. For GA stage functions: The feature has full documentation and test coverage. The feature will remain stable and upgrades will be guaranteed to be smooth. This feature API is not subject to disruptive changes. This feature will be supported with high priority by the community and long-term support will be guaranteed. Feature Stage List \u00b6 This list records the feature stages from the 1.8 release. Feature Default Stage Since Until Namespaced Subnet true GA 1.8 Distributed Gateway true GA 1.8 Active-backup Centralized Gateway true GA 1.8 ECMP Centralized Gateway false Beta 1.8 Subnet ACL true Alpha 1.9 Subnet Isolation (Will be replaced by ACL later) true Beta 1.8 Underlay Subnet true GA 1.8 Multiple Pod Interface true Beta 1.8 Subnet DHCP false Alpha 1.10 Subnet with External Gateway false Alpha 1.8 Cluster Inter-Connection with OVN-IC false Beta 1.8 Cluster Inter-Connection with Submariner false Alpha 1.9 VIP Reservation true Alpha 1.10 Create Custom VPC true Beta 1.8 Custom VPC Floating IP/SNAT/DNAT true Alpha 1.10 Custom VPC Static Route true Alpha 1.10 Custom VPC Policy Route true Alpha 1.10 Custom VPC Security Group true Alpha 1.10 Container Bandwidth QoS true GA 1.8 linux-netem QoS true Alpha 1.9 Prometheus Integration false GA 1.8 Grafana Integration false GA 1.8 IPv4/v6 DualStack false GA 1.8 Default VPC EIP/SNAT false Beta 1.8 Traffic Mirroring false GA 1.8 NetworkPolicy true Beta 1.8 Webhook false Alpha 1.10 Performance Tunning false Beta 1.8 Interconnection with Routes in Overlay Mode false Alpha 1.8 BGP Support false Alpha 1.9 Cilium Integration false Alpha 1.10 Custom VPC Peering false Alpha 1.10 Mellanox Offload false Alpha 1.8 Corigine Offload false Alpha 1.10 Windows Support false Alpha 1.10 DPDK Support false Alpha 1.10 OpenStack Integration false Alpha 1.9 Single Pod Fixed IP/Mac true GA 1.8 Workload with Fixed IP true GA 1.8 StatefulSet with Fixed IP true GA 1.8 VM with Fixed IP false Beta 1.9 Load Balancer Type Service in Default VPC false Alpha 1.11 Load Balance in Custom VPC false Alpha 1.11 DNS in Custom VPC false Alpha 1.11 Underlay and Overlay Interconnection false Alpha 1.11 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Feature Stage"},{"location":"en/reference/feature-stage/#feature-stage","text":"In Kube-OVN, feature stage is classified into Alpha , Beta and GA , based on the degree of feature usage, documentation and test coverage.","title":"Feature Stage"},{"location":"en/reference/feature-stage/#definition-of-stage","text":"For Alpha stage functions: The feature is not fully documented and well tested. This feature may change or even be removed in the future. This feature API is not guaranteed to be stable and may be removed. Community provides low priority support for this feature and long-term support cannot be guaranteed. Since feature stability and long-term support cannot be guaranteed, it can be tested and verified, but is not recommended for production use. For Beta stage functions: This feature is partially documented and tested, but complete coverage is not guaranteed. This feature may change in the future and the upgrade may affect the network, but it will not be removed as a whole. This feature API may change in the future and the fields may be adjusted, but not removed as a whole. This feature will be supported by the community in the long term. It can be used on non-critical services as the functionality will be supported for a long time, but it is not recommended for critical production service as there is a possibility of changes in functionality and APIs that may break the network. For GA stage functions: The feature has full documentation and test coverage. The feature will remain stable and upgrades will be guaranteed to be smooth. This feature API is not subject to disruptive changes. This feature will be supported with high priority by the community and long-term support will be guaranteed.","title":"Definition of Stage"},{"location":"en/reference/feature-stage/#feature-stage-list","text":"This list records the feature stages from the 1.8 release. Feature Default Stage Since Until Namespaced Subnet true GA 1.8 Distributed Gateway true GA 1.8 Active-backup Centralized Gateway true GA 1.8 ECMP Centralized Gateway false Beta 1.8 Subnet ACL true Alpha 1.9 Subnet Isolation (Will be replaced by ACL later) true Beta 1.8 Underlay Subnet true GA 1.8 Multiple Pod Interface true Beta 1.8 Subnet DHCP false Alpha 1.10 Subnet with External Gateway false Alpha 1.8 Cluster Inter-Connection with OVN-IC false Beta 1.8 Cluster Inter-Connection with Submariner false Alpha 1.9 VIP Reservation true Alpha 1.10 Create Custom VPC true Beta 1.8 Custom VPC Floating IP/SNAT/DNAT true Alpha 1.10 Custom VPC Static Route true Alpha 1.10 Custom VPC Policy Route true Alpha 1.10 Custom VPC Security Group true Alpha 1.10 Container Bandwidth QoS true GA 1.8 linux-netem QoS true Alpha 1.9 Prometheus Integration false GA 1.8 Grafana Integration false GA 1.8 IPv4/v6 DualStack false GA 1.8 Default VPC EIP/SNAT false Beta 1.8 Traffic Mirroring false GA 1.8 NetworkPolicy true Beta 1.8 Webhook false Alpha 1.10 Performance Tunning false Beta 1.8 Interconnection with Routes in Overlay Mode false Alpha 1.8 BGP Support false Alpha 1.9 Cilium Integration false Alpha 1.10 Custom VPC Peering false Alpha 1.10 Mellanox Offload false Alpha 1.8 Corigine Offload false Alpha 1.10 Windows Support false Alpha 1.10 DPDK Support false Alpha 1.10 OpenStack Integration false Alpha 1.9 Single Pod Fixed IP/Mac true GA 1.8 Workload with Fixed IP true GA 1.8 StatefulSet with Fixed IP true GA 1.8 VM with Fixed IP false Beta 1.9 Load Balancer Type Service in Default VPC false Alpha 1.11 Load Balance in Custom VPC false Alpha 1.11 DNS in Custom VPC false Alpha 1.11 Underlay and Overlay Interconnection false Alpha 1.11 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Feature Stage List"},{"location":"en/reference/iptables-rules/","text":"Iptables Rules \u00b6 Kube-OVN uses ipset and iptables to implement gateway NAT functionality in the default VPC overlay Subnets. The ipset used is shown in the following table: Name\uff08IPv4/IPv6\uff09 Type Usage ovn40services/ovn60services hash:net Service CIDR ovn40subnets/ovn60subnets hash:net Overlay Subnet CIDR and NodeLocal DNS IP address ovn40subnets-nat/ovn60subnets-nat hash:net Overlay Subnet CIDRs that enable NatOutgoing ovn40subnets-distributed-gw/ovn60subnets-distributed-gw hash:net Overlay Subnet CIDRs that use distributed gateway ovn40other-node/ovn60other-node hash:net Internal IP addresses for other Nodes ovn40local-pod-ip-nat/ovn60local-pod-ip-nat hash:ip Deprecated ovn40subnets-nat-policy hash:net All subnet cidrs configured with natOutgoingPolicyRules ovn40natpr-418e79269dc5-dst hash:net The dstIPs corresponding to the rule in natOutgoingPolicyRules ovn40natpr-418e79269dc5-src hash:net The srcIPs corresponding to the rule in natOutgoingPolicyRules The iptables rules (IPv4) used are shown in the following table: Table Chain Rule Usage Note filter INPUT -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -s 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" Used to count packets from the subnet to the external network \"10.16.0.0/16\" is the cidr of the subnet, the \"ovn-subnet-gateway\" before the \",\" in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the \"ovn-default\" after the \",\" is the name of the subnet filter FORWARD -d 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" Used to count packets from the external network accessing the subnet \"10.16.0.0/16\" is the cidr of the subnet, the \"ovn-subnet-gateway\" before the \",\" in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the \"ovn-default\" after the \",\" is the name of the subnet filter OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0 Clear traffic tag to prevent SNAT UDP: bad checksum on VXLAN interface nat PREROUTING -m comment --comment \"kube-ovn prerouting rules\" -j OVN-PREROUTING Enter OVN-PREROUTING chain processing -- nat POSTROUTING -m comment --comment \"kube-ovn postrouting rules\" -j OVN-POSTROUTING Enter OVN-POSTROUTING chain processing -- nat OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set ovn40services dst -j MARK --set-xmark 0x4000/0x4000 Adding masquerade tags to Pod access service traffic Used when the built-in LB is turned off nat OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (TCP) Only used when kube-proxy is using ipvs mode nat OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (UDP) Only used when kube-proxy is using ipvs mode nat OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source Use node IP as the source address for access from node to overlay Pods via service IP\u3002 Works only when kube-proxy is using ipvs mode nat OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE Perform SNAT for specific tagged traffic -- nat OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j MASQUERADE Perform SNAT for Service traffic between Pods passing through the node -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a distributed gateway, SNAT is not required. -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j MASQUERADE For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a centralized gateway, SNAT is required. -- nat OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN No SNAT is performed when the Pod IP is exposed to the outside world -- nat OVN-POSTROUTING -s 10.16.0.0/16 -m set ! --match-set ovn40subnets dst -j SNAT --to-source 192.168.0.101 When the Pod accesses the network outside the cluster, if the subnet is NatOutgoing and a centralized gateway with the specified IP is used, perform SNAT 10.16.0.0/16 is the Subnet CIDR\uff0c192.168.0.101 is the specified IP of gateway node nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j MASQUERADE When the Pod accesses the network outside the cluster, if NatOutgoing is enabled on the subnet, perform SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT ovn40subnets-nat-policy is all subnet segments configured with natOutgoingPolicyRules nat OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j MASQUERADE --random-fully When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90001/0x90001, it will do SNAT nat OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90002/0x90002, it will not do SNAT nat OVN-NAT-POLICY -s 10.0.11.0/24 -m comment --comment natPolicySubnet-net1 -j OVN-NAT-PSUBNET-aa98851157c5 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 10.0.11.0/24 represents the CIDR of the subnet net1, and the rules under the OVN-NAT-PSUBNET-aa98851157c5 chain correspond to the natOutgoingPolicyRules configuration of this subnet nat OVN-NAT-PSUBNET-xxxxxxxxxxxx -m set --match-set ovn40natpr-418e79269dc5-src src -m set --match-set ovn40natpr-418e79269dc5-dst dst -j MARK --set-xmark 0x90002/0x90002 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 418e79269dc5 indicates the ID of a rule in natOutgoingPolicyRules, which can be viewed through status.natOutgoingPolicyRules[index].RuleID, indicating that srcIPs meets ovn40natpr-418e79269dc5-src, and dstIPS meets ovn40natpr-418e79269dc5- dst will be marked with tag 0x90002 mangle OVN-OUTPUT -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003 Introduce kubelet's detection traffic to tproxy with a specific mark mangle OVN-PREROUTING -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.3 --tproxy-mark 0x90004/0x90004 Introduce kubelet's detection traffic to tproxy with a specific mark \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Iptables Rules"},{"location":"en/reference/iptables-rules/#iptables-rules","text":"Kube-OVN uses ipset and iptables to implement gateway NAT functionality in the default VPC overlay Subnets. The ipset used is shown in the following table: Name\uff08IPv4/IPv6\uff09 Type Usage ovn40services/ovn60services hash:net Service CIDR ovn40subnets/ovn60subnets hash:net Overlay Subnet CIDR and NodeLocal DNS IP address ovn40subnets-nat/ovn60subnets-nat hash:net Overlay Subnet CIDRs that enable NatOutgoing ovn40subnets-distributed-gw/ovn60subnets-distributed-gw hash:net Overlay Subnet CIDRs that use distributed gateway ovn40other-node/ovn60other-node hash:net Internal IP addresses for other Nodes ovn40local-pod-ip-nat/ovn60local-pod-ip-nat hash:ip Deprecated ovn40subnets-nat-policy hash:net All subnet cidrs configured with natOutgoingPolicyRules ovn40natpr-418e79269dc5-dst hash:net The dstIPs corresponding to the rule in natOutgoingPolicyRules ovn40natpr-418e79269dc5-src hash:net The srcIPs corresponding to the rule in natOutgoingPolicyRules The iptables rules (IPv4) used are shown in the following table: Table Chain Rule Usage Note filter INPUT -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter INPUT -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through -- filter FORWARD -s 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" Used to count packets from the subnet to the external network \"10.16.0.0/16\" is the cidr of the subnet, the \"ovn-subnet-gateway\" before the \",\" in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the \"ovn-default\" after the \",\" is the name of the subnet filter FORWARD -d 10.16.0.0/16 -m comment --comment \"ovn-subnet-gateway,ovn-default\" Used to count packets from the external network accessing the subnet \"10.16.0.0/16\" is the cidr of the subnet, the \"ovn-subnet-gateway\" before the \",\" in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the \"ovn-default\" after the \",\" is the name of the subnet filter OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0 Clear traffic tag to prevent SNAT UDP: bad checksum on VXLAN interface nat PREROUTING -m comment --comment \"kube-ovn prerouting rules\" -j OVN-PREROUTING Enter OVN-PREROUTING chain processing -- nat POSTROUTING -m comment --comment \"kube-ovn postrouting rules\" -j OVN-POSTROUTING Enter OVN-POSTROUTING chain processing -- nat OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set ovn40services dst -j MARK --set-xmark 0x4000/0x4000 Adding masquerade tags to Pod access service traffic Used when the built-in LB is turned off nat OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (TCP) Only used when kube-proxy is using ipvs mode nat OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (UDP) Only used when kube-proxy is using ipvs mode nat OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source Use node IP as the source address for access from node to overlay Pods via service IP\u3002 Works only when kube-proxy is using ipvs mode nat OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE Perform SNAT for specific tagged traffic -- nat OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j MASQUERADE Perform SNAT for Service traffic between Pods passing through the node -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a distributed gateway, SNAT is not required. -- nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j MASQUERADE For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a centralized gateway, SNAT is required. -- nat OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN No SNAT is performed when the Pod IP is exposed to the outside world -- nat OVN-POSTROUTING -s 10.16.0.0/16 -m set ! --match-set ovn40subnets dst -j SNAT --to-source 192.168.0.101 When the Pod accesses the network outside the cluster, if the subnet is NatOutgoing and a centralized gateway with the specified IP is used, perform SNAT 10.16.0.0/16 is the Subnet CIDR\uff0c192.168.0.101 is the specified IP of gateway node nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j MASQUERADE When the Pod accesses the network outside the cluster, if NatOutgoing is enabled on the subnet, perform SNAT -- nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT ovn40subnets-nat-policy is all subnet segments configured with natOutgoingPolicyRules nat OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j MASQUERADE --random-fully When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90001/0x90001, it will do SNAT nat OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90002/0x90002, it will not do SNAT nat OVN-NAT-POLICY -s 10.0.11.0/24 -m comment --comment natPolicySubnet-net1 -j OVN-NAT-PSUBNET-aa98851157c5 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 10.0.11.0/24 represents the CIDR of the subnet net1, and the rules under the OVN-NAT-PSUBNET-aa98851157c5 chain correspond to the natOutgoingPolicyRules configuration of this subnet nat OVN-NAT-PSUBNET-xxxxxxxxxxxx -m set --match-set ovn40natpr-418e79269dc5-src src -m set --match-set ovn40natpr-418e79269dc5-dst dst -j MARK --set-xmark 0x90002/0x90002 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 418e79269dc5 indicates the ID of a rule in natOutgoingPolicyRules, which can be viewed through status.natOutgoingPolicyRules[index].RuleID, indicating that srcIPs meets ovn40natpr-418e79269dc5-src, and dstIPS meets ovn40natpr-418e79269dc5- dst will be marked with tag 0x90002 mangle OVN-OUTPUT -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003 Introduce kubelet's detection traffic to tproxy with a specific mark mangle OVN-PREROUTING -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.3 --tproxy-mark 0x90004/0x90004 Introduce kubelet's detection traffic to tproxy with a specific mark \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Iptables Rules"},{"location":"en/reference/kube-ovn-api/","text":"Kube-OVN API Reference \u00b6 Based on Kube-OVN v1.12.0, we have compiled a list of CRD resources supported by Kube-OVN, listing the types and meanings of each field of CRD definition for reference. Generic Condition Definition \u00b6 Property Name Type Description type String Type of status status String The value of status, in the range of True , False or Unknown reason String The reason for the status change message String The specific message of the status change lastUpdateTime Time The last time the status was updated lastTransitionTime Time Time of last status type change In each CRD definition, the Condition field in Status follows the above format, so we explain it in advance. Subnet Definition \u00b6 Subnet \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Subnet metadata ObjectMeta Standard Kubernetes resource metadata information spec SubnetSpec Subnet specific configuration information status SubnetStatus Subnet status information SubnetSpec \u00b6 Property Name Type Description default Bool Whether this subnet is the default subnet vpc String The vpc which the subnet belongs to, default is ovn-cluster protocol String IP protocol, the value is in the range of IPv4 , IPv6 or Dual namespaces []String The list of namespaces bound to this subnet cidrBlock String The range of the subnet, e.g. 10.16.0.0/16 gateway String The gateway address of the subnet, the default value is the first available address under the CIDRBlock of the subnet excludeIps []String The range of addresses under this subnet that will not be automatically assigned provider String Default value is ovn . In the case of multiple NICs, the value is <name>.<namespace> of the NetworkAttachmentDefinition, Kube-OVN will use this information to find the corresponding subnet resource gatewayType String The gateway type in overlay mode, either distributed or centralized gatewayNode String The gateway node when the gateway mode is centralized, node names can be comma-separated natOutgoing Bool Whether the outgoing traffic is NAT externalEgressGateway String The address of the external gateway. This parameter and the natOutgoing parameter cannot be set at the same time policyRoutingPriority Uint32 Policy route priority. Used to control the forwarding of traffic to the external gateway address after the subnet gateway policyRoutingTableID Uint32 The TableID of the local policy routing table, should be different for each subnet to avoid conflicts private Bool Whether the subnet is a private subnet, which denies access to addresses inside the subnet if the subnet is private allowSubnets []String If the subnet is a private subnet, the set of addresses that are allowed to access the subnet vlan String The name of vlan to which the subnet is bound vips []String The virtual-ip parameter information for virtual type lsp on the subnet logicalGateway Bool Whether to enable logical gateway disableGatewayCheck Bool Whether to skip the gateway connectivity check when creating a pod disableInterConnection Bool Whether to enable subnet interconnection across clusters enableDHCP Bool Whether to configure dhcp configuration options for lsps belong this subnet dhcpV4Options String The DHCP_Options record associated with lsp dhcpv4_options on the subnet dhcpV6Options String The DHCP_Options record associated with lsp dhcpv6_options on the subnet enableIPv6RA Bool Whether to configure the ipv6_ra_configs parameter for the lrp port of the router connected to the subnet ipv6RAConfigs String The ipv6_ra_configs parameter configuration for the lrp port of the router connected to the subnet acls []Acl The acls record associated with the logical-switch of the subnet u2oInterconnection Bool Whether to enable interconnection mode for Overlay/Underlay enableLb *Bool Whether the logical-switch of the subnet is associated with load-balancer records enableEcmp Bool Centralized subnet, whether to enable ECMP routing Acl \u00b6 Property Name Type Description direction String Restrict the direction of acl, which value is from-lport or to-lport priority Int Acl priority, in the range 0 to 32767 match String Acl rule match expression action String The action of the rule, which value is in the range of allow-related , allow-stateless , allow , drop , reject SubnetStatus \u00b6 Property Name Type Description conditions []SubnetCondition Subnet status change information, refer to the beginning of the document for the definition of Condition v4AvailableIPs Float64 Number of available IPv4 IPs v4availableIPrange String The available range of IPv4 addresses on the subnet v4UsingIPs Float64 Number of used IPv4 IPs v4usingIPrange String Used IPv4 address ranges on the subnet v6AvailableIPs Float64 Number of available IPv6 IPs v6availableIPrange String The available range of IPv6 addresses on the subnet v6UsingIPs Float64 Number of used IPv6 IPs v6usingIPrange String Used IPv6 address ranges on the subnet sctivateGateway String The currently working gateway node in centralized subnet of master-backup mode dhcpV4OptionsUUID String The DHCP_Options record identifier associated with the lsp dhcpv4_options on the subnet dhcpV6OptionsUUID String The DHCP_Options record identifier associated with the lsp dhcpv6_options on the subnet u2oInterconnectionIP String The IP address used for interconnection when Overlay/Underlay interconnection mode is enabled IP Definition \u00b6 IP \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IP metadata ObjectMeta Standard Kubernetes resource metadata information spec IPSpec IP specific configuration information IPSepc \u00b6 Property Name Type Description podName String Pod name which assigned with this IP namespace String The name of the namespace where the pod is bound subnet String The subnet which the ip belongs to attachSubnets []String The name of the other subnets attached to this primary IP (field deprecated) nodeName String The name of the node where the pod is bound ipAddress String IP address, in v4IP,v6IP format for dual-stack cases v4IPAddress String IPv4 IP address v6IPAddress String IPv6 IP address attachIPs []String Other IP addresses attached to this primary IP (field is deprecated) macAddress String The Mac address of the bound pod attachMacs []String Other Mac addresses attached to this primary IP (field deprecated) containerID String The Container ID corresponding to the bound pod podType String Special workload pod, can be StatefulSet , VirtualMachine or empty Underlay configuration \u00b6 Vlan \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all instances of this resource will be kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vlan metadata ObjectMeta Standard Kubernetes resource metadata information spec VlanSpec Vlan specific configuration information status VlanStatus Vlan status information VlanSpec \u00b6 Property Name Type Description id Int Vlan tag number, in the range of 0~4096 provider String The name of the ProviderNetwork to which the vlan is bound VlanStatus \u00b6 Property Name Type Description subnets []String The list of subnets to which the vlan is bound conditions []VlanCondition Vlan status change information, refer to the beginning of the document for the definition of Condition ProviderNetwork \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value ProviderNetwork metadata ObjectMeta Standard Kubernetes resource metadata information spec ProviderNetworkSpec ProviderNetwork specific configuration information status ProviderNetworkStatus ProviderNetwork status information ProviderNetworkSpec \u00b6 Property Name Type Description defaultInterface String The name of the NIC interface used by default for this bridge network customInterfaces []CustomInterface The special NIC configuration used by this bridge network excludeNodes []String The names of the nodes that will not be bound to this bridge network exchangeLinkName Bool Whether to exchange the bridge NIC and the corresponding OVS bridge name CustomInterface \u00b6 Property Name Type Description interface String NIC interface name used for underlay nodes []String List of nodes using the custom NIC interface ProviderNetworkStatus \u00b6 Property Name Type Description ready Bool Whether the current bridge network is in the ready state readyNodes []String The name of the node whose bridge network is ready notReadyNodes []String The name of the node whose bridge network is not ready vlans []String The name of the vlan to which the bridge network is bound conditions []ProviderNetworkCondition ProviderNetwork status change information, refer to the beginning of the document for the definition of Condition Vpc Definition \u00b6 Vpc \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vpc metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcSpec Vpc specific configuration information status VpcStatus Vpc status information VpcSpec \u00b6 Property Name Type Description namespaces []String List of namespaces bound by Vpc staticRoutes []*StaticRoute The static route information configured under Vpc policyRoutes []*PolicyRoute The policy route information configured under Vpc vpcPeerings []*VpcPeering Vpc interconnection information enableExternal Bool Whether vpc is connected to an external switch StaticRoute \u00b6 Property Name Type Description policy String Routing policy, takes the value of policySrc or policyDst cidr String Routing cidr value nextHopIP String The next hop information of the route PolicyRoute \u00b6 Property Name Type Description priority Int32 Priority for policy route match String Match expression for policy route action String Action for policy route, the value is in the range of allow , drop , reroute nextHopIP String The next hop of the policy route, separated by commas in the case of ECMP routing VpcPeering \u00b6 Property Name Type Description remoteVpc String Name of the interconnected peering vpc localConnectIP String The local ip for vpc used to connect to peer vpc VpcStatus \u00b6 Property Name Type Description conditions []VpcCondition Vpc status change information, refer to the beginning of the documentation for the definition of Condition standby Bool Whether the vpc creation is complete, the subnet under the vpc needs to wait for the vpc creation to complete other proceeding default Bool Whether it is the default vpc defaultLogicalSwitch String The default subnet under vpc router String The logical-router name for the vpc tcpLoadBalancer String TCP LB information for vpc udpLoadBalancer String UDP LB information for vpc tcpSessionLoadBalancer String TCP Session Hold LB Information for Vpc udpSessionLoadBalancer String UDP session hold LB information for Vpc subnets []String List of subnets for vpc vpcPeerings []String List of peer vpcs for vpc interconnection enableExternal Bool Whether the vpc is connected to an external switch VpcNatGateway \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value VpcNatGateway metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcNatSpec Vpc gateway specific configuration information VpcNatSpec \u00b6 Property Name Type Description vpc String Vpc name which the vpc gateway belongs to subnet String The name of the subnet to which the gateway pod belongs lanIp String The IP address assigned to the gateway pod selector []String Standard Kubernetes selector match information tolerations []VpcNatToleration Standard Kubernetes tolerance information VpcNatToleration \u00b6 Property Name Type Description key String The key information of the taint tolerance operator String Takes the value of Exists or Equal value String The value information of the taint tolerance effect String The effect of the taint tolerance, takes the value of NoExecute , NoSchedule , or PreferNoSchedule tolerationSeconds Int64 The amount of time the pod can continue to run on the node after the taint is added The meaning of the above tolerance fields can be found in the official Kubernetes documentation Taint and Tolerance . IptablesEIP \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesEIP metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesEipSpec IptablesEIP specific configuration information used by vpc gateway status IptablesEipStatus IptablesEIP status information used by vpc gateway IptablesEipSpec \u00b6 Property Name Type Description v4ip String IptablesEIP v4 address v6ip String IptablesEIP v6 address macAddress String The assigned mac address, not actually used natGwDp String Vpc gateway name IptablesEipStatus \u00b6 Property Name Type Description ready Bool Whether IptablesEIP is configured complete ip String The IP address used by IptablesEIP, currently only IPv4 addresses are supported redo String IptablesEIP crd creation or update time nat String The type of IptablesEIP, either fip , snat , or dnat conditions []IptablesEIPCondition IptablesEIP status change information, refer to the beginning of the documentation for the definition of Condition IptablesFIPRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesFIPRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesFIPRuleSpec The IptablesFIPRule specific configuration information used by vpc gateway status IptablesFIPRuleStatus IptablesFIPRule status information used by vpc gateway IptablesFIPRuleSpec \u00b6 Property Name Type Description eip String Name of the IptablesEIP used for IptablesFIPRule internalIp String The corresponding internal IP address IptablesFIPRuleStatus \u00b6 Property Name Type Description ready Bool Whether IptablesFIPRule is configured or not v4ip String The v4 IP address used by IptablesEIP v6ip String The v6 IP address used by IptablesEIP natGwDp String Vpc gateway name redo String IptablesFIPRule crd creation or update time conditions []IptablesFIPRuleCondition IptablesFIPRule status change information, refer to the beginning of the documentation for the definition of Condition IptablesSnatRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesSnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesSnatRuleSpec The IptablesSnatRule specific configuration information used by the vpc gateway status IptablesSnatRuleStatus IptablesSnatRule status information used by vpc gateway IptablesSnatRuleSpec \u00b6 Property Name Type Description eip String Name of the IptablesEIP used by IptablesSnatRule internalIp String IptablesSnatRule's corresponding internal IP address IptablesSnatRuleStatus \u00b6 Property Name Type Description ready Bool Whether the configuration is complete v4ip String The v4 IP address used by IptablesSnatRule v6ip String The v6 IP address used by IptablesSnatRule natGwDp String Vpc gateway name redo String IptablesSnatRule crd creation or update time conditions []IptablesSnatRuleCondition IptablesSnatRule status change information, refer to the beginning of the documentation for the definition of Condition IptablesDnatRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesDnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesDnatRuleSpec The IptablesDnatRule specific configuration information used by vpc gateway status IptablesDnatRuleStatus IptablesDnatRule status information used by vpc gateway IptablesDnatRuleSpec \u00b6 Property Name Type Description eip Sting Name of IptablesEIP used by IptablesDnatRule externalPort Sting External port used by IptablesDnatRule protocol Sting Vpc gateway dnat protocol type internalIp Sting Internal IP address used by IptablesDnatRule internalPort Sting Internal port used by IptablesDnatRule IptablesDnatRuleStatus \u00b6 Property Name Type Description ready Bool Whether the configuration is complete v4ip String The v4 IP address used by IptablesDnatRule v6ip String The v6 IP address used by IptablesDnatRule natGwDp String Vpc gateway name redo String IptablesDnatRule crd creation or update time conditions []IptablesDnatRuleCondition IptablesDnatRule Status change information, refer to the beginning of the documentation for the definition of Condition VpcDns \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value VpcDns metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcDnsSpec VpcDns specific configuration information status VpcDnsStatus VpcDns status information VpcDnsSpec \u00b6 Property Name Type Description vpc String Name of the vpc where VpcDns is located subnet String The subnet name of the address assigned to the VpcDns pod VpcDnsStatus \u00b6 Property Name Type Description conditions []VpcDnsCondition VpcDns status change information, refer to the beginning of the document for the definition of Condition active Bool Whether VpcDns is in use For detailed documentation on the use of VpcDns, see Customizing VPC DNS . SwitchLBRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value SwitchLBRule metadata ObjectMeta Standard Kubernetes resource metadata information spec SwitchLBRuleSpec SwitchLBRule specific configuration information status SwitchLBRuleStatus SwitchLBRule status information SwitchLBRuleSpec \u00b6 Property Name Type Description vip String Vip address of SwitchLBRule namespace String SwitchLBRule's namespace selector []String Standard Kubernetes selector match information sessionAffinity String Standard Kubernetes service sessionAffinity value ports []SlrPort List of SwitchLBRule ports For detailed configuration information of SwitchLBRule, you can refer to Customizing VPC Internal Load Balancing . SlrPort \u00b6 Property Name Type Description name String Port name port Int32 Port number targetPort Int32 Target port of SwitchLBRule protocol String Protocol type SwitchLBRuleStatus \u00b6 Property Name Type Description conditions []SwitchLBRuleCondition SwitchLBRule status change information, refer to the beginning of the document for the definition of Condition ports String Port information service String Name of the service Security Group and Vip \u00b6 SecurityGroup \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have a value of SecurityGroup metadata ObjectMeta Standard Kubernetes resource metadata information spec SecurityGroupSpec Security Group specific configuration information status SecurityGroupStatus Security group status information SecurityGroupSpec \u00b6 Property Name Type Description ingressRules []*SgRule Inbound security group rules egressRules []*SgRule Outbound security group rules allowSameGroupTraffic Bool Whether lsps in the same security group can interoperate and whether traffic rules need to be updated SgRule \u00b6 Property Name Type Description ipVersion String IP version number, ipv4 or ipv6 protocol String The value of icmp , tcp , or udp priority Int Acl priority. The value range is 1-200, the smaller the value, the higher the priority. remoteType String The value is either address or securityGroup remoteAddress String The address of the other side remoteSecurityGroup String The name of security group on the other side portRangeMin Int The starting value of the port range, the minimum value is 1. portRangeMax Int The ending value of the port range, the maximum value is 65535. policy String The value is allow or drop SecurityGroupStatus \u00b6 Property Name Type Description portGroup String The name of the port-group for the security group allowSameGroupTraffic Bool Whether lsps in the same security group can interoperate, and whether the security group traffic rules need to be updated ingressMd5 String The MD5 value of the inbound security group rule egressMd5 String The MD5 value of the outbound security group rule ingressLastSyncSuccess Bool Whether the last synchronization of the inbound rule was successful egressLastSyncSuccess Bool Whether the last synchronization of the outbound rule was successful Vip \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vip metadata ObjectMeta Standard Kubernetes resource metadata information spec VipSpec Vip specific configuration information status VipStatus Vip status information VipSpec \u00b6 Property Name Type Description namespace String Vip's namespace subnet String Vip's subnet v4ip String Vip IPv4 ip address v6ip String Vip IPv6 ip address macAddress String Vip mac address parentV4ip String Not currently in use parentV6ip String Not currently in use parentMac String Not currently in use attachSubnets []String This field is deprecated and no longer used VipStatus \u00b6 Property Name Type Description conditions []VipCondition Vip status change information, refer to the beginning of the documentation for the definition of Condition ready Bool Vip is ready or not v4ip String Vip IPv4 ip address, should be the same as the spec field v6ip String Vip IPv6 ip address, should be the same as the spec field mac String The vip mac address, which should be the same as the spec field pv4ip String Not currently used pv6ip String Not currently used pmac String Not currently used OvnEip \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnEip metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnEipSpec OvnEip specific configuration information for default vpc status OvnEipStatus OvnEip status information for default vpc OvnEipSpec \u00b6 Property Name Type Description externalSubnet String OvnEip's subnet name v4ip String OvnEip IP address macAddress String OvnEip Mac address type String OvnEip use type, the value can be fip , snat or lrp OvnEipStatus \u00b6 Property Name Type Description conditions []OvnEipCondition OvnEip status change information, refer to the beginning of the documentation for the definition of Condition v4ip String The IPv4 ip address used by ovnEip macAddress String Mac address used by ovnEip OvnFip \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnFip metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnFipSpec OvnFip specific configuration information in default vpc status OvnFipStatus OvnFip status information in default vpc OvnFipSpec \u00b6 Property Name Type Description ovnEip String Name of the bound ovnEip ipName String The IP crd name corresponding to the bound Pod OvnFipStatus \u00b6 Property Name Type Description ready Bool OvnFip is ready or not v4Eip String Name of the ovnEip to which ovnFip is bound v4Ip String The ovnEip address currently in use macAddress String OvnFip's configured mac address vpc String The name of the vpc where ovnFip is located conditions []OvnFipCondition OvnFip status change information, refer to the beginning of the document for the definition of Condition OvnSnatRule \u00b6 Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnSnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnSnatRuleSpec OvnSnatRule specific configuration information in default vpc status OvnSnatRuleStatus OvnSnatRule status information in default vpc OvnSnatRuleSpec \u00b6 Property Name Type Description ovnEip String Name of the ovnEip to which ovnSnatRule is bound vpcSubnet String The name of the subnet configured by ovnSnatRule ipName String The IP crd name corresponding to the ovnSnatRule bound Pod OvnSnatRuleStatus \u00b6 Property Name Type Description ready Bool OvnSnatRule is ready or not v4Eip String The ovnEip address to which ovnSnatRule is bound v4IpCidr String The cidr address used to configure snat in the logical-router vpc String The name of the vpc where ovnSnatRule is located conditions []OvnSnatRuleCondition OvnSnatRule status change information, refer to the beginning of the document for the definition of Condition \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Kube-OVN API Reference"},{"location":"en/reference/kube-ovn-api/#kube-ovn-api-reference","text":"Based on Kube-OVN v1.12.0, we have compiled a list of CRD resources supported by Kube-OVN, listing the types and meanings of each field of CRD definition for reference.","title":"Kube-OVN API Reference"},{"location":"en/reference/kube-ovn-api/#generic-condition-definition","text":"Property Name Type Description type String Type of status status String The value of status, in the range of True , False or Unknown reason String The reason for the status change message String The specific message of the status change lastUpdateTime Time The last time the status was updated lastTransitionTime Time Time of last status type change In each CRD definition, the Condition field in Status follows the above format, so we explain it in advance.","title":"Generic Condition Definition"},{"location":"en/reference/kube-ovn-api/#subnet-definition","text":"","title":"Subnet Definition"},{"location":"en/reference/kube-ovn-api/#subnet","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Subnet metadata ObjectMeta Standard Kubernetes resource metadata information spec SubnetSpec Subnet specific configuration information status SubnetStatus Subnet status information","title":"Subnet"},{"location":"en/reference/kube-ovn-api/#subnetspec","text":"Property Name Type Description default Bool Whether this subnet is the default subnet vpc String The vpc which the subnet belongs to, default is ovn-cluster protocol String IP protocol, the value is in the range of IPv4 , IPv6 or Dual namespaces []String The list of namespaces bound to this subnet cidrBlock String The range of the subnet, e.g. 10.16.0.0/16 gateway String The gateway address of the subnet, the default value is the first available address under the CIDRBlock of the subnet excludeIps []String The range of addresses under this subnet that will not be automatically assigned provider String Default value is ovn . In the case of multiple NICs, the value is <name>.<namespace> of the NetworkAttachmentDefinition, Kube-OVN will use this information to find the corresponding subnet resource gatewayType String The gateway type in overlay mode, either distributed or centralized gatewayNode String The gateway node when the gateway mode is centralized, node names can be comma-separated natOutgoing Bool Whether the outgoing traffic is NAT externalEgressGateway String The address of the external gateway. This parameter and the natOutgoing parameter cannot be set at the same time policyRoutingPriority Uint32 Policy route priority. Used to control the forwarding of traffic to the external gateway address after the subnet gateway policyRoutingTableID Uint32 The TableID of the local policy routing table, should be different for each subnet to avoid conflicts private Bool Whether the subnet is a private subnet, which denies access to addresses inside the subnet if the subnet is private allowSubnets []String If the subnet is a private subnet, the set of addresses that are allowed to access the subnet vlan String The name of vlan to which the subnet is bound vips []String The virtual-ip parameter information for virtual type lsp on the subnet logicalGateway Bool Whether to enable logical gateway disableGatewayCheck Bool Whether to skip the gateway connectivity check when creating a pod disableInterConnection Bool Whether to enable subnet interconnection across clusters enableDHCP Bool Whether to configure dhcp configuration options for lsps belong this subnet dhcpV4Options String The DHCP_Options record associated with lsp dhcpv4_options on the subnet dhcpV6Options String The DHCP_Options record associated with lsp dhcpv6_options on the subnet enableIPv6RA Bool Whether to configure the ipv6_ra_configs parameter for the lrp port of the router connected to the subnet ipv6RAConfigs String The ipv6_ra_configs parameter configuration for the lrp port of the router connected to the subnet acls []Acl The acls record associated with the logical-switch of the subnet u2oInterconnection Bool Whether to enable interconnection mode for Overlay/Underlay enableLb *Bool Whether the logical-switch of the subnet is associated with load-balancer records enableEcmp Bool Centralized subnet, whether to enable ECMP routing","title":"SubnetSpec"},{"location":"en/reference/kube-ovn-api/#acl","text":"Property Name Type Description direction String Restrict the direction of acl, which value is from-lport or to-lport priority Int Acl priority, in the range 0 to 32767 match String Acl rule match expression action String The action of the rule, which value is in the range of allow-related , allow-stateless , allow , drop , reject","title":"Acl"},{"location":"en/reference/kube-ovn-api/#subnetstatus","text":"Property Name Type Description conditions []SubnetCondition Subnet status change information, refer to the beginning of the document for the definition of Condition v4AvailableIPs Float64 Number of available IPv4 IPs v4availableIPrange String The available range of IPv4 addresses on the subnet v4UsingIPs Float64 Number of used IPv4 IPs v4usingIPrange String Used IPv4 address ranges on the subnet v6AvailableIPs Float64 Number of available IPv6 IPs v6availableIPrange String The available range of IPv6 addresses on the subnet v6UsingIPs Float64 Number of used IPv6 IPs v6usingIPrange String Used IPv6 address ranges on the subnet sctivateGateway String The currently working gateway node in centralized subnet of master-backup mode dhcpV4OptionsUUID String The DHCP_Options record identifier associated with the lsp dhcpv4_options on the subnet dhcpV6OptionsUUID String The DHCP_Options record identifier associated with the lsp dhcpv6_options on the subnet u2oInterconnectionIP String The IP address used for interconnection when Overlay/Underlay interconnection mode is enabled","title":"SubnetStatus"},{"location":"en/reference/kube-ovn-api/#ip-definition","text":"","title":"IP Definition"},{"location":"en/reference/kube-ovn-api/#ip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IP metadata ObjectMeta Standard Kubernetes resource metadata information spec IPSpec IP specific configuration information","title":"IP"},{"location":"en/reference/kube-ovn-api/#ipsepc","text":"Property Name Type Description podName String Pod name which assigned with this IP namespace String The name of the namespace where the pod is bound subnet String The subnet which the ip belongs to attachSubnets []String The name of the other subnets attached to this primary IP (field deprecated) nodeName String The name of the node where the pod is bound ipAddress String IP address, in v4IP,v6IP format for dual-stack cases v4IPAddress String IPv4 IP address v6IPAddress String IPv6 IP address attachIPs []String Other IP addresses attached to this primary IP (field is deprecated) macAddress String The Mac address of the bound pod attachMacs []String Other Mac addresses attached to this primary IP (field deprecated) containerID String The Container ID corresponding to the bound pod podType String Special workload pod, can be StatefulSet , VirtualMachine or empty","title":"IPSepc"},{"location":"en/reference/kube-ovn-api/#underlay-configuration","text":"","title":"Underlay configuration"},{"location":"en/reference/kube-ovn-api/#vlan","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all instances of this resource will be kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vlan metadata ObjectMeta Standard Kubernetes resource metadata information spec VlanSpec Vlan specific configuration information status VlanStatus Vlan status information","title":"Vlan"},{"location":"en/reference/kube-ovn-api/#vlanspec","text":"Property Name Type Description id Int Vlan tag number, in the range of 0~4096 provider String The name of the ProviderNetwork to which the vlan is bound","title":"VlanSpec"},{"location":"en/reference/kube-ovn-api/#vlanstatus","text":"Property Name Type Description subnets []String The list of subnets to which the vlan is bound conditions []VlanCondition Vlan status change information, refer to the beginning of the document for the definition of Condition","title":"VlanStatus"},{"location":"en/reference/kube-ovn-api/#providernetwork","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value ProviderNetwork metadata ObjectMeta Standard Kubernetes resource metadata information spec ProviderNetworkSpec ProviderNetwork specific configuration information status ProviderNetworkStatus ProviderNetwork status information","title":"ProviderNetwork"},{"location":"en/reference/kube-ovn-api/#providernetworkspec","text":"Property Name Type Description defaultInterface String The name of the NIC interface used by default for this bridge network customInterfaces []CustomInterface The special NIC configuration used by this bridge network excludeNodes []String The names of the nodes that will not be bound to this bridge network exchangeLinkName Bool Whether to exchange the bridge NIC and the corresponding OVS bridge name","title":"ProviderNetworkSpec"},{"location":"en/reference/kube-ovn-api/#custominterface","text":"Property Name Type Description interface String NIC interface name used for underlay nodes []String List of nodes using the custom NIC interface","title":"CustomInterface"},{"location":"en/reference/kube-ovn-api/#providernetworkstatus","text":"Property Name Type Description ready Bool Whether the current bridge network is in the ready state readyNodes []String The name of the node whose bridge network is ready notReadyNodes []String The name of the node whose bridge network is not ready vlans []String The name of the vlan to which the bridge network is bound conditions []ProviderNetworkCondition ProviderNetwork status change information, refer to the beginning of the document for the definition of Condition","title":"ProviderNetworkStatus"},{"location":"en/reference/kube-ovn-api/#vpc-definition","text":"","title":"Vpc Definition"},{"location":"en/reference/kube-ovn-api/#vpc","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vpc metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcSpec Vpc specific configuration information status VpcStatus Vpc status information","title":"Vpc"},{"location":"en/reference/kube-ovn-api/#vpcspec","text":"Property Name Type Description namespaces []String List of namespaces bound by Vpc staticRoutes []*StaticRoute The static route information configured under Vpc policyRoutes []*PolicyRoute The policy route information configured under Vpc vpcPeerings []*VpcPeering Vpc interconnection information enableExternal Bool Whether vpc is connected to an external switch","title":"VpcSpec"},{"location":"en/reference/kube-ovn-api/#staticroute","text":"Property Name Type Description policy String Routing policy, takes the value of policySrc or policyDst cidr String Routing cidr value nextHopIP String The next hop information of the route","title":"StaticRoute"},{"location":"en/reference/kube-ovn-api/#policyroute","text":"Property Name Type Description priority Int32 Priority for policy route match String Match expression for policy route action String Action for policy route, the value is in the range of allow , drop , reroute nextHopIP String The next hop of the policy route, separated by commas in the case of ECMP routing","title":"PolicyRoute"},{"location":"en/reference/kube-ovn-api/#vpcpeering","text":"Property Name Type Description remoteVpc String Name of the interconnected peering vpc localConnectIP String The local ip for vpc used to connect to peer vpc","title":"VpcPeering"},{"location":"en/reference/kube-ovn-api/#vpcstatus","text":"Property Name Type Description conditions []VpcCondition Vpc status change information, refer to the beginning of the documentation for the definition of Condition standby Bool Whether the vpc creation is complete, the subnet under the vpc needs to wait for the vpc creation to complete other proceeding default Bool Whether it is the default vpc defaultLogicalSwitch String The default subnet under vpc router String The logical-router name for the vpc tcpLoadBalancer String TCP LB information for vpc udpLoadBalancer String UDP LB information for vpc tcpSessionLoadBalancer String TCP Session Hold LB Information for Vpc udpSessionLoadBalancer String UDP session hold LB information for Vpc subnets []String List of subnets for vpc vpcPeerings []String List of peer vpcs for vpc interconnection enableExternal Bool Whether the vpc is connected to an external switch","title":"VpcStatus"},{"location":"en/reference/kube-ovn-api/#vpcnatgateway","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value VpcNatGateway metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcNatSpec Vpc gateway specific configuration information","title":"VpcNatGateway"},{"location":"en/reference/kube-ovn-api/#vpcnatspec","text":"Property Name Type Description vpc String Vpc name which the vpc gateway belongs to subnet String The name of the subnet to which the gateway pod belongs lanIp String The IP address assigned to the gateway pod selector []String Standard Kubernetes selector match information tolerations []VpcNatToleration Standard Kubernetes tolerance information","title":"VpcNatSpec"},{"location":"en/reference/kube-ovn-api/#vpcnattoleration","text":"Property Name Type Description key String The key information of the taint tolerance operator String Takes the value of Exists or Equal value String The value information of the taint tolerance effect String The effect of the taint tolerance, takes the value of NoExecute , NoSchedule , or PreferNoSchedule tolerationSeconds Int64 The amount of time the pod can continue to run on the node after the taint is added The meaning of the above tolerance fields can be found in the official Kubernetes documentation Taint and Tolerance .","title":"VpcNatToleration"},{"location":"en/reference/kube-ovn-api/#iptableseip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesEIP metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesEipSpec IptablesEIP specific configuration information used by vpc gateway status IptablesEipStatus IptablesEIP status information used by vpc gateway","title":"IptablesEIP"},{"location":"en/reference/kube-ovn-api/#iptableseipspec","text":"Property Name Type Description v4ip String IptablesEIP v4 address v6ip String IptablesEIP v6 address macAddress String The assigned mac address, not actually used natGwDp String Vpc gateway name","title":"IptablesEipSpec"},{"location":"en/reference/kube-ovn-api/#iptableseipstatus","text":"Property Name Type Description ready Bool Whether IptablesEIP is configured complete ip String The IP address used by IptablesEIP, currently only IPv4 addresses are supported redo String IptablesEIP crd creation or update time nat String The type of IptablesEIP, either fip , snat , or dnat conditions []IptablesEIPCondition IptablesEIP status change information, refer to the beginning of the documentation for the definition of Condition","title":"IptablesEipStatus"},{"location":"en/reference/kube-ovn-api/#iptablesfiprule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesFIPRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesFIPRuleSpec The IptablesFIPRule specific configuration information used by vpc gateway status IptablesFIPRuleStatus IptablesFIPRule status information used by vpc gateway","title":"IptablesFIPRule"},{"location":"en/reference/kube-ovn-api/#iptablesfiprulespec","text":"Property Name Type Description eip String Name of the IptablesEIP used for IptablesFIPRule internalIp String The corresponding internal IP address","title":"IptablesFIPRuleSpec"},{"location":"en/reference/kube-ovn-api/#iptablesfiprulestatus","text":"Property Name Type Description ready Bool Whether IptablesFIPRule is configured or not v4ip String The v4 IP address used by IptablesEIP v6ip String The v6 IP address used by IptablesEIP natGwDp String Vpc gateway name redo String IptablesFIPRule crd creation or update time conditions []IptablesFIPRuleCondition IptablesFIPRule status change information, refer to the beginning of the documentation for the definition of Condition","title":"IptablesFIPRuleStatus"},{"location":"en/reference/kube-ovn-api/#iptablessnatrule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesSnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesSnatRuleSpec The IptablesSnatRule specific configuration information used by the vpc gateway status IptablesSnatRuleStatus IptablesSnatRule status information used by vpc gateway","title":"IptablesSnatRule"},{"location":"en/reference/kube-ovn-api/#iptablessnatrulespec","text":"Property Name Type Description eip String Name of the IptablesEIP used by IptablesSnatRule internalIp String IptablesSnatRule's corresponding internal IP address","title":"IptablesSnatRuleSpec"},{"location":"en/reference/kube-ovn-api/#iptablessnatrulestatus","text":"Property Name Type Description ready Bool Whether the configuration is complete v4ip String The v4 IP address used by IptablesSnatRule v6ip String The v6 IP address used by IptablesSnatRule natGwDp String Vpc gateway name redo String IptablesSnatRule crd creation or update time conditions []IptablesSnatRuleCondition IptablesSnatRule status change information, refer to the beginning of the documentation for the definition of Condition","title":"IptablesSnatRuleStatus"},{"location":"en/reference/kube-ovn-api/#iptablesdnatrule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource have the value IptablesDnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec IptablesDnatRuleSpec The IptablesDnatRule specific configuration information used by vpc gateway status IptablesDnatRuleStatus IptablesDnatRule status information used by vpc gateway","title":"IptablesDnatRule"},{"location":"en/reference/kube-ovn-api/#iptablesdnatrulespec","text":"Property Name Type Description eip Sting Name of IptablesEIP used by IptablesDnatRule externalPort Sting External port used by IptablesDnatRule protocol Sting Vpc gateway dnat protocol type internalIp Sting Internal IP address used by IptablesDnatRule internalPort Sting Internal port used by IptablesDnatRule","title":"IptablesDnatRuleSpec"},{"location":"en/reference/kube-ovn-api/#iptablesdnatrulestatus","text":"Property Name Type Description ready Bool Whether the configuration is complete v4ip String The v4 IP address used by IptablesDnatRule v6ip String The v6 IP address used by IptablesDnatRule natGwDp String Vpc gateway name redo String IptablesDnatRule crd creation or update time conditions []IptablesDnatRuleCondition IptablesDnatRule Status change information, refer to the beginning of the documentation for the definition of Condition","title":"IptablesDnatRuleStatus"},{"location":"en/reference/kube-ovn-api/#vpcdns","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value VpcDns metadata ObjectMeta Standard Kubernetes resource metadata information spec VpcDnsSpec VpcDns specific configuration information status VpcDnsStatus VpcDns status information","title":"VpcDns"},{"location":"en/reference/kube-ovn-api/#vpcdnsspec","text":"Property Name Type Description vpc String Name of the vpc where VpcDns is located subnet String The subnet name of the address assigned to the VpcDns pod","title":"VpcDnsSpec"},{"location":"en/reference/kube-ovn-api/#vpcdnsstatus","text":"Property Name Type Description conditions []VpcDnsCondition VpcDns status change information, refer to the beginning of the document for the definition of Condition active Bool Whether VpcDns is in use For detailed documentation on the use of VpcDns, see Customizing VPC DNS .","title":"VpcDnsStatus"},{"location":"en/reference/kube-ovn-api/#switchlbrule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have this value as kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value SwitchLBRule metadata ObjectMeta Standard Kubernetes resource metadata information spec SwitchLBRuleSpec SwitchLBRule specific configuration information status SwitchLBRuleStatus SwitchLBRule status information","title":"SwitchLBRule"},{"location":"en/reference/kube-ovn-api/#switchlbrulespec","text":"Property Name Type Description vip String Vip address of SwitchLBRule namespace String SwitchLBRule's namespace selector []String Standard Kubernetes selector match information sessionAffinity String Standard Kubernetes service sessionAffinity value ports []SlrPort List of SwitchLBRule ports For detailed configuration information of SwitchLBRule, you can refer to Customizing VPC Internal Load Balancing .","title":"SwitchLBRuleSpec"},{"location":"en/reference/kube-ovn-api/#slrport","text":"Property Name Type Description name String Port name port Int32 Port number targetPort Int32 Target port of SwitchLBRule protocol String Protocol type","title":"SlrPort"},{"location":"en/reference/kube-ovn-api/#switchlbrulestatus","text":"Property Name Type Description conditions []SwitchLBRuleCondition SwitchLBRule status change information, refer to the beginning of the document for the definition of Condition ports String Port information service String Name of the service","title":"SwitchLBRuleStatus"},{"location":"en/reference/kube-ovn-api/#security-group-and-vip","text":"","title":"Security Group and Vip"},{"location":"en/reference/kube-ovn-api/#securitygroup","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have a value of SecurityGroup metadata ObjectMeta Standard Kubernetes resource metadata information spec SecurityGroupSpec Security Group specific configuration information status SecurityGroupStatus Security group status information","title":"SecurityGroup"},{"location":"en/reference/kube-ovn-api/#securitygroupspec","text":"Property Name Type Description ingressRules []*SgRule Inbound security group rules egressRules []*SgRule Outbound security group rules allowSameGroupTraffic Bool Whether lsps in the same security group can interoperate and whether traffic rules need to be updated","title":"SecurityGroupSpec"},{"location":"en/reference/kube-ovn-api/#sgrule","text":"Property Name Type Description ipVersion String IP version number, ipv4 or ipv6 protocol String The value of icmp , tcp , or udp priority Int Acl priority. The value range is 1-200, the smaller the value, the higher the priority. remoteType String The value is either address or securityGroup remoteAddress String The address of the other side remoteSecurityGroup String The name of security group on the other side portRangeMin Int The starting value of the port range, the minimum value is 1. portRangeMax Int The ending value of the port range, the maximum value is 65535. policy String The value is allow or drop","title":"SgRule"},{"location":"en/reference/kube-ovn-api/#securitygroupstatus","text":"Property Name Type Description portGroup String The name of the port-group for the security group allowSameGroupTraffic Bool Whether lsps in the same security group can interoperate, and whether the security group traffic rules need to be updated ingressMd5 String The MD5 value of the inbound security group rule egressMd5 String The MD5 value of the outbound security group rule ingressLastSyncSuccess Bool Whether the last synchronization of the inbound rule was successful egressLastSyncSuccess Bool Whether the last synchronization of the outbound rule was successful","title":"SecurityGroupStatus"},{"location":"en/reference/kube-ovn-api/#vip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value Vip metadata ObjectMeta Standard Kubernetes resource metadata information spec VipSpec Vip specific configuration information status VipStatus Vip status information","title":"Vip"},{"location":"en/reference/kube-ovn-api/#vipspec","text":"Property Name Type Description namespace String Vip's namespace subnet String Vip's subnet v4ip String Vip IPv4 ip address v6ip String Vip IPv6 ip address macAddress String Vip mac address parentV4ip String Not currently in use parentV6ip String Not currently in use parentMac String Not currently in use attachSubnets []String This field is deprecated and no longer used","title":"VipSpec"},{"location":"en/reference/kube-ovn-api/#vipstatus","text":"Property Name Type Description conditions []VipCondition Vip status change information, refer to the beginning of the documentation for the definition of Condition ready Bool Vip is ready or not v4ip String Vip IPv4 ip address, should be the same as the spec field v6ip String Vip IPv6 ip address, should be the same as the spec field mac String The vip mac address, which should be the same as the spec field pv4ip String Not currently used pv6ip String Not currently used pmac String Not currently used","title":"VipStatus"},{"location":"en/reference/kube-ovn-api/#ovneip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnEip metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnEipSpec OvnEip specific configuration information for default vpc status OvnEipStatus OvnEip status information for default vpc","title":"OvnEip"},{"location":"en/reference/kube-ovn-api/#ovneipspec","text":"Property Name Type Description externalSubnet String OvnEip's subnet name v4ip String OvnEip IP address macAddress String OvnEip Mac address type String OvnEip use type, the value can be fip , snat or lrp","title":"OvnEipSpec"},{"location":"en/reference/kube-ovn-api/#ovneipstatus","text":"Property Name Type Description conditions []OvnEipCondition OvnEip status change information, refer to the beginning of the documentation for the definition of Condition v4ip String The IPv4 ip address used by ovnEip macAddress String Mac address used by ovnEip","title":"OvnEipStatus"},{"location":"en/reference/kube-ovn-api/#ovnfip","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources are kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnFip metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnFipSpec OvnFip specific configuration information in default vpc status OvnFipStatus OvnFip status information in default vpc","title":"OvnFip"},{"location":"en/reference/kube-ovn-api/#ovnfipspec","text":"Property Name Type Description ovnEip String Name of the bound ovnEip ipName String The IP crd name corresponding to the bound Pod","title":"OvnFipSpec"},{"location":"en/reference/kube-ovn-api/#ovnfipstatus","text":"Property Name Type Description ready Bool OvnFip is ready or not v4Eip String Name of the ovnEip to which ovnFip is bound v4Ip String The ovnEip address currently in use macAddress String OvnFip's configured mac address vpc String The name of the vpc where ovnFip is located conditions []OvnFipCondition OvnFip status change information, refer to the beginning of the document for the definition of Condition","title":"OvnFipStatus"},{"location":"en/reference/kube-ovn-api/#ovnsnatrule","text":"Property Name Type Description apiVersion String Standard Kubernetes version information field, all custom resources have kubeovn.io/v1 kind String Standard Kubernetes resource type field, all instances of this resource will have the value OvnSnatRule metadata ObjectMeta Standard Kubernetes resource metadata information spec OvnSnatRuleSpec OvnSnatRule specific configuration information in default vpc status OvnSnatRuleStatus OvnSnatRule status information in default vpc","title":"OvnSnatRule"},{"location":"en/reference/kube-ovn-api/#ovnsnatrulespec","text":"Property Name Type Description ovnEip String Name of the ovnEip to which ovnSnatRule is bound vpcSubnet String The name of the subnet configured by ovnSnatRule ipName String The IP crd name corresponding to the ovnSnatRule bound Pod","title":"OvnSnatRuleSpec"},{"location":"en/reference/kube-ovn-api/#ovnsnatrulestatus","text":"Property Name Type Description ready Bool OvnSnatRule is ready or not v4Eip String The ovnEip address to which ovnSnatRule is bound v4IpCidr String The cidr address used to configure snat in the logical-router vpc String The name of the vpc where ovnSnatRule is located conditions []OvnSnatRuleCondition OvnSnatRule status change information, refer to the beginning of the document for the definition of Condition \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OvnSnatRuleStatus"},{"location":"en/reference/kube-ovn-pinger-args/","text":"Kube-OVN-Pinger args Reference \u00b6 Based on the Kube-OVN v1.12.0 version, We have compiled the parameters supported by Kube-ovn-pinger, and listed the value types, meanings, and default values of each field defined by the parameters for reference Args Describeption \u00b6 Arg Name Type Description Default Value port Int metrics port 8080 kubeconfig String Path to kubeconfig file with authorization and master location information. If not set use the inCluster token. \"\" ds-namespace String kube-ovn-pinger daemonset namespace \"kube-system\" ds-name String kube-ovn-pinger daemonset name \"kube-ovn-pinger\" interval Int interval seconds between consecutive pings 5 mode String server or job Mode \"server\" exit-code Int exit code when failure happens 0 internal-dns String check dns from pod \"kubernetes.default\" external-dns String check external dns resolve from pod \"\" external-address String check ping connection to an external address \"114.114.114.114\" network-mode String The cni plugin current cluster used \"kube-ovn\" enable-metrics Bool Whether to support metrics query true ovs.timeout Int Timeout on JSON-RPC requests to OVS. 2 system.run.dir String OVS default run directory. \"/var/run/openvswitch\" database.vswitch.name String The name of OVS db. \"Open_vSwitch\" database.vswitch.socket.remote String JSON-RPC unix socket to OVS db. \"unix:/var/run/openvswitch/db.sock\" database.vswitch.file.data.path String OVS db file. \"/etc/openvswitch/conf.db\" database.vswitch.file.log.path String OVS db log file. \"/var/log/openvswitch/ovsdb-server.log\" database.vswitch.file.pid.path String OVS db process id file. \"/var/run/openvswitch/ovsdb-server.pid\" database.vswitch.file.system.id.path String OVS system id file. \"/etc/openvswitch/system-id.conf\" service.vswitchd.file.log.path String OVS vswitchd daemon log file. \"/var/log/openvswitch/ovs-vswitchd.log\" service.vswitchd.file.pid.path String OVS vswitchd daemon process id file. \"/var/run/openvswitch/ovs-vswitchd.pid\" service.ovncontroller.file.log.path String OVN controller daemon log file. \"/var/log/ovn/ovn-controller.log\" service.ovncontroller.file.pid.path String OVN controller daemon process id file. \"/var/run/ovn/ovn-controller.pid\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kube-ovn-pinger args description"},{"location":"en/reference/kube-ovn-pinger-args/#kube-ovn-pinger-args-reference","text":"Based on the Kube-OVN v1.12.0 version, We have compiled the parameters supported by Kube-ovn-pinger, and listed the value types, meanings, and default values of each field defined by the parameters for reference","title":"Kube-OVN-Pinger args Reference"},{"location":"en/reference/kube-ovn-pinger-args/#args-describeption","text":"Arg Name Type Description Default Value port Int metrics port 8080 kubeconfig String Path to kubeconfig file with authorization and master location information. If not set use the inCluster token. \"\" ds-namespace String kube-ovn-pinger daemonset namespace \"kube-system\" ds-name String kube-ovn-pinger daemonset name \"kube-ovn-pinger\" interval Int interval seconds between consecutive pings 5 mode String server or job Mode \"server\" exit-code Int exit code when failure happens 0 internal-dns String check dns from pod \"kubernetes.default\" external-dns String check external dns resolve from pod \"\" external-address String check ping connection to an external address \"114.114.114.114\" network-mode String The cni plugin current cluster used \"kube-ovn\" enable-metrics Bool Whether to support metrics query true ovs.timeout Int Timeout on JSON-RPC requests to OVS. 2 system.run.dir String OVS default run directory. \"/var/run/openvswitch\" database.vswitch.name String The name of OVS db. \"Open_vSwitch\" database.vswitch.socket.remote String JSON-RPC unix socket to OVS db. \"unix:/var/run/openvswitch/db.sock\" database.vswitch.file.data.path String OVS db file. \"/etc/openvswitch/conf.db\" database.vswitch.file.log.path String OVS db log file. \"/var/log/openvswitch/ovsdb-server.log\" database.vswitch.file.pid.path String OVS db process id file. \"/var/run/openvswitch/ovsdb-server.pid\" database.vswitch.file.system.id.path String OVS system id file. \"/etc/openvswitch/system-id.conf\" service.vswitchd.file.log.path String OVS vswitchd daemon log file. \"/var/log/openvswitch/ovs-vswitchd.log\" service.vswitchd.file.pid.path String OVS vswitchd daemon process id file. \"/var/run/openvswitch/ovs-vswitchd.pid\" service.ovncontroller.file.log.path String OVN controller daemon log file. \"/var/log/ovn/ovn-controller.log\" service.ovncontroller.file.pid.path String OVN controller daemon process id file. \"/var/run/ovn/ovn-controller.pid\" \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Args Describeption"},{"location":"en/reference/metrics/","text":"Metrics \u00b6 This document lists all the monitoring metrics provided by Kube-OVN. ovn-monitor \u00b6 OVN status metrics: Type Metric Description Gauge kube_ovn_ovn_status OVN Health Status. The values are: (2) for standby or follower, (1) for active or leader, (0) for unhealthy. Gauge kube_ovn_failed_req_count The number of failed requests to OVN stack. Gauge kube_ovn_log_file_size The size of a log file associated with an OVN component. Gauge kube_ovn_db_file_size The size of a database file associated with an OVN component. Gauge kube_ovn_chassis_info Whether the OVN chassis is up (1) or down (0), together with additional information about the chassis. Gauge kube_ovn_db_status The status of OVN NB/SB DB, (1) for healthy, (0) for unhealthy. Gauge kube_ovn_logical_switch_info The information about OVN logical switch. This metric is always up (1). Gauge kube_ovn_logical_switch_external_id Provides the external IDs and values associated with OVN logical switches. This metric is always up (1). Gauge kube_ovn_logical_switch_port_binding Provides the association between a logical switch and a logical switch port. This metric is always up (1). Gauge kube_ovn_logical_switch_tunnel_key The value of the tunnel key associated with the logical switch. Gauge kube_ovn_logical_switch_ports_num The number of logical switch ports connected to the OVN logical switch. Gauge kube_ovn_logical_switch_port_info The information about OVN logical switch port. This metric is always up (1). Gauge kube_ovn_logical_switch_port_tunnel_key The value of the tunnel key associated with the logical switch port. Gauge kube_ovn_cluster_enabled Is OVN clustering enabled (1) or not (0). Gauge kube_ovn_cluster_role A metric with a constant '1' value labeled by server role. Gauge kube_ovn_cluster_status A metric with a constant '1' value labeled by server status. Gauge kube_ovn_cluster_term The current raft term known by this server. Gauge kube_ovn_cluster_leader_self Is this server consider itself a leader (1) or not (0). Gauge kube_ovn_cluster_vote_self Is this server voted itself as a leader (1) or not (0). Gauge kube_ovn_cluster_election_timer The current election timer value. Gauge kube_ovn_cluster_log_not_committed The number of log entries not yet committed by this server. Gauge kube_ovn_cluster_log_not_applied The number of log entries not yet applied by this server. Gauge kube_ovn_cluster_log_index_start The log entry index start value associated with this server. Gauge kube_ovn_cluster_log_index_next The log entry index next value associated with this server. Gauge kube_ovn_cluster_inbound_connections_total The total number of inbound connections to the server. Gauge kube_ovn_cluster_outbound_connections_total The total number of outbound connections from the server. Gauge kube_ovn_cluster_inbound_connections_error_total The total number of failed inbound connections to the server. Gauge kube_ovn_cluster_outbound_connections_error_total The total number of failed outbound connections from the server. ovs-monitor \u00b6 ovsdb and vswitchd status metrics: Type Metric Description Gauge ovs_status OVS Health Status. The values are: health(1), unhealthy(0). Gauge ovs_info This metric provides basic information about OVS. It is always set to 1. Gauge failed_req_count The number of failed requests to OVS stack. Gauge log_file_size The size of a log file associated with an OVS component. Gauge db_file_size The size of a database file associated with an OVS component. Gauge datapath Represents an existing datapath. This metrics is always 1. Gauge dp_total Represents total number of datapaths on the system. Gauge dp_if Represents an existing datapath interface. This metrics is always 1. Gauge dp_if_total Represents the number of ports connected to the datapath. Gauge dp_flows_total The number of flows in a datapath. Gauge dp_flows_lookup_hit The number of incoming packets in a datapath matching existing flows in the datapath. Gauge dp_flows_lookup_missed The number of incoming packets in a datapath not matching any existing flow in the datapath. Gauge dp_flows_lookup_lost The number of incoming packets in a datapath destined for userspace process but subsequently dropped before reaching userspace. Gauge dp_masks_hit The total number of masks visited for matching incoming packets. Gauge dp_masks_total The number of masks in a datapath. Gauge dp_masks_hit_ratio The average number of masks visited per packet. It is the ration between hit and total number of packets processed by a datapath. Gauge interface Represents OVS interface. This is the primary metric for all other interface metrics. This metrics is always 1. Gauge interface_admin_state The administrative state of the physical network link of OVS interface. The values are: down(0), up(1), other(2). Gauge interface_link_state The state of the physical network link of OVS interface. The values are: down(0), up(1), other(2). Gauge interface_mac_in_use The MAC address in use by OVS interface. Gauge interface_mtu The currently configured MTU for OVS interface. Gauge interface_of_port Represents the OpenFlow port ID associated with OVS interface. Gauge interface_if_index Represents the interface index associated with OVS interface. Gauge interface_tx_packets Represents the number of transmitted packets by OVS interface. Gauge interface_tx_bytes Represents the number of transmitted bytes by OVS interface. Gauge interface_rx_packets Represents the number of received packets by OVS interface. Gauge interface_rx_bytes Represents the number of received bytes by OVS interface. Gauge interface_rx_crc_err Represents the number of CRC errors for the packets received by OVS interface. Gauge interface_rx_dropped Represents the number of input packets dropped by OVS interface. Gauge interface_rx_errors Represents the total number of packets with errors received by OVS interface. Gauge interface_rx_frame_err Represents the number of frame alignment errors on the packets received by OVS interface. Gauge interface_rx_missed_err Represents the number of packets with RX missed received by OVS interface. Gauge interface_rx_over_err Represents the number of packets with RX overrun received by OVS interface. Gauge interface_tx_dropped Represents the number of output packets dropped by OVS interface. Gauge interface_tx_errors Represents the total number of transmit errors by OVS interface. Gauge interface_collisions Represents the number of collisions on OVS interface. kube-ovn-pinger \u00b6 Network quality related metrics: Type Metric Description Gauge pinger_ovs_up If the ovs on the node is up Gauge pinger_ovs_down If the ovs on the node is down Gauge pinger_ovn_controller_up If the ovn_controller on the node is up Gauge pinger_ovn_controller_down If the ovn_controller on the node is down Gauge pinger_inconsistent_port_binding The number of mismatch port bindings between ovs and ovn-sb Gauge pinger_apiserver_healthy If the apiserver request is healthy on this node Gauge pinger_apiserver_unhealthy If the apiserver request is unhealthy on this node Histogram pinger_apiserver_latency_ms The latency ms histogram the node request apiserver Gauge pinger_internal_dns_healthy If the internal dns request is unhealthy on this node Gauge pinger_internal_dns_unhealthy If the internal dns request is unhealthy on this node Histogram pinger_internal_dns_latency_ms The latency ms histogram the node request internal dns Gauge pinger_external_dns_health If the external dns request is healthy on this node Gauge pinger_external_dns_unhealthy If the external dns request is unhealthy on this node Histogram pinger_external_dns_latency_ms The latency ms histogram the node request external dns Histogram pinger_pod_ping_latency_ms The latency ms histogram for pod peer ping Gauge pinger_pod_ping_lost_total The lost count for pod peer ping Gauge pinger_pod_ping_count_total The total count for pod peer ping Histogram pinger_node_ping_latency_ms The latency ms histogram for pod ping node Gauge pinger_node_ping_lost_total The lost count for pod ping node Gauge pinger_node_ping_count_total The total count for pod ping node Histogram pinger_external_ping_latency_ms The latency ms histogram for pod ping external address Gauge pinger_external_lost_total The lost count for pod ping external address kube-ovn-controller \u00b6 kube-ovn-controller status metrics\uff1a Type Metric Description Histogram rest_client_request_latency_seconds Request latency in seconds. Broken down by verb and URL Counter rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host Counter lists_total Total number of API lists done by the reflectors Summary list_duration_seconds How long an API list takes to return and decode for the reflectors Summary items_per_list How many items an API list returns to the reflectors Counter watches_total Total number of API watches done by the reflectors Counter short_watches_total Total number of short API watches done by the reflectors Summary watch_duration_seconds How long an API watch takes to return and decode for the reflectors Summary items_per_watch How many items an API watch returns to the reflectors Gauge last_resource_version Last resource version seen for the reflectors Histogram ovs_client_request_latency_milliseconds The latency histogram for ovs request Gauge subnet_available_ip_count The available num of ip address in subnet Gauge subnet_used_ip_count The used num of ip address in subnet kube-ovn-cni \u00b6 kube-ovn-cni status metrics: Type Metric Description Histogram cni_op_latency_seconds The latency seconds for cni operations Counter cni_wait_address_seconds_total Latency that cni wait controller to assign an address Counter cni_wait_connectivity_seconds_total Latency that cni wait address ready in overlay network Counter cni_wait_route_seconds_total Latency that cni wait controller to add routed annotation to pod Histogram rest_client_request_latency_seconds Request latency in seconds. Broken down by verb and URL Counter rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host Counter lists_total Total number of API lists done by the reflectors Summary list_duration_seconds How long an API list takes to return and decode for the reflectors Summary items_per_list How many items an API list returns to the reflectors Counter watches_total Total number of API watches done by the reflectors Counter short_watches_total Total number of short API watches done by the reflectors Summary watch_duration_seconds How long an API watch takes to return and decode for the reflectors Summary items_per_watch How many items an API watch returns to the reflectors Gauge last_resource_version Last resource version seen for the reflectors Histogram ovs_client_request_latency_milliseconds The latency histogram for ovs request \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Metrics"},{"location":"en/reference/metrics/#metrics","text":"This document lists all the monitoring metrics provided by Kube-OVN.","title":"Metrics"},{"location":"en/reference/metrics/#ovn-monitor","text":"OVN status metrics: Type Metric Description Gauge kube_ovn_ovn_status OVN Health Status. The values are: (2) for standby or follower, (1) for active or leader, (0) for unhealthy. Gauge kube_ovn_failed_req_count The number of failed requests to OVN stack. Gauge kube_ovn_log_file_size The size of a log file associated with an OVN component. Gauge kube_ovn_db_file_size The size of a database file associated with an OVN component. Gauge kube_ovn_chassis_info Whether the OVN chassis is up (1) or down (0), together with additional information about the chassis. Gauge kube_ovn_db_status The status of OVN NB/SB DB, (1) for healthy, (0) for unhealthy. Gauge kube_ovn_logical_switch_info The information about OVN logical switch. This metric is always up (1). Gauge kube_ovn_logical_switch_external_id Provides the external IDs and values associated with OVN logical switches. This metric is always up (1). Gauge kube_ovn_logical_switch_port_binding Provides the association between a logical switch and a logical switch port. This metric is always up (1). Gauge kube_ovn_logical_switch_tunnel_key The value of the tunnel key associated with the logical switch. Gauge kube_ovn_logical_switch_ports_num The number of logical switch ports connected to the OVN logical switch. Gauge kube_ovn_logical_switch_port_info The information about OVN logical switch port. This metric is always up (1). Gauge kube_ovn_logical_switch_port_tunnel_key The value of the tunnel key associated with the logical switch port. Gauge kube_ovn_cluster_enabled Is OVN clustering enabled (1) or not (0). Gauge kube_ovn_cluster_role A metric with a constant '1' value labeled by server role. Gauge kube_ovn_cluster_status A metric with a constant '1' value labeled by server status. Gauge kube_ovn_cluster_term The current raft term known by this server. Gauge kube_ovn_cluster_leader_self Is this server consider itself a leader (1) or not (0). Gauge kube_ovn_cluster_vote_self Is this server voted itself as a leader (1) or not (0). Gauge kube_ovn_cluster_election_timer The current election timer value. Gauge kube_ovn_cluster_log_not_committed The number of log entries not yet committed by this server. Gauge kube_ovn_cluster_log_not_applied The number of log entries not yet applied by this server. Gauge kube_ovn_cluster_log_index_start The log entry index start value associated with this server. Gauge kube_ovn_cluster_log_index_next The log entry index next value associated with this server. Gauge kube_ovn_cluster_inbound_connections_total The total number of inbound connections to the server. Gauge kube_ovn_cluster_outbound_connections_total The total number of outbound connections from the server. Gauge kube_ovn_cluster_inbound_connections_error_total The total number of failed inbound connections to the server. Gauge kube_ovn_cluster_outbound_connections_error_total The total number of failed outbound connections from the server.","title":"ovn-monitor"},{"location":"en/reference/metrics/#ovs-monitor","text":"ovsdb and vswitchd status metrics: Type Metric Description Gauge ovs_status OVS Health Status. The values are: health(1), unhealthy(0). Gauge ovs_info This metric provides basic information about OVS. It is always set to 1. Gauge failed_req_count The number of failed requests to OVS stack. Gauge log_file_size The size of a log file associated with an OVS component. Gauge db_file_size The size of a database file associated with an OVS component. Gauge datapath Represents an existing datapath. This metrics is always 1. Gauge dp_total Represents total number of datapaths on the system. Gauge dp_if Represents an existing datapath interface. This metrics is always 1. Gauge dp_if_total Represents the number of ports connected to the datapath. Gauge dp_flows_total The number of flows in a datapath. Gauge dp_flows_lookup_hit The number of incoming packets in a datapath matching existing flows in the datapath. Gauge dp_flows_lookup_missed The number of incoming packets in a datapath not matching any existing flow in the datapath. Gauge dp_flows_lookup_lost The number of incoming packets in a datapath destined for userspace process but subsequently dropped before reaching userspace. Gauge dp_masks_hit The total number of masks visited for matching incoming packets. Gauge dp_masks_total The number of masks in a datapath. Gauge dp_masks_hit_ratio The average number of masks visited per packet. It is the ration between hit and total number of packets processed by a datapath. Gauge interface Represents OVS interface. This is the primary metric for all other interface metrics. This metrics is always 1. Gauge interface_admin_state The administrative state of the physical network link of OVS interface. The values are: down(0), up(1), other(2). Gauge interface_link_state The state of the physical network link of OVS interface. The values are: down(0), up(1), other(2). Gauge interface_mac_in_use The MAC address in use by OVS interface. Gauge interface_mtu The currently configured MTU for OVS interface. Gauge interface_of_port Represents the OpenFlow port ID associated with OVS interface. Gauge interface_if_index Represents the interface index associated with OVS interface. Gauge interface_tx_packets Represents the number of transmitted packets by OVS interface. Gauge interface_tx_bytes Represents the number of transmitted bytes by OVS interface. Gauge interface_rx_packets Represents the number of received packets by OVS interface. Gauge interface_rx_bytes Represents the number of received bytes by OVS interface. Gauge interface_rx_crc_err Represents the number of CRC errors for the packets received by OVS interface. Gauge interface_rx_dropped Represents the number of input packets dropped by OVS interface. Gauge interface_rx_errors Represents the total number of packets with errors received by OVS interface. Gauge interface_rx_frame_err Represents the number of frame alignment errors on the packets received by OVS interface. Gauge interface_rx_missed_err Represents the number of packets with RX missed received by OVS interface. Gauge interface_rx_over_err Represents the number of packets with RX overrun received by OVS interface. Gauge interface_tx_dropped Represents the number of output packets dropped by OVS interface. Gauge interface_tx_errors Represents the total number of transmit errors by OVS interface. Gauge interface_collisions Represents the number of collisions on OVS interface.","title":"ovs-monitor"},{"location":"en/reference/metrics/#kube-ovn-pinger","text":"Network quality related metrics: Type Metric Description Gauge pinger_ovs_up If the ovs on the node is up Gauge pinger_ovs_down If the ovs on the node is down Gauge pinger_ovn_controller_up If the ovn_controller on the node is up Gauge pinger_ovn_controller_down If the ovn_controller on the node is down Gauge pinger_inconsistent_port_binding The number of mismatch port bindings between ovs and ovn-sb Gauge pinger_apiserver_healthy If the apiserver request is healthy on this node Gauge pinger_apiserver_unhealthy If the apiserver request is unhealthy on this node Histogram pinger_apiserver_latency_ms The latency ms histogram the node request apiserver Gauge pinger_internal_dns_healthy If the internal dns request is unhealthy on this node Gauge pinger_internal_dns_unhealthy If the internal dns request is unhealthy on this node Histogram pinger_internal_dns_latency_ms The latency ms histogram the node request internal dns Gauge pinger_external_dns_health If the external dns request is healthy on this node Gauge pinger_external_dns_unhealthy If the external dns request is unhealthy on this node Histogram pinger_external_dns_latency_ms The latency ms histogram the node request external dns Histogram pinger_pod_ping_latency_ms The latency ms histogram for pod peer ping Gauge pinger_pod_ping_lost_total The lost count for pod peer ping Gauge pinger_pod_ping_count_total The total count for pod peer ping Histogram pinger_node_ping_latency_ms The latency ms histogram for pod ping node Gauge pinger_node_ping_lost_total The lost count for pod ping node Gauge pinger_node_ping_count_total The total count for pod ping node Histogram pinger_external_ping_latency_ms The latency ms histogram for pod ping external address Gauge pinger_external_lost_total The lost count for pod ping external address","title":"kube-ovn-pinger"},{"location":"en/reference/metrics/#kube-ovn-controller","text":"kube-ovn-controller status metrics\uff1a Type Metric Description Histogram rest_client_request_latency_seconds Request latency in seconds. Broken down by verb and URL Counter rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host Counter lists_total Total number of API lists done by the reflectors Summary list_duration_seconds How long an API list takes to return and decode for the reflectors Summary items_per_list How many items an API list returns to the reflectors Counter watches_total Total number of API watches done by the reflectors Counter short_watches_total Total number of short API watches done by the reflectors Summary watch_duration_seconds How long an API watch takes to return and decode for the reflectors Summary items_per_watch How many items an API watch returns to the reflectors Gauge last_resource_version Last resource version seen for the reflectors Histogram ovs_client_request_latency_milliseconds The latency histogram for ovs request Gauge subnet_available_ip_count The available num of ip address in subnet Gauge subnet_used_ip_count The used num of ip address in subnet","title":"kube-ovn-controller"},{"location":"en/reference/metrics/#kube-ovn-cni","text":"kube-ovn-cni status metrics: Type Metric Description Histogram cni_op_latency_seconds The latency seconds for cni operations Counter cni_wait_address_seconds_total Latency that cni wait controller to assign an address Counter cni_wait_connectivity_seconds_total Latency that cni wait address ready in overlay network Counter cni_wait_route_seconds_total Latency that cni wait controller to add routed annotation to pod Histogram rest_client_request_latency_seconds Request latency in seconds. Broken down by verb and URL Counter rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host Counter lists_total Total number of API lists done by the reflectors Summary list_duration_seconds How long an API list takes to return and decode for the reflectors Summary items_per_list How many items an API list returns to the reflectors Counter watches_total Total number of API watches done by the reflectors Counter short_watches_total Total number of short API watches done by the reflectors Summary watch_duration_seconds How long an API watch takes to return and decode for the reflectors Summary items_per_watch How many items an API watch returns to the reflectors Gauge last_resource_version Last resource version seen for the reflectors Histogram ovs_client_request_latency_milliseconds The latency histogram for ovs request \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"kube-ovn-cni"},{"location":"en/reference/ovs-ovn-customized/","text":"OVS/OVN Customization \u00b6 Upstream OVN/OVS was originally designed with the goal of a general purpose SDN controller and data plane. Due to some specific usage of the Kubernetes network,Kube-OVN only focused on part of the features. In order to achieve better performance, stability and specific features, Kube-OVN has made some modifications to the upstream OVN/OVS. Users using their own OVN/OVS with Kube-OVN controllers need to be aware of the possible impact of the following changes: Did not merge into the upstream modification. 38df6fa3f7 Adjust the election timer to avoid large-scale cluster election jitter. d4888c4e75 add fdb update logging. d4888c4e75 fdb: fix mac learning in environments with hairpin enabled. 9a81b91368 ovsdb-tool: add optional server id parameter for \"join-cluster\" command. 0700cb90f9 Destination non-service traffic bypasses conntrack to improve performance on a particular data path. c48049a64f ECMP algorithm is adjusted from dp_hash to hash to avoid the hash error problem in some kernels. 64383c14a9 Fix kernel Crash issue under Windows. 08a95db2ca Support for github action builds on Windows. 680e77a190 Windows uses tcp listening by default. 05e57b3227 add support for windows. 0181b68be1 br-int controller: listen on 127.0.0.1:6653 by default. b3801ecb73 modify src route priority. 977e569539 fix reaching resubmit limit in underlay. 45a4a22161 ovn-nbctl: do not remove LB if vips is empty. 540592b9ff Replaces the Mac address as the destination address after DNAT to reduce additional performance overhead. 10972d9632 Fix vswitchd ofport_usage memory leak. Merged into upstream modification: 20626ea909 Multicast traffic bypasses LB and ACL processing stages to improve specific data path performance. a2d9ff3ccd Deb build adds compile optimization options. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVS/OVN Customization"},{"location":"en/reference/ovs-ovn-customized/#ovsovn-customization","text":"Upstream OVN/OVS was originally designed with the goal of a general purpose SDN controller and data plane. Due to some specific usage of the Kubernetes network,Kube-OVN only focused on part of the features. In order to achieve better performance, stability and specific features, Kube-OVN has made some modifications to the upstream OVN/OVS. Users using their own OVN/OVS with Kube-OVN controllers need to be aware of the possible impact of the following changes: Did not merge into the upstream modification. 38df6fa3f7 Adjust the election timer to avoid large-scale cluster election jitter. d4888c4e75 add fdb update logging. d4888c4e75 fdb: fix mac learning in environments with hairpin enabled. 9a81b91368 ovsdb-tool: add optional server id parameter for \"join-cluster\" command. 0700cb90f9 Destination non-service traffic bypasses conntrack to improve performance on a particular data path. c48049a64f ECMP algorithm is adjusted from dp_hash to hash to avoid the hash error problem in some kernels. 64383c14a9 Fix kernel Crash issue under Windows. 08a95db2ca Support for github action builds on Windows. 680e77a190 Windows uses tcp listening by default. 05e57b3227 add support for windows. 0181b68be1 br-int controller: listen on 127.0.0.1:6653 by default. b3801ecb73 modify src route priority. 977e569539 fix reaching resubmit limit in underlay. 45a4a22161 ovn-nbctl: do not remove LB if vips is empty. 540592b9ff Replaces the Mac address as the destination address after DNAT to reduce additional performance overhead. 10972d9632 Fix vswitchd ofport_usage memory leak. Merged into upstream modification: 20626ea909 Multicast traffic bypasses LB and ACL processing stages to improve specific data path performance. a2d9ff3ccd Deb build adds compile optimization options. \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"OVS/OVN Customization"},{"location":"en/reference/tunnel-protocol/","text":"Tunnel Protocol Selection \u00b6 Kube-OVN uses OVN/OVS as the data plane implementation and currently supports Geneve , Vxlan and STT tunnel encapsulation protocols. These three protocols differ in terms of functionality, performance and ease of use. This document will describe the differences in the use of the three protocols so that users can choose according to their situation. Geneve \u00b6 The Geneve protocol is the default tunneling protocol selected during Kube-OVN deployment and is also the default recommended tunneling protocol for OVN. This protocol is widely supported in the kernel and can be accelerated using the generic offload capability of modern NICs. Since Geneve has a variable header, it is possible to use 24bit space to mark different datapaths users can create a larger number of virtual networks. If you are using Mellanox or Corigine SmartNIC OVS offload, Geneve requires a higher kernel version. Upstream kernel of 5.4 or higher, or other compatible kernels that backports this feature. Due to the use of UDP encapsulation, this protocol does not make good use of the TCP-related offloads of modern NICs when handling TCP over UDP, and consumes more CPU resources when handling large packets. Vxlan \u00b6 Vxlan is a recently supported protocol in the upstream OVN, which is widely supported in the kernel and can be accelerated using the common offload capabilities of modern NICs. Due to the limited length of the protocol header and the additional space required for OVN orchestration, there is a limit to the number of datapaths that can be created, with a maximum of 4096 datapaths and a maximum of 4096 ports under each datapath. Also, inport -based ACLs are not supported due to header length limitations. Vxlan offloading is supported in common kernels if using Mellanox or Corigine SmartNIC. Due to the use of UDP encapsulation, this protocol does not make good use of the TCP-related offloads of modern NICs when handling TCP over UDP, and consumes more CPU resources when handling large packets. STT \u00b6 The STT protocol is an early tunneling protocol supported by the OVN that uses TCP-like headers to take advantage of the TCP offload capabilities common to modern NICs and significantly increase TCP throughput. The protocol also has a long header to support full OVN capabilities and large-scale datapaths. This protocol is not supported in the kernel. To use it, you need to compile an additional OVS kernel module and recompile the new version of the kernel module when upgrading the kernel. This protocol is not currently supported by the SmartNic and cannot use the offloading capability of OVS offloading. References \u00b6 https://ipwithease.com/vxlan-vs-geneve-understand-the-difference/ OVN FAQ What is Geneve \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Tunnel Protocol Selection"},{"location":"en/reference/tunnel-protocol/#tunnel-protocol-selection","text":"Kube-OVN uses OVN/OVS as the data plane implementation and currently supports Geneve , Vxlan and STT tunnel encapsulation protocols. These three protocols differ in terms of functionality, performance and ease of use. This document will describe the differences in the use of the three protocols so that users can choose according to their situation.","title":"Tunnel Protocol Selection"},{"location":"en/reference/tunnel-protocol/#geneve","text":"The Geneve protocol is the default tunneling protocol selected during Kube-OVN deployment and is also the default recommended tunneling protocol for OVN. This protocol is widely supported in the kernel and can be accelerated using the generic offload capability of modern NICs. Since Geneve has a variable header, it is possible to use 24bit space to mark different datapaths users can create a larger number of virtual networks. If you are using Mellanox or Corigine SmartNIC OVS offload, Geneve requires a higher kernel version. Upstream kernel of 5.4 or higher, or other compatible kernels that backports this feature. Due to the use of UDP encapsulation, this protocol does not make good use of the TCP-related offloads of modern NICs when handling TCP over UDP, and consumes more CPU resources when handling large packets.","title":"Geneve"},{"location":"en/reference/tunnel-protocol/#vxlan","text":"Vxlan is a recently supported protocol in the upstream OVN, which is widely supported in the kernel and can be accelerated using the common offload capabilities of modern NICs. Due to the limited length of the protocol header and the additional space required for OVN orchestration, there is a limit to the number of datapaths that can be created, with a maximum of 4096 datapaths and a maximum of 4096 ports under each datapath. Also, inport -based ACLs are not supported due to header length limitations. Vxlan offloading is supported in common kernels if using Mellanox or Corigine SmartNIC. Due to the use of UDP encapsulation, this protocol does not make good use of the TCP-related offloads of modern NICs when handling TCP over UDP, and consumes more CPU resources when handling large packets.","title":"Vxlan"},{"location":"en/reference/tunnel-protocol/#stt","text":"The STT protocol is an early tunneling protocol supported by the OVN that uses TCP-like headers to take advantage of the TCP offload capabilities common to modern NICs and significantly increase TCP throughput. The protocol also has a long header to support full OVN capabilities and large-scale datapaths. This protocol is not supported in the kernel. To use it, you need to compile an additional OVS kernel module and recompile the new version of the kernel module when upgrading the kernel. This protocol is not currently supported by the SmartNic and cannot use the offloading capability of OVS offloading.","title":"STT"},{"location":"en/reference/tunnel-protocol/#references","text":"https://ipwithease.com/vxlan-vs-geneve-understand-the-difference/ OVN FAQ What is Geneve \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"References"},{"location":"en/reference/underlay-topology/","text":"Underlay Traffic Topology \u00b6 This document describes the forwarding path of traffic in Underlay mode under different scenarios. Pods in Same Node and Same Subnet \u00b6 Internal logical switches exchange packets directly, without access to the external network. Pods in Different Nodes and Same Subnet \u00b6 Packets enter the physic switch via the node NIC and are exchanged by the physic switch. Pods in Same Node and Different Subnets \u00b6 Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. Here br-provider-1 and br-provider-2 can be the same OVS bridge\uff0cmultiple subnet can share a Provider Network\u3002 Pods in Different Nodes and Different Subnets \u00b6 Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. Access to External \u00b6 Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. The communication between nodes and Pods follows the same logic. Overview without Vlan Tag \u00b6 Overview with Vlan Tag \u00b6 Pod visit Service IP \u00b6 Kube-OVN configures load balancing for each Kubernetes Service on a logical switch on each subnet. When a Pod accesses other Pods by accessing the Service IP, a network packet is constructed with the Service IP as the destination address and the MAC address of the gateway as the destination MAC address. After the network packet enters the logical switch, load balancing will intercept and DNAT the network packet to modify the destination IP and port to the IP and port of one of the Endpoint corresponding to the Service. Since the logical switch does not modify the Layer 2 destination MAC address of the network packet, the network packet will still be delivered to the physic gateway after entering the physic switch, and the physic gateway will be required to forward the network packet. Service Backend is the Same Node and Same Subnet Pod \u00b6 Service Backend is the Same Node and Different Subnets Pod \u00b6 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Underlay Traffic Topology"},{"location":"en/reference/underlay-topology/#underlay-traffic-topology","text":"This document describes the forwarding path of traffic in Underlay mode under different scenarios.","title":"Underlay Traffic Topology"},{"location":"en/reference/underlay-topology/#pods-in-same-node-and-same-subnet","text":"Internal logical switches exchange packets directly, without access to the external network.","title":"Pods in Same Node and Same Subnet"},{"location":"en/reference/underlay-topology/#pods-in-different-nodes-and-same-subnet","text":"Packets enter the physic switch via the node NIC and are exchanged by the physic switch.","title":"Pods in Different Nodes and Same Subnet"},{"location":"en/reference/underlay-topology/#pods-in-same-node-and-different-subnets","text":"Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. Here br-provider-1 and br-provider-2 can be the same OVS bridge\uff0cmultiple subnet can share a Provider Network\u3002","title":"Pods in Same Node and Different Subnets"},{"location":"en/reference/underlay-topology/#pods-in-different-nodes-and-different-subnets","text":"Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers.","title":"Pods in Different Nodes and Different Subnets"},{"location":"en/reference/underlay-topology/#access-to-external","text":"Packets enter the physic network via the node NIC and are exchanged and routed and forwarded by physic switches and routers. The communication between nodes and Pods follows the same logic.","title":"Access to External"},{"location":"en/reference/underlay-topology/#overview-without-vlan-tag","text":"","title":"Overview without Vlan Tag"},{"location":"en/reference/underlay-topology/#overview-with-vlan-tag","text":"","title":"Overview with Vlan Tag"},{"location":"en/reference/underlay-topology/#pod-visit-service-ip","text":"Kube-OVN configures load balancing for each Kubernetes Service on a logical switch on each subnet. When a Pod accesses other Pods by accessing the Service IP, a network packet is constructed with the Service IP as the destination address and the MAC address of the gateway as the destination MAC address. After the network packet enters the logical switch, load balancing will intercept and DNAT the network packet to modify the destination IP and port to the IP and port of one of the Endpoint corresponding to the Service. Since the logical switch does not modify the Layer 2 destination MAC address of the network packet, the network packet will still be delivered to the physic gateway after entering the physic switch, and the physic gateway will be required to forward the network packet.","title":"Pod visit Service IP"},{"location":"en/reference/underlay-topology/#service-backend-is-the-same-node-and-same-subnet-pod","text":"","title":"Service Backend is the Same Node and Same Subnet Pod"},{"location":"en/reference/underlay-topology/#service-backend-is-the-same-node-and-different-subnets-pod","text":"\u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Service Backend is the Same Node and Different Subnets Pod"},{"location":"en/start/one-step-install/","text":"One-Click Installation \u00b6 Kube-OVN provides a one-click installation script to help you quickly install a highly available, production-ready Kube-OVN container network with Overlay networking by default. Helm Chart installation is supported since Kube-OVN v1.12.0, and the default deployment is Overlay networking. If you need Underlay/Vlan networking as the default container network\uff0cplease read Underlay Installation Before installation please read Prerequisites first to make sure the environment is ready. Script Installation \u00b6 Download the installation script \u00b6 We recommend using the stable release version for production environments, please use the following command to download: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh If you are interested in the latest features of the master branch, please use the following command to download: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh Modify Configuration Options \u00b6 Open the script using the editor and change the following variables to the expected: REGISTRY = \"kubeovn\" # Image Repo VERSION = \"v1.12.0\" # Image Tag POD_CIDR = \"10.16.0.0/16\" # Default subnet CIDR don't overlay with SVC/NODE/JOIN CIDR SVC_CIDR = \"10.96.0.0/12\" # Be consistent with apiserver's service-cluster-ip-range JOIN_CIDR = \"100.64.0.0/16\" # Pod/Host communication Subnet CIDR, don't overlay with SVC/NODE/POD CIDR LABEL = \"node-role.kubernetes.io/master\" # The node label to deploy OVN DB IFACE = \"\" # The name of the host NIC used by the container network, or if empty use the NIC that host Node IP in Kubernetes TUNNEL_TYPE = \"geneve\" # Tunnel protocol\uff0cavailable options: geneve, vxlan or stt. stt requires compilation of ovs kernel module You can also use regular expression to math NIC names\uff0csuch as IFACE=enp6s0f0,eth.* . Run the Script \u00b6 bash install.sh Wait Kube-OVN ready. Helm Chart Installation \u00b6 Since the installation of Kube-OVN requires setting some parameters, to install Kube-OVN using Helm, you need to follow the steps below. View the node IP address \u00b6 $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-ovn-control-plane NotReady control-plane 20h v1.26.0 172 .18.0.3 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 kube-ovn-worker NotReady <none> 20h v1.26.0 172 .18.0.2 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 Remove cluster master node taint \u00b6 $ kubectl taint node kube-ovn-control-plane node-role.kubernetes.io/control-plane:NoSchedule- node/kube-ovn-control-plane untainted This step can be skipped if you are sure that you do not need to schedule the pod at the master node. Add label to node \u00b6 $ kubectl label node -lbeta.kubernetes.io/os = linux kubernetes.io/os = linux --overwrite node/kube-ovn-control-plane not labeled node/kube-ovn-worker not labeled $ kubectl label node -lnode-role.kubernetes.io/control-plane kube-ovn/role = master --overwrite node/kube-ovn-control-plane labeled # The following labels are used for the installation of dpdk images and can be ignored in non-dpdk cases $ kubectl label node -lovn.kubernetes.io/ovs_dp_type! = userspace ovn.kubernetes.io/ovs_dp_type = kernel --overwrite node/kube-ovn-control-plane labeled node/kube-ovn-worker labeled Add Helm Repo information \u00b6 $ helm repo add kubeovn https://kubeovn.github.io/kube-ovn/ \"kubeovn\" has been added to your repositories $ helm repo list NAME URL kubeovn https://kubeovn.github.io/kube-ovn/ $ helm search repo kubeovn NAME CHART VERSION APP VERSION DESCRIPTION kubeovn/kube-ovn 0 .1.0 1 .12.0 Helm chart for Kube-OVN Run helm install to install Kube-OVN \u00b6 The Node0IP, Node1IP, and Node2IP parameters are the IP addresses of the cluster master nodes, respectively. For other parameters, you can refer to the variable definitions in the values.yaml file. # Single master node environment install $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } # Using the node information above as an example, execute the install command $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = 172 .18.0.3 NAME: kube-ovn LAST DEPLOYED: Fri Mar 31 12 :43:43 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None # Highly Available Cluster Installation $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } , ${ Node1IP } , ${ Node2IP } , --set replicaCount = 3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"One-Click Installation"},{"location":"en/start/one-step-install/#one-click-installation","text":"Kube-OVN provides a one-click installation script to help you quickly install a highly available, production-ready Kube-OVN container network with Overlay networking by default. Helm Chart installation is supported since Kube-OVN v1.12.0, and the default deployment is Overlay networking. If you need Underlay/Vlan networking as the default container network\uff0cplease read Underlay Installation Before installation please read Prerequisites first to make sure the environment is ready.","title":"One-Click Installation"},{"location":"en/start/one-step-install/#script-installation","text":"","title":"Script Installation"},{"location":"en/start/one-step-install/#download-the-installation-script","text":"We recommend using the stable release version for production environments, please use the following command to download: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh If you are interested in the latest features of the master branch, please use the following command to download: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"Download the installation script"},{"location":"en/start/one-step-install/#modify-configuration-options","text":"Open the script using the editor and change the following variables to the expected: REGISTRY = \"kubeovn\" # Image Repo VERSION = \"v1.12.0\" # Image Tag POD_CIDR = \"10.16.0.0/16\" # Default subnet CIDR don't overlay with SVC/NODE/JOIN CIDR SVC_CIDR = \"10.96.0.0/12\" # Be consistent with apiserver's service-cluster-ip-range JOIN_CIDR = \"100.64.0.0/16\" # Pod/Host communication Subnet CIDR, don't overlay with SVC/NODE/POD CIDR LABEL = \"node-role.kubernetes.io/master\" # The node label to deploy OVN DB IFACE = \"\" # The name of the host NIC used by the container network, or if empty use the NIC that host Node IP in Kubernetes TUNNEL_TYPE = \"geneve\" # Tunnel protocol\uff0cavailable options: geneve, vxlan or stt. stt requires compilation of ovs kernel module You can also use regular expression to math NIC names\uff0csuch as IFACE=enp6s0f0,eth.* .","title":"Modify Configuration Options"},{"location":"en/start/one-step-install/#run-the-script","text":"bash install.sh Wait Kube-OVN ready.","title":"Run the Script"},{"location":"en/start/one-step-install/#helm-chart-installation","text":"Since the installation of Kube-OVN requires setting some parameters, to install Kube-OVN using Helm, you need to follow the steps below.","title":"Helm Chart Installation"},{"location":"en/start/one-step-install/#view-the-node-ip-address","text":"$ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-ovn-control-plane NotReady control-plane 20h v1.26.0 172 .18.0.3 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9 kube-ovn-worker NotReady <none> 20h v1.26.0 172 .18.0.2 <none> Ubuntu 22 .04.1 LTS 5 .10.104-linuxkit containerd://1.6.9","title":"View the node IP address"},{"location":"en/start/one-step-install/#remove-cluster-master-node-taint","text":"$ kubectl taint node kube-ovn-control-plane node-role.kubernetes.io/control-plane:NoSchedule- node/kube-ovn-control-plane untainted This step can be skipped if you are sure that you do not need to schedule the pod at the master node.","title":"Remove cluster master node taint"},{"location":"en/start/one-step-install/#add-label-to-node","text":"$ kubectl label node -lbeta.kubernetes.io/os = linux kubernetes.io/os = linux --overwrite node/kube-ovn-control-plane not labeled node/kube-ovn-worker not labeled $ kubectl label node -lnode-role.kubernetes.io/control-plane kube-ovn/role = master --overwrite node/kube-ovn-control-plane labeled # The following labels are used for the installation of dpdk images and can be ignored in non-dpdk cases $ kubectl label node -lovn.kubernetes.io/ovs_dp_type! = userspace ovn.kubernetes.io/ovs_dp_type = kernel --overwrite node/kube-ovn-control-plane labeled node/kube-ovn-worker labeled","title":"Add label to node"},{"location":"en/start/one-step-install/#add-helm-repo-information","text":"$ helm repo add kubeovn https://kubeovn.github.io/kube-ovn/ \"kubeovn\" has been added to your repositories $ helm repo list NAME URL kubeovn https://kubeovn.github.io/kube-ovn/ $ helm search repo kubeovn NAME CHART VERSION APP VERSION DESCRIPTION kubeovn/kube-ovn 0 .1.0 1 .12.0 Helm chart for Kube-OVN","title":"Add Helm Repo information"},{"location":"en/start/one-step-install/#run-helm-install-to-install-kube-ovn","text":"The Node0IP, Node1IP, and Node2IP parameters are the IP addresses of the cluster master nodes, respectively. For other parameters, you can refer to the variable definitions in the values.yaml file. # Single master node environment install $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } # Using the node information above as an example, execute the install command $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = 172 .18.0.3 NAME: kube-ovn LAST DEPLOYED: Fri Mar 31 12 :43:43 2023 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None # Highly Available Cluster Installation $ helm install kube-ovn kubeovn/kube-ovn --set MASTER_NODES = ${ Node0IP } , ${ Node1IP } , ${ Node2IP } , --set replicaCount = 3 \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Run helm install to install Kube-OVN"},{"location":"en/start/prepare/","text":"Prerequisites \u00b6 Kube-OVN is a CNI-compliant network system that depends on the Kubernetes environment and the corresponding kernel network module for its operation. Below are the operating system and software versions tested, the environment configuration and the ports that need to be opened. Software Version \u00b6 Kubernetes >= 1.23. Docker >= 1.12.6, Containerd >= 1.3.4. OS: CentOS 7/8, Ubuntu 16.04/18.04/20.04. For other Linux distributions, please make sure geneve , openvswitch , ip_tables and iptable_nat kernel modules exist. Attention \uff1a For CentOS kernel version 3.10.0-862 bug exists in netfilter modules that lead Kube-OVN embed nat and lb failure.Please update kernel and check Floating IPs broken after kernel upgrade to Centos/RHEL 7.5 - DNAT not working . Kernel version 4.18.0-372.9.1.el8.x86_64 in Rocky Linux 8.6 has a TCP connection problem TCP connection failed in Rocky Linux 8.6 \uff0cplease update kernel to 4.18.0-372.13.1.el8_6.x86_64 or later\u3002 For kernel version 4.4, the related openvswitch module has some issues for ct\uff0cplease update kernel version or manually compile openvswitch kernel module. When building Geneve tunnel IPv6 in kernel should be enabled\uff0ccheck the kernel bootstrap options with cat /proc/cmdline .Check Geneve tunnels don't work when ipv6 is disabled for the detail bug info. Environment Setup \u00b6 Kernel should enable IPv6, if kernel bootstrap options contain ipv6.disable=1 , it should be set to 0 . kube-proxy works, Kube-OVN can visit kube-apiserver from Service ClusterIP. Make sure kubelet enabled CNI and find cni-bin and cni-conf in default directories, kubelet bootstrap options should contain --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d . Make sure no other CNI installed or has been removed\uff0ccheck if any config files still exist in /etc/cni/net.d/ . Ports Need Open \u00b6 Component Port Usage ovn-central 6641/tcp, 6642/tcp, 6643/tcp, 6644/tcp ovn-db and raft server listen ports ovs-ovn Geneve 6081/udp, STT 7471/tcp, Vxlan 4789/udp tunnel ports kube-ovn-controller 10660/tcp metrics port kube-ovn-daemon 10665/tcp metrics port kube-ovn-monitor 10661/tcp metrics port \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Prerequisites"},{"location":"en/start/prepare/#prerequisites","text":"Kube-OVN is a CNI-compliant network system that depends on the Kubernetes environment and the corresponding kernel network module for its operation. Below are the operating system and software versions tested, the environment configuration and the ports that need to be opened.","title":"Prerequisites"},{"location":"en/start/prepare/#software-version","text":"Kubernetes >= 1.23. Docker >= 1.12.6, Containerd >= 1.3.4. OS: CentOS 7/8, Ubuntu 16.04/18.04/20.04. For other Linux distributions, please make sure geneve , openvswitch , ip_tables and iptable_nat kernel modules exist. Attention \uff1a For CentOS kernel version 3.10.0-862 bug exists in netfilter modules that lead Kube-OVN embed nat and lb failure.Please update kernel and check Floating IPs broken after kernel upgrade to Centos/RHEL 7.5 - DNAT not working . Kernel version 4.18.0-372.9.1.el8.x86_64 in Rocky Linux 8.6 has a TCP connection problem TCP connection failed in Rocky Linux 8.6 \uff0cplease update kernel to 4.18.0-372.13.1.el8_6.x86_64 or later\u3002 For kernel version 4.4, the related openvswitch module has some issues for ct\uff0cplease update kernel version or manually compile openvswitch kernel module. When building Geneve tunnel IPv6 in kernel should be enabled\uff0ccheck the kernel bootstrap options with cat /proc/cmdline .Check Geneve tunnels don't work when ipv6 is disabled for the detail bug info.","title":"Software Version"},{"location":"en/start/prepare/#environment-setup","text":"Kernel should enable IPv6, if kernel bootstrap options contain ipv6.disable=1 , it should be set to 0 . kube-proxy works, Kube-OVN can visit kube-apiserver from Service ClusterIP. Make sure kubelet enabled CNI and find cni-bin and cni-conf in default directories, kubelet bootstrap options should contain --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d . Make sure no other CNI installed or has been removed\uff0ccheck if any config files still exist in /etc/cni/net.d/ .","title":"Environment Setup"},{"location":"en/start/prepare/#ports-need-open","text":"Component Port Usage ovn-central 6641/tcp, 6642/tcp, 6643/tcp, 6644/tcp ovn-db and raft server listen ports ovs-ovn Geneve 6081/udp, STT 7471/tcp, Vxlan 4789/udp tunnel ports kube-ovn-controller 10660/tcp metrics port kube-ovn-daemon 10665/tcp metrics port kube-ovn-monitor 10661/tcp metrics port \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Ports Need Open"},{"location":"en/start/sealos-install/","text":"One-Click Deployment of Kubernetes and Kube-OVN with sealos \u00b6 sealos , a distribution of Kubernetes, helps users quickly initialize a container cluster from scratch. By using sealos, users can deploy a Kubernetes cluster with Kube-OVN installed in minutes with a single command. Download sealos \u00b6 AMD64 ARM64 wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_amd64.tar.gz \\ && tar zxvf sealos_4.0.0_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_arm64.tar.gz \\ && tar zxvf sealos_4.0.0_linux_arm64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin Deploy Kubernetes and Kube-OVN \u00b6 ```bash sealos run labring/kubernetes:v1.24.3 labring/kube-ovn:v1.10.5 \\ --masters [masters ips seperated by comma] \\ --nodes [nodes ips seperated by comma] -p [your-ssh-passwd] ``` Wait to finish \u00b6 ```bash [Step 6/6] Finish ,,,, ,::, ,,::,,,, ,,,,,::::::::::::,,,,, ,,,::::::::::::::::::::::,,, ,,::::::::::::::::::::::::::::,, ,,::::::::::::::::::::::::::::::::,, ,::::::::::::::::::::::::::::::::::::, ,:::::::::::::,, ,,:::::,,,::::::::::, ,,:::::::::::::, ,::, ,:::::::::, ,:::::::::::::, :x, ,:: :, ,:::::::::, ,:::::::::::::::, ,,, ,::, ,, ,::::::::::, ,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x ,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x ,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx ,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx ,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,:: ,::::, ,::::, ,::::, ,::::, ,::::, ,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,, ,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::, ,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::, ,::::, ,::::, ,::::, ,::::::::::::,,,,, ,,,, ,::::, ,,,, ,,,::::,,,, ,::::, ,,::, Thanks for choosing Kube-OVN! For more advanced features, please read https://github.com/kubeovn/kube-ovn#documents If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose 2022-08-10T16:31:34 info succeeded in creating a new cluster, enjoy it! 2022-08-10T16:31:34 info ___ ___ ___ ___ ___ ___ /\\ \\ /\\ \\ /\\ \\ /\\__\\ /\\ \\ /\\ \\ /::\\ \\ /::\\ \\ /::\\ \\ /:/ / /::\\ \\ /::\\ \\ /:/\\ \\ \\ /:/\\:\\ \\ /:/\\:\\ \\ /:/ / /:/\\:\\ \\ /:/\\ \\ \\ _\\:\\~\\ \\ \\ /::\\~\\:\\ \\ /::\\~\\:\\ \\ /:/ / /:/ \\:\\ \\ _\\:\\~\\ \\ \\ /\\ \\:\\ \\ \\__\\ /:/\\:\\ \\:\\__\\ /:/\\:\\ \\:\\__\\ /:/__/ /:/__/ \\:\\__\\ /\\ \\:\\ \\ \\__\\ \\:\\ \\:\\ \\/__/ \\:\\~\\:\\ \\/__/ \\/__\\:\\/:/ / \\:\\ \\ \\:\\ \\ /:/ / \\:\\ \\:\\ \\/__/ \\:\\ \\:\\__\\ \\:\\ \\:\\__\\ \\::/ / \\:\\ \\ \\:\\ /:/ / \\:\\ \\:\\__\\ \\:\\/:/ / \\:\\ \\/__/ /:/ / \\:\\ \\ \\:\\/:/ / \\:\\/:/ / \\::/ / \\:\\__\\ /:/ / \\:\\__\\ \\::/ / \\::/ / \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ Website :https://www.sealos.io/ Address :github.com/labring/sealos ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Use Sealos to Deploy Kubernetes and Kube-OVN"},{"location":"en/start/sealos-install/#one-click-deployment-of-kubernetes-and-kube-ovn-with-sealos","text":"sealos , a distribution of Kubernetes, helps users quickly initialize a container cluster from scratch. By using sealos, users can deploy a Kubernetes cluster with Kube-OVN installed in minutes with a single command.","title":"One-Click Deployment of Kubernetes and Kube-OVN with sealos"},{"location":"en/start/sealos-install/#download-sealos","text":"AMD64 ARM64 wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_amd64.tar.gz \\ && tar zxvf sealos_4.0.0_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_arm64.tar.gz \\ && tar zxvf sealos_4.0.0_linux_arm64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin","title":"Download sealos"},{"location":"en/start/sealos-install/#deploy-kubernetes-and-kube-ovn","text":"```bash sealos run labring/kubernetes:v1.24.3 labring/kube-ovn:v1.10.5 \\ --masters [masters ips seperated by comma] \\ --nodes [nodes ips seperated by comma] -p [your-ssh-passwd] ```","title":"Deploy Kubernetes and Kube-OVN"},{"location":"en/start/sealos-install/#wait-to-finish","text":"```bash [Step 6/6] Finish ,,,, ,::, ,,::,,,, ,,,,,::::::::::::,,,,, ,,,::::::::::::::::::::::,,, ,,::::::::::::::::::::::::::::,, ,,::::::::::::::::::::::::::::::::,, ,::::::::::::::::::::::::::::::::::::, ,:::::::::::::,, ,,:::::,,,::::::::::, ,,:::::::::::::, ,::, ,:::::::::, ,:::::::::::::, :x, ,:: :, ,:::::::::, ,:::::::::::::::, ,,, ,::, ,, ,::::::::::, ,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x ,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x ,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx ,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx ,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,:: ,::::, ,::::, ,::::, ,::::, ,::::, ,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,, ,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::, ,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::, ,::::, ,::::, ,::::, ,::::::::::::,,,,, ,,,, ,::::, ,,,, ,,,::::,,,, ,::::, ,,::, Thanks for choosing Kube-OVN! For more advanced features, please read https://github.com/kubeovn/kube-ovn#documents If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose 2022-08-10T16:31:34 info succeeded in creating a new cluster, enjoy it! 2022-08-10T16:31:34 info ___ ___ ___ ___ ___ ___ /\\ \\ /\\ \\ /\\ \\ /\\__\\ /\\ \\ /\\ \\ /::\\ \\ /::\\ \\ /::\\ \\ /:/ / /::\\ \\ /::\\ \\ /:/\\ \\ \\ /:/\\:\\ \\ /:/\\:\\ \\ /:/ / /:/\\:\\ \\ /:/\\ \\ \\ _\\:\\~\\ \\ \\ /::\\~\\:\\ \\ /::\\~\\:\\ \\ /:/ / /:/ \\:\\ \\ _\\:\\~\\ \\ \\ /\\ \\:\\ \\ \\__\\ /:/\\:\\ \\:\\__\\ /:/\\:\\ \\:\\__\\ /:/__/ /:/__/ \\:\\__\\ /\\ \\:\\ \\ \\__\\ \\:\\ \\:\\ \\/__/ \\:\\~\\:\\ \\/__/ \\/__\\:\\/:/ / \\:\\ \\ \\:\\ \\ /:/ / \\:\\ \\:\\ \\/__/ \\:\\ \\:\\__\\ \\:\\ \\:\\__\\ \\::/ / \\:\\ \\ \\:\\ /:/ / \\:\\ \\:\\__\\ \\:\\/:/ / \\:\\ \\/__/ /:/ / \\:\\ \\ \\:\\/:/ / \\:\\/:/ / \\::/ / \\:\\__\\ /:/ / \\:\\__\\ \\::/ / \\::/ / \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ \\/__/ Website :https://www.sealos.io/ Address :github.com/labring/sealos ``` \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Wait to finish"},{"location":"en/start/underlay/","text":"Underlay Installation \u00b6 By default, the default subnet uses Geneve to encapsulate cross-host traffic, and build an overlay network on top of the infrastructure. For the case that you want the container network to use the physical network address directly, you can set the default subnet of Kube-OVN to work in Underlay mode, which can directly assign the address resources in the physical network to the containers, achieving better performance and connectivity with the physical network. Limitation \u00b6 Since the container network in this mode uses physical network directly for L2 packet forwarding, L3 functions such as SNAT/EIP, distributed gateway/centralized gateway in Overlay mode cannot be used. VPC level isolation is also not available for underlay subnet. Comparison with Macvlan \u00b6 The Underlay mode of Kube-OVN is very similar to the Macvlan, with the following major differences in functionality and performance: Macvlan performs better in terms of throughput and latency performance metrics due to its shorter kernel path and the fact that it does not require OVS for packet processing. Kube-OVN provides arp-proxy functionality through flow tables to mitigate the risk of arp broadcast storms on large-scale networks. Since Macvlan works at the bottom of the kernel and bypasses the host netfilter, Service and NetworkPolicy functionality requires additional development. Kube-OVN provides Service and NetworkPolicy capabilities through the OVS flow table. Kube-OVN Underlay mode provides additional features such as address management, fixed IP and QoS compared to Macvlan. Environment Requirements \u00b6 In Underlay mode, the OVS will bridge a node NIC to the OVS bridge and send packets directly through that node NIC, relying on the underlying network devices for L2/L3 level forwarding capabilities. You need to configure the corresponding gateway, Vlan and security policy in the underlying network device in advance. For OpenStack VM environments, you need to turn off PortSecurity on the corresponding network port. For VMware vSwitch networks, MAC Address Changes , Forged Transmits and Promiscuous Mode Operation should be set to allow . For Hyper-V virtualization, MAC Address Spoofing should be enabled in VM nic advanced features. Public clouds, such as AWS, GCE, AliCloud, etc., do not support user-defined Mac, so they cannot support Underlay mode network. In this scenario, if you want to use Underlay, it is recommended to use the VPC-CNI provided by the corresponding public cloud vendor.. The network interface that is bridged into ovs can not be type of Linux Bridge. For management and container networks using the same NIC, Kube-OVN will transfer the NIC's Mac address, IP address, route, and MTU to the corresponding OVS Bridge to support single NIC deployment of Underlay networks. OVS Bridge name format is br-PROVIDER_NAME \uff0c PROVIDER_NAME is the name of ProviderNetwork (Default: provider). Specify Network Mode When Deploying \u00b6 This deployment mode sets the default subnet to Underlay mode, and all Pods with no subnet specified will run in the Underlay network by default. Download Script \u00b6 wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh Modify Configuration Options \u00b6 ENABLE_ARP_DETECT_IP_CONFLICT # disable vlan arp conflict detection if necessary NETWORK_TYPE # set to vlan VLAN_INTERFACE_NAME # set to the NIC that carries the Underlay traffic, e.g. eth1 VLAN_ID # The VLAN Tag need to be added\uff0cif set 0 no vlan tag will be added POD_CIDR # The Underlay network CIDR\uff0c e.g. 192.168.1.0/24 POD_GATEWAY # Underlay physic gateway address, e.g. 192.168.1.1 EXCLUDE_IPS # Exclude ranges to avoid conflicts between container network and IPs already in use on the physical network, e.g. 192.168.1.1..192.168.1.100 ENABLE_LB # If Underlay Subnet needs to visit Service set it to true EXCHANGE_LINK_NAME # If swap the names of the OVS bridge and the bridge interface under the default provider-network. Default to false. LS_DNAT_MOD_DL_DST # If DNAT translate MAC addresses to accelerate service access. Default to true. Run the Script \u00b6 bash install.sh Dynamically Create Underlay Networks via CRD \u00b6 This approach dynamically creates an Underlay subnet that Pod can use after installation. Create ProviderNetwork \u00b6 ProviderNetwork provides the abstraction of host NIC to physical network mapping, unifies the management of NICs belonging to the same network, and solves the configuration problems in complex environments with multiple NICs on the same machine, inconsistent NIC names and inconsistent corresponding Underlay networks. Create ProviderNetwork as below: apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: net1 spec: defaultInterface: eth1 customInterfaces: - interface: eth2 nodes: - node1 excludeNodes: - node2 Note: The length of the ProviderNetwork resource name must not exceed 12. defaultInterface : The default node NIC name. When the ProviderNetwork is successfully created, an OVS bridge named br-net1 (in the format br-NAME ) is created in each node (except excludeNodes) and the specified node NIC is bridged to this bridge. customInterfaces : Optionally, you can specify the NIC to be used for a specific node. excludeNodes : Optional, to specify nodes that do not bridge the NIC. Nodes in this list will be added with the net1.provider-network.ovn.kubernetes.io/exclude=true tag. Other nodes will be added with the following tags: Key Value Description net1.provider-network.ovn.kubernetes.io/ready true bridge work finished, ProviderNetwork is ready on this node net1.provider-network.ovn.kubernetes.io/interface eth1 The name of the bridged NIC in the node. net1.provider-network.ovn.kubernetes.io/mtu 1500 MTU of bridged NIC in node If an IP has been configured on the node NIC, the IP address and the route on the NIC are transferred to the corresponding OVS bridge. Create VLAN \u00b6 Vlan provides an abstraction to bind Vlan Tag and ProviderNetwork. Create a VLAN as below: apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan1 spec: id: 0 provider: net1 id : VLAN ID/Tag\uff0cKube-OVN will add this Vlan tag to traffic, if set 0, no tag is added. provider : The name of ProviderNetwork. Multiple VLAN can use a same ProviderNetwork. Create Subnet \u00b6 Bind Vlan to a Subnet as below\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 Simply specify the value of vlan as the name of the VLAN to be used. Multiple subnets can refer to the same VLAN. Create Pod \u00b6 You can create containers in the normal way, check whether the container IP is in the specified range and whether the container can interoperate with the physical network. For fixed IP requirements, please refer to Fixed Addresses Logical Gateway \u00b6 For cases where no gateway exists in the physical network, Kube-OVN supports the use of logical gateways configured in the subnet in Underlay mode. To use this feature, set spec.logicalGateway to true for the subnet: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 logicalGateway : true When this feature is turned on, the Pod does not use an external gateway, but a Logical Router created by Kube-OVN to forward cross-subnet communication. Interconnection of Underlay and Overlay Networks \u00b6 If a cluster has both Underlay and Overlay subnets, by default, Pods in the Overlay subnet can access the Pod IPs in the Underlay subnet via a gateway using NAT. From the perspective of Pods in the Underlay subnet, the addresses in the Overlay subnet are external, and require the underlying physical device to forward, but the underlying physical device does not know the addresses in the Overlay subnet and cannot forward. Therefore, Pods in the Underlay subnet cannot access Pods in the Overlay subnet directly via Pod IPs. If you need to enable communication between Underlay and Overlay networks, you need to set the u2oInterconnection of the subnet to true . In this case, Kube-OVN will use an additional Underlay IP to connect the Underlay subnet and the ovn-cluster logical router, and set the corresponding routing rules to enable communication. Unlike the logical gateway, this solution only connects the Underlay and Overlay subnets within Kube-OVN, and other traffic accessing the Internet will still be forwarded through the physical gateway. Specify logical gateway IP \u00b6 After the interworking function is enabled, an IP from the subnet will be randomly selected as the logical gateway. If you need to specify the logical gateway of the Underlay Subnet, you can specify the field u2oInterconnectionIP . Specify custom VPC for Underlay Subnet connection \u00b6 By default, the Underlay Subnet will communicate with the Overlay Subnet on the default VPC. If you want to specify to communicate with a certain VPC, after setting u2oInterconnection to true , specify the subnet.spec.vpc field as the name of the VPC. Notice \u00b6 If there are IP addresses on the host nic, the OS you are using is Ubuntu, and the networking is configured with Netplan, it's recomanded to set NetworkManager as the renderer and to set static IP addresses for the nic (disable DHCP): network : renderer : NetworkManager ethernets : eth0 : dhcp4 : no addresses : - 172.16.143.129/24 version : 2 If the host networking service is NetworkManager, Kube-OVN will remove the host nic from the managed devices (managed by NetworkManager is no) after creating ProviderNetwork: root@ubuntu:~# nmcli device status DEVICE TYPE STATE CONNECTION eth0 ethernet unmanaged netplan-eth0 If you want to change the host nic's IP/route configuration, you need to set the nic managed by NetworkManager manually: nmcli device set eth0 managed yes After setting managed to yes\uff0cKube-OVN will transfer IP and routes on the nic to the OVS bridge, and remove the nic from the managed devices again. Notice \uff1aIf the host nic's MAC is changed, Kube-OVN will not change the OVS bridge's MAC unless kube-ovn-cni is restarted. Known Issues \u00b6 When the physical network is enabled with hairpin, Pod network is abnormal \u00b6 When physical networks enable hairpin or similar behaviors, problems such as gateway check failure when creating Pods and abnormal network communication of Pods may occur. This is because the default MAC learning function of OVS bridge does not support this kind of network environment. To solve this problem, it is necessary to turn off hairpin (or modify the relevant configuration of physical network), or update the Kube-OVN version. When there are a large number of Pods, gateway check for new Pods fails \u00b6 If there are a large number of Pods running on the same node (more than 300), it may cause packet loss due to the OVS flow table resubmit times exceeding the upper limit of ARP broadcast packets. 2022-11-13T08:43:46.782Z|00222|ofproto_dpif_upcall(handler5)|WARN|Flow: arp,in_port=331,vlan_tci=0x0000,dl_src=00:00:00:25:eb:39,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.131.240,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:25:eb:39,arp_tha=ff:ff:ff:ff:ff:ff bridge(\"br-int\") ---------------- 0. No match. >>>> received packet on unknown port 331 <<<< drop Final flow: unchanged Megaflow: recirc_id=0,eth,arp,in_port=331,dl_src=00:00:00:25:eb:39 Datapath actions: drop 2022-11-13T08:44:34.077Z|00224|ofproto_dpif_xlate(handler5)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,in_port=13483,vlan_tci=0x0000,dl_src=00:00:00:59:ef:13,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.152.3,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:59:ef:13,arp_tha=ff:ff:ff:ff:ff:ff To solve this issue, modify the OVN NB option bcast_arp_req_flood to false : kubectl ko nbctl set NB_Global . options:bcast_arp_req_flood = false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Underlay Installation"},{"location":"en/start/underlay/#underlay-installation","text":"By default, the default subnet uses Geneve to encapsulate cross-host traffic, and build an overlay network on top of the infrastructure. For the case that you want the container network to use the physical network address directly, you can set the default subnet of Kube-OVN to work in Underlay mode, which can directly assign the address resources in the physical network to the containers, achieving better performance and connectivity with the physical network.","title":"Underlay Installation"},{"location":"en/start/underlay/#limitation","text":"Since the container network in this mode uses physical network directly for L2 packet forwarding, L3 functions such as SNAT/EIP, distributed gateway/centralized gateway in Overlay mode cannot be used. VPC level isolation is also not available for underlay subnet.","title":"Limitation"},{"location":"en/start/underlay/#comparison-with-macvlan","text":"The Underlay mode of Kube-OVN is very similar to the Macvlan, with the following major differences in functionality and performance: Macvlan performs better in terms of throughput and latency performance metrics due to its shorter kernel path and the fact that it does not require OVS for packet processing. Kube-OVN provides arp-proxy functionality through flow tables to mitigate the risk of arp broadcast storms on large-scale networks. Since Macvlan works at the bottom of the kernel and bypasses the host netfilter, Service and NetworkPolicy functionality requires additional development. Kube-OVN provides Service and NetworkPolicy capabilities through the OVS flow table. Kube-OVN Underlay mode provides additional features such as address management, fixed IP and QoS compared to Macvlan.","title":"Comparison with Macvlan"},{"location":"en/start/underlay/#environment-requirements","text":"In Underlay mode, the OVS will bridge a node NIC to the OVS bridge and send packets directly through that node NIC, relying on the underlying network devices for L2/L3 level forwarding capabilities. You need to configure the corresponding gateway, Vlan and security policy in the underlying network device in advance. For OpenStack VM environments, you need to turn off PortSecurity on the corresponding network port. For VMware vSwitch networks, MAC Address Changes , Forged Transmits and Promiscuous Mode Operation should be set to allow . For Hyper-V virtualization, MAC Address Spoofing should be enabled in VM nic advanced features. Public clouds, such as AWS, GCE, AliCloud, etc., do not support user-defined Mac, so they cannot support Underlay mode network. In this scenario, if you want to use Underlay, it is recommended to use the VPC-CNI provided by the corresponding public cloud vendor.. The network interface that is bridged into ovs can not be type of Linux Bridge. For management and container networks using the same NIC, Kube-OVN will transfer the NIC's Mac address, IP address, route, and MTU to the corresponding OVS Bridge to support single NIC deployment of Underlay networks. OVS Bridge name format is br-PROVIDER_NAME \uff0c PROVIDER_NAME is the name of ProviderNetwork (Default: provider).","title":"Environment Requirements"},{"location":"en/start/underlay/#specify-network-mode-when-deploying","text":"This deployment mode sets the default subnet to Underlay mode, and all Pods with no subnet specified will run in the Underlay network by default.","title":"Specify Network Mode When Deploying"},{"location":"en/start/underlay/#download-script","text":"wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/install.sh","title":"Download Script"},{"location":"en/start/underlay/#modify-configuration-options","text":"ENABLE_ARP_DETECT_IP_CONFLICT # disable vlan arp conflict detection if necessary NETWORK_TYPE # set to vlan VLAN_INTERFACE_NAME # set to the NIC that carries the Underlay traffic, e.g. eth1 VLAN_ID # The VLAN Tag need to be added\uff0cif set 0 no vlan tag will be added POD_CIDR # The Underlay network CIDR\uff0c e.g. 192.168.1.0/24 POD_GATEWAY # Underlay physic gateway address, e.g. 192.168.1.1 EXCLUDE_IPS # Exclude ranges to avoid conflicts between container network and IPs already in use on the physical network, e.g. 192.168.1.1..192.168.1.100 ENABLE_LB # If Underlay Subnet needs to visit Service set it to true EXCHANGE_LINK_NAME # If swap the names of the OVS bridge and the bridge interface under the default provider-network. Default to false. LS_DNAT_MOD_DL_DST # If DNAT translate MAC addresses to accelerate service access. Default to true.","title":"Modify Configuration Options"},{"location":"en/start/underlay/#run-the-script","text":"bash install.sh","title":"Run the Script"},{"location":"en/start/underlay/#dynamically-create-underlay-networks-via-crd","text":"This approach dynamically creates an Underlay subnet that Pod can use after installation.","title":"Dynamically Create Underlay Networks via CRD"},{"location":"en/start/underlay/#create-providernetwork","text":"ProviderNetwork provides the abstraction of host NIC to physical network mapping, unifies the management of NICs belonging to the same network, and solves the configuration problems in complex environments with multiple NICs on the same machine, inconsistent NIC names and inconsistent corresponding Underlay networks. Create ProviderNetwork as below: apiVersion: kubeovn.io/v1 kind: ProviderNetwork metadata: name: net1 spec: defaultInterface: eth1 customInterfaces: - interface: eth2 nodes: - node1 excludeNodes: - node2 Note: The length of the ProviderNetwork resource name must not exceed 12. defaultInterface : The default node NIC name. When the ProviderNetwork is successfully created, an OVS bridge named br-net1 (in the format br-NAME ) is created in each node (except excludeNodes) and the specified node NIC is bridged to this bridge. customInterfaces : Optionally, you can specify the NIC to be used for a specific node. excludeNodes : Optional, to specify nodes that do not bridge the NIC. Nodes in this list will be added with the net1.provider-network.ovn.kubernetes.io/exclude=true tag. Other nodes will be added with the following tags: Key Value Description net1.provider-network.ovn.kubernetes.io/ready true bridge work finished, ProviderNetwork is ready on this node net1.provider-network.ovn.kubernetes.io/interface eth1 The name of the bridged NIC in the node. net1.provider-network.ovn.kubernetes.io/mtu 1500 MTU of bridged NIC in node If an IP has been configured on the node NIC, the IP address and the route on the NIC are transferred to the corresponding OVS bridge.","title":"Create ProviderNetwork"},{"location":"en/start/underlay/#create-vlan","text":"Vlan provides an abstraction to bind Vlan Tag and ProviderNetwork. Create a VLAN as below: apiVersion: kubeovn.io/v1 kind: Vlan metadata: name: vlan1 spec: id: 0 provider: net1 id : VLAN ID/Tag\uff0cKube-OVN will add this Vlan tag to traffic, if set 0, no tag is added. provider : The name of ProviderNetwork. Multiple VLAN can use a same ProviderNetwork.","title":"Create VLAN"},{"location":"en/start/underlay/#create-subnet","text":"Bind Vlan to a Subnet as below\uff1a apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 Simply specify the value of vlan as the name of the VLAN to be used. Multiple subnets can refer to the same VLAN.","title":"Create Subnet"},{"location":"en/start/underlay/#create-pod","text":"You can create containers in the normal way, check whether the container IP is in the specified range and whether the container can interoperate with the physical network. For fixed IP requirements, please refer to Fixed Addresses","title":"Create Pod"},{"location":"en/start/underlay/#logical-gateway","text":"For cases where no gateway exists in the physical network, Kube-OVN supports the use of logical gateways configured in the subnet in Underlay mode. To use this feature, set spec.logicalGateway to true for the subnet: apiVersion : kubeovn.io/v1 kind : Subnet metadata : name : subnet1 spec : protocol : IPv4 cidrBlock : 172.17.0.0/16 gateway : 172.17.0.1 vlan : vlan1 logicalGateway : true When this feature is turned on, the Pod does not use an external gateway, but a Logical Router created by Kube-OVN to forward cross-subnet communication.","title":"Logical Gateway"},{"location":"en/start/underlay/#interconnection-of-underlay-and-overlay-networks","text":"If a cluster has both Underlay and Overlay subnets, by default, Pods in the Overlay subnet can access the Pod IPs in the Underlay subnet via a gateway using NAT. From the perspective of Pods in the Underlay subnet, the addresses in the Overlay subnet are external, and require the underlying physical device to forward, but the underlying physical device does not know the addresses in the Overlay subnet and cannot forward. Therefore, Pods in the Underlay subnet cannot access Pods in the Overlay subnet directly via Pod IPs. If you need to enable communication between Underlay and Overlay networks, you need to set the u2oInterconnection of the subnet to true . In this case, Kube-OVN will use an additional Underlay IP to connect the Underlay subnet and the ovn-cluster logical router, and set the corresponding routing rules to enable communication. Unlike the logical gateway, this solution only connects the Underlay and Overlay subnets within Kube-OVN, and other traffic accessing the Internet will still be forwarded through the physical gateway.","title":"Interconnection of Underlay and Overlay Networks"},{"location":"en/start/underlay/#specify-logical-gateway-ip","text":"After the interworking function is enabled, an IP from the subnet will be randomly selected as the logical gateway. If you need to specify the logical gateway of the Underlay Subnet, you can specify the field u2oInterconnectionIP .","title":"Specify logical gateway IP"},{"location":"en/start/underlay/#specify-custom-vpc-for-underlay-subnet-connection","text":"By default, the Underlay Subnet will communicate with the Overlay Subnet on the default VPC. If you want to specify to communicate with a certain VPC, after setting u2oInterconnection to true , specify the subnet.spec.vpc field as the name of the VPC.","title":"Specify custom VPC for Underlay Subnet connection"},{"location":"en/start/underlay/#notice","text":"If there are IP addresses on the host nic, the OS you are using is Ubuntu, and the networking is configured with Netplan, it's recomanded to set NetworkManager as the renderer and to set static IP addresses for the nic (disable DHCP): network : renderer : NetworkManager ethernets : eth0 : dhcp4 : no addresses : - 172.16.143.129/24 version : 2 If the host networking service is NetworkManager, Kube-OVN will remove the host nic from the managed devices (managed by NetworkManager is no) after creating ProviderNetwork: root@ubuntu:~# nmcli device status DEVICE TYPE STATE CONNECTION eth0 ethernet unmanaged netplan-eth0 If you want to change the host nic's IP/route configuration, you need to set the nic managed by NetworkManager manually: nmcli device set eth0 managed yes After setting managed to yes\uff0cKube-OVN will transfer IP and routes on the nic to the OVS bridge, and remove the nic from the managed devices again. Notice \uff1aIf the host nic's MAC is changed, Kube-OVN will not change the OVS bridge's MAC unless kube-ovn-cni is restarted.","title":"Notice"},{"location":"en/start/underlay/#known-issues","text":"","title":"Known Issues"},{"location":"en/start/underlay/#when-the-physical-network-is-enabled-with-hairpin-pod-network-is-abnormal","text":"When physical networks enable hairpin or similar behaviors, problems such as gateway check failure when creating Pods and abnormal network communication of Pods may occur. This is because the default MAC learning function of OVS bridge does not support this kind of network environment. To solve this problem, it is necessary to turn off hairpin (or modify the relevant configuration of physical network), or update the Kube-OVN version.","title":"When the physical network is enabled with hairpin, Pod network is abnormal"},{"location":"en/start/underlay/#when-there-are-a-large-number-of-pods-gateway-check-for-new-pods-fails","text":"If there are a large number of Pods running on the same node (more than 300), it may cause packet loss due to the OVS flow table resubmit times exceeding the upper limit of ARP broadcast packets. 2022-11-13T08:43:46.782Z|00222|ofproto_dpif_upcall(handler5)|WARN|Flow: arp,in_port=331,vlan_tci=0x0000,dl_src=00:00:00:25:eb:39,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.131.240,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:25:eb:39,arp_tha=ff:ff:ff:ff:ff:ff bridge(\"br-int\") ---------------- 0. No match. >>>> received packet on unknown port 331 <<<< drop Final flow: unchanged Megaflow: recirc_id=0,eth,arp,in_port=331,dl_src=00:00:00:25:eb:39 Datapath actions: drop 2022-11-13T08:44:34.077Z|00224|ofproto_dpif_xlate(handler5)|WARN|over 4096 resubmit actions on bridge br-int while processing arp,in_port=13483,vlan_tci=0x0000,dl_src=00:00:00:59:ef:13,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.213.152.3,arp_tpa=10.213.159.254,arp_op=1,arp_sha=00:00:00:59:ef:13,arp_tha=ff:ff:ff:ff:ff:ff To solve this issue, modify the OVN NB option bcast_arp_req_flood to false : kubectl ko nbctl set NB_Global . options:bcast_arp_req_flood = false \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"When there are a large number of Pods, gateway check for new Pods fails"},{"location":"en/start/uninstall/","text":"Uninstall \u00b6 If you need to remove the Kube-OVN and replace it with another network plugin, please follow the steps below to remove all the corresponding Kube-OVN component and OVS configuration to avoid interference with other network plugins. Feel free to contact us with an Issue to give us feedback on why you don't use Kube-OVN to help us improve it. Delete Resource in Kubernetes \u00b6 Download and run the script below to delete resource created in Kubernetes: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/cleanup.sh bash cleanup.sh Cleanup Config and Logs on Every Node \u00b6 Run the following commands on each node to clean up the configuration retained by ovsdb and openvswitch: rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn rm -fr /var/log/kube-ovn Reboot Node \u00b6 Reboot the machine to ensure that the corresponding NIC information and iptable/ipset rules are cleared to avoid the interference with other network plugins: reboot \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Uninstall"},{"location":"en/start/uninstall/#uninstall","text":"If you need to remove the Kube-OVN and replace it with another network plugin, please follow the steps below to remove all the corresponding Kube-OVN component and OVS configuration to avoid interference with other network plugins. Feel free to contact us with an Issue to give us feedback on why you don't use Kube-OVN to help us improve it.","title":"Uninstall"},{"location":"en/start/uninstall/#delete-resource-in-kubernetes","text":"Download and run the script below to delete resource created in Kubernetes: wget https://raw.githubusercontent.com/kubeovn/kube-ovn/master/dist/images/cleanup.sh bash cleanup.sh","title":"Delete Resource in Kubernetes"},{"location":"en/start/uninstall/#cleanup-config-and-logs-on-every-node","text":"Run the following commands on each node to clean up the configuration retained by ovsdb and openvswitch: rm -rf /var/run/openvswitch rm -rf /var/run/ovn rm -rf /etc/origin/openvswitch/ rm -rf /etc/origin/ovn/ rm -rf /etc/cni/net.d/00-kube-ovn.conflist rm -rf /etc/cni/net.d/01-kube-ovn.conflist rm -rf /var/log/openvswitch rm -rf /var/log/ovn rm -fr /var/log/kube-ovn","title":"Cleanup Config and Logs on Every Node"},{"location":"en/start/uninstall/#reboot-node","text":"Reboot the machine to ensure that the corresponding NIC information and iptable/ipset rules are cleared to avoid the interference with other network plugins: reboot \u5fae\u4fe1\u7fa4 Slack Twitter Support","title":"Reboot Node"}]}
\ No newline at end of file
diff --git a/v1.12.x/sitemap.xml.gz b/v1.12.x/sitemap.xml.gz
index b39539e7f..7e0e9f6d1 100644
Binary files a/v1.12.x/sitemap.xml.gz and b/v1.12.x/sitemap.xml.gz differ
diff --git a/v1.12.x/zh/advance/ovn-eip-fip-snat/index.html b/v1.12.x/zh/advance/ovn-eip-fip-snat/index.html
index 762c4bb46..af44cfec3 100644
--- a/v1.12.x/zh/advance/ovn-eip-fip-snat/index.html
+++ b/v1.12.x/zh/advance/ovn-eip-fip-snat/index.html
@@ -21,7 +21,7 @@
 <span class=w>          </span>-<span class=w> </span>--external-gateway-switch<span class=o>=</span>external204<span class=w> </span>
 
 <span class=c1>### 以上配置都和下面的公网网络配置 vlan id 和资源名保持一致，目前仅支持指定一个 underlay 公网作为默认外部公网。</span>
-</code></pre></div> <p>该配置项的设计和使用主要考虑了如下因素：</p> <ul> <li>基于该配置项可以对接到 provider network，vlan，subnet 的资源。</li> <li>基于该配置项可以将默认 vpc enable_eip_snat 功能对接到已有的 vlan，subnet 资源，同时支持公网 ip 的 ipam。</li> <li>如果仅使用默认 vpc 的 enable_eip_snat 模式, 且仅使用旧的基于 pod annotaion 的 eip fip snat，那么这个配置无需配置。</li> <li>基于该配置可以不使用默认 vpc enable_eip_snat 流程，仅通过对应到 vlan，subnet 流程，可以兼容仅自定义 vpc 使用 eip snat 的使用场景。</li> </ul> <h3 id=11-underlay>1.1 准备 underlay 公网网络<a class=headerlink href=#11-underlay title="Permanent link">&para;</a></h3> <div class=highlight><pre><span></span><code><span class=c1># 准备 provider-network， vlan， subnet</span>
+</code></pre></div> <p>该配置项的设计和使用主要考虑了如下因素：</p> <ul> <li>基于该配置项可以对接到 provider network，vlan，subnet 的资源。</li> <li>基于该配置项可以将默认 vpc enable_eip_snat 功能对接到已有的 vlan，subnet 资源，同时支持公网 ip 的 ipam。</li> <li>如果仅使用默认 vpc 的 enable_eip_snat 模式, 且仅使用旧的基于 pod annotaion 的 fip snat，那么这个配置无需配置。</li> <li>基于该配置可以不使用默认 vpc enable_eip_snat 流程，仅通过对应到 vlan，subnet 流程，可以兼容仅自定义 vpc 使用 eip snat 的使用场景。</li> </ul> <h3 id=11-underlay>1.1 准备 underlay 公网网络<a class=headerlink href=#11-underlay title="Permanent link">&para;</a></h3> <div class=highlight><pre><span></span><code><span class=c1># 准备 provider-network， vlan， subnet</span>
 <span class=c1># cat 01-provider-network.yaml</span>
 apiVersion:<span class=w> </span>kubeovn.io/v1
 kind:<span class=w> </span>ProviderNetwork
@@ -120,14 +120,14 @@
 Route<span class=w> </span>Table<span class=w> </span>&lt;main&gt;:
 <span class=w>                </span><span class=m>0</span>.0.0.0/0<span class=w>              </span><span class=m>10</span>.5.204.254<span class=w> </span>dst-ip
 <span class=c1># 目前该路由已自动维护</span>
-</code></pre></div> <h2 id=2-ovn-eip>2. ovn-eip<a class=headerlink href=#2-ovn-eip title="Permanent link">&para;</a></h2> <p>该功能和 iptables-eip 设计和使用方式基本一致，ovn-eip 目前有三种 type</p> <ul> <li>nat: 用于 ovn dnat，fip, snat, 这些 nat 类型会记录在 status 中</li> <li>lrp: 用于 vpc 和公网相连的资源</li> <li>node-ext-gw: 用于 ovn 基于 bfd 的 ecmp 静态路由场景</li> </ul> <div class=highlight><pre><span></span><code>---
+</code></pre></div> <h2 id=2-ovn-eip>2. ovn-eip<a class=headerlink href=#2-ovn-eip title="Permanent link">&para;</a></h2> <p>该功能和 iptables-eip 设计和使用方式基本一致，ovn-eip 目前有三种 type</p> <ul> <li>nat: 用于 ovn dnat，fip, snat, 这些 nat 类型会记录在 status 中</li> <li>lrp: Resources connected to the public network from a vpc can be used by nat</li> <li>lsp: 用于 ovn 基于 bfd 的 ecmp 静态路由场景，在网关节点上提供一个 ovs internal port 作为 ecmp 路由的下一跳</li> </ul> <div class=highlight><pre><span></span><code>---
 kind:<span class=w> </span>OvnEip
 apiVersion:<span class=w> </span>kubeovn.io/v1
 metadata:
 <span class=w>  </span>name:<span class=w> </span>eip-static
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>fip
+<span class=w>  </span>type:<span class=w> </span>nat
 
 <span class=c1># 动态分配一个 eip 资源，该资源预留用于 fip 场景</span>
 </code></pre></div> <h3 id=21-ovn-fip-pod-fip>2.1 ovn-fip 为 pod 绑定一个 fip<a class=headerlink href=#21-ovn-fip-pod-fip title="Permanent link">&para;</a></h3> <div class=highlight><pre><span></span><code><span class=c1># k get po -o wide -n vpc1 vpc-1-busybox01</span>
@@ -146,7 +146,7 @@
 <span class=w>  </span>name:<span class=w> </span>eip-static
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>fip
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnFip
@@ -207,7 +207,7 @@
 <span class=w>  </span>name:<span class=w> </span>eip-for-vip
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>fip
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnFip
@@ -268,7 +268,7 @@
 <span class=w>  </span>name:<span class=w> </span>snat-for-subnet-in-vpc
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>snat
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnSnatRule
@@ -286,7 +286,7 @@
 <span class=w>  </span>name:<span class=w> </span>snat-for-pod-vpc-ip
 spec:
 <span class=w>  </span>externalSubnet:<span class=w> </span>external204
-<span class=w>  </span>type:<span class=w> </span>snat
+<span class=w>  </span>type:<span class=w> </span>nat
 
 ---
 kind:<span class=w> </span>OvnSnatRule
@@ -417,7 +417,7 @@
 <span class=c1># kubectl get odnat eip-dnat </span>
 NAME<span class=w>       </span>EIP<span class=w>        </span>PROTOCOL<span class=w>   </span>V4EIP<span class=w>       </span>V4IP<span class=w>          </span>INTERNALPORT<span class=w>   </span>EXTERNALPORT<span class=w>   </span>IPNAME<span class=w>          </span>READY
 eip-dnat<span class=w>   </span>eip-dnat<span class=w>   </span>tcp<span class=w>        </span><span class=m>10</span>.5.49.4<span class=w>   </span><span class=m>192</span>.168.0.4<span class=w>   </span><span class=m>22</span><span class=w>             </span><span class=m>22</span><span class=w>             </span>test-dnat-vip<span class=w>   </span><span class=nb>true</span>
-</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> 最后更新: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年8月3日</span> <br> 创建日期: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年3月3日</span> </small> </div> <h2 id=__comments>评论</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=zh-CN crossorigin=anonymous async>
+</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> 最后更新: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年8月8日</span> <br> 创建日期: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年3月3日</span> </small> </div> <h2 id=__comments>评论</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=zh-CN crossorigin=anonymous async>
 </script> <script>
     var palette = __md_get("__palette")
     if (palette && typeof palette.color === "object")
diff --git a/v1.12.x/zh/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html b/v1.12.x/zh/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html
index 89a57b637..f981f0b12 100644
--- a/v1.12.x/zh/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html
+++ b/v1.12.x/zh/advance/ovn-l3-ha-based-ecmp-with-bfd/index.html
@@ -10,7 +10,7 @@
 
 pod--&gt;vpc-subnet--&gt;vpc--&gt;snat--&gt;ecmp--&gt;external-subnet--&gt;gw-node1-ovnext0--&gt; node1-external-switch
 external-subnet--&gt;gw-node2-ovnext0--&gt; node2-external-switch
-external-subnet--&gt;gw-node3-ovnext0--&gt; node3-external-switch</code></pre> <p>该功能的使用方式和<a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> 基本一致，一致的部分包括 install.sh 的部署部分，provider-network，vlan，subnet 的准备部分。</p> <p>至于不相同的部分，会在以下部分具体阐述，主要包括 node-ext-gw 类型的 ovn-eip 的创建，以及基于 vpc enable_bfd 自动维护 bfd 以及 ecmp 静态路由。</p> <h2 id=1>1. 部署<a class=headerlink href=#1 title="Permanent link">&para;</a></h2> <h3 id=11-underlay>1.1 准备 underlay 公网网络<a class=headerlink href=#11-underlay title="Permanent link">&para;</a></h3> <h3 id=12-vpc-eip_snat>1.2 默认 vpc 启用 eip_snat<a class=headerlink href=#12-vpc-eip_snat title="Permanent link">&para;</a></h3> <h3 id=13-vpc-eip-snat-fip>1.3 自定义 vpc 启用 eip snat fip 功能<a class=headerlink href=#13-vpc-eip-snat-fip title="Permanent link">&para;</a></h3> <p>以上部分和 <a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> 完全一致，这些功能验证通过后，可以直接基于如下方式，将 vpc 切换到基于 ecmp 的 bfd 静态路由，当然也可以切回。</p> <p>自定义 vpc 使用该功能之前，需要先提供好网关节点，至少需要提供 2 个以上网关节点，注意当前实现 ovn-eip 的名字必须和网关节点名保持一致，目前没有做该资源的自动化维护。</p> <div class=highlight><pre><span></span><code><span class=c1># cat gw-node-eip.yaml</span>
+external-subnet--&gt;gw-node3-ovnext0--&gt; node3-external-switch</code></pre> <p>该功能的使用方式和<a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> 基本一致，一致的部分包括 install.sh 的部署部分，provider-network，vlan，subnet 的准备部分。</p> <p>至于不相同的部分，会在以下部分具体阐述，主要包括 lsp 类型的 ovn-eip 的创建，以及基于 vpc enable_bfd 自动维护 bfd 以及 ecmp 静态路由。</p> <h2 id=1>1. 部署<a class=headerlink href=#1 title="Permanent link">&para;</a></h2> <h3 id=11-underlay>1.1 准备 underlay 公网网络<a class=headerlink href=#11-underlay title="Permanent link">&para;</a></h3> <h3 id=12-vpc-eip_snat>1.2 默认 vpc 启用 eip_snat<a class=headerlink href=#12-vpc-eip_snat title="Permanent link">&para;</a></h3> <h3 id=13-vpc-eip-snat-fip>1.3 自定义 vpc 启用 eip snat fip 功能<a class=headerlink href=#13-vpc-eip-snat-fip title="Permanent link">&para;</a></h3> <p>以上部分和 <a href=../ovn-eip-fip-snat/ >ovn-eip-fip-snat.md</a> 完全一致，这些功能验证通过后，可以直接基于如下方式，将 vpc 切换到基于 ecmp 的 bfd 静态路由，当然也可以切回。</p> <p>自定义 vpc 使用该功能之前，需要先提供好网关节点，至少需要提供 2 个以上网关节点，注意当前实现 ovn-eip 的名字必须和网关节点名保持一致，目前没有做该资源的自动化维护。</p> <div class=highlight><pre><span></span><code><span class=c1># cat gw-node-eip.yaml</span>
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnEip</span>
 <span class=nt>apiVersion</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">kubeovn.io/v1</span>
@@ -18,7 +18,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">pc-node-1</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">external204</span>
-<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">node-ext-gw</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">lsp</span>
 
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnEip</span>
@@ -27,7 +27,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">pc-node-2</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">external204</span>
-<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">node-ext-gw</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">lsp</span>
 
 <span class=nn>---</span>
 <span class=nt>kind</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">OvnEip</span>
@@ -36,7 +36,7 @@
 <span class=w>  </span><span class=nt>name</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">pc-node-3</span>
 <span class=nt>spec</span><span class=p>:</span>
 <span class=w>  </span><span class=nt>externalSubnet</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">external204</span>
-<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">node-ext-gw</span>
+<span class=w>  </span><span class=nt>type</span><span class=p>:</span><span class=w> </span><span class="l l-Scalar l-Scalar-Plain">lsp</span>
 </code></pre></div> <p>由于这个场景目前设计上是供 vpc ecmp 出公网使用，所以以上在没有 vpc 启用 bfd 的时候，即不存在带有 enable bfd 标签的 lrp 的 ovn eip 的时候，网关节点不会触发创建网关网卡，也无法成功启动对端 bfd 会话的监听。</p> <h2 id=2-vpc-ecmp-bfd-l3-ha>2. 自定义 vpc 启用 ecmp bfd L3 HA 公网功能<a class=headerlink href=#2-vpc-ecmp-bfd-l3-ha title="Permanent link">&para;</a></h2> <div class=highlight><pre><span></span><code><span class=c1># cat 01-vpc-ecmp-enable-external-bfd.yml</span>
 kind:<span class=w> </span>Vpc
 apiVersion:<span class=w> </span>kubeovn.io/v1
@@ -258,7 +258,7 @@
 <span class=c1># 而且 ovnext ns 中的对端 bfd 会话也自动移除</span>
 <span class=c1># 该切换过程保持 vpc subnet 内保持 ping 未看到(秒级)丢包</span>
 <span class=c1># 再切换回去 也未看到(秒级)丢包</span>
-</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> 最后更新: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年5月9日</span> <br> 创建日期: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年3月3日</span> </small> </div> <h2 id=__comments>评论</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=zh-CN crossorigin=anonymous async>
+</code></pre></div> <p><a class=md-button href=https://ma.alauda.cn/p/0f39a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 4C5.36 4 2 6.69 2 10c0 1.89 1.08 3.56 2.78 4.66L4 17l2.5-1.5c.89.31 1.87.5 2.91.5A5.22 5.22 0 0 1 9 14c0-3.31 3.13-6 7-6 .19 0 .38 0 .56.03C15.54 5.69 12.78 4 9.5 4m-3 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m5 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1M16 9c-3.31 0-6 2.24-6 5s2.69 5 6 5c.67 0 1.31-.08 1.91-.25L20 20l-.62-1.87C20.95 17.22 22 15.71 22 14c0-2.76-2.69-5-6-5m-2 2.5a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1m4 0a1 1 0 0 1 1 1 1 1 0 0 1-1 1 1 1 0 0 1-1-1 1 1 0 0 1 1-1Z"/></svg></span> 微信群</a> <a class=md-button href=https://communityinviter.com/apps/kube-ovn/kube-ovn/ target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M6 15a2 2 0 0 1-2 2 2 2 0 0 1-2-2 2 2 0 0 1 2-2h2v2m1 0a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5a2 2 0 0 1-2 2 2 2 0 0 1-2-2v-5m2-8a2 2 0 0 1-2-2 2 2 0 0 1 2-2 2 2 0 0 1 2 2v2H9m0 1a2 2 0 0 1 2 2 2 2 0 0 1-2 2H4a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5m8 2a2 2 0 0 1 2-2 2 2 0 0 1 2 2 2 2 0 0 1-2 2h-2v-2m-1 0a2 2 0 0 1-2 2 2 2 0 0 1-2-2V5a2 2 0 0 1 2-2 2 2 0 0 1 2 2v5m-2 8a2 2 0 0 1 2 2 2 2 0 0 1-2 2 2 2 0 0 1-2-2v-2h2m0-1a2 2 0 0 1-2-2 2 2 0 0 1 2-2h5a2 2 0 0 1 2 2 2 2 0 0 1-2 2h-5Z"/></svg></span> Slack</a> <a class=md-button href=https://twitter.com/KubeOvn target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M22.46 6c-.77.35-1.6.58-2.46.69.88-.53 1.56-1.37 1.88-2.38-.83.5-1.75.85-2.72 1.05C18.37 4.5 17.26 4 16 4c-2.35 0-4.27 1.92-4.27 4.29 0 .34.04.67.11.98C8.28 9.09 5.11 7.38 3 4.79c-.37.63-.58 1.37-.58 2.15 0 1.49.75 2.81 1.91 3.56-.71 0-1.37-.2-1.95-.5v.03c0 2.08 1.48 3.82 3.44 4.21a4.22 4.22 0 0 1-1.93.07 4.28 4.28 0 0 0 4 2.98 8.521 8.521 0 0 1-5.33 1.84c-.34 0-.68-.02-1.02-.06C3.44 20.29 5.7 21 8.12 21 16 21 20.33 14.46 20.33 8.79c0-.19 0-.37-.01-.56.84-.6 1.56-1.36 2.14-2.23Z"/></svg></span> Twitter</a> <a class=md-button href=https://ma.alauda.cn/p/2f53a target=_blank><span class=twemoji><svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="m20 8-8 5-8-5V6l8 5 8-5m0-2H4c-1.11 0-2 .89-2 2v12a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2V6a2 2 0 0 0-2-2Z"/></svg></span> Support</a></p> <hr> <div class=md-source-file> <small> 最后更新: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年8月8日</span> <br> 创建日期: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">2023年3月3日</span> </small> </div> <h2 id=__comments>评论</h2> <script src=https://giscus.app/client.js data-repo=kubeovn/kube-ovn data-repo-id="MDEwOlJlcG9zaXRvcnkxNzcwNjg5NjE=" data-category=Announcements data-category-id=DIC_kwDOCo3boc4CAj60 data-mapping=pathname data-reactions-enabled=1 data-emit-metadata=0 data-input-position=bottom data-theme=light data-lang=zh-CN crossorigin=anonymous async>
 </script> <script>
     var palette = __md_get("__palette")
     if (palette && typeof palette.color === "object")
diff --git a/versions.json b/versions.json
index 7b4df844c..0bc829df5 100644
--- a/versions.json
+++ b/versions.json
@@ -1 +1 @@
-[{"version": "v1.13.x", "title": "v1.13.x (dev)", "aliases": ["dev"]}, {"version": "v1.12.x", "title": "v1.12.x (current)", "aliases": ["current", "stable"]}, {"version": "v1.11.x", "title": "v1.11.x", "aliases": []}, {"version": "v1.10.x", "title": "v1.10.x", "aliases": []}]
\ No newline at end of file
+[{"version": "v1.13.x", "title": "v1.13.x (dev)", "aliases": ["dev"]}, {"version": "v1.12.x", "title": "v1.12.x (current)", "aliases": ["stable", "current"]}, {"version": "v1.11.x", "title": "v1.11.x", "aliases": []}, {"version": "v1.10.x", "title": "v1.10.x", "aliases": []}]
\ No newline at end of file