Skip to content

Commit

Permalink
Merge pull request #1004 from ywc689/kni-replacement
Browse files Browse the repository at this point in the history 
Kni replacement
  • Loading branch information
@ywc689
ywc689 authored Dec 6, 2024
2 parents 9c87f87 + 7eb35f8 commit e3ef22a
Show file tree
Hide file tree
Showing 32 changed files with 1,327 additions and 374 deletions.
7 changes: 4 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ $ tar xf dpdk-20.11.10.tar.xz

### DPDK patchs

There are some patches for DPDK to support extra features needed by DPVS. Apply them if needed. For example, there's a patch for DPDK `kni` driver for hardware multicast, apply it if you are to launch `ospfd` on `kni` device.
There are some patches for DPDK to support extra features needed by DPVS. Apply them if needed. For example, there's a patch for DPDK `rte_kni` driver for hardware multicast, apply it if you want to use `rte_kni` as your management network for such exception data path as SSH, OSPF, BGP, etc.

> Notes: It's assumed we are in DPVS root directory where you have installed dpdk-stable-20.11.10 source codes. Please note it's not mandatory, just for convenience.
Expand Down Expand Up @@ -146,7 +146,7 @@ Next, install kernel modules required by DPDK and DPVS.
Depending on your NIC and system, NIC may require binding a DPDK-compitable driver, such as `vfio-pci`, `igb_uio`, or `uio_pci_generic`. Refer to [DPDK doc](https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html) for more details. In this test, we use the linux standard UIO kernel module `uio_pci_generic`.

* KNI kernel module:
KNI kernel module `rte_kni.ko` is required by DPVS as the exception data path which processes packets not dealt with in DPVS to kernel stack.
KNI kernel module `rte_kni.ko` is required as a solution to the exception data path to handle all packets not processed in DPVS.

```bash
$ modprobe uio_pci_generic
Expand All @@ -164,7 +164,8 @@ $ ./usertools/dpdk-devbind.py -b uio_pci_generic 0000:06:00.0
> 2. `dpdk-devbind.py -u` can be used to unbind driver and switch it back to Linux driver like `ixgbe`. Use `lspci` or `ethtool -i eth0` to check the NIC's PCI bus-id. Please refer to [DPDK Doc:Binding and Unbinding Network Ports to/from the Kernel Modules](https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules) for more details.
> 3. NVIDIA/Mellanox NIC uses bifurcated driver which doesn't rely on UIO/VFIO driver, so not bind any DPDK driver kernel module, but [NVIDIA MLNX_OFED/EN](https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/) is required. Refer to [Mellanox DPDK](https://enterprise-support.nvidia.com/s/article/mellanox-dpdk) for its PMD and [Compilation Prerequisites](https://doc.dpdk.org/guides/platform/mlx5.html#linux-prerequisites) for OFED installation.
> 4. A kernel module parameter `carrier` has been added to `rte_kni.ko` since [DPDK v18.11](https://elixir.bootlin.com/dpdk/v18.11/source/kernel/linux/kni/kni_misc.c), and the default value for it is "off". We need to load `rte_kni.ko` with extra parameter `carrier=on` to make KNI devices work properly.
> 5. Multiple DPVS instances can run on a single server if there are enough NICs or VFs within one NIC. Refer to [tutorial:Multiple Instances](https://github.com/iqiyi/dpvs/blob/devel/doc/tutorial.md#multi-instance) for details.
> 5. Following the DPDK technical board decision and refinement, the KNI kernel module, library and PMD was removed from the DPDK 23.11 release (refer to [ABI and API Deprecation(DPDK 22.11)](https://doc.dpdk.org/guides-22.11/rel_notes/deprecation.html)). As a replacement solution, DPVS has supported [virtio-user as exception path](https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html), which is default off now and can be enabled with `CONFIG_KNI_VIRTIO_USER` in config.mk.
> 6. Multiple DPVS instances can run on a single server if there are enough NICs or VFs within one NIC. Refer to [tutorial:Multiple Instances](https://github.com/iqiyi/dpvs/blob/devel/doc/tutorial.md#multi-instance) for details.
## Build DPVS

Expand Down
7 changes: 3 additions & 4 deletions conf/dpvs.conf.items
View file
Original file line number Diff line number Diff line change
Expand Up @@ -201,10 +201,9 @@ ipv6_defs {
<init> method "hlist" <"hlist"/"lpm">
recycle_time 10 <10, 1-36000>
lpm {
<init> lpm6_max_rules 1024 <1024, 16-2147483647>
<init> lpm6_num_tbl8s 65536 <65536, 16-2147483647>
<init> rt6_array_size 65536 <65536, 16-2147483647>
<init> rt6_hash_bucket 256 <256, 2-2147483647>
<init> lpm6_max_rules 1024 <1024, 16-65536>
<init> lpm6_num_tbl8s 16384 <16384, 256-1048576>
<init> rt6_hash_bucket 256 <256, 16-65536>
}
}
}
Expand Down
2 changes: 2 additions & 0 deletions config.mk
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ export CONFIG_IXGEB_PMD=y
export CONFIG_DPVS_LOG=y
export CONFIG_PDUMP=y
export CONFIG_ICMP_REDIRECT_CORE=n
export CONFIG_KNI_VIRTIO_USER=n

# debugging and logging
export CONFIG_DEBUG=n
Expand All @@ -31,3 +32,4 @@ export CONFIG_MSG_DEBUG=n
export CONFIG_DPVS_MP_DEBUG=n
export CONFIG_DPVS_NETIF_DEBUG=n
export CONFIG_DPVS_ICMP_DEBUG=n
export CONFIG_DPVS_ROUTE_DEBUG=n
Binary file removed doc/pics/kni.png
View file
Binary file not shown.
4 changes: 4 additions & 0 deletions doc/pics/kni.svg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
92 changes: 92 additions & 0 deletions doc/pics/tutorial.drawio
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
<mxfile host="app.diagrams.net" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" border="50" scale="3" compressed="false" locked="false" version="24.7.13">
<diagram id="eFARZ7ye3ZyHCSxxOhFb" name="kni">
<mxGraphModel dx="989" dy="545" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
<mxCell id="j9tfoosSE3qILXxTiQy9-3" value="&lt;font style=&quot;font-size: 24px;&quot;&gt;&lt;b style=&quot;&quot;&gt;DPVS&lt;/b&gt;&lt;/font&gt;&lt;div&gt;&lt;br&gt;&lt;/div&gt;&lt;div&gt;&lt;br&gt;&lt;/div&gt;&lt;div&gt;&lt;br&gt;&lt;/div&gt;&lt;div&gt;&lt;br&gt;&lt;/div&gt;&lt;div&gt;&lt;br&gt;&lt;/div&gt;&lt;div&gt;&lt;br&gt;&lt;/div&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#1ba1e2;strokeColor=#006EAF;fontColor=#ffffff;" vertex="1" parent="1">
<mxGeometry x="450" y="490" width="211.88" height="120" as="geometry" />
</mxCell>
<mxCell id="j9tfoosSE3qILXxTiQy9-5" value="&lt;font style=&quot;font-size: 14px;&quot;&gt;&lt;b&gt;dpdk0&lt;/b&gt;&lt;/font&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;" vertex="1" parent="1">
<mxGeometry x="501.88" y="523" width="70" height="30" as="geometry" />
</mxCell>
<mxCell id="j9tfoosSE3qILXxTiQy9-7" value="&lt;font style=&quot;font-size: 14px;&quot;&gt;&lt;b&gt;dpdk1&lt;/b&gt;&lt;/font&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;" vertex="1" parent="1">
<mxGeometry x="581.88" y="523" width="70" height="30" as="geometry" />
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-5" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;strokeColor=#FF0000;strokeWidth=2;startArrow=classicThin;startFill=0;endArrow=classicThin;endFill=0;dashed=1;" edge="1" parent="1" source="j9tfoosSE3qILXxTiQy9-5" target="O6jyKfrmbOcH9zmBw9ze-1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="IBzB3Yzv9ICFWQqCQZ6M-7" value="&lt;font style=&quot;font-size: 18px;&quot;&gt;&lt;b&gt;kernel&amp;nbsp;&lt;/b&gt;&lt;/font&gt;&lt;div&gt;&lt;font style=&quot;font-size: 18px;&quot;&gt;&lt;b&gt;TCP/IP Stack&lt;/b&gt;&lt;/font&gt;&lt;div&gt;&lt;font style=&quot;font-size: 18px;&quot;&gt;&lt;br&gt;&lt;/font&gt;&lt;/div&gt;&lt;div&gt;&lt;font style=&quot;font-size: 18px;&quot;&gt;&lt;br&gt;&lt;/font&gt;&lt;/div&gt;&lt;/div&gt;" style="rounded=1;whiteSpace=wrap;html=1;strokeColor=#CC6600;strokeWidth=2;" vertex="1" parent="1">
<mxGeometry x="150" y="670" width="210" height="110" as="geometry" />
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-7" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.25;entryY=0;entryDx=0;entryDy=0;strokeWidth=2;dashed=1;startArrow=classicThin;startFill=0;endArrow=classicThin;endFill=0;strokeColor=#FF0000;" edge="1" parent="1" source="IBzB3Yzv9ICFWQqCQZ6M-3" target="IBzB3Yzv9ICFWQqCQZ6M-4">
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="246" y="660" />
<mxPoint x="188" y="660" />
</Array>
</mxGeometry>
</mxCell>
<mxCell id="IBzB3Yzv9ICFWQqCQZ6M-3" value="&lt;div&gt;&lt;span style=&quot;color: rgb(0, 0, 0);&quot;&gt;&lt;font style=&quot;font-size: 18px;&quot;&gt;&lt;b&gt;Kernel Based&lt;/b&gt;&lt;/font&gt;&lt;/span&gt;&lt;/div&gt;&lt;div&gt;&lt;span style=&quot;color: rgb(0, 0, 0);&quot;&gt;&lt;font size=&quot;3&quot; style=&quot;&quot;&gt;&lt;b&gt;Applications&lt;/b&gt;&lt;/font&gt;&lt;/span&gt;&lt;br&gt;&lt;/div&gt;&lt;div&gt;&lt;span style=&quot;color: rgb(0, 0, 0);&quot;&gt;&lt;font style=&quot;font-size: 14px;&quot;&gt;&lt;br&gt;&lt;/font&gt;&lt;/span&gt;&lt;/div&gt;&lt;div&gt;&lt;span style=&quot;color: rgb(0, 0, 0);&quot;&gt;&lt;font style=&quot;font-size: 14px;&quot;&gt;sshd|keepalived|ospfd|bird|...&lt;/font&gt;&lt;/span&gt;&lt;/div&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffff88;strokeColor=#36393d;" vertex="1" parent="1">
<mxGeometry x="140" y="490" width="212.5" height="122" as="geometry" />
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-8" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.5;entryY=1;entryDx=0;entryDy=0;strokeWidth=2;dashed=1;startArrow=classicThin;startFill=0;endArrow=classicThin;endFill=0;strokeColor=#FF0000;" edge="1" parent="1" source="IBzB3Yzv9ICFWQqCQZ6M-4" target="IBzB3Yzv9ICFWQqCQZ6M-8">
<mxGeometry relative="1" as="geometry">
<Array as="points">
<mxPoint x="205" y="790" />
<mxPoint x="435" y="790" />
</Array>
</mxGeometry>
</mxCell>
<mxCell id="IBzB3Yzv9ICFWQqCQZ6M-4" value="&lt;font color=&quot;#cc6600&quot; style=&quot;font-size: 14px;&quot;&gt;&lt;b&gt;dpdk0.kni&lt;/b&gt;&lt;/font&gt;" style="rounded=1;whiteSpace=wrap;html=1;strokeColor=#CC6600;" vertex="1" parent="1">
<mxGeometry x="170" y="730" width="70" height="30" as="geometry" />
</mxCell>
<mxCell id="IBzB3Yzv9ICFWQqCQZ6M-6" value="&lt;font color=&quot;#cc6600&quot; style=&quot;font-size: 14px;&quot;&gt;&lt;b&gt;dpdk1.kni&lt;/b&gt;&lt;/font&gt;" style="rounded=1;whiteSpace=wrap;html=1;strokeColor=#CC6600;" vertex="1" parent="1">
<mxGeometry x="270" y="730" width="70" height="30" as="geometry" />
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-9" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;exitX=0.75;exitY=0;exitDx=0;exitDy=0;strokeWidth=2;dashed=1;startArrow=classicThin;startFill=0;endArrow=classicThin;endFill=0;strokeColor=#FF0000;" edge="1" parent="1" source="IBzB3Yzv9ICFWQqCQZ6M-8" target="j9tfoosSE3qILXxTiQy9-5">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="IBzB3Yzv9ICFWQqCQZ6M-8" value="&lt;font style=&quot;font-size: 18px;&quot;&gt;KNI kmod&lt;/font&gt;&lt;div&gt;&lt;font style=&quot;font-size: 12px;&quot;&gt;rte_kni.ko|vhost.ko&lt;/font&gt;&lt;/div&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#666666;gradientColor=#b3b3b3;" vertex="1" parent="1">
<mxGeometry x="380" y="670" width="110" height="50" as="geometry" />
</mxCell>
<mxCell id="oO2gNdg5D-aUiUnC3wbW-1" value="&lt;div&gt;&lt;font style=&quot;font-size: 18px;&quot;&gt;Linux Drivers&lt;/font&gt;&lt;/div&gt;&lt;font style=&quot;font-size: 12px;&quot;&gt;UIO|VFIO|Bifurcated Driver&lt;/font&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;gradientColor=#b3b3b3;strokeColor=#666666;" vertex="1" parent="1">
<mxGeometry x="497.88" y="670" width="164" height="50" as="geometry" />
</mxCell>
<mxCell id="oO2gNdg5D-aUiUnC3wbW-3" value="" style="endArrow=none;dashed=1;html=1;rounded=0;strokeWidth=2;dashPattern=1 1;strokeColor=#808080;" edge="1" parent="1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="100" y="640" as="sourcePoint" />
<mxPoint x="740" y="640" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="oO2gNdg5D-aUiUnC3wbW-4" value="" style="endArrow=none;dashed=1;html=1;rounded=0;strokeWidth=2;dashPattern=1 1;strokeColor=#808080;" edge="1" parent="1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="94" y="805" as="sourcePoint" />
<mxPoint x="734" y="805" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-4" value="" style="group" vertex="1" connectable="0" parent="1">
<mxGeometry x="510" y="830" width="132.87" height="64" as="geometry" />
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-1" value="" style="sketch=0;pointerEvents=1;shadow=0;dashed=0;html=1;strokeColor=none;fillColor=#505050;labelPosition=center;verticalLabelPosition=bottom;verticalAlign=top;outlineConnect=0;align=center;shape=mxgraph.office.devices.nic;" vertex="1" parent="O6jyKfrmbOcH9zmBw9ze-4">
<mxGeometry width="52.87" height="30" as="geometry" />
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-2" value="" style="sketch=0;pointerEvents=1;shadow=0;dashed=0;html=1;strokeColor=none;fillColor=#505050;labelPosition=center;verticalLabelPosition=bottom;verticalAlign=top;outlineConnect=0;align=center;shape=mxgraph.office.devices.nic;" vertex="1" parent="O6jyKfrmbOcH9zmBw9ze-4">
<mxGeometry x="80" width="52.87" height="30" as="geometry" />
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-3" value="&lt;font style=&quot;font-size: 18px;&quot;&gt;&lt;b&gt;NICs&lt;/b&gt;&lt;/font&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="O6jyKfrmbOcH9zmBw9ze-4">
<mxGeometry x="39.879999999999995" y="34" width="60" height="30" as="geometry" />
</mxCell>
<mxCell id="IBzB3Yzv9ICFWQqCQZ6M-1" value="&lt;b&gt;&lt;font style=&quot;font-size: 14px;&quot;&gt;DPDK PMD&lt;/font&gt;&lt;/b&gt;" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;" vertex="1" parent="1">
<mxGeometry x="501.88" y="569" width="150" height="34" as="geometry" />
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-10" value="&lt;font style=&quot;font-size: 18px;&quot;&gt;socket&lt;/font&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
<mxGeometry x="186" y="617" width="60" height="30" as="geometry" />
</mxCell>
<mxCell id="O6jyKfrmbOcH9zmBw9ze-13" value="&lt;b&gt;&lt;font style=&quot;font-size: 10px;&quot; color=&quot;#ffffff&quot;&gt;rte_ring or&amp;nbsp;&lt;/font&gt;&lt;span style=&quot;font-size: 10px; background-color: initial; color: rgb(255, 255, 255);&quot;&gt;vhost_user&lt;/span&gt;&lt;/b&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;rotation=90;" vertex="1" parent="1">
<mxGeometry x="439.5" y="569.5" width="82.24" height="11" as="geometry" />
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
Loading

0 comments on commit e3ef22a

Please sign in to comment.