E810-C 100G网卡中间加入被测设备后流量上不去 #513

gyl30 · 2024-12-20T07:19:35Z

gyl30
Dec 20, 2024

dperf 非常好用，之前经常用来测试各种网卡的速度，这次测试 100G 网卡时遇到了一点问题，请教一下。
我现在有 3 台机器，每台机器都有 2 张 100G 的网卡。三台机器两两之间使用 dperf 互相打流都可以跑满线速。
但是如果在 dperf client 和 dperf server 两台机器之间加一台机器跑 dpdk-testpmd 用来转发流量，那么带宽仅有50G 左
右,想请教下为什么会出现这种现象？哪里可能出现问题呢？拓扑和配置如下

dperf 使用的提交是 fcf5035
dpdk 版本 22.11
系统环境是 ubuntu22.04
内核版本是 6.8.0-49-generic
系统 /etc/default/grub 设置如下

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt isolcpus=1-21 nohz_full=1-21 rcu_nocbs=1-21 idle=poll nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=0 intel_idle.max_cstate=0 hpet=disable mce=off tsc=reliable numa_balancing=disable"

大页配置和网卡绑定参数 3 台机器都一样，如下

大页配置

root@seclead-Super-Server:~/dep/bin# ./dpdk-hugepages.py -s
Node Pages Size Total
0    16    1Gb    16Gb

Hugepages mounted on /dev/hugepages

网卡绑定信息

root@seclead-Super-Server:~/dep/bin# ./dpdk-devbind.py --status-dev net

Network devices using DPDK-compatible driver
============================================
0000:65:00.0 'Ethernet Controller E810-C for QSFP 1592' drv=vfio-pci unused=ice
0000:65:00.1 'Ethernet Controller E810-C for QSFP 1592' drv=vfio-pci unused=ice

Network devices using kernel driver
===================================
0000:17:00.0 'Ethernet Controller XL710 for 40GbE QSFP+ 1583' if=enp23s0f0np0 drv=i40e unused=vfio-pci 
0000:17:00.1 'Ethernet Controller XL710 for 40GbE QSFP+ 1583' if=enp23s0f1np1 drv=i40e unused=vfio-pci *Active*
0000:18:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp24s0f0 drv=ixgbe unused=vfio-pci *Active*
0000:18:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp24s0f1 drv=ixgbe unused=vfio-pci *Active*
0000:b5:00.0 'Ethernet Connection X722 for 1GbE 37d1' if=eno1np0 drv=i40e unused=vfio-pci 
0000:b5:00.1 'Ethernet Connection X722 for 1GbE 37d1' if=eno2np1 drv=i40e unused=vfio-pci *Active*
0000:b6:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp182s0f0 drv=ixgbe unused=vfio-pci *Active*
0000:b6:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp182s0f1 drv=ixgbe unused=vfio-pci *Active*

客户端配置

root@seclead-Super-Server:~/dperf# cat /root/dperf/build/client-dperf-udp-100.conf
#daemon
mode            client
protocol        udp
tx_burst        128
launch_num      10
cpu             0
rss             auto

duration        360m
payload_size    1400
#jumbo
#packet_size     9000
keepalive       1ms
cps             900
cc              8400



#numa2
port            0000:65:00.1    192.168.116.28   192.168.116.45      d0:9f:d9:70:4f:10
#               addr_start      num
client          192.168.116.28     1

#               addr_start      num
server          192.168.116.45    1

#               port_start      num
listen          80              1

服务端配置

#daemon
mode            server
protocol        udp
tx_burst        128
cpu             0
rss             auto
duration        360m
payload_size    1400
#jumbo
#packet_size     9000
keepalive       1ms

#numa2
port            0000:65:00.0    192.168.116.45   192.168.116.28   d0:9f:d9:70:55:b9

#               addr_start      num
client         192.168.116.28     1

#               addr_start      num
server         192.168.116.45     1

#               port_start      num
listen          80              1

可以跑满限速的拓扑

加入 dpdk-testpmd 仅有 50G 左右的拓扑

dpdk-testpmd 启动参数

dpdk-testpmd -l 0-7 -n 4 --proc-type=primary --socket-mem=2048 --file-prefix=dpdk -- --nb-ports=2 --nb-cores=7 --rxq=8 --txq=8 --forward-mode=mac --portmask
=0x3

尝试修改过 dpdk-testpmd 队列和线程数，测试结果没有变化

dpdk-testpmd 的输出

root@seclead-Super-Server: dpdk-testpmd -l 0-7 -n 4 --proc-type=primary --socket-mem=2048 --file-prefix=dpdk -- --nb-ports=2 --nb-cores=7 --rxq=8 --txq=8 --forward-mode=mac --portmask
=0x3                                                                                                                                                                                               
EAL: Detected CPU lcores: 32                                                                                                                                                                       
EAL: Detected NUMA nodes: 1                                                                                                                                                                        
EAL: Detected static linkage of DPDK                                                                                                                                                               
EAL: Multi-process socket /var/run/dpdk/dpdk/mp_socket                                                                                                                                             
EAL: Selected IOVA mode 'VA'                                                                                                                                                                       
EAL: No available 2048 kB hugepages reported                                                                                                                                                       
EAL: VFIO support initialized                                                                                                                                                                      
EAL: Using IOMMU type 1 (Type 1)                                                                                                                                                                   
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:65:00.0 (socket 0)                                                                                                                         
ice_load_pkg_type(): Active package is: 1.3.26.0, ICE OS Default Package (double VLAN mode)                                                                                                        
EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:65:00.1 (socket 0)                                                                                                                         
ice_load_pkg_type(): Active package is: 1.3.26.0, ICE OS Default Package (double VLAN mode)                                                                                                        
TELEMETRY: No legacy callbacks, legacy socket not created                                                                                                                                          
Set mac packet forwarding mode                                                                                                                                                                     
testpmd: create a new mbuf pool <mb_pool_0>: n=203456, size=2176, socket=0                                                                                                                         
testpmd: preferred mempool ops selected: ring_mp_mc                                                                                                                                                
Configuring Port 0 (socket 0)                                                                                                                                                                      
ice_set_rx_function(): Using AVX2 OFFLOAD Vector Rx (port 0).                                                                                                                                      
Port 0: D0:9F:D9:70:51:F0                                                                                                                                                                          
Configuring Port 1 (socket 0)                                                                                                                                                                      
ice_set_rx_function(): Using AVX2 OFFLOAD Vector Rx (port 1).                                                                                                                                      
                                                                                                                                                                                                   
Port 0: link state change event                                                                                                                                                                    
Port 1: D0:9F:D9:70:51:F1                                                                                                                                                                          
Checking link statuses...                                                                                                                                                                          
Done                                                                                                                                                                                               
No commandline core given, start packet forwarding                                                                                                                                                 
mac packet forwarding - ports=2 - cores=7 - streams=16 - NUMA support enabled, MP allocation mode: native                                                                                          
Logical Core 1 (socket 0) forwards packets on 2 streams:                                                                                   Logical Core 1 (socket 0) forwards packets on 2 streams:                                                                                                                                           
  RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01                                                                                                                            
  RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00                                                                                                                            
Logical Core 2 (socket 0) forwards packets on 2 streams:                                                                                                                                           
  RX P=0/Q=1 (socket 0) -> TX P=1/Q=1 (socket 0) peer=02:00:00:00:00:01                                                                                                                            
  RX P=1/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00
Logical Core 3 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=2 (socket 0) -> TX P=1/Q=2 (socket 0) peer=02:00:00:00:00:01
  RX P=1/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00
Logical Core 4 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=3 (socket 0) -> TX P=1/Q=3 (socket 0) peer=02:00:00:00:00:01
  RX P=1/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00
Logical Core 5 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=4 (socket 0) -> TX P=1/Q=4 (socket 0) peer=02:00:00:00:00:01
  RX P=1/Q=4 (socket 0) -> TX P=0/Q=4 (socket 0) peer=02:00:00:00:00:00
Logical Core 6 (socket 0) forwards packets on 3 streams:
  RX P=0/Q=5 (socket 0) -> TX P=1/Q=5 (socket 0) peer=02:00:00:00:00:01
  RX P=1/Q=5 (socket 0) -> TX P=0/Q=5 (socket 0) peer=02:00:00:00:00:00
  RX P=0/Q=6 (socket 0) -> TX P=1/Q=6 (socket 0) peer=02:00:00:00:00:01
Logical Core 7 (socket 0) forwards packets on 3 streams:
  RX P=1/Q=6 (socket 0) -> TX P=0/Q=6 (socket 0) peer=02:00:00:00:00:00
  RX P=0/Q=7 (socket 0) -> TX P=1/Q=7 (socket 0) peer=02:00:00:00:00:01
  RX P=1/Q=7 (socket 0) -> TX P=0/Q=7 (socket 0) peer=02:00:00:00:00:00

  mac packet forwarding packets/burst=32
  nb forwarding cores=7 - nb forwarding ports=2
  port 0: RX queue number: 8 Tx queue number: 8
    Rx offloads=0x80000 Tx offloads=0x10000
    RX queue: 0
      RX desc=1024 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=1024 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32
  port 1: RX queue number: 8 Tx queue number: 8
    Rx offloads=0x80000 Tx offloads=0x10000
    RX queue: 0
      RX desc=1024 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=1024 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32
Press enter to exit

Port 1: link state change event
^C
Signal 2 received, preparing to exit...
Telling cores to stop...
Waiting for lcores to finish...

  ------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 1/Queue= 3 -------
  RX-packets: 78340841       TX-packets: 78340841       TX-dropped: 0             

  ------- Forward Stats for RX Port= 1/Queue= 3 -> TX Port= 0/Queue= 3 -------
  RX-packets: 78340841       TX-packets: 78340841       TX-dropped: 0             

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 78340841       RX-dropped: 0             RX-total: 78340841
  TX-packets: 78340841       TX-dropped: 0             TX-total: 78340841
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 78340841       RX-dropped: 57056201      RX-total: 135397042
  TX-packets: 78340841       TX-dropped: 0             TX-total: 78340841
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 156681682      RX-dropped: 57056201      RX-total: 213737883
  TX-packets: 156681682      TX-dropped: 0             TX-total: 156681682
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Stopping port 0...
Stopping ports...
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
Port 1 is closed
Done

Bye...

dperf client 的输出

dperf server 的输出

pengjianzhang · 2024-12-20T07:29:59Z

pengjianzhang
Dec 20, 2024
Maintainer

物理网卡不需要用rss
多用几个CPU试试
用最新的代码
tx-burst可以少点，testpmd扛不住，有很多丢包啊；testpmd多开一些CPU、队列

如果喜欢dperf，加个star吧，谢谢

0 replies

gyl30 · 2024-12-20T07:36:19Z

gyl30
Dec 20, 2024
Author

我注意到testpmd 使用的 burst 是 32（mac packet forwarding packets/burst=32），testpmd 使用15个核心和 16个队列，我将 dperf client 的 burs 设置了32，然后删除 dperf client 中的 rss auto 配置后，依然没有效果

dpdk-testpmd -l 0-15 -n 4 --proc-type=primary --socket-mem=2048 --file-prefix=dpdk -- --nb-ports=2 --nb-cores=15 --rxq=16 --txq=16 --forward-mode=mac --portmask=0x3

输出如下

  mac packet forwarding packets/burst=32
  nb forwarding cores=15 - nb forwarding ports=2
  port 0: RX queue number: 16 Tx queue number: 16
    Rx offloads=0x80000 Tx offloads=0x10000
    RX queue: 0
      RX desc=1024 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=1024 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32
  port 1: RX queue number: 16 Tx queue number: 16
    Rx offloads=0x80000 Tx offloads=0x10000
    RX queue: 0
      RX desc=1024 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=1024 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32


Press enter to exit                                                                                                                                                                       [20/1932]
                                                                                                                                                                                                   
Port 0: link state change event                                                                                                                                                                    
                                                                                                                                                                                                   
Port 1: link state change event                                                                                                                                                                    
^C                                                                                                                                                                                                 
Signal 2 received, preparing to exit...                                                                                                                                                            
Telling cores to stop...                                                                                                                                                                           
Waiting for lcores to finish...                                                                                                                                                                    
                                                                                                                                                                                                   
  ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 1/Queue= 0 -------                                                                                                                     
  RX-packets: 2              TX-packets: 2              TX-dropped: 0                                                                                                                              
                                                                                                                                                                                                   
  ------- Forward Stats for RX Port= 1/Queue= 0 -> TX Port= 0/Queue= 0 -------                                                                                                                     
  RX-packets: 2              TX-packets: 2              TX-dropped: 0                                                                                                                              
                                                                                                                                                                                                   
  ------- Forward Stats for RX Port= 0/Queue=10 -> TX Port= 1/Queue=10 -------                                                                                                                     
  RX-packets: 409746656      TX-packets: 409746656      TX-dropped: 0                                                                                                                              
                                                                                                                                                                                                   
  ------- Forward Stats for RX Port= 1/Queue=14 -> TX Port= 0/Queue=14 -------                                                                                                                     
  RX-packets: 409746947      TX-packets: 409746947      TX-dropped: 0                                                                                                                              

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 409746943      RX-dropped: 6             RX-total: 409746949
  TX-packets: 409746949      TX-dropped: 0             TX-total: 409746949
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 409747936      RX-dropped: 356938197     RX-total: 766686133
  TX-packets: 409746658      TX-dropped: 0             TX-total: 409746658
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 819494879      RX-dropped: 356938203     RX-total: 1176433082
  TX-packets: 819493607      TX-dropped: 0             TX-total: 819493607
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

0 replies

gyl30 · 2024-12-20T07:46:52Z

gyl30
Dec 20, 2024
Author

这是 dperf client 的输出，不管怎么调参数 bitsRx 总是到 49,652,720,544 左右，感觉像是哪里设置不对

0 replies

pengjianzhang · 2024-12-20T07:49:44Z

pengjianzhang
Dec 20, 2024
Maintainer

用多个CPU
达到极限了，出现了发送丢包

0 replies

gyl30 · 2024-12-20T07:50:09Z

gyl30
Dec 20, 2024
Author

忘记 CPU 配置了 Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz，单 socket ，32 核心

2 replies

pengjianzhang Dec 20, 2024
Maintainer

dperf用4个CPU试试
cpu 0 1 2 3

gyl30 Dec 20, 2024
Author

配置了多 CPU 反而流量上不去了

gyl30 · 2024-12-20T07:52:46Z

gyl30
Dec 20, 2024
Author

这个极限是 dperf client 还是 testpmd呢？ bitsTx 有 96,900,784,960，比 bitsRx 多了一倍左右，dperf client 的发送应该不是瓶颈吧

1 reply

pengjianzhang Dec 20, 2024
Maintainer

发送应该可以打到100Gbps
配4个CPU试试

是否达到极限看是否出现丢包，都出现了丢包，都达到极限了

gyl30 · 2024-12-20T07:54:50Z

gyl30
Dec 20, 2024
Author

从这个 testpmd 的输出来看，是 port 1 接收的时候被丢弃了

  ------- Forward Stats for RX Port= 0/Queue= 1 -> TX Port= 1/Queue= 1 -------                                                                                                                     
  RX-packets: 185078625      TX-packets: 185078625      TX-dropped: 0                                                                                                                              
                                                                                                                                                                                                   
  ------- Forward Stats for RX Port= 1/Queue= 7 -> TX Port= 0/Queue= 7 -------                                                                                                                     
  RX-packets: 185078625      TX-packets: 185078625      TX-dropped: 0                                                                                                                              
                                                                                                                                                                                                   
  ---------------------- Forward statistics for port 0  ----------------------                                                                                                                     
  RX-packets: 185078625      RX-dropped: 0             RX-total: 185078625                                                                                                                         
  TX-packets: 185078625      TX-dropped: 0             TX-total: 185078625                                                                                                                         
  ----------------------------------------------------------------------------                                                                                                                     
                                                                                                                                                                                                   
  ---------------------- Forward statistics for port 1  ----------------------                                                                                                                     
  RX-packets: 185078625      RX-dropped: 160281267     RX-total: 345359892                                                                                                                         
  TX-packets: 185078625      TX-dropped: 0             TX-total: 185078625                                                                                                                         
  ----------------------------------------------------------------------------                                                                                                                     
                                                                                                                                                                                                   
  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++                                                                                                                      
  RX-packets: 370157250      RX-dropped: 160281267     RX-total: 530438517                                                                                                                         
  TX-packets: 370157250      TX-dropped: 0             TX-total: 370157250                                                                                                                         
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

0 replies

gyl30 · 2024-12-20T08:00:59Z

gyl30
Dec 20, 2024
Author

dperf client 的配置改成如下了

#daemon
mode            client
protocol        udp
tx_burst        32
launch_num      10
cpu             0-3
rss             auto

duration        360m
payload_size    1400
#jumbo
#packet_size     9000
keepalive       1ms
cps             900
cc              8400



#numa2
port            0000:65:00.1    192.168.116.28   192.168.116.45      d0:9f:d9:70:4f:10
#               addr_start      num
client          192.168.116.28     1

#               addr_start      num
server          192.168.116.45    1

#               port_start      num
listen          80              1

配置了多 cpu 后，没有 rss 报错了，Error: 'rss' is required if cpu num is not equal to server ip num，所以我添加了 rss 配置项。
现在跑了一段时间，流量一直上不去，看占用，除了 CPU 0 其他 CPU 也没用上

7 replies

pengjianzhang Dec 20, 2024
Maintainer

瓶颈在testpmd，testpmd丢包了

gyl30 Dec 20, 2024
Author

dperf 和 testpmd 都是使用 dpdk，dperf 可以发出更高的流量，有哪些优化吗？我尝试排查一下

pengjianzhang Dec 20, 2024
Maintainer

参考 https://dperf.org/
请在周围小伙伴之间宣传一下，加个star，谢谢~

gyl30 Dec 20, 2024
Author

好的，多谢，已 start

pengjianzhang Dec 20, 2024
Maintainer

很多用户反馈 dperf性能的确比某些工具性能高

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E810-C 100G网卡中间加入被测设备后流量上不去 #513

{{title}}

Replies: 8 comments 10 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

E810-C 100G网卡中间加入被测设备后流量上不去 #513

gyl30 Dec 20, 2024

Replies: 8 comments · 10 replies

pengjianzhang Dec 20, 2024 Maintainer

gyl30 Dec 20, 2024 Author

gyl30 Dec 20, 2024 Author

pengjianzhang Dec 20, 2024 Maintainer

gyl30 Dec 20, 2024 Author

pengjianzhang Dec 20, 2024 Maintainer

gyl30 Dec 20, 2024 Author

gyl30 Dec 20, 2024 Author

pengjianzhang Dec 20, 2024 Maintainer

gyl30 Dec 20, 2024 Author

gyl30 Dec 20, 2024 Author

pengjianzhang Dec 20, 2024 Maintainer

gyl30 Dec 20, 2024 Author

pengjianzhang Dec 20, 2024 Maintainer

gyl30 Dec 20, 2024 Author

pengjianzhang Dec 20, 2024 Maintainer

gyl30
Dec 20, 2024

Replies: 8 comments 10 replies

pengjianzhang
Dec 20, 2024
Maintainer

gyl30
Dec 20, 2024
Author

gyl30
Dec 20, 2024
Author

pengjianzhang
Dec 20, 2024
Maintainer

gyl30
Dec 20, 2024
Author

pengjianzhang Dec 20, 2024
Maintainer

gyl30 Dec 20, 2024
Author

gyl30
Dec 20, 2024
Author

pengjianzhang Dec 20, 2024
Maintainer

gyl30
Dec 20, 2024
Author

gyl30
Dec 20, 2024
Author

pengjianzhang Dec 20, 2024
Maintainer

gyl30 Dec 20, 2024
Author

pengjianzhang Dec 20, 2024
Maintainer

gyl30 Dec 20, 2024
Author

pengjianzhang Dec 20, 2024
Maintainer