Pktgen build error Ubuntu 22.04

[tpapastergiou3]

Hello,

I have installed the latest DPDK release (24.03.0) and I can build it without any issues.
I'm using Ubuntu 22.04.
However, when trying to install the latest Pktgen version (24.05.3) I get the following:

make
>>> Use 'make help' for more commands

./tools/pktgen-build.sh build
>>  SDK Path          : /data/test
>>  Install Path      : /data/test/Pktgen-DPDK
>>  Build Directory   : /data/test/Pktgen-DPDK/builddir
>>  Target Directory  : usr/local
>>  Build Path        : /data/test/Pktgen-DPDK/builddir
>>  Target Path       : /data/test/Pktgen-DPDK/usr/local

 Build and install values:
   lua_enabled       : -Denable_lua=false

>>> Ninja build in '/data/test/Pktgen-DPDK/builddir' buildtype=release
meson configure -Dbuildtype=release -Denable_lua=false /data/test/Pktgen-DPDK/builddir
ninja: Entering directory `/data/test/Pktgen-DPDK/builddir'
[0/1] Regenerating build files.
The Meson build system
Version: 0.61.2
Source dir: /data/test/Pktgen-DPDK
Build dir: /data/test/Pktgen-DPDK/builddir
Build type: native build
Program cat found: YES (/usr/bin/cat)
Project name: pktgen
Project version: 24.05.3
C compiler for the host machine: ccache cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -mavx: YES (cached)
Compiler for C supports arguments -mavx2: YES (cached)
Compiler for C supports arguments -Wno-pedantic: YES (cached)
Compiler for C supports arguments -Wno-format-truncation: YES (cached)
Found pkg-config: /usr/bin/pkg-config (0.29.2)
Found CMake: /usr/bin/cmake (3.22.1)
Run-time dependency libfgen found: NO (tried pkgconfig and cmake)
Dependency libdpdk found: YES 24.03.0 (cached)
Message: prefix: /usr/local libdir: lib/x86_64-linux-gnu
Message: DPDK lib path: /usr/local/lib/x86_64-linux-gnu
Library rte_net_bond found: YES
Program python3 found: YES (/usr/bin/python3)
WARNING: Python files installed by Meson might not be found by python interpreter.
 This warning can be avoided by setting "python.platlibdir" option.
WARNING: Python files installed by Meson might not be found by python interpreter.
 This warning can be avoided by setting "python.purelibdir" option.
Library rte_net_i40e found: YES
Library rte_net_ixgbe found: YES
Library rte_net_ice found: YES
Library rte_bus_vdev found: YES
Dependency threads found: YES unknown (cached)
Dependency numa found: YES 2.0.14 (cached)
Dependency pcap found: YES 1.10.1 (cached)
Library dl found: YES
Library m found: YES
Program doxygen found: NO
Program sphinx-build found: NO
Build targets in project: 9
NOTICE: Future-deprecated features used:
 * 0.56.0: {'Dependency.get_pkgconfig_variable'}

pktgen 24.05.3

  User defined options
    backend   : ninja
    buildtype : release
    enable_lua: false

Found ninja-1.10.1 at /usr/bin/ninja
Cleaning... 0 files.
[1/4] Compiling C object lib/hmap/libhmap.a.p/hmap.c.o
FAILED: lib/hmap/libhmap.a.p/hmap.c.o
ccache cc -Ilib/hmap/libhmap.a.p -Ilib/hmap -I../lib/hmap -I/usr/local/include -I/usr/include/libnl3 -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -O3 -march=native -mavx -mavx2 -DALLOW_EXPERIMENTAL_API -D_GNU_SOURCE -Wno-pedantic -Wno-format-truncation -fPIC -include rte_config.h -march=native -mrtm -MD -MQ lib/hmap/libhmap.a.p/hmap.c.o -MF lib/hmap/libhmap.a.p/hmap.c.o.d -o lib/hmap/libhmap.a.p/hmap.c.o -c ../lib/hmap/hmap.c
../lib/hmap/hmap.c:5:10: fatal error: bsd/sys/queue.h: No such file or directory
    5 | #include <bsd/sys/queue.h>        // for TAILQ_FOREACH_SAFE
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
[2/4] Compiling C object app/pktgen.p/pktgen-port-cfg.c.o
FAILED: app/pktgen.p/pktgen-port-cfg.c.o
ccache cc -Iapp/pktgen.p -Iapp -I../app -Ilib/common -I../lib/common -Ilib/utils -I../lib/utils -Ilib/vec -I../lib/vec -Ilib/plugin -I../lib/plugin -Ilib/cli -I../lib/cli -Ilib/lua -I../lib/lua -Ilib/hmap -I../lib/hmap -I/usr/local/include -I/usr/include/libnl3 -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -Werror -O3 -march=native -mavx -mavx2 -DALLOW_EXPERIMENTAL_API -D_GNU_SOURCE -Wno-pedantic -Wno-format-truncation -pthread -include rte_config.h -march=native -mrtm '-D__PROJECT_VERSION="24.05.3"' -MD -MQ app/pktgen.p/pktgen-port-cfg.c.o -MF app/pktgen.p/pktgen-port-cfg.c.o.d -o app/pktgen.p/pktgen-port-cfg.c.o -c ../app/pktgen-port-cfg.c
In file included from /usr/local/include/rte_bitops.h:20,
                 from /usr/local/include/rte_cman.h:12,
                 from /usr/local/include/rte_ethdev.h:163,
                 from ../lib/common/pg_compat.h:15,
                 from ../lib/common/pg_delay.h:14,
                 from ../app/pktgen-port-cfg.c:9:
../app/pktgen-port-cfg.c: In function ‘pktgen_config_ports’:
../app/pktgen-port-cfg.c:168:34: error: format ‘%ld’ expects argument of type ‘long int’, but argument 3 has type ‘int’ [-Werror=format=]
  168 |                 pktgen_log_panic("Unable to allocate %'ld pkt_seq_t headers", NUM_TOTAL_PKTS);
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  ~~~~~~~~~~~~~~
      |                                                                               |
      |                                                                               int
../app/pktgen-port-cfg.c:168:17: note: in expansion of macro ‘pktgen_log_panic’
  168 |                 pktgen_log_panic("Unable to allocate %'ld pkt_seq_t headers", NUM_TOTAL_PKTS);
      |                 ^~~~~~~~~~~~~~~~
../app/pktgen-port-cfg.c:168:57: note: format string is defined here
  168 |                 pktgen_log_panic("Unable to allocate %'ld pkt_seq_t headers", NUM_TOTAL_PKTS);
      |                                                      ~~~^
      |                                                         |
      |                                                         long int
      |                                                      %'d
cc1: all warnings being treated as errors
ninja: build stopped: subcommand failed.
make: *** [Makefile:15: build] Error 1

Any pointers are greatly appreciated

[joygetsit]

You may not have the /bsd/ directory and its header files in your system. Install it in your linux kernel using libbsd-dev package.

[KeithWiles]

You can install the package libbsd-dev using sudo apt-get install libbsd-dev looks like I need to update the docs.

[tpapastergiou3]

I did a clean OS reinstall (ubuntu 22.04 server) and reinstalled all the packages that are listed as dependencies to install dpdk/pktgen. Other steps:

add vm.nr_hugepages=2048 in /etc/sysctl.conf
add huge /mnt/huge hugetlbfs defaults 0 0 in /etc/fstab
reboot

# cat /proc/meminfo | grep -i "huge"
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         4194304 kB

• Installed mellanox drivers using ./mlnxofedinstall --add-kernel-support --upstream-libs --dpdk --skip-repo
• enabled iommu

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-107-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro iommu=pt intel_iommu=on

• To install dpdk I did:

wget https://fast.dpdk.org/rel/dpdk-24.03.tar.xz
tar xJf dpdk-24.03.tar.xz
cd dpdk-24.03

export RTE_SDK=$PWD
export RTE_TARGET=x86_64-native-linuxapp-gcc

meson setup build
cd build
ninja
meson install
Ldconfig

• To install pktgen I did:

wget https://github.com/pktgen/Pktgen-DPDK/archive/refs/tags/pktgen-24.03.0.tar.gz
tar -xvzf pktgen-24.03.0.tar.gz
cd Pktgen-DPDK-pktgen-24.03.0
make

Everything compiled without issue. When I'm running pktgen though I get:

# ./usr/local/bin/pktgen -l 1-10 -- -T -P -m "[2-10:2-10].2"

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 96
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:98:00.0 (socket 1)
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:98:00.1 (socket 1)
EAL: Using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(1)
EAL: Ignore mapping IO port bar(4)
EAL: Probe PCI driver: net_i40e (8086:1583) device: 0000:99:00.0 (socket 1)
TELEMETRY: No legacy callbacks, legacy socket not created
    0: mlx5_pci        9      1   15b3:101d/0000:98:00.0
    1: mlx5_pci       10      1   15b3:101d/0000:98:00.1
    2: net_i40e        0      1   8086:1583/0000:99:00.0

Initialize Port 2 -- RxQ 9, TxQ 9
!PANIC!: Cannot create mbuf pool (Default RX  2:1) port 2, queue 1, nb_mbufs 24576, socket_id 0: Cannot allocate memory
EAL: PANIC in pktgen_mbuf_pool_create():
Cannot create mbuf pool (Default RX  2:1) port 2, queue 1, nb_mbufs 24576, socket_id 0: Cannot allocate memory
0: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (rte_dump_stack+0x42) [7fe1357463f2]
1: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (__rte_panic+0xd4) [7fe13571e2bf]
2: ./usr/local/bin/pktgen (557ed3169000+0x35649) [557ed319e649]
3: ./usr/local/bin/pktgen (557ed3169000+0xb59a) [557ed317459a]
4: /lib/x86_64-linux-gnu/libc.so.6 (7fe135386000+0x29d90) [7fe1353afd90]
5: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0x80) [7fe1353afe40]
6: ./usr/local/bin/pktgen (557ed3169000+0xbb05) [557ed3174b05]
Aborted (core dumped)

I get the same errors no matter which port I use.
Also, it appears that files are created under /dev/hugepages (probably before the program crashes) and I delete them manually since they do not appear to be freed automatically.
Any ideas on how to debug this?

[KeithWiles]

Do you have a reason to use DPDK 23.04 and Pktgen 23.04 as I have fixed a number of problems in Pktgen 24.05.4, which should work with DPDK latest version 24.07.0-rc0. The latest version of DPDK does not compile on Ubuntu 24.04 LTS as the compile has found an issue with ./lib/pcapng/rte_pcapng.c file and memcpy. I have a local fix for that problem using pragma to disable the error.

If you can please update to the latest Pktgen version.

To rebuild Pktgen you can use make or make rebuild instead of the meson/ninja commands.

Add the -v option to the pktgen command line. I assume you are running as root here correct?

Rebuild Pktgen with debug make clean debug will give more information but will be slower.

Try using -m 2.2 instead of [2-10:2-20].2 which could have been simpler [m "[2-10].2" does the same thing, but try with only one lcore.

You don't need to remove hugepage alone unless you need to free the hugepages.

You are using 9 lcores for both rx/tx on port 2 in the example above and the port is on socket 1. The message Cannot create mbuf pool (Default RX 2:1) port 2, queue 1, nb_mbufs 24576, socket_id 0 state socket 0, which is wrong.