Skip to content

Tips for Debugging

Jump to bottom
Michel Machado edited this page May 19, 2021 · 25 revisions

This page gives some tips on debugging Gatekeeper using pktgen and gdb

Table of Contents

Generating Packets on DPDK Interfaces

This section describes how to install and use the DPDK pktgen program to generate packets. We provide an example based on our "XIA" server setup.

Server Physical Setup

The XIA server has four DPDK-enabled ports that can be used to send and receive packets. The diagram below shows the four ports (along with their PCI addresses): 

    +-----------------------------------------+
    |  -----               -----              |
    |  | o | A (83:00.0)   | o | B (83:00.1)  |
    |  --|--               --|--              |
    |    |                   |                |
    |  --|--               --|--              |
    |  | o | C (85:00.0)   | o | D (85:00.1)  |
    |  -----               -----              |
    +-----------------------------------------+

Note that ports A and C are connected, and ports B and D are connected. The MAC addresses of the four ports are A (e8:ea:6a:06:1f:7c), B (e8:ea:6a:06:1f:7d), C (e8:ea:6a:06:21:b2), and D (e8:ea:6a:06:21:b3). The port MAC addresses may be used for generating specific packets in the following sections.

When you run multiple DPDK applications at the same time (such as two instances of pktgen, or pktgen and another DPDK application), you need to blacklist the PCI devices, or ports in this case, that you don't want to use. Depending on how you blacklist the devices, the port numbers that are assigned by DPDK will change.

For example, if you blacklist ports A, B, and C in one instance of pktgen, then port D is known as port 0 in that instance. If, at the same time, you blacklist ports A, C, and D in another instance of pktgen, then in that instance port B is known as port 0. To avoid ambiguity, in this document we will refer to the ports using these letter designations (A, B, C, D) instead of by numbers.

Setting up pktgen

First, obtain the pktgen source code and build it: 

    $ git clone http://dpdk.org/git/apps/pktgen-dpdk
    $ cd pktgen-dpdk
    $ make

Once it compiles, do a setup step: 

    $ sudo -E ./setup.sh

The application should be ready to run at this point, but you may also encounter the need to perform the following step (still in the pktgen-dpdk directory): 

    $ cp Pktgen.lua app/app/x86_64-native-linuxapp-gcc/

Running Two Instances of pktgen

This demonstration will show how we can generate packets on port A and send them to port C. To begin, open two terminals.

In the first terminal, go to the dpdk-pktgen directory and run: 

    $ cd app/app/x86_64-native-linuxapp-gcc/
    $ sudo ./pktgen -l 0-2 --socket-mem 256 --file-prefix pg1 -b 83:00.1 -b 85:00.0 -b 85:00.1 -- -T -P -m "[1:2].0"

The "-c 7" option specifies which lcores are available for use, which is lcores 0, 1, and 2. The "-m [1:2].0" part specifies that lcores 1 and 2 will handle rx/tx on port 0, and lcore 0 is automatically assigned to the pktgen program for displaying statistics.

The "--socket-mem 256" option puts a limit on the memory used, which is often needed when multiple DPDK applications are run at the same time. The "--file-prefix pg1" option specifies a special file prefix to use for this DPDK application's meta information, which again is needed if there are multiple DPDK applications running (which we will have in the next part).

The "-b" options blacklist different ports -- in other words, excludes those ports from being used in the application. In this command, we only want to use port A (at PCI location 83:00.0), so we blacklist the other three ports: B (83:00.1), C (85:00.0), and D (85:00.1).

The "-T" option gives the display statistics some color and the "-P" makes all ports run in promiscuous mode.

In the second terminal, run the same command, but with different lcores and blacklisted ports: 

    $ cd app/app/x86_64-native-linuxapp-gcc/
    $ sudo ./pktgen -c 70 --socket-mem 256 --file-prefix pg2 -b 83:00.0 -b 83:00.1 -b 85:00.1 -- -T -P -m "[5:6].0"

This does the same as the first command, but instead allows lcores 4, 5, and 6 to be used, uses a different file prefix, and blacklists all ports except for port C.

More information about the command-line parameters is here.

Once the applications are running, go to the terminal running pktgen on port A. You can start packets flowing using: 

    $ Pktgen> start 0

Remember that port numbering always starts from 0 within an application, so since we blacklisted all other ports, port 0 means port A.

When you do this, you should see the second terminal's statistics being updated. You could also start packets flowing on the second terminal (port C) and see the packets being received on port A -- on the second terminal, port C will also be called port 0, since that is the only active port in that application.

You can stop packets flowing with: 

    $ Pktgen> stop 0

And quit with: 

    $ Pktgen> quit

Debugging Gatekeeper

Since Gatekeeper is compiled with the -g option, gdb can be directly used with Gatekeeper. After it is compiled, you can run: 

    $ sudo gdb ./build/gatekeeper

Note that, to have more debugging information, it's better to compile Gatekeeper without the -O3 option.

From there, you can use gdb as described in its documentation. You will need to run Gatekeeper without hugepages, so once gdb starts, you can run: 

    (gdb) run

Compiling DPDK with debuf information

Sometimes one has to investigate the code of DPDK with gdb as well, and having DPDK compiled with debug information included is very helpful in these cases. To compile the DPDK and sample applications with debugging information included and the optimization level set to 0, the EXTRA_CFLAGS environment variable should be set before compiling as follows: 

    export EXTRA_CFLAGS='-O0 -g'

Once the variable EXTRA_CFLAGS is exported, just call make as you would to compile DPDK.

Disassembling BPF programs

Flow entries may have BPF programs associated with them, and one may need to verify that these programs were correctly compiled. To do so, one can disassemble a BPF program as follows: 

 $ llvm-objdump -S -no-show-raw-insn -section=init declined.bpf
 
 declined.bpf:	file format ELF64-BPF
 
 Disassembly of section init:
 declined_init:
       0:	r0 = 0
       1:	exit

The option -no-show-raw-insn suppresses the hexadecimal representation of the instructions, which is only useful if one is debugging the BPF VM itself.

Gatekeeper BPF programs have two sections: "init" and "pkt". Section "init" is used to initialize a BPF state, whereas the section "pkt" is used to process a packet associate with a flow.

Generating a Debian package for debugging

Building a debug-enabled Debian package for Gatekeeper can be useful because it eases the uninstallation procedure, which must be done manually when performing the the build process with the setup.sh script.

The following procedure assumes one is at the root of a recursively-cloned gatekeeper repository, i.e. git clone --recursive http://github.com/AltraMayor/gatekeeper.git followed by cd gatekeeper.

1) Edit the Makefile to disable compiler optimizations and enable debug information: 

-EXTRA_CFLAGS += -O3 -g -Wfatal-errors -DALLOW_EXPERIMENTAL_API \
+EXTRA_CFLAGS += -O0 -ggdb -Wfatal-errors -DALLOW_EXPERIMENTAL_API \

2) Edit the debian/rules file and add the EXTRA_CFLAGS to do the same when building Gatekeeper: 

-               make config T="$(RTE_TARGET)"; \
+               make config T="$(RTE_TARGET)" EXTRA_CFLAGS="-O0 -ggdb"; \

3) Build the package: 

$ tar --exclude-vcs -zcvf ../gatekeeper_1.0.0.orig.tar.gz -C .. gatekeeper
$ debuild -uc -us

4) Install the gatekeeper package and corresponding debug symbols package. 

# dpkg -i ../gatekeeper_1.0.0-0_amd64.deb ../gatekeeper-dbgsym_1.0.0-0_amd64.ddeb

The package can then be controlled via systemd with the usual commands. For example, 

 # systemctl start gatekeeper   # start the service
 # systemctl stop gatekeeper    # stop the service
 # systemctl enable gatekeeper  # enable startup on boot
 # systemctl disable gatekeeper # disable startup on boot

To run Gatekeeper under GDB, it is first necessary to perform manually a few steps that the systemd unit performs before startup. These assume the GATEKEEPER_INTERFACES environment variable has been set in /etc/gatekeeper/envvars: 

# . /etc/gatekeeper/envvars
# install -oroot -ggatekeeper -m0770 -d /var/run/gatekeeper /var/run/dpdk/rte
# /usr/share/gatekeeper/devbind.sh $GATEKEEPER_INTERFACES
# gdb -ex 'handle SIG33 nostop noprint' --args /usr/sbin/gatekeeper $DPDK_ARGS -- $GATEKEEPER_ARGS

Notice that the directory must be the root of a recursively-cloned gatekeeper repository when calling gdb. Otherwise, gdb will not find the source code.

Finally, it's worth remembering that the Debian package of Gatekeeper also installs gkctl. The associated scripts are available at /usr/share/gatekeeper/. For example, to call script show_fib6.lua, enter: gkctl /usr/share/gatekeeper/show_fib6.lua.

Clone this wiki locally