XDP packets are sent with 2 seconds delays

[richd-de]

I wrote a very simple function which sends XDP packets, based on https://github.com/xdp-project/bpf-examples/blob/master/AF_XDP-example/xdpsock.c#L1535. There should be one packet sent once every 100ms. The problem is that all the packets are sent in ~10us after every 2 seconds ( I attached a Whireshark screenshot ). Any ideea why and how to fix this?
This is the function:

static int send_xdp_packet(struct xsk_socket_info* xsk, uint32_t* frame_nb, int batch_size)
{
uint32_t idx, idx_tx, tv_sec, tv_usec;
unsigned int i, rcvd;

if (xsk_ring_prod__reserve(&xsk->tx, batch_size, &idx_tx) != batch_size) {
    perror("Error xsk_ring_prod__reserve() failed");
    return (-1);
}
uint32_t len = PKT_SIZE;
// write entries in the Tx ring
xsk_ring_prod__tx_desc(&xsk->tx, idx_tx)->addr = *frame_nb * XSK_UMEM__DEFAULT_FRAME_SIZE;
xsk_ring_prod__tx_desc(&xsk->tx, idx_tx)->len = PKT_SIZE;
xsk_ring_prod__tx_desc(&xsk->tx, idx_tx)->options = 0;
*frame_nb = (*frame_nb + 1) % NUM_FRAMES;

++numOfPacketsSent;

xsk_ring_prod__submit(&xsk->tx, batch_size);

kick_tx(xsk);

rcvd = xsk_ring_cons__peek(&xsk->umem->cq, batch_size, &idx);
if (rcvd > 0) {
	xsk_ring_cons__release(&xsk->umem->cq, rcvd);
}

return 0;

}

static void kick_tx(struct xsk_socket_info* xsk)
{
int ret = sendto(xsk_socket__fd(xsk->xsk), NULL, 0, MSG_DONTWAIT, NULL, 0);
if (ret < 0) {
perror("kick_tx");
close(xsk_socket__fd(xsk->xsk));
exit(EXIT_FAILURE);
}
else if ((errno == ENOBUFS) || (errno == EAGAIN) || (errno == EBUSY) || (errno == ENETDOWN)) {
printf("WARNING kick_tx() sendto() returned error: %d\n", errno);
}
}

int main() {
/* ... */
xdp_mode = XDP_FLAGS_DRV_MODE;
xsks = xsk_configure_socket(umem, rx, tx, ifName, xdp_mode);
sockfd = xsk_socket__fd(xsks->xsk);
uint32_t frame_nb = 0;

    gen_eth_hdr_data();

    for (size_t i = 0; i < NUM_FRAMES; i++)   // NUM_FRAMES = 1
           gen_eth_frame(umem, i * opt_xsk_frame_size, PKT_SIZE, pkt_data);

 	while (true) {
	uint32_t frame_nb = 0;
        if (useXDP) {
		ssize_t n = send_xdp_packet(xsks, &frame_nb, opt_batch_size * frames_per_pkt); // opt_batch_size * frames_per_pkt = 1
        }

             usleep(100000);
  }

}

[magnus-karlsson]

Just as few questions.

• What mode are you running in? skb or zero-copy?
• What NIC are you using?
• You do have a sendto() in your kick_tx function I hope?

[richd-de]

Thank you for the reply!

1. I am running the app in driver mode, but I tried with zero-copy and had the same behavior.
2. Bellow is the NIC info :
   $ ~ sudo lshw -class network
   *-network
   description: Ethernet interface
   product: Ethernet Controller I225-LM
   vendor: Intel Corporation
   physical id: 0
   bus info: pci@0000:01:00.0
   logical name: enp1s0
   version: 03
   serial: 24:5e:be:6b:20:67
   capacity: 1Gbit/s
   width: 32 bits
   clock: 33MHz
   capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd
   autonegotiation
   configuration: autonegotiation=on broadcast=yes driver=igc driverversion=6.1.54-rt15 duplex=full firmware=1057:8754
   ip=198.168.68.203 latency=0 link=yes multicast=yes port=twisted pair

$ ~ sudo ethtool -i enp1s0
driver: igc
version: 6.1.54-rt15
firmware-version: 1057:8754
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

3. Yes, I use sendto() to kick the kernel. I edited the kick() function from the initial post, sorry about the confusion.

[magnus-karlsson]

Just to see if there is a problem with batching, could you send a packet every 200 ms instead and see if you get 10 packets every 2 seconds or 20 packets every 4 seconds?

Do you have another NIC you could try with, to see if you get the same behavior?

[richd-de]

With one packet sent every 200ms I get 10 packets every 2 seconds. Unfortunately, for the moment I don't have access to another NIC which supports XDP.

[richd-de]

Everything works as expected now, only after I enabled the XDP_COPY flag.

[magnus-karlsson]

Good to know. Then I would guess it is a bug in the i225 zero-copy driver, since it is not used when you use the XDP_COPY flag. We do have a test in our test suite that makes sure that sending a single packet works and is received quickly. Though I have never run it on an i225 since I do not have one in my lab. I would try reporting this to the person that has implemented this support.

commit 9acf59a752d4c686739117d3b3129e60af1ba5c1
Author: Andre Guedes [email protected]
Date: Thu Apr 22 23:25:55 2021 -0700

igc: Enable TX via AF_XDP zero-copy

Add support for transmitting packets via AF_XDP zero-copy mechanism.

[magnus-karlsson]

Can you check something for me please? Send one packet every 1 ms and see what you get. Just so I can pinpoint the problem better.

[richd-de]

Sorry for the late reply. With XDP_ZEROCOPY it sends only 1015 packets every 2 seconds when one packet is sent every 1ms. It also sends only 1015 packets every 2 seconds when the packets are sent every 0.1ms (100 us). With XDP_COPY the packets are sent correctly for both sending intervals.

[magnus-karlsson]

Weird. It is like the NIC driver in zero-copy mode has a 2 second timer that fires and then sends the packets.