can't use pktgen to send udp traffic to public server from local VM / public server

[Fuji1405116]

we setup the full dpdk environment in VM. then changed the config.ini a bit by changing the "port0" mac & ip to dpdk binded ethernet address and ip assigned to that interface. Then in main.c, I changed the recv_ip to my public server ip. I started the pktgen after make, I am seeing those "vm_dpdk_ip > public_server_ip" packets from host pc through tcpdump command but I am not actually getting those udp packets in the server (checked via tcpdump). The server is just simple linux server which was meant to receive packets and my VM's dpdk binded ip was meant to sent. my main target was to check whether I am able send/receive udp packets using dpdk in public internet. What am I doing wrong here ? Thanks in advance

[Fuji1405116]

I setup dpdk enviroment in public server1 to send packet to public server2 (not connected via ethernet, but possible to send udp packets,right?) I tried starting pktgen in sending mode in server and it was saying :
SYSCALL: Sendto failed to put packet in the TX ring Total: 2047 Free: 0
though it got solved somehow after I reconfigured by dpdk-setup.sh
a few times (which I ignored previously in VM) and then start printing some receiving packets:

POLLBODY: Received non-IPv4 packet, showing content below:
Dumping payload [len = 60]:
    0: ff ff ff ff ff ff 20 4e 71 3a 2a 40 08 06 00 01 ...... Nq:*@....
   16: 08 00 06 04 00 01 20 4e 71 3a 2a 40 95 14 bb 91 ...... Nq:*@....
   32: 00 00 00 00 00 00 95 14 bb 97 00 00 00 00 00 00 ................
   48: 00 00 00 00 00 00 00 00 00 00 00 00 
POLLBODY: Received non-IPv4 packet, showing content below:
Dumping payload [len = 60]:
    0: ff ff ff ff ff ff 20 4e 71 3a 2a 40 08 06 00 01 ...... Nq:*@....
   16: 08 00 06 04 00 01 20 4e 71 3a 2a 40 48 15 19 c1 ...... Nq:*@H...
   32: 00 00 00 00 00 00 48 15 19 cb 00 00 00 00 00 00 ......H.........
   48: 00 00 00 00 00 00 00 00 00 00 00 00 
POLLBODY: Received non-IPv4 packet, showing content below:
Dumping payload [len = 60]:
    0: ff ff ff ff ff ff 20 4e 71 3a 2a 40 08 06 00 01 ...... Nq:*@....
   16: 08 00 06 04 00 01 20 4e 71 3a 2a 40 48 15 19 c1 ...... Nq:*@H...
   32: 00 00 00 00 00 00 48 15 19 ce 00 00 00 00 00 00 ......H.........
   48: 00 00 00 00 00 00 00 00 00 00 00 00 
POLLBODY: Received non-IPv4 packet, showing content below:
Dumping payload [len = 60]:
    0: ff ff ff ff ff ff 20 4e 71 3a 2a 40 08 06 00 01 ...... Nq:*@....
   16: 08 00 06 04 00 01 20 4e 71 3a 2a 40 95 14 bc 81 ...... Nq:*@....
   32: 00 00 00 00 00 00 95 14 bc a3 00 00 00 00 00 00 ................
   48: 00 00 00 00 00 00 00 00 00 00 00 00 
POLLBODY: Received non-IPv4 packet, showing content below:
Dumping payload [len = 60]:
    0: ff ff ff ff ff ff 20 4e 71 3a 2a 40 08 06 00 01 ...... Nq:*@....
   16: 08 00 06 04 00 01 20 4e 71 3a 2a 40 48 15 19 c1 ...... Nq:*@H...
   32: 00 00 00 00 00 00 48 15 19 df 00 00 00 00 00 00 ......H.........
   48: 00 00 00 00 00 00 00 00 00 00 00 00

I tried to set the receiving ip from main.c and sending ip+mac from the config.ini (DPDK bound mac and IP) but no improvements. what possible wrong thing can I be doing here you think?
my config.ini:

lcores_primary=2
lcores_secondary=4
n_mem_channels=2

[port0]
mac_addr=e4:43:4b:4e:82:00
ip_addr=149.20.189.126

[port0_dst]
#mac_addr=68:05:ca:95:fa:64
ip_addr=72.21.27.26

I have set receiving ip in main.c too by
#define IP_RECV "172.31.100.1"
my current start log:

sudo ./pktgen -c ../../config.ini -f send -s 10 -r 10 -l pktgen-send.log -h
Do not know how to parse section:port0_dst name:ip_addr
Application args: ./pktgen -l 2 -n 2 --proc-type=primary -- -f send -s 10 -r 10 -l pktgen-send.log -h 
Poller args: ./pktgen -l 4 -n 2 --proc-type=secondary 
EAL: Detected 56 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected 56 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_i40e (8086:1572) device: 0000:01:00.1 (socket 0)
EAL: No legacy callbacks, legacy socket not created
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_14352_84004a4bef8ac
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_i40e (8086:1572) device: 0000:01:00.1 (socket 0)
EAL: No legacy callbacks, legacy socket not created
POLLINIT: Waiting to initialize IPC...
INIT: Initialized port 0.
MONITOR: Checking link status of port 0.
INIT: Using the same port for RX and TX
INIT: Waiting for the poller to complete its inialization...
POLLINIT: IPC initialized
App: UDPDK Intialized
Parsing app arguments...
SEND mode
SYSCALL: Binding port 10000 to sock_id 0
Entering send loop
Sending a packet every 0.100000000 s
USER1: rte_ip_frag_table_create: allocated of 25165952 bytes at socket 0
POLLINIT: Created IP fragmentation table
POLLBODY: Received non-IPv4 packet, showing content below:
Dumping payload [len = 119]:
    0: 01 80 c2 00 00 00 00 1c 73 08 7b 87 00 69 42 42 ........s.{..iBB
   16: 03 00 00 03 02 4e 80 00 00 1c 73 08 7b 83 00 00 .....N....s.{...
   32: 00 00 80 00 00 1c 73 08 7b 83 80 04 00 00 14 00 ......s.{.......
   48: 02 00 0f 00 00 00 40 00 00 00 00 00 00 00 00 00 ......@.........
   64: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
   80: 00 00 00 00 00 00 00 00 00 00 ac 36 17 7f 50 28 ...........6..P(
   96: 3c d4 b8 38 21 d8 ab 26 de 62 00 00 00 00 80 00 <..8!..&.b......
  112: 00 1c 73 08 7b 83 14 
POLLBODY: Received non-IPv4 packet, showing content below:
Dumping payload [len = 119]:
    0: 01 80 c2 00 00 00 00 1c 73 08 7b 87 00 69 42 42 ........s.{..iBB
   16: 03 00 00 03 02 4e 80 00 00 1c 73 08 7b 83 00 00 .....N....s.{...
   32: 00 00 80 00 00 1c 73 08 7b 83 80 04 00 00 14 00 ......s.{.......
   48: 02 00 0f 00 00 00 40 00 00 00 00 00 00 00 00 00 ......@.........
   64: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
   80: 00 00 00 00 00 00 00 00 00 00 ac 36 17 7f 50 28 ...........6..P(
   96: 3c d4 b8 38 21 d8 ab 26 de 62 00 00 00 00 80 00 <..8!..&.b......
  112: 00 1c 73 08 7b 83 14 
^CCaught signal 2 in sendrecv main process
INTR: Killing the poller process (14352)...
POLLINTR: Received request to stop
POLLBODY: Polling process exiting.
Closing log...
INTR: Killing the poller process (14352)...
CLOSE: Killing the poller process (14352)...
CLOSE: ...killed!
CLOSE: Closing socket 0 that was left open

[leoll2]

Hello Fuji,
I see some issues in your config.ini:

• The destination mac_addr is commented out
• The destination ip_addr should not be in config.ini (indeed, the logs print "Do not know how to parse section:port0_dst name:ip_addr"). It should be in main.c as #define IP_RECV "72.21.27.26"

I can confirm that the sender is failing to send packets, otherwise you would see stats being printed periodically on the console. What network card are you using? What is your DPDK (not UDPDK) configuration? Have you ever sent data with DPDK from that host?

By the way, you said that you could see the outbound packets in tcpdump; did you use dpdk-pdump or similar?

As for the non-IP received packets you see in the logs, they are STP and ARP ("broadcast ARP Who has 149.20.187.151? Tell 149.20.187.145") coming from other hosts in the network, so you can ignore them.

[Fuji1405116]

I am totally new to DPDK. I just learned 1 days ago that to fulfill my actual purpose I have to use Open vSwitch in the same server running dpdk. I actually tried UDPDk in local VM.. And tried to send packets from VM1(using UDPDK) to VM2, both VM were confugure with NAT and they using vfio driver. I didn't usw pdump to catch packets, can you shed some light on that? And isn't it possible to send dpdk packets from 1 local vm to another local vm using ether net address?
Thanks for the reply.

[leoll2]

I didn't usw pdump to catch packets, can you shed some light on that?

As far as I know, tcpdump doesn't work with DPDK because DPDK takes full control of the NIC whereas tcpdump reads from the regular network stack. You can learn more about pdump here: https://doc.dpdk.org/guides-16.07/sample_app_ug/pdump.html

And isn't it possible to send dpdk packets from 1 local vm to another local vm using ether net address?

It depends. If there is just a switch in-between, that should be possible. If you have routers (layer 3 devices) in the middle, the lack of ARP support may represent an obstacle unless you set a static ARP table.

[Fuji1405116]

yeah I will try to use Open vSwitch to integrate with dpdk to send/receive data first, then I will look into integrating UDPDK with Open vSwitch and report back. So if I use pure dpdk with open vSwitch then I have to do some extra work to make it possible for UDP packet sending right that you have already done in UDPDK? And thanks for the help..!

[Fuji1405116]

lats update : I was able to send udp packets using apps/pktgen from VM1 (using UDPDK) to Another VM2 (normal) and confirmed by receiving UDP packets from java app, and I used qemu/kvm and my vm's network were configured via NAT, actually they were connected through virtual ethernet cable which we confirmed by "arping" each-other. Same went successful for 2 public server but they were connected in same switch (confirmed via 'arping' each other). I think that's it for UDPDK as to send 1 public server to another remote public server, we have to use Open vSwitch.
But @leoll2 , when I was receiving packets in receiver end I couldn't see stats printing , and ran out problems sometimes like "can't put packet to tx ring" or "failed, total : 2048, free : 0" but then after restarting the pktgen app, it got fixed, Any idea why this is happening?

[Fuji1405116]

@leoll2 , I was able to send UDP packets even to remote server by setting target mac as gateway (next hop) and target ip as destination ip and I can receive those udp packets from that remote public server. Now I can't receive udp packets using pktgen, it seems like the code is stuck in "udpdk_recvfrom()" function and isn't proceeding. And another thing, whenever I run pkgen recv/send mode for the first few seconds, it seems to poll some large amount of non ipv4 packets everytime though I am not sending anything. can you shed on light what could be the possible reason for the code to stuck in udpdk_recvfrom() function?

[leoll2]

Now I can't receive udp packets using pktgen, it seems like the code is stuck in "udpdk_recvfrom()" function and isn't proceeding. Can you shed on light what could be the possible reason for the code to stuck in udpdk_recvfrom() function?

Hard to tell, what I can suggest is to instrument the source code of udpdk_recv with some log lines, then recompile. Even if you're not familiar with UDPDK architecture, there are quite a few comments in the code.

whenever I run pkgen recv/send mode for the first few seconds, it seems to poll some large amount of non ipv4 packets everytime though I am not sending anything

They are just packets coming from other hosts/devices in the network, not necessarily targeted to your host/application. If you are curious, you can inspect their content using this tool. Anyway they are ignored, so don't worry too much.

I was able to send UDP packets even to remote server by setting target mac as gateway (next hop) and target ip as destination ip and I can receive those udp packets from that remote public server.

May I ask you more details on how you configured UDPDK and (possibly) the devices in the middle? It could be helpful for other people trying to solve the same problem. If I understand correctly, you configured the sender with destination IP_ADDR <server ip> and destination mac_addr=<next hop in the network>? Is there a virtual/real switch/router in your setup?

[Fuji1405116]

Sending is working properly currently.
Lets say I need to send udp packets from server A to server B. And I am using dpdk in server A. There is no managed switch between A or B, they are just two independent public server with public ip. So I ran "arp" command in A to find the gateway mac which I later set in dst mac (config.ini). And changed the dst ip (server B ip) in main.c & now udp packets are flowing to server B from server A (from dpdk-binded port) using pktge in sending mode.
But I am little confused about the receiving procedure. I inspected (adding logs and recompile) the recv() functions where it got stuck and saw that it is stuck in busy waiting for packet dequeue which actually a library function of dpdk : rte_ring_dequeue() (so no packets actually came?). Lets say now I am running pktgen on server A in receiving mode (lets assume A1 is public ip and A2 is the dpdk-binded ip in A) so it will start polling. But the udp packet now I am sending from server B is containg dst ip address as server A's dpdk binded ip A1. Prevuously in sending mode I didnt have to bother about my dpdk-binded ip but as now I am receiving how this ip will get resolved from geteway? Or gateway will atleast broadcast packets and my dpdk-binded nic port will get it anyway by polling?
I assume in my case packets aren't even reaching to server A from server B. Can u suggest what to do so?

[Fuji1405116]

from pktgen related question (stackoverflow) I came to know that the way I sent the packets is not a standard way and I will have to create tap/tun interface to send receive dpdk accelarated packets from/to another public server. So when I will send a dpdk accelarated udp packets to another public server then it will also help the neighbour nodes to resolve dpdk-binded ip which will help in receiving udp packets destined for dpdk-binded ip. I am not very clear about using tap/tun interface and use that to send receive packets, can you suggest how to implement that with your UDPDK for sending/receiving?