How to improve network performance between VMs

[Milon-yang]

Hi,

I use iperf to test the network performance, the result shows the sending and receiving performance of network data between VMs is 35MB/sec. What maximum data throughput does L4RE support？Can we further improve its performance, and how to?

Thanks!

[icedieler]

Hi,
I assume you used our virtual virtio peer-to-peer link to connect the VMs? The achievable performance then depends on the VM configuration (e.g. vCPU to CPU assignment), the type of traffic (TCP or UDP) and also the memory bandwith of the platform you are using.
Our virtio peer-to-peer link only (mem)copies network packets between the VMs and does no checksumming or whatsoever. That means the memory bandwith is the upper limit here. In the VMs you have to account for the processing of the network packets.
From your question I defer that you were expecting a better performance. What was you expectation?
Cheers,
Matthias.

[alacko]

Hi, are you doing network between different cores or VMs running on the same core?

[Milon-yang]

Hi, are you doing network between different cores or VMs running on the same core?

I did it on the same core

[Milon-yang]

Hi, I assume you used our virtual virtio peer-to-peer link to connect the VMs? The achievable performance then depends on the VM configuration (e.g. vCPU to CPU assignment), the type of traffic (TCP or UDP) and also the memory bandwith of the platform you are using. Our virtio peer-to-peer link only (mem)copies network packets between the VMs and does no checksumming or whatsoever. That means the memory bandwith is the upper limit here. In the VMs you have to account for the processing of the network packets. From your question I defer that you were expecting a better performance. What was you expectation? Cheers, Matthias.

Yes, I need to improve the network bandwidth between the VMs.
Do you have any suggestion or guidance note ？

[alacko]

Hi, are you doing network between different cores or VMs running on the same core?

I did it on the same core

Ok, could you put the VMs on different cores and possibly also the switch to another one and compare?

[Milon-yang]

Hi,

I try to create 2 VMs, one occupies cpu0, while another VM occupies cpu3. then use l4vio_net_p2p for virtual network between VMs.
The network performance is still as same as in same cpu.

[alacko]

Please try to increase the queue size with the virtio-net component using the -s switch, like -s 4096. Increasing the MTU size in Linux might also be helpful.

[Milon-yang]

Thanks for your suggestion.
I tried to increasing the MTU size in Linux, the network performance could be improved. for example, I changed the value from 1500 to 9000, the network bandwidth improved to 110M/sec.

But when I changed virtqueue size from 256 (defalut value) to 4096, the network bandwidth only improved to 40M/sec.

By the way, I just changed the var "vq_max_num" in l4/pkg/virtio-net/server/p2p-link/net.cc, I don't know if I changed it correctly.

[alacko]

If it works it's change correctly I guess. The value can probably be even higher. And please combine with the MTU increase.
What you can also try to put the network component on a separate core on its own (i.e. use 3 cores for this setup).

[Milon-yang]

I increased the mtu value to 65535 (the biggest value), the network bandwidth improved to 310M/sec.
But when increase virtqueue size combined with mtu value, the performance can not be further improved, In other words, increasing virtqueue size didn't work very well.

In fact, I want to improve the bandwidth to 1GB/sec at least. Is there any other ways?

Thanks.

[icedieler]

I increased the mtu value to 65535 (the biggest value), the network bandwidth improved to 310M/sec.

Is it MB/s or Mbits/s?

In fact, I want to improve the bandwidth to 1GB/sec at least. Is there any other ways?

Do you happen to know the memory bandwidth of your platform? 10G sounds very ambitious to me.

[Milon-yang]

It is MBytes/sec
I experimented on PCU2.0, the memory bandwidth is 8G.

[jiangfenghua123]

the memory bandwidth of pcu is 51.2GB/s