Configure SR IOV with network config manager

SR-IOV is a specification that allows a PCIe device to appear to be multiple separate physical PCIe devices. The SR-IOV specification was created and is maintained by the PCI SIG, with the idea that a standard specification will help promote interoperability. SR-IOV works by introducing the idea of physical functions (PFs) and virtual functions (VFs). Physical functions (PFs) are full-featured PCIe functions; virtual functions (VFs) are “lightweight” functions that lack configuration resources Good new is systemd-networkd supports it with this commit and here is the documentation. Now how we can configure it via nmctl. I used kernel module netdevsim to configure and test it.

➜  ~  modprobe netdevsim                                                                                                                    
➜  ~  lsmod | grep netdevsim 
netdevsim             102400  0
psample                20480  1 netdevsim

➜  ~  echo "10 1" > /sys/bus/netdevsim/new_device 

➜  ~ sudo echo "99 1" > /sys/bus/netdevsim/new_device

➜  ~ ip -d link show eni99np1
287: eni99np1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ca:28:ff:4e:73:2a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 portname p1 switchid 82ae398327c5db81a27dc2756c43f00315f442de1779fcfbfc582bbb3e62cb parentbus netdevsim parentdev netdevsim99 

➜  ~ echo "3" > /sys/bus/netdevsim/devices/netdevsim99/sriov_numvfs

➜  ~ ip -d link show eni99np1                                      
287: eni99np1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ca:28:ff:4e:73:2a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 portname p1 switchid 82ae398327c5db81a27dc2756c43f00315f442de1779fcfbfc582bbb3e62cb parentbus netdevsim parentdev netdevsim99 
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
It's ready now. Let's configure it via `nmctl`
➜  ~ nmctl add-sr-iov eni99np1 vf 0 vlanid 5 qos 1 macspoofck yes qrss True trust yes linkstate yes macaddr 00:11:22:33:44:55

➜  ~ ip -d link show eni99np1                                                                                                
287: eni99np1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ca:28:ff:4e:73:2a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 portname p1 switchid 82ae398327c5db81a27dc2756c43f00315f442de1779fcfbfc582bbb3e62cb parentbus netdevsim parentdev netdevsim99 
    vf 0     link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff, vlan 5, qos 1, spoof checking on, link-state enable, trust on, query_rss on

➜  ~ sudo cat /etc/systemd/network/10-eni99np1.network
[Match]
Name=eni99np1


[SR-IOV]
VirtualFunction=0
VLANId=5
QualityOfService=1
MACSpoofCheck=yes
QueryReceiveSideScaling=yes
Trust=yes
LinkState=yes
MACAddress=00:11:22:33:44:55

As seen nmctl has generated SR-IOV configuration in systemd-networkd format and since nmctl reloads the configuration, systemd-networkd also configured the VF. Similarly we can configure other VF's

➜  ~ nmctl add-sr-iov eni99np1 vf 1 vlanid 6 qos 2 macspoofck yes qrss True trust yes linkstate yes macaddr 00:11:22:33:44:56

➜  ~ nmctl add-sr-iov eni99np1 vf 1 vlanid 6 qos 2 macspoofck yes qrss True trust yes linkstate yes macaddr 00:11:22:33:44:5

➜  ~ ip -d link show eni99np1                                                                                                
287: eni99np1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ca:28:ff:4e:73:2a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 portname p1 switchid 82ae398327c5db81a27dc2756c43f00315f442de1779fcfbfc582bbb3e62cb parentbus netdevsim parentdev netdevsim99 
    vf 0     link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff, vlan 5, qos 1, spoof checking on, link-state enable, trust on, query_rss on
    vf 1     link/ether 00:11:22:33:44:56 brd ff:ff:ff:ff:ff:ff, vlan 6, qos 2, spoof checking on, link-state enable, trust on, query_rss on
    vf 2     link/ether 00:11:22:33:44:57 brd ff:ff:ff:ff:ff:ff, vlan 7, qos 3, spoof checking on, link-state enable, trust on, query_rss on

And .network generated by nmctl

➜  ~ cat /etc/systemd/network/10-eni99np1.network
[Match]
Name=eni99np1

[SR-IOV]
VirtualFunction=0
VLANId=5
QualityOfService=1
MACSpoofCheck=yes
QueryReceiveSideScaling=yes
Trust=yes
LinkState=yes
MACAddress=00:11:22:33:44:55


[SR-IOV]
VirtualFunction=1
VLANId=6
QualityOfService=2
MACSpoofCheck=yes
QueryReceiveSideScaling=yes
Trust=yes
LinkState=yes
MACAddress=00:11:22:33:44:56


[SR-IOV]
VirtualFunction=2
VLANId=7
QualityOfService=3
MACSpoofCheck=yes
QueryReceiveSideScaling=yes
Trust=yes
LinkState=yes
MACAddress=00:11:22:33:44:57