Passthru my NVidia RTX 2080ti to nomadOS : bhyve: pci_passthru: map_pptdev_mmio failed

[Marietto2008]

Hello.

I'm enjoying myself with a new experiment. I'm trying to pass my nVidia graphic card inside a bhyve VM,this time running NomadOS and not Linux. Below you can see the parameters that I've used on bhyve :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/$vmdisk3'p2'/bhyve/img/Linux/ubuntu2204-cuda510.img,bootindex=1 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 10,virtio-net,tap1 \
-s 11,virtio-9p,sharename=/ \
-s 13,hda,play=/dev/dsp,rec=/dev/dsp \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:1 < /dev/null & sleep 2 && vncviewer 0:1

and this is the messages that I see on the monitor :

2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) map
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) map
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) map
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) map
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) map
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) map
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) map
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) map
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) map
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) map
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) map
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) map
Unhandled ps2 mouse command 0xe1
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) unmap
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) unmap
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000ff000000 (size           880000) map
bhyve: pci_passthru: vm_map_pptdev_mmio nvidia low failed
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000ff881000 (size           77f000) map
bhyve: pci_passthru: vm_map_pptdev_mmio nvidia low failed
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000ff000000 (size           880000) unmap
bhyve: pci_passthru: vm_unmap_pptdev_mmio nvidia low failed
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000ff881000 (size           77f000) unmap
bhyve: pci_passthru: vm_unmap_pptdev_mmio nvidia low failed
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) map
2/0/0 modify_mmio 0x0000000082881000 -> 0x00000000c2881000 (size           77f000) map

bhyve: pci_passthru: map_pptdev_mmio failed
Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 602.

it seems to be a bug in nomadOS. I haven't any problem passthruing the graphic card inside a bhyve / Linux VM. So the error could be related to bhyve and NomadOS only.

Look closely at the messages:

2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000c2000000 (size           880000) map

…
2/0/0 modify_mmio 0x0000000082000000 -> 0x00000000ff000000 (size           880000) map

The end address changes from c2000000 to ff000000. That’s wrong. An OS can write 0xFFFFFFFF to a BAR to get it’s size. PCI specification says that an OS should disable BARs before it does this check. Maybe NomadOS doesn’t follow the specification which confuses bhyve ? What do you think ? thanks.

[bsdlme]

I'm afraid that our expertise in bhyve is not good enough to debug this. You might have more success asking on [email protected] mailing list, pointing to this Issue.