Add AMD GPU to Talos VM (hosted on Proxmox)

[1moere1]

Problem

• The AMD GPU is not visible in the Talos VM

Situation

• Talos Worker VM running on a proxmox node with an AMD Radeon RX 550 GPU
• I create a vfio device and attached it to the talos worker VM
• I added the siderolabs/amdgpu module to the talos worker that is running on node03
• I added a bind mount to the extraMounts (/dev/dri to /dev/dri with bind, rshared, rw as options)

Proxmox insights

# lspci -nnk -d 1002:67ff
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] [1002:67ff] (rev ef)
        Subsystem: Hewlett-Packard Company Baffin [Radeon RX 550 640SP / RX 560/560X] [103c:83e2]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

# qm>  info pci
...
  Bus  1, device   0, function 0:
    VGA controller: PCI device 1002:67ff
      PCI subsystem 103c:83e2
      IRQ 10, pin A
      BAR0: 64 bit prefetchable memory at 0xd0000000 [0xdfffffff].
      BAR2: 64 bit prefetchable memory at 0xe0000000 [0xe01fffff].
      BAR4: I/O at 0x5000 [0x50ff].
      BAR5: 32 bit memory at 0xfe800000 [0xfe83ffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      id "hostpci0.0"
  Bus  1, device   0, function 1:
    Audio controller: PCI device 1002:aae0
      PCI subsystem 103c:aae0
      IRQ 10, pin B
      BAR0: 64 bit memory at 0xfe860000 [0xfe863fff].
      id "hostpci0.1"
...

# cat /etc/pve/qemu-server/513.conf
agent: 1
boot: order=net0;scsi0
cores: 4
cpu: x86-64-v2-AES
hostpci0: 0000:01:00,pcie=1
hotplug: disk,usb
machine: q35
memory: 16384
meta: creation-qemu=8.1.5,ctime=1734520425
name: lab-talos-wk03
net0: virtio=BC:24:11:F0:38:F6,bridge=vmbr0,firewall=1,tag=30
numa: 0
onboot: 1
ostype: l26
scsi0: data-ssd:vm-513-disk-0,iothread=1,size=30G
scsi1: data-ssd:vm-513-disk-1,iothread=1,size=250G
scsihw: virtio-scsi-single
smbios1: uuid=71d9881e-8abb-4bb4-b7f0-fa45f58f9ef3
sockets: 1
vmgenid: c79572b6-0516-4518-bbbb-80d1c92b892f

Talos insights

$ talosctl -n 192.168.30.113 ls /dev/dri -l
NODE             MODE         UID   GID   SIZE(B)   LASTMOD           LABEL   NAME
192.168.30.113   drwx------   0     0     40        Jan 31 15:29:56           .

$ talosctl -n 192.168.30.113 read /proc/modules
virtio_balloon 32768 0 - Live 0xffffffffc05f8000
iTCO_wdt 12288 1 - Live 0xffffffffc0623000
iTCO_vendor_support 12288 1 iTCO_wdt, Live 0xffffffffc05cd000
watchdog 40960 1 iTCO_wdt, Live 0xffffffffc05b4000
virtio_pci 40960 0 - Live 0xffffffffc060a000
virtio_pci_legacy_dev 12288 1 virtio_pci, Live 0xffffffffc0601000
ahci 45056 0 - Live 0xffffffffc059a000
virtio_pci_modern_dev 24576 1 virtio_pci, Live 0xffffffffc05e0000
lpc_ich 28672 0 - Live 0xffffffffc05cf000
libahci 49152 1 ahci, Live 0xffffffffc05ba000
i2c_i801 36864 0 - Live 0xffffffffc05a7000
mfd_core 12288 1 lpc_ich, Live 0xffffffffc059e000
i2c_smbus 12288 1 i2c_i801, Live 0xffffffffc0594000

[smira]

First, try talosctl get pcidevices on Talos, if it doesn't list your GPU, you're passing it through wrong way.

[1moere1]

Hmm yes it's visible:

192.168.30.113   hardware    PCIDevice   0000:06:10.0   1         Display controller         VGA compatible controller   Advanced Micro Devices, Inc. [AMD/ATI]   Baffin [Radeon RX 550 640SP / RX 560/560X]
192.168.30.113   hardware    PCIDevice   0000:06:10.1   1         Multimedia controller      Audio device                Advanced Micro Devices, Inc. [AMD/ATI]   Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]

And also appearing in logs

ansible@jump:/opt/talos/00_k8s$ talosctl --talosconfig clusterconfig/talosconfig -n 192.168.30.113 dmesg | grep 0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:23.717484452Z]: pci 0000:06:10.0: [1002:67ff] type 00 class 0x030000 conventional PCI endpoint
 DEVICE=+pci:0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:23.722508452Z]: pci 0000:06:10.0: BAR 0 [mem 0xc0000000-0xcfffffff 64bit pref]
 DEVICE=+pci:0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:23.730504452Z]: pci 0000:06:10.0: BAR 2 [mem 0xd0000000-0xd01fffff 64bit pref]
 DEVICE=+pci:0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:23.738505452Z]: pci 0000:06:10.0: BAR 4 [io  0x4000-0x40ff]
 DEVICE=+pci:0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:23.746508452Z]: pci 0000:06:10.0: BAR 5 [mem 0xfde00000-0xfde3ffff]
 DEVICE=+pci:0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:23.754507452Z]: pci 0000:06:10.0: ROM [mem 0xfde80000-0xfde9ffff pref]
 DEVICE=+pci:0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:23.755270452Z]: pci 0000:06:10.0: supports D1 D2
 DEVICE=+pci:0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:23.914493452Z]: pci 0000:06:10.0: vgaarb: bridge control possible
 DEVICE=+pci:0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:23.914790452Z]: pci 0000:06:10.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
 DEVICE=+pci:0000:06:10.0
192.168.30.113: kern:    info: [2025-01-31T20:51:24.032621452Z]: pci 0000:06:10.1: D0 power state depends on 0000:06:10.0

[smira]

Look for amdgpu driver - whether it picks up, reports any errors, etc.