[Bug]: VirtioSerialDxe doesn't interface properly with Serial Consoles attached to Virtualization Framework VMs

[t0rr3sp3dr0]

Is there an existing issue for this?

• I have searched existing issues

Bug Type

• Firmware
• Tool
• Unit Test

Code first?

• Yes

What packages are impacted?

OvmfPkg

Which targets are impacted by this bug?

RELEASE, DEBUG, NOOPT

Current Behavior

Applications using the VirtioSerialDxe driver in a Virtualization Framework VM are not able to read or write to the attached serial console.

Expected Behavior

VirtioSerialDxe should allow reads and writes from/to VirtIO Serial Consoles attached to Virtualization Framework VMs, like it does in QEMU VMs and like the Linux Driver does in Virtualization Framework VMs.

Steps To Reproduce

1. In an Intel Mac, install all dependencies to build edk2 and install vfkit (https://github.com/crc-org/vfkit?tab=readme-ov-file#installation)

2. Acquire the source code of edk2

git clone https://github.com/tianocore/edk2

3. Build OvmfPkg

cd ./edk2/OvmfPkg
bash ./build.sh

4. Create a bootable image

mkdir -p ./UNTITLED/EFI/BOOT
cp ./edk2/Build/OvmfX64/DEBUG_XCODE5/X64/Shell.efi ./UNTITLED/EFI/BOOT/BOOTx64.efi
cp ./edk2/Build/OvmfX64/DEBUG_XCODE5/X64/VirtioSerialDxe.efi ./UNTITLED/VirtioSerialDxe.efi
echo 'load FS0:\VirtioSerialDxe.efi' > ./UNTITLED/startup.nsh
hdiutil create -layout GPTSPUD -fs MS-DOS -srcfolder ./UNTITLED -format UDTO -ov ./UNTITLED.cdr

5. Start the Virtualization Framework VM with vfkit

vfkit \
    --bootloader efi,variable-store=$(mktemp),create \
    --device virtio-blk,path=./UNTITLED.cdr \
    --device virtio-input,keyboard \
    --device virtio-input,pointing \
    --device virtio-gpu,width=1280,height=800 \
    --device virtio-serial,logFile=./serial.txt \
    --gui

6. Check that nothing was written to ./serial.txt. Note that this file should at least the control sequences sent by TerminalDxe upon connecting.

Build Environment

- OS(s): macOS Sonoma
- Tool Chain(s): XCODE5

Version Information

Commit: fc140c5eae8ac9cc183f6b96bfe4b595b7119301

AFAIK, no version of VirtioSerialDxe ever worked with Virtualization Framework VMs. Above is the current hash of the master branch.

Urgency

Medium

Are you going to fix this?

Someone else needs to fix it

Do you need maintainer feedback?

Maintainer feedback requested

Anything else?

I've been trying to debug this for days and unfortunately I couldn't find the root cause of the problem. I found a related issue that is being fixed in #10575, but that patch does not address this problem fully.

I also ran Linux on the same VM and added some debug logs to their driver to try identify something that our driver doing differently, but it only helped to a certain point. And honestly, I think the VirtioSerialDxe driver is more in line with the VirtIO spec than the virtio_console driver.

The Linux driver does some unexpected stuff:

1. Ignore all messages received
2. Send VIRTIO_SERIAL_PORT_OPEN
3. Send VIRTIO_SERIAL_PORT_READY
4. Send VIRTIO_SERIAL_DEVICE_READY

Surprisingly this works.

Faking the device features presented to include VIRTIO_SERIAL_F_MULTIPORT makes the Linux driver behave more in line with our implementation:

1. Send VIRTIO_SERIAL_DEVICE_READY
2. Receive VIRTIO_SERIAL_DEVICE_ADD
3. Send VIRTIO_SERIAL_PORT_READY
4. Receive VIRTIO_SERIAL_CONSOLE_PORT
5. Send VIRTIO_SERIAL_PORT_OPEN

This also works and will be exactly what VirtioSerialDxe will do when #10575 gets merged.

The VirtIO Serial device in Virtualization Framework supports features VIRTIO_F_INDIRECT_DESC, VIRTIO_F_EVENT_IDX, VIRTIO_F_VERSION_1, and VIRTIO_F_RING_PACKED. The final set of negotiated features with the driver is just VIRTIO_F_VERSION_1. The device doesn't support Legacy VirtIO and throws an error if we try to negotiate that.

In my debugging, I could see that the driver sends the bytes all the way to VirtioSerialRingSendBuffer and successfully calls SetQueueNotify. But the serial console never gets those bytes. At this point, I think the problem is not related to the control messages being exchanged between the device and the driver. I guess it's some misconfiguration in the VirtIO rings or in the PCI layer.

I have almost no experience with VirtIO, UEFI, PCI, and drivers in general, so my debugging abilities don't go too far. I've been simply comparing and trying to match the behaviors of VirtioSerialDxe in Virtualization Framework vs QEMU and virtio_console in Linux, but unfortunatly that was not enough. I would appreciate some help from people with more expertise.

Btw, since we don't have serial, I've modified OvmfPkg to use UefiDebugLibConOut instead of BaseDebugLibSerialPort to allow us to see the debug messages: t0rr3sp3dr0@72382d2.