Why does the page fault handler only support traps caused by accessing the PLIC?

[ZhongkaiXu]

Hello, when I was reviewing the RISC-V part, I noticed that in the guest_page_fault_handler function, it seems that only traps related to the PLIC region are handled. Is this because it hasn't been fully implemented yet, or is it due to hardware limitations that don't support other data aborts?
I'm asking this question because I tried to let the VM access the PCI-ECAM region, which caused a trap into the hypervisor, but through QEMU's trace, I found that some access instructions didn't seem to be caught by the hypervisor and were directly returned as 0 to the VM.

[josecm]

I'm asking this question because I tried to let the VM access the PCI-ECAM region, which caused a trap into the hypervisor, but through QEMU's trace, I found that some access instructions didn't seem to be caught by the hypervisor and were directly returned as 0 to the VM.

That is indeed strange. When you say "tried to let the VM access the PCI-ECAM region", you mean you added this MMIO region to the VM's configuration dev list? If so, this is because the hypervisor is indeed passing through that region to the guest and the value from the MMIO registers you read is indeed zero. If not, the behavior you are observing shouldn't happen. First, the hardware should trap the access. Second, either there is an emulation handler registered within the hypervisor (and here you are correct, the only emulation handler at the moment for RISC-V is for the PLIC), or the hypervisor should simply stop the guest.

Maybe if you give me you VM configuration and tell me which MMIO addresses you are accessing I could look into it.

[ZhongkaiXu]

Apologies if my previous explanation was not clear.

I’m specifically discussing the setup of emul_access.addr before passing the struct emul_access to the handler. In the code, vaddr_t addr = CSRR(CSR_HTVAL) << 2; , we obtain the GPA for this exception, meaning the lowest two bits of the GPA from HTVAL are zero. However, when the VM accesses certain PCIe-ECAM registers, the address's lowest two bits may not be zero.

For instance, when accessing a register with an offset of 0x6 (0b110) and a GPA of 0x30000006, after trapping into the hypervisor, the GPA retrieved from HTVAL is 0x30000004. So we need to get the lowest two bits ( 0b10 in this example ) from STVAL and get the complete GPA 0x30000006.

I hope this clarifies my point. Does this make sense to you now?

[josecm]

It makes sense. I get your point. This code assumed any MMIO register access must be aligned to 4 bytes, but maybe we should account for byte and halfword accesses. Can you apply this patch and check if it makes sense for you and fixes your issue?

From 40f6242099e70f35fe7401c848c4fc096e779587 Mon Sep 17 00:00:00 2001
From: Jose Martins <[email protected]>
Date: Wed, 30 Oct 2024 18:48:25 +0000
Subject: [PATCH] fix(riscv): add low order bits to faulting address

Signed-off-by: Jose Martins <[email protected]>
---
 src/arch/riscv/sync_exceptions.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/arch/riscv/sync_exceptions.c b/src/arch/riscv/sync_exceptions.c
index e44c5414..7b7ca3fc 100644
--- a/src/arch/riscv/sync_exceptions.c
+++ b/src/arch/riscv/sync_exceptions.c
@@ -80,7 +80,7 @@ static inline bool is_pseudo_ins(uint32_t ins)
 
 static size_t guest_page_fault_handler(void)
 {
-    vaddr_t addr = csrs_htval_read() << 2;
+    vaddr_t addr = (csrs_htval_read() << 2) | (csrs_stval_read() & 0x3);
 
     emul_handler_t handler = vm_emul_get_mem(cpu()->vcpu->vm, addr);
     if (handler != NULL) {
-- 
2.34.1

[ZhongkaiXu]

That's exactly what I needed; it solves my issue. Thank you for your patience and response!

[josecm]

I'm creating a PR with this fix then!

[danielRep]

FYI, this change #190 is already in the mainline.

[ZhongkaiXu]

I added a PCIe node to the VM's device tree, but I didn’t map the address regionfor reg property which is the ECAM, in the page table of the VM. When the VM probes the PCIe device and accesses the ECAM, it traps into the hypervisor. I added an MMIO handler similar to the instruction parsing process of the PLIC handler, where the ECAM handler does not perform any special operations—it simply accesses and returns values from the hypervisor. The expected effect should be the same as if this region were directly pass through to the VM (without trapping into the hypervisor).

The problem is that I found that for each access to the configuration space corresponding to a specific BDF, the access instruction with offset 0x3d is always missing. From the hypervisor side, this access instruction does not appear, and from the Linux side, the access result is always 0. I can see the VM's access to other offsets in the hypervisor, and the hypervisor returns the correct results, but certain access instructions for specific offsets are consistently missing, which is quite confusing.

The result is that when I run the command lspci -vvv in Linux, I can see that interrupts are not correctly assigned for each device. This is because the content at offset 0x3D corresponds to the interrupt-pin register, which represents the type of INTx used by the device. However, when I directly map the ECAM region to the VM and run the same command, the interrupts are correctly assigned, and I can observe the VM's access to offset 0x3D in the QEMU trace pci_* logs.

Below is the PCIe node I added to the VM's device tree, which I dumped from qemu-system-riscv64 in qemu-9.0.1:

pci@30000000 {
    interrupt-map-mask = <0x1800 0x00 0x00 0x07>;
    interrupt-map = <0x00 0x00 0x00 0x01 0x09 0x20 0x00 0x00 0x00 0x02 0x09 0x21 0x00 0x00 0x00 0x03 0x09 0x22 0x00 0x00 0x00 0x04 0x09 0x23 0x800 0x00 0x00 0x01 0x09 0x21 0x800 0x00 0x00 0x02 0x09 0x22 0x800 0x00 0x00 0x03 0x09 0x23 0x800 0x00 0x00 0x04 0x09 0x20 0x1000 0x00 0x00 0x01 0x09 0x22 0x1000 0x00 0x00 0x02 0x09 0x23 0x1000 0x00 0x00 0x03 0x09 0x20 0x1000 0x00 0x00 0x04 0x09 0x21 0x1800 0x00 0x00 0x01 0x09 0x23 0x1800 0x00 0x00 0x02 0x09 0x20 0x1800 0x00 0x00 0x03 0x09 0x21 0x1800 0x00 0x00 0x04 0x09 0x22>;
    ranges = <0x1000000 0x00 0x00 0x00 0x3000000 0x00 0x10000 0x2000000 0x00 0x40000000 0x00 0x40000000 0x00 0x40000000 0x3000000 0x04 0x00 0x04 0x00 0x04 0x00>;
    reg = <0x00 0x30000000 0x00 0x10000000>;
    dma-coherent;
    bus-range = <0x00 0xff>;
    linux,pci-domain = <0x00>;
    device_type = "pci";
    compatible = "pci-host-ecam-generic";
    #size-cells = <0x02>;
    #interrupt-cells = <0x01>;
    #address-cells = <0x03>;
};