Emulate GICv3 on macos

[p1-0tr]

This PR adds support for GICv3 emulation on mac hosts.

Before merging I need to:

• move GICv3 logic which made it's way into the hvf package
• either delete the GICv2 implementation or provide some way of switching between v2 and v3
• do some more testing on platforms other than aarch64 macOS

Any feedback will be greatly appreciated :)

[slp]

Took an initial look at the PR and so far looks pretty good. A couple of comments:

• Please run cargo fmt on the code before committing, or configure your editor to run rustfmt after saving the each file.
• It's fine to drop GICv2. We don't need to be compatible guests not supporting GICv3 and I'd prefer to avoid keeping dead code around.
• Don't worry about GICv3 on non-macOS targets. We use the in-kernel implementation provided by KVM on Linux.

Thanks for the contribution!

[tylerfanelli]

Seems there's a few clippy warnings in the CI, could you address those?

[p1-0tr]

Seems there's a few clippy warnings in the CI, could you address those?

Should be all cleaned up now. I checked on arm64 macOS and x86 Linux (... and arm64 Linux 😅).

[tylerfanelli]

Thanks! All checks passed. Do I need to do anything special to test, or is it as simple as pulling your branch and running on macOS?

[p1-0tr]

Do I need to do anything special to test, or is it as simple as pulling your branch and running on macOS?

Yeah, should be as simple as pulling the branch and running on macOS. The simplest smoke test would be to run the chroot_vm example, and inspect the kernel log:

[p1-0tr] examples : DYLD_LIBRARY_PATH=../../libkrunfw ./chroot_vm rootfs_ubuntu /bin/sh
# dmesg | grep GIC
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: GICv3 features: 16 PPIs, RSS
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000003ff50000
[    0.006067] GICv3: CPU1: found redistributor 1 region 0:0x000000003ff70000
[    0.006341] GICv3: CPU2: found redistributor 2 region 0:0x000000003ff90000
[    0.006597] GICv3: CPU3: found redistributor 3 region 0:0x000000003ffb0000

[tylerfanelli]

Thanks, I've tried testing but noticed you're linking with libkrunfw. On macOS, the libkrun-efi flavor is built, and this traditionally does not link with libkrunfw. In which way are you building?

[p1-0tr]

In which way are you building?

sh build_on_krunvm.sh
make

otool -L target/release/libkrun.1.9.8.dylib 
target/release/libkrun.1.9.8.dylib:
    /Users/piotrs/repositories/libkrun/target/release/deps/libkrun.dylib (compatibility version 0.0.0, current version 0.0.0)
    /System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor (compatibility version 1.0.0, current version 210.2.2)
    libkrunfw.4.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)

To build the EFI variant:

EFI=1 make

Though the chroot_vm example links libkrun not libkrun-efi, so I assume you want to use the boot_efi example. In any case, both variants should work so long as the kernel you boot has the GICv3 driver (CONFIG_ARM_GIC_V3=y).

[slp]

@p1-0tr I've started tested this one (on an M3 Pro with macOS 15.1.1), but with both the EFI and regular flavors the kernel is hanging on boot for me if the microVM has 4 or more vCPUs (works fine with 1 to 3 vCPUs). Is this working well for you with >3 vCPUs?

(I haven't started debugging it yet as I thought I should ask first)

[p1-0tr]

Is this working well for you with >3 vCPUs?

I'll re-test first thing tomorrow morning. I am pretty sure the changes work with more than 4 CPUs (I had the code running with 32 cores on an 8 core machine, after that over provisioning seems to go bad), but I've not confirmed with the chroot_vm example (my normal test environment is a bit weird, so I may have missed something which causes this issue).

[p1-0tr]

@slp Hmm, the chroot_vm example seems to boot fine (I am using libkrunfw v4.4.2).

[p1-0tr] examples : DYLD_LIBRARY_PATH=../../libkrunfw ./chroot_vm rootfs_ubuntu /bin/sh
# cat /proc/cpuinfo
processor	: 0
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x61
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

processor	: 1
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x61
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

processor	: 2
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x61
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

processor	: 3
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x61
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

processor	: 4
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x61
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

processor	: 5
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x61
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

processor	: 6
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x61
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

processor	: 7
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x61
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

I didn't change much about the example itself:

 diff --git a/examples/chroot_vm.c b/examples/chroot_vm.c
index 86321ae..40aa2fe 100644
--- a/examples/chroot_vm.c
+++ b/examples/chroot_vm.c
@@ -234,7 +234,7 @@ int main(int argc, char *const argv[])
     }
 
     // Configure the number of vCPUs (1) and the amount of RAM (512 MiB).
-    if (err = krun_set_vm_config(ctx_id, 4, 4096)) {
+    if (err = krun_set_vm_config(ctx_id, 8, 4096)) {
         errno = -err;
         perror("Error configuring the number of vCPUs and/or the amount of RAM");
         return -1;

I'm on an M1 mbp running macOS 14.6.1 . How do you test the EFI variant (I've not had much luck with boot_efi)?

[slp]

The problem was that IPIs weren't working properly, as a result of a mismatch of the vCPU IDs between HVF and the userspace thread. With #242 in, this PR works fine for me.

Could you please rebase it? I think we should be able to merge it afterwards.

[slp]

Successfully tested on M1 and M3, both the regular and the EFI flavors.

Thanks for the contribution!