Orchestration of various Virtio devices (reopened) #41

torgeiru · 2025-09-28T15:08:30Z

Orchestration of VirtioFS, VirtioPMEM and VirtioCON (for testing).
Dependency used for testing upcoming modern Virtio, VirtioFS and VirtioPMEM.

VirtioFS: Adding necessary qemu_args and starting VirtioFSD. Installed pinned VirtioFSD as a dependency. VirtioCON: Redirecting output to a given path for console VirtioPMEM: Using a image path for the persistent memory

Removed debug stuff

Future enhancement is having a schema option for printing VirtioFSD debug messages.

alfreb

Virtiofs parts look great! With the few minor comments. Pmem and virtioconsole I have to do some more reading to fully understand - I'll do that!

alfreb · 2025-10-02T13:21:33Z

default.nix

+
+  virtiofsd_pinned ? import (fetchTarball {
+    name = "nixpkgs-pinned-for-virtiofsd";
+    url = "https://github.com/NixOS/nixpkgs/archive/13e8d35b7d6028b7198f8186bc0347c6abaa2701.tar.gz";


I don't think we need this pin - we can inherit the toplevel pin, typically from IncldueOS' pinned.nix

We can easily remove it. VirtioFSD should work with the latest nixpkgs. We need to check for breaking changes
when updating nixpkgs in the future.

alfreb · 2025-10-02T13:22:32Z

vmrunner/vm.schema.json

+          "type" : "number"
+        }
+      }
+    },


So new device types for virtio console, fs and pmem - nice!

alfreb · 2025-10-02T13:23:41Z

vmrunner/vmrunner.py

+        virtiofsd_args = ["virtiofsd", "--socket", socket, "--shared-dir", shared, "--sandbox", "none"]
+        self._virtiofsd_proc = subprocess.Popen(virtiofsd_args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) # pylint: disable=consider-using-with
+
+        time.sleep(0.1)


We don't want to sleep - wait for a specific event if there's a race. You poll just below - there's probably a way to find out if the process is ready

I reckon we can generate the socket and then wait for it to show up (the socket path).

Found this TODO in the CI script for virtio-fs (virtio-fs-ci):

# TODO This is racy. Wait for the listening message on virtiofsd stderr instead. while [ ! -e vhost-fs.sock ]; do sleep 0.1 done

May have changed because that belongs to an old VirtioFSD, but helpful for knowing when process is ready.

alfreb · 2025-10-02T14:08:07Z

vmrunner/vmrunner.py

+
+        virtiofs_args = []
+        if "virtiofs" in self._config:
+            socket = "/tmp/virtiofsd.sock"


This will be problematic when we run multiple vms simultaneously; I think you can just have python create a tempfile here

Do you think it would be okay to use something like mktemp and unlink the socket just for getting a conflict-free name?

Perhaps you could use Python's tempfile to create a temporary directory and put the socket there? Then you avoid the race and could get automatic cleanup when the script exits

That seems like a clean way of achieving the wanted result.

alfreb · 2025-10-02T14:26:40Z

vmrunner/vmrunner.py

        mem_arg = []
        if "mem" in self._config:
-            mem_arg = ["-m", str(self._config["mem"])]
+            mem_arg = ["-m", f"size={self._config["mem"]},maxmem={64}G"]


1TB should be the least we support 😁

…o, made it possible with multiple pmem devices per VM

torgeiru · 2025-10-04T15:09:24Z

Ok. I have done the requested changes and made it possible to have multiple PMEM devices per VM.
I got an error with 1024G and opted with 1000G because of limited BUS width (QEMU complained).

I have tested and it worked. The sockets are cleaned up automatically from the tip ^^·

The functionality can be tested by doing this:

Clone the Virtio branch of my IncludeOS fork together this vmrunner and my diskbuilder
Run tests modern_virtio/integration (VirtioCON), fs/virtiofs (VirtioFS) and fs/virtiopmem (VirtioPMEM) like this:

nix-shell --argstr unikernel <test-path> --argstr vmrunner <vmrunner-realpath> --argstr diskbuilder <diskbuilder-realpath> --run "./test.py"

Note: This enables modern Virtio console and not legacy. We have a legacy VirtioCON driver (legacy boolean option possibly). In the future, we must consider multiple backends like path and socket. Socket could be hooked with socat for interactivity.

…s anyway

mazunki · 2025-10-10T13:08:45Z

$ nix-shell \
    --argstr unikernel ./test/fs/integration/virtiofs  \
    --argstr vmrunner ~/repos/includeos/vmrunner       \
    --argstr diskbuilder ~/repos/includeos/diskbuilder \
    --run ./test.py 

$ nix-shell                                                  \
        --argstr unikernel ./test/modern_virtio/integration  \
        --argstr vmrunner ~/repos/torgeiru/vmrunner          \
        --argstr diskbuilder ~/repos/torgeiru/diskbuilder    \
        --run "./test.py"

both work fine on NixOS for me, but fail on Gentoo with the following errors after fixing pthread-related linking issues (see includeos/IncludeOS#2310):

VirtioFS test (test/fs/integration/virtiofs): libc++abi: terminating due to uncaught exception of type os::detail::Machine_access_error: Requested machine parts not found,
VirtioCON test (./test/modernvirtio/integration):<vm> /home/maz/repos/includeos/IncludeOS/test/modern_virtio/integration/service.cpp:36:main Expects failed: found_device == true

The virtio-pmem test fails on both machines for me:

<vm>       [ FAT32 ] Running tests for FAT32
<vm>                 [x] VirtioBlk disk created
<vm>                 [x] Disk not empty
<vm>                 [x] Disk size is 2048 sectors
<vm> libc++abi: terminating due to uncaught exception of type fs::Disk::Err_not_mounted: Filesystem not ready - make sure to init_fs before accessing
<vm> 
<vm>  **** PANIC ****
<vm> CPU: 0, Reason: TKILL called
<vm> Heap is at: 0x5dd000 / 0x5fd6fff  (diff=94347263)
<vm> Heap area: 1924 / 94059 Kb (allocated 1840 kb)
<vm> Total memory use: ~38% (38539264 of 100495360 b)
<vm> *** Found 0 plugin constructors:
<vm> 
<vm> *** Backtrace:
<vm> symtab or strtab is empty, indicating image may be stripped
<vm> [0] 0x00000000002013b3 + 0x000: 0x2013b3
<vm> [1] 0x0000000000265462 + 0x000: 0x265462
<vm> [2] 0x00000000002ca18d + 0x000: 0x2ca18d
<vm> [3] 0x000000000026f59a + 0x000: 0x26f59a
<vm> [4] 0x000000000026f4fc + 0x000: 0x26f4fc
<vm> [5] 0x00000000002030f2 + 0x000: 0x2030f2
<vm> [6] 0x0000000000269a66 + 0x000: 0x269a66
<vm> [7] 0x000000000021a88d + 0x000: 0x21a88d
<vm> [8] 0x00000000002c582a + 0x000: 0x2c582a
<vm> [9] 0x000000000021c45a + 0x000: 0x21c45a

torgeiru · 2025-10-10T13:30:22Z

Yes. I forgot to push private stuff will do. Changed something so that I could have multiple PMEM devices per VM.

mazunki · 2025-10-10T15:14:02Z

so that I could have multiple PMEM devices per VM.

Sounds like a priority! Do you think you could write a mini tutorial on how one could load multiple files as an example? Perhaps a directory or repository which could potentially be merged into includeos:examples/?

alfreb · 2025-10-14T12:13:50Z

I'm getting similar errors as @mazunki on Ubuntu, and the issues are also picked up by ./test.sh, which we need to stay green, manually until we have them in CI:

alfreb@testctrl:~/repos/torgeiru/IncludeOS$ export CCACHE_FLAG="--arg withCcache true"
alfreb@testctrl:~/repos/torgeiru/IncludeOS$ ./test.sh 
...
🚧 Step 8.4
📂 ./test/fs/integration/virtiofs/
⚙️  Running this command:
nix-shell --pure --arg withCcache true --argstr unikernel ./test/fs/integration/virtiofs/ --run ./test.py

...
libc++abi: terminating due to uncaught exception of type os::detail::Machine_access_error: Requested machine parts not found

👷⛔ Step 8.4 failed (nix-shell --pure --arg withCcache true --argstr unikernel ./test/fs/integration/virtiofs/ --run ./test.py)

🚧 Step 8.5
📂 ./test/fs/integration/virtiopmem/
⚙️  Running this command:
nix-shell --pure --arg withCcache true --argstr unikernel ./test/fs/integration/virtiopmem/ --run ./test.py
...
libc++abi: terminating due to uncaught exception of type os::detail::Machine_access_error: Requested machine parts not found
...
👷⛔ Step 8.5 failed (nix-shell --pure --arg withCcache true --argstr unikernel ./test/fs/integration/virtiopmem/ --run ./test.py)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

👷🧰 2 / 8 steps failed. There's some work left to do. 🛠  

Failed tests:
step 7.3: nix-shell --pure --arg withCcache true --argstr unikernel ./test/net/integration/dns/ --run ./test.py
step 7: 
step 8.4: nix-shell --pure --arg withCcache true --argstr unikernel ./test/fs/integration/virtiofs/ --run ./test.py
step 8.5: nix-shell --pure --arg withCcache true --argstr unikernel ./test/fs/integration/virtiopmem/ --run ./test.py
step 8:

Note that several networking related tests will fail until you follow the network config instructions found in shell.nix:
https://github.com/includeos/IncludeOS/blob/main/shell.nix#L80
(they populate with actual nix paths when you run it). Also, the DNS test here is flaky - so it some times fail - that's unrelated. Made an issue for that here includeos/IncludeOS#2313

Let me know when the virtio branch has passing tests so I can rerun this.

mazunki · 2025-10-14T12:16:47Z

--arg withCcache true

Is this good enough before merging PRs? I use it myself for quick iteration, but thought traces of old stuff could mess things up.

alfreb · 2025-10-14T12:18:30Z

--arg withCcache true

Is this good enough before merging PRs? I use it myself for quick iteration, but thought traces of old stuff could mess things up.

No, in CI I would not have this enabled. That said - I think it can give false negatives, but I don't expect false positives. So if it fails with CCache it has to be fixed, but passing with ccache does not necessarily imply it's reproducible.

torgeiru · 2025-10-14T12:44:20Z

Okay. I know that test.sh fails because it assumes that my vmrunner changes are upstream, they are not (until this PR gets merged). It says machine parts are not available indicating this (or found_device Expects failure), it doesn't expose the device to the VM. Mazunki ran the tests using this vmrunner (--argstr vmrunner) and it passes (except for VirtioPMEM).

@mazunki You got requested parts not found of gentoo, could you dump other previous errors and the command you used?

The VirtioPMEM stuff I need to investigate further (diskbuilder may not be reproducible and buffer overflow issues). To test VirtioPMEM stuff, create a raw file and write to it (have demoed this for Mazunki sometime ago). Use strings on the image to observe the string. The VirtioPMEM test in Virtio demonstrates the functionality by grabbing a DAX driver object and then executes some start_addr method. Write to this and flush using the driver flush method (to be certain that changes are persistent).

Alternatively, do a loop mount and build the image (dd 1MiB and format) according to the test and modify as you will. I can modify the test to do this instead of diskbuilder on request (loop mounting requires sudo). I think I will be able to fix diskbuilder but I have so many other things to think about hehe.

TLDR:
Comment out the new Virtio tests in the test.sh and manually type nix-shell --argstr unikernel <test-path> --argstr vmrunner <vmrunner-path> --argstr diskbuilder <diskbuilder-path> (diskbuilder argstr may not be needed because upstream now but unfortunately broken / non-reproducible).

torgeiru · 2025-10-14T13:51:28Z

Ran this using nix-shell --argstr unikernel ./test/fs/integration/virtiopmem/ --argstr vmrunner /home/littledragon/Skrivebord/vmrunner --run ./test.py.

<vm>       [ FAT32 ] Running tests for FAT32
<vm>                 [x] VirtioBlk disk created
<vm>                 [x] Disk not empty
<vm>                 [x] Disk size is 2048 sectors
<vm>         [ FAT ] Initializing FAT32 filesystem
<vm>                 [ofs=0  size=2048 (1048576 bytes)]
<vm> 
<vm>                 [x] Filesystem auto-initializedd
<vm>                 [x] Filesystem recognized as FAT32
<vm>                 [x] Listing root directory
<vm>                 [x] Exactly four ents in root dir
<vm>                 [x] Ent is a file
<vm>                 [x] Ents name is 'banana.txt'
<vm>       [ FAT32 ] Remounting disk.
<vm>                 [x] Disk not empty
<vm>                 [x] Disk size is 1048576 bytes
<vm>         [ FAT ] Initializing FAT32 filesystem
<vm>                 [ofs=0  size=2048 (1048576 bytes)]
<vm> 
<vm>                 [x] Filesystem mounted on VBR1
<vm>       [ FAT32 ] Shallow banana
<vm>                 [x] Stat /banana.txt
<vm>                 [x] Stat file in root dir
<vm>                 [x] Entity is file
<vm>                 [x] Entity is not directory
<vm>                 [x] Name is 'banana.txt'
<vm>       [ FAT32 ] Read file
<vm>                 [x] read_file: Read /banana.txt asynchronously
<vm>      ____                           ___
<vm>     |  _ \  ___              _   _.' _ `.
<vm>  _  | [_) )' _ `._   _  ___ ! \ | | (_) |    _
<vm> |:;.|  _ <| (_) | \ | |' _ `|  \| |  _  |  .:;|
<vm> |   `.[_) )  _  |  \| | (_) |     | | | |.',..|
<vm> ':.   `. /| | | |     |  _  | |\  | | |.' :;::'
<vm>  !::,   `-!_| | | |\  | | | | | \ !_!.'   ':;!
<vm>  !::;       ":;:!.!.\_!_!_!.!-'-':;:''    '''!
<vm>  ';:'        `::;::;'             ''     .,  .
<vm>    `:     .,.    `'    .::... .      .::;::;'
<vm>      `..:;::;:..      ::;::;:;:;,    :;::;'
<vm>        "-:;::;:;:      ':;::;:''     ;.-'
<vm>            ""`---...________...---'""
<vm> 
<vm>                 [x] Correct shallow banana
<vm>       [ FAT32 ] Deep banana
<vm>                 [x] Stat /dir1/dir2/dir3/dir4/dir5/dir6/banana.txt
<vm>                 [x] Stat file in deep dir
<vm>                 [x] Entity is file
<vm>                 [x] Entity is not directory
<vm>                 [x] Name is 'banana.txt'
<vm>       [ FAT32 ] Read inside stat
<vm>                 [x] read: Read /dir1/dir2/dir3/dir4/dir5/dir6/banana.txt asynchronously
<vm>                 [x] Correct deep fried banana
<vm>       [ FAT32 ] SUCCESS

[ SUCCESS ] <VMRunner> All tests passed

The ^^ is the VirtioPMEM test (copy of the VirtioBLK test) running (no need for the diskbuilder argstr because upstream). The FAT driver understands the image and therefore it passes. Still, it is interesting when the diskbuilder issues appear. I am running debian on my overdue laptop btw.

I will investigate these things, but I am currently pretty darn close to a breakthrough with VirtioFS DAX (have succesfully built legacy QEMU and VirtioFSD with DAX patches, hellish to find because no documentation).

torgeiru · 2025-10-16T14:28:35Z

Should I close this PR for now and send a smaller PR next time (one device per PR). Will postpone the Virtio integrations?

alfreb · 2025-10-16T15:30:35Z

Should I close this PR for now and send a smaller PR next time (one device per PR). Will postpone the Virtio integrations?

No, please leave it open, I'll try to test it without your IncludeOS changes.

torgeiru · 2025-10-16T16:44:06Z

Things will be easier with vmrunner changes upstream :) Promise to split up IncludeOS PRs for a better review experience

mazunki · 2025-10-17T16:33:19Z

@mazunki You got requested parts not found of gentoo, could you dump other previous errors and the command you used?

Did you figure this out? If not, includeos/IncludeOS#2316 might help you.

alfreb · 2025-10-18T09:31:28Z

nix-shell --argstr unikernel ./test/fs/integration/virtiopmem/ --argstr vmrunner /home/littledragon/Skrivebord/vmrunner --run ./test.py

This worked for me now - thanks!

alfreb

Thanks! I got the virtiopmem example to work now, and tested both other options manually. I can see virtiocon come up as a recognized PCI device and virtiofs spawns a virtiofsd daemon when the shared folder exists.

                +--[ Simple comm. controller , VirtIO (0x1043) ]
                +--[ Storage controller, VirtIO (0x105a) ]

These two entries show up in the PCI discovery when virtiofs and virtiocon entries are added to vm.json, and I think that's enough to validate these changes.

alfreb · 2025-10-18T09:45:14Z

vmrunner/vmrunner.py

+
+        info("Successfully started VirtioFSD!")
+
+        while not os.path.exists(socket):


This will hang with no helpful information if the shared folder doesn't exist:

* <VMRunner>: Booting ./fs_virtiopmem.elf.bin directly without bootloader (multiboot / -kernel args) * <VMRunner>: Successfully started VirtioFSD!

You can reproduce this by adding this to vm.json

"virtiofs" : { "shared" : "./this_is_shared" }

And then booting with boot, when ./this_is_shared does not exist. After creating the directory this_is_shared, the unikernel starts. So we should check if the provided shared folder exists and is a folder, and if not error out. But since it works in the happy path, and only takes effect if you actually add a virtiofs entry to vm.json, I'm happy to merge this - just add the check in a separate PR.

This is because poll() returns immediately and error might not have been raised before poll() have returned (this is a race). I quickly tried to add a time.sleep(0.1) and it didn't hang because poll then returns the error code and the failure exception is raised. We need another signal. Any suggestions on how to solve this? I know that tests use a "SUCCESS" to indicate a successful test. I think a similar pattern could be used here (error is printed early on error, a watch).

torgeiru · 2025-10-18T11:50:00Z

Thank you!

Did you try to run the ./test.sh now for the Virtio branch? Should work now when vmrunner stuff upstream.

I have do some warning cleanup, constexpr and stuff but the queue implementation I think is solid and clean.

torgeiru added 6 commits September 27, 2025 03:26

Added schema and logic for handling several Virtio devices

cbd3f8f

VirtioFS: Adding necessary qemu_args and starting VirtioFSD. Installed pinned VirtioFSD as a dependency. VirtioCON: Redirecting output to a given path for console VirtioPMEM: Using a image path for the persistent memory

Added a maxmem option for mem_arg

9b45ce0

Finished vmrunner Virtio part hopefully

39a0ec2

Removed debug stuff

Making so that VirtioFSD doesn't say anything

fab5ab3

Future enhancement is having a schema option for printing VirtioFSD debug messages.

Passhtrough pinned VirtioFSD

f3d06e1

Removed mistake

acdfb89

alfreb requested changes Oct 2, 2025

View reviewed changes

torgeiru added 2 commits October 3, 2025 21:32

Removing duplicated pinned nixpkgs

2bc9aea

Fixed random tmp file for VirtioFS socket and removed racy sleep. Als…

79eaede

…o, made it possible with multiple pmem devices per VM

torgeiru added 2 commits October 4, 2025 23:59

Removed redundant ifs. Schema requirements raises missing requirement…

5657b7b

…s anyway

Removed one more redundant exception raise

afffb7d

alfreb approved these changes Oct 18, 2025

View reviewed changes

alfreb merged commit 02b9df4 into includeos:master Oct 18, 2025

torgeiru deleted the virtio branch October 18, 2025 12:05


		info("Successfully started VirtioFSD!")

		while not os.path.exists(socket):

Uh oh!

Orchestration of various Virtio devices (reopened) #41

Orchestration of various Virtio devices (reopened) #41

Uh oh!

Conversation

torgeiru commented Sep 28, 2025

Uh oh!

alfreb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torgeiru Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torgeiru Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torgeiru Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torgeiru commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mazunki commented Oct 10, 2025

Uh oh!

torgeiru commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mazunki commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alfreb commented Oct 14, 2025

Uh oh!

mazunki commented Oct 14, 2025

Uh oh!

alfreb commented Oct 14, 2025

Uh oh!

torgeiru commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

torgeiru commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

torgeiru commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alfreb commented Oct 16, 2025

Uh oh!

torgeiru commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mazunki commented Oct 17, 2025

Uh oh!

alfreb commented Oct 18, 2025

Uh oh!

alfreb left a comment

Choose a reason for hiding this comment

Uh oh!

alfreb Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

torgeiru Oct 4, 2025 •

edited

Loading

torgeiru Oct 17, 2025 •

edited

Loading

torgeiru Oct 4, 2025 •

edited

Loading

torgeiru commented Oct 4, 2025 •

edited

Loading

torgeiru commented Oct 10, 2025 •

edited

Loading

mazunki commented Oct 10, 2025 •

edited

Loading

torgeiru commented Oct 14, 2025 •

edited

Loading

torgeiru commented Oct 14, 2025 •

edited

Loading

torgeiru commented Oct 16, 2025 •

edited

Loading

torgeiru commented Oct 16, 2025 •

edited

Loading

alfreb Oct 18, 2025 •

edited

Loading

torgeiru Oct 18, 2025 •

edited

Loading

torgeiru commented Oct 18, 2025 •

edited

Loading