Instruction for Using Kernel Modules/Headers Distributed as VHD in WSL2

[Locietta]

I noticed that the WSL2 kernel team decided to distributes kernel modules as VHD/VHDX files in the latest 6.6.y kernel releases, along with instructions for building modules.vhdx. I am maintaining a rolling wsl2 kernel project and would like to know:

1. How to properly use the modules.vhdx (e.g., via a configuration toggle in the .wslconfig file?).
2. Whether kernel headers can similarly be distributed as VHD/VHDX files.

Use Case Context

I currently use Scoop to distribute my kernel releases. Distributing both kernel modules and headers as VHD/VHDX files would make it much easier to integrate headers & modules into scoop update utility. So users don't have to manually update headers and modules per distro per update.

Thanks for your insights!

[github-actions]

Logs are required for review from WSL team

If this a feature request, please reply with '/feature'. If this is a question, reply with '/question'.
Otherwise please attach logs by following the instructions below, your issue will not be reviewed unless they are added. These logs will help us understand what is going on in your machine.

How to collect WSL logs

Download and execute collect-wsl-logs.ps1 in an administrative powershell prompt:

Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/WSL/master/diagnostics/collect-wsl-logs.ps1" -OutFile collect-wsl-logs.ps1
Set-ExecutionPolicy Bypass -Scope Process -Force
.\collect-wsl-logs.ps1

The script will output the path of the log file once done.

If this is a networking issue, please use collect-networking-logs.ps1, following the instructions here

Once completed please upload the output files to this Github issue.

Click here for more info on logging
If you choose to email these logs instead of attaching to the bug, please send them to [email protected] with the number of the github issue in the subject, and in the message a link to your comment in the github issue and reply with '/emailed-logs'.

[Locietta]

/question

[github-actions]

Diagnostic information

Found '/question', adding tag 'question'

[affirm-bats-yodel]

I don't really know this is the right way to do, but...

1. Build the kernel via Instructions on WSL2-Linux-Kernel/README.md
2. copy bzImage and *.vhdx that contains a loadable modules to the outside of the WSL Environment.
3. mount *.vhdx to WSL Environment using: wsl --mount --vhd *.vhdx --name kernel-modules and check path /mnt/wsl/kernel-modules is available on WSL Environment.
4. copy /mnt/wsl/kernel-modules/lib/modules/6.6.75.1-microsoft-standard-WSL2+/ to /usr/lib/modules/
5. update .wslconfig to point newer kernel path.
6. wsl --shutdown and wsl to restart.
7. $ uname -r to verify the kernel has been change to: 6.6.75.1-microsoft-standard-WSL2+
8. $ ls /usr/lib/modules/$(uname -r) to verify modules are available.

Disclaimer

1. Module should've copied to /usr/lib/modules/ when Environment changed.

[Locietta]

I don't really know this is the right way to do, but...

1. Build the kernel via Instructions on WSL2-Linux-Kernel/README.md
2. copy bzImage and *.vhdx that contains a loadable modules to the outside of the WSL Environment.
3. mount *.vhdx to WSL Environment using: wsl --mount --vhd *.vhdx --name kernel-modules and check path /mnt/wsl/kernel-modules is available on WSL Environment.
4. copy /mnt/wsl/kernel-modules/lib/modules/6.6.75.1-microsoft-standard-WSL2+/ to /usr/lib/modules/
5. update .wslconfig to point newer kernel path.
6. wsl --shutdown and wsl to restart.
7. $ uname -r to verify the kernel has been change to: 6.6.75.1-microsoft-standard-WSL2+
8. $ ls /usr/lib/modules/$(uname -r) to verify modules are available.

Disclaimer

1. Module should've copied to /usr/lib/modules/ when Environment changed.

Yeh, this works. We can also add script containing wsl --mount --vhd xxx.vhdx to task scheduler (triggered by system startup and Log: Microsoft-Windows-Hyper-V-Worker/Admin like this) so it will be automatically mounted whenever WSL boots. And on distro side, adding a systemd service that creates symlink from /mnt/wsl/... to the proper location during boot.

Maybe something like this👇

[Unit]
Description=Create kernel modules symlink
After=local-fs.target
RequiresMountsFor=/mnt/wsl/kernel-modules

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'rm -rf /lib/modules/* && ln -sfn /mnt/wsl/kernel-modules/lib/modules/$(uname -r) /lib/modules/$(uname -r)'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

I just wonder if there's any option in .wslconfig that might handle all of this under the hood.

[nunix]

I also think there's a new setting coming soon.

In the meantime, I went a slightly different route from:

1. Build the modules and the final path to vhdx: sudo $HOME/wsl2-linux-kernel/Microsoft/scripts/gen_modules_vhdx.sh "$HOME/wsl2-linux-kernel/modules/lib/modules/6.6.75.1-microsoft-standard-WSL2+" modules.vhdx
2. Copy the vhdx to Windows FS
3. Create a directory that will serve as mountpoint: sudo mkdir /usr/lib/modules/6.6.75.1-microsoft-standard-WSL2+
4. Add a bind mount in FSTAB: /mnt/wsl/kernel-modules /usr/lib/modules/6.6.75.1-microsoft-standard-WSL2+ none bind 0 0
5. Exit the distro and shutdown WSL: wsl --shutdown
6. Edit $env:USERPROFILE\.wslconfig to point to the new kernel
7. Mount the vhdx: wsl --mount --vhd --name kernel-modules modules.vhdx
8. Launch your distro and check the mountpoints: mount | grep modules # 2 mountpoints should appear
9. Try to load a module and check its status: sudo modprobe kvm && lsmod

This solution, like the others, requires that we always mount the disk first, and if we don't repeat the mountpoint + FSTAB steps, we'll need to always start "modules distro" first, and load there the needed modules too. I would say quite cumbersome.

So repeating mountpoint + FSTAB steps is recommended if you have multiple distros.
Also, in comparison to copying the modules to /usr/lib/modules/$(uname -r) directory, this makes it more "light", however the copy method as one (very?) big advantage: the modules load at distro boot, while the bind mount allows the modules to be mounted on demand only (no preload).

Hope this brings some light on the current workaround and I truly think/hope there's some .wslconfig option appearing soon.

PS: there's an intrusive way of modifying the kernel and loading the modules "like native", however is not documented nor encouraged ... but possible ;)

The Corsair

[affirm-bats-yodel]

On Kernel v5 (5.15.167.4-microsoft-standard-WSL2), the modules are mounted via overlay, seems like the mount is issued on initrd side that I cannot find any source of lowerdir

mount | grep modules

none on /lib/modules/5.15.167.4-microsoft-standard-WSL2 type overlay (rw,nosuid,nodev,noatime,lowerdir=/modules,upperdir=/lib/modules/5.15.167.4-microsoft-standard-WSL2/rw/upper,workdir=/lib/modules/5.15.167.4-microsoft-standard-WSL2/rw/work)

[zcobol]

@affirm-bats-yodel the actual modules are in the system.vhd located at C:\Program Files\WSL\system.vhd. You can start wsl --system to launches a shell for the system distribution.