Skip to content

Run bhyve on EspressoBin

Jump to bottom
Andrei Martin edited this page Sep 5, 2022 · 8 revisions

Setup

The setup consists in two repositories: the FreeBSD source code and the arm64 utils, which contains the building scripts, config files and auxiliary files (e.g. Device Trees, kernel configs). The arm64 bhyve can be found on the projects/bhyvearm64 branch.

root@freebsd:~# mkdir demo
root@freebsd:~# cd demo
root@freebsd:~/demo# git clone https://github.com/FreeBSD-UPB/freebsd-src freebsd
root@freebsd:~/demo# cd freebsd/
root@freebsd:~/demo/freebsd# git checkout projects/bhyvearm64-v11Jan2021
root@freebsd:~/demo/freebsd# cd ..
root@freebsd:~/demo# git clone https://github.com/FreeBSD-UPB/bhyvearm64-utils
root@freebsd:~/demo# cd bhyvearm64-utils/
root@freebsd:~/demo/bhyvearm64-utils# git checkout projects/bhyvearm64-v11Jan2021

To build the host, the config/arm64/host.json has to be updated:

  • kernconf should be GENERIC
  • update the paths to root/demo/<...>
  • start building the world and the host kernel

To build the guest, the config/arm64/guest_kernel.json has to be updated:

  • kernconf should be ESPRESSOBIN_GENERIC
  • update the paths to root/demo/<...>
  • start building the guest kernel

Cross-compiling the FreeBSD guest and host kernels

Host

To cross-compile the FreeBSD world and host you have to load the filemon module and run the build.sh script with the host.json config file and buildworld/buildkernel build parameter.

root@freebsd:~/demo/bhyvearm64-utils# cp freebsd/sys/arm64/conf/GENERIC ../freebsd/sys/arm64/conf/
root@freebsd:~/demo/bhyvearm64-utils# kldload filemon
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/host.json --build buildworld
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/host.json --build buildkernel
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/host.json --build installworld
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/host.json --build installkernel

Guest

In order to build the guest some files are needed, the kernelconfig and the device tree. The first step is to update the MFS_IMAGE variable in the ESPRESSOBIN_GUEST kernel config, then to copy ESPRESSOBIN_GUEST and armada-3720-espressobin-guest.dts in the FreeBSD repository.

MFS_IMAGE=/root/demo/bhyvearm64-utils/files_guest/ramdisk-guest.img
root@freebsd:~/demo/bhyvearm64-utils# cp freebsd/sys/arm64/conf/ESPRESSOBIN_GUEST ~/demo/freebsd/sys/arm64/conf/
root@freebsd:~/demo/bhyvearm64-utils# cp freebsd/sys/dts/arm64/armada-3720-espressobin-guest.dts ~/demo/freebsd/sys/dts/arm64
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/guest_kernel.json --build buildkernel

Note: The guest kernel will be build with small filesystem built directly into the binary. The generation of this filesystem is done by the build script.

U-Boot and storage

Firstly, we have to configure U-Boot on ESPRESSObin. Here are some tutorial on how to do that:

Secondly, we have to store the host's filesystem on a microSD card:

  1. Make two partitions: FAT32 and UFS
  2. Copy the built filesystem (arm64_rootfs) on both partitions
  3. Copy the virtio.img and virtio_run.sh files from the utils repository on the UFS partition
  4. Copy the guest kernel binary, in the same directory as previous files(3).

On U-Boot, set some variables with the paths to the device tree and the loader file and a command which boots the host.

setenv fdt_name 'boot/dtb/marvell/armada-3720-espressobin.dtb'
setenv image_name 'boot/loader.efi'
setenv bootcmd 'mmc dev 0; fatload mmc 0:1 $kernel_addr $image_name;fatload mmc 0:1 $fdt_addr $fdt_name; bootefi $kernel_addr $fdt_addr'
Marvell>> run bootcmd
OK set currdev=disk0p2
OK boot
mountroot> ufs:/dev/mmcsd0s2

The FreeBSD host started. You can run the virtio_run.sh script and start a guest.

Custom bootloader (bootrom)

Clone the following repository on a Linux system and switch to the bhyvearm64 branch: https://github.com/FreeBSD-UPB/u-boot

Run the instructions:

gmake bhyve_arm64_defconfig
gmake CROSS_COMPILE=aarch64-none-elf- -j2

These will generate the u-boot.bin file.

Download an ISO image with the desired OS (FreeBSD or Linux). And clone the utils repository (bhyvearm64-utils), which has the bootrom_run.sh script. Copy these three files on the microSD card, in the /root path.

At this point you can insert the microSD card, start the board and run the script. The virtual machine will start.

Update the custom bootloader

  • arch/arm/dts/bhyve.dts is the virtual machine's device tree. Each update on the hardware specs should be reflected in this file
  • board/emulation/bhyve-arm/bhyve-arm.c contains the memory mappings that the bootloader should be aware of. A new MMIO device should be also added here.

First setup

  • After you compile and copy the operating system on the microSD card, there is a directory called files_bootrom. Copy them on the card. These contain all the some networking scripts, ssh keys and all the required files in order to run pkg in install packages (such as GDB).

  • When the board starts, run the following instructions:

    mount -uf / pwd_mkdb -p /etc/master.passwd pkg update pkg upgrade -f pkg install gdb

  • GDB will tell you that it is not able to find some libraries. Actually, these can be found in /usr/local/lib/ and can be copied in /lib/. After that, your debugger will work.

Clone this wiki locally