Development environment setup using QEMU

Usually, we use QEMU with simulated NVMe device to perform development work. We use malloc0 instead of actual NVMe device in bdev.conf to run SPDK applications. Doing so allows multiple users to share the same physical server to perform development task.

1. Setup a QEMU with customize kernel

• Create f100M using dd if=/dev/urandom of=f100M bs=1048576 count=500

• Build kernel and make sure the following parameters are enabled in .config:

  • CONFIG_UIO_PCI_GENERIC=y (after manually set CONFIG_UIO=y, hit make will trigger the setup for CONFIG_UIO_PCI_GENERIC)
  • CONFIG_VIRTIO_PCI=y

• We start QEMU as follows:

sudo qemu-system-x86_64 -enable-kvm -curses -m 4096 -smp 4 -redir tcp:4444::22 -hda my-disk.img -hdb my-seed.img -drive file=f100M,if=none,id=D22 -device nvme,drive=D22,serial=foo -cpu host -kernel kbuild2/arch/x86_64/boot/bzImage -append "root=/dev/sda1"

• Install necessary packages in QEMU:

sudo apt-get update
sudo apt install make
sudo apt install build-essential
sudo apt install libnuma-dev
sudo apt-get install uuid-dev
sudo apt-get install libssl-dev

• Setup SPDK as follows:

git clone https://github.com/spdk/spdk.git
cd spdk
git checkout v18.07
git submodule update --init
./configure --prefix=/home/ubuntu/spdk_install
make -j8

• sudo scripts/setup.sh in spdk and run sudo ./hello_bdev under spdk/examples/bdev/hello_world/

• Optionally, you can put rustfs on the physical server and share the directory with your QEMU so that you can run your code in QEMU to make sure your change doesn't crash and run on the physical server to gather performance metrics. See ref1 and ref2 on setup directory sharing between host and guest.