Checkpoint for bhyve using libvdsk

In order to use this checkpoint implementation you need to:

• get the implementation using the following commands:

root@host # git clone https://github.com/FreeBSD-UPB/libvdsk
root@host # cd libvdsk
root@host # git checkout bhyve_checkpoint

• build and install the implementation for bhyve as follows:

NOTE: The following command MUST be run after the commands above.

root@host # make clean && make -DWITH_BHYVE_SNAPSHOT && make install

NOTE: The make clean command is important in order to use the last version of your code. It may work without it but if it doesn't, clean first.

The project proposes a checkpoint mechanism for bhyve using the qcow2 disk format provided the by livdsk library. The implementation provides two types of checkpoint:

1. Internal

The internal checkpoint uses the qcow2 disk format capability to store the snapshot information using new clusters inside the disk. On restore, in order to avoid copying all the clusters only the L1 table is copied.

In order to create an internal checkpoint the following command needs to be used:

bhyvectl --checkpoint <checkpoint_file_name> --vm=<guest_name>

The above command works only if the guest virtual machine is "alive" (powered on) because the checkpoint command is sent using an UNIX socket.

In order to restore an internal checkpoint the following command can be used:

bhyve -c 1 -m 512M -H -A -P -s 0:0,hostbridge -s 1:0,lpc -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 2:0,virtio-net,tap0 -s 3:0,ahci-hd,<qcow2_disk_name> -l com1,stdio -r <checkpoint_file_name> <guest_name> # (*)

NOTE: Some of the parameters that can be seen above can be changed because they represent the guest configuration. The command will restore the first found checkpoint with the name provided by the (*) command.

For deletion the following command can be used:

bhyvectl --del-checkpoint <checkpoint_file_name> --vm=<guest_name>

NOTE: The command will delete the first found checkpoint with the same name as <checkpoint_file_name> provided to the command.

2. External

The external checkpoint use the qcow2 base file and overlay file concepts. After a checkpoint creation finished successfully a new qcow2 file will be created based on the provided checkpoint file name and it will have as base the disk before the checkpoint creation was requested.

In order to create an external checkpoint the following command can be used:

bhyvectl --ext-checkpoint <checkpoint_file_name> --vm=<guest_name>

If you want to restore a guest just use the resulted file in order to boot the virtual machine or you can use a command similar with (*) while specifying the resulted file if you want to have the state of the guest as well.

For deleting an external checkpoint file the following command can be used:

bhyvectl --del-ext-checkpoint <checkpoint_file_name> --vm=<guest_name>

Recommendations and restrictions

• ahci disks should be used for starting the guest because the solution was tested using these type of disks.
• for using the commands provided above for checkpoint creation and deletion the guest must be powered on; for restore the guest must be rebooted using commands similar with the (*) command
• the deletion operation does not have a checkpoint merge mechanism implemented so any deletion will result in losing the data of the deleted checkpoint
• the libvdsk qcow2 implementation does not provide a cluster freeing method so creating and deleting checkpoints multiple times increases the size of disk, and it may also corrupt it so a rollback mechanism should be used in order to have a stable state of the disk

NOTE: The branch from here provides a working intergration between freebsd-src implementation and libvdsk implementation which can be used as a starting point in order to bring the checkpoint mechanism directly to bhyve implementation.