feat: support additional virtualized NVMe disks #85
Conversation
Any node for which nvme_size is specified in settings.yml will automatically have one additional virtual NVMe disk created, which will appear as /dev/nvme0n1. This can be useful for testing additional disks with Longhorn. The way it works is a bit obscure because libvirt only seems to support virtio, scsi and sata disks, so we have to resort to hacking qemu arguments in as described in this blog post: http://blog.frankenmichl.de/2018/02/13/add-nvme-device-to-vm/ This means we have to go create the volume manually first with `virsh` then pass its path to qemu. Matters are complicated further by volumes being owned by root by default, while libvirt runs VMs as the qemu user. For normal libvirt volumes, file ownership changes happen automatically, but for these volumes we're hacking in by hand, that doesn't happen, so we have to explicitly specify ownership, and to do _that_ we have to define the volume via XML, hence the nasty `virsh vol-create-as ... | sed ... | virsh vol-create` invocation. The other wrinkle is that we need to know the exact path to the disk image. Once a volume has been created, you can run `virsh vol-path --pool default VOLUME_NAME` to get the path, but we need to know the path when setting libvirt.qemuargs, whereas we don't want to actually create the volume until the `vagrant up` trigger for the node in question. If we create the volume outside the trigger, it gets created on _every_ evaluation of the Vagrant file, even for unrelated VMs... So, we call `virsh pool-dumpxml default` and get the pool path from that, then stick the volume name on the end. To test, try something like this in settings.yml: ``` harvester_network_config: cluster: - ip: 192.168.0.30 # [...] nvme_size: 10G ``` Signed-off-by: Tim Serong <[email protected]>
|
Does the network interface name change after adding an NVME disk? I have an impression that the ens5/ens6 or something changed after adding the second disk. |
|
Doesn't seem to make any difference. I just gave it a quick test checking ...and second with an NVMe: |
|
@m-ildefons you might be interested in this, as I see you're adding multiple disk support in #86 :-) |
|
When running the setup script i got this error: I have no default pool on my system: |
|
I would suggest that you add the line |
|
By the way I think we can have a more general naming cuz the created qcow2 file can also be used as other disk types, not only NVMe |
|
OK, that's weird. I don't think I did anything special to be able to see the default pool... I am in the |
|
Oh, wait a minute, my I'm going to have to experiment a bit more here. Out of curiosity @votdev @albinsun, what do you see if you run |
|
Actually, if you don't mind :-) please try the updated version of this PR which explicitly specifies |
|
|
I'm sorry, it fails again. |
All good, at least we got slightly further :-) but it's annoying that there's not more detail in the output as to what's failing now :-/ I'm not sure offhand how to further instrument that trigger script... I don't suppose you could try running this nasty thing directly from a shell and see if it prints anything interesting...?
|
This command succeeds when running in CLI. Will try how it behaves when there are no nodes deployed. |
Signed-off-by: Tim Serong <[email protected]>
tserong
force-pushed
the
wip-virtual-nvme
branch
from
066cd34 to
d539797
Compare
|
Latest code successfully create nvme0.qcow2 but seems not in hardware list yet. Tried to manually add disk but fail with |
Any node for which nvme_size is specified in settings.yml will automatically have one additional virtual NVMe disk created, which will appear as /dev/nvme0n1. This can be useful for testing additional disks with Longhorn. The way it works is a bit obscure because libvirt only seems to support virtio, scsi and sata disks, so we have to resort to hacking qemu arguments in as described in this blog post:
http://blog.frankenmichl.de/2018/02/13/add-nvme-device-to-vm/
This means we have to go create the volume manually first with
virshthen pass its path to qemu. Matters are complicated further by volumes being owned by root by default, while libvirt runs VMs as the qemu user. For normal libvirt volumes, file ownership changes happen automatically, but for these volumes we're hacking in by hand, that doesn't happen, so we have to explicitly specify ownership, and to do that we have to define the volume via XML, hence the nastyvirsh vol-create-as ... | sed ... | virsh vol-createinvocation.The other wrinkle is that we need to know the exact path to the disk image. Once a volume has been created, you can run
virsh vol-path --pool default VOLUME_NAMEto get the path, but we need to know the path when setting libvirt.qemuargs, whereas we don't want to actually create the volume until thevagrant uptrigger for the node in question. If we create the volume outside the trigger, it gets created on every evaluation of the Vagrant file, even for unrelated VMs... So, we callvirsh pool-dumpxml defaultand get the pool path from that, then stick the volume name on the end.To test, try something like this in settings.yml: