VirtualBox: Support VDI with package command

[auriusbendikas]

Currently vagrant imports VirtualBox boxes as VMDK images just like specified in OVA. However since version 5.0 (please correct me) VBoxManage supports "--options importtovdi", which will convert VMDK image to VDI during import from OVF.

Currently there is no way to enable this option via Vagrantfile configuration.
It would be very welcome enhancement if it could be done something like this (as specified in #7378):

config.vm.provider "virtualbox" do |v|
  v.use_vdi = true
end

This opens possibilities for further customizing VDI image with features not supported by VMDK, like enabling SSD TRIM support, resizing and etc.

References

• VirtualBox provider: Extension to allow disk resizing on VM import #7378
• The command 'vagrant up' creates a VMDK disk instead of VDI #3567

[briancain]

@auriusbendikas - is it not possible to do something like:

Vagrant.configure("2") do |config|
  # ...

  config.vm.provider "virtualbox" do |vb|
    vb.customize ["import", :id, "--options", "importtovdi"]
  end
end

for enabling the option?

edit: I haven't tested that yet, but I'm wondering if it's possible to define a virtualbox provider customize command that enables this option.

[auriusbendikas]

Hi @briancain, Just tried it today.

A customization command failed:

["import", :id, "--options", "importtovdi"]

The following error was experienced:

#<Vagrant::Errors::VBoxManageError: There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["import", "b512253a-f94e-46d5-81e8-8a2cd1345db6", "--options", "importtovdi"]

Stderr: VBoxManage.exe: error: Appliance file must have .ovf or .ova extension
VBoxManage.exe: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component ApplianceWrap, interface IAppliance, callee IUnknown
VBoxManage.exe: error: Context: "Read(Bstr(pszAbsFilePath).raw(), progressRead.asOutParam())" at line 324 of file VBoxManageAppliance.cpp
>

Please fix this customization and try again.

[bpow]

We can't use the :id here since the import needs to refer to the ova or ovf provided by the box. Is this available as a variable to be used in the Configure block somehow?

[bpow]

Actually, looking into this some more, I think customize-commands are being called after the import has already taken place, so it is probably too late?

[auriusbendikas]

You are absolutely right. The modifications are needed in the OVF import logic. If you look at the "References" section of the issue description specifically "GH-7378". This non merged pull request implements support for use_vdi property for every virtualbox driver, which in turn modifies import procedure of the OVF image where it is supported. It is a shame this pull request was not merged. In my opinion this functionality should be turned on by default on driver versions that support it.

[AntonioMeireles]

@chrisroberts, (as you're the one who vetoed #7378)

any plans to revisit this ? IMHO this is a key missing feature from the VirtualBox provider ...

Thanks in advance!

[joso9001]

This is something that would mean quite a bit for me!

Currently, one stopping me from going full Vagrant (I do use it quite a bit already) from manual installations is how much easier it is to work with VDI in VirtualBox. I'm aware you can convert it (#3567), but it seems like unnecessary work when there's support in VirtualBox for importing to VDI already.

I'm a big fan of having the virtual disks stay around 2 GB with the discard option, even after long-term use.

[JockeTF]

Whoops, seems I used the wrong account before. For the record, simply replacing:

output = execute("import", "-n", ovf)

With the following:

output = execute("import", "-n", "--options", "importtovdi", ovf)

In plugins/providers/virtualbox/driver/version_6_0.rb makes it use VDI for me.

Would it make sense to make this the default for VirtualBox?

[AntonioMeireles]

@JockeTF, i 'd suggest to keep surprises to a minimum that short term the ideal would be to add a new option to the provider (say the virtualbox.use_vdi referenced at the beggining of this thread which would default to false, per current behaviour). Later on the road, with things well tested its default would flip to true.

[thiagotalma]

That would be quite interesting.
.vmdk disks can not be compressed.

@briancain , do you suggest any workaround?

[briancain]

Hi, if the goal is essentially #7378, then my new pull request #11349 will do this out of the box. You will be able to resize disk images with Vagrant without having to worry about the format, even on the first vagrant up of a box. Vagrant will detect if the disk is vmdk and convert it to vdi to resize, then reclone back to preserve the original format.

If you really just want the primary disk to be vdi instead of vmdk, I think that's actually up to the box maintainer to choose. So you could in theory bring up a box, use vboxmanage clonevm to convert it to a vdi and reattach the image, then run vagrant package to get a box with the primary image as vdi.

[vadbes46]

@briancain this pull request for resizing disks is amazing
but can you just improve it a little bit not to convert the disk back to original format - vmdk?
just one option keep_in_vdi
it is needed for a primary disk imported from various boxes

[briancain]

@vadbes46 - Sure, that makes sense 👍 if you wanted to make a new feature request issue for that, that would be great!

[vadbes46]

@briancain maybe it could be done within this issue so it can be closed? For me it seems to fully cover the original request.

Thanks & Regards.

[vadbes46]

currently did a workaround for myself - little custom provision plugin which executes only once at first up - it halts machine, recreates primary disk and reattaches it(using your code as an example), then powers on and continues provision

Thanks

[patrickjmccarty]

@briancain I believe you are mistaken when you write above "I think that's actually up to the box maintainer to choose". The vagrant package command always exports to VMDK, regardless of what format your base VM was originally created with, because VMDK is the only disk format supported by OVF it seems. There is no way that I can find currently (short of hacking Vagrant's code) to easily and efficiently import a box as a VDI image, which is the preferred disk format when using VirtualBox. That's the feature that this issue is requesting. VirtualBox has made it very easy to achieve by providing the "--options importtovdi" argument; Vagrant just needs to expose that option to allow a Vagrantfile to enable it, as was originally proposed in this issue.

[gavenkoa]

I tried https://www.vagrantup.com/docs/disks/configuration to pass --discard on --nonrotational on for TRIM support (with export VAGRANT_EXPERIMENTAL="disks").

Unfortunately Vagrant 2.2.10 doesn't support syntax config.vm.disk yet:

    # config.vm.disk :disk, name: "ydisk", size: "10", primary: true, provider_config: {"virtualbox__discard": "on", "virtualbox__nonrotational" => "on"}
    config.vm.disk :disk, name: "ydisk", size: "10", primary: true, provider_config: {"virtualbox" => {"discard" => "on", "nonrotational" => "on"}}

fails on:

 INFO runner: Running action: environment_unload #<Vagrant::Action::Builder:0x0000000006af1b48>
C:/opt/Vagrant/embedded/gems/2.2.10/gems/vagrant-2.2.10/lib/vagrant/util/numeric.rb:40:in `string_to_bytes': undefined method `to_sym' for nil:NilClass (NoMethodError)
        from C:/opt/Vagrant/embedded/gems/2.2.10/gems/vagrant-2.2.10/plugins/kernel_v2/config/disk.rb:171:in `validate'
        from C:/opt/Vagrant/embedded/gems/2.2.10/gems/vagrant-2.2.10/plugins/kernel_v2/config/vm.rb:943:in `block in validate'
        from C:/opt/Vagrant/embedded/gems/2.2.10/gems/vagrant-2.2.10/plugins/kernel_v2/config/vm.rb:942:in `each'
        from C:/opt/Vagrant/embedded/gems/2.2.10/gems/vagrant-2.2.10/plugins/kernel_v2/config/vm.rb:942:in `validate'

[rgaufman]

I found this example online:

HOME_DISK = '/path-to-disk.vdi'.freeze

Vagrant.configure('2') do |config|
  config.vm.provider 'virtualbox' do |vb|
    vb.memory = 4096
    vb.cpus = 4

    if ARGV[0] == 'up' && !File.exist?(HOME_DISK)
      vb.customize ['createhd',
                    '--filename', HOME_DISK,
                    '--format', 'VDI',
                    '--size', 40000]

      vb.customize ['storageattach', :id,
                    '--storagectl', 'SATA',
                    '--port', 0,
                    '--device', 0,
                    '--type', 'hdd',
                    '--medium', HOME_DISK]
    end
  end
  ...

Is this a good idea, or no? - Any better way to achieve this?

EDIT: This doesn't work, it creates the VDI but it's not used by the VM box: h.vm.box = 'ubuntu/focal64'.

Any solution for this?

[ALameLlama]

Would love to support for this, so we can configure homestead to use VDI using the Vagrantfile.

Because VMDK can't trim without converting, we've had issues with the disk reaching 70GB and needing to re-provision.

currently I've got my driver edited with @JockeTF solution and it's working but would be nice to have support for this.