persistent disks

[mritalian]

Hi,

I am wondering, how to create persistent disks outside of a proxmox_virtual_environment_vm so that I may attach them when the VM is created, then delete them later?

Without doing something like that, non-ephemeral storage becomes considerably more difficult and less performant (i.e. nfs, iscsi, etc...)

I'm willing to work on the implementation if needed.

Thanks

[bpg]

Hi @mritalian!

So far I didn't have such a use case in my environments. If you have some ideas how this can be implemented directly in PVE that would be a good starting point. Perhaps try to create this configuration in PVE UI and document the steps, then we can see what is needed from the provider to implement that?

Thanks!

[otopetrik]

It looks like PVE has no support for disks not owned by a VM (or a template).

But there might be a workaround. One VM (101) can use disks owned by a different VM (102).
Using qm set 101 --scsi1 local-zfs:vm-102-disk-0 works, and 101.conf contains something like:

...
scsi0: local-zfs:vm-101-disk-0
scsi1: local-zfs:vm-102-disk-0
...

VM 102 then exists just as a container for data disks - no boot disk, no boot order, etc.

Ideally PVE would have additional VM type - in addition to existing "VM" and "template".
Using tags to mark such "data VMs" and using a hookscript to ensure that the VM is never started might be enough.

It might also be a good idea to use hookscript for VMs using data disks to perform some sort of locking, to make sure no other VM is using disks from the same data VM.

Using single disk in multiple VMs at the same time would require use of a shared-disk clustered filesystem (like GFS2).

Backups might work:

• pve-zsync does not seem to use qemu-guest agent to perform guest-fsfreeze-freeze and guest-fsfreeze-thaw, so behavior should be the same by just running pve-zsync against data VM 102.
  Given that pve-zsync does not work on linked clones, I have no idea what would happen if it ran against a VM which uses disks from another VM.

• When using Proxmox Backup Server, it might just work by backing up VM 102 (and including the disks owned by VM 101).
  After destroying VM 102 and creating a replacement VM (e.g VM 103) using the same disks from VM 101,
  the Proxmox Backup Server should deduplicate data on the receiving side when receiving the first backup of VM 103.

• Using Proxmox Backup Server for VM 101 is probably a very bad idea, it would try to backup an image which changes underneath the running backup process. It likely would have no idea about running VM 102 qemu process and the backup would not be consistent.

Second option is to handle data disks completely outside PVE control, just create zvol manually (e.g. using terraform null_resource and ssh) and use full path in VM configuration, like when using disk passthrough.

The provider's disk argument would have to be modified to:

• support custom name argument (for using existing disks from other VMs)
• support empty string datastore_id (for using full path), including handling a disk without the : separator.

[mritalian]

Thanks for the useful info. In the state of things now it seems my only options are, in sequence

1. shell script: create the volumes manually
2. terraform: create vms using terraform but somehow tell it not to power on
3. shell script: attach the volumes to the created vms
4. shell script: power on vm

and at destroy time:

• use at destroy provisioner on vm resource to detach the disk before vm is destroyed

Using shell_script resource instead of null_resource + provisioner, you can add some state to all of these, so it's less a bunch of arbitrary scripts running in the wild, and more a set of resources with a state and lifecycle to them, which is more terraform-like

As far as enhancing this provider, the main thing of value I see at this point is some mechanism to specify volume to attach to vm at create time, with abilty to remember which disks "belong" to the vm and should be deleted with it, and which were simply attached & should not be deleted with the vm. Just for comparison, vSphere provider handles this behaviour in the provider (i.e. deleting the vm from the console always deletes all disks irrespective of whether they were created with the vm or not, but destroying the resource through tf doesn't destroy disks that were created outside of the resource).

[otopetrik]

Thanks for the useful info. In the state of things now it seems my only options are, in sequence

1. shell script: create the volumes manually

No, volumes can be created using terraform:

E.g.

resource "proxmox_virtual_environment_vm" "data_vm" {
  ...

  vm_id     = 501 
  name     = "test-data-vm"
  tags     = ["data-vm"]
  started = false
  # data disk to be used by another VM
  disk {
    datastore_id = "local-zfs"
    file_format  = "raw"
    interface = "scsi0"
    size = 20
  }
}

Volumes can be created in a never-to-be-run VMs, this way pve-zsync could still be used to quickly backup data disks (e.g. every few minutes from a single NVMe to a pair of mirrored HDDs or to another machine).

2. terraform: create vms using terraform but somehow tell it not to power on

3. shell script: attach the volumes to the created vms

4. shell script: power on vm

The following is a proof-of-concept showing what is possible now:

resource "proxmox_virtual_environment_vm" "data_user_vm" {
  ...

  vm_id     = 502 
  name     = "test-data-user-vm"

  # boot disk
  disk {
    datastore_id = "local-zfs"
    file_format  = "raw"
    interface = "scsi0"
    size = 8 
  }

  connection {
    type = "ssh"
    user = "root"
    # proxmox host
    host = "pve-01.lan"
  }

  # attach data disk
  provisioner "remote-exec" {
    inline = [ "qm set ${self.vm_id} --scsi1 local-zfs:vm-501-disk-0"]
  }

}

At the moment, it has many unwanted qualities:

• It attaches the disk late, in provisioner, after the VM has already started. Because of disk hotplug, it is not a huge issue, but it is still a possible race condition - VM could boot faster than provisioner attaches the disk.
• The provider has no idea about the second attached disk, running terraform apply again would try to remove the disk or re-create the VM. This is the main issue.
• Changes in provisioner code do not automatically taint the VM. E.g. adding second "qm set" call to the provisioner will not the script to be re-run.

Making this actually usable would require some changes to the provider.
Instead of using remote-exec provisioner, it would need to support something like this:

resource "proxmox_virtual_environment_vm" "data_user_vm" {
  ...
  # data disk
  disk {
    existing = "local-zfs:vm-501-disk-0"
    interface = "scsi1"
  }
}

and ideally a version for volumes not managed by PVE at all (also useful for passing whole physical HDDs directly to VMs)

resource "proxmox_virtual_environment_vm" "data_user_vm" {
  ...
  # data disk
  disk {
    existing = "/dev/zvol/rpool/not-managed-by-pve/data-disk-123"
    interface = "scsi1"
  }
}

and at destroy time:

* use at destroy provisioner on vm resource to detach the disk before vm is destroyed

After running terraform destroy -target proxmox_virtual_environment_vm.data_user_vm on the above terraform code,
calling zfs list rpool/data/vm-501-disk-0 still shows the disk as existing.

It seems that when PVE destroys a VM, it destroys only disks "owned" by the VM.
Probably just the ones prefiexed by vm-501-, but it would be nice to have it somewhere documented as a guarantee. Change to that behavior could get ugly.

Using shell_script resource instead of null_resource + provisioner, you can add some state to all of these, so it's less a bunch of arbitrary scripts running in the wild, and more a set of resources with a state and lifecycle to them, which is more terraform-like

Yes, that looks like a better way to wrap zfs create and zfs destroy.

As far as enhancing this provider, the main thing of value I see at this point is some mechanism to specify volume to attach to vm at create time, with abilty to remember which disks "belong" to the vm and should be deleted with it, and which were simply attached & should not be deleted with the vm.

Fortunately it seems that there is no need to remember that. It looks like ``terraform destroy -target proxmox_virtual_environment_vm.data_user_vmdestroys only disks with name indicating they belong to the target VM. And when using full/dev/...` path to a block device, there is no reason why PVE should known, how to actually "delete" such device (is it zvol, LVM LV, a loopback ... ?)

It seems that adding something like existing= argument to disk block (and all the code behind it...) should be enough.

[mritalian]

Hi. This is very useful info. I much prefer this way of creating volumes. I will work with this new info and report back my results.

[mritalian]

Because anything worth doing is worth overdoing...

variable proxmox_parameters {
  type = object({
    user = string
    password = string
    endpoint = string
    node_name = string
    tags = optional(map(string))
  })
  sensitive = true
}

variable "attach_disks" {
  type = list(object({
    disk_id = optional(string)
    mount_point = string # A unique place where to mount the primary partition, i.e. /mnt/sdb1
  }))
  default = []
}

[...]


resource "proxmox_virtual_environment_vm" "vm" {

[....]

  provisioner "local-exec" {
    environment = {
      NODE = var.proxmox_parameters.node_name
      VMID = self.vm_id
      USERNAME = var.proxmox_parameters.user
      PASSWORD = var.proxmox_parameters.password
      ENDPOINT = trimsuffix(var.proxmox_parameters.endpoint, "/") # includes https://, port, and possibly trailing slash
    }
    interpreter = ["/bin/bash", "-ce"]
    command = <<-EOT
      # Perform Login
      LOGIN_RESPONSE=$(curl -k -d "username=$${USERNAME}&password=$${PASSWORD}" $${ENDPOINT}/api2/json/access/ticket) || { echo 'Login failed'; exit 1; }
      TICKET=$(echo $${LOGIN_RESPONSE} | jq -r .data.ticket) || { echo 'Ticket extraction failed'; exit 1; }
      CSRF=$(echo $${LOGIN_RESPONSE} | jq -r .data.CSRFPreventionToken) || { echo 'CSRF token extraction failed'; exit 1; }
      # Perform SCSI Disk Assignment
      ${join(" && ",
          [for idx in range(length(var.attach_disks)):
            format("curl -k -b \"PVEAuthCookie=$${TICKET}\" -X PUT -H \"CSRFPreventionToken: $${CSRF}\" -d \"scsi%d=%s\" $${ENDPOINT}/api2/json/nodes/$${NODE}/qemu/$${VMID}/config", 1 + length(var.create_disks) + idx, var.attach_disks[idx].disk_id)
        ]
      )}
      # Log out (optional and not necessarily provided by Proxmox API)
    EOT
  }
[...]
}

This has the added benefit of not requiring ssh access, at the expense of being super fragile. Still, with your poitners the solution very workable. That said, disk attachment is a somewhat rudimentary thing, I feel this could easily be added to the provider; I'm willing to assist in that.

[otopetrik]

#606 should fix it. @mritalian, could you verify that ? Thanks!

[mritalian]

Very cool. Will try it out.