Snapshotting volumes on a Microsoft Azure Government cloud

[max3903]

I was able to install Velero 1.6 on a Microsoft Azure Government cloud (see #3786 for reference) and setup a daily schedule to backup the entire cluster.

Everything seems to work fine except the snapshots of persistent volumes: I get a ResourceNotFound error in the logs of the velero pod:

time="2021-06-09T21:43:25Z" level=info msg="Processing item" backup=velero/backup-1643 logSource="pkg/backup/backup.go:354" name=odoo-postgresql-0 namespace=odoo-dev progress= resource=pods
time="2021-06-09T21:43:25Z" level=info msg="Backing up item" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:121" name=odoo-postgresql-0 namespace=odoo-dev resource=pods
time="2021-06-09T21:43:25Z" level=info msg="Executing custom action" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:327" name=odoo-postgresql-0 namespace=odoo-dev resource=pods
time="2021-06-09T21:43:25Z" level=info msg="Executing podAction" backup=velero/odoo-dev-backup-1643 cmd=/velero logSource="pkg/backup/pod_action.go:51" pluginName=velero
time="2021-06-09T21:43:25Z" level=info msg="Adding pvc data-odoo-postgresql-0 to additionalItems" backup=velero/odoo-dev-backup-1643 cmd=/velero logSource="pkg/backup/pod_action.go:67" pluginName=velero
time="2021-06-09T21:43:25Z" level=info msg="Done executing podAction" backup=velero/odoo-dev-backup-1643 cmd=/velero logSource="pkg/backup/pod_action.go:77" pluginName=velero
time="2021-06-09T21:43:25Z" level=info msg="Backing up item" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:121" name=data-odoo-postgresql-0 namespace=odoo-dev resource=persistentvolumeclaims
time="2021-06-09T21:43:25Z" level=info msg="Executing custom action" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:327" name=data-odoo-postgresql-0 namespace=odoo-dev resource=persistentvolumeclaims
time="2021-06-09T21:43:25Z" level=info msg="Executing PVCAction" backup=velero/odoo-dev-backup-1643 cmd=/velero logSource="pkg/backup/backup_pv_action.go:49" pluginName=velero
time="2021-06-09T21:43:25Z" level=info msg="Backing up item" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:121" name=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 namespace= resource=persistentvolumes
time="2021-06-09T21:43:25Z" level=info msg="Executing takePVSnapshot" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:405" name=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 namespace= resource=persistentvolumes
time="2021-06-09T21:43:25Z" level=info msg="label \"topology.kubernetes.io/zone\" is not present on PersistentVolume, checking deprecated label..." backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:432" name=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 namespace= persistentVolume=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 resource=persistentvolumes
time="2021-06-09T21:43:25Z" level=info msg="label \"failure-domain.beta.kubernetes.io/zone\" is not present on PersistentVolume" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:435" name=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 namespace= persistentVolume=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 resource=persistentvolumes
time="2021-06-09T21:43:25Z" level=info msg="Got volume ID for persistent volume" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:462" name=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 namespace= persistentVolume=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 resource=persistentvolumes volumeSnapshotLocation=default
time="2021-06-09T21:43:25Z" level=info msg="Getting volume information" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/item_backupper.go:483" name=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 namespace= persistentVolume=pvc-598f4ea6-eddd-4435-99b8-9c535598aad0 resource=persistentvolumes volumeID=kubernetes-dynamic-pvc-598f4ea6-eddd-4435-99b8-9c535598aad0
time="2021-06-09T21:43:25Z" level=info msg="1 errors encountered backup up item" backup=velero/odoo-dev-backup-1643 logSource="pkg/backup/backup.go:427" name=odoo-postgresql-0
time="2021-06-09T21:43:25Z" level=error msg="Error backing up item" backup=velero/odoo-dev-backup-1643 error="error getting volume info: rpc error: code = Unknown desc = compute.DisksClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code=\"ResourceNotFound\" Message=\"The Resource 'Microsoft.Compute/disks/kubernetes-dynamic-pvc-598f4ea6-eddd-4435-99b8-9c535598aad0' under resource group '****' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix\"" logSource="pkg/backup/backup.go:431" name=odoo-postgresql-0

and:

$ velero backup logs odoo-dev-backup-1643
An error occurred: request failed: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.
RequestId:54580233-701e-0072-2478-5d7e86000000
Time:2021-06-09T21:43:33.6879806Z</Message></Error>

It appears that the volumes of the cluster belong to a different resource group (Infrastructure Resource Group) than the one of the cluster and cannot be changed on the cluster configuration in the Azure portal.

Does anyone have a workaround? How do you take snapshots of volumes? Is it specific to government cloud?

Thanks!

CC: @shooter2134

[shooter2134]

Did you find a workaround ?

[max3903]

Nope, we stayed away from volumes for production environments and used blob storage.