Velero backs up only metadata and not the data itself (pvc data)

[devopsInterviews]

Hi all, We are experiencing issues with backing up and restoring data on our vanilla Kubernetes cluster running on vSphere 7.3 to a remote MinIO S3 server using Velero, the vSphere plugin, and the AWS plugin. We are seeking your assistance to resolve this issue.

Environment Details:
Kubernetes Version: Vanilla Kubernetes Cluster (1.26.1)
vSphere Version: 7.3
Backup Solution: Velero
velero version: 1.13.2
Plugins Used:
velero/velero-plugin-for-aws
1.6.1
vsphereveleroplugin/velero-plugin-for-vsphere
1.4.2
velero/velero-plugin-for-csi
0.7.1

Object Storage: MinIO S3 on a remote server
Current Setup:
We have deployed Velero with the following pods in the velero namespace:

kubectl get pods -n velero

NAME READY STATUS RESTARTS AGE
backup-driver-88bd6cd86-ktlh7 1/1 Running 0 13m
datamgr-for-vsphere-plugin-49dc8 1/1 Running 0 12m
datamgr-for-vsphere-plugin-7flrm 1/1 Running 0 12m
datamgr-for-vsphere-plugin-9xswz 1/1 Running 0 12m
datamgr-for-vsphere-plugin-lv78l 1/1 Running 0 12m
velero-9c8d49fbc-6ndgp 1/1 Running 0 14m

Issue Details:
Backup Process: We are able to initiate backups using Velero, and the metadata (e.g., namespaces, services, pods, PVC definitions) appears to be backed up correctly.
Restore Process: During the restore process, all the Kubernetes objects (services, pods, PVCs, etc.) are restored successfully. However, the actual data inside the PVCs is not restored. The PVCs remain empty.

Observations:

Metadata Backup: The metadata is correctly migrated and visible after restoration.
Data Absence: The data within the PVCs is not restored, resulting in empty PVCs even though the PVC objects themselves are recreated.
Pod Logs: We have examined the logs of the datamgr-for-vsphere-plugin pods and observed errors related to invalid object names during the upload process to MinIO.
Log Excerpt from datamgr-for-vsphere-plugin Pod:

time="2024-06-26T12:59:28Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=upload error="Failed to upload snapshot, ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc, to durable object storage. XMinioInvalidObjectName: Object name contains unsupported characters.\n\tstatus code: 400, request id: 17DC8F5886D5214C, host id: dd9025bab4ad464b049177c46595eb6ebf374d3b3fd1af9251148b658df7ac2e3e8" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/upload_controller.go:380" error.function="github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller.(*uploadController).processUpload" key=velero/upload-2d9093bb-3e07-4e19-ab94-ae97e614a2dc logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/controller/generic_controller.go:138"

Log Excerpt from Restore Command:

error preparing persistentvolumeclaims/mynamespace/pvc-prometheus: rpc error: code = Unknown desc = Failed to create a CloneFromSnapshot CR: Phase=Failed, err=cloneFromSnapshot: Failed at calling SnapshotManager CreateVolumeFromSnapshotWithMetadata with peId pvc:mynamespace/pvc-prometheus, err: InternalError: We encountered an internal error, please try again.: cause(The filename, directory name, or volume label syntax is incorrect.)\n\tstatus code: 500, request id: 17DC8CD5B1B0338C, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8

Our Understanding of the Backup and Restore Process:
Backup Process:

Velero initiates a backup process, capturing both metadata and data snapshots.
The metadata is successfully backed up and stored in MinIO.
Data snapshots (IVDs) are taken but fail to upload to MinIO due to invalid object names.
Restore Process:

Velero restores the metadata correctly.
Attempts to restore data from MinIO fail, leading to empty PVCs.

Installation Command:
Here is the command we used to install Velero:

velero install --provider aws --bucket velero --secret-file credentials-velero --plugins velero/velero-plugin-for-aws:v1.6.1,vsphereveleroplugin/velero-plugin-for-vsphere:v1.4.2,velero/velero-plugin-for-csi:v0.7.1 --snapshot-location-config region=minio --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://:9000

We seek your guidance on the following points:

Resolution for Invalid Object Names: How can we ensure that object names generated during the snapshot process are valid and compatible with MinIO? We don't have control over them as far as I know.
Ensuring Data Backup and Restore: Steps we can take to ensure that both metadata and data are correctly backed up and restored.
We appreciate any insights or recommendations you can provide to help us resolve this issue.

Thank you for your support.

Best regards !

[blackpiglet]

@xing-yang
Could you please help to take a look at this issue?

[devopsInterviews]

@blackpiglet @xing-yang
We would really appreciate any help.

[blackpiglet]

The vSphere plugin issue needs expertise from @xing-yang.

I have one suggestion too. Please remove the CSI plugin from the environment, although this may not be the cause of your issue.

[devopsInterviews]

@blackpiglet
We already tried to remove the CSI plugin but we still faced the same issue.
@xing-yang If you have any suggestions it would be great.

[BarelElbaz]

+1

[BarelElbaz]

We have further investigation:
There is a difference between running it on Windows and Linux hosts, it seems that the issue occurs only on Windows (Running Minio as exe)
To reproduce, we used the latest versions of the Velero plugin and Minio, and here are the results:
On linux:

pip3 install minio
echo hello > file.txt
#Create a "velero" bucket in minio

Run the following script

from minio import Minio
from minio.error import S3Error

def main():
   # Initialize minio client object.
   minioClient = Minio(
       "localhost:9000",
       access_key="minioadmin",
       secret_key="minioadmin",
       secure=False
   )

   # Define the object name with unsupported characters.
   object_name = "ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc"
   bucket_name = "velero"
   file_path = "file.txt"
   content_type = "application/text"

   # Upload the file
   try:
       minioClient.fput_object(bucket_name, object_name, file_path, content_type=content_type)
       print(f"Successfully uploaded {object_name} to bucket {bucket_name}")
   except S3Error as err:
       print(f"Error occurred: {err}")

if __name__ == "__main__":
   main()

The upload passed successfully:

Successfully uploaded ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc to bucket velero

Try the same on a Windows host:
Install minio as exe (https://dl.min.io/server/minio/release/windows-amd64/minio.exe)
and run the script:

C:\minio>C:\minio\myscript.py
Error occurred: S3 operation failed; code: XMinioInvalidObjectName, message: Object name contains unsupported characters., resource: /velero/ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc, request_id: 17DDCC6B9550F1A0, host_id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, bucket_name: velero, object_name: ivd:90197568-1d73-4fce-88569-dfbe4b553a05:2d9093bb-35e07-4e19-ab94-ae97e614a2dc

It's probably an issue with minio's windows flavor because of a limitation in the NTFS file system (which doesn't support colon ':' in filenames):

Here is the code that verifies it in Minio:
https://github.com/minio/minio/blob/f736702da8f3fe0967f9a25bab6c8d7e27d002a4/cmd/object-api-utils.go#L197-L223

[devopsInterviews]

@xing-yang @blackpiglet
Do you have any estimation for this bug?
Because currently we can not use Velero and can not backup our data at all..

[blackpiglet]

@devopsInterviews
Since you use the vSphere plugin, please create a downstream ticket to trace this issue.
There is more guarantee of engineer effort for the downstream ticket.
The upstream OSS issue works in a best-effort way if the engineer is busy with other tasks.

[devopsInterviews]

@blackpiglet
We did create a ticket in the Vsphere plugin repo. On the same day we opened it here.
Unfortunately, no one even looked on the ticket there..
The link to the ticket:

vmware-tanzu/velero-plugin-for-vsphere#586

[blackpiglet]

Not sure whether you are using a commercial version of the vSphere k8s environment, e.g. TKGs, TKGm, or TKGI.
If you use one of them, you can create a ticket to escalate the issue.

[devopsInterviews]

@blackpiglet

Currently we do not use any commercial version and we depend on the open source community.
Is the only way to promote Windows support is via commercial version? Or do you believe we can get the help without that.