VolumeSnapshotContents Synced across clusters are rePUT because of incorrect source

[anshulahuja98]

What steps did you take and what happened:

Velero syncs VolumeSnapshotContents across clusters with same BSL as part of Backup Sync flow.

Here earlier Velero used to sync them with spec.source.snapshotHandle set, whereas now it is populated with VolumeHandle.
This leads to CSI driver to again trigger createSnapshot - https://github.com/kubernetes-csi/external-snapshotter/blame/master/pkg/sidecar-controller/snapshot_controller.go#L88

This further leads to throttling /high load on the CSI driver leading to other consequences.

What did you expect to happen:

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version):
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[anshulahuja98]

Initial fix is done in #7924

[Lyndon-Li]

@anshulahuja98 @blackpiglet For 1.14.1, I think the initial fix #7924 is enough right?
@anshulahuja98 Could you also cherry pick #7924 to 1.13 branch?

[anshulahuja98]

@Lyndon-Li we also need to merge - #7983
Can you please help with review

[anshulahuja98]

Post that I can help with cherry pick of both commits

[blackpiglet]

Put more comments about this fix for the user to understand how this fix works and its limitations.

The fix resets the VolumeSnapshotContent.Spec.Source from the VolumeHandle to SnapshotHandle on the backup sync process. This makes the VolumeSnapshotContents a static type avoiding the CSI snapshot controller to recreate its VolumeSnapshot.

But this modification is not persisted in the backup's metadata in the Backup's object storage, so the VolumeSnapshotContent in the backup's volumesnapshotcontents.json.gz is different from the synced VolumeSnapshotContent created in the k8s cluster.

That should be fixed in the v1.15.0 Velero release by removing the VolumeSnapshotContent from the backup metadata.

[Lyndon-Li]

Reopen for cherry-picks