When executing velero backup delete, object data of ${bucket_name}/kopia/${namespace} cannot be deleted from s3 (minio).

[aki-kamada]

When executing velero backup delete, object data of ${bucket_name}/kopia/${namespace} cannot be deleted from s3 (minio)
I confirmed that it can delete ${bucket_name}/backups/${backup_name}.
But, it can't delete ${bucket_name}/kopia/${namespace}.

Is it bug? If this is bug, I will open ticket for bug.

• use version
  1.12.0

• using velero backup command with parametter "--snapshot-move-data"
  $ velero backup create 20231002-1656-kamap2 --snapshot-move-data --selector velerobackup=enable --namespace kube-storage --include-namespaces kamap2

• How to reproduce

##### I deleted backup resource by "20231002-1656-kamap2".
$ velero backupo delete 20231002-1656-kamap2 -n kube-storage

##### I confirmed that it was deleted.
$ velero backup get --namespace=kube-storage
NAME                                               STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
20230929-163700-kongskgr                           Completed   0        0          2023-09-29 16:38:00 +0900 JST   25d       default            velerobackup=enable
20230929-171100-kongskgr                           Completed   0        0          2023-09-29 17:11:45 +0900 JST   25d       default            velerobackup=enable
kong-customplugin-backup                           Completed   0        0          2023-09-25 17:08:37 +0900 JST   21d       default            velerobackup=enable
test-schedule-kamada-202310021818-20231002093043   Completed   0        0          2023-10-02 18:30:43 +0900 JST   29d       default            velerobackup=enable


##### I confirmed that backuprepository resource is existing with the name of "kamap2-default-kopia-5mmvt".
$ velero repo get --namespace=kube-storage
NAME                                    STATUS   LAST MAINTENANCE
kamada-default-kopia-gx92k              Ready    2023-10-03 16:55:40 +0900 JST
kamap2-default-kopia-5mmvt              Ready    2023-10-03 17:00:40 +0900 JST
kong-customplugin-default-kopia-zmgcs   Ready    2023-10-03 17:30:40 +0900 JST



##### I confirmed velero log that it was delete 20231002-1656-kamap2. But I found that some objects show error under the kopia/20231002-1656-kamap2 directory.
time="2023-10-02T08:03:30Z" level=info msg="Removing existing deletion requests for backup" backup=20231002-1656-kamap2 controller=backup-deletion deletebackuprequest=kube-storage/20231002-1656-kamap2-db5j7 logSource="pkg/controller/backup_deletion_controller.go:483"
time="2023-10-02T08:03:30Z" level=info msg="invoking DeleteItemAction plugins" item=20231002-1656-kamap2-7phnl logSource="internal/delete/delete_item_action_handler.go:116" namespace=kube-storage
time="2023-10-02T08:03:30Z" level=info msg="Executing DataUploadDeleteAction" backup=20231002-1656-kamap2 cmd=/velero controller=backup-deletion deletebackuprequest=kube-storage/20231002-1656-kamap2-db5j7 logSource="pkg/datamover/dataupload_delete_action.go:33" pluginName=velero
time="2023-10-02T08:03:31Z" level=info msg="Starting to check for items in namespace" logSource="internal/delete/delete_item_action_handler.go:100" namespace=kamap2
time="2023-10-02T08:03:31Z" level=info msg="invoking DeleteItemAction plugins" item=deployment-mount300 logSource="internal/delete/delete_item_action_handler.go:116" namespace=kamap2
time="2023-10-02T08:03:31Z" level=info msg="invoking DeleteItemAction plugins" item=kamap2 logSource="internal/delete/delete_item_action_handler.go:116" namespace=
time="2023-10-02T08:03:31Z" level=info msg="Starting to check for items in namespace" logSource="internal/delete/delete_item_action_handler.go:100" namespace=kamap2
time="2023-10-02T08:03:31Z" level=info msg="invoking DeleteItemAction plugins" item=my-first-pvc300-velero logSource="internal/delete/delete_item_action_handler.go:116" namespace=kamap2
time="2023-10-02T08:03:31Z" level=info msg="Removing PV snapshots" backup=20231002-1656-kamap2 controller=backup-deletion deletebackuprequest=kube-storage/20231002-1656-kamap2-db5j7 logSource="pkg/controller/backup_deletion_controller.go:293"
time="2023-10-02T08:03:31Z" level=info msg="Removing pod volume snapshots" backup=20231002-1656-kamap2 controller=backup-deletion deletebackuprequest=kube-storage/20231002-1656-kamap2-db5j7 logSource="pkg/controller/backup_deletion_controller.go:318"
time="2023-10-02T08:03:31Z" level=info msg="Removing snapshot data by data mover" backup=20231002-1656-kamap2 controller=backup-deletion deletebackuprequest=kube-storage/20231002-1656-kamap2-db5j7 logSource="pkg/controller/backup_deletion_controller.go:326"
time="2023-10-02T08:03:31Z" level=info msg="Founding existing repo" backupLocation=default logSource="pkg/repository/ensurer.go:85" repositoryType=kopia volumeNamespace=kamap2
time="2023-10-02T08:03:32Z" level=info msg="Deleted snapshot 90c17bf3125d2c1bfd7585beea09f8d8, namespace: kamap2, repo type: kopia" logSource="pkg/controller/backup_deletion_controller.go:555"
time="2023-10-02T08:03:32Z" level=info msg="Removing local datauploads" backup=20231002-1656-kamap2 controller=backup-deletion deletebackuprequest=kube-storage/20231002-1656-kamap2-db5j7 logSource="pkg/controller/backup_deletion_controller.go:333"
time="2023-10-02T08:03:32Z" level=info msg="Removing backup from backup storage" backup=20231002-1656-kamap2 controller=backup-deletion deletebackuprequest=kube-storage/20231002-1656-kamap2-db5j7 logSource="pkg/controller/backup_deletion_controller.go:353"
time="2023-10-02T08:03:33Z" level=info msg="Removing restores" backup=20231002-1656-kamap2 controller=backup-deletion deletebackuprequest=kube-storage/20231002-1656-kamap2-db5j7 logSource="pkg/controller/backup_deletion_controller.go:359"
time="2023-10-02T08:03:33Z" level=info msg="Reconciliation done" backup=20231002-1656-kamap2 controller=backup-deletion deletebackuprequest=kube-storage/20231002-1656-kamap2-db5j7 logSource="pkg/controller/backup_deletion_controller.go:446"


----- snip

time="2023-10-03T08:00:40Z" level=info msg="Running maintenance on backup repository" backupRepo=kube-storage/kamap2-default-kopia-5mmvt logSource="pkg/controller/backup_repository_controller.go:285"
time="2023-10-03T08:00:41Z" level=warning msg="active indexes [xn0_02dcbd0f786db8b67ca39513209f22a2-s20197d85c6bd7e85121-c1 xn0_72efdaeab14f9ea3f08c14d266a0cb49-sb8453497be0c47b5121-c1 xn0_b384096e694dc3db59aee2a7a71e2ba1-s7ddd096a2027a547121-c1] deletion watermark 0001-01-01 00:00:00 +0000 UTC" logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:101" sublevel=error
time="2023-10-03T08:00:41Z" level=warning msg="active indexes [xn0_02dcbd0f786db8b67ca39513209f22a2-s20197d85c6bd7e85121-c1 xn0_72efdaeab14f9ea3f08c14d266a0cb49-sb8453497be0c47b5121-c1 xn0_b384096e694dc3db59aee2a7a71e2ba1-s7ddd096a2027a547121-c1] deletion watermark 0001-01-01 00:00:00 +0000 UTC" logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:101" sublevel=error
time="2023-10-03T08:00:41Z" level=info msg="Running quick maintenance..." logModule=kopia/maintenance logSource="pkg/kopia/kopia_log.go:94"
time="2023-10-03T08:00:41Z" level=info msg="Running quick maintenance..." logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:94"
time="2023-10-03T08:00:41Z" level=warning msg="active indexes [xn0_02dcbd0f786db8b67ca39513209f22a2-s20197d85c6bd7e85121-c1 xn0_72efdaeab14f9ea3f08c14d266a0cb49-sb8453497be0c47b5121-c1 xn0_b384096e694dc3db59aee2a7a71e2ba1-s7ddd096a2027a547121-c1] deletion watermark 0001-01-01 00:00:00 +0000 UTC" logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:101" sublevel=error
time="2023-10-03T08:00:41Z" level=info msg="Finished quick maintenance." logModule=kopia/maintenance logSource="pkg/kopia/kopia_log.go:94"
time="2023-10-03T08:00:41Z" level=info msg="Finished quick maintenance." logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:94"
time="2023-10-03T08:01:37Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=kube-storage/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:152"
time="2023-10-03T08:01:38Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=kube-storage/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:137"
I1003 08:01:56.843944       1 request.go:690] Waited for 1.045119366s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/hnc.x-k8s.io/v1alpha2?timeout=32s
time="2023-10-03T08:02:47Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=kube-storage/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:152"

It can't delete kopia's object storage from s3(minio).

It can delete well backup/20231002-1656-kamap2 from s3(minio).

[shubham-pampattiwar]

I believe you might be facing this similar issue: #6575
Fix PR: #6587
Fix will be available in velero 1.12.1

[aki-kamada]

@shubham-pampattiwar
Thank you for your comment.
I confirmed this issue and PR that this code already merged at 1.12.0. So, I tried to open ticket #6916.

Regards,
Aki

[mallancrime]

To determine if this is a bug, you can take the following steps:

Review Velero's documentation and release notes to check if there are any known issues related to the version you are using (1.12.0).
Explore Velero's GitHub repository or community forums to see if others have reported a similar issue.
If no solution is found, consider opening a bug ticket to bring attention to the problem. Provide detailed information about your setup, the steps to reproduce the issue, and any error messages you encounter.
Ensure that you include the version details, specific commands used, and any relevant configurations to help the Velero community address the potential bug effectively. This collaborative approach can contribute to resolving the issue and improving the overall functionality of Velero.