Not restoring from the backup I replicated from another minio (PartiallyFailed)

[zynpsnltrkk]

I have a velero deployment in my Kubernetes cluster. this Velero is looking at Minio storage running in a separate VM. and this Minio is also working replicated from another Minio. I can get backup an application already in my kubernetes cluster and then restore it successfully. but I am getting the following error while restoring the backups I have provided from other Minio.

❯ velero backup get -n test
NAME             STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
test-6           Completed   0        0          2021-03-25 16:37:43 +0300 +03   29d       default            <none>

❯ velero restore create restore-test-6 --from-backup test-6 --restore-volumes -n zeynep
Restore request "restore-test-6" submitted successfully.
Run `velero restore describe restore-test-6` or `velero restore logs restore-test-6` for more details.

❯ velero restore get -n zeynep
NAME             BACKUP   STATUS            STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
restore-test-6   test-6   PartiallyFailed   2021-03-25 20:30:24 +0300 +03   2021-03-25 20:30:25 +0300 +03   4        11         2021-03-25 20:30:23 +0300 +03   <none>

❯ velero restore describe restore-test-6 -n zeynep--details
Name:         restore-test-6
Namespace:    zeynep
Labels:       <none>
Annotations:  <none>

Phase:  PartiallyFailed (run 'velero restore logs restore-test-6' for more information)

Started:    2021-03-25 20:20:55 +0300 +03
Completed:  2021-03-25 20:20:57 +0300 +03
Warnings:   <error getting warnings: Get "http://x.x.x.x:18050/velero/restores/restore-test-6/restore-restore-test-6-results.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20210325%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20210325T172411Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=82706676bfbaa9d655e3da160f7377b86a1cf0d3c222888d55c4d6d12a937526": dial tcp x.x.x.x:18050: connect: connection refused>

Errors:  <error getting errors: Get "http://x.x.x.x:18050/velero/restores/restore-test-6/restore-restore-test-6-results.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20210325%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20210325T172411Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=82706676bfbaa9d655e3da160f7377b86a1cf0d3c222888d55c4d6d12a937526": dial tcp x.x.x.x:18050: connect: connection refused>

Backup:  test-6

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  true

Restic Restores:
  Failed:
    zeynep/mysqlha-0: conf, data, scripts
    zeynep/mysqlha-1: conf, data
    zeynep/mysqlha-2: conf, data

It s the velero pods logs

 ~ kc logs velero -n zeynep 
time="2021-03-25T14:33:38Z" level=error msg="unable to successfully complete restic restores of pod's volumes" error="[pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=s3:http://x.x.x.x:18050/velero/restic/zeynep --password-file=/tmp/velero-restic-credentials-zeynep157736290 --cache-dir=/scratch/.cache/restic 71c9b412 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: unable to load index 3393b879: ciphertext verification failed\n: exit status 1, pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=s3:http://x.x.x.x:18050/velero/restic/zeynep --password-file=/tmp/velero-restic-credentials-zeynep114312537 --cache-dir=/scratch/.cache/restic 285ab961 --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: unable to load index 26faab8e: ciphertext verification failed\n: exit status 1, pod volume restore failed: error restoring volume: error running restic restore, cmd=restic restore --repo=s3:http://x.x.x.x:18050/velero/restic/zeynep --password-file=/tmp/velero-restic-credentials-zeynep448406756 --cache-dir=/scratch/.cache/restic fad31ceb --target=., stdout=, stderr=: error getting snapshot size: error running command, stderr=Fatal: unable to load index 4a64e9e5: ciphertext verification failed\n: exit status 1]" logSource="pkg/restore/restore.go:1296" restore=zeynep/test5-restore
time="2021-03-25T17:46:22Z" level=error msg="Error getting backup metadata from backup store" backup=test-6 backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = error getting object backups/test-6/velero-backup.json: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 166FA6F9B872B80B, host id: " error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store.go:261" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).GetBackupMetadata" logSource="pkg/controller/backup_sync_controller.go:209"

Does anyone have an idea about this?

[zynpsnltrkk]

solution: If you are using a minio cluster with distributed erasure mode, you should give permission to all targets from all sources you replicate. The main problem was in replication.

[vivekwalia28]

Could you elaborate it a bit more as I am facing the same issue and ran out of all the ideas of restore getting complete.
Thanks!

[zynpsnltrkk]

hi @vivekwalia28 my environment was running on aws but on-prem style. I had two clusters, they both was working on 3 nodes and different networks. I was added all the source machines public ips to the security group of the target machines over the relevant port. Then I connected to the target machines and gave the same permission over the firewall for all source public ips.

ufw allow from source-ip to any port minio-port

I had given these privileges to a single source from a single target machine before. Since I didn't use load balancer, it couldn't actually carry all the data while replication from source to target. so even though I could see the backup on the target k8s cluster, when I restored it, it was partially failed.

I compared the data from source and target to understand. The commands below can help with this.

mc admin trace -v source_minio
mc admin info source_minio
mc ls source_minio