One backup stuck 'In progress' and subsequent ones in 'new'

[rombert]

Hi,

I have a basic Velero setup with helm and AWS (actually going to Linode Object Storage ). This used to be working but broke at some point, I think when I upgraded from 1.3.1 to 1.5.3 . The helm deployment (with FluxCD) is

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: velero
  namespace: lmn-system
spec:
  releaseName: velero
  chart:
    repository: https://vmware-tanzu.github.io/helm-charts
    name: velero
    version: 2.16.0
  values:
    image:
      tag: v1.5.3
    configuration:
      logLevel: debug
      provider: aws
      backupStorageLocation:
        name: default
        bucket: bucket003
        config:
          region: eu-central-1
          s3Url: https://eu-central-1.linodeobjects.com
    snapshotsEnabled: false
    deployRestic: true
    initContainers:
    - name: velero-plugin-for-aws
      image: velero/velero-plugin-for-aws:v1.1.0
      volumeMounts:
      - mountPath: /target
        name: plugins

and I have 'hourly' and 'daily' schedules set up.

I noticed at some point that I have lots of backups in the New state, and that they were not getting processed or expired. I tried deleting them with the velero CLI tool, but they would not get processed. I checked the velero deployment and it was running with no issues. Then I resorted to deleting all backups using kubectl delete backup --all and restarting velero. That removed most of the backups, except a couple that were marked as PartiallyFailed.

Right now I am back again with:

• one backup set to in progress (started about 2 hours ago)
• a number of backups set to new

The only potential problems I see in the velero logs are of the form:

time="2021-04-09T05:51:50Z" level=error msg="Error backing up item" backup=lmn-system/hourly-20210408224631 error="timed out waiting for all PodVolumeBackups to complete" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/restic/backupper.go:172" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:431" name=prometheus-server-55fd8f9f88-wddqx

From the restic logs this might be interesting:

time="2021-04-08T21:51:38Z" level=info msg="Backup starting" backup=lmn-system/manual001 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:191" name=manual001-v76qm namespace=lmn-system
time="2021-04-08T21:51:38Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=lmn-system/manual001 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:342" name=manual001-v76qm namespace=lmn-system pvcUID=b97384ff-0c4a-4071-96fc-d9a9f02d0958
time="2021-04-08T21:51:38Z" level=info msg="No completed pod volume backup found for PVC" backup=lmn-system/manual001 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:372" name=manual001-v76qm namespace=lmn-system pvcUID=b97384ff-0c4a-4071-96fc-d9a9f02d0958
time="2021-04-08T21:51:38Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=lmn-system/manual001 controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:283" name=manual001-v76qm namespace=lmn-system

I am unsure what to do next, any help would be appreciated. Thanks!