How to exclude specific CSI volumes from backup?

[Zalfsten]

I have the same task as #7746: I would like to use the backup facilities of by PostgreSQL operator instead of velero to backup the persistent volume of my database.

It's no problem to introduce a backup hook to trigger an operator backup, but still velero will also create a normal CSI snapshot and store it in S3 -- which wastes a lot resources. How do I exclude my PostgreSQL volumes from backup?

I'm using the builtin velero CSI plugin for volume snapshots.

The annotation backup.velero.io/backup-volumes-excludes has no effect for me. It could be that it's not evaluated by the CSI plugin.

Any ideas how to fix this?

[sseago]

backup.velero.io/backup-volumes-excludes wouldn't help you -- that is intended to opt out of fs-backup (when that's the default) and use snapshots instead.

You might be able to use volume policies (applying the skip policy) if the available filters are able to narrow down to just the PostgresQL volumes, although I don't know if that's enough to uniquely describe the volumes of interest in your case. Here's the doc for that:
https://velero.io/docs/main/resource-filtering/#resource-policies

[Zalfsten]

Thanks for your answer, @sseago

I've thought about that. I think I could use this to exclude the backup of the PostgreSQL PVCs, but actually I need them for a full restore. I would like to backup all kubernetes manifests -- just not the PostgreSQL volume contents.

Actually I do not understand good enough, how velero works internally. When does velero trigger the CSI plugin? -- When it encounters a PVC to backup? Or when it encounters a PV? Or maybe a Pod?
In case it's the PV, I could exclude the PVs from backup, because they get created automatically when I create/restore a PVC...

[Zalfsten]

Oh... maybe it' possible... I wasn't aware that there is something like the skip policy. I will try out

[Zalfsten]

Finally I've had a look. It doesn't work out of the box, because all resource policy rules I can define depend on storage characteristics -- like size or storageClass. I would like to define rules depending on the application I run -- like PostgreSQL.
Of course I could introduce a new storageClass as workaround that I only use for PostgreSQL volume, so I can use this storage class to define a skip policy. I will try that out... But I'm really annoyed, because all I do for some time now is lining up workarounds to get my nextcloud backup running.

[Zalfsten]

ok, it works this way, though it's an ugly workaround.

[sseago]

@Zalfsten "Of course I could introduce a new storageClass as workaround that I only use for PostgreSQL volume, so I can use this storage class to define a skip policy. ... ok, it works this way, though it's an ugly workaround."

I agree, it is an ugly workaround. Ideally we should extend the selection mechanism for volume policies to allow label selectors -- then you'd just need to label your PG volumes appropriately and add the label to the skip policy. It doesn't work that way today, but it's certainly something we could add in the future.

[mame-leadec]

@Zalfsten Could you elaborate on your solution? I’m encountering the same issue. I'm trying to use the CloudNativeOperator alongside Velero.

While backup and restore hooks aren't an issue, I'd like to exclude the backup of any PODs and PVCs associated with 'postgresql'.

I also tried to write a custom velero plugin:

1. Which either supresses the resources --> doesnt work well in combination with CSI Plugin

2. Or which adds the annotation "velero.io/exclude-from-backup=true" to any resource associated with 'postgresql' --> but it applies it only to the persisted resources, not the live Kubernetes resources.

[Zalfsten]

@mame-leadec, sorry for my late answer.

First of all you have to distinguish between the the backup of kubernetes objects (pods, services, pvc and the like) -- which can be represented as yaml manifests -- and the backup of volume content. The former is the main purpose of velero, the latter can also be accomplished using plugins.
As far as I know, volume backup is always triggered by the corresponding kubernetes object backup -- although I cannot tell if the PV, the PVC or the POD backup triggers CSI volume snapshots.
You can exclude kubernetes objects from backup, this will then implicitly exclude the backup/snapshot of associated volumes. But once the CSI snapshot plugin is active, it's not easy to turn off backups for specific volumes while still retaining the corresponding kubernetes object backup.
This is the scenario I was looking a solution for.

The (workaround) solution suggested by @sseago makes use of velero resource policies that can be assigned to Backup or BackupSchedule objects. A skip policy will achieve what I need. Unfortunately there are no means to assign resource policies selectively to single objects. So the only solution I've found, is to define a special StorageClass that will be excluded from volume snapshots in general.

The solution roughly looks like this:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
  name: postgres
parameters: {}
provisioner: ""
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "false"
  labels:
    velero.io/csi-volumesnapshot-class: "true"
  name: postgres-backup-snapshots
parameters: {}
driver: ""
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: skip-postgres-volume-backup
  namespace: velero
data:
  skip-postgres-volume-backup.yaml: |
    version: v1
    volumePolicies:
      - conditions:
          storageClass:
            - postgres
        action:
          type: skip
---
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: ""
  namespace: velero
spec:
  schedule: "1 1 * * *"
  template:
    storageLocation: default
    resourcePolicy:
      kind: configmap
      name: skip-postgres-volume-backup
    hooks:
      resources:
        - name: backup-postgresql-master
          labelSelector:
            matchLabels:
              application: spilo
              spilo-role: master
          pre:
            - exec:
                container: postgres
                command:
                  - "/usr/bin/bash"
                  - "-c"
                  - "su -l postgres -c 'envdir /run/etc/wal-e.d/env /scripts/postgres_backup.sh /home/postgres/pgdata/pgroot/data'"

[mame-leadec]

@Zalfsten Thank you for your reply. Maybe I should consider your solution. However, that would mean additional complexity on our storage side ...

A side question: this solution only excludes CSI volumes, how do you handle resources like pods when restoring?