Backing up a cluster with PV makes the entire cluster unusable

[misteruly]

version velero-v1.5.2

Minio deployed separately

❯ docker run -p 9000:9000
--name minio1
-v /private/mnt/data:/data
-e "MINIO_ACCESS_KEY=minio"
-e "MINIO_SECRET_KEY=minio123"
minio/minio server /data

velero install
--provider aws
--bucket backup
--secret-file examples/minio/credentials-velero
--use-volume-snapshots=false
--plugins velero/velero-plugin-for-aws:v1.0.0
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.0.3:9000

deploy nginx-pv

---

apiVersion: v1
kind: Namespace
metadata:
name: nginx-example
labels:
app: nginx

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-logs
namespace: nginx-example
labels:
app: nginx
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Mi

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: nginx-example
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
annotations:
pre.hook.backup.velero.io/container: fsfreeze
pre.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--freeze", "/var/log/nginx"]'
post.hook.backup.velero.io/container: fsfreeze
post.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--unfreeze", "/var/log/nginx"]'
spec:
volumes:
- name: nginx-logs
persistentVolumeClaim:
claimName: nginx-logs
containers:
- image: nginx:1.17.6
name: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/var/log/nginx"
name: nginx-logs
readOnly: false
- image: ubuntu:bionic
name: fsfreeze
securityContext:
privileged: true
volumeMounts:
- mountPath: "/var/log/nginx"
name: nginx-logs
readOnly: false
command:
- "/bin/bash"
- "-c"
- "sleep infinity"

apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: my-nginx
namespace: nginx-example
spec:
ports:

• port: 80
  targetPort: 80
  selector:
  app: nginx
  type: LoadBalancer

backup nginx

velero backup create nginx-example-backup --include-namespaces nginx-example

velero logs

time="2020-12-29T05:58:16Z" level=info msg="Getting items for group" backup=velero/nginx-example-backup group=servicemesh.kubesphere.io/v1alpha2 logSource="pkg/backup/item_collector.go:76"
time="2020-12-29T05:58:16Z" level=info msg="Getting items for resource" backup=velero/nginx-example-backup group=servicemesh.kubesphere.io/v1alpha2 logSource="pkg/backup/item_collector.go:165" resource=servicepolicies
time="2020-12-29T05:58:16Z" level=info msg="Listing items" backup=velero/nginx-example-backup group=servicemesh.kubesphere.io/v1alpha2 logSource="pkg/backup/item_collector.go:291" namespace=nginx-example resource=servicepolicies
time="2020-12-29T05:58:16Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup group=servicemesh.kubesphere.io/v1alpha2 logSource="pkg/backup/item_collector.go:297" namespace=nginx-example resource=servicepolicies
time="2020-12-29T05:58:16Z" level=info msg="Getting items for resource" backup=velero/nginx-example-backup group=servicemesh.kubesphere.io/v1alpha2 logSource="pkg/backup/item_collector.go:165" resource=strategies
time="2020-12-29T05:58:16Z" level=info msg="Listing items" backup=velero/nginx-example-backup group=servicemesh.kubesphere.io/v1alpha2 logSource="pkg/backup/item_collector.go:291" namespace=nginx-example resource=strategies
time="2020-12-29T05:58:16Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup group=servicemesh.kubesphere.io/v1alpha2 logSource="pkg/backup/item_collector.go:297" namespace=nginx-example resource=strategies
time="2020-12-29T05:58:16Z" level=info msg="Getting items for group" backup=velero/nginx-example-backup group=app.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76"
time="2020-12-29T05:58:16Z" level=info msg="Getting items for resource" backup=velero/nginx-example-backup group=app.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=applications
time="2020-12-29T05:58:16Z" level=info msg="Listing items" backup=velero/nginx-example-backup group=app.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=nginx-example resource=applications
time="2020-12-29T05:58:16Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup group=app.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=nginx-example resource=applications
time="2020-12-29T05:58:16Z" level=info msg="Getting items for group" backup=velero/nginx-example-backup group=snapshot.storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:76"
time="2020-12-29T05:58:16Z" level=info msg="Getting items for resource" backup=velero/nginx-example-backup group=snapshot.storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=volumesnapshots
time="2020-12-29T05:58:16Z" level=info msg="Listing items" backup=velero/nginx-example-backup group=snapshot.storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=nginx-example resource=volumesnapshots
time="2020-12-29T05:58:16Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup group=snapshot.storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=nginx-example resource=volumesnapshots
time="2020-12-29T05:58:16Z" level=info msg="Getting items for resource" backup=velero/nginx-example-backup group=snapshot.storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=volumesnapshotcontents
time="2020-12-29T05:58:16Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup group=snapshot.storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=volumesnapshotcontents
time="2020-12-29T05:58:16Z" level=info msg="Getting items for resource" backup=velero/nginx-example-backup group=snapshot.storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=volumesnapshotclasses
time="2020-12-29T05:58:16Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-example-backup group=snapshot.storage.k8s.io/v1beta1 logSource="pkg/backup/item_collector.go:192" resource=volumesnapshotclasses
time="2020-12-29T05:58:16Z" level=info msg="Getting items for group" backup=velero/nginx-example-backup group=extensions/v1beta1 logSource="pkg/backup/item_collector.go:76"
time="2020-12-29T05:58:16Z" level=info msg="Getting items for resource" backup=velero/nginx-example-backup group=extensions/v1beta1 logSource="pkg/backup/item_collector.go:165" resource=ingresses
time="2020-12-29T05:58:16Z" level=info msg="Listing items" backup=velero/nginx-example-backup group=extensions/v1beta1 logSource="pkg/backup/item_collector.go:291" namespace=nginx-example resource=ingresses
time="2020-12-29T05:58:16Z" level=info msg="Retrieved 0 items" backup=velero/nginx-example-backup group=extensions/v1beta1 logSource="pkg/backup/item_collector.go:297" namespace=nginx-example resource=ingresses
time="2020-12-29T05:58:16Z" level=info msg="Collected 34 items matching the backup spec from the Kubernetes API (actual number of items backed up may be more or less depending on velero.io/exclude-from-backup annotation, plugins returning additional related items to back up, etc.)" backup=velero/nginx-example-backup logSource="pkg/backup/backup.go:302" progress=
time="2020-12-29T05:58:16Z" level=info msg="Processing item" backup=velero/nginx-example-backup logSource="pkg/backup/backup.go:378" name=nginx-deployment-f96b7fd86-fvgxn namespace=nginx-example progress= resource=pods
time="2020-12-29T05:58:16Z" level=info msg="Backing up item" backup=velero/nginx-example-backup logSource="pkg/backup/item_backupper.go:121" name=nginx-deployment-f96b7fd86-fvgxn namespace=nginx-example resource=pods
time="2020-12-29T05:58:16Z" level=info msg="running exec hook" backup=velero/nginx-example-backup hookCommand="[/sbin/fsfreeze --freeze /var/log/nginx]" hookContainer=fsfreeze hookName="" hookOnError=Fail hookPhase=pre hookSource=annotation hookTimeout="{30s}" hookType=exec logSource="pkg/podexec/pod_command_executor.go:124" name=nginx-deployment-f96b7fd86-fvgxn namespace=nginx-example resource=pods

Now the cluster is unavailable and the CPU value has soared

load average: 61.92, 57.97, 40.27

4202 root 20 0 774512 575992 43256 S 11.0 3.5 11:41.03 kube-apiserver
1112 root 20 0 2923300 118736 35792 S 9.3 0.7 7:54.55 kubelet 1492 root 20 0 2529364 144016 31352 S 3.0 0.9 4:04.85 dockerd
13810 root 20 0 1781228 42388 18064 S 2.7 0.3 4:00.48 calico-node 3311 root 20 0 10.1g 107312 22760 D 1.0 0.7 3:12.25 etcd
9973 10001 20 0 142520 32372 15740 S 0.7 0.2 0:11.58 metrics-server

Don't know what causes it

[nrb]

@misteruly Can you remove this section and re-try it?

 annotations:
pre.hook.backup.velero.io/container: fsfreeze
pre.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--freeze", "/var/log/nginx"]'
post.hook.backup.velero.io/container: fsfreeze
post.hook.backup.velero.io/command: '["/sbin/fsfreeze", "--unfreeze", "/var/log/nginx"]'

The freezing may have caused some unexpected behavior.

Also, is this a local installation on your computer?

[misteruly]

@nrb This is the test environment.How to migrate PV data from the original cluster to the new cluster and associate it with POD???

[nrb]

@misteruly Velero will automatically associate pods with their PVs, but the fsfreeze hooks are not always necessary. They are only needed with restic.

[carlisia]

This discussion has been moved to an issue: #3345.