can't restore the ebs volume properly #3247

ozbillwang · 2020-12-17T20:08:29Z

ozbillwang
Dec 17, 2020

What steps did you take and what happened:
Deploy a jenkins helm chart with below command

$ cat storageClass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: jenkins-standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
  - debug
volumeBindingMode: Immediate

$ kubectl apply -f storageClass.yaml

# enable persistence to true and set the storageClass in its helm chart's `values.yaml`

persistence:
  enabled: true
  storageClass: jenkins-standard

# deploy the jenkins helm chart
$ kubectl create ns jenkins
$ kubens jenkins
$ helm install my-jenkins jenkins/jenkins -f values.yaml

# Do the port-forward, I can login jenkins, then create a new job and run it.

# Do a velero backup
$ elero backup create firstbackup --include-namespaces jenkins

$ velero backup get
NAME          STATUS      ERRORS   WARNINGS   CREATED                          EXPIRES   STORAGE LOCATION   SELECTOR
firstbackup   Completed   0        0          2020-12-18 05:45:37 +1100 AEDT   29d       default            <none>

# So the backup is successful, I can see a new ebs snapshop create as well in aws console

# but when run the restore, I got error "partiallyfailed"
$ velero restore get
NAME                         BACKUP        STATUS            STARTED                          COMPLETED                        ERRORS   WARNINGS   CREATED                          SELECTOR
firstbackup-20201218055633   firstbackup   PartiallyFailed   2020-12-18 05:56:35 +1100 AEDT   2020-12-18 06:06:36 +1100 AEDT   1        0          2020-12-18 05:56:35 +1100 AEDT   <none>

$ velero restore logs firstbackup-20201218055633
time="2020-12-17T18:56:35Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:467" restore=velero/firstbackup-20201218055633
time="2020-12-17T18:56:35Z" level=info msg="Starting restore of backup velero/firstbackup" logSource="pkg/restore/restore.go:363" restore=velero/firstbackup-20201218055633
time="2020-12-17T18:56:35Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:401" resource=volumesnapshotclass.snapshot.storage.k8s.io restore=velero/firstbackup-20201218055633
time="2020-12-17T18:56:35Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:401" resource=volumesnapshotcontents.snapshot.storage.k8s.io restore=velero/firstbackup-20201218055633
time="2020-12-17T18:56:35Z" level=info msg="Skipping restore of resource because it cannot be resolved via discovery" logSource="pkg/restore/restore.go:401" resource=volumesnapshots.snapshot.storage.k8s.io restore=velero/firstbackup-20201218055633
time="2020-12-17T18:56:35Z" level=info msg="Restoring cluster level resource 'persistentvolumes'" logSource="pkg/restore/restore.go:726" restore=velero/firstbackup-20201218055633

# Jenkins is in pending status
$ kk get all
NAME               READY   STATUS    RESTARTS   AGE
pod/my-jenkins-0   0/2     Pending   0          60m

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/my-jenkins         ClusterIP   100.64.32.232    <none>        8080/TCP    60m
service/my-jenkins-agent   ClusterIP   100.66.220.202   <none>        50000/TCP   60m

NAME                          READY   AGE
statefulset.apps/my-jenkins   0/1     60m

NAME                              SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/my-jenkins-backup   0 2 * * *   False     0        <none>          60m

What did you expect to happen:

Jenkins get restored.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

kubectl logs deployment/velero -n velero
velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
velero backup logs <backupname>
velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
velero restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

Velero version (use velero version): 1.5.2
Velero features (use velero client config get features):
Kubernetes version (use kubectl version): 1.18.12
Kubernetes installer & version: kops 1.18.2
Cloud provider or hardware configuration: aws
OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

👍 for "I would like to see this bug fixed as soon as possible"
👎 for "There are more important bugs to focus on right now"

Answered by carlisia

Jan 25, 2021

This discussion has been moved to an issue: #3342.

View full answer

carlisia · 2020-12-17T21:39:16Z

carlisia
Dec 17, 2020

Hi @ozbillwang, can we please have the output of the following cmds:

velero describe restore firstbackup-20201218055633

kubectl logs deployment/velero -n velero

0 replies

ozbillwang · 2020-12-17T21:57:17Z

ozbillwang
Dec 17, 2020
Author

The original restore job has been deleted, I paste another one, they have same error

$ velero describe restore firstbackup-20201218071742
Name:         firstbackup-20201218071742
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  PartiallyFailed (run 'velero restore logs firstbackup-20201218071742' for more information)

Started:    2020-12-18 07:17:43 +1100 AEDT
Completed:  2020-12-18 07:27:44 +1100 AEDT

Errors:
  Velero:     <none>
  Cluster:  error waiting on in-cluster persistentvolume pvc-0397eba7-0c28-4941-918e-7bf4c3a36a95: timed out waiting for the condition
  Namespaces: <none>

Backup:  firstbackup

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

the velero log is very long, i attach here.

velero.log.gz

0 replies

ozbillwang · 2020-12-17T21:58:21Z

ozbillwang
Dec 17, 2020
Author

I found another error, it is related to the volume restore problem, my feeling is, velero can't work with KOPS properly

$ kk describe pod/wordpress-6857459697-k7zx8
Name:           wordpress-6857459697-k7zx8
Namespace:      staging
Priority:       0
Node:           <none>
Labels:         app=wordpress
                pod-template-hash=6857459697
                tier=frontend
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/wordpress-6857459697
Containers:
  wordpress:
    Image:      wordpress:4.8-apache
    Port:       80/TCP
    Host Port:  0/TCP
    Environment:
      WORDPRESS_DB_HOST:      wordpress-mysql
      WORDPRESS_DB_PASSWORD:  <set to the key 'password' in secret 'mysql-pass'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nt488 (ro)
      /var/www/html from wordpress-persistent-storage (rw)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  wordpress-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  wp-pv-claim
    ReadOnly:   false
  default-token-nt488:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nt488
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  59m (x4 over 59m)    default-scheduler  0/4 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 2 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  59m (x2 over 59m)    default-scheduler  0/4 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  57m (x5 over 58m)    default-scheduler  0/5 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 3 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  4m2s (x38 over 57m)  default-scheduler  0/5 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 4 pod has unbound immediate PersistentVolumeClaims.

I follow the online document https://www.eksworkshop.com/intermediate/280_backup-and-restore/, get same error. This time it deploys mysql with wordpress. The only difference is I don't use EKS, my kubernetes cluster is installed by KOPS.

0 replies

ozbillwang · 2020-12-18T01:26:19Z

ozbillwang
Dec 18, 2020
Author

I create a new KOPS cluster, and do the lab (https://www.eksworkshop.com/intermediate/280_backup-and-restore/) again, it works.

So maybe my first setup with jenkins helm chart are some issues, I need dig out.

0 replies

ozbillwang · 2020-12-18T02:15:37Z

ozbillwang
Dec 18, 2020
Author

create another new kops cluster, do the jenkins change again, get same issue. Not sure why.

0 replies

ozbillwang · 2020-12-18T02:27:03Z

ozbillwang
Dec 18, 2020
Author

I fixed the issue.

When delete jenkins helm chart, the jenkins home (/var/jenkins_home) is a AWS EBS, it is not automatically cleaned. After manually delete it from AWS console, I can successfully restore jenkins with velero, all setting is back.

That's so cool.

And the restore made the helm list command work as well. Too good than I expected.

0 replies

ozbillwang · 2020-12-18T02:28:45Z

ozbillwang
Dec 18, 2020
Author

By the way, why I can't keep the exist EBS to get restore successfully?

I think it could be a bug?

0 replies

carlisia · 2021-01-25T21:16:05Z

carlisia
Jan 25, 2021

This discussion has been moved to an issue: #3342.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

can't restore the ebs volume properly #3247

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 8 comments

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

can't restore the ebs volume properly #3247

ozbillwang Dec 17, 2020

Replies: 8 comments

carlisia Dec 17, 2020

ozbillwang Dec 17, 2020 Author

ozbillwang Dec 17, 2020 Author

ozbillwang Dec 18, 2020 Author

ozbillwang Dec 18, 2020 Author

ozbillwang Dec 18, 2020 Author

ozbillwang Dec 18, 2020 Author

carlisia Jan 25, 2021

ozbillwang
Dec 17, 2020

carlisia
Dec 17, 2020

ozbillwang
Dec 17, 2020
Author

ozbillwang
Dec 17, 2020
Author

ozbillwang
Dec 18, 2020
Author

ozbillwang
Dec 18, 2020
Author

ozbillwang
Dec 18, 2020
Author

ozbillwang
Dec 18, 2020
Author

carlisia
Jan 25, 2021