Skip to content

Commit

Permalink
Merge pull request #1687 from zimnx/mz/restore-docs-1.11
Browse files Browse the repository at this point in the history 
[v1.11] Update documentation about restoring from backup
  • Loading branch information
@scylla-operator-bot
scylla-operator-bot[bot] authored Jan 15, 2024
2 parents 7b2ad54 + 069132d commit 3dd1616
Showing 1 changed file with 187 additions and 59 deletions.
246 changes: 187 additions & 59 deletions docs/source/nodeoperations/restore.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,88 +2,216 @@

This procedure will describe how to restore from backup taken using [Scylla Manager](../manager.md) to a fresh **empty** cluster of any size.

First identify to which snapshot you want to restore. To get list of available snapshot execute following command on Scylla Manager Pod.
```bash
sctool backup list -c <CLUSTER_ID> --all-clusters -L <BACKUP_LOCATION>
```
:::{caution}
Due to a [bug](https://github.com/scylladb/scylla-manager/issues/3679) in Scylla Manager not supporting ScyllaClusters having both non-TLS and TLS CQL ports open, you have to disable the TLS certificate management
in Scylla Operator.
This step will no longer be required, when the bug is fixed.

Where:
* `CLUSTER_ID` - is a name of a cluster or ID under which ScyllaCluster was registered. You can find it in ScyllaCluster Status.
* `BACKUP_LOCATION` - is a location where backup is stored. For example, for bucket called `backups` stored in AWS S3, location is `s3:backups`.
To disable TLS certificate management in Scylla Operator add `--feature-gates="AutomaticTLSCertificates=false"` flag to Scylla Operator deployment.

```bash
sctool backup list -c simple-cluster --all-clusters -L s3:backups
Snapshots:
- sm_20201227144037UTC (409MiB)
- sm_20201228145917UTC (434MiB)
Keyspaces:
- users (9 tables)
- system_auth (2 tables)
- system_distributed (3 tables)
- system_schema (13 tables)
- system_traces (5 tables)
```console
kubectl -n scylla-operator patch deployment/scylla-operator --type=json -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--feature-gates=AutomaticTLSCertificates=false"}]'
```

To get the list of files use:
:::

In the following example, the ScyllaCluster, which was used to take the backup, is called `source`. Backup will be restored into the ScyllaCluster named `target`.

::::{tab-set}
:::{tab-item} Source ScyllaCluster
```yaml
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
name: source
namespace: scylla
spec:
agentVersion: 3.2.5
version: 5.4.1
developerMode: true
backups:
- name: foo
location:
- s3:source-backup
keyspace:
- '*'
datacenter:
name: us-east-1
racks:
- name: us-east-1a
members: 1
storage:
capacity: 1Gi
resources:
limits:
cpu: 1
memory: 1Gi
```
:::
:::{tab-item} Target ScyllaCluster
```yaml
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
name: target
namespace: scylla
spec:
agentVersion: 3.2.5
version: 5.4.1
developerMode: true
datacenter:
name: us-east-1
racks:
- name: us-east-1a
members: 1
storage:
capacity: 1Gi
resources:
limits:
cpu: 1
memory: 1Gi
```
:::
::::
Make sure your target cluster is already registered in Scylla Manager. To get a list of all registered clusters, execute the following command:
```console
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool cluster list
+--------------------------------------+---------------------------------------+---------+-----------------+
| ID | Name | Port | CQL credentials |
+--------------------------------------+---------------------------------------+---------+-----------------+
| af1dd5cd-0406-4974-949f-dc9842980080 | scylla/target | default | set |
| ebd82268-efb7-407e-a540-3619ae053778 | scylla/source | default | set |
+--------------------------------------+---------------------------------------+---------+-----------------+
```

```bash
sctool backup files -c <CLUSTER_ID> -L <BACKUP_LOCATION> -T <SNAPSHOT_TAG>
Identify the tag of a snapshot which you want to restore. To get a list of all available snapshots, execute following command:
```console
kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool backup list -c <CLUSTER_ID> --all-clusters -L <BACKUP_LOCATION>
```

Where:
* `SNAPSHOT_TAG` - name of snapshot you want to restore.
* `CLUSTER_ID` - the name or ID of a registered cluster with access to `BACKUP_LOCATION`.
* `BACKUP_LOCATION` - the location in which the backup is stored.

In this example, `BACKUP_LOCATION` is `s3:source-backup`. Use the name of cluster which has access to the backup location for `CLUSTER_ID`.
In this example, it's `scylla/target`.

```console
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool backup list -c scylla/target --all-clusters -L s3:source-backup
backup/ff36d7e0-af2e-458c-afe6-868e0f3396b2
Snapshots:
- sm_20240105115931UTC (409MiB, 1 nodes)
Keyspaces:
- system_schema (15 tables)
- users (9 tables)

Before we start restoring the data, we have to restore the schema.
The first output line is a path to schemas archive, for example:
```bash
s3://backups/backup/schema/cluster/ed63b474-2c05-4f4f-b084-94541dd86e7a/task_287791d9-c257-4850-aef5-7537d6e69d90_tag_sm_20201228145917UTC_schema.tar.gz ./
```

To download this archive you can use AWS CLI tool `aws s3 cp`.
In the below commands, we are restoring the `sm_20240105115931UTC` snapshot. Replace it with a tag of a snapshot that you want to restore.
Restoring consist of two steps. First, you'll restore the schema, and then the data.
To restore schema, create a restore task manually on target ScyllaCluster by executing following command:
```console
kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c <CLUSTER_ID> -L <BACKUP_LOCATION> -T <SNAPSHOT_TAG> --restore-schema
```

This archive contains a single CQL file for each keyspace in the backup.
```bash
tar -ztvf task_287791d9-c257-4850-aef5-7537d6e69d90_tag_sm_20201228145917UTC_schema.tar.gz
-rw------- 0/0 12671 2020-12-28 13:17 users.cql
-rw------- 0/0 2216 2020-12-28 13:17 system_auth.cql
-rw------- 0/0 921 2020-12-28 13:17 system_distributed.cql
-rw------- 0/0 12567 2020-12-28 13:17 system_schema.cql
-rw------- 0/0 4113 2020-12-28 13:17 system_traces.cql
Where:
* `CLUSTER_ID` - a name or ID of a cluster you want to restore into.
* `BACKUP_LOCATION` - the location in which the backup is stored.
* `SNAPSHOT_TAG` - a tag of a snapshot that you want to restore.

When the task is created, the command will output the ID of a restore task.
```console
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c scylla/target -L s3:source-backup -T sm_20240105115931UTC --restore-schema
restore/57228c52-7cf6-4271-8c8d-d446ff160747
```

Extract this archive and copy each schema file to one of the cluster Pods by:
```bash
kubectl -n scylla cp users.cql simple-cluster-us-east-1-us-east-1a-0:/tmp/users.cql -c scylla
Use the following command to check progress of the restore task:
```console
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool progress -c scylla/target restore/57228c52-7cf6-4271-8c8d-d446ff160747
Restore progress
Run: 0dd20cdf-abc4-11ee-951c-6e7993cf42ed
Status: DONE - restart required (see restore docs)
Start time: 05 Jan 24 12:15:02 UTC
End time: 05 Jan 24 12:15:09 UTC
Duration: 6s
Progress: 100% | 100%
Snapshot Tag: sm_20240105115931UTC

+---------------+-------------+----------+----------+------------+--------+
| Keyspace | Progress | Size | Success | Downloaded | Failed |
+---------------+-------------+----------+----------+------------+--------+
| system_schema | 100% | 100% | 214.150k | 214.150k | 214.150k | 0 |
+---------------+-------------+----------+----------+------------+--------+
```

To import schema simply execute:
```bash
kubectl -n scylla exec simple-cluster-us-east-1-us-east-1a-0 -c scylla -- cqlsh -f /tmp/users.cql
As suggested in the progress output, you will need to execute a rolling restart of the ScyllaCluster.
```console
kubectl patch scyllacluster target --type merge -p '{"spec": {"forceRedeploymentReason": "schema restored"}}'
```

Once the schema is recreated we can proceed to downloading data files.
Use the following commands to wait until restart is finished:
```console
$ kubectl wait --for='condition=Progressing=False' -n scylla scyllaclusters.scylla.scylladb.com/target
scyllacluster.scylla.scylladb.com/target condition met

First let's save a list of snapshot files to file called `backup_files.out`:
$ kubectl wait --for='condition=Degraded=False' -n scylla scyllaclusters.scylla.scylladb.com/target
scyllacluster.scylla.scylladb.com/target condition met

```bash
kubectl -n scylla-manager exec deployment.apps/scylla-manager-controller -- sctool backup files -c simple-cluster -L s3:backups -T sm_20201228145917UTC > backup_files.out
$ kubectl wait --for='condition=Available=True' -n scylla scyllaclusters.scylla.scylladb.com/target
scyllacluster.scylla.scylladb.com/target condition met
```

We will be using `sstableloader` to restore data. `sstableloader` needs a specific directory structure to work namely: `<keyspace>/<table>/<contents>`
To create this directory structure and download all the files execute these commands:
```bash
mkdir snapshot
cd snapshot
# Create temporary directory structure.
cat ../backup_files.out | awk '{print $2}' | xargs mkdir -p
# Download snapshot files.
cat ../backup_files.out | xargs -n2 aws s3 cp
To restore the tables content, create a restore task manually on target ScyllaCluster by executing the following command:
```console
kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c <CLUSTER_ID> -L <BACKUP_LOCATION> -T <SNAPSHOT_TAG> --restore-tables
```

To load data into cluster pass cluster address to `sstableloader` together with path to data files and credentials:
```bash
sstableloader -d 'simple-cluster-us-east-1-us-east-1a-0.scylla.svc,simple-cluster-us-east-1-us-east-1a-1.scylla.svc,simple-cluster-us-east-1-us-east-1a-2.scylla.svc' ./users/data_0 --username scylla --password <password>
Where:
* `CLUSTER_ID` - a name or ID of a cluster you want to restore into.
* `BACKUP_LOCATION` - the location in which the backup is stored.
* `SNAPSHOT_TAG` - a tag of a snapshot that you want to restore.

When the task is created, the command will output the ID of a restore task.
```console
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c scylla/target -L s3:source-backup -T sm_20240105115931UTC --restore-tables
restore/63642069-bed5-4def-ba0f-68c49e47ace1
```

Depending on how big is your data set, this operation may take some time.
Once it finishes, data from the snapshot is restored and you may clean up the host.
Use the following command to check progress of the restore task:
```console
$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool progress -c scylla/target restore/63642069-bed5-4def-ba0f-68c49e47ace1
Restore progress
Run: ab015cef-abc8-11ee-9521-6e7993cf42ed
Status: DONE
Start time: 05 Jan 24 12:48:04 UTC
End time: 05 Jan 24 12:48:15 UTC
Duration: 11s
Progress: 100% | 100%
Snapshot Tag: sm_20240105115931UTC

+-------------+-------------+--------+---------+------------+--------+
| Keyspace | Progress | Size | Success | Downloaded | Failed |
+-------------+-------------+--------+---------+------------+--------+
| users | 100% | 100% | 409MiB | 409MiB | 409MiB | 0 |
+-------------+-------------+--------+---------+------------+--------+

Post-restore repair progress
Run: ab015cef-abc8-11ee-9521-6e7993cf42ed
Status: DONE
Start time: 05 Jan 24 12:48:04 UTC
End time: 05 Jan 24 12:48:15 UTC
Duration: 11s
Progress: 100%
Intensity: 1
Parallel: 0
Datacenters:
- us-east-1

+-------------+--------------+----------+----------+
| Keyspace | Table | Progress | Duration |
+-------------+--------------+----------+----------+
| users | users | 100% | 0s |
+-------------+--------------+----------+----------+

```

0 comments on commit 3dd1616

Please sign in to comment.