diff --git a/docs/source/nodeoperations/restore.md b/docs/source/nodeoperations/restore.md index b4d85573cff..f8ac607d9d6 100644 --- a/docs/source/nodeoperations/restore.md +++ b/docs/source/nodeoperations/restore.md @@ -2,88 +2,216 @@ This procedure will describe how to restore from backup taken using [Scylla Manager](../manager.md) to a fresh **empty** cluster of any size. -First identify to which snapshot you want to restore. To get list of available snapshot execute following command on Scylla Manager Pod. -```bash -sctool backup list -c --all-clusters -L -``` +:::{caution} +Due to a [bug](https://github.com/scylladb/scylla-manager/issues/3679) in Scylla Manager not supporting ScyllaClusters having both non-TLS and TLS CQL ports open, you have to disable the TLS certificate management +in Scylla Operator. +This step will no longer be required, when the bug is fixed. -Where: -* `CLUSTER_ID` - is a name of a cluster or ID under which ScyllaCluster was registered. You can find it in ScyllaCluster Status. -* `BACKUP_LOCATION` - is a location where backup is stored. For example, for bucket called `backups` stored in AWS S3, location is `s3:backups`. +To disable TLS certificate management in Scylla Operator add `--feature-gates="AutomaticTLSCertificates=false"` flag to Scylla Operator deployment. -```bash -sctool backup list -c simple-cluster --all-clusters -L s3:backups -Snapshots: - - sm_20201227144037UTC (409MiB) - - sm_20201228145917UTC (434MiB) -Keyspaces: - - users (9 tables) - - system_auth (2 tables) - - system_distributed (3 tables) - - system_schema (13 tables) - - system_traces (5 tables) +```console +kubectl -n scylla-operator patch deployment/scylla-operator --type=json -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--feature-gates=AutomaticTLSCertificates=false"}]' ``` -To get the list of files use: +::: + +In the following example, the ScyllaCluster, which was used to take the backup, is called `source`. Backup will be restored into the ScyllaCluster named `target`. + +::::{tab-set} +:::{tab-item} Source ScyllaCluster +```yaml +apiVersion: scylla.scylladb.com/v1 +kind: ScyllaCluster +metadata: + name: source + namespace: scylla +spec: + agentVersion: 3.2.5 + version: 5.4.1 + developerMode: true + backups: + - name: foo + location: + - s3:source-backup + keyspace: + - '*' + datacenter: + name: us-east-1 + racks: + - name: us-east-1a + members: 1 + storage: + capacity: 1Gi + resources: + limits: + cpu: 1 + memory: 1Gi +``` +::: +:::{tab-item} Target ScyllaCluster +```yaml +apiVersion: scylla.scylladb.com/v1 +kind: ScyllaCluster +metadata: + name: target + namespace: scylla +spec: + agentVersion: 3.2.5 + version: 5.4.1 + developerMode: true + datacenter: + name: us-east-1 + racks: + - name: us-east-1a + members: 1 + storage: + capacity: 1Gi + resources: + limits: + cpu: 1 + memory: 1Gi +``` +::: +:::: + +Make sure your target cluster is already registered in Scylla Manager. To get a list of all registered clusters, execute the following command: +```console +$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool cluster list ++--------------------------------------+---------------------------------------+---------+-----------------+ +| ID | Name | Port | CQL credentials | ++--------------------------------------+---------------------------------------+---------+-----------------+ +| af1dd5cd-0406-4974-949f-dc9842980080 | scylla/target | default | set | +| ebd82268-efb7-407e-a540-3619ae053778 | scylla/source | default | set | ++--------------------------------------+---------------------------------------+---------+-----------------+ +``` -```bash -sctool backup files -c -L -T +Identify the tag of a snapshot which you want to restore. To get a list of all available snapshots, execute following command: +```console +kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool backup list -c --all-clusters -L ``` Where: -* `SNAPSHOT_TAG` - name of snapshot you want to restore. +* `CLUSTER_ID` - the name or ID of a registered cluster with access to `BACKUP_LOCATION`. +* `BACKUP_LOCATION` - the location in which the backup is stored. + +In this example, `BACKUP_LOCATION` is `s3:source-backup`. Use the name of cluster which has access to the backup location for `CLUSTER_ID`. +In this example, it's `scylla/target`. + +```console +$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool backup list -c scylla/target --all-clusters -L s3:source-backup +backup/ff36d7e0-af2e-458c-afe6-868e0f3396b2 +Snapshots: + - sm_20240105115931UTC (409MiB, 1 nodes) +Keyspaces: + - system_schema (15 tables) + - users (9 tables) -Before we start restoring the data, we have to restore the schema. -The first output line is a path to schemas archive, for example: -```bash -s3://backups/backup/schema/cluster/ed63b474-2c05-4f4f-b084-94541dd86e7a/task_287791d9-c257-4850-aef5-7537d6e69d90_tag_sm_20201228145917UTC_schema.tar.gz ./ ``` -To download this archive you can use AWS CLI tool `aws s3 cp`. +In the below commands, we are restoring the `sm_20240105115931UTC` snapshot. Replace it with a tag of a snapshot that you want to restore. +Restoring consist of two steps. First, you'll restore the schema, and then the data. +To restore schema, create a restore task manually on target ScyllaCluster by executing following command: +```console +kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c -L -T --restore-schema +``` -This archive contains a single CQL file for each keyspace in the backup. -```bash -tar -ztvf task_287791d9-c257-4850-aef5-7537d6e69d90_tag_sm_20201228145917UTC_schema.tar.gz --rw------- 0/0 12671 2020-12-28 13:17 users.cql --rw------- 0/0 2216 2020-12-28 13:17 system_auth.cql --rw------- 0/0 921 2020-12-28 13:17 system_distributed.cql --rw------- 0/0 12567 2020-12-28 13:17 system_schema.cql --rw------- 0/0 4113 2020-12-28 13:17 system_traces.cql +Where: +* `CLUSTER_ID` - a name or ID of a cluster you want to restore into. +* `BACKUP_LOCATION` - the location in which the backup is stored. +* `SNAPSHOT_TAG` - a tag of a snapshot that you want to restore. + +When the task is created, the command will output the ID of a restore task. +```console +$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c scylla/target -L s3:source-backup -T sm_20240105115931UTC --restore-schema +restore/57228c52-7cf6-4271-8c8d-d446ff160747 ``` -Extract this archive and copy each schema file to one of the cluster Pods by: -```bash -kubectl -n scylla cp users.cql simple-cluster-us-east-1-us-east-1a-0:/tmp/users.cql -c scylla +Use the following command to check progress of the restore task: +```console +$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool progress -c scylla/target restore/57228c52-7cf6-4271-8c8d-d446ff160747 +Restore progress +Run: 0dd20cdf-abc4-11ee-951c-6e7993cf42ed +Status: DONE - restart required (see restore docs) +Start time: 05 Jan 24 12:15:02 UTC +End time: 05 Jan 24 12:15:09 UTC +Duration: 6s +Progress: 100% | 100% +Snapshot Tag: sm_20240105115931UTC + ++---------------+-------------+----------+----------+------------+--------+ +| Keyspace | Progress | Size | Success | Downloaded | Failed | ++---------------+-------------+----------+----------+------------+--------+ +| system_schema | 100% | 100% | 214.150k | 214.150k | 214.150k | 0 | ++---------------+-------------+----------+----------+------------+--------+ ``` -To import schema simply execute: -```bash -kubectl -n scylla exec simple-cluster-us-east-1-us-east-1a-0 -c scylla -- cqlsh -f /tmp/users.cql +As suggested in the progress output, you will need to execute a rolling restart of the ScyllaCluster. +```console +kubectl patch scyllacluster target --type merge -p '{"spec": {"forceRedeploymentReason": "schema restored"}}' ``` -Once the schema is recreated we can proceed to downloading data files. +Use the following commands to wait until restart is finished: +```console +$ kubectl wait --for='condition=Progressing=False' -n scylla scyllaclusters.scylla.scylladb.com/target +scyllacluster.scylla.scylladb.com/target condition met -First let's save a list of snapshot files to file called `backup_files.out`: +$ kubectl wait --for='condition=Degraded=False' -n scylla scyllaclusters.scylla.scylladb.com/target +scyllacluster.scylla.scylladb.com/target condition met -```bash -kubectl -n scylla-manager exec deployment.apps/scylla-manager-controller -- sctool backup files -c simple-cluster -L s3:backups -T sm_20201228145917UTC > backup_files.out +$ kubectl wait --for='condition=Available=True' -n scylla scyllaclusters.scylla.scylladb.com/target +scyllacluster.scylla.scylladb.com/target condition met ``` -We will be using `sstableloader` to restore data. `sstableloader` needs a specific directory structure to work namely: `//` -To create this directory structure and download all the files execute these commands: -```bash -mkdir snapshot -cd snapshot -# Create temporary directory structure. -cat ../backup_files.out | awk '{print $2}' | xargs mkdir -p -# Download snapshot files. -cat ../backup_files.out | xargs -n2 aws s3 cp +To restore the tables content, create a restore task manually on target ScyllaCluster by executing the following command: +```console +kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c -L -T --restore-tables ``` -To load data into cluster pass cluster address to `sstableloader` together with path to data files and credentials: -```bash -sstableloader -d 'simple-cluster-us-east-1-us-east-1a-0.scylla.svc,simple-cluster-us-east-1-us-east-1a-1.scylla.svc,simple-cluster-us-east-1-us-east-1a-2.scylla.svc' ./users/data_0 --username scylla --password +Where: +* `CLUSTER_ID` - a name or ID of a cluster you want to restore into. +* `BACKUP_LOCATION` - the location in which the backup is stored. +* `SNAPSHOT_TAG` - a tag of a snapshot that you want to restore. + +When the task is created, the command will output the ID of a restore task. +```console +$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool restore -c scylla/target -L s3:source-backup -T sm_20240105115931UTC --restore-tables +restore/63642069-bed5-4def-ba0f-68c49e47ace1 ``` -Depending on how big is your data set, this operation may take some time. -Once it finishes, data from the snapshot is restored and you may clean up the host. +Use the following command to check progress of the restore task: +```console +$ kubectl -n scylla-manager exec -ti deployment.apps/scylla-manager -- sctool progress -c scylla/target restore/63642069-bed5-4def-ba0f-68c49e47ace1 +Restore progress +Run: ab015cef-abc8-11ee-9521-6e7993cf42ed +Status: DONE +Start time: 05 Jan 24 12:48:04 UTC +End time: 05 Jan 24 12:48:15 UTC +Duration: 11s +Progress: 100% | 100% +Snapshot Tag: sm_20240105115931UTC + ++-------------+-------------+--------+---------+------------+--------+ +| Keyspace | Progress | Size | Success | Downloaded | Failed | ++-------------+-------------+--------+---------+------------+--------+ +| users | 100% | 100% | 409MiB | 409MiB | 409MiB | 0 | ++-------------+-------------+--------+---------+------------+--------+ + +Post-restore repair progress +Run: ab015cef-abc8-11ee-9521-6e7993cf42ed +Status: DONE +Start time: 05 Jan 24 12:48:04 UTC +End time: 05 Jan 24 12:48:15 UTC +Duration: 11s +Progress: 100% +Intensity: 1 +Parallel: 0 +Datacenters: + - us-east-1 + ++-------------+--------------+----------+----------+ +| Keyspace | Table | Progress | Duration | ++-------------+--------------+----------+----------+ +| users | users | 100% | 0s | ++-------------+--------------+----------+----------+ + +```