From b4d5705be44718fa97af53efbe5519432878c35e Mon Sep 17 00:00:00 2001 From: Seshachalam <104052572+seshachalam-yv@users.noreply.github.com> Date: Thu, 5 Dec 2024 16:34:51 +0530 Subject: [PATCH] Docs: Enhance immutable snapshots guide with detailed instructions (#811) --- docs/usage/enabling_immutable_snapshots.md | 373 +++++++++++++++++++++ docs/usage/immutable_snapshots.md | 82 ----- 2 files changed, 373 insertions(+), 82 deletions(-) create mode 100644 docs/usage/enabling_immutable_snapshots.md delete mode 100644 docs/usage/immutable_snapshots.md diff --git a/docs/usage/enabling_immutable_snapshots.md b/docs/usage/enabling_immutable_snapshots.md new file mode 100644 index 000000000..6cfc483ea --- /dev/null +++ b/docs/usage/enabling_immutable_snapshots.md @@ -0,0 +1,373 @@ +# Enabling Immutable Snapshots in `etcd-backup-restore` + +This guide walks you through the process of enabling immutable snapshots in `etcd-backup-restore` by leveraging bucket-level immutability features provided by cloud storage providers like Google Cloud Storage (GCS) and Azure Blob Storage (ABS). Enabling immutability ensures that your backups are tamper-proof and comply with regulatory requirements. + +--- + +## Terminology + +- **Bucket / Container**: A storage resource in cloud storage services where objects (such as snapshots) are stored. GCS uses the term **bucket**, while ABS uses **container**. + +- **Immutability Policy**: A configuration that specifies a minimum period during which objects in a bucket/container are protected from deletion or modification. + +- **Immutability Period**: The duration defined by the immutability policy during which objects remain immutable. + +- **Immutability**: The property of an object being unmodifiable after creation, until the immutability period expires. + +- **Locking**: The action of making an immutability policy permanent, preventing any reduction or removal of the immutability period. + +- **ETag**: An identifier representing a specific version of a policy or object, used for concurrency control. + +--- + +## Overview + +Currently, `etcd-backup-restore` supports bucket-level immutability for GCS and ABS. + +- **Immutability Policy**: You can add an immutability policy to a bucket/container to specify an immutability period. + - When an immutability policy is set, objects in the bucket/container can only be deleted or replaced once their age exceeds the immutability period. + - The policy retroactively applies to existing objects as well as new objects added to the bucket/container. + +> [!CAUTION] +> Locking an immutability policy is an irreversible action. + +- **Locking an Immutability Policy**: You can lock a bucket's/container's immutability policy to permanently enforce it. + - Once locked, you cannot remove the immutability policy or reduce its immutability period. + - You cannot delete a bucket/container with a locked immutability policy unless every object's age crosses the immutability period, and thus expiring. + - You can increase the immutability period of a locked policy if needed. + - A locked bucket/container can only be deleted once all objects present in the bucket/container are deleted. + + +--- + +## Configure Bucket-Level Immutability + +By configuring an immutability policy on your storage bucket/container, you ensure that all snapshots are stored in an immutable (Write Once, Read Many) state for a specified duration. This prevents snapshots from being modified or deleted until they reach the end of the immutability period. + +### Configure an Immutability Policy + +You can set a time-based immutability policy on your bucket/container. The immutability policy specifies the minimum duration for which the objects must remain immutable. This configuration can also be achieved using the cloud provider's respective console/portal. + +#### Google Cloud Storage (GCS) + +To configure an immutability policy on a GCS bucket: + +1. **Set the Immutability Policy** + + Use the `gcloud` command-line tool to set the immutability period: + + ```bash + gcloud storage buckets update gs://[BUCKET_NAME] \ + --retention-period [IMMUTABILITY_PERIOD] + ``` + + - Replace `[IMMUTABILITY_PERIOD]` with the desired immutability period (e.g., `4d` for four days, `1y` for one year). + - Replace `[BUCKET_NAME]` with the name of your bucket. + + **Example:** + + ```bash + gcloud storage buckets update gs://my-bucket \ + --retention-period 4d + ``` + +#### Azure Blob Storage (ABS) + +To configure an immutability policy on an Azure Blob Storage container: + +1. **Create the Immutability Policy** + + Use the Azure CLI to create the immutability policy: + + ```bash + az storage container immutability-policy create \ + --resource-group [RESOURCE_GROUP] \ + --account-name [STORAGE_ACCOUNT] \ + --container-name [CONTAINER_NAME] \ + --period [IMMUTABILITY_PERIOD] + ``` + + - Replace `[RESOURCE_GROUP]`, `[STORAGE_ACCOUNT]`, and `[CONTAINER_NAME]` with your specific values. + - Replace `[IMMUTABILITY_PERIOD]` with the desired immutability period in days. + + **Example:** + + ```bash + az storage container immutability-policy create \ + --resource-group myResourceGroup \ + --account-name myStorageAccount \ + --container-name my-container \ + --period 4 + ``` + +### Modify an Unlocked Immutability Policy + +You can modify an unlocked immutability policy to adjust the immutability period or to allow additional writes to the bucket/container. + +#### Google Cloud Storage (GCS) + +To modify the immutability policy: + +1. **Set a New Immutability Period** + + ```bash + gcloud storage buckets update gs://[BUCKET_NAME] \ + --retention-period [NEW_IMMUTABILITY_PERIOD] + ``` + + **Example:** + + ```bash + gcloud storage buckets update gs://my-bucket \ + --retention-period 7d + ``` + +2. **Remove the Immutability Policy** + + ```bash + gcloud storage buckets update gs://[BUCKET_NAME] \ + --clear-retention-period + ``` + +#### Azure Blob Storage (ABS) + +To modify an unlocked immutability policy: + +1. **Retrieve the Policy's ETag** + + ```bash + etag=$(az storage container immutability-policy show \ + --account-name [STORAGE_ACCOUNT] \ + --container-name [CONTAINER_NAME] \ + --query etag --output tsv) + ``` + +2. **Extend the Immutability Period** + + ```bash + az storage container immutability-policy extend \ + --resource-group [RESOURCE_GROUP] \ + --account-name [STORAGE_ACCOUNT] \ + --container-name [CONTAINER_NAME] \ + --period [NEW_IMMUTABILITY_PERIOD] \ + --if-match $etag + ``` + + **Example:** + + ```bash + az storage container immutability-policy extend \ + --resource-group myResourceGroup \ + --account-name myStorageAccount \ + --container-name my-container \ + --period 7 \ + --if-match $etag + ``` + +3. **Delete an Unlocked Policy** + + ```bash + az storage container immutability-policy delete \ + --resource-group [RESOURCE_GROUP] \ + --account-name [STORAGE_ACCOUNT] \ + --container-name [CONTAINER_NAME] \ + --if-match $etag + ``` + +### Lock the Immutability Policy + +Locking the immutability policy makes it irreversible and ensures that the policy cannot be reduced or removed. This provides compliance with regulatory requirements. + +#### Google Cloud Storage (GCS) + +To lock the immutability policy: + +> [!CAUTION] +> Locking an immutability policy is an irreversible action. + +1. **Lock the Policy** + + ```bash + gcloud storage buckets update gs://[BUCKET_NAME] \ + --lock-retention-period + ``` + + **Example:** + + ```bash + gcloud storage buckets update gs://my-bucket \ + --lock-retention-period + ``` + + +#### Azure Blob Storage (ABS) + +To lock the immutability policy: + +> [!WARNING] +> Once you have thoroughly tested a time-based immutability policy, you can proceed to lock the policy. A locked policy ensures compliance with regulations such as SEC 17a-4(f) and other regulatory requirements. While you can extend the immutability interval for a locked policy up to **_`five times`_**, it cannot be shortened. +> After a policy is locked, it cannot be deleted. However, you can delete the blob once the immutability interval has expired. + +1. **Retrieve the Policy's ETag** + + ```bash + etag=$(az storage container immutability-policy show \ + --account-name [STORAGE_ACCOUNT] \ + --container-name [CONTAINER_NAME] \ + --query etag --output tsv) + ``` + +> [!CAUTION] +> Once locked, the policy cannot be removed or decreased. + +2. **Lock the Policy** + + ```bash + az storage container immutability-policy lock \ + --resource-group [RESOURCE_GROUP] \ + --account-name [STORAGE_ACCOUNT] \ + --container-name [CONTAINER_NAME] \ + --if-match $etag + ``` + + **Example:** + + ```bash + az storage container immutability-policy lock \ + --resource-group myResourceGroup \ + --account-name myStorageAccount \ + --container-name my-container \ + --if-match $etag + ``` + +--- + +## Ignoring Snapshots During Restoration + +In certain scenarios, you might want `etcd-backup-restore` to ignore specific snapshots present in the object store during the restoration of etcd's data directory. When snapshots were mutable, operators could simply delete these snapshots, and subsequent restorations would not include them. However, once immutability is enabled, it is no longer possible to delete these snapshots. + +Many cloud providers allow you to add custom annotations or tags to objects to store additional metadata. These annotations or tags are separate from the object's main data and do not affect the object itself. This feature is available for immutable objects as well. + +We leverage this feature to signal to `etcd-backup-restore` to ignore certain snapshots during restoration. The annotation or tag to be added to a snapshot is: + +- **Key:** `x-etcd-snapshot-exclude` +- **Value:** `true` + +### Adding the Exclusion Tag + +#### Google Cloud Storage (GCS) + +To add the custom metadata: + +1. **Using the `gcloud` CLI** + + ```bash + gcloud storage objects update gs://[BUCKET_NAME]/[SNAPSHOT_PATH] \ + --custom-metadata x-etcd-snapshot-exclude=true + ``` + + **Example:** + + ```bash + gcloud storage objects update gs://my-bucket/snapshots/snapshot.db \ + --custom-metadata x-etcd-snapshot-exclude=true + ``` + +2. **Using the Google Cloud Console** + + - Navigate to your bucket in the Google Cloud Console. + - Locate the snapshot object. + - Click on the object to view its details. + - In the **Custom metadata** section, add: + - **Key:** `x-etcd-snapshot-exclude` + - **Value:** `true` + - Save the changes. + +#### Azure Blob Storage (ABS) + +To add the tag: + +1. **Using the `az` CLI** + + ```bash + az storage blob tag set \ + --account-name [STORAGE_ACCOUNT] \ + --container-name [CONTAINER_NAME] \ + --name [SNAPSHOT_NAME] \ + --tags x-etcd-snapshot-exclude=true + ``` + + **Example:** + + ```bash + az storage blob tag set \ + --account-name myStorageAccount \ + --container-name my-container \ + --name snapshots/snapshot.db \ + --tags x-etcd-snapshot-exclude=true + ``` + +2. **Using the Azure Portal** + + - Navigate to your storage account and container. + - Locate the snapshot blob. + - Click on the blob to view its details. + - In the **Blob index tags** section, add: + - **Name:** `x-etcd-snapshot-exclude` + - **Value:** `true` + - Save the changes. + +After adding the annotation or tag, `etcd-backup-restore` will ignore these snapshots during the restoration process. + +--- + +## Setting the Immutability Period + +When configuring the immutability period, consider setting it to align with your delta snapshot retention period, typically **four days**. + +- **Why Four Days?** This duration provides a buffer for identifying and resolving issues, especially over weekends. + +- **Example Scenario:** If an issue occurs on a Friday, the four-day immutability period allows administrators until Monday or Tuesday to debug and recover data without the risk of backups being altered or deleted. + +--- + +## Best Practices + +- **Thorough Testing Before Locking:** + + Before locking the immutability policy, perform comprehensive testing of your backup and restoration procedures. Ensure that: + + - Snapshots are being created and stored correctly. + - Restoration from snapshots works as expected. + - You are familiar with the process of adding exclusion tags if needed. + + Locking the immutability policy is irreversible, so it's crucial to confirm that your setup is fully functional. + +- **Use Exclusion Tags Wisely:** Apply exclusion tags to snapshots only when absolutely necessary. Overuse of exclusion tags can lead to unintentionally skipping important delta snapshots during the restoration process, potentially compromising data integrity. + +--- + +## Conclusion + +Enabling immutable snapshots in `etcd-backup-restore` significantly enhances the security and reliability of your backups by preventing unintended modifications or deletions. By leveraging bucket/container-level immutability features provided by GCS and ABS, you can meet compliance requirements and ensure data integrity. + +It's essential to carefully plan your immutability periods and exercise caution when locking immutability policies, as these actions have long-term implications. Use the snapshot exclusion feature judiciously to maintain control over your restoration processes. + +By following best practices and regularly reviewing your backup and immutability strategies, you can ensure that your data remains secure, compliant, and recoverable when needed. + +--- + +**References:** + +- **Google Cloud Storage (GCS)** + - [Bucket Lock Documentation](https://cloud.google.com/storage/docs/bucket-lock) + - [Use and lock retention policies](https://cloud.google.com/storage/docs/using-bucket-lock) + - [Custom Metadata](https://cloud.google.com/storage/docs/metadata#custom-metadata) + +- **Azure Blob Storage (ABS)** + - [Immutable Storage for Azure Blob Storage](https://learn.microsoft.com/azure/storage/blobs/immutable-storage-overview) + - [Configure Immutability Policies](https://learn.microsoft.com/azure/storage/blobs/immutable-policy-configure-container-scope) + - [Blob Index Tags](https://learn.microsoft.com/azure/storage/blobs/storage-index-tags-overview) + +--- + diff --git a/docs/usage/immutable_snapshots.md b/docs/usage/immutable_snapshots.md deleted file mode 100644 index 6ba9c11d1..000000000 --- a/docs/usage/immutable_snapshots.md +++ /dev/null @@ -1,82 +0,0 @@ -# Immutable Snapshots - -## Overview of Immutable Objects - -Several cloud providers offer functionality to create immutable objects within their storage services. Once an object is uploaded, it cannot be modified or deleted for a set period, known as the **immutability period**. These are referred to as **immutable objects**. - -Currently, etcd-backup-restore supports the use of immutable objects on the following cloud platforms: - -- Google Cloud Storage -- Azure Blob Storage - -## Enabling and using Immutable Snapshots with etcd-backup-restore - -Etcd-backup-restore supports immutable objects, typically at what cloud providers call the "bucket level." During the creation of a bucket, it is configured to render objects immutable for a specific duration from the moment of their upload. This feature can be enabled through: - -- **Google Cloud Storage**: [Bucket Lock](https://cloud.google.com/storage/docs/bucket-lock) -- **Azure Blob Storage**: [Container-level WORM Policies](https://learn.microsoft.com/en-us/azure/storage/blobs/immutable-container-level-worm-policies) - -It is also possible to enable immutability retroactively by making appropriate API calls to your cloud provider, allowing the immutable snapshots feature to be used with existing buckets. For information on such configurations, please refer to your cloud provider's documentation. - -Setting the immutability period at the bucket level through the CLI can be done through the following commands: - -- Google Cloud Storage would be done like so: - - ```sh - gcloud storage buckets update --retention-period= - ``` - -- Azure Blob Storage Container would be done like so: - - ```sh - az storage container immutability-policy create --resource-group --account-name --container-name --period - ``` - -The behaviour of bucket's objects uploaded before a bucket is set to immutable varies among storage providers. etcd-backup-restore manages these objects and will perform garbage collection according to the configured garbage collection policy and the object's immutability expiry. - -> Note: If immutable snapshots are not enabled then the object's immutability expiry will be considered as zero, hence causing no effect on current functionality. - -## Current Capabilities - -etcd-backup-restore does not require configurations related to the immutability period of bucket's objects as this information is derived from the bucket's existing immutability settings. The etcd-backup-restore process also verifies the immutability expiry time of an object prior to initiating its garbage collection. - -Therefore, it is advisable to configure your garbage collection policies based on the duration you want your objects to remain immutable. - -## Storage Considerations - -Making objects immutable for extended periods can increase storage costs since these objects cannot be removed once uploaded. Storing outdated snapshots beyond their utility does not significantly enhance recovery capabilities. Therefore, consider all factors before enabling immutability for buckets, as this feature is irreversible once set by cloud providers. - -## Ignoring Snapshots From Restoration - -There might be certain cases where operators would like `etcd-backup-restore` to ignore particular snapshots present in the object store during restoration of etcd's data-dir. -When snapshots were mutable, operators could simply delete these snapshots, and the restoration that follows this would not include them. -Once immutability is turned on, however, it would not be possible to do this. - -Various cloud providers provide functionality to add custom annotations/tags to objects to add additional information to objects. These additional annotations/tags are orthogonal to the object's metadata, and therefore do not affect the object itself. This feature is thus available for objects which are immutable as well. - -We leverage this feature to signal to etcd-backup-restore to not consider certain snapshots during restoration. -The annotation/tag that is to be added to a snapshot for this is `x-etcd-snapshot-exclude=true`. - -You can add these tags through for the following providers like so: - -- **Google Cloud Storage**: as specified in the [docs](https://cloud.google.com/sdk/gcloud/reference/storage/objects/update?hl=en). (GCS calls this Custom Metadata). - - ```sh - gcloud storage objects update gs://bucket/your-snapshot --custom-metadata=x-etcd-snapshot-exclude=true - ``` - - or: - - Use the Google Cloud Console to add custom metadata to the object in the `Custom metadata` section of the object. - -- **Azure Blob Storage**: as specified in the [docs](https://learn.microsoft.com/en-us/cli/azure/storage/blob/tag?view=azure-cli-latest#az-storage-blob-tag-set). (ABS calls this tags). - - ```sh - az storage blob tag set --container-name your-container --name your-snapshot --tags "x-etcd-snapshot-exclude"="true" - ``` - - or - - Use the Azure Portal to add the tag in the `Blob index tags` section of the blob. - -Once these annotations/tags are added, etcd-backup-restore will ignore those snapshots during restoration.