Skip to content

Latest commit

 

History

History
492 lines (326 loc) · 29.7 KB

encryption-at-rest.md

File metadata and controls

492 lines (326 loc) · 29.7 KB
Raw
title summary aliases
Encryption at Rest
Learn how to enable encryption at rest to protect sensitive data.
/docs/dev/encryption at rest/

Encryption at Rest

Note:

If your cluster is deployed on AWS and uses the EBS storage, it is recommended to use the EBS encryption. See AWS documentation - EBS Encryption. You are using the non-EBS storage on AWS such as the local NVMe storage, it is recommended to use encryption at rest introduced in this document.

Encryption at rest means that data is encrypted when it is stored. For databases, this feature is also referred to as TDE (transparent data encryption). This is opposed to encryption in flight (TLS) or encryption in use (rarely used). Different things could be doing encryption at rest (such as SSD drive, file system, and cloud vendor), but by having TiKV do the encryption before storage this helps ensure that attackers must authenticate with the database to gain access to data. For example, when an attacker gains access to the physical machine, data cannot be accessed by copying files on disk.

Encryption support in different TiDB components

In a TiDB cluster, different components use different encryption methods. This section introduces the encryption supports in different TiDB components such as TiKV, TiFlash, PD, and Backup & Restore (BR).

When a TiDB cluster is deployed, the majority of user data is stored on TiKV and TiFlash nodes. Some metadata is stored on PD nodes (for example, secondary index keys used as TiKV Region boundaries). To get the full benefits of encryption at rest, you need to enable encryption for all components. Backups, log files, and data transmitted over the network should also be considered when you implement encryption.

TiKV

TiKV supports encryption at rest. This feature allows TiKV to transparently encrypt data files using AES or SM4 in CTR mode. To enable encryption at rest, an encryption key must be provided by the user and this key is called master key. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest.

You can use Key Management Service (KMS) for both cloud and self-hosted deployments or supply the plaintext master key in a file.

TiKV currently does not exclude encryption keys and user data from core dumps. It is advised to disable core dumps for the TiKV process when using encryption at rest. This is not currently handled by TiKV itself.

TiKV tracks encrypted data files using the absolute path of the files. As a result, once encryption is turned on for a TiKV node, the user should not change data file paths configuration such as storage.data-dir, raftstore.raftdb-path, rocksdb.wal-dir and raftdb.wal-dir.

SM4 encryption is only supported in v6.3.0 and later versions of TiKV. TiKV versions earlier than v6.3.0 only support AES encryption. SM4 encryption might lead to 50% to 80% degradation on throughput.

TiFlash

TiFlash supports encryption at rest. Data keys are generated by TiFlash. All files (including data files, schema files, and temporary files) written into TiFlash (including TiFlash Proxy) are encrypted using the current data key. The encryption algorithms, the encryption configuration (in the tiflash-learner.toml file supported by TiFlash, and the meanings of monitoring metrics are consistent with those of TiKV.

If you have deployed TiFlash with Grafana, you can check the TiFlash-Proxy-Details -> Encryption panel.

SM4 encryption is only supported in v6.4.0 and later versions of TiFlash. TiFlash versions earlier than v6.4.0 only support AES encryption.

PD

Encryption-at-rest for PD is an experimental feature, which is configured in the same way as in TiKV.

Backups with BR

BR supports S3 server-side encryption (SSE) when backing up data to S3. A customer-owned AWS KMS key can also be used together with S3 server-side encryption. See BR S3 server-side encryption for details.

Logging

TiKV, TiDB, and PD info logs might contain user data for debugging purposes. The info log and this data in it are not encrypted. It is recommended to enable log redaction.

TiKV encryption at rest

Overview

TiKV currently supports encrypting data using AES128, AES192, AES256, or SM4 (only in v6.3.0 and later versions), in CTR mode. TiKV uses envelope encryption. As a result, two types of keys are used in TiKV when encryption is enabled.

  • Master key. The master key is provided by user and is used to encrypt the data keys TiKV generates. Management of master key is external to TiKV.
  • Data key. The data key is generated by TiKV and is the key actually used to encrypt data.

The same master key can be shared by multiple instances of TiKV. The recommended way to provide a master key in production is via KMS. Currently, TiKV supports KMS encryption on AWS, Google Cloud, and Azure. To enable KMS encryption, you need to create a customer master key (CMK) through KMS, and then provide the CMK key ID to TiKV using the configuration file. If TiKV fails to get access to the KMS CMK, it will fail to start or restart.

Alternatively, if using custom key is desired, supplying the master key via file is also supported. The file must contain a 256 bits (or 32 bytes) key encoded as hex string, end with a newline (namely, \n), and contain nothing else. Persisting the key on disk, however, leaks the key, so the key file is only suitable to be stored on the tempfs in RAM.

Data keys are passed to the underlying storage engine (namely, RocksDB). All files written by RocksDB, including SST files, WAL files, and the MANIFEST file, are encrypted by the current data key. Other temporary files used by TiKV that may include user data are also encrypted using the same data key. Data keys are automatically rotated by TiKV every week by default, but the period is configurable. On key rotation, TiKV does not rewrite all existing files to replace the key, but RocksDB compaction are expected to rewrite old data into new data files, with the most recent data key, if the cluster gets constant write workload. TiKV keeps track of the key and encryption method used to encrypt each of the files and use the information to decrypt the content on reads.

Regardless of data encryption method, data keys are encrypted using AES256 in GCM mode for additional authentication. This required the master key to be 256 bits (32 bytes), when passing from file instead of KMS.

Configure encryption

To enable encryption, you can add the encryption section in the configuration files of TiKV and PD:

[security.encryption]
data-encryption-method = "aes128-ctr"
data-key-rotation-period = "168h" # 7 days

  • data-encryption-method specifies the encryption algorithm. The possible values are "aes128-ctr", "aes192-ctr", "aes256-ctr", "sm4-ctr" (only for v6.3.0 and later versions), and "plaintext". The default value is "plaintext", which means that encryption is disabled by default.

    • For a new TiKV cluster or an existing TiKV cluster, only data written after encryption has been enabled is guaranteed to be encrypted.
    • To disable encryption after it is enabled, remove data-encryption-method from the configuration file or set its value to "plaintext", and then restart TiKV.
    • To change the encryption algorithm, replace the value of data-encryption-method with a supported encryption algorithm, and then restart TiKV. After the replacement, as new data is written in, the encryption files generated by the previous encryption algorithm are gradually rewritten to files generated by the new encryption algorithm.

  • data-key-rotation-period specifies how often TiKV rotates keys.

If encryption is enabled (that is, the value of data-encryption-method is not "plaintext"), you must specify a master key in either of the following ways:

Specify a master key via KMS

TiKV supports KMS encryption for three platforms: AWS, Google Cloud, and Azure. Depending on the platform where your service is deployed, you can choose one of them to configure KMS encryption.

Warning:

Currently, specifying a master key using Google Cloud KMS is experimental. It is not recommended that you use it in production environments. This feature might be changed or removed without prior notice. If you find a bug, you can report an issue on GitHub.

Step 1. Create a master key

To create a key on AWS, take the following steps:

  1. Go to the AWS KMS on the AWS console.
  2. Make sure that you have selected the correct region on the top right corner of your console.
  3. Click Create key and select Symmetric as the key type.
  4. Set an alias for the key.

You can also perform the operations using the AWS CLI:

aws --region us-west-2 kms create-key
aws --region us-west-2 kms create-alias --alias-name "alias/tidb-tde" --target-key-id 0987dcba-09fe-87dc-65ba-ab0987654321

The --target-key-id to enter in the second command is in the output of the first command.

Step 2. Configure the master key

To specify the master key using AWS KMS, add the [security.encryption.master-key] configuration after the [security.encryption] section in the TiKV configuration file:

[security.encryption.master-key]
type = "kms"
key-id = "0987dcba-09fe-87dc-65ba-ab0987654321"
region = "us-west-2"
endpoint = "https://kms.us-west-2.amazonaws.com"

The key-id specifies the key ID for the KMS CMK. The region is the AWS region name for the KMS CMK. The endpoint is optional and you do not need to specify it normally unless you are using an AWS KMS-compatible service from a non-AWS vendor or need to use a VPC endpoint for KMS.

You can also use multi-Region keys in AWS. For this, you need to set up a primary key in a specific region and add replica keys in the regions you require.

Step 1. Create a master key

To create a key on Google Cloud, take the following steps:

  1. Go to the Key Management page in the Google Cloud console.

  2. Click Create key ring. Enter a name for the key ring, select a location of the key ring, and then click Create. Note that the location of the key ring needs to cover the region where the TiDB cluster is deployed.

  3. Select the key ring you created in the previous step, and then click Create Key on the key ring details page.

  4. Enter a name for the key, set the key information as follows, and then click Create.

    • Protection level: Software or HSM
    • Key Material: Generated key
    • Purpose: Symmetric encrypt/decrypt

You can also perform this operation using the gcloud CLI:

gcloud kms keyrings create "key-ring-name" --location "global"
gcloud kms keys create "key-name" --keyring "key-ring-name" --location "global" --purpose "encryption" --rotation-period "30d"

Make sure to replace the values of "key-ring-name", "key-name", "global", and "30d" in the preceding command with the names and configurations corresponding to your actual key.

Step 2. Configure the master key

To specify the master key using Google Cloud KMS, add the [security.encryption.master-key] configuration after the [security.encryption] section:

[security.encryption.master-key]
type = "kms"
key-id = "projects/project-name/locations/global/keyRings/key-ring-name/cryptoKeys/key-name"
vendor = "gcp"

[security.encryption.master-key.gcp]
credential-file-path = "/path/to/credential.json"
  • key-id specifies the key ID of the KMS CMK.
  • credential-file-path specifies the path of the authentication credentials file, which currently supports two types of credentials: Service Account and Authentication User. If the TiKV environment is already configured with application default credentials, there is no need to configure credential-file-path.

Step 1. Create a master key

To create a key on Azure, refer to the instructions in Set and retrieve a key from Azure Key Vault using the Azure portal.

Step 2. Configure the master key

To specify the master key using Azure KMS, add the [security.encryption.master-key] configuration after the [security.encryption] section in the TiKV configuration file:

[security.encryption.master-key]
type = 'kms'
key-id = 'your-kms-key-id'
region = 'region-name'
endpoint = 'endpoint'
vendor = 'azure'

[security.encryption.master-key.azure]
tenant-id = 'tenant_id'
client-id = 'client_id'
keyvault-url = 'keyvault_url'
hsm-name = 'hsm_name'
hsm-url = 'hsm_url'
# The following four fields are optional, used to set client authentication credentials. You can configure them according to the requirements of your scenario.
client_certificate = ""
client_certificate_path = ""
client_certificate_password = ""
client_secret = ""

Except vendor, you need to modify the values of other fields in the preceding configuration to the corresponding configuration of the actual key.

Specify a master key via a file

To specify a master key that's stored in a file, the master key configuration would look like the following:

[security.encryption.master-key]
type = "file"
path = "/path/to/key/file"

Here path is the path to the key file. The file must contain a 256 bits (or 32 bytes) key encoded as hex string, end with a newline (\n) and contain nothing else. Example of the file content:

3b5896b5be691006e0f71c3040a29495ddcad20b14aff61806940ebd780d3c62

Rotate the master key

To rotate master key, you have to specify both of the new master key and old master key in the configuration, and restart TiKV. Use security.encryption.master-key to specify the new master key, and use security.encryption.previous-master-key to specify the old master key. The configuration format for security.encryption.previous-master-key is the same as encryption.master-key. On restart TiKV must access both of the old and new master key, but once TiKV is up and running, TiKV will only need access to the new key. It is okay to leave the encryption.previous-master-key configuration in the configuration file from that on. Even on restart, TiKV only tries to use the old key if it fails to decrypt existing data using the new master key.

Currently online master key rotation is not supported, so you need to restart TiKV. It is advised to do a rolling restart to a running TiKV cluster serving online query.

Here is an example configuration for rotating the KMS CMK:

[security.encryption.master-key]
type = "kms"
key-id = "50a0c603-1c6f-11e6-bb9e-3fadde80ce75"
region = "us-west-2"

[security.encryption.previous-master-key]
type = "kms"
key-id = "0987dcba-09fe-87dc-65ba-ab0987654321"
region = "us-west-2"

Monitoring and debugging

To monitor encryption at rest, if you deploy TiKV with Grafana, you can look at the Encryption panel in the TiKV-Details dashboard. There are a few metrics to look for:

  • Encryption initialized: 1 if encryption is initialized during TiKV startup, 0 otherwise. In case of master key rotation, after encryption is initialized, TiKV do not need access to the previous master key.
  • Encryption data keys: number of existing data keys. The number is bumped by 1 after each time data key rotation happened. Use this metrics to monitor if data key rotation works as expected.
  • Encrypted files: number of encrypted data files currently exists. Compare this number to existing data files in the data directory to estimate portion of data being encrypted, when turning on encryption for a previously unencrypted cluster.
  • Encryption meta file size: size of the encryption meta data files.
  • Read/Write encryption meta duration: the extra overhead to operate on metadata for encryption.

For debugging, the tikv-ctl command can be used to dump encryption metadata such as encryption method and data key id used to encryption the file, as well as list of data keys. Since the operation can expose sensitive data, it is not recommended to use in production. Please refer to TiKV Control document.

Compatibility between TiKV versions

To reduce the overhead caused by I/O and mutex contention when TiKV manages the encryption metadata, an optimization is introduced in TiKV v4.0.9 and controlled by security.encryption.enable-file-dictionary-log in the TiKV configuration file. This configuration parameter takes effect only on TiKV v4.0.9 or later versions.

When it is enabled (by default), the data format of encryption metadata is unrecognizable by TiKV v4.0.8 or earlier versions. For example, assume that you use TiKV v4.0.9 or later with encryption at rest and the default enable-file-dictionary-log configuration. If you downgrade your cluster to TiKV v4.0.8 or earlier, TiKV will fail to start, with an error in the info log similar to the following one:

[2020/12/07 07:26:31.106 +08:00] [ERROR] [mod.rs:110] ["encryption: failed to load file dictionary."]
[2020/12/07 07:26:33.598 +08:00] [FATAL] [lib.rs:483] ["called `Result::unwrap()` on an `Err` value: Other(\"[components/encryption/src/encrypted_file/header.rs:18]: unknown version 2\")"]

To avoid the error above, you can first set security.encryption.enable-file-dictionary-log to false and start TiKV with v4.0.9 or later. Once TiKV starts successfully, the data format of encryption metadata is downgraded to the version recognizable to earlier TiKV versions. At this point, you can then downgrade your TiKV cluster to an earlier version.

TiFlash encryption at rest

Overview

The encryption algorithm currently supported by TiFlash is consistent with that supported by TiKV, including AES128, AES192, AES256, and SM4 (only in v6.4.0 and later versions), in CTR mode. TiFlash also uses envelope encryption. Therefore, two types of keys are used in TiFlash when encryption is enabled.

  • Master key. The master key is provided by user and is used to encrypt the data keys TiFlash generates. Management of master key is external to TiFlash.
  • Data key. The data key is generated by TiFlash and is the key actually used to encrypt data.

The same master key can be shared by multiple instances of TiFlash, and can also be shared among TiFlash and TiKV. The recommended way to provide a master key in production is via AWS KMS. Alternatively, if using custom key is desired, supplying the master key via file is also supported. The specific method to generate master key and the format of the master key are the same as TiKV.

TiFlash uses the current data key to encrypt all data placed on the disk, including data files, Schema files, and temporary data files generated during calculations. Data keys are automatically rotated by TiFlash every week by default, and the period is configurable. On key rotation, TiFlash does not rewrite all existing files to replace the key, but background compaction tasks are expected to rewrite old data into new data files, with the most recent data key, if the cluster gets constant write workload. TiFlash keeps track of the key and encryption method used to encrypt each of the files and use the information to decrypt the content on reads.

Key creation

To create a key on AWS, refer to the steps to create a key for TiKV.

Configure encryption

To enable encryption, you can add the encryption section in the tiflash-learner.toml configuration file:

[security.encryption]
data-encryption-method = "aes128-ctr"
data-key-rotation-period = "168h" # 7 days

Alternatively, add the following contents in the TiUP cluster template:

server_configs:
  tiflash-learner:
    security.encryption.data-encryption-method: "aes128-ctr"
    security.encryption.data-key-rotation-period: "168h" # 7 days

Possible values for data-encryption-method are "aes128-ctr", "aes192-ctr", "aes256-ctr", "sm4-ctr" (only in v6.4.0 and later versions) and "plaintext". The default value is "plaintext", which means encryption is not turned on. data-key-rotation-period defines how often TiFlash rotates the data key. Encryption can be turned on for a fresh TiFlash cluster, or an existing TiFlash cluster, though only data written after encryption is enabled is guaranteed to be encrypted. To disable encryption, remove data-encryption-method in the configuration file, or reset it to "plaintext", and restart TiFlash. To change encryption method, update data-encryption-method in the configuration file and restart TiFlash. To change the encryption algorithm, replace data-encryption-method with a supported encryption algorithm and then restart TiFlash. After the replacement, as new data is written in, the encryption file generated by the previous encryption algorithm is gradually rewritten to a file generated by the new encryption algorithm.

The master key has to be specified if encryption is enabled (that is,data-encryption-method is not "plaintext"). To specify an AWS KMS CMK as the master key, add the encryption.master-key section after the encryption section in the tiflash-learner.toml configuration file:

[security.encryption.master-key]
type = "kms"
key-id = "0987dcba-09fe-87dc-65ba-ab0987654321"
region = "us-west-2"
endpoint = "https://kms.us-west-2.amazonaws.com"

Alternatively, add the following contents in the TiUP cluster template:

server_configs:
  tiflash-learner:
    security.encryption.master-key.type: "kms"
    security.encryption.master-key.key-id: "0987dcba-09fe-87dc-65ba-ab0987654321"
    security.encryption.master-key.region: "us-west-2"
    security.encryption.master-key.endpoint: "https://kms.us-west-2.amazonaws.com"

The meanings of the preceding configuration items are the same as those of TiKV.

To specify a master key that is stored in a file, add the following configuration in the tiflash-learner.toml configuration file:

[security.encryption.master-key]
type = "file"
path = "/path/to/key/file"

Alternatively, add the following contents in the TiUP cluster template:

server_configs:
  tiflash-learner:
    security.encryption.master-key.type: "file"
    security.encryption.master-key.path: "/path/to/key/file"

The meanings of the preceding configuration items and the content format of the key file are the same as those of TiKV.

Rotate the master key

To rotate the master key of TiFlash, follow the steps to rotate the master key of TiKV. Currently, TiFlash does not support online master key rotation, either. Therefore, you need to restart TiFlash to make the rotation effective. It is recommended to do a rolling restart to a running TiFlash cluster serving online query.

To rotate the KMS CMK, add the following contents in the tiflash-learner.toml configuration file:

[security.encryption.master-key]
type = "kms"
key-id = "50a0c603-1c6f-11e6-bb9e-3fadde80ce75"
region = "us-west-2"

[security.encryption.previous-master-key]
type = "kms"
key-id = "0987dcba-09fe-87dc-65ba-ab0987654321"
region = "us-west-2"

Alternatively, add the following contents in the TiUP cluster template:

server_configs:
  tiflash-learner:
    security.encryption.master-key.type: "kms"
    security.encryption.master-key.key-id: "50a0c603-1c6f-11e6-bb9e-3fadde80ce75"
    security.encryption.master-key.region: "us-west-2"
    security.encryption.previous-master-key.type: "kms"
    security.encryption.previous-master-key.key-id: "0987dcba-09fe-87dc-65ba-ab0987654321"
    security.encryption.previous-master-key.region: "us-west-2"

Monitoring and debugging

To monitor encryption at rest, if you deploy TiFlash with Grafana, you can look at the Encryption panel in the TiFlash-Proxy-Details dashboard. The meaning of monitoring items is the same as that of TiKV.

For debugging, since TiFlash reuses TiKV's logic for managing encrypted metadata, the tikv-ctl command can be used to dump encryption metadata such as encryption method and data key ID used to encryption the file, as well as list of data keys. This operation can expose sensitive data and is therefore not recommended in production. Refer to TiKV Control for more details.

Compatibility between TiKV versions

TiFlash also optimizes encrypted metadata operations in v4.0.9, and its compatibility requirements are the same as those of TiKV. For details, see Compatibility between TiKV versions.

BR S3 server-side encryption

To enable S3 server-side encryption when backup to S3 using BR, pass --s3.sse argument and set value to "aws:kms". S3 will use its own KMS key for encryption. Example:

tiup br backup full --pd <pd-address> --storage "s3://<bucket>/<prefix>" --s3.sse aws:kms

To use a custom AWS KMS CMK that you created and owned, pass --s3.sse-kms-key-id in addition. In this case, both the BR process and all the TiKV nodes in the cluster would need access to the KMS CMK (for example, via AWS IAM), and the KMS CMK needs to be in the same AWS region as the S3 bucket used to store the backup. It is advised to grant access to the KMS CMK to BR process and TiKV nodes via AWS IAM. Refer to AWS documentation for usage of IAM. For example:

tiup br backup full --pd <pd-address> --storage "s3://<bucket>/<prefix>" --s3.sse aws:kms --s3.sse-kms-key-id 0987dcba-09fe-87dc-65ba-ab0987654321

When restoring the backup, both --s3.sse and --s3.sse-kms-key-id should NOT be used. S3 will figure out encryption settings by itself. The BR process and TiKV nodes in the cluster to restore the backup to would also need access to the KMS CMK, or the restore will fail. Example:

tiup br restore full --pd <pd-address> --storage "s3://<bucket>/<prefix>"

BR Azure Blob Storage server-side encryption

When backing up data to Azure Blob Storage using BR, you can specify either an encryption scope or an encryption key for server-side encryption.

Method 1: use an encryption scope

To specify an encryption scope for the backup data, you can use one of the following two ways:

  • Include the --azblob.encryption-scope option in the backup command and set it to the scope name:

    tiup br backup full --pd <pd-address> --storage "azure://<bucket>/<prefix>" --azblob.encryption-scope scope1

  • Include encryption-scope in the URI and set it to the scope name:

    tiup br backup full --pd <pd-address> --storage "azure://<bucket>/<prefix>?encryption-scope=scope1"

For more information, see the Azure documentation: Upload a blob with an encryption scope.

When restoring the backup, you do not need to specify the encryption scope. Azure Blob Storage automatically decrypts the data. For example:

tiup br restore full --pd <pd-address> --storage "azure://<bucket>/<prefix>"

Method 2: use an encryption key

To specify an encryption key for the backup data, you can use one of the following three ways:

  • Include the --azblob.encryption-key option in the backup command and set it to an AES256 encryption key:

    tiup br backup full --pd <pd-address> --storage "azure://<bucket>/<prefix>" --azblob.encryption-key <aes256-key>

  • Include encryption-key in the URI and set it to an AES256 encryption key. If the key contains URI reserved characters such as & and %, you need to percent-encode it first:

    tiup br backup full --pd <pd-address> --storage "azure://<bucket>/<prefix>?encryption-key=<aes256-key>"

  • Set the AZURE_ENCRYPTION_KEY environment variable to an AES256 encryption key. Before running, make sure that you remember the encryption key in the environment variable to avoid forgetting it.

    export AZURE_ENCRYPTION_KEY=<aes256-key>
tiup br backup full --pd <pd-address> --storage "azure://<bucket>/<prefix>"

For more information, see the Azure documentation: Provide an encryption key on a request to Blob storage.

When restoring the backup, you need to specify the encryption key. For example:

  • Include the --azblob.encryption-key option in the restore command:

    tiup br restore full --pd <pd-address> --storage "azure://<bucket>/<prefix>" --azblob.encryption-key <aes256-key>

  • Include encryption-key in the URI:

    tiup br restore full --pd <pd-address> --storage "azure://<bucket>/<prefix>?encryption-key=<aes256-key>"

  • Set the AZURE_ENCRYPTION_KEY environment variable:

    export AZURE_ENCRYPTION_KEY=<aes256-key>
tiup br restore full --pd <pd-address> --storage "azure://<bucket>/<prefix>"