diff --git a/content/embeds/aur-rds-mysql-create-debezium-user.md b/content/embeds/aur-rds-mysql-create-debezium-user.md new file mode 100644 index 0000000000..949ba4415c --- /dev/null +++ b/content/embeds/aur-rds-mysql-create-debezium-user.md @@ -0,0 +1,36 @@ +## Create Debezium user + +The Debezium connector needs a user account to connect to MySQL. This +user must have appropriate permissions on all databases where you want Debezium +to capture changes. + +1. Connect to your database as an admin user and create a new user for the connector: + + ```sql + CREATE USER ''@'%' IDENTIFIED BY ''; + ``` + + Replace `` and `` with a username and password for the new user. + + The `%` means that the user can connect from any client. If you want to restrict the user to connect only from the RDI host, replace `%` with the IP address of the RDI host. + +1. Grant the user the necessary permissions: + + ```sql + GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO ''@'%'; + ``` + + Replace `` with the username of the Debezium user. + + You can also grant SELECT permissions for specific tables only. The other permissions are global and cannot be restricted to specific tables. + + ```sql + GRANT RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO ''@'%'; + GRANT SELECT ON . TO ''@'%'; + ``` + +1. Finalize the user's permissions: + + ```sql + FLUSH PRIVILEGES; + ``` \ No newline at end of file diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql.md deleted file mode 100644 index b4017b661d..0000000000 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -Title: Prepare AWS Aurora and PostgreSQL for RDI -aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/my-sql-mariadb/ -alwaysopen: false -categories: -- docs -- integrate -- rs -- rdi -description: Prepare AWS Aurora/PostgreSQL databases to work with RDI -group: di -linkTitle: Prepare AWS Aurora/PostgreSQL -summary: Redis Data Integration keeps Redis in sync with the primary database in near - real time. -type: integration -weight: 5 ---- - -Follow the steps in the sections below to prepare an -[AWS Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) -database to work with RDI. - -## 1. Create a parameter group - -In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), -navigate to **Parameter groups > Create parameter group**. You will see the panel shown -below: - -{{}} - -Enter the following information: - -| Name | Value | -| :-- | :-- | -| **Parameter group name** | rdi-aurora-pg | -| **Description** | Enable logical replication for RDI | -| **Engine Type** | Aurora PostgreSQL | -| **Parameter group family** | aurora-postgresql15 | -| **Type** | DB Cluster Parameter Group | - -Select **Create** to create the parameter group. - -## 2. Edit the parameter group - -Navigate to **Parameter groups** in the console. Select the `rdi-aurora-pg` -group you have just created and then select **Edit** . You will see this panel: - -{{}} - -Search for the `rds.logical_replication` parameter and set its value to 1. Then, -select **Save Changes**. - -## 3. Select the new parameter group - -Go back to your target database on the RDS console, select **Modify** and then -scroll down to **Additional Configuration**. Set -the **DB Cluster Parameter Group** to the value `rdi-aurora-pg` that you have just added: - -{{}} diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md new file mode 100644 index 0000000000..061188eb18 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md @@ -0,0 +1,22 @@ +--- +Title: Prepare AWS RDS and Aurora databases for RDI +aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/ +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Learn how to prepare AWS RDS and Aurora databases for RDI. +group: di +linkTitle: Prepare AWS RDS and Aurora +summary: Prepare AWS Aurora and AWS RDS databases to work with Redis Data Integration. +hideListLinks: false +type: integration +weight: 5 +--- + +You can use RDI with databases on [AWS Relational Database Service (RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) and [AWS Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html). + +The pages in this section give detailed instructions to get your source +database ready for Debezium to use: \ No newline at end of file diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md new file mode 100644 index 0000000000..9df6c5cb0d --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md @@ -0,0 +1,101 @@ +--- +Title: Prepare AWS Aurora MySQL/AWS RDS MySQL for RDI +aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql/ +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Enable CDC features in your source databases +group: di +hideListLinks: false +linkTitle: Prepare AWS Aurora/RDS MySQL +summary: Prepare AWS Aurora MySQL and AWS RDS MySQL databases to work with Redis Data Integration. +type: integration +weight: 2 +--- + +Follow the steps in the sections below to prepare an [AWS Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) or [AWS RDS MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.MySQL.html) database. +database to work with RDI. + +Select the steps for your database type. + +{{< multitabs id="rds-aur-mysql" + tab1="AWS Aurora MySQL" + tab2="AWS RDS MySQL" >}} + +## Add an Aurora reader node + +RDI requires that your Aurora MySQL database has at least one replica or reader node. + +To add a reader node to an existing database, select **Add reader** from the **Actions** menu of the database and [add a reader node](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-replicas-adding.html). + +You can also create one during database creation by selecting **Create an Aurora Replica or Reader node in a different AZ (recommended for scaled availability)** under **Availability & durability > Multi-AZ deployment**. + +## Create and apply parameter group + +RDI requires some changes to database parameters. On AWS Aurora, you change these parameters via a parameter group. + +1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings: + + | Name | Value | + | :-- | :-- | + | **Parameter group name** | Enter a suitable parameter group name, like `rdi-mysql` | + | **Description** | (Optional) Enter a description for the parameter group | + | **Engine Type** | Choose **Aurora MySQL**. | + | **Parameter group family** | Choose **aurora-mysql8.0**. | + | **Type** | Select **DB Parameter Group**. | + + Select **Create** to create the parameter group. + +1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters: + + | Name | Value | + | :-- | :-- | + | `binlog_format` | `ROW` | + | `binlog_row_image` | `FULL` | + | `gtid_mode` | `ON` | + | `enforce_gtid_consistency` | `ON` | + + Select **Save Changes** to apply the changes to the parameter group. + +1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created. + + Select **Save changes** to apply the parameter group to the new database. + +{{< embed-md "aur-rds-mysql-create-debezium-user.md" >}} + +-tab-sep- + +## Create and apply parameter group + +RDI requires some changes to database parameters. On AWS RDS, you change these parameters via a parameter group. + +1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings: + + | Name | Value | + | :-- | :-- | + | **Parameter group name** | Enter a suitable parameter group name, like `rdi-mysql` | + | **Description** | (Optional) Enter a description for the parameter group | + | **Engine Type** | Choose **MySQL Community**. | + | **Parameter group family** | Choose **mysql8.0**. | + + Select **Create** to create the parameter group. + +1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters: + + | Name | Value | + | :-- | :-- | + | `binlog_format` | `ROW` | + | `binlog_row_image` | `FULL` | + + Select **Save Changes** to apply the changes to the parameter group. + +1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created. + + Select **Save changes** to apply the parameter group to the new database. + +{{< embed-md "aur-rds-mysql-create-debezium-user.md" >}} + +{{< /multitabs >}} diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md new file mode 100644 index 0000000000..ec2dee5207 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md @@ -0,0 +1,82 @@ +--- +Title: Prepare AWS Aurora PostgreSQL/AWS RDS PostgreSQL for RDI +aliases: +- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aur-pgsql/ +- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql/ +- /integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql/ +alwaysopen: false +categories: +- docs +- integrate +- rs +- rc +- rdi +description: Prepare AWS Aurora PostgreSQL databases to work with RDI +group: di +linkTitle: Prepare AWS Aurora PostgreSQL +summary: Prepare AWS Aurora PostgreSQL databases to work with Redis Data Integration. +type: integration +weight: 1 +--- + +Follow the steps in the sections below to prepare an +[AWS Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) or [AWS RDS PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html) +database to work with RDI. + +## Create and apply parameter group + +RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group. + +1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings: + + | Name | Value | + | :-- | :-- | + | **Parameter group name** | Enter a suitable parameter group name, like `rdi-aurora-pg` or `rdi-rds-pg` | + | **Description** | (Optional) Enter a description for the parameter group | + | **Engine Type** | Choose **Aurora PostgreSQL** for Aurora PostgreSQL or **PostgreSQL** for AWS RDS PostgreSQL. | + | **Parameter group family** | Choose **aurora-postgresql15** for Aurora PostgreSQL or **postgresql13** for AWS RDS PostgreSQL. | + + Select **Create** to create the parameter group. + +1. Navigate to **Parameter groups** in the console. Select the group you have just created and then select **Edit**. Change the following parameters: + + | Name | Value | + | :-- | :-- | + | `rds.logical_replication` | `1` | + + Select **Save Changes** to apply the changes to the parameter group. + +1. Go back to your database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created. + + Select **Save changes** to apply the parameter group to your database. + +## Create Debezium user + +The Debezium connector needs a user account to connect to PostgreSQL. This +user must have appropriate permissions on all databases where you want Debezium +to capture changes. + +1. Connect to PostgreSQL as the `postgres` user and create a new user for the connector: + + ```sql + CREATE ROLE WITH LOGIN PASSWORD '' VALID UNTIL 'infinity'; + ``` + + Replace `` and `` with a username and password for the new user. + +1. Grant the user the necessary replication permissions: + + ```sql + GRANT rds_replication TO ; + ``` + + Replace `` with the username of the Debezium user. + +1. Connect to your database as the `postgres` user and grant the new user access to one or more schemas in the database: + + ```sql + GRANT SELECT ON ALL TABLES IN SCHEMA TO ; + ``` + + Replace `` with the username of the Debezium user and `` with the schema name. + diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver.md new file mode 100644 index 0000000000..5d7dbca456 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver.md @@ -0,0 +1,104 @@ +--- +Title: Prepare Microsoft SQL Server on AWS RDS for RDI +aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver/ +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Enable CDC features in your source databases +group: di +hideListLinks: false +linkTitle: Prepare Microsoft SQL Server on AWS RDS +summary: Prepare Microsoft SQL Server on AWS RDS databases to work with Redis Data Integration. +type: integration +weight: 3 +--- + +Follow the steps in the sections below to prepare a [Microsoft SQL Server on AWS RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.SQLServer.html) database to work with RDI. + +{{< note >}} +Change Data Capture (CDC) is not supported on SQL Server Express Edition. Only the Standard, Enterprise, and Developer editions support CDC and are supported by RDI. +{{< /note >}} + +## Create the Debezium user + +The Debezium connector needs a user account to connect to SQL Server. This +user must have appropriate permissions on all databases where you want Debezium +to capture changes. + +1. Connect to your database as an admin user and create a new user for the connector: + + ```sql + USE master + GO + CREATE LOGIN WITH PASSWORD = '' + GO + USE + GO + CREATE USER FOR LOGIN + GO + ``` + + Replace `` and `` with a username and password for the new user and replace `` with the name of your database. + +1. Grant the user the necessary permissions: + + ```sql + USE master + GO + GRANT VIEW SERVER STATE TO + GO + USE + GO + EXEC sp_addrolemember N'db_datareader', N'' + GO + ``` + + Replace `` with the username of the Debezium user and replace `` with the name of your database. + +## Enable CDC on the database + +Change Data Capture (CDC) must be enabled for the database and for each table you want to capture. + +1. Enable CDC for the database by running the following command: + + ```sql + EXEC msdb.dbo.rds_cdc_enable_db '' + GO + ``` + + Replace `` with the name of your database. + +1. Enable CDC for each table you want to capture by running the following commands: + + ```sql + USE + GO + EXEC sys.sp_cdc_enable_table + @source_schema = N'', + @source_name = N'
', + @role_name = N'', + @supports_net_changes = 0 + GO + ``` + + Replace `` with the name of your database, `` with the name of the schema containing the table, `
` with the name of the table, and `` with the name of a new role that will be created to manage access to the CDC data. + + {{< note >}} +The value for `@role_name` can’t be a fixed database role, such as `db_datareader`. +Specifying a new name will create a corresponding database role that has full access to the +captured change data. + {{< /note >}} + +1. Add the Debezium user to the CDC role: + + ```sql + USE + GO + EXEC sp_addrolemember N'', N'' + GO + ``` + + Replace `` with the name of the role you created in the previous step and replace `` with the username of the Debezium user. \ No newline at end of file