Skip to content

Cherry-pick RDI with RDS/Aurora instructions from preview-rc-rdi to main #1707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions content/embeds/aur-rds-mysql-create-debezium-user.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
## Create Debezium user

The Debezium connector needs a user account to connect to MySQL. This
user must have appropriate permissions on all databases where you want Debezium
to capture changes.

1. Connect to your database as an admin user and create a new user for the connector:

```sql
CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
```

Replace `<username>` and `<password>` with a username and password for the new user.

The `%` means that the user can connect from any client. If you want to restrict the user to connect only from the RDI host, replace `%` with the IP address of the RDI host.

1. Grant the user the necessary permissions:

```sql
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO '<username>'@'%';
```

Replace `<username>` with the username of the Debezium user.

You can also grant SELECT permissions for specific tables only. The other permissions are global and cannot be restricted to specific tables.

```sql
GRANT RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO '<username>'@'%';
GRANT SELECT ON <database>.<table> TO '<username>'@'%';
```

1. Finalize the user's permissions:

```sql
FLUSH PRIVILEGES;
```

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
Title: Prepare AWS RDS and Aurora databases for RDI
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/
alwaysopen: false
categories:
- docs
- integrate
- rs
- rdi
description: Learn how to prepare AWS RDS and Aurora databases for RDI.
group: di
linkTitle: Prepare AWS RDS and Aurora
summary: Prepare AWS Aurora and AWS RDS databases to work with Redis Data Integration.
hideListLinks: false
type: integration
weight: 5
---

You can use RDI with databases on [AWS Relational Database Service (RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) and [AWS Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html).

The pages in this section give detailed instructions to get your source
database ready for Debezium to use:
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
---
Title: Prepare AWS Aurora MySQL/AWS RDS MySQL for RDI
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql/
alwaysopen: false
categories:
- docs
- integrate
- rs
- rdi
description: Enable CDC features in your source databases
group: di
hideListLinks: false
linkTitle: Prepare AWS Aurora/RDS MySQL
summary: Prepare AWS Aurora MySQL and AWS RDS MySQL databases to work with Redis Data Integration.
type: integration
weight: 2
---

Follow the steps in the sections below to prepare an [AWS Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) or [AWS RDS MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.MySQL.html) database.
database to work with RDI.

Select the steps for your database type.

{{< multitabs id="rds-aur-mysql"
tab1="AWS Aurora MySQL"
tab2="AWS RDS MySQL" >}}

## Add an Aurora reader node

RDI requires that your Aurora MySQL database has at least one replica or reader node.

To add a reader node to an existing database, select **Add reader** from the **Actions** menu of the database and [add a reader node](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-replicas-adding.html).

You can also create one during database creation by selecting **Create an Aurora Replica or Reader node in a different AZ (recommended for scaled availability)** under **Availability & durability > Multi-AZ deployment**.

## Create and apply parameter group

RDI requires some changes to database parameters. On AWS Aurora, you change these parameters via a parameter group.

1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:

| Name | Value |
| :-- | :-- |
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-mysql` |
| **Description** | (Optional) Enter a description for the parameter group |
| **Engine Type** | Choose **Aurora MySQL**. |
| **Parameter group family** | Choose **aurora-mysql8.0**. |
| **Type** | Select **DB Parameter Group**. |

Select **Create** to create the parameter group.

1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:

| Name | Value |
| :-- | :-- |
| `binlog_format` | `ROW` |
| `binlog_row_image` | `FULL` |
| `gtid_mode` | `ON` |
| `enforce_gtid_consistency` | `ON` |

Select **Save Changes** to apply the changes to the parameter group.

1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.

Select **Save changes** to apply the parameter group to the new database.

{{< embed-md "aur-rds-mysql-create-debezium-user.md" >}}

-tab-sep-

## Create and apply parameter group

RDI requires some changes to database parameters. On AWS RDS, you change these parameters via a parameter group.

1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:

| Name | Value |
| :-- | :-- |
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-mysql` |
| **Description** | (Optional) Enter a description for the parameter group |
| **Engine Type** | Choose **MySQL Community**. |
| **Parameter group family** | Choose **mysql8.0**. |

Select **Create** to create the parameter group.

1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:

| Name | Value |
| :-- | :-- |
| `binlog_format` | `ROW` |
| `binlog_row_image` | `FULL` |

Select **Save Changes** to apply the changes to the parameter group.

1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.

Select **Save changes** to apply the parameter group to the new database.

{{< embed-md "aur-rds-mysql-create-debezium-user.md" >}}

{{< /multitabs >}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
---
Title: Prepare AWS Aurora PostgreSQL/AWS RDS PostgreSQL for RDI
aliases:
- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aur-pgsql/
- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql/
- /integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql/
alwaysopen: false
categories:
- docs
- integrate
- rs
- rc
- rdi
description: Prepare AWS Aurora PostgreSQL databases to work with RDI
group: di
linkTitle: Prepare AWS Aurora PostgreSQL
summary: Prepare AWS Aurora PostgreSQL databases to work with Redis Data Integration.
type: integration
weight: 1
---

Follow the steps in the sections below to prepare an
[AWS Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) or [AWS RDS PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html)
database to work with RDI.

## Create and apply parameter group

RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group.

1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:

| Name | Value |
| :-- | :-- |
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-aurora-pg` or `rdi-rds-pg` |
| **Description** | (Optional) Enter a description for the parameter group |
| **Engine Type** | Choose **Aurora PostgreSQL** for Aurora PostgreSQL or **PostgreSQL** for AWS RDS PostgreSQL. |
| **Parameter group family** | Choose **aurora-postgresql15** for Aurora PostgreSQL or **postgresql13** for AWS RDS PostgreSQL. |

Select **Create** to create the parameter group.

1. Navigate to **Parameter groups** in the console. Select the group you have just created and then select **Edit**. Change the following parameters:

| Name | Value |
| :-- | :-- |
| `rds.logical_replication` | `1` |

Select **Save Changes** to apply the changes to the parameter group.

1. Go back to your database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.

Select **Save changes** to apply the parameter group to your database.

## Create Debezium user

The Debezium connector needs a user account to connect to PostgreSQL. This
user must have appropriate permissions on all databases where you want Debezium
to capture changes.

1. Connect to PostgreSQL as the `postgres` user and create a new user for the connector:

```sql
CREATE ROLE <username> WITH LOGIN PASSWORD '<password>' VALID UNTIL 'infinity';
```

Replace `<username>` and `<password>` with a username and password for the new user.

1. Grant the user the necessary replication permissions:

```sql
GRANT rds_replication TO <username>;
```

Replace `<username>` with the username of the Debezium user.

1. Connect to your database as the `postgres` user and grant the new user access to one or more schemas in the database:

```sql
GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <username>;
```

Replace `<username>` with the username of the Debezium user and `<schema>` with the schema name.

Loading