Skip to content

Commit

Permalink
changefeed: add private link access steps
Browse files Browse the repository at this point in the history
  • Loading branch information
grovecai committed Oct 17, 2024
1 parent 6a2b86b commit 5e94493
Show file tree
Hide file tree
Showing 3 changed files with 54 additions and 10 deletions.
3 changes: 3 additions & 0 deletions TOC-tidb-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,9 @@
- [To Kafka Sink](/tidb-cloud/changefeed-sink-to-apache-kafka.md)
- [To TiDB Cloud Sink](/tidb-cloud/changefeed-sink-to-tidb-cloud.md)
- [To Cloud Storage](/tidb-cloud/changefeed-sink-to-cloud-storage.md)
- Reference
- [Setup Self Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-self-hosted-kafka-pls.md)
- [Setup Self Hosted Kafka Private Service Connect in GCP](/tidb-cloud/setup-self-hosted-kafka-psc.md)
- Disaster Recovery
- [Recovery Group Overview](/tidb-cloud/recovery-group-overview.md)
- [Get Started](/tidb-cloud/recovery-group-get-started.md)
Expand Down
48 changes: 40 additions & 8 deletions tidb-cloud/changefeed-sink-to-apache-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,21 @@ Before creating a changefeed to stream data to Apache Kafka, you need to complet

### Network

Make sure that your TiDB cluster can connect to the Apache Kafka service.
Make sure that your TiDB cluster can connect to the Apache Kafka service. There are 3 kinds of network connection methods can be used to connect to Kafka.
1. Private Connect
2. VPC Peering
3. Public IP

If you want a quick try, you can choose **Public IP**. If you want cost-effective, you can choose **VPC Peering**, trade off VPC CIDR conflict and security. If you want to get rid of VPC CIDR conflict and satisfy security compliance, **Private Connect** is the choice, but it will introduce extra [Private Data Link Cost](/tidbcloud/tidb-cloud-billing-ticdc-rcu.md#private-data-link-cost)

#### Private Connect
Private Connect leverages Private Link or Private Service Connect technologies which provided by cloud vendors, that allow the resources in your VPC to connect to services in other VPCs using private IP addresses, as if those services were hosted directly in your VPC.

Currently, we only support Private Connect to self-hosted Kafka.
1. If your Apache Kafka service already or will be setup in AWS, please follow [Setup Self Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-self-hosted-kafka-pls.md) to make sure the network connection is set up properly.
2. If your Apache Kafka service already or will be setup in GCP, please follow [Setup Self Hosted Kafka Private Service Connect in GCP](/tidb-cloud/setup-self-hosted-kafka-psc.md) to make sure the network connection is set up properly.

#### VPC Peering

If your Apache Kafka service is in an AWS VPC that has no internet access, take the following steps:

Expand All @@ -39,7 +53,7 @@ If your Apache Kafka service is in an AWS VPC that has no internet access, take

3. If the Apache Kafka URL contains hostnames, you need to allow TiDB Cloud to be able to resolve the DNS hostnames of the Apache Kafka brokers.

1. Follow the steps in [Enable DNS resolution for a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/modify-peering-connections.html#vpc-peering-dns).
1. Follow the steps in [Enable DNS resolution for a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-dns.html).
2. Enable the **Accepter DNS resolution** option.

If your Apache Kafka service is in a Google Cloud VPC that has no internet access, take the following steps:
Expand All @@ -49,6 +63,10 @@ If your Apache Kafka service is in a Google Cloud VPC that has no internet acces

You must add the CIDR of the region where your TiDB Cloud cluster is located to the ingress firewall rules. The CIDR can be found on the **VPC Peering** page. Doing so allows the traffic to flow from your TiDB cluster to the Kafka brokers.

#### Public IP

If you want to provide Public IP access to your Apache Kafka service, you need to assign Public IPs to all you Kafka brokers. It is not recommend to use Public IP in production environment for security consideration.

### Kafka ACL authorization

To allow TiDB Cloud changefeeds to stream data to Apache Kafka and create Kafka topics automatically, ensure that the following permissions are added in Kafka:
Expand All @@ -65,14 +83,28 @@ For example, if your Kafka cluster is in Confluent Cloud, you can see [Resources

## Step 2. Configure the changefeed target

1. Under **Brokers Configuration**, fill in your Kafka brokers endpoints. You can use commas `,` to separate multiple endpoints.
2. Select an authentication option according to your Kafka authentication configuration.

TODO:
1. For **Kafka Provider**, we only provide **Self-hosted Kafka** option, we will support more later.
> **Note:**
> Currently, we treat all the Apache Kafka Services as self-hosted since we didn't make any special integration to different Kafka Providers, such as Amazon MSK, Confluent ... It doesn't mean that we can not connect to Amazon MSK or Confluent Kafka. If the Kafka Provider can provide standard network connection methods, just like VPC Peering, Public IP, Private Link and Private Service Connect, we definitely can connect to them. You may have question "Can you connect to Amazon MSK by multi VPC which is powered by Private Link technology?" Sorry, we haven't supported it yet since it's not a standard Private Link, but may be later.
2. Select **Connectivity Method** by your Apache Kafka Service setup.
1. If you select **VPC Peering** or **Public IP**, fill in your Kafka brokers endpoints. You can use commas `,` to separate multiple endpoints.
2. If you select **Private Link**
1. Make sure you select the same **Kafka Type**, **Suggested Kafka Endpoint Service AZ** and fill the same unique ID in **Kafka Advertised Listener Pattern** when you [Setup Self Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-self-hosted-kafka-pls.md) in **Network** section.
2. Double-check the **Kafka Advertised Listener Pattern** by clicking the button **Check usage and generate**, which will show message to help you validate the unique ID.
3. Fill the **Endpoint Service Name** which is configured in [Setup Self Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-self-hosted-kafka-pls.md)
4. Fill the **Boostrap Ports**, suggest at least one port for one AZ. You can use commas `,` to separate multiple ports.
3. If you select **Private Service Connect**
1. Make sure you fill the same unique ID in **Kafka Advertised Listener Pattern** when you [Setup Self Hosted Kafka Private Service Connect in GCP](/tidb-cloud/setup-self-hosted-kafka-psc.md) in **Network** section.
2. Double-check the **Kafka Advertised Listener Pattern** by clicking the button **Check usage and generate**, which will show message to help you validate the unique ID.
3. Fill the **Service Attachment** which is configured in [Setup Self Hosted Kafka Private Service Connect in GCP](/tidb-cloud/setup-self-hosted-kafka-psc.md)
4. Fill the **Boostrap Ports**, suggest provide more than one ports. You can use commas `,` to separate multiple ports.
2. Select an **Authentication** option according to your Kafka authentication configuration.
- If your Kafka does not require authentication, keep the default option **Disable**.
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the user name and password of your Kafka account for authentication.
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.

3. Select your Kafka version. If you do not know that, use Kafka V2.
4. Select a desired compression type for the data in this changefeed.
3. Select your **Kafka Version**. If you do not know that, use Kafka V2.
4. Select a desired **Compression** type for the data in this changefeed.
5. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
6. Click **Next** to check the configurations you set and go to the next page.

Expand Down
13 changes: 11 additions & 2 deletions tidb-cloud/tidb-cloud-billing-ticdc-rcu.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@ aliases: ['/tidbcloud/tidb-cloud-billing-tcu']

# Changefeed Billing

## RCU Cost

TiDB Cloud measures the capacity of [changefeeds](/tidb-cloud/changefeed-overview.md) in TiCDC Replication Capacity Units (RCUs). When you [create a changefeed](/tidb-cloud/changefeed-overview.md#create-a-changefeed) for a cluster, you can select an appropriate specification. The higher the RCU, the better the replication performance. You will be charged for these TiCDC changefeed RCUs.

## Number of TiCDC RCUs
### Number of TiCDC RCUs

The following table lists the specifications and corresponding replication performances for changefeeds:

Expand All @@ -26,6 +28,13 @@ The following table lists the specifications and corresponding replication perfo
>
> The preceding performance data is for reference only and might vary in different scenarios. It is strongly recommended that you conduct a real workload test before using the changefeed feature in a production environment. For further assistance, contact [TiDB Cloud support](/tidb-cloud/tidb-cloud-support.md#get-support-for-a-cluster).
## Price
### Price

To learn about the supported regions and the price of TiDB Cloud for each TiCDC RCU, see [Changefeed Cost](https://www.pingcap.com/tidb-cloud-pricing-details/#changefeed-cost).

## Private Data Link Cost

If you choose "Private Link" or "Private Service Connect" network connectivity method, we will charge you extra "Private Data Link" cost which in [Data Transfer Cost](https://www.pingcap.com/tidb-dedicated-pricing-details/#data-transfer-cost) category.

The price of "Private Data Link" will be **$ 0.01 per GiB**, same as **Data Processed** of [AWS Interface Endpoint pricing](https://aws.amazon.com/privatelink/pricing/#Interface_Endpoint_pricing) and **Consumer data processing** of [Google Cloud Private Service Connect pricing](https://cloud.google.com/vpc/pricing#psc-forwarding-rules)

0 comments on commit 5e94493

Please sign in to comment.