Add setup self hosted Kafka PSC for GCP

pingcap · Nov 7, 2024 · f77fbed · f77fbed
1 parent 14247e6
commit f77fbed
Show file tree

Hide file tree

Showing 2 changed files with 63 additions and 7 deletions.
diff --git a/tidb-cloud/setup-self-hosted-kafka-pls.md b/tidb-cloud/setup-self-hosted-kafka-pls.md
@@ -27,15 +27,14 @@ Let's show how to connect to a three AZ Kafka private link service in AWS by exa
     - Manage Load Balancer
     - Manage Endpoint Service
     - Connect to EC2 Nodes to configure Kafka nodes
-2. Make sure that you create a TiDB Cloud Dedicated cluster first. Align Kafka deployment info with TiDB Cluster.
+2. Make sure that you create a TiDB Cloud Dedicated cluster in AWS first. Align Kafka deployment info with TiDB Cluster.
    1. In the [TiDB Cloud console](https://tidbcloud.com), navigate to the cluster overview page of the TiDB cluster, and then click **Changefeed** in the left navigation pane.
    2. In the overview page, you can find the region of TiDB Cluster, make sure your Kafka cluster will be deployed to the same region.
    3. Click **Create Changefeed**
       1. Select **Kafka** as **Target Type**.
-      2. Select **Self-hosted Kafka** as **Kafka Provider**
-      3. Select **Private Link** as **Connectivity Method**
-   4. Take note the AWS account arn in **Reminders before proceeding** information, which your will use it to authorize TiDB Cloud to create endpoint for the Kafka private link service.
-   5. Select **Kafka Type**, confirm you will deploy Kafka cluster to **Single AZ** or **3 AZ**. Here we select **3 AZ**. Take note of the AZ IDs you want to deploy your Kafka cluster. If you don't know the relationship between your AZ names and AZ IDs, please refer to [AWS document](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html) to find it.
+      2. Select **Private Link** as **Connectivity Method**
+   4. Take note the principal of TiDB Cloud AWS account in **Reminders before proceeding** information, which your will use it to authorize TiDB Cloud to create endpoint for the Kafka private link service.
+   5. Select **Number of AZs**, confirm you will deploy Kafka cluster to **Single AZ** or **3 AZ**. Here we select **3 AZ**. Take note of the AZ IDs you want to deploy your Kafka cluster. If you don't know the relationship between your AZ names and AZ IDs, please refer to [AWS document](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html) to find it.
    6. Pick a unique **Kafka Advertised Listener Pattern** for your Kafka Private Link service
       1. Input a unique random string can only include numbers or lowercase letters, which will be used to generate **Kafka Advertised Listener Pattern** later.
       2. Click **Check usage and generate** button to check if the random string is unique and generate **Kafka Advertised Listener Pattern** which will be used to assemble EXTERNAL advertised listener for kafka brokers. 

diff --git a/tidb-cloud/setup-self-hosted-kafka-psc.md b/tidb-cloud/setup-self-hosted-kafka-psc.md
@@ -5,6 +5,63 @@ summary: This document explains how to set up private service connect for self-h
 
 # Setup Self-hosted Kafka Private Service Connect in Google Cloud
 
+This document explains how to set up private service connect for self-hosted Kafka in Google Cloud and how to make it work with TiDB Cloud.
+
+The main idea is the same as we do in AWS:
+1. TiDB Cloud VPC connects to Kafka VPC through limit private endpoints.
+2. Kafka clients need to talk directly to all Kafka brokers.
+3. Therefore, we need to map every Kafka brokers to different ports to make every broker is unique in TiDB Cloud VPC.
+4. We will leverage Kafka bootstrap mechanism and Google Cloud resources to achieve the mapping.
+
 There are two ways to setup private service connect for self-hosted Kafka in Google Cloud:
-1. by PSC port mapping: new kafka cluster & existed kafka cluster
-2. by kafka proxy: new kafka cluster & existed kafka cluster
+1. Using PSC port mapping mechanism, which requires static port-broker mapping configuration. Require to reconfigure existing Kafka cluster to add a group of EXTERNAL listener and advertised listener. 
+2. Using [Kafka-proxy](https://github.com/grepplabs/kafka-proxy), which introduces a extra running process as proxy between Kafka clients and Kafka brokers, the proxy will dynamic configure port-broker mapping and forward requests. No need to reconfigure existing Kafka cluster.
+
+
+## Prerequisites
+1. Make sure you have authorization to set up Kafka private service connect in your own Google Cloud account. 
+    - Manage VM Nodes
+    - Manage VPC
+    - Manage Subnet
+    - Manage Load Balancer
+    - Manage Private Service Connect
+    - Connect to VM Nodes to configure Kafka nodes
+2. Make sure that you create a TiDB Cloud Dedicated cluster in Google Cloud first. Align Kafka deployment info with TiDB Cluster.
+   1. In the [TiDB Cloud console](https://tidbcloud.com), navigate to the cluster overview page of the TiDB cluster, and then click **Changefeed** in the left navigation pane.
+   2. In the overview page, you can find the region of TiDB Cluster, make sure your Kafka cluster will be deployed to the same region.
+   3. Click **Create Changefeed**
+      1. Select **Kafka** as **Target Type**.
+      2. Select **Private Service Connect** as **Connectivity Method**
+   4. Take note the Google Cloud project in **Reminders before proceeding** information, which your can use it to authorize auto-accept endpoint creation request from TiDB Cloud.
+   5. Take note of the **Suggested Kafka Zones**. Here are the Zones where the TiDB Cluster is deployed. It is recommended that Kafka to be deployed in these Zones as well to reduce cross-zone traffic.
+   6. Pick a unique **Kafka Advertised Listener Pattern** for your Kafka private service connect
+      1. Input a unique random string can only include numbers or lowercase letters, which will be used to generate **Kafka Advertised Listener Pattern** later.
+      2. Click **Check usage and generate** button to check if the random string is unique and generate **Kafka Advertised Listener Pattern** which will be used to assemble EXTERNAL advertised listener for kafka brokers, or configure Kafka-proxy. 
+
+Please take note of all this deployment information, use them to configure your Kafka private service connect.
+Example of deployment information.
+
+| Information                        | Value                                                                                                                        |
+|------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
+| Region                             | Oregon (us-west1)                                                                                                            |
+| Google Cloud project of TiDB Cloud | tidbcloud-prod-000                                                                                                           |
+| Zones                              | 1. us-west1-a <br/> 2. us-west1-b <br/> 3. us-west1-c                                                                        |
+| Kafka Advertised Listener Pattern  | The unique random string: abc <br/> Generated pattern: &lt;broker_id&gt;.abc.us-west1.gcp.3199745.tidbcloud.com:&lt;port&gt; |
+
+
+
+## Setup Self-hosted Kafka Private Service Connect by PSC Port Mapping
+TODO: graph
+
+### First, Setup Kafka Cluster
+#### Deploy a New Kafka Cluster
+#### Reconfigure a Running Kafka Cluster
+### Second, Expose Kafka Cluster as Private Link Service
+### Third, Connect from TiDB Cloud
+
+## Setup Self-hosted Kafka Private Service Connect by Kafka-proxy
+TODO: graph
+
+### First, Setup Kafka Cluster
+### Second, Expose Kafka Cluster as Private Link Service
+### Third, Connect from TiDB Cloud