You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tidb-cloud/changefeed-overview.md
+32-8Lines changed: 32 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,21 +9,21 @@ TiDB Cloud changefeed helps you stream data from TiDB Cloud to other data servic
9
9
10
10
> **Note:**
11
11
>
12
-
> - Currently, TiDB Cloud only allows up to 100 changefeeds per cluster.
12
+
> - Currently, TiDB Cloud only allows up to 100 changefeeds per <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent>.
13
13
> - Currently, TiDB Cloud only allows up to 100 table filter rules per changefeed.
14
14
> - For [{{{ .starter }}}](/tidb-cloud/select-cluster-tier.md#starter) and [{{{ .essential }}}](/tidb-cloud/select-cluster-tier.md#essential) clusters, the changefeed feature is unavailable.
15
15
16
16
## View the Changefeed page
17
17
18
18
To access the changefeed feature, take the following steps:
19
19
20
-
1. In the [TiDB Cloud console](https://tidbcloud.com), navigate to the [**Clusters**](https://tidbcloud.com/project/clusters) page of your project.
20
+
1. In the [TiDB Cloud console](https://tidbcloud.com), <CustomContentplan="dedicated">navigate to the [**Clusters**](https://tidbcloud.com/project/clusters) page of your project.</CustomContent><CustomContentplan="premium">navigate to the [**TiDB Instances**](https://tidbcloud.com/tidbs) page.</CustomContent>
21
21
22
22
> **Tip:**
23
23
>
24
24
> You can use the combo box in the upper-left corner to switch between organizations, projects, and clusters.
25
25
26
-
2. Click the name of your target cluster to go to its overview page, and then click **Data** > **Changefeed** in the left navigation pane. The changefeed page is displayed.
26
+
2. Click the name of your target <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent> to go to its overview page, and then click **Data** > **Changefeed** in the left navigation pane. The changefeed page is displayed.
27
27
28
28
On the **Changefeed** page, you can create a changefeed, view a list of existing changefeeds, and operate the existing changefeeds (such as scaling, pausing, resuming, editing, and deleting a changefeed).
29
29
@@ -36,22 +36,46 @@ To create a changefeed, refer to the tutorials:
36
36
-[Sink to TiDB Cloud](/tidb-cloud/changefeed-sink-to-tidb-cloud.md)
37
37
-[Sink to cloud storage](/tidb-cloud/changefeed-sink-to-cloud-storage.md)
38
38
39
-
## Query Changefeed RCUs
39
+
## Query changefeed capacity
40
+
41
+
<CustomContentplan="dedicated">
42
+
43
+
For TiDB Cloud Dedicated, you can query the TiCDC Replication Capacity Units (RCUs) of a changefeed.
40
44
41
45
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB cluster.
42
46
2. Locate the corresponding changefeed you want to query, and click **...** > **View** in the **Action** column.
43
47
3. You can see the current TiCDC Replication Capacity Units (RCUs) in the **Specification** area of the page.
44
48
49
+
</CustomContent>
50
+
<CustomContentplan="premium">
51
+
52
+
For {{{ .premium }}}, you can query the TiCDC Changefeed Capacity Units (CCUs) of a changefeed.
53
+
54
+
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB instance.
55
+
2. Locate the corresponding changefeed you want to query, and click **...** > **View** in the **Action** column.
56
+
3. You can see the current TiCDC Changefeed Capacity Units (CCUs) in the **Specification** area of the page.
57
+
58
+
</CustomContent>
59
+
45
60
## Scale a changefeed
46
61
62
+
<CustomContentplan="dedicated">
63
+
47
64
You can change the TiCDC Replication Capacity Units (RCUs) of a changefeed by scaling up or down the changfeed.
48
65
49
66
> **Note:**
50
67
>
51
68
> - To scale a changefeed for a cluster, make sure that all changefeeds for this cluster are created after March 28, 2023.
52
69
> - If a cluster has changefeeds created before March 28, 2023, neither the existing changefeeds nor newly created changefeeds for this cluster support scaling up or down.
53
70
54
-
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB cluster.
71
+
</CustomContent>
72
+
<CustomContentplan="premium">
73
+
74
+
You can change the TiCDC Changefeed Capacity Units (CCUs) of a changefeed by scaling up or down the changfeed.
75
+
76
+
</CustomContent>
77
+
78
+
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent>.
55
79
2. Locate the corresponding changefeed you want to scale, and click **...** > **Scale Up/Down** in the **Action** column.
56
80
3. Select a new specification.
57
81
4. Click **Submit**.
@@ -60,7 +84,7 @@ It takes about 10 minutes to complete the scaling process (during which the chan
60
84
61
85
## Pause or resume a changefeed
62
86
63
-
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB cluster.
87
+
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent>.
64
88
2. Locate the corresponding changefeed you want to pause or resume, and click **...** > **Pause/Resume** in the **Action** column.
65
89
66
90
## Edit a changefeed
@@ -69,7 +93,7 @@ It takes about 10 minutes to complete the scaling process (during which the chan
69
93
>
70
94
> TiDB Cloud currently only allows editing changefeeds in the paused status.
71
95
72
-
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB cluster.
96
+
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent>.
73
97
2. Locate the changefeed you want to pause, and click **...** > **Pause** in the **Action** column.
74
98
3. When the changefeed status changes to `Paused`, click **...** > **Edit** to edit the corresponding changefeed.
75
99
@@ -84,7 +108,7 @@ It takes about 10 minutes to complete the scaling process (during which the chan
84
108
85
109
## Delete a changefeed
86
110
87
-
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB cluster.
111
+
1. Navigate to the [**Changefeed**](#view-the-changefeed-page) page of your target TiDB <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent>.
88
112
2. Locate the corresponding changefeed you want to delete, and click **...** > **Delete** in the **Action** column.
Copy file name to clipboardExpand all lines: tidb-cloud/changefeed-sink-to-apache-kafka.md
+58-5Lines changed: 58 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,17 +7,31 @@ summary: This document explains how to create a changefeed to stream data from T
7
7
8
8
This document describes how to create a changefeed to stream data from TiDB Cloud to Apache Kafka.
9
9
10
+
<CustomContentplan="dedicated">
11
+
10
12
> **Note:**
11
13
>
12
14
> - To use the changefeed feature, make sure that your TiDB Cloud Dedicated cluster version is v6.1.3 or later.
13
15
> - For [{{{ .starter }}}](/tidb-cloud/select-cluster-tier.md#starter) and [{{{ .essential }}}](/tidb-cloud/select-cluster-tier.md#essential) clusters, the changefeed feature is unavailable.
14
16
17
+
</CustomContent>
18
+
<CustomContentplan="premium">
19
+
20
+
> **Note:**
21
+
>
22
+
> For [{{{ .starter }}}](/tidb-cloud/select-cluster-tier.md#starter) and [{{{ .essential }}}](/tidb-cloud/select-cluster-tier.md#essential) clusters, the changefeed feature is unavailable.
23
+
24
+
</CustomContent>
25
+
15
26
## Restrictions
16
27
17
-
- For each TiDB Cloud cluster, you can create up to 100 changefeeds.
28
+
- For each TiDB Cloud <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent>, you can create up to 100 changefeeds.
18
29
- Currently, TiDB Cloud does not support uploading self-signed TLS certificates to connect to Kafka brokers.
19
30
- Because TiDB Cloud uses TiCDC to establish changefeeds, it has the same [restrictions as TiCDC](https://docs.pingcap.com/tidb/stable/ticdc-overview#unsupported-scenarios).
20
31
- If the table to be replicated does not have a primary key or a non-null unique index, the absence of a unique constraint during replication could result in duplicated data being inserted downstream in some retry scenarios.
32
+
33
+
<CustomContentplan="dedicated">
34
+
21
35
- If you choose Private Link or Private Service Connect as the network connectivity method, ensure that your TiDB cluster version meets the following requirements:
22
36
23
37
- For v6.5.x: version v6.5.9 or later
@@ -30,6 +44,8 @@ This document describes how to create a changefeed to stream data from TiDB Clou
30
44
- If you want to distribute changelogs by primary key or index value to Kafka partition with a specified index name, make sure the version of your TiDB cluster is v7.5.0 or later.
31
45
- If you want to distribute changelogs by column value to Kafka partition, make sure the version of your TiDB cluster is v7.5.0 or later.
32
46
47
+
</CustomContent>
48
+
33
49
## Prerequisites
34
50
35
51
Before creating a changefeed to stream data to Apache Kafka, you need to complete the following prerequisites:
@@ -39,12 +55,14 @@ Before creating a changefeed to stream data to Apache Kafka, you need to complet
39
55
40
56
### Network
41
57
42
-
Ensure that your TiDB cluster can connect to the Apache Kafka service. You can choose one of the following connection methods:
58
+
Ensure that your TiDB <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent> can connect to the Apache Kafka service. You can choose one of the following connection methods:
43
59
44
60
- Private Connect: ideal for avoiding VPC CIDR conflicts and meeting security compliance, but incurs additional [Private Data Link Cost](/tidb-cloud/tidb-cloud-billing-ticdc-rcu.md#private-data-link-cost).
45
61
- VPC Peering: suitable as a cost-effective option, but requires managing potential VPC CIDR conflicts and security considerations.
46
62
- Public IP: suitable for a quick setup.
47
63
64
+
<CustomContentplan="dedicated">
65
+
48
66
<SimpleTab>
49
67
<divlabel="Private Connect">
50
68
@@ -87,6 +105,35 @@ It is **NOT** recommended to use Public IP in a production environment.
87
105
88
106
</div>
89
107
</SimpleTab>
108
+
</CustomContent>
109
+
110
+
<CustomContentplan="premium">
111
+
112
+
<SimpleTab>
113
+
<divlabel="Private Connect">
114
+
115
+
Private Connect leverages **Private Link** or **Private Service Connect** technologies from cloud providers to enable resources in your VPC to connect to services in other VPCs using private IP addresses, as if those services were hosted directly within your VPC.
116
+
117
+
TiDB Cloud currently supports Private Connect only for self-hosted Kafka. It does not support direct integration with MSK, Confluent Kafka, or other Kafka SaaS services. To connect to these Kafka SaaS services via Private Connect, you can deploy a [kafka-proxy](https://github.com/grepplabs/kafka-proxy) as an intermediary, effectively exposing the Kafka service as self-hosted Kafka.
118
+
119
+
If your Apache Kafka service is hosted on AWS, follow [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) to configure the network connection and obtain the **Bootstrap Ports** information, and then follow [Set Up Private Endpoint for Changefeeds](/tidb-cloud/premium/set-up-sink-private-endpoint-premium.md) to create a private endpoint.
120
+
121
+
</div>
122
+
<divlabel="Public IP">
123
+
124
+
If you want to provide Public IP access to your Apache Kafka service, assign Public IP addresses to all your Kafka brokers.
125
+
126
+
It is **NOT** recommended to use Public IP in a production environment.
127
+
128
+
</div>
129
+
130
+
<divlabel="VPC Peering">
131
+
132
+
Currently, the VPC Peering feature for {{{ .premium }}} instances is only available upon request. To request this feature, click **?** in the lower-right corner of the [TiDB Cloud console](https://tidbcloud.com) and click **Request Support**. Then, fill in "Apply for VPC Peering for {{{ .premium }}} instance" in the **Description** field and click **Submit**.
133
+
134
+
</div>
135
+
</SimpleTab>
136
+
</CustomContent>
90
137
91
138
### Kafka ACL authorization
92
139
@@ -100,7 +147,7 @@ For example, if your Kafka cluster is in Confluent Cloud, you can see [Resources
100
147
## Step 1. Open the Changefeed page for Apache Kafka
101
148
102
149
1. Log in to the [TiDB Cloud console](https://tidbcloud.com).
103
-
2. Navigate to the cluster overview page of the target TiDB cluster, and then click **Data** > **Changefeed** in the left navigation pane.
150
+
2. Navigate to the overview page of the target TiDB <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent>, and then click **Data** > **Changefeed** in the left navigation pane.
104
151
3. Click **Create Changefeed**, and select **Kafka** as **Destination**.
105
152
106
153
## Step 2. Configure the changefeed target
@@ -140,6 +187,8 @@ The steps vary depending on the connectivity method you select.
140
187
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
141
188
142
189
</div>
190
+
191
+
<CustomContentplan="dedicated">
143
192
<divlabel="Private Service Connect (Google Cloud)">
144
193
145
194
1. In **Connectivity Method**, select **Private Service Connect**.
@@ -158,6 +207,9 @@ The steps vary depending on the connectivity method you select.
158
207
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
159
208
160
209
</div>
210
+
</CustomContent>
211
+
212
+
<CustomContentplan="dedicated">
161
213
<divlabel="Private Link (Azure)">
162
214
163
215
1. In **Connectivity Method**, select **Private Link**.
@@ -176,6 +228,7 @@ The steps vary depending on the connectivity method you select.
176
228
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
177
229
178
230
</div>
231
+
</CustomContent>
179
232
</SimpleTab>
180
233
181
234
## Step 3. Set the changefeed
@@ -219,7 +272,7 @@ The steps vary depending on the connectivity method you select.
219
272
6. If you select **Avro** as your data format, you will see some Avro-specific configurations on the page. You can fill in these configurations as follows:
220
273
221
274
- In the **Decimal** and **Unsigned BigInt** configurations, specify how TiDB Cloud handles the decimal and unsigned bigint data types in Kafka messages.
222
-
- In the **Schema Registry** area, fill in your schema registry endpoint. If you enable **HTTP Authentication**, the fields for user name and password are displayed and automatically filled in with your TiDB cluster endpoint and password.
275
+
- In the **Schema Registry** area, fill in your schema registry endpoint. If you enable **HTTP Authentication**, the fields for user name and password are displayed and automatically filled in with your TiDB <CustomContentplan="dedicated">cluster</CustomContent><CustomContentplan="premium">instance</CustomContent> endpoint and password.
223
276
224
277
7. In the **Topic Distribution** area, select a distribution mode, and then fill in the topic name configurations according to the mode.
225
278
@@ -272,7 +325,7 @@ The steps vary depending on the connectivity method you select.
272
325
273
326
## Step 4. Configure your changefeed specification
274
327
275
-
1. In the **Changefeed Specification** area, specify the number of Replication Capacity Units (RCUs) to be used by the changefeed.
328
+
1. In the **Changefeed Specification** area, specify the number of <CustomContentplan="dedicated">Replication Capacity Units (RCUs)</CustomContent><CustomContentplan="premium">Changefeed Capacity Units (CCUs)</CustomContent> to be used by the changefeed.
276
329
2. In the **Changefeed Name** area, specify a name for the changefeed.
277
330
3. Click **Next** to check the configurations you set and go to the next page.
0 commit comments