Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Kafka] Support native format read/write kafka record #8724

Open
wants to merge 18 commits into
base: dev
Choose a base branch
from
Open

[Feature][Kafka] Support native format read/write kafka record #8724

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
63f1966
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 14, 2025
56aa7a1
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 14, 2025
afe9986
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 14, 2025
21a6d71
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 14, 2025
3e35347
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 15, 2025
6059e42
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 15, 2025
44d2da8
Merge remote-tracking branch 'refs/remotes/upstream/dev' into feature…
CosmosNi Feb 17, 2025
e03d62a
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 17, 2025
959d263
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 17, 2025
91dce70
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 17, 2025
db197b8
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 17, 2025
273f8c5
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 17, 2025
585ad48
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 18, 2025
5b48456
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 18, 2025
bb2c96f
Merge remote-tracking branch 'upstream/dev' into feature-kafka-native
CosmosNi Feb 18, 2025
0e92b1c
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 18, 2025
e4ff9c4
[Feature][Kafka] Support native format read/write kafka record
CosmosNi Feb 20, 2025
84c904f
Merge remote-tracking branch 'origin/feature-kafka-native' into featu…
CosmosNi Feb 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions docs/en/connector-v2/sink/Kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,21 +30,21 @@ They can be downloaded via install-plugin.sh or from the Maven central repositor

## Sink Options

| Name | Type | Required | Default | Description |
|-----------------------|--------|----------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| topic | String | Yes | - | When the table is used as sink, the topic name is the topic to write data to. |
| bootstrap.servers | String | Yes | - | Comma separated list of Kafka brokers. |
| kafka.config | Map | No | - | In addition to the above parameters that must be specified by the `Kafka producer` client, the user can also specify multiple non-mandatory parameters for the `producer` client, covering [all the producer parameters specified in the official Kafka document](https://kafka.apache.org/documentation.html#producerconfigs). |
| semantics | String | No | NON | Semantics that can be chosen EXACTLY_ONCE/AT_LEAST_ONCE/NON, default NON. |
| partition_key_fields | Array | No | - | Configure which fields are used as the key of the kafka message. |
| partition | Int | No | - | We can specify the partition, all messages will be sent to this partition. |
| assign_partitions | Array | No | - | We can decide which partition to send based on the content of the message. The function of this parameter is to distribute information. |
| transaction_prefix | String | No | - | If semantic is specified as EXACTLY_ONCE, the producer will write all messages in a Kafka transaction,kafka distinguishes different transactions by different transactionId. This parameter is prefix of kafka transactionId, make sure different job use different prefix. |
| format | String | No | json | Data format. The default format is json. Optional text format, canal_json, debezium_json, ogg_json and avro.If you use json or text format. The default field separator is ", ". If you customize the delimiter, add the "field_delimiter" option.If you use canal format, please refer to [canal-json](../formats/canal-json.md) for details.If you use debezium format, please refer to [debezium-json](../formats/debezium-json.md) for details. |
| field_delimiter | String | No | , | Customize the field delimiter for data format. |
| common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](../sink-common-options.md) for details |
| protobuf_message_name | String | No | - | Effective when the format is set to protobuf, specifies the Message name |
| protobuf_schema | String | No | - | Effective when the format is set to protobuf, specifies the Schema definition |
| Name | Type | Required | Default | Description |
|-----------------------|--------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| topic | String | Yes | - | When the table is used as sink, the topic name is the topic to write data to. |
| bootstrap.servers | String | Yes | - | Comma separated list of Kafka brokers. |
| kafka.config | Map | No | - | In addition to the above parameters that must be specified by the `Kafka producer` client, the user can also specify multiple non-mandatory parameters for the `producer` client, covering [all the producer parameters specified in the official Kafka document](https://kafka.apache.org/documentation.html#producerconfigs). |
| semantics | String | No | NON | Semantics that can be chosen EXACTLY_ONCE/AT_LEAST_ONCE/NON, default NON. |
| partition_key_fields | Array | No | - | Configure which fields are used as the key of the kafka message. |
| partition | Int | No | - | We can specify the partition, all messages will be sent to this partition. |
| assign_partitions | Array | No | - | We can decide which partition to send based on the content of the message. The function of this parameter is to distribute information. |
| transaction_prefix | String | No | - | If semantic is specified as EXACTLY_ONCE, the producer will write all messages in a Kafka transaction,kafka distinguishes different transactions by different transactionId. This parameter is prefix of kafka transactionId, make sure different job use different prefix. |
| format | String | No | json | Data format. The default format is json. Optional text format, canal_json, debezium_json, ogg_json , avro and native.If you use json or text format. The default field separator is ", ". If you customize the delimiter, add the "field_delimiter" option.If you use canal format, please refer to [canal-json](../formats/canal-json.md) for details.If you use debezium format, please refer to [debezium-json](../formats/debezium-json.md) for details. |
| field_delimiter | String | No | , | Customize the field delimiter for data format. |
| common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](../sink-common-options.md) for details |
| protobuf_message_name | String | No | - | Effective when the format is set to protobuf, specifies the Message name |
| protobuf_schema | String | No | - | Effective when the format is set to protobuf, specifies the Schema definition |


## Parameter Interpretation
Expand Down Expand Up @@ -268,4 +268,4 @@ sink {
"""
}
}
```
```
Loading
Loading