Using Kafka Connect still need to have "CONNECT_OFFSET_STORAGE_TOPIC" #159
-
|
Since this sink connector stores state in ClickHouse, it does not seem, when pairing with Kafka Connect images |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 1 reply
-
|
I did not see any option to avoid this so far https://docs.confluent.io/platform/current/connect/design.html#:~:text=However%2C%20Kafka%20Connect%20does%20require,topics%20used%20for%20this%20storage. And it is the only supported way to directly talk to self-managed Kafka https://clickhouse.com/docs/en/integrations/kafka#self-managed-kafka-connectivity |
Beta Was this translation helpful? Give feedback.
-
|
Please help to see what the reply is on this. thank you!! @yonzhang2012 @yonzhang |
Beta Was this translation helpful? Give feedback.
-
|
@Bowenislandsong It depends on what you mean by storing offsets - there's the normal kafka offset, and we also store the offsets when exactly once is enabled. If you set For the offset topic - to my knowledge, that's part of Kafka Connect (and not something the connector can control) and not removable. |
Beta Was this translation helpful? Give feedback.
-
|
@mzitnik Unless I'm mistaken? |
Beta Was this translation helpful? Give feedback.
@Bowenislandsong It depends on what you mean by storing offsets - there's the normal kafka offset, and we also store the offsets when exactly once is enabled. If you set
<exactlyOnce>false</exactlyOnce>that should disable that part of it.For the offset topic - to my knowledge, that's part of Kafka Connect (and not something the connector can control) and not removable.