Distributed cluster support

[huishougongming]

How streams support the computation requirement of receiving data from the designated topics in the Kafka cluster, processing it, and then re writing it to Kafka or other distributed databases

[bossenti]

Hi @huishougongming,

Probably I don't get your question right, but the scenario described is possible by using our Kafka adapter allowing you to get data from Kafka as a stream. This stream can be processed within StreamPipes and published to Kafka using our sink or any of our other integrations.

Does this answer your inquiry?

[huishougongming]

Thank you for your clarification
It's like this, there is currently a Kafka cluster with 20 nodes, and each topic has 10 or more partitions. There are three issues

1. How to fill in multiple host names for adapter and sink?
2. When the streams service restarts, it always receives this data from the latest location, including the mqtt adapter, which leads to data loss during the downtime. How can this be resolved?
3. The data received through the adapter will enter the topic automatically created by local Kafka, and then output through sink. What is the logic for creating this topic in local Kafka? For example, how to set the number of partitions?

[bossenti]

How to fill in multiple host names for adapter and sink?

As of now, we don't support this scenario for both. Although, this should be easy to implement, so feel free to file us a feature request.

When the streams service restarts, it always receives this data from the latest location, including the mqtt adapter, which leads to data loss during the downtime. How can this be resolved?

I think you lost me here. How does the MQTT adapter come into play? Aren't we talking about Kafka?

The data received through the adapter will enter the topic automatically created by local Kafka, and then output through sink. What is the logic for creating this topic in local Kafka? For example, how to set the number of partitions?

The whole data handling in StreamPipes happens internally. There is no need to care about topic configuration by the end user. Currently, there is no option to influence the number of partitions used by StreamPipes afaik.