support kafka topics & queues?

[jstrachan]

it would be cool to be able to use kafka as a store for messages on a topic then to be able to consume from any topic using queue like semantics.

e.g. if a topic was configured in the gateway to be a kafka topic, the gateway could write the message into a kafka topic. For consumers, it could read from kafka (keeping track in ZK of where it is etc)

Then for queues, we could have a 'kafka queue' microservice which for each logical queue on a topic, we have a single process as the owner, which then consumes messages from the topic and load balances them across consumers on any of the available message gateways.

The kafka queue service would consume a kafka topic and send each message to one of the message-gateway processes that have registered a consumer with it (with retry logic & acks etc).

I wonder if the easiest way to implement the kafka queue service is to just use a regular Artemis Broker and consume messages from the kafka topic and in effect send a KafkaMessage to the queue in the Artemis broker; which is a special kind of message that just refers to the offset and number of bytes in the Kafka topic? then Artemis can do all the usual persistence, load balancing, flow control, acking and so forth?

Then when the gateway gets the KafkaMessage it then transforms it into the actual payload from the Kafka topic? Then we get to have JMS queues on a kafka topic and reuse kafka for the message persistence; with Artemis implementing each queue and doing consumer flow control?