Is it possible to achieve concurrent processing of messages across different partitions on the same topic using smallrye ? #2331
Unanswered
kirankolli-addepar
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Quarkus version = 2.16.5.Final
I am trying to achieve concurrent processing of messages that are in different partitions but same topic. I also want to have explicit control on the commit of each message and do not want to commit until the message has been processed. Since the messages are coming in different partitions based on the key I believe this should be easily achievable.
However what I am finding is that there is serial processing of messages and a message on a different partition is not picked up until the currently processing message is committed. Why is this and how do I resolve this ?
My code:
My properties are set as below :
As you can see from above I have set the partitions property to three and the worker pool size to three as well.
Also when I change the annotation of
@Blocking("worker-pool")
to@Blocking(value = "worker-pool", ordered = false)
what I find is that messages in the same partition are being consumed in a concurrent fashion i.e. skipping the order which is not what I want at all. I would like concurrent processing across partitions but that order within a partition be respected.Also posted to :
https://stackoverflow.com/questions/77311211/quarkus-how-to-achieve-concurrent-processing-of-messages-from-different-kafka
Beta Was this translation helpful? Give feedback.
All reactions