-
Notifications
You must be signed in to change notification settings - Fork 135
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
there is no way to control partition strategy #75
Comments
In addition to the use case mentioned, sending samples of a metric to multiple partitions can introduce out-of-order sample errors. With multiple partitions, there is no guarantee that messages will be consumed in order (this is only guaranteed for a partition, not across multiple partitions). To guarantee ordering of samples, all samples for a metric name+label set should go to the same partition. This is not the behavior of the current code with |
The kafka client used in this project uses kafka.PartitionAny. This strategy doesn't let the user control how the data is spread onto the partitions on the topic. As a result, writing downstream processes that are partition-based is hampered. For example, if I was writing an application that computed gauge from a counter or an alert based on the duration of some condition, the application would need to have a centralized state so that partition consumers could consult the historical state to evaluate the result. However, if the metrics were reliably hashed onto a partition, the consumers could be operated with consumer-specific state (or in-memory state).
The text was updated successfully, but these errors were encountered: