WIFI-13597: fix: modified kafka manager to use poll in producer #104
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://telecominfraproject.atlassian.net/browse/WIFI-13597 NOTE: This fix is port of Telecominfraproject/wlan-cloud-ucentralgw#360 Summary of changes: - Modified code in KafkaManager to use poll instead of flush for every messages sent. flush is used only on empty internal notification queue in idle times. Signed-off-by: Ivan Chvets <[email protected]>
acapparelli
approved these changes
Jun 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
During latest experiments with large number of APs, we narrowed down that memory is consumed by Kafka internal queue on GW (producer). And with large number of messages producer cannot keep up with emptying this queue.
One noticeable suspect was identified in
flush()call hereLooks like, flushing on every message slows down producer to 100 messages per second.
The solution was to use
poll()to allow for faster message transmission in peak times.Related Jira: https://telecominfraproject.atlassian.net/browse/WIFI-13597
NOTE: This fix is port of Telecominfraproject/wlan-cloud-ucentralgw#360
Summary of changes: