-
Notifications
You must be signed in to change notification settings - Fork 367
[improve] modify the negativeACK structure to reduce memory overhead #1410
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR refactors the negative acknowledgments (nack) tracker to improve memory efficiency and timing precision by implementing changes from Pulsar Java's PIP-393. The implementation replaces a simple map-based approach with a more sophisticated time-ordered structure using bitmaps to reduce memory overhead.
- Replaced map-based storage with AVL tree ordered by time and roaring bitmaps for entry tracking
- Added configurable precision bit parameter to control timing granularity and memory usage
- Updated test infrastructure to support the new nack precision bit parameter
Reviewed Changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
File | Description |
---|---|
pulsar/negative_acks_tracker.go | Core refactoring replacing map with AVL tree and roaring bitmaps for memory-efficient nack tracking |
pulsar/consumer.go | Added NackPrecisionBit configuration option to ConsumerOptions |
pulsar/consumer_impl.go | Propagated nack precision bit parameter to partition consumer options |
pulsar/consumer_partition.go | Updated partition consumer to pass nack precision bit to tracker constructor |
pulsar/negative_acks_tracker_test.go | Updated test calls to include new nack precision bit parameter |
pulsar/consumer_test.go | Enhanced existing nack test and added new precision bit validation test |
pulsar/consumer_zero_queue_test.go | Updated zero queue consumer nack test to handle non-deterministic message ordering |
go.mod | Added dependencies for roaring bitmaps and AVL tree data structures |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please also take a look at the Copilot comments. Seems that there are some comments that havn't been addressed.
Motivation
The main content of this PR is to implement the Pulsar Java community's PIP-393 proposal in the Go client.
Negative Acknowledgement handling in now Pulsar Go client exhibits inefficiencies related to memory usage, timing precision, and batch-level contention.
This effort aims to bring parity with the refinements implemented in the Java client (and CPP client), notably around memory usage and redelivery coordination.
Modifications
Refactor the
negativeAcksTracker
to solve the above problems.Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
If
yes
was chosen, please highlight the changesDocumentation