Skip to content

Conversation

gy-deng
Copy link
Contributor

@gy-deng gy-deng commented Aug 26, 2025

Motivation

The main content of this PR is to implement the Pulsar Java community's PIP-393 proposal in the Go client.

Negative Acknowledgement handling in now Pulsar Go client exhibits inefficiencies related to memory usage, timing precision, and batch-level contention.
This effort aims to bring parity with the refinements implemented in the Java client (and CPP client), notably around memory usage and redelivery coordination.

Modifications

Refactor the negativeAcksTracker to solve the above problems.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • Functional testing of negative acks is covered by existing tests (e.g., TestConsumerNack)
  • Extended integration test for evaluating redelivery precision across varying negative acknowledgement bit-widths (i.e., TestNegativeAckPrecisionBitCnt

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (no)
  • The wire protocol: (no)

Documentation

  • Does this pull request introduce a new feature? (no)

Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the negative acknowledgments (nack) tracker to improve memory efficiency and timing precision by implementing changes from Pulsar Java's PIP-393. The implementation replaces a simple map-based approach with a more sophisticated time-ordered structure using bitmaps to reduce memory overhead.

  • Replaced map-based storage with AVL tree ordered by time and roaring bitmaps for entry tracking
  • Added configurable precision bit parameter to control timing granularity and memory usage
  • Updated test infrastructure to support the new nack precision bit parameter

Reviewed Changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
pulsar/negative_acks_tracker.go Core refactoring replacing map with AVL tree and roaring bitmaps for memory-efficient nack tracking
pulsar/consumer.go Added NackPrecisionBit configuration option to ConsumerOptions
pulsar/consumer_impl.go Propagated nack precision bit parameter to partition consumer options
pulsar/consumer_partition.go Updated partition consumer to pass nack precision bit to tracker constructor
pulsar/negative_acks_tracker_test.go Updated test calls to include new nack precision bit parameter
pulsar/consumer_test.go Enhanced existing nack test and added new precision bit validation test
pulsar/consumer_zero_queue_test.go Updated zero queue consumer nack test to handle non-deterministic message ordering
go.mod Added dependencies for roaring bitmaps and AVL tree data structures

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Copy link
Member

@RobertIndie RobertIndie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also take a look at the Copilot comments. Seems that there are some comments that havn't been addressed.

@gy-deng gy-deng requested a review from RobertIndie October 12, 2025 06:25
@gy-deng gy-deng requested a review from RobertIndie October 14, 2025 07:51
@thetumbled thetumbled added this to the v0.17.0 milestone Oct 18, 2025
@thetumbled thetumbled merged commit d99d2c0 into apache:master Oct 18, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants