Skip to content

Releases: confluentinc/parallel-consumer

0.3.0.0

16 Dec 18:40
Compare
Choose a tag to compare

Features

  • Queueing and pressure system now self tuning, performance over default old tuning values (softMaxNumberMessagesBeyondBaseCommitOffset and maxMessagesToQueue) has doubled.
    • These options have been removed from the system.
  • Offset payload encoding back pressure system
    • If the payload begins to take more than a certain threshold amount of the maximum available, no more messages will be brought in for processing, until the space need beings to reduce back below the threshold.
      This is to try to prevent the situation where the payload is too large to fit at all, and must be dropped entirely.
    • See Proper offset encoding back pressure system so that offset payloads can't ever be too large https://github.com/confluentinc/parallel-consumer/issues/47[#47]
    • Messages that have failed to process, will always be allowed to retry, in order to reduce this pressure.

Improvements

  • Default ordering mode is now KEY ordering (was UNORDERED).
    • This is a better default as it's the safest mode yet high performing mode.
      It maintains the partition ordering characteristic that all keys are processed in log order, yet for most use cases will be close to as fast as UNORDERED when the key space is large enough.
  • https://github.com/confluentinc/parallel-consumer/issues/37[Support BitSet encoding lengths longer than Short.MAX_VALUE #37] - adds new serialisation formats that supports wider range of offsets - (32,767 vs 2,147,483,647) for both BitSet and run-length encoding.
  • Commit modes have been renamed to make it clearer that they are periodic, not per message.
  • Minor performance improvement, switching away from concurrent collections.

Fixes

  • Maximum offset payload space increased to correctly not be inversely proportional to assigned partition quantity.
  • Run-length encoding now supports compacted topics, plus other bug fixes as well as fixes to Bitset encoding.