You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In reference to the discussion from this issue #2548 (comment)
The backpressure mechanism in the SmallRye Pulsar connector does not consider memory limit settings. While Pulsar clients can be configured with memoryLimitBytes, which is shared among all consumers and producers used by the client, the backpressure mechanism solely relies on the maxPendingMessages set for the producer. Consequently, it will refrain from sending more messages to the producer queue. However, disregarding the memory limit may result in a MemoryBufferIsFullError if a certain number of messages (lower than maxPendingMessages) with large payload sizes exceed memoryLimitBytes.
In order to address this, backpressure should also verify whether the client can reserve memory for the message to be sent via the producer. This could be achieved by introducing a new processor, such as the MemoryLimitAwareSendProcessor, which would check memory availability before sending the message to the producer.
The text was updated successfully, but these errors were encountered:
In reference to the discussion from this issue #2548 (comment)
The backpressure mechanism in the SmallRye Pulsar connector does not consider memory limit settings. While Pulsar clients can be configured with
memoryLimitBytes
, which is shared among all consumers and producers used by the client, the backpressure mechanism solely relies on themaxPendingMessages
set for the producer. Consequently, it will refrain from sending more messages to the producer queue. However, disregarding the memory limit may result in aMemoryBufferIsFullError
if a certain number of messages (lower thanmaxPendingMessages
) with large payload sizes exceedmemoryLimitBytes
.In order to address this, backpressure should also verify whether the client can reserve memory for the message to be sent via the producer. This could be achieved by introducing a new processor, such as the
MemoryLimitAwareSendProcessor
, which would check memory availability before sending the message to the producer.The text was updated successfully, but these errors were encountered: