Bug: Multiple Topics in @KafkaListener Stops Working After Upgrading to 3.3.2

[Andre-Satorres]

Affected Version

• Works in: 3.3.1
• Fails in: 3.3.2

Description

After upgrading from Spring Kafka 3.3.1 to 3.3.2, a @KafkaListener annotated method listening to multiple topics stopped working properly in tests.

I have two topics configured:

@Component
class EventKafkaConsumer(private val eventProcessor: EventProcessor) {

    @KafkaListener(topics = ["\${kafka.consumer.topic-a}", "\${kafka.consumer.topic-b}"])
    suspend fun listen(events: List<Event>) {
        eventProcessor.process(events)
    }
}

Observed Behavior

• I have two tests, each producing messages to a different topic.
• I have a single assert to check if any message was consumed by the application. I also added logs right under the KafkaListener method.
• When running both tests together, the second test always fails. There is no logs for the failed one, but I can see in the kafka the message there. It's just never consumed at all. Not even the deserializer is invoked.
• Running the second test alone, it passes. Proof that my implementation is correct, and it only fails when both run together.
• If I split the single @KafkaListener method into two separate methods, each with its own @KafkaListener, one for the first topic and another for the second topic, both tests pass.
• If I revert back to 3.3.1, keeping the single @KafkaListener for multiple topics, both tests pass.

Expected Behavior

• The same @KafkaListener should continue consuming from both topics in ** 3.3.2**, just like it does in 3.3.1.
• Running tests for both topics should not interfere with each other.

Steps to Reproduce

1. Use Spring Kafka 3.3.2.
2. Create a single @KafkaListener method listening to multiple topics.
3. Write two test cases, each producing messages to one of the topics, asserting that the messge was consumed.
4. Run both tests together: the second test always fails.
5. Run the second test alone: it passes.
6. Change to two separate @KafkaListener methods, one for each topic: both tests pass.
7. Downgrade to 3.3.1: both tests pass with a single @KafkaListener.

Additional Information

• Using "io.projectreactor.kafka:reactor-kafka"

Could this be a bug introduced in Spring Kafka or related to how Kafka consumers are initialized in 3.3.2?

[artembilan]

Any chances to have a simple project from you where we can reproduce the problem?
Thanks.

[akoufa]

@artembilan We have the same issue in 2 different microservices. The only difference is we are only consuming a single topic. After the update from 3.3.1 to 3.3.2 the event consuming stops after the first Kafka poll is done. It seems that the offsets are not commited and we don't receive any new events after that. We had to force downgrade to 3.3.1 and the microservices continued to consume succesfully.

We are using a Kotlin suspend function for the @KafkaListener and we have enabled isAsyncAcks .

[artembilan]

Oh, I see! So, this is different report for the same issue: #3744.
Since that one has been fixed, closing this one as duplicate .

The release with the fix next Tuesday.

Thank you for always upgrading !