Shutdown error leads to inconsistent container shutdown

[simonhir]

Hi together,

we discovered a strange behavior with our Spring Cloud Stream applications which in my opinion has it's origin in Spring Kafka.
Our Kafka cluster seams to swap it's certificate (something we are already digging into) which leads to following exception in Spring Kafka.

Log

2025-04-19T22:25:19.919589656+02:00 - ERROR - org.apache.kafka.clients.NetworkClient[KafkaConsumerDestination{consumerDestinationName='dms-test', partitions=0, dlqName='null'}.container-0-C-1]: [Consumer clientId=consumer-dms-2, groupId=dms] Connection to node 1 (cluster-kafka-tls-1-eventbus-k.example.com/10.165.30.126:443) failed authentication due to: SSL handshake failed

2025-04-19T22:25:19.921337573+02:00 - ERROR - org.springframework.kafka.listener.KafkaMessageListenerContainer[KafkaConsumerDestination{consumerDestinationName='dms-test', partitions=0, dlqName='null'}.container-0-C-1]: Authentication/Authorization Exception and no authExceptionRetryInterval set
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)
	... 33 common frames omitted
Wrapped by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271)
	at sun.security.validator.Validator.validate(Validator.java:256)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296)
	... 28 common frames omitted
Wrapped by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alert.createSSLException(Alert.java:130)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138)
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476)
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273)
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260)
	at java.security.AccessController.doPrivileged(AccessController.java:714)
	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:443)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:532)
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:381)
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:301)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:595)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:252)
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.pollForFetches(LegacyKafkaConsumer.java:687)
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.poll(LegacyKafkaConsumer.java:618)
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.poll(LegacyKafkaConsumer.java:591)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:874)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1692)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1667)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1445)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1335)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at java.lang.Thread.run(Thread.java:1583)
Wrapped by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed

2025-04-19T22:25:19.929342143+02:00 - ERROR - org.springframework.kafka.listener.KafkaMessageListenerContainer[KafkaConsumerDestination{consumerDestinationName='dms-test', partitions=0, dlqName='null'}.container-0-C-1]: Fatal consumer exception; stopping container

2025-04-19T22:25:19.968904361+02:00 - INFO - org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder$2[KafkaConsumerDestination{consumerDestinationName='dms-test', partitions=0, dlqName='null'}.container-0-C-1]: dms: partitions revoked: [dms-test-0, dms-test-1, dms-test-2, dms-test-3, dms-test-4]

2025-04-19T22:25:19.970496326+02:00 - ERROR - org.apache.kafka.clients.NetworkClient[KafkaConsumerDestination{consumerDestinationName='dms-test', partitions=0, dlqName='null'}.container-0-C-1]: [Consumer clientId=consumer-dms-2, groupId=dms] Connection to node 0 (cluster-kafka-tls-0-eventbus-k.example.com/10.165.30.126:443) failed authentication due to: SSL handshake failed

2025-04-19T22:25:19.971367174+02:00 - ERROR - org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer[KafkaConsumerDestination{consumerDestinationName='dms-test', partitions=0, dlqName='null'}.container-0-C-1]: [Consumer clientId=consumer-dms-2, groupId=dms] Failed to close coordinator with a timeout(ms)=30000
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)
	... 24 common frames omitted
Wrapped by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271)
	at sun.security.validator.Validator.validate(Validator.java:256)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296)
	... 19 common frames omitted
Wrapped by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alert.createSSLException(Alert.java:130)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138)
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:443)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:532)
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:381)
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:301)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:595)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:322)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1500)
Wrapped by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed

2025-04-19T22:25:20.062508017+02:00 - ERROR - org.apache.kafka.clients.NetworkClient[KafkaConsumerDestination{consumerDestinationName='dms-test', partitions=0, dlqName='null'}.container-0-C-1]: [Consumer clientId=consumer-dms-2, groupId=dms] Connection to node 2 (cluster-kafka-tls-2-eventbus-k.example.com/10.165.30.126:443) failed authentication due to: SSL handshake failed

2025-04-19T22:25:20.06300635+02:00 - ERROR - org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer[KafkaConsumerDestination{consumerDestinationName='dms-test', partitions=0, dlqName='null'}.container-0-C-1]: [Consumer clientId=consumer-dms-2, groupId=dms] Failed to close fetcher with a timeout(ms)=30000
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)
	... 39 common frames omitted
Wrapped by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271)
	at sun.security.validator.Validator.validate(Validator.java:256)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296)
	... 34 common frames omitted
Wrapped by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alert.createSSLException(Alert.java:130)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138)
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:443)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:532)
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:381)
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:301)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:595)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)
	at org.apache.kafka.clients.consumer.internals.Fetcher.maybeCloseFetchSessions(Fetcher.java:131)
	at org.apache.kafka.clients.consumer.internals.Fetcher.closeInternal(Fetcher.java:168)
	at org.apache.kafka.clients.consumer.internals.AbstractFetch.lambda$close$11(AbstractFetch.java:478)
	at org.apache.kafka.common.internals.IdempotentCloser.close(IdempotentCloser.java:161)
	at org.apache.kafka.common.internals.IdempotentCloser.close(IdempotentCloser.java:139)
	at org.apache.kafka.clients.consumer.internals.AbstractFetch.close(AbstractFetch.java:478)
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.lambda$close$3(LegacyKafkaConsumer.java:1148)
	at org.apache.kafka.common.utils.Utils.swallow(Utils.java:1042)
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.close(LegacyKafkaConsumer.java:1148)
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.close(LegacyKafkaConsumer.java:1104)
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.close(LegacyKafkaConsumer.java:1092)
	at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1757)
	at org.springframework.kafka.core.DefaultKafkaConsumerFactory$ExtendedKafkaConsumer.close(DefaultKafkaConsumerFactory$ExtendedKafkaConsumer.java:515)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.wrapUp(KafkaMessageListenerContainer.java:1967)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1393)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at java.lang.Thread.run(Thread.java:1583)
Wrapped by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed

2025-04-19T22:25:20.077263175+02:00 - ERROR - org.springframework.kafka.listener.KafkaMessageListenerContainer[KafkaConsumerDestination{consumerDestinationName='dms-test', partitions=0, dlqName='null'}.container-0-C-1]: Error while stopping the container
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)
	... 24 common frames omitted
Wrapped by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271)
	at sun.security.validator.Validator.validate(Validator.java:256)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296)
	... 19 common frames omitted
Wrapped by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alert.createSSLException(Alert.java:130)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195)
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138)
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:443)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(S

The problem is that this tiggeres (as described for the authExceptionRetryInterval property) a shutdown which fails and puts the application in an inconsistent state, from which it never recovers. Health checks are still up but the application never reconnects to the Kafka cluster and just remains in this state.

An option would be to set the authExceptionRetryInterval property, but that seems like a workaround for propably a bug in the error handling for the shutdown to me.
As I'm not realy femilar how this should correctly be handled by either Spring Kafka or some custom logic in my application it would be great if someone has some input for me.

[artembilan]

That log is just not readable.
Can you, please, reformat it the way that it can be scrolled and gazed easily?

It also would be great if you can share some simple project where we can reproduce the problem.
For now it is not clear what is going on, and mostly because I cannot read that log.

Thanks for understanding.

[simonhir]

@artembilan i replaced the log with a more readable version.

As we our self can't reproduce the behavior and are currently looking for the cause, I'm not really able to provide a reproducible example.
It would help me to know how Spring Kafka should react to such exceptions (thrown by the Kafka client library) like the above one.

[artembilan]

Thanks for editing stack trace!
Now it is very clear what is going on.

So, the logic there in the ListenerConsumer.run() like this:

				catch (AuthenticationException | AuthorizationException ae) {
					if (this.authExceptionRetryInterval == null) {
						ListenerConsumer.this.logger.error(ae,
								"Authentication/Authorization Exception and no authExceptionRetryInterval set");
						this.fatalError = true;
						exitThrowable = ae;
						break;
					}
					else {
						ListenerConsumer.this.logger.error(ae,
								"Authentication/Authorization Exception, retrying in "
										+ this.authExceptionRetryInterval.toMillis() + " ms");
						publishRetryAuthEvent(ae);
						failedAuthRetry = true;
						// We can't pause/resume here, as KafkaConsumer doesn't take pausing
						// into account when committing, hence risk of being flooded with
						// GroupAuthorizationExceptions.
						// see: https://github.com/spring-projects/spring-kafka/pull/1337
						sleepFor(this.authExceptionRetryInterval);
					}
				}

If no authExceptionRetryInterval, we mark this container as this.fatalError = true; and exit from this main poll loop like this in the end of run():

wrapUp(exitThrowable);

That one calls:

	public void stopAbnormally(Runnable callback) {
		this.lifecycleLock.lock();
		try {
			doStop(callback, false);
			publishContainerStoppedEvent();
		}
		finally {
			this.lifecycleLock.unlock();
		}
	}

This means that there is no way for this KafkaMessageListenerContainer to recover itself.

I'm not sure what health check you talk about, but you definitely can react to that ContainerStoppedEvent emitted from the mentioned stopAbnormally().

The exception you are observing is really recoverable when you fix that certificate issue on the Kafka broker side.
And that's exactly what is that authExceptionRetryInterval for:

					else {
						ListenerConsumer.this.logger.error(ae,
								"Authentication/Authorization Exception, retrying in "
										+ this.authExceptionRetryInterval.toMillis() + " ms");
						publishRetryAuthEvent(ae);
						failedAuthRetry = true;
						// We can't pause/resume here, as KafkaConsumer doesn't take pausing
						// into account when committing, hence risk of being flooded with
						// GroupAuthorizationExceptions.
						// see: https://github.com/spring-projects/spring-kafka/pull/1337
						sleepFor(this.authExceptionRetryInterval);
					}