Apache Kafka 3.2 / Confluent platform 7.2 Kafka Connect service does not see classpath classes when running with Java 17

[MilanMasek]

Greetings,

What is working with Java11 does not work with Java17.

I'm running Kafka 3.2 / Confluent Platform 7.2 with Kafka Connect service in Distributed mode and with installed Ignite kafka-ext module.

When service is started with Java 11, all works fine and classes are loaded and found. But I need for particular reason switch to the Java 17 and in that case Ignite classes can't be found. Jars are, however, added to the java classpath of the Kafka Connect process. Question is - how is this possible and how to fix it? Does this require some extra steps or approach because of Java 17?

The Kafka connect process is started as follows:
cp-kafk+ 1195088 255 5.6 6174016 915684 ? Ssl 09:54 0:40 /etc/alternatives/jre_17/bin/java -Xms256M -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/var/log/kafka -Dlog4j.configuration=file:/etc/kafka/connect-log4j.properties -cp /usr/share/java/kafka/*:/usr/share/java/confluent-common/*:/usr/share/java/kafka-serde-tools/*:/usr/share/java/monitoring-interceptors/*:/usr/share/java/ignite-kafka-connector-ignite-2.14.0/*:/usr/bin/../share/java/kafka/*:/usr/bin/../share/java/confluent-telemetry/* -javaagent:/opt/prometheus/jmx_prometheus_javaagent.jar=8077:/opt/prometheus/kafka_connect.yml org.apache.kafka.connect.cli.ConnectDistributed /etc/kafka/connect-distributed.properties
I have all relevant jars stored in the /usr/share/java/ignite-kafka-connector-ignite-2.14.0 directory and it is added in the classpath as visible above. The plugin kafka-ext is loaded and Added as shown in the connect log. However, later it fails with NoClassDefFoundError. I've also tried to put relevant jar ignite-core (where the class is) at the default path, but no luck either. The connector's worker initialization fails with
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.internal.util.IgniteUtils
The class is in the ignite-core jar, which is placed in the mentioned directory.
$ jar -tf ignite-core-2.14.0.jar | grep -i igniteutils.class org/apache/ignite/internal/util/IgniteUtils.class

Full log:
[2023-05-05 09:55:16,083] INFO [ignite-eps-shared:cache-betradar.2.kafka-topic:dr-ignite-betradar|worker] Instantiated connector ignite-eps-shared:cache-betradar.2.kafka-topic:dr-ignite-betradar with version 7.2.1-ccs of type class org.apache.ignite.stream.kafka.connect.IgniteSourceConnector (org.apache.kafka.connect.runtime.Worker:274) [2023-05-05 09:55:16,083] INFO [ignite-eps-shared:cache-betradar.2.kafka-topic:dr-ignite-betradar|worker] Finished creating connector ignite-eps-shared:cache-betradar.2.kafka-topic:dr-ignite-betradar (org.apache.kafka.connect.runtime.Worker:299) [2023-05-05 09:55:16,085] ERROR [ignite-eps-shared:cache-betradar.2.kafka-topic:dr-ignite-betradar|task-0] Failed to start task ignite-eps-shared:cache-betradar.2.kafka-topic:dr-ignite-betradar-0 (org.apache.kafka.connect.runtime.Worker:549) org.apache.kafka.connect.errors.ConnectException: Instantiation error at org.apache.kafka.connect.runtime.isolation.Plugins.newPlugin(Plugins.java:80) at org.apache.kafka.connect.runtime.isolation.Plugins.newConverter(Plugins.java:286) at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:521) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1421) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getTaskStartingCallable$22(DistributedHerder.java:1434) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.KafkaException: Could not instantiate class org.apache.ignite.stream.kafka.connect.serialization.CacheEventConverter at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:396) at org.apache.kafka.connect.runtime.isolation.Plugins.newPlugin(Plugins.java:78) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:392) ... 9 more Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.internal.util.IgniteUtils at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.<init>(AbstractNodeNameAwareMarshaller.java:36) at org.apache.ignite.marshaller.jdk.JdkMarshaller.<init>(JdkMarshaller.java:86) at org.apache.ignite.marshaller.jdk.JdkMarshaller.<init>(JdkMarshaller.java:80) at org.apache.ignite.marshaller.jdk.JdkMarshaller.<clinit>(JdkMarshaller.java:71) at org.apache.ignite.stream.kafka.connect.serialization.CacheEventDeserializer.<clinit>(CacheEventDeserializer.java:34) at org.apache.ignite.stream.kafka.connect.serialization.CacheEventConverter.<init>(CacheEventConverter.java:33) ... 15 more
I've setup the classpath of the custom classes for Kafka Connect service and expecting classes to be found and loaded.

Reference: https://stackoverflow.com/questions/76182817/apache-kafka-3-2-confluent-platform-7-2-does-not-see-classpath-classes-when-ru