ClassCastException: class io.netty.channel.epoll.EpollSocketChannel

[vdesabou]

Using Confluent Platform(CP) 7.0.1, I get the following error with the connector (tested with 0.32.0 and 0.40.0):

[2022-06-23 07:44:54,564] ERROR [zeebe-sink|task-2] WorkerSourceTask{id=zeebe-sink-2} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:206)
java.lang.ExceptionInInitializerError
at io.grpc.netty.NettyChannelBuilder.<clinit>(NettyChannelBuilder.java:83)
at io.camunda.zeebe.client.impl.ZeebeClientImpl.buildChannel(ZeebeClientImpl.java:127)
at io.camunda.zeebe.client.impl.ZeebeClientImpl.<init>(ZeebeClientImpl.java:83)
at io.camunda.zeebe.client.impl.ZeebeClientBuilderImpl.build(ZeebeClientBuilderImpl.java:300)
at io.zeebe.kafka.connect.util.ZeebeClientHelper.buildClient(ZeebeClientHelper.java:62)
at io.zeebe.kafka.connect.source.ZeebeSourceTask.start(ZeebeSourceTask.java:52)
at org.apache.kafka.connect.runtime.WorkerSourceTask.initializeAndStart(WorkerSourceTask.java:229)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:197)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:254)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassCastException: class io.netty.channel.epoll.EpollSocketChannel
at java.base/java.lang.Class.asSubclass(Class.java:3640)
at io.grpc.netty.Utils.epollChannelType(Utils.java:322)
at io.grpc.netty.Utils.<clinit>(Utils.java:114)
... 14 more
[25.480s][info][class,load  ] org.apache.kafka.connect.runtime.WorkerTask$$Lambda$865/0x0000000100955c40 source: org.apache.kafka.connect.runtime.WorkerTask
[2022-06-23 07:44:54,566] ERROR [zeebe-sink|task-0] WorkerSourceTask{id=zeebe-sink-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:206)
java.lang.NoClassDefFoundError: Could not initialize class io.grpc.netty.NettyChannelBuilder
at io.camunda.zeebe.client.impl.ZeebeClientImpl.buildChannel(ZeebeClientImpl.java:127)
at io.camunda.zeebe.client.impl.ZeebeClientImpl.<init>(ZeebeClientImpl.java:83)
at io.camunda.zeebe.client.impl.ZeebeClientBuilderImpl.build(ZeebeClientBuilderImpl.java:300)
at io.zeebe.kafka.connect.util.ZeebeClientHelper.buildClient(ZeebeClientHelper.java:62)
at io.zeebe.kafka.connect.source.ZeebeSourceTask.start(ZeebeSourceTask.java:52)
at org.apache.kafka.connect.runtime.WorkerSourceTask.initializeAndStart(WorkerSourceTask.java:229)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:197)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:254)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

The problem is that the connector depends on netty-transport-classes-epoll-4.1.73.Final.jar but it is not included in the connector (at least on the version shipped on Confluent Hub).

Since the connector does not ship with this dependency, it behaves differently depending on the deployments:

In CP 7.0.1, there is a package io.netty.channe.epoll, but it is from a different jar (so not working):

io.netty.channel.epoll.Epoll source: file:/usr/share/java/confluent-security/connect/netty-all-4.1.68.Final.jar

in CP 7.1.1, it is present there:

io.netty.channel.epoll.Epoll source: file:/usr/share/java/confluent-security/connect/netty-transport-classes-epoll-4.1.73.Final.jar

After adding missing netty-transport-classes-epoll-4.1.73.Final.jar in lib folder, the connector works fine.
Can you ship this dependency directly in the connector ?

[berndruecker]

Thanks for the great description of this issue! I will have a look at it hopefully next week

[berndruecker]

I am not sure why the epoll piece is used here, and not in other scenarios, and thus does not come as a transitive dependency (from the zeebe-java-client).

However, other connectors (like e.g. Redis) did resolve this by excluding netty libraries completely, see jcustenborder/kafka-connect-redis@252c416.

Could you confirm that this is also working in your case @vdesabou? Then I would also go down this route and exclude it when building the connectors - sounds like the better solution to me if Confluent also bundles Netty.