smallrye-pulsar cannot publish message in sample Quarkus project

[vszakd]

Hello,
I have a very simple demo project, made following the Quarkus quickstarts.

The project cannot publish because of this error:

WARN  [org.apa.pul.cli.imp.ProducerImpl] (vert.x-eventloop-thread-5) [quote-requests] [external-out] error while create opSendMsg by batch message container: java.lang.NoClassDefFoundError: com/scurrilous/circe/checksum/Crc32cIntChecksum
	at org.apache.pulsar.common.protocol.Commands.serializeCommandSendWithSize(Commands.java:1650)
	at org.apache.pulsar.common.protocol.Commands.newSend(Commands.java:568)
	at org.apache.pulsar.common.protocol.Commands.newSend(Commands.java:515)
	at org.apache.pulsar.client.impl.ProducerImpl.sendMessage(ProducerImpl.java:854)
	at org.apache.pulsar.client.impl.BatchMessageContainerImpl.createOpSendMsg(BatchMessageContainerImpl.java:255)
	at org.apache.pulsar.client.impl.ProducerImpl.batchMessageAndSend(ProducerImpl.java:2166)
	at org.apache.pulsar.client.impl.ProducerImpl.batchFlushTask(ProducerImpl.java:2149)
	at org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:54)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ClassNotFoundException: com.scurrilous.circe.checksum.Crc32cIntChecksum
	... 18 more

build.gradle

dependencies {
    implementation 'io.quarkus:quarkus-smallrye-reactive-messaging-pulsar'
    implementation enforcedPlatform("${quarkusPlatformGroupId}:${quarkusPlatformArtifactId}:${quarkusPlatformVersion}")
    implementation 'io.quarkus:quarkus-resteasy-reactive-jackson'
    implementation 'io.quarkus:quarkus-arc'
    testImplementation 'io.quarkus:quarkus-junit5'
    testImplementation 'io.rest-assured:rest-assured'
}

implementation

    @Outgoing("external-out")
    public Multi<Double> generate() {
        // Build an infinite stream of random prices
        // It emits a price every second
        return Multi.createFrom().ticks().every(Duration.ofSeconds(5))
                .map(x -> random.nextDouble());
    }

Thanks.

[ozangunalp]

Are you running it on JVM or native binary?

[ozangunalp]

Also a dependency tree would help, because that class is available in the org.apache.bookkeeper:circe-checksum module pulled by the pulsar-client-original dependency.

[vszakd]

I did further tests, it only happens with Gradle. The same identical project with the same dependencies works fine on Maven. I will send a dependency tree soon.

…

On Wed, 7 Feb 2024 at 11:25, Ozan Gunalp ***@***.***> wrote: Also a dependency tree would help, because that class is available in the org.apache.bookkeeper:circe-checksum module pulled by the pulsar-client-original dependency. — Reply to this email directly, view it on GitHub <#2472 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA2RWV53FO42VGTE7AWB53YSNJBDAVCNFSM6AAAAABCXEC4USVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZRG4ZTIOBSHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[vszakd]

JVM

…

On Wed, 7 Feb 2024 at 11:18, Ozan Gunalp ***@***.***> wrote: Are you running it on JVM or native binary? — Reply to this email directly, view it on GitHub <#2472 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA2RWXEBCGI35JUDF5FZATYSNIGTAVCNFSM6AAAAABCXEC4USVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZRG4YTSOJRGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ozangunalp]

Interesting, they are all compile-time dependencies, I don't see how Gradle can resolve it differently.

[vszakd]

Just tried with two brand-new Maven and Gradle projects, created from the Quarkus "New project" wizard in IntelliJ IDEA using Quarkus 3.7.2.

Attached the two dependency trees.

maven-dep.txt

gradle-dep.txt

[ozangunalp]

After much searching, I found out why this happens. It is not obvious but possible to resolve this.

The problem stems from the fact that the dependency artifact resolution in Maven and Gradle are different when it comes to the default packaging of transitive dependencies.

In Maven, the dependency type defaults to "jar". But in Gradle, a dependency is resolved by default to the type marked in the artifact description. For example, the org.apache.bookkeeper:circe-checksum artifacts packaging in pom file is nar, but there are artifacts published with type nar jar, see https://repo1.maven.org/maven2/org/apache/bookkeeper/circe-checksum/4.16.1/.
Therefore in Maven the transitive dependency is resolved to the jar type and included in the Quarkus app as expected. But in Gradle the nar type is resolved and filtered out from the dependencies. The same is true for native-image compilation with GraalVM.

So as to the resolution, I think the easiest way is to adjust the dependency resolution strategy in Gradle. You can either add explicit dependencies to the jar type artifacts:

    implementation("org.apache.bookkeeper:circe-checksum:4.16.1") {
        artifact {
            type = "jar"
        }
    }
    implementation("org.apache.bookkeeper:cpu-affinity:4.16.1") {
        artifact {
            type = "jar"
        }
    }

or add dependency substitutions like

configurations.all {    
    resolutionStrategy.dependencySubstitution {
        substitute(module("org.apache.bookkeeper:circe-checksum"))
            .using(module("org.apache.bookkeeper:circe-checksum:4.16.1"))
            .withoutClassifier()
    }
}

Even in Kotlin DSL I find advanced Gradle configurations are insanely complex so I'd suggest just declaring explicit dependencies.
Maybe you can find a better solution in Gradle dependency resolutions rules

[cescoffier]

@ozangunalp should we explicitly declare that dependency in the Pulsar connector extension (runtime)?

[ozangunalp]

@cescoffier It won't change anything. The transitive dependency is already declared with jar type, default in Maven. Gradle doesn't resolve it as jar, but nar.

[vszakd]

@ozangunalp great find! Thanks.

[vszakd]

Should I open an issue with Quarkus to update the docs?

[cescoffier]

yes, please.

[vszakd]

Issue opened quarkusio/quarkus#40628