Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GraalVM native-image builds based on the Kafka 2.6.0 release #31

Merged
merged 25 commits into from
Nov 12, 2020

Conversation

solsson
Copy link
Owner

@solsson solsson commented Aug 15, 2020

Based on scripts that worked for 2.5.0 builds fail for 2.6.0 CLIs:

Exception in thread "main" com.oracle.svm.core.jdk.UnsupportedFeatureError: Unsupported constructor java.lang.invoke.MemberName.<init>(Class, String, MethodType, byte) is reachable: All methods from java.lang.invoke should have been replaced during image building.
	at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:86)
	at java.lang.invoke.MemberName.<init>(MemberName.java:812)
	at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:2030)
	at java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:1102)
	at scala.runtime.Statics$VM.mkHandle(Statics.java:161)
	at scala.runtime.Statics$VM.<clinit>(Statics.java:155)
	at com.oracle.svm.core.hub.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:350)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:270)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at scala.runtime.Statics.releaseFence(Statics.java:148)
	at scala.collection.immutable.$colon$colon.<init>(List.scala:623)
	at scala.collection.immutable.List.prependedAll(List.scala:153)
	at scala.collection.immutable.List$.from(List.scala:651)
	at scala.collection.immutable.List$.from(List.scala:648)
	at scala.collection.IterableFactory.apply(Factory.scala:103)
	at scala.collection.IterableFactory.apply$(Factory.scala:103)
	at scala.collection.immutable.List$.apply(List.scala:648)
	at scala.collection.SeqFactory$Delegate.apply(Factory.scala:304)
	at scala.collection.SeqFactory$Delegate.apply(Factory.scala:303)
	at scala.collection.SeqFactory$Delegate.apply(Factory.scala:304)
	at kafka.server.ConfigType$.<clinit>(DynamicConfigManager.scala:41)
	at com.oracle.svm.core.hub.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:350)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:270)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at kafka.admin.ConfigCommand$.<clinit>(ConfigCommand.scala:67)
	at com.oracle.svm.core.hub.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:350)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:270)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at kafka.admin.ConfigCommand.main(ConfigCommand.scala)

I guess this is because of the upgrade from Scala 1.12 to Scala 1.13.

The zookeeper build works, but still suffers from the issue reported in #29:

[QuorumPeer[myid=3](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.Learner - FOLLOWING - LEADER ELECTION TOOK - 0 MS
[QuorumPeer[myid=3](plain=0.0.0.0:2181)(secure=disabled)] WARN org.apache.zookeeper.server.ZooKeeperServer - Failed to register with JMX
java.lang.NullPointerException
	at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
	at org.apache.zookeeper.server.quorum.LearnerZooKeeperServer.registerJMX(LearnerZooKeeperServer.java:127)
	at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:73)
	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1253)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:517)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
[QuorumPeer[myid=3](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.Learner - Getting a diff from the leader 0x0
[QuorumPeer[myid=3](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.Learner - Learner received NEWLEADER message
[QuorumPeer[myid=3](plain=0.0.0.0:2181)(secure=disabled)] ERROR org.apache.zookeeper.server.quorum.QuorumPeer - Error writing next dynamic config file to disk: 
[QuorumPeer[myid=3](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.Learner - Learner received UPTODATE message
[QuorumPeer[myid=3](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.CommitProcessor - Configuring CommitProcessor with 8 worker threads.
[QuorumPeer[myid=3](plain=0.0.0.0:2181)(secure=disabled)] WARN org.apache.zookeeper.server.ZooKeeperServer - Failed to register with JMX
java.lang.NullPointerException
	at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:108)
	at org.apache.zookeeper.server.quorum.LearnerZooKeeperServer.registerJMX(LearnerZooKeeperServer.java:105)
	at org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:461)
	at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:572)
	at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:89)
	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1253)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:517)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
[WorkerReceiver[myid=3]] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), FOLLOWING (my state)0 (n.config version)

solsson added 16 commits August 15, 2020 07:10
recommend this setting, and failing at build time is preferrable over runtime
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved type during parsing: io.netty.util.internal.logging.Log4J2Logger. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing type is then reported at run time when it is accessed the first time.
Detailed message:
Trace:
	at parsing io.netty.util.internal.logging.Log4J2LoggerFactory.newInstance(Log4J2LoggerFactory.java:33)
Call path from entry point to io.netty.util.internal.logging.Log4J2LoggerFactory.newInstance(String):
	no path found from entry point to target method
@solsson
Copy link
Owner Author

solsson commented Aug 16, 2020

GraalVM's @Substitution support is a clear USP for native-image builds, notwithstanding its usefulness for Java targeting Kubernetes pods. It's nice to be able to get rid of runtime code you don't need.

The zookeeper JMX NullPointerException from 2.5.0 is fixed now, assuming the substitution has no adverse side effects.

It's still very possible that the native-image zookeeper build can somehow be the culprit in Yolean/kubernetes-kafka#310. The difference now is that there shouldn't be any stacktraces in logs.

The use of MemberName was fixed using --initialize-at-build-time as recommended in scala/bug#1163.

@solsson solsson changed the title WIP native-image builds based on the Kafka 2.6.0 release GraalVM native-image builds based on the Kafka 2.6.0 release Aug 16, 2020
@solsson
Copy link
Owner Author

solsson commented Aug 16, 2020

This result for zookeeper with --initialize-at-build-time is quite interesting:

[ForkJoinPool-2-worker-1] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.quorum.Learner - TCP NoDelay set to: true
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.quorum.Leader - TCP NoDelay set to: true
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.quorum.Leader - zookeeper.leader.maxConcurrentSnapshots = 10
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.quorum.Leader - zookeeper.leader.maxConcurrentSnapshotTimeout = 5
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:host.name=125fd877a5fb
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.version=11.0.7
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.vendor=GraalVM Community
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.home=/opt/graalvm
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.class.path=/opt/graalvm/lib/svm/builder/pointsto.jar:/opt/graalvm/lib/svm/builder/objectfile.jar:/opt/graalvm/lib/svm/builder/llvm-wrapper-shadowed.jar:/opt/graalvm/lib/svm/builder/svm.jar:/opt/graalvm/lib/svm/builder/svm-llvm.jar:/opt/graalvm/lib/svm/builder/javacpp-shadowed.jar:/opt/graalvm/lib/svm/builder/llvm-platform-specific-shadowed.jar
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.io.tmpdir=/tmp
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.compiler=<NA>
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.name=Linux
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.arch=amd64
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.version=5.4.0-42-generic
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.name=nonroot
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.home=/home/nonroot
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.dir=/
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.memory.free=888MB
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.memory.max=11072MB
[ForkJoinPool-2-worker-9] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.memory.total=981MB
Warning: class initialization of class org.apache.zookeeper.server.admin.JettyAdminServer failed with exception java.lang.NoClassDefFoundError: org/eclipse/jetty/server/Connector. This class will be initialized at run time because option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=org.apache.zookeeper.server.admin.JettyAdminServer to explicitly request delayed initialization of this class.
Warning: class initialization of class org.apache.zookeeper.common.SSLContextAndOptions failed with exception java.lang.NoClassDefFoundError: io/netty/handler/ssl/CipherSuiteFilter. This class will be initialized at run time because option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=org.apache.zookeeper.common.SSLContextAndOptions to explicitly request delayed initialization of this class.
WARNING GR-10238: VarHandle for static field is currently not fully supported. Static field private static volatile java.lang.System$Logger jdk.internal.event.EventHelper.securityLogger is not properly marked for Unsafe access!
[/home/nonroot/zookeeper-server-start:26]     (clinit):     240.40 ms,  2.32 GB
[/home/nonroot/zookeeper-server-start:26]   (typeflow):   8,125.27 ms,  2.32 GB
[/home/nonroot/zookeeper-server-start:26]    (objects):   7,795.15 ms,  2.32 GB
[/home/nonroot/zookeeper-server-start:26]   (features):     484.84 ms,  2.32 GB
[/home/nonroot/zookeeper-server-start:26]     analysis:  17,260.02 ms,  2.32 GB
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a direct/mapped ByteBuffer in the image heap. A direct ByteBuffer has a pointer to unmanaged C memory, and C memory from the image generator is not available at image run time.A mapped ByteBuffer references a file descriptor, which is no longer open and mapped at run time.   To see how this object got instantiated use -H:+TraceClassInitialization. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image run time by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace: 
	at parsing org.apache.zookeeper.server.persistence.FilePadding.padFile(FilePadding.java:78)
Call path from entry point to org.apache.zookeeper.server.persistence.FilePadding.padFile(FileChannel): 
	at org.apache.zookeeper.server.persistence.FilePadding.padFile(FilePadding.java:76)
	at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:232)
	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:485)
	at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:572)
	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:122)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:517)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
	at com.oracle.svm.core.code.IsolateEnterStub.PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df(generated:0)

com.oracle.svm.core.util.UserError$UserException: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a direct/mapped ByteBuffer in the image heap. A direct ByteBuffer has a pointer to unmanaged C memory, and C memory from the image generator is not available at image run time.A mapped ByteBuffer references a file descriptor, which is no longer open and mapped at run time.   To see how this object got instantiated use -H:+TraceClassInitialization. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image run time by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace: 
	at parsing org.apache.zookeeper.server.persistence.FilePadding.padFile(FilePadding.java:78)
Call path from entry point to org.apache.zookeeper.server.persistence.FilePadding.padFile(FileChannel): 
	at org.apache.zookeeper.server.persistence.FilePadding.padFile(FilePadding.java:76)
	at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:232)
	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:485)
	at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:572)
	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:122)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:517)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
	at com.oracle.svm.core.code.IsolateEnterStub.PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df(generated:0)

	at com.oracle.svm.core.util.UserError.abort(UserError.java:79)
	at com.oracle.svm.hosted.FallbackFeature.reportAsFallback(FallbackFeature.java:217)
	at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:753)
	at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:538)
	at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:451)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a direct/mapped ByteBuffer in the image heap. A direct ByteBuffer has a pointer to unmanaged C memory, and C memory from the image generator is not available at image run time.A mapped ByteBuffer references a file descriptor, which is no longer open and mapped at run time.   To see how this object got instantiated use -H:+TraceClassInitialization. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image run time by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace: 
	at parsing org.apache.zookeeper.server.persistence.FilePadding.padFile(FilePadding.java:78)
Call path from entry point to org.apache.zookeeper.server.persistence.FilePadding.padFile(FileChannel): 
	at org.apache.zookeeper.server.persistence.FilePadding.padFile(FilePadding.java:76)
	at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:232)
	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:485)
	at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:572)
	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:122)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:517)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
	at com.oracle.svm.core.code.IsolateEnterStub.PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df(generated:0)

	at com.oracle.graal.pointsto.constraints.UnsupportedFeatures.report(UnsupportedFeatures.java:126)
	at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:750)
	... 8 more
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a direct/mapped ByteBuffer in the image heap. A direct ByteBuffer has a pointer to unmanaged C memory, and C memory from the image generator is not available at image run time.A mapped ByteBuffer references a file descriptor, which is no longer open and mapped at run time.   To see how this object got instantiated use -H:+TraceClassInitialization. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image run time by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
	at com.oracle.svm.hosted.image.DisallowedImageHeapObjectFeature.error(DisallowedImageHeapObjectFeature.java:130)
	at com.oracle.svm.core.image.DisallowedImageHeapObjects.check(DisallowedImageHeapObjects.java:85)
	at com.oracle.svm.hosted.image.DisallowedImageHeapObjectFeature.replacer(DisallowedImageHeapObjectFeature.java:122)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.replaceObject(AnalysisUniverse.java:534)
	at com.oracle.svm.hosted.ameta.AnalysisConstantReflectionProvider.replaceObject(AnalysisConstantReflectionProvider.java:217)
	at com.oracle.svm.hosted.ameta.AnalysisConstantReflectionProvider.interceptValue(AnalysisConstantReflectionProvider.java:188)
	at com.oracle.svm.hosted.ameta.AnalysisConstantReflectionProvider.readValue(AnalysisConstantReflectionProvider.java:102)
	at com.oracle.svm.hosted.ameta.AnalysisConstantReflectionProvider.readFieldValue(AnalysisConstantReflectionProvider.java:78)
	at jdk.internal.vm.compiler/org.graalvm.compiler.nodes.util.ConstantFoldUtil$1.readValue(ConstantFoldUtil.java:51)
	at jdk.internal.vm.compiler/org.graalvm.compiler.core.common.spi.JavaConstantFieldProvider.readConstantField(JavaConstantFieldProvider.java:84)
	at com.oracle.svm.hosted.ameta.AnalysisConstantFieldProvider.readConstantField(AnalysisConstantFieldProvider.java:72)
	at jdk.internal.vm.compiler/org.graalvm.compiler.nodes.util.ConstantFoldUtil.tryConstantFold(ConstantFoldUtil.java:47)
	at com.oracle.svm.hosted.phases.ConstantFoldLoadFieldPlugin.tryConstantFold(ConstantFoldLoadFieldPlugin.java:64)
	at com.oracle.svm.hosted.phases.ConstantFoldLoadFieldPlugin.handleLoadStaticField(ConstantFoldLoadFieldPlugin.java:60)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genGetStatic(BytecodeParser.java:4882)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genGetStatic(BytecodeParser.java:4849)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5334)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3423)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:3230)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1088)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:982)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:84)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:214)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
	at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:225)
	at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:352)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:322)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureParsed(MethodTypeFlow.java:311)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.addContext(MethodTypeFlow.java:112)
	at com.oracle.graal.pointsto.DefaultAnalysisPolicy$DefaultVirtualInvokeTypeFlow.onObservedUpdate(DefaultAnalysisPolicy.java:228)
	at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:470)
	at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:542)
	at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:530)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
	... 5 more
Error: Image build request failed with exit status 1
The command '/bin/sh -c native-image   --no-server   --install-exit-handlers   -H:+ReportExceptionStackTraces   --no-fallback   -H:ConfigurationFileDirectories=/home/nonroot/native-config   --initialize-at-build-time   --allow-incomplete-classpath   -Djava.awt.headless=true   -Dkafka.logs.dir=/opt/kafka/bin/../logs   -cp ${classpath}   -H:Name=zookeeper-server-start   org.apache.zookeeper.server.quorum.QuorumPeerMain   /home/nonroot/zookeeper-server-start' returned a non-zero code: 1

Yolean/kubernetes-kafka#310
because when I investigated the timeouts there I found reports
that similar errors had been fixed by cleaning up (or trashing) persisted state.

> Detected a direct/mapped ByteBuffer in the image heap.
> A direct ByteBuffer has a pointer to unmanaged C memory,
> and C memory from the image generator is not available at image run time.
> A mapped ByteBuffer references a file descriptor,
> which is no longer open and mapped at run time.

Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Detected a direct/mapped ByteBuffer in the image heap. A direct ByteBuffer has a pointer to unmanaged C memory, and C memory from the image generator is not available at image run time.A mapped ByteBuffer references a file descriptor, which is no longer open and mapped at run time.   To see how this object got instantiated use -H:+TraceClassInitialization. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image run time by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace:
	at parsing org.apache.zookeeper.server.persistence.FilePadding.padFile(FilePadding.java:78)
Call path from entry point to org.apache.zookeeper.server.persistence.FilePadding.padFile(FileChannel):
	at org.apache.zookeeper.server.persistence.FilePadding.padFile(FilePadding.java:76)
that do include the .class files, I fail to get rid of:

Error: Unsupported features in 2 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved method during parsing: io.netty.buffer.Unpooled.wrappedBuffer(byte[]). To diagnose the issue you can use the --allow-incomplete-classpath option. The missing method is then reported at run time when it is accessed the first time.
Trace:
	at parsing org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:265)
Call path from entry point to org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode():
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:240)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:301)
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved method during parsing: org.eclipse.jetty.servlet.ServletContextHandler.<init>(int). To diagnose the issue you can use the --allow-incomplete-classpath option. The missing method is then reported at run time when it is accessed the first time.
Trace:
	at parsing org.apache.zookeeper.server.admin.JettyAdminServer.<init>(JettyAdminServer.java:90)
Call path from entry point to org.apache.zookeeper.server.admin.JettyAdminServer.<init>(String, int, int, String):
	at org.apache.zookeeper.server.admin.JettyAdminServer.<init>(JettyAdminServer.java:77)
	at org.apache.zookeeper.server.admin.JettyAdminServer.<init>(JettyAdminServer.java:71)
down after we've resolved classpath issues.

The stdout prints of config are gone now, but this print remains:
INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
initialization at runtime, for the same reason as the admincmd build:
"All methods from java.lang.invoke should have been replaced during image building"

This reverts commit e8da8a7.
@solsson
Copy link
Owner Author

solsson commented Aug 16, 2020

At 695f84f the two initial issues were resolved and autobuild passed, pushing:

solsson/kafka:native-kafka-topics@sha256:7707dbef80265f114acb7179b2502ab435444cd5063498cde0b48a8822283338
solsson/kafka:native-kafka-configs@sha256:12253fbe020513df8fc19330cf50dd6ce75a4211c688c2258bc648f9cef80eef
solsson/kafka:native-kafka-consumer-groups@sha256:58a5778cbd1c5c1089dc9f079999c8bc452a1a2f66c277b475ddcd073e5ad89c
solsson/kafka:native-zookeeper-server-start@sha256:7bed6180490cf94ee037da2be4f1e99077d4e81d6107c24a00cd2c912d21c9dc

Thus I merged that to master. After 695f84f we're expanding the classpath to try to, eventually, get rid of --allow-incomplete-classpath and hopefully support TLS and JettyAdminServer,

@solsson solsson merged commit e2ac792 into master Nov 12, 2020
@som-snytt
Copy link

The use of MemberName was fixed using --initialize-at-build-time as recommended in

scala/bug#11634

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants