Ozone 1.3.0 + Spark 3.5.2 : Unable to create OzoneClient object #7118

kmizumar · 2024-08-26T07:10:20Z

kmizumar
Aug 26, 2024

Hello experts, I need your help.

Our spark driver fails to create an OzoneClient object with the following stack, after showing tons of INFO RetryInvocationHandler: java.lang.IllegalStateException, while invoking $Proxy36.submitRequest over nodeId=om-06,nodeAddress=<REDACTED>:9862 after 499 failover attempts. Trying to failover immediately.

24/08/26 06:24:50 ERROR OMFailoverProxyProviderBase: Failed to connect to OMs: [om-05, om-06, om-30]. Attempted 500 failovers.
24/08/26 06:24:50 ERROR OzoneClientFactory: Couldn't create RpcClient protocol exception: 
java.lang.IllegalStateException
    at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:494)
    at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:119)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:162)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
    at jdk.proxy3/jdk.proxy3.$Proxy36.submitRequest(Unknown Source)
    at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
    at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:239)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:148)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:67)
    at <REDACTED>.tools.Main$.main(Main.scala:74)
    at <REDACTED>.tools.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" java.io.IOException: Couldn't create RpcClient protocol
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:148)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:67)
    at <REDACTED>.tools.Main$.main(Main.scala:74)
    at <REDACTED>.tools.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalStateException
    at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:494)
    at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:119)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:162)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
    at jdk.proxy3/jdk.proxy3.$Proxy36.submitRequest(Unknown Source)
    at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
    at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:239)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
    ... 16 more
24/08/26 06:24:50 INFO ShutdownHookManager: Shutdown hook called
24/08/26 06:24:50 INFO ShutdownHookManager: Deleting directory /tmp/spark-b16b5c9f-bbe2-4256-8609-047787f0bf9c
24/08/26 06:24:50 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system...
24/08/26 06:24:50 INFO MetricsSystemImpl: s3a-file-system metrics system stopped.
24/08/26 06:24:50 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.

We're doing nothing special in creating the object and confirmed our core-site.xml and ozone-site.xml work fine with another simple Ozone Java API client.

    val ozoneClient = OzoneClientFactory.getRpcClient()

I suspect this is a kind of dependency problem.
Could anyone please shed light on this? How can we investigate the root cause of this issue?

$ cat build.sbt 
ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.14"

lazy val scopeVersion = "4.1.0"
lazy val sparkVersion = "3.5.2"
//lazy val ozoneVersion = "1.4.0"
lazy val ozoneVersion = "1.3.0"

lazy val root = (project in file("."))
  .settings(
    name := "cais-tools",
    idePackagePrefix := Some("<REDACTED>.tools"),
    assemblySettings,
    assembly / mainClass := Some("<REDACTED>.tools.Main"),
    libraryDependencies ++= Seq(
      "com.github.scopt" %% "scopt" % scopeVersion,
      "org.apache.spark" %% "spark-core" % sparkVersion % Provided,
      "org.apache.spark" %% "spark-sql" % sparkVersion % Provided,
      "org.apache.spark" %% "spark-kubernetes" % sparkVersion % Provided,
      "org.apache.spark" %% "spark-hadoop-cloud" % sparkVersion % Provided,
      "org.apache.ozone" % "ozone-client" % ozoneVersion,
    )
  )

lazy val assemblySettings = Seq(
  assembly / assemblyJarName := s"${name.value}-${version.value}.jar",
  assembly / assemblyMergeStrategy := {
    case PathList("META-INF", _*) => MergeStrategy.discard
    case x => MergeStrategy.first
  }
)

#6549 looks similar, but I'm not confident.

yazynin · 2024-08-26T18:24:59Z

yazynin
Aug 26, 2024

Why you use ozone 1.3.0? this installation worked before?

0 replies

kmizumar · 2024-08-27T04:05:18Z

kmizumar
Aug 27, 2024
Author

Here are a few reasons why we are using Ozone 1.3.0 as a client:

The target Ozone cluster is running on code based on version 1.3.0.
We have achieved the expected operations without issues when using the Ozone Java API for operations from a Java client.

We encounter this problem while trying to combine Spark to perform large-scale parallel execution of individually executable data processing tasks.

0 replies

siddhantsangwan · 2024-08-27T06:46:48Z

siddhantsangwan
Aug 27, 2024
Collaborator

Not an expert in this area, but I can help you in ensuring the basics are correct.
Have you configured these correctly?

ozone.om.service.ids = omservice
ozone.om.address.omservice.om0 = <om-node1-host>:9862
ozone.om.address.omservice.om1 = <om-node2-host>:9862
ozone.om.address.omservice.om2 = <om-node3-host>:9862
ozone.om.nodes.omservice = om0,om1,om2

<property>
    <name>fs.ofs.impl</name
    <value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value>
</property>

Following from this issue, I think Spark may have to use this jar:

share/ozone/lib/ozone-filesystem-hadoop3-client-1.3.0.jar

Check if you have this jar in the classpath.

1 reply

kmizumar Aug 28, 2024
Author

Check if you have this jar in the classpath.

Interesting, I added the following line to my libraryDependencies

      "org.apache.ozone" % "ozone-filesystem-hadoop3-client" % ozoneVersion,

then the call stack changed as follows:

24/08/28 13:23:08 INFO OzoneClientFactory: Creating OzoneClient with default configuration.
Exception in thread "main" java.lang.NoSuchMethodError: 'org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMResponse org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolPB.submitRequest(com.google.protobuf.RpcController, org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest)'
    at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
    at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:239)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:148)
    at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:67)
    at <REDACTED>.tools.Main$.main(Main.scala:74)
    at <REDACTED>.tools.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
24/08/28 13:23:08 INFO ShutdownHookManager: Shutdown hook called
24/08/28 13:23:08 INFO ShutdownHookManager: Deleting directory /tmp/spark-c77ccce7-0c89-4d7c-8026-279632539459
24/08/28 13:23:08 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system...
24/08/28 13:23:08 INFO MetricsSystemImpl: s3a-file-system metrics system stopped.
24/08/28 13:23:08 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
Stream closed EOF for user-kmizumar/cais-tools-verify-8d869d91992718c4-driver (spark-kubernetes-driver)

Is this a packaging issue?

siddhantsangwan · 2024-09-11T12:06:00Z

siddhantsangwan
Sep 11, 2024
Collaborator

@kmizumar What's the latest on this, were you able to resolve the problem? @GeorgeJahad will you be able to help here?

0 replies

GeorgeJahad · 2024-09-12T04:36:27Z

GeorgeJahad
Sep 12, 2024
Collaborator

Sorry, I'm not working on Ozone right now and don't have time to look. If it is the same problem as the one I was working on, then this is the key comment:
https://issues.apache.org/jira/browse/HDDS-6926?focusedCommentId=17754164&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17754164

In particular:

This is the ozone jar file that should be used with Spark and any other tool that uses the hadoop-client jar:
hadoop-ozone/ozonefs-hadoop3-client/target/ozone-filesystem-hadoop3-client-1.3.0-SNAPSHOT.jar

It should be used with spark instead of this jar file:
ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ozone 1.3.0 + Spark 3.5.2 : Unable to create OzoneClient object #7118

{{title}}

Replies: 5 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Ozone 1.3.0 + Spark 3.5.2 : Unable to create OzoneClient object #7118

kmizumar Aug 26, 2024

Replies: 5 comments · 1 reply

yazynin Aug 26, 2024

kmizumar Aug 27, 2024 Author

siddhantsangwan Aug 27, 2024 Collaborator

kmizumar Aug 28, 2024 Author

siddhantsangwan Sep 11, 2024 Collaborator

GeorgeJahad Sep 12, 2024 Collaborator

kmizumar
Aug 26, 2024

Replies: 5 comments 1 reply

yazynin
Aug 26, 2024

kmizumar
Aug 27, 2024
Author

siddhantsangwan
Aug 27, 2024
Collaborator

kmizumar Aug 28, 2024
Author

siddhantsangwan
Sep 11, 2024
Collaborator

GeorgeJahad
Sep 12, 2024
Collaborator