"java.net.NoRouteToHostException: No route to host (Host unreachable)" when connecting to Hive metastore from Spark

[paulpaul1076]

When I try to connect to hive metastore from spark 3.5.0 operator, I get this exception:

New Text Document.txt

It worked previously in Spark 3.4.0. Do you know why this could be? And what is the easiest way to check connectivity from your spark pods to hive metastore pods?

Thanks!

More info:

Stackable version of hive metastore is 23.7.0
Stackable version of spark is 23.11.0

Could this be the problem?

[adwk67]

I've had a look at the list of breaking changes in the last release and there are none that relate to Hive resource definitions per se. The intention with SDP releases is to issue operators in a version that have been tested and verified to work together, so mixing the operator release versions is not recommended. Without having any more information it looks as though this is network related. Can you provide the hive metastore resource definition?

[adwk67]

Just to be clear: so spark 3.4.0 worked with this:-

Stackable version of hive metastore is 23.7.0
Stackable version of spark is 23.11.0

[paulpaul1076]

@adwk67 no, spark 3.4.0's stackable version was 23.7.0. Then we upgraded to spark 3.5.0 with stackable version 23.11.0.

[paulpaul1076]

@adwk67 okay, we just checked with stackable verison 23.11.0. And the problem remains after upgrading to spark 3.5.0 with stackable version 23.11.0.

What could be the reason for this connectivity issue?

[maltesander]

Hi @paulpaul1076 ,

the NoRouteToHostException basically states that the client (spark) cannot connect to the server (hive). This maybe caused by several things.

• Hive server not working correctly
• Network / Firewall problems
• DNS problems

Could you please check:

• Is the hive server up and running correctly?
• If yes, can you ping it / connect via another client?
• Does it run in the same namespace as spark?
• Are you connecting via IP / Service / FQDN?

and provide the full hive and spark cluster yaml (remove / redact secrets, s3 access/secret keys etc.)

What hive version are you running? What is your kubernetes / openshift cluster version?

[paulpaul1076]

We figured it out, some incorrect IP address was getting resolved from the hostname. That was the issue. We put in the correct hive.metastore.uris ip into the spark job and it started working, thanks! Now we will just figure out why the wrong ip was getting resolved but that should be easy.