Skip to content

Commit

Permalink
[SPARK-37345][CORE] Add java.security.jgss/sun.security.krb5 to DEF…
Browse files Browse the repository at this point in the history
…AULT_MODULE_OPTIONS

### What changes were proposed in this pull request?

Add `java.security.jgss/sun.security.krb5` to `DEFAULT_MODULE_OPTIONS`.

### Why are the changes needed?

To support Spark runs on Java 17:
```
Exception in thread "main" java.lang.IllegalArgumentException: Can't get Kerberos realm
	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:306)
	at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:352)
	at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:50)
	at org.apache.spark.deploy.SparkHadoopUtil$.instance$lzycompute(SparkHadoopUtil.scala:397)
	at org.apache.spark.deploy.SparkHadoopUtil$.instance(SparkHadoopUtil.scala:397)
	at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:418)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:423)
	at org.apache.spark.executor.YarnCoarseGrainedExecutorBackend$.main(YarnCoarseGrainedExecutorBackend.scala:81)
	at org.apache.spark.executor.YarnCoarseGrainedExecutorBackend.main(YarnCoarseGrainedExecutorBackend.scala)
Caused by: java.lang.IllegalAccessException: class org.apache.hadoop.security.authentication.util.KerberosUtil cannot access class sun.security.krb5.Config (in module java.security.jgss) because module java.security.jgss does not export sun.security.krb5 to unnamed module 3a0baae5
	at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:392)
	at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:674)
	at java.base/java.lang.reflect.Method.invoke(Method.java:560)
	at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:85)
	at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
	... 9 more
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manual test:
```
export JAVA_HOME=/home/yumwang/jdk-17.0.1
export PATH=${JAVA_HOME}/bin:${PATH}

bin/spark-sql  --conf spark.yarn.dist.archives=/home/yumwang/jdk-17.0.1_linux-x64_bin.tar.gz --conf spark.executorEnv.JAVA_HOME=./jdk-17.0.1_linux-x64_bin.tar.gz/jdk-17.0.1/
```

Closes apache#34615 from wangyum/SPARK-37345.

Authored-by: Yuming Wang <[email protected]>
Signed-off-by: Yuming Wang <[email protected]>
  • Loading branch information
wangyum committed Nov 17, 2021
1 parent 1b84a6b commit 02b0545
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ public class JavaModuleOptions {
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
"--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
"--add-opens=java.base/sun.security.action=ALL-UNNAMED",
"--add-opens=java.base/sun.util.calendar=ALL-UNNAMED"};
"--add-opens=java.base/sun.util.calendar=ALL-UNNAMED",
"--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED"};

/**
* Returns the default Java options related to `--add-opens' and
Expand Down

0 comments on commit 02b0545

Please sign in to comment.