Skip to content

Commit

Permalink
[CELEBORN-763] Add --add-opens to bootstrap shell scripts
Browse files Browse the repository at this point in the history 
### What changes were proposed in this pull request?

Add --add-opens to bootstrap shell scripts

### Why are the changes needed?

Additional `--add-opens` is required for Java 17, notes, the `--add-opens` list is copied from Spark and was used for UT, I am not sure each of them is required but at least the UT passed with them.

Details supplied by cfmcgrady

[JEP 403](https://openjdk.java.net/jeps/403) targeted for [JDK 17](https://openjdk.java.net/projects/jdk/17/) will remove `--illegal-access` flag. That will be equivalent to `--illegal-access=deny`.

this means using reflection to invoke protected methods of exported `java.*` APIs will no longer work. For example:

```shell
> /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/jshell
|  欢迎使用 JShell -- 版本 17.0.7
|  要大致了解该版本, 请键入: /help intro

jshell> java.nio.ByteBuffer direct = java.nio.ByteBuffer.allocateDirect(1);
direct ==> java.nio.DirectByteBuffer[pos=0 lim=1 cap=1]

jshell> direct.getClass().getDeclaredConstructor(long.class, int.class).setAccessible(true);
|  异常错误 java.lang.reflect.InaccessibleObjectException：Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module 34c45dca
|        at AccessibleObject.checkCanSetAccessible (AccessibleObject.java:354)
|        at AccessibleObject.checkCanSetAccessible (AccessibleObject.java:297)
|        at Constructor.checkCanSetAccessible (Constructor.java:188)
|        at Constructor.setAccessible (Constructor.java:181)
|        at (#2:1)

jshell>

```

```shell
>  /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/jshell -R --add-opens=java.base/java.nio=ALL-UNNAMED
|  欢迎使用 JShell -- 版本 17.0.7
|  要大致了解该版本, 请键入: /help intro

jshell> java.nio.ByteBuffer direct = java.nio.ByteBuffer.allocateDirect(1);
direct ==> java.nio.DirectByteBuffer[pos=0 lim=1 cap=1]

jshell> direct.getClass().getDeclaredConstructor(long.class, int.class).setAccessible(true);

jshell>
```

### Does this PR introduce _any_ user-facing change?

Yes, for Java 17 support.

### How was this patch tested?

CI and review

Closes #1677 from pan3793/CELEBORN-763.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 5b3f43d)
Signed-off-by: Cheng Pan <[email protected]>
  • Loading branch information
@pan3793
pan3793 committed Jul 5, 2023
1 parent 24e3f51 commit 09c90ee
Show file tree
Hide file tree
Showing 4 changed files with 49 additions and 5 deletions.
4 changes: 2 additions & 2 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -123,11 +123,11 @@
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED</extraJavaTestArgs>
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED</extraJavaTestArgs>

<mavenCentralId>central</mavenCentralId>
<mavenCentralName>Maven Central</mavenCentralName>
Expand Down
15 changes: 14 additions & 1 deletion sbin/restart-worker.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,21 @@ CELEBORN_JAVA_OPTS="$CELEBORN_WORKER_JAVA_OPTS"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS -Xmx$CELEBORN_WORKER_MEMORY"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS -XX:MaxDirectMemorySize=$CELEBORN_WORKER_OFFHEAP_MEMORY"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS -Dio.netty.tryReflectionSetAccessible=true"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --illegal-access=warn"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.lang=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.lang.invoke=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.lang.reflect=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.io=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.net=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.nio=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.util=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.util.concurrent=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.nio.cs=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.security.action=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.util.calendar=ALL-UNNAMED"
export CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS"

if [ "$WORKER_INSTANCE" = "" ]; then
Expand Down
20 changes: 19 additions & 1 deletion sbin/start-master.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,24 @@ if [ "$CELEBORN_MASTER_MEMORY" = "" ]; then
CELEBORN_MASTER_MEMORY="1g"
fi

export CELEBORN_JAVA_OPTS="-Xmx$CELEBORN_MASTER_MEMORY $CELEBORN_MASTER_JAVA_OPTS"
CELEBORN_JAVA_OPTS="$CELEBORN_MASTER_JAVA_OPTS"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS -Xmx$CELEBORN_MASTER_MEMORY"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS -Dio.netty.tryReflectionSetAccessible=true"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --illegal-access=warn"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.lang=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.lang.invoke=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.lang.reflect=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.io=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.net=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.nio=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.util=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.util.concurrent=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.nio.cs=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.security.action=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.util.calendar=ALL-UNNAMED"
export CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS"

exec "${CELEBORN_HOME}/sbin/celeborn-daemon.sh" start org.apache.celeborn.service.deploy.master.Master 1 "$@"
15 changes: 14 additions & 1 deletion sbin/start-worker.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,21 @@ CELEBORN_JAVA_OPTS="$CELEBORN_WORKER_JAVA_OPTS"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS -Xmx$CELEBORN_WORKER_MEMORY"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS -XX:MaxDirectMemorySize=$CELEBORN_WORKER_OFFHEAP_MEMORY"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS -Dio.netty.tryReflectionSetAccessible=true"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --illegal-access=warn"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.lang=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.lang.invoke=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.lang.reflect=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.io=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.net=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.nio=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.util=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.util.concurrent=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.nio.cs=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.security.action=ALL-UNNAMED"
CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS --add-opens=java.base/sun.util.calendar=ALL-UNNAMED"
export CELEBORN_JAVA_OPTS="$CELEBORN_JAVA_OPTS"

if [ "$WORKER_INSTANCE" = "" ]; then
Expand Down

0 comments on commit 09c90ee

Please sign in to comment.