[CELEBORN-879] Add `dev/dependencies.sh` for audit dependencies #1797

cfmcgrady · 2023-08-08T04:34:58Z

What changes were proposed in this pull request?

As title

Why are the changes needed?

As title

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GA

codecov · 2023-08-08T05:12:04Z

Codecov Report

Merging #1797 (71a60a6) into main (30d979f) will increase coverage by 0.14%.
Report is 1 commits behind head on main.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1797      +/-   ##
==========================================
+ Coverage   46.45%   46.59%   +0.14%     
==========================================
  Files         163      163              
  Lines       10135    10135              
  Branches      934      934              
==========================================
+ Hits         4707     4721      +14     
+ Misses       5117     5105      -12     
+ Partials      311      309       -2

see 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

AngersZhuuuu · 2023-08-08T06:24:21Z

Add this to github CI?

pan3793 · 2023-08-08T06:45:23Z

@cfmcgrady thanks for working on this.

IMO, the dependencies auditing in Celeborn is a bit complex compared with Spark, since Celeborn is composed of client, master, and worker, all of them have different runtime jars.

But I think we can merge the classpath of master and worker, which simplifies both packaging scripts and dependencies auditing, then the dependencies auditing only needs to take care of client and server dependencies.

dependencise-client-flink-1.14
dependencise-client-flink-...
dependencise-client-spark-2.4
dependencise-client-spark-...
dependencise-server

pan3793 · 2023-08-08T06:47:24Z

And I think we can focus on sbt for this new feature, mvn has some cache issues which are hard to overcome

cfmcgrady · 2023-08-08T09:07:44Z

Add this to github CI?

I intend to introduce a GitHub Action job once sbt and maven dependencies have been aligned. @AngersZhuuuu

…t/SelfDescribing for SBT CI ### What changes were proposed in this pull request? As title ### Why are the changes needed? Recently, I came across an issue in the SBT CI process that can result in failure due to the `NoClassDefFoundError` exception. ``` [error] Uncaught exception when running org.apache.celeborn.common.unsafe.PlatformUtilSuite: java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing [error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing [error] at java.lang.ClassLoader.defineClass1(Native Method) [error] at java.lang.ClassLoader.defineClass(ClassLoader.java:757) [error] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) [error] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) [error] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) [error] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) [error] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) [error] at java.security.AccessController.doPrivileged(Native Method) [error] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) [error] at java.lang.ClassLoader.loadClass(ClassLoader.java:419) [error] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) [error] at java.lang.ClassLoader.loadClass(ClassLoader.java:352) [error] at org.junit.runner.Computer.getSuite(Computer.java:28) [error] at org.junit.runner.Request.classes(Request.java:77) [error] at org.junit.runner.Request.classes(Request.java:92) [error] at com.novocode.junit.JUnitTask.execute(JUnitTask.java:52) [error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) [error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [error] at java.lang.Thread.run(Thread.java:750) [error] Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException: org.hamcrest.SelfDescribing [error] at java.net.URLClassLoader.findClass(URLClassLoader.java:387) [error] at java.lang.ClassLoader.loadClass(ClassLoader.java:419) [error] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) [error] at java.lang.ClassLoader.loadClass(ClassLoader.java:352) [error] at java.lang.ClassLoader.defineClass1(Native Method) [error] at java.lang.ClassLoader.defineClass(ClassLoader.java:757) [error] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) [error] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) [error] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) [error] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) [error] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) [error] at java.security.AccessController.doPrivileged(Native Method) [error] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ``` Upon further investigation, I found that the root cause is SBT's sometimes inability to resolve Maven dependencies cached within GA. ```shell ./build/sbt "show celeborn-common/update" ``` ``` [info] org.hamcrest:hamcrest-core:1.3:default: (MISSING) Artifact(hamcrest-core, jar, jar, None, Vector(), Some(file:/home/runner/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar), Map(), None, false) ``` This PR addresses the random issue by disabling the Maven cache for SBT CI. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GA #1797 pass GA after disabled maven cache. Closes #1818 from cfmcgrady/sbt-ci. Authored-by: Fu Chen <[email protected]> Signed-off-by: Fu Chen <[email protected]>

cfmcgrady · 2023-08-21T08:25:14Z

Audit the dependencies module by module as suggested.
As Maven remains the predominant build tool among users, the dependencies list in this PR is generated using Maven.
cc @pan3793 @waitinfuture

cfmcgrady · 2023-08-22T11:14:02Z

the diff between sbt and maven, - for maven, + for sbt

diff --git a/dev/deps/dependencies-client-flink-1.14 b/dev/deps/dependencies-client-flink-1.14
index 7e8c870f..bbc28ab9 100644
--- a/dev/deps/dependencies-client-flink-1.14
+++ b/dev/deps/dependencies-client-flink-1.14
@@ -25,7 +25,8 @@ hadoop-client-api/3.2.4//hadoop-client-api-3.2.4.jar
 hadoop-client-runtime/3.2.4//hadoop-client-runtime-3.2.4.jar
 htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
 jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
-jsr305/1.3.9//jsr305-1.3.9.jar
+jna/4.2.2//jna-4.2.2.jar
+jsr305/3.0.2//jsr305-3.0.2.jar
 jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
 leveldbjni-all/1.8//leveldbjni-all-1.8.jar
 lz4-java/1.8.0//lz4-java-1.8.0.jar
@@ -48,6 +49,7 @@ netty-codec-xml/4.1.93.Final//netty-codec-xml-4.1.93.Final.jar
 netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
 netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
 netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler-ssl-ocsp/4.1.93.Final//netty-handler-ssl-ocsp-4.1.93.Final.jar
 netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
 netty-resolver-dns-classes-macos/4.1.93.Final//netty-resolver-dns-classes-macos-4.1.93.Final.jar
 netty-resolver-dns-native-macos/4.1.93.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.93.Final-osx-aarch_64.jar
@@ -73,6 +75,6 @@ ratis-thirdparty-misc/1.0.4//ratis-thirdparty-misc-1.0.4.jar
 scala-library/2.12.15//scala-library-2.12.15.jar
 scala-reflect/2.12.15//scala-reflect-2.12.15.jar
 shims/0.9.32//shims-0.9.32.jar
-slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+slf4j-api/2.0.7//slf4j-api-2.0.7.jar
 snakeyaml/1.33//snakeyaml-1.33.jar
 zstd-jni/1.5.2-1//zstd-jni-1.5.2-1.jar
diff --git a/dev/deps/dependencies-client-flink-1.15 b/dev/deps/dependencies-client-flink-1.15
index 7e8c870f..bbc28ab9 100644
--- a/dev/deps/dependencies-client-flink-1.15
+++ b/dev/deps/dependencies-client-flink-1.15
@@ -25,7 +25,8 @@ hadoop-client-api/3.2.4//hadoop-client-api-3.2.4.jar
 hadoop-client-runtime/3.2.4//hadoop-client-runtime-3.2.4.jar
 htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
 jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
-jsr305/1.3.9//jsr305-1.3.9.jar
+jna/4.2.2//jna-4.2.2.jar
+jsr305/3.0.2//jsr305-3.0.2.jar
 jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
 leveldbjni-all/1.8//leveldbjni-all-1.8.jar
 lz4-java/1.8.0//lz4-java-1.8.0.jar
@@ -48,6 +49,7 @@ netty-codec-xml/4.1.93.Final//netty-codec-xml-4.1.93.Final.jar
 netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
 netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
 netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler-ssl-ocsp/4.1.93.Final//netty-handler-ssl-ocsp-4.1.93.Final.jar
 netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
 netty-resolver-dns-classes-macos/4.1.93.Final//netty-resolver-dns-classes-macos-4.1.93.Final.jar
 netty-resolver-dns-native-macos/4.1.93.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.93.Final-osx-aarch_64.jar
@@ -73,6 +75,6 @@ ratis-thirdparty-misc/1.0.4//ratis-thirdparty-misc-1.0.4.jar
 scala-library/2.12.15//scala-library-2.12.15.jar
 scala-reflect/2.12.15//scala-reflect-2.12.15.jar
 shims/0.9.32//shims-0.9.32.jar
-slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+slf4j-api/2.0.7//slf4j-api-2.0.7.jar
 snakeyaml/1.33//snakeyaml-1.33.jar
 zstd-jni/1.5.2-1//zstd-jni-1.5.2-1.jar
diff --git a/dev/deps/dependencies-client-flink-1.17 b/dev/deps/dependencies-client-flink-1.17
index 7e8c870f..bbc28ab9 100644
--- a/dev/deps/dependencies-client-flink-1.17
+++ b/dev/deps/dependencies-client-flink-1.17
@@ -25,7 +25,8 @@ hadoop-client-api/3.2.4//hadoop-client-api-3.2.4.jar
 hadoop-client-runtime/3.2.4//hadoop-client-runtime-3.2.4.jar
 htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
 jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
-jsr305/1.3.9//jsr305-1.3.9.jar
+jna/4.2.2//jna-4.2.2.jar
+jsr305/3.0.2//jsr305-3.0.2.jar
 jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
 leveldbjni-all/1.8//leveldbjni-all-1.8.jar
 lz4-java/1.8.0//lz4-java-1.8.0.jar
@@ -48,6 +49,7 @@ netty-codec-xml/4.1.93.Final//netty-codec-xml-4.1.93.Final.jar
 netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
 netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
 netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler-ssl-ocsp/4.1.93.Final//netty-handler-ssl-ocsp-4.1.93.Final.jar
 netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
 netty-resolver-dns-classes-macos/4.1.93.Final//netty-resolver-dns-classes-macos-4.1.93.Final.jar
 netty-resolver-dns-native-macos/4.1.93.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.93.Final-osx-aarch_64.jar
@@ -73,6 +75,6 @@ ratis-thirdparty-misc/1.0.4//ratis-thirdparty-misc-1.0.4.jar
 scala-library/2.12.15//scala-library-2.12.15.jar
 scala-reflect/2.12.15//scala-reflect-2.12.15.jar
 shims/0.9.32//shims-0.9.32.jar
-slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+slf4j-api/2.0.7//slf4j-api-2.0.7.jar
 snakeyaml/1.33//snakeyaml-1.33.jar
 zstd-jni/1.5.2-1//zstd-jni-1.5.2-1.jar
diff --git a/dev/deps/dependencies-client-spark-2.4 b/dev/deps/dependencies-client-spark-2.4
index 917a0444..2dd3df6c 100644
--- a/dev/deps/dependencies-client-spark-2.4
+++ b/dev/deps/dependencies-client-spark-2.4
@@ -24,11 +24,9 @@ guava/14.0.1//guava-14.0.1.jar
 hadoop-client-api/3.2.4//hadoop-client-api-3.2.4.jar
 hadoop-client-runtime/3.2.4//hadoop-client-runtime-3.2.4.jar
 htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
-jackson-annotations/2.6.7//jackson-annotations-2.6.7.jar
-jackson-core/2.6.7//jackson-core-2.6.7.jar
-jackson-databind/2.6.7.3//jackson-databind-2.6.7.3.jar
 jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
-jsr305/1.3.9//jsr305-1.3.9.jar
+jna/4.2.2//jna-4.2.2.jar
+jsr305/3.0.2//jsr305-3.0.2.jar
 jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
 leveldbjni-all/1.8//leveldbjni-all-1.8.jar
 lz4-java/1.4.0//lz4-java-1.4.0.jar
@@ -51,6 +49,7 @@ netty-codec-xml/4.1.93.Final//netty-codec-xml-4.1.93.Final.jar
 netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
 netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
 netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler-ssl-ocsp/4.1.93.Final//netty-handler-ssl-ocsp-4.1.93.Final.jar
 netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
 netty-resolver-dns-classes-macos/4.1.93.Final//netty-resolver-dns-classes-macos-4.1.93.Final.jar
 netty-resolver-dns-native-macos/4.1.93.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.93.Final-osx-aarch_64.jar
@@ -76,6 +75,6 @@ ratis-thirdparty-misc/1.0.4//ratis-thirdparty-misc-1.0.4.jar
 scala-library/2.11.12//scala-library-2.11.12.jar
 scala-reflect/2.11.12//scala-reflect-2.11.12.jar
 shims/0.9.32//shims-0.9.32.jar
-slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+slf4j-api/2.0.7//slf4j-api-2.0.7.jar
 snakeyaml/1.33//snakeyaml-1.33.jar
 zstd-jni/1.4.4-3//zstd-jni-1.4.4-3.jar
diff --git a/dev/deps/dependencies-client-spark-3.0 b/dev/deps/dependencies-client-spark-3.0
index 5005ba52..5ebe3463 100644
--- a/dev/deps/dependencies-client-spark-3.0
+++ b/dev/deps/dependencies-client-spark-3.0
@@ -25,7 +25,8 @@ hadoop-client-api/3.2.4//hadoop-client-api-3.2.4.jar
 hadoop-client-runtime/3.2.4//hadoop-client-runtime-3.2.4.jar
 htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
 jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
-jsr305/1.3.9//jsr305-1.3.9.jar
+jna/4.2.2//jna-4.2.2.jar
+jsr305/3.0.2//jsr305-3.0.2.jar
 jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
 leveldbjni-all/1.8//leveldbjni-all-1.8.jar
 lz4-java/1.7.1//lz4-java-1.7.1.jar
@@ -48,6 +49,7 @@ netty-codec-xml/4.1.93.Final//netty-codec-xml-4.1.93.Final.jar
 netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
 netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
 netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler-ssl-ocsp/4.1.93.Final//netty-handler-ssl-ocsp-4.1.93.Final.jar
 netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
 netty-resolver-dns-classes-macos/4.1.93.Final//netty-resolver-dns-classes-macos-4.1.93.Final.jar
 netty-resolver-dns-native-macos/4.1.93.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.93.Final-osx-aarch_64.jar
@@ -73,6 +75,6 @@ ratis-thirdparty-misc/1.0.4//ratis-thirdparty-misc-1.0.4.jar
 scala-library/2.12.10//scala-library-2.12.10.jar
 scala-reflect/2.12.10//scala-reflect-2.12.10.jar
 shims/0.9.32//shims-0.9.32.jar
-slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+slf4j-api/2.0.7//slf4j-api-2.0.7.jar
 snakeyaml/1.33//snakeyaml-1.33.jar
 zstd-jni/1.4.4-3//zstd-jni-1.4.4-3.jar
diff --git a/dev/deps/dependencies-client-spark-3.1 b/dev/deps/dependencies-client-spark-3.1
index 6e8fd970..772fd3f2 100644
--- a/dev/deps/dependencies-client-spark-3.1
+++ b/dev/deps/dependencies-client-spark-3.1
@@ -25,7 +25,8 @@ hadoop-client-api/3.2.4//hadoop-client-api-3.2.4.jar
 hadoop-client-runtime/3.2.4//hadoop-client-runtime-3.2.4.jar
 htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
 jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
-jsr305/1.3.9//jsr305-1.3.9.jar
+jna/4.2.2//jna-4.2.2.jar
+jsr305/3.0.2//jsr305-3.0.2.jar
 jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
 leveldbjni-all/1.8//leveldbjni-all-1.8.jar
 lz4-java/1.7.1//lz4-java-1.7.1.jar
@@ -48,6 +49,7 @@ netty-codec-xml/4.1.93.Final//netty-codec-xml-4.1.93.Final.jar
 netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
 netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
 netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler-ssl-ocsp/4.1.93.Final//netty-handler-ssl-ocsp-4.1.93.Final.jar
 netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
 netty-resolver-dns-classes-macos/4.1.93.Final//netty-resolver-dns-classes-macos-4.1.93.Final.jar
 netty-resolver-dns-native-macos/4.1.93.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.93.Final-osx-aarch_64.jar
@@ -73,6 +75,6 @@ ratis-thirdparty-misc/1.0.4//ratis-thirdparty-misc-1.0.4.jar
 scala-library/2.12.10//scala-library-2.12.10.jar
 scala-reflect/2.12.10//scala-reflect-2.12.10.jar
 shims/0.9.32//shims-0.9.32.jar
-slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+slf4j-api/2.0.7//slf4j-api-2.0.7.jar
 snakeyaml/1.33//snakeyaml-1.33.jar
 zstd-jni/1.4.8-1//zstd-jni-1.4.8-1.jar
diff --git a/dev/deps/dependencies-client-spark-3.2 b/dev/deps/dependencies-client-spark-3.2
index 6fbebf79..32e829d7 100644
--- a/dev/deps/dependencies-client-spark-3.2
+++ b/dev/deps/dependencies-client-spark-3.2
@@ -25,7 +25,8 @@ hadoop-client-api/3.2.4//hadoop-client-api-3.2.4.jar
 hadoop-client-runtime/3.2.4//hadoop-client-runtime-3.2.4.jar
 htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
 jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
-jsr305/1.3.9//jsr305-1.3.9.jar
+jna/4.2.2//jna-4.2.2.jar
+jsr305/3.0.2//jsr305-3.0.2.jar
 jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
 leveldbjni-all/1.8//leveldbjni-all-1.8.jar
 lz4-java/1.7.1//lz4-java-1.7.1.jar
@@ -48,6 +49,7 @@ netty-codec-xml/4.1.93.Final//netty-codec-xml-4.1.93.Final.jar
 netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
 netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
 netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler-ssl-ocsp/4.1.93.Final//netty-handler-ssl-ocsp-4.1.93.Final.jar
 netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
 netty-resolver-dns-classes-macos/4.1.93.Final//netty-resolver-dns-classes-macos-4.1.93.Final.jar
 netty-resolver-dns-native-macos/4.1.93.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.93.Final-osx-aarch_64.jar
@@ -73,6 +75,6 @@ ratis-thirdparty-misc/1.0.4//ratis-thirdparty-misc-1.0.4.jar
 scala-library/2.12.15//scala-library-2.12.15.jar
 scala-reflect/2.12.15//scala-reflect-2.12.15.jar
 shims/0.9.32//shims-0.9.32.jar
-slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+slf4j-api/2.0.7//slf4j-api-2.0.7.jar
 snakeyaml/1.33//snakeyaml-1.33.jar
 zstd-jni/1.5.0-4//zstd-jni-1.5.0-4.jar
diff --git a/dev/deps/dependencies-client-spark-3.3 b/dev/deps/dependencies-client-spark-3.3
index 7e8c870f..bbc28ab9 100644
--- a/dev/deps/dependencies-client-spark-3.3
+++ b/dev/deps/dependencies-client-spark-3.3
@@ -25,7 +25,8 @@ hadoop-client-api/3.2.4//hadoop-client-api-3.2.4.jar
 hadoop-client-runtime/3.2.4//hadoop-client-runtime-3.2.4.jar
 htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
 jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
-jsr305/1.3.9//jsr305-1.3.9.jar
+jna/4.2.2//jna-4.2.2.jar
+jsr305/3.0.2//jsr305-3.0.2.jar
 jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
 leveldbjni-all/1.8//leveldbjni-all-1.8.jar
 lz4-java/1.8.0//lz4-java-1.8.0.jar
@@ -48,6 +49,7 @@ netty-codec-xml/4.1.93.Final//netty-codec-xml-4.1.93.Final.jar
 netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
 netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
 netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler-ssl-ocsp/4.1.93.Final//netty-handler-ssl-ocsp-4.1.93.Final.jar
 netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
 netty-resolver-dns-classes-macos/4.1.93.Final//netty-resolver-dns-classes-macos-4.1.93.Final.jar
 netty-resolver-dns-native-macos/4.1.93.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.93.Final-osx-aarch_64.jar
@@ -73,6 +75,6 @@ ratis-thirdparty-misc/1.0.4//ratis-thirdparty-misc-1.0.4.jar
 scala-library/2.12.15//scala-library-2.12.15.jar
 scala-reflect/2.12.15//scala-reflect-2.12.15.jar
 shims/0.9.32//shims-0.9.32.jar
-slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+slf4j-api/2.0.7//slf4j-api-2.0.7.jar
 snakeyaml/1.33//snakeyaml-1.33.jar
 zstd-jni/1.5.2-1//zstd-jni-1.5.2-1.jar
diff --git a/dev/deps/dependencies-client-spark-3.4 b/dev/deps/dependencies-client-spark-3.4
index 8c570f49..591177f5 100644
--- a/dev/deps/dependencies-client-spark-3.4
+++ b/dev/deps/dependencies-client-spark-3.4
@@ -25,7 +25,8 @@ hadoop-client-api/3.2.4//hadoop-client-api-3.2.4.jar
 hadoop-client-runtime/3.2.4//hadoop-client-runtime-3.2.4.jar
 htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
 jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
-jsr305/1.3.9//jsr305-1.3.9.jar
+jna/4.2.2//jna-4.2.2.jar
+jsr305/3.0.2//jsr305-3.0.2.jar
 jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
 leveldbjni-all/1.8//leveldbjni-all-1.8.jar
 lz4-java/1.8.0//lz4-java-1.8.0.jar
@@ -48,6 +49,7 @@ netty-codec-xml/4.1.93.Final//netty-codec-xml-4.1.93.Final.jar
 netty-codec/4.1.93.Final//netty-codec-4.1.93.Final.jar
 netty-common/4.1.93.Final//netty-common-4.1.93.Final.jar
 netty-handler-proxy/4.1.93.Final//netty-handler-proxy-4.1.93.Final.jar
+netty-handler-ssl-ocsp/4.1.93.Final//netty-handler-ssl-ocsp-4.1.93.Final.jar
 netty-handler/4.1.93.Final//netty-handler-4.1.93.Final.jar
 netty-resolver-dns-classes-macos/4.1.93.Final//netty-resolver-dns-classes-macos-4.1.93.Final.jar
 netty-resolver-dns-native-macos/4.1.93.Final/osx-aarch_64/netty-resolver-dns-native-macos-4.1.93.Final-osx-aarch_64.jar
@@ -56,12 +58,8 @@ netty-resolver-dns/4.1.93.Final//netty-resolver-dns-4.1.93.Final.jar
 netty-resolver/4.1.93.Final//netty-resolver-4.1.93.Final.jar
 netty-transport-classes-epoll/4.1.93.Final//netty-transport-classes-epoll-4.1.93.Final.jar
 netty-transport-classes-kqueue/4.1.93.Final//netty-transport-classes-kqueue-4.1.93.Final.jar
-netty-transport-native-epoll/4.1.87.Final/linux-aarch_64/netty-transport-native-epoll-4.1.87.Final-linux-aarch_64.jar
-netty-transport-native-epoll/4.1.87.Final/linux-x86_64/netty-transport-native-epoll-4.1.87.Final-linux-x86_64.jar
 netty-transport-native-epoll/4.1.93.Final/linux-aarch_64/netty-transport-native-epoll-4.1.93.Final-linux-aarch_64.jar
 netty-transport-native-epoll/4.1.93.Final/linux-x86_64/netty-transport-native-epoll-4.1.93.Final-linux-x86_64.jar
-netty-transport-native-kqueue/4.1.87.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.87.Final-osx-aarch_64.jar
-netty-transport-native-kqueue/4.1.87.Final/osx-x86_64/netty-transport-native-kqueue-4.1.87.Final-osx-x86_64.jar
 netty-transport-native-kqueue/4.1.93.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.93.Final-osx-aarch_64.jar
 netty-transport-native-kqueue/4.1.93.Final/osx-x86_64/netty-transport-native-kqueue-4.1.93.Final-osx-x86_64.jar
 netty-transport-native-unix-common/4.1.93.Final//netty-transport-native-unix-common-4.1.93.Final.jar
@@ -77,6 +75,6 @@ ratis-thirdparty-misc/1.0.4//ratis-thirdparty-misc-1.0.4.jar
 scala-library/2.12.17//scala-library-2.12.17.jar
 scala-reflect/2.12.17//scala-reflect-2.12.17.jar
 shims/0.9.32//shims-0.9.32.jar
-slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+slf4j-api/2.0.7//slf4j-api-2.0.7.jar
 snakeyaml/1.33//snakeyaml-1.33.jar
 zstd-jni/1.5.2-5//zstd-jni-1.5.2-5.jar

waitinfuture

LGTM, thanks! Merging to main.

waitinfuture · 2023-08-26T07:56:05Z

dev/dependencies.sh

+    MVN_MODULES="worker,master"
+    ;;
+esac
+


nit: one blank line

### What changes were proposed in this pull request? As title ### Why are the changes needed? This PR ensures dependency alignment between SBT and Maven, based on the audit results implemented in #1797 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GA and Review Closes #1831 from cfmcgrady/align-deps-2. Authored-by: Fu Chen <[email protected]> Signed-off-by: zky.zhoukeyong <[email protected]>

zwangsheng · 2023-08-28T01:37:30Z

Hi @cfmcgrady, would you like to add how to use this script audit dependencies into ${PROJECT_HOME}/CONTRIBUTING.md, since we added how to build configurations docs in this.

cfmcgrady · 2023-08-28T02:09:41Z

Hi @cfmcgrady, would you like to add how to use this script audit dependencies into ${PROJECT_HOME}/CONTRIBUTING.md, since we added how to build configurations docs in this.

Good catch, Thank you for the reminder, let me fix the ci first.

…t/SelfDescribing for SBT CI ### What changes were proposed in this pull request? As title ### Why are the changes needed? Recently, I came across an issue in the SBT CI process that can result in failure due to the `NoClassDefFoundError` exception. ``` [error] Uncaught exception when running org.apache.celeborn.common.unsafe.PlatformUtilSuite: java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing [error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing [error] at java.lang.ClassLoader.defineClass1(Native Method) [error] at java.lang.ClassLoader.defineClass(ClassLoader.java:757) [error] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) [error] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) [error] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) [error] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) [error] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) [error] at java.security.AccessController.doPrivileged(Native Method) [error] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) [error] at java.lang.ClassLoader.loadClass(ClassLoader.java:419) [error] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) [error] at java.lang.ClassLoader.loadClass(ClassLoader.java:352) [error] at org.junit.runner.Computer.getSuite(Computer.java:28) [error] at org.junit.runner.Request.classes(Request.java:77) [error] at org.junit.runner.Request.classes(Request.java:92) [error] at com.novocode.junit.JUnitTask.execute(JUnitTask.java:52) [error] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) [error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [error] at java.lang.Thread.run(Thread.java:750) [error] Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException: org.hamcrest.SelfDescribing [error] at java.net.URLClassLoader.findClass(URLClassLoader.java:387) [error] at java.lang.ClassLoader.loadClass(ClassLoader.java:419) [error] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) [error] at java.lang.ClassLoader.loadClass(ClassLoader.java:352) [error] at java.lang.ClassLoader.defineClass1(Native Method) [error] at java.lang.ClassLoader.defineClass(ClassLoader.java:757) [error] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) [error] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) [error] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) [error] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) [error] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) [error] at java.security.AccessController.doPrivileged(Native Method) [error] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ``` Upon further investigation, I found that the root cause is SBT's sometimes inability to resolve Maven dependencies cached within GA. ```shell ./build/sbt "show celeborn-common/update" ``` ``` [info] org.hamcrest:hamcrest-core:1.3:default: (MISSING) Artifact(hamcrest-core, jar, jar, None, Vector(), Some(file:/home/runner/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar), Map(), None, false) ``` This PR addresses the random issue by disabling the Maven cache for SBT CI. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GA apache/celeborn#1797 pass GA after disabled maven cache. Closes #1818 from cfmcgrady/sbt-ci. Authored-by: Fu Chen <[email protected]> Signed-off-by: Fu Chen <[email protected]>

### What changes were proposed in this pull request? As title ### Why are the changes needed? This PR ensures dependency alignment between SBT and Maven, based on the audit results implemented in apache/celeborn#1797 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GA and Review Closes #1831 from cfmcgrady/align-deps-2. Authored-by: Fu Chen <[email protected]> Signed-off-by: zky.zhoukeyong <[email protected]>

cfmcgrady mentioned this pull request Aug 15, 2023

[CELEBORN-898][INFRA] Fix java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing for SBT CI #1818

Closed

cfmcgrady marked this pull request as draft August 16, 2023 03:21

cfmcgrady added 5 commits August 21, 2023 10:44

audit dependencies

7fc0304

install

158b9cf

nit

c6d2043

includeScope

f28072e

style

0269ade

cfmcgrady force-pushed the audit-deps branch from 618364c to 0269ade Compare August 21, 2023 02:45

cfmcgrady added 12 commits August 21, 2023 13:45

sbt server

a670e7e

mvn build-classpath

73022cc

add dependencies-client-spark-2.4

b3676d8

sbt_build_client_classpath

1dc5f77

dev/deps/dependencies-client-spark-3.0

316dc8a

dev/deps/dependencies-client-spark-3.1

b588554

dev/deps/dependencies-client-spark-3.2

6745dc0

dev/deps/dependencies-client-spark-3.3

ca1038d

dev/deps/dependencies-client-spark-3.4

e38c51a

dev/deps/dependencies-client-flink-1.14

8316fdf

dev/deps/dependencies-client-flink-1.15

a00b676

dev/deps/dependencies-client-flink-1.17

71a60a6

cfmcgrady marked this pull request as ready for review August 21, 2023 08:25

cfmcgrady mentioned this pull request Aug 24, 2023

[CELEBORN-906][BUILD] Aligning dependencies between SBT and Maven #1831

Closed

waitinfuture approved these changes Aug 26, 2023

View reviewed changes

dev/dependencies.sh

MVN_MODULES="worker,master"

;;

esac

Copy link

Contributor

waitinfuture Aug 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: one blank line

waitinfuture closed this in 49b6b10 Aug 26, 2023

cfmcgrady deleted the audit-deps branch August 28, 2023 03:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CELEBORN-879] Add `dev/dependencies.sh` for audit dependencies #1797

[CELEBORN-879] Add `dev/dependencies.sh` for audit dependencies #1797

cfmcgrady commented Aug 8, 2023

codecov bot commented Aug 8, 2023 •

edited

Loading

AngersZhuuuu commented Aug 8, 2023

pan3793 commented Aug 8, 2023 •

edited

Loading

pan3793 commented Aug 8, 2023

cfmcgrady commented Aug 8, 2023

cfmcgrady commented Aug 21, 2023 •

edited

Loading

cfmcgrady commented Aug 22, 2023

waitinfuture left a comment

waitinfuture Aug 26, 2023

zwangsheng commented Aug 28, 2023

cfmcgrady commented Aug 28, 2023

[CELEBORN-879] Add dev/dependencies.sh for audit dependencies #1797

[CELEBORN-879] Add dev/dependencies.sh for audit dependencies #1797

Conversation

cfmcgrady commented Aug 8, 2023

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

codecov bot commented Aug 8, 2023 • edited Loading

Codecov Report

AngersZhuuuu commented Aug 8, 2023

pan3793 commented Aug 8, 2023 • edited Loading

pan3793 commented Aug 8, 2023

cfmcgrady commented Aug 8, 2023

cfmcgrady commented Aug 21, 2023 • edited Loading

cfmcgrady commented Aug 22, 2023

waitinfuture left a comment

Choose a reason for hiding this comment

waitinfuture Aug 26, 2023

Choose a reason for hiding this comment

zwangsheng commented Aug 28, 2023

cfmcgrady commented Aug 28, 2023

[CELEBORN-879] Add `dev/dependencies.sh` for audit dependencies #1797

[CELEBORN-879] Add `dev/dependencies.sh` for audit dependencies #1797

codecov bot commented Aug 8, 2023 •

edited

Loading

pan3793 commented Aug 8, 2023 •

edited

Loading

cfmcgrady commented Aug 21, 2023 •

edited

Loading