TEZ-4387: use new doclet API in Java 9+ #296

abstractdog · 2023-06-23T12:43:04Z

wThis PR ensures about JDK 8-17 compile-time compatibility with JDK dependent source folders in tez-javadoc-tools module:

cd tez-tools && mvn clean install -DskipTests

JDK 8-16:

export JAVA_HOME=`/usr/libexec/java_home -v 1.8`
export JAVA_HOME=`/usr/libexec/java_home -v 11`

JDK 17:

export JAVA_HOME=`/usr/libexec/java_home -v 17`

original code was contributed by @LA-Toth (which utilized different modules), now I'm trying to hit the PR until all the precommits problems disappear

ayushtkn · 2024-12-22T16:06:24Z

@abstractdog If we can get this in, we might be able to have compile time support for JDK-17

abstractdog · 2024-12-23T10:11:45Z

@abstractdog If we can get this in, we might be able to have compile time support for JDK-17

yeah, let me take a look
after resolving conflicts, I'll have to check compilation failures in precommit (most probably related to the fact that this patch introduced different profiles for different JDK versions

pushed again to get a new precommit result (old one is not available)

abstractdog · 2024-12-23T11:19:50Z

JDK17 compilation passed successfully locally, added a JDK17 build step to github actions

abstractdog · 2024-12-24T16:32:14Z

the current issue is that some modules are JDK17+ compatible, and are successfully skipped when you build the whole project, but the precommit enters into the folder of the changed submodule and builds it, hence the sub-module skipping won't work
precommit needs to be prepared for this scenario (or some additional maven hack is needed into the submodule's pom.xml)

https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-296/9/artifact/out/patch-mvninstall-tez-tools_tez-javadoc-tools-base.txt

cd /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-296/src/tez-tools/tez-javadoc-tools-base

mvn --batch-mode -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-296/yetus-m2/tez-master-patch-1 -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dspotbugs.skip=true

abstractdog · 2024-12-25T13:43:14Z

latest patch is okay from precommit point of view, I need to check what's happening on JDK17 in terms of javadoc generation:

mvn clean install javadoc:aggregate -DskipTests -pl tez-api -pl tez-mapreduce -pl tez-runtime-library
mvn site -f tez-api

the latter command throws:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on project tez-api: Error generating maven-javadoc-plugin:3.0.1:aggregate report:
[ERROR] Exit code: 1 - error: Class org.apache.hadoop.classification.tools.IncludePublicAnnotationsStandardDoclet is not a valid doclet.
[ERROR]   Note: As of JDK 13, the com.sun.javadoc API is no longer supported.
[ERROR]
[ERROR] Command line was: /Library/Java/JavaVirtualMachines/jdk-17.jdk/Contents/Home/bin/javadoc @options @packages
[ERROR]
[ERROR] Refer to the generated Javadoc files in '/Users/laszlobodor/apache/tez/tez-api/target/site/apidocs' dir.

as the doc generation works on JDK8, maybe we can postpone the solution as a followup ticket, but I'm trying to handle it now hoping that it's an easy one

ayushtkn

one checkstyle warning, rest looks good in the build.
@abstractdog as of now we are just compiling, how do we know if the tests would be fine with higher JDK versions?

...tools/tez-javadoc-tools-base/src/main/java/org/apache/tez/tools/javadoc/util/HtmlWriter.java

abstractdog · 2024-12-25T21:50:13Z

thanks @ayushtkn for the comments, I'll address them

currently I'm testing a different approach which doesn't involve separate modules, only separate folders in the original tez-javadoc-tools module, which looks a bit more lightweight

abstractdog · 2024-12-26T07:42:48Z

compile time compatibility of tez-javadoc-tools is done
created TEZ-4597 as a follow-up to take care of full javadoc generation on JDK17, in the meantime, it works flawlessly on JDK8

abstractdog · 2024-12-26T10:04:21Z

latest precommit is clean (besides the already known TestAnalyzer failure)
@ayushtkn

tez-yetus · 2024-12-26T12:23:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 15s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	3m 51s	Maven dependency ordering for branch
+1 💚	mvninstall	12m 12s	master passed
+1 💚	compile	2m 42s	master passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	compile	2m 28s	master passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+1 💚	checkstyle	1m 42s	master passed
+1 💚	javadoc	1m 49s	master passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	1m 32s	master passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+0 🆗	spotbugs	5m 51s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	6m 51s	master passed
-0 ⚠️	patch	6m 4s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 23s	the patch passed
+1 💚	compile	2m 39s	the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javac	2m 39s	the patch passed
+1 💚	compile	2m 34s	the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+1 💚	javac	2m 34s	the patch passed
+1 💚	checkstyle	0m 21s	tez-tools/tez-javadoc-tools: The patch generated 0 new + 0 unchanged - 23 fixed = 0 total (was 23)
+1 💚	checkstyle	0m 52s	root: The patch generated 0 new + 0 unchanged - 23 fixed = 0 total (was 23)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 4s	The patch has no ill-formed XML file.
+1 💚	javadoc	1m 35s	the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	1m 32s	the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+1 💚	findbugs	6m 34s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 26s	tez-javadoc-tools in the patch passed.
-1 ❌	unit	78m 17s	root in the patch failed.
+1 💚	asflicense	1m 6s	The patch does not generate ASF License warnings.
		136m 39s

Reason	Tests
Failed junit tests	tez.analyzer.TestAnalyzer

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-296/17/artifact/out/Dockerfile
GITHUB PR	#296
JIRA Issue	TEZ-4387
Optional Tests	dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle
uname	Linux 438c1a349e78 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `c77d37e`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-296/17/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-296/17/testReport/
Max. process+thread count	1554 (vs. ulimit of 5500)
modules	C: tez-tools/tez-javadoc-tools . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-296/17/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn

Thanx @abstractdog for the changes, apart from the change in License stuff, changes LGTM

...vadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/doclet/ConfigStandardDoclet.java

tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/Writer.java

tez-tools/tez-javadoc-tools/src/main/java-8-16/org/apache/tez/tools/javadoc/util/XmlWriter.java

abstractdog · 2024-12-30T12:35:21Z

Thanx @abstractdog for the changes, apart from the change in License stuff, changes LGTM

thanks! addressed them in the latest commit

tez-yetus · 2024-12-30T15:10:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	21m 23s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	4m 4s	Maven dependency ordering for branch
+1 💚	mvninstall	12m 44s	master passed
+1 💚	compile	2m 39s	master passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	compile	2m 27s	master passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+1 💚	checkstyle	1m 41s	master passed
+1 💚	javadoc	1m 44s	master passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	1m 27s	master passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+0 🆗	spotbugs	5m 48s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	6m 49s	master passed
-0 ⚠️	patch	6m 0s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 23s	the patch passed
+1 💚	compile	2m 54s	the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javac	2m 54s	the patch passed
+1 💚	compile	2m 38s	the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+1 💚	javac	2m 38s	the patch passed
+1 💚	checkstyle	0m 17s	tez-tools/tez-javadoc-tools: The patch generated 0 new + 0 unchanged - 23 fixed = 0 total (was 23)
+1 💚	checkstyle	0m 52s	root: The patch generated 0 new + 0 unchanged - 23 fixed = 0 total (was 23)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 4s	The patch has no ill-formed XML file.
+1 💚	javadoc	1m 24s	the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04
+1 💚	javadoc	1m 25s	the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
+1 💚	findbugs	6m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 24s	tez-javadoc-tools in the patch passed.
-1 ❌	unit	76m 14s	root in the patch failed.
+1 💚	asflicense	1m 3s	The patch does not generate ASF License warnings.
		155m 54s

Reason	Tests
Failed junit tests	tez.analyzer.TestAnalyzer

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-296/18/artifact/out/Dockerfile
GITHUB PR	#296
JIRA Issue	TEZ-4387
Optional Tests	dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle
uname	Linux ebb4f1b38fb1 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `c77d37e`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu122.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~22.04-ga
unit	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-296/18/artifact/out/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-296/18/testReport/
Max. process+thread count	1593 (vs. ulimit of 5500)
modules	C: tez-tools/tez-javadoc-tools . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-296/18/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog mentioned this pull request Jun 23, 2023

TEZ-4387: use new doclet API in Java 9+ #187

Closed