Better Java Module System support for Maven 4

[hendrikebbers]

More and more projects will use the java module system in future. Maven as the most prominent build tool in the Java ecosystem should have a great support for the java module system.

[khmarbaise]

Questions comming into my mind

• How to setup a build to use JPMS
  • maven-jmod-plugin (requires some support)
  • maven-jlink-plugin
• Testing with JPMS
• Multi Module Build?

[khmarbaise]

Yes, but if you have JPMS via jmod etc. you could create your own JRE via JLink ... that would the full path to a self contained delivery..

[cowwoc]

On a somewhat related note, there is a lot of confusion about how to deploy a multirelease JAR file. The compiler plug-in makes it easy, but the documentation is far from obvious. In fact, I don't think it even includes a usage example for the "multiReleaseOutput" option.

Is there a separate discussion thread for this?

[desruisseaux]

No separated thread as far as I know, but the multi-sources proposal in this thread covers the case of multi-releases. That proposal aims to make all the following use-cases easier with one single solution:

• Module Source Hierarchy
• Multi-releases
• Sources that are not main or test (e.g. "snippet" since Java 18)
• Other use cases currently addressed by build-helper-maven-plugin.

Note that if this proposal is accepted, the way to do multi-releases with the new compiler plugin would be different than what the current plugin does. It should be easier and more obvious. However, the current way may not be supported anymore. So maybe it is not a bad thing if its documentation is not yet too accessible.

[hendrikebbers]

Do you think that it makes sense to extract that to a new topic?

[desruisseaux]

I think so, because it addresses more topics than only JPMS as listed above, but also because it is not a blocker issue for JPMS. If not done, we cannot do "Multi Source Hierarchy" (i.e. many Java modules in a single Maven module) but we can still use JPMS with one Java module per Maven module.

[jbee]

As I said in out chat, for me it is hard to imagine the middle ground where the Maven build is based on the Java's module-info.java files while also merging that information with the pom.xml information for properties that aren't sufficiently defined by the module-info.java.

From a user's perspective I would want to mainly modularize my project using Java modules and use a single maven module for adding in the other aspects about the project. Assuming that not everyone will be wanting such a model I would think that if there are multiple maven sub-modules in such a project the dependencies between the (sub)modules should not be required to be stated including the module structure as this does follow from the Java module declaration. TLDR; don't make me redefine information that is implied by the module-info.java.

On top I would want to be able to opt-out of Maven features or maybe have a different "mode" when incorporating the Java modules so that the compilation only ever can be the expected sequence of javac and jar tool calls. Something like this might even be a precondition to be able to merge the two worlds without making it super complicated. TLDR; when merging opt for the simplest model possible even if that removes support for "advanced" Maven features.

I also want to mention that doing a major change like this surely needs to touch the code in many central places. This is by nature the opposite of stability for those that just want to keep using maven "as is". So the way such a change is made must make sense with stability in mind. For example, if this is first added to a new major version the issue is little as a major version change is usually done very conscious and after looking at the pros and cons. It will be hard balancing the different users (without Java modules vs with Java modules) against each other.

[khmarbaise]

That would a full change in the idea of Maven and just say the modules (JPMS) is the basic part.. which is valid from a perspective but my guess is the minority of the project will use the JPMS (but that is a different story). If you like to go full via JPMS modules I would suggest to take deep look into https://github.com/sormuras/bach

[desruisseaux]

Actually the work described in the "Changes in Maven compiler plugin" section below is already taking that direction. In particular, the following:

I would want to mainly modularize my project using Java modules and use a single maven module for adding in the other aspects about the project.

would be enabled by the "Change for Maven 4 core not yet done or discussed" section (without forcing developers to use it if they don't want). I do not know yet if the latter proposal will be accepted, but the work on the compiler plugin already anticipates on that. In this model, the Maven POM would need only a <dependencyManagement> section, the <dependencies> section may not be needed anymore (except maybe for tests which have additional dependencies compared to module-info declaration).

[rmannibucau]

this is something to discuss on the list cause the current work is mainly to drop the heuristic about using module path or class path but doing that is exactly the same in practise due to JPMS capabilities and JPMS vs packaging design so ultimately we would just move the issue we had to another section without enabling any new use case so sounds dangerous, better to stick to what is known deterministic and working and not think we can be more clever than others IMHO.

[jbee]

Thinking a bit on how Maven and Java Module information can merge more smooth might be to look at a set of annotations defined by Maven to be put in the module-info.java to add in information that is missing on elements that otherwise would need some "stub" with reference in the pom.xml just to link the two together again.

Christian and me have been looking at annotation like this in context of bach and there we came to the conclusion that just using programmatic setup is easier and more powerful at the same time so it is not worth it in the programmatic approach bach has. But for Maven with declarative basis of the pom.xml it might suit better to also have annotations as a declarative complement.

[ascheman]

Regarding (build time) dependency management I wonder if it shouldn't be possible to provision ClassPaths from the module-info.java through a Maven extension like it was done by Polyglot?

If other coordinates for publication of the resulting artifact(s) like GAV, developer, licence, repository and other information could be moved to a module-info.java file (or similar file) as well, the POM could be completely removed (or drastically reduced).

For multi module builds I still would like to define common dependency meta data only once, like it is done in dependencyManagement in a parent POM or via BOM import.

[cstamas]

Am just putting this here, the ongoing Maven4 work: https://github.com/Geomatys/maven-compiler-plugin/wiki
Ping @desruisseaux

[desruisseaux]

Thanks @cstamas for pinging me. Work for JPMS are ongoing in both Maven 4 core and in the Maven compiler plugin. The implementation of some of the proposals described below are already advanced, but it is still possible to modify the approach according feedbacks.

Change already done in Maven 4 core API

Problem

Placing a JAR on the class-path or on the module-path makes a difference: services are not detected by java.util.ServiceLoader in the same way, the behaviour of java.lang.reflect is different because of encapsulation, the way to load a JAR file at run-time is very different (ClassLoader versus ModuleLayer), etc. Unfortunately, Maven 3 tries to automatically detect where to place a JAR file using heuristic rules, with no easy way for the developers to control that choice. The choice is not always correct. Even if it was correct in a majority of cases, developers sometime have good reasons to want a less usual choice.

Solution in Maven 4-alpha

Maven 4 tries to resolve that problem by giving to users a way to control where to place a JAR file. It uses the <type> element, which already exists in Maven 3 but got new values in Maven 4. For example:

<dependency>
  <groupId>org.foo</groupId>
  <artifactId>bar</artifactId>
  <type>modular-jar</type>
</dependency>

Some values for <type> are:

• jar: Maven chooses automatically whether to place the dependency on the class-path or on the module-path. Whether it should stay the default type for compatibility with Maven 3, and whether the heuristic rules should be exactly the same as Maven 3 or a modified version of them, have not been decided.
• module-jar: unconditionally place the dependency on the module-path. No heuristic, even if the JAR has a module-info.
• classpath-jar: unconditionally place the dependency on the class-path. No heuristic, even if the JAR has no module-info.

This approach is opportunistically expanded to other kinds of paths:

• processor: replace the <annotationProcessorPaths> element in Maven compiler plugin configuration.
• modular-processor: like processor but unconditionally place the JAR on the processor module path.
• classpath-processor: like processor but unconditionally place the JAR on the processor class path.
• doclet: replace the <docletArtifact> element in Maven javadoc plugin configuration.
• etc. for taglet path, agent path, or any other kinds of path. They would all be managed as ordinary dependencies instead of plugin configuration, with only a different type.

Note however that while those types have already been introduced in Maven 4, no plugin take them in account yet. The first plugin to be updated is the compiler plugin (discussed below).

Change for Maven 4 core not yet done or discussed

The way to specify the directory of source code in Maven 3 is too minimalist for the evolution of the Java language in the last decade:

<build>
  <sourceDirectory>src/main/java</sourceDirectory>
</build>

If for any reason the code is spreads between two or more directories, developers have to use build-helper-maven-plugin. Doing multi-releases is difficult, and JPMS with Module Source Hierarchy is practically impossible. I would like a new way to specify the source directories, more like below:

<sources>
  <source module="org.foo.bar" release="11">src/org.foo.bar/java</source>
  <source module="org.foo.biz" release="11">src/org.foo.biz/java</source>
  <source module="org.foo.biz" release="17">src/org.foo.biz/java17</source>
<sources>

Notes:

• This example has two JPMS modules, org.foo.bar and org.foo.biz. For a non-JPMS project, the module attribute would be absent.
• The org.foo.biz module uses multi-releases with two releases: 11 and 17. If the release attribute is omitted, the default would be the default of the javac tool (not anymore a Maven default hard-coded in some plugins).
• For a JPMS project, the <dependencies> section become effectively more like <dependencyManagement>: which dependencies are really used depend on the module-info content. The <dependencyManagement> section may become obsolete.

I have not yet done a formal proposal for this change on the Maven developer mailing list, but hope to do so after a working version of the new compiler plugin become available. Note that while the above structure does not exist yet in the Maven model, the new Maven compiler plugin is already ready for it. It already uses a (source directory, module name, Java release) structure internally.

Change in Maven compiler plugin

The changes in Maven 4 core were relatively minor. But the change in Maven compiler plugin is a major rewrite. All plexus dependencies are removed. The Plexus compiler is replaced by javax.tools.JavaCompiler interface. The Plexus directory scanner is replaced by java.nio, etc. The ways to manage source directories and dependencies are as described above. The incremental build has been opportunistically be rewritten. Default values are the defaults of the standard Java tools, not anymore the Maven opinion of what defaults should be. As of today, the main code is close to complete, but the tests are not yet executable. In the next few months, I will fix the tests, debug, then start to play with the compiler plugin on a branch of the Apache SIS project, which is already structured as a Modular Source Hierarchy and is currently built with Gradle. The work is a little bit slow because I work on it only during the weekends, and the next months will be busy with "Community over code" conference in Europe and another meeting.

After we get a workable compiler plugin, the next plugins to update will be javadoc, tests and JAR. But I think they will be easier to upgrade. Except Javadoc, for which I would propose to merge with the compiler plugin, but I keep this debate for another time.

[hendrikebbers]

All those ideas sound super cool! I'm currently migrating a big (gradle) project (https://github.com/hashgraph/hedera-services) to java modules. We still have a lot of problems and I assume not all of them would be solved by your ideas but the additional type and especially the definition of multiple source sets looks really promising.

[jbee]

I would like a new way to specify the source directories, more like below:

<sources>
 <source module="org.foo.bar" release="11">src/org.foo.bar/java</source>
 <source module="org.foo.biz" release="11">src/org.foo.biz/java</source>
 <source module="org.foo.biz" release="17">src/org.foo.biz/java17</source>
<sources>

Indeed! You also might want to think in code spaces as "containers" for these to know enough on how they layer semantically. E.g. "main" and "test".

[desruisseaux]

Agree, I also think that an attribute for code spaces or containers may be needed, as opposed to XML elements like <sources> versus <testSources>. One reason is that Maven has been used to think for years as if only two containers existed: main and test. But since Java 18, we have a third container: snippet, and we have no guarantee that no other container will appear in the future.

[jjohannes]

This is a large topic. I apologise in advanced for another long comment. It may be a good idea to put these things in some kind of specification document so that different aspects can be discussed separately and do not get mixed up.

My thoughts on the topic:

I would consider taking the Java Module System into central consideration when thinking about the "next step" of improving Java build tools (both Maven and Gradle). It has the potential to make the tools better even for projects that do not use the Module System. In other words, when we discuss the future of the build tools, we should stop thinking about the "Projects using the Module System" being the "special case". But rather seeing "Projects using the Module System" as the default case (we want to get to). That will help to reach the goal that it actually becomes the default in the future, which will then allow the whole ecosystem to profit and use all the JDK tooling (jlink etc) seamlessly.

Then, one can re-evaluate the convention-over-configuration status for "standard" Java projects. The Module System allows us to move responsibility from the build tools into Java itself. Which in turn allows for more conventions and the build tools to move into the background. Which tackles one of the major problems they have right now (IMO): There is still too much configuration needed in cases where projects are (more or less) "standard Java projects".

I emphasize with what @ascheman wrote above. In my mind a perfect build system for the Module System can move into the background. Which means there are no additional files in the single projects (of a multi-project) needed as everything can be a convention or a central configuration. I did a proof-of-concept for that with Gradle: https://github.com/jjohannes/java-module-system

Here is a slide from my presentation on the topic, which attempts to split up responsibilities between the Module System and the Build Tool. That is, all that can be expressed in the Module System about Modules and their relationship is done there. Everything else that is needed to retrieve third party modules and build the software is done by the build tool:

These are the major functionalities we have in Gradle / Gradle plugins that would be needed in a corresponding Maven approach:

• Deriving the dependencies from module-info This is done by the java-module-dependencie Gradle plugin. The plugin contains a properties file that maps Module Names to GAs (based on what is in Maven Central today). The user can extend this list by providing their own properties file. I think the same thing could be implement for Maven with an Extension (what @ascheman already suggested). The versions of the modules are managed in a BOM (called platform project in Gradle). Similar can be done in a Maven setup today by defining a BOM with <dependencyManagement>.
  Feels to me like no new "Maven core" features would be needed to write such an Extension.

• Deal with variants of Modules Each published Java Module comes in variations. E.g. there are several Jars containing a Module with the name javafx.graphics to choose from. In the end however, there can only be one javafx.graphics ending up on the Module Path as the Module System cannot deal with multiple versions/variants of the same module. Hence sorting this out is a responsibility of the build tool. The first variation dimension is the version. This is solved. Both Maven and Gradle detect version conflicts and resolve them and allow only one version of a certain GA to end up on the Module Path (or Classpath). But there are other dimensions of variation. E.g. javafx.graphics has different implementations for different Operating Systems. E.g. jakarta.mail can be only an API (jakarta.mail:jakarta.mail-api) or an API+Implementation (org.eclipse.angus:jakarta.mail).
  Gradle has a variant aware dependency management system – based on the concepts of Attributes and Capabilities – that allows us to deal with such multi-dimensional variation and express things like "choose the Linux variant for all Modules" or "prefer the Implementation of Jakarta Mail over only the API" centrally. Similar to how we define versions in a central place.
  I think Maven has functionality here as well through using profiles. But I lack the experience here to say what it is missing (or not missing). But something is needed to do "variant selection beyond versions" in the setup I sketch here. Today, shortcomings in this area (e.g. wrong Jar on Module Path) are solved by using <excludes> on individual dependencies which this setup (that derives dependencies from module-info) no longer allows.
  This is also a topic that cross-cuts build tools as it influences the metadata published with Modules. Gradle went ahead here with it's own format to allow Modules to express things in their Metadata like: "This Jar is for Linux" or "org.eclipse.angus:jakarta.mail is another implementation of jakarta.mail:jakarta.mail-api (and only one is allows on the Module Path)". For the good of the whole ecosystem, it would be great if such things could be expressed in a common format. Maybe by extending the Consumer POM format in some way. (Note for reference: I wrote this Maven plugin to publish some of this information in the Gradle format with Maven.)

• Deal with wrong/incomplete Metadata and Jars The setup I envision works well if "the world is perfect". Which means all the third party Modules used are complete in their description – both Jar and Metadata. But that's not the case in reality. So the idea is to make the world perfect by patching what is needed in third party Jars. Like an adapter layer or filter you put around your "own world" through which you only allow perfect Modules to enter.
  In Jars today, often the "module-info.class" is still missing completely. In Gradle we have extra-java-module-info to (again centrally) define this missing information. In Maven, Moditec supports this.
  In Metadata, a lot of information is missing today. Like the information for which operating system a certain classified Jar is (see example above). Or just dependencies that are wrong or wrongly scoped. Gradle has the concept of Component Metadata Rules that allows you to express how the metadata should be adjusted/enriched before it is used to build and resolve the dependency graph. I wonder if something similar would be interesting for Maven (?) I think today, such problems are solved by using <exclude>. Which is probably usable here as well as long as you use it inside <dependencyManagement> of your "central BOM" only. One would have to check how far this goes and which use cases cannot be solved by that. I think the discussion on include for dependencyManagement is also about this.

Note that the topics of defining versions and variant selection centrally and centrally patching metadata to make it "perfect for my world" are the same for non-Module (classspath) projects. But thinking about this with the Module System in mind makes it clearer why it is useful to think of these as aspects of dependency management as something that should be dealt with "somewhere central" (and not in individual project POMs).

I don't feel strongly about "how" these things work or will work in Maven. I just wanted to offer these thoughts as something to create a vision for a "perfect Java Module System project setup" from – with as much "convention-over-configuration" as possible. If this sounds useful, I think an interesting experiment would be to prototype the Maven Extension sketched and check how my Gradle example could be done similarly in Maven.

[desruisseaux]

I also wish that "Projects using the Module System" become the default case. Note: just for completeness, maybe the slide should also contain the following elements in the Java's module-info side?

• Services used (would be in the "Module dependencies" section)
• Services offered
• Qualified exports (packages visible to only to some specific modules)
• Which modules are authorized to do reflection on which packages
• Javadoc

I mention that for emphasis that if the overlap between module-info and pom.xml is resolved by generating (at least partially) one from the other, then I think that the source should be module-info, as it is more than just a list of dependencies.

[jjohannes]

Completely agree with all of this. Thanks for the addition.
(The slide was focussing on the dependency management aspects that are covered by the build system in traditional setups, but you are right that it can be misunderstood.)

[rmannibucau]

For the record while I can agree supporting a bit better JPMS is a nice addition i hope the default stays plain classpath (or loader depending the env you evolve in) since it is current (and future for some years) ecosystem so nothing should create/require a module-info at any stage IMHO.

[desruisseaux]

The proposal is not to make module-info mandatory, neither to create a module-info automatically. The previous comment clarified that module-info would be considered as source code provided by developers.

The proposal is to reverse the current thinking in the way that build tools are designed: instead of having Maven or Gradle designed around the idea of class-path and handling modules as a special case, do the opposite: Maven or Gradle designed around the idea of modules, and having class-path as a special case. This approach has some consequences on the way that tools API is designed. But it will not force users to do JPMS, neither restrict them to modular dependencies.

In Maven case, I don't think that such shift will happen in Maven 4. I guess that the two models will co-exist in Maven 4, then maybe some methods like getClassPath() would be deprecated in Maven 5.

[rmannibucau]

If the outcome is "the dependency defines its usage" i'm fine with it since the default will stay the current - ie type=jpms is ok while default stays "jar" == classpath.
If the outcome is to change anything about current default case I would be happy for it to not happen like that - I don't want any impact of the mainstream users (JPMS is not yet anything important enough to consider it whatever the debate is about the chicken or the egg, and the java ecosystem is too diverse to see it as a primary thing yet since a lot of big OSS projects just reject it - not judging there if ambiguous).

[desruisseaux]

Yes, in Maven 4 the dependency <type> defines its usage. Note that the current default in Maven 3 is not jar == classpath, but rather jar == heuristic rules. This behaviour stay the same in Maven 4. One open question is whether the heuristic rules should stay exactly as in Maven 3, or slightly modified. This question has not been debated yet.

[rmannibucau]

What was debated is behavior will be 1-1 with model version 4.0.0, guess rest can be refined to be 1-1 with the type (jar=classpath, others are dispatched to where needed pby)

[cowwoc]

I have a slightly more minimalistic take on this.

I'm thinking we could declare all dependencies in module-info.java and optionally add extra metadata using Maven.

For example, any dependency found in module-info.java that can be automatically mapped to a GAV will get "reasonable defaults" like compile-scope. You'd need to add a way to map entries in module-info.java to a GAV if the default resolution method does not work.

As others have hinted, maybe we could do away with <dependencies> and add any extra information (like the version number) using <dependencyManagement>. (Non-modularized code could still use <dependencies> but configuring modularized libraries would become easier)

Thoughts?

[rmannibucau]

4 is decided due to v3 compat, this is why i spoke of v5.
Now i could say i have 10+ years of such tries of failures so do it the maven way, try outside, show adoption and we integrate it, for now we only have proof of the opposite

[jbee]

@rmannibucau IMHO you kind of make a self-fulfilling prophecy type of argument. The poor JPMS support in maven could be seen as a major contributor to the lack of adoption.

[cstamas]

Maven4 is "Maven3 compatible" (I guess this is "v3 compat" you mention), in a way it can run Maven3 plugins just fine. But nothing stops (and in fact this is the goal) to have compiler as Maven4 plugin (to use Maven4 API). Hence, the "v3 compat" is basically about "Maven4 can run Maven3 builds" (read: projects using Maven3 plugins). But IF a Maven4 plugin can do more, the better. And project using Maven4 plugins, is not subject of of "v3 compat" IMHO.

[desruisseaux]

I fully agree with jbee, this is the "chicken and egg problem" already mentioned on the developer mailing list. Regarding compatibility, nothing will change for developers using only the class-path.

[rmannibucau]

You keep saying that but it is wrong since if there was a real need people would have asked for compat way earlier, see what was happening in EE land before it moves to jakarta so ko, lack of support is never sufficient to explain the unlove a feature gets, no guess there.
As said years ago and recently, until spring enforces jpms - will be unlikely - or java removes the classpath, jpms will stay like that by construction.

[hendrikebbers]

We splitted the discussion in 4 sub-discussions:

• Maven supports compilation and execution on classpath or module path #35
• Better support for JLink #36
• How multiple source sets can be managed in Maven #37
• Better integration of Java modules as dependencies in the future of Maven #38