How multiple source sets can be managed in Maven

[hendrikebbers]

This is based on the discussion in #11

An idea (copied from #11) might look like this:

<sources>
  <source module="org.foo.bar" release="11">src/org.foo.bar/java</source>
  <source module="org.foo.biz" release="11">src/org.foo.biz/java</source>
  <source module="org.foo.biz" release="17">src/org.foo.biz/java17</source>
<sources>

Goal of this milestone should not be to have a final 100% solid integration in Maven but to have a definition for the future with a working prototype. Next to that parts of the prototype that does not make Maven 4 incompatible can be merged directly to Maven.

[rmannibucau]

Why not using the built in support of compiler (+resources if needed) plugin thru executions? 🤔

[desruisseaux]

Duplicating the same information in many places is entropy. The claim that few peoples care in practice is not backed by facts such as the very existence of this "Support & Care for Apache Maven" project and the fact that they have chosen JPMS as a priority. Users requests for JPMS support exist since at least 2018 (e.g. MNG-6390 — even if that particular one has been closed as "not a problem").

[rmannibucau]

Can you detail your entropy comoutation? You assume it will behave as today but in practose it will not. Common example is you dont want javadoc to use j17 sources when you build with j11 as baseline etc so it is NOT comparable to handle mjars and standard/current mode.
Entropy does not increase by using plugin conf.

Facts are just the low requests compared to some other features, this bugtracker is a bad reference IMO cause does not represent list activity nor users feedbacks.
Mjar and jpms always got solved in ways where users were happy (mass statement if you review slack or forums for ex) so yes this is far to be a big thing and even inside maven community mjar usage is not a shared and ack as good thing.

[cstamas]

Mjar and jpms always got solved in ways where users were happy (mass statement if you review slack or forums for ex) so yes this is far to be a big thing and even inside maven community mjar usage is not a shared and ack as good thing.

This sentence is simply untrue: just look at "modern bits" or Maven ecosystem, like latest new projects (mvnd), or latest resolver or latest indexer, they all use MJAR feature.

[rmannibucau]

Does not mean it is wrong, just that it passed either under radars (or lack of interest case for these modules) or in force like resolver...

All that must be discussed on the list and hopefully there will be a consensus but take care that this bugtracker opens tickets on topics where there is not yet any agreement (mainly by lack of exchange).
Now if you think you are numerous enough to pass in force you can ignore that too, I dislike it as an asf guy but ill also not fight and can just leave if it goes that way to avoid more friction than we already have.

[cstamas]

Facts are just the low requests compared to some other features, this bugtracker is a bad reference IMO cause does not represent list activity nor users feedbacks.

If I am about to agree with you, that Martin is "biased" and him pointing at JIRA issues is "not representational", I have to state at the same time that your point is "not representational" either: you are referencing to a code base that had literally little if no changes (sans "update deps" I do not count) happened since 15 years, and it was pretty much in state "as it was done" (before or around 15 years ago).

[desruisseaux]

My goal with the reference to MNG-6390 was not to said that requests are numerous (I have no data for that), but that they are not recent.

[rmannibucau]

Yep but it is like jsr223, when it got created we got a few requests, then nothing for years and recently when a "core" pushed enough it became a maven plugin.
No issue to do the same with mjar (in compiler) or jpms but shouldnt impact the core model yet IMHO (to recheck for v5 if it changed is what I mean).

[jjohannes]

Because this was compared to what Gradle offers in earlier comments, I would like to share some info on that. Maybe it helps to put the scope here into perspective. In Gradle these things are called Source Set and Source Directory Set (small but important destiction!). Both Source Set and Source Directory Set can be dynamically added to a project. In fact, these are not core concepts but only concepts added by the java plugins that are implemented on top of the dynamic "task graph" and "dependency management" systems of Gradle core. I also have a video and further doc links on the topic.

Example

└── src
    ├── main           <- Source Set
    │   ├── java          <- Source Directory Set (it's called Set because in can have several folders, but in practice it usually has one)
    │   ├── kotlin        <- Source Directory Set
    │   └── java21        <- Source Directory Set
    ├── testFixtures   <- Source Set
    │   └── java          <- Source Directory Set
    ├── integrationTest <- Source Set
    │   └── java          <- Source Directory Set
    └── test           <- Source Set
        ├── java          <- Source Directory Set
        └── groovy        <- Source Directory Set

• A Source Set has a separate context. It may be completely disconnected from other source sets, almost as if it was a separat (sub)project. It has it's own dependency scopes. E.g. if you add the testFixtures source set, you have testFixturesImplementation and testFixturesApi scopes for declaring dependencies. And there is a testFixturesCompileClasspath and a testFixturesRuntimeClasspath. So each source set has its own dependencies.
• All Source Directory Sets that belong to a source set share the same context but usually have a separate task operating on each of them. E.g. main/java is processed by a Java compile task and main/kotlin by a Kotlin compile task. But both tasks get the same compileClasspath as input.

In Maven...
....I would say there are already plugins which behave similar to the Source Directory Set concept. For example the kotlin-maven-plugin. I think this is quite comparable because they also work in the same context (lifecycle) with the same dependency scopes. Maybe this concept can be generalized so that it becomes easier to add new Source Directory Sets. One use case is to have several Java source folders for different Java versions where the compile goal would be executed for each of them with different compiler options.
For the concept of Source Set I cannot really imagine how it would work in Maven. As you have the fixed compile and test-compile lifecycle phases and the fixed dependency scopes, it does not really map. You could think about that you can define a "source set" and then map it to one of the existing phases and dependency scopes, but then there would be conceptually not much difference to a Source Directory Set I think.

[cstamas]

@gnodet ping, and this is what i wanted to achieve with (different kind of) "scopes" and "project paths", if you remember

[rmannibucau]

Hmm, I still fail to see how you make it work, do you move the full compiler+javadoc+jlink+... configurations to source tag? If not we just end with <source>src/org.foo.biz/java17</source> which is what we kind of already have else does it mean we drop default plugin configuration to move it elsewhere (would be very negative since we just forced to define the plugin and therefore the configuration is not an issue)?

The other issue in the original snippet is:

  <source module="org.foo.bar" release="11">src/org.foo.bar/java</source>
  <source module="org.foo.biz" release="11">src/org.foo.biz/java</source>

Maven community - I exclude myself there ;) - always rejected this idea to build 2 (GA/not related) jars in a single module and make 1 module = 1 multimodule project, so it sounds very fishy and build a mjar is not that hard and would still require more setup anyway (I don't think it would be auto enabled just cause there are 2 sources).

[desruisseaux]

The Gradle's concept of "source set" may map approximately (but not completely) to the type attribute in above proposal. I think that both approaches would be equivalent for JPMS. The type approach may be easier to map to the existing Maven design.

About building many JAR files in a single Maven module, even if the community rejected that proposal in the past, it may be time to reconsider. Because the Java language has evolved. In a Modular Source Hierarchy, those JAR files are not unrelated. Their content are the result of a single call to javac, it is not one separated compilation per JAR file. So before to reject this idea, there is two questions to answer:

• Can you compile this simple test case with Maven or Gradle modules without Java compiler warnings? That test case demonstrates the problem with Gradle, but it is the same problem with Maven. The README.md explains in details.
• If no, is it Maven's decision to not allow developers to build their project cleanly, at least regarding those standard Java warnings or errors?

Unless you can answer "yes" to one of above questions, multi-JAR files is a natural consequence of one way (among 2 or 3 ways) that Java standard compiler works today, and are necessary for allowing developers to build safer code and documentation in cases like above use case. Anyway, it is not in this thread that we can said "no, this safety doesn't deserve to be given to developers". This is a question for the community.

[rmannibucau]

The sample mixes topics.

1. Building a mjar without warnings -> we already do since day 1 (think log4j does if it didnt change)
2. Module hierarchy (jre/jdk layout) is not supported by maven/gradle but is by mx or jdk toolkit, this is the opiniated choice of the build and maven does not this one. Agree it can be changed for v5 but it will also make it harder for all the mainstream cases where it is not relevant - keep in mind it was design for the jdk monolitic build

Anyway belongs to the list, anything discuss there has very few chance to be asf friendly

[desruisseaux]

Building a mjar

If "mjar" means "multi-releases", the sample is not at all about this topic. It is only about Module Source Hierarchy.

Module hierarchy (jre/jdk layout) …snip… keep in mind it was design for the jdk monolitic build

Module Source Hierarchy is for any project that wish to use it. The OpenJDK project uses it, but there is nothing specific to OpenJDK. Not allowing developers to use it if they want seems more opinionated than giving this freedom.

[rmannibucau]

Ok, agree on mjar - thought at some point you want to merge the "same" module to do a mjar since it is/can be somehow related.

Still module source hierarchy was for the jdk itself originally.
While it can meet any project - and I'm thinking to Tomcat more than any other maven based projects - it must not go against the build tools and it does for maven (and gradle at some point).
If we stick to javac in the reasoning we'll miss the part that maven is a pipeline and the pipeline has assumptions, this does break it since 1 module = multiple main artifacts.

Side note agree with Gradle comment that it is probably not a good default case - indeed you can always enable it overriding default plugins: gradle/gradle#25974 (comment).

[desruisseaux]

Side note agree with Gradle comment that it is probably not a good default case

That Gradle comment was based on a misunderstanding of what Module Source Hierarchy is. I'm in the Community Over Code Europe conference and had a talk with a Gradle engineer yesterday. They reopened the issue following that discussion.

it must not go against the build tools

It seems to me that Maven should adapt to Java, not the opposite.

[rmannibucau]

It seems to me that Maven should adapt to Java, not the opposite.

You probably noted this is not the direction java takes, only the jdk, it is not even promoted by the oracle evangelists....

[desruisseaux]

There is no way it can be widely used as long as the tools do not support it. Note also that the Oracle evangelists do not neither said "don't use it". My guess is that they just acknowledge that they can't said "use it" if developers are unable to do so with Maven or Gradle.

[rmannibucau]

There is no way it can be widely used as long as the tools do not support it.

I know it is your thing so let do it, please make it possible to create one folder per half of class - there is no user but until it is in maven there will be none ;).

My guess is that they just acknowledge that they can't said "use it" if developers are unable to do so with Maven or Gradle.

You said it was the same for JPMS but they promoted it a little bit emphasing the target was primarly the JDK itself so I guess this is not right.

Also note there is not a single use case we don't enable without that feature too so not sure that it is something we should focus on until we fixed all issues of the maven bugtracker - which can be saner than targetting new features for now.

[desruisseaux]

You said it was the same for JPMS but they promoted it a little bit emphasing the target was primarly the JDK itself so I guess this is not right.

I remember to have seen this statement from an Oracle employee, but there is also material such as Youtube videos of Oracle employees showing how to use JPMS in our own projects. There is also some new Java features that requires JPMS, such as JEP-476 (we way agree or not with that JEP, this is not the point). In addition of already existing tools that are clearly targeted to users, not the JDK, such as jlink.

[rmannibucau]

Does not change much that we dont need this thread feature to enable all jvm use cases...so noop is fine for me

[desruisseaux]

Noop may be fine for you, but not for other developers on this thread. Nobody is forcing you to use JPMS if you don't want. But conversely, please don't block those who want to use JPMS (and no, it is hardly possible with today's Maven. We tried very hard).

[rmannibucau]

I dont block anyone to do - even helped actually, I'm just trying to keep maven simple and not become ant as proposed in this thread for very few people.