Allow marking a module "available for parallel compilation", and compile all of the scala sources in parallel

[kaepaokaawela]

In my work project, I have a lot of modules that I've structured like the following: xxx-interface and xxx-impl.

The implementation module contains a lot of code that are simply implementations of the interface, and as such, between the files inside xxx-impl, there are no dependencies. Naturally the implementation module has much more code and is slower to compile.

If there is a way to mark a module as "parallel compilation", such that each and one of those files are invisible to each other so we can compile all of them in parallel, it would be a game changer for me.

• The compilation result should remain the same
• Referring to other files should result in a not found error at compile time (bonus points to manually allow some files to depend on some specific other files)

[lihaoyi]

There currently isn't such a way, but maybe one could be developed. @bishabosha did some work on parallelizing compilation in Scala 3, and there's some prior work with Hydra in Scala 2, but Mill doesn't have anything built in. We use the same compilation toolchain as the rest of the Scala ecosystem

[kaepaokaawela]

@lolgab Thank you very much for the help.
For me the requirements are quite simple, so I have slightly tweaked your first version, and it works like a charm:

trait ParallelModule extends CommonModule { outer =>
  val fileSrc = millSourcePath / "src"
  def fileSrcInput = T.input { PathRef(fileSrc) }
  interp.watch(fileSrc)

  def inputModuleDeps: Seq[JavaModule]

  val submoduleFiles = os
    .walk(fileSrc)
    .filter(f => f.ext == "scala" || f.ext == "java")
    .map(_.relativeTo(fileSrc))

  def allSourceFiles = T { List[PathRef]() }

  object file extends Cross[CrossFile](submoduleFiles.map(_.toString.replace("/", "_")))
  trait CrossFile extends CommonModule with Cross.Module[String] {
    def allSourceFiles = T { List(PathRef(fileSrc / crossValue)) }
    def sources = Seq.empty[PathRef]
    def moduleDeps = inputModuleDeps
  }

  def moduleDeps = file.crossModules
}

  object implementation extends ParallelModule {
    def ivyDeps = super.ivyDeps() ++ Agg(
      // stuff
    )

    def inputModuleDeps = Seq(interface, definition)
  }

About the tree shaped dependencies -- interesting idea, but I wonder if linear dependency levels (0, 1, 2, 3, ...) using prefixes in the directory names foo/src/00-shared-fragments foo/src/01-implementations may be easier to deal with. Inside each level, none of the sources can see each other, but they can all see the ones above.

Obviously trees can express linear dependencies, but having to dig through the directory hierarchy may be a bit cumbersome. Personally my project structure isn't complex enough to require trees to describe, so just a thought 🙂

[kaepaokaawela]

Sad news. Previously, it took 30 seconds to compile the whole thing single threaded.
After I parallelized the module compilation, the longest file took 24 seconds (there 30+ files right now).

In my machine, parallelization makes compilation take more or less the same (wall clock), and requires more cpu and ram, so I am going to revert this change.

[lolgab]

You can try to split in less groups, maybe 2 groups and see if you see some benefit.

[bishabosha]

each time you compile that makes a separate time the compiler has to initialise the entire classpath. It would be much more efficient for the compiler itself to become batch-aware

[kaepaokaawela]

I see. This module sits at the near bottom of the module hierarchy, so the visible classpath is quite big. Also, each file of the module contains Scala 3 code that uses a lot of macros (quill).

I don't know which one contributes more to the initialization overhead. Let me try to fiddle with the granularity and see if I can find an optimal level of parallelism.

If only there is a way to pre-heat and reuse the compiler state, or if the compiler is to be batch aware.. (but then this is not a mill issue anymore)