Build packages in parallel. #35
Conversation
MXE Octave doesn't (currently?) support building with parallel `make` initially. However, once Octave has been (cross-)built, building multiple packages in parallel works fine. And it is more efficient to do so, e.g., because some parts of the build steps of any of the built packages (like running a configure script) cannot be parallelized. Building multiple packages in parallel allows using more CPU threads at the same time in these cases. To allow that, invoke `make` twice: First, to build Octave (without parallel `make`). And then again if the previous invocation was successful, to build the remaining tools and packages (with four parallel `make` processes). That should be the equivalent of running the following command locally: `make JOBS=8 && make -j4 JOBS=2 all 7z-dist zip-dist nsis-installer`
There was a problem hiding this comment.
@mmuetzel This looks reasonable to me. Shall I apply this config to the live system?
Unless you want to replicate the full system for testing on your local system using Docker compose https://github.com/gnu-octave/octave-buildbot/tree/main/test, we can only test "in production".
The metric you want to compare is if the package build will be faster than with current sequential build of packages using multiple CPU threads, right?
Just let me know if we want to revert of make follow-up changes.
|
Yeah. Ever since @jwe gave the hint to building multiple packages after Octave has been built, I'm using that approach locally. And from what I can see, building with MXE Octave is faster with that. Also, jwe's buildbots are using that approach. And @jwe builds the Windows installer like that. I tried to change the rules after reading the buildbot documentation. I hope I got the conditions (like for I'd appreciate if you could update the buildbot master with these changes after the merge. |
|
Great, then I'll update the configuration later when I am back on the PC. |
|
The changes should be active with the next build. |
|
It looks like this triggered the following errors:
Maybe, we need to use the full namespace here? That would probably be |
The buildbots are failing after gnu-octave#35 with errors like the following: * builtins.NameError: name 'SUCCESS' is not defined * builtins.NameError: name 'SKIPPED' is not defined That might be because these constants need to be fully qualified. Add the full namespace where these constants are used.
* Attempt to fix errors about undefined names. The buildbots are failing after #35 with errors like the following: * builtins.NameError: name 'SUCCESS' is not defined * builtins.NameError: name 'SKIPPED' is not defined That might be because these constants need to be fully qualified. Add the full namespace where these constants are used. * master.cfg import undefined symbols --------- Co-authored-by: Kai Torben Ohlhus <[email protected]>
MXE Octave doesn't (currently?) support building with parallel
makeinitially. However, once Octave has been (cross-)built, building multiple packages in parallel works fine. And it is more efficient to do so, e.g., because some parts of the build steps of any of the built packages (like running a configure script) cannot be parallelized. Building multiple packages in parallel allows using more CPU threads at the same time in these cases.To allow that, invoke
maketwice: First, to build Octave (without parallelmake). And then again if the previous invocation was successful, to build the remaining tools and packages (with four parallelmakeprocesses).That should be the equivalent of running the following command locally:
make JOBS=8 && make -j4 JOBS=2 all 7z-dist zip-dist nsis-installer@siko1056: I don't know how to test these changes. Does this look reasonable to you?