No options to control number of parallel compiling processes

[jinluchang]

I don't see a way to control the number of parallel compiling processes in the documentation. Reading the code, it seems that there is currently no way to specify this number but to rely on ninja's default parameter:

meson-python/mesonpy/__init__.py

Line 796 in ef67e0e

self._meson('compile')

Ninja and Meson's philosophy prevent this issue from being solved by using an environment variable. So as far as I can see, there has to be an option added to meson-python to support altering this option (and perhaps other meson compile options).

Any help would be really appreciated.

[rgommers]

Thanks for the request @jinluchang. This will be enabled by gh-167.

[eli-schwartz]

Ninja and Meson's philosophy prevent this issue from being solved by using an environment variable.

It's really nothing to do with Meson. This is comparable to MAKEFLAGS=-j4 make, although Make permits many things in MAKEFLAGS that it should probably not.

Meson supports some environment variables during configuration, but "freezes" them at the time of configuration. For example, $CC/$CFLAGS.

Ninja, sadly, rejects the request for $NINJAFLAGS, not out of any principled objection to environment variables, but apparently because ninja focuses on speed, and getenv(2) is out of scope as a result (I guess it's just way too slow, lol) but "we suggest you write a python script called /usr/local/bin/ninja for cross-platform compatibility, and have it invoke the real ninja with your system's preferred -j option hardcoded". Astounding.

... Using https://github.com/michaelforney/samurai you can set SAMUFLAGS=-j4 and then run pip install X where X uses this build backend; if meson-python picks up samurai as the ninja program to run (some distros symlink samurai as ninja, either by default or as an option) then samurai will pick up that environment variable.

Note: you will need #175 to avoid installing a PyPI-specific version of ninja that overrides whichever system one you might have.

[jinluchang]

@rgommers Thank you! #167 is exactly what I hoped for. Perhaps I can close this issue now?

@eli-schwartz I agree with your point. It would be great to have #175 too. At present, I have to create a dummy package to fill this dependency https://github.com/jinluchang/Ninja-dummy. Having a dependency that downloads during compile time can be quite inconvenient sometimes scikit-build/ninja-python-distributions#127 (comment).

[henryiii]

#175 is just waiting for #202 and #203 (IIRC). I'll try to finish it off as soon as those go in. (Though I've forgotten why #202 was needed for it).

Just curious, why are making a dumpy package? If you want to control compile time completely, such as for distributions, you can use --no-isolation --skip-dependency-check/-nx (build) or --no-build-isolation (pip).

[jinluchang]

@henryiii Good suggestion! I didn't think of that.

[rgommers]

@jinluchang this should work now that gh-167 has been merged. I just tested that this does what you'd expect:

pip install . --config-settings compile_args="-j1"

[jinluchang]

Great! Thanks.