Splitting a large Meson project into multiple pyprojects

[tobiasdiez]

At Sagemath, we have the following wish-list for our Meson build:

• Have one monolithic project that compiles all Cython files and includes all Python files. This is mostly used for local development.
• Have one "standard" project (shipped to pypi) that includes almost all of the monolithic project, except for certain addons or extensions.
• Each of these addons should be shipped to pypi as a separate project.

One important aspect is that these addon's have additional external dependencies that are not (necessarily) shared by the "standard" installation.

I wanted to ask if someone has experience with a similar setup and can recommend an approach for implementing it?
The following options come to mind:

• Extract the addons to Meson subprojects that manage their own dependencies and installation instructions. The monolytic and standard project then include these subjects based on a "toggle".
• Use tags to mark the files belonging to a certain addon, and then toggle their installation by providing the correct --tags argument. I don't know though if it is possible to check for the presence of a dependency conditional on the provided install tag.
• Introduce a boolean variable for each addon, and then guard the installation of the relevant Cython/Python files using if-statements.

[rgommers]

Interesting question - and great to see Sagemath working on a Meson build!

I don't know though if it is possible to check for the presence of a dependency conditional on the provided install tag.

It isn't. Install tags are purely an install-time feature; all targets gets built even if they don't get installed. It works well for something like "don't include the test suite in the wheel", but not for conditional dependency detection or other conditional build steps.

Extract the addons to Meson subprojects that manage their own dependencies and installation instructions.

Subprojects are designed for external dependencies; my gut feel here is that they can be made to work for your use case, but aren't the best choice given that your project is in a single large repo and the addons you're trying to enable/disable aren't really external.

Introduce a boolean variable for each addon, and then guard the installation of the relevant Cython/Python files using if-statements.

This seems like the way to go. Important question: are these addons each in their own subdirectory? If so, then I think this shouldn't be hard: move the dependency() calls that you now all have in the top-level meson.build to the meson.build file in the addons subdirectory, and then based on the boolean variable use subdir('addon1') one level up to include it in the full build yes or no. This will then also solve the problem that for publishing the addon as a separate package, you need a separate pyproject.toml file (or amend the existing one) with a new project name, description, license, etc.

[tobiasdiez]

Thanks a lot! These are very valuable insights.

These addons are mostly in their own subfolders, but not always (we might or might not want to change this - the jury is not yet out on this). There are also ideas floating in the room to extract at least some of the addons to completely independent repositories. I have the feeling that subprojects would serve as a temporary middle way where the addons are still in the same repo but a bit more independent from the main code. Does this change your gutt feeling about using subprojects for this purpose or do you still prefer the boolean-variable approach?

This will then also solve the problem that for publishing the addon as a separate package, you need a separate pyproject.toml file

Just to make sure that I parse this sentence correctly: one still needs to have a separate pyproject.toml + main meson.build file for each addon - just that the latter is really simple and sets only the project metadata and then includes the meson build file of the addon, right?

[rgommers]

Does this change your gutt feeling about using subprojects for this purpose or do you still prefer the boolean-variable approach?

Not sure without trying. Both methods are easy to evolve into moving the add-on to a separate repo. And for the add-ons that aren't completely isolated, I think subprojects aren't an option. One advantage of subprojects is that it's slightly cleaner to have a separate project() call (see code outline below). I'd probably still try the boolean flag option first, but the differences may be minor/subtle.

one still needs to have a separate pyproject.toml + main meson.build file for each addon - just that the latter is really simple and sets only the project metadata and then includes the meson build file of the addon, right?

I think yes, but now I am unsure of parsing that sentence, so let me sketch it out. The repo will look like:

# top-level of the repo
meson.build       # contains a project() call for the main project
meson.options   # defines the `addon1_standalone` option (default: false)
pyproject.toml   # contains all metadata for the main project
    src/
        meson.build     # contains `if use_addon1; subdir(addon1)` 
        addon1/
            meson.build      # contains `dependency(my-dep-for-addon1)`
            meson.options   # also defines the `addon1_standalone` option (default: true)
            pyproject.toml  # contains all metadata for addon1

The src/addon1/meson.build should have a separate project() call to ensure that that meson.build file can be used standalone:

if get_option('addon1_standalone')   # build option, defined in meson.options 
  project(
    'addon1',
    'c', 'cython'
  )  # add meson_version, default_options, etc. as needed
endif

Thesrc/addon1/pyproject.toml file should contain that same flag [build-system] and [project] tables with metadata for addon1.

Then to build your wheels:

# from top-level directory
python -m build --wheel  # build wheel for main project
cd src/addon1
python -m build --wheel  # build wheel for addon1

[tobiasdiez]

Ah, this is an interesting idea. But sadly it doesn't seem to work. With the if get_option(...) around project, Meson complains that project is not the first statement in the meson file if I compile the addon separately. But without it the compilation of the main project fails because of a "Second call to project()". Any idea how to workaround this limitation? Eg, is it possible to have a main meson build file that is not called meson.build?

I did a bit more research and it seems people recommend subprojects in this case (e.g. mesonbuild/meson#6404 and mesonbuild/meson#9929). An alternative approach would be to have an <root>/addons folder that only contains the pyproject.toml + main meson build file for each addon, and a symlink to the main addon source code.

[rgommers]

Ahh that's too bad. The table in meson#6406-comment summarizes it well:

There will be some limitations though I expect, for the "have one monolithic project for local development" use case. For one, the add-on has to be a namespace package for it to show up in-tree in the monolithic mode (but that's okay still). More importantly, I suspect editable installs for the monolithic mode won't work for the add-ons in subprojects.