module for creation of toolset projects for libs

[grisumbras]

The PR

1. adds a module to add customisable targets for external dependencies;
2. adds a vcpkg support module that allows searching for vcpkg-installed packages;
3. modifies pkg-config module to be usable with the previous two without need for extensive configuration by users.

The functionality is thoroughly described in the documentation also added by the PR. The intention is to allow project
authors to request external dependencies, and the users to be able to conveniently describe how to find them.

vckpg support is partcularly relevant on Windows, where it quickly becomes one of the most popular package managers, if not the default one.

Together all changes should allow (in the best case scenario) something like the following to work:

# -config.jam
using vcpkg ;

# Jamfile
import external ;
external.lib /foo//somelib : : <header>foo/somelib.hpp ;
exe myapp : main.cpp /foo//somelib ;

Also, if the project for external dependency already exists at the time of invocation of external.lib or external.alias, the
module does not try to create it again, and does not create a target either. This makes it trivially overridable by e.g. Conan generators.

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no api changes)
• Documentation content changes
• Other (please describe):

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

• I searched the discussions
• I searched the closed and open issues
• I read the contribution guidelines
• I added myself to the copyright attributions for significant changes
• I checked that tests pass locally with my changes
• I added tests that prove my fix is effective or that my feature works
• I added necessary documentation (if appropriate)

I couldn't figure out how to mock existence of pre-installed packages, so no tests.

[grisumbras]

@grafikrobot can you comment on this PR?

[grafikrobot]

@grisumbras sorry it's taking a long time to look at this. I've had some personal things to deal with. Unfortunately it's going to take me even longer. As I now have to finish writing some wg21 wording on a paper in the next 10 days.

[grafikrobot]

Sorry it took so long for me to look at this. But finally had time, the whole weekend free time, for this. I have to say I'm really confused by the changes even after hours of reading the code and imagining how this is meant to be used. So instead of a bunch single comments to specific code parts I'm going to list some questions I have roughly from beginning to end:

• Why does ac.jam need to know anything about vcpkg?
• What do the changes in project.jam do?
• There's a some, seemingly undocumented, use of env variables in external.jam.
• Why is external.jam doing vcpkg specific things if this is supposed to be reusable?
• Why does external.jam use ac.jam?
• Does external.jam support non-global targets?
• What do the changes in pkg-config.jam do?
• Does vcpkg.jam handle multiple roots?
• Why does vcpkg.jam not specify address-model for some variants?

I also have some UI design comments/questions:

• Why doesn't this integrate with the existing package support. I.e. this seems to behave entirely different than existing package support. Why?
• Having users do special external.lib/alias targets directly in jamfiles will not help in the long term. As users will not be able to switch package managers without lots of changes. And it would make it hard to intermix multiple package managers.

From a practical perspective I don't see why we need these many changes to various parts just for vcpkg support. I don't see how this is going to be maintainable long term. In that it looks like there's a lot of disparate coupling in what you're doing.

My current inclination is to drop this PR entirely and start a different approach from scratch.

[grisumbras]

I will first answer design/UI questions.

Why doesn't this integrate with the existing package support. I.e. this seems to behave entirely different than existing package support. Why?

This does integrate with PM support, to the extent it is possible. The core of this PR is external.jam, which is supposed to be used by library authors to declare their dependencies (this declares targets). The mechanism to set up those targets uses the same system as ac.jam, that is it searches in known directories for headers and library binaries.
Users who do not use a PM that creates such targets for b2 will be able to configure what those known directories are.
Users who do use a PM that creates targets for b2, will not have a problem, because the external.jam "disables" itself in that case.

Having users do special external.lib/alias targets directly in jamfiles will not help in the long term. As users will not be able to switch package managers without lots of changes. And it would make it hard to intermix multiple package managers.

As I've mentioned, the module "disables" itself when it sees that a project that it tries to create already exists. I've experimented with Conan, and Conan-created project (using your b2 generator) was successfully overriding targets created by external.jam. So if a user changes PM no change is actually required at all.

Implementation questions:

Why does ac.jam need to know anything about vcpkg?

My original intention was to add the functionality to ac.jam, but then I realised the change would be more significant, and I moved it to a separate module. But I've left vcpkg support in ac.jam in order for existing library modules (e.g. openssl.jam) to benefit from it. ac.jam's works by searching in several known directories. I added vcpkg packages directories to the list of those known directories, because vcpkg increasingly becomes the default way for devs to install libraries on Windows. I can make this change more generic, to allow other potential PMs of this kind.

What do the changes in project.jam do?

They allow external.jam to not create a project when a PM already did that. In my experiments, it's a common occurance that b2 starts to load a PM-generated project, but before it is done, it loads the jamfle, which in turn invokes external.lib/alias. So, external.jam could not reliably see if a PM already created a project it wants to create. So, I added a list of projects that are being loaded currently.

There's a some, seemingly undocumented, use of env variables in external.jam.

Can you be more specific? As far as I can see, use of env variables was copied from ac.jam.

Why is external.jam doing vcpkg specific things if this is supposed to be reusable?

I answered this previously, but to reiterate, I added vcpkg package directories to the list of directories where files are searched for. This is kind of the same as looking in /usr/local on *nix. I can make it more generic to allow other such PMs to benefit from this, but I don't expect anyone using it.

Why does external.jam use ac.jam?

Library discovery in external.jam is based on ac.jam (it's basically an extension). Classes in external.jam inherit from ac-library from ac.jam.

Does external.jam support non-global targets?

Not at the moment. Do you think there's benefit in such support? I have a suspicion it would hinder the ability to disable external.jam-creation of targets when a PM already created them.

What do the changes in pkg-config.jam do?

When experimenting with using pkg-config with external.jam, I've realised that I've made pkg-config module unnecessarily hard to use. So, I've added condition argument to its using rule, which pins a registered pkg-config configuration to a particular combination of properties. I also added target /pkg-config//prefix as an implementation detail (it allows using b2's variant resolution mechanism to choose the appropriate pkg-config configuration).

Does vcpkg.jam handle multiple roots?

It does. But possibly, I didn't understand your question. Can you be more specific?

Why does vcpkg.jam not specify address-model for some variants?

Oh, I think, I forgot to add address-model to 64-bit variants.

Updated: added address-model to all variants that lacked it.

[grisumbras]

@grafikrobot can we return to discussing this PR?