[question] Howto avoid "Requirement 'x' not in lockfile 'requires'" with input from conan graph build-order

[marlamb]

Hi,
I am using conan graph build-order to determine the build order for the complete dependency tree of our central project given a central lockfile. Then I use the result to build the dependencies one by one, again using the central lockfile, to ensure consistency. Now it turns out that in some situations the central lockfile does not have a requirement in the correct section (requires vs build-requires). E.g. I have a requirement to libcurl, which in turn has a build requirement to libtool, which in turn has a requirement to automake. In my lockfile libtool and automake are part of the build-requires, which makes sense. But when I do the conan create <path_to_libtool_recipe> --lockfile=<central_lockfile> <settings_and_options_parsed_from_conan_build_order_json> it fails with

ERROR: Requirement automake/... not in lockfile 'requires'

I also saw the problem the exact other way around (often also in test packages).
I read about --lockfile-partial, which (if I understand it correctly) is also not what I want, as it would loosen almost everything, but in general I of course want that the exact libs/recipe revisions from the lockfile are used.

My question is: is there a way to avoid the error while still ensuring that only defined versions/recipe revs are used? Note that using conan install would not be ideal for me, as I would like to execute also the test packages (recipes might change along and I want to ensure that they still work as intended).

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @marlamb

Thanks for your question.

I'd say it is very likely that the lockfile or conan graph build-order capture might be missing something.
These commands must be executed with the same arguments as the expected posterior create/install commands. Same profiles, settings, etc. But most importantly the same --build=missing or similar policy.

If you don't use --build=missing when capturing the lockfile or the build-order, then they might be missing some of the tool-requires or their transitive dependencies.

Could you please try to check this?

If this is not then:

E.g. I have a requirement to libcurl, which in turn has a build requirement to libtool, which in turn has a requirement to automake. In my lockfile libtool and automake are part of the build-requires, which makes sense. But when I do the conan create <path_to_libtool_recipe> --lockfile=<central_lockfile> <settings_and_options_parsed_from_conan_build_order_json> it fails with

We would like to have a bit more of details to reproduce this. If you can share the exact commands you are using to capture this lockfile, that would help to reproduce and understand. Thanks!

[marlamb]

Hi @memsharded , thanks for the quick reply. I created a minimal example that reproduces the error. It is greatly simplified and does steps manually instead of reading some of the files, but I hope it helps nevertheless to detect the underlying issue.

I use the following file structure

conan_17363
├── experiment.bash
└── recipes
    ├── build_tool
    │   └── conanfile.py
    ├── consumer
    │   └── conanfile.py
    └── transitive_dep
        └── conanfile.py

The content of the files is:

# recipes/build_tool/conanfile.py

from conan import ConanFile

class BuildToolConan(ConanFile):
    name = "build_tool"
    version = "1.0.0"

    def requirements(self) -> None:
        self.requires("transitive_dep/2.0.0@marlamb/testing")

# recipes/consumer/conanfile.py

from conan import ConanFile

class ConsumerConan(ConanFile):
    name = "consumer"
    version = "0.0.1"

    def build_requirements(self) -> None:
        self.tool_requires("build_tool/1.0.0@marlamb/testing")

# recipes/transitive_dep/conanfile.py

from conan import ConanFile

class TransitiveDepConan(ConanFile):
    name = "transitive_dep"
    version = "2.0.0"

and finally the experiments.bash file

#!/usr/bin/env bash

set -e

ROOT_DIR="$(dirname -- "$(readlink -f -- "$0";)";)"

VENV_DIR="${ROOT_DIR}/venv"
rm -rf ${VENV_DIR}
virtualenv ${VENV_DIR}
source ${VENV_DIR}/bin/activate
pip install conan

CONAN_DIR="${ROOT_DIR}/conan_home"
rm -rf ${CONAN_DIR}

export CONAN_HOME="${CONAN_DIR}"
conan remote disable "*"
conan profile detect


RECIPES_DIR="${ROOT_DIR}/recipes"
BUILD_TOOL="${RECIPES_DIR}/build_tool/conanfile.py"
TRANSITIVE_DEP="${RECIPES_DIR}/transitive_dep/conanfile.py"
CONSUMER="${RECIPES_DIR}/consumer/conanfile.py"

USER_CHANNEL="--user marlamb --channel testing"

conan export "${BUILD_TOOL}" $(echo "${USER_CHANNEL}")
conan export "${TRANSITIVE_DEP}" $(echo "${USER_CHANNEL}")
conan export "${CONSUMER}" $(echo "${USER_CHANNEL}")

LOCKFILE="${ROOT_DIR}/conan.lock"

conan lock create --lockfile-out="${LOCKFILE}" --no-remote $(echo "${USER_CHANNEL}") "${CONSUMER}"

conan graph build-order --build=missing $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}" --format=json --order-by=configuration --reduce "${CONSUMER}" > "${ROOT_DIR}/build_order.json"

# In reality now the `build_order.json` gets parsed to determine all settings/options for the subsequent "conan create" commands.
# In this minimal example no settings/options exist, so the next commands would be

conan create "${TRANSITIVE_DEP}" $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}"
# The following fails with
# "ERROR: Requirement 'transitive_dep/2.0.0@marlamb/testing' not in lockfile 'requires'"
# which is obviously true, looking at the content of ${LOCKFILE}.
conan create "${BUILD_TOOL}" $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}"

I saw the requires field in the info section of build_tool within the build_order.json, but I don't see a way passing it to conan create.

[memsharded]

Thanks very much for the detailed reproducible case, it really helped! This helps a lot to quickly identify issues.

The missing thing is that:

conan create "${BUILD_TOOL}" $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}"

Is missing the --build-require specifier

conan create "${BUILD_TOOL}" $(echo "${USER_CHANNEL}") --lockfile="${LOCKFILE}" --build-require

Note that you want to build the build_tool that will work as a tool_require in the "build" context, to be used later as such by the consumer. But that conan create is creating by default things in the "host" context, unless you add the --build-require. The --build-require does the effective (and necessary in many cases) of the build-host profiles, so it will not build the package for the host context, as such application would be useless for any cross-building scenario, but it will build it for the "build" context, so the binary will exist when it is used as tool-requires.

In the conan graph build-order you should see the --tool-requires=build_tool/version --build=build_tool/version, because that build order is intended to be used with conan install, not conan create, because that is easier than having to find the url and commit and do the correct checkout for the conan create.

Please let me know if this helps.

[marlamb]

Yes, indeed this seems to solve the issue. I will try that out on a bigger problem in the next days.

I was hoping to be able to use conan graph build-order also in combination with conan create in an intended way. In my system I have full control over the recipes, but I have potential changes that might break the test packages. Therefore conan create seems to be the best choice.

I extended my example a little in order to check that I got everything right also with respect to cross-build situations, but still have some questions (will follow after the example).

It now looks like this:

conan_17363
├── config
│   └── profiles
│       ├── debug
│       └── release
├── experiment.bash
└── recipes
    ├── build_tool
    │   └── conanfile.py
    ├── consumer
    │   └── conanfile.py
    ├── runtime_dep
    │   └── conanfile.py
    └── transitive_dep
        └── conanfile.py

The files are:

# config/profiles/debug
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux

# config/profiles/release
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux

# recipes/build_tool/conanfile.py
from conan import ConanFile

class BuildToolConan(ConanFile):
    name = "build_tool"
    version = "1.0.0"
    settings = "build_type"

    def requirements(self) -> None:
        self.requires("transitive_dep/2.0.0@marlamb/testing")

# recipes/consumer/conanfile.py
from conan import ConanFile

class ConsumerConan(ConanFile):
    name = "consumer"
    version = "0.0.1"
    settings = "build_type"
    
    def requirements(self) -> None:
        self.requires("runtime_dep/1.2.3@marlamb/testing")

# recipes/runtime_dep/conanfile.py
from conan import ConanFile

class RuntimeDepConan(ConanFile):
    name = "runtime_dep"
    version = "1.2.3"
    settings = "build_type"

    def build_requirements(self) -> None:
        self.tool_requires("build_tool/1.0.0@marlamb/testing")

# recipes/transitive_dep/conanfile.py
from conan import ConanFile

class TransitiveDepConan(ConanFile):
    name = "transitive_dep"
    version = "2.0.0"
    settings = "build_type"

#!/usr/bin/env bash

set -e

ROOT_DIR="$(dirname -- "$(readlink -f -- "$0";)";)"

VENV_DIR="${ROOT_DIR}/venv"
rm -rf ${VENV_DIR}
virtualenv ${VENV_DIR}
source ${VENV_DIR}/bin/activate
pip install conan

CONAN_DIR="${ROOT_DIR}/conan_home"
rm -rf ${CONAN_DIR}

export CONAN_HOME="${CONAN_DIR}"
conan config install "${ROOT_DIR}"/config
conan remote disable "*"

BUILD_ORDER_FILE="${ROOT_DIR}/build_order.json"
rm -f "${BUILD_ORDER_FILE}"
LOCKFILE="${ROOT_DIR}/conan.lock"
rm -f "${LOCKFILE}"

RECIPES_DIR="${ROOT_DIR}/recipes"
TRANSITIVE_DEP="${RECIPES_DIR}/transitive_dep/conanfile.py"
BUILD_TOOL="${RECIPES_DIR}/build_tool/conanfile.py"
RUNTIME_DEP="${RECIPES_DIR}/runtime_dep/conanfile.py"
CONSUMER="${RECIPES_DIR}/consumer/conanfile.py"

USER_CHANNEL="--user marlamb --channel testing"

conan export "${TRANSITIVE_DEP}" $(echo "${USER_CHANNEL}")
conan export "${BUILD_TOOL}" $(echo "${USER_CHANNEL}")
conan export "${RUNTIME_DEP}" $(echo "${USER_CHANNEL}")
conan export "${CONSUMER}" $(echo "${USER_CHANNEL}")

conan lock create \
  --lockfile-out="${LOCKFILE}" \
  --no-remote \
  $(echo "${USER_CHANNEL}") \
  --profile:build=release \
  --profile:host=release \
  --profile:host=debug \
  "${CONSUMER}"

# Determine profile:host and profile:build from desired consumer profile.
conan graph build-order \
  --build=missing \
  $(echo "${USER_CHANNEL}") \
  --lockfile="${LOCKFILE}" \
  --profile:host=debug \
  --profile:build=release \
  --format=json \
  --order-by=configuration \
  --reduce \
  "${CONSUMER}" > "${BUILD_ORDER_FILE}"

# Extract settings (and options) from "info" section in build.order.json file.
# Decide for "--build-require" based on "--tool-requires" in "build_args" section in build.order.json file.
conan create \
  "${TRANSITIVE_DEP}" \
  $(echo "${USER_CHANNEL}") \
  --lockfile="${LOCKFILE}" \
  --profile:host=debug \
  --profile:build=release \
  --settings:host=build_type=Release \
  --build-require

conan create \
  "${BUILD_TOOL}" \
  $(echo "${USER_CHANNEL}") \
  --lockfile="${LOCKFILE}" \
  --profile:host=debug \
  --profile:build=release \
  --settings:host=build_type=Release \
  --build-require

conan create \
  "${RUNTIME_DEP}" \
  $(echo "${USER_CHANNEL}") \
  --lockfile="${LOCKFILE}" \
  --profile:host=debug \
  --profile:build=release \
  --settings:host=build_type=Debug

The key point is obviously where I assemble the arguments for the conan create calls. To me it looks like the settings put into the "info" section are correct and also take into account the cross-building scenario. But it needs the --build-require in order to work also with the lockfile. In this small example it would also work without parsing the settings at all, as the profiles already contain everything and the --build-require puts the correct settings/options into place. Do you have a recommendation what information should be used? Using both looks somehow overdone, but I still like the idea to just give the settings/options to the conan create, as it is very explicit.

[memsharded]

I was hoping to be able to use conan graph build-order also in combination with conan create in an intended way. In my system I have full control over the recipes, but I have potential changes that might break the test packages. Therefore conan create seems to be the best choice.

Not necessarily, see my comments in the PR for the CI Tutorial: conan-io/docs#3799 (comment)

For running conan create it would be necessary to:

• Use the scm approach to store the SCM coordinates in conandata.yml
• For a given package reference, retrieve the scm coordinates from its conandata.yml
• Do the git clone and checkout for those coordinates
• conan create in the cloned folder

Steps 2 and 3 will be automated by the new conan workspace open command.

In any case, given the above link, I'd probably recommend conan install --requires=<ref> --build=<ref> as the conan graph build-order returns, focus on the overall CI process, and leave building from conan create for later.

If you haven't yet, please check conan-io/docs#3799, hopefully it will be published soon.

[marlamb]

Thanks for pointing out to the PR, it contains really a lot valuable information! Now (I think) I understand, why you are building the graph-order mainly for conan install. If I am not mistaken, it is because of the asynchronous and highly decoupled nature within your system (people providing recipes are others than consuming the packages) and hence the design to separate the package and the product pipelines is natural.

But I don't think that this is the only possible design. Within the system I use no scm problem exists by design, all recipes and consumers have a consistent state at any time. Therefore no separation into two pipelines it required, one is sufficient. For that pipeline it has turned out that test_packages have quite some value to identify errors in packaging early, preventing unnecessary uploads to the Artifactory. That is why I prefer conan create. If I would go with conan install, the test packages would never be built.

Do you think it is possible to treat conan install and conan create as equal consumers of the graph-order, enabling a higher variety of (intended) CI designs? Although the --build-require helps, it feels a little hacky to me.

In any case, given the above link, I'd probably recommend conan install --requires=<ref> --build=<ref> as the conan graph build-order returns, focus on the overall CI process, and leave building from conan create for later.

I don't get the "leave building from conan create for later": if I already built everything with conan install, why should I go for a conan create later? That would result in building everything twice, wouldn't it?

[memsharded]

Steps 2 and 3 will be automated by the new conan workspace open command.

Note: the conan workspace open command is being released as dev-testing in Conan 2.10 today

Do you think it is possible to treat conan install and conan create as equal consumers of the graph-order, enabling a higher variety of (intended) CI designs? Although the --build-require helps, it feels a little hacky to me.

To clarify, a conan create is a higher layer that does:

• conan export .
• conan install --requires=pkg/version --build=pkg/version
• conan test test_package pkg/version

Which is slightly different to the conan install . over the current recipe, because that install the dependencies of the current pkg recipe, while the conan install --requires=pkg/version install that pkg/version and its dependencies.

I don't get the "leave building from conan create for later": if I already built everything with conan install, why should I go for a conan create later? That would result in building everything twice, wouldn't it?

When building all packages in a dependency graph that didn't get changes to their source code, but they need to be rebuilt because some of their dependencies did change, then the conan install --requires=pkg/version --build=pkg/version and conan create flows are absolutely identical, they will build exactly the same thing in the Conan cache. The only difference is that the conan create will also run after the test_package, but on the other hand it requires to do the clone+checkout of the specific repo and commit, which is not straightforward if your packages haven't captured the scm info inside the conandata.yml.

So what I mean didn't rebuild twice the same thing, I am just suggesting that implementing the CI pipeline with conan install ---requires=pkg/version --build=pkg/version is simpler to do, and it is always possible to replace that with a conan create or add a conan test step after to make sure to execute the test_package. But still the overall pipeline architecture and functionality is the same, it is just this very local detail about the running of the test_package.

Please let me know if this clarifies the questions.