[question] conan 2.12.0 KeyError on python-require

[jwatson0]

Describe the bug

This just started happening in our CI builds as soon as we picked up conan 2.12.0. Reverting to 2.11.0 resolves the problem.
I haven't narrowed it down yet, but I wanted to get this report up quickly. Not sure yet if this is a regression or a known breaking change.

We have a somewhat complex conan config that uses a python-require to load a custom generator for our lbstanza language compiler.
The conan dependency is slm -> lbstanzagenerator_pyreq

This is the error we're getting with 2.12.0:

ERROR: Package 'slm/0.6.25' not resolved: slm/0.6.25: Cannot load recipe.
Error loading conanfile at '/home/user/client/.conan2/p/slm021eaf83fc180/e/conanfile.py': Error loading conanfile at '/home/user/client/.conan2/p/lbsta3c7344737aed1/e/conanfile.py': Unable to load conanfile in /home/user/client/.conan2/p/lbsta3c7344737aed1/e/conanfile.py
                 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: lbstanzagenerator_pyreq/0.6.18

lbstanzagenerator_pyreq conanfile.py:

from conan import ConanFile
from conan.tools.files import save
from conans.model.conanfile_interface import ConanFileInterface
from conans.model.pkg_type import PackageType
from conans.model.build_info import _Component
from io import TextIOWrapper
from pathlib import Path

class LBStanzaGeneratorPyReq(ConanFile):
    name = "lbstanzagenerator_pyreq"
    version = "0.6.18"
    package_type = "python-require"

# LBStanza Generator class
class LBStanzaGenerator:
   # ... snip

slm conanfile.py

class ConanSlmPackage(ConanFile):
  package_type = "application"
  python_requires = "lbstanzagenerator_pyreq/[>=0.6.18 <0.7.0]"

conan list:

  lbstanzagenerator_pyreq
    lbstanzagenerator_pyreq/0.6.18
  slm
    slm/0.6.25

These conan packages are in a private artifactory, not on conan-center.

I'll update this when I narrow down the issue, but let me know if this looks relevant to a recent change.

How to reproduce it

conan install . -pr:b linux-x86_64-default -pr:h linux-x86_64-gcc-11 --output-folder build --build never

[memsharded]

Hi @jwatson0

Thanks for your report.

It seems your code is using some non-public, non-documented private imports in:

from conans.model.conanfile_interface import ConanFileInterface
from conans.model.pkg_type import PackageType
from conans.model.build_info import _Component

These imports have been moved in Conan 2.12 to the new conan.internal namespace, meaning that they can't be used from there either, but as part of the process of restructuring the project and getting rid of the old conans folder, which dates to some package issues with Python 2.7.

It is likely that some import like PackageType might be at some point documented publicly, we are doing an effort to stabilize the public API, but others like _Component are even private for Conan codebase itself, so they are extremely unlikely they will be exposed publicly any time.

If you want to share your usages of those things in the code, I might try to help to guide how it would be possible to get rid of them.

[jwatson0]

Ah, yes. I went rummaging through the python datastructures looking for the data that I knew was in there, but I couldn't find through the public interface. I would very much like to update this to the public interfaces, and hopefully the data is available in the restructured APIs.

Background:
We have a non-C language compiler [lbstanza.org] that links with conan packages and uses gcc for the final link step. The compiler has its own build system and config file that needs the gcc include and link flags for the libraries that it links with. We used to use the conanbuildinfo.args in conan v1, but that doesn't seem to be available in conan v2, so I wrote a custom generator to output the data in the lbstanza config file format. We basically want to allow a user to link an arbitrary conan package library into their stanza build without having conan or cmake at build-time.

The conan generator needs to support the following:

• compiling both application binaries, static libs, and shared libs
• list conan cache paths to each lib for linking at build time
• detect whether the dependency is static, shared, or header-only
• both absolute path and relative path for different build use cases
• preserve lib linking order (i.e. libssl first, then libcrypto)
• support linux, mac, windows, (i.e. - .a / .so / .dylib / .dll)
• runtime shared linker search path hints in output lib where possible
  • "-Wl,-rpath,$ORIGIN/deps/curl/lib"

The generator source is available here:

• https://github.com/StanzaOrg/slm/blob/master/slm_builder/conan_lbstanza_generator/conanfile.py

The steps that use the private python interfaces are basically this:

• for each direct and transitive conan dependency:
  • Get a list of include paths, library paths, and library names:
    • conans.model.conanfile_interface.ConanFileInterface.cpp_info.get_sorted_components() -> .libs
  • Detect whether the library is static or shared:
    • PackageType.SHARED
  • And I think this listed similar data for conan packages that were separated into components?
    • conans.model.build_info._Component.libs

The output is similar to this:

• dependency: libcurl -> openssl -> zlib

• example: libcurl/8.10.1 static linux
  -I .conan2/p/libcu1c64887adb924/p/include
.conan2/p/libcu1c64887adb924/p/lib/libcurl.a
-I .conan2/p/opens67cc4fdb55110/p/include
.conan2/p/opens67cc4fdb55110/p/lib/libssl.a
.conan2/p/opens67cc4fdb55110/p/lib/libcrypto.a
-I .conan2/p/zlibda21a13ecea3f/p/include
.conan2/p/zlibda21a13ecea3f/p/lib/libz.a

• example: libcurl/8.10.1 shared linux
  -I .conan2/p/libcu1c64887adb924/p/include
-L .conan2/p/libcu1c64887adb924/p/lib -l curl
-I .conan2/p/opens67cc4fdb55110/p/include
-L .conan2/p/opens67cc4fdb55110/p/lib -l ssl -l crypto
-I .conan2/p/zlibda21a13ecea3f/p/include
-L .conan2/p/zlibda21a13ecea3f/p/lib -l z

We also experimented with a custom deployer, but that didn't work as well for us.

I haven't yet looked for any interfaces newly available in conan 2.12.0, and I may have overlooked some obvious existing interface. The doc on custom generators got me started, but I think the detailed data that I needed was missing from the dependency output.

Do you think this data is available now in a public interface?

Thanks,
Jason

[memsharded]

We have made this recent release a good effort to keep pushing the public API, and we have started to move things and design to make parts of the model public. The first step was the Remote, see https://docs.conan.io/2/reference/extensions/python_api/RemotesAPI.html#conan.api.model.Remote, I know it is small, but at least something to start with. The work to be done here is huge, and resources limited, so it might take a while. But it is at least moving forward and we will keep doing efforts in every release to keep documenting and stabilizing the API.

We have also added a warning for next release to warn users against internal imports and encourage to submit tickets with the use cases to analyze if it makes sense to make them part of the public API.

As a recommendation, I'd suggest to drop the types such as depinst: ConanFileInterface. The ConanFileInterface was not really used here for instantiating objects or anything else, just as a Python typing hint, so it can be dropped. I know it is not as convenient for development, but it is worth the gain in stability, less potential of breaking things.

[memsharded]

So, in short, no, those classes are not public in the API yet.
I'd suggest:

• For things like PackageType.SHARED, you can safely replace them for the public documented alternative, the recipe package_type = "shared-library" is guaranteed to not break, so you can use "shared-library" safely, and dinst.package_type == "shared-library" is also error-checked, as the left side is PackageType.
• _Component is also used only for typing, can be safely dropped from the code.

[jwatson0]

Thanks, I'll update to use that.