What is the recommend way to install a conda dependency in a poetry env?

[ThibaultGROUEIX]

• [X ] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

[finswimmer]

Hello @ThibaultGROUEIX,

you post is lacking a lot of information. What are you trying to do and why? What have you done so far und what problems/questions arises on that way?

fin swimmer

[ThibaultGROUEIX]

Hi @finswimmer ,
thanks for getting back to me!
I am trying to install the package IGL in my poetry environment. https://anaconda.org/conda-forge/igl
Here is my pyproject:

[tool.poetry]
name = "njf"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.9"

[tool.poetry.dev-dependencies]
pytest = "^7.0.1"
black = "^22.1.0"
tox = "^3.24.5"
poethepoet = "^0.12.3"
awscli = "1.17.0"

[tool.black]
line-length = 140

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

I have tried poetry add igl, but this fails because IGL is not hosted on pypi but on anaconda.
Thanks a lot!

[ThibaultGROUEIX]

Hi @finswimmer ,
I would be very helpful to have some guidelines regarding this question. It does not seem documented anywhere.
Thanks for your help,
Thibault

[finswimmer]

Hello @ThibaultGROUEIX,

there is no way to install conda packages via poetry. The conda ecosystem has its own way how packages are build and the metadata are provided, also the repositories work differently.

In #5138 (comment) @volkerjaenisch mentioned that he is working on an enhancement. I don't know how far he get it until now and if it's even possible at all.

fin swimmer

[volkerjaenisch]

@ThibaultGROUEIX

It is possible. I wrote some prototypish code that is capable of transforming a given anaconda installation and a given conda env into Packages on a local PyPI-Server. Then I did a (not really intrusive 100line) patch on Poetry that enables Poetry to install the Conda packages (utilizing the local PyPI).

Cool, eh! But here they come - the many gory details and hacks.

An Anconda-Installation consists not only of Python-Packages (PP) but also of Non-Python-Packages (NPP) (C-Libs, Fortran-Libs, etc.). These NPP unfortunately cannot be installed via a python-Package (Wheel)[Wheels miss the necessary hooks]. Therefore the packages (FakePythonPackages (FPP)) on the local PyPI-Server look like a real wheels but they are not transformed conda-Packages! They are real Python-Wheel packages, but they are empty of code and reflect only dependencies of the associated Conda-Packages (The interdependence of the PPs as well as that between PPs and NPPs and NPPs to NPPs) and include all the metadata (two JSON files from each Conda-Package) to find and install the associated Conda-PP or Conda-NPP.
The patched Poetry knows what a FPP looks like (The JSON-Files are a unique hint) and treats them differently to PPs invoking code that installs the Conda-package linked at from the FPP package mimicking the Conda mechanism of hardlinking the files in the Conda-Package to site-packages (and some other locations).

It works at first glance, but it is not really tested. I did this research/prototyping work for a really nice customer and there is a good chance that they may allow to make it open-source. But opening the source is only one part but bringing the code to production is another big junk of work. This effort is only reasonable if there is real interest of the community for such a hacked, complex solution. Also this work is based on a lot of reverse engineering Anconda/Conda. If Anaconda changes its packaging strategy anything may vanish in thin air.

Cheers,
Volker