Miniforge (aka mamba) buildpack support

[sgaist]

Hi,

paketo-buildpacks already provides support for conda environment through the miniconda and update-conda-env buildpacks however, the conda command currently suffers from slowdowns when it's resolving package dependencies (see their performance documentation page. While there are several techniques to speed things up, they are not always applicable.

As an alternative, mamba has been created with a different solver library that speed things up and is fully compatible with conda.

As a simple example, when one would like to build an image for pytorch following the instructions from their get started locally selector which results in conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia, the resolution time from conda is just not comparable to the one from mamba. The latter takes seconds to minutes while the former can take several hours. In this case, there is mostly none of the conda suggested optimizations but it's a pretty common scenario.

With that in mind, I would like to propose the addition of mamba support for the creation of conda environments using buildpacks.

There are mainly two ways that this could be done:

1. Install mamba as part of the base environment in the miniconda buildpack (mamba on top of miniconda is vehemently not recommended in the mamba documentation)
2. Create a miniforge buildpack which is more or less the same as the miniconda buildpack but using the miniforge installer

Beside following the recommendation of the mamba projects, the following is also a deal breaker for option 1: if in the process of building the environment, there's a new version of python installed, it will break the following command invocations as the package will not be known anymore.

In any case, for both options, conda-env-update would need to be modified so that either conda or mamba can be used through for example an environment variable. There might be better ways to do that but I don't know the intricacies of buildpacks well enough currently. Note that we could also have a mamba-env-update buildpack but it would just fragment the process as the main change here is really the command itself. The other conda-env-update changes required would be using the command in the log messages so they would be accurate rather than the hard coded conda.

Finally, the python-start detection should also allow the selection of conda VS mamba and create a build plan accordingly. That's maybe also something that could be passed as environment variable.

What do you guys think about it ?

[sgaist]

After some fiddling and discussions, a way simpler solution could be used: allow the miniconda buildpack user to switch to the mamba solver.

More details about how one does that to a conda installation can be found in this anaconda blog article.

On the plus side, this allow to keep everything as it is now, the only new thing required is a way to pass the solver desired to the build pack. AFAIK, it seems to be through the use of an environment variable.

With that in mind, I think the miniconda VS miniforge question is still valid although less important.

[sgaist]

An implementation proposition can be found at paketo-buildpacks/miniconda#480

[millsks]

This thread is a little older so this may not be news, but just as a heads up, as of conda 23.10.0 the conda-libmamba-solver is the default solver.

https://conda.org/blog/2023-11-06-conda-23-10-0-release/