Create environment from environment.yml

[matthewfeickert]

To build ontop of these images in an easier manner, it would be quite useful if the images were constructed from an environment.yml file that was copied into the image and then built from. So replacing

docker-coffea-base/base/Dockerfile

Line 11 in a879ea1

RUN mamba install --yes \

with

RUN mamba install --yes --file /environment.yml

This would also make it easier to be able to create a lock file for the environment with conda-lock and then to build on top of these images by being able to extend the environment.yml and conda-lock files as needed.

Is this in scope?

[oshadura]

@lgray do you have any preferences?

(for reference Matthew is working on the generic IRIS-HEP Analysis System docker image and we had an idea to use docker-coffea-base as a baseline image for it)

[nsmith-]

I am trying this now as part of making a coffea2023 image, and of course there is an issue in that any conda-installable pytorch conflicts with nomkl and if we pip-install it then we cannot at the same time pip-install torch-scatter, etc. since their setup seems to require the ability to import torch. Perhaps we just take the hit and start using Intel MKL?

[matthewfeickert]

I am trying this now as part of making a coffea2023 image, and of course there is an issue in that any conda-installable pytorch conflicts with nomkl and if we pip-install it then we cannot at the same time pip-install torch-scatter, etc. since their setup seems to require the ability to import torch. Perhaps we just take the hit and start using Intel MKL?

@nsmith- I don't know the specifics of why you do/don't want PyTorch with nomkl, but if you haven't already I would make an Issue with torch-scatter as requiring torch to be available at install time and not using a PEP 518 build-system to provide it at build time should be considered a bad bug these days. (edit: c.f. rusty1s/pytorch_scatter#265)

[nsmith-]

The original impetus was to reduce the image size, but now that our image has ballooned to 1.4GB, saving the 200MB for MKL seems less important. In https://github.com/CoffeaTeam/docker-coffea-base/blob/coffea2023/base/environment.yml I managed to build a first image and manually uploaded it to dockerhub.

[nsmith-]

FYI we're consolidating the images/environments in CoffeaTeam/af-images#1