Skip to content

Commit

Permalink
enhanced README
Browse files Browse the repository at this point in the history
  • Loading branch information
lukas.gehrig committed Oct 23, 2023
1 parent 4e19b7b commit 1bb832e
Showing 1 changed file with 13 additions and 1 deletion.
14 changes: 13 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,22 @@ Errors in the build-string can occur if they have an `-` (see [Build number and
## About MPI
**General**: Currently, there are three different options on how a conda-wheel can be build with mpi. It's either "nompi", "openmpi" or "mpich". It seems like other packages like [h5py](https://anaconda.org/conda-forge/h5py/files) include this in their build-string and so should we. This makes it clear to others which packages use mpi and which mpi implementation they were built with by using `conda list | grep mpi`. The setup of the github workflow and the conda-recipe should be build accordingly so that once you build a wheel, you just have to care about setting `MPI` and `MPI_VERSION` in `build_base.yml` accordingly. So, we just have to care about that the according mpi-wheels of all Karabo-dependencies are avilable at [anaconda.org](https://anaconda.org/) (whether it's our channel or not) and that they're build with the same MPI (openmpi or mpich). Be aware that openmpi nor mpich is windows-compatible. But since we don't support windows you shouldn't have to worry too much.

**Installation of MPI-wheels**: Installation is fairly straight-forward. If there are mpi-wheels and no-mpi-wheels available, the no-mpi-wheels are chosen per default. If installing an mpi-build is needed (and let the dependencies figure out whether it is mpich or openmpi), just install it like the following: `conda install -c conda-forge <package>=*=mpi*`. If you want to be sure that it's mpich or openmpi, just install it with the very same installation instruction by appending `openmpi` or `mpich` at the end.
**Installation of MPI-wheels**: Installation is fairly straight-forward. If there are mpi-wheels and no-mpi-wheels available, the no-mpi-wheels are chosen per default. If installing an mpi-build is needed (and let the dependencies figure out whether it is mpich or openmpi), just install it like the following: `conda install -c conda-forge <package>=*=mpi*`. If you want to be sure that it's mpich or openmpi, just install it with the very same installation instruction by appending `openmpi` or `mpich` at the end. Note that there are tools available which are based on mpi, but are not specific to it's implementation like `dask-mpi` or `mpi4py`. These tools just rely just that there is any mpi installed and therefore do not have a specific mpi-wheel.

**Using Native MPI**: There are cases where using an MPI compiled on the target architecture is needed. We don't have to consider this use-case for the feedstock since there is a solution according to [conda-forge docs](https://conda-forge.org/docs/user/tipsandtricks.html). In this use-casee, the installer has to link the mpi of the environment to an MPI available at a standard-location as follows: `conda install "mpich=x.y.z=external_*"` or `conda install "openmpi=x.y.z=external_*"`.

**MPI Implementation Support**: Since Karabo `v0.20.0` we used to support openmpi-builds. Because mpich is more suited for hpc-environments (like cscs only supports mpich), we decided to switch the mpi-dependent builds to mpich from `v0.21.0` onwards. However, in theory we could support nompi, openmpi and mpich-based builds. This could easily be achieved by putting these three variables in the according `conda_build_config.yaml` and zip them with `python`. However, because we have limited resources for setup & testing, this is not implemented yet. Be aware that the following dependencies have mpi-wheels which need to be referenced accordingly if they're in a `meta.yaml`. Please keep them updated.

External dependencies & which mpi-wheels they provide:
- [h5py](https://anaconda.org/conda-forge/h5py/files): nompi, openmpi, mpich
- [hdf5](https://anaconda.org/conda-forge/hdf5/files): nompi, openmpi, mpich (is a dep of `h5py`)
- [casacore](https://anaconda.org/conda-forge/casacore/files): nompi, openmpi

Feedstock Dependencies:
- [fftw3](https://github.com/i4Ds/Karabo-Feedstock/blob/main/fftw3/meta.yaml): mpich
- [pfft](https://github.com/i4Ds/Karabo-Feedstock/blob/main/pfft/meta.yaml): mpich
- [pinocchio](https://github.com/i4Ds/Karabo-Feedstock/blob/main/pinocchio/meta.yaml): mpich

## Feedstock Build Dependencies
**Active builds**: In case you intend to create a new build which also needs underlying changes of other builds, it's useful to know all feedstock-dependencies to know which one(s) to build first. The following provides an overview of the dependencies. In the brackets are the transversal dependencies if they're not already present as direct dependency. Please keep the list updated so that only karabo-latest builds are listed and that they have the correct direct and transversal dependencies:
- aotools: -
Expand Down

0 comments on commit 1bb832e

Please sign in to comment.