fixing versions for dependencies

[dkazanc]

As we do releases more often now, I think we need to be more careful with what we install in our conda environment. For example, with a recent issue of Astra toolbox moving to conda-forge and dropping support of the older cudatoolkit versions there was an error in the environment installed by default.

To be on the safe side, we probably need to fix all the packages in our environment so that the environment will be reproduced exactly for the future releases? It is convenient that we install Htttomo + libraries through pip so that the packages in already created environment will not be changed. We should generate a list of explicit packages from the working environment and reuse it until we actually need to update some of the packages in that environment.
And the main environment file should also have the main packages versions fixed as well, e.g.:

  - astra-toolbox == 2.1.1
  - python == 1.25
  - numpy == 1.25
  - cupy==12.3.0
  - cudatoolkit==11.2

I hope that this will reduce the chance of breaking things with our local installations especially.

[yousefmoazzam]

My preferred approach to defining dependencies is to put version constraints where it's needed only, and let the dependency resolver figure out the rest. The more version constraints there are, the more fragile the installation will be (any incompatibility that is a consequence of manually specified version constraints will be rejected by the dependency resolver).

For example, in the astra case, I recall from slack that first there was an attempt to manually specify the version and build of libastra via:

- libastra[version='==2.2.0', build=*cuda11.8h658bec2*]

and that resulted in a "Could not solve environment specs" error from the dependency resolver.

We then saw that relaxing the constraint to simply ask for a "cuda" build version of astra-toolbox via the following:

- astra-toolbox[build='*cuda*']

was all that was needed, we didn't need to say anything about the astra-toolbox version.

If it were the case that using some astra-toolbox version (which was a "cuda" build) didn't work with httomo, then I'd advocate putting a version constraint. But we only needed a build constraint to make mamba's dependency resolver happy, so I don't see a reason to fix the astra-toolbox version.

Applying similar reasoning to other packages mentioned in the original post, we know that a cupy version > 12.3.0 causes issues for httomo, so constraining the version to be <= 12.3.0 is the most sensible, in my opinion (rather than fixing it at specifically 12.3.0). For numpy, whatever the latest version httomo and co. works with, do a similar thing, rather than fixing a specific version.

I think an argument could be made to fix the cudatoolkit version to a specific version, if the cudatoolkit version had a significant impact on httomo behaviour (such as memory estimation needing to change between cudatoolkit versions, due to CuFFT being updated and using different amounts of memory for an FFT or something). But this is something that is yet to be shown, so I wouldn't yet advocate for this either.

It's up to you, but I don't think it's that useful to specify a single version for dependencies where there's no known compatibility issue between the version of that dependency and httomo, and it makes the installation more fragile.

As for generating explicit dependency version lists for existing conda envs being used in production for reproducibility purposes, yeah, this seems reasonable to me.