containerized conda deployment with --containerize

[akhanf]

Now that we have #351, we can update our docker build to use --containerize

@Dhananjhay maybe you could explore this? (I haven't tried it out yet myself)..

[Dhananjhay]

@akhanf is it possbile to use the latest version of miniconda inside the hippunfold_deps autotop container? Right now, when I try to run hippunfold with --sdm apptainer conda flag, I get the following error:

error:CreateCondaEnvironmentException:
Conda must be version 24.7.1 or later, found version 4.10.3. Please update conda to the latest version. Note that you can also install conda into the snakemake environment without modifying your main conda installation.

This is because the docker image specified in the container directive of rules in Snakefile, i.e., config["singularity"]["autotop"] is used as a base image for the conda env docker image, which I've temporarily hosted on my custom Dockerhub repo docker pull dhananjhay/hippunfold_conda:2.0.0. If we were to update Conda version inside the base image, then I believe we might be able to run hippunfold with --sdm flag. I've been trying to change Conda's version in the autotop docker image but I can't seem to build it locally. Everytime I try it errors out with this trail:

#0 942.7 [ 88%] Building CXX object reg-apps/CMakeFiles/reg_tools.dir/reg_tools.cpp.o
#0 942.7 cd /opt/niftyreg-1.3.9/reg-apps && /usr/bin/c++ -D_BUILD_NR_DEV -D_USE_SSE -I/opt/niftyreg-1.3.9/src/reg-lib -I/opt/niftyreg-1.3.9/src/reg-io -I/opt/niftyreg-1.3.9/src/reg-io/nifti -I/opt/niftyreg-1.3.9/src/reg-io/png -I/opt/niftyreg-1.3.9 -I/opt/niftyreg-1.3.9/src/reg-io/nrrd -I/opt/niftyreg-1.3.9/src/reg-io/nrrd/NrrdIO -fopenmp -O3 -DNDEBUG   -fPIC -MD -MT reg-apps/CMakeFiles/reg_tools.dir/reg_tools.cpp.o -MF CMakeFiles/reg_tools.dir/reg_tools.cpp.o.d -o CMakeFiles/reg_tools.dir/reg_tools.cpp.o -c /opt/niftyreg-1.3.9/src/reg-apps/reg_tools.cpp
#0 946.7 /opt/niftyreg-1.3.9/src/reg-apps/reg_tools.cpp: In function ‘int main(int, char**)’:
#0 946.7 /opt/niftyreg-1.3.9/src/reg-apps/reg_tools.cpp:223:24: error: narrowing conversion of ‘3’ from ‘int’ to ‘bool’ [-Wnarrowing]
#0 946.7   223 |         bool boolX[8]={3,1,0,0,0,0,0,0};
#0 946.7       |                        ^
#0 946.7 /opt/niftyreg-1.3.9/src/reg-apps/reg_tools.cpp:225:24: error: narrowing conversion of ‘3’ from ‘int’ to ‘bool’ [-Wnarrowing]
#0 946.7   225 |         bool boolY[8]={3,0,1,0,0,0,0,0};
#0 946.7       |                        ^
#0 946.7 /opt/niftyreg-1.3.9/src/reg-apps/reg_tools.cpp:227:24: error: narrowing conversion of ‘3’ from ‘int’ to ‘bool’ [-Wnarrowing]
#0 946.7   227 |         bool boolZ[8]={3,0,0,1,0,0,0,0};
#0 946.7       |                        ^
#0 947.5 make[2]: *** [reg-apps/CMakeFiles/reg_tools.dir/build.make:76: reg-apps/CMakeFiles/reg_tools.dir/reg_tools.cpp.o] Error 1
#0 947.5 make[2]: Leaving directory '/opt/niftyreg-1.3.9'
#0 947.5 make[1]: *** [CMakeFiles/Makefile2:666: reg-apps/CMakeFiles/reg_tools.dir/all] Error 2
#0 947.5 make[1]: Leaving directory '/opt/niftyreg-1.3.9'
#0 947.5 make: *** [Makefile:156: all] Error 2

Now is this a MacOS specific issue I don't know, I'll dig around a little bit more and try to resolve this build issue.

[akhanf]

Ya I noticed the issue that we can't seem to use the hippunfold_deps singularity container when running hippunfold in conda, even if we're not enabling ---use-conda (which seems like a snakemake bug/limitation to me), but yeah it would be good to avoid this. I'm wondering if we should just deprecate the hippunfold_deps container in favour of the a new conda based one (eg with --containerize). As long as we would still be able to run it as a traditional bids app (ie single container with all envs included and entrypoint as run.py) then I don't see the need to keep using hippunfold_deps. What do you think?

[akhanf]

Also I think --sdm apptainer isn't any different from --use-singularity (or --use-apptainer), just updated cli naming scheme.

[akhanf]

Ya I noticed the issue that we can't seem to use the hippunfold_deps singularity container when running hippunfold in conda, even if we're not enabling ---use-conda (which seems like a snakemake bug/limitation to me),

Actually I think this is the expected behaviour when both conda and singularity directives are used in a rule (ie instead of using the dependencies as is in the container it makes a conda env with the container miniconda).

[Dhananjhay]

I'm wondering if we should just deprecate the hippunfold_deps container in favour of the a new conda based one (eg with --containerize). As long as we would still be able to run it as a traditional bids app (ie single container with all envs included and entrypoint as run.py) then I don't see the need to keep using hippunfold_deps. What do you think?

I don't know, I like the bifurcation between Singularity and Conda right now. I feel this approach provides a mutual safety net—if one platform encounters issues, we can fall back on the other. Deprecating hippunfold_deps might introduce some additional maintenance work, as the current Conda Dockerfile is quite minimalistic, focusing solely on environment setup. For example, the download_extract_template rule requires the unzip command-line tool, which isn't included in the Conda Dockerfile. But your idea about just using --containerize Dockerfile seems interesting and I'd like to give it a shot!

Additionally, I’m curious about the level of priority package managers place on updating software versions on Conda-Forge. Is it treated as an afterthought, or do they actively maintain it? This could impact the long-term viability of a purely Conda-based approach.