Allow explicitly building without mpi4py?

[AlexanderRichert-NOAA]

Version: 1.5.8
System: A Cray with SLES 15.3, but I expect this problem will occur on any system
Description of problem: The netcdf4-python installer auto-detects mpi4py by trying to load the module, and will build with it if available. This is a problem for us because we are trying to install through Spack, which requires having some Python version loaded, and our Python installation happens to include mpi4py but we do not want netcdf4-python to use it (partly because we don't need it, and partly because if we wanted it, we would want it to point to an mpi4py copy inside our Spack-based installation, not the Python installation that's only used at build time to run Spack). We are working on finding a way to modify our $PYTHONPATH to work around this issue, but would it be possible to have with/without mpi4py as an explicitly configurable build option as opposed to the current auto-detection scheme?

[jswhit]

Actually, setup.py looks for #define NC_HAS_PARALLEL in netcdf_meta.h to determine whether to add parallel support to the build. If it finds it, it then imports mpi4py to find out where the mpi4py header files are. So, if you configure the spack-stack netcdf C library to not have parallel support then netcdf4-python won't have it either.

[AlexanderRichert-NOAA]

Thanks for the info. I think what makes our case a bit unusual is that we are compiling NetCDF with MPI support, but the Python module without it. It sounds like maybe the Spack recipe should automatically determine MPI support based on whether NetCDF-C was built with it rather than having it as an independently configurable option.

[jswhit]

it would be relatively simple to disable parallel support by patching the setup.py file for netcdf4-python, if you wanted to apply this diff in the spack recipe

--- a/setup.py
+++ b/setup.py
@@ -392,7 +392,8 @@ if 'sdist' not in sys.argv[1:] and 'clean' not in sys.argv[1:] and '--version' n
        (netcdf_lib_version > "4.4" and netcdf_lib_version < "4.5"):
         has_cdf5_format = True

-    has_parallel_support = check_has_parallel_support(inc_dirs)
+    #has_parallel_support = check_has_parallel_support(inc_dirs)
+    has_parallel_support = False
     has_has_not = "has" if has_parallel_support else "does not have"
     print(f"netcdf lib {has_has_not} parallel functions")

[climbfuji]

@AlexanderRichert-NOAA FWIW, I ended up applying this patch to force building without parallel features, because I ran into this problem: nanoporetech/ont_fast5_api#21 (with py-netcdf4/1.7.1-post2 and mpi4py 4.0.1). I think we need to apply the patch when ~mpi is set for py-netcdf4 (and we need to change our variant request in common packages.yaml).

@jswhit auto-detecting and guessing what user wants is more often than not not the best idea. There should be an explicit flag for py-netcdf4 that the user can set. If they want parallel capabilities, then check that everything is built with those options. Otherwise, don't care and build w/o parallel features.