issm: install/benchmarking on gadi

[fmccormack]

I'm after an install of the Ice-sheet and Sea-level System Model (ISSM) on gadi.

ISSM is open source. Standard install requires autotools, cmake, and petsc, as well as triangle, chaco and m1qn3. Basic install instructions can be found in the user guide (https://issm.jpl.nasa.gov/files/userguide.pdf), and examples of configure scripts for different clusters can be found on the wiki (https://issm.ess.uci.edu/trac/issm/wiki). We wouldn't use the python or matlab wrappers (with-wrappers=no in configure script) when running on gadi.

I've had ISSM running in the past on gadi, but with more recent versions of ISSM I've had issues. A recent version of ISSM didn't compile with the system petsc modules, so I installed a local version of petsc. With this configuration the job would run, but occasionally no output would be written and there were no error messages. My postdoc Johanna was able to get the most recent version of ISSM installed with a system module of petsc (see attached "install_issm" script; you'll also need to export the variable $ISSM_DIR to be the install directory), but I haven't tested this build thoroughly. I'd be keen for advice on appropriate optimisation flags with this.

If the Underworld folks have advice on petsc builds and scaling that'd be great.

There are many different solver options for running ISSM (see examples in the attached paper), which are configured at the user end. Once we get an optimised build, I'd be very keen to perform benchmarking tests for parallel performance.

install_issm.txt
environment.txt
gmd-10-155-2017.pdf

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/ice-sheet-model-install-and-benchmarking-on-gadi/912/4

[aidanheerdegen]

The result of 5 minutes of googling:

The Parallel Ice Sheet Model (PISM) has a spack package and also relies on PETSc:

https://www.pism.io/docs/installation/spack.html

Firedrake has a spack package

https://github.com/firedrakeproject/firedrake-spack

and also has PETSc as a dependency. @angus-g have you tried a firedrake spack install?

Also note there are a bunch of open PETSc issues, most of which are probably not relevant, but worth looking through, or searching if/when there are "snags".

Not having to support python wrappers, at least in the first instance, makes building a lot simpler.

[aidanheerdegen]

The Center for Efficient Exascale Discretizations (CEED) project utilises spack and includes PETSc:

https://ceed.exascaleproject.org/ceed-2.0/

And have spack configurations for eight different platforms, which is also of interest in seeing what is required in "real-world" scenarios to get stuff working.

[fmccormack]

Thanks for that info @aidanheerdegen.

How do we go about building ISSM using spack packages (i.e. petsc, triangle, m1qn3 and chaco)? Do you want me to do that or what would you like me to do?

[fmccormack]

Hi @aidanheerdegen - just following up on my previous message. Are you able to provide guidance about where to install ISSM and how you'd like that to proceed? Thanks heaps

[fmccormack]

Hi again @aidanheerdegen, we're planning to submit a group NCMAS that will involve ISSM modelling. I'd like to have some benchmarking done well in advance of that, if possible. Is there any updates on getting ISSM installed using spack?

[aidanheerdegen]

Sorry for the lack of a response @fmccormack. Can we catch up at the Workshop? I've roped in @angus-g to see if he can get the PetSc spack build working on gadi, as this is something he needs for his work too. Ain't collaboration grand!

[angus-g]

Alright, I have think I've got things mostly working! Here's a summary of what I've done.

Spack configuration

I started off with a regular install of Spack, and Micael's COSIMA version of the Spack configuration files (https://github.com/COSIMA/spack-config/tree/main/config/system). I ended up making some tweaks to these files to suit my development needs. The whole reference repository is here: https://github.com/angus-g/gadi-spack/tree/master/config

I think the most important and relevant changes were in packages.yaml:

1. I removed [email protected], because the spec [email protected] would pick up that version, and it would refuse to link correctly.
2. I modified [email protected], to include the +cluster mpi_family=openmpi variants. This means that the system package is recognised as having BLACS and ScaLAPACK included, which is necessary for PETSc later on.

Intel oneAPI package modification

Next up is the issue of the MKL installation on Gadi being different to the paths expected by Spack (ACCESS-NRI/spack-config#1). For this, I just defined a package that inherits from the builtin IntelOneapiMkl package, but specifies a different component path property (https://github.com/angus-g/gadi-spack/blob/master/gadi/packages/intel-oneapi-mkl/package.py). We can use this as a dependency for PETSc.

PETSc build

This was the starting point for most of this. The spec I ended up building was:

[email protected] %[email protected] +mumps +scalapack ^[email protected] ^intel-oneapi-mkl+scalapack+cluster mpi_family=openmpi

This builds with the latest Intel compiler and MKL, includes MUMPS and ScaLAPACK in PETSc (we also get METIS and SuperLU_DIST as default variants on the PETSc package). Referring to the configure lines @fmccormack attached in the first post, maybe we also need Chaco and ParMETIS, at least for ISSM in general. We can build PETSc with +parmetis, and there is a chaco package we could bring in as a dependency too.

ISSM package

Just to demonstrate how it's done, I added a simple ISSM package to my Spack repository (https://github.com/angus-g/gadi-spack/blob/master/gadi/packages/issm/package.py). This includes modified m1qn3 and triangle packages. This was mostly based on the "system petsc build" configure invocation (below), but it's very simple to add other variants/configure parameters.

Configure invocation

## System petsc build
./configure \
    --prefix=$ISSM_DIR \
            --with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \
            --with-mpi-include="$OPENMPI_BASE/include"  \
            --with-petsc-dir="$PETSC_ROOT" \
            --with-metis-dir="$PETSC_ROOT" \
            --with-mumps-dir="$PETSC_ROOT" \
            --with-scalapack-lib="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" \
            --with-mkl-libflags="-L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm" \
            --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \
            --with-wrappers=no \
            --enable-debugging \
            --enable-development \
            --enable-shared \
            --without-kriging \
            --with-numthreads=16  CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif90

I don't have any way of testing the build, but it did at least complete successfully and build an executable.

[micaeljtoliveira]

@angus-g Nice work! I've used it to fix the MKL support in the COSIMA Spack instance.

[aidanheerdegen]

This is awesome @angus-g, thanks heaps.

[fmccormack]

Thanks @angus-g - I'll have a look at this tomorrow and let you know how it goes!

[aidanheerdegen]

Awesome work @angus-g, thanks again. This is merged, so the next step is some instructions on how to set up a working spack configuration for @fmccormack and collaborators.

There is an existing topic on the forum about an attempt I made to set up a shared spack configuration. It hasn't had any attention in some time, and needs to be reconfigured at the least, and probably redesigned. This is one of the things we've got on our list but it is a long list.

I don't think this is necessarily how I'd suggest Felicity utilise spack, but it has a fair bit of useful information about what is required to configure spack for use at NCI.

[micaeljtoliveira]

@aidanheerdegen Happy to install this in the COSIMA spack instance on gadi to make it available to the community.

[aidanheerdegen]

Happy to install this in the COSIMA spack instance on gadi to make it available to the community.

That is a very nice offer @micaeljtoliveira, and I'll leave it up to @fmccormack to say definitively yes or no, but I believe the intention was to do a number of compilations with different options/dependencies and do some benchmarking so it sounds like they'd like to do the building/installation themselves.

Now it may be that how you've configured the COSIMA spack instance would be a good model to follow, indeed ACCESS-NRI may well do just that! :)