Cokurt GenTen Pele

Build instructions

Currently, we are building the GenTen libs and VTK-m, and building VTK-m cannot be build on the login node (you will run out of memory). Thus, you need to build the dependencies (via the python command below) using a batch job or interactive node. To get an interactive node use this command:

bsub -XF -W 120 -nnodes 1 -P CSC340 -Is /bin/bash

When the allocation is granted, you will have 120 minute to complete the build, and as of the time of this writing (9/28/2021), it will be enough time to build vtkm. You will only have to build the dependencies once, and any changes to ascent can be compiled on the login node (with the correct modules loaded).

ml cuda/11.0.2 
ml gcc/9.1.0
ml cmake/3.20.2
git clone --recursive https://github.com/Alpine-DAV/ascent.git
cd ascent
git checkout genten
python3 scripts/uberenv/uberenv.py --spack-config-dir=./scripts/uberenv_configs/spack_configs/configs/olcf/summit --spec="+genten ^conduit@develop ^vtk-h@develop"
mkdir build
mkdir install
cd build 
cmake -C ../uberenv_libs/[email protected] ../src/ -DCMAKE_INSTALL_PREFIX=path_to_install
make install
cd ../..

Note: the cmake file in uberenv_libs will be named differently.

Building Pele with a Make-based build system

Neither GenTen nor Kokkos natively integrate with a Make build system. While Ascent provides a path for this, we are not currently maintaining a path for this in the file:

ascent_install_path/share/ascent/ascent_config.mk

To add the correct lines to link against Kokkos and GenTen (and its cublas deps) the several libraries need to be added to the ASCENT_VTKM_LIB_FLAGS variable.

    ASCENT_VTKM_LIB_FLAGS = -L $(ASCENT_VTKM_DIR)/lib \
                            -lvtkm_rendering-$(ASCENT_VTKM_VERSION) \
                            -lvtkm_filter_contour-$(ASCENT_VTKM_VERSION) \
                            -lvtkm_filter_gradient-$(ASCENT_VTKM_VERSION) \
                            -lvtkm_filter_extra-$(ASCENT_VTKM_VERSION) \
                            -lvtkm_filter_common-$(ASCENT_VTKM_VERSION) \
                            -lvtkm_worklet-$(ASCENT_VTKM_VERSION) \
                            -lvtkm_source-$(ASCENT_VTKM_VERSION) \
                            -lvtkm_io-$(ASCENT_VTKM_VERSION) \
                            -lvtkm_lodepng-$(ASCENT_VTKM_VERSION) \
                            -lvtkm_cont-$(ASCENT_VTKM_VERSION) \
                            -lvtkmdiympi_nompi \
														/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgt_higher_moments.a /gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgentenlib.a /gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgenten_mathlibs_c.a /gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/kokkos-3.4.00-b3gns6simasdq7qtmp5fnnv6le7dzwpn/lib64/libkokkoscontainers.a /gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/kokkos-3.4.00-b3gns6simasdq7qtmp5fnnv6le7dzwpn/lib64/libkokkoscore.a

Above you have to scroll to the right to see the extra libs but I'll list them here:

/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgt_higher_moments.a
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgentenlib.a
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgenten_mathlibs_c.a 
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/kokkos-3.4.00-b3gns6simasdq7qtmp5fnnv6le7dzwpn/lib64/libkokkoscontainers.a 
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/kokkos-3.4.00-b3gns6simasdq7qtmp5fnnv6le7dzwpn/lib64/libkokkoscore.a

More concisely, these are the libs in linking order:

libgt_higher_moments.a
libgentenlib.a
libgenten_mathlibs_c.a 
libkokkoscontainers.a 
libkokkoscore.a

Your paths will be different, but if you followed the instructions above, then they will be located inside the uberenv_libs directory in a path similar to uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/

IMPORTANT NOTE: each time you make install ascent this config file will be overwritten as part of the installation process, but its contents will not change unless you rebuilt the dependencies. Once you make the changes to the file, keep a copy around so you don't have to grep for the libs again.