Building on SCOREC RHEL7 with Cabana

The instructions in the 'setup' section should be run once to get the stack of software built. After that, the instructions in the 'develop' section should be followed to rebuild pumipic after making source code changes.

CUDA flags for the NVIDIA Turing GPUs are included below. The scorec workstations cranium and blockade have Turing GPUs.

setup

Create a directory to work from. It will contain all source code and build directories.

mkdir pumipicDev #this can be any name - just be consistent
cd pumipicDev

Create an environment script envRhel7_turing.sh with the following contents:

export root=$PWD 
module use /opt/scorec/spack/dev/lmod/linux-rhel7-x86_64/Core
module unuse /opt/scorec/spack/lmod/linux-rhel7-x86_64/Core
module load gcc/7.4.0-c5aaloy cuda/10.2
module load mpich/3.3.1-bfezl2l
module load cmake

function getname() {
  name=$1
  machine=`hostname -s`
  buildSuffix=${machine}-cuda
  echo "build-${name}-${buildSuffix}"
}
export engpar=$root/`getname engpar`/install # This is where engpar will be (or is) installed
export kk=$root/`getname kokkos`/install   # This is where kokkos will be (or is) installed
export oh=$root/`getname omegah`/install  # This is where omega_h will be (or is) installed
export cab=$root/`getname cabana`/install # This is where cabana will be (or is) installed
export pumipic=$root/`getname pumipic`/install # This is where PumiPIC will be (or is) installed
export CMAKE_PREFIX_PATH=$engpar:$kk:$kk/lib64/cmake:$oh:$cab:$pumipic:$CMAKE_PREFIX_PATH
export MPICH_CXX=$root/kokkos/bin/nvcc_wrapper

cm=`which cmake`
echo "cmake: $cm"
echo "kokkos install dir: $kk"

Create a file named buildAll_turing.sh with the following contents:

#!/bin/bash -e

#kokkos
cd $root
git clone [email protected]:kokkos/kokkos.git
cd kokkos
git checkout 3.1.00
cd -
mkdir -p $kk
cd $_/..
cmake ../kokkos \
  -DCMAKE_CXX_COMPILER=$root/kokkos/bin/nvcc_wrapper \
  -DKokkos_ARCH_TURING75=ON \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ENABLE_OPENMP=off \
  -DKokkos_ENABLE_CUDA=on \
  -DKokkos_ENABLE_CUDA_LAMBDA=on \
  -DKokkos_ENABLE_DEBUG=on \
  -DKokkos_ENABLE_PROFILING=on \
  -DCMAKE_INSTALL_PREFIX=$PWD/install
make -j 24 install

#engpar
cd $root
git clone [email protected]:SCOREC/EnGPar.git
cd EnGPar
cd -
mkdir -p $engpar
cd $_/..
cmake ../EnGPar \
  -DCMAKE_INSTALL_PREFIX=$engpar \
  -DCMAKE_C_COMPILER="mpicc" \
  -DCMAKE_CXX_COMPILER="mpicxx" \
  -DCMAKE_CXX_FLAGS="-std=c++11" \
  -DENABLE_PARMETIS=OFF \
  -DENABLE_PUMI=OFF \
  -DIS_TESTING=OFF
make install -j8 

#omegah
cd $root
git clone [email protected]:SCOREC/omega_h.git
cd omega_h
[ -d $oh ] && rm -rf ${oh%%install}
mkdir -p $oh 
cd ${oh%%install}
cmake ../omega_h \
  -DCMAKE_INSTALL_PREFIX=$oh \
  -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_Kokkos=ON \
  -DOmega_h_USE_CUDA=on \
  -DOmega_h_CUDA_ARCH=75 \
  -DOmega_h_USE_MPI=on  \
  -DBUILD_TESTING=on  \
  -DCMAKE_CXX_COMPILER=`which mpicxx` \
  -DKokkos_PREFIX=$kk/lib64/cmake
make VERBOSE=1 -j8 install
ctest -E warp_test_parallel # see https://github.com/SCOREC/pumi-pic/pull/65#issuecomment-824335130

#cabana
cd $root
git clone [email protected]:ECP-copa/Cabana.git cabana
cd cabana
git checkout 0.3.0
[ -d $cab ] && rm -rf ${cab%%install}
mkdir -p $cab
cd ${cab%%install}
cmake ../cabana \
  -DCMAKE_BUILD_TYPE="Debug" \
  -DCMAKE_CXX_COMPILER=$root/kokkos/bin/nvcc_wrapper \
  -DCabana_ENABLE_TESTING=ON \
  -DCabana_ENABLE_EXAMPLES=ON \
  -DCabana_ENABLE_Cuda=ON \
  -DCMAKE_INSTALL_PREFIX=$cab
make -j 24 install
ctest

#pumipic
cd $root
git clone [email protected]:SCOREC/pumi-pic.git
cd pumi-pic
git submodule init 
git submodule update # download test data
cd -
[ -d $pumipic ] && rm -rf ${pumipic%%install}
mkdir -p $pumipic
cd ${pumipic%%install}
echo $CMAKE_PREFIX_PATH
cmake ../pumi-pic/  \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DIS_TESTING=ON \
  -DPS_IS_TESTING=ON \
  -DENABLE_CABANA=ON \
  -DTEST_DATA_DIR=../pumi-pic/pumipic-data \
  -DOmega_h_PREFIX=$oh \
  -DEnGPar_PREFIX=$engpar \
  -DCMAKE_INSTALL_PREFIX=$PWD/install
make -j8 install
ctest -E destroy_test

Make the script executable:

chmod +x buildAll_turing.sh

Source the environment script from this work directory:

source envRhel7_turing.sh

Run the build script:

./buildAll_turing.sh

develop

To resume work on pumipic run the following commands to setup your environment:

cd pumi-pic
source envRhel7_turing.sh

Note, envRhel7_turing.sh MUST be sourced from the top-level work directory (pumipicDev).

Always do this before either of the following steps.

C++ source changes only

Assuming changes existing pumipic C++ source/header files were made you can just run make as follows:

cd build-pumipic-*/ #the machine name will be in the directory name
make

CMake changes

If CMake files were changed (i.e., to add new C++ source files) then you should delete the contents of the build directory (rm -rf build-pumipic-*), then rerun cmake, and make:

mkdir $pumipic
cd $pumipic/..
cmake ../pumi-pic/  \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DIS_TESTING=ON \
  -DPS_IS_TESTING=ON \
  -DENABLE_CABANA=ON \
  -DTEST_DATA_DIR=../pumi-pic/pumipic-data \
  -DOmega_h_PREFIX=$oh \
  -DEnGPar_PREFIX=$engpar \
  -DCMAKE_INSTALL_PREFIX=$PWD/install
make install -j8