How to reduce trilinos build times? #12796

jrobcary · 2024-03-05T17:48:11Z

jrobcary
Mar 5, 2024

I am trying to speed up trilinos builds. My latest build, on 4 cores:

Building trilinos-sercuda in /home/research/cary/projects/xsimall-dev2/builds/trilinos-14.0.0/sercuda using vcloud.txcorp.com-trilinos-sercuda-build.sh at 2024-03-04-06:53:46.

and it is currently at

[ 97%] Building CXX object packages/muelu/src/CMakeFiles/muelu.dir/Utils/ExplicitInstantiation/ETI_MueLu_ToggleCoordinatesTransferFactory.cpp.o

more than a day later:

Tue Mar 5 10:35:59 MST 2024

My thought is to reduce the number of template instantiations to ordinals = {int, ssizet}, floating point = {float, double}, cpu and cuda, openmp off. Is this possible? Other ideas for speeding up builds?

Thx.

cgcgcg · 2024-03-05T18:23:17Z

cgcgcg
Mar 5, 2024
Collaborator

Just for reference: I recently built the Trilinos solver stack based on Tpetra on a 9 year old laptop in ~ 3h. Can you post your CMake invocation?

0 replies

jrobcary · 2024-03-05T19:43:21Z

jrobcary
Mar 5, 2024
Author

Serial:

env NVCC_WRAPPER_DEFAULT_COMPILER='/home/bilder/contrib/gcc-9.4.0/bin/g++' PATH='/home/research/cary/opt/dev1/contrib-gcc940/geant4-11.1.1-sersh/bin:/home/research/cary/opt/dev1/volatile-gcc940/veccore/bin:/home/research/cary/opt/dev1/contrib-gcc940/swig-4.0.1-ser/bin:/home/research/cary/opt/dev1/contrib-gcc940/qt-5.14.2-sersh/bin:/home/research/cary/opt/dev1/volatile-gcc940/vc/bin:/home/research/cary/opt/dev1/contrib-gcc940/autotools-lt-2.4.6/bin:/home/research/cary/opt/dev1/contrib-gcc940/hdf5-1.12.1-ser/bin:/home/research/cary/opt/dev1/contrib-gcc940/pcre-8.36-ser/bin:/home/research/cary/opt/dev1/contrib-gcc940/mpich-4.0.2-shared/bin:/home/research/cary/opt/dev1/contrib-gcc940/cmake-3.26.4-ser/bin:/home/research/cary/opt/dev1/contrib-gcc940/Python-3.10.4-sersh/bin:/home/research/cary/opt/dev1/contrib-gcc940/openssl-1.1.1k-sersh/bin:/home/research/cary/opt/dev1/contrib-gcc940/xz-5.2.4-ser/bin:/home/bilder/contrib/texlive/bin/x86_64-linux:/home/research/cary/opt/dev1/contrib-gcc940/texlive/bin/x86_64-linux:/home/research/cary/opt/dev1/volatile-gcc940/bin:/home/research/cary/opt/dev1/contrib-gcc940/bin:/home/bilder/contrib/gcc-9.4.0/bin:/home/research/cary/txcenv:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/cuda-11.8/bin' '/home/research/cary/opt/dev1/contrib-gcc940/cmake-3.26.4-ser/bin/cmake' \
  -DCMAKE_INSTALL_PREFIX:PATH=/home/research/cary/opt/dev1/contrib-gcc940/trilinos-14.0.0-sercuda \
  -DCMAKE_BUILD_TYPE:STRING=Release \
  -DCMAKE_COLOR_MAKEFILE:BOOL=FALSE \
  -DCMAKE_VERBOSE_MAKEFILE:BOOL=TRUE \
  -DCMAKE_C_COMPILER:FILEPATH='/home/bilder/contrib/gcc-9.4.0/bin/gcc' \
  -DCMAKE_CXX_COMPILER:FILEPATH='/home/research/cary/projects/xsimall-dev1/builds/trilinos-14.0.0/packages/kokkos/bin/nvcc_wrapper' \
  -DCMAKE_CUDA_COMPILER:FILEPATH='/usr/local/cuda-11.8/bin/nvcc' \
  -DCMAKE_Fortran_COMPILER:FILEPATH='/home/bilder/contrib/gcc-9.4.0/bin/gfortran' \
  -DCMAKE_C_FLAGS:STRING='-pthread -pipe -fPIC' \
  -DCMAKE_CXX_FLAGS:STRING='-pthread -pipe -fPIC -fopenmp ' \
  -DCMAKE_CUDA_FLAGS='' \
  -DCMAKE_Fortran_FLAGS:STRING='-pipe -fPIC' \
  -DDART_TESTING_TIMEOUT:STRING=600 \
  -DML_ENABLE_SuperLU:BOOL=OFF \
  -DKokkosKernels_ENABLE_EXPERIMENTAL:BOOL=OFF \
  -DTrilinos_ENABLE_SuperLU5_API=ON \
  -DPYTHON_EXECUTABLE:FILEPATH=/home/research/cary/opt/dev1/contrib-gcc940/Python-3.10.4-sersh/bin/python3 \
  -DTpetra_INST_SERIAL:BOOL=ON \
  -DTrilinos_ENABLE_EXAMPLES:BOOL=FALSE \
  -DTrilinos_ENABLE_FLOAT:BOOL=ON \
  -DTpetra_INST_INT_LONG_LONG:BOOL=ON \
  -DCMAKE_EXE_LINKER_FLAGS:STRING='-L/home/bilder/contrib/gcc-9.4.0/lib64 -Wl,-rpath,/home/bilder/contrib/gcc-9.4.0/lib64' \
  -DTPL_BLAS_LIBRARIES:FILEPATH='/home/research/cary/opt/dev1/contrib-gcc940/lapack-3.7.0-ser/lib64/libblas.a' \
  -DTPL_LAPACK_LIBRARIES:FILEPATH='/home/research/cary/opt/dev1/contrib-gcc940/lapack-3.7.0-ser/lib64/liblapack.a' \
  -DTrilinos_ENABLE_Teuchos:BOOL=ON \
  -DTrilinos_ENABLE_Triutils:BOOL=ON \
  -DTrilinos_ENABLE_Epetra:BOOL=ON \
  -DTrilinos_ENABLE_Amesos:BOOL=ON \
  -DTrilinos_ENABLE_Ifpack:BOOL=ON \
  -DTrilinos_ENABLE_AztecOO:BOOL=ON \
  -DTrilinos_ENABLE_ML:BOOL=ON \
  -DTrilinos_ENABLE_EpetraExt:BOOL=ON \
  -DTrilinos_ENABLE_Galeri:BOOL=ON \
  -DTrilinos_ENABLE_Shards:BOOL=ON \
  -DTrilinos_ENABLE_Intrepid:BOOL=ON \
  -DTrilinos_ENABLE_Komplex:BOOL=ON \
  -DTrilinos_ENABLE_NOX:BOOL=ON \
  -DTrilinos_ENABLE_Kokkos:BOOL=ON \
  -DTrilinos_ENABLE_Tpetra:BOOL=ON \
  -DTrilinos_ENABLE_Xpetra:BOOL=ON \
  -DTrilinos_ENABLE_MueLu:BOOL=ON \
  -DTrilinos_ENABLE_Amesos2:BOOL=ON \
  -DTrilinos_ENABLE_Belos:BOOL=ON \
  -DTrilinos_ENABLE_Ifpack2:BOOL=ON \
  -DTPL_ENABLE_BinUtils:BOOL=OFF \
  -DTPL_ENABLE_SuperLU:BOOL=ON \
  -DSuperLU_INCLUDE_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/superlu-5.2.2-ser/include' \
  -DSuperLU_LIBRARY_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/superlu-5.2.2-ser/lib' \
  -DSuperLU_LIBRARY_NAMES:STRING='superlu' \
  -DTPL_ENABLE_HDF5:BOOL=ON \
  -DHDF5_INCLUDE_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/hdf5-1.12.1-ser/include' \
  -DHDF5_LIBRARY_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/hdf5-1.12.1-ser/lib' \
  -DHDF5_LIBRARY_NAMES:STRING='hdf5_hl;hdf5;hdf5_tools;z;dl' \
  -DTPL_ENABLE_Netcdf:BOOL=ON \
  -DNetcdf_INCLUDE_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/netcdfc-stable.r8003-ser/include' \
  -DNetcdf_LIBRARY_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/netcdfc-stable.r8003-ser/lib64;/home/research/cary/opt/dev1/contrib-gcc940/hdf5-1.12.1-ser/lib' \
  -DNetcdf_LIBRARY_NAMES:STRING='netcdf;hdf5_hl;hdf5;z;dl' \
  -DTrilinos_ENABLE_OpenMP:BOOL=ON \
  -DTpetra_INST_SERIAL:BOOL='ON' \
  -DKokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE:BOOL='TRUE' \
  -DKokkos_ENABLE_SERIAL:BOOL='ON' \
  -DTPL_ENABLE_CUDA:BOOL='ON' \
  -DTpetra_ENABLE_CUDA:BOOL='ON' \
  -DKokkos_ENABLE_CUDA:BOOL='ON' \
  -DKokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE:BOOL='TRUE' \
  -DKokkos_ENABLE_CUDA_UVM:BOOL='ON' \
  -DKokkos_ENABLE_CUDA_LAMBDA:BOOL='ON' \
  -DKokkos_ARCH_TURING75:BOOL='ON' \
  -DCMAKE_CUDA_ARCHITECTURES='75' \
  /home/research/cary/projects/xsimall-dev1/builds/trilinos-14.0.0

Parallel:

env NVCC_WRAPPER_DEFAULT_COMPILER='/home/research/cary/opt/dev1/contrib-gcc940/mpich-4.0.2-shared/bin/mpicxx' PATH='/home/research/cary/opt/dev1/contrib-gcc940/geant4-11.1.1-sersh/bin:/home/research/cary/opt/dev1/volatile-gcc940/veccore/bin:/home/research/cary/opt/dev1/contrib-gcc940/swig-4.0.1-ser/bin:/home/research/cary/opt/dev1/contrib-gcc940/qt-5.14.2-sersh/bin:/home/research/cary/opt/dev1/volatile-gcc940/vc/bin:/home/research/cary/opt/dev1/contrib-gcc940/autotools-lt-2.4.6/bin:/home/research/cary/opt/dev1/contrib-gcc940/hdf5-1.12.1-ser/bin:/home/research/cary/opt/dev1/contrib-gcc940/pcre-8.36-ser/bin:/home/research/cary/opt/dev1/contrib-gcc940/mpich-4.0.2-shared/bin:/home/research/cary/opt/dev1/contrib-gcc940/cmake-3.26.4-ser/bin:/home/research/cary/opt/dev1/contrib-gcc940/Python-3.10.4-sersh/bin:/home/research/cary/opt/dev1/contrib-gcc940/openssl-1.1.1k-sersh/bin:/home/research/cary/opt/dev1/contrib-gcc940/xz-5.2.4-ser/bin:/home/bilder/contrib/texlive/bin/x86_64-linux:/home/research/cary/opt/dev1/contrib-gcc940/texlive/bin/x86_64-linux:/home/research/cary/opt/dev1/volatile-gcc940/bin:/home/research/cary/opt/dev1/contrib-gcc940/bin:/home/bilder/contrib/gcc-9.4.0/bin:/home/research/cary/txcenv:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/cuda-11.8/bin' '/home/research/cary/opt/dev1/contrib-gcc940/cmake-3.26.4-ser/bin/cmake' \
  -DCMAKE_INSTALL_PREFIX:PATH=/home/research/cary/opt/dev1/contrib-gcc940/trilinos-14.0.0-parcuda \
  -DCMAKE_BUILD_TYPE:STRING=Release \
  -DCMAKE_COLOR_MAKEFILE:BOOL=FALSE \
  -DCMAKE_VERBOSE_MAKEFILE:BOOL=TRUE \
  -DTPL_ENABLE_MPI:BOOL=ON \
  -DCMAKE_C_COMPILER:FILEPATH='/home/research/cary/opt/dev1/contrib-gcc940/mpich-4.0.2-shared/bin/mpicc' \
  -DCMAKE_CXX_COMPILER:FILEPATH='/home/research/cary/projects/xsimall-dev1/builds/trilinos-14.0.0/packages/kokkos/bin/nvcc_wrapper' \
  -DCMAKE_CUDA_COMPILER:FILEPATH='/usr/local/cuda-11.8/bin/nvcc' \
  -DCMAKE_Fortran_COMPILER:FILEPATH='/home/research/cary/opt/dev1/contrib-gcc940/mpich-4.0.2-shared/bin/mpif90' \
  -DCMAKE_C_FLAGS:STRING='-pthread -pipe -fPIC' \
  -DCMAKE_CXX_FLAGS:STRING='-pthread -pipe -fPIC -DHAVE_SUPERLUDIST_ENUM_NAMESPACE -DHAVE_SUPERLUDIST_LUSTRUCTINIT_2ARG -fopenmp ' \
  -DCMAKE_CUDA_FLAGS='' \
  -DCMAKE_Fortran_FLAGS:STRING='-pipe -fPIC' \
  -DDART_TESTING_TIMEOUT:STRING=600 \
  -DML_ENABLE_SuperLU:BOOL=OFF \
  -DKokkosKernels_ENABLE_EXPERIMENTAL:BOOL=OFF \
  -DTrilinos_ENABLE_SuperLU5_API=ON \
  -DPYTHON_EXECUTABLE:FILEPATH=/home/research/cary/opt/dev1/contrib-gcc940/Python-3.10.4-sersh/bin/python3 \
  -DTpetra_INST_SERIAL:BOOL=ON \
  -DTrilinos_ENABLE_EXAMPLES:BOOL=FALSE \
  -DTrilinos_ENABLE_FLOAT:BOOL=ON \
  -DTpetra_INST_INT_LONG_LONG:BOOL=ON \
  -DCMAKE_EXE_LINKER_FLAGS:STRING='-L/home/bilder/contrib/gcc-9.4.0/lib64 -Wl,-rpath,/home/bilder/contrib/gcc-9.4.0/lib64' \
  -DTPL_BLAS_LIBRARIES:FILEPATH='/home/research/cary/opt/dev1/contrib-gcc940/lapack-3.7.0-ser/lib64/libblas.a' \
  -DTPL_LAPACK_LIBRARIES:FILEPATH='/home/research/cary/opt/dev1/contrib-gcc940/lapack-3.7.0-ser/lib64/liblapack.a' \
  -DTrilinos_ENABLE_Teuchos:BOOL=ON \
  -DTrilinos_ENABLE_Triutils:BOOL=ON \
  -DTrilinos_ENABLE_Epetra:BOOL=ON \
  -DTrilinos_ENABLE_Amesos:BOOL=ON \
  -DTrilinos_ENABLE_Ifpack:BOOL=ON \
  -DTrilinos_ENABLE_AztecOO:BOOL=ON \
  -DTrilinos_ENABLE_ML:BOOL=ON \
  -DTrilinos_ENABLE_EpetraExt:BOOL=ON \
  -DTrilinos_ENABLE_Galeri:BOOL=ON \
  -DTrilinos_ENABLE_Shards:BOOL=ON \
  -DTrilinos_ENABLE_Intrepid:BOOL=ON \
  -DTrilinos_ENABLE_Komplex:BOOL=ON \
  -DTrilinos_ENABLE_NOX:BOOL=ON \
  -DTrilinos_ENABLE_Kokkos:BOOL=ON \
  -DTrilinos_ENABLE_Tpetra:BOOL=ON \
  -DTrilinos_ENABLE_Xpetra:BOOL=ON \
  -DTrilinos_ENABLE_MueLu:BOOL=ON \
  -DTrilinos_ENABLE_Amesos2:BOOL=ON \
  -DTrilinos_ENABLE_Belos:BOOL=ON \
  -DTrilinos_ENABLE_Ifpack2:BOOL=ON \
  -DTPL_ENABLE_BinUtils:BOOL=OFF \
  -DTPL_ENABLE_SuperLUDist:BOOL=ON \
  -DSuperLUDist_INCLUDE_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/superlu_dist-5.4.0-par/include' \
  -DSuperLUDist_LIBRARY_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/superlu_dist-5.4.0-par/lib' \
  -DSuperLUDist_LIBRARY_NAMES:STRING='superlu_dist' \
  -DTPL_ENABLE_SuperLUDist_Without_ParMETIS:BOOL=TRUE \
  -DAmesos_Allow_SuperLUDist_Without_ParMETIS=ON \
  -DTPL_ENABLE_HDF5:BOOL=ON \
  -DHDF5_INCLUDE_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/hdf5-1.12.1-par/include' \
  -DHDF5_LIBRARY_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/hdf5-1.12.1-par/lib' \
  -DHDF5_LIBRARY_NAMES:STRING='hdf5_hl;hdf5;hdf5_tools;z;dl' \
  -DTPL_ENABLE_Netcdf:BOOL=ON \
  -DNetcdf_INCLUDE_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/netcdfc-stable.r8003-par/include' \
  -DNetcdf_LIBRARY_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/netcdfc-stable.r8003-par/lib64;/home/research/cary/opt/dev1/contrib-gcc940/hdf5-1.12.1-par/lib' \
  -DNetcdf_LIBRARY_NAMES:STRING='netcdf;hdf5_hl;hdf5;z;dl' \
  -DTPL_ENABLE_HYPRE:BOOL=ON \
  -DHYPRE_INCLUDE_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/hypre-2.20.0-parsh/include' \
  -DHYPRE_LIBRARY_DIRS:PATH='/home/research/cary/opt/dev1/contrib-gcc940/hypre-2.20.0-parsh/lib' \
  -DHYPRE_LIBRARY_NAMES:STRING='HYPRE' \
  -DIfpack2_ENABLE_HYPRE=OFF \
  -DTpetra_INST_SERIAL:BOOL='ON' \
  -DKokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE:BOOL='TRUE' \
  -DKokkos_ENABLE_SERIAL:BOOL='ON' \
  -DTPL_ENABLE_CUDA:BOOL='ON' \
  -DTpetra_ENABLE_CUDA:BOOL='ON' \
  -DKokkos_ENABLE_CUDA:BOOL='ON' \
  -DKokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE:BOOL='TRUE' \
  -DKokkos_ENABLE_CUDA_UVM:BOOL='ON' \
  -DKokkos_ENABLE_CUDA_LAMBDA:BOOL='ON' \
  -DKokkos_ARCH_TURING75:BOOL='ON' \
  -DCMAKE_CUDA_ARCHITECTURES='75' \
  -DHAVE_TEUCHOS_COMPLEX:BOOL=FALSE \
  /home/research/cary/projects/xsimall-dev1/builds/trilinos-14.0.0

0 replies

cgcgcg · 2024-03-05T19:55:22Z

cgcgcg
Mar 5, 2024
Collaborator

Do you need the non-Kokkos stack in a CUDA build?
Unless there is a clear use case for it, disable float.
UVM isn't needed anymore, unless your application needs it. In that case, make sure that the flags are right, see Tpetra: Adding cmake support for making Tpetra allocate in shared space #12622
In order to speed up repeated builds, consider using ccache.

0 replies

jrobcary · 2024-03-05T21:04:53Z

jrobcary
Mar 5, 2024
Author

Thx.

Do you need the non-Kokkos stack in a CUDA build?
Our old framework uses Epetra stuff, and we need to keep that working.
Unless there is a clear use case for it, disable float.
Our current solvers can switch to float, greating improving performance when it is sufficient.
UVM isn't needed anymore, unless your application needs it. In that case, make sure that the flags are right, see
#12622
Disabling
In order to speed up repeated builds, consider using ccache.
Looking into this.

Is there a minimum version of trilinos for this?

1 reply

cgcgcg Mar 5, 2024
Collaborator

#12622 got merged in December. ccache has been working for ages. -D CMAKE_CXX_COMPILER_LAUNCHER=ccache should do the trick.

jrobcary · 2024-04-14T15:08:37Z

jrobcary
Apr 14, 2024
Author

Seems done.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trilinos

How to reduce trilinos build times? #12796

{{title}}

Replies: 5 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Trilinos

How to reduce trilinos build times? #12796

jrobcary Mar 5, 2024

Replies: 5 comments · 1 reply

cgcgcg Mar 5, 2024 Collaborator

jrobcary Mar 5, 2024 Author

cgcgcg Mar 5, 2024 Collaborator

jrobcary Mar 5, 2024 Author

cgcgcg Mar 5, 2024 Collaborator

jrobcary Apr 14, 2024 Author

jrobcary
Mar 5, 2024

Replies: 5 comments 1 reply

cgcgcg
Mar 5, 2024
Collaborator

jrobcary
Mar 5, 2024
Author

cgcgcg
Mar 5, 2024
Collaborator

jrobcary
Mar 5, 2024
Author

cgcgcg Mar 5, 2024
Collaborator

jrobcary
Apr 14, 2024
Author