Skip to content

tracking memory defects

Jump to bottom
Luc Berger edited this page Aug 15, 2019 · 18 revisions

Explanations and instructions on how to track common memory defects in MueLu

Using clang tools

clang and llvm provide comprehensive tools to analyze the quality of code from formatting tools to compiler assisted debugging to just-in-time code optimization. One such tools is the address sanitizer library that can be linked against at compilation time to keep track of memory leaks, uninitialized memory, out-of-bound access...

Requirements

Of course you need a clang compiler to use the clang address sanitizer, at the time of writing clang/7.0.1 is used through the sems modules. See below a list of modules loaded to perform the build.

[lberge@trappist muelu]$ module list
Currently Loaded Modulefiles:
  1) sems-env                   3) sems-clang/7.0.1           5) sems-cmake/3.12.2
  2) sems-gcc/5.3.0             4) sems-openmpi/1.10.1        6) sems-ninja_fortran/1.8.2

You also need to tell clang to instrument the code and to link against the sanitizer library. Below is a minimal configuration script that enables the sanitizer at build and link time.

CXX_WARNINGS_CLANG="-Wall -Wno-vla -Wno-pragmas -Wno-unknown-pragmas -Wno-unused-local-typedefs -Wno-deprecated-declarations -Wno-address -Wno-inline -DTRILINOS_HIDE_DEPRECATED_HEADER_WARNINGS -fsanitize=address -fno-omit-frame-pointer"

echo "Configure sanitized version of Trilinos to identify memory issues"
cd ${TRILINOS_BUILD_ROOT}/sanitizer
rm -f CMakeCache.txt
rm -rf CMakeFiles
cmake \
  -G "Ninja" \
  -D CMAKE_BUILD_TYPE:STRING="DEBUG" \
  -D BUILD_SHARED_LIBS=ON \
  -D CMAKE_LINKER:FILEPATH=${SEMS_CLANG_ROOT}/bin/clang++ \      <---- you need to specify the clang linker!
  -D CMAKE_EXE_LINKER_FLAGS:STRING="-fsanitize=address" \        <---- the sanitizer flag is also needed by the linker
  -D CMAKE_CXX_FLAGS="${CXX_WARNINGS_CLANG}" \                   <---- defined above and contains flag for sanitizer
  -D Trilinos_ENABLE_CXX11=ON \
  -D Trilinos_ENABLE_Fortran=OFF \
  -D TPL_ENABLE_MPI=ON \
  -D Trilinos_ENABLE_OpenMP=ON \
  -D Trilinos_ENABLE_FLOAT=OFF \
  -D Trilinos_ENABLE_COMPLEX_DOUBLE=OFF \
  -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
  -D Teuchos_GLOBALLY_REDUCE_UNITTEST_RESULTS=ON \
  -D Trilinos_ENABLE_MueLu=ON \
  -D   MueLu_ENABLE_TESTS=ON \
  -D   MueLu_ENABLE_EXAMPLES=ON \
  -D   MueLu_ENABLE_Experimental=ON \
  -D   MueLu_ENABLE_Kokkos_Refactor=ON \
  -D   MueLu_ENABLE_Kokkos_Refactor_Use_By_Default=ON \
  -D   MueLu_CXX_FLAGS="${CXX_WARNINGS_CLANG}" \                 <---- this is somewhat redundant but allows to set flag such as -Werror in MueLu only
  -D Trilinos_ENABLE_Xpetra=ON \
  -D   Xpetra_ENABLE_TESTS=ON \
  -D   Xpetra_ENABLE_Experimental=ON \
  -D   Xpetra_ENABLE_Kokkos_Refactor=ON \
  -D CMAKE_INSTALL_PREFIX:PATH=${TRILINOS_BUILD_ROOT}/sanitizer \
  ${TRILINOS_SOURCE} 2>&1 | tee ${TRILINOS_BUILD_ROOT}/sanitizer/configure_sanitizer.log

Example output

Once the build a build of trilinos has been perform, simply running an example will generate a stack trace if a memory misuse is detected. I recommend however to suppress the output from memory issues in mpi in order to get a more readable report. To suppress output from specific libraries you need to set the following environment variable export LSAN_OPTIONS=suppressions=/path/to/suppression/file/ASAN.supp and you can find an example of a suppression file below.

[lberge@trappist Trilinos_lucbv_builds]$ cat ASAN.supp
leak:libopen-pal
leak:libmpi

Finally running the MueLu_SimpleTpetra_MPI_4 example at the time of writing produces the following output:

[lberge@trappist muelu]$ ctest -R MueLu_SimpleTpetra_MPI_4 -VV
UpdateCTestConfiguration  from :/home/lberge/Research/Trilinos_lucbv_builds/sanitizer/packages/muelu/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/lberge/Research/Trilinos_lucbv_builds/sanitizer/packages/muelu/DartConfiguration.tcl
Test project /home/lberge/Research/Trilinos_lucbv_builds/sanitizer/packages/muelu
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 14
    Start 14: MueLu_SimpleTpetra_MPI_4

14: Test command: /projects/sems/install/rhel7-x86_64/sems/compiler/clang/7.0.1/openmpi/1.10.1/bin/mpiexec "-np" "4" "/home/lberge/Research/Trilinos_lucbv_builds/sanitizer/packages/muelu/example/basic/MueLu_Simple.exe" "--linAlgebra=Tpetra" "--xml=simple.xml"
14: Test timeout computed to be: 10000000
14: ========================================================
14: Xpetra::Parameters
14:  Linear algebra library: Tpetra
14: Galeri::Xpetra::Parameters<long long>
14:  Matrix type: Laplace2D
14:  Problem size: 10000 (100x100)
14: Processor subdomains in x direction: 2
14: Processor subdomains in y direction: 2
14: Processor subdomains in z direction: -1
14: ========================================================
14: Galeri complete.
14: ========================================================
14: *********** MueLu ParameterList ***********
14: verbosity = low   [unused]
14: coarse: max size = 1000   [unused]
14: coarse: type = Klu2   [unused]
14: cycle type = V   [unused]
14: multigrid algorithm = sa   [unused]
14: transpose: use implicit = 1   [unused]
14: max levels = 10   [unused]
14: number of equations = 1   [unused]
14: sa: use filtered matrix = 1   [unused]
14: aggregation: type = uncoupled   [unused]
14: aggregation: drop scheme = classical   [unused]
14: smoother: type = RELAXATION   [unused]
14: use kokkos refactor = 0   [unused]
14: smoother: params -> 
14:  relaxation: type = Jacobi   [unused]
14:  relaxation: sweeps = 1   [unused]
14:  relaxation: damping factor = 0.8   [unused]
14: *******************************************
14: =================================================================
14: =================================================================
14: ==36955==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffcd41456e0 at pc 0x000000a0cdf2 bp 0x7ffcd4145240 sp 0x7ffcd4145238
14: =================================================================
14: READ of size 8 at 0x7ffcd41456e0 thread T0
14: ==36957==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7fff4fe37b20 at pc 0x000000a0cdf2 bp 0x7fff4fe37680 sp 0x7fff4fe37678
14: READ of size 8 at 0x7fff4fe37b20 thread T0
14: ==36956==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffde50526e0 at pc 0x000000a0cdf2 bp 0x7ffde5052240 sp 0x7ffde5052238
14: =================================================================
14: READ of size 8 at 0x7ffde50526e0 thread T0
14: ==36954==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffd7122aac0 at pc 0x000000a0cdf2 bp 0x7ffd7122a620 sp 0x7ffd7122a618
14: READ of size 8 at 0x7ffd7122aac0 thread T0
14:     #0 0xa0cdf1 in Kokkos::Impl::SharedAllocationTracker::operator=(Kokkos::Impl::SharedAllocationTracker const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:401:23
14:     #1 0xa0cdf1 in Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> >::operator=(Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> > const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/core/src/Kokkos_View.hpp:1984
14:     #2 0xa0cbae in Kokkos::StaticCrsGraph<int, Kokkos::LayoutLeft, Kokkos::OpenMP, unsigned long, void>::operator=(Kokkos::StaticCrsGraph<int, Kokkos::LayoutLeft, Kokkos::OpenMP, unsigned long, void> const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/containers/src/Kokkos_StaticCrsGraph.hpp:340:13
14:     #0 0xa0cdf1 in Kokkos::Impl::SharedAllocationTracker::operator=(Kokkos::Impl::SharedAllocationTracker const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:401:23
14:     #1 0xa0cdf1 in Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> >::operator=(Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> > const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/core/src/Kokkos_View.hpp:1984
14:     #2 0xa0cbae in Kokkos::StaticCrsGraph<int, Kokkos::LayoutLeft, Kokkos::OpenMP, unsigned long, void>::operator=(Kokkos::StaticCrsGraph<int, Kokkos::LayoutLeft, Kokkos::OpenMP, unsigned long, void> const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/containers/src/Kokkos_StaticCrsGraph.hpp:340:13
14:     #0 0xa0cdf1 in Kokkos::Impl::SharedAllocationTracker::operator=(Kokkos::Impl::SharedAllocationTracker const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:401:23
14:     #1 0xa0cdf1 in Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> >::operator=(Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> > const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/core/src/Kokkos_View.hpp:1984
14:     #2 0xa0cbae in Kokkos::StaticCrsGraph<int, Kokkos::LayoutLeft, Kokkos::OpenMP, unsigned long, void>::operator=(Kokkos::StaticCrsGraph<int, Kokkos::LayoutLeft, Kokkos::OpenMP, unsigned long, void> const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/containers/src/Kokkos_StaticCrsGraph.hpp:340:13
14:     #0 0xa0cdf1 in Kokkos::Impl::SharedAllocationTracker::operator=(Kokkos::Impl::SharedAllocationTracker const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/core/src/impl/Kokkos_SharedAlloc.hpp:401:23
14:     #1 0xa0cdf1 in Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> >::operator=(Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::MemoryTraits<0u> > const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/core/src/Kokkos_View.hpp:1984
14:     #2 0xa0cbae in Kokkos::StaticCrsGraph<int, Kokkos::LayoutLeft, Kokkos::OpenMP, unsigned long, void>::operator=(Kokkos::StaticCrsGraph<int, Kokkos::LayoutLeft, Kokkos::OpenMP, unsigned long, void> const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos/containers/src/Kokkos_StaticCrsGraph.hpp:340:13
14:     #3 0x7f2fb92e7bde in KokkosSparse::CrsMatrix<double, int, Kokkos::OpenMP, void, unsigned long>::operator=(KokkosSparse::CrsMatrix<double, int, Kokkos::OpenMP, void, unsigned long> const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos-kernels/src/sparse/KokkosSparse_CrsMatrix.hpp:384:7
14:     #4 0x7f2fb93e575f in Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::local_matrix_type const Tpetra::MMdetails::merge_matrices<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, unsigned long) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:3446:19
14:     #5 0x7f2fb93d9797 in Tpetra::MMdetails::KernelWrappers<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >::mult_A_B_newmatrix_kernel_wrapper(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Teuchos::RCP<Tpetra::Import<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_OpenMP.hpp:239:26
14:     #6 0x7f2fb9141b93 in void Tpetra::MMdetails::mult_A_B_newmatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:2068:3
14:     #3 0x7fa31ee64bde in KokkosSparse::CrsMatrix<double, int, Kokkos::OpenMP, void, unsigned long>::operator=(KokkosSparse::CrsMatrix<double, int, Kokkos::OpenMP, void, unsigned long> const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos-kernels/src/sparse/KokkosSparse_CrsMatrix.hpp:384:7
14:     #7 0x7f2fb912f49e in void Tpetra::MatrixMatrix::Multiply<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:256:5
14:     #4 0x7fa31ef6275f in Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::local_matrix_type const Tpetra::MMdetails::merge_matrices<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, unsigned long) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:3446:19
14:     #5 0x7fa31ef56797 in Tpetra::MMdetails::KernelWrappers<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >::mult_A_B_newmatrix_kernel_wrapper(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Teuchos::RCP<Tpetra::Import<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_OpenMP.hpp:239:26
14:     #6 0x7fa31ecbeb93 in void Tpetra::MMdetails::mult_A_B_newmatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:2068:3
14:     #7 0x7fa31ecac49e in void Tpetra::MatrixMatrix::Multiply<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:256:5
14:     #3 0x7f0f94234bde in KokkosSparse::CrsMatrix<double, int, Kokkos::OpenMP, void, unsigned long>::operator=(KokkosSparse::CrsMatrix<double, int, Kokkos::OpenMP, void, unsigned long> const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos-kernels/src/sparse/KokkosSparse_CrsMatrix.hpp:384:7
14:     #4 0x7f0f9433275f in Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::local_matrix_type const Tpetra::MMdetails::merge_matrices<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, unsigned long) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:3446:19
14:     #5 0x7f0f94326797 in Tpetra::MMdetails::KernelWrappers<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >::mult_A_B_newmatrix_kernel_wrapper(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Teuchos::RCP<Tpetra::Import<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_OpenMP.hpp:239:26
14:     #6 0x7f0f9408eb93 in void Tpetra::MMdetails::mult_A_B_newmatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:2068:3
14:     #7 0x7f0f9407c49e in void Tpetra::MatrixMatrix::Multiply<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:256:5
14:     #3 0x7f81f45e9bde in KokkosSparse::CrsMatrix<double, int, Kokkos::OpenMP, void, unsigned long>::operator=(KokkosSparse::CrsMatrix<double, int, Kokkos::OpenMP, void, unsigned long> const&) /home/lberge/Research/Trilinos_lucbv/packages/kokkos-kernels/src/sparse/KokkosSparse_CrsMatrix.hpp:384:7
14:     #4 0x7f81f46e775f in Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::local_matrix_type const Tpetra::MMdetails::merge_matrices<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, unsigned long) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:3446:19
14:     #5 0x7f81f46db797 in Tpetra::MMdetails::KernelWrappers<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >::mult_A_B_newmatrix_kernel_wrapper(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Teuchos::RCP<Tpetra::Import<int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_OpenMP.hpp:239:26
14:     #6 0x7f81f4443b93 in void Tpetra::MMdetails::mult_A_B_newmatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:2068:3
14:     #7 0x7f81f443149e in void Tpetra::MatrixMatrix::Multiply<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:256:5
14:     #8 0x7f2fd34a6fae in Xpetra::MatrixMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Multiply(Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/xpetra/sup/Utils/Xpetra_MatrixMatrix.hpp:1627:9
14:     #8 0x7fa339023fae in Xpetra::MatrixMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Multiply(Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/xpetra/sup/Utils/Xpetra_MatrixMatrix.hpp:1627:9
14:     #8 0x7f0fae3f3fae in Xpetra::MatrixMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Multiply(Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/xpetra/sup/Utils/Xpetra_MatrixMatrix.hpp:1627:9
14:     #8 0x7f820e7a8fae in Xpetra::MatrixMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Multiply(Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/xpetra/sup/Utils/Xpetra_MatrixMatrix.hpp:1627:9
14:     #9 0x7fa333fe1e66 in Xpetra::MatrixMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Multiply(Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >, Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/xpetra/sup/Utils/Xpetra_MatrixMatrix.hpp:1713:7
14:     #9 0x7f2fce464e66 in Xpetra::MatrixMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Multiply(Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >, Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/xpetra/sup/Utils/Xpetra_MatrixMatrix.hpp:1713:7
14:     #9 0x7f0fa93b1e66 in Xpetra::MatrixMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Multiply(Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >, Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/xpetra/sup/Utils/Xpetra_MatrixMatrix.hpp:1713:7
14:     #10 0x7fa335187e8a in MueLu::RAPFactory<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Build(MueLu::Level&, MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Misc/MueLu_RAPFactory_def.hpp:169:16
14:     #11 0x7fa33926871a in MueLu::TwoLevelFactoryBase::CallBuild(MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/../MueCentral/MueLu_TwoLevelFactoryBase.hpp:151:7
14:     #10 0x7f2fcf60ae8a in MueLu::RAPFactory<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Build(MueLu::Level&, MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Misc/MueLu_RAPFactory_def.hpp:169:16
14:     #11 0x7f2fd36eb71a in MueLu::TwoLevelFactoryBase::CallBuild(MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/../MueCentral/MueLu_TwoLevelFactoryBase.hpp:151:7
14:     #9 0x7f8209766e66 in Xpetra::MatrixMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Multiply(Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > const&, bool, Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >, Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Teuchos::RCP<Teuchos::ParameterList> const&) /home/lberge/Research/Trilinos_lucbv/packages/xpetra/sup/Utils/Xpetra_MatrixMatrix.hpp:1713:7
14:     #10 0x7f0faa557e8a in MueLu::RAPFactory<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Build(MueLu::Level&, MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Misc/MueLu_RAPFactory_def.hpp:169:16
14:     #11 0x7f0fae63871a in MueLu::TwoLevelFactoryBase::CallBuild(MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/../MueCentral/MueLu_TwoLevelFactoryBase.hpp:151:7
14:     #10 0x7f820a90ce8a in MueLu::RAPFactory<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Build(MueLu::Level&, MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Misc/MueLu_RAPFactory_def.hpp:169:16
14:     #11 0x7f820e9ed71a in MueLu::TwoLevelFactoryBase::CallBuild(MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/../MueCentral/MueLu_TwoLevelFactoryBase.hpp:151:7
14:     #12 0x7f2fd26f095c in Teuchos::RCP<Xpetra::Operator<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >& MueLu::Level::Get<Teuchos::RCP<Xpetra::Operator<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, MueLu::FactoryBase const*) /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/../MueCentral/MueLu_Level.hpp:203:14
14:     #12 0x7fa33826d95c in Teuchos::RCP<Xpetra::Operator<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >& MueLu::Level::Get<Teuchos::RCP<Xpetra::Operator<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, MueLu::FactoryBase const*) /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/../MueCentral/MueLu_Level.hpp:203:14
14:     #13 0x7f2fcfcc1ac1 in MueLu::TopRAPFactory<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Build(MueLu::Level&, MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/MueCentral/MueLu_TopRAPFactory_def.hpp:76:38
14:     #13 0x7fa33583eac1 in MueLu::TopRAPFactory<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Build(MueLu::Level&, MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/MueCentral/MueLu_TopRAPFactory_def.hpp:76:38
14:     #12 0x7f0fad63d95c in Teuchos::RCP<Xpetra::Operator<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >& MueLu::Level::Get<Teuchos::RCP<Xpetra::Operator<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, MueLu::FactoryBase const*) /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/../MueCentral/MueLu_Level.hpp:203:14
14:     #14 0x7f2fceee23ac in MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Setup(int, Teuchos::RCP<MueLu::FactoryManagerBase const>, Teuchos::RCP<MueLu::FactoryManagerBase const>, Teuchos::RCP<MueLu::FactoryManagerBase const>) /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/MueCentral/MueLu_Hierarchy_def.hpp:377:24
14:     #15 0x7f2fd1e75e1e in MueLu::HierarchyManager<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::SetupHierarchy(MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/MueLu_HierarchyManager.hpp:241:20
14:     #16 0x7f2fd1e7240a in MueLu::ParameterListInterpreter<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::SetupHierarchy(MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/MueLu_ParameterListInterpreter_def.hpp:2203:23
14:     #14 0x7fa334a5f3ac in MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Setup(int, Teuchos::RCP<MueLu::FactoryManagerBase const>, Teuchos::RCP<MueLu::FactoryManagerBase const>, Teuchos::RCP<MueLu::FactoryManagerBase const>) /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/MueCentral/MueLu_Hierarchy_def.hpp:377:24
14:     #17 0xc923f2 in Teuchos::RCP<MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > > MueLu::CreateXpetraPreconditioner<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >, Teuchos::ParameterList const&) /home/lberge/Research/Trilinos_lucbv/packages/muelu/adapters/xpetra/MueLu_CreateXpetraPreconditioner.hpp:97:19
14:     #15 0x7fa3379f2e1e in MueLu::HierarchyManager<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::SetupHierarchy(MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/MueLu_HierarchyManager.hpp:241:20
14:     #16 0x7fa3379ef40a in MueLu::ParameterListInterpreter<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::SetupHierarchy(MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/Interface/MueLu_ParameterListInterpreter_def.hpp:2203:23
14:     #17 0xc923f2 in Teuchos::RCP<MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > > MueLu::CreateXpetraPreconditioner<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >, Teuchos::ParameterList const&) /home/lberge/Research/Trilinos_lucbv/packages/muelu/adapters/xpetra/MueLu_CreateXpetraPreconditioner.hpp:97:19
14:     #18 0xb21ace in void PreconditionerSetup<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&, Teuchos::RCP<Xpetra::MultiVector<Teuchos::ScalarTraits<double>::coordinateType, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&, Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&, Teuchos::ParameterList&, bool, bool, bool, int, Teuchos::RCP<MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&, Teuchos::RCP<Xpetra::Operator<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&) /home/lberge/Research/Trilinos_lucbv/packages/muelu/example/basic/../../test/scaling/DriverCore.hpp:210:12
14:     #19 0x73c613 in int main_<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Teuchos::CommandLineProcessor&, Xpetra::UnderlyingLib&, int, char**) /home/lberge/Research/Trilinos_lucbv/packages/muelu/example/basic/Simple.cpp:203:5
14:     #18 0xb21ace in void PreconditionerSetup<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Teuchos::RCP<Xpetra::Matrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&, Teuchos::RCP<Xpetra::MultiVector<Teuchos::ScalarTraits<double>::coordinateType, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&, Teuchos::RCP<Xpetra::MultiVector<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&, Teuchos::ParameterList&, bool, bool, bool, int, Teuchos::RCP<MueLu::Hierarchy<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&, Teuchos::RCP<Xpetra::Operator<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> > >&) /home/lberge/Research/Trilinos_lucbv/packages/muelu/example/basic/../../test/scaling/DriverCore.hpp:210:12
14:     #20 0x70101a in Automatic_Test_ETI(int, char**) /home/lberge/Research/Trilinos_lucbv/packages/muelu/example/basic/../../test/unit_tests/MueLu_Test_ETI.hpp:158:18
14:     #21 0x7028a1 in main /home/lberge/Research/Trilinos_lucbv/packages/muelu/example/basic/Simple.cpp:254:10
14:     #19 0x73c613 in int main_<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >(Teuchos::CommandLineProcessor&, Xpetra::UnderlyingLib&, int, char**) /home/lberge/Research/Trilinos_lucbv/packages/muelu/example/basic/Simple.cpp:203:5
14:     #20 0x70101a in Automatic_Test_ETI(int, char**) /home/lberge/Research/Trilinos_lucbv/packages/muelu/example/basic/../../test/unit_tests/MueLu_Test_ETI.hpp:158:18
14:     #21 0x7028a1 in main /home/lberge/Research/Trilinos_lucbv/packages/muelu/example/basic/Simple.cpp:254:10
14:     #13 0x7f0faac0eac1 in MueLu::TopRAPFactory<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::Build(MueLu::Level&, MueLu::Level&) const /home/lberge/Research/Trilinos_lucbv/packages/muelu/src/MueCentral/MueLu_TopRAPFactory_def.hpp:76:38
14:     #22 0x7f2fa993d3d4 in __libc_start_main (/lib64/libc.so.6+0x223d4)
14:     #23 0x5ec3bc in _start (/home/lberge/Research/Trilinos_lucbv_builds/sanitizer/packages/muelu/example/basic/MueLu_Simple.exe+0x5ec3bc)
14: 
14:     #22 0x7fa30f4ba3d4 in __libc_start_main (/lib64/libc.so.6+0x223d4)
14: Address 0x7fff4fe37b20 is located in stack of thread T0    #23 0x5ec3bc in _start (/home/lberge/Research/Trilinos_lucbv_builds/sanitizer/packages/muelu/example/basic/MueLu_Simple.exe+0x5ec3bc)
14:  at offset 832 in frame
14: 
14: Address 0x7ffcd41456e0 is located in stack of thread T0    #0 0x7f2fb93e46bf in Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::local_matrix_type const Tpetra::MMdetails::merge_matrices<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, unsigned long) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:3425
14:  at offset 832 in frame
14: 
14:   This frame has 27 object(s):
14:     #0 0x7fa31ef616bf in Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >::local_matrix_type const Tpetra::MMdetails::merge_matrices<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > >(Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Tpetra::CrsMatrixStruct<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, unsigned long) /home/lberge/Research/Trilinos_lucbv/packages/tpetra/core/ext/TpetraExt_MatrixMatrix_def.hpp:3425
14:     [32, 176) 'ref.tmp' (line 3434)
14:     [240, 384) 'ref.tmp1' (line 3435)
14: 
14:     [448, 472) 'ref.tmp11' (line 3438)
14:   This frame has 27 object(s):
14:     [512, 536) 'ref.tmp12' (line 3438)
14:     [32, 176) 'ref.tmp' (line 3434)
14:     [576, 600) 'ref.tmp30' (line 3438)
14:     [240, 384) 'ref.tmp1' (line 3435)
14:     [640, 664) 'ref.tmp32' (line 3438)
14:     [448, 472) 'ref.tmp11' (line 3438)
14:     [704, 728) 'Ik_' (line 3442)
14:     [512, 536) 'ref.tmp12' (line 3438)
14:     [768, 792) 'ref.tmp110' (line 3443)
14:     [576, 600) 'ref.tmp30' (line 3438)
14:     [832, 976) 'ref.tmp111' (line 3443) <== Memory access at offset 832 is inside this variable
14:     [640, 664) 'ref.tmp32' (line 3438)
14:     [704, 728) 'Ik_' (line 3442)
14:     [1040, 1184) 'Iks' (line 3445)
14:     [768, 792) 'ref.tmp110' (line 3443)
14:     [1248, 1272) 'Mrowptr' (line 3449)
14:     [832, 976) 'ref.tmp111' (line 3443) <== Memory access at offset 832 is inside this variable
14:     [1312, 1344) 'ref.tmp148' (line 3456)
14:     [1376, 1377) 'ref.tmp149' (line 3456)
14:     [1040, 1184) 'Iks' (line 3445)
...

Using valgrind

Home

Intro

Xpetra
MueLu
MueMex
Kokkos refactor

Developer environment

using git
building
debugging
checking code

Communication

Mailing lists
github issues

Testing

unit-tests
cdash dashboard
nightly tests
memory checks

Clone this wiki locally