Kokkos Refactor in MueLu (and Xpetra)

MueLu relies heavily on Xpetra to provide a multigrid package that can work both with Epetra and Tpetra underlying linear algebra. Xpetra was created to provide a unique interface to both Epetra and Tpetra objects. That being said MueLu also aims at performing well on next generation platforms (manycore, GPUs, the next big thing). To do so we decided to use Kokkos for code portability and also because Tpetra is using Kokkos. However Epetra does not support Kokkos and this means that some code in MueLu will require Tpetra as the underlying linear algebra package.

What are the build/run requirements of refactored code

The goal of this section is to defined constraints that the refactored code needs to comply with but also what additional freedom is granted in to refactored code vs. "regular" code.

Build requirements

The goal is to have all of MueLu, including the kokkos refactored code, to build with both the Epetra stack and the Tpetra stack. In practice this means that a good chunk of the refactored code needs to be guarded with the following pre-processor instructions:

in Xpetra

#ifdef HAVE_XPETRA_KOKKOS_REFACTOR
#ifdef HAVE_XPETRA_TPETRA

in MueLu

#ifdef HAVE_MUELU_KOKKOS_REFACTOR

this allow us to write code that uses Kokkos but builds when Tpetra is disabled. One problem arising from this approach is that Xpetra objects typically need to implement stubs that throw errors when the underlying linear algebra is Epetra. It is also our philosophy that the Epetra path in Xpetra should not depend on Tpetra at all and vice-versa.

Run requirements

The Kokkos refactor branches of Xpetra and MueLu are not meant to run with an underlying Epetra linear algebra, only Tpetra supports fully the new features enable with the Kokkos refactor. For this reason examples, tests and unit tests that employ the Kokkos refactor are not tested with linAlgebra=Epetra. However some features might work with Epetra and OpenMP even though they are not officially supported.