Fix missing MPI in custom-matrix-format example

[MarcelKoch]

This PR fixes the missing MPI in the custom-matrix-format example when using the MPI wrapper as CXX compiler. There are actually two ways of fixing this implemented here:

1. remove the <ginkgo/ginkgo.hpp> include in the cuda file. Now the MPI headers are not required when compiling the cuda code anymore.
2. Use the CMake option `MPI_ASSUME_NO_BUILTIN_MPI to disable using MPI from the compiler wrapper directly.

Both of these fix the issue. I think having both of them is a good idea, because 1. the include is just not necessary, and 2. this would allow users to include <ginkgo/ginkgo.hpp> in device code.

[MarcelKoch]

Fixes part of #1367

[yhmtsai]

sorry, I do not get the usage of MPI_ASSUME_NO_BUILTIN_MPI.
could you elaborate it on two cases from gcc and mpicc?

[upsj]

LGTM! One minor nit (we already have way too many configuration options)

[upsj]

Do we need to make this controllable at all? It seems to be relevant only in very limited circumstances, where either mpiexec is missing from the environment, or the user has a MPI compiler wrapper loaded that is not the one specified by CXX_COMPILER

[MarcelKoch]

TBH, I'm not 100% sure how cmake finds MPI with this option on. Perhaps there are some weird cases, where it doesn't use the same MPI as the MPI wrapper would have. So I added the option to turn it off. I also marked it as advanced to indicate that users generally need not to worry about it.

[MarcelKoch]

@yhmtsai If you set CXX=mpicxx then compiling with it doesn't require adding include paths or libraries for MPI. That gets picked up by cmake, when it finds the MPI package. So, on the cmake level, linking to MPI doesn't require any additional flags. This will lead to errors, when something is compiled not using mpicxx, but still including/linking to MPI. The MPI_ASSUME_NO_BUILTIN_MPI flag tells cmake to somehow ignore the MPI compiler wrapper and add the needed include path and libraries regardless.

[upsj]

This whole setting has me a bit confused - theoretically, CMake should interrogate the compiler wrapper for the location of the mpi.h file and add the include flag to the MPI target, but that doesn't seem to be happening here? From FindMPI.cmake: # Given the new design of FindMPI, INCLUDE_DIRS will always be located, even under implicit linking.

[MarcelKoch]

@upsj The comment from FindMPI.cmake is not in the official documentation, so I don't think we can rely on that. I tried again (without MPI_ASSUME_NO_BUILTIN_MPI) with the latest cmake release (3.26.4) and they still not set the include directories.
So maybe that is a bug in cmake, or the comment is just not very clear.

[upsj]

@MarcelKoch yes, I'd really like to get a full trace of the execution, because this seems like a bug to me. For this to be fixed, we need MPI_CUDA_INCLUDE_DIRS MPI_CXX_COMPILER_INCLUDE_DIRS to be set, which seems to be missing.

[MarcelKoch]

@upsj I will open an cmake issue to see what's going on. From a brief look at the source, you're right, it always sets the include directories, but if a compiler wrapper is used, the value of the used variable is empty. So the source code comment is moot.

[MarcelKoch]

Here is the issue, let's see what happens: https://gitlab.kitware.com/cmake/cmake/-/issues/25085

[upsj]

I no longer think this fix is necessary, instead we should be propagating the C++ compiler as CUDA host compiler correctly.

there is a more general fix

[MarcelKoch]

Fixed by #1368 and #1205