(GTL DEBUG: 0) cuIpcGetMemHandle: invalid argument, CUDA_ERROR_INVALID_VALUE on polaris

[ndkeen]

After #6101 which brings in kokkos 4.2, we see runtime error with a test like:
ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.pm-gpu_gnugpu
(also on similar machine ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.muller-gpu_gnugpu)

0: (GTL DEBUG: 0) cuIpcGetMemHandle: invalid argument, CUDA_ERROR_INVALID_VALUE, line no 97

A fix that seems to work is to add this build flag:
Kokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC=OFF

This will also be needed for scream repo.
@bartgol @mahf708

[bartgol]

I just opened a quick follow up PR, to clean up some deprecated code issues (they did not cause falls in e3sm bc deprecated code in looks was allowed).

I can add a fix for this in that PR.

[ndkeen]

Fine with me. Just include the change to make the muller-gpu file same as pm-gpu.

[gsever]

Just to note that the same issue shows up running a recent E3SM/SCREAM version on ALCF's Polaris:

(GTL DEBUG: 0) cuIpcGetMemHandle: invalid argument, CUDA_ERROR_INVALID_VALUE, line no 148

Adding -DKokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC=OFF in ./cime_config/machines/cmake_macros/gnugpu_polaris.cmake fixes the runtime error as posted here.

[mahf708]

Do we have a polaris machine entry? reopening to ensure it is added there

edit: looks like we do, so we should add this there.

@gsever could you issue a PR? Or @rljacob can advise who's the machine POC for polaris

[gsever]

Yes, there is Polaris setup in the master, but it fails minimum with this error:

ERROR: module command /usr/share/lmod/lmod/libexec/lmod python load cmake/3.23.2 craype-x86-rome PrgEnv-gnu/8.3.3 failed with message:
Lmod has detected the following error: The following module(s) are unknown: "PrgEnv-gnu/8.3.3" "cmake/3.23.2"

Changes are needed in:

./cime_config/machines/config_machines.xml
./cime_config/machines/config_batch.xml
./cime_config/machines/cmake_macros/gnugpu_polaris.cmake
./externals/ekat/extern/kokkos/bin/nvcc_wrapper

This file should also be included:
./components/eamxx/cmake/machine-files/polaris.cmake

I recall @amametjanov was handling Polaris changes earlier. I could help with further testing.

[mahf708]

@amametjanov re the last item there (file in eamxx/cmake/machine-files): that's also what's needed for anvil jfyi

[ndkeen]

ok to close? or create another issue specific to polaris?

[mahf708]

Will be automatically closed when #6985 is merged