set -mllvm -amdgpu-function-calls=true for HIP builds

[BenWibking]

Describe the proposal
We should automatically add -mllvm -amdgpu-function-calls=true to the compiler flags when -DAMREX_GPU_BACKEND=HIP (AMD GPUs). This works around compiler bugs for large GPU kernels (e.g., chemistry networks).

Describe alternatives you've considered
Alternatively, don't use AMD GPUs.

Additional context
See also: AMReX-Astro/Microphysics#1489.

[psharda]

I like the alternative you have considered.

[psharda]

On a more serious note, this is easy to do for a single machine (like moth) because one can just update the corresponding profile. However, it does not seem to work in general because amrex sets amdgpu-function-calls=false in its CMake files, which does not seem to be overridden if we set the opposite in the CMake for quokka. This is the case for Setonix -- I had to go into quokka/extern/amrex and manually change it to true, which is of course sub-optimal.

[BenWibking]

On a more serious note, this is easy to do for a single machine (like moth) because one can just update the corresponding profile. However, it does not seem to work in general because amrex sets amdgpu-function-calls=false in its CMake files, which does not seem to be overridden if we set the opposite in the CMake for quokka. This is the case for Setonix -- I had to go into quokka/extern/amrex and manually change it to true, which is of course sub-optimal.

Can you open an issue on the AMReX GitHub repo for this problem? Allowing downstream codes to change this is something they need to fix in their CMake settings.

[BenWibking]

The underlying problem is that the AMD GPU compiler does register allocation incorrectly: https://discourse.llvm.org/t/the-current-state-of-spilling-function-calls-and-related-problems/2863.

Unfortunately, there is nothing we can do about this until they rewrite their compiler.