MPICH_GPU_SUPPORT_ENABLED

@xylar , I agree. I will work on setting MPICH_GPU_SUPPORT_ENABLED in config_machines.xml.

For your information, MPICH_GPU_SUPPORT_ENABLED is already set to 1 in config_machines.xml, but only for GPU compilation. It might need to be set to 0 for non-GPU compilation.

I doubt it needs to be set to zero for non-gpu runs but it won't hurt.

Also, I could be wrong but I think the environment variable isn't being set for Frontier yet in config_machine.xml. Is there a PR already that I missed?

@xylar , MPICH_GPU_SUPPORT_ENABLED on Frontier is actually set to 0 if the compiler name includes gpu. I just checked with config_machines.xml. Sorry for the confusion. The Scream team sets MPICH_GPU_SUPPORT_ENABLED to 1, and we are trying to use the machine and compiler configurations from Scream as a base for Frontier. I think we need to check whether we want to always set the environment variable to 1 or set it to 0 when we are not compiling E3SM for GPU runs.

@philipwjones , I think this PR can be merged as it is now, and the MPICH flag can be removed once it is added in E3SM. However, this PR may not be helpful in fixing the issue on Polaris. I don’t see any urgent need for this PR with Omega.

However, this PR may not be helpful in fixing the issue on Polaris.

Correct, this won't make a difference in Polaris because it doesn't source the Omega environment script.

I doubt it needs to be set to zero for non-gpu runs but it won't hurt.

The MPICH_GPU_SUPPORT_ENABLED variable doesn't need to be set to zero for non-GPU runs, but it cannot be set to one. If MPICH_GPU_SUPPORT_ENABLED=1 during a non-GPU run, the code crashes during MPI initialization because the flag causes it to look for the GPU transport layer library, which is not linked during CPU-only builds. Setting it to 0 in the environment script would be helpful, since I've had situations where, for example, I'm using an interactive node on Frontier testing both the GPU and CPU builds and forgotten to explicitly reset MPICH_GPU_SUPPORT_ENABLED to zero when running the CPU build after the GPU build. Putting this in the environment script would be one less step to remember.

@brian-oneill, I still feel like, philosophically, this is not the kind of flag that Omega should be setting at all. This is part of the larger E3SM infrastructure and we need Omega standalone runs to work as closely as possible to its eventual coupled state. Setting environment variables in standalone that don't get set in coupled mode is a dangerous road to go down.

I think it would be okay to pursue setting MPICH_GPU_SUPPORT_ENABLED=0 for CPU runs in an E3SM PR. We could see if there's push-back for some reason.

And anything Omega does differently in standalone mode that E3SM leaves me in a quandary about how to handle this in both mache and Polaris. mache tries to follow E3SM as closely as possible (and the more we do this, the more I can automatically update as E3SM evolves). And Polaris tries not to set any environment variables itself and just use what mache tells it for a given machine, compiler and MPI configuration.

I agree, that all sounds reasonable. Mainly was chiming in to state that I think it should to be set explicitly in either case.

@xylar, @brian-oneill, I agree with both of you. MPICH_GPU_SUPPORT_ENABLED should be set in the E3SM configuration rather than in Omega. I also prefer setting it explicitly to either 0 or 1, as I encountered the same issue Brian described. In fact, I am currently updating the E3SM Frontier configuration, which will most likely include the MPICH_GPU_SUPPORT_ENABLED setting as discussed here. We can then downstream the update to Omega later.