Update modules on alcf polaris

[amametjanov]

Update modules on alcf polaris. Also,

• update queues
• use cray wrappers for serial-gnu
• update cmake for gnugpu builds
• add eamxx cmake machine file
• run small eam and mpas-o cases on 1 polaris node
• add MOAB_ROOT env-var

Fixes #6422

[BFB]

[amametjanov]

Testing:

• e3sm_gpucxx + gnugpu: 2 of 2 PASS -- https://my.cdash.org/viewTest.php?buildid=2818477
• e3sm_gpuacc + nvidiagpu: 2 of 2 PASS -- https://my.cdash.org/viewTest.php?buildid=2828733
• e3sm_integration + gnu: 134 of 136 PASS -- https://my.cdash.org/viewTest.php?buildid=2821061

[rljacob]

Adding @bartgol since this touches some code in eamxx and so testing needs to run on SNL.

[bartgol]

The code in eamxx is a mach file, so testing on SNL machine, albeit triggered, is irrelevant. Still, I'll approve and retrigger, so we get more shiny green check marks...

[gsever]

I would like to note a few points regarding this Polaris update.

1. Is the default build setting “-O2” for CUDA instead of “-O3”? “F2010-SCREAMv1” compset used to build in ~400s vs. now ~600s with the current settings on a compute node with 32-cores.

2. Despite the working status of “gnugpu” compiler option, shouldn’t it be specified more explicitly in config_machines.xml, instead of

<modules compiler="gnugpu">
  <command name="load">nvhpc-mixed</command>

      <modules compiler="gnugpu">
	<command name="load">PrgEnv-gnu/8.5.0</command>
	<command name="load">cudatoolkit-standalone/12.2.2</command>
      </modules>

Likewise, listing other modules in the file with specific versions to ease testing and reproduction.

3. Queue specs in config_batch.xml doesn’t seem to target longer production runs with max allowed runtime of 3h - https://docs.alcf.anl.gov/polaris/running-jobs/

4. While for testing purposes there is merit to executing CPU tests on a GPU-specific machine, it would be much more useful if a configuration is deployed on ALCF’s CRUX for practical CPU-based production runs. It may be more helpful to see other GPU tests deployed on Polaris instead (eg, E3SM_EAMXX builds).