MYNN SFC OpenACC Acceleration

[grantfirl]

PR Author Checklist:

• I have linked PR's from all sub-components involved in section below.

• I am confirming reviews are completed in ALL sub-component PR's.

• I have run the full RT suite on either Hera/Cheyenne AND have attached the log to this PR below this line:

  • LOG:
    RegressionTests_hera.log

• I have added the list of all failed regression tests to "Anticipated changes" section.

• I have filled out all sections of the template.

Description

See ufs-community/ccpp-physics#97 for a complete description.

Overview from the developer:

With very minimal changes to the original code of the scheme, the MYNN surface scheme has been enhanced with OpenACC statements which introduce the capability for offloading computational execution to OpenACC-supported accelerator devices (e.g. Nvidia GPUs). Since the scheme operates by looping multiple times over independent vertical columns, the overall computational strategy maps well to GPU hardware where multiple iterations of each loop can be run in parallel with SIMD methods. Data movement has been optimized to ensure data transfers from host memory to device memory are limited as data movement is a significant source of latency when performing offloading to accelerator devices. Performance increases on a GPU ranged from a 3.3x slowdown to a 41.9x speedup versus CPU execution (See the Performance section for more information).

Linked Issues and Pull Requests

Associated UFSWM Issue to close

Subcomponent Pull Requests

NOAA-EMC/fv3atm#693
ufs-community/ccpp-physics#97

Blocking Dependencies

None

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Anticipated Changes

Input data

• No changes are expected to input data.
• Changes are expected to input data:
  • New input data.
  • Updated input data.

Regression Tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

Tests effected by changes in this PR:

Libraries

• Not Needed
• Needed
  • Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
  • Add issue link from JCSDA/spack-stack following this item

Code Managers Log

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.
  • N/A

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Jet
  • Gaea
  • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[grantfirl]

I'm rerunning RTs on Hera since the first set ran into memory issues apparently.

[zach1221]

I'm rerunning RTs on Hera since the first set ran into memory issues apparently.

Sounds good @grantfirl . Keep us posted. Once those Hera logs are ready I think we can start working this PR.

[grantfirl]

I'm rerunning RTs on Hera since the first set ran into memory issues apparently.

Sounds good @grantfirl . Keep us posted. Once those Hera logs are ready I think we can start working this PR.

@zach1221 OK, RTs on Hera finished successfully, and I attached the log in the description.

[zach1221]

ORTs passed. Jenkins-ci logs attached.
Jenkins-ci 1891 ORTs.pdf

[epic-cicd-jenkins]

Automated RT Failure Notification
Machine: jet
Job: RT
[RT] Repo location: /lfs4/HFIP/hfv3gfs/role.epic/autort/pr/1508030375/20230912235213/ufs-weather-model
Please make changes and add the following label back: jet-RT

[jkbk2004]

Jet system issue. System is very crowded. Skip jet!

[zach1221]

Jet system issue. System is very crowded. Skip jet!

Sure. I should have the hera logs ready soon.

[jkbk2004]

Jet queue is not moving even today. So confirm to skip jet. @grantfirl we can start merging process. @zach1221 FYI

[zach1221]

Jet queue is not moving even today. So confirm to skip jet. @grantfirl we can start merging process. @zach1221 FYI

I've followed up with Grant on ccpp-physics pr #97 .

[jkbk2004]

Jet queue is not moving even today. So confirm to skip jet. @grantfirl we can start merging process. @zach1221 FYI

I've followed up with Grant on ccpp-physics pr #97 .

@zach1221 Thanks!

[grantfirl]

The ccpp-physics PR is merged and the fv3atm PR is updated and ready for merge. @zach1221 @jkbk2004