Optimal Hera node configuration for running C1152 grid -- layout and quilting I/O groups in config.ufs

[mdtoyNOAA]

I am trying to run the 'gfsfcst' task with the C1152 grid on Hera but am having trouble getting the proper node configuration. The model immediately crashes with "Out of Memory" error messages. Can someone please advise on what the proper settings should be for these variables:
layout_x_gfs
layout_y_gfs
WRITE_GROUP_GFS
WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS

Thank you.

[JessicaMeixner-NOAA]

@mdtoyNOAA I'm not sure about the full set of settings. That being said, I recently had memory errors with C768 on hera and having WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS=20 was the key to solving that. Might also try more threads for the atm component?

[mdtoyNOAA]

@JessicaMeixner-NOAA I also had memory errors with C768 and got it to work with
WRITE_GROUP_GFS=1
WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS=16

So far, with C1152 I've tried up to:
WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS=24 But I will go higher (and maybe increase the write groups to 2) to see if I can get it to work. Thank you.

BTW, I'm able to get the model to integrate (without I/O) with layout_GFS=16,16

[WalterKolczynski-NOAA]

The workflow team would recommend not trying to run C1152 on Hera at all. We recommend using one of the bigger supported machines (preferably WCOSS). If my math is correct, the configuration you tried that didn't work is already >10% of the entire Hera compute for atm-only. You will likely struggle to get it working, struggle to get it through the queue, and then if you do manage to run forecasts, a few will quickly tank the fairshare for the whole project.

[mdtoyNOAA]

I was able to get a successful test run on Hera with output provided using WRITE_GROUP_GFS=1 and WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS=32 (I'm going to see if I can get away with fewer write-tasks). The total nodes I requested was 173. I am only doing 24-hour forecasts and the wall clock time for each run would be no more than 1 hour. I think this should be okay, but please let me know if you disagree. I don't have access to WCOSS.

[WalterKolczynski-NOAA]

If you've got it working, then it's between you and the portfolio manager. But it is not a resolution we currently support in workflow.

[mdtoyNOAA]

Sounds good. I won't be running the full workflow -- I'll only be running the 'gfsfcst' task and analyzing the raw netCDF files myself.

[aerorahul]

Is this conversation resolved with the tag for HR3 prototype?

[JessicaMeixner-NOAA]

The "optimal" configuration will likely be an ongoing discussion but I do think @mdtoyNOAA can run now, so closing for now is likely okay. It can always be re-opened.