[develop] Make get_obs tasks day-dependent in workflow; other improvements and bug fixes

[gsketefian]

DESCRIPTION OF CHANGES:

This PR fixes multiple bugs in the verification (vx) and other parts of the SRW App, the main one being that the get_obs tasks as well as some of the vx pre-processing tasks currently do not work for an experiment with multiple cycles if those cycles overlap in time (bug discovered by @michelleharrold and @willmayfield). Fixes and changes made by this PR are described in more detail below.

Changes related to get_obs tasks:

• Make get_obs tasks in the ROCOTO workflow obs-day-based as opposed to cycle-based. Thus, for each day for which obs are needed for vx (and for each obs type that is needed for vx), there is now a get_obs workflow task.

• Move the functionality in the ex-script exregional_get_verif_obs.sh to the new python script get_obs.py. The new exregional_get_verif_obs.sh is now a very short script that just calls get_obs.py.

• The new get_obs.py script, along with changes to setup.py to calculate the times at which various types of obs need to be retrieved, ensure that no clobbering of retrieved obs files occurs (this currently does occur if cycles overlap).

• In config_defaults.yaml, introduce new variables specifying the obs availability interval for each of the four obs types (CCPA, NOHRSC, MRMS, NDAS) that might be retrieved. These variables are [CCPA|NOHRSC|MRMS|NDAS]_OBS_AVAIL_INTVL_HRS.

• setup.py now checks that multiple consistency constraints and requirements on the temporal vx parameters in the SRW configuration file (e.g. the accumulation periods, the obs availability intervals, the forecast output interval) are satisfied that would otherwise cause errors in the workflow. (setup.py calls functions in the (renamed) script set_cycle_and_obs_timeinfo.py to run these checks.) If such inconsistencies exist, the parameters are either adjusted to fix them or, if that is not possible, the experiment generation process is stopped.

• In config_defaults.yaml, introduce flags that determine whether or not to delete the raw obs directories and files that the get_obs tasks create after the raw obs have been copied/moved/renamed to their final/processed locations. These new flags are REMOVE_RAW_OBS_[CCPA|NOHRSC|MRMS|NDAS].

• In config_defaults.yaml, move the base directories for the obs, i.e. [CCPA|NOHRSC|MRMS|NDAS]_OBS_DIR, from the platform section to the verification section so that they are near the METplus obs file name templates (OBS_...FN_TEMPLATE) for which they serve as base directories.

• The processed/final files that the get_obs tasks create are now located and named as specified by the combination of the obs base directory (e.g. CCPA_OBS_DIR) and the obs file name template (e.g. CCPA_APCP_FN_TEMPLATE). Currently, the processed/final file that the get_obs tasks first look for are, say for CCPA, {CCPA_OBS_DIR}/{CCPA_APCP_FN_TEMPLATE}, but if these files don't exist and the obs need to be retrieved, the retrieved and processed file names are not necessarily given by this template. With this PR, the raw files are renamed and moved after retrieval to ensure they are located at {..._OBS_DIR}/{..._FN_TEMPLATE}.

• Retrieve only 6-hourly NOHRSC snow accumulation obs, not 24-hourly accumulations. Currently, 24-hour accumulated obs are also retrieved (although there doesn't seem to be a WE2E test for it).

• Modify the configuration file parm/data_locations.yml for retrieve_data.py to extract all files in an archive at a time (i.e. per call to retrieve_data.py) instead of extracting only one obs file out of an archive for each call to retrieve_data.py. This speeds up the data retrieval significantly since a large portion of the get_obs tasks' wallclock time is spent establishing a connection to HPSS.

• Modify parm/data_locations.yml to account for the change in prebpufr (NDAS) obs file names on May 22, 2024. This is currently causing get_obs_ndas tasks to fail for cycles at or after this date. (Bug found by @michelleharrold.)

• Fix vx task dependencies to work with new obs-day-based get_obs tasks. Now, all get_obs tasks (i.e. for all obs days) for a given obs type must be complete before any vx tasks for that obs type can launch. This doesn't cause any significant delay because the get_obs tasks run in parallel and get at most one day's worth of obs.

Changes related to vx pre-processing tasks (PcpCombine_obs and Pb2nc_obs):

• Add PcpCombine_obs tasks for both 6-hour and 24-hour accumulations of NOHRSC obs. The one for 6-hour accumulation simply converts the grib2 obs files to NetCDF, while the one for 24-hour accumulation adds the 6-hour grib2 obs to obtain a NetCDF file for 24-hour obs accumulations.

• Place all output from PcpCombine_obs tasks (both for CCPA and NOHRSC) under the cycle directories, just as is done for the analogous PcpCombine_fcst tasks for forecasts. This is because accumulations, even for obs, depend on the start time of the cycle, e.g. 6-hour CCPA accumulations needed to verify a set of forecasts that start at 00Z will be different than 6-hour CCPA accumulations needed to verify a set of forecasts that start at 03Z. (Currently, the output files from these tasks are placed in the metprd directory under the main experiment directory without consideration for the start times of the accumulations.)

• Make the Pb2nc_obs task for NDAS obs-day-dependent (unlike the PcpCombine_obs tasks, which are cycle-dependent). This can be done because unlike accumulations, the result of the Pb2nc_obs task does not depend on the starting time of the cycles; it only depends on a given valid time. Also, keep the output of the Pb2nc_obs task in the cycle-independent directory metprd directly under the main experiment directory.

Small, self-contained bug fixes and improvements:

• Move evaluation of METplus time strings out of what used to be set_vx_fhr_list.sh (now renamed to set_leadhrs.sh) and into a new bash script (bash_utils/eval_METplus_timestr_tmpl.sh) to make it easier to change this functionality to python later on.

• Allow WE2E test names to include dots since dots (like underscores) are handy to use as separators in the test name.

• Add the two new SRW config parameters VX_CONFIG_[DET|ENS]_FN in config_defaults.yaml that specify the yaml configuration files to use for deterministic and ensemble verification. The default values for these are the files vx_config_[det|ens]_fn.yaml in parm/metplus. These parameters allow a user to specify other user-created yaml files in this directory to use for the vx configuration so that the default files, which are under version control, do not have to be changed.

• Change some metatask and task names for clarity and consistency.

• Add an option to mrms_pull_topofhour.py to not assume that there is a valid-date subdirectory under the specified source directory and to not add such a subdirectory under the specified output directory when generating output. This is handy when calling this script from the new get_obs.py script.

• Fix bug in parm/wflow/verify_det.yaml so that all tasks have a cycldefs statement by default. This bug was causing GridStat workflow tasks for CCPA and NOHRSC obs to be created for cycles not defined for the workflow (these extraneous cycles probably correspond to the default set of cycles that a task gets assigned by ROCOTO when it does not contain an explicit cycledefs statement). (Bug found by @michelleharrold, solution by @mkavulich.)

• Fix bug in scripts/exregional_run_met_gridstat_or_pointstat.sh to append a string for the cycle date ("_YYYYMDDHH") to the name of the metplus log file for deterministic GridStat and PointStat tasks. This was causing the metplus log file for GridStat for a given cycle tasks to be overwritten by those for other cycles. (Bug found by @michelleharrold.)

• Fix bug in parm/default_workflow.yaml in "cycled_from_second" section in which the starting YYYYMMDDHH value of the cycledef can contain an HH value that is larger than 23. This currently happens because this HH is obtained directly from INCR_CYCL_FREQ without checking whether that value is less than 24.

• Fix bug In launch_FV3LAM_wflow.sh to change double quotes to single quotes to prevent failure in the interpretation of the command by cron.

New WE2E tests added:

The following new WE2E tests were added to the verification subdirectory under test_configs to test various aspects of the new code:

• get_obs_hpss.do_vx_det.multicyc.cycintvl_07hr_inits_vary_fcstlen_09hr.ncep-hrrr
• get_obs_hpss.do_vx_det.multicyc.cycintvl_11hr_inits_vary_fcstlen_03hr.ncep-hrrr
• get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_00z_fcstlen_03hr.ncep-hrrr
• get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_12z_fcstlen_03hr.nssl-mpas
• get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_12z_fcstlen_48hr.nssl-mpas
• get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_21z_fcstlen_03hr.ncep-hrrr
• get_obs_hpss.do_vx_det.multicyc.cycintvl_96hr_inits_12z_fcstlen_48hr.nssl-mpas
• get_obs_hpss.do_vx_det.singlecyc.init_00z_fcstlen_36hr.winter_wx.SRW

The purpose of each of these is described in the description section of the corresponding test config file.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• derecho.intel
• gaea.intel
• hera.gnu
• hera.intel
• hercules.intel
• jet.intel
• orion.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

Three sets of WE2E tests were conducted on Hera/intel:

1. The "fundamental" suite consisting of:

   • grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
   • grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot
   • grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16
   • grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR
   • grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
   • grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0

2. The existing verification tests, consisting of:

   • `MET_ensemble_verification
   • MET_ensemble_verification_only_vx
   • MET_ensemble_verification_only_vx_time_lag
   • MET_ensemble_verification_winter_wx
   • MET_verification
   • MET_verification_only_vx
   • MET_verification_winter_wx

3. The newly added get_obs/verification tests, consisting of:

   • get_obs_hpss.do_vx_det.multicyc.cycintvl_07hr_inits_vary_fcstlen_09hr.ncep-hrrr
   • get_obs_hpss.do_vx_det.multicyc.cycintvl_11hr_inits_vary_fcstlen_03hr.ncep-hrrr
   • get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_00z_fcstlen_03hr.ncep-hrrr
   • get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_12z_fcstlen_03hr.nssl-mpas
   • get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_12z_fcstlen_48hr.nssl-mpas
   • get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_21z_fcstlen_03hr.ncep-hrrr
   • get_obs_hpss.do_vx_det.multicyc.cycintvl_96hr_inits_12z_fcstlen_48hr.nssl-mpas
   • get_obs_hpss.do_vx_det.singlecyc.init_00z_fcstlen_36hr.winter_wx.SRW

All tests were successful.

DOCUMENTATION:

I am not familiar with the new RST file setup and will need help moving my documentation from comments in the code to the RST files.

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
  Possibly; I'm not sure what exactly the documentation requirements are currently.
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

CONTRIBUTORS (optional):

@michelleharrold @mkavulich @JeffBeck-NOAA @willmayfield

[gspetro-NOAA]

@gsketefian I noticed that the tech doc test is failing. You can follow the instructions in the slides here or in the documentation to update. I'm hoping it will be pretty straightforward now that I've put together the instructions, but let me know if you need help.

[MichaelLueken]

The verification WE2E tests have successfully passed on Hera Intel:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
vx-det_multicyc_fcst-overlap_ncep-hrrr_20241113161443              COMPLETE               3.69
vx-det_multicyc_long-fcst-overlap_nssl-mpas_20241113161445         COMPLETE              12.29
MET_verification_only_vx_20241113161447                            COMPLETE               0.28
vx-det_multicyc_first-obs-00z_ncep-hrrr_20241113161449             COMPLETE               0.78
vx-det_multicyc_last-obs-00z_ncep-hrrr_20241113161450              COMPLETE               0.85
vx-det_multicyc_no-fcst-overlap_ncep-hrrr_20241113161452           COMPLETE               1.89
MET_verification_20241113161454                                    COMPLETE              14.84
vx-det_multicyc_long-fcst-no-overlap_nssl-mpas_20241113161455      COMPLETE              13.13
MET_ensemble_verification_only_vx_time_lag_20241113161457          COMPLETE               4.33
MET_ensemble_verification_20241113161500                           COMPLETE              47.35
vx-det_multicyc_no-00z-obs_nssl-mpas_20241113161502                COMPLETE               0.78
MET_verification_winter_wx_20241113161504                          COMPLETE              24.39
vx-det_long-fcst_custom-vx-config_gfs_20241113161506               COMPLETE               0.70
vx-det_long-fcst_custom-vx-config_aiml-fourcastnet_20241113161507  COMPLETE               0.64
vx-det_long-fcst_custom-vx-config_aiml-graphcast_20241113161509    COMPLETE               0.59
vx-det_long-fcst_custom-vx-config_aiml-panguweather_2024111316151  COMPLETE               0.63
MET_ensemble_verification_winter_wx_20241113161513                 COMPLETE             160.75
vx-det_long-fcst_winter-wx_SRW-staged_20241113161515               COMPLETE               1.06
MET_ensemble_verification_only_vx_20241113161517                   COMPLETE               1.15
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             290.12

as well as the verification tests on Hera GNU:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
vx-det_multicyc_fcst-overlap_ncep-hrrr_20241113173041              COMPLETE               7.32
vx-det_multicyc_long-fcst-overlap_nssl-mpas_20241113173044         COMPLETE              26.93
MET_verification_only_vx_20241113173045                            COMPLETE               0.31
vx-det_multicyc_first-obs-00z_ncep-hrrr_20241113173047             COMPLETE               1.30
vx-det_multicyc_last-obs-00z_ncep-hrrr_20241113173048              COMPLETE               1.39
vx-det_multicyc_no-fcst-overlap_ncep-hrrr_20241113173050           COMPLETE               3.17
MET_verification_20241113173052                                    COMPLETE              18.97
vx-det_multicyc_long-fcst-no-overlap_nssl-mpas_20241113173053      COMPLETE              27.87
MET_ensemble_verification_only_vx_time_lag_20241113173055          COMPLETE              12.94
MET_ensemble_verification_20241113173058                           COMPLETE              72.61
vx-det_multicyc_no-00z-obs_nssl-mpas_20241113173059                COMPLETE               1.30
MET_verification_winter_wx_20241113173101                          COMPLETE              32.84
vx-det_long-fcst_custom-vx-config_gfs_20241113173102               COMPLETE               0.79
vx-det_long-fcst_custom-vx-config_aiml-fourcastnet_20241113173104  COMPLETE               0.78
vx-det_long-fcst_custom-vx-config_aiml-graphcast_20241113173106    COMPLETE               0.78
vx-det_long-fcst_custom-vx-config_aiml-panguweather_2024111317310  COMPLETE               0.77
MET_ensemble_verification_winter_wx_20241113173110                 COMPLETE             212.75
vx-det_long-fcst_winter-wx_SRW-staged_20241113173112               COMPLETE               1.41
MET_ensemble_verification_only_vx_20241113173114                   COMPLETE               1.55
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             425.78

[gsketefian]

@gspetro-NOAA I fixed up as much of the docs as I could, but I ran into one issue that I don't have a good fix for. The problem is that the scripts get_obs.py and eval_metplus_timestr_tmpl.py (and, indirectly, set_leadhrs.py) need the environment variable METPLUS_ROOT to import a module from METplus, but METPLUS_ROOT is not available when making the docs (maybe we have to load the modules for running vx). First, I got errors like:

WARNING: autodoc: failed to import module 'eval_metplus_timestr_tmpl'; the following exception was raised:
Traceback (most recent call last):
  File "/home/Gerard.Ketefian/miniconda3/lib/python3.12/site-packages/sphinx/ext/autodoc/importer.py", line 58, in import_module
    return importlib.import_module(modname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/Gerard.Ketefian/miniconda3/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/scratch2/BMC/fv3lam/Gerard.Ketefian/AGILE3/ufs-srweather-app/ush/eval_metplus_timestr_tmpl.py", line 7, in <module>
    sys.path.append(os.environ['METPLUS_ROOT'])
                    ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "<frozen os>", line 685, in __getitem__
KeyError: 'METPLUS_ROOT'

Then I thought to define METPLUS_ROOT in Makefile itself (for the specific case of Hera; see my most recent changes in Makefile). That got me around finding METplus, but now I get errors like this:

WARNING: autodoc: failed to import module 'eval_metplus_timestr_tmpl'; the following exception was raised:
No module named 'dateutil'

I decided to stop there and see if you have any ideas. I'll tag @mkavulich too since he's the original author of the use of METPLUS_ROOT in eval_metplus_timestr_tmpl.py.

[gsketefian]

@MichaelLueken FYI that I made that change of the wallclock time in verify_ens.yaml. I'm now waiting to hear back from @gspetro-NOAA and/or @mkavulich regarding a fix to the doc build issue I ran into (see my previous message). Thanks.

[MichaelLueken]

@gsketefian and @gspetro-NOAA -

Unfortunately, it looks like the modification to doc/Makefile won't work for RtD and the GHA test. In the Doc Tests GHA, I see the following in the Build documentation section:

WARNING: autodoc: failed to import module 'eval_metplus_timestr_tmpl'; the following exception was raised:
No module named 'metplus' [autodoc.import_object]
WARNING: autodoc: failed to import module 'get_obs'; the following exception was raised:
No module named 'metplus' [autodoc.import_object]
WARNING: autodoc: failed to import module 'set_leadhrs'; the following exception was raised:
No module named 'metplus' [autodoc.import_object]

Additionally, the technical documentation for the above three scripts in Read the Docs are empty.

The modification made to doc/Makefile will allow the documentation to find metplus on Hera, but not on any other machine or on the GitHub runner, leading to the warning message.

In the new python scripts, I see:

try:
    sys.path.append(os.environ['METPLUS_ROOT'])
except:
    print("\nERROR ERROR ERROR\n")
    print("Environment variable METPLUS_ROOT must be set to use this script\n")
    raise
from metplus.util import string_template_substitution as sts

Since there is now a dependency on METplus, will we need to include both MET and METplus in the environment used to run the Doc Tests GHA?

It looks like dateutil is a METplus requirement. So, if MET and METplus are required, then python-dateutil will also be required within the build environment for technical documentation.

[gspetro-NOAA]

@MichaelLueken @gsketefian Is there a way to put the try/except statement under if __name__ == "__main__":? @maddenp helped me with a similar error. Based on what he said, Sphinx's autodoc imports the Python module it is trying to produce docs for. When import happens, all the top-level code (outside of functions, classes, etc.) has to be executed, so lines like

try:
    sys.path.append(os.environ['METPLUS_ROOT'])
except:
    print("\nERROR ERROR ERROR\n")
    print("Environment variable METPLUS_ROOT must be set to use this script\n")
    raise
from metplus.util import string_template_substitution as sts

get executed. This causes an error precisely because the 'METPLUS_ROOT' directory is not available. Putting this code under if __name__ == "__main__": might be one way to solve the problem. It seems like you can put the try/except block there and leave the import statement below where it is. (Note that you'll likely have to do this for every script where the try/except statement is outside if __name__ == "__main__":.) If the import statement alone causes a problem, it can be added to the autodoc_mock_imports list in conf.py.

[maddenp-noaa]

Your linter might complain about an import statement somewhere other than at the top of the file, but there are good reasons to suppress linter complaints and this might be one of them.

Alternatively, with from importlib import import_module at the top of the module, you can do sts = import_module("metplus.util.string_template_substitution") close to where sts is needed to do a just-in-time dynamic import.

[MichaelLueken]

Thanks, @gspetro-NOAA and @maddenp-noaa, for options to try to get the technical documentation to work!

I think the main issue here is that it is impossible to set metplus if the environmental variable, METPLUSROOT, isn't available. The modification that @gsketefian made to doc/Makefile allows us to get around the environmental variable issue, but then metplus isn't available.

As @gspetro-NOAA noted, adding metplus to the autodoc_mock_imports list in conf.py allowed the build to successfully complete.

@gsketefian -

In doc/conf.py, please add metplus to the autodoc_mock_imports list (line 255):

autodoc_mock_imports = ["f90nml","cartopy","mpl_toolkits.basemap","fill_jinja_template",
   "matplotlib","numpy","uwtools","mpl_toolkits","metplus",
   ]

This is how I was able to get around the issue with metplus and dateutil.

[gsketefian]

@MichaelLueken @gspetro I followed your suggestion of adding "metplus" to autodoc_mock_imports in conf.py, and that worked. Also, I restored theMakefile for the documentation to its original version and instead defined the environment variable METPLUS_ROOT in conf.py. There, I set it to just a blank string since it just needs to exist; it's not used directly to load METplus (its contents get added to the system path, but that path already contains the location of METplus due to the addition of "metplus" to autodoc_mock_imports). Hopefully the doc test will succeed now.

[MichaelLueken]

@gsketefian -

Thank you for addressing the documentation issues! The Test Docs GHA test is passing now and the technical documentation for the new ush/ python scripts are being generated in RtD!

Since the new verification WE2E tests are successfully running on Hera GNU and Hera Intel, I will approve these changes and start the automated testing.

[MichaelLueken]

The rerun of the Jenkins tests on Derecho have successfully completed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km_20241118074436                      COMPLETE              41.66
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              86.90
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE              15.68
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024111807444  COMPLETE              67.76
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             744.00
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20241118  COMPLETE              66.49
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              24.85
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              19.42
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024111807444  COMPLETE              38.33
pregen_grid_orog_sfc_climo_20241118074449                          COMPLETE              20.17
specify_template_filenames_20241118074451                          COMPLETE              20.97
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1146.23

The rest of the tests successfully passed on Friday evening.

Since the last conversations have been resolved, I will now move forward with merging this PR.

[gsketefian]

@MichaelLueken Thanks for shepherding this through!