Enable parallel metp jobs and fix race condition with the gfscleanup job

[DavidHuber-NOAA]

Description

This brings in a change to EMC_verif-global that offsets the start time of parallel instances when running metp jobs. This prevents Python from attempting to create the same directory in multiple instances.
Simultaneously, this also fixes an issue with the gfscleanup job potentially running before the metp jobs.
Resolves #2906
Resolves #2899

Type of change

• New feature (adds functionality)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)
  • EMC verif-global PR Workaround: Sleep each METplus job by jobnum seconds to avoid race EMC_verif-global#135
  • GDAS
  • GFS-utils
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils
  • UFS-weather-model
  • wxflow

How has this been tested?

Standalone test on Orion with 40 parallel instances over a 30-cycle span.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes

[DavidHuber-NOAA]

I added a one-line fix to enkf.yaml.j2 that @AntonMFernando-NOAA found. This code has not been used as the eomg jobs are not runnable at this time, but it will help when others try to use code samples.

[DavidHuber-NOAA]

I finished running tests on Hera. All metp jobs passed. Marking this PR as ready for review.

[emcbot]

CI Update on Wcoss2 at 09/23/24 01:50:13 PM
============================================
Cloning and Building global-workflow PR: 2907
with PID: 116296 on host: clogin03

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Mon Sep 23 13:56:25 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 09/23/24 02:34:19 PM
Case setup: Completed for experiment C48_ATM_fde2acb4
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_fde2acb4
Case setup: Skipped for experiment C48_S2SWA_gefs_fde2acb4
Case setup: Completed for experiment C48_S2SW_fde2acb4
Case setup: Completed for experiment C96_atm3DVar_extended_fde2acb4
Case setup: Skipped for experiment C96_atm3DVar_fde2acb4
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_fde2acb4
Case setup: Completed for experiment C96C48_hybatmDA_fde2acb4
Case setup: Completed for experiment C96C48_ufs_hybatmDA_fde2acb4

[emcbot]

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_fde2acb4 *** SUCCESS *** at 09/23/24 03:56:07 PM
Experiment C48_S2SW_fde2acb4 *** SUCCESS *** at 09/23/24 04:14:13 PM
Experiment C96C48_hybatmDA_fde2acb4 *** SUCCESS *** at 09/23/24 05:07:22 PM
Experiment C96C48_hybatmaerosnowDA_fde2acb4 *** SUCCESS *** at 09/23/24 05:42:27 PM
Experiment C96C48_ufs_hybatmDA_fde2acb4 *** SUCCESS *** at 09/23/24 07:14:22 PM
Experiment C96_atm3DVar_extended_fde2acb4 *** SUCCESS *** at 09/24/24 02:35:40 AM