Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable parallel metp jobs and fix race condition with the gfscleanup job #2907

Open
wants to merge 9 commits into
base: develop
Choose a base branch
from
Open

Enable parallel metp jobs and fix race condition with the gfscleanup job #2907

wants to merge 9 commits into from

Conversation

DavidHuber-NOAA
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA commented Sep 12, 2024 •
edited
Loading

Description

This brings in a change to EMC_verif-global that offsets the start time of parallel instances when running metp jobs. This prevents Python from attempting to create the same directory in multiple instances.
Simultaneously, this also fixes an issue with the gfscleanup job potentially running before the metp jobs.
Resolves #2906
Resolves #2899

Type of change

  • New feature (adds functionality)

Change characteristics

How has this been tested?

Standalone test on Orion with 40 parallel instances over a 30-cycle span.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes

@DavidHuber-NOAA
Copy link
Contributor Author

I added a one-line fix to enkf.yaml.j2 that @AntonMFernando-NOAA found. This code has not been used as the eomg jobs are not runnable at this time, but it will help when others try to use code samples.

@DavidHuber-NOAA DavidHuber-NOAA mentioned this pull request Sep 13, 2024
Open
@DavidHuber-NOAA DavidHuber-NOAA changed the title Enable parallel metp jobs Enable parallel metp jobs and fix race condition with the gfscleanup job Sep 13, 2024
@DavidHuber-NOAA
Merge remote-tracking branch 'origin/develop' into feature/parallel_metp
91c0ffe 
* origin/develop:
  Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917)
  Cleanup job for GEFS (NOAA-EMC#2919)
  Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912)
  add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922)
  Support gefs C48 on Azure (NOAA-EMC#2881)
  Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)
@DavidHuber-NOAA DavidHuber-NOAA marked this pull request as ready for review September 16, 2024 12:59
@DavidHuber-NOAA
Copy link
Contributor Author

I finished running tests on Hera. All metp jobs passed. Marking this PR as ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@malloryprow malloryprow malloryprow approved these changes

@RussTreadon-NOAA RussTreadon-NOAA Awaiting requested review from RussTreadon-NOAA

@WalterKolczynski-NOAA WalterKolczynski-NOAA Awaiting requested review from WalterKolczynski-NOAA

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Re-enable parallel metp jobs Incorrect gfs job dependencies caused gfsmetpg2g1 failure
3 participants
@DavidHuber-NOAA @WalterKolczynski-NOAA @malloryprow