Enable DA cycling on gaea #3032

jswhit2 · 2024-10-23T17:35:24Z

Description

currently DA cycling on gaea is disabled. These changes remove that restriction.

Resolves #3121
Resolves #2985

Many of the changes in this PR are duplicated in #3106. I suggest that PR be merged first.

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? /NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

ran a cycled test with coupled 3dvar (0.5 degree atmosphere and 0.25 degree ocean).

Checklist

CoryMartin-NOAA

Did not test but looks reasonable. This is for C5, do we know how easy it would be to work on C6?

jswhit2 · 2024-10-23T19:39:05Z

c6 is basically the same - there are only subtle differences in some of the system libraries. This means that some of the executables compiled on c5 won't run on c6 (the model does, but GSI doesn't). So, the biggest obstacle to running on c6 will be to update all the build systems to have separate c5/c6 modulefiles.

CoryMartin-NOAA · 2024-10-23T19:40:27Z

Thanks @jswhit2 for the explanation

WalterKolczynski-NOAA

It appears this PR is specifically to enable cycling for the JEDI system? The atm-only GSI-based test at least appears to already be enabled on Gaea.

If this PR enables cycling with the JEDI system, those tests (in ci/cases/pr) should probably be tested and enabled.

WalterKolczynski-NOAA · 2024-10-30T10:54:26Z

env/GAEA.env

+ ;;
+ *)
+    # Some other job not yet defined here
+    echo "WARNING: The job step ${step} does not specify Hercules-specific resources"


Suggested change

echo "WARNING: The job step ${step} does not specify Hercules-specific resources"

echo "WARNING: The job step ${step} does not specify Gaea-specific resources"

@WalterKolczynski-NOAA I have been running the coupled atm/ocean 3dvar, so I wasn't aware it already worked for atm only DA. When running coupled DA without these mods, load_ufsda_modules.sh quits with the message "WARNING: UFSDA NOT SUPPORTED ON THIS PLATFORM".

WalterKolczynski-NOAA · 2024-10-30T11:13:00Z

Should coordinate with @AnilKumar-NOAA as well, who has been working on the Gaea port.

AnilKumar-NOAA · 2024-10-30T12:34:30Z

Yes, I'll port and test on gaea but only after all recent OS upgrades on gaea PR's are merged. We are waiting on that.

DavidBurrows-NCO · 2024-10-30T13:20:00Z

Hi @jswhit2 @WalterKolczynski-NOAA @CoryMartin-NOAA @AnilKumar-NOAA...Walter is correct, the C96_atm3DVar test case PR2885 was merged for Gaea-C5. However, with the recent C5 OS upgrade, all submodules within the global-workflow fail to build. After talking with Rahul, we decided to submit PRs to build submodules on both C5 and C6 and to follow the filename updates of the ufs-wx-model PR2448, e.g., having a gaeac5 and gaeac6 module files, having detect machine scripts return gaeac5 or gaeac6, etc. I have submitted PRs for Gaea-C5 and C6 builds for ufs-utils, gfs-utils, upp, gsi, gsi-monitor, and gsi-utils. I could set up a GDAS C5 and C6 build PR if you want/need. I'm in training full time through Thursday just to note.

jswhit2 · 2024-11-01T21:52:34Z

I could set up a GDAS C5 and C6 build PR if you want/need. I'm in training full time through Thursday just to note.

yes please, that would be great. I'd be happy to do the testing needed to make sure things work on c5/c6 once all the builds work.

DavidBurrows-NCO · 2024-11-04T21:41:41Z

I could set up a GDAS C5 and C6 build PR if you want/need. I'm in training full time through Thursday just to note.

yes please, that would be great. I'd be happy to do the testing needed to make sure things work on c5/c6 once all the builds work.

@jswhit2 Sounds good. I submitted an issue to build GDASApp on Gaea-C5 and C6 here. I will keep you updated on build progress. Is test data already set up on both systems?

DavidBurrows-NCO · 2024-11-05T12:26:18Z

@jswhit As @CoryMartin-NOAA said, C6 build was already set up. I made adjustments to C5 module files and machine ID to work with how the global-workflow will be configured, i.e., using gaeac5 and gaeac6 throughout global-workflow and submodules. I have a branch here that builds on both C5 and C6 with those C5 updates. @jswhit Let me know if you want to run tests on this set up, or since the changes are minimal, I may just submit this PR for building. Thanks

DavidBurrows-NCO · 2024-11-05T12:58:29Z

@jswhit Since your PR is for the global-workflow side, I submitted the GDAS PR1361 .

…al-workflow (#1361) After the recent Gaea-C5 OS upgrade, GDASApp fails to build. This issue corrects Gaea-C5 build and updates the build scripts to conform to ufs-wx-model (following ufs-wx-model ufs-community/ufs-weather-model#2448) and eventual global-workflow updates. Refs NOAA-EMC/global-workflow 3011 NOAA-EMC/global-workflow#3011 Refs NOAA-EMC/global-workflow 3032 NOAA-EMC/global-workflow#3032 Resolves #1360

WalterKolczynski-NOAA · 2024-11-12T16:39:42Z

Is there anything you're waiting on me for, or are @jswhit and @DavidBurrows-NCO / @AnilKumar-NOAA still coordinating things?

jswhit · 2024-11-14T16:25:15Z

@DavidBurrows-NCO everything builds on c6, thanks. I updated this PR to include changes in global-workflow to build and run on both c5/c6.

jswhit · 2024-11-15T16:26:04Z

@DavidBurrows-NCO I am currently testing 2021032400-2021032600, so if you could bring those dumps over too that would be great. I'm running the coupled 3dvar, and my test yaml is at /ncrc/home2/Jeffrey.S.Whitaker/C192mx025_3DVarAOWCDA.yaml.

DavidBurrows-NCO · 2024-11-15T17:01:00Z

2021032400-2021032600

@jswhit Is this the subset you are expecting?

ls -d /scratch1/NCEPDEV/global/glopara/dump/{gdas,gfs}.2021032{4..6}*
/scratch1/NCEPDEV/global/glopara/dump/gdas.20210324
/scratch1/NCEPDEV/global/glopara/dump/gdas.20210325
/scratch1/NCEPDEV/global/glopara/dump/gdas.20210326
/scratch1/NCEPDEV/global/glopara/dump/gfs.20210324
/scratch1/NCEPDEV/global/glopara/dump/gfs.20210325
/scratch1/NCEPDEV/global/glopara/dump/gfs.20210326

I looked into the subdirectories and notice that /scratch1/NCEPDEV/global/glopara/dump/gfs.20210324/06 is missing. @aerorahul Should I open an issue for this?

aerorahul · 2024-11-15T17:02:35Z

2021032400-2021032600

@jswhit Is this the subset you are expecting?
ls -d /scratch1/NCEPDEV/global/glopara/dump/{gdas,gfs}.2021032{4..6}*
/scratch1/NCEPDEV/global/glopara/dump/gdas.20210324
/scratch1/NCEPDEV/global/glopara/dump/gdas.20210325
/scratch1/NCEPDEV/global/glopara/dump/gdas.20210326
/scratch1/NCEPDEV/global/glopara/dump/gfs.20210324
/scratch1/NCEPDEV/global/glopara/dump/gfs.20210325
/scratch1/NCEPDEV/global/glopara/dump/gfs.20210326
I looked into the subdirectories and notice that /scratch1/NCEPDEV/global/glopara/dump/gfs.20210324/06 is missing. @aerorahul Should I open an issue for this?

probably. Tagging @KateFriedman-NOAA as she keeps the dump dataset synced

KateFriedman-NOAA · 2024-11-15T17:07:26Z

I looked into the subdirectories and notice that /scratch1/NCEPDEV/global/glopara/dump/gfs.20210324/06 is missing.

I'll get that filled in on Hera...stay tuned...

KateFriedman-NOAA · 2024-11-15T17:22:53Z

The 2021032406 dump files for gdas and gfs are now available on Hera. Does anyone need this cycle on WCOSS2 or MSU as well?

jswhit · 2024-11-15T17:39:53Z

2021032400-2021032600

@jswhit Is this the subset you are expecting?
ls -d /scratch1/NCEPDEV/global/glopara/dump/{gdas,gfs}.2021032{4..6}*
/scratch1/NCEPDEV/global/glopara/dump/gdas.20210324
/scratch1/NCEPDEV/global/glopara/dump/gdas.20210325
/scratch1/NCEPDEV/global/glopara/dump/gdas.20210326
/scratch1/NCEPDEV/global/glopara/dump/gfs.20210324
/scratch1/NCEPDEV/global/glopara/dump/gfs.20210325
/scratch1/NCEPDEV/global/glopara/dump/gfs.20210326
I looked into the subdirectories and notice that /scratch1/NCEPDEV/global/glopara/dump/gfs.20210324/06 is missing. @aerorahul Should I open an issue for this?
probably. Tagging @KateFriedman-NOAA as she keeps the dump dataset synced

Yes, that should do it. I also need the ocean obs for the same time period (on hera they are at /scratch1/NCEPDEV/da/common/, I've already copied them to c5 and c6 /gpfs/f5/nggps_psd/proj-shared/Jeffrey.S.Whitaker/da/common/, /gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/da/common/)

KateFriedman-NOAA · 2024-11-18T13:35:45Z

Yes, that should do it. I also need the ocean obs for the same time period (on hera they are at /scratch1/NCEPDEV/da/common/, I've already copied them to c5 and c6 /gpfs/f5/nggps_psd/proj-shared/Jeffrey.S.Whitaker/da/common/, /gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/da/common/)

@jswhit2 Please open a new issue to copy ocean obs into shared global space, thanks!

DavidBurrows-NCO · 2024-11-18T18:38:53Z

Yes, that should do it.

@jswhit I've transferred the requested data to C6 at /gpfs/f6/bil-fire8/world-shared/global/glopara/dump/{gdas,gfs}.2021032{4..6}*
Fix files are already on C6 at: /gpfs/f6/bil-fire8/world-shared/global/glopara/fix

ush/parsing_namelists_CICE.sh

DavidHuber-NOAA · 2024-11-26T18:47:09Z

Adding the blocked label until #3106 is merged.

DeniseWorthen · 2024-11-26T18:54:39Z

The fix to the ice_in file is included in ufs-community/ufs-weather-model#2505 using the additional hist_suffix setting.

changes to enable DA cycling on gaea

d11184e

jswhit2 requested review from WalterKolczynski-NOAA, aerorahul, CoryMartin-NOAA and DavidHuber-NOAA as code owners October 23, 2024 17:35

CoryMartin-NOAA previously approved these changes Oct 23, 2024

View reviewed changes

jswhit2 mentioned this pull request Oct 25, 2024

UFS DA cycling on gaea #2985

Open

WalterKolczynski-NOAA changed the title ~~changes to enable DA cycling on gaea~~ Enable DA cycling on gaea Oct 30, 2024

WalterKolczynski-NOAA requested changes Oct 30, 2024

View reviewed changes

fix machine name

cbc8953

jswhit2 dismissed CoryMartin-NOAA’s stale review via cbc8953 October 30, 2024 18:07

DavidBurrows-NCO mentioned this pull request Nov 4, 2024

Add/fix build capability for Gaea-C5 and Gaea-C6 NOAA-EMC/GDASApp#1360

Closed

DavidBurrows-NCO mentioned this pull request Nov 5, 2024

Update build scripts on Gaea-C5 to conform with ufs-wx-model and global-workflow NOAA-EMC/GDASApp#1361

Merged

jswhit2 and others added 4 commits November 14, 2024 08:07

Merge branch 'NOAA-EMC:develop' into develop

1cb4f3b

updates for c5/c6

7129d0c

new files for c5/c6

7ed5e8d

gaea -> gaeac5

f2a4679

no longer needed

a0ce4ec

jswhit2 requested a review from KateFriedman-NOAA as a code owner November 15, 2024 16:14

update c5 modulefile

3e87509

jswhit2 added 8 commits November 15, 2024 14:29

update max_tasks_per_node for c6

d4888c4

update

936c057

fix for build gdas app on c5,c6

bed74a0

update

f384bc2

update links for fix dirs

07e7cd5

update

775d04d

not needed

e30007c

update

38a1060

KateFriedman-NOAA mentioned this pull request Nov 19, 2024

stage input data (fixed files, obs and ics) for cycled DA on gaea c5/c6 #3108

Open

fix output frequency for cice history files (to every 3h, not every dt)

c2c87d4

jswhit2 requested review from junwang-noaa and DeniseWorthen as code owners November 20, 2024 23:21

jswhit2 added 2 commits November 20, 2024 18:21

update

594cd52

Merge remote-tracking branch 'upstream/develop' into develop

ba89ebd

DavidHuber-NOAA reviewed Nov 21, 2024

View reviewed changes

ush/parsing_namelists_CICE.sh Show resolved Hide resolved

jswhit2 mentioned this pull request Nov 21, 2024

ice model dumps history files every timestep then crashes with latest UFSWM #3121

Open

DavidHuber-NOAA added the blocked Issue is currently being blocked by another issue label Nov 26, 2024

DeniseWorthen mentioned this pull request Nov 26, 2024

Update CICE (2024-11) including pio_finalize bug fix ufs-community/ufs-weather-model#2505

Open

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable DA cycling on gaea #3032

Enable DA cycling on gaea #3032

jswhit2 commented Oct 23, 2024 •

edited

Loading

CoryMartin-NOAA left a comment

jswhit2 commented Oct 23, 2024

CoryMartin-NOAA commented Oct 23, 2024

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA Oct 30, 2024

jswhit Oct 30, 2024

WalterKolczynski-NOAA commented Oct 30, 2024

AnilKumar-NOAA commented Oct 30, 2024

DavidBurrows-NCO commented Oct 30, 2024

jswhit2 commented Nov 1, 2024

DavidBurrows-NCO commented Nov 4, 2024

DavidBurrows-NCO commented Nov 5, 2024

DavidBurrows-NCO commented Nov 5, 2024

WalterKolczynski-NOAA commented Nov 12, 2024

jswhit commented Nov 14, 2024

jswhit commented Nov 15, 2024

DavidBurrows-NCO commented Nov 15, 2024

aerorahul commented Nov 15, 2024

KateFriedman-NOAA commented Nov 15, 2024

KateFriedman-NOAA commented Nov 15, 2024

jswhit commented Nov 15, 2024

KateFriedman-NOAA commented Nov 18, 2024

DavidBurrows-NCO commented Nov 18, 2024

DavidHuber-NOAA commented Nov 26, 2024

DeniseWorthen commented Nov 26, 2024

	echo "WARNING: The job step ${step} does not specify Hercules-specific resources"
	echo "WARNING: The job step ${step} does not specify Gaea-specific resources"

Enable DA cycling on gaea #3032

Are you sure you want to change the base?

Enable DA cycling on gaea #3032

Conversation

jswhit2 commented Oct 23, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

jswhit2 commented Oct 23, 2024

CoryMartin-NOAA commented Oct 23, 2024

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA Oct 30, 2024

Choose a reason for hiding this comment

jswhit Oct 30, 2024

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Oct 30, 2024

AnilKumar-NOAA commented Oct 30, 2024

DavidBurrows-NCO commented Oct 30, 2024

jswhit2 commented Nov 1, 2024

DavidBurrows-NCO commented Nov 4, 2024

DavidBurrows-NCO commented Nov 5, 2024

DavidBurrows-NCO commented Nov 5, 2024

WalterKolczynski-NOAA commented Nov 12, 2024

jswhit commented Nov 14, 2024

jswhit commented Nov 15, 2024

DavidBurrows-NCO commented Nov 15, 2024

aerorahul commented Nov 15, 2024

KateFriedman-NOAA commented Nov 15, 2024

KateFriedman-NOAA commented Nov 15, 2024

jswhit commented Nov 15, 2024

KateFriedman-NOAA commented Nov 18, 2024

DavidBurrows-NCO commented Nov 18, 2024

DavidHuber-NOAA commented Nov 26, 2024

DeniseWorthen commented Nov 26, 2024

jswhit2 commented Oct 23, 2024 •

edited

Loading