Move the variational scripts to this repository #2920

guillaumevernieres · 2024-09-13T18:12:37Z

Description

This PR "moves and refactors" the variational DA exscripts that were in the GDASapp to this repository. The ens. var. feature will be replicated/moved in a subsequent PR.

fixes #Refactoring of the marine DA scripts GDASApp#977

Waiting for 2 companion PR's:

ci testing: We should be able to turn on the WCDA test on Hercules

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES/NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

The latest tests were done on Hecules, earlier ones on Hera

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT

ush/python/pygfs/task/marine_bmat.py

ci/cases/gfsv17/ocnanal.yaml

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT

ush/python/pygfs/task/marine_bmat.py

ush/python/pygfs/utils/marine_da_utils.py

DavidNew-NOAA

Looks good to me. No comments beyond what others have already said.

ush/python/pygfs/utils/marine_da_utils.py

ush/python/pygfs/task/marine_analysis.py

This PR needs to be merged before - [NOAA-EMC/global-workflow#2920 It depends on - [NOAA-EMC/jcb-gdas#26 and will require an update to the `jcb-gdas` submodules. It will break the GDASApp marine CI until the `g-w` pr is merged.

aerorahul · 2024-09-20T17:26:46Z

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

+# Generate COM variables from templates
+YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_OBS
+
+RUN=${GDUMP} YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \


Why is this GDUMP? This should be RUN.

GDUMP is associated with the guess; that is the previous cycle gPDY/gcyc -- just clarifying why the q.

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

aerorahul · 2024-09-20T17:32:20Z

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

-export CDUMP=${CDUMP:-${RUN:-"gfs"}}
-export CDATE=${CDATE:-${PDY}${cyc}}
+# Ignore possible spelling error (nothing is misspelled)
+# shellcheck disable=SC2153
 export GDUMP=${GDUMP:-"gdas"}


I doubt, GDUMP is used in a finalize step, but it might be needed in the __init__. Just check if that is the case.

aerorahul · 2024-09-20T17:45:12Z

jobs/rocoto/marineanlchkpt.sh

@@ -8,11 +8,10 @@ source "${HOMEgfs}/ush/preamble.sh"
 status=$?
 [[ ${status} -ne 0 ]] && exit "${status}"

-export job="ocnanalrun"
-export jobid="${job}.$$"
+export job="marineanlchkpt"


why remove jobid?

no reason, will fix.

Co-authored-by: Rahul Mahajan <[email protected]>

emcbot · 2024-09-21T06:26:52Z

CI Passed on Hera in Build# 2
Built and ran in directory /scratch1/NCEPDEV/global/CI/2920


Experiment C48mx500_3DVarAOWCDA_259df531 Completed 2 Cycles: *SUCCESS* at Fri Sep 20 16:09:33 UTC 2024
Experiment C48_ATM_259df531 Completed 1 Cycles: *SUCCESS* at Fri Sep 20 16:21:49 UTC 2024
Experiment C48_S2SWA_gefs_259df531 Completed 1 Cycles: *SUCCESS* at Fri Sep 20 17:48:00 UTC 2024
Experiment C48_S2SW_259df531 Completed 1 Cycles: *SUCCESS* at Fri Sep 20 21:47:20 UTC 2024
Experiment C96_atm3DVar_259df531 Completed 3 Cycles: *SUCCESS* at Fri Sep 20 22:05:53 UTC 2024
Experiment C96C48_hybatmDA_259df531 Completed 3 Cycles: *SUCCESS* at Fri Sep 20 22:05:53 UTC 2024
Experiment C96C48_ufs_hybatmDA_259df531 Completed 2 Cycles: *SUCCESS* at Sat Sep 21 06:26:41 UTC 2024
Experiment C96C48_hybatmaerosnowDA_259df531 Completed 3 Cycles: *SUCCESS* at Sat Sep 21 06:26:41 UTC 2024

emcbot · 2024-09-21T07:05:15Z

CI Update on Wcoss2 at 09/21/24 07:05:14 AM
============================================
Cloning and Building global-workflow PR: 2920
with PID: 103281 on host: clogin03

emcbot · 2024-09-21T07:48:09Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Sat Sep 21 07:11:43 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 09/21/24 07:47:47 AM
Case setup: Completed for experiment C48_ATM_98ace069
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_98ace069
Case setup: Skipped for experiment C48_S2SWA_gefs_98ace069
Case setup: Completed for experiment C48_S2SW_98ace069
Case setup: Completed for experiment C96_atm3DVar_extended_98ace069
Case setup: Skipped for experiment C96_atm3DVar_98ace069
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_98ace069
Case setup: Completed for experiment C96C48_hybatmDA_98ace069
Case setup: Completed for experiment C96C48_ufs_hybatmDA_98ace069

emcbot · 2024-09-21T10:51:12Z

CI Passed on Hercules in Build# 3
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2920


Experiment C48_ATM_98ace069 Completed 1 Cycles: *SUCCESS* at Sat Sep 21 04:19:26 CDT 2024
Experiment C96C48_hybatmDA_98ace069 Completed 3 Cycles: *SUCCESS* at Sat Sep 21 05:20:12 CDT 2024
Experiment C96_atm3DVar_98ace069 Completed 3 Cycles: *SUCCESS* at Sat Sep 21 05:26:03 CDT 2024
Experiment C48_S2SW_98ace069 Completed 1 Cycles: *SUCCESS* at Sat Sep 21 05:44:17 CDT 2024
Experiment C48_S2SWA_gefs_98ace069 Completed 1 Cycles: *SUCCESS* at Sat Sep 21 05:50:54 CDT 2024

guillaumevernieres and others added 7 commits August 28, 2024 09:11

...

c9e41b1

var init ok, run needs tlc

53a4de8

updated checkpoint task

ba78b00

working finalize

4509b0c

Merge branch 'develop' into feature/marinevar

2b667f2

...

317e351

removed envar test yaml

ea6ae91

guillaumevernieres requested review from aerorahul, WalterKolczynski-NOAA and AndrewEichmann-NOAA September 13, 2024 18:12

github-advanced-security bot found potential problems Sep 13, 2024

View reviewed changes

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT Fixed Show fixed Hide fixed

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT Fixed Show fixed Hide fixed

guillaumevernieres added 4 commits September 13, 2024 13:30

norms, try 1

5c7eb42

Merge branch 'develop' into feature/marinevar

b4d46aa

norms, try 2

3b15dc6

norms, try 3

90ad1d3

guillaumevernieres mentioned this pull request Sep 13, 2024

Move the variational scripts to the g-w NOAA-EMC/GDASApp#1282

Merged

AndrewEichmann-NOAA reviewed Sep 13, 2024

View reviewed changes

aerorahul requested a review from DavidNew-NOAA September 14, 2024 14:44

guillaumevernieres added 2 commits September 16, 2024 17:54

addressed review comments

f1d7f8b

Merge branch 'develop' into feature/marinevar

f84ae82

DavidNew-NOAA approved these changes Sep 17, 2024

View reviewed changes

aerorahul reviewed Sep 17, 2024

View reviewed changes

ush/python/pygfs/utils/marine_da_utils.py Outdated Show resolved Hide resolved

aerorahul reviewed Sep 17, 2024

View reviewed changes

ush/python/pygfs/task/marine_analysis.py Show resolved Hide resolved

DavidNew-NOAA mentioned this pull request Sep 17, 2024

Incorporate JEDI class into marine jobs #2931

Open

guillaumevernieres mentioned this pull request Sep 18, 2024

Use jinja template for the obs list NOAA-EMC/GDASApp#1285

Open

guillaumevernieres added 2 commits September 18, 2024 11:00

module tidy

9e8bbad

Merge branch 'develop' into feature/marinevar

a286b1e

guillaumevernieres requested a review from aerorahul September 18, 2024 16:07

updated gdas and jcb-gdas hashes

a4d8391

aerorahul added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Sep 20, 2024

aerorahul reviewed Sep 20, 2024

View reviewed changes

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE Outdated Show resolved Hide resolved

aerorahul reviewed Sep 20, 2024

View reviewed changes

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE Outdated Show resolved Hide resolved

aerorahul reviewed Sep 20, 2024

View reviewed changes

guillaumevernieres and others added 2 commits September 20, 2024 16:49

Update jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

77f3ddc

Co-authored-by: Rahul Mahajan <[email protected]>

Update jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

98ace06

Co-authored-by: Rahul Mahajan <[email protected]>

emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Sep 21, 2024

WalterKolczynski-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Sep 21, 2024

emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Sep 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move the variational scripts to this repository #2920

Move the variational scripts to this repository #2920

guillaumevernieres commented Sep 13, 2024 •

edited

Loading

DavidNew-NOAA left a comment

aerorahul Sep 20, 2024

aerorahul Sep 20, 2024

aerorahul Sep 20, 2024

aerorahul Sep 20, 2024

guillaumevernieres Sep 20, 2024

emcbot commented Sep 21, 2024

emcbot commented Sep 21, 2024

emcbot commented Sep 21, 2024

emcbot commented Sep 21, 2024

Move the variational scripts to this repository #2920

Are you sure you want to change the base?

Move the variational scripts to this repository #2920

Conversation

guillaumevernieres commented Sep 13, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

DavidNew-NOAA left a comment

Choose a reason for hiding this comment

aerorahul Sep 20, 2024

Choose a reason for hiding this comment

aerorahul Sep 20, 2024

Choose a reason for hiding this comment

aerorahul Sep 20, 2024

Choose a reason for hiding this comment

aerorahul Sep 20, 2024

Choose a reason for hiding this comment

guillaumevernieres Sep 20, 2024

Choose a reason for hiding this comment

emcbot commented Sep 21, 2024

emcbot commented Sep 21, 2024

emcbot commented Sep 21, 2024

emcbot commented Sep 21, 2024

guillaumevernieres commented Sep 13, 2024 •

edited

Loading