Add snow DA update and recentering for the EnKF forecasts #2690

CoryMartin-NOAA · 2024-06-14T14:30:38Z

Description

Apologies in that this is a fairly substantial PR, but a lot of things were needed to make this happen. This PR adds the capability to update the ensemble of snow states by recentering the ensemble mean to the deterministic snow analysis and applying increments as appropriate.

Resolves #2585

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? YES in the sense that results will change
Does this change require a documentation update? Maybe?

How has this been tested?

Cycled test on Hera

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

…-NOAA/global-workflow into feature/snow-ensemble

emcbot · 2024-08-21T19:11:08Z

CI Update on Wcoss2 at 08/21/24 07:11:07 PM
============================================
Cloning and Building global-workflow PR: 2690
with PID: 108102 on host: clogin03

emcbot · 2024-08-21T20:11:05Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Wed Aug 21 19:21:02 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 08/21/24 08:10:40 PM
Case setup: Completed for experiment C48_ATM_b3f8e5b3
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_b3f8e5b3
Case setup: Skipped for experiment C48_S2SWA_gefs_b3f8e5b3
Case setup: Completed for experiment C48_S2SW_b3f8e5b3
Case setup: Completed for experiment C96_atm3DVar_extended_b3f8e5b3
Case setup: Skipped for experiment C96_atm3DVar_b3f8e5b3
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_b3f8e5b3
Case setup: Completed for experiment C96C48_hybatmDA_b3f8e5b3
Case setup: Completed for experiment C96C48_ufs_hybatmDA_b3f8e5b3

emcbot · 2024-08-21T22:07:36Z

Experiment C96C48_ufs_hybatmDA_b3f8e5b3 FAIL on Wcoss2 at 08/21/24 10:07:35 PM

Error logs:

/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2690/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_b3f8e5b3/logs/2024022400/enkfgdasatmensanlinit.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2690/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_b3f8e5b3/logs/2024022400/gdasatmanlinit.log
/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2690/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_b3f8e5b3/logs/2024022400/gfsatmanlinit.log

Follow link here to view the contents of the above file(s): (link)

WalterKolczynski-NOAA · 2024-08-22T17:10:58Z

@RussTreadon-NOAA can you take a look at why this and #2738 are both failing on the UFSDA atm test? If it doesn't have a quick fix, we may need to temporarily disable it to get v17 stuff in.

RussTreadon-NOAA · 2024-08-22T17:28:19Z

CoryMartin-NOAA:feature/snow-ensemble points at gdas.cd @ 0431b26. This is the same GDASApp hash as PR #2833. This GDASApp hash adds radiances to the atmospheric variational and local ensemble DA analysis jobs.

g-w PR #2833 contains g-w changes needed to copy JEDI radiance bias correction files to COMROOT. This PR does not contain these g-w changes. As a result each of the init jobs failed because they can not copy the atms_n20 bias correction files

enkfgdasatmensanlinit.log:    raise OSError(f"unable to copy {source} to {target}")
enkfgdasatmensanlinit.log:OSError: unable to copy /lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2690/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_b3f8e5b3/gdas.20240223/18//analysis/atmos/gdas.t18z.atms_n20.satbias.nc to /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96C48_ufs_hybatmDA_b3f8e5b3/enkfgdas.2024022400/enkfgdasatmensanl_00/obs/gdas.t18z.atms_n20.satbias.nc
gdasatmanlinit.log:    raise OSError(f"unable to copy {source} to {target}")
gdasatmanlinit.log:OSError: unable to copy /lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2690/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_b3f8e5b3/gdas.20240223/18//analysis/atmos/gdas.t18z.atms_n20.satbias.nc to /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96C48_ufs_hybatmDA_b3f8e5b3/gdas.2024022400/gdasatmanl_00/obs/gdas.t18z.atms_n20.satbias.nc
gfsatmanlinit.log:    raise OSError(f"unable to copy {source} to {target}")
gfsatmanlinit.log:OSError: unable to copy /lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT/PR/2690/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_b3f8e5b3/gdas.20240223/18//analysis/atmos/gdas.t18z.atms_n20.satbias.nc to /lfs/h2/emc/stmp/terry.mcguinness/RUNDIRS/C96C48_ufs_hybatmDA_b3f8e5b3/gfs.2024022400/gfsatmanl_00/obs/gdas.t18z.atms_n20.satbias.nc

We can not run g-w CI C96C48_ufs_hybatmDA until g-w PR #2833 is merged into g-w develop. The other option is to back up the gdas.cd hash in this PR to a GDASApp hash before 0431b26.

0431b26 add lgetkf observer and solver templates (#1252)
925ebc8 update da-utils hash (#1255)
a58e705 Add DA-Utils to GDASApp (#1253)
eec4559 Add an entry to assimilate new snow dump files (sfcsno) replacing adpsfc_snow (#1245)
c5794ab fix for FMS paths that are too long in snow YAMLs (#1250)
9e38433 resolve inconsistencies between bufr2ioda python converters and jcb templates (#1241)
c68ef9d output gaussian increment for aerosols (#1248)
67e8c50 Add specificHumidity to ADPUPA BUFR DUMP Python API (#1246)
f3fa26d Stage atmospheric backgrounds and UFS cubed-sphere history files (#1236)
cc4568d Use ensemble member for mask in snow recentering (#1242)

WalterKolczynski-NOAA · 2024-08-22T18:32:13Z

While #2833 is close, this one is a bit further along, so let's do this:

@CoryMartin-NOAA Please add wcoss2 to the skip_ci_on_hosts list in ci/cases/pr/C96C48_ufs_hybatmDA.yaml

@RussTreadon-NOAA Please turn the test back on as part of your PR after this one gets merged

CoryMartin-NOAA · 2024-08-22T18:40:30Z

@WalterKolczynski-NOAA done, thanks!

RussTreadon-NOAA · 2024-08-22T18:44:21Z

@WalterKolczynski-NOAA , I'll try my best to remember to undo 1cb436e in PR #2833 but I may forget. Let's try to help each other to remember this important detail.

emcbot · 2024-08-22T18:46:02Z

CI Update on Wcoss2 at 08/22/24 06:46:01 PM
============================================
Cloning and Building global-workflow PR: 2690
with PID: 105346 on host: clogin03

emcbot · 2024-08-22T19:50:11Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Thu Aug 22 18:51:58 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 08/22/24 07:49:53 PM
Case setup: Completed for experiment C48_ATM_1cb436e7
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_1cb436e7
Case setup: Skipped for experiment C48_S2SWA_gefs_1cb436e7
Case setup: Completed for experiment C48_S2SW_1cb436e7
Case setup: Completed for experiment C96_atm3DVar_extended_1cb436e7
Case setup: Skipped for experiment C96_atm3DVar_1cb436e7
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_1cb436e7
Case setup: Completed for experiment C96C48_hybatmDA_1cb436e7
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_1cb436e7

emcbot · 2024-08-23T07:07:16Z

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_1cb436e7 *** SUCCESS *** at 08/22/24 08:56:11 PM
Experiment C48_S2SW_1cb436e7 *** SUCCESS *** at 08/22/24 09:14:16 PM
Experiment C96C48_hybatmDA_1cb436e7 *** SUCCESS *** at 08/22/24 09:56:22 PM
Experiment C96C48_hybatmaerosnowDA_1cb436e7 *** SUCCESS *** at 08/22/24 10:42:23 PM
Experiment C96_atm3DVar_extended_1cb436e7 *** SUCCESS *** at 08/23/24 07:00:38 AM

* origin/develop: Fix gdas build on Gaea and add Gaea to available CI list (NOAA-EMC#2857) Support ATM forecast only on Google (NOAA-EMC#2832) Add GEFS C48 support on AWS (NOAA-EMC#2818) Update omega calculation (NOAA-EMC#2751) Add snow DA update and recentering for the EnKF forecasts (NOAA-EMC#2690)

* origin/develop: Create JEDI class (NOAA-EMC#2805) Restructure the bufr sounding job (NOAA-EMC#2853) Add an archive task to GEFS system to archive files locally (NOAA-EMC#2816) Reenable Orion Cycling Support (NOAA-EMC#2877) Eliminate race conditions and remove DATAROOT last in cleanup (NOAA-EMC#2893) Update aerosol climatology to 2013-2024 mean (NOAA-EMC#2888) Add ability to run CI test C96_atm3DVar.yaml to Gaea-C5 (NOAA-EMC#2885) Support global-workflow GEFS C48 on Google Cloud (NOAA-EMC#2861) Add 3 and 9 hr increment files to IC staging (NOAA-EMC#2876) Add diffusion/diag B for aerosol DA and some other needed changes (NOAA-EMC#2738) Correct ocean `MOM.res_#` stage copy (NOAA-EMC#2868) Support coupling on AWS (NOAA-EMC#2859) Add JEDI ATM lgetkf observer and solver jobs (NOAA-EMC#2833) Fix gdas build on Gaea and add Gaea to available CI list (NOAA-EMC#2857) Support ATM forecast only on Google (NOAA-EMC#2832) Add GEFS C48 support on AWS (NOAA-EMC#2818) Update omega calculation (NOAA-EMC#2751) Add snow DA update and recentering for the EnKF forecasts (NOAA-EMC#2690) support ATM forecast only on Azure (NOAA-EMC#2827) Convert staging job to python and yaml (NOAA-EMC#2651) Fixed test on UNAVAILBLE in python Rocoto check (NOAA-EMC#2842)

CoryMartin-NOAA added 30 commits May 9, 2024 18:37

start to add new task

b0656aa

add new j-job

d16119d

Save to do some initial testing

b4c97f8

more updates

3667349

Merge branch 'NOAA-EMC:develop' into feature/snow-ensemble

accc4f4

Merge branch 'develop' into feature/snow-ensemble

e565f21

end of day commit

81b7b6e

End of day/week commit

3aaa8f6

Merge branch 'NOAA-EMC:develop' into feature/snow-ensemble

7085716

save

29641b2

Merge branch 'feature/snow-ensemble' of https://github.com/CoryMartin…

531136a

…-NOAA/global-workflow into feature/snow-ensemble

Merge branch 'NOAA-EMC:develop' into feature/snow-ensemble

7491c35

snow changes

d57e12f

end of day commit

89fd461

stage files is working

f26822e

save updates

2460af1

more updates

057996d

make norm happy

146dc60

Merge branch 'develop' into feature/snow-ensemble

cff98a0

norm is an angry old man

d7afb7b

remove esnowanl from enkfgfs

eb5716d

Merge branch 'feature/snow-ensemble' of https://github.com/CoryMartin…

256e182

…-NOAA/global-workflow into feature/snow-ensemble

end of day commit

6a57a64

fregrid working but seems a bit hacky

ebb3c31

norms and remove commented out code

2bdee22

Merge branch 'develop' into feature/snow-ensemble

99fdfbe

update snow ens

42b3fbe

Merge branch 'develop' into feature/snow-ensemble

a950c80

update some submodules

f380223

update some things

73894dc

emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Aug 21, 2024

emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Aug 21, 2024

emcbot added CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Aug 21, 2024

skip wcoss2 for ci for now

1cb436e

WalterKolczynski-NOAA added CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS and removed CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed labels Aug 22, 2024

emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Aug 22, 2024

emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Aug 22, 2024

emcbot added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Aug 23, 2024

WalterKolczynski-NOAA approved these changes Aug 23, 2024

View reviewed changes

WalterKolczynski-NOAA merged commit 2ce2116 into NOAA-EMC:develop Aug 23, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add snow DA update and recentering for the EnKF forecasts #2690

Add snow DA update and recentering for the EnKF forecasts #2690

CoryMartin-NOAA commented Jun 14, 2024 •

edited

Loading

emcbot commented Aug 21, 2024

emcbot commented Aug 21, 2024

emcbot commented Aug 21, 2024

WalterKolczynski-NOAA commented Aug 22, 2024

RussTreadon-NOAA commented Aug 22, 2024

WalterKolczynski-NOAA commented Aug 22, 2024

CoryMartin-NOAA commented Aug 22, 2024

RussTreadon-NOAA commented Aug 22, 2024

emcbot commented Aug 22, 2024

emcbot commented Aug 22, 2024

emcbot commented Aug 23, 2024

Add snow DA update and recentering for the EnKF forecasts #2690

Add snow DA update and recentering for the EnKF forecasts #2690

Conversation

CoryMartin-NOAA commented Jun 14, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

emcbot commented Aug 21, 2024

emcbot commented Aug 21, 2024

emcbot commented Aug 21, 2024

WalterKolczynski-NOAA commented Aug 22, 2024

RussTreadon-NOAA commented Aug 22, 2024

WalterKolczynski-NOAA commented Aug 22, 2024

CoryMartin-NOAA commented Aug 22, 2024

RussTreadon-NOAA commented Aug 22, 2024

emcbot commented Aug 22, 2024

emcbot commented Aug 22, 2024

emcbot commented Aug 23, 2024

CoryMartin-NOAA commented Jun 14, 2024 •

edited

Loading