add capability to dynamically generate GSI *info files #3472

jswhit2 · 2025-03-18T18:53:12Z

Description

Currently config.anal has an if/then/else loop that sets env vars SATINFO, OZINFO, CONVINFO that contain paths to different gsi info files for different periods over the last several years. For reanalysis, we need a solution that works back to 1979. Rather than add to the if/then/else block in config.anal, I have created a set of scripts that generate *info files dynamically given a date (https://github.com/NOAA-PSL/build_gsinfo). What is needed is a way to use those scripts in global-workflow. Note this is only needed for GSI - for JEDI the needed functionality is included in observation_chronicle (https://github.com/NOAA-EMC/jcb-gdas/tree/develop/observation_chronicle/atmosphere).

Resolves #3293

Requires NOAA-EMC/GSI-fix#28

Enabled via USE_BUILD_GSINFO env var that can be set to YES in config.base (default is NO).
3 new scripts added (create_satinfo.sh, create_ozinfo.sh, create_convinfo.sh). These scripts generate the GSI *info for a given analysis date using data from build_gsinfo (which will live inside GSI-fix once NOAA-EMC/GSI-fix#28 is merged).

The OBS_INPUT table in the GSI namelist is removed from exglobal_atmos_analysis.sh to allow for separate options for NCEP ops and reanalysis. Both versions are included as text files in build_gsinfo/obs_input. The OBS_INPUT env var can be used to choose which version do use. The NCEP ops version is the default in config.anal.

A workaround for NOAA-EMC/GSI#752 is included in exglobal_atmos_analysis.sh (pointing to a separate directory for HIRS coefficient files). This hack can be removed once NOAA-EMC/GSI#783 is merged.

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)?NO
Does this change require a documentation update? YES
Does this change require an update to any of the following submodules? YES
- [ x] GSI-fix add NOAA-PSL/build_gsinfo as a submodule GSI-fix#28

How has this been tested?

Clone and build on gaeac6 and hera
Cycled test on gaeac6 and hera

Checklist

Any dependent changes have been merged and published
[ x] My code follows the style guidelines of this project
[ x] I have performed a self-review of my own code
[x ] I have commented my code, particularly in hard-to-understand areas
[ x] I have documented my code, including function, input, and output descriptions
[ x] My changes generate no new warnings
[ x] New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added
Any new scripts have been added to the .github/CODEOWNERS file with owners
I have made corresponding changes to the system documentation if necessary

github-advanced-security

ShellCheck found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

jswhit2 · 2025-03-18T18:54:09Z

@jack-woollen @jswhit2 Jeff I didn't figure out how to update your branch with the convinfo file containing complete satwnd definitions. Instead you can copy from /work2/noaa/da/jwoollen/RAEXPS/scripts/2021ozn1/build_gsinfo/convinfo/merged_convinfo.txt.

CoryMartin-NOAA · 2025-03-19T19:11:40Z

parm/config/gfs/config.anal

+else
+   export OBS_INPUT=${FIXgfs}/gsi/build_gsinfo/obs_input/obs_input_ops.txt
+fi
+export HIRS_FIX=/gpfs/f6/drsa-precip4/proj-shared/Jeffrey.S.Whitaker/hirs_fix


noting a hard coded Gaea path here

Thanks @CoryMartin-NOAA - I think this needs to default to the CRTM_FIX directory and be configurable in config.anal. We won't need any of this when CRTM is updated though (hopefully).

CoryMartin-NOAA · 2025-03-19T19:12:27Z

scripts/create_convinfo.sh

@@ -0,0 +1,18 @@
+# create global_ozinfo file for a given date


Need a shebang defined here?

CoryMartin-NOAA · 2025-03-19T19:12:46Z

scripts/create_ozinfo.sh

@@ -0,0 +1,27 @@
+# create global_ozinfo file for a given date


Another shebang needed

Generally: These new scripts likely need to go in ush/ and need the style issues addressed

scripts/exglobal_atmos_analysis.sh

ush/create_ozinfo.sh

+echo '!                                  error  error variational qc'
+# loop over satellites
+cd "${FIXgfs}/gsi/build_gsinfo/ozinfo" || exit 1
+grep -v '^ *#' satellites | while IFS= read -r sat


ush/create_satinfo.sh

+echo '!sensor/instr/sat      chan iuse  error  error_cld  ermax   var_b    var_pg  icld_det icloud iaerosol'
+# loop over satellites
+cd "${FIXgfs}/gsi/build_gsinfo/satinfo" || exit 1
+grep -v '^ *#' satellites | while IFS= read -r sat


ush/create_satinfo.sh

ush/create_ozinfo.sh

ush/create_satinfo.sh

ClaraDraper-NOAA

@jswhit2 To include the soil analysis in the reanalysis / scout runs, we'll need to make changes to the convinfo and anavinfo. For the NRT system, those changes are here.

Can you please add these changes into your [conv/anav]info files? I'm not sure if it's better to include them always, or as an option.

jswhit · 2025-04-03T18:46:18Z

@ClaraDraper-NOAA I think you forgot to include the link. Would you be able to tell us specifically which lines to add to the. reanalysis version of convinfo (https://github.com/NOAA-PSL/build_gsinfo/blob/main/convinfo/merged_convinfo.txt)? The anavinfo changes won't be needed for the scout runs until we start running ensemble DA.

ClaraDraper-NOAA · 2025-04-03T19:01:14Z

@jswhit2 To include the soil analysis in the reanalysis / scout runs, we'll need to make changes to the convinfo and anavinfo. For the NRT system, those changes are here.

Can you please add these changes into your [conv/anav]info files? I'm not sure if it's better to include them always, or as an option.

I did forget the link! It's here

ClaraDraper-NOAA · 2025-04-03T19:04:11Z

If we want to assimilate 2m obs into the scout run (maybe not a great idea???), to see the changes you need, in fix/gsi/

diff global_convinfo_2mObs.txt global_convinfo.txt

If we just want to monitor them, you can leave convinfo as is. For monitoring or assim, we need to add t2m and q2m to the met_guess and state_derivative namelists in anavinfo. There's an example here on hera:

/scratch2/BMC/gsienkf/Clara.Draper/gerrit-hera/global-workflow_CNTRL/fix/global_anavinfo_2mDiag.l127.txt

jswhit · 2025-04-03T19:21:09Z

OK got it - thanks @ClaraDraper-NOAA

DavidHuber-NOAA · 2025-04-10T12:25:20Z

Moving to draft while upstream issues are addressed. @jswhit feel free to re-mark this as ready for review when it is so.

DavidHuber-NOAA

A few minor nit-picks. There are also a number of shell-check issues noted here: https://github.com/NOAA-EMC/global-workflow/actions/runs/14601534590/job/40960623756?pr=3472. Could you please also address these?

DavidHuber-NOAA · 2025-05-09T17:53:39Z

scripts/exglobal_atmos_analysis.sh

@@ -359,14 +367,26 @@ cd ${DATA} || exit 99
 # Fixed files
 ${NLN} ${BERROR}       berror_stats
 ${NLN} ${SATANGL}      satbias_angle
-${NLN} ${SATINFO}      satinfo
+if [[ ${SATINFO} == "generate" ]]; then
+   sh ${USHgfs}/create_satinfo.sh  ${PDY}${cyc} > satinfo


Prefer that the create_satinfo.sh file be called directly. If it isn't already, that script should be made executable.

Suggested change

sh ${USHgfs}/create_satinfo.sh ${PDY}${cyc} > satinfo

${USHgfs}/create_satinfo.sh ${PDY}${cyc} > satinfo

DavidHuber-NOAA · 2025-05-09T17:54:03Z

scripts/exglobal_atmos_analysis.sh

 ${NLN} ${RADCLOUDINFO} cloudy_radiance_info.txt
 ${NLN} ${ATMSFILTER}   atms_beamwidth.txt
 ${NLN} ${ANAVINFO}     anavinfo
-${NLN} ${CONVINFO}     convinfo
+if [[ ${CONVINFO} == "generate" ]]; then
+   sh ${USHgfs}/create_convinfo.sh  ${PDY}${cyc} > convinfo


Suggested change

sh ${USHgfs}/create_convinfo.sh ${PDY}${cyc} > convinfo

${USHgfs}/create_convinfo.sh ${PDY}${cyc} > convinfo

DavidHuber-NOAA · 2025-05-09T17:54:58Z

scripts/exglobal_atmos_analysis.sh

 ${NLN} ${vqcdat}       vqctp001.dat
 ${NLN} ${INSITUINFO}   insituinfo
-${NLN} ${OZINFO}       ozinfo
+if [[ ${OZINFO} == "generate" ]]; then
+   sh ${USHgfs}/create_ozinfo.sh  ${PDY}${cyc} > ozinfo


Suggested change

sh ${USHgfs}/create_ozinfo.sh ${PDY}${cyc} > ozinfo

${USHgfs}/create_ozinfo.sh ${PDY}${cyc} > ozinfo

DavidHuber-NOAA · 2025-05-09T17:57:23Z

ush/create_satinfo.sh

+    for f in "${sat}"/*; do
+        if [[ ${f} != "${sat}/readme" ]]; then # skip readme file
+           datex=$(basename "${f}")
+	   if [[ ${date} -ge ${datex} ]]; then
+              usedate=${datex}
+           fi
+        fi
+    done


Please use consistent white spacing and avoid tab characters.

DavidHuber-NOAA · 2025-05-09T17:57:56Z

ush/create_satinfo.sh

+        echo "date not found for ${sat}"
+	exit 1


jswhit2 · 2025-05-23T16:28:31Z

A few minor nit-picks. There are also a number of shell-check issues noted here: https://github.com/NOAA-EMC/global-workflow/actions/runs/14601534590/job/40960623756?pr=3472. Could you please also address these?

@DavidHuber-NOAA those shell-check errors do not appear to coming from lines I modified so I hesitate to change them.

jswhit2 and others added 9 commits February 4, 2025 15:33

introduce new env var USE_BUILD_GSINFO

3182e17

add scripts for generating GSI info files when USE_BUILD_GSINFO=YES

a97e131

update

e4094ad

Merge branch 'develop' into build_gsinfo

87cae24

Merge branch 'develop' into build_gsinfo

16182d3

use OBS_INPUT envar to point to obs_input namelist block table text file

3fdd95c

fix for bad hirs coeff files (workaround until new crtm)

a8a3473

add OBS_INPUT and HIRS_FIX

b589b9b

Merge branch 'NOAA-EMC:develop' into build_gsinfo

1ab1ef6

jswhit2 requested review from RussTreadon-NOAA and CoryMartin-NOAA as code owners March 18, 2025 18:53

github-advanced-security bot found potential problems Mar 18, 2025

View reviewed changes

CoryMartin-NOAA reviewed Mar 19, 2025

View reviewed changes

jswhit2 added 6 commits March 25, 2025 18:08

move scripts from scripts to ush, add shebang

49b5ed8

address shell linting errors

dd77bb4

address shell linting errors

0e87789

address shell linting errors

3fcc4f1

fixed crtm2 hirs2 coeff files now included in build_gsinfo

1e7b9f8

address shell linting errors

339dfdb

github-advanced-security bot found potential problems Mar 25, 2025

View reviewed changes

jswhit2 added 3 commits March 25, 2025 19:11

address more shell linting errors

25ab1b0

scripts now in USHgfs

25b7871

add check for existence of satellites file

11ad3f0

github-advanced-security bot found potential problems Mar 25, 2025

View reviewed changes

ush/create_ozinfo.sh Fixed Show fixed Hide fixed

ush/create_satinfo.sh Fixed Show fixed Hide fixed

jswhit2 added 3 commits March 25, 2025 15:35

update

1dd3450

Merge branch 'develop' into build_gsinfo

901be87

update

ac22e7d

ClaraDraper-NOAA requested changes Apr 3, 2025

View reviewed changes

DavidHuber-NOAA marked this pull request as draft April 10, 2025 12:25

jswhit2 marked this pull request as ready for review April 15, 2025 18:46

Merge branch 'develop' into build_gsinfo

62607ab

jswhit2 added a commit to NOAA-PSL/global-workflow that referenced this pull request Apr 23, 2025

include changes from NOAA-EMC#3472

c6e3099

DavidHuber-NOAA requested changes May 9, 2025

View reviewed changes

danholdaway requested a review from aerorahul May 21, 2025 19:50

Jeffrey Whitaker added 7 commits May 23, 2025 10:07

make executable, use BUILD_GSINFO_DIR, remove tabs

e572cd4

use BUILD_GSINFO_DIR

9fbe816

create_*info.sh now executable.

1e53c68

use OBS_INPUT_DIR

e252934

fix typo

2814d4f

add ignore_2mQM

b27e1d9

Merge branch 'develop' into build_gsinfo

0c3167e

		@@ -0,0 +1,18 @@
		# create global_ozinfo file for a given date

		@@ -0,0 +1,27 @@
		# create global_ozinfo file for a given date

	sh ${USHgfs}/create_satinfo.sh ${PDY}${cyc} > satinfo
	${USHgfs}/create_satinfo.sh ${PDY}${cyc} > satinfo

	sh ${USHgfs}/create_convinfo.sh ${PDY}${cyc} > convinfo
	${USHgfs}/create_convinfo.sh ${PDY}${cyc} > convinfo

	sh ${USHgfs}/create_ozinfo.sh ${PDY}${cyc} > ozinfo
	${USHgfs}/create_ozinfo.sh ${PDY}${cyc} > ozinfo

add capability to dynamically generate GSI *info files #3472

Are you sure you want to change the base?

add capability to dynamically generate GSI *info files #3472

Uh oh!

Conversation

jswhit2 commented Mar 18, 2025

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

jswhit2 commented Mar 18, 2025 • edited by jack-woollen Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Check notice

Check notice

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ClaraDraper-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

jswhit commented Apr 3, 2025

Uh oh!

ClaraDraper-NOAA commented Apr 3, 2025

Uh oh!

ClaraDraper-NOAA commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jswhit commented Apr 3, 2025

Uh oh!

DavidHuber-NOAA commented Apr 10, 2025

Uh oh!

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jswhit2 commented May 23, 2025

Uh oh!

Uh oh!

jswhit2 commented Mar 18, 2025 •

edited by jack-woollen

Loading

ClaraDraper-NOAA commented Apr 3, 2025 •

edited

Loading