Format changes #773
Format changes #773
Conversation
…e name is 11 characters and the gsi is set for 10. Also, the channel numbers for iasi-ng can exceed 9999 so 5 digits are needed.
…o format_changes
|
@RussTreadon-NOAA I am not able to assign reviewers. I suggest @emilyhcliu as she will be working on the metop-sg-a1 microwave sounder. If anyone else plans to work on a metop-sg-a1 instrument, they may want to participate. Other potential reviewers would be Andrew Collard, Dave Huber or Innocent Souopgui. FYI the branch name is "format_changes" |
|
@wx20jjung : @emilyhcliu has been added as a reviewer. Please reach out to the potential reviewers you listed to see who else can review your changes. GSI PRs need to peer reviews and approvals. |
|
@ADCollard, @DavidHuber-NOAA @InnocentSouopgui-NOAA Would any of you be willing to review my "format_changes" branch? This is my changes for adding metop-sg-a1 instruments. |
Happy to |
|
Thank you @ADCollard |
|
@wx20jjung Is there any reason why we are only increasing the length by one character. Maybe that is enough, but is it worth making it a little larger to reduce the risk of doing this exercise again? |
I second this. |
I will review. |
|
This is not a big concern, but a flag that can slow down the review: |
|
I have 2 reasons for only increasing the length by 1. The GSI sunset date is approaching and metop-sg-a? could be the last satellite added, especially with such a long name. The other reason, the current length of 10 has been adequate for decades. I can make it any length as long as there is a consensus. My goal was to make changing the length of all the variables involved simple. In this case the original variable is dplat(). dplat() is a public (common) variable and can be added with a use statement. I originally tried to make the length of platid dependent on the length of dplat(). My tests did not work. The alternative was to change platid to dplat. |
|
In the format change: The 2A12 does leave at least one space between them. |
|
@wx20jjung Agreed. Just a suggestion since |
There was a problem hiding this comment.
My main comment:
in src/gsi/read_diag.f90
was the changes here tested with binary files? I didn't see where those files are created to check if the creation is updated accordingly.
for instance subroutine read_radiag_header_bin reads record including satid that the size just changed from 10 to 11.
in src/gsi/statsrad.f90
Good catch (A16 => A20) for the 3rd element in the format # 1102, 16 was smaller than the size of the variable
Great update (2a10 to 2A12) in format # 1115, ensuring a space for readability
|
@InnocentSouopgui-NOAA I am not sure how to test the _bin (binary) option. The defaults are netcdf. Any suggestions? In setuprad.f90 the header is written by subroutine init_binary_diag_ A little further down the write statement is: To be consistent, satid should be the same length as dplat(). |
|
Thanks @wx20jjung, That write in setuprad solves my concern. |
|
WCOSS2 (Dogwood) ctests This is an expected result ... still nice to get confirmation of no failures. |
|
Hercules ctests Did not run ctests on Orion due to (1) problem with |
|
@wx20jjung , do we need to make any changes in I see the following There are other |
|
@RussTreadon-NOAA Your examples are of concern. I did not look through src/enkf and should have. I will review the src/enfk files next week and make the necessary changes. |
|
Thank you @wx20jjung for checking code in |
|
@russ Treadon - NOAA Federal ***@***.***>
The only character string that needs extending is for the satellite name
i.e. metop-sg-a1. I see where the instrument and satellite names are
combined but not where the satellite is separated out. The character
string is long enough for the combined length (character(len=20)) for
iasi-ng_metop-sg-a1. Unfortunately, dplat has a different meaning in these
subroutines.
Thanks to Steve and Iliana,I think I have enough proxy data to do a cycled
diag file write and read now. I will check through the enkf output to see
if I can find any length problems.
…On Mon, Jul 15, 2024 at 3:23 PM RussTreadon-NOAA ***@***.***> wrote:
Thank you @wx20jjung <https://github.com/wx20jjung> for checking code in
src/enkf to see which files need to be updated. Getting to this next week
is fine.
—
Reply to this email directly, view it on GitHub
<#773 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMPASA4VSZNPMYNR5WDOP53ZMQORFAVCNFSM6AAAAABKXA4FTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRZGIZDAMJSGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Thank you @wx20jjung for the update. Please commit the necessary changes to |
… IASI (AVHRR) to avoid zero values for cluster size and radiance values. I also removed an unused variable.
|
Thank you @wx20jjung for updating |
|
@wx20jjung , please commit changes to |
|
In order to test the format changes for the enkf portion, I have to
incorporate the various read and write components of iasi-ng_metop-sg-1a
(and metimage_metop-sg-1a) within the workflow and gsi. I only have proxy
data for iasi-ng_metop-sg-1aThis is not a clean test.
The enkfgdaseobs is reading the file as expected.
read_obs_check: bufr file date is 2023042518 iasingbufr iasi-ng
metop-sg-a1
data type metimage_metop-sg-a1not used in info file -- do not read file
READ_OBS: read 113 iasi-ng iasi-ng_metop-sg-a1 using ntasks= 16 0
3 999999 999999
The enkfgdasdiag also seems to be processing the files as expected with the
generation of the diag file:
diag_iasi-ng_metop-sg-a1_ges.2023042518_ensmean.nc4.
The enkfgdaseupd output is limited. The diag file exists as expected and
has a reasonable size;
33167491 Jul 27 00:37 diag_iasi-ng_metop-sg-a1_ges.2023042518_ensmean.nc4
Output from stderr file is reasonable:
4364 iasi-ng_metop-sg-a1 chan= 29 var= 1.000 varch_cld= 0.000 use=
1 ermax= 2.000 b_rad= 10.00 pg_rad= 0.00 icld_det= 1 icloud=-1
iaeros=-1
0: 4364 iasi-ng_metop-sg-a1 0.115006 0.000000 0.000000
0.016922 -0.374549 0.000000 0.000000 11.261273 -40.967182
60.998313 -28.848502 4.773325
On s4, enkfgdaseupd completed as expected.
End eupd.sh at 12:32:24 with error code 0 (time elapsed: 00:09:15)
I made no changes to any ./src/enkf files. This branch now also contains
the changes for #775.
Here are the ctest results from hera:
Test project /scratch1/NCEPDEV/jcsda/Jim.Jung/scrub/ctests_russ/update/build
Start 1: global_4denvar
Start 2: rtma
Start 3: rrfs_3denvar_rdasens
Start 4: hafs_4denvar_glbens
Start 5: hafs_3denvar_hybens
Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens ............. Passed 614.33 sec
2/6 Test #2: rtma ............................. Passed 971.77 sec
3/6 Test #5: hafs_3denvar_hybens .............. Passed 1045.52 sec
4/6 Test #6: global_enkf ...................... Passed 1147.59 sec
5/6 Test #4: hafs_4denvar_glbens .............. Passed 1225.27 sec
6/6 Test #1: global_4denvar ................... Passed 1970.79 sec
100% tests passed, 0 tests failed out of 6
Total Test time (real) = 1970.81 sec
Jet continues to be slow. Will post results when they finish.
…On Mon, Jul 29, 2024 at 6:46 AM RussTreadon-NOAA ***@***.***> wrote:
@wx20jjung <https://github.com/wx20jjung> , please commit changes to
src/enkf to wx20jjung:format_changes. Once this is done, we can do one
more round of ctests. We can then schedule this PR for merger into develop
pending successful ctests results.
—
Reply to this email directly, view it on GitHub
<#773 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMPASA5OLB3UZ2GUDWTE22TZOYMRFAVCNFSM6AAAAABKXA4FTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJVGU4TSMBQGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Here are my stats from Jet:
Test project
/mnt/lfs4/NESDIS/nesdis-rdo1/Jim.Jung/noscrub/ctests/update/build
Start 3: rrfs_3denvar_rdasens
Start 4: hafs_4denvar_glbens
Start 2: rtma
Start 1: global_4denvar
Start 5: hafs_3denvar_hybens
Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............***Failed 667.91 sec
2/6 Test #2: rtma ............................. Passed 1996.96 sec
3/6 Test #6: global_enkf ......................***Failed 2005.69 sec
4/6 Test #5: hafs_3denvar_hybens .............. Passed 2070.87 sec
5/6 Test #4: hafs_4denvar_glbens ..............***Failed 2247.57 sec
6/6 Test #1: global_4denvar ................... Passed 2387.81 sec
The rrfs_3denvar_rdasens test failed with a missing file
forrtl: severe (24): end-of-file during read, unit 24, file
/mnt/lfs4/NESDIS/nesdis-rdo1/Jim.Jung/ptmp/tmpreg_rrfs_3denvar_rdasens/rrfs_3denvar_rdasens_loproc_updat/coupler.res.
Both the hafs_4denvar_glbens and global_enkf failed the run time limit
tests.
The runtime for global_enkf_hiproc_updat is 159.765957 seconds. This has
exceeded maximum allowable threshold time of 144.564952 seconds,
resulting in Failure timethresh2 of the regression test.
The runtime for hafs_4denvar_glbens_hiproc_updat is 299.077362 seconds.
This has exceeded maximum allowable threshold time of 268.541397 seconds,
resulting in Failure of timethresh2 the regression test.
On Mon, Jul 29, 2024 at 11:27 AM Jim Jung - NOAA Affiliate <
***@***.***> wrote:
… In order to test the format changes for the enkf portion, I have to
incorporate the various read and write components of iasi-ng_metop-sg-1a
(and metimage_metop-sg-1a) within the workflow and gsi. I only have proxy
data for iasi-ng_metop-sg-1aThis is not a clean test.
The enkfgdaseobs is reading the file as expected.
read_obs_check: bufr file date is 2023042518 iasingbufr iasi-ng
metop-sg-a1
data type metimage_metop-sg-a1not used in info file -- do not read file
READ_OBS: read 113 iasi-ng iasi-ng_metop-sg-a1 using ntasks= 16 0
3 999999 999999
The enkfgdasdiag also seems to be processing the files as expected with
the generation of the diag file:
diag_iasi-ng_metop-sg-a1_ges.2023042518_ensmean.nc4.
The enkfgdaseupd output is limited. The diag file exists as expected and
has a reasonable size;
33167491 Jul 27 00:37 diag_iasi-ng_metop-sg-a1_ges.2023042518_ensmean.nc4
Output from stderr file is reasonable:
4364 iasi-ng_metop-sg-a1 chan= 29 var= 1.000 varch_cld= 0.000 use=
1 ermax= 2.000 b_rad= 10.00 pg_rad= 0.00 icld_det= 1 icloud=-1
iaeros=-1
0: 4364 iasi-ng_metop-sg-a1 0.115006 0.000000 0.000000
0.016922 -0.374549 0.000000 0.000000 11.261273 -40.967182
60.998313 -28.848502 4.773325
On s4, enkfgdaseupd completed as expected.
End eupd.sh at 12:32:24 with error code 0 (time elapsed: 00:09:15)
I made no changes to any ./src/enkf files. This branch now also contains
the changes for #775.
Here are the ctest results from hera:
Test project
/scratch1/NCEPDEV/jcsda/Jim.Jung/scrub/ctests_russ/update/build
Start 1: global_4denvar
Start 2: rtma
Start 3: rrfs_3denvar_rdasens
Start 4: hafs_4denvar_glbens
Start 5: hafs_3denvar_hybens
Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens ............. Passed 614.33 sec
2/6 Test #2: rtma ............................. Passed 971.77 sec
3/6 Test #5: hafs_3denvar_hybens .............. Passed 1045.52 sec
4/6 Test #6: global_enkf ...................... Passed 1147.59 sec
5/6 Test #4: hafs_4denvar_glbens .............. Passed 1225.27 sec
6/6 Test #1: global_4denvar ................... Passed 1970.79 sec
100% tests passed, 0 tests failed out of 6
Total Test time (real) = 1970.81 sec
Jet continues to be slow. Will post results when they finish.
On Mon, Jul 29, 2024 at 6:46 AM RussTreadon-NOAA ***@***.***>
wrote:
> @wx20jjung <https://github.com/wx20jjung> , please commit changes to
> src/enkf to wx20jjung:format_changes. Once this is done, we can do one
> more round of ctests. We can then schedule this PR for merger into
> develop pending successful ctests results.
>
> —
> Reply to this email directly, view it on GitHub
> <#773 (comment)>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AMPASA5OLB3UZ2GUDWTE22TZOYMRFAVCNFSM6AAAAABKXA4FTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJVGU4TSMBQGM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
|
Thank you @wx20jjung. |
|
WCOSS2 test |
|
WCOSS2 debug test global_4denvar global_enkf This failure is reported in issue #776. The same failure occurs when running the debug hafs_3denvar_hybens and hafs_4denvar_glbens This failure is not due to changes in this PR. The debug rtma The same error occurs when running the debug rrfs_3denvar_rdasens_ |
* origin/develop: Move to contrib spack-stack on Jet (NOAA-EMC#787) a quick workaround for increasing the mpi task numbers on orion for ctest :: rrfs_3denvar_rdasens (NOAA-EMC#788) Recover the capability of handling model fields from operation gfs.v16.3 (NOAA-EMC#785) fix a bug in deter_sfc_gmi (NOAA-EMC#781) add safeguard to thompson_reff (NOAA-EMC#779) Fix incorrect usage of real(i_kind) in mg_input.f90 (NOAA-EMC#760) Transition to Thompson Microphysics for Microwave All-sky Assimilation (NOAA-EMC#743) Format changes for EUMETSAT metop-sg and CADS debug fix (NOAA-EMC#773) Update global_4denvar and global_enkf ctests to reflect GFS v17 (NOAA-EMC#774) fix for cris-fsr memory corruption (NOAA-EMC#767) Gnssrwnd1.0 (NOAA-EMC#747)
Description
The next generation polar satellites from EUMETSAT, metop-sg-a?, have more characters (11) in their name than the GSI is currently coded to use (10). Also, the new IASI instrument, IASI-NG, has more than 10,000 channels. The current GSI uses I4 in formats for write statements, These changes increase this to I5.
Without these changes issues will arise with various files like the satinfo file when the new satellites are added. There are other internal issues like generating the radstat files and loading the correct coefficient files for the CRTM, where the name would be clipped and potentially conflict with follow-on satellites e.g. iasi-ng_metop-sg-a1 would conflict with iasi-ng_metop-sg-a2 because the 1 and 2 in the name would be clipped.
I had concerns about how to best code the character() changes. See discussion #769
I followed Innocent Souopgui's comments in my coding.
The changes in this PR are format changes only in preparation for metop-sg-a? This fixes #770.
This PR also fixes #775
Type of change
How Has This Been Tested?
I have tested these changes on S4 with proxy data for IASI-NG_metop-sg-a1 in a short, 2 cycle experiment. I added IASI-NG channels to the satinfo file and checked the various outputs (bias correction files, gsistat file output radstat file names, and etc. Current satellites work properly and all of the new satellite and channel number format problems I could find are fixed. These changes should not change results with the current satellites and have not in my current tests.
My ctests on hera passed. Both rrfs ctests on jet (develop and update) failed. All other tests on jet passed. The GSI is not able to fully test these changes until supporting read subroutines for metop-sg-a? are included.
My jet statistics:
1/6 Test #3: rrfs_3denvar_rdasens .............***Failed 1980.84 sec
2/6 Test #2: rtma ............................. Passed 3552.39 sec
3/6 Test #5: hafs_3denvar_hybens .............. Passed 3558.76 sec
4/6 Test #4: hafs_4denvar_glbens .............. Passed 3737.34 sec
5/6 Test #6: global_enkf ...................... Passed 3935.60 sec
6/6 Test #1: global_4denvar ................... Passed 3970.37 sec
My hera statistics:
1/6 Test #3: rrfs_3denvar_rdasens ............. Passed 1637.22 sec
2/6 Test #6: global_enkf ...................... Passed 2087.80 sec
3/6 Test #1: global_4denvar ................... Passed 2929.82 sec
4/6 Test #5: hafs_3denvar_hybens .............. Passed 18541.04 sec
5/6 Test #4: hafs_4denvar_glbens .............. Passed 18605.18 sec
6/6 Test #2: rtma ............................. Passed 18648.58 sec
Checklist