Create a GSI branch with configuration to use CRTM-2.4.1-jedi.1 #93
Comments
|
@CoryMartin-NOAA @RussTreadon-NOAA |
|
Thank you @emilyhcliu for creating a GSI fork which can use I updated a working copy of I am currently working through the sequence of steps to clone, build, setup, and run jobs. |
|
@emilyhcliu , where my I find the run script you used to test the |
|
@RussTreadon-NOAA is the issue a shared library is missing? I think, if so, then |
|
@CoryMartin-NOAA , yes the initial problem was the shared library. Thank you for the pointer. I added the crtm path to With the addition of A check of The log file contains 690927 lines of I built It would be helpful to examine a |
|
I found the following in the EMC JEDI Discussions google space
This is a g-w change since EID version controls |
@RussTreadon-NOAA I was out this morning to see dentist. There are three scripts (provided by Cory for our previous code sprint) You just need to modify the path to GSI in ps. I already turned off iodaconv.sh. |
|
Thank you @emilyhcliu for pointing me at the scripts you use to run |
|
@emilyhcliu , this is very odd. Using my @CoryMartin-NOAA , have you run |
|
@RussTreadon-NOAA no I have not, I thought you had to use big endian, since presumably the GSI is compiled with big endian and then the BERROR_STATS file, and others, will also be big endian. |
Agreed! The gsi code is compiled with big endian compiler flags. The GSI static-B file is big endian. The CRTM coefficients being provided to The EMC JEDI Discussions note and comments above indicate that we need to use little endian coefficients
|
|
Change With this change |
|
@RussTreadon-NOAA @emilyhcliu can we use |
Good suggestion. I tried. |
|
@RussTreadon-NOAA @CoryMartin-NOAA I added the Big_Endian files for crtm-v2.4.0_emc.3. CRTM_FIX=/work/noaa/da/eliu/JEDI-GDAS/crtm-v2.4.0_emc.3-fix/Big_Endian The crtm_v2.4.1-jedi-1 fix files have We want n21 in the filename. And, we also use amsua_metop-a_v2.SpcCoeff.bin in our operational GFS. N21 and the amsua_metop-a_v2 files are in crtm-v2.4.0_emc.3-fix only. |
|
@emilyhcliu , unfortunately, setting The executable aborted with the previously mentioned byte-swapped error. Here's the error message from a representative task, 36 |
|
Reverting Perhaps the issue is with the |
|
Can we ask the library team to install CRTM 2.4.1-jedi.1 on Orion? Then we could directly load Just a thought. |
|
@emilyhcliu and @CoryMartin-NOAA , I will pause work on this issue until we have a clear path forward. |
|
This is very strange, why would it work for @emilyhcliu but not you, @RussTreadon-NOAA . Am I correct in understanding we seem to get the same error regardless of big or little endian coefficients? |
|
Yes, @CoryMartin-NOAA , your understanding is correct. Given your comment, I did the following this morning
With With With
A run using |
|
As an additional test, do the following
Summary: g-w and stand-alone script behavior is consistent when run from my Orion account. |
|
Hmm, ugh, I suggest we wait for @emilyhcliu before digging further as she apparently has the magic touch to get this working |
Agreed! |
|
Issues sorted out in GSI gdas-validation test. I was using the wrong
@emilyhcliu , your working copy I recommend that we do not modify Look in Orion I retained your local modification to |
|
@RussTreadon-NOAA I updated the branch with your suggestion and did a single-cycle run (observer only). It ran to completion without issues. |
|
Thank you @emilyhcliu . Let me keep debugging |
|
Debugging found differences in some fix files, gsi namelist settings, and processing of HIRS dump files.
With the above additions to expdir @emilyhcliu explained that the HIRS change is required when using CRTM-2.4.1-jedi.1. What about the fix file and gsi namelist changes? Which of these changes do we need or want to include in gdas-validation? |
|
@emilyhcliu , @ADCollard , & @CoryMartin-NOAA Additional differences between run using Emily's stand-alone script and the gdas-validation g-w gdasanal job:
If someone can point me at the GSI configuration we want to use for gdas-validation, I can create a JEDI-T2O branch in which |
|
We are using the focus cycle (2021080100) for our evaluation and should be using the focus cycle for the code sprint. So, people can use the 2021080100 geoval and obs files from the UFO evaluation as a reference. These files contain GSI output (e.g. HofX and some derived variables). The GSI workflow for the code sprint provides a way for people to re-run the focus cycle the GFS processing from prep step to observer part of the first outer loop. And people can configure it to run with different configurations or cycle. For example, for me (working on radiances), I can change the all-sky related namelist and configuration files (anavinfo, satinfo, ...etc) for the latest update (ta2tb is true and use updated anavinfo... etc) in the operational system. @RussTreadon-NOAA I think we should keep namelist setting and configuration as the same as we run the 2021080100. For the code sprint, we are not seeking bit-identical result since we will be checking end-to-end comparion between GDAS and JEDI. They are some fundamental difference between the two in current status. So, bottom line, we need to have the following setup in the GSI workflow: We should also turn off Hilbert curve for aircraft data. @CoryMartin-NOAA and @ADCollard found that the switch for Hilbert curve is hardwired in GSI. So, we need to turn it off in the code, not from the script. @CoryMartin-NOAA and @ADCollard, could you give guidence for this? |
|
@emilyhcliu @RussTreadon-NOAA The Hilbert Curve code starts at line 3007 of read_prepbufr.f90. The entire if-block statrting |
|
Namelist Would this suffice? |
Not only would it suffice, but we should probably add this to the develop branch.... |
|
One question, one suggestion, and one request
|
OK, we can open an issue and get it into |
I tested this change in GSI tag Operations run the global GSI with logical I'm confused. The original code in The comment in the code along with @ADCollard 's guidance suggest that this block should only be executed when |
|
Seems my understanding of logical Logical |
|
The GDAS-validation sprint begins Monday, 11/13. Next week (6-10 Nov) is a short work week (Friday, 11/10 is the Veterans day holiday). I'm not available 11/10 through 11/12. Work remains to prepare Here's a partial listing of our work items:
What other pre-sprint work should be added to the above list? |
|
Given that Orion is slow, do we move to Hera? I think the savings in runtime will be offset by the longer job queues though... Thanks @RussTreadon-NOAA I think this is a good list. The biggest one is item 3 , @ADCollard and @emilyhcliu what all should we turn off in GSI? I know we need to turn off FGAT, the time thinning error inflation, VarQC. But what else? |
|
I'm wrestling with the move to Hera, too. The 11/13 sprint is step one of gdas validation in that it will focus on the observer (ufo), right? If true, we will have a step two gdas validation at a later date where we compare gsi.x & fv3jedi_var.x minimization (solver) including varbc. Not all this work will be done on Orion (or Hercules). So long term I think we want to extend The gdas validation sprint isn't the only thing DAD staff are working on. Are we using using Hera for GFS v17 tests? GFS v17 includes JEDI based marine, land, and aerosol DA. Maybe we reserve Hera for GFS v17 & related JEDI work and keep gdas validation on Orion for the time being. Thoughts? Comments? |
|
I think the extension to Hera is fairly straightforward, the only real sticking point would be mirroring the input data to Hera from Orion. We would need to stage FMS restarts, Gaussian history files, bias correction files, (and observations should be in the glopara space already). This isn't difficult, but it does take up space. Hera space is at a premium compared to Orion. I also agree that Hera is probably better spent on the GFS T2O specific tasks and use Orion for this lower readiness level testing. GSI may be running slow on Orion, but it still runs. |
|
Agreed. The GSI still runs on Orion ... it's just slow. One suggestion from the Orion helpdesk is to recompile the stack we use to build GSI. I asked in g-w issue #1996 about getting this done. GSI slowness on Orion will be addressed. We also have the possibility of using Hercules in the future, though it seems some executables also run slow on Hercules. |
|
Conduct the following test on Orion.
The job ran to completion. The run directory is The With the above changes |
|
I am also using |
|
Great! Shall we update |
|
The following branches have been created in the following repositories for possible use in the GDAS validation sprint:
It is not clear if all the changes in JEDI-T2O branch Two additional considerations
|
Lines 3007 to 3165 have been commented out in the snapshot of |
For the upcoming end-to-end ode sprint, we would like to have a GSI branch with a configuration to use CRTM 2.4.1-jedi.1, which is consistent with the CRTM used in GDASApp.
GSI
The GSI branch created for this PR is in the following GSI forked repository:
GSI-crtm_v2.4.1-jedi.1
These are changes to GSI.
CRTM
The CRTM version 2.4.1-jedi.1 is built on ORION in the following location:
/work/noaa/da/eliu/JEDI-GDAS/crtm_v2.4.1-jedi.1-intel2022
The associated coefficients files are in the following location:
/work/noaa/da/eliu/JEDI-GDAS/crtm_v2.4.1-jedi.1-fix/Little_Endian
There are three sets of CRTM coefficients we can:
I tested three sets, and they all worked fine.
The first one contains the coefficients we use in the operation + N21 coefficients
The second one is the one linked to GDASApp from the run_ufo_hofx_test.sh
The third one is the coefficients packed with crtm_v2.4.1-jedi.1 tag.
The first set of coefficients is good for our purpose for UFO evaluation with GSI.
The text was updated successfully, but these errors were encountered: