Bring feature/gdas-validation up to date with develop #861
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-EMC#808) ### Description: - Remove duplicate code (NetCDFToIodaConverter2d.h) and implement the channel dimension to the original constructors. - Update the VIIRS ioda converter ctest with reference test. --------- Co-authored-by: ypwang19 <[email protected]>
This makes the ctest DMPDIR for observation files more like the DMPDIR used in model runs, in order to provide for closer-to-life ctests, and in anticipation of the R2D2-ectomy.
Tested on containers and Hera. The variational test for `soca` and `fv3-jedi` are ok now, but I see failures in land DA ... - fixes NOAA-EMC#814
#### What this PR fixed - Two new ioda converters <del>that work with `mktime` in a C library function</del> that need to ensure consistent behavior across different machines with different time zones (e.g., Orion (CST) and Hera (UTC)) - <del>Set the time zone explicitly using the `tzset` function with UTC time zone</del> - Used ```oops/utils``` to make dateime to Julian date so now code is a lot simpler than before Fixed NOAA-EMC#783 --------- Co-authored-by: Guillaume Vernieres <[email protected]>
…AA-EMC#821) - Moved a simple OceanMask struct that reads in regional ocean masks to base class - Those are an implementation of the feature in all converters Close NOAA-EMC#775 - Tested with one full obs file for each converter: 
The obs prepped and converted to IODA are copied to `ROTDIR` under obs. Also, the order of `JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP` and `JGLOBAL_PREP_OCEAN_OBS` tasks are swapped , as `JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP` creates the target directory. --------- Co-authored-by: Guillaume Vernieres <[email protected]>
This PR adds in changes to clone recursively in CI, plus a fix to not create infinite loops in symlinks (still not clear how it happens but a `rm` should do the trick) Note the stable driver script will be heavily modified as part of the switch to submodules.
There were a number of changes needed to some of the converter scripts. The changes are outlined in issue NOAA-EMC#785 . From that issue: Some updates were made to the following bufr2ioda python sripts. 1. acft_profiles prepbufr script - added units to instantaneousAltitudeRate (should be ms-1) - added ObsError for all variables - added windEastward to QM - dateTime code fix to make it run faster. Missing is now _, 2. ADPSFC prepbufr script - changes to the SFCSHP script that came up during the code sprint needed to be done to ADPSFC as well - I added ObsError for stationPressure. - I edited the height variable down from heightOfStation. - I added stationElevation to MetaData. - datetime fix 3. the conventional ps script. - Added ObsSubType - Added ObsError - Added stationElevation to MetaData - heightOfStation -> height - fixed types - ObsError and QM ==0 are changed to "Missing" - dateTime codefix. Runs slightly faster. Missing is no longer 0, it is _, --------- Co-authored-by: Nicholas Esposito <[email protected]> Co-authored-by: Nicholas Esposito <[email protected]> Co-authored-by: Cory Martin <[email protected]> Co-authored-by: Nicholas Esposito <[email protected]> Co-authored-by: Emily Liu <[email protected]>
) Same as NOAA-EMC#803 but into develop! --------- Co-authored-by: RussTreadon-NOAA <[email protected]> Co-authored-by: Emily Liu <[email protected]> Co-authored-by: emilyhcliu <[email protected]> Co-authored-by: BrettHoover-NOAA <[email protected]> Co-authored-by: Brett Hoover <[email protected]> Co-authored-by: Xuanli Li <[email protected]> Co-authored-by: RussTreadon-NOAA <[email protected]>
The ctest for JGLOBAL_PREP_OCEAN_OBS runs on cycle 2018-04-15-12 like the rest of the g-w task ctests, and processes Metop B GHRSST using the GHRSST IODA converter, and puts them in the appropriate ROTDIR obs directory for later use. Another step towards NOAA-EMC#815 @ShastriPaturi you should probably give this a closer look --------- Co-authored-by: Guillaume Vernieres <[email protected]>
…MC#834) This PR fixes two bugs: 1. Missing datetime "units" 2. Ghrsst2ioda converter outputs sea surface temperature with unit "C" All outputs are placed at each issue at NOAA-EMC#831 and NOAA-EMC#830 Close NOAA-EMC#831 , Close NOAA-EMC#830
@emilyhcliu found that permission restrictions on `bufr2ioda_subpfl_argo_profiles.py` prevent it from being executed. [`feature/chmod`](https://github.com/NOAA-EMC/GDASApp/tree/feature/chmod) was created and the file permissions changed so that `bufr2ioda_subpfl_argo_profiles.py` is executable. Fixes NOAA-EMC#788
- The UFO filters are rearranged by using the explicit filter ordering with `obs pre filters`, `obs prior filters` and `obs post filters` options. - Add `Temporal Thinning filter` into the `obs post filters` section and and arrange it after the `Background Check filter` and before the `Buddy check filter`. - Rename the IMS IODA file name and copy it into the `rundir/obs` directory for the late use in the snow analysis in the global workflow (NOAA-EMC/global-workflow#2033). Co-authored-by: Cory Martin <[email protected]>
This PR is prepared for NOAA-EMC/global-workflow#2199
Removes soca ctests related to R2D2, associated data, and any lines in scripts, yamls, or other config files referencing it. SOCA ctests now run on obs prepped by the prep obs task. Almost takes care of NOAA-EMC#815 as references are still present in global-workflow task config files. `logging.info` has been replaced by `print` because logging hasn't been working --------- Co-authored-by: Guillaume Vernieres <[email protected]>
Add `bound to include` to manipulate the DA window following the recent upgrades in JEDI on the DA window manipulation (PRs [ufo #3056](JCSDA-internal/ufo#3056); [oops #2386](JCSDA-internal/oops#2386) and [ioda NOAA-EMC#1121](JCSDA-internal/ioda#1121)) for handling the loss of the observations due to the difference in NCEP dump convention and IODA convention.
…AA-EMC#847) - I moved all the untested obs `yaml` under `obs/old`, assuming that they probably are all wrong since the removal of R2D2 - I added (fake) AMSR2 icec cdl files - Same as above but for ADT, also skipping the concatenation post ioda processing and concatenating with the new ioda converters - I've made an effort to try to use the provider's file naming convention, but do check.
MSU Hercules is available for use. This PR adds `hercules.lua` to GDASApp `modulefiles/GDAS`. Fixes NOAA-EMC#773 --------- Co-authored-by: Cory Martin <[email protected]>
This PR creates git submodules for each external JEDI library rather than using `ecbuild_bundle` to get the library source code through git. There is also a new set of scripts to automate the update process of the submodules (both an update and a git add, in separate scripts). Peppered through here are also some minor changes for CI testing that are needed due to this change + the global-workflow using submodules. --------- Co-authored-by: Guillaume Vernieres <[email protected]> Co-authored-by: RussTreadon-NOAA <[email protected]> Co-authored-by: RussTreadon-NOAA <[email protected]>
A little (tall?) birdie has informed us that we apparently should not be using DRIPCG, and seems we are using DRIPCG in our tests. This PR removes that. Note that the global-workflow will likely need modified in `config.atmanl` to use the different YAML name.
We'll have to test and readjust the hybrid B weights, since the correlation central block is now normalized! (thanks @travissluka) , but the plumbing is mostly sorted out. - fixes NOAA-EMC#826 ### Issues not addressed yet - Better vertical decorrelation scales - The horizontal diffusion shouldn't be initialized at every cycle.
#### What this PR includes: - Fix test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY by updating `marine_gdas_plots.yaml` format and it related `marine_eva_post.py` - <del>As of now, ctests passed on `Hera Machine` and `Orion Machine` is still needed to check</del> - ctests passed on both `Hera Machine` and `Orion Machine`, see NOAA-EMC#829 Fixes NOAA-EMC#829 Thanks to @CoryMartin-NOAA
…-EMC#844) This PR adds bufr2ioda python API converters for marine monthly in situ data based on the work done during the "End-to-End Code Sprint" by the atmosphere group. partially resolves NOAA-EMC#711 --------- Co-authored-by: Guillaume Vernieres <[email protected]>
|
@emilyhcliu , @ADCollard , and @CoryMartin-NOAA : Is this PR necessary? Are developers still using |
|
I think we are in agreement that we don't need it and we can just use develop |
|
Got it. Close as not needed. At some point we should delete |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
feature/gdas-validationis 25 commits behinddevelop. This PR bringsfeature/gdas-validationup to date withdevelop.Resolves #859