…-EMC#808)

### Description:

- Remove duplicate code (NetCDFToIodaConverter2d.h) and implement the
channel dimension to the original constructors.

- Update the VIIRS ioda converter ctest with reference test.

---------

Co-authored-by: ypwang19 <[email protected]>

This makes the ctest DMPDIR for observation files more like the DMPDIR
used in model runs, in order to provide for closer-to-life ctests, and
in anticipation of the R2D2-ectomy.

Tested on containers and Hera.
The variational test for `soca` and `fv3-jedi` are ok now, but I see
failures in land DA ...
- fixes NOAA-EMC#814

#### What this PR fixed
- Two new ioda converters <del>that work with `mktime` in a C library
function</del> that need to ensure consistent behavior across different
machines with different time zones (e.g., Orion (CST) and Hera (UTC))
- <del>Set the time zone explicitly using the `tzset` function with UTC
time zone</del>
- Used ```oops/utils``` to make dateime to Julian date so now code is a
lot simpler than before

Fixed NOAA-EMC#783

---------

Co-authored-by: Guillaume Vernieres <[email protected]>

…AA-EMC#821)

- Moved a simple OceanMask struct that reads in regional ocean masks to
base class
- Those are an implementation of the feature in all converters

Close NOAA-EMC#775

- Tested with one full obs file for each converter:

![ocean_flag_combined](https://github.com/NOAA-EMC/GDASApp/assets/141867620/e114f1c4-f45a-420e-8336-26075cdfa301)

The obs prepped and converted to IODA are copied to `ROTDIR` under obs.
Also, the order of `JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP` and
`JGLOBAL_PREP_OCEAN_OBS` tasks are swapped , as
`JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP` creates the target directory.

---------

Co-authored-by: Guillaume Vernieres <[email protected]>

This PR adds in changes to clone recursively in CI, plus a fix to not
create infinite loops in symlinks (still not clear how it happens but a
`rm` should do the trick)

Note the stable driver script will be heavily modified as part of the
switch to submodules.

There were a number of changes needed to some of the converter scripts.
The changes are outlined in issue NOAA-EMC#785 .

From that issue:
Some updates were made to the following bufr2ioda python sripts.

1. acft_profiles prepbufr script
      - added units to instantaneousAltitudeRate (should be ms-1)
      - added ObsError for all variables
      - added windEastward to QM
      - dateTime code fix to make it run faster. Missing is now _, 

2. ADPSFC prepbufr script - changes to the SFCSHP script that came up
during the code sprint needed to be done to ADPSFC as well
      - I added ObsError for stationPressure. 
      - I edited the height variable down from heightOfStation. 
      - I added stationElevation to MetaData.
      - datetime fix

3. the conventional ps script.
      - Added ObsSubType
      - Added ObsError
      - Added stationElevation to MetaData
      - heightOfStation -> height
      - fixed types
      - ObsError and QM ==0 are changed to "Missing"
- dateTime codefix. Runs slightly faster. Missing is no longer 0, it is
_,

---------

Co-authored-by: Nicholas Esposito <[email protected]>
Co-authored-by: Nicholas Esposito <[email protected]>
Co-authored-by: Cory Martin <[email protected]>
Co-authored-by: Nicholas Esposito <[email protected]>
Co-authored-by: Emily Liu <[email protected]>

)

Same as NOAA-EMC#803  but into develop!

---------

Co-authored-by: RussTreadon-NOAA <[email protected]>
Co-authored-by: Emily Liu <[email protected]>
Co-authored-by: emilyhcliu <[email protected]>
Co-authored-by: BrettHoover-NOAA <[email protected]>
Co-authored-by: Brett Hoover <[email protected]>
Co-authored-by: Xuanli Li <[email protected]>
Co-authored-by: RussTreadon-NOAA <[email protected]>

The ctest for JGLOBAL_PREP_OCEAN_OBS runs on cycle 2018-04-15-12 like
the rest of the g-w task ctests, and processes Metop B GHRSST using the
GHRSST IODA converter, and puts them in the appropriate ROTDIR obs
directory for later use.

Another step towards NOAA-EMC#815

@ShastriPaturi you should probably give this a closer look

---------

Co-authored-by: Guillaume Vernieres <[email protected]>

…MC#834)

This PR fixes two bugs:

1. Missing datetime "units"
2. Ghrsst2ioda converter outputs sea surface temperature with unit "C"

All outputs are placed at each issue at
NOAA-EMC#831 and
NOAA-EMC#830

Close NOAA-EMC#831 , Close
NOAA-EMC#830

@emilyhcliu found that permission restrictions on
`bufr2ioda_subpfl_argo_profiles.py` prevent it from being executed.
[`feature/chmod`](https://github.com/NOAA-EMC/GDASApp/tree/feature/chmod)
was created and the file permissions changed so that
`bufr2ioda_subpfl_argo_profiles.py` is executable.

Fixes NOAA-EMC#788

- The UFO filters are rearranged by using the explicit filter ordering
with `obs pre filters`, `obs prior filters` and `obs post filters`
options.
- Add `Temporal Thinning filter` into the `obs post filters` section and
and arrange it after the `Background Check filter` and before the `Buddy
check filter`.
- Rename the IMS IODA file name and copy it into the `rundir/obs`
directory for the late use in the snow analysis in the global workflow
(NOAA-EMC/global-workflow#2033).

Co-authored-by: Cory Martin <[email protected]>

This PR is prepared for
NOAA-EMC/global-workflow#2199

Removes soca ctests related to R2D2, associated data, and any lines in
scripts, yamls, or other config files referencing it. SOCA ctests now
run on obs prepped by the prep obs task.

Almost takes care of NOAA-EMC#815 as
references are still present in global-workflow task config files.

`logging.info` has been replaced by `print` because logging hasn't been
working

---------

Co-authored-by: Guillaume Vernieres <[email protected]>

Add `bound to include` to manipulate the DA window following the recent
upgrades in JEDI on the DA window manipulation (PRs [ufo
#3056](JCSDA-internal/ufo#3056); [oops
#2386](JCSDA-internal/oops#2386) and [ioda
NOAA-EMC#1121](JCSDA-internal/ioda#1121)) for handling
the loss of the observations due to the difference in NCEP dump
convention and IODA convention.

…AA-EMC#847)

- I moved all the untested obs `yaml` under `obs/old`, assuming that
they probably are all wrong since the removal of R2D2
- I added (fake) AMSR2 icec cdl files
- Same as above but for ADT, also skipping the concatenation post ioda
processing and concatenating with the new ioda converters
- I've made an effort to try to use the provider's file naming
convention, but do check.

MSU Hercules is available for use. This PR adds `hercules.lua` to
GDASApp `modulefiles/GDAS`.

Fixes NOAA-EMC#773

---------

Co-authored-by: Cory Martin <[email protected]>

This PR creates git submodules for each external JEDI library rather
than using `ecbuild_bundle` to get the library source code through git.

There is also a new set of scripts to automate the update process of the
submodules (both an update and a git add, in separate scripts).

Peppered through here are also some minor changes for CI testing that
are needed due to this change + the global-workflow using submodules.

---------

Co-authored-by: Guillaume Vernieres <[email protected]>
Co-authored-by: RussTreadon-NOAA <[email protected]>
Co-authored-by: RussTreadon-NOAA <[email protected]>

A little (tall?) birdie has informed us that we apparently should not be
using DRIPCG, and seems we are using DRIPCG in our tests. This PR
removes that. Note that the global-workflow will likely need modified in
`config.atmanl` to use the different YAML name.

We'll have to test and readjust the hybrid B weights, since the
correlation central block is now normalized! (thanks @travissluka) , but
the plumbing is mostly sorted out.
- fixes NOAA-EMC#826

### Issues not addressed yet
- Better vertical decorrelation scales
- The horizontal diffusion shouldn't be initialized at every cycle.

#### What this PR includes:

- Fix test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY

by updating `marine_gdas_plots.yaml` format and it related
`marine_eva_post.py`

- <del>As of now, ctests passed on `Hera Machine` and `Orion Machine` is
still needed to check</del>
- ctests passed on both `Hera Machine` and `Orion Machine`, see
NOAA-EMC#829

Fixes NOAA-EMC#829

Thanks to @CoryMartin-NOAA

…-EMC#844)

This PR adds bufr2ioda python API converters for marine monthly in situ
data based on the work done during the "End-to-End Code Sprint" by the
atmosphere group.

partially resolves NOAA-EMC#711

---------

Co-authored-by: Guillaume Vernieres <[email protected]>