…te obsfile for satwnd.abi_goes-16

…s/0.35.0

Two updates for GOES AMV

1. BUFR Converter:
- remove wind direction and speed
- change sensorZenithAngle to satelliteZenithAngle
2. UFO Filters:
- change `sensorZenithAngle` to `satelliteZenithAngle`
- add `linear obs operator` section
The declaration of `linear obs operator" is necessary for GOES AMV since
there is no linearized component for applying the 10-meter factor.

The increments look reasonable:
```
0: ----------------------------------------------------------------------------------------------------
  0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-1.642335e-01 Max:+1.660654e-01 RMS:+2.746685e-04
  0: northward_wind                               | Min:-2.120068e-01 Max:+1.663539e-01 RMS:+2.631295e-04
  0: air_temperature                              | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ----------------------------------------------------------------------------------------------------
```

The UFO filter results vs. GSI  -- looks good


![ufo_abi_goes-16_omf_windEastward_qc_time1](https://github.com/NOAA-EMC/GDASApp/assets/36091766/017aeabc-d6ca-4643-b037-16f46dde76a0)


![gsi_abi_goes-16_omf_windEastward_qc_time1](https://github.com/NOAA-EMC/GDASApp/assets/36091766/d99599a2-3f36-4ef6-aa2e-4611500b8c43)

Rather than each new ob type being manually added to the list, we can
use `glob.glob` to get the list of available observation types.

Two updates for MetOp SCATWIND; one minor updates for GOES AMV 

1. BUFR Converter:
    - remove wind direction and speed
    - add `stationElevation` as zero 

2. UFO Filters:
    - add linear obs operator section
The declaration of `linear obs operator" is necessary for GOES AMV since
there is no
      linearized component for applying the 10-meter factor.
- add `Gaussian Thinning` filter to thin data with 75 km box (equal box
size, prefer data close to central of the box)

3. For GOES AMV YAML files, change output data extension from `nc4` to
`nc`

UFO vs. GSI O-F and QC comparison: O-F and data count passed look close
![ufo_ascat_metop-b_omf_windEastward_qc
(1)](https://github.com/NOAA-EMC/GDASApp/assets/36091766/f234543e-e0b5-4c6e-add9-3c7365c0c469)

![gsi_ascat_metop-b_omf_windEastward_qc](https://github.com/NOAA-EMC/GDASApp/assets/36091766/37f4fba8-bd2a-4a54-b45e-4055e3ef3142)

3Dvar increments are reasonable with MetOp-B SCATWIND alone.
```
0: ----------------------------------------------------------------------------------------------------
  0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-1.741092e+00 Max:+1.712287e+00 RMS:+2.282584e-03
  0: northward_wind                               | Min:-1.825957e+00 Max:+1.930385e+00 RMS:+2.496075e-03
  0: air_temperature                              | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ----------------------------------------------------------------------------------------------------

```

3DVar increments are also reasonable with MetOp-B + MetOp-A assimilated

…oda.x (#730)

This PR allows for all obtypes to be processed in two loops.

We still need to figure out a way to process these in parallel
(somewhat).

Using python multiprocessing to generate obs in parallel. The current
list of obs goes from 10+ minutes to completing in ~5.5 minutes.

@RussTreadon-NOAA noted that `obs linear operator` should be `linear obs
operator` in the GOES-17 satwind YAML. This fixes that.

This reduces runtime further to run everything in one parallel pool.
Now, things complete in under 4 minutes (as of now).

Adding satwinds from the Advanced Himawari Imager (AHI) from Himawari-8
to GDASApp end-to-end testing

new files include:
parm/atm/obs/config/satwind_ahi_h8.yaml: QC filter YAML for AHI
Himawari-8 satwinds
parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.json: JSON containing data
format, sensor, and satellite information for AHI Himawari-8 satwinds
ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.py: bufr2ioda code for
extracting AHI Himawari-8 satwinds from BUFR

modified files include:
ush/ioda/bufr2ioda/run_bufr2ioda.py: added `"satwind_amv_ahi"` to list
`BUFR_py`

See #741 for details on testing. JEDI/GSI comparisons look good with GSI
thinning turned off, but there are large ob-count disparities both
before and after QC when comparing GSI+thinning to JEDI+thinning. The
tested `Gaussian Thinning` filter is included in the YAML but commented
out with a note.

---------

Co-authored-by: Brett Hoover <[email protected]>
Co-authored-by: Cory Martin <[email protected]>

merge gdas-validation

Instead of writing the filled out `.json` and `.yaml` files to
`$COM_OBS`, we write them to `$DATA` so that they get purged unless
`KEEPDATA="YES"`

…msua

Updating GNSSRO converter and adding the yaml file to GDASApp end-to-end
testing based on [#712](#712).

New file:
parm/atm/obs/config/gnssro.yaml: Added QC filters in the YAML

Updated files:
parm/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.json: Updated the satellite
information
ush/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.py: Updated pccf, qfro, and
satelliteAscending flag

Refer to [issue #750](#750)
for testing details. JEDI/GSI comparisons revealed a cutoff near
obs=0.03 Rad due to super refraction 2. The number of observations after
QC in JEDI is also inconsistent with GSI.

List of items to be added in this PR:
- BUFR Converter: Add YAML for converting ATMS BUFR to IODA
   - The ObsValue is antenna temperature
   - Input BUFR is normal feel 

- Testing YAML: re-evaluation results due to update CRTM from v2.3.0 to
v2.4.1-jedi.1
   - atms_npp_noqc.yaml --- 100% replication
   - atms_npp.yaml  --- 100 % replication
   - atms_n20_noqc.yaml --- 100 % replication
- atms_n20.yaml --- ~ 1100% replication: two channel 7 observations
difference
  
- Config YAML: 
* add two read routine QC: data thinning and removal of data from scan
edge
   - atms_npp.yaml
   - atms_n20.yaml   

- Paring PR in JCSDA UFO repository
- This PR also has a [paring PR in JCSDA UFO
repository](JCSDA-internal/ufo#3094).
   - It is OK to merge this one into GDASApp before the paring UFO PR
- The related test data (obs and geoval files from GSI) for UFO
Evaluation have been updated (due to CRTM-2.4.1 update) on ORION and
HERA.


Notes:
- End-to-end testing completed without problem. 
- However, the filter results show one of the filters, 88-165 GHz
scattering index check, tossed more data compared to GSI.
- The excess screening from the 88-165 GHz scattering index check is
under investigation and will be reported separately.
- Please check comments below for validation results

…8, n15, metop-b and metop-c

…nvention.

…o-end testing (#764)

Adding satwinds from the Spinning Enhanced Visible and InfraRed Imager
(SEVIRI) from METEOSAT-8 to GDASApp end-to-end testing

new files include:
parm/atm/obs/config/satwind_seviri_m8.yaml: QC filter YAML for SEVIRI
METEOSAT-8 satwinds
parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json: JSON containing
data format, sensor, and satellite information for SEVIRI METEOSAT-8
satwinds
ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.py: bufr2ioda code for
extracting SEVIRI METEOSAT-8 satwinds from BUFR

End-to-end testing results are provided in #758 - Note: Thinning filter
is provided in YAML but turned off due to irreconcilable differences
with GSI in testing.

---------

Co-authored-by: Brett Hoover <[email protected]>

…msua

…msua

Small code-change to include SEVIRI/METEOSAT-11 satwinds in end-to-end
validation. This uses the hook-ins already provided by the
SEVIRI/METEOSAT-8 satwinds code that was merged in
#764.

New file: parm/atm/obs/config/satwind_seviri_m11.yaml - this is nearly a
direct copy of parm/atm/obs/config/satwind_seviri_m8.yaml, since they
utilize the same filters. Only the header information has changed.

Modified file: parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json -
METEOSAT-11 was added to the `satellite_info` group

End-to-end testing looks very similar to what was seen for
SEVIRI/METEOSAT-8 satwinds, detailed in
#758. A brief run-down:

(LW)IR winds are in QC agreement down to a 0.02% difference
(cloud-top)WV winds are in QC agreement down to a 0.20% difference
VIS winds differ by 4.5%, but there are only 22 accepted winds in GSI
and 23 in JEDI, the difference is a single satwind

All ob, HofX, and ob-minus-HofX differences look good

Increment summary, from gdasatmanlrun.log:
```
  0: ----------------------------------------------------------------------------------------------------
  0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-3.479765e-01 Max:+3.430433e-01 RMS:+4.990993e-04
  0: northward_wind                               | Min:-4.226316e-01 Max:+3.376561e-01 RMS:+4.889429e-04
  0: air_temperature                              | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+1.618770e-20 RMS:+1.293217e-23
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+1.474788e-19 RMS:+2.167418e-22
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ----------------------------------------------------------------------------------------------------
```

Co-authored-by: Brett Hoover <[email protected]>

…msua

In [ previous PR](#757) for
ATMS, the end-to-end testing was done without data thinning. And the
excessive QC filtering in clear-sky areas, compared to GSI, was found.
(See
[plots](#757 (comment)))

This PR includes two fixes for ATMS in the end-to-end testing and one
new feature:
1. fix the excess QC filtering in clear-sky areas. ---> This has a
paring [UFO PR #3122](JCSDA-internal/ufo#3122)
2. add data thinning 
3. add the diagnostic flags (QC) --- **This can [reproduce QC
flags](#768 (comment))
from GSI in UFO**

**To test updates in this PR, please check out the UFO branch:
[feature/satrad](https://github.com/JCSDA-internal/ufo/tree/feature/satrad)
from JCSDA-internal in gdas-validation. This branch consolidates all
proposed code changes to UFO for gdas validation:
UFO PR #JCSDA-internal/ufo#3122
UFO PR #JCSDA-internal/ufo#3121
UFO PR #JCSDA-internal/ufo#3094

---------

Co-authored-by: Cory Martin <[email protected]>

…msua

…AA-EMC/GDASApp into feature/gdas-validation_amsua

…tion_amsua_test

merge together

This PR includes the following:
1. YAML name change - add sensor (ABI) to the filename of observation
YAML for SATWIND from GOES platforms
2. YAML name change - add sensor (ASCAT) to the filename of observation
YAML for SCATWIND from MetOp platforms
3. update the YAML name changes in the list for 3dvar 
4. Add `CO2` in obs forward operator and add `linear obs operator`
without CO2 for ATMS

These changes have no impact on 3DVar results.

This PR includes the following updates for conventional surface pressure
1. Add observation YAML under configuration
2. Update observation YAML under testing
3. Add conv_ps to 3dvar obs list
4. Set window shift to true (time boundary is inclusive)
5. Validate 3dvar for surface (station) pressure (end-to-end)
- [QC
validation](#803 (comment))
-
[Increments](#803 (comment))

See [PR #792 ](#792) for the
BUFR converter work to convert the following surface pressure
observation into IODA:
- SFC - 181, 187
- SFCSHIP - 180, 183
- SONDE - 120

The related UFO evaluation (using GSI obs and geovals) results for
surface pressure are documented
[here](https://docs.google.com/presentation/d/1OvVMYOutxVyk0GE93Jzoh3hQVoj-dCgrwsVWDDlPT1s/edit?usp=sharing).