…te obsfile for satwnd.abi_goes-16

…s/0.35.0

Two updates for GOES AMV

1. BUFR Converter:
- remove wind direction and speed
- change sensorZenithAngle to satelliteZenithAngle
2. UFO Filters:
- change `sensorZenithAngle` to `satelliteZenithAngle`
- add `linear obs operator` section
The declaration of `linear obs operator" is necessary for GOES AMV since
there is no linearized component for applying the 10-meter factor.

The increments look reasonable:
```
0: ----------------------------------------------------------------------------------------------------
  0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-1.642335e-01 Max:+1.660654e-01 RMS:+2.746685e-04
  0: northward_wind                               | Min:-2.120068e-01 Max:+1.663539e-01 RMS:+2.631295e-04
  0: air_temperature                              | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ----------------------------------------------------------------------------------------------------
```

The UFO filter results vs. GSI  -- looks good


![ufo_abi_goes-16_omf_windEastward_qc_time1](https://github.com/NOAA-EMC/GDASApp/assets/36091766/017aeabc-d6ca-4643-b037-16f46dde76a0)


![gsi_abi_goes-16_omf_windEastward_qc_time1](https://github.com/NOAA-EMC/GDASApp/assets/36091766/d99599a2-3f36-4ef6-aa2e-4611500b8c43)

Rather than each new ob type being manually added to the list, we can
use `glob.glob` to get the list of available observation types.

)

Two updates for MetOp SCATWIND; one minor updates for GOES AMV 

1. BUFR Converter:
    - remove wind direction and speed
    - add `stationElevation` as zero 

2. UFO Filters:
    - add linear obs operator section
The declaration of `linear obs operator" is necessary for GOES AMV since
there is no
      linearized component for applying the 10-meter factor.
- add `Gaussian Thinning` filter to thin data with 75 km box (equal box
size, prefer data close to central of the box)

3. For GOES AMV YAML files, change output data extension from `nc4` to
`nc`

UFO vs. GSI O-F and QC comparison: O-F and data count passed look close
![ufo_ascat_metop-b_omf_windEastward_qc
(1)](https://github.com/NOAA-EMC/GDASApp/assets/36091766/f234543e-e0b5-4c6e-add9-3c7365c0c469)

![gsi_ascat_metop-b_omf_windEastward_qc](https://github.com/NOAA-EMC/GDASApp/assets/36091766/37f4fba8-bd2a-4a54-b45e-4055e3ef3142)

3Dvar increments are reasonable with MetOp-B SCATWIND alone.
```
0: ----------------------------------------------------------------------------------------------------
  0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-1.741092e+00 Max:+1.712287e+00 RMS:+2.282584e-03
  0: northward_wind                               | Min:-1.825957e+00 Max:+1.930385e+00 RMS:+2.496075e-03
  0: air_temperature                              | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ----------------------------------------------------------------------------------------------------

```

3DVar increments are also reasonable with MetOp-B + MetOp-A assimilated

…oda.x (NOAA-EMC#730)

This PR allows for all obtypes to be processed in two loops.

We still need to figure out a way to process these in parallel
(somewhat).

Using python multiprocessing to generate obs in parallel. The current
list of obs goes from 10+ minutes to completing in ~5.5 minutes.

@RussTreadon-NOAA noted that `obs linear operator` should be `linear obs
operator` in the GOES-17 satwind YAML. This fixes that.

This reduces runtime further to run everything in one parallel pool.
Now, things complete in under 4 minutes (as of now).

…A-EMC#746)

Adding satwinds from the Advanced Himawari Imager (AHI) from Himawari-8
to GDASApp end-to-end testing

new files include:
parm/atm/obs/config/satwind_ahi_h8.yaml: QC filter YAML for AHI
Himawari-8 satwinds
parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.json: JSON containing data
format, sensor, and satellite information for AHI Himawari-8 satwinds
ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.py: bufr2ioda code for
extracting AHI Himawari-8 satwinds from BUFR

modified files include:
ush/ioda/bufr2ioda/run_bufr2ioda.py: added `"satwind_amv_ahi"` to list
`BUFR_py`

See NOAA-EMC#741 for details on testing. JEDI/GSI comparisons look good with GSI
thinning turned off, but there are large ob-count disparities both
before and after QC when comparing GSI+thinning to JEDI+thinning. The
tested `Gaussian Thinning` filter is included in the YAML but commented
out with a note.

---------

Co-authored-by: Brett Hoover <[email protected]>
Co-authored-by: Cory Martin <[email protected]>

Instead of writing the filled out `.json` and `.yaml` files to
`$COM_OBS`, we write them to `$DATA` so that they get purged unless
`KEEPDATA="YES"`

…#753)

Updating GNSSRO converter and adding the yaml file to GDASApp end-to-end
testing based on [NOAA-EMC#712](NOAA-EMC#712).

New file:
parm/atm/obs/config/gnssro.yaml: Added QC filters in the YAML

Updated files:
parm/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.json: Updated the satellite
information
ush/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.py: Updated pccf, qfro, and
satelliteAscending flag

Refer to [issue NOAA-EMC#750](NOAA-EMC#750)
for testing details. JEDI/GSI comparisons revealed a cutoff near
obs=0.03 Rad due to super refraction 2. The number of observations after
QC in JEDI is also inconsistent with GSI.

List of items to be added in this PR:
- BUFR Converter: Add YAML for converting ATMS BUFR to IODA
   - The ObsValue is antenna temperature
   - Input BUFR is normal feel 

- Testing YAML: re-evaluation results due to update CRTM from v2.3.0 to
v2.4.1-jedi.1
   - atms_npp_noqc.yaml --- 100% replication
   - atms_npp.yaml  --- 100 % replication
   - atms_n20_noqc.yaml --- 100 % replication
- atms_n20.yaml --- ~ 1100% replication: two channel 7 observations
difference
  
- Config YAML: 
* add two read routine QC: data thinning and removal of data from scan
edge
   - atms_npp.yaml
   - atms_n20.yaml   

- Paring PR in JCSDA UFO repository
- This PR also has a [paring PR in JCSDA UFO
repository](JCSDA-internal/ufo#3094).
   - It is OK to merge this one into GDASApp before the paring UFO PR
- The related test data (obs and geoval files from GSI) for UFO
Evaluation have been updated (due to CRTM-2.4.1 update) on ORION and
HERA.


Notes:
- End-to-end testing completed without problem. 
- However, the filter results show one of the filters, 88-165 GHz
scattering index check, tossed more data compared to GSI.
- The excess screening from the 88-165 GHz scattering index check is
under investigation and will be reported separately.
- Please check comments below for validation results

…o-end testing (NOAA-EMC#764)

Adding satwinds from the Spinning Enhanced Visible and InfraRed Imager
(SEVIRI) from METEOSAT-8 to GDASApp end-to-end testing

new files include:
parm/atm/obs/config/satwind_seviri_m8.yaml: QC filter YAML for SEVIRI
METEOSAT-8 satwinds
parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json: JSON containing
data format, sensor, and satellite information for SEVIRI METEOSAT-8
satwinds
ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.py: bufr2ioda code for
extracting SEVIRI METEOSAT-8 satwinds from BUFR

End-to-end testing results are provided in NOAA-EMC#758 - Note: Thinning filter
is provided in YAML but turned off due to irreconcilable differences
with GSI in testing.

---------

Co-authored-by: Brett Hoover <[email protected]>

)

Small code-change to include SEVIRI/METEOSAT-11 satwinds in end-to-end
validation. This uses the hook-ins already provided by the
SEVIRI/METEOSAT-8 satwinds code that was merged in
NOAA-EMC#764.

New file: parm/atm/obs/config/satwind_seviri_m11.yaml - this is nearly a
direct copy of parm/atm/obs/config/satwind_seviri_m8.yaml, since they
utilize the same filters. Only the header information has changed.

Modified file: parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json -
METEOSAT-11 was added to the `satellite_info` group

End-to-end testing looks very similar to what was seen for
SEVIRI/METEOSAT-8 satwinds, detailed in
NOAA-EMC#758. A brief run-down:

(LW)IR winds are in QC agreement down to a 0.02% difference
(cloud-top)WV winds are in QC agreement down to a 0.20% difference
VIS winds differ by 4.5%, but there are only 22 accepted winds in GSI
and 23 in JEDI, the difference is a single satwind

All ob, HofX, and ob-minus-HofX differences look good

Increment summary, from gdasatmanlrun.log:
```
  0: ----------------------------------------------------------------------------------------------------
  0: Increment print | number of fields = 8 | cube sphere face size: C768
  0: eastward_wind                                | Min:-3.479765e-01 Max:+3.430433e-01 RMS:+4.990993e-04
  0: northward_wind                               | Min:-4.226316e-01 Max:+3.376561e-01 RMS:+4.889429e-04
  0: air_temperature                              | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: surface_pressure                             | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: specific_humidity                            | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: cloud_liquid_ice                             | Min:+0.000000e+00 Max:+1.618770e-20 RMS:+1.293217e-23
  0: cloud_liquid_water                           | Min:+0.000000e+00 Max:+1.474788e-19 RMS:+2.167418e-22
  0: ozone_mass_mixing_ratio                      | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
  0: ----------------------------------------------------------------------------------------------------
```

Co-authored-by: Brett Hoover <[email protected]>

In [ previous PR](NOAA-EMC#757) for
ATMS, the end-to-end testing was done without data thinning. And the
excessive QC filtering in clear-sky areas, compared to GSI, was found.
(See
[plots](NOAA-EMC#757 (comment)))

This PR includes two fixes for ATMS in the end-to-end testing and one
new feature:
1. fix the excess QC filtering in clear-sky areas. ---> This has a
paring [UFO PR #3122](JCSDA-internal/ufo#3122)
2. add data thinning 
3. add the diagnostic flags (QC) --- **This can [reproduce QC
flags](NOAA-EMC#768 (comment))
from GSI in UFO**

**To test updates in this PR, please check out the UFO branch:
[feature/satrad](https://github.com/JCSDA-internal/ufo/tree/feature/satrad)
from JCSDA-internal in gdas-validation. This branch consolidates all
proposed code changes to UFO for gdas validation:
UFO PR #JCSDA-internal/ufo#3122
UFO PR #JCSDA-internal/ufo#3121
UFO PR #JCSDA-internal/ufo#3094

---------

Co-authored-by: Cory Martin <[email protected]>

…MC#789)

This PR includes the following:
1. YAML name change - add sensor (ABI) to the filename of observation
YAML for SATWIND from GOES platforms
2. YAML name change - add sensor (ASCAT) to the filename of observation
YAML for SCATWIND from MetOp platforms
3. update the YAML name changes in the list for 3dvar 
4. Add `CO2` in obs forward operator and add `linear obs operator`
without CO2 for ATMS

These changes have no impact on 3DVar results.

)

This PR includes the following updates for conventional surface pressure
1. Add observation YAML under configuration
2. Update observation YAML under testing
3. Add conv_ps to 3dvar obs list
4. Set window shift to true (time boundary is inclusive)
5. Validate 3dvar for surface (station) pressure (end-to-end)
- [QC
validation](NOAA-EMC#803 (comment))
-
[Increments](NOAA-EMC#803 (comment))

See [PR NOAA-EMC#792 ](NOAA-EMC#792) for the
BUFR converter work to convert the following surface pressure
observation into IODA:
- SFC - 181, 187
- SFCSHIP - 180, 183
- SONDE - 120

The related UFO evaluation (using GSI obs and geovals) results for
surface pressure are documented
[here](https://docs.google.com/presentation/d/1OvVMYOutxVyk0GE93Jzoh3hQVoj-dCgrwsVWDDlPT1s/edit?usp=sharing).