Create a converter for amusa from bufr to ioda #754

xincjin-NOAA · 2023-11-20T18:31:37Z

Amsua has two NCEP bufr files which need to be converted to ioda format and merged together.

The two ncep bufr example files are gdas.t00z.esamua.tm00.bufr_d and gdas.t00z.1bamua.tm00.bufr_d.

created one json file for the converter:: bufr2ioda_ncep_amsua.json
created two yaml files for the converter: bufr2ioda_combine_ncep_esamua.yml, bufr2ioda_combine_ncep_1bamua_ta.yml
fix files are added: amsua_metop-c_v2.ACCoeff.nc [might need more]
the following python scripts are added to transfer bufr to ioda format and merge them.
- antcorr_application.py
- bufr2ioda_base.py
- bufr2ioda_combine_ncep_amsua.py
- utils.py
run_bufr2ioda.py is updated for the converter

To test the changes in this PR, we will need to checkout feature/query_python_ext for iodaconv.
https://github.com/JCSDA-internal/ioda-converters/tree/feature/query_python_ext

…te obsfile for satwnd.abi_goes-16

…s/0.35.0

Two updates for GOES AMV 1. BUFR Converter: - remove wind direction and speed - change sensorZenithAngle to satelliteZenithAngle 2. UFO Filters: - change `sensorZenithAngle` to `satelliteZenithAngle` - add `linear obs operator` section The declaration of `linear obs operator" is necessary for GOES AMV since there is no linearized component for applying the 10-meter factor. The increments look reasonable: ``` 0: ---------------------------------------------------------------------------------------------------- 0: Increment print | number of fields = 8 | cube sphere face size: C768 0: eastward_wind | Min:-1.642335e-01 Max:+1.660654e-01 RMS:+2.746685e-04 0: northward_wind | Min:-2.120068e-01 Max:+1.663539e-01 RMS:+2.631295e-04 0: air_temperature | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: surface_pressure | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: specific_humidity | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_ice | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_water | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ozone_mass_mixing_ratio | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ---------------------------------------------------------------------------------------------------- ``` The UFO filter results vs. GSI -- looks good ![ufo_abi_goes-16_omf_windEastward_qc_time1](https://github.com/NOAA-EMC/GDASApp/assets/36091766/017aeabc-d6ca-4643-b037-16f46dde76a0) ![gsi_abi_goes-16_omf_windEastward_qc_time1](https://github.com/NOAA-EMC/GDASApp/assets/36091766/d99599a2-3f36-4ef6-aa2e-4611500b8c43)

Rather than each new ob type being manually added to the list, we can use `glob.glob` to get the list of available observation types.

Two updates for MetOp SCATWIND; one minor updates for GOES AMV 1. BUFR Converter: - remove wind direction and speed - add `stationElevation` as zero 2. UFO Filters: - add linear obs operator section The declaration of `linear obs operator" is necessary for GOES AMV since there is no linearized component for applying the 10-meter factor. - add `Gaussian Thinning` filter to thin data with 75 km box (equal box size, prefer data close to central of the box) 3. For GOES AMV YAML files, change output data extension from `nc4` to `nc` UFO vs. GSI O-F and QC comparison: O-F and data count passed look close ![ufo_ascat_metop-b_omf_windEastward_qc (1)](https://github.com/NOAA-EMC/GDASApp/assets/36091766/f234543e-e0b5-4c6e-add9-3c7365c0c469) ![gsi_ascat_metop-b_omf_windEastward_qc](https://github.com/NOAA-EMC/GDASApp/assets/36091766/37f4fba8-bd2a-4a54-b45e-4055e3ef3142) 3Dvar increments are reasonable with MetOp-B SCATWIND alone. ``` 0: ---------------------------------------------------------------------------------------------------- 0: Increment print | number of fields = 8 | cube sphere face size: C768 0: eastward_wind | Min:-1.741092e+00 Max:+1.712287e+00 RMS:+2.282584e-03 0: northward_wind | Min:-1.825957e+00 Max:+1.930385e+00 RMS:+2.496075e-03 0: air_temperature | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: surface_pressure | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: specific_humidity | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_ice | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_water | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ozone_mass_mixing_ratio | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ---------------------------------------------------------------------------------------------------- ``` 3DVar increments are also reasonable with MetOp-B + MetOp-A assimilated

…oda.x (#730) This PR allows for all obtypes to be processed in two loops. We still need to figure out a way to process these in parallel (somewhat).

Using python multiprocessing to generate obs in parallel. The current list of obs goes from 10+ minutes to completing in ~5.5 minutes.

@RussTreadon-NOAA

@RussTreadon-NOAA noted that `obs linear operator` should be `linear obs operator` in the GOES-17 satwind YAML. This fixes that.

This reduces runtime further to run everything in one parallel pool. Now, things complete in under 4 minutes (as of now).

Adding satwinds from the Advanced Himawari Imager (AHI) from Himawari-8 to GDASApp end-to-end testing new files include: parm/atm/obs/config/satwind_ahi_h8.yaml: QC filter YAML for AHI Himawari-8 satwinds parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.json: JSON containing data format, sensor, and satellite information for AHI Himawari-8 satwinds ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.py: bufr2ioda code for extracting AHI Himawari-8 satwinds from BUFR modified files include: ush/ioda/bufr2ioda/run_bufr2ioda.py: added `"satwind_amv_ahi"` to list `BUFR_py` See #741 for details on testing. JEDI/GSI comparisons look good with GSI thinning turned off, but there are large ob-count disparities both before and after QC when comparing GSI+thinning to JEDI+thinning. The tested `Gaussian Thinning` filter is included in the YAML but commented out with a note. --------- Co-authored-by: Brett Hoover <[email protected]> Co-authored-by: Cory Martin <[email protected]>

merge gdas-validation

Instead of writing the filled out `.json` and `.yaml` files to `$COM_OBS`, we write them to `$DATA` so that they get purged unless `KEEPDATA="YES"`

…msua

emilyhcliu · 2024-03-19T03:47:55Z

The merge is a bit odd and tricky.
The AMSU-A yaml files under parm/atm/obs/config in your branch before merging with the develop are new yaml files except for amsua_n19. So, when you merge your branch with the develop (yaml file has j2 extension under config), you will get a conflict for amsua_n19.yaml

The amsua_n19.yaml.j2 in the develop is old. Could you confirm that the amsua_n19.yaml.j2 in the merged branch has all the updates (e.g. thinning and some changes in the filters) from your branch?
Also, it looks like you will need to manually add j2 extension for the merged amsua yaml files under config.

xincjin-NOAA · 2024-03-19T13:49:40Z

@emilyhcliu I will check those and make a test run then.

Thanks,

xincjin-NOAA · 2024-03-20T00:09:53Z

Merged the develop into this branch and tested on orion.

xincjin-NOAA · 2024-03-20T00:20:56Z

@ADCollard and @emilyhcliu , After a small modifying of QC filters on the GSI run the OmF and final ObsError difference between GSI and JEDI become smaller(the plots were updated), though there still a little differences. We also plot the maps of them and don't find any odd behaviors. We conclude that the differences are caused mainly by the difference of filters.

Not sure if we still want to make more tests to make the differences smaller.

ADCollard · 2024-03-20T14:15:08Z

@xincjin-NOAA Can you confirm that the plots have been updated (channel and maps plots)? They say they were most recently edited last week.

Having said that they do look a little better than I remember. Do you have the unthinned versions of these plots?

Can you also explain what was changed in the GSI to improve the agreement?

Thanks.

xincjin-NOAA · 2024-03-20T14:58:21Z

@ADCollard Yes I updated the plots (might be it is better just add new plots later). @emilyhcliu ran the new GSI with some changes in the filters. I am going to see if I can find the results from the unthinned runs.

xincjin-NOAA · 2024-03-20T15:22:39Z

Figure 3

**Obs comparisons between those from GSI and IODA(after converter and not thinned) **

GSI	JEDI

GSI	JEDI

Conclusion
The Obs are almost the same after convert to ioda format.

ADCollard · 2024-03-20T16:12:40Z

@xincjin-NOAA Can you plot O-B after QC? Thanks.

xincjin-NOAA · 2024-03-20T17:07:57Z

@ADCollard These are from the files directly converted from the bufr file and no QC yet in case of IODA. Is O-F the same as O-B? The O-F plots above are after QC.

ADCollard · 2024-03-20T17:18:58Z

@xincjin-NOAA Yes O-F is the same as O-B. I thought you had produced the O-B plots after QC but before thinning? Maybe @emilyhcliu did?

emilyhcliu · 2024-03-21T15:33:24Z

@xincjin-NOAA Are you now running the end-to-end with the latest T2O fix? If so, then your result from GADS is high resolution (C768). Please confirm if you are running end-to-end in C768 resolution. Check your config.base and search CASE.
Let me know which resolution you are using. I will point you to the correct IODA file from GSI.

emilyhcliu · 2024-03-21T15:51:24Z

@xincjin-NOAA I checked out your branch and will test them today.

xincjin-NOAA · 2024-03-21T18:04:41Z

Figure 4

Here are the channel statistics plots for results without thinning and with QC. ( the amsua n19 file is merged from two files)

O-F Bias/Standard Deviation

GSI	JEDI

Final ObsError

GSI	JEDI

Numbers PassedQC

GSI	JEDI

Maps for Channel 3 .

O - F

GSI	JEDI

Final ObsError

GSI	JEDI

There are no large differences found.

xincjin-NOAA · 2024-03-22T03:11:35Z

Figure 5

Here are the channel statistics plots for results without thinning and after QC for a single 1bamsua file.

O-F Bias/Standard Deviation

GSI	JEDI

Final ObsError

GSI	JEDI

Numbers PassedQC

GSI	JEDI

Maps for Channel 3 .

O - F

GSI	JEDI

Final ObsError

GSI	JEDI

They agree well with each other in general.

xincjin-NOAA · 2024-03-22T03:33:13Z

Summary

Figure 1 , the channel statistics plots (#obs, bias/std for OMF and final obs error) for JEDI and GSI runs show general agreement between GSI and JEDI results, except for the channel 3 which show some differences in O-F and Final-error. In order to understand this we explored different causes as follow:

The maps for channel 3 (Figure 2) show that there are not any place where significant differences existed.
Figure 3 demonstrated that there are no differences between GSI and JEDI after converted to IODA data format.
Figure 4, the plots from results without thinning and after QC, show no large differences
Figure 5, the plots which is drawn based on the results with a single amsua file without merging, also show no large differences, though some differences are existed here or there.

Therefore, we can conclude that the small differences come from different methods used in GSI and JEDI in terms of thinning, different QC filters in general. The difference identified in Figure 1 for channel 3 is from different thinning methods between GSI and JEDI.

xincjin-NOAA · 2024-03-22T03:50:19Z

There are two concerns left from this PR:

find a place to store the Antenna Coefficient NetCDF files
merge two .nc files is okay for now, but it is better to implement it in the ObSpace

create another issue for this: #987

emilyhcliu · 2024-03-22T03:57:39Z

@xincjin-NOAA The result for the unthinned case looks OK in general. The obs errors for low-peaking channels 1-2 have slightly lower final obs error standard deviation values. I will run end-to-end with your branch to check the initial/final them obs error and compared with GSI. I am a bit curious about the slight difference for channels 1-2. I will post the results of my test here.

One more comment:
Please add error handling in the Python converter for missing input files or missing subsets
- the need to add the error handling was identified by @RussTreadon-NOAA (see PR #937 for more infomation)
- here is the example of error handling added in the converter

emilyhcliu · 2024-03-22T04:03:39Z

There are two concerns left from this PR:

find a place for *.nc files

merge two .nc files

create another issue for this: #987

Your amsua python code can merge the 1bamua and esamua, isn't it?

For your first concern, do you mean we need to find a place to store the Antenna Coefficient NetCDF files?

xincjin-NOAA · 2024-03-22T11:34:14Z

@emilyhcliu

For the first, Yes. @CoryMartin-NOAA commented that it is better to merge the two .nc files in Obspace, though it is okay for now.

I will put details there.

xincjin-NOAA · 2024-03-25T17:22:48Z

After some experiments by turn off different combinations of filters and found that the step 10 which is the filter of Grosscheck filter might play a key role in the differences found. I am going to explore the map of them to see if we can find something.

Figure 6

Final ObsError

Before	After

xincjin-NOAA · 2024-03-27T18:28:52Z

Numbers in Gross Check

Channel	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
IODA	165	164	54	9	56	27	0	0	163	24	28	76	157	54	39
GSI	148	130	59	8	62	40	0	0	156	18	29	69	149	52	36

The numbers in channel 1, 2, are much larger than that for channel 3. This can explain the feature noticed above.

Figure 7

Map for Gross Check

GSI	JEDI

Gross filter are quite different between GSI and IODA.

xincjin-NOAA · 2024-03-29T00:48:42Z

Figure 8 Comparisons of the Initial ObsError between GSI and JEDI.

Initial ObsError

GSI	JEDI

xincjin-NOAA · 2024-03-29T00:58:44Z

Figure 9 Map comparisons of the Initial ObsError, retrieved clw, and hofx0 between GSI and JEDI (un-thinned data, Channel 1)

Initial ObsError

GSI	JEDI

Retrieved CLW

GSI	JEDI

Hofx0

GSI	JEDI

xincjin-NOAA · 2024-03-29T13:48:10Z

Figures 8 and 9 showed un-thinned statics of initial errors and related retrieved clw and hofx0.

Figure 8 show the initial errors agree well between GSI and JEDI with a little higher in GSI for channels 2 and 3 as noticed by @emilyhcliu above.
From the map in Figure 9 we can see a little difference between GSI and JEDI in the initial error and related clw and hofx0

In summary, we think these differences, (mean and std differences are 1%, and 4%, respectively) are acceptable considering there are many differences between two systems. Any comments from @emilyhcliu and @ADCollard?

xincjin-NOAA · 2024-03-29T13:51:05Z

@emilyhcliu error handling in the codes were checked and added some more.

CoryMartin-NOAA · 2024-11-01T20:35:43Z

This PR is 11 months old. Is it still relevant? Do we need to close it and open a new one later?

RussTreadon-NOAA and others added 30 commits November 9, 2023 13:48

update prototype_3d observation list to process satwind_goes-16, upda…

9285f5f

…te obsfile for satwnd.abi_goes-16

Merge develop at 957a2fc into feature/gdas-validation

9189e6f

update GDAS/orion.lua to use spack-stack-1.5.1, bufr/12.0.1, and atla…

f5320c8

…s/0.35.0

Remove a double-declaired "obs post filters" in satwnd GOES YAMLs.

8c34966

Merge branch 'develop' into feature/gdas-validation

78fc82f

Make the python bufr2ioda driver more generic (#725)

dda513f

Rather than each new ob type being manually added to the list, we can use `glob.glob` to get the list of available observation types.

Merge branch 'develop' into feature/gdas-validation

d368a2a

Have the driver script process both python scripts and YAMLs + bufr2i…

5c305aa

…oda.x (#730) This PR allows for all obtypes to be processed in two loops. We still need to figure out a way to process these in parallel (somewhat).

Use parallel processing to speed up obs processing (#733)

9fc4d73

Using python multiprocessing to generate obs in parallel. The current list of obs goes from 10+ minutes to completing in ~5.5 minutes.

temp

8f6b9c1

Fix YAML so minimization works for satwind G17 (#743)

0970ec8

@RussTreadon-NOAA noted that `obs linear operator` should be `linear obs operator` in the GOES-17 satwind YAML. This fixes that.

Consolidate parallel processing (#745)

ae22466

This reduces runtime further to run everything in one parallel pool. Now, things complete in under 4 minutes (as of now).

Merge branch 'develop' into feature/gdas-validation

59db885

amsua converter initial set

673c086

merge gdas-validation

0092f53

merge gdas-validation

forgot undating run_bufr2ioda.py

820d845

a working version

e4bc8e0

Write the bufr2ioda config files in DATA (#748)

10f01db

Instead of writing the filled out `.json` and `.yaml` files to `$COM_OBS`, we write them to `$DATA` so that they get purged unless `KEEPDATA="YES"`

remove a json file

bcfefae

Merge branch 'feature/gdas-validation' into feature/gdas-validation_a…

37409a7

…msua

correct code norm

8762fbd

a few more

2d32978

a few more

c6ebf2f

a few more

1a1d09e

a few more

d982094

dele last line

a910c82

no new line at the end of the file

56810e2

make it working after merging develop

c49d17d

xincjin-NOAA mentioned this pull request Mar 22, 2024

find a place for .nc file and need a solution to merge two .nc files #987

Open

some clean and log

ba8a1fa

xincjin-NOAA mentioned this pull request Jul 24, 2024

A new version of bufr2ioda converter for amsua using bufr_query interface #1231

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a converter for amusa from bufr to ioda #754

Create a converter for amusa from bufr to ioda #754

xincjin-NOAA commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Mar 19, 2024 •

edited

Loading

xincjin-NOAA commented Mar 19, 2024

xincjin-NOAA commented Mar 20, 2024

xincjin-NOAA commented Mar 20, 2024

ADCollard commented Mar 20, 2024

xincjin-NOAA commented Mar 20, 2024

xincjin-NOAA commented Mar 20, 2024 •

edited

Loading

ADCollard commented Mar 20, 2024

xincjin-NOAA commented Mar 20, 2024 •

edited

Loading

ADCollard commented Mar 20, 2024

emilyhcliu commented Mar 21, 2024 •

edited

Loading

emilyhcliu commented Mar 21, 2024

xincjin-NOAA commented Mar 21, 2024 •

edited

Loading

xincjin-NOAA commented Mar 22, 2024 •

edited

Loading

xincjin-NOAA commented Mar 22, 2024 •

edited

Loading

xincjin-NOAA commented Mar 22, 2024 •

edited

Loading

emilyhcliu commented Mar 22, 2024

emilyhcliu commented Mar 22, 2024

xincjin-NOAA commented Mar 22, 2024

xincjin-NOAA commented Mar 25, 2024 •

edited

Loading

xincjin-NOAA commented Mar 27, 2024 •

edited

Loading

xincjin-NOAA commented Mar 29, 2024 •

edited

Loading

xincjin-NOAA commented Mar 29, 2024 •

edited

Loading

xincjin-NOAA commented Mar 29, 2024 •

edited

Loading

xincjin-NOAA commented Mar 29, 2024

CoryMartin-NOAA commented Nov 1, 2024

Create a converter for amusa from bufr to ioda #754

Are you sure you want to change the base?

Create a converter for amusa from bufr to ioda #754

Conversation

xincjin-NOAA commented Nov 20, 2023 • edited Loading

emilyhcliu commented Mar 19, 2024 • edited Loading

xincjin-NOAA commented Mar 19, 2024

xincjin-NOAA commented Mar 20, 2024

xincjin-NOAA commented Mar 20, 2024

ADCollard commented Mar 20, 2024

xincjin-NOAA commented Mar 20, 2024

xincjin-NOAA commented Mar 20, 2024 • edited Loading

ADCollard commented Mar 20, 2024

xincjin-NOAA commented Mar 20, 2024 • edited Loading

ADCollard commented Mar 20, 2024

emilyhcliu commented Mar 21, 2024 • edited Loading

emilyhcliu commented Mar 21, 2024

xincjin-NOAA commented Mar 21, 2024 • edited Loading

xincjin-NOAA commented Mar 22, 2024 • edited Loading

xincjin-NOAA commented Mar 22, 2024 • edited Loading

xincjin-NOAA commented Mar 22, 2024 • edited Loading

emilyhcliu commented Mar 22, 2024

emilyhcliu commented Mar 22, 2024

xincjin-NOAA commented Mar 22, 2024

xincjin-NOAA commented Mar 25, 2024 • edited Loading

xincjin-NOAA commented Mar 27, 2024 • edited Loading

xincjin-NOAA commented Mar 29, 2024 • edited Loading

xincjin-NOAA commented Mar 29, 2024 • edited Loading

xincjin-NOAA commented Mar 29, 2024 • edited Loading

xincjin-NOAA commented Mar 29, 2024

CoryMartin-NOAA commented Nov 1, 2024

xincjin-NOAA commented Nov 20, 2023 •

edited

Loading

emilyhcliu commented Mar 19, 2024 •

edited

Loading

xincjin-NOAA commented Mar 20, 2024 •

edited

Loading

xincjin-NOAA commented Mar 20, 2024 •

edited

Loading

emilyhcliu commented Mar 21, 2024 •

edited

Loading

xincjin-NOAA commented Mar 21, 2024 •

edited

Loading

xincjin-NOAA commented Mar 22, 2024 •

edited

Loading

xincjin-NOAA commented Mar 22, 2024 •

edited

Loading

xincjin-NOAA commented Mar 22, 2024 •

edited

Loading

xincjin-NOAA commented Mar 25, 2024 •

edited

Loading

xincjin-NOAA commented Mar 27, 2024 •

edited

Loading

xincjin-NOAA commented Mar 29, 2024 •

edited

Loading

xincjin-NOAA commented Mar 29, 2024 •

edited

Loading

xincjin-NOAA commented Mar 29, 2024 •

edited

Loading