Diagnostic and cmorization for CMUG WP5.4 LST

[valeriupredoi]

Description

This is a PR opened to streamline the process of integration of the CMUG WP5.4-LST data; work is done by @morobking and I have a mere technical support role here.

Also, @hb326 I knows you involved in the CMUG project, maybe you could have a bit of time help out/take a look/review this when ready? 🍺

• Closes CMUG WP5.4 CCI LST with Uncertainty propagation #1945
• Link to documentation:

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Code is written according to the code quality guidelines
• 🛠 Documentation is available
• 🛠 Tests run successfully
• 🛠 The list of authors is up to date
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

• 🧪 Recipe runs successfully
• 🧪 Recipe is well documented
• 🧪 Figure(s) and data look as expected from literature
• 🛠 Provenance information has been added

New or updated data reformatting script

• 🛠 Documentation is available
• 🛠 The dataset has been added to the CMOR check recipe
• 🧪 Numbers and units of the data look physically meaningful

---

To help with the number of pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[axel-lauer]

I thought it might be worth to copy some of my comments on #1945 here, so we can hopefully revive working on this pull request.

I checked out ESMValCore branch https://github.com/ESMValGroup/ESMValCore/tree/ESACCI_LST_uncerts and
ESMValTool branch https://github.com/ESMValGroup/ESMValTool/tree/CMUG-WP5.4-LST
successfully ran the cmorizer: cmorize_obs -c ./config-user.yml -o ESACCI_LST_UNCERT

There are, however, lots of "debug" messages such as "BOOO", "HELLOE!" or warning messages such as "Multiple coordinates for axis Y.". This might require another look.

I also ran recipe_esacci_lst_uncerts.yml : esmvaltool run ./ESMValTool/esmvaltool/recipes/recipe_esacci_lst_uncerts.yml --config_file ./config-user.yml

The ESMValTool preprocessor finishes successfully, but then the script lst_uncert/lst_uncert.py crashes with division by zero.

[valeriupredoi]

cheers @axel-lauer 🍺 @morobking please don't include scary messages in your code 😆 Let me know if you need any help with this 👍

[morobking]

They spook me into fixing things :)

Has something changed in how v1 to v2 uses the user config yml?
I'm getting this message when running the recipe as it stands:
esmvalcore.experimental.config._validated_config.InvalidConfigParameter: write_plots is not a valid config parameter.

[morobking]

@valeriupredoi Have you seen my question on the error?
esmvalcore.experimental.config._validated_config.InvalidConfigParameter: write_plots is not a valid config parameter.
I'm trying things but is there an exact thing to do?

[zklaus]

@morobking, this is due to ESMValGroup/ESMValCore#1300 where these settings were removed. A little bit more explanation can be found in ESMValGroup/ESMValCore#93. Basically, you have two options: Either you relocate those options to the recipe, or you remove them from the diagnostic. Most diagnostic developers remove them from the diagnostic, opting to always store both netcdf and image files.

[morobking]

@axel-lauer Thanks for the email Axel.
I have the above question about esmvalcore.cmor.check.CMORCheckError
and
related to this PR,
I want to be able to get multiple model ensemble members into the diagnostic using for example
ts:
mip: Amon
preprocessor: lst_preprocessor
additional_datasets:
- {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r(1:2)i1p1f2, start_year: 2010, end_year: 2010, grid: gn}
but I only see one instance of ts when I look at the data the diagnostic is loading.
If I use a seperate dataset section in the recipe it looks for all the variables I know arent available in the models that are only in the CCI data. Is there a solution for this?

Finally for now, I'm getting this error in a seperate piece of work
_intersect_modulus
raise ValueError(
ValueError: coordinate's range greater than coordinate's unit's modulus
Is this related to SciTools/iris#3907 where I see ESMValTool comes up that conversation?

This is coming from trying to get ERA5-Land data to load in a diagnostic. I see only the pr variable has a cmor fix, so have written something to quickly cmorizse ts and ts. That all runs fine, including util.fix_coord on time latitude and longitude. But when the recipe runs I get that Value Error when it tries to extract a region in the preprocessor.

[axel-lauer]

I want to be able to get multiple model ensemble members into the diagnostic using for example ts: mip: Amon preprocessor: lst_preprocessor additional_datasets: - {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r(1:2)i1p1f2, start_year: 2010, end_year: 2010, grid: gn} but I only see one instance of ts when I look at the data the diagnostic is loading. If I use a seperate dataset section in the recipe it looks for all the variables I know arent available in the models that are only in the CCI data. Is there a solution for this?

After updating https://github.com/ESMValGroup/ESMValCore/tree/ESACCI_LST_uncerts with the latest main I could successfully run both of these two options:

      ts:
        mip: Amon
        preprocessor: lst_preprocessor
        additional_datasets:
          - {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r1i1p1f2, start_year: 2010, end_year: 2010, grid: gn}
          - {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r2i1p1f2, start_year: 2010, end_year: 2010, grid: gn}

and

      ts:
        mip: Amon
        preprocessor: lst_preprocessor
        additional_datasets:
          - {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r(1:2)i1p1f2, start_year: 2010, end_year: 2010, grid: gn}

Both versions run successfully and produce the same plot:

If I am not mistaken, I would think from the log file [log.txt] (https://github.com/ESMValGroup/ESMValTool/files/8511364/log.txt) that both ensemble members are read by the diagnostic. So maybe simply updating ESMValCore with the latest main branch does the trick? It could also be that I do not fully understand the actual problem... In that case (which is totally possible) please provide some more instructions on how to reproduce the problem.

Finally for now, I'm getting this error in a seperate piece of work _intersect_modulus raise ValueError( ValueError: coordinate's range greater than coordinate's unit's modulus Is this related to SciTools/iris#3907 where I see ESMValTool comes up that conversation?

This is coming from trying to get ERA5-Land data to load in a diagnostic. I see only the pr variable has a cmor fix, so have written something to quickly cmorizse ts and ts. That all runs fine, including util.fix_coord on time latitude and longitude. But when the recipe runs I get that Value Error when it tries to extract a region in the preprocessor.

@morobking Could you guide me to a recipe I could use for testing? I will have a look then.

[morobking]

Thank you @axel-lauer

On the multiple ensemble question:
I have updated the branch with git pull
In the diagnostic python esmvaltool/diag_scripts/lst_uncert/lst_uncert.py
I have these lines to load the data
input_metadata = config['input_data'].values()

loaded_data = {}
ancestor_list = []
for dataset, metadata in group_metadata(input_metadata, 'dataset').items():
    cubes, ancestors = _get_input_cubes(metadata)
    loaded_data[dataset] = cubes

When I do a print(loaded_data) I get
{'UKESM1-0-LL': {'ts': <iris 'Cube' of surface_temperature / (K) (time: 12; latitude: 8; longitude: 5)>},
'ESACCI_LST_UNCERTS': {'tsLocalSfcErrDay': <iris 'Cube' of uncertainty from locally correlated errors on surface scales daytime / (K) (time: 12; latitude: 200; longitude: 200)>, 'tsLocalSfcErrNight': <iris 'Cube' of uncertainty from locally correlated errors on surface scales night time / (K) (time: 12; latitude: 200; longitude: 200)>, 'tsLocalAtmErrDay': <iris 'Cube' of uncertainty from locally correlated errors on atmospheric scales / (K) (time: 12; latitude: 200; longitude: 200)>, 'tsLocalAtmErrNight': <iris 'Cube' of uncertainty from locally correlated errors on atmospheric scales / (K) (time: 12; latitude: 200; longitude: 200)>, 'tsLSSysErrDay': <iris 'Cube' of uncertainty from large-scale systematic errors / (K) (time: 12)>, 'tsLSSysErrNight': <iris 'Cube' of uncertainty from large-scale systematic errors / (K) (time: 12)>, 'tsUnCorErrDay': <iris 'Cube' of uncertainty from uncorrelated errors / (K) (time: 12; latitude: 200; longitude: 200)>, 'tsUnCorErrNight': <iris 'Cube' of uncertainty from uncorrelated errors / (K) (time: 12; latitude: 200; longitude: 200)>}}

Here there is only one entry for UKESM data but in the recipe
ts:
mip: Amon
preprocessor: lst_preprocessor
additional_datasets:
- {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r1i1p1f2, start_year: 2010, end_year: 2010, grid: gn}
- {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r2i1p1f2, start_year: 2010, end_year: 2010, grid: gn}

The same output occurs for print(loaded_data) when I use the r(1:2)i1p1f2 syntax or other CMIP models in the recipe when defining the ts variable.

My question is how to get the different ensemble members available in the python code of the diagnostic?

I'll keep the other 2 questions seperate.

[axel-lauer]

On the multiple ensemble question: I have updated the branch with git pull In the diagnostic python esmvaltool/diag_scripts/lst_uncert/lst_uncert.py I have these lines to load the data input_metadata = config['input_data'].values()

loaded_data = {}
ancestor_list = []
for dataset, metadata in group_metadata(input_metadata, 'dataset').items():
    cubes, ancestors = _get_input_cubes(metadata)
    loaded_data[dataset] = cubes

Here there is only one entry for UKESM data but in the recipe ts: mip: Amon preprocessor: lst_preprocessor additional_datasets: - {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r1i1p1f2, start_year: 2010, end_year: 2010, grid: gn} - {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r2i1p1f2, start_year: 2010, end_year: 2010, grid: gn}

The same output occurs for print(loaded_data) when I use the r(1:2)i1p1f2 syntax or other CMIP models in the recipe when defining the ts variable.

My question is how to get the different ensemble members available in the python code of the diagnostic?

Thanks for the additional explanation, I think now I get it. I believe the problem is in function _get_input_cubes in lst_uncert.py and not in the recipe or preprocessor. As the dictionary keys used in _get_input_cubes for each dataset are simply the variable name (short_name), entries for different ensemble members are overwritten for UKESM1-0-LL. I am not exactly sure what is supposed to happen later in the code, but an option might be to use a list of cubes (instead of a dictionary) and add new items with append() or altenatively, use a more unique key for the dictionary than just short_name, possibly containing the ensemble member if present.

[morobking]

@axel-lauer Thank you, that solves the first problem 👍

My second question is around the new tsDay and tsNight vatiables I have had to make from the CCI LST data.
When I run the recipe with them included I get this error:
File "/home/users/robking/ESMValCore/esmvalcore/cmor/check.py", line 235, in report_errors
raise CMORCheckError(msg)
esmvalcore.cmor.check.CMORCheckError: There were errors in variable ts:
ts: does not match coordinate rank
in cube:
surface_temperature / (kelvin) (time: 1; latitude: 3600; longitude: 7200)
Dimension coordinates:
time x - -
latitude - x -
longitude - - x
Attributes:
Conventions CF-1.7
source_file /work/scratch-nopw/morobking/OBS/Tier2/ESACCI_LST_UNCERTS/OBS_ESACCI_L...
var tsDay

I do not get this for the other new CCI LST uncertainty variables, like tsLSErrDay etc...
Is there something to be looking for to fix, or something I can do in the recipe?

The third question concerns a different branch, so I'll start a conversation there about that soon.

[axel-lauer]

My second question is around the new tsDay and tsNight vatiables I have had to make from the CCI LST data. When I run the recipe with them included I get this error: File "/home/users/robking/ESMValCore/esmvalcore/cmor/check.py", line 235, in report_errors raise CMORCheckError(msg) esmvalcore.cmor.check.CMORCheckError: There were errors in variable ts: ts: does not match coordinate rank in cube: surface_temperature / (kelvin) (time: 1; latitude: 3600; longitude: 7200) Dimension coordinates: time x - - latitude - x - longitude - - x Attributes: Conventions CF-1.7 source_file /work/scratch-nopw/morobking/OBS/Tier2/ESACCI_LST_UNCERTS/OBS_ESACCI_L... var tsDay

I do not get this for the other new CCI LST uncertainty variables, like tsLSErrDay etc... Is there something to be looking for to fix, or something I can do in the recipe?

I think there are some possible reasons for this problem. Here is what I would do:

ESMValTool:

1. Change the variable name in the cmorized files from "lst" to "tsDay" / "tsNight" and rerun the cmorizer.

ESMValCore:

2. Make sure that the dimensions in the custom cmor tables CMOR_tsDay.dat and CMOR_tsNight.dat are correct. At the moment, the tables say "dimensions: time". I believe this should rather be something like "dimensions: longitude latitude time". Btw. maybe also check if the ordering of the dimensions in your other custom CMOR_tables (e.g. CMOR_tsLocalSfcErr.dat") is correct.

I think the most important thing to check is 2), item 1) is probably rather a cosmetic problem.

[morobking]

@axel-lauer Thank you - your point 2 is the answer! Not sure I would have found that, nor sure how I missed that when originally setting up the cmor tables.

My third question is about a something on a different branch, I'll set up a draft PR there.

[MargaridaS]

Hello @axel-lauer

I contact you because I found another dataset with the same error ValueError: coordinate's range greater than coordinate's unit's modulus than the one obtained by @morobking:

Finally for now, I'm getting this error in a seperate piece of work _intersect_modulus raise ValueError( ValueError: coordinate's range greater than coordinate's unit's modulus Is this related to SciTools/iris#3907 where I see ESMValTool comes up that conversation?

This is coming from trying to get ERA5-Land data to load in a diagnostic. I see only the pr variable has a cmor fix, so have written something to quickly cmorizse ts and ts. That all runs fine, including util.fix_coord on time latitude and longitude. But when the recipe runs I get that Value Error when it tries to extract a region in the preprocessor.

The dataset for my case is Ghrsstostia-v2.0, launched with the recipe:
OBS_ostia_2007_extract_region.txt and with the file: tos_200701.zip. Is this a bug? How could it be solved?

Thank you very much in advance.

Kind regards,

Marga

[axel-lauer]

@MargaridaS I am sorry, but I cannot remember the details. I believe it was @valeriupredoi and @zklaus helping. Can one of you remember the problem with the coordinates?
I briefly checked your file tos_200701.nc and I saw the longitudes are running form -180 to 180. I believe CMOR standard would be 0 to 360 instead. Not sure if that causes the problem but it might be worth checking.

[MargaridaS]

@MargaridaS I am sorry, but I cannot remember the details. I believe it was @valeriupredoi and @zklaus helping. Can one of you remember the problem with the coordinates? I briefly checked your file tos_200701.nc and I saw the longitudes are running form -180 to 180. I believe CMOR standard would be 0 to 360 instead. Not sure if that causes the problem but it might be worth checking.

Hi @axel-lauer

Thank you for your prompt answer. I changed longitudes to 0 to 360 but the same error appears. I see a bug was reported to iris some time ago: SciTools/iris#3907 . Could this be a type of variables error?

Best regards,

Marga

[MargaridaS]

Hi @axel-lauer

Setting manually the first bound to zero and the last bound to 360 solved the issue.
Thank you very much for your attention.

Best,
Marga