Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adapt ESMValTool to new configuration #3761

Merged
merged 34 commits into from
Oct 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
6685cd0
Remove obsolete config-user
schlunma Oct 2, 2024
d15683e
Fix CI test
schlunma Oct 2, 2024
6891007
Adapt doc
schlunma Oct 2, 2024
42403fa
Adapt recipe filler
schlunma Oct 2, 2024
6328ade
Adapt CMORizers
schlunma Oct 2, 2024
9e185bc
Adapt diags
schlunma Oct 2, 2024
0119432
Adapt recipes
schlunma Oct 2, 2024
1aad242
Adapt generate.py
schlunma Oct 2, 2024
d3eab26
Adapt NCL logging
schlunma Oct 2, 2024
78f8315
Undo changes in individual CMORizers
schlunma Oct 2, 2024
0aea81d
Undo changes in individual CMORizers
schlunma Oct 2, 2024
429c767
Typo
schlunma Oct 2, 2024
7b14a54
Merge branch 'main' into new_config
schlunma Oct 7, 2024
2fc5a46
Fix NCL code style
schlunma Oct 7, 2024
56db54d
Backwards-compatible CMORizer tests
schlunma Oct 7, 2024
f5a13a2
Backwards-compatible diagnostic run tests
schlunma Oct 7, 2024
c49c6a8
isort
schlunma Oct 7, 2024
8cae713
Remove print
schlunma Oct 7, 2024
0e9f378
Do not test deprecation warnings from ESMValCore
schlunma Oct 7, 2024
dfcc120
Expand environment variables when reading config dirs for CMORizer
schlunma Oct 15, 2024
697e60f
Do not use bare Exception
schlunma Oct 15, 2024
f31f9b7
Add FAQ how to read multiple config dirs
schlunma Oct 15, 2024
d67b4bc
Update esmvaltool/cmorizers/data/datasets.yml
schlunma Oct 15, 2024
cf85b57
remove "user" before configuration
schlunma Oct 15, 2024
2f842ec
Merge remote-tracking branch 'public/main' into new_config
schlunma Oct 15, 2024
51ef776
Better line breaks in doc
schlunma Oct 15, 2024
8725b41
Merge branch 'main' into new_config
schlunma Oct 15, 2024
918471c
Update doc/sphinx/source/recipes/recipe_oceans.rst
schlunma Oct 16, 2024
4db3491
Update doc/sphinx/source/recipes/recipe_shapeselect.rst
schlunma Oct 16, 2024
ce62d9f
`rootpath` -> ``rootpath``
schlunma Oct 16, 2024
75e2b33
`auxiliary_data_dir` -> ``auxiliary_data_dir``
schlunma Oct 16, 2024
c95184f
Merge remote-tracking branch 'public/main' into new_config
schlunma Oct 16, 2024
1847f2f
Merge remote-tracking branch 'public/main' into new_config
schlunma Oct 21, 2024
f0e7565
Restored original recipe_filler
schlunma Oct 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -216,8 +216,8 @@ jobs:
conda activate esmvaltool
mkdir -p ~/climate_data
esmvaltool config get_config_user
echo "search_esgf: when_missing" >> ~/.esmvaltool/config-user.yml
cat ~/.esmvaltool/config-user.yml
echo "search_esgf: when_missing" >> ~/.config/esmvaltool/config-user.yml
cat ~/.config/esmvaltool/config-user.yml
for recipe in esmvaltool/recipes/testing/recipe_*.yml; do
esmvaltool run "$recipe"
done
Expand Down
274 changes: 0 additions & 274 deletions config-user-example.yml

This file was deleted.

9 changes: 5 additions & 4 deletions doc/sphinx/source/community/dataset.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,15 @@ and run the recipe, to make sure the CMOR checks pass without warnings or errors

To test a pull request for a new CMORizer script:

#. Download the data following the instructions included in the script and place
it in the ``RAWOBS`` path specified in your ``config-user.yml``
#. Download the data following the instructions included in the script and
place it in the ``RAWOBS`` ``rootpath`` specified in your
:ref:`configuration <esmvalcore:config_options>`
#. If available, use the downloading script by running
``esmvaltool data download --config_file <config-file> <dataset>``
#. Run the cmorization by running ``esmvaltool data format <config-file> <dataset>``
#. Copy the resulting data to the ``OBS`` (for CMIP5 compliant data) or ``OBS6``
(for CMIP6 compliant data) path specified in your
``config-user.yml``
(for CMIP6 compliant data) ``rootpath`` specified in your
:ref:`configuration <esmvalcore:config_options>`
#. Run ``recipes/examples/recipe_check_obs.yml`` with the new dataset to check that
the data can be used

Expand Down
23 changes: 11 additions & 12 deletions doc/sphinx/source/community/diagnostic.rst
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ If it is just a few simple scripts or packaging is not possible (i.e. for NCL) y
and paste the source code into the ``esmvaltool/diag_scripts`` directory.

If you have existing code in a compiled language like
C, C++, or Fortran that you want to re-use, the recommended way to proceed is to add Python bindings and publish
C, C++, or Fortran that you want to reuse, the recommended way to proceed is to add Python bindings and publish
the package on PyPI so it can be installed as a Python dependency. You can then call the functions it provides
using a Python diagnostic.

Expand Down Expand Up @@ -134,9 +134,8 @@ Diagnostic output
Typically, diagnostic scripts create plots, but any other output such as e.g.
text files or tables is also possible.
Figures should be saved in the ``plot_dir``, either in both ``.pdf`` and
``.png`` format (preferred), or
respect the ``output_file_type`` specified in the
:ref:`esmvalcore:user configuration file`.
``.png`` format (preferred), or respect the :ref:`configuration option
<esmvalcore:config_options>` ``output_file_type`` .
Data should be saved in the ``work_dir``, preferably as a ``.nc``
(`NetCDF <https://www.unidata.ucar.edu/software/netcdf/>`__) file, following the
`CF-Conventions <https://cfconventions.org/>`__ as much as possible.
Expand Down Expand Up @@ -181,7 +180,7 @@ human inspection.
In addition to provenance information, a caption is also added to the plots.

Provenance information from the recipe is automatically recorded by ESMValCore, whereas
diagnostic scripts must include code specifically to record provenance. See below for
diagnostic scripts must include code specifically to record provenance. See below for
documentation of provenance attributes that can be included in a recipe.
When contributing a diagnostic, please make sure it records the provenance,
and that no warnings related to provenance are generated when running the recipe.
Expand Down Expand Up @@ -252,7 +251,7 @@ for example
plot_types:
errorbar: error bar plot

To use these items, include them in the provenance record dictionary in the form
To use these items, include them in the provenance record dictionary in the form
:code:`key: [value]`
i.e. for the example above as
:code:`'plot_types': ['errorbar']`.
Expand All @@ -275,8 +274,8 @@ Always use :func:`esmvaltool.diag_scripts.shared.run_diagnostic` at the end of y
with run_diagnostic() as config:
main(config)

Create a ``provenance_record`` for each diagnostic file (i.e. image or data
file) that the diagnostic script outputs. The ``provenance_record`` is a
Create a ``provenance_record`` for each diagnostic file (i.e. image or data
file) that the diagnostic script outputs. The ``provenance_record`` is a
dictionary of provenance items, for example:

.. code-block:: python
Expand All @@ -296,15 +295,15 @@ dictionary of provenance items, for example:
'statistics': ['mean'],
}

To save a matplotlib figure, use the convenience function
:func:`esmvaltool.diag_scripts.shared.save_figure`. Similarly, to save Iris cubes use
To save a matplotlib figure, use the convenience function
:func:`esmvaltool.diag_scripts.shared.save_figure`. Similarly, to save Iris cubes use
:func:`esmvaltool.diag_scripts.shared.save_data`. Both of these functions take
``provenance_record`` as an argument and log the provenance accordingly.
Have a look at the example Python diagnostic in
`esmvaltool/diag_scripts/examples/diagnostic.py <https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/diag_scripts/examples/diagnostic.py>`_
for a complete example.

For any other files created, you will need to make use of a
For any other files created, you will need to make use of a
:class:`esmvaltool.diag_scripts.shared.ProvenanceLogger` to log provenance. Include the
following code directly after the file is saved:

Expand Down Expand Up @@ -489,7 +488,7 @@ This includes the following items:
* In-code documentation (comments, docstrings)
* Code quality (e.g. no hardcoded pathnames)
* No Codacy errors reported
* Re-use of existing functions whenever possible
* Reuse of existing functions whenever possible
* Provenance implemented

Run recipe
Expand Down
Loading