Merge branch 'main' into add-ocean-variable-access-live-cmoriser

ESMValGroup · Jan 29, 2025 · c8cc31a · c8cc31a
2 parents fc8b6d8 + 1e34201
commit c8cc31a
Show file tree

Hide file tree

Showing 96 changed files with 4,605 additions and 1,160 deletions.
diff --git a/.github/workflows/create-condalock-file.yml b/.github/workflows/create-condalock-file.yml
@@ -27,7 +27,7 @@ jobs:
         with:
           auto-update-conda: true
           activate-environment: esmvaltool-fromlock
-          python-version: "3.13"
+          python-version: "3.12"  # switch to 3.13 when mamba>2 available
           miniforge-version: "latest"
           use-mamba: true
       - name: Update and show conda config

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -33,13 +33,13 @@ repos:
       - id: codespell
         additional_dependencies: [tomli]  # required for Python 3.10
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: "v0.8.0"
+    rev: "v0.9.2"
     hooks:
       - id: ruff
         args: [--fix]
       - id: ruff-format
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: 'v1.13.0'
+    rev: 'v1.14.1'
     hooks:
       - id: mypy
         additional_dependencies:

diff --git a/conda-linux-64.lock b/conda-linux-64.lock
diff --git a/doc/conf.py b/doc/conf.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 #
 # ESMValTool documentation build configuration file, created by
-# sphinx-quickstart on Tue Jun  2 11:34:13 2015.
+# sphinx-quickstart on Tue Jun 2 11:34:13 2015.
 #
 # This file is execfile()d with the current directory set to its
 # containing dir.

diff --git a/doc/contributing.rst b/doc/contributing.rst
@@ -158,7 +158,7 @@ These include in particular:
   branch. If a strong objection is raised the backward-incompatible
   change should not be merged until the objection is resolved.
 - 🛠 Information required for the “*backward-incompatible changes*”
-  section in the PR  that introduces the *backward-incompatible change*
+  section in the PR that introduces the *backward-incompatible change*
   available.
 
 .. _scientific_relevance:

diff --git a/doc/quickstart/configure.rst b/doc/quickstart/configure.rst
diff --git a/doc/quickstart/find_data.rst b/doc/quickstart/find_data.rst
@@ -107,18 +107,27 @@ The following native reanalysis/observational datasets are supported under the
 To use these datasets, put the files containing the data in the directory that
 you have :ref:`configured <config_options>` for the ``rootpath`` of the
 ``native6`` project, in a subdirectory called
-``Tier{tier}/{dataset}/{version}/{frequency}/{short_name}``.
+``Tier{tier}/{dataset}/{version}/{frequency}/{short_name}`` (assuming you are
+using the ``default`` DRS for ``native6``).
 Replace the items in curly braces by the values used in the variable/dataset
 definition in the :ref:`recipe <recipe_overview>`.
-Below is a list of native reanalysis/observational datasets currently
-supported.
 
-.. _read_native_era5:
+.. _read_native_era5_nc:
 
-ERA5
-^^^^
+ERA5 (in netCDF format downloaded from the CDS)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+ERA5 data can be downloaded from the Copernicus Climate Data Store (CDS) using
+the convenient tool `era5cli <https://era5cli.readthedocs.io>`__.
+For example for monthly data, place the files in the
+``/Tier3/ERA5/version/mon/pr`` subdirectory of your ``rootpath`` that you have
+configured for the ``native6`` project (assuming you are using the ``default``
+DRS for ``native6``).
 
-- Supported variables: ``cl``, ``clt``, ``evspsbl``, ``evspsblpot``, ``mrro``, ``pr``, ``prsn``, ``ps``, ``psl``, ``ptype``, ``rls``, ``rlds``, ``rsds``, ``rsdt``, ``rss``, ``uas``, ``vas``, ``tas``, ``tasmax``, ``tasmin``, ``tdps``, ``ts``, ``tsn`` (``E1hr``/``Amon``), ``orog`` (``fx``)
+- Supported variables: ``cl``, ``clt``, ``evspsbl``, ``evspsblpot``, ``mrro``,
+  ``pr``, ``prsn``, ``ps``, ``psl``, ``ptype``, ``rls``, ``rlds``, ``rsds``,
+  ``rsdt``, ``rss``, ``uas``, ``vas``, ``tas``, ``tasmax``, ``tasmin``,
+  ``tdps``, ``ts``, ``tsn`` (``E1hr``/``Amon``), ``orog`` (``fx``).
 - Tier: 3
 
 .. note:: According to the description of Evapotranspiration and potential Evapotranspiration on the Copernicus page
@@ -131,6 +140,85 @@ ERA5
   of both liquid and solid phases to vapor (from underlying surface and vegetation)."
   Therefore, the ERA5 (and ERA5-Land) CMORizer switches the signs of ``evspsbl`` and ``evspsblpot`` to be compatible with the CMOR standard used e.g. by the CMIP models.
 
+.. _read_native_era5_grib:
+
+ERA5 (in GRIB format available on DKRZ's Levante or downloaded from the CDS)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+ERA5 data in monthly, daily, and hourly resolution is `available on Levante
+<https://docs.dkrz.de/doc/dataservices/finding_and_accessing_data/era_data/index.html#era-data>`__
+in its native GRIB format.
+
+.. note::
+  ERA5 data in its native GRIB format can also be downloaded from the
+  `Copernicus Climate Data Store (CDS)
+  <https://cds.climate.copernicus.eu/datasets>`__.
+  For example, hourly data on pressure levels is available `here
+  <https://cds.climate.copernicus.eu/datasets/reanalysis-era5-pressure-levels?tab=download>`__.
+  Reading self-downloaded ERA5 data in GRIB format is experimental and likely
+  requires additional setup from the user like setting up the proper directory
+  structure for the input files and/or creating a custom :ref:`DRS
+  <config_option_drs>`.
+
+To read these data with ESMValCore, use the :ref:`rootpath
+<config_option_rootpath>` ``/pool/data/ERA5`` with :ref:`DRS
+<config_option_drs>` ``DKRZ-ERA5-GRIB`` in your configuration, for example:
+
+.. code-block:: yaml
+
+  rootpath:
+    ...
+    native6:
+      /pool/data/ERA5: DKRZ-ERA5-GRIB
+    ...
+
+The `naming conventions
+<https://docs.dkrz.de/doc/dataservices/finding_and_accessing_data/era_data/index.html#file-and-directory-names>`__
+for input directories and files for native ERA5 data in GRIB format on Levante
+are
+
+* input directories: ``{family}/{level}/{type}/{tres}/{grib_id}``
+* input files: ``{family}{level}{typeid}_{tres}_*_{grib_id}.grb``
+
+All of these facets have reasonable defaults preconfigured in the corresponding
+:ref:`extra facets<extra_facets>` file, which is available here:
+:download:`native6-era5.yml
+</../esmvalcore/config/extra_facets/native6-era5.yml>`.
+If necessary, these facets can be overwritten in the recipe.
+
+Thus, example dataset entries could look like this:
+
+.. code-block:: yaml
+
+  datasets:
+    - {project: native6, dataset: ERA5, timerange: '2000/2001',
+       short_name: tas, mip: Amon}
+    - {project: native6, dataset: ERA5, timerange: '2000/2001',
+       short_name: cl, mip: Amon, tres: 1H, frequency: 1hr}
+    - {project: native6, dataset: ERA5, timerange: '2000/2001',
+       short_name: ta, mip: Amon, type: fc, typeid: '12'}
+
+The native ERA5 output in GRIB format is stored on a `reduced Gaussian grid
+<https://confluence.ecmwf.int/display/CKB/ERA5:+data+documentation#ERA5:datadocumentation-SpatialgridSpatialGrid>`__.
+By default, these data are regridded to a regular 0.25°x0.25° grid as
+`recommended by the ECMWF
+<https://confluence.ecmwf.int/display/CKB/ERA5%3A+What+is+the+spatial+reference#heading-Interpolation>`__
+using bilinear interpolation.
+
+To disable this, you can use the facet ``automatic_regrid: false`` in the
+recipe:
+
+.. code-block:: yaml
+
+  datasets:
+    - {project: native6, dataset: ERA5, timerange: '2000/2001',
+       short_name: tas, mip: Amon, automatic_regrid: false}
+
+- Supported variables: ``albsn``, ``cl``, ``cli``, ``clt``, ``clw``, ``hur``,
+  ``hus``, ``o3``, ``prw``, ``ps``, ``psl``, ``rainmxrat27``, ``sftlf``,
+  ``snd``, ``snowmxrat27``, ``ta``, ``tas``, ``tdps``, ``toz``, ``ts``, ``ua``,
+  ``uas``, ``va``, ``vas``, ``wap``, ``zg``.
+
 .. _read_native_mswep:
 
 MSWEP
@@ -140,7 +228,10 @@ MSWEP
 - Supported frequencies: ``mon``, ``day``, ``3hr``.
 - Tier: 3
 
-For example for monthly data, place the files in the ``/Tier3/MSWEP/version/mon/pr`` subdirectory of your ``native6`` project location.
+For example for monthly data, place the files in the
+``/Tier3/MSWEP/version/mon/pr`` subdirectory of your ``rootpath`` that you have
+configured for the ``native6`` project (assuming you are using the ``default``
+DRS for ``native6``).
 
 .. note::
   For monthly data (``V220``), the data must be postfixed with the date, i.e. rename ``global_monthly_050deg.nc`` to ``global_monthly_050deg_197901-201710.nc``
@@ -289,20 +380,23 @@ For some variables, extra facets are necessary; otherwise ESMValCore cannot
 read them properly.
 Supported keys for extra facets are:
 
-==================== ====================================== =================================
-Key                  Description                            Default value if not specified
-==================== ====================================== =================================
-``channel``          Channel in which the desired variable  No default (needs to be specified
-                     is stored                              in extra facets or recipe if
-                                                            default DRS is used)
-``postproc_flag``    Postprocessing flag of the data        ``''`` (empty string)
-``raw_name``         Variable name of the variable in the   CMOR variable name of the
-                     raw input file                         corresponding variable
-``raw_units``        Units of the variable in the raw       If specified, the value given by
-                     input file                             the ``units`` attribute in the
-                                                            raw input file; otherwise
-                                                            ``unknown``
-==================== ====================================== =================================
+===================== ====================================== =================================
+Key                   Description                            Default value if not specified
+===================== ====================================== =================================
+``channel``           Channel in which the desired variable  No default (needs to be specified
+                      is stored                              in extra facets or recipe if
+                                                             default DRS is used)
+``postproc_flag``     Postprocessing flag of the data        ``''`` (empty string)
+``raw_name``          Variable name of the variable in the   CMOR variable name of the
+                      raw input file                         corresponding variable
+``raw_units``         Units of the variable in the raw       If specified, the value given by
+                      input file                             the ``units`` attribute in the
+                                                             raw input file; otherwise
+                                                             ``unknown``
+``reset_time_bounds`` Boolean if time bounds are deleted,    ``False``
+                      and automatically recalculated by
+                      iris
+===================== ====================================== =================================
 
 .. note::
 
@@ -642,6 +736,8 @@ first discuss the ``drs`` parameter: as we've seen in the previous section, the
 DRS as a standard is used for both file naming conventions and for directory
 structures.
 
+.. _config_option_drs:
+
 Explaining ``drs: CMIP5:`` or ``drs: CMIP6:``
 ---------------------------------------------
 Whereas ESMValCore will by default use the CMOR standard for file naming (please

diff --git a/doc/quickstart/output.rst b/doc/quickstart/output.rst
@@ -150,6 +150,6 @@ Here is an example metadata.yml file:
 As you can see, this is effectively a dictionary with several items including
 data paths, metadata and other information.
 
-There are  several tools available in python which are built to read and parse
+There are several tools available in python which are built to read and parse
 these files. The tools are available in the shared directory in the diagnostics
 directory.
diff --git a/doc/recipe/preprocessor.rst b/doc/recipe/preprocessor.rst
@@ -2870,8 +2870,46 @@ Other
 
 Miscellaneous functions that do not belong to any of the other categories.
 
-Clip
-----
+.. _cumulative_sum:
+
+``cumulative_sum``
+------------------
+
+This function calculates cumulative sums along a given coordinate.
+
+The ``cumulative_sum`` preprocessor supports the following arguments in the
+recipe:
+
+* ``coord`` (:obj:`str`): Coordinate over which the cumulative sum is
+  calculated.
+  Must be 0D or 1D.
+* ``weights`` (array-like, :obj:`bool`, or ``None``, default: ``None``):
+  Weights for the calculation of the cumulative sum.
+  Each element in the data is multiplied by the corresponding weight before
+  summing.
+  Can be an array of the same shape as the input data, ``False`` or ``None``
+  (no weighting), or ``True`` (calculate the weights from the coordinate
+  bounds; only works if each coordinate point has exactly 2 bounds).
+* ``method`` (:obj:`str`, default: ``"sequential"``): Method used to perform
+  the cumulative sum.
+  Only relevant if the cube has `lazy data
+  <https://scitools-iris.readthedocs.io/en/stable/userguide/real_and_lazy_data.html>`__.
+  See :func:`dask.array.cumsum` for details.
+
+Example:
+
+.. code-block:: yaml
+
+    preprocessors:
+      preproc_cumulative_sum:
+        cumulative_sum:
+          coord: time
+          weights: true
+
+See also :func:`esmvalcore.preprocessor.cumulative_sum`.
+
+``clip``
+--------
 
 This function clips data values to a certain minimum, maximum or range. The function takes two
 arguments:
@@ -2892,7 +2930,7 @@ The example below shows how to set all values below zero to zero.
 .. _histogram:
 
 ``histogram``
--------------------
+-------------
 
 This function calculates histograms.
 

diff --git a/environment.yml b/environment.yml
@@ -40,6 +40,7 @@ dependencies:
   - python-stratify >=0.3
   - pyyaml
   - requests
+  - rich
   - scipy >=1.6
   - shapely >=2.0.0
   - yamale