Merge pull request #168 from cerfacs-globc/fix/slice_mode_on_seasons

Fix slice_mode for seasons

.github/CONTRIBUTING.rst

@@ -13,7 +13,7 @@ Contents:
 
 
 Getting started
----------------
+===============
 
 Let's first see how to get icclim up and running.
 icclim sources are stored in `github <https://github.com/cerfacs-globc/icclim>`_.
@@ -51,7 +51,7 @@ You can now:
     See `numpy's guide <https://numpy.org/doc/stable/dev/development_workflow.html#writing-the-commit-message>`_ for additional keywords.
 
 Documentation contribution
---------------------------
+==========================
 
 One of the most important aspect of any open source project is its documentation.
 It's both the entry point for most new users and the gallery of the project.
@@ -83,7 +83,7 @@ By default it will serve the documentation on ``localhost://8000``.
 
 
 Add new standard indices
-------------------------
+========================
 
 Existing index in xclim
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -142,7 +142,7 @@ This process is described in section `Existing index in xclim`_ above.
 
 
 Add new operators for user indices
-----------------------------------
+==================================
 
 icclim provides a convenient way to quickly write simple index, we call this features "user indices".
 A few operators are already available but if you think icclim could benefit from adding new ones, your contribution is most welcomed.
@@ -168,7 +168,7 @@ For the unit tests, you can add them in ``tests/unit_tests/test_user_indices.py`
 For the documentation, you should add an explanation of the operator behavior in section ``user_index`` of ``doc/references/icclim_index_api.rst``.
 
 Improve icclim API
-------------------
+==================
 
 icclim features pre-processing and post-processing steps over the index computation:
 This includes:
@@ -194,7 +194,7 @@ For post-processing steps:
 
 
 Other contributions
--------------------
+===================
 
 If you would like to see any other change in icclim not listed here, you can always open an issue on
 `icclim's github <https://github.com/cerfacs-globc/icclim>`_ and we will work with you on how to implement it.

README.rst

@@ -63,7 +63,7 @@ For a detailed description of each ECA&D index, please visit: https://www.ecad.e
 ..
   Pytest Coverage Comment:Begin
 
-.. |coverage| image:: https://img.shields.io/badge/Coverage-90%25-brightgreen.svg
+.. |coverage| image:: https://img.shields.io/badge/Coverage-92%25-brightgreen.svg
         :target: https://github.com/cerfacs-globc/icclim/blob/master/README.rst#code-coverage
         :alt: Code coverage
 

doc/source/dev/ci.rst

@@ -1,16 +1,15 @@
 Continuous integration
 ======================
 
-icclim continuous integration aims to assist development by:
-- Avoiding introducing bugs in the code base .
-- Ensuring all new code follow the same code style.
-- Measuring how much icclim code base is tested by automated unit tests. This is known as code coverage.
-- Making sure the documentation generation is functioning well.
+icclim continuous integration (CI) aims to assist development by:
+    - Avoiding introducing bugs in the code base.
+    - Ensuring all new code follow the same code style.
+    - Measuring how much icclim code base is tested by automated unit tests. This is known as code coverage.
+    - Making sure the documentation generation is functioning well.
 
 These goals are reached using multiple tools:
-- pre-commit CI enforce the code style (Black + flake8 + isort) is followed by
-committing changes directly on new pull request and blocking merge if necessary.
-The relevant file is `.pre-commit-config.yaml`.
-- readthedocs, which serve our documentation is also configured to run the documentation generation on
-each new pull request.
-- github actions are used to run unit tests and report the results in each pull request.
+    - pre-commit CI enforce the code style (Black + flake8 + isort) is followed by
+      committing changes directly on new pull request and blocking merge if necessary.
+      The relevant file is `.pre-commit-config.yaml`.
+    - readthedocs, which serve our documentation is also configured to run the documentation generation on each new pull request.
+    - github actions are used to run unit tests and report the results in each pull request.

doc/source/dev/release_process.rst

@@ -31,11 +31,11 @@ Release process
 
         python3 -m pip install --index-url https://test.pypi.org/simple/ icclim
 
-.. note::
+    .. note::
 
-    It may fail due to missing dependencies in test.pypi.
-    In that case, create the environment from icclim environment.yml file to
-    pull all needed dependencies from conda.
+        It may fail due to missing dependencies in test.pypi.
+        In that case, create the environment from icclim environment.yml file to
+        pull all needed dependencies from conda.
 
 #. Upload to pypi for real.
 

doc/source/references/release_notes.rst

@@ -1,8 +1,17 @@
 Release history
 ===============
 
-5.2.1-dev
----------
+5.2.1
+-----
+[maint] Made Frequency part of SliceMode union.
+[fix] slice_mode seasonal samplings was giving wrong results for quite a few indices. This has been fixed and the performances should also be improved by the fix.
+However, now seasonal slice_mode does not allow to use xclim missing values mechanisms.
+[fix] user_index ExtremeMode config was not properly parsed when a string was used.
+[fix] user_index Anomaly operator was not properly using the `ref_time_range` to setup a reference period as it should.
+[fix] user_index Sum and Mean operators were broken due to a previous refactoring and a lack of unit tests, it is now fixed and tested.
+[maint] Changed how `rechunker` dependency is pinned to add flexibility. We want a version above '0.3' but not the '0.4'.
+[maint] For the newly generate API, on `custom_index` function, the parameter `user_index` is now mandatory.
+
 
 5.2.0
 -----

environment.yml

@@ -13,7 +13,7 @@ dependencies:
  - netCDF4>=1.5.7
  - cftime>=1.5.0
  - pyyaml>=6.0
- - rechunker>=0.5
+ - rechunker>=0.3.3
  - psutil
  - zarr
  - fsspec

icclim/__init__.py

@@ -3,4 +3,4 @@
 from .main import index, indice, indices  # noqa
 from .pre_processing.rechunk import create_optimized_zarr_store  # noqa
 
-__version__ = "5.2.0"
+__version__ = "5.2.1"

icclim/_generated_api.py

@@ -4012,6 +4012,7 @@ def ww(
 
 
 def custom_index(
+    user_index: UserIndexDict,
     in_files: str | list[str] | Dataset | DataArray,
     var_name: str | list[str] | None = None,
     slice_mode: SliceMode = Frequency.YEAR,
@@ -4022,7 +4023,6 @@ def custom_index(
     ignore_Feb29th: bool = False,
     out_unit: str | None = None,
     netcdf_version: str | NetcdfVersion = NetcdfVersion.NETCDF4,
-    user_index: UserIndexDict = None,
     save_percentile: bool = False,
     logs_verbosity: Verbosity | str = Verbosity.LOW,
 ) -> Dataset:
@@ -4074,10 +4074,6 @@ def custom_index(
         ``optional`` Output unit for certain indices: "days" or "%" (default: "days").
     netcdf_version : str | icclim.models.netcdf_version.NetcdfVersion
         ``optional`` NetCDF version to create (default: "NETCDF3_CLASSIC").
-    user_index : UserIndexDict
-        ``optional`` A dictionary with parameters for user defined index.
-        See :ref:`Custom indices`.
-        Ignored for ECA&D indices.
     save_percentile : bool
         ``optional`` True if the percentiles should be saved within the resulting netcdf
          file (default: False).
@@ -4090,6 +4086,7 @@ def custom_index(
     This function has been auto-generated.
     """
     return icclim.index(
+        user_index=user_index,
         in_files=in_files,
         var_name=var_name,
         slice_mode=slice_mode,
@@ -4100,7 +4097,6 @@ def custom_index(
         ignore_Feb29th=ignore_Feb29th,
         out_unit=out_unit,
         netcdf_version=netcdf_version,
-        user_index=user_index,
         save_percentile=save_percentile,
         logs_verbosity=logs_verbosity,
     )

icclim/main.py

@@ -28,7 +28,7 @@
 from icclim.models.user_index_config import UserIndexConfig
 from icclim.models.user_index_dict import UserIndexDict
 from icclim.pre_processing.input_parsing import read_dataset, update_to_standard_coords
-from icclim.user_indices.dispatcher import compute_user_index
+from icclim.user_indices.calc_operation import CalcOperation, compute_user_index
 
 log: IcclimLogger = IcclimLogger.get_instance(Verbosity.LOW)
 
@@ -221,7 +221,7 @@ def index(
         DEPRECATED, use user_index instead.
 
     """
-    _setup(callback, callback_percentage_start_value, logs_verbosity)
+    _setup(callback, callback_percentage_start_value, logs_verbosity, slice_mode)
     index_name, user_index = _handle_deprecated_params(
         index_name, indice_name, transfer_limit_Mbytes, user_index, user_indice
     )
@@ -312,14 +312,19 @@ def _handle_deprecated_params(
     return index_name, user_index
 
 
-def _setup(callback, callback_percentage_start_value, logs_verbosity):
+def _setup(callback, callback_start_value, logs_verbosity, slice_mode):
     # make xclim input daily check a warning instead of an error
+    # TODO: it might be safer to feed a context manager which will setup
+    #       and teardown these confs
     xclim.set_options(data_validation="warn")
+    if Frequency.is_seasonal(slice_mode):
+        # for now seasonal slice_modes missing values cannot be checked
+        xclim.set_options(check_missing="skip")
     # keep attributes through xarray operations
     xr.set_options(keep_attrs=True)
     log.set_verbosity(logs_verbosity)
     log.start_message()
-    callback(callback_percentage_start_value)
+    callback(callback_start_value)
 
 
 def _compute_ecad_index_dataset(
@@ -359,12 +364,13 @@ def _compute_user_index_dataset(
     )
     user_indice_da = compute_user_index(user_indice_config)
     user_indice_da.attrs["units"] = _get_unit(config.out_unit, user_indice_da)
-    if config.freq.post_processing is not None:
-        user_indice_da, time_bounds = config.freq.post_processing(user_indice_da)
-        result_ds[user_indice_config.index_name] = user_indice_da
-        result_ds["time_bounds"] = time_bounds
-    else:
+    if user_indice_config.calc_operation is CalcOperation.ANOMALY:
+        # with anomaly time axis disappear
         result_ds[user_indice_config.index_name] = user_indice_da
+        return result_ds
+    user_indice_da, time_bounds = config.freq.post_processing(user_indice_da)
+    result_ds[user_indice_config.index_name] = user_indice_da
+    result_ds["time_bounds"] = time_bounds
     return result_ds
 
 

icclim/models/constants.py

@@ -26,3 +26,14 @@
 MODIFIABLE_UNIT = "MODIFIABLE_UNIT"  # fields: out_unit
 MODIFIABLE_THRESHOLD = "MODIFIABLE_THRESHOLD"  # fields: threshold
 MODIFIABLE_QUANTILE_WINDOW = "MODIFIABLE_QUANTILE_WINDOW"  # fields: window_width
+
+# Map of months index to their short name, used to get a pandas frequency anchor
+MONTHS_MAP = {1:"JAN",  2:"FEB", 3:"MAR", 4:"APR", 5:"MAY", 6:"JUN", 7:"JUL", 8:"AUG", 9:"SEP", 10:"OCT", 11:"NOV", 12:"DEC" }
+
+# Season defined by their month numbers
+AMJJAS_MONTHS = [*range(4, 9)]
+ONDJFM_MONTHS = [10, 11, 12, 1, 2, 3]
+DJF_MONTHS = [12, 1, 2]
+MAM_MONTHS = [*range(3, 6)]
+JJA_MONTHS = [*range(6, 9)]
+SON_MONTHS = [*range(9, 12)]