CHANGELOG.rst

Changelog

v0.11.0 - unreleased

New Features

• Integrated MESMER-M into the code base, see Integration of MESMER-M.
• Added number of observations to the output of the AR process (#395). By Victoria Bauer.
• Add python 3.13 to list of supported versions (#547). By Mathias Hauser.

Breaking changes

• Switch random number generation for drawing emulations from np.random.seed() to np.random.default_rng() (#495). By Victoria Bauer.

• Using Cholesky decomposition for finding covariance localization radius and drawing from the multivariate normal distribution (#408) By Victoria Bauer.

• Removed support for python 3.9 (#513) By Mathias Hauser.

• The supported versions of some dependencies were changed (#399, #405, and #503):

  Package	Old	New
  cartopy	not specified	0.22
  dask	not specified	2023.8
  joblib	not specified	1.3
  netcdf4	not specified	1.6
  numpy	not specified	1.24
  packaging	not specified	23.1
  pandas	2.0	no change
  pooch	not specified	1.7
  properscoring	not specified	0.1
  pyproj	not specified	3.6
  regionmask	0.8	0.10
  scikit-learn	not specified	1.3
  scipy	not specified	1.11
  shapely	not specified	2.0
  statsmodels	not specified	0.14
  xarray	2023.04	2023.7

Deprecations

• Deprecated mask_3D_frac_approx as the functionality is now offered in regionmask v0.12.0 (#451).

Bug fixes

• Averaging standard deviations for the AR parameters of global variability over several ensemble members and scenarios now averages the variances (#499). By Victoria Bauer.

Documentation

• Updated and extended the development Guide (#511, #523)

Internal Changes

• Start testing the minimum versions of required dependencies (#398). By Mathias Hauser.
• Restore compatibility with pandas version v2.2 and xarray version v2024.02 (#404). By Mathias Hauser.
• Explicitly include all required dependencies (#448).
• Unshallow the mesmer git repository on rtd (#456). By Victoria Bauer.
• Use ruff instead of isort and flake8 to lint the code base (#490). By Mathias Hauser.

Integration of MESMER-X

In the release the MESMER-X functionality is integrated into the MESMER Codebase.

• Add MESMER-X functionality to the code base (#432)
• Some general refactoring and clean-up (#437, #465, #466, #467, #468, #469, #470, #502)

• Add unit tests (#526,

  #533, #534, #540)

• Add integration tests (#524,

  #550 #553)

Integration of MESMER-M

This release integrates MESMER-M into the existing MESMER infrastructure. This includes some refactoring, bugfixes and enhancements of the MESMER-M functionality. Note that this led to some numerical changes compared to the MESMER-M publication (Nath et al., 2022).

• move MESMER-M scripts into mesmer ( #419, and #421).
• move the harmonic model and power transformer functionalities to the stats module ( #484).
• add example script for MESMER-M workflow (#491)
• add integration tests for MESMER-M (#501)

Auto-Regression

• Implement functions performing the monthly (cyclo-stationary) auto-regression and adapt these functions to work with xarray. This includes extracting the drawing of spatially correlated innovations to a stand-alone function. (#473)
• Remove the bounds of -1 and 1 on the slope of the cyclo-stationary AR(1) process. This bound is not necessary since cyclo-stationarity is also given if the slopes of a few months are (slightly) larger than one. We now return the residuals of the cyclo-stationary AR(1) process to fit the covariance matrix on these residuals. As a consequence, adjustment of the covariance matrix with the AR slopes is no longer necessary. After this, no adjustment is necessary anymore. (#480) Compare discussion in #472.
• Implement function to localize the empirical covarince matrix for each month individually to use in drawing of spatially correlated noise in the AR process. (#479)

Yeo-Johnson power transformer

• Ensure the power transformer yields the correct normalization for more cases ( #440):
  • expand the upper bound of the first coefficient from 1 to \infty, i.e. to 1e10 ( #446, #501)
  • remove jacobian rosen_der from fit ( #447)
  • change optimization method from SLSQP to Nelder-Mead ( #455)
• adjust the first guess to assume the data is normally distributed ( #429)
• make (back-) transformations more stable by using np.expm1 and np.log1p (#494)
• rewrite power transformer to work with xarray, and refactor from a class structure to functions ( #442, and #474)
• fix small code issues and clean the docstrings ( #436, #444, #439, #475, and #425)
• add tests (#430)

Harmonic model

• Performance and other optimizations:
  • only fit orders up to local minimum and use coeffs from precious order as first guess (#443)
  • infer the harmonic model order from the coefficients (#434)
• return residuals instead of the loss for the optimization (#460)
• remove fitting of linear regression with yearly temperature (#415 <https://github.com/MESMER-group/mesmer/pull/415/>_ and #488) in line with (Nath et al. 2022).
• add helper function to upsample yearly data to monthly resolution ( #418, and #435)
• de-duplicate the expression of months in their harmonic form (#415) move creation of the month array to the deepest level (#487).
• fix indexing of harmonic model coefficients (#415)
• Refactor variable names, small code improvements, optimization, fixes and clean docstring (#415, #424, #433, #512, and #574).
• add tests ( #431, and #458)
• add function to generate fourier series using xarray (#478)

By Victoria Bauer and Mathias Hauser.

v0.10.0 - 2024.01.04

version 0.10.0 fixes the bug in the legacy calibration and is thus not numerically backward compatible. It also updated the supported python, pandas and xarray versions. Updating the pandas version will create an error when trying to load pickled mesmer bundles, requiring to use mesmer version v0.9.0 for this.

Bug fixes

Ensure de-duplicating the historical ensemble members conserves their order. Previously, the legacy calibration routines used np.unique, which shuffles them. See #338 for details. (#339). By Mathias Hauser.

Breaking changes

• Removed support for python 3.7 and python 3.8 ( #163, #365, #367, and #371). By Mathias Hauser.

• The supported versions of some dependencies were changed (#369):

  Package	Old	New
  pandas	<2.0	>=2.0
  xarray	not specified	>=2023.04

New Features

• Add python 3.12 to list of supported versions (#368). By Mathias Hauser.

v0.9.0 - 2023.12.19

version 0.9.0 is a big step towards rewriting mesmer. All statistical functionality was extracted and works for xarray data objects. It also contains data handling functions to prepare climate model data using xarray.

• The restructured code is fully functional and can be used to calibrate and emulate temperature. However, it is still missing wrappers which encapsulate the full chain and helpers to simplify calibrating several scenarios and ensemble members.
• This version still contains the legacy routines to train and emulate temperature. It should have no numerical changes, only minimal changes in usage, and offers speed gains over v0.8.3.

Known bugs

For the legacy training, the influence of the global variability is underestimated, because the historical ensemble members are shuffled "randomly". This is kept in v0.9.0 for backward compatibility and will be fixed in a follow-up bug fix release. For details see #338.

New Features

• Extracted statistical functionality for linear regression:

  • Create :py:class:`mesmer.stats.LinearRegression` which encapsulates fit, predict, etc. methods around linear regression (#134). By Mathias Hauser.
  • Add xarray wrapper for fitting a linear regression ( #123 and #142). By Mathias Hauser.
  • Add add fit_intercept argument to the linear_regression fitting methods and functions (#144). By Mathias Hauser.
  • Allow to pass 1-dimensional targets to :py:meth:`mesmer.stats.LinearRegression.fit` (#221). By Mathias Hauser.
  • Allow to exclude predictor variables in :py:meth:`mesmer.stats.LinearRegression.predict` (#354). By Mathias Hauser.
  • Fixed two bugs related to (non-dimension) coordinates ( #332, #333 and #334). By Mathias Hauser.

• Extracted statistical functionality for auto regression:

  • Add mesmer.stats.fit_auto_regression: xarray wrapper to fit an auto regression model (#139). By Mathias Hauser.
  • Have mesmer.stats.fit_auto_regression return the variance instead of the standard deviation ( #306, and #318). By Mathias Hauser.
  • Add draw_auto_regression_correlated and draw_auto_regression_uncorrelated: to draw samples of a (spatially-)correlated and uncorrelated auto regression model ( #322, #161 and #313). By Mathias Hauser.
  • Add mesmer.stats.select_ar_order to select the order of an auto regressive model (#176). By Mathias Hauser.

• Extracted functions dealing with the spatial covariance and its localization:

  • Add xarray wrappers :py:func:`mesmer.stats.adjust_covariance_ar1` and :py:func:`mesmer.stats.find_localized_empirical_covariance` (#191). By Mathias Hauser.
  • Refactor and extract numpy-based functions dealing with the spatial covariance and its localization (#167 and #184). By Mathias Hauser.
  • Allow to pass 1 x n arrays to :py:func:`mesmer.stats.adjust_covariance_ar1` (#224). By Mathias Hauser.

• Update LOWESS smoothing:

  • Extract the LOWESS smoothing for xarray objects: :py:func:`mesmer.stats.lowess`. (#193, #283, and #285). By Mathias Hauser.
  • Allow to pool data along a dimension to estimate the LOWESS smoothing. (#331). By Mathias Hauser.

• Added helper functions to process xarray-based model data:

  • Added functions to stack regular lat-lon grids to 1D grids and unstack them again (#217). By Mathias Hauser.
  • Added functions to mask the ocean and Antarctica ( #219 and #314). By Mathias Hauser.
  • Added functions to calculate the weighted global mean (#220 and #287). By Mathias Hauser.
  • Added functions to wrap arrays to [-180, 180) and [0, 360), respectively (#270 and #273). By Mathias Hauser.

• The aerosol data is now automatically downloaded using pooch. (#267). By Mathias Hauser.

• Added helper functions to estimate and superimpose volcanic influence (#336). By Mathias Hauser.

• Added additional tests for the calibration step (#209):

  • one scenario (SSP5-8.5) and two ensemble members (#211)
  • two scenarios (SSP1-2.6 and SSP5-8.5) with one and two ensemble members, respectively (#214)
  • different selection of predictor variables (tas**2 and hfds) for different scenarios (#291)

  By Mathias Hauser.

• Allow passing xr.DataArray to gaspari_cohn (#298). By Mathias Hauser.

• Allow passing xr.DataArray to geodist_exact (#299). By Zeb Nicholls and Mathias Hauser.

• Add calc_gaspari_cohn_correlation_matrices a function to calculate Gaspari-Cohn correlation matrices for a range of localisation radii (#300). By Zeb Nicholls and Mathias Hauser.

• Add a helper function to load tas and (potentially) hfds for several ESMs from cmip-ng archive at ETHZ (#326). By Mathias Hauser.

Breaking changes

• Localization radii that lead to singular matrices are now skipped (#187). By Mathias Hauser.
• Refactor and split :py:func:`train_l_prepare_X_y_wgteq` into two functions: :py:func:`get_scenario_weights` and :py:func:`stack_predictors_and_targets` (#143). By Mathias Hauser.
• Moved gaspari_cohn & calc_geodist_exact from io.load_constant_files to core.computation (#158). By Yann Quilcaille.
• The function mask_percentage has been renamed to :py:func:`core.regionmaskcompat.mask_3D_frac_approx` (#202). By Mathias Hauser.
• Removed :py:func:`mesmer.io.load_constant_files.infer_interval_breaks` and the edges from the lat and lon dictionaries i.e., lon["e"] and lat["e"] (#233). By Mathias Hauser.
• Deprecated the reg_type argument to :py:func:`mesmer.io.load_constant_files.load_regs_ls_wgt_lon_lat` and the reg_dict argument to :py:func:`mesmer.utils.select.extract_land`. These arguments no longer have any affect (#235). By Mathias Hauser.
• Removed ref["type"] == "first", i.e., calculating the anomaly w.r.t. the first ensemble member (#247). By Mathias Hauser.
• Renamed mesmer.calibrate_mesmer._calibrate_and_draw_realisations to mesmer.calibrate_mesmer._calibrate_tas (#66). By Mathias Hauser.

Deprecations

• The function mesmer.create_emulations.create_emus_gt has been renamed to :py:func:`create_emulations.gather_gt_data` (#246). By Mathias Hauser.
• The function mesmer.utils.select.extract_time_period is now deprecated and will be removed in a future version. Please raise an issue if you use this function (#243). By Mathias Hauser.

Bug fixes

• Fix three issues with :py:func:`core.regionmaskcompat.mask_3D_frac_approx`. Note that these issues are only relevant if passing xarray objects and/ or masks close to the poles (#202 and #218). By Mathias Hauser.

Documentation

• Add development/contributing docs (#121). By Zeb Nicholls.

Internal Changes

• Refactor the mesmer internals to use the new statistical core, employ helper functions etc.:

  • Use :py:func:`mesmer.utils.separate_hist_future` in :py:func:`mesmer.calibrate_mesmer.train_gt` (#281).
  • Use of :py:class:`mesmer.stats.LinearRegression` in
    • :py:func:`mesmer.calibrate_mesmer.train_gt_ic_OLSVOLC` (#145).
    • :py:func:`mesmer.create_emulations.create_emus_lv_OLS` and :py:func:`mesmer.create_emulations.create_emus_OLS_each_gp_sep` (#240).

  By Mathias Hauser.

• Restore compatibility with regionmask v0.9.0 (#136). By Mathias Hauser.

• Renamed the interpolation keyword of np.quantile to method changed in numpy v1.22.0 (#137). By Mathias Hauser.

• Add python 3.10 and python 3.11 to list of supported versions (#162 and #284). By Mathias Hauser.

• Move contents of setup.py to setup.cfg (#169). By Mathias Hauser.

• Use pyproject.toml for the build-system and setuptools_scm for the __version__ (#188). By Mathias Hauser.

• Moved the climate model data manipulation functions (#237). By Mathias Hauser.

v0.8.3 - 2021-12-23

New Features

• Add mesmer.stats._linear_regression (renamed to mesmer.stats._fit_linear_regression_np in #142). Starts the process of refactoring the codebase (#116). By Zeb Nicholls.

Bug fixes

• Initialize llh_max to -inf to ensure the cross validation loop is entered (#110). By Jonas Schwaab.

Documentation

• Fix copyright notice and release version in documentation (#127). By Zeb Nicholls.

Internal Changes

• Automatically upload the code coverage to codecov.io after the test suite has run (#99). By Mathias Hauser.
• Internal refactor: moved a number of inline comments to their own line (especially if this allows to have the code on one line instead of several) and other minor cleanups (#98). By Mathias Hauser.
• Refactor io.load_cmipng_tas and io.load_cmipng_hfds to de-duplicate their code and add tests for them (#55). By Mathias Hauser.

v0.8.2 - 2021-10-07

Bug fixes

• Reintroduce ability to read in cmip5 data from the cmip5-ng archive at ETH (#90). By Lea Beusch.

Internal Changes

• Reproduce the test files because of a change in regionmask which affected the mesmer tests (#95). By Mathias Hauser.
• Refactor and speed up of the Gaspari-Cohn function and the calculation of the great circle distance (#85, #88). By Mathias Hauser.
• The geopy package is no longer a dependency of mesmer (#88). By Mathias Hauser.
• Convert README from Markdown to reStructuredText to fix package build errors. Also allows to include the README in the docs to avoid duplication (#102). By Mathias Hauser.

v0.8.1 - 2021-07-15

• Update example script (#80).

v0.8.0 - 2021-07-13

• First release on PyPI and conda (#79).