From a5c43b1034a7159a3919cd7bec7ae695c070125d Mon Sep 17 00:00:00 2001 From: Mattia Almansi Date: Tue, 4 Jun 2024 18:41:44 +0200 Subject: [PATCH] new version of the notebook with the new template --- ...emble-product_climate-monitoring_q01.ipynb | 586 ++++++++---------- 1 file changed, 251 insertions(+), 335 deletions(-) diff --git a/Satellite_ECVs/Ocean_ECVs/satellite_satellite-sea-surface-temperature-ensemble-product_climate-monitoring_q01.ipynb b/Satellite_ECVs/Ocean_ECVs/satellite_satellite-sea-surface-temperature-ensemble-product_climate-monitoring_q01.ipynb index 574c8459..9ba49e91 100644 --- a/Satellite_ECVs/Ocean_ECVs/satellite_satellite-sea-surface-temperature-ensemble-product_climate-monitoring_q01.ipynb +++ b/Satellite_ECVs/Ocean_ECVs/satellite_satellite-sea-surface-temperature-ensemble-product_climate-monitoring_q01.ipynb @@ -1,67 +1,125 @@ { "cells": [ { - "attachments": {}, "cell_type": "markdown", - "id": "ad16ba25", + "id": "239f935f-3cce-4ef2-a12f-6a64bd127010", "metadata": {}, "source": [ - "# Monitoring climate variability using satellite-derived SST\n", - "### *Does these datasets provide a reliable representation of SST climatology and its variability?*\n", - "This notebook aims at evaluating the reliability and representativeness of sea surface temperature (SST) climatology and its variability in satellite-based long-term climate data records (CDRs). \n", - "To this end, two different SST CDRs have been intercompared on a common 30-year (1982-2011) period. These datasets, available from the CADS, are the following:\n", - "- Sea surface temperature daily gridded data from 1981 to 2016 derived from a multi-product satellite-based ensemble - from the Group for High Resolution Sea Surface Temperature (GHRSST) multi-product ensemble (GMPE) produced by the European Space Agency SST Climate Change Initiative (ESA SST CCI) (GMPE in the following);\n", - "- Sea Surface Temperature daily data from 1981 to present derived from satellite observations, ESA CCI SST Level-4 dataset v2.1 (ESA CCI SST in the following);\n", - "#### **Main Takeaways - Assessment outcomes in a nutshell**\n", + "# Satellite Sea Surface Temperature intercomparison of ESACCI and GMPE for Climate Monitoring" + ] + }, + { + "cell_type": "markdown", + "id": "61a01290-1cd2-4e7e-b526-590f2697a1be", + "metadata": {}, + "source": [ + "Production date: 30-05-2024\n", + "\n", + "Produced by: CNR-ISMAR, Daniele Ciani, Andrea Pisano, Vincenzo de Toma" + ] + }, + { + "cell_type": "markdown", + "id": "eaf44d60-a59c-4266-86df-3438ba661eb2", + "metadata": {}, + "source": [ + "# Use Case: Monitoring climate variability using satellite-derived SST\n", + "## Quality assessment question\n", + "* **Does these datasets provide a reliable representation of SST climatology and its variability?**" + ] + }, + { + "cell_type": "markdown", + "id": "b149e903-5d09-4220-82e3-d370ddf399ae", + "metadata": {}, + "source": [ + "**‘Context’** This notebook aims at evaluating the reliability and representativeness of sea surface temperature (SST) climatology and its variability in satellite-based long-term climate data records (CDRs). \n", + "To this end, two different SST CDRs have been intercompared on a common 30-year (1982-2011) period. " + ] + }, + { + "cell_type": "markdown", + "id": "7b867552-f181-4d7b-945e-303dd81eacab", + "metadata": {}, + "source": [ + "## Quality Assessment Statement\n", "- Both datasets provide consistent results, well reproducing SST mean values and the main patterns of SST variability on a variety of scales, ranging from seasonal to (multi-)annual time scales;\n", "- Both datasets can be exploited for climate studies. As an example, El-Niño/La-Niña events are well characterized by both GMPE and ESA CCI SST products\n", ";\n", "- ESA SST CCI dataset should be used with caution in the first decade of the CDR. Significant discrepancies with respect to the ensemble mean GMPE dataset were indeed observed in the 1982-1992 period;\n", - "- Both datasets have been described in the literature (Martin et al. 2012, Merchant et al., 2019; Fiedler et al., 2019)." + "- Both datasets have been described in the literature ([[2]](https://doi.org/10.1016/j.rse.2018.12.015), [[3]](https://doi.org/10.1016/j.dsr2.2012.04.013), [[4]](https://doi.org/10.1038/s41597-019-0236-x))." ] }, { - "attachments": {}, "cell_type": "markdown", - "id": "73106113", + "id": "19526d81-a387-492f-82ac-f59dba07a6e6", "metadata": {}, "source": [ - "## Import packages" + "![Test image](SST_q1_thumbnail.jpg)\n", + "\n", + "*Nino 3.4 Time series calculated according to [[5]](https://climatedataguide.ucar.edu/climate-data/nino-sst-indices-nino-12-3-34-4-oni-and-tni)*" ] }, { - "cell_type": "code", - "execution_count": 1, - "id": "157d88d6-d5a1-4c4c-a879-6584e89dd1b7", - "metadata": { - "tags": [ - "hide-input" - ] - }, - "outputs": [], + "cell_type": "markdown", + "id": "7b9ef5d7-d18e-468d-9363-8b8adc54ac91", + "metadata": {}, "source": [ - "import cartopy.crs as ccrs\n", - "import matplotlib.pyplot as plt\n", - "import tqdm\n", - "import xarray as xr\n", - "from c3s_eqc_automatic_quality_control import diagnostics, download, plot, utils\n", + "# Methodology\n", + "\n", + "This Jupyter Notebook inter-compares the performances of the European Space Agency (ESA) Climate Change Initiative (CCI) SST (ESA-CCI) dataset v2.1 and the GHRSST multi product ensemble (GMPE) SST.\n", + "\n", + "Respectively, catalogue entries available from the CDS are the following:\n", + "* Sea surface temperature daily gridded data from 1981 to 2016 derived from a multi-product satellite-based ensemble - from the Group for High Resolution Sea Surface Temperature (GHRSST) multi-product ensemble (GMPE) produced by the European Space Agency SST Climate Change Initiative (ESA SST CCI) (GMPE in the following);\n", + "* Sea Surface Temperature daily data from 1981 to present derived from satellite observations, ESA CCI SST Level-4 dataset v2.1 (ESA CCI SST in the following);\n", + "\n", + "The methodology employed is suitable for investigating the representation of SST variability over climatological time scales. \n", + "\n", + "-The climatology has been defined over a common 30-year long period 1982-2011. \n", + "\n", + "-Annual and seasonal mean, as well as standard deviation, have been computed over the time axis. \n", + "\n", + "-Time series of globally averaged SST anomalies were derived by subtracting the annual cycle from the product’s monthly mean time series, representing the mean of each month over the entire period (1982-2011). \n", + "\n", + "This processing aims to minimize the impact of the seasonal component, thereby enhancing the contribution of higher and lower frequency components. \n", + "\n", + "The analysis and results are organised in the following steps, which are detailed in the sections below:' \n", "\n", - "plt.style.use(\"seaborn-v0_8-notebook\")" + "**1. [Import packages and define parameters for the requests to the CDS](#import-packages-and-define-parameters-for-the-request-to-the-cds)**\n", + "\n", + "**2. [Define functions to cache](#define-function-to-cache)**\n", + "\n", + "**3. [Download and transform step](#download-and-transform-step)**\n", + "\n", + "**4. [Plot Annual Maps, Bias, Seasonal Maps and Timeseries, with discussions](#plot-annual-maps,-bias,-seasonal-maps-and-timeseries,-with-discussion)**\n", + " * 4.1 Plot Annual Maps\n", + " * 4.2 Plot Bias maps\n", + " * 4.3 Plot Seasonal Maps\n", + " * 4.4 Plot of global monthly mean time series, Eastern Tropical Pacific SST (El Niño 3), and East Central Tropical Pacific SST (El Niño 3.4)\n" ] }, { "attachments": {}, "cell_type": "markdown", - "id": "b6e64735", + "id": "73106113", + "metadata": {}, + "source": [ + "# Analysis and results \n", + "## 1. Import packages and define parameters for the requests to the CDS" + ] + }, + { + "cell_type": "markdown", + "id": "09e1110b-861f-4b95-bafb-c809c2e3e80f", "metadata": {}, "source": [ - "## Define Parameters" + "In the following cell the necessary packages are imported, defining the request to the CDS to download ESACCI and GMPE SST from 1982 to 2011. Regions to calculate nino3 and nino3.4, and global timeseries are defined. All informations are stored in dictionaries." ] }, { "cell_type": "code", - "execution_count": 2, - "id": "81c4e187-558a-4b76-ad18-7d00a5a69d3a", + "execution_count": 1, + "id": "41770ece-5c41-4839-af2b-ca3de8f99171", "metadata": { "tags": [ "hide-input" @@ -69,6 +127,16 @@ }, "outputs": [], "source": [ + "import cartopy.crs as ccrs\n", + "import matplotlib.pyplot as plt\n", + "import tqdm\n", + "import xarray as xr\n", + "from c3s_eqc_automatic_quality_control import diagnostics, download, plot, utils\n", + "import warnings\n", + "warnings.simplefilter(\"ignore\")\n", + "\n", + "plt.style.use(\"seaborn-v0_8-notebook\")\n", + "\n", "# Time\n", "year_start = 1982\n", "year_stop = 2011\n", @@ -78,29 +146,8 @@ " \"nino3.4\": {\"lon_slice\": slice(-170, -120), \"lat_slice\": slice(-5, 5)},\n", " \"nino3\": {\"lon_slice\": slice(-150, -90), \"lat_slice\": slice(-5, 5)},\n", " \"global\": {\"lon_slice\": slice(-180, 180), \"lat_slice\": slice(-90, 90)},\n", - "}" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "id": "56a16b25", - "metadata": {}, - "source": [ - "## Define Parameters" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "326a98d3-884f-4c54-abb0-f543a2aa11f4", - "metadata": { - "tags": [ - "hide-input" - ] - }, - "outputs": [], - "source": [ + "}\n", + "\n", "# Requests\n", "request_dicts = {\n", " \"ESACCI\": {\n", @@ -141,12 +188,20 @@ "id": "5ef55fea", "metadata": {}, "source": [ - "## Functions to cache" + "## 2. Define functions to cache" + ] + }, + { + "cell_type": "markdown", + "id": "ba5c5543-3313-45f8-a08b-0aa54581e8ca", + "metadata": {}, + "source": [ + "Functions for rechunking to optimize caching of intermediate results, get nan values on land and convert to degree celsius in the ocean are defined below." ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 2, "id": "e319a354-d6b2-40fa-83e6-e9c70a631af3", "metadata": { "tags": [ @@ -209,12 +264,20 @@ "id": "59ddd578-1703-47f3-bc84-6d4d8b877a11", "metadata": {}, "source": [ - "## Download and transform" + "## 3. Download and transform step" + ] + }, + { + "cell_type": "markdown", + "id": "356cfe9b-5494-4127-854a-797320f1cb76", + "metadata": {}, + "source": [ + "Tuples and dictionaries containing infomations such as reduction operations (mean and standard deviation) are defined, and the request is set up, transforming the data while downloading data, according to the proposed diagnostics. " ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 3, "id": "6c2a4c0a-b185-44cb-9eca-26cde95da101", "metadata": { "tags": [ @@ -233,36 +296,10 @@ "name": "stderr", "output_type": "stream", "text": [ - "annual: 100%|██████████| 30/30 [00:08<00:00, 3.49it/s]\n", - "spatial reduction: 100%|██████████| 3/3 [00:00<00:00, 7.67it/s]\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "product='ESACCI' reduction='mean'\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "season: 100%|██████████| 4/4 [00:00<00:00, 7.42it/s]\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "product='ESACCI' reduction='std'\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "season: 100%|██████████| 4/4 [00:00<00:00, 5.78it/s]\n" + "annual: 100%|██████████| 30/30 [00:04<00:00, 7.35it/s]\n", + "spatial reduction: 100%|██████████| 3/3 [00:00<00:00, 11.65it/s]\n", + "season: 100%|██████████| 4/4 [00:00<00:00, 18.33it/s]\n", + "season: 100%|██████████| 4/4 [00:00<00:00, 21.79it/s]\n" ] }, { @@ -276,86 +313,10 @@ "name": "stderr", "output_type": "stream", "text": [ - "annual: 100%|██████████| 30/30 [00:07<00:00, 4.08it/s]\n", - "spatial reduction: 100%|██████████| 3/3 [00:00<00:00, 7.75it/s]\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "product='GMPE' reduction='mean'\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "season: 100%|██████████| 4/4 [00:00<00:00, 8.24it/s]\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "product='GMPE' reduction='std'\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "season: 100%|██████████| 4/4 [00:00<00:00, 6.95it/s]\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - " Size: 438kB\n", - "Dimensions: (time: 10957, product: 2, reduction: 1, region: 3)\n", - "Coordinates:\n", - " * time (time) datetime64[ns] 88kB 1982-01-01T12:00:00 ... 2011-12-...\n", - " * product (product) object 16B 'ESACCI' 'GMPE'\n", - " * reduction (reduction) object 8B 'mean'\n", - " * region (region) object 24B 'global' 'nino3' 'nino3.4'\n", - " dayofyear (time) int64 88kB dask.array\n", - "Data variables:\n", - " analysed_sst (product, reduction, region, time) float32 263kB dask.array\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n", - "/data/common/miniforge3/envs/wp5/lib/python3.11/site-packages/dask/array/numpy_compat.py:53: RuntimeWarning: invalid value encountered in divide\n", - " x = np.divide(x1, x2, out)\n" + "annual: 100%|██████████| 30/30 [00:03<00:00, 9.68it/s]\n", + "spatial reduction: 100%|██████████| 3/3 [00:00<00:00, 10.58it/s]\n", + "season: 100%|██████████| 4/4 [00:00<00:00, 20.04it/s]\n", + "season: 100%|██████████| 4/4 [00:00<00:00, 19.47it/s]\n" ] } ], @@ -417,7 +378,6 @@ " stringify_dates=True,\n", " )\n", "\n", - " #This was inside the reduction loop before\n", " reduction=\"mean\" \n", " for region, slices in tqdm.tqdm(regions.items(), desc=\"spatial reduction\"):\n", " ds = download.download_and_transform(\n", @@ -431,13 +391,9 @@ " product=[product], reduction=[reduction], region=[region]\n", " )\n", " )\n", - " #End of block moved out of the loop\n", " \n", " seasonal_ds_list = []\n", " for reduction in reductions:\n", - " # Spatial\n", - " print(f\"{product=} {reduction=}\")\n", - " \n", " # Seasonal\n", " func = getattr(diagnostics, f\"seasonal_weighted_{reduction}\")\n", " tmp_ds_list = []\n", @@ -464,8 +420,7 @@ " datasets_seasonal[product] = xr.concat(seasonal_ds_list, \"reduction\")\n", "\n", "ds_timeseries = xr.merge(timeseries_ds_list)\n", - "del tmp_ds_list, annual_ds_list, seasonal_ds_list, timeseries_ds_list\n", - "print(ds_timeseries)" + "del tmp_ds_list, annual_ds_list, seasonal_ds_list, timeseries_ds_list" ] }, { @@ -473,13 +428,21 @@ "id": "5c6bb95d-d229-441f-b44c-ad7cb5f1d699", "metadata": {}, "source": [ - "## Plot annual maps" + "## 4. Plot Annual Maps, Bias, Seasonal Maps and Timeseries, with discussions" + ] + }, + { + "cell_type": "markdown", + "id": "23d4cdb3-3614-494f-84cb-deec2e14e387", + "metadata": {}, + "source": [ + "In the following cells results of the diagnostics are shown and discussed." ] }, { "cell_type": "code", - "execution_count": 6, - "id": "f57da93f-8e26-42f2-9fc0-170ec4c67834", + "execution_count": 4, + "id": "66de59c4-da03-498d-9423-07c7b7745539", "metadata": { "tags": [ "hide-input" @@ -494,20 +457,8 @@ " ds[\"latitude\"] = ds[\"latitude\"].round(3)\n", " ds[\"longitude\"] = ds[\"longitude\"].round(3)\n", " datasets.append(ds.expand_dims(product=[product]))\n", - "ds = xr.concat(datasets, \"product\")" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "ed9f3ec7-7f76-4f61-b190-15e2ddbfec5f", - "metadata": { - "tags": [ - "hide-input" - ] - }, - "outputs": [], - "source": [ + "ds = xr.concat(datasets, \"product\")\n", + "\n", "ds_annual_low_res = ds.sel(reduction=\"mean\")\n", "datasets = []\n", "for reduction in (\"mean\", \"std\"):\n", @@ -517,53 +468,16 @@ ] }, { - "cell_type": "code", - "execution_count": 8, - "id": "4a45ab81-d6bc-40cd-a636-3ed6a028a456", - "metadata": { - "tags": [ - "hide-input" - ] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " Size: 219kB\n", - "Dimensions: (time: 10957, reduction: 1)\n", - "Coordinates:\n", - " * time (time) datetime64[ns] 88kB 1982-01-01T12:00:00 ... 2011-12-...\n", - " product Size: 219kB\n", - "Dimensions: (time: 10957, reduction: 1)\n", - "Coordinates:\n", - " * time (time) datetime64[ns] 88kB 1982-01-01T12:00:00 ... 2011-12-...\n", - " product