From f99d3a8b2d8a42650efdc7e6df2777cb60311b38 Mon Sep 17 00:00:00 2001 From: Mattia Almansi Date: Mon, 3 Jun 2024 10:04:07 +0200 Subject: [PATCH] some updates to EOBS --- ...ations-europe_climate-monitoring_q01.ipynb | 1204 +++++++++++++++-- 1 file changed, 1075 insertions(+), 129 deletions(-) diff --git a/In_Situ/insitu_insitu-gridded-observations-europe_climate-monitoring_q01.ipynb b/In_Situ/insitu_insitu-gridded-observations-europe_climate-monitoring_q01.ipynb index b7855ed1..a36e35e4 100644 --- a/In_Situ/insitu_insitu-gridded-observations-europe_climate-monitoring_q01.ipynb +++ b/In_Situ/insitu_insitu-gridded-observations-europe_climate-monitoring_q01.ipynb @@ -9,8 +9,8 @@ } }, "source": [ - "# Use Case: Monitoring climate change on the Iberian Peninsula\n", - "## User Question: How consistent are surface air temperature climatologies from reanalysis and observations over time? How does this depend on the choice of reference period?\n", + "# Use Case: Assessment of Climate Change.\n", + "## User Question: How consistent is the 30-year air temperature climatology over time? How does this depend on the choice of reference period?\n", "\n", "### About\n", "\n", @@ -18,32 +18,33 @@ "\n", "**This Use Case aims to address the following User Questions:**\n", "\n", - "1) How consistent is the 30-year air temperature climatology over time, in Europe?\n", + "1) How consistent is the 30-year air temperature climatology over time? \n", "\n", "2) How does this depend on the choice of reference period?\n", "\n", "**This notebook comprises the following steps:**\n", "1. Define the AoI, search and download E-OBS\n", "2. Inspect and view data\n", - "3. Calculate the climatology for alternative 30-year periods\n", - "4. Calculate the probability density function of each alternative 30-year period\n", - "5. Calculate and plot climatological season maps\n", - "6. Calculate seasonal probability density function of each alternative 30-year period\n", + "3. Calculate the climatology for alternative 30-year periods, and the ensemble spread\n", + "4. Calculate the probability density Function (PDF) of each alternative 30-year period, and the ensemble spread\n", + "5. Calculate and plot seasonal PDFs and the climatological season maps, and the ensemble spread\n", + "6. Calculate seasonal probability density function of each alternative 30-year period, and the ensemble spread\n", "7. Main takeaways\n", + "8. References\n", "\n", - "**The main takeaways of this Use Case are:**\n", + "**The Quality Statements of this Use Case are:**\n", "\n", - "- There is a visible difference between air temperature climatologies from different periods, as the earliest 30-years subset (1951-1980) shows an overall “colder” annual mean air temperature, while the latest 30-years subset (1991-2020) shows the “warmest” one, over the Iberian Peninsula.\n", + "- Mean daily air temperature (TG) from EOBS offers complete temporal-spatial coverage over the AoI, showing a climate change-driven warming rate consistent with those reported in the literature.\n", "\n", - "- In addition, these changes are progressive along the time dimension, i.e., each climatology reveals an increased mean annual temperature, compared to the 10-years previous one. \n", + "- Furthermore, the observed asymmetry in the TG probability density function underscores the importance of assessing changes across the entire distribution, rather than focusing solely on mean values.\n", "\n", - "- These deviations aren’t the same across all of the statistical distribution, i.e., there are greater upwards deviations in the upper values of the mean annual air temperature, when compared with the lowest ones. The Probability Density Function (PDF) graphs thus show that there is a shift towards the right but also changes in the simmetry with larger increase of warmer temperatures.\n", + "- Nevertheless, TG ensemble spread shows inhomogeneous spatial patterns that arise from the irregular density of in-situ stations available, across the region. Indeed, this inhomogeneity indicates that EOBS accuracy is location-dependent. \n", "\n", - "- These changes are also visible at seasonal level, with less variance on the Winter Season PDF (DJF) when compared with the Summer Season PDF (JJA).\n", + "- As described in this dataset's documentation, the usage of this dataset should be with caution in interpreting trends and extremes, considering the potential limitations in accuracy and completeness, especially in regions with sparse observational data (Cornes et al., 2018).\n", "\n", - "- Furthermore, there is an evident spatial pattern regarding the changes observed on the Climatological Mean Maps, with significant differences between the south and westernmost parts of the Iberia Peninsula (warmer), compared to the interior and northernmost regions (colder). \n", + "- Furthermore, potential biases should be considered from these techniques and consider the spatial variability in data density when interpreting results. For instance, it may also be influenced by factors such as changes in observational practices, station location shifts, or urbanization effects, which can introduce uncertainties in the interpretation of long-term trends.\n", "\n", - "- Whenever using climatological data for their application, the user should carefully consider that the results are sensible to the chosen time of interest. This is especially true to applications which are highly sensitive to air temperature differences (e.g., estimation of energy needs, analysis of extreme events based on percentiles).\n", + "- As an indicator of the confidence intervals associated with individual grid cells, the ensemble spread may provide insights into the reliability of data in specific locations, aiding in the assessment of uncertainties (Carvalho, Cardoso Pereira, and Rocha, 2021).\n", "\n" ] }, @@ -82,23 +83,9 @@ "hide-input" ] }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: cdsapi in /home/anolive/miniconda3/envs/C3S_BOPEN/lib/python3.10/site-packages (0.6.1)\n", - "Requirement already satisfied: requests>=2.5.0 in /home/anolive/miniconda3/envs/C3S_BOPEN/lib/python3.10/site-packages (from cdsapi) (2.31.0)\n", - "Requirement already satisfied: tqdm in /home/anolive/miniconda3/envs/C3S_BOPEN/lib/python3.10/site-packages (from cdsapi) (4.65.0)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in /home/anolive/miniconda3/envs/C3S_BOPEN/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi) (3.1.0)\n", - "Requirement already satisfied: idna<4,>=2.5 in /home/anolive/miniconda3/envs/C3S_BOPEN/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi) (3.4)\n", - "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/anolive/miniconda3/envs/C3S_BOPEN/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi) (2.0.3)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /home/anolive/miniconda3/envs/C3S_BOPEN/lib/python3.10/site-packages (from requests>=2.5.0->cdsapi) (2023.5.7)\n" - ] - } - ], + "outputs": [], "source": [ - "!pip install cdsapi" + "# !pip install cdsapi" ] }, { @@ -143,15 +130,15 @@ "\n", "E-OBS comprises a set of spatially continuous Essential Climate Variables (ECVs) from the Surface Atmosphere, following the Global Climate Observing System (GCOS) convention, provided as the mean and spread of the spatial prediction ensemble algorithm, at regular latitude-longitude grid intervals (at a 0.1° and 0.25° spatial resolution), and covering a long time-period, from 1 January 1950 to present-day. In addition to the land surface elevation, E-OBS includes daily air temperature (mean, maximum and minimum), precipitation amount, wind speed, sea-level pressure and shortwave downwelling radiation. \n", "\n", - "The latest E-OBS version, **E-OBSv25.0e**, was released in April 2022 and its main difference from the previous E-OBSv24.0e is the amount of input data used.\n", + "The latest E-OBS version, **E-OBSv29.0e**, was released in March 2024 and its main difference from the previous versions is the amount of input data used. Existing networks of synoptic, climatological and (manual) rain-gauge station data are updated with the latest measurements that are received directly from the national or regional meteorological services. \n", "\n", - "Having selected the correct dataset, we now need to specify what product type, variables, temporal and geographic coverage we are interested in. In this Use Case, the ensemble mean of air temperature (TG) will be used, considering the last version available. These can all be selected in the “Download data” tab from the CDS. In this tab a form appears in which we will select the following parameters to download, for example:\n", + "Having selected the correct dataset, we now need to specify what product type, variables, temporal and geographic coverage we are interested in. In this Use Case, the ensemble mean of air temperature (TG) will be used, considering the last version available. In addition, the ensemble spread of air temperature shall also be used as an indicator of data quality. These can all be selected in the “Download data” tab from the CDS. In this tab a form appears in which we will select the following parameters to download, for example:\n", "\n", - "* **Product Type:** Ensemble mean \n", - "* **Variable:** Mean temperature \n", + "* **Product Type:** Ensemble mean (Ensemble spread)\n", + "* **Variable:** Mean temperature\n", "* **Grid resolution:** 0.25 \n", "* **Period:** Full period \n", - "* **Version:** 25.0e \n", + "* **Version:** 29.0e \n", "* **Format:** Zip file (.zip)\n", "\n", "At the end of the download form, select “**Show API request**”. This will reveal a block of code, which you can simply copy and paste into a cell of your Jupyter Notebook …\n", @@ -165,6 +152,7 @@ "execution_count": 3, "id": "f7e88a56", "metadata": { + "metadata": {}, "tags": [ "hide-input" ] @@ -181,7 +169,9 @@ "colors = [\"deepskyblue\", \"green\", \"gold\", \"darkorange\", \"red\"]\n", "\n", "# Variable name\n", - "varname = \"tg\"" + "varname1 = \"tg\"\n", + "\n", + "varname2 = \"tg_spread\"" ] }, { @@ -239,7 +229,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████| 1/1 [00:01<00:00, 1.04s/it]\n" + "100%|██████████| 1/1 [00:00<00:00, 1.46it/s]\n" ] } ], @@ -253,7 +243,7 @@ " \"variable\": \"mean_temperature\",\n", " \"grid_resolution\": \"0.25deg\",\n", " \"period\": \"full_period\",\n", - " \"version\": \"25.0e\",\n", + " \"version\": \"29.0e\",\n", " },\n", ")\n", "ds = download.download_and_transform(\n", @@ -263,6 +253,44 @@ ")" ] }, + { + "cell_type": "code", + "execution_count": 6, + "id": "8db063b4", + "metadata": { + "tags": [ + "hide-input" + ] + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 1/1 [00:00<00:00, 15.27it/s]\n" + ] + } + ], + "source": [ + "# Define request\n", + "request = (\n", + " \"insitu-gridded-observations-europe\",\n", + " {\n", + " \"format\": \"zip\",\n", + " \"product_type\": \"ensemble_spread\",\n", + " \"variable\": \"mean_temperature\",\n", + " \"grid_resolution\": \"0.25deg\",\n", + " \"period\": \"full_period\",\n", + " \"version\": \"29.0e\",\n", + " },\n", + ")\n", + "dss = download.download_and_transform(\n", + " *request,\n", + " transform_func=regionalise_and_dayofyear_reindex,\n", + " transform_func_kwargs={\"lon_slice\": lon_slice, \"lat_slice\": lat_slice},\n", + ")" + ] + }, { "cell_type": "markdown", "id": "a360c081", @@ -280,13 +308,620 @@ "source": [ "### Basic statistics\n", "\n", - "To understand better the E-OBS data structure and check the aggregated Daily Mean Temperature (TG), we will first need to retrieve the TG variable from the multidimensional netCDF data structure and calculate the descriptive statistics." + "To understand better the E-OBS data structure and check the aggregated Daily Mean Temperature (TG), we will first need to retrieve the TG variable from the multidimensional netCDF data structure and calculate the descriptive statistics." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "46abc22e", + "metadata": { + "tags": [ + "hide-input" + ] + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
<xarray.Dataset>\n",
+       "Dimensions:    (latitude: 32, longitude: 44, dayofyear: 366, period: 5)\n",
+       "Coordinates:\n",
+       "  * latitude   (latitude) float64 36.12 36.38 36.62 36.88 ... 43.38 43.62 43.88\n",
+       "  * longitude  (longitude) float64 -9.875 -9.625 -9.375 ... 0.375 0.625 0.875\n",
+       "  * dayofyear  (dayofyear) int64 1 2 3 4 5 6 7 8 ... 360 361 362 363 364 365 366\n",
+       "  * period     (period) object '1951-1980' '1961-1990' ... '1991-2020'\n",
+       "    season     (dayofyear) object dask.array<chunksize=(366,), meta=np.ndarray>\n",
+       "Data variables:\n",
+       "    TG         (period, dayofyear, latitude, longitude) float32 dask.array<chunksize=(5, 366, 32, 44), meta=np.ndarray>\n",
+       "Attributes:\n",
+       "    E-OBS_version:  29.0e\n",
+       "    Conventions:    CF-1.4\n",
+       "    References:     http://surfobs.climate.copernicus.eu/dataaccess/access_eo...\n",
+       "    history:        Fri Mar 22 10:08:04 2024: ncks --no-abc -d time,0,27027 /...\n",
+       "    NCO:            netCDF Operators version 5.1.4 (Homepage = http://nco.sf....
" + ], + "text/plain": [ + "\n", + "Dimensions: (latitude: 32, longitude: 44, dayofyear: 366, period: 5)\n", + "Coordinates:\n", + " * latitude (latitude) float64 36.12 36.38 36.62 36.88 ... 43.38 43.62 43.88\n", + " * longitude (longitude) float64 -9.875 -9.625 -9.375 ... 0.375 0.625 0.875\n", + " * dayofyear (dayofyear) int64 1 2 3 4 5 6 7 8 ... 360 361 362 363 364 365 366\n", + " * period (period) object '1951-1980' '1961-1990' ... '1991-2020'\n", + " season (dayofyear) object dask.array\n", + "Data variables:\n", + " TG (period, dayofyear, latitude, longitude) float32 dask.array\n", + "Attributes:\n", + " E-OBS_version: 29.0e\n", + " Conventions: CF-1.4\n", + " References: http://surfobs.climate.copernicus.eu/dataaccess/access_eo...\n", + " history: Fri Mar 22 10:08:04 2024: ncks --no-abc -d time,0,27027 /...\n", + " NCO: netCDF Operators version 5.1.4 (Homepage = http://nco.sf...." + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Specify the old variable name\n", + "old_variable_name = 'tg'\n", + "\n", + "# Specify the new variable name, long name and units\n", + "new_variable_name = 'TG'\n", + "new_long_name = 'Daily Mean Temperature'\n", + "new_units = '°C'\n", + "\n", + "# Specify the new variable units>\n", + "\n", + "# Change the variable name and long name\n", + "ds = ds.rename({old_variable_name: new_variable_name})\n", + "ds[new_variable_name].attrs['long_name'] = new_long_name\n", + "ds[new_variable_name].attrs['units'] = new_units\n", + "\n", + "ds" ] }, { "cell_type": "code", - "execution_count": 6, - "id": "cb24491a", + "execution_count": 9, + "id": "3da4ae0b", "metadata": { "tags": [ "hide-input" @@ -660,29 +1295,29 @@ " fill: currentColor;\n", "}\n", "
<xarray.Dataset>\n",
-       "Dimensions:    (longitude: 44, latitude: 32, dayofyear: 366, period: 5)\n",
+       "Dimensions:    (latitude: 32, longitude: 44, dayofyear: 366, period: 5)\n",
        "Coordinates:\n",
-       "  * longitude  (longitude) float64 -9.875 -9.625 -9.375 ... 0.375 0.625 0.875\n",
        "  * latitude   (latitude) float64 36.12 36.38 36.62 36.88 ... 43.38 43.62 43.88\n",
+       "  * longitude  (longitude) float64 -9.875 -9.625 -9.375 ... 0.375 0.625 0.875\n",
        "  * dayofyear  (dayofyear) int64 1 2 3 4 5 6 7 8 ... 360 361 362 363 364 365 366\n",
        "  * period     (period) object '1951-1980' '1961-1990' ... '1991-2020'\n",
        "    season     (dayofyear) object dask.array<chunksize=(366,), meta=np.ndarray>\n",
        "Data variables:\n",
-       "    tg         (period, dayofyear, latitude, longitude) float32 dask.array<chunksize=(5, 366, 32, 44), meta=np.ndarray>\n",
+       "    TG Spread  (period, dayofyear, latitude, longitude) float32 dask.array<chunksize=(5, 366, 32, 44), meta=np.ndarray>\n",
        "Attributes:\n",
-       "    E-OBS_version:  25.0e\n",
+       "    E-OBS_version:  29.0e\n",
        "    Conventions:    CF-1.4\n",
        "    References:     http://surfobs.climate.copernicus.eu/dataaccess/access_eo...\n",
-       "    history:        Mon Mar 28 14:53:05 2022: ncks --no-abc -d time,0,26297 /...\n",
-       "    NCO:            netCDF Operators version 4.7.5 (Homepage = http://nco.sf....