From 3838464a7f664665a7ea5a0413fe75f8404c01ba Mon Sep 17 00:00:00 2001 From: RuthBowyer Date: Mon, 18 Nov 2024 15:40:53 +0000 Subject: [PATCH 01/13] Updates in line with current plans --- README.md | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index dd649cc1..f9001b4e 100644 --- a/README.md +++ b/README.md @@ -4,29 +4,30 @@ ![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg) ![CI](https://github.com/alan-turing-institute/clim-recal/actions/workflows/ci.yaml/badge.svg) -Welcome to `clim-recal`, a specialized resource designed to tackle systematic errors or biases in **Regional Climate Models (RCMs)**. As researchers, policy-makers, and various stakeholders explore publicly available RCMs, they need to consider the challenge of biases that can affect the accurate representation of climate change signals. +Welcome to `clim-recal`, a specialised resource designed to prepare data to tackle systematic errors or biases in **Regional Climate Models (RCMs)**. As researchers, policy-makers, and various stakeholders explore publicly available RCMs, they may wish to consider the challenge of biases that can affect the accurate representation of climate change signals. -`clim-recal` provides both a **broad review** of available bias correction methods as well as **software**, **practical tutorials** and **guidance** that helps users apply these methods methods to various datasets. +`clim-recal` provides a data-processing pipeline for extracting parts of the **UK Climate Projections 2018 Convection Permitting model (UKCP18-CPM)** in order to apply and assess **bias correction methods**. -`clim-recal` is an **extensive software library and guide to application of Bias Correction (BC) methods**: +In future, `clim-recal` aims to provide both a **broad review** of available **bias-correction methods** as well as assessment of these **methods** and **software** that helps users apply these methods methods to various datasets. + +`clim-recal:` - Contains accessible information about the [why and how of bias correction for climate data](#why-bias-correction) -- Is a software library for for the application of BC methods (see our full pipeline for bias-correction of the ground-breaking local-scale (2.2km) [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). `clim-recal` brings together different software packages in `python` and `R` that implement a variety of bias correction methods, making it easy to apply them to data and compare their outputs. +- Is a software library for pre-processing climate data to ready it for bias-correction (see our full pipeline for bias-correction of the ground-breaking local-scale (2.2km) [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). `clim-recal` brings together different software packages in `python` and `R` that implement a variety of bias correction methods, making it easy to apply them to data and compare their outputs. - Was developed in partnership with the MetOffice to ensure the propriety, quality, and usability of our work - Provides a framework for open additions of new software libraries/bias correction methods (in planning) -# Overview: Bias Correction Pipeline +# Overview: Data-processing Pipeline -`clim-recal` is a debiasing pipeline, with the following steps: +`clim-recal` is a data-processing pipeline, with the following steps: 1. **Set-up & data download** *We provide custom scripts to facilitate download of data* 2. **Preprocessing** *This includes reprojecting, resampling & splitting the data prior to bias correction* -3. **Apply bias correction** - *Our pipeline embeds two distinct methods of bias correction* -4. **Assess the debiased data** - *We have developed a way to assess the quality of the debiasing step across multiple alternative methods* + +Our team are currently working on applying and assessing different methods of bias correction. There's lots of collaboration opportunities as a result - [please get in touch!](docs/contributing.md) +In future, we may then embed the best method(s) for this dataset to the `clim-recal` pipeline, following expert review of the data assessments. For a quick start on bias correction, refer to our [pipeline guide](python/README.md). @@ -44,7 +45,6 @@ We are in the process of developing comprehensive documentation for our code bas ## To use `clim-recal` programmatically - There are extensive [`API Reference`](docs/reference) within the python code. -- Comments within `R` scripts ## To contribute to `clim-recal` @@ -52,8 +52,9 @@ We are in the process of developing comprehensive documentation for our code bas # The Datasets -## UKCP18 +## UKCP18-CPM The [UK Climate Projections 2018 (UKCP18)](https://www.metoffice.gov.uk/research/approach/collaboration/ukcp) dataset offers insights into the potential climate changes in the UK. UKCP18 is an advancement of the UKCP09 projections and delivers the latest evaluations of the UK's possible climate alterations in land and marine regions throughout the 21st century. This crucial information aids in future Climate Change Risk Assessments and supports the UK’s adaptation to climate change challenges and opportunities as per the National Adaptation Programme. +We make use of the [Convection Permitting Model (CPM)](). ADD MORE HERE ## HADS [HadUK-Grid](https://www.metoffice.gov.uk/research/climate/maps-and-data/data/haduk-grid/haduk-grid) is a comprehensive collection of climate data for the UK, compiled from various land surface observations across the country. This data is organized into a uniform grid to ensure consistent coverage throughout the UK at up to 1km x 1km resolution. The dataset, spanning from 1836 to the present, includes a variety of climate variables such as air temperature, precipitation, sunshine, and wind speed, available on daily, monthly, seasonal, and annual timescales. @@ -82,11 +83,7 @@ If you have suggestions on the repository, or would like to include a new method All are welcome and appreciated. # Future plans -- **Finish refactor for BC**: The infrastructure for testing bias correction methods needs some reworking and documentation. -- **Release BC results**: Provide results from example BC runs. -- **More BC Methods**: Further bias correction of UKCP18 products. *This is planned for a future release and is not available yet.* -- **Pipeline for adding new methods**: *This is planned for a future release and is not available yet.* - +- **Adding in bias correction method to pipeline** - following our sister project reviewing bias correction methods applied to this dataset, we may incorporate the selected method(s) to the pipeline. ## Acknowledgements From e309a6b16be900f1e8cba2cea531f2b9adcf2831 Mon Sep 17 00:00:00 2001 From: RuthBowyer Date: Mon, 18 Nov 2024 15:45:50 +0000 Subject: [PATCH 02/13] decapitalise cities to avoid confusion with command line instruction --- docs/download.qmd | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/download.qmd b/docs/download.qmd index 4ebe3a43..7fa26ca0 100644 --- a/docs/download.qmd +++ b/docs/download.qmd @@ -34,21 +34,21 @@ grep -iE "resample.*cpm.*rainfall.*01.*_[0-9]{8}-[0-9]{8}.*" data-v1.0.txt | xar ## Crops ### HADS -For a given region `` (either `Scotland`, `Glasgow`, `Manchester` or `London`), for measurement `` (either `tasmax`, `tasmin` or `pr`), the monthly data can be downloaded and decompressed with: +For a given region `` (either `scotland`, `glasgow`, `manchester` or `london`), for measurement `` (either `tasmax`, `tasmin` or `pr`), the monthly data can be downloaded and decompressed with: ```shell grep -iE "crop.*hads.*.*.*_[0-9]{8}-[0-9]{8}.*" data-v1.0.txt | xargs -n 1 curl -O; gunzip *.nc.gz ``` -For example, for region is `Manchester`, measure is `tasmax`: +For example, for region is `manchester`, measure is `tasmax`: ```shell grep -iE ".*crop.*hads.*manchester.*tasmax.*_[0-9]{8}-[0-9]{8}\.nc\.gz" data-v1.0.txt | xargs -n 1 curl -O; gunzip *.nc.gz ``` ### CPM -For a given region `` (either `Scotland`, `Glasgow`, `Manchester` or `London`), for measurement `` (either `tasmax`, `tasmin` or `pr`), for run `` (either `01`, `05`, `06`, `07`, `08`), the yearly data can be downloaded and decompressed with: +For a given region `` (either `scotland`, `glasgow`, `manchester` or `london`), for measurement `` (either `tasmax`, `tasmin` or `pr`), for run `` (either `01`, `05`, `06`, `07`, `08`), the yearly data can be downloaded and decompressed with: ```shell grep -iE "crop.*cpm.*.*.*_[0-9]{8}-[0-9]{8}.*" data-v1.0.txt | xargs -n 1 curl -O; gunzip *.nc.gz ``` -For example, for region `Manchester`, measure `tasmax`, run `01`: +For example, for region `manchester`, measure `tasmax`, run `01`: ```shell grep -iE ".*crop.*cpm.*manchester.*tasmax.*01_[0-9]{8}-[0-9]{8}\.nc\.gz" data-v1.0.txt | xargs -n 1 curl -O; gunzip *.nc.gz ``` From 7a004191c91399ad33c90315d5d9b053db80e80b Mon Sep 17 00:00:00 2001 From: RuthBowyer Date: Mon, 18 Nov 2024 16:00:26 +0000 Subject: [PATCH 03/13] Updates to read me - seperating out with sister project --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index f9001b4e..8be34c03 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Welcome to `clim-recal`, a specialised resource designed to prepare data to tack `clim-recal` provides a data-processing pipeline for extracting parts of the **UK Climate Projections 2018 Convection Permitting model (UKCP18-CPM)** in order to apply and assess **bias correction methods**. -In future, `clim-recal` aims to provide both a **broad review** of available **bias-correction methods** as well as assessment of these **methods** and **software** that helps users apply these methods methods to various datasets. +In future, our [sister project] aims to provide both a **broad review** of available **bias-correction methods** as well as assessment of these **methods** and **software** that helps users apply these methods methods to various datasets.The results of this work may then be integrated back to `clim-recal`. `clim-recal:` @@ -49,12 +49,14 @@ We are in the process of developing comprehensive documentation for our code bas ## To contribute to `clim-recal` - See the [Contributing](docs/contributing.md) section below +- To contribute to our sister project on assessing bias correction methods for these data, please contact Ruth Bowyer. + # The Datasets ## UKCP18-CPM The [UK Climate Projections 2018 (UKCP18)](https://www.metoffice.gov.uk/research/approach/collaboration/ukcp) dataset offers insights into the potential climate changes in the UK. UKCP18 is an advancement of the UKCP09 projections and delivers the latest evaluations of the UK's possible climate alterations in land and marine regions throughout the 21st century. This crucial information aids in future Climate Change Risk Assessments and supports the UK’s adaptation to climate change challenges and opportunities as per the National Adaptation Programme. -We make use of the [Convection Permitting Model (CPM)](). ADD MORE HERE +We make use of the [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). This dataset represents a much finer resolution of climate model (2.2km grid) than typical climate-models, representing a step forward in the ability to simulate small scale behavior (in particular 'atmospheric convection'), and the influence of mountains, coastlines and urban areas. As a result, the CPM provides access to credible climate information important for small-scale weather features and also on local (kilometre) scale; which is particularly important for improving our understanding of climate change in cities. ## HADS [HadUK-Grid](https://www.metoffice.gov.uk/research/climate/maps-and-data/data/haduk-grid/haduk-grid) is a comprehensive collection of climate data for the UK, compiled from various land surface observations across the country. This data is organized into a uniform grid to ensure consistent coverage throughout the UK at up to 1km x 1km resolution. The dataset, spanning from 1836 to the present, includes a variety of climate variables such as air temperature, precipitation, sunshine, and wind speed, available on daily, monthly, seasonal, and annual timescales. @@ -63,9 +65,7 @@ We make use of the [Convection Permitting Model (CPM)](). ADD MORE HERE Regional climate models contain systematic errors, or biases in their output [^1]. Biases arise in RCMs for a number of reasons, such as the assumptions in the general circulation models (GCMs), and in the downscaling process from GCM to RCM. -Researchers, policy-makers and other stakeholders wishing to use publicly available RCMs need to consider a range of "bias correction” methods (sometimes referred to as "bias adjustment" or "recalibration"). Bias correction methods offer a means of adjusting the outputs of RCM in a manner that might better reflect future climate change signals whilst preserving the natural and internal variability of climate [^2]. - -Part of the `clim-recal` project is to review several bias correction methods. This work is ongoing and you can find our initial [taxonomy here](https://docs.google.com/spreadsheets/d/18LIc8omSMTzOWM60aFNv1EZUl1qQN_DG8HFy1_0NdWk/edit?usp=sharing). When we've completed our literature review, it will be submitted for publication in an open peer-reviewed journal. +Researchers, policy-makers and other stakeholders wishing to use publicly available RCMs need to consider a range of "bias correction” methods (sometimes referred to as "bias adjustment" or "recalibration"). Bias correction methods offer a means of adjusting the outputs of RCM in a manner that might better reflect future climate change signals whilst preserving the natural and internal variability of climate [^2]. The `clim-recal` pipeline provides preprocessed data, including the innovative [UKCP18-CPM datasets](# The Datasets), to faciliate the assessment of these methods without requiring the whole (very large) dataset. Our work is however, just like climate data, intended to be dynamic, and we are in the process of setting up a pipeline for researchers creating new methods of bias correction to be able to submit their methods for inclusion on in the `clim-recal` repository. From 05c426e1fcdb867023eeafc6cf62858547c5af9a Mon Sep 17 00:00:00 2001 From: RuthBowyer Date: Mon, 18 Nov 2024 16:07:45 +0000 Subject: [PATCH 04/13] Add link to new repo --- README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 8be34c03..f685fe97 100644 --- a/README.md +++ b/README.md @@ -8,11 +8,10 @@ Welcome to `clim-recal`, a specialised resource designed to prepare data to tack `clim-recal` provides a data-processing pipeline for extracting parts of the **UK Climate Projections 2018 Convection Permitting model (UKCP18-CPM)** in order to apply and assess **bias correction methods**. -In future, our [sister project] aims to provide both a **broad review** of available **bias-correction methods** as well as assessment of these **methods** and **software** that helps users apply these methods methods to various datasets.The results of this work may then be integrated back to `clim-recal`. +In future, our [sister project](https://github.com/Urban-Analytics-Technology-Platform/bias-correction-for-cpm) aims to provide both a **broad review** of available **bias-correction methods** as well as assessment of these **methods** and **software** that helps users apply these methods methods to various datasets.The results of this work may then be integrated back to `clim-recal`. `clim-recal:` -- Contains accessible information about the [why and how of bias correction for climate data](#why-bias-correction) - Is a software library for pre-processing climate data to ready it for bias-correction (see our full pipeline for bias-correction of the ground-breaking local-scale (2.2km) [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). `clim-recal` brings together different software packages in `python` and `R` that implement a variety of bias correction methods, making it easy to apply them to data and compare their outputs. - Was developed in partnership with the MetOffice to ensure the propriety, quality, and usability of our work - Provides a framework for open additions of new software libraries/bias correction methods (in planning) @@ -83,7 +82,7 @@ If you have suggestions on the repository, or would like to include a new method All are welcome and appreciated. # Future plans -- **Adding in bias correction method to pipeline** - following our sister project reviewing bias correction methods applied to this dataset, we may incorporate the selected method(s) to the pipeline. +- **Adding in bias correction method to pipeline** - following our [sister project](https://github.com/Urban-Analytics-Technology-Platform/bias-correction-for-cpm) reviewing bias correction methods applied to this dataset, we may incorporate the selected method(s) to the pipeline. ## Acknowledgements From 62aa6384ba13d8e9bd36920ff9267ccf999a2c75 Mon Sep 17 00:00:00 2001 From: RuthBowyer Date: Mon, 25 Nov 2024 10:59:59 +0000 Subject: [PATCH 05/13] Updates to readme - simplifying to avoid confusion --- README.md | 49 +++++++++++++++++++++++-------------------------- 1 file changed, 23 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index f685fe97..b819ce5f 100644 --- a/README.md +++ b/README.md @@ -4,32 +4,39 @@ ![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg) ![CI](https://github.com/alan-turing-institute/clim-recal/actions/workflows/ci.yaml/badge.svg) -Welcome to `clim-recal`, a specialised resource designed to prepare data to tackle systematic errors or biases in **Regional Climate Models (RCMs)**. As researchers, policy-makers, and various stakeholders explore publicly available RCMs, they may wish to consider the challenge of biases that can affect the accurate representation of climate change signals. +Welcome to `clim-recal`, a specialised resource which provides a data-processing pipeline for extracting parts of the **UK Climate Projections 2018 Convection Permitting model (UKCP18-CPM)** in order to apply and assess **bias correction methods** via adjustment to and comaprison with the **Had-UK grid**. -`clim-recal` provides a data-processing pipeline for extracting parts of the **UK Climate Projections 2018 Convection Permitting model (UKCP18-CPM)** in order to apply and assess **bias correction methods**. - -In future, our [sister project](https://github.com/Urban-Analytics-Technology-Platform/bias-correction-for-cpm) aims to provide both a **broad review** of available **bias-correction methods** as well as assessment of these **methods** and **software** that helps users apply these methods methods to various datasets.The results of this work may then be integrated back to `clim-recal`. `clim-recal:` -- Is a software library for pre-processing climate data to ready it for bias-correction (see our full pipeline for bias-correction of the ground-breaking local-scale (2.2km) [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). `clim-recal` brings together different software packages in `python` and `R` that implement a variety of bias correction methods, making it easy to apply them to data and compare their outputs. +- Is a software library for pre-processing climate data to ready it for bias-correction - Was developed in partnership with the MetOffice to ensure the propriety, quality, and usability of our work - Provides a framework for open additions of new software libraries/bias correction methods (in planning) # Overview: Data-processing Pipeline -`clim-recal` is a data-processing pipeline, with the following steps: +Regional climate models (RCMs) contain systematic errors, or biases in their output [^1]. Biases arise in RCMs for a number of reasons, such as the assumptions in the general circulation models (GCMs), and in the downscaling process from GCM to RCM. + +Researchers, policy-makers and other stakeholders wishing to use publicly available RCMs need to consider a range of "bias correction” methods (sometimes referred to as "bias adjustment" or "recalibration"). +Bias correction methods offer a means of adjusting the outputs of RCM in a manner that might better reflect future climate change signals whilst preserving the natural and internal variability of climate [^2]. + +However, in order to apply and assess these methods, the climate model of interest needs to be overlaid to corresponding observation data. This can be a time-consuming and laborious process where data is spatially and temporally very granular. + +The `clim-recal` pipeline addresses this by providing preprocessed data, including the innovative [UKCP18-CPM datasets](# The Datasets), to facilitate the assessment of these methods on aligned, reprojected data, without requiring the whole (very large) dataset. + +`clim-recal` is a data-processing pipeline, with the following steps: 1. **Set-up & data download** *We provide custom scripts to facilitate download of data* 2. **Preprocessing** *This includes reprojecting, resampling & splitting the data prior to bias correction* -Our team are currently working on applying and assessing different methods of bias correction. There's lots of collaboration opportunities as a result - [please get in touch!](docs/contributing.md) -In future, we may then embed the best method(s) for this dataset to the `clim-recal` pipeline, following expert review of the data assessments. For a quick start on bias correction, refer to our [pipeline guide](python/README.md). +Our work is however, just like climate data, intended to be dynamic, and we welcome collaboration from researchers who wish to further our aims! + + # Documentation We are in the process of developing comprehensive documentation for our code base to supplement the guidance provided in this and other `README.md` files. In the interim, there is documentation available in the following forms: @@ -48,8 +55,6 @@ We are in the process of developing comprehensive documentation for our code bas ## To contribute to `clim-recal` - See the [Contributing](docs/contributing.md) section below -- To contribute to our sister project on assessing bias correction methods for these data, please contact Ruth Bowyer. - # The Datasets @@ -60,30 +65,16 @@ We make use of the [Convection Permitting Model (CPM)](https://www.metoffice.gov ## HADS [HadUK-Grid](https://www.metoffice.gov.uk/research/climate/maps-and-data/data/haduk-grid/haduk-grid) is a comprehensive collection of climate data for the UK, compiled from various land surface observations across the country. This data is organized into a uniform grid to ensure consistent coverage throughout the UK at up to 1km x 1km resolution. The dataset, spanning from 1836 to the present, includes a variety of climate variables such as air temperature, precipitation, sunshine, and wind speed, available on daily, monthly, seasonal, and annual timescales. -# Why Bias Correction? - -Regional climate models contain systematic errors, or biases in their output [^1]. Biases arise in RCMs for a number of reasons, such as the assumptions in the general circulation models (GCMs), and in the downscaling process from GCM to RCM. - -Researchers, policy-makers and other stakeholders wishing to use publicly available RCMs need to consider a range of "bias correction” methods (sometimes referred to as "bias adjustment" or "recalibration"). Bias correction methods offer a means of adjusting the outputs of RCM in a manner that might better reflect future climate change signals whilst preserving the natural and internal variability of climate [^2]. The `clim-recal` pipeline provides preprocessed data, including the innovative [UKCP18-CPM datasets](# The Datasets), to faciliate the assessment of these methods without requiring the whole (very large) dataset. - -Our work is however, just like climate data, intended to be dynamic, and we are in the process of setting up a pipeline for researchers creating new methods of bias correction to be able to submit their methods for inclusion on in the `clim-recal` repository. - -[^1]: Senatore et al., 2022, -[^2]: Ayar et al., 2021, - # Contributing -If you have suggestions on the repository, or would like to include a new method (see below) or library, please +If you have suggestions on the repository, please: - raise an [issue](https://github.com/alan-turing-institute/clim-recal/issues) - [get in touch](mailto:clim-recal@turing.ac.uk) - see our [contributing](docs/contributing.md) section, which includes details on contriubting to the documentation. All are welcome and appreciated. -# Future plans -- **Adding in bias correction method to pipeline** - following our [sister project](https://github.com/Urban-Analytics-Technology-Platform/bias-correction-for-cpm) reviewing bias correction methods applied to this dataset, we may incorporate the selected method(s) to the pipeline. - ## Acknowledgements Prior to 12th September 2024 we included a reference to the [python-cmethods](https://github.com/btschwertfeger/python-cmethods) library, written by Benjamin Thomas Schwertfeger. @@ -98,6 +89,12 @@ Inadvertently, we did not identify that the license for the `python-cmethods` li * Added the citation below. -## Citation +## Citations + +[^1]: Senatore et al., 2022, +[^2]: Ayar et al., 2021, **python-cmethods**: Benjamin T. Schwertfeger. (2024). btschwertfeger/python-cmethods: v2.3.0 (v2.3.0). Zenodo. https://doi.org/10.5281/zenodo.12168002 + + + From c48cd7a8c73a364f22dd3283a5b0f2a26cf2c045 Mon Sep 17 00:00:00 2001 From: RuthBowyer Date: Mon, 25 Nov 2024 11:09:03 +0000 Subject: [PATCH 06/13] I don't think we need to include these R scripts in the docs, so removing --- _quarto.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/_quarto.yml b/_quarto.yml index 8df764c8..a140ae70 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -9,9 +9,6 @@ project: - "README.md" - "setup-instructions.md" - "!clim-recal.Rproj" - - "R/README.md" - - "R/misc/Identifying_Runs.md" - - "R/comparing-r-and-python/HADs-reprojection/WIP-Comparing-HADs-grids.md" - "docs/cpm_projection.qmd" - "docs/reference" - "docs/contributing.md" From dd668d6a95b6e790db5fa8f2d31fb009b8ed16b3 Mon Sep 17 00:00:00 2001 From: RuthBowyer Date: Mon, 25 Nov 2024 11:09:22 +0000 Subject: [PATCH 07/13] Added a note about run selection --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index b819ce5f..8d6ac265 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,8 @@ We are in the process of developing comprehensive documentation for our code bas The [UK Climate Projections 2018 (UKCP18)](https://www.metoffice.gov.uk/research/approach/collaboration/ukcp) dataset offers insights into the potential climate changes in the UK. UKCP18 is an advancement of the UKCP09 projections and delivers the latest evaluations of the UK's possible climate alterations in land and marine regions throughout the 21st century. This crucial information aids in future Climate Change Risk Assessments and supports the UK’s adaptation to climate change challenges and opportunities as per the National Adaptation Programme. We make use of the [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). This dataset represents a much finer resolution of climate model (2.2km grid) than typical climate-models, representing a step forward in the ability to simulate small scale behavior (in particular 'atmospheric convection'), and the influence of mountains, coastlines and urban areas. As a result, the CPM provides access to credible climate information important for small-scale weather features and also on local (kilometre) scale; which is particularly important for improving our understanding of climate change in cities. +The UKCP18-CPM is comprised of 12 ensemble members (or runs). In addition to run 1, we selected the runs which represented the mean, 2nd highest and 2nd lowest daily tasmax values across the whole sequence (runs 5, 6, 7 & 8) to provide users with enough uncertainty in their estimates to appropriately assess bias correction methods. + ## HADS [HadUK-Grid](https://www.metoffice.gov.uk/research/climate/maps-and-data/data/haduk-grid/haduk-grid) is a comprehensive collection of climate data for the UK, compiled from various land surface observations across the country. This data is organized into a uniform grid to ensure consistent coverage throughout the UK at up to 1km x 1km resolution. The dataset, spanning from 1836 to the present, includes a variety of climate variables such as air temperature, precipitation, sunshine, and wind speed, available on daily, monthly, seasonal, and annual timescales. From bbac394d9ac13823f46aebb714f2ceff01daac71 Mon Sep 17 00:00:00 2001 From: Andy Smith Date: Wed, 27 Nov 2024 13:23:44 +0000 Subject: [PATCH 08/13] fix linting for README.md --- README.md | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 8d6ac265..41c5eda9 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,12 @@ ![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg) ![CI](https://github.com/alan-turing-institute/clim-recal/actions/workflows/ci.yaml/badge.svg) -Welcome to `clim-recal`, a specialised resource which provides a data-processing pipeline for extracting parts of the **UK Climate Projections 2018 Convection Permitting model (UKCP18-CPM)** in order to apply and assess **bias correction methods** via adjustment to and comaprison with the **Had-UK grid**. +Welcome to `clim-recal`, a specialised resource which provides a data-processing pipeline for extracting parts of the **UK Climate Projections 2018 Convection Permitting model (UKCP18-CPM)** in order to apply and assess **bias correction methods** via adjustment to and comaprison with the **Had-UK grid**. `clim-recal:` -- Is a software library for pre-processing climate data to ready it for bias-correction +- Is a software library for pre-processing climate data to ready it for bias-correction - Was developed in partnership with the MetOffice to ensure the propriety, quality, and usability of our work - Provides a framework for open additions of new software libraries/bias correction methods (in planning) @@ -17,12 +17,12 @@ Welcome to `clim-recal`, a specialised resource which provides a data-processing Regional climate models (RCMs) contain systematic errors, or biases in their output [^1]. Biases arise in RCMs for a number of reasons, such as the assumptions in the general circulation models (GCMs), and in the downscaling process from GCM to RCM. -Researchers, policy-makers and other stakeholders wishing to use publicly available RCMs need to consider a range of "bias correction” methods (sometimes referred to as "bias adjustment" or "recalibration"). -Bias correction methods offer a means of adjusting the outputs of RCM in a manner that might better reflect future climate change signals whilst preserving the natural and internal variability of climate [^2]. +Researchers, policy-makers and other stakeholders wishing to use publicly available RCMs need to consider a range of "bias correction” methods (sometimes referred to as "bias adjustment" or "recalibration"). +Bias correction methods offer a means of adjusting the outputs of RCM in a manner that might better reflect future climate change signals whilst preserving the natural and internal variability of climate [^2]. -However, in order to apply and assess these methods, the climate model of interest needs to be overlaid to corresponding observation data. This can be a time-consuming and laborious process where data is spatially and temporally very granular. +However, in order to apply and assess these methods, the climate model of interest needs to be overlaid to corresponding observation data. This can be a time-consuming and laborious process where data is spatially and temporally very granular. -The `clim-recal` pipeline addresses this by providing preprocessed data, including the innovative [UKCP18-CPM datasets](# The Datasets), to facilitate the assessment of these methods on aligned, reprojected data, without requiring the whole (very large) dataset. +The `clim-recal` pipeline addresses this by providing preprocessed data, including the innovative [UKCP18-CPM datasets](# The Datasets), to facilitate the assessment of these methods on aligned, reprojected data, without requiring the whole (very large) dataset. `clim-recal` is a data-processing pipeline, with the following steps: @@ -30,11 +30,11 @@ The `clim-recal` pipeline addresses this by providing preprocessed data, includi *We provide custom scripts to facilitate download of data* 2. **Preprocessing** *This includes reprojecting, resampling & splitting the data prior to bias correction* - + For a quick start on bias correction, refer to our [pipeline guide](python/README.md). -Our work is however, just like climate data, intended to be dynamic, and we welcome collaboration from researchers who wish to further our aims! +Our work is however, just like climate data, intended to be dynamic, and we welcome collaboration from researchers who wish to further our aims! # Documentation @@ -60,9 +60,9 @@ We are in the process of developing comprehensive documentation for our code bas ## UKCP18-CPM The [UK Climate Projections 2018 (UKCP18)](https://www.metoffice.gov.uk/research/approach/collaboration/ukcp) dataset offers insights into the potential climate changes in the UK. UKCP18 is an advancement of the UKCP09 projections and delivers the latest evaluations of the UK's possible climate alterations in land and marine regions throughout the 21st century. This crucial information aids in future Climate Change Risk Assessments and supports the UK’s adaptation to climate change challenges and opportunities as per the National Adaptation Programme. -We make use of the [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). This dataset represents a much finer resolution of climate model (2.2km grid) than typical climate-models, representing a step forward in the ability to simulate small scale behavior (in particular 'atmospheric convection'), and the influence of mountains, coastlines and urban areas. As a result, the CPM provides access to credible climate information important for small-scale weather features and also on local (kilometre) scale; which is particularly important for improving our understanding of climate change in cities. +We make use of the [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). This dataset represents a much finer resolution of climate model (2.2km grid) than typical climate-models, representing a step forward in the ability to simulate small scale behavior (in particular 'atmospheric convection'), and the influence of mountains, coastlines and urban areas. As a result, the CPM provides access to credible climate information important for small-scale weather features and also on local (kilometre) scale; which is particularly important for improving our understanding of climate change in cities. -The UKCP18-CPM is comprised of 12 ensemble members (or runs). In addition to run 1, we selected the runs which represented the mean, 2nd highest and 2nd lowest daily tasmax values across the whole sequence (runs 5, 6, 7 & 8) to provide users with enough uncertainty in their estimates to appropriately assess bias correction methods. +The UKCP18-CPM is comprised of 12 ensemble members (or runs). In addition to run 1, we selected the runs which represented the mean, 2nd highest and 2nd lowest daily tasmax values across the whole sequence (runs 5, 6, 7 & 8) to provide users with enough uncertainty in their estimates to appropriately assess bias correction methods. ## HADS [HadUK-Grid](https://www.metoffice.gov.uk/research/climate/maps-and-data/data/haduk-grid/haduk-grid) is a comprehensive collection of climate data for the UK, compiled from various land surface observations across the country. This data is organized into a uniform grid to ensure consistent coverage throughout the UK at up to 1km x 1km resolution. The dataset, spanning from 1836 to the present, includes a variety of climate variables such as air temperature, precipitation, sunshine, and wind speed, available on daily, monthly, seasonal, and annual timescales. @@ -97,6 +97,3 @@ Inadvertently, we did not identify that the license for the `python-cmethods` li [^2]: Ayar et al., 2021, **python-cmethods**: Benjamin T. Schwertfeger. (2024). btschwertfeger/python-cmethods: v2.3.0 (v2.3.0). Zenodo. https://doi.org/10.5281/zenodo.12168002 - - - From 3f2a90d0c221fc63316de598324ed44c84b3662a Mon Sep 17 00:00:00 2001 From: Ruth Bowyer <105492883+RuthBowyer@users.noreply.github.com> Date: Mon, 2 Dec 2024 10:32:01 +0000 Subject: [PATCH 09/13] Update README.md Co-authored-by: Sam Greenbury <50113363+sgreenbury@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 41c5eda9..c55ed275 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ ![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg) ![CI](https://github.com/alan-turing-institute/clim-recal/actions/workflows/ci.yaml/badge.svg) -Welcome to `clim-recal`, a specialised resource which provides a data-processing pipeline for extracting parts of the **UK Climate Projections 2018 Convection Permitting model (UKCP18-CPM)** in order to apply and assess **bias correction methods** via adjustment to and comaprison with the **Had-UK grid**. +Welcome to `clim-recal`, a specialised resource which provides a data-processing pipeline for extracting parts of the **UK Climate Projections 2018 Convection Permitting model (UKCP18-CPM)** in order to apply and assess **bias correction methods** via adjustment to and comparison with the **HadUK-Grid**. `clim-recal:` From bde554b663507f03d67bc9f376e2ed9416c1aad7 Mon Sep 17 00:00:00 2001 From: Ruth Bowyer <105492883+RuthBowyer@users.noreply.github.com> Date: Mon, 2 Dec 2024 10:32:15 +0000 Subject: [PATCH 10/13] Update README.md Co-authored-by: Sam Greenbury <50113363+sgreenbury@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c55ed275..18994981 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ The `clim-recal` pipeline addresses this by providing preprocessed data, includi *This includes reprojecting, resampling & splitting the data prior to bias correction* -For a quick start on bias correction, refer to our [pipeline guide](python/README.md). +For a quick start on installing and running the pipeline, refer to our [pipeline guide](python/README.md). Our work is however, just like climate data, intended to be dynamic, and we welcome collaboration from researchers who wish to further our aims! From 1059cce3345827bbdcc14d3dcf04ddf64502e85a Mon Sep 17 00:00:00 2001 From: RuthBowyer Date: Tue, 3 Dec 2024 21:04:14 +0000 Subject: [PATCH 11/13] Inciorporating review suggestions to read me -m resolve conflict --- README.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 18994981..e9eb258a 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,8 @@ Bias correction methods offer a means of adjusting the outputs of RCM in a manne However, in order to apply and assess these methods, the climate model of interest needs to be overlaid to corresponding observation data. This can be a time-consuming and laborious process where data is spatially and temporally very granular. -The `clim-recal` pipeline addresses this by providing preprocessed data, including the innovative [UKCP18-CPM datasets](# The Datasets), to facilitate the assessment of these methods on aligned, reprojected data, without requiring the whole (very large) dataset. + +The `clim-recal` pipeline addresses this by providing preprocessed data, including the innovative [UKCP18-CPM datasets](#the-datasets), to facilitate the assessment of these methods on aligned, reprojected data, without requiring the whole (very large) dataset. `clim-recal` is a data-processing pipeline, with the following steps: @@ -60,9 +61,19 @@ We are in the process of developing comprehensive documentation for our code bas ## UKCP18-CPM The [UK Climate Projections 2018 (UKCP18)](https://www.metoffice.gov.uk/research/approach/collaboration/ukcp) dataset offers insights into the potential climate changes in the UK. UKCP18 is an advancement of the UKCP09 projections and delivers the latest evaluations of the UK's possible climate alterations in land and marine regions throughout the 21st century. This crucial information aids in future Climate Change Risk Assessments and supports the UK’s adaptation to climate change challenges and opportunities as per the National Adaptation Programme. -We make use of the [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). This dataset represents a much finer resolution of climate model (2.2km grid) than typical climate-models, representing a step forward in the ability to simulate small scale behavior (in particular 'atmospheric convection'), and the influence of mountains, coastlines and urban areas. As a result, the CPM provides access to credible climate information important for small-scale weather features and also on local (kilometre) scale; which is particularly important for improving our understanding of climate change in cities. -The UKCP18-CPM is comprised of 12 ensemble members (or runs). In addition to run 1, we selected the runs which represented the mean, 2nd highest and 2nd lowest daily tasmax values across the whole sequence (runs 5, 6, 7 & 8) to provide users with enough uncertainty in their estimates to appropriately assess bias correction methods. +We make use of the [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). This dataset represents a much finer spatial resolution of climate model (2.2km grid) than typical climate-models, representing a step forward in the ability to simulate small scale behavior (in particular 'atmospheric convection'), and the influence of mountains, coastlines and urban areas. As a result, the CPM provides access to credible climate information important for small-scale weather features and also on local (kilometre) scale; which is particularly important for improving our understanding of climate change in cities. + +The UKCP18-CPM represents a high-emission scenario (RCP 8.5). + +The UKCP18-CPM is comprised of 12 ensemble members (or runs), driven by the same 12km Regional Climate Model (Strand 3 12km RCM ensemble). In addition to run 1, we selected the following runs: + +- Run 05: Represents the ensemble member with the second lowest mean annual tasmax of all ensembles members +- Run 06: Represents the ensemble member with the second highest mean annual tasmax of all ensembles members +- Run 07 & Run 08: Represent the ensemble members with the average mean annual tasmax of all ensemble members + +We believe that this combination will provide users with enough uncertainty in their estimates to appropriately assess bias correction methods. + ## HADS [HadUK-Grid](https://www.metoffice.gov.uk/research/climate/maps-and-data/data/haduk-grid/haduk-grid) is a comprehensive collection of climate data for the UK, compiled from various land surface observations across the country. This data is organized into a uniform grid to ensure consistent coverage throughout the UK at up to 1km x 1km resolution. The dataset, spanning from 1836 to the present, includes a variety of climate variables such as air temperature, precipitation, sunshine, and wind speed, available on daily, monthly, seasonal, and annual timescales. @@ -73,7 +84,7 @@ The UKCP18-CPM is comprised of 12 ensemble members (or runs). In addition to run If you have suggestions on the repository, please: - raise an [issue](https://github.com/alan-turing-institute/clim-recal/issues) - [get in touch](mailto:clim-recal@turing.ac.uk) -- see our [contributing](docs/contributing.md) section, which includes details on contriubting to the documentation. +- see our [contributing](docs/contributing.md) section, which includes details on contributing to the documentation. All are welcome and appreciated. From 62965e1b03030dc0a4a17bceef7699e939ae1479 Mon Sep 17 00:00:00 2001 From: RuthBowyer Date: Tue, 3 Dec 2024 21:04:23 +0000 Subject: [PATCH 12/13] Recaptilise letters --- docs/download.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/download.qmd b/docs/download.qmd index 7fa26ca0..069623ea 100644 --- a/docs/download.qmd +++ b/docs/download.qmd @@ -34,7 +34,7 @@ grep -iE "resample.*cpm.*rainfall.*01.*_[0-9]{8}-[0-9]{8}.*" data-v1.0.txt | xar ## Crops ### HADS -For a given region `` (either `scotland`, `glasgow`, `manchester` or `london`), for measurement `` (either `tasmax`, `tasmin` or `pr`), the monthly data can be downloaded and decompressed with: +For a given region `` (either `Scotland`, `Glasgow`, `Manchester` or `London`), for measurement `` (either `tasmax`, `tasmin` or `pr`), the monthly data can be downloaded and decompressed with: ```shell grep -iE "crop.*hads.*.*.*_[0-9]{8}-[0-9]{8}.*" data-v1.0.txt | xargs -n 1 curl -O; gunzip *.nc.gz ``` @@ -44,11 +44,11 @@ grep -iE ".*crop.*hads.*manchester.*tasmax.*_[0-9]{8}-[0-9]{8}\.nc\.gz" data-v1. ``` ### CPM -For a given region `` (either `scotland`, `glasgow`, `manchester` or `london`), for measurement `` (either `tasmax`, `tasmin` or `pr`), for run `` (either `01`, `05`, `06`, `07`, `08`), the yearly data can be downloaded and decompressed with: +For a given region `` (either `Scotland`, `Glasgow`, `Manchester` or `London`), for measurement `` (either `tasmax`, `tasmin` or `pr`), for run `` (either `01`, `05`, `06`, `07`, `08`), the yearly data can be downloaded and decompressed with: ```shell grep -iE "crop.*cpm.*.*.*_[0-9]{8}-[0-9]{8}.*" data-v1.0.txt | xargs -n 1 curl -O; gunzip *.nc.gz ``` -For example, for region `manchester`, measure `tasmax`, run `01`: +For example, for region `Manchester`, measure `tasmax`, run `01`: ```shell grep -iE ".*crop.*cpm.*manchester.*tasmax.*01_[0-9]{8}-[0-9]{8}\.nc\.gz" data-v1.0.txt | xargs -n 1 curl -O; gunzip *.nc.gz ``` From cda835d8b2405e8a4fcc1363e635c1ed7ccf95eb Mon Sep 17 00:00:00 2001 From: Andy Smith Date: Fri, 6 Dec 2024 16:20:19 +0000 Subject: [PATCH 13/13] fixes linting --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index e9eb258a..e61fac83 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ Bias correction methods offer a means of adjusting the outputs of RCM in a manne However, in order to apply and assess these methods, the climate model of interest needs to be overlaid to corresponding observation data. This can be a time-consuming and laborious process where data is spatially and temporally very granular. -The `clim-recal` pipeline addresses this by providing preprocessed data, including the innovative [UKCP18-CPM datasets](#the-datasets), to facilitate the assessment of these methods on aligned, reprojected data, without requiring the whole (very large) dataset. +The `clim-recal` pipeline addresses this by providing preprocessed data, including the innovative [UKCP18-CPM datasets](#the-datasets), to facilitate the assessment of these methods on aligned, reprojected data, without requiring the whole (very large) dataset. `clim-recal` is a data-processing pipeline, with the following steps: @@ -62,17 +62,17 @@ We are in the process of developing comprehensive documentation for our code bas ## UKCP18-CPM The [UK Climate Projections 2018 (UKCP18)](https://www.metoffice.gov.uk/research/approach/collaboration/ukcp) dataset offers insights into the potential climate changes in the UK. UKCP18 is an advancement of the UKCP09 projections and delivers the latest evaluations of the UK's possible climate alterations in land and marine regions throughout the 21st century. This crucial information aids in future Climate Change Risk Assessments and supports the UK’s adaptation to climate change challenges and opportunities as per the National Adaptation Programme. -We make use of the [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). This dataset represents a much finer spatial resolution of climate model (2.2km grid) than typical climate-models, representing a step forward in the ability to simulate small scale behavior (in particular 'atmospheric convection'), and the influence of mountains, coastlines and urban areas. As a result, the CPM provides access to credible climate information important for small-scale weather features and also on local (kilometre) scale; which is particularly important for improving our understanding of climate change in cities. +We make use of the [Convection Permitting Model (CPM)](https://www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP-Convection-permitting-model-projections-report.pdf). This dataset represents a much finer spatial resolution of climate model (2.2km grid) than typical climate-models, representing a step forward in the ability to simulate small scale behavior (in particular 'atmospheric convection'), and the influence of mountains, coastlines and urban areas. As a result, the CPM provides access to credible climate information important for small-scale weather features and also on local (kilometre) scale; which is particularly important for improving our understanding of climate change in cities. The UKCP18-CPM represents a high-emission scenario (RCP 8.5). The UKCP18-CPM is comprised of 12 ensemble members (or runs), driven by the same 12km Regional Climate Model (Strand 3 12km RCM ensemble). In addition to run 1, we selected the following runs: -- Run 05: Represents the ensemble member with the second lowest mean annual tasmax of all ensembles members +- Run 05: Represents the ensemble member with the second lowest mean annual tasmax of all ensembles members - Run 06: Represents the ensemble member with the second highest mean annual tasmax of all ensembles members - Run 07 & Run 08: Represent the ensemble members with the average mean annual tasmax of all ensemble members -We believe that this combination will provide users with enough uncertainty in their estimates to appropriately assess bias correction methods. +We believe that this combination will provide users with enough uncertainty in their estimates to appropriately assess bias correction methods. ## HADS