From 99217dbebe0b2bb114315e268ad2893c214c2245 Mon Sep 17 00:00:00 2001 From: Quarto GHA Workflow Runner Date: Wed, 20 Nov 2024 10:31:05 +0000 Subject: [PATCH] Built site for gh-pages --- .nojekyll | 2 +- docs/deprecated_pipeline.html | 2 +- docs/reproducibility.qmd | 59 ++++++++++++++++++++++++++++++++++- sitemap.xml | 48 ++++++++++++++-------------- 4 files changed, 84 insertions(+), 27 deletions(-) diff --git a/.nojekyll b/.nojekyll index 03c97d05..7d16130c 100644 --- a/.nojekyll +++ b/.nojekyll @@ -1 +1 @@ -382bf7c7 \ No newline at end of file +5c23243f \ No newline at end of file diff --git a/docs/deprecated_pipeline.html b/docs/deprecated_pipeline.html index f7882c47..6bd5a3d9 100644 --- a/docs/deprecated_pipeline.html +++ b/docs/deprecated_pipeline.html @@ -431,7 +431,7 @@

"tasmax, tasmin, pr/rainfall" +* `--all-regions` => "Glasgow, London, Manchester, Scotland" +* `--run 01`, `--run 05`, `--run 06`, --run 07`, `--run 08` => The data from CPM runs 01, 05, 06 and 07. + +An summary of the operation of this script: + +* Creates temporary directories to hold one year of CPM and HADs data on a local, fast disk. +* Loops through each year of data (1980 through to 2080). For each year it: + * Copies the relevant CPM and HAD files into the working directory, whilst maintaining the directory structure. + * Runs Clim-recal using the options above. + * Deletes certain extraneous crop files. (Due to a bug, certain output files are created multiple times. As a workaround we simply deleted the extra files by calling `bash/remove-extra-cropfiles.py` from run-pipeline-iteratively shell script). + + +## bash/combine-iterative-runs.sh + +A side effect of running the pipeline iteratively, is that the outputs for each year are placed in their own timestamped directory. This script uses rsync to combine these into a single coherent output directory. + +# Verifying results + +In order to assert that the results produced by the pipeline it is necessary to have a method to compare the outputs of different executions of the pipeline. Because netCDF files can store their creation date within their header, it is not possible to rely on a checksum of the entire file to assure reproducibility. + +Therefore we just select the last 10k bytes of data from each file. We generate the checksums of the file subsets using this script: + +`bash/generate_trailing_checksums.sh` + +This script requires two arguments: +- The directory of files to create checksums for. All "*.nc" file within this directory +- The number of trailing bytes to use in teh checksum calculation (this is passed as an argument to `tail`) + +The script produces a sorted list of relative file paths and their checksums, in a text file named `manifest_last_bytes_$2.txt`. The manifest files for two executions of the pipeline should be comparable with using the standard *NIX `diff` command. diff --git a/sitemap.xml b/sitemap.xml index 20f50088..3123bcaa 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,98 +2,98 @@ https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.ceda_ftp_download.html - 2024-11-06T16:05:44.012Z + 2024-11-20T10:27:19.623Z https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.utils.core.html - 2024-11-06T16:05:44.200Z + 2024-11-20T10:27:19.815Z https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.utils.xarray.html - 2024-11-06T16:05:44.312Z + 2024-11-20T10:27:19.927Z https://alan-turing-institute.github.io/clim-recal/docs/cpm_projection.html - 2024-11-06T16:04:48.298Z + 2024-11-20T10:26:24.387Z https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.resample.html - 2024-11-06T16:05:44.100Z + 2024-11-20T10:27:19.715Z https://alan-turing-institute.github.io/clim-recal/docs/deprecated_pipeline.html - 2024-11-06T16:04:48.298Z + 2024-11-20T10:26:24.387Z https://alan-turing-institute.github.io/clim-recal/docs/docker-configurations.html - 2024-11-06T16:04:48.298Z + 2024-11-20T10:26:24.387Z https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.pipeline.html - 2024-11-06T16:05:44.008Z + 2024-11-20T10:27:19.619Z https://alan-turing-institute.github.io/clim-recal/notebooks/cpm_projection_diff_plots.html - 2024-11-06T16:04:48.354Z + 2024-11-20T10:26:24.443Z https://alan-turing-institute.github.io/clim-recal/docs/contributing.html - 2024-11-06T16:04:48.298Z + 2024-11-20T10:26:24.383Z https://alan-turing-institute.github.io/clim-recal/R/misc/Identifying_Runs.html - 2024-11-06T16:04:48.058Z + 2024-11-20T10:26:24.143Z https://alan-turing-institute.github.io/clim-recal/setup-instructions.html - 2024-11-06T16:04:48.374Z + 2024-11-20T10:26:24.463Z https://alan-turing-institute.github.io/clim-recal/README.html - 2024-11-06T16:04:48.070Z + 2024-11-20T10:26:24.159Z https://alan-turing-institute.github.io/clim-recal/R/README.html - 2024-11-06T16:04:48.054Z + 2024-11-20T10:26:24.139Z https://alan-turing-institute.github.io/clim-recal/R/comparing-r-and-python/HADs-reprojection/WIP-Comparing-HADs-grids.html - 2024-11-06T16:04:48.054Z + 2024-11-20T10:26:24.139Z https://alan-turing-institute.github.io/clim-recal/python/README.html - 2024-11-06T16:04:48.370Z + 2024-11-20T10:26:24.459Z https://alan-turing-institute.github.io/clim-recal/notebooks/cpm_projection_diff_plots_linear_nearest.html - 2024-11-06T16:04:48.366Z + 2024-11-20T10:26:24.455Z https://alan-turing-institute.github.io/clim-recal/docs/download.html - 2024-11-06T16:04:48.298Z + 2024-11-20T10:26:24.387Z https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.debiasing.debias_wrapper.html - 2024-11-06T16:05:44.140Z + 2024-11-20T10:27:19.755Z https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.config.html - 2024-11-06T16:05:44.040Z + 2024-11-20T10:27:19.655Z https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.data_loader.html - 2024-11-06T16:05:44.024Z + 2024-11-20T10:27:19.639Z https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.utils.server.html - 2024-11-06T16:05:44.236Z + 2024-11-20T10:27:19.851Z https://alan-turing-institute.github.io/clim-recal/docs/reference/clim_recal.utils.data.html - 2024-11-06T16:05:44.348Z + 2024-11-20T10:27:19.963Z https://alan-turing-institute.github.io/clim-recal/docs/reference/index.html - 2024-11-06T16:05:43.988Z + 2024-11-20T10:27:19.603Z