notebooks/DEACoastlines_generation_CLI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "69da8225-7681-4de4-b1b5-be4fdabf52c7",
   "metadata": {
    "tags": []
   },
   "source": [
    "# DEA Coastlines generation using command line tools <img align=\"right\" src=\"https://github.com/GeoscienceAustralia/dea-notebooks/raw/develop/Supplementary_data/dea_logo.jpg\">\n",
    "This notebook demonstrates how to run a DEA Coastlines analysis using command line tools."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81b6c48d-db3b-4f30-82fa-67c96e64cc1c",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e436c72-fa98-489e-bc7d-787ead143e76",
   "metadata": {},
   "source": [
    "Set working directory to top level of repo to ensure links work correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c0c743c8-8d0d-4f71-9a70-c4132adaaa86",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan/Robbi/dea-coastlines\n"
     ]
    }
   ],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0ec2b5f-36a1-423b-add1-80f50f07361a",
   "metadata": {},
   "source": [
    "Update required packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb80bb63-817e-4098-a92d-19a3701d8958",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pip install -r requirements.in --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccd976ef-d89d-494b-9c84-84120bf963b9",
   "metadata": {},
   "source": [
    "Set analysis parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4729d261-6b27-4a61-a937-366c5174ce05",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "study_area = 64\n",
    "raster_version = 'testing2023'\n",
    "vector_version = 'testing2023'\n",
    "continental_version = 'testing2023'\n",
    "config_path = 'configs/dea_coastlines_config.yaml'\n",
    "\n",
    "# study_area = 9\n",
    "# raster_version = 'development'\n",
    "# vector_version = 'development'\n",
    "# continental_version = 'development'\n",
    "# config_path = 'configs/dea_coastlines_config_development.yaml'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9906d6b-01a4-4c94-8dd0-178eb281bd6a",
   "metadata": {},
   "source": [
    "### Run DEA Coastlines analysis\n",
    "#### Run tidally-constrained raster generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dc5632c4-f872-46e5-9762-80b8556f3622",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: python -m coastlines.raster [OPTIONS]\n",
      "\n",
      "Options:\n",
      "  --config_path TEXT              Path to the YAML config file defining inputs\n",
      "                                  to use for this analysis. These are\n",
      "                                  typically located in the `dea-\n",
      "                                  coastlines/configs/` directory.  [required]\n",
      "  --study_area TEXT               A string providing a unique ID of an\n",
      "                                  analysis gridcell that will be used to run\n",
      "                                  the analysis. This should match a row in the\n",
      "                                  \"id\" column of the provided analysis\n",
      "                                  gridcell vector file.  [required]\n",
      "  --raster_version TEXT           A unique string proving a name that will be\n",
      "                                  used for output raster directories and\n",
      "                                  files. This can be used to version different\n",
      "                                  analysis outputs.  [required]\n",
      "  --start_year INTEGER            The first annual shoreline you wish to be\n",
      "                                  included in the final outputs. To allow low\n",
      "                                  data pixels to be gapfilled with additional\n",
      "                                  satellite data from neighbouring years, the\n",
      "                                  full timeseries of satellite data loaded in\n",
      "                                  this step will include one additional year\n",
      "                                  of preceding satellite data (i.e. if\n",
      "                                  `--start_year 2000`, satellite data from\n",
      "                                  1999 onward will be loaded for gapfilling\n",
      "                                  purposes). Because of this, we recommend\n",
      "                                  that at least one year of satellite data\n",
      "                                  exists in your datacube prior to\n",
      "                                  `--start_year`.\n",
      "  --end_year INTEGER              The final annual shoreline you wish to be\n",
      "                                  included in the final outputs. To allow low\n",
      "                                  data pixels to be gapfilled with additional\n",
      "                                  satellite data from neighbouring years, the\n",
      "                                  full timeseries of satellite data loaded in\n",
      "                                  this step will include one additional year\n",
      "                                  of ensuing satellite data (i.e. if\n",
      "                                  `--end_year 2020`, satellite data up to and\n",
      "                                  including 2021 will be loaded for gapfilling\n",
      "                                  purposes). Because of this, we recommend\n",
      "                                  that at least one year of satellite data\n",
      "                                  exists in your datacube after `--end_year`.\n",
      "  --tide_centre FLOAT             The central tide height used to compute the\n",
      "                                  min and max tide height cutoffs. Tide\n",
      "                                  heights will be masked so all satellite\n",
      "                                  observations are approximately centred over\n",
      "                                  this value. The default is 0.0 which\n",
      "                                  represents 0 m Above Mean Sea Level.\n",
      "  --buffer FLOAT                  The distance (in degrees) to buffer the\n",
      "                                  study area grid cell extent. This buffer is\n",
      "                                  important for ensuring that generated\n",
      "                                  rasters overlap along the boundaries of\n",
      "                                  neighbouring study areas so that we can\n",
      "                                  extract seamless vector shorelines. Defaults\n",
      "                                  to 0.05 degrees, or roughly 5 km at the\n",
      "                                  equator.\n",
      "  --aws_unsigned / --no-aws_unsigned\n",
      "                                  Whether to use sign AWS requests for S3\n",
      "                                  access\n",
      "  --overwrite / --no-overwrite    Whether to overwrite tiles with existing\n",
      "                                  outputs, or skip these tiles entirely.\n",
      "  --help                          Show this message and exit.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python -m coastlines.raster --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fca95348-eda9-4071-a9a1-a3f9219c5cf0",
   "metadata": {},
   "source": [
    "Example analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47397eb8-ddec-46c6-9f7c-431a50bb7eae",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m coastlines.raster --config_path {config_path} --study_area {study_area} --raster_version {raster_version} --start_year 1988 --end_year 2023"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e0f60ac-c68d-4b6f-a19b-257679588ea1",
   "metadata": {},
   "source": [
    "#### Run vector annual shoreline and rates of change statistics generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0ed72c21-ba89-4fad-9521-f5493aad9d66",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: python -m coastlines.vector [OPTIONS]\n",
      "\n",
      "Options:\n",
      "  --config_path TEXT              Path to the YAML config file defining inputs\n",
      "                                  to use for this analysis. These are\n",
      "                                  typically located in the `dea-\n",
      "                                  coastlines/configs/` directory.  [required]\n",
      "  --study_area TEXT               A string providing a unique ID of an\n",
      "                                  analysis gridcell that was previously used\n",
      "                                  to generate raster files. This is used to\n",
      "                                  identify the raster files that will be used\n",
      "                                  as inputs for shoreline extraction, and\n",
      "                                  should match a row in the \"id\" column of the\n",
      "                                  provided analysis gridcell vector file.\n",
      "                                  [required]\n",
      "  --raster_version TEXT           A unique string providing a name that was\n",
      "                                  used to generate raster files. This is used\n",
      "                                  to identify the raster files that will be\n",
      "                                  used as inputs for shoreline extraction.\n",
      "                                  [required]\n",
      "  --vector_version TEXT           A unique string proving a name that will be\n",
      "                                  used for output vector directories and\n",
      "                                  files. This allows multiple versions of\n",
      "                                  vector files to be generated from the same\n",
      "                                  input raster data, e.g. for testing\n",
      "                                  different water index thresholds or indices.\n",
      "                                  If not provided, this will default to the\n",
      "                                  same string supplied to \"--raster_version\".\n",
      "  --water_index TEXT              A string giving the name of the computed\n",
      "                                  water index to use for shoreline extraction.\n",
      "                                  Defaults to \"mndwi\".\n",
      "  --index_threshold FLOAT         The water index threshold used to extract\n",
      "                                  subpixel precision shorelines. Defaults to\n",
      "                                  0.00.\n",
      "  --start_year INTEGER            The first annual shoreline to extract from\n",
      "                                  the input raster data.\n",
      "  --end_year INTEGER              The final annual shoreline to extract from\n",
      "                                  the input raster data.\n",
      "  --baseline_year INTEGER         The annual shoreline used as a baseline from\n",
      "                                  which to generate the rates of change point\n",
      "                                  statistics. This is typically the most\n",
      "                                  recent annual shoreline in the dataset (i.e.\n",
      "                                  the same as `--end_year`).\n",
      "  --aws_unsigned / --no-aws_unsigned\n",
      "                                  Whether to use sign AWS requests for S3\n",
      "                                  access\n",
      "  --overwrite / --no-overwrite    Whether to overwrite tiles with existing\n",
      "                                  outputs, or skip these tiles entirely.\n",
      "  --help                          Show this message and exit.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python -m coastlines.vector --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dede36a-c26b-4d98-a359-7080dc3f2fd9",
   "metadata": {},
   "source": [
    "Example analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c23665e0-6d46-4185-b9f0-04a50d856174",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m coastlines.vector --config_path {config_path} --study_area {study_area} --raster_version {raster_version} --start_year 1988 --end_year 2023 --baseline_year 2023"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa26bb5f-c618-418d-bc87-c01a33fd612c",
   "metadata": {},
   "source": [
    "#### Run continental-scale layer generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "518bf214-ce70-4baa-a2ad-28302986fe01",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: python -m coastlines.continental [OPTIONS]\n",
      "\n",
      "Options:\n",
      "  --vector_version TEXT           A unique string proving a name that was used\n",
      "                                  for output vector directories and files.\n",
      "                                  This is used to identify the tiled annual\n",
      "                                  shoreline and rates of change layers that\n",
      "                                  will be combined into continental-scale\n",
      "                                  layers.  [required]\n",
      "  --continental_version TEXT      A unique string proving a name that will be\n",
      "                                  used for output continental-scale layers.\n",
      "                                  This allows multiple versions of\n",
      "                                  continental-scale layers to be generated\n",
      "                                  from the same input vector data, e.g. for\n",
      "                                  testing different hotspot of coastal change\n",
      "                                  summary layers. If not provided, this will\n",
      "                                  default to the string provided to \"--\n",
      "                                  vector_version\".\n",
      "  --shorelines BOOLEAN            A boolean indicating whether to combine\n",
      "                                  tiled annual shorelines layers into a single\n",
      "                                  continental-scale annual shorelines layer.\n",
      "  --ratesofchange BOOLEAN         A boolean indicating whether to combine\n",
      "                                  tiled rates of change statistics layers into\n",
      "                                  a single continental-scale rates of change\n",
      "                                  statistics layer.\n",
      "  --hotspots BOOLEAN              A boolean indicating whether to generate\n",
      "                                  continental-scale coastal change hotspot\n",
      "                                  summary layers (this can be very slow\n",
      "                                  depending on the radii requested).\n",
      "  --hotspots_radius INTEGER       The distance (in metres) used to generate\n",
      "                                  coastal change hotspots summary layers. This\n",
      "                                  controls the spacing of each summary point,\n",
      "                                  and the radius used to aggregate rates of\n",
      "                                  change statistics around each point. The\n",
      "                                  default generates three hotspot layers with\n",
      "                                  radii 10000 m, 5000 m and 1000 m. To specify\n",
      "                                  multiple custom radii, repeat this argument,\n",
      "                                  e.g. `--hotspots_radius 1000\n",
      "                                  --hotspots_radius 5000`.\n",
      "  --baseline_year INTEGER         The annual shoreline used to generate the\n",
      "                                  hotspot summary points. This is typically\n",
      "                                  the most recent annual shoreline in the\n",
      "                                  dataset.\n",
      "  --shapefiles BOOLEAN            A boolean indicating whether to export\n",
      "                                  outputs as a zipped ESRI Shapefile dataset\n",
      "                                  (this can be slow depending on the size of\n",
      "                                  the analysis).\n",
      "  --include-styles / --no-include-styles\n",
      "                                  Set this to indicate whether to include\n",
      "                                  styles in the output OGC GeoPackage file.\n",
      "  --help                          Show this message and exit.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!python -m coastlines.continental --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51249ca9-35ca-4baa-b6a1-67f149a6fef0",
   "metadata": {},
   "source": [
    "Example analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22e3dca9-6a5e-4e6a-ac2c-c3f3b1ae904b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m coastlines.continental --vector_version {vector_version} --continental_version {continental_version} --shorelines True --ratesofchange True --hotspots True --baseline_year 2023"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe55ab27-807a-44c8-9408-595934893faa",
   "metadata": {},
   "source": [
    "## Example combined analysis\n",
    "This demonstrates how the three components of DEA Coastlines (raster generation, vector generation and continental layers generation) can be applied automatically to a sequence of input study area grid cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b804fec-39f4-48a9-8525-a78fbf06e084",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Study areas\n",
    "study_areas = [283]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c39e6533-f4fd-4a3c-90a5-44f19825c2f5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Run raster and vector generation for each study area\n",
    "for study_area in study_areas:\n",
    "    print(study_area)\n",
    "    !python -m coastlines.raster --config_path {config_path} --study_area {study_area} --raster_version {raster_version} --start_year 1988 --end_year 2022\n",
    "    !python -m coastlines.vector --config_path {config_path} --study_area {study_area} --raster_version {raster_version} --vector_version {vector_version} --start_year 1988 --end_year 2022 --baseline_year 2022\n",
    "\n",
    "# When complete, combine into single continental outputs\n",
    "!python -m coastlines.continental --vector_version {vector_version} --continental_version {continental_version} --shorelines True --ratesofchange True --hotspots True --baseline_year 2022"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f83a811-d875-4c27-9f9e-3e644119b49f",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Clean up and remove files\n",
    "Warning: this will delete all data processed with a matching unique version name. Only run these cells if you no longer need these processed outputs!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "585af34b-9cde-4ff2-8fa5-5c2baaab2e79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# rm -rf data/interim/raster/{raster_version}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3069daf-5d71-4c70-8740-d02efb1fc0f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# rm -rf data/interim/vector/{vector_version}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38d2209f-e978-4735-835b-68381358517a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# rm -rf data/processed/{continental_version}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d560564-e485-4a02-8f19-0dea1ba42bd3",
   "metadata": {
    "tags": []
   },
   "source": [
    "***\n",
    "\n",
    "## Additional information"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "457f851c-3b51-4c80-8430-d893fdbd05ec",
   "metadata": {},
   "source": [
    "**License:** The code in this notebook is licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). \n",
    "Digital Earth Australia data is licensed under the [Creative Commons by Attribution 4.0](https://creativecommons.org/licenses/by/4.0/) license.\n",
    "\n",
    "**Contact:** For assistance with any of the Python code or Jupyter Notebooks in this repository, please post a [Github issue](https://github.com/GeoscienceAustralia/dea-coastlines/issues/new).\n",
    "\n",
    "**Last modified:** July 2023\n",
    "\n",
    "**To cite:**\n",
    "\n",
    "> Bishop-Taylor, R., Nanson, R., Sagar, S., Lymburner, L. (2021). Mapping Australia's dynamic coastline at mean sea level using three decades of Landsat imagery. Remote Sensing of Environment, 267, 112734. Available: https://doi.org/10.1016/j.rse.2021.112734\n",
    ">\n",
    "> Nanson, R., Bishop-Taylor, R., Sagar, S., Lymburner, L., (2022). Geomorphic insights into Australia's coastal change using a national dataset derived from the multi-decadal Landsat archive. Estuarine, Coastal and Shelf Science, 265, p.107712. Available: https://doi.org/10.1016/j.ecss.2021.107712\n",
    ">\n",
    "> Bishop-Taylor, R., Sagar, S., Lymburner, L., Alam, I., Sixsmith, J. (2019). Sub-pixel waterline extraction: characterising accuracy and sensitivity to indices and spectra. Remote Sensing, 11 (24):2984. Available: https://doi.org/10.3390/rs11242984"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}