planet_API_101_BoulderCreekAugust2020.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qwDLxTfEPIK5"
      },
      "source": [
        "## Analyzing the 2020 Boulder Creek Fires with PlanetScope Imagery\n",
        "\n",
        "A Python notebook that provides a step-by-step guide for beginners to examine the devastating wildfires that occurred in Boulder Creek in August 2020 using satellite imagery from PlanetScope.\n",
        "\n",
        "The notebook covers various aspects of the analysis, including searching, filtering, ordering, and visualizing the satellite data. By following this guide, beginners can gain insights into the extent and severity of the fires, aiding in recovery efforts, evaluating firefighting strategies, and planning for future wildfire mitigation.\n",
        "\n",
        "This notebook is based on a workshop given by Planet.com's Developers for GISDay@Stanford 2023, with a focus on adapting the analysis to the specific context of the 2020 Boulder Creek Fires. [workshop given by Planet.com's Developers for GISDay@Stanford 2023](https://hello.planet.com/data/s/sdtjXQAPQpmBBbc)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOavDbXrPIK8"
      },
      "source": [
        "This tutorial is an introduction to [Planet](https://www.planet.com)'s Data and Orders API using the official [Python client](https://github.com/planetlabs/planet-client-python), the `planet` module. It shows you how to create bulk orders, use Planet Labs' tools, and deliver to the cloud.\n",
        "\n",
        "## Requirements\n",
        "\n",
        "This tutorial assumes familiarity with the [Python](https://python.org) programming language throughout. Python modules used in this tutorial are:\n",
        "* [IPython](https://ipython.org/) and [Jupyter](https://jupyter.org/)\n",
        "* [planet](https://github.com/planetlabs/planet-client-python)\n",
        "* [geojsonio](https://pypi.python.org/pypi/geojsonio)\n",
        "* [rasterio](https://rasterio.readthedocs.io/en/latest/index.html)\n",
        "* [asyncio](https://docs.python.org/3/library/asyncio.html)\n",
        "\n",
        "You should also have an account on the Planet Platform and retrieve your API key from your [account page](https://www.planet.com/account/).\n",
        "\n",
        "## Useful links\n",
        "* [Planet Client V2 Documentation](https://github.com/planetlabs/planet-client-python)\n",
        "* [Planet Data API reference](https://developers.planet.com/docs/apis/data/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SdbGGuaOPIK9"
      },
      "source": [
        "This tutorial will cover the basic operations possible with the Python client, particularly those that interact with the Data API and Orders API"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6zGyVciEPIK-"
      },
      "source": [
        "## Set up\n",
        "\n",
        "In order to interact with the Planet API using the client, we need to import the necessary packages & define helper functions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xQu8CYqHPIK-"
      },
      "outputs": [],
      "source": [
        "#general packages\n",
        "import os\n",
        "import json\n",
        "import glob\n",
        "import asyncio\n",
        "import requests\n",
        "import nest_asyncio\n",
        "import matplotlib.pyplot as plt\n",
        "from requests.auth import HTTPBasicAuth\n",
        "from datetime import datetime, timedelta\n",
        "from google.colab import userdata\n",
        "\n",
        "#geospatial packages\n",
        "import rasterio\n",
        "import geopandas as gpd\n",
        "from shapely.geometry import shape\n",
        "from shapely.ops import unary_union\n",
        "import folium\n",
        "\n",
        "\n",
        "#planet SDK\n",
        "from planet import Auth, reporting, Session, OrdersClient, order_request, data_filter\n",
        "\n",
        "\n",
        "# We will also create a small helper function to print out JSON with proper indentation.\n",
        "def indent(data):\n",
        "    print(json.dumps(data, indent=2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LFYB8G1hQV4M"
      },
      "source": [
        "# Installing packages\n",
        "\n",
        "You probably encountered some errors after running that cell. Here, we will install the missing packages. If any other packages were missing from your Python Environment, they should be listed in the error messages, and you can simply add them to the end of the `!pip` command line to add them to the install sequence.\n",
        "\n",
        "Run the following code cell, then return to the previous cell and try it again.\n",
        "\n",
        "Continue to install missing packages, using the `!pip install` command, until you no longer recieve error messages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yXINvoPSQEaZ"
      },
      "outputs": [],
      "source": [
        "# You can install packages that aren't currently installed in your Python Notebook using !pip install <package name>\n",
        "# In this case, we will install the Planet Package:\n",
        "!pip install rasterio planet folium"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j9oyYUMTFCaU"
      },
      "source": [
        "## Authentication"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-MU1LNz9PIK_"
      },
      "source": [
        "Here, you will paste your API Key when prompted. It will be used to authenticate when ordering data.\n",
        "\n",
        "Be sure to go to **Edit>Clear all outputs** to clear the console output that results, before sharing this notebook, or uploading it to a public repository, such as GitHub.\n",
        "\n",
        "Additionally, regularly resetting your API Key on Planet.com can help keep your account and access secure.\n",
        "\n",
        "You can also authenticate via the CLI using [`auth init`](https://planet-sdk-for-python-v2.readthedocs.io/en/latest/cli/cli-reference/?h=auth#auth:~:text=message%20and%20exit.-,auth,-%C2%B6), this will store your API key as an environment variable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vp2KqqPoPILA"
      },
      "outputs": [],
      "source": [
        "# if your Planet API Key is not set as an environment variable, you can paste it below\n",
        "if 'PL_API_KEY' in os.environ:\n",
        "    API_KEY = os.environ['PL_API_KEY']\n",
        "else:\n",
        "    API_KEY = input(\"PASTE_API_KEY_HERE AND HIT RETURN:   \")\n",
        "    os.environ['PL_API_KEY'] = API_KEY\n",
        "\n",
        "client = Auth.from_key(API_KEY)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Using Google Colab Secret Manager"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from google.colab import secrets\n",
        "\n",
        "# Get the API key from the secret manager\n",
        "API_KEY = secrets.get_secret('planet_api')\n",
        "\n",
        "# Set the API key as an environment variable\n",
        "os.environ['PL_API_KEY'] = API_KEY\n",
        "\n",
        "# Create a client for the Planet API\n",
        "client = Auth.from_key(API_KEY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y3Md-d7CPILC"
      },
      "source": [
        "## August 2020 Boulder Creek fires\n",
        "\n",
        "The Boulder Creek fires in August 2020 exemplify the devastating impact of wildfires on communities, ecosystems, and resources. This event, characterized by rapid spread and extensive damage, highlighted the urgent necessity for enhanced monitoring, management, and recovery strategies in the face of increasingly frequent and severe wildfires. The utilization of high-resolution satellite imagery, such as that provided by PlanetScope, is pivotal in this context. It offers a detailed assessment of the affected areas, enabling better understanding and documentation of the fire's extent and severity. By integrating this imagery analysis within a Python notebook, researchers and disaster response teams can efficiently process and interpret data to support recovery efforts, evaluate the effectiveness of firefighting strategies, and plan for future wildfire mitigation. This analytical approach not only aids in immediate response and recovery but also contributes to long-term resilience and preparedness, underscoring the critical role of technology in addressing environmental challenges.\n",
        "\n",
        "## Searching an Area of Interest (AOI)\n",
        "\n",
        "This code reads a **GeoJSON** file named `\"lakelagunita.geojson\"`\n",
        "into a Python object using the `json.loads(f.read())` method, which converts the file's JSON content into a Python dictionary. It then accesses the `'features'` list from this dictionary, selects the first feature (`[0]` indicating the first item in the list), and retrieves the `'geometry'` information associated with that feature. Essentially, it extracts the geometric data (such as coordinates defining points, lines, or polygons) of the first geographic feature described in the GeoJSON file.\n",
        "\n",
        "The `geom_all` variable stores the geometry of the first feature from the file, demonstrating how to access nested data in JSON format. The `geom_large` dictionary defines a new geometry directly in the code, illustrating how to construct a GeoJSON feature programmatically. You could use either of these methods for providing the **Area of Interest (AOI)**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AzBisAirPILC"
      },
      "source": [
        "Let's also read in a GeoJSON geometry into a variable so we can use it during testing. *The geometry can only have one polygon to work with the data API*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6e7dyoerPILD"
      },
      "outputs": [],
      "source": [
        "with open(\"BoulderCreek.geojson\") as f:\n",
        "    geom_all = json.loads(f.read())['features'][0]['geometry']\n",
        "geom_large = {\n",
        "      \"type\": \"Feature\",\n",
        "      \"properties\": {},\n",
        "      \"geometry\": {\n",
        "        \"coordinates\": [\n",
        "          [\n",
        "            [\n",
        "              -122.22305932516572,\n",
        "              37.195384463702354\n",
        "            ],\n",
        "            [\n",
        "              -122.22305932516572,\n",
        "              37.040779763332225\n",
        "            ],\n",
        "            [\n",
        "              -121.95486734792922,\n",
        "              37.040779763332225\n",
        "            ],\n",
        "            [\n",
        "              -121.95486734792922,\n",
        "              37.195384463702354\n",
        "            ],\n",
        "            [\n",
        "              -122.22305932516572,\n",
        "              37.195384463702354\n",
        "            ]\n",
        "          ]\n",
        "        ],\n",
        "        \"type\": \"Polygon\"\n",
        "      }\n",
        "    }\n",
        "print(geom_all)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NZOAVnEcta_N"
      },
      "source": [
        "## Display the AOI\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kbjxpKaztcS3"
      },
      "outputs": [],
      "source": [
        "m = folium.Map([37.4264632718164,-122.1828114547462], zoom_start=15, tiles=\"cartodbpositron\")\n",
        "\n",
        "geojson_data = 'BoulderCreek.geojson'\n",
        "\n",
        "folium.GeoJson(geojson_data, name=\"Boulder Creek\").add_to(m)\n",
        "\n",
        "folium.LayerControl().add_to(m)\n",
        "\n",
        "m"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uwQyT1ZCPILD"
      },
      "source": [
        "## Creating a Filter"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8X8AetVPILD"
      },
      "source": [
        "This code block sets up filters for querying geospatial data:\n",
        "\n",
        "- **Item Types**: Defines the type of satellite imagery to search for, in this case, \"PSScene\".\n",
        "- **Geometry Filter**: Uses a predefined geometry (`geom_large`) to filter data to only include imagery that intersects with this area.\n",
        "- **Date Range Filter**: Specifies a date range to filter the imagery, selecting images acquired between December 10, 2022, and September 30, 2023.\n",
        "- **Combined Filter**: Combines the geometry and date range filters using an \"AND\" logic, meaning both conditions must be met for an image to be included in the search results.\n",
        "\n",
        "The cloud cover filter is *commented out*, indicating it's not currently used but can be applied to restrict results to images with less than 80% cloud cover.\n",
        "\n",
        "The possible filters include `and_filter`, `date_range_filter`, `range_filter` and so on, mirroring the options supported by the Planet API. Additional filters are described [here](https://planet-sdk-for-python-v2.readthedocs.io/en/latest/python/sdk-guide/#filter:~:text=(main())-,Filter,-%C2%B6)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S3E5I4K2PILD"
      },
      "outputs": [],
      "source": [
        "# Define the filters we'll use to find our data\n",
        "\n",
        "item_types = [\"PSScene\"]\n",
        "\n",
        "#Geometry filter\n",
        "geom_filter = data_filter.geometry_filter(geom_large)\n",
        "\n",
        "#Date range filter\n",
        "date_range_filter = data_filter.date_range_filter(\n",
        "    \"acquired\", gt = datetime(month=08, day=01, year=2020),\n",
        "    lt = datetime(month=12, day=30, year=2020))\n",
        "#Cloud cover filter\n",
        "#cloud_cover_filter = data_filter.range_filter('clear_percent', gt = 80)\n",
        "\n",
        "#Combine all of the filters\n",
        "combined_filter = data_filter.and_filter([geom_filter, date_range_filter])#, cloud_cover_filter])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9rizbiFQEg-i"
      },
      "source": [
        "## Print the `combined_filter`\n",
        "\n",
        "Print the filter so you can see what the results look like"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R2t-AfQDPILE",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "combined_filter"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aJ87JbeNPILE"
      },
      "source": [
        "## Set the `item_types`\n",
        "\n",
        "This code block is setting up an API request object for querying a geospatial data service. It specifies the type of data to search for, \"PSScene\", using the \"item_types\" key. Additionally, it includes a \"filter\" key that incorporates a previously defined combined_filter, which likely combines several criteria (like geographic area, date range, etc.) to narrow down the search results. This request object can then be used with the service's API to fetch data that matches the given criteria.\n",
        "\n",
        "* \"PSScene\", Planetscope Scenes  \n",
        "* \"REOrthoTile\" for RapidEye OrthoTiles,   \n",
        "* \"REScene\" for unorthorectified RapidEye strips,  \n",
        "* \"SkySatScene\" for SkySat imagery,  \n",
        "* \"SkySatCollect\" for orthorectified SkySat composites  \n",
        "\n",
        "Additional `item_types` can be found at https://developers.planet.com/docs/apis/data/items-assets/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w0FSGwrpPILE"
      },
      "outputs": [],
      "source": [
        "item_type = \"PSScene\"\n",
        "\n",
        "# API request object\n",
        "search_request = {\n",
        "  \"item_types\": [item_type],\n",
        "  \"filter\": combined_filter\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JglD5K4YIoAQ"
      },
      "outputs": [],
      "source": [
        "search_request"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0vXcs31VKq74"
      },
      "source": [
        "## POST to the Planet API\n",
        "\n",
        "This code sends a POST request to the Planet API to search for images matching specific criteria defined in search_request. It uses basic authentication with an API key. The response, assumed to contain image data, is parsed from JSON to extract and print the number of image IDs found, showing how many images matched the search filters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bco5z_1tPILE"
      },
      "outputs": [],
      "source": [
        "# fire off the POST request\n",
        "search_result = \\\n",
        "  requests.post(\n",
        "    'https://api.planet.com/data/v1/quick-search',\n",
        "    auth=HTTPBasicAuth(API_KEY, ''),\n",
        "    json=search_request)\n",
        "\n",
        "# extract image IDs only\n",
        "image_ids = [feature['id'] for feature in search_result.json()['features']]\n",
        "print(len(image_ids))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lWk6ndQXPILF"
      },
      "source": [
        "## Pagination links\n",
        "\n",
        "This code accesses the pagination links from the JSON response of a search query made to an API.\n",
        "\n",
        "Pagination is used to break down large datasets into smaller, manageable chunks or \"pages\" of data. `_links` would typically contain URLs to navigate through these pages, allowing the client to request subsequent sets of results (like \"next\" page or \"previous\" page) without retrieving all data at once. This is efficient for both the server and client, especially when dealing with large amounts of data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O589UzIcPILF"
      },
      "outputs": [],
      "source": [
        "search_result.json()['_links']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FNWfkCYTPILF"
      },
      "source": [
        "# Using the SDK\n",
        "\n",
        "Using the REST API directly requires manually crafting HTTP requests, handling authentication, parsing responses, and managing session states, offering more control and flexibility but requiring more code to handle the interaction.\n",
        "\n",
        "In contrast, using the SDK (Software Development Kit) abstracts and simplifies the process of interacting with the API by providing pre-built functions and methods, handling low-level details like session management and request retries. It allows for more Pythonic code, with asynchronous capabilities and direct integration into Python applications.\n",
        "\n",
        "This code performs an asynchronous search request using the Planet SDK, retrieving a list of items that match the specified combined_filter and item_types criteria. It uses an asynchronous session to make the request and asynchronously iterates over the search results up to a limit of 500 items, gathering them into a list called `item_list`. This approach allows for non-blocking network requests, making the code efficient for handling I/O-bound tasks like web requests in a concurrent manner.\n",
        "\n",
        "If the number of items requested is more than 500, the client will automatically fetch more pages of results in order to get the exact number requested.\n",
        "\n",
        "Then we can save the output to be visualized as a geojson"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Se1tGqQnPILF"
      },
      "outputs": [],
      "source": [
        "async with Session() as sess:\n",
        "    cl = sess.client('data')\n",
        "    item_list = [i async for i in cl.search(search_filter=combined_filter, item_types=item_types,limit=500)]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PvP4h3DEPILF"
      },
      "source": [
        "## Print the # of items in your Search Result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ay06VX_sPILF"
      },
      "outputs": [],
      "source": [
        "len(item_list)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W1iR6YNpR6RX"
      },
      "source": [
        "## Adding our previous Filters\n",
        "\n",
        "This code block sets up a cloud cover filter to only include images with greater than 80% clarity, combines it with other filters (geometric and date range), and then performs an asynchronous search with the Planet API to retrieve up to 500 items matching these criteria. It uses an asynchronous session for efficient network operations, collecting the search results into a list named item_list."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8yeI1ljNPILF"
      },
      "outputs": [],
      "source": [
        "#Cloud cover filter\n",
        "cloud_cover_filter = data_filter.range_filter('clear_percent', gt = 80)\n",
        "\n",
        "#Combine all of the filters\n",
        "combined_filter = data_filter.and_filter([geom_filter, date_range_filter, cloud_cover_filter])\n",
        "\n",
        "async with Session() as sess:\n",
        "    cl = sess.client('data')\n",
        "    item_list = [i async for i in cl.search(search_filter=combined_filter, item_types=item_types,limit=500)]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "euMOtjkkUefH"
      },
      "source": [
        "## Print the newly filtered # of items in your Search Result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5mgDpRJjPILF"
      },
      "outputs": [],
      "source": [
        "len(item_list)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8pdUxWqFPILG"
      },
      "source": [
        "## Print the Search results\n",
        "\n",
        "Now, we can iterate through our search results.\n",
        "\n",
        "This code iterates through the list of items, `item_list`, and prints each item's ID and item type. It accesses the 'id' directly from each item dictionary and the 'item_type' from the nested 'properties' dictionary within each item."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5yeF4C-KPILG",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "for item in item_list:\n",
        "    print(item['id'], item['properties']['item_type'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nBRml7HcPILG"
      },
      "source": [
        "## Save all of our scene footprints as a GeoJSON file\n",
        "\n",
        "This code block creates a **GeoJSON** file representing a collection of spatial features (like satellite images or scenes) from item_list. It first checks if an 'output' directory exists, creating it if not. If a file named `'results01.geojson'` already exists in this directory, it deletes the file. Then, it iterates over item_list, constructing GeoJSON feature objects for each item by including their geometry and properties, and appends these to a feature collection. Finally, it writes this collection as a GeoJSON string to 'results02.geojson'."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6X5yn38TPILG"
      },
      "outputs": [],
      "source": [
        "scene_geoms = {\n",
        "  \"type\": \"FeatureCollection\",\n",
        "  \"features\": []\n",
        "}\n",
        "\n",
        "if not os.path.isdir('output'):\n",
        "    os.mkdir('output')\n",
        "else:\n",
        "    if os.path.isfile('output/results01.geojson'):\n",
        "        os.remove('output/results01.geojson')\n",
        "\n",
        "with open('output/results01.geojson','w') as f:\n",
        "    for item in item_list:\n",
        "        geom_out =     {\n",
        "          \"type\": \"Feature\",\n",
        "          \"properties\": item['properties'],\n",
        "          \"geometry\": item['geometry']\n",
        "        }\n",
        "        scene_geoms['features'].append(geom_out)\n",
        "    jsonStr = json.dumps(scene_geoms)\n",
        "    f.write(jsonStr)\n",
        "    f.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T_s07_s1xub0"
      },
      "source": [
        "# Display the image footprints"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y5lu486ex0Yo"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "\n",
        "m = folium.Map([37.4264632718164,-122.1828114547462], zoom_start=10, tiles=\"cartodbpositron\")\n",
        "\n",
        "geojson_data = './output/results01.geojson'\n",
        "\n",
        "folium.GeoJson(geojson_data, name=\"Image Footprints\").add_to(m)\n",
        "\n",
        "folium.LayerControl().add_to(m)\n",
        "\n",
        "m"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_VMx-FlzV9R8"
      },
      "source": [
        "# Examine an Item\n",
        "\n",
        "The code `item_list[0]` accesses the first item in the list named `item_list`, expected to contain data about geospatial features, specifically from the Planet API. The output shown is a Python dictionary representing a geospatial feature, including links to its data (`_links`), permissions available (`_permissions`), a list of `assets`, the `geometry` defining its spatial footprint, a unique identifier (`id`), and various properties such as a`cquisition time`, `cloud cover`, and more, related to the satellite image.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I5qAOaKFPILG"
      },
      "outputs": [],
      "source": [
        "item_list[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TkJlOgRYPILG"
      },
      "source": [
        "## Ordering\n",
        "\n",
        "Searching using the Data API involves querying the Planet database to find satellite images that match specific criteria (like date, location, and cloud cover). It's about discovering what data is available.\n",
        "\n",
        "Ordering using the Orders API, on the other hand, is the next step after identifying the desired images. It involves requesting the processing and delivery of specific datasets, possibly with additional operations like format conversion or applying specific filters, to make the data ready for analysis or integration into applications."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A2ep8YUAPILG"
      },
      "source": [
        "Now that we have all of the imagery that we want to order we need to package it in a way that the Orders API can handle. Breaking it up by week."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4MJqRf2yWlbR"
      },
      "source": [
        "This code organizes a list of items (each representing a satellite image with an acquisition date) into groups based on their acquisition dates, with each group covering a span of 30 days.\n",
        "\n",
        "It first sorts the items by date in ascending order.\n",
        "\n",
        "Then, it iterates through these items, grouping them together if they fall within the same 30-day period.\n",
        "\n",
        "If an item's date is outside the current 30-day window, it starts a new group.\n",
        "\n",
        "Finally, it prints the number of these groups."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XJ8AD767PILG"
      },
      "outputs": [],
      "source": [
        "grouped_items = []\n",
        "current_group = []\n",
        "#reverse the list since it comes in last date first\n",
        "reversed_items = sorted(item_list, key=lambda item: item['properties']['acquired'])\n",
        "\n",
        "#Select the earliest item\n",
        "group_start_date = datetime.strptime(reversed_items[0]['properties']['acquired'], \"%Y-%m-%dT%H:%M:%S.%fZ\")\n",
        "\n",
        "for item in reversed_items:\n",
        "    time_object = item['properties']['acquired']\n",
        "    time = datetime.strptime(time_object, \"%Y-%m-%dT%H:%M:%S.%fZ\")\n",
        "\n",
        "    if time < group_start_date + timedelta(days=7):\n",
        "        current_group.append(item)\n",
        "\n",
        "    else:\n",
        "        grouped_items.append(current_group)\n",
        "        current_group = [item]\n",
        "        group_start_date = time\n",
        "if current_group:\n",
        "    grouped_items.append(current_group)\n",
        "\n",
        "print(len(grouped_items))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Xs3VJDNPILG"
      },
      "source": [
        "## Querying a property of our image groups\n",
        "\n",
        "Lets see what the cloud cover is for our scenes\n",
        "\n",
        "This code iterates over the first group of satellite images in `grouped_items` and prints the `clear_percent` property for each image. The clear_percent indicates the percentage of the image not obscured by clouds, providing insight into the image's clarity and suitability for analysis or visual inspection.\n",
        "\n",
        "The output represents the `clear_percent` values of satellite images from the first group in grouped_items. Each number indicates the percentage of the image area that is free from cloud cover, with higher numbers suggesting clearer conditions. The values range from 81% to 100%, indicating varying levels of clarity across the images, with several images having very high clarity (99% to 100%)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FWb54ziaPILG"
      },
      "outputs": [],
      "source": [
        "for item in grouped_items[0]:\n",
        "    print(item['properties']['clear_percent'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kE6GHV8BPILG"
      },
      "source": [
        "# Sort the images to prioritize the clearer images\n",
        "\n",
        "This code sorts each group of satellite images within `grouped_items` by their `clear_percent` value in descending order, ensuring each group's most unobscured images are listed first. It then compiles these sorted groups into a new list, `sorted_items`. Finally, it prints the `clear_percent` values of all images in the first sorted group, displaying them from the highest to the lowest percentage of clarity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FVWtFN-ZPILH"
      },
      "outputs": [],
      "source": [
        "sorted_items = []\n",
        "for group in grouped_items:\n",
        "    sorted_group = sorted(group, key=lambda item: item['properties']['clear_percent'], reverse=True)\n",
        "    sorted_items.append(sorted_group)\n",
        "\n",
        "\n",
        "for item in sorted_items[0]:\n",
        "    print(item['properties']['clear_percent'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N_vjXNUFYZ9M"
      },
      "source": [
        "## A function to calculate intersection\n",
        "\n",
        "This code defines a function get_overlap that calculates the area of overlap between two geometries. It uses the shape function from the shapely.geometry module to convert the input geometries into shape objects. Then, it computes the intersection of these two shapes, which represents the overlapping area, and returns this intersection as the result."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lTIpvGDxPILH"
      },
      "outputs": [],
      "source": [
        "def get_overlap(geometry1, geometry2):\n",
        "    \"\"\"Calculate the area of overlap between two geometries.\"\"\"\n",
        "    shape1 = shape(geometry1)\n",
        "    shape2 = shape(geometry2)\n",
        "\n",
        "    # Compute the intersection of the two geometries.\n",
        "    intersection = shape1.intersection(shape2)\n",
        "\n",
        "    return intersection"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8_bGSeuxPILH"
      },
      "source": [
        "## Evaluating coverage\n",
        "\n",
        "Look for the minimum about on scenes to cover the entire AOI\n",
        "\n",
        "This code iterates through groups of satellite images (`sorted_items`), selecting a minimum set of images per group that collectively cover the target area (`geom_all`). For each image, it checks if its geometry overlaps with the target area. If there's an existing overlap (`intersection`), it calculates the union of the new and existing overlaps, adding the image to the weekly list if the union expands the covered area.\n",
        "\n",
        "The goal is to compile lists (`minimum_sorted_list`) of the fewest images needed to cover the target area each week, optimizing for spatial coverage and minimizing cloud cover.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nlHvr5H7PILH"
      },
      "outputs": [],
      "source": [
        "minimum_sorted_list = []\n",
        "\n",
        "\n",
        "for week_items in sorted_items:\n",
        "    intersection = False\n",
        "    weekly_minimum_list = []\n",
        "    for item in week_items:\n",
        "        #for each scene itterate through every geometry and check if it overlaps with the scene\n",
        "        overlap = get_overlap(geom_all, item['geometry'])\n",
        "        if intersection:\n",
        "            new_intersection = unary_union([overlap,intersection])\n",
        "\n",
        "            #If the new interseciton is bigger then the old then add the scene to the order\n",
        "            if round(new_intersection.area, 8) > round(intersection.area, 8):\n",
        "                intersection = new_intersection\n",
        "                weekly_minimum_list.append(item)\n",
        "        else:\n",
        "            if overlap.area > 0:\n",
        "                intersection = overlap\n",
        "                weekly_minimum_list.append(item)\n",
        "    print(len(week_items), \" to \", len(weekly_minimum_list))\n",
        "\n",
        "    if len(weekly_minimum_list) > 0:\n",
        "        minimum_sorted_list.append(weekly_minimum_list)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v7SdRm5QZZX0"
      },
      "source": [
        "## Evaluating cloud cover before and after\n",
        "\n",
        "This code block is comparing and displaying the clear_percent properties of satellite images from two different lists: `sorted_items[0]` and `minimum_sorted_list[0]`.\n",
        "\n",
        "The first loop prints the `clear_percent` of each image in the first sorted group, showing how clear each image is.\n",
        "\n",
        "After printing \"Now\", the second loop does the same for the first group of images that have been determined to minimally cover a target area, likely optimized for both coverage and clarity.\n",
        "\n",
        "This allows for a comparison of image clarity before and after optimization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NsrkmpY1PILH"
      },
      "outputs": [],
      "source": [
        "for item in sorted_items[0]:\n",
        "    print(item['properties']['clear_percent'])\n",
        "print(\"Now\")\n",
        "for item in minimum_sorted_list[0]:\n",
        "    print(item['properties']['clear_percent'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vRViS3ucPILH"
      },
      "source": [
        "## Clarity of the final selections\n",
        "\n",
        "Now lets print the average clear percent of each scene we are ordering.\n",
        "\n",
        "This code calculates and prints the average `clear_percent` for each group of satellite images in `minimum_sorted_list`.\n",
        "\n",
        "It iterates through each group, collecting the `clear_percent` values into a list, then computes the average of these values for the group, indicating the overall clarity of images selected for minimal coverage."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aEyUkgTLPILH"
      },
      "outputs": [],
      "source": [
        "for group in minimum_sorted_list:\n",
        "    clear = []\n",
        "    for item in group:\n",
        "        clear.append(int(item['properties']['clear_percent']))\n",
        "    print(sum(clear)/len(clear))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iPW1zrNjPILH"
      },
      "source": [
        "## Order images for mosaicking with clearer images on top\n",
        "\n",
        "We need to reverse the order of the scenes one more time because when mosaicing the last scene is stacked at the top.\n",
        "\n",
        "This code sorts each group of satellite images in `minimum_sorted_list` by their `clear_percent` in ascending order, to ensure that when these images are used in a mosaic, the clearest images (last in the sorted list) are placed on top.\n",
        "\n",
        "After sorting, it prints the `clear_percent` of each image in the first sorted group, demonstrating the order in which they will be layered in the mosaic, with lower clarity images at the bottom and higher clarity images on top."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wfZ1lwiGPILH"
      },
      "outputs": [],
      "source": [
        "order_items = []\n",
        "for group in minimum_sorted_list:\n",
        "    sorted_group = sorted(group, key=lambda item: item['properties']['clear_percent'])\n",
        "    order_items.append(sorted_group)\n",
        "\n",
        "for item in order_items[0]:\n",
        "    print(item['properties']['clear_percent'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2j6i6z4HPILH"
      },
      "source": [
        "# Place a Order\n",
        "Create the order structure using `planet` functions\n",
        "\n",
        "## Create an `assemble_order()' function and test it\n",
        "\n",
        "This code defines an asynchronous function assemble_order that constructs a request for ordering satellite imagery from the Planet API.\n",
        "\n",
        "It specifies the image IDs to be included in the order, applies a series of processing tools (clipping to a specified area, performing a band math operation, and creating a composite), and builds the order request with these specifications.\n",
        "\n",
        "The function then awaits this order assembly process with a test order name and a specific image ID, preparing the order request for submission."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ymL-34jcPILI"
      },
      "outputs": [],
      "source": [
        "async def assemble_order(name,item_ids):\n",
        "    products = [\n",
        "        order_request.product(item_ids, 'analytic_udm2', 'PSScene')\n",
        "    ]\n",
        "\n",
        "    clip = order_request.clip_tool(aoi=geom_all)\n",
        "    bandmath = order_request.band_math_tool(b1='(b2-b4)/(b2+b4)*100+100', pixel_type='8U')\n",
        "    composite = order_request.composite_tool()\n",
        "\n",
        "\n",
        "\n",
        "    tools = [clip,bandmath,composite]\n",
        "\n",
        "    request = order_request.build_request(\n",
        "        name, products=products, tools=tools)\n",
        "    return request\n",
        "\n",
        "request =  await assemble_order(\"test\",['20230207_180504_51_24b6'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DgDOXS54cFr7"
      },
      "source": [
        "`print(request)` statement will output the assembled order request details created by the assemble_order function.\n",
        "\n",
        "This will include the name of the order, specified products (image IDs with their types and item type), and the tools applied (clip, band math, and composite) along with their configurations.\n",
        "\n",
        "The result is a structured representation of the order that is ready to be submitted to the Planet API for processing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BoegQsHMPILI"
      },
      "outputs": [],
      "source": [
        "request"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LlCh5b4jPILI"
      },
      "source": [
        "## Create a function to order imagery\n",
        "\n",
        "This code defines an asynchronous function, do_order, which takes an order request as input, creates an order with the Planet API using an OrdersClient session, waits for the order to complete, and then downloads the ordered data to a directory named after the order.\n",
        "\n",
        "It handles the order creation, monitoring, and downloading process asynchronously, allowing for efficient management of potentially long-running network operations involved in ordering satellite imagery."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XZCBERnOPILI"
      },
      "outputs": [],
      "source": [
        "async def do_order(request):\n",
        "    async with Session() as sess:\n",
        "        cl = OrdersClient(sess)\n",
        "        #with reporting.StateBar(state='creating') as bar:\n",
        "        order = await cl.create_order(request)\n",
        "        #bar.update(state='created', order_id=order['id'])\n",
        "\n",
        "        await cl.wait(order['id'],max_attempts=0)#, callback=bar.update_state)\n",
        "        os.mkdir(request['name'])\n",
        "\n",
        "        # if we get here that means the order completed. Yay! Download the files.\n",
        "        await cl.download_order(order['id'],directory=request['name'])\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oXCsMgl1PILI"
      },
      "source": [
        "## Create all orders\n",
        "\n",
        "This code iterates over groups of satellite images (`order_items`), constructing a unique order name for each group based on a prefix and the acquisition date of the first image in each group.\n",
        "\n",
        "It then creates an order for each group of images by calling the `assemble_order()` function with the constructed order name and the IDs of the images in the group, appending the future object representing the asynchronous operation to `order_list`.\n",
        "\n",
        "Finally, it prints the total number of orders prepared."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ifFnaAvCPILI"
      },
      "outputs": [],
      "source": [
        "order_list = []\n",
        "folder_list= []\n",
        "name = \"lake_lagunita_cloud_\"\n",
        "for group in order_items:\n",
        "    ids = []\n",
        "    order_name = name + group[0]['properties']['acquired'][:10]\n",
        "    print(order_name)\n",
        "    folder_list.append(order_name)\n",
        "    for item in group:\n",
        "        ids.append(item['id'])\n",
        "    order_list.append(await assemble_order(order_name,ids))\n",
        "print(len(order_list))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t7zPFZhAd1RZ"
      },
      "source": [
        "## Submitting and monitoring the orders\n",
        "\n",
        "This code uses `asyncio` and `nest_asyncio` to run multiple asynchronous operations (`do_order` function calls for each order in `order_list`) in parallel within a single event loop, effectively managing concurrent execution.\n",
        "\n",
        "`nest_asyncio.apply()` makes it possible to overcome limitations when running `asyncio` in environments like Jupyter notebooks, which already run in an event loop.\n",
        "\n",
        "This method is used to efficiently process multiple orders simultaneously, reducing overall completion time compared to sequential execution.\n",
        "\n",
        "***NOTE: This code block will sometimes throw errors, even though orderes have been submitted successfully. Be patient, and watch your Files location, for the incoming images. You should ultimately end up with the same number of images, as groups, that you caluclated earlier.***"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DBTAsfpxPILI"
      },
      "outputs": [],
      "source": [
        "# asyncio, the Python package that provides the API to run and manage coroutines\n",
        "\n",
        "nest_asyncio.apply()\n",
        "\n",
        "#now all you need to do to have them run in parallel is to create an array of order requests\n",
        "async with Session() as sess:\n",
        "    tasks = [do_order(o) for o in order_list]\n",
        "    await asyncio.gather(*tasks)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCz6oQGtPILI"
      },
      "source": [
        "## Visualize the output!\n",
        "\n",
        "This code block searches for \"composite.tif\" files in subdirectories, sorts them, and then plots them on a grid of 4x4 subplots using `matplotlib`.\n",
        "\n",
        "Each image is opened with `rasterio`, read into an array, and displayed using imshow with a \"Greens\"  (Dark to Light Green) colormap. You can find more about matplotlib colormaps, [here](https://matplotlib.org/stable/users/explain/colors/colormaps.html).\n",
        "\n",
        "The title of each subplot is set to a date extracted from the filename. Axes are turned off for clarity, and the layout is adjusted for tight spacing. This is a way to visually compare satellite images or processed data tiles based on their acquisition dates."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L700Nf15PILJ"
      },
      "outputs": [],
      "source": [
        "files = []\n",
        "# for folder in folder_list:\n",
        "#     files.extend(glob.glob(folder+\"/*/composite.tif\"))\n",
        "\n",
        "files.extend(glob.glob(\"*/*/composite.tif\"))\n",
        "files.sort()\n",
        "nrow = 4\n",
        "ncol = 4\n",
        "\n",
        "f, axes = plt.subplots(nrow, ncol, figsize=(3*ncol, 3*nrow))\n",
        "for file, ax in zip(files, axes.flatten()):\n",
        "    with rasterio.open(file) as src:\n",
        "        arr = src.read()\n",
        "\n",
        "    ax.imshow(arr[0], cmap=\"Greens\")\n",
        "\n",
        "\n",
        "    date = file.split(\"_\")[-1].split('/')[0]\n",
        "    ax.set_title(date)\n",
        "\n",
        "for ax in axes.flatten():\n",
        "    ax.axis(\"off\")\n",
        "plt.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JHUyCHZCg_kp"
      },
      "source": [
        "# Useful utilities"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s7Gc13XDbU_1"
      },
      "outputs": [],
      "source": [
        "# This line packages the contents of your Files folder, for download\n",
        "\n",
        "!zip -r /content/file.zip /content/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xNN9PqOEbUnn"
      },
      "outputs": [],
      "source": [
        "#  This code downloads the packaged files and prompts for a directory to save to.\n",
        "\n",
        "from google.colab import files\n",
        "files.download(\"/content/file.zip\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Tzn5dlBUtN7H"
      },
      "outputs": [],
      "source": [
        "# This function deletes all contents of a folder, recursively. USE WITH CAUTiON!!\n",
        "\n",
        "import os\n",
        "import shutil\n",
        "\n",
        "def delete_contents(folder):\n",
        "    for item in os.listdir(folder):\n",
        "        item_path = os.path.join(folder, item)\n",
        "        if os.path.isfile(item_path):\n",
        "            os.remove(item_path)\n",
        "        elif os.path.isdir(item_path):\n",
        "            shutil.rmtree(item_path)\n",
        "    print(\"All files and folders have been deleted.\")\n",
        "\n",
        "# Uncomment the following line to run this funtion the /content/ folder\n",
        "# in your colab notebook, or alter the directory path to target another folder\n",
        "\n",
        "# delete_contents('/content')\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}