OPERA_S3_Access.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cc0a01d3-d0e0-4b0f-8c72-8416e8aa6a2d",
   "metadata": {},
   "source": [
    "# OPERA Direct S3 Access\n",
    "\n",
    "Access OPERA L2-RTC and L2-CSLC in-place\n",
    "\n",
    "Author: Alex Lewandowski; Alaska Satellite Facility\n",
    "\n",
    "Notebook based on following ASF documentation and examples:\n",
    "- [ASF OPERA Data Discovery](https://asf.alaska.edu/datasets/daac/opera/)\n",
    "- [ASF S3 Credential Instructions](https://cumulus.asf.alaska.edu/s3credentialsREADME?)\n",
    "\n",
    "---\n",
    "\n",
    "<div class=\"alert alert-danger\" style=\"display: flex; align-items: center; font-family: 'Times New Roman', Times, serif; background-color: 'rgba(200,0,0,0.2)'\">\n",
    "  <div style=\"width: 95%;\">\n",
    "    <h2><b>Warning: Direct S3 Access with EDL Bearer Tokens is only possible in-region (AWS us-west-2)</b></h2>\n",
    "      <ul>\n",
    "    <li>This notebook cannot be run locally.</li>\n",
    "    <li>It must be run from an EC2 instance in the AWS us-west-2 region where the data are stored.</li>\n",
    "    <li>You can run this from OpenSARLab or from another Jupyter Lab installation running in us-west-2.</li>\n",
    "  </div>\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\" style=\"display: flex; align-items: center; font-family: 'Times New Roman', Times, serif; background-color: #d1ecf1;\">\n",
    "  <div style=\"display: flex; align-items: center; width: 10%;\">\n",
    "    <a href=\"https://github.com/ASFOpenSARlab/opensarlab_OPERA-CSLC_Recipe_Book/issues\">\n",
    "      <img src=\"https://opensarlab-docs.asf.alaska.edu/opensarlab-notebook-assets/logos/github_issues.png\" alt=\"GitHub logo over the word Issues\" style=\"width: 100%;\">\n",
    "    </a>\n",
    "  </div>\n",
    "  <div style=\"width: 95%;\">\n",
    "    <b>Did you find a bug? Do you have a feature request?</b>\n",
    "    <br/>\n",
    "    Explore GitHub Issues on this Jupyter Book's GitHub repository. Find solutions, add to the discussion, or start a new bug report or feature request: <a href=\"https://github.com/ASFOpenSARlab/opensarlab_OPERA-CSLC_Recipe_Book/issues\">opensarlab_OPERA-CSLC_Recipe_Book Issues</a>\n",
    "  </div>\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-info\" style=\"display: flex; align-items: center; justify-content: space-between; font-family: 'Times New Roman', Times, serif; background-color: #d1ecf1;\">\n",
    "  <div style=\"display: flex; align-items: center; width: 10%; margin-right: 10px;\">\n",
    "    <a href=\"mailto:uso@asf.alaska.edu\">\n",
    "      <img src=\"https://opensarlab-docs.asf.alaska.edu/opensarlab-notebook-assets/logos/ASF_support_logo.png\" alt=\"ASF logo\" style=\"width: 100%\">\n",
    "    </a>\n",
    "  </div>\n",
    "  <div style=\"width: 95%;\">\n",
    "    <b>Have a question related to SAR, OPERA CSLCs, or ASF data access?</b>\n",
    "    <br/>\n",
    "    Contact ASF User Support: <a href=\"mailto:uso@asf.alaska.edu\">uso@asf.alaska.edu</a>\n",
    "  </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## 0. Import Required Software"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3707b1c-d253-4bb5-820f-aa59715f2bc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ast\n",
    "from datetime import datetime\n",
    "from getpass import getpass\n",
    "import io\n",
    "import json\n",
    "import re\n",
    "import urllib.request\n",
    "\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import display\n",
    "\n",
    "import boto3\n",
    "import h5py\n",
    "import pandas as pd\n",
    "import s3fs\n",
    "import xarray as xr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "527cfc6e-0bca-44db-90df-f3edc2a595bb",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 1. Select some OPERA product IDs\n",
    "\n",
    "**L2-RTC-S1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "010fc721-2561-4b13-86b0-6bfa425d9efa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # code assumes opera_ids contains a single product type\n",
    "# opera_ids = [\n",
    "#     \"OPERA_L2_RTC-S1_T137-292339-IW3_20240201T020001Z_20240201T081115Z_S1A_30_v1.0\",\n",
    "#     \"OPERA_L2_RTC-S1_T137-292339-IW3_20240201T020001Z_20240201T114538Z_S1A_30_v1.0\",\n",
    "#     \"OPERA_L2_RTC-S1_T137-292339-IW3_20240120T020002Z_20240120T142521Z_S1A_30_v1.0\",\n",
    "# ]\n",
    "\n",
    "# prefix = \"OPERA_L2_RTC-S1\" if \"RTC\" in opera_ids[0] else \"OPERA_L2_CSLC-S1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0345c66d-4029-45f8-a45d-fde6d8c1a4b8",
   "metadata": {},
   "source": [
    "**L2-RTC-S1-STATIC**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08e40d17-e14a-4d73-848a-93147a31db78",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # code assumes opera_ids contains a single product type\n",
    "# opera_ids = [\n",
    "#     \"OPERA_L2_RTC-S1-STATIC_T137-292339-IW3_20140403_S1A_30_v1.0\",\n",
    "# ]\n",
    "\n",
    "# prefix = \"OPERA_L2_RTC-S1\" if \"RTC\" in opera_ids[0] else \"OPERA_L2_CSLC-S1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12c5f3c7-d03b-4d72-836b-f952405fc94c",
   "metadata": {},
   "source": [
    "**L2-CSLC-S1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e62bcfd-97d0-4671-b219-d1152b6a0600",
   "metadata": {},
   "outputs": [],
   "source": [
    "# code assumes opera_ids contains a single product type\n",
    "opera_ids = [\n",
    "    \"OPERA_L2_CSLC-S1_T137-292339-IW3_20240201T020001Z_20240202T175818Z_S1A_VV_v1.0\",\n",
    "    \"OPERA_L2_CSLC-S1_T137-292339-IW3_20240201T020001Z_20240202T175821Z_S1A_VV_v1.0\",\n",
    "    \"OPERA_L2_CSLC-S1_T137-292339-IW3_20240120T020002Z_20240121T075842Z_S1A_VV_v1.0\",\n",
    "]\n",
    "\n",
    "prefix = \"OPERA_L2_RTC-S1\" if \"RTC\" in opera_ids[0] else \"OPERA_L2_CSLC-S1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc25d887-62fd-4908-8490-376554fc643e",
   "metadata": {},
   "source": [
    "**L2-CSLC-S1-STATIC**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f3895b0-452d-4d12-8e31-90953dde3c51",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # code assumes opera_ids contains a single product type\n",
    "# opera_ids = [\n",
    "#     \"OPERA_L2_CSLC-S1-STATIC_T137-292339-IW3_20140403_S1A_v1.0\",\n",
    "# ]\n",
    "\n",
    "# prefix = \"OPERA_L2_RTC-S1\" if \"RTC\" in opera_ids[0] else \"OPERA_L2_CSLC-S1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed65cd3d-9090-4136-a65e-5590631ba08b",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 2. Request S3 access credentials\n",
    "\n",
    "**Enter your Earthdata Login Bearer Token**\n",
    "\n",
    "[Instructions for creating an EDL bearer token](https://urs.earthdata.nasa.gov/documentation/for_users/user_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c89e395a-45d4-45c2-9d03-7cd42d6cd3bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "token = getpass(\"Enter your EDL Bearer Token\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05c63a18-36a1-44df-ae57-4f809df9778b",
   "metadata": {},
   "outputs": [],
   "source": [
    "event = {\n",
    "    \"CredentialsEndpoint\": \"https://cumulus.asf.alaska.edu/s3credentials\",\n",
    "    \"BearerToken\": token,\n",
    "    \"Bucket\": \"asf-cumulus-prod-opera-products\",\n",
    "    \"Prefix\": prefix,\n",
    "    \"StaticPrefix\": f\"{prefix}_STATIC\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bdaa317-6f49-4a1d-9c54-7cee7b6c2bb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get temporary download credentials\n",
    "tea_url = event[\"CredentialsEndpoint\"]\n",
    "bearer_token = event[\"BearerToken\"]\n",
    "req = urllib.request.Request(\n",
    "    url=tea_url,\n",
    "    headers={\"Authorization\": f\"Bearer {bearer_token}\"}\n",
    ")\n",
    "with urllib.request.urlopen(req) as f:\n",
    "    creds = json.loads(f.read().decode())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e03dca66-f3ca-403d-afa3-0291c9093e6a",
   "metadata": {},
   "source": [
    "## 3. If accessing RTC or RTC-STATIC data, select a layer type\n",
    "\n",
    "This is not necessary for CSLC and CSLC-STATIC, as their data layers are stored in multidimensional HDF5 files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eeff72e0-7a7b-4616-a974-f58db3e03c73",
   "metadata": {},
   "outputs": [],
   "source": [
    "rtc = 'RTC' in event[\"Prefix\"]\n",
    "static = 'STATIC' in opera_ids[0]\n",
    "\n",
    "if rtc:\n",
    "    if static:\n",
    "        file = {\n",
    "            'Incidence angle (ellipsoidal)': '_incidence_angle.tif',\n",
    "            'Local-incidence angle': '_local_incidence_angle.tif', \n",
    "            'No. of looks': '_number_of_looks.tif',\n",
    "            'Layover Shadow Mask layer': '_mask.tif',\n",
    "            'RTC Area Normalization Factor (ANF) gamma0 to beta0': '_rtc_anf_gamma0_to_beta0.tif',\n",
    "            'RTC Area Normalization Factor (ANF) gamma0 to sigma0': '_rtc_anf_gamma0_to_sigma0.tif'\n",
    "        }\n",
    "    else:\n",
    "        file = {\n",
    "            'VH RTC': '_VH.tif',\n",
    "            'VV RTC': '_VV.tif',\n",
    "            'Layover Shadow Mask layer': '_mask.tif',\n",
    "        }\n",
    "elif 'CSLC'  not in event['Prefix']:\n",
    "    raise Exception(\"Unrecognized Product Type\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a67ea41c-a45b-40cc-b0cc-58f010647345",
   "metadata": {},
   "outputs": [],
   "source": [
    "if rtc:\n",
    "    print(\"Select a product type\")\n",
    "    product_choice = widgets.RadioButtons(\n",
    "        options=file,\n",
    "        description='',\n",
    "        disabled=False,\n",
    "        layout={'width': '500px'}\n",
    "    )\n",
    "    display(product_choice)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24def021-b145-4f8e-82a9-6419dd60f7c4",
   "metadata": {},
   "source": [
    "## 4. Open a single OPERA product\n",
    "\n",
    "Access the first product in `opera_ids`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7effab4-65cf-4c98-bf1b-c7219615f573",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = f\"{opera_ids[0]}{product_choice.value}\" if rtc else f\"{opera_ids[0]}.h5\"\n",
    "object_key = f\"{event['StaticPrefix']}/{opera_ids[0]}/{filename}\" if static else f\"{event['Prefix']}/{opera_ids[0]}/{filename}\" \n",
    "\n",
    "fs = s3fs.S3FileSystem(key=creds['accessKeyId'], secret=creds['secretAccessKey'], token=creds['sessionToken'])\n",
    "\n",
    "# Define S3 path\n",
    "s3_path = f\"{event['Bucket']}/{object_key}\"\n",
    "\n",
    "# Open the file as a file-like object using s3fs\n",
    "with fs.open(s3_path, mode='rb') as f:\n",
    "    if '.h5' in object_key and not rtc:\n",
    "        ds = xr.open_dataset(f, engine='h5netcdf', group=\"data\") # can't seem to specify a specific group here, like \"data/VV\"\n",
    "        with h5py.File(f, 'r') as h5f:\n",
    "            attributes = h5f.attrs\n",
    "            for attr, value in attributes.items():\n",
    "                try:\n",
    "                    ds.attrs[attr] = value.decode('utf-8')\n",
    "                except AttributeError:\n",
    "                    ds.attrs[attr] = value\n",
    "    else:\n",
    "        ds = xr.open_dataarray(f, engine=\"rasterio\")\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e880947-f435-471a-8db7-14924c889c36",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 5. Create an RTC time-series \n",
    "\n",
    "**Build a sorted list of OPERA L2-RTC-S1 IDs**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc6f7682-a333-4b70-8c81-e1e6828a3d1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "opera_rtc_ids = sorted([\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20231004T020005Z_20240122T203351Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20231016T020005Z_20231016T154509Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20231028T020005Z_20231029T045555Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20231109T020004Z_20231114T103429Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20231121T020004Z_20231206T001313Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20231203T020004Z_20231203T100451Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20231203T020004Z_20231203T122715Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20231215T020003Z_20231215T142550Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20231227T020002Z_20231230T152233Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20240108T020002Z_20240109T091409Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20240120T020002Z_20240120T142521Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20240201T020001Z_20240201T081115Z_S1A_30_v1.0',\n",
    "    'OPERA_L2_RTC-S1_T137-292339-IW3_20240201T020001Z_20240201T114538Z_S1A_30_v1.0'\n",
    "])\n",
    "opera_rtc_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef3c533f-9de1-42cb-8a2d-b25d9d8daec8",
   "metadata": {},
   "source": [
    "**Create a pandas DataFrame of the time series**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60b8b5f9-0575-44db-aea6-0aea27cfce87",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dt(opera_id, date_regex):\n",
    "    acquisition_time = re.search(date_regex, opera_id)\n",
    "    try:\n",
    "        return acquisition_time.group(0)\n",
    "    except AttributeError:\n",
    "        raise Exception(f\"Acquisition timestamp not found in scene ID: {opera_id}\") \n",
    "\n",
    "acquisition_date_regex = r\"(?<=_)\\d{8}T\\d{6}Z(?=_\\d{8}T\\d{6})\"\n",
    "process_dt_regex = r\"(?<=\\d{8}T\\d{6}Z_)\\d{8}T\\d{6}Z(?=_S1)\"\n",
    "\n",
    "acquisition_dt = pd.to_datetime([get_dt(id, acquisition_date_regex) for id in opera_rtc_ids])\n",
    "process_dt = pd.to_datetime([get_dt(id, process_dt_regex) for id in opera_rtc_ids])\n",
    "\n",
    "times_series_df = (pd.DataFrame(data={\n",
    "    'OPERA L2-RTC-S1 ID': opera_rtc_ids, \n",
    "    'AcquisitionDateTime': acquisition_dt,\n",
    "    'ProcessDateTime': process_dt\n",
    "})\n",
    ".sort_values(by='ProcessDateTime')\n",
    ".drop_duplicates(subset=['AcquisitionDateTime'], keep='last')\n",
    ".drop('ProcessDateTime', axis=1)\n",
    ".sort_values(by='AcquisitionDateTime')\n",
    ".reset_index(drop=True))\n",
    "\n",
    "times_series_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b51a77a-da4d-4d6e-9811-3efbfd65a0d9",
   "metadata": {},
   "source": [
    "**Build a time series of both polarizations in an xarray Dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6239be4-db46-4c96-b1ef-25933c449d93",
   "metadata": {},
   "outputs": [],
   "source": [
    "fs = s3fs.S3FileSystem(key=creds['accessKeyId'], secret=creds['secretAccessKey'], token=creds['sessionToken'])\n",
    "\n",
    "polarizations = ['VV', 'VH']\n",
    "da_stack = []\n",
    "\n",
    "for t, row in times_series_df.iterrows():\n",
    "    opera_id = row['OPERA L2-RTC-S1 ID']\n",
    "    time = pd.to_datetime(row['AcquisitionDateTime'])\n",
    "    polarization_stack = []\n",
    "\n",
    "    for polarization in polarizations:\n",
    "        filename = f\"{opera_id}_{polarization}.tif\"\n",
    "        object_key = f\"OPERA_L2_RTC-S1/{opera_id}/{filename}\"\n",
    "        s3_path = f\"s3://{event['Bucket']}/{object_key}\"\n",
    "\n",
    "        with fs.open(s3_path, mode='rb') as f:\n",
    "            da = xr.open_dataarray(f, engine=\"rasterio\")\n",
    "            da = da.expand_dims(time=pd.Index([time], name='time'))\n",
    "            polarization_stack.append(da)\n",
    "\n",
    "    da_polarized = xr.concat(polarization_stack, dim=pd.Index(polarizations, name='polarization'))\n",
    "    da_stack.append(da_polarized)\n",
    "\n",
    "ds = xr.concat(da_stack, dim='time')\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75da6ce5-98fc-4501-b3be-30061c7c8791",
   "metadata": {},
   "source": [
    "**Access the VV time series**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c058c15-f62f-490a-a5a6-cc8a1f95ca92",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "my_time = pd.to_datetime('20231203T020004Z')\n",
    "ds.sel(polarization='VV')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d447adc0-abf7-4b0a-bc5d-0b252ad26518",
   "metadata": {},
   "source": [
    "**Access the VH RTC from the 1st timestep**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10398bfb-7559-40b9-b6c6-0780256de848",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ds.sel(polarization='VH').isel(time=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20c318fb-b6d5-43e6-87fb-11f9962fffdd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "osl_opera_cslc_jbook_env [conda env:.local-osl_opera_cslc_jbook_env]",
   "language": "python",
   "name": "conda-env-.local-osl_opera_cslc_jbook_env-osl_opera_cslc_jbook_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}