7-Map_Validation.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "2eb2fc88-25b8-4d5a-8a2c-b8e40cbe6178",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2023 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "#\n",
    "# Developed for Google LLC by RedCastle Resources Inc\n",
    "# https://www.redcastleresources.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae243bdf-1298-4668-a561-3709d7a26995",
   "metadata": {
    "id": "title:generic,gcp"
   },
   "source": [
    "# <img width=50px  src = 'https://apps.fs.usda.gov/lcms-viewer/src/assets/images/lcms-icon.png'> Lab 7: LCMS Map Validation\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/redcastle-resources/lcms-training/blob/main/7-Map_Validation.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/redcastle-resources/lcms-training/blob/main/7-Map_Validation.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/redcastle-resources/lcms-training/blob/main/7-Map_Validation.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      Open in Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>\n",
    "<br/><br/><br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4603c77e-11e2-45bc-9ddd-b75c4a170d8b",
   "metadata": {
    "id": "overview:mlops"
   },
   "source": [
    "## 7.0: Overview and Introduction\n",
    "\n",
    "\n",
    "This notebook teaches how to assess map accuracy of LCMS outputs.\n",
    "\n",
    "While this notebook teaches the specific steps LCMS takes to assess map accuracy, your assessment methods can possibly be much simpler if you have a large training sample, simple or systematic random sample design, and/or a map output that always takes the most confident prediction from the model. \n",
    "\n",
    "### 7.0.1: Background\n",
    "\n",
    "* While previous tutorials have provided a good depiction of the model error, for a number of reasons, this is not a statistically valid depiction of the map error. The following are reasons why the model error is not the same as the map error in the case of LCMS' current methods:\n",
    "    * Our sample is not a simple random sample - rather it is a stratified random sample\n",
    "    * For Change, we introduce additional logic that makes the default model output different than the map output\n",
    "* LCMS also lacks sufficient training data to leave a subset out of the actual model calibration\n",
    "    * This means we need to simulate the model error using either a bootstrapping technique or k-fold cross validation\n",
    "* If your particular project has sufficient training data to leave a subset out of the actual model calibration, it is a simple or systematic random sample (not stratified random), and the default model predicted class is the same as your final map class, you can skip this step and use the model errors from the methods used in Module 5.\n",
    "\n",
    "### 7.0.2: Objective\n",
    "\n",
    "In this tutorial, you learn how to assess the map accuracy of LCMS map outputs. \n",
    "\n",
    "This tutorial uses the following Google Cloud services:\n",
    "\n",
    "- `Google Earth Engine`\n",
    "- `Google Cloud Storage`\n",
    "\n",
    "The steps performed include:\n",
    "\n",
    "- Understanding the difference between model and map accuracy\n",
    "- Simulating map accuracy with k-fold cross validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3afa3bd3-afcf-413c-9367-b8693cc6e921",
   "metadata": {},
   "source": [
    "### 7.0.3: Before you begin\n",
    "\n",
    "#### If you are working in Workbench: Set your current URL under `workbench_url`\n",
    "This gives the Map Viewer a url in which to host the viewer we will be generating. \n",
    "* This will be in your URL/search bar at the top of the browser window you are currently in\n",
    "* It will look something like `https://1234567890122-dot-us-west3.notebooks.googleusercontent.com/` (See the image below)\n",
    "\n",
    "![workspace url](img/workspace-url.png)\n",
    "\n",
    "#### Set a folder to use for all exports under `export_path_root` \n",
    "* This folder should be an assets folder in an existing GEE project.\n",
    "* By default, this folder is the same as the pre-baked folder (where outputs have already been created). \n",
    "* If you would like to create your own outputs, specify a different path for `export_path_root`, but leave the `pre_baked_path_root` as it was. This way, the pre-baked outputs can be shown at the end, instead of waiting for all exports to finish.\n",
    "  * It will be something like `projects/projectID/assets/newFolder`\n",
    "* This folder does not have to already exist. If it does not exist, it will be created\n",
    "\n",
    "**If you are working in Qwiklabs and wish to export:** Copy the project ID from the 'Start Lab' screen into the `projectID` field in `export_path_root`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0061d85b-fc89-4ce7-9800-12d3e19ed61a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Export path root: projects/rcr-gee/assets/lcms-training\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "workbench_url = 'https://1307eb830a12e633-dot-us-central1.notebooks.googleusercontent.com/lab/tree/lcms-training/7-Map_Validation.ipynb'\n",
    "pre_baked_path_root  = 'projects/rcr-gee/assets/lcms-training'\n",
    "export_path_root = pre_baked_path_root\n",
    "\n",
    "# to export to a custom project, fill in your project ID and uncomment the lines below\n",
    "# projectID = 'qwiklabs-gcp-00-00000000' # change this to match your project id\n",
    "# export_path_root = f'projects/{projectID}/assets/lcms-training'\n",
    "\n",
    "print('Export path root:', export_path_root)\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7764a53b-ed4d-4f76-b641-946d215cd775",
   "metadata": {},
   "source": [
    "#### Installation\n",
    "First, install necessary Python packages. Uncomment the first line to upgrade geeViz if necessary.\n",
    "\n",
    "Note that we're also importing additional custom scripts from the lcms_scripts folder in this repo. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4fae6857-1dfc-448b-b92f-85ec010f8460",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "#Module imports\n",
    "#!python -m pip install geeViz --upgrade\n",
    "try:\n",
    "    import geeViz.getImagesLib as getImagesLib\n",
    "except:\n",
    "    !python -m pip install geeViz\n",
    "    import geeViz.getImagesLib as getImagesLib\n",
    "\n",
    "import geeViz.changeDetectionLib as changeDetectionLib\n",
    "import geeViz.assetManagerLib as aml\n",
    "import geeViz.taskManagerLib as tml\n",
    "import geeViz.gee2Pandas as g2p\n",
    "import inspect,operator,os,glob,json,warnings\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd  \n",
    "import numpy as np\n",
    "\n",
    "import lcms_scripts.accuracy_and_sampling_lib2 as asl\n",
    "from importlib import reload\n",
    "\n",
    "try:\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    from sklearn.ensemble import RandomForestClassifier\n",
    "    from sklearn.model_selection import GroupKFold\n",
    "    from sklearn.metrics import accuracy_score,classification_report,balanced_accuracy_score,cohen_kappa_score\n",
    "    from sklearn import metrics \n",
    "except:\n",
    "    !pip install -U scikit-learn\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    from sklearn.ensemble import RandomForestClassifier\n",
    "    from sklearn.model_selection import GroupKFold\n",
    "    from sklearn.metrics import accuracy_score,classification_report,balanced_accuracy_score,cohen_kappa_score\n",
    "    from sklearn import metrics \n",
    "\n",
    "ee = getImagesLib.ee\n",
    "Map = getImagesLib.Map\n",
    "\n",
    "# Can set the port used for viewing map outputs\n",
    "Map.port = 1235\n",
    "print('Done')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e57f53e6-27ad-49a7-9107-7fd65b6aa095",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Set up your work environment\n",
    "\n",
    "Create a folder in your export path where you will export the composites. In addition, create a blank image collection where your composites will live.\n",
    "\n",
    "Currently, when running within Colab or Workbench, geeView uses a different project to authenticate through, so you may need to make your asset public to view from within Colab.\n",
    "\n",
    "**Warning!!**\n",
    "\n",
    "* **Read-only access is provided for all authenticated GEE users for pre-baked outputs**\n",
    "* **If you are using the pre-baked output location (`export_path_root = pre_baked_path_root`), you will see errors for any operation that tries to write, delete, or change access permissions to any asset in the pre-baked output location.**\n",
    "* **This is expected and will not stop you from successfully running this notebook - ignore these error messages should they appear.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ed03b3ae-31ec-4caf-9add-5d9d1eb1bec7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "# Bring in all folders/collections that are needed\n",
    "# These must already exist as they are created in previous notebooks\n",
    "\n",
    "export_timeSync_folder = f'{export_path_root}/lcms-training_module-4_timeSync'\n",
    "\n",
    "export_assembledLCMSOutputs_collection = f'{export_path_root}/lcms-training_module-6_assembledLCMSOutputs'\n",
    "\n",
    "# This is the pre-made TimeSync data\n",
    "# Creating this dataset is not covered in this set of notebooks\n",
    "timeSync_featureCollection = 'projects/lcms-292214/assets/R8/PR_USVI/TimeSync/18_PRVI_AllPlots_TimeSync_Annualized_Table_secLC'\n",
    "\n",
    "# The model options table (created in module 5.1, but stored in the lcms-training repository)\n",
    "model_options_csv_filename = './tables/LCMS_model_options_table.csv'\n",
    "change_thresholds_json_filename = './tables/LCMS_change_thresholds.json'\n",
    "\n",
    "# Set up folder to put accuracy files\n",
    "local_map_acc_folder = '/tmp/lcms-training/local_map_accuracy'\n",
    "\n",
    "if not os.path.exists(local_map_acc_folder):os.makedirs(local_map_acc_folder)\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "de0a034d-86b6-4cdb-8de9-b177250d8d0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "# set up map\n",
    "Map.clearMap()\n",
    "\n",
    "# reset port if necessary\n",
    "Map.port = 1237\n",
    "Map.proxy_url = workbench_url\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "388f479b",
   "metadata": {},
   "source": [
    "\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a35b6b8",
   "metadata": {},
   "source": [
    "## 7.1: Prepare input data\n",
    "\n",
    "\n",
    "\n",
    "### 7.1.2: Bring in Reference Data\n",
    "\n",
    "* First, we'll need to repeat steps from Module 5 and download our reference data to a local location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e5d6f281-1c62-44ca-a56e-00769912a05f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LCMS class code, names, and colors: {'Change_class_names': ['Stable', 'Slow Loss', 'Fast Loss', 'Gain', 'Non-Processing Area Mask'], 'Change_class_palette': ['3d4551', 'f39268', 'd54309', '00a398', '1b1716'], 'Change_class_values': [1, 2, 3, 4, 5], 'Land_Cover_class_names': ['Trees', 'Tall Shrubs & Trees Mix (SEAK Only)', 'Shrubs & Trees Mix', 'Grass/Forb/Herb & Trees Mix', 'Barren & Trees Mix', 'Tall Shrubs (SEAK Only)', 'Shrubs', 'Grass/Forb/Herb & Shrubs Mix', 'Barren & Shrubs Mix', 'Grass/Forb/Herb', 'Barren & Grass/Forb/Herb Mix', 'Barren or Impervious', 'Snow or Ice', 'Water', 'Non-Processing Area Mask'], 'Land_Cover_class_palette': ['005e00', '008000', '00cc00', 'b3ff1a', '99ff99', 'b30088', 'e68a00', 'ffad33', 'ffe0b3', 'ffff00', 'aa7700', 'd3bf9b', 'ffffff', '4780f3', '1b1716'], 'Land_Cover_class_values': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], 'Land_Use_class_names': ['Agriculture', 'Developed', 'Forest', 'Non-Forest Wetland', 'Other', 'Rangeland or Pasture', 'Non-Processing Area Mask'], 'Land_Use_class_palette': ['efff6b', 'ff2ff8', '1b9d0c', '97ffff', 'a1a1a1', 'c2b34a', '1b1716'], 'Land_Use_class_values': [1, 2, 3, 4, 5, 6, 7], 'study_area': 'PRUSVI', 'year': 1985}\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "# Bring in raw TS data\n",
    "timeSyncData = ee.FeatureCollection(timeSync_featureCollection)\n",
    "timeSync_fields = timeSyncData.first().toDictionary().keys().getInfo()\n",
    "# Now lets bring in all training data and prep it for modeling\n",
    "assets = ee.data.listAssets({'parent': export_timeSync_folder})['assets']\n",
    "\n",
    "# You may need to change the permissions for viewing model outputs in geeViz\n",
    "# Uncomment this if needed\n",
    "# for asset in assets:aml.updateACL(asset['name'],writers = [],all_users_can_read = True,readers = [])\n",
    "\n",
    "# Read in each year of extracted TimsSync data\n",
    "training_data = ee.FeatureCollection([ee.FeatureCollection(asset['name']) for asset in assets]).flatten()\n",
    "\n",
    "# Bring in existing LCMS data for the class names, numbers, and colors\n",
    "lcms_viz_dict = ee.ImageCollection(\"USFS/GTAC/LCMS/v2022-8\").filter('study_area==\"PRUSVI\"').first().toDictionary().getInfo()\n",
    "\n",
    "                                             \n",
    "print('LCMS class code, names, and colors:',lcms_viz_dict)\n",
    "\n",
    "# Specify the field for each product\n",
    "reference_field_dict = {'Land_Cover':{'field':'DOM_SEC_LC'},\n",
    "                        'Land_Use':{'field':'DOM_LU'},\n",
    "                        'Change':{'field':'CP'}\n",
    "                       }\n",
    "\n",
    "# Get the field names for prediction\n",
    "# Find any field that was not in the original TimeSync data and assume that is a predictor variable\n",
    "all_fields = training_data.first().toDictionary().keys().getInfo()\n",
    "predictor_field_names = [field for field in all_fields if field not in timeSync_fields]\n",
    "\n",
    "# Filter out any non null values (any training plot with missing predictor data will cause the model to fail entirely)\n",
    "training_data = training_data.filter(ee.Filter.notNull(predictor_field_names))\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "43e33c57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/lcms-training/local_modeling\\timeSync_training_table.csv  already exists\n",
      "Done\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BARREN</th>\n",
       "      <th>BARREN-GRA</th>\n",
       "      <th>BARREN-IMP</th>\n",
       "      <th>BARREN-SHR</th>\n",
       "      <th>BARREN-TRE</th>\n",
       "      <th>BARREN-TS</th>\n",
       "      <th>CHANGE_DUR</th>\n",
       "      <th>CP_Code</th>\n",
       "      <th>DOM_LU_Code</th>\n",
       "      <th>DOM_SEC_LC_Code</th>\n",
       "      <th>...</th>\n",
       "      <th>swir2_LT_dur</th>\n",
       "      <th>swir2_LT_fitted</th>\n",
       "      <th>swir2_LT_mag</th>\n",
       "      <th>swir2_LT_slope</th>\n",
       "      <th>wetness_LT_diff</th>\n",
       "      <th>wetness_LT_dur</th>\n",
       "      <th>wetness_LT_fitted</th>\n",
       "      <th>wetness_LT_mag</th>\n",
       "      <th>wetness_LT_slope</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>19777.0</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.0</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "      <td>19777.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.005360</td>\n",
       "      <td>0.098094</td>\n",
       "      <td>0.004955</td>\n",
       "      <td>0.002174</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.327350</td>\n",
       "      <td>1.459372</td>\n",
       "      <td>3.307933</td>\n",
       "      <td>4.352885</td>\n",
       "      <td>...</td>\n",
       "      <td>26.305810</td>\n",
       "      <td>0.079596</td>\n",
       "      <td>0.005514</td>\n",
       "      <td>0.000503</td>\n",
       "      <td>-0.008305</td>\n",
       "      <td>27.750417</td>\n",
       "      <td>-0.073550</td>\n",
       "      <td>-0.011920</td>\n",
       "      <td>-0.000660</td>\n",
       "      <td>2011.236494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.073016</td>\n",
       "      <td>0.297449</td>\n",
       "      <td>0.070221</td>\n",
       "      <td>0.046579</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.767562</td>\n",
       "      <td>1.035758</td>\n",
       "      <td>1.297727</td>\n",
       "      <td>4.435511</td>\n",
       "      <td>...</td>\n",
       "      <td>11.267701</td>\n",
       "      <td>0.039772</td>\n",
       "      <td>0.033840</td>\n",
       "      <td>0.005742</td>\n",
       "      <td>0.029825</td>\n",
       "      <td>11.333125</td>\n",
       "      <td>0.044698</td>\n",
       "      <td>0.043909</td>\n",
       "      <td>0.006246</td>\n",
       "      <td>6.040297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.006700</td>\n",
       "      <td>-0.210300</td>\n",
       "      <td>-0.070100</td>\n",
       "      <td>-0.372316</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.360283</td>\n",
       "      <td>-0.393000</td>\n",
       "      <td>-0.176800</td>\n",
       "      <td>2000.517822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>0.053306</td>\n",
       "      <td>-0.011300</td>\n",
       "      <td>-0.000481</td>\n",
       "      <td>-0.022232</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>-0.097389</td>\n",
       "      <td>-0.034600</td>\n",
       "      <td>-0.001229</td>\n",
       "      <td>2006.169922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.069844</td>\n",
       "      <td>0.006200</td>\n",
       "      <td>0.000230</td>\n",
       "      <td>-0.007874</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>-0.067405</td>\n",
       "      <td>-0.014300</td>\n",
       "      <td>-0.000466</td>\n",
       "      <td>2011.295898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>0.095467</td>\n",
       "      <td>0.021800</td>\n",
       "      <td>0.000900</td>\n",
       "      <td>0.005050</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>-0.045750</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.000423</td>\n",
       "      <td>2016.476685</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>0.373600</td>\n",
       "      <td>0.300500</td>\n",
       "      <td>0.204700</td>\n",
       "      <td>0.212600</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>0.072426</td>\n",
       "      <td>0.212600</td>\n",
       "      <td>0.053150</td>\n",
       "      <td>2021.411011</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 162 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        BARREN    BARREN-GRA    BARREN-IMP    BARREN-SHR    BARREN-TRE  \\\n",
       "count  19777.0  19777.000000  19777.000000  19777.000000  19777.000000   \n",
       "mean       0.0      0.005360      0.098094      0.004955      0.002174   \n",
       "std        0.0      0.073016      0.297449      0.070221      0.046579   \n",
       "min        0.0      0.000000      0.000000      0.000000      0.000000   \n",
       "25%        0.0      0.000000      0.000000      0.000000      0.000000   \n",
       "50%        0.0      0.000000      0.000000      0.000000      0.000000   \n",
       "75%        0.0      0.000000      0.000000      0.000000      0.000000   \n",
       "max        0.0      1.000000      1.000000      1.000000      1.000000   \n",
       "\n",
       "       BARREN-TS    CHANGE_DUR       CP_Code   DOM_LU_Code  DOM_SEC_LC_Code  \\\n",
       "count    19777.0  19777.000000  19777.000000  19777.000000     19777.000000   \n",
       "mean         0.0     14.327350      1.459372      3.307933         4.352885   \n",
       "std          0.0      6.767562      1.035758      1.297727         4.435511   \n",
       "min          0.0      1.000000      1.000000      1.000000         1.000000   \n",
       "25%          0.0      9.000000      1.000000      3.000000         1.000000   \n",
       "50%          0.0     17.000000      1.000000      3.000000         1.000000   \n",
       "75%          0.0     21.000000      1.000000      3.000000         9.000000   \n",
       "max          0.0     21.000000      4.000000      6.000000        14.000000   \n",
       "\n",
       "       ...  swir2_LT_dur  swir2_LT_fitted  swir2_LT_mag  swir2_LT_slope  \\\n",
       "count  ...  19777.000000     19777.000000  19777.000000    19777.000000   \n",
       "mean   ...     26.305810         0.079596      0.005514        0.000503   \n",
       "std    ...     11.267701         0.039772      0.033840        0.005742   \n",
       "min    ...      1.000000         0.006700     -0.210300       -0.070100   \n",
       "25%    ...     17.000000         0.053306     -0.011300       -0.000481   \n",
       "50%    ...     28.000000         0.069844      0.006200        0.000230   \n",
       "75%    ...     38.000000         0.095467      0.021800        0.000900   \n",
       "max    ...     38.000000         0.373600      0.300500        0.204700   \n",
       "\n",
       "       wetness_LT_diff  wetness_LT_dur  wetness_LT_fitted  wetness_LT_mag  \\\n",
       "count     19777.000000    19777.000000       19777.000000    19777.000000   \n",
       "mean         -0.008305       27.750417          -0.073550       -0.011920   \n",
       "std           0.029825       11.333125           0.044698        0.043909   \n",
       "min          -0.372316        1.000000          -0.360283       -0.393000   \n",
       "25%          -0.022232       19.000000          -0.097389       -0.034600   \n",
       "50%          -0.007874       32.000000          -0.067405       -0.014300   \n",
       "75%           0.005050       38.000000          -0.045750        0.010000   \n",
       "max           0.212600       38.000000           0.072426        0.212600   \n",
       "\n",
       "       wetness_LT_slope          year  \n",
       "count      19777.000000  19777.000000  \n",
       "mean          -0.000660   2011.236494  \n",
       "std            0.006246      6.040297  \n",
       "min           -0.176800   2000.517822  \n",
       "25%           -0.001229   2006.169922  \n",
       "50%           -0.000466   2011.295898  \n",
       "75%            0.000423   2016.476685  \n",
       "max            0.053150   2021.411011  \n",
       "\n",
       "[8 rows x 162 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now will download the training table to a local location\n",
    "\n",
    "local_model_data_folder = '/tmp/lcms-training/local_modeling'\n",
    "local_training_csv = os.path.join(local_model_data_folder,'timeSync_training_table.csv')\n",
    "\n",
    "\n",
    "if not os.path.exists(local_model_data_folder):os.makedirs(local_model_data_folder)\n",
    "\n",
    "# Download the training data from a featureCollection to a local CSV\n",
    "# This function will automatically break the featureCollection into 5000 feature featureCollections\n",
    "# if it is larger than the 5000 feature limit set by GEE\n",
    "g2p.featureCollection_to_csv(training_data,local_training_csv,overwrite = False)\n",
    "\n",
    "# Once the table is store locally, read it in\n",
    "training_df = pd.read_csv(local_training_csv)\n",
    "\n",
    "print('Done')\n",
    "\n",
    "training_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0493893",
   "metadata": {},
   "source": [
    "#### 7.2.1.1 Filter to predictors used\n",
    "\n",
    "* Filter out to only have rows from the non correlated top 30 predictors\n",
    "* No single set of prectors works best over all models and all model performance metrics\n",
    "* Any subset of predictors could be used here, but this one should work well\n",
    "* Good options are: `Non-correlated Predictors`, `All Predictors Top 30`, or `Non-correlated Predictors Top 30`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "86b925a6-f238-436f-8abe-3cd7b6536f21",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Product Name</th>\n",
       "      <th>Model Name</th>\n",
       "      <th>OOB Acc</th>\n",
       "      <th>Overall Acc</th>\n",
       "      <th>Balanced Acc</th>\n",
       "      <th>Kappa</th>\n",
       "      <th>Var Imp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Change</td>\n",
       "      <td>Non-correlated Predictors Top 30</td>\n",
       "      <td>0.884917</td>\n",
       "      <td>0.846736</td>\n",
       "      <td>0.281699</td>\n",
       "      <td>0.094835</td>\n",
       "      <td>['swir2_LT_slope', 'swir2_LT_fitted', 'NDVI_LT...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Land_Cover</td>\n",
       "      <td>Non-correlated Predictors Top 30</td>\n",
       "      <td>0.974010</td>\n",
       "      <td>0.696878</td>\n",
       "      <td>0.286296</td>\n",
       "      <td>0.484086</td>\n",
       "      <td>['red_CCDC_fitted', 'red_LT_fitted', 'NBR_LT_f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Land_Use</td>\n",
       "      <td>Non-correlated Predictors Top 30</td>\n",
       "      <td>0.993578</td>\n",
       "      <td>0.787196</td>\n",
       "      <td>0.626276</td>\n",
       "      <td>0.659068</td>\n",
       "      <td>['red_LT_fitted', 'green_CCDC_fitted', 'red_CC...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Product Name                        Model Name   OOB Acc  Overall Acc  \\\n",
       "3        Change  Non-correlated Predictors Top 30  0.884917     0.846736   \n",
       "7    Land_Cover  Non-correlated Predictors Top 30  0.974010     0.696878   \n",
       "11     Land_Use  Non-correlated Predictors Top 30  0.993578     0.787196   \n",
       "\n",
       "    Balanced Acc     Kappa                                            Var Imp  \n",
       "3       0.281699  0.094835  ['swir2_LT_slope', 'swir2_LT_fitted', 'NDVI_LT...  \n",
       "7       0.286296  0.484086  ['red_CCDC_fitted', 'red_LT_fitted', 'NBR_LT_f...  \n",
       "11      0.626276  0.659068  ['red_LT_fitted', 'green_CCDC_fitted', 'red_CC...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "predictor_set = 'Non-correlated Predictors Top 30'\n",
    "model_options = pd.read_csv(model_options_csv_filename)\n",
    "\n",
    "model_options = model_options[model_options['Model Name'] == predictor_set]\n",
    "\n",
    "display(model_options)\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "868a5764",
   "metadata": {},
   "source": [
    "## 7.2: Prepare validation framework\n",
    "### 7.2.1: K-Fold Cross-validation\n",
    "\n",
    "* LCMS does not have enough training samples to simply ommit 20% or so from training our final models\n",
    "* Since our assemblage process introduces differences between the model predicted class, and our sample is based on a stratified random sample design, we cannot simply use the out-of-bag samples from the random forest model\n",
    "* We have to use a method that will simulate the map accuracy that can account for the likelihood of each samples inclusion (strata weights), as well as also allow us to introduce any assemblage rules that are not typically part of the underlying random forest model\n",
    "* Our methods roughly follow guidance from [Stehman 2014](./lit/Stehman_2014.pdf)\n",
    "* We first divide our reference sample into a train and test set over K-folds and train a random forest model for each fold\n",
    "* E.g. if we have 5 folds, then for each fold, 80% of the training data are used to train the model, and the remaining 20% are held out for comparing the reference and predicted classes. This ensures all samples are held out once ($20 \\%  * 5 folds = 100\\%$).\n",
    "![Cross_Validation](https://scikit-learn.org/stable/_images/grid_search_cross_validation.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2b1aff8d-b726-4fed-8cd9-d8851c947279",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold 1 has 17797 training samples and 1980 test samples\n",
      "Fitting model for fold 1, Change\n",
      "Fitting model for fold 1, Land_Cover\n",
      "Fitting model for fold 1, Land_Use\n",
      "Fold 2 has 17797 training samples and 1980 test samples\n",
      "Fitting model for fold 2, Change\n",
      "Fitting model for fold 2, Land_Cover\n",
      "Fitting model for fold 2, Land_Use\n",
      "Fold 3 has 17805 training samples and 1972 test samples\n",
      "Fitting model for fold 3, Change\n",
      "Fitting model for fold 3, Land_Cover\n",
      "Fitting model for fold 3, Land_Use\n",
      "Fold 4 has 17805 training samples and 1972 test samples\n",
      "Fitting model for fold 4, Change\n",
      "Fitting model for fold 4, Land_Cover\n",
      "Fitting model for fold 4, Land_Use\n",
      "Fold 5 has 17797 training samples and 1980 test samples\n",
      "Fitting model for fold 5, Change\n",
      "Fitting model for fold 5, Land_Cover\n",
      "Fitting model for fold 5, Land_Use\n",
      "Fold 6 has 17798 training samples and 1979 test samples\n",
      "Fitting model for fold 6, Change\n",
      "Fitting model for fold 6, Land_Cover\n",
      "Fitting model for fold 6, Land_Use\n",
      "Fold 7 has 17798 training samples and 1979 test samples\n",
      "Fitting model for fold 7, Change\n",
      "Fitting model for fold 7, Land_Cover\n",
      "Fitting model for fold 7, Land_Use\n",
      "Fold 8 has 17798 training samples and 1979 test samples\n",
      "Fitting model for fold 8, Change\n",
      "Fitting model for fold 8, Land_Cover\n",
      "Fitting model for fold 8, Land_Use\n",
      "Fold 9 has 17799 training samples and 1978 test samples\n",
      "Fitting model for fold 9, Change\n",
      "Fitting model for fold 9, Land_Cover\n",
      "Fitting model for fold 9, Land_Use\n",
      "Fold 10 has 17799 training samples and 1978 test samples\n",
      "Fitting model for fold 10, Change\n",
      "Fitting model for fold 10, Land_Cover\n",
      "Fitting model for fold 10, Land_Use\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "# Set up k-fold cross-validation\n",
    "\n",
    "products = ['Change','Land_Cover','Land_Use']\n",
    "KFoldInfo = {}\n",
    "# kfoldinfo_pickle_filename = pickleName+'.p'\n",
    "# KFoldInfo['TrainingData'] = training_df.copy()\n",
    "\n",
    "# strata = allTrainingData[stratColumn].squeeze()\n",
    "groups = training_df['PLOTID'].squeeze()\n",
    "k = 10\n",
    "n_jobs = 4\n",
    "gkf = GroupKFold(n_splits=k)\n",
    "foldNum = 1\n",
    "seed = 999\n",
    "nTrees = 50\n",
    "\n",
    "# Fit and Train model\n",
    "# Set up a random forest model\n",
    "rf = RandomForestClassifier(n_estimators = nTrees, random_state=seed,oob_score=False,n_jobs = n_jobs)\n",
    "\n",
    "# Set up k fold in\n",
    "KFoldInfo['STRATUM'] = []\n",
    "KFoldInfo['STRATUM_PIXEL_COUNT'] = []\n",
    "KFoldInfo['STRATUM_PIXEL_PCT'] = []\n",
    "\n",
    "# Iterate over k folds\n",
    "for train_index, test_index in gkf.split(training_df, training_df, groups):\n",
    "    KFoldInfo[foldNum] = {}\n",
    "    \n",
    "    print(f'Fold {foldNum} has {len(train_index)} training samples and {len(test_index)} test samples')\n",
    "\n",
    "    # Indices of training and test samples\n",
    "    KFoldInfo[foldNum]['Indices'] = {\\\n",
    "        'Train': train_index,\n",
    "        'Test': test_index}\n",
    "\n",
    "    # Run model and predict probabilities\n",
    "    KFoldInfo[foldNum]['Probabilities'] = {}\n",
    "    KFoldInfo[foldNum]['Predictions'] = {}\n",
    "    KFoldInfo[foldNum]['Model'] = {}\n",
    "    KFoldInfo[foldNum]['Ref'] = {}\n",
    "    \n",
    "    # Pull the tran and test plots\n",
    "    k_train,k_test = training_df.iloc[train_index], training_df.iloc[test_index]\n",
    "    \n",
    "    # Get the strata info\n",
    "    KFoldInfo['STRATUM'].extend(k_test['STRATUM'])\n",
    "    KFoldInfo['STRATUM_PIXEL_COUNT'].extend(k_test['STRATUM_PIXEL_COUNT'])\n",
    "    KFoldInfo['STRATUM_PIXEL_PCT'].extend(k_test['STRATUM_PIXEL_PCT'])\n",
    "    for product_name in products:\n",
    "        print(f'Fitting model for fold {foldNum}, {product_name}')\n",
    "        \n",
    "        # Pull predictors from table from 5.1\n",
    "        # Some parsing is needed to read it in properly\n",
    "        predictor_variable_names = model_options[model_options['Product Name'] == product_name]['Var Imp'].values[0]\n",
    "        predictor_variable_names = predictor_variable_names[1:-1]\n",
    "        predictor_variable_names=predictor_variable_names.replace(\"'\",\"\").split(', ')\n",
    "        \n",
    "        # Get X and Y points for each group  \n",
    "        kx_train = k_train[predictor_variable_names]\n",
    "        ky_train = k_train[reference_field_dict[product_name]['field']+'_Code']\n",
    "        \n",
    "        kx_test = k_test[predictor_variable_names]\n",
    "        ky_test = k_test[reference_field_dict[product_name]['field']+'_Code']\n",
    "        \n",
    "        rf.fit(kx_train,ky_train)\n",
    "        \n",
    "        # Get predicted classes or probabilities for each Test Point\n",
    "        # For change, get the probabilities, for all others, get the classes\n",
    "        if product_name in ['Land_Cover','Land_Use']:\n",
    "            ky_pred = rf.predict(kx_test)\n",
    "            \n",
    "        else:\n",
    "            ky_pred = rf.predict_proba(kx_test)\n",
    "        KFoldInfo[foldNum]['Predictions'][product_name] = ky_pred\n",
    "        KFoldInfo[foldNum]['Ref'][product_name] = ky_test\n",
    "        \n",
    "    foldNum+=1\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30487d7e",
   "metadata": {},
   "source": [
    "### 7.2.2: Organize Reference and Predicted Values\n",
    "\n",
    "* Organize reference and predicted values into simple dictionary by product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "29c1825a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Change': {'refs': 0        1\n",
      "1        1\n",
      "2        1\n",
      "3        1\n",
      "4        4\n",
      "        ..\n",
      "19772    1\n",
      "19773    1\n",
      "19774    1\n",
      "19775    1\n",
      "19776    1\n",
      "Length: 19777, dtype: int64, 'preds': 0         [0.92, 0.0, 0.02, 0.06]\n",
      "1         [0.74, 0.0, 0.12, 0.14]\n",
      "2          [0.9, 0.0, 0.04, 0.06]\n",
      "3        [0.68, 0.02, 0.08, 0.22]\n",
      "4         [0.9, 0.02, 0.04, 0.04]\n",
      "                   ...           \n",
      "19772     [0.84, 0.0, 0.08, 0.08]\n",
      "19773    [0.68, 0.02, 0.16, 0.14]\n",
      "19774    [0.62, 0.02, 0.22, 0.14]\n",
      "19775     [0.62, 0.02, 0.06, 0.3]\n",
      "19776      [0.94, 0.0, 0.06, 0.0]\n",
      "Length: 19777, dtype: object}, 'Land_Cover': {'refs': 0        10\n",
      "1        10\n",
      "2        10\n",
      "3        10\n",
      "4        10\n",
      "         ..\n",
      "19772    12\n",
      "19773    12\n",
      "19774    12\n",
      "19775    14\n",
      "19776    14\n",
      "Length: 19777, dtype: int64, 'preds': 0        10\n",
      "1         1\n",
      "2        10\n",
      "3        10\n",
      "4        10\n",
      "         ..\n",
      "19772    12\n",
      "19773    12\n",
      "19774     1\n",
      "19775     1\n",
      "19776    14\n",
      "Length: 19777, dtype: int64}, 'Land_Use': {'refs': 0        4\n",
      "1        4\n",
      "2        1\n",
      "3        6\n",
      "4        6\n",
      "        ..\n",
      "19772    2\n",
      "19773    2\n",
      "19774    2\n",
      "19775    5\n",
      "19776    5\n",
      "Length: 19777, dtype: int64, 'preds': 0        4\n",
      "1        3\n",
      "2        4\n",
      "3        2\n",
      "4        6\n",
      "        ..\n",
      "19772    2\n",
      "19773    2\n",
      "19774    2\n",
      "19775    3\n",
      "19776    5\n",
      "Length: 19777, dtype: int64}}\n"
     ]
    }
   ],
   "source": [
    "ref_pred_dict = {}\n",
    "\n",
    "for product_name in products:\n",
    "    preds = []\n",
    "    refs = []\n",
    "    for foldNum in range(1,k+1):\n",
    "        predsFold = KFoldInfo[foldNum]['Predictions'][product_name]\n",
    "        refsFold = KFoldInfo[foldNum]['Ref'][product_name]\n",
    "        preds.extend(predsFold)\n",
    "        refs.extend(refsFold)\n",
    "        \n",
    "    refs = pd.Series(refs)\n",
    "    preds = pd.Series(preds)\n",
    "    ref_pred_dict[product_name] = {'refs':refs,'preds':preds}\n",
    "   \n",
    "print(ref_pred_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29dcfc2b",
   "metadata": {},
   "source": [
    "### 7.2.3: Apply Thresholds to Change\n",
    "\n",
    "* Since the Change outputs use a specific set of thresholds and logic covered in Module 6, we must replicate this process to ensure the predicted Change classes reflect what would be on the map.\n",
    "* There are many additional considerations for representing time series map accuracy\n",
    "    * Often, since our input predictor data may not indicate a change occurred until the year after, we will allow for a prediction the year after to count as correct. These kind of fuzzy logic exceptions are not included in this example, but may be appropriate to consider. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "432b0f36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Change Predictions: [[0.92 0.   0.02 0.06]\n",
      " [0.74 0.   0.12 0.14]\n",
      " [0.9  0.   0.04 0.06]\n",
      " ...\n",
      " [0.62 0.02 0.22 0.14]\n",
      " [0.62 0.02 0.06 0.3 ]\n",
      " [0.94 0.   0.06 0.  ]]\n",
      "[0.92 0.74 0.9  ... 0.62 0.62 0.94]\n",
      "Output Change Predictions: [1. 1. 1. ... 1. 1. 1.]\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "product_name = 'Change'\n",
    "\n",
    "# Open class thresholds used from Module 6\n",
    "o = open(change_thresholds_json_filename,'r')\n",
    "change_thresholds = json.load(o)\n",
    "o.close()\n",
    "\n",
    "# Pull the predictions from the k-fold cross validation\n",
    "# Currently they contain the probability of all predictions\n",
    "raw_change_preds = np.array(ref_pred_dict[product_name]['preds'].values.tolist())\n",
    "print('Input Change Predictions:',raw_change_preds)\n",
    "\n",
    "# Specify which classes to assemble and the index (value - 1) where they are located in the prediction\n",
    "# probability table\n",
    "change_classes = ['Slow Loss','Fast Loss','Gain']\n",
    "change_indices = [1,2,3]\n",
    "\n",
    "# Find the max probability\n",
    "max_prob = np.max(raw_change_preds, axis=1)\n",
    "print(max_prob)\n",
    "\n",
    "# Set up a constant output of 1 (Stable)\n",
    "out = np.ones(len(max_prob))\n",
    "\n",
    "# Find max conf class\n",
    "change_class_max_conf_preds = raw_change_preds.argmax(axis=1)+1\n",
    "\n",
    "# Iterate across each class \n",
    "for change_class,change_index in list(zip(change_classes,change_indices)):\n",
    "    \n",
    "    # Pull the predictions for the class\n",
    "    preds_class = raw_change_preds[:,change_index]\n",
    "\n",
    "    # Find if it is the max and above the threshold\n",
    "    isMax = preds_class == max_prob\n",
    "    aboveThresh = preds_class > change_thresholds[change_class]\n",
    "    isChange = isMax & aboveThresh\n",
    "\n",
    "    # If it is the max and above the threshold, recode the stable to the respective value (index +1)\n",
    "    change_value = change_index+1\n",
    "    out[isChange] = change_value\n",
    "\n",
    "print('Output Change Predictions:',out)\n",
    "# Reset the predictions for Change\n",
    "ref_pred_dict[product_name]['preds'] = pd.Series(out)\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "925c2db1-2c48-4b83-8804-8dfea44327ff",
   "metadata": {},
   "source": [
    "## 7.3 Compute Accuracy\n",
    "\n",
    "### 7.3.1: Set up strata and strata weights\n",
    "\n",
    "* In order to account for the probability of including a sample from a given stratum, we need to know the proportion of the total sample location population for each stratum \n",
    "* Note that our strata are not the same as our map classes - we set the strata before we make the map. the goal of  the strata is to make sure we appropriately sample classes that are underrepresented.\n",
    "* The number of pixels in each class will be used as weights in our accuracy assessmentWe can never expect a model-based output to match this level of detail.\n",
    "The question becomes whether the maps are useful and do they at least align with the literature that employed similar methods.\n",
    "Reviewing the maps, we can see they are a reasonable depiction of what they intend to represent, but should be used with these error rates in mind. They are generally appropriate to use over larger areas, but not for your favorite pixel.\n",
    "\n",
    "\n",
    "![strata](img/lab-7/strata-PRUSVI.png)\n",
    "\n",
    "Source: [LCMS Methods v2020-6](https://data.fs.usda.gov/geodata/rastergateway/LCMS/LCMS_v2020-6_Methods.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b036565a-7412-43af-b4df-a6e90c959780",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(1, 1450331), (2, 113212), (3, 110101), (4, 344785), (5, 71185), (6, 92958), (7, 3519261), (8, 3325898), (9, 865643), (10, 258676), (11, 102288)]\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "# Set up lookup dictionary to convert the numeric codes to names\n",
    "strata_code_name_dict = ee.Dictionary({1:\"DEVELOPED\",\n",
    "                                       2: \"WATER\",\n",
    "                                       3: \"BARREN\",\n",
    "                                       4: \"AGRICULTURE\",\n",
    "                                       5: \"NON-FORESTED WETLAND\",\n",
    "                                       6: \"FORESTEED WETLAND\",\n",
    "                                       7: \"RANGE-GRASS-SHRUB-HERB\",\n",
    "                                       8: \"EVERGREEN\",\n",
    "                                       9: \"DECIDUOUS\",\n",
    "                                       10: \"CLOUD FOREST\",\n",
    "                                       11: \"COASTAL MIXED FOREST\"\n",
    "                                      })\n",
    "\n",
    "stratum = [i for i in KFoldInfo['STRATUM']]\n",
    "\n",
    "stratum_counts = KFoldInfo['STRATUM_PIXEL_COUNT']\n",
    "stratum_pct = [(n/100) for n in KFoldInfo['STRATUM_PIXEL_PCT']]\n",
    "strata_dict = dict(set(zip(stratum,stratum_counts)))\n",
    "strata_pct_dict = dict(set(zip(stratum,stratum_pct)))\n",
    "strata_dict_sorted = sorted(strata_dict.items(), key = lambda x:x[0])\n",
    "strata_pct_dict_sorted = sorted(strata_pct_dict.items(), key = lambda x:x[0])\n",
    "\n",
    "print(strata_dict_sorted)\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9243663d-a8c8-439c-a2a0-45b0b2e50e2c",
   "metadata": {},
   "source": [
    "### 7.3.2: Look at confusion matrices\n",
    "\n",
    "* We will first look at the confusion matrices from the cross-validation\n",
    "* These will be very similar to the matrices shown in Module 5\n",
    "* But, here we area assessing map error- rather than model error\n",
    "\n",
    "* Notice the impact of the change assemblage (thresholds) is very minor. While the overall accuraccy is not changed much, the map is much different (as shown in module 6).\n",
    "\n",
    "* Remember that we are not mapping slow loss in Puerto Rico since it is so uncommon in our training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f6272717",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Assembled Change Map Confusion Matrix:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"5\" halign=\"left\">Observed</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Stable</th>\n",
       "      <th>Slow Loss</th>\n",
       "      <th>Fast Loss</th>\n",
       "      <th>Gain</th>\n",
       "      <th>User's Accuracy (1 - commission error)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">Predicted</th>\n",
       "      <th>Stable</th>\n",
       "      <td>2559.70</td>\n",
       "      <td>2.75</td>\n",
       "      <td>146.61</td>\n",
       "      <td>438.88</td>\n",
       "      <td>82.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Slow Loss</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fast Loss</th>\n",
       "      <td>4.53</td>\n",
       "      <td>0.00</td>\n",
       "      <td>13.32</td>\n",
       "      <td>1.36</td>\n",
       "      <td>69.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gain</th>\n",
       "      <td>24.17</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.15</td>\n",
       "      <td>18.08</td>\n",
       "      <td>40.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Producer's Accuracy (1 - omission error)</th>\n",
       "      <td>98.91</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.37</td>\n",
       "      <td>4.07</td>\n",
       "      <td>81.43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   Observed            \\\n",
       "                                                     Stable Slow Loss   \n",
       "Predicted Stable                                    2559.70      2.75   \n",
       "          Slow Loss                                    0.00      0.00   \n",
       "          Fast Loss                                    4.53      0.00   \n",
       "          Gain                                        24.17      0.00   \n",
       "          Producer's Accuracy (1 - omission error)    98.91      0.00   \n",
       "\n",
       "                                                                      \\\n",
       "                                                   Fast Loss    Gain   \n",
       "Predicted Stable                                      146.61  438.88   \n",
       "          Slow Loss                                     0.00    0.00   \n",
       "          Fast Loss                                    13.32    1.36   \n",
       "          Gain                                          2.15   18.08   \n",
       "          Producer's Accuracy (1 - omission error)      8.37    4.07   \n",
       "\n",
       "                                                                                           \n",
       "                                                   User's Accuracy (1 - commission error)  \n",
       "Predicted Stable                                                                    82.07  \n",
       "          Slow Loss                                                                   NaN  \n",
       "          Fast Loss                                                                 69.59  \n",
       "          Gain                                                                      40.72  \n",
       "          Producer's Accuracy (1 - omission error)                                  81.43  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Change Max Conf:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"5\" halign=\"left\">Observed</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Stable</th>\n",
       "      <th>Slow Loss</th>\n",
       "      <th>Fast Loss</th>\n",
       "      <th>Gain</th>\n",
       "      <th>User's Accuracy (1 - commission error)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">Predicted</th>\n",
       "      <th>Stable</th>\n",
       "      <td>2564.42</td>\n",
       "      <td>2.75</td>\n",
       "      <td>148.00</td>\n",
       "      <td>441.65</td>\n",
       "      <td>82.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Slow Loss</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fast Loss</th>\n",
       "      <td>4.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12.55</td>\n",
       "      <td>1.04</td>\n",
       "      <td>69.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gain</th>\n",
       "      <td>19.48</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.53</td>\n",
       "      <td>15.63</td>\n",
       "      <td>42.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Producer's Accuracy (1 - omission error)</th>\n",
       "      <td>99.08</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.92</td>\n",
       "      <td>3.55</td>\n",
       "      <td>81.48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   Observed            \\\n",
       "                                                     Stable Slow Loss   \n",
       "Predicted Stable                                    2564.42      2.75   \n",
       "          Slow Loss                                    0.00      0.00   \n",
       "          Fast Loss                                    4.50      0.00   \n",
       "          Gain                                        19.48      0.00   \n",
       "          Producer's Accuracy (1 - omission error)    99.08      0.00   \n",
       "\n",
       "                                                                      \\\n",
       "                                                   Fast Loss    Gain   \n",
       "Predicted Stable                                      148.00  441.65   \n",
       "          Slow Loss                                     0.00    0.00   \n",
       "          Fast Loss                                    12.55    1.04   \n",
       "          Gain                                          1.53   15.63   \n",
       "          Producer's Accuracy (1 - omission error)      7.92    3.55   \n",
       "\n",
       "                                                                                           \n",
       "                                                   User's Accuracy (1 - commission error)  \n",
       "Predicted Stable                                                                    82.00  \n",
       "          Slow Loss                                                                   NaN  \n",
       "          Fast Loss                                                                 69.67  \n",
       "          Gain                                                                      42.60  \n",
       "          Producer's Accuracy (1 - omission error)                                  81.48  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Assembled Land Cover Map Confusion Matrix:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"12\" halign=\"left\">Observed</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Trees</th>\n",
       "      <th>Shrubs &amp; Trees Mix</th>\n",
       "      <th>Grass/Forb/Herb &amp; Trees Mix</th>\n",
       "      <th>Barren &amp; Trees Mix</th>\n",
       "      <th>Shrubs</th>\n",
       "      <th>Grass/Forb/Herb &amp; Shrubs Mix</th>\n",
       "      <th>Barren &amp; Shrubs Mix</th>\n",
       "      <th>Grass/Forb/Herb</th>\n",
       "      <th>Barren &amp; Grass/Forb/Herb Mix</th>\n",
       "      <th>Barren or Impervious</th>\n",
       "      <th>Water</th>\n",
       "      <th>User's Accuracy (1 - commission error)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"12\" valign=\"top\">Predicted</th>\n",
       "      <th>Trees</th>\n",
       "      <td>1786.16</td>\n",
       "      <td>133.57</td>\n",
       "      <td>160.10</td>\n",
       "      <td>4.13</td>\n",
       "      <td>50.08</td>\n",
       "      <td>56.66</td>\n",
       "      <td>5.60</td>\n",
       "      <td>86.78</td>\n",
       "      <td>1.64</td>\n",
       "      <td>39.69</td>\n",
       "      <td>7.99</td>\n",
       "      <td>77.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shrubs &amp; Trees Mix</th>\n",
       "      <td>0.34</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.06</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.69</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>23.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grass/Forb/Herb &amp; Trees Mix</th>\n",
       "      <td>10.82</td>\n",
       "      <td>3.40</td>\n",
       "      <td>14.64</td>\n",
       "      <td>0.32</td>\n",
       "      <td>6.24</td>\n",
       "      <td>10.62</td>\n",
       "      <td>1.04</td>\n",
       "      <td>18.70</td>\n",
       "      <td>0.35</td>\n",
       "      <td>1.66</td>\n",
       "      <td>0.00</td>\n",
       "      <td>20.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Barren &amp; Trees Mix</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shrubs</th>\n",
       "      <td>4.41</td>\n",
       "      <td>0.37</td>\n",
       "      <td>9.56</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12.33</td>\n",
       "      <td>14.14</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.90</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>30.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grass/Forb/Herb &amp; Shrubs Mix</th>\n",
       "      <td>6.62</td>\n",
       "      <td>0.93</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12.81</td>\n",
       "      <td>9.62</td>\n",
       "      <td>0.00</td>\n",
       "      <td>17.16</td>\n",
       "      <td>0.34</td>\n",
       "      <td>1.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>19.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Barren &amp; Shrubs Mix</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grass/Forb/Herb</th>\n",
       "      <td>54.75</td>\n",
       "      <td>11.30</td>\n",
       "      <td>62.60</td>\n",
       "      <td>0.20</td>\n",
       "      <td>24.50</td>\n",
       "      <td>61.70</td>\n",
       "      <td>1.12</td>\n",
       "      <td>190.80</td>\n",
       "      <td>6.48</td>\n",
       "      <td>11.82</td>\n",
       "      <td>0.14</td>\n",
       "      <td>45.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Barren &amp; Grass/Forb/Herb Mix</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.35</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Barren or Impervious</th>\n",
       "      <td>9.65</td>\n",
       "      <td>0.70</td>\n",
       "      <td>12.25</td>\n",
       "      <td>0.13</td>\n",
       "      <td>3.41</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.69</td>\n",
       "      <td>19.71</td>\n",
       "      <td>10.57</td>\n",
       "      <td>224.08</td>\n",
       "      <td>0.32</td>\n",
       "      <td>78.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Water</th>\n",
       "      <td>0.53</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.13</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.28</td>\n",
       "      <td>4.71</td>\n",
       "      <td>81.85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Producer's Accuracy (1 - omission error)</th>\n",
       "      <td>95.05</td>\n",
       "      <td>0.68</td>\n",
       "      <td>5.23</td>\n",
       "      <td>0.00</td>\n",
       "      <td>10.49</td>\n",
       "      <td>5.85</td>\n",
       "      <td>0.00</td>\n",
       "      <td>57.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>81.54</td>\n",
       "      <td>69.54</td>\n",
       "      <td>70.53</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   Observed  \\\n",
       "                                                      Trees   \n",
       "Predicted Trees                                     1786.16   \n",
       "          Shrubs & Trees Mix                           0.34   \n",
       "          Grass/Forb/Herb & Trees Mix                 10.82   \n",
       "          Barren & Trees Mix                           0.00   \n",
       "          Shrubs                                       4.41   \n",
       "          Grass/Forb/Herb & Shrubs Mix                 6.62   \n",
       "          Barren & Shrubs Mix                          0.00   \n",
       "          Grass/Forb/Herb                             54.75   \n",
       "          Barren & Grass/Forb/Herb Mix                 0.00   \n",
       "          Barren or Impervious                         9.65   \n",
       "          Water                                        0.53   \n",
       "          Producer's Accuracy (1 - omission error)    95.05   \n",
       "\n",
       "                                                                       \\\n",
       "                                                   Shrubs & Trees Mix   \n",
       "Predicted Trees                                                133.57   \n",
       "          Shrubs & Trees Mix                                     0.68   \n",
       "          Grass/Forb/Herb & Trees Mix                            3.40   \n",
       "          Barren & Trees Mix                                     0.00   \n",
       "          Shrubs                                                 0.37   \n",
       "          Grass/Forb/Herb & Shrubs Mix                           0.93   \n",
       "          Barren & Shrubs Mix                                    0.00   \n",
       "          Grass/Forb/Herb                                       11.30   \n",
       "          Barren & Grass/Forb/Herb Mix                           0.00   \n",
       "          Barren or Impervious                                   0.70   \n",
       "          Water                                                  0.00   \n",
       "          Producer's Accuracy (1 - omission error)               0.68   \n",
       "\n",
       "                                                                                \\\n",
       "                                                   Grass/Forb/Herb & Trees Mix   \n",
       "Predicted Trees                                                         160.10   \n",
       "          Shrubs & Trees Mix                                              0.70   \n",
       "          Grass/Forb/Herb & Trees Mix                                    14.64   \n",
       "          Barren & Trees Mix                                              0.00   \n",
       "          Shrubs                                                          9.56   \n",
       "          Grass/Forb/Herb & Shrubs Mix                                    0.91   \n",
       "          Barren & Shrubs Mix                                             0.00   \n",
       "          Grass/Forb/Herb                                                62.60   \n",
       "          Barren & Grass/Forb/Herb Mix                                    0.00   \n",
       "          Barren or Impervious                                           12.25   \n",
       "          Water                                                           0.00   \n",
       "          Producer's Accuracy (1 - omission error)                        5.23   \n",
       "\n",
       "                                                                              \\\n",
       "                                                   Barren & Trees Mix Shrubs   \n",
       "Predicted Trees                                                  4.13  50.08   \n",
       "          Shrubs & Trees Mix                                     0.00   2.06   \n",
       "          Grass/Forb/Herb & Trees Mix                            0.32   6.24   \n",
       "          Barren & Trees Mix                                     0.00   0.00   \n",
       "          Shrubs                                                 0.00  12.33   \n",
       "          Grass/Forb/Herb & Shrubs Mix                           0.00  12.81   \n",
       "          Barren & Shrubs Mix                                    0.00   0.00   \n",
       "          Grass/Forb/Herb                                        0.20  24.50   \n",
       "          Barren & Grass/Forb/Herb Mix                           0.02   0.00   \n",
       "          Barren or Impervious                                   0.13   3.41   \n",
       "          Water                                                  0.00   1.13   \n",
       "          Producer's Accuracy (1 - omission error)               0.00  10.49   \n",
       "\n",
       "                                                                                 \\\n",
       "                                                   Grass/Forb/Herb & Shrubs Mix   \n",
       "Predicted Trees                                                           56.66   \n",
       "          Shrubs & Trees Mix                                               0.00   \n",
       "          Grass/Forb/Herb & Trees Mix                                     10.62   \n",
       "          Barren & Trees Mix                                               0.00   \n",
       "          Shrubs                                                          14.14   \n",
       "          Grass/Forb/Herb & Shrubs Mix                                     9.62   \n",
       "          Barren & Shrubs Mix                                              0.00   \n",
       "          Grass/Forb/Herb                                                 61.70   \n",
       "          Barren & Grass/Forb/Herb Mix                                     0.01   \n",
       "          Barren or Impervious                                             0.67   \n",
       "          Water                                                            0.00   \n",
       "          Producer's Accuracy (1 - omission error)                         5.85   \n",
       "\n",
       "                                                                        \\\n",
       "                                                   Barren & Shrubs Mix   \n",
       "Predicted Trees                                                   5.60   \n",
       "          Shrubs & Trees Mix                                      0.00   \n",
       "          Grass/Forb/Herb & Trees Mix                             1.04   \n",
       "          Barren & Trees Mix                                      0.00   \n",
       "          Shrubs                                                  0.06   \n",
       "          Grass/Forb/Herb & Shrubs Mix                            0.00   \n",
       "          Barren & Shrubs Mix                                     0.00   \n",
       "          Grass/Forb/Herb                                         1.12   \n",
       "          Barren & Grass/Forb/Herb Mix                            0.00   \n",
       "          Barren or Impervious                                    0.69   \n",
       "          Water                                                   0.06   \n",
       "          Producer's Accuracy (1 - omission error)                0.00   \n",
       "\n",
       "                                                                    \\\n",
       "                                                   Grass/Forb/Herb   \n",
       "Predicted Trees                                              86.78   \n",
       "          Shrubs & Trees Mix                                  0.69   \n",
       "          Grass/Forb/Herb & Trees Mix                        18.70   \n",
       "          Barren & Trees Mix                                  0.00   \n",
       "          Shrubs                                              0.90   \n",
       "          Grass/Forb/Herb & Shrubs Mix                       17.16   \n",
       "          Barren & Shrubs Mix                                 0.00   \n",
       "          Grass/Forb/Herb                                   190.80   \n",
       "          Barren & Grass/Forb/Herb Mix                        0.00   \n",
       "          Barren or Impervious                               19.71   \n",
       "          Water                                               0.01   \n",
       "          Producer's Accuracy (1 - omission error)           57.01   \n",
       "\n",
       "                                                                                 \\\n",
       "                                                   Barren & Grass/Forb/Herb Mix   \n",
       "Predicted Trees                                                            1.64   \n",
       "          Shrubs & Trees Mix                                               0.00   \n",
       "          Grass/Forb/Herb & Trees Mix                                      0.35   \n",
       "          Barren & Trees Mix                                               0.00   \n",
       "          Shrubs                                                           0.00   \n",
       "          Grass/Forb/Herb & Shrubs Mix                                     0.34   \n",
       "          Barren & Shrubs Mix                                              0.00   \n",
       "          Grass/Forb/Herb                                                  6.48   \n",
       "          Barren & Grass/Forb/Herb Mix                                     0.00   \n",
       "          Barren or Impervious                                            10.57   \n",
       "          Water                                                            0.00   \n",
       "          Producer's Accuracy (1 - omission error)                         0.00   \n",
       "\n",
       "                                                                         \\\n",
       "                                                   Barren or Impervious   \n",
       "Predicted Trees                                                   39.69   \n",
       "          Shrubs & Trees Mix                                       0.00   \n",
       "          Grass/Forb/Herb & Trees Mix                              1.66   \n",
       "          Barren & Trees Mix                                       0.00   \n",
       "          Shrubs                                                   0.00   \n",
       "          Grass/Forb/Herb & Shrubs Mix                             1.01   \n",
       "          Barren & Shrubs Mix                                      0.02   \n",
       "          Grass/Forb/Herb                                         11.82   \n",
       "          Barren & Grass/Forb/Herb Mix                             1.35   \n",
       "          Barren or Impervious                                   224.08   \n",
       "          Water                                                    0.28   \n",
       "          Producer's Accuracy (1 - omission error)                81.54   \n",
       "\n",
       "                                                           \\\n",
       "                                                    Water   \n",
       "Predicted Trees                                      7.99   \n",
       "          Shrubs & Trees Mix                         0.00   \n",
       "          Grass/Forb/Herb & Trees Mix                0.00   \n",
       "          Barren & Trees Mix                         0.00   \n",
       "          Shrubs                                     0.00   \n",
       "          Grass/Forb/Herb & Shrubs Mix               0.00   \n",
       "          Barren & Shrubs Mix                        0.00   \n",
       "          Grass/Forb/Herb                            0.14   \n",
       "          Barren & Grass/Forb/Herb Mix               0.00   \n",
       "          Barren or Impervious                       0.32   \n",
       "          Water                                      4.71   \n",
       "          Producer's Accuracy (1 - omission error)  69.54   \n",
       "\n",
       "                                                                                           \n",
       "                                                   User's Accuracy (1 - commission error)  \n",
       "Predicted Trees                                                                     77.01  \n",
       "          Shrubs & Trees Mix                                                        23.49  \n",
       "          Grass/Forb/Herb & Trees Mix                                               20.17  \n",
       "          Barren & Trees Mix                                                          NaN  \n",
       "          Shrubs                                                                    30.51  \n",
       "          Grass/Forb/Herb & Shrubs Mix                                              19.00  \n",
       "          Barren & Shrubs Mix                                                        0.00  \n",
       "          Grass/Forb/Herb                                                           45.48  \n",
       "          Barren & Grass/Forb/Herb Mix                                               0.00  \n",
       "          Barren or Impervious                                                      78.76  \n",
       "          Water                                                                     81.85  \n",
       "          Producer's Accuracy (1 - omission error)                                  70.53  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Assembled Land Use Map Confusion Matrix:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"7\" halign=\"left\">Observed</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Agriculture</th>\n",
       "      <th>Developed</th>\n",
       "      <th>Forest</th>\n",
       "      <th>Non-Forest Wetland</th>\n",
       "      <th>Other</th>\n",
       "      <th>Rangeland or Pasture</th>\n",
       "      <th>User's Accuracy (1 - commission error)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"7\" valign=\"top\">Predicted</th>\n",
       "      <th>Agriculture</th>\n",
       "      <td>4.66</td>\n",
       "      <td>0.72</td>\n",
       "      <td>3.78</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.51</td>\n",
       "      <td>41.41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Developed</th>\n",
       "      <td>8.29</td>\n",
       "      <td>476.40</td>\n",
       "      <td>29.17</td>\n",
       "      <td>0.97</td>\n",
       "      <td>2.85</td>\n",
       "      <td>26.61</td>\n",
       "      <td>85.85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Forest</th>\n",
       "      <td>5.14</td>\n",
       "      <td>59.48</td>\n",
       "      <td>1750.64</td>\n",
       "      <td>12.50</td>\n",
       "      <td>17.26</td>\n",
       "      <td>206.22</td>\n",
       "      <td>85.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Non-Forest Wetland</th>\n",
       "      <td>0.24</td>\n",
       "      <td>0.13</td>\n",
       "      <td>13.48</td>\n",
       "      <td>13.24</td>\n",
       "      <td>0.16</td>\n",
       "      <td>11.01</td>\n",
       "      <td>42.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Other</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.25</td>\n",
       "      <td>1.09</td>\n",
       "      <td>5.89</td>\n",
       "      <td>0.06</td>\n",
       "      <td>92.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rangeland or Pasture</th>\n",
       "      <td>32.89</td>\n",
       "      <td>58.89</td>\n",
       "      <td>75.67</td>\n",
       "      <td>18.42</td>\n",
       "      <td>0.00</td>\n",
       "      <td>369.94</td>\n",
       "      <td>64.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Producer's Accuracy (1 - omission error)</th>\n",
       "      <td>9.94</td>\n",
       "      <td>81.16</td>\n",
       "      <td>93.56</td>\n",
       "      <td>34.07</td>\n",
       "      <td>58.37</td>\n",
       "      <td>58.19</td>\n",
       "      <td>81.46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      Observed            \\\n",
       "                                                   Agriculture Developed   \n",
       "Predicted Agriculture                                     4.66      0.72   \n",
       "          Developed                                       8.29    476.40   \n",
       "          Forest                                          5.14     59.48   \n",
       "          Non-Forest Wetland                              0.24      0.13   \n",
       "          Other                                           0.00      0.00   \n",
       "          Rangeland or Pasture                           32.89     58.89   \n",
       "          Producer's Accuracy (1 - omission error)        9.94     81.16   \n",
       "\n",
       "                                                             \\\n",
       "                                                     Forest   \n",
       "Predicted Agriculture                                  3.78   \n",
       "          Developed                                   29.17   \n",
       "          Forest                                    1750.64   \n",
       "          Non-Forest Wetland                          13.48   \n",
       "          Other                                        0.25   \n",
       "          Rangeland or Pasture                        75.67   \n",
       "          Producer's Accuracy (1 - omission error)    93.56   \n",
       "\n",
       "                                                                              \\\n",
       "                                                   Non-Forest Wetland  Other   \n",
       "Predicted Agriculture                                            0.00   0.00   \n",
       "          Developed                                              0.97   2.85   \n",
       "          Forest                                                12.50  17.26   \n",
       "          Non-Forest Wetland                                    13.24   0.16   \n",
       "          Other                                                  1.09   5.89   \n",
       "          Rangeland or Pasture                                  18.42   0.00   \n",
       "          Producer's Accuracy (1 - omission error)              34.07  58.37   \n",
       "\n",
       "                                                                         \\\n",
       "                                                   Rangeland or Pasture   \n",
       "Predicted Agriculture                                              5.51   \n",
       "          Developed                                               26.61   \n",
       "          Forest                                                 206.22   \n",
       "          Non-Forest Wetland                                      11.01   \n",
       "          Other                                                    0.06   \n",
       "          Rangeland or Pasture                                   369.94   \n",
       "          Producer's Accuracy (1 - omission error)                58.19   \n",
       "\n",
       "                                                                                           \n",
       "                                                   User's Accuracy (1 - commission error)  \n",
       "Predicted Agriculture                                                               41.41  \n",
       "          Developed                                                                 85.85  \n",
       "          Forest                                                                    85.43  \n",
       "          Non-Forest Wetland                                                        42.52  \n",
       "          Other                                                                     92.00  \n",
       "          Rangeland or Pasture                                                      64.76  \n",
       "          Producer's Accuracy (1 - omission error)                                  81.46  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "reload(asl)\n",
    "# Ignore divide by 0 warnings from NumPy\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# Iterate across each product\n",
    "for product in products:\n",
    "    product_title = product.replace('_',' ')\n",
    "\n",
    "    # Set up output filename\n",
    "    cm_file = os.path.join(local_map_acc_folder,f'LCMS_{product}_Map_Confusion_Matrix.csv')\n",
    "\n",
    "    # Get the class numbers, names, and values \n",
    "    class_numbers = list(set(ref_pred_dict[product]['refs']))\n",
    "    class_names = lcms_viz_dict[f'{product}_class_names'][:-1]\n",
    "    class_values = lcms_viz_dict[f'{product}_class_values'][:-1]\n",
    "    lookup_dict = dict(list(zip(class_values,class_names)))\n",
    "\n",
    "    # Get rid of any missing values\n",
    "    assessment_classes = [n for n in class_values if n  in class_numbers]\n",
    "    labels = [lookup_dict[n] for n in class_values if n  in class_numbers]\n",
    "\n",
    "    # Get the confusion matrix\n",
    "    print(f'Final Assembled {product_title} Map Confusion Matrix:')\n",
    "    cm = asl.getConfusionMatrix(ref_pred_dict[product]['refs'],ref_pred_dict[product]['preds'],stratum,stratum_pct,strata_dict,assessment_classes,labels)\n",
    "    display(cm)\n",
    "    cm.to_csv(cm_file)\n",
    "    \n",
    "    # Show the confusion matrix of change if we simply used the max model confidence (only slightly different)\n",
    "    if product == 'Change':\n",
    "        print('Change Max Conf:')\n",
    "        cm = asl.getConfusionMatrix(ref_pred_dict[product]['refs'],change_class_max_conf_preds,stratum,stratum_pct,strata_dict,assessment_classes,labels)\n",
    "        display(cm)\n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30197741",
   "metadata": {},
   "source": [
    "### 7.3.3: Run Stehman 2014 Accuracy Method\n",
    "\n",
    "* Once the reference and predicted classes are organized for each LCMS output, we can run a method that reflects the method outlined in [Stehman 2014](https://www.tandfonline.com/doi/abs/10.1080/01431161.2014.930207) for each LCMS output product.\n",
    "* The outputs are stored as `.csv` files \n",
    "\n",
    "* The documentation of the function used is shown [here](https://github.com/redcastle-resources/lcms-training/blob/048c6c19b33b793a2ff0ef07fe212038acba87d7/lcms_scripts/accuracy_and_sampling_lib2.py#L343)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "98ddfb83-5b91-446c-8438-5a51ea06388b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing accuracy for: Change\n",
      "Computing accuracy for: Land_Cover\n",
      "Computing accuracy for: Land_Use\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "\n",
    "reload(asl)\n",
    "acc_results = {}\n",
    "for product_name in products:\n",
    "    print(f'Computing accuracy for: {product_name}')\n",
    "    acc_file = os.path.join(local_map_acc_folder,f'LCMS_{product_name}_Map_Accuracy_Results.csv')\n",
    "    \n",
    "    accuracy, balanced_accuracy, users, producers, kappa, f1_score, areas, accuracy_error, usersError, producersError, area_errors = asl.get_write_stratified_accuracies(\\\n",
    "        ref_pred_dict[product_name]['refs'],            # The correct classifications\n",
    "        ref_pred_dict[product_name]['preds'],      # The predicted classifications\n",
    "        stratum,       # The strata of the same plots as above\n",
    "        strata_dict,         # Dictionary of the number of pixels in each stratum - defined in LCMSVariables - used for weighting\n",
    "        lcms_viz_dict[f'{product_name}_class_values'][:-1], # Class names - used for looping through classes for users/producers accuracies and areas\n",
    "        lcms_viz_dict[f'{product_name}_class_names'][:-1],\n",
    "        method = product_name,        # This is just a run name, used for printing out accuracies in file. Not really used anymore\n",
    "        accFile = acc_file)\n",
    "\n",
    "    acc_results[product_name] = accuracy, balanced_accuracy, users, producers, kappa, f1_score, areas, accuracy_error, usersError, producersError, area_errors \n",
    "\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c599c24",
   "metadata": {},
   "source": [
    "### 7.3.4: Look at Accuracy Outputs\n",
    "\n",
    "* Once finished, we can look at the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3a97f328",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metric</th>\n",
       "      <th>Value</th>\n",
       "      <th>Confidence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Accuracy</td>\n",
       "      <td>0.814346</td>\n",
       "      <td>0.003340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Balanced Accuracy</td>\n",
       "      <td>0.295959</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Kappa</td>\n",
       "      <td>0.076810</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F1 Score</td>\n",
       "      <td>0.814346</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Users Accuracy: Stable</td>\n",
       "      <td>0.820721</td>\n",
       "      <td>0.003404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Users Accuracy: Slow Loss</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Users Accuracy: Fast Loss</td>\n",
       "      <td>0.695901</td>\n",
       "      <td>0.000545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Users Accuracy: Gain</td>\n",
       "      <td>0.407214</td>\n",
       "      <td>0.000640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Producers Accuracy: Stable</td>\n",
       "      <td>0.989129</td>\n",
       "      <td>0.003404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Producers Accuracy: Slow Loss</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Producers Accuracy: Fast Loss</td>\n",
       "      <td>0.083734</td>\n",
       "      <td>0.000545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Producers Accuracy: Gain</td>\n",
       "      <td>0.040674</td>\n",
       "      <td>0.000640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Design-Based Area Estimation: Stable</td>\n",
       "      <td>0.813544</td>\n",
       "      <td>0.003346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Design-Based Area Estimation: Slow Loss</td>\n",
       "      <td>0.001011</td>\n",
       "      <td>0.000239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Design-Based Area Estimation: Fast Loss</td>\n",
       "      <td>0.048848</td>\n",
       "      <td>0.001865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Design-Based Area Estimation: Gain</td>\n",
       "      <td>0.136598</td>\n",
       "      <td>0.002960</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     Metric     Value  Confidence\n",
       "0                                  Accuracy  0.814346    0.003340\n",
       "1                         Balanced Accuracy  0.295959         NaN\n",
       "2                                     Kappa  0.076810         NaN\n",
       "3                                  F1 Score  0.814346         NaN\n",
       "4                    Users Accuracy: Stable  0.820721    0.003404\n",
       "5                 Users Accuracy: Slow Loss       NaN    0.000000\n",
       "6                 Users Accuracy: Fast Loss  0.695901    0.000545\n",
       "7                      Users Accuracy: Gain  0.407214    0.000640\n",
       "8                Producers Accuracy: Stable  0.989129    0.003404\n",
       "9             Producers Accuracy: Slow Loss  0.000000    0.000000\n",
       "10            Producers Accuracy: Fast Loss  0.083734    0.000545\n",
       "11                 Producers Accuracy: Gain  0.040674    0.000640\n",
       "12     Design-Based Area Estimation: Stable  0.813544    0.003346\n",
       "13  Design-Based Area Estimation: Slow Loss  0.001011    0.000239\n",
       "14  Design-Based Area Estimation: Fast Loss  0.048848    0.001865\n",
       "15       Design-Based Area Estimation: Gain  0.136598    0.002960"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metric</th>\n",
       "      <th>Value</th>\n",
       "      <th>Confidence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Accuracy</td>\n",
       "      <td>0.705311</td>\n",
       "      <td>0.003776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Balanced Accuracy</td>\n",
       "      <td>0.217255</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Kappa</td>\n",
       "      <td>0.347829</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F1 Score</td>\n",
       "      <td>0.705311</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Users Accuracy: Trees</td>\n",
       "      <td>0.770108</td>\n",
       "      <td>0.003822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Users Accuracy: Tall Shrubs &amp; Trees Mix (SEAK ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Users Accuracy: Shrubs &amp; Trees Mix</td>\n",
       "      <td>0.234875</td>\n",
       "      <td>0.000109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Users Accuracy: Grass/Forb/Herb &amp; Trees Mix</td>\n",
       "      <td>0.201728</td>\n",
       "      <td>0.000591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Users Accuracy: Barren &amp; Trees Mix</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Users Accuracy: Tall Shrubs (SEAK Only)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Users Accuracy: Shrubs</td>\n",
       "      <td>0.305139</td>\n",
       "      <td>0.000529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Users Accuracy: Grass/Forb/Herb &amp; Shrubs Mix</td>\n",
       "      <td>0.190034</td>\n",
       "      <td>0.000475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Users Accuracy: Barren &amp; Shrubs Mix</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Users Accuracy: Grass/Forb/Herb</td>\n",
       "      <td>0.454836</td>\n",
       "      <td>0.001973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Users Accuracy: Barren &amp; Grass/Forb/Herb Mix</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Users Accuracy: Barren or Impervious</td>\n",
       "      <td>0.787648</td>\n",
       "      <td>0.001714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Users Accuracy: Snow or Ice</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Users Accuracy: Water</td>\n",
       "      <td>0.818476</td>\n",
       "      <td>0.000357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Producers Accuracy: Trees</td>\n",
       "      <td>0.950548</td>\n",
       "      <td>0.003822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Producers Accuracy: Tall Shrubs &amp; Trees Mix (S...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Producers Accuracy: Shrubs &amp; Trees Mix</td>\n",
       "      <td>0.006839</td>\n",
       "      <td>0.000109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Producers Accuracy: Grass/Forb/Herb &amp; Trees Mix</td>\n",
       "      <td>0.052331</td>\n",
       "      <td>0.000591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Producers Accuracy: Barren &amp; Trees Mix</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Producers Accuracy: Tall Shrubs (SEAK Only)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Producers Accuracy: Shrubs</td>\n",
       "      <td>0.104899</td>\n",
       "      <td>0.000529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Producers Accuracy: Grass/Forb/Herb &amp; Shrubs Mix</td>\n",
       "      <td>0.058463</td>\n",
       "      <td>0.000475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Producers Accuracy: Barren &amp; Shrubs Mix</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Producers Accuracy: Grass/Forb/Herb</td>\n",
       "      <td>0.570084</td>\n",
       "      <td>0.001973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Producers Accuracy: Barren &amp; Grass/Forb/Herb Mix</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Producers Accuracy: Barren or Impervious</td>\n",
       "      <td>0.815439</td>\n",
       "      <td>0.001714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Producers Accuracy: Snow or Ice</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Producers Accuracy: Water</td>\n",
       "      <td>0.695429</td>\n",
       "      <td>0.000357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Design-Based Area Estimation: Trees</td>\n",
       "      <td>0.579064</td>\n",
       "      <td>0.003842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Design-Based Area Estimation: Tall Shrubs &amp; Tr...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Design-Based Area Estimation: Shrubs &amp; Trees Mix</td>\n",
       "      <td>0.045360</td>\n",
       "      <td>0.001822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Design-Based Area Estimation: Grass/Forb/Herb ...</td>\n",
       "      <td>0.074527</td>\n",
       "      <td>0.002334</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Design-Based Area Estimation: Barren &amp; Trees Mix</td>\n",
       "      <td>0.001670</td>\n",
       "      <td>0.000335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Design-Based Area Estimation: Tall Shrubs (SEA...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Design-Based Area Estimation: Shrubs</td>\n",
       "      <td>0.035601</td>\n",
       "      <td>0.001544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Design-Based Area Estimation: Grass/Forb/Herb ...</td>\n",
       "      <td>0.044931</td>\n",
       "      <td>0.001825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Design-Based Area Estimation: Barren &amp; Shrubs Mix</td>\n",
       "      <td>0.002898</td>\n",
       "      <td>0.000427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Design-Based Area Estimation: Grass/Forb/Herb</td>\n",
       "      <td>0.103598</td>\n",
       "      <td>0.002534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Design-Based Area Estimation: Barren &amp; Grass/F...</td>\n",
       "      <td>0.006161</td>\n",
       "      <td>0.000667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Design-Based Area Estimation: Barren or Imperv...</td>\n",
       "      <td>0.095032</td>\n",
       "      <td>0.001976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Design-Based Area Estimation: Snow or Ice</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Design-Based Area Estimation: Water</td>\n",
       "      <td>0.011157</td>\n",
       "      <td>0.000554</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               Metric     Value  Confidence\n",
       "0                                            Accuracy  0.705311    0.003776\n",
       "1                                   Balanced Accuracy  0.217255         NaN\n",
       "2                                               Kappa  0.347829         NaN\n",
       "3                                            F1 Score  0.705311         NaN\n",
       "4                               Users Accuracy: Trees  0.770108    0.003822\n",
       "5   Users Accuracy: Tall Shrubs & Trees Mix (SEAK ...       NaN    0.000000\n",
       "6                  Users Accuracy: Shrubs & Trees Mix  0.234875    0.000109\n",
       "7         Users Accuracy: Grass/Forb/Herb & Trees Mix  0.201728    0.000591\n",
       "8                  Users Accuracy: Barren & Trees Mix       NaN    0.000000\n",
       "9             Users Accuracy: Tall Shrubs (SEAK Only)       NaN    0.000000\n",
       "10                             Users Accuracy: Shrubs  0.305139    0.000529\n",
       "11       Users Accuracy: Grass/Forb/Herb & Shrubs Mix  0.190034    0.000475\n",
       "12                Users Accuracy: Barren & Shrubs Mix  0.000000    0.000000\n",
       "13                    Users Accuracy: Grass/Forb/Herb  0.454836    0.001973\n",
       "14       Users Accuracy: Barren & Grass/Forb/Herb Mix  0.000000    0.000000\n",
       "15               Users Accuracy: Barren or Impervious  0.787648    0.001714\n",
       "16                        Users Accuracy: Snow or Ice       NaN    0.000000\n",
       "17                              Users Accuracy: Water  0.818476    0.000357\n",
       "18                          Producers Accuracy: Trees  0.950548    0.003822\n",
       "19  Producers Accuracy: Tall Shrubs & Trees Mix (S...       NaN    0.000000\n",
       "20             Producers Accuracy: Shrubs & Trees Mix  0.006839    0.000109\n",
       "21    Producers Accuracy: Grass/Forb/Herb & Trees Mix  0.052331    0.000591\n",
       "22             Producers Accuracy: Barren & Trees Mix  0.000000    0.000000\n",
       "23        Producers Accuracy: Tall Shrubs (SEAK Only)       NaN    0.000000\n",
       "24                         Producers Accuracy: Shrubs  0.104899    0.000529\n",
       "25   Producers Accuracy: Grass/Forb/Herb & Shrubs Mix  0.058463    0.000475\n",
       "26            Producers Accuracy: Barren & Shrubs Mix  0.000000    0.000000\n",
       "27                Producers Accuracy: Grass/Forb/Herb  0.570084    0.001973\n",
       "28   Producers Accuracy: Barren & Grass/Forb/Herb Mix  0.000000    0.000000\n",
       "29           Producers Accuracy: Barren or Impervious  0.815439    0.001714\n",
       "30                    Producers Accuracy: Snow or Ice       NaN    0.000000\n",
       "31                          Producers Accuracy: Water  0.695429    0.000357\n",
       "32                Design-Based Area Estimation: Trees  0.579064    0.003842\n",
       "33  Design-Based Area Estimation: Tall Shrubs & Tr...  0.000000    0.000000\n",
       "34   Design-Based Area Estimation: Shrubs & Trees Mix  0.045360    0.001822\n",
       "35  Design-Based Area Estimation: Grass/Forb/Herb ...  0.074527    0.002334\n",
       "36   Design-Based Area Estimation: Barren & Trees Mix  0.001670    0.000335\n",
       "37  Design-Based Area Estimation: Tall Shrubs (SEA...  0.000000    0.000000\n",
       "38               Design-Based Area Estimation: Shrubs  0.035601    0.001544\n",
       "39  Design-Based Area Estimation: Grass/Forb/Herb ...  0.044931    0.001825\n",
       "40  Design-Based Area Estimation: Barren & Shrubs Mix  0.002898    0.000427\n",
       "41      Design-Based Area Estimation: Grass/Forb/Herb  0.103598    0.002534\n",
       "42  Design-Based Area Estimation: Barren & Grass/F...  0.006161    0.000667\n",
       "43  Design-Based Area Estimation: Barren or Imperv...  0.095032    0.001976\n",
       "44          Design-Based Area Estimation: Snow or Ice  0.000000    0.000000\n",
       "45                Design-Based Area Estimation: Water  0.011157    0.000554"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metric</th>\n",
       "      <th>Value</th>\n",
       "      <th>Confidence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Accuracy</td>\n",
       "      <td>0.814593</td>\n",
       "      <td>0.003218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Balanced Accuracy</td>\n",
       "      <td>0.486290</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Kappa</td>\n",
       "      <td>0.592546</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F1 Score</td>\n",
       "      <td>0.814593</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Users Accuracy: Agriculture</td>\n",
       "      <td>0.414056</td>\n",
       "      <td>0.000319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Users Accuracy: Developed</td>\n",
       "      <td>0.858515</td>\n",
       "      <td>0.002583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Users Accuracy: Forest</td>\n",
       "      <td>0.854253</td>\n",
       "      <td>0.003687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Users Accuracy: Non-Forest Wetland</td>\n",
       "      <td>0.425218</td>\n",
       "      <td>0.000474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Users Accuracy: Other</td>\n",
       "      <td>0.920012</td>\n",
       "      <td>0.000369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Users Accuracy: Rangeland or Pasture</td>\n",
       "      <td>0.647561</td>\n",
       "      <td>0.002611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Producers Accuracy: Agriculture</td>\n",
       "      <td>0.099354</td>\n",
       "      <td>0.000319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Producers Accuracy: Developed</td>\n",
       "      <td>0.811577</td>\n",
       "      <td>0.002583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Producers Accuracy: Forest</td>\n",
       "      <td>0.935557</td>\n",
       "      <td>0.003687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Producers Accuracy: Non-Forest Wetland</td>\n",
       "      <td>0.340653</td>\n",
       "      <td>0.000474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Producers Accuracy: Other</td>\n",
       "      <td>0.583743</td>\n",
       "      <td>0.000369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Producers Accuracy: Rangeland or Pasture</td>\n",
       "      <td>0.581939</td>\n",
       "      <td>0.002611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Design-Based Area Estimation: Agriculture</td>\n",
       "      <td>0.022053</td>\n",
       "      <td>0.000991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Design-Based Area Estimation: Developed</td>\n",
       "      <td>0.191608</td>\n",
       "      <td>0.002873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Design-Based Area Estimation: Forest</td>\n",
       "      <td>0.574897</td>\n",
       "      <td>0.003722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Design-Based Area Estimation: Non-Forest Wetland</td>\n",
       "      <td>0.016693</td>\n",
       "      <td>0.000974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Design-Based Area Estimation: Other</td>\n",
       "      <td>0.016747</td>\n",
       "      <td>0.000734</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Design-Based Area Estimation: Rangeland or Pas...</td>\n",
       "      <td>0.178001</td>\n",
       "      <td>0.003194</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               Metric     Value  Confidence\n",
       "0                                            Accuracy  0.814593    0.003218\n",
       "1                                   Balanced Accuracy  0.486290         NaN\n",
       "2                                               Kappa  0.592546         NaN\n",
       "3                                            F1 Score  0.814593         NaN\n",
       "4                         Users Accuracy: Agriculture  0.414056    0.000319\n",
       "5                           Users Accuracy: Developed  0.858515    0.002583\n",
       "6                              Users Accuracy: Forest  0.854253    0.003687\n",
       "7                  Users Accuracy: Non-Forest Wetland  0.425218    0.000474\n",
       "8                               Users Accuracy: Other  0.920012    0.000369\n",
       "9                Users Accuracy: Rangeland or Pasture  0.647561    0.002611\n",
       "10                    Producers Accuracy: Agriculture  0.099354    0.000319\n",
       "11                      Producers Accuracy: Developed  0.811577    0.002583\n",
       "12                         Producers Accuracy: Forest  0.935557    0.003687\n",
       "13             Producers Accuracy: Non-Forest Wetland  0.340653    0.000474\n",
       "14                          Producers Accuracy: Other  0.583743    0.000369\n",
       "15           Producers Accuracy: Rangeland or Pasture  0.581939    0.002611\n",
       "16          Design-Based Area Estimation: Agriculture  0.022053    0.000991\n",
       "17            Design-Based Area Estimation: Developed  0.191608    0.002873\n",
       "18               Design-Based Area Estimation: Forest  0.574897    0.003722\n",
       "19   Design-Based Area Estimation: Non-Forest Wetland  0.016693    0.000974\n",
       "20                Design-Based Area Estimation: Other  0.016747    0.000734\n",
       "21  Design-Based Area Estimation: Rangeland or Pas...  0.178001    0.003194"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get each output .txt file and open it \n",
    "results_files = glob.glob(os.path.join(local_map_acc_folder,'*Accuracy*csv'))\n",
    "for results_file in results_files:\n",
    "    results = pd.read_csv(results_file)\n",
    "    display(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c44c1bc7",
   "metadata": {},
   "source": [
    "### 7.3.5: Interpret results\n",
    "* Notice the accuracies are low for many classes - especially change classes\n",
    "* Stable generally has high accuracy, and skews the overall accuracy, but the balanced accuracy and the user's and producer's accuracy are low. \n",
    "* While it is true there is a lot of disagreement between our TimeSync-based reference data and our maps, this has been shown to be the reality throughout the literature\n",
    "    * E.g. \n",
    "        * [Cohen et al 2018](./lit/Cohen_et_al_2018_Landtrendr_Ensemble.pdf)\n",
    "            * Tested various LandTrendr band outputs for detecting change. Omission error rates were between ~0.45 and ~0.60 and commission error rates were > 0.825.\n",
    "        * [Healey et al 2018](./lit/Healey_et_al_2018_RSE_LCMS_Approach.pdf)\n",
    "            * Tested various change algorithms for predicting change much like LCMS does now. The best output had a balanced omission and commission error rate of 0.4.\n",
    "        * [Stehman et al 2021](./lit/Stehman_et_al_2021_LCMAP_Acc.pdf)\n",
    "            * Used an independent TimeSync-based reference dataset to validate USGS Land Change Monitoring Assessment and Projection (LCMAP) outputs. The change (any spectral break by CCDC - covered in Module 3), had a commission error rate of 0.86 and omission error rate of 0.83 (Table 8).\n",
    "    * Each of these papers relied on TimeSync-based reference data that had analysts include every piece of information available through the use of tools such as Google Earth's high resolution time lapse, other additional high resolution aerial photography, ancillary datasets such as the Monitoring Trends in Burn Serverity (MTBS), etc.\n",
    "    * We can never expect a model-based output to match this level of detail.\n",
    "    * The question becomes whether the maps are useful and do they at least align with the literature that employed similar methods.\n",
    "    * Reviewing the maps, we can see they are a reasonable depiction of what they intend to represent, but should be used with these error rates in mind. They are generally appropriate to use over larger areas, but not for your favorite pixel.\n",
    "    * Additional considerations is often with time series mapping, correct classifications will be allowed if the model predicted the class in the exact year or an adjacent year. The method outlined in this module did not employ such methods.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf28db5",
   "metadata": {},
   "source": [
    "## 7.4 Copy outputs to Google Cloud Storage\n",
    "### 7.4.1 Set up Google Cloud Storage Bucket\n",
    "\n",
    "* This step will set up a location in Google Cloud Storage for the accuracy results `.txt` files to be copied to\n",
    "* You can set up your own export location by specifying a unique name for the bucket\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "77a6309e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Creating gs://lcms-training-outputs-accuracy/...\n",
      "ServiceException: 409 A Cloud Storage bucket named 'lcms-training-outputs-accuracy' already exists. Try another name. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.\n"
     ]
    }
   ],
   "source": [
    "pre_baked_accuracy_output_bucket = f'gs://lcms-training-outputs-accuracy'\n",
    "\n",
    "projectID = 'qwiklabs-gcp-00-00000000' # change this to match your project id\n",
    "personal_accuracy_output_bucket = f'{projectID}-outputs-accuracy'\n",
    "\n",
    "# set accuracy output bucket \n",
    "accuracy_output_bucket = pre_baked_accuracy_output_bucket\n",
    "\n",
    "# You may need to run these functions if running outside Workbench\n",
    "# In order to avoid permissions issues, ensure the projectName is the same as the one you used for authenticating \n",
    "# to GEE\n",
    "# !gcloud auth login\n",
    "# !gcloud projects list\n",
    "# !gcloud config set project projectName\n",
    "\n",
    "!gsutil mb {accuracy_output_bucket}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf861e85",
   "metadata": {},
   "source": [
    "### 7.4.2: Copy outputs to Google Cloud Storage\n",
    "\n",
    "* Copy outputs to a bucket. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9f7197e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Copying file://\\tmp\\lcms-training\\local_map_accuracy\\LCMS_Land_Cover_Map_Accuracy_Results.csv [Content-Type=application/vnd.ms-excel]...\n",
      "Copying file://\\tmp\\lcms-training\\local_map_accuracy\\LCMS_Land_Use_Map_Accuracy_Results.csv [Content-Type=application/vnd.ms-excel]...\n",
      "/ [0/6 files][    0.0 B/  8.0 KiB]   0% Done                                    \n",
      "Copying file://\\tmp\\lcms-training\\local_map_accuracy\\LCMS_Change_Map_Accuracy_Results.csv [Content-Type=application/vnd.ms-excel]...\n",
      "/ [0/6 files][    0.0 B/  8.0 KiB]   0% Done                                    \n",
      "/ [0/6 files][    0.0 B/  8.0 KiB]   0% Done                                    \n",
      "Copying file://\\tmp\\lcms-training\\local_map_accuracy\\LCMS_Change_Map_Confusion_Matrix.csv [Content-Type=application/vnd.ms-excel]...\n",
      "Copying file://\\tmp\\lcms-training\\local_map_accuracy\\LCMS_Land_Cover_Map_Confusion_Matrix.csv [Content-Type=application/vnd.ms-excel]...\n",
      "/ [0/6 files][    0.0 B/  8.0 KiB]   0% Done                                    \n",
      "/ [0/6 files][    0.0 B/  8.0 KiB]   0% Done                                    \n",
      "Copying file://\\tmp\\lcms-training\\local_map_accuracy\\LCMS_Land_Use_Map_Confusion_Matrix.csv [Content-Type=application/vnd.ms-excel]...\n",
      "/ [0/6 files][    0.0 B/  8.0 KiB]   0% Done                                    \n",
      "/ [1/6 files][  8.0 KiB/  8.0 KiB]  99% Done                                    \n",
      "/ [2/6 files][  8.0 KiB/  8.0 KiB]  99% Done                                    \n",
      "/ [3/6 files][  8.0 KiB/  8.0 KiB]  99% Done                                    \n",
      "/ [4/6 files][  8.0 KiB/  8.0 KiB]  99% Done                                    \n",
      "-\n",
      "- [5/6 files][  8.0 KiB/  8.0 KiB]  99% Done                                    \n",
      "- [6/6 files][  8.0 KiB/  8.0 KiB] 100% Done                                    \n",
      "\n",
      "Operation completed over 6 objects/8.0 KiB.                                      \n"
     ]
    }
   ],
   "source": [
    "!gsutil -m cp {local_map_acc_folder}/*.csv {accuracy_output_bucket}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "38ac4281",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://console.cloud.google.com/storage/browser/lcms-training-outputs-accuracy\n"
     ]
    }
   ],
   "source": [
    "# Open this link to view the accuracy outputs\n",
    "print(f'https://console.cloud.google.com/storage/browser/{os.path.basename(accuracy_output_bucket)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "937fed79",
   "metadata": {},
   "source": [
    "## Lab 7 challenge:\n",
    "\n",
    "**If you are accessing this lab through Qwiklabs, this challenge will be assessed for completion in Lab 7.**\n",
    "1. Change the thresholds used to find fast loss and gain and apply to the raw predicted Change probabilities\n",
    "\n",
    "    * Use `{'Slow Loss':0,'Fast Loss': 0.05, 'Gain': 0.15}` as your thresholds\n",
    "\n",
    "    * Use the `raw_change_preds` variable from earlier in the module as your raw change predictions\n",
    "\n",
    "    * Use many of the same methods as used section 7.1.5\n",
    "<br>\n",
    "\n",
    "2. Create a confusion matrix of the resulting assembled predicted \n",
    "\n",
    "    * Use `ref_pred_dict['Change']['refs']` as your observed values\n",
    "\n",
    "    * Use the `asl.getConfusionMatrix` function to get the confusion matrix\n",
    "        \n",
    "    * Take note the results change very very little from the results from section 7.2.1.\n",
    "    <br> \n",
    "3. Save confusion matrix to a csv file.\n",
    "\n",
    "   * Save csv to this path: `\"/tmp/challenge/module_7_challenge_answer.csv\"`\n",
    "     * **Note: The path to the csv must exactly match the path above.** \n",
    "    <br>\n",
    "    \n",
    "    * Create the `\"/tmp/challenge\"` folder if it does not already exist.\n",
    "      \n",
    "        Example:\n",
    "    ```python\n",
    "        out_csv = \"/tmp/challenge/module_7_challenge_answer.csv\"\n",
    "        if not os.path.exists(os.path.dirname(out_csv)):os.makedirs(os.path.dirname(out_csv))\n",
    "    ```\n",
    "<br>\n",
    "\n",
    "4.  Check that the output csv exists.\n",
    "    \n",
    "    * Example: \n",
    "    ```python\n",
    "        print(os.path.exists(out_csv))\n",
    "    ```\n",
    "<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1af39663",
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert challenge code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59822586",
   "metadata": {},
   "source": [
    "## Done with Module 7"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-11.m111",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-11:m111"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}