aq_by_weather.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ae929fb3-0316-494b-accb-1951756d4d32",
   "metadata": {
    "tags": []
   },
   "source": [
    "# ML Demo: Predicting Air Quality w/ ASDI NOAA + OpenAQ Datasets\n",
    "This demo consists of a Jupyter Notebook that connects to Amazon Sustainability Data Initiative (ASDI) datasets from NOAA and OpenAQ to build a Machine Learning (ML) model to predict air quality levels using weather data via a Binary Classification [AutoGluon](https://auto.gluon.ai/stable/index.html) model. The project's purpose is to demonstrate using two different types of ASDI datasets (files in Amazon S3 and HTTPS APIs) within a Jupyter Notebook, such as provided by Amazon SageMaker Studio Lab. This demo is NOT for scientific or health purposes.\n",
    "\n",
    "This Jupyter Notebook can be run for free using Amazon SageMaker Studio Lab and open-source Amazon Sustainability Data Initiative (ASDI) datasets without needing an AWS account.\n",
    "\n",
    "[![Open in SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/aws-samples/aws-smsl-predict-airquality-via-weather/blob/main/aq_by_weather.ipynb)\n",
    "\n",
    "## BACKGROUND: 1 out of 8 deaths in the world is due to poor air quality*\n",
    "This demo explores correlations between weather and air quality since we know factors like temperatures, wind speeds, etc, affect certain air quality parameters. Predicting air quality can get into highly sophisticated ML techniques, but this demo shows how merging NOAA GSOD weather data with OpenAQ air quality data to build an ML model using AutoGluon (AutoML from AWS) can result in prediction accuracy of ~75-90% using Binary Classification models for various high-pollution areas and air quality parameters (mostly tested for 2.5 micron Particulate Matter and some Ground Level Ozone).\\\n",
    "*Source: [OpenAQ.org](https://OpenAQ.org)\n",
    "\n",
    "## INSTRUCTIONS\n",
    "- Configure your environment using the repo's environment.yml file or by running the *pip install* commands below.\n",
    "- If you want to run this Notebook yourself, first use the _Clear All Outputs_ option.\n",
    "- CELL #3: Review the Classes and Variables defined in this cell to see defaults and how ASDI access occurs.\n",
    "- CELL #4: Review the pre-defined AQParams and AQScenarios in this cell. You can edit these and/or add your own.\n",
    "- CELL #5: Select a scenario via the drop-down to use throughout the Notebook. This will drive the ML process.\n",
    "- Step through the remaining cells in the Notebook to access data, merge data, and build a Binary Classification Autogluon model.\n",
    "- NOTE: NOAA and OpenAQ input data is saved locally as CSV files and re-used in order to minimize unnecessary data access and compute resources. Be aware of this feature and clear the local data files, if your intention is to re-fetch the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5a41f10b-f9ed-433a-8d32-bb718f23d811",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# Set up your environment according to the repo's environment.yml file or run the following...\n",
    "# Comment these out, once installed or otherwise not needed.\n",
    "# This creates an empty pip_requirements.txt file used to suppress 'already satisfied' output.\n",
    "import os\n",
    "with open('pip_requirements.txt', mode='a'): pass\n",
    "%pip install boto3        -r pip_requirements.txt | grep -v 'already satisfied'\n",
    "%pip install pandas       -r pip_requirements.txt | grep -v 'already satisfied'\n",
    "%pip install numpy        -r pip_requirements.txt | grep -v 'already satisfied'\n",
    "%pip install requests     -r pip_requirements.txt | grep -v 'already satisfied'\n",
    "%pip install ipywidgets   -r pip_requirements.txt | grep -v 'already satisfied'\n",
    "%pip install scikit-learn -r pip_requirements.txt | grep -v 'already satisfied'\n",
    "%pip install autogluon    -r pip_requirements.txt | grep -v 'already satisfied'\n",
    "%pip install matplotlib   -r pip_requirements.txt | grep -v 'already satisfied'\n",
    "%pip install nbconvert    -r pip_requirements.txt | grep -v 'already satisfied'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e844d780-d995-4bdc-98ae-570bc8253265",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import statements for packages used...\n",
    "import os, glob, shutil, sys, requests, json\n",
    "import boto3\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import ipywidgets as widgets\n",
    "\n",
    "from botocore import UNSIGNED\n",
    "from botocore.config import Config\n",
    "from io import StringIO\n",
    "from datetime import datetime\n",
    "from types import SimpleNamespace\n",
    "from IPython.display import clear_output\n",
    "\n",
    "# The following is required for matplotlib plots to display in some envs...\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9ea427ea-8ce7-4dc5-80e1-c674f7f569ee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classes and Variables are ready.\n"
     ]
    }
   ],
   "source": [
    "# CELL #3: Review the Variables and Classes defined in this cell to see defaults and how ASDI access occurs...\n",
    "\n",
    "# class AQParam => Used to define attributes for the (6) main OpenAQ parameters.\n",
    "class AQParam:\n",
    "    def __init__(self, id, name, unit, unhealthyThresholdDefault, desc):\n",
    "        self.id                        = id\n",
    "        self.name                      = name\n",
    "        self.unit                      = unit\n",
    "        self.unhealthyThresholdDefault = unhealthyThresholdDefault\n",
    "        self.desc                      = desc\n",
    "    \n",
    "    def isValid(self):\n",
    "        if(self is not None and self.id > 0 and self.unhealthyThresholdDefault > 0.0 and \n",
    "           len(self.name) > 0 and len(self.unit) > 0 and len(self.desc) > 0):\n",
    "            return True\n",
    "        else:\n",
    "            return False\n",
    "            \n",
    "    def toJSON(self):\n",
    "        return json.dumps(self, default=lambda o: o.__dict__, sort_keys=True, indent=2)\n",
    "\n",
    "# class AQScenario => Defines an ML scenario including a Location w/ NOAA Weather Station ID \n",
    "#                     and the target OpenAQ Param.\n",
    "# Note: OpenAQ data mostly begins sometime in 2016, so using that as a default yearStart value.\n",
    "class AQScenario:\n",
    "    def __init__(self, location=None, noaaStationID=None, aqParamTarget=None, unhealthyThreshold=None, \n",
    "                 yearStart=2016, yearEnd=2022, aqRadiusMiles=10, featureColumnsToDrop=None):\n",
    "        self.location           = location\n",
    "        self.name               = location + \"_\" + aqParamTarget.name\n",
    "        self.noaaStationID      = noaaStationID\n",
    "        self.noaaStationLat     = 0.0\n",
    "        self.noaaStationLng     = 0.0\n",
    "        self.openAqLocIDs       = []\n",
    "        \n",
    "        self.aqParamTarget      = aqParamTarget\n",
    "        \n",
    "        if unhealthyThreshold and unhealthyThreshold > 0.0:\n",
    "            self.unhealthyThreshold = unhealthyThreshold\n",
    "        else:\n",
    "            self.unhealthyThreshold = self.aqParamTarget.unhealthyThresholdDefault\n",
    "        \n",
    "        self.yearStart          = yearStart\n",
    "        self.yearEnd            = yearEnd\n",
    "        self.aqRadiusMiles      = aqRadiusMiles\n",
    "        self.aqRadiusMeters     = aqRadiusMiles * 1610 # Rough integer approximation is fine here.\n",
    "        \n",
    "        self.modelFolder        = \"AutogluonModels\"\n",
    "            \n",
    "    def getSummary(self):\n",
    "        return f\"Scenario: {self.name} => {self.aqParamTarget.desc} ({self.aqParamTarget.name}) with UnhealthyThreshold > {self.unhealthyThreshold} {self.aqParamTarget.unit}\"\n",
    "    \n",
    "    def getModelPath(self):\n",
    "        return f\"{self.modelFolder}/aq_{self.name}_{self.yearStart}-{self.yearEnd}/\"\n",
    "    \n",
    "    def updateNoaaStationLatLng(self, noaagsod_df_row):\n",
    "        # Use a NOAA row to set Lat+Lng values used for the OpenAQ API requests...\n",
    "        if(noaagsod_df_row is not None and noaagsod_df_row['LATITUDE'] and noaagsod_df_row['LONGITUDE']):\n",
    "            self.noaaStationLat = noaagsod_df_row['LATITUDE']\n",
    "            self.noaaStationLng = noaagsod_df_row['LONGITUDE']\n",
    "            print(f\"NOAA Station Lat,Lng Updated for Scenario: {self.name} => {self.noaaStationLat},{self.noaaStationLng}\")\n",
    "        else:\n",
    "            print(\"NOAA Station Lat,Lng COULD NOT BE UPDATED.\")\n",
    "    \n",
    "    def isValid(self):\n",
    "        if(self is not None and self.aqParamTarget is not None and\n",
    "           self.yearStart > 0 and self.yearEnd > 0 and self.yearEnd >= self.yearStart and \n",
    "           self.aqRadiusMiles > 0 and self.aqRadiusMeters > 0 and self.unhealthyThreshold > 0.0 and \n",
    "           len(self.name) > 0 and len(self.noaaStationID) > 0):\n",
    "            return True\n",
    "        else:\n",
    "            return False\n",
    "            \n",
    "    def toJSON(self):\n",
    "        return json.dumps(self, default=lambda o: o.__dict__, sort_keys=True, indent=2)\n",
    "\n",
    "# class AQbyWeatherApp => Main app class with settings, AQParams, AQScenarios, and data access methods...\n",
    "class AQbyWeatherApp:\n",
    "    def __init__(self, mlTargetLabel='isUnhealthy', mlEvalMetric='accuracy', mlTimeLimitSecs=None):\n",
    "        self.mlTargetLabel   = mlTargetLabel\n",
    "        self.mlEvalMetric    = mlEvalMetric\n",
    "        self.mlTimeLimitSecs = mlTimeLimitSecs\n",
    "        self.mlIgnoreColumns = ['DATE','NAME','LATITUDE','LONGITUDE','day','unit','average','parameter']\n",
    "        \n",
    "        self.defaultColumnsNOAA   = ['DATE','NAME','LATITUDE','LONGITUDE',\n",
    "                                     'DEWP','WDSP','MAX','MIN','PRCP','MONTH'] # Default relevant NOAA columns\n",
    "        self.defaultColumnsOpenAQ = ['day','parameter','unit','average']       # Default relevant OpenAQ columns\n",
    "        \n",
    "        self.aqParams    = {} # A list to save AQParam objects\n",
    "        self.aqScenarios = {} # A list to save AQScenario objects\n",
    "        \n",
    "        self.selectedScenario = None\n",
    "    \n",
    "    def addAQParam(self, aqParam):\n",
    "        if aqParam and aqParam.isValid():\n",
    "            self.aqParams[aqParam.name] = aqParam\n",
    "            return True\n",
    "        else:\n",
    "            return False\n",
    "    \n",
    "    def addAQScenario(self, aqScenario):\n",
    "        if aqScenario and aqScenario.isValid():\n",
    "            self.aqScenarios[aqScenario.name] = aqScenario\n",
    "            if(self.selectedScenario is None):\n",
    "                self.selectedScenario = self.aqScenarios[next(iter(self.aqScenarios))] # Default selectedScenario to 1st item.\n",
    "            return True\n",
    "        else:\n",
    "            return False\n",
    "    \n",
    "    def getFilenameNOAA(self):\n",
    "        if self and self.selectedScenario and self.selectedScenario.isValid():\n",
    "            return f\"dataNOAA_{self.selectedScenario.name}_{self.selectedScenario.yearStart}-{self.selectedScenario.yearEnd}_{self.selectedScenario.noaaStationID}.csv\"\n",
    "        else:\n",
    "            return \"\"\n",
    "    \n",
    "    def getFilenameOpenAQ(self):\n",
    "        if self and self.selectedScenario and self.selectedScenario.isValid() and len(self.selectedScenario.openAqLocIDs) > 0:\n",
    "            idString = \"\"\n",
    "            for i in range(0, len(self.selectedScenario.openAqLocIDs)):\n",
    "                idString = idString + str(self.selectedScenario.openAqLocIDs[i]) + \"-\"\n",
    "            idString = idString[:-1]\n",
    "            return f\"dataOpenAQ_{self.selectedScenario.name}_{self.selectedScenario.yearStart}-{self.selectedScenario.yearEnd}_{idString}.csv\"\n",
    "        else:\n",
    "            return \"\"\n",
    "    \n",
    "    def getFilenameOther(self, prefix):\n",
    "        if self and self.selectedScenario and self.selectedScenario.isValid():\n",
    "            return f\"{prefix}_{self.selectedScenario.name}_{self.selectedScenario.yearStart}-{self.selectedScenario.yearEnd}.csv\"\n",
    "    \n",
    "    def getNoaaDataFrame(self):\n",
    "        # ASDI Dataset Name: NOAA GSOD\n",
    "        # ASDI Dataset URL : https://registry.opendata.aws/noaa-gsod/\n",
    "        # NOAA GSOD README : https://www.ncei.noaa.gov/data/global-summary-of-the-day/doc/readme.txt\n",
    "        # NOAA GSOD data in S3 is organized by year and Station ID values, so this is straight-forward\n",
    "        # Example S3 path format => s3://noaa-gsod-pds/{yyyy}/{stationid}.csv\n",
    "        # Let's start with a new DataFrame and load it from a local CSV or the NOAA data source...\n",
    "        noaagsod_df = pd.DataFrame()\n",
    "        filenameNOAA = self.getFilenameNOAA()\n",
    "\n",
    "        if os.path.exists(filenameNOAA):\n",
    "            # Use local data file already accessed + prepared...\n",
    "            print('Loading NOAA GSOD data from local file: ', filenameNOAA)\n",
    "            noaagsod_df = pd.read_csv(filenameNOAA)\n",
    "        else:\n",
    "            # Access + prepare data and save to a local data file...\n",
    "            noaagsod_bucket = 'noaa-gsod-pds'\n",
    "            print(f'Accessing and preparing data from ASDI-hosted NOAA GSOD dataset in Amazon S3 (bucket: {noaagsod_bucket})...')\n",
    "            s3 = boto3.client('s3', config=Config(signature_version=UNSIGNED))\n",
    "\n",
    "            for year in range(self.selectedScenario.yearStart, self.selectedScenario.yearEnd + 1):\n",
    "                key = f'{year}/{self.selectedScenario.noaaStationID}.csv'                                    # Compute the key to get\n",
    "                csv_obj = s3.get_object(Bucket=noaagsod_bucket, Key=key)                                     # Get the S3 object\n",
    "                csv_string = csv_obj['Body'].read().decode('utf-8')                                          # Read object contents to a string\n",
    "                noaagsod_df = pd.concat([noaagsod_df, pd.read_csv(StringIO(csv_string))], ignore_index=True) # Use the string to build the DataFrame\n",
    "\n",
    "            # Perform some Feature Engineering to append potentially useful columns to our dataset... (TODO: Add more to optimize...)\n",
    "            # It may be true that Month affects air quality (ie: seasonal considerations; tends to have correlation for certain areas)\n",
    "            noaagsod_df['MONTH'] = pd.to_datetime(noaagsod_df['DATE']).dt.month\n",
    "\n",
    "            # Trim down to the desired key columns... (do this last in case engineered columns are to be removed)\n",
    "            noaagsod_df = noaagsod_df[self.defaultColumnsNOAA]\n",
    "            \n",
    "        return noaagsod_df\n",
    "        \n",
    "    def getOpenAqDataFrame(self):\n",
    "        # ASDI Dataset Name: OpenAQ\n",
    "        # ASDI Dataset URL : https://registry.opendata.aws/openaq/\n",
    "        # OpenAQ API Docs  : https://docs.openaq.org/#/v2/\n",
    "        # OpenAQ S3 data is only organized by date folders, so each folder is large and contains all stations.\n",
    "        # Because of this, it's better to query ASDI OpenAQ data using the CloudFront-hosted API.\n",
    "        # Note that some days may not have values and will get filtered out via an INNER JOIN later.\n",
    "        # Let's start with a new DataFrame and load it from a local CSV or the NOAA data source...\n",
    "        aq_df = pd.DataFrame()\n",
    "        aq_reqUrlBase = \"https://api.openaq.org/v2\" # OpenAQ ASDI API Endpoint URL Base (ie: add /locations OR /averages)\n",
    "        \n",
    "        if self.selectedScenario.noaaStationLat == 0.0 or self.selectedScenario.noaaStationLng == 0.0:\n",
    "            print(\"NOAA Station Lat/Lng NOT DEFINED. CANNOT PROCEED\")\n",
    "            return aq_df\n",
    "        \n",
    "        if len(self.selectedScenario.openAqLocIDs) == 0:\n",
    "            # We must start by querying nearby OpenAQ Locations for their IDs...\n",
    "            print('Accessing ASDI-hosted OpenAQ Locations (HTTPS API)...')\n",
    "            aq_reqParams = {\n",
    "                'limit': 10,\n",
    "                'page': 1,\n",
    "                'offset': 0,\n",
    "                'sort': 'desc',\n",
    "                'order_by': 'location',\n",
    "                'parameter': self.selectedScenario.aqParamTarget.name,\n",
    "                'coordinates': f'{self.selectedScenario.noaaStationLat},{self.selectedScenario.noaaStationLng}',\n",
    "                'radius': self.selectedScenario.aqRadiusMeters,\n",
    "                'isMobile': 'false',\n",
    "                'sensorType': 'reference grade',\n",
    "                'dumpRaw': 'false'\n",
    "            }\n",
    "            aq_resp = requests.get(aq_reqUrlBase + \"/locations\", aq_reqParams)\n",
    "            aq_data = aq_resp.json()\n",
    "            if aq_data['results'] and len(aq_data['results']) >= 1:\n",
    "                for i in range(0, len(aq_data['results'])):\n",
    "                    self.selectedScenario.openAqLocIDs.append(aq_data['results'][i]['id'])\n",
    "                print(f\"OpenAQ Location IDs within {self.selectedScenario.aqRadiusMiles} miles ({self.selectedScenario.aqRadiusMeters}m) \" +\n",
    "                      f\"of NOAA Station {self.selectedScenario.noaaStationID} at \" +\n",
    "                      f\"{self.selectedScenario.noaaStationLat},{self.selectedScenario.noaaStationLng}: {self.selectedScenario.openAqLocIDs}\")\n",
    "            else:\n",
    "                print(f\"NO OpenAQ Location IDs found within {self.selectedScenario.aqRadiusMiles} miles ({self.selectedScenario.aqRadiusMeters}m) \" +\n",
    "                      f\"of NOAA Station {self.selectedScenario.noaaStationID}. CANNOT PROCEED.\")\n",
    "        \n",
    "        if len(self.selectedScenario.openAqLocIDs) >= 1:\n",
    "            filenameOpenAQ = self.getFilenameOpenAQ()\n",
    "\n",
    "            if os.path.exists(filenameOpenAQ):\n",
    "                # Use local data file already accessed + prepared...\n",
    "                print('Loading OpenAQ data from local file: ', filenameOpenAQ)\n",
    "                aq_df = pd.read_csv(filenameOpenAQ)\n",
    "            else:\n",
    "                # Access + prepare data (NOTE: calling OpenAQ API one year at a time to avoid timeouts)\n",
    "                print('Accessing ASDI-hosted OpenAQ Averages (HTTPS API)...')\n",
    "\n",
    "                for year in range(self.selectedScenario.yearStart, self.selectedScenario.yearEnd + 1):\n",
    "                    aq_reqParams = {\n",
    "                        'date_from': f'{year}-01-01',\n",
    "                        'date_to': f'{year}-12-31',\n",
    "                        'parameter': [self.selectedScenario.aqParamTarget.name],\n",
    "                        'limit': 366,\n",
    "                        'page': 1,\n",
    "                        'offset': 0,\n",
    "                        'sort': 'asc',\n",
    "                        'spatial': 'location',\n",
    "                        'temporal': 'day',\n",
    "                        'location': self.selectedScenario.openAqLocIDs,\n",
    "                        'group': 'true'\n",
    "                    }\n",
    "                    aq_resp = requests.get(aq_reqUrlBase + \"/averages\", aq_reqParams)\n",
    "                    aq_data = aq_resp.json()\n",
    "                    if aq_data['results'] and len(aq_data['results']) >= 1:\n",
    "                        aq_df = pd.concat([aq_df, pd.json_normalize(aq_data['results'])], ignore_index=True)\n",
    "\n",
    "                # Using the joined dataframe, trim down to the desired key columns...\n",
    "                aq_df = aq_df[self.defaultColumnsOpenAQ]\n",
    "\n",
    "                # Perform some Label Engineering to add our binary classification label => {0=OKAY, 1=UNHEALTHY}\n",
    "                aq_df[self.mlTargetLabel] = np.where(aq_df['average'] <= self.selectedScenario.unhealthyThreshold, 0, 1)\n",
    "        \n",
    "        return aq_df\n",
    "    \n",
    "    def getMergedDataFrame(self, noaagsod_df, aq_df):\n",
    "        if len(noaagsod_df) > 0 and len(aq_df) > 0:\n",
    "            merged_df = pd.merge(noaagsod_df, aq_df, how=\"inner\", left_on=\"DATE\", right_on=\"day\")\n",
    "            merged_df = merged_df.drop(columns=self.mlIgnoreColumns)\n",
    "            return merged_df\n",
    "        else:\n",
    "            return pd.DataFrame()\n",
    "    \n",
    "    def getConfusionMatrixData(self, cm):\n",
    "        cmData = SimpleNamespace()\n",
    "        cmData.TN = cm[0][0]\n",
    "        cmData.TP = cm[1][1]\n",
    "        cmData.FN = cm[1][0]\n",
    "        cmData.FP = cm[0][1]\n",
    "        \n",
    "        cmData.TN_Rate = cmData.TN/(cmData.TN+cmData.FP)\n",
    "        cmData.TP_Rate = cmData.TP/(cmData.TP+cmData.FN)\n",
    "        cmData.FN_Rate = cmData.FN/(cmData.FN+cmData.TP)\n",
    "        cmData.FP_Rate = cmData.FP/(cmData.FP+cmData.TN)\n",
    "        \n",
    "        cmData.TN_Output = f\"True Negatives  (TN): {cmData.TN} of {cmData.TN+cmData.FP} => {round(cmData.TN_Rate * 100, 2)}%\"\n",
    "        cmData.TP_Output = f\"True Positives  (TP): {cmData.TP} of {cmData.TP+cmData.FN} => {round(cmData.TP_Rate * 100, 2)}%\"\n",
    "        cmData.FN_Output = f\"False Negatives (FN): {cmData.FN} of {cmData.FN+cmData.TP} => {round(cmData.FN_Rate * 100, 2)}%\"\n",
    "        cmData.FP_Output = f\"False Positives (FP): {cmData.FP} of {cmData.FP+cmData.TN} => {round(cmData.FP_Rate * 100, 2)}%\"\n",
    "        \n",
    "        return cmData\n",
    "            \n",
    "print(\"Classes and Variables are ready.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "599478b3-5abd-4bc3-9888-d3dee71b1c09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AQbyWeather.aqParams: 6\n",
      "AQbyWeather.aqScenarios: 15 (Default Selected: bakersfield_pm25)\n"
     ]
    }
   ],
   "source": [
    "# CELL #4: Review the pre-defined AQParams and AQScenarios in this cell. You can edit these and/or use your own...\n",
    "# AQParams are added with default thresholds, which can be overridden on a per-AQScenario basis.\n",
    "# These AQParams are based on the OpenAQ /parameters API call where isCore=true (https://api.openaq.org/v2/parameters).\n",
    "# Default thresholds where provided using data from EPA.gov (https://www.epa.gov/criteria-air-pollutants/naaqs-table).\n",
    "# Confirm and adjust params or thresholds as needed for your needs... Not for scientific or health purposes.\n",
    "\n",
    "# Instantiate main App class with explicit mlTargetLabel and mlEvalMetric provided...\n",
    "AQbyWeather = AQbyWeatherApp(mlTargetLabel='isUnhealthy', mlEvalMetric='accuracy')\n",
    "\n",
    "# Define and add new AQParams...\n",
    "AQbyWeather.addAQParam(AQParam( 1, \"pm10\", \"µg/m³\", 150.0, \"Particulate Matter < 10 micrometers\"))\n",
    "AQbyWeather.addAQParam(AQParam( 2, \"pm25\", \"µg/m³\",  12.0, \"Particulate Matter < 2.5 micrometers\"))\n",
    "AQbyWeather.addAQParam(AQParam( 7, \"no2\",  \"ppm\",   100.0, \"Nitrogen Dioxide\"))\n",
    "AQbyWeather.addAQParam(AQParam( 8, \"co\",   \"ppm\",     9.0, \"Carbon Monoxide\"))\n",
    "AQbyWeather.addAQParam(AQParam( 9, \"so2\",  \"ppm\",    75.0, \"Sulfur Dioxide\"))\n",
    "AQbyWeather.addAQParam(AQParam(10, \"o3\",   \"ppm\",   0.070, \"Ground Level Ozone\"))\n",
    "\n",
    "# Define available AQ Scenarios for certain locations with their associated NOAA GSOD StationID values...\n",
    "# NOAA GSOD Station Search: https://www.ncei.noaa.gov/access/search/data-search/global-summary-of-the-day\n",
    "# TODO: Someday consider how to OPTIONALLY append more scenarios via an optional JSON file\n",
    "#       (ie: without adding a dependecy outside the .ipynb file)\n",
    "# NOTE: For Ozone Scenarios, we're generally using 0.035 ppm to override the default threshold.\n",
    "AQbyWeather.addAQScenario(AQScenario(\"bakersfield\", \"72384023155\", AQbyWeather.aqParams[\"pm25\"], None))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"bakersfield\", \"72384023155\", AQbyWeather.aqParams[\"pm10\"], None)) # Attempt at pm10 prediction.\n",
    "AQbyWeather.addAQScenario(AQScenario(\"bakersfield\", \"72384023155\", AQbyWeather.aqParams[\"o3\"],  0.035))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"fresno\",      \"72389093193\", AQbyWeather.aqParams[\"pm25\"], None))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"fresno\",      \"72389093193\", AQbyWeather.aqParams[\"o3\"],  0.035))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"visalia\",     \"72389693144\", AQbyWeather.aqParams[\"pm25\"], None))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"visalia\",     \"72389693144\", AQbyWeather.aqParams[\"o3\"],  0.035))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"san-jose\",    \"72494693232\", AQbyWeather.aqParams[\"pm25\"], None))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"san-jose\",    \"72494693232\", AQbyWeather.aqParams[\"o3\"],  0.035))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"los-angeles\", \"72287493134\", AQbyWeather.aqParams[\"pm25\"], None))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"los-angeles\", \"72287493134\", AQbyWeather.aqParams[\"o3\"],  0.035))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"phoenix\",     \"72278023183\", AQbyWeather.aqParams[\"pm25\"], None))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"phoenix\",     \"72278023183\", AQbyWeather.aqParams[\"o3\"],  0.035))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"fairbanks\",   \"70261026411\", AQbyWeather.aqParams[\"pm25\"], None))\n",
    "AQbyWeather.addAQScenario(AQScenario(\"lahore-pk\",   \"41640099999\", AQbyWeather.aqParams[\"pm25\"], None))\n",
    "\n",
    "print(f\"AQbyWeather.aqParams: {str(len(AQbyWeather.aqParams))}\")\n",
    "print(f\"AQbyWeather.aqScenarios: {str(len(AQbyWeather.aqScenarios))} (Default Selected: {AQbyWeather.selectedScenario.name})\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3395018f-87bd-4a7f-8ee5-7e90d38bfb04",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** CHOOSE YOUR OWN ADVENTURE HERE ***\n",
      "Please select a Scenario via the following drop-down-list...\n",
      "(NOTE: If you change Scenario, you must re-run remaining cells to see changes.)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "54d02042d8c74215baadef6fc83953d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(options=('bakersfield_pm25', 'bakersfield_pm10', 'bakersfield_o3', 'fresno_pm25', 'fresno_o3', 'visal…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# CELL #5: Select a Scenario via DROP DOWN LIST to use throughout the Notebook. This will drive the ML process...\n",
    "# A default \"value\" is set to avoid issues. Change this default to run the Notebook from start-to-finish for that Scenario.\n",
    "print(\"*** CHOOSE YOUR OWN ADVENTURE HERE ***\")\n",
    "print(\"Please select a Scenario via the following drop-down-list...\")\n",
    "print(\"(NOTE: If you change Scenario, you must re-run remaining cells to see changes.)\")\n",
    "ddl = widgets.Dropdown(options=AQbyWeather.aqScenarios.keys(), \n",
    "                       value=AQbyWeather.aqScenarios[\"bakersfield_pm25\"].name) # <-- DEFAULT / FULL-RUN VALUE\n",
    "ddl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ffed30d9-1725-405b-a87f-e2aac0be5634",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n",
      "{\n",
      "  \"aqParamTarget\": {\n",
      "    \"desc\": \"Particulate Matter < 2.5 micrometers\",\n",
      "    \"id\": 2,\n",
      "    \"name\": \"pm25\",\n",
      "    \"unhealthyThresholdDefault\": 12.0,\n",
      "    \"unit\": \"\\u00b5g/m\\u00b3\"\n",
      "  },\n",
      "  \"aqRadiusMeters\": 16100,\n",
      "  \"aqRadiusMiles\": 10,\n",
      "  \"location\": \"bakersfield\",\n",
      "  \"modelFolder\": \"AutogluonModels\",\n",
      "  \"name\": \"bakersfield_pm25\",\n",
      "  \"noaaStationID\": \"72384023155\",\n",
      "  \"noaaStationLat\": 0.0,\n",
      "  \"noaaStationLng\": 0.0,\n",
      "  \"openAqLocIDs\": [],\n",
      "  \"unhealthyThreshold\": 12.0,\n",
      "  \"yearEnd\": 2022,\n",
      "  \"yearStart\": 2016\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "if ddl.value:\n",
    "    AQbyWeather.selectedScenario = AQbyWeather.aqScenarios[ddl.value]\n",
    "    print(AQbyWeather.selectedScenario.getSummary())\n",
    "    print(AQbyWeather.selectedScenario.toJSON())\n",
    "else:\n",
    "    print(\"Please select a Scenario via the above drop-down-list.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9c3a647c-0fee-4b49-8576-0ee355629f0e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n",
      "Accessing and preparing data from ASDI-hosted NOAA GSOD dataset in Amazon S3 (bucket: noaa-gsod-pds)...\n",
      "NOAA Station Lat,Lng Updated for Scenario: bakersfield_pm25 => 35.4344,-119.0542\n",
      "noaagsod_df.shape = (2346, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DATE</th>\n",
       "      <th>NAME</th>\n",
       "      <th>LATITUDE</th>\n",
       "      <th>LONGITUDE</th>\n",
       "      <th>DEWP</th>\n",
       "      <th>WDSP</th>\n",
       "      <th>MAX</th>\n",
       "      <th>MIN</th>\n",
       "      <th>PRCP</th>\n",
       "      <th>MONTH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43440</td>\n",
       "      <td>-119.05420</td>\n",
       "      <td>29.2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-01-02</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43440</td>\n",
       "      <td>-119.05420</td>\n",
       "      <td>32.8</td>\n",
       "      <td>2.4</td>\n",
       "      <td>57.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-01-03</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43440</td>\n",
       "      <td>-119.05420</td>\n",
       "      <td>32.6</td>\n",
       "      <td>4.3</td>\n",
       "      <td>68.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-01-04</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43440</td>\n",
       "      <td>-119.05420</td>\n",
       "      <td>32.0</td>\n",
       "      <td>4.9</td>\n",
       "      <td>68.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-01-05</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43440</td>\n",
       "      <td>-119.05420</td>\n",
       "      <td>42.8</td>\n",
       "      <td>4.5</td>\n",
       "      <td>64.9</td>\n",
       "      <td>42.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2341</th>\n",
       "      <td>2022-06-04</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43424</td>\n",
       "      <td>-119.05524</td>\n",
       "      <td>50.5</td>\n",
       "      <td>7.3</td>\n",
       "      <td>91.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2342</th>\n",
       "      <td>2022-06-05</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43424</td>\n",
       "      <td>-119.05524</td>\n",
       "      <td>45.1</td>\n",
       "      <td>6.4</td>\n",
       "      <td>90.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2343</th>\n",
       "      <td>2022-06-06</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43424</td>\n",
       "      <td>-119.05524</td>\n",
       "      <td>58.7</td>\n",
       "      <td>7.5</td>\n",
       "      <td>90.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2344</th>\n",
       "      <td>2022-06-07</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43424</td>\n",
       "      <td>-119.05524</td>\n",
       "      <td>47.8</td>\n",
       "      <td>5.4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2345</th>\n",
       "      <td>2022-06-08</td>\n",
       "      <td>BAKERSFIELD AIRPORT, CA US</td>\n",
       "      <td>35.43424</td>\n",
       "      <td>-119.05524</td>\n",
       "      <td>43.5</td>\n",
       "      <td>4.3</td>\n",
       "      <td>91.9</td>\n",
       "      <td>75.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2346 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            DATE                        NAME  LATITUDE  LONGITUDE  DEWP  WDSP  \\\n",
       "0     2016-01-01  BAKERSFIELD AIRPORT, CA US  35.43440 -119.05420  29.2   3.0   \n",
       "1     2016-01-02  BAKERSFIELD AIRPORT, CA US  35.43440 -119.05420  32.8   2.4   \n",
       "2     2016-01-03  BAKERSFIELD AIRPORT, CA US  35.43440 -119.05420  32.6   4.3   \n",
       "3     2016-01-04  BAKERSFIELD AIRPORT, CA US  35.43440 -119.05420  32.0   4.9   \n",
       "4     2016-01-05  BAKERSFIELD AIRPORT, CA US  35.43440 -119.05420  42.8   4.5   \n",
       "...          ...                         ...       ...        ...   ...   ...   \n",
       "2341  2022-06-04  BAKERSFIELD AIRPORT, CA US  35.43424 -119.05524  50.5   7.3   \n",
       "2342  2022-06-05  BAKERSFIELD AIRPORT, CA US  35.43424 -119.05524  45.1   6.4   \n",
       "2343  2022-06-06  BAKERSFIELD AIRPORT, CA US  35.43424 -119.05524  58.7   7.5   \n",
       "2344  2022-06-07  BAKERSFIELD AIRPORT, CA US  35.43424 -119.05524  47.8   5.4   \n",
       "2345  2022-06-08  BAKERSFIELD AIRPORT, CA US  35.43424 -119.05524  43.5   4.3   \n",
       "\n",
       "       MAX   MIN  PRCP  MONTH  \n",
       "0     55.0  27.0   0.0      1  \n",
       "1     57.0  27.0   0.0      1  \n",
       "2     68.0  34.0   0.0      1  \n",
       "3     68.0  39.0   0.0      1  \n",
       "4     64.9  42.1   0.0      1  \n",
       "...    ...   ...   ...    ...  \n",
       "2341  91.0  64.0   0.0      6  \n",
       "2342  90.0  63.0   0.0      6  \n",
       "2343  90.0  63.0   0.0      6  \n",
       "2344  91.0  66.0   0.0      6  \n",
       "2345  91.9  75.9   0.0      6  \n",
       "\n",
       "[2346 rows x 10 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# GET NOAA GSOD WEATHER DATA...\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "noaagsod_df = AQbyWeather.getNoaaDataFrame()\n",
    "\n",
    "if(len(noaagsod_df) >= 1):\n",
    "    # Update NOAA Station Lat/Lng...\n",
    "    AQbyWeather.selectedScenario.updateNoaaStationLatLng(noaagsod_df.iloc[0])\n",
    "    \n",
    "    # Save DataFrame to CSV...\n",
    "    noaagsod_df.to_csv(AQbyWeather.getFilenameNOAA(), index=False)\n",
    "\n",
    "    # Output DataFrame properties...\n",
    "    print('noaagsod_df.shape =', noaagsod_df.shape)\n",
    "    display(noaagsod_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2893d81f-a488-464d-b374-dafbfc0ae3fb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n",
      "Accessing ASDI-hosted OpenAQ Locations (HTTPS API)...\n",
      "OpenAQ Location IDs within 10 miles (16100m) of NOAA Station 72384023155 at 35.4344,-119.0542: [6885, 28777, 6883, 230814, 887]\n",
      "Accessing ASDI-hosted OpenAQ Averages (HTTPS API)...\n",
      "aq_df.shape = (2195, 5)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>parameter</th>\n",
       "      <th>unit</th>\n",
       "      <th>average</th>\n",
       "      <th>isUnhealthy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-03-06</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>2.0000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-03-10</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>10.5000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-03-11</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>5.4286</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-03-12</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>4.1667</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-03-13</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>6.0000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2190</th>\n",
       "      <td>2022-06-10</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>9.2045</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2191</th>\n",
       "      <td>2022-06-11</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>9.4583</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2192</th>\n",
       "      <td>2022-06-12</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>6.1667</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2193</th>\n",
       "      <td>2022-06-13</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>3.5435</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2194</th>\n",
       "      <td>2022-06-14</td>\n",
       "      <td>pm25</td>\n",
       "      <td>µg/m³</td>\n",
       "      <td>7.3000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2195 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             day parameter   unit  average  isUnhealthy\n",
       "0     2016-03-06      pm25  µg/m³   2.0000            0\n",
       "1     2016-03-10      pm25  µg/m³  10.5000            0\n",
       "2     2016-03-11      pm25  µg/m³   5.4286            0\n",
       "3     2016-03-12      pm25  µg/m³   4.1667            0\n",
       "4     2016-03-13      pm25  µg/m³   6.0000            0\n",
       "...          ...       ...    ...      ...          ...\n",
       "2190  2022-06-10      pm25  µg/m³   9.2045            0\n",
       "2191  2022-06-11      pm25  µg/m³   9.4583            0\n",
       "2192  2022-06-12      pm25  µg/m³   6.1667            0\n",
       "2193  2022-06-13      pm25  µg/m³   3.5435            0\n",
       "2194  2022-06-14      pm25  µg/m³   7.3000            0\n",
       "\n",
       "[2195 rows x 5 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# GET OPENAQ AIR QUALITY DAILY AVERAGES DATA...\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "aq_df = AQbyWeather.getOpenAqDataFrame() # Gets nearby Location IDs THEN gets associated daily averages.\n",
    "    \n",
    "if len(aq_df) > 0:\n",
    "    # Output DataFrame properties...\n",
    "    print('aq_df.shape =', aq_df.shape)\n",
    "    display(aq_df)\n",
    "    aq_df.to_csv(AQbyWeather.getFilenameOpenAQ(), index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a676054-ca63-47a2-832c-c136d1f0ecb4",
   "metadata": {},
   "source": [
    "## Merging the NOAA and OpenAQ data into one dataframe...\n",
    "We'll be using Autogluon's [_TabularPredictor_](https://auto.gluon.ai/stable/tutorials/tabular_prediction/index.html) on this merged dataframe to predict our target label column. For best results, we need to ensure we have enough data and that we have enough of each target label classification {0,1}.\n",
    "\n",
    "Review the shape of the merged_df table and the target label bar chart to determine if you have enough data and enough variation in the target label. Ideally, we should have thousands of total rows and at least hundreds of each class value.\n",
    "\n",
    "In the following cell, we can also visualize correlations between different columns in our merged_df table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6a873736-65b4-41a5-82e0-794ef33188b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n",
      "merged_df.shape = (2184, 7)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DEWP</th>\n",
       "      <th>WDSP</th>\n",
       "      <th>MAX</th>\n",
       "      <th>MIN</th>\n",
       "      <th>PRCP</th>\n",
       "      <th>MONTH</th>\n",
       "      <th>isUnhealthy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>47.3</td>\n",
       "      <td>6.8</td>\n",
       "      <td>72.0</td>\n",
       "      <td>51.1</td>\n",
       "      <td>0.02</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>44.8</td>\n",
       "      <td>3.7</td>\n",
       "      <td>82.9</td>\n",
       "      <td>44.1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>42.8</td>\n",
       "      <td>6.9</td>\n",
       "      <td>82.9</td>\n",
       "      <td>48.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>43.2</td>\n",
       "      <td>2.5</td>\n",
       "      <td>73.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>0.23</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>43.5</td>\n",
       "      <td>3.8</td>\n",
       "      <td>66.9</td>\n",
       "      <td>45.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2179</th>\n",
       "      <td>50.5</td>\n",
       "      <td>7.3</td>\n",
       "      <td>91.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2180</th>\n",
       "      <td>45.1</td>\n",
       "      <td>6.4</td>\n",
       "      <td>90.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2181</th>\n",
       "      <td>58.7</td>\n",
       "      <td>7.5</td>\n",
       "      <td>90.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2182</th>\n",
       "      <td>47.8</td>\n",
       "      <td>5.4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2183</th>\n",
       "      <td>43.5</td>\n",
       "      <td>4.3</td>\n",
       "      <td>91.9</td>\n",
       "      <td>75.9</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2184 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      DEWP  WDSP   MAX   MIN  PRCP  MONTH  isUnhealthy\n",
       "0     47.3   6.8  72.0  51.1  0.02      3            0\n",
       "1     44.8   3.7  82.9  44.1  0.00      3            0\n",
       "2     42.8   6.9  82.9  48.0  0.00      3            0\n",
       "3     43.2   2.5  73.0  45.0  0.23      3            0\n",
       "4     43.5   3.8  66.9  45.0  0.00      3            0\n",
       "...    ...   ...   ...   ...   ...    ...          ...\n",
       "2179  50.5   7.3  91.0  64.0  0.00      6            0\n",
       "2180  45.1   6.4  90.0  63.0  0.00      6            0\n",
       "2181  58.7   7.5  90.0  63.0  0.00      6            0\n",
       "2182  47.8   5.4  91.0  66.0  0.00      6            0\n",
       "2183  43.5   4.3  91.9  75.9  0.00      6            0\n",
       "\n",
       "[2184 rows x 7 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEDCAYAAADZUdTgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQVElEQVR4nO3df4xlZX3H8fdHVvAHjcuPCYFddGlca1Cj4hSxtIaIVVDjEoMUS8uKpJs2YLXUyNKmob9MIG1KpRqaLaxASkCKVDaKUoooaQ3IoID8LBMEdzf8GORHtaiIfvvHPBuv4w6zO3e4A/O8X8nknvN9nnOe5yaTz5x97jl3U1VIkvrwgsWegCRpdAx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOLFvsCTyTvffeu1atWrXY05Ck55Wbbrrpkaoa217bczr0V61axcTExGJPQ5KeV5LcP1ubyzuS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHVkztBPsjHJw0luG6j9XZK7ktya5N+TLB9oOy3JZJK7k7xzoH5Eq00mWb/g70SSNKcdeTjrfOBTwIUDtauB06rq6SRnAqcBpyY5EDgWeA2wH/CfSV7Vjvk08NvAFuDGJJuq6o6FeRuLa9X6Ly72FJaU+85492JPQVqy5rzSr6rrgEdn1P6jqp5uu9cDK9v2GuCSqvpxVX0HmAQObj+TVXVvVT0FXNL6SpJGaCHW9D8EfKltrwA2D7RtabXZ6pKkERoq9JP8OfA0cNHCTAeSrEsykWRiampqoU4rSWKI0E/yQeA9wHH18/9dfSuw/0C3la02W/2XVNWGqhqvqvGxse1+SZwkaZ7mFfpJjgA+Dry3qp4caNoEHJtktyQHAKuBbwA3AquTHJBkV6Y/7N003NQlSTtrzrt3klwMHAbsnWQLcDrTd+vsBlydBOD6qvrDqro9yaXAHUwv+5xUVT9t5zkZuArYBdhYVbc/C+9HkvQM5gz9qvrAdsrnPUP/TwCf2E79SuDKnZqdJGlB+USuJHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHVkztBPsjHJw0luG6jtmeTqJPe01z1aPUnOTjKZ5NYkBw0cs7b1vyfJ2mfn7UiSnsmOXOmfDxwxo7YeuKaqVgPXtH2AI4HV7WcdcA5M/5EATgfeDBwMnL7tD4UkaXTmDP2qug54dEZ5DXBB274AOGqgfmFNux5YnmRf4J3A1VX1aFU9BlzNL/8hkSQ9y+a7pr9PVT3Qth8E9mnbK4DNA/22tNpsdUnSCA39QW5VFVALMBcAkqxLMpFkYmpqaqFOK0li/qH/UFu2ob0+3Opbgf0H+q1stdnqv6SqNlTVeFWNj42NzXN6kqTtmW/obwK23YGzFrhioH58u4vnEOCJtgx0FfCOJHu0D3Df0WqSpBFaNleHJBcDhwF7J9nC9F04ZwCXJjkRuB84pnW/EngXMAk8CZwAUFWPJvkb4MbW76+rauaHw5KkZ9mcoV9VH5il6fDt9C3gpFnOsxHYuFOzkyQtqDlDX9Lz26r1X1zsKSwZ953x7sWewtD8GgZJ6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHhgr9JH+S5PYktyW5OMmLkhyQ5IYkk0k+m2TX1ne3tj/Z2lctyDuQJO2weYd+khXAHwPjVfVaYBfgWOBM4KyqeiXwGHBiO+RE4LFWP6v1kySN0LDLO8uAFydZBrwEeAB4G3BZa78AOKptr2n7tPbDk2TI8SVJO2HeoV9VW4G/B77LdNg/AdwEPF5VT7duW4AVbXsFsLkd+3Trv9d8x5ck7bxhlnf2YPrq/QBgP+ClwBHDTijJuiQTSSampqaGPZ0kacAwyztvB75TVVNV9RPgcuBQYHlb7gFYCWxt21uB/QFa+8uA7808aVVtqKrxqhofGxsbYnqSpJmGCf3vAockeUlbmz8cuAO4Fji69VkLXNG2N7V9WvtXqqqGGF+StJOGWdO/gekPZL8JfLudawNwKnBKkkmm1+zPa4ecB+zV6qcA64eYtyRpHpbN3WV2VXU6cPqM8r3Awdvp+yPg/cOMJ0kajk/kSlJHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHhgr9JMuTXJbkriR3JnlLkj2TXJ3knva6R+ubJGcnmUxya5KDFuYtSJJ21LBX+p8EvlxVrwZeD9wJrAeuqarVwDVtH+BIYHX7WQecM+TYkqSdNO/QT/Iy4K3AeQBV9VRVPQ6sAS5o3S4Ajmrba4ALa9r1wPIk+853fEnSzhvmSv8AYAr4TJJvJTk3yUuBfarqgdbnQWCftr0C2Dxw/JZWkySNyDChvww4CDinqt4I/B8/X8oBoKoKqJ05aZJ1SSaSTExNTQ0xPUnSTMOE/hZgS1Xd0PYvY/qPwEPblm3a68OtfSuw/8DxK1vtF1TVhqoar6rxsbGxIaYnSZpp3qFfVQ8Cm5P8WisdDtwBbALWttpa4Iq2vQk4vt3FcwjwxMAykCRpBJYNefyHgYuS7ArcC5zA9B+SS5OcCNwPHNP6Xgm8C5gEnmx9JUkjNFToV9XNwPh2mg7fTt8CThpmPEnScHwiV5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6MnToJ9klybeSfKHtH5DkhiSTST6bZNdW363tT7b2VcOOLUnaOQtxpf8R4M6B/TOBs6rqlcBjwImtfiLwWKuf1fpJkkZoqNBPshJ4N3Bu2w/wNuCy1uUC4Ki2vabt09oPb/0lSSMy7JX+PwIfB37W9vcCHq+qp9v+FmBF214BbAZo7U+0/pKkEZl36Cd5D/BwVd20gPMhybokE0kmpqamFvLUktS9Ya70DwXem+Q+4BKml3U+CSxPsqz1WQlsbdtbgf0BWvvLgO/NPGlVbaiq8aoaHxsbG2J6kqSZ5h36VXVaVa2sqlXAscBXquo44Frg6NZtLXBF297U9mntX6mqmu/4kqSd92zcp38qcEqSSabX7M9r9fOAvVr9FGD9szC2JOkZLJu7y9yq6qvAV9v2vcDB2+nzI+D9CzGeJGl+fCJXkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjoy79BPsn+Sa5PckeT2JB9p9T2TXJ3knva6R6snydlJJpPcmuSghXoTkqQdM8yV/tPAn1bVgcAhwElJDgTWA9dU1WrgmrYPcCSwuv2sA84ZYmxJ0jzMO/Sr6oGq+mbb/j5wJ7ACWANc0LpdABzVttcAF9a064HlSfad7/iSpJ23IGv6SVYBbwRuAPapqgda04PAPm17BbB54LAtrSZJGpGhQz/J7sDngI9W1f8OtlVVAbWT51uXZCLJxNTU1LDTkyQNGCr0k7yQ6cC/qKoub+WHti3btNeHW30rsP/A4Stb7RdU1YaqGq+q8bGxsWGmJ0maYZi7dwKcB9xZVf8w0LQJWNu21wJXDNSPb3fxHAI8MbAMJEkagWVDHHso8PvAt5Pc3Gp/BpwBXJrkROB+4JjWdiXwLmASeBI4YYixJUnzMO/Qr6r/AjJL8+Hb6V/ASfMdT5I0PJ/IlaSOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1JGRh36SI5LcnWQyyfpRjy9JPRtp6CfZBfg0cCRwIPCBJAeOcg6S1LNRX+kfDExW1b1V9RRwCbBmxHOQpG4tG/F4K4DNA/tbgDcPdkiyDljXdn+Q5O4Rza0HewOPLPYk5pIzF3sGWiTP+d/P59Hv5itmaxh16M+pqjYAGxZ7HktRkomqGl/seUjb4+/naIx6eWcrsP/A/spWkySNwKhD/0ZgdZIDkuwKHAtsGvEcJKlbI13eqaqnk5wMXAXsAmysqttHOYfOuWym5zJ/P0cgVbXYc5AkjYhP5EpSRwx9SeqIoS9JHXnO3aevhZPk1Uw/8byilbYCm6rqzsWblaTF5JX+EpXkVKa/5iLAN9pPgIv9ojs9lyU5YbHnsJR5984SleR/gNdU1U9m1HcFbq+q1YszM+mZJfluVb18seexVLm8s3T9DNgPuH9Gfd/WJi2aJLfO1gTsM8q59MbQX7o+ClyT5B5+/iV3LwdeCZy8WJOSmn2AdwKPzagH+Prop9MPQ3+JqqovJ3kV019nPfhB7o1V9dPFm5kEwBeA3avq5pkNSb468tl0xDV9SeqId+9IUkcMfUnqiKGvJSHJM374l+QHM/Y/mORTcxxzWJIvLND8/jLJxwbG3m+g7b4key/EONJcDH0tCVX1G4s9h53wQaZvp5VGztDXkrDtSj7JvkmuS3JzktuS/NYOHHt+krOTfD3JvUmOHmjePcllSe5KclGStGPelORrSW5KclWSfVv9D5LcmOSWJJ9L8pIZYx0NjAMXtTm+uDV9OMk3k3w7yauTvCDJPUnG2nEvSDK5bV+aL0NfS83vAldV1RuA1wM37+Bx+wK/CbwHOGOg/kamn3k4EPhV4NAkLwT+CTi6qt4EbAQ+0fpfXlW/XlWvB+4EThwcpKouAyaA46rqDVX1w9b0SFUdBJwDfKyqfgb8K3Bca387cEtVTe3g+5G2y/v0tdTcCGxswfz57d0HPmDwfuXPt6C9I8ngE6HfqKotAEluBlYBjwOvBa5uF/67AA+0/q9N8rfAcmB3pv+XuB1xeXu9CXhf294IXAH8I/Ah4DM7eC5pVl7pa0mpquuAtzL9INr5SY5vTT9s3zu0zZ7AIwP7Px7Yziz1nzJ9oRSmv7/oDe3ndVX1jtbnfODkqnod8FfAi3Zw6tvG2TYGVbUZeCjJ25h+yO5LO3guaVaGvpaUJK8AHqqqfwHOBQ5qTV8Dfq/1eTFwDHDtPIe5GxhL8pZ2vhcmeU1r+xXggfYvjeNmOf77rd+OOJfpZZ5/80lqLQRDX0vNYcAtSb4F/A7wyVb/CPC+tkRzPdMhet18Bqiqp4CjgTOT3ML05wbb7h76C+AG4L+Bu2Y5xfnAP8/4IHc2m5heJnJpRwvCr2GQnsOSjANnVdWcdyFJO8IPcqXnqPaf3fwRsy8TSTvNK31J6ohr+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakj/w9ogwL5p/SQzwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Merge the NOAA GSOD weather data with our OpenAQ data by DATE...\n",
    "# Perform another column drop to remove columns we don't want as features/inputs.\n",
    "# This column removal will NOT be necessary once we can use Autogluon ignore_columns param (TBD).\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "merged_df = AQbyWeather.getMergedDataFrame(noaagsod_df, aq_df)\n",
    "\n",
    "if(len(merged_df > 0)):\n",
    "    # Output DataFrame properties...\n",
    "    print('merged_df.shape =', merged_df.shape)\n",
    "    display(merged_df)\n",
    "    merged_df.groupby([AQbyWeather.mlTargetLabel]).size().plot(kind=\"bar\")\n",
    "    merged_df.to_csv(AQbyWeather.getFilenameOther(\"dataMERGED\"), index=False)\n",
    "\n",
    "# TODO: Explore other ways to join the data and/or handle null values..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c5dee84f-da68-4b10-a075-a5cab419cd16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAHYCAYAAAB6CS2aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwKUlEQVR4nO3debwnVX3n/9ebRhaDhiaNBAFBlhkXkFZ6mJ9kIwoJTozwcxiFmAgTHQajJi44aphRY+JIEqKGyE/SgwhukIgorcFHDy7EBRcabWgWCavSHaJii4qgSPfn90fVhS+Xe7u/dH9Pf+/t7+vZj3rcqnNOVZ2666c/51RVqgpJkiS1sc24OyBJkrQ1M9iSJElqyGBLkiSpIYMtSZKkhgy2JEmSGjLYkiRJashgawtKsi7JyiTXJrkqyWuTbNPXHZ7kh3391HJEkncmedXAMZYnOXtg+2+SvCbJPknu7fe7LslZU8cep0fQ/3uTfCPJ9Um+luTEgTa7Jflk/zm7LsklffmcvObZJKkkHxzY3jbJ95J8clq7jyf5yrSyM5K8aWD71CRntu/16Gzs+pOcmOTd/fpbktyT5HED7e/e8r3eNAM/69ck+UiSR89Q/okkOw/sc0qSb/b1VyR5cV9+WZIb+u//LyX592O6rAcM872c5JgkV/c/06uSHDNQd26SNUm277cXJbktyUEDv//WJrm1X/90//N+zbR+vCXJKZt5LZdvpP7uadsPfJ9uYJ/Dp/9cb0b/HrjG/tyPH6i7LcmiUZxHbc3ZP0xbqXuranFVPRU4EngO8OaB+i/09VPLp4EvAYcB9IHEIuCpA/scBkz9sri5qhYDTwOeAhzT8mKGNGz/b66qp1fVk4HjgFcl+a99m7cCl1bVwVX1FOANA/vPxWuezU+AA5Ps2G8fCawZbND/8T0E+MUk+w5U/U/gxCT79uUvBU5t3+WR2uj1T3Mn8NrmvWpj6mf9QOA+4OQZytcCLwdIcjLd5+PQ/vv52UAGjveiqjoYOA/46y10DRuywa9lkoOB04Gj+5/p5wGnJ3nawDHWAX84eNCqWjX1+w9YBryu3z6i1YVU1WGtjt3AicDjN9ZIc4/B1phU1XeBk4BXJMkGml4OPLNffypwDfDjJAv7/xU+Gfj6tGPf3++3/8g7/sgN0/+1gztU1S3Aa4A/7ot2B1YP1F89/SRz7Jo35BLgd/r144Hzp9U/H/gEcAFd0AlAVf2ILrh6d7+8qaruat3ZBjZ2/YPOAV6YZJfmvWrrC8z8ffllYI9+/U+Bl/VfZ6rqR1V13gz7fH6WY43Dhr6WpwD/u6puBeg/vh143UCbdwGvTrJt+67ObipzlWT3JJ8fyDz+2hD7nttnnS9PckuSYweqd0pyYZ+t/NDU7/kkhyT55yRXpsv0796X/7c+o3lVko9OZUMHznUssAT4UN/HqUD3lUm+3mcPn5RkmyQ3Jtm132+bJDdNbWs8DLbGqA8qFgBTQyW/locOI+5XVf8K3J/kCXRZoC8DX6ULYJYAq6rqvsHj9j+kzwZWbalrmc0w/af7n/90Xwee1K+fCbw3yefSDZ897H92c+maN+IC4LgkO9Bl4746rX7qj9b5/foDqup8YCHw2Kr6wBboawsbu/5Bd9MFXH+yJTrWQh9IPIdp35dJFtB9vy5L8ljgMf3vg4353enHGqMNfS2fClw5rf0KHprV/jbwReAPHsE59xv8HcmDGcNR+D1geZ9VOxhYOeR+uwO/CjwXOG2g/OnAq+gy7vsCv5LkUcDfAcdW1SF0399v69tfVFX/oc9gXg+8ZPAkVXUh3efwRX22796+6s6qegbwHuCUqloPfBB4UV9/BHBVVX1vyOtRA2P9H4Ue5gtV9dwZyi+nC1QOA95B97/hw4Af0g3TTdmv/wVUwMVV9am23R3asP0f9EC2r6qW90NnR9H94fpGkgP76rl6zTOqqquT7EMXSF0yWJdkN+AA4ItVVUl+nuTAqrqmr9+T7hf7+iQ7VdW8mcM0ZUPXP4szgJVJTm/asdHbsf++hC6z9d5p5XvQ/UG9FPiFIY73oST3ArcBrxxpTzfRJnwtZ/J24GLgn4ZsPzVtAOjmM23ieWdyBXBOHxB9vKpWbqDt4HvuPt4HONf1P8NTvlZVq/t+rgT2Ae4CDgQu7RNdC4A7+vYHJvkLYGdgJ2D5kP2+qP94JV1mHLog7mK67OEfAu8b8lhqxGBrjPoAYh3wXbrhtNlMzXs6iG4Y7na6uSw/4qE/RA/5RTSHDNv/QU+n+2MEQFWtBT4MfLifePrrdL9c5uo1b8gyuvkshwO/NFD+ArrM1a39L+LH0v0hm5qb9bd0c/ye3H8cHJKZT2a7/oepqruSfJh+btM8cu8s35f3VtXiPhO7HHh5VZ2R5O4k+24gu/WiqlrRrLebbrav5XV0cw+vGig7BLh2cOequrEPRF7QtJdDqKrPJ/l1uqHRc5O8o6reD9ybZLuBEYRd6OYTTvnZwHpmKV9H9/c2wLVV9Uwe7lzgmKq6Kt0NQocP2fWp80ydg6q6Pcl3kjwLOJQHs1waE4cRx6QfPz8LeHdt/G3gl9OlqNdW1bo+8NiZbihug3fSzBGPqP/9/5ZPp0u3k+RZefBurscA+9ENQcxX5wB/VlXTh4OOB46qqn2qah+6P07HASR5Dt1w8/uBPween+QpW67LIzXb9c/mHcB/Zyv6z2FV3UM3J/G1/VDj24Ez+yFFkuyU/m7EOW62r+XpwBv7n+Wpn+k/Bf5mhmO8jW6O11gl2Rv4TlX9H+Bs4Bl91T8Dv9+32ZEuMPzcJp7mBmDXJM/sj/eoJFNDq48B7ugza7MFRz/u2w3jbLrhxI9U1bpN7K9GxGBry9qxn2twLfBp4P8CfzZQP33O1tRky1V0d/ENPg5gFfDDqhr8H9ZcNUz/90v/6AfgH4Ezqmoq63UIsCLJ1XRzvs6uqiu2UN9HrqpWV9UZg2X9H6O9Gfgc9ZOKf5jkN+iGA/6oOj+hy2pt8PbzuWqm699I+zuBjwHbt+vVlldV3wCupguy30P3B/yKdI83+AKwfozdG8psX8t+CO71wCeSfJPupo//MdPQXFVdy7SbfMbkcOCqJN8AXkiXSYZuzuDz+wzcV+iCl89vygn67NixwF8muYpuXtjU3ZD/i27e25eAb85yiHOBs6ZNkJ/NMrrhSIcQ54BsPKkiSZLmkyRLgHdW1UbvqlR7W01aXpIkQZI3AC/DuVpzhpktSZKkhpyzJUmS1JDBliRJUkMGW5IkaauR5Jwk3820F5cP1Cfda5ZuSvey9GcM1J3Qv+7oxiQnjKpPBlvzQJKTxt2HcfC6J4vXPVkm9bphsq99CzmX7o0js3kO3ds6DqB7R/F7ANK9h/XNwH+kexjsm5MsHEWHDLbmh0n9wfS6J4vXPVkm9bphsq+9uf45aGs30ORo4P39cwu/Auyc7oXgvw1cWlVrq+oHdK/T2lDQNjSDLUmSNEn2oHtt3JTVfdls5ZvN52wNadEuC2qfvR41lnM/YY9tWXLwDmN5RseqtbuO47QALFi4kO332mss133QLt8bx2kBv95j68CYTOp1b7vzmK87G2/SyoKFO7P9E8Zz7ffdvvrOqtoiP+i//Zu/UN9fO/o3BV159c+uBX46ULS0qpaO/EQjZLA1pH32ehRfW77XuLuxxe3/4ZPH3YWx+NrvnTXuLozFfhdM5td7km1z/xijjjFat8OcfxtSE9965eu+taXO9f216/ja8ieM/LgLdr/xp1W1ZDMOsQYY/IO+Z1+2hoe+AHxP4LLNOM8DHEaUJEkjV8D6Bv9GYBnw4v6uxP+H7j29dwDLgd9KsrCfGP9bfdlmM7MlSZIaKNbVls8gJjmfLkO1KMlqujsMHwVQVWcBlwD/CbgJuAf4r33d2iR/DlzRH+qtVbWhifZDM9iSJElbjao6fiP1Bbx8lrpzgHNG3SeDLUmSNHLdMOLE3fsxI+dsSZIkNWRmS5IkNTGiCe3znsGWJEkauaJYVw4jgsOIkiRJTZnZkiRJTThBvmNmS5IkqSEzW5IkaeQKWGdmCzCzJUmS1JSZLUmS1IRztjoGW5IkaeQKfPRDz2FESZKkhsxsSZKkJnx+fMfMliRJUkNmtiRJ0sgV5aMfegZbkiRp9ArWGWsBDiNKkiQ1ZWZLkiSNXOEE+SlmtiRJkhoysyVJkhoI68i4OzEnGGxJkqSRK2C9E+QBhxElSZKamrOZrSTrgFXAo4D7gfcD76yq9UkOBy4Gbh3Y5RTgd4BvVdW7+mMsB26vqpf2238DrAEuAq4HbgC2Az4P/FFVOZdPkqQRcRixM2eDLeDeqloMkORxwIeBxwJv7uu/UFXPHdwhyc7AC4B3JdkGWNTvM+Uw4NX9+s1VtTjJtsBngWPogjBJkqSRmRfDiFX1XeAk4BVJNhQmXw48s19/KnAN8OMkC5NsDzwZ+Pq0Y9/f77f/yDsuSdKEKrrM1qiX+WguZ7YeoqpuSbIAeFxf9GtJVg40+c9VdXOS+5M8gS6L9WVgD7oA7IfAqqq6bzBeS/Jo4NnAm7bAZUiSNDHW1/wMjkZt3gRbM3jYMGLvcrpA6zDgHXTB1mF0wdaXBtrt1wdrBVxcVZ+afqAkJ9Fl1HjCHvP5UyVJksZl3kQQSfYF1gHfpRsOnM2X6IKrg+iGEW8HXgv8CHjfQLubp+aEzaaqlgJLAZYcvIM3sEqSNKSpYUTNkzlbSXYFzgLeXVUbC3ouB54LrK2qdVW1FtiZbijx8qYdlSRJmmYuZ7Z27If5ph798AG6YcEp0+ds/UVVXUj3uIhFdHcvTlkF7FRVdzbtsSRJAqAI6+ZHTqe5ORtsVdWCDdRdBvziLHXreOjjHqiqE6dt3wYcuLl9lCRJ2pg5G2xJkqT5zbsROwZbkiRp5Jwg/yAHUyVJkhoysyVJkhoI68qcDpjZkiRJasrMliRJGrkC1pvTAQy2JElSI06Q7xhySpIkNWRmS5IkjVyVE+Sn+FmQJElqyMyWJElqYr1ztgCDLUmS1ED3BHkH0MBhREmSpKbMbEmSpAacID/Fz4IkSVJDZrYkSdLIjfMJ8kmOAv4WWACcXVWnTat/J/Cb/eajgcdV1c593TpgVV/37ap63ub2x2BLkiRtNZIsAM4EjgRWA1ckWVZV1021qapXD7R/JfD0gUPcW1WLR9kngy1JktTEuhrLox8OBW6qqlsAklwAHA1cN0v744E3t+yQwZYkSRq5Iq0e/bAoyYqB7aVVtXRgew/g9oHt1cB/nOlASfYGngh8dqB4h/749wOnVdXHN7fDBluSJGk+ubOqlozoWMcBF1bVuoGyvatqTZJ9gc8mWVVVN2/OSQy2JElSE+vH8+iHNcBeA9t79mUzOQ54+WBBVa3pP96S5DK6+VybFWz56AdJkrQ1uQI4IMkTk2xHF1Atm94oyZOAhcCXB8oWJtm+X18E/Aqzz/UampktSZI0cuN6XU9V3Z/kFcByukc/nFNV1yZ5K7CiqqYCr+OAC6qqBnZ/MvD3SdbTJaROG7yLcVMZbEmSpJErMq67EamqS4BLppW9adr2W2bY73LgoFH3x2FESZKkhsxsSZKkJsb1BPm5xmBrSKvW7sr+Hz553N3Y4m76vbPG3YWx2O+CyftaA9R2tfFGW6nadjKvfbs7JvPPwK5Xjmd4a9y+Ne4OTKjJ/CmTJElNVcG68Tz6Yc4x2JIkSQ2E9UxmBnE6Q05JkqSGzGxJkqSRKxxGnOJnQZIkqSEzW5IkqYlxPEF+LvKzIEmS1JCZLUmSNHJFWD+m1/XMNQZbkiSpCYcRO34WJEmSGjKzJUmSRq6A9T76ATCzJUmS1JSZLUmS1EBY5+t6AIMtSZLUgMOID/KzIEmS1JCZLUmS1ITDiB0zW5IkSQ2Z2ZIkSSNXFeds9Qy2JElSE+sMtgCHESVJkpoysyVJkkaugPVOkAfMbEmSJDVlZkuSJDUQ52z1/CxIkiQ1ZGZLkiSNXPe6HudsgcGWJElqZJ0DaMAcHUZM8s4krxrYXp7k7IHtv0nymiT3JvlGkuuTfC3JiQNtdkvyySRXJbkuySV9+T79fiv78rOSzMnPgyRJmv/mambrS8ALgHf1gdAi4LED9YcBrwZurqqnAyTZF7goSarqfcBbgUur6m/7+qcN7H9zVS1Osi3wWeAY4KLG1yRJ0sQo4jBib65mdC4HntmvPxW4BvhxkoVJtgeeDKwd3KGqbgFeA/xxX7Q7sHqg/urpJ6mq+/tz7T/qC5AkSYI5GmxV1b8C9yd5Al0W68vAV+kCsCXAKuC+GXb9OvCkfv1M4L1JPpfk1CSPn944yaOBZ/fHkyRJI7SebUa+zEdzdRgRuozTYf3yDmCPfv2HdMOMM3kgX1lVy/uhxaOA5wDfSHJgX71fkpV0N0tcXFWfmvFgyUnASQALFi7c3OuRJGliVME6hxGBuR1sfYkuuDqIbhjxduC1wI+A982yz9OB66c2qmot8GHgw0k+Cfw6cCX9nK2NdaCqlgJLAbbfa6/a1AuRJEmTay7n4y4Hngusrap1feC0M91Q4uXTGyfZBzgd+Lt++1n9MCFJHgPsB3x7i/RckiSxvjLyZT6ay5mtVXR3IX54WtlOVXVnkp3ohgO/AewA/Bg4o6rO7dseArw7yf10QeXZVXVFH5RJkiRtEXM22KqqdTz0cQ9U1YkD67cBO25g/78G/nqG8tuAAx+2gyRJGpnu0Q9zeQBty5mzwZYkSZrf1jE/h/1GzZBTkiSpITNbkiRp5HwR9YPMbEmSJDVkZkuSJDXgBPkpfhYkSZIaMrMlSZKaWO/diICZLUmS1MDUuxFHvQwjyVFJbkhyU5I3zFB/YpLvJVnZLy8dqDshyY39csIoPhdmtiRJ0lYjyQLgTOBIYDVwRZJlVXXdtKb/UFWvmLbvLsCbgSV0N1Re2e/7g83pk5ktSZLUxPraZuTLEA4FbqqqW6rqPuAC4Oghu/zbwKVVtbYPsC4Fjtqkix9gsCVJkrYmewC3D2yv7sum+89Jrk5yYZK9HuG+j4jBliRJGrnu3YijX4BFSVYMLCdtQvc+AexTVU+jy16dN8prn845W5IkqYlGdyPeWVVLNlC/BthrYHvPvuwBVfX9gc2zgb8a2PfwaftetqkdnWJmS5IkbU2uAA5I8sQk2wHHAcsGGyTZfWDzecD1/fpy4LeSLEyyEPitvmyzmNmSJEkjN653I1bV/UleQRckLQDOqaprk7wVWFFVy4A/TvI84H5gLXBiv+/aJH9OF7ABvLWq1m5unwy2JEnSVqWqLgEumVb2poH1NwJvnGXfc4BzRtkfgy1JktSE70bsGGxJkqTRe/DuwYlnyClJktSQmS1JkjRyhS+inmJmS5IkqSEzW5IkqQnnbHXMbEmSJDVkZkuSJI3cuB5qOhcZbEmSpCYMtjoOI0qSJDVkZmtIB+3yPb72e2eNuxtb3H4XnDzuLozFzcdN3tcaJvfrDbDNTyfzf+A/f8z6cXdhLP7tsHH3YEwu2HKnKnyo6RQzW5IkSQ2Z2ZIkSU34UNOOwZYkSRq9coL8FIcRJUmSGjKzJUmSRs7nbD3IzJYkSVJDZrYkSVITZrY6BluSJGnkfM7WgxxGlCRJasjMliRJaqLMbAFmtiRJkpoysyVJkprwCfIdM1uSJEkNmdmSJEkjV76u5wEGW5IkqQknyHccRpQkSWrIzJYkSWrAh5pOMbMlSZLUkJktSZLUhHO2OgZbkiRp5ArvRpziMKIkSVJDZrYkSdLoVfesLZnZkiRJasrMliRJasJ3I3YMtiRJ0sgV3o04xWFESZKkhsxsSZKkBnyC/JR5mdlKUkk+OLC9bZLvJfnktHYfT/KVaWVnJHnTwPapSc5s32tJkjSJ5mtm6yfAgUl2rKp7gSOBNYMNkuwMHALcnWTfqrqlr/qfwMqBYO2lwNO3TLclSZocPvqhMy8zW71LgN/p148Hzp9W/3zgE8AFwHFThVX1I+BU4N398qaquqt1ZyVJ0mSaz8HWBcBxSXYAngZ8dVr9VAB2fr/+gKo6H1gIPLaqPjDbCZKclGRFkhXf+/66kXZekqStXVVGvsxH8zbYqqqrgX3oAqlLBuuS7AYcAHyxqv4F+HmSAwfq9wR2Bx6fZKcNnGNpVS2pqiW7/tKCBlchSdLWqcpga8q8DbZ6y4DTefgQ4gvoMle3JrmNB4OyKX8LvBn4x/6jJElSE/N1gvyUc4C7qmpVksMHyo8HjqqqLwMkeSLwaeDUJM8BHge8H3g0cHWS91XVdVu055IkbeV89ENnXme2qmp1VZ0xWJZkH2Bv4CsD7W4FfpjkN4B3AX9UnZ8Ar6ObKC9JkrYCSY5KckOSm5K8YYb61yS5LsnVST6TZO+BunVJVvbLslH0Z15mtqrqYfOsquoy4LJ+c48Z6p/Rr/77aeUXAReNtoeSJGkcj35IsgA4k+6xUKuBK5IsmzaC9Q1gSVXdk+RlwF8BL+zr7q2qxaPs07zObEmSpLlrTBPkDwVuqqpbquo+uqcXHP3QftXnquqefvMrwJ4jvfBpDLYkSdLWZA/g9oHt1cww4jXgJcCnBrZ36B/79JUkx4yiQ/NyGFGSJM1tRbNHNSxKsmJge2lVLd2UAyX5fWAJ8BsDxXtX1Zok+wKfTbKqqm7ejP4abEmSpHnlzqpasoH6NcBeA9t7Mu2VfgBJjqB7o8xvVNXPpsqrak3/8ZYkl9G90m+zgi2HESVJUhPVYBnCFcABSZ6YZDu6V/Y95K7CJE8H/h54XlV9d6B8YZLt+/VFwK8Am/1oKDNbkiRp9PonyG/x01bdn+QVwHJgAXBOVV2b5K3AiqpaBvw1sBPwkSQA366q5wFPBv4+yXq6hNRpo3gOp8GWJEnaqlTVJUx7lV9VvWlg/YhZ9rscOGjU/THYkiRJbYzhOVtzkXO2JEmSGjKzJUmSmhjHnK25yGBLkiQ1MY7X9cxFDiNKkiQ1ZGZLkiSNXOEw4hQzW5IkSQ2Z2ZIkSaNXgJktwMyWJElSU2a2JElSE96N2DHYkiRJbRhsAQ4jSpIkNWVmS5IkNRAf/dAzsyVJktSQmS1JktSGc7YAgy1JktRC+QT5KQ4jSpIkNWRmS5IkteEwImCwNbRVa3dlvwtOHnc3trjabjJ/Uibxaw1w83FnjbsLY7P/+ZP5NV8/oT/jrHd4S1uOwZYkSWrEoBYMtiRJUisTmjidzgnykiRJDZnZkiRJbZjZAsxsSZIkNWVmS5IkjV4BPtQUMLMlSZLUlJktSZLURDlnCzDYkiRJrRhsAQ4jSpIkNWVmS5IkteEEecDMliRJUlNmtiRJUhNxzhZgsCVJkloonCDfcxhRkiSpITNbkiSpgThBvmdmS5IkqSEzW5IkqQ3nbAEGW5IkqRWDLcBhREmSpKbMbEmSpDbMbAFmtiRJkpoysyVJkkav8NEPPTNbkiRJDZnZkiRJTfhuxI7BliRJasNgC3AYUZIkqamtIthKUkk+OLC9bZLvJflkv31iknf3629Jck+Sxw20v3vL91qSJLWQ5KgkNyS5KckbZqjfPsk/9PVfTbLPQN0b+/Ibkvz2KPqzVQRbwE+AA5Ps2G8fCazZQPs7gdc275UkSdqikiwAzgSeAzwFOD7JU6Y1ewnwg6raH3gn8Jf9vk8BjgOeChwF/H/98TbL1hJsAVwC/E6/fjxw/gbangO8MMkuzXslSdKESo1+GcKhwE1VdUtV3QdcABw9rc3RwHn9+oXAs5OkL7+gqn5WVbcCN/XH2yxbU7B1AXBckh2ApwFf3UDbu+kCrj/Z0AGTnJRkRZIV637yk9H1VJKkSVAZ/QKLpv4298tJ0866B3D7wPbqvmzGNlV1P/BD4JeG3PcR22ruRqyqq/sx1+PpslwbcwawMsnpGzjmUmApwPZ77eU9FZIkjd+dVbVk3J14JLamzBbAMuB0NjyECEBV3QV8GHh54z5JkjR5qtGycWuAvQa29+Th87gfaJNkW+AXge8Pue8jtrUFW+cAf1ZVq4Zs/w7gv7MVZfgkSZpwVwAHJHliku3oJrwvm9ZmGXBCv34s8Nmqqr78uP5uxScCBwBf29wObVVBRlWtphseHLb9nUk+Bry6Xa8kSZpQY5iAU1X3J3kFsBxYAJxTVdcmeSuwoqqWAe8FPpDkJmAtXUBG3+4fgeuA+4GXV9W6ze3TVhFsVdVOM5RdBlzWr58LnNuvv2Vau9cAr2nbQ0mSJs+4XtdTVZcwbf52Vb1pYP2nwH+ZZd+3AW8bZX+2tmFESZKkOWWryGxJkqQ5yPv4ATNbkiRJTZnZkiRJbZjZAsxsSZIkNWVmS5IkjdwjeJfhVs9gS5IktdG9y3DiOYwoSZLUkJktSZLUhsOIgJktSZKkpsxsSZKkJpwg3zHYkiRJbRhsAQ4jSpIkNWVmS5IkjZ7P2XqAmS1JkqSGzGxJkqQ2zGwBBluSJKkVgy3AYURJkqSmzGxJkqQmnCDfMbMlSZLUkMGWJElSQwZbkiRJDTlnS5IkteGcLcBgS5IkteAT5B/gMKIkSVJDZra0QbXtZP63ZJufZtxdGIv9zz953F0Ym5uOP2vcXRiLf/f+l427C2Ox7Y8n82d8i5vMPyEPY2ZLkiSpITNbkiSpDTNbgMGWJElqIDhBforDiJIkSQ2Z2ZIkSW2Y2QLMbEmSJDVlZkuSJI2eDzV9gMGWJElqw2ALcBhRkiSpKTNbkiSpDTNbgJktSZKkpsxsSZKkJpwg3zGzJUmS1JCZLUmS1IaZLcBgS5IktVAYbPUcRpQkSWrIzJYkSWrCCfIdM1uSJEkNmdmSJEltmNkCzGxJkqRGUqNfNqs/yS5JLk1yY/9x4QxtFif5cpJrk1yd5IUDdecmuTXJyn5ZPMx5DbYkSdKkeAPwmao6APhMvz3dPcCLq+qpwFHAu5LsPFD/uqpa3C8rhzmpwZYkSWqjGiyb52jgvH79POCYh3W56l+q6sZ+/V+B7wK7bs5JDbYkSdKk2K2q7ujX/w3YbUONkxwKbAfcPFD8tn548Z1Jth/mpE6QlyRJo9fuoaaLkqwY2F5aVUunNpJ8GvjlGfY79SHdq6pk9llgSXYHPgCcUFXr++I30gVp2wFLgdcDb91Yhw22JEnSyKVfGrizqpbMVllVR8xWl+Q7SXavqjv6YOq7s7R7LPBPwKlV9ZWBY09lxX6W5H3AKcN02GFESZI0KZYBJ/TrJwAXT2+QZDvgY8D7q+rCaXW79x9DN9/rmmFOarAlSZLamHsT5E8DjkxyI3BEv02SJUnO7tu8APh14MQZHvHwoSSrgFXAIuAvhjnpvBlGTLKO7uK2Ba6nG0O9Z1r5rcAfVNVd/T6nAC8Ffgr8HPi7qnp/ksuA3fvyu4E/rKobtuwVSZKkLamqvg88e4byFXTxAlX1QeCDs+z/rE0573zKbN3bP9PiQOA+4OQZytcCLwdIcjJwJHBoVS2m++QODh+/qKoOprv186+30DVIkjQx5tpDTcdlPgVbg74A7D9D+ZeBPfr1PwVeVlU/AqiqH1XVeTPs8/lZjiVJkrTZ5l2wlWRb4Dl0Q4eD5QvoslfL+rsIHlNVtwxxyN+dfqyBY56UZEWSFet+8pPN7LkkSRNm7s3ZGot5M2cL2DHJyn79C8B7p5XvQTeX61LgF4Y43oeS3AvcBrxypgb9czuWAmy/117z9EssSdKY+JcTmF/B1r393KsZy5M8GlgOvLyqzkhyd5J9N5DdelE/IU6SJKmZeTeMOJuqugf4Y+C1/VDj24Ez+yFFkuyU5MXj7KMkSROjweT4+TpBfj5ltjaqqr6R5GrgeOA9wE7AFUl+Tvfoh78ZZ/8kSdLkmTfBVlXtNEx5Vf3uwOZf9cv0fQ4faeckSdLDzdNM1KjNm2BLkiTNL/N12G/Utpo5W5IkSXORmS1JktSGmS3AzJYkSVJTZrYkSVITztnqGGxJkqTRm8ev1xk1hxElSZIaMrMlSZLaMLMFmNmSJElqysyWJEkaueAE+SlmtiRJkhoysyVJktowswUYbEmSpEZSRlvgMKIkSVJTZrYkSdLo+VDTB5jZkiRJasjMliRJasJHP3QMtiRJUhsGW4DDiJIkSU2Z2ZIkSU04jNgxsyVJktSQmS1JktSGmS3AYEuSJLVQDiNOcRhRkiSpITNbkiSpDTNbgJktSZKkpsxsPQLb3J9xd2GL2+6OyfwW+flj1o+7C2OxfrvJ/W/ov3v/y8bdhbH4lxe/Z9xdGIsnffEPxt2FrV5wztaUyfxLKkmS2iujLXAYUZIkqSkzW5IkqQmHETtmtiRJkhoysyVJkkav8NEPPTNbkiRJDZnZkiRJTWQyn6LzMAZbkiSpDYcRAYcRJUmSmjKzJUmSmvDRDx0zW5IkaSIk2SXJpUlu7D8unKXduiQr+2XZQPkTk3w1yU1J/iHJdsOc12BLkiSNXtG9rmfUy+Z5A/CZqjoA+Ey/PZN7q2pxvzxvoPwvgXdW1f7AD4CXDHNSgy1JktREavTLZjoaOK9fPw84ZuhrSQI8C7jwke5vsCVJkibFblV1R7/+b8Bus7TbIcmKJF9Jckxf9kvAXVV1f7+9GthjmJM6QV6SJLXRZoL8oiQrBraXVtXSqY0knwZ+eYb9Tn1I16oqmTVXtndVrUmyL/DZJKuAH25qhw22JEnSfHJnVS2ZrbKqjpitLsl3kuxeVXck2R347izHWNN/vCXJZcDTgY8COyfZts9u7QmsGabDDiNKkqSRC3NyztYy4IR+/QTg4of1O1mYZPt+fRHwK8B1VVXA54BjN7T/TAy2JEnS6LW4E3Hz70Y8DTgyyY3AEf02SZYkObtv82RgRZKr6IKr06rqur7u9cBrktxEN4frvcOc1GFESZI0Earq+8CzZyhfAby0X78cOGiW/W8BDn2k5zXYkiRJTfgE+Y7DiJIkSQ2Z2ZIkSW2Y2QLMbEmSJDVlZkuSJDXhnK2OwZYkSRq9AtYbbYHDiJIkSU2NLdhKUkk+OLC9bZLvJfnkQNkxSa5Ocn2SVQMvgyTJuUnWDD7lNcltSQ5KsrJf1ia5tV//dJJ9klwzrR9vSXLKFrhkSZImSzVY5qFxDiP+BDgwyY5VdS9wJAPvGEpyMHA6cGRV3ZrkicClSW6pqqv7ZuuAPwTeM7VfVa0CFvfHOBf4ZFVd2G/v0/qiJEmSBo17GPES4Hf69eOB8wfqTgH+d1XdCtB/fDvwuoE27wJencS5Z5IkzTFz8N2IYzHuYOsC4LgkOwBPA746UPdU4Mpp7Vf05VO+DXwR+INHcM79BoYZVwInP+JeS5KkjZt770Yci7FmhKrq6n5o73i6LNemeDvdW7f/acj2N1fV4qmNJG+ZrWGSk4CTALbdeeEmdk+SJE2ycWe2AJbRzc06f1r5dcAh08oOAa4dLKiqG4GVwAtG3bGqWlpVS6pqyTY7/cKoDy9J0lbNYcTOXJjrdA5wV1WtSnL4QPnpwEeSfLaqbuszYH8KHDvDMd7G8JktSZKkLWbswVZVrQbOmKF8ZZLXA59I8ijg58D/qKqVM7S9NsnXgWe07q8kSRrCPH5Uw6iNLdiqqp1mKLsMuGxg+yLgoln2P3Ha9vOHaHMbcOC0srcM2WVJkjSkAJmnE9pHbS7M2ZIkSdpqjX0YUZIkbaXWj7sDc4OZLUmSpIbMbEmSpCacs9UxsyVJktSQmS1JkjR6PvrhAQZbkiSpgfn7LsNRcxhRkiSpITNbkiSpifn6LsNRM7MlSZLUkJktSZLUhnO2AIMtSZLUQkF8gjzgMKIkSVJTZrYkSVIbDiMCZrYkSZKaMrMlSZLaMLEFGGxJkqRGfBF1x2FESZKkhsxsSZKkNsxsAWa2JEmSmjKzJUmSRq8AH2oKmNmSJElqysyWJEkauVDejdgz2JIkSW0YbAEOI0qSJDVlZkuSJLVhZgsw2BpeYN0Ok3dbxa5XZtxdGIt/O2zcPRiT9ZP59QbY9seTee1P+uIfjLsLY/HNX/3AuLswFgvG3YEJZbAlSZJGz0c/PMBgS5IkNeHdiB0nyEuSJDVksCVJktqoGv2yGZLskuTSJDf2HxfO0OY3k6wcWH6a5Ji+7twktw7ULR7mvAZbkiRpUrwB+ExVHQB8pt9+iKr6XFUtrqrFwLOAe4D/O9DkdVP1VbVymJMabEmSpAYaZLU2fw7Y0cB5/fp5wDEbaX8s8KmqumdzTmqwJUmSRq9oFWwtSrJiYDnpEfRqt6q6o1//N2C3jbQ/Djh/Wtnbklyd5J1Jth/mpN6NKEmS5pM7q2rJbJVJPg388gxVpw5uVFUlmTVVlmR34CBg+UDxG+mCtO2ApcDrgbdurMMGW5IkqY0xPGerqo6YrS7Jd5LsXlV39MHUdzdwqBcAH6uqnw8ceyor9rMk7wNOGaZPDiNKkqRJsQw4oV8/Abh4A22PZ9oQYh+gkSR0872uGeakZrYkSVITc/ChpqcB/5jkJcC36LJXJFkCnFxVL+239wH2Av552v4fSrIrEGAlcPIwJzXYkiRJE6Gqvg88e4byFcBLB7ZvA/aYod2zNuW8BluSJKmNuZfZGguDLUmSNHoFrDfYAifIS5IkNWVmS5IkNTCSJ75vFcxsSZIkNWRmS5IktWFmCzDYkiRJrRhsAQ4jSpIkNWVmS5IkjZ6PfniAmS1JkqSGzGxJkqQGCmr9uDsxJxhsSZKkNpwgDziMKEmS1JSZLUmSNHpOkH/ARjNbSS7fSP3d07ZPTPLujexzeJJPDtfFjfbvLUlOGTj34wfqbkuyaBTnkSRJ2hQbDbaq6rAt0ZERORF4/MYaSZKkLaBq9Ms8NExm6+7+4+5JPp9kZZJrkvzaEPuem+SMJJcnuSXJsQPVOyW5MMk3k3woSfp9Dknyz0muTLI8ye59+X9LckWSq5J8NMmjp53rWGAJ8KG+jzv2Va9M8vUkq5I8Kck2SW5Msmu/3zZJbpraliRJGqVHMkH+94DlVbUYOBhYOeR+uwO/CjwXOG2g/OnAq4CnAPsCv5LkUcDfAcdW1SHAOcDb+vYXVdV/qKqDgeuBlwyepKouBFYAL6qqxVV1b191Z1U9A3gPcEpVrQc+CLyorz8CuKqqvjfk9UiSpGGY2QIe2QT5K4Bz+oDo41W1cgNtBz8bH+8DnOuS7DZQ/rWqWg2QZCWwD3AXcCBwaZ/oWgDc0bc/MMlfADsDOwHLh+z3Rf3HK4Hn9+vnABcD7wL+EHjfTDsmOQk4CWDBwp2HPJ0kSeqeszU/g6NRGzqzVVWfB34dWAOcm+TFfdW9SbYbaLoLcOfA9s8G1jNL+Tq6wC/AtX1manFVHVRVv9W3ORd4RVUdBPwZsMOQXZ86z9Q5qKrbge8keRZwKPCpmXasqqVVtaSqlizYaachTydJkvSgoYOtJHsD36mq/wOcDTyjr/pn4Pf7NjsCLwA+t4n9uQHYNckz++M9KslT+7rHAHf0mbUXzbL/j/t2wzibbjjxI1W1bhP7K0mSZlLA+vWjX+ahRzJn63DgqiTfAF4I/G1f/ifA8/uhwK/QBS+f35TOVNV9wLHAXya5im5e2NTdkP8L+CrwJeCbsxziXOCsaRPkZ7OMbjhyxiFESZKkUdjonK2q2qn/eB5w3gz1a+gmv8+074mzHOsy4LKB8lcMrK+kG66cfqz30E1yn17+loH1jwIfHajeZ6BuBV3AOOVguonxswVukiRpczhnC5jQJ8gneQPwMmYfjpQkSZvLYAuY0HcjVtVpVbV3VX1x3H2RJElbt4nMbEmSpNbKdyP2JjKzJUmStKWY2ZIkSaNX0D3TXAZbkiSpDYcRAYcRJUmSmjKzJUmS2vDRD4CZLUmSpKbMbEmSpNGrmrfvMhw1M1uSJEkNmdmSJEltOGcLMNiSJEmNlMOIgMOIkiRJTZnZkiRJDZTDiD0zW5IkSQ2Z2ZIkSaNX+LqensGWJElqwxdRAw4jSpIkNWVmS5IkjVwB5TAiYGZLkiSpKTNbkiRp9Kqcs9UzsyVJkpqo9TXyZXMk+S9Jrk2yPsmSDbQ7KskNSW5K8oaB8icm+Wpf/g9JthvmvAZbkiRpUlwDPB/4/GwNkiwAzgSeAzwFOD7JU/rqvwTeWVX7Az8AXjLMSQ22JElSG7V+9MvmdKfq+qq6YSPNDgVuqqpbquo+4ALg6CQBngVc2Lc7DzhmmPMabEmSJD1oD+D2ge3VfdkvAXdV1f3TyjfKCfJDuu/21Xd+65Wv+9aYTr8IuHMcJx7XBffGdt1cMJazThnfdY+X1z1ZxnrdC8Z14s44r33vLXWiH/OD5Z+uCxc1OPQOSVYMbC+tqqVTG0k+DfzyDPudWlUXN+jPRhlsDamqdh3XuZOsqKpZJ/JtrbzuyeJ1T5ZJvW6YnGuvqqPGdN4jNvMQa4C9Brb37Mu+D+ycZNs+uzVVvlEOI0qSJD3oCuCA/s7D7YDjgGVVVcDngGP7dicAQ2XKDLYkSdJESPL/JlkNPBP4pyTL+/LHJ7kEoM9avQJYDlwP/GNVXdsf4vXAa5LcRDeH673DnNdhxPlh6cabbJW87snidU+WSb1umOxrH6uq+hjwsRnK/xX4TwPblwCXzNDuFrq7FR+RdFkxSZIkteAwoiRJUkMGW5IkSQ0ZbEmSJDVksCVJktSQwZYkSVJDBluSJEkNGWxJkiQ19P8DjsDDCNe8TPYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x576 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize correlations in our merged dataframe...\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "correlations = merged_df.corr()\n",
    "fig = plt.figure(figsize=(10, 8))\n",
    "ax = fig.add_subplot(111)\n",
    "cax = ax.matshow(correlations, vmin=-1, vmax=1)\n",
    "fig.colorbar(cax)\n",
    "ticks = np.arange(0, len(merged_df.columns), 1)\n",
    "ax.set_xticks(ticks)\n",
    "ax.set_yticks(ticks)\n",
    "ax.set_xticklabels(merged_df.columns)\n",
    "ax.set_yticklabels(merged_df.columns)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "219a6c62-70eb-41ad-ae88-5ccd27f28aab",
   "metadata": {},
   "source": [
    "## Begin ML on Merged Dataframe...\n",
    "In order to build, validate, and test our ML model, we'll begin by creating three dataframes:\n",
    "1. **train_df:** This is split from the full merged_df data as the larger chunk and used for training.\n",
    "2. **validate_df:** This is split from the full merged_df data as the smaller chunk and used for validation.\n",
    "3. **test_df:** This is the validation dataframe, but with the true target label values removed to enable model testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "66bce085-f0ac-418f-b3e9-5e1da7cefbd9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n",
      "Number of training samples: 1463\n",
      "Number of validation samples: 721\n"
     ]
    }
   ],
   "source": [
    "# Additional import statements for autogluon+sklearn and split out train_df + validate_df data...\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "from autogluon.tabular import TabularPredictor\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "train_df, validate_df = train_test_split(merged_df, test_size=0.33, random_state=1)\n",
    "print('Number of training samples:', len(train_df))\n",
    "print('Number of validation samples:', len(validate_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cff0133a-7dfe-4187-8dda-026d0695ec92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DEWP</th>\n",
       "      <th>WDSP</th>\n",
       "      <th>MAX</th>\n",
       "      <th>MIN</th>\n",
       "      <th>PRCP</th>\n",
       "      <th>MONTH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>41.9</td>\n",
       "      <td>4.7</td>\n",
       "      <td>89.1</td>\n",
       "      <td>55.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>994</th>\n",
       "      <td>40.8</td>\n",
       "      <td>2.5</td>\n",
       "      <td>62.1</td>\n",
       "      <td>37.9</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1820</th>\n",
       "      <td>42.2</td>\n",
       "      <td>5.5</td>\n",
       "      <td>90.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1442</th>\n",
       "      <td>50.0</td>\n",
       "      <td>4.3</td>\n",
       "      <td>73.9</td>\n",
       "      <td>52.0</td>\n",
       "      <td>0.06</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>596</th>\n",
       "      <td>39.7</td>\n",
       "      <td>2.7</td>\n",
       "      <td>66.0</td>\n",
       "      <td>39.9</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>849</th>\n",
       "      <td>53.0</td>\n",
       "      <td>5.5</td>\n",
       "      <td>93.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>208</th>\n",
       "      <td>37.2</td>\n",
       "      <td>5.6</td>\n",
       "      <td>91.9</td>\n",
       "      <td>57.9</td>\n",
       "      <td>0.00</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>288</th>\n",
       "      <td>37.1</td>\n",
       "      <td>2.8</td>\n",
       "      <td>66.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1226</th>\n",
       "      <td>39.3</td>\n",
       "      <td>6.8</td>\n",
       "      <td>99.0</td>\n",
       "      <td>62.1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2129</th>\n",
       "      <td>42.3</td>\n",
       "      <td>5.9</td>\n",
       "      <td>73.9</td>\n",
       "      <td>42.1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>721 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      DEWP  WDSP   MAX   MIN  PRCP  MONTH\n",
       "241   41.9   4.7  89.1  55.0  0.00     11\n",
       "994   40.8   2.5  62.1  37.9  0.00      1\n",
       "1820  42.2   5.5  90.0  63.0  0.00      5\n",
       "1442  50.0   4.3  73.9  52.0  0.06      4\n",
       "596   39.7   2.7  66.0  39.9  0.00     12\n",
       "...    ...   ...   ...   ...   ...    ...\n",
       "849   53.0   5.5  93.0  64.0  0.00      8\n",
       "208   37.2   5.6  91.9  57.9  0.00     10\n",
       "288   37.1   2.8  66.0  36.0  0.00     12\n",
       "1226  39.3   6.8  99.0  62.1  0.00      9\n",
       "2129  42.3   5.9  73.9  42.1  0.00      4\n",
       "\n",
       "[721 rows x 6 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create the test_df data and remove the target label column...\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "test_df=validate_df.drop([AQbyWeather.mlTargetLabel], axis=1)\n",
    "display(test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be7097cd-6b74-4824-8eec-06d526ee7f67",
   "metadata": {},
   "source": [
    "## Run AutoGluon TabularPredictor.fit(...)\n",
    "- Instantiate TabularPredictor with explicit values for label, eval_metric, and path.\n",
    "- Call TabularPredictor.fit(...) using our train_df tabular data as the main input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "499df50a-239b-4173-a91f-f86941d69b15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: path already exists! This predictor may overwrite an existing predictor! path=\"AutogluonModels/aq_bakersfield_pm25_2016-2022/\"\n",
      "Presets specified: ['best_quality']\n",
      "Beginning AutoGluon training ...\n",
      "AutoGluon will save models to \"AutogluonModels/aq_bakersfield_pm25_2016-2022/\"\n",
      "AutoGluon Version:  0.4.2\n",
      "Python Version:     3.9.13\n",
      "Operating System:   Linux\n",
      "Train Data Rows:    1463\n",
      "Train Data Columns: 6\n",
      "Label Column: isUnhealthy\n",
      "Preprocessing data ...\n",
      "AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).\n",
      "\t2 unique label values:  [0, 1]\n",
      "\tIf 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "Selected class <--> label mapping:  class 1 = 1, class 0 = 0\n",
      "Using Feature Generators to preprocess the data ...\n",
      "Fitting AutoMLPipelineFeatureGenerator...\n",
      "\tAvailable Memory:                    15321.24 MB\n",
      "\tTrain Data (Original)  Memory Usage: 0.07 MB (0.0% of available memory)\n",
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n",
      "\tStage 1 Generators:\n",
      "\t\tFitting AsTypeFeatureGenerator...\n",
      "\tStage 2 Generators:\n",
      "\t\tFitting FillNaFeatureGenerator...\n",
      "\tStage 3 Generators:\n",
      "\t\tFitting IdentityFeatureGenerator...\n",
      "\tStage 4 Generators:\n",
      "\t\tFitting DropUniqueFeatureGenerator...\n",
      "\tTypes of features in original data (raw dtype, special dtypes):\n",
      "\t\t('float', []) : 5 | ['DEWP', 'WDSP', 'MAX', 'MIN', 'PRCP']\n",
      "\t\t('int', [])   : 1 | ['MONTH']\n",
      "\tTypes of features in processed data (raw dtype, special dtypes):\n",
      "\t\t('float', []) : 5 | ['DEWP', 'WDSP', 'MAX', 'MIN', 'PRCP']\n",
      "\t\t('int', [])   : 1 | ['MONTH']\n",
      "\t0.0s = Fit runtime\n",
      "\t6 features in original data used to generate 6 features in processed data.\n",
      "\tTrain Data (Processed) Memory Usage: 0.07 MB (0.0% of available memory)\n",
      "Data preprocessing and feature engineering runtime = 0.04s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n",
      "\tTo change this, specify the eval_metric parameter of Predictor()\n",
      "AutoGluon will fit 2 stack levels (L1 to L2) ...\n",
      "Fitting 13 L1 models ...\n",
      "Fitting model: KNeighborsUnif_BAG_L1 ...\n",
      "\t0.7416\t = Validation score   (accuracy)\n",
      "\t0.0s\t = Training   runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: KNeighborsDist_BAG_L1 ...\n",
      "\t0.7573\t = Validation score   (accuracy)\n",
      "\t0.0s\t = Training   runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: LightGBMXT_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "2022-06-14 22:38:13,323\tWARNING services.py:1856 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 4294967296 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=4.77gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.\n",
      "\t0.8373\t = Validation score   (accuracy)\n",
      "\t5.38s\t = Training   runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: LightGBM_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8182\t = Validation score   (accuracy)\n",
      "\t6.25s\t = Training   runtime\n",
      "\t0.03s\t = Validation runtime\n",
      "Fitting model: RandomForestGini_BAG_L1 ...\n",
      "\t0.812\t = Validation score   (accuracy)\n",
      "\t0.77s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: RandomForestEntr_BAG_L1 ...\n",
      "\t0.8059\t = Validation score   (accuracy)\n",
      "\t0.67s\t = Training   runtime\n",
      "\t0.15s\t = Validation runtime\n",
      "Fitting model: CatBoost_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8278\t = Validation score   (accuracy)\n",
      "\t6.0s\t = Training   runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "Fitting model: ExtraTreesGini_BAG_L1 ...\n",
      "\t0.8141\t = Validation score   (accuracy)\n",
      "\t0.77s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: ExtraTreesEntr_BAG_L1 ...\n",
      "\t0.8072\t = Validation score   (accuracy)\n",
      "\t0.69s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: NeuralNetFastAI_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8018\t = Validation score   (accuracy)\n",
      "\t12.3s\t = Training   runtime\n",
      "\t0.12s\t = Validation runtime\n",
      "Fitting model: XGBoost_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8148\t = Validation score   (accuracy)\n",
      "\t5.06s\t = Training   runtime\n",
      "\t0.07s\t = Validation runtime\n",
      "Fitting model: NeuralNetTorch_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8291\t = Validation score   (accuracy)\n",
      "\t17.65s\t = Training   runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: LightGBMLarge_BAG_L1 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8195\t = Validation score   (accuracy)\n",
      "\t8.32s\t = Training   runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "Fitting model: WeightedEnsemble_L2 ...\n",
      "\t0.8373\t = Validation score   (accuracy)\n",
      "\t0.98s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Fitting 11 L2 models ...\n",
      "Fitting model: LightGBMXT_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8401\t = Validation score   (accuracy)\n",
      "\t6.48s\t = Training   runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: LightGBM_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8483\t = Validation score   (accuracy)\n",
      "\t7.34s\t = Training   runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: RandomForestGini_BAG_L2 ...\n",
      "\t0.8346\t = Validation score   (accuracy)\n",
      "\t0.83s\t = Training   runtime\n",
      "\t0.15s\t = Validation runtime\n",
      "Fitting model: RandomForestEntr_BAG_L2 ...\n",
      "\t0.8325\t = Validation score   (accuracy)\n",
      "\t0.77s\t = Training   runtime\n",
      "\t0.15s\t = Validation runtime\n",
      "Fitting model: CatBoost_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8442\t = Validation score   (accuracy)\n",
      "\t11.68s\t = Training   runtime\n",
      "\t0.04s\t = Validation runtime\n",
      "Fitting model: ExtraTreesGini_BAG_L2 ...\n",
      "\t0.8189\t = Validation score   (accuracy)\n",
      "\t0.73s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: ExtraTreesEntr_BAG_L2 ...\n",
      "\t0.8257\t = Validation score   (accuracy)\n",
      "\t0.68s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: NeuralNetFastAI_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8325\t = Validation score   (accuracy)\n",
      "\t12.15s\t = Training   runtime\n",
      "\t0.23s\t = Validation runtime\n",
      "Fitting model: XGBoost_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8476\t = Validation score   (accuracy)\n",
      "\t6.37s\t = Training   runtime\n",
      "\t0.07s\t = Validation runtime\n",
      "Fitting model: NeuralNetTorch_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8284\t = Validation score   (accuracy)\n",
      "\t11.78s\t = Training   runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: LightGBMLarge_BAG_L2 ...\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.8401\t = Validation score   (accuracy)\n",
      "\t13.94s\t = Training   runtime\n",
      "\t0.11s\t = Validation runtime\n",
      "Fitting model: WeightedEnsemble_L3 ...\n",
      "\t0.8483\t = Validation score   (accuracy)\n",
      "\t0.93s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 177.76s ... Best model: \"WeightedEnsemble_L3\"\n",
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/aq_bakersfield_pm25_2016-2022/\")\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<autogluon.tabular.predictor.predictor.TabularPredictor at 0x7f1c2de96af0>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use AutoGluon TabularPredictor to fit a model for our training data...\n",
    "display(AQbyWeather.selectedScenario.getSummary()) #Using display for consistent/sequential output order.\n",
    "predictor = TabularPredictor(label=AQbyWeather.mlTargetLabel, \n",
    "                             eval_metric=AQbyWeather.mlEvalMetric, \n",
    "                             path=AQbyWeather.selectedScenario.getModelPath())\n",
    "predictor.fit(train_data=train_df, time_limit=AQbyWeather.mlTimeLimitSecs, verbosity=2, presets='best_quality')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "021921b8-6133-46b3-aed5-8e733e627aab",
   "metadata": {},
   "source": [
    "## Evaluating Our Autogluon Model...\n",
    "- Use TabularPredictor.feature_importance(...) with our validation_df to get feature importance data.\\\n",
    "  (this data is displayed further below)\n",
    "- Use TabularPredictor.leaderboard(...) with our validation_df to get individual model performance data.\\\n",
    "  (this data is displayed further below)\n",
    "- Display TabularPredictor.evaluate(...) to output various model quality characteristics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ace98288-c321-4ea9-ac20-b163d9753af1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing feature importance via permutation shuffling for 6 features using 721 rows with 5 shuffle sets...\n",
      "\t63.95s\t= Expected runtime (12.79s per shuffle set)\n",
      "\t15.71s\t= Actual runtime (Completed 5 of 5 shuffle sets)\n",
      "Evaluation: accuracy on test data: 0.7947295423023578\n",
      "Evaluations on test data:\n",
      "{\n",
      "    \"accuracy\": 0.7947295423023578,\n",
      "    \"balanced_accuracy\": 0.7923707440100882,\n",
      "    \"mcc\": 0.5820230438897135,\n",
      "    \"roc_auc\": 0.874984237074401,\n",
      "    \"f1\": 0.7620578778135049,\n",
      "    \"precision\": 0.7476340694006309,\n",
      "    \"recall\": 0.7770491803278688\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Get dataframes for feature importance + model leaderboard AND get+display model evaluation...\n",
    "display(AQbyWeather.selectedScenario.getSummary()) #Using display for consistent/sequential output order.\n",
    "featureimp_df   = predictor.feature_importance(validate_df)\n",
    "leaderboard_df  = predictor.leaderboard(validate_df, silent=True)\n",
    "modelEvaluation = predictor.evaluate(validate_df, auxiliary_metrics=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6d9a3a88-aab0-49f8-a6d4-ba20474bfd0b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>score_test</th>\n",
       "      <th>score_val</th>\n",
       "      <th>pred_time_test</th>\n",
       "      <th>pred_time_val</th>\n",
       "      <th>fit_time</th>\n",
       "      <th>pred_time_test_marginal</th>\n",
       "      <th>pred_time_val_marginal</th>\n",
       "      <th>fit_time_marginal</th>\n",
       "      <th>stack_level</th>\n",
       "      <th>can_infer</th>\n",
       "      <th>fit_order</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NeuralNetTorch_BAG_L2</td>\n",
       "      <td>0.812760</td>\n",
       "      <td>0.828435</td>\n",
       "      <td>1.367442</td>\n",
       "      <td>1.244300</td>\n",
       "      <td>75.653306</td>\n",
       "      <td>0.148462</td>\n",
       "      <td>0.127412</td>\n",
       "      <td>11.780471</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RandomForestEntr_BAG_L2</td>\n",
       "      <td>0.804438</td>\n",
       "      <td>0.832536</td>\n",
       "      <td>1.344696</td>\n",
       "      <td>1.265710</td>\n",
       "      <td>64.646867</td>\n",
       "      <td>0.125717</td>\n",
       "      <td>0.148821</td>\n",
       "      <td>0.774032</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ExtraTreesGini_BAG_L2</td>\n",
       "      <td>0.803051</td>\n",
       "      <td>0.818865</td>\n",
       "      <td>1.352479</td>\n",
       "      <td>1.281073</td>\n",
       "      <td>64.600034</td>\n",
       "      <td>0.133500</td>\n",
       "      <td>0.164185</td>\n",
       "      <td>0.727199</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>LightGBMXT_BAG_L1</td>\n",
       "      <td>0.801664</td>\n",
       "      <td>0.837321</td>\n",
       "      <td>0.072428</td>\n",
       "      <td>0.043975</td>\n",
       "      <td>5.381871</td>\n",
       "      <td>0.072428</td>\n",
       "      <td>0.043975</td>\n",
       "      <td>5.381871</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>WeightedEnsemble_L2</td>\n",
       "      <td>0.801664</td>\n",
       "      <td>0.837321</td>\n",
       "      <td>0.074507</td>\n",
       "      <td>0.046960</td>\n",
       "      <td>6.365828</td>\n",
       "      <td>0.002078</td>\n",
       "      <td>0.002985</td>\n",
       "      <td>0.983958</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>CatBoost_BAG_L2</td>\n",
       "      <td>0.801664</td>\n",
       "      <td>0.844156</td>\n",
       "      <td>1.232805</td>\n",
       "      <td>1.160382</td>\n",
       "      <td>75.556857</td>\n",
       "      <td>0.013825</td>\n",
       "      <td>0.043494</td>\n",
       "      <td>11.684022</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>XGBoost_BAG_L2</td>\n",
       "      <td>0.801664</td>\n",
       "      <td>0.847573</td>\n",
       "      <td>1.281233</td>\n",
       "      <td>1.191477</td>\n",
       "      <td>70.239682</td>\n",
       "      <td>0.062253</td>\n",
       "      <td>0.074588</td>\n",
       "      <td>6.366847</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>LightGBMLarge_BAG_L2</td>\n",
       "      <td>0.801664</td>\n",
       "      <td>0.840055</td>\n",
       "      <td>1.448996</td>\n",
       "      <td>1.226459</td>\n",
       "      <td>77.812670</td>\n",
       "      <td>0.230016</td>\n",
       "      <td>0.109571</td>\n",
       "      <td>13.939835</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NeuralNetFastAI_BAG_L2</td>\n",
       "      <td>0.801664</td>\n",
       "      <td>0.832536</td>\n",
       "      <td>1.493159</td>\n",
       "      <td>1.346558</td>\n",
       "      <td>76.021008</td>\n",
       "      <td>0.274179</td>\n",
       "      <td>0.229669</td>\n",
       "      <td>12.148173</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>LightGBMXT_BAG_L2</td>\n",
       "      <td>0.800277</td>\n",
       "      <td>0.840055</td>\n",
       "      <td>1.286816</td>\n",
       "      <td>1.161328</td>\n",
       "      <td>70.349639</td>\n",
       "      <td>0.067837</td>\n",
       "      <td>0.044440</td>\n",
       "      <td>6.476804</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>RandomForestGini_BAG_L2</td>\n",
       "      <td>0.800277</td>\n",
       "      <td>0.834586</td>\n",
       "      <td>1.347594</td>\n",
       "      <td>1.263777</td>\n",
       "      <td>64.703335</td>\n",
       "      <td>0.128615</td>\n",
       "      <td>0.146888</td>\n",
       "      <td>0.830500</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>RandomForestGini_BAG_L1</td>\n",
       "      <td>0.798890</td>\n",
       "      <td>0.812030</td>\n",
       "      <td>0.132431</td>\n",
       "      <td>0.155190</td>\n",
       "      <td>0.770934</td>\n",
       "      <td>0.132431</td>\n",
       "      <td>0.155190</td>\n",
       "      <td>0.770934</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>LightGBMLarge_BAG_L1</td>\n",
       "      <td>0.797503</td>\n",
       "      <td>0.819549</td>\n",
       "      <td>0.101114</td>\n",
       "      <td>0.046891</td>\n",
       "      <td>8.318884</td>\n",
       "      <td>0.101114</td>\n",
       "      <td>0.046891</td>\n",
       "      <td>8.318884</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>ExtraTreesEntr_BAG_L2</td>\n",
       "      <td>0.797503</td>\n",
       "      <td>0.825701</td>\n",
       "      <td>1.349804</td>\n",
       "      <td>1.281102</td>\n",
       "      <td>64.551041</td>\n",
       "      <td>0.130825</td>\n",
       "      <td>0.164214</td>\n",
       "      <td>0.678206</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ExtraTreesGini_BAG_L1</td>\n",
       "      <td>0.796117</td>\n",
       "      <td>0.814081</td>\n",
       "      <td>0.139428</td>\n",
       "      <td>0.155610</td>\n",
       "      <td>0.767929</td>\n",
       "      <td>0.139428</td>\n",
       "      <td>0.155610</td>\n",
       "      <td>0.767929</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>RandomForestEntr_BAG_L1</td>\n",
       "      <td>0.794730</td>\n",
       "      <td>0.805878</td>\n",
       "      <td>0.127543</td>\n",
       "      <td>0.152630</td>\n",
       "      <td>0.670142</td>\n",
       "      <td>0.127543</td>\n",
       "      <td>0.152630</td>\n",
       "      <td>0.670142</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>ExtraTreesEntr_BAG_L1</td>\n",
       "      <td>0.794730</td>\n",
       "      <td>0.807245</td>\n",
       "      <td>0.140027</td>\n",
       "      <td>0.158820</td>\n",
       "      <td>0.690295</td>\n",
       "      <td>0.140027</td>\n",
       "      <td>0.158820</td>\n",
       "      <td>0.690295</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>LightGBM_BAG_L2</td>\n",
       "      <td>0.794730</td>\n",
       "      <td>0.848257</td>\n",
       "      <td>1.261405</td>\n",
       "      <td>1.155427</td>\n",
       "      <td>71.213395</td>\n",
       "      <td>0.042426</td>\n",
       "      <td>0.038539</td>\n",
       "      <td>7.340560</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>WeightedEnsemble_L3</td>\n",
       "      <td>0.794730</td>\n",
       "      <td>0.848257</td>\n",
       "      <td>1.263417</td>\n",
       "      <td>1.158216</td>\n",
       "      <td>72.141569</td>\n",
       "      <td>0.002012</td>\n",
       "      <td>0.002789</td>\n",
       "      <td>0.928173</td>\n",
       "      <td>3</td>\n",
       "      <td>True</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>CatBoost_BAG_L1</td>\n",
       "      <td>0.793343</td>\n",
       "      <td>0.827751</td>\n",
       "      <td>0.009780</td>\n",
       "      <td>0.050919</td>\n",
       "      <td>5.999756</td>\n",
       "      <td>0.009780</td>\n",
       "      <td>0.050919</td>\n",
       "      <td>5.999756</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>LightGBM_BAG_L1</td>\n",
       "      <td>0.793343</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.056508</td>\n",
       "      <td>0.032668</td>\n",
       "      <td>6.247002</td>\n",
       "      <td>0.056508</td>\n",
       "      <td>0.032668</td>\n",
       "      <td>6.247002</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>NeuralNetTorch_BAG_L1</td>\n",
       "      <td>0.790569</td>\n",
       "      <td>0.829118</td>\n",
       "      <td>0.164121</td>\n",
       "      <td>0.106688</td>\n",
       "      <td>17.652812</td>\n",
       "      <td>0.164121</td>\n",
       "      <td>0.106688</td>\n",
       "      <td>17.652812</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>NeuralNetFastAI_BAG_L1</td>\n",
       "      <td>0.789182</td>\n",
       "      <td>0.801777</td>\n",
       "      <td>0.198543</td>\n",
       "      <td>0.116403</td>\n",
       "      <td>12.301760</td>\n",
       "      <td>0.198543</td>\n",
       "      <td>0.116403</td>\n",
       "      <td>12.301760</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>XGBoost_BAG_L1</td>\n",
       "      <td>0.779473</td>\n",
       "      <td>0.814764</td>\n",
       "      <td>0.059501</td>\n",
       "      <td>0.073944</td>\n",
       "      <td>5.063055</td>\n",
       "      <td>0.059501</td>\n",
       "      <td>0.073944</td>\n",
       "      <td>5.063055</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>KNeighborsUnif_BAG_L1</td>\n",
       "      <td>0.748960</td>\n",
       "      <td>0.741627</td>\n",
       "      <td>0.008464</td>\n",
       "      <td>0.012038</td>\n",
       "      <td>0.004562</td>\n",
       "      <td>0.008464</td>\n",
       "      <td>0.012038</td>\n",
       "      <td>0.004562</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>KNeighborsDist_BAG_L1</td>\n",
       "      <td>0.747573</td>\n",
       "      <td>0.757348</td>\n",
       "      <td>0.009090</td>\n",
       "      <td>0.011113</td>\n",
       "      <td>0.003834</td>\n",
       "      <td>0.009090</td>\n",
       "      <td>0.011113</td>\n",
       "      <td>0.003834</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      model  score_test  score_val  pred_time_test  \\\n",
       "0     NeuralNetTorch_BAG_L2    0.812760   0.828435        1.367442   \n",
       "1   RandomForestEntr_BAG_L2    0.804438   0.832536        1.344696   \n",
       "2     ExtraTreesGini_BAG_L2    0.803051   0.818865        1.352479   \n",
       "3         LightGBMXT_BAG_L1    0.801664   0.837321        0.072428   \n",
       "4       WeightedEnsemble_L2    0.801664   0.837321        0.074507   \n",
       "5           CatBoost_BAG_L2    0.801664   0.844156        1.232805   \n",
       "6            XGBoost_BAG_L2    0.801664   0.847573        1.281233   \n",
       "7      LightGBMLarge_BAG_L2    0.801664   0.840055        1.448996   \n",
       "8    NeuralNetFastAI_BAG_L2    0.801664   0.832536        1.493159   \n",
       "9         LightGBMXT_BAG_L2    0.800277   0.840055        1.286816   \n",
       "10  RandomForestGini_BAG_L2    0.800277   0.834586        1.347594   \n",
       "11  RandomForestGini_BAG_L1    0.798890   0.812030        0.132431   \n",
       "12     LightGBMLarge_BAG_L1    0.797503   0.819549        0.101114   \n",
       "13    ExtraTreesEntr_BAG_L2    0.797503   0.825701        1.349804   \n",
       "14    ExtraTreesGini_BAG_L1    0.796117   0.814081        0.139428   \n",
       "15  RandomForestEntr_BAG_L1    0.794730   0.805878        0.127543   \n",
       "16    ExtraTreesEntr_BAG_L1    0.794730   0.807245        0.140027   \n",
       "17          LightGBM_BAG_L2    0.794730   0.848257        1.261405   \n",
       "18      WeightedEnsemble_L3    0.794730   0.848257        1.263417   \n",
       "19          CatBoost_BAG_L1    0.793343   0.827751        0.009780   \n",
       "20          LightGBM_BAG_L1    0.793343   0.818182        0.056508   \n",
       "21    NeuralNetTorch_BAG_L1    0.790569   0.829118        0.164121   \n",
       "22   NeuralNetFastAI_BAG_L1    0.789182   0.801777        0.198543   \n",
       "23           XGBoost_BAG_L1    0.779473   0.814764        0.059501   \n",
       "24    KNeighborsUnif_BAG_L1    0.748960   0.741627        0.008464   \n",
       "25    KNeighborsDist_BAG_L1    0.747573   0.757348        0.009090   \n",
       "\n",
       "    pred_time_val   fit_time  pred_time_test_marginal  pred_time_val_marginal  \\\n",
       "0        1.244300  75.653306                 0.148462                0.127412   \n",
       "1        1.265710  64.646867                 0.125717                0.148821   \n",
       "2        1.281073  64.600034                 0.133500                0.164185   \n",
       "3        0.043975   5.381871                 0.072428                0.043975   \n",
       "4        0.046960   6.365828                 0.002078                0.002985   \n",
       "5        1.160382  75.556857                 0.013825                0.043494   \n",
       "6        1.191477  70.239682                 0.062253                0.074588   \n",
       "7        1.226459  77.812670                 0.230016                0.109571   \n",
       "8        1.346558  76.021008                 0.274179                0.229669   \n",
       "9        1.161328  70.349639                 0.067837                0.044440   \n",
       "10       1.263777  64.703335                 0.128615                0.146888   \n",
       "11       0.155190   0.770934                 0.132431                0.155190   \n",
       "12       0.046891   8.318884                 0.101114                0.046891   \n",
       "13       1.281102  64.551041                 0.130825                0.164214   \n",
       "14       0.155610   0.767929                 0.139428                0.155610   \n",
       "15       0.152630   0.670142                 0.127543                0.152630   \n",
       "16       0.158820   0.690295                 0.140027                0.158820   \n",
       "17       1.155427  71.213395                 0.042426                0.038539   \n",
       "18       1.158216  72.141569                 0.002012                0.002789   \n",
       "19       0.050919   5.999756                 0.009780                0.050919   \n",
       "20       0.032668   6.247002                 0.056508                0.032668   \n",
       "21       0.106688  17.652812                 0.164121                0.106688   \n",
       "22       0.116403  12.301760                 0.198543                0.116403   \n",
       "23       0.073944   5.063055                 0.059501                0.073944   \n",
       "24       0.012038   0.004562                 0.008464                0.012038   \n",
       "25       0.011113   0.003834                 0.009090                0.011113   \n",
       "\n",
       "    fit_time_marginal  stack_level  can_infer  fit_order  \n",
       "0           11.780471            2       True         24  \n",
       "1            0.774032            2       True         18  \n",
       "2            0.727199            2       True         20  \n",
       "3            5.381871            1       True          3  \n",
       "4            0.983958            2       True         14  \n",
       "5           11.684022            2       True         19  \n",
       "6            6.366847            2       True         23  \n",
       "7           13.939835            2       True         25  \n",
       "8           12.148173            2       True         22  \n",
       "9            6.476804            2       True         15  \n",
       "10           0.830500            2       True         17  \n",
       "11           0.770934            1       True          5  \n",
       "12           8.318884            1       True         13  \n",
       "13           0.678206            2       True         21  \n",
       "14           0.767929            1       True          8  \n",
       "15           0.670142            1       True          6  \n",
       "16           0.690295            1       True          9  \n",
       "17           7.340560            2       True         16  \n",
       "18           0.928173            3       True         26  \n",
       "19           5.999756            1       True          7  \n",
       "20           6.247002            1       True          4  \n",
       "21          17.652812            1       True         12  \n",
       "22          12.301760            1       True         10  \n",
       "23           5.063055            1       True         11  \n",
       "24           0.004562            1       True          1  \n",
       "25           0.003834            1       True          2  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# View Autogluon Individual Model Leaderboard...\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "display(leaderboard_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "80cb9042-12b7-4659-beed-20c71bb3f13b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>importance</th>\n",
       "      <th>stddev</th>\n",
       "      <th>p_value</th>\n",
       "      <th>n</th>\n",
       "      <th>p99_high</th>\n",
       "      <th>p99_low</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>WDSP</th>\n",
       "      <td>0.110125</td>\n",
       "      <td>0.022908</td>\n",
       "      <td>0.000212</td>\n",
       "      <td>5</td>\n",
       "      <td>0.157293</td>\n",
       "      <td>0.062957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MONTH</th>\n",
       "      <td>0.102913</td>\n",
       "      <td>0.012382</td>\n",
       "      <td>0.000025</td>\n",
       "      <td>5</td>\n",
       "      <td>0.128408</td>\n",
       "      <td>0.077418</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MAX</th>\n",
       "      <td>0.051318</td>\n",
       "      <td>0.003923</td>\n",
       "      <td>0.000004</td>\n",
       "      <td>5</td>\n",
       "      <td>0.059395</td>\n",
       "      <td>0.043240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PRCP</th>\n",
       "      <td>0.033564</td>\n",
       "      <td>0.005668</td>\n",
       "      <td>0.000094</td>\n",
       "      <td>5</td>\n",
       "      <td>0.045235</td>\n",
       "      <td>0.021894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MIN</th>\n",
       "      <td>0.014147</td>\n",
       "      <td>0.007685</td>\n",
       "      <td>0.007328</td>\n",
       "      <td>5</td>\n",
       "      <td>0.029970</td>\n",
       "      <td>-0.001676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DEWP</th>\n",
       "      <td>0.009986</td>\n",
       "      <td>0.008459</td>\n",
       "      <td>0.028801</td>\n",
       "      <td>5</td>\n",
       "      <td>0.027404</td>\n",
       "      <td>-0.007432</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       importance    stddev   p_value  n  p99_high   p99_low\n",
       "WDSP     0.110125  0.022908  0.000212  5  0.157293  0.062957\n",
       "MONTH    0.102913  0.012382  0.000025  5  0.128408  0.077418\n",
       "MAX      0.051318  0.003923  0.000004  5  0.059395  0.043240\n",
       "PRCP     0.033564  0.005668  0.000094  5  0.045235  0.021894\n",
       "MIN      0.014147  0.007685  0.007328  5  0.029970 -0.001676\n",
       "DEWP     0.009986  0.008459  0.028801  5  0.027404 -0.007432"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='feature'>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAAEiCAYAAAAGQD92AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvYklEQVR4nO3de7iVZZn48e/tRsXkYCI55gkYDxxEQDcUEmiaSJeM1ISJh4LBpkQtxxm9ol+NmjP109FfWUqaBR5KBwg7MB6SyiglVNCQ0/aAioI6SSiKmgfg/v2xFtvNdgPvxr332hu+n+taF+/7HN51v3vp5uZZz/O8kZlIkiRJ2rqdKh2AJEmS1FaYPEuSJEkFmTxLkiRJBZk8S5IkSQWZPEuSJEkFmTxLkiRJBbWrdACNsddee2W3bt0qHYYkSZK2Yw899NBfM7NrQ3VtKnnu1q0b8+fPr3QYkiRJ2o5FxDObq3PahiRJklSQybMkSZJUkMmzJEmSVFCbmvMsSZK0vXvnnXdYuXIlb775ZqVD2e61b9+e/fbbj5133rlwH5NnSZKkVmTlypV07NiRbt26ERGVDme7lZmsXr2alStX0r1798L9nLYhSZLUirz55pt06dLFxLmZRQRdunRp9Ai/ybMkSVIrY+LcMrbl52zyLEmSpE0cddRRLfp+y5cv59Zbb23R99xWznlu5Wp69ircttejNc0YiSRJqoRuE+9o0ustv+zErbb505/+1KTvuSXr1q2rTZ5PO+20FnvfbeXIsyRJkjbRoUMHAGbPns3RRx/NqFGj6NGjBxMnTuSWW25h0KBB9O3blyeffBKAcePGcdZZZ1FdXc0hhxzC7bffDpTmb//TP/0Tffv2ZcCAAfz+978H4MYbb+Skk07i2GOP5bjjjmPixInce++99O/fn+9+97ssX76coUOHcsQRR3DEEUfUJvOzZ8/mmGOOYfTo0fTs2ZPTTz+dzARg3rx5HHXUUfTr149Bgwaxdu1a1q9fz4UXXsjAgQM5/PDD+eEPf/i+fzaOPEuSJGmzHnnkEWpqathzzz3p0aMHX/jCF3jwwQf53ve+x9VXX81VV10FlKZePPjggzz55JN8/OMfZ9myZUyaNImIYNGiRTz66KMMHz6cxx9/HICHH36YhQsXsueeezJ79myuvPLK2qT7jTfe4De/+Q3t27fniSee4NRTT2X+/PkA/PnPf2bJkiV8+MMfZsiQIcyZM4dBgwZxyimnMG3aNAYOHMirr77KbrvtxuTJk+ncuTPz5s3jrbfeYsiQIQwfPrxRu2vUZ/IsSZKkzRo4cCD77LMPAH//93/P8OHDAejbt2/tSDLAZz/7WXbaaScOPvhgevTowaOPPsp9993Hl7/8ZQB69uzJgQceWJs8H3/88ey5554Nvuc777zDueeey4IFC6iqqqrtAzBo0CD2228/APr378/y5cvp3Lkz++yzDwMHDgSgU6dOAMyaNYuFCxcyY8YMAF555RWeeOIJk2dJkiQ1j1133bX2eKeddqo932mnnVi3bl1tXf2dK7a2k8Xuu+++2brvfve77L333jzyyCNs2LCB9u3bNxhPVVXVJjHUl5lcffXVnHDCCVuMpTEKzXmOiBER8VhELIuIiQ3UD4uIhyNiXUSMrle3PiIWlF8z65R3j4gHytecFhG7vP/bkSRJUiX87Gc/Y8OGDTz55JM89dRTHHrooQwdOpRbbrkFgMcff5xnn32WQw899D19O3bsyNq1a2vPX3nlFfbZZx922mknfvKTn7B+/fotvvehhx7KCy+8wLx58wBYu3Yt69at44QTTuDaa6/lnXfeqY3h9ddff1/3udWR54ioAiYBxwMrgXkRMTMzl9Zp9iwwDriggUv8LTP7N1B+OfDdzJwaEdcBZwLXNi58SZIktQYHHHAAgwYN4tVXX+W6666jffv2nH322UyYMIG+ffvSrl07brzxxk1Gjjc6/PDDqaqqol+/fowbN46zzz6bz3zmM9x8882MGDFii6PUALvssgvTpk3jy1/+Mn/729/Ybbfd+O1vf8sXvvAFli9fzhFHHEFm0rVrV375y1++r/uMjSsUN9sgYjBwSWaeUD7/GkBm/t8G2t4I3J6ZM+qUvZaZHeq1C2AV8HeZua7+e2xOdXV1bpwsvqNwqzpJknYsNTU19OpV/O//1mDcuHGMHDmS0aNHb71xK9PQzzsiHsrM6obaF5m2sS+wos75ynJZUe0jYn5E3B8RnyqXdQHWZObGSSqNvaYkSZLU4lpiweCBmflcRPQA7omIRcArRTtHxBeBL0Lp6wBJkiS1LjfeeGOlQ2gxRUaenwP2r3O+X7mskMx8rvznU8BsYACwGtgjIjYm75u9ZmZen5nVmVndtWvXom8rSZIkNbkiyfM84ODy7hi7AGOAmVvpA0BEfDAidi0f7wUMAZZmaaL174GNE2PGAr9qbPCSJElSS9pq8lyel3wucDdQA0zPzCURcWlEnAQQEQMjYiVwMvDDiFhS7t4LmB8Rj1BKli+rs0vHV4F/jYhllOZAT27KG5MkSZKaWqE5z5l5J3BnvbKL6hzPozT1on6/PwF9N3PNp4BBjQlWkiRJqqRCD0mRJEnSjuuqq67ijTfeaLDuxhtv5Nxzz22wrkOHDg2Wt2U+nluSJKk1u6RzE1+v8KZnta666irOOOMMPvCBDzRtLG2QI8+SJEmq9frrr3PiiSfSr18/DjvsML75zW/y/PPP8/GPf5yPf/zjANxwww0ccsghDBo0iDlz5tT2ffrppxk8eDB9+/blG9/4xibXveKKKxg4cCCHH344F198MQATJ05k0qRJtW0uueQSrrzyyha4y21n8ixJkqRav/71r/nwhz/MI488wuLFi/mXf/kXPvzhD/P73/+e3//+97zwwgtcfPHFzJkzh/vuu4+lS5fW9j3vvPOYMGECixYtYp999qktnzVrFk888QQPPvggCxYs4KGHHuKPf/wjp5xyCtOnT69tN336dE455ZQWvd/GMnmWJElSrb59+/Kb3/yGr371q9x777107rzptJEHHniAY445hq5du7LLLrtskuzOmTOHU089FYDPfe5zteWzZs1i1qxZDBgwgCOOOIJHH32UJ554ggEDBvDiiy/y/PPP88gjj/DBD36Q/fffn9bMOc+SJEmqdcghh/Dwww9z55138o1vfIPjjjuuUf0j4j1lmcnXvvY1vvSlL72n7uSTT2bGjBn87//+b6sfdQZHniVJklTH888/zwc+8AHOOOMMLrzwQh5++GE6duzI2rVrAfjIRz7CH/7wB1avXs0777zDz372s9q+Q4YMYerUqQDccsstteUnnHACU6ZM4bXXXgPgueee48UXXwTglFNOYerUqcyYMYOTTz65pW5zmznyLEmSpFqLFi3iwgsvZKeddmLnnXfm2muvZe7cuYwYMaJ27vMll1zC4MGD2WOPPejfv39t3+9973ucdtppXH755YwaNaq2fPjw4dTU1DB48GCgtIXdT3/6Uz70oQ/Rp08f1q5dy7777rvJPOnWKkpPym4bqqurc/78+ZUOo0XV9OxVuG2vR2uaMRJJktQSampq6NWr+N//en8a+nlHxEOZWd1Qe6dtSJIkSQWZPEuSJEkFmTxLkiRJBZk8S5IkSQWZPEuSJEkFmTxLkiRJBZk8S5IkSQX5kBRJkqRWrO9NfZv0eovGLmrS6zXGMcccw5VXXkl1dYNbKLcJjjxLkiRJBZk8S5Ikqdby5cvp2bMnp59+Or169WL06NG88cYb72n361//mpNPPrn2fPbs2YwcORKACRMmUF1dTZ8+fbj44osbfJ8OHTrUHs+YMYNx48YBsGrVKj7zmc8wcOBABg4cyJw5c5rw7t4/k2dJkiRt4rHHHuPss8+mpqaGTp068YMf/OA9bT7xiU/wwAMP8PrrrwMwbdo0xowZA8C3vvUt5s+fz8KFC/nDH/7AwoULC7/3eeedx/nnn8+8efO47bbb+MIXvtA0N9VETJ4lSZK0if33358hQ4YAcMYZZ3Dfffe9p027du0YMWIE//M//8O6deu44447GDVqFADTp0/niCOOYMCAASxZsoSlS5cWfu/f/va3nHvuufTv35+TTjqJV199lddee61pbqwJFFowGBEjgO8BVcCPM/OyevXDgKuAw4ExmTmjXN4fuBboBKwHvpWZ08p1NwJHA6+ULzMuMxe8r7uRWpmanr0Kt+31aE0zRiJJUnERscXzjcaMGcM111zDnnvuSXV1NR07duTpp5/myiuvZN68eXzwgx9k3LhxvPnmm1t8j7r1GzZs4P7776d9+/ZNdDdNa6sjzxFRBUwCPgn0Bk6NiN71mj0LjANurVf+BvD5zOwDjACuiog96tRfmJn9y68F23QHkiRJalLPPvssc+fOBeDWW2/lYx/7WIPtjj76aB5++GF+9KMf1U7ZePXVV9l9993p3Lkzf/nLX7jrrrsa7Lv33ntTU1PDhg0b+MUvflFbPnz4cK6++ura8wULFjTRXTWNIiPPg4BlmfkUQERMBUYBtePvmbm8XLehbsfMfLzO8fMR8SLQFVjzfgOXJEnaEVRia7lDDz2USZMmMX78eHr37s2ECRMabFdVVcXIkSO58cYbuemmmwDo168fAwYMoGfPnptM/6jvsssuY+TIkXTt2pXq6uraqRnf//73Oeecczj88MNZt24dw4YN47rrrmueG90GRZLnfYEVdc5XAh9p7BtFxCBgF+DJOsXfioiLgN8BEzPzrcZeV5IkSU2rXbt2/PSnPy3U9pprruGaa67ZpOzGG29ssO3s2bNrj0ePHs3o0aPf02avvfZi2rRphWNtaS2yYDAi9gF+AvxTZm4cnf4a0BMYCOwJfHUzfb8YEfMjYv6qVataIlxJkiSpQUWS5+eA/euc71cuKyQiOgF3AF/PzPs3lmfmC1nyFnADpekh75GZ12dmdWZWd+3atejbSpIkaRt069aNxYsXb1L26U9/mv79+2/yuvvuuysUYWUVmbYxDzg4IrpTSprHAKcVuXhE7AL8Arh54w4cder2ycwXorTU8lPA4gYuIUmSpAqru6BvR7fVkefMXAecC9wN1ADTM3NJRFwaEScBRMTAiFgJnAz8MCKWlLt/FhgGjIuIBeVX/3LdLRGxCFgE7AX8Z1PemCRJktTUCu3znJl3AnfWK7uozvE8StM56vf7KdDgbPPMPLZRkUqSJEkV5hMGJUmSpIJMniVJkqSCCk3bkCRJUmXU9OzVpNfr9WjNNvW75557uOCCC3j77bc58sgjmTx5Mu3atePll19m/PjxPPnkk7Rv354pU6Zw2GGHNXiN5cuXM3LkyPfs5gFw0UUXMWzYMD7xiU9sNoZLLrmEDh06cMEFF2zTPTQFR54lSZK0RRs2bGDs2LFMnTqVxYsXc+CBB9Y+UfDb3/42/fv3Z+HChdx8882cd9552/Qel1566RYT59bC5FmSJEm1li9fTs+ePTn99NPp1asXo0ePZsWKFeyyyy4ccsghABx//PHcdtttACxdupRjjy3tA9GzZ0+WL1/OX/7yl81ef/369fzzP/8zffr0Yfjw4fztb38DYNy4ccyYUdrZ+M4776Rnz54ceeSRfOUrX2HkyJG1/ZcuXcoxxxxDjx49+P73v98sP4MtMXmWJEnSJh577DHOPvtsampq6NSpE9OnT2fdunXMnz8fgBkzZrBixQoA+vXrx89//nMAHnzwQZ555hlWrly52Ws/8cQTnHPOOSxZsoQ99tijNgnf6M033+RLX/oSd911Fw899BD1nzD96KOPcvfdd/Pggw/yzW9+k3feeacpb32rTJ4lSZK0if33358hQ4YAcMYZZzBnzhymTp3K+eefz6BBg+jYsSNVVVUATJw4kTVr1tC/f3+uvvpqBgwYUFvXkO7du9O/f38AjjzySJYvX75J/aOPPkqPHj3o3r07AKeeeuom9SeeeCK77rore+21Fx/60Ie2OMrdHFwwKEmSpE2UHgC96fngwYO59957AZg1axaPP/44AJ06deKGG24AIDPp3r07PXr02Oy1d91119rjqqqq2mkbRdXvv27dukb1f78ceZYkSdImnn32WebOnQvArbfeysc+9jFefPFFAN566y0uv/xyzjrrLADWrFnD22+/DcCPf/xjhg0bRqdOnbb5vQ899FCeeuqp2hHpadOmvY87aXqOPEuSJLVi27q13Ptx6KGHMmnSJMaPH0/v3r2ZMGECF198MbfffjsbNmxgwoQJtYsEa2pqGDt2LBFBnz59mDx58vt67912240f/OAHjBgxgt13352BAwc2xS01mcjMSsdQWHV1dW6cqL6jaMzejpX4n0tb5ucnSWqsmpoaevVq2r2dG2NLezG3lNdee40OHTqQmZxzzjkcfPDBnH/++c3yXg39vCPiocysbqi90zYkSZLUqvzoRz+if//+9OnTh1deeYUvfelLlQ6pltM2tqDbxDsKt11+2YnNGIkkSVLL6Nat2/sedV69ejXHHXfce8p/97vf0aVLl632P//885ttpPn9MnmWJElSk+rSpQsLFiyodBjNwmkbkiRJUkEmz5IkSVJBJs+SJElSQSbPkiRJUkEuGJQkSWrFJp11T5Ne75zrjt2mfvfccw8XXHABb7/9NkceeSSTJ0+mXbt2vPzyy4wfP54nn3yS9u3bM2XKFA477LDNXqdDhw689tpr2xp+xTnyLEmSpC3asGEDY8eOZerUqSxevJgDDzyQm266CYBvf/vb9O/fn4ULF3LzzTdz3nnnVTja5mXyLEmSpFrLly+nZ8+enH766fTq1YvRo0ezYsUKdtllFw455BAAjj/+eG677TYAli5dWvuo7p49e7J8+XL+8pe/bPV9MpMLL7yQww47jL59+zJt2jQAzjnnHGbOnAnApz/9acaPHw/AlClT+PrXv97k99tYhaZtRMQI4HtAFfDjzLysXv0w4CrgcGBMZs6oUzcW+Eb59D8z86Zy+ZHAjcBuwJ3AedmWnhWuVs0H3EiStO0ee+wxJk+ezJAhQxg/fjzTp09n3bp1zJ8/n+rqambMmMGKFSsA6NevHz//+c8ZOnQoDz74IM888wwrV65k77333uJ7/PznP2fBggU88sgj/PWvf2XgwIEMGzaMoUOHcu+993LSSSfx3HPP8cILLwBw7733MmbMmGa/963Z6shzRFQBk4BPAr2BUyOid71mzwLjgFvr9d0TuBj4CDAIuDgiPliuvhb4Z+Dg8mvENt+FJEmSmsz+++/PkCFDADjjjDOYM2cOU6dO5fzzz2fQoEF07NiRqqoqACZOnMiaNWvo378/V199NQMGDKit25L77ruPU089laqqKvbee2+OPvpo5s2bV5s8L126lN69e7P33nvzwgsvMHfuXI466qhmve8iiow8DwKWZeZTABExFRgFLN3YIDOXl+s21Ot7AvCbzHypXP8bYEREzAY6Zeb95fKbgU8Bd72Pe5EkSVITiIj3nA8ePJh7770XgFmzZvH4448D0KlTJ2644QagNBWje/fu9OjRY5vfe99992XNmjX8+te/ZtiwYbz00ktMnz6dDh060LFjx22+blMpMud5X2BFnfOV5bIiNtd33/LxtlxTkiRJzejZZ59l7ty5ANx666187GMf48UXXwTgrbfe4vLLL+ess84CYM2aNbz99tsA/PjHP2bYsGF06tRpq+8xdOhQpk2bxvr161m1ahV//OMfGTRoEAAf/ehHueqqq2qncVx55ZUMHTq0OW610Vr9VnUR8UXgiwAHHHBAhaORJElqWdu6tdz7ceihhzJp0iTGjx9P7969mTBhAhdffDG33347GzZsYMKECbWLBGtqahg7diwRQZ8+fZg8eXKh9/j0pz/N3Llz6devHxHBf/3Xf/F3f/d3QCmxnjVrFgcddBAHHnggL730UptKnp8D9q9zvl+5rIjngGPq9Z1dLt+vyDUz83rgeoDq6moXFEqSJDWzdu3a8dOf/nSTsiuuuIIrrrjiPW0HDx5cO4WjiI17PEfEZq955plncuaZZwKw88478/rrrzcm/GZVZNrGPODgiOgeEbsAY4CZBa9/NzA8Ij5YXig4HLg7M18AXo2Ij0ZpUs3ngV9tQ/ySJElSi9nqyHNmrouIcyklwlXAlMxcEhGXAvMzc2ZEDAR+AXwQ+IeI+GZm9snMlyLiPygl4ACXblw8CJzNu1vV3YWLBSVJkiquW7duLF68+H1dY/Xq1Rx33HHvKf/d735Hly5d3te1K63QnOfMvJPSXsx1yy6qczyPTadh1G03BZjSQPl8YPPPbpQkSVKb1KVLFxYsWFDpMJqFTxiUJElqZXxuXMvYlp+zybMkSVIr0r59e1avXm0C3cwyk9WrV9O+fftG9Wv1W9VJkiTtSPbbbz9WrlzJqlWrKh3Kdq99+/bst1+DM483y+RZkiSpFdl5553p3r17pcPQZjhtQ5IkSSrI5FmSJEkqyORZkiRJKsjkWZIkSSrI5FmSJEkqyORZkiRJKsjkWZIkSSrI5FmSJEkqyORZkiRJKsjkWZIkSSrI5FmSJEkqyORZkiRJKsjkWZIkSSrI5FmSJEkqyORZkiRJKsjkWZIkSSrI5FmSJEkqqFDyHBEjIuKxiFgWERMbqN81IqaV6x+IiG7l8tMjYkGd14aI6F+um12+5sa6DzXljUmSJElNrd3WGkREFTAJOB5YCcyLiJmZubROszOBlzPzoIgYA1wOnJKZtwC3lK/TF/hlZi6o0+/0zJzfNLcitW2TzrqncNtzrju2GSORJEmbU2TkeRCwLDOfysy3ganAqHptRgE3lY9nAMdFRNRrc2q5ryRJktQmFUme9wVW1DlfWS5rsE1mrgNeAbrUa3MK8N/1ym4oT9n49waSbUmSJKlVaZEFgxHxEeCNzFxcp/j0zOwLDC2/PreZvl+MiPkRMX/VqlUtEK0kSZLUsCLJ83PA/nXO9yuXNdgmItoBnYHVderHUG/UOTOfK/+5FriV0vSQ98jM6zOzOjOru3btWiBcSZIkqXkUSZ7nAQdHRPeI2IVSIjyzXpuZwNjy8WjgnsxMgIjYCfgsdeY7R0S7iNirfLwzMBJYjCRJktSKbXW3jcxcFxHnAncDVcCUzFwSEZcC8zNzJjAZ+ElELANeopRgbzQMWJGZT9Up2xW4u5w4VwG/BX7UJHckSZIkNZOtJs8AmXkncGe9sovqHL8JnLyZvrOBj9Yrex04spGxSpIkSRXlEwYlSZKkgkyeJUmSpIJMniVJkqSCTJ4lSZKkgkyeJUmSpIJMniVJkqSCTJ4lSZKkgkyeJUmSpIJMniVJkqSCCj1hUG3DpLPuaVT7c647tpkikSRJ2j458ixJkiQVZPIsSZIkFWTyLEmSJBXknGdJakBNz16Nat/r0ZpmikSS1Jo48ixJkiQVZPIsSZIkFWTyLEmSJBVk8ixJkiQVZPIsSZIkFWTyLEmSJBVk8ixJkiQVVCh5jogREfFYRCyLiIkN1O8aEdPK9Q9ERLdyebeI+FtELCi/rqvT58iIWFTu8/2IiCa7K0mSJKkZbDV5jogqYBLwSaA3cGpE9K7X7Ezg5cw8CPgucHmduiczs3/5dVad8muBfwYOLr9GbPttSJIkSc2vyBMGBwHLMvMpgIiYCowCltZpMwq4pHw8A7hmSyPJEbEP0Ckz7y+f3wx8CrirkfFL2g51m3hH4bbLLzuxGSORJGlTRaZt7AusqHO+slzWYJvMXAe8AnQp13WPiD9HxB8iYmid9iu3ck1JkiSpVSky8vx+vAAckJmrI+JI4JcR0acxF4iILwJfBDjggAOaIURJkiSpmCIjz88B+9c5369c1mCbiGgHdAZWZ+ZbmbkaIDMfAp4EDim3328r16Tc7/rMrM7M6q5duxYIV5IkSWoeRZLnecDBEdE9InYBxgAz67WZCYwtH48G7snMjIiu5QWHREQPSgsDn8rMF4BXI+Kj5bnRnwd+1QT3I0mSJDWbrU7byMx1EXEucDdQBUzJzCURcSkwPzNnApOBn0TEMuAlSgk2wDDg0oh4B9gAnJWZL5XrzgZuBHajtFDQxYKSJElq1QrNec7MO4E765VdVOf4TeDkBvrdBty2mWvOBw5rTLCSJElSJTX3gkFJklpcTc9ehdv2erSmGSORtL3x8dySJElSQSbPkiRJUkFO25CkJjDprHsKtz3numObMRJJUnNy5FmSJEkqyORZkiRJKsjkWZIkSSrI5FmSJEkqyORZkiRJKsjkWZIkSSrI5FmSJEkqyORZkiRJKsjkWZIkSSrI5FmSJEkqyMdzS5KaVLeJdxRuu/yyE5sxEklqeo48S5IkSQWZPEuSJEkFmTxLkiRJBZk8S5IkSQW5YFCSJLUqNT17FW7b69GaZoxEei+TZ0mSJG1W35v6Fm67aOyiZoykdSg0bSMiRkTEYxGxLCImNlC/a0RMK9c/EBHdyuXHR8RDEbGo/OexdfrMLl9zQfn1oSa7K0mSJKkZbHXkOSKqgEnA8cBKYF5EzMzMpXWanQm8nJkHRcQY4HLgFOCvwD9k5vMRcRhwN7BvnX6nZ+b8JroXSZIkVdCOMOWmyMjzIGBZZj6VmW8DU4FR9dqMAm4qH88AjouIyMw/Z+bz5fIlwG4RsWtTBC5JkiS1tCLJ877AijrnK9l09HiTNpm5DngF6FKvzWeAhzPzrTplN5SnbPx7RESjIpckSZJaWItsVRcRfShN5fhSneLTM7MvMLT8+txm+n4xIuZHxPxVq1Y1f7CSJEnSZhRJnp8D9q9zvl+5rME2EdEO6AysLp/vB/wC+HxmPrmxQ2Y+V/5zLXArpekh75GZ12dmdWZWd+3atcg9SZIkSc2iyFZ184CDI6I7pSR5DHBavTYzgbHAXGA0cE9mZkTsAdwBTMzMORsblxPsPTLzrxGxMzAS+O37vZmKuqRzI9q+0nxxSJIkqdlsNXnOzHURcS6lnTKqgCmZuSQiLgXmZ+ZMYDLwk4hYBrxEKcEGOBc4CLgoIi4qlw0HXgfuLifOVZQS5x814X1JklTIpLPuKdz2nOuO3XojSdu1Qg9Jycw7gTvrlV1U5/hN4OQG+v0n8J+bueyRxcOUJEmSKq9FFgxKkiRJ2wOTZ0mSJKkgk2dJkiSpIJNnSZIkqSCTZ0mSJKkgk2dJkiSpIJNnSZIkqSCTZ0mSJKmgQg9JkSRJ279uE+8o3Hb5ZSc2YyRS6+XIsyRJklSQI8/SJZ0b2f6V5olDkiS1eo48S5IkSQWZPEuSJEkFOW1DkiRpO+CCz5Zh8ixJkrSjacx6n+4HNF8cbZDJsyRJarzGJF8utNZ2xORZkiQ1q7439W1U++nNFIfUFFwwKEmSJBXkyLMkqXL86l9SG+PIsyRJklSQI8+S2jZHLiVJLajQyHNEjIiIxyJiWURMbKB+14iYVq5/ICK61an7Wrn8sYg4oeg1JUmSpNZmq8lzRFQBk4BPAr2BUyOid71mZwIvZ+ZBwHeBy8t9ewNjgD7ACOAHEVFV8JqSJElSq1Jk5HkQsCwzn8rMt4GpwKh6bUYBN5WPZwDHRUSUy6dm5luZ+TSwrHy9IteUJEmSWpUic573BVbUOV8JfGRzbTJzXUS8AnQpl99fr+++5eOtXXO71Zj9Lt3rUpIkqfWIzNxyg4jRwIjM/EL5/HPARzLz3DptFpfbrCyfP0kpGb4EuD8zf1ounwzcVe62xWvWufYXgS8CHHDAAUc+88wz2363kiRpuzLprHsKtz3numObMRI1VmM+O2jZzy8iHsrM6obqikzbeA7Yv875fuWyBttERDugM7B6C32LXBOAzLw+M6szs7pr164FwpUkSZKaR5FpG/OAgyOiO6UEdwxwWr02M4GxwFxgNHBPZmZEzARujYjvAB8GDgYeBKLANSVJkrbI0WS1tK0mz+U5zOcCdwNVwJTMXBIRlwLzM3MmMBn4SUQsA16ilAxTbjcdWAqsA87JzPUADV2z6W9PkiRJajqFHpKSmXcCd9Yru6jO8ZvAyZvp+y3gW0WuKUmSJLVmPp5bkiRJKsjkWZIkSSrI5FmSJEkqyORZkiRJKqjQgkFJkiSpKbXVbQYdeZYkSZIKMnmWJEmSCjJ5liRJkgoyeZYkSZIKMnmWJEmSCjJ5liRJkgoyeZYkSZIKMnmWJEmSCorMrHQMhUXEKuCZSsfRjPYC/lrpILRN/OzaNj+/tsvPrm3z82vbtufP78DM7NpQRZtKnrd3ETE/M6srHYcaz8+ubfPza7v87No2P7+2bUf9/Jy2IUmSJBVk8ixJkiQVZPLculxf6QC0zfzs2jY/v7bLz65t8/Nr23bIz885z5IkSVJBjjxLkiRJBZk8S5IkSQWZPEuSJEkFmTxLkiRJBbWrdAA7qoj4FHAQsCgz765wOGqEiPjXLdVn5ndaKhY1XkRcDfyfzFxbr7wncE1mfqIykWlrIuJg4Erg74FFwAWZ+Vxlo5J2HOYuJY48V0BE/AA4H+gC/EdE/HuFQ1LjdKzzuqDeeccKxqVi/hdYEBGnAUTEByLiv4D/ASZVNDJtzRTgduAzwMPA1ZUNR40REWsj4tXya22d8zciYl2l49OWmbu8y63qKiAiFgP9MnN9RHwAuDczj6x0XGq8iPhzZg6odBxqnIjoQSnx6gh8GJgO/GdmvlHRwLRFEbEgM/vXOX84M4+oYEh6HyKiA3AO8CXgF5n5bxUOSVtg7vIup21UxtuZuR4gM9+IiKh0QNpm/uuzbdpQ/rMdUAXUmDi3Ce0jYgCw8XfmbnXPM/PhikWmwiJiD+BfgM8DtwIDM3N1JWNSIeYuZY48V0BEvAEs23hKaf7esvJxZubhlYpNjePIV9tT/qpxLPD1zJwWEfsC3wO6AhMyc2lFA9RmRcRsNv8P1szMY1swHDVSROwF/BtwCqUpOFdn5iuVjUpFmbu8y+S5AiLiwC3VZ+YzLRWLGi8iFvHuX+AHsekvkx3qF0hbFBHfA77RwILBTwLfycxelYlM2r5FxOvAKuAGYG39ehdbt27mLu9y2kYF1P8PLCK6AMOAZzPzocpEpUYYWekAtO0y87zNlN9V/stdrVREnEFp0Ocn9co/B6zPzFsrE5kKuoJ3Bx5cXN32fBr4E/BwZu7QCzwdea6AiLgdmJiZiyNiH0qrxudT+grk+sy8qpLxacsiYlZmDq90HGoaEdEbOLX8WpOZ1RUOSZsREQ8Ax2Xma/XKdwf+uKMuXpJaQkRcCRwF9KS0VeQcSsn0nzLzpUrG1tJMnisgIpZkZp/y8f8Bembm5yOiIzDHr/1bN3fYaPsiohvvJszvAAcC1Zm5vIJhaSu2tMYgIhb6u7N1i4jvb6k+M7/SUrFo20XELkA1pUR6cPm1JjN7VzSwFuS0jcp4p87xccCPADJzbURsaLiLWpHOEfGPm6vMzJ+3ZDBqnIiYC3QCpgKfycwnIuJpE+c2YbeI2D0zN5leUx542KVCMak4pyVuH3aj9Du0c/n1PKWR6B2GyXNlrIiILwMrgSOAXwNExG7AzpUMTIV0pjTvuaFtehIweW7d/gLsC+xNaYeNJ3DLwbZiMjAjIs7auHak/C3CpHKdWrHMvKnSMWjbRcT1QB9Kiz0foDRl4zuZ+XJFA6sAk+fKOBO4FPgEcEpmrimXf5TSKmS1bs9k5vhKB6Ftk5mfiojOwD8Cl5Qf+bxHRAzKzAcrHJ62IDOvjIjXgD+WH7AB8BpwWWZeW8HQVEBEzNxSfWae1FKxaJscAOxKacDhOUoDgGsqGVClOOdZaiTnPG9fImJv4LPAGOCAzNy/wiGpgPJUjY3T3XYCTs3MWyoclrYgIlYBK4D/pjRyucm3d5n5h0rEpeLKD0bpQ2m+81HAYcBLwNzMvLiSsbUkk+cKiYixwFcorVoFqAG+n5k3Vy4qFRERfTJzSaXjUNOLiAN3pL1K25qI6ETpcc77Ar8Cfls+vwB4JDNHVTA8bUVEVAHHU1qoezhwB/Df/j5teyJiP2AIpQR6JNAlM/eoaFAtyOS5AsqJ878A/0ppm7qgNPf5CuCq+nuYqnWJiKfZdI5s1DnPzPz7lo9KRfnVcdsVEb8CXgbmUlps/SFK//+dl5kLKhiaGikidqWURF8BfDMzr6lwSNqKiDiP0s4aR1Ha+OBPdV6LMnOH2fDA5LkCIuJ+YEz91f3lhS9TM/OjlYhLxZQfalPXTpS+9r+A0ubxn2n5qFSUXx23XRGxKDP7lo+rgBcoTbV5s7KRqahy0nwipcS5GzATmJKZz1UyLm1dRHyH8t7OmflCpeOpJBcMVkanhrbFyszl5a8l1Ypl5mqA8jzLzwEXAguAEzNzaQVDUzF/x7tfHZ+GXx23JbXbfGbm+ohYaeLcdkTEzZTmyN5JabR5cYVDUuPcnpn3AERE98x8emNFRPzjjrRNqyPPFRARD23uSVhbqlPrEBE7A+OB84H7KK30X1bZqLQt/Oq4bYmI9cDrvPttwW7AG+XzzEwHH1qx8nMMNu7R/Z6pb35+rVvdhxTVf2DRlh5gtD1y5LkyekXEwgbKA+jR0sGo0Z4G1gFXAc8Ch0dE7ZPNdqR/fbdVDXx1/H3gF5WMSVuXmVWVjkHbLjN3qnQMel9iM8cNnW/XTJ4r4+uU5g29xKZPG1Tb8FtKoyb9yq+6fEhKK+dXx21XRLQHzgIOAhZSmiu7rrJRSTuM3MxxQ+fbNadtVEBEXElptWpPSo+0nEN5xWpmvlTJ2KTtnV8dt10RMY3SgMO9wCcpPbDovMpGJe0YImIN8EdKvyuHlo8pn38sMz9YodBanMlzBUXELkA1pUR6cPm1JjN7VzQwbVVEHEZpoWCfctES4MrMXFS5qKTtW73dNtoBD+5I8yylSoqIo7dUvyPtVOS0jcraDegEdC6/nqc0Eq1WLCJGAVcC/xf4f+XiauDnEXFBZv6qYsFJ27e6u22sKz3sTFJLqJscR0TXctmqykVUOY48V0BEXE9pxHItpX1m7wfuz8yXKxqYComIR4BRm9mn+1eZWX8etKQmUGe3DSh9VexuG1ILKT+a+yLgy5SebxCUFs9fnZmXVjK2lubK18o4ANgV+F/gOWAlsKaSAalR2m1un25g5xaPRtpBZGZVZnYqvzpmZrs6xybOUvM6H/gYMDAz9yzPcf4IMCQizq9saC3LkecKKf8Lrg+l+c5HUVr9/xIwNzMvrmRs2rLyyPM/ZOaz9coPBP4nMw9vuKckSW1TRPwZOD4z/1qvvCswKzMHVCayluec5wrJ0r9aFpdXr75Sfo0EBgEmz63bxcBvI+LbwEPlsmpgIvDVikUlSVLz2bl+4gylec/lh4ftMEyeKyAivsK7I87vUN6mDpiCCwZbvcz8ZUQ8DfwbpblfAEuBz2bmI5WLTJKkZvP2NtZtd5y2UQER8R3Keztn5guVjkeSJGlL6i3Y3aQKaJ+ZO8zos8mz1EgRMXNL9Zl5UkvFIkmSWpbTNqTGGwysAP6b0laDbjYrSdIOwpFnqZEiogo4HjgVOBy4A/jvzFxS0cAkSVKzc59nqZEyc31m/jozxwIfBZYBsyPi3AqHJkmSmpnTNqRtEBG7AidSGn3uBnwf+EUlY5IkSc3PaRtSI0XEzZQeanMnMDUzF1c4JEmS1EJMnqVGiogNvLtdT93/gYLS8298TLAkSdspk2dJkiSpIBcMSpIkSQWZPEuSJEkFmTxLUisWEV+JiJqIuKWR/bpFxGnNFZck7ahMniWpdTsbOD4zT29kv25Ao5Pn8kOAJEmbYfIsSa1URFwH9ADuioivR8SUiHgwIv4cEaPKbbpFxL0R8XD5dVS5+2XA0IhYEBHnR8S4iLimzrVvj4hjysevRcT/i4hHgMERcUb5fRZExA9NqCXpXSbPktRKZeZZwPPAx4HdgXsyc1D5/IqI2B14kdLI9BHAKZQe2AMwEbg3M/tn5ne38la7Aw9kZj9gdfk6QzKzP7AeaOyotyRtt3zCoCS1DcOBkyLigvJ5e+AASsn1NRHRn1Kie8g2XHs9cFv5+DjgSGBeRADsRilBlyRh8ixJbUUAn8nMxzYpjLgE+AvQj9K3iW9upv86Nv22sX2d4zczc32d97kpM7/WFEFL0vbGaRuS1DbcDXw5ysPBETGgXN4ZeCEzNwCfAzbOT14LdKzTfznQPyJ2ioj9gUGbeZ/fAaMj4kPl99kzIg5s0juRpDbM5FmS2ob/AHYGFkbEkvI5wA+AseXFfj1599HxC4H1EfFIRJwPzAGeBpZSmhf9cENvkplLgW8AsyJiIfAbYJ/muSVJant8PLckSZJUkCPPkiRJUkEmz5IkSVJBJs+SJElSQSbPkiRJUkEmz5IkSVJBJs+SJElSQSbPkiRJUkEmz5IkSVJB/x+4SfqT0TgfAQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# View and Plot Feature Importance... (this various from Scenario to Scenario)\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "display(featureimp_df)\n",
    "featureimp_df.drop(columns=[\"n\"]).plot(kind=\"bar\", figsize=(12, 4), xlabel=\"feature\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7456a6b0-2236-47db-b6a8-dc11f7b3d35c",
   "metadata": {},
   "source": [
    "## Using + Testing Our Model...\n",
    "Now that we've built our Autogluon ML Model, we can use our test_df dataframe to make actual target label predictions and then compare those predictions against the known true label values to determine how the model is performing across each binary classification { 0=OKAY, 1=Unhealthy }.  Further below, we'll plot a _Confusion Matrix_ and the **Summary** explains how to interpret results and covers **Key Learnings**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "af1ec4ac-5e75-406b-8fe1-980ecf8d6986",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "241     1\n",
       "994     1\n",
       "1820    0\n",
       "1442    0\n",
       "596     1\n",
       "       ..\n",
       "849     0\n",
       "208     1\n",
       "288     1\n",
       "1226    0\n",
       "2129    0\n",
       "Name: isUnhealthy, Length: 721, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Load + Use Our Model (this line is unnecessary, but shows how to load a built model)...\n",
    "predictor = TabularPredictor.load(AQbyWeather.selectedScenario.getModelPath())\n",
    "\n",
    "# Make Predictions, which are saved to an array: y_pred\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "y_pred = predictor.predict(test_df)\n",
    "display(y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "efa500e0-4d0b-47be-95fd-6b45a5dea0f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "241     1\n",
       "994     1\n",
       "1820    0\n",
       "1442    0\n",
       "596     1\n",
       "       ..\n",
       "849     1\n",
       "208     1\n",
       "288     1\n",
       "1226    0\n",
       "2129    0\n",
       "Name: isUnhealthy, Length: 721, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get true label values as an array: y_true\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "y_true = validate_df[AQbyWeather.mlTargetLabel]\n",
    "display(y_true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "43d7e462-2462-4fb1-9b6c-c7cd6cd56554",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n",
      "True Negatives  (TN): 336 of 416 => 80.77%\n",
      "True Positives  (TP): 237 of 305 => 77.7%\n",
      "False Negatives (FN): 68 of 305 => 22.3%\n",
      "False Positives (FP): 80 of 416 => 19.23%\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x7f1a60178220>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbAAAAFzCAYAAABfMlwwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeKElEQVR4nO3deZhdVZnv8e+bIjNJIIQhJKFBRuNAtCPihIgDQ6ug1wGHFpUWtcWZvkq37UC39+G2CGo7otJiO2Kjgoqi4oD0RSEgpkkQCYMkIQSSkImMVfXeP84uOGBSVank1Kl19vfzPPvhnHWGvSrkya/etdZeOzITSZJKM6rdHZAkaSgMMElSkQwwSVKRDDBJUpEMMElSkQwwSVKRdmt3B5pNm9qVB84a3e5uSNy2YFK7uyABsLF3HVt6N8Wu/t7jnzMxV67qGfLnb5i/+crMPGEXdmmHjagAO3DWaK67cla7uyFx0uxnt7sLEgDXrr2sJd+7YlUPv7ty5pA/P3r67dN2YXeGZEQFmCRpuCQ92dvuTuwUA0ySaiiBXsreiclFHJKkIlmBSVJN9eIQoiSpMEnSU/hm7gaYJNWUc2CSJLWBFZgk1VACPYVXYAaYJNVU6UOIBpgk1VCCizgkSWUqexG9izgkSYWyApOkGkrSRRySpAIl9JSdXwaYJNVRYzPfshlgklRLQQ+7/D6Zw8pFHJKkIlmBSVINJdDrHJgkqUSlDyEaYJJUQ429EMsOMOfAJElFsgKTpJrqzbIrMANMkmqoE4YQDTBJqqEk6Cl8FskAk6SaKn0Isez4lSTVlhWYJNWQc2CSpEIFPVn2IJwBJkk11NiN3gCTJBWo9CHEsuNXklRbVmCSVEOZzoFJkgrVW/gQogEmSTXUWEZfdgVWdu8lSbVlBSZJteQcmCSpQF4HJkkqVo+b+UqSStN3O5WhHgOJiHERcV1E/CEiFkTER6r2gyLidxGxKCK+HRFjqvax1fNF1esHDnQOA0yS1AqbgeMy80hgDnBCRBwN/F/ggsw8BHgAOL16/+nAA1X7BdX7+mWASVJN9eaoIR8DyYb11dPR1ZHAccB/Ve0XA6dUj0+unlO9/tyI6HeM0zkwSaqhXXAd2LSImNf0/MLMvLD5DRHRBdwAHAJ8BrgdWJ2Z3dVblgAzqsczgMUAmdkdEWuAvYAV2+uAASZJNZTEzi7iWJGZc/s9R2YPMCci9gC+BxyxMyd8NIcQJUktlZmrgV8CTwP2iIi+4mkmsLR6vBSYBVC9PgVY2d/3GmCSVFO9jBryMZCI2LuqvIiI8cDzgVtoBNnLqredBlxWPb68ek71+i8yM/s7h0OIklRDmbR6J47pwMXVPNgo4JLM/GFELAS+FRH/Cvwe+HL1/i8D/xkRi4BVwKkDncAAk6RaipbuRp+Z84EnbaP9DuCobbRvAl6+I+cwwCSphpKWV2AtV3bvJUm1ZQUmSTVV+v3ADDBJqqEk6C18M18DTJJqygpMklSchEHtaTiSld17SVJtWYFJUi0FPS28Dmw4GGCSVEOdMIRogElSTZVegZUdv5Kk2rICk6QaygyHECVJZSp9L0QDTJJqKKGlu9EPBwNMkmopiq/Ayu69JKm2rMAkqYYa14E5hChJKpCb+UqSiuPtVCRJxeotvAIru/eSpNqyApOkGsqEHocQJUklcg5MklScxiKOsmeRyu69JKm2rMAkqaZKvx+YASZJNeROHJKkQpU/B2aAFWTLpuC9Lz2ErVtG0dMNz/qbNbzuH+7l/PfM4k/zJ0DCjMds5qxP3M34ib0A/PryPfjax/eDSB4zexNnf/bPbf4p1KlOed0Sjn/ZvWTCXX+ayAX/dDhT997M+z/+RybtsZVFCyZx3vsPp3tr2f9odhJvp9KPiDgB+CTQBXwpM89t5fk63eixyb9953bGT+yleyu855RDecpxa3nzR5YycVIjsL7w4f25/KJpvPLt97H0jjF8+9/34fzLbmPSHj2sXuHvK2qNvfbZzItfu5S3vGguWzZ3cfb5C3n2SffxlGNW8b2LZ3D1j/fhzA/dxgteei9XfHv/dndXHaJlvwpFRBfwGeBEYDbwqoiY3arz1UEED1VW3VuDnq1BBA+FVyZs3jSKvl+qfvz1vXjR61cwaY8eAPaY1t2WfqseurqSMeN6GdWVjB3Xy6r7x/DEp67mmp/uDcDPv78vT3vuyjb3Un36LmQe6jEStPJX8qOARZl5B0BEfAs4GVjYwnN2vJ4eOPP4w7nnrjG86PUrOOLJGwA4712zuP4XkzngsE2c8cGlACy5YxwA737xIfT2Bq9977085Tnr2tZ3da6V943lu/8xi4uv+h1bNnVx4//bg0ULJvHgut3o7Wn8Y7di+Rj22ndzm3uqZqXPgbWy9zOAxU3Pl1RtjxARZ0TEvIiYd//KnhZ2pzN0dcHnfn4rX79hIbfeNIG7/tgIqbM+sZhv/H4BBxy6mV9fvifQCLuld47lY5cu4uzP/plPnDWL9Wu62tl9dajdJ2/l6ONW8IbnH8Vrj30q48b38tfPWtXubqkffbvRD/UYCdoev5l5YWbOzcy5e+/lP66DtfuUHo58+nqu/+Wkh9q6uuDYkx/gmiumADBt+laOfsFadhsN+x2whZkHb2bpnWPa1WV1sDlPW829S8ex9oEx9HSP4r9/No3ZT1rLxEndjOpKAKbtu4WVy8e2uadq1ksM+RgJWhlgS4FZTc9nVm0aotUrux6qoDZvDG68ehKzmkIpE669cgqzDm4M0zz9hDXMv3Z3ANas7GLJ7WOZfsCW9nReHe3+ZWM54sh1jB3XAyRzjn6Au2+fwPzr9uCZL7gfgOedspzf/mKv9nZUHaWVc2DXA4dGxEE0gutU4NUtPF/HW7V8NOe98wB6e4PeXjjmRas56nlree8ph7BhfReZ8JjZG3n7uUsAmHvsOm789STe9OwjGNWVvOmf72HyVIdptevdOn8y1/x0Gp/6rxvp6QnuuGV3fnzJdK7/9VTed94fed077+L2W3bnykv3a3dXVemEC5kjM1v35REnAZ+gsYz+osz8aH/vn3vkuLzuyln9vUUaFifNfna7uyABcO3ay1jTff8uT5qpj907n3/R/xry5y95+hduyMy5u7BLO6ylFwZl5hXAFa08hyRpCEbQYoyhavsiDkmShsKtGSSphhK3kpIkFar0IUQDTJJqqBNWIRpgklRTpQeYizgkSUWyApOkGurbC7FkBpgk1ZSrECVJ5cny58AMMEmqoU5YhegiDklSkazAJKmmSq/ADDBJqiFXIUqSipWFB5hzYJKkXS4iZkXELyNiYUQsiIh3Vu0fjoilEXFTdZzU9JmzI2JRRNwaEccPdA4rMEmqqRZfB9YNvDczb4yIScANEfGz6rULMvO85jdHxGzgVOBxwP7AzyPisMzc7m3kDTBJqqFs8XVgmbkMWFY9XhcRtwAz+vnIycC3MnMzcGdELAKOAq7d3gccQpSkmsqMIR/AtIiY13Scsb3zRMSBwJOA31VNZ0bE/Ii4KCL2rNpmAIubPraE/gPPCkyS6mmnVyGuyMy5A54lYnfgUuBdmbk2Ij4H/AuNa6n/Bfg48MahdMAKTJLUEhExmkZ4fT0zvwuQmcszsycze4Ev0hgmBFgKzGr6+MyqbbsMMEmqqZ0cQuxXRATwZeCWzDy/qX1609teAtxcPb4cODUixkbEQcChwHX9ncMhREmqoWHYC/EZwN8C/xMRN1Vt/wi8KiLmVF24C3gzQGYuiIhLgIU0VjC+rb8ViGCASVI9ZWMlYsu+PvMa2OY6/Sv6+cxHgY8O9hwGmCTVVOn3A3MOTJJUJCswSaqhpPy9EA0wSaold6OXJBWqlYs4hoNzYJKkIlmBSVJNOQcmSSpOpgEmSSqUizgkSUVyEYckSW1gBSZJNeUcmCSpOMngbosykhlgklRThU+BGWCSVEsdsIzeRRySpCJZgUlSXRU+hmiASVJNlT6EaIBJUk15IbMkSW1gBSZJNeQdmSVJZUrAAJMklaj0OTADTJLqqvAAcxGHJKlIVmCSVEtu5itJKlXhQ4gGmCTVkZv5SpLUHlZgklRXDiFKkspU9hDidgMsIv6dfvI5M9/Rkh5JkoZHB1dg84atF5Kk4depAZaZFzc/j4gJmbmh9V2SJGlgA65CjIinRcRC4I/V8yMj4rMt75kkqXX6NvMd6jECDGYZ/SeA44GVAJn5B+CYFvZJkjQMMod+jASDWoWYmYsjHpG4Pa3pjiRp2IyQIBqqwQTY4oh4OpARMRp4J3BLa7slSWq5ETIUOFSDGUJ8C/A2YAZwDzCnei5JUtsMWIFl5grgNcPQF0nSMIrChxAHswrxMRHxg4i4PyLui4jLIuIxw9E5SVKL5E4eI8BghhC/AVwCTAf2B74DfLOVnZIktdpOLKEfIXNngwmwCZn5n5nZXR1fA8a1umOSJPWnv70Qp1YPfxwR7we+RaNwfCVwxTD0TZLUSiNkKHCo+lvEcQONH6+vVnxz02sJnN2qTkmShkGnBlhmHjScHZEkDbNODbBmEfF4YDZNc1+Z+dVWdUqS1GJ9eyEWbMAAi4gPAcfSCLArgBOBawADTJLUNoNZhfgy4LnAvZn5BuBIYEpLeyVJarnIoR8jwWCGEDdmZm9EdEfEZOA+YFaL+yVJarUREkRDNZgAmxcRewBfpLEycT1wbSs7JUnSQAazF+LfVw8/HxE/ASZn5vzWdkuS1GojZShwqPq7kPnJ/b2WmTfu6s78af4Ejt9/zq7+WmmHLf3uzHZ3QQJgy1mj292FIYmIWTQW++1LY7Dywsz8ZLVJxreBA4G7gFdk5gPRuOnkJ4GTgA3A6wfKmf4qsI/381oCxw3y55AkjUStXUbfDbw3M2+MiEnADRHxM+D1wFWZeW61y9P7gffRWOF+aHU8Ffhc9d/t6u9C5ufskh9BkjTytHhX+cxcBiyrHq+LiFto3FfyZBqXZgFcDPyKRoCdDHw1MxP4bUTsERHTq+/ZpsEso5ckdaKdu53KtIiY13Scsb3TRMSBwJOA3wH7NoXSvTSGGKERboubPrakatuuQe3EIUnSo6zIzLkDvSkidgcuBd6VmWsbU10NmZkRQ19KYgUmSTXV6guZI2I0jfD6emZ+t2peHhHTq9en07i2GGApj7zGeGbVtl2DuSNzRMRrI+KD1fMDIuKowXVfkjRitfCOzNWqwi8Dt2Tm+U0vXQ6cVj0+Dbisqf11VeYcDazpb/4LBjeE+Fmgl8aqw3OAdTQS9SmD+KwkaaRq7XVgzwD+FvifiLipavtH4Fzgkog4Hfgz8IrqtStoLKFfRGMZ/RsGOsFgAuypmfnkiPg9QLVef8yO/BSSpJGl1XsaZuY1PHw/yUd77jben8DbduQcg5kD2xoRXVRZHRF706jIJElqm8FUYJ8CvgfsExEfpbE7/Qda2itJUut1+v3AMvPrEXEDjZIvgFMy85aW90yS1Fqduhdin4g4gMaE2g+a2zLz7lZ2TJLUWh27mW+TH9HI6QDGAQcBtwKPa2G/JEmt1ukBlplPaH5e7VL/99t5uyRJw2KHt5Kqdhbud4dgSdII1+Jl9MNhMHNg72l6Ogp4MnBPy3okSRoenR5gwKSmx9005sQubU13JEnDppMDrLqAeVJmnjVM/ZEkaVC2G2ARsVtmdkfEM4azQ5Kk4dHJc2DX0ZjvuikiLge+AzzY92LT1viSJA27wcyBjQNW0tiNvu96sAQMMEkqWQdXYPtUKxBv5uHg6lP4jy1JNdfhy+i7gN3Z9nb4hf/YkqTS9RdgyzLznGHriSRpeBVeivQXYGXvsy9J6l8HB9hf3DFTktQZgg6eA8vMVcPZEUnSMCs8wEa1uwOSJA3FDu9GL0nqAB2+jF6S1MkMMElSkQoPMOfAJElFsgKTpJpyDkySVCYDTJJUnMQAkySVqfQhRBdxSJKKZAUmSXVVeAVmgElSTZU+hGiASVJdGWCSpOJ0wCpEF3FIkopkBSZJNRTVUTIDTJLqqvAhRANMkmqq9FWIzoFJkopkBSZJdVV4BWaASVJdGWCSpOJk+XNgBpgk1VXhAeYiDklSkazAJKmmHEKUJJXJAJMklaj0Csw5MElSkazAJKmOOuB2KgaYJNWVASZJKk1Q/hyYASZJdVV4gLmIQ5JUJANMkmoqMod8DPjdERdFxH0RcXNT24cjYmlE3FQdJzW9dnZELIqIWyPi+MH03wCTpDrKnTwG9hXghG20X5CZc6rjCoCImA2cCjyu+sxnI6JroBMYYJJUU5FDPwaSmVcDqwbZlZOBb2Xm5sy8E1gEHDXQhwwwSaqrnavApkXEvKbjjEGe9cyImF8NMe5Ztc0AFje9Z0nV1i8DTJI0FCsyc27TceEgPvM54GBgDrAM+PjOdMBl9JJUU8N9HVhmLn/o3BFfBH5YPV0KzGp668yqrV9WYJJUV61dxPEXImJ609OXAH0rFC8HTo2IsRFxEHAocN1A32cFJkl1NMjFGEMVEd8EjqUxV7YE+BBwbETMaZydu4A3A2Tmgoi4BFgIdANvy8yegc5hgEmSdrnMfNU2mr/cz/s/Cnx0R85hgElSXRW+lZQBJkk15Ga+kqRyDWJLqJHMAJOkmiq9AnMZvSSpSFZgklRHO3E910hhgElSTUVvu3uwcwwwSaorKzC108TJPbz7vMUceMQmMuH898xi86ZRvOPcJYwZ10tPd/Dps2dy600T2t1VdZCuFVvZ81NLGLW6BwIefP6ePPjCvZj0jeWMv34dGUHvlC4eePsMeqeOZvfvr2D81WsAiJ5kt6WbWfYfh5OT/CeonUpfxNGyvz0RcRHwQuC+zHx8q85Td289ZynzfjWJfz3jQHYb3cvY8ck/feEuvnb+vsz75WSectxaTv/APfzvlx3S7q6qg+QoWHPafmw9eDyxsYd9zrqDzUdOZP0p01j36n0BmPijlUy+5H5Wv2V/1p8yjfWnTANg3PXr2P0HKw0v7bRWrkL8Ctu+G6d2kQmTenjC0Q/yk29MBaB76ygeXNtFJkyc1NhGbOLkHlYtH93ObqoD9U4dzdaDxwOQ47vYOnMsXSu7yQkP30Q3NvWS8ZefHX/NGjY8a/JwdVXbkzSuAxvqMQK07FegzLw6Ig5s1fcL9jtgC2tWdvHeCxbzmMdt5Lb5E/jcP+/P5z84g//zzTt40weXEZG8+8WHtrur6mBd921h9J2b2HJYI9Amf30543+1mpzQxYpzDnzEe2NzL+N+v57Vf7dfG3qqRyt9CLHt14FFxBl9d/TcyuZ2d6coXV3JIU/YyA+/uhdve8HhbNowileeeR8vPG0lX/jQ/rx27my+8OEZvOf8xQN/mTQEsbGHqf+2mDVv3O+h6mvta/Zl+RcPZ8MxU5j440feUX7c9evYfMR4hw9HimG+ncqu1vYAy8wL++7oOZqx7e5OUVYsG839y0Zz6+8nAnDND6dwyBM28vyXr+KaK6YAcPUPpnDYnA3t7KY6VXcy9WOL2XjMFDYd/ZdDghuPmcL4a9c+om38NWvY+Mwpw9VDdbi2B5iG7oH7R7PinjHMPHgTAHOetZ67bxvHyuWjeeLTHmy0PXM999zpLwbaxTLZ8zNL6Z4xlvUvnvZQc9c9D4+ijLtuHd0zHv67Fw/2MHbhBjYd5fzXSNC3me9Qj5HAOr5wn/nADN736bvZbXRy791j+Pi7Z3HtlZN56zn30NWVbNk8ik/8w8x2d1MdZswfNzDh12vY+ldj2fs9twOw9jX7MPGqB9ht6RYYBd17j2b1m/d/6DPjf7eWTUdOJMf5e/OIMIIWYwxVK5fR/8XdODNzuzcz09DcsWA8bz/xsEe0Lbhud8484bDtfELaeVseO5Gl333cX7Rv/utJ2/3MhuP2ZMNxe7ayW9pBI6WSGqpWrkLc1t04JUkjReEBZi0vSSqSc2CSVFMOIUqSypNAb9kJZoBJUl2VnV8GmCTVVelDiC7ikCQVyQpMkurKC5klSSUqfQjRAJOkOhpBu8oPlXNgkqQiWYFJUg01dqMvuwQzwCSprnrb3YGdY4BJUk1ZgUmSyuMiDkmS2sMKTJJqyTsyS5IK5YXMkqQyWYFJkoqTEIUvo3cRhySpSFZgklRXDiFKkopUdn4ZYJJUV6XvxOEcmCSpSFZgklRXhVdgBpgk1VHibvSSpPIEWfwcmAEmSXVVeIC5iEOSVCQrMEmqq8IrMANMkurIRRySpFK5iEOSVKbCA8xFHJKkXS4iLoqI+yLi5qa2qRHxs4i4rfrvnlV7RMSnImJRRMyPiCcP5hwGmCTVUjYqsKEeA/sKcMKj2t4PXJWZhwJXVc8BTgQOrY4zgM8N5gQGmCTVUdLSAMvMq4FVj2o+Gbi4enwxcEpT+1ez4bfAHhExfaBzOAcmSXU1/KsQ983MZdXje4F9q8czgMVN71tStS2jHwaYJGkopkXEvKbnF2bmhYP9cGZmROzUKhIDTJJqaieX0a/IzLk7+JnlETE9M5dVQ4T3Ve1LgVlN75tZtfXLOTBJqqvWLuLYlsuB06rHpwGXNbW/rlqNeDSwpmmocbuswCSpjhLobd11YBHxTeBYGkONS4APAecCl0TE6cCfgVdUb78COAlYBGwA3jCYcxhgklRLO1VJDfztma/azkvP3cZ7E3jbjp7DIURJUpGswCSprgrfSsoAk6S6MsAkScVp8SKO4WCASVItJWTZNwRzEYckqUhWYJJUV86BSZKK4xyYJKlYhVdgzoFJkopkBSZJdVV4BWaASVIttXYvxOFggElSHSXQW/Z1YAaYJNVV4RWYizgkSUWyApOkuiq8AjPAJKmW0guZJUkFSkg385UkafhZgUlSXTmEKEkqkos4JEnFyfRCZklSoQqvwFzEIUkqkhWYJNVUOoQoSSqPu9FLkkqUuIxeklQod+KQJGn4WYFJUg0lkA4hSpKKk1n8EKIBJkk1VXoF5hyYJKlIVmCSVFeFDyFGjqAL2SLifuDP7e5H4aYBK9rdCQn/Lu4qf5WZe+/qL42In9D4fzRUKzLzhF3Vn6EYUQGmnRcR8zJzbrv7Ifl3Ua3mHJgkqUgGmCSpSAZY57mw3R2QKv5dVEs5ByZJKpIVmCSpSAZYB4mIEyLi1ohYFBHvb3d/VE8RcVFE3BcRN7e7L+psBliHiIgu4DPAicBs4FURMbu9vVJNfQVo6/VBqgcDrHMcBSzKzDsycwvwLeDkNvdJNZSZVwOr2t0PdT4DrHPMABY3PV9StUlSRzLAJElFMsA6x1JgVtPzmVWbJHUkA6xzXA8cGhEHRcQY4FTg8jb3SZJaxgDrEJnZDZwJXAncAlySmQva2yvVUUR8E7gWODwilkTE6e3ukzqTO3FIkopkBSZJKpIBJkkqkgEmSSqSASZJKpIBJkkqkgGmtouInoi4KSJujojvRMSEnfiur0TEy6rHX+pvQ+OIODYinj6Ec9wVEdMG2/6o96zfwXN9OCLO2tE+SnVggGkk2JiZczLz8cAW4C3NL0bEbkP50sz8u8xc2M9bjgV2OMAkjQwGmEaa3wCHVNXRbyLicmBhRHRFxMci4vqImB8RbwaIhk9X90H7ObBP3xdFxK8iYm71+ISIuDEi/hARV0XEgTSC8t1V9fesiNg7Ii6tznF9RDyj+uxeEfHTiFgQEV8CYqAfIiK+HxE3VJ8541GvXVC1XxURe1dtB0fET6rP/CYijtglf5pSBxvSb7ZSK1SV1onAT6qmJwOPz8w7qxBYk5lPiYixwH9HxE+BJwGH07gH2r7AQuCiR33v3sAXgWOq75qamasi4vPA+sw8r3rfN4ALMvOaiDiAxq4mjwU+BFyTmedExN8Ag9lZ4o3VOcYD10fEpZm5EpgIzMvMd0fEB6vvPhO4EHhLZt4WEU8FPgscN4Q/Rqk2DDCNBOMj4qbq8W+AL9MY2rsuM++s2l8APLFvfguYAhwKHAN8MzN7gHsi4hfb+P6jgav7viszt3evqucBsyMeKrAmR8Tu1TleWn32RxHxwCB+pndExEuqx7Oqvq4EeoFvV+1fA75bnePpwHeazj12EOeQas0A00iwMTPnNDdU/5A/2NwEvD0zr3zU+07ahf0YBRydmZu20ZdBi4hjaYTh0zJzQ0T8Chi3nbdndd7Vj/4zkNQ/58BUiiuBt0bEaICIOCwiJgJXA6+s5simA8/Zxmd/CxwTEQdVn51ata8DJjW976fA2/ueRMSc6uHVwKurthOBPQfo6xTggSq8jqBRAfYZBfRVka+mMTS5FrgzIl5enSMi4sgBziHVngGmUnyJxvzWjRFxM/AFGiMI3wNuq177Ko1d0B8hM+8HzqAxXPcHHh7C+wHwkr5FHMA7gLnVIpGFPLwa8iM0AnABjaHEuwfo60+A3SLiFuBcGgHa50HgqOpnOA44p2p/DXB61b8FwMmD+DORas3d6CVJRbICkyQVyQCTJBXJAJMkFckAkyQVyQCTJBXJAJMkFckAkyQVyQCTJBXp/wMuIKtervoM8gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get Confusion Matrix (CM), Get Additional CM Data, View CM...\n",
    "# Learn More: https://towardsdatascience.com/confusion-matrix-what-is-it-e859e1bbecdc\n",
    "#             https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "cm = confusion_matrix(y_true, y_pred)\n",
    "\n",
    "# Print Confusion Matrix data...\n",
    "cmData = AQbyWeather.getConfusionMatrixData(cm)\n",
    "print(cmData.TN_Output)\n",
    "print(cmData.TP_Output)\n",
    "print(cmData.FN_Output)\n",
    "print(cmData.FP_Output)\n",
    "\n",
    "# Plot Confusion Matrix...\n",
    "cmd = ConfusionMatrixDisplay(confusion_matrix=cm)\n",
    "fig, ax = plt.subplots(figsize=(8, 6))\n",
    "cmd.plot(ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "152d9203-6987-4fbb-a33e-acef22cfb993",
   "metadata": {},
   "source": [
    "## SUMMARY...\n",
    "This demo showed how to use a SageMaker Studio Lab Jupyter Notebook to connect to open-source Amazon Sustainability Data Initiative (ASDI) datasets both via Amazon S3 (for NOAA weather data) and via HTTPS API (for OpenAQ air quality averages). The NOAA weather data and OpenAQ measurement averages are merged into a single table with irrelevant features dropped. This table is then used to build an Autogluon [TabularPredictor](https://auto.gluon.ai/stable/tutorials/tabular_prediction/index.html) ML model to predict a binary classification { 0=OKAY, 1=Unhealthy } based on weather features.\n",
    "\n",
    "### Key Learnings\n",
    "- The relationship between weather and air quality varies geographically and this was confirmed.\n",
    "  - Feature Importance review shows differing factors in models built for different locations.\n",
    "  - Academic research indicates the relationship can change by season and an engineered “MONTH” feature was important for most models.\n",
    "  - An engineered “DAYOFWEEK” feature was expected to be important, but observed correlation was minimal and removing the feature improved most models.\n",
    "- Predicting a target measurement was infeasible, but predicting a binary “OKAY” vs “unhealthy” classification based on threshold values yielded decent results (~75-90% accuracy metrics).\n",
    "- For more on production-level air quality predictions, research how climatology and deeper statistical analysis are merged into 3D models that incorporate emissions models, meteorological models, and chemical models.\n",
    "\n",
    "### Confusion Matrix and Saving Final Results\n",
    "The Confusion Matrix above plots the relationships between:\n",
    "- True Positives (1=>1; see bottom-right)\n",
    "- True Negatives (0=>0; see top-left)\n",
    "- False Positives (1=>0; see top-right)\n",
    "- False Negatives (0=>1; see bottom-left)\n",
    "\n",
    "_The validation dataframe with predictions appended as the final column are saved below as a CSV for additional analysis._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c1c7d5e7-0d51-4170-b1dd-ba09b97632bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scenario: bakersfield_pm25 => Particulate Matter < 2.5 micrometers (pm25) with UnhealthyThreshold > 12.0 µg/m³\n",
      "Results saved to dataRESULTS_bakersfield_pm25_2016-2022.csv. DONE.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DEWP</th>\n",
       "      <th>WDSP</th>\n",
       "      <th>MAX</th>\n",
       "      <th>MIN</th>\n",
       "      <th>PRCP</th>\n",
       "      <th>MONTH</th>\n",
       "      <th>isUnhealthy</th>\n",
       "      <th>PREDICTION</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>41.9</td>\n",
       "      <td>4.7</td>\n",
       "      <td>89.1</td>\n",
       "      <td>55.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>994</th>\n",
       "      <td>40.8</td>\n",
       "      <td>2.5</td>\n",
       "      <td>62.1</td>\n",
       "      <td>37.9</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1820</th>\n",
       "      <td>42.2</td>\n",
       "      <td>5.5</td>\n",
       "      <td>90.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1442</th>\n",
       "      <td>50.0</td>\n",
       "      <td>4.3</td>\n",
       "      <td>73.9</td>\n",
       "      <td>52.0</td>\n",
       "      <td>0.06</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>596</th>\n",
       "      <td>39.7</td>\n",
       "      <td>2.7</td>\n",
       "      <td>66.0</td>\n",
       "      <td>39.9</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>849</th>\n",
       "      <td>53.0</td>\n",
       "      <td>5.5</td>\n",
       "      <td>93.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>208</th>\n",
       "      <td>37.2</td>\n",
       "      <td>5.6</td>\n",
       "      <td>91.9</td>\n",
       "      <td>57.9</td>\n",
       "      <td>0.00</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>288</th>\n",
       "      <td>37.1</td>\n",
       "      <td>2.8</td>\n",
       "      <td>66.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1226</th>\n",
       "      <td>39.3</td>\n",
       "      <td>6.8</td>\n",
       "      <td>99.0</td>\n",
       "      <td>62.1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2129</th>\n",
       "      <td>42.3</td>\n",
       "      <td>5.9</td>\n",
       "      <td>73.9</td>\n",
       "      <td>42.1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>721 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      DEWP  WDSP   MAX   MIN  PRCP  MONTH  isUnhealthy  PREDICTION\n",
       "241   41.9   4.7  89.1  55.0  0.00     11            1           1\n",
       "994   40.8   2.5  62.1  37.9  0.00      1            1           1\n",
       "1820  42.2   5.5  90.0  63.0  0.00      5            0           0\n",
       "1442  50.0   4.3  73.9  52.0  0.06      4            0           0\n",
       "596   39.7   2.7  66.0  39.9  0.00     12            1           1\n",
       "...    ...   ...   ...   ...   ...    ...          ...         ...\n",
       "849   53.0   5.5  93.0  64.0  0.00      8            1           0\n",
       "208   37.2   5.6  91.9  57.9  0.00     10            1           1\n",
       "288   37.1   2.8  66.0  36.0  0.00     12            1           1\n",
       "1226  39.3   6.8  99.0  62.1  0.00      9            0           0\n",
       "2129  42.3   5.9  73.9  42.1  0.00      4            0           0\n",
       "\n",
       "[721 rows x 8 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create and save final results...\n",
    "print(AQbyWeather.selectedScenario.getSummary())\n",
    "resultsFile = AQbyWeather.getFilenameOther(\"dataRESULTS\")\n",
    "results_df = pd.DataFrame()\n",
    "results_df['PREDICTION'] = pd.DataFrame(y_pred)\n",
    "results_df = pd.concat([validate_df, results_df], axis=1)\n",
    "results_df.to_csv(resultsFile, index=False)\n",
    "print(f\"Results saved to {resultsFile}. DONE.\")\n",
    "display(results_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2da4da59-1326-4b15-95e4-3f86a877cfed",
   "metadata": {},
   "source": [
    "## Thanks and Clean-up...\n",
    "**Thanks for your time! I hope this demo was insightful...**\n",
    "\n",
    "If you'd like to clean up locally generated files, change _isCleanupTime_ to **True** below and run the cell.\n",
    "\n",
    "The End."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c1e368f6-c54d-4a6c-bf9d-9a6667628761",
   "metadata": {},
   "outputs": [],
   "source": [
    "# OPTIONAL: Clean Up Generated Files...\n",
    "isCleanupTime = False\n",
    "if(isCleanupTime):\n",
    "    # Remove local Autogluon Models + Folder...\n",
    "    shutil.rmtree(AQbyWeather.selectedScenario.modelFolder)\n",
    "    \n",
    "    # Remove local CSV data files...\n",
    "    for filename in glob.glob(\"*.csv\"):\n",
    "        os.remove(filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d6db503-8406-4cb7-b49c-ada5b8918780",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "aq_by_weather:Python",
   "language": "python",
   "name": "conda-env-aq_by_weather-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}