AlphabetSoupCharity_Optimzation4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7f014505-3aa1-4511-bb78-3313577024f5",
   "metadata": {},
   "source": [
    "## Dependancies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "acba0fd8-5039-4e6a-b728-e2999a8f095f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import pandas as pd\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b4d0186-ca03-46cc-9d1a-2a251e389f82",
   "metadata": {},
   "source": [
    "- Additional Optimization the Model "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "044eb570-c465-419f-b8fc-f46371fbe874",
   "metadata": {},
   "source": [
    "##### Methods to optimize model:\n",
    "\n",
    "- Adjust the input data to ensure that no variables or outliers are causing confusion in the model, such as:\n",
    "    - Dropping more or fewer columns.\n",
    "    - Creating more bins for rare occurrences in columns.\n",
    "    - Increasing or decreasing the number of values for each bin.\n",
    "- Add more neurons to a hidden layer.\n",
    "- Add more hidden layers.\n",
    "- Use different activation functions for the hidden layers.\n",
    "- Add or reduce the number of epochs to the training regimen."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a56364c-ad5a-4d83-9450-9b5dfbd87abc",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "de38ba2d-bd25-4721-aa53-532ccae99ba2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EIN</th>\n",
       "      <th>NAME</th>\n",
       "      <th>APPLICATION_TYPE</th>\n",
       "      <th>AFFILIATION</th>\n",
       "      <th>CLASSIFICATION</th>\n",
       "      <th>USE_CASE</th>\n",
       "      <th>ORGANIZATION</th>\n",
       "      <th>STATUS</th>\n",
       "      <th>INCOME_AMT</th>\n",
       "      <th>SPECIAL_CONSIDERATIONS</th>\n",
       "      <th>ASK_AMT</th>\n",
       "      <th>IS_SUCCESSFUL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10520599</td>\n",
       "      <td>BLUE KNIGHTS MOTORCYCLE CLUB</td>\n",
       "      <td>T10</td>\n",
       "      <td>Independent</td>\n",
       "      <td>C1000</td>\n",
       "      <td>ProductDev</td>\n",
       "      <td>Association</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>5000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10531628</td>\n",
       "      <td>AMERICAN CHESAPEAKE CLUB CHARITABLE TR</td>\n",
       "      <td>T3</td>\n",
       "      <td>Independent</td>\n",
       "      <td>C2000</td>\n",
       "      <td>Preservation</td>\n",
       "      <td>Co-operative</td>\n",
       "      <td>1</td>\n",
       "      <td>1-9999</td>\n",
       "      <td>N</td>\n",
       "      <td>108590</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10547893</td>\n",
       "      <td>ST CLOUD PROFESSIONAL FIREFIGHTERS</td>\n",
       "      <td>T5</td>\n",
       "      <td>CompanySponsored</td>\n",
       "      <td>C3000</td>\n",
       "      <td>ProductDev</td>\n",
       "      <td>Association</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10553066</td>\n",
       "      <td>SOUTHSIDE ATHLETIC ASSOCIATION</td>\n",
       "      <td>T3</td>\n",
       "      <td>CompanySponsored</td>\n",
       "      <td>C2000</td>\n",
       "      <td>Preservation</td>\n",
       "      <td>Trust</td>\n",
       "      <td>1</td>\n",
       "      <td>10000-24999</td>\n",
       "      <td>N</td>\n",
       "      <td>6692</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10556103</td>\n",
       "      <td>GENETIC RESEARCH INSTITUTE OF THE DESERT</td>\n",
       "      <td>T3</td>\n",
       "      <td>Independent</td>\n",
       "      <td>C1000</td>\n",
       "      <td>Heathcare</td>\n",
       "      <td>Trust</td>\n",
       "      <td>1</td>\n",
       "      <td>100000-499999</td>\n",
       "      <td>N</td>\n",
       "      <td>142590</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        EIN                                      NAME APPLICATION_TYPE  \\\n",
       "0  10520599              BLUE KNIGHTS MOTORCYCLE CLUB              T10   \n",
       "1  10531628    AMERICAN CHESAPEAKE CLUB CHARITABLE TR               T3   \n",
       "2  10547893        ST CLOUD PROFESSIONAL FIREFIGHTERS               T5   \n",
       "3  10553066            SOUTHSIDE ATHLETIC ASSOCIATION               T3   \n",
       "4  10556103  GENETIC RESEARCH INSTITUTE OF THE DESERT               T3   \n",
       "\n",
       "        AFFILIATION CLASSIFICATION      USE_CASE  ORGANIZATION  STATUS  \\\n",
       "0       Independent          C1000    ProductDev   Association       1   \n",
       "1       Independent          C2000  Preservation  Co-operative       1   \n",
       "2  CompanySponsored          C3000    ProductDev   Association       1   \n",
       "3  CompanySponsored          C2000  Preservation         Trust       1   \n",
       "4       Independent          C1000     Heathcare         Trust       1   \n",
       "\n",
       "      INCOME_AMT SPECIAL_CONSIDERATIONS  ASK_AMT  IS_SUCCESSFUL  \n",
       "0              0                      N     5000              1  \n",
       "1         1-9999                      N   108590              1  \n",
       "2              0                      N     5000              0  \n",
       "3    10000-24999                      N     6692              1  \n",
       "4  100000-499999                      N   142590              1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import and read the charity_data.csv.\n",
    "import pandas as pd \n",
    "application_df = pd.read_csv(\"resources/charity_data.csv\")\n",
    "application_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "88b2b259-3776-473b-95b1-aac1a3245f9c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EIN                       0\n",
       "NAME                      0\n",
       "APPLICATION_TYPE          0\n",
       "AFFILIATION               0\n",
       "CLASSIFICATION            0\n",
       "USE_CASE                  0\n",
       "ORGANIZATION              0\n",
       "STATUS                    0\n",
       "INCOME_AMT                0\n",
       "SPECIAL_CONSIDERATIONS    0\n",
       "ASK_AMT                   0\n",
       "IS_SUCCESSFUL             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for null values\n",
    "application_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5537d34f-62cf-4797-a166-3793c2c88b92",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the non-beneficial ID columns 'EIN'\n",
    "application_df.drop([\"EIN\"], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7a6b2a79-d5e9-4eae-8ba3-f291dd2f4d32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NAME</th>\n",
       "      <th>APPLICATION_TYPE</th>\n",
       "      <th>AFFILIATION</th>\n",
       "      <th>CLASSIFICATION</th>\n",
       "      <th>USE_CASE</th>\n",
       "      <th>ORGANIZATION</th>\n",
       "      <th>STATUS</th>\n",
       "      <th>INCOME_AMT</th>\n",
       "      <th>SPECIAL_CONSIDERATIONS</th>\n",
       "      <th>ASK_AMT</th>\n",
       "      <th>IS_SUCCESSFUL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BLUE KNIGHTS MOTORCYCLE CLUB</td>\n",
       "      <td>T10</td>\n",
       "      <td>Independent</td>\n",
       "      <td>C1000</td>\n",
       "      <td>ProductDev</td>\n",
       "      <td>Association</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>5000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AMERICAN CHESAPEAKE CLUB CHARITABLE TR</td>\n",
       "      <td>T3</td>\n",
       "      <td>Independent</td>\n",
       "      <td>C2000</td>\n",
       "      <td>Preservation</td>\n",
       "      <td>Co-operative</td>\n",
       "      <td>1</td>\n",
       "      <td>1-9999</td>\n",
       "      <td>N</td>\n",
       "      <td>108590</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ST CLOUD PROFESSIONAL FIREFIGHTERS</td>\n",
       "      <td>T5</td>\n",
       "      <td>CompanySponsored</td>\n",
       "      <td>C3000</td>\n",
       "      <td>ProductDev</td>\n",
       "      <td>Association</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>SOUTHSIDE ATHLETIC ASSOCIATION</td>\n",
       "      <td>T3</td>\n",
       "      <td>CompanySponsored</td>\n",
       "      <td>C2000</td>\n",
       "      <td>Preservation</td>\n",
       "      <td>Trust</td>\n",
       "      <td>1</td>\n",
       "      <td>10000-24999</td>\n",
       "      <td>N</td>\n",
       "      <td>6692</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GENETIC RESEARCH INSTITUTE OF THE DESERT</td>\n",
       "      <td>T3</td>\n",
       "      <td>Independent</td>\n",
       "      <td>C1000</td>\n",
       "      <td>Heathcare</td>\n",
       "      <td>Trust</td>\n",
       "      <td>1</td>\n",
       "      <td>100000-499999</td>\n",
       "      <td>N</td>\n",
       "      <td>142590</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       NAME APPLICATION_TYPE  \\\n",
       "0              BLUE KNIGHTS MOTORCYCLE CLUB              T10   \n",
       "1    AMERICAN CHESAPEAKE CLUB CHARITABLE TR               T3   \n",
       "2        ST CLOUD PROFESSIONAL FIREFIGHTERS               T5   \n",
       "3            SOUTHSIDE ATHLETIC ASSOCIATION               T3   \n",
       "4  GENETIC RESEARCH INSTITUTE OF THE DESERT               T3   \n",
       "\n",
       "        AFFILIATION CLASSIFICATION      USE_CASE  ORGANIZATION  STATUS  \\\n",
       "0       Independent          C1000    ProductDev   Association       1   \n",
       "1       Independent          C2000  Preservation  Co-operative       1   \n",
       "2  CompanySponsored          C3000    ProductDev   Association       1   \n",
       "3  CompanySponsored          C2000  Preservation         Trust       1   \n",
       "4       Independent          C1000     Heathcare         Trust       1   \n",
       "\n",
       "      INCOME_AMT SPECIAL_CONSIDERATIONS  ASK_AMT  IS_SUCCESSFUL  \n",
       "0              0                      N     5000              1  \n",
       "1         1-9999                      N   108590              1  \n",
       "2              0                      N     5000              0  \n",
       "3    10000-24999                      N     6692              1  \n",
       "4  100000-499999                      N   142590              1  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "application_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6fcaec9f-ae41-4221-972c-4f34b3068a21",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NAME                      19568\n",
       "APPLICATION_TYPE             17\n",
       "AFFILIATION                   6\n",
       "CLASSIFICATION               71\n",
       "USE_CASE                      5\n",
       "ORGANIZATION                  4\n",
       "STATUS                        2\n",
       "INCOME_AMT                    9\n",
       "SPECIAL_CONSIDERATIONS        2\n",
       "ASK_AMT                    8747\n",
       "IS_SUCCESSFUL                 2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Determine the number of unique values in each column.\n",
    "application_df.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ea975c27-20f3-4ac6-9186-2ed981de933c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "T3     27037\n",
       "T4      1542\n",
       "T6      1216\n",
       "T5      1173\n",
       "T19     1065\n",
       "T8       737\n",
       "T7       725\n",
       "T10      528\n",
       "T9       156\n",
       "T13       66\n",
       "T12       27\n",
       "T2        16\n",
       "T25        3\n",
       "T14        3\n",
       "T29        2\n",
       "T15        2\n",
       "T17        1\n",
       "Name: APPLICATION_TYPE, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Look at APPLICATION_TYPE value counts for binning\n",
    "application_count = application_df[\"APPLICATION_TYPE\"].value_counts()\n",
    "application_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2d095404-ca57-495e-a31a-5cc74a22e109",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['T8',\n",
       " 'T7',\n",
       " 'T10',\n",
       " 'T9',\n",
       " 'T13',\n",
       " 'T12',\n",
       " 'T2',\n",
       " 'T25',\n",
       " 'T14',\n",
       " 'T29',\n",
       " 'T15',\n",
       " 'T17']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(application_count.loc[application_count < 1000].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6c10dd9c-4e40-4ebb-9263-263192c96c0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "T3       27037\n",
       "Other     2266\n",
       "T4        1542\n",
       "T6        1216\n",
       "T5        1173\n",
       "T19       1065\n",
       "Name: APPLICATION_TYPE, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Choose a cutoff value and create a list of application types to be replaced\n",
    "# use the variable name `application_types_to_replace`\n",
    "application_types_to_replace = list(application_count.loc[application_count < 1000].index)\n",
    "\n",
    "# Replace in dataframe\n",
    "for app in application_types_to_replace:\n",
    "    application_df['APPLICATION_TYPE'] = application_df['APPLICATION_TYPE'].replace(app,\"Other\")\n",
    "\n",
    "# Check to make sure binning was successful\n",
    "application_df['APPLICATION_TYPE'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6d47acfa-a587-4edb-8464-823381fcfa49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "C1000    17326\n",
       "C2000     6074\n",
       "C1200     4837\n",
       "C3000     1918\n",
       "C2100     1883\n",
       "         ...  \n",
       "C4120        1\n",
       "C8210        1\n",
       "C2561        1\n",
       "C4500        1\n",
       "C2150        1\n",
       "Name: CLASSIFICATION, Length: 71, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Look at CLASSIFICATION value counts for binning\n",
    "classification_count = application_df['CLASSIFICATION'].value_counts()\n",
    "classification_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9129203f-b7f8-4213-a483-75589e6bec09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "C1000    17326\n",
       "C2000     6074\n",
       "C1200     4837\n",
       "C3000     1918\n",
       "C2100     1883\n",
       "C7000      777\n",
       "C1700      287\n",
       "C4000      194\n",
       "C5000      116\n",
       "C1270      114\n",
       "C2700      104\n",
       "C2800       95\n",
       "C7100       75\n",
       "C1300       58\n",
       "C1280       50\n",
       "C1230       36\n",
       "C1400       34\n",
       "C7200       32\n",
       "C2300       32\n",
       "C1240       30\n",
       "C8000       20\n",
       "C7120       18\n",
       "C1500       16\n",
       "C1800       15\n",
       "C6000       15\n",
       "C1250       14\n",
       "C8200       11\n",
       "C1238       10\n",
       "C1278       10\n",
       "C1235        9\n",
       "C1237        9\n",
       "C7210        7\n",
       "C2400        6\n",
       "C1720        6\n",
       "C4100        6\n",
       "C1257        5\n",
       "C1600        5\n",
       "C1260        3\n",
       "C2710        3\n",
       "C0           3\n",
       "C3200        2\n",
       "C1234        2\n",
       "C1246        2\n",
       "C1267        2\n",
       "C1256        2\n",
       "Name: CLASSIFICATION, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You may find it helpful to look at CLASSIFICATION value counts >1\n",
    "classification_count[classification_count >1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4ad1751c-3b01-4f2e-ac3b-8a7eb8c89317",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['C7000',\n",
       " 'C1700',\n",
       " 'C4000',\n",
       " 'C5000',\n",
       " 'C1270',\n",
       " 'C2700',\n",
       " 'C2800',\n",
       " 'C7100',\n",
       " 'C1300',\n",
       " 'C1280',\n",
       " 'C1230',\n",
       " 'C1400',\n",
       " 'C7200',\n",
       " 'C2300',\n",
       " 'C1240',\n",
       " 'C8000',\n",
       " 'C7120',\n",
       " 'C1500',\n",
       " 'C1800',\n",
       " 'C6000',\n",
       " 'C1250',\n",
       " 'C8200',\n",
       " 'C1238',\n",
       " 'C1278',\n",
       " 'C1235',\n",
       " 'C1237',\n",
       " 'C7210',\n",
       " 'C2400',\n",
       " 'C1720',\n",
       " 'C4100',\n",
       " 'C1257',\n",
       " 'C1600',\n",
       " 'C1260',\n",
       " 'C2710',\n",
       " 'C0',\n",
       " 'C3200',\n",
       " 'C1234',\n",
       " 'C1246',\n",
       " 'C1267',\n",
       " 'C1256',\n",
       " 'C2190',\n",
       " 'C4200',\n",
       " 'C2600',\n",
       " 'C5200',\n",
       " 'C1370',\n",
       " 'C1248',\n",
       " 'C6100',\n",
       " 'C1820',\n",
       " 'C1900',\n",
       " 'C1236',\n",
       " 'C3700',\n",
       " 'C2570',\n",
       " 'C1580',\n",
       " 'C1245',\n",
       " 'C2500',\n",
       " 'C1570',\n",
       " 'C1283',\n",
       " 'C2380',\n",
       " 'C1732',\n",
       " 'C1728',\n",
       " 'C2170',\n",
       " 'C4120',\n",
       " 'C8210',\n",
       " 'C2561',\n",
       " 'C4500',\n",
       " 'C2150']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(classification_count.loc[classification_count < 1000].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "5bcf9070-04d6-4b38-ab02-fe8df7b9dc9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "C1000    17326\n",
       "C2000     6074\n",
       "C1200     4837\n",
       "Other     2261\n",
       "C3000     1918\n",
       "C2100     1883\n",
       "Name: CLASSIFICATION, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Choose a cutoff value and create a list of classifications to be replaced\n",
    "# use the variable name `classifications_to_replace`\n",
    "classifications_to_replace = list(classification_count.loc[classification_count < 1000].index)\n",
    "\n",
    "# Replace in dataframe\n",
    "for cls in classifications_to_replace:\n",
    "    application_df['CLASSIFICATION'] = application_df['CLASSIFICATION'].replace(cls,\"Other\")\n",
    "    \n",
    "# Check to make sure binning was successful\n",
    "application_df['CLASSIFICATION'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e51dd51d-f1f7-4916-bd18-8e694adb32ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert categorical data to numeric with `pd.get_dummies`\n",
    "application_dummies = pd.get_dummies(application_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e6458374-a657-485e-be6f-120dbffea8bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>STATUS</th>\n",
       "      <th>ASK_AMT</th>\n",
       "      <th>IS_SUCCESSFUL</th>\n",
       "      <th>NAME_1 DAY RANCH RESCUE AND RURAL OKLAHOMA ANIMAL RESOURCE INC</th>\n",
       "      <th>NAME_100 BLACK MEN OF AMERICA</th>\n",
       "      <th>NAME_100 BLACK MEN OF MEMPHIS INC</th>\n",
       "      <th>NAME_100 BLACK MEN OF WEST GEORGIA INC</th>\n",
       "      <th>NAME_1150 WEBSTER STREET INC</th>\n",
       "      <th>NAME_116TH CAVALRY REGIMENT CHAPTER OF THE US CAVALRY &amp; ARMOR ASSOCIATION</th>\n",
       "      <th>NAME_13TH BOMB SQUADRON ASSOCIATION</th>\n",
       "      <th>...</th>\n",
       "      <th>INCOME_AMT_1-9999</th>\n",
       "      <th>INCOME_AMT_10000-24999</th>\n",
       "      <th>INCOME_AMT_100000-499999</th>\n",
       "      <th>INCOME_AMT_10M-50M</th>\n",
       "      <th>INCOME_AMT_1M-5M</th>\n",
       "      <th>INCOME_AMT_25000-99999</th>\n",
       "      <th>INCOME_AMT_50M+</th>\n",
       "      <th>INCOME_AMT_5M-10M</th>\n",
       "      <th>SPECIAL_CONSIDERATIONS_N</th>\n",
       "      <th>SPECIAL_CONSIDERATIONS_Y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>108590</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>6692</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>142590</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 19609 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   STATUS  ASK_AMT  IS_SUCCESSFUL  \\\n",
       "0       1     5000              1   \n",
       "1       1   108590              1   \n",
       "2       1     5000              0   \n",
       "3       1     6692              1   \n",
       "4       1   142590              1   \n",
       "\n",
       "   NAME_1 DAY RANCH RESCUE AND RURAL OKLAHOMA ANIMAL RESOURCE INC  \\\n",
       "0                                                  0                \n",
       "1                                                  0                \n",
       "2                                                  0                \n",
       "3                                                  0                \n",
       "4                                                  0                \n",
       "\n",
       "   NAME_100 BLACK MEN OF AMERICA  NAME_100 BLACK MEN OF MEMPHIS INC  \\\n",
       "0                              0                                  0   \n",
       "1                              0                                  0   \n",
       "2                              0                                  0   \n",
       "3                              0                                  0   \n",
       "4                              0                                  0   \n",
       "\n",
       "   NAME_100 BLACK MEN OF WEST GEORGIA INC  NAME_1150 WEBSTER STREET INC  \\\n",
       "0                                       0                             0   \n",
       "1                                       0                             0   \n",
       "2                                       0                             0   \n",
       "3                                       0                             0   \n",
       "4                                       0                             0   \n",
       "\n",
       "   NAME_116TH CAVALRY REGIMENT CHAPTER OF THE US CAVALRY & ARMOR ASSOCIATION  \\\n",
       "0                                                  0                           \n",
       "1                                                  0                           \n",
       "2                                                  0                           \n",
       "3                                                  0                           \n",
       "4                                                  0                           \n",
       "\n",
       "   NAME_13TH BOMB SQUADRON ASSOCIATION  ...  INCOME_AMT_1-9999  \\\n",
       "0                                    0  ...                  0   \n",
       "1                                    0  ...                  1   \n",
       "2                                    0  ...                  0   \n",
       "3                                    0  ...                  0   \n",
       "4                                    0  ...                  0   \n",
       "\n",
       "   INCOME_AMT_10000-24999  INCOME_AMT_100000-499999  INCOME_AMT_10M-50M  \\\n",
       "0                       0                         0                   0   \n",
       "1                       0                         0                   0   \n",
       "2                       0                         0                   0   \n",
       "3                       1                         0                   0   \n",
       "4                       0                         1                   0   \n",
       "\n",
       "   INCOME_AMT_1M-5M  INCOME_AMT_25000-99999  INCOME_AMT_50M+  \\\n",
       "0                 0                       0                0   \n",
       "1                 0                       0                0   \n",
       "2                 0                       0                0   \n",
       "3                 0                       0                0   \n",
       "4                 0                       0                0   \n",
       "\n",
       "   INCOME_AMT_5M-10M  SPECIAL_CONSIDERATIONS_N  SPECIAL_CONSIDERATIONS_Y  \n",
       "0                  0                         1                         0  \n",
       "1                  0                         1                         0  \n",
       "2                  0                         1                         0  \n",
       "3                  0                         1                         0  \n",
       "4                  0                         1                         0  \n",
       "\n",
       "[5 rows x 19609 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preview numeric dataframe\n",
    "application_dummies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2b3c53ff-e1cf-4979-a261-17bcdf89bfc0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Split our preprocessed data into our features and target arrays\n",
    "X = application_dummies.drop('IS_SUCCESSFUL', axis=1).values\n",
    "y = application_dummies['IS_SUCCESSFUL'].values\n",
    "\n",
    "# Split the preprocessed data into a training and testing dataset\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 42, stratify=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9f5ba0e1-2067-408c-9e7f-a31f12d6a296",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a StandardScaler instances\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Fit the StandardScaler\n",
    "X_scaler = scaler.fit(X_train)\n",
    "\n",
    "# Scale the data\n",
    "X_train_scaled = X_scaler.transform(X_train)\n",
    "X_test_scaled = X_scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5fa421d1-72fd-4152-8933-01f28195f860",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25724, 19608)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check X_train_scaled shape\n",
    "X_train_scaled.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "fbe6b0cd-e7a5-4d28-94df-25489f587530",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8575, 19608)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check X_test_scaled shape\n",
    "X_test_scaled.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0d2722d9-b222-489e-ba9e-0466664e7473",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19608"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check feature input for model\n",
    "feat_input=X.shape[1]\n",
    "feat_input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4b796ce-3a49-46ba-be1e-8a0d94dff097",
   "metadata": {},
   "source": [
    "## Compile, Train and Evaluate the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c22aa05e-9714-4f02-a502-1b7d87e7d5cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a method that creates a new Sequential model with hyperparameter options\n",
    "def create_model(hp):\n",
    "    nn_model = tf.keras.models.Sequential()\n",
    "\n",
    "    # Allow kerastuner to decide which activation function to use in hidden layers\n",
    "    activation = hp.Choice('activation',['relu','tanh'])\n",
    "    \n",
    "    # Allow kerastuner to decide number of neurons in first layer\n",
    "    nn_model.add(tf.keras.layers.Dense(units=hp.Int('first_units',\n",
    "        min_value=1,\n",
    "        max_value=30,\n",
    "        step=5), activation=activation, input_dim=feat_input))\n",
    "\n",
    "    # Allow kerastuner to decide number of hidden layers and neurons in hidden layers\n",
    "    for i in range(hp.Int('num_layers', 1, 5)):\n",
    "        nn_model.add(tf.keras.layers.Dense(units=hp.Int('units_' + str(i),\n",
    "            min_value=1,\n",
    "            max_value=30,\n",
    "            step=5),\n",
    "            activation=activation))\n",
    "    \n",
    "    nn_model.add(tf.keras.layers.Dense(units=1, activation=\"sigmoid\"))\n",
    "\n",
    "    # Compile the model\n",
    "    nn_model.compile(loss=\"binary_crossentropy\", optimizer='adam', metrics=[\"accuracy\"])\n",
    "    \n",
    "    return nn_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6439088a-b3bf-4f6c-a3c2-1665c741066a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reloading Oracle from existing project ./untitled_project/oracle.json\n",
      "INFO:tensorflow:Reloading Tuner from ./untitled_project/tuner0.json\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-08-09 13:46:44.346333: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "# Import the kerastuner library\n",
    "import keras_tuner as kt\n",
    "\n",
    "tuner = kt.Hyperband(\n",
    "    create_model,\n",
    "    objective=\"val_accuracy\",\n",
    "    max_epochs=20,\n",
    "    hyperband_iterations=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a2287f89-037f-4c4f-bb79-aaf7056e98af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trial 50 Complete [00h 00m 44s]\n",
      "val_accuracy: 0.7723615169525146\n",
      "\n",
      "Best val_accuracy So Far: 0.7940524816513062\n",
      "Total elapsed time: 00h 21m 33s\n",
      "\n",
      "Search: Running Trial #51\n",
      "\n",
      "Value             |Best Value So Far |Hyperparameter\n",
      "tanh              |tanh              |activation\n",
      "16                |26                |first_units\n",
      "3                 |1                 |num_layers\n",
      "11                |11                |units_0\n",
      "66                |6                 |units_1\n",
      "51                |26                |units_2\n",
      "21                |56                |units_3\n",
      "1                 |96                |units_4\n",
      "7                 |7                 |tuner/epochs\n",
      "0                 |3                 |tuner/initial_epoch\n",
      "1                 |2                 |tuner/bracket\n",
      "0                 |1                 |tuner/round\n",
      "\n",
      "Epoch 1/7\n",
      "804/804 [==============================] - 12s 13ms/step - loss: 0.5020 - accuracy: 0.7566 - val_loss: 0.4699 - val_accuracy: 0.7887\n",
      "Epoch 2/7\n",
      "804/804 [==============================] - 7s 8ms/step - loss: 0.1421 - accuracy: 0.9507 - val_loss: 0.5077 - val_accuracy: 0.7266\n",
      "Epoch 3/7\n",
      "804/804 [==============================] - 4s 5ms/step - loss: 0.1002 - accuracy: 0.9617 - val_loss: 0.7033 - val_accuracy: 0.6132\n",
      "Epoch 4/7\n",
      "804/804 [==============================] - 4s 5ms/step - loss: 0.0948 - accuracy: 0.9627 - val_loss: 0.4855 - val_accuracy: 0.7524\n",
      "Epoch 5/7\n",
      "804/804 [==============================] - 5s 6ms/step - loss: 0.0924 - accuracy: 0.9634 - val_loss: 0.5351 - val_accuracy: 0.6918\n",
      "Epoch 6/7\n",
      "660/804 [=======================>......] - ETA: 0s - loss: 0.0919 - accuracy: 0.9632"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/0y/7pmlhq5x2bjbv4z7b2s37nwm0000gn/T/ipykernel_73273/1874305172.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Run the kerastuner search for best hyperparameters\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtuner\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train_scaled\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mvalidation_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test_scaled\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/keras_tuner/engine/base_tuner.py\u001b[0m in \u001b[0;36msearch\u001b[0;34m(self, *fit_args, **fit_kwargs)\u001b[0m\n\u001b[1;32m    181\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    182\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_trial_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrial\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 183\u001b[0;31m             \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_trial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mfit_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    184\u001b[0m             \u001b[0;31m# `results` is None indicates user updated oracle in `run_trial()`.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    185\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresults\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/keras_tuner/tuners/hyperband.py\u001b[0m in \u001b[0;36mrun_trial\u001b[0;34m(self, trial, *fit_args, **fit_kwargs)\u001b[0m\n\u001b[1;32m    382\u001b[0m             \u001b[0mfit_kwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"epochs\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"tuner/epochs\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    383\u001b[0m             \u001b[0mfit_kwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"initial_epoch\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"tuner/initial_epoch\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 384\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mHyperband\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_trial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mfit_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    385\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    386\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_build_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/keras_tuner/engine/tuner.py\u001b[0m in \u001b[0;36mrun_trial\u001b[0;34m(self, trial, *args, **kwargs)\u001b[0m\n\u001b[1;32m    293\u001b[0m             \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_checkpoint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    294\u001b[0m             \u001b[0mcopied_kwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"callbacks\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 295\u001b[0;31m             \u001b[0mobj_value\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_build_and_fit_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mcopied_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    296\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    297\u001b[0m             \u001b[0mhistories\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj_value\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/keras_tuner/engine/tuner.py\u001b[0m in \u001b[0;36m_build_and_fit_model\u001b[0;34m(self, trial, *args, **kwargs)\u001b[0m\n\u001b[1;32m    220\u001b[0m         \u001b[0mhp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrial\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhyperparameters\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    221\u001b[0m         \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_try_build\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 222\u001b[0;31m         \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhypermodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    223\u001b[0m         tuner_utils.validate_trial_results(\n\u001b[1;32m    224\u001b[0m             \u001b[0mresults\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moracle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mobjective\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"HyperModel.fit()\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/keras_tuner/engine/hypermodel.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, hp, model, *args, **kwargs)\u001b[0m\n\u001b[1;32m    138\u001b[0m             \u001b[0mIf\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0ma\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mit\u001b[0m \u001b[0mshould\u001b[0m \u001b[0mbe\u001b[0m \u001b[0mthe\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mobjective\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    139\u001b[0m         \"\"\"\n\u001b[0;32m--> 140\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    141\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    142\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/keras/utils/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     62\u001b[0m     \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     63\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 64\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     65\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# pylint: disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m       \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_process_traceback_frames\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__traceback__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m   1407\u001b[0m                 _r=1):\n\u001b[1;32m   1408\u001b[0m               \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1409\u001b[0;31m               \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1410\u001b[0m               \u001b[0;32mif\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_sync\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1411\u001b[0m                 \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/tensorflow/python/util/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    148\u001b[0m     \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    149\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 150\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    151\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    152\u001b[0m       \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_process_traceback_frames\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__traceback__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    913\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    914\u001b[0m       \u001b[0;32mwith\u001b[0m \u001b[0mOptionalXlaContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jit_compile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 915\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    916\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    917\u001b[0m       \u001b[0mnew_tracing_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexperimental_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    945\u001b[0m       \u001b[0;31m# In this case we have created variables on the first call, so we run the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    946\u001b[0m       \u001b[0;31m# defunned version which is guaranteed to never create variables.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 947\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateless_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# pylint: disable=not-callable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    948\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateful_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    949\u001b[0m       \u001b[0;31m# Release the lock early so that multiple threads can perform the call\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2452\u001b[0m        filtered_flat_args) = self._maybe_define_function(args, kwargs)\n\u001b[1;32m   2453\u001b[0m     return graph_function._call_flat(\n\u001b[0;32m-> 2454\u001b[0;31m         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access\n\u001b[0m\u001b[1;32m   2455\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2456\u001b[0m   \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m   1859\u001b[0m       \u001b[0;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1860\u001b[0m       return self._build_call_outputs(self._inference_function.call(\n\u001b[0;32m-> 1861\u001b[0;31m           ctx, args, cancellation_manager=cancellation_manager))\n\u001b[0m\u001b[1;32m   1862\u001b[0m     forward_backward = self._select_forward_and_backward_functions(\n\u001b[1;32m   1863\u001b[0m         \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m    500\u001b[0m               \u001b[0minputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    501\u001b[0m               \u001b[0mattrs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 502\u001b[0;31m               ctx=ctx)\n\u001b[0m\u001b[1;32m    503\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    504\u001b[0m           outputs = execute.execute_with_cancellation(\n",
      "\u001b[0;32m~/opt/anaconda3/envs/PythonData/lib/python3.7/site-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 55\u001b[0;31m                                         inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m     56\u001b[0m   \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Run the kerastuner search for best hyperparameters\n",
    "tuner.search(X_train_scaled,y_train,epochs=20,validation_data=(X_test_scaled,y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8073b80b-ce08-45a6-aa24-7e4f8f74b138",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 'tanh', 'first_units': 26, 'num_layers': 1, 'units_0': 11, 'units_1': 6, 'units_2': 26, 'units_3': 56, 'units_4': 96, 'tuner/epochs': 7, 'tuner/initial_epoch': 3, 'tuner/bracket': 2, 'tuner/round': 1, 'tuner/trial_id': '0038'}\n",
      "{'activation': 'tanh', 'first_units': 16, 'num_layers': 1, 'units_0': 21, 'units_1': 81, 'units_2': 51, 'units_3': 96, 'units_4': 16, 'tuner/epochs': 20, 'tuner/initial_epoch': 7, 'tuner/bracket': 2, 'tuner/round': 2, 'tuner/trial_id': '0012'}\n",
      "{'activation': 'tanh', 'first_units': 26, 'num_layers': 1, 'units_0': 11, 'units_1': 6, 'units_2': 26, 'units_3': 56, 'units_4': 96, 'tuner/epochs': 3, 'tuner/initial_epoch': 0, 'tuner/bracket': 2, 'tuner/round': 0}\n"
     ]
    }
   ],
   "source": [
    "# Get top 3 model hyperparameters and print the values\n",
    "top_hyper = tuner.get_best_hyperparameters(3)\n",
    "for param in top_hyper:\n",
    "    print(param.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b07f74ab-d744-4c4a-bcd6-8dcd274692f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "268/268 - 1s - loss: 0.5207 - accuracy: 0.7941 - 1s/epoch - 5ms/step\n",
      "Loss: 0.5207111239433289, Accuracy: 0.7940524816513062\n",
      "268/268 - 1s - loss: 0.4722 - accuracy: 0.7939 - 699ms/epoch - 3ms/step\n",
      "Loss: 0.472161203622818, Accuracy: 0.7939358353614807\n",
      "268/268 - 1s - loss: 0.5170 - accuracy: 0.7938 - 700ms/epoch - 3ms/step\n",
      "Loss: 0.5169849991798401, Accuracy: 0.7938192486763\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the top 3 models against the test dataset\n",
    "top_model = tuner.get_best_models(3)\n",
    "for model in top_model:\n",
    "    model_loss, model_accuracy = model.evaluate(X_test_scaled,y_test,verbose=2)\n",
    "    print(f\"Loss: {model_loss}, Accuracy: {model_accuracy}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a4749e72-4a7d-4bd0-b4b9-1fa398cf4940",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'activation': 'tanh',\n",
       " 'first_units': 26,\n",
       " 'num_layers': 1,\n",
       " 'units_0': 11,\n",
       " 'units_1': 6,\n",
       " 'units_2': 26,\n",
       " 'units_3': 56,\n",
       " 'units_4': 96,\n",
       " 'tuner/epochs': 7,\n",
       " 'tuner/initial_epoch': 3,\n",
       " 'tuner/bracket': 2,\n",
       " 'tuner/round': 1,\n",
       " 'tuner/trial_id': '0038'}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get first best model hyperparameters\n",
    "best_hyper = tuner.get_best_hyperparameters(1)[0]\n",
    "best_hyper.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "21675628-9ebc-40cc-8688-60801b9c3c53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'activation': 'tanh',\n",
       " 'first_units': 16,\n",
       " 'num_layers': 1,\n",
       " 'units_0': 21,\n",
       " 'units_1': 81,\n",
       " 'units_2': 51,\n",
       " 'units_3': 96,\n",
       " 'units_4': 16,\n",
       " 'tuner/epochs': 20,\n",
       " 'tuner/initial_epoch': 7,\n",
       " 'tuner/bracket': 2,\n",
       " 'tuner/round': 2,\n",
       " 'tuner/trial_id': '0012'}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get second best model hyperparameters\n",
    "second_hyper = tuner.get_best_hyperparameters(2)[1]\n",
    "second_hyper.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "2dbdf142-d12e-4e40-8b2a-a55911a7f084",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.\n",
      "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.iter\n",
      "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_1\n",
      "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_2\n",
      "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.decay\n",
      "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.learning_rate\n",
      "268/268 - 1s - loss: 0.4722 - accuracy: 0.7939 - 604ms/epoch - 2ms/step\n",
      "Loss: 0.472161203622818, Accuracy: 0.7939358353614807\n"
     ]
    }
   ],
   "source": [
    "# Compare the performance to the second-best model\n",
    "second_model = tuner.get_best_models(2)[1]\n",
    "model_loss, model_accuracy = second_model.evaluate(X_test_scaled,y_test,verbose=2)\n",
    "print(f\"Loss: {model_loss}, Accuracy: {model_accuracy}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38a3b651-9ac7-4f2b-bca9-7636ae689ee7",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "#### 79% Accuracy\n",
    "#### 0.5% Loss"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PythonData",
   "language": "python",
   "name": "pythondata"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}