t81_558_class_04_2_multi_class.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_04_2_multi_class.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 4: Training for Tabular Data**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "* Part 4.1: Encoding a Feature Vector for Keras Deep Learning [[Video]](https://www.youtube.com/watch?v=Vxz-gfs9nMQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_1_feature_encode.ipynb)\n",
    "* **Part 4.2: Keras Multiclass Classification for Deep Neural Networks with ROC and AUC** [[Video]](https://www.youtube.com/watch?v=-f3bg9dLMks&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_2_multi_class.ipynb)\n",
    "* Part 4.3: Keras Regression for Deep Neural Networks with RMSE [[Video]](https://www.youtube.com/watch?v=wNhBUC6X5-E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_3_regression.ipynb)\n",
    "* Part 4.4: Backpropagation, Nesterov Momentum, and ADAM Neural Network Training [[Video]](https://www.youtube.com/watch?v=VbDg8aBgpck&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_4_backprop.ipynb)\n",
    "* Part 4.5: Neural Network RMSE and Log Loss Error Calculation from Scratch [[Video]](https://www.youtube.com/watch?v=wmQX1t2PHJc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_04_5_rmse_logloss.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.2: Multiclass Classification with ROC and AUC\n",
    "\n",
    "* **Binary Classification** - Classification between two possibilities (positive and negative).  Common in medical testing, does the person have the disease (positive) or not (negative).\n",
    "* **Classification** - Classification between more than 2.  The iris dataset (3-way classification).\n",
    "* **Regression** - Numeric prediction.  How many MPG does a car get? (covered in next video)\n",
    "\n",
    "In this class session we will look at some visualizations for all three.\n",
    "\n",
    "\n",
    "It is important to evaluate the level of error in the results produced by a neural network.  In this part we will look at how to evaluate error for both classification and regression neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary Classification and ROC Charts\n",
    "\n",
    "Binary classification occurs when a neural network must choose between two options, which might be true/false, yes/no, correct/incorrect, or buy/sell.  To see how to use binary classification, we will consider a classification system for a credit card company.  This classification system must decide how to respond to a new potential customer.  This system will either \"issue a credit card\" or \"decline a credit card.\"  \n",
    "\n",
    "When you have only two classes that you can consider, the objective function's score is the number of false positive predictions versus the number of false negatives. False negatives and false positives are both types of errors, and it is important to understand the difference. For the previous example, issuing a credit card would be the positive.  A false positive occurs when a credit card is issued to someone who will become a bad credit risk.  A false negative happens when a credit card is declined to someone who would have been a good risk.  \n",
    "\n",
    "Because only two options exist, we can choose the mistake that is the more serious type of error, a false positive or a false negative.  For most banks issuing credit cards, a false positive is worse than a false negative.  Declining a potentially good credit card holder is better than accepting a credit card holder who would cause the bank to undertake expensive collection activities.\n",
    "\n",
    "Consider the following program that uses the [wcbreast_wdbc dataset](https://data.heatonresearch.com/data/t81-558/wcbreast_wdbc.csv) to classify if a breast tumor is cancerous (malignant) or not (benign).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>...</th>\n",
       "      <th>worst_symmetry</th>\n",
       "      <th>worst_fractal_dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>...</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>...</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>567</th>\n",
       "      <td>927241</td>\n",
       "      <td>M</td>\n",
       "      <td>...</td>\n",
       "      <td>0.4087</td>\n",
       "      <td>0.12400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>568</th>\n",
       "      <td>92751</td>\n",
       "      <td>B</td>\n",
       "      <td>...</td>\n",
       "      <td>0.2871</td>\n",
       "      <td>0.07039</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>569 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id diagnosis  ...  worst_symmetry  worst_fractal_dimension\n",
       "0    842302         M  ...          0.4601                  0.11890\n",
       "1    842517         M  ...          0.2750                  0.08902\n",
       "..      ...       ...  ...             ...                      ...\n",
       "567  927241         M  ...          0.4087                  0.12400\n",
       "568   92751         B  ...          0.2871                  0.07039\n",
       "\n",
       "[569 rows x 32 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/wcbreast_wdbc.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "pd.set_option('display.max_columns', 5)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ROC curves can be a bit confusing.  However, they are very common.  It is important to know how to read them.  Even their name is confusing.  Do not worry about their name, it comes from electrical engineering (EE).\n",
    "\n",
    "Binary classification is common in medical testing.  Often you want to diagnose if someone has a disease.  This can lead to two types of errors, know as false positives and false negatives:\n",
    "\n",
    "* **False Positive** - Your test (neural network) indicated that the patient had the disease; however, the patient did not have the disease.\n",
    "* **False Negative** - Your test (neural network) indicated that the patient did not have the disease; however, the patient did have the disease.\n",
    "* **True Positive** - Your test (neural network) correctly identified that the patient had the disease.\n",
    "* **True Negative** - Your test (neural network) correctly identified that the patient did not have the disease.\n",
    "\n",
    "Types of errors can be seen in Figure 4.ETYP. \n",
    "\n",
    "**Figure 4.ETYP: Type of Error**\n",
    "![Type of Error](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_4_errors.png \"Type of Error\")\n",
    "\n",
    "Neural networks classify in terms of probability of it being positive. However, at what probability do you give a positive result?  Is the cutoff 50%? 90%?  Where you set this cutoff is called the threshold.  Anything above the cutoff is positive, anything below is negative.  Setting this cutoff allows the model to be more sensitive or specific:\n",
    "\n",
    "More info on Sensitivity vs Specificity: [Khan Academy](https://www.youtube.com/watch?v=Z5TtopYX1Gc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEJCAYAAACqmv3eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUxdvG8e+kh5KQAIoSOkgJJQQp0qQKKCI9QRCwAIoFu/zs7bUXBERpKkgwAVGK0qsFpIdQFem9twDp5/1jllAMkITdnS3P57pyZbPt3HuyeTI7Z86MsiwLIYQQzudjOoAQQngrKcBCCGGIFGAhhDBECrAQQhgiBVgIIQyRAiyEEIZIAXYCpdQrSqmNSqkkpVSiUqq+HZ97qe17WaXU/Zdcf7tSaqi9tmMvSqlM2z5Yp5Rao5RqmIvHJDsjWw7bfVMp9byJbV+Lg99PM5VSRWyXn1JKbVZKxSmlOiilBttrO0LzMx3A0yml7gDaA9GWZaUqpYoBAfZ6fsuyLhSwssD9wETb9auAVfbajh2dtywrCkAp1QZ4H7jTbCT34YT3092X/DgQaGdZ1g7bz9PttR2hSQvY8W4BjlqWlQpgWdZRy7L2K6XqKKWWKKVWK6XmKKVuAVBKLVZKfaiUWqGU+kcp1cR2faTtukRby6eS7foLrcMPgCa2259RSjVTSv2ilPJRSu280KqxPeZfpdTNSqniSqkpSqmVtq9GTt0zEAKcsGUqpJRaYGsVr1dK3Xflna92H1vrf7NSarStZThXKRVsu62iUmr+JS3uCrbrX7C95iSl1FuXbOMVpdTfSqn5QGVn7IQ8utr7aecl75sVSqmKAFf7Hdv25be2/ZiklOpiu36nUqqYUuproDww3fZ+6quUGm67z81KqZ9t+3Rdbj7FiKuwLEu+HPgFFAISgX+AEejWnj+wFChuu08M8I3t8mLgU9vlu4H5tsvDgJ62ywFAsO1ysu17M+CXS7ab/TPwBfCg7XL9S55zItDYdrk0sNkJ+yPTtj+2AKeAOrbr/YAQ2+ViwL+AuuI15ngfdOs/A4iy3TYJ6GW7vBzoZLscBBQA7gJG2R7rA/wCNAXqAOtt9wmxPf/zpt9D13s/2a7fCbxiu9z7kt99jr9j4ENgyCXPG3bJ8xTL4XJfYLjtcgLwtO2yLxBqer+465d0QTiYZVnJSqk6QBOgOfrN+y5QHZinlAL9Jj5wycN+sn1fjS4uAMuAV5RSEcBPlmVtzUOMBOB14Fsg1vYzQCugmi0DQIhSqrBlWWfy8Nx5dWkXxB3AeKVUdXQxfE8p1RTIAkoCNwMHL3ns1e4DsMOyrETb5dVAWaVUYaCkZVk/A1iWlWLb7l3oIrzWdv9CQCWgMPCzZVnnbPdzuY/cOb2fLumb/eGS75/bLuf4O7ZdH3vJ857IQ4wW6CKPZVmZ6H+kIh+kADuB7U26GFislFoPPA5stCzrjqs8JNX2PRPb78iyrIlKqeXAPcAcpdQjlmUtzGWEZUBFpVRxoCP6HwDo1t8dlmWdz+trsgfLspbZ+jCLo1v7xdEt4nSl1E50i/VSPa9xn9RL7pcJBKMLdk4U8L5lWSMvu1KppwGXnxwlh/dTnws3XXo32/ccf8dKV2SXf62eTvqAHUwpVflCf61NFLAZKG5rAaKU8ldKRV7necoD2y3LGoo+GFLzirucQbfg/sPSnxV/Bj5DfwQ9ZrtpLvDEJduIyvULswOlVBV06/8YEAocthXW5kCZHB6Sm/tksyzrNLBXKdXRtr1ApVQBYA7wkFKqkO36kkqpm4DfgE5KqWBbK/Fe+7xS+7nK+2mX7XLMJd+X2S5f7Xd85fVheYixAHjM9jhfpVRIHh4rLiEF2PEKAeOUUpuUUklANXR3QFfgQ6XUOnSf3vUOZMQAG5RSiUAVYPwVtycBGbaDIs/k8PgEoBcXux8AngJutx2E2QQ8msfXlh/BtgOFibYsfWwtujhbllXolu6WHB6bm/tc6QHgKdu+XwqUsCxrLrpvdJmtBfkjUNiyrDW2TInAFOD3G3mhDpLT++lN222Btk9Jg4AL74Gr/Y7fBcKUUhts78HmecgwCGhu23ergWs2HsTVXTjIIYRwY7bumNstyzpqOovIPWkBCyGEIdICFkIIQ6QFLIQQhkgBFkIIQ6QACyGEIVKAhRDCECnAQghhSJ5ORS5WrJhVtmxZB0URefH3338DULmyK07Y5VyyLy6SfeGaVq9efdSyrOJXXp+nAly2bFlWrXLFKWa9T7NmzQBYvHix0RyuQPbFRbIvXJNSaldO10sXhBBCGCIFWAghDJECLIQQhsh8wEKIbOnp6ezdu5eUlBTTUdxSUFAQERER+Pv75+r+UoCFENn27t1L4cKFKVu2LJesoiFywbIsjh07xt69eylXrlyuHiNdEEKIbCkpKRQtWlSKbz4opShatGiePj1IARZCXEaKb/7ldd9JARZCuBSlFM8991z2z5988glvvvmm3bfz3nvvXfZzw4bXW5TG/qQACyFcSmBgID/99BNHjzp2cY8rC/DSpUsdur2cSAEWQrgUPz8/+vfvz+eff/6f244cOUKXLl2oW7cudevW5c8//8y+vnXr1kRHRzNgwADKlCmTXcA7duxInTp1iIyMZNSoUQAMHjyY8+fPExUVRc+ePQEoVKgQADExMcycOTN7m3379mXKlClkZmbywgsvULduXWrWrMnIkZctqp0vUoCFEC7n8ccfJy4ujlOnTl12/aBBg3jmmWdYuXIlU6ZM4ZFHHgHgrbfeokWLFqxZs4ZOnTqxe/fu7Md88803rF69mlWrVjF06FCOHTvGBx98QHBwMImJicTFxV22jdjYWBIS9Nq1aWlpLFiwgLvvvpuxY8cSGhrKypUrWblyJaNHj2bHjh039DplGJoQ4qqafdfsP9d1j+zOwLoDOZd+jrvj7v7P7X2j+tI3qi9Hzx2l66Sul922uO/iXG03JCSE3r17M3ToUIKDg7Ovnz9/Pps2bcr++fTp05w5c4Y//viDn3/+GYC2bdsSFhaWfZ+hQ4dm37Znzx62bt1K0aJFr7rtdu3a8dRTT5Gamsrs2bNp2rQpwcHBzJ07l6SkJH788UcATp06xdatW3M95CwnUoCFEC7p6aefJjo6mgcffDD7uqysLJYtW3ZZUQY9BjcnixcvZv78+SxbtowCBQrQrFmz6w4TCwoKolmzZsyZM4eEhAR69OiRvY1hw4bRpk2bG3xlF0kBFkJc1bVarAX8C1zz9mIFiuW6xZuT8PBwunfvztixY3nooYcAuOuuuxg+fDgvvPACAImJiURFRdG4cWMmTZrESy+9xNy5czlx4gSgW6lhYWEUKFCALVu28Ndff2U/v7+/P+np6TmetRYbG8uYMWNYtWoV3333HQBt2rThq6++okWLFvj7+/PPP/9QsmRJChYsmO/XKH3AQgiX9dxzz102GmLo0KGsWrWKmjVrUq1aNb7++msA3njjDebOnUt0dDSzZs3illtuoXDhwrRt25aMjAxq1qzJa6+9RoMGDbKfq3///tSsWTP7INyl7rrrLn777TdatWpFQEAAAI888gjVqlUjOjqa6tWrM2DAADIyMm7o9eVpWfrbb7/dkvmAXYPM+3qR7IuLbnRfbN68mapVq9ovkJOkpqbi6+uLn58fy5Yt47HHHiMxMdFIlpz2oVJqtWVZt195X+mCEEK4vd27d9O9e3eysrIICAhg9OjRpiPlihRgIYTbq1SpEmvXrjUdI8+kD1gIIQyRAiyEEIZIARZCCEOkAAshhCFSgIUQLsXX15eoqCiqV69Ot27dOHfuXJ4ev3//frp21adAJyYmXjaxzvTp0/nggw/smvdGSAEWQriUC5PkbNiwgYCAgOyTLXLr1ltvzZ6v4coC3KFDBwYPHmzXvDdCCrAQwmU1adKEf//9l+PHj9OxY0dq1qxJgwYNSEpKAmDJkiVERUURFRVF7dq1OXPmDDt37qR69eqkpaXx+uuvk5CQQFRUFAkJCXz33Xc88cQTnDp1irJly5KVlQXAuXPnKFWqFOnp6Wzbto22bdtSp04dmjRpwpYtWxz2+qQACyFcUkZGBrNmzaJGjRq88cYb1K5dm6SkJN577z169+4N6NUyvvzySxITE/n9998vm6QnICCAt99+m5iYGBITE4mJicm+LTQ0lFq1arFkyRIAZsyYQZs2bfD396d///4MGzaM1atX88knnzBw4ECHvUY5EUMIkbOnnwZ7n84bFQVDhlzzLhcmSgfdAn744YepX78+U6ZMAaBFixYcO3aMU6dO0ahRI5599ll69uxJ586diYiIyHWUmJgYEhISaN68OfHx8QwcOJDk5GSWLl1Kt27dsu+XmpqajxeaO1KAhRAu5UIf8KVymrNGKcXgwYO55557mDlzJg0aNGD+/PkEBQXlajsdOnTgf//7H8ePH2f16tW0aNGCs2fPUqRIEafNIyEFWAiRs+u0VJ2padOmxMXF8dprr7F48WKKFStGSEgI27Zto0aNGtSoUYNly5axZcuW7NYzQOHChTlz5kyOz1moUCHq1avHoEGDaN++Pb6+voSEhFCuXDkmT55Mt27dsCyLpKQkatWq5ZDXJX3AQgiX9+abb2ZPQzl48GDGjRsHwJAhQ6hevTq1atUiODiYdu3aXfa45s2bs2nTpuyDcFeKiYlhwoQJl/UPx8XFMXbsWGrVqkVkZCTTpk1z2OuSFrAQwqUkJyf/57rw8PAcC+GwYcP+c13ZsmXZsGFD9uNWrlx52e19+/bNvty1a9f/dG+UK1eO2bNn5yd6nkkLWAghDJECLIQQhkgBFkIIQ6QACyEuk5dlysTl8rrvpAALIbIFBQVx7NgxKcL5YFkWx44dy/U4ZJBREEKIS0RERLB3716OHDliOopbCgoKytPZeFKAhRDZ/P39KVeunOkYXkO6IIQQwhApwEIIYYgUYCGEMEQKsBBCGCIFWAghDJECLIQQhkgBFkIIQ6QACyGEIVKAhRDCECnAQghhiBRgIYQwRAqwEEIYIgVYCCEMkdnQ3MyJ8ydYvm85O0/uJDMrk10nd1GmSBnTsYRpR4/CX3/Bjh1gWTBlCtxxB9x6q+lk4hqkALuJ/Wf289ivjzH97+n6ilOgUGRkZQCwct9KUjJSaFKmicGUwun+/RcGDICFCy9epxR07aov16kDo0ZBdLSZfOKapAvCTZxLP8fSPUt5pckrLOi9gMalGtOkdBPKh5UH4PXFr3Pnd3fy1KynOJt21nBa4XCZmfD551CzJqxeDe+8A0uWQJMm0Lixbg1/+ins3w/16sGrr0JqqunU4gpSgF1YZlYmE9dPxLIsKoZXZNfTu3i3xbu0KNcCXx9flFIopQCY3G0yT9R7gmErhhE9KpoDZw4YTi8cJiUF7r4bnn0WWraEjRt1gW3aFHx89Ff9+vr2jRuhVy/4v//TXRInTphOLy4hBdhFWZbFgF8G0POnnizYsQCAAv4Frnr/QgGFGNpuKAt7L2Tf6X3cNeEuTpyXPzaPk5EB998Pc+fCV1/B9OlQsuTV7x8WBt99B1On6mLcvj2cO+e0uOLapAC7qMHzBzN27VheafIKrcq3yvXjmpdrzrTYadS4qcY1C7ZwQ5YFjz4KP/8MX3yhL9s+AV3XfffBxIm6a6JrV0hLc2xWkStSgF3Q58s+56OlH/FonUd5p/k7eX58y/ItmdhlIoF+gaRkpJCZlemAlMLpXn0Vxo6F116Dp57K++O7dIGvv4ZZs+Chh3RBF0ZJAXYxh88e5uWFL9OpSieG3z08u483P46dO0b0yGi+WP6FHRMKIxYtgvfeg0cegbfeyv/z9OsHb78NcXEwYYL98ol8kQLsYm4qeBOL+ixiTIcx+Pr43tBzhQeHUzG8Ii8veJnNRzbbKaFwutOn4cEHoVIl3fVwA/+UAXj5ZWjUCJ58EvbutU9GkS9SgF3IzpM7AWgQ0YDw4PAbfj6lFKPuHUXBgIL0mdone8ywcDPPPw979uiDaQXs0K/v66ufKz1dt6ilK8IYKcAuYvne5VQaVomJ6yfa9XlLFCrBiLtHsHL/Sj768yO7PrdwgtmzYfRoXYQbNrTf81asCB99BHPmwJgx9ntekSdSgF1AZlYm/Wb045ZCt3BPpXvs/vwx1WPoHtmd6X9Pl1awOzl/Hvr3h2rVbqzf92oee0yPI37uOTh0yP7PL65LTkV2AePXjWf94fVM7jaZ0KBQh2zjmw7fEOwfjI+S/7luY9gw3fWwaBEEBdn/+X184MsvITJSn0k3fLj9tyGuSf4aDTuffp7XFr1G/ZL16VK1i8O2UzCgID7KhxPnT7D9xHaHbUfYybFjetTDPfdAs2aO207lyrqVPXIkbN3quO2IHEkBNmzNgTWcSj3FR60/uqEhZ7mRZWVxx9g7eHj6w1hy4MW1vfcenDkDH3zg+G298QYEBurREcKppAAb1qh0I/Y8s4emZZo6fFs+yocn6j3B4p2LmfXvLIdvT+TTzp26O6BPH6he3fHbu/lmfZDvxx9h+XLHb09kkwJs0PpD67EsiyJBRZy2zf51+lMxvCIvzX9JzpBzVa+9pvtn337bedt87jldiF98UYalOZEUYEMOJR+i7ui6vPvbu07dboBvAO82f5cNhzcwdctUp25b5MLWrfostSefhIgI5223cGFd+H/7TX8Jp5ACbMiwFcNIy0yje2R3p2+7a7WuVAqvxLK9y5y+bXEdn34KAQF6Kklne+ghuOkm+PBD52/bS8kwNAPOpJ7hy5Vf0rFKRyoXq+z07fv6+LK6/2oKBxZ2+rbFNRw8qM9Q69MHSpRw/vaDg/UkP6++CklJerJ34VDSAjZgzJoxnEw5yUuNXjKW4ULxPZQsA/BdxtCheprI5583l2HgQChYUJ8lJxxOCrABkzdN5s4yd1I/or7RHNP/nk7E5xEkHkw0mkOgJ9wZMUJPGVmpkrkcYWF6XHB8POzaZS6Hl5ACbMDivouZ0Nn8VIBNyzQl2C9Y5ohwBaNGwalT8JK5T0XZnnlGz7j22Wemk3g8KcBOZFkWGVkZBPgGEBHixCPcV1EkqAgD6gwgYWMCu05Ka8eY9HQYMgRatIDbbzedBkqVgp499SQ9x4+bTuPRpAA70W+7fqPcF+VYd3Cd6SjZnqz/JAAjV480nMSLTZsG+/bB00+bTnLRM8/otePGjTOdxKNJAXaiEatGcDbtLJWKGuzju0Lp0NK0v60949aNkxMzTPnqKyhTRq907Cpq1dKrKH/1FWRlmU7jsaQAO8nB5IP8tPkn+kb1dbnFMj9u/TErHllxwytwiHzYsgUWLoQBA/RE6a5k4EB9YsjChaaTeCwpwE4yds1YMrIyePT2R01H+Y/bit5GyZBrLG0uHOfrr8HfHx5+2HSS/+raFYoV06MzhENIAXaCzKxMRq4eSavyrbit6G2m4+Ton2P/0Pr71qw/tN50FO9x9qw+8aJrV30GmqsJCtJnx02fLmvHOYgUYCe4sDbbW80csKqBnRQNLsrvu37nq1VfmY7iPeLj9dCzxx4zneTqBgzQfcCjR5tO4pGkADuBj/KhbcW2NCxlxzW97KxogaLEVI/h+6TvOZN6xnQc7zBihJ5usnFj00murnx5aNdOj1NOTzedxuNIAXawXSd3MXj+YA6cOWA6ynU9WudRktOSSdiYYDqK51u7Ftas0S1MB0/Ef8MefVTPUzFzpukkHkcKsIONWzeOD//8kNTMVNNRrqtBRAOqFKvCt4nfmo7i+b79Vs96dv/9ppNcX7t2eq7gb+V9YW8yG5oDZVlZfJf4HS3KtaBskbKm41yXUor/Nf4fR84ewbIshy+R5LVSU/Wcvx07Qni46TTX5+cHDzygz9Y7fNg1Dxi6KWkBO9Bvu35jx8kdPBj1oOkouda7Vm+ea/icFF9HmjFDn+L7oPu8L3jwQcjIgAnm5zDxJFKAHei7xO8ICQyhc9XOpqPkybn0c8RviJcz4xzl22+hZElo3dp0ktyrVg3q1dPZZckiu5EC7ED+Pv70reV6Z75dz6yts+gxpQfzts8zHcXz7N8Ps2dD796ud+bb9Tz4IGzYAKtXm07iMaQAO9DoDqP5ot0XpmPkWfvb2hMeHC4H4xzh++/1uNq+fU0nybvYWH1yhhyMsxspwA6y/cR20xHyLdAvkF41ejF1y1SOn5fpCO3GsvSZb40bw22ueUbkNRUpAp07w8SJkJJiOo1HkALsANtPbKfC0AqMXTPWdJR86xvVl7TMNCZvnGw6iudYtUpPvtOnj+kk+de3L5w8Cb/+ajqJR5AC7ABxSXEAtK7gRgdZrhBVIopqxauxdO9S01E8x4QJEBio535wVy1a6AVDZTSEXcg4YDuzLIu49XE0LdOU0qGlTcfJN6UUvz/4O2FBYaajeIaMDD33Q/v2+qO8u/L1hR494Msv9VA6dxjH7MKkBWxnqw+s5u9jf9OrRi/TUW5YeHA4SiksGXZ04+bP1ycx9HL/9wW9eunVm3/80XQStycF2M4mrp9IgG8AXau58cfMS3z4x4fcMfYOKcI3Ki5Ot3zbtTOd5MbVrg1VqujXJG6IFGA7e+PON/j1/l8JC/aMj+6hQaEs37dclq6/EWfPws8/Q7duug/Y3SmlF+387TfYvdt0GrcmBdjOQoNCaVW+lekYdtOtWjf8ffyJWy+tnXybNk0XYU/ofrjgwiRCEyeazeHmpADb0f/99n9uPfQsJ0ULFKVdpXZMXD9RTk3Or7g4vdS7K8/7m1fly0PDhno0hHRP5ZsUYDs5l36OD/78gGV7l5mOYne9avTiQPIBFu9cbDqK+zlyBObM0S1GHw/7c+vVCzZuhKQk00ncloe9I8yZuXUmyWnJ3F/DDeZ3zaP2t7Xn+Tuep0yRMqajuJ8pUyAz0z3m/c2rrl31sLQEmcA/v6QA20n8hnhuLngzd5a503QUuwv2D+bjuz6mYnhF01HcT3y8nkmsRg3TSeyveHFo1Uq/RumGyBcpwHZwOvU0v279le6R3fH1cbMZrnIpy8pi0Y5FrDu4znQU97Fvnx4pEBvr+ssO5VdsLOzYAStWmE7ilqQA28Gh5EM0iGhAj+o9TEdxmIysDLpM6sInyz4xHcV9TJqkW4YxMaaTOE7Hjnpppfh400nckhRgO6hUtBKL+izijlJ3mI7iMAG+AXSp2oWpW6ZyPv286TjuIT4eoqPdc+az3CpSBO6+W/cDZ8oombySAnyDktOSOXruqOkYThFbPZbktGRmbpXVca9r+3b9sTw21nQSx4uNhQMH4I8/TCdxO1KAb9AP63+gxCcl+Pf4v6ajOFyzss24ueDN/LDhB9NRXN+FkQHdu5vN4Qzt20OBAtINkQ9SgG9Q/MZ4KoRXoEJYBdNRHM7Xx5fukd1ZuX8lGVkZpuO4tvh4faJCGS8YulewIHTooCfnSU83ncatSAG+AQeTD7J452JiImO8ZhXhd5q/w7antuHnIzOZXtXmzfrkBG/ofrggNhaOHoUFC0wncStSgG/Aj5t+JMvKIibSg49yXyE0KFSK7/UkJOhhZ+488XpetWkDISF65IfINSnAN2DSxklEFo8k8qZI01Gc6pd/fqHy8MqcTDlpOorrsSxdhO68E265xXQa5wkKgvvu07O+paWZTuM2pADfgHEdxzGy/UjTMZyueIHi/HPsH6ZumWo6iuvZsEF3QXjy2N+riYnR68XNm2c6iduQAnwDyoWVo1HpRqZjOF29kvUoW6QskzbKx83/mDRJT7rTubPpJM7XurUeFyzdELkmBTif/jf/f8zdNtd0DCOUUnSv1p152+dx7Nwx03Fch2Xp/t8WLeCmm0yncb6AAOjUCaZOlWXrc0kKcD7sPrWbD/78gFX7V5mOYkz3yO5kZGVIN8Sl1q2DrVu9Y+zv1XTvDqdPw1zvbJzklRTgfJi8cTKgi5C3ir4lmkH1B1G1eFXTUVxHQoKentEbux8uaNlSr5QsU1TmiownyodJmyYRfUu0V0/PqJRiSNshpmO4jgujH1q1gqJFTacxx99f/wOKj4fz5yE42HQilyYt4DzacWIHK/atoHs17239XmrTkU1e3RWTbfVqPf+DN3c/XBATA8nJMGuW6SQuTwpwHu07s48qxap4dffDBZZl0TmhMy/Oe9F0FPMSEnTrr1Mn00nMa9ZMT9Yu3RDXJQU4jxqXbszmxzdTLqyc6SjGKaXoHtmdJbuWcDD5oOk45lzofrjrLggLM53GPD8/6NIFfvlFrwYtrkoKcB6cST1Dakaq6RguJSYyhiwriymbppiOYs7y5bB7t3eefHE1MTFw7hz8+qvpJC5NCnAeDFsxjBKfluBUyinTUVxG5E2RVCtejUmbvHjw/aRJegxshw6mk7iOJk2gRAk5KeM6pADnQcLGBKoVr0ZoUKjpKC4lJjKGlftWeufcEFlZusi0aweh8r7I5uurJyP69Vc4c8Z0GpclBTiXthzdQtKhJBn9kIMn6z3JwecPUiSoiOkozrd0qV58U0Y//FdMjD4jbsYM00lclhTgXJq0cRIKRddqXjTFYC6FBYcREhhiOoYZkybpmcDuvdd0EtfTsCGULCndENcgBTiXJm2cROPSjSkZUtJ0FJe0fO9yGoxpwO5Tu01HcZ7MTJg8WS9KWbiw6TSux8cHunXT44FPyXGTnEgBzgXLshjTYQzvtXzPdBSXVbRAUZbvW559mrZX+P13OHhQRj9cS0yMnh942jTTSVySFOBcUErRIKIBjUs3Nh3FZVUMr0idW+qQsNGLBt8nJOj10O65x3QS11W/vl4XT07KyJEU4OuwLItXFrxC4sFE01FcXmz1WFbuX8m249tMR3G89HS9CGWHDroIi5wppVvBc+fCMZm69EpSgK9jzYE1vPfHe6zev9p0FJd34fRsr2gFL1yoF6H0poU38ys2FjIy4KefTCdxOVKAryN+Qzz+Pv50qirn+F9P6dDSPFXvKaoW84IpKuPj9bjfNm1MJ3F9UVFw2216n4nLyHSU15BlZZGwMYE2FdsQHhxuOo5b+KLdF6YjOF5qqm7NdekCgYGm07g+pXQr+J134MAB71qs9DqkBXwNy/YsY8/pPcRGysfMvDh89jAr9q0wHcQpjNYAABjSSURBVMNxZs/Wqz5I90PuxcToSYt+/NF0EpciBfga9pzeQ0RIBB0qyzn+edHzp570+qkXlmWZjuIY8fFQrJhe+03kTrVqULOmdENcQQrwNcRWj2XX07soHCiD7PMiJjKGrce3eubIkbNnYfp0Pc+Bn/Tg5UlsrD51e9cu00lchhTgq0hOS8ayLHyU7KK86ly1M34+fkxcP9F0FPubMUNPsyjdD3l34YQVaQVnk+pyFU/NeoroUdGe+zHagcKDw2lbsS3xG+PJsrJMx7GvuDiIiNDTLYq8KV9en5gx0QP/MeeTFOAcpGSkMGXzFKJKRKGUMh3HLfWs0ZO9p/ey9sBa01Hs5+hRfQCuRw89z4HIu549ISkJNmwwncQlyLsoB7/88wunU0/Ts0ZP01Hc1n2V72PHoB3UubWO6Sj2M3myPqGgp7wv8q17dz1XsLSCASnAOYpbH0eJQiVoXra56ShuK9g/mLJFypqOYV8TJ148mi/y5+aboVUrvS+zPKx7Kh+kAF/hxPkTzNw6k9jIWHx9fE3HcWuHkg/R4YcOzPjbAybk3rUL/vhDt36lW+rG9Oyp9+eyZaaTGCfjaK5QMKAgCV0TvON0Wge7MEVloF8g91Z28wnLL3xk7tHDbA5P0LEjBAfrA5qNGplOY5S0gK8Q4BtAxyodqVyssukobs/Px4+YyBhm/D3DvRcytSxdLBo2hHLlTKdxf4ULw3336ZUy0tNNpzFKCvAl9p3ex5uL3+Rg8kHTUTzG/TXuJzUzlZ82u/FMWOvXw8aNcP/9ppN4jvvv19NTzpljOolRUoAvMSFpAm8teYvktGTTUTxG/ZL1qRBWge+TvjcdJf/Gj9dnvcnCm/bTpo0+nft7N35f2IH0AdtYlsW4deNoVKoRFcMrmo7jMZRSvNDwBY6eO4plWe43rjojAyZM0KteFC9uOo3nCAjQ/emjRsGJExAWZjqREdICtlm1fxWbj26md63epqN4nAG3D+CVpq+4X/EFvZLDoUPQp4/pJJ6nd289tacXr5osBdhm3LpxBPoGZq/qIOwrJSOFGX/PcL9Tu8ePh/BwvfKxsK86dfS46vHjTScxRgqwTUpGCt0ju1MkqIjpKB4pYUMCHeI78Nfev0xHyb2TJ2HqVP1RWSZetz+l9CeLpUth61bTaYyQAmwzpsMYxnUcZzqGx+pctTPBfsGMW+dG+3jSJP0RWbofHKdXLz2vhpe2gqUAo1dwANyzj9JNFA4sTOeqnUnYmEBKRorpOLkzfjxUrQq33246iee69VZ9avL333vlqcleX4CPnjtKxGcRDFs+zHQUj9enVh9Oppx0j1OT//0X/vxTt37lH7Nj9emjT03+7TfTSZzO6wvwhKQJpGel06xsM9NRPF6Lci2ICIlg7ra5pqNc37ff6o/GvXqZTuL5OnaEkBD45hvTSZzOqwuwZVmMWTOGeiXrUePmGqbjeDxfH1+WPbyMUfeOMh3l2jIydAFu1w5KljSdxvMVKKDPjJs8WR/49CJeXYD/2vsXG49spF90P9NRvEZESARKKdcejjZrll4+vZ+8L5ymXz9ISdFzbngRry7A36z9hoL+BYmJjDEdxat8tfIrokZGkZmVaTpKzsaMgRIlZOyvM0VHQ+3aMHq0nvzIS3h1Af6w9YdMi50mqx47WbECxUg6lMS87fNMR/mv/fvh11+hb1/w9zedxrv06wfr1sGaNaaTOI1XF+Dw4HBalm9pOobX6VC5A8UKFGPMmjGmo/zXd99BZiY8/LDpJN6nRw89T/Do0aaTOI3XFuB+0/sxdctU0zG8UqBfIH1q9WHa39Oyx2C7hKwsGDsWmjeHijIhk9MVKQLduunJ78+eNZ3GKbyyAK87uI4xa8ew+9Ru01G81sO1HyYjK4NxiS50ZtyiRbB9OzzyiOkk3qtfPzhzxmsm6PHKAjxi5QiC/ILoVVPGeJpStXhV3m/5Pm0qtjEd5aIRI6BoUejUyXQS79WokZ6gZ8QI00mcwusK8MmUk0xYP4H7q99PeHC46ThebXDjwdS82UVWGN6zB6ZN032/wcGm03gvpWDgQFi1ClasMJ3G4byuAI9LHMe59HM8Xu9x01EEsObAGj5d+qnpGDBypO4Dfuwx00lE79563bjhw00ncTivK8ClQ0vzcO2Hib4l2nQUAcz4ewbPz3uef4//ay5Eaqo+8n7vvVC2rLkcQitcWBfhhAQ4csR0GofyugLcqWonxnRwweFPXqp/nf74+fgxYqXBPr8ff4TDh+Fx+VTkMh5/HNLS9EkxHsyrCvC0LdM4meJd55q7ulsK30KXql34NvFbzqYZGnr05Zdw2216WkThGqpWhRYt4Ouv9dwcHsprCvCOEzvolNCJz5Z9ZjqKuMIT9Z7gZMpJJq6f6PyNr1kDy5bpFpeP1/w5uIcnnoDdu+GXX0wncRiveccNXT4UXx9f+tfpbzqKuEKjUo24s8ydnM847/yNf/45FCokq164onvvhdKl9e/IQ3nFsvQnzp9g9JrRxFaPJSIkwnQccQWlFIv6LHL+iiR79kB8PDz5JISGOnfb4vr8/ODpp+HZZ/WQtHr1TCeyO69oAX+96mvOpp/l+TueNx1FXMWFKSqX7VnmvI1+8YWeeWvQIOdtU+TNI4/of46ffGI6iUN4RQHecGQDrcu3plaJWqajiGsYu3YsDb9pyKr9qxy/sVOnYNQoiImBMmUcvz2RP4ULw6OPwpQp+jRxD+MVBTiucxzTYqeZjiGuo3tkd0ICQ/hkqRNaOyNH6jkHnpdPRS7vqafA1xc+87wD6B5dgLOsLPaf2Q9AsL+cXurqQgJDGFBnAJM3TWbHiR2O21Bamu5+aNlSTwIuXNutt+q1+b75Bo4eNZ3Grjy6AM/cOpOyQ8qydM9S01FELg2qPwhf5evY4YJxcXridWn9uo/nnoPz5/WYbQ/isQXYsizeXvI2JUNKUvfWuqbjiFwqGVKS3rV6M2fbHNIz0+2/gYwM+L//g6goaONCM7GJa4uMhA4dYMgQ3X/vITy2AM/6dxYr96/k1Sav4u8rS8u4k49bf8z6x9Y75vcWFwfbtsGbb+qZt4T7eOMNvWry0KGmk9iNRxZgy7J4c/GblC1Slt61epuOI/IoLDiMQL9AUjNSOZVix9ZORga8845u/XboYL/nFc4RHa1/b5995jGtYI8swBsOb2DNgTXS+nVjKRkpVP2yKq8vet1+TyqtX/fnYa1gjyzANW6uwT9P/iOtXzcW5BdE87LNGbl6ZPZIlhsirV/P4GGtYI8rwKdTTwNQPqy8tH7d3CtNXyEjK4P3f3//xp/s+++l9espLrSCv/jCdJIb5lEFOCMrg4ZjGzJolpxa6gnKh5XnodoP8fXqr9l6bGv+n+jcOXj9dbj9dmn9eoLoaL1u38cfw6FDptPcEI8qwN+u/ZaNRzZyZ9k7TUcRdvJ287cJ9A1k3LobWD15yBDYu1fPJyCtX8/wwQeQkqI/0bgxjynAyWnJvLboNRqVakSnKrKqracoUagEq/uv5p3m7+TvCQ4dgvffh/vugzvlH7PHuO02vX7f6NGwaZPpNPnmMQX44z8/5tDZQ3x616fOn9ZQOFTlYpVRSnHk7BEsy8rbg996S7eUPvrIMeGEOa+/DgULwksvmU6Sbx5RgNMy0/gm8RtiImOoH1HfdBzhAEmHkig/tDyTNk7K/YM2b9Yznj36qG4xCc9SrBi88opeMWPhQtNp8sUjCnCAbwBJjyYxpO0Q01GEg0QWj6RCWAVemPcCyWnJ13/AhXl+CxbULSXhmZ56Sk8nOmgQpDvg1HUHc/sCvOvkLjKzMgkLDqNEoRKm4wgH8fXx5cu7v2TP6T28ufjN6z/ghx9g3jw970Px4g7PJwwJCtInZWzY4JbTVbp1AT6ffp6W41vS6+depqMIJ2hUuhH9ovsx5K8hJB5MvPodT5yAZ56BunX1gRrh2Tp0gI4ddX//DgdOY+oAbl2A/+/3/2PbiW30i+5nOopwkg9afUCxAsWYu23u1e/0v//peWNHjtQTeQvPN3So/l0PHKi7n9yE2xbgTUc28dGfH/FAzQdoUa6F6TjCScKDw9n8+GZebPRizndYulQX3qeflsnWvUmpUvDuuzB7NkyebDpNrrllAU7PTKfP1D4UDizMp3d9ajqOcLKw4DAAVuxbwbbj2y7ecPYsPPig/mN86y1D6YQxTzwBderA44/DwYOm0+SKWxbgA8kHOJN6hlHtR1G8oBxg8Ubn0s/RfmJ7ev3cCwvbR85nn4WtW2HcOChUyGxA4Xy+vjB+PCQn63/EbtAV4ZYFuHRoaZIeS6JLtS6mowhDCvgXYFi7Yfy19y92ndyl+3xHjdLLDDVvbjqeMKVaNT1HxOzZMGKE6TTX5VYF+FTKKV5e8DJn084S4BtgOo4wLKZ6DL1q9uLAsV1Yf2/RU02+k89TloXnePxxaNdO/zN28dOU3aYAW5bFIzMe4aM/P2LjkY2m4wgXMbz1EKodU1iZmZz65isIDDQdSZimlF5BuVAhiInRXRIuym0K8Pt/vM+Pm37kg1YfUK9kPdNxhIsIfe1dQs9bHI8oSuEoeV8ImxIlYOJE3QLu29dl+4PdogD/8s8vvLrwVXrW6MlzdzxnOo5wFePH66kmS5akWPnq+CgfTqacNJ1KuIrWrfUkTFOm6DMiXZDLF+C0zDSemPkEtW+pzeh7R8tMZ0JbuRL699cH3CpUAODvo39z27Db+Hbtt4bDCZfx7LPQqxe89hpMn246zX+4fAEO8A1g7gNz+TnmZ4L9g03HEa7gn3/gnnvglltg0qTsSdYrhFeg5s01GfDLAGZtnWU4pHAJSunRMXXqQI8esGyZ6USXcdkCfOTsEYavGI5lWdxW9DZKh5Y2HUm4gt27oVUrfXnOHD0loY2fjx9Tuk+h+k3V6TKpC7/v+t1QSOFSgoP1lJW33gp33w3r1plOlM0lC/CplFO0jWvLC/NeYPuJ7abjCFdx6JDu1zt9GubOzXGO39CgUGb3mk3p0NK0/6E9Gw/LiBmBPig3f74eGXHXXfqEHRfgcgX42LljtJnQhqRDSfzU/ScqhFcwHUm4gr17oUUL/f3XX/WY36u4qeBNzHtgHrGRsZQPK+/EkMKllSmjpyi1LH3swAXGCLtUAd5zag9Nvm1C4sFEfuz2I+0qtTMdSbiCLVugUSPYs0cX30aNrvuQUqGlGHnvSIL9gzl27hiLdy52fE7h+qpUgQULIDMTmjQx3ifsUgU48WAiB5MPMqfXHO6rcp/pOMIVrFgBjRvrdd2WLIFmzfL8FC/Oe5HW37dm4vqJ9s8n3E+NGnrWvPBwaNkSZs40FsUlCvDWY7o/5t7K97J90HZZVl5oY8dC06YQEgJ//pnv6SU/a/MZjUo1oudPPXlx3otkZGXYOahwO+XK6fdU1apw773w3nuQleX0GEYLcHpmOs/OeZaqX1blr71/AVAkqIjJSMIVnD8PDz8MjzyiPyYuXw4VK+b76UKDQpnTaw6P3f4YHy/9mJbjW3Io+ZAdAwu3dNNN+lNVTIxe3PO++/RqKk5krABvPLyRxt825vO/Pmdg3YFE3xJtKopwJStXQr16+lz+V1/Vs1rZYU23QL9ARtwzgu87fc+xc8dkMiehFSoEcXEwfLge1hgVpUfYOImRAvzhHx9Se2Rtth3fRkLXBIa2Gyp/EN7u3Dk9e1WDBnD8uO6Xe+cduy8p1KtmL9Y9uo6w4DBSM1J5ds6zHEx2j8m7hYMopWdQ+/13PWa4TRs9n/Dx4w7ftNMKsGVZWLYJMZRSdIvsxubHN9M9sruzIghXlJUFEyboeVw//RT69dPDg9o5bgSMr48u6iv2rWDEyhFUGV6FD//4kPPp5x22TeEG6teHxER4+WX4/ns9YmL4cEhLc9gmHV6ALcti4Y6FNPymIfEb4gF4oeELxHWOk9UsvFlWlj47KToaHnhAH5FetAi+/hpCQ50SoUmZJqx7dB1NyjRh8ILBVBpWidGrR5NlOf9gjHARQUF64p5VqyAyEp58Uh+oi4uDDPsfvHVYAU7NSGX8uvFEj4qm5fiW7Dm1B39ffwCZUMebnTunF82MjNRHn8+cgR9+0G/4fAwxu1GVi1VmRo8ZLOm7hFKhpRi7diw+Sv9ZSIvYi0VFwcKFMGsWFC6sJ/QpX16vtnHSfjPuOawA3xd/H32m9iEtM43R945m65Nb6Vqtq6M2J1xZVpbuX+vfH0qWhEcfhYIFddfD5s0QGws+ZkdENi3TlKUPLWVGjxkAHD13lFs/u5Wuk7oydctU0jId9zFUuCiloG1bWLMGpk3Ts+69+KJ+D/fsqQ8Q32Cr2O9GM6ZkpLBk5xJmbp3Jgh0L+POhPwkNCuXpBk/z7B3P0rp8a2nxeqOzZ3ULYuZMffbanj266HbufHF4mYu9L5RS2d1iGVkZ9K7Zm/iN8UzZPIWwoDDaVmzL63e+TpViVQwnFU7l4wMdOuivtWv1J7hJk/SE78WL6wl+7r5bzzFRJG/DaPNcgDOyMvDz8WP53uW8MO8FVuxbQWpmKkF+QbQo14JDZw8RGhRK24pt8/rUwl1ZFuzcqd+cS5fqAe6rV0N6ui66rVvD++9Dx476ZzdQolAJvmj3BZ+2+ZT52+cTvyGeWf/O4q1mern7aVumMfvf2TQq3YjaJWpTuVhl/HxuuD0jXF3t2vo4xRdf6O6JSZP0PMPjxulCXauWPlW+USN934oVrzmSJ0/vmMSDiUxImkDfqL4UCihEamYqA+sOpFX5VjQv21zm6/VkmZlw8KBuye7eDdu26Rml/vkH1q/XM5QBBARA3brwzDO68DZp4tbrtPn5+NG2YlvaVmxLlpWV3T+87cQ2Jm6YyNervwYg0DeQGjfXYNnDy/Dz8WPNgTWcTTtLREgEJUNKyjBLTxMYqBsUHTvqbojly/X44T/+0GPYhw/X9ytQAKpXv+rT5KkAhwWHUSm8EgCRN0Wy/JHl+X8Bwj4yMnQfa2bmf78yMvRXeroeSpOerudUSE3V38+f119nz+qFC8+c0YX0xAn9dfw4HDkChw/r75mZl2+7RAmoVEkfoKhV6+JXUJCZfeFgF4ovwLN3PMug+oPYdGQTiQcTWXdoHfvP7M9uBb+15C2m/31xBYaiwUWpVaIWC3ovAGDIX0PYd3ofIYEhFA4sTOGAwpQMKZn9yXHtgbWkZ6UT6BtIgG8A/r7+FAooRIlCJQA9ZSvoIXU+ygdf5Zs9vE44mZ/fxVYv6L+59ev1vMPr1kFS0tUfmpftlAktQ6PS15+JyqMdPpy9BA6Qu8X+rnafS6+/cPnS71devvTrAn//3GfPDX9/CAu7+FW2rD4z7aabICICSpXSXxUq6LOIvJivjy81bq5BjZtr8AAPXHbb520+5/G6j7Pn1B72nN7D4bOHKeh/sftl/vb5LNyxkPMZF0da1C9ZP7sA95nah/WH11/2nC3LtWR+7/kA1B5Zmx0nd1x2e8cqHe36+kQ++fnp7odL5y65yvEOZeVhtVCl1BFg1w3Gu1HFgKOGM7gK2RcXyb64SPbFRa6yL8pYlvWfEx/yVIBdgVJqlWVZt5vO4QpkX1wk++Ii2RcXufq+cInpKIUQwhtJARZCCEPcsQCPMh3Ahci+uEj2xUWyLy5y6X3hdn3AQgjhKdyxBSyEEB7BrQuwUup5pZSllCpmOospSqmPlVJblFJJSqmflVJet6aTUqqtUupvpdS/SqnBpvOYopQqpZRapJTarJTaqJQaZDqTSUopX6XUWqXUL6azXI3bFmClVCmgNbDbdBbD5gHVLcuqCfwD/M9wHqdSSvkCXwLtgGpAD6VUNbOpjMkAnrMsqyrQAHjci/cFwCBgs+kQ1+K2BRj4HHgR8OpObMuy5lqWdWFOvL+ACJN5DKgH/GtZ1nbLstKAeOA+w5mMsCzrgGVZa2yXz6CLT0mzqcxQSkUA9wBjTGe5FrcswEqpDsA+y7LWmc7iYh4CZpkO4WQlgT2X/LwXLy06l1JKlQVqA946YcsQdAPNpZc3cdn585RS84ESOdz0CvAycJdzE5lzrX1hWdY0231eQX8EjXNmNheQ00n2Xv2pSClVCJgCPG1Z1mnTeZxNKdUeOGxZ1mqlVDPTea7FZQuwZVmtcrpeKVUDKAess030HgGsUUrVsyzLI5e3vdq+uEAp1QdoD7S0vG9c4V6g1CU/RwD7DWUxTinljy6+cZZl/WQ6jyGNgA5KqbuBICBEKTXBsqxehnP9h9uPA1ZK7QRutyzLFSbccDqlVFvgM+BOy7KOmM7jbEopP/TBx5bAPmAlcL9lWRuNBjNA6RbJOOC4ZVlPm87jCmwt4Octy2pvOktO3LIPWFxmOFAYmKeUSlRKfW06kDPZDkA+AcxBH3Sa5I3F16YR8ADQwvZeSLS1AoWLcvsWsBBCuCtpAQshhCFSgIUQwhApwEIIYYgUYCGEMEQKsBBCGCIFWAghDJECLIQQhkgBFkIIQ/4f+7m/AdsGas0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import scipy.stats as stats\n",
    "import math\n",
    "\n",
    "mu1 = -2\n",
    "mu2 = 2\n",
    "variance = 1\n",
    "sigma = math.sqrt(variance)\n",
    "x1 = np.linspace(mu1 - 5*sigma, mu1 + 4*sigma, 100)\n",
    "x2 = np.linspace(mu2 - 5*sigma, mu2 + 4*sigma, 100)\n",
    "plt.plot(x1, stats.norm.pdf(x1, mu1, sigma)/1,color=\"green\", \n",
    "         linestyle='dashed')\n",
    "plt.plot(x2, stats.norm.pdf(x2, mu2, sigma)/1,color=\"red\")\n",
    "plt.axvline(x=-2,color=\"black\")\n",
    "plt.axvline(x=0,color=\"black\")\n",
    "plt.axvline(x=+2,color=\"black\")\n",
    "plt.text(-2.7,0.55,\"Sensitive\")\n",
    "plt.text(-0.7,0.55,\"Balanced\")\n",
    "plt.text(1.7,0.55,\"Specific\")\n",
    "plt.ylim([0,0.53])\n",
    "plt.xlim([-5,5])\n",
    "plt.legend(['Negative','Positive'])\n",
    "plt.yticks([])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import zscore\n",
    "\n",
    "# Prepare data - apply z-score to ALL x columns\n",
    "# Only do this if you have no categoricals (and are sure you \n",
    "# want to use z-score across the board)\n",
    "x_columns = df.columns.drop('diagnosis').drop('id')\n",
    "for col in x_columns:\n",
    "    df[col] = zscore(df[col])\n",
    "\n",
    "# Convert to numpy - Regression\n",
    "x = df[x_columns].values\n",
    "y = df['diagnosis'].map({'M':1,\"B\":0}).values # Binary classification, \n",
    "                                              # M is 1 and B is 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "# Plot a confusion matrix.\n",
    "# cm is the confusion matrix, names are the names of the classes.\n",
    "def plot_confusion_matrix(cm, names, title='Confusion matrix', \n",
    "                            cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(names))\n",
    "    plt.xticks(tick_marks, names, rotation=45)\n",
    "    plt.yticks(tick_marks, names)\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    \n",
    "\n",
    "# Plot an ROC. pred - the predictions, y - the expected output.\n",
    "def plot_roc(pred,y):\n",
    "    fpr, tpr, _ = roc_curve(y, pred)\n",
    "    roc_auc = auc(fpr, tpr)\n",
    "\n",
    "    plt.figure()\n",
    "    plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver Operating Characteristic (ROC)')\n",
    "    plt.legend(loc=\"lower right\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Chart Example\n",
    "\n",
    "The following code demonstrates how to implement a ROC chart in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 426 samples, validate on 143 samples\n",
      "Epoch 1/1000\n",
      "426/426 - 1s - loss: 0.6823 - accuracy: 0.8873 - val_loss: 0.6580 - val_accuracy: 0.9650\n",
      "Epoch 2/1000\n",
      "426/426 - 0s - loss: 0.6171 - accuracy: 0.9343 - val_loss: 0.5181 - val_accuracy: 0.9650\n",
      "Epoch 3/1000\n",
      "426/426 - 0s - loss: 0.4115 - accuracy: 0.9413 - val_loss: 0.2385 - val_accuracy: 0.9580\n",
      "Epoch 4/1000\n",
      "426/426 - 0s - loss: 0.1872 - accuracy: 0.9507 - val_loss: 0.0922 - val_accuracy: 0.9720\n",
      "Epoch 5/1000\n",
      "426/426 - 0s - loss: 0.1031 - accuracy: 0.9718 - val_loss: 0.0613 - val_accuracy: 0.9790\n",
      "Epoch 6/1000\n",
      "426/426 - 0s - loss: 0.0807 - accuracy: 0.9789 - val_loss: 0.0530 - val_accuracy: 0.9790\n",
      "Epoch 7/1000\n",
      "426/426 - 0s - loss: 0.0725 - accuracy: 0.9812 - val_loss: 0.0471 - val_accuracy: 0.9930\n",
      "Epoch 8/1000\n",
      "426/426 - 0s - loss: 0.0648 - accuracy: 0.9812 - val_loss: 0.0467 - val_accuracy: 0.9930\n",
      "Epoch 9/1000\n",
      "426/426 - 0s - loss: 0.0577 - accuracy: 0.9789 - val_loss: 0.0529 - val_accuracy: 0.9860\n",
      "Epoch 10/1000\n",
      "426/426 - 0s - loss: 0.0533 - accuracy: 0.9836 - val_loss: 0.0507 - val_accuracy: 0.9860\n",
      "Epoch 11/1000\n",
      "426/426 - 0s - loss: 0.0487 - accuracy: 0.9836 - val_loss: 0.0477 - val_accuracy: 0.9860\n",
      "Epoch 12/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "426/426 - 0s - loss: 0.0449 - accuracy: 0.9859 - val_loss: 0.0476 - val_accuracy: 0.9860\n",
      "Epoch 00012: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1f618f96448>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',\n",
    "                kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(1,activation='sigmoid',kernel_initializer='random_normal'))\n",
    "model.compile(loss='binary_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "    patience=5, verbose=1, mode='auto', restore_best_weights=True)\n",
    "\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "          callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3gU5fbA8e8hVCF0RSAqKL0EkI6CFGkWQEXADhZEilIUBVEQvSiKXECa/FDRi4KKAipYAPWCIkiH0BEpUVSaoRi4CTm/P2YSlpBslpDdSTmf59knmZ13Zs68uztn5p2Zd0RVMcYYY1KTy+sAjDHGZG6WKIwxxvhlicIYY4xfliiMMcb4ZYnCGGOMX5YojDHG+GWJIpsTkc0i0tzrODILERkqItM9WvYMEXnJi2VnNBG5R0S+See06f5OisiPIlInPdOml4g8LiKvhHKZmY0lihASkT0iEisiJ0TkD3fDUSiYy1TV6qr6fTCXkUhE8onIyyKyz13PnSLylIhIKJafQjzNRSTa9z1VHaWqDwdpeeJuVKJE5KSIRIvIxyJSMxjLSy8RGSEiMy9mHqr6vqq2CWBZ5yXH9H4nReRW4LiqrnOHR4hInPt7+ltElotI42TTFBWRKe7v7R8R2SQiPVKY990istqd1wER+VJErndHTwPuFZHLLjTm7MISRejdqqqFgNpAHWCIx/FcMBHJncqoj4FWwE1AOHAf0BMYH4QYREQy2/d3PPAE8DhQHKgEzANuzugF+fkMgs7DZfcC/pPsvQ/d31NJ4Duc7yAAIpIXWAxcBTQGigBPAa+IyECfcgOBccAooBRwJTAZ6AigqqeAL4H7g7JWWYGq2itEL2APcKPP8KvAAp/hRsBy4G9gA9DcZ1xx4B3gd+AoMM9n3C3Aene65UBk8mUCZYBYoLjPuDrAISCPO/wgsNWd/9fAVT5lFegD7AR+TWHdWgGngCuSvd8QOANUcIe/B14GfgZigPnJYvJXB98D/wJ+dNelAtDDjfk4sBt41C1b0C2TAJxwX2WAEcBMt0w5d70eAPa5dfGsz/IKAO+69bEVGAxEp/LZVnTXs4Gfz38GMAlY4Ma7ErjGZ/x4YD9wDFgDNPUZNwKYA8x0xz8MNAB+cuvqADARyOszTXVgEXAE+BMYCrQD/gfEuXWywS1bBHjLnc9vwEtAmDuuu1vn/3bn9ZL73g/ueHHH/eV+phuBGjg7CXHu8k4Anyf/HQBhbly/uHWyhmTfIbdcXvfzjEhWJzN9hqu5n+el7vBDbkwFk82rqxtPYXe9TwB3pvHbvQf4zuttiFcvzwPISa9kP5AIYBMw3h0uCxzG2RvPBbR2hxO/9AuAD4FiQB7gBvf9a90fQ0P3R/eAu5x8KSzzW+ARn3heA6a6/3cCdgFVgdzAMGC5T1l1NzrFgQIprNsrwH9TWe+9nN2Af+9uiGrgbMw/4eyGO606+B5ng17djTEPzt76NTgbqxuAf4Br3fLNSbZhJ+VE8X84SaEWcBqo6rtObp1H4GwAU0sUvYC9aXz+M3A2tA3c+N8HZvuMvxco4Y4bBPwB5PeJO879nHK58dbFSay53XXZCvR3y4fjbPQHAfnd4YbJ68Bn2fOAN93P5DKcRJ74mXUH4oF+7rIKcG6iaIuzgS/qfg5VgdI+6/ySn9/BUzi/g8rutLWAEinUXXXgpJ/PMq/7eR0CcrvvzQbeTWFeud31aYuTOOMTp/Hz2V0LHPF6G+LVK7MduucE80TkOM6e41/AcPf9e4GFqrpQVRNUdRGwGrhJREoD7YFeqnpUVeNU9b/udI8Ab6rqSlU9o6rv4mzsGqWw7A+Au8BpugG6ue8BPAq8rKpbVTUe5zC8tohc5TP9y6p6RFVjU5h3SZwNU0oOuOMT/UdVo1T1JPAc0EVEwvzVgc+0M1R1s6rGu/WwQFV/Ucd/gW+ApqnEkZoXVDVWVTfgHMXUct/vAoxy6zwamOBnHiX8rL+vT1X1Z7eO38dpggRAVWeq6mF33V4H8uFsQBP9pKrz3LqJVdU1qrrCLb8HZ0N/g1v2FuAPVX1dVU+p6nFVXZlSQCJSCuf71V9VT6rqXzhHCN18iv2uqm+4y0r++cfhJKIqgLjfoUDqApwjo2Gqut39DDeo6uEUyhXFOeJIrouI/I1ztPEI0NmtW0jlO+mOP+SOLwEc8pkmNcdxjj5yJEsUoddJVcNx9narcHYDehVwp3tS7m/3y389UBq4Amdv5mgK87sKGJRsuitwmlmSmwM0FpEyQDOcvellPvMZ7zOPIzh7eGV9pt/vZ70OubGmpLQ7PqX57MU5MiiJ/zpIMQYRaS8iK0TkiFv+Js5NSoH4w+f/f4DECwzKJFuev/U/TOrrH8iyEJFBIrJVRGLcdSnCueuSfN0ricgX7onaYzjJPbH8FTjNOYG4CuczOOBT72/iHFmkuGxfqvotTrPXJOBPEZkmIoUDXHagcR7FSUbJfaSqRXHOLUThHGUlSvE76Z5jKemOPwyUDOC8SzhOs1qOZInCI+7e7wxgjPvWfpw97aI+r4Kq+oo7rriIFE1hVvuBfyWb7hJVnZXCMv/G2ePuAtwNzFJV9ZnPo8nmU0BVl/vOws8qLQYaisgVvm+KSAOcjcG3Pm/7lrkSZ4/0UBp1cF4MIpIPp+lqDFDK3WAsxElwacUbiAM4TU4pxZ3cEiBCROqlZ0Ei0hR4GuezKeauSwxn1wXOX58pwDagoqoWxmnrTyy/H6dJLiXJ57Mf5yi0pE+9F1bV6n6mOXeGqhNUtS5OE1ElnCalNKdLI05fO3EOhMumNFJVD+EcFY9wj8DB+U62F5GCyYrfgbO+K3DO8ZzCadLzpyrO0WaOZInCW+OA1iJSG+ck5a0i0lZEwkQkv3t5Z4R7GP8lMFlEiolIHhFp5s7j/4BeItLQvRKooIjcLCIp7X2B09R0P86P5QOf96cCQ0SkOoCIFBGROwNdEVVdjLOx/EREqrvr0AineWWKqu70KX6viFQTkUuAkcAcVT3jrw5SWWxenOaZg0C8iLQHfC/Z/BMoISLpbTL4CKdOirkbqL6pFXTXbzIwy405rxt/NxF5JoBlheO0lR8EcovI8zgnW9Oa5hhwQkSqAI/5jPsCuFxE+otz2XK4iDR0x/0JlEu8asz9fn0DvC4ihUUkl4hcIyI3EAARqe9+//IAJ3E2vGd8lnW1n8mnAy+KSEX3+xspIiWSF1LVOJwNf6oxqeo2nIswBrtv/QeIBj4WkXLu76YtThPiCFWNUdUY4Hlgkoh0EpFL3HLtReRVn9nfgPMbzJEsUXhIVQ8C7wHPqep+nMvxhuJsLPbj7JUlfkb34ex5b8M5t9HfncdqnLbZiTiH57twTjSm5jOcK3T+dNvkE2OZC4wGZrvNGFE47dYX4g6cSxS/wrmSZCbOlTT9kpX7D87R1B84J1ofd2NIqw7OoarH3Wk/wln3u931Sxy/DZgF7HabVFJqjvNnJM6G5lecjdQcnD3R1DzO2SaYv3GaVG4DPg9gWV/jbIh24DTHncJ/UxfAkzjrfBxnh+HDxBFu3bQGbsWp551AC3d04iWkh0Vkrfv//TiJdwtOXc4hsKY0cBLa/7nT7cVpzkk8Un4LqObW/7wUph2L8/l9g5P03sI5WZ6SN3F+B/68BvQUkctU9TTOFX/7ca4wO+Yu71lVfS1xAlUdCwzEuYAj8XvXF+cEPyKSH6dJ8900lp1tydmWB2OCT0S+x7lSxZO7oy+GiDwGdFPVgPa0TcYTkR+AfuredBeiZfbDuWR3cJqFsynPbtoxJrNz27qvxmnHrohzqelET4PK4VT1+rRLZfgy3wj1MjMbSxTGpC4vTnNHeZympNk45yGMyVGs6ckYY4xfdjLbGGOMX1mu6alkyZJarlw5r8MwxpgsZc2aNYdU9dL0TJvlEkW5cuVYvXq112EYY0yWIiJ70zutNT0ZY4zxyxKFMcYYvyxRGGOM8csShTHGGL8sURhjjPHLEoUxxhi/gpYoRORtEflLRKJSGS8iMkFEdonIRhG5NlixGGOMSb9gHlHMwHkebWra43S0VhHnIexTghiLMcaYdAraDXequlREyvkp0hF4z33C2goRKSoipS/gWbtB9cHKfcxf/5vXYRhjzEX5a8c6diyZfVHz8PLO7LKc+2CWaPe98xKFiPTEOergyiuvDElw89f/xpYDx6hWOtBH/xpjTOZx6vhRNn4yiT0rFlKwRKDPoEqZl4lCUngvxa5sVXUaMA2gXr16Ievutlrpwnz4aONQLc4YYzLMHXfcQfTqbxgyZAjDhg2jYMHkjw4PnJdXPUVz7sPqI4DfPYrFGGOyvM2bN/Pbb06T+ejRo1m/fj2jRo3ikksuuaj5enlE8RnQV0RmAw2BmEDOT+w+eJKub/4U9OCs2ckYk1WcPHmSF198kddff5177rmHGTNmUKFChQybf9AShYjMApoDJUUkGhgO5AFQ1anAQpwHlu8C/gF6BDLf2LgzwQj3PNVKF6Zj7bIhWZYxxqTXggUL6NOnD3v37uXBBx9k9OjRGb6MLPeEu+JXVdUje7d6HYYxxnhu8uTJ9OnTh2rVqjF16lSaNm2aalkRWaOq9dKznCz3PApjjMnJ4uPjOXjwIKVLl6ZLly7ExsbSr18/8ubNG7Rl2hGFMcZkET///DOPPvoouXPnZsWKFYSFhQU87cUcUVhfT8YYk8n9/fff9O7dm0aNGvHXX3/x9NNPkytX6Dbf1vRkjDGZ2KZNm2jdujUHDx7k8ccfZ+TIkRQuHNorMi1RGGNMJhQXF0eePHmoVKkSLVq04KmnnuLaa73pO9WanowxJhM5ffo0I0eOpHr16pw4cYJ8+fIxa9Ysz5IEWKIwxphM49tvvyUyMpLhw4dz7bXXcvr0aa9DAixRGGOM52JjY7nvvvto1aoV8fHxfPXVV8yePZsSJUp4HRpgicIYYzyXP39+Dh06xLBhw4iKiqJt27Zeh3QOSxTGGOOBjRs30rZtW6KjoxERFixYwIsvvkiBAgW8Du08liiMMSaETp48mXQF09q1a9m5cydASO+LuFCZNzJjjMlmPvvsM6pVq8aYMWN48MEH2b59Oy1atPA6rDTZfRTGGBMi8+bNo3Dhwvzwww9cd911XocTMOvryRhjgiQuLo4JEybQokULrr32Wo4fP07+/PnJkydPyGOxvp6MMSaTWbFiBfXq1ePJJ5/ko48+AiA8PNyTJHGxLFEYY0wGOnr0KL169aJJkyYcOXKEuXPn8vLLL3sd1kWxRGGMMRlo2rRpTJ8+nQEDBrBlyxY6deqEiHgd1kWxcxTGGHORtm/fzsGDB7n++us5ffo027dvJzIy0uuwzmHnKIwxxgOnTp1i+PDhREZG0qdPH1SVfPnyZbokcbEsURhjTDosWrSImjVrMnLkSDp37szXX3+d5ZuYUmP3URhjzAVaunQpbdq0oWLFiixatIgbb7zR65CCyo4ojDEmAGfOnGHTpk0ANG3alLfeeouNGzdm+yQBliiMMSZN69ato0mTJlx33XX8+eefiAgPPvgg+fPn9zq0kLBEYYwxqTh+/DgDBw6kXr167NmzhylTpnDZZZd5HVbI2TkKY4xJQUxMDDVr1mT//v08+uijvPzyyxQrVszrsDxhicIYY3wcO3aMwoULU6RIEXr27EmrVq1o3Lix12F5ypqejDEGpwO/V199lYiICNauXQvAsGHDcnySADuiMMYYfvzxR3r16kVUVBSdOnXi0ksv9TqkTMWOKIwxOVq/fv24/vrriYmJYf78+cydO5crrrjC67AyFUsUxpgcx7ePu8svv5wnn3ySLVu20KFDBw+jyrysU0BjTI6ybds2evXqxYABA+jYsaPX4YSMdQpojDFpiI2N5bnnniMyMpINGzYQGxvrdUhZRlAThYi0E5HtIrJLRJ5JYXwREflcRDaIyGYR6RHMeIwxOdOSJUuoWbMmL730Et26dWP79u1069bN67CyjKBd9SQiYcAkoDUQDawSkc9UdYtPsT7AFlW9VUQuBbaLyPuq+r9gxWWMyXmio6PJnTs3S5YsoWXLll6Hk+UE8/LYBsAuVd0NICKzgY6Ab6JQIFycvnkLAUeA+CDGZIzJAc6cOcPUqVPJmzcvjzzyCPfffz/dunUjX758XoeWJQWz6akssN9nONp9z9dEoCrwO7AJeEJVE5LPSER6ishqEVkdFxcXrHiNMdnA2rVradSoEX379uXrr78GQEQsSVyEYCaKlJ7gkfwSq7bAeqAMUBuYKCKFz5tIdZqq1lPVenny5Mn4SI0xWd6xY8d44oknqF+/Pvv372fWrFl8/PHHXoeVLQQzUUQDvnetROAcOfjqAXyqjl3Ar0CVIMZkjMmmNmzYwMSJE+nVqxfbtm2jW7du2faJc6EWzESxCqgoIuVFJC/QDfgsWZl9QCsAESkFVAZ2BzEmY0w28uuvv/L2228DzsOEdu3axaRJkyhatKjHkWUvQUsUqhoP9AW+BrYCH6nqZhHpJSK93GIvAk1EZBOwBHhaVQ8FKyZjTPbwv//9j5dffplq1aoxaNAgjh49CkD58uU9jix7sjuzjTFZyrJly+jVqxdbtmzh9ttvZ/z48URERHgdVqZ3MXdmW++xxpgs4+DBg7Rp04ZSpUrx+eefc8stt3gdUo5gXXgYYzI1VWXRokUAXHrppXzxxRds3rzZkkQIWaIwxmRamzdv5oYbbqBNmzZ8//33ALRq1YqCBQt6G1gOY4nCGJPp/PPPPwwdOpTatWuzefNmpk+fTrNmzbwOK8eycxTGmExFVWnRogU///wzDzzwAK+99po9cc5jliiMMZnCgQMHuOyyywgLC2Po0KEUKVKE5s2bex2WwZqejDEeO3PmDBMmTKBy5cpMnjwZgI4dO1qSyEQsURhjPLN69WoaNGjAE088QZMmTbjpppu8DsmkIOBEISJ2mYExJsO8+uqrNGjQgAMHDvDhhx/y5Zdfcs0113gdlklBmolCRJqIyBacbjgQkVoiMjnokRljsh1VJfFRAQ0aNKBPnz5s3bqVLl26WAd+mViaXXiIyEqgM/CZqtZx34tS1RohiO881oWHMVnTL7/8Qu/evalRowavv/661+HkOBfThUdATU+quj/ZW2fSszBjTM5z+vRpXnrpJWrUqMFPP/1kzUtZUCCXx+4XkSaAut2FP47bDGWMMf6sWbOGe++9l23btnHnnXcybtw4ypQp43VY5gIFkih6AeNxHmMaDXwD9A5mUMaY7KFQoUKICAsXLqR9+/Zeh2PSKZBEUVlV7/F9Q0SuA34MTkjGmKwqISGBd955h59++onp06dTuXJloqKiyJXLrsTPygL59N4I8D1jTA4WFRVFs2bNePjhh9m5cycnT54EsCSRDaR6RCEijYEmwKUiMtBnVGEgLNiBGWOyhpMnTzJy5EjGjh1LkSJFeOedd3jggQfsctdsxF/TU16gkFsm3Of9YziXyxpjDKdOneKdd97h/vvv59VXX6VEiRJeh2QyWCD3UVylqntDFE+a7D4KY7wXHR3NhAkTePnllwkLC+PIkSMUL17c67CMH8G+j+IfEXlNRBaKyLeJr/QszBiTtcXHx/Pvf/+bqlWrMnHiRNavXw9gSSKbCyRRvA9sA8oDLwB7gFVBjMkYkwmtXLmSevXqMXDgQJo1a8bmzZupW7eu12GZEAjk8tgSqvqWiDyhqv8F/isi/w12YMaYzCMhIYEePXoQExPDnDlzuP322+1kdQ4SSKKIc/8eEJGbgd+BiOCFZIzJDFSVOXPm0K5dO8LDw/n0008pW7Ys4eHhaU9sspVAmp5eEpEiwCDgSWA60D+oURljPLVz507atm1Lly5dmDZtGgBVqlSxJJFDpXlEoapfuP/GAC0g6c5sY0w2c/r0aUaPHs2oUaPIly8fEydOpFevXl6HZTzm74a7MKALTh9PX6lqlIjcAgwFCgB1QhOiMSZU+vTpw1tvvUW3bt0YO3YspUuX9jokkwmkeh+FiMwArgB+BhoCe4HGwDOqOi9UASZn91EYk7H++usvEhISuPzyy9m5cye7d++mbdu2XodlMtjF3Efhr+mpHhCpqgkikh84BFRQ1T/SsyBjTOaSkJDA9OnTefrpp2nTpg0ffvghFStWpGLFil6HZjIZfyez/6eqCQCqegrYYUnCmOxh48aNXH/99Tz66KPUrl2bF154weuQTCbm74iiiohsdP8X4Bp3WABV1cigR2eMyXBz5syhW7duFCtWjPfee497773X7okwfvlLFFVDFoUxJuiOHTtG4cKFad68OX369GH48OHW9YYJSJqdAmY2djLbmAuzb98++vXrx++//86KFSsIC7OnBOREwe4UMN1EpJ2IbBeRXSLyTCplmovIehHZbF2DGJNx4uLiGDNmDFWrVmXx4sV06dKFrLZjaDKHQLrwSBf3PoxJQGucZ22vEpHPVHWLT5miwGSgnaruE5HLghWPMTnJ3r176dChAxs3buTWW2/ljTfe4KqrrvI6LJNFBZQoRKQAcKWqbr+AeTcAdqnqbnces4GOwBafMncDn6rqPgBV/esC5m+MSUZVEREuv/xySpUqxdy5c+nYsaOdrDYXJc2mJxG5FVgPfOUO1xaRzwKYd1lgv89wtPuer0pAMRH5XkTWiMj9gYVtjPGlqsycOZP69etz4sQJ8uXLxzfffEOnTp0sSZiLFsg5ihE4Rwd/A6jqeqBcANOl9O1M3kCaG6gL3Ay0BZ4TkUrnzUikp4isFpHVcXFxyUcbk6Nt376dVq1acd9995E7d24OHz7sdUgmmwkkUcSrakw65h2N0wVIogicLsqTl/lKVU+q6iFgKVAr+YxUdZqq1lPVenny5ElHKMZkP/Hx8QwfPpzIyEjWrl3LlClTWL58uZ2LMBkukEQRJSJ3A2EiUlFE3gCWBzDdKqCiiJQXkbxANyB5k9V8oKmI5BaRS3D6lLJrX40JQFhYGMuWLaNz585s376dXr16kStXUC9kNDlUIN+qfkB14DTwAU5342k+j0JV44G+wNc4G/+PVHWziPQSkV5uma045z424nQ+OF1Vo9KzIsbkBH/88QcPPvgg+/fvR0RYuHAh77//PqVKlfI6NJONpXnDnYjUUdV1IYonTXbDncmJzpw5w7Rp0xgyZAixsbHMnDmTO++80+uwTBYS7BvuxorINhF5UUSqp2chxpj0W7duHU2aNKF3797Uq1ePTZs2WZIwIZVmolDVFkBz4CAwTUQ2iciwYAdmjHFMnDiRPXv28P7777No0SIqVTrvwkBjguqC+noSkZrAYKCrquYNWlR+WNOTye5UlXnz5lGuXDnq1KnD0aNHAShWrJjHkZmsLKhNTyJSVURGiEgUMBHniqeI9CzMGOPfnj176NChA7fffjvjxo0DnARhScJ4KZAuPN4BZgFtVDX5fRDGmAwQFxfH2LFjeeGFF8iVKxdjxozhiSee8DosY4AAEoWqNgpFIMbkZG+++SbPPPMMnTp1Yvz48Vx55ZVeh2RMklQThYh8pKpdRGQT53a9YU+4MyYDHD58mD179lC3bl0eeeQRKlSoQLt27bwOy5jz+DuiSDzuvSUUgRiTU6gq7733Hk8++STh4eHs2LGDfPnyWZIwmVaqJ7NV9YD7b29V3ev7AnqHJjxjspetW7fSokULunfvTsWKFZk3bx65cwftsTDGZIhAbrhrncJ77TM6EGOyuw0bNlCrVi02btzItGnT+OGHH4iMtBZck/n5O0fxGM6Rw9UistFnVDjwY7ADMya7iI6OJiIigsjISF544QUeeughLrvMHuZoso5Ub7gTkSJAMeBlwPd518dV9UgIYkuR3XBnsorff/+dAQMGsHDhQrZt20bZssmf22VM6ATrhjtV1T1AH+C4zwsRKZ6ehRmTE5w5c4aJEydStWpV5s+fz+DBgylZsqTXYRmTbv7Oon2Ac8XTGpzLY32fWKfA1UGMy5gs6dSpUzRr1oxVq1bRunVrJk+eTIUKFbwOy5iLkmqiUNVb3L/lQxeOMVlTXFwcefLkIX/+/LRo0YKBAwfStWtXe161yRYC6evpOhEp6P5/r4iMFRG7bdQYnHsi5syZQ4UKFVi7di0Ao0ePplu3bpYkTLYRyOWxU4B/RKQWTs+xe4H/BDUqY7KA3bt3c/PNN3PnnXdSokQJewypybYC+WbHq3NpVEdgvKqOx7lE1pgca+zYsVSvXp1ly5Yxbtw4fv75Z2rXru11WMYERSC3hB4XkSHAfUBTEQkD8gQ3LGMytxMnTnDTTTcxfvx4IiKs132TvQXyzOzLgbuBVaq6zD0/0VxV3wtFgMnZfRTGC4cOHeKpp57itttuo0OHDiQkJFhTk8lSgvrgIlX9A3gfKCIitwCnvEoSxoRaQkICb7/9NpUrV2bmzJns2rULwJKEyVECueqpC/AzcCfQBVgpIp2DHZgxXtuyZQvNmzfnoYceolq1aqxfv56BAwd6HZYxIRfIOYpngfqq+heAiFwKLAbmBDMwY7y2evVqNm/ezFtvvUX37t3tKMLkWIGco9ikqjV9hnMBG3zfCyU7R2GCaeHChRw+fJj77rsPVeXo0aMUL2491pisL6jnKICvRORrEekuIt2BBcDC9CzMmMwqOjqazp07c/PNNzNx4kRUFRGxJGEMgZ3Mfgp4E4gEagHTVPXpYAdmTCjEx8czfvx4qlatyoIFC/jXv/7FsmXL7K5qY3z4ex5FRWAMcA2wCXhSVX8LVWDGhMKaNWvo378/7dq1Y9KkSVx9tfV1aUxy/o4o3ga+AO7A6UH2jZBEZEyQxcTE8OmnnwLQsGFDVq5cycKFCy1JGJMKf1c9havq/7n/bxeRtaEIyJhgUVU++ugj+vfvz+HDh9mzZw9lypShQYMGXodmTKbm74giv4jUEZFrReRaoECyYWOyjF9++YX27dvTrVs3ypYty/LlyylTpozXYRmTJfg7ojgAjPUZ/sNnWIGWwQrKmIx0/Phx6tatS0JCAhMmTKB3796EhYV5HZYxWYa/Bxe1CGUgxmS0jRs3EhkZSXh4OG+99RaNGjWy51Ybkw52q6nJdg4ePMgDDzxArVq1WLjQueXnjjvusCRhTDoFNWESF4cAABxsSURBVFGISDsR2S4iu0TkGT/l6ovIGetDylyMhIQEpk+fTuXKlZk1axZDhw6lefPmXodlTJYXSF9P6eI+t2IS0BqIBlaJyGequiWFcqOBr4MVi8kZ7rjjDubNm0ezZs2YMmUK1apV8zokY7KFQHqPFfdZ2c+7w1eKSCDXEzYAdqnqblX9HzAb5yl5yfUDPgH+uoC4jQHg5MmTxMfHA3DXXXcxY8YMvv/+e0sSxmSgQJqeJgONgbvc4eM4RwppKQvs9xmOdt9LIiJlgduAqf5mJCI9RWS1iKyOi4sLYNEmJ/j888+pVq0akydPBqBLly488MAD1v2GMRkskETRUFX7AKcAVPUokDeA6VL6tSbvqnYc8LSqnvE3I1Wdpqr1VLVenjz2FNacbv/+/dx+++106NCB8PBw6tat63VIxmRrgZyjiHPPIygkPY8iIYDpooErfIYjgN+TlakHzHb3AEsCN4lIvKrOC2D+JgeaOXMmvXr1IiEhgVdeeYUBAwaQN28g+y3GmPQKJFFMAOYCl4nIv4DOwLAAplsFVBSR8sBvQDecZ28nUdXyif+LyAzgC0sSJiWJ3X5HRETQvHlz3njjDcqXL5/2hMaYi5ZmolDV90VkDdAKpzmpk6qm+eQgVY0Xkb44VzOFAW+r6mYR6eWO93tewhiAv//+myFDhlCwYEHGjBlD8+bN7ZJXY0IszUQhIlcC/wCf+76nqvvSmlZVF5LsIUepJQhV7Z7W/EzOoarMmjWLgQMHcvDgQQYMGJB0VGGMCa1Amp4W4JyfECA/UB7YDlQPYlwmB/v111/p2bMnixcvpn79+nz55ZfUqVPH67CMybECaXo659nYbs+xjwYtIpPjxcXFsXHjRiZNmsSjjz5qHfgZ47ELvjNbVdeKSP1gBGNyriVLlrBgwQLGjh1LpUqV2Lt3L/nz5/c6LGMMgZ2jGOgzmAu4FjgYtIhMjvLnn38yaNAg3n//fa655hqeffZZSpQoYUnCmEwkkBvuwn1e+XDOWaTUFYcxAUtISODNN9+kSpUqfPTRRzz33HNs2rSJEiVKeB2aMSYZv0cU7o12hVT1qRDFY3KImJgYhg0bRu3atZkyZQpVqlTxOiRjTCpSPaIQkdxu1xr22FOTIU6cOMHYsWM5c+YMxYoVY+XKlXz77beWJIzJ5PwdUfyMkyTWi8hnwMfAycSRqvppkGMz2cj8+fPp168f+/fvp3bt2rRs2ZKrr77a67CMMQEI5BxFceAwzjOybwFudf8ak6a9e/fSsWNHOnXqRNGiRfnxxx9p2dIet25MVuLviOIy94qnKM7ecJcoeS+wxpxHVencuTNbtmzh1VdfpX///ljvv8ZkPf4SRRhQiMC6CzcmyYoVK6hevTrh4eFMmzaN4sWLc9VVV3kdljEmnfwligOqOjJkkZgs78iRIwwZMoRp06bx/PPP88ILL1jXG8ZkA/4ShfW+ZgKiqsycOZNBgwZx5MgRBg0axFNP2RXVxmQX/hJFq5BFYbK0oUOH8sorr9CoUSMWLVpErVq1vA7JGJOBUk0UqnoklIGYrOXUqVOcOHGCkiVL0qNHD6666ip69uxJrlyBXEhnjMlK7FdtLtiiRYuoWbMmjzzyCACVKlWiV69eliSMyabsl20C9scff3D33XfTpk0bRIS+fft6HZIxJgQuuJtxkzN999133HbbbcTGxjJixAiefvpp6+HVmBzCEoXxKy4ujjx58hAZGUnr1q3517/+RaVKlbwOyxgTQtb0ZFJ0/PhxBgwYQNOmTTlz5gwlSpTg448/tiRhTA5kicKcQ1X59NNPqVq1KuPHj6dOnTqcPn3a67CMMR6yRGGSHDp0iFtvvZU77riDkiVLsnz5cqZMmcIll1zidWjGGA9ZojBJwsPD+fPPPxk7diyrV6+mUaNGXodkjMkELFHkcD/88APt27fnxIkT5MuXj5UrVzJgwABy57brHIwxDksUOdThw4d5+OGHadq0KVu2bGH37t0AdtOcMeY8tlXIYVSVGTNmULlyZWbMmMFTTz3Fli1biIyM9Do0Y0wmZe0LOdB7771H5cqVmTp1KjVr1vQ6HGNMJmdHFDlAbGwsw4cPJzo6GhHhk08+YdmyZZYkjDEBsUSRzX399dfUqFGDkSNHMn/+fACKFStm5yKMMQGzrUU29fvvv9O1a1fatWtHnjx5+Pbbb+nTp4/XYRljsiBLFNnUSy+9xPz58xk5ciQbNmygRYsWXodkjMmiRFW9juGCFL+qqh7Zu9XrMDKlNWvWJHXgd/jwYY4ePUqFChW8DssYkwmIyBpVrZeeaYN6RCEi7URku4jsEpFnUhh/j4hsdF/LRcSeoZkOx44d4/HHH6dBgwYMHToUgBIlSliSMMZkiKAlChEJAyYB7YFqwF0iUi1ZsV+BG1Q1EngRmBaseLIjVeXjjz+mSpUqTJw4kccee4yZM2d6HZYxJpsJ5n0UDYBdqrobQERmAx2BLYkFVHW5T/kVQEQQ48l2PvjgA+69917q1KnD/PnzqV+/vtchGWOyoWAmirLAfp/haKChn/IPAV+mNEJEegI9AQqVviaj4suS/ve//7F7926qVKlC586diY2NpXv37tY3kzEmaIJ5jkJSeC/FM+ci0gInUTyd0nhVnaaq9VS1Xp48eTIwxKxl6dKl1K5dmzZt2nDq1Cny5cvHww8/bEnCGBNUwUwU0cAVPsMRwO/JC4lIJDAd6Kiqh4MYT5Z16NAhevTowQ033EBsbCxTp06151UbY0ImmLuiq4CKIlIe+A3oBtztW0BErgQ+Be5T1R1BjCXL2r17N/Xr1+fYsWM888wzPPfcc/YgIWNMSAUtUahqvIj0Bb4GwoC3VXWziPRyx08FngdKAJNFBCA+vdf5ZjfHjh2jcOHClC9fnh49etC9e3dq1KjhdVjGmBzIbrjLZP755x9efPFFpk2bxoYNG4iIsAvBjDEX72JuuLOzoJnIggUL6Nu3L3v27KFHjx4UKFDA65CMMcYSRWYQHx/PXXfdxZw5c6hatSr//e9/adasmddhGWMMYJ0Ceiqx2S937tyUKlWKUaNGsX79eksSxphMxRKFR1atWkXDhg1Zu3YtABMnTmTIkCHkzZvX48iMMeZclihCLCYmhr59+9KwYUOio6M5fNhuHTHGZG6WKEIosQO/KVOm0LdvX7Zt20br1q29DssYY/yyk9khtHXrVsqWLcvnn39OvXp2u4gxJmuw+yiC6PTp07z22mvUqlWLW2+9lbi4OHLlykVYWJjXoRljcphM++CinOy7776jVq1aPPfccyxZsgSAPHnyWJIwxmQ5ligy2F9//cUDDzxAy5YtiYuL48svv2TcuHFeh2WMMelmiSKDffPNN8yaNYtnn32WqKgo2rVr53VIxhhzUewcRQbYtGkT27dvp3Pnzqgqv/76K1dffbXXYRljTBI7R+GRkydPMnjwYOrUqcPgwYOJi4tDRCxJGGOyFbs8Np0+//xz+vbty759+3jooYcYPXo0Ofnpe+Z8cXFxREdHc+rUKa9DMTlI/vz5iYiIyNDtkSWKdIiKiqJDhw5Ur16dZcuWcf3113sdksmEoqOjCQ8Pp1y5crjPWzEmqFSVw4cPEx0dTfny5TNsvtb0FKD4+Hi+//57AGrUqMEXX3zBunXrLEmYVJ06dYoSJUpYkjAhIyKUKFEiw49iLVEEYOXKldSrV49WrVqxc+dOAG6++WZrajJpsiRhQi0Y3zlLFH4cPXqUxx57jMaNG3Po0CE+/vhjKlSo4HVYxhgTUpYoUnH69Gnq1KnDtGnT6N+/P1u3buX222+3PUSTpYSFhVG7dm1q1KjBrbfeyt9//500bvPmzbRs2ZJKlSpRsWJFXnzxRXwvl//yyy+pV68eVatWpUqVKjz55JNerIJf69at4+GHH/Y6jFQtXbqUa6+9lty5czNnzpxUy61Zs4aaNWtSoUIFHn/88aTP4fTp03Tt2pUKFSrQsGFD9uzZA8DBgwdDeo+WJYpkfvvtNwDy5cvHiBEjWL16NWPHjiU8PNzjyIy5cAUKFGD9+vVERUVRvHhxJk2aBEBsbCwdOnTgmWeeYceOHWzYsIHly5czefJkwLlgo2/fvsycOZOtW7cSFRWV4Zd9x8fHX/Q8Ro0aRb9+/UK6zAtx5ZVXMmPGDO6++26/5R577DGmTZvGzp072blzJ1999RUAb731FsWKFWPXrl0MGDCAp59+GoBLL72U0qVL8+OPPwZ9HcCuekpy6tQpRo8ezahRo/joo4/o2LEj3bt39zosk0288Plmtvx+LEPnWa1MYYbfWj3g8o0bN2bjxo0AfPDBB1x33XW0adMGgEsuuYSJEyfSvHlz+vTpw6uvvsqzzz5LlSpVAOcpjL179z5vnidOnKBfv36sXr0aEWH48OHccccdFCpUiBMnTgAwZ84cvvjiC2bMmEH37t0pXrw469ato3bt2sydO5f169dTtGhRACpUqMCPP/5Irly56NWrF/v27QNg3LhxXHfddecs+/jx42zcuJFatWoB8PPPP9O/f39iY2MpUKAA77zzDpUrV2bGjBksWLCAU6dOcfLkST7//HP69evHpk2biI+PZ8SIEXTs2JE9e/Zw3333cfLkScB5mFiTJk0Crt+UlCtXDoBcuVLfJz9w4ADHjh2jcePGANx///3MmzeP9u3bM3/+fEaMGAFA586d6du3L6qKiNCpUyfef//98+olGCxRAEuWLOGxxx5j586d3HXXXTRs2NDrkIzJUGfOnGHJkiU89NBDgNPsVLdu3XPKXHPNNZw4cYJjx44RFRXFoEGD0pzviy++SJEiRdi0aRPgnNdLy44dO1i8eDFhYWEkJCQwd+5cevTowcqVKylXrhylSpXi7rvvZsCAAVx//fXs27ePtm3bsnXruT0yrF69mho1aiQNV6lShaVLl5I7d24WL17M0KFD+eSTTwD46aef2LhxI8WLF2fo0KG0bNmSt99+m7///psGDRpw4403ctlll7Fo0SLy58+ftC1YvXr1efE3bdqU48ePn/f+mDFjuPHGG9Nc/+R+++03IiIikoYjIiKSWjZ+++03rrjiCsBJ1kWKFOHw4cOULFmSevXqMWzYsAteXnrk+ETRv39/xo8fT4UKFfjmm2/sQUImKC5kzz8jxcbGUrt2bfbs2UPdunWTvt+Je6UpuZDzcIsXL2b27NlJw8WKFUtzmjvvvDOpF+WuXbsycuRIevTowezZs+natWvSfLds2ZI0zbFjxzh+/Pg5TcAHDhzg0ksvTRqOiYnhgQceYOfOnYgIcXFxSeNat25N8eLFAac/ts8++4wxY8YATmvCvn37KFOmDH379mX9+vWEhYWxY8eOFONftmxZmut4IVLqRinxM/A37rLLLuP333/P0FhSkyMTRUJCAqpKWFgYDRo04Pnnn2fIkCHkz5/f69CMyVCJ5yhiYmK45ZZbmDRpEo8//jjVq1dn6dKl55TdvXs3hQoVIjw8nOrVq7NmzZqkZp3UpJZwfN9Lfk1/wYIFk/5v3Lgxu3bt4uDBg8ybNy9pDzkhIYGffvqJAgUK+F0333k/99xztGjRgrlz57Jnzx6aN2+e4jJVlU8++YTKlSufM78RI0ZQqlQpNmzYQEJCQqrbg4w+ooiIiCA6OjppODo6mjJlyiSN279/PxEREcTHxxMTE5OU8E6dOuW3fjJSjjuZvWHDBpo0aZJ0Uu/uu+/mhRdesCRhsrUiRYowYcIExowZQ1xcHPfccw8//PADixcvBpwjj8cff5zBgwcD8NRTTzFq1KikveqEhATGjh173nzbtGnDxIkTk4YTm55KlSrF1q1bk5qWUiMi3HbbbQwcOJCqVatSokSJFOe7fv3686atWrUqu3btShqOiYmhbNmyAMyYMSPVZbZt25Y33ngjaW993bp1SdOXLl2aXLly8Z///IczZ86kOP2yZctYv379ea/0JAmA0qVLEx4ezooVK1BV3nvvPTp27AhAhw4dePfddwHnXE/Lli2TkvCOHTvOaXoLphyTKE6cOMGgQYOoW7cuu3fv5vLLL/c6JGNCqk6dOtSqVYvZs2dToEAB5s+fz0svvUTlypWpWbMm9evXp2/fvgBERkYybtw47rrrLqpWrUqNGjU4cODAefMcNmwYR48epUaNGtSqVYvvvvsOgFdeeYVbbrmFli1bUrp0ab9xde3alZkzZyY1OwFMmDCB1atXExkZSbVq1Zg6dep501WpUoWYmJikvfvBgwczZMgQrrvuulQ38uAcecTFxREZGUmNGjV47rnnAOjduzfvvvsujRo1YseOHecchaTXqlWriIiI4OOPP+bRRx+levWzTZC1a9dO+n/KlCk8/PDDVKhQgWuuuYb27dsD8NBDD3H48GEqVKjA2LFjeeWVV5Km+e6777j55psvOsZA5IhuxhcvXkyPHj2Ijo6mZ8+evPLKKwG1pRpzMbZu3UrVqlW9DiNb+/e//014eHimvpciWJo1a8b8+fNT3Jal9N2zbsbTkDdvXooXL86PP/7Im2++aUnCmGziscceI1++fF6HEXIHDx5k4MCBIduWZcsjiri4OMaNG0dMTAwvvfQS4LSx+ruW2ZiMZkcUxit2RJGG5cuXU7duXQYPHpx0Mg383/BiTLBktR0xk/UF4zuXbbaeR44coWfPnlx33XX8/fffzJs3j08++cQShPFM/vz5OXz4sCULEzKJz6PI6Ks4s819FIcPH+aDDz7gySefZPjw4RQqVMjrkEwOl3h9/MGDB70OxeQgiU+4y0hZOlFs376dDz/8kOeff56KFSuyd+/epOuwjfFanjx5MvQpY8Z4JajtMiLSTkS2i8guEXkmhfEiIhPc8RtF5NpA5hsbG8vzzz9PZGQk//73v9m/fz+AJQljjAmCoB1RiEgYMAloDUQDq0TkM1Xd4lOsPVDRfTUEprh/UxV36gQ1a9bkl19+4Z577uH111+nVKlSwVkJY4wxQW16agDsUtXdACIyG+gI+CaKjsB76pztWyEiRUWktKqefwuo6+ShA+QqcjWLFy+mVatWQQzfGGMMBDdRlAX2+wxHc/7RQkplygLnJAoR6Qn0dAdP79y5Myq9/apkMyWBQ14HkUlYXZxldXGW1cVZldMukrJgJoqU+ipOfp1gIGVQ1WnANAARWZ3em0ayG6uLs6wuzrK6OMvq4iwROf/hGgEK5snsaOAKn+EIIHnn6YGUMcYY46FgJopVQEURKS8ieYFuwGfJynwG3O9e/dQIiPF3fsIYY0zoBa3pSVXjRaQv8DUQBrytqptFpJc7fiqwELgJ2AX8A/QIYNbTghRyVmR1cZbVxVlWF2dZXZyV7rrIcp0CGmOMCS3rCMkYY4xfliiMMcb4lWkTRbC6/8iKAqiLe9w62Cgiy0WklhdxhkJadeFTrr6InBGRzqGML5QCqQsRaS4i60Vks4j8N9QxhkoAv5EiIvK5iGxw6yKQ86FZjoi8LSJ/iUhUKuPTt91U1Uz3wjn5/QtwNZAX2ABUS1bmJuBLnHsxGgErvY7bw7poAhRz/2+fk+vCp9y3OBdLdPY6bg+/F0VxekK40h2+zOu4PayLocBo9/9LgSNAXq9jD0JdNAOuBaJSGZ+u7WZmPaJI6v5DVf8HJHb/4Sup+w9VXQEUFRH/T3HPmtKsC1VdrqpH3cEVOPejZEeBfC8A+gGfAH+FMrgQC6Qu7gY+VdV9AKqaXesjkLpQIFxEBCiEkyjiQxtm8KnqUpx1S026tpuZNVGk1rXHhZbJDi50PR/C2WPIjtKsCxEpC9wGTA1hXF4I5HtRCSgmIt+LyBoRuT9k0YVWIHUxEaiKc0PvJuAJVU0ITXiZSrq2m5n1eRQZ1v1HNhDweopIC5xEcX1QI/JOIHUxDnhaVc84O4/ZViB1kRuoC7QCCgA/icgKVd0R7OBCLJC6aAusB1oC1wCLRGSZqh4LdnCZTLq2m5k1UVj3H2cFtJ4iEglMB9qr6uEQxRZqgdRFPWC2myRKAjeJSLyqzgtNiCET6G/kkKqeBE6KyFKgFpDdEkUgddEDeEWdhvpdIvIrUAX4OTQhZhrp2m5m1qYn6/7jrDTrQkSuBD4F7suGe4u+0qwLVS2vquVUtRwwB+idDZMEBPYbmQ80FZHcInIJTu/NW0McZygEUhf7cI6sEJFSOD2p7g5plJlDurabmfKIQoPX/UeWE2BdPA+UACa7e9Lxmg17zAywLnKEQOpCVbeKyFfARiABmK6qKV42mZUF+L14EZghIptwml+eVtVs1/24iMwCmgMlRSQaGA7kgYvbbloXHsYYY/zKrE1PxhhjMglLFMYYY/yyRGGMMcYvSxTGGGP8skRhjDHGL0sUJlNye35d7/Mq56fsiQxY3gwR+dVd1loRaZyOeUwXkWru/0OTjVt+sTG680mslyi3N9SiaZSvLSI3ZcSyTc5ll8eaTElETqhqoYwu62ceM4AvVHWOiLQBxqhq5EXM76JjSmu+IvIusENV/+WnfHegnqr2zehYTM5hRxQmSxCRQiKyxN3b3yQi5/UaKyKlRWSpzx53U/f9NiLykzvtxyKS1gZ8KVDBnXagO68oEenvvldQRBa4zzaIEpGu7vvfi0g9EXkFKODG8b477oT790PfPXz3SOYOEQkTkddEZJU4zwl4NIBq+Qm3QzcRaSDOs0jWuX8ru3cpjwS6urF0dWN/213OupTq0ZjzeN1/ur3sldILOIPTidt6YC5OLwKF3XElce4sTTwiPuH+HQQ86/4fBoS7ZZcCBd33nwaeT2F5M3CfXQHcCazE6VBvE1AQp2vqzUAd4A7g/3ymLeL+/R5n7z0pJp8yiTHeBrzr/p8XpyfPAkBPYJj7fj5gNVA+hThP+Kzfx0A7d7gwkNv9/0bgE/f/7sBEn+lHAfe6/xfF6fepoNeft70y9ytTduFhDBCrqrUTB0QkDzBKRJrhdEdRFigF/OEzzSrgbbfsPFVdLyI3ANWAH93uTfLi7Imn5DURGQYcxOmFtxUwV51O9RCRT4GmwFfAGBEZjdNctewC1utLYIKI5APaAUtVNdZt7oqUs0/kKwJUBH5NNn0BEVkPlAPWAIt8yr8rIhVxegPNk8ry2wAdRORJdzg/cCXZsw8ok0EsUZis4h6cJ5PVVdU4EdmDs5FLoqpL3URyM/AfEXkNOAosUtW7AljGU6o6J3FARG5MqZCq7hCRujh95rwsIt+o6shAVkJVT4nI9zjdXncFZiUuDuinql+nMYtYVa0tIkWAL4A+wAScvoy+U9Xb3BP/36cyvQB3qOr2QOI1Buwchck6igB/uUmiBXBV8gIicpVb5v+At3AeCbkCuE5EEs85XCIilQJc5lKgkztNQZxmo2UiUgb4R1VnAmPc5SQX5x7ZpGQ2TmdsTXE6ssP9+1jiNCJSyV1milQ1BngceNKdpgjwmzu6u0/R4zhNcIm+BvqJe3glInVSW4YxiSxRmKzifaCeiKzGObrYlkKZ5sB6EVmHcx5hvKoexNlwzhKRjTiJo0ogC1TVtTjnLn7GOWcxXVXXATWBn90moGeBl1KYfBqwMfFkdjLf4DzbeLE6j+4E51kiW4C1IhIFvEkaR/xuLBtwutV+Fefo5kec8xeJvgOqJZ7MxjnyyOPGFuUOG+OXXR5rjDHGLzuiMMYY45clCmOMMX5ZojDGGOOXJQpjjDF+WaIwxhjjlyUKY4wxflmiMMYY49f/A2fVORsqkkA/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred = model.predict(x_test)\n",
    "plot_roc(pred,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiclass Classification Error Metrics\n",
    "\n",
    "If you want to predict more than one outcome, you will need more than one output neuron.  Because a single neuron can predict two outcomes, a neural network with two output neurons is somewhat rare.  If there are three or more outcomes, there will be three or more output neurons. The following sections will examine several metrics for evaluating classification error. The following classification neural network will be used to evaluate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1500 samples, validate on 500 samples\n",
      "Epoch 1/1000\n",
      "1500/1500 - 1s - loss: 1.5542 - accuracy: 0.4193 - val_loss: 1.1433 - val_accuracy: 0.4980\n",
      "Epoch 2/1000\n",
      "1500/1500 - 0s - loss: 1.1488 - accuracy: 0.4760 - val_loss: 1.0920 - val_accuracy: 0.4980\n",
      "Epoch 3/1000\n",
      "1500/1500 - 0s - loss: 1.0847 - accuracy: 0.5027 - val_loss: 1.0355 - val_accuracy: 0.5260\n",
      "Epoch 4/1000\n",
      "1500/1500 - 0s - loss: 0.9356 - accuracy: 0.6240 - val_loss: 0.9037 - val_accuracy: 0.6320\n",
      "Epoch 5/1000\n",
      "1500/1500 - 0s - loss: 0.8340 - accuracy: 0.6693 - val_loss: 0.8218 - val_accuracy: 0.6740\n",
      "Epoch 6/1000\n",
      "1500/1500 - 0s - loss: 0.7877 - accuracy: 0.6800 - val_loss: 0.8232 - val_accuracy: 0.6580\n",
      "Epoch 7/1000\n",
      "1500/1500 - 0s - loss: 0.7555 - accuracy: 0.6980 - val_loss: 0.8054 - val_accuracy: 0.6640\n",
      "Epoch 8/1000\n",
      "1500/1500 - 0s - loss: 0.7544 - accuracy: 0.6827 - val_loss: 0.8253 - val_accuracy: 0.6780\n",
      "Epoch 9/1000\n",
      "1500/1500 - 0s - loss: 0.7387 - accuracy: 0.6973 - val_loss: 0.8564 - val_accuracy: 0.6280\n",
      "Epoch 10/1000\n",
      "1500/1500 - 0s - loss: 0.7405 - accuracy: 0.6867 - val_loss: 0.7672 - val_accuracy: 0.6740\n",
      "Epoch 11/1000\n",
      "1500/1500 - 0s - loss: 0.7015 - accuracy: 0.7007 - val_loss: 0.7592 - val_accuracy: 0.7000\n",
      "Epoch 12/1000\n",
      "1500/1500 - 0s - loss: 0.7129 - accuracy: 0.6833 - val_loss: 0.7828 - val_accuracy: 0.6580\n",
      "Epoch 13/1000\n",
      "1500/1500 - 0s - loss: 0.6940 - accuracy: 0.7073 - val_loss: 0.7529 - val_accuracy: 0.7040\n",
      "Epoch 14/1000\n",
      "1500/1500 - 0s - loss: 0.7011 - accuracy: 0.6980 - val_loss: 0.7475 - val_accuracy: 0.6860\n",
      "Epoch 15/1000\n",
      "1500/1500 - 0s - loss: 0.6844 - accuracy: 0.6940 - val_loss: 0.7666 - val_accuracy: 0.6720\n",
      "Epoch 16/1000\n",
      "1500/1500 - 0s - loss: 0.6720 - accuracy: 0.7047 - val_loss: 0.7458 - val_accuracy: 0.7020\n",
      "Epoch 17/1000\n",
      "1500/1500 - 0s - loss: 0.6699 - accuracy: 0.7153 - val_loss: 0.7630 - val_accuracy: 0.6820\n",
      "Epoch 18/1000\n",
      "1500/1500 - 0s - loss: 0.6722 - accuracy: 0.7127 - val_loss: 0.7374 - val_accuracy: 0.7000\n",
      "Epoch 19/1000\n",
      "1500/1500 - 0s - loss: 0.6722 - accuracy: 0.7100 - val_loss: 0.7517 - val_accuracy: 0.6840\n",
      "Epoch 20/1000\n",
      "1500/1500 - 0s - loss: 0.6666 - accuracy: 0.7113 - val_loss: 0.7515 - val_accuracy: 0.6880\n",
      "Epoch 21/1000\n",
      "1500/1500 - 0s - loss: 0.6581 - accuracy: 0.7193 - val_loss: 0.7997 - val_accuracy: 0.6500\n",
      "Epoch 22/1000\n",
      "1500/1500 - 0s - loss: 0.6487 - accuracy: 0.7147 - val_loss: 0.7422 - val_accuracy: 0.6880\n",
      "Epoch 23/1000\n",
      "Restoring model weights from the end of the best epoch.\n",
      "1500/1500 - 0s - loss: 0.6407 - accuracy: 0.7260 - val_loss: 0.7542 - val_accuracy: 0.6880\n",
      "Epoch 00023: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1fbd556fc88>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification neural network\n",
    "import numpy as np\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(100, input_dim=x.shape[1], activation='relu',\n",
    "                kernel_initializer='random_normal'))\n",
    "model.add(Dense(50,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(25,activation='relu',kernel_initializer='random_normal'))\n",
    "model.add(Dense(y.shape[1],activation='softmax',\n",
    "                kernel_initializer='random_normal'))\n",
    "model.compile(loss='categorical_crossentropy', \n",
    "              optimizer=tensorflow.keras.optimizers.Adam(),\n",
    "              metrics =['accuracy'])\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, \n",
    "                        verbose=1, mode='auto', restore_best_weights=True)\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "          callbacks=[monitor],verbose=2,epochs=1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Accuracy\n",
    " \n",
    "Accuracy is the number of rows where the neural network correctly predicted the target class.  Accuracy is only used for classification, not regression.\n",
    "\n",
    "$ accuracy = \\frac{c}{N} $\n",
    "\n",
    "Where $c$ is the number correct and $N$ is the size of the evaluated set (training or validation). Higher accuracy numbers are desired.\n",
    "\n",
    "As we just saw, by default, Keras will return the percent probability for each class. We can change these prediction probabilities into the actual iris predicted with **argmax**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = model.predict(x_test)\n",
    "pred = np.argmax(pred,axis=1) \n",
    "# raw probabilities to chosen class (highest probability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the actual iris flower predicted, we can calculate the percent accuracy (how many were correctly classified)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score: 0.7\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "y_compare = np.argmax(y_test,axis=1) \n",
    "score = metrics.accuracy_score(y_compare, pred)\n",
    "print(\"Accuracy score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Classification Log Loss\n",
    "\n",
    "Accuracy is like a final exam with no partial credit.  However, neural networks can predict a probability of each of the target classes.  Neural networks will give high probabilities to predictions that are more likely.  Log loss is an error metric that penalizes confidence in wrong answers. Lower log loss values are desired.\n",
    "\n",
    "The following code shows the output of predict_proba:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numpy array of predictions\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.    , 0.1082, 0.7827, 0.1084, 0.0008, 0.    , 0.    ],\n",
       "       [0.    , 0.7501, 0.2489, 0.    , 0.0009, 0.    , 0.    ],\n",
       "       [0.    , 0.7138, 0.284 , 0.0001, 0.0021, 0.0001, 0.    ],\n",
       "       [0.    , 0.3254, 0.6667, 0.0063, 0.0015, 0.    , 0.    ],\n",
       "       [0.    , 0.0529, 0.6657, 0.2809, 0.0004, 0.    , 0.    ]],\n",
       "      dtype=float32)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As percent probability\n",
      "[ 0.     10.819  78.2673 10.8365  0.0765  0.0006  0.    ]\n",
      "Log loss score: 0.737357779353857\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "\n",
    "# Don't display numpy in scientific notation\n",
    "np.set_printoptions(precision=4)\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "# Generate predictions\n",
    "pred = model.predict(x_test)\n",
    "\n",
    "print(\"Numpy array of predictions\")\n",
    "display(pred[0:5])\n",
    "\n",
    "print(\"As percent probability\")\n",
    "print(pred[0]*100)\n",
    "\n",
    "score = metrics.log_loss(y_test, pred)\n",
    "print(\"Log loss score: {}\".format(score))\n",
    "\n",
    "# raw probabilities to chosen class (highest probability)\n",
    "pred = np.argmax(pred,axis=1) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Log loss](https://www.kaggle.com/wiki/LogarithmicLoss) is calculated as follows:\n",
    "\n",
    "$ \\mbox{log loss} = -\\frac{1}{N}\\sum_{i=1}^N {( {y}_i\\log(\\hat{y}_i) + (1 - {y}_i)\\log(1 - \\hat{y}_i))} $\n",
    "\n",
    "\n",
    "You should use this equation only as an objective function for classifications that have two outcomes. The variable y-hat is the neural network’s prediction, and the variable y is the known correct answer.  In this case, y will always be 0 or 1.  The training data have no probabilities. The neural network classifies it either into one class (1) or the other (0).  \n",
    "\n",
    "The variable N represents the number of elements in the training set the number of questions in the test.  We divide by N because this process is customary for an average.  We also begin the equation with a negative because the log function is always negative over the domain 0 to 1.  This negation allows a positive score for the training to minimize.\n",
    "\n",
    "You will notice two terms are separated by the addition (+).  Each contains a log function.  Because y will be either 0 or 1, then one of these two terms will cancel out to 0.  If y is 0, then the first term will reduce to 0.  If y is 1, then the second term will be 0.  \n",
    "\n",
    "If your prediction for the first class of a two-class prediction is y-hat, then your prediction for the second class is 1 minus y-hat.  Essentially, if your prediction for class A is 70% (0.7), then your prediction for class B is 30% (0.3).  Your score will increase by the log of your prediction for the correct class.  If the neural network had predicted 1.0 for class A, and the correct answer was A, your score would increase by log (1), which is 0. For log loss, we seek a low score, so a correct answer results in 0.  Some of these log values for a neural network's probability estimate for the correct class:\n",
    "\n",
    "* -log(1.0) = 0\n",
    "* -log(0.95) = 0.02\n",
    "* -log(0.9) = 0.05\n",
    "* -log(0.8) = 0.1\n",
    "* -log(0.5) = 0.3\n",
    "* -log(0.1) = 1\n",
    "* -log(0.01) = 2\n",
    "* -log(1.0e-12) = 12\n",
    "* -log(0.0) = negative infinity\n",
    "\n",
    "As you can see, giving a low confidence to the correct answer affects the score the most.  Because log (0) is negative infinity, we typically impose a minimum value.  Of course, the above log values are for a single training set element.  We will average the log values for the entire training set.\n",
    "\n",
    "The log function is useful to penalizing wrong answers.  The following code demonstrates the utility of the log function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\ipykernel_launcher.py:12: RuntimeWarning: divide by zero encountered in log\n",
      "  if sys.path[0] == '':\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAE0CAYAAAASSJRcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5icd1338c93z+fdZM/JZpM0TZrm0JZ0E3oAuimltghUQHgqUrEPUAFRq4KiRRAVlIqKXI+P2kvwQQUiWk4XlFKKpEXbtE3SQxLSNOc02U32kOz5PPN9/phJ2Cab7D07O3vPzL5f1zXXHO77nvnOt9M7n/zym9+YuwsAAADA7MoJuwAAAAAgGxG0AQAAgBQgaAMAAAApQNAGAAAAUoCgDQAAAKQAQRsAAABIAYI2AKQ5MztiZrfMwvPcambfCrjv02a2NtnXBID5jKANAPPHZyT9RcB9PyfpT1JYCwBkPYI2AMwDZrZRUqW7bwt4yHckbTazxhSWBQBZjaANABnCzArN7PNm1ha/fN7MCidt/z0za49ve5+ZuZldHt98u6THJu17g5l1mdmS+P2rzazHzFZLkruPSNoh6da5e4cAkF0I2gCQOe6TdJ2kayRdLWmTpI9LkpndJul3JN0i6XJJN5137HpJ+87ecfcnJP2jpC+bWbGkf5X0cXd/cdIxe+OvAwCYAYI2AGSOX5b0J+7e4e6dkj4l6a74tndK+md33+PuQ/Ftk1VJ6j/vsT+WVCnpaUltkv7uvO398eMAADNA0AaAzLFI0tFJ94/GHzu77eVJ2ybflqQzksonP+Du45L+n6R1kv7K3f28Y8ol9SRXMgDMXwRtAMgcbZKWTrrfHH9MktolNU3atuS8Y1+QtGryA2a2WNInJf2zpL+aPN877kpJzydZMwDMWwRtAMgcX5P0cTOrNbMaSZ+Q9G/xbV+XdLeZXWlmJfFtkz2kSfO2zcwUG83+oqT3KhbU/3TS9kJJ10r6YWreCgBkP4I2AGSOP5O0XbHR6V2SdsYfk7t/X9IXJP1Y0gFJT8aPGY1v3ymp18xeHX/8NyXVS/qj+JSRuxUL6q+Nb3+LpK3ufnbEHACQILtwSh4AINOZ2ZWSdksqdPeJ+GO3SvqQu/9CgOOfkvRed9+d2koBIHsRtAEgS5jZWyV9T1KppC9LigYJ1QCA1GDqCABkj1+T1CnpoKSIpA+GWw4AzG+MaAMAAAApwIg2AAAAkAIEbQAAACAF8sIuIBE1NTW+bNmy0F5/cHBQpaWlob1+pqN/M0fvkkP/kkP/Zo7eJYf+JYf+JWfHjh1d7l6bzHNkVNBetmyZtm/fHtrrb926Va2traG9fqajfzNH75JD/5JD/2aO3iWH/iWH/iXHzI4m+xxMHQEAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAgRtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSINSgbWa3mdk+MztgZh8LsxYAAABgNoUWtM0sV9LfSbpd0hpJv2Rma8KqBwAAAJhNYY5ob5J0wN0PufuYpC2S7gixHgDAecYmohqPethlAEBGygvxtRdLennS/eOSXh1SLQAwL0Sjrp7hcXUPjKprYEzdg6PqHhiL3R+MXXcPjKk7frtvZELvvrJAbwi7cADIQOYezkiFmb1D0s+5+/vi9++StMndf+O8/e6RdI8k1dfXX7tly5Y5r/WsgYEBlZWVhfb6mY7+zRy9S04298/dNRKR+sdcfaOu/vHYdd9Y7NIfv449FttvqrO+SSorkCoKTBUFpvL4paLAtKJ0TOsas7N/qZbNn725QP+SQ/+Ss3nz5h3u3pLMc4Q5on1c0pJJ95sktZ2/k7s/IOkBSWppafHW1tY5KW4qW7duVZivn+no38zRu+RkWv/cXX3DE+ocGFFH/6g6z14Gfjb6HBtxHlPXwKhGJ6JTPk9ZYZ6qywpVXVqgxfWFqikrUHVpoarLClRdVqia0th1dVmBFpQUKDfHpnyeTOtfOqF3yaF/yaF/4QszaD8jaaWZLZd0QtKdkt4VYj0AkFIj45Fzgbmzf/SVITr+eFf89ljkwvCcn2uqiQfj6tJCXV5XFrs/KTCfu11aoKL83BDeJQDgrNCCtrtPmNmHJf1AUq6kL7n7nrDqAYCZiEZdp4fG1NE3qo7+kVcE6ckBurN/VP0jExccbyZVlxaopqxQteWFWlFbqtryQtXG79eWF6quvFC1ZUWqKM6T2dSjzgCA9BPmiLbc/SFJD4VZAwBMxd3VPzqhjr4Rnewd1am+EZ3qH9Gp3hGd6hs9d7ujf1QTU6zKUVaYdy4wX9lQodetLLwgQNeWx0ae83L57TAAyEahBm0ACMPZKRwn+0Z0qm9EJ+OB+fzbQ2ORC44tL8pTQ0WR6iuKdN2K6nO368oLVVcRG3muKS9QSQGnVwCY7/iTAEBWGR6LqK13WO09I2rrHdbJ3hHt2Duqfz70dCxI942oZ2j8guMK8nLioblQaxdV6ObVdaqvKFR9PEg3VBSprqKQAA0ACIw/MQBkjNGJiE72jqitZ0TtvcNq741f94yoLX57qhBdUSAtrR1T04IStSxboPryItVX/ixA11cUqrI4n/nPAIBZRdAGkBbcXZ0Dozp+ZljHzwyrrWdY7T3DauuNTedo7x1W18DYBcdVleSrsbJYiyqLtKG5SouqitVYWRR7rCoWprf9z0/U2vqaEN4VAGA+I2gDmBPRqKujf1THzwzpRM9wPFAP6fiZYZ04M6zjPcMaO2896PLCPDVWxULz2kUVaqwsVmNVkRbFrxsri5jKAQBIW/wJBWBWRKKuk30jsdD8igAdu93eM3LB2tDVpQVqWlCs1Y3lumVNvRZXFatpQbGaFpSosapIFUX5Ib0bAACSR9AGENjg6ISOnR6KXbqHdPT0oI6dHtax7kEdPzN8wTJ3teWFalpQrPWLK3X7ukYtXhAL0ksWFGtRVTGj0QCArMafcgDOOTtP+lh3LEwfjV+fvd01MPqK/SuK8rS0ulRrF1fq9vWNWrKgJD4iHQvS/DIhAGA+I2gD84y7q7N/VIe6BnV40uXleKCevHa0mbSoslhLFhbr9avr1FxdouaFJVpaXaKlC0tVWcLUDgAALoagDWSpvpFxHe6MheifheoBHe4c1OCkMF2Ql6Ol8fB8w4oaLY2H6ebq2Oh0YR6j0gAAzARBG8hg45GojnYP6kDHpCAdD9WTl8LLMalpQYmW15SqZelCXVZbquU1sUtjZbFyc1g/GgCA2UbQBjLAWMT107Y+7e/o18GOAe2PX450Db7iC4i15YVaXlOqW66sPxekL6st1ZKFJYxMAwAwxwjaQBoZGJ04F6QPdAzoQEe/9ncM6Fj3kPyHP5EUG51eVl2qy+vKdOuaeq2sL9OK2jItrylVOcvhAQCQNgjaQAjGI1Ed6hzUiyf79OLJfr3Y3qeXTg3oRM/wuX3yc02X1ZRp3eJKXVM1rje8ep1W1pVrWQ2j0wAAZAKCNpBC7rFfQ9zbHgvU+072a297nw52Dmg8EpvykZ9rWlFbpo3LFuhd9c26vK5MK+vK1LywRHm5OZKkrVu3qvWqRWG+FQAAkCCCNjBLxiai2t/Rrz0n+vTT9r5zo9U9Q+Pn9mmsLNIVDeVqvaJOVzaW64qGcl1WU6aCvJwQKwcAAKlA0AZmYGQ8on0n+7W7rVe7T/Rq94k+7TvZf+4nxksKcrWqvly3r2vQ6oYKXdFQrtUN5aoqKQi5cgAAMFcI2sA0BkcntLe9Lxao22LX+zsGFImv9lFZnK91iyt0943LtHZxpdYtqtCy6lLlsGQeAADzGkEbmCQSde3v6Ndzx3r03Muxy0un+nV2Bb3q0gKtW1yp119Zp/WLK7V2UaWaFhTLjFANAABeiaCNee1U34iePReqz2jX8d5zv5pYWZyvq5dU6dY19VrfVKX1iytVX1FIqAYAAIEQtDFvTESi+ml7n545ckY7jp7Ws8d61N47Iim28seaxgq9/domXbOkStcsqdLymlJCNQAAmDGCNrLW4OiEnnu5R88cOa3tR85o57EzGoqPVjctKNbGZQtjobq5SmsaK1SUz9rUAABg9hC0kTXODI7pqcPdeubIGW0/clq72/oUibrMpCsbKvSOa5u0cflCtSxdqIbKorDLBQAAWY6gjYzVPzKupw+f1hMHu/XEwW7tbe+TJBXm5ehVzVX6UOsKtSxbqFc1V6mCnyYHAABzjKCNjDE8FtGOo2f0xMEuPXGwW7tO9CoSdRXm5ahl2QJ95NZVun5FtdYvruIHYAAAQOgI2khb7q59p/r12L5OPfZSp7YfOaOxSFR5Oaarl8RGrK9fUa0NzQuYXw0AANIOQRtppXd4XP9zoOtcuD7ZF1sVZHVDud5zw1LdcHmNNi5bqLJCProAACC9kVYQKnfXS6cG9OjeU9q6r0M7j/UoEnWVF+bpNStr1HpFrV63qlaNlcVhlwoAAJAQgjbmXCTq2nH0jB7Zc1I/3HtKR7uHJElrF1XoAzddpptW1elVzVXKz2WeNQAAyFwEbcyJ4bGInu2Y0Pf+43n96MUOnR4cU0Fujq5fUa33v/YyvWFNveorWHIPAABkD4I2UmZkPKKt+zr13Rfa9KO9HRoej6i86KRuXl2nN6yp102ralXOsnsAACBLEbQxq8YmovrvA5367vPteuSnpzQwOqGFpQV664bFWhzp0Pt/YTNL7wEAgHmBoI2kubuePnxa33z2hL6/+6R6h8dVUZSnN65v0JuuWqQbVlQrLzdHW7duJWQDAIB5g6CNGXv59JC+sfOEHtx5XMdOD6mkIFc/t7ZBb7qqUa9dWUuoBgAA8xpBGwkZGpvQ93ed1H/uOK4nD3VLkm5YUa17b1mp29Y1qKSAjxQAAIBE0EZABzr69W/bjunBHcfVPzqh5oUl+p03rNJbX7VYSxaWhF0eAABA2iFo46LGJqJ65Kcn9W/bjmrbodMqyM3R7esb9K5Nzdq0fKHMLOwSAQAA0hZBGxfoHhjVvzx5VF99+pg6+0fVtKBYv3/bar2jpUk1ZYVhlwcAAJARCNo452DngL7434f14I7jGp2I6ubVdbrruqV63apa5eYweg0AAJAIgjb0zJHTeuDxQ3p07ynl5+bo7Rua9N7XLNfldWVhlwYAAJCxCNrz2JMHu/X5R1/SU4dPa0FJvn7j5pX6leuXMj0EAABgFhC05xl315OHuvX5R/fr6cOnVVdeqE++eY3u3Nis4oLcsMsDAADIGgTteWTH0dP67MP7zgXsP37zGt25qVlF+QRsAACA2RZK0Dazv5T0Zkljkg5Kutvde8KoZT443DWo+x9+Ud/ffVK15YX61FvW6n9tXELABgAASKGwRrR/KOkP3H3CzD4r6Q8k/X5ItWSt7oFRfeFH+/WVp46pIC9Hv33LKr3/dcv59UYAAIA5EEricvdHJt3dJukXw6gjW0Wirq88dVR/+YN9GhqL6M6NS3TvLatUW86XHAEAAOZKOgxt/m9J/x52Edni2WNn9Eff3q3dJ/p04+XV+tRb1uryuvKwywIAAJh3zN1T88Rmj0pqmGLTfe7+7fg+90lqkfQ2v0ghZnaPpHskqb6+/totW7akpN4gBgYGVFaWnmtLD427vv7SmB57eUKVhaZfWl2gTQ25afUz6encv3RH75JD/5JD/2aO3iWH/iWH/iVn8+bNO9y9JZnnSFnQnvaFzd4j6QOSXu/uQ0GOaWlp8e3bt6e2sEvYunWrWltbQ3v9i3nspU597MEXdKpvRHffuFz33rJS5UX5YZd1gXTtXyagd8mhf8mhfzNH75JD/5JD/5JjZkkH7bBWHblNsS8/3hQ0ZONC/SPj+vT39mrLMy/r8royfeNDN+qaJVVhlwUAAACFN0f7/0gqlPTD+NSGbe7+gZBqyUg7jp7Rb37tWbX3DusDN63QvbesZLk+AACANBLWqiOXh/G62SAadf3j44f0uUf2aVFVkf7zgzdoQ/OCsMsCAADAedJh1REE1D0wqt/++vN6/KVO/fxVjfrzt61XRRrOxQYAAABBO2PsaevV+7+8XV2DY/r0W9fpXZua02pFEQAAALwSQTsDPLSrXb/79edVVZKvBz9wg9Y3VYZdEgAAAKZB0E5j7q6//dF+ff7R/drQXKV/uOta1ZUXhV0WAAAAAiBop6lI1PXxb+3W154+prdvaNJn3rZOhXmsKgIAAJApCNppaGQ8onu3PKeH95zUr29eoY/cegXzsQEAADIMQTvNDI9F9N4vP6MnDnbrj960Ru99zfKwSwIAAMAMELTTyMh4LGRvO9Stv37n1XrbhqawSwIAAMAMEbTTxMh4RO//l+168lC3/uodhGwAAIBMlxN2AZAmIlF9+Ks79ZP9Xfrs268iZAMAAGQBgnbI3F2f+M4ePbq3Q396x1q9s2VJ2CUBAABgFhC0Q/b3jx3UV586pg+2rtBd1y8LuxwAAADMEoJ2iB7a1a77H96nO65ZpI/eekXY5QAAAGAWEbRDcqCjXx/5j+e1oblK9//iVcrJYZ1sAACAbELQDkH/yLju+dcdKinI09+/+1p+8REAACALsbzfHHN3fewbu3S0e0hfed+rVV9RFHZJAAAASAFGtOfYN589oe+90K7fvXWVrrusOuxyAAAAkCIE7Tl0omdYn/z2Hm1ctkC/9roVYZcDAACAFCJozxF310e+/ryi7vrrd16jXL78CAAAkNUI2nPkwZ0n9OShbt3382u0ZGFJ2OUAAAAgxQjac6BnaEyfeWivNjRX6c6N/PIjAADAfEDQngOffXifeofH9em3rme9bAAAgHmCoJ1ie9v7tOWZY3rP9ct0ZWNF2OUAAABgjhC0U+z+h19UeWGefuv1K8MuBQAAAHOIoJ1C2w5168f7OvWhzZersiQ/7HIAAAAwhwjaKeLuuv/hF9VQUaRfvWFZ2OUAAABgjk0btM3sw2a2YC6KySbbDp3WzmM9+vXNK1SUnxt2OQAAAJhjQUa0GyQ9Y2ZfN7PbzIxlMwL4+8cOqqasQO9oYTk/AACA+WjaoO3uH5e0UtIXJf2qpP1m9hkz4zfEL2JPW68ef6lTd9+4nNFsAACAeSrQHG13d0kn45cJSQsk/aeZ3Z/C2jLWP/3ksMoK8/Tu65aGXQoAAABCkjfdDmb2m5LeI6lL0j9J+qi7j5tZjqT9kn4vtSVmljODY/rernbduXGJKotZaQQAAGC+mjZoS6qR9DZ3Pzr5QXePmtmbUlNW5npw53GNTUT1rlc3h10KAAAAQjRt0Hb3T1xi297ZLSezubu++vQxbWiu0uoGfgUSAABgPmMd7Vm0/egZHeoc1LtezdxsAACA+Y6gPYu+81ybivJzdPu6hrBLAQAAQMgI2rNkIhLVQ7va9for61VaGGTqOwAAALIZQXuWPHmoW92DY3rzVYvCLgUAAABpgKA9S777fLvKCvPUekVt2KUAAAAgDRC0Z0E06vrRix1qvaKWX4IEAACAJIL2rNjT1qeugVHdvLou7FIAAACQJgjas+DH+zpkJt20imkjAAAAiCFoz4If7+vQ1U1Vqi4rDLsUAAAApAmCdpJ6h8b13Ms9jGYDAADgFUIN2mb2ETNzM6sJs45kbD96Wu7S9Suqwy4FAAAAaSS0oG1mSyS9QdKxsGqYDU8fPq2C3Bxds6Qq7FIAAACQRsIc0f4bSb8nyUOsIWlPHzmtq5oqWdYPAAAArxBK0Dazt0g64e7Ph/H6s2VobEK7jvdq0/KFYZcCAACANGPuqRlQNrNHJTVMsek+SX8o6VZ37zWzI5Ja3L3rIs9zj6R7JKm+vv7aLVu2pKTeIAYGBlRWVnbu/t7uiD77zIh++9pCXV2bF1pdmeL8/iE4epcc+pcc+jdz9C459C859C85mzdv3uHuLck8R8rSobvfMtXjZrZe0nJJz5uZJDVJ2mlmm9z95BTP84CkBySppaXFW1tbU1XytLZu3arJr7//8UOS9urdt7+Wpf0COL9/CI7eJYf+JYf+zRy9Sw79Sw79C9+cD8O6+y5J535CcboR7XS260SvFlUWEbIBAABwAdbRTsLutl6tW1wZdhkAAABIQ6EHbXdflomj2QOjEzrcNUjQBgAAwJRCD9qZ6qdtfXKX1hO0AQAAMAWC9gztO9UvSVrdWB5yJQAAAEhHBO0ZOtgxoNKCXDVUFIVdCgAAANIQQXuGDnYOaEVdmeJLFAIAAACvQNCeoYMdA1pRyyLwAAAAmBpBewYGRyfU1juiFbWlYZcCAACANEXQnoHDXYOSxIg2AAAALoqgPQNng/ayGka0AQAAMDWC9gwcPzMsSWpaUBxyJQAAAEhXBO0ZONEzpMrifJUX5YddCgAAANIUQXsGTpwZ1uIqRrMBAABwcQTtGTjRM6zFTBsBAADAJRC0E+TujGgDAABgWgTtBPUOj2twLMIXIQEAAHBJBO0EtfWMSJIWMaINAACASyBoJ6hrYFSSVFteGHIlAAAASGcE7QR19seCdk0ZQRsAAAAXR9BO0NkR7ZqygpArAQAAQDojaCeoa2BURfk5KivMC7sUAAAApDGCdoK6BsZUU1YoMwu7FAAAAKQxgnaCugZGmZ8NAACAaRG0E9TZT9AGAADA9AjaCYpNHeGLkAAAALg0gnYC3F19w+OqLMkPuxQAAACkOYJ2Asaj0lgkqspigjYAAAAujaCdgMFxlySCNgAAAKZF0E7A0ETsuqKIoA0AAIBLI2gnYIgRbQAAAARE0E7A2akjFQRtAAAATIOgnYCzU0cY0QYAAMB0CNoJODt1pKIoL+RKAAAAkO4I2glg6ggAAACCImgnYHjCVZyfq/xc2gYAAIBLIzEmYDQilRTkhl0GAAAAMgBBOwGjEamYoA0AAIAACNoJGI04I9oAAAAIhKCdgNiINiuOAAAAYHoE7QSMRVwl+YxoAwAAYHoE7QQwRxsAAABBEbQTMBpxgjYAAAACIWgnYCwipo4AAAAgEIJ2Alh1BAAAAEERtBMwOsGqIwAAAAgmtKBtZr9hZvvMbI+Z3R9WHUFNRKKacKmYqSMAAAAIIJThWTPbLOkOSVe5+6iZ1YVRRyKGxyOSpOIC/hEAAAAA0wsrNX5Q0l+4+6gkuXtHSHUENjYRlSQV5jGiDQAAgOmFFbRXSXqtmT1lZo+Z2caQ6ghsPOKSpPxcRrQBAAAwPXP31Dyx2aOSGqbYdJ+kT0v6L0m/JWmjpH+XdJlPUYyZ3SPpHkmqr6+/dsuWLSmpdzqdQ1F99PFhvW99gV6zOD+UGjLdwMCAysrKwi4jI9G75NC/5NC/maN3yaF/yaF/ydm8efMOd29J5jlSNkfb3W+52DYz+6Ckb8SD9dNmFpVUI6lziud5QNIDktTS0uKtra2pKXgaBzoGpMcf0/q1a9R6zeJQash0W7duVVj//TIdvUsO/UsO/Zs5epcc+pcc+he+sOZBfEvSzZJkZqskFUjqCqmWQMYjsTnaBUwdAQAAQABhLQr9JUlfMrPdksYkvWeqaSPp5FzQziNoAwAAYHqhBG13H5P07jBee6bOrjrClyEBAAAQBKkxoLEIQRsAAADBkRoDOru8H1NHAAAAEASpMaDxCb4MCQAAgOBIjQGdmzqSZyFXAgAAgExA0A6I5f0AAACQCFJjQKw6AgAAgESQGgMaYx1tAAAAJIDUGBBfhgQAAEAiSI0BnV3eLy+XL0MCAABgegTtgCai8aCdQ8sAAAAwPVJjQFGPBW1yNgAAAIIgNgYUiY9o5xpTRwAAADA9gnZA54J2DkEbAAAA0yNoBxR1l0kyRrQBAAAQAEE7oEjUxWA2AAAAgiJoBxRxF4PZAAAACIqgHVCUEW0AAAAkgKAdUCRKswAAABAc2TGgqDOiDQAAgOAI2gHxZUgAAAAkgqAdUIQRbQAAACSAoB1Q7MuQJG0AAAAEQ9AOiKkjAAAASARBO6BI/JchAQAAgCAI2gGxjjYAAAASQdAOKOIiaAMAACAwgnZAjGgDAAAgEQTtgPgyJAAAABJB0A4oto42SRsAAADBELQDYuoIAAAAEkHQDijiTrMAAAAQGNkxoEjUxcwRAAAABEXQDshZ3g8AAAAJIGgHFOWXIQEAAJAAgnZA7mLqCAAAAAIjaAfk8rBLAAAAQAYhaCeAAW0AAAAERdAOyBnQBgAAQAII2gGRswEAAJAIgnYC+DIkAAAAgiJoB8WQNgAAABJA0A7IxTraAAAACI6gHRBfhgQAAEAiQgnaZnaNmW0zs+fMbLuZbQqjjkQxRxsAAABBhTWifb+kT7n7NZI+Eb+f1hjQBgAAQCLCCtouqSJ+u1JSW0h1BObOHG0AAAAElxfS694r6Qdm9jnFwv4NIdUBAAAApIR5ir7lZ2aPSmqYYtN9kl4v6TF3f9DM3inpHne/5SLPc4+ke+J3r5C0LxX1BlQjqSvE18909G/m6F1y6F9y6N/M0bvk0L/k0L/kXOHu5ck8QcqC9iVf1KxXUpW7u5mZpF53r5juuLCZ2XZ3bwm7jkxF/2aO3iWH/iWH/s0cvUsO/UsO/UvObPQvrDnabZJuit++WdL+kOoAAAAAUiKsOdrvl/S3ZpYnaUQ/mxoCAAAAZIVQgra7/7eka8N47SQ9EHYBGY7+zRy9Sw79Sw79mzl6lxz6lxz6l5yk+xfKHG0AAAAg2/ET7AAAAEAKELQlmdltZrbPzA6Y2cem2G5m9oX49hfMbEPQY+eDAP375XjfXjCzJ8zs6knbjpjZLjN7zsy2z23l6SFA/1rNrDfeo+fM7BNBj812AXr30Ul9221mETNbGN/GZ8/sS2bWYWa7L7Kdc99FBOgd571LCNA/znuXEKB/nPsuwsyWmNmPzWyvme0xs9+aYp/ZO/e5+7y+SMqVdFDSZXTsp54AAARrSURBVJIKJD0vac15+7xR0vclmaTrJD0V9NhsvwTs3w2SFsRv3362f/H7RyTVhP0+0rx/rZK+O5Njs/mS6PuX9GZJ/zXp/rz+7MV78DpJGyTtvsh2zn0z7x3nveT6x3kvif6dty/nvlf2o1HShvjtckkvpTL3MaItbZJ0wN0PufuYpC2S7jhvnzsk/YvHbJNUZWaNAY/NdtP2wN2fcPcz8bvbJDXNcY3pLJnP0Hz//CX6/n9J0tfmpLIM4e6PSzp9iV04913EdL3jvHdpAT57FzPvP3tSwv3j3DeJu7e7+8747X5JeyUtPm+3WTv3EbRjzX150v3jurDhF9snyLHZLtEevFexvyWe5ZIeMbMdFvsV0PkmaP+uN7Pnzez7ZrY2wWOzVeD3b2Ylkm6T9OCkh+f7Zy8Izn2zg/PezHDeSxLnvkszs2WSXiXpqfM2zdq5L6x1tNOJTfHY+UuxXGyfIMdmu8A9MLPNiv2B85pJD9/o7m1mVifph2b2Yvxv6vNFkP7tlLTU3QfM7I2SviVpZcBjs1ki7//Nkv7H3SePAM33z14QnPuSxHlvxjjvzQ7OfRdhZmWK/QXkXnfvO3/zFIfM6NzHiHbsbyNLJt1vUuyXK4PsE+TYbBeoB2Z2laR/knSHu3effdzd2+LXHZK+qdg/y8wn0/bP3fvcfSB++yFJ+WZWE+TYLJfI+79T5/3TKZ+9QDj3JYHz3sxx3ps1nPumYGb5ioXsr7j7N6bYZdbOfQRt6RlJK81suZkVKPah/M55+3xH0q/Ev4V6naRed28PeGy2m7YHZtYs6RuS7nL3lyY9Xmpm5WdvS7pV0pTfoM5iQfrXYGYWv71Jsf9vu4Mcm+UCvX8zq5R0k6RvT3qMz14wnPtmiPNecjjvJY9z39Tin6svStrr7n99kd1m7dw376eOuPuEmX1Y0g8U+zbpl9x9j5l9IL79HyQ9pNg3UA9IGpJ096WODeFthCZg/z4hqVrS/42fNyfcvUVSvaRvxh/Lk/RVd384hLcRmoD9+0VJHzSzCUnDku702Nef5/XnL2DvJOmtkh5x98FJh8/7z54kmdnXFFvdocbMjkv6pKR8iXPfdAL0jvPeJQToH+e9SwjQP4lz38XcKOkuSbvM7Ln4Y38oqVma/XMfvwwJAAAApABTRwAAAIAUIGgDAAAAKUDQBgAAAFKAoA0AAACkAEEbAAAASAGCNgAAAJACBG0AAAAgBQjaAJCFzGyjmb1gZkXxX4PbY2brwq4LAOYTfrAGALKUmf2ZpCJJxZKOu/ufh1wSAMwrBG0AyFJmViDpGUkjkm5w90jIJQHAvMLUEQDIXgsllUkqV2xkGwAwhxjRBoAsZWbfkbRF0nJJje7+4ZBLAoB5JS/sAgAAs8/MfkXShLt/1cxyJT1hZje7+3+FXRsAzBeMaAMAAAApwBxtAAAAIAUI2gAAAEAKELQBAACAFCBoAwAAAClA0AYAAABSgKANAAAApABBGwAAAEgBgjYAAACQAv8f8v90kBNbuZkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib.pyplot import figure, show\n",
    "from numpy import arange, sin, pi\n",
    "\n",
    "#t = arange(1e-5, 5.0, 0.00001)\n",
    "#t = arange(1.0, 5.0, 0.00001) # computer scientists\n",
    "t = arange(0.0, 1.0, 0.00001)  # data     scientists\n",
    "\n",
    "fig = figure(1,figsize=(12, 10))\n",
    "\n",
    "ax1 = fig.add_subplot(211)\n",
    "ax1.plot(t, np.log(t))\n",
    "ax1.grid(True)\n",
    "ax1.set_ylim((-8, 1.5))\n",
    "ax1.set_xlim((-0.1, 2))\n",
    "ax1.set_xlabel('x')\n",
    "ax1.set_ylabel('y')\n",
    "ax1.set_title('log(x)')\n",
    "\n",
    "show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[ 21   1   0   0   0   0   0]\n",
      " [  4 193  52   0   0   0   0]\n",
      " [  0  52 134   1   0   0   0]\n",
      " [  0   0  12   2   0   0   0]\n",
      " [  0   8   0   0   0   0   0]\n",
      " [ 10   7   0   0   0   0   0]\n",
      " [  3   0   0   0   0   0   0]]\n",
      "Normalized confusion matrix\n",
      "[[0.95 0.05 0.   0.   0.   0.   0.  ]\n",
      " [0.02 0.78 0.21 0.   0.   0.   0.  ]\n",
      " [0.   0.28 0.72 0.01 0.   0.   0.  ]\n",
      " [0.   0.   0.86 0.14 0.   0.   0.  ]\n",
      " [0.   1.   0.   0.   0.   0.   0.  ]\n",
      " [0.59 0.41 0.   0.   0.   0.   0.  ]\n",
      " [1.   0.   0.   0.   0.   0.   0.  ]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAEmCAYAAADr3bIaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAf7klEQVR4nO3debwcVZn/8c83CfumIQECJAT4YRBQIokgIJu4gCKLymaGQcUJERDHZUYURhj8MfpScUMRgyC4sYkICsMyjIg4MJCEEFllSwSJkIUdDCQ880edhuZyb3fdvl1dXX2/b1/1Snd19TnPDfG555w655QiAjMza2xE2QGYmVWBk6WZWQ5OlmZmOThZmpnl4GRpZpaDk6WZWQ5OlsOIpNUk/UbSk5IuGkI50yRd3c7YyiJpF0n3lB2HdT95nmX3kfRh4DPAlsDTwFzglIi4YYjlHgZ8EtgpIpYPOdAuJymALSLivrJjsepzy7LLSPoM8G3gP4D1gQnA6cB+bSh+E+DPwyFR5iFpVNkxWIVEhI8uOYB1gGeAAxtcswpZMn0kHd8GVkmf7Q48DHwWeAxYCHw0ffbvwAvAi6mOI4CTgJ/VlT0RCGBUev8R4AGy1u2DwLS68zfUfW8n4BbgyfTnTnWfXQd8GfhjKudqYMwAP1st/n+ti39/4L3An4GlwBfrrt8euBF4Il37PWDl9Nn16Wd5Nv28B9eV/3ngb8BPa+fSdzZPdWyX3m8ILAZ2L/vfho/yD7csu8uOwKrAJQ2uOR54GzAZ2JYsYZxQ9/kGZEl3I7KE+H1Jr4+IE8laqxdExJoRcVajQCStAXwX2Dsi1iJLiHP7uW40cHm6dl3gm8Dlktatu+zDwEeB9YCVgc81qHoDsr+DjYAvAWcC/wBMAXYBviRps3TtCuDTwBiyv7s9gaMAImLXdM226ee9oK780WSt7On1FUfE/WSJ9OeSVgd+DJwTEdc1iNeGCSfL7rIusDgad5OnASdHxGMRsYisxXhY3ecvps9fjIgryFpVk1qM5yVgG0mrRcTCiLijn2veB9wbET+NiOURcR5wN/D+umt+HBF/jojngQvJEv1AXiQbn30ROJ8sEX4nIp5O9d8BvBkgImZHxE2p3vnAD4HdcvxMJ0bEshTPq0TEmcC9wP8C48h+OZk5WXaZJcCYJmNpGwIL6t4vSOdeLqNPsn0OWHOwgUTEs2Rd1xnAQkmXS9oyRzy1mDaqe/+3QcSzJCJWpNe1ZPZo3efP174v6Q2Sfivpb5KeIms5j2lQNsCiiPh7k2vOBLYBTouIZU2utWHCybK73Aj8nWycbiCPkHUhayakc614Fli97v0G9R9GxFUR8S6yFtbdZEmkWTy1mP7aYkyD8QOyuLaIiLWBLwJq8p2G0z8krUk2DnwWcFIaZjBzsuwmEfEk2Tjd9yXtL2l1SStJ2lvS19Jl5wEnSBoraUy6/mctVjkX2FXSBEnrAF+ofSBpfUn7prHLZWTd+RX9lHEF8AZJH5Y0StLBwFbAb1uMaTDWAp4Cnkmt3k/0+fxRYLPXfKux7wCzI+LjZGOxZww5SusJTpZdJiK+STbH8gRgEfAQcAzw63TJ/wdmAfOAPwFz0rlW6roGuCCVNZtXJ7gRZHfVHyG7Q7wb6eZJnzKWAPuka5eQ3cneJyIWtxLTIH2O7ObR02St3gv6fH4ScK6kJyQd1KwwSfsBe5ENPUD232E7SdPaFrFVlielm5nl4JalmVkOTpZmZjk4WZqZ5eBkaWaWQ1dtJDB63TGx8fi+U/Y6a9TIZtP0ild+BGavtmDBfBYvXtzWf5oj194kYvlrFlENKJ5fdFVE7NXOGAajq5LlxuM34bf//T+lxjB2rZVLrR9Acrq07rLzDlPbXmYsf55VJjWd0fWyv8/9frPVWYXqqmRpZsOJQNUZCXSyNLNyCKhQL8rJ0szK45almVkzghEjyw4iNydLMyuPu+FmZk0Id8PNzJqTW5ZmZrm4ZWlmloNblmZmzXhSuplZc56UbmaWk1uWZmbNCEZ6UrqZWWMVm2dZaKSSfi1ptqQ7JE0vsi4zqyAp/1GyoluWH4uIpZJWA26RdHF6dOrLUhKdDrDRxuMLDsfMuke17oYXHemxkm4DbgLGA1v0vSAiZkbE1IiYOnrdsQWHY2ZdxS1LkLQ78E5gx4h4TtJ1wKpF1WdmFdTGlqWks4F9gMciYpt07gJgUrrkdcATETFZ0kTgLuCe9NlNETGjUflFdsPXAR5PiXJL4G0F1mVmVdP+FuM5wPeAn9RORMTBr1SnU4En666/PyIm5y28yGR5JTBD0jyy7H1TgXWZWRW1sWUZEdenFuNrq8kebHUQ8I5Wyy8sWUbEMmDvoso3sx4wuJblGEmz6t7PjIiZOb+7C/BoRNxbd25TSbcCTwEnRMQfGhXgeZZmVpJB3w1fHBGtPmbyUOC8uvcLgQkRsUTSFODXkraOiKcGKsDJ0szKITryWAlJo4APAFNq51LPd1l6PVvS/cAbgFn9FoKTpZmVpmPzLN8J3B0RD79cszQWWBoRKyRtRjat8YFGhVRnRqiZ9Z42zrOUdB5wIzBJ0sOSjkgfHcKru+AAuwLz0jzwXwIzImJpo/LdsjSz8rT3bvihA5z/SD/nLgYuHkz5TpZmVp4uWJmTl5OlmZVD1Vob7mRpZuVxy9LMrDk5WZqZNZY9gsfJsiWjRorRa6xUagyjt/9kqfUDXHn+yWWHwA6bjy47BOt1EhrhZGlm1pRblmZmOThZmpnl4GRpZtaM0lERTpZmVgohtyzNzPJwsjQzy8HJ0swsBydLM7NmfIPHzKw5IUaM8K5DZmZNuRtuZpZHdXKlk6WZlUTValkWNmAgaaKk24sq38yqT1Luo2xuWZpZabohCeZV9K2oUZLOlTRP0i8lrV5wfWZWEbXlju1qWUo6W9Jj9T1aSSdJ+qukuel4b91nX5B0n6R7JL2nWflFJ8tJwMyIeDPwFHBU3wskTZc0S9KsxYsWFRyOmXUVDeJo7hxgr37OfysiJqfjCgBJW5E9T3zr9J3TJY1sVHjRyfKhiPhjev0z4O19L4iImRExNSKmjhk7tuBwzKxrqL1jlhFxPbA0Z+37AedHxLKIeBC4D9i+0ReKTpbR5L2ZDWODTJZjar3QdEzPWc0xaSjwbEmvT+c2Ah6qu+bhdG5ARSfLCZJ2TK8PBW4ouD4zqxCNUO4DWFzrhaZjZo4qfgBsDkwGFgKn1qru59qGjbmik+VdwOGS5gGjyQI3MwOKnzoUEY9GxIqIeAk4k1e62g8D4+su3Rh4pFFZhU0dioj5wFZFlW9m1daJ+ZOSxkXEwvT2AKB2p/wy4BeSvglsCGwB3NyoLM+zNLPStDNZSjoP2J1sbPNh4ERgd0mTybrY84EjASLiDkkXAncCy4GjI2JFo/KdLM2sNO1MlhFxaD+nz2pw/SnAKXnLd7I0s/JUZwGPk6WZladKyx2dLM2sHBXbdcjJ0sxKIaBCudLJ0szKIkaMqE62dLI0s9K4G25m1ozcDTcza0rgbnirBIwaWe6jMa88/+RS6wf41Hm3lh0CNx7/jrJDqFQXzVpTpf/EXZUszWx4qdIvRCdLMyuHxyzNzJrL5llWJ1s6WZpZSbrjEbd5OVmaWWkqlCudLM2sJPLUITOzpjxmaWaWU4VypZOlmZXHLUszsxwqlCudLM2sJN7818ysuapt/lvurhVmNozp5WeH5zmaliadLekxSbfXnfu6pLslzZN0iaTXpfMTJT0vaW46zmhWvpOlmZVGyn/kcA6wV59z1wDbRMSbgT8DX6j77P6ImJyOGc0Kd7I0s3KkSel5j2Yi4npgaZ9zV0fE8vT2JmDjVsMtPFlK+sfUBL5N0k+Lrs/MqqE2KX0Q3fAxkmbVHdMHWeXHgP+se7+ppFsl/V7SLs2+XOgNHklbA8cDO0fEYkmj+7lmOjAdYPyECUWGY2ZdZpB3wxdHxNQW6zkeWA78PJ1aCEyIiCWSpgC/lrR1RDw1UBlFtyzfAfwyIhYDRMTSvhdExMyImBoRU8eOGVtwOGbWTdo8ZjlAHToc2AeYFhEBEBHLImJJej0buB94Q6Nyip46JCAKrsPMKqroeZaS9gI+D+wWEc/VnR8LLI2IFZI2A7YAHmhUVtEty2uBgyStmwJ8TTfczIapQbQq8+RUSecBNwKTJD0s6Qjge8BawDV9pgjtCsyTdBvwS2BGfz3feoW2LCPiDkmnAL+XtAK4FfhIkXWaWTWozZv/RsSh/Zw+a4BrLwYuHkz5ha/giYhzgXOLrsfMqqdKK3i83NHMSjOiQtnSydLMSlOhXOlkaWblkGCkHythZtact2gzM8uhQrly4GQp6TQaTCiPiGMLicjMhgWRTR+qikYty1kdi8LMhqUKDVkOnCzT/MiXSVojIp4tPiQzGxZyburbLZoud5S0o6Q7gbvS+20lnV54ZGbW8zqxkUa75Fkb/m3gPUBth47byNZVmpm1TGST0vMeZct1NzwiHurTXF5RTDjl22Hz8vf6+K/P7VZ2CEQX7BXVBf//sIJV6b9xnmT5kKSdgJC0MnAsqUtuZjYUVRqzzJMsZwDfATYC/gpcBRxdZFBm1vt6bgVP2uV8WgdiMbNhpjqpMt/d8M0k/UbSovRM3kvTzsJmZkPSzueGFy3P3fBfABcC44ANgYuA84oMysx6X3Y3PP9RtjzJUhHx04hYno6f4efqmNlQDaJV2Q0ty0Zrw2tzaH4n6TjgfLIkeTBweQdiM7Me1wU5MLdGN3hmkyXH2o9zZN1nAXy5qKDMbHjohhZjXo3Whm/ayUDMbHipjVlWRa4VPJK2AbYCVq2di4ifFBWUmQ0P7WxZSjob2Ad4LCK2SedGAxcAE4H5wEER8Xj67AvAEWQrEo+NiKsalZ9n6tCJwGnp2AP4GrBvaz+OmVlGgpFS7iOHc4C9+pw7Drg2IrYArk3vkbQVcAiwdfrO6ZJGNio8z93wDwF7An+LiI8C2wKr5InczKyRdu46FBHXA0v7nN6PVx7FfS6wf9358yNiWUQ8CNwHbN+o/Dzd8Ocj4iVJyyWtDTwGeFK6mQ3ZILvhYyTVb0o+MyJmNvnO+hGxECAiFkpaL53fCLip7rqH07kB5UmWsyS9DjiT7A75M8DNOb73KpJOAp6JiG8M9rtm1psGOWS5OCKmtqvqfs41nD+eZ234UenlGZKuBNaOiHktBGdm9jLRkX0qH5U0LrUqx5H1jCFrSY6vu25j4JFGBQ04Zilpu74HMBoYlV43Jel4SfdI+i9gUp7vmNkwMYjxyiHk1MuAw9Prw4FL684fImkVSZsCW9Ckx9yoZXlqg88CeEejgiVNIbvb9JZUzxyybnzf66YD0wHGT5jQqEgz6zFtnjp0HrA72djmw8CJwFeBCyUdAfwFOBAgIu6QdCFwJ7AcODoiGm5q3mhS+h5DjH0X4JKIeC79IJcNUM9MYCbAlClTvebcbBjJMx0nr4g4dICP9hzg+lOAU/KWn2tS+hA4+ZlZv0S1lju2M7H3dT1wgKTVJK0FvL/Ausysgqq0RVthLcuImCPpAmAusAD4Q1F1mVn19NxjJZS1k6cBm0XEyZImABtERNO5loMdEzCz4aVCuTJXN/x0YEegNnj6NPD9wiIys2GjA1OH2iZPN3yHiNhO0q0AEfF4eiSumVnLsi3auiAL5pQnWb6YduMIAEljgZcKjcrMhoUi7zC3W55YvwtcAqwn6RTgBuA/Co3KzIaFnuqGR8TPJc0mm9gpYP+IuKvwyMysp0kdWRveNnnuhk8AngN+U38uIv5SZGBm1vsqlCtzjVlezisPLlsV2BS4h2yHYTOzllVp6lCebvib6t+nHYeOHOByM7NcRI9NSu8rrcx5axHBmNkw0iXLGPPKM2b5mbq3I4DtgEWFRWRmw4b63bC8O+VpWa5V93o52RjmxcWEY2bDRU89NzxNRl8zIv6lQ/GU7u8vNNz/syPWXLXonfPMukNPJEtJoyJied5HSJiZDVaV9rNs1IS5mWx8cm7a5fwi4NnahxHxq4JjM7Me1lPd8GQ0sITsmTu1+ZYBOFmaWeu6ZBljXo2S5XrpTvjtvJIka/y4CDMbsl5Z7jgSWJMWHkZuZtZML3XDF0bEyR2LxMyGGTGyTS1LSZOAC+pObQZ8CXgd8E+8Mjf8ixFxRSt1NEqWFcr5ZlY12dMd21NWRNwDTIaXpzz+lWxryY8C34qIbwy1jkbJst9n7ZqZtUVxyx33BO6PiAXtnJo04Oa/EbG0bbWYmfVjRNrTMs8BjJE0q+6YPkCxhwDn1b0/RtI8SWdLen3Lsbb6RTOzoah1wwexU/riiJhad8x8TZnZ88H2JZsXDvADYHOyLvpC4NRW4/W6OjMrTQFTh/YG5kTEowC1PwEknQn8ttWC3bI0s9IU8AyeQ6nrgksaV/fZAWTzxlvilqWZlUK0t7UmaXXgXbx6c/KvSZpMNjd8PkPYuNzJ0szKofZupBERzwHr9jl3WLvKL7QbLukfJN0saa6kH6b5T2ZmQLrJk/MoW2HJUtIbgYOBnSNiMrACmNbPddNrUwEWLfYG7GbDhYCRUu6jbEV2w/cEpgC3pKb2asBjfS9Kt/9nAkyZMtVrzs2GkS7IgbkVmSwFnBsRXyiwDjOrLFVq898ixyyvBT4kaT0ASaMlbVJgfWZWIbW74XmPshXWsoyIOyWdAFwtaQTwInA0sKCoOs2sWqrUsix06lBEXMCrt00yM3tZdVKl51maWVnaPM+yaE6WZlaKdq/gKZqTpZmVxi1LM7MceuUZPGZmhcm64dXJlk6WZlaaCvXCnSzNrCxCblmamTXnlqWZWRMeszQzy2Nwj4sonZOlmZXGybJFKyJ4dtnyUmMYWaWJX2YV5xs8ZmZNCE9KNzPLpYDnhhfGydLMStPObrik+cDTZM/7Wh4RUyWNJtsmciLZo3APiojHWym/Spt+mFkPqXXD8x457RERkyNianp/HHBtRGxB9vSG41qN18nSzEqiQf2vRfsB56bX5wL7t1qQk6WZlSPNs8x7AGNqj81Ox/Q+JQbZY2xm1322fkQsBEh/rtdquB6zNLPSDLK9uLiue92fnSPikfSQxGsk3T2U2PpysjSzUmRjlu27wRMRj6Q/H5N0CbA98KikcRGxUNI44LFWy3c33MxKo0EcDcuR1pC0Vu018G7gduAy4PB02eHApa3G6palmZWnfQ3L9YFL0mMqRgG/iIgrJd0CXCjpCOAvwIGtVuBkaWalaVc3PCIeALbt5/wSYM921OFkaWalqc76nQ6NWUo6VtJdkn7eifrMrCLaNWjZAZ1qWR4F7B0RD3aoPjPrclkO7IIsmFPhLUtJZwCbAZdJ+nTR9ZlZRQx+UnqpCm9ZRsQMSXuRrdlcXHR9ZlYdXZADcyt9nqWk6bXlS0sWO5eaDSsVGrMsPVlGxMyImBoRU9cdM6bscMysYzqykUbbeOqQmZWmG8Yi83KyNLNSdEnvOreOJMuImNiJesysWlShpqVblmZWmgrlSidLMytPhXKlk6WZlaRig5ZOlmZWmm6YEpSXk6WZlUJ4zNLMLJcK5UonSzMrUYWypZOlmZXGY5ZmZjmMqE6udLI0sxI5WZqZNVa1ndK7KlmOkFh1pZGlxjCySv0Csypr4w7oksYDPwE2AF4CZkbEdySdBPwTsChd+sWIuKKVOroqWZrZ8NLGpsly4LMRMUfSWsBsSdekz74VEd8YagVOlmZWnjZly4hYCCxMr5+WdBewUXtKz5S+U7qZDVeD3il9TO0RNOmY3m+p0kTgLcD/plPHSJon6WxJr281WidLMyvNIJ/uuLj2CJp0zHxteVoTuBj454h4CvgBsDkwmazleWqrsTpZmlkpBvOssjy9dUkrkSXKn0fErwAi4tGIWBERLwFnAtu3Gq+TpZmVp03ZUtmW62cBd0XEN+vOj6u77ADg9lZD9Q0eMyvNiPZtO7QzcBjwJ0lz07kvAodKmgwEMB84stUKnCzNrDTtSpURccMAxbU0p7I/TpZmVo42TkrvBCdLMytRdbKlk6WZlcI7pZuZ5VShXOlkaWblccvSzCwHb9GWSPo3YBrwELAYmN2O3T/MrEdUJ1cWlywlTQU+SLagfRQwB5hdVH1mVj0VypWFtizfDlwaEc8DSPpNfxelnUOmA4yfMKHAcMysm0htXcFTuCLXhuf6W4iImbVdRMaMGVtgOGbWddq5k0bBikyWNwDvl7Rq2jbpfQXWZWYVVKFcWVw3PCJukXQZcBuwAJgFPFlUfWZWPRXqhRe+Rds3ImISsD8wCd/gMbOXDXqn9FIVPc9ypqStgFWBcyNiTsH1mVlFeLljnYj4cJHlm5l1ilfwmFlp3LI0M8uhG8Yi83KyNLNSZJPSy44iPydLMyuPk6WZWXPuhpuZ5VClGzx+briZlaadyx0l7SXpHkn3STqu3bE6WZpZedqULSWNBL4P7A1sRfa88K3aGaqTpZmVpo3LHbcH7ouIByLiBeB8YL92xtpVY5a3zpm9eM1VRiwYQhFjyHZkL5Nj6J4YoDvi6IUYNmlXIDW3zpl91eora8wgvrKqpFl172dGxMz0eiOyJzLUPAzsMNQY63VVsoyIIW1oKWlWRExtVzyOodoxdEscjqF/EbFXG4vrr+kZbSzf3XAz6wkPA+Pr3m8MPNLOCpwszawX3AJsIWlTSSsDhwCXtbOCruqGt8HM5pcUzjFkuiEG6I44HEPBImK5pGOAq4CRwNkRcUc761BEW7v1ZmY9yd1wM7McnCzNCiBp9bJjsPZysrS2krS+VKUVv+0naV/gtHSjwXqEk2UbSBqR/hzuSWIj4ASypWal/l2UVb+kdYFjgVOBjSWNLiMOa7+eSZaSViur7oh4Kb3cRNKoMhOFpDeWVTfZvLbZwFuAD5ScMDcEkNTpGR8vAMuBfwO+CbzU+PLiSdqg7F9evaAnkmWaMvA1SV+RtE4H691J0iHp9dHAxcCPgeNqrc1OkvQJ4OuS1i+hbkU2teIlYEvg88B+ZfyfNP17OEPSV4GjJK3Sqboj4mngWuD9wLyIeKLkX55vAr4MfNAJc2gqP89S0lHAgcCHgTnARpK+HBH3dqD61wNfSa25iSmOicDuwFclHVfX6ixUGiebAewbEY92os56ERGSpgGfBI4APgbsAYySdHF0aI6apP2Bg4B9gUuAOyJiWSfqrnMh2b/F70l6PCK+1eH6AZC0D3AMsCZpbXcn/1v0mkq3LCWtDWxHNlv/g8Ct6aPvStqi6Poj4nJgeqp7VEQ8ANwA/BRYn2yrqE7ZELggIhZIWqmD9dabBFwYEfOAfwHuI0ueB3awVbMO8G1gf+BF4DMAkt7QofqJiAURcS3ZL/BPpJZuR6XexXHApyLi7cCvyH6Jl9La7wWVTpYR8RRwNLAecEBamH848FbgsE7cjYyIa4DjgfdKOjgiXkit2jWBTo4fLgB2kTQpIl4EkHRYaml1yhxgZ0lbR8SLEXEa2WqKbcn+PjphPvB14IiIeHdEvCDpWODjnf4lEhG3Ah8CTpJ0ZCfrJhs7HQHUNqc5C1gJ+BTwng7H0hMq3w2PiGWSniPr7r2JbDH9lcCP0r52nYjhUkmHkbVotwJuBjYlSx6d8kdgZ+BwSf8DrEV2V/bQDsZwHdkvqkMl/TewGrAUOC2N5XXCbOBS4CVJuwMTyH6BHl77JdJJETEvxfF8h+t9XNIvgT0kPRERt0u6iKwndLCk35UwPFFpPbHcMQ3g/zPwTrLu70ERcXcJcexPdpPnIuC4iJjf4frHkW14ui/wJPCV1CXuZAwbAh9Ix3LgsxHxpw7HMI7s72BfYAnw9U7H0A0kbUw2jj2F7Bf3/sA/AicBJ0TEbeVFVz09kSwBUhdrA+CliPhriXHsBsyPiKFsYjzUGFYG6FTLeoAY1iD79/VMiTGsBFBGi7JbpHH9nciGQq4A1iDbVONdZdwIrLKeSZZm1pikPYCvAEe6VTl4TpZmw0Qanli5zF5PlTlZmpnlUOmpQ2ZmneJkaWaWg5OlmVkOTpY9QtIKSXMl3S7poqFsPivpHEkfSq9/lCbaD3Tt7pJ2aqGO+dJrnxk90Pk+1wxqOpKkkyR9brAxmtVzsuwdz0fE5IjYhmyp24z6DyWNbKXQiPh4RNzZ4JLdyebxmfU0J8ve9Afg/6VW3+8k/QL4k6SRkr4u6RZJ82rrlZX5nqQ7JV1Ottae9Nl1kqam13tJmiPpNknXSppIlpQ/nVq1u0gaK+niVMctknZO311X0tWSbpX0Q6DpZg6Sfi1ptqQ7JE3v89mpKZZrJY1N5zaXdGX6zh8kbdmOv0wz6IG14fZqyja73ZtsfTzA9sA2EfFgSjhPRsRb0xLRP0q6mmyz3knAm8iWi94JnN2n3LHAmcCuqazREbFU0hnAMxHxjXTdL4BvRcQNkiaQPZr0jcCJwA0RcbKk95GtUW7mY6mO1YBb0vZiS8hWocyJiM9K+lIq+xiylSkzIuJeSTsApwPvaOGv0ew1nCx7x2qS5qbXfyDbZWYn4OaIeDCdfzfw5tp4JNl2ZlsAuwLnRcQK4JG0CUZfbwOur5UVEUsHiOOdwFZ1u4CtLWmtVMcH0ncvl/R4jp/pWEkHpNfjU6xLyDYYviCd/xnwK0lrpp/3orq6O7bpr/U+J8ve8XxETK4/kZLGs/WngE9GxFV9rnsv0Gx1gnJcA9nQzo4R8apddlIsuVdApJ163pnKek7SdcCqA1weqd4n+v4dmLWLxyyHl6vINqNdCbINcdOGF9cDh6QxzXFkO5z3dSOwm6RN03drD+J6mmw7uJqrybrEpOtqyet6YFo6tzfZLvONrAM8nhLllmQt25oRZPtEQrbB7g1pb9MHJR2Y6pCkbZvUYZabk+Xw8iOy8cg5km4HfkjWu7gEuBf4E/AD4Pd9vxgRi8jGGX8l6TZe6Qb/BjigdoOHbA/NqekG0p28clf+34FdJc0hGw74S5NYryTbo3Qe2TNkbqr77Flga0mzycYkT07npwFHpPjuINuuzqwtvDbczCwHtyzNzHJwsjQzy8HJ0swsBydLM7McnCzNzHJwsjQzy8HJ0swsh/8DgxqiT0W+S4QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAEmCAYAAAAA6gkZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3debxcVZnu8d+ThEAgEIYAYgYIiGFQQQggkwZEBJRJVEAaEOGGyGQ757Z2i9A2eoGrrYAhIjI4MAhikFygLxoBlSYDEEgQzQUyAAoJM0Qh8N4/9jpSKarq1MmpXXvXqefLpz5U7b1rrbfqnLxnrb3XWlsRgZmZvdmgogMwMysrJ0gzszqcIM3M6nCCNDOrwwnSzKwOJ0gzszqcIDuApJmSTkrPj5F0a4vL30JSSBrSynJ7qVOSfiTpGUl396OcvSU91MrYiiJprKQXJQ0uOhbLOEECkh6V9FdJ61RsO0nSzALDqikifhIR+xcdRwvsBXwAGB0Ru65uIRFxR0SMb11Y+Ui/Y/s1OiYiFkfE8Ih4rV1xWWNOkG8YAnymv4WklpG/195tDjwaES8VHUgZtLP1bs3zP+Q3nAt8QdL6tXZK2kPSLEnPpf/vUbFvpqRvSPod8DKwZeqyniLpz5JekHS2pK0k/UHS85KukTQ0vX8DSb+S9FTqcv5K0ug6cXxS0p3p+ZdSl6zn8aqky9K+EZJ+KOkJSY9J+veerpukwZLOk7RM0sPAhxp9MZLGSLo+xbdc0gVp+yBJX5W0SNKTkq6QNCLt6+m2Hy9pcarrK2nficAlwO4p7q9Xfq6KekPS29LzgyQtSN/lY5K+kLZPlLS04j3bpp/Hs5LmSzqkYt9lki6UdFMq578lbVXnM/fEf4KkJennMlnSLpLmpfIvqDh+K0m/Tt/PMkk/6fldknQlMBa4MX3eL1WUf6KkxcCvK7YNkbShpKWSDk5lDJe0UNJxjX5W1mIR0fUP4FFgP+B64N/TtpOAmen5hsAzwLFkLc2j0+uN0v6ZwGJg+7R/DSCA6cB6afvfgduALYERwALg+PT+jYAjgLWBdYFrgRsq4psJnJSefxK4s8ZnGAM8DhyUXt8AXAysA2wC3A2cnPZNBv6Y3rMh8JsU75Aa5Q4G7gO+ncpaC9gr7fsUsDB9puHp+7sy7dsilfkDYBiwQ/oOtq31OWp9rvT+t6XnTwB7p+cbADul5xOBpen5GimefwGGAvsCLwDj0/7LgKeBXdPP6SfAVXV+J3rin5o+8/7A39L3ugkwCngSeF86/m1kpwzWBDYGbge+U/07VqP8K9L3Oqxi25B0zP7AX1J9PwB+XvS/lW57FB5AGR68kSDfATyXfsErE+SxwN1V7/kD8Mn0fCZwVtX+APaseD0H+HLF6/Mr/wFVvXdH4JmK1zNpkCDTP65/lA9smpLRsIpjjgZ+k57/GphcsW9/6ifI3YGn6uy7DTil4vV44NWUfHr+sY+u2H83cFStz1Hnc1UmyMXAycB6VcdM5I0EuXdKKIMq9v8MODM9vwy4pGLfQcAf6/wMeuIfVbFtOXBkxevrgH+u8/7DgHuqf8dqlL9ljW1DKrZ9D7if7I/fRkX/W+m2h7vYFSLiAeBXwJSqXW8FFlVtW0TWiuixpEaRf614vqLG6+EAktaWdHHqqj5P1vpYX81fzfwh8FBEfCu93pysNfVE6go+S9aa3KTi81TGW/3ZKo0BFkXEyhr7qr+XRWTJcdOKbX+peP4y6TOvhiPIEtoiSb+VtHudeJZExOtVMVX+nPoaT7M/w00kXZW6/88DPwZG9lI21P69qTSN7A/3jyJieRPlWQs5Qb7Z14D/war/qB4nSzqVxgKPVbzuz7JInydrfe0WEesB703b1dsbJU1J7z2xYvMSshbkyIhYPz3Wi4jt0/4nyBJfj7ENqlgCjFXtiwjV38tYYCWrJpFmvUR2igEASW+p3BkRsyLiULIkfwNwTZ14xmjVi2TVP6e8nEP2O/Cu9DP8J1b9+dX7/aj7e5P+QF5M1g3/dM/5WGsfJ8gqEbEQuBo4o2LzDODtkj6RTqAfCWxH1tpshXXJWiPPStqQLEn3StKBKc7DImJFxWd4ArgVOF/SeuliylaS3pcOuQY4Q9JoSRvw5hZzpbvJEuo3Ja0jaS1Je6Z9PwM+K2mcpOHAfwBX12lt9uY+YHtJO0paCziz4nMOVTb+c0REvAo8D9QaCvPfZIn2S5LWkDQROBi4ajXi6at1gRfJfoajgC9W7f8r2bnavviX9P9PAecBV/ShV2Et4ARZ21lkJ84BSF2bD5O19JYDXwI+HBHLWlTfd8jOIy4D7gJubvJ9R5KdL31Qb1zJnpr2HUd2oWIB2QWlnwObpX0/AG4hS0pzyS6u1BTZmLyDyS5CLAaWpnoBLgWuJDsl8AjZRYzTm4y9up4/kX3v/xf4M3Bn1SHHAo+m7utkshZadRmvAIcAB5J9lxcBx0XEH1cnpj76OrAT2Tnsm3jzd3oO8NV0yuMLvRUmaWfgc2TxvwZ8i6y12eiPmbWY0olgMzOr4hakmVkdTpBmNiBIujRNWHigzn5J+m4acD9P0k69lekEaWYDxWXAAQ32HwhsnR6TgO/3VqATpJkNCBFxO9lMqXoOBa6IzF1kY403a3A8pZogrzXWDq05otAYdnj7qN4PytmgXkc/mrXXokWPsmzZspb+Zg5eb/OIlSt6PzCJFU/NJxsp0WNaREzrQ5WjWHVg/tK07Yl6byhXglxzBGvueGLvB+bo17d+vdD6AYYN9VA3K5c9d5vQ8jJj5QrWHP/xpo//270X/i0i+hNIrQTfcBhPqRKkmXUTQXtXBlzKqjPIRpPNvqrL5yDNrBgCpOYf/TcdOC5dzX4P8FyadVaXW5BmVpwWtiAl/YxsdaeRaY3Qr5Et2kJETCWbMnwQ2ZJ4LwMn9FamE6SZFUQwqHXn2yPi6F72B3BqX8p0gjSz4rSm65wbJ0gzK4Zo90WaPnOCNLOCtOziS26cIM2sOG5BmpnV4RakmVktbR8o3mdOkGZWjJ6B4iXmBGlmxXEL0sysFsHgci/M4gRpZsXogHGQuUYn6QZJcyTNlzQpz7rMrAO1d7GKPsu7BfmpiHha0jBglqTr0i1U/yElzix5rrlezuGYWXn4KvYZkg5Pz8eQ3QtilQSZVgSeBjBo+Ga+B61ZN+nWq9iSJgL7AbtHxMuSZgJr5VWfmXWgLm5BjgCeSclxG+A9OdZlZp2mwHOLzcozQd4MTJY0D3gIuCvHusysE3VrCzIi/k52H1ozs9q6uAVpZtaAr2KbmdUmWnrLhTw4QZpZQdyCNDOrz+cgzczqcAvSzKwOtyDNzGqQz0GamdXnFqSZWW1ygjQze7PsljROkE3bcfwobr/trEJj2OZz0wutH+DGL+1bdAhsO8prc1rOJDTICdLMrCa3IM3M6nCCNDOrwwnSzKwWpUeJOUGaWSGE3II0M6vHCdLMrA4nSDOzOpwgzcxq8UUaM7PahBg0qNyr+ZQ7OjMb0CQ1/WiirAMkPSRpoaQpNfaPkHSjpPskzZd0Qm9lOkGaWXHUh0ejYqTBwIVkt5reDjha0nZVh50KLIiIHYCJwPmShjYq1wnSzIqhlrYgdwUWRsTDEfEKcBVwaNUxAayrrLDhwNPAykaF5pYgJW0h6YG8yjezztfHBDlS0uyKx6SKokYBSypeL03bKl0AbAs8DtwPfCYiXm8Uny/SmFlh+jjMZ1lETKhXVI1tUfX6g8C9wL7AVsB/SbojIp6vV2HeXewhki6XNE/SzyWtnXN9ZtYheqYatqiLvRQYU/F6NFlLsdIJwPWRWQg8AmzTqNC8E+R4YFpEvAt4Hjil+gBJk3qazMueeirncMysVFp0kQaYBWwtaVy68HIUUL369WLg/QCSNiXLTw83KjTvBLkkIn6Xnv8Y2Kv6gIiYFhETImLCyI03zjkcMyuNFl6kiYiVwGnALcCDwDURMV/SZEmT02FnA3tIuh+4DfhyRCxrVG7e5yCrzwFUvzazLtbKqYYRMQOYUbVtasXzx4H9+1Jm3i3IsZJ2T8+PBu7MuT4z6yAapKYfRcg7QT4IHC9pHrAh8P2c6zOzDtLKmTR5yK2LHRGPko1oNzN7kyITX7M8DtLMCuMEaWZWhxOkmVk95c6PTpBmVhy3IM3MapETpJlZTQJKnh+dIM2sKGJQQQPAm+UEaWaFcRfbzKwWuYttZlaTwF3svhAwZHCxt8m5/PQ3rcjWdsdcfFfRITDn6x8oOoTSd7+s/8r+Iy5VgjSz7lL2P4JOkGZWDJ+DNDOrLRsHWe4M6QRpZgXxcmdmZnWVPD86QZpZQeRhPmZmNfkcpJlZAyXPj06QZlYctyDNzOooeX50gjSzgnjBXDOz2rxgrplZXR4obmZWV8nzoxOkmRWkAwaK5774oqTjJM2TdJ+kK/Ouz8w6Q89A8WYfRci1BSlpe+ArwJ4RsUzShjWOmQRMAhgzdmye4ZhZyZT9HGTeLch9gZ9HxDKAiHi6+oCImBYREyJiwsYjN845HDMrE6n5RxHyPgcpIHKuw8w6VLe3IG8DPi5pI4BaXWwz61J9aD0OyBZkRMyX9A3gt5JeA+4BPplnnWbWGeRxkBARlwOX512PmXWekudHj4M0s+IMKnmGLPYm1GbW1Vp5DlLSAZIekrRQ0pQ6x0yUdK+k+ZJ+21uZbkGaWSEkGNyimTSSBgMXAh8AlgKzJE2PiAUVx6wPXAQcEBGLJW3SW7luQZpZYVo4k2ZXYGFEPBwRrwBXAYdWHfMJ4PqIWAwQEU/2VqgTpJkVpo9d7JGSZlc8JlUUNQpYUvF6adpW6e3ABpJmSpoj6bje4qvbxZb0PRoM8o6IM3or3MysHpEN9emDZRExoUFx1arz1xBgZ+D9wDDgD5Luiog/1auw0TnI2Y0iNTPrrxYu5rMUGFPxejTweI1jlkXES8BLkm4HdgD6niDT+MV/kLROKtjMrP9au0rPLGBrSeOAx4CjyM45VvolcIGkIcBQYDfg240K7fUcpKTdJS0AHkyvd5B0Ud/jNzNbVauG+UTESuA04BayXHVNmsk3WdLkdMyDwM3APOBu4JKIeKBRuc0M8/kO8EFgeqrkPknvbeJ9ZmZ1idYOFI+IGcCMqm1Tq16fC5zbbJlNjYOMiCVVTeHXmq2g0+z5tpFFh8DihdWnTtpv6dMrig6BMRutXXQIlrOST6RpKkEukbQHEJKGAmeQuttmZv0xEBarmAz8J9mYosfI+vin5hmUmQ18rZxJk5deE2RaDfyYNsRiZl2m3OmxuavYW0q6UdJTkp6U9EtJW7YjODMb2Mp+065mphr+FLgG2Ax4K3At8LM8gzKzgS+7it38owjNJEhFxJURsTI9fozvM2Nm/dWH1mPpbvtacf+Y36S11a4iS4xHAje1ITYzG+BKfhG74UWaOWQJsecjnFyxL4Cz8wrKzLpDxw7ziYhx7QzEzLpLzznIMmtqJo2kdwDbAWv1bIuIK/IKysy6Q8e2IHtI+howkSxBzgAOBO4EnCDNbLVJMLjkCbKZq9gfJVtg8i8RcQLZ+mlr5hqVmXWFVt60Kw/NdLFXRMTrklZKWg94EvBAcTPrt47vYgOz093AfkB2ZftFsrXU+kTSmcCLEXFeX99rZgNTyfNjU3OxT0lPp0q6GVgvIublG5aZDXRCLV0PMg+NBorv1GhfRMztrXBJXwGOI7vb2FNkLVAzMyjw3GKzGrUgz2+wL4B9GxUsaWey+0K8O9UzlxoJMt26cRLAmLFjewnXzAaSjj0HGRH79LPsvYFfRMTLAJKm16lnGjANYOedJ3iOt1kXaWYYTZGaGijeD054ZlaTKH8LMs8EfjtwuKRhktYFDs6xLjPrQGVf7iy3FmREzJV0NXAvsAi4I6+6zKzzDIhbLihrAx8DbBkRZ0kaC7wlInodCxkR3wC+0f8wzWwgKnl+bKqLfRGwO3B0ev0CcGFuEZlZ1xgIUw13i4idJN0DEBHPpNu/mpmttmy5s3I3IZtJkK9KGky6Ii1pY+D1XKMys65Q9mE+zcT3XeAXwCaSvkG21Nl/5BqVmXWFju9iR8RPJM0hW/JMwGER8WDukZnZgCZ18FzsHumq9cvAjZXbImJxnoGZ2cBX8vzY1DnIm3jj5l1rAeOAh4Dtc4zLzLpA2Yf5NNPFfmfl67TKz8l1Djcza4oYAAPFq6UZMrvkEYyZdZECpxA2q5lzkJ+reDkI2IlsbUczs34R5c6QzbQg1614vpLsnOR1+YRjZt2i4++LnQaID4+IL7YpnsJtsMtpRYfAM7MuKDoEs7bo2AQpaUhErGx06wUzs/4o+3qQjVqQd5Odb7w3rQZ+LfBSz86IuD7n2MxsAOuELnYzUw03BJaT3YPmw2QL3344z6DMrAv0YZphMw1NSQdIekjSQklTGhy3i6TXJH20tzIbtSA3SVewH+CNgeI9fCsFM+u3Vk01TNdLLgQ+ACwFZkmaHhELahz3LeCWZsptlCAHA8Oh5nV4J0gz65cWd7F3BRZGxMMAkq4CDgUWVB13OtkonKbGcjdKkE9ExFmrEaiZWRPE4L61IEdKml3xelq6KyrAKGBJxb6lwG6r1CaNAg4nO13Y7wRZ8tOnZtbJsrsa9uktyyJiQoPiqlX3dL8DfDkiXmv26nmjBPn+pkowM1sdrZ1quBQYU/F6NPB41TETgKtSchwJHCRpZUTcUK/QugkyIp5e/VjNzHrXwvUgZwFbSxoHPAYcBXyi8oCIGNfzXNJlwK8aJUfI8bavZmaNrEYXu640qeU0sqvTg4FLI2K+pMlp/9TVKdcJ0swK08oVxSNiBjCjalvNxBgRn2ymTCdIMytMyWcaOkGaWTFE+e9q6ARpZsVQ+ReryDWBS/onSXdLulfSxWmaj5kZkC7UNPkoQm4JUtK2wJHAnhGxI/AacEyN4yZJmi1p9lPLvFC5WbcQMFhq+lGEPLvY7wd2Jps0DjAMeLL6oDRVaBrAzjtP8Bxvsy5S8h52rglSwOUR8T9zrMPMOpa6+hzkbcBHJW0CIGlDSZvnWJ+ZdZCeq9jNPoqQWwsyIhZI+ipwq6RBwKvAqcCivOo0s85S9hZkrsN8IuJq4Oo86zCzzlXu9OhxkGZWlA4YB+kEaWaF8EwaM7MG3II0M6uj7Ld9dYI0s0JkXexyZ0gnSDMrTMl72E6QZlYUIbcgzcxqcwvSzKwGn4M0M6tHbkGamdXlBNkH/2/5Sxz5o1mFxnDOf36u0PrNuokv0piZ1SA8UNzMrK5W3hc7D06QZlYYd7HNzGpwF9vMrC7PpDEzq83jIM3M6it5fnSCNLNiZOcgy50inSDNrDDlTo9OkGZWpJJnSCdIMyuMu9hmZnWUOz226a6Lks6Q9KCkn7SjPjPrEOrDowDtakGeAhwYEY+0qT4zK7ks75W7DZl7C1LSVGBLYLqkz+Zdn5l1iDRQvNlHEXJvQUbEZEkHAPtExLK86zOzzlHu9mObzkE2ImmSpNmSZr/ywrNFh2Nm7dTCc5CSDpD0kKSFkqbU2H+MpHnp8XtJO/RWZuEJMiKmRcSEiJgwdN31iw7HzNpGffqvYUnSYOBC4EBgO+BoSdtVHfYI8L6IeBdwNjCttwg9zMfMCtPCc4u7Agsj4uGsXF0FHAos6DkgIn5fcfxdwOjeCi28BWlm3akvveuUR0f2nI5Lj0kVxY0CllS8Xpq21XMi8H96i7EtLciI2KId9ZhZZ1HfmpDLImJCvaJqbIs6de5DliD36q1Cd7HNrDAt7GIvBcZUvB4NPP7m+vQu4BKycdnLeyvUXWwzK0wLL2LPAraWNE7SUOAoYPoqdUljgeuBYyPiT83E5xakmRWjhVMII2KlpNOAW4DBwKURMV/S5LR/KvBvwEbARalrv7JBlx1wgjSzArVyqmFEzABmVG2bWvH8JOCkvpTpBGlmhRC+J42ZWV0lz49OkGZWoJJnSCdIMytM2Zc7c4I0s8IMKnd+dII0swI5QZqZvVknrCheqgT5/JPLufWiywuN4epZFxRav1nXKHCl8GaVKkGaWXcpeX50gjSzApU8QzpBmllBel8pvGhOkGZWGJ+DNDOroYWL+eTGCdLMilPyDOkEaWaFGVTyPrYTpJkVptzp0QnSzIrigeJmZo2UO0M6QZpZIbyiuJlZAyXPj06QZlYctyDNzOro6qmGkv4VOAZYAiwD5kTEeXnWaWYdpNz5Mb8EKWkCcATw7lTPXGBOXvWZWecpeX7MtQW5F/DLiFgBIOnGWgdJmgRMAmCN4TmGY2ZlIpV/Js2gHMtu6pNHxLSImBAREzRkWI7hmFnpqA+PAuSZIO8EDpa0lqThwIdyrMvMOlDJ82N+XeyImCVpOnAfsAiYDTyXV31m1nlK3sPOtQUJcF5EjAcOA8bjizRm9g/q039FyHsc5DRJ2wFrAZdHxNyc6zOzDtH1Uw0j4hN5lm9mlifPpDGzwnR1C9LMrJGunmpoZlZPNlC86Cgac4I0s+I4QZqZ1eYutplZHWW/SJP3QHEzs7paOdVQ0gGSHpK0UNKUGvsl6btp/zxJO/VWphOkmRWnRRlS0mDgQuBAYDvg6DRJpdKBwNbpMQn4fm/hOUGaWWFaONVwV2BhRDwcEa8AVwGHVh1zKHBFZO4C1pe0WaNCS3UOMlY8texv9164qB9FjCRbuXy1DVvjwv68vSUxtIBjeEMZ4hgIMWzeqkB63DN3zi1rD9XIPrxlLUmzK15Pi4hp6fkosjsX9FgK7Fb1/lrHjAKeqFdhuRJkxMb9eb+k2RExoVXxOIbOjqEscTiG2iLigBYWV6uJGatxzCrcxTazgWApMKbi9Wjg8dU4ZhVOkGY2EMwCtpY0TtJQ4ChgetUx04Hj0tXs9wDPRUTd7jWUrIvdAtN6PyR3jiFThhigHHE4hpxFxEpJpwG3AIOBSyNivqTJaf9UYAZwELAQeBk4obdyFdGwC25m1rXcxTYzq8MJ0iwHktYuOgbrPydIaylJm0pln2GbL0mHAN9LFwusgzlBtoCkQen/3Z4YRgFfJZvmVeh3UVT9kjYCzgDOB0ZL2rCIOKw1BkyClDSsqLoj4vX0dHNJQ4pMDpK2LapusjFlc4B3Ax8pOEm+FUBSu0dqvAKsBP4V+N/A640Pz5+ktxT9B6tTDYgEmS7v/y9J50ga0cZ695B0VHp+KnAd8CNgSk+rsp0kfRo4V9KmBdStyIZEvA5sA3wZOLSIf5jp92GqpG8Cp0has111R8QLwG3AwcC8iHi24D+Y7wTOBo5wkuy7jh8HKekU4GPAJ4C5wChJZ0fEn9tQ/QbAOanVtkWKYwtgIvBNSVMqWpe5Sue9JgOHRMRf21FnpYgISccApwMnAp8C9gGGSLou2jSeTNJhwMeBQ4BfAPMj4u/tqLvCNWS/ixdIeiYivt3m+gGQ9GHgNGA4aS51O38WA0FHtyAlrQfsRDZq/gjgnrTru5K2zrv+iLiJbNmkI4AhEfEwcCdwJbAp2bJL7fJW4OqIWCRpjTbWW2k8cE1EzAO+SDYg93TgY21svYwAvgMcBrwKfA5A0tvbVD8RsSgibiP7o/3p1KJtq9SLmAJ8JiL2Aq4n+8NdSKu+U3V0goyI54FTgU2Aw9Pk9+OBXYBj23EVMSL+C/gKcJCkIyPildR6HQ6083zgImBvSeMj4lUAScemFlW7zAX2lLR9RLwaEd8jm9WwA9n30Q6PAucCJ0bE/hHxiqQzgJPa/YcjIu4BPgqcKenkdtZNdi50ENCzAMwPgTWAzwAfbHMsHavju9gR8XdJL5N15d5JNhn9ZuCStC5cO2L4paRjyVqu2wF3A+PIEka7/A7YEzhe0u+Bdcmuph7dxhhmkv1xOlrSr4FhwNPA99K5uXaYA/wSeF3SRGAs2R/N43v+cLRTRMxLcaxoc73PSPo5sI+kZyPiAUnXkvV4jpT0mwJOPXScATHVMJ2E/2dgP7Ku7ccj4o8FxHEY2YWaa4EpEfFom+vfjGxR0EOA54BzUne3nTG8FfhIeqwEPh8R97c5hs3IvoNDgOXAue2OoQwkjSY7L70z2R/rw4DjgDOBr0bEfcVF1xkGRIIESN2ntwCvR8RjBcbxPuDRiOjPwr/9jWEoQLta0HViWIfs9+vFAmNYA6CIlmNZpPP0e5Cd5pgBrEO2cMUHiriY12kGTII0s8Yk7QOcA5zs1mNznCDNukQ69TC0yN5Np3GCNDOro6OH+ZiZ5ckJ0sysDidIM7M6nCAHCEmvSbpX0gOSru3Pgq2SLpP00fT8kjT4vd6xEyXtsRp1PCq9+Z7I9bZXHdOnoUOSzpT0hb7GaOYEOXCsiIgdI+IdZNPMJlfulDR4dQqNiJMiYkGDQyaSjbMzG3CcIAemO4C3pdbdbyT9FLhf0mBJ50qaJWlez/xgZS6QtEDSTWRz20n7ZkqakJ4fIGmupPsk3SZpC7JE/NnUet1b0saSrkt1zJK0Z3rvRpJulXSPpIupfRP3VUi6QdIcSfMlTarad36K5TZJG6dtW0m6Ob3nDknbtOLLtO7V8XOxbVXKFog9kGw+OsCuwDsi4pGUZJ6LiF3S9MzfSbqVbIHb8cA7yaZqLgAurSp3Y+AHwHtTWRtGxNOSpgIvRsR56bifAt+OiDsljSW7Dee2wNeAOyPiLEkfIpsT3JtPpTqGAbPSUl3LyWaDzI2Iz0v6t1T2aWQzRCZHxJ8l7QZcBOy7Gl+jGeAEOZAMk3Rven4H2eotewB3R8Qjafv+wLt6zi+SLQ22NfBe4GcR8RrweFpootp7gNt7yoqIp+vEsR+wXcWKWutJWjfV8ZH03pskPdPEZzpD0uHp+ZgU63KyRXmvTtt/DFwvaXj6vNdW1N22hXJtYHKCHDhWRMSOlRtSonipchNwekTcUnXcQUBvMwbUxDGQnbbZPSJWWb0mxdL0rIS0As5+qayXJc0E1qpzeKR6n63+Dsz6w+cgu8stZAu4rgHZIrJpUYnbgaPSOcrNyFYCr/YH4H2SxqX39tyM6gWypdV63ErW3SUd15OwbgeOSdsOJFuNvZERwDMpOW5D1oLtMYhsnUXIFqW9M60N+oikj6U6JGmHXuowa8gJsrtcQnZ+ca6kB4CLyXoRvwD+DFQooXoAAACISURBVNwPfB/4bfUbI+IpsvOG10u6jze6uDcCh/dcpCFbg3JCugi0gDeupn8deK+kuWRd/cW9xHoz2Rqf88juqXJXxb6XgO0lzSE7x3hW2n4McGKKbz7Z0m9mq81zsc3M6nAL0sysDidIM7M6nCDNzOpwgjQzq8MJ0sysDidIM7M6nCDNzOr4//KcQb6jzhY5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn import svm, datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "# Compute confusion matrix\n",
    "cm = confusion_matrix(y_compare, pred)\n",
    "np.set_printoptions(precision=2)\n",
    "print('Confusion matrix, without normalization')\n",
    "print(cm)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm, products)\n",
    "\n",
    "# Normalize the confusion matrix by row (i.e by the number of samples\n",
    "# in each class)\n",
    "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "print('Normalized confusion matrix')\n",
    "print(cm_normalized)\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cm_normalized, products, \n",
    "        title='Normalized confusion matrix')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}