03. _Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Probability Scoring Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- (Wiki) In decision theory, a **score function**, or **scoring rule**, measures the accuracy of probabilistic predictions. It is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive outcomes. The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one. A score can be thought of as either a measure of the \"calibration\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Predicting probabilities instead of class labels for a classification problem can provide additional nuance and uncertainty for the predictions.\n",
    "- More sophisticated metrics to be used to interpret and evaluate the predicted probabilities. \n",
    "- Score function examples:\n",
    "    - **Log loss score**: heavily penalizes predicted probabilities far away from their expected value\n",
    "    - **Brier score**: gentler than log loss but still penalizes proportional to the distance from the expected value\n",
    "    - **AUROC**: summarizes the likelihood of the model predicting a higher probability for true positive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import log_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Log loss = $-\\sum_{c=1}^{M}y_{o,c} \\log (p_{o,c})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_probs = [x for x in np.linspace(0, 1, num=100)]\n",
    "# evaluate predictions for a 0 true value\n",
    "losses_0 = [log_loss(y_true=[0], y_pred=[x], labels=[0,1]) for x in pred_probs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f640c371320>]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXuElEQVR4nO3dfZAcd33n8fd3nvZZ0soay7ItIRzs2MYhMtkodpEAAZz4uBSGulRiVyVxqlwnIHBF7pJUSFKVwN0lRaoC3CVFkSjBwUl4cpwAKg6S+Iw5h1RsWINsZBtjI2wsIUur59U+zEx3f++P7tldybva0e70zPxWn1fVVD/9ZvvbXvmrn76/X3ebuyMiIuEpdDsAERFZGSVwEZFAKYGLiARKCVxEJFBK4CIigSp18mSbNm3y7du3d/KUIiLBe/TRR4+6e/Xc/R1N4Nu3b2d8fLyTpxQRCZ6ZPb/YfpVQREQCpQQuIhKoZRO4mfWb2dfM7DEze8LM3p/t/7iZfc/M9mafHfmHKyIiTa3UwGvAG9z9jJmVga+a2ZeyY7/l7vflF56IiCxl2QTu6cNSzmSb5eyjB6iIiHRZSzVwMyua2V7gCHC/uz+SHfpDM3vczD5sZn25RSkiIi/RUgJ399jddwBXAjvN7Abgd4BrgR8HNgK/vdh3zWyXmY2b2fjExESbwhYRkQuaheLuJ4EHgVvd/ZCnasBfAzuX+M5udx9z97Fq9SXz0EVE1rRDp2b44L88zfeOTrX9Z7cyC6VqZhuy9QHgFuDbZrYl22fAW4F9bY9ORCRwPzg5y599+Vm+f3y67T+7lVkoW4B7zKxImvDvdfcvmNmXzawKGLAXeEfboxMRCVwUJwCUCtb2n93KLJTHgRsX2f+GtkcjIrLGREk6aS+PBK47MUVEcjSXwIvtT7dK4CIiOcqzhKIELiKSo/keuBK4iEhQorhZA1cJRUQkKFGSlVDUAxcRCUuzB15WD1xEJCzNHnhRPXARkbA0BzHLmoUiIhKWZgmlqAQuIhKWRnMeuG7kEREJS9wsoagGLiISlmYNXCUUEZHAaBqhiEigoiTBDArqgYuIhKURey69b1ACFxHJVZwkudS/QQlcRCRXjdhzeQ4KKIGLiOQqTpxyDnPAQQlcRCRXkUooIiJhSgcxu5TAzazfzL5mZo+Z2RNm9v5s/8vN7BEze9bMPmNmlVwiFBEJWJx4Lk8ihNZ64DXgDe7+o8AO4FYzuwn4Y+DD7v4K4ARwVy4RiogErBEn3ZtG6Kkz2WY5+zjwBuC+bP89wFtziVBEJGBx0uVZKGZWNLO9wBHgfuC7wEl3j7ImB4ArlvjuLjMbN7PxiYmJdsQsIhKMRuwUu3kjj7vH7r4DuBLYCVzb6gncfbe7j7n7WLVaXWGYIiJhipIklycRwgXOQnH3k8CDwM3ABjMrZYeuBA62OTYRkeDFiXdvGqGZVc1sQ7Y+ANwCPEWayH8+a3Yn8PlcIhQRCVieg5il5ZuwBbjHzIqkCf9ed/+CmT0JfNrM/ifwTeBjuUQoIhKwPO/EXDaBu/vjwI2L7N9PWg8XEZElNGKnv9wDNXAREbkw6SCmnoUiIhKcKO7iIKaIiKxclHhvTCMUEZELEydOSW/kEREJTyNOKKmEIiISnkhv5BERCVOUdPlZKCIisjI98ywUERG5MHGsQUwRkSA1kkQ1cBGREEWxaxaKiEho3J0oUQIXEQlOnDgAJT0LRUQkLNFcAlcPXEQkKHMJXCUUEZGwRHECoGmEIiKhUQlFRCRQUdwsoagHLiISlCjJSijqgYuIhGW+B96lBG5mW83sQTN70syeMLP3ZPvfZ2YHzWxv9nlzLhGKiAQqynke+LJvpQci4Dfc/RtmNgI8amb3Z8c+7O5/kktkIiKBmyuh5NQDXzaBu/sh4FC2PmlmTwFX5BKNiMga0vUSykJmth24EXgk2/VuM3vczO42s9ElvrPLzMbNbHxiYmJVwYqIhKRZQil3+1Z6MxsG/gH4dXc/DXwU+CFgB2kP/YOLfc/dd7v7mLuPVavVNoQsIhKG5o08xW72wM2sTJq8P+Hu/wjg7ofdPXb3BPhLYGcuEYqIBKrrN/KYmQEfA55y9w8t2L9lQbO3AfvaH56ISLjyvpGnlVkorwF+GfiWme3N9v0ucIeZ7QAceA54ey4RiogEqpHzjTytzEL5KrDY2b/Y/nBERNaOuJdmoYiISOvm54HrWSgiIkGZn0aoHriISFCag5hdnUYoIiIXrpHNA+/6jTwiInJhmi81Vg9cRCQwjW7fyCMiIisTN0somoUiIhKW5iyUonrgIiJhaWSzUNQDFxEJTJz0wNMIRUTkws31wFVCEREJS5w4xYKRPtS1/ZTARURy0kiS3MonoAQuIpKbKHbKSuAiIuFpllDyogQuIpKTRpzk9hwUUAIXEclNnHhut9GDEriISG4asef2MgdQAhcRyU2UJN3tgZvZVjN70MyeNLMnzOw92f6NZna/mT2TLUdzi1JEJEBRDwxiRsBvuPv1wE3Au8zseuC9wAPufjXwQLYtIiKZKE5yew4KtJDA3f2Qu38jW58EngKuAG4D7sma3QO8Na8gRURC1FODmGa2HbgReATY7O6HskMvApuX+M4uMxs3s/GJiYlVhCoiEpZ0ELMHEriZDQP/APy6u59eeMzdHfDFvufuu919zN3HqtXqqoIVEQlJOojZ5VkoZlYmTd6fcPd/zHYfNrMt2fEtwJF8QhQRCVMUd3kQ09LHaH0MeMrdP7Tg0B7gzmz9TuDz7Q9PRCRcUeK5PUoWoNRCm9cAvwx8y8z2Zvt+F/gAcK+Z3QU8D/xCPiGKiIQpSvK9kWfZBO7uXwWW+ivkje0NR0Rk7YjipDcGMUVE5MJEcQ9NIxQRkdZFSaJnoYiIhCjqpRt5RESkdZGeRigiEqa0hKIeuIhIcDSIKSISqHQeuBK4iEhworgHnoUiIiIXTrNQREQCpRKKiEiA3D19oYOmEYqIhCVK0lckqAcuIhKYKM4SuAYxRUTCEiUJQK7PA1cCFxHJQbMH3tU38oiIyIVrZD1wlVBERAITaxBTRCRMc4OYSuAiImGZm0aoQUwRkbBEcVYD7+aNPGZ2t5kdMbN9C/a9z8wOmtne7PPm3CIUEQlQswfe7WmEHwduXWT/h919R/b5YnvDEhEJ2/w0wi72wN39IeB4bhGIiKxB89MIe7MG/m4zezwrsYwu1cjMdpnZuJmNT0xMrOJ0IiLh6OVphB8FfgjYARwCPrhUQ3ff7e5j7j5WrVZXeDoRkbA0emEQczHuftjdY3dPgL8EdrY3LBGRsMU9Moj5Ema2ZcHm24B9S7UVEbkYdeJZKKXlGpjZp4DXA5vM7ADwB8DrzWwH4MBzwNtzi1BEJEDNEko5x2ehLJvA3f2ORXZ/LIdYRETWjGYJRU8jFBEJTKNXa+AiInJ+cdKjs1BEROT8Gnqhg4hImJqzUPIcxFQCFxHJQbOEoh64iEhgGrEGMUVEgjT3LBSVUEREwjL3NEKVUEREwqJ3YoqIBCrSnZgiImGK4oRSwTBTAhcRCUqceK5v4wElcBGRXDRiz/U2elACFxHJRZQk6oGLiIQoSjzXGSigBC4ikot0EFMlFBGR4EQaxBQRCVMUq4QiIhKkdBBTJRQRkeD0RA/czO42syNmtm/Bvo1mdr+ZPZMtR3ONUkQkML1SA/84cOs5+94LPODuVwMPZNsiIpJJpxF2uYTi7g8Bx8/ZfRtwT7Z+D/DWNsclIhK05rNQ8rTSvx42u/uhbP1FYPNSDc1sl5mNm9n4xMTECk8nIhKWKO6NEsp5ubsDfp7ju919zN3HqtXqak8nIhKEKOndG3kOm9kWgGx5pH0hiYiEr1cGMRezB7gzW78T+Hx7whERWRuiXngaoZl9Cvh34IfN7ICZ3QV8ALjFzJ4B3pRti4hIJi2h5NsDLy3XwN3vWOLQG9sci4jImhHEIKaIiLyUHicrIhKoKNazUEREgqQeuIhIoHp5GqGIiJxHQ2/kEREJU6wSiohImNJphOqBi4gEpxM38iiBi4i0WZI4iaNBTBGR0DSSBICySigiImGJk/QJ20WVUEREwtKI0wSuGriISGCaPXAlcBGRwERxWgPXNEIRkcBEWQ+8rFkoIiJhieLmIKZ64CIiQZmfRqgeuIhIUDSNUEQkUI3mIGbOJZRl34l5Pmb2HDAJxEDk7mPtCEpEJGRxhwYxV5XAMz/t7kfb8HNERNaERqwSiohIkJrzwHv9WSgO/IuZPWpmuxZrYGa7zGzczMYnJiZWeToRkd4XyiDmT7r7q4H/ALzLzF57bgN33+3uY+4+Vq1WV3k6EZHe1wjhRh53P5gtjwCfBXa2IygRkZA9/eJpAPrLxVzPs+IEbmZDZjbSXAd+BtjXrsBEREL02Asn+ZN//g6vu6bKdZety/Vcq5mFshn4rJk1f84n3f2f2hKViEiAjk/V+bVPfIPqSB//6xd3UMi5Br7iBO7u+4EfbWMsIiLBihPnPZ/+JhOTNe57582MDlVyP2c75oGLiFzUTk03+M37HuNfnznKH73tR3jVlRs6cl4lcBGRVXjshZO865Pf4MVTs/z+z13PHTu3duzcSuAiIitwerbB7v+3n7946LtcOtLP37/jZm7cNtrRGJTARUQuwGwj5u8efp6PPPgsJ6Yb3Lbjct7/lleyYTD/mve5lMBFRFrw/LEpPvHI97l3/AVOTjf4qas38du3XssNV6zvWkxK4CIiSzh2psaX9r3IFx7/AQ/vP06xYPzsKzfzKzdv56arLul2eErgIiJN7s4zR87w4LeP8JWnJ/jac8eJE+eq6hD/9U3X8Is/vpXL1vd3O8w5SuAictFKEmf/0Sm+/txxHt5/jIf3H+Pw6RoA1142wttfexU/96rLuW7LCNlNiz1FCVxELgruzqFTs+w7eIp9B0+x98Ap9n7/BKdnIwCqI338xMs38ppXbOJ111S5fMNAlyNenhK4iKw5p6YbPDsxybNHzvDtFyf59qFJnj48yfGpOgAFg2s2j/AfX7WFG7eO8mPbR7lq01BP9rLPRwlcRII0XY94/th09pniuWNTfHdiiu8dnWJisjbXbqBc5JrLRrjlus1cf/k6brhiPddtGWGwEn76C/8KRGTNcXdOz0QcOj3DD07OcPDEDAdPznLgxDQvnJjhwPFpjmW96aZLhiq8fNMQr7umyisuHebqS4d5xaXDbB0dzP2hUt2iBC4iHTVTj5mYrDFxZpYjp2scPj3L4cl0+eKpWV48PcvhU7NM1eOzvlcuGpdvGGDr6CC3XL+ZrRsH2bZxkO2XDLHtkkHWD5S7dEXdowQuIqsSxQknZxocn6pz7Eyd41N1jk/VODZV5+iZGsfOpMujZ+pMTNY4U4te8jPKRaM63Mdl6/u59rKRdBBx/QBbNvSzZf0AV44OUB3uW7M96ZVSAhcRIH0c6uRsg1MzDU5OZ8uZBqem65ycbnBiusHJmeZ6nRNTdU5k7RZjBqODFS4ZqnDJcIVXXr6O6kgfm4b7uHSkj+pIH5eO9LN5XR+jgxUl5xVQAhdZA9yd2UbCZK3B5GzEmdmIydmIydl0+3S2bK6fnmlky4hT2frk7Et7xgsNVYpsGKywYbDM6GCFK0cHGc3WLxmuzCXrjcMVNg5V2DhYoZTzW9kvdkrgIl3SiBOmazFT9YipWsRUPWa6FnGmFjFdjzlTy/bXIs7U4nRZT5PzVNZucjZiKtsXZS/SPZ/hvhLr+kuM9JdZP1Dm8g1pyWLdQLrdXG4YKLN+cOGyQqWkZNxrlMBFlpAkTi1KmGnETNcjZhsxM/Wzt6frMTONmJl6uj5dj5mppwl4em5/uj1Vi5ipx2mirkc04uUTLqSliKFKiaG+IkN9JUb6Sgz1ldg2NMhw//z2SH+Z4f4Sw31FRvrKjGSJeqS/xLrsWFFlijVFCVyCkSROPU6YbcTUosWXs43mMmY2SqhlSXQ2WngsSbez/TP1+WMz2afZ7kJVigUGKkUGykUG+4oMVooMlktsHKqwdXSQgUqRoUqRwb4Sg+V02dwe7isyWCkxnCXkoUqasAfKRdWHZVFK4NKSKE6oxwn1KP3Uovnt2ty+eO54PU6oNdJ9tazNwna1aP54fe54mjSb67WF69l3V6pg0F9OE2t/uUhfuTC3PlgpsXGoSH+2b6BSnGvbTMb95QIDlTTp9mf7B5uJujLfTjVf6aRVJXAzuxX430AR+Ct3/0BboroINXuXjTihETuNLDmetX3WvoR6dG67hHrsc9vNZe2c7Xr23XQZZ/t9LvHOJ+V4bruF8mpLKqUCfdmnUizQXy7O7ysXGekvsSlbT9ulybOvlLbrLxfoL6UJuL+UJeNS+nP6y/PLvmabcpH+UpFy0YK7TVpkOStO4GZWBD4C3AIcAL5uZnvc/cl2BXch3J04caIkTWpR7DSSdLlwvZnoFraLkjShRQvaNI83Yic6J7FGSZrsoiShEaU/e7F251uP4rMTdtyuDLlAwdKEWS6mybJSKrx0u1hgsFLK9huVUnHuWF9pvk1lwXpfubBImzRhNrebCfes9sWCSgEibbSaHvhO4Fl33w9gZp8GbgPansD/9IFn+Nzeg0RZolssAbc6ILRaxYJRKRYoFY1yMU166TJdLxXO3tdfTtuXs++ctZ4l01LB5pNrYT7JlopG37ltigXKzWWxmXQLc+dbmHA1YCWytq0mgV8BvLBg+wDwE+c2MrNdwC6Abdu2rehEl470cd2WdZQLRilLZqUFybJULMwfKxrlQrpcuL+ZXJtJ9NzvNxNkqWBzybNcLMz9rGay1D/DRaRX5D6I6e67gd0AY2NjK+om375zG7fvXFnyFxFZq1YzZH4Q2Lpg+8psn4iIdMBqEvjXgavN7OVmVgFuB/a0JywREVnOikso7h6Z2buBfyadRni3uz/RtshEROS8VlUDd/cvAl9sUywiInIBdNuYiEiglMBFRAKlBC4iEiglcBGRQJl7Z25BBzCzCeD5FX59E3C0jeGEQNd8cdA1XxxWc80vc/fquTs7msBXw8zG3X2s23F0kq754qBrvjjkcc0qoYiIBEoJXEQkUCEl8N3dDqALdM0XB13zxaHt1xxMDVxERM4WUg9cREQWUAIXEQlUzyVwM7vVzJ42s2fN7L2LHO8zs89kxx8xs+2dj7K9Wrjm/2ZmT5rZ42b2gJm9rBtxttNy17yg3X8yMzezoKectXK9ZvYL2e/5CTP7ZKdjbLcW/lxvM7MHzeyb2Z/tN3cjznYys7vN7IiZ7VviuJnZn2b/TR43s1ev6oTu3jMf0sfSfhe4CqgAjwHXn9Pm14A/z9ZvBz7T7bg7cM0/DQxm6++8GK45azcCPAQ8DIx1O+6cf8dXA98ERrPtS7sddweueTfwzmz9euC5bsfdhut+LfBqYN8Sx98MfAkw4CbgkdWcr9d64HMvSnb3OtB8UfJCtwH3ZOv3AW+0sF9Uuew1u/uD7j6dbT5M+vajkLXyewb4H8AfA7OdDC4HrVzvfwY+4u4nANz9SIdjbLdWrtmBddn6euAHHYwvF+7+EHD8PE1uA/7GUw8DG8xsy0rP12sJfLEXJV+xVBt3j4BTwCUdiS4frVzzQneR/g0esmWvOfun5VZ3/z+dDCwnrfyOrwGuMbN/M7OHzezWjkWXj1au+X3AL5nZAdL3CvyXzoTWVRf6//t55f5SY2kfM/slYAx4XbdjyZOZFYAPAb/a5VA6qURaRnk96b+wHjKzH3H3k12NKl93AB939w+a2c3A35rZDe6edDuwUPRaD7yVFyXPtTGzEuk/vY51JLp8tPRyaDN7E/B7wFvcvdah2PKy3DWPADcAXzGz50hrhXsCHshs5Xd8ANjj7g13/x7wHdKEHqpWrvku4F4Ad/93oJ/0gU9rWVtfBt9rCbyVFyXvAe7M1n8e+LJnowOBWvaazexG4C9Ik3fotVFY5prd/ZS7b3L37e6+nbTu/xZ3H+9OuKvWyp/rz5H2vjGzTaQllf2dDLLNWrnm7wNvBDCz60gT+ERHo+y8PcCvZLNRbgJOufuhFf+0bo/aLjFK+x3SEezfy/b9d9L/gSH9Jf898CzwNeCqbsfcgWv+v8BhYG/22dPtmPO+5nPafoWAZ6G0+Ds20rLRk8C3gNu7HXMHrvl64N9IZ6jsBX6m2zG34Zo/BRwCGqT/qroLeAfwjgW/549k/02+tdo/17qVXkQkUL1WQhERkRYpgYuIBEoJXEQkUErgIiKBUgIXEQmUEriISKCUwEVEAvX/AcFN6zYYyQT6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(pred_probs, losses_0, label='true=0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "actual_labels = [\"bam\", \"ham\", \"spam\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.1094237467877998e-15"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(actual_labels,  [[1, 0, 0], [0, 1, 0], [0, 0, 1]]) # 모두 정답"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "34.538776394910684"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(actual_labels,  [[0, 1, 0], [1, 0, 0], [0, 1, 0]]) # 완벽히 틀린 경우"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.14462152754328852"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 적당히 틀린 경우. accuracy=1.0 이지만. Probability Score는 0이 아님\n",
    "log_loss(actual_labels,  [[0.8, 0, 0.2], [0.1, 0.9, 0], [0, 0.1, 0.9]]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Brier Score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$BS = \\frac{1}{N} \\sum_{t=1}^N(f_t - o_t)^2  $$\n",
    "where $f_{t}$ is the probability that was predicted, $o_{t}$ the actual label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import brier_score_loss\n",
    "\n",
    "pred_probs = [x for x in np.linspace(0, 1, num=100)]\n",
    "# evaluate predictions for a 0 true value\n",
    "losses_1 = [brier_score_loss(y_true=[0], y_prob=[x], pos_label=[1]) for x in pred_probs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f640c30fd30>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXRU9d3H8feX7EAgECJLSAjIoqwCYdPW2qcuSFVcakVFxYJoW2sfW59Wu2iLdrO2tlbqjoiIiNa2qFjqgooLkLBDEAgRSNiysIaQdX7PH0k9KQIZyGTuLJ/XOTlnZu4l93Mzkw83v7uZcw4REQl/rbwOICIigaFCFxGJECp0EZEIoUIXEYkQKnQRkQgR69WCO3Xq5LKysrxavIhIWFq+fHmpcy7tWNM8K/SsrCxyc3O9WryISFgys23Hm6YhFxGRCKFCFxGJECp0EZEIoUIXEYkQKnQRkQjRZKGb2QwzKzazdceZbmb2iJnlm9kaMxsW+JgiItIUf7bQZwJjTzD9YqBPw9dU4LHmxxIRkZPVZKE75z4A9p5glvHALFdvCZBiZl0DFVBEJFL4fI5fvZFH4d6KFvn+gRhDTwcKGz0vanjtC8xsqpnlmlluSUlJABYtIhI+/vJuPk8t/owP80tb5PsHdaeoc+5J51y2cy47Le2YZ66KiESk9zeV8Kd3NnHlsHQmjMhokWUEotB3AI3TdW94TUREgB37j/D9uSvp1zmZX10+CDNrkeUEotDnAzc2HO0yGjjgnNsVgO8rIhL2qmrr+M4LK6irczw2cThJ8TEttqwmL85lZi8C5wGdzKwIuA+IA3DOPQ4sAMYB+UAFcHNLhRURCTfTXstjdeF+Hp84nJ6d2rTosposdOfctU1Md8B3A5ZIRCRCvJxbyAtLt3PbV05n7MAuLb48nSkqItIC1u04wE//sY5zeqdy14V9g7JMFbqISIDtr6jmttnLSW0TzyMThhIbE5yq9ewGFyIikajO57hj7iqKD1bx0q2jSW2bELRlq9BFRALo4bc28cGmEn5z5SCGZnYI6rI15CIiEiAL1+/m0UX5TBiRwbUjM4O+fBW6iEgA5BeX88N5qxnSvT2/uGyAJxlU6CIizXSosoapz+eSENuKxyYOJzGu5U4eOhGNoYuINIPP5/jBvNVsK6vghSmj6JaS5FkWbaGLiDTDo4vyeStvDz/7+pmM7pXqaRYVuojIKXpnwx4efnsTVw5NZ9LZWV7HUaGLiJyK/OJy/nfuKgZ0a8evr2y5KyieDBW6iMhJOtiwEzQ+thVP3JDt2U7Qo2mnqIjISfD5HP87dxXbG3aCpnu4E/Ro2kIXETkJf3xrE+9+Wsx9l/ZnlMc7QY+mQhcR8dPra3Z+fiboxNE9vI7zBSp0ERE/rNtxgLteXk12jw5MGz8wJHaCHk2FLiLShNLyKqbOyqVD63gemzic+NjQrE7tFBUROYHqWh/fnr2cssPVvHLb2aQlB+9yuCdLhS4ichzOOX7+j3XkbN3HX64dyqDu7b2OdEKh+XeDiEgImPnxVl7KLeT2r/bm0iHdvI7TJBW6iMgxLN5cwv2v53Fh/8784ILg3BO0uVToIiJH2VJSzndeWEHfzsk8fM1ZtGoVeke0HIsKXUSkkf0V1Ux5Lpf4mFY8dWM2bRLCZ1dj+CQVEWlhNXU+vjtnBTv2HeGFW0aR0bG115FOigpdRIT6I1p++dp6Psov46GrhzAiq6PXkU6ahlxERIDnPt7K7CXbufXcXnxjeHev45wSFbqIRL1FG4uZ9noeF/TvzI/HnuF1nFOmQheRqLZx9yG+N2clZ3Rpx5/C6IiWY1Ghi0jUKjlUxeTncmgdH8Mzk8LriJZjCe/0IiKnqLKmjqnP51JaXsW8W8fQtX3o3KjiVKnQRSTq+HyOu15ezcrt+3l84jAGd0/xOlJA+DXkYmZjzWyjmeWb2d3HmJ5pZovMbKWZrTGzcYGPKiISGA+/vYnX1+zi7ovPYOzArl7HCZgmC93MYoDpwMVAf+BaM+t/1Gw/A+Y554YCE4C/BjqoiEggvJxbyF/ezeea7AxuPbeX13ECyp8t9JFAvnOuwDlXDcwFxh81jwPaNTxuD+wMXEQRkcD4OL+Ue15dyzm9U3ngitC861Bz+FPo6UBho+dFDa819gtgopkVAQuA7x3rG5nZVDPLNbPckpKSU4grInJq8osPcevs5fTs1Ia/Xj+cuJjIO8gvUGt0LTDTOdcdGAc8b2Zf+N7OuSedc9nOuey0tLQALVpE5MRKDlUx6dkcEmJjmDFpBO2T4ryO1CL8KfQdQEaj590bXmtsMjAPwDn3CZAIdApEQBGR5qiormXKczmUlVfzzE3ZYXfBrZPhT6HnAH3MrKeZxVO/03P+UfNsB74GYGZnUl/oGlMREU/V+Rx3vLiStTsO8Mi1QxmSERmHJx5Pk4XunKsFbgcWAhuoP5plvZlNM7PLGmb7IXCLma0GXgQmOedcS4UWEWmKc45pr63n7Q3F3HfpAC7o39nrSC3OrxOLnHMLqN/Z2fi1exs9zgPOCWw0EZFT9/Tiz3juk21M+VJPbjo7y+s4QRF5u3lFJOrNX72TXy3YwNcHd+Un4870Ok7QqNBFJKIsKSjjrnmrGZnVkT9cPSSsr554slToIhIxNu85xNRZuWR0TOLJG4eTGBfjdaSgUqGLSETYfaCSm2YsIyEuhpk3jySldbzXkYJOhS4iYe9gZQ2Tnl3GwcpaZt48IqKPNT8RFbqIhLWq2jpunbWc/OJyHp84nAHd2nsdyTO6HrqIhC2fz/GDeav5pKCMh68Zwpf6RPcJ6tpCF5Gw5Jzjl6+t5401u/jJuDO4Ymh3ryN5ToUuImHpr+9t+fzEoannnu51nJCgQheRsDMvp5DfL9zI5Wd1i6oTh5qiQheRsLJw/W7ufnUN5/ZN48FvRNeJQ01RoYtI2FhaUMb3XlzJoO4pPHb9MOJjVWGN6achImEhb+dBpszKJaNDEs9OGkGbBB2kdzQVuoiEvK2lh7lxxjLaJsQya/IoOraJvrNA/aFCF5GQtudgJROfWUqdz8fzk0eSnpLkdaSQpUIXkZB1oKKGG59Zxr7D1cy8eSS9T0v2OlJI0yCUiISkw1W1TJq5jM9KD/PszSMi/vZxgaBCF5GQU1Vbx9Tnc1lTdIDp1w3jnN7RfUq/vzTkIiIhpbbOxx0vruSj/DIevGowYwd28TpS2FChi0jI8PkcP/rbGhau38N9l/bnquG6PsvJUKGLSEhwzvGL19bz6ood3Hl+X24+p6fXkcKOCl1EQsKDCzcy65NtTD23F3d8rbfXccKSCl1EPDd9UT6PvbeF60dlcs/FZ2Cm67OcChW6iHjqmQ8/4/cLN3LF0HTuHz9QZd4MKnQR8cycpdu5//U8Lh7Yhd9/Y7CunNhMKnQR8cSrK4r46T/W8tV+afx5wlBiY1RHzaWfoIgE3etrdnLXy6sZ0yuVxyYO12VwA0Q/RREJqoXrd/P9uavI7tGRp2/KJjEuxutIEUOFLiJBs+jTYm6fs4LB3dsz4+YRtI7X1UcCSYUuIkHx/qYSbp29nH5dkpl580ja6gYVAedXoZvZWDPbaGb5Znb3ceb5ppnlmdl6M5sT2JgiEs4+3FzK1Fm59E5ry+zJo2ifFOd1pIjU5H+RZhYDTAcuAIqAHDOb75zLazRPH+Ae4Bzn3D4zO62lAotIePl4SylTZuXQs1MbZk8ZRUpr3W2opfizhT4SyHfOFTjnqoG5wPij5rkFmO6c2wfgnCsObEwRCUdLCsqYPDOXzI6teWGKbh3X0vwp9HSgsNHzoobXGusL9DWzj8xsiZmNPdY3MrOpZpZrZrklJSWnllhEwsKSgjJufjaH7h2SeGHKaFLbJngdKeIFaqdoLNAHOA+4FnjKzL5wexHn3JPOuWznXHZaWlqAFi0ioWZpQ5mnd0hizi2jSUtWmQeDP4W+A8ho9Lx7w2uNFQHznXM1zrnPgE3UF7yIRJklBWVMejaHbimJzLlllMo8iPwp9Bygj5n1NLN4YAIw/6h5/kH91jlm1on6IZiCAOYUkTDw8ZbSz4dZ5k4dw2nJiV5HiipNFrpzrha4HVgIbADmOefWm9k0M7usYbaFQJmZ5QGLgP9zzpW1VGgRCT0f5ZfyrZk5ZHTUMItXzDnnyYKzs7Ndbm6uJ8sWkcB6f1MJU2flkpXahhduGUUn7QBtMWa23DmXfaxpOlVLRJrlnQ17+PbsFZx+WltmTx6po1k8pEIXkVO2cP1ubp+zgjO6tOP5ySN10pDHVOgickpeW72TO19axcD09jz3rZE6nT8E6OJcInLSXllexPfnrmRoZgrPT1aZhwptoYvISZmzdDs/+ftazumdylM3ZusSuCFE74SI+O3pxQU88MYGvtovjccmDtfNKUKMCl1EmuSc4y/v5vPHtzZx8cAu/HnCUN02LgSp0EXkhJxz/PZfn/LE+wVcOSydB68arBs6hygVuogcV53P8fN/rmPO0u1MHJ3JtMsG0qqVeR1LjkOFLiLHVFPn466XV/PPVTv59nmn86OL+mGmMg9lKnQR+YLKmjpun7OCtzcU86Ox/fjOeb29jiR+UKGLyH85WFnDlOdyydm6l/svH8gNo3t4HUn8pEIXkc+Vlldx04xlbNx9iD9dcxbjzzr65mQSylToIgJA0b4KbnxmGTsPHOHpm7I5r5/u9R5uVOgiwqe7D3LTjGUcqa7jhSmjGN6jo9eR5BSo0EWiXM7WvUyemUNSfAwv33Y2/bokex1JTpEKXSSKvZ23h+/OWUF6ShKzJo+ke4fWXkeSZlChi0SpucvqL7I1ML09z04aoRtTRAAVukiUaXxdlq/0TeOv1w+jTYKqIBLoXRSJIrV1Pu6dv545S7dz5bB0fnfVYOJ0XZaIoUIXiRIV1bXc8eJK3t5QrFP5I5QKXSQKlJVX8a3ncllbtJ/7xw/ghjFZXkeSFqBCF4lwBSXl3Dwzh90HKnl84nAuHNDF60jSQlToIhEsd+tepszKpZUZL04dzbDMDl5HkhakQheJUG+s2cWd81aRnpLEzJtH0CO1jdeRpIWp0EUijHOOx97fwoP/2kh2jw48dWM2HdrEex1LgkCFLhJBaup8/Ozv63gpt5DLhnTjwW8M1o2co4gKXSRCHKio4TtzlvNRfhl3/E9v7rygrw5LjDIqdJEIsK3sMDfPzKFwbwUPXT2Ebwzv7nUk8YAKXSTMLS0o47bZy3HA7MmjGNUr1etI4hEVukgYm5dTyE//sZaMjq2ZcdMIsjrpSJZo5tdFHMxsrJltNLN8M7v7BPNdZWbOzLIDF1FEjlbnczzweh4/+tsaRvdK5e/fOUdlLk1voZtZDDAduAAoAnLMbL5zLu+o+ZKB7wNLWyKoiNQ7WFnD919cyaKNJdw0pgc/v6Q/sbrAluDfFvpIIN85V+CcqwbmAuOPMd/9wO+AygDmE5FGPis9zBXTP2Lx5lLuv3wgvxw/UGUun/Pnk5AOFDZ6XtTw2ufMbBiQ4Zx740TfyMymmlmumeWWlJScdFiRaLZ4cwnjH/2QvYereX7yKG4Y3cPrSBJimv1fu5m1Av4I/LCpeZ1zTzrnsp1z2Wlpac1dtEhUcM7x1AcF3DRjGV3bJzH/9i8x5nQdySJf5M9RLjuAjEbPuze89h/JwEDgvYaTGLoA883sMudcbqCCikSjypo67nl1LX9fuYOxA7rwh28O0d2F5Lj8+WTkAH3MrCf1RT4BuO4/E51zB4BO/3luZu8Bd6nMRZpnx/4j3Pb8ctbtPMBdF/blu1/trTM/5YSaLHTnXK2Z3Q4sBGKAGc659WY2Dch1zs1v6ZAi0ebj/FJuf3ElNbU+nrohm/P7d/Y6koQBv/52c84tABYc9dq9x5n3vObHEolOzjmeXvwZv3lzA6enteWJG4bTK62t17EkTGgwTiRElFfV8uNX1vDG2l1cPLALv796CG01Xi4nQZ8WkRCQX1zObbOXU1BSzj0Xn8HUc3tpvFxOmgpdxGOvr9nJj19ZQ2JcDLOnjOLs0zs1/Y9EjkGFLuKR6lofv16wgZkfb2VYZgrTrx9G1/ZJXseSMKZCF/HAzv1H+O6cFazcvp9vndOTe8adQZxO4ZdmUqGLBNmiT4u5c94qausc068bxtcHd/U6kkQIFbpIkNTW+Xjo35t4/P0t9O/ajunXD6OnLnkrAaRCFwmCHfuPcMeLK1m+bR/Xjcrk3kv66+bNEnAqdJEW9lbeHu56eTW1dT4euXYolw3p5nUkiVAqdJEWUlVbx28WfMrMj7cyoFs7Hr1OQyzSslToIi0gv7icO15cSd6ug0w6O4t7xp1BQqyGWKRlqdBFAsg5x0s5hfzytTwS41rxzE3ZfO1MXVhLgkOFLhIg+yuquefVtby5bjdjeqXy8DVn0aV9otexJIqo0EUC4OMtpfzgpdWUHa7inovP4JYv96JVK12LRYJLhS7SDFW1dfzh35t4anEBPVPb8NSN5zCoe3uvY0mUUqGLnKKNuw/x/bkr+XT3ISaOzuQn486kdbx+pcQ7+vSJnKQ6n+OZDwt4aOEm2iXFMmNSNv9zhnZ8ivdU6CInoXBvBT+ct5plW/dy0YDO/PqKQaS2TfA6lgigQhfxi3OOF5Zu59cLNhBjxh+uHsKVw9J1EwoJKSp0kSbs3H+EH/9tDYs3l/LlPp347VWDSU/Rdcsl9KjQRY7DOcfcnEJ+9cYGfM7xwOUDuX5UprbKJWSp0EWOoWhfBfe8upbFm0sZ0yuV3101mMzU1l7HEjkhFbpIIz6f4/kl2/jdvz7FgAcuH8h1IzN1kpCEBRW6SIP84nLu/tsacrft49y+afz6ioF076CtcgkfKnSJetW1Ph5/fwuPvptPUnyMjmCRsKVCl6i2fNs+7nl1DZv2lHPJ4K7cd+kA0pJ1XLmEJxW6RKUDR2p48F+fMmfZdrq2S9TZnhIRVOgSVZxzvLZmF9Ney2Pv4Sq+dU5P7rygL20T9Ksg4U+fYokaBSXl3PvP9XyYX8qg9PbMvHkEA9N1ZUSJHCp0iXhHquv463v5PPF+AQlxrZg2fgDXj+pBjA5FlAjjV6Gb2Vjgz0AM8LRz7rdHTf8BMAWoBUqAbznntgU4q8hJcc6xcP0e7n89jx37j3D5Wd34ydfP5LRk3UVIIlOThW5mMcB04AKgCMgxs/nOubxGs60Esp1zFWb2beBB4JqWCCzij/zicqa9nscHm0ro1zmZuVNHM7pXqtexRFqUP1voI4F851wBgJnNBcYDnxe6c25Ro/mXABMDGVLEXwcra3jk7c3M/HgrSXEx/PyS/tw4pgdxMa28jibS4vwp9HSgsNHzImDUCeafDLx5rAlmNhWYCpCZmelnRJGm1fkcL+cW8tC/N1J2uJpvDs/grov66ZhyiSoB3SlqZhOBbOArx5runHsSeBIgOzvbBXLZEr0+2VLG/a/nkbfrIMN7dGDGpBEM7p7idSyRoPOn0HcAGY2ed2947b+Y2fnAT4GvOOeqAhNP5PgKSsr57Zuf8u+8PaSnJPGXa4dyyeCuOmVfopY/hZ4D9DGzntQX+QTgusYzmNlQ4AlgrHOuOOApRRrZe7iaR97ZzOwl20iIbcX/XdSPyV/qSWJcjNfRRDzVZKE752rN7HZgIfWHLc5wzq03s2lArnNuPvB7oC3wcsPW0Xbn3GUtmFui0JHqOmZ89BmPv7eFw9W1TBiZyZ3n99U4uUgDv8bQnXMLgAVHvXZvo8fnBziXyOdq63y8sryIh9/exJ6DVZx/Zmd+PLYffTonex1NJKToTFEJWc453ly3m4cWbqSg9DBDM1N49LphjMjq6HU0kZCkQpeQ45zjvU0l/OHfG1m34yB9TmvLkzcM54L+nbXDU+QEVOgSUj7eUsof/72J3G376N4hiYeuHsIVQ9N13RURP6jQJSQsKSjj4bc2sfSzvXRul8ADlw/km9kZxMfqDE8Rf6nQxTPOOZYU7OWRdzbzSUEZackJ3Hdpf64dmalDEEVOgQpdgs45x4f5pfzlnXyWbd1LWnICP/v6mUwc3UNFLtIMKnQJGp/P8faGPUxflM/qogN0aZfILy8bwDUjMlTkIgGgQpcWV1PnY/6qnTzxwRY27Skns2NrfnPlIK4clk5CrIpcJFBU6NJiyqtqeSmnkGcWF7DzQCX9Oifzp2vO4pLBXYnV5WxFAk6FLgG352Alz360lReWbuNQZS0jszryqysGcV6/NB1HLtKCVOgSMGuLDvDMhwW8vmYXPue4eGBXpny5J0MzO3gdTSQqqNClWWrqfCxcv5vnPt5KztZ9tE2I5cYxWUw6O4vM1NZexxOJKip0OSUlh6qYu2w7Lyzdzu6DlWR0TOJnXz+Ta0ZkkJwY53U8kaikQhe/OefI2bqP55ds41/rdlFT5/hyn048cPlAvnrGaTo9X8RjKnRp0v6Kav62YgcvLttOfnE5yYmx3DA6i+tHZ3J6Wluv44lIAxW6HJPP5/ikoIyXcgr51/rdVNf6OCsjhQevGswlQ7rSOl4fHZFQo99K+S+Feyt4dcUOXllRSOHeI7RLjGXCiAwmjMikf7d2XscTkRNQoQuHKmt4c+1uXl1ZxJKCvZjBmF6p3HVhPy4a0EWn5YuECRV6lKqqreODTaX8c9UO3srbQ1Wtj6zU1vzwgr5cMSyd7h10yKFIuFGhR5HaOh9LCvby2uqdvLluFwcra+nQOo5rRmRwxdB0zspI0ZmcImFMhR7haup8LCkoY8HaXSxcv4e9h6tpEx/DRQO6cOlZ3fhS707E6boqIhFBhR6BKqprWby5lIXrd/N23h4OVtbSOj6G88/szLhBXTmvX5rGxUUikAo9Quw5WMmiT4t5e8MeFm8uparWR7vEWM7v35mxA7pwbl+VuEikU6GHqTqfY1Xhft7fWMy7G4tZt+MgAOkpSVw7MpML+ndmZM+OGk4RiSIq9DBSuLeCj/JLWZxfyoebSzlwpIZWBkMzO/B/F/Xja2eeRr/OydqxKRKlVOghrPhQJZ9sKWNJQRmfbClja1kFAJ3bJXBB/86c1y+NL/XuRErreI+TikgoUKGHCOcchXuPkLttL8s+q/8qKD0MQHJCLKN6deTGMVl8uU8nep/WVlvhIvIFKnSPHKmuY+2OA6wq3MfK7fvJ3baPkkNVACQnxjIyqyPXjMhgzOmpDOjWXlcyFJEmqdCD4Eh1HRt2H2T9zoOsLdrPmqIDbC4up87nAMjomMQ5p6cyPKsjwzM70K9LsgpcRE6aCj2AfD7Hjv1H2Lj7EBv3HOLT3YfYsOsgBSXlNHQ3HVrHMah7Cuef2ZmzMlI4KzOFTm0TvA0uIhFBhX4KDhypYXtZBZ+VHaagpJyCksNsKSlnS0k5lTW+z+fr3iGJM7okM25QVwZ0a8eAbu1IT0nS+LeItAi/Ct3MxgJ/BmKAp51zvz1qegIwCxgOlAHXOOe2BjZqcNT5HGWHq9hzoIqdB46w+0AlO/cfoWjfEQr3VbB9bwX7K2o+n98MurVPoldaG0b36kGf09rSp3Nb+nZO1q3YRCSomix0M4sBpgMXAEVAjpnNd87lNZptMrDPOdfbzCYAvwOuaYnA/qqp81FRVUdFTS2Hq+o4VFnDocpaDlXWsv9INfsrath3uJq9FdWUlVdTdriK4oNVlB2u/nxs+z/iY1vRPSWJ9A5JjBvUlR4dW9MjtQ1ZnVqTldpGZ2CKSEjwZwt9JJDvnCsAMLO5wHigcaGPB37R8PgV4FEzM+fcfzdjAMzLKeTJxQX4nMO5+i3q2joftT5HTZ2Pqtr6r6NL+VgS41qR2iaB1LbxdGqbQP+u7TgtOZHT2iXQpV0iXdsn0TUlkY6t42mlnZQiEuL8KfR0oLDR8yJg1PHmcc7VmtkBIBUobTyTmU0FpgJkZmaeUuCU1nENZ0NCKzNaGcTGtCK2lREbYyTGxpAQ14rE2BiS4mNokxBL6/gYkhNjaZcYR3JiHCmt42ifFKctaxGJKEHdKeqcexJ4EiA7O/uUtt4vHNCFCwd0CWguEZFI4M+Vm3YAGY2ed2947ZjzmFks0J76naMiIhIk/hR6DtDHzHqaWTwwAZh/1DzzgZsaHn8DeLclxs9FROT4mhxyaRgTvx1YSP1hizOcc+vNbBqQ65ybDzwDPG9m+cBe6ktfRESCyK8xdOfcAmDBUa/d2+hxJXB1YKOJiMjJ0N0PREQihApdRCRCqNBFRCKECl1EJEKYV0cXmlkJsO0U/3knjjoLNQponaOD1jk6NGedezjn0o41wbNCbw4zy3XOZXudI5i0ztFB6xwdWmqdNeQiIhIhVOgiIhEiXAv9Sa8DeEDrHB20ztGhRdY5LMfQRUTki8J1C11ERI6iQhcRiRAhXehmNtbMNppZvpndfYzpCWb2UsP0pWaWFfyUgeXHOv/AzPLMbI2ZvWNmPbzIGUhNrXOj+a4yM2dmYX+Imz/rbGbfbHiv15vZnGBnDDQ/PtuZZrbIzFY2fL7HeZEzUMxshpkVm9m640w3M3uk4eexxsyGNXuhzrmQ/KL+Ur1bgF5APLAa6H/UPN8BHm94PAF4yevcQVjnrwKtGx5/OxrWuWG+ZOADYAmQ7XXuILzPfYCVQIeG56d5nTsI6/wk8O2Gx/2BrV7nbuY6nwsMA9YdZ/o44E3AgNHA0uYuM5S30D+/ObVzrhr4z82pGxsPPNfw+BXga2YWzndzbnKdnXOLnHMVDU+XUH8HqXDmz/sMcD/wO6AymOFaiD/rfAsw3Tm3D8A5VxzkjIHmzzo7oF3D4/bAziDmCzjn3AfU3x/ieMYDs1y9JUCKmXVtzjJDudCPdXPq9OPN45yrBf5zc+pw5c86NzaZ+v/hw1mT69zwp2iGc+6NYAZrQf68z32Bvmb2kZktMbOxQUvXMvxZ518AE82siPr7L3wvONE8c7K/700K6k2iJXDMbCKQDXzF6ywtycxaAX8EJnkcJdhiqR92OY/6v8I+MLNBzrn9nqZqWdcCM51zfzCzMdTfBW2gc87ndbBwEcpb6NF4c2p/1hkzOx/4KXCZc64qSNlaSlPrnAwMBN4zs63UjzXOD/Mdo/68z0XAfOdcjXPuM2AT9QUfrvxZ58nAPADn3AGIxuYAAAEWSURBVCdAIvUXsYpUfv2+n4xQLvRovDl1k+tsZkOBJ6gv83AfV4Um1tk5d8A518k5l+Wcy6J+v8Flzrlcb+IGhD+f7X9Qv3WOmXWifgimIJghA8yfdd4OfA3AzM6kvtBLgpoyuOYDNzYc7TIaOOCc29Ws7+j1nuAm9hKPo37LZAvw04bXplH/Cw31b/jLQD6wDOjldeYgrPPbwB5gVcPXfK8zt/Q6HzXve4T5US5+vs9G/VBTHrAWmOB15iCsc3/gI+qPgFkFXOh15mau74vALqCG+r+4JgO3Abc1eo+nN/w81gbic61T/0VEIkQoD7mIiMhJUKGLiEQIFbqISIRQoYuIRAgVuohIhFChi4hECBW6iEiE+H8S7AQZ21DSIAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(pred_probs, losses_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ROC AUC Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# roc curve\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import roc_curve\n",
    "from matplotlib import pyplot\n",
    "# generate 2 class dataset\n",
    "X, y = make_classification(n_samples=1000, n_classes=2)\n",
    "# split into train/test sets\n",
    "trainX, testX, trainy, testy = train_test_split(X, y, test_size=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(500, 20)\n",
      "(500,)\n"
     ]
    }
   ],
   "source": [
    "print(trainX.shape)\n",
    "print(trainy.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.06540962  0.05569037 -1.76097949 -0.25088038 -2.06328798 -0.75387301\n",
      "  0.35076608  3.52514219  0.17562864 -1.20576816  0.29359279 -0.22094019\n",
      "  1.67877923  0.98146232 -1.06815427 -1.41501886  0.24429494 -0.67137172\n",
      "  2.23431233 -0.10425883]\n",
      "label: 0\n"
     ]
    }
   ],
   "source": [
    "print(trainX[0])\n",
    "print('label:', trainy[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/parkchanwoo/anaconda3/envs/torch/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='warn', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LogisticRegression()\n",
    "model.fit(trainX, trainy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "probs = model.predict_proba(testX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500, 2)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# keep probabilities for the positive outcome only\n",
    "probs = probs[:, 1]\n",
    "# calculate roc curve\n",
    "fpr, tpr, thresholds = roc_curve(testy, probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3yV5d3H8c8vYSaEEZIwAkmAsIeKERwUWSK4qIoWV21rS5dP++hThbrqaK0djg6rYitVW6sWRaNgbbUsFZQ4mkAUZJMAssMICRnX88dJ7CkN5EBOzn3Ofb7v1ysvzrjJ+d0k+XLluq/zu8w5h4iIxL4ErwsQEZHwUKCLiPiEAl1ExCcU6CIiPqFAFxHxiRZevXBaWprLycnx6uVFRGLS+++/v9M5l97Qc54Fek5ODgUFBV69vIhITDKzjUd7TlMuIiI+oUAXEfEJBbqIiE8o0EVEfEKBLiLiE40Gupk9YWbbzWzFUZ43M/u1ma0xs0IzGx7+MkVEpDGhjND/CEw6xvOTgb51H9OBR5peloiIHK9G16E75xabWc4xDpkCPOUCfXiXmVlHM+vmnNsaphpFpCEFs6FojtdVyHGocY6qmlra9DgZJt8X9s8fjjcWZQKbg+6X1D32X4FuZtMJjOLJysoKw0uLHEU8hN3GtwJ/Zo/ytg4JSdmhKtbtPEBigjG0h8Oa4TUi+k5R59wsYBZAXl6edtaQ8GgovOMh7LJHwdCpkPdVryuRYyg7VMVP53/Ms6s2k9M5ifsuHYb17twsrxWOQC8Fegbd71H3mEh4NDbabii8FXYSBWpqHZc+8g7rdhzgm2f35oYJ/WjTMrHZXi8cgZ4PXG9mzwIjgTLNn8sJOVpwNzbaVnhLlNlz8DAdk1qSmGD8YGJ/undsw7AeHZv9dRsNdDP7CzAGSDOzEuBHQEsA59yjwHzgPGANUA7op0qOT32QHy24FdgSI5xzvPRRKXe9UsyMSQO4YkQWk4Z0jdjrh7LK5YpGnnfAd8NWkcSWcFx8DA5yBbfEqC17D3Hr3CIWrNrBKVkdycvuFPEaPGufKzGusVH18VCQS4x7+aNSbp27gppaxx0XDOLaM3NITGiOdSzHpkCX0AWPxjWqFvlch7YtOblnR356yVB6piZ5VocCXUKfNgkOcQW5xLHqmlr+8NZ6qmpquX5cX8b0z+DsfumYRX5UHkyBHi+OFdqhTpsoxEUo3rKPGS8UUlRaxvnDuuGcw8w8D3NQoPtDKCPsY4W2glqkUZXVNfz2n2t4ZOFaOia15HdXDWfykK5REeT1FOjR6nhWj4QywlZoizTJhp3lPLpoLRed3J3bzx9Ep+RWXpf0XxTo0apoDmwrgq5DGz9WYS3SLA5WVvOP4s/44imZ9O+awps3jiGrs3cXPRujQI829SPz+jD/6jyvKxKJS0s+3cEPXyyidO8hhmS2JzcjJarDHBTo0eFYywFFJKLKyqv4yfxini8ooXdaMs9NP4PcjBSvywqJAt1rBbPh1f8N3NZyQBFP1dQ6Ln30HdbvPMh3xvThe+P7NmszrXBToEfSsdq8XvCQQlzEI7sPHqZj20AzrZvO7U9mx7YMyezgdVnHTZtER1L93Hiw7FEKcxGPOOd44f0Sxv5yIc8uD+zTc+7grjEZ5qAReuTpQqdIVCjZU84tc1ewePUOTs3uxIheqV6X1GQK9ObS0PRKqMsQRaRZzf2whNvmrsABd100mGtOzybBg2Za4aZAD7djdSHsOlQrV0SiQGpya07NSeXei4fQo1N0L0U8Hgr0cKsPc61WEYkaVTW1PL5kHdU1ju+N78vZ/dIZ3Tctqt62Hw4K9HAJfkNQ9ijNk4tEiRWlZcx4oZCVW/Zx4Undo6qZVrgp0JuqoSkWTauIeK6iqoZfv/kpjy1eR6ekVjx69XAmDenmdVnNSoHeVMGjck2xiESNjbvKeXzJOi45JZPbzh9Eh6SWXpfU7BTox+vI1SvquSISNQ5WVvP6ym1cMrwH/bum8M//G+PpDkKRpkA/Xkd2QdTKFZGosGj1Dm55sYgtZYcY1qMDuRkpcRXmoEAPXfBcuS56ikSNPQcPc8+8Yl78oJQ+6cn89Zux00wr3BTooTpyOaKIeK6+mdbGXeVcPzaX68flxlQzrXBToIeiYLZG5iJRZNeBSjoltSIxwZg5aQCZndoyuHts9l8JJzXnOpaC2TD7/H+3t9XIXMRTzjmeL9jM2F8u5C/LNwEwcXBXhXkdjdCPdKzNJrQkUcQzm3eXc8vcIpZ8upMROamc0buz1yVFHQU6HD3EFeQiUeHFD0q47aUVGHDPF4dw1YgsXzTTCjcFunYMEol6ae1aM6JXKj+5eCiZHdt6XU7Uiu9ADw5zbTIhEjWqamp5bNFaamrh+xP6MrpfOqP7pXtdVtSL70Cvn2ZRmItEjRWlZdw0p5CPt+5jysn/bqYljYvfQA9eiqgwF/FcRVUND73xKY8vWUdqciseu+ZUzh3c1euyYkpIyxbNbJKZrTKzNWY2s4Hns8xsgZl9aGaFZnZe+EsNs/rRuZYiikSFTbvL+cNb65g6vAdv3HC2wvwENDpCN7NE4GHgHKAEWG5m+c654qDDbgOed849YmaDgPlATjPUG14anYt4an9FFX9bsY3L8nrSr0sKC34wxlc7CEVaKFMuI4A1zrl1AGb2LDAFCA50B7Svu90B2BLOIkXEfxZ8sp1b5xaxbV8Fp2R1JDcjRWHeRKEEeiawOeh+CTDyiGPuBP5uZv8DJAMTGvpEZjYdmA6QlZV1vLWKiA/sPniYe14tZu6HpfTNaMecb58Zt820wi1cF0WvAP7onLvfzM4AnjazIc652uCDnHOzgFkAeXl5LkyvLSIxoqbWMfWRd9i0u5zvje/Ld8f2oXWL+G2mFW6hBHop0DPofo+6x4JdB0wCcM4tNbM2QBqwPRxFikhs27G/ks7JgWZat5w3kMxObRnYrX3jf1GOSyirXJYDfc2sl5m1AqYB+UccswkYD2BmA4E2wI5wFhoW9c22Zp8f2KRCRJqVc47nlm9i3P0Leea9QDOtCYO6KMybSaMjdOdctZldD7wOJAJPOOdWmtndQIFzLh/4P+BxM7uBwAXSrzjnom9KJXi3Ie00JNKsNu0qZ+aLhbyzdhcje6UyKjfN65J8L6Q5dOfcfAJLEYMfuyPodjFwVnhLayba/1Ok2c15v4TbX1pBYoLxk4uHcMVpaqYVCfH7TlERaTZd2rfmzD6d+fHFQ+jWQc20IkWBLiJNdri6lkcWrqXWOW44px9f6JvOF/qqmVakxc+ORfW9W0QkrP61eS8X/uYtHnxjNZt3lxONl8/iRfyM0NW7RSSsDh2u4YF/rOIPb60nI6UNv/9yHhMGdfG6rLgWP4EO6t0iEkab95Tz5DsbmTYii5mTB9C+TUuvS4p78RXoItIk++qaaV1e10xr4U1j6K4dhKKGAl1EQvLPTz7jlhdXsH1/BcOzOpGb0U5hHmUU6CJyTLsOVHL3q8W8/NEW+ndJ4dFrTiU3o53XZUkD4iPQg3cnEpGQ1dQ6Lnt0KZv3lHPDhH58e0wfWrWIn8VxscbfgV4wO7C6pX65ola4iIRk+/4K0pJbk5hg3Hr+QHp0SqJ/V7W4jXb+/q+2vndL9ihtBC0Sgtpax5/f3ci4Xy7iz3XNtMYP7KIwjxH+HqGDereIhGjDzoPMfLGQZet2c2afzpytd3rGHP8Huog06vmCzdz+0gpaJSZw3yVD+dJpPTFTM61Yo0AXETI7tmV0v3TumTKErh3aeF2OnCAFukgcqqyu4XcL1uKc48aJ/TkrN42z1K885inQReLMh5v2MOOFQlZ/doBLh/fAOafpFZ9QoIvEifLD1dz/99U88fZ6urZvwxNfyWPcADXT8hMFukicKN1ziKeXbeSqkVnMmDSAFDXT8h0FuoiPlR2q4rWirUwbkUXfLiksummMdhDyMQW6iE/9feU2bntpBbsOHiYvJ5XcjHYKc59ToIv4zM4DldyZv5JXC7cyoGsKv782T8204oQCXcRHamodUx95hy17K/jBxH588+w+tEz0d4cP+TcFuogPfLavgvR2gWZaP7pwMD06taVvF/VfiTf6r1skhtXWOp5etpHx9y/iz+9uBGDsgAyFeZzy7whdPdDF59btOMDMF4t4b/1uRuWmMaZ/htclicf8G+hFcwJ/qge6+NBzyzdxx8srad0igZ9PHcZlp/bQuz3Fx4EOgdG5eqCLD/XolMSY/oFmWhnt1UxLAvwd6CI+UVldw2/eXAPAD85VMy1pmD8vitbPn4v4wPsbd3Per5bw2wVr2L6/Auec1yVJlPLnCF3z5+IDByur+cXrq3hy6Qa6d2jLk18bwdn9tIuQHF1II3Qzm2Rmq8xsjZnNPMoxl5tZsZmtNLNnwlvmCdD8ucS4LXsP8cx7m/jy6dm8fsNohbk0qtERupklAg8D5wAlwHIzy3fOFQcd0xf4IXCWc26PmXm3fkrLFSWGlZVXMa9oK1eODDTTWnLzWLrooqeEKJQplxHAGufcOgAzexaYAhQHHfMN4GHn3B4A59z2cBcaMk23SIz624pt3P7yCnYfPMzI3qn0SW+nMJfjEsqUSyawOeh+Sd1jwfoB/czsbTNbZmaTGvpEZjbdzArMrGDHjh0nVnEoNN0iMWT7/gq+8+f3+daf3ie9XWte/u5Z9ElXMy05fuG6KNoC6AuMAXoAi81sqHNub/BBzrlZwCyAvLw8XaqXuFdT67j80aVsKavgpnP7M310bzXTkhMWSqCXAj2D7veoeyxYCfCuc64KWG9mqwkE/PKwVCniM1vLDtElpU2gmdZFg+nZKUktbqXJQhkKLAf6mlkvM2sFTAPyjzjmJQKjc8wsjcAUzLow1iniC7W1jj++vZ7x9y/iT/XNtPpnKMwlLBodoTvnqs3seuB1IBF4wjm30szuBgqcc/l1z000s2KgBrjJOberOQv/LwWzAxdEtxVB16ERfWmRUKzZfoCZLxRSsHEPo/ulM26AmmlJeIU0h+6cmw/MP+KxO4JuO+DGuo/IK5gNr/5v4Hb2KK1wkajz7HubuCN/JW1bJnL/ZSdxyfBMNdOSsPPHO0Xrlype8JBWt0hUyuqcxISBGdx10RDSU1p7XY74lD8CHbRUUaJKRVUNv37zUwBunjSAM/ukcWYfNdOS5qX1USJhVrBhN+f9egm/W7iW3QcPq5mWRIx/RugiHjtQWc0v/vYJTy3bSGbHtjz1tRGMVv8ViSAFukiYbCs7xLPLN3PtGTncdG5/klvrx0siS99xIk2w5+BhXi3ayjWnZ5ObEWimpR2ExCsKdJET4JzjtRXbuOPlFewtr+LMPp3pk95OYS6eUqCLHKft+yq4/eUVvL7yM4ZmduCpr41UMy2JCgp0keNQU+u47LGlbCur4IeTB3DdqF60UDMtiRIKdJEQbNl7iK7tA8207p4yhJ6d2tJbo3KJMrE/tNCG0NKMamods49opnV2v3SFuUSl2B+ha4ciaSZrtu/n5jmFfLBpL2P6pzN+YBevSxI5ptgPdNDb/iXsnnl3E3fmryS5dSIPfukkvniymmlJ9PNHoIuEWU5aEhMHd+HOiwaT1k7NtCQ2KNBFCDTTevCN1RjGzMlqpiWxKfYvioo00bvrdjH5V0t4bNE69ldUqZmWxCyN0CVu7a+o4md/+4Q/LdtEVmoSz3x9JGfmalQusUuBLnHrs32VzHm/hK+P6sWNE/uR1Eo/DhLb9B0scWX3wcPMK9zCNWfkkJvRjiU3j9MOQuIbCnSJC845Xi3cyp35K9lXUcVZuWn0Tm+nMBdfUaCL7322r4Jb567gjY8/Y1iPDvx56ki901N8SYEuvlZT67i8rpnWrecN5Ktn5aiZlviWAl18qWRPOd06tCUxwbhnyhCyUpPISUv2uiyRZqWhivhKTa3j90vWMeGBRfxpWaCZ1uh+6QpziQsaoYtvrNq2n5tfKORfm/cyfkAGEwermZbEFwW6+MKflm3krldWktKmJb+adjIXndRdzbQk7ijQJaY55zAzcjPacd7QbtxxwSA6q5mWxCkFusSkQ4dreOAfq0hIMH44eSCn9+7M6b07e12WiKd0UVRiztK1u5j0q8U8vmQ95ZU1aqYlUkcjdIkZ+yqq+On8T/jLe5vI7pzEM98YqRa3IkEU6BIztu+r5KUPS5k+ujc3TOhH21aJXpckElVCmnIxs0lmtsrM1pjZzGMcd6mZOTPLC1+JEs92Hajkj2+vByA3ox1vzRjLLecNVJiLNKDREbqZJQIPA+cAJcByM8t3zhUfcVwK8H3g3eYoVOKLc478f23hzvyVHKisZnS/dHqnt9MKFpFjCGWEPgJY45xb55w7DDwLTGnguHuAnwEVYaxP4tCWvYe47skCvv/sR2R3Tmbe976gZloiIQhlDj0T2Bx0vwQYGXyAmQ0Hejrn5pnZTUf7RGY2HZgOkJWVdfzViu9V19QybdYyduyv5PYLBvGVM3NITNAbhERC0eSLomaWADwAfKWxY51zs4BZAHl5eVprJp/bvLuc7h3b0iIxgXsvHkpWahJZnZO8LkskpoQy5VIK9Ay636PusXopwBBgoZltAE4H8nVhVEJRXVPLrMVrmfDAIp5eugGAUX3TFOYiJyCUEfpyoK+Z9SIQ5NOAK+ufdM6VAZ8vBjazhcAPnHMF4S1V/ObjrfuY8UIhhSVlnDOoC5OHdvO6JJGY1migO+eqzex64HUgEXjCObfSzO4GCpxz+c1dpPjP00s3cNcrxXRo25LfXnkK5w/tpmZaIk0U0hy6c24+MP+Ix+44yrFjml6W+FV9M61+XVK48KTu3H7BIFKTW3ldlogv6J2iEhHlh6v55euraZFo3HLeQEb27sxINdMSCavYbs5VMBs2vuV1FdKIt9fs5NyHFvPE2+s5XF2rZloizSS2R+hFcwJ/Dp3qbR3SoLJDVdw772OeK9hMr7Rknv/mGYzolep1WSK+FduBDpA9CvK+6nUV0oCdByp5pXAL3zq7D/87oS9tWqr/ikhziv1Al6iyY38lr/xrC18b1Ys+6e14a8Y4XfQUiRAFuoSFc46XPirlrleKKa+sYeyADHqlJSvMRSJIgS5NVrr3ELfOLWLhqh0Mz+rIz6cOo1dastdlicQdBbo0SaCZ1lJ2HTjMnRcO4poz1ExLxCsKdDkhm3aVk9kp0EzrvkuGkZWaRM9U9V8R8VJsr0OXiKuuqeWRhWuZ8OAinlq6AYCzctMU5iJRQCN0CdnKLWXMeKGQFaX7OHdwF85XMy2RqKJAl5A8+c4G7nm1mI5JrXjkquHqjCgShRTockz1zbQGdE1hysmZ3H7BQDomaSmiSDRSoEuDDlZW84vXV9Ey0bj1/EFqpiUSA3RRVP7L4tU7mPjgYp5cuoGqGqdmWiIxQiN0+VxZeRX3zCtmzvsl9E4PNNM6LUfNtERiRewGen3r3OxRXlfiGzsPVvJa0Va+M6YP3xuvZloisSZ2A12tc8Ni+/4K8j/awte/0PvzZlqd1H9FJCbFbqCDWuc2gXOOFz4o5Z5XizlUVcP4gV3olZasMBeJYbEd6HJCNu8u55a5RSz5dCd52Z2471I10xLxAwV6nKmuqeWKx5ex5+Bh7pkymKtGZpOgZloivqBAjxMbdh6kZ2oSLRIT+PnUQDOtHp3Uf0XET7QO3eeqamp5eMEaJj64+PNmWmf2SVOYi/iQRug+tqK0jJvnFFK8dR/nD+3GBcO6e12SiDQjBbpPzX57PT+e9zGpya149OpTmTSkq9cliUgzU6D7TH0zrcHdO3DJKZncdv4gOiS19LosEYkABbpPHKis5ud/+4RWiQncdsEgRvRKZUQvvW1fJJ7ooqgPLFy1nXMfXMzTyzbiQM20ROKURugxbM/Bw9wzr5gXPyglN6Mdc751Jqdmd/K6LBHxiAI9hu0pP8zfV37G98bl8t1xubRuoWZaIvEspCkXM5tkZqvMbI2ZzWzg+RvNrNjMCs3sTTPLDn+pArB9XwWzFq/FOUfv9Ha8PWMcN07srzAXkcYD3cwSgYeBycAg4AozG3TEYR8Cec65YcAc4OfhLjTeOed4fvlmxj+wiPv/vpoNu8oBtIJFRD4XypTLCGCNc24dgJk9C0wBiusPcM4tCDp+GXB1OIuMd5t3l/PDF4t4a81ORvRK5b5LhqqZloj8l1ACPRPYHHS/BBh5jOOvA15r6Akzmw5MB8jKygqxxPhW30xrb3kVP/7iEK4ckaVmWiLSoLBeFDWzq4E84OyGnnfOzQJmAeTl5Wlt3TGs33mQrLpmWr+YehLZnZPo3rGt12WJSBQL5aJoKdAz6H6Pusf+g5lNAG4FLnLOVYanvPhTVVPLb978lHMfXMyT72wA4Iw+nRXmItKoUEboy4G+ZtaLQJBPA64MPsDMTgEeAyY557aHvco4UViyl5vnFPLJtv1ceFJ3LjpZzbREJHSNBrpzrtrMrgdeBxKBJ5xzK83sbqDAOZcP/AJoB/zVzAA2Oecuasa6feeJt9bz43nFpKe05vEv53HOoC5elyQiMSakOXTn3Hxg/hGP3RF0e0KY64ob9c20hvXowJdO68nMyQPp0FZLEUXk+Omdoh7ZX1HFfa99QusWidxx4SDyclLJy1EzLRE5cWrO5YEFn2xn4oOL+ct7m2iRaGqmJSJhoRF6BO0+eJi7X1nJSx9toV+XdvzuqjM5JUvNtEQkPBToEVR2qIo3P97O98f35btjc2nVQr8giUj4KNCb2bayCl76qJRvju5Nr7Rk3po5Thc9RaRZKNCbiXOOZ5dv5t55H1NVW8ukwV3JSUtWmItIs1GgN4ONuw4y84Uilq7bxem9U7nvkmHkqJmWiDQzBXqYVdfUcuXj71J2qIp7Lx7KtNN6qpmWiESEAj1M1u44QHZdM637Lw800+rWQf1XRCRytMyiiQ5X1/LQG6uZ9NBinlq6EYDTe3dWmItIxGmE3gQfbd7LjDmFrPpsP1NO7s4XT8n0uiQRiWMK9BP0h7fW85N5xWSktOEP1+YxfqCaaYmItxTox6m+mdbJPTswbUQWMycPoH0bLUUUEe8p0EO0r6KKn87/hDYtE/jRhYM5NTuVU7PVTEtEoocuiobgjeLPOOeBRTy3fBOtWiSomZaIRCWN0I9h14FK7nqlmPx/bWFA1xRmXZPHST07el2WiEiDFOjHsL+imgWrtnPDhH58e0wfNdMSkaimQD/Clr2HmPthKd8Z04ectGTenjlOFz1FJCYo0OvU1jqeeW8T9732CTW1jvOHdiMnLVlhLiIxQ4EOrN95kJkvFPLu+t2clduZn148jKzOSV6XJSJyXOI+0Ktrarn69++yr6KKn186jMvyemCmZloiEnviNtDXbN9PTudkWiQm8OCXTia7cxJd2rfxuiwRkRMWd8s2KqtreOAfq5n00BKerGumNaJXqsJcRGJeXI3QP9i0hxlzCvl0+wEuOSWTS9RMS0R8JG4C/fHF67j3tY/p1r4Ns796GmP7Z3hdkohIWPk+0GtrHQkJxvDsjlw1MosZkwaQoqWIIuJDvg30skNV/GReMW1bJnLXlCFqpiUivufLi6Kvr9zGOQ8s4oUPSklu3ULNtEQkLvhqhL7zQCU/enkl84q2Mqhbe574ymkMyezgdVkiIhERe4FeMBuK5sC2Iug69D+eOlBRzZJPd3DTuf2ZPro3LRN9+QuIiEiDYi/Qg8N86FRK9x5i7gclfHdsLjlpybzzw/G0ax17pyUi0lQhDWHNbJKZrTKzNWY2s4HnW5vZc3XPv2tmOeEu9D90HUrtta/ydNVYJj6wiIcXrGXjrnIAhbmIxK1G08/MEoGHgXOAEmC5meU754qDDrsO2OOcyzWzacDPgC81R8EAh6pquHbWMt7bsJsv9E3j3ouH0jNVzbREJL6FMpwdAaxxzq0DMLNngSlAcKBPAe6suz0H+K2ZmWuG5SUOx8db9/GJ28cvpg5j6qlqpiUiAqEFeiawOeh+CTDyaMc456rNrAzoDOwMPsjMpgPTAbKysk6oYOs6jO5JFbwx+Wwy1H9FRORzEZ1wds7NAmYB5OXlndjoffJ9dA1nUSIiPhHKRdFSoGfQ/R51jzV4jJm1ADoAu8JRoIiIhCaUQF8O9DWzXmbWCpgG5B9xTD5wbd3tqcA/m2P+XEREjq7RKZe6OfHrgdeBROAJ59xKM7sbKHDO5QN/AJ42szXAbgKhLyIiERTSHLpzbj4w/4jH7gi6XQFcFt7SRETkeOi98SIiPqFAFxHxCQW6iIhPKNBFRHzCvFpdaGY7gI0n+NfTOOJdqHFA5xwfdM7xoSnnnO2cS2/oCc8CvSnMrMA5l+d1HZGkc44POuf40FznrCkXERGfUKCLiPhErAb6LK8L8IDOOT7onONDs5xzTM6hi4jIf4vVEbqIiBxBgS4i4hNRHehRtzl1BIRwzjeaWbGZFZrZm2aW7UWd4dTYOQcdd6mZOTOL+SVuoZyzmV1e97VeaWbPRLrGcAvhezvLzBaY2Yd139/neVFnuJjZE2a23cxWHOV5M7Nf1/17FJrZ8Ca/qHMuKj8ItOpdC/QGWgH/AgYdccx3gEfrbk8DnvO67gic81ggqe72t+PhnOuOSwEWA8uAPK/rjsDXuS/wIdCp7n6G13VH4JxnAd+uuz0I2OB13U0859HAcGDFUZ4/D3gNMOB04N2mvmY0j9A/35zaOXcYqN+cOtgU4Mm623OA8RbbO0Y3es7OuQXOufK6u8sI7CAVy0L5OgPcA/wMqIhkcc0klHP+BvCwc24PgHNue4RrDLdQztkB7etudwC2RLC+sHPOLSawP8TRTAGecgHLgI5m1q0prxnNgd7Q5tSZRzvGOVcN1G9OHatCOedg1xH4Hz6WNXrOdb+K9nTOzYtkYc0olK9zP6Cfmb1tZsvMbFLEqmseoZzzncDVZlZCYP+F/4lMaZ453p/3RkV0k2gJHzO7GsgDzva6luZkZgnAA8BXPC4l0loQmHYZQ+C3sMVmNtQ5t9fTqprXFcAfnXP3m9kZBHZBG+Kcq/W6sFgRzSP0eNycOpRzxswmALcCFznnKiNUW3Np7JxTgCHAQjPbQGCuMT/GL4yG8nUuAYSgCP0AAAErSURBVPKdc1XOufXAagIBH6tCOefrgOcBnHNLgTYEmlj5VUg/78cjmgM9HjenbvSczewU4DECYR7r86rQyDk758qcc2nOuRznXA6B6wYXOecKvCk3LEL53n6JwOgcM0sjMAWzLpJFhlko57wJGA9gZgMJBPqOiFYZWfnAl+tWu5wOlDnntjbpM3p9JbiRq8TnERiZrAVurXvsbgI/0BD4gv8VWAO8B/T2uuYInPMbwGfAR3Uf+V7X3NznfMSxC4nxVS4hfp2NwFRTMVAETPO65gic8yDgbQIrYD4CJnpdcxPP9y/AVqCKwG9c1wHfAr4V9DV+uO7foygc39d667+IiE9E85SLiIgcBwW6iIhPKNBFRHxCgS4i4hMKdBERn1Cgi4j4hAJdRMQn/h8KbAyvUqq7EgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot([0, 1], [0, 1], linestyle='--')\n",
    "pyplot.plot(fpr, tpr)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:torch]",
   "language": "python",
   "name": "conda-env-torch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}