resnet_finue_tune.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transfer learning  Based on ResNet34 for Methylation Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network in this notebook is an implementation of the ResNet-50 [1] architecture on the CelebA face dataset [2] to train a gender classifier.  \n",
    "\n",
    "\n",
    "References\n",
    "    \n",
    "- [1] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778). ([CVPR Link](https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html))\n",
    "\n",
    "- [2] Zhang, K., Tan, L., Li, Z., & Qiao, Y. (2016). Gender and smile classification using deep convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 34-38).\n",
    "\n",
    "The ResNet-50 architecture is similar to the ResNet-34 architecture shown below (from [1]):\n",
    "\n",
    "\n",
    "![](images/resnets/resnet34/resnet34-arch.png)\n",
    "\n",
    "However, in ResNet-50, the skip connection uses a bottleneck (from [1]):\n",
    "\n",
    "\n",
    "![](images/resnets/resnet50/resnet50-arch-1.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following figure illustrates residual blocks with skip connections such that the input passed via the shortcut matches the dimensions of the main path's output, which allows the network to learn identity functions.\n",
    "\n",
    "![](images/resnets/resnet-ex-1-1.png)\n",
    "\n",
    "\n",
    "The ResNet-34 architecture actually uses residual blocks with skip connections such that the input passed via the shortcut matches is resized to dimensions of the main path's output. Such a residual block is illustrated below:\n",
    "\n",
    "![](images/resnets/resnet-ex-1-2.png)\n",
    "\n",
    "The ResNet-50 uses a bottleneck as shown below:\n",
    "\n",
    "![](images/resnets/resnet-ex-1-3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import PosixPath\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 365,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torchvision\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from torchvision import datasets, models, transforms\n",
    "import matplotlib.pyplot as plt\n",
    "import torch.nn.functional as F\n",
    "from torchsummary import summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 545,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Hyper-paremeters\n",
    "BATCH_SIZE = 8\n",
    "NUM_EPOCHS = 30\n",
    "LEARNING_RATE = 0.00001\n",
    "RANDOM_SEED = 42\n",
    "\n",
    "\n",
    "## Model Architecture\n",
    "NUM_CLASSES = 2\n",
    "\n",
    "DEVICE = torch.device(\"cuda: 0\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset explore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GSE74845\n",
    "\n",
    "[Dataset Link](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE74845)\n",
    "\n",
    "The development of non-invasive primary cancer preventive measures in humans require a thorough understanding of the initial cancer-driving molecular mechanisms. High grade serous extra-uterine M llerian cancers (HGSEMC; formerly classified as ovarian/tubal/peritoneal cancer) present at a very late stage and less than 40% of women survive 5 years. Although the recent TCGA initiatives revealed key molecular changes in established cancers, very little is known about the initial molecular alterations in cancer development. Analysis of normal tissue at extensively high risk prior to the development of any microscopic alterations is critical. BRCA1/2 mutation carriers have an up to 30 40 fold increased risk to develop ovarian cancer, preferentially HGSEMC. Despite a plethora of evidence linking mutations in BRCA1 or BRCA2 to cancer development the core components such as organ specificity (i.e. to breast and Fallopian Tube) are still missing. The Fallopian Tube of BRCA mutation carriers offers a unique opportunity to study carcinogenesis because these cancers originate only from the distal (i.e. fimbrial) end of the Fallopian Tube (which is in close proximity to the ovary), and not from the proximal end (which is close to the uterus). The ovary which is in extreme close proximity to the fimbriae provides an excellent soil for cancer cells which are likely shed from the fimbriae and once the cancer has been discovered the big bulk of tumour is usually present on the ovary and hence the majority of HGSEMC are also referred to as ovarian cancer . To study the earliest steps of human carcinogenesis we performed epigenome-wide DNA methylation (DNAme) analyses (using the Illumina 450k DNA methylation bead-array assay assessing DNAme at ~480 000 CpG sites) in 215 microscopic normal Fallopian Tube samples of BRCA1/2 mutation carriers (n=56) and controls (n=59) who had their tubes/ovaries removed for prophylactic or other reasons, respectively (for 52 and 49 individuals respectively we analysed both fimbrial and proximal Fallopian Tubes). In order to adjust for any epigenetic effects which are not of immediate importance to the carcinogenic process, for each volunteer we analysed both the fimbrial (at risk) and the proximal (non at risk) portion of the tubes separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 546,
   "metadata": {},
   "outputs": [],
   "source": [
    "root_dir = PosixPath(\".\")\n",
    "dataset_path = \"GSE74845_series_matrix.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 369,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_matrix  = pd.read_table(root_dir/dataset_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 370,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_REF</th>\n",
       "      <th>GSM2132967</th>\n",
       "      <th>GSM2132968</th>\n",
       "      <th>GSM2132969</th>\n",
       "      <th>GSM2132970</th>\n",
       "      <th>GSM2132971</th>\n",
       "      <th>GSM2132972</th>\n",
       "      <th>GSM2132973</th>\n",
       "      <th>GSM2132974</th>\n",
       "      <th>GSM2132975</th>\n",
       "      <th>...</th>\n",
       "      <th>GSM2133183</th>\n",
       "      <th>GSM2133184</th>\n",
       "      <th>GSM2133185</th>\n",
       "      <th>GSM2133186</th>\n",
       "      <th>GSM2133187</th>\n",
       "      <th>GSM2133188</th>\n",
       "      <th>GSM2133189</th>\n",
       "      <th>GSM2133190</th>\n",
       "      <th>GSM2133191</th>\n",
       "      <th>GSM2133192</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cg00000029</td>\n",
       "      <td>0.413567</td>\n",
       "      <td>0.341079</td>\n",
       "      <td>0.381044</td>\n",
       "      <td>0.418526</td>\n",
       "      <td>0.308511</td>\n",
       "      <td>0.415981</td>\n",
       "      <td>0.325744</td>\n",
       "      <td>0.611465</td>\n",
       "      <td>0.426866</td>\n",
       "      <td>...</td>\n",
       "      <td>0.486653</td>\n",
       "      <td>0.393736</td>\n",
       "      <td>0.565041</td>\n",
       "      <td>0.559429</td>\n",
       "      <td>0.503834</td>\n",
       "      <td>0.489559</td>\n",
       "      <td>0.488975</td>\n",
       "      <td>0.626374</td>\n",
       "      <td>0.684211</td>\n",
       "      <td>0.383244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cg00000108</td>\n",
       "      <td>0.943061</td>\n",
       "      <td>0.750692</td>\n",
       "      <td>0.913595</td>\n",
       "      <td>0.932710</td>\n",
       "      <td>0.914242</td>\n",
       "      <td>0.901047</td>\n",
       "      <td>0.916935</td>\n",
       "      <td>0.946930</td>\n",
       "      <td>0.935397</td>\n",
       "      <td>...</td>\n",
       "      <td>0.939712</td>\n",
       "      <td>0.914754</td>\n",
       "      <td>0.954812</td>\n",
       "      <td>0.960274</td>\n",
       "      <td>0.950043</td>\n",
       "      <td>0.944714</td>\n",
       "      <td>0.928777</td>\n",
       "      <td>0.944578</td>\n",
       "      <td>0.947195</td>\n",
       "      <td>0.947318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>cg00000109</td>\n",
       "      <td>0.799719</td>\n",
       "      <td>0.917194</td>\n",
       "      <td>0.885665</td>\n",
       "      <td>0.889474</td>\n",
       "      <td>0.866218</td>\n",
       "      <td>0.927703</td>\n",
       "      <td>0.863381</td>\n",
       "      <td>0.864142</td>\n",
       "      <td>0.873000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.957135</td>\n",
       "      <td>0.908891</td>\n",
       "      <td>0.951368</td>\n",
       "      <td>0.852136</td>\n",
       "      <td>0.892192</td>\n",
       "      <td>0.958426</td>\n",
       "      <td>0.938942</td>\n",
       "      <td>0.915518</td>\n",
       "      <td>0.870629</td>\n",
       "      <td>0.933809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cg00000165</td>\n",
       "      <td>0.212454</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.420250</td>\n",
       "      <td>0.325410</td>\n",
       "      <td>0.282497</td>\n",
       "      <td>0.421712</td>\n",
       "      <td>0.341264</td>\n",
       "      <td>0.342029</td>\n",
       "      <td>0.280949</td>\n",
       "      <td>...</td>\n",
       "      <td>0.518261</td>\n",
       "      <td>0.464912</td>\n",
       "      <td>0.558984</td>\n",
       "      <td>0.418291</td>\n",
       "      <td>0.431217</td>\n",
       "      <td>0.413011</td>\n",
       "      <td>0.395184</td>\n",
       "      <td>0.598854</td>\n",
       "      <td>0.409190</td>\n",
       "      <td>0.471264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cg00000236</td>\n",
       "      <td>0.893922</td>\n",
       "      <td>0.911511</td>\n",
       "      <td>0.868725</td>\n",
       "      <td>0.805311</td>\n",
       "      <td>0.890320</td>\n",
       "      <td>0.916782</td>\n",
       "      <td>0.921667</td>\n",
       "      <td>0.872765</td>\n",
       "      <td>0.911796</td>\n",
       "      <td>...</td>\n",
       "      <td>0.865964</td>\n",
       "      <td>0.896430</td>\n",
       "      <td>0.894705</td>\n",
       "      <td>0.796170</td>\n",
       "      <td>0.914917</td>\n",
       "      <td>0.903630</td>\n",
       "      <td>0.848537</td>\n",
       "      <td>0.917473</td>\n",
       "      <td>0.840046</td>\n",
       "      <td>0.901202</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 217 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       ID_REF  GSM2132967  GSM2132968  GSM2132969  GSM2132970  GSM2132971  \\\n",
       "0  cg00000029    0.413567    0.341079    0.381044    0.418526    0.308511   \n",
       "1  cg00000108    0.943061    0.750692    0.913595    0.932710    0.914242   \n",
       "2  cg00000109    0.799719    0.917194    0.885665    0.889474    0.866218   \n",
       "3  cg00000165    0.212454    0.294118    0.420250    0.325410    0.282497   \n",
       "4  cg00000236    0.893922    0.911511    0.868725    0.805311    0.890320   \n",
       "\n",
       "   GSM2132972  GSM2132973  GSM2132974  GSM2132975     ...      GSM2133183  \\\n",
       "0    0.415981    0.325744    0.611465    0.426866     ...        0.486653   \n",
       "1    0.901047    0.916935    0.946930    0.935397     ...        0.939712   \n",
       "2    0.927703    0.863381    0.864142    0.873000     ...        0.957135   \n",
       "3    0.421712    0.341264    0.342029    0.280949     ...        0.518261   \n",
       "4    0.916782    0.921667    0.872765    0.911796     ...        0.865964   \n",
       "\n",
       "   GSM2133184  GSM2133185  GSM2133186  GSM2133187  GSM2133188  GSM2133189  \\\n",
       "0    0.393736    0.565041    0.559429    0.503834    0.489559    0.488975   \n",
       "1    0.914754    0.954812    0.960274    0.950043    0.944714    0.928777   \n",
       "2    0.908891    0.951368    0.852136    0.892192    0.958426    0.938942   \n",
       "3    0.464912    0.558984    0.418291    0.431217    0.413011    0.395184   \n",
       "4    0.896430    0.894705    0.796170    0.914917    0.903630    0.848537   \n",
       "\n",
       "   GSM2133190  GSM2133191  GSM2133192  \n",
       "0    0.626374    0.684211    0.383244  \n",
       "1    0.944578    0.947195    0.947318  \n",
       "2    0.915518    0.870629    0.933809  \n",
       "3    0.598854    0.409190    0.471264  \n",
       "4    0.917473    0.840046    0.901202  \n",
       "\n",
       "[5 rows x 217 columns]"
      ]
     },
     "execution_count": 370,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_matrix.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 371,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(470426, 217)"
      ]
     },
     "execution_count": 371,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 372,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_matrix.drop(\"ID_REF\", inplace=True, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 373,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_names = \"GSM2132967 GSM2132968 GSM2132969 GSM2132970 GSM2132971 GSM2132972 GSM2132973 GSM2132974 GSM2132975 GSM2132976 GSM2132977 GSM2132978 GSM2132979 GSM2132980 GSM2132981 GSM2132982 GSM2132983 GSM2132984 GSM2132985 GSM2132986 GSM2132987 GSM2132988 GSM2132989 GSM2132990 GSM2132991 GSM2132992 GSM2132993 GSM2132994 GSM2132995 GSM2132996 GSM2132997 GSM2132998 GSM2132999 GSM2133000 GSM2133001 GSM2133002 GSM2133003 GSM2133004 GSM2133005 GSM2133006 GSM2133007 GSM2133008 GSM2133009 GSM2133010 GSM2133011 GSM2133012 GSM2133013 GSM2133014 GSM2133015 GSM2133016 GSM2133017 GSM2133018 GSM2133019 GSM2133020 GSM2133021 GSM2133022 GSM2133023 GSM2133024 GSM2133025 GSM2133026 GSM2133027 GSM2133028 GSM2133029 GSM2133030 GSM2133031 GSM2133032 GSM2133033 GSM2133034 GSM2133035 GSM2133036 GSM2133037 GSM2133038 GSM2133039 GSM2133040 GSM2133041 GSM2133042 GSM2133043 GSM2133044 GSM2133045 GSM2133046 GSM2133047 GSM2133048 GSM2133049 GSM2133050 GSM2133051 GSM2133052 GSM2133053 GSM2133054 GSM2133055 GSM2133056 GSM2133057 GSM2133058 GSM2133059 GSM2133060 GSM2133061 GSM2133062 GSM2133063 GSM2133064 GSM2133065 GSM2133066 GSM2133067 GSM2133068 GSM2133069 GSM2133070 GSM2133071 GSM2133072 GSM2133073 GSM2133074 GSM2133076 GSM2133079 GSM2133081 GSM2133083 GSM2133085 GSM2133088 GSM2133090 GSM2133092 GSM2133093 GSM2133094 GSM2133095 GSM2133096 GSM2133097 GSM2133098 GSM2133099 GSM2133100 GSM2133101 GSM2133102 GSM2133103 GSM2133104 GSM2133105 GSM2133106 GSM2133107 GSM2133108 GSM2133109 GSM2133110 GSM2133111 GSM2133112 GSM2133113 GSM2133114 GSM2133115 GSM2133116 GSM2133117 GSM2133118 GSM2133119 GSM2133120 GSM2133121 GSM2133122 GSM2133123 GSM2133124 GSM2133125 GSM2133126 GSM2133127 GSM2133128 GSM2133129 GSM2133130 GSM2133131 GSM2133132 GSM2133133 GSM2133134 GSM2133135 GSM2133136 GSM2133137 GSM2133138 GSM2133139 GSM2133140 GSM2133141 GSM2133142 GSM2133143 GSM2133144 GSM2133145 GSM2133146 GSM2133147 GSM2133148 GSM2133149 GSM2133150 GSM2133151 GSM2133152 GSM2133153 GSM2133154 GSM2133155 GSM2133156 GSM2133157 GSM2133158 GSM2133159 GSM2133160 GSM2133161 GSM2133162 GSM2133163 GSM2133164 GSM2133165 GSM2133166 GSM2133167 GSM2133168 GSM2133169 GSM2133170 GSM2133171 GSM2133172 GSM2133173 GSM2133174 GSM2133175 GSM2133176 GSM2133177 GSM2133178 GSM2133179 GSM2133180 GSM2133181 GSM2133182 GSM2133183 GSM2133184 GSM2133185 GSM2133186 GSM2133187 GSM2133188 GSM2133189 GSM2133190 GSM2133191 GSM2133192\"\n",
    "sample_title = \"Patient-23-fim    Patient-29-fim    Patient-28-fim    Patient-29-prox   Patient-24-fim    Patient-30-fim    Patient-24-prox   Patient-30-prox   Patient-27-prox   Patient-15-fim    Patient-27-fim    Patient-15-prox   Patient-17-fim    Patient-22-fim    Patient-18-prox   Patient-25-prox   Patient-18-fim    Patient-25-fim    Patient-31-prox   Patient-26-prox   Patient-31-fim    Patient-26-fim    Patient-22-prox   Patient-32-fim    Patient-32-prox   Patient-19-prox   Patient-14-prox   Patient-20-fim    Patient-14-fim    Patient-20-prox   Patient-16-prox   Patient-21-fim    Patient-16-fim    Patient-21-prox   Patient-19-fim    Patient-47-fim    Patient-47-prox   Patient-33-fim    Patient-42-prox   Patient-33-prox   Patient-46-fim    Patient-40-fim    Patient-46-prox   Patient-40-prox   Patient-48-fim    Patient-67-fim    Patient-48-prox   Patient-67-prox   Patient-69-fim    Patient-56-prox   Patient-69-prox   Patient-56-fim    Patient-70-fim    Patient-57-prox   Patient-70-prox   Patient-57-fim    Patient-50-prox   Patient-61-fim    Patient-54-fim    Patient-61-prox   Patient-63-fim    Patient-36-prox   Patient-63-prox   Patient-36-fim    Patient-49-prox   Patient-38-prox   Patient-49-fim    Patient-38-fim    Patient-34-fim    Patient-60-prox   Patient-34-prox   Patient-60-fim    Patient-35-prox   Patient-35-fim    Patient-37-fim    Patient-41-fim    Patient-37-prox   Patient-41-prox   Patient-39-prox   Patient-44-prox   Patient-39-fim    Patient-44-fim    Patient-43-prox   Patient-65-prox   Patient-43-fim    Patient-65-fim    Patient-45-prox   Patient-66-prox   Patient-45-fim    Patient-53-prox   Patient-64-prox   Patient-53-fim    Patient-64-fim    Patient-51-fim    Patient-51-prox   Patient-58-fim    Patient-52-fim    Patient-59-prox   Patient-52-prox   Patient-59-fim    Patient-55-prox   Patient-62-prox   Patient-55-fim    Patient-62-fim    Patient-58-prox   Patient-68-fim    Patient-68-prox   Patient-82-fim    Patient-93-prox   Patient-86-prox   Patient-93-fim    Patient-86-fim    Patient-103-prox  Patient-90-prox   Patient-103-fim   Patient-90-fim    Patient-73-prox   Patient-94-fim    Patient-95-prox   Patient-99-prox   Patient-95-fim    Patient-99-fim    Patient-96-prox   Patient-101-prox  Patient-96-fim    Patient-101-fim   Patient-97-prox   Patient-104-prox  Patient-97-fim    Patient-104-fim   Patient-71-prox   Patient-76-prox   Patient-71-fim    Patient-76-fim    Patient-74-fim    Patient-77-prox   Patient-74-prox   Patient-77-fim    Patient-75-prox   Patient-109-fim   Patient-75-fim    Patient-109-prox  Patient-111-prox  Patient-92-fim    Patient-111-fim   Patient-92-prox   Patient-105-fim   Patient-84-fim    Patient-105-prox  Patient-84-prox   Patient-115-fim   Patient-87-fim    Patient-115-prox  Patient-87-prox   Patient-98-fim    Patient-72-fim    Patient-98-prox   Patient-72-prox   Patient-100-fim   Patient-78-fim    Patient-100-prox  Patient-102-fim   Patient-79-fim    Patient-102-prox  Patient-80-fim    Patient-80-prox   Patient-85-prox   Patient-81-fim    Patient-88-fim    Patient-81-prox   Patient-88-prox   Patient-83-fim    Patient-89-fim    Patient-83-prox   Patient-89-prox   Patient-85-fim    Patient-91-fim    Patient-91-prox   Patient-108-prox  Patient-114-fim   Patient-108-fim   Patient-114-prox  Patient-110-prox  Patient-106-fim   Patient-110-fim   Patient-106-prox  Patient-112-fim   Patient-107-prox  Patient-112-prox  Patient-113-fim   Patient-5-prox    Patient-113-prox  Patient-5-fim Patient-1-prox    Patient-12-fim    Patient-1-fim Patient-12-prox   Patient-4-prox    Patient-13-prox   Patient-4-fim Patient-13-fim    Patient-2-fim Patient-7-prox    Patient-2-prox    Patient-8-fim Patient-3-prox    Patient-8-prox    Patient-3-fim Patient-9-prox    Patient-6-fim Patient-9-fim Patient-7-fim Patient-10-fim    Patient-10-prox   Patient-11-fim    Patient-11-prox\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 374,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_dict = {sample: label for sample, label in zip(sample_names.split(), sample_title.split())}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 375,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'GSM2132967': 'Patient-23-fim',\n",
       " 'GSM2132968': 'Patient-29-fim',\n",
       " 'GSM2132969': 'Patient-28-fim',\n",
       " 'GSM2132970': 'Patient-29-prox',\n",
       " 'GSM2132971': 'Patient-24-fim',\n",
       " 'GSM2132972': 'Patient-30-fim',\n",
       " 'GSM2132973': 'Patient-24-prox',\n",
       " 'GSM2132974': 'Patient-30-prox',\n",
       " 'GSM2132975': 'Patient-27-prox',\n",
       " 'GSM2132976': 'Patient-15-fim',\n",
       " 'GSM2132977': 'Patient-27-fim',\n",
       " 'GSM2132978': 'Patient-15-prox',\n",
       " 'GSM2132979': 'Patient-17-fim',\n",
       " 'GSM2132980': 'Patient-22-fim',\n",
       " 'GSM2132981': 'Patient-18-prox',\n",
       " 'GSM2132982': 'Patient-25-prox',\n",
       " 'GSM2132983': 'Patient-18-fim',\n",
       " 'GSM2132984': 'Patient-25-fim',\n",
       " 'GSM2132985': 'Patient-31-prox',\n",
       " 'GSM2132986': 'Patient-26-prox',\n",
       " 'GSM2132987': 'Patient-31-fim',\n",
       " 'GSM2132988': 'Patient-26-fim',\n",
       " 'GSM2132989': 'Patient-22-prox',\n",
       " 'GSM2132990': 'Patient-32-fim',\n",
       " 'GSM2132991': 'Patient-32-prox',\n",
       " 'GSM2132992': 'Patient-19-prox',\n",
       " 'GSM2132993': 'Patient-14-prox',\n",
       " 'GSM2132994': 'Patient-20-fim',\n",
       " 'GSM2132995': 'Patient-14-fim',\n",
       " 'GSM2132996': 'Patient-20-prox',\n",
       " 'GSM2132997': 'Patient-16-prox',\n",
       " 'GSM2132998': 'Patient-21-fim',\n",
       " 'GSM2132999': 'Patient-16-fim',\n",
       " 'GSM2133000': 'Patient-21-prox',\n",
       " 'GSM2133001': 'Patient-19-fim',\n",
       " 'GSM2133002': 'Patient-47-fim',\n",
       " 'GSM2133003': 'Patient-47-prox',\n",
       " 'GSM2133004': 'Patient-33-fim',\n",
       " 'GSM2133005': 'Patient-42-prox',\n",
       " 'GSM2133006': 'Patient-33-prox',\n",
       " 'GSM2133007': 'Patient-46-fim',\n",
       " 'GSM2133008': 'Patient-40-fim',\n",
       " 'GSM2133009': 'Patient-46-prox',\n",
       " 'GSM2133010': 'Patient-40-prox',\n",
       " 'GSM2133011': 'Patient-48-fim',\n",
       " 'GSM2133012': 'Patient-67-fim',\n",
       " 'GSM2133013': 'Patient-48-prox',\n",
       " 'GSM2133014': 'Patient-67-prox',\n",
       " 'GSM2133015': 'Patient-69-fim',\n",
       " 'GSM2133016': 'Patient-56-prox',\n",
       " 'GSM2133017': 'Patient-69-prox',\n",
       " 'GSM2133018': 'Patient-56-fim',\n",
       " 'GSM2133019': 'Patient-70-fim',\n",
       " 'GSM2133020': 'Patient-57-prox',\n",
       " 'GSM2133021': 'Patient-70-prox',\n",
       " 'GSM2133022': 'Patient-57-fim',\n",
       " 'GSM2133023': 'Patient-50-prox',\n",
       " 'GSM2133024': 'Patient-61-fim',\n",
       " 'GSM2133025': 'Patient-54-fim',\n",
       " 'GSM2133026': 'Patient-61-prox',\n",
       " 'GSM2133027': 'Patient-63-fim',\n",
       " 'GSM2133028': 'Patient-36-prox',\n",
       " 'GSM2133029': 'Patient-63-prox',\n",
       " 'GSM2133030': 'Patient-36-fim',\n",
       " 'GSM2133031': 'Patient-49-prox',\n",
       " 'GSM2133032': 'Patient-38-prox',\n",
       " 'GSM2133033': 'Patient-49-fim',\n",
       " 'GSM2133034': 'Patient-38-fim',\n",
       " 'GSM2133035': 'Patient-34-fim',\n",
       " 'GSM2133036': 'Patient-60-prox',\n",
       " 'GSM2133037': 'Patient-34-prox',\n",
       " 'GSM2133038': 'Patient-60-fim',\n",
       " 'GSM2133039': 'Patient-35-prox',\n",
       " 'GSM2133040': 'Patient-35-fim',\n",
       " 'GSM2133041': 'Patient-37-fim',\n",
       " 'GSM2133042': 'Patient-41-fim',\n",
       " 'GSM2133043': 'Patient-37-prox',\n",
       " 'GSM2133044': 'Patient-41-prox',\n",
       " 'GSM2133045': 'Patient-39-prox',\n",
       " 'GSM2133046': 'Patient-44-prox',\n",
       " 'GSM2133047': 'Patient-39-fim',\n",
       " 'GSM2133048': 'Patient-44-fim',\n",
       " 'GSM2133049': 'Patient-43-prox',\n",
       " 'GSM2133050': 'Patient-65-prox',\n",
       " 'GSM2133051': 'Patient-43-fim',\n",
       " 'GSM2133052': 'Patient-65-fim',\n",
       " 'GSM2133053': 'Patient-45-prox',\n",
       " 'GSM2133054': 'Patient-66-prox',\n",
       " 'GSM2133055': 'Patient-45-fim',\n",
       " 'GSM2133056': 'Patient-53-prox',\n",
       " 'GSM2133057': 'Patient-64-prox',\n",
       " 'GSM2133058': 'Patient-53-fim',\n",
       " 'GSM2133059': 'Patient-64-fim',\n",
       " 'GSM2133060': 'Patient-51-fim',\n",
       " 'GSM2133061': 'Patient-51-prox',\n",
       " 'GSM2133062': 'Patient-58-fim',\n",
       " 'GSM2133063': 'Patient-52-fim',\n",
       " 'GSM2133064': 'Patient-59-prox',\n",
       " 'GSM2133065': 'Patient-52-prox',\n",
       " 'GSM2133066': 'Patient-59-fim',\n",
       " 'GSM2133067': 'Patient-55-prox',\n",
       " 'GSM2133068': 'Patient-62-prox',\n",
       " 'GSM2133069': 'Patient-55-fim',\n",
       " 'GSM2133070': 'Patient-62-fim',\n",
       " 'GSM2133071': 'Patient-58-prox',\n",
       " 'GSM2133072': 'Patient-68-fim',\n",
       " 'GSM2133073': 'Patient-68-prox',\n",
       " 'GSM2133074': 'Patient-82-fim',\n",
       " 'GSM2133076': 'Patient-93-prox',\n",
       " 'GSM2133079': 'Patient-86-prox',\n",
       " 'GSM2133081': 'Patient-93-fim',\n",
       " 'GSM2133083': 'Patient-86-fim',\n",
       " 'GSM2133085': 'Patient-103-prox',\n",
       " 'GSM2133088': 'Patient-90-prox',\n",
       " 'GSM2133090': 'Patient-103-fim',\n",
       " 'GSM2133092': 'Patient-90-fim',\n",
       " 'GSM2133093': 'Patient-73-prox',\n",
       " 'GSM2133094': 'Patient-94-fim',\n",
       " 'GSM2133095': 'Patient-95-prox',\n",
       " 'GSM2133096': 'Patient-99-prox',\n",
       " 'GSM2133097': 'Patient-95-fim',\n",
       " 'GSM2133098': 'Patient-99-fim',\n",
       " 'GSM2133099': 'Patient-96-prox',\n",
       " 'GSM2133100': 'Patient-101-prox',\n",
       " 'GSM2133101': 'Patient-96-fim',\n",
       " 'GSM2133102': 'Patient-101-fim',\n",
       " 'GSM2133103': 'Patient-97-prox',\n",
       " 'GSM2133104': 'Patient-104-prox',\n",
       " 'GSM2133105': 'Patient-97-fim',\n",
       " 'GSM2133106': 'Patient-104-fim',\n",
       " 'GSM2133107': 'Patient-71-prox',\n",
       " 'GSM2133108': 'Patient-76-prox',\n",
       " 'GSM2133109': 'Patient-71-fim',\n",
       " 'GSM2133110': 'Patient-76-fim',\n",
       " 'GSM2133111': 'Patient-74-fim',\n",
       " 'GSM2133112': 'Patient-77-prox',\n",
       " 'GSM2133113': 'Patient-74-prox',\n",
       " 'GSM2133114': 'Patient-77-fim',\n",
       " 'GSM2133115': 'Patient-75-prox',\n",
       " 'GSM2133116': 'Patient-109-fim',\n",
       " 'GSM2133117': 'Patient-75-fim',\n",
       " 'GSM2133118': 'Patient-109-prox',\n",
       " 'GSM2133119': 'Patient-111-prox',\n",
       " 'GSM2133120': 'Patient-92-fim',\n",
       " 'GSM2133121': 'Patient-111-fim',\n",
       " 'GSM2133122': 'Patient-92-prox',\n",
       " 'GSM2133123': 'Patient-105-fim',\n",
       " 'GSM2133124': 'Patient-84-fim',\n",
       " 'GSM2133125': 'Patient-105-prox',\n",
       " 'GSM2133126': 'Patient-84-prox',\n",
       " 'GSM2133127': 'Patient-115-fim',\n",
       " 'GSM2133128': 'Patient-87-fim',\n",
       " 'GSM2133129': 'Patient-115-prox',\n",
       " 'GSM2133130': 'Patient-87-prox',\n",
       " 'GSM2133131': 'Patient-98-fim',\n",
       " 'GSM2133132': 'Patient-72-fim',\n",
       " 'GSM2133133': 'Patient-98-prox',\n",
       " 'GSM2133134': 'Patient-72-prox',\n",
       " 'GSM2133135': 'Patient-100-fim',\n",
       " 'GSM2133136': 'Patient-78-fim',\n",
       " 'GSM2133137': 'Patient-100-prox',\n",
       " 'GSM2133138': 'Patient-102-fim',\n",
       " 'GSM2133139': 'Patient-79-fim',\n",
       " 'GSM2133140': 'Patient-102-prox',\n",
       " 'GSM2133141': 'Patient-80-fim',\n",
       " 'GSM2133142': 'Patient-80-prox',\n",
       " 'GSM2133143': 'Patient-85-prox',\n",
       " 'GSM2133144': 'Patient-81-fim',\n",
       " 'GSM2133145': 'Patient-88-fim',\n",
       " 'GSM2133146': 'Patient-81-prox',\n",
       " 'GSM2133147': 'Patient-88-prox',\n",
       " 'GSM2133148': 'Patient-83-fim',\n",
       " 'GSM2133149': 'Patient-89-fim',\n",
       " 'GSM2133150': 'Patient-83-prox',\n",
       " 'GSM2133151': 'Patient-89-prox',\n",
       " 'GSM2133152': 'Patient-85-fim',\n",
       " 'GSM2133153': 'Patient-91-fim',\n",
       " 'GSM2133154': 'Patient-91-prox',\n",
       " 'GSM2133155': 'Patient-108-prox',\n",
       " 'GSM2133156': 'Patient-114-fim',\n",
       " 'GSM2133157': 'Patient-108-fim',\n",
       " 'GSM2133158': 'Patient-114-prox',\n",
       " 'GSM2133159': 'Patient-110-prox',\n",
       " 'GSM2133160': 'Patient-106-fim',\n",
       " 'GSM2133161': 'Patient-110-fim',\n",
       " 'GSM2133162': 'Patient-106-prox',\n",
       " 'GSM2133163': 'Patient-112-fim',\n",
       " 'GSM2133164': 'Patient-107-prox',\n",
       " 'GSM2133165': 'Patient-112-prox',\n",
       " 'GSM2133166': 'Patient-113-fim',\n",
       " 'GSM2133167': 'Patient-5-prox',\n",
       " 'GSM2133168': 'Patient-113-prox',\n",
       " 'GSM2133169': 'Patient-5-fim',\n",
       " 'GSM2133170': 'Patient-1-prox',\n",
       " 'GSM2133171': 'Patient-12-fim',\n",
       " 'GSM2133172': 'Patient-1-fim',\n",
       " 'GSM2133173': 'Patient-12-prox',\n",
       " 'GSM2133174': 'Patient-4-prox',\n",
       " 'GSM2133175': 'Patient-13-prox',\n",
       " 'GSM2133176': 'Patient-4-fim',\n",
       " 'GSM2133177': 'Patient-13-fim',\n",
       " 'GSM2133178': 'Patient-2-fim',\n",
       " 'GSM2133179': 'Patient-7-prox',\n",
       " 'GSM2133180': 'Patient-2-prox',\n",
       " 'GSM2133181': 'Patient-8-fim',\n",
       " 'GSM2133182': 'Patient-3-prox',\n",
       " 'GSM2133183': 'Patient-8-prox',\n",
       " 'GSM2133184': 'Patient-3-fim',\n",
       " 'GSM2133185': 'Patient-9-prox',\n",
       " 'GSM2133186': 'Patient-6-fim',\n",
       " 'GSM2133187': 'Patient-9-fim',\n",
       " 'GSM2133188': 'Patient-7-fim',\n",
       " 'GSM2133189': 'Patient-10-fim',\n",
       " 'GSM2133190': 'Patient-10-prox',\n",
       " 'GSM2133191': 'Patient-11-fim',\n",
       " 'GSM2133192': 'Patient-11-prox'}"
      ]
     },
     "execution_count": 375,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 376,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_matrix.dropna(axis = 0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 377,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(470425, 216)"
      ]
     },
     "execution_count": 377,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 378,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([470425])"
      ]
     },
     "execution_count": 378,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.from_numpy(data_matrix.iloc[:,1].values).size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 379,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transform_target(target):\n",
    "    if target.split(\"-\")[-1] == \"prox\":\n",
    "        return 0\n",
    "    else:\n",
    "        return 1   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 380,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_dict = {target : transform_target(target_dict[target]) for target in target_dict}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 381,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.41356674, 0.94306078, 0.79971923, ..., 0.42618182, 0.65923307,\n",
       "       0.92669805])"
      ]
     },
     "execution_count": 381,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_matrix.iloc[:,0].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 382,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(216, 110)"
      ]
     },
     "execution_count": 382,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(target_dict),sum(target_dict.values())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 547,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GSEDataset(Dataset):\n",
    "    def __init__(self, data_matrix, target_dict):\n",
    "        self.data_matrix = data_matrix\n",
    "        self.target_dict = target_dict\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.target_dict)\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        x = torch.from_numpy(self.data_matrix.iloc[:,idx].values).float()\n",
    "        y = torch.tensor(self.target_dict[self.data_matrix.columns[idx]])\n",
    "        return x,y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 548,
   "metadata": {},
   "outputs": [],
   "source": [
    "ges_dataset =  GSEDataset(data_matrix, target_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 549,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_size = int(0.8 * len(ges_dataset))\n",
    "test_size = len(ges_dataset) - train_size\n",
    "\n",
    "train_dataset, test_dataset = torch.utils.data.random_split(ges_dataset, [train_size, test_size])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 550,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataloader = DataLoader(dataset = train_dataset,\n",
    "                             batch_size=BATCH_SIZE,\n",
    "                             shuffle=True,\n",
    "                             num_workers=4)\n",
    "\n",
    "test_dataloader = DataLoader(dataset = test_dataset,\n",
    "                             batch_size=BATCH_SIZE,\n",
    "                             shuffle=False,\n",
    "                             num_workers=4)\n",
    "\n",
    "dataset_loader = {\"train\": train_dataloader, \"test\": test_dataloader}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transfer Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 551,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "resnet = torchvision.models.resnet18(pretrained=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 552,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 conv1  ->  Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
      "--------------------------------------------------------------------------------\n",
      "1 bn1  ->  BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "--------------------------------------------------------------------------------\n",
      "2 relu  ->  ReLU(inplace=True)\n",
      "--------------------------------------------------------------------------------\n",
      "3 maxpool  ->  MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
      "--------------------------------------------------------------------------------\n",
      "4 layer1  ->  Sequential(\n",
      "  (0): BasicBlock(\n",
      "    (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (1): BasicBlock(\n",
      "    (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      ")\n",
      "--------------------------------------------------------------------------------\n",
      "5 layer2  ->  Sequential(\n",
      "  (0): BasicBlock(\n",
      "    (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (downsample): Sequential(\n",
      "      (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "      (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (1): BasicBlock(\n",
      "    (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      ")\n",
      "--------------------------------------------------------------------------------\n",
      "6 layer3  ->  Sequential(\n",
      "  (0): BasicBlock(\n",
      "    (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (downsample): Sequential(\n",
      "      (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "      (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (1): BasicBlock(\n",
      "    (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      ")\n",
      "--------------------------------------------------------------------------------\n",
      "7 layer4  ->  Sequential(\n",
      "  (0): BasicBlock(\n",
      "    (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (downsample): Sequential(\n",
      "      (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (1): BasicBlock(\n",
      "    (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      ")\n",
      "--------------------------------------------------------------------------------\n",
      "8 avgpool  ->  AdaptiveAvgPool2d(output_size=(1, 1))\n",
      "--------------------------------------------------------------------------------\n",
      "9 fc  ->  Linear(in_features=512, out_features=1000, bias=True)\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for index, (name, layer) in enumerate(resnet.named_children()):\n",
    "    print(index, name, \" -> \", layer)\n",
    "    print(\"--\"* 40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 553,
   "metadata": {},
   "outputs": [],
   "source": [
    "x,y = next(iter(dataset_loader[\"train\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 554,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([8, 470425]), torch.Size([8]))"
      ]
     },
     "execution_count": 554,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.size(), y.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 555,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5, 5, 31, 607]"
      ]
     },
     "execution_count": 555,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def div_prime(num):\n",
    "    res = []\n",
    "    while num != 1:\n",
    "        for i in range(2, int(num+1)):\n",
    "            if num % i == 0:\n",
    "                res.append(i)\n",
    "                num = num // i\n",
    "                break\n",
    " \n",
    "    return res\n",
    "div_prime(470425)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 556,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GESMODEL(torch.nn.Module):\n",
    "    def __init__(self, num_class = NUM_CLASSES):\n",
    "        super(GESMODEL, self).__init__()\n",
    "        \n",
    "        \n",
    "        self.pre = torch.nn.Sequential(\n",
    "            torch.nn.Conv2d(in_channels=5, out_channels=3, kernel_size= 1, stride=1, padding=0, bias = False),\n",
    "            torch.nn.ReLU(inplace = True),\n",
    "            torch.nn.BatchNorm2d(3),\n",
    "            torch.nn.AdaptiveAvgPool2d((224,224)),\n",
    "            \n",
    "        )\n",
    "        \n",
    "\n",
    "        self.base_model = models.resnet34(pretrained=True)\n",
    "#         for param in self.base_model.parameters():\n",
    "#             param.requires_grad = False\n",
    "            \n",
    "        self.base_layers = list(self.base_model.children())\n",
    "        \n",
    "            \n",
    "        \n",
    "        self.middle = nn.Sequential(*self.base_layers[:6])\n",
    "        \n",
    "        self.last = nn.Sequential(\n",
    "            torch.nn.Conv2d(in_channels = 128, out_channels = 2, kernel_size = 1, stride = 1, padding = 0, bias = False),\n",
    "            torch.nn.BatchNorm2d(2),\n",
    "            torch.nn.AdaptiveAvgPool2d(1),\n",
    "        )\n",
    "        \n",
    "        \n",
    "        for layer in self.pre.modules():\n",
    "            if isinstance(layer, torch.nn.Conv2d):\n",
    "                n = layer.kernel_size[0] * layer.kernel_size[1] * layer.out_channels\n",
    "                layer.weight.data.normal_(0, (2. / n)**.5)\n",
    "            elif isinstance(layer, torch.nn.BatchNorm2d):\n",
    "                layer.weight.data.fill_(1)\n",
    "                layer.bias.data.zero_()\n",
    "                \n",
    "        for layer in self.last.modules():\n",
    "            if isinstance(layer, torch.nn.Conv2d):\n",
    "                n = layer.kernel_size[0] * layer.kernel_size[1] * layer.out_channels\n",
    "                layer.weight.data.normal_(0, (2. / n)**.5)\n",
    "            elif isinstance(layer, torch.nn.BatchNorm2d):\n",
    "                layer.weight.data.fill_(1)\n",
    "                layer.bias.data.zero_()\n",
    "                \n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = x.view(-1, 5,155,607)\n",
    "        x = self.pre(x)\n",
    "        x = self.middle(x)\n",
    "#         print(x.size()) # torch.Size([2, 512, 7, 7])\n",
    "        x = self.last(x)\n",
    "        logits = torch.squeeze(x)\n",
    "        probas = F.softmax(logits, dim = 1)\n",
    "        return logits,probas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 557,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = GESMODEL().to(DEVICE)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 558,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "            Conv2d-1          [-1, 3, 155, 607]              15\n",
      "              ReLU-2          [-1, 3, 155, 607]               0\n",
      "       BatchNorm2d-3          [-1, 3, 155, 607]               6\n",
      " AdaptiveAvgPool2d-4          [-1, 3, 224, 224]               0\n",
      "            Conv2d-5         [-1, 64, 112, 112]           9,408\n",
      "            Conv2d-6         [-1, 64, 112, 112]           9,408\n",
      "       BatchNorm2d-7         [-1, 64, 112, 112]             128\n",
      "       BatchNorm2d-8         [-1, 64, 112, 112]             128\n",
      "              ReLU-9         [-1, 64, 112, 112]               0\n",
      "             ReLU-10         [-1, 64, 112, 112]               0\n",
      "        MaxPool2d-11           [-1, 64, 56, 56]               0\n",
      "        MaxPool2d-12           [-1, 64, 56, 56]               0\n",
      "           Conv2d-13           [-1, 64, 56, 56]          36,864\n",
      "           Conv2d-14           [-1, 64, 56, 56]          36,864\n",
      "      BatchNorm2d-15           [-1, 64, 56, 56]             128\n",
      "      BatchNorm2d-16           [-1, 64, 56, 56]             128\n",
      "             ReLU-17           [-1, 64, 56, 56]               0\n",
      "             ReLU-18           [-1, 64, 56, 56]               0\n",
      "           Conv2d-19           [-1, 64, 56, 56]          36,864\n",
      "           Conv2d-20           [-1, 64, 56, 56]          36,864\n",
      "      BatchNorm2d-21           [-1, 64, 56, 56]             128\n",
      "      BatchNorm2d-22           [-1, 64, 56, 56]             128\n",
      "             ReLU-23           [-1, 64, 56, 56]               0\n",
      "             ReLU-24           [-1, 64, 56, 56]               0\n",
      "       BasicBlock-25           [-1, 64, 56, 56]               0\n",
      "       BasicBlock-26           [-1, 64, 56, 56]               0\n",
      "           Conv2d-27           [-1, 64, 56, 56]          36,864\n",
      "           Conv2d-28           [-1, 64, 56, 56]          36,864\n",
      "      BatchNorm2d-29           [-1, 64, 56, 56]             128\n",
      "      BatchNorm2d-30           [-1, 64, 56, 56]             128\n",
      "             ReLU-31           [-1, 64, 56, 56]               0\n",
      "             ReLU-32           [-1, 64, 56, 56]               0\n",
      "           Conv2d-33           [-1, 64, 56, 56]          36,864\n",
      "           Conv2d-34           [-1, 64, 56, 56]          36,864\n",
      "      BatchNorm2d-35           [-1, 64, 56, 56]             128\n",
      "      BatchNorm2d-36           [-1, 64, 56, 56]             128\n",
      "             ReLU-37           [-1, 64, 56, 56]               0\n",
      "             ReLU-38           [-1, 64, 56, 56]               0\n",
      "       BasicBlock-39           [-1, 64, 56, 56]               0\n",
      "       BasicBlock-40           [-1, 64, 56, 56]               0\n",
      "           Conv2d-41           [-1, 64, 56, 56]          36,864\n",
      "           Conv2d-42           [-1, 64, 56, 56]          36,864\n",
      "      BatchNorm2d-43           [-1, 64, 56, 56]             128\n",
      "      BatchNorm2d-44           [-1, 64, 56, 56]             128\n",
      "             ReLU-45           [-1, 64, 56, 56]               0\n",
      "             ReLU-46           [-1, 64, 56, 56]               0\n",
      "           Conv2d-47           [-1, 64, 56, 56]          36,864\n",
      "           Conv2d-48           [-1, 64, 56, 56]          36,864\n",
      "      BatchNorm2d-49           [-1, 64, 56, 56]             128\n",
      "      BatchNorm2d-50           [-1, 64, 56, 56]             128\n",
      "             ReLU-51           [-1, 64, 56, 56]               0\n",
      "             ReLU-52           [-1, 64, 56, 56]               0\n",
      "       BasicBlock-53           [-1, 64, 56, 56]               0\n",
      "       BasicBlock-54           [-1, 64, 56, 56]               0\n",
      "           Conv2d-55          [-1, 128, 28, 28]          73,728\n",
      "           Conv2d-56          [-1, 128, 28, 28]          73,728\n",
      "      BatchNorm2d-57          [-1, 128, 28, 28]             256\n",
      "      BatchNorm2d-58          [-1, 128, 28, 28]             256\n",
      "             ReLU-59          [-1, 128, 28, 28]               0\n",
      "             ReLU-60          [-1, 128, 28, 28]               0\n",
      "           Conv2d-61          [-1, 128, 28, 28]         147,456\n",
      "           Conv2d-62          [-1, 128, 28, 28]         147,456\n",
      "      BatchNorm2d-63          [-1, 128, 28, 28]             256\n",
      "      BatchNorm2d-64          [-1, 128, 28, 28]             256\n",
      "           Conv2d-65          [-1, 128, 28, 28]           8,192\n",
      "           Conv2d-66          [-1, 128, 28, 28]           8,192\n",
      "      BatchNorm2d-67          [-1, 128, 28, 28]             256\n",
      "      BatchNorm2d-68          [-1, 128, 28, 28]             256\n",
      "             ReLU-69          [-1, 128, 28, 28]               0\n",
      "             ReLU-70          [-1, 128, 28, 28]               0\n",
      "       BasicBlock-71          [-1, 128, 28, 28]               0\n",
      "       BasicBlock-72          [-1, 128, 28, 28]               0\n",
      "           Conv2d-73          [-1, 128, 28, 28]         147,456\n",
      "           Conv2d-74          [-1, 128, 28, 28]         147,456\n",
      "      BatchNorm2d-75          [-1, 128, 28, 28]             256\n",
      "      BatchNorm2d-76          [-1, 128, 28, 28]             256\n",
      "             ReLU-77          [-1, 128, 28, 28]               0\n",
      "             ReLU-78          [-1, 128, 28, 28]               0\n",
      "           Conv2d-79          [-1, 128, 28, 28]         147,456\n",
      "           Conv2d-80          [-1, 128, 28, 28]         147,456\n",
      "      BatchNorm2d-81          [-1, 128, 28, 28]             256\n",
      "      BatchNorm2d-82          [-1, 128, 28, 28]             256\n",
      "             ReLU-83          [-1, 128, 28, 28]               0\n",
      "             ReLU-84          [-1, 128, 28, 28]               0\n",
      "       BasicBlock-85          [-1, 128, 28, 28]               0\n",
      "       BasicBlock-86          [-1, 128, 28, 28]               0\n",
      "           Conv2d-87          [-1, 128, 28, 28]         147,456\n",
      "           Conv2d-88          [-1, 128, 28, 28]         147,456\n",
      "      BatchNorm2d-89          [-1, 128, 28, 28]             256\n",
      "      BatchNorm2d-90          [-1, 128, 28, 28]             256\n",
      "             ReLU-91          [-1, 128, 28, 28]               0\n",
      "             ReLU-92          [-1, 128, 28, 28]               0\n",
      "           Conv2d-93          [-1, 128, 28, 28]         147,456\n",
      "           Conv2d-94          [-1, 128, 28, 28]         147,456\n",
      "      BatchNorm2d-95          [-1, 128, 28, 28]             256\n",
      "      BatchNorm2d-96          [-1, 128, 28, 28]             256\n",
      "             ReLU-97          [-1, 128, 28, 28]               0\n",
      "             ReLU-98          [-1, 128, 28, 28]               0\n",
      "       BasicBlock-99          [-1, 128, 28, 28]               0\n",
      "      BasicBlock-100          [-1, 128, 28, 28]               0\n",
      "          Conv2d-101          [-1, 128, 28, 28]         147,456\n",
      "          Conv2d-102          [-1, 128, 28, 28]         147,456\n",
      "     BatchNorm2d-103          [-1, 128, 28, 28]             256\n",
      "     BatchNorm2d-104          [-1, 128, 28, 28]             256\n",
      "            ReLU-105          [-1, 128, 28, 28]               0\n",
      "            ReLU-106          [-1, 128, 28, 28]               0\n",
      "          Conv2d-107          [-1, 128, 28, 28]         147,456\n",
      "          Conv2d-108          [-1, 128, 28, 28]         147,456\n",
      "     BatchNorm2d-109          [-1, 128, 28, 28]             256\n",
      "     BatchNorm2d-110          [-1, 128, 28, 28]             256\n",
      "            ReLU-111          [-1, 128, 28, 28]               0\n",
      "            ReLU-112          [-1, 128, 28, 28]               0\n",
      "      BasicBlock-113          [-1, 128, 28, 28]               0\n",
      "      BasicBlock-114          [-1, 128, 28, 28]               0\n",
      "          Conv2d-115            [-1, 2, 28, 28]             256\n",
      "     BatchNorm2d-116            [-1, 2, 28, 28]               4\n",
      "AdaptiveAvgPool2d-117              [-1, 2, 1, 1]               0\n",
      "================================================================\n",
      "Total params: 2,696,089\n",
      "Trainable params: 2,696,089\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 1.79\n",
      "Forward/backward pass size (MB): 157.70\n",
      "Params size (MB): 10.28\n",
      "Estimated Total Size (MB): 169.77\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "summary(model, (470425,))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 559,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_accuracy(model, data_loader, device):\n",
    "    model.eval()\n",
    "    correct_pred, num_examples = 0, 0\n",
    "    for i, (features, targets) in enumerate(data_loader):\n",
    "            \n",
    "        features = features.to(device)\n",
    "        targets = targets.to(device)\n",
    "\n",
    "        logits, probas = model(features)\n",
    "        _, predicted_labels = torch.max(probas, 1)\n",
    "        num_examples += targets.size(0)\n",
    "        correct_pred += (predicted_labels == targets).sum()\n",
    "    return correct_pred.float()/num_examples * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 560,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, data_loader, optimizer, num_epochs,batch_size, device,metric_func, random_seed = 7):\n",
    "    # Manual seed for deterministic data loader\n",
    "    torch.manual_seed(random_seed)\n",
    "    \n",
    "    loss_list = []\n",
    "    train_acc_list, valid_acc_list = [], []\n",
    "    \n",
    "    for epoch in range(num_epochs):\n",
    "        start = time.time()\n",
    "        # set training mode\n",
    "        model.train() \n",
    "        for batch_idx, (features, targets) in enumerate(data_loader[\"train\"]):\n",
    "            features = features.to(device)\n",
    "            targets = targets.to(device)\n",
    "\n",
    "\n",
    "            ## forward pass\n",
    "            logits, probas = model(features)\n",
    "            loss = F.cross_entropy(logits,targets)\n",
    "\n",
    "            # backward pass\n",
    "            # clear the gradients of all tensors being optimized\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            ### Login\n",
    "            loss_list.append(loss.item())\n",
    "\n",
    "            print ('Epoch: {0:03d}/{1:03d} | Batch {2:03d}/{3:03d} | Loss: {4:.2f}'.format(\n",
    "                epoch+1, num_epochs, batch_idx, \n",
    "                     len(train_dataset)//batch_size, loss))\n",
    "        \n",
    "        end = time.time()\n",
    "        with torch.set_grad_enabled(False):\n",
    "            train_acc = metric_func(model, data_loader[\"train\"], device)\n",
    "            valid_acc = metric_func(model, data_loader[\"test\"], device)\n",
    "            \n",
    "            print('Epoch: {0:03d}/{1:03d} train acc: {2:.3f} % | val acc: {3:.3f} % | time: {4:.3f} s'.format(\n",
    "                  epoch+1, num_epochs, train_acc, valid_acc, end-start))\n",
    "            \n",
    "\n",
    "            \n",
    "            train_acc_list.append(train_acc)\n",
    "            valid_acc_list.append(valid_acc)\n",
    "            \n",
    "    return model, loss_list, train_acc_list, valid_acc_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 561,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/030 | Batch 000/021 | Loss: 0.70\n",
      "Epoch: 001/030 | Batch 001/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 002/021 | Loss: 0.70\n",
      "Epoch: 001/030 | Batch 003/021 | Loss: 0.71\n",
      "Epoch: 001/030 | Batch 004/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 005/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 006/021 | Loss: 0.68\n",
      "Epoch: 001/030 | Batch 007/021 | Loss: 0.70\n",
      "Epoch: 001/030 | Batch 008/021 | Loss: 0.70\n",
      "Epoch: 001/030 | Batch 009/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 010/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 011/021 | Loss: 0.70\n",
      "Epoch: 001/030 | Batch 012/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 013/021 | Loss: 0.70\n",
      "Epoch: 001/030 | Batch 014/021 | Loss: 0.70\n",
      "Epoch: 001/030 | Batch 015/021 | Loss: 0.73\n",
      "Epoch: 001/030 | Batch 016/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 017/021 | Loss: 0.70\n",
      "Epoch: 001/030 | Batch 018/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 019/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 020/021 | Loss: 0.69\n",
      "Epoch: 001/030 | Batch 021/021 | Loss: 0.68\n",
      "Epoch: 001/030 train acc: 50.581 % | val acc: 50.000 % | time: 2.603 s\n",
      "Epoch: 002/030 | Batch 000/021 | Loss: 0.68\n",
      "Epoch: 002/030 | Batch 001/021 | Loss: 0.69\n",
      "Epoch: 002/030 | Batch 002/021 | Loss: 0.69\n",
      "Epoch: 002/030 | Batch 003/021 | Loss: 0.67\n",
      "Epoch: 002/030 | Batch 004/021 | Loss: 0.70\n",
      "Epoch: 002/030 | Batch 005/021 | Loss: 0.70\n",
      "Epoch: 002/030 | Batch 006/021 | Loss: 0.69\n",
      "Epoch: 002/030 | Batch 007/021 | Loss: 0.70\n",
      "Epoch: 002/030 | Batch 008/021 | Loss: 0.69\n",
      "Epoch: 002/030 | Batch 009/021 | Loss: 0.67\n",
      "Epoch: 002/030 | Batch 010/021 | Loss: 0.68\n",
      "Epoch: 002/030 | Batch 011/021 | Loss: 0.70\n",
      "Epoch: 002/030 | Batch 012/021 | Loss: 0.69\n",
      "Epoch: 002/030 | Batch 013/021 | Loss: 0.70\n",
      "Epoch: 002/030 | Batch 014/021 | Loss: 0.68\n",
      "Epoch: 002/030 | Batch 015/021 | Loss: 0.67\n",
      "Epoch: 002/030 | Batch 016/021 | Loss: 0.69\n",
      "Epoch: 002/030 | Batch 017/021 | Loss: 0.69\n",
      "Epoch: 002/030 | Batch 018/021 | Loss: 0.70\n",
      "Epoch: 002/030 | Batch 019/021 | Loss: 0.69\n",
      "Epoch: 002/030 | Batch 020/021 | Loss: 0.68\n",
      "Epoch: 002/030 | Batch 021/021 | Loss: 0.70\n",
      "Epoch: 002/030 train acc: 57.558 % | val acc: 72.727 % | time: 2.464 s\n",
      "Epoch: 003/030 | Batch 000/021 | Loss: 0.67\n",
      "Epoch: 003/030 | Batch 001/021 | Loss: 0.68\n",
      "Epoch: 003/030 | Batch 002/021 | Loss: 0.68\n",
      "Epoch: 003/030 | Batch 003/021 | Loss: 0.69\n",
      "Epoch: 003/030 | Batch 004/021 | Loss: 0.68\n",
      "Epoch: 003/030 | Batch 005/021 | Loss: 0.67\n",
      "Epoch: 003/030 | Batch 006/021 | Loss: 0.70\n",
      "Epoch: 003/030 | Batch 007/021 | Loss: 0.68\n",
      "Epoch: 003/030 | Batch 008/021 | Loss: 0.69\n",
      "Epoch: 003/030 | Batch 009/021 | Loss: 0.68\n",
      "Epoch: 003/030 | Batch 010/021 | Loss: 0.68\n",
      "Epoch: 003/030 | Batch 011/021 | Loss: 0.68\n",
      "Epoch: 003/030 | Batch 012/021 | Loss: 0.68\n",
      "Epoch: 003/030 | Batch 013/021 | Loss: 0.69\n",
      "Epoch: 003/030 | Batch 014/021 | Loss: 0.70\n",
      "Epoch: 003/030 | Batch 015/021 | Loss: 0.67\n",
      "Epoch: 003/030 | Batch 016/021 | Loss: 0.67\n",
      "Epoch: 003/030 | Batch 017/021 | Loss: 0.70\n",
      "Epoch: 003/030 | Batch 018/021 | Loss: 0.69\n",
      "Epoch: 003/030 | Batch 019/021 | Loss: 0.69\n",
      "Epoch: 003/030 | Batch 020/021 | Loss: 0.69\n",
      "Epoch: 003/030 | Batch 021/021 | Loss: 0.68\n",
      "Epoch: 003/030 train acc: 70.349 % | val acc: 79.545 % | time: 2.469 s\n",
      "Epoch: 004/030 | Batch 000/021 | Loss: 0.68\n",
      "Epoch: 004/030 | Batch 001/021 | Loss: 0.67\n",
      "Epoch: 004/030 | Batch 002/021 | Loss: 0.68\n",
      "Epoch: 004/030 | Batch 003/021 | Loss: 0.67\n",
      "Epoch: 004/030 | Batch 004/021 | Loss: 0.68\n",
      "Epoch: 004/030 | Batch 005/021 | Loss: 0.67\n",
      "Epoch: 004/030 | Batch 006/021 | Loss: 0.69\n",
      "Epoch: 004/030 | Batch 007/021 | Loss: 0.68\n",
      "Epoch: 004/030 | Batch 008/021 | Loss: 0.70\n",
      "Epoch: 004/030 | Batch 009/021 | Loss: 0.70\n",
      "Epoch: 004/030 | Batch 010/021 | Loss: 0.66\n",
      "Epoch: 004/030 | Batch 011/021 | Loss: 0.67\n",
      "Epoch: 004/030 | Batch 012/021 | Loss: 0.68\n",
      "Epoch: 004/030 | Batch 013/021 | Loss: 0.67\n",
      "Epoch: 004/030 | Batch 014/021 | Loss: 0.68\n",
      "Epoch: 004/030 | Batch 015/021 | Loss: 0.69\n",
      "Epoch: 004/030 | Batch 016/021 | Loss: 0.68\n",
      "Epoch: 004/030 | Batch 017/021 | Loss: 0.67\n",
      "Epoch: 004/030 | Batch 018/021 | Loss: 0.68\n",
      "Epoch: 004/030 | Batch 019/021 | Loss: 0.67\n",
      "Epoch: 004/030 | Batch 020/021 | Loss: 0.67\n",
      "Epoch: 004/030 | Batch 021/021 | Loss: 0.70\n",
      "Epoch: 004/030 train acc: 75.000 % | val acc: 79.545 % | time: 2.498 s\n",
      "Epoch: 005/030 | Batch 000/021 | Loss: 0.67\n",
      "Epoch: 005/030 | Batch 001/021 | Loss: 0.67\n",
      "Epoch: 005/030 | Batch 002/021 | Loss: 0.68\n",
      "Epoch: 005/030 | Batch 003/021 | Loss: 0.68\n",
      "Epoch: 005/030 | Batch 004/021 | Loss: 0.67\n",
      "Epoch: 005/030 | Batch 005/021 | Loss: 0.67\n",
      "Epoch: 005/030 | Batch 006/021 | Loss: 0.66\n",
      "Epoch: 005/030 | Batch 007/021 | Loss: 0.69\n",
      "Epoch: 005/030 | Batch 008/021 | Loss: 0.67\n",
      "Epoch: 005/030 | Batch 009/021 | Loss: 0.68\n",
      "Epoch: 005/030 | Batch 010/021 | Loss: 0.67\n",
      "Epoch: 005/030 | Batch 011/021 | Loss: 0.69\n",
      "Epoch: 005/030 | Batch 012/021 | Loss: 0.68\n",
      "Epoch: 005/030 | Batch 013/021 | Loss: 0.69\n",
      "Epoch: 005/030 | Batch 014/021 | Loss: 0.70\n",
      "Epoch: 005/030 | Batch 015/021 | Loss: 0.65\n",
      "Epoch: 005/030 | Batch 016/021 | Loss: 0.67\n",
      "Epoch: 005/030 | Batch 017/021 | Loss: 0.69\n",
      "Epoch: 005/030 | Batch 018/021 | Loss: 0.69\n",
      "Epoch: 005/030 | Batch 019/021 | Loss: 0.64\n",
      "Epoch: 005/030 | Batch 020/021 | Loss: 0.66\n",
      "Epoch: 005/030 | Batch 021/021 | Loss: 0.69\n",
      "Epoch: 005/030 train acc: 78.488 % | val acc: 79.545 % | time: 2.478 s\n",
      "Epoch: 006/030 | Batch 000/021 | Loss: 0.67\n",
      "Epoch: 006/030 | Batch 001/021 | Loss: 0.67\n",
      "Epoch: 006/030 | Batch 002/021 | Loss: 0.65\n",
      "Epoch: 006/030 | Batch 003/021 | Loss: 0.65\n",
      "Epoch: 006/030 | Batch 004/021 | Loss: 0.69\n",
      "Epoch: 006/030 | Batch 005/021 | Loss: 0.68\n",
      "Epoch: 006/030 | Batch 006/021 | Loss: 0.66\n",
      "Epoch: 006/030 | Batch 007/021 | Loss: 0.69\n",
      "Epoch: 006/030 | Batch 008/021 | Loss: 0.67\n",
      "Epoch: 006/030 | Batch 009/021 | Loss: 0.66\n",
      "Epoch: 006/030 | Batch 010/021 | Loss: 0.65\n",
      "Epoch: 006/030 | Batch 011/021 | Loss: 0.68\n",
      "Epoch: 006/030 | Batch 012/021 | Loss: 0.67\n",
      "Epoch: 006/030 | Batch 013/021 | Loss: 0.67\n",
      "Epoch: 006/030 | Batch 014/021 | Loss: 0.68\n",
      "Epoch: 006/030 | Batch 015/021 | Loss: 0.68\n",
      "Epoch: 006/030 | Batch 016/021 | Loss: 0.66\n",
      "Epoch: 006/030 | Batch 017/021 | Loss: 0.67\n",
      "Epoch: 006/030 | Batch 018/021 | Loss: 0.67\n",
      "Epoch: 006/030 | Batch 019/021 | Loss: 0.67\n",
      "Epoch: 006/030 | Batch 020/021 | Loss: 0.66\n",
      "Epoch: 006/030 | Batch 021/021 | Loss: 0.64\n",
      "Epoch: 006/030 train acc: 81.977 % | val acc: 81.818 % | time: 2.481 s\n",
      "Epoch: 007/030 | Batch 000/021 | Loss: 0.68\n",
      "Epoch: 007/030 | Batch 001/021 | Loss: 0.65\n",
      "Epoch: 007/030 | Batch 002/021 | Loss: 0.69\n",
      "Epoch: 007/030 | Batch 003/021 | Loss: 0.66\n",
      "Epoch: 007/030 | Batch 004/021 | Loss: 0.68\n",
      "Epoch: 007/030 | Batch 005/021 | Loss: 0.67\n",
      "Epoch: 007/030 | Batch 006/021 | Loss: 0.65\n",
      "Epoch: 007/030 | Batch 007/021 | Loss: 0.67\n",
      "Epoch: 007/030 | Batch 008/021 | Loss: 0.69\n",
      "Epoch: 007/030 | Batch 009/021 | Loss: 0.67\n",
      "Epoch: 007/030 | Batch 010/021 | Loss: 0.65\n",
      "Epoch: 007/030 | Batch 011/021 | Loss: 0.66\n",
      "Epoch: 007/030 | Batch 012/021 | Loss: 0.65\n",
      "Epoch: 007/030 | Batch 013/021 | Loss: 0.65\n",
      "Epoch: 007/030 | Batch 014/021 | Loss: 0.62\n",
      "Epoch: 007/030 | Batch 015/021 | Loss: 0.68\n",
      "Epoch: 007/030 | Batch 016/021 | Loss: 0.65\n",
      "Epoch: 007/030 | Batch 017/021 | Loss: 0.68\n",
      "Epoch: 007/030 | Batch 018/021 | Loss: 0.68\n",
      "Epoch: 007/030 | Batch 019/021 | Loss: 0.66\n",
      "Epoch: 007/030 | Batch 020/021 | Loss: 0.66\n",
      "Epoch: 007/030 | Batch 021/021 | Loss: 0.69\n",
      "Epoch: 007/030 train acc: 84.884 % | val acc: 81.818 % | time: 2.478 s\n",
      "Epoch: 008/030 | Batch 000/021 | Loss: 0.63\n",
      "Epoch: 008/030 | Batch 001/021 | Loss: 0.65\n",
      "Epoch: 008/030 | Batch 002/021 | Loss: 0.65\n",
      "Epoch: 008/030 | Batch 003/021 | Loss: 0.64\n",
      "Epoch: 008/030 | Batch 004/021 | Loss: 0.67\n",
      "Epoch: 008/030 | Batch 005/021 | Loss: 0.67\n",
      "Epoch: 008/030 | Batch 006/021 | Loss: 0.63\n",
      "Epoch: 008/030 | Batch 007/021 | Loss: 0.65\n",
      "Epoch: 008/030 | Batch 008/021 | Loss: 0.68\n",
      "Epoch: 008/030 | Batch 009/021 | Loss: 0.69\n",
      "Epoch: 008/030 | Batch 010/021 | Loss: 0.63\n",
      "Epoch: 008/030 | Batch 011/021 | Loss: 0.64\n",
      "Epoch: 008/030 | Batch 012/021 | Loss: 0.66\n",
      "Epoch: 008/030 | Batch 013/021 | Loss: 0.66\n",
      "Epoch: 008/030 | Batch 014/021 | Loss: 0.68\n",
      "Epoch: 008/030 | Batch 015/021 | Loss: 0.65\n",
      "Epoch: 008/030 | Batch 016/021 | Loss: 0.64\n",
      "Epoch: 008/030 | Batch 017/021 | Loss: 0.65\n",
      "Epoch: 008/030 | Batch 018/021 | Loss: 0.66\n",
      "Epoch: 008/030 | Batch 019/021 | Loss: 0.67\n",
      "Epoch: 008/030 | Batch 020/021 | Loss: 0.68\n",
      "Epoch: 008/030 | Batch 021/021 | Loss: 0.70\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 008/030 train acc: 84.302 % | val acc: 81.818 % | time: 2.482 s\n",
      "Epoch: 009/030 | Batch 000/021 | Loss: 0.66\n",
      "Epoch: 009/030 | Batch 001/021 | Loss: 0.66\n",
      "Epoch: 009/030 | Batch 002/021 | Loss: 0.61\n",
      "Epoch: 009/030 | Batch 003/021 | Loss: 0.64\n",
      "Epoch: 009/030 | Batch 004/021 | Loss: 0.67\n",
      "Epoch: 009/030 | Batch 005/021 | Loss: 0.65\n",
      "Epoch: 009/030 | Batch 006/021 | Loss: 0.63\n",
      "Epoch: 009/030 | Batch 007/021 | Loss: 0.63\n",
      "Epoch: 009/030 | Batch 008/021 | Loss: 0.63\n",
      "Epoch: 009/030 | Batch 009/021 | Loss: 0.67\n",
      "Epoch: 009/030 | Batch 010/021 | Loss: 0.68\n",
      "Epoch: 009/030 | Batch 011/021 | Loss: 0.63\n",
      "Epoch: 009/030 | Batch 012/021 | Loss: 0.64\n",
      "Epoch: 009/030 | Batch 013/021 | Loss: 0.62\n",
      "Epoch: 009/030 | Batch 014/021 | Loss: 0.67\n",
      "Epoch: 009/030 | Batch 015/021 | Loss: 0.61\n",
      "Epoch: 009/030 | Batch 016/021 | Loss: 0.63\n",
      "Epoch: 009/030 | Batch 017/021 | Loss: 0.68\n",
      "Epoch: 009/030 | Batch 018/021 | Loss: 0.70\n",
      "Epoch: 009/030 | Batch 019/021 | Loss: 0.69\n",
      "Epoch: 009/030 | Batch 020/021 | Loss: 0.64\n",
      "Epoch: 009/030 | Batch 021/021 | Loss: 0.66\n",
      "Epoch: 009/030 train acc: 86.628 % | val acc: 84.091 % | time: 2.481 s\n",
      "Epoch: 010/030 | Batch 000/021 | Loss: 0.64\n",
      "Epoch: 010/030 | Batch 001/021 | Loss: 0.67\n",
      "Epoch: 010/030 | Batch 002/021 | Loss: 0.63\n",
      "Epoch: 010/030 | Batch 003/021 | Loss: 0.59\n",
      "Epoch: 010/030 | Batch 004/021 | Loss: 0.64\n",
      "Epoch: 010/030 | Batch 005/021 | Loss: 0.64\n",
      "Epoch: 010/030 | Batch 006/021 | Loss: 0.64\n",
      "Epoch: 010/030 | Batch 007/021 | Loss: 0.65\n",
      "Epoch: 010/030 | Batch 008/021 | Loss: 0.64\n",
      "Epoch: 010/030 | Batch 009/021 | Loss: 0.63\n",
      "Epoch: 010/030 | Batch 010/021 | Loss: 0.64\n",
      "Epoch: 010/030 | Batch 011/021 | Loss: 0.64\n",
      "Epoch: 010/030 | Batch 012/021 | Loss: 0.62\n",
      "Epoch: 010/030 | Batch 013/021 | Loss: 0.61\n",
      "Epoch: 010/030 | Batch 014/021 | Loss: 0.68\n",
      "Epoch: 010/030 | Batch 015/021 | Loss: 0.66\n",
      "Epoch: 010/030 | Batch 016/021 | Loss: 0.63\n",
      "Epoch: 010/030 | Batch 017/021 | Loss: 0.67\n",
      "Epoch: 010/030 | Batch 018/021 | Loss: 0.64\n",
      "Epoch: 010/030 | Batch 019/021 | Loss: 0.61\n",
      "Epoch: 010/030 | Batch 020/021 | Loss: 0.63\n",
      "Epoch: 010/030 | Batch 021/021 | Loss: 0.65\n",
      "Epoch: 010/030 train acc: 88.372 % | val acc: 86.364 % | time: 2.488 s\n",
      "Epoch: 011/030 | Batch 000/021 | Loss: 0.62\n",
      "Epoch: 011/030 | Batch 001/021 | Loss: 0.67\n",
      "Epoch: 011/030 | Batch 002/021 | Loss: 0.68\n",
      "Epoch: 011/030 | Batch 003/021 | Loss: 0.64\n",
      "Epoch: 011/030 | Batch 004/021 | Loss: 0.64\n",
      "Epoch: 011/030 | Batch 005/021 | Loss: 0.62\n",
      "Epoch: 011/030 | Batch 006/021 | Loss: 0.61\n",
      "Epoch: 011/030 | Batch 007/021 | Loss: 0.57\n",
      "Epoch: 011/030 | Batch 008/021 | Loss: 0.60\n",
      "Epoch: 011/030 | Batch 009/021 | Loss: 0.63\n",
      "Epoch: 011/030 | Batch 010/021 | Loss: 0.66\n",
      "Epoch: 011/030 | Batch 011/021 | Loss: 0.62\n",
      "Epoch: 011/030 | Batch 012/021 | Loss: 0.64\n",
      "Epoch: 011/030 | Batch 013/021 | Loss: 0.63\n",
      "Epoch: 011/030 | Batch 014/021 | Loss: 0.62\n",
      "Epoch: 011/030 | Batch 015/021 | Loss: 0.62\n",
      "Epoch: 011/030 | Batch 016/021 | Loss: 0.59\n",
      "Epoch: 011/030 | Batch 017/021 | Loss: 0.62\n",
      "Epoch: 011/030 | Batch 018/021 | Loss: 0.65\n",
      "Epoch: 011/030 | Batch 019/021 | Loss: 0.63\n",
      "Epoch: 011/030 | Batch 020/021 | Loss: 0.62\n",
      "Epoch: 011/030 | Batch 021/021 | Loss: 0.68\n",
      "Epoch: 011/030 train acc: 90.116 % | val acc: 86.364 % | time: 2.486 s\n",
      "Epoch: 012/030 | Batch 000/021 | Loss: 0.66\n",
      "Epoch: 012/030 | Batch 001/021 | Loss: 0.55\n",
      "Epoch: 012/030 | Batch 002/021 | Loss: 0.61\n",
      "Epoch: 012/030 | Batch 003/021 | Loss: 0.58\n",
      "Epoch: 012/030 | Batch 004/021 | Loss: 0.60\n",
      "Epoch: 012/030 | Batch 005/021 | Loss: 0.59\n",
      "Epoch: 012/030 | Batch 006/021 | Loss: 0.65\n",
      "Epoch: 012/030 | Batch 007/021 | Loss: 0.66\n",
      "Epoch: 012/030 | Batch 008/021 | Loss: 0.62\n",
      "Epoch: 012/030 | Batch 009/021 | Loss: 0.62\n",
      "Epoch: 012/030 | Batch 010/021 | Loss: 0.63\n",
      "Epoch: 012/030 | Batch 011/021 | Loss: 0.66\n",
      "Epoch: 012/030 | Batch 012/021 | Loss: 0.61\n",
      "Epoch: 012/030 | Batch 013/021 | Loss: 0.61\n",
      "Epoch: 012/030 | Batch 014/021 | Loss: 0.67\n",
      "Epoch: 012/030 | Batch 015/021 | Loss: 0.60\n",
      "Epoch: 012/030 | Batch 016/021 | Loss: 0.62\n",
      "Epoch: 012/030 | Batch 017/021 | Loss: 0.61\n",
      "Epoch: 012/030 | Batch 018/021 | Loss: 0.64\n",
      "Epoch: 012/030 | Batch 019/021 | Loss: 0.64\n",
      "Epoch: 012/030 | Batch 020/021 | Loss: 0.56\n",
      "Epoch: 012/030 | Batch 021/021 | Loss: 0.51\n",
      "Epoch: 012/030 train acc: 90.116 % | val acc: 84.091 % | time: 2.533 s\n",
      "Epoch: 013/030 | Batch 000/021 | Loss: 0.61\n",
      "Epoch: 013/030 | Batch 001/021 | Loss: 0.56\n",
      "Epoch: 013/030 | Batch 002/021 | Loss: 0.64\n",
      "Epoch: 013/030 | Batch 003/021 | Loss: 0.57\n",
      "Epoch: 013/030 | Batch 004/021 | Loss: 0.63\n",
      "Epoch: 013/030 | Batch 005/021 | Loss: 0.67\n",
      "Epoch: 013/030 | Batch 006/021 | Loss: 0.64\n",
      "Epoch: 013/030 | Batch 007/021 | Loss: 0.60\n",
      "Epoch: 013/030 | Batch 008/021 | Loss: 0.58\n",
      "Epoch: 013/030 | Batch 009/021 | Loss: 0.57\n",
      "Epoch: 013/030 | Batch 010/021 | Loss: 0.57\n",
      "Epoch: 013/030 | Batch 011/021 | Loss: 0.60\n",
      "Epoch: 013/030 | Batch 012/021 | Loss: 0.59\n",
      "Epoch: 013/030 | Batch 013/021 | Loss: 0.60\n",
      "Epoch: 013/030 | Batch 014/021 | Loss: 0.60\n",
      "Epoch: 013/030 | Batch 015/021 | Loss: 0.62\n",
      "Epoch: 013/030 | Batch 016/021 | Loss: 0.60\n",
      "Epoch: 013/030 | Batch 017/021 | Loss: 0.61\n",
      "Epoch: 013/030 | Batch 018/021 | Loss: 0.59\n",
      "Epoch: 013/030 | Batch 019/021 | Loss: 0.57\n",
      "Epoch: 013/030 | Batch 020/021 | Loss: 0.59\n",
      "Epoch: 013/030 | Batch 021/021 | Loss: 0.70\n",
      "Epoch: 013/030 train acc: 91.279 % | val acc: 84.091 % | time: 2.491 s\n",
      "Epoch: 014/030 | Batch 000/021 | Loss: 0.63\n",
      "Epoch: 014/030 | Batch 001/021 | Loss: 0.59\n",
      "Epoch: 014/030 | Batch 002/021 | Loss: 0.65\n",
      "Epoch: 014/030 | Batch 003/021 | Loss: 0.59\n",
      "Epoch: 014/030 | Batch 004/021 | Loss: 0.63\n",
      "Epoch: 014/030 | Batch 005/021 | Loss: 0.57\n",
      "Epoch: 014/030 | Batch 006/021 | Loss: 0.54\n",
      "Epoch: 014/030 | Batch 007/021 | Loss: 0.58\n",
      "Epoch: 014/030 | Batch 008/021 | Loss: 0.57\n",
      "Epoch: 014/030 | Batch 009/021 | Loss: 0.54\n",
      "Epoch: 014/030 | Batch 010/021 | Loss: 0.61\n",
      "Epoch: 014/030 | Batch 011/021 | Loss: 0.59\n",
      "Epoch: 014/030 | Batch 012/021 | Loss: 0.61\n",
      "Epoch: 014/030 | Batch 013/021 | Loss: 0.62\n",
      "Epoch: 014/030 | Batch 014/021 | Loss: 0.72\n",
      "Epoch: 014/030 | Batch 015/021 | Loss: 0.54\n",
      "Epoch: 014/030 | Batch 016/021 | Loss: 0.60\n",
      "Epoch: 014/030 | Batch 017/021 | Loss: 0.58\n",
      "Epoch: 014/030 | Batch 018/021 | Loss: 0.57\n",
      "Epoch: 014/030 | Batch 019/021 | Loss: 0.58\n",
      "Epoch: 014/030 | Batch 020/021 | Loss: 0.56\n",
      "Epoch: 014/030 | Batch 021/021 | Loss: 0.65\n",
      "Epoch: 014/030 train acc: 91.860 % | val acc: 84.091 % | time: 2.500 s\n",
      "Epoch: 015/030 | Batch 000/021 | Loss: 0.60\n",
      "Epoch: 015/030 | Batch 001/021 | Loss: 0.53\n",
      "Epoch: 015/030 | Batch 002/021 | Loss: 0.53\n",
      "Epoch: 015/030 | Batch 003/021 | Loss: 0.58\n",
      "Epoch: 015/030 | Batch 004/021 | Loss: 0.60\n",
      "Epoch: 015/030 | Batch 005/021 | Loss: 0.59\n",
      "Epoch: 015/030 | Batch 006/021 | Loss: 0.58\n",
      "Epoch: 015/030 | Batch 007/021 | Loss: 0.57\n",
      "Epoch: 015/030 | Batch 008/021 | Loss: 0.55\n",
      "Epoch: 015/030 | Batch 009/021 | Loss: 0.57\n",
      "Epoch: 015/030 | Batch 010/021 | Loss: 0.56\n",
      "Epoch: 015/030 | Batch 011/021 | Loss: 0.58\n",
      "Epoch: 015/030 | Batch 012/021 | Loss: 0.52\n",
      "Epoch: 015/030 | Batch 013/021 | Loss: 0.55\n",
      "Epoch: 015/030 | Batch 014/021 | Loss: 0.52\n",
      "Epoch: 015/030 | Batch 015/021 | Loss: 0.57\n",
      "Epoch: 015/030 | Batch 016/021 | Loss: 0.58\n",
      "Epoch: 015/030 | Batch 017/021 | Loss: 0.62\n",
      "Epoch: 015/030 | Batch 018/021 | Loss: 0.61\n",
      "Epoch: 015/030 | Batch 019/021 | Loss: 0.60\n",
      "Epoch: 015/030 | Batch 020/021 | Loss: 0.56\n",
      "Epoch: 015/030 | Batch 021/021 | Loss: 0.60\n",
      "Epoch: 015/030 train acc: 93.605 % | val acc: 81.818 % | time: 2.500 s\n",
      "Epoch: 016/030 | Batch 000/021 | Loss: 0.57\n",
      "Epoch: 016/030 | Batch 001/021 | Loss: 0.52\n",
      "Epoch: 016/030 | Batch 002/021 | Loss: 0.51\n",
      "Epoch: 016/030 | Batch 003/021 | Loss: 0.59\n",
      "Epoch: 016/030 | Batch 004/021 | Loss: 0.53\n",
      "Epoch: 016/030 | Batch 005/021 | Loss: 0.55\n",
      "Epoch: 016/030 | Batch 006/021 | Loss: 0.54\n",
      "Epoch: 016/030 | Batch 007/021 | Loss: 0.54\n",
      "Epoch: 016/030 | Batch 008/021 | Loss: 0.66\n",
      "Epoch: 016/030 | Batch 009/021 | Loss: 0.56\n",
      "Epoch: 016/030 | Batch 010/021 | Loss: 0.55\n",
      "Epoch: 016/030 | Batch 011/021 | Loss: 0.53\n",
      "Epoch: 016/030 | Batch 012/021 | Loss: 0.53\n",
      "Epoch: 016/030 | Batch 013/021 | Loss: 0.54\n",
      "Epoch: 016/030 | Batch 014/021 | Loss: 0.51\n",
      "Epoch: 016/030 | Batch 015/021 | Loss: 0.50\n",
      "Epoch: 016/030 | Batch 016/021 | Loss: 0.55\n",
      "Epoch: 016/030 | Batch 017/021 | Loss: 0.56\n",
      "Epoch: 016/030 | Batch 018/021 | Loss: 0.51\n",
      "Epoch: 016/030 | Batch 019/021 | Loss: 0.51\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 016/030 | Batch 020/021 | Loss: 0.52\n",
      "Epoch: 016/030 | Batch 021/021 | Loss: 0.50\n",
      "Epoch: 016/030 train acc: 92.442 % | val acc: 77.273 % | time: 2.477 s\n",
      "Epoch: 017/030 | Batch 000/021 | Loss: 0.58\n",
      "Epoch: 017/030 | Batch 001/021 | Loss: 0.47\n",
      "Epoch: 017/030 | Batch 002/021 | Loss: 0.56\n",
      "Epoch: 017/030 | Batch 003/021 | Loss: 0.52\n",
      "Epoch: 017/030 | Batch 004/021 | Loss: 0.54\n",
      "Epoch: 017/030 | Batch 005/021 | Loss: 0.61\n",
      "Epoch: 017/030 | Batch 006/021 | Loss: 0.54\n",
      "Epoch: 017/030 | Batch 007/021 | Loss: 0.46\n",
      "Epoch: 017/030 | Batch 008/021 | Loss: 0.47\n",
      "Epoch: 017/030 | Batch 009/021 | Loss: 0.59\n",
      "Epoch: 017/030 | Batch 010/021 | Loss: 0.51\n",
      "Epoch: 017/030 | Batch 011/021 | Loss: 0.51\n",
      "Epoch: 017/030 | Batch 012/021 | Loss: 0.45\n",
      "Epoch: 017/030 | Batch 013/021 | Loss: 0.60\n",
      "Epoch: 017/030 | Batch 014/021 | Loss: 0.46\n",
      "Epoch: 017/030 | Batch 015/021 | Loss: 0.47\n",
      "Epoch: 017/030 | Batch 016/021 | Loss: 0.47\n",
      "Epoch: 017/030 | Batch 017/021 | Loss: 0.60\n",
      "Epoch: 017/030 | Batch 018/021 | Loss: 0.51\n",
      "Epoch: 017/030 | Batch 019/021 | Loss: 0.55\n",
      "Epoch: 017/030 | Batch 020/021 | Loss: 0.48\n",
      "Epoch: 017/030 | Batch 021/021 | Loss: 0.42\n",
      "Epoch: 017/030 train acc: 93.605 % | val acc: 79.545 % | time: 2.484 s\n",
      "Epoch: 018/030 | Batch 000/021 | Loss: 0.44\n",
      "Epoch: 018/030 | Batch 001/021 | Loss: 0.46\n",
      "Epoch: 018/030 | Batch 002/021 | Loss: 0.52\n",
      "Epoch: 018/030 | Batch 003/021 | Loss: 0.49\n",
      "Epoch: 018/030 | Batch 004/021 | Loss: 0.53\n",
      "Epoch: 018/030 | Batch 005/021 | Loss: 0.60\n",
      "Epoch: 018/030 | Batch 006/021 | Loss: 0.49\n",
      "Epoch: 018/030 | Batch 007/021 | Loss: 0.49\n",
      "Epoch: 018/030 | Batch 008/021 | Loss: 0.61\n",
      "Epoch: 018/030 | Batch 009/021 | Loss: 0.54\n",
      "Epoch: 018/030 | Batch 010/021 | Loss: 0.43\n",
      "Epoch: 018/030 | Batch 011/021 | Loss: 0.47\n",
      "Epoch: 018/030 | Batch 012/021 | Loss: 0.60\n",
      "Epoch: 018/030 | Batch 013/021 | Loss: 0.38\n",
      "Epoch: 018/030 | Batch 014/021 | Loss: 0.60\n",
      "Epoch: 018/030 | Batch 015/021 | Loss: 0.58\n",
      "Epoch: 018/030 | Batch 016/021 | Loss: 0.47\n",
      "Epoch: 018/030 | Batch 017/021 | Loss: 0.55\n",
      "Epoch: 018/030 | Batch 018/021 | Loss: 0.49\n",
      "Epoch: 018/030 | Batch 019/021 | Loss: 0.50\n",
      "Epoch: 018/030 | Batch 020/021 | Loss: 0.47\n",
      "Epoch: 018/030 | Batch 021/021 | Loss: 0.54\n",
      "Epoch: 018/030 train acc: 94.186 % | val acc: 84.091 % | time: 2.486 s\n",
      "Epoch: 019/030 | Batch 000/021 | Loss: 0.52\n",
      "Epoch: 019/030 | Batch 001/021 | Loss: 0.60\n",
      "Epoch: 019/030 | Batch 002/021 | Loss: 0.46\n",
      "Epoch: 019/030 | Batch 003/021 | Loss: 0.49\n",
      "Epoch: 019/030 | Batch 004/021 | Loss: 0.54\n",
      "Epoch: 019/030 | Batch 005/021 | Loss: 0.43\n",
      "Epoch: 019/030 | Batch 006/021 | Loss: 0.44\n",
      "Epoch: 019/030 | Batch 007/021 | Loss: 0.57\n",
      "Epoch: 019/030 | Batch 008/021 | Loss: 0.48\n",
      "Epoch: 019/030 | Batch 009/021 | Loss: 0.42\n",
      "Epoch: 019/030 | Batch 010/021 | Loss: 0.49\n",
      "Epoch: 019/030 | Batch 011/021 | Loss: 0.41\n",
      "Epoch: 019/030 | Batch 012/021 | Loss: 0.50\n",
      "Epoch: 019/030 | Batch 013/021 | Loss: 0.43\n",
      "Epoch: 019/030 | Batch 014/021 | Loss: 0.44\n",
      "Epoch: 019/030 | Batch 015/021 | Loss: 0.48\n",
      "Epoch: 019/030 | Batch 016/021 | Loss: 0.70\n",
      "Epoch: 019/030 | Batch 017/021 | Loss: 0.58\n",
      "Epoch: 019/030 | Batch 018/021 | Loss: 0.55\n",
      "Epoch: 019/030 | Batch 019/021 | Loss: 0.46\n",
      "Epoch: 019/030 | Batch 020/021 | Loss: 0.45\n",
      "Epoch: 019/030 | Batch 021/021 | Loss: 0.40\n",
      "Epoch: 019/030 train acc: 94.767 % | val acc: 84.091 % | time: 2.480 s\n",
      "Epoch: 020/030 | Batch 000/021 | Loss: 0.52\n",
      "Epoch: 020/030 | Batch 001/021 | Loss: 0.47\n",
      "Epoch: 020/030 | Batch 002/021 | Loss: 0.44\n",
      "Epoch: 020/030 | Batch 003/021 | Loss: 0.42\n",
      "Epoch: 020/030 | Batch 004/021 | Loss: 0.49\n",
      "Epoch: 020/030 | Batch 005/021 | Loss: 0.35\n",
      "Epoch: 020/030 | Batch 006/021 | Loss: 0.42\n",
      "Epoch: 020/030 | Batch 007/021 | Loss: 0.45\n",
      "Epoch: 020/030 | Batch 008/021 | Loss: 0.44\n",
      "Epoch: 020/030 | Batch 009/021 | Loss: 0.47\n",
      "Epoch: 020/030 | Batch 010/021 | Loss: 0.52\n",
      "Epoch: 020/030 | Batch 011/021 | Loss: 0.45\n",
      "Epoch: 020/030 | Batch 012/021 | Loss: 0.65\n",
      "Epoch: 020/030 | Batch 013/021 | Loss: 0.46\n",
      "Epoch: 020/030 | Batch 014/021 | Loss: 0.53\n",
      "Epoch: 020/030 | Batch 015/021 | Loss: 0.31\n",
      "Epoch: 020/030 | Batch 016/021 | Loss: 0.49\n",
      "Epoch: 020/030 | Batch 017/021 | Loss: 0.40\n",
      "Epoch: 020/030 | Batch 018/021 | Loss: 0.37\n",
      "Epoch: 020/030 | Batch 019/021 | Loss: 0.46\n",
      "Epoch: 020/030 | Batch 020/021 | Loss: 0.46\n",
      "Epoch: 020/030 | Batch 021/021 | Loss: 0.62\n",
      "Epoch: 020/030 train acc: 94.767 % | val acc: 86.364 % | time: 2.495 s\n",
      "Epoch: 021/030 | Batch 000/021 | Loss: 0.37\n",
      "Epoch: 021/030 | Batch 001/021 | Loss: 0.40\n",
      "Epoch: 021/030 | Batch 002/021 | Loss: 0.66\n",
      "Epoch: 021/030 | Batch 003/021 | Loss: 0.41\n",
      "Epoch: 021/030 | Batch 004/021 | Loss: 0.36\n",
      "Epoch: 021/030 | Batch 005/021 | Loss: 0.42\n",
      "Epoch: 021/030 | Batch 006/021 | Loss: 0.48\n",
      "Epoch: 021/030 | Batch 007/021 | Loss: 0.39\n",
      "Epoch: 021/030 | Batch 008/021 | Loss: 0.45\n",
      "Epoch: 021/030 | Batch 009/021 | Loss: 0.42\n",
      "Epoch: 021/030 | Batch 010/021 | Loss: 0.36\n",
      "Epoch: 021/030 | Batch 011/021 | Loss: 0.52\n",
      "Epoch: 021/030 | Batch 012/021 | Loss: 0.41\n",
      "Epoch: 021/030 | Batch 013/021 | Loss: 0.37\n",
      "Epoch: 021/030 | Batch 014/021 | Loss: 0.44\n",
      "Epoch: 021/030 | Batch 015/021 | Loss: 0.55\n",
      "Epoch: 021/030 | Batch 016/021 | Loss: 0.42\n",
      "Epoch: 021/030 | Batch 017/021 | Loss: 0.45\n",
      "Epoch: 021/030 | Batch 018/021 | Loss: 0.51\n",
      "Epoch: 021/030 | Batch 019/021 | Loss: 0.40\n",
      "Epoch: 021/030 | Batch 020/021 | Loss: 0.43\n",
      "Epoch: 021/030 | Batch 021/021 | Loss: 0.43\n",
      "Epoch: 021/030 train acc: 95.349 % | val acc: 81.818 % | time: 2.485 s\n",
      "Epoch: 022/030 | Batch 000/021 | Loss: 0.46\n",
      "Epoch: 022/030 | Batch 001/021 | Loss: 0.32\n",
      "Epoch: 022/030 | Batch 002/021 | Loss: 0.44\n",
      "Epoch: 022/030 | Batch 003/021 | Loss: 0.45\n",
      "Epoch: 022/030 | Batch 004/021 | Loss: 0.52\n",
      "Epoch: 022/030 | Batch 005/021 | Loss: 0.37\n",
      "Epoch: 022/030 | Batch 006/021 | Loss: 0.45\n",
      "Epoch: 022/030 | Batch 007/021 | Loss: 0.51\n",
      "Epoch: 022/030 | Batch 008/021 | Loss: 0.36\n",
      "Epoch: 022/030 | Batch 009/021 | Loss: 0.54\n",
      "Epoch: 022/030 | Batch 010/021 | Loss: 0.46\n",
      "Epoch: 022/030 | Batch 011/021 | Loss: 0.47\n",
      "Epoch: 022/030 | Batch 012/021 | Loss: 0.36\n",
      "Epoch: 022/030 | Batch 013/021 | Loss: 0.33\n",
      "Epoch: 022/030 | Batch 014/021 | Loss: 0.35\n",
      "Epoch: 022/030 | Batch 015/021 | Loss: 0.42\n",
      "Epoch: 022/030 | Batch 016/021 | Loss: 0.42\n",
      "Epoch: 022/030 | Batch 017/021 | Loss: 0.36\n",
      "Epoch: 022/030 | Batch 018/021 | Loss: 0.41\n",
      "Epoch: 022/030 | Batch 019/021 | Loss: 0.41\n",
      "Epoch: 022/030 | Batch 020/021 | Loss: 0.52\n",
      "Epoch: 022/030 | Batch 021/021 | Loss: 0.35\n",
      "Epoch: 022/030 train acc: 95.930 % | val acc: 79.545 % | time: 2.520 s\n",
      "Epoch: 023/030 | Batch 000/021 | Loss: 0.45\n",
      "Epoch: 023/030 | Batch 001/021 | Loss: 0.39\n",
      "Epoch: 023/030 | Batch 002/021 | Loss: 0.32\n",
      "Epoch: 023/030 | Batch 003/021 | Loss: 0.45\n",
      "Epoch: 023/030 | Batch 004/021 | Loss: 0.45\n",
      "Epoch: 023/030 | Batch 005/021 | Loss: 0.40\n",
      "Epoch: 023/030 | Batch 006/021 | Loss: 0.39\n",
      "Epoch: 023/030 | Batch 007/021 | Loss: 0.27\n",
      "Epoch: 023/030 | Batch 008/021 | Loss: 0.32\n",
      "Epoch: 023/030 | Batch 009/021 | Loss: 0.41\n",
      "Epoch: 023/030 | Batch 010/021 | Loss: 0.48\n",
      "Epoch: 023/030 | Batch 011/021 | Loss: 0.36\n",
      "Epoch: 023/030 | Batch 012/021 | Loss: 0.43\n",
      "Epoch: 023/030 | Batch 013/021 | Loss: 0.51\n",
      "Epoch: 023/030 | Batch 014/021 | Loss: 0.38\n",
      "Epoch: 023/030 | Batch 015/021 | Loss: 0.43\n",
      "Epoch: 023/030 | Batch 016/021 | Loss: 0.59\n",
      "Epoch: 023/030 | Batch 017/021 | Loss: 0.44\n",
      "Epoch: 023/030 | Batch 018/021 | Loss: 0.42\n",
      "Epoch: 023/030 | Batch 019/021 | Loss: 0.35\n",
      "Epoch: 023/030 | Batch 020/021 | Loss: 0.37\n",
      "Epoch: 023/030 | Batch 021/021 | Loss: 0.52\n",
      "Epoch: 023/030 train acc: 95.930 % | val acc: 81.818 % | time: 2.484 s\n",
      "Epoch: 024/030 | Batch 000/021 | Loss: 0.36\n",
      "Epoch: 024/030 | Batch 001/021 | Loss: 0.35\n",
      "Epoch: 024/030 | Batch 002/021 | Loss: 0.46\n",
      "Epoch: 024/030 | Batch 003/021 | Loss: 0.41\n",
      "Epoch: 024/030 | Batch 004/021 | Loss: 0.40\n",
      "Epoch: 024/030 | Batch 005/021 | Loss: 0.34\n",
      "Epoch: 024/030 | Batch 006/021 | Loss: 0.42\n",
      "Epoch: 024/030 | Batch 007/021 | Loss: 0.42\n",
      "Epoch: 024/030 | Batch 008/021 | Loss: 0.39\n",
      "Epoch: 024/030 | Batch 009/021 | Loss: 0.44\n",
      "Epoch: 024/030 | Batch 010/021 | Loss: 0.33\n",
      "Epoch: 024/030 | Batch 011/021 | Loss: 0.42\n",
      "Epoch: 024/030 | Batch 012/021 | Loss: 0.29\n",
      "Epoch: 024/030 | Batch 013/021 | Loss: 0.42\n",
      "Epoch: 024/030 | Batch 014/021 | Loss: 0.39\n",
      "Epoch: 024/030 | Batch 015/021 | Loss: 0.41\n",
      "Epoch: 024/030 | Batch 016/021 | Loss: 0.39\n",
      "Epoch: 024/030 | Batch 017/021 | Loss: 0.35\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 024/030 | Batch 018/021 | Loss: 0.40\n",
      "Epoch: 024/030 | Batch 019/021 | Loss: 0.62\n",
      "Epoch: 024/030 | Batch 020/021 | Loss: 0.35\n",
      "Epoch: 024/030 | Batch 021/021 | Loss: 0.70\n",
      "Epoch: 024/030 train acc: 97.093 % | val acc: 86.364 % | time: 2.493 s\n",
      "Epoch: 025/030 | Batch 000/021 | Loss: 0.47\n",
      "Epoch: 025/030 | Batch 001/021 | Loss: 0.46\n",
      "Epoch: 025/030 | Batch 002/021 | Loss: 0.40\n",
      "Epoch: 025/030 | Batch 003/021 | Loss: 0.34\n",
      "Epoch: 025/030 | Batch 004/021 | Loss: 0.36\n",
      "Epoch: 025/030 | Batch 005/021 | Loss: 0.39\n",
      "Epoch: 025/030 | Batch 006/021 | Loss: 0.48\n",
      "Epoch: 025/030 | Batch 007/021 | Loss: 0.34\n",
      "Epoch: 025/030 | Batch 008/021 | Loss: 0.49\n",
      "Epoch: 025/030 | Batch 009/021 | Loss: 0.36\n",
      "Epoch: 025/030 | Batch 010/021 | Loss: 0.33\n",
      "Epoch: 025/030 | Batch 011/021 | Loss: 0.37\n",
      "Epoch: 025/030 | Batch 012/021 | Loss: 0.43\n",
      "Epoch: 025/030 | Batch 013/021 | Loss: 0.40\n",
      "Epoch: 025/030 | Batch 014/021 | Loss: 0.32\n",
      "Epoch: 025/030 | Batch 015/021 | Loss: 0.47\n",
      "Epoch: 025/030 | Batch 016/021 | Loss: 0.38\n",
      "Epoch: 025/030 | Batch 017/021 | Loss: 0.41\n",
      "Epoch: 025/030 | Batch 018/021 | Loss: 0.37\n",
      "Epoch: 025/030 | Batch 019/021 | Loss: 0.37\n",
      "Epoch: 025/030 | Batch 020/021 | Loss: 0.38\n",
      "Epoch: 025/030 | Batch 021/021 | Loss: 0.41\n",
      "Epoch: 025/030 train acc: 97.674 % | val acc: 88.636 % | time: 2.498 s\n",
      "Epoch: 026/030 | Batch 000/021 | Loss: 0.40\n",
      "Epoch: 026/030 | Batch 001/021 | Loss: 0.39\n",
      "Epoch: 026/030 | Batch 002/021 | Loss: 0.32\n",
      "Epoch: 026/030 | Batch 003/021 | Loss: 0.31\n",
      "Epoch: 026/030 | Batch 004/021 | Loss: 0.31\n",
      "Epoch: 026/030 | Batch 005/021 | Loss: 0.39\n",
      "Epoch: 026/030 | Batch 006/021 | Loss: 0.29\n",
      "Epoch: 026/030 | Batch 007/021 | Loss: 0.32\n",
      "Epoch: 026/030 | Batch 008/021 | Loss: 0.35\n",
      "Epoch: 026/030 | Batch 009/021 | Loss: 0.40\n",
      "Epoch: 026/030 | Batch 010/021 | Loss: 0.35\n",
      "Epoch: 026/030 | Batch 011/021 | Loss: 0.35\n",
      "Epoch: 026/030 | Batch 012/021 | Loss: 0.39\n",
      "Epoch: 026/030 | Batch 013/021 | Loss: 0.40\n",
      "Epoch: 026/030 | Batch 014/021 | Loss: 0.35\n",
      "Epoch: 026/030 | Batch 015/021 | Loss: 0.52\n",
      "Epoch: 026/030 | Batch 016/021 | Loss: 0.50\n",
      "Epoch: 026/030 | Batch 017/021 | Loss: 0.45\n",
      "Epoch: 026/030 | Batch 018/021 | Loss: 0.37\n",
      "Epoch: 026/030 | Batch 019/021 | Loss: 0.35\n",
      "Epoch: 026/030 | Batch 020/021 | Loss: 0.40\n",
      "Epoch: 026/030 | Batch 021/021 | Loss: 0.43\n",
      "Epoch: 026/030 train acc: 95.930 % | val acc: 84.091 % | time: 2.511 s\n",
      "Epoch: 027/030 | Batch 000/021 | Loss: 0.40\n",
      "Epoch: 027/030 | Batch 001/021 | Loss: 0.31\n",
      "Epoch: 027/030 | Batch 002/021 | Loss: 0.35\n",
      "Epoch: 027/030 | Batch 003/021 | Loss: 0.34\n",
      "Epoch: 027/030 | Batch 004/021 | Loss: 0.34\n",
      "Epoch: 027/030 | Batch 005/021 | Loss: 0.43\n",
      "Epoch: 027/030 | Batch 006/021 | Loss: 0.45\n",
      "Epoch: 027/030 | Batch 007/021 | Loss: 0.34\n",
      "Epoch: 027/030 | Batch 008/021 | Loss: 0.32\n",
      "Epoch: 027/030 | Batch 009/021 | Loss: 0.31\n",
      "Epoch: 027/030 | Batch 010/021 | Loss: 0.33\n",
      "Epoch: 027/030 | Batch 011/021 | Loss: 0.34\n",
      "Epoch: 027/030 | Batch 012/021 | Loss: 0.25\n",
      "Epoch: 027/030 | Batch 013/021 | Loss: 0.28\n",
      "Epoch: 027/030 | Batch 014/021 | Loss: 0.27\n",
      "Epoch: 027/030 | Batch 015/021 | Loss: 0.34\n",
      "Epoch: 027/030 | Batch 016/021 | Loss: 0.33\n",
      "Epoch: 027/030 | Batch 017/021 | Loss: 0.34\n",
      "Epoch: 027/030 | Batch 018/021 | Loss: 0.30\n",
      "Epoch: 027/030 | Batch 019/021 | Loss: 0.35\n",
      "Epoch: 027/030 | Batch 020/021 | Loss: 0.31\n",
      "Epoch: 027/030 | Batch 021/021 | Loss: 0.39\n",
      "Epoch: 027/030 train acc: 97.093 % | val acc: 81.818 % | time: 2.498 s\n",
      "Epoch: 028/030 | Batch 000/021 | Loss: 0.40\n",
      "Epoch: 028/030 | Batch 001/021 | Loss: 0.33\n",
      "Epoch: 028/030 | Batch 002/021 | Loss: 0.36\n",
      "Epoch: 028/030 | Batch 003/021 | Loss: 0.28\n",
      "Epoch: 028/030 | Batch 004/021 | Loss: 0.28\n",
      "Epoch: 028/030 | Batch 005/021 | Loss: 0.31\n",
      "Epoch: 028/030 | Batch 006/021 | Loss: 0.29\n",
      "Epoch: 028/030 | Batch 007/021 | Loss: 0.28\n",
      "Epoch: 028/030 | Batch 008/021 | Loss: 0.39\n",
      "Epoch: 028/030 | Batch 009/021 | Loss: 0.35\n",
      "Epoch: 028/030 | Batch 010/021 | Loss: 0.31\n",
      "Epoch: 028/030 | Batch 011/021 | Loss: 0.41\n",
      "Epoch: 028/030 | Batch 012/021 | Loss: 0.31\n",
      "Epoch: 028/030 | Batch 013/021 | Loss: 0.38\n",
      "Epoch: 028/030 | Batch 014/021 | Loss: 0.42\n",
      "Epoch: 028/030 | Batch 015/021 | Loss: 0.29\n",
      "Epoch: 028/030 | Batch 016/021 | Loss: 0.39\n",
      "Epoch: 028/030 | Batch 017/021 | Loss: 0.32\n",
      "Epoch: 028/030 | Batch 018/021 | Loss: 0.35\n",
      "Epoch: 028/030 | Batch 019/021 | Loss: 0.28\n",
      "Epoch: 028/030 | Batch 020/021 | Loss: 0.34\n",
      "Epoch: 028/030 | Batch 021/021 | Loss: 0.74\n",
      "Epoch: 028/030 train acc: 98.256 % | val acc: 86.364 % | time: 2.508 s\n",
      "Epoch: 029/030 | Batch 000/021 | Loss: 0.36\n",
      "Epoch: 029/030 | Batch 001/021 | Loss: 0.29\n",
      "Epoch: 029/030 | Batch 002/021 | Loss: 0.39\n",
      "Epoch: 029/030 | Batch 003/021 | Loss: 0.30\n",
      "Epoch: 029/030 | Batch 004/021 | Loss: 0.28\n",
      "Epoch: 029/030 | Batch 005/021 | Loss: 0.26\n",
      "Epoch: 029/030 | Batch 006/021 | Loss: 0.36\n",
      "Epoch: 029/030 | Batch 007/021 | Loss: 0.29\n",
      "Epoch: 029/030 | Batch 008/021 | Loss: 0.51\n",
      "Epoch: 029/030 | Batch 009/021 | Loss: 0.46\n",
      "Epoch: 029/030 | Batch 010/021 | Loss: 0.33\n",
      "Epoch: 029/030 | Batch 011/021 | Loss: 0.33\n",
      "Epoch: 029/030 | Batch 012/021 | Loss: 0.40\n",
      "Epoch: 029/030 | Batch 013/021 | Loss: 0.29\n",
      "Epoch: 029/030 | Batch 014/021 | Loss: 0.42\n",
      "Epoch: 029/030 | Batch 015/021 | Loss: 0.32\n",
      "Epoch: 029/030 | Batch 016/021 | Loss: 0.51\n",
      "Epoch: 029/030 | Batch 017/021 | Loss: 0.37\n",
      "Epoch: 029/030 | Batch 018/021 | Loss: 0.30\n",
      "Epoch: 029/030 | Batch 019/021 | Loss: 0.30\n",
      "Epoch: 029/030 | Batch 020/021 | Loss: 0.38\n",
      "Epoch: 029/030 | Batch 021/021 | Loss: 0.38\n",
      "Epoch: 029/030 train acc: 98.837 % | val acc: 88.636 % | time: 2.503 s\n",
      "Epoch: 030/030 | Batch 000/021 | Loss: 0.28\n",
      "Epoch: 030/030 | Batch 001/021 | Loss: 0.34\n",
      "Epoch: 030/030 | Batch 002/021 | Loss: 0.27\n",
      "Epoch: 030/030 | Batch 003/021 | Loss: 0.32\n",
      "Epoch: 030/030 | Batch 004/021 | Loss: 0.29\n",
      "Epoch: 030/030 | Batch 005/021 | Loss: 0.31\n",
      "Epoch: 030/030 | Batch 006/021 | Loss: 0.38\n",
      "Epoch: 030/030 | Batch 007/021 | Loss: 0.36\n",
      "Epoch: 030/030 | Batch 008/021 | Loss: 0.47\n",
      "Epoch: 030/030 | Batch 009/021 | Loss: 0.55\n",
      "Epoch: 030/030 | Batch 010/021 | Loss: 0.31\n",
      "Epoch: 030/030 | Batch 011/021 | Loss: 0.33\n",
      "Epoch: 030/030 | Batch 012/021 | Loss: 0.29\n",
      "Epoch: 030/030 | Batch 013/021 | Loss: 0.34\n",
      "Epoch: 030/030 | Batch 014/021 | Loss: 0.30\n",
      "Epoch: 030/030 | Batch 015/021 | Loss: 0.36\n",
      "Epoch: 030/030 | Batch 016/021 | Loss: 0.29\n",
      "Epoch: 030/030 | Batch 017/021 | Loss: 0.29\n",
      "Epoch: 030/030 | Batch 018/021 | Loss: 0.32\n",
      "Epoch: 030/030 | Batch 019/021 | Loss: 0.31\n",
      "Epoch: 030/030 | Batch 020/021 | Loss: 0.31\n",
      "Epoch: 030/030 | Batch 021/021 | Loss: 0.43\n",
      "Epoch: 030/030 train acc: 99.419 % | val acc: 88.636 % | time: 2.494 s\n"
     ]
    }
   ],
   "source": [
    "model, loss_list, train_acc_list, valid_acc_list = train_model(model, \n",
    "            dataset_loader, \n",
    "            optimizer, \n",
    "            NUM_EPOCHS, \n",
    "            device = DEVICE, \n",
    "            batch_size = BATCH_SIZE,\n",
    "            metric_func = compute_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 562,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydd5hcVd2A33PvtO3JlvReIQkp9N6r0hSlCYg0URA+UBBUEAEFQUEEpCNdiiJFeg0lhQRSIL2Tusn2Ou3e8/1xy9ypO7vZNpv7Ps8+O3PrubM753d+XUgpcXFxcXHZdVF6egAuLi4uLj2LKwhcXFxcdnFcQeDi4uKyi+MKAhcXF5ddHFcQuLi4uOzieHp6AO2lvLxcjho1qqeH4eLi4pJTfPnll1VSyopU+3JOEIwaNYr58+f39DBcXFxccgohxIZ0+1zTkIuLi8sujisIXFxcXHZxXEHg4uLisovjCgIXFxeXXRxXELi4uLjs4riCwMXFxWUXxxUELi4uLrs4riBwcXFx6WTqWyK8vmhLTw8ja3IuoczFxcWlt3PF8wuYuXIHU4eVMLKsoKeH0yauRuDi4uLSyWypawUgFNV7eCTZ4QoCFxcXl10cVxC4uLi47OK4ggBYVdnIc3O/7elhuLi4uPQIrrMYOOGeT4nqkrP3G9HTQ3FxcXHpdlyNAIjq0vit5YZjx8XFJTeQsqdHkB2uIHCQKx5+FxcXl87EFQQOXEHg4uLSmQjR0yPIDlcQOAhGtJ4egksv4PQHZ/OPj1f39DBcXLoNVxA4cDUCF4Av1tdwx9srenoYLn0A10eQg7gagYuLy67ILiUIIprOZc99xTeb61PudzUCFxeXzsT1EQBCiOOFECuEEKuFENel2H+3EGKh+bNSCFHXleNZsa2RNxZv5VcvLYptjIbtl65G4OLisivSZYJACKEC9wMnAJOAs4QQk5zHSCmvklJOl1JOB+4FXu6q8XyycgcXPzUfgEK/kUcXXfYm3FrBcv+PKaQlTiPQdMnkG9/m2bkbUl7v7x+s4uT7Pku5750l23h69vpOHX+uUNkQpCEY6elhuLj0ClwfAewLrJZSrpVShoHngVMyHH8W8K+uGszaHU0011cBUBQwBMHij/8NQEBEOE99l/98uck+fumWBprDGn95J7XT8K73VrJ4U2oT00+f/pIbXl2Sct+2+iBnPTwn7l65wuXPfcXUm97JeMx+f/qAI+78uHsG5NImG6qbGXXdG7y/tLKnh9JptIY1Rl33Bk/NXt/TQ0lLrpiELLpSEAwFNjrebzK3JSGEGAmMBj7sqsHsv+1ZPvNfSQGt+D0qAJGG7WyVpXyoTedCz1u8tmizffxX39YCMG5AIeGozt3vraQlHAWgORSNu/bq7Y2c+9hcWsPxpiWZYjmweFMds9dWc/f7K+O2RzWdVxduTnlOb+F/i7fSEIy2eVx1c7jNY3qaCb97ix88MKunh9HlLDIXK68s3NzGkblDdXMIgIdmru3hkaSnF3+NU9KVgiCVTEz38ZwJ/FtKmdJIL4S4RAgxXwgxf8eOHR0ajDryAIpFKyeps6lrDfPMrDWMDK3gG30UH+nTKRONVFBH2DQPNTU2sqdYybz1tTz62Vru+WAVVz6/0DAZ/T62KpZS8ofXl/LpqirmrKuOu+fo699k8aZ4t0ezKUwS/1GemLWeK59fyH8XJH9hG4MRnvh8Xa8WErlGOKozf0NtTw+jy7G+hH3xP8f9PnQeXSkINgHDHe+HAel6t51JBrOQlPJhKeXeUsq9KyoqOjSYkgkHslwfzoXqWzSs+4pz3t2TQfp2/q0dxgY5EIBRotJebRy/9Bpe9t9EOfV2TPl7SysZ+5s3464biur4VONjjKSIOjr5vs+55X9L7fdNIUPWedV4Obm90bjvDvP3/R+t5ri7PwHg2n8v5qbXl7JwoyFUPlhWmaR97AzvL61k1uqqnb5OX/hi9oVnSEkfeixh2l168yO5pqEY84DxQojRQggfxmT/WuJBQoiJQH9gdheOhfLCAPdEv894ZTP/8d0EwN2R03hH35tTjjgYgDHKVnY0hvjq9QcZW28MZ7oSn2G6t1jOvd6/48FY2QcjGl5LEGiJ/5rG+8c+WwfAyspGbnjlGwA8avxHb/3jaOZEdOc7K1hR2QjAIlMAeFWFlZWNXPjkfH77ytcd/iwSueip+Zz96Nydvo7z+XM1AkvTe/P00n4Ue9LsO8+VY3NsTtBlgkBKGQUuB94BlgEvSimXCCFuFkKc7Dj0LOB52cVLMUURPPCnm9k65FjyRJgtspR7tNM4dtIgTjvqIML5gzhe+YJ3vt7MuC9vYZlulKQ2BIHESxSQ3OR9ipPUORyqLAaMSf7tJdsAWFHZaE+AhyqL+MZ/IWNFzNSz7uvZHK98AUjUhCWD9V5PmIi++raWLfVBwMiDaDE1gdXbmwCob43w9jfbsvoMZq2psrWKnSHdn8qppdS29H4/QSq0PqYRWP9mupsi063k2r9Rl+YRSCnflFJOkFKOlVL+0dx2o5TyNccxN0kpk3IMuor8H/6D2yJn8cvIzxhQ5OfXJ+wGioq+x+kconxN7WePUUwTD0RPYpU+lMs9r7LYfzEv+m7mVOVzpijr0aXgSs/LHKh8w/0frkRBx0+YmR++Rctbv2es2MxTvj9TKIKcrM4mz2s4pw9Y/Bse9P2NQ5SvCWs6mi656Ml5zFlbba/c4uWA5J4HH0DFmGC/rWmxTUcRTXL1CwuZ9od3ufSZL9lU25LyeWevqeae91cBcPYjczn1/s+z+pwyleSOplk1tzq0gEg0x74JJn1twnRXzy7ZsMs1pinuV85D2kkArLn+KFTF+KoEppwEc//On7yPATBbn8yJcg7j2UyxaGGGWM0M32q+0UexPrAbJ4bf5jnfn6iVhfQXxup8m+xP6Ve1fOCP3e8432L+5TkHdI3CRiPK4Wnf7exV9YDtb5i7roafHDQaiDdNnKV+yG3ex/hd5Cc8ox3Dlc8vtPdFNJ2XHY5lSxP5elM9U4YW23bUsx6ZA8CVR4/P+jNavq2B4//2Kf88fx+O2G1A0v6IptvmMCdWVBVAuB29Hb7cUMuw/nkMLA5kfU5X0Vc1gr5kGrLozX8q10fQyxGOv5AlBAAYtg/6d+6y3+6gHzdEfsLckT/l15GLeUQ7iYej3+Wc8PXMHHqJfVx/0URQegEYJOKjULbJ/uymr6a4ZQPsWI7iCIraR4nlJzQGo9z3wQoOURbz8Cdr+GTlDmaIVdxmCqV9leVJz5G4Yt9SF+TpORs46b7PeOTT5LC69ti+3/raMDV9sb6Gu95dwf8Wb4kzB4XTlOJwagTpjnly1nqO+MvHcdtOe2AWJ96bOjmvu+lrPgJLJ+hLj9WXhVtPsctpBGkRAmXfCzn1v03o5penklLWTLqcF1Z8TUBRQEAQndKKIUxd9ggv+G7mS30Cv4tewLu+a5mgbOaB6El8ok+lH00s0MfxacG1XBB9lap5GykHTgndzKv+G5msrKdc1DNMVPHn6Blc5/kXl3je4MfhX3Pe4zoL/HfaQztZnc2d0dPZaEY3QbJj+rzHv7Bfv79sO5ccOjZu/+cpooI0XRJJsXJfV9UMwMjSfK572XBKH31L7N7O1X5lQ5B1Vc3sP6YszkeQ6roAv3/NSLSTUiKEsAWGZfLqaRJ9NLlOrq1Ms0HkgMGrN2srqdglBUFxwJM2MeqwI0+gIRhh8efrAeifb6z2PYqCRxUEIzpThpZQWlbBCdV/ts87IXw7OgKJwv5jSnlrbQ0AwVFHc+aa12H+x9TLfBbJsWyS5fzC84p97qWe1+3XBynfMFOfhmYqa/+MHsdPPO/wqf8qpgUfpp5CAKK6zsnKLGoo4jN9D67yvMQZ6sfsH7qPhlajxEN1U2xydQoKgBfnb+Tafy9O+RlsqWsF4icR5yTvXO0f+ZePaQ5rrL/9u/E+gjZMQy1hjQK/h7osnMrVTSH2uvV9nrpgXw6d0LHw4Wzpc6Yh83cfeyygbz5TT7HLmYYAPr/uSBbccEzKfVcdM4EbT4yVROqX7wOgvNBnO30LAx7uPWtPwMg8BtBQOXXGcO74wVTOP3C0fX7R/j+2X/82ciEg0PqPSTu2SzxvsNp/DuWigXui3+fW6Dl8kX8oAM/7buFI5StjXNEq/u67j2d8twFwpee/DBK1DKUK3fyGnPvYF6lvAmmFAMRW/GGH1lFdV085RpaqUxA0OwREqLmeC9S3UNDT+ggs4dJkZmdbWchxZroErDDa+z7q+mYxfU8jMMNH3VmzW8k1TWyXFARFAS/9C3xp91tfniK/B7/X+IjGDSgiYAqCfPM3wO6DiwEo8KncfcZ0Tt97OPm+2H7GH8MtkR+xVB/Jm/p+AIR8ZQBcHv4F72h7AzAp+DhHhP4KgEfoaFLwZPRYNFTmDr/YuJeykcd9f0FFY/dQbCL3E1tVn+n5iGYzaW1lZQM+IvxcfYXHvHdi5TVYdvC7vfczy3953LO/tmgLG2uMCKSgOcn7CTPm0Yl86L8akCnyJYwJdMziu7nR+zRHKV+lPAawhemrCzcTjurUmoLAKgSYiuKAoZXVt3R9Mbu+qhH0JXJtks0FdknTUDa8ctlBDCoOMKDIzzXHTeTsfUfYSVc+j8KUocXccdpUvjN1MNceN9EWEgB+T7x8PfqCW3ju6y3oc74F4MuJV/PBFg/v6Pvwjr4P50ytoOXLWtbJwWjTz2PT8nlcUH8BNRhC5syj9wNH7bs1gXN5KPpd+/1fvQ8CoEvBLzyvsKRlIne9N4yzxLvcEnjCPm5GdDUL5HhawlEUdL6nWqGkEhBsqG7min8tsI+vN01MP1A/QZEaxaKVIlptjcCpGUR0nfwmo1LrAFGX1lmc51VpCWv86c3lbG8IMX1EPyCzILA0HGs8XUkfUwgcjtW+Qx+T1b2CXVIjyIbpw/sxqCSAogguO2Ic/Qt85HljGcRCCE7fZziFfg/DS/OpKIrFjJaYfoUz9jYqbBwwtoxbT92DM/cZzkHjyqhTy/hz9CwieIjg4dg9J9rnqqfey9sHPMsaadTn23NEP/KKSpPG91PPG/brE1UjRPSagY/Q5C3je/ID/v7BKr6jxJuG/uv/PQOopTmkcbPnn7HxYjiHP18dXyvJKic9Rmy1tw0UNWxrCHLTa0viiu/VNIdRWo3zx4nNfLxiO7e9uSxp3E6Bub66mUbTVxPwpv9XtLSLutZkf4KuS/7z5aakKKpQVOOOt5cnFQhsi75nGjJ+631w9ux7T9RzuIKgHfz88HFAzC+QjokDi3jjioO5/bQ94rbfftpUnr1ofyobgva2iw8ZzfiBhQwuCXDu/iMBKCuMCZU/fm8PfB6VB6Mn8X/hn7NX8AFui5zFZ9pkfh25OO76dXkjWTXwBI5UFlBKA4rQ+UYfxc2Rc1mlG4LlBPULmkJRzvF8YJ83RBgT+ObaZs5V32W4MEoWb6g2TETlIlZue7Co4aMV23li1noWOQrqHXrbu5Q0GBVVx4otPDv3Wx76JDmM1e+Y8HXZtlMZYqGywUjysS8v2MwvX1rEo2YZD4uX5m/iHx+v4e8frmrz+k76WvioFWHTXjkQjGi8OH9jr/QtWGGjvXBoSeRKiKtrGmoHR08ayPrbv9vmcUIIJg8pSbv/Z4ePxaMIrj1+N3ymGWn29UfZ+8tM/8W+o0rZfXAxUkpuj55l739IO8lOitsWGMuo4DI+1Kcz1edhY9lJzNj0DF8FLgXgpeihPK6dwJPascz0X8V+yjI2VDfjlxUMF0Yl12FiB8vkSArqV3GN9wnW6IO5JHI1M81K2eXUU62UUaZXM1hUs/BbQwDUOMpNTxTf4hdRWqSf8Uos0U3TZZwjOM+hEWi6tE1Imb4uiZnMOxpDzFpTxSnTh1JjFgmsSgg/tSb0llD7ah71NR8BHTQN3fXeSh7+ZC398rwcO3lQpw9rZ8ilP1GujNXVCHqAgcUBfnfiJFsIJFJqCoJG06whMnjHzjz1VLZOPI+NciABr0pd8cS4/S9qhzOgyI/P62OxPobdxLdc/OQXDKCWp6NHo0vBJGHY9kfWGD6DscpWPvBfwynKZ+QRpFzUs0SOIYKHUaKSpVsbAKhuigmCaYqx+n9ZO5jBooZCDG0iFI2fiJ2mIV06HM8ZvjCJWsO5j83lyucX0hCM2F+0xI/IEj7tndj7nGnI/N3elb2V19GYRf+J7iaX/kKuIHDpMCV5ho+hKdS2czTgVRldUQAYTuzGkMbRoTsA2CJLmSd3QxGCAcV+VsphjBKVTBer8Yso8/SJrJJDOVX9jAvUtzi46oW4a9/j+wf3eu9lgKjj20gx1f5hjBWxSuJVzbFV+B5iLTWykDm6EXo7WJh5FAnmHK/QycMwjckUpqEdjSFOvf9ztta32tsSzTXLtxnhpFEtpngnCktbEKSIXso0KXaWRvD2N1sZdd0bbK5rbfvgLiTTIiJXif39cmSWzQFcQdALGVRi1Nz58QGj2jzW71UYXWYIgtrmMGMrClkth3H7uGf5buhPgLHyHlDkpzowCkVIfqjOBGCevhtPaMcxWqnkRu/TlGhGiYyQjFkMj1YX0F80sVoOJdp/HFP8sZaHTo1ghNjOWjmEbbI/AAPNchuJ5aivrP8zywIX2OOy7P/WBPzi/I0s3FjHE7PW2+ekC0X9vxcWcvtbyeU3ICYIUhXIy+QH6CwfwUvzjVaky7Y0dMr1OsrOJpT1ZjmSC6tt10fg0mECXjWlL8LnUZLCMsdWFNqVS7c2BDl+yiA++OVhbK0L8uA3RrirLiVXHT0Bz9ZG+ADO8nxEpezHVsqYV3oy9a2vUhI1SlD8PXoqd0VPZwhVDBB1vOK/EYCv9dFQXsCgbR/hIUoUT1zmcoloZossoxJTEJBaEBzQajTb8RNG6hpFjWsAmdTUR3HMQNE0JUE/WRnrVpc4X1mrxlTRMplW/Z1dfbSnp4HY2rmnR9J55IIAsMiVsboaQY6w8MZjePOKg+33//jRntzxg6kMLA4wvDTf2Gj+142tKCTfH++UPXBcOftOm2pv2yLLAXjmov1pyRsMwD3R73NP9DRjP+UslOP4WcHdvKHty9dyDHmDdkNFs53MJfXL+ImZSVwimqmnkGavcV1LI2iNaFQ3hYxSElrM1FVGA2c0PMHFX5/F49470aJh8xGMZ3BO7NE0GkEmLIGZaoXv/HJ+mdCuMlFIzFpdxSsp2oe2RW9ZSVufZ65MSC49g6sR5Aj98n325OZVBd/ZY7C9b0hJgBtOnMQxu8cKwxX4Yn9aey4sjJWU/lXkp0waXMygkgBzyw5icOPXfKhNR8ORFQ20lk/hsur/A6B4+GQA9hDrKKCV2+tuJuANM0xUUUIz9bIAfHnURQsY6PAR7HXr+5QW+PjqFzFHdqloYM+QkedwpLqQwdqWuLFaE+mKF27giKWPAo9m/oASJt5QBkHg3HbaA7P46oZjbAd94vFWEuGpM4Zmvn8aejr80tYIMgzj8Ds/Yr/RZfz5B1PTH9SLsJ7FlW2dh6sR5BBWDwBfUptLwYUHj2ZEWb69zVnm4trjzQlYMbZVFU9ijRxqm00WjLqIU0M3s0iOS7rnxIFFsfsPm0Gtbwinqx9xrDofr4zwtrYP56tvUyRaqZMFeFVBpezPQGGEmD41ez1ghJrqcx60r1UmGglE6lmnG8Krn2bkMlhfcss0NHHZ3ykRLXZznmyxBEEqs1Liqt9pvuq8xKveoRLYGkGGaXN9dQsvzN/YXUPaaXLJzJUrmpgrCHIIKxkrmwqcBY6SDT/ab2Rsx6838PmhTwOxSU9VVBamEAIAU4Y68iFUD2v7HcDB6hKu8LzCOjmYefoEVGGWgKAAj6KYgqCWUhq4edl3OExZRAlNKHPuty9VRj39aGC5NFqClsrauDElOnkLaSVDXbqk0sS2IEgVNZQgG5xHdHZCWU/PA/bquYMD6Y0TWW8cUzpyRWi5giCHyPd5eO+qQ7n7jOlZHKum3pHXD4/PiDKyJj2rUuilh41NOnz68H5x75vyYiaSpXIkVTImKOplAaoiOGjGFAaIWqYo6ygRLTzp+zN3eB+Ou85dvgfxCY1luiGkKmQtOxpD9gr2gY/XsNQRcVMkWuIcyInUtYR59NO1aLqkviXCGrOnc2skWZNI1AicuQOdlUfQe3wE5u92ntdLhp+Rnja7ZaKjGd09hesjyDHGO0w1mUgsfJdqn/VPaplGnFm/FkP65bHPqP6cMt0QAFtK94UNsFIfym2RsxmjxPIKNskKhAC1eDCDlXq7dAXAcep8AA4L3cVM/9X29o2ygmbpZ4Co45k5G+KKvs1fvRmrIHgJLWxXBJouOU6Zx3fVOVwZuQxprmWen2eYNiqK/Nz+1nK21hu5CqmK3yWaf5zaR2dnFvf0RKD3QYN6LmQR5IomYOEKgj6KEIKLDh6dsuewldFsTRK2IPAlCw9VEbx06YH2+/p+kxgbfNp2KpfIZnvfOjnYqJc6fD8UGeV2r+Hg/WDQhRy1zWi7uUEOYrE+mqmKURuohmLqKaCIFjYHo4xp+pITlRX8Tz8ArSVWy8jQCCAS1XjIdzcAr2oH8pm+ByFiJcVnra62hQCk7p2cuOp3JrV1lmmot6yo+2b4aO48S66M1DUN9WF+d+IkDhpXnrTdEgTW6tcynwRSaASJeFUlLrJoi4xVRq2hyMhkHX8semEsqiky4pC4a5wd/q39eq0cRLPMo9wXpikU4awVV3Kf717Gis3QGgvtLKIFVQi0xm32tsd8f+WP3sfjrp24ok+lEWi6zkXqGxyuGCW3nYKg86t09nDUUAd9BL15AuvNY0skV4SWqxHsglimISugxioDEfCq/P6kSUblzh/tmbIyqE+NX+s2UMiPw7+mkFbsdbCiEDzjBfIfO5QF+jhKxu0LjorYTcSimzbJCpoJUCyCNAfDdnTQAcpStObYcUVm7SK5fWXc/Y8yO7ZZKAIqqKVC1LNUjkqpEXg2zuJ33mf5Vq/g0PCMmEN5ywLyq2pSfWTtprf4CKxps6PTUe95jhg76wDvDmwfQQ+PI1tcQbALkmgasmLoK4r8nL73cH5y0Oi053rUZCVy90O+z4Mz18RtUwdN5rrIRbyn7cWjgXzuiJzBWhnTEp6OHs1IUYlEoUkGKBdB/M2xxK3JYj01TbGWnuWi3vAfbF8ad598QswQq1ggxwECTYc3/b+hQtQzKvhcSo1AaTDu008YZi1b4D18OPsD8FzSOVLKDtXt6enJSrcnzVyZkrKh9z9LrpniXNPQLog/QRBcfcwE7j5jGodnEZZq2dBP23MYZ+4znIoiP6UF3qRre1WV57UjqaYEr6pw1c0PMmj/0+3jbohewHmR6wFDQygQrQRat9v7JyvrGb75Tfv9dGUNmpSom+eyScbMXX4R4b/+33O6+rE5Pp0Ks39CMc2pO6UFDd+DpX384MHZWT93tiSGs/YUHfEVb6huZqXZJ7o3y49cEG45METAFQS7JD7VsPFbC+GAV+V7M4ZlteK1Vs8FfpXbT5vKvN8eHZezYPUXVhxB/15Vwasq9DM7t508bQjnHzjK3t9MgHzZii9kmGU+1aYwVVnHyaoxQX+gzWCGshopJd5tC1igj+O40O18o8euMVUYZbCdEUAPee9Oad5STN+Djyi26aSNb2yq4nWJn0kqenoe6EgTl8Pu/JglPVwsLxO5Mrka5MZgXUGwC2IlpnXEMWrZ0z1K7F/H2W+4KJBsbfSYfoVys/OaLmWsPhLQJAPkyRbyIoYg+ESPlTp4IXo48/SJDBY1lOvVeBo2slwfwQo5gj9Fz7aPKxfGxOVcuR+gLk2pEYiQoRF4hUYRrUnnjRCVSeekm+xfXbiZ8b99iw3VzXHbe4ttXe+ARtDbyYXw0VzLI3AFwS6IVaqiQ4LAnFk8DqfxCVMGU2wKgFSCwGsKjaH98wCobAjGhXAGRT5+rYX8iLFSf1U7yN736+jFrJTDADjO7MG82uznvEQfxUbdMGeNMfskaLq0S2EDRLVYY5U5a6v55+frUIKxsFSrDaez1PUn/quSspjTFb57d4khNBZurEu5v6d7Bctc8Ky2k9+8/HVPDyFrcuVTdwXBLoi6E8tVq3+A1yEIfB6FXx5r1DMqCniTzrGExrB+liAI2WGex0wayI8O2wMVjUHaNhpkPgfNmELD/tfwf+GfA4L6fCP7+DBlEYDtdD502gQOCd/Dg9ETGSm2I9DRdEkJzbRKwwFerNXbk+GZD8/httcXUbj6dXtsZZiCIKEm0eCSvLj3ifstLMGXrpNXZ5asuOf9VawybfftJVcmpGyYn1Ax1mXncQXBLkhxnoefHjqG5y7av93nHj/F6F970rQhcdutrmoTBhYmnWNpIENMQdAv32uXwJg2rITCAaMA+KHnExbo48jzqRQf/zu+KDoagB8dOQOA3ZVvAdgmS3n8/L257AijJMYGORC/iDCYGj5bvok8EWaFqUUMELVxIaTnqO+jyChz9N0BKDNNSlFNgmpWIJUCD/ETe7rmOOkEgSVrO0sjaAxGuPv9lZzx8Jx2nWfdvw8pBDlFrnzubvjoLogQguu/s3uHzh0/sChl05xTpg+hKODhiInJmcyW9lDg9/DgOXsxY0Q/Sgt8tIQ1w2lcGXNMvqAdzhAzsc36Dnny+6OjMkjUonnyaSQPr6rYpba3yjIAhnpqsebvFfoIpitrGSDqCEd1/B7jmgcoS1ilD+UX4cuZF7jMjjCKRjWkFmGLLGOoqGaQHotggpgmlIjlKE/XVjSDH7ldWI78UIraSZmIRQ3lyIzUHnLgkXIhsglcjcClkxBCcNTuA+OihSycuQfHTxnEwOIAXlXh0sPGGtnMpbG8hXVysK0tWKvZfJ+XkLcYwLT/CzyKwlBTw2iQhuN591KFQWYfBMuvUESL4TD+5mWe9v6JClFPlSyhFqNmUxmGEIpEwwgk63RD4ymXVXHPkE4jsHwdtS3xgsByFnZWETv7fu3NEO5jLgLnxJoLj5QLY4QuFgRCiOOFECuEEKuFENelOeZ0IcRSIcQSIURyJo9LzuPJVD8aIC/m3F0vB5JnrvSt73y+TyXoNaqgfhs2qp16VYGiCEaU5tOEIeHo3UwAACAASURBVBD6q0G7M9oq06FcLFqMSfyd33KI+g3TlTU0kkcUD7WykAqzb0I02GLe3xAE/pZtcX0K0rXLtMpdb0nTpD5TEbuX5m/k0qe/TLs/7jqmBGivqamvmYb6ynP0NrpMEAghVOB+4ARgEnCWEGJSwjHjgeuBg6SUk4H/66rxuPQc3hTZyHEIgTz3FR6KfpdWAhT4401DeT6VkCkI6jFKaFtaxvtXH0azDABGzwJbI9ANjaCYFj589UlojFVJbTCvsc3RQEePGBO5JQgGUcOlz8Qm6cqGEHe9tzLJ+WsJgsWb6pmzNlZt1cony+Qsvubfi3l7Sax20urtjexoDCUd9+6SbUz7w7txn0m25EKoZXvItefIFcHVlRrBvsBqKeVaKWUYeB44JeGYi4H7pTS6kkgpt+PS51Db0ggAMfYIbov+CIiVw45pBB5CPkMQNEpj9W9pGT6PQqOpERTQSoWoJyxVtlFKWKoUiRYaV84EYJk+PO4albLUbqmphQ1BUE8BddJotfnxih32+P7w+hL+/sEqZq6M/xcNRQ2toaY5zJkPz0myCbdnBX/0XZ9wyB0f8uina+PyEu54Z4X92rp+bXM4u4v2sRITzs8zF54pV3wzXSkIhgLO/nebzG1OJgAThBCfCyHmCCGOT3UhIcQlQoj5Qoj5O3bsSHWISx8i3zYNWT6CmEbQaBasc2oZzaYgyJctFNBqmooEDWZ56+FKNWv0wXyh72Zewzh+qyxlsCkI9LBRujoofTTKfApFvKnHSkyrbnJMwFIyou4Lbvc8jA/DR2BpCJboa2/4aDCic+sbyzjLER0UP/nBXe+uYMYt7zFzZey7MGt1Fa3hZEdyrkxE2ZIDc388OTLerhQEqZaBiR+LBxgPHA6cBTwqhOiXdJKUD0sp95ZS7l1R0XY9HJfcxnIWW/8sAa9KxGf4BqxJ3JnQdtlRuxFRAuTLFgpFkBYMU1GjzKNItDJc7GCzLGeTNP53SjBW29tkKWU04CVKNGz4CEJ4acFPPvEmGmtM9a0Op/C6mfzs219ypudjpgijv0LiZNzRPAJnOKpz8ovqkr9/uBqAmabGsq6qmbMfnctvX0lOtIoVnevQMLqFLXWtfLg8OZs7FU7B1osfKefoSkGwCRjueD8M2JLimFellBEp5TpgBYZgcNmFyTMn3QPGltnvdTPGX6aIxrn6mAl484rJl83kE7R9Bs2igGKaGcZ2NskKOxFNxVi176AfipD0pxERNMJIG8mnNYUgsCbSLXWxpjc0xP6dLaez1dvBCvfMxjSUKrLIuSWdCaS2xdBOLOFktedMNe6O5jN0x2R78n2fccET87M61vkYvVm4WeTAEIGuFQTzgPFCiNFCCB9wJvBawjGvAEcACCHKMUxFa7twTC45gLX6/usPp/HuVYcatYwUw1xkTeKjywviT/IXcnDDmwwStbZGUK/2Y5qyhv6ikeVyOHmTv8O3e13HX6JGFdQaaYSQlopG1CbDaVsp+9Mq/fQXjXzku4ojzX4HVnjo5rqW2D2DsfwHqy3nuqpY5U7ILo8gVdayc/JPp1RYgsCa5FMVDcwF01BVU5b+DnJj8neSK+PtMkEgpYwClwPvAMuAF6WUS4QQNwshTjYPeweoFkIsBT4CrpFSVqe+osuugiUIAl6VCVaPZlMQeNG4/oTdkvsiTPwOAOPFJppMjaDSO5xSYayS13nHcdcZe9Kw58+pM3MIfnCwUdyuv2hEbd4KGOaiFvxMVdYxWqnkBs/TANSZk+5mZ5hoq+FfCEmPLQh+9Ohcjr37E9sums1KPJX5yLkl3TUsh7G1O5VPvq8Vnevp2k3tJRcEMXRxZrGU8k3gzYRtNzpeS+Bq88elj3HTSZPSFmPLhOUsdlLdfzoAX+rjOSBVOOqE42D2fXE+gurACCwLT0O/Sfg8SlzJ7AljRsEXRlKZt7mOBplPCwGKiooxG6JRj1Ey40zxLtVKMXNqD47ds6WaRlFIrZ5nF69LJBsfwWerqpLahGZjArG0FJlBI7BOTjeB5kLkjZPcGm3u4JaYcOkyzs/Q6SwTlkbgpGbAfuwTvJ8d9OdQTwpBEIjFGDSbgmBlYDrUwyfaHig+M8TUce1AP6McRn/RiL95C9tFGecfOIroqoB9zDBhOGQv87xKpezPWy370RKOGsKqpYZ6UUyT4qdEiy9DbZGNILgkRVKZHmcaSn2NppDhULb2ptII2rp7W3KgO6tpZ9MFLj6zuPeLhVyRs64gcOl15KUQBKoi2IGRgXzcpIEpToplJ7dIo+/BVu9wfpD3CKtqJZPMWkNOjSCv2Oh0Vi4ayG9az2I5CI8iCBITBOWigUJaGEitYUJCo6Y5bAiC1hpq9EI0NUCx7vAdEJuAO8OU0dZq3nI2p+qKprdhG+pNphYp2+7joGehKfUmcmCIgFtryKUX4suQiXzi1MEMKA4k73AIAqvkhBDg6T+cegoJmM148hwmmIJAgEjxCCaKjRQ0bWCtHIKqCoLCuH5YGsfe5H0KRUgCIsJYsYU6yyTTtJ3tehERX7EdkmphTcAdDR+NjxrKfIy9O4NGkFaYtH9oXUZWQqk3DTgLcsX01qYgMEtFuLh0GymjX2yHaJoloy8WRbRGxkpkW9FFlg3eWRRPUQTRAXtwrDIfRUZZrQ3EqygUaUa9os/1KQD8QP3EPmeyWM+ctdWMuu4NQnWVbNdL0P0llIgEQWAOOFWtoXeXbGPZ1jZaQTpOSydLEgvKpUzcMfelLZrXxkTVndNYNjIzF8xBENNscmO02WkEq4UQdybWCXJx6U6sCSttuQohqMwbR5Us5jXtQHOTsAVBS4qsWwC9bCKKMK69SVagKsKePP+rHRJ/LILJynpufWMZKhq+cA076IfI65+kEdhF4lLMbpc8/SUn3PNpVs8L6VeV1nbrtyIEdS1hbn9ruV022zrTKoWRfI2Mw+hWssq5cJqGunAsO0tv+lyzIRtBMBVYiZH1O8cs91DcxeNy2QV568pDeOS8vVPusybWTDbkJyY/wb6hfxDCZ287aJzhB2gIpu4XIArK7NdbZBleVfBM8cVcEr6K1/QD+VCbbu//1j+ByWIDAKU0oCDZIUvwFJTiFxH8GOGcZ6kf8MOqf+Ah2uF+BNmEjyaahoSAm/+3lAdnruHdpUamriUk0gnC3jRhZWUZ6k0DzoYcGW6bzmIpZSPwCPCIEOJQ4F/A3UKIfwO3SClXd/EYXXYRdh9czO6DU68xrO9/pjabwuNFd6xtrjxqPJOHlHDf2TOYaOUjJKAUlNqvt8lSVEWhXhQxV98HgMe0EzhSXQhAZeHuTA6+i0C3G9rskCX4C43yF8U0swMfN3ieIb8hxInKQHQ5LsunT3zetkspJGYNK0LYNZEilkZgHpNWEPSimSobjUCmfdM76U2fbyay8hEIIU4WQvwXuAf4KzAGeJ2EHAEXl65Cc0x26XD2PRhbUcBeIw0H8olThzA+jSDwOARBCB9eVcRNSPP1iXyi7cGD0ZPY1G8fikULZ6gfUyyMKKEGCvAUGPcpEc34iGkGp6qfd0rP4rTO4oReA0Ik+1fiavOkuFAn983ZKbIzDfWiAWfA9hHkxnCzCh9dhZH1e6eUcpZj+79NDcHFpcuxJtRUHdAsVCW2rsn0BZx5zeH2a4/DNGRcQ9j3GlmWz4ZqOC9yPQA/Kx9D9erbmS5WUyXNIngyLyYIaGakqEQVku3qQA6Ri/k0EkuoW7ujKbk0RhraYxqyhKQQsQDSmLYQOz4U1VMkrvWemSoroRTnI+g9Y0+kF32sWZGVj0BKeWGCEABASnlFF4zJxSUJa8LK1OPGuS/TynFkWQEjy8wJ2Qw71aQxhXpUBSvA5rbv7cH+Y2Iag9ejslEOYIiophCj1EQTefgKjWPKRYPdIW2u7wBUISkKGf0LZq+p5si/zuSl+ZuyfN7Ur1Mdo2mWtuSMVonXFgCaQ1ES6U0aQTbzeq8abxbkikDIRhAMEEK8LoSoEkJsF0K8KoQY0+Ujc3FxYGsEGUxDcRpBthfOMzKSF8mxgGFe0swicIUBD89fcoCd1+BVBJtlGUNFld2voEnm4zUFwUO+uynD8B1sEUbWsgwb0USLNhmagbMgXbak1wjiQ1QFJGkEzlVzSj9BL5qosvMR9KIBZ0GujDYbQfAc8CIwCBgCvIThMHZx6TaslWC2PoLLj8jSSVtQzhXhy7ko/KvYvfTU9/KoCptlBaPFNiYIY2XfSB5qfqy8xShhROtskoYgiLYaE3+DWSq6JM8LgEAn22kifWax8dspJC0fQaqm9c3hVBpB5jF0p00+K0GQI5nFMR9BLx6kg2wEgZBSPi2ljJo/z5A7gs6lj+CMjEmH12xWM214P3649/C0xyXymn4gNRjRSsGIljZnwasKXtQOQxGS8zzvEZEqIbx48mNZzeOUzUSlwlZpaAnRkFH91ApfLQwYbrl1gXO42fNEVuNrK6HMGVprfTzWM0zc+jrXeZ4DoDmUqoNZG3TjNz0bs09cfkUXjmVnyZH53yYbQfCREOI6IcQoIcRIIcS1wBtCiFIhRGmbZ7u4dAJF5gRaVuhLe4zXLEanpajvny3BiE5UTycIFA458GCqpCE0Gs2WmD6/3z5mnNhCLUXsiJiNdIKmIGg1VuMC7Kii8zzvZTeodILAMg3ZM6hIqjd09MqbuNTzP/yEUyaVtZ1Z3H0zWjar51ybYDsy3Nlrqnlx/sa2D+xEsokaOsP8/dOE7RdgPKfrL3Dpcn6w13CiuuT0DCt9q49xNE05hWxojWh2NrClfViToUcV3HjiJNg8CrYtpknmmfcVcOH78NjR7K58yyJ9DJubVQgAEcNHYGkEwahOKe3zE+hSx0eEMN647RFNEo7GBJdTWZIAzVX2+yliHboenykNbU+siftDUY3/LdrK9/cc2mal0PaSa47gTOxM+OhZjxj9qjP9r3c2bWoEUsrRGX5cIeDSLaiK4Ef7jYxrWp+I5dTdGbt2MKLZztdUGoEQAhExcggWm//+XkWB/JhyvEgfSzOGliAizWi6tIVTa1ijVLRRZ8gxFoBH1T+zyH8x+QSTjvn+A587BJdDGEggGOuRMEjUpqx71NaKP3Hv395fxS9fWsR7S7PrMdwe2ptHkBv291wYY3YJZV4hxBVCiH+bP5cLIbxtnefi0t34bNNQx798+4wqtc9PzGK2fBBMOB6AP0TOA8zchvxYPsJCfSxBfOhSkE+IxmDEnnCDUY1SYWgEUZn563fz/5YCcLi6iDwRph/JPYm/2dxgawSKM48ACeHY8R6iqXsjt1MjqKw3hFFDMNnxvLO011ncm8mVcVpkYxp6APAC/zDfn2tuu6irBuXi0hG8tkbQsfMX33QsxQEvW+pa+f1rSygvMu385vU8Vnjq0X9gj4+m00h+7OS8WOTQAjkeiUILfvIJ2mWrAUIRnVIMjSDR1JPIym3xJiS/iKRcYMYSyhzmLAmEYz0SvEJL3RKzLUGQZkXbFQ1r+mAV6pwRCNkIgn2klNMc7z8UQizqqgG5uHQUa8XeUY2gOGBMzD8+cBQ/PnBUiuubgkD1xAsBk29805gSXsQ6OcgYhyefQq2VutZInGlokJl0FmxDEKiKiJtJAsSavA+mmq0YWohm1hV68+tt9n5dAuFYRVQPWkrTUNvhoxl3dyrZTJq5EjXUF8tQa0KIsdYbM5ksdQUrF5cexPIRtFcQXHv8RPYc0a/N42zTUBruKr+FvYMPAIIDxpTh6TeEQaKGxz5bx6ZaIwGtNaIxSTEqmMo21tUeVUA05hfwY2gWRylfMjvwCw5TjPXYTa8vTTo32TSkpTYNZRwBSbNzW8d/umoHG2ta2jgqNX0pj8AiF8YI2QmCazBCSD8WQswEPgR+2bXDcnFpP94O+gh+fvg4Xv75QWn3W1fL5KgGkN48qjBqEAW8CrJkBMNEFcsXf0FVneG4bY1oTDJLWZeJRvYTy9JeTxEiblUfEIZGMEMxCv5OFWsyP1gkNiGn1Qja+KwS90qHGSoV5z72BYfe+VHmcaUhOyd/jsysJrmSCZ3xP1sIoQCtwHjgCvNnopSyY39pF5cuxJqoU014O0OxmcPgaUMj8DgEhd+jovYfwXhlM+/5r+Ut33UcriwkFI4wQlTatY1e8N/C/spSDlEWJ11PVeIFwSHKYooTGuCkQ6YyDbUx6X+xrib1ddpJRz/+7BLKOnZtl8xkFARSSh34q5QyJKVcLKVcJKUMddPYXFzahWW6aWuV214mDzErjbYRKeOMNvV7FXwVo+33Y5RtPOG7g/zQDvwiymw91vDved+tPO27Pel6qhBxq/rLPK/xgPdvtkkpk2lJQpxpyIuWcsXt3Hb6Q7OTr9ONto0+mVCWI+PNxjT0rhDiNNHZ2SMuLp2M5SOIdrIguPrYCQgBewwtyXics/zF2IpC1P6jko4Z3rocgFn65DbvqyoC5j0Wt+0gdQm/8LzS9qCljIsaStctre2oodTvu2I2aG+JiVwgV0abjSC4GqPQXEgI0SCEaBQiy4wYF5duxA4f7WRBsOeI/qy77bsM6ZeX8ThLEOw9sj8/P3ws9B9p77s7choAv236IwAz9enJF0jAowDzHkm73ycizPdfytWeF5P2GVFDTYQUI7rJIzSWbmngp0/Pt7uXGce14SPo1lpDfVEjyI0BZ5NZXCSlVKSUPillsfne7Vns0uuwEso6WyPIFqtpzvFTBhn+gpJYiYClMiYU/qftxxI5ipeiyX2dxotNXKK+DkCZtiPj/YpopVw0cIXnFXYT38btk1JCsJ6gp5iIVPGg8fjn63hnSSXLtzryE9Z8xD5iedp7dMYK/ObXl/L0nA1tHpdtGeqTlFkME9t3elwuMbLJLP4gm20uLj1NVzmLM7HkD8fZrwv9hlPZtqL6C2Gv8/lL+a18rcf8BUv1UYBR9dTJKcpnvOe/lt94/0UJTQyOZm5iUyFi3c9e9d0Qt0+TQGstrWoRUQxBENsX+3zGvH0OL/lvbvtBgbe+3kooYmgTzuJ2EU3ngY/XpCxqB/D45+u44ZVv2rx+MKJx46vf2HWZUhINc6/vPp733ZrVmF2yI60gEEIEzOqi5UKI/la1USHEKIy+BC4uvQpfF5mGUmH1Pijwx3IyBxYb9YVqmh3xFCfdw5p+B7KNMp6MHgNA2MzjLOoX3ybzHt8/7NcjRSV+jOv8OPzrlGOwEtPAyDr2EHNm67o0BIGnmAgqXqcgaMfnY8mMWWuq+NmzX/H2EiNpzekjeG7ut/z57eU88snarK+bimfnfstTszdw7wer0h6jthiF9FKV22gvizfV8dd3V+z0dTKRI5ahjBrBT4Evgd3M39bPq8D9XT80F5f24fWYmcXd8O2b85uj+PTaI+K2DSoOALCtPj6wzhIWs00HsWUmOufQPdJef5SoxKcbeQObZVnKYwaJ+HDPYmLOYU1KaKmhRS1O0ghsE4yWHAWVaNO24uDrW+JX6Vc+v5Dr/mOEvDaZLTBTdkBrB1b2dSqntoXaapiE6smu93Mmznx4Dvd+uDqtJtMZ5HwegZTyHinlaOBXUsoxjoqj06SU93XjGF1cssIyDXXHKqy80M/w0vgyE9b7xC9/vs9oGP+2vi8HBe9hlj4FAC2/Iu31h4gqVM3IKg4S34PhNe0AZmpTGYihEfxP2x+AEhHLG9AcGoGGGqct2BpBS6xMtUWismB9lqmihJ6fZ9TMTyzb3VGy8RGozaYgkIU7dS8Av+lTqmvJYIraRcjGWXyvEOJAIcTZQojzrJ/uGJyLS3uwzDXjBuz8JNERDhxbxg0nTuJ3350Ut720wJjIDxlfzmZik78MpA9HzRMhFN0QBCEZLwhapZ9G8vAKYyW7SZYDUOJINtM0HVprCZo+AqdpyDadNcZqE9n70paUSD/Jx9qIpj0kLascPZyzEeCeFkMQ1Mmd1wistqE1zeE2juw41jOt2NbIqOveYPV243m3NwZZuLEuw5ndSzbO4qeBvwAHA/uYP3tnc3EhxPFCiBVCiNVCiOtS7D9fCLFDCLHQ/HErmrp0GCEEz120H89fsn+P3f/Cg0fbE79FWaHhOxjWP59h/WMhqKoi2JLG7BMgjEczTExBfJzn8BME8dIsY9fZbAkCh0aAFgY9QlDJJ4oHVcTsLbbpbMuCpPsmTsaWYMi02LeOWV/d/hpDx9z9SdJ1MqGGjMmzmUC775VISb7xd6pt6XpB8MbiLeZvQ/h+555POfX+z7vsvu0lm+qjewOTZDsDYoUQKoYv4RhgEzBPCPGalDKxQtYLUsrL23NtF5d0HDiuvKeHkESZKRiqm0J2ZBEYppTn9aO5Wn0h6ZwiNRpnGvpEn8YCfRwzlNW04kd3rOEsYeLUCDD9C1HhMTWCFKahtUalGKu/MoCu6xynzONDfQYRPDHTUIbns6aG1xZtyfQxtEk2TmzF7PjWVsG+bOhnagS1zV1nGrKeSFWs0GZDIFc1dZ3w6QjZJJR9AwzqwLX3BVZLKddKKcPA88ApHbiOi0tOM3mIkXaz7+hS218AhkbwkDyVn4evSDonX4RR9RCaFEQwzomaX9cgvjjn7xZTIyiO0wiMyU3DgyY8ccfbS7pWY3XtIzYRik1f8JDvbn7reSbr5+usIC09gz/CQjFLbvjY+cY4lmmoKzUCC6tOVU/luLRFNhpBObBUCPEFYIdDSClPbuO8oYCzA/MmYL8Ux50mhDgUWAlcJaXs3q7NLi5dzMiyAub/7mhK833MXBlLEjNaYQre1/eKO75Z+slXw0T1kOkoNiYRa34MSp/RpAZ4KnoMm6Thd/A7+hVI3RQEQjWdxSnCR6PG19kbF3ZqrFinmxVOY9VG08/OnRel1fZ1RNQQdk7h1VGsaK62akjtDNbnZ7U93ZnueV1JNoLgpg5eO9V/TuKn8DrwLyllSAhxKfAkcGTShYS4BLgEYMSIER0cjotLz1Fu+gkuO2Icn64yonVURSAEhBwNam6PnMkp6izyRIRWWxAYqBiTdCt+fKYgsNpiAgSck2PUmNy2NWmMRbUdy+BYlZo+CL8pCIQAaU74FcIom52Naaiz6v9Yw8p0OdXSCMTOT95WkcJopnjVncR6FCuQIdKF99oZMiWU7QYgpZwJzJFSzrR+cGgGGdgEDHe8HwbEGRGllNWOaqaPAPFLo9hxD0sp95ZS7l1RkT7kzsWlt7P/mDLGm1FNqhB2hm5IGsJgsRxDEJ/hLE4QBFZjmhb8rNSNr9YGOdA2HV3jfdHubxAMG1+rVVVBNOGJixpauLGOJ2etR5oagaFdGFZ3GTH8EhUYZqOsOgR0mmkoex+BN8E0dO5jc3n00/YltFnCrUsnZ/ORPL1cI8jkI3jO8TqxPu0/aJt5wHghxGghhA84E3jNeYAQYrDj7clA+i4dLi59DMXx7auURoe0oPQRlD4CImIIAkfoqOUD2CFLeFA7ie+F/sCXciLO9fpTZjnr2kZj5RyRhrPY49AIHpy5ht+/tsQWBGCUqRZCQNQwL/nM4zPlEVh0NJM78bxsLqNELR9BvGno01VV3PpG+6YP63ahLtUITNNQF1XG7SwyCQKR5nWq90lIKaPA5cA7GBP8i1LKJUKIm4UQln/hCiHEErMH8hXA+VmP3MUlR4mZCxR7gq2kP2CszoN4ySOERw8RcmgEVlmFHbI/OgoL5Pika1u+g/lrjXj7KCoa8RqBjUMQ+IgYGkE0XtnPLnw0/b5MJE6K2QQmqhHLR7DzpiHrdpFo95Uk0bTcEwQyzetU71NfQMo3pZQTpJRjpZR/NLfdKKV8zXx9vZRyspmtfISUMn0ZRBeXPoYiYiuqu6I/BGCFPpxW/PgJ49ODtDoEQaEwzDbbZea+CIA98UdQCQtPvP/AwjHpTxQbieqSmvrGuEPs/gMZE8o6NrklmkmyETqWRmAJvJ3Bul9Xmoasj8ZyFndE+HUHmQTBMCHE34UQ9zpeW++HdtP4XFz6HM4vvxWNM1ufzKjgc9RQTBAffhkmoLfQLI3EqbeuPMQ+p5q2BYEVJRRFpVb0o1ykyGLVQgRN38TL/psAuO/9hDSfLLzFHRUEVky9fZ0s5uPO1AisOblLBYH523ZMJzxkL5EDGaOGrnG8np+wL/G9i4tLOxEi9fwalF58hMmTrTRRys8OH8vug4tZMu03iAXPxiWTpWIw1bYzNYqHHZRRRj0qGhqxPAaiYUSCcq/oYZyHbKxt5anZ65PqKjnJJAjS+Q+aQlHufCe+8mdWzmIzUS7RWdwRLIEc7haNILWPQJcSpROS43aWtIJASvlkdw7ExWVXIW4qSDEHNJFPvt5MC9AsA4wwJ+GN43/MpXOntHn92YFfcHrI6E8QQaVKKUPVdCqoYxtWSQuJ0EL4RfzElOiE/e+Czfx3wWbuP3vPtPfL5CNIl2Pwl3dW8NTs+GY12ayOhR5NOc6OEDMN9ZyPINNnJ6XMmL/RmWSTWezi4tIlpLa8b5Wl+GSIIaKGJvLsSaQ9Rd2sOPuoVKlRjMnf2b/Ai4ZA8qlmCJYN+gDjvA6stDPZudOt8lvCyffJRiOwBEEeYfph+DP2uOmdbIaZhHW7cDeWoU4yhyUW+XO8706zkSsIXFx6kFQrPqtkBEATeXZ5bbUdkmCMMFJ2oqg0eYyIpFJHq3FrRf2JPpV5+gQ2mtnJ1nZNxt/LWtn3p4GShKYwmWz76falWgnbzuL0l0PIKDO1qShCcoz6JdDxzOCYj6DrZlxrMrd+JzuLU48JurfxvSsIXFy6G8c3PFWCkbMRTbMM2HVqlBSC4PcnTeKJn+yTtH2w2bQmgoeQtwiIL0pnTfghvESkx8489qfJ2LWybxcELmVR4JK4fanMP9+alUjTmYasST9ACGFmTNuZxSnPMBB6lNXSiFXZ2S5lsjuihszf1vMm/r0TPx/n/u6MKMqmDPUdQohiIYRXCPGBEKJKCHFOdwzOxaUv8tC5e3HWFmUtkgAAIABJREFUvsMZXV5gd/dyslEOsF/HaQQptIfyQj+HTxzAv6Lx3dIGmFFCUVSCHqPo3cnqLBb6LyaPoJ2lHMZLBI9tErIEhCqkPUFDrHtYKlKZdM5+dE7afWCshL1EWR74CTd7njC3tTHx6YY5ywqp9aCnPOesh+ew+w1vZ76WY2zhaNeXfbBGmfg5Jn4+zve9TSM4VkrZAJyIUTZiAvERRS4uLu1g/MAibvv+1DhTT7/8WL2hemKNdZplwA49TNUBrNisoHl99GKOD91ub7dKRERQiZiC4Ah1Ef1EMyPEdtuHEJYewnjsKBynj2Bd4BwGU21cJ8HG4+x4lmr+ttpWposa0qXkEMVodXmKOsu4TsojHZgVVa1saw9R6luTncaz11bTGmnb7m/dr2tLTEjzV2qNQCbc2rm/s2o4ZUM2gsD6D/0ORoG4mkwHu7i4tJ/EZjbLy4xG95tlOR4z9FBJ+Lbedfo0Dh0f8ycslyOYFHwciBWNi6Li8XppJtbIJopKAbFeB5E4QRA/sc5QjEbyiSvZIaLafm1NWPd47+Mc9T0AigNGQGK62jq6jGktlimszYnPdBSHzWBHj9DZUhfMfE6my1nO4m4wDdlZzG04i7Ve7Cx+XQixHKNBzQdCiAqg45++i4tLEqX5MUHwnT0G8eHE33NQ8B6+kLvbPoJE09D39xyW5GxuwU9IeuwEsige/B6FRlFkH+MnwmRlPQAr5TC7ub2CHt/lDGwBkrhqLiWWgaxL6Ecjp6izuNX7TyCmqaQLj5RS2sLHEkrWLV6av5GPV2xPPsksrR3FQ0SqqGhsrW9NfYMssMNHu7DEhO0sJrVGkGQa6qFaRNn0LL4OOADYW0oZAZpxG8y4uHQqPk/sq3jytKEIX77d39hqsp7KWZyMoI5CKswIoYhU8XkUmpSYIAgQZppYQ1gtZI0cYvsIbvH8k5PV+PqSlhkmMdqlQLTa4ay6lOxpag7N0ii3XeDz2PtSIWXMDJUnQuY249iGYJTz/zkv+STNOD6CioaCB32n+g13h7M4di/jd7KPIP64eGdxV48qRjbO4h8CUSmlJoT4HfAMMKTLR+bisgvh9Bf0z/fafgGAooCxuk7lI0jFDrOSKRhmIJ+q8IVvX3ubX0ToL5po9pUhUQhLD14R5XvqZ8njMqOJtGiY33ietbcXEkSXsNct77H38r/wuO8vANSZ/o0Cv5GenN40FNMI8syGOm1OfKZpSEMlggcP0Z0q62xZabo2s9gYn90LKKnERAbTUDe6i7MxDd0gpWwUQhwMHIfRPOaBrh2Wi8uuwUe/OpyXf35gnCAY0i8v7r3V5zhTHsH3Z8TKfzl7EEcwNIKvAvvb2/yE8RMhLPzmMYaPYIU0ehxEZWxasIrXDamawyWeN+ztBRgmmermMD9RYtsj0hhrP9PUlW6itqKGAPIJATILH0HEfiYNBRWdyE4IAmui7Y7w0USBYJH0Xk+/ryvJRhBY7vfvAg9IKV8FfBmOd3FxyZLR5QXsOaK/bf//xZHjGF6aj0eNfTULTcdrqvBRi7vOmM7fzpgOQKXsb2+P4sGrKuy2x972Nj8R/IQJC+NrbAmCLaYA+U30QvtYKzpISwhvKRCp3YSaOaUUmL2ZnXP7Ryu2c9lzXyGlRJPSjlxShKQfTW0LAjNqKCpV26+h7cQk3q0JZWnHkD7zuFflEQCbhRAPAacDbwoh/Fme5+LikiW7DTZs+EfsZuQQeB2rf8ve3pZlyPIhWDZ3ME1DHoURA2PRRQHCBESYjQ3GpBMx+xXkEWaRPoZ6WWAfa1UxDctYeCsYpqFUWHZ/uxumYzL71YuLeGPxVtZXt8Q5iwFGicosTEOmmcrWCLSdavSSLqSzM0nUCBIn92RncfK53UE2E/rpGM1ljpdS1gGluHkELi6dylVHT+CFS/ZnzxHGat6pEVgmobZKTFi7n4keQ7P080z0KMJ48HkUvB6Fo0J3AhAQEQKE7TaYYVS8RMkzt0UctSitiX1VZXyfgl97n+dg5WvUhIY3AVMIaSkm2WrTsTtvXU2csxhgtNjaLtNQFBUv2k6t5m1hlSAIghHNzozuLBJLTSRut+i14aNSyhZgDXCcEOJyYICU8t0uH5mLyy6ER1XYb0ystITTWWzRliCwTEcL5Hgmh/7J76IXAgK/quBVFGqkoXX4CccJgiuOnoRH6BSIVlqlP26CtjSCDduT04d+43kuLowUnI7f1CtggIZgxHYW18t8NCkYpWxr2yZumYZQiUoVVeho2TQxSEVLDXs1fWxcL+EaFz81n0Pv/Khj103A/hzS7E/KI+jFJSauBJ4FBpg/zwghftHVA3Nx2ZXxJGaP0Xb10XThpT6PglcVdtvLgOkstgQBqmH2KaKFVnwMdFQptXod+1OUfV4mh1Mi4uv95JmOX2tCS1VrKKzp6KazuIUA1ZRQQV0WGkGs2Y7lI+iwRvDWr7ms6lZ2E98mFcb7dFWVcbtONBnpaQRCsvO4l2oEwIXAfmaLyRuB/YGLu3ZYLi67Ntbq35lf0Fb4aLr9Po+CR1Xsid9PhIAI293JUI3tJaKZVvz8T9vfzva1ooacguCJ6LHGaehmxI9j3ELiRbOTw1LZ3yNRI17HK6JEpMoOWUKFqG974tVjGoHlI+hw6GfEMP0crXyZpBFYpCuYlw1Wol+iSShTbSFI0Ag6fPf2k40gEBBnCNTIonm9i4tLx7FMQ1boKLRdgkFN8232qQo+VUFHISxVAsIwDYVsjcD4XUwLrdJPNSWcGPqjMQ6r9ISjR/BN0fNZoQ/DT4R8h2P62ehRgKEVxExDyeOJaDr9I5UMFzsI42WH7EeFqKM5nFwfqKopxIptpvlJcwoCw0cQinRQEPiN+ks/87yGlKmL1+1MrZ/E66W7UlIeQQ+ZhjK1qrT4JzBXCPFf8/2pwGNdNyQXFxfLWZzvi/WNbMsMkk4jyPOpdpmKID7yCREgQggfN5w4CTw7jHsK3a7sGbXq+ZiCwNII3taMktdhPBQQ5DmvITBOCd3MJMXoOFZAMKWzOPYcOvdsPQcUWKaPoIoSxotNKQvIHXHnxzSGoqy//bt2QllUqkRQUdEJZWgqo+syfTZ2i2H+KRAh8gij6dL+jGLnp7101li5CukEYybTkHNfOKozf0MNY8oLGVQS2PmBJZCNs/gu4CdADVAL/ERK+bdOH4mLi4uNFT7q1AgylYKG9OWUh/fPtzWMSlnKXspK/CJCEC9n7DPcXh0DtGIlmRkCKNE09MvIpca98LKbshHFbHXZQoA1ulFwYIqyLmX4qD1OhzlHR1AlS8xqqcnHNjrLdD/zfSCmEXjQCCU88/UvL7ZfZwwtba6yXxbQmvLYnTENWeycaSj2uq41zNmPzOX9ZZU7PaZUZNQIhBAKsFhKOQX4qktG4OLikoS1ki1wCIKxAwoYUhJgS33qGP7qNHV3RpYX2I1lNsoKjlQXAoZ9N8+rQl4sAa3Vqi1kCoIj1IUskyP5oToTMAQAGHkFBSJW8K0VHxvkOELSy17KSr42J7RUdn+nQBshtlMjC/EJjQKCcVVSk5BW3oNK1PIRJAiCf32x0X6d0bTTEhMEhSLIr/+zGK+q8JcfTrO370x+ge0jsIaerkFPguyOG7NMPi7bMiPtJaNGIKXUgUVCiBFdcncXF5eUtJr2cqcgyPd5mHX9UWnPOXX6UH5++FiG/397Zx4fVXku/u8ze3YIYQ8IVBRZA8YVRNxRkKJSBf2pXLXiVYpLW4vaehVvvdTrrdbiUqvVat2XulfqhopLFS0im4AYARGEAElIJpNZ3t8fZ5kzk5lJAhmSkPf7+eSTOee855xnJpP3Oc/yPk9x4mTauzBgu5oqVfzp/xD51ghKOxRBvWkRWIrgCNdq/ur7HQe7NgFxSyGElzxHfCCo/DTgZRd5FFJrT6I9vvwTl7pfTpDHWdKhQILsxEhr7SqJqajpiJoWgU+iGV1DGSfy0G62u4xFdnkEeXHpZp79bFPCkL3x0TdeOGbtJ2l/0sriqNMiaDwuXRxob2nOZXsDK8zuZC9ZP9kRR6PRAHbnsny/u4mRcXJ8bq6dOMReiQxw97ljcLnEdg29GxtlVwh9OHqKeWJcEdSZiiB9PoixvyHJmWCdF1Je/GKsEyDSQP/P5nO99wm7NhE0LvK201zf0LWZrSfDuAkrNx6JNXINOcnoGorUU+Uy3nd+unIZe2ERHBJZTT/Haun0weKkeybECOKvLVlS9bhuDZoTLL45K3fWaDRpOWJgMS6Bnx4zqMXnFgbi5SAOG2BMdl5zXcIrsaN4JXRU4gkpXENN0UBiyQkrNTWEDz+mIrj3aPv4wbKRz9VBgBH0DkqAHGVMwDuVUbG0q9Q0K2cyl5CRNSSxjFlDadNRlYJwkF2+IoAEJeVkb2IEd+z+JfjhHj4zb7kHJSZU49f73DUkIgeKyFil1LvOH4w/1aZ052k0mr2nR2GA9f8zidH9uzY9OIkF5462X1uxBq8ng/Hvj/cqqLctgsyEkhSBMqeSEF5TEQCVa+3jXRwLz8KRGC5HETurdHUP2cXl7hcJkNifAGDKgniJ7ArVk4jZjyCjayjdRB5tABRVLqNcd56jbpLznq2SNZTkElIkusYydihz7G9L19CdQCqnXZ15TKPRtEN6FAbshWgeUxF4Mi1LdjxlBptZWDikUjsTLEVQUZnY6awLTkUQxW+Worg5fL7tGvqZ++9c632KC91GBZuvt8WvsWxTFSHlYfnAi6km37QIMi8oS+vaCRsWwE4xFIHTNbQ3PYOvePxzjrj1zYRHeVGGooqnkcKvno1nNiXfQ4XjcRenUrIURFsEiwcopZYl71RKLQEGZEUajUbTqtgWgeNR8pYfD0s7Pqj8jUpZ3Bs5HYBqlQvAKcN6kpeTOrvHihHsro8k7O/ibIEZM5TAbeFzeCh6KlXkEVPCAS6jPWXEnJZO/P27jiso/BIh6vLYY9xEqc/gGmpKEewyLYIC4gXmnHGFlsYIXl32PVurQ7Ah3uUtEDZahjpbVr70xWb7eMItlj/HhGeG8RP3ooRzjNdtpwgyrVrIkOOl0WjaC1YhOmfBur5d0//7BvElVD4FuD1yNs9Fx3FOw28AKO2aSzczw+e68MUMqH/cHmtYBA00JLlsLvG8yjD5xtgIG0/gVlwhipsaybXH1qWYeqz1DFExXFIRPHiIMbb+Xf7gXZDSz592Io8YY6ukgHrlpZvZ1jP5nJZYBNtqHKU2HjrVfilmfSRn1pBzLk+IY2z7CoBhUmGPjcti/G4LRfCpiDSqKSQiF4MZAdFoNO0Sa7pINXFkyjwJ4rfdSKtj/fh7dCxR3Pw8fDmr1AEAeNxC12glAN+qngnnW8FibzTRNdRHdvCq/wbj/pF6c2w8zlAt8bTW3BS9DqxSF3FF4MJDlF813M2P3R8yyvV1o3PSTuSWIlJ+tlNEiVTZh+ocZS6aaxEEG6Ic9ts3E+S0SXYNAeLIyEq4RdSwlHqZRf+cC8osWZrVtnoPyJQ1dBXwdxE5j/jEX47RneyM7Iij0WhagxMO6cFrX25JWbo6U6ezoPLb50xs+F3KMR6X0C1mLMjarLolHLNiBLmRGruP4S6Vl+AakmhjRVAjBaC+ByAvqZAdgM9c2WwpgqhyG5VRzcm+B7sanfPsZ5v4+ckHN34DpkUQwsd2VUQJcUVQ61jJ3FzPUH04rjwGyJaEY2IGxdP3I3DsMGsp9ZLKRmMtpZa2ZMZektYiUEptVUodjZE+WmH+3KyUOkoptSXdeU5EZKKIfCUi60RkboZx00REiUh5ujEajab53HFOGe9fe1xC9VILy0pIcFGYT6lBfAnxhFR4XC7+4J/F6lg/NqnuCcdCyktAwhQow93y04Zr2K6KEu8fNSb6ekeq6m53fMw13meZ7n474RzrSXvlNkOJhPHgVWGqzW5qPRylsy3++PY6pixYzK2vrUo84HBNbTcrn1rUNjgVQWpN8Mk3OxImfyc9k+SwgsX+0A7u895BgapJWKKRaBEYiqC37Gh0/zZLH40LoN5RSv3R/Hm7qfEWIuIG7gZOBYYCM0RkaIpxBcAc4F/NF1uj0WTC73HTrzg35THrodI5qWygN2BMjhkzjDAqoy52jWFiw+/s4nQWlkVgWQA7Vb7dxxggh3rEVAROi6DOXZhwnfneBxLfj9nfeNUPxiReQy55qtYOLPeQxhYBGNlG97+3PnGnGSyux8cO6ZLgGqoNZXYNfbWlhrP/9BH/k6xcTDxJHdusHgqjNz3CRPennBX7Z8JSvQRlY5bZ7kY1iY6h7LuGstl7+HBgnVJqvVKqAXgS+HGKcbcAt0GaJqgajaZVsWIEzknlcs9NzGm4giCBJi0Ct8uVtgCepQi6mpnnu8gn5phmZrr/iSeaGCwGqPd2aXStS9yvIhiuFcsiaDDTVqtUHl4idlpqd6mikFqW+n/KSv9/8ILv1wnXSgjKRuKKYKcUUUy1fR+nRZBKEWyuMs5dvz3u6nJO5smtO8XcdpuTfPL6iwRFYMYIPBKjkLr24RpqBfoCGx3bm8x9NiIyGuinlHol04VE5FIRWSIiS7Zt29b6kmo0nYBjBpdw5QmDbQXgDBpvk268FBsLNN0S0+uWtCWxQ3jxS4O9gGyXKrCf2sHwodsWgYpPiqkUwa+9j/GMb55xT0sRmBPpLgyXkLUGoJ/8wJnu9+kiteRKiDJXohWwbbdxz111DfxtsZGdU4+XndIVtyi7vIUzRpDKM1RnWgzOMh7ORWCWRfB44Bxjh7kqzWWmzO5scCWUxUi4RzR+b+Pza1ySOluuoeaUmNhTUklsvzOzsukdwMymLqSUuh+4H6C8vHzva8NqNJ2QRy8+AoBPKwwftDNo7JyQmnINuV2StqtXSPnwErVTMneRl+Aa6i2VREJBcCVaBA2+xooAoNy1BhcxO1gcJm4ROBnjWscY17q0MlfubqBnYYC73lrH7oot4IW1lVF65XQBBSVSxQ5VmLAuIdXK5DrTYnD2iXB+FB6sCqmGwnIpyyIwFEFyjaZUFgFAMTUJ8QPbIuiArqFNQD/Hdimw2bFdAAwHFolIBUYLzJd0wFijyS7xGEGa403MNh63iwcvLGfyyN6NjlnF50plOzUqhwge+yn+B9WFXrIDUqSPKm/qeAYYHc98tkVgKgLy0o63sNw9ABWVtQy98XU+/Ho7AXNVcz0+NoSM+1pxAmdZ61SuoaAZJM5xKIJoCteQ9Z5FJbqGIiqxiGCq9FGALlKT6BoyB2bK+NobsqkIPgUGi8hAEfEB0wG7aqlSqkopVaKUGqCUGgB8DEwxVy5rNJosEY8RpJ5UmsqfF+DQA4pZcO6YRsc2qB4AjJa17DKLyf0ifBlPRiawMFpOL9lhN7lxKgLxGArky9gA3ogeyuOR41kUNXoD5BLiKs9zQNwiqFZNK4J8R9jxvTXbqGuIsnpLDTmmIgjis8tMWCmkmeoAQXydgVUePByN8a0jXmC5hhrE7OsQjaCiEYZsXwiAVxJjCInB4ggNHuMzK6YmIVxs/UmyVX00a4pAKRUBZgMLgVXA00qpFSIyT0SmZOu+Go0mM6nSR53+6HATDeEzKYqvldGl7Eeu79lpFpPboHoyN3IpOyigUILkSOP0UeXLs8//afjnXB+5hJeiRpXUHAkx3v2lKaUh9PdJ6xcsdjkUxJeBS7jO8xgA23fHn7adFsE2M7W1ewqLIFX10jozhpDjNZ7sb31tFec+EE94dEuia+jFzzfy9FOP2McHyhau9jxrWysqyTVU7zWKDBZIMHWwuAO6hlBKvaaUOkgp9SOl1G/NfTcqpRr1M1BKTdDWgEaTfWzXUJpZJV3LS4tMiuJb1dOOCVgWgUXQ7INQhPEE7bQINnc7ijsjZ3Jz+AJ7n1VqwlkddLvZWGc7RXzS5bRG99+genBZw1X29izPqwDsqI0vUgtIAyHlQeFip8qlQbnjrqFo5hhBrWkRWAH1D9dVJhy3LAJLEbiJ8f6a+LKryzwvc6XnecplDdB4HcGOmOGqyiWU1L/Yqj7awSwCjUbTPnE14RraHUosk3DBUQckbGeyCCJ42GG6W6zy0hZW/MDKKHIGiwM+H3dGprGTxv2T7/PeAcDjnMo6VWofr/U1tgqC+BOC03Wm8nFm6gRoIGTeO6bEKDNhuoZCCRZB4/dnKUmrOF1yw3vbNWS6sNwSoyhFEedn/PMAxeotNZz+x8X8UF1PLNLAd3UeYkrIlfoEiyDbjWm0ItBoOhhDehXQo6B5fQNS4UqxjsA56Tjr7QBMGpEYFM7Y+QvYLsYEvTPZIjAnditV0wqo3jxlGD0LG78fq5OaVZV0mzuxrlGDp6jROfXKx3plyLsm1hcvEVzEEqyYAA0JSmi7MuoNHSBbmLjqOnpgrA5OZRFY7z1mK4LEKdRSBJa14yJGl5TV/A057nprLV9+V8UXm6ogGiaMh1oC5CaV2YivLE55qb1GKwKNpoPx+lXj+eSGE/f4fLNZWbOfLpMnu6ZiCDvE8HM3sgiU4eoplhrCyug7DHDh0QNSujyCSVVIc9yJCmppzzN5MjKBV6JH2vuiuPha9WVw/SM8Ej0Zr0QpoSphAVxAGhI6sW1XRRzj+pLTXJ8wdMebPOC7HUgdLI7GkiyCJLndVvqoGSz2EKWrqiEqbqIqcawzmJ3jdaOiYcK4CeInh/qUMQLtGtJoNK1CKosgE94k90dTWUVW569KlVg2wmp600VqEp7IwahflExdUre0hbmTE7aVL4+5kUv5efgy3o6WAVAgRm8Br8/PNmXI0V2qCJsTeHd2cYb7A/wStq+zODYCj8Q41v0FACNd31BIbcpgsaVQLIWQPDF7ktJHDYugmqCni+2Ossh1NsRRCqINhPFQp/zkSWKMIF5iQisCjUbTCqSqNZRpak+efA7skZ9mpMGz/qn8d/g8nouOT9hvuYYOc62hQBL7B6R60rX8+wAvRo9G+QoSjvtMBRXCx5+jk4B4FVKv20Wl2flsUG7QnsAvMYPHvRzF4V6MGr2VSyVeteAg2ZhS4VmWgGUUJStJyyKwYwTEyCdIvTuv0WKywRLv+BtTChUzXEN1pmsosVWl8TtLekArAo2ms9HUOoJMvDpnHFNG9ck4ZrOnHw9EJ1Gb1L/KObEnk0oROF1LARoIeBMXYzlrIn0bM+IHVvG5qmCYHWbgucRVY5fEsNxRTnZSQFQJpbLd3newa1Ma11CiRZBsyXglsRSG21wVHRG/vc/iQd//2a9jMYVEw0RwU4efXOoTUkuVdg1pNJrWxHYNpfnvP//IA3j8kiMY3teYSJ0T4rA+RU3GFtIpmFSdxyxSlbUI4ePhEY8CsCI2oJEicMYutlAMwEvm0/0xg0voV9ofgBKptuMayUXh3C4hhosdGNZDg/ioV14OkK2kCoVEUsQIPEQoptq8foyoElvhuFD4aSDi8jWyCAwsxaIgGqZBGa6hXEm0CLLdszibtYY0Gk07JsE1ZE40f5hexqnDe+PzuOzjLezhnlbBBGmZRQCwq2gIk0K3skr1Z6I36enb4ZaJ4WJk/Z+pNZVN76IAt110MtzioSvVRMxZ3Vo4ZvVf9rqFaExRqYroLtWEJIcgQgF1GS0C65jHLVzreYpLPa8yvP4BPESJ4LZTWD1E8RIhIj5iygMCUSW4xTjfT5gQPuN60QbTIgjQg13UJqwjMH7rrCGNRtMqZGqEPqRXod3M5j/GDgCgf5q+Bk5uO2uk/TqtReBwDZ0Q+t+EY+kUgc/jYoUaQAwXfk+SRZB0TjV59pN4TGE41HO70YW4a6gnO9kY68740B3xccAOM54QEj81KocCSa0ILEvghX9vJhyN4XG7OMa1DIAJri9wE0tQBC5ihkUgPts1tMOxViLHTBONxoCY4RoK4jNdQ/Di0u/47Nudbdq8XqPR7IdYE1yqOcU5t54xupSK+ZPomudrPDCJiSN6MaZ/F/O6qSernQ6ff3KLy7SKwOH+8SWlsboz9E2wJ3F/IQXU2iuGe8guvlQD2WW6giyXkRWPqJcANeRSSF3KYLG1LxiOsuDtdXhdwreqFwALfH/kJNcSoriJKkM2K0bwza6onQXlzKayVk1HlUIiIerx0aC8eCSKAq58ciln3fuhzhrSaDStS6ba9unmmWcuO4rbpo1MfRCjKmaqhjeAbWEox3ST7CZKV/ra2WrTn+wayuQnsebwQCH5qs7e3VN2slV1jQ8zx1llrevxU61yKZBgY0UQCXHejrsZKhWA0ePA7XLRVeILxga6thLBZVsEbonhVWFCeOyCfJb1AZBr1l2KRSO4Yg0ElZ+BvbrgJZK0jsD4rYPFGo2mVUi1StWac9I9zR82oJizy/ulPGZcS+wa/QHThVOc56Ni/iRG90vVayAp7TLNBOfMDGqUs99MiyDPrG0UIESh1NnrC5xYZa3r8VNDDgVJHcIA6j+4h4m1L3Kx5x/G/V1CgdrNEa7VLIqOsgvYRXETNhePGa6hMCG8fBMzVjxHcHN5wxzAUUfJbJ8ZxIdy+fARSVlrSKePajSaVsGaTwsD3hTH9mymEYH/O3sUPz/pII7+keH2sfzazvn0x+67OTv0G8AI6P7p/EOB9IogU/39TA107HsGisgzLQIrtdRpEVhYweOIclGjcimQukYlJr76wKiV2c3KEHIJ43c8DUAMYXFsuHEN3NSFzXRPYvglTEh5WaeMBo3DXd9QaSoNqxKrK2IpAj/K7cVLNGU/Au0a0mg0rcKBPfKZe+oQ7j6vcT+BPZ1m3C6hR0GAn50w2FGr35zJHBPad9KLT9QhAFw0diCnDDP86+kmuExNcpILvjmJOVxDubFaQHGl53kAfsCwCM4aU8pLs432nNW2ReClhlwKSHINKcWgkNGw/kDXd8b9XUK3BqPX1v9FfmKX0IgoN9tqrXRVI0bQgJc3Y8bn/XZsDLWma8yyCMRUBCG84PIarqEU/QhxjWapAAAd/UlEQVS0a0ij0bQKIsJlx/6InoXxvH6VIW7QHJzn5fnd5n2M7bmnDUl9jmNSs4K5o/p1oWL+JIrNALXT+yMONXX4gOKMFoHTNZSr6hgmFZzlfh+IWwT/c+YIRpYaSsHK3vmO7tSoXCOP39ExjFANBRIkpsS2CDxuFyX1FbwXHcEKNdAuoWHECAzZ3A7XUAgfw+ofZG74EjtGkmspgrDxO6j84PYZDWwc5U+1a0ij0ewz9nSicc7JuWZjd2uiHtO/K6/NOcY86mjr6DgnaFY8zTEDwunEWPvbU/n61tN4+rKjMiote1VuoAi/qrcnejBaZt55TpkdiH559jgmn3gCAM8HR1Njroj2NOyOX3D3VgDWq97kSAMeIkaMIFzJJlUCxGsjxZzBYociAKjFaN9Za1oPeWa9IYk6YgRuY6yKxushxfSCMo1Gs6/Y03lGUlgEbsfKMmvxl9Pd4nRzHNTTyKS5ZNyghOs5/eQKlRA8zlQO2z6v6wAAzvO8ZR87atiBTB3d194eUVqE6nsWhy5soJIiert3GPKFd9sy3/fyh1yBoQgOZDP5BHG7BH+szl4xXW+uk3ARI2YqAj9hXKJoUInxGCs4bTXpcYfjMQLcZvcyh0XSkXsWazSaDkJrLljKMy0Cp1vHyvBxzt1O11D3Aj8V8ydx4lCjZtDxQ7oD8ZaQqbCaxBx3cHfOPaJ/wjHbNTR8Gg3iY5xruX0sx9f4+VdEqMQI4NaYgWNP2EgLvXfROlavNTqKWb0OCqQOjyi8sXp7NbNlEfgkYlsEOeYTfyipzlAQPw3KTZEYisCKEdQrH+I2XEzRcNyKie6l664ptCLQaDQ2raII/FZ3rvi1LDeRMyUy09Ptf08dwfvXHkd+IL3TwoorFOf5uXjcwIRjtsJxuah1d7FLS/w6/B8EfOmVC0A1hiKQkHHO2h922+fbioAgAQnjQtlBYlsRELFXOFsNZpIVAQhV5DNMKgDF85+sAyyLwFAEb6/4zh5tKWrJ0oytFYFGo7FpjQfOHHOidTuCAFaGj7PGf6aMIJ/HRb/i3ATXkCRFDiyLwOdxNSpI5zwv6I6vaH4memxGKwPgmsnlAHzz3RYGzH2Vr7bU0EN20aDcbFKGpVJAEF/MeIq3FIDTNWRZBJYiSK48Ckbdo/HuL5nufoeAOa4eHy6PMfbNLzfaY+3GNNoi0Gg02SK+oGzvr2VZFbne+NO8Va7ZmZufKesnFSqpa4K1XuGsMX0JeBKnMmcJ55CpCELKQwgvAW/maa/8IMPNtPF7I0Bcu3Udl3leppYc222UL3W4wsb6hGSLYJfKd7iGTItANVYEFse4lpEjRjygXvlsi8AqaQ3x/gc6WKzRaLJO8lP3nnBAcS6zxg9i+uFxv33cNRQf15yc+ME90zfBGdQ9n4r5RkOa2lAk4ZhTZTR4jGsY9YSkSYtAAkaswOp2dkPBa9AAXWW3nVGUTxBPxDhuxQisrmvWfWJKyMGY4Bu7huL4iJKPYV3sJgfxGOd4zZLZ8zwP0WvrBGCgTh/VaDTZpzXWK7lcwnWnHcLAkjx7n+Uacj6pN+fptndRDr+ZPLTJccmuoVgKi8By6ySPbXwxoyhcgTk5V3viBfJ2mxZBgQRxhY1Ab13S4jCrllAUl71OwFIEBf74s/fCqOGC6iPbjWqnCLsJ4HLHYw0AP3G/S0nNakAvKNNoNNkky1kpVtpnuvTRTKjkoj8psK41rE+heU78WIPXUARW0becJoLFePyExWtbBCjDL3NfZLJtERQQxGXm/lvltdeoUgCejk4AjPUElmvIihG8+fNj7dvMCl/N05FjKZEqCqmjXnJRuHB5Dcuiu+xEiJEjDYTFsDp01pBGo8k62XI9uFO4hlp7UquYP4lfnnKweZ/4jYJe44nesgiacg2BYUUUYiiCQHQ3O1Q+8yPnEsJLg3JTIHUMqDRWKlvrCL5RvRlY/zfeiBlP+g14bKvCihEkvmdhK13pRjVFUkudy7CgXB5DETzs+18CpmupwWUpgpZ8Is1HKwKNRmPTVBvKPcWKEZw6vJe9r7kWQUtkSqVcviydwR8iZ/CU+aTeHEUQ9hTYFkFutIZqZbm5hDrJJZ8gB1R9AsQVDCSW2t5NDsVmiWrLNZT8nrerIjwSo7/8QK1YiiBeotvKOgq7/Ihk8e+TlavuY8LhMJs2baK+vr6tRdFkiUAgQGlpKV5v+qCbZs9pzayhVIgIn1x/AkW5Xv7x69eBxAVnrYWlCJwWQU5hMbdFfmJvN7WOACDqK6Cg1niaz4vttlcCA+wmly6uIDnhKt7Jn0R1fV7Ka+xWOQw2C9TZzeyTPmCrdPUg2czaBsO1FCEu30D5HoB/rKluccvQlrBfKIJNmzZRUFDAgAEDsqYxNW2HUorKyko2bdrEwIEDmz5Bs8dk87+nR2Fi8/qWuoaaMxEeOaiYc4/oz+UTfmTv65qb2GEt4GlaESh/AQVSCUC+2s1OFZ/sa8mlUOrIjVRR7SpMdwl8xGsFWRaBs59zz0I/lTWGIiiW3dTEjPhDOBhvdHOL9yHALEaXRfYL11B9fT3dunXTSmA/RUTo1q2btvjaIYO6p34abg7JrSdbA4/bxa1njKC0a7zPclFuohXZZLAYUP5CCqgjh3oGqwq+Vn3sY7sll95SiYso1ZJeEXgkar9OFSMoDHjtVcwQL4Xt6XaAve8Ql7GorI7sKoL9wiKA7PnONO0D/ffNLs3JzEnFKz8bR3041vTAFHg9++Y5NNkiaE6MQPkKKZAgo13rCEiYd2Jl9rGD+/fB9a0RKK6WInt/rs9NXUN88rfSP8HhGnLECETiDXEAvlfFAIwuK+fsxc8S2rySF/03Ao1be7Y2+4VFoNFoWoeWKtxcn8fuHdBSWmoR7OmzQJecJIugGYogv6Q33aiiBKPG0GYVX0uQX9zLXh+wrjb+3m/58fCEa6R0DTnehCDUOCwCK+gsIow+sJQv1IH2sQ7tGhKRiSLylYisE5G5KY5fJiJfishSEVksIk2vHGmniAjnn3++vR2JROjevTuTJ08G4KWXXmL+/PkZr7F582amTZsGwMMPP8zs2bNbJMOtt97a5JiZM2fy7LPPtui6e8LSpUt57bXXsn4fTesw/6yRlHbNadYk2Vp4W6gI9jRYWpyfFCPwpb7v4z89gpdnjwMgr+eB+CTKwaZrpt5Rr4hu8fjDN7t9FAQ8zBo/qJGicloEqbKGRLDXJUBi9pG1wtuyGKymN9kia4pARNzA3cCpwFBgRoqJ/nGl1AilVBlwG/D7bMmTbfLy8li+fDnBoJFp8MYbb9C3b7zm+ZQpU5g7t5EuTKBPnz57NUk3RxHsK7Qi6FicPqoPi391fNZWrqbC52lm+uhe3qcw4OV+szcypLcIjv5RCSNKTVdPVyMpYbhUABDxFMQHFscVQQ05lPXrwnWnHdJIUb0eO8x+3WB64ZM/Xme66UanIjDHVZlB6jAefnb8gWSLbMYIDgfWKaXWA4jIk8CPgZXWAKVUtWN8HrDXCVI3v7yClZurmx7YAob2KeS/Th/W5LhTTz2VV199lWnTpvHEE08wY8YM3n/f8CU+/PDDLFmyhAULFjBz5kwKCwtZsmQJW7Zs4bbbbmPatGlUVFQwefJkli83aqdv3LiRiRMn8s0333DuuefyX//1XwBMnTqVjRs3Ul9fz5VXXsmll17K3LlzCQaDlJWVMWzYMB577DEeeeQRbr/9dkSEkSNH8uijjwLw3nvv8fvf/z7h3smkOvfbb7/loosuYtu2bXTv3p2HHnqI/v3788wzz3DzzTfjdrspKirizTff5MYbbyQYDLJ48WKuu+46zjnnnNb6c2j2E5prEbRG1uTJw+LrF5osMQFQbCoC1zeElZuIJwfMxV1WsxuAWhVIm/30q/ClnOH+AIhP+E7X25BeBazeEs8Q2qB62q+tUY9ET+IG1+N06daLa046qGm595BsKoK+wEbH9ibgiORBInIFcA3gA47PojxZZ/r06cybN4/JkyezbNkyLrroIlsRJPP999+zePFiVq9ezZQpU1JOxp988gnLly8nNzeXww47jEmTJlFeXs5f/vIXiouLCQaDHHbYYZx11lnMnz+fBQsWsHTpUgBWrFjBb3/7Wz744ANKSkrYsWNHs++d7tzZs2dzwQUXcOGFF/KXv/yFOXPm8MILLzBv3jwWLlxI37592bVrFz6fj3nz5tmKT6NJhW8fBYuTaZYCKuyLcnnpFquhUhXgcbu45qSDGDe4BHJr7WG15NhWVLLCSlV62sn8s0YyvG8RmM3TYg7rwNIXz/im8kjwZAZ17ZLVhIlsKoJUUjdS7kqpu4G7ReRc4NfAhY0uJHIpcClA//79kw8n0Jwn92wxcuRIKioqeOKJJzjttNMyjp06dSoul4uhQ4eydevWlGNOOukkunUzglRnnnkmixcvpry8nLvuuou///3vgGE1rF271h5n8fbbbzNt2jRKSox+qsXFxc2+d7pzP/roI55//nkAzj//fK699loAxo4dy8yZMzn77LM588wzM39IGo1JS2ME+xSXG/HnQ3AnRdTidglzThhsHGuI++trCdjunlSZVyeGbuMQ2ZDyFgGvm0uOGcSsf1ydNgbgcbsI4Wtxye6Wkk1FsAno59guBTZnGP8kcG+qA0qp+4H7AcrLy7O4vm7vmTJlCr/4xS9YtGgRlZWVacf5/fEsgHSpe8lPACLCokWLePPNN/noo4/Izc1lwoQJKfPrlVJpnyCaunemc1PJd9999/Gvf/2LV199lbKyMtsq0Wgy0VxF0GaJw/WGi9kjMbt6KgDeeKZPDFfGhXHrVCnrzGJ06VjoiCVYWMFiy9rI1MSnNcimSv4UGCwiA0XEB0wHXnIOEJHBjs1JwNosyrNPuOiii7jxxhsZMWLEXl/rjTfeYMeOHQSDQV544QXGjh1LVVUVXbt2JTc3l9WrV/Pxxx/b471eL+GwkbJ2wgkn8PTTT9vKyOkaaop05x599NE8+eSTADz22GOMG2dkWHz99dccccQRzJs3j5KSEjZu3EhBQQE1NTWpb6DRkJ0FZa3KBS/ynipjdsPPEoPoSRO/VUivIENbTSc3nT6UV342LuMY6xbd8oyHtupgOMPovSdrfwmlVASYDSwEVgFPK6VWiMg8EZliDpstIitEZClGnKCRW6ijUVpaypVXXtkq1xo3bhznn38+ZWVlnHXWWZSXlzNx4kQikQgjR47kN7/5DUceeaQ9/tJLL2XkyJGcd955DBs2jBtuuIFjjz2WUaNGcc011zT7vunOveuuu3jooYfs4PEf/vAHAH75y18yYsQIhg8fzvjx4xk1ahTHHXccK1eupKysjKeeeqpVPg/N/oXXvW+f9f/fkf0Z3b9L808YeAxXcD2vxI7K6JoJho000VOG9eJ/zhzBCUOMctc3TxnG+UfGVwn/ZaZRlXTm2IFGbCAF+Wa/Autu/YqN9NIfqrO7ql72dEVhW1FeXq6WLFmSsG/VqlUccsghbSSRZl+h/877BwPmvgpgdxdrio/XVzL9/o/544zRnD6qT9MntCKjbv4nVcEwQ3oV8PpV4+MHbjIm8gH1jzOmfxeev3ysfeiyRz/j9RVbuOe8MXTL83HO/YbVnu79Wp8HwOJfHUdp11w++rqSGX/+mAcuKOeSR5ZkPL+5iMhnSqnyVMf2mxITGo1m/+TIQd34+LoT6FUUaHpwlkiOA9zAFUTDRonordWhhGNWb2UB8vwtm2KtGklH/agbq2+ZaKe6XjQ2u8UWtSLQaDTtnrZSAlWmb37l94lrk56LHkO92VH+u13BhGOWk0Wk5YrAiaUE9tYSaA7tPFqj0Wg07Y9MpTis2ksBr5u8ZlQ6feCClN6afYq2CDQajaaF/P6cMt5YuZWyfl04pFdiKepfTx7KkF4FHHtQ94RqpOk4cWjPJsdkG60INBqNpoUcd3APjju4R8pj+X4PM02f/r4s4rc3aNeQRqPRZIlsLwRrLbQiaCXcbjdlZWUMHz6c008/nV27dmXlPkcffXRWrqvRaDovWhG0Ejk5OSxdupTly5dTXFzM3XffnZX7fPjhh1m5bmsQjTbtD9Vo3r/2ON64enzTAzsZPQuz23wmE/tfjOAfc2HLl617zV4j4NTMTWWcHHXUUSxbtgyARYsWcfvtt/PKK68ARgXP8vJyZs6cyYABA7jwwgt5+eWXCYfDPPPMMwwZMoSbbrqJDRs2sH79ejZs2MBVV13FnDlzAMjPz2f37t0sWrSIm266iZKSEpYvX86hhx7K3/72N0SE1157jWuuuYaSkhLGjBnD+vXr7ftbVFRUcP7551Nba1RSXLBgAUcffTTnnHMOF154oV00b+bMmZx++ulMnTqVuXPnsmjRIkKhEFdccQWzZs1i0aJF3HzzzfTu3ZulS5eycuXKlGWyAR588EF+97vf0adPHwYPHozf72fBggVs27aNyy67jA0bjOJcd955J2PHjkWzf9KvOLfpQe2MN685NqvX/+T6E5rVSzlb7H+KoI2JRqO89dZbXHzxxc0aX1JSwueff84999zD7bffzgMPPADA6tWreeedd6ipqeHggw/mP//zP/F6E8va/vvf/2bFihX06dOHsWPH8sEHH1BeXs6sWbN47733GDhwIDNmzEh53x49evDGG28QCARYu3YtM2bMYMmSJUyfPp2nnnqK0047jYaGBt566y3uvfdeHnzwQYqKivj0008JhUKMHTuWk08+GYiXyx440AiQpSqTHQqFuOWWW/j8888pKCjg+OOPZ9SoUQBceeWVXH311YwbN44NGzZwyimnsGrVqj36/DWabHBgj/ymB+0FPQrbbrEc7I+KoAVP7q2J1RSmoqKCQw89lJNOOqlZ51llmw899FC7xDPApEmT8Pv9+P1+evTowdatWyktTaxiePjhh9v7rHvn5+czaNAge1KeMWMG999/f6P7hsNhZs+ezdKlS3G73axZswYwmuvMmTOHUCjE66+/zvjx48nJyeGf//wny5YtszuoVVVVsXbtWnw+H4cffrh9PyBlmewtW7Zw7LHH2iWtf/KTn9j3fPPNN1m50u5XRHV1NTU1NRQUOLpCaTSarLH/KYI2wooRVFVVMXnyZO6++27mzJmDx+MhFovZ45JLRlslod1uN5FIpNH+VMcyjWlu7ag77riDnj178sUXXxCLxQgEjCeSQCDAhAkTWLhwIU899ZRtUSil+OMf/8gpp5yScJ1FixaRl5eXsJ2qTHYmuWKxGB999BE5OTlpx2g0muyhg8WtTFFREXfddRe333474XCYAw44gJUrVxIKhaiqquKtt97K6v2HDBnC+vXrqaioAEhb+bOqqorevXvjcrl49NFHEwK906dP56GHHuL999+3J/5TTjmFe++91y5zvWbNGju+kHzdVGWyDz/8cN5991127txJJBLhueees885+eSTEzqZ6X4Gmv2JD+Yez6tzMpedbmu0IsgCo0ePZtSoUTz55JP069ePs88+2y4PPXr06KzeOycnh3vuuYeJEycybtw4evbsSVFR45K3l19+OX/961858sgjWbNmTcJT/cknn8x7773HiSeeiM9nLJe/5JJLGDp0KGPGjGH48OHMmjUrpZWSrkx23759uf766zniiCM48cQTGTp0qC3XXXfdxZIlSxg5ciRDhw7lvvvuy8ZHo9G0CX275DCsT+qy0+0FXYZ6P2T37t3k5+ejlOKKK65g8ODBXH311W0tli1XJBLhjDPO4KKLLuKMM85o9vn676zZ17y1aivhaIyJw3u3tSh7TaYy1Noi2A/585//TFlZGcOGDaOqqopZs2a1tUgA3HTTTfaiu4EDBzJ16tS2FkmjycgJh/TcL5RAU2iLQNNh0H9njWbP6RQWQUdTaJqWof++Gk322C8UQSAQoLKyUk8W+ylKKSorK+0UV41G07rsF+sISktL2bRpE9u2bWtrUTRZIhAINFpQp9FoWof9QhF4vd6Ela0ajUajaT77hWtIo9FoNHuOVgQajUbTydGKQKPRaDo5HW4dgYhsA77dw9NLgO2tKM6+oqPKDR1Xdi33vkXLnX0OUEp1T3WgwymCvUFElqRbUNGe6ahyQ8eVXcu9b9Fyty3aNaTRaDSdHK0INBqNppPT2RRB41ZdHYOOKjd0XNm13PsWLXcb0qliBBqNRqNpTGezCDQajUaThFYEGo1G08npNIpARCaKyFcisk5E5ra1PE5E5C8i8oOILHfsKxaRN0Rkrfm7q7lfROQu830sE5ExbSh3PxF5R0RWicgKEbmyI8guIgER+UREvjDlvtncP1BE/mXK/ZSI+Mz9fnN7nXl8QFvI7ZDfLSL/FpFXOpjcFSLypYgsFZEl5r52/V0xZekiIs+KyGrzu35UR5C7JXQKRSAibuBu4FRgKDBDRIa2rVQJPAxMTNo3F3hLKTUYeMvcBuM9DDZ/LgXu3UcypiIC/FwpdQhwJHCF+bm2d9lDwPFKqVFAGTBRRI4EfgfcYcq9E7jYHH8xsFMpdSBwhzmuLbkSWOXY7ihyAxynlCpz5N639+8KwB+A15VSQ4BRGJ99R5C7+Sil9vsf4ChgoWP7OuC6tpYrScYBwHLH9ldAb/N1b+Ar8/WfgBmpxrX1D/AicFJHkh3IBT4HjsBYIepJ/s4AC4GjzNcec5y0kbylGBPP8cArgHQEuU0ZKoCSpH3t+rsCFALfJH9u7V3ulv50CosA6AtsdGxvMve1Z3oqpb4HMH/3MPe3y/diuh1GA/+iA8huuleWAj8AbwBfA7uUUpEUstlym8ergG77VmKbO4FrgZi53Y2OITeAAv4pIp+JyKXmvvb+XRkEbAMeMt1xD4hIHu1f7hbRWRSBpNjXUfNm2917EZF84DngKqVUdaahKfa1iexKqahSqgzjCftwIFUzZEu2diG3iEwGflBKfebcnWJou5LbwVil1BgM98kVIjI+w9j2IrsHGAPcq5QaDdQSdwOlor3I3SI6iyLYBPRzbJcCm9tIluayVUR6A5i/fzD3t6v3IiJeDCXwmFLqeXN3h5AdQCm1C1iEEePoIiJWsyanbLbc5vEiYMe+lRSAscAUEakAnsRwD91J+5cbAKXUZvP3D8DfMRRwe/+ubAI2KaX+ZW4/i6EY2rvcLaKzKIJPgcFmdoUPmA681MYyNcVLwIXm6wsx/O/W/gvM7IQjgSrLRN3XiIgADwKrlFK/dxxq17KLSHcR6WK+zgFOxAgAvgNMM4cly229n2nA28p0AO9LlFLXKaVKlVIDML7DbyulzqOdyw0gInkiUmC9Bk4GltPOvytKqS3ARhE52Nx1ArCSdi53i2nrIMW++gFOA9Zg+IJvaGt5kmR7AvgeCGM8UVyM4ct9C1hr/i42xwpGBtTXwJdAeRvKPQ7D7F0GLDV/TmvvsgMjgX+bci8HbjT3DwI+AdYBzwB+c3/A3F5nHh/UDr4zE4BXOorcpoxfmD8rrP/B9v5dMWUpA5aY35cXgK4dQe6W/OgSExqNRtPJ6SyuIY1Go9GkQSsCjUaj6eRoRaDRaDSdHK0INBqNppOjFYFGo9F0crQi0HRaRGS3+XuAiJzbyte+Pmn7w9a8vkbTmmhFoNEYBf9apAjMiraZSFAESqmjWyiTRrPP0IpAo4H5wDFmnfyrzYJ0/ysin5o15WcBiMgEMfovPI6xWAgRecEsorbCKqQmIvOBHPN6j5n7LOtDzGsvN2vzn+O49iJH3fvHzJXbGk3W8TQ9RKPZ75kL/EIpNRnAnNCrlFKHiYgf+EBE/mmOPRwYrpT6xty+SCm1wyxV8amIPKeUmisis5VR1C6ZMzFWqo4CSsxz3jOPjQaGYdSm+QCjttDi1n+7Gk0i2iLQaBpzMka9mKUYZbW7YTQaAfjEoQQA5ojIF8DHGMXGBpOZccATyqh+uhV4FzjMce1NSqkYRrmOAa3ybjSaJtAWgUbTGAF+ppRamLBTZAJGGWLn9okYzV/qRGQRRn2fpq6djpDjdRT9/6nZR2iLQKOBGqDAsb0Q+E+zxDYicpBZMTOZIoxWkHUiMgSjlLVF2Do/ifeAc8w4RHdgPEZBOI2mzdBPHBqNUVUyYrp4HsboUTsA+NwM2G4DpqY473XgMhFZhtGS8GPHsfuBZSLyuTJKRVv8HaOd5BcYlVuvVUptMRWJRtMm6OqjGo1G08nRriGNRqPp5GhFoNFoNJ0crQg0Go2mk6MVgUaj0XRytCLQaDSaTo5WBBqNRtPJ0YpAo9FoOjn/H3nsUGGrUnj9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss_list, label='Minibatch cost')\n",
    "plt.plot(np.convolve(loss_list, \n",
    "                     np.ones(5,)/5, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 563,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU1fn48c9JyALZSVgCIYR930JEFFQQZBMFEVTUCojS+lXR+msrLnWrttRS61qtCoiWRQQpqIAixQouQMISVgEFQhZCgGQSSEK28/vjTGKELJNk9jzv1yuvmbkzc++5mWSee7bnKK01QgghBICPqwsghBDCfUhQEEIIUUGCghBCiAoSFIQQQlSQoCCEEKJCE1cXoCGioqJ0XFycq4shhBAeJSkp6bTWukVVz3l0UIiLiyMxMdHVxRBCCI+ilDpe3XPSfCSEEKKCBAUhhBAVJCgIIYSo4LA+BaXUAmA8cEpr3du6rTnwIRAHHANu0VpnK6UU8AowDsgHpmutd9TnuMXFxaSmplJYWNjwkxAABAYGEhMTg5+fn6uLIoRwMEd2NL8HvA68X2nbHGCj1nquUmqO9fGjwFigi/XncuBN622dpaamEhISQlxcHCbWiIbQWnPmzBlSU1Pp0KGDq4sjhHAwhzUfaa2/Bs5etHkCsMh6fxEwsdL297XxPRCulIquz3ELCwuJjIyUgGAnSikiIyOl5iVEI+HsPoVWWusMAOttS+v2tsCJSq9LtW67hFJqllIqUSmVmJWVVeVBJCDYl/w+hWg83KWjuapvnSpzemut39ZaJ2itE1q0qHLuhRBCeKXcwmK+PpTFy18eYm+axSHHcPbktUylVLTWOsPaPHTKuj0VaFfpdTFAupPLZhdnzpxhxIgRAJw8eRJfX1/Kg9e2bdvw9/evdR8zZsxgzpw5dOvWrdrXvPHGG4SHh3PHHXfYp+BCCLdSVqb5MescO1Ky2XE8hx0p2RzJOofWoBREBgfQu22Y3Y/r7KCwBpgGzLXerq60/QGl1DJMB7OlvJnJ00RGRrJr1y4AnnnmGYKDg/nd7373i9dordFa4+NTdUVt4cKFtR7n/vvvb3hhhRBuw1JQzK4TOew4ns2OlGx2ncghr7AEgLCmfgyIDeeGfm2Ij42gX7swQgIdMxrQkUNSlwLDgCilVCrwNCYYLFdKzQRSgCnWl6/FDEc9ghmSOsNR5XKVI0eOMHHiRIYOHcrWrVv59NNPefbZZ9mxYwcFBQXceuutPPXUUwAMHTqU119/nd69exMVFcVvfvMb1q1bR7NmzVi9ejUtW7bkySefJCoqiocffpihQ4cydOhQ/vvf/2KxWFi4cCFXXnkl58+f56677uLIkSP07NmTw4cP8+6779K/f38X/zaEaNzKyjRHss5VBIAdKTkcOXUOAB8FXVuFML5vG+Jjw4lvH0HHqCCn9e05LChoradW89SIKl6rAbtf+j77yT72p+fadZ8924Ty9A296vXe/fv3s3DhQt566y0A5s6dS/PmzSkpKWH48OFMnjyZnj17/uI9FouFa665hrlz5/LII4+wYMEC5syZc8m+tdZs27aNNWvW8Nxzz7F+/Xpee+01WrduzcqVK9m9ezfx8fH1KrcQomEs+cXsPGG+/HemZLMrJYe8C6YWENHMjwGxEUzo14b49hH0axdOcIDr0tJ5dEI8T9OpUycuu+yyisdLly5l/vz5lJSUkJ6ezv79+y8JCk2bNmXs2LEADBw4kM2bN1e570mTJlW85tixYwBs2bKFRx99FIB+/frRq1f9gpkQou7yi0pYuSONxd8f5+DJPMDUArq1DuXG/qYZaEBsOB2cWAuwhVcHhfpe0TtKUFBQxf3Dhw/zyiuvsG3bNsLDw7nzzjurnAtQuWPa19eXkpKSKvcdEBBwyWtMBUwI4UwnLYUs+u4YS7amYCkopl9MGL8f3Y0BseH0jXFtLcAW7l06L5abm0tISAihoaFkZGTw+eefM2bMGLseY+jQoSxfvpyrrrqKPXv2sH//frvuXwh3VlqmOZSZx86UHJr4KCbFt6WJr+NG4e9JtTB/y098mpxBmdaM7tWamUM7MLB9hFvVBGojQcFF4uPj6dmzJ71796Zjx44MGTLE7sd48MEHueuuu+jbty/x8fH07t2bsDD7D2ETwh1kny8y7fbHc9h5IpvdJyycu/BzzXrx1uP8bUo/urYKsdsxS8s0Xx7IZP6Wo2w7epYgf1/uuiKOGUPiaNe8md2O40zKk5sYEhIS9MWL7Bw4cIAePXq4qETupaSkhJKSEgIDAzl8+DCjRo3i8OHDNGlS92sB+b0Kd3M4M4+tR8+yIyWbnSk5HD19HgBfH0X31iEVbfbxsRHsTbfw1Op9nCss4aGRXfj11R0bVGs4f6GEjxJPsPDbYxw/k0/b8KbMGBLHLZe1I9RBQ0XtSSmVpLVOqOo5qSl4sXPnzjFixAhKSkrQWvOvf/2rXgFBCHdx8ZU5QGSQPwNiI5iSEEN8bAR9Y8Jo5v/Lv/O4qCAGd4zk6dX7+NvnP/D5vpP8bXI/urWuW60hPaeARd8eY8m2FPIKS4iPDecPo7szulcrhzZNOZN8Q3ix8PBwkpKSXF0MIRqsqivzJ8b1YHSv1rRr3tSmNvuo4ADeuCOecckZPLV6L+Nf28xDI7rw62s64VfLF/quEznM33KUtXsy0Foztnc0d1v7C7yNBAUhhNsqvzJfui2F3MISBtjhyvz6vtEM7ticp9fsY94Xh1hvrTX0iA79xetKyzQb9p/k3c1HSTyeTUhAE+4eEse0K+OIifDM/gJbSFAQQridS67M+0Qzc2gH4mPtc2UeGRzA67fHc32fDP64ei83vr6FB6/twn3DOlFYXMryxFTe+/YoJ84W0K55U54a35NbLmvn9sNJ7cH7z1AI4TJZeRfYmZLNnjQL+UWlNr1n94kcp12Zj+0TzeUdI3l6zT5e2nCI1bvSOJV7gbwLJSS0j+CJcT24rmdrfH08Z0hpQ0lQEELYRXFpGQcyctlxPJudJ0xWzxNnCwAzIqipn69N+2kZEsBT43syJSHGYUnfKmse5M9rUwdwfZ9o/rHhEMO6t2Tm0A70bxfu8GO7IwkKdjZs2DAee+wxRo8eXbHt5Zdf5tChQ/zzn/+s8j3BwcGcO3eO9PR0Zs+ezYoVK6rc77x580hIqHIUWcVxZs2aRbNm5qpq3LhxLFmyhPDwxvnHLRzrVG5hRS6fHSnZJKdauFBSBkCr0ADiYyO4a3Ac8e3D6dUmjEAbg4KrjOndmjG9W7u6GC4nQcHOpk6dyrJly34RFJYtW8bf/va3Wt/bpk2bKgOCrV5++WXuvPPOiqCwdu3aeu9LiMqKSsrYn5FrDQAmvXNajqkF+PkqerUJ447L2xPf3swLiA4L9KhZvOJnEhTsbPLkyTz55JNcuHCBgIAAjh07Rnp6Ov3792fEiBFkZ2dTXFzM888/z4QJE37x3mPHjjF+/Hj27t1LQUEBM2bMYP/+/fTo0YOCgoKK1913331s376dgoICJk+ezLPPPsurr75Keno6w4cPJyoqik2bNhEXF0diYiJRUVG89NJLLFiwAIB77rmHhx9+mGPHjjF27FiGDh3Kt99+S9u2bVm9ejVNmzZ16u9MuJ/M3MKfm4GOmz6B8lpAdFggA2LDmTEkjgGxEfRqE+r2tQBhO+8OCuvmwMk99t1n6z4wdm61T0dGRjJo0CDWr1/PhAkTWLZsGbfeeitNmzZl1apVhIaGcvr0aQYPHsyNN95Y7dXUm2++SbNmzUhOTiY5OfkXaa9feOEFmjdvTmlpKSNGjCA5OZnZs2fz0ksvsWnTJqKion6xr6SkJBYuXMjWrVvRWnP55ZdzzTXXEBERweHDh1m6dCnvvPMOt9xyCytXruTOO++0z+9KuNTFX+zHz+bb9L6S0jKy84sB8Pf1oXfbUO4c3J742Aji24cTHSYXDd7Mu4OCi5Q3IZUHhQULFqC15vHHH+frr7/Gx8eHtLQ0MjMzad266jbMr7/+mtmzZwPQt29f+vbtW/Hc8uXLefvttykpKSEjI4P9+/f/4vmLbdmyhZtuuqkiS+ukSZPYvHkzN954Ix06dKhYdKdy2m3hWYpKytiXbqlo49+ZklPRvFP+xX5tt5b42DCKRinoGBVEfHtTCwhoIrWAxsS7g0INV/SONHHiRB555JGKVdXi4+N57733yMrKIikpCT8/P+Li4qpMlV1ZVbWIo0ePMm/ePLZv305ERATTp0+vdT815bcqT7kNJu125WYq4b4yLAXsTPl56ca96bkUWZt32oY3pX9sOHcP7UB8bDg95Ytd1IF3BwUXCQ4OZtiwYdx9991MnWoWoLNYLLRs2RI/Pz82bdrE8ePHa9zH1VdfzeLFixk+fDh79+4lOTkZMCm3g4KCCAsLIzMzk3Xr1jFs2DAAQkJCyMvLu6T56Oqrr2b69OnMmTMHrTWrVq3igw8+sP+JC4e4UFLK3rTcihrAjpRsMizmQsC/iQ9924Yx7Yry5p0IWoUGurjEwpNJUHCQqVOnMmnSJJYtWwbAHXfcwQ033EBCQgL9+/ene/fuNb7/vvvuY8aMGfTt25f+/fszaNAgwKygNmDAAHr16nVJyu1Zs2YxduxYoqOj2bRpU8X2+Ph4pk+fXrGPe+65hwEDBkhTkR0UFJVy+twFu6ZJzi0s5utDWRUpoPel5VJU+nMtICGuOfGx4QyIjaBndCj+TbwjEZtwD5I6W9hEfq+X+ubIaf6wIpm0nAIui4tg5tCOXNezVb1nv6acyWfBN0f5KPEE54tKCWjiQ9+YMGsK6AjiY8NpKbUAYQeSOlsIOzp3oYQ/rz3Akq0pdIgK4rcju/JR0gl+8+8kYps3Y8aQOKYk2JYnR2tN4vFs3t38E1/sz8RXKW7o14Y7B8fSNya81uydQtibBAUh6mDL4dM8ujKZdEsB917Vgf83qhuBfr7cP7wTX+w3ef6f/WQ/L204xNRBsUy7Mo624ZcO4SwuLWPtngzmbzlKcqqF8GZ+3HdNJ+66Io7WYVIbEK7jlUFBay2zKe3Ik5sY7SWvsJg/rz3I0m0pdIwKYsVvrmBg++YVzzfx9WFcn2jG9YlmZ0o287ccrfgZ07s19wztwIDYCCz5xSzZlsL73x0jw1JIx6ggnp/Ym5vjY2jqLyOEhOt5XVAIDAzkzJkzREZGSmCwA601Z86cITCw8V69bj6cxZyVe8iwFDDr6o48cl3XGmfwDoiN4PXbI0irtBbAZ8kZ9IwO5ejp8xQUl3Jlp0ien9ib4TbOHRDCWbyuo7m4uJjU1NRax+4L2wUGBhITE4Ofn/uvPWtPuYXF/GXtAZZuO0HHFkH8bXK/eq20dc66atjHO9Lo0iqYe4Z2pGeb0NrfKISD1NTR7HVBQYiGyi8q4etDp3n2k31k5hZy79Ud+e3ImmsHQngSGX0kRDW01qSczWdHpYlhBzLyKC3TdGoRxMr7rmSAnVb7EsITSFAQjUp+UQm7T1gqgsDOlGzOnC8CIMjfl37twrnvmk7Etw9nSOcoSQ8hGh0JCsLraa357qczLNhylE0/ZFFaZppMO7YIYnj3lgyINWsAdG0V0qiWXRSiKhIUhNcqKinj0+R03t18lP0ZuUQG+XPPVR0Y3CGSAbHhhDfzd3URhXA7EhSE18k+X8Tircd5/7vjnMq7QJeWwfz15j5M6N9WOouFqIVLgoJS6iHgXkAB72itX1ZKNQc+BOKAY8AtWutsV5RPeKYjp86x4JujfLwjlcLiMq7u2oJ5UzpwVZcombMihI2cHhSUUr0xAWEQUASsV0p9Zt22UWs9Vyk1B5gDPOrs8gn3kFtYTNKxbMpsGDKdX1TKxztS2fRDFv5NfJg0oC13D+1A11YhTiipEN7FFTWFHsD3Wut8AKXU/4CbgAnAMOtrFgFfIUGhUfrvwUwe+3gPmbkXbH5PVLA/vx3ZlTsGxxIVHFD7G4QQVXJFUNgLvKCUigQKgHFAItBKa50BoLXOUEq1rOrNSqlZwCyA2NhY55RYOIUlv5jnPt3Pyh2pdGsVwouT+xHRrPZZ1ApFl1bB0l8ghB04PShorQ8opf4KbADOAbuBkjq8/23gbTAzmh1SSOF0Gw+Y2sGZ80U8eG1nHri2s8wREMIFXNLRrLWeD8wHUEr9GUgFMpVS0dZaQjRwyhVlE85lyS/m2U/28fHONLq3DmH+tMvoExPm6mIJ0Wi5avRRS631KaVULDAJuALoAEwD5lpvV7uibMJ5vtyfyeOrTO1g9rWdeeDaLrK0pBAu5qp5CiutfQrFwP1a62yl1FxguVJqJpACTHFR2YSD5eQX8ewn+1llrR0smH4ZvdtK7UA4SN5JyMuANgNcXZKaWdLgp68AG1vFYwZBi652L4armo+uqmLbGWCEC4ojnOiLfSd5fNVecvKLmD2iCw8M7yy1A+E4qUmw9FYoyIH7t0JkJ1eXqGontsHS2yD/jO3vuf4l7wkKovHJPl/EM5/sY/WudHpEh/LeDKkdCAc78AmsvBeCW0BxIXz+BNy+zNWlutT+1fDxLAiJhjs+gqAWtr2vqWOy90pQEA73+b6TPGGtHTw8sgv/N0xqB8KBtIbv34TPH4e28TD1Q9i1GL58Go5shM5u0iChNXz7Gmx4CtoNgtuWQFCUq0slQUE4ztnzRTyzZh9rdqfTMzqU9+8eJCuOCccqK4X1j8G2f0H38TDpHfBvBoPvg6T3TKDosAV8XbyKYGkJrPsDJM6HnhPhpn+Bn3sseStBQTjE+r0ZPPmfvVgKinnkuq7cN6wTfr5SOxAOVHQeVsyEQ+vgigfguufAxzrXpUkAjP4zLJsKiQvg8l+7rpwXzsGKGXD4CxjyEIx4Bnzc539DgoKwq7Pni3hq9V4+Tc6gd9tQPph5OT2ipXYgHCzvJCy5FU4mw7h5MOjeS1/TbSx0HAabXoDekyEo0tmlhNx0WHILZO6H8f+AhLudX4ZaSFAQdrN2TwZ//M9ecguL+d2orvz6GqkdCCc4dQAWTzEjd25bCt3GVP06pWD0X+CtofDVn+H6vzu3nCf3moBQaIHbP4Qu1zn3+DaSoCAaLPt8EU+u3stn1trB4imX07211A6EE/z0FXz4K/BrCjPW1j4XoVVPuGwmbH/XXKW36uWUYnJkIyyfBgHBMGMdRPd1znHrQYKCaJDjZ84zbcE20nIK+P3obsy6umPNtYPMfaYKbYvQNs77pxU/O3XQjOd3dWdsbXYtgTUPQmQXM5QzvJ1t7xv2GOz5CNbPgbvWmBqEIyUtgk9/Cy17wO3LIaytY4/XQBIURL3tPpHDzEXbKSnTLJs1mIHtm1f/Yq3hu9fhiz9i84xNFIx6Hq643/H/uMJ8Rv970TStJMyE8S+5ukTVS02C/9xn+ghueR8C6zDnpVlzGP4ErP0dHPwMeox3VCkh5Xv4ZDZ0uhamLIJA969BS1AQ9bLp4Cn+b/EOIoP9WXT3IDq1CK7+xb8YfjcBrniw9i95reHbV+GLJyD7GIyZC77y5+owJUXwyUOwewmEtYOkhaZ5pXVvV5fsUmVl5u8puBXc+m8IqMdiSgNnwPb55u+r80jHDActK4N1j0JIG1NO/yD7H8MB5L9M1NmH21N4fNVeekSbvEUtQ2r4h6o8/O7K2TDyWduH301ZBF8+ZSb4WE7AzfNNm6ywr4IcWP4rOPq1aVoZNAteG2iaV6Z94n61tD0fQVoiTHyzfgEBzAXGmL/ABxPh+3/CVY/Yt4xgAmzGLutcCc8ICAAyNETYTGvNPzYc4tGVexjSOYpls66oOSDkZsDCsXDkS5OnZdSf6jYe28fHNB+Nm2eCysKxZp/CfrKPw4LRcPw7mPgWDJtjmleufQKObTapItzJhXNmZnKbeOh7W8P21Wk4dLseNv/dDGm1p8Jc+PJZiLkM+nhWbk8JCsImxaVlzFm5h1c2HmbywBjmT0sgOKCGimbmPnh3BJz9yXSuXTaz/gcfdC9MXQZnfoR3R5p9i4ZL22F+n7kZ8KuPof/Un5+Lnw4te8IXT5q8Qe7im5dNxtOxf7XPhK9Rf4LSItj4XMP3Vdnmv8P5UzDmr+5X06qFBIVGJrewmE0/nCI9p8Dm95y/UMK97yfyYeIJZl/bmb9N7lvzCKMjG2H+aNBlZvidPcZjdx0Nd6+DshJYMAZ+3NTwfTZmB9fCe9ebtvR7NkCHq3/5fHnzSs5x+P4N15TxYtnH4ZtXoc8tJleQPUR2gsH/Z3IjpSXZZ59nfjRNUv1uh5iB9tmnEymtPXdFy4SEBJ2YmOjqYniMvMJibnv7e/al5wLQOjSQ+PbhDGgXQXz7cHq1CbtkneOsvAvc/d529qVbeH5iH26/vJZ1sXe8D5887Ljhd5ZUWHwLnP4Bxr8M8b+y7/4bg63/Mh2gbQaYSVTBVS6Hbiy7wwTgB5MgNNp5ZazK8rvg8AZ4ING+f1cX8uDVeIhoDzM3NPzKfuntcPR/5ncW0to+ZbQzpVSS1jqhqueko7mRuFBSym/+ncTBk3nMndSHwuJSdqTksPNENmv3mPZUP19FzzZhxMeGEx8bQXRYII8s301W3gXeuSuBET1aVX8AreG/z8PmedBpBEx5zzHD78Ji4O715gtizQPmSnb4Ex5XRXeJslLTHPT9P3+ZLK4mo56HNwbBxmfhprecU86qHN1sUkwPf8L+FxoBITDyaVh9v+nE7ntL/ff14yb44TMY8bTbBoTaSE2hESgr08xetpNPkzP4+5R+3Dww5hfPn8orZGdKDjtSstmZkkNyag6FxWUARAb5M3/6ZfRvF179AUou/PwPFT/NpA9w9MSn0mL47BFTM+kzxcxlwI6BwccXWvRw/2Gw507ZOBlQw9fz4OCnprlk1PM/J4urzZfPwpaX4J6NEFPlxaVjlZXCv6426SEe2G5mL9v9GGXwznDz+3wwsX6jhUpLTAqN4ny4f5vbZD2titQUGjGtNX/6bD+fJmcwZ2z3SwICQMuQQEb3as3oXubKpri0jIMZeRzIyOXKzpHERNRwNZl/1jQxpHwLI5+BIQ8756rd1w9ueBUi4kwn4Z6P7H+M9kPhtn87bDGTBjt9xHwJldjYP6R8YOyLdc8QetUjps193aOmecXZGT13LILMvab26YiAAOacxr4IC0aZdBRTFtZ9uGvSQsg6YOYkuHFAqI3UFLzcm1/9yF/XH+TuIR344/geKHt+YZ/9Cf492bTz3/Qm9L7Zfvuui/RdtqfOsFX2MTP0MSLOpFCIiLPv/u1h8S1w/FuY+Ab42FAzC4+t/2S0XUvMDOKb3oZ+t9ZvH/VRkAOvxUOL7jD9MyekpHgPPn3E5Ei6fblJtWKL/LOmnK37OCd1RgNJTaGRWpGUyl/XH+TGfm148no7B4TyNWW1hmlrIHaw/fZdV236mx97i+4Hy243wzanLnNN00l1Dn8Jhz+H6/5kZok7Wt/bYNs7JlB2v955kwj/96L5wh3zF+d80Q6cDqEx8NE0eGeEuSCwJZB+Ndc0b42Z6/YBoTYyJNVLbTp4ikdXJjO0cxTzpvTDx8eOf6j7VsF7402+mXu+dG1AcKS4Ieb8/IPM8E13mchVWgyfPwbNO8Hlv3HOMX18zNyAvAzY8g/nHDPrkFlBLf4uE6CdpctIM5gBzPDnI1/W/PpTB5yfddWBJCh4oZ0p2fzf4h30iA7hrV8NtN96yFrDN6/AR9PNcMaZX5px3t4sqovpYG3dx6Ro/u4N83twpe3vwulDZiWxJv7OO267QdD3VpN2JPuY44/3+ePg1wyu/aPjj3Wx1n3g3o2m2XDxLZC4sOrXaW2W/wwIMSOjvIAEBS/zY9Y57n5vOy1DA1g4fVDNs47rorTEjPbZ8BT0mgR3rXbNylWuEBRlcgD1uMF8Ua39vfl9uML5M/DVX0zWza6jnX/8kc+YUUtfOPiL+tAXcGQDXPMoBLdw7LGqE9rGTJjsNBw+fRi+fMaMUqrsh3Xw0yaTM6pZDVmCPYgEBS+SmVvIXfO34eujeP/uQbQICbDPji/kmf6DxAUw9LcmMZ0Hj66oF7+mJkHflQ/C9nfgwztMHh5n2/SCOe5oJ7WxXyy0DQx9BA6sMXMHHKGkyATfyM4mOZ8rBYTA1A9NVtUt/4CVM39O+1FywWRZjerWsDQubkaCgpewFBQzbcE2cvKLeG/GINpH2ikrY266SUT343/hhlesV4qN9M/m4gR9742zfyK1mpzca4Y9DroXWnZ33nEvduUDEBZrsqiWldp//9vfgTOHnd88Vh3fJmY95eueg30fw/sTTI1t61tmBN6YP7v/gkR1IKOPvEBxaRmzF23hXFYaC2/pT++Q85B7vuE7zjlh+g8u5MEdy03eeWG+lMNj4aMZP49QadXTscfU2nwJB4abTKau5NfUJJL7aJqZQ2DPxefPZcFXfzV/a11G2W+/DaUUDHnIfO4f/xrmjzRl7TrW6/4vJCh4gZXL3+e1jMcI9cuHVXbeeYi1XbV1Hzvv2MOVJ+hbfIupMTyQ5Ng+lgOfmFTW4+a5x2S6nhPM5L4NT5vlMDtc1fB9FuSYtTeKz5tagjsO7ex1k/mfWDYVSgph9AuuLpHdyeQ1D7f/s9fpuu2PnG7agdYjbVjRrC6UD3QZDSE15Dxq7DL3m1nFCTNMeg9HKC6ENy4D/2D49Wb3Sb2RcwL+fbNpQpnwRsMmteWkwOIpJsPohNehXwPXSnA0SxqcOwltPS8LKsjkNe9UVkbeumfouf0Vkvzj6fXAxxDsBleQjU2rnnDZPaYd3FHj1L9/w3xp3rXafQICQHg7mPm5Gaq7apZJTnj17+t+YZK2wwxkKC6EO1dCx2scU157Cmtr/8R8bqKR9hh6uOJCSlfMJGT7K3ykRxB5zyoCJSC4zrA5ZiLfukftP4chNwO+/rvJatpxmH33bQ9NI+DOj82M500vmMSIJUW2v/+HdWZioG8AzPzCMwKCl5Og4Gnyz8IHE/Hd/zFzi2+j2aTXiWtVQwZT4XjNmpuJS8c2myyk9rTxWSgrNqOe3FUTf5NWe9hjJnHe4ptN/0Bttv7LpBFp0d3MHHfliCpRwSVBQSn1W6XUPqXUXqXUUqVUoFKqg1Jqq1LqsKzJg2AAABo5SURBVFLqQ6WUG4xFczPW5ShLU5N4oOhB8gc9yPX9bEzYJRxr4AyzfOXnT9hv+crURNi91KQFb97BPvt0FKVMjWniW2a95wWjTZNXVcpKzSzgdX8wo3emfyb9Vm7E6UFBKdUWmA0kaK17A77AbcBfgX9orbsA2YD3zAaxhxPbYP51lOafZUbZkxyLHs0T1/dwdalEuV8sX/nPhu+vrMw0RwW3gqv+X8P35yz9p5r1nnMzzHDdtB2/fL4o3yyQ9P0/zboOt35Q+0I/wqlc1XzUBGiqlGoCNAMygGuBFdbnFwETXVQ292NNQKcDwri/6Yvs1N154/Z4AprYuEiKcI6Ow0zb/9fzzJdiQ+z5CNISzWTBuub1d7UOV5v+gSaBpr/g4Fqz/dwp6+PPzIL2Y/5i+0I/wmmcHhS01mnAPCAFEwwsQBKQo7UuTyiTClTZta+UmqWUSlRKJWZlZTmjyLYrKzU5cez2U1wpAV1/5sW+wfqMIF6c3Nd+M5aFfY36k+kD2Phc/fdx4ZxJUd0m3nTgeqKW1n6CFt1Nv8F/X4B3R0DWQbhtCQx2UnZXUWe1jm9TSj0ALNZaZ9vjgEqpCGAC0AHIAT4Cxlbx0iqHcWit3wbeBjNPwR5lsosdH8CaB6mm2A3TaxIbuj3DG0v2Mv3KOMb2cfEC6qJ6zTuaZpFvXjZDVWPqOI695AJ8MtukqL7lfc9OKRLSyvQXfHwvfP0iBLU0j9vGu7pkoga2DHpuDWxXSu0AFgCf64bNeBsJHNVaZwEopT4GrgTClVJNrLWFGMDOS2k5kNbw/ZsmjbS9r+xC23AidgKPvPYNfWPCeGycjNBwe1f/znQQr/tD3ZavLMiGZXfC8S1m4fd2gxxbTmfwb2aCW/JyM+s57NLlYIV7qTUoaK2fVEr9ERgFzABeV0otB+ZrrX+sxzFTgMFKqWZAATACSAQ2AZOBZcA0YHU99u0a6Tvh1D64/iW7Z0ssKinjgbe+BZB+BE8REGK+1Ff/n+kbsGWmb/YxM6M3+xhMehf6TnF0KZ3Hx9d0QAuPYNMljLVmcNL6UwJEACuUUi/W9YBa662YDuUdwB5rGd4GHgUeUUodASKB+XXdt8vs/MB0qvWZbPdd/3ntAXanWvjb5H60ay6jNDxGv6lmIaIvn649xXZqolny89wp+NV/vCsgCI9Ta1BQSs1WSiUBLwLfAH201vcBA4F6rdSutX5aa91da91ba/0rrfUFrfVPWutBWuvOWuspWusL9dm30xXlw54VJkFYYJhdd71+bwbvfXuMGUPiGNO7tV33LRzMxwfGvmj6Br55ufrXHfjEjMjxDzIds3FDnFdGIapgS00hCpiktR6ttf5Ia10MoLUuA8Y7tHSe4MAncCEXBvzKrrtNOZPP7z9Kpl9MGI+NlfkIHqndIOgzBb55FbKP//I5rc3Snh/+Clr1NkubRnVxTTmFqMSWoLAWOFv+QCkVopS6HEBrfcBRBfMYOz8w67i2t98V3oWSUu5fsgOl4PXb4+23xrJwvpHPmjb1DZWWrywtMUt6fv449BgP0z913ZKTQlzElm+bN4HKjaLnrdvE2Z9MvpsBd9p16OCfPzvAnjQL86ZIP4LHC2trljDdvxqObTH9Cx/eYbKqXvEATHnfLFojhJuwZUiqqjwEVWtdZp2JLHYtMWsO9Lvdbrv8LDmDRd8d556hHRjVS/oRvMKVD8KO92HtH0w6jJN7zGI5g+51dcmEuIQtl7c/WTub/aw/DwE/Obpgbq+s1ASFTiPsllf92OnzPLoymf7twvnDGJmP4DX8mpr1fU/tg9NH4LalEhCE27Lliv83wKvAk5jpuhuBWY4slEf4cRPkppllA+2gsNj0I/j6KF6/fYD0I3ibXjdB/hmIHSxLmwq3ZsvktVOYLKaisp3vQ7NI6DbOLrt7/rP97EvP5d27EoiJkH4Er6OU1A6ER7Al91EgJo11LyCwfLvW+m4Hlsu9nT9jMj8OutcsMNJAn+xO59/fpzDr6o6M7Cl55YUQrmNLG8UHmPxHo4H/YfIS5TmyUG5vz3KTCdMOcxOOnj7PYx/vIT42nN+P7maHwgkhRP3ZEhQ6a63/CJzXWi8Crgcab6Oo1iYjapt4s2h7AxQWl/J/i3fQxFfx+u3x+PlKP4IQwrVs+RYqtt7mKKV6A2FAnMNK5O7Kk9/FN7yW8Nyn+zmQkctLt/SjTbiMVRdCuJ4to4/etq6B8CSwBggG/ljzW7zYzg+gSVPoXa+0TxVW70pjydYUfn1NR67tLv0IQgj3UGNQUEr5ALnWBXa+Bjo6pVTuyk7J737MOsfjH+8hoX0Evxsl/QhCCPdRY/ORNendA04qi/urSH53Z713UVJaxgNLduLfxIfXbh8g/QhCCLdiyzfSBqXU75RS7ZRSzct/HF4yd7TzA4joAHFD672LpdtSOJCRy18m9SE6TPoRhBDuxZY+hfL5CPdX2qZpbE1J5cnvrn3STESqB0tBMS9tOMTgjs0ZLXmNhBBuyJYZzR2cURC3Z4fkd29sOkJOQTFPXt8TVc/AIoQQjmTLjOa7qtqutX7f/sVxU3ZIfnf8zHkWfnOUyfEx9G5r3xXahBDCXmxpPrqs0v1AYARmfeXGExTKk9+N+Uu9dzF33UH8fH34ncxaFkK4MVuajx6s/FgpFYZJfdF4lCe/6zq2Xm/fdvQs6/ae5JHrutIqNLD2NwghhIvUZzxkPtB4FpMtT37X97Z6Jb8rK9P86dP9RIcFcu9VjatvXgjheWzpU/gEM9oITBDpCSx3ZKHcSvKH1uR39Zub8J9daexJs/CPW/vR1N/XzoUTQgj7sqVPYV6l+yXAca11qoPK4352LYa2A+uV/C6/qIQX1/9A35gwJvSzz+psQgjhSLYEhRQgQ2tdCKCUaqqUitNaH3NoydxBcSFk7oVhj9fr7e98fZSTuYW8OnUAPj4yBFUI4f5s6VP4CCir9LjUus375aaZ2/B2dX7rSUshb/3vR8b1ac2gDo1zArgQwvPYEhSaaK2Lyh9Y7zd8uTFPYLG2koXF1Pmt8774gdIyzaNjutu5UEII4Ti2BIUspdSN5Q+UUhOA044rkhsprymE1q0/YG+ahZU7Upk+JI72kUEOKJgQQjiGLX0KvwEWK6Vetz5OBaqc5ex1ymsKdQgKWpshqBHN/Ll/eGcHFUwIIRzDlslrPwKDlVLBgNJaN571mS0nIKgF+Nk+4eyL/ZlsPXqWP03oRVhTPwcWTggh7K/W5iOl1J+VUuFa63Na6zylVIRS6nlnFM7lLGl16k8oKinjL2sP0LllMFMHxTqwYEII4Ri29CmM1VrnlD+wrsI2znFFciOW1Do1Hb3/3TGOncnniet70EQWzxFCeCBbvrl8lVIB5Q+UUk2BgBpeXyOlVDel1K5KP7lKqYeti/dsUEodtt5G1PcYdqG16WgOs204avb5Il7deJirukQxrGsLBxdOCCEcw5ag8G9go1JqplJqJrABWFTfA2qtf9Ba99da9wcGYnIprQLmABu11l2AjdbHrlOYA0XnbG4+Wrb9BLmFJbJWghDCo9nS0fyiUioZGAkoYD3Q3k7HHwH8qLU+bh3qOsy6fRHwFfConY5TdxbrcFQb10/YmZJNx6ggurUOcWChhBDCsWxt+D6JmdV8M+aL/ICdjn8bsNR6v5XWOgPAetuyqjcopWYppRKVUolZWVl2KkYVKiau2dZ8lJxqoU+MLJ4jhPBs1dYUlFJdMV/aU4EzwIeYIanD7XFgpZQ/cCPwWF3ep7V+G3gbICEhQdfy8vrLtX2OwqncQk7mFtI3JtxhxRFCCGeoqfnoILAZuEFrfQRAKfVbOx57LLBDa51pfZyplIrWWmcopaKBU3Y8Vt1ZUsHHD4Jb1frS5FQLAH2lpiCE8HA1NR/djGk22qSUekcpNQLTp2AvU/m56QhgDTDNen8asNqOx6o7SyqERoNP7S1syWkWfBT0ahPqhIIJIYTjVPuNp7VepbW+FeiO6fT9LdBKKfWmUmpUQw6qlGoGXAd8XGnzXOA6pdRh63NzG3KMBrPYPhw1OTWHLi1DaOZvS9YQIYRwX7VeBmutz2utF2utxwMxwC4aOFxUa52vtY7UWlsqbTujtR6hte5ivT3bkGM0mI0T17TW7Em1SNOREMIr1Gnardb6rNb6X1rrax1VILdQVgp56TbNUUjLKeDM+SIJCkIIryC5GKpyLhPKSmwKCnsqOpll5JEQwvNJUKhKxcS12oPC7lQLfr6K7tEyaU0I4fkkKFTFcsLc2lJTSMuhe+tQApr4OrhQQgjheBIUqmLj4jplZVpmMgshvIoEharkpoF/CATW/GV//Gw+eYUl9JOgIITwEhIUqmJJNYnwasl2mpxqlpno01Y6mYUQ3kGCQlUsqTb1JySnWgho4kPXVsFOKJQQQjieBIWq2DhxLTk1h15tQmWVNSGE15Bvs4sVF0D+6VpTXJSWafam5cr8BCGEV5GgcLHcdHNby+I6R06do6C4VGYyCyG8igSFi1UsrlNzn0J5J7PUFIQQ3kSCwsVsDgoWggOa0DEqyAmFEkII55CgcDEbJ64lp1no3TYUHx97LjEhhBCuJUHhYrmpENQSmgRU+5KikjIOpOfST5qOhBBeRoLCxconrtXgUGYeRaVlkt5CCOF1JChczJJWa3/Cbmsns9QUhBDeRoJCZVpbJ67V0sl8wkJEMz9iIpo6qWBCCOEcEhQqK8yB4vO1jzxKs9AnJhxVS24kIYTwNBIUKqsYjlp9n0JBUSmHMvPo21b6E4QQ3keCQmUVK65Vn+Jif0YupWVaZjILIbySBIXKbFhxTWYyCyG8mQSFyiyp4ONn5ilUY0+qhZYhAbQOC3RiwYQQwjkkKFSWmwahbcCn+l/L7tQcaToSQngtCQqV1bK4Tl5hMT+dPi9NR0IIryVBobJaJq7tTctFa2QmsxDCa0lQKFdWam0+qn446p40ayezDEcVQngpCQrlzmWCLq2xprA71ULb8KZEBlefLE8IITyZBIVyNqyjsCfVQr92UksQQngvCQrlagkK2eeLSDmbT5+20skshPBeEhTK1RIU9qRZAOgnncxCCC/mkqCglApXSq1QSh1USh1QSl2hlGqulNqglDpsvY1waqEsqeAfAoFVf+mXz2TuJZ3MQggv5qqawivAeq11d6AfcACYA2zUWncBNlofO09uzcNRk1MtdIwKIqypnxMLJYQQzuX0oKCUCgWuBuYDaK2LtNY5wARgkfVli4CJTi2Y5USN2VGTUy0yP0EI4fVcUVPoCGQBC5VSO5VS7yqlgoBWWusMAOttlQmIlFKzlFKJSqnErKws+5Wqholrp3ILOZlbKDOZhRBezxVBoQkQD7yptR4AnKcOTUVa67e11gla64QWLVrYp0TFBZB/utoV15JTTSez5DwSQng7VwSFVCBVa73V+ngFJkhkKqWiAay3p5xWotx0c1tNTSE5zYKPgl5tQp1WJCGEcAWnBwWt9UnghFKqm3XTCGA/sAaYZt02DVjttEJVrKNQdZ9CcmoOXVqG0My/idOKJIQQruCqb7kHgcVKKX/gJ2AGJkAtV0rNBFKAKU4rTcWKa5fWFLTW7Em1cG336tdYEEIIb+GSoKC13gUkVPHUCGeXBfh54loVyfDScgo4c76Ivu2kk1kI4f1kRjOY5qOgltDk0kR3e8o7mWXSmhCiEZCgADVOXNudasHPV9E9OsTJhRJCCOeToADWFdeq7mTek5ZD99ahBDTxdXKhhBDC+SQoaG2duNbukqeKSspIPmGR+QlCiEZDgkJBNhSfr7KT+b8HM8m7UMLInq1cUDAhhHA+CQq51Q9HXZGUSqvQAK7uYqeZ00II4eYkKFSzjsKpvEI2/ZDFTQNi8PVRLiiYEEI4nwSFaoLC6p3plJZpJg+sPp22EEJ4GwkKllTw8TPzFKy01qxISmVAbDidWwa7sHBCCOFcEhQsqRDaBnx+/lXsSbPwQ2ae1BKEEI2OBIXcS4ejrkhKxb+JD+P7tnFRoYQQwjUkKFw0ce1CSSmrd6UzuldrWXpTCNHoNO6gUFZq1lKo1Mm88cApLAXFTJGmIyFEI9S4g0LeSdClv5i49lHiCVqHBjKkc5QLCyaEEK7RuINCxcQ106dwKreQ/x3KYlJ8W5mbIIRolBp3ULhoxbVVO9Mo08ioIyFEo9XIg8LPKS601nyUlMrA9hF0bCFzE4QQjVMjDwqpEBAKgWHsTrVw5NQ5qSUIIRo1CQrWTuYVSScI9PPh+r7RLi6UEEK4TuMOCrmpEBZDYXEpa3alM6ZXa0IDZW6CEKLxatxBwTpxbcP+THILS5g88NKFdoQQojFpvEGhuADyz0BYDCuSUmkTFsgVnSJdXSohhHCpxhsUrCOPcvxasflwFjcPlHUThBCi8QaFXLOOwlcn/SnTcHO8jDoSQojGGxSsi+usOKK5LC6CuKggFxdICCFcr9EHhW1nmjJFOpiFEAJo5EEht0kkvn6BjJO5CUIIATTioFCac4LjJeGM7d2a4IAmri6OEEK4hUYbFPKzUjhRGsnkBOlgFkKIco0zKGiN//k08vxbMbiDzE0QQohyjTIoZGRmEKAv0KpdZ3xkboIQQlRwSVBQSh1TSu1RSu1SSiVatzVXSm1QSh223kY46vhfb98JQK8ePR11CCGE8EiurCkM11r311onWB/PATZqrbsAG62PHWJU2xIAWrTt5KhDCCGER3Kn5qMJwCLr/UXAREcdKKLklLkT1rbmFwohRCPjqqCggS+UUklKqVnWba201hkA1tuWVb1RKTVLKZWolErMysqq39FD20D38RBU5SGEEKLRctUA/SFa63SlVEtgg1LqoK1v1Fq/DbwNkJCQoOt19O7Xmx8hhBC/4JKagtY63Xp7ClgFDAIylVLRANbbU64omxBCNGZODwpKqSClVEj5fWAUsBdYA0yzvmwasNrZZRNCiMbOFc1HrYBVSqny4y/RWq9XSm0HliulZgIpwBQXlE0IIRo1pwcFrfVPQL8qtp8BRji7PEIIIX7mTkNShRBCuJgEBSGEEBUkKAghhKggQUEIIUQFpXX95n+5A6VUFnD8os1RwGkXFMdRvO18wPvOydvOB7zvnLztfKBh59Rea92iqic8OihURSmVWCnJnsfztvMB7zsnbzsf8L5z8rbzAcedkzQfCSGEqCBBQQghRAVvDApvu7oAduZt5wPed07edj7gfefkbecDDjonr+tTEEIIUX/eWFMQQghRTxIUhBBCVPCaoKCUGqOU+kEpdUQp5bD1nZ1JKXVMKbVHKbVLKZXo6vLUh1JqgVLqlFJqb6VtzZVSG5RSh623Ea4sY11Ucz7PKKXSrJ/TLqXUOFeWsS6UUu2UUpuUUgeUUvuUUg9Zt3vyZ1TdOXnk56SUClRKbVNK7baez7PW7R2UUlutn9GHSil/uxzPG/oUlFK+wCHgOiAV2A5M1Vrvd2nBGkgpdQxI0Fp77KQbpdTVwDngfa11b+u2F4GzWuu51gAeobV+1JXltFU15/MMcE5rPc+VZasP64JW0VrrHdZ1TpIw66NPx3M/o+rO6RY88HNSZp2BIK31OaWUH7AFeAh4BPhYa71MKfUWsFtr/WZDj+ctNYVBwBGt9U9a6yJgGTDBxWUSgNb6a+DsRZsnAIus9xdh/mE9QjXn47G01hla6x3W+3nAAaAtnv0ZVXdOHkkb56wP/aw/GrgWWGHdbrfPyFuCQlvgRKXHqXjwH0ElGvhCKZWklJrl6sLYUSutdQaYf2CgpYvLYw8PKKWSrc1LHtPUUplSKg4YAGzFSz6ji84JPPRzUkr5KqV2YZYp3gD8CORorUusL7Hbd563BAVVxTbPbxeDIVrreGAscL+16UK4nzeBTkB/IAP4u2uLU3dKqWBgJfCw1jrX1eWxhyrOyWM/J611qda6PxCDaRnpUdXL7HEsbwkKqUC7So9jgHQXlcVutNbp1ttTwCrMH4M3yLS2+5a3/55ycXkaRGudaf2nLQPewcM+J2s79Upgsdb6Y+tmj/6MqjonT/+cALTWOcBXwGAgXClVvnqm3b7zvCUobAe6WHvj/YHbgDUuLlODKKWCrJ1kKKWCgFHA3prf5THWANOs96cBq11YlgYr//K0ugkP+pysnZjzgQNa65cqPeWxn1F15+Spn5NSqoVSKtx6vykwEtNPsgmYbH2Z3T4jrxh9BGAdXvYy4Ass0Fq/4OIiNYhSqiOmdgBmLe0lnnhOSqmlwDBMmt9M4GngP8ByIBZIAaZorT2i87aa8xmGaZLQwDHg1+Xt8e5OKTUU2AzsAcqsmx/HtMF76mdU3TlNxQM/J6VUX0xHsi/mQn651vo563fEMqA5sBO4U2t9ocHH85agIIQQouG8pflICCGEHUhQEEIIUUGCghBCiAoSFIQQQlSQoCCEEKKCBAUhaqCUKq2UVXOXPTPwKqXiKmdbFcIdNKn9JUI0agXW9AJCNApSUxCiHqxrXfzVmud+m1Kqs3V7e6XURmvStY1KqVjr9lZKqVXWnPi7lVJXWnflq5R6x5on/wvrjFUhXEaCghA1a3pR89GtlZ7L1VoPAl7HzKbHev99rXVfYDHwqnX7q8D/tNb9gHhgn3V7F+ANrXUvIAe42cHnI0SNZEazEDVQSp3TWgdXsf0YcK3W+idr8rWTWutIpdRpzAIvxdbtGVrrKKVUFhBTOQ2BNa3zBq11F+vjRwE/rfXzjj8zIaomNQUh6k9Xc7+611Slcq6aUqSfT7iYBAUh6u/WSrffWe9/i8nSC3AHZulEgI3AfVCxYEqoswopRF3IVYkQNWtqXfGq3Hqtdfmw1ACl1FbMxdVU67bZwAKl1O+BLGCGdftDwNtKqZmYGsF9mIVehHAr0qcgRD1Y+xQStNanXV0WIexJmo+EEEJUkJqCEEKIClJTEEIIUUGCghBCiAoSFIQQQlSQoCCEEKKCBAUhhBAV/j81TFtNqRae8QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), train_acc_list, label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), valid_acc_list, label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 564,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test ACC: 88.64%\n"
     ]
    }
   ],
   "source": [
    "with torch.set_grad_enabled(False):\n",
    "    test_acc = compute_accuracy(model=model,\n",
    "                           data_loader=dataset_loader[\"test\"],\n",
    "                           device=DEVICE)\n",
    "\n",
    "print(f'Test ACC: {test_acc:.2f}%')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tryit",
   "language": "python",
   "name": "tryit"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}